THE RKGBLOG

Jim Novo: Are You Sure Your Marketing Really Works?

jim novo

Are people responding to your marketing , or are they placing orders and your marketing had nothing to do with it? That’s the critical issue.
— Jim Novo

Jim Novo is one smart guy.

Jim learned direct marketing during his decade at HSN, then he shifted over to online marketing and web analytics.

Jim writes, blogs, and consults.

He even runs a successful pet retailing site on the side, anonymously — just so he has a “lab” to prove his theories.

I’ve been a Novo fan since I read Drilling Down in my cataloging days, and I follow his current blog with interest. (Sample: check out this amazing and far-ranging Novo post, Lab Store: Web Merchandising.)

I recently caught up with Jim by phone to get his thoughts on control groups, the incremental value of email, merchandising strategy, and brand voice.

Listen to podcast: rkgblog_interview_Jim_Novo.mp3


Jim Novo Transcript

Alan Kaufman: This is Alan Rimm Kaufman and I’m here today with Jim Novo.

Jim Novo: Hi, Alan!

Alan Kaufman: Jim, you have a very long and varied history in direct marketing and online marketing. Can you tell us about your past?

Jim Novo: Sure. I started off in cable television, before it was computerized at all. We were working on microfiche. When that industry became computerized, it was an amazing time because all of a sudden we could collect all this information from customer interaction that we’d never been able to keep before.

Real simple things like statusing a call. A customer calls in: well why are they calling? So you could set up reason codes and find out why. I started looking at that information and it was just this huge window into the customer experience.

I was looking at things like why do people want to disconnect their cable TV. One of the things I realized was it was basically because they were confused about the technology. They didn’t understand how to hook up their VCR. They didn’t know how the computerized boxes worked. They had trouble with remote controls. Typical new technology stuff.

Alan Kaufman: Sure.

Jim Novo: People just didn’t get it. We’d leave the technical manual for the converter box with the customer. It’s almost impossible to read. There was no consumer stuff. That’s typical of new technology, right? You get the basic sort of engineering look at the box. You don’t get, “here’s how you hook up your VCR if you have this kind of television set” kind of stuff.

So one of things I did was I created this sort of new customer kit that explained all this stuff to people, diagrams of how to hook up their VCRs, all that kind of stuff.

The installation kits that would go out with the new customer installs were consecutively numbered. So I was thinking, well I can test this. I’ll just put this new customer kit into the installation kits that have even numbers and I won’t put one in the ones that have odd numbers.

Alan Kaufman: Outstanding.

Jim Novo: Then I’ll just look at the customer churn rates out one month, three months, six months.

Lo and behold! I didn’t know it was called a controlled test but I has just done one!

The churn rate for the customers that got all this explanation about how to use their boxes was 19% lower than the people who didn’t get it.

Wow, I thought, that’s pretty amazing.

I mean, you could actually do marketing, you could actually sort of influence the customer experience and it would really drop to the bottom line. You would have financial impact.

So that was kind of the very beginning of me sort of looking at the world like a marketing guy.

Then I left cable and I went to Home Shopping Network and it was pretty much exactly the same thing all over again.

Alan Kaufman: How long were you at HSN?

Jim Novo: For just over 10 years.

Alan Kaufman: That was a huge period of growth for HSN.

Jim Novo: Once again, I was really lucky.

I was there when cable was computerized and of course HSN was built from the ground up on computers. Every transaction in that place was time stamped, which was the first time anybody had ever kind of thought about that. So we had just unbelievable amounts of data.

home shopping network

It was really, really important that things were time stamped because we started to get this whole notion of time and how important time was in terms of customer behavior, how long it had been since they last purchased, and those kind of ideas.

It was the same kind of basic idea all over again: a new technology, so people have questions.

We’d actually get calls from people asking if they could buy.

That’s pretty shocking when you’ve got this entire channel that’s devoted to selling stuff to people and they would call in and say, “Can I buy this?”

You’re thinking, “excuse me, what part of what we’re doing isn’t making sense?” How could someone watch HSN and ask permission to buy? It’ was just the most bizarre thing but it was because it was brand new. It probably sounds weird to other people but when you launch a channel like this and nobody’s ever seen anything like it before people go, “what the hell is this?”

Alan Kaufman: What have you been up to since HSN?

Jim Novo: We did a lot of Internet stuff beginning in 1994 or so at HSN. That was pretty exciting.

I could see that the Web was going to be the next new customer technology interface challenge. When I left

HSN I wrote a book about how the customer experience and marketing and technology lessons that we had learned at HSN had all played out and how you could really measure the impact of changes on customer experience and marketing and show how you could bring profit to the bottom line. That’s what the book is about. Then, from the book came consulting stuff. Here I am 10 years later.

drilling down

The ideas just applied to the Web, it was easy to sort of get involved with the early days of Web analytics.

Alan Kaufman: One of the things that I’ve heard you describe is the similarities between selling over the television and selling through a website. You’ve said that HSN is like Web marketing, only 10 years before. The channel is always on and other marketing effects are pulsing on and off but that the TV is always running. The martker’s challenge is allocating orders to the main core channel versus the extra efforts being used to drive traffic.

You made this tremendous analogy noting that a website is also like that.

The website is always there, always on, picking up orders. Meanwhile there are also pulses of e-mail, affiliate efforts, paid search, and so on. The marketer needs to parcel out the impact of just the website and also determine the impact of all these other pushes. Can you say more about that analogy?

Jim Novo: Sure. I guess that – we actually had a presentation on part of this at the DMA together. The issue is that when you’re talking about response the question is, are people really responding to your marketing or are they simply marching into the site and placing orders and your marketing had really nothing to do with it? That’s the critical issue.

A lot of times people count something as response when it’s not really adding incremental value to the customer base.

So the idea is to use control groups. Take a certain part of the customer base and hold it back and not send them the e-mail or whatever and then measure the net effect of the e-mail, the incremental effect on actual customer purchases and value. What you find is that a lot of customer, particularly best customers are people who would have made the purchase without your marketing anyway. Especially if you’re offering people a discount — a lot of times you’re giving them a discount that you didn’t have to give.

It’s that opportunity cost that most offline like catalogers pay a lot of attention to because they don’t want to be giving up margin when they don’t have to.

That was a really huge issue at HSN because the TV was so powerful that it could create such incredible demand.

So we had to show that the marketing was to be incremental to that.

Alan Kaufman: In Web marketing, many firms are concerned about multi-channel marketing, struggling to understand the allocation of orders across channels.

Too many folks focus on the technical aspect of tracking and figuring out the touches before an order. I believe that event if you had perfect visibility into every marketing touch and there was no technological challenges, for example, even if you could see that the person bought on Monday morning and they had received an e-mail the Friday before and clicked on a couple of search ads a few days prior and received a catalog two weeks prior — even if you had all that perfect data, then you STILL wouldn’t how to allocate credit across those. Which marketing touch or combination of marketing touches really influenced the consumer? Which one if any spurred the sale?

Jim Novo: I would completely agree. This I think in a lot of cases channel tracking is similar to a “time spent” of idea. Also, how much of your time do you really want to spend on trying to be accurate as opposed to be precise?

Alan Kaufman: A lot of people simply take the last touch and give that credit. If you’re mailing catalogs quarterly and sending e-mails weekly, the last touch is almost always going to be an e-mail whether or not the email is what really had the impact.

Jim Novo: That’s for sure. That’s where that whole idea of control groups comes in because that’s really the only way to measure if you’re generating any kind of incrementality.

Alan Kaufman: When we spoke at the DMA together, you had a flamboyant quote about the billions of dollars which e-mail has taken credit for that might not be real. Can you tell that story again for our listeners?

Jim Novo: Well yeah. Email campaigns could be either taking credit for that they shouldn’t have or the flip side of that which is that they’re dramatically under reporting what they should be taking credit for. The reason is exactly what we were just talking about with this idea of control groups and finding out what the incrementality is of different marketing kinds of touch points.

Alan Kaufman: But in particularly with e-mail, do you think that folks’ e-mail tracking reporting is tending to be overestimating or underestimating the true value?

Jim Novo: Well I think it’s pretty accurate in terms of response. In other words somebody clicked on this and clicked on that and so forth.

The question is would they have come to the site and bought anyway if you hadn’t sent the e-mail.

That’s not such a terribly important idea unless you’re doing discounts. If you’re doing discounts then you run into that idea of, “Well he would have bought anyway without the discount.”

And that’s the opportunity right there. That’s the opportunity cost that you have to measure because otherwise you could be misallocating credit for sales and you could misallocating funding in the marketing department.

Alan Kaufman: Certainly in the catalog world, marketers are struggling with the same issue. The million dollar question is “what kind of incremental value are the books generating?”

Jim Novo: Right. At HSN we found – I think it will also be found on the Web — that catalogs can be used to extend the purchasing cycle of a customer.

But you have to figure out which segments of customers and which product lines and so forth did that work for.

So for example at Home Shopping it never worked with jewelry. We could sell tons of jewelry on television and then take our best jewelry customers and mail out catalogs to them and they wouldn’t buy any more than they bought from TV. Those sales were completely non-incremental. The cost of the book was just marketing thrown down the tube because it didn’t increase their sales. Those campaigns were completely cannibalistic to the TV.

gold necklace

What we would do is take customers that were good jewelry buyers and mail half of them a catalog and half of them not a catalog. Then when you looked at what happened afterwards they spent at the same rate.

Alan Kaufman: Regardless of the extra book cost?

Jim Novo: Yes.

So you’re completely upside down in the book cost, and then if you offer a discount on the book then you’re out that, too.

Alan Kaufman: Even worse.

Jim Novo: But what we did find was that for doll buyers and other different segments of customers the books were incredibly incremental. They would drive all kinds of additional sales to TV.

collectible dolls

One of the strategies that we used was we said to ourselves, “So we’ve got this new product on television. Why don’t we put new product on television and old product in the books?”

So the catalogs sort of became an adjunct – it expanded the opportunity for customers to buy dolls rather than replicate the opportunity to buy dolls and that’s where you saw the incremental sales.

Alan Kaufman: That’s really neat. Earlier, you talked about the time element in customer behavior. Can you go more into that?

Jim Novo: Yes.

People tend to do marketing on sort of a calendar type basis, monthly, weekly and so forth.

What we found was that that was really not nearly as powerful as matching up marketing to where the customer was in their relationship with you, how long it had been since they’d last interacted with you and so forth.

If you gave people a message that acknowledged that sort of state of the relationship, “Well we haven’t seen you for awhile,” or, “You haven’t bought anything for awhile and we miss you,” or that kind of thing (I’m not saying you have to use those words) — it’s treating people differently as their relationship changes with you rather than dumping the same stuff to people on a weekly or a monthly basis. It generates much longer relationships and much more profitable relationships to do it that way.

Alan Kaufman: So relationship-based rather than calendar-based?

Jim Novo: Yes. In the ‘90’s it was called Relationship Marketing. There were books written on it by Regis McKenna and other people and that was really the practice that we started to follow because it had been successful in a lot of B2B kind of scenarios.

So we were thinking, “Well why wouldn’t this work for B2C?”

It does.

It works so much better than trying to do a calendar based kind of thing.

You have some of the e-mail companies now are starting to look at this kind of life cycle marketing kind of stuff and some people are incredibly successful with it. It’s more effort but it makes you more money.

Alan Kaufman: A slightly different topic. You do consulting now, but you are also still a retailer. You’re running an anonymous “lab store” which you describe as specialty pet niche retailing site run by your wife.

Jim Novo: Yes. Well back in ’98 when I left Home Shopping, I had all this knowledge and I wrote the book and stuff but people weren’t really all that interested. It was kind of like everybody was doing IPOs and there was so much money around nobody really cared. Remember?

So I thought to myself, “Well what if I just do my own store then I can prove how all this stuff works using my own store and using the data from my own store and that way I’ll have a lot of examples and so forth to show people.”

So the thought was, well it would build some credibility and maybe it would even make a little money.

So we implemented all of this stuff, all the customer experience stuff, all the marketing strategy and merchandising and all of those things that we had learned at Home Shopping on this website. It eventually got so big that my wife could quit her job.

So it’s really quite a fun store. I do the marketing and the site, I taught myself some HTML. It’s not a real complicated site or anything. It doesn’t have a lot of wuzzy, fuzzy stuff because I’m not a technical guy.

But it works.

That’s why customers like it — because it’s really simple, easy to order.

We do all kinds of customer experience stuff, new customer kits and all kinds of things like that that really drive their performance. But the interesting thing about it is it’s really about the productivity.

Instead of, “Let’s grow as big as we can possibly grow,” the question is, “How much money can one person make by spending the least time they possibly can doing the business?”

So it’s all about productivity.

It’s not about how big can it be.

It’s, “If we spend $1.00 can we make $5.00 and then how do we make $6.00 and how do we make $7.00 for spending that same dollar?”

Alan Kaufman: You’ve blogged about how you’ve used testing to show what works best for that site.

In some cases, results for your site contradict commonly accepted best practices.

For example, you’ve described in your blog removing the site search box from your small specialty pet site.

Jim Novo:
Yes.

Maybe it’s just because of the fact that there’s so many technically oriented people involved in web marketing is that everything is very tactical and everybody is always looking for best practices and that kind of thing.

It strikes me that a lot of what’s done on the Web lacks a strategic view that is broader than just the mechanical “you get a response when you do this and that” thinking. There really has to be a strategy of some kind that then unfolds through the entire thing.

So the example that you just brought up was search.

Site search is great when you’ve got tons and tons of products and there’s really no good way to do navigation.

In some ways search is just a substitute for poor navigation, if you know what I mean. It’s like, “Well we can’t get the navigation right so let’s put a search box type of thing.”

When we designed the navigation on our site and we thought through the product and we thought about bundling and we thought about all the things that you need to think about when you put up a high productivity retail site because it’s a strategic stuff that I’m talking about.

We intentionally designed the navigation based on our product bundling. The idea is that if the navigation was really clear what would happen is people would be exposed to more products and be more likely to buy larger order if we kind of had them go through the process of looking at all the items by going through a really flat hierarchy or really good navigational experience as opposed to a search box.

That, combined with the fact that we offer flat rate shipping (which means that no matter how many items you order your shipping cost is going to be the same) tremendously drives up higher average orders.

Alan Kaufman: Sure.

Jim Novo: Because people would be exposed to stuff and they’re like, “Wow, I didn’t know the store had that and I didn’t know this product even existed,” because we invent stuff for the site all the time.

We invent our own products.

It’s like, “Who would’ve searched for a product that doesn’t exist anywhere else?” Do you know what I’m saying? How would they search for that?

So by driving them through navigation that’s really well designed we figured we would drive up the average order. So of course I know I have to test it, right?

So we put up a search box and low and behold while that search box is on the site the average order fell by like 20%. So what was happening was people were just going to the search box and saying, “I know I want this,” and searching for it and finding it and adding it to the cart and checking out and you’re missing that opportunity for people to add other things.

I think this whole idea of serendipity which is discovering things you didn’t know exist is a real issue for the Web because everything is so search based.


Alan Kaufman: Sure.

Jim Novo: People don’t get exposed to a lot of stuff because they’re always searching for what they want —

Alan Kaufman: They know what they want already.

Jim Novo: -Right — as opposed to what they never knew about. That search only gives you what you already know you want is a real sort of macro problem for the Web, I think.

Alan Kaufman: StumbleUpon versus Google.

Jim Novo: Yes, absolutely.

Alan Kaufman: Another fascinating thing you described was you have a very narrow merchandise selection on this site.

Jim Novo: Right.

Alan Kaufman: Chris Anderson has written about the long tail and the advantages of carrying every single SKU in your category, the NetFlix model.

But you tested and on your own niche site you came up with some different results.

Jim Novo: Well I think it goes back to strategy again. The strategy is to be the most productive that we possibly can, spend the least amount of time and make the most money in that store. If you’re talking about Amazon that’s not their model.

Alan Kaufman: The Amazon model is “comprehensive everything.”

Jim Novo: Yes. Amazon is about comprehensive everything and they don’t really care about the productivity of it. Amazon is trying to grow bigger and bigger and bigger. They’re not trying to become more and more and more profitable necessarily. That is the Amazon strategy.

One of the things that I learned at Home Shopping because the way that business model works is that you have this thing called “air time” and air time is incredibly precious because there’s only 24 hours a day of it. Do you know what I’m saying?

Alan Kaufman: “Time and tide waits for no man”.

Jim Novo: Absolutely, there’s no more time in the day than 24 hours. So you’re always trying to make the air time more productive. You’re always trying to do higher profit per minute and it’s all about profit per minutes, profit per minute, profit per minute.

I guess that’s where I got this notion of productivity.

So the guy who invented Home Shopping, Bud Paxon, used to say that “there’s always someone in America somewhere who wants to buy a telephone shaped like a tomato.”

If you think about that, one of the things that used to happen is that we would cycle an awful lot of different kinds of product through there and we would basically respond to what the customer said about the product through their ordering. So if product went up on the air and it had a really high productivity rate in terms of profit per minute then we would keep putting it up there until they bought as much as they wanted. Do you know what I’m saying?

HSN

Alan Kaufman: Sure.

Jim Novo: In other words, sell them what they want to buy as opposed to having this thing and trying to sell it to them when they don’t really want to buy it. So that was the whole idea of it.

So the idea was to always sell what people wanted to buy rather than trying to reverse engineer it and convince them to buy something that they weren’t sure that they wanted to buy.

So that’s the philosophy we have in the lab store.

We essentially carry a narrow selection but all of those products sell really, really well.

They’re all high demand product and it’s a core customer belief kind of thing.

And if customers want to go out and expand on the products that we offer by buying somewhere else, that’s fine. They can do that. I don’t have a problem with that really.

I don’t want to hold 5,000 items just so that I can give people the opportunity to buy one of them once a year. Do you know what I’m saying? I want to sell everything I have in the store every single day and that’s the difference between the strategy.

Alan Kaufman: Remind of your inventory terms of the store?

Jim Novo: It is between 20 and 21 turns.

Alan Kaufman: That’s amazing.

Jim Novo:Where in a typical offline store it’s more like six or eight turns or something like that. We are around three times more —

Alan Kaufman: –productive than the typical store. Wow.

When you say “productivity” are you talking about margin per sales dollar, or margin per minute of time you put into the store?

Jim Novo: Margin per dollar spent. So it would be not only the time we put into it but obviously the expense that we put into it.

It’s fine to have a long tail. I’m not saying the long tail is wrong – I just want to be clear — I’m not saying there’s anything wrong with that strategy. I’m just saying there are other strategies out there.

Alan Kaufman: Sure. Search boxes aren’t bad. Long tail merchandising isn’t necessarily bad. What can be bad is doing something without a coherent strategy, without thinking about what you’re doing it and why.

Jim Novo: It just goes back to the conversation we had just a minute ago about the Web being so tactical. People aren’t thinking often strategically, aren’t thinking at a high enough level. They’re just saying, “Well this guy did this and that worked. So it’s going to work for me.”

Alan Kaufman: Then everyone thinks they have to do the same thing.

Jim Novo: Right. It depends on context. Search boxes are just absolutely mandatory in a long tail business. The question is, “Do you want to be in a long tail business?” The question is not, “Do you want to have a search box or not?”

The question is, “Do I want to be in a long tail business,” because that has all kinds of ramifications and implications in terms of out of stock for example.

You can be in a huge long tail business but how many times are you out of stock on the order?

We are never out of stock. We don’t even have any kind of inventory management type of thing that plugs into the website and says, “Well this item’s out of stock.”

We are never out of stock on anything ever.

That’s just part of our strategy. People always know they can always get it with us and they’re never going to have to wait. They’re going to receive it in three to five days, always. That’s just part of the business philosophy.

Alan Kaufman: You’re not achieving that by keeping piles of excess inventory, not at 21 turns.

Jim Novo: Absolutely. You’ve got to have some place to store all that stuff, right?

Alan Kaufman: So you’re managing to stay in stock always by being narrowly focused, and still keep the inventory churning.

Jim Novo: Right. I’m sure we give up revenue by not having a broad selection. One of our major competitors offers about 25 different kinds of pet toys. We offer the five toys that everybody wants. It’s just a different strategy. That means that we can buy in tremendous bulk.

That’s the other thing in retail anyway. What happens is if you’re buying stuff onesie, twosie so that you can have a long tail business you’re paying premium price on it. We buy in bulk. We go to these pet shows and they’ll be mostly retail store owners purchasing there and they’re coming and they place their order for 5 of the unit and we come in and place the order for 200 of the unit. So we get tremendous efficiencies that way. Not only in your cost of product because you’re buying it at a huge discount versus everybody else but also just in operational efficiency in terms of getting the product in and so forth.

It really makes a big difference. All of it has to be plugged together as a strategy. You have to first say to yourself, “we’re going to be like HSN, we’re going to operate on a productivity model” as opposed to a long tail model or a gross sales model or whatever model you want to talk about.

Alan Kaufman: Can you give us your most important tips for online retailers as we move into 2008?

Jim Novo: Sure. I think we’ve probably touched on a couple of them. I think for people interested in the strategic aspect of the Web — I really think it’s going to be this idea of voice. The voice of the site, the voice of your customer experience, the personality that you give yourself which some people would say is our brand. For some people that would equate to brand.

I think of it more as personality because I like it to be more of a two-way exchange and involve the customer experience as part of that rather than just sort of a statement of who we are. It’s about being who you are.

On our pet site, for example, all the communications, all of our customer communications come from our site mascot. Our animal mascot is a voice that we use and it’s a cartoon figure that goes along with it and this voice speaks on the website and speaks in all the customer service e-mails. Every year this figure, this brand voice gets Christmas cards from our customers, and our character also gets valentine day cards from our customers, people send it their pet pictures and all that kind of stuff.

christmas card

Do you see what I mean? That’s a real voice. That’s a real personality.

When that’s happening it clear to you that the customer is engaged in that.

You don’t send Christmas cards to a company mascot unless you’re engaged in the whole concept of the idea.

So what happens is just about everything you do in marketing and service can flow from that voice, whatever you voice is.

Some voices are going to be high end and, “We provide great service,” and other voices are going to be, “We’re the cheapest you’re going to find anywhere,” and whatever it is.

It doesn’t have to be one or the other but it has to be clear what it is, because everything else flows from that.

Alan Kaufman: The clear and consistent voice and personality of the online retailer’s brand.

Jim Novo: Yes.

My other tips for 2009?

On the tactical level, I would say it’s this idea of control groups.

I think people should start messing around with this. They might not actually implement it in any kind of wide way at the job that they’re at but in their next job it’s a really big deal.

If you can walk in to your boss at your next job and say, “You guys aren’t using control groups? We ought to really start this and see what kind of profit that we’re really generating with these different marketing things.”

In your current job it might be hard to stand up and say, “Oh by the way, we haven’t been measuring this correctly”, but maybe you could in your next job.

Alan Kaufman: “Hey, everything we’ve been doing is wrong!”

Jim Novo: That’s not going to work too well –-

Alan Kaufman: – in many places that might get you the next job more quickly!

Jim Novo: -But even without changing jobs there are ways to mess around with control groups and just get some experience and just try it a couple of times.

Hold out 200 customers, an insignificant number and then do your mailing and then compare the sales generated, the average sale per customer generated from people who would get the e-mail versus the people who don’t.

Just start messing around with controls. Track those people over time. Because what you may find, and I think it’s pretty much certain you will find, is that you actually generate more sales from your e-mail than you think you do, because people respond in lots of different ways over time, not necessarily through the marketing and / or tracking device, or whatever it is you sent with the e-mail. So what you’re not measuring is all the people who responded in other ways because of your e-mail, but not through your e-mail, if you know what I mean. Holding that group out and tracking it over time you can really find out some amazing stuff that you really don’t know right now.

Alan Kaufman: Jim, thanks for the conversation. To summarize your two tips for 2008: have a consistent brand voice and personality, and use hold-outs and controls.

Jim Novo: That’s right.

Alan Kaufman: You’ve been more than generous with your time today Jim. If folks wanted to find you online where should they go?

Jim Novo: It’s JimNovo.com is the website and Blog.JimNovo.com is the blog.


Listen to podcast: rkgblog_interview_Jim_Novo.mp3

Technorati Tags:

  • Alan Rimm-Kaufman
    Alan Rimm-Kaufman founded the Rimm-Kaufman Group...
  • Comments
    One Response to “Jim Novo: Are You Sure Your Marketing Really Works?”
    1. Jim Novo says:

      Thanks for posting this, Alan!

      One clarification I should include before I get slaughtered by the statistical crowd:

      When I said “hold out 200 customers”, I should have stated which segment I was thinking about — Active Best Customers — because that is where the effects are most pronounced, as we discussed earlier in the piece.

      If you don’t do a lot of segmentation or have a gigantic list, holding back 200 won’t show you much.

      So what I probably should have said was “1/4 percent of any segment” so the idea would scale from the smallest indie retailer up to Amazon regardless of their approach to e-mail.

      The idea of “Begin messing around with controls” is not meant to be a statistical test: it’s a way to explore and understand the behavior so the correct marketing approach can be designed *before* a valid test is launched.