Almost any question can be answered, cheaply, quickly and finally, by a test campaign. And that’s the way to answer them—not by arguments around a table. Go to the court of last resort—the buyers of your product.

—Claude Hopkins, Scientific Advertising, 1923

Savvy catalogers have long used testing to improve their mail businesses. As the web matures, catalogers are bringing the same discipline to their online marketing efforts.

This article offers 10 tips for running direct marketing tests in the online world. The first 7 are common to online and offline. The last 3 are unique to online marketing, for they exploit the web’s high speed and low cost.

Tip #1: Move Bigger Levers First: List, Offer, Creative

In descending importance, the three essential elements of a direct marketing campaign are “list” (who, and how many, people receive the offer), “offer” (what merchandise you offer those people, at what price, and with what service), and “creative” (how is the merchandise presented, described, and displayed.)

When testing, always move the bigger levers first.

If you goal is to double online sales, your best bet is doubling qualified traffic to your site. This is generally easier than doubling average order value or doubling site conversion. (Worthy goals, too, but harder to achieve.)

If you’ve not yet tested paid search, paid inclusion, local search, affiliates, Ebay, Amazon, and so on—get out and do so. Such “list” tests offer you the greatest chance of really bumping sales.

After “list”, focus on “offer.” Is your site presenting the right merchandise at the right prices? What about shipping fees: would the conversion lift from a free free shipping offset the cost? Suggestion: when setting a minimum order size for an offer, place it above your average order size.

Finally, focus on your creative—how your site looks and works. Does your homepage highlight the breadth of your merchandise? Are your product detail pages clear, with relevant information above the fold (visible without scrolling)? Is your checkout processes smooth, fast, and intuitive?

Tip #2: Test Shouts, Not Whispers

Testing takes effort, attention, and sometimes money.

Don’t test tiny tweaks.

Favor bold tests that have the potential to really change your business. Suggestion: a sure sign of a bold test is that it may make some insiders slightly uncomfortable.

Subtle tests will, at best, yield subtle results, often too small to detect.

Tip #3: Keep Test Notebooks

For each test, document what you tested, why, and what happened. Short pre-test and post-test summaries keep you from repeating mistakes or wasting time on questions already answered.

Before the test, write down a clear hypothesis of what you’re trying to prove or disprove. Here’s an example:

Test #6, October 2005: Our hypothesis is that bringing visitors into our site from paid search to the new simplified product page template will increase conversions relative to the current grid-style product page template.

Before the test, also record your decision metrics and the roll-out plans.

If the new pages increase closing by a significant amount, that is 50 more orders than the control for the week, we’ll discard the grid template in favor of the simpler template.

After the test, record your numeric results, your interpretation, and suggestions for next steps.

Results: Despite one large order, the simple treatment stunk, actually reducing conversion a bit. Next steps: keep the grid, test another challenger later this month.

Given the value of test notebooks—indeed, they become as your marketing department’s shared institutional memory—its worth maintaining two copies.

Tip #4: Test One Thing At A Time

To isolate the effect of a variable, traditional testing mandates changing a single factor at a time.

Testing online is usually cheaper and faster than a traditional in-the-mail test. Because of this, there’s less need to cram everything into one massive test. Start with a series of fast, simple one-factor tests—you’ll quickly learn what matters.

More advanced marketing teams should look into multivariate testing (also known as MVT, scientific testing design of experiments, or Taguchi testing). MVT offers marketers the chance to vary many factors at once in a statistically valid way.

MVT test design and analysis require more skill—a little training for your team from a seminar or statistical consultant goes a long way here.

Tip #5: Separate Signal From Noise

All tests have some element of random statistical noise. Suppose you took a mailing list of 10,000 people, randomly split it into two cells of 5000 people each, and on the same day mailed each cell the same catalog. Even with exactly the same treatment to both groups, one cell by chance alone will have a few more orders, and thus a higher response rate.

Be sure you can distinguish marketing signal from marketplace noise. Familiarize yourself with basic statistical significance calculations. As a very rough rule of thumb, if you plot conversion rates over time, a test needs to increase conversion by more than 1.5 times the normal range of variability to be significant.

If your team isn’t using these statistical significance formulas yet, you can get training from the DMA (http://www.the-dma.org/seminars/statistics/), or we’ve put a Excel spreadsheet with these basic formulas on our website (http://www.rimmkaufman.com/statistics)

Tip #6: Assign Unique Tracking Codes

Online testing without proper tracking is like flying an airplane blindfolded.

To be able to read your tests, each cell needs a unique tracking code. Make sure your tracking application can report key metrics for each code—visits, sales, and ad expense.

Make sure your tracking application allows you as many keycodes as you need.

Some marketers embed meaning in the various digits of the code, where a code like “0510E02″ might mean the second (”02″) email test (”E”) sent in October, 2005 (”0510″).

Other marketers use sequential numbers for their cells, and log the meaning of each code in a spreadsheet or database.

In either case, make sure your team maintains scrupulous electronic records matching keycodes to their corresponding cells and campaigns, with accompanying information regarding the creative and offer. Many a test has been lost to keycode mismanagement.

Tip #7: Manage Expectations

Many ideas when tested will prove to be duds. Be patient and try again. Make sure your team understands that most tests produce null results. Because your current marketing approach represents years or decades of thoughtful improvement, many alternatives won’t test out better. And the more successes your testing program obtains, the harder it becomes to move the needle.

Should the difficulty of hitting home runs stop you from stepping up to the plate?

Not at all. While testing wins may be scarce, the big wins than can follow more efficient marketing are well worth the effort.

Tip # 8: Online vs. Offline Testing: A Difference in Kind, Not Degree

In the offline world, versioning is expensive and often limited. You face physical limits on what you can vary due to printing and bindery constraints.

In the online world, versioning is usually inexpensive, often free.

Suppose you’re testing an holiday sales email. You might have six decent ideas for the subject line, two possibilities for the large seasonal image, and three reasonable candidates for the lead “hero” product. Considering all combinations of subject, image, and product, you could send 36 different email versions (6 x 2 x 3).

Which version will prove most effective?

If your testing framework can support it, and if the creative can be built in a “cookie-cutter” format, then send them all.

This brute-force “shotgun” approach can be surprisingly effective at finding winners.

(Note this isn’t multivariate testing described in #4 above—MVT would attempt to test all 36 version with only 12 or so cells.)

Tip #9: Testing Within, Not Between

In the offline world, tests are typically slow events, requiring time to prepare, launch, and analyze.

With online testing, results come immediately, often within hours. Smart marketers can exploit this early information to test within a campaign, rather than just between campaigns.

Here’s how.

Suppose you mail a monthly email to your house file. Getting the subject line right can have a large impact on open and conversion rates.

Many catalogers test multiple subject lines on each mailing. Then, after the campaign, the marketing team gathers over coffee to review results by subject line, learning what worked best, so as to improve future emails.

Instead, pull off a random 10% of your email file to mail first. Randomly nth these emails across the various subject line ideas, and mail them. Wait six hours. Evaluate the subject line ideas using their 6-hour open rates for initial blast. (You’d prefer to use conversion rates, but for many catalogers six hours is likely too short to see meaningful sales.) Then—and here’s the trick—mail the remaining 90% of your file using the winning subject line from the initial blast.

This allows you to reap the benefits of sending the better email immediately, not down the road in a future email.

Tip #10: Automated Testing

In the offline world, a test is a discrete marketing effort. Each test takes careful planning and execution by your marketing team. Testing is anything but automatic.

The online world is beginning to offer an exciting new opportunity: automated test robots.

Automated testing may not yet be supported by your e-commerce platform or any of your advertising partners. Within 12 months, they will.

Here’s how automated testing works. You load up a set of different marketing messages (be these ads on an advertising network, or featured products on your homepage) and instruct the platform as to which metric you’re seeking to optimize (such as click-through rate, clicks, or sales). The platform then rotates through the set of messages, tracking which performs best. As the platform learns which message performs best, it preferentially serves the winner, improving your results.

Early examples of automated test robots include Google’s “automatically optimize ad serving for my ads” adwords option (http://www.adwords.google.com), and Offermatica’s “mbox” platform (http://www.offermatica.com).

Online Testing Gives Insights

Claude Hopkins’ praise for testing rings as true today as when written 80 years ago.

May your online tests surprise you, bringing you fresh insights to grow your business!

Alan Rimm-Kaufman, PhD, leads the Rimm-Kaufman Group, a direct marketing service and consulting firm helping catalogers with online marketing and paid search. Alan may be reached via his website at http://www.rimmkaufman.com.

Direct Marketing Significance Calculator