Joel had a lousy experience at the Starbucks on 58th Street and 8th in NYC and he wrote a column about it for Inc. magazine.

Joel’s gripe was with an angry headset-wearing employee, a Starbucks “order expediter”. From research on starbucksgossip.typepad.com, Joel discovered that

Expediters are not really there to see to it that a customer’s order is filled more quickly, they believe. Rather, expediters exist solely to prevent people in line from giving up and wandering off, maybe to go to the Dunkin’ Donuts around the corner. Once a customer places an order, the logic goes, he or she feels an ethical obligation to wait for it to be filled, no matter how long the process takes. Expediters are there to lock in that order as soon as possible.

Joel continues (emphasis and eliding mine)

All of this fancy optimization stuff is called Operations Research. […]

If you’re planning to expand your business to a certain scale, you must first establish procedures and build systems to get predictable outcomes […] It’s a real academic field of study, and it’s really hard and really important. You need to hire pretty smart people to do studies and experiments and collect the statistics and then figure out what it all means. […]

Starbucks is great at operations research. […]

But as it has grown, Starbucks seems to have lost its knack for figuring out whether policies dreamed up at HQ are really going to work in the field.

Systems need to be flexible, and [managers] need to be wary of procedures that, applied blindly, can cross the line into something that looks more like antagonism […]

We’ll get to the PPC bidding angle soon, but first some background on Operations Research, aka “OR”. I earned a PhD in OR at MIT, and so feel obligated to wade in.

OR is a quirky discipline. It was born in Britain during WWII to solve military logistics problems. OR is an odd brew of optimization techniques, statistics, probability, and mathematical modeling.

To its credit, OR helped the Allies win the war by providing efficient plans for shuttling men and materiel across the Atlantic. OR helps cities place fire and police stations in smart locations. OR helps manufacturers run factories more efficiently. And OR helps UPS get packages delivered on time. You can read more OR success stories here and here.

To its discredit, OR is why everyone on your last flight paid a different price, why Long-Term Capital Management nearly blew up the US financial markets in 1999, and why Joel had a bad experience at Starbucks in Manhattan.

You see, optimization algorithms love going to extremes.

The machines think, “if more is better, than most is best.”

But in the real world, extremism often isn’t optimal.

Take an agricultural production planning model. If, say, corn turns out to marginally more profitable than soybeans, the computer will suggest planting every single darned acre in corn, because that solution ekes out a bit more profit.

A smart farmer will hedge her bets and back off from the extreme solution a bit, because the machine’s recommendation is too brittle. She’ll plant mostly corn, but she’ll also put in a few acres of soy and even some wheat, just in case, computer be darned. Smart humans see the larger picture.

You can steer optimization algorithms away from extreme points by penalizing extremities in the objective function or by excluding them from the feasible region. But that revised problem will have its own extremes, and most algorithms will march over to the best corner or the top of the highest peak and happily camp there.

When LTCM imploded, they had thousands of trades on their books, but all were highly correlated — the machines led the traders to place effectively the same bet over and over again.

Extremism. Brittleness. Inflexibility. These are the Achilles’ heel of mathematical optimization.

Optimization algorithms are fantastic tenchologies — don’t get me wrong. Optimization can perform amazing feats. The complexity of modern life requires optimization in many areas. But you can’t just hand the keys over to the machines and go home.

OK, back to pay-per-click bidding.

At RKG, we use a portfolio optimization bidding algorithm to bid millions of ads for our clients, and do so to maximize client profitability. We need machines and we need algorithms and we need APIs, because at scale the PPC bidding problem is far too huge and far too hard to attempt manually.

By default, our system applies two rules when bidding client ads:

1. ECON RULE: Bid ads to their economic efficiency target. Optimize bids based on resulting sales and the client’s economic objectives.
2. SANITY RULE: Keep bids within sane levels.

The ECON rules are the heart of RKG’s bidding system. They are our company’s “secret sauce”, and are far more sophisticated than the short description above suggests. Our clients can instruct us to bid to their choices of economic targets, be those ad-to-sales ratios, return-on-ad-spends, estimated margins, actual margins, net actual margins, what have you.

The real magic of our system is how we handle the many low-traffic ads which, considered alone, have insufficient data to determine good bids. That’s the portfolio optimization piece. Our algorithms use smart clustering and aggregation to bid wisely. Our algorithm is backed by significant math and stats, many many many lines of code, ongoing testing, and years of experience. We’re continually working on improving it, both internally and via formal collaborations with stats profs at Santa Clara, Rice, and Rochester universities.

The SANITY rules are just governors. They take the form, “Never bid more than ‘X’ or less than ‘Y’ per click, regardless of economics”. These overrides keep bids sane. Suppose a particularly lucky ad experienced, over some time frame, two clicks and generated one $2000 sale. Suppose that client instructed us to spend 25% of tracked revenue on advertising. With sales-per-click at one thousand bucks ($2000 AOV x 1 order / 2 clicks), the client’s 4-to-1 sales-to-advertising ratio target would suggest we could bid up to $250 per click.

Sweeping variability and time effects under the rug, the raw economics suggest we could bid up to $250. But we sure as heck never would bid that high — for most retailers going above a few bucks CPC just isn’t prudent.

But these two types of rules alone aren’t enough. Our system provides a salad bar of additional rule types we can use if needed to better serve our clients.

Retail is fundamentally a human experience. Retail is about real people trading money and goods between themselves. And like all human endeavors, retail (like online advertising as a whole) is messy. By that I mean retailing is full of special cases and exceptions.

A retailer is ill-served by an inflexible or brittle bidding system, one that seizes the keys and sends the merchants home. A strong PPC bid platform rests on solid economic foundations, but also accommodates the messy realities of real retail.

What do I mean by messy realities?

• “Ski season starts soon. Last year our telemark gear sold like gangbusters, but recall we had a killer sale on that whole category in ‘07, so this year those terms will likely perform differently.”
• “The CEO wants our ad to appear on page one on Google for the term ‘digital camera’, always, regardless of it’s performance.”
• “Push our home and garden products harder this week to support a national store advertising effort, regardless of how they’re selling on the site.”
• “We want to apply different ROAS targets to different parts of our advertising campaigns, reflecting different margin percentages on different product lines.”
• “Make sure we don’t advertise products in states we can’t sell them, like lead solder into California.”

In addition our basic economic-based bidding, our bid platform allows our client service analysts to layer on additional rules to modify or override certain bids on certain ads which meet certain conditions during certain time periods so as to meet client objectives. Most bids don’t require tinkering. But, when they do, we’ve built tremendous flexibility into our bidding system to allow it.

(An aside: the term “rules-based bidding” has a negative connotation in paid search, evoking the “keep me in position five regardless” approach popular with some firms in recent years. We at RKG typically avoid the word “rules” when describing our technology in our marketing pamphlets for this reason.)

Some bid platforms treat PPC bidding as a black box, “set-and-forget” math optimization problem. We’d suggest online advertisers are better served by more flexible approaches.

We believe the best bidding approach is a hybrid one, smart machines teamed with smart people.

Computers handle what computers do best: large-scale stats, brute-force arithmetic, and executing strategy.

People handle what people do best: special cases, overrides, sanity checks, and developing the strategy.

As Joel experienced in the 58th Street Starbucks, going to extremes can lead to suboptimal outcomes.

Kudos to Baron Schwartz (RKG alum), on his just-published book, the definitive tome for high performance mysql.

Many congrats to the rocking RKG softball team, who won last night, and so ended the season in first place.

And the president is coming to town tomorrow.

From all of us here at RKG, we wish our readers a safe and happy 4th.

RKGDuck slides a snippet of perl (or a big honking mass of perl, should you so need) into the Windows cut-and-paste clipboard buffer. This lets you do powerful text transformations within and between all Windows apps, using the familiar Control-C (copy) and Control-V (paste) mechanism.

After cajoling your local perl geek to write a tiny filter for your task, you can share that functionality easily with less-technical users, and you don’t have to install perl on the end-user machines.

Here’s a new RKGDuck video showing how in a couple minutes you can write a powerful filter to help clean up keyword lists.

You can watch other how-to videos and download the app itself on the RkGDuck homepage.

RKG provides RKGDuck free to the community under GNU General Public License. RKGDuck us distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantibility or fitness for a particular purpose.

If you’ve found RKGDuck helpful and can share your experience, or if you’ve hit any snags, do comment below.

Quack!

YAPC::NA 2008 was a wonderful geek conference which occurred last week in Chicago. RKG strongly supports the open-source movement, and we do our best to give something back to the community through sponsorships and code.

Kudos to the other fine organizations who chipped in:

Google, O’Reilly, ITA Software, Summersault, Grant Street Group, Deutsche Börse Group, CheetahMail, Rakudo, OpenMake, Rimm-Kaufman Group, Active State, Knowmad, WhitePages, Apress , PackT Publishing, Pragmatic Bookshelf, No Starch Press, PlainBlack, CarSpot, Where2GetIt, VirtualMin, Allant, CBS Sports, DRW Trading, Free Software Magazine

Open source code is typically high-quality, lets you avoid licensing fees, and most important, you can lift the hood and fix any bits that come loose.

RKG is among the many many companies that owe a large debt of thanks to the open source movement.

But, of the many small businesses using open source, only a small fraction give back to the community.

One way to support open source is donate to the organizations and conferences.

Again this year, RKG is making a donation to help the good folks at YAPC::NA2008, and urge others using perl to do the same.

Another way is to contribute code.

Bug fixes. New features. New applications. What have you.

If your company uses open source, and you improve the code, seriously consider sharing it back.

While we don’t release the proprietary code which provides RKG and our clients with unique technological and/or business advantages, we do share back “housekeeping” code.

For example, for both RKG and for the community, Baron Schwartz developed table archiving nibblers, table synchronization helpers, server monitoring scripts, and other useful mysql tools. He bundled these apps as MaatKit and released them under the GNU General Public License.

What a honor for Baron that last week at the MySql conference he was given a “MySql Community Member Of The Year” award. Kudos, Baron!

It doesn’t matter if your donation is large or small. It doesn’t matter if you give money or code.

What does matter is this: if you’re benefiting from the Open Source Movement, try to give something back.

It makes good business sense. And it is the right thing to do.

Bidding and the analysis that goes into fine tuning a flexible bidding system are critically important to a PPC program and are often neglected. Folks tend to spend too much time flipping copy, and not enough time studying the numbers. There is much more money to be made with the latter than with the former. I used to think the issue was simply not knowing what to with the numbers, but now I realize that some systems don’t really give folks access to the raw numbers needed for meaningful analysis.

In any analysis, step 1 is “Gather the raw data”. What happens if that data is hard to access, takes time and requires lots of manual manipulation to bring together in a useful format? The answer is obvious: not much analysis is going to take place.

Yet a number of professional SEM firms are in exactly this boat. Many SEM systems don’t use the atomic (ad level) cost and click data from the engines. They use estimates, and periodically do data pulls from the engines for reports back to the client.

This came as a total shock to me. How could it be that big companies — much much larger than us, who’ve been in this game longer than us — not get this right?

If the sales data is on one server and the cost and click data is on another server doing the analyses that are critical to success becomes impossible. Pulling the data together should take moments, but will take hours if numbers have to come from different servers.

An important technical question to ask any prospective SEM agency is: Do you have an API connection with each engine, and if so, do you store the daily ad level costs on your server? If their answer is “no” and you’re looking for first class results, you’ll need to look elsewhere.

In world where everything is transparent, what is going to happen is that people who are good, who come from the heart, are going to win.

And people who are bad, who have horrible intentions, are going to lose.

Understand this: anything you do is going to be known.

– Gary Varynerchuk of WineLibraryTV

Confused about paid search?

Wondering how you can increase your website conversion?

Stuck on a thorny web marketing issue?

Give us a shot, we’ll try to help if we can. Send your question to

and put WMQA, for Web Marketing Questions Answered, somewhere in the email subject. (If your RSS reader stripped the image, the address is wmqa at our domain.)

We’ll do our best to answer your question on this blog, if we can. We won’t be able to answer every question, so we’ll select questions which are interesting, which we can answer, and which may help others. Respecting your privacy, we’ll strip identifying information before posting. We also look forward to our great readers chiming in with their views via comments.

Cheers –

WMQA

The show runs May 19-22 at the Gaylord Palms Resort.

The folks at the DMA have generously sent along a speaker discount — the attached form can help you save up to 25% off registration.

Getting The Most Out of Your PPC Program
Ryan Gibson
May 19, 2008
9:45a-10:45a

We’ll review best practices for developing keyword lists, building effective adgroups, and writing compelling copy. We’ll present explicit formulas for rational economic bidding and provide spreadsheet models to determine optimal PPC ad budgets. We’ll also touch on key advanced topics, including advertising on brand names; match-type optimization; mining the long tail; and off-site conversion tracking.

Conversion Is Your Ultimate Secret Weapon: Tracking & Optimizing Search Traffic
Larry Becker
May 19, 2008
12:45p-1:45p

You must measure your paid and organic search conversions to determine where to invest, where to cut, and where to fine-tune. In this session, we’ll review different tracking technologies, including redirectors, cookies, tracking URLs, and on-page JavaScript. We’ll show methods for tracking search-driven sales into the call center and stores. We’ll discuss the limitations of “last touch gets credit” allocation schemes and propose alternatives. We’ll discuss why “conversion is the ultimate secret weapon” and present concrete approaches to increase sales per visitor.

Growth Through Smart Bidding: Wine Enthusiast & RKG Share Strategies
George Michie & Glenn Edelman
May 21, 2008
10:45a- 12:00p

Grow your PPC program through smarter bidding! In this session, you’ll hear the secrets behind portfolio bid management systems and why the industry standard position-based systems guarantee inefficiency. Hear how Wine Enthusiast increased non-brand sales significantly and improved efficiency through smart long-tail management and flexible tools. You will learn:

10 advanced tricks and tips to improve your bottom line

The single most important fact about search marketing

Beyond the talks, we’re also offering 1:1s at the “medical center” and hosting a lunch table.

Lunch with the Experts
May 21, 2008
Noon

Ryan Gibson hosts a table discussing hot topics in paid search marketing.

“Ask the Doctor” Multichannel Merchant Medical Centers
Time TBA

Larry Becker, (who also returns as as a judge in this year’s MCM website awards) will provide one-on-one consulting on usability and website effectiveness.

Ryan Gibson will provide one-on-one consulting on paid search marketing.

If you’d like to meet up in Orlando, give us a call. We hope to see you at the show!

The DMA has graciously offered us a discount to share. Use code SPKVIP for $100 off the current price.

I’ll be sharing a panel discussion with Sarab Singh of GlobalSpec and Chris Hulse of Business.com:

Best Practices in B-to-B Search
Wednesday March 5th
8:45a- 10:00a

This session will provide a detailed look into online search for the B2B community. The panel will discuss what makes B2B search/online advertising unique and provide information on the best ways to reach B2B customers through search. It will provide you with actionable information to drive your search marketing and answer many questions.