Measuring the success of our optimization efforts turned out to be harder than any of us initially thought. It occurred to me that since my situation is anything but unique, it might make sense to write about it. Hopefully others have had similar experiences and we can raise awareness on this rather new topic…
You’ve made the case for advanced optimization, implemented loads of slick technology, and deployed across some or all of your paid search programs. Did you do the right thing? Did you make more money? If so, how much more?
To answer the question ‘did I make more money’ implies a baseline. Trouble is, you don’t really have one. What you’re really asking is ‘did I make more money than I would have otherwise’, and it’s hard to tell with paid search, because the market is inherently dynamic, as are the programs we manage in these markets. This makes benchmarking really tough, and scientific testing nearly impossible.
What you need here is a way to (somewhat) objectively measure success, a methodology not only that people can agree upon, but also one where execution is manageable. Let’s look at a few ideas that might work for you, starting from the simplest and moving toward the more complex:
Trending Success Metrics Over Time.
Look at trends in revenue, profit, ROI, whatever matters to you most from week to week, month to month, etc. Look at overall program performance before you started optimizing, and over time as optimization took over. You just might find out that ROI has increased steadily quarter over quarter, for instance. This approach is relatively easy, and works fine in a steady-state environment, but if you’re subject to seasonality or a shifting market it may not work for you. For example, your optimization efforts may look like a home run in a booming economy or a high season, and conversely you might be inclined to run back to your cube and update your resume if you’re trending results in a low season or a rough economy. What if this method doesn’t give you any conclusive results?
Comparing Paid Search to Your Overall Business
If seasonality or macroeconomic trends are clouding your view of campaign performance, try comparing your results to those of your business as a whole. This may give you the baseline you need, particularly if your paid search campaign is in a somewhat mature state. For example, if your company saw ten percent earnings growth and your SEM profit increased by twenty percent over the same period, you may be able to claim ten percent growth attributed to paid search. That’s not bad for a day’s work, assuming the lift in profit is more than you paid for optimization. But what if your paid search program is on a totally different trajectory than your overall business?
If the first two methods don’t work for whatever reason, here’s another way to look at it. You probably have parts of your portfolio that are managed by automated algorithms, and portions that aren’t. Try looking at these parts separately to understand what happened. Select a cohort of keywords that was managed by each method, and analyze what happened over time. You’ll need to account for seasonality, so ideally look at year-over-year comparisons, or if you don’t have that long a timeline, try to normalize for the seasonality with historical data (try to figure out how much your paid search business as a whole changes with the season). Then, look at the performance of each cohort and compare. For example, if you’re managing through an economic downturn and you see a downward trend in cohort A, but cohort B shows a flat trend line, even though you’re not seeing growth in cohort B you may conclude that it’s doing better by comparison, and you can now quantify the ‘growth’ or ‘lift’.
As I hint at above, once you understand your lift or growth from advanced optimization, you will want to calculate your ROI on the optimization effort as a whole. Particularly if you’re in a big company, you should always be able to hold your projects up against any other effort your company could otherwise use the money to pursue. If you can show that you increased profit by a million dollars at, say, a fifty percent ROI, odds are good that your management will be asking you if you can do more optimization projects, thereby eliminating the need for you to rush back to your cube and update that resume.
Again, I think the key here is to find a method that you and your management can agree upon and a process that is manageable. And once you do, you’ll be well served to have the data and analyses in your hip pocket well before anyone comes asking for them.