Adometry Combines Attribution with Optimization

So…my last two posts on attribution systems (MMA and VisualIQ ) were among the least popular ever, right down there with Marketing Lessons from Chernobyl (which, let’s face it, was in pretty poor taste). But vox populi isn’t always vox Dei, eh? I think it’s an important topic, so here we go again.

The lucky recipient of that less-than-stirring introduction is Adometry, which in no way deserves any disrespect. From humble beginnings in click fraud prevention, they have grown in recent years to be one of the leaders in algorithmic response attribution. Their latest expansion moves them beyond digital channels to offline media including direct mail, television, and print. They have also moved from attributing past results to using predictive models to optimize current and future campaigns. Impressive.

The core of Adometry’s attribution methodology is to compile the sequence of marketing messages seen by each individual, and then compare results of individuals whose sequence differs by only one message. Any difference in results is then attributed to that message. This is conceptually simple, but requires clever treatments to handle low volumes for specific sequences and to isolate the impact of attributes such as placement, time slot, creative, and list segment. Adometry also lets users model against multiple events in the customer life cycle, such as sign-ups, first purchase, and repeat purchase. It calls these all conversions, which I personally found a bit confusing but suppose would quickly get used to.

The system also classifies each conversion as attributable, multi-touch, and multi-channel, depending on whether it was linked to at least one message (attributable), to multiple messages (multi-touch) and to messages in multiple channels (multi-channel). For each category, it shows the conversion count and revenue: so, for example, you see the number and revenue for multi-touch repeat purchases. That’s a lot of information to digest, but does give a great deal of insight into the effect of different promotions and channels on different parts of the business. This encourages marketers to look beyond any single measure, such as cost per order, that tells only a small part of the business story.

The system’s optimization process begins with the attribution analysis, but then adds auto-generated predictive models to estimate the impact of future ad plans, including interactions across channels. Users can enter scenarios with budgets for multiple channels and campaigns, and then apply other constraints such as limits on the change in spending per channel. They also define output measures for the system to optimize against: like other optimization systems, Adometry can only optimize against a single measure, but this can be a composite of several items. For each scenario, the system will determine the optimal budget allocation and show the expected results across each output measure. Users can modify the recommended plan and have the system re-forecast the results. The final plan can be output to a spreadsheet for further editing. Adometry can also be connected directly to ad buying platforms, including systems for real time bidding on individual impressions.   The company says optimization typically yields a 20% to 40% improvement in ad-to-sales ratios.

The database of marketing messages per individual can be used for other types of analysis. These include reach and frequency reports, which show the number of individuals reached in total, reached in each channel, and reached exclusively for each channel. The reports count impressions as well as individuals; show how many people were reached in each combination of channels; show the number of people with each number of impressions (one, two, three, etc.); and show the current member count in each funnel stage.

Adometry’s data comes primarily from tags embedded in advertisements, emails, and other online messages, which drop cookies to identify who sees which message. The system can also draw data from Web server logs or third party tags. Adometry can further enrich its database by appending external information about individuals, using both online and offline sources. This lets it profile the audiences associated with different events, channels, campaigns, and other attributes. Optimization models can use data that can’t be tied to specific individuals, such as weather, economic conditions,  and mass media like television and print. The system can also verify which ads were actually seen by individuals, providing more precise inputs to the attribution calculations.

Pricing for Adometry is based on the number of channels and volume of data. It starts around $100,000 per year for the smallest clients with enough volume to use the system effectively (about 30 to 50 million impressions per month). Currently, more than 50 companies use Adometry’s attribution services.

Read More

6 Key Marketing Measures That Don't Include Revenue

Ask most marketers how they measure performance, and they’ll tell you they look at results: incremental revenue or return on investment if they’re available, or response rates if they're not. Industry experts take a similar approach, focusing largely on the need for better revenue measures. The situation – and barely concealed frustration – is captured perfectly in the headline from a recent Forrester Consulting study sponsored by Silverpop: “Response Metrics Are Used To Evaluate Success, Leaving Customer Or Business Impact Metrics Largely Ignored”.

I agree that revenue is important, but humbly suggest that there’s more to marketing measurement than ROI. Marketers need several types of information – and shouldn’t let the quest for performance measures prevent them from meeting other requirements.

Here are six non-value measures that marketers should build into their reporting systems.

• Benchmarks. Sure, marketing’s job is to generate revenue, but is generating $1 million good or bad? The only way to know is by placing the number in context, which could be this year’s marketing plan or last year’s actual results. Even if marketers can’t measure revenue, they can  set benchmarks for metrics like number of leads generated, funnel conversion rates, or cost per order. In fact, measuring components that contribute to results gives better insights into marketing performance than reporting on the results themselves. For this reason, marketers who can’t directly measure marketing-generated revenue should think twice before creating complex indirect estimates that are hard to understand and have limited credibility. The money would probably be better spent on reports that provide a clearer picture of what’s actually happening.

• Projections. Past results are interesting but the future is more important. Again, the real need is to understand the factors that determine future results, such as response rates and funnel velocity. Changes in these can give early warning of risks and opportunities.  The trick is to distinguish real trends from random variations, so marketers react quickly without chasing too many false alarms.

• Operations. It’s easy to make mistakes in setting up a marketing program, especially one with multiple stages, lead scoring models, and decision rules. Even careful testing can’t always capture all program steps or contingencies. Marketers need reports on the number of people in each program stage and receiving each message, and they need a model that lets them know whether those numbers are reasonable. Reports should show both program-to-date and weekly or daily results: cumulative data show major errors such as bad program logic, and short-term results capture small problems, such as a missed processing step, that could get lost in a program-to-date aggregate.

• Exceptions. Projections and benchmarks put data in context, but marketers don't have time to comb through every figure.  They need exception reports to highlight the most important variations, both positive and negative.  Marketers also need tools to drill into the exceptions so they can understand what happened and identify new opportunities.

• Testing.  Formal tests are the most certain way to understand the impact of marketing projects, but they often require special reporting tools such as ways to compare results for different customer groups over time.  Incremental revenue is the ultimate measure for test evaluation, but often other metrics such as response rates or velocity are easier to capture and more directly relevant.  Reaping the full benefit of tests also requires systems to distribute results and catalog findings for future reference.

• Strategic Goals. Marketing plans should be based on corporate strategy, but long-term goals fade into the background once marketers start making tactical choices based on day-to-day results. The reporting system should provide direct measures of strategic objectives – things like penetration of new market segments and exploration of new channels – so marketers can see the cumulative impact of deviations from the original plans. Many strategic goals, such as process change, staff training, and systems deployment, are not measured in revenue at all.

The types of measures I’ve just described don’t replace revenue and ROI reporting.  Rather, they meet needs that revenue reports alone cannot. Ideally, all these types of information will be combined in a marketing dashboard that provides a quick overview of critical information and allows drilling into details when necessary. Marketers should realize that the contents of this dashboard will change over time as their focus shifts to different programs and strategic goals. They should also recognize that good reporting will generate new questions as it uncovers risks and opportunities that would otherwise have gone undetected.  The system should make those questions easier to answer, but marketers shouldn't expect their total work to decrease.  What they can expect is that better reporting will increase the value created by their efforts: yet another new metric, Return on Reporting, should go up.

Read More

How Do You Measure the Influence of Marketing Messages?

My review of Coremetrics Lifestyle raised the issue of measuring the impact of marketing materials on customer behavior. Of course, this is just one piece of the marketing attribution puzzle. But it’s worth a separate discussion because it’s such a common question – and, unlike so many measurement problems, this one actually has an answer.

Let’s start with the original impetus. This was an “influence” report that showed the percentage of people reaching a marketing stage who had received specific marketing treatments (or had other attributes such as source, product history, demographic, etc.). The idea was that treatments received by a higher percentage of customers were more influential. In other words, if 100% of new buyers saw a white paper offer and just 50% saw a Webinar invitation, then the white paper has more influence than the Webinar.

Plausible, yes. But wrong.

Let’s think through the example. What if the white paper is offered to everyone? Yes, 100% of new buyers saw it, but so did 100% of non-buyers. We know exactly nothing about whether it made its recipients more or less likely to purchase.

Now, let’s say just 10% of prospects see the Webinar invitation, compared with 50% of buyers. Can we say it has a positive influence? Still no: maybe the Webinar attracts hot prospects who would have purchased anyway. It’s even possible that the Webinar offer annoys people and actually reduces purchase rates. You can’t tell from these figures.

In other words, it’s not enough to know what was seen by customers who became buyers (or, more generally, by people who took any particular action). You also need to know what was seen by non-buyers and, ideally, to compare results for groups that are similar except for that particular treatment.

So, what measures do make sense for assessing influence?

- the simplest measure compares the result rate of treated customers with results for non-treated customers. You might find that 20% of people who receive a white paper became buyers, compared with 10% of people who don’t receive the white paper. These two figures can be combined in a single ratio: 20% of treated / 10% of non-treated = 2.0. The higher the ratio, the more it seems that receiving the white paper increased the likelihood that someone would purchase. But it’s no more than a suggestion: maybe the white paper was sent to people who were stronger prospects to begin with.

- a more advanced measure adjusts for the audience by attempting to limit the non-treated group (e.g., non-buyers) to customers similar to the target group. This could be done by building a statistical model that uses all other attributes to predict behavior. Or, you could apply lead scores or funnel stage definitions. Whatever the technique, the result is to divide the audience into groups that are expected to behave similarly. The calculation would then compare results of treated vs. non-treated customers in each same group. So, a report might find that 40% of “stage 3 leads” (whatever they are) made a purchase after attending a Webinar, while just 15% of “stage 3 leads” made a purchase if they didn't attend a Webinar. Again, the treated and non-treated figures could be combined in a ratio (40% / 15% = 2.7)

- of course, the only true measure is a structured test. This ensures that the only difference between the treated and non-treated groups is the treatment itself. Without such tests, there's a good chance that the customers selected for treatment would have performed differently in any event.

A proper reporting system would present the ratios along with actual result rates, trends over time, the number of customers receiving each treatment, and comparisons with ratios for other treatments. These figures help marketers focus their energies on the most valuable opportunities. Still, the starting point is always a comparison of treated vs. non-treated performance: without that, the numbers could mean anything.

Read More

Right On Interactive Offers Lifecycle Reporting

Summary: Right On Interactive has added great life stage reporting to the data integration and output generation features of its earlier 5Buckets product. It could supplement a traditional marketing automation system or perhaps replace one. Either way, it’s worth a look to see what you’re missing.

When I reviewed Right On Interactive in a July 2009 post, the company was selling its 5Buckets marketing software as a multi-channel output generation tool that complemented conventional marketing automation systems. Since then, Right On has expanded its functions, dropped the 5Buckets name, and repositioned itself as a marketing automation alternative focused on “customer lifecycle marketing”. It’s tempting to discuss the business strategy behind this, but I assume that you Dear Reader are a marketer and therefore it's not your problem So let’s look at what the system actually does.

We'll start with the standard marketing automation functions. These are what you need if Right On is really to substitute for one of the better-known products:

• data management: Right On can import files from any source, placing the data into standard structures or custom tables. Users can link the imported data to any other table, allowing complex data structures. They can also load data to the system API. This is more powerful than many marketing automation products, which are largely limited to a company, contact and activity history files.

• segmentation: users can define segments using a step-by-step query builder or by writing SQL. The query builder supports complex relationships. This is competitive with or better than standard marketing automation systems.

• campaign design: users can define campaigns with multiple “tactics” . Each tactic has its own action, schedule, metrics, documents, and start and end dates. Contacts can enter a tactic from an assigned segment or flow from a previous tactic based on their response and a user-specified waiting period. These features let Right On support multi-step campaigns although complex designs would be a challenge.

• create emails and forms: Right On uses ExactTarget for email and form creation. The integration is fairly smooth since the editing features are accessed within the Right On interface. A native solution is under development but the current approach should work for unless you have a particular aversion to ExactTarget.

• campaign actions: each tactic can execute one action. These include sending an email via Salesforce.com or ExactTarget, creating a Salesforce.com task, generating an output file, and sending emails to Foursquare friends. This covers the basic needs, although most other products also offer options such as changing data and adding a contact to a list or campaign.

• CRM integration: Right On can synchronize data with Salesforce.com and Microsoft CRM on a regular basis. It can also pull file segments and Salesforce.com campaign members as lists. This makes it roughly equivalent to other products. Right On also has a connector with location-based social network Foursquare.

• campaign reporting: users can manually enter campaign costs, target revenue, and actual revenue. The system will capture responses and use the results for response reporting, cost per response and return on investment. This is pretty standard stuff, although many other systems can also import opportunity revenue automatically from Salesforce.com – a feature still in Right On’s future.

• lead scoring: users can define separate scores for customer fit and activities. Customer fit is based on static attributes such as title while activity score is based on events such as email opens or Twitter posts. There’s also an engagement index that compares the actual activity score with the maximum possible score had the contact responded to every promotion. The scoring rules are built in the usual fashion, by assigning to points to different attribute values or different events, although the interface is nicer than most. Contacts are rescored nightly. The scores are stored on the customer record and can trigger an action to send the contact to Salesforce.com. This is on par with other products.

In other words, when you just look at standard features, Right On is no better than adequate. But that's not the whole picture. There's also "customer lifecycle marketing".

What that means in practice is users can assign contacts to lifecycle stages. This lets the system track contacts as they move through the buying, on-boarding and retention processes. Specifically, it generates reports on the number of contacts in each stage, stage-to-stage conversion rates, and average time spent in each stage. It stores each contact’s stage and score histories so it can report on trends in these metrics as well.

Digging a bit deeper: users define the stages by creating segmentation rules similar to standard queries. The system checks each contact against the rules, assigning the contact to the latest stage for which they qualify. The actual stages can be whatever the user wants. Right On's default set holds two lead stages (investigate and evaluate) and two customer stages (value and advocate).

But there's more. Right On creates scatter plots of contacts in each stage, using customer fit and activity scores as dimensions. The resulting “lifecycle map” is a graphic representation of the shape and quality of the company's contact inventories. The plots are interactive: users can select a group on the plot to create a new segment and can drill down to see the details of the individual contacts. They can view reports and maps for all contacts or selected segments.



Right On recognizes that its data could be used to project future business and to correlate stage changes with marketing campaigns, although it hasn’t yet built these features. Once it does, the system will go a long way to providing the stage-based marketing measurement that I’ve been arguing marketers really need. (You can also view my Marketo-sponsored Webinar on the topic.)

So where does this leave us?

I’m lukewarm about Right On as a primary marketing automation system but see great value in its lifecycle reporting. Pricing is relatively modest – starting at just under $1,700 per month for up to 50,000 contacts – so larger firms may be able to use both Right On and a conventional marketing automation product. Smaller companies will probably have to choose one or the other.

Read More

Omniture Study Suggests Marketers Doubt Value of Analytics Investment

Not to beat a dead horse, but Wednesday’s eMarketer reported on yet another survey that touched on the question of why marketers don’t measure. Although the Omniture 2010 Online Analytics Survey is obviously limited to Web analytics, the answers probably apply to other types of measurement as well.

I wasn’t able to get a copy of the full survey results, despite two requests to Omniture and even filling it out myself, which was supposed to yield a copy that compared my answers with my peers'. Perhaps I’m peerless. But the snippets published in eMarketer are enough for now.


Specifically, eMarketer reported that the leading challenge in Web analytics was “talent”, cited by 58.4% of respondents. Assuming that “talent” is really a polite way of saying “skilled staff”, this suggests that lack of education, not lack of time, is the critical roadblock to better measurement. I’ve been betting the reason is time, but would reconsider in the face of new evidence.

But wait.

When I took the survey, the question about “talent” actually defined it as "lack of skill/time". So it’s perfectly possible that marketers picking "talent" really saw lack of time as the most important challenge.

My position is arguably strengthened by the relatively low ranking of "support/training" (37.6%) and "budget" (31.7%) in the answers. Those can certainly improve skills but they can’t expand the manager's available time. Even hiring more staff wouldn't do that.

On the other hand, the second- and third-ranked challenges were "actionability" (47.3%) and "finding insights" (41.5%) which both suggest doubts that Web analytics can deliver real value. This would show a need for education – but, as I wrote in my comment on the original Why Marketers Don't Measure post, it's a need for education in the fundamental utility of measurement, not education in specific techniques.

Bottom line: the Omniture survey confirms that marketers won’t invest in analytics until they’re convinced it’s the best use of their limited resources. Efforts to expand adoption of analytics should start with that.

Read More

VisualIQ Measures Marketing Impacts Across All Channels

Summary: VisualIQ combines customer-level transactions and contact history with traditional aggregate data to produce better marketing performance measurement. It hasn't solved the problem of identifying the same customer across channels, but it's trying.

I was going to start this post by writing that last-click attribution has recently come under fire, but the first Google hit on the topic brings up a study from 2007. So maybe the criticism isn’t particularly new. But the fact remains that, now more than ever, marketers are trying to measure the impact of all contacts on customer behavior.

Broadly speaking, the problem is attacked in two ways. One, most common among consumer goods manufacturers and others who do not sell directly to their customers, uses aggregated data in marketing mix models to find correlations between marketing efforts and total sales. The other, favored by banks, retailers, communications providers and others who do sell directly to known buyers, assesses the impact of each contact with specific individuals. Last-click attribution is a particular challenge for online marketers because they fall between these two situations: they can often identify their buyers but not trace their full contact history.

VisualIQ, founded in 2005 as Connexion.a, proposes to straddle these worlds by combining aggregate-level models with customer-specific contact history. They haven’t found a magic bullet: like everyone else, VisualIQ tracks online customers through cookies, with all the limits that implies. But VisualIQ strives to make the best use of what’s available by unifying data from as many online campaigns as possible, linking cookies with online transactions, and then linking online transactions to offline identities.

This approach offers some general advantages and two specific capabilities. The general advantages come from assembling all advertising and customer transaction information in one database. This allows VisualIQ to analyze campaign results, do whatever identity matching is possible, and to isolate the impact of source, contact frequency, demographics, location and other variables. VisualIQ, a hosted service, has invested heavily in technology to analyze massive data sets along such dimensions.

The first specific capability is relating pre-purchase contacts to actual purchases for individual customers, thus moving beyond last-click attribution. Although this is subject to the limits of cookie-based tracking, VisualIQ does what it can to build a unified identity by sharing the same cookie IDs across as many online channels as possible. The second capability is building mix models with data from actual customer contacts instead of market-level estimates or surveys. VisualIQ says it has found this yields more accurate results than traditional information.

This is all good stuff and VisualIQ has packaged it nicely in a tiered set of offerings. These range from campaign-level reporting to customer-based insights to predictive modeling and simulation, with prices for the simplest system starting as low as $5,000 to $10,000 per month. The company has had considerable success, counting major banks, retailers, and communications firms as clients. Note that these are all industries that sell to their customers directly.

But VisualIQ’s specific offerings are just part of the story. What’s really important is setting explicit goals of linking identities across channels and measuring cross-channel marketing impacts. These are arguably the core challenges in marketing measurement today. This focus has led VisualIQ to look for alternatives to cookies and to use existing methods to combine online and offline information for the same person.

The company is also seeking to make it easier to apply its results. Today, it basically generates reports that suggest better media allocations and advertising contents. But it is working to automatically feed those findings as rules into execution systems such as ad servers and ad exchanges. This brings marketers closer to the ultimate goal of self-optimizing programs. Other vendors are also pursuing self-optimization, but VisualIQ promises the advantage of decisions based on data from all channels rather than a single channel or, heaven forbid, just the last click.

Read More

5 Steps to Marketing Measurement Maturity

Summary: marketing performance measurement can start with simple response tracking, and grow in stages to show business impact, track the buying process, optimize results and demonstrate strategic alignment. Each stage adds new data, systems, measures and processes.

I’ll be talking about marketing measurement this Tuesday at Silverpop’s B2B Marketing University seminar in Palo Alto, with a repeat performance in Boston on November 4. The core of my presentation will be a 5-step measurement maturity model for B2B marketers. This post will give you a brief summary.

A little background: marketers’ objectives for performance measurement generally fall into five broad categories: measure response, show business impact, track the buying process, optimize results and demonstrate marketing alignment with business strategy. Each category has different requirements for data, systems, measures, and processes. Because some of these requirements overlap, there’s a natural progression starting with the simplest requirements and adding new requirements for each stage. This progression leads to a maturity model. Here are the details.

1. Measure Response. The most basic requirement is simply to count the number of responses to each marketing program. This stage also includes the closely related step of calculating the cost of those responses for a simple cost-per-response measure that can be used for a rough ranking of investments.

Key data needs for this stage include mechanisms to capture responses and campaign costs and to link responses to campaigns. This requires a campaign management system to execute the campaigns and track their costs, and a marketing database that stores the identity, promotion history and response history of leads generated by the campaigns.

2. Show Business Impact of Marketing Campaigns. The cost per response tells little about the business impact of a marketing program. You must also know the value of those responses. At a minimum, this requires linking marketing leads to closed sales. This typically means importing closed opportunity records from a sales automation system and linking those opportunities back to the original marketing lead. Add the cost data already gathered in stage 1 (response measurement) and you can calculate a simple Return on Investment based on revenue / acquisition cost.

Of course, true ROI is based on profits, not revenue, and incorporates all incremental costs, not just the initial marketing expense. A proper business impact measurement thus requires capturing the full marketing, sales and product costs associated with new leads, as well as their actual or estimated long-term value. These give a ROI measure that comes reasonably close to showing the true business impact of each acquisition campaign.

In terms of new requirements, this step adds sales opportunity data, which implies integration with a sales force automation or CRM system. It also requires processes to assign opportunities to leads and leads to campaigns, and, optionally, ways to import and connect lifetime costs and revenues.

Keep in mind that this approach only applies to lead acquisition campaigns, not to campaigns that nurture existing leads or customers, or branding campaigns that don’t generate a direct response. This is a smaller issue for B2B marketers than many consumer marketers, who often have no direct way to identify the customers acquired or influenced by their activities.

3. Understand the Buying Process. This stage looks at the impact of non-acquisition marketing treatments on moving customers through the buying process. It requires defining stages within the buying process, tracking the movement of individual leads through those stages over time, recording the marketing treatments applied to those individuals, and measuring the correlation (if any) of treatments to stage changes. The result is both a more detailed understanding of the buying process and a way to decide which treatments are most valuable.

Meeting these requirements implies capturing more information about leads, including static attributes (company, job title, etc.) and behaviors such as Web site visits and email responses. These can be used in scoring models or other tools that decide which stage a lead is in at any given moment. The system must also keep a history of each lead’s stages over time, so it can correlate stage changes with treatments. The treatments themselves are already captured in the marketing database needed for response measurement, so they do not represent a new requirement.

4. Optimize Results. Once you’re tracking lead movement through process stages and measuring the impact of individual treatments, you’re ready to build an end-to-end model that calculates the impact of each small change on final sales. You can then optimize the combination of treatments across the process. For example, you might find you can spend less on acquiring new leads if you spend more on nurturing existing ones, to produce the same sales volume at lower cost.

The calculations for this type of optimization are fairly simple, and they don’t require any new information beyond the previous stage in the maturity model. But you do need more precise information about the impact of different treatments, which means formal testing of alternative treatments and careful analysis of results. You’ll also need a business simulation model that can estimate the impact of changes on near-term revenues and costs, since the company still has its quarterly targets to hit. You'll also need to define you goals -- higher revenue? higher profit rate? lower marketing costs? -- so you know what to optimize.

Ideally, you’ll also add optimization software that can automatically find the best of all possible treatment combinations. But few B2B marketers have the data volume or statistical skills required for this, so you’ll probably end up manually running a variety of scenarios through your simulation model instead.

5. Demonstrate Strategic Alignment. Delivering the optimal set of treatments won’t satisfy your CEO if your marketing programs don’t align with the larger business strategy. That alignment may in fact require campaigns that produce low short-term returns, such as investing in a new product or market segment.

To demonstrate alignment, you must first show that your planned marketing activities support business goals, and then show that those activities have yielded the expected results. For example, a strategy based on selling to a new group of customers might yield a marketing plan with 10% of acquisition spending aimed at generating leads from that segment, and a goal of that segment accounting for 5% of new leads received.

The exact requirements for this stage will depend on your specific strategic goals. But it’s likely that you’ll need more information about the purposes of your marketing spending (so you can show which funds are supporting which strategic goals) and more about lead attributes and behaviors (so you can show that results are in line with expectations).

Incidentally, demonstrating strategic alignment doesn’t depend directly understanding the buying process (stage 3) or optimizing results (stage 4). So a company that has reached stage 2 in the maturity model could jump immediately to demonstrating strategic alignment if desired.

Final Thought: once you get past counting response, all later stages in the maturity model assume you are measuring your marketing performance against the ultimate goals of closed sales and long-term customer value. This is increasingly necessary as marketing remains involved with leads even after they are officially transferred to sales. It implies that marketing and sales must integrate their systems, so they can view, coordinate, analyze and ultimately optimize sales and marketing activities across the entire buying cycle.

In companies where the marketing's responsibility still ends with the hand-off of qualified leads to sales, the maturity model could be adjusted to optimize only through that stage. But that's an increasingly obsolete approach.

Read More

LucidEra's Failure: More Evidence that Marketers Won't Pay for Measurement

I’m just catching up with what happened while I was on vacation these past two weeks. One piece of news is the demise of LucidEra, which this blog profiled almost exactly one year ago. According to SearchDataManagement.com, the company said it shut down because it couldn’t raise new funds or find a buyer.

There has been some learned discussion of the causes of LucidEra’s collapse on Timo Elliot’s BI Questions Blog. Much seems to focus on the apparent operating costs. These must have been substantial, since the company raised $15.6 million in 2007 and, presumably, has since spent it all.

Still, I think the fundamental problem was a lack of customers. When I spoke with LucidEra in June 2008 they said they had about 40 paying clients. When I spoke with them again in October 2008, the number was 50 and it was still at 50 when we spoke in April 2009. In other words, LucidEra was making very few sales or, even worse, was able to make new sales but couldn’t retain its customers. [For more insight based on comments by LucidEra managers, see this post on the Datadoodle blog.]

With the benefit of 20/20 hindsight, LucidEra’s strategic decision to focus on building sales analysis applications primarily for Salesforce.com was a mistake. Bear in mind that there are about 60,000 Salesforce.com customers – selling to 50 of them is less than 0.1% penetration.

I suspect LucidEra’s price point, around $3,000 per month depending on the details, was too rich for many of its prospective clients. Not that they couldn’t actually afford it – but they didn’t want to spend that much money on sales analysis.

This is not surprising. I reluctantly concluded some time ago that marketers (and presumably sales managers) are not willing to spend money on measurement systems even though they consistently say in surveys that better measurement is a high priority. For recent evidence along these lines, see the 2009 Marketing ROI and Measurements Study published by Lenskold Group and sponsored by MarketSphere, which found that “6 in 10 firms (59%) indicate having an increased demand for marketing measurements, analysis and reporting in 2009 without the budget necessary for those measurement efforts.”

Many analysts and other on-demand business intelligence vendors have been quick to assert that LucidEra’s failure does not reflect a problem with the notion of on-demand BI in general. I agree, since I see the key to LucidEra's demise as its uniquely narrow focus on sales analysis. Indeed, competitors including Birst and GoodData have leapt to offer a new home to orphaned LucidEra clients.

Still, the apparently high costs to sustain a small client base suggests the economics of this business are not as attractive as they seem. LucidEra's Darren Cunningham did tell me that their costs were particularly high because they were not a multi-tenant solution and had to manage the entire BI stack to support a single application. Presumably other on-demand BI vendors can run more cheaply. Still there does seem to be a little more reason for caution in approaching on-demand BI vendors, even though there is not (yet) any cause for alarm.

Read More

Making Some Changes

Maybe it's that spring has finally arrived, but, for whatever reason, I have several changes to announce.

- Most obviously, I've changed the look of this blog itself. Partly it's because I was tired of the old look, but mostly it's to allow me to take advantage of new capabilities now provided by Blogger. The most interesting is a new polling feature. You'll see the first poll at the right.

- I've also started a new blog "MPM Toolkit" at http://mpmtoolkit.blogspot.com/. In this case, MPM stands for Marketing Performance Measurement. This has a been a topic of growing professional interest to me; in fact, I have a book due out on the topic this fall (knock wood). I felt the subject was different enough from what I've been writing about here to justify a blog of its own. Trying to keep the brand messages clear, as it were.

- I have resumed working under the Raab Associates Inc. umbrella, and am now a consultant rather than partner with Client X Client. This has more to do with accounting than anything else. It does, however, force me to revisit my portion of the Raab Associates Web site, which has not been updated since the (Bill) Clinton Administration. I'll probably set up a new separate site fairly soon.

Sorry to bore you with personal details. I'll make a more substantive post tomorrow.

Read More

Marketing Performance Measurement: No Answers to the Really Tough Questions

I recently ran a pair of two-day workshops on marketing performance measurement. My students had a variety of goals, but the two major ones they mentioned were the toughest issues in marketing: how to allocate resources across different channels and how to measure the impact of marketing on brand value.

Both questions have standard answers. Channel allocation is handled by marketing mix models, which analyze historical data to determine the relative impact of different types of spending. Brand value is measured by assessing the important customer attitudes in a given market and how a particular brand matches those attitudes.

Yet, despite my typically eloquent and detailed explanations, my students found these answers unsatisfactory. Cost was one obstacle for most of them; lack of data was another. They really wanted something simpler.

I’d love to report I gave it to them, but I couldn't. I had researched these topics thoroughly as preparation for the workshops and hadn’t found any alternatives to the standard approaches; further research since then still hasn’t turned up anything else of substance. Channel allocation and brand value are inherently complex and there just are no simple ways to measure them.

The best I could suggest was to use proxy data when a thorough analysis is not possible due to cost or data constraints. For channel allocation, the proxy might be incremental return on investment by channel: switching funds from low ROI to high ROI channels doesn’t really measure the impact of the change in marketing mix, but it should lead to an improvement in the average level of performance. Similarly, surveys to measure changes in customer attitudes toward a brand don’t yield a financial measure of brand value, but do show whether it is improving or getting worse. Some compromise is unavoidable here: companies not willing or able to invest in a rigorous solution must accept that their answers will be imprecise.

This round of answers was little better received than the first. Even ROI and customer attitudes are not always available, and they are particularly hard to measure in multi-channel environments where the result of a particular marketing effort cannot easily be isolated. You can try still simpler measures, such as spending or responses for channel performance or market share for brand value. But these are so far removed from the original question that it’s difficult to present them as meaningful answers.

The other approach I suggested was testing. The goal here is to manufacture data where none exists, thereby creating something to measure. This turned out to be a key concept throughout the performance measurement discussions. Testing also shows that marketers are at least doing something rigorous, thereby helping satisfy critics who feel marketing investments are totally arbitrary. Of course, this is a political rather than analytical approach, but politics are important. The final benefit of testing is it gives a platform for continuous improvement: even though you may not know the absolute value of any particular marketing effort, a test tells whether one option or another is relatively superior. Over time, this allows a measurable gain in results compared with the original levels. Eventually it may provide benchmarks to compare different marketing efforts against each other, helping with both channel allocation and brand value as well.

Even testing isn’t always possible, as my students were quick to point out. My answer at that point was simply that you have to seek situations where you can test: for example, Web efforts are often more measurable than conventional channels. Web results may not mirror results in other channels, because Web customers may themselves be very different from the rest of the world. But this again gets back to the issue of doing the best with the resources at hand: some information is better than none, so long as you keep in mind the limits of what you’re working with.

I also suggested that testing is more possible than marketers sometimes think, if they really make testing a priority. This means selecting channels in part on the basis of whether testing is possible; designing programs so testing is built in; and investing more heavily in test activities themselves (such as incentives for survey participants). This approach may ultimately lead to a bias in favor of testable channels—something that seems excessive at first: you wouldn’t want to discard an effective channel simply because you couldn’t test it. But it makes some sense if you realize that testable channels can be improved continuously, while results in untestable channels are likely to stagnate. Given this dynamic, testable channels will sooner or later become more productive than untestable channels. This holds even if the testable channels are less efficient at the start.

I offered all these considerations to my students, and may have seen a few lightbulbs switch on. It was hard to tell: by the time we had gotten this far into the discussion, everyone was fairly tired. But I think it’s ultimately the best advice I could have given them: focus on testing and measuring what you can, and make the best use possible of the resulting knowledge. It may not directly answer your immediate questions, but you will learn how to make the most effective use of your marketing resources, and that’s the goal you are ultimately pursuing.

Read More

Marketing Performance Involves More than Ad Placement

I received a thoughtful e-mail the other day suggesting that my discussion of marketing performance measurement had been limited to advertising effectiveness, thereby ignoring the other important marketing functions of pricing, distribution and product development. For once, I’m not guilty as charged. At a minimum, a balanced scorecard would include measures related to those areas when they were highlighted as strategic. I’d further suggest that many standard marketing measures, such as margin analysis, cross-sell ratios, and retail coverage, address those areas directly.

Perhaps the problem is that so many marketing projects are embedded in advertising campaigns. For example, the way you test pricing strategies is to offer different prices in the marketplace and see how customers react. Same for product testing and cross-sales promotions. Even efforts to improve distribution are likely to boil down to campaigns to sign up new dealers, training existing ones, distribute point of sale materials, and so on. The results will nearly always be measured in terms of sales results, exactly as you measure advertising effectiveness.

In fact, since everything is measured through advertising it and recording the results, the real problem may be how to distinguish “advertising” from the other components of the marketing mix. In classic marketing mix statistical models, the advertising component is representing by ad spend, or some proxy such as gross rating points or market coverage. At a more tactical level, the question is the most cost-effective way to reach the target audience, independent of the message content (which includes price, product and perhaps distribution elements, in addition to classic positioning). So it does make sense to measure advertising effectiveness (or, more precisely, advertising placement effectiveness) as a distinct topic.

Of course, marketing does participate in activities that are not embodied directly in advertising or cannot be tested directly in the market. Early-stage product development is driven by market research, for example. Marketing performance measurement systems do need to indicate performance in these sorts of tasks. The challenge here isn’t finding measures—things like percentage of sales from new products and number of research studies completed (lagging and leading indicators, respectively) are easily available. Rather, the difficulty is isolating the contribution of “marketing” from the contribution of other departments that also participate in these projects. I’m not sure this has a solution or even needs one: maybe you just recognize that these are interdisciplinary teams and evaluate them as such. Ultimately we all work for the same company, eh? Now let’s sing Kumbaya.

In any event, I don’t see a problem using standard MPM techniques to measure more than advertising effectiveness. But it’s still worth considering the non-advertising elements explicitly to ensure they are not overlooked.

Read More

Is Marketing ROI Important?

You may have noticed that my discussions of marketing performance measurement have not stressed Return on Marketing Investment as an important metric. Frankly, this surprises even me: ROMI appears every time I jot down a list of such measures, but it never quite fits into the final schemes. To use the categories I proposed yesterday, ROMI isn’t a measure of business value, of strategic alignment, or of marketing efficiency. I guess it comes closest to the efficiency category, but the efficiency measures tend to be more simple and specific, such as a cost per unit or time per activity. Although ROMI could be considered the ultimate measure of marketing efficiency, it is too abstract to fit easily into this group.

Still, my silence doesn’t mean I haven’t been giving ROMI much thought. (I am, after all, a man of many secrets.) In fact, I spent some time earlier this week revisiting what I assume is the standard work on the topic, James Lenskold’s excellent Marketing ROI. Lenksold takes a rigorous and honest view of the subject, which means he discusses the challenges as well as the advantages. I came away feeling ROMI faces two major issues: the practical one of identifying exactly which results are caused by a particular marketing investment, and the more conceptual one of how to deal with benefits that depend in part on future marketing activities.

The practical issue of linking results to investments has no simple solution: there’s no getting around the fact that life is complex. But any measure of marketing performance faces the same challenge, so I don’t see this as a flaw in ROMI itself. The only thing I would say is that ROMI may give a false illusion of precision that persists no matter how many caveats are presented along with the numbers.

How to treat future, contingent benefits is also a problem any methodology must face. Lenskold offers several options, from treating several investments into a single investment for analytical purposes, to reporting the future benefits separately from the immediate ROMI, to treating investments with long-term results (e.g. brand building) as overhead rather than marketing. Since he covers pretty much all the possibilities, one of them must be the right answer (or, more likely, different answers will be right in different situations). My own attitude is this isn’t something to agonize over: all marketing decisions (indeed, all business decisions) require assumptions about the future, so it’s not necessary to isolate future marketing programs as something to treat separate from, say, future product costs. Both will result in part from future business decisions. When I calculate lifetime value, I certainly include the results of future marketing efforts in the value stream. Were I to calculate ROMI, I’d do the same.

So here's what it comes down to. Even though I'm attracted to the idea of ROMI, I find it isn't concrete enough to replace specific marketing efficiency measures like cost per order, but is still too narrow to provide the strategic insight gained from lifetime value. (This applies unless you define ROMI to include the results of future marketing decisions, but then it's really the same as incremental LTV.)

Now you know why ROMI never makes my list of marketing performance measures.

Read More

Marketing Performance: Plan, Simulate, Measure

Let’s dig a bit deeper into the relationships I mentioned yesterday among systems for marketing performance measurement, marketing planning, and marketing simulation (e.g., marketing mix models, lifetime value models). You can think of marketing performance measures as falling into three broad categories:

- measures that show how marketing investments impact business value, such as profits or stock price

- measures that show how marketing investments align with business strategy

- measures that show how efficiently marketing is doing its job (both in terms of internal operations and of cost per unit – impression, response, revenue, etc.)

We can put aside the middle category, which is really a special case related to Balanced Scorecard concepts. Measures in this are traditional Balanced Scorecard measures of business results and performance drivers. By design, the Balanced Scorecard focuses on just a few of these measures, so it is not concerned with the details captured in the marketing planning system. (Balanced Scorecard proponents recognize the importance of such plans; they just want to manage them elsewhere). Also, as I’ve previously commented, Balanced Scorecard systems don’t attempt to precisely correlate performance drivers to results, even though they do use strategy maps to identify general causal relationships between them. So Balanced Scorecard systems also don’t need marketing simulation systems, which do attempt to define those correlations.

This leaves the high-level measures of business value and the low-level measures of efficiency. Clearly the low-level measures rely on detailed plans, since you can only measure efficiency by looking at performance of individual projects and then the project mix. (For example: measuring cost per order makes no sense unless you specify the product, channel, offer and other specifics. Only then can you determine whether results for a particular campaign were too high or too low, by comparing them with similar campaigns.)

But it turns out that even the high-level measures need to work from detailed plans. The problem here is that aggregate measures of marketing activity are too broad to correlate meaningfully with aggregate business results. Different marketing activities affect different customer segments, different business measures (revenue, margins, service costs, satisfaction, attrition), and different time periods (some have immediate effects, others are long-term investments). Past marketing investments also affect current period results. So a simple correlation of this period marketing costs vs. this period business results makes no sense. Instead, you need to look at the details of specific marketing efforts, past and present, to estimate how they each contribute to current business results. (And you need to be reasonably humble in recognizing that you’ll never really account for results precisely—which is why marketing mix models start with a base level of revenue that would occur even if you did nothing.) The logical place to capture those detailed marketing effort is the marketing planning system.

The role of simulation systems in high-level performance reporting is to convert these detailed marketing plans into estimates of business impact from each program. The program results can then be aggregated to show the impact of marketing as a whole.

Of course, if the simulation system is really evaluating individual projects, it can also provide measures for the low-level marketing efficiency reports. In fact, having those sorts of measures is the only way the low-level system can get beyond comparing programs only against other similar programs, to allow comparisons across different program types. This is absolutely essential if marketers are going to shift resources from low- to high-yield activities and therefore make sure they are optimizing return on the marketing budget as a whole. (Concretely: if I want to compare direct mail to email, then looking at response rate won’t do. But if I add a simulation system that calculates the lifetime value acquired from investments in both, I can decide which one to choose.)

So it turns out that planning and simulation systems are both necessary for both high-level and low-level marketing performance measurement. The obvious corollary is that the planning system must capture the data needed for the simulation system to work. This would include tags to identify the segments, time periods and outcomes the each program is intended to affect. Some of these will be part of the planning system already, but other items will be introduced only to make simulation work.

Read More

Looking for Balanced Scorecard Software

I haven’t been able to come up with an authoritative list of major balanced scorecard software vendors. UK-based consultancy 2GC lists more than 100 in a helpful database with little blurbs on each, but they include performance management systems that are not necessarily for balanced scorecards. The Balanced Scorecard Collaborative, home of balanced scorecard co-inventor David P. Norton, lists two dozen products they have certified as meeting true balanced scorecard criteria. Of these, more than half belong non-specialist companies including enterprise software (Oracle, Peoplesoft [now Oracle], SAP, Infor, Rocket Software) and broad business intelligence systems (Business Objects, Cognos, Hyperion [now Oracle], Information Builders, Pilot Software [now SAP], SAS). Most of these firms have purchased specialist products. The remaining vendors (Active Strategy, Bitam, Consist FlexSI, Corporater, CorVu, InPhase, Intalev, PerformanceSoft [now Actuate], Procos, Prodacapo, QPR and Vision Grupos Consultores) are a combination of performance management specialists and regional consultancies.

That certified products are available from all the major enterprise and business intelligence vendors shows the basic functions needed for balanced scorecard are well understood and widely available. I’m sure there are differences among the products but suspect their choice of system will rarely be critical to project success or failure. The core functions are creation of strategy maps and cascading scorecards. I suspect systems vary more widely in their ability to import and transform scorecard data. A number of products also include project management functions such as task lists and milestone reporting. This is probably outside of the core requirements for balanced scorecard but does make sense in the larger context of providing tools to help meet business goals.

If your idea of a good time is playing with this sort of system (and whose isn’t?), Strategy Map offers a fully functional personal version for free.

Read More

Why Balanced Scorecards Haven't Succeeded at Marketing Measurement

All this thinking about the overwhelming number of business metrics has naturally led me consider balanced scorecards as a way to organize metrics effectively. I think it’s fair to say that balanced scorecards have had only modest success in the business world: the concept is widely understood, but far from universally employed.

Balanced scorecards make an immense amount of sense. A disciplined scorecard process begins with strategy definition followed by a strategy map, which identifies the measures most important to a business and how they are relate to each other and final results. Once the top-level scorecard is built, subsidiary scorecards report on components that contribute to the top-level measures, providing more focused information and targets for lower-level managers.

That’s all great. But my problem with scorecards, and I suspect the reason they haven’t been used more widely, is they don’t make a quantifiable link between scorecard measures and business results. Yes, something like on-time arrivals may be a critical success factor for an airline, and thus appear on its scorecard. That scorecard will even give a target value to compare with actual performance. But it won’t show the financial impact of missing the target—for example, every 1% shortfall vs. the target on-time arrival rate translates into $10 million in lost future value. Proponents would argue (a) this value is impossible to calculate because there are so many intervening factors and (b) so long as managers are rewarded for meeting targets (or punished for not meeting them), that’s incentive enough. But I believe senior managers are rightfully uncomfortable setting those sorts of targets and reward systems unless the relationships between the targets and financial results are known. Otherwise, they risk disproportionately rewarding the selected behaviors, thereby distorting management priorities and ultimately harming business results.

Loyal readers of this blog might expect me to propose lifetime value as a better alternative. It probably is, but the lukewarm response it elicits from most managers has left me cautious. Whether managers don’t trust LTV calculations because they’re too speculative, or (more likely) are simply focused on short-term results, it’s pretty clear that LTV will not be the primary measurement tool in most organizations. I haven’t quite given up hope that LTV will ultimately receive its due, but for now feel it makes more sense to work with other measures that managers find more compelling.

Read More

So Many Measures, So Little Time

I’ve been collating lists of marketing performance metrics from different sources, which is exactly as much fun as it sounds. One result that struck me was how little overlap I found: on two big lists of just over 100 metrics each, there were only 24 in common. These were fundamental concepts like market share, customer lifetime value, gross rating points, and clickthrough rate. Oddly enough, some metrics that I consider very basic were totally absent, such as number of campaigns and average campaign size. (These are used to measure staff productivity and degree of targeting.) I think the lesson here is that there is an infinite number of possible metrics, and what’s important is finding or inventing the right ones for each situation. A related lesson is that there is no agreed-upon standard set of metrics to start from.

I also found I could divide the metrics into three fundamental groups. Two were pretty much expected: corporate metrics related to financial results, customers and market position (i.e., brand value); and execution metrics related to advertising, retail, salesforce, Internet, dealers, etc. The third group, which took me a while to recognize, was product metrics: development cost, customer needs, number of SKUs, repair cost, revenue per unit, and so on. Most discussions of the topic don’t treat product metrics as a distinct category, but it’s clearly different from the other two. Of course, many product attributes are not controlled by marketing, particularly in the short term. But it’s still important to know about them since they can have a major impact on marketing results.

Incidentally, this brings up another dimension that I’ve found missing in most discussions, which often classify metrics in a sequence of increasing sophistication, such as activity measures, results measures and leading indicators. Such schemes have no place for metrics based on external factors such as competitor behavior, customer needs, or economic conditions--even though such metrics are present in the metrics lists. Such items are by definition beyond the control of the marketers being measured, so in a sense it’s wrong to consider them as marketing performance metrics. But they definitely impact marketing results, so, like product attributes, they are needed as explanatory factors in any analysis.

Read More