Salespeople: One Question Matters Most

Back in December, the Sales Lead Management Association and LEADTRACK published a survey on lead management practices that I haven’t previously had time to write about. (The survey is still available on the SLMA Web site.) It contained 10 questions, which is about as many as I can easily grasp.

The two clearest answers came from questions about the information salespeople want and why they don’t follow up on inquiries. By far the most desired piece of information about a lead was purchasing time frame: this was cited by 41% of respondents, compared with budget (17%), application (15%), lead score (15%) and authority (12%). I guess it’s a safe bet that salespeople jump quickly on leads who are about to purchase and pretty much ignore the others, so this finding strongly reinforces the need for nurturing campaigns that allow marketers to keep in contact with leads who are not yet ready to buy.

Note that none of listed categories included behavioral information such as email clickthroughs or Web page visits, which demand generation vendors make so much of. I doubt they would have ranked highly had they been included. Although behavioral data provides some insights into a lead’s state of mind, it's useful to be reminded that wholly pragmatic facts about time frame are a salesperson's paramount concern.

The other clear message from the survey was that the main reason leads are not followed up is “not enough info”. This was cited by 55% of respondents, compared with 14% for “inquired before, never bought”, 12% for “no system to organize leads”, 10% for “no phone number”, 7% for "geo undesirable" and 2% because of "no quota on product". This is an unsurprising result, since (a) good information is often missing and (b) salespeople don’t like to waste time on unqualified leads. Based on the previous question, we can probably assume that the critical piece of necessary information is time frame. So this answer reinforces the importance of gathering that information and passing it on.

One set of answers that surprised me a bit were that 77% or 80% of salespeople were working with an automated lead management system, either “CRM/lead management” or “Software as a Service”. I’ve given two figures because the question was purposely asked two different ways to check for consistency. The categories don’t make much sense to me because they overlap: products like Salesforce.com are both CRM systems and SaaS. Still, this doesn't affect the main finding that nearly everyone has some type of automated system to “update lead status” and “manage your inquires” (the two different questions that were asked). This is higher market penetration than I expected, although I do recognize that those questions deal more with lead management (a traditional sales automation function) than lead generation (the province of demand generation systems). Still, to the extent that CRM systems can offer demand generation functions, there may be a more limited market for demand generation than the vendors expect.

One final interesting set of figures had to do with marketing measurement. The survey found that 23% of companies measure ROI for all lead generation tactics, 30% measure it for some tactics, and 47% don’t measure it at all. The authors of the survey report seem to find these numbers distressingly low, particularly in comparison with the 80% of companies that have a system in place and, at least in theory, are capturing the data needed for measurement. I suppose I come at this from a different perspective, having seen so many surveys over the years showing that most companies don’t do much measurement. To me, 23% measuring everything seems unbelievably high. (For example, Jim Lenskold's 2008 Marketing ROI and Measurements Study found 26% of respondents measured ROI on some or all campaigns; the combination of "some" and "all" in the SLMA study is 53%.) Either way, of course, there is plenty of room for improvement, and that's what really counts.

Read More

New Best Practices White Paper

As promised, I've written a white paper with the 37 Marketing Automation Best Practices listed in last week's post. This is in the Resources section of the Raab Guide to Demand Generation Systems site; you have to register and then log in. Registration is free.

Read More

Best Practices for Marketing Automation and Demand Generation Campaigns

I enjoyed my little presentation on BrightTalk last Wednesday, which you can still view by clicking here. (If that doesn’t work, go to the BrightTalk site and key my name into the site search function. This will also bring up a roundtable discussion from Tuesday, which I think was interesting as well.) The BrightTalk platform itself worked nicely and was about as simple as possible. They offer a limited version for free (one 30 minute Webinar per month), which is worth considering if you’d like to dip your toe into this sort of thing. The next level up is $949 per month, which is rather pricey compared with $99 per month for Go To Webinar, a platform we’ve used here which does roughly the same thing. I'm not saying they're identical: BrightTalk lets you upload your slides rather than sharing the screen of your PC, which makes it more reliable, and seems to offer some promotional services too. So you’d want to look more closely at both paid services before making a choice.

But I digress. The heart of my presentation on Wednesday ended up as a list of 37 “best practices” for marketing automation / demand generation programs. I’ll probably embed them in a white paper for the Raab Guide Web site in the near future, but for now I thought I’d share them here. (If you want the full slide deck, complete with moderately witty speaker notes, drop me at email at draab@raabassociates.com.)

A bit of context: the presentation listed a sequence of steps for marketing campaign creation, deployment and analysis. The best practices are organized around those steps.

Step 1: Gather Data. The marketer assembles information about the target audience. Best practices here involve the types of data, and, in particular, expanding beyond traditional sources.
• leads, promotions, responses, orders: these are the traditional data sources used in most marketing systems. Best practices would link actual orders back to the individual leads, and do accurate customer data integration.

• external demographics, preferences, contact names: the best practice here is to supplement internal data with external sources such as D&B, Hoovers, LexisNexis, ZoomInfo, etc. More information allows more accurate targeting and better lead scoring.

• social networks: these can be another source of contact names, and sometimes of introductions via mutual friends. A close look at what individuals have said and done in these networks could provide deep insight into a particular person’s needs, attitudes and interests, but this is more an activity for salespeople than marketers.

• summarized activity detail: marketing systems gather an overwhelming mass of detail about prospect activity, down to every click on every Web page. Best practice is to make this more usable by flagging summaries such as “three visits in the past seven days” and making them available for segmentation and event-triggered marketing.

• self-adjusting surveys: once a lead has answered a survey question, the system should automatically replace that question with new one. This builds a richer customer profile and avoids annoying the lead by asking the same question twice. For a bonus best practice, the system should choose the next question based on user-defined rules that select the most useful question for each individual.

• order detail, payments, customer service: the best practice is to gather information beyond the basic order history from operational systems. This also allows more precise targeting and may uncover opportunities that are otherwise invisible, such as follow-up to service problems.

• near real-time updates: fast access to information about lead behaviors allows quick response, particularly in the form of event-triggered messages. This can be critical to engaging a prospect when her interest is at its peak, and before she turns to a competitor.

• household and company levels: consumers should be grouped by households and business leads by their company, division or project. This grouping permits selections and scoring based on activity of the entire group, which may display patterns that are not visible when looking at just a single individual.

Step 2: Design Campaign. The marketer now designs the flow of the campaign itself. Traditional marketing programs use a small number of simple campaigns, each designed from scratch and often used just once. Even traditional campaigns often include multiple steps, so this itself isn’t listed separately as a new best practice.

• many specialized campaigns: the best practice marketer deploys many campaigns, each tailored to a specific customer segment or business need. These are more effective because they are more tightly targeted.

• cross sell, up sell and retention campaigns: demand generation focuses primarily on campaigns to acquire new leads. The best practice is to supplement these with campaigns that help sell more to existing customers and to retain those customers. Marketing automation has generally included these types of campaigns, at least in theory, but many firms could productively expand their efforts in these areas.

• share and reuse components (structure, rules, lists): when marketers are running many specialized campaigns, they have greater opportunity to share common components, and greater benefit from doing so. Sharing makes it possible to build more complex, sophisticated components and to ensure consistency both in how each customer is treated and in how company policies are implemented.

• new channels (search, Web ads, mobile, social): these new channels are often more efficient than traditional channels, and many have other benefits such as being easier to measure. Best practice marketers test new channels aggressively to find out what works and how they can best be used. Even if the new channels are not immediately cost-effective, marketers can limit their investment but still build some experience that will be useful later.

• multiple channels in same campaign: true multichannel campaigns contact customers through different media. A mix of media allows you to reach customers who are responsive in different channels, thereby boosting the aggregate response. Channels may also be chosen based on stated customer preferences and the nature of a particular contact. Marketing automation systems make it easy to switch between media within a single campaign.

Step 3: Develop Content. This step creates the actual marketing materials needed by each step in the campaign design. These are emails, call scripts, landing pages, brochures, and so on.

• rule-selected content blocks and best offers: content is tailored to individuals not simply by inserting data elements (“Dear [First_Name]” but by executing rules that select different messages based on the situation. For example, a rule might send different messages based on the customer’s account balance.

• map drip-marketing message to buyer stage: best practice nurturing campaigns deliver messages that move the lead through a sequence of stages, typically starting with general information and becoming more product oriented. This is more effective than sending the same message to everyone or always sending product information.

• standard templates: messages are built using standard templates that share a desired look-and-feel and contain common elements such as headers and footers. This provides consistency, saves work, and ensures that policies are followed.

• share and reuse components (items, content blocks, images): like shared campaign components, shared marketing contents minimize the work needed to create many different, tailored campaigns. Sharing also makes it easy to deploy changes, such as a new price or new logo, without individually modifying every item in every campaign.

• unified content management system across channels: even though most marketing materials are channel-specific, many components such as images and text blocks can in fact be shared across different channels. Managing these through a single system further saves work, supports sharing, and ensures consistency.

Step 4: Execute Campaign. The campaign is deployed to actual customers. Best practice campaigns often run continuously, rather than being executed once and then replaced with something new. This lets marketers refine them over time, testing different treatments for different conditions and keeping the winners.

• separate treatments by segment: messages and campaign flows are tailored to the needs of each segment. This could be done by creating one campaign with variations for different segments or by creating separate campaigns for each segment. Which works best depends largely on your particular marketing automation system. Either way, shared components should keep the redundant work to a minimum.

• statistical modeling for segmentation: predictive model scores can often define segments more accurately than manual segmentations. Perhaps more important, they can be less labor-intensive to create, allowing marketers to build more segments and rebuild them more often. This matters because best practice marketing involves so many specialized campaigns and is constantly adjusting to new conditions.

• change campaign flow based on responses, events, activities: best practice campaigns change lead treatments in response to their behaviors. Thus, instead of a fixed sequence of treatments, they send leads down different branches and, in some cases, move them from one campaign to another. Changes may be triggered by activities within the campaign, such as response to a message or data provided within a form, or by information recorded elsewhere and reported to the marketing automation system.

• advanced scoring (complex rules, activity patterns, event depreciation, point caps): simple lead scoring formulas are often inaccurate predictors of future behavior. Best practice scoring may involve complex calculations based on relationships among several data elements, summarized activity detail, reduced value assigned to less recent events, and caps on the number of points assigned for any single type of activity. A related challenge for system designers is making complex formulas reasonably easy to set up and understand.

• company-level scores and activity tracking: the best practice campaign can use aggregated company or household data to calculate scores, guide individual treatments, and issue alerts. This allows more appropriate treatment than looking at each individual in isolation.

• multiple scores per lead: for companies with several products, the best practice to calculate a separate lead score for each. The scores may also have different thresholds for sending the lead to sales.

• define score formula jointly with sales: the salesperson is the ultimate judge of whether a lead is qualified. But many marketing departments still set up lead scoring formulas without sales input. Best practice is to work together on defining the criteria and then to periodically review the results to see if the formula can be improved.

• let sales return leads for more nurturing: traditional lead management is a one-way street, with leads sent from marketing to sales and then never heard from again. Best practice marketers allow salespeople to return leads to marketing for further nurturing. This improves the chances of a lead ultimately making a purchase, even if it doesn’t happen right away.

Step 5: Analyze Results. Learning from past campaigns may be the most important best practice of all. Having many targeted campaigns allows for continuous incremental improvement, achieved by quickly evaluating the return on each project and adjusting future programs based on the results.

• advanced response attribution: traditional methods often credit a lead to whichever campaign contacted them first, or whichever generated the first response. Best practice marketers look more deeply at the factors which may have influenced a lead’s behavior, often applying sophisticated analytics to estimate the incremental impact of different campaigns.

• standard metrics, within and across channels: resources can only be allocated to their optimal use if return on investment can be compared across campaigns. This requires standard metrics, which must be calculated consistently and clearly understood throughout the organization.

• formal test designs (a/b, multivariate): traditional marketers often do little testing, and the tests they do are often poorly designed. Best practice marketing involves continuous, formal testing designed to answer specific questions and lead to actionable results.

• capture immediate and long-term results: initial response rate or cost per lead fails to take into account the value of the leads generated, which can differ hugely from campaign to campaign. Best practice requires measuring the long-term value and building it into standard campaign metrics.

• evaluate on customer profitability, not revenue: customers with the same revenue can vary greatly in the actual profit they bring to the company, depending on the profit margins of their purchases and other costs such as customer support. Best practice metrics include accurate profitability measures, preferably drawn from an activity-based costing system.

• continually assess and reallocate spending: best practice marketers have a formal process to shift resources to the most productive marketing investments. These will change as campaigns are refined, business conditions evolve, and new opportunities emerge. A formal assessment process is essential because organizations otherwise tend to resist change.

Infrastructure. Individual campaigns are made possible by an underlying infrastructure that has best practices of its own.

• consolidated systems (multi-channel content management, campaign management and analytics): today’s marketing systems can usually handle multiple channels, so decommissioning older channel-based systems may save money as well as making multi-channel campaigns easier to execute. Consolidated multi-channel analytics, which may occur outside of the marketing automation system, are particularly important for gaining a complete view of each customer.

• advanced system training: marketing departments often provide workers with the minimum training needed to gain competency in their tools. Best practice departments recognize that additional training can make users more productive, particularly as the tools themselves add new capabilities that users would otherwise not be able to exploit.

• advanced analytics training: analytics play a central role in the continuous improvement process. Solid analytics training ensures that users can set up proper tests and interpret the results. Because data and tools are often already available, lack of training is frequently the main obstacle that prevents marketers from using analytics effectively.

• formal processes: best practice marketers develop, document and enforce formal, consistent business processes. This both ensures that work is done efficiently and makes it possible to execute changes when opportunities arise.

• cross-department cooperation: working with sales, service, finance and other departments is essential to sharing systems, data and metrics. A cross-department perspective ensures that each department considers the impact of its decisions on the rest of the company and on the customers themselves.

Summary

The best practice vision is many marketing campaigns, each precisely targeted, efficiently executed, and carefully designed to yield the greatest possible value. The campaigns are supported by detailed analysis to understand results and identify potential improvements. This information is quickly fed into new campaigns, ensuring that the company continually evolves its approaches and makes the best possible use of marketing resources. Continuous optimization is the ultimate best practice that all other practices should support.

Read More

Company-Level Data in Demand Generation Systems

I had an interesting email conversation last month with a Raab Guide buyer about the nuances of company-level data management in demand generation systems. He started from the perfectly reasonable premise that the demand generation system should give an overview of activity for all leads associated with a given company. This is a topic we do cover in the Guide, but not in the detail he needed. The conversation has sharpened my own thinking on the subject and prompted a couple of conversations with vendors. I think it’s worth discussing here in some detail.

There are really two separate issues at play. The first is how the demand generation system treats company-level data. This, to start at the very beginning, is data about the company associated with an individual. Typically this is the company they work for, although there might be another relationship such as consultant or services vendor. From a database design standpoint, having one company record that is linked to multiple individuals avoids redundant data and, therefore, potential inconsistencies between company data for different people. (The technical term for this is “normalization”, meaning that the database is designed so a given piece of information is stored only once.) Some of the demand generation systems do indeed have a separate company table: it is one of the marks of a sophisticated design.

But the wrinkle here is that most CRM systems in general, and Salesforce.com in particular, also have separate company and individual levels. In fact, Salesforce.com actually has two types of individuals: “leads” which are unattached individuals, and “contacts”, which are individuals associated with an “account” (typically a company, although it might a smaller entity such as a division or department). Most demand generation systems make little distinction between CRM “leads” and “contacts”, converting them both to the same record type when data is loaded or synchronized.

The common assumption among demand generation vendors is that the CRM system is the primary source of information for which individuals are associated with which companies and for the company information itself. This makes sense, given that the salespeople who manage the CRM data are much closer to the companies and individuals than the marketers who run the demand generation system. Demand generation systems therefore generally import the company / individual relationships as part of their synchronization process. Systems with a separate company table store the imported company data in it; those without a separate company table copy the (same) company data onto each individual record. So far so good.

However, this raises the question of whether the demand generation system should be permitted to override company / individual relationships defined in the CRM system or to edit the company (or, for that matter, individual) data itself. I have to get back to the vendors and ask the question, but I believe that most vendors do NOT let the demand generation system assign individuals to companies or change the imported relationships. (In at least some cases, users can choose whether or not to allow such changes.) Interestingly enough, these limits apply even to systems that infer a visitor’s company from their IP address and show it in reports. Whether demand generation can change company-level data and have those changes flow back into the CRM system is less clear: again, it may be matter of configuration in some products. The vendors who don’t provide this capability will argue, probably correctly, that few marketers really want to do this and or in fact should do it.

So what information DOES originate in the demand generation system? Basically, it is new individuals, which correspond to “lead” records in Salesforce.com, and attributes for existing individuals, which may relate to either CRM “leads” or “contacts”. The demand generation system may also capture company information, but this is stored on the individual record and kept distinct from the company information in the CRM system. When a new individual is sent from demand generation to the CRM system, it is set up in CRM as a “lead” (that is, unattached to a company). The CRM user can later convert it to a contact within an account. But the demand generation system cannot generally set up accounts and contacts itself. (Again, let me stress that there may be some exceptions—I’ll let you know when I find out.)

Bottom line: leads, lead data and contact data may originate in either demand generation or CRM, and the synchronization is truly bi-directional in that changes made in either system will be copied to the other. But accounts and account / contact relationships are only maintained in the CRM system: most demand generation systems simply copy that data and don’t allow changes. Thus, it is essentially a unidirectional synchronization.

This leads us to the second issue, which is company-level reporting. It’s touted by most demand generation vendors, but a closer look reveals that some actually rely on the account-level reporting in Salesforce.com to deliver it. This isn’t necessarily a problem, since the Salesforce.com report can incorporate activities captured by the demand generation system. Per the earlier discussion, these activities will be linked to individuals (leads or contacts in Salesforce.com terms). Of course, if the individuals have not been linked to a company in the CRM system, they cannot be included in company-level reports.

One problem with relying on CRM for company-level reporting is that the demand generation system may have some nice reporting capabilities that you’d want to use. So there is some advantage to capturing the company / individual links within the demand generation system, even if the links themselves are created in CRM. (Not to get too technical here, but those “links” could simply be a company ID on the individual record, even if there is no separate company-level table in the data model. So lacking a company table does not preclude company-level reporting.)

A second issue is that some marketers want to use company-level information in their lead scoring calculations. This is another question we ask explicitly in the Guide, and only some companies say they can do it. Whether they make it simple is another question—some products who claim the facility in fact require considerable effort to make it happen. Again, the vendors who don’t offer this would presumably say that it isn’t very important to their clients.

I hope this clarifies both the mechanics of handling company-level data and some of underlying business issues. It’s one of those aspects of demand generation systems where vendor differences are real but subtle, and where the true importance of those differences is clear only to experienced users. All I can do in the Guide and this blog is try to describe things as clearly and accurately as possible, and thereby help you to make sound judgments about your own business needs.

Read More

Raab Marketing Automation Webinar on January 14

Just a brief note to let you know that I'll be giving a Webinar on "How to Get the Most Value from Your Marketing Automation System" on Wednesday, January 14 at 4 p.m. Eastern / 1 p.m. Pacific. You can register at http://www.brighttalk.com/webcasts/2017/attend This is part of a two-day Marketing Automation Summit organized by BrightTALK, a vendor of webcasting technology that sponsors channels of specialized content, as well as running regular online conferences. As near as I can tell, BrightTALK makes its money selling the technology and related services: according to their Web site, you can have your own "channel" with unlimited webcasts for $949 per month. Whether their conferences are simply a way to promote this or have another revenue stream attached, I don't know.

In any case, my presentation on the 14th will look at short-term ways of getting value from existing systems, on the theory that money for new systems will be scarce in the near future. This is a bit of a switch from my usual focus on system selection. But system selection always boils down to how the system will be used, so it isn't much of a stretch.

Tune in if you have a chance, and if you're, say, recovering from a skiing accident over the holidays, you can listen to the full lineup of sessions starting on January 13. I'll also be participating in a round table discussion at 7 p.m. Eastern / 4 p.m. Pacific on the 13th.

Read More

ADVIZOR's In-Memory Database Supports Powerful Visualization

Back when I was writing a great deal about QlikView, I proposed that its fundamental value came from empowering business analysts to do work for themselves that would otherwise require IT support. (See, for example, this post, which has the virtue of pretty graphics.) This same notion of considering which users do which work has permiated my ideas of usability measurement for demand generation systems and usability in general. But to get back specifically to business intelligence systems, I think there is a particularly large gap between the capabilities available to business analysts and those available to IT. That is, even though the business intelligence systems like Cognos and Business Objects give analysts many ways to slice and present prepared data, they do not let analysts add new data or restructure existing data to meet new needs. This still requires the IT staff to design new data cubes and loading processes.

This gap is partly filled by analytical technologies such as columnar systems and database appliances, which can give good performance without schemas tailored for each task. But those systems are purchased and managed by the IT department, so they still leave analysts largely reliant on IT’s tender mercies.

A much larger portion of the gap is filled by products like QlikView, which the analysts can largely control for themselves. These can be divided into two subcategories: database engines like QlikView and illuminate, and visualization tools like Tableau and TIBCO Spotfire. The first group lets analysts do complex data manipulation and queries without extensive data modeling, while the latter group lets them do complex data exploration and presentation without formal programming. This distinction is not absolute: the database tools offer some presentation functions, and the visualization tools support some data manipulation. Both capabilities must be available for the analysts the work independently.

This brings us to ADVIZOR from ADVIZOR Solutions. ADVIZOR features an in-memory database and some data manipulation, but its primary strength is visualization. This includes at least fifteen chart types, including some with delightfully cool names like Multiscape, Parabox and Data Constellations. Analysts can easily configure these by selecting a few menu options. The charts are also somewhat interactive, allowing users to select records by clicking on a shape or drawing a box around data points. Some settings can be changed within the chart, such as selecting a measure to report on. Others, such as specifying the dimensions, require modifying the chart setup. The distinction won’t matter much to business analysts, who will have the tools build and modify the charts. But final consumers of the analyses typically run a viewer that does not permit changes to the underlying graph configuration.

On the other hand, that in-memory database can link several charts within a dashboard so selections made on one chart are immediately reflected in all others. This is arguably the greatest strength of the system, since it lets users slice data across many dimensions without writing complex queries. Colors are also consistent from one chart to the next, so that, for example, the colors assigned to different customer groups in a bar chart determine the color of the dot assigned to each customer in a scatter plot. Selecting a single group by clicking on its bar would turn the dots of all the other customers to gray. Keeping the excluded records visible in this fashion may yield more insight than simply removing them, although the system could also do that. These adjustments appear almost instantly even where millions of records are involved.

Dashboards can easily be shared with end-users through either a zero-footprint Web client or a downloadable object. Both use the Microsoft .NET platform, so Mac and Linux users need not apply. Images of ADVIZOR dashboards can easily be exported to Office documents, and can actually be manipulated from within Powerpoint if they are connected to the underlying dashboard. It’s also easy to export results such as lists of selected records.

Circling back to that database: it employs technology developed at Bell Labs during the 1990’s to support interactive visualization. The data model itself is a fairly standard one of tables linked by keys. Users can import data from text files, relational databases, Excel, Access, text files, Business Objects or Salesforce.com. They can map the table relationships and add some transformations and calculated fields during or after the import process. Although the mapping and transformations are executed interactively, the system records the sequence so the user can later edit it or repeat it automatically.

The import is fairly quick: the vendor said that an extract of three to four gigabytes across thirty tables runs in about twenty minutes, of which about five minutes is the build itself. The stored data is highly compressed but expands substantially when loaded into RAM: in the previous example, the three to four GB are saved as a 70 MB project file, but need 1.4 GB of RAM. The current version of ADVIZOR runs on 32 bit systems which limits it to 2-4 GB of RAM, although a 64 bit version is on track for release in January 2009. This will allow much larger implementations.

Pricing of ADVIZOR starts at $499 for a desktop version limited to Excel, Access or Salesforce.com source data and without table linking. (A 30-day trial version of this costs $49.) The full version starts at around $10,000, with additional charges for different types of user seats and professional services. Few clients pay less than $20,000 and a typical purchase is $50,000 to $60,000. Most buyers are business analysts or managers with limited technical skills, so the company usually helps set up their initial data loads and applications. ADVIZOR was introduced in 2004 and has several thousand end users, with a particular concentration in fund-raising for higher education. The bulk of ADVIZOR sales come through vendors who have embedded it within their own products.

Read More

Simplifying Demand Generation Usability Assessment: No Obvious Answers

My feelings are hurt, people. No one has commented on last week’s post about usability measurement. I know it’s not the world’s most fascinating topic but I really wanted some feedback. And I do, after all, know how many people visit the site each day. Based on those numbers, there are a lot of you who have chosen not to help me.

Oh well, no grudges here -- ‘tis the season and all that. I’m guessing the reason for the lack of comment is that the proposed methodology was too complex for people to take the time to assess and critique. In fact, the length of the post itself may be an obstacle, but that in turn reflects the complexity of the approach it describes. Fair enough.

So the question is, how do you simplify the methodology and still provide something useful?

- One approach would be to reduce the scope of the assessment. Instead of defining scenarios for all types of demand generation processes, pick a single process and just analyze that. Note that I said “single” not “simple”, because you want to capture the ability of the systems to do complicated things as well. This is a very tempting path and I might try it because it’s easy to experiment with. But it still raises all the issues of how you determine which tasks are performed by which types of users and how you account for the cost of hand-offs between those users. This strikes me as a very important dimension to consider, but I also recognize that it introduces quite a bit of complexity and subjectivity into the process. I also recognize that measuring even a single process will require measuring system set-up, content creation and other preliminary tasks. Thus, you still need to do a great deal of work to get metrics on one task, and that task isn’t necessarily representative of the relative strengths of the different vendors. This seems like a lot of effort for a meager result.

- Another approach would be to ask users rather than trying to run the tests independently. That is, you would do a survey that lists the various scenarios and asks users to estimate the time they require and how this is distributed among different user types. That sounds appealing insofar as now someone else does the work, but I can’t imagine how you would get enough data to be meaningful, or how you would ensure different users’ responses were consistent.

- A variation of this approach would be to ask vastly simpler questions – say, estimate the time and skill level needed for a half-dozen or so typical processes including system setup, simple outbound email campaign, setting up a nurturing campaign, etc. You might get more answers and on the whole they’d probably be more reliable, but you’re still at the mercy of the respondents’ honesty. Since vendors would have a major incentive to game the system, this is a big concern. Nor is it clear what incentives users would have to participate, how you screen out people with axes to grind, or whether users would be constrained by non-disclosure agreements from participating. Still, this may be the most practical of all the approaches I’ve come up with, so perhaps it’s worth pursuing. Maybe we get a sample of ten clients from each vendor? Sure they’d be hand-picked, but we could still hope their answers would accurately reflect any substantial differences in workload by vendor.

- Or, we could ask the vendors to run their own tests and report the results. But who would believe them? Forget it.

- Maybe we just give up on any kind of public reporting, and provide buyers with the tools to conduct their own evaluations. I think this is a sound idea and actually mentioned it in last week’s post. Certainly the users themselves would benefit, although it’s not clear how many buyers really engage in a detailed comparative analysis before making their choice. (For what it’s worth, our existing usability worksheet is the third most popular download from the Raab Guide site. I guess that’s good news.) But if the results aren’t shared, there is no benefit to the larger community. We could offer the evaluation kit for free in return for sharing the results, but I doubt this is enough of an incentive. And you’d still have the issue of ensuring that reported results are legitimate.

So there you have it, folks. I’m between a rock and a hard place. No matter how much I talk about usability, the existing Raab Guide mostly lists features, and people will use it to compare systems on that basis. But I can’t find a way to add usability to the mix that’s both objective and practical. Not sure where to go next.

Read More