Mud Season

Ahh, spring! Mud season in these parts and, as if on queue, Nucleus Research has delivered a small monsoon of data about SAP's (NYSE: SAP) customers and held up a mirror to its latest marketing blitz.

The skinny: SAP claims in a global ad campaign that its customers average 32 percent better profitability than the companies that use Brand X for things like ERP, supply chain management and CRM. Nucleus Research, which has been known to crash and spoil vendors' self-congratulatory parties, has said not so fast.

According to Nucleus, a population of reference customers from SAP's Web site showed they were 20 percent less profitable than their industry peers. What gives?

Read Between the Lines

If all you read is the headline, you might conclude that Nucleus has provided us with a valuable service and taken the big corporation to task. The fine print might reveal something more -- or less.

The issue is that Nucleus did its research on 82 customers that were bragged about on the SAP Web site -- a minuscule portion of SAP's 32,000 customers, about a quarter of 1 percent to be precise -- and, of course, there is the issue of whether or not this is a representative sample.

People who like to see the glass as half empty (and it was cheap whiskey too) will probably say, of course these customers aren't representative. They ought to represent SAP's best if they're referenceable. On the other hand, people who say the glass is half full might point out that no one stays on top forever -- life's vicissitudes and all that.

Not to put too fine a point on all this, but there are also some questionable circumstances. For example, the research group that provided the data on which the ad campaign is based, Stratascope, is headed up by Juergen Kuebler, a former -- you guessed it -- SAP executive. Moreover, the data is culled from a report that has so far not been shared with the public.

It's hard to pick a favorite in this mess. SAP has played fast and loose with its customer data for years, and Nucleus seems to have made a career out of planting a pie in the face of an unsuspecting high flying corporation. I have never been a fan of the idea of splitting the difference on something like this -- of saying, "A pox on both your houses." Maybe it makes me a bad negotiator, or maybe it simply says that things should be run to ground, if possible, and that we shouldn't just give up.

Creative Counting

Several memorable research reports by other firms like Credit Suisse First Boston (CSFB) have shown that in previous years, SAP was so intent on capturing the pole position in the CRM market share race that it invented a novel way of counting deployments. Surprise! SAP became the market share leader. Maybe they really are today, who knows? I don't keep close tabs on market share, because I don't think it measures anything other than sales. I think a lot of people feel that way -- at least in part -- because the definition got blurred. Now the trophy is all but retired.

Nucleus, on the other hand, first came to prominence, at least to me, when it did the same kind of research on the Siebel customer base. Actually, it was a handful of Siebel customers promoted on the Siebel Web site, and the metric then was customer satisfaction.

At that time, Siebel was promoting its customer satisfaction scores, which were in the 90 percent range, as proof of its wonderfulness. It was such a small sample that Nucleus researchers were able to interview the Siebel customers individually. I forget the actual results, but they were substantially at variance with Siebel's 90 percent claims and a small scandal ensued.

At that point I asked Siebel for access to their customers to conduct a statistically valid survey which would, hopefully, get to the bottom of it all. Siebel complied, and what I found was very interesting.

We got over 300 completed responses to our survey, and the data proved Siebel's assertion of 90 percent customer satisfaction. We also conducted a small number of direct interviews at random with Siebel customers. What we found in direct interviews was that although customers were satisfied with their purchases, they were not so enthusiastic when it came to all of the work involved in deploying a new version of what was then client-server software.

Loyalty vs. Satisfaction

From that, we saw the smoldering discontent with client-server software and the evolving importance of loyalty versus simple satisfaction. The interviews constituted what is called a qualitative survey -- not enough data to say much with statistical confidence, but important in a Bob Dylan, "You don't need a weatherman to know which way the wind blows" kind of way. It was an eye-opening experience and it set the stage for a lot that has happened in the years since in areas like on-demand computing and the escalating importance of customer loyalty over satisfaction.

I hate to sound like I am splitting the difference, but I don't know who was right then and I don't know who is right now. Certainly all the players know how to use statistical analysis and, as we all know, you can often get statistics to support your thesis. That's basically where Disraeli's famous quote, "Lies, damn lies, and statistics," comes from.

If there's a lesson in all this, I think it is that when a tool like ERP, supply chain management, or CRM becomes ubiquitous, it ceases to have a major impact on competitive advantage -- you either have it and continue to compete, or you never adopted it and are probably out of business or heading that way.

The winners don't look back because they know the importance of always looking ahead for the next innovation.