Just the other week, IDC star analyst Mike Fauscette wrote a post on a topic near and dear to my heart: What are the right measures of your R&D spend?  I submit this is an extremely important topic for any software company and no less so for companies in the travel space.  Even if you’re not in the business of selling software, technology is increasingly important for travel companies being able to increase bookings, margins and deliver a great experience to their customers.

The context of the post was a presentation he and several other analysts received at the Oracle OpenWorld financial analysts summit.  Oracle was trying to demonstrate their commitment to innovation and keeping their technology at the forefront of the industry…and specifically ahead of SAP and HP.

Fauscette noted that instead of using a common metric –  Percent of Revenue – Oracle used raw spend.  While both are common metrics, I don’t think that either is an effective measure that companies should use in evaluating the effectiveness of their spend.  Mike found both measures wanting as they don’t have any direct linkage to the performance of the underlying business:

“I could spend bunches of $$ and research and develop lots of things that nobody wanted and while my spend as a % of revenue would be very high (and probably increasing as my revenues fell through the floor, at least for a little while), I could never call that success.”

We need to come up with ways to measure the effectiveness of your R&D spend, not just the amount. Specifically, I’d like to help you better pinpoint whether the activities you’re pursuing are helping meet the objectives of the your business or not..

What I’m going to do is talk a little about types of R&D and then discuss what metrics you ought to be using to evaluate what you spent on them.

Why Measure R&D Spend at All?

Good question. If you’re a public company it may be a reporting requirement or a common metric that financial analysts use to forecast your stock price and set their ratings.

Some may say that it’s not even worth doing or measure deeply. They’ll say it’s hard to do. They may use the old line about measuring advertising expenditure: “Half of our budget is working great. I just don’t know which half.”

Or they may just say that they’re staying within their budget, so leave them alone.   This is the single most important reason why I think that Percent of Revenue is the worst possible metric. Many companies “set” their budgets based on a percent of revenue. No other rationale. Now I’ll say this — % of revenue is easy to do and easy to measure, but it doesn’t tell you anything.  Helluva way to run a railroad.

But you’re not one of “them”, right? You’re smarter than that.

Big R, Little r

As I stated before, the metrics you use must support the business objectives you’re trying to achieve. And so you must first understand how your R&D expenditures support those aims.

R&D is a term that is often misused and misunderstood. In the classical sense Research (what I call “Big R”) is an effort to explore and create advanced technology which may or may not have a direct impact on today’s business, while Development is the industrialization of new technology into products for sale.  However many companies mistakenly conflate the two terms to mean the same thing. Thus when many companies refer to R&D, they’re talking mostly about development activities which I’d call “little r”.

Similarly, many companies misuse the term innovation. Clayton Christensen segments innovation into “disruptive” and “incremental”. Disruptive innovations alter the status quo in the industry – think the iPod, the iPhone, Software-as-a-Service (e.g. Salesforce.com) and Cloud Computing.  Incremental innovations are just what they sound like…they move the ball forward, but not dramatically (e.g. Microsoft Office 2010).

The truth is that most companies spend the majority of their resources on bug fixes and feature enhancements, simply trying to hold on to customers and revenues via a traditional upgrade cycle, while trying to convince others (and maybe themselves) that the new versions incorporate many innovations (“New and Improved!”, “Your shirts will be 10% whiter!”). But in most cases these are merely features masquerading as innovation.

How much you spend on Big R v. Little R, or Disruptive v. Incremental innovation are strategic decisions which you must make first.

And there aren’t any hard and fast rules of how much you should be spending, both in the aggregate or on specific products. So much of that depends on:

• Organizational Maturity : e.g. startups should spend much higher proportionally to revenues than an established company)
• Scale: You can’t simply benchmark your % of revenues against Oracle if you’re a $100M company. You may want to compete with larger companies in the marketplace, but don’t enjoy the economies of scale that your competitors may have. So don’t try to benchmark blindly against them.
• Business models: This is the “apples to oranges” discussion.  Different companies have different revenues. A company that’s pins growth on new license sales should look at investment rates differently than a company that’s dependant on software maintenance. And even different still are long tail revenue-based companies, primarily SaaS companies, who use a subscription or usage based model.

But once you have your strategies and objectives in place – and it’s critical that the objectives are tied to achieving over-arching business goals, not merely pursuing technology for technology’s sake –   it’s important to measure the progress you’re making, which leads us to our last section.

What Are the Right Metrics?

There are of course many metrics which can be used in evaluating the effectiveness of your R&D expenditures. Let me name a few, some of which I’ll debunk, others I’ll suggest you add to your list if you don’t already use them:

• Often Used, Marginally Valuable
  • % of revenue: As noted at the top, not really valuable in any way other than a gross and inaccurate way to compare one company’s spend versus another. Or simply a way to build a top-down budget.
  • # of patents: Another often used metric, yet mostly directional in value. Many companies use this metric to try to gauge how “innovative” they are. But the question is really how many of these patents actually impact the business.  Do they help drive revenue or control costs? A patent, or any new feature, that isn’t monetized doesn’t have any intrinsic value and falls into the category of an invention (cool new thing) rather than an innovation (cool new thing that customers want and are willing to pay for).
• Revenue- and Margin-based. This is where it actually gets interesting. Are the fruits of your labor actually improving the health of the business?
  • Revenue: Pretty basic. Are they going up, down or are they stagnant. If it’s either of the latter two it means that you’re either not spending your resources on building products or services that meet your target customers needs.
  • Vitality Index: Revenue is a very gross measure and there are many factors that impact it beyond R&D spend, making it less valuable. So let me introduce you to a concept you likely haven’t heard before, the Vitality Index (VI). VI is a measure of how much of your revenues are driven by products or services introduced in the past year (which are more likely a product of your current R&D spend). The higher your VI score, the greater the direct impact your R&D is having on business growth. The other benefit of this measure is that new products generally return higher margins than older products, so the higher the VI, the better the long term profit prospects of the company.
  • Customer Retention/Churn Rate: This is extremely important as it’s far more costly to acquire a new customer than keep one. It’s also more indicative of energy spent on bug fixes and new feature introduction than disruptive innovations.
• Cost-Based.
  • Cost of Rework as a % of Total Budget: This is a great one because it highlights wasted energy. By definition this activity adds no value to the organization. It may help reduce attrition from angry customers, but it will not add a single customer to the business. To expect that rework should be zero is not reasonable, but like golf, the lower your score, the better. So watch this for trends and use it to identify inefficiencies in your development organization.
  • Defect Injection Rate: The number of total known defects discovered during a product development cycle. This is the flip side to re-work as each of the defects ought to be fixed, although many are often not because they don’t rise to a level of importance (i.e. impact on sales) that merits the effort. But it is an important indicator of the effectiveness of your engineering effectiveness and is what generates the high cost and wasted effort of re-work, as noted above. Then there’s the matter of where those defects came from (bad requirements? bad coding?), but that’s a whole ‘nother post.
  • Defect Leakage: Worse than the number of defects that you find, is the number of defects your customers find.  That is as long as they are still customers. If this happens to frequently you can expect real (negative) impacts to customer satisfaction, customer retention and your corporate reputation as a reliable provider of technology.
  • Variance to Budget by Product/Initiative: Self explanatory, but it’s important to look at the performance at the detail level rather than in the aggregate. It will help you identify underperforming teams.
  • Variance to Release Schedule: Extremely important as missed release dates provide a black eye for the organization and represent lost revenue opportunities that can’t be recovered.  It’s not a strict financial measure but has a direct financial impact on the company.

What’s your POV? Are you using these metrics? Do you have others that you’d like to share? Will you do anything different tomorrow than you did today?

Ness Software Product Labs has a strategic consulting practice that helps organizations evaluate the effectiveness of their R&D operations, both by helping establish a structured metrics program and comparing current processes to industry best practices. The final result is the creation of an actionable plan to enhance software engineering and testing practices tied to expected results.

NB: Hat tip to Dr. Jerry Smith, a former colleague who helped me develop some of my thoughts around R&D metrics and introduced me to the concept of the Vitality Index.

Last week I listened to an interesting podcast by Forrester analysts Mike Gualtieri and Jeffrey Hammond entitled Talent Matters: Why Application Development Cannot Be Industrialized.   At first I thought this could have been an anti-outsourcing discussion, but after listening and thinking about it, it’s really about a deeper understanding of the importance of having exceptional talent in your software engineering organization.  And it’s a concept that any technology-driven business – including travel technology companies, travel suppliers and travel distribution and intermediaries –  need to be thinking long and hard about.

The conversation can be boiled down pretty simply: application development and testing professionals are not a fungible asset.  Deep down (maybe we don’t have to go so deep), we all know this is true, yet every development project is estimated using the productivity of an “average” developer.  And worse yet, they refer to them as resources.  And according to Gualtieri and Hammond, this is where it all goes horribly wrong.  And they’re right.

It doesn’t matter what estimation method you use – Function Point Analysis, analogy-based techniques, parametric models  – these methods are used to estimate the man-months needed to complete the project.  But not all developers are the same.  There are rock-stars, low performers and everything in between.   So while an excellent guide, the accuracy of the estimates really depend on who are the members of the development team.

Art v. Science

In fact the best developers are ‘artists’ to borrow a phrase from Seth Godin’s latest book, “Linchpin”.  The boys from Forrester talk about attributes of the best developers: creative, passionate, disciplined.  But while artists may be the best developers, not all developers are artists. So as an industry, we have tried to put some more structure around the science of software development practices and to make software engineering a “profession”.  We have also leaned on methodologies and tools to enable average developers to deliver above average results.  It’s helped the industry move forward and develop great software.

The fallacy of the software factory

But the mistake that perhaps many firms make is to think that these methodologies and tools would turn their software development teams into software factories.  Nowhere has this approach towards trying to create software factories been more pronounced than in the IT outsourcing business.  That’s because under the traditional outsourcing model success (i.e. margins) is achieved by trying to break any task down into its most basic components so that those activities can be completed by the most junior and cheapest resources (there’s that word again).

This is not to say that methodologies, tools, reusable libraries are not valuable and don’t make software engineering teams more productive. But they’re not a cure-all for having good software engineers. Tools and methodologies are more like guiderails to reduce mistakes and help less-seasoned developers accomplish more advanced tasks, but don’t necessarily guarantee well written, high-performance software..   So I’m in general agreement with their point of view.  But I think it’s important to talk about software engineering maturity.

Software Product Engineering versus IT Application Development

Now let me shift the conversation slightly from a basic application development perspective (which Hammond and Gualtieri were focusing on) to a software product engineering perspective.  Let me start by saying that I believe that software product engineering is a different animal than developing IT applications.  Software products and platforms – whether sold directly to customers or simply provide the infrastructure to deliver your products or services to your customers – requires a complexity, a thoughtfulness, in development, nay architecture and design, that are not often found in applications designed for internal users.  And Forrester agrees. In fact they have written a report about how some companies whose businesses are technology driven, but don’t sell software per se, are starting to try to re-organize their software development organizations in the image of an ISVs R&D organization. Forrester calls these companies ‘product-centric’ and more and more travel companies of all stripes fit this description.

If you need a rule of thumb, think about who’s using the software in question: is it a customer – who may choose to switch to a competitor if they’re not happy with their experience or the performance of the app – or is it a app or other computing resource used by an employee whose only alternative is to find somewhere else to work? If it’s the former, you’re probably talking about an application that needs a more developed software product engineering approach.

Software product engineering activities differ from IT organizations because:

• They are more mature in software engineering practices when measured against maturity models like CMMi
• Adhere to formalized software engineering practices and use of common coding practices
• Recognize, appreciate, and adhere to release schedules
• Quality is of supreme importance.  Testing is important, but quality is designed in.
• QA teams use test planning, test strategy, as opposed to an ad-hoc approach to testing

I do want to avoid painting all IT organizations with too broad a brush. Some IT organizations are more mature than others and have put more investment into putting software engineering practices, focused on architecture and enforced adherence to coding standards across the organization.  However, it’s more likely that you’ll see that with teams whose code is tied closely to generating revenue or supporting the delivery of the company’s product or service to their customers.  They adopt a “product-centric” approach, as Forrester would say, to software development because they realize that bad software negatively impacts the company’s performance.

Talent Is Even More Important in Software Product Engineering Organizations

My firm, Ness Software Product Labs, is not the same as the big ITO firms. Our mission is to help extend and enhance the capabilities of software product engineering organizations – helping companies build and test software that drives revenues – rather than managing internal IT applications and infrastructure.  To accomplish that goal, Ness SPL believes that it requires a different philosophical approach from what you see from the large ITO firms. It informs the type of people we hire, the way we collaborate with our clients and the kinds of relationships and engagement models we employ.

But in the context of this subject, I want to focus on the talent issue again.  Software product engineering is not ITO.  Architecting, designing, building and testing products that are tied to revenue, that require high levels of performance, scalability and resiliency is not a task to be done by lowest-common-denominator individuals. We believe it requires highly talented individuals…artists with discipline.  We spend a lot of time developing best practices, solution accelerators, and identifying the best tools to use.  We spend at least as much time finding the right people and developing their talent and offer them challenges that most ITO firms can’t.  We don’t try to create a software factory, we try to create an environment where talented engineers can thrive.

And the crucible in which we do that work is not for the faint of heart.  We are part of the software engineering organizations of some of the leading technology firms on the planet: Amadeus the number one travel technology provider in the world; NAVTEQ, the leading mapping and location-based services platform, PayPal the leading alternative payments platform, OpenText a billion dollar Enterprise Content Management firm. And the list goes on.  Our approach and the talent that we hire is borne out by the clients who choose us to be a part of their R&D organization, not to run the plumbing.

A lot of stuff happened while I was OpenTravel’s 2010 Advisory Forum in Seattle this past week:

1. I’d be remiss if I didn’t start with the successful IPO of Amadeus (AMA.MC), the largest GDS and leader in global travel technology solutions…and Ness Software Product Labs’ signature client in the travel technology space.
2. Apple buys Siri.  It’s not just mobile-web search, Siri also has the ability to learn preferences and not just provide better results back, but help complete activities or business processes like making dinner reservations or booking a flight.  It should be interesting to see how the iPhone experience evolves as Siri deepens the integration with iPhone OS.
3. Salesforce.com and VMWare announce VMForce.  Looks like Salesforce finally admitted that APEX was holding back the Force.com Platform-as-a-Service business and partnered with VMWare to enable direct Java-based development on the rest of the Force.com infrastructure. Good analysis by Enterprise Irregulars Phil Wainwright, Bob Warfield and Brian Sommer.
4. 5 reasons iPhone v. Android isn’t Mac v. Windows. Well said.
5. Another Top 5 list.  This time it’s the untapped or under-exploited opportunities in travel.  Mobile payments, re-booking top the list, but read on to see what other opportunities lie in wait.
6. Steve Jobs’ official position on Apple v. Adobe Flash.  Nothing terribly new (or not already said by Daring Fireball’s John Gruber), but if it’s written by Steve…and he gives it almost 1700 words, you should at least read it.
7. Excellent analysis of what Apple’s announcement of iAds will mean.
8. Tech Crunch’s Michael Arrington announces that with the recent announcements at the f8 conference we’re officially in the ‘age of Facebook’.
9. Forrester’s JP Gownder offers an opinion on why consumers will pay more for products and uses the Mac as a shining example.  I only have two issues with the analysis.  First I think that Gownder mistakenly equates repeat purchase with brand loyalty.  But more importantly, the piece reads as if the three categories can have similar impact. As I’m in the middle of Steven Sinek’s “Start With Why”, I believe “self-selection” (as Gownder calls it) reason is by far the most influential. But if you read the book, self-selection has a lot to do with whether or not a company has a compelling “why”.
10. On top of the Amadeus IPO, two big mergers in the travel industry:  Hertz buys Dollar Thrifty and United and Continental Airlines are set to merge under the United moniker. And it increasingly looks like Google will buy ITA.  A lot of activity for sure.

Well I finally landed a new gig with Ness Technologies.  While still in the outsourced product development space it’s a bit of a different mission for me.

My task is to leverage the experience Ness has garnered in the travel technology space through it’s long-standing work with Amadeus, a leading travel global distribution system (GDS) provider based in Europe.  Essentially it’s a chance to build a business within a business almost completely on my own.  A heck of a challenge and one I’m looking forward to.  I’m ready to take the leap.

Used with permission from Eric Chan (maveric2003)

The fun will be in figuring out yet another new industry with the travel biz.  Already see that there is a lot of lingo to learn, new buyer titles and profiles and emerging market dynamics.  Should be fun.

The good news for me is that there seems to be a lot of potential to leverage topics and technologies that are already near and dear to me (Cloud, SaaS, SOA, social media).

And while I expect that a lot of the posts on the blog should start reflecting issues in the travel technology space, I will continue to put in my two cents — here, on Twitter and in my new role as contributor to eBizQ — on the goings on in the software business and impact of emerging technologies.  In fact, later today I’ll be heading to the Microsoft New England R&D Center for Cloud Camp Boston and hope to emerge with some insights and video to share.

In an earlier post about the hype around outcome-based outsourcing engagements, I talked about how it’s a great conversation starter, but doesn’t necessarily end up being the way that contracts eventually get written.  Not that it’s about bait-and-switch (although sometimes it is).  But it begs a question: how pervasive is outcome-based outsourcing?

Today I had an excellent, wide-ranging discussion with Forrester analyst Bill Martorelli about outcome-based outsourcing.  Bill is currently working on an upcoming research report about outcomes which he will be presenting at the Forrester Services and Sourcing Forum 2009 and we were comparing notes and I was looking for some feedback on the messaging I’ve developed for my company, Symphony Services.  We had a pretty big disparity in our estimates.  We were talking pretty broadly about the ITO and outsourced product development (OPD) markets, but I said that it’s probably less than 5% of total contracts, definitely less than 10 percent.  Bill thought that it was larger, perhaps twice that, but as we talked about what really counts as true outcome orientation versus simple output-based, fixed-time/fixed-price or SLA-supported contracts, he felt that my number probably wasn’t too bad.

But what this really highlights is that even some of the brightest minds in the business don’t have a good handle on how much of the outsourcing contracts are outcome-based.  What do you think is the right number, or do you think it’s just so much hype?