What is the ROI of Test?


For me, this is a momentous occasion. After being quoted a couple of times in other people’s blogs, I’m finally breaking out on my own. I’m very excited to see where this path will go.

I struggled a bit thinking through what should be the topic of my first post. I finally landed on a topic that got brought up over lunch with a friend of mine in the industry. It was about 4 or 5 years ago. At that time, my boss, who was not in test, was giving open positions more and more to the Development team. I felt that the test schedule, which was already at risk, was going to be impossible to achieve if at least some of those positions weren’t QA. I explained my plight and asked my companion “Given N testers on a team, what is the ROI for the N+1th tester?”   He responded with more than a little cynicism, “Brent, I don’t know. But tell your boss if he wants to see QA’s value, get rid of your testers”.  I was one with his frustration. Those of us in the business know our importance but it’s hard to quantify.

Delivery of quality is a subjective thing and how Testers should do it is just as subjective. It’s a topic for another day, but I’ve begun to collect definitions of a Tester’s Job and, not surprisingly, it’s already amazingly diverse. There are several ways to produce software and make it shippable.  Due to this, it seems a different perspective on the role of QA in the production of software is created for each.

Consider the following for your project:

  1. Customer Resiliency:   How risk adverse is your customer? Will customers be ok with bugs?
  2. Business Resiliency:   How risk adverse is your business team? Is there a window of opportunity  that must be perfect? Do we have to get it right the first time?
  3. Speed: How fast does your team release?  Is it twice a  month or thrice a decade?
  4. Complexity: This is a  measure of the number of components that must interact to make a solid system.
  5. Complication: This is a measure of the design and implementation of the code used to make up a component
  6. Test Confidence: This is a measure of your test collateral and its ability to cover the product.
  7. Test Speed: How long does it take us to use the test collateral to determine state of the product?
  8. Development Skill: What is level of mastery of the craft that exists in your team as a whole?
  9. Testing Skill: What is level of mastery of the craft that exists in your team as a whole?
  10. Ignorance (props to Armour): Are you measuring the above? Do you know what you need to do? Do you know which one is the bottleneck in your team?
  11. Irrational Optimism: How hopeful is the team that the plan is correct and/or on track?

It’s very similar to taking a car trip.   We need to know where we are, where we are going, how we are going to get there, what to pack, etc.  If we make the wrong choices, like travel someplace we haven’t been without a map, or packing only winter clothes for a trip to someplace sunny, or taking the wrong kind of car for the terrain, we could end up paying a lot more than expected to get to our destination.

Your team has all of these chararcteristics in varying degrees.  Depending on where your team stands on each of them has a potential of fundamentally changing the team’s approach to software development.   An obvious example:  if you ship every three years,
you are likely to take a lot more of a preventative approach to bugs and “find ‘em before the customer does”.    However, if you ship every 3 weeks, you will likely focus on taking smart risks with your testing and get better at reducing the time it takes to react to a failure.

When I think of Test, I am not referring to Testers or a Test team per se.   I am referring to what portion of the payroll budget is applied to Testing activities.  While it is easy to chat about things like Dev to Test ratios, figuring out how much of that money should be spent testing vs. developing depends entirely on the culture of your team.  If your team registers high in all of the negative things, you may need to spend *a lot* in order to mitigate the risk and save your business.

Accordingly, Testing is a form of Software Business Insurance.   The more coverage you have the less at-risk you are.  So there’s really only one answer to my question.   It Depends!   Coverage mitigates risk.   If you don’t feel at-risk, then, until proven otherwise, coverage will not be valuable to you.

One day, perhaps, someone from the Actuarial world will join us in Testing and show us how to determine this more precisely, or maybe outsourced Testing companies will model their fee structure after the insurance industry.   Until then, in order to hire the next tester, you should be prepared to articulate the risk you see in your project and the value your business will gain in comparison to the options.  Back to the car trip metaphor, if you know the person driving the car isn’t a great driver, you could mitigate the risk and buy supplemental insurance or you could send the person to Better Driving School.

I recently met up with my friend. His group has recently undergone a major reorg and QA got trimmed back in a big way. It seems the group’s Head looked at the cost of the QA team and asked “What do these guys do?”  Worry not for my friend, though.   He’s taken his entire team and is going to Development.   His team has strong testing and development skills and many of the other characteristics above are positives for him.  I doubt he’ll need much supplemental coverage.

Advertisements

10 thoughts on “What is the ROI of Test?

  1. Re: 1. Customer Resiliency: How risk adverse is your customer? Will customers be ok with bugs?

    If you’re doing this, it’s also worth considering: is today’s customer *exactly the same* as the customer your sales team is selling to *right now*?

  2. Good post. I like the idea of comparing testing with insurance. But which customers would be okay with an application which is not done right, the first time?

    • @testerab I totally agree!

      Nitin, the way I interpret the spirit of your question, I would answer it “No customer ever.” One of my favorite quotes (I wish I knew who to credit for it) “Cheap, Fast, or Good: Pick 2”. If we focus on quality and timeliness, that will necessate applying more resources to bear to analyze the customer, build up the right test collateral, deploy and test thoroughly, etc. This could make the product’s price too high to be valued.

      However, one phenomenon we’ve noticed in the new web services world is that the customer is ok with lower than expected quality if, amongst other things. they feel like they are an *active* part of the design team. “Hey, yesterday, I told them it sucked and today, it’s fixed… They even thanked me for the bug report.” These customers are likely to be more interested in how quickly can you fix it vs. how thorough were you at prevention in the first place.

  3. At a very basic level, the return for development and testing on the N+1th person is just the next set of tasks in your prioritized to-do list. I find that when some people analyze the return, they assume that the next development tasks will produce a better return, but they question whether the testing tasks will produce a better return. Does adding the N+1th feature really earn your company more money? Can you really quantify the number of additional users and/or purchases your product will process with the N+1th feature? Probably not. It’s almost as hard as determining whether completing N+1 testing tasks will delight the customers even more and produce more customers or purchases.

    • Mike, thank you for your comments and your insight. It has also been my experience that folks will look at the Dev’s workload (Coding only) with an eye to either growing a return or growing market share (or both) and then use that same eye to judge the value of the testing tasks. I think those of us in the craft need to change the discussion. Testing is more about mitigating risk. Testing tasks do not inheritly add to the upside; they minimize the downside of where your business assumptions may be wrong. If your business believes it’s got all of it’s characteristics aligned, it should feel comfortable decreasing that investment.

  4. Great to see you blogging, Brent. This was a timely read for me, as I am working through some of the same questions as we speak. Looking forward to reading more of your thoughts and fantastic insight!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s