For me, this is a momentous occasion. After being quoted a couple of times in other people’s blogs, I’m finally breaking out on my own. I’m very excited to see where this path will go.
I struggled a bit thinking through what should be the topic of my first post. I finally landed on a topic that got brought up over lunch with a friend of mine in the industry. It was about 4 or 5 years ago. At that time, my boss, who was not in test, was giving open positions more and more to the Development team. I felt that the test schedule, which was already at risk, was going to be impossible to achieve if at least some of those positions weren’t QA. I explained my plight and asked my companion “Given N testers on a team, what is the ROI for the N+1th tester?” He responded with more than a little cynicism, “Brent, I don’t know. But tell your boss if he wants to see QA’s value, get rid of your testers”. I was one with his frustration. Those of us in the business know our importance but it’s hard to quantify.
Delivery of quality is a subjective thing and how Testers should do it is just as subjective. It’s a topic for another day, but I’ve begun to collect definitions of a Tester’s Job and, not surprisingly, it’s already amazingly diverse. There are several ways to produce software and make it shippable. Due to this, it seems a different perspective on the role of QA in the production of software is created for each.
Consider the following for your project:
- Customer Resiliency: How risk adverse is your customer? Will customers be ok with bugs?
- Business Resiliency: How risk adverse is your business team? Is there a window of opportunity that must be perfect? Do we have to get it right the first time?
- Speed: How fast does your team release? Is it twice a month or thrice a decade?
- Complexity: This is a measure of the number of components that must interact to make a solid system.
- Complication: This is a measure of the design and implementation of the code used to make up a component
- Test Confidence: This is a measure of your test collateral and its ability to cover the product.
- Test Speed: How long does it take us to use the test collateral to determine state of the product?
- Development Skill: What is level of mastery of the craft that exists in your team as a whole?
- Testing Skill: What is level of mastery of the craft that exists in your team as a whole?
- Ignorance (props to Armour): Are you measuring the above? Do you know what you need to do? Do you know which one is the bottleneck in your team?
- Irrational Optimism: How hopeful is the team that the plan is correct and/or on track?
It’s very similar to taking a car trip. We need to know where we are, where we are going, how we are going to get there, what to pack, etc. If we make the wrong choices, like travel someplace we haven’t been without a map, or packing only winter clothes for a trip to someplace sunny, or taking the wrong kind of car for the terrain, we could end up paying a lot more than expected to get to our destination.
Your team has all of these chararcteristics in varying degrees. Depending on where your team stands on each of them has a potential of fundamentally changing the team’s approach to software development. An obvious example: if you ship every three years,
you are likely to take a lot more of a preventative approach to bugs and “find ‘em before the customer does”. However, if you ship every 3 weeks, you will likely focus on taking smart risks with your testing and get better at reducing the time it takes to react to a failure.
When I think of Test, I am not referring to Testers or a Test team per se. I am referring to what portion of the payroll budget is applied to Testing activities. While it is easy to chat about things like Dev to Test ratios, figuring out how much of that money should be spent testing vs. developing depends entirely on the culture of your team. If your team registers high in all of the negative things, you may need to spend *a lot* in order to mitigate the risk and save your business.
Accordingly, Testing is a form of Software Business Insurance. The more coverage you have the less at-risk you are. So there’s really only one answer to my question. It Depends! Coverage mitigates risk. If you don’t feel at-risk, then, until proven otherwise, coverage will not be valuable to you.
One day, perhaps, someone from the Actuarial world will join us in Testing and show us how to determine this more precisely, or maybe outsourced Testing companies will model their fee structure after the insurance industry. Until then, in order to hire the next tester, you should be prepared to articulate the risk you see in your project and the value your business will gain in comparison to the options. Back to the car trip metaphor, if you know the person driving the car isn’t a great driver, you could mitigate the risk and buy supplemental insurance or you could send the person to Better Driving School.
I recently met up with my friend. His group has recently undergone a major reorg and QA got trimmed back in a big way. It seems the group’s Head looked at the cost of the QA team and asked “What do these guys do?” Worry not for my friend, though. He’s taken his entire team and is going to Development. His team has strong testing and development skills and many of the other characteristics above are positives for him. I doubt he’ll need much supplemental coverage.