Are you moving quality upstream? Part 2

Row, row, row your boat,

Gently down up the stream.

Merrily, merrily, merrily, merrily,

Life is but a dream.

Last week, I wrote about a phenomenon that I’ve noticed in the QA industry regarding our ability to move quality upstream or more precisely, our inability.  Great thanks to fellow blogger, Phil Kirkham, whose comment led me to the term that describes it all: “Risk Compensation”.

We’ve all seen charts like this:


We all know that the cost of fixing a bug goes up huge as it travels down a stream.   This is usually the chart we show younger testers to inspire them to “move quality upstream”.      Note, though, that quality is *not* mentioned in the chart.   This is because it’s assumed true that we know exactly what level of quality is needed throughout the entire software development cycle.

Really all this chart says is “if there is an important invalid assumption in your project code, it is better to find that sooner than later.”


But do we do that?

I wish I had access to a study that showed this same chart over time on the same team.   (Seriously, if you have something like this, please send it my way.   Even better would be something measuring unit of production over cost of production over time)

I suspect that all we have done is increased the cost of a project for the entire period.  (Imagine the same chart shifted up one or two lines)   We do more and more testing up front, but still do more and more testing in the middle and the end.    I’ve seen, on some teams, the cost to actually go up much, much more on the back end of a project.  They interpret “moving quality upstream” to mean we need more automation earlier.   Sadly, they end up spending their time and money trying to maintain an increasingly burdensome automation suite meant to validate a static set of assumptions that are increasingly proving to be non-static.

In teams like this, your testers become full time automation maintainers and in the worst of cases, the actual testing of coverage gaps becomes a task for the customer (which is a sign, by some, that quality slipped downstream again).

So what is Quality?

So if we are going to move it, we need to know what it is.   My view: the quality of your product is the aggregated subjective opinion your customers have of your product (more details will be the subject of a future post).   If your product works, works the way your customers expect, and makes them happy, that’s higher quality than the reverse.   The point:   the quality of your product is determined by the customers of your product.

Where are we now?

In order to move something forward, we need to get a baseline of where we are at now.   Since quality is determined by our customers, this means we need to get it in their hands.  I find the modern techniques of constructing a Minimum Viable Product combined with plenty of instrumentation intriguing.   The trick is to Fail Fast whenever the customer is behaving differently from our expectations.   Then take the data that we have and Learn Fast to discover what it is we need to change.

Take a step forward

Winding this back to Risk Compensation, moving forward is often a sociological challenge, not a technical one.

What we want to do is Recover Fast and get a new version of the product in the customer’s hand in order to start the loop over.  But to do this we need to change the minds of people:

1)      Test to choose to focus more on understanding the depth and breadth of your customer’s concerns

2)      Dev to choose to keep quality high WITHOUT relying on the safety net

3)      Customer to choose to use your product more and more and pay more and more for it.

If you haven’t figured it out yet, the idea of micro-releases really appeals to me.    Faster, smaller release cycles make it harder to lower the quality through bug churn.  More importantly, we get more rapid feedback from the actual measurer of quality in the production line, The Customer.

IMHO, no one can sustainably ship garbage to their customer.  They will either change their mind or go out of business.

A hodge podge of other ideas came to mind through the process of writing this post.   I will share those next time.

Are you moving quality upstream? Part 1

I hate the cliché “Move Quality Upstream”. I mean it. I detest it.

Those words have justified changes in staffing, automation strategy, process changes, milestone dates, etc. The end result: More of the things that changed, but the product still falls apart at the same places. Long stabilization milestones are still needed. All that work to move quality upstream and it gets flushed back down. Code velocity probably increased and the cost of doing business certainly increased, but did quality move even an iota forward?   The phrase tells us what to do, but not how to do it or how to know we acheived it.   It’s inactionable.


The NFL has had similar problems. They have been improving helmet technology for years, but concussion rates have kept pace. One study I read recently  –  but not so recently that I can find it again… Grrr! – mentioned that the improvement in the helmet tech might just be the cause.

Basically, how it works:

  •          NFL players are incented to hit hard ($$, glory, competitive nature, etc.)
  •          The better helmets function, the more confidence the player has in safety
  •          The more the player believes he’s safe, the riskier behavior he takes on
  •          the riskier behavior he takes, the higher the chance of injury

I was watching the 49er’s pummel the Seahawks the other day (Go Niners!) and the referee called a penalty against one of the players. The tackler had initiated a helmet to helmet tackle and this is now a NO-NO in the NFL. The announcer mentioned “the referees were all asked to call the penalty even *if* they weren’t 100% sure”.

Will this fix the problem?  Maybe.   I think it could help.  This shows that the NFL is viewing these concussions as a behavior problem of the player and not a failure of the equipment.  The penalty is one of the worst in the game.  However, the powerful incentives still exist in the game and penalties happen all of the time.  How often will it be the cause of a loss?  Probably rare.

Ok, Brent.  Neat story, but…

“What does this have to do with Testing?” you ask.    As I mentioned last time, Testing minimizes risk “of injury”.  Far too often, we are the ever-improving Helmet and our development team plays the part of the NFL player.  The more we improve, the more dev trusts the safety net we’ve provided, the faster dev goes…

But not better.   Instead of moving quality upstream, we’ve made the stream faster.

I’ll go into what to do about it next time.

What is the ROI of Test?

For me, this is a momentous occasion. After being quoted a couple of times in other people’s blogs, I’m finally breaking out on my own. I’m very excited to see where this path will go.

I struggled a bit thinking through what should be the topic of my first post. I finally landed on a topic that got brought up over lunch with a friend of mine in the industry. It was about 4 or 5 years ago. At that time, my boss, who was not in test, was giving open positions more and more to the Development team. I felt that the test schedule, which was already at risk, was going to be impossible to achieve if at least some of those positions weren’t QA. I explained my plight and asked my companion “Given N testers on a team, what is the ROI for the N+1th tester?”   He responded with more than a little cynicism, “Brent, I don’t know. But tell your boss if he wants to see QA’s value, get rid of your testers”.  I was one with his frustration. Those of us in the business know our importance but it’s hard to quantify.

Delivery of quality is a subjective thing and how Testers should do it is just as subjective. It’s a topic for another day, but I’ve begun to collect definitions of a Tester’s Job and, not surprisingly, it’s already amazingly diverse. There are several ways to produce software and make it shippable.  Due to this, it seems a different perspective on the role of QA in the production of software is created for each.

Consider the following for your project:

  1. Customer Resiliency:   How risk adverse is your customer? Will customers be ok with bugs?
  2. Business Resiliency:   How risk adverse is your business team? Is there a window of opportunity  that must be perfect? Do we have to get it right the first time?
  3. Speed: How fast does your team release?  Is it twice a  month or thrice a decade?
  4. Complexity: This is a  measure of the number of components that must interact to make a solid system.
  5. Complication: This is a measure of the design and implementation of the code used to make up a component
  6. Test Confidence: This is a measure of your test collateral and its ability to cover the product.
  7. Test Speed: How long does it take us to use the test collateral to determine state of the product?
  8. Development Skill: What is level of mastery of the craft that exists in your team as a whole?
  9. Testing Skill: What is level of mastery of the craft that exists in your team as a whole?
  10. Ignorance (props to Armour): Are you measuring the above? Do you know what you need to do? Do you know which one is the bottleneck in your team?
  11. Irrational Optimism: How hopeful is the team that the plan is correct and/or on track?

It’s very similar to taking a car trip.   We need to know where we are, where we are going, how we are going to get there, what to pack, etc.  If we make the wrong choices, like travel someplace we haven’t been without a map, or packing only winter clothes for a trip to someplace sunny, or taking the wrong kind of car for the terrain, we could end up paying a lot more than expected to get to our destination.

Your team has all of these chararcteristics in varying degrees.  Depending on where your team stands on each of them has a potential of fundamentally changing the team’s approach to software development.   An obvious example:  if you ship every three years,
you are likely to take a lot more of a preventative approach to bugs and “find ‘em before the customer does”.    However, if you ship every 3 weeks, you will likely focus on taking smart risks with your testing and get better at reducing the time it takes to react to a failure.

When I think of Test, I am not referring to Testers or a Test team per se.   I am referring to what portion of the payroll budget is applied to Testing activities.  While it is easy to chat about things like Dev to Test ratios, figuring out how much of that money should be spent testing vs. developing depends entirely on the culture of your team.  If your team registers high in all of the negative things, you may need to spend *a lot* in order to mitigate the risk and save your business.

Accordingly, Testing is a form of Software Business Insurance.   The more coverage you have the less at-risk you are.  So there’s really only one answer to my question.   It Depends!   Coverage mitigates risk.   If you don’t feel at-risk, then, until proven otherwise, coverage will not be valuable to you.

One day, perhaps, someone from the Actuarial world will join us in Testing and show us how to determine this more precisely, or maybe outsourced Testing companies will model their fee structure after the insurance industry.   Until then, in order to hire the next tester, you should be prepared to articulate the risk you see in your project and the value your business will gain in comparison to the options.  Back to the car trip metaphor, if you know the person driving the car isn’t a great driver, you could mitigate the risk and buy supplemental insurance or you could send the person to Better Driving School.

I recently met up with my friend. His group has recently undergone a major reorg and QA got trimmed back in a big way. It seems the group’s Head looked at the cost of the QA team and asked “What do these guys do?”  Worry not for my friend, though.   He’s taken his entire team and is going to Development.   His team has strong testing and development skills and many of the other characteristics above are positives for him.  I doubt he’ll need much supplemental coverage.