MyTestCareer++;

It has almost been 4 months since I have last blogged. Several big things (big to me, at least) have occurred in that period. The least relevant to this post, but the hardest on my schedule, has been a house move. We are still living 33% of the time out of boxes, but things for my family are slowly getting back to normal at our new location. In addition, the old house has most of its laundry list completed and will appear on the market very soon. <Fingers crossed> It has definitely been a rough adventure. If I wait another 15 years to move again, that will be fine by me.

That isn’t the only change for me. Today’s post, as is often the case, the result of deep thinking of themes that are problematic to me. As example, I wrote my last post when I was thinking through what I wanted to work on next in my career and where I wanted to do it. As a result, I have made a non-trivial decision to move back into a test role (Gory details? check out my LinkedIn profile). Several really great things happened after my last post. I has offered several great Development Lead roles and even a Development Manager role. These all were very interesting and if I hadn’t already committed to the new team, I might have taken that DM job. I definitely still think of it often. Why? Even though I am back in Test now, I suspect the Test discipline is not my final destination.

I have gone back to test because I have some new personal and “organizational” goals to aggressively pursue and I believe the best place to do that and contribute to the delivery of ground-breaking, customer delighting products is in that role.

To explain further, I want to improve lives and solve problems. I want to contribute to building businesses that satisfy a customer need and excel at doing it. In particular, I want to advance the science of producing Quality in the context of that business. This is a strong passion of mine.

In my last year in Development, I had zero testers. My team had to own their testing themselves. As a result, I learned 2 very important things. 1) Far too much of our focus on quality is around code correctness and 2) Testers are *not* needed to deliver improvements there. In fact, Testers who grew up the way I did, might actually be slowing down the product and contributing to a much more expensive bottom line. I think Whittaker is right when he claims “All that testing is getting in the way of quality“.

My developers, without a test team, had to focus on testing things themselves. Since we were an Agile team, we shipped each ticket when it was completed. Every day we were releasing something and every day if someone on my team messed up, the whole team would swarm to fix or revert. We discovered over time, what tests helped, what hurt, what policies protected the customer, and what were just burdensome costs. Over time, we learned to rely *more* on design practices make changes quickly and effortlessly than on costly overtesting. As we focused on our ability to react *quickly* to failure, our ability to prevent failure grew and grew. Our designs became much more cohesive, much less coupled together. Easily detangled and simple to test.

With code correctness managed, this enabled us to turn to quality and customer satisfaction of our feature (link here for what I mean by Quality). This ended up being a much harder and interesting problem. The team I left behind is still aggressively working hard to nail this. They are an Agile team and able to self-optimize for what they learn on a weekly basis. They will get there. I have no doubt of it.

They are working towards having pinpoint accuracy around customer insight and quality for their customer base. They will achieve QWAN.

So why the new team? And why test?

The test org of my new team is going a different direction than most. They are aggressively shifting functional testing over to development and changing their focus to higher value output. They want to create a much, much stronger investment in infrastructure that reduces the friction to high value customer insight and work prioritization. They want to shift to a monthly release cycle and a service orientation.

I went there to help them execute the shift and bring about an age of Data Science and Agility in the organization.

Just found out on Friday that I am going to be the leader for one of the key initiatives needed to make this happen.

My boss told me this along with the phrase “be careful of what you ask for….”

This is going to be one of the biggest tasks I have ever tackled. I do not yet know everything that I and my team will need to do, but it’s going to be very exciting and the final goal very rewarding.

TEST is a four letter word

(note: this claim has not been tested in all global markets)

Last Friday, I attended a meeting with several technical Test leaders from around the company.    We were focused on Test PR.  Specifically, we wondered what the Test community could do to get college students really excited about the role or, alternatively, find those CompSci folks that would thrive more in the Test Discipline than in Dev or PM.

It was a productive meeting – the ideas generated will be great.  One thing, though, bugged me about meeting.   We spent a large amount of time trying to converge on a definition of Test that we felt was valuable for the exercise.

Earlier the same day, I met with a tester who currently works in Office Test.  This tester had not been out of school for long and was frustrated with their job.  They came to me to sort this out: was it their job, their manager, the team they were on, Microsoft?  

“I really love interacting with my Developer.   I’m really helping to make them productive and they appreciate me for it”. 

 This person was upset because they knew they were providing real value, but their manager was “old school” and focused on the tester’s low bug count.

On the web, there are examples as well.  Here are a couple of recent ones:

• Alan Page puzzles over the Test Job (and whether or not he’s doing it)
• Scott Barber tries to scope the Value Add of Test (and does so quite beautifully, imho)

This is a pattern that I’ve noticed a lot of lately (2 or 3 times a week).   Folks are trying to define the value/purpose/meaning of the Test position.  

And struggling.

 

It occurred to me that the problem might be Test itself.   Not the discipline, mind you, the word.

Ever since I learned Design Patterns, I have been fascinated by the idea that a single simple word can immediately invoke an entire concept.  If you were looking at a piece of my code and stated, “Brent, don’t you think this code would work better as a Singleton?”  I would know exactly what you meant. 

Definitions are key to understanding.  Singleton is nice.   It has multiple definitions, but in the CompSci context, only one.  You know it or you don’t.

Test doesn’t quite do the same.  If I say I programmed a solution, it means I wrote a bunch of code that I believe works to solve the problem I intended.    If I hand that code over to someone to Test, what are they going to do with/to it?  Is it clear?

Alan shares his perspective on the problem:  

“Roles that testers play on teams vary. They vary a lot. You can’t compare them. That’s ok, and (IMO) part of the growth of the role. I find it disturbing and detrimental when testers not only assume that their version of testing is “the way”, or that some roles (or people in those roles) do not qualify as testing.”

I tend to agree.   I do not consider myself part of the “Context-Driven” School of testing.   I think Testing, by its nature, is context-driven and therefore including that term is redundant.   (Though, it does occur to me that this viewpoint *might* actually make me a member of the school.    I’ve struggled to find the other Schools of Test to compare.  (Edited:  it’s here.))

I find myself puzzling:  How many other industries that use the word Test have this very same ‘Context-driven’ property? 

For example, if my doctor were to schedule a CA 125 test for me, would the lab technician recognize that I was a male and question the validity of the test?   If the results came back low (or high), would the lab technician offer me nutritional tips to improve my experience?     Would the technician defer my sample to someone else while they worked hard to improve the efficiency of the paperwork handoff between the doctor and the technician?

To all three I answer:  Maybe, but I doubt it.

IMHO, over the years, Test has gotten overloaded so much that it has lost much of its meaning.   I am reminded of another article.  In this one, Michael Bolton makes a distinction between Checking and Testing.  I love this.   While I don’t necessarily agree with what he finally landed on regarding the definition of Testing.  I can roll with it.  Michael is helping to disambiguate the terms.  I’ve personally found these terms helpful when communicating with others.

So how did we get here?  My hypothesis is we got here through business demands.

Scott says this:

“When it comes to software, on average, testing is on [the business’] radar, and testing is not providing the value they are looking for (or, at least is not being communicated in a way they understand).”

Many business leaders have differing expectations of what they want from test.  When Software testing began, it was essentially for scientific endeavors.    It should be of no surprise to find that it grew from debugging as a mechanism to “prove” correctness of a program (ie prove it works).  This then grew to finding bugs.   However, I believe as commercialized software began to dominate, making it “user friendly” became a differentiator for most businesses.  Suddenly “correctness” meant quality and we hired hoards of testers to pound and find bugs.  The cost of recurring Test passes forced us to learn to automate in order to move forward.   Since Testing has gone through such dramatic shifts and very quickly, it really shouldn’t be surprising that many folks still cling to older definitions and older value propositions

We are at another transitional stage.  Quality is now not necessarily about prevention as it once was, but about quick reaction.   We are going faster and faster, and storing more and more and more data.  I believe the Testers of the future are Data Scientists. 

One thing I am digging already is the cool new title.  I hope we don’t overload this one too.

There’s no Hope in Test

I bet that got your attention.

No,  I am not echoing James Whittaker’s and others’ thoughts decrying the role of the Test Generalist.  They may or may not be right.   I think it makes sense, and only time will tell for sure.

I did not say:

There’s no Hope FOR Test.

I said:

There’s no Hope IN Test.

While PM/Dev/Business Team may hope it works right, or ships on time, or sells enough, Hope is *not* a valid Testing Technique.  Hope is another way of saying “Assume”.

Neither has a place in Test.

Testing is a analytical role and requires strong critical thinking skills.  In a nut shell, in order to mitigate risk to the business, Testers need to be able to outthink their peers.  We need to find the right answers.  These require the right questions.

Which requires Thought!

Testers do not Hope.  They prove or disprove.

We are paid to find invalid assumptions, not make them ourselves.

—–

Originally, this was the intent of the speech.  Prove it works, don’t hope it does.  However, I realized there is another important meaning to consider.

I recently was in a meeting with Eric Brechner and Test Leaders from around Microsoft. We were chatting about his blog entry, “Test don’t get No Respect“.

He talks of a reality Testers can be in.   Misunderstood.  Paid less than their counterparts. Tasked to do brute force work.

I tire of the Test Zombies and the path of Intellectual Laziness they follow.    These are the folks who view Testing as both an Ends and a Means.  They automate because they automate.  They do what Dev/PM tells them, because that’s what they are used to.  They complain about Dev/PM doing this, because they are used to that as well.  They forgotten their role and value to the process.  If this is what we are purging from the software engineering process, so be it.  Good Riddance.

They aren’t thinking in a fundamentally thoughtful role.   This can even progress to Career-ending potential.   These folks HOPE that Test isn’t dead.  It would affect their livelihood, but are doing nothing about it.  ‘cept Hoping.

Folks, true testing isn’t dead.  As long as there is software, Testing will be needed.    But you need to make sure you are proving your value to the business.  New techniques will be invented.  New requirements will form.  Business plans will be disrupted.  These will require new approaches.    This may change Testing forevermore.

After a quick set of searches on Indeed.com, I gathered these stats:

software testing -   6,304 new jobs (edited from 111,047 (see comments))

java – 84,554 new jobs

C++ – 41,106 new jobs

c# – 40,837 new jobs

cobol – 3,031 new jobs

I look around nowadays and wonder if this what it was like when mainframe developers were told they were dying.   As you can see, COBOL is not dead, but it’s demand is woefully diminished.

But I wonder what happened to those folks who hoped their skill would be valued forever.

The Test Lead’s Job

Last time, I wrote about the Tester’s job, so when I got the following request from James Waletzky, a Development Lead colleague, friend and fellow blogger (though, it’s been nine months, James. We miss you.), I felt it was a perfect follow-up:

Hey guys,

What resources would you pass along to a new test lead outside of MS? Assume the guy is quite green and has focused on manual testing all along.

Some initial suggestions that perhaps you could add 2-3 of your most influential books, blogs, websites, etc to.
– http://angryweasel.com/blog/
– http://testastic.wordpress.com/
– Alan’s book – how we test software at Microsoft (http://www.hwtsam.com/)

Thanks,

James.

Resources and References

In my opinion, most of what a test lead does can be broken down into the categories below.  I have given some information that I find the most valuable today in the test lead role.

People

It’s important that you learn to understand the needs of your team, peers, customers, etc.

• Please understand me – Learning to understand why people think and act a certain way is key to working with them.
• Four Steps to the Epiphany – Gain skills in understanding *actually* what your customer wants.  Do not rely on your PM.  Do it yourself.
• HTWFAIP – Not just a catchy name.    This stuff really works.  And you don’t feel Machiavellian afterwards.

Product

You will still need to understand the product and deeply.  I like to read up on the competition as well as try to find universities that are researching similar technology (or pieces of the technology).   They often end up being great places to recruit, but almost always end up being a source of great whitepapers.

Project

You need to understand how your product engineering executes and/or how it should execute.

• The Lean Startup , Kanban, and Implementing Lean Software Development are my choices for Agile books.  They are fantastic!  
• I do not like the Agile Manifesto.   I have seen it be used wrong far too often to recommend it to the general populace.
• You need to understand the Waterfall method as well.   It does have its place in the world.    Accountability and predictability are what it offers.  If your team is trying to defend a position in the market or simply needs to execute, deploying agile is going to be expensive.  If you are trying to iterate realtime and get the right value to your customer every N weeks, Agile is a no-brainer.
• Anything by Alan Shalloway and his team is great.  I could have put this in product or people.   The folks at .NetObjectives are doing fantastic work and deserve your attention and money.
• Influencer - if you take the time to master this, you will be able to change darn near anything that needs to be changed.  Highly recommended!

 

Test Discipline

You are a test lead.  Not a PM lead.  Not a Dev Lead.  Not a release manager.   Make yourself a part of advancing the science within your own team.

• Start with my blogroll.  All of those authors are there for a reason.  Not all are test, but all are there to help you build quality.  Follow them on twitter.
• Poor Man’s Test Conferencing – I like to find curriculum for Test conferences and track them for latest advances.   Seeing what people want to teach is a great way to make sure you are staying on top of the discipline.

More (hopefully) helpful advice

Unlike the Tester’s Job, I don’t have a clever 7 word description to sum it all up…

Yet!

Until then, I would start off the discussion by congratulating them. Moving into a lead role is acknowledgement of solid skills and a lot of hard work. Something to be proud of, for sure. I would share that having been a manager of testers for more years than I care to remember, I can vouch that this is the single most important Rite of Passage for someone choosing to try out QA management. Good luck!

Three Most Important Things

These are the three most important things, imho, to master as a Test Lead.

Leader

You will need to manage work; you need to get things done.    You now need to focus on getting the best out of your team, not just out of yourself.   Managers focus on tasks, assigning people to them in order to benefit the business.  Leaders focus on people, assigning tasks to them to benefit the business and to benefit the person.  Be a leader.

It may be a subtle difference, but here’s a very important litmus test you can use to tell if you are being a leader or just a manager:

Leaders have followers.

I can’t believe how many people seem to forget this simple characteristic.  You can not be promoted to leadership.   People have to choose to follow you.

My advice:

•          Learn Active Listening Skills  – Understand your peoples’ concerns.
•          Get your people to like you. 
  •    Why?  Human psychology – It’s impossible to follow someone who you do not like.
  •    How?  The easiest way to get someone to like you is to like them first. 
  •    Dale Carnegie offers courses you can take to improve, but his signature book is still fantastic

 

Teacher

Moving to become a lead is hard.  Unless it happened due to the Peter Principle, you got to this position because you are awesome at something.   But now that you are here, you will need to get good at teaching your team to be awesome at that same thing. 

When I look back at my career, the people who were my best managers were the ones who were actively helping to guide my education in my job. 

My advice:

•    Constantly look for ways to help your team grow and encourage them to do so.
  •  Each of your employees should have a stronger resume once they complete being on your team.  Help them get there.
  •  Your team is stronger if you put each of them into roles that enable them to emphasize their strengths, but work on their weaknesses.
  •  Learn yourself.  Lead by example.   Share it with your team and ask for their advice on how to make it better.  Blog about it.
  •  The Socratic Method – Learn it, live it, love it.

 

Respect

As a Lead, you live and die in your career, by the success of your people, peers, and managers.  You are now a big part of the business.  If you cannot get the respect of your peers as example, you will not last long on that team in a leadership role.  Note: this may be okay.  Sometimes the culture of the team just simply may not be a fit for your leadership style.  If you realize you do not have the respect of your peers, you need to fix it ASAP.  If you can’t, you need to consider going to a culture that is more your style.

The only advice I will offer you

• You have to work this out yourself.  It is a Trial by Fire.   To earn respect, you have to do what you think is right.  You cannot follow the words of me, your boss, anyone or thing and expect people to respect you.  They will be respecting that person or thing.
• I used to share the following with people I was teaching to be a Lead:

“In the dictionary, there are 2 definitions of respect.  The first: awe and amazement.  The second: fear.  I do not care which definition you use.”

 I rarely share this with folks nowadays.   Depending on the circumstance, I have more tools in my toolbox and don’t need to resort to using the second definition.  BUT…  It DOES work…  Warning though:  if you have any sort of heart, it will make you feel dirty to keep doing it.  

Oh Yeah… One Last Thing

As a Test lead, you will need to be ready to answer the following question:

Why wasn’t this found before?  

My current favorite answer is Michael Bolton’s:   “Because it was buried so deep that not even the developer could find it.”

However, that answer will get you crucified in the middle of the likely heated triage discussion in which the question gets raised.

It’s best to be prepared, calm and thoughtful instead.  My 2^nd favorite answer (perhaps more productive):  “I agree we need to understand that.  After the current emergency is behind us, let’s sit down and find out.  We can fix it together.  I will schedule the meeting now.”

 

Good luck!  Hope this all helps.

 

The Tester’s Job

In my first blog, I talked about Test’s value to the business.   It’s insurance against risk.    For the longest time, my answer to *how* to do this was one of my most precious assests.  If, as a manager, I was faced with a difficult decision, I could use my definition of a Tester’s Job to guide me to the right place.  It enabled me to set a vision and a goal.  Hugely valuable!

Here’s what it was:

A tester’s job is to effectively identify, efficiently execute, and promptly report on the minimum number of test cases need to achieve the customer’s quality bar for the product.

To me, it was brilliant.  It wrapped up all the things we need to do to be successful.   Test Cases, Bug reporting, Automation, Customer Empathy, helping Dev get the results they needed in time, etc.   It’s all there.   Highly efficient.  There was one BIG problem with it.

IT SUCKS!

An employee of mine at the time, Isidro Hegouaburu, said it best (imagine this being said with a rich Argentinian accent):

“Brent,  your definition is really helpful and it tells me what to do, but it depresses me.  I don’t want to do that job.  It’s not sexy!”

Ouch!   That hurt!  What was worse was: he was right!   Customer Delight is an Emotion after all; Emotion is key to quality.   To define a tester job, a champion of quality, so stoically was to demean it.

Ugh.

So I threw it away and began a search to find a better one.   I talked with several leaders and got their opinion over the years.

• To find bugs
• To move Quality Upstream
• To inspect
• To manage and mitigate risk
• To make the product better
• To measure
• To make developers better
• To ask questions

I’m sure you have heard these yourself.    I find these mostly unsatisfying.  QA tries to do all of these at times.  These definitions attempt to take one task and make it *the* priority.   I wanted something goal-driven and encapsulated it all (much like my long and painful previous definition did).

Also, most of these came from people who’s experience was mostly Waterfall-based, I wanted my new definition to cover the entire gammut of project execution styles, especially Agile, since many are switching to that.

the Ah Ha Moment

I often refer to an NFL player metaphor when I consider test.    I think of Developers as players and Testers as the armor being worn.

I was admiring Frank Gore, in particular.   I was amazed at his ability to deflect oncoming traffic and to get to his goal.  Then it clicked for me:

A tester’s job is to accelerate the achievement of shippable quality

Since dev puts the quality into the product, our job was to speed them through that potential quagmire and help them get the code (ball) to the goal.  We will use a variety of tactics to get that done (including many of those mentioned in the defintions above)

One team meeting a couple of months ago, I tested this out.   Since what we had scheduled for that week had been taken care of already,  I asked each of my employees to tell me either the most important or the fastest thing each of them should do that week to accelerate the achievement of shippable quality.

Their responses were each fantastic, IMHO.  This definition encouraged them to think of quality and what it would take to ship and what was the current bottleneck for each of them.  My favorite answer that day was the Unit tests from dev were so poor that, if necessary, the QA team should fund writing them and making them easier for dev to write in the future.

I’d love to hear what you think and what you think this definition might be missing.

Merry Christmas!

Speed date your way to better quality

I often see that people get stuck in Intellectual Laziness Loops, especially in Test.    It can be a highly monotonous discipline.    Finding the same sets of bugs over and over, running the same set of tests over and over, fixing the same set of automation over and over, etc.   This is wearing on the psyche.   If your team is in constant firedrill mode, it’s just easier to be a Test Zombie.

During ship time, I spend a lot of time thinking about how to avoid this.

So when one of my employees came to me a month before our latest ship date and stated “I don’t think people are doing enough end to end scenario testing and I want to do something about it.”.   I was thrilled.

The Problem:

We only had a couple of days to organize  it.   We needed to find which people hadn’t been talking to each other but should have been.  We needed tests generated, bugs found…. The whole she-bang.  And fast, the product was about to lockdown.

Speed dating to the rescue:

We were talking about how do we match folks together quickly and efficiently.  We thought Speed Dating was a perfect paradigm and might be pique the team’s interest.

What we did:

• First, we found folks on each team and pre-socialized the idea.  We asked what they thought and would they attend.  Because we were inviting them to contribute to the idea, they were much more intested in attending.
• We arranged a meeting for the whole test team.  But told them it was entirely optional, but should be fun.
• On the meeting day, my employee gave everyone a sheet of paper for notetaking and stated the rules:
  • pair off with who you’d like, but with a preference for someone who you are unclear what they do
  • You have 15 minutes to: Share what each of you own in the product, find where your two areas intersect, and what tests you would need to do to cover the intersection.
  • After 15 minutes, you’ll find another partner.
• We did this four times only that day

The Result:

It was a huge hit.  Even after the scheduled meeting was over, the conversation continued on for hours.   The number one benefit was folks learned what each other was doing.   Several folks came up to me that day and stated “I did not know that person was working on [a tool]; I can’t wait to use it/collaborate with them/etc”.

It was fun to watch in action.   Unlike speed dating though, even the mismatched “couples” were successful.   One pair I observed worked really hard to find their intersection point.  “and then we make calls to the database layer.  Do you guys use that layer?  no?  hmm..   and them we call down to….”  And so on.   As a result. these two walked away with an understanding of each other’s architecture and problems.  Knowledge 2 – Ignorance 0.

At the end of the week, we walked away with several new bugs, several new test cases, better team relationships, and a bunch of people who were excited to do it the next time  (especially the folks who couldn’t attend…  “Man, I think I missed something cool”)

Are you moving quality upstream? Part 1

I hate the cliché “Move Quality Upstream”. I mean it. I detest it.

Those words have justified changes in staffing, automation strategy, process changes, milestone dates, etc. The end result: More of the things that changed, but the product still falls apart at the same places. Long stabilization milestones are still needed. All that work to move quality upstream and it gets flushed back down. Code velocity probably increased and the cost of doing business certainly increased, but did quality move even an iota forward?   The phrase tells us what to do, but not how to do it or how to know we acheived it.   It’s inactionable.

Football

The NFL has had similar problems. They have been improving helmet technology for years, but concussion rates have kept pace. One study I read recently  -  but not so recently that I can find it again… Grrr! – mentioned that the improvement in the helmet tech might just be the cause.

Basically, how it works:

•          NFL players are incented to hit hard ($$, glory, competitive nature, etc.)
•          The better helmets function, the more confidence the player has in safety
•          The more the player believes he’s safe, the riskier behavior he takes on
•          the riskier behavior he takes, the higher the chance of injury

I was watching the 49er’s pummel the Seahawks the other day (Go Niners!) and the referee called a penalty against one of the players. The tackler had initiated a helmet to helmet tackle and this is now a NO-NO in the NFL. The announcer mentioned “the referees were all asked to call the penalty even *if* they weren’t 100% sure”.

Will this fix the problem?  Maybe.   I think it could help.  This shows that the NFL is viewing these concussions as a behavior problem of the player and not a failure of the equipment.  The penalty is one of the worst in the game.  However, the powerful incentives still exist in the game and penalties happen all of the time.  How often will it be the cause of a loss?  Probably rare.

Ok, Brent.  Neat story, but…

“What does this have to do with Testing?” you ask.    As I mentioned last time, Testing minimizes risk “of injury”.  Far too often, we are the ever-improving Helmet and our development team plays the part of the NFL player.  The more we improve, the more dev trusts the safety net we’ve provided, the faster dev goes…

But not better.   Instead of moving quality upstream, we’ve made the stream faster.

I’ll go into what to do about it next time.

What is the ROI of Test?

For me, this is a momentous occasion. After being quoted a couple of times in other people’s blogs, I’m finally breaking out on my own. I’m very excited to see where this path will go.

I struggled a bit thinking through what should be the topic of my first post. I finally landed on a topic that got brought up over lunch with a friend of mine in the industry. It was about 4 or 5 years ago. At that time, my boss, who was not in test, was giving open positions more and more to the Development team. I felt that the test schedule, which was already at risk, was going to be impossible to achieve if at least some of those positions weren’t QA. I explained my plight and asked my companion “Given N testers on a team, what is the ROI for the N+1th tester?”   He responded with more than a little cynicism, “Brent, I don’t know. But tell your boss if he wants to see QA’s value, get rid of your testers”.  I was one with his frustration. Those of us in the business know our importance but it’s hard to quantify.

Delivery of quality is a subjective thing and how Testers should do it is just as subjective. It’s a topic for another day, but I’ve begun to collect definitions of a Tester’s Job and, not surprisingly, it’s already amazingly diverse. There are several ways to produce software and make it shippable.  Due to this, it seems a different perspective on the role of QA in the production of software is created for each.

Consider the following for your project:

1. Customer Resiliency:   How risk adverse is your customer? Will customers be ok with bugs?
2. Business Resiliency:   How risk adverse is your business team? Is there a window of opportunity  that must be perfect? Do we have to get it right the first time?
3. Speed: How fast does your team release?  Is it twice a  month or thrice a decade?
4. Complexity: This is a  measure of the number of components that must interact to make a solid system.
5. Complication: This is a measure of the design and implementation of the code used to make up a component
6. Test Confidence: This is a measure of your test collateral and its ability to cover the product.
7. Test Speed: How long does it take us to use the test collateral to determine state of the product?
8. Development Skill: What is level of mastery of the craft that exists in your team as a whole?
9. Testing Skill: What is level of mastery of the craft that exists in your team as a whole?
10. Ignorance (props to Armour): Are you measuring the above? Do you know what you need to do? Do you know which one is the bottleneck in your team?
11. Irrational Optimism: How hopeful is the team that the plan is correct and/or on track?

It’s very similar to taking a car trip.   We need to know where we are, where we are going, how we are going to get there, what to pack, etc.  If we make the wrong choices, like travel someplace we haven’t been without a map, or packing only winter clothes for a trip to someplace sunny, or taking the wrong kind of car for the terrain, we could end up paying a lot more than expected to get to our destination.

Your team has all of these chararcteristics in varying degrees.  Depending on where your team stands on each of them has a potential of fundamentally changing the team’s approach to software development.   An obvious example:  if you ship every three years,
you are likely to take a lot more of a preventative approach to bugs and “find ‘em before the customer does”.    However, if you ship every 3 weeks, you will likely focus on taking smart risks with your testing and get better at reducing the time it takes to react to a failure.

When I think of Test, I am not referring to Testers or a Test team per se.   I am referring to what portion of the payroll budget is applied to Testing activities.  While it is easy to chat about things like Dev to Test ratios, figuring out how much of that money should be spent testing vs. developing depends entirely on the culture of your team.  If your team registers high in all of the negative things, you may need to spend *a lot* in order to mitigate the risk and save your business.

Accordingly, Testing is a form of Software Business Insurance.   The more coverage you have the less at-risk you are.  So there’s really only one answer to my question.   It Depends!   Coverage mitigates risk.   If you don’t feel at-risk, then, until proven otherwise, coverage will not be valuable to you.

One day, perhaps, someone from the Actuarial world will join us in Testing and show us how to determine this more precisely, or maybe outsourced Testing companies will model their fee structure after the insurance industry.   Until then, in order to hire the next tester, you should be prepared to articulate the risk you see in your project and the value your business will gain in comparison to the options.  Back to the car trip metaphor, if you know the person driving the car isn’t a great driver, you could mitigate the risk and buy supplemental insurance or you could send the person to Better Driving School.

I recently met up with my friend. His group has recently undergone a major reorg and QA got trimmed back in a big way. It seems the group’s Head looked at the cost of the QA team and asked “What do these guys do?”  Worry not for my friend, though.   He’s taken his entire team and is going to Development.   His team has strong testing and development skills and many of the other characteristics above are positives for him.  I doubt he’ll need much supplemental coverage.