“Test doesn’t understand the customer”

One thing I learned long ago is: if you want to make a breakthrough, you’ve got to try to look at the problem multiple ways.  Get different perspectives.   This is one reason why I am very fond of mentor relationships.   I currently have 4 mentors and about twice that number of mentees.   Maybe I’ll talk to their importance in some future blog, but needless to say, these folks are invaluable.    Especially those who did not grow their careers following the same path I did.

This is true of Mr. R.  He was an executive who had ascended via the Program Management path.  I had asked him to be my mentor because he had a shrewd eye for managing the business of software.  Something I was looking to understand more of and to master.  At the time, my title was Test Manager, which is essentially a middle manager role between the executives and frontline managers.  Mr. R’s role was to manage a Product Business.   This meant engineering, documentation, marketing, etc.

On the first day of our mentorship, I asked his opinion around something I had casually observed at Microsoft.   What I did not expect was how fundamentally his answer would shake my world.

Me:  “…In addition, given your experience, there’s another pattern I’ve observed that I’d love your opinion on.  I’ve noticed that around 15 out of 20 folks in your role got there via Program Management, about 4 via development, and around 1 out of 20 via test.”

Mr. R: “I don’t know if that’s totally accurate, but seems about right.”

Me:  “So why do you think test so rarely gets the role?”

Mr. R: “Oh, That’s easy!  It’s because Test doesn’t understand the customer.”

I remember the impact of those words.  Particularly, “That’s easy”.   Mr. R did not even pause to consider.  He knew that was the correct answer and he knew it confidently.

I have a reputation of not being afraid of any 1:1 debate and what this guy just told me would have been fighting words in just about any other situation.  Test is the damned Customer Defender, after all.   I flipped the Bozo bit.  I thought: “Well, I guess I’m going to have to find someone else to be my mentor for this.  This guy’s clueless.”  Thankfully, the problem solver side of me still didn’t have closure.   “Can’t have him telling everyone Test doesn’t understand the customer.  That’s not good for the Test discipline.  Let’s correct his thinking and go get lunch”, my thoughts continued.


I couldn’t.

Nothing was coming to mind.  What did test, as a whole, do to understand the customer?  I wasn’t suffering Thinker’s Block. I simply was unable to come up with any argument to disprove him.

I thanked Mr. R for his time and said that I needed to think some more on this.

And think I did…   For the next several weeks.

I examined the activities that Testers did.  Test cases, specs, automation, bugs, etc.   Failing to find anything really relevant, I told several peer Test Managers my story and asked them the same question.   It was like I had just told them there was no Santa Claus.  Some fought.  And hard, but eventually they all came to the same conclusion:  Test had wrapped itself up in the Emperor’s Clothes.

So the “Test Thinktank” worked the problem.  We discovered there were 3 places where Test had a touchpoint with the customer:

  1. Indirectly through PM’s feature requirements
  2. Vaguely through trolling newsgroups.  Though, this usually resulted in aggregation of complaints.  Ie. Understanding what it takes to stop irritating the customer.
  3. Actually interacting with the customer

Points #2 and #3, we noticed happened rarely.   Ironically, Testers were often too busy Testing.   Testers who were especially good at interacting with the customer were often considered misplaced and quickly moved to the PM organization.  Testers who weren’t good at it would be uninvited from future events.  This would be due to the natural tester mindset.  Testers, who were very good at finding things wrong, would find themselves sharing that info with the customers.   In general, it’s considered a Bad Business idea to send your Engineering team’s resident pessimists to the people who write the check.

Another thing has happened to me recently as well.  Those of you who find this blog via my LinkedIn profile are already aware that while my soul belongs to Test, my title no longer does.  I’m in a frontline Dev management role.

I feel like a sheep in wolves clothing.


I’m doing dev work, but I feel more like a SuperTester, than the Developer I expected to feel like.  I don’t know if it is because of all of my Test training or because I don’t have a Test team assigned to me as a partner, but I spend a *lot* of time thinking about quality.   A lot more than I did in test.  I also wonder what I would do differently if I had a test team as a partner.    I’m pretty sure I don’t want one right now.

There are a couple of reasons why:

  1. The Burden of Accountability & Responsibility is about 3 times stronger.   The mother of the baby feels a lot more responsible than the midwife to make sure the child becomes a responsible adult.   I felt really accountable in Test, but it is just stronger when you know that you and your team is writing the code that will affect customers.
  2. I feel closer to the customer than ever before.  Based on prior experience with other Dev managers, I don’t think Dev, in general, understands how good they got it.   They are directly providing quality to the customer.  I… Am… Actually… Delivering… Quality…   It’s fantastic.  In addition, I’ve really discovered that I don’t like my team writing code that the customer doesn’t use.   It’s a waste.   This forces me to understand my customer goals and intent, fail fast, and quickly iterate, so I can maximize the value my team is providing.  Non-theoretically.
  3. I do not want my team getting sloppy and relying on a crutch.  I think far too many Test/Dev pairings end up with Test cleaning up after Dev’s messes; not about effectively partnering to maximize customer value.  It’s an easy pattern to fall back to.   I want my team mastering the ability to produce excellent designs/code on their own first (producing fewer features, if necessary).

As I sit back now, it’s still hard for me to disprove Mr. R.  My own experience is that I actually understand what Quality means to my Customer better than ever before.  I understand their intent and desires better than ever before.  My hope for today’s post is that you simply consider for yourself if you actually understand your customer.

There’s a motto that I use frequently on my team: “We don’t know what we’re doing”.  I’ve communicated to my team that the motto’s intent is to remind each and every one of us that until we verified with the customer that we delivered something they wanted.  Our understanding of the customer is simply an assumption we should constantly try to clarify.


9 thoughts on ““Test doesn’t understand the customer”

  1. Somewhat controversial title, though after reading the post it seems appropriate to your context. I’m sure plenty of other testers can relate to it though, I’ve worked ( as a dev and a tester ) at places where there was zero customer contact and all work was done through specs.

    Do you think your post applies to places that are doing ATDD/BDD with close customer collaboration and demos to the customer every 2 weeks ?

    At the recent STC Test Bash there were 2 talks aimed at getting testers closer to the customers – Visualing Quality ( http://www.shino.de/2012/03/23/testbash-visualizing-quality-a-random-walk-of-ideas/ ) and Survival Of the Fit Tester ( http://www.shino.de/2012/03/23/testbash-survival-of-the-fit-tester/ ) – both of them with plenty of ways and ideas for a tester to understand what a customer wants

    “it is just stronger when you know that you and your team is writing the code that will affect customers” – how do you and the devs know this ? How do you get the devs to understand that what they are typing on their keyboard has an impact and how ?

    • I completely understand your feelings about the title. Likely, it’s exactly how I felt when Mr. R said those same words. It is my hope that the reader will have that same feeling invoked while reading.

      At present, my own team doesn’t do ATDD/BDD, but close customer collaboration and frequent demos/releases to the customer are both highly important to us. I think it would be very hard for test (or anyone) to maintain a stance of customer ignorance under such frequent customer scrutiny. My team has also gained a lot of value from using User Stories in the form of “As a [customer], I want [to do something] so that [goal will be achieved].”

      FYI, about BDD/ATDD, even though we are not yet doing either, the team is still doing Unit tests and asserts after the code is written. I’m training the team towards Agile and away from Waterfall and I’ve choosen to do this stepwise. Once they mastered a concept, teach them a new one. Each of my team is required to go to training on TDD, Design Patterns, and Refactoring by June of this year as well, so I’ll be aggressive on teaching them that very soon.

      I would add Big Data research to your list of links. Using telemetry to understand your customer’s behaviour and choices will also help.

      I know the accountability is stronger from personal experience. How does my team know it is writing the code that will affect customers? “How do you get the devs to understand that what they are typing on their keyboard has an impact and how ?”

      Now those are thought provoking questions. In my team’s case, we know it because we interact with the customer very frequently. They tell us when it has impacted them (positively or negatively). Vicarious Experiences are powerful motivators.

      I think the trick is to 1) make the customer non-theoretical. Demos will do this, but so will telemetry, so do forums, etc. and 2) frequently connect to the customer with the intent of understanding their current behavior on your product and their desired behavior.

      I am reminded of a tweet I read after I wrote my “JeDI” post. The person was commenting that it seemed to him that most were moving to Agile due to productivity gains, not customer connection.

      I remember thinking that capturing productivity gains from Agile without customer connection is impossible. It occurs to me now that might be the “secret sauce” to Agile. If you are not grounded in the customer, your team can very rapidly progress towards massive dysfunciton.

  2. Nice article, Brent!

    I don’t think it’s just test that fails to understand the customer – all disciplines suffer from it, even PM. During my career in a dev role, there have been countless times that we THOUGHT we were building the right thing for the customer, only to be stymied in a usability test where they couldn’t even get past the most basic concept. “How could they not get it?” was a frequent thought. We often fall into the trap thinking that the customer is us, when it often isn’t – even for developer tools or APIs. We are just too close to the work.

    This experience is what, in part, brought me to agile. Get something in front of them quickly. Get feedback before it is too late. Make refinements. Embrace change.

    We can THINK we know the customer, but we don’t know what we don’t know until we get working software in front of them and fully understand their goals and scenarios. It is not just test that is guilty of failing on this countless times, but all of us.

  3. Pingback: Quality is a 4 letter word « Testastic

  4. Pingback: Losing Our Way - Steve Rowe's Blog - Site Home - MSDN Blogs

  5. Pingback: In pursuit of quality: shifting the Program Manager Mindset | Testastic

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s