Making QWAN actionable for Testers


Shortly after writing my last post, QWANtifying Quality, I was having lunch with a longtime friend of mine who gave me some feedback. He loved the post, but he wanted something more to make it actionable. I received similar feedback from others… It was fairly esoteric. Today’s post will try to connect the dots. If you are reading this site for the first time, I recommend you go back and read the prior post before continuing. Though, in its conclusion, I share an opinion:

Delighting customers is good. Helping them live to their fullest potential is better.

For those who just want to know my answer to How? In a nutshell, it’s follow the Lean principles of “Fail Fast, Learn Fast, Recover Fast” and all that goes with it.

Otherwise, read on. I think Alexander’s knowledge on Quality sheds some great insight into how/why Agile actually works:

I asked my friend to explain to me his routine for watching TV. He likes to grab a huge pillow, a blanket, a bowl of snacks, and two drinks. When he enters into TV mode, he doesn’t want to get up for a while. He puts his drinks and bowl on a side table, covers himself up and then props his legs up on a coffee table. TV on. Laissez les bon temps rouler!

I asked him: how would it feel if you didn’t have that side table? He mentioned it would clearly be a loss. By holding his beverages and snacks in close proximity, the table is helping him to live. That table, the pillow, blanket… all of it… worked together to maximize his comfort level.

Using Alexander’s terms: my friend’s design of his living room has encouraged a Pattern of Events. Essentially, this is a verb or set of verbs that the design encourages over and over. In this case, my friend interacts with all of these objects in the same way most times he watches TV. It helps my friend live better. In addition, this pattern is deemed “alive”.

Why are Patterns of Events important?

Because they give you insight into the quality of your design. “Design” in this case can refer equally to a living room or a software package. I’ve said before that quality is the aggregated subjective POV of your customers. Today I’ll put it differently:

Quality is the aggregation of the Patterns of Events created by the design.

What this means:

  • Patterns are those interactions that repeat themselves over and over.
  • Interactions are more important than opinions. Actions do speak louder than the words of your users.
  • If humans don’t interact with the design, the quality is, at best, indeterminate and at worst (and most likely), low.
  • If humans do interact with the design, then the quality is the overall sum of Living patterns + Dead patterns. If the Deadening patterns are the stronger force, net quality will be low.

NOTE: A living pattern is one that either frees us or amuses us. Similarly, a dead pattern is one that traps us or adds difficulty to achieving our goals. Dead patterns often feel like “work”. One example of a dead pattern that immediately comes to mind is my Ping Pong table.

My youngest son and I love to play it together. Unfortunately, the table is currently folded up next to bicycles, baseball equipment, treadmill, etc. on one side of the garage. Getting to it is a challenge. My wife’s car occupies the other side of the garage. I installed a really nice light on her side of the garage, so we can play table tennis at night, but do we? No. There’s too much friction. I have to move out the car, untangle the table from the other stuff, move the table over, etc. So while the table has the potential of improving our lives, it doesn’t. There are too many dead patterns working against it.

Similarly, imagine what my friend’s living room would feel like to him if he moved his side table right next to the TV. Do you think he would still continue to use it? My buddy knows subconsciously the patterns for his living room. Until some new pattern comes into play, he will always design his living room in a similar way, even though he may not know why he did it.

Creators of software need to tap into the subconscious of its users to figure out the patterns. Note: Alexander mentions you cannot know QWAN while you are experiencing it. While that may not be entirely true, I think what he means is it is a lot easier to understand other people’s subconscious programming than your own. The second you tap into your own subconscious your ability to “see” what it’s doing dissipates. By definition, it’s operating under your conscious mind’s radar.

Ok, I get it. How do I use this?

I tell people the first thing they need to do is think of their software as a set of questions, not a set of answers. There are a lot of folks in Test who want to do everything possible to prevent harm to the customer and there is value in that. Quality means lighting up Living patterns. You simply cannot do that up front. Until you have it validated by your customer, it is just a guess.

Never forget that.

The questions you want to ask are going to be a variant of “Does this feel right to you?”, but they are likely going to be specific for your areas and feature. Another way to put it: The customer is already going to test your product. In addition, she/he is going to execute the Test cases that matters to them. Unfortunately, they will not tell you whether or not the test pass or failed. You will have to reverse engineer that from the telemetry. By knowing the questions up front, the design will be influenced to make this easier.

Next, add instrumentation and an instrumentation pipeline. Easy instrumentation method: Log the timestamp, object, and event for every interaction the user performs. Create a background process to send that info to a well-known site for aggregation. Can’t afford to receive and store all that info? Then, surgically, instrument the information that will help you to answer your questions.

Step 3: Get users using it. Slap a “Beta” label on it and get it out to your customers. Let them use it as *they* see fit.

Lastly, Analytics and lots of it. You need to look at heat maps and sequences of events. You need to look at failure/success rates, if present. You will have to make educated guesses, if not. I consider myself only a beginner at this Data Scientist stuff, but I can say it does get easier and easier over time to make an educated guess, ship something that tests that guess, then observe the behavior again. It’s hugely rewarding when you figure something out and the data validates it.

A note: If you have the ability to allow folks to customize their experience, I highly recommend you enable it. This isn’t just skinning. By observing how they arrange their experiences, they are giving you great insight into their behavior and Pattern of Events. This is no different than letting folks rearrange their own living room. Let them design how they like it. I love the Windows Phone live tile and pinning features as a great example. In addition, it’s never a bad idea to enable the like/dislike button on your design. If customers actually want to tell you that the test case they tried passed/failed, you should reduce that friction as much as possible.

Another note: Many folks challenge my optimize for reaction approach to quality. “Brent, does that mean you believe we shouldn’t test anything?” No, of course not. But I believe the proactive testing should be mostly focused on preventing those things that are most likely to cause an allergic reaction to users. I don’t think it takes much effort to strike the right balance. My suggestion: Get the right people in a room, spend 30 minutes, and come up with a 10 Minute Test Plan. By arbitrarily limiting time spent, the group will work together and focus on the stuff most important to prevent the customer from experiencing pain.

Making it real with an example

Here’s something I’ve observed with my children when they are on the XBOX. When they are on the XBOX dashboard, they almost never use the Voice interface, but they almost always use it when they are in a video application, such as Netflix.

After observing this behavior for many weeks, what I’ve noticed is:

  • The kids spend very little time on the dashboard. They use it to get to where they want to go. They always have a controller in hand when they are on the dashboard.
  • In Netflix, they almost always use the controller to navigate show selection
  • Once a show is selected, they become almost allergic to the controller.
    • Voice becomes the preferred mechanism to interact with XBOX.
    • Controller is avoided. Accidental button pushes causes the show to skip ahead, back, etc.

In order to improve quality, here’s some possible conclusions:

  • On the dashboard, the negative force appears to be voice misrecognitions.
    • Clearly, improving the engine should boost quality
    • But, optimizing for navigating with short easy to recognize statements could help too. Like “start Netflix”.
  • In Netflix, any option that would prevent accidental pushes would improve quality. It’s a pain to navigate back to where the show was before the accidental push.
    • Allow keys to be remapped
    • Allow option to turn off controller when show is playing
    • Require dbl-press of buttons… etc…
    • OR instead of preventing, make it easy to undo the mistake. Some ability to go back to where I was before that button press.

If I had access and the data existed, some questions I would ask of the telemetry before implementing any code change:

  • What are the common patterns for using voice on the dashboard? How are users being successful with voice on the dashboard? How are they failing?
    • Do my kid’s usage pattern show up?
    • What are the most recognized terms being used? How does word count affect success rate?
  • How often does the *same* Netflix show get restarted after a controller button push?

There are a lot more I could ask. What’s key is that you are trying to understand the customer’s behavior and if *they* viewed what they are doing as positive or negative.

The next thing to do is to use your knowledge and get a fix out there. Ask the same questions and see if the success rate improved. Do this frequently enough, you’ll find your product iterating towards solving customer needs and improving quality. One thing I have really come to value: By pushing to find Living patterns, this process really helps to filter out those things your customer doesn’t care about. I hate it when my team delivers something that the customer doesn’t use. By following this model, we spend the minimal amount of time on those things while still pushing the envelope and innovating.