In pursuit of quality: shifting the Program Manager Mindset

Hello all and a very happy New Year’s wish to you. It has been a LOOOOOOONG time, since I have written a post (as several folks have reminded me). While it remains my goal to post at least once a month, I know I won’t always achieve that goal. These past several months have been a particular drain on my time, probably my most valuable commodity at this time in my life. One of my New Year’s resolutions is to reprioritize how I spend my time. My goal is to focus more deeply on my personal retraining and on my family. I believe many of my readers are considering or following a similar path forward as am I. With luck, the time will avail itself, so I can publish what I’ve learned and share.

As part of that retraining, I’ve gone back to school. I am quite happy to report that I am now done with my first year of a MS in Analytics and thus far, I’m a straight A student. Honestly, though, it is a bit odd going back to school in your mid-40’s. I really could care less about the grades (it has been decades since someone has asked me my GPA). I am very passionate about learning the topic AND I have school age children who are watching my every move. I just simply can’t have teenagers calling me a hypocrite. The drive to learn the subject matter is really helpful to get the grades. I wish I had such a similar motivation the first time through college. J Thus far, I have been able to bring every class I have taken back into the applied context of my work and have done something valuable with it. I’ve got a ways to go, for sure, but the journey is been a lot of work and a lot of fun.

At work, I am helping to land something *very cool* for shifting the world into a more Data-Driven culture. I am on the Power BI (Business Intelligence) team in the SQL divisions. We are enhancing our SaaS offering and a preview for the public is available now with our new features. Check it out. The price tag for the public offering? Absolutely free.

I am quite proud to be a part of this effort. I hope you check it out and provide feedback.

News Update:

Since it has been a while since I last posted, I thought I’d cover a couple of quick topics:

  • Layoffs – Several of my brothers and sisters at Microsoft got laid off in the last few months. Most of the impacted folks that I knew have already landed new jobs and are reporting being even happier than they were. There are still a few people that I know are looking. If you are looking or seeking, please feel welcome to send me a note on LinkedIn and/or Twitter. I’ll see if I can leverage my network to help broker new relationships for people.
  • The AB Testing Podcast is back after a couple of months hiatus. (ok, this is old news if you are one of the “three”) You are welcome to check us out and send any question, comment, or feedback you’d like. Alan and I are both change agents of a sort and believers in collaboration and community for accelerating achieving goals. I, for one, absolutely adore the Mailbag segment. Any way we can improve the podcast, talk on something that might improve your life, or share your success story, send us a note. We’ll talk about it “on the air”.
  • Other agents of change: I’d like to call out others who are putting “pen to paper” in to help the community grow and converge.
    • Ben Bourland – Has started a blog recently and is journaling his experiences and insights on the shift from Test to Quality.
    • Steve Rowe – A long time blogger and QA manager in the Windows org. He is also trying to influence change towards data-driven.
    • Michael Hunter – Another long time test blogger and friend did a live presentation recently at SASQAG. The link to the video is here. His talk is on how he came to the realization that he had strengths that were valuable in many disciplines and that actually testing was not something he had every actually enjoyed. I share this in case his journey inspires others to let go of their fear that the discipline is changing.

More Changes

About a year ago, I wrote a post with the intent of helping Testers manage the change. A lot has changed in my company as well as others around us, and it is now fairly clear (to me, at least): Test as a dedicated team is a thing of the past. Testers have shifted into Data Analysts roles, development roles, infrastructure roles, or specialists in NFRs (non-functional requirements) such as End to End Integration and/or Performance. Very few “Testers” still exist. However, the transition is far from done. Many Testers have just been bolted onto Development teams under the guise of combined engineering, but the actions of those involved haven’t changed. In some teams (thankfully, including mine), Testers are gone and their skillset is slowly being incorporated into the team as a whole. However, it’s a slow difficult change for the prior development team to fill the void. A wise manager once told me, “Brent, be patient. It’s a marathon, not a sprint.” This change is far from over, but definitely moving in a more productive direction.

However, one role that persists that continues to show very little sign of change is PM. Program managers historically have owned the requirements for defining what we are bringing customers, then driving a schedule towards delivering it. Most PMs spend time speaking with customers and build a strong intuition on what customers want. They are generally very charismatic and likeable and work hard to try to make people happy. While I have never been in PM, back in the day, my father was Director of PM for multiple companies. We now have very interesting conversations about our differing experiences. For example, one commonality my father seemed to share with other PM’s I know was a drive to get into the executive role. I have had an opportunity on multiple occasions to work directly for the executive in my team. I think my dad was more than a little disappointed when I have told him “I have very little drive or desire to do that job.”

I was once told by a mentor that “Test doesn’t understand the customer” and that this was the main reason why PM advanced to executive more often than not. I think most PM’s I have spoken with in various degrees or another share this opinion. However, it is my belief that the next big culture shift that is about to come. Right now, PM, too, either won’t understand the customer and/or their ability to act on the appropriate window of opportunity will be too slow. I believe the primary root cause for this will be an over reliance on their intuition and a reluctance to test it out. PM’s are used to being able to rely on their soft skills and having a long time to react to the market. In comparison to techniques used by competition, the result is slow and often just plain wrong. Moving forward requires a mindshift in the program management organization.


I have mentioned that in today’s world: speed to market on value adds is paramount. Old school PM Techniques, unfortunately, are too slow and do not scale. Thus far, PM has remained relatively unscathed as part of these changes, but I firmly believe their judgment day is coming next. Their careers are at stake. Consider this: as data science takes further hold into organizations and teams light up the ability for individual engineers to make correct and actionable decisions on their own, the need for a team Customer Expert (ie. PM) becomes *dramatically* reduced. For those paying attention, one will notice that need for PM to take on the role of schedule meister has already been cut way back. Unlike Test, I believe PM will be still needed, but I do believe we are nearing a time where we will see their numbers greatly reduced. PM must learn to adapt and apply this new knowledge to not only survive, but thrive. IMHO, it will be those who learn to balance their intuition with the data towards actionable and valuable decision making that will differentiate themselves from the pack.

Better Together

I have had multiple conversations with the Data Science groups throughout the company and a new series of problems are becoming quite clear:

  1. No Customers – Complaints from the data guys that they are building assets that should be being used, but aren’t.
  2. DRIP (Data Rich, Insight Poor) – The items they are building that DO get used are, in essence, Scorecards and Dashboards filled with Vanity Metrics that aren’t shifting the needle towards anything the business values.

There’s an obvious symbiotic relationship between program management and data science. These folks need to be working together in concert like a well-oiled machine. I really don’t understand why they aren’t, but it’s clear from the number of people talking to me about it, this isn’t happening or not happening sufficiently enough. I think a big part of it though is PM doesn’t know how to take advantage of the data teams and the data teams don’t know how to express their value in a way that resonates with the PM in a non-threatening fashion. By that I mean a win-win scenario, I’ve spoken with several PM’s that very strongly believe this is all “data crap” and their intuition is all that is needed. True, Steve Jobs did it. I would argue that Mr. Jobs was special. He was one in a million whose intuition just happened to be right. The rest of us are wrong all of the time. The positive thing though: PM doesn’t have to do it alone (and, in my belief, they probably can’t). Your data science team is ready, willing, and eager to help.

Show Me the Evidence

I believe PM need to invest more heavily into understanding how to do Evidence-based decision making (aka Hypothesis Testing). A key principle for their lives going forward should be: No matter how right you think you are, due to uncertainty, there is *some* chance you are wrong. Therein lies the problem, there is always some risk associated with uncertainty (such as wasting time/resources on a problem customers don’t care about us solving). Please feel free to leverage your intuition; this new world is *NOT* about intuition versus facts, but rather intuition validated by facts. Both… Together… Intuition on its own can very quickly lead you in the wrong direction. Likewise, facts on their own can lead you to optimizing your current business, but will not help you to find the breakthrough game-changer with higher business potential.

Hypothesis testing in a nutshell

The goal of hypothesis testing is to be able to confidently select the best available next action to take.

NOTE: “Best” is relative. Good enough *IS* in fact Good enough. One common error is see PM’s making a lot lately is deferring a decision until they have 100% accurate and precise information. You simply do not need this. Leveraging heuristics and a solid understanding of how to use confidence intervals will take you far. For example, Douglas Hubbard’s Rule of Five tells us that there is a greater than 93% probability that the median of a population is between the smallest and largest values in a random sample of only five items. Do you *really* need to know the median? Or is knowing the range all you need?


  • Idea Phase
    • Write down your hypotheses individually
      • These are in a form of a statement (not a question)
      • These must also reflect a business KPI.
  • Knowledge Phase (knowledge is information in action. The ability to use information)
    • For each hypothesis, enumerate your possible actions
      • What actions will you take if the hypothesis was true?
      • What actions will you take if false?
  • Information Phase (information is data organized in a meaningful way)
    • Now you need to enumerate the questions needed in order to confidently select the appropriate action.
      • Battle confirmation bias – the human tendency to search for, interpret, and remember information in a way that confirms one’s preconceptions
  • Data Phase (data are raw facts that are meaningless by themselves)
    • The last phase is to simply enumerate the data points you need to answer your questions.
      • Most of the time you will be measuring customer behavior. Measure the behavior you want customers to take, instead of trying to measure everything.
      • Be wary of Hawthorne’s effect: People’s behavior will change in accordance to how they know they are being measured.
    • Get these instrumented or build a heuristic based on your existing instrumentation that can be used to answer your questions.

Hubbard provides 4 very useful measurement assumptions to consider when you are designing your data phase:

  1. Your problem is not as unique as you think
  2. you have more data than you think
  3. you need less data than you think
  4. an adequate amount of new data is more accessible than you think

Example: (oversimplified)

Hypothesis: Within my product, enabling Users to share with other users via Facebook will cause a notable increase in Acquisition and Engagement KPIs.

Possible Actions: (not complete)

  • True
    • No action
    • Optimize – make it really easy for users to share
    • Enhance – improve the sharing content to entice receivers
  • False
    • Abandon future feature development or cut
    • If acquisition improves, but not engagement, develop new hypothesis.
    • If engagement, but not acquisition, improves, develop new hypothesis.

Possible Questions: (not complete)

  • Which users shared?
  • Which users received invites?
  • Which receivers entered the service due to a sharing invite?
  • How does acquisition via sharing compare to other acquisition means?
  • How does sharer’s engagement level compare with those who don’t?
  • How does receiver’s engagement level compare with those who don’t?

Data Needed:

  • Users who share & receive
  • Receivers’ acquisition date & means
  • Acquisition rate correlated with receiving rate
  • Engagement rate correlated with sharing rate
  • Engagement rate correlated with receiving rate

If PMs and Data teams work together to craft their hypotheses (and experiments), more precise and accurate decisions will be made. Ideally, with the minimal amount of new instrumentation having to be added to the product. Be prepared to be wrong… A lot… But celebrate it. The faster you are wrong also turns out to be the faster you will *stop* being wrong by building fast, actionable knowledge towards business goals.

One final note:

There is one other phenomenon that seems to becoming common in the post-tester world that needs to be addressed.

PM, you need to STOP testing for your developers. Yes, yes, I know. “But Brent, I am now being held accountable for the customer satisfaction level for my feature and what I keep getting is bugs, bugs, and bugs.” Remember that this is a system problem. We’ve changed the software development system by removing testers from it, but by no means does this imply that this was done with the right tooling and training in place. Nor does it mean that you and your peers in PM have figured out how to truly determine the right number of features your dev team can produce in order to confidently get to done, done, done. There is much work to do here, for sure. You need to play your part.

How? Stop being the safety net. It is absolutely ok for you to be a part of the team and help everyone improve the quality of the product under development, but always consider whether or not your role in doing this is becoming more and more required. This might imply a dysfunction in the system and the team has learned that they can avoid doing testing themselves by shipping their crap to you. It’s super seductive for a dev. I, myself, have found my directs doing this, at least a couple of times in the past several months. It’s also seductive for you. Usually, when you find that critical issue, you will get praised for how well you really saved my bacon. Both sides get an endorphin rush.

Instead, I encourage you to strike a different balance between improving and shipping your features and playing your new role in this system. My recommendation is to take on the strict role of the PO for your team. A key responsibility of the Product Owner is own the Acceptance phase. Acceptance does not mean you find bugs. It means you own the decision as to whether or not the feature is ready to ship to your customer.

You can do this easily by doing 2 things:

  1. Making sure your team must doing an Acceptance Interview with you before they do final checkin into the shipping build.
    1. It is an interview only. DO NOT open the product. This is for the short term only and is required in order to set expectations: it is not your job to disprove/prove that they are done. It is theirs. It is your job to be satisfied that they have done so. Once your team has shown signs of using the new expected behaviors, feel free to change the interview process if it makes everyone’s life easier, but until then, resist the urge.
    2. Usually, you will have 2 lists for acceptance: 1) the requirements (acceptance criteria) defined when the item was still in the backlog, and 2) the non-functional requirements (perf, scale, localization, security, etc) that all features must be scrutinized against.
    3. For each requirement, simply ask the developer: “how did your prove readiness for this requirement?”. If their answer is vague, follow up with more precision questioning.
    4. Using your best judgment, if you don’t like what you are hearing, then Reject with your rationale.
  2. Being brave. For a short period of time, you will be causing disruption in the system. Your devs may have already gotten used to your new role. Change of this sort almost always causes anxiety. This anxiety may come with consequences depending on your team’s culture. In addition, your actions will likely result in several of your key stories/features not completing in this sprint. Remember your job is first and foremost to satisfy your customer. Your preference should be to do that alongside your team, but if your team wants something else, you may need to “go it alone” for a little while. Remember also everyone on your team are professionals, but they may have some bad behaviors due to existing muscle memory. Your job should be to help coach them to learn new ways.

As always, thank you for reading and I appreciate and welcome any feedback you’d like to offer.

In Pursuit of Quality: Shifting the Tester mindset

Last time, I wrote a book review on Lean Analytics. Towards the end of that post, I lamented that I see a lot of testers in my neck of the woods trying to map their old way of thinking into what’s coming next. Several folks (Individual contributors and managers of same) have come to me wondering why should test move into this world of “data crap” and why is how they have been previously operating so wrong now. It is my hope today to explain this out.

But first, before continuing, I’d like to try something new and offer you a poll to take.

Please consider the following:

So which did you pick? Over time, it will be interesting to me to track how people are viewing this simple comparison. I have been doing this example for almost a year now. When I first started it, about 1 in 2 testers polled would select the bug-free code. Whereas with testers I talk to lately, about 1 in 3 will select it. I definitely view this as a good sign and that folks are starting to reflect on these changes and adapting. My ideal world is that 1 year from now the ratio is closer to 1 in 10.

Why is this poll so hard for folks?

Primarily, it is due to our training. Test was the last line of defense – a safety net – needed to assure we didn’t do a recall when we released product out to manufacturing. When I first started in the software development world, 1.44 floppy disks were the prevailing way customers installed new software on to their system. Windows NT 3.1, as example, required 22 of them. It was horrible. Installation of a new machine would take the better part of the day, disks would be asked for out of order, and lastly, people would often get to the end of the install to discover that a setting they were asked for at the very beginning was wrong and that it was easier to just redo install than to hunt through the manual to figure out how to fix it after the install.

Customers who got their system up and running successfully and found a major bug afterwards would be quite sore with us. Thankfully, I have not heard of this is quite some time, but back then, Microsoft had the reputation of shipping quality in version 3.0. There was a strong and successful push within the company to get our testers trained with a singular mission: find the bugs before our customers do and push to get them fixed. I was proud to state back then that Microsoft was the best in the world at doing this.

The problem I am attempting to address is the perceived value loss in Test’s innate ability to prevent bugs from hitting the customer. A couple of months ago I presented to a group of testers and one of the questions asked “All of this reacting to customer stuff is great, but how can we prevent bugs in the first place?” Thankfully, someone else answered that question more helpfully as my initial response would’ve been “Stop trying to do so“.

The core of the issue, imo, is that we have continued to view our efforts as statically valuable. That our efforts to find bugs up front (assuring code correctness) will always be highly regarded. Unfortunately, we neglected to notice that the world was changing. That, in fact, it was more dynamic: Our need to get correctness right before shipping was actually tied to another variable: Our ability to react to bugs found by customers after shipping. The longer the time it takes us to react, the more we need to prevent correctness issues.

“Quality redefinition” – from correctness to customer value

A couple of years ago, I wrote a blog, Quality is a 4 letter word. Unfortunately, it seems that I wrote it well before it’s time. I have received feedback recently from folks stating that series of posts were quite helpful to them now. One such person had read it then and had a violent allergic reaction to the post:

“Brent, you can’t redefine quality”.

“I’m not!”, I replied, “We’ve *always* had it wrong! But up until now, it’s been okay. Now we need to journey in a different direction.”

While I now refer to the 4 pillars of Quality differently, their essence remains the same. I encourage you to read that post.

The wholeness of Quality should now be evaluated on 4 fronts:

  • Features that customers use to create value
  • The correctness of those features
  • The extent to which those features feel finished/polished
  • The context in which those features should be used for maximum value.

Certainly, correctness is an important aspect of quality, but usage is a significantly greater one. If you take anything away from today’s post, please take this:

Fixing correctness issues on a piece of code that no one is using is a waste of time & resources.

We need to change

In today’s world, with services lighting up left and right, we need to shift to a model that allows us to identify and improve Quality faster. This is a market differentiator.

It is my belief that in the short term, the best way to do this is to focus on the following strategy:

    • Pre-production
      • Train your testers to rewrite their automation such that Pass/Fail is not determined by the automation, but rather, leveraging the instrumentation and data exhaust outputted by the system. Automation becomes a user simulator, but testers grow muscle in using product logs to evaluate the truth. This set of measurements can be directly applied to production traffic when the code ships live.
      • Train your testers to be comfortable with tweaking and adding instrumentation to enable measurement of the above.
    • Next, move to Post-production
      • Leverage their Correctness skillset and their new measurement muscle to learn to understand the system behavior under actual usage load
      • This is an evaluation of QoS, Quality of Service. What you want Testers learning is what & why the system does what it does under prodtraffic.
      • You can start here in order to grow their muscle in statistical analysis.
    • Then, focus their attention on Customer Behavior
      • Teach them to look for patterns in the data that show:
        • Places in the code where customers are trying to achieve some goal but encountering pain (errors, crashes, etc) or friction (latency issues, convoluted paths to goal, etc). This is very easy to find generally.
        • Places in the code where customers are succeeding in achieving goal and are walking away delighted. These are patterns that create entertainment or freedom for the customer. Unlike the above, this is much harder to find, it will require hypothesis testing, flighting, and experimentation, but are significantly more valuable to the business at hand.
      • Being stronger in stats muscle will be key here. Since Quality is a subjective point of view, this will force Test away from a world of absolutes (pass/fail) and into one of probabilities (likelihood of adding value to customers vs. not). Definitely, it is wise to befriend your local Data Scientist and get them to share the magic. This will help you and your team to scale sustainably.
      • This is an evaluation of QoE, Quality of Experience. What you want Testers learning is what & why the Customers do what they do
    • You will then want to form up a dynamic set of metrics and KPI’s that capture the up-to-date learnings and help the organization quickly operationalize their goals of taking action towards adding customer value. This will generate Quality!

Lastly, while executing on these mindshifts, it will be paramount to remain balanced. The message of this blog is NOT that we should stop preventing bugs (despite my visceral response above). Bugs, in my world view, fall into 2 camps: Catastrophes and Other. In order to have Quality high, it is critical that we continue to work to prevent Catastrophe class bugs from hitting our customers. At the same time, we need to build infrastructure that will enable us to react very quickly.

I simply ask you to consider that:

    As the speed to which we can react to our customers INCREASES, the number of equivalence classes of bugs that fall into Catastrophe class DECREASES. Sacrificing speed to delivery, in the name of quality, makes delivering actual Quality so much harder. Usage, now, defines Quality better than correctness.

Ken Johnston: a friend, former manager, and co-author of the book “How we test at Microsoft” recently published a blog on something he calls “MVQ”.    Ken is still quite active on the Test Conference scene (one next month), but if you ever get the chance to ask, ask him “If he were to starting writing the Second Edition of his book, how much of the content would still be important?“. His response is quite interesting, but I’ll not steal that thunder here. J

Here’s a graphic from his post for your consideration. I think it presents a very nice balance:

Thank you for reading.

What would a Leader do?

Just over 2 years ago I wrote my most viewed blog post to date: The Tester’s Job. As 2013 comes to a close, there is much hullaballoo happening in the Microsoft Test community. After a series of non-trivial reorgs, it is clear Microsoft is making a huge step away from the traditional role for Testers and even the word Test is being stricken from the vernacular with our fearless leaders’ titles being replaced with a new one Director of Quality. Followers of this blog know this is change I felt was coming for a while now as well as aggressively supported.

Last May, my own team underwent this change with some great successes and some even greater learnings which I presented recently to a set of Test leaders. This presentation has become the closest thing to viral that I’ve ever experienced. Mostly, I believe, because folks are anxious about the change and are eager to use whatever data they can find to help predict their own future. The deck mentions some pretty significant changes needed to make the paradigm shift happen. The most controversial being a very large change in the Dev to Test ratio (a number that was historically used to determine the “right size” of test teams during reorgs). In my experience, some folks are more comfortable with innovating and being a starter. Whereas, other folks are superb at executing and getting work closed. Between the two, I have always been much more interested in being on the frontlines of change. Accordingly, I’ve never much been afraid of change, and view it as an opportunity to explore something new. I thoroughly enjoy seeing how these new learnings can be used to grow myself, my team, and whatever product I happen to be working on. And as a result, my love of the New and of Learning has helped to make me quite adaptable over the years. However, I understand that not everyone’s built this way and even those who are, the changes coming might be more than they can tolerate.

One colleague sent me the following in email after seeing my presentation:

Sobering. This is a lot of where we are headed in [my group], but without (so far) shifting any resources.  This may be really hard on test from the sounds of it.  Are there any suggestions on how to lessen the pain? Do we just rip the band aid and give people retraining?”

Since this is something that I am getting a lot of questions about lately, I felt it would be a good topic for a post. As I mentioned last time, Spider organizations don’t scale to the degree we will need them, so we need to build up Starfish muscle. While this does mean a move towards Headless organizations, it by no means describes one that is leaderless. In fact, leaders become critical. One of the challenges with this shift is that people are so accustomed to living in Spider organizations that they forget, nay, afraid to lead. This is the first change I think people need to do.

Here’s a very simple strategy I have found that helps me when times are troubling:

  • Ask and answer: What would a Leader do? – If there were an actual leader right here and now, what would they be doing? Why? What goal would they be trying to achieve? How would they go about it?
  • Be that leader – Why not you? Everyone has the ability to lead. It’s just easier not to. Choose to lead. Bravery is doing the right thing even though you are afraid.

So what would a leader do in these times? Here’s what I think:

  1. Keep their head on straight. There are 4 key things people need from their leaders. Trust, Hope, Compassion, and Stability. You cannot provide *any* of these if you join the panic. Imagine the state of the fireman who is going UP the stairs of a burning building.
  2. Manage the Change
    1. Explain what, why, and how the change is occurring. All three! Many times I see leaders leave one of them out. Folks need all three in order to triangulate on the correct direction.
    2. Explain the goal and new direction. Telling folks where to head is easier and more beneficial than telling them what to avoid. “We need to ship weekly” is better than “We need to stop shipping every 3 years” as examples.
    3. Enlist others in making the change happen. People are more likely to follow something that they contributed to creating. I’ve always been a fan of enlisting the team to come up with the logistics and dates and to place themselves into the positions that they are the most passionate about.
    4. Pull the trigger. Be the tipping point to get the momentum going.
  3. Train ThemselvesThis is probably the single most important item. You cannot help others if you have not helped yourself. You need to learn more about the new world you are heading towards. Dive in. Then and only then will you be in a position to guide others. Seek out internal experts. Change jobs. Go back to school. Head to conferences. I read recently that if you just read 1 hour a day in the field of your choosing, you would be an international expert on that topic in just 7 years. Those investments add up very quickly. Do not underestimate it. (I, myself, will be starting my Masters in Analytics on January 6th! (very excited about it))
  4. Train Others – You need to distribute what you have learned. A couple of things I have been doing lately:
    1. When someone asks me to talk to them on a topic, I assume they will not be the last. So organize it as a presentation.
    2. Record it, so it can be shared.
    3. Create a local community and ask each interested person if they’d like to join it. I can no longer find the reference, but something like 80% people will join if they are just asked. Try to drive participation on the community to make it self-sustaining.
  5. Get out of the way – Remain as the bottleneck to the change for as little time as possible. Someone may need to stay at the helm in order to make sure the momentum continues in the right direction, but once it is clear that it has, get out of the decision making process and let the team be empowered.

Rip the Band-Aid?

    To be honest, I am a proponent of Band-Aid ripping in these situations. People are afraid to make changes due to the unknown consequences. As Brad Pitt’s character asked in MoneyBall, “Which would you prefer, a clean shot in the head or 5 shots to the body and bleed to death?” The longer you wait, the harder the pain will be for those involved. But DO NOT rip the Band-Aid without a plan.

One last note: One very popular question people have been asking me lately is do I think that Test is dying? I believe Test (as we know it) is like a chicken with its head cut off. It’s dead, but the rest of the body is still flapping about and doesn’t quite know it yet. I have now been at the company for 20 years and, in that time, have seen a number of these big transitions occur in the Test discipline. I find it wise to remember: Each time these transitions occurred, a fairly large number of people were affected, but as a whole, we improved and became more valuable to the company. I think this time around will be no different. My view is that, given our innate ability to code and test coupled with our passionate pursuit of quality, our staff is well suited for being the engineers of the future and perhaps better than any of the other disciplines. However, whichever way the wind blows, it’s clear we will need to change. My New Year’s Resolution is to help anyone and everyone I can to help make this migration. After all, it’s what a leader would do.



It has almost been 4 months since I have last blogged. Several big things (big to me, at least) have occurred in that period. The least relevant to this post, but the hardest on my schedule, has been a house move. We are still living 33% of the time out of boxes, but things for my family are slowly getting back to normal at our new location. In addition, the old house has most of its laundry list completed and will appear on the market very soon. <Fingers crossed> It has definitely been a rough adventure. If I wait another 15 years to move again, that will be fine by me.

That isn’t the only change for me. Today’s post, as is often the case, the result of deep thinking of themes that are problematic to me. As example, I wrote my last post when I was thinking through what I wanted to work on next in my career and where I wanted to do it. As a result, I have made a non-trivial decision to move back into a test role (Gory details? check out my LinkedIn profile). Several really great things happened after my last post. I has offered several great Development Lead roles and even a Development Manager role. These all were very interesting and if I hadn’t already committed to the new team, I might have taken that DM job. I definitely still think of it often. Why? Even though I am back in Test now, I suspect the Test discipline is not my final destination.

I have gone back to test because I have some new personal and “organizational” goals to aggressively pursue and I believe the best place to do that and contribute to the delivery of ground-breaking, customer delighting products is in that role.

To explain further, I want to improve lives and solve problems. I want to contribute to building businesses that satisfy a customer need and excel at doing it. In particular, I want to advance the science of producing Quality in the context of that business. This is a strong passion of mine.

In my last year in Development, I had zero testers. My team had to own their testing themselves. As a result, I learned 2 very important things. 1) Far too much of our focus on quality is around code correctness and 2) Testers are *not* needed to deliver improvements there. In fact, Testers who grew up the way I did, might actually be slowing down the product and contributing to a much more expensive bottom line. I think Whittaker is right when he claims “All that testing is getting in the way of quality“.

My developers, without a test team, had to focus on testing things themselves. Since we were an Agile team, we shipped each ticket when it was completed. Every day we were releasing something and every day if someone on my team messed up, the whole team would swarm to fix or revert. We discovered over time, what tests helped, what hurt, what policies protected the customer, and what were just burdensome costs. Over time, we learned to rely *more* on design practices make changes quickly and effortlessly than on costly overtesting. As we focused on our ability to react *quickly* to failure, our ability to prevent failure grew and grew. Our designs became much more cohesive, much less coupled together. Easily detangled and simple to test.

With code correctness managed, this enabled us to turn to quality and customer satisfaction of our feature (link here for what I mean by Quality). This ended up being a much harder and interesting problem. The team I left behind is still aggressively working hard to nail this. They are an Agile team and able to self-optimize for what they learn on a weekly basis. They will get there. I have no doubt of it.

They are working towards having pinpoint accuracy around customer insight and quality for their customer base. They will achieve QWAN.

So why the new team? And why test?

The test org of my new team is going a different direction than most. They are aggressively shifting functional testing over to development and changing their focus to higher value output. They want to create a much, much stronger investment in infrastructure that reduces the friction to high value customer insight and work prioritization. They want to shift to a monthly release cycle and a service orientation.

I went there to help them execute the shift and bring about an age of Data Science and Agility in the organization.

Just found out on Friday that I am going to be the leader for one of the key initiatives needed to make this happen.

My boss told me this along with the phrase “be careful of what you ask for….”

This is going to be one of the biggest tasks I have ever tackled. I do not yet know everything that I and my team will need to do, but it’s going to be very exciting and the final goal very rewarding.

It’s YOUR career. Lead it!

Microsoft’s review model is an odd beast. Some things about it I adore, some things about it I abhor, and some things that just don’t make sense to me. Why *do* we go through our “midyear” discussion in March and then write up our Annual reviews in June (3 mos later), then get our reviews back 3 months later? On which review are we supposed to take credit for the awesome things we did from that June to September? Shouldn’t next year’s goals be written right after we submit our reviews (tweaked when we find out our reviews)?

This post is not about complaining about the review system. No review system is perfect and it turns out the Peter Principle (“people get promoted to the point of incompetency”) is true. It was proven in 2010 by the Nobel Peace prize winners of Management Science (here). Microsoft follows a general “Up or Out” strategy, which is considered a typical “defense” to Peter Principle happening in your org.

But wanna hear something really neat?

These same guys proved that promoting the most competent people from the lower levels up is not as successful of a strategy as alternating between the Best and Worst when promoting. Why? Because being good at working at one level, does not mean you have the skills needed to do what is required at the new higher level.  Some of the “incompetent” folks *actually* have the skills to outperform their peers at the higher levels. If you are a teacher and/or a strategist, you aren’t going to be valued as much as an individual, but if you can land a leadership role where you have to coach folks, you can nail it.

Let that soak in. Alternating between best and least competent people is a *more* successful strategy to improve the overall corporate efficiency.

That tweaks my noodle a bit, but I’ve seen situations like this. Years ago, I told an employee (who had zero passion for testing) that I would give him 30 days to find a new job in Microsoft or I would accept his resignation. I also told him if he applied for a test role that I would not give his new manager my recommendation, but a PM role, he would have it 100%! With that motivation in hand, he did land a PM job… The next several reviews he received were stellar.

He was now competent in the position he was in.

This time of year is typically where all employees are done hearing their reviews (I hope so!!!) and have written up their goals for next year. People should now be looking ahead figuring out how to maximize their career growth and build solid value into the products their team is producing. It is common for me to get my opinion asked for by employees or mentees about how to get a better review score or get promoted.

My answer is personalized for each individual, but in general, I let people know first and foremost, they need to own their own career. I can help them understand how the game is played, but ultimately, they are the ones who must do the work.

Usually, people who ask me this question is because they have one of the following facets of their career out of whack:

  • A product that drives their passion
  • A team that highly values the application of their strengths
  • A job that keeps them challenged


You need to find a product that will capture your heart. Sometimes mentees will come to me saying “I am thinking of joining that team, because I can learn [some important skill]”… Probing deeper, I will find that’s the only thing they value in the team. I usually respond by asking “Do you know what you will do next year once you have learned it and are bored and/or unhappy?” If they don’t, I recommend they look for a different team and work with them to try to find another means for them to acquire that skill. Passion is what keeps you engaged. If the product doesn’t do it for you, your value to the team will be constrained. Happy people are productive people.


I will write an article on this more deeply in the future, but I highly recommend some quality introspection around your strengths and how they are utilized in the organization. My favorite framework to use is StrengthFinders. Their tagline is “Do you have the opportunity to do what you do best every day?”.   Studies have shown (available off of the strengthfinders site) that a strategy around heavily self-investing in what you are good at naturally will yield a nearly 10x productivity rate. In this blog’s context though, your strengths catalog your *youness*… In a business context, this means the value you naturally bring to the workplace. It’s important to know this so you can help identify those roles in your workplace that will value what you bring,

My strengths (as example):

  1. Ideation
  2. Restorative
  3. Individualization
  4. Relator
  5. Analytical

At a rudimentary level, it shows that I have talents in Innovating, Team Building, and Problem Solving. I love being a manager on teams that are on the cutting edge and trying to find that breakthrough that will take them to the next level.

As a corollary, I do not enjoy, nor thrive, in Individual Contributor roles that are purely execution in nature. These are very important roles, no doubt, but not something I am best suited for.

Other frameworks exist to help you identify your strengths, if you decide that strengthfinders is a “cross between good advice and a horoscope”.  The point is to find and use what will help you thrive. Both you and the team you are on should value your strengths and invest in them. If you find yourself in a position of feeling like square peg being forced into a round hole on a daily basis, talk with your manager and find a new role. If none exists, find a new team.


In order to grow, you need to be challenged. The following diagram highlights Csikszentmihalyi’s Theory of Flow (not be confused with Kanban’s usage of the same term).

The chart above shows that the secret to happiness is to grow your skills and challenges at the same time. Growing your impact on your organization is quite similar to this process. Investing in your strengths is equivalent to investing in your skills (above). Being constantly in the flow will make you happier.

At Microsoft, there is a concept of a Business Justification for each role. What this means, in essence, is there is a business need for someone with a mastery of certain skills to be in the role. So what do you do when you have grown your strengths to the point that exceeds the business justification for your role? (or you have “automated” your role, so that it now has a lower level business justification?)

Really you have two choices:

  • Find new challenges to add to your role (and continue the flow)
  • Find a new role

Your manager should always be willing to partner with you on the first bullet above. If they are not, then you really need to find that new role elsewhere.

One last point here: If you are someone in upper levels (senior, principal, partner), then think about the business justification for your role in comparison to your peer group. If it implies a lower level justification by comparison (for example: a Senior can do your job and you are a Principal, and the rest of your peer group has Principal scoped jobs), then you *also* need to do one of the above choices if you want to have a chance to level the playing field for the review process. Now is the time to do this thinking, not 6 mos from now.

Alone in the Dark

By definition, you can’t know what you don’t know you don’t know (Armour’s 2nd order of ignorance), but you can help defeat it by constructing a process which will enable you to learn those things that matter. Yes, *you* need to be the one that leads your career, but you don’t have to do it alone, nor without a clue. Surround yourself with those things and people that will enable you to learn. No matter where you are, I view getting a mentor as crucial. I, myself, have several mentors and value their honest assessments highly. I am lucky in that I don’t get offended easily and surround myself with “say it like it is” folks.


If your manager is telling you “You need to grow your visibility”, then shame on them. Your manager is optimizing for a dysfunctional loophole in how the process works. Yes, this tactic will work. It can and will get you promoted, but it elevates Peter’s Principle. I believe Visibility is a 4 letter word… It’s offensive. Unfortunately, this advice encourages folks to do what it takes to be visible. By making visibility a goal, you are not learning the skills to maintain and sustain yourself at the new level.

You are helping to ensure the Peter Principle comes true for *you*.

There’s a reason why it usually takes 2-3 years to get promoted. You are mastering a new set of skills to help you scale to the higher positions and win there. Don’t make visibility the goal; instead, making adding solid business value and growing your impact to that business the priority… Nail this and visibility will come with it, but as a result of your success not as the goal. Plus you will be more prepared to handle your next promotion.

Choose a job you love, and you will never have to work a day in your life.

Need more to read?

Some other great blog posts on Microsoft Review and career advice from their authors:

Anita George:

How Important is the “How”?

If You Want It, Then You Aren’t Ready For It

Up, Up, and Away!

Eric Brechner:

Level Up

Permanently High Plateau

It’s not going to be okay

Alan Page:

Three Surprises

Dead Man’s Curve

Stack Overflow