Modern Testing Principles Explained

Nearly 8 years ago, I wrote my very first blog with a great deal of nervousness.   “Why does the world need yet another person espousing the value of test to software teams?”, I thought.   I am glad I did though. It greatly helped my communication skills and were instrumental in focusing my thoughts.    Looking back you can almost follow my own transformation into Modern approaches to Quality. Hell, just the first 5 posts contain many of the thoughts that are now expressed in the Modern Testing Principles (co-authored with Alan Page), including its mission statement of #ataosq.

Today officially marks another “milestone” of sorts for me in my shift away from Traditional Testing. This will be my last post on Testastic.    Why?  In essence, I no longer identify with the Testing brand.   I am a data scientist now;  Customer concerns and the delivery of actual quality is still (and will be forever) an important part of my job and my mission.

BUT, Test is now firmly entrenched in my mind as something I do and not something I am. 

Am I done blogging?   Who knows?!?  A wise friend once told me “I blog because I have something to say and when I don’t, I don’t”.  When/if I do come back to blogging, it will be under some new name that I identify with.  I am fond of the idea that I could further help Quality focused people enter into the data arena specifically. Maybe it will be oriented in that direction.

Before you worry (or rejoice), no, I am not leaving the podcast.  It has always been about Software Quality (not specifically Testing) and will remain so for the foreseeable future.   We named it ‘AB Testing’ as it enables us to talk on a broad array of topics.

The rest of this post is a follow up on Modern Testing Principles that I have been meaning to write for months.   It is my hope that it adds some form of clarity to the community at large.  

What are the Modern Testing Principles?

They are a path forward.  We authored them because we noted that a good deal of Test was struggling.  Struggling to make sense of several changes occurring in how software was being produced and their role in it.  Trying to make sense of memes such as “Test is Dead” or practices like “Testing in Production”.  In the traditional testing culture, these are nonsensical concepts. Several testers were (and are) left still believing that “what got us here, can still get us there”.

The Modern Testing Principles were developed to help Testers leverage their strengths into new positive directions that align better with producing software in the Age of the Customer. They are the antidote to Traditional testing.

Modern Testing is specifically named to reflect a need to leverage the benefits of contemporary practices and learnings.   The name “Modern Testing” is intended to distinguish present-day techniques, activities, and processes from their traditional “Test Last” counterparts.  It is absolutely expected that new learnings and techniques will continue to be discovered and leveraged in the pursuit of improving business’ ability to realize customer value.   Adapting and incorporating these learnings to towards “Accelerating the achievement of shippable quality” is thus a key tenet to Modern Testing.

Why are these needed?  

An astute observer of contemporary software development practices will notice that the principles, in fact, do not reflect anything truly Modern, nor anything about Testing.   Many companies have realized that techniques invented in the waterfall era of software development, no longer scale to current business demands.  As businesses adjust to the new landscape, many unprepared Testers are being left out in the cold.   Several studies have been coming out over the last few years which are giving credibility to a growing belief that while Test is not dead yet, it is dying.  Test is shifting from being a role to being an activity. As these efforts progress, Test, who has always been viewed as a cost to a business, is also being viewed as a bottleneck and irrelevant. BUT, not all software development business have adapted yet, so there is time for many.  Enough time for those, who care to do so, to learn new ways to apply their valuable skills into new positive directions. The Modern Testing Principles are for these Testers. They are guiding light for those looking for a way to create positive change in their lives in the face of a changing Test landscape.  

Markets have always been won by companies being able to adapt quickly to customer needs faster than their competition.   However, the software landscape has changed from the 1980’s waterfall process on several key aspects. Many companies today have learned and reaped the rewards of adaptable software development practices with low cycle times.  As such, it is easier than ever to compete; small teams can build great solutions at a fraction of the time and effort than ever in history. Lower switching costs give customers more freedom to move to competition in friction-free manner.  The online world grants more visibility to customer behavior telemetry than ever before seen. Additionally, software engineering techniques and processes have changed to allow continuous updates to software which makes risks smaller than ever to deliver solutions to problems that customers have today.

Important Note: one day a series of discoveries will occur that will remove the need for human involvement in quality evaluation and adaptation. As a data scientist, I suspect it will surprise no one that I believe Machine Learning will play a huge role in that. When those discoveries are found and implemented, Modern Testing approaches, just like Traditional Testing approaches before it, should immediately be abandoned.

Who is a Modern Tester?

Short answer: No one.  It is not a title. There will certainly never be a certification for it.  Modern Testing is a direction and a pursuit with a specific focus: Accelerating the achievement of shippable quality. A Modern Tester is a systems thinker who has experience in Traditional methods and continuously is learning to find more effective ways to achieve quality.  Some specific notions that a Modern Tester might reflect upon:

  1. The world has changed

    • Technologies like AI, Open Source, Big Data, and the Cloud have made it easier for competitors to win
    • Customers are empowered and likely indifferent.   They can switch to your competitor in seconds.
    • Customers care may only about their problems being solved.
    • They need those solutions today.   Not in 6 mos. Today.
    • Traditional Test approaches (such as scheduled test passes and separate Dev/Test disciplines) were required when we needed to ship Manufacturing and it was expensive to re-ship software.  They are may only be optional now.
    • Businesses need to care about Code Correctness and Craftsmanship in order to sustain speed of delivery.  Their Customers, though, don’t.
    • Several very successful companies have shipped bugs (and continue to do so) and don’t have Testers.
  2. Several Traditional Testing methods might no longer be needed or are unable to scale to current demands; keeping them alive may harm business goals by:
    • Creating unnecessary delays
    • Creating unnecessary costs
    • Being focused on code and spec correctness instead of quality
    • Enabling a detrimental ‘safety net’ culture and failing to deliver on the promise of moving quality upstream
    • Treating Test activities as an innate specialization that cannot be taught
    • Favoring dogmatic isolationism (“specialization”) over community and Whole team approaches
    • Being a passive contributor to decision making
    • Favoring Intuition and theory over work that measurably moves key positive business KPIs, such Customer Satisfaction or Cycle Time.
    • Failing to ACTUALLY measure and then, reduce risk

Explaining the Principles

In this section, I will explain the rationale around each of the principles.  More detail can be found in a variety of places. Visiting is an obvious first start, but I would also include with Episode 67 of the podcast and continuing through to 93.   The Ministry of Testing also has a fantastic synthesis here (and comes with a totally awesome printable graphic).   Lastly, many of the #Three (the term of endearment for our regular listeners) have created their own presentations or references.  My current favorite of which is a brilliant mind map of Modern Testing by Maciej Wyrodek.   Several of the #Three are active participants in MT community discussions on our Slack channel.   Click here to join.  (Don’t forget to go to the #iwantasticker channel.  The stickers are cool). I will also add additional book references for those who wish to understand the principles deeper.

My Nicknames

For each of the principles, I have “short titles” for them that simplify their intent. These are:

  1. Business First
  2. Bottlenecks out
  3. Learn
  4. Leadership
  5. Real Quality
  6. Product Hypotheses
  7. Embrace the Fear

#1 – Our priority is improving the business.

Book Reference: The Lean Startup by Eric Ries

This has turned out to be one of the least controversial principles in the list. Honestly, I fear this is due to a misunderstanding of the principle. In business leadership circles, Test is most often seen as a cost of doing business. Almost an insurance policy. But is it? When Test is questioned on the return, the answers are generally vague or subjective. The Test Community at Large does not have a good answer for this. The most common answer I’ve seen is “reducing cost”. However, digging in deeper often reveals that this is false. Its theoretical.

Test is generally very strong in Customer Empathy. The Modern Tester adds to this a non-theoretical look at what the goals the business is trying to achieve, looks deeply at how their work contributes to it, objectively measures it and becomes part of the solution moving that KPI. They remove the theory and execute in terms of ROI towards business goals. Generally, businesses are trying to do one of 2 things: Grow or Reap tangible rewards. Find out from your business leaders what your company is trying to do, connect your work to it, measure it, and improve. I highly recommend keeping an active eye on Anne-Marie Charrett and her work on Quality Engineering. She is going deep on this topic and imho, it is absolutely a critical step.

Last, we get feedback quite often around “Profit isn’t everything”. We agree, but it is important. A businesses primary mission is to serve its customers. Profit is a necessary secondary mission, though, so the company can exist and grow.

#2 – We accelerate the team, and use models like Lean Thinking and the Theory of Constraints to help identify, prioritize and mitigate bottlenecks from the system.

Book Reference: The Principles of Product Development Flow by Donald Reinertsen

The Traditional Tester is building the reputation of slowing down ship cycles. The Modern Tester attacks this head on and works to instead speed them up. This is done by identifying bottlenecks and other causes of delay and work to remove them. Reinertsen’s book talks directly about Flow and goes deep into what practices work and why. While is not a book written for QA specifically, it does cover many topics on what causes delays. For example, a recent scientific study published by Dr. Nancy Forsgren in her book ‘Accelerate’, calls out that 1) Developers primarily creating and maintaining tests is correlated with business performance, but 2) Automation owned and maintained by QA is NOT. Why? 1) the code naturally becomes “more testable when Developers write the tests” and 2) they “care about them and will invest more effort into maintaining them”. QA has known for years that ‘moving quality upstream’ will result in the removal of many delays in the system. In this case, Dr. Forsgren’s conclusion is the same as my number one suggestion: Teach your dev’s TDD; help them succeed. Once that is done, discover the next major cause of delay and root it out too.

#3 – We are a force for continuous improvement, helping the team adapt and optimize in order to succeed, rather than providing a safety net to catch failures.

Book Reference: Implementing Lean Software Development by Mary/Tom Poppendieck

The primary goal here is to incorporate a constant incremental “learning loop” into the Development process that shifts Code Correctness concerns and accountability to the people who write the code. I hear quite often that “Dev doesn’t want to do test” as a justification for keeping the Test Last approaches alive. First off, creating an identity for a discipline (QA) based on what a different discipline (dev) doesn’t want to do is not good. Modern Testers seek missions of direct value to the business. Next, Developers are perfectly capable of writing solid code and this includes Test Code. They are just not always incentivized to do so. Furthermore, with a good test team as their safety net, they don’t have to. The safety net keeps the customer protected from code correctness concerns; however, it also keeps dev insulated from those consequences and encourages the ‘it’s your job, not mine’ approach. Instead, start shifting away from this protector role and begin the process of teaching dev to succeed on their own.

#4 – We care deeply about the quality culture of our team, and we coach, lead, and nurture the team towards a more mature quality culture.

Book Reference: The Starfish and the Spider by Ori Brafman

“Culture eats strategy for breakfast, lunch, and dinner.” Creating and sustaining an inclusive community of people is perhaps one of the best ways to create and sustain positive initiatives. There’s no real good way to create successful software if one ignores people. People are key, especially in knowledge work. As an example, I have recently been diving into the science behind the push for Diversity and Inclusiveness. Multiple studies lately have shown that Women in leadership roles not only improve business outcomes, but greatly so. One failing of the movement (imho) is the lack of explaining why. Not surprisingly, it is NOT because what body parts a human does/does not have. Women are statistically more likely to value relationships between other people, collaborate, and include other ideas towards common goals.   These are all learnable skills.  Improvements in Knowledge Work requires new ideas. Ideas are born from old ideas coming together. Diversity grows the number of ‘old ideas’ in the mix. Diversity is not possible without inclusiveness. Building a community where people share their efforts and thoughts towards commons goals creates a breeding ground of practical ideas, including ones you can use now.  Leadership (especially servant leadership) is necessary to make these changes stick and self-sustain.

#5 – We believe that the customer is the only one capable to judge and evaluate the quality of our product

Book Reference: The Four Steps to the Epiphany by Steve Blank

One of the more controversial principles. When correctly interpreted, it can create an identity crisis for those entrenched in Traditional Test culture. It states the customer is the only real judge of quality. The corollary: Test is not a judge.

Who is the Customer? Whoever holds the power over the final buying decision. For example, are you working on a B2B product? Then I hope you are actively helping *your* customers to help *their* customers.

Why is the Customer the only one capable? Because Quality is not about bugs or features. It’s about the application of a solution to a recurring problem.  A problem your customer has.  Quality is a subjective point of view and the only subjective point of view that matters: the customer.  A friend of the podcast, Steve Rowe, said it best. “Test has lost its way“. The Modern Tester cares about quality over code correctness and requirements. And the Customer is the only one that knows whether or not the product is solving their problem. Quality is customer happiness.  (not leads to…   is)

#6 – We use data extensively to deeply understand customer usage and then close the gaps between product hypotheses and business impact.

Book Reference: Lean Analytics by Alistair Croll

Without the data, you are just guessing. It may be an informed guess, but a guess nevertheless. Our job is to deliver value to the customer. Fixing bugs or adding features that don’t matter to the customer is simply wasting time. Time is the one crucial resource that once it is gone, you can not recover. Data is the key ingredient needed to take your “product requirements” (aka guesses) as a start and triangulate the actual needs of your customer. You don’t have to shift to being a Data Scientist as I did. (but it was absolutely a worthwhile journey for me.) However, it is critical that you become accustomed to using data for driving decisions. Traditional Testing was about being an information provider. Our suggestion is go further. Guide the decision and translate it into action. I did a webinar on this for AST. If you can power through all of ‘Ums’ I do in the presentation (I can’t), I do think there’s a lot of helpful content in here for you to start your journey.

#7 – We expand testing abilities and knowhow across the team; understanding that this may reduce (or eliminate) the need for a dedicated testing specialist.

Book Reference: More Agile Testing: Learning Journeys for the Whole Team by Janet Gregory/Lisa Crispin

We acknowledge that principle #7 is scary. Some interpret it as Alan and Brent’s version of “Test is dead.” Others interpret it as Alan and Brent are saying to “Kill Test”. I will be very clear. We are saying none of these things. However, we are saying Testing is an activity (and a necessary one) and further, we are saying this is an activity better owned across the whole team. Even if it eliminates your job as a testing specialist.

Both Alan and I agree with the growing sentiment that Dev and Test positions will eventually merge, however, you don’t have to. Instead, I offer you this suggestion: whether we are eventually proven right or wrong, following the principles will grow you in your capabilities for delivering business results. If we are wrong, you will be a stronger test specialist for the experience. If we are right, then you will be prepared and have options and experiences that will make you even more valuable than before.

In recent years, I’ve had the opportunity to connect with Lisa Crispin a number of times and walked away better for the experience. They say never meet your heroes, because they will disappoint you. Lisa has not. In fact, I am proud of her. She’s acting on the opportunity the changes created for the Quality focused folk amongst us, which “led her down a whole new [exciting] path”.  Keep an active eye on Lisa as well.

Lisa Crispin and Janet Gregory have already been working the Whole team aspect for years. Heck, if all you do is start a book club with your devs, that will be a fantastic start.

While we can not guarantee your experience will match ours, but literally everyone I know who has made the journey away from Traditional Test has stated they would never go back.   Its quite a journey and not always easy, but the “grass is greener” here.   It is my hope that we will get their stories up on the Podcast soon.


Well, that’s it then. Thanks for taking the time to read this and for allowing me to help guide in the written form. I really appreciate the Quality Community and look forward to collaboration going forward. Hopefully, I will start a new blog site soon under a different theme, but until then, feel free to connect with me on Twitter or the Podcast’s slack chanel.



3 thoughts on “Modern Testing Principles Explained

  1. Pingback: Testing Bits – May 26th – June 1st, 2019 | Testing Curator Blog

  2. It’s sadden me to see that it is your last post here, I enjoyed your writing.
    I am really happy you like the mindmap, it is always rewarding to see fruits of your work being used.

  3. #1 – Our priority is improving the business.

    yes, we know, that’s why it’s better to release low-quality products (like MS, Apple, Google does) and say “that’s ok, if customers will find an issue – we will fix” instead to not deliver crap to customers.
    Revenue is the goal

    #5 – We believe that the customer is the only one capable to judge and evaluate the quality of our product

    let’s translate – “we believe that our customers can test for free for us so we won’t pay those people for doing a hard job to prevent issues and not allow us to deliver buggy stuff”
    all those insiders, early previews and public betas are just ways to save money by not hiring people who are skilled enough to really work on identifying issues with process/design/product itself and make top management of companies to earn more money to their pockets.

    There is no real shift in testing or any “quality w/o testing” – it’s just having never fixed bugs for years reported by thousands of people in insiders program for any MS product and getting more and more problems in the same area. Customers know, aha

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s