Unicorns, Data Scientists, and other mythical creatures

Hi all!  It has been a while since I’ve written a blog and since my last post in January, lots of exciting things have happened to me.   Those who have been following me on LinkedIn or listening to the AB Testing Podcast know that I have taken a new job as the data scientist manager in the Azure team.  In the short time since I have started in March, I can easily say this is absolutely the best job I have ever had.   In no small sense, it really feels like a job I was meant to do from the start of my career.   It’s funny to me.   As I sit and type this, I am reminded of my mentor in High School, my calculus teacher.   He was a cranky man, who in one part loved his job, but in another felt like he had accepted second best in life.   He frequently mentioned that one day when *his* high school math teacher died, he would go piss on that grave.   He felt that his mentors had not set him up to succeed in life and was driven to not repeat that error with his students.   A story for another day, but soon after college, I would learn that same sense of responsibility to teach those what I wasn’t taught.   My mentor wanted me to be an Actuary.   I loved math (still do) and went to college with that in mind.   I bolted on a Computer Science degree once I learned how much I loved it, but upon graduating, my Math degree would encapsulate knowledge that, in general, I would not use for 20 years.   Until about 4 years ago.     Needless to say, I really love my job, what I am doing, and problems I am solving.   I do wish, now and again, that I could wind back the clock and be where I am today, but have another 20 years to master my new direction.  C’est la vie.  So much cool stuff to learn and use and so little time.

Very similar to my last team, my new team did not have much experience in their new space when I started and I am grateful to be on the ground floor of what we are building there.  Much like in many places in my company, many of the folks are old SDETs and dealing with the change is an ongoing challenge, but not one that I am unfamiliar with.   Honestly, it is going nicely in my humble opinion, but as more and more people are learning what data can do for a business, the pressure to hire and train more data scientists is ever increasing.     In the last 9 months, spontaneous 1:1s with folks have increased by an order of magnitude with folks who are: 1) looking to hire a “data scientist”, 2) looking to become one, or 3) looking to preserve their current position.    Today’s post is mostly about issue #1.  Although, #2 is also interesting to me as the majority of these folks have been Program Managers lately (which might indicate a sign of change in that discipline).   #3?   I would say that’s the majority of what I speak to on my blog and on the podcast, but if there are specific questions, send me a tweet.   We will feature it as part of our Mailbag segment on the podcast.  We love the mailbag!

This post was inspired by a talk I attended at this year’s Strata conference in New York.  The presenter, Katie Kent, did a talk on Unicorn hunting in the data science world, which I thought was fantastic.  Her company Galvanize.com offers a 12 week immersive course that claims to prep you for a Data Science role with a 94% success rate.   I haven’t researched this myself, but maybe I will test it with a few employees of mine and report back.   Might be another alternative for the #2 issues I mentioned above.  I can say Katie’s talk was great and it resonated.   Many of the discussions I have had recently was *exactly* in this problem space.   Managers coming to me or my manager wondering how to quickly learn from us being vanguards and asking how to take advantage of Data Science and “maybe if I can get an open position, you can help me hire one?”.


Yup, that’s right.   One!


Katie’s talk was about Unicorn Hunting.   The elusive Unicorn was the perfect singular Data Scientist a company could hire that could solve all of its needs in the data science space.    I regret to inform all of you except Katie.   They are extremely rare (perhaps, rarer than the Unicorn) and if you can find one, you will probably not be able to afford him/her.  (note: if you can though, you should!).   The challenge here is this perfect Data Scientist would have to be an expert in too many very distinct fields.  The ones I am aware of that exist are indeed Rockstars (in the Stats world), but these aren’t the folks you will successfully hire into your one-off position.

One new experience for me on my new team has been to hire Data Scientists.   Almost all of the folks I have interviewed have been PhD’s, but the best have only achieved Master’s degrees.   To date, I have not met a candidate with only a Bachelor’s degree.  <aside> This, by itself, is interesting.   If one can become a data scientist in only 12 weeks, why not 4 years?    </aside>   Master’s candidates are great, I think, because they have 1) learned some depth and 2) have stayed grounded in the application of their craft.   I’ve learned that there’s a big debate in academia with respect to post-doctoral individuals and whether or not to join Industry.     They are pushed to push the boundaries of the science and I think somewhere along the way, the drive to apply it to real life dissipates.  Masters students are more applied scientists, in my humble opinion, than theorists and as a result, more immediately useful to a business.  This is an exaggeration, of course, but highlights another cause for the Data Scientist shortage.  The PhD folks that do venture into industry and survive it are helpful.     They are able to pull the practiced, but old, learnings of Data Science closer to what’s currently known which causes an acceleration of all involved.

However, this is knowledge work and there’s too much of it.   Due to the cognitive limitations of any single human, it will *always* be rare for just 1 person to be even good enough for what is needed end-to-end.

Depending on who you talk to, there are multiple definitions of a Data Scientist’s job.  My present favorite: A Data Scientist helps a business drive action by understanding and exploiting relationships present in the data.   There are 4 key principles buried in that definition.

4 Key Data Science Principles:

  1. Actionability – the recommendations must be interesting, valuable, and within the means of the business
  2. Credibility – In this business, Objective Truth is *everything*.   It is ok to communicate confidence intervals, but it is not ok to be wrong. Data Science teams get, AT MOST, one time to present wrong data, insights, recommendations to an executive. It is wise to remember this.
  3. Understanding Relationships – this is the bread and butter work most data scientists are hired to do.   There are a vast sea of techniques to use for digging knowledge out of data. One must also have the ability to understand what it means.   Domain Knowledge is critical.
  4. Data – lots and lots and lots of it

To be able to succeed, one must turn data into knowledge and make knowledge work.   Sounds good as a t-shirt motto, but in practice, this is hard.   It takes deep knowledge from several disciplines to turn this into something efficient that scales to not only the amount of data being processed, but also the timeline the business requires in order to benefit from the discoveries.


In my experience, it takes a team to pull this off.


In my observations, here’s what you need:

  1. Data scientists – starting with the obvious.   However, what may not be obvious is that Data Science is a very wide umbrella.   But there are 2 major branches, and likely, you will need both.
    1. Applied Statistics – The ability to prove/disprove known hypotheses in a deductive manner.   You have a belief already in hand and you are trying to prove/disprove it. Applied Stats techniques tend to be faster than Machine Learning.   A few simple histograms are easy to pull together as an (exaggerated) example.
    2. Data Mining – Using Machine Learning techniques in an inductive manner. You start without a preformed belief, but instead a goal (such as predict when a customer will churn) and let interesting patterns within the data unveil themselves. (note: interesting is a Data Science term and it can be measured. )   Machine learning techniques, in my experience, handles Big Data problems better.   They can scale to the size of data better.
  2. Data Engineers – Engineering the movement, storage, indexing, parallelization, cleansing, and normalization of data is a very hard problem and MOST data scientists do not know how to do this.   As Big data grows, this role, already critical, becomes even more so.   Credibility starts with the data and these guys are key to caretaking and monitoring it. They should be paying attention to not only traditional RDBMS solutions, but to technologies such as Hadoop, Splunk, Azure Data Lake, etc.   Each of these solutions come with their own pro’s and con’s and you need someone who knows what they are doing. These folks should understand the architectures end to end from data emission to visualization. There is NO silver bullet and you need a person who understands the trade offs.   Every executive wants 1) a cheap solution, 2) near real time, and 3) inclusive of all the data. Cheap, Fast, or good: Pick 2.
  3. Computer Scientist – Especially in distributed computing.   The current state of the art for Big Data is to parallelize and send your code to the machine that is storing the data to do calculations (Map Reduce, in nutshell). This greatly reduces the time spent as code is easier to move than the data, but even so many of the techniques are O(N2).   Polynomial time is too slow (or expensive even with millions of machines running in parallel).   There’s an active quest for O(N) and O(1) solutions to Data Science problems as well as clever approaches to Data Structures that helps to improve speed and storage costs.   One new item that I have not spent nearly enough time on is in the use of heuristics.   More here later.
  4. Domain Knowledge Expert – Even if you got the people with the skills above, you still need to be able to understand what the data *means* in order to move forward. Typically, the data is being emitted from telemetry from lots of product developers.   Unreasonable to expect 1 person can know everything about how the product works, but you will NOT succeed if your in-house data science team knows nothing.
  5. Business Expert – You need to be able to understand what the business goals are and how to translate your uncovered insights to support decision making.   This takes art in communication and visualization.
  6. Agile – An agile coach is needed in order to pull together these folks and get them working together towards common goals.   It does NOT make sense to over focus on *any* of the above specialties.   all roles are necessary to succeed and since this team is working to improve the business, adaptability is key.   As new knowledge is gained, the team needs to be able to shift in the new direction sustainably.   This happens A LOT!
  7. Manager – Really I wanted to put Orchestrator here, but you really need someone who is a Systems Thinker who is crafting strategies to the best effect for the business.   These folks need to work together.


You can scale these “requirements” up or down depending on the problems your team is facing, but the people on the team should be working in unison like a choreographed dance or an orchestra.   They should be one team and not individual teams and focus on vertical slices of “value” being delivered.  No one person can do all of the above.  3 to 4 might be the minimum to produce a team that is outputting something that is considered valuable.  Imho, 5-7 folks with the above skills is about right as long as you have the right depth and breadth.

Lastly, AB podcast listeners will not that I frown on specialists.   This is true here as well.   Every person on your team should be able to perform at least 2 of the functions above (ideally, 3) with at least 1 place where they can competently achieve deep results.  In addition, it’s a good idea to minimize the overall overlap of expertise on your team and rely on your Agile coach to create an environment of knowledge sharing and team cooperation.

In closing:

While if you are patient, you will be able to create a team of folks who, together, have mastery of the above and if you are also lucky, keep that team small, but don’t try for the Unicorn.   They do exist, but are too hard and too expensive.  Even if you manage to land one, if the business isn’t prepped to include them in the strategy, you will likely not get the value you hope for from them.   Knowledge work cannot be done in a silo in the fleeting windows of opportunity many are encountering today.

Thanks for reading and Happy Holidays!

In pursuit of quality: shifting the Program Manager Mindset

Hello all and a very happy New Year’s wish to you. It has been a LOOOOOOONG time, since I have written a post (as several folks have reminded me). While it remains my goal to post at least once a month, I know I won’t always achieve that goal. These past several months have been a particular drain on my time, probably my most valuable commodity at this time in my life. One of my New Year’s resolutions is to reprioritize how I spend my time. My goal is to focus more deeply on my personal retraining and on my family. I believe many of my readers are considering or following a similar path forward as am I. With luck, the time will avail itself, so I can publish what I’ve learned and share.

As part of that retraining, I’ve gone back to school. I am quite happy to report that I am now done with my first year of a MS in Analytics and thus far, I’m a straight A student. Honestly, though, it is a bit odd going back to school in your mid-40’s. I really could care less about the grades (it has been decades since someone has asked me my GPA). I am very passionate about learning the topic AND I have school age children who are watching my every move. I just simply can’t have teenagers calling me a hypocrite. The drive to learn the subject matter is really helpful to get the grades. I wish I had such a similar motivation the first time through college. J Thus far, I have been able to bring every class I have taken back into the applied context of my work and have done something valuable with it. I’ve got a ways to go, for sure, but the journey is been a lot of work and a lot of fun.

At work, I am helping to land something *very cool* for shifting the world into a more Data-Driven culture. I am on the Power BI (Business Intelligence) team in the SQL divisions. We are enhancing our SaaS offering and a preview for the public is available now with our new features. Check it out. The price tag for the public offering? Absolutely free.

I am quite proud to be a part of this effort. I hope you check it out and provide feedback.

News Update:

Since it has been a while since I last posted, I thought I’d cover a couple of quick topics:

  • Layoffs – Several of my brothers and sisters at Microsoft got laid off in the last few months. Most of the impacted folks that I knew have already landed new jobs and are reporting being even happier than they were. There are still a few people that I know are looking. If you are looking or seeking, please feel welcome to send me a note on LinkedIn and/or Twitter. I’ll see if I can leverage my network to help broker new relationships for people.
  • The AB Testing Podcast is back after a couple of months hiatus. (ok, this is old news if you are one of the “three”) You are welcome to check us out and send any question, comment, or feedback you’d like. Alan and I are both change agents of a sort and believers in collaboration and community for accelerating achieving goals. I, for one, absolutely adore the Mailbag segment. Any way we can improve the podcast, talk on something that might improve your life, or share your success story, send us a note. We’ll talk about it “on the air”.
  • Other agents of change: I’d like to call out others who are putting “pen to paper” in to help the community grow and converge.
    • Ben Bourland – Has started a blog recently and is journaling his experiences and insights on the shift from Test to Quality.
    • Steve Rowe – A long time blogger and QA manager in the Windows org. He is also trying to influence change towards data-driven.
    • Michael Hunter – Another long time test blogger and friend did a live presentation recently at SASQAG. The link to the video is here. His talk is on how he came to the realization that he had strengths that were valuable in many disciplines and that actually testing was not something he had every actually enjoyed. I share this in case his journey inspires others to let go of their fear that the discipline is changing.

More Changes

About a year ago, I wrote a post with the intent of helping Testers manage the change. A lot has changed in my company as well as others around us, and it is now fairly clear (to me, at least): Test as a dedicated team is a thing of the past. Testers have shifted into Data Analysts roles, development roles, infrastructure roles, or specialists in NFRs (non-functional requirements) such as End to End Integration and/or Performance. Very few “Testers” still exist. However, the transition is far from done. Many Testers have just been bolted onto Development teams under the guise of combined engineering, but the actions of those involved haven’t changed. In some teams (thankfully, including mine), Testers are gone and their skillset is slowly being incorporated into the team as a whole. However, it’s a slow difficult change for the prior development team to fill the void. A wise manager once told me, “Brent, be patient. It’s a marathon, not a sprint.” This change is far from over, but definitely moving in a more productive direction.

However, one role that persists that continues to show very little sign of change is PM. Program managers historically have owned the requirements for defining what we are bringing customers, then driving a schedule towards delivering it. Most PMs spend time speaking with customers and build a strong intuition on what customers want. They are generally very charismatic and likeable and work hard to try to make people happy. While I have never been in PM, back in the day, my father was Director of PM for multiple companies. We now have very interesting conversations about our differing experiences. For example, one commonality my father seemed to share with other PM’s I know was a drive to get into the executive role. I have had an opportunity on multiple occasions to work directly for the executive in my team. I think my dad was more than a little disappointed when I have told him “I have very little drive or desire to do that job.”

I was once told by a mentor that “Test doesn’t understand the customer” and that this was the main reason why PM advanced to executive more often than not. I think most PM’s I have spoken with in various degrees or another share this opinion. However, it is my belief that the next big culture shift that is about to come. Right now, PM, too, either won’t understand the customer and/or their ability to act on the appropriate window of opportunity will be too slow. I believe the primary root cause for this will be an over reliance on their intuition and a reluctance to test it out. PM’s are used to being able to rely on their soft skills and having a long time to react to the market. In comparison to techniques used by competition, the result is slow and often just plain wrong. Moving forward requires a mindshift in the program management organization.


I have mentioned that in today’s world: speed to market on value adds is paramount. Old school PM Techniques, unfortunately, are too slow and do not scale. Thus far, PM has remained relatively unscathed as part of these changes, but I firmly believe their judgment day is coming next. Their careers are at stake. Consider this: as data science takes further hold into organizations and teams light up the ability for individual engineers to make correct and actionable decisions on their own, the need for a team Customer Expert (ie. PM) becomes *dramatically* reduced. For those paying attention, one will notice that need for PM to take on the role of schedule meister has already been cut way back. Unlike Test, I believe PM will be still needed, but I do believe we are nearing a time where we will see their numbers greatly reduced. PM must learn to adapt and apply this new knowledge to not only survive, but thrive. IMHO, it will be those who learn to balance their intuition with the data towards actionable and valuable decision making that will differentiate themselves from the pack.

Better Together

I have had multiple conversations with the Data Science groups throughout the company and a new series of problems are becoming quite clear:

  1. No Customers – Complaints from the data guys that they are building assets that should be being used, but aren’t.
  2. DRIP (Data Rich, Insight Poor) – The items they are building that DO get used are, in essence, Scorecards and Dashboards filled with Vanity Metrics that aren’t shifting the needle towards anything the business values.

There’s an obvious symbiotic relationship between program management and data science. These folks need to be working together in concert like a well-oiled machine. I really don’t understand why they aren’t, but it’s clear from the number of people talking to me about it, this isn’t happening or not happening sufficiently enough. I think a big part of it though is PM doesn’t know how to take advantage of the data teams and the data teams don’t know how to express their value in a way that resonates with the PM in a non-threatening fashion. By that I mean a win-win scenario, I’ve spoken with several PM’s that very strongly believe this is all “data crap” and their intuition is all that is needed. True, Steve Jobs did it. I would argue that Mr. Jobs was special. He was one in a million whose intuition just happened to be right. The rest of us are wrong all of the time. The positive thing though: PM doesn’t have to do it alone (and, in my belief, they probably can’t). Your data science team is ready, willing, and eager to help.

Show Me the Evidence

I believe PM need to invest more heavily into understanding how to do Evidence-based decision making (aka Hypothesis Testing). A key principle for their lives going forward should be: No matter how right you think you are, due to uncertainty, there is *some* chance you are wrong. Therein lies the problem, there is always some risk associated with uncertainty (such as wasting time/resources on a problem customers don’t care about us solving). Please feel free to leverage your intuition; this new world is *NOT* about intuition versus facts, but rather intuition validated by facts. Both… Together… Intuition on its own can very quickly lead you in the wrong direction. Likewise, facts on their own can lead you to optimizing your current business, but will not help you to find the breakthrough game-changer with higher business potential.

Hypothesis testing in a nutshell

The goal of hypothesis testing is to be able to confidently select the best available next action to take.

NOTE: “Best” is relative. Good enough *IS* in fact Good enough. One common error is see PM’s making a lot lately is deferring a decision until they have 100% accurate and precise information. You simply do not need this. Leveraging heuristics and a solid understanding of how to use confidence intervals will take you far. For example, Douglas Hubbard’s Rule of Five tells us that there is a greater than 93% probability that the median of a population is between the smallest and largest values in a random sample of only five items. Do you *really* need to know the median? Or is knowing the range all you need?


  • Idea Phase
    • Write down your hypotheses individually
      • These are in a form of a statement (not a question)
      • These must also reflect a business KPI.
  • Knowledge Phase (knowledge is information in action. The ability to use information)
    • For each hypothesis, enumerate your possible actions
      • What actions will you take if the hypothesis was true?
      • What actions will you take if false?
  • Information Phase (information is data organized in a meaningful way)
    • Now you need to enumerate the questions needed in order to confidently select the appropriate action.
      • Battle confirmation bias – the human tendency to search for, interpret, and remember information in a way that confirms one’s preconceptions
  • Data Phase (data are raw facts that are meaningless by themselves)
    • The last phase is to simply enumerate the data points you need to answer your questions.
      • Most of the time you will be measuring customer behavior. Measure the behavior you want customers to take, instead of trying to measure everything.
      • Be wary of Hawthorne’s effect: People’s behavior will change in accordance to how they know they are being measured.
    • Get these instrumented or build a heuristic based on your existing instrumentation that can be used to answer your questions.

Hubbard provides 4 very useful measurement assumptions to consider when you are designing your data phase:

  1. Your problem is not as unique as you think
  2. you have more data than you think
  3. you need less data than you think
  4. an adequate amount of new data is more accessible than you think

Example: (oversimplified)

Hypothesis: Within my product, enabling Users to share with other users via Facebook will cause a notable increase in Acquisition and Engagement KPIs.

Possible Actions: (not complete)

  • True
    • No action
    • Optimize – make it really easy for users to share
    • Enhance – improve the sharing content to entice receivers
  • False
    • Abandon future feature development or cut
    • If acquisition improves, but not engagement, develop new hypothesis.
    • If engagement, but not acquisition, improves, develop new hypothesis.

Possible Questions: (not complete)

  • Which users shared?
  • Which users received invites?
  • Which receivers entered the service due to a sharing invite?
  • How does acquisition via sharing compare to other acquisition means?
  • How does sharer’s engagement level compare with those who don’t?
  • How does receiver’s engagement level compare with those who don’t?

Data Needed:

  • Users who share & receive
  • Receivers’ acquisition date & means
  • Acquisition rate correlated with receiving rate
  • Engagement rate correlated with sharing rate
  • Engagement rate correlated with receiving rate

If PMs and Data teams work together to craft their hypotheses (and experiments), more precise and accurate decisions will be made. Ideally, with the minimal amount of new instrumentation having to be added to the product. Be prepared to be wrong… A lot… But celebrate it. The faster you are wrong also turns out to be the faster you will *stop* being wrong by building fast, actionable knowledge towards business goals.

One final note:

There is one other phenomenon that seems to becoming common in the post-tester world that needs to be addressed.

PM, you need to STOP testing for your developers. Yes, yes, I know. “But Brent, I am now being held accountable for the customer satisfaction level for my feature and what I keep getting is bugs, bugs, and bugs.” Remember that this is a system problem. We’ve changed the software development system by removing testers from it, but by no means does this imply that this was done with the right tooling and training in place. Nor does it mean that you and your peers in PM have figured out how to truly determine the right number of features your dev team can produce in order to confidently get to done, done, done. There is much work to do here, for sure. You need to play your part.

How? Stop being the safety net. It is absolutely ok for you to be a part of the team and help everyone improve the quality of the product under development, but always consider whether or not your role in doing this is becoming more and more required. This might imply a dysfunction in the system and the team has learned that they can avoid doing testing themselves by shipping their crap to you. It’s super seductive for a dev. I, myself, have found my directs doing this, at least a couple of times in the past several months. It’s also seductive for you. Usually, when you find that critical issue, you will get praised for how well you really saved my bacon. Both sides get an endorphin rush.

Instead, I encourage you to strike a different balance between improving and shipping your features and playing your new role in this system. My recommendation is to take on the strict role of the PO for your team. A key responsibility of the Product Owner is own the Acceptance phase. Acceptance does not mean you find bugs. It means you own the decision as to whether or not the feature is ready to ship to your customer.

You can do this easily by doing 2 things:

  1. Making sure your team must doing an Acceptance Interview with you before they do final checkin into the shipping build.
    1. It is an interview only. DO NOT open the product. This is for the short term only and is required in order to set expectations: it is not your job to disprove/prove that they are done. It is theirs. It is your job to be satisfied that they have done so. Once your team has shown signs of using the new expected behaviors, feel free to change the interview process if it makes everyone’s life easier, but until then, resist the urge.
    2. Usually, you will have 2 lists for acceptance: 1) the requirements (acceptance criteria) defined when the item was still in the backlog, and 2) the non-functional requirements (perf, scale, localization, security, etc) that all features must be scrutinized against.
    3. For each requirement, simply ask the developer: “how did your prove readiness for this requirement?”. If their answer is vague, follow up with more precision questioning.
    4. Using your best judgment, if you don’t like what you are hearing, then Reject with your rationale.
  2. Being brave. For a short period of time, you will be causing disruption in the system. Your devs may have already gotten used to your new role. Change of this sort almost always causes anxiety. This anxiety may come with consequences depending on your team’s culture. In addition, your actions will likely result in several of your key stories/features not completing in this sprint. Remember your job is first and foremost to satisfy your customer. Your preference should be to do that alongside your team, but if your team wants something else, you may need to “go it alone” for a little while. Remember also everyone on your team are professionals, but they may have some bad behaviors due to existing muscle memory. Your job should be to help coach them to learn new ways.

As always, thank you for reading and I appreciate and welcome any feedback you’d like to offer.

Systems Thinking: Why I am done with agility as a goal

    Recently, I was writing up a presentation where I was going to state that the New Tester’s job definition was to “accelerate business agility”. One of my peers looked at it and remarked “Isn’t that sort of redundant?”. After some discussion, it became clear that “agility” did not have a clear well-understood definition.

To be clear, I am MOST definitely not done with Agile methods, but as best as I will be able to, I am done with using the word ‘agility’ to describe it. If one looks this up in your favorite dictionary, you will find it described as “moving quickly”. While moving quickly is certainly a valuable goal, it is pitifully insufficient in the modern day software world and if not tempered correctly, can actually lead to more pain than what you may have started with. When I now give talks on Agile, my usual starting point is to first clarify that Agile is NOT about moving quickly, so much as it is about changing direction quickly. So in a nutshell, Agile is not about agility. One problem I am trying to unwind is the dominance of strong-willed, high paid folks proclaiming that Agility is the goal and quite simply, they do not know what they are talking about as evidenced by the typical lack of details explaining behavior and/or success changes their team should be making. This causes their reports to “follow” this guidance, but left to their own devices to make it up. A few clever folks actually study it and realize that shifting to Agile is quite a paradigm shift to succeed and hard to do. This can be a slow process, which seems to contradict the goal of “moving quickly”, so gets abandoned for a faster version of Waterfall or similar dysfunctional hybrid. There’s a common phrase in MBA classes, “Pick 2: Cheap, fast, or good”. This implies a singular focus on fast is likely to deliver crap and at a high cost.

One quick test to see if your leader understands: Ask how much are we going to invest in real-time learning. Then observe how those words align with actions. Moving fast without learning along the way is definitely NOT Agile, but more importantly, it is wrought with peril.

Many of my recent blog posts are on the topic of leadership lately. If you find yourself in such a role and are trying to lead a team towards Agile, my guidance is that you think carefully about the goals and behaviors you are expecting and use the word that describes it better. If you don’t know what you want, then get trained. In my experience, using Agile methods is very painful if the team leadership does not know what, why, and how to use them.

Consider these word alternatives:

  • Nimble: quick to understand, think, devise, etc.
  • Dexterity: the ability to move skillfully
  • Adaptability: the ability to change (or be changed) to fit changed circumstances

These ALL make more sense to me than “moving quickly”, but adaptability is what fits the bill the best in my mind.

    In my last post, I focused on one aspect of the shift paradigm shift happening in the world of test towards the goal of improving adaptability. I have mentioned before my passion (and the primary reason I write this blog) is about Quality. However, to make a business well-functioning in this modern age, a singular focus on changing the paradigm on quality is not sufficient. As Test makes its shift, other pieces of the system must take up the slack. For example, a very common situation happening is that Test simply stops testing in favor of higher value activities. Dev then needs to take up that slack. If they don’t (and most likely they won’t initially), then they will ship bugs to customer and then depending of customer impact, cause chaos as dev attempts to push testing back. We need to consider the whole system, not just one part of it.

A couple of months ago, I was asked to begin thinking through the next phase of shifting the org towards better adaptability. Almost immediately, I rattled off the following list of paradigm shifts that need to be done to the system as a whole.








Spider teams

Starfish teams


Quality (value)



NIH is bad

NIH is Awesome

Large batch

Small Batch







Green is good

Red is good






Shared Accountability


Hopefully, you can see that moving quickly is certainly a part of this, but more importantly, this list shows a series of changes needed for focus, sharing, understanding the current environment, and learning…

Recently, I have come upon some material from Dr. Amjad Umar (currently, a senior strategist at the UN and one of my favorite professors) where he argues that companies should be plan-fully considering the overall “smartness” of their systems. He states that technologies alone cannot improve smartness. But you can improve it by starting with the right combination of changes to your existing People, Processes, and Technology. Smartness, by the way, is analogous to Adaptability.

I have taken his concept and broadened it to something I call “Umar’s Smartness Cube”. I think it nicely describes at a high level what needs to be considered when one makes System changes. The goal of the whole cube, of course, is to improve Business Value.

How to use this to improve your system:

  1. First determine and objectively measure the goal you are trying to achieve.
  2. Consider the smartness cube and enumerate opportunities to improve the above goal.
  3. Consider tradeoffs between other elements to achieve goals better. For example, maybe we don’t need the world’s best technical widget if we just change the process for using what we have to reduce the training burden.
  4. Prioritize these opportunities (I like to use (BizValue+TimeCriticality)/Cost)
  5. Get them in a backlog that acts like a priority queue and start executing.


This, of course, is over-simplified, but hopefully, sets you in an actionable direction for “accelerating the adaptability of your Business (system)”.

As thinking-in-progress, any feedback is appreciated.

AB Testing – Episode 0100

– Alan’s upcoming presentations at Star East
– We explore the potential topic for Alan’s 5 minute Lightning talk
– We spend a good deal of time on the continuing need for Leaders to help resuscitate Test Zombies.
– very briefly talk about Gamification for Engagement
– And Alan drops a surprise bomb on me

It’s been awhile since I’ve actually written something and I have a couple of topics I am itching to talk about. I will try to get one of those out this weekend.

Want to subscribe to the podcast?
RSS feed

Also, on the Windows Phone store. Search for “AB Testing”.

AB Testing Podcast – Episode 2 – Leading change

The Angry Weasel and I have put together another podcast. In this episode, we talk about problems we see in leading change towards a better direction. We cover some changes we face, change management, and reasons change fails. We talk about the importance of “why” and leverage the “Waterfall vs. Agile” religious war, as an example.

We managed to keep our “shinythingitis” to a minimum of slightly less than 70% of the time. :) Enjoy!

Want to subscribe to the podcast?
RSS feed

A/B Testing Podcast goes live

So one day, a colleague and friend, Alan Page,  said “Hey, Brent, why don’t we start a podcast?”.   After much discourse (Alan pushing me) and challenges (me finding the time), I am happy to announce that I suck at it, but am doing it anyways.  I am very thankful to have an old pro who is letting me tag along for the ride. Anyways, if you’ve got a 30 min to kill and want to hear some more from a couple of guys who are trying to help lead change, please check it out. AB, in this context, stands for Alan/Brent and in “Episode 1” we explore Testing vs. Quality as well as recent changes happening at Microsoft.  Enjoy.  Feedback is always welcome.   We are likely to keep going until someone tells us that the pain is unbearable.  :)

Download:AB Testing – “Episode” 1

Or play it now:

Want to subscribe to the podcast? Here’s the RSS feed.