The Combined Engineering Software Model

Happy New Year, everyone! I find myself looking forward to 2016 despite, perhaps, some of the worrisome predictions I may have made in our most recent podcast.  2016 should be the year when I finally complete my Master’s degree, celebrate 20 years of marriage with my adorable wife, take a really exciting vacation (TBD) with my family (my eldest turns 18 this year, so want to do a “big bang”), and make some serious progress advancing the principles of customer focused quality and engineering in my Data Science job. Big year!   I hope your year turns out even better than mine will be. J

Today’s topic is something that Alan and I have gotten lots of tweet, emails, and questions.   It IS something we’ve talked about a lot on the podcast, but not in a very cogent fashion.   A quick search on the web yields nothing, so I’d thought I would add some detail.

Please note: on the podcast, Alan and I will often switch between discussing Combined Engineering and Unified Engineering models. These are the same things.     The model is called Combined Engineering, but honestly, we both think Unified Engineering is better.   I think of an Oil and Vinegar salad dressing.   You can shake it really hard and combine it together, but it’s really not cohesive and will separate again. You will have to shake it up again to reap the rewards. Too much work.   Unified in my mind is more like Ranch dressing.   Several ingredients melded together to make a whole new awesome thing.

A quick note

This is not my model. It was first deployed by a pair of leaders in the Bing team (I am leaving out their names to “protect the innocent”). One who grew up in Test and the other in Dev. Both are brilliant guys and have ascended to high ranking positions in my company.   However, I was in Bing when the model was piloted and was instrumental, a year and a half later, to helping my next organization move to the model when I left Bing.  About 2 years after the model was first piloted, it hit the rest of the company like gangbusters. The majority of the company was switched to this model now and dedicated positions in Test are remarkably rare.   I’ve met only 1 person in the last year who’s title is still Test Manager and honestly, I felt sad for her.   It takes time and effort to do the transition and she and a team were already about 1 year behind the rest of the testers in the company. I remember thinking at the time: “How can I help this person? Surely being in Test as well as being in management is going to make this person a target for the next set of layoffs (God forbid that they happen again sometime soon)”.

I am not fully aware of what was the inspiration for the guys that piloted the CE model, but I am very positive that it was grounded in Agile.

History

As of this month, I have been with the company for 22 years and during that time I have experienced 3 very different organizational models.   The first was the PUM (Product Unit Manager) model where a business was run entirely by one person, the PUM, who would have middle managers for each of the disciplines reporting to them. The PUM would report to a VP and the middle managers would have frontline managers as directs.   The frontline managers would generally partner with the managers of the other disciplines and their collective teams would work to add features together, but with clear dev, test, and PM boundaries. The second was known as the Functional model. It took the idea of clear distinctions between the disciplines even further. PUM roles were eliminated and in replaced by 3 new heads, one for each discipline. This continued up the entire organizational stack.   Directors and even VPs in Test, were quite common. The idea of this model was that each discipline would be far more efficient due to allowing them further control over optimizing their craft. It would reduce waste and duplicated effort.   Lastly, the CE model.   I think of this model as mostly the polar opposite of the Functional model.   It rejects the notion that discipline optimization is the key for producing ROI, and suggests getting teams to use their very different skills together towards business goals in a tight knit fashion is more effective.  It is important to note that on all teams that I have encountered, the CE model has almost no impact on PM. However, Dev and Test are combined into the same frontline team a single leader and are accountable for both the development and testing of the features they are producing.

CE Goal

OK, there’s a LOT to discuss with respect to the Combined Engineering model, but absolutely beyond a shadow of doubt the first and foremost should be the goal of it. Be forewarned: in no way should “implementing CE” be the goal.   In order to execute on a business strategy, CE is one of many implementations to consider for your organizational model, but it should be considered alongside the business outcomes and strategies.   The goal in a nutshell is to create an environment where empowered individuals with complementary, but different specializations, are working together towards common team goals.

If it makes sense for your business, do enter into CE, but don’t do it flippantly. There can be a lot of change incurred with the model and such change brings risks.   Consequences I have seen include significant slowdown in project effectiveness and speed and massive morale decrease.   Where change management strategy has been neglected, I’ve seen those teams lose their best people which further disturbs the downward spiral.   However, a strong implementation results in the opposite.   Huge performance and morale boosts.   Teams that feel like families and work together to achieve the best and most important goals for the business.

Combined Engineering, accordingly, is a very strong complement to your favorite Agile Methodology.   I’ve regaled on the importance of considering software development as knowledge work.   Dig into that statement further than just surface level. As Peter Drucker tells us, knowledge work is distinct in that usually the task is not known, but rather, must be determined.   On an assembly line, the work is known upfront. In Knowledge work, there is only outcomes and a selection of choices. Each person on a knowledge working team holds a set of the pieces for the business jigsaw puzzle. CE works to activate Knowledge Sharing amongst team members and get the collective knowledge of the team working together towards business goals.   It creates collaboration and unity and from this, high ROI productivity.

When to implement

My Inner Agile Coach screams “always” as a response to when to implement CE as part of the business strategy, but this is not true. If your product has static and relatively stable goals and/or long release cycles, then your product likely has no fast feedback loop between the engineers and customers. No real reason to change or adapt. Software being released to governments, packaged products or software embedded in your coffee maker or car are examples.   Honestly, I would push to make product changes to enable fast feedback loops before I would consider a CE model.   Otherwise, the benefits are unlikely to be worth the cost, pain, and risk incurred from the changes.

However, if you are able to ship at “internet speed”, have a strong leadership that is trying to push decision making down to its soon-to-be-empowered staff, have the ability to react quickly to competitive pressures and/or customers’ demands AND have a discipline-segregated workforce, then IMHO, this is a no-brainer.

How to Implement

On paper, the implementation is cake. I recommend using the simplest implementation.   Yes, it won’t be perfect, but you can change it 3 to 6 months later once you’ve learned the next change problem to resolve.   Making everything “perfect” up front takes time and creates anxiety.   Better to just start by making the smallest amount of required change NOW. Here are the simple (naïve) steps:

  • All frontline Dev/Test managers are now Engineering managers and no longer representatives for just their discipline.   Managers, who were once partners (in the functional model), are now peers.  (note for simplicity, I will continue to use their former titles though)
  • The product they used to co-own as peers should split in half as vertical slices. One half going to the former Test manager and the other to the former Dev manager.
  • Team accountabilities are the same as they were previously except now instead of being split by discipline each leader owns this within their team.   Both teams are responsible for feature development, unit tests, test harness improvements, etc.
  • The individual developers and testers need to reorg. How?
    • Split into 2 teams of equal sizes
    • The TOP 20% Best Testers go automatically to the Dev Manager’s team
    • The TOP 20% Best Developers go automatically to the Test Manager’s team
    • The Test/Dev Manager work together to appropriately place the rest of the individuals into each team.
  • Create Monthly team based goals for each vertical slice and do so *without* acknowledging where the team members came from. Ie.   The Test manager should not have easy to achieve development goals, nor should the Dev manager have softened quality goals.
  • The Test manager gets a new mandate:
    • He was given the best developers in order to bootstrap his success and help transform the rest of his team.   He should use these folks to aggressively train the others.
    • Goal: At the end of 6 months, every former developer should have created and executed their own test suite and every former tester should have checked in their own bug fixes/features into the product.
    • Training will be offered, of course, but the accountability for success falls to the Test manager
    • He has 6 months to achieve that goal. At the end of the period, each of the developers that were forced to move will be given the opportunity to change teams, if desired.
  • The Dev manager gets a similar mandate especially regarding the best testers
  • Obviously, this relies on a strong leadership team that can communicate the expectations and outcomes in a fashion that alleviates concerns. We are forcing people to move teams.   Most uncool, but the 6 month escape clause helps here and honestly, you really do need these experts to help with the training.
  • Lastly, you will need a public “incentive” plan that clearly remarks on how performance reviews will be judged based on the ROI of the features that individuals produce.    You will not be able to get a good review score by either the previous test review standard or dev’s equivalent.   You are NOT incorporating testers into the dev team (or vice versa), nor are you now going to judge Testers by the long practiced dev standard.   A new Engineering standard will need to be developed that makes sense for your business/company, which will be 50% weighted towards test’s benefit and 50% weighted towards devs. (Remember, you will change this later when you learn more).   The goal here is to breakdown the old us vs. them discipline thinking. Your ideal standard will be one that equivalently makes both dev & test comfortable that they can succeed.

The agile-wary reader will notice that, in essence, you are trying to take a team of specialists (dev) and a team of generalists (test) and merge them into more fluid teams of generalizing specialists and specializing generalists.   What I have found, is that the developers have a much harder time with this than the testers.   The testers will be mostly concerned about all of the bugs that will be shipped to the customer now without their protection.   They will also be concerned how they will be judged by the dev standard. Leadership simply needs to stick to their guns on the 6 month objective.  Consistently sticking to these goals are key to them being achieved. You should not call it a failure or a success until you’ve seen movements in the new “muscle memory” of the team.

If it ain’t broke…

Many will tell you the old model wasn’t broken so why are we changing it. Rarely is it the case that I have found their assertion true. More often than not, the reason these folks are proclaiming it is not broken is because they are not measuring the right goals. The name of the game in today’s software world is speed and the single most precious resource you have: calendar time. We no longer can afford to find huge design flaws a month before we ship and we certainly can’t afford the month long test pass at the end.   These are common consequences of the functional model and optimizing for discipline strengths.   They are harmful to today’s business strategies.   They are too slow.

 

Optional ingredients

To close, I will list out some challenges I have encountered alongside recommendations

Lack of training – training is key for success. We’ve already given teams their own in-house experts, but this may not be enough.   Listen carefully to the struggles of the team and bring in outside help in necessary.   Looking at external coding bootcamps for your former testers as well as design patterns, algorithms, etc. Treat your high ranking individuals as the leaders they should be and get them presenting brownbags and setting the example of the transformative outcomes you are expecting.

Specialist vs. Generalist battle – many devs will proclaim loadly and proudly that they are specialists and they don’t want to be generalists.   Much of this comes from how they got good review scores in the past (focused on owning deep complex code).   Respect their strengths, but be firm on expecting them to broaden.   At the end of the day, you don’t want either specialists or generalists, you want people who can go deep when it matters but you want those same folks to be able fluidly move to higher ROI tasks. This creates stronger adaptability – a key business imperative.

Shared assets and true specialities – Some topic of concern such as security, performance, and government compliance are deep topics by their nature.   It is very unreasonable to expect one team to be able to master these AND do development AND do testing. Consider funding a separate team to own this for all. Their goal is to create those assets that make it easier for the regular engineering teams to just “do the right thing” automatically.   Test Harnesses and Bug databases and other internal “products” should be owned by a single team with a strong sense of customer focus.   Their job is to accelerate the success of the product engineering teams and prevent duplication of effort.   Customer focus is key here.    Too often I see this teams build what they want to build and senior leadership force others to use it. Far too often, I see that these systems fail to achieve the needs of the engineering team due to this.   They need to be built to achieve goals, not to look good for the veep.

Area Owners – I recommend heavily to ban individuals from owning areas. I encourage centers of gravity and pursue areas of passion, but make it clear they these areas are owned by the team and anyone can and will work on it.   This helps to remove the reliance on specialists to solve problems and removes a key bottleneck from your system flow.   Lastly, it really amps up knowledge sharing on the team.   If multiple people are familiar with an area of the product that’s broken, then the odds of a fast and righteous fix being implemented is much, much higher than otherwise.

Command and Control – Leadership is the single easiest way to break or stall this transformation. Get rid of command and control methods.   As mentioned above, they don’t align with knowledge work anyways.   Tell your people the outcomes you want and the criteria for what defines it’s completeness.   They are professionals they will be able to figure out the next action to take as well as the subsequent ones.

 

 

I hope this is helpful.   Please feel free to post any comments and questions below.

 

Happy New Year!

 

Advertisements

Systems Thinking: Why I am done with agility as a goal

    Recently, I was writing up a presentation where I was going to state that the New Tester’s job definition was to “accelerate business agility”. One of my peers looked at it and remarked “Isn’t that sort of redundant?”. After some discussion, it became clear that “agility” did not have a clear well-understood definition.

To be clear, I am MOST definitely not done with Agile methods, but as best as I will be able to, I am done with using the word ‘agility’ to describe it. If one looks this up in your favorite dictionary, you will find it described as “moving quickly”. While moving quickly is certainly a valuable goal, it is pitifully insufficient in the modern day software world and if not tempered correctly, can actually lead to more pain than what you may have started with. When I now give talks on Agile, my usual starting point is to first clarify that Agile is NOT about moving quickly, so much as it is about changing direction quickly. So in a nutshell, Agile is not about agility. One problem I am trying to unwind is the dominance of strong-willed, high paid folks proclaiming that Agility is the goal and quite simply, they do not know what they are talking about as evidenced by the typical lack of details explaining behavior and/or success changes their team should be making. This causes their reports to “follow” this guidance, but left to their own devices to make it up. A few clever folks actually study it and realize that shifting to Agile is quite a paradigm shift to succeed and hard to do. This can be a slow process, which seems to contradict the goal of “moving quickly”, so gets abandoned for a faster version of Waterfall or similar dysfunctional hybrid. There’s a common phrase in MBA classes, “Pick 2: Cheap, fast, or good”. This implies a singular focus on fast is likely to deliver crap and at a high cost.

One quick test to see if your leader understands: Ask how much are we going to invest in real-time learning. Then observe how those words align with actions. Moving fast without learning along the way is definitely NOT Agile, but more importantly, it is wrought with peril.

Many of my recent blog posts are on the topic of leadership lately. If you find yourself in such a role and are trying to lead a team towards Agile, my guidance is that you think carefully about the goals and behaviors you are expecting and use the word that describes it better. If you don’t know what you want, then get trained. In my experience, using Agile methods is very painful if the team leadership does not know what, why, and how to use them.

Consider these word alternatives:

  • Nimble: quick to understand, think, devise, etc.
  • Dexterity: the ability to move skillfully
  • Adaptability: the ability to change (or be changed) to fit changed circumstances

These ALL make more sense to me than “moving quickly”, but adaptability is what fits the bill the best in my mind.

    In my last post, I focused on one aspect of the shift paradigm shift happening in the world of test towards the goal of improving adaptability. I have mentioned before my passion (and the primary reason I write this blog) is about Quality. However, to make a business well-functioning in this modern age, a singular focus on changing the paradigm on quality is not sufficient. As Test makes its shift, other pieces of the system must take up the slack. For example, a very common situation happening is that Test simply stops testing in favor of higher value activities. Dev then needs to take up that slack. If they don’t (and most likely they won’t initially), then they will ship bugs to customer and then depending of customer impact, cause chaos as dev attempts to push testing back. We need to consider the whole system, not just one part of it.

A couple of months ago, I was asked to begin thinking through the next phase of shifting the org towards better adaptability. Almost immediately, I rattled off the following list of paradigm shifts that need to be done to the system as a whole.

 

From

To

Prevention

Reaction

QoS

QoE

Spider teams

Starfish teams

Correctness

Quality (value)

Intuition

Truth

NIH is bad

NIH is Awesome

Large batch

Small Batch

Schedule

Throughput

Vanity

Action

Hero

Team

Green is good

Red is good

Yearly

Daily

Absolutes

Probabilities

Ownership

Shared Accountability

 

Hopefully, you can see that moving quickly is certainly a part of this, but more importantly, this list shows a series of changes needed for focus, sharing, understanding the current environment, and learning…

Recently, I have come upon some material from Dr. Amjad Umar (currently, a senior strategist at the UN and one of my favorite professors) where he argues that companies should be plan-fully considering the overall “smartness” of their systems. He states that technologies alone cannot improve smartness. But you can improve it by starting with the right combination of changes to your existing People, Processes, and Technology. Smartness, by the way, is analogous to Adaptability.

I have taken his concept and broadened it to something I call “Umar’s Smartness Cube”. I think it nicely describes at a high level what needs to be considered when one makes System changes. The goal of the whole cube, of course, is to improve Business Value.

How to use this to improve your system:

  1. First determine and objectively measure the goal you are trying to achieve.
  2. Consider the smartness cube and enumerate opportunities to improve the above goal.
  3. Consider tradeoffs between other elements to achieve goals better. For example, maybe we don’t need the world’s best technical widget if we just change the process for using what we have to reduce the training burden.
  4. Prioritize these opportunities (I like to use (BizValue+TimeCriticality)/Cost)
  5. Get them in a backlog that acts like a priority queue and start executing.

 

This, of course, is over-simplified, but hopefully, sets you in an actionable direction for “accelerating the adaptability of your Business (system)”.

As thinking-in-progress, any feedback is appreciated.

AB Testing Podcast – Episode 2 – Leading change

The Angry Weasel and I have put together another podcast. In this episode, we talk about problems we see in leading change towards a better direction. We cover some changes we face, change management, and reasons change fails. We talk about the importance of “why” and leverage the “Waterfall vs. Agile” religious war, as an example.

We managed to keep our “shinythingitis” to a minimum of slightly less than 70% of the time. 🙂 Enjoy!

Want to subscribe to the podcast?
RSS feed
iTunes

There’s no Hope in Test

I bet that got your attention.

No,  I am not echoing James Whittaker’s and others’ thoughts decrying the role of the Test Generalist.  They may or may not be right.   I think it makes sense, and only time will tell for sure.

I did not say:

There’s no Hope FOR Test.

I said:

There’s no Hope IN Test.

While PM/Dev/Business Team may hope it works right, or ships on time, or sells enough, Hope is *not* a valid Testing Technique.  Hope is another way of saying “Assume”.

Neither has a place in Test.

Testing is a analytical role and requires strong critical thinking skills.  In a nut shell, in order to mitigate risk to the business, Testers need to be able to outthink their peers.  We need to find the right answers.  These require the right questions.

Which requires Thought!

Testers do not Hope.  They prove or disprove.

We are paid to find invalid assumptions, not make them ourselves.

—–

Originally, this was the intent of the speech.  Prove it works, don’t hope it does.  However, I realized there is another important meaning to consider.

I recently was in a meeting with Eric Brechner and Test Leaders from around Microsoft. We were chatting about his blog entry, “Test don’t get No Respect“.

He talks of a reality Testers can be in.   Misunderstood.  Paid less than their counterparts. Tasked to do brute force work.

I tire of the Test Zombies and the path of Intellectual Laziness they follow.    These are the folks who view Testing as both an Ends and a Means.  They automate because they automate.  They do what Dev/PM tells them, because that’s what they are used to.  They complain about Dev/PM doing this, because they are used to that as well.  They forgotten their role and value to the process.  If this is what we are purging from the software engineering process, so be it.  Good Riddance.

They aren’t thinking in a fundamentally thoughtful role.   This can even progress to Career-ending potential.   These folks HOPE that Test isn’t dead.  It would affect their livelihood, but are doing nothing about it.  ‘cept Hoping.

Folks, true testing isn’t dead.  As long as there is software, Testing will be needed.    But you need to make sure you are proving your value to the business.  New techniques will be invented.  New requirements will form.  Business plans will be disrupted.  These will require new approaches.    This may change Testing forevermore.

After a quick set of searches on Indeed.com, I gathered these stats:

software testing –   6,304 new jobs (edited from 111,047 (see comments))
java – 84,554 new jobs
C++ – 41,106 new jobs
c# – 40,837 new jobs
cobol – 3,031 new jobs

I look around nowadays and wonder if this what it was like when mainframe developers were told they were dying.   As you can see, COBOL is not dead, but it’s demand is woefully diminished.

But I wonder what happened to those folks who hoped their skill would be valued forever.

What is the ROI of Test?

For me, this is a momentous occasion. After being quoted a couple of times in other people’s blogs, I’m finally breaking out on my own. I’m very excited to see where this path will go.

I struggled a bit thinking through what should be the topic of my first post. I finally landed on a topic that got brought up over lunch with a friend of mine in the industry. It was about 4 or 5 years ago. At that time, my boss, who was not in test, was giving open positions more and more to the Development team. I felt that the test schedule, which was already at risk, was going to be impossible to achieve if at least some of those positions weren’t QA. I explained my plight and asked my companion “Given N testers on a team, what is the ROI for the N+1th tester?”   He responded with more than a little cynicism, “Brent, I don’t know. But tell your boss if he wants to see QA’s value, get rid of your testers”.  I was one with his frustration. Those of us in the business know our importance but it’s hard to quantify.

Delivery of quality is a subjective thing and how Testers should do it is just as subjective. It’s a topic for another day, but I’ve begun to collect definitions of a Tester’s Job and, not surprisingly, it’s already amazingly diverse. There are several ways to produce software and make it shippable.  Due to this, it seems a different perspective on the role of QA in the production of software is created for each.

Consider the following for your project:

  1. Customer Resiliency:   How risk adverse is your customer? Will customers be ok with bugs?
  2. Business Resiliency:   How risk adverse is your business team? Is there a window of opportunity  that must be perfect? Do we have to get it right the first time?
  3. Speed: How fast does your team release?  Is it twice a  month or thrice a decade?
  4. Complexity: This is a  measure of the number of components that must interact to make a solid system.
  5. Complication: This is a measure of the design and implementation of the code used to make up a component
  6. Test Confidence: This is a measure of your test collateral and its ability to cover the product.
  7. Test Speed: How long does it take us to use the test collateral to determine state of the product?
  8. Development Skill: What is level of mastery of the craft that exists in your team as a whole?
  9. Testing Skill: What is level of mastery of the craft that exists in your team as a whole?
  10. Ignorance (props to Armour): Are you measuring the above? Do you know what you need to do? Do you know which one is the bottleneck in your team?
  11. Irrational Optimism: How hopeful is the team that the plan is correct and/or on track?

It’s very similar to taking a car trip.   We need to know where we are, where we are going, how we are going to get there, what to pack, etc.  If we make the wrong choices, like travel someplace we haven’t been without a map, or packing only winter clothes for a trip to someplace sunny, or taking the wrong kind of car for the terrain, we could end up paying a lot more than expected to get to our destination.

Your team has all of these chararcteristics in varying degrees.  Depending on where your team stands on each of them has a potential of fundamentally changing the team’s approach to software development.   An obvious example:  if you ship every three years,
you are likely to take a lot more of a preventative approach to bugs and “find ‘em before the customer does”.    However, if you ship every 3 weeks, you will likely focus on taking smart risks with your testing and get better at reducing the time it takes to react to a failure.

When I think of Test, I am not referring to Testers or a Test team per se.   I am referring to what portion of the payroll budget is applied to Testing activities.  While it is easy to chat about things like Dev to Test ratios, figuring out how much of that money should be spent testing vs. developing depends entirely on the culture of your team.  If your team registers high in all of the negative things, you may need to spend *a lot* in order to mitigate the risk and save your business.

Accordingly, Testing is a form of Software Business Insurance.   The more coverage you have the less at-risk you are.  So there’s really only one answer to my question.   It Depends!   Coverage mitigates risk.   If you don’t feel at-risk, then, until proven otherwise, coverage will not be valuable to you.

One day, perhaps, someone from the Actuarial world will join us in Testing and show us how to determine this more precisely, or maybe outsourced Testing companies will model their fee structure after the insurance industry.   Until then, in order to hire the next tester, you should be prepared to articulate the risk you see in your project and the value your business will gain in comparison to the options.  Back to the car trip metaphor, if you know the person driving the car isn’t a great driver, you could mitigate the risk and buy supplemental insurance or you could send the person to Better Driving School.

I recently met up with my friend. His group has recently undergone a major reorg and QA got trimmed back in a big way. It seems the group’s Head looked at the cost of the QA team and asked “What do these guys do?”  Worry not for my friend, though.   He’s taken his entire team and is going to Development.   His team has strong testing and development skills and many of the other characteristics above are positives for him.  I doubt he’ll need much supplemental coverage.