Article Overview
We ruminated on what to write about this column. I thought software product development would be a good topic, but what to write about, I was not so sure. Lately, I have been teaching and speaking on the topic of testing software (embedded) products, but perhaps this was not a topic of interest for project managers. On the other hand, I have seen projects go sideways due to testing, insufficient testing, poor test preparation, or unexpected testing results near the product launch window (end of the project). I was stuck with this quandary, what topic. However, one never knows where one will get inspiration.
Table of content
I watch very little television. I read, and write and occasionally play Planet Side 2 (I’m Super Scorpian1). I do enjoy watching baking shows and recently discovered Baking Impossible on Netflix. This show couples’ engineers with bakers to produce edible works of engineering. The individuals are referred to as Bakineers. The teams get briefs (like a specification) of the expectations of them from a baking and engineering perspective. The teams have a specific amount of time to plan and create their idea that is to fulfill the specification. The time runs out, and 3 judges assess the respective team product against the baking and engineering specification criteria. The judges vote to remove a team from the competition and for the $100,000 prize.
It might seem odd to bring a baking show to a product development discussion. It is not odd but prudent. Watching the beginning of season 1 we see the teams focusing on building and creating their edible engineering works. In the beginning most teams bring their creations to the judges seemingly untested and the developed product does not work when demonstrated before the judges. It is not too many challenges (episodes) into the series, that we see most of the teams learning the importance of test as you build, the consequence of failures it seems.
I found this a bit surprising although, perhaps that is on me. In 30 years of product development, I have noticed that testing the product in parallel with the development of the product is at times not a priority. We think we know all we need to know. There are few instances when this approach works, and for the record, I learned the testing in parallel lesson early in my product development career. I would design the embedded hardware, and write the code for an industrial application. I had written a bit of code and it worked perfectly when I tested it. In a moment of hubris, I went on to write large swaths of code, multiple features, and when I went to exercise the features, nothing worked. Not even the parts that had once worked.
Good judgment comes from experience, and experience comes from bad judgment.
~Rita Mae Brown
The third challenge in season 1, titled Sweet Chain Reactions, was a Rube Goldberg specification. One of the teams had the baker prompt the engineer to test the product. The engineer’s reply, we do not have time for that. The baker’s requested more than one time, for the engineer to test the product. The engineer’s reply was the same, we do not have time for that. At the next demonstration, this team failed, and subsequently voted off by the judges.
There is only one thing more painful than learning from experience, and that is not learning from experience.
~Laurence J. Peter
The idea we have no time for testing is not as rare as it should be, and it is not always engineers that get in the way, but project managers. Testing is essential to product development and to project management. This starts with assumptions we have made at the beginning of the project to define scope, strategy selection, in making estimates of duration and costs, and many project attributes. We should devise methods to refute our assumptions, some sort of test. We say refute, as evidence we perceive as confirming what we believe, is subject to confirmation bias. That is our interpreting the evidence in error, favorable to our existing beliefs. One bit of evidence that refutes, means we are wrong and need to take appropriate action in response. To get this evidence will often require testing, or experimenting. In this way testing should start at the very beginning of the project.
As engineers, we can use models and simulations early in the development process, these tools were not available to the Bakineers. There are many other ways to test, I have seen engineers build parts out of cardboard to determine fit, and to assess interfaces for connectors and pinouts. Anything that helps us learn or understand. We recently wrote a book with Shawn P. Quigley on learning while doing the work.
Project managers, like the baker on that team, must understand the testing needs of the product and project It seems, sadly, that engineers may downplay or perhaps even overlook this testing need. “We do not have time for that” is not a good answer. The fact is that testing should occur throughout the development. Writing specifications? Testers should be reviewing that material for two reasons, first to be prepare equipment for the testing, and more importantly, to provide a constructive, objective review of the specification material. We stress this in our product development discussions and speaking events. To avoid or at least reduce any potential rework, it is best to understand the capability of the things we are working as soon as possible. Anything you think you know should be approached with a certain amount of skepticism. A successful way to address is to ask questions such as:
- What makes you think this?
- How do you know that?
- What can we do to disprove?
- What would we predict if we run a specific experiment?
Every time you think you know something and do not test or experiment to explore, the project is at risk. Choosing to short-change the testing, likely introduces risks. We should have a good idea of what exactly is that risk introduced due to forgoing testing. Historical information can help us assess where things have gone poorly (and well) similar projects, if we have this record, and the projects recorded have some analog to our project.
We should be careful to get out of an experience all the wisdom that is in it -- not like the cat that sits on a hot stove lid. She will never sit down on a hot lid again -- and that is well; but also, she will never sit down on a cold one anymore.
~ Mark Twain
In project management, we often must make decisions on incomplete knowledge. We should be aware of that and act accordingly. Acting accordingly can include finding ways to test and explore the things we must understand to be successful. Testing is an important element to product development and project planning and risk mitigation.
1 Netflix: Series '' Baking Impossible ''