Testing Practices and Processes Defined

One key aspect of any project is the testing that is performed during Development by the delivery team and during Testing by the delivery team and the customer User Acceptance Testing (UAT) team. Testing, of course, is that last critical step to ensure both the requirements are met and the system is a working, functioning system and is ready for deployment.

Therefore, I thought it appropriate to present at this time some testing documentation I’ve been carrying around for quite some time on the subject. This information is from the Carnegie Mellon Software Engineering Institute (SEI) and describes the various types of testing that are commonly carried out during the development process up to deployment.

The following is an overview of each testing approach.

Design model validation: Each phase in the development process that creates a model of the product or some portion of it should include testing activities that verify the syntax of the model and validate it against the required system. The test can serve as the exit criteria for that phase. We use “model” in a broad sense, to refer to non-software assets that represent a product for the purpose of either making predictions about the product implementation or prescribing constraints for other assets. A business case for a product line is a model; it predicts how profitable the product line will be. Software designs are models; they predict behavior and also impose constraints on implementations.

Unit testing: Testing for implementation defects begins with the most basic unit of code development. This unit may be a function, class, or component. This kind of testing occurs during coding; therefore, the intention is to direct the testing search to those portions of the code that are most likely to contain faults–complex control structures, for example. As each unit is constructed, it is tested to ensure that it (1) does everything that its specification claims and (2) does not do anything it should not. A test case associates a set of input values with the result that should be produced by a correctly functioning system. The functional testing strategy uses the specification of the unit to determine which inputs to use in the testing. This strategy provides evidence that the unit does everything it is supposed to. A second strategy, termed structural testing, selects test inputs on the basis of the structure of the code that implements the functionality of the unit. This strategy provides evidence that the unit does not do anything it is not supposed to.

Subsystem integration testing: The integration of basic units, even those that have been adequately unit tested, may produce failures resulting from the interaction of the units. Timing discrepancies and type/subtype relationships can be the source of these errors. The tests are constructed from the use cases used to represent the full product’s requirements. The integration test plan should describe tests that have been systematically selected from the interactions among the units being integrated. Protocol descriptions between pairs of units or flows through sets of units that implement a specific pattern of behavior can be used to select the test cases. Test cases should include instances in which the error-handling capability of the units is evaluated, such as when one unit throws an exception that should be caught by another unit.

System integration testing: When some critical mass of subsystems has been fully developed and tested, the focus shifts to representative tests of the completed application as a whole to determine whether a product does what it is supposed to do. These representative tests are selected to cover the complete specification for the portion of functionality that has been produced. The amount of testing a specific function receives is based either on its frequency of use (operational profiles) or on the criticality of the function (risk-based testing). Special forms of system testing include load testing (to determine if the software can handle the anticipated amount of work), stress testing (to determine if the software can handle an unanticipated amount of work), and performance testing (to determine if the software can handle the anticipated amount of work in the required time).

In addition to testing as the exit criteria for process phases, the next five types of tests described are applied to verify certain product properties.

Regression testing: Regression testing is used to ascertain that the software under test that exhibited the expected behavior prior to a change continues to exhibit that behavior after the change. Regression tests are constructed, and periodically applied, to determine whether the software under test remains correct and consistent over time. Regression testing is triggered by changes that affect a predefined scope of assets or that affect certain critical assets. The actual test cases used in regression testing are no different from any other test cases. The regression test suite is a sample of the functional tests from the original test suites administered prior to any changes.

Conformance testing: Conformance testing determines whether the software under test can be used in a specific role in an application. The conformance test set should cover all the required interactions between all the components that will participate in the application.

Acceptance testing: To validate the claims of the manufacturer or provider, the consumer performs acceptance testing. The acceptance test is more realistic than the system test, since the application being tested is sited in the consumer’s actual environment.

Deployment testing: Deployment testing is conducted by the development organization prior to releasing the software to customers for acceptance testing. Where acceptance testing focuses on the functionality of the delivered product, deployment testing covers all the unique system configurations on which the product is to be deployed. This testing focuses on the interaction between the product and platform-specific libraries, device drivers, and operating systems. During the deployment testing phase, the application’s ability to deploy or install itself is also tested.

Reliability models: Testing is used to estimate the reliability of a software component or system; however, establishing the reliability of a piece of software through testing is a costly process. The test cases are selected based on the expected frequency of use of each product feature.

When Project Management is Fun

Ok, “fun” may be a stretch…but not really. When do we dislike our jobs the most? When they are dead ends. When we’re micro-managed. When we don’t feel leadership is leading us. When we feel that we aren’t given the tools or authority to succeed.

I’ve had one or more positions where all of this happened….maybe not all in one job (though I can think of one specific job that encompassed most of these). I’m sure we all can think of a job where we were pushed down and we were unhappy. We tend to not be at our most productive selves in these types of jobs and under these types of conditions.

I’ve always found that I am at my most productive when the work is fun and when I have a great deal of control over that work. The same goes for the projects that I lead. If I have executive management that has a great deal of confidence in my skills and basically lets me run the show with the understanding I’ll shout out when there are needs as well as keeping them up-to-date on status, then I usually truly enjoy.

In fact, I find that managing projects with the some or all of the following 6 characteristics end up being the most “fun” for me to manage:

• Job Autonomy – This basically means being allowed a great degree of freedom and discretion in a job. When I’m allowed to lead a project and do the job well, make the right choices, and decisions and lead the customer without an excess of executive involvement, work is more fun, more productive and I can form a much better relationship with the customer.

• Focused Resources – Having skilled, focused resources that you know you can count on to get the job done certainly makes managing the project with confidence a lot easier. Knowing those resources are focused on your project tasks when they should be gives you the confidence to make appropriate promises to the customer and have your word stick.

• No Micro-Managing – Somewhat similar to Job Autonomy. I’m at my worst when I’m micro-managed and second-guessed. That’s for managers with nothing better to do. I once was leading a team onsite with a customer and had 5 other projects I was also leading. I was onsite with this team for 2 weeks – meaning I had 10 other weekly status meetings during those two weeks, 10 other status reports to delivery, 10 other project schedules to get out, etc. Unfortunately, I also had a senior manager there doing nothing more than walking around trying to make sure my team was focused on the onsite customer – I think because HE was being micro managed as well. It made it difficult to serve my other customers for this company with this individual constantly trying to get me back to focusing on the onsite customer – even though the technical resources on the project we engaged in the tasks we were actually there to perform (I was mainly there to just oversee that…which doesn’t require 24/7 oversight – and I could certainly afford 2-3 hours per day to devote to other projects).

• Requires Some Innovation – Leading a project that isn’t just a slam-dunk is more challenging to me and therefore more fun. One that requires more out of the box thinking definitely gets my attention – and usually challenges my team even more and keeps them excited and engaged.

• Cool Technology – We’ve all been here…any project that requires some cool technology makes the work more fun, right? What’s more fun…a project that requires an out of the box CRM system to be tested and implemented or a project that calls for a slot machine load testing simulator to be built from scratch, heavily tested, and implemented?

• Some Technical Hands-On – Coming from a technical background, I appreciate the chance to get my feet wet sometimes rather than just ‘manage’ resources and project deliverables. Even if it’s just cleansing and manipulating data and loading it into the final solution. Anything technically hands-on that changes the pace for me adds to the fun of managing a successful project.

Summary

There are projects out there that are cool and fun and cutting edge and there are others that are as dull as burnt toast. The best we can hope for is that we get a decent mix of both. We need some easy ones once in awhile – especially if our load is usually heavy. But the really challenging projects involving some and cool technology can help keep us fresh and having fun in our Project Management positions that, let’s face it, can be somewhat dry at times….at least that what my wife always imagines.

Startup Teams with HP on Cloud-Based Testing

I received my latest copy of InformationWeek recently and found this article interesting – especially since all discussions these days seem to center on either Cloud Computing or Agile Development.

Skytap is a startup that tabs themselves as the leading provider of cloud-based virtual labs that deliver 100% self-service provisioning of complex IT environments without any architectural changes. Cloud computing is poised to become the defining technology of the 21^st century and Skytap’s goal is to make serving up virtual machines over the internet as ubiquitous as delivering html to a browser. They are working to maximize efficiencies, minimize costs, eliminate unnecessary hardware, outsourcing, eco-efficient computing, and doing more with less.

I’ve worked many very large-scale government contracts where testing was a massive onsite effort involving additional hardware, software, and bodies in a compressed and stressful timeframe. Cloud-based testing would have made those experiences much more sane. And it was solely my responsibility at the time to make those tests happen and help ensure their success.

Likewise, my time in the gaming industry involved load testing for slot data management software. It’s necessary to test slot machines against large loads of usage – the last thing a very large casino gaming entity wants to happen is for their slot system to crash on a Saturday night due to heavy customer usage!

Without further ado, here is the article written by Charles Babcock for InformationWeek…

“Startup Skytap has cut a couple of powerful alliances for it’s cloud computing services, most recently joining forces with Hewlett-Packard to make it easier for companies to stress-test software against thousands of simulated end users without taxing their won data centers.

Skytap – named one of the InformationWeek Startup 50 in April, shortly after getting $7 million in venture funding – offers a Virtual Lab where developers try out applications by building test environments from its library of operating systems, databases, and middleware. Skytap already partners with Microsoft to enable Visual Studio Team System testing.

Skytap is providing HP’s LoadRunner testing tool to build test scenarios that push an application’s limits. The tests can be set up, managed, and torn down through HP’s Quality Center, which uses Skytap’s cloud computing resources to execute the actual test. The tests run as virtual workloads under VMware’s ESX Server.

Cloud computing is seen as a lower-cost way to offload workload spikes from the data center, and testing and quality assurance are likely prospects. Other cloud computing services such as Amazon Elastic Compute Cloud also can be used for testing. Companies pay Skytap from $1,000 to $10,000 a month for cloud-based testing.”