MANAGEMENT VISIONS: January 30, 2006

"TESTING 1, 2, 3"

Not long ago you heard me whine about how poorly commercial software is being tested by the major vendors. Normally, shoddy workmanship results in clients distrusting the products and vendors. Surprisingly, this is not the case in the IT Industry. Instead of holding vendors accountable by their customers, sloppy workmanship has become an acceptable form of behavior in this field.

Let me share with you the standard approach to testing as used by today's software companies:

1. Announce a general release of the product to the press.
2. Write a section of code and test/debug it.
3. Assemble the code and issue it to customers in "Beta" form.
4. Let the customers play with it and report problems as they occur.
5. From this, the vendor develops and prioritizes a punchlist of corrections.
6. The punchlist is addressed with the product delivery date in mind.
7. The first release of the product is issued with much fanfare (even though the punchlist has not been fully corrected).
8. More errors are reported by the customer base and added to the punchlist.
9. The second release of the product is issued as a fixpak. This release, in reality, represents the finished product.

The intent of beta code is more for public relations and publicity as opposed to delivering a viable product offering. Instead of thoroughly testing their products themselves, vendors let their customers do it for them. It shouldn't be this way.

I don't have a problem with the beta-test concept as such, so long as the vendor has legitimately tested the product themselves prior to issuing the release. They're not. Is it that they are incapable of testing it themselves? Maybe. Most likely though, it is cheaper to have outsiders test it for them. However, beta testing represents a disorganized test of the product. Something much more structured should be done prior to this by the vendor.

As we have discussed on more than one occasion, a system is a product that can be engineered and manufactured like any other product. Let's think about the design of a product, such as an automobile. The overall product is first broken down into a series of assemblies, such as the chassis, body, engine, drive train, and so on. Each assembly is then "exploded" (or sub-divided) into subassemblies, and components.

After the product has been decomposed top-down into its assemblies, parallel teams of designers concurrently refine their portion of the product with little interaction between assembly design teams.

Parts such as nuts and bolts are carefully identified, classified, and reused not only within the product but with other products. One classic example of this is years ago when General Motors was able to slip a Pontiac Engine into an Oldsmobile. This demonstrated the power of standardization and the reusability of parts.

After a product has been fully designed, the product assembly line "implodes" the product (bottom-up) with components making up subassemblies, and then the subassemblies into assemblies, and then assemblies into the final product. Again, think about the assembly line for automobiles where the pieces and parts of the car are built in smaller increments until the final assemblies are put together and the key is turned and the car is driven off the assembly line.

This approach to product design is obviously an old one, but it can be easily adapted to the design and development of an information system.

Under "PRIDE"-ISEM an information system is a 4-level hierarchical structure consisting of the system (representing the product) which is decomposed into sub-systems representing business processes. Each sub-system is divided into procedures (both administrative and computer). Administrative procedures are broken into operational steps or manual tasks. Computer procedures are broken into programs.

As in the product analogy, the system is designed top-down into sub-systems. Once this has been done, parallel teams of designers can work concurrently developing separate sub-systems.

When the final pieces and parts have been designed, they are developed and tested "bottom-up." For example, unit tests are performed on programs, a string test of the programs within a computer procedure is then performed; this is followed by a test of all of the procedures in each sub-system, and finally parallel testing of all of the sub-systems in the system is performed. If successful, the key is turned and the car is driven off of the assembly line.

The so-called "parts" of the system-product are the data components representing data elements, records, and files. Like product parts, the data resources should be carefully identified, classified, and reused not only within the system, but with other systems, thereby promoting integration.

The point is, like products, systems are designed by explosion (top-down if you will) and tested and implemented by implosion (or bottom-up).

The early phases of "PRIDE"-ISEM are primarily used to design the system into refined components. They are also used to specify the testing criteria for the later phases; to illustrate.

PHASE 2 - SYSTEM DESIGN - The Test Plan at this level will be used later in Phase 8 and should cover how parallel testing of the sub-systems should be performed. Key file maintenance sub-systems should be tested and installed prior to simple "read-only" sub-systems.

PHASE 3 - SUB-SYSTEM DESIGN - The Test Plan at this level will be used later in Phase 7 and should address human/machine interaction. Since inputs and outputs should be fully defined by this phase, the Test Plan should include:

- Design standards to be observed for Graphical User Interfaces and report layouts. This includes the use of Help text.
- Default values and validation rules for collecting data.
- Ease of use (intuitiveness).
- The processing of optional parameters.
- Verify the overall processing logic.
- Benchmark tests.

PHASE 4-II - SOFTWARE ENGINEERING - The Test Plan at this level will be used later in Phase 6 and should check operational dependencies between programs and stress testing. Input/File/Output layouts should be verified as well as volume testing. The assemblage of test data should be included.

BOTTOM-UP TESTING

The remaining phases in "PRIDE"-ISEM are used to test the system from the bottom-up.

PHASE 5 - SOFTWARE MANUFACTURING - Following the production of the executable program, the programmer assembles pertinent test data to test and debug the individual program. To do so, the programmer consults the Test Plan, Program Specifications, and pertinent Software Structure Diagrams resulting from Phase 4-II. This becomes the "roadmap" for testing the program.

There are a variety of testing and debugging tools available on the market. Use of such tools should be encouraged.

When the program has been tested to the satisfaction of the programmer, the results are reviewed with the computer procedure designer and Quality Assurance. This review assures that all standards have been followed, and the program has been tested to conform to specifications. The test data is saved for later reference and the test results are filed for future reference.

PHASE 6 - SOFTWARE TESTING - The purpose of this phase is to test and correct errors in the overall computer procedure. This is sometimes called a "string test," "job stream test," or "stress test." Whereas the "unit test" performed in Phase 5 was concerned with data format, the emphasis in Phase 6 is on volume testing.

It is the objective of the Computer Procedure Designer to assure that the procedure meets the sub-system specifications. In addition to testing for correct processing logic, they also check the operational considerations of timing, re-start, etc.

Following testing, the results are reviewed with Quality Assurance and the Sub-System Designer. This assures compliance with all standards and that the Sub-System requirements are met. When the testing is satisfactorily accomplished, the test data is filed for future reference.

PHASE 7 - SUB-SYSTEM TEST - Testing in Phase 7 is performed in a similar manner as in Phase 6, except at a higher level. Whereas Phase 6 "Software Testing," is performed by Software Engineering for all of the programs within the computer procedure, Phase 7 is performed by Systems Engineering for all of the procedures in the sub-system, both administrative and computer.

During Computer Procedure Design, Software Engineering prepared the computer procedure that will be used by DP Operations. In order to test this procedure, it is recommended that DP Operations actually perform the tests under Software Engineering's direction. In this way, the operating procedures are tested along with the programs used to implement the procedure. Often the operating people will point out areas for procedure improvement.

During Phase 3, Systems Engineering specified a testing criteria for the overall sub-system. This included the validation of input/output processing, along with anticipated transaction volume for the sub-system. Phase 7, therefore, is used to verify the test criteria and benchmarks.

Following Phase 7, the sub-system is essentially ready for operation. However, considerable care should be exercised to avoid a premature installation of the sub-system for day-to-day operation. This situation could cause panic conditions if other parts of the system are not ready. This is why Phase 8 is used to release sub-systems.

PHASE 8 - SYSTEM OPERATION - A final test of the system is performed where the various sub-systems are tested in parallel. This is similar to the testing in Phases 6 and 7, but at the highest level. It is not uncommon to invite user personnel and DP operations to participate in the test. The Administrative Procedure Manuals and Computer Run Books, as created in the various Phases 4-I's and 4-II's, are used as part of the formal walk-through. The test data as created in previous phases is assembled and used as the basis for the test. Where available, actual data should be tested to simulate actual system situations. If any errors are discovered, they are corrected immediately and testing is resumed. Changes to the sub-systems may also be proposed; however, these should be considered carefully. In some instances, it may be more appropriate to suspend any modifications or improvements temporarily until use of the new system has settled into routine operation.

CONCLUSION

Whether you use "PRIDE"-ISEM or not, the concept of defining refined levels of test plans "top-down", and performing corresponding tests "bottom-up" is a simple and effective approach. It offers a structure that is easy to manage and provides for a rigorous test of the system. Test data generators and debugging aids are useful and should be encouraged, but simple organization and a methodical approach to testing can have a more dramatic effect. Following this, if you still want to perform a beta test, you have my blessing.

Always remember this, most beta test programs are designed more for promoting the product as opposed to actually testing it. Think about it; its a "no-brainer" for the vendor. Customers begin to talk about the product while assuming the testing costs of the vendor. Even if something were to go wrong (and frequently does), the vendor is not liable and simply shrugs off the snafus due to beta code.

I used to beta-test software products for vendors, but I no longer have the time nor inclination to do the manufacturer's work for them anymore. Further, I no longer rush out to buy the latest release of "any" software product; I have been burned too many times by the vendors. As far as I'm concerned, the software vendors really need to clean up their act when it comes to testing. If they really want us to test their products for them, let us know where we should send the bill.

OUR BRYCE'S LAW OF THE WEEK therefore is...
"Most beta test programs are designed more for promoting the product as opposed to actually testing it."

IN OUR "DOWN THE ROAD" SECTION

A special conference entitled "Cincinnati Technology: the Next Generation" will be held on Saturday, February 18th at the Netherlands Hilton in downtown Cincinnati, OH. Among the speakers will be yours truly to discuss the "PRIDE" approach to Information Resource Management. For information or to register, contact the First Rule Group at 513/375-3291.

On March 6th-8th, the Gartner Business Intelligence Summit 2006 will be held at the Hyatt Regency in Chicago, IL. For info, contact Gartner at 203/316-6757

The 17th International Conference of the Information Resource Management Association will be held May 21st-24th at the Wyndham Hotel in Washington D.C. For information, call IRMA headquarters in PA at 717/533-8879

If you have got an upcoming IRM related event you want mentioned, please e-mail the date, time and location of the event to timb001@attglobal.net

MY "PET PEEVE OF THE WEEK" IS "AUTOMOTIVE REPAIRS"

Recently, I had to replace the battery in my GMC Envoy. I haven't had to replace a car battery in quite some time. Back in the 1970's when I worked at a gas station on my summer vacation from school it used to take me about 5-10 minutes to replace a battery. But this time it took me at least an hour to undo the safety box and bar around the battery and undo the cables which were strategically placed in an awkward position to make it as difficult as possible to use a wrench to unscrew the posts. I just couldn't get over how the car manufacturers have complicated such a simple and mundane task as changing a battery. Small wonder the dealerships are charging us excessively for repair work.

Another thing that bothers me is how the major automotive repair shops try to rip us off for excessive work. For example, I took my other car into a repair shop recently to have a headlight replaced. After the service shop looked it over, they called me with a long list of other items they wanted to replace, including shocks and tires. They were embarrassed when I told them I had both the shocks and the tires replaced just last year. As a consumer, I feel like I am being harvested every time I bring a car into these shops. Not only do they want to fix your immediate problem, but they go out of their way to find new ones. What bothers me is that many people fall for this old gag. What these service shops don't realize is that they are creating more distrust and suspicion than good service. It has taken me a long time to find a good mechanic that I trust and believe me, I have seen a lot of mechanics, most looking for nothing more than a quick buck. The guy I've got now knows his stuff and charges a fair price. And believe me, he is worth his weight in gold.

Such is my Pet Peeve of the Week.

AND FINALLY...

I received an e-mail from a Martin Dimond in Cincinnati who wrote me regarding last week's interview with Mike Snyder.
Martin writes:

"Mike mentioned the types of skills people need in the corporate IT world for Systems Development. Could you elaborate on this?"

Thanks Martin for your note,
Good question. Mike and I have talked about this at length. Basically, we see true systems people as somewhat extroverted in nature and programmers as introverted. A systems person is also more of a generalist who understand business and total systems. The programmer is more of a detailist and in tune with technology. Because the systems person is more related to people, he or she must have good interpersonal communication skills for such things as interviewing, making a presentation, and salesmanship. They also have to have good writing skills in order to develop proposals and record information requirements. They must also have basic business skills such as how to develop a cost/benefit analysis, return on investment, and calculating break even points, or being able to perform work measurement and work simplification. They also have to be able to effectively communicate systems requirements to the programmers. In this capacity, they play the role of translator.

Unfortunately, the schools are not producing such people. Instead, they are producing more and more technicians who have to be debriefed in order to make them productive in the workplace.

Interestingly, some of the best systems people I have met do not have a computer background and come from the fields of music, engineering, and library science; which are all creative fields based on discipline and organization. Also, I have found very few people who are both good at systems as well as programming. Each requires a different set of skills and perspective. Typically, a person is good at one job but not the other.

Again, Thanks for your e-mail. Keep those cards and letters coming.

Folks, don't forget to check out our BRYCE'S CRASH COURSE IN MANAGEMENT which is a free on-line multimedia presentation offering pragmatic advice on how to discharge the duties of a manager, whether it be for a commercial or non-profit enterprise. Frankly, for someone aspiring to be a manager or for a new manager, it will be the best 45 minutes you can invest in yourself. Check it out on the cover of our corporate web page at: http://www.phmainstreet.com/mba

For a complete listing of my essays, see the "PRIDE" Special Subject Bulletins section of our corporate web site.

MBA is an international management consulting firm specializing in Information Resource Management. We offer training, consulting, and writing services in the areas of Enterprise Engineering, Systems Engineering, Data Base Engineering, Project Management, Methodologies and Repositories. For information, call us at 727/786-4567.

Management Visions is a presentation of M. Bryce & Associates, a division of M&JB Investment Company of Palm Harbor, Florida, USA. The program is produced on a weekly basis and updated on Sundays. It is available in versions for RealPlayer, Microsoft Media Player, and MP3 suitable for Podcasting. See our web site for details. You'll find our broadcast listed in several Podcast and Internet Search engines, as well as Apples' iTunes.

If you have any questions or would like to be placed on our e-mailing list to receive notification of future broadcasts, please e-mail it to timb001@phmainstreet.com

We accept MP3 files with your voice for possible inclusion in the broadcast.

Management Visions accepts advertising. For rates, please contact yours truly directly.

Our corporate web page is at:

http://phmainstreet.com/mba/

This is Tim Bryce reporting.

Since 1971: "Software for the finest computer - the Mind."

END