November 20, 2006

"TESTING 1, 2, 3..."

Have you ever noticed how poorly commercial software is being tested by the major vendors these days? I have. Normally, shoddy workmanship results in clients distrusting the products and vendors. Surprisingly, this is not the case in the IT Industry. Instead of holding vendors accountable by their customers, sloppy workmanship has become an acceptable form of behavior in this field.

Let me share with you the standard approach to testing as used by today's software companies:

1. Announce a general release of the product to the press.

2. Write a section of code and test/debug it.

3. Assemble the code and issue it to customers in "Beta" form.

4. Let the customers play with it and report problems as they occur.

5. From this, the vendor develops and prioritizes a punchlist of corrections.

6. The punchlist is addressed with the product delivery date in mind.

7. The first release of the product is issued with much fanfare (even though the punchlist has not been fully corrected).

8. More errors are reported by the customer base and added to the punchlist.

9. The second release of the product is issued as a fixpak. This release, in reality, represents the finished product.

The intent of beta code is more for public relations and publicity as opposed to delivering a viable product offering. Instead of thoroughly testing their products themselves, vendors let their customers do it for them. It shouldn't be this way.

I don't have a problem with the beta-test concept as such, so long as the vendor has legitimately tested the product themselves prior to issuing the release. They're not. Is it that they are incapable of testing it themselves? Maybe. Most likely though, it is cheaper to have outsiders test it for you. However, beta testing represents a disorganized test of the product. Something much more structured should be done prior to this by the vendor.

A SYSTEM IS A PRODUCT - THE "EXPLOSION/IMPLOSION" PROCESS

As we have discussed on more than one occasion in these bulletins, a system is a product that can be engineered and manufactured like any other product. Let's think about the design of a product, such as an automobile. The overall product is first broken down into a series of assemblies, such as the chassis, body, engine, drive train, and so on. Each assembly is then "exploded" (sub-divided) into subassemblies, and components.

After the product has been divided top-down into its assemblies, parallel teams of designers concurrently refine their portion of the product with little interaction between assembly design teams.

Parts such as nuts and bolts are carefully identified, classified, and reused not only within the product but with other products. One classic example of this is years ago when General Motors was able to slip a Pontiac Engine into an Oldsmobile. This demonstrated the power of standardization and reusability of parts.

After a product has been fully designed, the product assembly line "implodes" the product (bottom-up) with components making up subassemblies, and then the subassemblies into assemblies, and then assemblies into the final product. Again, think about the assembly line for automobiles where the pieces and parts of the car are built in smaller increments until the final assemblies are put together and the key is turned and the car is driven off the assembly line.

This approach to product design is obviously an old one, but it can be easily adapted to the design and development of an information system.

Under "PRIDE"-ISEM an information system is a 4-level hierarchical structure consisting of the system (representing the product) which is decomposed into sub-systems representing business processes. Each sub-system is divided into procedures (both administrative and computer). Administrative procedures are broken into operational steps or manual tasks. Computer procedures are broken into programs.

As in the product analogy, the system is designed top-down into sub-systems. Once this has been done, parallel teams of designers can work concurrently developing separate sub-systems.

When the final pieces and parts have been designed, they are developed and tested "bottom-up." For example, unit tests are performed on programs, a string test of the programs within a computer procedure is then performed; this is followed by a test of all of the procedures in each sub-system, and finally parallel testing of all of the sub-systems in the system is performed. If successful, the key is turned and the car is driven off of the assembly line.

The so-called "parts" of the system-product are the data components representing data elements, records, and files. Like product parts, the data resources should be carefully identified, classified, and reused not only within the system, but with other systems, thereby promoting integration.

The point is, like products, systems are designed by explosion (top-down if you will) and tested and implemented by implosion (or bottom-up).

TOP-DOWN TEST PLANS

The early phases of "PRIDE"-ISEM are primarily used to design the system into refined components. They are also used to specify the testing criteria for the later phases; to illustrate.

PHASE 2 - SYSTEM DESIGN - The Test Plan at this level will be used later in Phase 8 and should cover how parallel testing of the sub-systems should be performed. Key file maintenance sub-systems should be tested and installed prior to simple "read-only" sub-systems.

PHASE 3 - SUB-SYSTEM DESIGN - The Test Plan at this level will be used later in Phase 7 and should address human/machine interaction. Since inputs and outputs should be fully defined by this phase, the Test Plan should include:

- Design standards to be observed for Graphical User Interfaces and report layouts. This includes the use of Help text.

- Default values and validation rules for collecting data.

- Ease of use (intuitiveness).

- The processing of optional parameters.

- Verify the overall processing logic.

- Benchmark tests.

PHASE 4-II - SOFTWARE ENGINEERING - The Test Plan at this level will be used later in Phase 6 and should check operational dependencies between programs and stress testing. Input/File/Output layouts should be verified as well as volume testing. The assemblage of test data should be included.

BOTTOM-UP TESTING

The remaining phases in "PRIDE"-ISEM are used to test the system from the bottom-up.

PHASE 5 - SOFTWARE MANUFACTURING - Following the production of the executable program, the programmer assembles pertinent test data to test and debug the individual program. To do so, the programmer consults the Test Plan, Program Specifications, and pertinent Software Structure Diagrams resulting from Phase 4-II. This becomes the "road map" for testing the program.

There are a variety of testing and debugging tools available on the market. Use of such tools should be encouraged.

When the program has been tested to the satisfaction of the programmer, the results are reviewed with the computer procedure designer and Quality Assurance. This review assures that all standards have been followed, and the program has been tested to conform to specifications. The test data is saved for later reference and the test results are filed for future reference.

PHASE 6 - SOFTWARE TESTING - The purpose of this phase is to test and correct errors in the overall computer procedure. This is sometimes called a "string test," "job stream test," or "stress test." Whereas the "unit test" performed in Phase 5 was concerned with data format, the emphasis in Phase 6 is on volume testing.

It is the objective of the Computer Procedure Designer to assure that the procedure meets the sub-system specifications. In addition to testing for correct processing logic, they also check the operational considerations of timing, restart, etc.

Following testing, the results are reviewed with Quality Assurance and the Sub-System Designer. This assures compliance with all standards and that the Sub-System requirements are met. When the testing is satisfactorily accomplished, the test data is filed for future reference.

PHASE 7 - SUB-SYSTEM TEST - Testing in Phase 7 is performed in a similar manner as in Phase 6, except at a higher level. Whereas Phase 6 "Software Testing," is performed by Software Engineering for all of the programs within the computer procedure, Phase 7 is performed by Systems Engineering for all of the procedures in the sub-system, both administrative and computer.

During Computer Procedure Design, Software Engineering prepared the computer procedure that will be used by DP Operations. In order to test this procedure, it is recommended that DP Operations actually perform the tests under Software Engineering's direction. In this way, the operating procedures are tested along with the programs used to implement the procedure. Often the operating people will point out areas for procedure improvement.

During Phase 3, Systems Engineering specified a testing criteria for the overall sub-system. This included the validation of input/output processing, along with anticipated transaction volume for the sub-system. Phase 7, therefore, is used to verify the test criteria and benchmarks.

Following Phase 7, the sub-system is essentially ready for operation. However, considerable care should be exercised to avoid a premature installation of the sub-system for day-to-day operation. This situation could cause panic conditions if other parts of the system are not ready. This is why Phase 8 is used to release sub-systems.

PHASE 8 - SYSTEM OPERATION - A final test of the system is performed where the various sub-systems are tested in parallel. This is similar to the testing in Phases 6 and 7, but at the highest level. It is not uncommon to invite user personnel and DP operations to participate in the test. The Administrative Procedure Manuals and Computer Run Books, as created in the various Phases 4-I's and 4-II's, are used as part of the formal walk-through. The test data as created in previous phases is assembled and used as the basis for the test. Where available, actual data should be tested to simulate actual system situations. If any errors are discovered, they are corrected immediately and testing is resumed. Changes to the sub-systems may also be proposed; however, these should be considered carefully. In some instances, it may be more appropriate to suspend any modifications or improvements temporarily until use of the new system has settled into routine operation.

CONCLUSION

Whether you use our "PRIDE"-ISEM or not, the concept of defining refined levels of test plans "top-down", and performing corresponding tests "bottom-up" is a simple and effective approach. It offers a structure that is easy to manage and provides for a rigorous test of the system. Test data generators and debugging aids are useful and should be encouraged, but simple organization and a methodical approach to testing can have a more dramatic effect. Following this, if you still want to perform a beta test, you have my blessing.

Always remember this, most beta test programs are designed more for promoting the product as opposed to actually testing it. Think about it; its a "no-brainer" for the vendor. Customers begin to talk about the product while assuming the testing costs of the vendor. Even if something were to go wrong (and frequently does), the vendor is not liable and simply shrugs off the snafus due to beta code.

I used to beta-test software products for vendors, but I no longer have the time nor inclination to do the manufacturer's work for them anymore. Further, I no longer rush out to buy the latest release of "any" software product; I have been burned too many times by the vendors. As far as I'm concerned, the software vendors really need to clean up their act when it comes to testing. If they really want us to test their products for them, let us know where we should send the bill.

OUR BRYCE'S LAW OF THE WEEK therefore is... "Most beta test programs are designed more for promoting the product as opposed to actually testing it."

"PRIDE" METHODOLOGIES FOR IRM

Friends, the "PRIDE" Methodologies for Information Resource Management (IRM) is a common sense solution for Enterprise Engineering, Systems Engineering, Data Base Engineering, and Project Management. The methodologies include defined work breakdown structures, deliverables, and review points that promote quality and the production of industrial-strength information systems. Building information resources is a science, not an art form. Our methodologies clearly explain the concepts that govern them, which remarkably, is derived from engineering/manufacturing practices. Now you can get these acclaimed methodologies for free at our corporate web site at: http://www.phmainstreet.com/mba/pride/

IN OUR "DOWN THE ROAD" SECTION

The Association of Management Consulting Firms will be holding their 60th Annual Meeting on December 6th-8th at the Harvard Club in New York City. For information, contact AMCF headquarters in New York at 212/551-7887 or visit their web page at: http://www.amcf.org/

If you have got an upcoming IRM related event you want mentioned, please e-mail the date, time and location of the event to timb001@phmainstreet.com

"BRYCE MANAGEMENT ANALYSIS" SERVICE

We've just introduced a new free service for managers to perform a self-analysis of their style of management, including leadership and corporate culture. Check it out at:

http://www.phmainstreet.com/mba/bma.htm

MBA DAILY PRODUCTIVITY ANALYZER INTRODUCED

Also be sure to check out our new "MBA Daily Productivity Analyzer" which is a free calculator to evaluate a person's personal productivity during the day. It is also available at our corporate web site.

http://www.phmainstreet.com/mba/mbaprod.htm

MY "PET PEEVE OF THE WEEK" IS "HOW DOES THE O.S. AFFECT PRODUCTIVITY?"

I've seen a lot of operating systems over the years. I would have to say my favorite character based system was VMS which was used on the DEC VAX. Hewlett Packard's MPE was a close second. Both were reliable operating systems that ran well for a number of years. On the desktop, my favorite was IBM's OS/2 which was way ahead of its time. It was very reliable and ran smoothly. In fact, I've still got a few OS/2 based machines in the office that have run quietly and uninterrupted for many years.

Then we come to Microsoft Windows. I've watched it grow and evolve over the years, from version 3.x to Win95, 98, NT, 2000, XP, and now Vista is in the offing. As you probably know, I've never been a fan of Microsoft products. Having seen other platforms and knowing how other programs work, I know how primitive MS products can be. Unfortunately, most people don't. Nonetheless, MS remains the standard on the desktop and will be with us for a long time.

I've been using Windows XP for some time now and I like to consider myself a power user of the computer. During the average day, I'll be doing a lot of basic text processing as well as word processing, designing and updating web pages, desktop publishing, creating and updating spreadsheets and data bases, recording or listening to multimedia (these podcasts for example), scanning images and burning CD's, as well as extensively using the Internet, such as e-mail, traveling the web, FTP transfer, and VoIP. Its quite common for me to have many things going on at one time. So much so that people wonder how I keep track of things, but I manage. The last thing I need is an operating system that is going to slow me down.

Having seen other operating systems first hand, I felt a noticeable loss in productivity going into the Windows world. Not only did I find MS apps clunky and awkward to use, I found the operating system unreliable. Even now, I can count on Windows crashing or locking up at least once or twice a week, of course at the most inconvenient time during the day. I run backups regularly to cover myself as I have become very paranoid about all of this. Basically, I feel I am being restrained by Microsoft. And its not just me; I know a lot of people, both friends and business associates who are becoming increasingly frustrated with Windows.

This got me thinking about how Microsoft truly affects productivity in the office these days. If we do not have faith in the operating system, that it might crash and burn, this will affect our work habits. For example, instead of doing our job, we find ourselves doing many more "saves" and backups of important data; that we find ourselves waiting agonizingly for the computer to reboot, or waiting for a program to run while Windows struggles with multitasking and memory management. These little "inconveniences" add up and distract us from focusing on our job. In my small office alone, I estimate that at least 10% of our time is distracted due to the nuances of Microsoft products. Let's assume this 10% is correct and equally applicable to big business. This would mean Microsoft products are having not only an adverse effect on productivity in the corporate world, but it is also having an adverse affect on our economy as well.

I find Microsoft's television ads amusing as they portray their products as "State-of-the-art" and models of efficiency. To me, Microsoft does not improve productivity in the workplace, it is actually an impediment. Maybe they're right when they call it "Windoze" as opposed to "Windows."

Such is my Pet Peeve of the Week.

NEW eBOOK: THE BRYCE IS RIGHT!

Folks, we've just released a new book on management entitled, "The Bryce is Right! Empowering Managers in today's Corporate Culture." This is a frank and candid description of the state of the art in management and includes essays on the problems in management today, along with some pragmatic advice on how to deal with them. Basically, this is a condensed course in management. As such, it is suited for managers, either those aspiring to become a manager or for those who need a refresher course. It will also be of interest to young people entering the work force, and is excellent for college curriculums.

Charles Cole of Lyndhurst, OH, said it is a "Very interesting book. Good work! It reminds me of some of the early works I read by W. Edwards Deming. Too bad the American corporate gurus of his day didn't pay him heed."

And Wolf Hager of Fort Myers, FL, says it is "A very impressive publication which requires careful reading and reminds me somewhat of Peter Drucker."

The price is just $20 plus tax. For more information on our book or to order on-line, see:

http://www.phmainstreet.com/mba/bryce1.htm

We have also just produced a new one-day training program of the same name. For more information on both the eBook and course, please visit our web site at:

http://www.phmainstreet.com/mba/bryce1.htm

While there, look for our new MS PowerPoint presentation describing both the book and the training program.

AND FINALLY...

I received an e-mail from a Bob Carlson in Los Angeles who wrote me regarding last week's essay on "Evaluating Purchased Packages."

Bob writes:

"I listened carefully to your November 13th broadcast and it all sounded good, but what you described sounded like overkill to me."

Thanks Bob for your note,

I guess its a matter of how much money you are willing to risk on a packaged solution. For small PC based apps priced at under $100, you probably won't do it. But for the major applications involving hundreds of thousands of dollars, or perhaps millions, it is hardly overkill; it is time well spent. In this industry, I have found there tends to be an inclination to leap before we look. Also, we're easily persuaded by facade as opposed to substance. All I was trying to point out by the essay is that there is a right way and a wrong way to evaluate purchased packages. So, is it really overkill? Hardly.

Again, Thanks for your e-mail. Keep those cards and letters coming.

MBA is an international management consulting firm specializing in Information Resource Management. We offer training, consulting, and writing services in the areas of Enterprise Engineering, Systems Engineering, Data Base Engineering, Project Management, Methodologies and Repositories. For information, call us at 727/786-4567. For a complete listing of my essays, see the "PRIDE" Special Subject Bulletins section of our corporate web site.

Our corporate web page is at:

http://phmainstreet.com/mba/

Management Visions is a presentation of M. Bryce & Associates, a division of M&JB Investment Company of Palm Harbor, Florida, USA. The program is produced on a weekly basis and updated on Sundays. It is available in versions for RealPlayer, Microsoft Media Player, and MP3 suitable for Podcasting. See our web site for details. You'll find our broadcast listed in several Podcast and Internet Search engines, as well as Apples' iTunes.

If you have any questions or would like to be placed on our e-mailing list to receive notification of future broadcasts, please e-mail it to timb001@phmainstreet.com

For a copy of past broadcasts, please contact me directly.

We accept MP3 files with your voice for possible inclusion in the broadcast.

There is no charge for adding a link to "Management Visions" on your web page, for details and HTML code, see the "Management Visions" web site.

Management Visions accepts advertising. For rates, please contact yours truly directly.

Copyright © 2006 by M&JB Investment Company of Palm Harbor, Florida, USA. All rights reserved. "PRIDE" is the registered trademark of M&JB Investment Company.

This is Tim Bryce reporting.

Since 1971: "Software for the finest computer - the Mind."

END