Testing in the stage where the validation of all system components occurs. testing is the basis for accepting a system, and its completion represents one of the more significant developmental milestones. Testing at any level may result in the re-design of a portion of the system.

The extent of testing required will vary considerably with the size of the system, but it remains as one of the most critical points during the system life cycle, and one that often causes the most trouble and lost time. It must be a constant consideration throughout all prior stages and their relevant activities. Listed below are the activity products which serve as background information to the testing activities:

It is advantageous to think of testing as falling into three major categories:

Initially, the designers have responsibility for manual procedures and facilities testing. The supervisor charged with implementation and conversion responsibilities will assume prime responsibility during the last phase of testing.

Program debugging is a primarily the responsibility of the programmer who wrote the program and was accomplished in the previous stage.

Subsystem and system tests are the responsibility of the designers and a specialized team, although programmers may execute the tests under their supervision. It is the designers who create the system logic, controlling information flow between programs and subsystems. The designers also know the objectives in detail, towards which the system design was directed.

Acceptance testing is a joint responsibility of the testing team and of user representatives. Acceptance testing is critical, as any failures subsequent to this activity will have a major impact on the user organization.

All of the testing categories - Manual procedures and facilities, Systems/subsystems, and Acceptance - are providing permanent materials and tools for continued system maintenance and enhancement. It is essential that this work be viewed as the creation of a permanent means of testing throughout the system life and not as a set of temporary disposable materials and tools.

The standard activities for this task are to:

The only activity which could be considered as being optional is that concerning site preparation and hardware installation, or that concerning testing training course, work aids, and human procedures. These are not always carried out due to the nature of the system or acceptance of hardware and site as constraints.

The specific products of the testing standard activities are:

During testing, three cycles of documentation will take place. The first is the establish what is planned in detail, which takes the form of a test plan. The next is to record the results of each activity. The third is to summarize the testing stage as a whole, citing overall methods, results, and conformance to the test plan. The documentation itself will reflect the progressive, accumulative approach to testing.

While extensive planning for testing was accomplished in Detail Design, more detailed planning will be required at this point by those responsible for conducting the tests. A detailed test plan is prepared for testing as a whole, and for each identifiable test which will take place. For the overall test plan, the information will be summary in nature while, for the specific test, it will be as detailed as possible. Certain parts will remain relatively unchanged, whereas other parts, such as schedules, may have to be drastically revised as testing proceeds. The results of each testing activity will be evaluated relative to this detailed test plan.

In addition, indoctrination is required for the computer room and associated personnel as to what the system plans expect of them. Standard procedures when a program halts, such as "always dump core" or "always write out register contents: must be presented.

This activity should be executed prior to the first test run being sent to the computer center. The absence of sufficient information form the computer operator, or a test package lost in scheduling, can nullify a perfectly good test shot and cause a significant delay in the completion of the system. This activity should result in a smooth relationship between operations and software personnel.

In some instances, this activity may have occurred earlier in the design cycle, but this is the latest point in time when it should take place since testing should be done. If possible, on the actual implementation configurations. In some installations, all hardware and facilities procurement is made external to system development and then becomes a basic constraint on that development. This activity would not apply in this case.

A delay in the preparation of the site or installation of the hardware can disrupt the entire schedule. In many cases, new systems will require only the installation of some additional equipment at an existing site. This installation of additional equipment should be examined in detail to ensure that it does not cause an unexpected disruption.

For an existing installation, a checklist will be used which contains:

The adding of a relatively minor amount of additional equipment can have a serious impact on the system if the existing equipment is in high load conditions: i.e. new disk drives causing channel overload.

If there is to be extensive replacement by one computer hardware configuration over a preceding one, the change-over can be very difficult, du to:

The installation of a computer hardware configuration in a new site may involve the construction of a new building or special area in a building, or modification of an area previously used for other purposes.

The site must be fully ready to receive the equipment when it is available for installation. Particular care should be taken to ensure that the equipment is moved into the site without damage. Many items of equipment are vulnerable to damage when subjected to very low levels of overstrain, overheating, being moved through a dusty area, etc.

Hardware and operating system checkout should be the sole responsibility of the vendor, but vendor’s checkout is not normally sufficient. This activity is crucial to the future performance of the system and should take place under the direct supervision of qualified personnel from the purchasing organization. If qualified personnel are not available, it may be advisable to use an independent contractor who has experience with the hardware and software and who is able to provide an objective evaluation of the checkout.

The facilities considered here can range from minor equipment additions to existing facilities, to completely new hardware systems and up to the inclusion of new buildings.

The magnitude of this task is obviously related to the above parameters. For the minor equipment changes, simple and brief acceptance procedures may be used. For completely new and massive facilities, extensive acceptance procedures will be required over a much longer period.

In this activity the best operational arrangement for program modules will be developed. The control for this activity can include system generation modifications, run time job control procedures, and/or compilation and edit procedures.

Whatever combination is used the system designer must carefully consider the architecture of the system used (virtual memory, hierarchy of memories, size of cache memory, I/O characteristics, etc.) as well as specific software requirements such as:

In any case the best source of information regarding the specific parameters which must be taken into account and the flexibility of the system are the manufactures manuals. These manuals are also a guide for evaluating trade-off involved in different approaches.

Additional functions which must be considered at this time include:

This is an ongoing activity in that the testing activities will tend to affect the size of the individual modules and the load module structure itself. In some instances, the load module structure and the job control will have to be modified many times before the testing is complete.

Previously developed training programs are tried out on groups of subjects in a practice training

environment in this activity. All courses to be used should have several practice dry runs in order to firm the content and smooth the presentation.

One of the required items is an agreed upon test(s) for any training program. To be able to evaluate any training it must be possible to test the results. A pair of tests before/after may be useful to measure the

results of training. It is desirable that the tests be oriented to the required behavior rather than to document and to prepare punching documents should be tested on actual inputs (selected for comprehensiveness) with a known standard based on actual worker averages.

For formal courses, this testing involves conducting practice sessions of the courses: formal training includes workshops. The two major benefits realized are the correction and improvement on the course material, and the development of style and expertise on the part of the instructions.

For on-the-job training, this involves the conducting of several training programs to determine whether the trainee can, infact, learn the required skills in this manner. Specification of ways to improve the on-the-job training will result from this activity. It must be noted that the individuals conducting on-the-job training will themselves need training.

Here, work aids will exercised to determine whether, in fact, they support the task in the manner anticipated. Suggested modifications and improvements should result from this activity.

All the above evaluations will recur throughout the life cycle of the training device. At this point, the particular device must be proved suitable for application to naive subjects.

In this activity, all human procedures completed in detail design are validated in a simulated work environment. Primarily, these procedures refer to alternative human response patterns in system operation. Although the specific details may differ, this validation is directly analogous to program testing. In general, validation should be done under the most realistic conditions possible, using operational personnel as test subjects. One prime purpose of this activity is to identify and correct human "bottlenecks". The level of testing required depends directly on the complexity of the system. However, testing required depends directly on the complexity of the system. However, regardless of the level, the actual conducting of these tests is extremely important. An other important feature of this testing is the fine tuning of the man/machine interface, such that subtle changes to a screen format, etc., may prove to have a strong benefit, from a human engineering standpoint, towards improving the overall human performance. This would normally follow directly from testing the training courses in that it can be an outgrowth of on-the-job training, and that trainees from those test training courses can be used to test the manual procedures.

This is a critical activity because the entire testing of the automated system and future reliability of that system will probably directly relate to the adequacy of the test data used. There is, of course, considerable difference between developing test data for a large, complicated system and doing so for a small, simple system. For the complex case it could take a large team of specialists many months to develop adequate test data, while the simple case could involve only one man for a few days.

Quite often, the services of an automated test data generator are employed. The test data generator may be either an "in house" product, or it may have been advisable to look outside the company to determine if there was a package available to do the job at a reasonable cost. The use of such a device would be most appropriate for large systems efforts. If this is done, it should have been planned during a previous stage (3.8) as last minute decision in areas so critical as this could be dangerous. In any event, a test data generator, if used, will have to be carefully analyzed and the parameters fully understood before use. The generator itself will also require testing. Even if a test data generator is used for much of the system testing, certain tests will have to be run using live data since there is often some difference between what data are supposed to be and what they actually are.

Quite often, programs developed to do the conversion which builds the live data base will be used to build a sample of that base for testing purposes. The control information required by the data base design and the data management techniques being employed must be inserted into the transformed records as the conversion procedures and logic are subjected to testing. The importance of the test data base cannot be over-stressed. Enough data must be used to create the anticipated operational conditions and enough volume to cause overloading of the system. An inadequate test data base could lead to the acceptance of a system that could not survive under a loaded operational environment.

At the same time, test transaction files should be created. This can be accomplished through a test data generator or by collecting a representative sample of live transactions. As in building the data base, there must be enough transaction data to test the load capacity of the software and facilities, as well as to test every possible path through the system. Coordination between the test data to test the load capacity of the software and facilities, as well as to test every possible path through the system. Coordination between the test data base and the test transactions is required.

Subsystem testing includes a careful testing of a program in its environment, namely, the adjacent programs. In order to ensure that nothing has been overlooked, each program will process inputs from prior programs and pass its outputs to subsequent programs. All interfaces will be tested.

In the case of batch systems, this includes job stream testing. Testing a job stream involves exercising all the programs, in sequence, required to transform initial subsystem inputs into ultimate subsystem results.

For on-line or real time systems, the process is to test transaction path. Testing a transaction path involves exercising all the programs or processes required for a unique input transaction type.

For both types of testing several purposes are served, which are to:

This activity can be accomplished only when the subsystem can be assembled and operated as an organized entity. The objectives of subsystem testing are to test for weaknesses not found in program debugging and to assure the designers that the subsystem will function as required under conditions closely related to those encountered in the operational environment. Considerable amount of manual procedure testing can be executed within subsystem testing. Attempts are made at this point to "break" the system with overload and unusual conditions.

In this activity, the project team has to create and execute a test procedure to ensure that the completed subsystem can operate without disturbance, its performance is up to standard, and that it meet the objectives originally set forth. Depending on the size of the system being developed, the subsystem test could result in being the system test. The primary concern is whether all the programs and processes can function together smoothly and whether any incompatibilities exist. The work load will be as in a normal work day and will then be expanded to overload conditions.

Of explicit importance in this phase of testing is the verification of all internal subsystem interfaces.

Of special significance are the an/machine interfaces which have not been tested prior to this point.

Once the subsystems have been independently tested, they must be brought together into a system test. This step is concerned with the testing of all relationships between subsystems and of the relationship between the system and the outside environment.

System testing takes on two difference meanings, depending on whether the implementation is phased by subsystem or not.

In the case where the system is to be implemented as an entity, it must be tested as a complete entity before proceeding. This is, when subsystem testing is completed, all those subsystems will undergo an integrated testing effort.

In the case where the system is to be implemented in phases by subsystem, each new subsystem must only be tested relative to the subsystems that have been, or are being, implemented. This is, when each subsystem is internally sound (completed subsystem testing), it is tested in the environment of the subsystems which have already been implemented. In this event, only when the last subsystem within the system is undergoing system test can this test be considered complete in terms of the system designed in the Preliminary Design stage. The first approach would normally be used in a small systems effort, where the second would be used in a large effort.

This constitutes a verification to the user that the system meet the system objectives and performace criteria, and is operationally sound.

Accceptance testing can lead to unnecessary problems if the parameters of the acceptance test have not been clearly defined. Acceptance test requirements were first specified in the Definition Study, were expanded in Preliminary Design, and were finalized in Detail Design. A typical type of difficulty is when the user asserts that the system does not do "X" and the designers and developers claim that the system was never supposed to be able to do "X". Early thought about the acceptance test can avoid these problems to a large extent.

Acceptance testing, like system implementation, amy ba phased or occur as one large formal test. As phased implementation, a phased test must not only demostrate the proper functioning of newly released functions or subsystems, but also demostrate the continued integrity of the system as a whole. A phased acceptance test has the advantages that:

In many large systems the process of conversion and implementation is so large that it is worth the additional effort and expenditure required to perform a pre-implementation acceptance test. This may be done by using systems exerciser or simulation methods. It will also train personnel and in general create a simulated working environment that relfects the real environment as closely as possibe.

This activity could be satisified in a number of ways. Normally, a parallel run with the current production methods would be appropriate. Whatever the procedure, it should be designed to do the following:

It is imperative that the user arrange and conduct his own acceptance test. This test should take place following the systems tests performed by the design and programming teams. It should be noted that an acceptance test may be conducted for the system as a whold or for any specific subsystem, depending on the degree to which the subsystems are phased.

Part of acceptance testing should be running the system during the timeframes proposed for its normal operation so that the impact on the computer center operatons can be evaluated as soon as possible. Prior to this time all development and testing will probably have been conducted in such a way as to minimize impact on production work, and the impact of the system estimated by the designers. Now is the opportunity to see what problems may be caused by running the system during the actual framework. Prior to this proint, estimates have been made concerning the effect of running the system in a production environment but now these estimates are validated.

Successful completion of the acceptance tests is the main criterion for final turnover of the system for implementation. The acceptance tests should therefore be through and complete. No part of the system should be implemented before the acceptance test has been completed. The resources used in conduting properly organized and comprehensive acceptance tests are well invested, because they lessen the possibility of trouble occurring after system turnover and verify to the user that the system performs as advertised.

The test will identify differences between what was expected of the system and what is actually does. Each unplanned difference should be examined and resolved. If the subsystem test were trhorugh, the number of differences should be minimal.

The number and effect of differences on the system can be considered as a direct reflection on any, or all, the activities which preceded this activity. The experience during the acceptance test should be reviewed and resolutions explained. During this period, future plans for the introduction of the system in the company can be reviewed and recommendations made.

The test report is prepared when all activities are complete. The report should follow the format established for reporting to management. All changes to the system and discrepancies should be reviewed and the impact on the system evaluated. All documentation should be updated.

Prior to this activity, all the products are considered as being working papers and are highly subject to change. The test result report is a final, technical report that should reflect the test results in a formal, concrete way: changes should be few and made only after careful consideration.

The report should be processed for review by all affected parties and submmitted fro managerial approval. Management will:

This analysis will be performed for testing as a whole and for each identifiable test. For individual tests, this information will form the basis for making corrections to system components, and for modifying test plans for tests not yet conducted. The complexity of the analysis will depend on the level being tested and on the nature of any problems which may have arisen. Suggested contents of this report now follow.