This paper presents the maintenance testing environment used for the CF-188 Operational Flight Program (OFP) as an example of the problems involved with the verification of such software. Among the special challenges in this testing environment are hardware interactions, and safety and timing issues. The difficulty of predicting what will be changed over the lifetime of the aircraft (at least 20 years) adds to the challenge. As well, verification during software maintenance differs from verification during software development. Much of the paper addresses issues involved in increasing the automation of testing of such a complex system
Keywords
system testing, regression testing, subsystem testing, real-time, safety-critical
Introduction
With the increase in modern systems such as the Tactical Command, Control and Communications System, the New Patrol Frigate, the CP-140 Aurora long range patrol aircraft, the CF-188 Hornet fighter aircraft, and the planned Sea King helicopter replacement, the Canadian Armed Forces is faced with the continuing and expensive task of maintaining large amounts of software.
This paper focuses on the CF-188 Operational Flight
Program
(OFP) to illustrate in some detail the problems associated
with the verification of modifications to such complex long-lived embedded software. It is based on work that was
completed over a year ago [9][10]. A brief update on the
current status is provided in the Conclusion. The CF- 188
OFP is broken up into several subsystems, each of which
runs on its own processor (or processors) and interacts with
different items of hardware on the aircraft, such as radar,
weapons, pilot input switches, and flight control surfaces.
The subsystems are connected by a series of buses, which are
multiplexed, and are therefore referred to as MUX buses. The
CF-188 OFP testing environment must provide suitable
interfaces for testing the integrated system and each
subsystem, providing either actual hardware or a suitable
emulator for each hardware device the system or subsystem
interacts with. The CF-188 OFP presents an additional
challenge because it is safety-critical hard-real-time
software. Safety-critical software is software which could
result in injury or damage should it fail to perform as
required [8]. Hard-real-time software is software which must
not only be logically correct, but must also satisfy stringent
timing constraints [7]. Verification that the software meets
the functional, safety and timing requirements is one of the
major cost drivers for the maintenance of weapon system
software. To make matters still more complex, verification
during software maintenance is often more difficult than
verification during software development, for a number of
reasons which will be explored later in the paper. The main
issue is the risk of introducing problems in parts of the
software that have not been modified, which could impact
the operational effectiveness of the entire fleet of aircraft.
The CF-188 Software Maintenance Process
The CF-188 software maintenance process consists of six phases: changes are proposed, validated, evaluated, approved, implemented, and verified.
In the proposal phase, system and/or subsystem problems or changes to system level or subsystem level requirements are identified and recorded on Software Trouble Reports (STRs).
Validation activities applied to each STR depend on whether the STR is based on a problem or a change in requirements. In the case of a problem report, a top-down testing approach is followed in order to duplicate and localize the source of the reported problem. Typically, test cases based on the operational profile of the system at the instant that the problem was first detected are executed on the system test rig in an attempt to localize the problem to a particular subsystem. Once the problem has been localized to a subsystem, testing is carried out on the applicable subsystem test rig and software development system in an attempt to localize the problem within the subsystem. In the case of a change in requirements, the process is simpler: the STR is assessed on the basis of its operational validity.
Once the STR has been validated, it is converted to a Software Change Request (SCR), which is then subjected to an engineering evaluation to provide an estimate of technical, operational, cost and schedule impacts. Technical evaluation activities for those SCRs which are problem based typically follow a bottom-up approach starting from the problem source. They include structural testing at the unit level to determine which units require modification to implement the change, and a combination of static and dynamic testing and analysis at the unit, subsystem, and system levels to identify potential integration impacts of the change. Technical evaluations for those SCRs which are requirements based typically follow a top-down approach starting at the level at which the requirement is changed. For example, the evaluation of a proposed change at the system level, such as the addition of a new weapon, commences with system level testing on the system test rig using a simulation of changed subsystems to examine integration issues at the system interfaces. It also includes subsystem level testing using stubs and/or drivers on the appropriate subsystem test rigs to examine integration issues at the subsystem interfaces. Finally, the appropriate software support environment is used to assess integration issues.
The approval phase of the CF-188 software maintenance process identifies SCRs which are to be implemented in future builds of a particular subsystem and allocates resources to complete the next build.
The implementation phase of the process includes all of those activities that are usually associated with software development, with some differences. The first difference is that modifications are designed and developed for only those units affected by the SCR. The second difference is the approach taken to incrementally build the new software baseline. First, the modifications made to incorporate each SCR are separately integrated into the baseline software in accordance with a bottom-up approach. Second, the new software baseline is created by integrating the modifications of individual SCRs incrementally. As well, the implementation step is contracted out, so the details of how it is conducted are usually hidden. The subsequent verification effort emphasizes the testing of requirements affected by the SCRs, based on maximizing code coverage of the modified software and on a suitable level of regression testing. [11].
The CF-188 OFP Verification Process
Verification must ensure that modifications are implemented correctly, in accordance with modified specifications, and that the modifications do not adversely affect unmodified software. In the case of the CF-188 OFP maintenance, verification emphasizes test planning and test execution at the subsystem and system levels [4].
Verification During Software Maintenance
Verification is defined as the "process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase" [6] or as the activity that "ensures that each step of the development process correctly echoes the intentions of the immediately preceding step" [3]. These definitions do not address the unique requirements of verification during software maintenance in that they fail to recognize that the maintenance phase starts at the end of the development phase, with an existing system. Verification during software maintenance must not only answer the question "are we building the product right?", it must also deal with whether the product was built right, and what has changed since. It must also deal with the fact that a copy of the software is in use on each aircraft in the fleet, and modifications can impact that use in a negative way.
Verification during software maintenance, especially in a subsystem/system integration environment such as that found in the case of the CF-188, is often more difficult than during software development. The maintainers of the software are usually not the developers. The maintainers' only visibility into the design decisions made during original development is either via the requirements and design specifications or via the source code. Often, as is the case with the CF-188 OFPs, the requirements and design specifications are inadequate for verification purposes. In the case of the CF-188, they do not decompose the software in a clearly defined manner and are incomplete [11]. For example, the timing constraint specifications for the CF-188 Mission Computer OFP are almost non-existent. It appears that the designers' concern about the timing behaviour was minimal due to the fact that they could design the original version of the OFP to fit the constraints imposed by the processors and memory [1]. For maintainers, this lack of timing specifications means that there is no way to predict accurately if modifications will result in timing problems.
The CF-188 OFP specifications are written in a combination of natural language and flow-charts, for which the verification techniques are limited. The benefits of re-engineering the specifications using other notations are outweighed by several factors, including the difficulty of contracting for software, the required level of effort, and the cost of training. Using formal notations can make this situation even worse, and there is dubious benefit in attempting to prove correctness, due to the questionable validity of proofs in an environment as complex as the CF-188 [5]. As another example, experience with the SCR Methodology on the A-7 and Darlington Nuclear Generating System re-engineering exercises indicates that the level of effort required to generate and prove the formal specification was very high and, while there was an improvement in the precision and readability of the specifications, of questionable value in terms of cost and product quality [2]. Tool support for formal specification notations improves this situation, but tools are generally still at a research stage and not yet ready for large-scale production use. Furthermore, many of the modifications to the OFP involve the integration of off-the-shelf (OTS) software developed by the United States Navy. The specifications for this OTS software would likely not be compatible with any one new specification method chosen.
There are advantages to testing in the maintenance phase. One of the principal ones is that it can be worthwhile making significant investments in the testing environment. That is certainly the case for the CF-188. Even so, funds are limited, so one of the issues addressed in this paper is how to assess which areas of improvement in the testing environment are most worthy of investment. The paper does not attempt to provide detailed cost-benefit analyses, but does present the factors and issues that are important in the decision making process.
Factors Affecting CF-188 Testing
The employment of testing techniques during the maintenance of the CF-188 is influenced by several factors. First, the majority of software maintenance is now contracted out. Second, the various subsystems have not been formally decomposed and documented. Third, the different subsystems were designed using different methodologies and are written in different programming languages.
The primary impact of the contractual arrangements is that the contract imposes formal testing only at the subsystem and system levels. As a result, subsystem and system tests follow contractually specified procedures, while lower levels of testing follow the contractor's internal procedures.
Because the various subsystems have not been formally decomposed and documented, unit specifications are either poor or non-existent, so testing of modified units in isolation is difficult. As well, new and modified units are integrated into an existing baseline. These two factors make it attractive to use integration testing to achieve unit testing, provided that the integration tests provide sufficient unit test coverage and observability.
The fact that the software within the various subsystems has been designed using different methodologies and programmed in different languages means that different software support environments are required for each subsystem. In general, the programming support environment that was used to develop the software for the subsystem was delivered with the subsystem. The result is that unit, integration, and subsystem level testing uses testing tools which are unique to each subsystem.
The testing tools employed in the CF-188 software maintenance environment include both static analysis and dynamic analysis tools. In this section, some of the testing tools used to test the various CF-188 subsystems and the CF-188 system are described and assessed. The major distinction for testing purposes is between Prime Mission Vehicle (PMV - referring to the aircraft itself) flight testing, and ground based testing using the Integrated Test Facility described later.
Test cases for both PMV flight tests and ITF system and subsystem testing (except as noted below in the detailed descriptions of subsystem testing) include design-based, equivalence partitioned and boundary value functional tests. They also include regression tests, to ensure that modifications have not impacted the implementation of those requirements which were not modified. Flight test cases are described in a set of flight test cards which provide the pilot with directions on how to execute the tests manually. These flight test cards are developed by hand, based on written requirements. ITF system and subsystem test case descriptions provide the test engineer with explicit directions on how to set up the test environment and how to execute the tests manually. As is the case for flight testing, these test cases are developed by hand based on published specifications. In both cases, the lack of a complete set of system requirements in a testable format makes the determination of test cases very difficult [11], but improved requirements are gradually being developed as changes are made.
Prime Mission Vehicle Flight Testing
Final software testing is flight testing on the actual aircraft, or PMV, using dynamic, functional black-box testing.
The primary advantage of flight testing is that the software is subjected to the actual operating environment, including actual real-world inputs and a completely integrated avionics suite. The primary disadvantages are safety issues, the high cost of flying hours, the limited controllability and observability of subsystems, and very limited on-board monitoring capabilities, which do not permit thorough logging of test results. Monitoring of flight tests primarily involves observations by the pilot, though some aircraft instrumentation, such as the Heads-Up-Display (HUD) camera and the Maintenance Signal Data Recording System (MSDRS), provides a limited recording capability. Also, the aircraft can be specially instrumented with high-speed cameras, video cameras, strain gauges, and special MUX data recording equipment.
The PMV does not have an automated test capability.
The CF-188 System Model
This section contains a partial list of the CF-188 avionics subsystems that will help in understanding following sections. The top level CF-188 Avionics System is decomposed into a number of subsystems, including:
- Mission Computer (MC)
- Radar Data Processor (RDP)
- Radar Signal Processor (RSP)
- Stores Management Set (SMS)
- Electronic Warfare (EW)
- Integrated Communications, Navigation and Identification (CNI)
- Multipurpose Display Group (MDG)
- Maintenance Signal Data Recording System (MSDRS)
Each subsystem has its own OFP; the aggregate of all OFPs is also referred to as the OFP. Most of these names are relatively self explanatory, but a few need some explanation: The MDG runs multiple displays for the pilot, and the SMS manages the armament. Other subsystems relevant to this paper include the Up-Front Control Panel (UFCP), which provides the primary cockpit control of the CNI subsystems, the Ground Proximity Warning System, which is an independent subsystem except for use of sensors and displays, and the Flight Control Computers, which are responsible for the fly-by-wire subsystem.
Current software practice would document the further decomposition of each subsystem into smaller modules. Unfortunately, the internal structure of the subsystems was largely not documented in the original system, although progress has been made in this direction over the past several years.
Integrated Test Facility (ITF)
The Integrated Test Facility (ITF) is a dynamic testing environment which is primarily used for system testing and subsystem testing of the Mission Computer (MC) subsystem. Test stations for other subsystems can be switched in and out, depending on the mode of operation of the ITF. A functional duplication is provided of those CF-18 subsystems required to perform testing of the MC subsystem, along with a simulation of the real-world operational environment. Certain avionics subsystems can be physically present as part of the ITF. Dynamic and static software simulators are available to represent the functionality of most subsystems. Dynamic software simulators provide data in real-time while static simulators simply respond to queries with an acknowledgment. The "real-world" environment is simulated using dynamic simulation models.
Avionics subsystems which can be physically incorporated into the ITF include actual radar receiver antennae, several devices for pilot input, and several displays. Avionics subsystems which are only modeled using dynamic software simulators include communication radios and the altimeter. Several other avionics subsystems are only represented by static software simulators which provide only a simple Avionics MUX bus interface, or which have limited dynamic functionality. These include the Ground Proximity Warning System, the Flight Control Computers, and the Maintenance Signal Data Recorder Set (MSDRS).
"Real-world" inputs to the ITF are simulated. Dynamic earth and atmospheric models are used to simulate the "real-world" flight environment. A Radar Target Generator (RTG) is used to inject simulated radar targets into the radar when the actual radar is used. An Electronic Warfare Emitter Generator (EWEG) is used to inject simulated threat emitter signals into the Radar Warning Receiver antennae and a Weapons Simulator is used to inject weapon signals into the Stores Management System.
The ITF incorporates several capabilities which permit non-intrusive monitoring and recording of up to 100 input and output simulation variables, environmental variables, discrete avionics variables and MUX bus words and variables as well as non-intrusive monitoring of several discrete signals. All recorded variable values are time-stamped with a universal reference time. This can later be used for timing analysis. Several actual CF-188 avionics displays and indicators permit the monitoring of data presentations intended for the pilot.
There is currently no acceptable tool for monitoring program execution within the MC subsystem.
The monitoring capabilities of the ITF have recently been upgraded to provide a more user-friendly graphical user interface which permits the tester to select various simulation variables, MUX words, specified variables within the MUX words, and discrete signal variables for analysis during the test execution. The values of these variables may be displayed in alphanumeric format or plotted on a graph against either the value of other variable values or the value of the time-stamps generated by the universal time code generator. The capability to plot variable values against each other permits some analysis of cause-effect relationships while the capability to plot variable values against the universal time permits some analysis of timing issues.
The ITF and its test stations have a limited automated test capability. Script files can be written in a special command language and automatically executed. This capability is limited in that the command language is sequential and does not support basic programming language features, such as loops, required for more complete automatic generation of test cases. Furthermore, there is no capability to compare actual test outputs automatically with expected results.
The primary advantages of testing on the ITF versus the PMV are the elimination of the flight safety issues, and the lower cost. For example, test cases on the ITF can include stress tests which exceed the safe limits of the flight envelope of the aircraft. Also, testing on the ITF provides a much better monitoring and control capability for the test cases. The primary disadvantage of testing on the ITF is that the software is not subjected to "real-world" stimuli. The simulated environmental inputs and nearly complete integrated avionics suite on the ITF minimize this disadvantage.
Test Support Hardware and Software in the ITF
Mission Computer Support System
The Mission Computer Support System (MCSS) is hosted on an IBM system and provides both static and dynamic analysis tools for the Mission Computer (MC) and the Multipurpose Display Group (MDG) subsystems. These tools are used in combination to compile MC and MDG source code, to debug MC software, and for unit and integration testing of MC units.
The static analysis capabilities within the MCSS include a compiler and a Data Base tool. The compiler has static verification capabilities which are common to many other compilers. For example, it checks the usage of symbols for multi-defined symbols and symbols not in the symbol table. The Mission Computer Data Base tool is an Oracle database application that provides information on MC internal and external parameters, multiplex bus word list, and MC routines. It is also used to provide cross referencing between these elements and to obtain the dynamic structure (calling tree) which represents the calling sequence of routines within the MC software. This tool is important and valuable. It is used to select regression tests by identifying where the impact of modified code is likely to be.
The dynamic analysis capabilities within this support system includes a Function Simulation Program (FSP) and a Multipurpose Display Group Simulator (MDGSIM). The FSP consists of an emulator and a User Control Program (UCP). The emulator provides bit-by-bit results identical to those which would be obtained if the program were executed on AYK-14 computer that the MC OFP normally runs on. The UCP controls the FSP execution, reads data in, schedules module calculations, and writes test data on files for subsequent printing. It provides the user with the capability to set values within the MC memory, to define the rate at which the MC modules are to be executed, to provide the logic to schedule the input of events when the elapsed time is reached and to define schedules to impose order on the individual MC modules. It also provides various diagnostic capabilities to verify the subsystem, including the capability to compare computation results to user-defined expected results, to dump memory contents, to inspect an absolute memory address, to trace execution, and to observe changes in parameter values. The UCP also includes a pathfind capability to aid in the conduct of structural testing by detecting and monitoring all paths taken by a set of test cases.
The FSP is used for both unit and integration testing. Integration testing is conducted by executing the integrated units in a specified sequence for a specified number of iterations, forcing the units through specified paths, and permitting the examination of full or partial results of the test.
The MDGSIM tool is used for unit testing of the MDG code. It provides a simulated graphical display of two displays used in the aircraft. Unit testing involves the downloading of the background and first pass cyclic information after which only cyclic parameters are updated under UCP. This tool is used primarily to verify symbols and the positioning of symbols on the displays.
Test cases for MC and MDG unit-level tests include structural path coverage tests to verify the control flow within modified units.
Data Reduction and Analysis Station
The Data Reduction and Analysis Station (DRAS) was intended to provide a post-test analysis capability of the MUX and discrete digital data recorded during the execution of tests on the ITF and the Radar Software Test Station (RSTS). The DRAS also includes a Video Cassette Recorder (VCR) capability to permit the playback of the video recordings of two repeater Digital Display Indicators (DDIs) connected to the ITF. The event history of selected recorded variables can be displayed, synchronized with the time-stamps inserted at the time of recording. Additionally, it is possible to use the time-stamps to manually synchronize the video recordings with other data, although the limitations of the standard VCR make this difficult. The DRAS also can compare test results gathered during different executions of a test case, search for variable values which are outside a prescribed upper and lower bound, and calculate the accuracy of some air-to-ground weapons.
Unfortunately, this station is not currently used because of its very poor user interface. Even if the interface is improved, it is very difficult to correlate the program execution traces from the special MC and radar test tools and the In Circuit Emulator (ICE) with the data playback on the DRAS This means that the analysis of much test data is restricted to the real-time analysis by the tester of the monitored variables and the various CF-188 displays. The requirement to repeat tests to verify that different events occurred correctly is costly, and there is no verification of real-time events that occur at rates higher than can be analysed by a human tester.
Test Stations Included in the ITF
A partial description of the test stations in the ITF follows.
Stores Management Set Test Station
The Stores Management Set Test Station (SMSTS) is a dynamic testing tool for the SMS subsystem and for initial testing of the integration of the SMS subsystem with the MC subsystem. It includes a Stores Management Processor (SMP), the encoders/decoders which are used to program the weapons on each weapon station, a weapon simulator which is used to replicate the signals generated by the weapons, and a semi-dynamic simulation of the mission computers.
The SMSTS incorporates several capabilities which permit non-intrusive monitoring and recording of input and output simulation variables, environmental variables, discrete avionics variables and MUX bus words. In addition, a test point panel is provided to permit attachment of multimeters, logic analysers and a strip chart recorder. All recorded variable values are time-stamped with a universal reference time. Monitoring of program execution within the Stores Management Processor (SMP) is possible through the use of an Intel 8080 ICE which provides the capability to insert breakpoints and examine memory and register contents.
SMS subsystem test cases describe subsystem stimuli and expected responses, but not subsystem performance requirements.
Radar Software Test Station/Electronic Warfare Integrated Test Bench; Radar Support Environment; Communications System Controller Test Station
These parts of the ITF are grouped here because they have similar characteristics, including the fact that test cases have only been developed specifically for them very recently, because changes are just now starting to be made to the subsystems they are associated with. They have nonetheless been important, because they have been needed for testing modifications to other parts of the CF-188 software.
The Radar Software Test Station (RSTS)/Electronic Warfare Integrated Test Bench (EWITB) is a dynamic testing tool which is primarily used for Radar and EW subsystem testing, though it is also used for subsystem testing of the Mission Computer (MC). Its design and capabilities are virtually identical to those of the ITF. Depending on the mode of the ITF, it runs as part of the ITF, or the ITF can run without it, in standalone mode. The primary difference between them is that the actual radar and the actual Electronic Warfare (EW) subsystems are integrated into RSTS and EWITB respectively (and hence into the ITF when it is operating in the mode in which it is integrated with the RSTS or EWITB). Other differences are that there is no actual Heads-Up Display (HUD), and programmable touch panels are used in place of avionics switches.
The Radar Support System (RSS) provides both static and dynamic analysis tools for the Radar Data Processor (RDP) subsystem. The static analysis tools include a propriety Hughes Assembly Program (HAP). The dynamic analysis capabilities include a Programming Support Environment (PSE) which provides capabilities comparable to those of an In-Circuit Emulator. It can be used to control the execution of the program in the RDP, to insert breakpoints, to examine memory and register contents, and to write values to memory and registers. There is currently no support for the Radar Signal Processor (RSP) subsystem.
The Communication System Controller Test Station (CSCTS) is a dynamic testing tool which is used for testing and debugging of the CSC subsystem, and for initial integration testing with the MC subsystem. It also provides partial coverage of the navigation and identification subsystems. Certain avionics subsystems are physically present on the CSCTS, while dynamic and static software simulators are used to represent the functionality of others.
The RSTS is the only one of theseparts of the ITF that has unique non-intrusive monitoring and recording capabilities can be applied to virtually all variables. All recorded variable values are time-stamped with a universal reference time generated by a global time code generator for timing analysis. Monitoring of program execution path coverage within the Radar Data Processor is possible through the use of the special Programming Support Environment (PSE) tool; however, this capability is severely limited by the fact that the PSE is only capable of recording data at a one millisecond rate.
Test cases for the Radar and CSC subsystems have only started to be developed very recently, since the first applicable SCRs have been approved for implementation in the past year. Test cases for the EW subsystem level tests have been developed recently, with the EWITB portion of the test station having been recently released for use. The lack of experience with test cases for these subsystems makes assessment of their potential for automation difficult.
System and subsystem testing in a software maintenance environment is needed both to verify the functionality of modifications, and to ensure that they have not impacted the implementation of those requirements which were not modified. Automating these tests would have many advantages:
- It would serve to improve the repeatability of the tests by minimizing the possibility of human errors during test input injection. This is essential for regression testing, which involves repeating past tests to ensure that the modifications to the software did not introduce faults into previously operational software. It would also be beneficial for the tests used to verify the functionality of modifications during a particular integration effort, as they are often used as regression tests during subsequent integration testing.
- The elimination of delays caused by human interaction would permit the execution of more tests within what is typically a limited amount of time allocated for testing.
- An automated capability would reduce the amount of post-test analysis of test data required by providing a capability to monitor and analyse more variables during the execution of a single test case than a human tester could. This can be especially important in a real-time situation where test cases may not be repeatable, since environmental variables cannot be completely controlled.
- Automated testing tends to improve the organization of testing by imposing discipline on the conduct of test planning and execution, and provides better measurement of testing coverage.
In the following sections, possible improvements to the automated testing capability are investigated for the CF-188 testing setup. Some of these improvements have now been implemented. A brief status report is given in the Conclusion.
Automating ITF Test Initialization or Reset
The ITF must be re-initialized between tests under certain conditions: to load different code into the processors, to reset the simulation environment, or to change a selection of emulators versus real subsystems. The initialization process is highly automated, but still requires some operator intervention. Complete automation of the initialization and reset procedures is not practical.
Automating the Injection of Test Inputs
Testing of real-time software at the subsystem and system levels requires that the software be subjected to "real-world" test inputs at the appropriate time. In the case of the CF-188, these "real-world" inputs include environmental inputs, such as atmospheric conditions, radar targets, EW emitters, various radio stations and navigation aids, and inputs made by the pilot. The ITF provides for the automatic injection of many, but not all of these inputs. In particular, environmental inputs, radar target return signals, EW emitter signals, weapons signals, communications signals from radio stations, and navigation aids signals are all automatically injected in real-time by the various dynamic simulation models and/or simulators. Inputs normally injected by the pilot are injected either manually via actual CF-188 flight controls in the ITF or, in some cases, semi-automatically using the ITF override capabilities.
The following sections examine the potential for increasing the automation of those inputs on the ITF which are currently injected manually or semi-automatically.
Automated HOTAS Control Issues
The Hands-On-Throttle And Stick (HOTAS) is the primary flight control in the CF-188. It provides the pilot with control of the throttles and flight control surfaces. It contains switches which permit the pilot to control other functions, such as the selection of radios, the selection of weapons, the release of weapons, the aiming of the radar antenna, and the selection of radar targets, to name only a few. The ITF provides software overrides for all significant HOTAS inputs in system and subsystem testing; but in some test cases, actions must respond to external stimuli in ways that are difficult or expensive to coordinate automatically.
For example, the tester may define initial flight simulation conditions such as the initial altitude, position, speed, etc of the CF-188 system, and then "fly" the system within the flight simulation using the actual aircraft flight controls. Some tests will require flight control inputs to compensate for inputs from the environmental simulation models, such as the varying wind conditions generated by the Atmospheric Model. Currently, the tester is required to interpret the effects of these atmospheric conditions and provide the appropriate control inputs. Such control inputs cannot currently be controlled or are not easily controlled by the ITF command language because it does not support loops; thus, there can be no such control-loop compensation. The full provision of this capability in the ITF test software would be the equivalent of providing an automatic test pilot. This would mean test software as complex and hard to verify as the software under test.
As another example, The Target Designator Controller (TDC) switch is used to position the TDC box over a radar target visible in the radar display or an actual target on the Heads-Up-Display (HUD), and to activate some of the Multipurpose Display Group (MDG) menu functions from the HOTAS. While the position of the TDC indicator and the activation of the MDG functions using the HOTAS inputs can be controlled using the ITF command language, this capability is not useful in practice. The tester interprets the displayed video images and decides what actions to take. This decision process and control is very difficult to automate, since it would require a capability to monitor the position of the targets on the appropriate display and a capability to coordinate the position of the TDC indicator and the target. In essence, this would require an additional processor and complex calculations linking parts of the ITF which currently operate independently.
Automated Avionics Subsystem Control Issues
There are several cockpit controls and displays which are used by the pilot to control the operation of various avionics subsystems within the CF-188. The MDG and the UFCP are the primary avionics control panels in the CF-188, while the ALR-67 Radar Warning Receiver (RWR) Control/Indicator is used to control the Electronic Warfare (EW) subsystem. The MDG includes three multifunction displays which each have twenty programmable switches around their display periphery, and a Heads-Up-Display (HUD). The ITF has a current capability to monitor but not override the ALR-67 and MDG programmable switches, and does not have a capability to enter data into the UFCP.
Automated Test Input Injection Requirements
The capability to control MDG switches automatically is required for fully automated system level and MC subsystem level testing of most avionics functions. The capability to automatically inject data into the UFCP is required for fully automated system level and MC subsystem testing of the navigation and communication functions. The capability to automatically control ALR-67 input data via the ALR-67 switches is required for fully automated system level testing with the EW subsystem.
There are two ways to provide the automatic control needed: an override capability for the switches and data entry devices, or the replacement of the control panel with an emulator. In addition, given that fully automated testing is not practical for some functions, the possibility of semi-automated testing is discussed.
Automated Switch and Data Overrides: The addition of overrides for the various switches and data entry devices would provide the capability to automatically inject the test inputs; but it would require a hardware modification to the various avionics control panels. This is difficult, because of the physical design of the various control panels, which are actual CF-188 avionics components. As well, the modified control panels would be unique in the fleet. This, in turn, would make their maintenance expensive and unreliable.
Emulators: Another option is to replace the various control panels with emulators. In the case of the UFCP and the ALR-67 RWR control panels, this option would provide the capability to control the injection of test inputs automatically into the Communications Set Controller (CSC) and the EW suite subsystem respectively. In the case of the MDG, however, this option would impact the integrity of the system level and MC subsystem level tests. In particular, the MDG display and control functions are controlled by processors within the MDG, whose activity is in turn coordinated by code that runs on the MC. One of the goals of system level and MC subsystem testing is to verify the real-time processing capability of the MDG code, both on its own processors, and on the MC. Emulation of the MDG would limit the ability to verify the real-time processing of the MDG code. Furthermore, there is reluctance among pilots to accept a system which has been verified using an MDG emulator, since the MDG is the primary avionics control for the pilots.
Semi-Automated Testing: There are several advantages to the implementation of a semi-automated test capability, in which the operator would be prompted for those inputs which cannot be automated. First, it would improve the repeatability of tests by automating most test inputs and controlling the injection of the others. Second, it has the potential to reduce the test execution time, although the amount of time saved would be highly dependant on the speed of the tester's response to the prompts. Third, it would not require any changes to the current ITF configuration.
There are also several disadvantages to the implementation of a semi-automated test capability. First, the inclusion of the prompts to the tester, and any error handling required to handle an incorrect or untimely input by the tester would increase the complexity of the test software. Second, there may be insufficient time for the tester to respond to a prompt during a real-time scenario.
The analysis of test results during test execution requires that the behaviour of the system/subsystem(s) be monitored and analyzed with respect to the expected results. For system level and subsystem level testing of real-time systems, this means that the system and/or subsystem outputs must be compared to the correct value and the correct timing must be verified. In addition, there is a requirement to measure the coverage of the test cases to help determine when sufficient testing has been done. Analysis can be conducted either in real-time during the execution of the test, or post-test using a record and playback capability.
The ITF has no capability to automatically analyze test data during the execution of a test. All real-time analysis must currently be carried out manually by the tester.
The following sections examine issues which impact the ability to provide an automated real-time analysis capability on the ITF and issues which impact the ability to automate post-test analysis on the DRAS.
Monitoring and Recording Issues
Limiting the recording of simulation, environmental, digital discrete signal and MUX bus word variables to a maximum of one hundred total monitored variables has little practical adverse impact on automated real-time analysis, since the analysis of more than one hundred variable values in real-time would require very complex test software. Furthermore, the ITF has the capability to record all MUX bus words and up to one hundred other variables; so additional analysis can be done post-test. On the other hand, there are times when test cases must be redone, with a different selection of variables to be recorded.
Real-time analysis of analog discrete signals cannot easily be automated, since these signals are monitored using equipment such as oscilloscopes, power meters, and strip chart recorders. This limitation has little impact on the automation of system level or MC subsystem testing since most test cases do not require this type of monitoring. Furthermore, these outputs can be verified either in real-time using non-automated tests or they can be recorded for post-test analysis.
Real-time analysis of data displayed on the various actual CF-188 displays, such as the MDG displays, the UFCP, and the ALR-67 control/indicator, cannot easily be automated since the test software in the ITF cannot monitor the display data. For example, the test software cannot easily be modified to analyze the correlation of the TDC indicator and a radar target on the radar display page on the MDG. The impact of this limitation is minimized by the ability to monitor and record the data sent to the displays. Furthermore, the actual CF-188 displays can be recorded for post-test analysis using a Video Cassette Recorder (VCR), although there are problems in making use of these recordings, due to the difficulty of coordinating them with other test outputs on playback.
Special test harnesses and tools external to the ITF provide a capability to monitor the execution of the MC and RDP subsystem code on their respective processors. However, they cannot be controlled remotely by the ITF. Furthermore, these tools are cannot directly measure path coverage, although they can generate program execution traces which can later be analyzed to determine path coverage. In-Circuit Emulators (ICE) can be used to monitor the execution of the CSC and SMS code, although they may cause system level faults, such as timing faults, that may not exist with the use of the actual processors.
Automated Real-Time Analysis Requirements
While the ITF provides some capabilities to automatically monitor and record test data, it was not designed to support automated real-time analysis. Fully automated real-time analysis requires a capability to compare actual monitored values to expected results. Assuming that the CF-188 specifications were sufficiently precise to permit either the determination of expected results for inclusion in a test program or the derivation of algorithms to calculate the expected values, the test command language and the test controller need several capabilities. First, they require a capability to calculate the expected results, possibly based on the values of one or more monitor variables, in real-time. Second, they require the capability to compare the monitored values with the expected results. Third, they require a capability to select or skip test procedures based on the results of the analysis, and finally, they require a capability to report the analysis results. Included in the latter is a requirement to be able to indicate which tests were and were not executed.
The provision of the capability in the test software to compute the expected result increases the complexity of the test software and imposes a requirement to verify that the test software computes the expected value correctly. In this context, it is possible, especially in a real-time system, that a measured variable may be correct either within a given tolerance of the expected value or within a given range of values. For example, a test of the "stall warning" function in a CF-188 system or MC subsystem test is dependant on a combination of variable values pertaining to air speed and the attitude of the aircraft. In this case, the test software could analyze the values of the pertinent variables and determine if the stall warning was correct at the appropriate time. Given that the test software in this example could be of similar complexity to the actual stall warning function software, this example also shows how the overall complexity of the software can be doubled, with resulting additional verification costs.
One requirement that is clear from the above is that the ITF command language and its sequential interpreter must be either upgraded or replaced with a capability which can provide real-time analysis.
New DRAS
Given that the current DRAS is based on late 1970's technology, an upgrade to its capabilities it is not considered practical. Instead, a new DRAS is needed to provide the post-test analysis capability of the test data that was recorded during test execution on the ITF. This new DRAS should provide a capability to display the test data in different formats, including a scrollable list of user-specified variables in a text-oriented tabular format which is keyed on the global reference time generated by the time code generator, a graphical plot format in which the values of user-selected variables are plotted versus the global time, and an event history format in which the changes in values of user-selected variables are listed against the global reference time at which they changed. The new DRAS should have an interactive query and reporting capability which supports queries based on such criteria as the magnitude of the minimum and maximum excursions, the average value of the variables, and the total number of transitions of the user-selected variables. It should provide a capability to analyze and display the program execution traces generated by the various ICEs and the special tools used for monitoring. The use of the VCR to record the CF-18 displays should be replaced with a digital record and playback capability which uses the global reference time. This would permit the correlation of the displayed data with other recorded data. Finally, the DRAS should provide a capability to analyze analog data that was recorded.
An Automated Test Capability for the ITF
While fully automated testing is not feasible for all test cases, such as those which require the acquisition of a target using the TDC and the radar antenna elevation controls, there are many test cases which do not require the acquisition of targets. These test cases can and should be automated. Automation of these test cases requires the implementation of emulators for the MDG, the UFCP, and the ALR-67 RWR control panels, and the implementation of a test language which is capable of controlling and logging the control flow of the test activities based on results while tests are in progress. The implementation of the emulators will provide both a means of automatically injecting test inputs and a means of automatically monitoring and/or recording the display outputs for analysis of the functional correctness of the system behaviour. Also, the global time-stamp capability within the ITF should be used to correlate the display data on the emulators with the MUX and digital discrete data captured by the current ITF monitoring and recording capability; however, this data cannot be used to analyse the correctness of the timing behaviour of the system, since the emulators may not exactly emulate the timing behaviour of the actual system components. For those tests which are designed either to verify the exact timing behaviour or the exact MDG display presentation of the system, either manual or semi-automated testing using the actual aircraft control panels will be required. Automated test cases should ideally commence immediately after the power-on initialization or reset to ensure that the processors are in a known initial state at the commencement of the test execution.
The CF-188 system and its subsystems are hard-real-time software systems. To perform properly, they must be logically correct, and must satisfy timing constraints. The maintenance of this software requires extensive use of testing, to verify the functionality and timing of the modified software, and to ensure that the modifications have not impacted other parts of the software. The goal of this paper was to examine how the degree of automated system and subsystem testing capability could be increased, based on the current CF-188 Integrated Test Facility (ITF).
Automation of testing on the PMV is out of the question, as it would require extensive modifications to at least one aircraft. This creates a number of problems, and leads to high expenses for marginal returns. By the time PMV testing is used, the software has already achieved a high degree of reliability, and problems that do appear during PMV testing can generally be found using the ITF. As well, there is no guarantee that more automated PMV testing would significantly improve diagnosis capabilities.
The paper has examined the CF-188 software maintenance process, organized around the various testing tools within the CF-188 software maintenance environment. Limitations in the application of testing techniques in the current CF-188 software maintenance environment were identified, one of which is the limited automation of system and subsystem testing capabilities.
Regression tests at the system and subsystem levels are candidates for further automation. Current ITF capabilities were examined to determine the extent to which further automation is feasible. This included an examination of possible improvements in automating inputs and analyzing test results, both during execution and post-test.
It was concluded that the automation of tests designed to verify exact timing requirements and to coordinate inputs with the exact presentation of data on the displays and indicators would not be practical. The problem is the high present and future cost of modifications to the avionics used in the ITF and of a capability to interpret and interact with displayed data to enable selection of subsequent test inputs. On the other hand, it is practical to automate the majority of test cases executed on the ITF by implementing emulators for the MDG, UFCP and the ALR-67 controls and displays, and improving the test control language so it would be capable of controlling and logging execution of a wider range of test cases.
Development of initial versions of MDG, UFCP and ALR-67 control and display emulators for incorporation in the ITF is now complete. These efforts commenced with the development of a digital record and playback capability for the control panels and display data of each of these control and display system. Subsequent phases of these projects will develop a more complete emulation of these control and display systems. A new version of the DRAS has been delivered, which can record pilot inputs and replay the flight on the ITF. There continue to be plans to investigate an improved test control scripting language. It is also of some interest that, while this paper has focused on testing in the traditional sense, there has been a software process improvement initiative underway at the same time, and code inspections are now part of the maintenance process.
Further research could be conducted to answer the following questions: How can the current CF-188 system and software specifications be improved from a testability point of view? How can test cases be automatically generated from existing documentation? What improvements are possible in non-intrusive monitoring of timing and program execution within target processors? How can automation of test coverage measurement and dynamic coverage control best be provided?
Bibliography
[1] D.W. Campbell, Timing Analysis in Hard-Real-Time Systems with Application to the CF-188, (M.Eng. Thesis), Royal Military College of Canada, May 1991.
[2] Craigen, D. Gerhart, S., and Ralston, T., "Case Study: Darlington Nuclear Generating Station', IEEE Software, 30-32, Jan 1994.
[3] Deutsch, M.S., Software Verification and Validation - Realistic Project Approaches, Prentice-Hall Inc., Englewood Cliffs, NJ, 1982.
[4] Falardeau, Capt J.D.G., "A Study of Computer Aided Software Engineering (CASE) Tools used in the CF-18 Mission Computer (MC) Software Development Process", 4 Software Engineering Squadron Report, May 1995
[5] Fetzer, J.H., 'Program Verification: The Very Idea', Communications of the ACM, Vol 31, No. 9, 1048-1063, Sep 1988
[6] IEEE Standard Glossary of Software Engineering Terminology, ANSI/IEEE Standard 610.12-1990, IEEE Press, New York, 1990.
[7] Jahanian, F. and Mok, A.K., "Safety Analysis of Timing Properties in Real-Time Systems", IEEE Transactions on Software Engineering, Vol 12, No. 9, 890-904, Sep 1986.
[8] Leveson, N.B. and Harvey, P.R., "Analyzing Software Safety", IEEE Transactions on Software Engineering, Vol SE-9, No. 9, pp. 569-579, Sep 1983
[9] Mack, A.W., A Study of an Automated Test Capability for the CF-18 Avionics Operational Support Equipment, CF-18 Software Support Re-engineering Technology Improvement Team Report, Ottawa, May 1995
[10] Mack, A.W., Subsystem and System Integration Testing in Software Maintenance: A Case Study of the CF-188", M.Eng. Thesis, Royal Military College of Canada, May 1996
[11] Working Group Report, 'A Testing Philosophy for the CF-188 Operational Flight Programs', CF18 Weapon System Software Unit, Cold Lake, Alberta, Canada, 3 May 1990
[12] Young, Dr N.J.B, 'Automating the Testing of Software', AGARD Conference Proceedings No.343, 41-1 - 41-13, Oct 1983