Paul Kingston, IVEX
Captain Tom Peters, Delta Air Lines
Chris Lehman, Atlantis Aerospace Corporation
Edward M. Boothe, Consultant, Flight Simulation and Training

Gerald D. Gibb, Steven Hampton, John A. Wise and John C. Wolf,
Embry-Riddle Aeronautical University

ABSTRACT

An Atlantis-sponsored analysis has indicated that the most cost-effective program would utilize Level 5 to 6 FTDs for a significant portion of a conventional air carrier training program. A demonstration project to validate the conclusion of the analysis was conducted at the Delta Air Lines Training Center. Groups of pilots from Delta Air Lines and from Embry-Riddle Aeronautical University were trained using both the conventional all simulator curriculum and a curriculum in which a Level 6 FTD with a visual system was substituted for a significant portion of the training. Performance measures were obtained using simulator generated data, evaluations conducted by Aircrew Program Designees (APDs) and independent check pilots from Embry-Riddle. The results from standard type-rating check rides indicate that comparatively few significant differences exist between the two training curricula. The results indicate that FTDs properly integrated with a simulator in an approved training program can produce pilot performance similar to those obtained from an all-simulator program. Further efforts should be expended to investigate assignment of training tasks to the appropriate level device. Since the initial and operating costs of a full flight simulator are significantly higher than that of an FTD, these efforts would be warranted.

INTRODUCTION

Air carriers, large and small, have long sought a low cost, highly effective means of pilot training and certification (licensing). Aviation authorities have been amenable to lower cost approaches as long as there was no compromise of training effectiveness or certification standards. The certification standards have required that pilots demonstrate proficiency in either an airplane or an advanced simulator when the simulator was used in an approved training program. Past stimuli to utilize part-task trainers in a training program without also using an advanced simulator have been unsuccessful. Operators have taken the position that if the airplane has to be used at all, the entire training may as well be done in it. This attitude and the cost of FTDs has been a continuing barrier to gaining the advantages offered by FTDs. Attitudes are changing, however, and more interest in the use of FTDs, especially in conjunction with advanced simulators, is becoming evident. The addition of a visual system to an FTD offers the potential of even greater utilization and effectiveness. Visual FTDs will very likely prove to be an excellent complement to a simulator flight training program or flight training programs using airplanes.

Background

In the mid-1980s, the Federal Aviation Administration (FAA) worked with the Regional Airline Association (RAA) to define a part-task device to meet RAA needs and in May 1987 published Advisory Circular (AC) 120-45 entitled "Advanced Training Devices (Airplane Only) Evaluation and Qualification." Accompanying AC120-45 was AC120-46 "Use of Advanced Training Devices", which delineated the flight training credits that could be derived from an Advanced Training Device (ATD). There were no ATDs entered into service because operators considered it too expensive for the benefits available and pilot certification would had to have been completed in an aircraft. Consequently, total training in airplanes continued among small operators and, unfortunately, so did training accidents.

In 1990 and 1991, the FAA defined seven flight training device levels to provide more flexibility for operators to utilize FTDs of suitable complexity and capability in their training programs. Slowly, the FTD has been "catching on" and there are now a number of the devices in service in flight training programs; about a dozen in the United States and about twice that worldwide. There are more in service in ground training programs, where the devices usually do not require qualification by the cognizant regulatory authority. FTDs have not, however, served as effectively and efficiently as they can in air carrier flight training and certification programs. In many cases, especially among large operators, the entire flight training program is done in an advanced simulator regardless of the training task and without regard to the minimum level of device required to support the particular training task. It appears to be often overlooked that an advanced simulator training program approved in accordance with Federal Aviation Regulation (FAR) 121, Appendix H "Advanced Simulation Plan" (Ref 1.) does not require total use of advanced simulators, but does require that the operator specify what simulators and training devices will be used.

Requirements Analysis

As late as 1988, John Rolfe stated "the psychological truth about the principles that make simulator training genuinely effective is unknown" (Ref. 2). There seems to be no new evidence since that time to explain the effectiveness of simulation. Despite this unknown, various documents such as regulations, advisory circular's and handbooks permit the assignment of particular training events and tasks to specific levels of training devices or simulators. For examples, FAR 121 Appendices E and F and FAA Order 8400.10 "Air Transportation Operations Inspector's Handbook" (Ref 3.) provide tables which assign training and certification tasks to specified levels of flight simulators and flight training devices. As implied above, there is no identified scientific data base to support these assignments. However, there is a forty plus years experience base to support simulator and FTD use, and there is experiential evidence that more training and checking could be allocated to lower level devices than is currently being done. None of the guidance and regulatory documents addresses training or pilot certification credits for the combination of an FTD and a visual system. FTDs are treated as nonvisual devices and simulators are treated as devices which do have visual systems. Although FTDs with a visual system are not given credit in any of the tables, AC 120-45A "Airplane Flight Training Device qualification" does acknowledge that such a possibility exists.

To identify the most effective and efficient combination of training equipment for a typical air carrier training program, Atlantis Aerospace Corporation undertook a detailed analysis in 1993. The study addressed transition training and a Boeing 737-400/500 curriculum. In the analysis, comparisons were made of many different combinations of flight management and guidance systems trainers (FMGST), levels 4, 5, and 6 FTDs with and without visual systems, and a full flight simulator (FFS). The objective was to off-load the FFS to the extent practicable to provide overall equivalent training as the lowest cost. The focus was on the capability of today's FTDs to support selected training tasks rather than their technical description or what they are permitted to do based on today's regulations and guidance.

The conclusion of the analysis was that the most cost effective training device is a"Level 5⁺⁺ FTD"; an FTD that is between FAA levels 5 and 6. The"Level 5⁺⁺ FTD" incorporates full functional systems, controls and displays, an enclosed cockpit, sounds, representative control loading, and level 5 aero modelling. The most cost effective training solution was to use an FMGST, "Visual Level 5⁺⁺ FTD" and a FFS. Since the report of this analysis is a proprietary Atlantis document, it is not available. Suffice it to say, however, that the cost savings using the above combination of devices to obtain equivalently effective training are most significant. There is no compromise of training quality of effectiveness from that obtained with the full use of an FFS throughout the curriculum.

The Atlantis report determined the most efficient and effective combination of devices for a transition training curriculum, based on analysis. It remained to be shown that the conclusion was viable in a "real" flight training program. To this end, four organizations combined efforts in early 1994 to experimentally demonstrate the conclusion from the Atlantis analysis. Atlantis Aerospace Corporation, IVEX Visual Systems, Embry-Riddle Aeronautical University, and Delta Air Lines combined talents and resources to demonstrate the use and advantage of a visual FTD in an air carrier training program. SEOS Displays Ltd. provided a display system for the experiment. The experiment involved control and test groups of pilots so that results of the two could be compared.

Purpose

The objective of the demonstration was to determine whether pilot training costs could be reduced, as indicated by the Atlantis study, while maintaining the integrity and quality of the training program. Intrinsic in this purpose is the requirement to not compromise training effectiveness or certification standards and not adversely affect aviation safety. There can, however, be more efficient flight simulator use by doing only necessary tasks in the simulator and doing other tasks in less costly devices. The key is to assign each task or event to the device which provides the necessary cues and environment for that task, but to not train in a more sophisticated device than necessary. It must be recognized, though, that it is not sufficient for the pilot to merely accomplish the task. He or she must accomplish the task with the same control strategy and similar control inputs to those that would be used in the respective aircraft. The objective is for the FTD or simulator to provide the same pilot stimulus for the task that the aircraft would provide. One must admit, however, that there is insufficient scientific data relating pilot stimulus and response to necessary cues to determine the correct device complexity required for each task.

The need for motion can be based on so-called "disturbance" inputs which derive from unusual events or disturbances of the flight path, as opposed to pilot-initiated deviations of the flight path. Events which are known to be independent of motion stimulus can be trained, and for that matter checked, in an FTD. Training of some of the events may benefit from a visual system. In this demonstration the visual system was used throughout the flight training portion of the program. There was no intent to identify which tasks would benefit by visual cues and which would not.

Current FTD and simulator task assignment is based primarily on realism. The issue is often how realistically does the device represent the total environment, not just how realistically it represents the given event or task. Realism is certainly an acceptable criteria for success, but it may lead to over specification of the needed training medium. However, since there is no data base except experience that identifies the cues required for given tasks, there is as yet, no other available criteria. The demonstration described in this paper does not attempt to relate pilot response to cues, per se, but is to show that many tasks can be off-loaded from the simulator to a simpler device. Hopefully the results will stimulate further study into cue analysis.

METHOD

Subjects

A total of forty-eight pilots, twenty-four volunteer pilots from Embry-Riddle Aeronautical University and twenty-four volunteer furloughed pilots from Delta Air Lines, participated. In each case, half the pilots were in a control group and the other half in a test group. The crew training concept was used and all pilots in the captain position possessed, or were eligible for, an airline transport pilot certificate.

The mean flight experience of the Embry-Riddle pilots was 1,300 hours, with a range of 800 to 10,000 hours. Their mean age was 26, with a range of 22 to 43 years of age. Each pilot held at least a commercial certificate with a multi-engine and instrument rating. The Embry-Riddle pilots were paired so as to avoid having two low experience pilots together as a crew, neither of which might possess the experience requirements for an air transport rating. After pairing, the subjects were randomly assigned to the control and test groups.

The mean flight experience of the Delta pilots was 3,700 hours, with a range of 1,800 to 8,000 hours. Their mean age was 36.5, with a range of 29 to 49 years of age. Each pilot held at least a commercial certificate with a multi-engine and instrument rating. The Delta subjects were furloughed pilots of varying experience, all of whom have previously served in a line capacity. They too were paired and then randomly assigned to the control and test groups.

Apparatus

A Level 6 FTD with a single-channel, two-window visual system (no Level 5 FTD was available) and a Level D flight simulator were used for the experiment. However, a Level 6 is actually more representative of a"Level 5⁺⁺ FTD" than is a Level 5. The results are of course applicable to the specific device used or to devices of similar technical description, especially since the Level 6 device is programmed with a specific airplane aerodynamic model which exceeds the Level 5 requirement. The simulator was a qualified device normally used in the Delta flight training program. The FTD with visual system was qualified by the Federal Aviation Administration (FAA) for use in the flight training demonstration program. FAA qualification is not required for devices used in ground training programs, and consequently the Level 6 FTD formerly used only in the ground training portion of Delta's program had not been previously qualified. All objective pilot performance data were collected using the data collection capabilities of the simulator.

Procedure

All subjects completed the normal ten day MD-88 ground school which was an integral part of the initial training program. The ground training program was unaltered for the demonstration program and utilized the Level 6 FTD, but did not use the visual system.

The pilots in the control groups received flight training in accordance with the Delta Air Lines standard all simulator (Level D) initial training program. The pilots in the test groups were trained in a program in which the visual FTD was used in lieu of the simulator for the first five of the nine training and certification days in the program. Some tasks which require a simulator were learned and practised in the FTD, but were then repeated in the simulator in the latter part of the training program.

The check rides were administered by an Aircrew Program Designee (APD) who did not know whether the pilot was trained in the all-simulator program or in the combined FTD and simulator program. All subjects completed the check ride in the MD-88 Level D flight simulator and were evaluated using standard performance criteria required by the FAA-approved Delta Airlines training program. Pass or fail was determined solely by the APD. Any pilot trainee needing more than the allotted time of the training program was given one additional day of training and a second check ride.

There was a second observer from the Embry-Riddle staff, whose function was data collection. The observer, a senior check pilot, completed a detailed special performance evaluation form during each check ride. The analysis of the data from the special performance evaluation complemented objective data collected using the simulator computer system. The second observer also managed the collection of the objective data. This involved initializing the computer for data collection before each maneuver.

Simulator-generated data was used in this study as a means to objectively assess and quantify pilot performance while mitigating evaluator biases. The maneuvers and performance parameters were selected based on meeting three criteria: a) the maneuver was a required task in the check ride, b) performance could be assessed by capturing relevant parameters, and c) a clear standard of target performance could be developed and used for comparison. All required tasks in the check ride could not be assessed since a clear reference point was unobtainable or because difficulty in identifying an initialization/termination point was encountered. Therefore, only maneuvers and parameters that could be precisely standardized across all check rides were used. No attempt was made to sample all check ride maneuvers or their components. The six maneuvers sampled and their associated performance parameters were:

RESULTS

All forty-eight pilots successfully completed the check ride eventually, although three initially failed, two in the control group and one in the test group. One pilot failed the oral examination but was allowed to continue the training and subsequently completed the check ride successfully. The APDs using a standard satisfactory/unsatisfactory evaluation form reported little or no differences between the pilots trained in the all-simulator program and the combined FTD and simulator program.

The data from the Embry-Riddle independent observer consisted of rating sheets with simple dichotomous scores. Pilot performance was evaluated only as to whether or not a procedure, checklist item, or performance item was successfully completed within the parameters of the ATP practical test standards, which are identical to the performance required on a rating ride. Rating sheets were used to assess eight flight maneuvers: a) precision approaches, b) visual approaches, c) approach to stalls, d) non-precision approaches, e) normal takeoffs, f) rejected takeoffs, g) V₁ cuts, and h) steep turns. Flight maneuvers were evaluated in the Level D simulator at the conclusion of the training program.

The frequency of missed items was too small to analyze each of the eight maneuvers independently. Consequently these data were combined across maneuvers to develop composite ratings. Non-parametric tests were performed on these data for each of the Embry-Riddle and Delta pilot groups, and the Embry-Riddle and Delta groups combined. No significant differences were found between training groups for Embry-Riddle pilots, Delta pilots, or the two groups combined.

The important performance criteria for this study, however, were the simulator captured data. Six sampled maneuvers were analyzed to determine if there were significant differences among pilots in critical flight performance measures. In each case Group 1 represents the control group (full flight simulator throughout training) and Group 2 represents the test group (combined FTD and full simulator). The reader is cautioned that complete performance data is not available in many instances as a result of simulator problems in capturing and transferring data. All analyses are conducted assuming unequal sample variances and using the probabilities for two-tailed tests.

The data reported below have been organized by maneuver. In all cases where there are no significant differences between Embry-Riddle and Delta pilots within training groups (i.e. control and test), the data have been collapsed to increase sample size. Results of these comparisons are not presented here. RMS values were obtained by squaring each deviation value, summing the resulting squares, dividing by the number of samples and taking the square root of the result for each individuals' performance. No distinction is made between first officers and captains.

1) Steep Turns: The mean RMS values for angle of bank (AOB), airspeed, and altitude deviations are presented in Tables 1 and 2. As can be observed from Table 1, there is a significant difference between training groups for AOB deviations. The control group had demonstrated significantly less deviation in maintaining a constant AOB. These data represent the Embry-Riddle and Delta groups combined, as there were no significant differences between these groups within each training group. However, when training groups are examined separately, there is also a significant difference (t=2.33, df=16, p<.03) between the Delta control and test groups. Those trained in the combined FTD/Simulator program had significantly greater AOB deviations (M=2.66, SD=1.09) from the 45^o standard than pilots trained completely in the simulator (M=1.80, SD=.69).

Regarding altitude and airspeed deviations from entry configuration of the steep turn maneuver, no significant differences were found between training groups within either the Embry-Riddle or Delta pilot populations. Furthermore, since no significant differences were found for either performance criteria between Embry-Riddle and Delta training groups, within each training group, the data were collapsed. In examining the data in Table 2 it is clear that there are no significant differences between training groups for either airspeed or altitude deviations.

Table 1 - Mean RMS Values for AOB Deviations: Steep Turns

where n = number of pilots, M = mean deviation, SD = standard deviation, t = testing value,
df = degrees of freedom, p = level of probability

Table 2 - Mean RMS Values for Airspeed and Altitude Deviations During Steep Turn Maneuvers

2) Rejected Takeoffs: The mean RMS values for the deviation from runway heading and the total distance to stop in feet are reported in Table 3. These data are for the Embry-Riddle and Delta groups combined, since there were no significant differences between these pilot groups within each training group for either performance measure. In addition, no differences between training groups were found within the Delta or Embry-Riddle groups for the two performance measures.

Distance to stop is derived from the distance travelled in feet at the point of engine failure to the point of zero forward ground speed. Heading deviations are calculated over the same parameters. Data is not available for several pilots, as the data collection subroutines were not initialized before the maneuver began. This is due to the Embry-Riddle observer not knowing whether a take-off roll would progress into a normal or aborted takeoff.

Table 3 - Mean RMS Values for Runway Heading Deviation and Distance to Stop

3) Engine Failure at V₁: The mean RMS values for heading deviation are presented in Table 4. There are no significant differences between training groups regarding this performance measure. The data were collapsed across Embry-Riddle and Delta groups for each training group, since there were no significant between groups differences within each training group.

Table 4 - Mean RMS Values for Heading Deviation

4) ILS Approach: The mean RMS values for horizontal deviations from the localizer and vertical deviations from the glideslope during the ILS approach are presented in Table 5. In comparing performance between training groups within Embry-Riddle and Delta groups separately, no differences were found for either vertical or horizontal deviations from the ILS course (results not shown). Further analyses indicated that there were no significant differences between Embry-Riddle and Delta training groups within each training group for the two performance measures. Consequently, the data for Embry-Riddle and Delta training groups were collapsed for each training group. These data are presented in Table 5 and yield no significant differences between training groups.

Table 5 - Mean RMS Values for Vertical and Horizontal ILS Deviations

5) Approach to Stall: The mean number of feet of altitude lost during the time from stall onset to stall recovery are reported in Table 6. As discussed earlier, secondary stalls during the recovery were treated as a continuation of the initial stall. The altitude lost between initial altitude and lowest altitude attained is the measure of performance, rather than cumulative altitude lost over multiple stall profiles.

No significant differences were found within Embry-Riddle or Delta groups between training groups. The data reported in Table 6 is from Embry-Riddle and Delta groups combined within each training group, since no significant differences were found between these training groups within each training group.

Table 6 - Mean Feet of Altitude Lost During Approach to Stall Maneuver

As can be seen in the above table, performance for the combined FTD and full flight simulator training group is better than that for the all full flight simulator group. This, however, is not a significant difference (p<.33).

6) Visual Approach: This maneuver was difficult to assess, as there were no standard profiles that could be developed for comparison, and the destination airport could vary from one check ride to the next. Consequently, distance in feet from runway centerline at the moment of touchdown is the only available performance measure.

There were no significant differences between training groups on this performance criterion for the Embry-Riddle group. Within the Delta training group, those in the test training group performed significantly better (6.85 mean feet from centerline) than those in the control group (14.05 mean feet from centerline); t=2.38, df=27, p<.02.

Data for the Embry-Riddle and Delta pilot groups could not be collapsed for analysis. Although there were no significant differences between Embry-Riddle and Delta control groups, the Delta test group performed at a significantly better level (t=2.45, df=18, p<.02) than the Embry-Riddle test group. Mean distance from centerline for the Delta group was 6.85 feet (SD=5.46, n=17) compared to 23.80 feet (SD=28.7, n=18) for the Embry-Riddle group.

CONCLUSIONS

The data given here are limited to the sampling of a few flight parameters of a select group of maneuvers from a full flight simulator during a type rating check ride. As such, they represent only a portion of flight performance assessment. However, the performance criteria selected were chosen solely on the basis of the capability of the simulator to capture these data and our ability to identify and distinguish among maneuvers. Consequently, the selected performance criteria are unbiased and objectively quantifiable. A few tasks, such as windshear training, that are critical to flight training but were not part of the flight check, were not evaluated. Some tasks are required by regulation to be trained in a simulator and should continue to be taught in a full flight simulator. The data given here demonstrate, however, that FTDs are a very effective complement to a full flight simulator in an airline training program.

These results provide strong support to transfer a significant portion of an air carrier's training program from a full flight simulator to an FTD. Based on the limited data presented here, it is inappropriate to provide complete conclusions about the capabilities of FTDs to supplement or replace simulators in the air carrier training environment. Nonetheless, the data does suggest that for some maneuvers FTDs are an acceptable substitute and provide a level of training and performance that is not significantly different from that acquired using full flight simulators.

It was found that four of the six selected maneuvers evaluated using the simulator-generated data revealed no significant differences for training group. In one case, it was apparent that the FTD/simulator combined training group actually performed at a higher level of proficiency than the control group. The total distance from the runway centerline on the visual approach task indicated better performance for the test group. In examining all the performance criteria collected and analyzed in this study there is a trend, although not significant, of better performance for the FTD test group than the control group.

The other significant difference found between training groups was for bank angle deviation from 45^o AOB during the steep turn maneuver. Training conducted completely within the simulator appears to result in better performance as assessed by this criterion. Perhaps the motion cues available under this training program assist in learning and performing this maneuver at a higher level of proficiency. It should be noted, however, that there were no significant differences between training groups for the other two performance criteria for this maneuver. This may indicate that the observed difference was a chance result, or that airspeed and altitude hold are not as contingent on motion. Further experimentation is required to investigate the effect of motion cues on pilot training and checking.

In summary, this research has examined the feasibility in an air carrier training program of transferring tasks from full flight simulators to FTDs with integrated visual systems. The data seem to support that at least some tasks can be transferred without a decrease in performance proficiency. Further work is required, however, to determine which tasks are more suited for training in an FTD rather than a simulator, and which tasks could be relegated solely to a visual FTD with no requirement to repeat the tasks in a simulator.

REFERENCES

1. Federal Aviation Regulation Part 121, Appendix H - Advance Simulation Plan, June, 1980.

2. Rolfe, J.M., Fidelity: An Evaluation of the Concept and the Implications for the Design, Procurement, Operation, and Evaluation of Flight Simulators, 1988.

3. Air Transportation Operations Inspector's Handbook, FAA Order 8400.10. Federal Aviation Administration, June 30, 1991.