Levels of Evaluation and Implications for Assessing Training Results - - PowerPoint PPT Presentation

Levels of Evaluation and Implications for Assessing Training

Results from the Secure LVC Advanced Training Environment Demonstration

J E S S I C A D . C O R T E Z , R E S E A R C H E R I / O P S Y C H O L O G I S T W A R F I G H T E R R E A D I N E S S R E S E A R C H D I V I S I O N , 2 9 A p r i l 2 0 2 0

Distribution A. Approved for public release; distribution unlimited. Case Number 88ABW-2018-4376

What is SLATE?

SLATE (Secure LVC [Live-Virtual-Constructive] Advanced Training Environment) IS A 40-MONTH ADVANCED TECHNOLOGY DEMONSTRATION

• Commanders of Air Force Materiel Command and Air Combat Command co-commissioned
• F-35 is Target Program
• Air Force Research Laboratory chartered to “vet the Critical Enabling Technologies” required for LVC

(“6 Pillars of SLATE”)

• Directed by: 711th Human Performance Wing, Airman Systems Directorate, Warfighter Readiness

Research Division

• F-15Es chosen as modified live players
• Aircraft with LVC-capable Operational Flight Program (OFP)
• F-16s as unmodified live players
• No change to aircraft OFP
• During mission, carry SLATE pod, demonstrating encrypted Air Combat Maneuvering Instrumentation
• U.S. Navy joined to augment modified live with F-18s as robust cross-platform players and additional pods

Case Number 88ABW-2018-4376

3

SLATE VIDEO

Video PA Cleared: 88ABW-2018-5681

“Training realism shall be maximized through use of the live training domain supplemented by integrated virtual and constructive capabilities.”

• -DoD Directive 1322.18

SLATE ATD Charter

Validate LVC infrastructure, interface control and data standards in a government owned, non-proprietary architecture Demonstrate Multiple Independent Levels of Security (MILS), encryption technology and rule sets Evaluate alternative large throughput, MILS capable encryptor, radio, data-link and 5G-ATW waveform Demonstrate Pod form factor Develop select weapon system OFP changes with vendors based upon clear guidelines, resulting in direct risk reduction and leave behind for Program of Record

Case Number 88ABW-2018-4376

Every effort being made for exportability across infrastructure

“THE 6 PILLARS OF SLATE”

THROUGH ALL 3 PHASES

1 2 3 4 5

6 Document training effectiveness benefits and

• pportunities with

integrated LVC capabilities

SLATE ATD: PILLAR SIX

Document Training Effectiveness Benefits and Opportunities With Integrated LVC Capabilities

Pillar Six Slides PA Cleared / Case Number 88ABW-2018-5740

6

Case Number 88ABW-2019-3130

Kirkpatrick’s Level 1 “reaction evaluation” measures whether aircrew are satisfied, motivated and interested in the relevance and applicability of LVC as a training and readiness construct.

SLATE ATD: Pillar Six

Pillar Six Slides PA Cleared / Case Number 88ABW-2018-5740

SLATE Phases – Pillar 6

Pillar Six Slides PA Cleared / Case Number 88ABW-2018-5740

• Phase I:
• Respondents: 8
• Entity Counts: 147 LVC weapons events
• F-15E ML, F-16 and F/A-18 sims
• LVC-Enabled Sorties Flown: 12
• Phase II:
• Respondents: 11
• Entity Counts: 160 LVC weapons events
• F-15E ML, F-16 UL, F-16 sims
• LVC-Enabled Sorties Flown: 14
• Phase III:
• Respondents: 35
• Entity Counts: 335 LVC weapons events
• F-15E ML, F/A-18E/F/G ML, F-16 and F/A-18 sims
• LVC-Enabled Sorties Flown: 71

Photos by RHA

SLATE Focus Point: Training Realism

Pillar Six Slides PA Cleared / Case Number 88ABW-2018-5740

Participant responses are a representative subset of a fuller set of comments

RESULTS

THROUGH ALL 3 PHASES

IMPLICATIONS

Participants’ Feedback Illustrates Aircrew:

• Look for realism that provides instant and

correct feedback

• Value a technology-rich innovative

learning environment

• Noted the careful planning of scenarios

By providing powerful new ways to engage aircrew in operational and strategic-level missions will increase their capacity to operate effectively and thrive under any environmental condition. Participant Responses: Positives

• “Overall LVC integration. The ability to have 3 types of entities
• n the network gives multiple opportunities and fighting

against ‘realistic’ threats.”

• “Flying in formation with virtual and constructive bros. ‘Seeing’

them in your sensors and being visual, although positions were not accurate at times, is game changing. Allows the

• pportunity to train as part of a larger force package.”

Participant Responses: Things to Work On

• “The single frequency war made communications nearly
• impossible. Severely degraded tactical operations and the

ability to focus on the fight.”

• “The Link-16 & SLATE pictures are not correlating in ALL cases

which causes confusion & possible poor tactical decisions.”

SLATE Focus Point: System Capabilities

Pillar Six Slides PA Cleared / Case Number 88ABW-2018-5740

Participant responses are a representative subset of a fuller set of comments

RESULTS

THROUGH ALL 3 PHASES

IMPLICATIONS

Participants’ Feedback Indicated:

• The scope and application of interactions within

the system must evolve in order to increase the system’s value

• The resources available to meet the challenges of

tomorrow and indicated they would welcome an extensive model library The system must meet the demands of a changing environment in order to increase the desirability of the LVC enterprise architecture. To increase its value, the system must evolve. Participant Responses: Positives

• "Mort cue. Real-time feedback helps highlight our mistakes or

validates our tactics and helps the learning sink in.“

• “Seeing fake entities on S/A radar worked fairly well seemed

more stable worked as they should with symbology separating what was real and fake."

Participant Responses: Things to Work On

• “Having constructive aircraft outside airspace does not help

tactical training with heavy civilian traffic because we do not fight aircraft outside the airspace. This needs some heavy admin discussion.”

• “More in-aircraft troubleshooting capabilities or instruction.”

SLATE Focus Point: Future Capabilities

Pillar Six Slides PA Cleared / Case Number 88ABW-2018-5740

Participant responses are a representative subset of a fuller set of comments

RESULTS

THROUGH ALL 3 PHASES

IMPLICATIONS

Participants Noted:

• The knowledge management system’s ability to

focus on roles and missions, tactics and situational awareness

• The system’s ability to determine performance on

measures divided into quality domains A firm grasp on the broad array of factors influencing aircrews’ performance across process, quality and efficiency measures translates into actionable ideas

• r practical innovations.

Participant Responses: Positives

• "LFE amounts of red air that can't be replicated at home station.“
• "Getting multiple threats to appear without generating all the

aircraft needed was amazing.“

• “8v8 line with another~8 constructive threats- can't fight that

at home.”

Participant Responses: Things to Work On

• "The ADCP software required to run SLATE appears to result in

abnormalities in normal avionics operations. Degraded the capability to focus on tactics by increasing aircrew workload and troubleshooting.“

• “Pods are too user intensive requiring more cross check time to

ensure proper operation precluding full up tactical maneuvering.”

SLATE Focus Point: User Satisfaction

Pillar Six Slides PA Cleared / Case Number 88ABW-2018-5740

Participant responses are a representative subset of a fuller set of comments

RESULTS

THROUGH ALL 3 PHASES

IMPLICATIONS

Participants’ Feedback on Analysis of the System:

• Focused on examining actions and procedures

that provide realism and produce more highly skilled aircrew

• Made suggestions regarding the development
• f adaptive thinking and improvements of the

LVC environment By increasing the understanding of tactical relationships, aircrew can modify their real-time thought processes using the results of new or better experiences provided by the LVC readiness construct and technological capability. Participant Responses: Positives

• "Getting multiple threats appear without generating all

the [Live] aircraft needed was amazing.“

• "Nice use of 3-D playback for debrief. Outstanding debrief

tool for surface to air threat engagements.“

• “Overall experience is really cool-can provide good

repetitions and learning.”

Participant Responses: Things to Work On

• “Having constructive aircraft outside airspace does not help

tactical training with heavy civilian traffic because we do not fight aircraft outside the airspace. This needs some heavy admin discussion.”

• “More in-aircraft troubleshooting capabilities or instruction.”

SLATE Focus Point: System Capabilities

Pillar Six Slides PA Cleared / Case Number 88ABW-2018-5740

Participant responses are a representative subset of a fuller set of comments

RESULTS

THROUGH ALL 3 PHASES

IMPLICATIONS

Participants’ Feedback Noted:

• The scope and application of interactions within

the system

• The resources available to meet the challenges of

tomorrow and indicated they would welcome an extensive model library The system must meet the demands of a changing environment in order to increase the desirability of the LVC enterprise architecture. To increase its value, the system must evolve. Participant Responses: Positives

• "Mort cue. Real-time feedback helps highlight our mistakes or

validates our tactics and helps the learning sink in.“

• “Seeing fake entities on S/A radar worked fairly well seemed

more stable worked as they should with symbology separating what was real and fake."

Participant Responses: Things to Work On

• “Having constructive aircraft outside airspace does not help

tactical training with heavy civilian traffic because we do not fight aircraft outside the airspace. This needs some heavy admin discussion.”

• “More in-aircraft troubleshooting capabilities or instruction.”

Pillar Six: Pilot Feedback (Positives)

Pillar Six Slides PA Cleared / Case Number 88ABW-2018-5740

Pillar Six: Pilot Feedback (Things to Work On)

Pillar Six Slides PA Cleared / Case Number 88ABW-2018-5740

SLATE ATD: PILLAR SIX

Conclusion and Take Away

Pillar Six Slides PA Cleared / Case Number 88ABW-2018-5740

17

Jessica Cortez, Ph.D. Researcher I/O Psychologist Wright-Patterson AFB, OH 1.937.938.4013 jessica.cortez.3.ctr@us.af.mil jessica.cortez@cubic.com