Taking Immersive Leap in Training of Landing Signal Officers Clay - - PowerPoint PPT Presentation

Taking Immersive Leap in Training

• f Landing Signal Officers

Amela ¡Sadagic ¡ Naval ¡Postgraduate ¡School ¡

Clay ¡(Larry) ¡“Sea ¡Fog” ¡Greunke ¡ US ¡Navy ¡

Overview

• Problem Space and Motivation
• Background
• Task Analysis
• Survey of User Domain
• Prototype System and Development
• Feasibility Study
• Discussion and Future Work

2 ¡

• Task: Facilitate the "safe and expeditious recovery" of

Naval aircraft aboard aircraft carriers.

• Time on task: 20 seconds between the pilot's start

position until landing, to guide pilot toward an acceptable position on the carrier.

• Originally just a single LSO - evolved into a team
• Main Roles:

– Deck Calling LSO – Controlling LSO – Backup LSO – CAG LSO

3 ¡

Who are the LSOs?

LSOs through the History

4 ¡

Problem Space & Motivation

5 ¡

• Issue: 108 landing-related mishaps between

2005 – Jul 2015. 99 involved LSOs performance.

• Training on-the-job: Majority of training for

junior LSOs occurs while a carrier is deployed

• ut to sea.
• Other training: 2 weeks of Initial Formal Ground

Training (IFGT) – an LSO spends 6 hours working with the large LSO Trainer (2H111). Agreement in domain: Additional training

• pportunities are very much needed.

Current training Solution:

2H111 Simulator

6 ¡

• LSO School in Oceana, VA
• Two stories tall room
• Several large screens +

physical interfaces / actual instruments

Our Objective

7 ¡

• Fill the training gap - Create conditions for

unlimited number of training opportunities unrestricted by location and time.

• Use Commercial off-the-shelf (COTS)

components to create a light-weight, portable, fully virtual system that replicates the aircraft carrier environment and allows LSOs to practice as individual or as a team.

Background

8 ¡

• Objective of any training system: Positive

transfer of training (skills & knowledge) [1][2][3][4]

• Issues that can get in the way in VR:

– Technical characteristics of VR system – Judgment of distance in VR [5][6][7][8] – Cybersickness [9][10][11][12] – Appropriateness of training approaches – Richness of scenarios – Length of exposure

• All elements were reviewed before designing

new LSO trainer.

Design Requirements

9 ¡

• 1. Support all sensory cues (system feedback)

and user input essential for training of LSOs à Perform task analysis vetted by Subject Matter Experts (SMEs).

• 2. Support all major capabilities of 2H111.
• 3. Acquire a thorough understanding of users'

training needs, domain view of benefits and shortcomings of training with 2H111. à Address shortcomings and bring additional capabilities non-existent in 2H111.

Task Analysis

10 ¡

• Analyzed the three most important positions on

the LSO team:

• 1. Controlling LSO,
• 2. Backup LSO, and
• 3. Deck Caller LSO
• Results vetted by SMEs from the LSO School
• Major difference between 2H111 and our

system: Absence of haptic feedback i.e. lack of physical instrumentation that 2H111 has (LSO can press real buttons and switches)

Survey of User Domain

11 ¡

• Our objective: Acquire an accurate

understanding about:

• 1. Current state of training practices in the LSO

community,

• 2. LSOs’ perception of elements of training with the

2H111 simulator – (a) benefits and good characteristics, and (b) negative issues and

• bstacles in training practice.
• Built full IRB documentation.
• Executed online survey using LimeSurvey.
• 35 LSOs responded.

Survey of User Domain:

Illustration of Results

12 ¡

Skills, knowledge and concepts qualified as most difficult to acquire by an LSO (more results in [13])

à This plays a role when deciding what to emphasize in new training system.

• 1. Eye calibration: Determining the aircraft position as it

relates to the ideal glideslope angle

• 2. Judgment: LSO's situational awareness

Design Goals for Prototype System

13 ¡

• 1. Support all major capabilities and training
• bjectives currently supported by the 2H111.
• 2. Use only COTS components - remain a

lightweight training system.

• 3. Invest considerable efforts to minimize potential

for cybersickness (extended exposure to training solution needed to acquire LSOs skills).

• 4. Integrate a variety of typical COTS input
• ptions for trainee and instructor.

Prototype System & Development:

System Architecture

14 ¡

Unity Application Virtual Environment Voice Recognition App. Xbox Controller Immersed LSO (a trainee) Xbox Controller Instructor LSO Keyboard Headphones Leap Motion Controller Oculus DK2 Head tracking Display Microphone

15 ¡

• Laptop platform: Alienware 17 R2 with Intel

Core i7-4980HQ CPU @ 2.80 GHz, 16 GB RAM and GeForce GTX 980M GPU

• Input devices: Two Xbox Controllers (one for a

trainee LSO + one for instructor), Leap Motion Controller, microphone

• Output devices: Oculus DK2, headphones
• Oculus DK2: 960 x 1080 per eye, max refresh rate of 75

Hz, field of view (FOV) 100 degrees, and weight of device: .97 lbs.

• Dev. Tools: Unity, Blender, Audacity

Prototype System & Development:

System Architecture & Tools

16 ¡

Prototype System & Development:

3D Assets

Feasibility Study (1 of 2)

17 ¡

• 1. Technical System Performance

Measurement: Frame rate is 37 – 60 FPS

• 2. Cross Comparison with 2H111:
• a. LSOs VR prototype portable, unlike 2H111
• b. 2H111 has restricted FOV
• c. 2H111 supports team training (all positions need

to be present), and LSOs VR prototype can support training of individual positions (other positions could be simulated - agents);

• d. 3rd person view possible in LSO prototype -

enable novel learning points

Feasibility Study (2 of 2)

18 ¡

• 3. Informal Demo Feedback (13 LSOs)
• 1. Speech Recognition System
• 2. Visual Cues:

– Daytime and nighttime aircraft recoveries – Signals from Arresting Gear Officers (AGO) visible – Strobe light patterns to be fixed – Add “day ID light” on the Super Hornet variants

• 3. Landing Signal Officer Display System (LSODS)
• 4. Manually Operated Visual Landing Aid System

(MOVLAS)

Feasibility Study

19 ¡

Daytime & nighttime recovery

MOVLAS & LSODS in 2H111 and VR

Feasibility Study

20 ¡

Portion of Deck Caller LSO’s perspective with Arresting Gear Officer being visible on the flight deck

Going Beyond 2H111 Capabilities

21 ¡

‘Magic book’ AR using a smartphone à added capability for top-down view of landing operation

Going Beyond 2H111 Capabilities

22 ¡

• Federation of LSO trainers for team training
• Utilizing VR to prototype and test new capabilities

(e.g. AR concepts) - https://youtu.be/d79oT3PxT2U

Future work

23 ¡

• 1. Development of additional capabilities
• 2. System validation (models and data)
• 3. Development of federation of simulations/

trainers

• 4. Formal usability study (GUI)
• 5. Training effectiveness study
• 6. System re-design
• 7. Transfer of training study

Conclusion

24 ¡

• 1. Major 2H111 training objectives supported
• 2. Prototype succeeded to remain lightweight – all

components COTS solutions

• 3. Achieved high frame rate – all LSOs operations

were possible (note: additional efforts could still be invested to improve frame rate in complex scenes)

• 4. Supported a range of COTS input devices and

interactive modalities. Time make immersive leap has come!

References

[1] R.P. Darken, and W.P. Banker. Navigating in Natural Environments: A Virtual Environment Training Transfer Study. Virtual Reality Annual International Symposium,

• 1998. Proceedings., IEEE 1998.

[2] N.E. Seymour, A. G. Gallagher, S.A. Roman, M.K. O’Brien, V.K. Bansal, D.K. Andersen, and R.M. Satava. Virtual Reality Training Improves Operating Room Performance. Annals of Surgery, Vol. 236, No. 4, 2002 Oct, pages 458–464. [3] J.M. Nolan, and J.M. Jones. Games For Training: Leveraging Commercial Off The Shelf Multiplayer Gaming Software For Infantry Squad Collective Training, Master Thesis, Naval Postgraduate School, 2005. [4] B. Brown. A Training Transfer Study of Simulation Games. Master thesis, Naval Postgraduate School, 2010. [5] P. Willemsen, M.B. Colton, S.H. Creem-Regehr, and W.B. Thompson. The effects of head-mounted display mechanics on distance judgments in virtual environments. APGV '04 Proceedings of the 1st Symposium on Applied perception in graphics and visualization, 2004. [6] W.B. Thompson, P. Willemsen, A.A. Gooch, S.H. Creem-Regehr, J.M. Loomis, and A.C.

• Beall. Journal Presence: Teleoperators and Virtual Environments, Vol. 13, No. 5, Oct.

2004, pages 560 – 571.

25 ¡

References

[7] J.M. Knapp, and J.M. Loomis. Limited Field of View of Head-Mounted Displays Is Not the Cause of Distance Underestimation in Virtual Environments. Journal Presence: Teleoperators and Virtual Environments, Vol. 13, No. 5, Oct. 2004, pages 572-577. [8] I.V. Piryankova, S. de la Rosa, U. Kloos, H.H. Bülthoff, and B.J. Mohler. Egocentric Distance Perception in Large Screen Immersive Displays. Journal Displays, Vol. 34,

• No. 2, 2013, pages 153-164.

[9] K.M. Stanney, and R.S. Kennedy. The Psychometrics of Cybersickness. Communications of the ACM, Vol. 40, No. 8, Aug. 1997, pages 66-68. [10] K.M. Stanney, R.S. Kennedy, and J.M. Drexler. Cybersickness Is Not Simulator

• Sickness. Proceedings of the Human Factors and Ergonomics Society 41st Annual

Meeting, l997. [11] R. Patterson, M. Winterbottom, and B. Pierce. Perceptual Issues in the Use of Head- mounted Visual Displays. Human Factors: Journal of Human Factors and Ergonomics Society, Vol. 48, No. 3, 2006, pages 555–573. [12] J. J. LaViola Jr. A Discussion of Cybersickness in Virtual Environments. ACM SIGCHI Bulletin, Vol. 32, No. 1, 2000, pages 47–56. [13] L. Greunke, "“Charlie,” Development of a Light-weight, Virtual Reality Trainer for the LSO Community: Time to Make the Leap Toward Immersive VR," Master Thesis, Naval Postgraduate School, 2015.

26 ¡

Q & A

As seen in Research Demo, Ballrooms A & B!