Experimental Evaluation of a Joint Cognitive System for 4D - - PowerPoint PPT Presentation

1

Challenge the future

1

Experimental Evaluation of a Joint Cognitive System for 4D Trajectory Management

The Third SESAR Innovation Days Rolf Klomp

2

Challenge the future

2

C-SHARE consortium

Max Mulder (coordinator) René van Paassen Rolf Klomp Gesa Praetorius Martijn Mooij Annemiek van Drunen Sorin Iacob Vincent Clot Dennis Nieuwenhuisen Pim van Leeuwen Tanja Bos Anneloes Maij

C-SHARE

3

Challenge the future

3

Towards higher levels of automation

A key pillar of the SESAR Master Plan: Allocate a central role for the human, but supported by a high degree of automation However, the role of the human is not well defined yet

ATM network demand Technological advances

4

Challenge the future

4

• bserved aircraft

metering fix

How can we effectively cope with complex situations in 4D trajectory planning?

5

Challenge the future

5

• bserved aircraft

metering fix

By taking a ‘leap of faith’ and always trusting automation?

6

Challenge the future

6

The challenge of human-systems integration

Human-machine interaction perception settings sensing action

Mismatch?

7

Challenge the future

7

• bserved aircraft

metering fix

C-SHARE aims to make automation more transparent

8

Challenge the future

8

The foundation for effective human-machine coordination is based on a common ground of the problem to solve

environment functional model (FAR) functional view (FAV)

9

Challenge the future

9

Solution Space Diagram Vector field method Discrete route-based info HMI designs Initial Travel Space implementation Final JCS experiment version

Applying the multi-disciplinary CSE and EID paradigms is an iterative process

10

Challenge the future 10

Δψ

+10kts +20kts +30kts VMAX

travel space

• bserved aircraft

metering fix

Breakdown of the Travel Space Representation

11

Challenge the future 11

• bserved aircraft

metering fix restricted field of travel safe field of travel

Breakdown of the Travel Space Representation

12

Challenge the future 12

The travel space describes the constraints for problem solving rather than a single optimised solution

• bserved aircraft

metering fix safe field of travel restricted field of travel

13

Challenge the future 13

Allowing for automation and the human operator to work together as team players

• bserved aircraft

metering fix automated advisory

14

Challenge the future 14

Example use of the Travel Space Representation

15

Challenge the future 15

The Travel Space is an egocentric representation

• bserved aircraft

metering fix ? ? ? ?

16

Challenge the future 16

Research has resulted in the development of a software- based Joint Cognitive System (JCS)

17

Challenge the future 17

C-SHARE second human-in-the-loop experiment

Aim of the study:

• To investigate the effectiveness of the Travel Space Representation under varying airspace

and traffic conditions

Independent variables:

• Traffic orderliness:

structured, unstructured

• Scale of perturbations:

small, medium, large

Dependent Measures:

• Loss of separation and overall performance
• Workload and acceptance
• Use of automated advisories

18

Challenge the future 18

Experiment set-up

Two sub-experiments:

• Manual Control task
• Automated advisories

Participants:

• (2x) 12 Participants
• 3 groups: ATCos, domain experts, students

Procedure:

• 45 minute briefing and training
• 6 experiment runs of 6 minutes (4x speed)
• 2 hours total

19

Challenge the future 19

Results

Loss of separation and overall performance:

• An expected breakdown in human-automation coordination during the more complex

scenarios did not occur during either sub-experiment

• In both experiments 2 loss of separation events occurred over of 2232 controlled flights
• Similarly 2 Restricted Area intrusions occurred

20

Challenge the future 20

ISA workload self-assessment

• In both experiments the relative workload increased significantly with perturbation scale,

however, never became unmanageable

• In sub-experiment 2 workload significantly decreased for unstructured traffic, likely due to

the availability of automated advisories

21

Challenge the future 21

CARS acceptance rating

• Acceptance was highest for students, followed by domain experts and finally ATCos
• Acceptance did not significantly decrease in more complex scenarios

22

Challenge the future 22

Automated advisories

• Use of advisories was highest for students, followed by domain experts and finally ATCos
• More advisories were used in complex scenarios
• The number of advisory rejections shows that the Travel Space enables the operator to

understand, veto, and re-direct automation

23

Challenge the future 23

Issues in human factors and system performance measurements

• Currently no methods to evaluate joint human-automation systems
• Methodologies focus heavily on the distinction between the human and

the system parts

• Results are biased by experience and are highly dependent of individual

strategies

• There is no clear baseline of a future operational ATM concept

24

Challenge the future 24

Overall project outcome

• Proven that C-SHARE approach to the design of decision support tools can

greatly benefit human-automation coordination

• One of the first working (scalable, multi-platform) joint human-automation

prototypes for trajectory-based operations

• Opportunities for further research have been identified

25

Challenge the future 25

Future research

Functional Purpose Abstract Functions Generalized Functions Physical Functions Physical Form

technology knowledge components whole system Bridging the gap in the Abstraction Decomposition Space

Flow based management C-SHARE

26

Challenge the future 26

Questions?

Contact: R.E.Klomp@TUDelft.nl

27

Challenge the future 27

Experimental Evaluation of a Joint Cognitive System for 4D Trajectory Management

The Third SESAR Innovation Days Rolf Klomp

28

Challenge the future 28

• “Out-of-the-loop” situation awareness
• Vigilance problems
• Transient workload peaks
• Skill degradation
• Overreliance
• Lack of trust
• …

There is a risk of introducing breakdowns in human-automation coordination

29

Challenge the future 29

Example displays using the ecological approach

30

Challenge the future 30

The multidisciplinary approach is based upon the proven CSE and EID frameworks

The five steps of CSE:

• Work Domain Analysis
• Control Task Analysis
• Strategies Analysis
• Cooperation Analysis
• Worker Competencies

what is the purpose of the system? what must be done? how can it be done? how can work be shared? how can operators best be supported?

31

Challenge the future 31

• verall system goals

underlying laws and principles realization of abstract functions

The Abstraction Hierarchy shows the opportunities, constraints and relationships of the problem domain

32

Challenge the future 32

The multidisciplinary approach is based upon the proven CSE and EID frameworks

EID: making visible the invisible

AUTOMATION

33

Challenge the future 33

Iterative approach

scope definition and task evaluation concept FAR JCS version 1 JCS version 2 conceptual evaluation HITL 2 workshop with stakeholders D4.4 D3.2 D3.3 HITL 1

design evaluation

D1.1 D2.2 D2.3

34

Challenge the future 34

C-SHARE first human-in-the-loop experiment

Aim of the study:

• To validate the Travel Space concept and obtain expert feedback in an early design stage
• Held with 3 former ATCOs involved in SESAR activities at the EEC in Brétigny

35

Challenge the future 35

C-SHARE final workshop

• 16 international domain experts and future end-users with a broad range of expertise
• Hands-on demonstration of the JCS
• SWOT analysis to identify strengths, weaknesses, opportunities and threats to guide future

research activities demonstration cubicle Interactive SWOT