[PPT] - Evaluation of Cyber Defense Exercises Using Visual Analytics Process PowerPoint Presentation

SLIDE 1

1

Evaluation of Cyber Defense Exercises Using Visual Analytics Process

Radek Ošlejšek, Jan Vykopal, Karolína Burská, and Vít Rusňák

IEEE Frontier in Education Conference, San Jose, USA, 2018

SLIDE 2

IEEE Frontier in Education Conference, San Jose, USA, 2018

2

KYPO Cyber Range

Cloud-based “simulator” of computer networks So powerful that we can organize cyber defense exercises, CDXs

–

Extreme use case

–

Comprehensive training for IT professionals

–

Realism, difficulty (2 days), work under stress, ...

–

Protection of complex critical infrastructure by Blue teams

–

Escalated attacks of a Red team

… but the preparation and organization is a nightmare :-(

SLIDE 3

IEEE Frontier in Education Conference, San Jose, USA, 2018

3

Cyber Defense Exercises – Current Problems

New scenarios are designed from scratch

–

No transfer of knowledge and experience between (changing)

rganizers

The lack of situational awareness

–

Monitoring the infrastructure, providing insight, ...

The lack of analytical tools

–

Evaluation of scenarios, improving their impact on learners

Reason:

–

too many involved people, non-formalized processes, changing data, unclear objectives => a lot of ad-hoc preparation and manual work.

SLIDE 4

IEEE Frontier in Education Conference, San Jose, USA, 2018

4

Cyber Defense Exercises – Life Cycle

L e a r n i n g a n d t r a i n i n g

b

j e c t i v e s B a c k g r

u

n d s t

r

y S c e n a r i

t

a s k s a n d i n j e c t s S c e n a r i

t

e c h n i c a l d e t a i l s S c

r

i n g d e s i g n S a n d b

x

d e p l

y

m e n t H a c k a t h

n

S c e n a r i

a

n d s a n d b

x

t w e a k i n g S a n d b

x

d e p l

y

m e n t D r y r u n e x e c u t i

n

F e e d b a c k i n c

r

p

r

a t i

n

F a m i l i a r i z a t i

n

p e r i

d

A c t u a l e x e r c i s e H

t

w a s h u p e v a l u a t i

n

W

r

k s h

p

f

r

B l u e t e a m s I n t e r n a l l e s s

n

s l e a r n e d Plan Do Check

A. Preparation
B. Dry run
C. Execution
D. Evaluation

White T eam Green T eam Red T eam Blue T eams

months a week weeks days

Adjust

SLIDE 5

IEEE Frontier in Education Conference, San Jose, USA, 2018

5

Our Goal

To clarify data, processes, and requirements
Systematically support organizers in their tasks by means of

interactive visualizations integrated into the cyber range

SLIDE 6

IEEE Frontier in Education Conference, San Jose, USA, 2018

6

Approach: Using a Visual Analytics Process

Knowledge generation model by Sacha et al.

–

Hypothesis-driven model extending the model of Keim et al. (the computer part) with hierarchically connected human loops Classification of

SLIDE 7

IEEE Frontier in Education Conference, San Jose, USA, 2018

7

Analytical Goals (Classification of Hypotheses)

G1: Evaluation of exercise and its parameters

–

To make an exercise useful and to keep learners motivated to finish it.

–

Hypotheses related to scenario difficulty, learners’ confidence and satisfaction, learners’ skills, and other qualitative aspects

G2: Behavioral analysis of learners

–

To reveal relevant facts about the motivation of learners, learning impact, their level of knowledge, etc.

–

Hypothesis related to the study of the behavior of learners during an exercise.

G3: Runtime situational awareness

–

We can consider situational awareness as a process of making simple runtime hypotheses in the users’ mind.

SLIDE 8

IEEE Frontier in Education Conference, San Jose, USA, 2018

8

Classification of Data

Scenario-specific data

–

Configuration data defined by organizers usually in the preparation phase

–

Division of learners to teams, network topology, penalties, ...

Exercise runtime data

–

A system-generated data gathered and stored during the execution phase of an exercise.

–

Obtained penalty points by individual teams, ...

Evaluation data

–

User-generated data providing qualitative information

–

Post-exercise surveys, online feedback data, notes of organizers, ...

SLIDE 9

IEEE Frontier in Education Conference, San Jose, USA, 2018

9

Classification of Visualizations

Exercise infrastructure view

–

Monitoring of services and infrastructure (G3 – situational awareness and G2 – behavioral analysis).

Visual insight into the exercise progression

–

Primary visualizations for G3 – situational awareness. Moreover, online validation of exercise parameters (G1 – exercise evaluation)

Interactive feedback visualizations

–

Interactive = learners provides comments, ranks events, etc. This data is used by organizers to reveal inappropriate exercise parameters (G1 – exercise evaluation) and to collect behavioral data (G2 – behavioral analysis)

SLIDE 10

IEEE Frontier in Education Conference, San Jose, USA, 2018

10

Model

Can be as simple as descriptive statistics or as complex as a

data mining algorithms

Statistical models are used extensively for CDX
Utilization of advanced models is exceptional and ad-hoc just

because of missing conceptual solution to repeated analytical tasks

SLIDE 11

IEEE Frontier in Education Conference, San Jose, USA, 2018

11

Case Study

Hypothesis:

–

The participants improve their skills

Data

–

Data from scoring and auditing systems

–

Pre- and post-exercise questionnaires

Model

–

Descriptive statistics

Visualizations

–

Feedback visualization

–

Statistical visualizations

SLIDE 12

IEEE Frontier in Education Conference, San Jose, USA, 2018

12

Exploration Loop: Actions and Findings

For the hypothesis “The participants improve their skills” Actions:

–

Organizers: Data definition, configuration of data sources (sub- systems) and dashboards (visualizations), evaluation

–

Learners: Filling questionnaires, interaction with the cyber range and the feedback visualization

Findings:

–

Majority of the learners confirmed they learned new skills or re-shaped existing ones.

–

Some learners did not learn anything new.

–

Some others admitted the lack of necessary skills.

SLIDE 13

IEEE Frontier in Education Conference, San Jose, USA, 2018

13

Insight and Knowledge

For the hypothesis “The participants improve their skills” Insight:

–

Fairly confirmed. Individual learners would be affected by their skills and skills of teammates. A novel ways of prerequisite testing are desired.

–

New hypotheses hypotheses have been derived:

The difficulty of the exercise was adequate for learners
Learners form well-balanced teams

Knowledge:

–

Knowledge is a “justified insight”. In our case study, it is necessary to repeat the exercise so that we get data of more participants

SLIDE 14

IEEE Frontier in Education Conference, San Jose, USA, 2018

14

Conclusion

We proved the applicability of VA process on complex cyber

defense exercises

We proposed a basic classification for hypotheses, data,

models, and visualizations and their mapping to CDX life cycle

Applying the VA process to the organization of cyber defense