Toward Architecture-based Reliability Estimation Roshanak Roshandel - - PowerPoint PPT Presentation

Toward Architecture-based Reliability Estimation

Roshanak Roshandel & Nenad Medvidovic Computer Science Department University of Southern California {roshande,neno}@usc.edu

Motivation

• Software reliability: probability that the system

performs its intended functionality without failure

• Software reliability techniques aim at reducing
• r eliminating failure of software systems
• Complementary to testing, rely on

implementation

• How do we go about building reliable

systems?

• How do we measure reliability early?

Software Architecture

• High-level abstractions describing

– Structure, Behavior, Constraints

• Coarse-grain building blocks, promote

separation of concerns, reuse

– Components, Connectors, Interfaces, Configurations

• Architectural decisions directly affect aspects of

software dependability

– Reliability

• ADLs, Formal modeling notations, related

analysis

– Often lack quantification and measurement

Architectural Reliability

• Lightly explored
• Require availability of implementation to:

– Build behavioral model of the software system – Obtain each component’s reliability

• Software architecture offers compositional

approaches to modeling and analysis

• The challenge is quantifying these results

– Presence of uncertainty – Unknown operational profile – Improper behavior

Archi chitect ectur ure

Local Reliability Local Reliability

Local Reliability

Global Reliability

M ar kov M ar kov M

• del

M

• del

M ar kov M ar kov M

• del

M

• del

M ar kov M ar kov M

• del

M

• del

Interfac e Protoco ls Static Behavio rs

Component

• nent

Interfac e Protocols Static Behavio rs Dynamic Behaviors

Component

• nent

Interface Protocols Static Behaviors Dynamic Behaviors

Component

• nent

“The Quartet” “The Quartet”

Comp Reliability Transition Probabilities

Architectural Models

Analysis Defects State Reliability Quantification

Classification Cost framework

( ( ), ) G t f θ r Domain Knowledge Random OR Model Extractor ITP Reliability Estimator Baum- Welch Algorithm Training data State-based Markov model Hidden Markov Modeling

Legend

Artifacts Major steps of the approach Numerical values Iterative process

ITP

Initial transition probabilities

Component Reliability

The Quartet

• 1. Interface
• Point by which a component interacts with other

components

• 2. Static behavior
• Discrete functionality of a component
• i.e., at particular “snapshots” during the system’s

execution

• 3. Dynamic behavior
• Continuous view of how a component arrives at

different states throughout its execution

• 4. Interaction protocol
• External view of the component
• Specifies its legal interactions with other components

in the system

Cruise Control Comp gas() brake() cruise() decelerate() accelerate () maintain()

PRO V gas( val : SpeedType) : SpeedType; PRO V br ake( val : SpeedType) : SpeedType; PRO V cr ui se( speed: SpeedType) ; Bool ean; STATE- VAR: cur Speed: SpeedType; i sCr ui si ng: Bool ean; I NVARI ANT: cur Speed M AX; O PERATI O NS:

• gas. pr eCond ( val > 0) ;
• gas. post Cond ( ~cur Speed = cur Speed + val ) ;

br ake. pr eCond ( val < 0) ; br ake. post Cond ( ~cur Speed = cur Speed + val AND i sCr ui si ng = f al se) ; cr ui se. pr eCond ( speed > 0) ; cr ui se. post Cond ( ~cur Speed = speed AND i sCr ui si ng = t r ue) ; INT ERFACES ST ATIC BEHAVIOR

≤

≤

stop

gas/accelerate

manual cruise

gas/accelerate brake[val +curSpeed >0] /decelerate cruise/maintain brake/decelerate gas/accelerate brake[val +curSpeed 0] /decelerate DYNAM IC BEHAVIOR

≤

S1 S2

gas() brake() cruise gas brake() INT ERACT ION PROT OCOLS

Comp Reliability Transition Probabilities

Architectural Models

Analysis Defects State Reliability Quantification

Classification Cost framework

( ( ), ) G t f θ r Domain Knowledge Random OR Model Extractor ITP Reliability Estimator Baum- Welch Algorithm Training data State-based Markov model Hidden Markov Modeling

Legend

Artifacts Major steps of the approach Numerical values Iterative process

ITP

Initial transition probabilities

Component Reliability

Interface Static Behaviors Interaction Protocols Dynamic Behaviors

Syntactic Semantic

Comp Reliability Transition Probabilities

Architectural Models

Analysis Defects State Reliability Quantification

Classification Cost framework

( ( ), ) G t f θ r Domain Knowledge Random OR Model Extractor ITP Reliability Estimator Baum- Welch Algorithm Training data State-based Markov model Hidden Markov Modeling

Legend

Artifacts Major steps of the approach Numerical values Iterative process

ITP

Initial transition probabilities

Component Reliability

Defect Quantification

• Architectural defects could affect system

Reliability

• Different defects affect the Reliability differently

– e.g., interface mismatch vs. protocol mismatch

• The cost of mitigating defects varies based on

the defect type

• Other (domain specific) factors may affect the

quantification

• Classification + Cost framework

Classification + Cost Framework

1 2

( ( ), ), ( ) [ ( ), ( ),..., ( )]

t n

c G t f where t t t t θ θ θ θ θ = = r r

• Pluggable/Adaptable
• Identify the important

factors within a domain

• For a defect class t
• f: Frequency of
• ccurrence
• And

vector of all relevant factors

• Result will be used in

reliability estimation

( ) t θ r

Directional Structural Usage Incomplete Interface Signatures Static Behavior Pre / Post Conditions Protocol Interaction Protocols Topological Error Behavioral Inconsistency Architectural Defect Directional Structural Usage Incomplete Interface Signatures Static Behavior Pre / Post Conditions Protocol Interaction Protocols Topological Error Behavioral Inconsistency

Comp Reliability Transition Probabilities

Architectural Models

Analysis Defects State Reliability Quantification

Classification Cost framework

( ( ), ) G t f θ r Domain Knowledge Random OR Model Extractor ITP Reliability Estimator Baum- Welch Algorithm Training data State-based Markov model Hidden Markov Modeling

Legend

Artifacts Major steps of the approach Numerical values Iterative process

ITP

Initial transition probabilities

Component Reliability

Reliability Techniques

• Non-Homogenous Poisson Processes, Binomial

Models, Software Reliability Growth Models, …

• Markovian Models

– Suited to architectural approaches – Consider a system’s structure, compositional – Stochastic processes – Informally, a finite state machine extended with transition probabilities

Our Reliability Model

• Built based on the dynamic behavioral model
• Assume Markov property

– Discrete Time Markov Chains

• Transition probabilities may be unknown
• Complex behavior results in lack of a

correspondence between events and states

• Event/action pairs to describe component

interactions Augmented Hidden Markov Models (AHMM)

Evaluation

• Uncertainty analysis

– Operational profile – Incorrect behavior

• Sensitivity analysis

– Traditional Markov-based sensitivity analysis combined with the defect quantification

• Complexity
• Scalability

Conclusion and Future Work

• Step toward closing the gap between

architectural specification and its effect on system’s reliability

• Handles two types of uncertainties associated

with early reliability estimation

• Preliminary results are promising
• Need further evaluation
• Build compositional models to estimate system

reliability based on estimated component reliabilities

Questions?

AHMM

1 1 1

: , { ,..., } : : : , { ,..., } : : , :{ ,..., } : : ( , , )

N t M K

S Set of all possible States S S S N Number of states q state at time t E Set of all events E E E M Number of events F Set of all actions F F F K Number of actions We now define A B is a Hidden Markov M λ π = = =

1

: : { }, Pr[ | ], 1 , : { ( )} ( ) Pr[ / | ], 1 ,1 ,1 :

ij ij t j t i j j m k t j

• del such that

A statetransition probability distribution A a a q S q S i j N B Interface probability distribution in state j B b m b m E F at t q S j N m M k K The initial proba π

+

= = = = ≤ ≤ = = = ≤ ≤ ≤ ≤ ≤ ≤

1

{ } Pr[ ],1 .

i i i

bility distribution q S i n π π π = = = ≤ ≤

Cruise Control Example

stop

gas/accelerate

manual cruise

gas/accelerate brake/decelerate cruise/maintain brake/decelerate gas/accelerate brake/decelerate DYNAM IC BEHAVIOR

Partial Markov Extension

stop gas/accelerate

cruise

gas/accelerate cruise/ maintain gas/accelerate brake/decelerate TRUE TRUE

manual

TRUE gas/accelerate gas/accelerate gas/ accelerate gas/accelerate gas/ accelerate brake/ decelerate brake/ decelerate brake/ decelerate

Transition Probabilities

0.15 0.8 0.05 0.018 0.36 0.622 0.02 0.85 0.13 stop manual cruise stop ITP manual cruise ⎡ ⎤ ⎢ ⎥ = ⎢ ⎥ ⎢ ⎥ ⎣ ⎦

cruise 0.6 1 maintain 0.6 0.6 cruise manual stop 0.01 1 decelerate 0.01 break manual cruise 0.01 1 decelerate 0.01 break manual manual 0.08 1 decelerate 0.08 0.1 break manual stop 0.008 0.4 accelerate gas manual cruise 0.012 0.6 accelerate 0.02 gas manual manual 0.08 1 accelerate 0.08 0.1 gas manual manual 0.2 1 TRUE 0.2 0.2 TRUE manual cruise 0.03 1 accelerate 0.03 gas cruise stop 0.02 1 accelerate 0.02 0.0 5 gas cruise cruise 0.1 1 TRUE 0.1 0.1 TRUE cruise manual 0.85 1 decelerate 0.85 0.8 5 break cruise manual 0.8 1 accelerate 0.8 gas stop cruise 0.05 1 accelerate 0.05 gas stop stop 0.05 1 accelerate 0.05 0.9 gas stop stop 0.1 1 TRUE 0.1 0.1 TRUE stop

• Dest. State

Total Pr Pr(O).Pr(R) Pr(R) Reaction Pr(O) Observation Origin State cruise 0.6 1 maintain 0.6 0.6 cruise manual stop 0.01 1 decelerate 0.01 break manual cruise 0.01 1 decelerate 0.01 break manual manual 0.08 1 decelerate 0.08 0.1 break manual stop 0.008 0.4 accelerate gas manual cruise 0.012 0.6 accelerate 0.02 gas manual manual 0.08 1 accelerate 0.08 0.1 gas manual manual 0.2 1 TRUE 0.2 0.2 TRUE manual cruise 0.03 1 accelerate 0.03 gas cruise stop 0.02 1 accelerate 0.02 0.0 5 gas cruise cruise 0.1 1 TRUE 0.1 0.1 TRUE cruise manual 0.85 1 decelerate 0.85 0.8 5 break cruise manual 0.8 1 accelerate 0.8 gas stop cruise 0.05 1 accelerate 0.05 gas stop stop 0.05 1 accelerate 0.05 0.9 gas stop stop 0.1 1 TRUE 0.1 0.1 TRUE stop

• Dest. State

Total Pr Pr(O).Pr(R) Pr(R) Reaction Pr(O) Observation Origin State

0.1178 0.8293 0.0529 0.0304 0.3672 0.6024 0.0135 0.8537 0.1328 P ⎡ ⎤ ⎢ ⎥ = ⎢ ⎥ ⎢ ⎥ ⎣ ⎦

Baum-Welch

Reliability Model

stop (R1) cruise (R3) manual (R2) T12 T21 T22 T23 T32 T33 F C R3 1-R1 1-R2 1-R3

• Adaptation of

Cheung1980

1 2 1 1 11 1 12 1 1 1 1 1 1 2 1 1 ( 1)1 1 ( 1)2 1

... ... 1 ... ... 1 ... ... 1 ... ... ... ... ... ... ... ... ... ... ... ˆ 1 ... ... ... ... ... ... ... ... ... ... ... 1

j N j N i i i i i i ij i iN i N N N N N N N

C F S S S S C F R RT RT RT RT S P R RT RT RT RT S R R T R T S S

− − − − − −

− = − −

1 ( 1) 1 ( 1) 1 2

... ... 1 ... ...

N N j N N N N N N N N N N Nj N NN

R T R T R R R T R T R T R T

− − − −

⎡ ⎤ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ − ⎢ ⎥ ⎣ ⎦

ˆ ( , )

n

P i j

Probability of reaching j from i after n steps.

1

ˆ ( , )

n comp

R P S C =

Example…

1 1 ˆ 0.1300 0.0670 0.7444 0.0864 1 0.0147 0.3626 0.5227 0.2400 P ⎡ ⎤ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ = ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ 0.1178 0.8293 0.0529 0.0304 0.3672 0.6024 0.0135 0.8537 0.1328 P ⎡ ⎤ ⎢ ⎥ = ⎢ ⎥ ⎢ ⎥ ⎣ ⎦

0.15 0.8 0.05 0.018 0.36 0.622 0.02 0.85 0.13 stop manual cruise stop ITP manual cruise ⎡ ⎤ ⎢ ⎥ = ⎢ ⎥ ⎢ ⎥ ⎣ ⎦

1(1,

) 0.7444 0.76 0.5657 %56

comp cruise comp comp

R Q cruise R R R

−

= × = × ≈ ⇒ ≈

Q Rstop=0.87, Rmanual=0.9, Rcruise=0.76

More on the AHMM

• For states Si and Sj, there may be several transitions

Em/Fk

• Probability of transition from Si to Sj by means of a

given Em and all possible actions Fk

• But do we know what these are at the architecture

level?

1 1

m k

M K ij ijE F m k

T P

= =

=∑∑

Parameter (re)estimation

• Baum-Welch algorithm

– Uses Expectation Maximization – Given a sequence of training data

• Calculates the probability of a given observation sequence

and the probability of transitions from Si to Sj

1 1 1 1 1 1

( ) ( )Pr ( | )Pr ( | ) ( ) Pr ( | )Pr ( | ) ( )

t t t t t t j t t t t t t j

i j q i q j x q i i q j q i x q j j α α β β

− − − −

= = = = = = = =

∑ ∑

System Reliability

comp B comp A comp C comp D conn1 conn2 Archictecture

comp A comp B conn1 comp C conn2 comp D

components connector communication link state transition concurrent state

Relationships

• Interface vs. Other Models

– Syntactic – Interface as the core – Static Behaviors constrain interfaces using pre/post- conditions – Transition labels on Dynamic Behaviors and Interaction Protocols relate to interface as well – Dynamic Behaviors and Interaction Protocol model may have additional transitions that do not relate to component’s interfaces

• hierarchy and abstraction

Relationships II

• Static Behaviors vs. Dynamic Behaviors

– Semantic – Transition Guard vs. Operation Pre-Condition

• Union Guard:

– State Invariant vs. Component Invariant – State Invariants vs. Operation Post-Condition

1 n i i

U G G U G P

=

= ∨ ⇒

StateInv CompInv => StateInv PostCond =>

Relationships III

• Dynamic Behaviors vs. Interaction Protocols

– Semantic – The dynamic behavioral model may be more general than the protocol of interactions; any execution trace obtained by the protocol model, must result in a legal execution of component’s dynamic behavioral model

• Static Behaviors vs. Interaction Protocols

– Static Behaviors Dynamic Behaviors Interaction Protocols – Dynamic Behavioral model acts as a conceptual bridge – Interaction protocols specifies the valid sequence by which the

component’s interfaces may be accessed, oblivious to the component’s internal state

• No direct conceptual relationship

Uncertainty Analysis

• Two sources of uncertainty:

– Unknown operation profile, and incorrect component behavior

• How important it is to estimate ITP accurately?

– Complexity of the behavioral model directly relates to the importance of correct ITP initialization

• How about slight changes to ITP? How well the

model can handle uncertainty?

Evaluation

• Uncertainty analysis

– Operational profile – Incorrect behavior

• Sensitivity analysis

– Traditional Markov-based sensitivity analysis combined with the defect quantification

• Complexity
• Scalability

Uncertainty Analysis

• Two sources of uncertainty:

– Unknown operation profile, and incorrect component behavior

• How important it is to estimate ITP accurately?

– Complexity of the behavioral model directly relates to the importance of correct ITP initialization

• How about slight changes to ITP? How well the

model can handle uncertainty?

Example

0.15 0.8 0.05 0.018 0.36 0.622 0.02 0.85 0.13 ITP ⎡ ⎤ ⎢ ⎥ = ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ 0.05 0.9 0.05 0.018 0.36 0.622 0.22 0.65 0.13 ITP ⎡ ⎤ ⎢ ⎥ ′ = ⎢ ⎥ ⎢ ⎥ ⎣ ⎦

• 93.33%

12.50% 80.00% . .% 555.55% 55.55%

• 48.23%

900.00%

• 23.52%

15.38% Rand Fluc ⎡ ⎤ ⎢ ⎥ = ⎢ ⎥ ⎢ ⎥ ⎣ ⎦

Sensitivity Analysis

0.2 0.4 0.6 0.8 1 1.2 1 2 3 4 R1 R2 R3 Comp Reliability

• Tied with the cost

framework can

• ffer cost-effective

mitigation strategies

Complexity and Scalability

• Complexity of event-based

Markov Model:

• Our event/action based model:

– N: num states, M: num events – K: num actions, T: length of training data

• M and K are fixed, but N can be

reduced using hierarchy

2

( ) O N M T × ×

2

( ) O N M K T × × ×