Improving quality modelling Stphane Vaucher PhD Defence November - - PowerPoint PPT Presentation

Improving quality modelling Stéphane Vaucher PhD Defence November 15, 2010

DIRO - Thesis defence - Stéphane Vaucher 2

Presentation structure

• Context
• Problems addressed
• Subjectivity
• Support for evolution
• Composition
• Conclusion

DIRO - Thesis defence - Stéphane Vaucher 3

Context

DIRO - Thesis defence - Stéphane Vaucher 4

Context

• Software systems are

– Large – Complex

• Typical banking/retail company

– >50 MLOCs of code – Mission critical

• Quality is important

– Bugs – Ability to modify existing code

DIRO - Thesis defence - Stéphane Vaucher 5

Testers Users C-level Managers Project Managers Developers QA team

Stakeholders

DIRO - Thesis defence - Stéphane Vaucher 6

Testers Users C-level Managers Project Managers Developer s QA team Easy to estimate? Help gain market share? Easy to use? Easy to modify? Passes quality audits? Conforms to spec?

Stakeholders

DIRO - Thesis defence - Stéphane Vaucher 7

Software Evolution

• Systems grow over time
• 8-12% annual employee turnaround
• Changes can improve/degrade quality

DIRO - Thesis defence - Stéphane Vaucher 8

Quality Evaluation Problem

• How to ensure software

quality?

• Large:

– Automation is essential

• State of the art:

– Use of ML or statistics to

build predictive models using historical data

DIRO - Thesis defence - Stéphane Vaucher 9

Metrics

Quality model

Estimation

DIRO - Thesis defence - Stéphane Vaucher 10

Metrics

Quality model

Estimation

➔Product ➔Process ➔Resources ➔Bugs ➔Effort ➔Operation

performance

➔Cost of

• wnership

DIRO - Thesis defence - Stéphane Vaucher 11

Context of my Research

• Contacted by a large company

– Quality problem: too many bugs

• Situation

– ~50% of systems are outsourced – No control over development process used – Want control over quality of systems

• Tried to build models using commercial/OS

tools based on “best practices”

– ISO9126 – Standard on product quality

DIRO - Thesis defence - Stéphane Vaucher 12

State of the Practice

• Most quality models are rule-based
• Do not leverage historical data
• Based on static analysis of the code

– (ex: size metrics)

• Example:

– Partner's quality portal – CAST software – Sonar – McCabe

DIRO - Thesis defence - Stéphane Vaucher 13

Presentation structure

• Context
• Problems addressed
• Subjectivity
• Support for evolution
• Composition
• Conclusion

DIRO - Thesis defence - Stéphane Vaucher 14

Problem

Many problems found in industry are not addressed in the literature and require research

DIRO - Thesis defence - Stéphane Vaucher 15

Problem 1

DIRO - Thesis defence - Stéphane Vaucher 16

Metrics

Quality model

Estimation Locate module to test Audit work Estimate coding task Code

Estimates are used by stakeholders

DIRO - Thesis defence - Stéphane Vaucher 17

Metrics

Quality model

Estimation

Estimates must correspond to expectations

DIRO - Thesis defence - Stéphane Vaucher 18

The notion of quality depends

• n the stakeholder

Quality is subjective

DIRO - Thesis defence - Stéphane Vaucher 19

Problem 2

DIRO - Thesis defence - Stéphane Vaucher 20

Tracking Quality Change

Version 1 Version 2

DIRO - Thesis defence - Stéphane Vaucher 21

Identify Changes Changes Assess quality Quality Assess quality Version 1 Version 2

Tracking Quality Change

DIRO - Thesis defence - Stéphane Vaucher 22

Quality changes with the system Quality models should help identify good and bad changes

DIRO - Thesis defence - Stéphane Vaucher 23

Abstraction level of systems

System Subsystem 2 Subsystem 1 Subsystem n

...

Package 2 Package 1 Package m

...

Class 2 Class 1 Class i

...

Method 2 Method 1 Method j

... ...

DIRO - Thesis defence - Stéphane Vaucher 24

Abstraction level of systems

System Manager Score

DIRO - Thesis defence - Stéphane Vaucher 25

Abstraction level of systems

Method Tester Score

DIRO - Thesis defence - Stéphane Vaucher 26

Composition of Quality

System Subsystem 1 Subsystem n

...

Package 1 Package m

...

Class 1 Class i

...

Method 1 Method j

... ...

Tester Score Manager Score

DIRO - Thesis defence - Stéphane Vaucher 27

The quality of a system depends

• n the quality of its parts

DIRO - Thesis defence - Stéphane Vaucher 28

Thesis

• Three dimensions for improvement of quality

models

– Support of subjectivity – Support for software evolution – Explicit support of software composition

DIRO - Thesis defence - Stéphane Vaucher 29

Presentation structure

• Context
• Problems addressed
• Subjectivity
• Support for evolution
• Composition
• Conclusion

DIRO - Thesis defence - Stéphane Vaucher 30

Supporting Subjectivity

DIRO - Thesis defence - Stéphane Vaucher 31

Subjectivity

• Quality audits were a source of conflict for
• ur partner
• Rule-based quality models

– (e.g. If LOC(method) > 10, bad) – Disagreements on rules/thresholds

Audit team Coding team Yes No

DIRO - Thesis defence - Stéphane Vaucher 32

Subjectivity: Approaches

• Fuzzy logic: how close is the code to the rules
• Bayes' theory: how probable is the code good

given its structure

• Focus on what most agree upon

DIRO - Thesis defence - Stéphane Vaucher 33

Subjectivity: Proposal

• Encode subjectivity as the probability that a

stakeholder would agree with the model

• P(quality=good|symptoms)

Yes No Yes

2/3

DIRO - Thesis defence - Stéphane Vaucher 34

Supporting Subjectivity

Detecting Design Defects

DIRO - Thesis defence - Stéphane Vaucher 35

Design defects

• Recurring bad design decisions
• From discussions with our partner
• Common defects defined by (Brown, 1996)
• Use of metrics to detect defects (Marinescu,

2004)

• State of the art defect detection (Moha, 2009)

– Domain-specific language to specify detection

rules

DIRO - Thesis defence - Stéphane Vaucher 36

Defects

• Blob: class that centralises functionality
• Spaghetti code: code without structure
• Functional decomposition: classes that are

function calls

DIRO - Thesis defence - Stéphane Vaucher 37

Data Set

• Original data set from (Moha, 2009)
• Gantt and Xerces: 2 medium open-source

packages

• Vote of 5 students (3 originally + 2 additional)
• Unanimous in ~30% of cases

DIRO - Thesis defence - Stéphane Vaucher 38

Symptoms

• Reused symptoms from Moha and Brown
• Analysis: symptoms are system-specific
• 2 options

– General symptoms: detection on a new system – Specific symptoms: detection on a new version

• f a known system

DIRO - Thesis defence - Stéphane Vaucher 39

Objectives

• Outperform state of the art detection

techniques

• Rank results in order of importance: how likely

someone will agree with an estimate

• 2 scenarios:

– Cross-project: learn on a project, apply to the

• ther

– Intra-project: learn and apply on the same

project (3-fold cross validation)

DIRO - Thesis defence - Stéphane Vaucher 40

Inspection Process

• Inspection in order of probability: P(Defect|Sympt.)
• Comparison with DECOR: a set of candidate

classes

• Recall: % of all defects returned
• Precision: % true positives returned

Class P(Spag|Symptoms) CoreDocumentImpl 99.89% AttrImpl 99.88% ... XPathException 0.11% Inspection order CoreDocumentImpl AttrImpl XPathException

DIRO - Thesis defence - Stéphane Vaucher 41

Sample output

1 2 3 4 5 6 7 8 9 10 11 12 13 14

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Bayes(P) Bayes(R) Bayes(F1) DECOR(P) DECOR(R) DECOR(F1)

Classes inspected

DIRO - Thesis defence - Stéphane Vaucher 42

Cross-project validation

• Same result set, but requires 1/2 inspections

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Bayes(P) Bayes(R) Bayes(F1) DECOR(P) DECOR(R) DECOR(F1)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1 2 3 4 5 6 7 8 9 10 11 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Bayes(P) Bayes(R) Bayes(F1) DECOR(P) DECOR(R) DECOR(F1)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Bayes(P) Bayes(R) Bayes(F1) DECOR(P) DECOR(R) DECOR(F1) 1234567891 1 1 1 2 1 3 1 4 1 5 1 6 1 7 1 8 1 9 2 2 1 2 2 2 3 2 4 2 5 2 6 2 7 2 8 2 9 3 3 1 3 2 3 3 3 4 3 5 3 6 3 7 3 8 3 9 4 4 1 4 2 4 3 4 4 4 5 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Bayes(P) Bayes(R) Bayes(F1) DECOR(P) DECOR(R) DECOR(F1)

DIRO - Thesis defence - Stéphane Vaucher 43

Intra-system validation

• Recall: 100%, very high precisions

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Precision Recall 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Precision Recall 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Precision Recall 1 2 3 4 5 6 7 8 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Precision Recall 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Precision Recall 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Precision Recall

DIRO - Thesis defence - Stéphane Vaucher 44

Subjectivity

• Management of subjectivity
• Cross-project : leads to equal/better precision

validation than DECOR

• Intra-project : leads to better precision than

DECOR and perfect recall

• Initial Bayesian approach published at

QSIC2009

• Follow up was accepted (with revisions) at JSS

DIRO - Thesis defence - Stéphane Vaucher 45

Presentation structure

• Context
• Problems addressed
• Subjectivity
• Support for evolution
• Composition
• Conclusion

DIRO - Thesis defence - Stéphane Vaucher 46

Supporting Evolution

DIRO - Thesis defence - Stéphane Vaucher 47

Systems are long lived

• Identify change patterns that

increase/decrease quality

• Quality will vary depending on maintenance

performed

• Changes are not homogeneously distributed

(on average 20% of classes change per minor version)

DIRO - Thesis defence - Stéphane Vaucher 48

Quality Signals

• Quality estimates are points of a signal vector

s=(q1, q2, ..., qn)

• Data mining to identify common trends of

quality evolution

DIRO - Thesis defence - Stéphane Vaucher 49

Quality Signal Matching

• Dynamic Time Warping: measures a distance

between signals by warping time

• Can match two modules that change in similar

manners, but at different moments/rates

DIRO - Thesis defence - Stéphane Vaucher 50

Quality Signal – Data Mining

• Two types of classification

– Clustering: used to identify unknown trends from

a group of signals

– Classification: used to label signals given a

group of known trends

DIRO - Thesis defence - Stéphane Vaucher 51

Supporting evolution

Tracking Blobs

DIRO - Thesis defence - Stéphane Vaucher 52

Study of Blobs

• Reminder:

– A blob is a class that centralises functionality

• Analysis of the introduction/removal of Blobs in

EclipseJDT and Xerces

• How are Blobs introduced?
• How are Blobs removed?

DIRO - Thesis defence - Stéphane Vaucher 53

Identified Patterns

DIRO - Thesis defence - Stéphane Vaucher 54

Quality Trend Distribution

• Eclipse JDT
• 58-63% of blobs are introduced and never

corrected/removed

– Xerces developers say it is by design

• Xerces

58% 10% 1% 6% 4% 9% 12%

Constant Sharp im- provement Slow im- provement Temporary Badness Temporary Relief Sharp de- gradationt Slow degrad- ation

63% 3% 11% 1% 3% 5% 14%

Constant Sharp Im- provement Slow Im- provement Temporary Badness Temporary Relief Sharp Degrada- tion Slow De- gradation

DIRO - Thesis defence - Stéphane Vaucher 55

Decreasing Quality

• Concerns 20-25% of Blobs
• Significant functionality is added to classes that

increase in responsibilities

– Classes double/triple in size (number of lines of

code/methods)

– Data holder classes are added (symptom used

for detection)

DIRO - Thesis defence - Stéphane Vaucher 56

Increasing Quality

• Few typical textbook corrections

– Only 7 cases when methods were moved out

• f the Blob and only one class extraction

found

• In Eclipse, blobs disappear as functionality is

added to peripheral classes

• Analysis covered period before the

widespread adoption of refactoring tools

DIRO - Thesis defence - Stéphane Vaucher 57

Evolution Conclusions

• Quality signals can be mined to understand the

evolution of a system

• Studying blobs, we found most are added by

design

• More blobs are moved than corrected
• Textbook refactorings are rare
• Published at WCRE 2010

DIRO - Thesis defence - Stéphane Vaucher 58

Presentation structure

• Context
• Problems addressed
• Subjectivity
• Support for evolution
• Composition
• Conclusion

DIRO - Thesis defence - Stéphane Vaucher 59

Software Quality Composition

DIRO - Thesis defence - Stéphane Vaucher 60

Abstraction level of systems

System Subsystem 2 Subsystem 1 Subsystem n

...

Package 2 Package 1 Package m

...

Class 2 Class 1 Class i

...

Method 2 Method 1 Method j

... ...

DIRO - Thesis defence - Stéphane Vaucher 61

Composition & Quality

• Quality models are built at a given level of

granularity

– Method-level : for testers – System-level : for managers

• How to combine information from a level to

another

– Max, min, averages

• Is there another way?

DIRO - Thesis defence - Stéphane Vaucher 62

General Approach

• Explicitly consider containment relationships
• Component: has internal quality
• Container: is composed of components each

with different levels of quality

• Aggregation strategy: assesses the importance
• f components on a container

DIRO - Thesis defence - Stéphane Vaucher 63

Multi-level Quality Evaluation

Score component 1 Component Model Metrics component 2 Component Model Component Model

…

Aggregation Model

…

Container Model

…

Container Metrics Weigthed Component Quality Score container Organisation components Metrics component 1 Metrics component n Score component 2 Score component n

DIRO - Thesis defence - Stéphane Vaucher 64

Quality Composition

Identifying high-maintenance classes

DIRO - Thesis defence - Stéphane Vaucher 65

High-Maintenance Classes

Score method 1 Method Model Metrics method 2 Method Model Method Model

…

Aggregation Model

…

Class Model

…

Class metrics Quality methods Score class Organisation methods Metrics Method 1 Metrics method n Score method 2 Score method n

DIRO - Thesis defence - Stéphane Vaucher 66

Method-level Model

• Built using freely available NASA dataset
• P(Changes=High|revg,LOC)

Method Change LOC Relative EVG

DIRO - Thesis defence - Stéphane Vaucher 67

Aggregation Model

• Weighs the importance of a method using an

importance function

• Naive strategy: use a standard aggregation

method

– max, median, average, sum

• Complex strategy: based on call-graph

analysis

DIRO - Thesis defence - Stéphane Vaucher 68

Call-graph Analysis

m1 m2 m3 m4 m5 m6 m7 m8 main Class 2 Class 3 Class 4 Class 1

• Call-graph: represents the call relationships

between methods in a system

DIRO - Thesis defence - Stéphane Vaucher 69

Call-graph Importance

• Use a measure of centrality

– Degree, PageRank, Markov Centrality,

Betweeness

m1 m2 m3 m4 m5 m6 m7 m8 main Class 2 Class 3 Class 4 Class 1

DIRO - Thesis defence - Stéphane Vaucher 70

Class-level Model

Changes Method Quality Method Importance DIT NOC Inheritance Aggregated Method-level Quality LCOM5 Cohesion Weighted Method Quality Weighted Method Quality Weighted Method Quality Cohesion Coupling Complexity

DIRO - Thesis defence - Stéphane Vaucher 71

Case Study – Data set

• Class-level

– Replication data from (Khomh, 2009) – 4 systems with # changes per class for training – Used ArgoUML (1237 classes) for test

• Method-level

– Available data from NASA – 2 systems OO with # changes per method

DIRO - Thesis defence - Stéphane Vaucher 72

Inspection efficiency

• # of changes identified after inspecting the N

first classes

• For N =50, PageRank and Centrality dominate

DIRO - Thesis defence - Stéphane Vaucher 73

Size-specific Inspections

• Change density after inspecting the N first

classes

• Maximum is the most efficient for

N < 20

• N >20, Degree and

PageRank are the most efficient

DIRO - Thesis defence - Stéphane Vaucher 74

Quality Composition

Evaluating Web Sites

DIRO - Thesis defence - Stéphane Vaucher 75

Quality Composition of Web Sites

• Applied our general composition model to

evaluate the navigability of Web sites

• Container: site
• Component: page
• Aggregation model: estimates the probability
• f a user visiting a page

DIRO - Thesis defence - Stéphane Vaucher 76

Quality Composition

• Applied model to good and random sites
• Found that our model can distinguish these

sites (p-value ~ 0)

• Presented at WSE2010

DIRO - Thesis defence - Stéphane Vaucher 77

Composition – Conclusion

• We proposed a general quality composition

model

• We can explicitly model the impact of a

component on a container

• Applied to the identification of high change

classes

– Found that central classes tend to change

more in ArgoUML

– Model outperformed standard mechanisms

• Applied to Web site evaluation

– Can correctly differentiate between

DIRO - Thesis defence - Stéphane Vaucher 78

Presentation structure

• Context
• Problems addressed
• Subjectivity
• Support for evolution
• Composition
• Conclusion

DIRO - Thesis defence - Stéphane Vaucher 79

Conclusion

• Contribution 1: Subjectivity

– Present Bayesian models to support

subjectivity

– Outperforms rule-based state of the art

techniques: 50% fewer classes to inspect

• Contribution 2: Evolution

– Quality estimates are signals that be mined – Clustering these signals can find change

patterns

– Patterns can be explored to identify good/bad

practices

Contribution 3: Composition

DIRO - Thesis defence - Stéphane Vaucher 80

Conclusion

• Both theoretical contributions and

experimentally tested

• Reuse of existing data set
• Results were released for replication

DIRO - Thesis defence - Stéphane Vaucher 81

Perspectives – currently underway

• Extending composition work to additional

systems

• Extending Web navigability study (with

Antoine Moulinart)

– Different aggregation models – Perfecting site and page models

• Use of crowdsourcing to gather quality data to

experiment with

DIRO - Thesis defence - Stéphane Vaucher 82

Collaborations

• Marouane Kessentini: use of meta-heuristics

to locate design defects

• Foutse Khomh: empirical study of good

practices

DIRO - Thesis defence - Stéphane Vaucher 83

Perspectives

• Composition between different other levels of

abstraction (e.g., class to package) and beyond containment relationships

• Call graph based metrics are very promising
• Technical debt calculations

DIRO - Thesis defence - Stéphane Vaucher 84

Publications

• Simon Allier, Houari Sahraoui, Salah Sadou, Stephane Vaucher Restructuring Object-Oriented Applications into

Component-Oriented Applications by using Consistency with Execution Traces Proceedings of the 13th International Symposium on Component Based Software Engineering (CBSE) 2010, Prague, Czech Republic.

• Simon Allier, Stephane Vaucher, Bruno Dufour, and Houari Sahraoui. Deriving Coupling Metrics from Call Graphs In Cristina

Marinescu and Jurgen J. Vinju, editors, 10th International Working Conference on Source Code Analysis and Manipulation (SCAM)}. IEEE Computer Society Press, 2010.

• Foutse Khomh, Stéphane Vaucher, Yann-Gael Guéhéneuc, and Houari Sahraoui. BDTEX: A GQM-based Bayesian

Approach for the Detection of Antipatterns. Journal of Systems and Software}, 2010. [Accepted with revisions]

• Marouane Kessentini, Stephane Vaucher, and Houari Sahraoui. Deviance from perfection is a better criterion than

closeness to evil when identifying risky code. In Elisabetta Di Nitto and Jamie Andrews, editors, 25th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE/ACM, 2010.

• Stephane Vaucher, Samuel Boclinville, Houari Sahraoui, and Naji Habra. Recommending improvements to web applications

using quality-driven heuristic search. In Darrell Long, Gottfried Vossen, and Jeffrey~Xu Yu, editors, 10th International Conference On Web Information Systems Engineering, pages 321--334. Springer, 2009.

• Stephane Vaucher, Foutse Khomh, Naouel Moha, and Yann-Gaël Guéhéneuc. Prevention and cure of software defects:

Lessons from the study of god classes. In Giuliano Antoniol and Andy Zaidman, editors, 16th Working Conference on Reverse Engineering (WCRE), pages 145--154. IEEE Computer Society Press, 2009.

• Stephane Vaucher and Houari Sahraoui. Do software libraries evolve differently than applications?: an empirical

investigation.In Symposium on Library-Centric Software Design (LCSD), pages 88--96, New York, NY, USA, 2007. ACM.

• Stephane Vaucher and Houari Sahraoui. Etude de la changeabilite des systemes orientes-objet. In Actes des journees

Langages et Modèles à Objets.Éditions Cépaduès, 2008.

• Stephane Vaucher and Houari Sahraoui. Multi-level evaluation of web sites. In 12th IEEE International Symposium on Web

System Evolution (WSE). IEEE Computer Society, 2010.

• Stephane Vaucher, Houari Sahraoui, and Jean Vaucher.Discovering New Change Patterns in Object-Oriented Systems. In

Andy Zaidman, Massimilano Di Penta, and Ahmed Hassan, editors, 15th Working Conference on Reverse Engineering (WCRE)}, pages 37--41, October 2008.

DIRO - Thesis defence - Stéphane Vaucher 85

Cross-project validation

• Same result set, but requires 1/2 inspections

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Bayes(P) Bayes(R) Bayes(F1) DECOR(P) DECOR(R) DECOR(F1)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1 2 3 4 5 6 7 8 9 10 11 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Bayes(P) Bayes(R) Bayes(F1) DECOR(P) DECOR(R) DECOR(F1)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Bayes(P) Bayes(R) Bayes(F1) DECOR(P) DECOR(R) DECOR(F1) 1234567891 1 1 1 2 1 3 1 4 1 5 1 6 1 7 1 8 1 9 2 2 1 2 2 2 3 2 4 2 5 2 6 2 7 2 8 2 9 3 3 1 3 2 3 3 3 4 3 5 3 6 3 7 3 8 3 9 4 4 1 4 2 4 3 4 4 4 5 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Bayes(P) Bayes(R) Bayes(F1) DECOR(P) DECOR(R) DECOR(F1)