Automated Test Data Generation on the Analyses of Feature Models A - - PowerPoint PPT Presentation

Automated Test Data Generation on the Analyses of Feature Models

A Metamorphic Testing Approach

ICST 2010, Paris Sergio Segura1, Robert M. Hierons2, David Benavides1 and Antonio Ruiz-Cortés1

1 Department of Computer Languages and Systems, Univesity of Seville, Spain 2 School of Information Systems, Computing and Mathematics, Brunel University, Uxbridge, Middlesex, UK

Introduction Problem Contribution Conclusions Future work

Introduction Problem Contribution Conclusions Future work

Introduction

Mandatory Optional Alternative (choose 1) Or Excludes Requires

A Feature Model represents a set of products in terms of features and relations among them. Feature Feature

Introduction

= {MobilePhone, Calls, Screen, Basic, GPS, Media, MP3}

A Feature Model represents a set of products in terms of features and relations among them.

Non valid product!

Introduction

Automated analysis of feature models:

Computer-aided extraction of information from feature models

14 How many products?

Introduction

Yes, feature “Basic” is dead Any error?

Automated analysis of feature models:

Computer-aided extraction of information from feature models

Introduction

Automated Analyses on FMs

Valid Product Void Filter Variability Products Number of Products Optimization Corrective Explanations Dead Features Explanations SAT Constraint Programming Description Logic BDD OWL

Operations Techniques Tools Testing Mechanisms

• ver 30 operations

Automated Test Data Generation Manual Test Suite

Introduction

Analysis tool

The model represents

2016 products

Input Output

How many products?

Combinatorial complexity! Combinatorial complexity!

Introduction

Metamorphic Testing

cos(x)

x = 42

• 0.399

cos(x) = cos(-x) cos(x)

x = - 42

• 0.399

Metamorphic Relation Follow-up test case

• 0.426

Introduction Problem Contribution Conclusions Future work

Problem

How can we avoid the oracle problem on the analyses of feature models? How can we automate the generation of test data?

Introduction Problem Contribution Conclusions Future work

Contribution

We propose a set of metamorphic relations on feature models We present an automated test data generator How can we avoid the oracle problem on the analyses of feature models? How can we automate the generation of test data?

Contribution

We propose a set of metamorphic relations on feature models We present an automated test data generator How can we avoid the oracle problem on the analyses of feature models? How can we automate the generation of test data?

Contribution

Metamorphic relations on Feature Models

products(FM) products(FM) P1 = {A,C} P2 = {A,B,C} Neighbours FM FM’

FM’ = FM + mandatory

P1 = {A,C,D} P2 = {A,B,C,D} Metamorphic relation

Contribution

Metamorphic relations on Feature Models

)) ' ) ( ( }) { ' ) ( ( ) ( ) ' ( ' ( ' ) ( # ) ' ( # P P P features pf f P P P features pf FM products P FM products P P FM products FM products                

products(FM) products(FM) P1 = {A,C} P2 = {A,B,C} Neighbours FM FM’

FM’ = FM + mandatory

P1 = {A,C,D} P2 = {A,B,C,D} Metamorphic relation (pf) (f)

Contribution

Metamorphic relations on Feature Models

)) ' ) ( ( }) { ' ) ( ( ) ( ) ' ( ' ( ' ) ( # ) ' ( # P P P features pf f P P P features pf FM products P FM products P P FM products FM products                

})) { ' ) ( ( ' ) ( ) ' ( ' ( ' ) }, { , ( ) ( # ) ' ( # f P P P features pf P P FM products P FM products P P pf FM filter FM products FM products                

)) ' ) ( ( }) { ' ) ( ( ) ( ) ' ( ' ( ' ) }, { , ( )# 1 (# ) ( # ) ' ( # P P P features pf c P P C c P features pf FM products P FM products P P pf FM filter C FM products FM products                       ))) ' ) ( ( ) ' ) ( ) ( ( ) ( ) ' ( ' ( ' ) }, { , ( )# 1 2 ( ) ( # ) ' ( #

#

P P P features pf S P P C S P features pf FM products P FM products P P pf FM filter FM products FM products

C

                       }) { }, { , ( \ ) ( ) ' ( g f FM filter FM products FM products  ) }, , { , ( \ ) ( ) ' (  g f FM filter FM products FM products  Metamorphic relation FM FM’

MANDATORY OPTIONAL ALTERNATIVE OR REQUIRES EXCLUDES

Contribution

We propose a set of metamorphic relations on feature models We present an automated test data generator How can we avoid the oracle problem on the analyses of feature models? How can we automate the generation of test data?

Contribution

Automated Test Data Generation

P1 = {A,C} P2 = {A,B,C} P1 = {A,C} P2 = {A,B,C,D} P3 = {A,B,C,E} P4 = {A,B,C,D,E}

• 1. Start with a trivial test case
• 2. Create a neighbour by adding a (random)

relationship to the model

• 2. Create a neighbour by adding a (random)

relationship to the model

• 3. Construct the set of products represented by the new

model according to our metamorphic relations

• 3. Construct the set of products represented by the new

model according to our metamorphic relations

Or

Contribution

Automated Test Data Generation

P1 = {A,C} P2 = {A,B,C,D} P3 = {A,B,C,E} P4 = {A,B,C,D,E} P1 = {A,C} P2 = {A,B,C} P1 = {A,C,F} P2 = {A,B,G} P3 = {A,B,C,D,F} P4 = {A,B,C,E,F} P5 = {A,B,C,D,G} P6 = {A,B,C,E,G} P7 = {A,B,C,D,E,G} P8 = {A,B,C,D,E,F}

Or Alternative Excludes

P1 = {A,C,F} P2 = {A,B,C,D,F} P3 = {A,B,C,E,F} P4 = {A,B,C,D,E,F}

Optional

P1 = {A,C,F} P2 = {A,B,C,D,F} P3 = {A,B,C,E,F} P4 = {A,B,C,D,E,F} P5 = {A,B,C,E,F,H} P6 = {A,B,C,D,E,F,H}

Yes, it represents at least one product Is the model consistent? How many products it represents? 6 products Does the model contain any dead feature? Yes, feature „G‟ is not included in any product. Metrics?

Commonality(B) = 83.3% Variability = 0.023

Contribution

Automated Test Data Generation

13,741,632 products

50 features 5% cross-tree constraints

Contribution

Evaluation

Reasoner LoC #Mutants Av Time (s) Max Time (s) Max TCs Score Sat4jReasoner 743 262 7.4 566.5 414 99.3% JavaBDDReasoner 625 302 1.6 111.7 350 98.7% JaCoPReasoner 686 185 0.7 8.3 83 100% Total 2054 749 3.2 566.5 414 99.3%

Mutation Testing Real faults and tools

FaMa Framework v1.0 alpha Batory’06 - SPLC

Score: 91.6% Average time: 20.2 seconds 2 defects found:

• Sat4jStagedConfigurations  Av Time: 0.5 seconds
• JaCoPProducts  Av Time: 142.9 seconds

6 operations. Timeout: 600s

Introduction Problem Contribution Conclusions Future work

Conclusions

• Contributions:
• Set of metamorphic relations on feature models overcoming the oracle problem.
• Automated test data generator relying on them.
• Main points of our approach:
• Efficient. Hundred of complex feature models and their associated set of products

can be generated within seconds.

• Flexible. It can be used to test any operation extracting information from the set of

products of a feature model (most of them)

• Effective. Mutation score of 99.3%. Two defects identified in a recent release of

FaMa.

Introduction Problem Contribution Conclusions Future work

Ongoing and Future Work

• Comparison of our automated generator and FaMa manual suite

 Results have revealed a great superiority of our automated approach.

• Combination of our manual suite and our automated test data generator.

 We have got significant improvements in terms of efficiency

• Evaluation with other tools.

Two defects found in SPLOT .

• Study the applicability of our approach in other domains.

http://www.splot-research.org

Thanks!

www.isa.us.es www.isa.us.es/fama sergiosegura@us.es