Tempest: Integrated OpenStack Testing Matthew Treinish - - PowerPoint PPT Presentation

Tempest: Integrated OpenStack Testing

Matthew Treinish mtreinish@kortar.org May 12, 2014

OpenStack is Complicated

Testing before Tempest

◮ Every project had its own unit tests ◮ Some projects had functional tests ◮ There was nothing to test the whole picture

What is Tempest?

◮ The official OpenStack integration test suite ◮ Only black-box testing, only interacts with OpenStack REST APIs ◮ Over 2300 tests covering every incubated and integrated project ◮ Used as the gating tests for every integrated project

Tempest Growth

◮ Started in late Diablo to solve the lack of integrated testing ◮ Became a gating test suite in the Folsom cycle ◮ Was one of the 5 most active projects in Icehouse ◮ Testing with tempest is now a project graduation requirement ◮ Being used in refstack as part of defcore validation

Growing Pains

◮ Huge continued increase in the number of projects and tests ◮ The common code to handle all the APIs becomes more complex ◮ Managing individual tests becomes very difficult ◮ Total run time continues to grow ◮ Tempest configuration becomes more involved to handle all the services and

configurations

Design Principles

◮ Should be able to run against any OpenStack Cloud from devstack to a public

cloud

◮ Everything explicit ◮ Only use public interfaces (REST APIs) ◮ Should be self cleaning ◮ Self Testing

Basic Tempest Architecture

API Tests

◮ Directly test the OpenStack REST APIs ◮ Uses a unified tempest REST client ◮ Fall into 2 categories positive and negative ◮ Negative tests are starting to be automatically generated from api schemas

Scenario Tests

◮ Through path tests of functionality ◮ Designed to simulate a typical end user action workflow ◮ Use official python-*clients ◮ Ideally will involve integration points between projects

CLI Tests

◮ Test the official client CLIs ◮ Quick sanity checks of functionality in an OpenStack environment ◮ Only perform read-only operations ◮ Mostly a matter of convenience for gating

Stress Tests

◮ Use tempest to generate load on a cloud ◮ Dedicated scenarios for generating load ◮ Can specify any existing tempest test to also be a stress generator ◮ All scenario tests are automatically added as stress tests ◮ Run stress tests serially or in parallel looping for a set period of time

3rd Party and Unit Tests

◮ 3rd Party Tests

◮ Tests for in-tree non-OpenStack APIs ◮ Currently only tests Nova’s EC2 implementation

◮ Unit Tests

◮ Tempest eventually reached a size where its operations were too complex to take for

granted

◮ Started late in Havana because of bugs in wrapper scripts ”super passing” the gate ◮ Unit testing coverage was expanded significantly during Icehouse

Parallel Tempest

◮ Tempest is capable of running it’s tests in parallel ◮ More closely simulates real load on a cloud ◮ Greatly improves speed of running tests:

◮ Serially gate runs took 1 hr with <1000 tests ◮ Current gate runs are 40min to 1 hr with 2300 tests

Issue with single user in parallel

Tenant Isolation and Parallel Execution

◮ Tenant isolation enables running each test class in isolation ◮ When enabled each test class creates an isolated user and tenant ◮ Execute tests in parallel at the class level

Branchless Tempest

◮ Starting with Icehouse tempest will no longer have stable branches ◮ For gating purposes we will use master tempest against stable branch proposed

commits

◮ All proposed tempest commits will run against master on the other projects and

the currently supported stable releases (Icehouse forward)

◮ This should help us ensure greater api consistency between releases

Tempest Configuration

◮ Around 200 config options currently ◮ Intent of config file is to tell tempest what can be run ◮ What should be run is the domain of the test runner ◮ Sample config file documents every option ◮ Intended to be able to cover any OpenStack configuration ◮ Should have sane defaults (ie match project defaults) ◮ To setup a basic config file:

◮ Configuring only basic auth and connection info should be needed ◮ Also specify flavor and image ids to use for creating servers ◮ List which services are available and any required configuration for those services

Future Work

◮ Expansion of autogenerated negative tests ◮ More Unit test coverage ◮ Expansion of feature flags in config file ◮ Configuration file validation and generation tooling ◮ Possibly a new service for running tempest

Where to get more information

◮ Tempest Repository ◮ Tempest Documentation ◮ IRC #openstack-qa channel on Freenode

Questions?