ERAU the FAA Research 2007-2009 CEH Tools Qualification Contract - - PowerPoint PPT Presentation

Real Time Safety ERAU College of Engineering

page 1

ERAU the FAA Research 2007-2009 CEH Tools Qualification

Contract DTFACT-07-C-00010

• Dr. Andrew J. Kornecki, Dr. Brian Butka

Embry Riddle Aeronautical University

• Dr. Janusz Zalewski

Florida Gulf Coast University FAA SW&CEH Conference Denver, CO August 20, 2008

Real Time Safety ERAU College of Engineering

page 2

Outline

Contract DTFACT-07-C-00010 “A Study on Tool Qualification for Complex Electronic Hardware”, Mar 15, 2007 – May 14, 2009

• Introduction
• CEH Tools and Development Process
• Tool Qualification
• Concerns and Possible Solutions
• Conclusions and Future Work

Real Time Safety ERAU College of Engineering

page 3

Introduction: CEH Tools Project

ERAU/FGCU Project: “A Study on Tool Qualification for

Complex Electronic Hardware” with an objective to produce report on the state of art in the CEH tool market and the programmable logic tool qualification issues

Explore safety issues in the assessment and

qualification of tools used in developing complex electronic hardware (CEH) for the aircraft

Typical devices are programmable logic devices (PAL,

PLA, GAL, FPGA, ASIC, etc.) used as components of programmable electronic hardware

RTCA DO-254, “Design Assurance Guidance for

Airborne Electronic Hardware”, Section 11.4, “Tool Assessment and Qualification” is the starting point

Real Time Safety ERAU College of Engineering

page 4

Literature Study

Perspectives:

• A broad view of the issues in designing CEH (26)
• A focus on safety issues in avionics applications

related to safety critical aspects of CEH (6)

• An industry practices in qualification of CEH tools

and compliance with DO-254 standard (18)

Literature review points:

• Plan to develop and verify PLD programs in the

same way as software programs

• Plan the safety argument from the start, and build

up evidence throughout development

• Use mature tools, amenable to qualification and

supported throughout the project life time

• Investigate the use of formal notations and analysis

techniques to increase verifiability

• Do not use programmable logic hardware just to

avoid developing safety-critical software

Real Time Safety ERAU College of Engineering

page 5

What is a tool?

RTCA DO-254/ED-80 provides guidance for design

assurance of airborne electronic hardware defining design assurance, lifecycle, processes (planning, design, V&V, CM, assurance, certification liaison), and the lifecycle data

No clear definition of a tool in the RTCA DO-254 or

associated CAST papers; paraphrasing DO-178B: “A computer program or a hardware device used to help develop, test, analyze, produce or modify hardware component, subsystem, system or its documentation”

A tool reduces, eliminates, or automates the

• bjectives of the design or verification process

Real Time Safety ERAU College of Engineering

page 6

RTCA DO-254

DO-254 distinguishes between two primary types of tools:

• Design Tools - Tools whose output is part of hardware

design and thus can introduce errors. For example, an ASIC router or a tool that creates a board or chip layout based on a schematic or other detailed requirement (used to generate the hardware item or the hardware design, thus an error in the tool could introduce an error in the hardware item)

• Verification Tools - Tools used to ensure performance

against predetermined standards or requirements. These tools do not introduce errors, but may fail to detect them. For example, an analog or digital circuit simulator or an automated test that measures actual circuit performance. (used to verify the hardware item, an error in the tool may cause the tool to fail to detect an error in the hardware item or hardware design)

Real Time Safety ERAU College of Engineering

page 7

CEH Tools Issues

ASIC/FPGA tools have a thick layer of abstraction between the user

Verilog or VHDL input and the tools output: the tool interprets the input, synthesizes/optimizes the logic, creates net lists, and translates the net list into a hardware layout

Synthesis includes typically the optimization of logic, timing, and

power aspects - a black box, with design as an input and “synthesized” design as an output

FPGA/CPLD vendors are interested in developing software required

to take a design (schematic or HDL) into a form that can be used to program a circuit – they offer cost-effective entry-level design environments (complete packages with design entry, simulators, libraries, and the back end)

Tool industry is dynamic and hardware keeps evolving, the software

developers for back-end tools have to attend to two primary activities, developing libraries for new EDA tools and simulators, while creating better fitters and routers for new hardware with more resources and more complex architectures

Evolving interchange standards such as EDIF and Verilog/VHDL

help standardize interfaces to CAD tools and simulators in form of EDIF-compatible library of design elements using Verilog or VHDL to implement the models necessary for simulation environment

Real Time Safety ERAU College of Engineering

page 8

• 1. Identify the tool
• 2. Identify the process

the tool supports

• 6. Establish

qualification baseline and problem reporting

• 7. Basic tool

qualification

• 9. Design tool

qualification yes yes no no no

• 10. Complete

the process

DO-254/ED-80 Tools Assessment and Qualification Process

• 3. Independent

assessment?

• 4. Tool is design A/B/C
• r verification A/B?
• 5. Relevant tool

history?

• 8. Tool is design tool

A/B?

no yes yes

Real Time Safety ERAU College of Engineering

page 9

Is Tool Qualification Required?

A company working on a design assurance level A

project writes a test bench that runs on a simulator and produces a pass/fail output

• The test bench automates the verification process

and is therefore a tool and thus must be assessed and qualified

Can relevant product service history get us out of tool

qualification?

• Many design tool suites have hundreds of users

doing diverse designs

• Relevant service history requires the tracking of

problem (bug) reports and their resolution

Traceability data for all functional failure paths

It is rare that all of the necessary data is available

Real Time Safety ERAU College of Engineering

page 10

Is Tool Qualification Required ?

(2)

The other way to avoid tool qualification is if the

tool outputs are independently assessed

From DO-254:

Independent assessment of a design tool’s output that is generated in whole or in part by the tool may be established by the verification activities performed on the item, such as component, netlist

• r assembly. In this case, the integrity of the end

item does not depend upon the correctness of the design tool output alone

Real Time Safety ERAU College of Engineering

page 11

Is Tool Qualification Required ?

(3)

The correctness of the design tool can be

assessed by verification tools and in-system hardware verification tests that are run

• Conventional debugging of the hardware using

logic analyzers etc. will also assess the correctness of the design tool

It seems that design tools would rarely need to be

qualified since it is unlikely that the output of the design tool is used without being verified both in simulation and in hardware

Real Time Safety ERAU College of Engineering

page 12

CEH: Logic Design Activities

Design Entry: HDL, schematic entry, integration of

IP cores

Synthesis: translation HDL design definition into

the logical or physical representation for specific hardware platform

Implementation & Configuration: assignment of the

logic created during design entry and synthesis into specific physical resources of the target device.

Verification: design verification ranging from

simulation to static timing analysis to equivalency checking via formal verification.

Board Level Integration: compatibility of

programmable logic design with the entire system.

Real Time Safety ERAU College of Engineering

page 13

Design Entry Synthesis Place & Route Programming Configuration Time Simulation Time Analysis Power Analysis Other Verification Behavioral Simulation Functional Simulation

Generic Design Flow Graph

Real Time Safety ERAU College of Engineering

page 14

The Concern

It is possible for errors to be introduced that occur while

translating the simulated design into hardware

Normal verification techniques will catch most of these

errors during hardware verification

But … some circuits are designed to operate only if the

hardware is malfunctioning Examples are:

• Triply redundant hardware with voting circuits

Used to mitigate failure in one path

• Metastability detection circuits

Can be used to detect single-event upsets due to

alpha particle radiation

These circuits are difficult to verify in working hardware

Real Time Safety ERAU College of Engineering

page 15

Example: Triply Redundant Hardware

Intended Implemented

In A In B

The synthesis tool can “optimize” the design in

an unexpected fashion

Real Time Safety ERAU College of Engineering

page 16

Example: Metastability Sensing Circuits

Intended Implemented

It is possible to detect metastability by

monitoring the Q and Qbar outputs of a single flip-flop

Real Time Safety ERAU College of Engineering

page 17

A Problem

The redundancy and metastability problems occur

when:

• design optimizations are applied during

synthesis, and

• the tool would not generate explicit warnings

that the optimization had occurred

The circuits should not operate if the hardware is

functioning normally, thus error like this may not be detected during verification of the hardware

Real Time Safety ERAU College of Engineering

page 18

Possible Solutions

Since the problems are caused by the synthesizer

performing optimizations, we could turn off all synthesizer optimizations

• It is a possible solution, but it could be difficult to

meet timing and area constraints without

• ptimizations

An experienced designer would recognize these

conditions ahead of time and configure the synthesizer appropriately

• Possible but difficult to verify whether or not the

designer has handled all possible areas of concern

Real Time Safety ERAU College of Engineering

page 19

Possible Solutions (2)

Catch the errors in hardware verification

• Add additional test circuitry to allow verification
• f all circuits

Difficult to anticipate early in the design so that it

can be written into the requirements

• Add additional verification tests to verify circuit
• peration

If the additional circuitry is in place to detect

single-event upsets due to alpha particle radiation we could add radiation susceptibility tests

No current standard for alpha particle radiation

• testing. Should it be in DO-160 rather than

handled in DO-254?

Real Time Safety ERAU College of Engineering

page 20

Possible Solutions (3)

The problems introduced by the synthesizer could

be detected by running simulations using the gate level netlist with back annotated timing data

• Running the full test suite on a gate level netlist

is feasible but it will be very slow and expensive

However DO-254 guidance does not require to do

this:

• Analyzing the implementation below the level of

that specified by the designer, such as at the gate or transistor level, is not necessary …. (DO-254, Appendix B page B-9)

• The designer worked at the HDL level not the

gate level

Real Time Safety ERAU College of Engineering

page 21

Other Likely Problems

At DAL level A and B elemental analysis is used to

assure that all functional failure paths have been covered

However, some failure mechanisms can exist

which occur at conditions that may not be identified as functional failure paths

Some failures are related to where within the

FPGA the actual hardware is located during the place and route phase

Real Time Safety ERAU College of Engineering

page 22

Non-Functional Failure Path Concerns

Two problems can be caused if large numbers of

devices within the FPGA connected to the same internal FPGA power supply switch at the same time

Resistance in the internal FPGA power bussing

can cause the voltage provided to circuits in a particular location to drop

Inductance in the FPGA power bussing can create

Simultaneous Switching Noise if many outputs switch simultaneously

Both mechanisms can cause errors directly or

change the circuit timing leading to errors elsewhere

Real Time Safety ERAU College of Engineering

page 23

Non-Functional Failure Path Concerns (2)

Much of the information needed to analyze these

conditions is proprietary to the FPGA vendor

Vendor supplied FPGA tools are the best at

addressing these issues

However our survey indicates that the majority of

designs are implemented using third party tools such as Mentor Graphics and Synplicity

Real Time Safety ERAU College of Engineering

page 24

Industry Survey - Methodology

Questionnaire distributed during the 2007 FAA

SW&CEH Conference, New Orleans, LA, July 24-26, 2007 on a dedicated CEH session attended by 54 individuals interested in the CEH and the application of DO-254

Posting the questionnaire on the web and linking from

www.do254.com website

Follow-up mailing to over 150 individuals engaged in

development of the aviation software and hardware

The questionnaire distributed internally in companies

engaged in the design of programmable logic devices

Collection of feedback at the Programmable Logic User

Group meeting in Clearwater, FL, November 15, 2007

As a result we obtained a sample of 45 recorded

responses

Real Time Safety ERAU College of Engineering

page 25

Industry Survey - Qualification

Only eight out of 45 respondents indicated that tool

qualification was attempted:

The responses were inconsistent and incomplete; here

is not verified list:

• Aldec – level A
• Synplify – level C, level A
• Questa – level A
• ModelSim – level A
• Actel Libero – level A

Two respondents listed several tools but without clear

statement if they were actually qualified and without specifying the qualification level/type (qualification activities have been performed and documented but never reviewed or challenged by the FAA)

Real Time Safety ERAU College of Engineering

page 26

Survey: Tool Landscape

Type of Devices

fpga 32% asic 18% cpld 18% soc 8% pla/pal/gal 24%

Vendors of Hardware Devices

actel 20% atmel 6% altera 22% cypress 11% lattice 12% quicklog 5% xilinx 24%

Vendors of Software Tools

synopsis 19% synplicity 19% mentor 27% aldec 9% cadence 8%

• thers

18%

Errors Encountered

yes 48% no 52%

Real Time Safety ERAU College of Engineering

page 27

Survey: Population

Educational Background

SW 17% HW 83%

Type of Work

tools 3% application 36% verification 15% proj.mgt 25% components 21%

Years of Expertise

<3 years 12% 3-6 years 19% 7-12 years 19% >12 years 50%

Work on DO-254 Projects

yes 33% no 67%

Real Time Safety ERAU College of Engineering

page 28

Industry Survey – Selected Comments

• We qualified a tool by comparing a logic analyzer trace from the actual system

with a simulation from the tool (including propagations delays and functionality). The DO-254 guidance is not useful and does not include what should be done with the data. Most of the tools are widely in use and do not have a negative safety impact. Mainly the repeatability and verification of the design itself affects safety

• All companies are attempting qualification differently and we have not seen any

increase in safety by doing a qualification. These tools are in use in thousands of designs in the world and the probability of a tool qualification effort finding an error is very low or negligible

• The tools are designed for rapid freeform development and not for requirements

based methodical design. Fine for commercial applications but not really targeted to high reliability design. Qualification is basically demonstrating the pedigree of the tool itself, most vendors either do not keep requirements based design and verification information or that information is considered proprietary or trades

• secret. Thus independent assessment of tool output will continue to be the

primary, and sometimes only, methodology available for “qualification”. As long as that fact doesn’t change, there will be very little progress

• Not having specific guidelines, there is too much uncertainty on what needs to be

done to assess or qualify the tools

• The issue is about the user competence about the task being performed - do

they know how to appropriately apply the tool? Guidance is extremely vague on the objectives of qualification. The use of the word "tool" is somewhat misleading. How the tool is applied is sometimes more of a risk than the tool itself. E.g. if a verification environment consists of a simulation testbench running on Modelsim, the tool itself should be less of an issue for assessment/qualification than the user developed simulation testbench

Real Time Safety ERAU College of Engineering

page 29

Observations

Vendor supplied FPGA tools are best at identifying

failures that require knowledge of the FPGA internals

Vendor supplied design tools are not commonly

used design tools

Vendor supplied tools often cannot close timing on

designs that are easily handled by third party tools

Vendor supplied tools can take advantage of

detailed knowledge of the hardware to implement unique features

Real Time Safety ERAU College of Engineering

page 30

Summary

For many designs normal the normal verification

process activities are sufficient to independently assess the tool outputs

But some design problems are difficult to assess

and require extra effort to identify and address

• Requires knowledgeable designers and DERs
• The risk is people do not know what they do not

know

DAL level A FPGA design and verification requires

an experienced team of design and verification engineers

Gate level verification of the design appears to be

necessary

Real Time Safety ERAU College of Engineering

page 31

Moving Forward

We are executing several test cases to investigate

issues including:

• Non Functional Failure Path issues such as

Simultaneous Switching Noise

• Comparing tool reported device timing to actual

measured timing to evaluate design robustness

• Un-programmed circuitry operation: observe

whether or not un-programmed logic blocks switch during normal device operation and what are the implications with respect to power supply and signal integrity

Real Time Safety ERAU College of Engineering

page 32

Questions?

Questions, comments, suggestions?

Andrew J. Kornecki

kornecka@erau.edu

Brian Butka

butkab@erau.edu

Janusz Zalewski

zalewski@fgcu.edu