Modeling and Control of Dynamic Systems Validation Darya - PowerPoint PPT Presentation

Modeling and Control of Dynamic Systems Validation Darya Krushevskaya Konstantin Tretyakov

Introduction � Model evaluation Experiment Experiment � In accordance with intended use of the model Select model Select model � Investigate particular � Investigate particular structure structure structure structure property Estimate model Estimate model Validate model Not accepted Accepted

Data � Test or validation set � Not used during training � Cross-validation � Partitioning of the data into subsets � Partitioning of the data into subsets

Validation 1. Evaluation of the residuals � Tests for correlation 2. Estimation of the average generalization error error 3. Visualization of the model’s ability to predict � Graphical comparison

Tests for Correlations I � Residuals should be uncorrelated with all linear and nonliniar combinations of past data � Complete test is unrealistic � Consider only few tests � Consider only few tests

Correlation Tests

Tests for Correlations II � Calculate correlation functions �(τ) � If the data are indeed uncorrelated, the values �(τ) are asymptotically normal with 1 1 distribution : distribution : � � ( ( 0 0 , , ) ) � � This suggests a simple statistical test τ ∈ [ − (| �(τ) | < 1.96/N ) for 20 , 20 ]

NNARX demo

NNARX demo

NNARX demo

NNARX demo

NNARX demo

NNARX demo

NNARX demo

NNARX demo

Estimation of the average generalization error

Visualization of the Predictions � Shows variation in accuracy of the prediction � Can show overfitting and possible systematic errors

Visualization of the Predictions � Underparametrized model

Visualization of the Predictions � Overparametrized model

Prediction intervals � Estimating reliability of predictions for a specific input � S ∈ M � Variance of the prediction error of regression � Variance of the prediction error of regression vector φ(t):

NNATX model evaluation � A 95% confidence interval is drawn

K-step ahead predictions � In case of fast sampling ≈ − y ( t ) y ( t 1 ) � Check that ŷ(t|�)�=�y(t�1) � K-step ahead prediction � K-step ahead prediction

K-step prediction demo

Summary � Model validation � Correlation functions � Estimation average generalization error � Visualization of predictions � Visualization of predictions

Variance � S ∈ M , thus � The covariance matrix:

The Noise variance � The noise variance: