Program 13:00 Welcome and introduction 13:20 Research progress on - - PDF document

26/05/2016 1

IWT-Tetra project User group meeting 13 May 2016

Program

• 13:00 Welcome and introduction
• 13:20 Research progress on RGB+LWIR pedestrian

detection

• 14:20 Hardware update and geometrical calibration issues

and solutions

• 14:45 Rule-based reasoning with real-life application demo
• 15:30 Discussion and planning of in-the-field tests
• 16:00 Conclusions and future work

Updated industrial users group

SPINOFF

Project abstract

• Camera-based safety and security

systems

• Real-time reaction on incidents?
• Manual monitoring
• Automatic processing and incident

detection

• Needed components:

1.

Very reliable detection of persons in camera images

2.

Reasoning system that can decide if an alarm must be generated

Enabling factors

• State-of-the-art person detection algorithms show

astonishing results

• Accuracy great on standard benchmark data sets
• EAVISE succeeded in running these in real-time on

limited hardware

• Both open source and commercial-grade

implementations available

• Price of LWIR-cameras descends steeply, with increasing

resolution

• Knowledge-representation based probabilistic reasoning
• ffers potential to analyse each situation

Project idea

• Making people detection reliable,

also in difficult circumstances (fog, smoke, rain, dust, motion blur, …):

• Combine RGB and LWIR

camera

• Adapt state-of-the-art person

detection algorithms for this sensor combination

• Use probabilistic KR for analysis
• f situation: must an alarm be

generated?

26/05/2016 2 Project goals

• Developing a sensor combination and software for ultra-

reliably detecting people in real-time

• Composing a real-life reference image database for

evaluating person detection techniques in difficult circumstances

• Studying techniques for automatic analysis of the observed

situation and classification as normal or abnormal

• Studying the certification procedure for camera-based

safety and security systems

• The demonstration and dissemination of the project

results via 5 real-life user cases

• Supporting industrial companies to adopt the developed

techniques in their products and services

People@VIPER

• Prof. Toon

Goedemé

• Prof. Joost

Vennekens (Dr.) Kristof Van Beeck Andy Warrens Kristof Van Engeland

• Dr. Floris De

Smedt New employee

Work packages and progress

WP1: Hardware WP4: Evaluation and dissemination 1.A Study on sensors 1.B Hardware imple- mentation & calibration 1.C Benchmark database WP2: Person detection 2.A Study on algorithms 2.B Person detection SW implementation 2.C Evaluation on Benchmark database WP3: Alarm system 3.A Study on AI 3.B Learning of ranking 3.C Online learning 3.D Evaluation 4.A User Cases 4.B Evaluation and documentation 4.C Study on certification and legal aspects 4.D Broad dissimination

Planning

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 WP1.A: Study on sensors WP1.B: Hardware realisation and calibration WP1.C: Benchmark database

MP8

WP2.A: Study on algoriths for person detection WP2.B: Implementation algorithm person detection

MP6

WP2.C: Evaluation on benchmark database

MP7

WP3.A: Study on learning alarm system WP3.B: Learning of ranking WP3.C: Online learning WP3.D: Evaluation on benchmark database

MP7

WP4.A: User cases

MP1 MP2 MP3 MP4 MP5

WP4.B: Evaluation and documentation

MP7

WP4.C: Certification & legal aspects

MP9

WP4.D: Broad dissemination and networking

MP9

Program

• 13:00 Welcome and introduction
• 13:20 Research progress on RGB+LWIR pedestrian

detection

• 14:20 Hardware update and geometrical calibration issues

and solutions

• 14:45 Rule-based reasoning with real-life application demo
• 15:30 Discussion and planning of in-the-field tests
• 16:00 Conclusions and future work

Program

• 13:00 Welcome and introduction
• 13:20 Research progress on RGB+LWIR pedestrian

detection

• 14:20 Hardware update and geometrical calibration issues

and solutions

• 14:45 Rule-based reasoning with real-life application demo
• 15:30 Discussion and planning of in-the-field tests
• 16:00 Conclusions and future work

26/05/2016 3

Research progress on RGB- LWIR pedestrian detection

Overview

• How does pedestrian detection work?
• KAIST dataset
• Performed experiments: comparitive study
• ACF – ACF+
• Amount of training data
• Trained model size
• Resolution of LWIR-image (simulate lower resolution sensor)
• Training set - Testing set

How does pedestrian detection work?

Pedestrian detection approach

16

• Create a model for pedestrians
• Examples of positives (pedestrians)
• Examples of negatives (non-pedestrians)
• Convert to feature representation
• Good distinction between pedestrians and

background

• Robust for scene changes (e.g. illumination)
• Train a model
• Machine Learning: Adaboost, Support Vector

Machines, Neural networks, …

• Distinction between “Pedestrian” and “Background”
• Intra-class variation: pedestrians can have many

appearances

Pedestrian detection approach

At every location…and multiple scales  sliding window

17

Search the model in the image features (Sliding Window):

…

• Calculate features at multiple

scales  Feature pyramid

• Similarity between the model and

the features forms the certainty of a pedestrian at that location

• A threshold defines the boundary

between “background” and “detection”

• Non-Maximum-Suppression
• Sliding window results in

clusters of detections around pedestrians

• NMS reduces this to only

the highest scoring detection

• f each cluster

87.81 68.71 68.46 26.89 8.405

Pedestrian detection approach

26/05/2016 4 Influence of the threshold value

19

Low threshold

• More pedestrians

found

• More mistakes

High threshold

• Less pedestrians

found

• Less mistakes

Measure accuracy

Precision vs. Recall Miss rate vs. False Positives per Image

20

Miss Rate: The share of pedestrians that is not found FPPI: Average number of false detections (non- pedestrian) per image Best point: bottom left Recall: share of pedestrians found Precision: share of detections that is a pedestrian Best point: top right

Used detector

21

• Channel based detectors
• Use both gradient and color information
• Feature values are calculated as the sum of pixel values in

rectangles

• AdaBoost Machine Learning
• Integral Channel Features [1]
• 30 000 random rectangles inside model window
• Each stage (2000) is a decision tree of features
• Aggregate Channel Features [2]
• Approximation of the features at most scales
• All possible squares of a specific size inside the model

window

[1] “Integral Channel Features”, Dollàr, Tum Perona and Belongie, BMCV 2009 [2] “Fast Feature Pyramids for Object Detection”, Dollàr, Appel, Belongie and Perona, PAMI 2014

KAIST dataset

Dataset: KAIST

• Color and LWIR
• for both day and night conditions

Previous work

24

• Results from literature
• “Multispectral pedestrian detection: Benchmark dataset

and baseline” CVPR 2015

• Add channels

calculated on LWIR

• Pixel information
• Gradient magnitude
• Gradient orientations
• Night experiments
• Reasonable
• ≥ 50px
• ≥ 65% visible
• Each 30th image

IR

26/05/2016 5

Performed experiments

• A. Warrens

RGB to LWIR extension

• Easy extension of existing RGB detectors to LWIR
• Avoids retraining
• Based on [3]

[3] “Far infra-red based pedestrian detection for driver-assistance systems based on candidate filters, Gradient-based filters and multi-frame approval matching”, Wang and Liu, December 2015

RGB to LWIR extension

• Start from an existing RGB-based detector
• ACF
• Classify as pedestrian if a peak in intensity takes place on

the LWIR-image

RGB to LWIR extension

73% reduction in false positives

RGB to LWIR extension

• This approach can not improve the recall, only the

precision

• Very limited range of ACF-detections used
• A larger range will increase processing time!
• “Resolution is not sufficient to distinguish pedestrians from

road” (A. Warrens)

• We need a stronger approach than “Thresholding”!

Performed experiments

• Dr. F. De Smedt

26/05/2016 6 Performed experiments: overview

• Our internal developed framework (PeopleDetect)
• Detection code
• Training code
• Extended to cope with LWIR-input
• Read RGB- and LWIR-images as separate inputs
• Determine influence of
• ACF – ACF+
• Amount of training data
• Trained model size
• Resolution of LWIR-image (simulate lower resolution sensor)
• Training set - Testing set

ACF – ACF+

ACF

• # weak classifiers
• 2048
• Depth of decision trees
• 2
• # negatives
• 5 000
• # accumulated negatives
• 10 000

ACF+ [4]

• # weak classifiers
• 4096
• Depth of decision trees
• 5
• # negatives
• 25 000
• # accumulated negatives
• 50 000

[4] “Local Decorrelation for Improved Pedestrian Detection”, Nam, Dollàr and Han, Advances in Neural Information Processing Systems 2014

ACF – ACF+

76,94% reduction in false detections 48,2% reduction in false detections 57,47% reduction in false detections

• We improve drastically over the state-of-the-art!
• ACF+
• Only 1405 weak classifiers used
• ACF+ does not use full potential

Amount of training data

• Influence of increasing the training material (5x) is negligible in

accuracy (but not in training time)

Trained model size

• According to [5] this could improve detection accuracy 50px 60px
• We used 50px 100px
• Influence of increasing the model size is negligible
• Requires upscale of the image
• More or less 3x slower detection time!

[5] “Filtered Channel Features for Pedestrian Detection” Zhang, Benenson and Schiele, CVPR 2015

Resolution of LWIR-image

• KAIST-dataset cameras
• RGB
• PointGrey Flea3
• 640x480
• LWIR
• FLIR-A35
• 320x256
• Later more on calibration approaches between Color

and LWIR

26/05/2016 7 Resolution of LWIR-image

• Simulate low resolution LWIR sensor (naïve approach)
• Downscale the LWIR-image
• What about bit-depth/range/…?
• Future experiments with low-cost/low-resolution sensors

will show the correctness of this simulation!

• Evaluation on each image (instead 30th) for more accurate

comparison

Resolution of LWIR-image

76,66% Smoothing

• r just

lucky?

• NOTE: the original resolution of the LWIR-camera is only 320x256
• Even at low resolutions the LWIR is very beneficial

Still improvement

• ver state-of-the-

art

Resolution of LWIR-image

Rescale Resolution ACF (# Weaks used) ACF+ (# Weaks used) Full (no rescale) 640x480 2048 1405 0.5 320x240 2048 1640 0.4 256x192 2048 1647 0.3 192x120 2048 1620 0.2 128x96 2048 1716 0.1 64x48 2048 2001

Training set – Testing set

• What is the influence of the trainingdata on the detection

accuracy?

• Night vs Day vs Both
• Is detection more accurate when trained for a specific

setting (e.g. night training for night evaluation)

• Comparison of the feature selection

Training set – Testing set (Day) Training set – Testing set (Night)

26/05/2016 8 Training set – Testing set (All)

Training set – Testing set (model comparison)

0,05 0,1 0,15 0,2 0,25 0,3 0,35 0,4

Color Raw pixel Color Gradient Magnitude Color HOG LWIR Raw pixel LWIR Gradient Magnitude LWIR HOG

Normalised weight Channel type

Comparison of used features per model (ACF+)

Night Day All

Conclusion

• We use the ACF detector
• Channel-based allows easy extension with LWIR
• Own implementation of training and detection code
• Comparative study of settings
• KAIST dataset
• ACF vs ACF+
• Even with a default training setup we improve drastically over the

current best published results

• We gain a lot in detection accuracy by using a stronger model

(ACF+)

• ACF+ is “to strong” for this detection task (not all weak-classifiers

used)

Conclusion

• Comparative study of settings
• Amount of training data used
• We compared training with “each 20th image” and “each 4th

image”

• Only a small accuracy gain reached
• Trained model size
• Literature has shown examples where a larger model better

exploits the data

• We do not see a similar accuracy gain
• Resolution of LWIR-image
• Naïve simulation of low-resolution sensors
• Also on lower resolutions we reach similar accuracy results

Conclusion

• Comparative study of settings
• What is the influence of the trainingdata on the detection

accuracy?

• Night vs Day vs Both
• Do we gain accuracy by training for a specific setting?
• Night training captures most important information
• Performs also good on day-images
• Training on both generalizes well for both day and night

images

• Using a separate models for day and night will not improve

accuracy

• We see a shift of feature selection from color to LWIR

when using more night images

Future work

• Improve accuracy further?
• Local Decorrelation Features, Filtered Channel

Features, …

• Will require longer detection times
• What is the complementarity between day/night model
• The Combinator?
• Improve localization (Movie)
• Adding scene constraints (ground constraint)
• Use more applied datasets from proposed applications
• Top-down view:
• perspective transformation

rigid detector adaption

• Low-cost sensors

26/05/2016 9 Future work

• How generic are the trained models?
• Currently trained in a similar environment as evaluation
• What if we use the trained model in other Color-LWIR

image pairs? (Other datasets)

• How well perform the “high resolution” models on low-

resolution inputs

• Further suggestions?

Movie

Questions?

Hardware and calibration

Overview:

• 1. Selected cameras
• 2. Interfacing
• 3. Calibration

Study on IR sensors

 Advantages:  Better people detection

through body heat sensing

 Sees clearly in complete

darkness without any illumination

 Works in bright sunlight,

through smoke, dust or even light fog.

 Disadvantages:  Expensive  Low image resolution

Study on IR sensors

heat wave radiation

26/05/2016 10 Market study (1/2)

Manufacturer Name Price Resolution Omron D6T-8L-06 41.38 € 1x8 Omron D6T-44L-06 42.59 € 4x4 Melexis MLX90621 (obsolete) 50.43 € 16x4 Melexis MLX90620 59.55 € 16x4 FLIR Lepton 500-0643-00 160.93 € 80x60 FLIR Lepton 500-0659-01 168.29 € 80x60 FLIR Lepton 500-0690-00 168.29 € 80x60 Seek Thermal Compact 299.00 € 206x156 Seek Thermal Compact XR 349.00 € 206x156 Fluke Ti90 1,195.99 € 80x60 FLIR Tau2 160 1,400.00 € 160x128 FLIR Tau2 168 1,400.00 € 168x128 FLIR Tau2 162 1,400.00 € 162x128 FLIR FLIR Vue 336 1,499.00 € 336x256 DRS Technology Tamarisk 320 1,894.43 € 320x240 Fluke Ti100 1,995.99 € 160x120

Market study (2/2)

Manufacturer Name Price Resolution FLIR Quark 336 2,500.00 € 336x256 FLIR Quark 640 2,500.00 € 640x512 FLIR Tau2 324 2,500.00 € 324x256 FLIR Tau2 336 2,500.00 € 336x256 FLIR Tau2 640 2,500.00 € 640x512 FLIR FLIR Vue 640 2,999.00 € 640x512 Mobotix FlexMount S15 3,634.00 € 336x252 Acal Tamarisk 320 3,674.82 € 320x240 COX CX320 3,877.14 € 320x240 COX CX320 3,877.14 € 384x288 Mobotix AllroundDual M15 3,998.00 € 336x252 Fluke Ti200 5,495.99 € 200x150 COX CX640 5,503.03 € 640x480 Fluke Ti300 6,195.99 € 240x180 Xenics Gobi-384 7,500.00 € 384x288 Fluke Ti400 7,995.99 € 320x240

Market study

 Selection from market study

Name Price Resolution Sensolid sensor with Lepton 3 ??? 160×120 SEEK Thermal Compact 299,00 € 206x156 FLIR AX8 840 € 80x60 IR (640x480 RGB) FLIR ThermiCam Lepton ??? 160×120

Camera Interfacing

Name Available? Interface Working? Sensolid sensor with Lepton 3 Yes USB Yes, but proprietary format and only in Windows SEEK Thermal Compact Yes USB Yes, on Android and in PC with libseek API on Linux (no autoscale yet) FLIR AX8 No Ethernet/IP, H264 n.a. FLIR ThermiCam Lepton Yes GigE - RTSP No

Geometrical Calibration

• We must know wich pixels of

the RGB camera correspond to which pixels of the IR camera

• Difficult because of baseline
• Standard checker board does

not work

• Not visible in IR

Geometrical calibration

26/05/2016 11 Program

• 13:00 Welcome and introduction
• 13:20 Research progress on RGB+LWIR pedestrian

detection

• 14:20 Hardware update and geometrical calibration issues

and solutions

• 14:45 Rule-based reasoning with real-life application demo
• 15:30 Discussion and planning of in-the-field tests
• 16:00 Conclusions and future work

How to combine RGB person detection with IR cameras?

Multiple options:

• 1. Person detection algorithms on IR data
• 2. IR preprocessing + person detection on RGB images
• 3. Person detection on RGB + IR verification
• 4. Integrated IR+RGB person detection

IR RGB

Integrating IR in RGB person detection

• Example: ICF detector

IR

Rule-based Reasoning with Real Life Application Demo

Kristof Van Engeland

Overview

• Motivation
• Goals
• Concept
• Decision Engine: Problog
• Use Case 1: Hygiene Monitor
• Requirements
• Description
• Pipeline
• Demo
• Conclusions and future work
• Questions

Motivation

• VIPER brings Computer Vision and AI together
• CV: Make existing pedestrian detecting algorithms more

robust (see Floris De Smedt, et al.)

• AI: Interpret visual data with a rule-based automatic

decision making system

26/05/2016 12 Goals

• Interpretation of situations happening on camera
• Make weighted decisions based on interpretation
• Automatic, yet correctable
• Robust and flexible code base

Concept

• “Layered cake” of features
• Higher levels  more

expensive to calculate

• Make decisions based on

information from every level

• Result: faster and more

efficient system

Decision Engine: Problog

• Imperative languages

describe the solution of a problem

• Problog =

Declarative Language

• Describe the problem

itself!

• Very modular
• Easy to add and remove

rules

• Probabilistic Logic Learning

Use Case 1: Description

• Goal: prototyping entire workflow
• Requirements:
• Easy access to test environment
• Multiple possible interactions with environment
• Multiple scenarios
• Related to a real, practical use case
• Relatively simple

Use Case 1: Description

• Research Question: how many people leave the bathroom

with dirty hands after a visit?

• Test environment: bathroom
• Possible interactions with:
• Door, sink, urinal, toilet, …
• Several possible scenarios:
• Wash hands only
• Use urinal / toilet only
• Uses urinal / toilet and wash hands
• …
• Not overly complex!

Use Case 1: Pipeline

Image Processing Data Transformation Problog Stream Generation Problog Inference

26/05/2016 13 Use Case 1: Pipeline

load calculate centers calculate distances MA filter threshold calculate average positions split setup stream events ↕ locations generate Problog: facts + evidence facts and evidence application specific logic

Features Used

N/A

begin/end sequence, position of person begin/end movement, location comparison sequence of events

Problog

Sample screenshot Sample input

fnr x_feet y_feet x_head y_head 159 103 359 32 236 1 160 118 359 44 239 2 161 122 364 46 243 3 162 116 359 67 261 4 163 144 364 81 264 … … … … … … 1357 1624 107 346 22 229 1358 1625 98 346 16 239 1359 1626 90 348 14 243 1360 1627 94 349 18 236 1361 1628 86 350 15 224

Movement between two frames Moving Average Filter

26/05/2016 14 Thresholding Split per person Schematic overview

T IN U OUT W

P(dirtyhands)

IN: enters T: toilet U: urinal W: washbasin OUT: exits

Problog code (core) Problog Code (generated)

Person 1 Person 2 Person 3

Online demo click here

26/05/2016 15 Demo: Showcase Demo: Results Use Case 1: Conclusions

• Output is relatively in line with expectations
• Real value of output depends on input parameters
• Generated events and evidence are reusable with different

application logic

Use Cases: Future Work

• Refinement of the pipeline:
• Support for streaming
• Better UI to define objects and rules
• Integration with detection algorithm itself
• Real Use Cases!
• Suggestions?

Thanks for listening!

Any questions?

Program

• 13:00 Welcome and introduction
• 13:20 Research progress on RGB+LWIR pedestrian

detection

• 14:20 Hardware update and geometrical calibration issues

and solutions

• 14:45 Rule-based reasoning with real-life application demo
• 15:30 Discussion and planning of in-the-field tests
• 16:00 Conclusions and future work

26/05/2016 16 Cases

Test case Industrial partner Complexity image processing Safety critical Complexity AI Responsa- bility of the system Security from fixed cameras Flir ITS * * ** * Security from UAVs DroneMatrix, Xtendit, Airobot, EUKA *** * ** * Safety from fixed cameras Havenbedrijf Antwerpen, WenZ, .. * *** * ** Safety from manned driving vehicles CNHi, GrootJebbink ** *** * * Safety from unmanned driving vehicles Octinion ** *** * *** Patient monitoring Sensolid, Alphatronics * ** *** **

GrootJebbink case

• Person’s safety around large vehicles
• First demo: real-time detection of persons

behind a construction vehicle

• Presented at BAUMA 2016 trade fair for

construction machinery in München 11-04- 2016 t/m 17-04-2016

Alphatronics case

Eerste testen Alphatronics case

Seek thermal camera, 1m boven bed

CNHi use case proposal 19th November 2015 95

• Person detection in dust for agricultural (larger crops

particles in Ag dust) and construction vehicles

• Person detection for commercial vehicles in difficult

environmental condition with high reliability (>99,xx %?) in:

• Blind spot all-around vehicle,
• Vehicle path.
• Data acquisition equipment and sensors from EAVISE.

CNHi acquire data in real condition.

VIPER project: interesting use cases for CNHi Bediening van beweegbare bruggen op afstand via camerabewaking: personendetectie op het brugdek

26/05/2016 17 Practical issues

• All feedback is always welcome via mail/tel/…
• Website: www.eavise.be/viper
• IWT/VLAIO e-tool “gebruikerspoll”
• Collects feedback after every user group meeting
• GitLab group

Thank you for you attention!

Contact:

toon.goedeme@kuleuven.be Joost.vennekens@kuleuven.be