Program 13:00h welcome and introduction (Toon Goedem) 13:30h Status - - PDF document

SLIDE 1

8/12/2016 1

IWT-Tetra project User group meeting December 8th, 2016

Program

• 13:00h welcome and introduction (Toon Goedemé)
• 13:30h Status update hardware and calibration (Bjorn Van

Tilt / Toon Goedemé)

• 14:00h Status update abnormal behaviour detection

(Kristof Van Engeland)

• 14:45h Coffee break
• 15:00h Status update person detection algorithms (Kristof

Van Beeck)

• 15:45h Discussion and further planning (Joost Vennekens)

Updated industrial users group Project abstract

• Camera-based safety and security

systems

• Real-time reaction on incidents?
• Manual monitoring
• Automatic processing and incident

detection

• Needed components:

1.

Very reliable detection of persons in camera images

2.

Reasoning system that can decide if an alarm must be generated

Enabling factors

• State-of-the-art person detection algorithms show

astonishing results

• Accuracy great on standard benchmark data sets
• EAVISE succeeded in running these in real-time on

limited hardware

• Both open source and commercial-grade

implementations available

• Price of LWIR-cameras descends steeply, with increasing

resolution

• Knowledge-representation based probabilistic reasoning
• ffers potential to analyse each situation

Project idea

• Making people detection reliable,

also in difficult circumstances (fog, smoke, rain, dust, motion blur, …):

• Combine RGB and LWIR

camera

• Adapt state-of-the-art person

detection algorithms for this sensor combination

• Use probabilistic KR for analysis
• f situation: must an alarm be

generated?

SLIDE 2

8/12/2016 2 Project goals

• Developing a sensor combination and software for ultra-

reliably detecting people in real-time

• Composing a real-life reference image database for

evaluating person detection techniques in difficult circumstances

• Studying techniques for automatic analysis of the observed

situation and classification as normal or abnormal

• Studying the certification procedure for camera-based

safety and security systems

• The demonstration and dissemination of the project

results via 5 real-life user cases

• Supporting industrial companies to adopt the developed

techniques in their products and services

People@VIPER

• Prof. Toon

Goedemé

• Prof. Joost

Vennekens

• Dr. Kristof

Van Beeck Andy Warrens Kristof Van Engeland

• Dr. Floris De

Smedt Maarten Vandersteegen

Work packages and progress

WP1: Hardware WP4: Evaluation and dissemination 1.A Study on sensors 1.B Hardware imple- mentation & calibration 1.C Benchmark database WP2: Person detection 2.A Study on algorithms 2.B Person detection SW implementation 2.C Evaluation on Benchmark database WP3: Alarm system 3.A Study on AI 3.B Learning of ranking 3.C Online learning 3.D Evaluation 4.A User Cases 4.B Evaluation and documentation 4.C Study on certification and legal aspects 4.D Broad dissimination

Planning

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 WP1.A: Study on sensors WP1.B: Hardware realisation and calibration WP1.C: Benchmark database

MP8

WP2.A: Study on algoriths for person detection WP2.B: Implementation algorithm person detection

MP6

WP2.C: Evaluation on benchmark database

MP7

WP3.A: Study on learning alarm system WP3.B: Learning of ranking WP3.C: Online learning WP3.D: Evaluation on benchmark database

MP7

WP4.A: User cases

MP1 MP2 MP3 MP4 MP5

WP4.B: Evaluation and documentation

MP7

WP4.C: Certification & legal aspects

MP9

WP4.D: Broad dissemination and networking

MP9

Deliverables

• Scientific publications:
• Van Beeck, Kristof, Goedemé, Toon; The Automatic Blind Spot

Camera: A Vision-Based Active Alarm System; European Conference on Computer Vision Workshops; Computer Vision – ECCV 2016 Amsterdam, The Netherlands

• Van Beeck, Kristof; The automatic blind spot camera: hard real-time

detection of moving objects from a moving camera; PhD thesis KU Leuven; 2016.

• Floris De Smedt and Toon Goedemé, How to reach top accuracy

for a visual warning system from a car, The sixth International Conference on Image Processing Theory, Tools and Applications (IPTA'16), Oulu, Finland

• Vulgarizing publications:
• Article in “Motion Control”: TRENDS EN ONTWIKKELINGEN BIJ

VISIESYSTEMEN; EAVISE ONDERZOEKT NIEUWE TECHNOLOGIEEN VOOR INDUSTRIELE TOEPASSINGEN

Master theses

• Bjorn Van Tilt: “Calibratiesysteem van gecombineerde

kleurencamera - IR sensor voor persoonsdetectie”

• Bert Dewinter: “Ontwikkeling van een persoonsdetector op

FPGA”

• Nele Allaert: “Automatische detectie van patientengedrag

in hun slaapomgeving”

• Mariia Zakharova: “People detection from drones with

RGB+IR cameras”

SLIDE 3

8/12/2016 3 Application cases

• Men’s room hygienic guard (intern)
• Detecting people on train rails (FLIR)
• Vehicle blind spot (GrootJebbink)
• Patient monitoring in bed (AlphaTronics)
• Safety around agricultural vehicles (CNHi, Octinion)
• Safety on bridges and locks (Port Antwerp, WenZ)
• Drone surveillance (DroneMatrix)

Program

• 13:00h welcome and introduction (Toon Goedemé)
• 13:30h Status update hardware and calibration (Bjorn Van

Tilt / Toon Goedemé)

• 14:00h Status update abnormal behaviour detection

(Kristof Van Engeland)

• 14:45h Coffee break
• 15:00h Status update person detection algorithms (Kristof

Van Beeck)

• 15:45h Discussion and further planning (Joost Vennekens)

Calibration of combined colour and IR-camera for pedestrian detection

Bjorn Van Tilt Toon Goedemé

Contents

• Situation
• Objectives
• Thermal camera
• Calibration and mapping
• Reading temperatures
• Future work

Situation

• Pedestrian detector
• Difficult scenario’s (night, fog, dust, …)
• Adding Thermal images to detectors
• Thermal camera’s getting cheaper
• But low resolutions

Objectives

• Using cheap thermal camera and webcam
• Seek Thermal Compact (206x156, 35°)
• Pleomax PWC-7300 Crystal Webcam

(1280x1024, 35°)

• Reading and synchronizing cameras
• Calibration of both cameras
• Removing lens distortion
• Mapping cameras on each other
• Calibration of IR to read temp
• Capture different scenarios

SLIDE 4

8/12/2016 4 Seek Thermal Compact

• Cheaper IR camera
• 206x156px
• For smartphones
• No Linux driver

Raw data After calibration images Dead pixel filter Removed thermal gradient

Calibration and mapping

• Something needed to be seen in

both camera’s

• Board with metal circles
• Resistors behind every circle for

heat

• Using 100 frames

Finding calibration pattern

B1

Removing lens distortion Mapping both cameras

• Fit homography
• Transform one world point from RGB to IR frame
• Disadvantage: Uses planes => no depth
• Not big issue => Cameras are mounted close to each
• ther

Calibration and mapping

From normal camera To LWIR camera

SLIDE 5

8/12/2016 5 Calibration and mapping

1 2 3 4 5 6 7 8 2 4 6 8 10 12 14 Distance [m] error [pixels]

Average pixel error

Avg error

Temperature readings

• Never accurate. Highly depended on

material properties

• Hard because camera isn’t held at specific temperature

Output Future work

• Capturing different scenarios
• Inside and outside
• Normal daylight
• Night time
• During fog
• During rain
• Test pedestrian detection algorithm on this data

Thank you for you attention!

Contact:

toon.goedeme@kuleuven.be Joost.vennekens@kuleuven.be

SLIDE 6

08/12/2016 1

Detecting abnormal behaviour in public locations

Kristof Van Engeland – EAVISE

Use Case 2: platform surveillance

• motivation: people behave dangerously around railroad tracks
• running around
• crossing the tracks
• suicide
• …
• but: specific impulsive behaviour is very difficult to predict!
• tough to define and/or acquire “positive” training samples
• movements can be very erratic
• detection and tracking not always accurate enough

Research scope

• Requirements:
• distinction: “normal” “abnormal” behaviour
• (semi-)automatic operation
• Questions:
• how much “history” do we need to track?
• desired accuracy? precision – recall?
• real-time possible?

Example video Pipeline proposal

• find people
• track movement
• find people
• track movement

Detection

• split camera view
• initialize model
• split camera view
• initialize model

Preparation

• map tracks to

regions

• train model with

tracks

• map tracks to

regions

• train model with

tracks

Training

• determine boundary

between categories

• filter paths
• determine boundary

between categories

• filter paths

Classification

• evaluate parameters
• repeat training with

new parameters

• manual adjustment
• evaluate parameters
• repeat training with

new parameters

• manual adjustment

Post- processing

Detection and Tracking

• sequence (track) created for every

person

• person leaving and reentering screen

counts as new track

• more info:

see presentation by Kristof Van Beeck

SLIDE 7

08/12/2016 2

Sequence Classification: methodologies

• features
• k-grams, patterns, time series shapelets, …
• sequence distance
• K-Nearest Neighbor, Support Vector Machines, Dynamic Time Warping, …
• models
• Naïve Bayes, Markov Model (K-order, Hidden, Variable Order)

Xing, Z., Pei, J., & Keogh, E. (2010). A brief survey on sequence classification.

Unsupervised Learning

• opposed to Supervised Learning
• advantages:
• labelling of data is not necessary
• “generic solution”: not bound to specific features
• computationally less intensive
• disadvantages:
• accuracy?
• results need more refining or post-processing

Model of Choice: Markov Chain (nth order)

• stochastic model
• limited memory:
• nly takes last n states into account
• finite state space

split camera view into “regions”

Region Clustering

• tried but rejected:
• K-Means (badly defined borders)
• DBSCAN (large amount of unclassifiable

points)

• solution: “agglomerative clustering”
• hierarchical clustering: bottom-up approach
• minimize average of distances

dendrogram

• connectivity constraints

speed gains!

• deterministic amount of clusters

Pedregosa et al. (2011). Scikit-learn: Machine learning in Python.

same region!

Region Clustering

• using hierarchical clusters and a kNN classifier
• {p1, p2, …, p99, p100} {R1, R1, …, R7, R7}

All detected paths projected on clusters

SLIDE 8

08/12/2016 3

Model Fitting

• 1st Order Markov Model
• Expectation – Maximization algorithm
• transition matrix: before training after training

Probabilities after training

likely occurences rare occurences

Probabilities after training

rare occurences likely occurences

Determining the threshold

• only interested in rarest occurrences
• using “Quantile function”
• = inf ∈ ℝ: ≤
• Inverse Cumulative Distribution function

Provisional results

• th. = -9

Manual adjustment

• Problem: too many false positives
• people walking the entire length of the platform, pacing up and down, …
• not enough data gathered, bad parameter tuning, …
• Solution: exclude certain “paths” from influencing the results
• exclude all paths containing [Ra, Rb, …]
• exclude all regions in a straight line between p1 and p2
• Dynamic Time Warping
• …

SLIDE 9

08/12/2016 4

Adjusted results Evaluation

• difficult to tune all important parameters:
• amount of clusters
• probability threshold
• model order
• general pipeline: no ground truth available
• train station case: ground truth available (after labelling)
• future work:
• vary between different parameter settings
• apply cross-validation and generate PR curve
• automatically pick the best settings for high accuracy

Evaluation: early test on annotated data Evaluation: early test on annotated data

Questions? Suggestions?

SLIDE 10

8/12/2016 1

Progress on pedestrian detection

Kristof Van Beeck

Outline

• Short recap of results previous meeting
• New FLIR – LWIR dataset
• Pedestrian detection on this dataset
• How?
• Initial experiments with KAIST model
• Newly trained model
• Various improvements
• Conclusions and future work

2

Recap of results previous meeting

• Presented internal developed pedestrian detection

framework

• Uses ACF as baseline (RGB only)
• Contains both training and detection code
• Applied on the KAIST dataset

3

Recap - FLIR dataset – Pedestrian detection - Conclusions

Recap of results previous meeting

• KAIST multispectral pedestrian dataset:
• Publically available
• 95.000 VGA (640 x 480, 20 Hz) image pairs (Color + LWIR)
• 103.128 annotations and 1.182 unique pedestrians
• LWIR is useful during night/misty conditions

4

Recap - FLIR dataset – Pedestrian detection - Conclusions

Recap of results previous meeting

• Added extra channels to integrate the LWIR information

5

ACF color channels

• 3 color channels (LUV)
• 6 gradient orientations
• 1 gradient magnitude

LWIR channels

• 1 intensity
• 6 gradient orientations
• 1 gradient magnitude

Gets better

Recap - FLIR dataset – Pedestrian detection - Conclusions

Recap of results previous meeting

• Tested many influences:
• ACF/ACF+, amount of training data, model size,

resolution of LWIR images

• Training set – test set

6

Recap - FLIR dataset – Pedestrian detection - Conclusions

SLIDE 11

8/12/2016 2

FLIR dataset

FLIR LWIR dataset

• Recently acquired dataset from FLIR
• LWIR only, train station Brugge
• Divided in two sets
• Each set again divided: crossings, humans, combined

8

Recap - FLIR dataset – Pedestrian detection - Conclusions

Brugge bend Brugge straight

FLIR LWIR dataset

• Specs:
• # of videos: combined: 27, crossing: 26, humans: 37
• Framerate of 7 FPS, resolution of 640x512
• Goal: detect people which cross the train tracks

9

Recap - FLIR dataset – Pedestrian detection - Conclusions

FLIR LWIR dataset

• Detecting this behavior is composed of two main parts:
• Perform accurate pedestrian detection and generate

tracks (CV team)

• Analyze detections/tracks for abnormal behavior (AI team)

10

Recap - FLIR dataset – Pedestrian detection - Conclusions

Pedestrian detection

Pedestrian detection

• Detection of pedestrians in only LWIR images is difficult:
• Advantage LWIR: able to visualize people in low light (night, mist)

conditions

• Disadvantage: less discriminative information (only grayscale

image, no color information)

• Challenge: see how far we get concerning detection

accuracy

• First naive approach: camera is fixed, perform background

subtraction, uses blobs as detection will this work?

12

Recap - FLIR dataset – Pedestrian detection - Conclusions

SLIDE 12

8/12/2016 3

Pedestrian detection

• Works sometimes (MOG technique):
• However many problems:
• Blobs merge, passing trains:

13

Recap - FLIR dataset – Pedestrian detection - Conclusions

Pedestrian detection

• People disappear in background if standing still for longer periods

(long memory needed), auto-contrast (shorter memory):

• Might be avoided with specific background parameters (threshold

foreground, length of memory)

• However, difficult to generalize and tune
• Therefore, we first develop a good pedestrian detector using

appearance features

14

Recap - FLIR dataset – Pedestrian detection - Conclusions

Pedestrian detection – how?

15

Input image Extract features Evaluate Model Perform NMS Detection Output

0.5 0.76

• What about multiple sizes?
• Sliding window in scale-space feature pyramid time consuming
• Detection scores = probability measure
• Low threshold = find many pedestrians, make mistakes
• High threshold = find fewer pedestrians with less or no mistakes

Recap - FLIR dataset – Pedestrian detection - Conclusions

Pedestrian detection – how?

16

• How is a detection model learned?

Use hundreds (or thousands) of images Images containing pedestrians (positives) Images containing background (negatives) Machine Learning (e.g. SVM, AdaBoost, Neural Networks,…)

Recap - FLIR dataset – Pedestrian detection - Conclusions

Pedestrian detection – how?

17

• Comparing CV algorithms
• Loop over multiple score thresholds
• Compare with ground truth

Manual annotations Correct detections (TP) Annotations not found (FN) Wrong detections (FP)

Recap - FLIR dataset – Pedestrian detection - Conclusions

Pedestrian detection – how?

18

• Plot these measures in graph

Find more pedestrians Make fewer mistakes Optimal Point

Recap - FLIR dataset – Pedestrian detection - Conclusions

SLIDE 13

8/12/2016 4

Pedestrian detection

• We first trained a new model based on KAIST LWIR data
• nly:
• Use LWIR image as grayscale input image
• Model size: 50 x 20.5 pixels (proven to be ideal)
• Aimed for 4096 weak classifiers, depth 5 decision trees

Reached 2701

• Every 20th frame
• Total of 2120 annotations (2500 images)
• 3496 positive patches, 50000 negatives (automatically sampled)

19

Recap - FLIR dataset – Pedestrian detection - Conclusions

Pedestrian detection

• Result:
• Seems to work, what about quantitative measures?

20

Recap - FLIR dataset – Pedestrian detection - Conclusions

Pedestrian detection

• For this we need annotation data
• Convince colleagues to collaborate in labelling session ☺
• Using VATIC tool
• Videos were renumbered (bruggeX), doubles were deleted

63 videos remained Ordered from more to less important (crossings) 27 videos were labeled:

21

Recap - FLIR dataset – Pedestrian detection - Conclusions

Crossings Combined (almost no crossings) 0,1,3,4,5,6,7,8,9,10,11,12,13,14 15,16,17,18,19,20,21,22,23,24,25,26,27

Pedestrian detection

• Label results:
• Total of 24831 frames, 30129 labels (4523 occluded)
• Total of 79 unique track IDs

22

Recap - FLIR dataset – Pedestrian detection - Conclusions

Pedestrian detection

• Labelled video (brugge9.avi):

23

Recap - FLIR dataset – Pedestrian detection - Conclusions

Pedestrian detection

• With this annotation data, validation data is calculated:

24

Recap - FLIR dataset – Pedestrian detection - Conclusions

No upscaling: pedestrians up to 50 px are detected (labels < 40px discarded) With upscaling: pedestrians up to 40 px are detected (labels < 35px discarded)

SLIDE 14

8/12/2016 5

Pedestrian detection

• Result of KAIST model on video:

25

Recap - FLIR dataset – Pedestrian detection - Conclusions

Pedestrian detection

• Not that good!
• However, we now have annotations
• Trained a new model with data (model specs same as KAIST)
• Interleaved annotations for training and test set
• Each 5th image
• Total of 2921 annotations (2429 images)
• 4212 positive patches, 50000 negatives

26

Recap - FLIR dataset – Pedestrian detection - Conclusions

Crossings Combined (almost no crossings) 0,1,3,4,5,6,7,8,9,10,11,12,13,14 15,16,17,18,19,20,21,22,23,24,25,26,27

Training pool Testing pool

Pedestrian detection

• Results:

27

Recap - FLIR dataset – Pedestrian detection - Conclusions

On test set

On training set for validation

At precision of 90%, recall of 75% (AP of 78.9%)

Pedestrian detection

28

Recap - FLIR dataset – Pedestrian detection - Conclusions

• Much better! (video in a few slides)
• Further improvements:
• Add scene constraints
• Add background subtraction
• Add basic tracking
• Scene constraints:
• Assume flat ground plane
• After calibration (based on annotations) each height at

specific position is known

• Reject detections which deviate from this constraint

Pedestrian detection

29

Recap - FLIR dataset – Pedestrian detection - Conclusions

• Applied on this scenario:
• Divided in two regions
• Extract annotations
• Fit first order function (plane)

Pedestrian detection

30

Recap - FLIR dataset – Pedestrian detection - Conclusions

• Height is now known at each position
• Eliminate detections which diverge (based on percentage)

Optimal point Original FLIR model Prune too much Prune too little

SLIDE 15

8/12/2016 6

Pedestrian detection

31

Recap - FLIR dataset – Pedestrian detection - Conclusions

• Zoomed in:

For fixed recall (75%) ~5% improvement in precision!

Pedestrian detection

• Resulting video (FLIR model + scene constraints)

32

Recap - FLIR dataset – Pedestrian detection - Conclusions

Pedestrian detection

• Incorporate background subtraction to increase detection

accuracy:

• For each detection, validate the percentage of foreground pixels in

background subtraction image

• Experimented with KNN background
• Use this percentage to reweight the detection scores
• Should increase the precision for fixed recall (eliminate FP)
• Note: is unable to increase the recall

33

Recap - FLIR dataset – Pedestrian detection - Conclusions

Pedestrian detection

• No significant increase in accuracy

34

Recap - FLIR dataset – Pedestrian detection - Conclusions

Slight increase in recall, precision drops: Good detections not always enough foreground pixels Too small? Change parameters of BGS?

Pedestrian detection

• Final improvement: tracking
• Cope with missing detections
• Track bounding box based on specific model, predict future position
• Match with new detection, keep prediction if none found
• Eliminate path if no detection found for multiple frames (TTL)
• Kalman filter, constant velocity motion model

35

Recap - FLIR dataset – Pedestrian detection - Conclusions

Pedestrian detection

• Accuracy results:

36

Recap - FLIR dataset – Pedestrian detection - Conclusions

Significant increase in recall! If TTL too high, precision drops (FP are longer maintained)

SLIDE 16

8/12/2016 7

Pedestrian detection

• Final video, compare initial KAIST model (red) with our final best

result (green):

37

Recap - FLIR dataset – Pedestrian detection - Conclusions

Conclusions and Future work

• We presented our experiments on the new FLIR dataset
• Started from KAIST model (AP = 52.3%)
• Trained own model
• Introduced several improvements (AP = ~84%)
• Achieved excellent accuracy
• Why not perfect?
• Small pedestrians
• Sometimes low contrast
• Sometimes correct detection,

but not labeled

38

Recap - FLIR dataset – Pedestrian detection - Conclusions

Conclusions and Future work

• Several improvements exist:
• Better evaluation of background subtraction

Evaluate more parameters

• Use blobs of background subtraction as pre-processing?
• Improve tracking (paths not always unique per person)

39

Recap - FLIR dataset – Pedestrian detection - Conclusions

Thank you for your attention!

• Questions?
• E-mails:

kristof.vanbeeck@kuleuven.be toon.goedeme@kuleuven.be

Discussion & Planning

Person detection: planning

42

• CNHi: collecting footage in dusty conditions
• WenZ/PortAntwerp: collecting footage
• DroneMatrix: experiments within scope of Mirador project

Abnormal behaviour detection: planning

• Validate current system
• Examples of abnormal behaviour — acting?
• Extension
• Shoplifting
• Footage of (non-IR) surveillance cameras: in progress