SLIDE 1 Real-Time Pedestrian Tracking, Prediction & Navigation
Dinesh Manocha
dm@cs.unc.edu http://gamma.cs.unc.edu
SLIDE 2 Collabo Collaborators
2
- Aniket Bera
- Jur van den Berg (UNC/Utah/Google/Otto/Uber)
- Andrew Best
- Sean Curtis (UNC/Boeing/TRI)
- Ernest Cheung
- Stephen Guy (UNC/Minnesota)
- Davik Kasik (Boeing)
- Sujeong Kim (UNC/SRI)
- Ming C Lin
- Rahul Narain (UNC/Berkeley/Minnesota)
- Sahil Narang
- Chonhyon Park (UNC/Zoox)
- Sachin Patil (UNC/Berkeley/Ottp/Uber)
- Ari Shapiro (ICT/USC)
- Jamie Snape (UNC/Kitware)
- Basim Zafar (Hajj Research Institute)
SLIDE 3
Pe Pedestrian Ve Vehicle In Interactio ions
SLIDE 4 Pe Pedestrian/Cro rowd Sim Simula lation tion & Pre Prediction
4
SLIDE 5 AC ACM New News: Avoi
ding th the Crush Crush (May 2016)
5
ACM NEWS
Avoiding the Crush
By Keith Kirkpatrick
May 3, 2016 Researchers have been developing models that mimic how people move in large groups. Earthquakes, floods, and hurricanes take lives around the world, due to their unpredictable nature and massive power. Humans are responsible for similar levels of carnage through human crushes or stampedes; for example, more than 2,200 Muslim pilgrims were killed at the Ratangarh Mata Temple in India killed 115 people and injured more than 100. To address such public safety issues, researchers have been developing computer models that mimic how people actually move when in large groups. Their ultimate goal: to develop reliable algorithms that can feed into models to accurately predict how crowds move, then design physical spaces to safely accommodate and control that movement to prevent deadly crushes.
SLIDE 6 Pe Pedestrian Mo Motion
Simula lation tion
- Non‐Velocity Based
- Rule‐based
- Boids
- Force‐based
- Cellular Automata (CA)
- Velocity Based
- Plan control based on velocity‐space considerations
- …
6
SLIDE 7 RVO RVO: “R “Reactiv eactive” e” Obs Obstacl acles
- Reciprocity Assumption
- RVOA
B(vB, vA) = {v’ A | 2v’ A – vA VOA B(vB)} [Berg et al. 2008]
7
X Y Vx Vy Workspace Velocity Space
SLIDE 8 Bene Benefit fit of
reciprocity: Pe Pedestrian mot motion
model
8
Velocity Obstacle (VO) Reciprocal Velocity Obstacle (RVO)
SLIDE 9 Me Menge: nge: Open Open Sour Source ce pedes pedestrian rian sim simula lation tion fr framework
9
http://gamma.cs.unc.edu/Menge
SLIDE 10 Where?
- Anomaly Detection
- Behavior Analysis
- Crowd Counting
- Driverless Cars
- Robotics
Pedestrian Tracking and Prediction
SLIDE 11 Crowd Video Crowd Video Input
Pedestrian Tracking and Prediction: Pipeline
SLIDE 12 Crowd Video Crowd Video EnKF State Estimation EnKF State Estimation Trajectory Extraction Trajectory Extraction Motion-model Driven Sensor Capture Input
Pedestrian Tracking and Prediction: Pipeline
SLIDE 13 Crowd Video Crowd Video EnKF State Estimation EnKF State Estimation Trajectory Extraction Trajectory Extraction Compute Pedestrian Clusters Compute Pedestrian Clusters Motion-model Driven Sensor Capture Input
Pedestrian Tracking and Prediction: Pipeline
SLIDE 14 Crowd Video Crowd Video EnKF State Estimation EnKF State Estimation Trajectory Extraction Trajectory Extraction Compute Pedestrian Clusters Compute Pedestrian Clusters Movement Flow Learning Movement Flow Learning Global Movement Patterns Global Movement Patterns Motion-model Driven Sensor Capture Input Crowd Flow Learning
Pedestrian Tracking and Prediction: Pipeline
SLIDE 15 Crowd Video Crowd Video EnKF State Estimation EnKF State Estimation Trajectory Extraction Trajectory Extraction Compute Pedestrian Clusters Compute Pedestrian Clusters Movement Flow Learning Movement Flow Learning Global Movement Patterns Global Movement Patterns Predicted State Predicted State Motion-model Driven Sensor Capture Input Crowd Flow Learning
Learning Microscopic and Macroscopic Motion Models Learning Microscopic and Macroscopic Motion Models
Local Movement Patterns Local Movement Patterns
Pedestrian Tracking and Prediction: Pipeline
SLIDE 16 Crowd Video Crowd Video EnKF State Estimation EnKF State Estimation Trajectory Extraction Trajectory Extraction Compute Pedestrian Clusters Compute Pedestrian Clusters Movement Flow Learning Movement Flow Learning Global Movement Patterns Global Movement Patterns Predicted State Predicted State Motion-model Driven Sensor Capture Input Crowd Flow Learning
Learning Microscopic and Macroscopic Motion Models Learning Microscopic and Macroscopic Motion Models
Local Movement Patterns Local Movement Patterns Prediction Feedback
Pedestrian Tracking and Prediction: Pipeline
SLIDE 17 State Estimation
17
Sensor Sensor
Noisy
z
x
Estimated state
Pedestrian Simulation Pedestrian Simulation Maximum Likelihood Estimation Maximum Likelihood Estimation
Q ) f(x
Error distribution Predicted states
EnKF EnKF
SLIDE 18 Local Features
18
For each time step, for each pedestrian
Pedestrian dynamics feature:
Position, Average velocity Goal velocity Entry points learning (Gaussian Mixture Model) Entry Points:
(temporal variance)
SLIDE 19 Global Features
19
Movement Flow Learning (K-means clustering)
SLIDE 20
SLIDE 21 What more do we need?
- Behavior Learning
- Culture sensitive
- Scene specific
SLIDE 22
Pedestrian/Crowd Classification
SLIDE 23 Synthetic Labeled Datasets for learning
http:///
gamma.cs.unc.edu/RCrowdT/Dataset http://arxiv.org/abs/1606.08998
SLIDE 24
LCrowdV: Synthesized Labeled datasets for pedestrian video analysis
SLIDE 25
Synthetic Labeled Data for Learning: pedestrian detection + behavior classification
SLIDE 26 Pedestrian Behavior Learning:Pipeline
Labeled Crowd Dataset Labeled Crowd Dataset Live Video Stream Live Video Stream State Estimation State Estimation Linear Regression Fitting Based on Eysenck Model Linear Regression Fitting Based on Eysenck Model Predict Future state based
State Space Predict Future state based
State Space Prediction/Navigation Prediction/Navigation
SLIDE 27 Pedestrian State Pedestrian State Pedestrian Path Prediction Pedestrian Path Prediction
Shy
Impulsive
Labeled Dataset
SLIDE 28
Video: International Trade Fair, New Delhi 2016
Real‐time Pedestrian Behavior Classification
SLIDE 29
Improved Tracking and Prediction using Behavior Classification
SLIDE 30
SLIDE 31
Panic Simulation
SLIDE 32
Robot Navigation
SLIDE 33 Robot Navigation
Kinematic model of the robot Robot’s position R(t, u) at time t given the control u can be derived as follows
SLIDE 34
Robot Navigation: Improvements
GVO: Generalized velocity obstacles for non-holonomic constraints
SLIDE 35
Ongoing Work: Vehicle Navigation Simulator
SLIDE 36 Pedestrian Modeling: Conclusions
- Improved motion models for dense scenarios
- Pedestrian dynamics learning using Bayesian Inference
- Simulation + perception + learning
- Use of synthetic labeled datasets for deep learning
- Results:
- Real-time pedestrian tracking and prediction in dense scenes
- Pedestrian behavior prediction
- Improved accuracy
SLIDE 37 Ongoing and Future Work
- Predict pedestrian’s mood and body language
- Integrate with autonomous vehicles
- Evaluate different motion planning algorithms
- Develop robust pedestrian navigation algorithms
- Model other behaviors: bicycles
SLIDE 38 Acknowledgments
38
- Collaborators:
- Boeing, Hajj Research Institute, Julich SuperComputing
Center, Intel, Relic, Willow Garage
- Funding:
- Army Research Office, National Science Foundation,
Intel, HajjCORE, Boeing, KAUST, Willow Garage
