IEEE ETFA 2018 Turin, September 6-th A single camera inspection - - PowerPoint PPT Presentation

IEEE ETFA 2018

Turin, September 6-th

A single camera inspection system to detect and localize obstacles on railways based on manifold Kalman filtering

Federica Fioretti1, Emanuele Ruffaldi2, Carlo Alberto Avizzano1

federica.fioretti.90@gmail.com (1) TeCIP Institute, Scuola Superiore Sant'Anna Pisa, Italy, (2) MMI s.p.a, Italy

Due to the raising need to enforce the surveillance of the railroad in order to guarantee the safety for its users, many systems have been designed involving different technologies:

• Infrastructure based systems, communicating with on board train sensors and actively cooperating with

sensory equipment mounted nearby the railway line

• Locomotive systems, on board sensors with no interactions with any wayside equipment.
• Unmanned rail vehicles (URV) represent an innovative solution, bringing several advantages for being a self

powered base for a larger number of sensors, with no traffic disruption. The Computer Vision system developed in this work is specifically aimed to an URV, to support its intrusion and obstacle detection tasks.

Railway surveillance

Goal

Assuming that the URV is equipped with a single camera placed at the head of the train, we are interested in reconstructing the trajectory performed by the vehicle and the localization of known objects nearby the railroad.

Issues and goals

Application issues

• Scarce amount of available images regarding italian railway signs
• Few structural elements in the environment
• Monocular camera usage
• Low cost sensor
• Incomplete knowledge of camera parameters
• Unavoidable reconstruction uncertainty due to projection

Vanishing point

Projection of a world point at infinity to the image plane.

Homography and Vanishing Points

Homography

Considering a couple of views, is the mapping of an image point from a frame to the other is related to their intrinsic parameters (", "′), their relative pose, defined by (&, '), nonethless to the 3D local coordinates of the point [*, +, ,].. /0

1 = "1&"34/0 + "1'/,

Operational concepts

3D points Camera model Camera matrix

!"[$", &"]

IMAGE 1 Matched points, lines Camera matrix

!([$(, &(]

IMAGE 2 Matched points, lines Homography Relative camera poses Further knowledge

HAAR Cascade Classifier process:

Objects detection

Positive images Negative images Annotation Create samples Train cascade Parameters Object Classifier

Classifiers Dataset size Minimum Hit Rate Maximum False Alarm rate Kilometer sign 60, 200 0.995 0.05 Poles 248, 200 0.95 0.2

Scheme:

Line detection

1. Extract the lower portion of B/W image, applying Hough Transform:

• 2. Apply Template matching to the first rail element:

Algorithm: Railway Extraction frame ← first frame iter ← 30 Loop: extract each frame of the video while frame is not empty do: Evaluation of the lowest stripe of the image: leftX[0] ← Hough line search rightX[0] ← Hough line search Iterative evaluation of the upper stripes of image: for i ← 1 to iter do: Template matching technique: leftX[i] ← Find position of the template close to leftX[i-1] rightX[i] ← Find position of the template close to rightX[i-1]

Algorithm: Railway Extraction frame ← first frame iter ← 30 Loop: extract each frame of the video while frame is not empty do: Evaluation of the lowest stripe of the image: leftX[0] ← Hough line search rightX[0] ← Hough line search Iterative evaluation of the upper stripes of image: for i ← 1 to iter do: Template matching technique: leftX[i] ← Find position of the template close to leftX[i-1] rightX[i] ← Find position of the template close to rightX[i-1]

Line detection

!"#$$%$&'()*/,-./* = ! + 2 Δ45678&9:# ! ;"#$$%$&'()*/,-./* = ;&9:#(!) End points of the segment fitting the railroad sleepers: For each point feature between the rails having coordinates (;, !):

FoE coordinate estimation

+ Principal point + Intersection point of the lines approximating the rails + Mean of the point of intersection among all matched points in a bounding box, considering the same fixed y coordinate In this fashion the displacement of the FoE along the horizon line w.r.t. the principal point, Δ"#$% can be registered and exploited to:

• Filter the matched set of points
• Obtain motion cues. In more detail, the vehicle’s yaw rate can be

estimated as: &' = Δ) Δ* = Δ"#+%,- .

/01234

Where:

• Δ) is the difference in yaw angle between consecutive

frames

• Δ* is the time interval between consecutive frames
• Δ"#+% is the FoE displacement along the horizon line w.r.t.

the principal point

• .

/ is the focal distance along the image x axis

• ,- is the tangental velocity
• 01234 is the near clipping plane distance of the camera

FoE Tracking

Camera pose recovery

UKF Algebraic Algorithm FoE tracking Image Flow

Lines Detected turn markings Rail Lines velocity initialization Yaw rate

Camera motion estimate Sensor Fusion

pose, velocity, acceleration

Object detection Point Feature matching Lines detection Algebraic Algorithm UKF FoE tracking Sensor Fusion

pose Threshold check to filter wrong matches Fused camera pose and motion

Line Constraint

Considering a 3D line ! seen by a camera, having extrinsic parameters [$, &] the back-projection of the imaged line is a plane ( with normal ). )× $+ − & = 0

Line-based Algorithm

4. [)1]×$2

3)2)4 3&̃4 − )1 ×$4 3)4)2 3&̃2 = 0

Two points lying on the edges of a sleeper line, can be locally reconstructed imposing their known euclidean distance: 1.435 m. Employing the parameters [$6, &̃6], [$

7, &̃7], associated to two

different views of the same railroad tie, the unknown projective scale 8 is recovered using the following: $6 9:; − 8&̃6 − $

7 9:< − 8&̃7 = 0

The linear velocity = can be computed as: = > = & > − &(> − 1) 8BCDEF)G &FCH

Pose – Algebraic Algorithm

1. To each j-th view a rotation matrix I

7 is

associated, according to the lo loca cal sp spheric ical co coordinates of the vanishing points 2. )j

3× $ 7! − &7 = 0

3. X! Y = 0, X = )1

3

)2

3$2

)1

3&2

)4

3$4

)4

3&4

, ! Y = [!3, 1]3

From ‘’Extrinsic calibration of heterogeneous cameras by line images’’ by Ly et al (2014)

Numerical test Accuracy achieved in simulation

Whenever at least two groups of parallel lines are devised across three frames, it is possible to estimate the camera pose. The algorithm is able to estimate the camera orientation with high accuracy. The non-zero error affecting the displacement estimation provokes small drift over time (~ 1% of the distance traveled). Subsampling four views with overall displacement ~10 m:

Pose – Algebraic Algorithm

State dynamics

ℰ " + 1 = ℰ " ⊞ [( " Δ*, , " Δ*]. , " + 1 = , " + /(")∆* ( " + 1 = ( " + 3(")Δ* / " = 45 (") 3 " = 47(") Where: ⊞ ∶ 9:(3)×ℝ> → 9:(3) ⊟ ∶ 9:(3)×9:(3) → ℝ> Applying the Lie Algebra: ℰ " + 1 = exp ((")"× ,(")"× 1 ℰ "

Pose – UKF on manifold

State variables:

• Camera orientation and translation

ℰ " = E " , * " ∈ 9: 3

• Linear velocity , ∈ ℝG
• Angular velocity ( ∈ ℝG
• Linear acceleration / ∈ ℝG
• Angular acceleration 3 ∈ ℝG

Output variable:

• non-homogeneous coordinates of each matched image

point H ∈ ℝ7

Sampling time: Δ* = 5 GI

⁄ s

Noise variables:

• Process noise [45, 47] ∈ ℝ> acting only on 3 and / with

associated covariance matrix K

• Measurement noise L ∈ ℝ7 with associated covariance

matrix M

Work flow

1. Initialization: ! "#$% & − 1 , * +#$% & − 1 from algebraic algorithm 2. UKF processes each feature tracked between consecutive frames: a) Prediction: , +- & = /(, + & − 1 , ! " & − 1 , * + & − 1 , 1) Handling sigma points 3 4 around , + & − 1 :

• 3 4 k = 3 4 & − 1 ⊞ !

" & − 1 Δ9, * + & − 1 Δ9

• Recompose sigma-points in ,

+- k

• :- & = ∑

<

4 (=)( >? 4@A

3 4 − , +- k) (3 4 − , +- k) B b) Correction: for each i-th feature

• Triangulation: (K,,

+ & − 1 , , +- k), C4,D-E, C4,D → G4

• Ĉ4,D = ℎ(J, ,

+ & − 1 , ! " & − 1 , * + & − 1 , G4)

• Innovation: ℐ4 = C4,D − C4,D-E
• ,

+4 & = , +4

• & ⊞ LMℐ4,

:4 & = :4

• & − LMNLM

3. RANSAC(, + & , : & , ℐ(&)) 4. Final , +(&) from fusion with ! "O,PQR & and * +#$% &

Pose – UKF on manifold

Actual Image points Two view matching Triangulation 3D points Projection 2D – 3D Intrinsics State prediction Correction Weighted Fusion Epipole (FoE) Tracking Algebraic algorithm Kalman Gain Estimated Image Point

• RANSAC

up to 50 iterations C & − 1 , C(&) G4 Ĉ(&) C(&) !S(&) *(&) Innovation

Repeat for each feature

Analysis and recostruction

Analysis and recostruction

Conclusions

• Employing visual input alone gathered from a single camera, the overall method succeded to

reconstruct the train trajectory, to detect and localize objects within the infrastructure with sufficient accuracy

• Approaches based on the projective geometry mathematics have been employed
• Further refinement and continuous estimation utilizing the UKF
• The augmentation of the state confers flexibility to Sensor Fusion: all the measurements available can

be integrated in the form they have been provided.

Thank you for your attention