iQmulus/TerraMobilita benchmark on Urban Analysis Bruno Vallet, - - PowerPoint PPT Presentation

▶
iqmulus terramobilita benchmark on urban analysis
SMART_READER_LITE
LIVE PREVIEW

iQmulus/TerraMobilita benchmark on Urban Analysis Bruno Vallet, - - PowerPoint PPT Presentation

Jan 27, 2024 151 likes •657 views

iQmulus/TerraMobilita benchmark on Urban Analysis Bruno Vallet, Mathieu Brdif, Batriz Marcotegui, Andres Serna, Nicolas Paparoditis 1 / 50 08/07/2014 First iQmulus workshop : Urban Analysis Benchmark Introduction 2 / 50 08/07/2014

slide-1
SLIDE 1

1 / 50 08/07/2014 First iQmulus workshop : Urban Analysis Benchmark

iQmulus/TerraMobilita benchmark on Urban Analysis

Bruno Vallet, Mathieu Brédif, Béatriz Marcotegui, Andres Serna, Nicolas Paparoditis

slide-2
SLIDE 2

2 / 50 08/07/2014 First iQmulus workshop : Urban Analysis Benchmark

Introduction

slide-3
SLIDE 3

08/07/2014 3 / 50 First iQmulus workshop : Urban Analysis Benchmark

Introduction

  • Mobile laser scanning (MLS) generates massive amount of data
  • Urban cores are obects of utmost interest :
  • Urban planning
  • Inventory and maintenance
  • Accessibility diagnostic
  • Need for tools to analyse MLS data acquired in urban cores
  • Need for a benchmark of existing tools
slide-4
SLIDE 4

08/07/2014 4 / 50 First iQmulus workshop : Urban Analysis Benchmark

Benchmark objectives

  • Trigger interest on MLS in scientific communities :
  • Computer vision
  • Photogrammetry/remote sensing
  • Geometry processing
  • Provide reliable and large scale ground truth for works on MLS
  • Define an ambitious goal for MLS based urban analysis
  • Provide an objective tool to compare the qualities of urban analysis

algorithms

slide-5
SLIDE 5

08/07/2014 5 / 50 First iQmulus workshop : Urban Analysis Benchmark

Guidelines

  • Fully controlled annotation of the data. For each point :
  • object/segment id
  • class label
  • Very generic semantic tree to provide an ontology for urban scenes
  • Evaluation :
  • Multicriteria : not a ranking but an evaluation of the pros and cons
  • f each benchmarked algorithm
  • Objective : no parameters/thresholds
slide-6
SLIDE 6

08/07/2014 6 / 50 First iQmulus workshop : Urban Analysis Benchmark

Outline

  • Dataset
  • Analysis problem statement
  • Ground truth production
  • Evaluation metrics
  • Participants & results
  • Conclusion
slide-7
SLIDE 7

7 / 50 08/07/2014 First iQmulus workshop : Urban Analysis Benchmark

Dataset

slide-8
SLIDE 8

08/07/2014 8 / 50 First iQmulus workshop : Urban Analysis Benchmark

Data

  • Acquisition with Stereopolis MLS :
  • 360° Riegl sensor, multiecho
  • Applanix georeferencing
  • Anisotropic resolution :
  • Across trajectory : Constant angular resolution (0.03°) => distance

dependant geometric resolution

  • Along trajectory : Constant time resolution (10ms) => speed

dependant geometric resolution

slide-9
SLIDE 9

08/07/2014 9 / 50 First iQmulus workshop : Urban Analysis Benchmark

Data attributes

  • Attributes :
  • X,Y,Z : coordinates of the echo in a geographical frame
  • X0,Y0,Z0 : coordinates of the laser center at the time this echo was

acquired

  • Reflectance : backscattered intensity corrected for distance
  • num_echo : number of the echo in case of multiple returns
  • Time : time at which the point was acquired
  • Data provided in ply file format for easy and generic attribute

handling.

slide-10
SLIDE 10

08/07/2014 10 / 50 First iQmulus workshop : Urban Analysis Benchmark

Data

slide-11
SLIDE 11

08/07/2014 11 / 50 First iQmulus workshop : Urban Analysis Benchmark

Area

  • 10+1 zones in the center of Paris (6ème arrondissement)
  • Each zone has 30 (12) million points corresponding to 2 minutes of

acquisition each and around 500m (depending on vehicle speed)

slide-12
SLIDE 12

12 / 50 08/07/2014 First iQmulus workshop : Urban Analysis Benchmark

Analysis problem statement

slide-13
SLIDE 13

08/07/2014 13 / 50 First iQmulus workshop : Urban Analysis Benchmark

Scene analysis

  • We call scene analysis the combination of :
  • A segmentation of the scene in individual objects surfaces
  • A semantic labellisation (classification) of these objects
  • Participants are asked to provide a ply file, adding for each point :
  • A segment identifier id (defining the segmentation)
  • A class label class (defining the classification)
slide-14
SLIDE 14

08/07/2014 14 / 50 First iQmulus workshop : Urban Analysis Benchmark

Introduction : segmentation

slide-15
SLIDE 15

08/07/2014 15 / 50 First iQmulus workshop : Urban Analysis Benchmark

Introduction : classification

slide-16
SLIDE 16

08/07/2014 16 / 50 First iQmulus workshop : Urban Analysis Benchmark

Targeted Communities

  • Classification specialists :
  • Interested in classification ground truth
  • Not interested in object individualization
  • Growing interest in contextual classification
  • Segmentation specialists :
  • Growing interest in semantics to assist the segmentation
  • Detection specialists :
  • Detectors for specific object types
  • The semantic and geometric problems are connected
slide-17
SLIDE 17

08/07/2014 17 / 50 First iQmulus workshop : Urban Analysis Benchmark

Scene analysis : semantics

  • The semantic tree is very detailed :
  • Surface classes :
  • Road
  • Curb
  • Sidewalk
  • Facade/building
  • Objects classes :
  • Dynamic/static
  • Natural/man made
  • Punctual/linear/extended
  • Participants can go as deep as they wish in the semantics tree
  • Evaluation will be performed accordingly
slide-18
SLIDE 18

08/07/2014 18 / 50 First iQmulus workshop : Urban Analysis Benchmark

Scene analysis : semantics

slide-19
SLIDE 19

08/07/2014 19 / 50 First iQmulus workshop : Urban Analysis Benchmark

Scene analysis : semantics

slide-20
SLIDE 20

20 / 50 08/07/2014 First iQmulus workshop : Urban Analysis Benchmark

Ground truth production

slide-21
SLIDE 21

08/07/2014 21 / 50 First iQmulus workshop : Urban Analysis Benchmark

Ground truth production tool

  • Requirements
  • Fast and easy navigation and annotation
  • Segmentation at point level
  • Interactivity/editability
  • We designed an inteface in sensor geometry :
  • Columns are points acquired consecutively
  • Consecutive columns correspond to points acquired at a time

interval equal to the time for the laser beam to finish a 360° sweep

slide-22
SLIDE 22

08/07/2014 22 / 50 First iQmulus workshop : Urban Analysis Benchmark

Data: sensor space

slide-23
SLIDE 23

08/07/2014 23 / 50 First iQmulus workshop : Urban Analysis Benchmark

Segmentation

  • In 2D, segmentation is created and maintained by a partition graph
  • User is provided with graph editing tools :
  • Create a node (at a pixel corner) possibly on an existing edge
  • Create an edge (along pixel boundaries) :
  • A straight line (Brezenham)
  • A minimal path for the cost :
  • Parameters = weights for Normal/Depth/Intensity difference term
  • User can interactively tune these parameters
  • Move an existing node (recomputes all adjacent edges)
slide-24
SLIDE 24

08/07/2014 24 / 50 First iQmulus workshop : Urban Analysis Benchmark

Other features

  • A segment can be split by any plane defined by :
  • Three points
  • Two points (vertical)
  • One point (vertical and orthogonal to beam direction
  • Plus an offset
  • Segments can be merged (necessary in case of occlusions)
  • Segments can be tagged by a label from the semantic tree
  • Zooming, Panning
  • Snapping
  • Import/Export point clouds with label/ids per points
  • Web based (javascript+webGL)
slide-25
SLIDE 25

08/07/2014 25 / 50 First iQmulus workshop : Urban Analysis Benchmark

Example

slide-26
SLIDE 26

08/07/2014 26 / 50 First iQmulus workshop : Urban Analysis Benchmark

Production details

  • Production of the learning dataset (12Mpts) with an alpha version of

the tool

  • For the 10 zones of the benchmark :
  • 10 participants
  • 2 days production each
  • Around 60% of the 300 Mpts annotated
  • Easy production management thanks to the web based tool :
  • Each participants gets a unique link alowing them to process a 30 Mpts

block

  • Their work is simply stored as a graph
  • Graphs are controlled and final ground truth ply files exported
slide-27
SLIDE 27

27 / 50 08/07/2014 First iQmulus workshop : Urban Analysis Benchmark

Evaluation metrics

slide-28
SLIDE 28

08/07/2014 28 / 50 First iQmulus workshop : Urban Analysis Benchmark

Multicriteria evaluation

  • Evaluate the algorithm result:
  • As a classification algorithm: confusion matrix
  • As a detection algorithm :
  • precision/recall for object classes
  • No notion of object for surface classes
slide-29
SLIDE 29

08/07/2014 29 / 50 First iQmulus workshop : Urban Analysis Benchmark

Precision/Recall

  • Need to answer the questions
  • Is a Ground truth (GT) object detected ?
  • Is an Algorithm result (AR) a good detection ?
  • Answer (and evaluation) requires to match objects from the GT to
  • bjects from the AR
  • This matching allows to define :
  • Precision = #(GT match AR)/#GT
  • Recall = #(AR match GT)/#AR
  • Thus precision/recall is defined on a subjective matching criterion
slide-30
SLIDE 30

08/07/2014 30 / 50 First iQmulus workshop : Urban Analysis Benchmark

Delocalisation

Ground truth Algorithm result

slide-31
SLIDE 31

08/07/2014 31 / 50 First iQmulus workshop : Urban Analysis Benchmark

Dilatation/Erosion

Dilatation Erosion Ground truth Algorithm result

slide-32
SLIDE 32

08/07/2014 32 / 50 First iQmulus workshop : Urban Analysis Benchmark

Scission/fusion

Split (N to 1) Merge (1 to M) Ground truth Algorithm result

slide-33
SLIDE 33

08/07/2014 33 / 50 First iQmulus workshop : Urban Analysis Benchmark

N to M associations

Ground truth Algorithm result

slide-34
SLIDE 34

08/07/2014 34 / 50 First iQmulus workshop : Urban Analysis Benchmark

Intersection/Union Ratio

  • Gives a « distance » between objets :
  • 0 = no intersection
  • 1 = perfect match
  • Matching often defined by a threshold on R
  • Above 0.5, no N to M matchings
  • But 0.5 is very strict
  • Precision/recall depends highly on this threshold
slide-35
SLIDE 35

08/07/2014 35 / 50 First iQmulus workshop : Urban Analysis Benchmark

Proposition

  • Give precision and recall as a function of this threshold :
  • No arbitrary (subjective) choice of a threshold
  • Compare algorithms by comparing curves
  • For thresholds below 0.5, also give the number of N to 1 and 1 to M

pairings

slide-36
SLIDE 36

36 / 50 08/07/2014 First iQmulus workshop : Urban Analysis Benchmark

Participants & results

slide-37
SLIDE 37

08/07/2014 37 / 50 First iQmulus workshop : Urban Analysis Benchmark

Participants

  • CMM - MINES ParisTech (Andres Serna, Beatriz Marcotegui):
  • Based on elevation images
  • Mathematical Morphology based image processing
  • Machine learning techniques
  • Does the full analysis (segmentation and classification)
  • Institute of Photogrammetry and Remote Sensing (IPF) – KIT (Martin

Weinmann) :

  • Extract a variety of low-level geometric features
  • Supervised classification based on careful feature selection
  • Only classification evaluated
slide-38
SLIDE 38

08/07/2014 38 / 50 First iQmulus workshop : Urban Analysis Benchmark

Ground truth

slide-39
SLIDE 39

08/07/2014 39 / 50 First iQmulus workshop : Urban Analysis Benchmark

CMM result

slide-40
SLIDE 40

08/07/2014 40 / 50 First iQmulus workshop : Urban Analysis Benchmark

IPF result

slide-41
SLIDE 41

08/07/2014 41 / 50 First iQmulus workshop : Urban Analysis Benchmark

Results for CMM

  • Classification (one 30 Mpts zone) :
  • Surface/object : 92.6%
  • Building/ground surface : 98.3%
  • Curb/sidewalk/road : 98.4%

GT/AR Road&side Curb road 71.8348 0.684865 sidewalk 25.7088 0.687476 curb 0.187855 0.896216

slide-42
SLIDE 42

08/07/2014 42 / 50 First iQmulus workshop : Urban Analysis Benchmark

Results for CMM

  • Classification (one 30 Mpts zone) :
  • Surface/object : 92.6%
  • Building/ground surface : 98.3%
  • Curb/sidewalk/road : 98.4%

GT/AR Road&side Curb road 71.8348 0.684865 sidewalk 25.7088 0.687476 curb 0.187855 0.896216

slide-43
SLIDE 43

08/07/2014 43 / 50 First iQmulus workshop : Urban Analysis Benchmark

Results for CMM

  • Classification (one 30 Mpts zone) :
  • Surface/object : 92.6%
  • Building/ground surface : 98.3%
  • Curb/sidewalk/road : 98.4% but curb (2.3%) confused for sidewalk

(0.7%) and road (0.7%) because of rasterization.

  • Static/mobile object : 91.8%
  • Pedestrian/2/4 wheelers : 99.3%

GT/AR pedestrian 2 wheelers 4+ wheelers pedestrian 1.63193 0.00262888 0.123962 2 wheelers 0.388468 0.653378 4+ wheelers 0.112435 0.0281088 97.0591

slide-44
SLIDE 44

08/07/2014 44 / 50 First iQmulus workshop : Urban Analysis Benchmark

Results for CMM

  • Detection (one 30 Mpts zone) : All objects

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 20 40 60 80 100 120 Precision recall 0.1 0.2 0.3 0.4 0.5 0.6 2 4 6 8 10 12 14 16 1-n m-1

slide-45
SLIDE 45

08/07/2014 45 / 50 First iQmulus workshop : Urban Analysis Benchmark

Results for CMM

  • Detection (one 30 Mpts zone) :
  • Static objects :

Dynamic objects

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 20 40 60 80 100 120 Precision Recall 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 20 40 60 80 100 120 Precision recall

slide-46
SLIDE 46

08/07/2014 46 / 50 First iQmulus workshop : Urban Analysis Benchmark

Results for IPF

  • Classification (learning dataset only)
  • Surface/object : 87.8%
  • Ground/Building surface : 93.7%
  • Static/mobile object : 91.5%
  • Pedestrian/2/4 wheelers : 68.5%

GT/AR pedestrian 2 wheelers 4+ wheelers pedestrian 4.06508 0.401652 0.0832806 2 wheelers 0.397657 8.72356 1.02395 4+ wheelers 10.4307 19.173 55.7012

slide-47
SLIDE 47

47 / 50 08/07/2014 First iQmulus workshop : Urban Analysis Benchmark

Conclusion

slide-48
SLIDE 48

08/07/2014 48 / 50 First iQmulus workshop : Urban Analysis Benchmark

Conclusion

  • Very challenging benchmark :
  • Large dataset, requiring a large amount of work for ground truth

production

  • Very detailed semantic tree
  • Difficult data:
  • Vehicle stops (point accumulations)
  • Transversal roads (very different scanning geometry)
  • Objectivity :
  • Manual production of the ground truth
  • Parameter free evaluation
slide-49
SLIDE 49

08/07/2014 49 / 50 First iQmulus workshop : Urban Analysis Benchmark

Perspectives

  • Perspectives :
  • Realeasing a larger part of the ground truth for learning
  • More targeted benchmarks (car type determination, static/mobile
  • bject determination, ...)
  • Benchmark will stay open for future participants
  • Having the participants provide an executable instead of a result :
  • Comparison of timings
  • More validity to the benchmark results (no fine parameter tuning)
  • Vector evaluation for surface limits
  • Correcting the anisotropy in pointwise evaluation
slide-50
SLIDE 50

50 / 50 08/07/2014 First iQmulus workshop : Urban Analysis Benchmark

Thank you for your attention Visit us at data.ign.fr/benchmarks/UrbanAnalysis