SLIDE 1
REAL TIME DUAL CAMERA SPECTRAL IMAGING BASED ON NVIDIA TEGRA SOC TO ASSESS UAV MISSIONS
Michele Moscaritolo, Alessandro Della Villa, Giacomo Benelli San Jose (CA) - 07 April 2016
REAL TIME DUAL CAMERA SPECTRAL IMAGING BASED ON NVIDIA TEGRA SOC TO - - PowerPoint PPT Presentation
REAL TIME DUAL CAMERA SPECTRAL IMAGING BASED ON NVIDIA TEGRA SOC TO - - PowerPoint PPT Presentation
REAL TIME DUAL CAMERA SPECTRAL IMAGING BASED ON NVIDIA TEGRA SOC TO ASSESS UAV MISSIONS Michele Moscaritolo, Alessandro Della Villa, Giacomo Benelli San Jose (CA) - 07 April 2016 PRESENTATION OVERVIEW Introduction Requirements Proposed
SLIDE 2
SLIDE 3
MILITARY COMMERCIAL HOBBY
UAV (Unmanned Aerial Vehicle) known as a drone (Dynamic Remotely Operated Navigation Equipment). An aircraft without a human pilot aboard.
SLIDE 4
DRONE APPLICATIONS
Inspection Agriculture Creative Survey & mapping Safety & Security
Energy Infrastructure Telecom Mining Infrastructure Construction Fire fjghting Surveillance SAR Farming Videography Photography Research Vegetation management
SLIDE 5
DRONE APPLICATIONS POSE A CHALLENGES
Inspection
Data acquisition & Analysis Computational power Real time
- nboard processing
Multi spectral sensor
Agriculture
ASSESSMENT OF PAYLOAD FOR:
Survey & mapping Safety & Security
SLIDE 6
PRESENTATION OVERVIEW
Introduction Requirements Proposed solution Development assessment Discussion
SLIDE 7
BASIC REQUIREMENTS
Data acquisition & Analysis Computational power Real time on board processing Multi spectral sensor Light and small sensor Reduce cost Scalable and modular User/Developer friendly 1 2 3 4 5 6 7 8
SLIDE 8
MULTISPECTRAL SOLUTIONS
Multispectral camera ideal “tool” to capture data in different scenario but: Expensive Cost ! Processing power hungry ! To satisfy this requirement we must fjnd an alternative solution
SLIDE 9
PRESENTATION OVERVIEW
Introduction Requirements Proposed solution Development assessment Discussion
SLIDE 10
ADVANCED UAV BASE ON COMPUTER VISION
IR+Visible image sensors coupled with high processing power of Jetson Tegra can be the ideal solution since Tegra can real time associate to each pixel Visible and IR Spetctrum
Nvidia Tegra Family K1, X1, ... DAYLIGHT CAMERA THERMAL CAMERA PROCESS UNIT
SLIDE 11
GOAL: ADVANCED UAVs
Flir Digital Sony optical zoom Nvidia Tegra Family K1, X1, ... SOFTWARE - SDK DAYLIGHT CAMERA PROCESS UNIT THERMAL CAMERA
Library of functions real-time computer vision and image acquisition and processing
SLIDE 12
PERFORMANCE COMPARISON
2 4 6 8 10 12 14 16 18 3 1 200 400 600 800 1000
Performance compare for Embedded Computer Vision and Image Processing Costs WATTS X1 NVIDIA 2016
- OCCHI II
GFLOPS
K1 NVIDIA oOCCHI I SMARTPHONES MULTISPECTRAL SOLUTION FPGA
SLIDE 13
PRESENTATION OVERVIEW
Introduction Requirements Proposed solution Development assessment Discussion
SLIDE 14
ARCHITECTURE
Custom Algorithms for UAV applications Metadata, Communication Protocol, Data Storage, Video Output Software + SDK
tegra
V4L
Tegra SoC GPU Sensor interface Camera Sensors Visible and Infrared (LWIR)
HDMI USB
SLIDE 15
SOFTWARE ARCHITECTURE
1 2 3 4 Receive commands from remote and provide output (HDMI or SD) Effjciently capture from generic USB cameras Deliver a constant frame rate for UAV fmy operations Take benefjt of GPU to combine sensor data In order to meet requirements we need a software capable to:
SLIDE 16
SOFTWARE ARCHITECTURE
GPU
Pre-process CORE 2 CORE 1 CORE 3 Visible capture MASTER THREAD IR capture Process
Output frame Frame SX Frame DX 8.33 fps 25 fps MAVLINK
Pre-process
SLIDE 17
SOFTWARE ARCHITECTURE FEATURES
Capture and Master have been isolated in separate threads attached to dedicated cores Capture and Master have fjxed pulse interval in order to deliver fjxed frame rate (8.33 fps or 25 fps) To respect time processing constrain GPU is used GPU processing pipelines can be confjgured setting up wanted transformations to exetend and customize functionalities 1 2 3 4
SLIDE 18
SOFTWARE ARCHITECTURE FEATURES
video replacement
SLIDE 19
TYPICAL SCENARIO
RAW DATA IMAGE ACQUISITION
Typical scenario Based on previous requirement roof inspection
SLIDE 20
IMAGE FILTERING & PRE-PROCESSING SEGMENTATION
SLIDE 21
MEASUREMENT & ANALYSIS
SLIDE 22
TYPICAL SCENARIO
RAW DATA IMAGE ACQUISITION
Typical scenario Based on previous requirement Body signature detection
SLIDE 23
IMAGE PROCESSING IMAGE REGISTRATION
SLIDE 24
MEASUREMENT & ANALYSIS
SLIDE 25
PRESENTATION OVERVIEW
Introduction Requirements Proposed solution Development assessment Discussion
SLIDE 26
OVERLAY SOLUTION
To adopt in different applications and drone missions To get relevant spectral information To have real time on board processing with Nvidia Tegra Scalar and modular platform has been developed for UAV Multi Spectral applications It offers less spectral information with respect to hyperspectral but it is 3-4 times cheaper.
Dual head camera with overlay solution is possible: 1 2 3 4
SLIDE 27
Michele Moscaritolo, Alessandro Della Villa, Giacomo Benelli San Jose (CA) - 07 April 2016