Detailed Design Review MSD 18047 - VIRTUAL CANE Agenda Project and - PowerPoint PPT Presentation

Detailed Design Review MSD 18047 - VIRTUAL CANE

Agenda Project and Concept Breakdown Picture Matching SIFT Obstacle Avoidance OpenAL IMU Motion Estimation Housing

Team Team Member Role Major Demo/Focus Obs Av., Pic Mat, Suhail Prasathong Team Lead Computer Engineering Obs Av ML Housing Deepti Chintalapudi Project Manager Industrial Engineering Research E J Team Member Electrical Engineering IMU/Gyro Josh Drezner Purchasing and Electrical Engineering BOM Documentation & Aziz Alorifi Communications Computer Engineering Housing OpenAL, Pic Mat Stuart Burtner Team Member Computer Engineering

Project Background ● To build a hands-free device to assist Visually Impaired Individuals regain some degree of independence ● Primary key challenges to overcome include orientation and localization ● A secondary key challenge is obstacle avoidance

Task & Update Tracking - POA ● IMU position tracking Feasibility Joshua ● Picture Matching Demo Suhail ● SIFT Demo Stuart ● Obstacle Avoidance Demo Suhail ● Machine learning demo Suhail ● OpenAL Audio Demo Stuart ● Triangulation Demo EJ ● Housing Demo Deepti ● Concept Breakdown VIdeo Aziz ● Edge Maintenance Aziz

Concept Breakdown Review

Picture Matching Demo

Picture Matching Demo ● Pre-SIFT implementation to explore picture matching possibilities ○ Uses opencv technology ○ Maps pixels but does not account for angle, just rotation ● Outcomes: ○ Matches pictures well ○ Does not handle angles unless multiple pre-defined angles are provided ○ This is a possible solution but definitely not an optimal solution

SIFT Algorithm Demo

SIFT Algorithm - Overview Match an input image to one existing in a database: ● Necessary component of the design - solves localization & orientation problem ● Must handle varying degrees of distance and angular skew from reference point When this system produces a match between an Input picture and an existing picture - sound will be played at the location of the match

SIFT Algorithm - Process Three ‘reference points’ taken across household: ● Two “2 - Dimensional” reference points ● One “3 - dimensional” reference point Fourteen test pictures taken: ● Many slightly skewed & scaled images ● Some false-positives (Should match no image) ● Some severely skewed images

SIFT Algorithm - Process Reference Point 1

SIFT Algorithm - Process Reference Point 2

SIFT Algorithm - Process Reference Point 3

SIFT Algorithm - Process Score = 65.2%

SIFT Algorithm - Process Score = 31.25% Score = 36.36%

SIFT Algorithm - Process No Match

SIFT Algorithm - Process Score = 84.06%

SIFT Algorithm - Process Score = 19.23% Score = 23.53%

SIFT Algorithm - Process Score = 0% Score = 45.45%

SIFT Algorithm - Process Score = 38.46%

SIFT Algorithm - Process Score = 0%

SIFT Algorithm - Process No Match

SIFT Algorithm - Process No Match

SIFT Algorithm - Process Score = 0%

SIFT Algorithm - Process No Match

SIFT Algorithm - Process No Match

SIFT Algorithm - Process Score = 13.79% Score = 15.0%

SIFT Algorithm - Process Score = 30.76%

SIFT Algorithm - Outcomes Overall: Sift does not fit our needs ● Does not work with high skew ● Only produces reasonable results in highly similar circumstances Moving forward: Try ASIFT ● Capable of matching at heavy skew ○ Image to the right produced 51 ASIFT matches ○ Originally produced 0 SIFT matches

Position Tracking/IMU Feasibility

IMU Position Tracking Feasibility - Process ● Integration: Drift was handled by creating a threshold of 0.15m/s 2 where any measurement below would be ● rounded to 0. Thus integration was only performed during assumed periods of movement ● Tests were done in which the IMU was moved a set distance at varying “speeds” to measure the accuracy and precision of the integration and error handling.

IMU Position Tracking Feasibility - Outcome ● Overall: Not Feasible. ● No repeatable test was produced Threshold varied from 0.1 up to 0.5 (m/s 2 ) ○ ○ Varied Time Delay from 75ms to 250ms ○ Equations changed to instantaneous readings: ● Next Steps: ○ Possibly look into gravity cancellation ○ Assist with Camera Triangulation Feasibility ○ Adjust scope of device so that it only works while standing still, and after movement the RP will need to be reestablished

Obstacle Avoidance

Obstacle Avoidance - Overview ● Create handsfree system to help user avoid obstacle ● Explore three major facets: ○ AI Poly API ○ Ultrasonic ○ IR Sensor ● Future potential - 3D Depth Sensor

Obstacle Avoidance - Process ● Step 1: Discussed and evaluated AI Poly Potential with Dr. Hochgraf and Machine Intelligence Lab ● Step 2: Considered Ultrasonic option

Obstacle Avoidance - Process ● Step 3: Tested 2 IR sensors in conjunction ○ Gathered IR information independently from left and right side ○ Goal is to expand this so it covers the user from all sides ● Step 4: Explored 3D Depth Sensor

Obstacle Avoidance - Outcome ● Outcome 1 - Ultrasonic is not an aesthetically sound option ● Outcome 2 - AI Poly does not provide any value for obstacle avoidance ● Outcome 3 - IR Sensors are difficult in terms of utility and are not as accurate as initially hoped

Obstacle Avoidance - Conclusion ● Final Conclusion: ○ Shelf obstacle avoidance till February ○ Accomplish primary goals of orientation and localization ○ If accomplished in allotted timeline, work towards obstacle avoidance

Obstacle Avoidance - Alternative/Simplification ● Demo using two main functions: ○ Machine learning of object and image recognition ○ Getting depth through triangulation

Obstacle Avoidance - Afterthought ● For the future, it was determined that RaspberryPi’s depth sensing camera is probably the best path forward ● Pi 3D sensing offers: ○ Built in obstacle avoidance library ○ Stationary and motion obstacle avoidance ○ Plug and play capabilities ○ $179.99

OpenAL

OpenAL - Overview Given an input X, Y, Z Coordinate & a rotation, play a sound ● Critical component for generating a reasonable sound output ● Must produce localizable sound - capable of distinguishing between sounds from the left, right, forward, and behind ● Must be capable of responding to rotational changes

OpenAL - Process A program was created to repeatedly play one sound: ● A .wav file is parsed and read into a format understandable by OpenAL ● At the command-line, X & Y location of the sound can be changed dynamically while the sound continues playing ● Similarly, the listener can be rotated such that it simulates the user ‘turning’ within an environment

OpenAL - Process

OpenAL - Outcome Results: ● Sound is localizable in front of and to either side of a listener ● Difficult to distinguish between forward and behind ● Rotational vector is not correctly implemented Next steps: ● Add reverb FX to sound (available within library) ● Search for methods to improve “behind -the- head” phenomenon ● Determine how to implement 360º rotation

Motion Estimation

Motion Estimation - Overview This serves as a means to track the velocity of an object in an image as well as a method of compression The technique to investigate is a Motion estimation method called spatio-temporal gradient, where by we estimate the motion in an image using Optical Flow

Motion Estimation - Process

Motion Estimation - Process

Motion Estimation - Demo

References Slides and Demo reference obtained from Image and Video Processing on Cousera - https://www.coursera.org/learn/digital

Housing

Housing - Overview Design & develop housing for both prototypes 1. Housing for IMU Prototype 2. Check viability of Product Design for final concept Determined that : - Housing HAS to be compact - especially prototype to be worn on the wrist - PLA can be used for proof-of-concept (melts at relatively high temp.) - ABS should be used for final housing (Stronger, holds more weight)

Housing - Process Plastic Casing for Ultrasonic Sensor+IMU Prototype Components : Speaker, Arduino, Battery Module, IMU Board, Circuit Board Potential Component Replacements : Battery Module, Smaller Circuit Board Design needs to be improved to fit all components in a more compact casing (currently quite big = 3.7 x 2.9 x 2 inches)

Housing - Process 2 Pi Cam Casings Safety Ergonomic Earphones for Glasses Glasses Design for PI Cam Concept Casing for Piping for External Electrical Wiring