Amaze Your Users With Augmented Reality Colin Cornaby 360iDev - - PowerPoint PPT Presentation

Amaze Your Users With Augmented Reality

Colin Cornaby 360iDev 2013

Colin Cornaby - @colincornaby

• Evolution of Smartphone Development And

Augmented Reality

• Available Augmented Reality SDKs and Technologies
• Design Considerations
• Questions

Contents

Colin Cornaby - @colincornaby

Bio

• Full time iOS/Android Developer.
• Moved to Objective-C/Cocoa in 2001.
• Started iOS development when the SDK was

announced.

• Worked on shareware, database applications, Final

Cut Pro, Photoshop and Premiere Plugins, and mobile development.

• Currently iOS Developer at Digimarc on the

Application’s Team.

Colin Cornaby - @colincornaby

Digimarc

• Research focused, publicly traded company based out
• f Beaverton, OR.
• Focused on Augmented Reality Research and SDKs.

Colin Cornaby - @colincornaby

Digimarc Discover

• Scans for barcodes
• Scans for our own special audio and visual content
• (Previously) Music Identification
• Saves content for later viewing

Why Augmented Reality?

Nintendo 3DS Augmented Reality Games (erste Deutsche Version) - http://www.youtube.com/watch?v=GryGuy3-ZIg

Augmented Reality For The Rest Of Us

(But We’ll Talk About How To Make 3D Characters Appear On A Card Too)

Colin Cornaby - @colincornaby

Early Mobile Development

• Concentration on navigating a “Network of Devices”
• Email
• Web Browser
• Other Devices or Computers

Colin Cornaby - @colincornaby

iPhone

• Great for talking to devices!
• Real web browser for talking to HTTP servers!
• Bluetooth and Wifi for talking to headsets and

Wifi routers!

• iPod Dock Connector for talking to computers!
• (Eventually) MobileMe for talking to the cloud!

Colin Cornaby - @colincornaby

• Showed how a mobile device could interact with

things around the user.

• Instead of abstracting devices, Maps abstracted

physical things, providing a new view of the world around the user.

• Early technology stitched together existing

technologies - maps, landmarks, place metadata - to provide a new abstraction for a physical environment and make for an amazing experience.

Maps Demo

Network Of Devices

Network Of Things

But Things Don’t Have IP Addresses!

(Usually)

So How Do We Talk To Things?

Colin Cornaby - @colincornaby

Augmented Reality

• “Augmented reality (AR) is a live, direct or indirect,

view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, graphics or GPS data.” - Wikipedia, http:// en.wikipedia.org/wiki/Augmented_reality

Context + Content

Colin Cornaby - @colincornaby

Context + Content

• Context is provided by the sensors on the device.
• The Maps application finds waypoints around you

based on location data provided by both GPS and the onboard Wifi.

• Content is the information the context is used to

filter.

• In the Maps application Apple’s waypoint database

is filtered by the user’s current location.

But I write an SSH

• client. Why does this

matter to me?

Colin Cornaby - @colincornaby

• Maybe your application can remember settings based
• n location.
• You could also scan content from printed materials

to add convenience for your users.

• Remind your user about something that’s nearby.
• Use sensors to build a usage pattern and have helpful

information ready for your users before they ask for it.

So How Can We Use These Sensors? (APIs)

Colin Cornaby - @colincornaby

Camera

• iOS devices include great, high quality image sensors

that can be manipulated by AVFoundation API.

• All current Apple devices ship with front cameras,

and most ship with rear cameras.

• The lowest end iPod Touch current has no rear

camera.

Colin Cornaby - @colincornaby

AVFoundation - built in AR!

• Apple has begun rolling out some AR functionality

through CoreImage.

• iOS 6 defines a CIDetector class that currently only

provides face recognition.

• API to pay attention to in future releases.

Colin Cornaby - @colincornaby

Barcodes

• Almost all products that are packaged have
• barcodes. This provides an easy path for any

application that is product or shopping oriented.

• But identifying what item the barcode belongs to can

be difficult. A barcode is a unique number, but it doesn’t provide any other identifying information about a product.

Colin Cornaby - @colincornaby

Barcodes

• There are some free-to-access databases that are

crowd sourced.

• Retailers are a great channel for getting information
• n products based on UPC code.
• Google Shopping can search for products based
• n UPC code.

Colin Cornaby - @colincornaby

Barcodes

• Retailer information can also be heavily guarded.
• There is concern from retailers that having an API

would let competitors run automated scripts to instantly price match.

• Delicious Monster (makers of Delicious Library) had

their iPhone application yanked from the App Store because Amazon didn’t want their API to be used on mobile devices.

Colin Cornaby - @colincornaby

Barcodes

• QR codes are another way to embed information on

real world objects, and they’re easy to create.

• They can hold any arbitrary data, not just URLs.
• They can also be ugly.

Colin Cornaby - @colincornaby

Barcodes

• Several SDKs available to read barcodes
• Zxing (pronounced Zebra Crossing)
• ZBar
• These APIs can read UPC, EAN, and QR codes,

among many other types.

Colin Cornaby - @colincornaby

Digimarc Discover SDK

• Digimarc’s Discover SDK allows you to read data

from images that have been embedded with invisible “watermarks.”

• Watermarks can be embedded into images via our

web portal or a Photoshop plugin.

Colin Cornaby - @colincornaby

Digimarc Discover SDK

• On detection, a 32 bit or 64 bit id is derived from

the hidden mark in the image. Once you have that payload, our servers can use that as a key to return your application actual content.

• Highly accurate, but the media that is being read

must have been previously marked.

• API is designed so that any Discover enabled

application can read any Discover enabled media, no siloing.

Colin Cornaby - @colincornaby

Digimarc Discover SDK

• Sports Illustrated Swimsuit Edition
• Cooking Light
• House Beautiful
• Packaging

Colin Cornaby - @colincornaby

Content Recognition/Metaio

• Metaio is an augmented reality platform built around

image recognition with an available SDK. Uses algorithms to try to match different characteristics of an image against a database.

• Several other comparable platforms such as Layar

and Aurasma.

• Because Metaio knows more details about the image

itself, the API can be used for projecting 3D objects

• r videos onto surfaces.

Metaio Demo

Metaio Hello World: http://docs.metaio.com/bin/view/Main/HelloAugmentedWorldExample

Metaio Demo

Colin Cornaby - @colincornaby

Content Recognition/

• Image analysis algorithms specific to this sort of AR are

making guesses as to which different images are supposed to match different pieces of content. Can lead to confusion of different images in an app.

• As an application’s database of known images gets

larger, risks of false positives increase.

• But if absolute reliability isn’t required, and you do

want features like 3D projection, this technology is a good choice.

• Doing this processing in the cloud is also possible

(Google Goggles).

Colin Cornaby - @colincornaby

OCR

• Optical Character Recognition
• Tesseract-ocr is an open source OCR SDK
• riginally created by HP and now maintained by

Google.

• Several real time text translation apps around OCR.

One of the first was an iOS app called “Word Lens”.

• Many other use cases. Reading business cards,

invoices, addresses, etc.

Word Lens

Colin Cornaby - @colincornaby

OpenCV

• “OpenCV (Open Source Computer

Vision Library) is an open source computer vision and machine learning software library. OpenCV was built to provide a common infrastructure for computer vision applications and to accelerate the use of machine perception in the commercial products.” - http://opencv.org/about.html

Colin Cornaby - @colincornaby

OpenCV

• “...can be used to detect and recognize faces, identify
• bjects, classify human actions in videos, track

camera movements, track moving objects, extract 3D models of objects, produce 3D point clouds from stereo cameras, stitch images together to produce a high resolution image of an entire scene, find similar images from an image database, remove red eyes from images taken using flash, follow eye movements, recognize scenery and establish markers to overlay it with augmented reality, etc.” - http://opencv.org/ about.html

Colin Cornaby - @colincornaby

OpenCV

• Open source and constantly evolving.
• Work on the iOS version is still ongoing.
• The most bleeding edge, and possibly the most

feature rich SDK for augmented reality.

Colin Cornaby - @colincornaby

CVFunhouse

• Demo project for OpenCV on iOS
• https://github.com/jeradesign/CVFunhouse

Canny Ege Detection

Hough Transform

Apps that Can See: Getting Data from CoreImage to Computer Vision Carl Brown Monday, 11:50, Salon C

Colin Cornaby - @colincornaby

Microphone/Audio

• All iOS devices have some sort of audio input.
• One advantage of audio is that it’s the interaction

with it is passive. It doesn’t necessarily require the user to do anything special with the device.

• Audio recording is also a supported background

mode.

Colin Cornaby - @colincornaby

Digimarc Discover Audio SDK

• Works very similarly to our image based SDK. Can

recognize a payload that has been embedded into an audio track, and then our servers can provide a link to related content.

• Very reliable and very fast, as long as you can supply

pre-hinted content.

• Great for TV commercials, retailers, hinted music,

etc.

• Hinted content can currently be created at our web

portal.

Colin Cornaby - @colincornaby

Digimarc Discover Audio SDK

• Identification of Marked Content
• Music Content
• Radio
• Television Shows/Commercials
• Locational Awareness

Colin Cornaby - @colincornaby

Gracenote

• The Gracenote SDK analyzes music through the

microphone to try to identify the track.

• It will also provide metadata such as the artist,

album, current time in the track, etc.

• Available freely for non commercial use.

Colin Cornaby - @colincornaby

Gracenote/Entourage

• Entourage is a second API from Gracenote that tries

to provide real time music recognition.

• No button to hit, Entourage is built to be

continuously listening.

• Contains capabilities for identifying TV content.
• Also has features to try to infer properties from
• audio. Is it music or speech? Did the track just

change?

Colin Cornaby - @colincornaby

Accuracy

• Like image recognition, can suffer from accuracy

issues.

• But it’s the best way to build an app that understands

content that you can’t control or pre-hint.

Colin Cornaby - @colincornaby

Location

• Location can used to build a great standalone or

complimentary augmented reality experience.

• Apple’s CoreLocation framework can provide course

and fine grain location.

• On devices with GPS hardware this location can be

very exact.

• Also supported as background mode. Can provide

feedback based on entering a location or location changes.

Portland, OR 45°31′12″N 122°40′55″W

Geocoding

45°31′12″N 122°40′55″W Portland, OR

Reverse Geocoding

Colin Cornaby - @colincornaby

Geocoding

• Both geocoding and reverse geocoding are

supporting by CoreLocation.

• You can even geocode and reverse geocode with the

native Address Book API data structures.

• Note: Under iOS 5 and earlier, these requests are

routed through Google, and they may have restrictions on how you can use these calls.

Colin Cornaby - @colincornaby

Location - Magnetometer

• Many iOS devices (iPhones and iPads) feature a

Magnetometer sensor that can act as a compass.

• Typically used to infer what direction the user is

facing in relation to points of interest in the world around them.

Magnetometer

http://www.joinpad.net/augmented-reality-2/

Augmented Reality Geolocation Math Dalmo Cirne Tuesday, 12:30, Salon C

Colin Cornaby - @colincornaby

Bluetooth LE Beacons

• Small, low power beacons suitable for providing

micro location.

• Bluetooth 4.0 means no pairing
• Pocket sized and last for years of a single battery
• Multiple

Vendors

• KS Technologies -http://www.kstechnologies.com/

products/ibeacon-alpha-program-participants-only

• Tōd - https://www.todhq.com

Colin Cornaby - @colincornaby

What does a beacon

• UUID
• This identifies a beacon as yours.
• Major ID
• Can be anything you want. Possible use as a
• category. Usual example is a store location, where

UUID defines the store chain.

• Minor ID
• Also anything you want. Usually a sub attribute of

major - i.e. a store aisle.

Colin Cornaby - @colincornaby

Bluetooth LE Beacons

• Beacons last years on a single battery
• They’re also portable
• Carry one on you, stick one in your car, put one

in your backpack, etc.

• Bluetooth LE means you don’t have to pair them.

Colin Cornaby - @colincornaby

Beacon Distance

• You can test the wireless strength of a signal to imply

distance from the beacon.

• Common example: Beacon broadcasts from cash

register in store. One action when the user enters the store, another action when they are at the cash register.

API for iOS?

Colin Cornaby - @colincornaby

Bluetooth Beacons

Colin Cornaby - @colincornaby

Bluetooth Beacons

CoreLocation

Colin Cornaby - @colincornaby

Bluetooth Beacons

CoreLocation Location is a supported background mode

Colin Cornaby - @colincornaby

Bluetooth Beacons

More Frameworks?

Colin Cornaby - @colincornaby

Bluetooth Beacons

iOS Device As Beacon?

Colin Cornaby - @colincornaby

KST Technologies - 360iDev

• KS Technologies has hidden several Bluetooth

beacons around 360iDev.

• Play with the technology - especially if you’re on iOS

7!

Colin Cornaby - @colincornaby

Write Your Own!

• iOS has a lot of tools for creating great augmented

reality experiences from scratch!

• Grand Central Dispatch for harnessing multiple

cores

• Accelerate/NEON for fast processing of audio or

video frames

• CoreAudio/AudioUnits for best in class audio

processing

• AVFoundation for video sources

Design Considerations

Colin Cornaby - @colincornaby

Power Usage

• CPU Power is advancing, and batteries aren’t.
• Will a user expect your application to last all day?
• Write better, more battery friendly algorithms.
• Or use lower power sensors to inform higher

powered sensors.

• Accelerometer or basic microphone data doesn’t

require much power to acquire.

Colin Cornaby - @colincornaby

Understanding Context

• Some information your application ingests might

need to be processed.

• OCR could read any string, and your application

may not understand the content.

• A QR code could contain any information.

Colin Cornaby - @colincornaby

QR code example

http://apple.com BEGIN:VCARD VERSION:3.0 N:Appleseed;Johnny The quick brown fox...

Colin Cornaby - @colincornaby

QR code example

• How do we know if a QR code contains a URL?
• It starts with http or https
• Ask if iOS can handle this URL
• It ends in .com
• It has a TLD

x x x x

NSDataDetector *detector = [NSDataDetector

• dataDetectorWithTypes:NSTextCheckingTypeLink
• error:&error];

NSArray *matches = [detector matchesInString:string

• ptions:0 range:NSMakeRange(0, [string length])];

Colin Cornaby - @colincornaby

Data Detectors

• Data Detectors are a great hidden gem on iOS.
• Given any arbitrary string, a data detector can find

and sanitize dates, addresses, links, phone numbers, and transit information.

• A basic use might be coding an OCR app that

could recognize a phone number or a link off a printed page, and then understand and use the content!

• Wrapped in the NSDataDetector class.

Colin Cornaby - @colincornaby

Privacy

• Device sensors may collect a lot of data on the user,

and require usage of data that has been collected by the user.

• Still an open question. What is the right balance for

users?

• If you’re collecting data or using device sensors, be

clear about what you are doing, why you’re doing it, and what data is being collected.

• iOS will ask the user’s permission automatically for

many different sensors.

Colin Cornaby - @colincornaby

Gamification

• Bridging virtual gaming to the real world.
• Scavenger hunt model
• Foursquare, Facebook, etc.
• Great way to keep users interacting with your brand
• r your physical presence.

Colin Cornaby - @colincornaby

Mis-identification

• Technologies that use algorithms to guess what

content is can sometimes guess wrong.

• Music identification SDKs and apps can get a song

wrong.

• Image matching algorithms can get confused and

return the wrong result.

Colin Cornaby - @colincornaby

No Results

• A song matching application may not be able to get a

result in a room that’s too noisy.

• A barcode reader might not be able to read a

barcode that’s on certain surfaces, or a QR code that is too complex.

• Users might try and use your application with objects

that your application was never designed to work with.

Colin Cornaby - @colincornaby

Manage Expectations

• Make clear to the user what sorts of objects your

application is intended to work with.

• Depending on the technology you are using, assume

your application could be wrong.

• Gracefully provide feedback to the user.
• Infer behaviors. If the user hasn’t triggered any

behaviors in your application, they could be trying to interact with something your application doesn’t understand.

What’s Next?

Colin Cornaby - @colincornaby

Wearable Computing

• Google Glass and the iWatch
• Opens the door to true always-on augmented

reality.

• Phones are great devices, but you have to take

your phone out of your pocket and activate the right app.

Colin Cornaby - @colincornaby

Wearable Computing

• Other Devices Just Provide Sensor Input From the

World Around You Or Your Own Body

• Fuelband
• Fitbit
• Glucose Meters

Apps that Can See: Getting Data from CoreImage to Computer Vision Carl Brown Monday, 11:50, Salon C Augmented Reality Geolocation Math Dalmo Cirne Tuesday, 12:30, Salon C Take the Red Pill and Leave the Matrix: Writing Apps for the Physical World Carl Brown Tuesday, 5:10, General Session

Colin Cornaby - @colincornaby

Questions?

Colin Cornaby @colincornaby (Twitter, App.net) colin.cornaby@digimarc.com Digimarc http://digimarc.com, http://developer.digimarc.com