nput " Input Presented by: Jim McKeeth Embarcadero - - PDF document

¡

K4 K4

Keynote ¡ 4/16/2015 ¡9:45 ¡AM ¡

¡ ¡ ¡

"Thought hought: : The he Fut Futur ure e of

• f Mobile
• bile

and and Embedded mbedded Applicat pplication ion Input nput"

¡

Presented by: Jim McKeeth Embarcadero Technologies

¡ ¡ ¡ ¡ ¡

Brought ¡to ¡you ¡by: ¡

¡ ¡ ¡ 340 ¡Corporate ¡Way, ¡Suite ¡300, ¡Orange ¡Park, ¡FL ¡32073 ¡ 888-­‑268-­‑8770 ¡·√ ¡904-­‑278-­‑0524 ¡·√ ¡sqeinfo@sqe.com ¡·√ ¡www.sqe.com

Jim McKeeth

Embarcadero Technologies As lead developer evangelist at Embarcadero Technologies, Jim McKeeth is a key part of Embarcadero’s developer community outreach. With more than twenty years of programming experience, Jim travels the world speaking at conferences and sharing his excitement and knowledge. He holds a patent for the swipe to unlock and pattern unlocks used on both iPhone and Android phones, plus a number of other computer- and software-related patents. When not traveling, Jim is an improvisational performer with ComedySportz Boise and enjoys spending time with his family. ¡

4/6/15 ¡ 1 ¡

Thought: The Future of Mobile and Embedded Application Input

JIM MCKEETH EMBARCADERO TECHNOLOGIES

JIM.MCKEETH@EMBARCADERO.COM

@JIMMCKEETH

About Jim McKeeth

• Lead World Wide Developer Evangelist
• Host of Podcast at Delphi.org
• Longtime developer
• Object Pascal, Java, C#, JavaScript, Objective-C, etc.
• Invented and patented swipe to unlock in 2000
• US Patent # 8352745 & 6766456, etc.
• Improvisational performer with ComedySportz Boise
• I am not a neuroscientist, but I find this all fascinating!

2 ¡

4/6/15 ¡ 2 ¡

3 ¡

• Founded 1993
• Provides Architect, Database and Developer Tools
• 3.2 Million Customers including 97% of Fortune 2000
• 35+ Product Awards – Consistent Innovation
• 500+ Employees in 29 Countries
• www.embarcadero.com

Embarcadero Technologies

Agenda

• The Science of Reading the Brain
• The Science of Direct Brain Input
• Consumer Brain-Computer Interfaces
• Demonstration - Volunteer flying the Quadricopter
• Summary & MindTunes

4 ¡

Images ¡from ¡Wikimedia ¡Commons ¡or ¡Product ¡websites.

4/6/15 ¡ 3 ¡

“In five years, science fiction will be real!”

5 ¡

• Claus Torp Jensen (@ClausTorpJensen)

Senior Technical Staff Member and Chief Architect API Economy at IBM (@ibmapimgt)

What will the world be like in five years?

Brain-Computer Interface

6 ¡

ST:TNG ¡Episode ¡93 ¡(4x19): ¡The ¡Nth ¡Degree ¡(1991) ¡ ¡

4/6/15 ¡ 4 ¡

Reading the Brain

The Science of

Polygraph (aka Lie Detector)

• Measures physical

manifestations of stress that come with lying

• Blood pressure, pulse, respiration,

skin conductivity, etc.

• Looks for changes.
• Requires specialized training.
• Invented in 1921

8 ¡

4/6/15 ¡ 5 ¡

Heart Rate Monitors

• Bluetooth GATT Profiles supports

all Bluetooth Smart Heart Rate Monitors.

• Your app doesn’t need to support

individual heart rate monitors.

• Assigned Number: 0x180D

9 ¡

https://developer.bluetooth.org/gatt/profiles/Pages/ProfilesHome.aspx

Magnetic Resonance Imaging (MRI)

• Uses oscillating magnetic fields at

different resonant frequencies.

• Functional MRI (fMRI) uses

software to analyze images.

• Shows activity (blood flow) in

different parts of the brain.

• Combined with brain mapping can

reveal thought information.

10 ¡

4/6/15 ¡ 6 ¡

Electroencephalography (EEG)

• Measures electrical activity along the

scalp.

• Detects voltage fluctuations resulting from

ionic current flows within neurons of brain.

• Millisecond-range resolution (faster than CT or

MRI)

• Typical clinical usage has 19+ input

electrodes as well as ground and reference.

• Invented between 1875 and 1924

11 ¡

Magnetoencephalography (MEG)

12 ¡

Uses Arrays of SQUIDs

(Superconducting QUantum Interference Devices)

4/6/15 ¡ 7 ¡

Direct Brain Input

The Science of

Change Perception of Reality

14 ¡

The ¡“headjack” ¡from ¡The ¡Matrix ¡movies ¡-­‑ ¡1999 ¡

4/6/15 ¡ 8 ¡

Transcranial Direct-Current Stimulation (tDCS)

• Constant, low current delivered via

electrodes.

• Results:
• Increase cognitive performance
• Enhance language & math ability
• Improve attention span
• Aid problem solving, memory &

coordination

• Recent DIY “Brain Hacking” Craze
• http://www.reddit.com/r/tDCS

15 ¡

Accelerated Learning tdcsplacements.com

http://thebrainstimulator.net/

tDCS with Mobile

• foc.us just launched new tDCS

headsets with Bluetooth API

• Targeting athletes and gamers
• Developer devices available
• http://www.foc.us/index.php/v2-tdcs-

developer-edition

• http://www.foc.us/bluetooth-api

16 ¡

4/6/15 ¡ 9 ¡

Transcranial Magnetic Stimulation (TMS)

• Similar strength to MRI
• Magnetic induction of electrical

currents

• Used to treat neurological conditions
• Produces:
• Flashes of lights in vision

(phosphene)

• Muscular activity

17 ¡

Ultrasonic Transcranial Stimulation Device

• 2000 patent from Sony (US6584357 B1)
• Finer targeting than TMS
• Theoretically invoking sensory experiences
• Continued research and patents (US7350522 B2)

18 ¡

Speculative work, “There were not any experiments done.”

4/6/15 ¡ 10 ¡

Ultrasonic Neuromodulation for Spatial Resolution

• Virginia Tech Carilion

Research Institute study published in Jan 2014 Nature Neuroscience

• Used “targeted ultrasound” to

boost “sensory perception”

• Targeted an area of the brain

as small as the size of an “M&M”

19 ¡

hHp://www.eurekalert.org/pub_releases/2014-­‑01/vt-­‑udt011014.php ¡ ¡

20 ¡

Using More than 10% of Our Brain?

LUCY

2014

We all use 1 00% of our Brain Capacity

4/6/15 ¡ 11 ¡

Interesting Milestones

Science Fiction Becoming Reality

Brain-to-Brain Input

22 ¡

The ¡“Hat” ¡from ¡Brainstorm ¡-­‑ ¡1983 ¡

4/6/15 ¡ 12 ¡

Brain-to-Brain Input at University of Washington in 2013

• Input from Computer
• EEG from brain
• Transmitted over campus internet
• Replayed through TMS to another

brain

• Output to computer

23 ¡ More information http://bit.ly/EEG2TMS & http://bit.ly/bci2cbi

See What You See - Hear What You Hear

24 ¡

The ¡“Monitor” ¡from ¡Ender’s ¡Game ¡– ¡2013 ¡

4/6/15 ¡ 13 ¡

See What You See

• In 2010 a team lead by Shinji Nishimoto

partially reconstructed YouTube videos based on brain recordings.

• Used fMRI
• Overlaid similar videos to create composite

video.

• http://youtu.be/nsjDnYxJ0bo

& http://youtu.be/EdGjB6XCXOA

25 ¡

http://bit.ly/ReconstructingVisual

Hear What You Hear

• 2012 - UC Berkeley & Brian Pasley decoded

internal neural processing of auditory information.

• Reconstructed sound on computer from

electrical signals in observer’s brains.

• Resulting sound was recognizable when

compared to original.

26 ¡

hHp://bit.ly/ReconstrucLngSpeech ¡ ¡

4/6/15 ¡ 14 ¡

Data Entry via BCI

27 ¡

ST:TNG ¡Episode ¡93 ¡(4x19): ¡The ¡Nth ¡Degree ¡(1991) ¡ ¡

Typing via BCI

• Brain researcher Jonathan

Wolpaw at Wadsworth Center in Albany, NY

• Wear EEG while looking at a

flashing matrix of characters

• Each character flashes with a

different pattern

• ~15 seconds to type a character
• The Verge http://embt.co/typebci

28 ¡

. . . . . . . . .

A

B C D E F . . G H I J

K

L . . M N

O

P Q R . . S T U V

W

X . . Y

Z

1 2 3 4 . . 5 6 7 8 9 . . . . . . . . .

4/6/15 ¡ 15 ¡

Uploading Your Consciousness

29 ¡

Transcendence ¡-­‑ ¡2014 ¡ ¡

Human Connectome Project

• Map of neural connection the brain
• The neurological version of the genome
• Unique to each person, changes over time
• Representation of memories, thoughts, etc.
• www.ted.com/talks/sebastian_seung
• humanconnectomeproject.org
• humanconnectome.org

30 ¡

4/6/15 ¡ 16 ¡

OpenWorm Project

• Mapped the 300+ neurons and 7000+

connections of a worm (C. elegans)

• Created a software model of connectome
• Connected to LEGO robotic

representation

• Behaved like worm (without training or

programming)

• http://bit.ly/openwormlego

& http://www.openworm.org/

31 ¡

32 ¡

h:p://www.gartner.com/newsroom/id/2819918

• bsolete

Gartner's 2014 Hype Cycle for Emerging Technologies August 11, 2014

BCI ¡

Hype Cycle

4/6/15 ¡ 17 ¡

Brain-Computer Interfaces

Read-Only Consumer

34 ¡

• EEG with 1 dry electrode
• Measures “attention,” “meditation” and

eye-blinks

• Bluetooth communication
• SDK for iOS, Android, PC & Mac
• $129.99 to $79.95 retail neurosky.com
• Hacking:
• http://makezine.com/2011/05/10/behind-the-brain-blinker/
• http://hackaday.com/2012/12/20/modifying-an-eeg-headset-for-

lucid-dreaming/

NeuroSky’s MindWave (also in XWave headsets)

4/6/15 ¡ 18 ¡

Emotiv EPOC

• 16 wet electrodes
• 14 EEG electrodes – read brain waves

+ 2 reference electrodes

• Gyroscope to read head movements
• 4 mental states, 13 conscious thoughts & facial expressions

– 4 Processing suites

• Wireless (encrypted) connection to Windows, Linux or Mac
• $399 desktop or $499 for Bluetooth Smart (coming soon)
• www.emotiv.com

35 ¡

InteraXon’s Muse

• 7 dry EEG sensors via Bluetooth
• Focuses on “brain fitness”
• 4 hour battery
• Basic Mac & Windows SDK now,

full SDK coming soon (incl. iOS & Android)

• Works with Windows, OS X, iOS, Android, etc.
• $299 available now
• choosemuse.com

36 ¡

4/6/15 ¡ 19 ¡

OpenBCI

• Open source hardware & software
• Bluetooth, Arduino, EEG
• 8 wet Electrodes per board (can

daisy chain boards)

• Full open EEG & BCI platform
• www.openbci.com starting at $449
• No iOS or Android yet

37 ¡

Emotiv Insight

• Expected eventually
• Starting at $299
• 5 dry sensors + 2 reference
• Bluetooth 4.0 LE (Smart)
• Battery Life: 4+ hours
• SDK: Android, iOS, Mac, Linux

and Windows Platforms

38 ¡

www.emoLv.com ¡-­‑ ¡Comparison ¡hHp://bit.ly/1cfliTn ¡ ¡

4/6/15 ¡ 20 ¡

No More Woof (“work in progress”)

• EEG for Dogs
• Recognize brain patterns and

match with mental states:

• Hungry, Playful, Curious, etc.
• Translates to human
• By Nordic Society for Invention

and Discovery (NSID)

• Raised $22K on IndiGoGo

39 ¡

http://www.nordicinvention.com/nmw.html

Emotiv EPOC

More Details on the

4/6/15 ¡ 21 ¡

Expressiv Suite

• Detect facial expressions
• Eyelid & eyebrow positions

– Wink, blink, furrow

• Horizontal eye movement
• Smile, laugh, clenching & smirking
• EEG sensors picking up signals to

muscles (not brain waves)

• Very fast (10ms)

41 ¡

Uses for Facial Detection

• Realistic avatar expressions
• Anything requiring fast input
• Facial gesture detection
• Basic eye tracking
• Fast input & responses

42 ¡

4/6/15 ¡ 22 ¡

Affectiv Suite

• Detect emotions / mental states
• Excitement
• Engagement or Boredom
• Meditation
• Frustration
• Passive detection
• Variable levels of each
• Short & long term tracking

43 ¡

Uses for Emotion Tracking

• Realtime media evaluation
• Stress analysis and

management

• Mood monitoring
• For more accuracy combine with
• ther data sources
• Eye tracking, heart rate, etc.

44 ¡

4/6/15 ¡ 23 ¡

Cognitiv Suite

• Detect conscious thoughts
• Requires training
• Detect 13 thought patterns with

intensity level

• Left, right, up, down, forward, pull
• CW, CCW, left, right, sway-back,

sway-forward, Disappear

• Not as fast as Expresiv

45 ¡

Uses for Cognitive Thought Identification

• Whole new input method
• Variable intensity for fine control
• Observe subconscious responses
• More training improves accuracy
• Improves the model
• Trains your brain to focus

46 ¡

4/6/15 ¡ 24 ¡

The Pieces

47 ¡

Emotiv EPOC Brain-Computer Interface Google Glass Wearable Computer Parrot.AR Drone 2 Simi-Autonomous

About Appmethod

• Integrated Development Environment
• Multi-platform visual designer
• Native compiler for Win32, Win64, OS X, iOS & Android
• Framework for rapid prototyping and accelerated

development with C++ or Object Pascal

• Architecture allowing shared code & UI across platforms
• Advanced language features like ARC, Generics, etc.

48 ¡

www.appmethod.com

Starts at $25/mo per platform

• r

C++ Android Phone for Free

4/6/15 ¡ 25 ¡

Laptop ¡

EDK ¡ My ¡ App ¡

How it Works

49 ¡

Brain ¡

Electrical ¡ Signals ¡ Wireless ¡ Signal ¡

AT*REF=1,290718208 ¡

Provides ¡ WiFi ¡ ¡ Hotspot ¡

Feedback ¡

Emotiv EPOC SDK + Parrot AR.Drone + Google Glass

Demonstration

Tweet @Appmethod and #IoT

• r G+, LinkedIn, Facebook, etc.

4/6/15 ¡ 26 ¡

Emotiv EPOC SDK

• Platforms
• Windows
• Mac OS X
• Ubuntu
• Fedora
• iOS & Android coming soon

51 ¡

• Languages
• C / C++ / Objective-C
• C# (.NET)
• Python
• Object Pascal / Delphi
• Java coming soon

MindTunes

• Three disabled musicians
• Collaborated with DJ Fresh and

Julien Castet (tech)

• Created dance track
• http://youtu.be/HyF4ZxGhPHw
• Full documentary
• http://youtu.be/PgfxKZiSCDQ

52 ¡

4/6/15 ¡ 27 ¡

More Information

• Email: jim.mckeeth@embarcadero.com
• Emotiv EPOC: www.emotiv.com
• Parrot AR.Drone: projects.ardrone.org
• Appmethod: www.appmethod.com
• My blog: delphi.org
• Full Slide Deck:

http://www.slideshare.net/jimmckeeth/jim-mc-keeth- wearable-thought-input

• Code: https://github.com/jimmckeeth/Delphi-Emotiv-EPOC

53 ¡