Fixing the image problem of the web using machine learning Chris - - PowerPoint PPT Presentation

▶
fixing the image problem of the web using machine learning
SMART_READER_LITE
LIVE PREVIEW

Fixing the image problem of the web using machine learning Chris - - PowerPoint PPT Presentation

Sep 01, 2023 338 likes •1.06k views

Fixing the image problem of the web using machine learning Chris Heilmann @codepo8, GotoCon Copenhagen, October 2016 Of innovation and impatience CHRIS HEILMANN @CODEPO8 Chris Heilmann @codepo8, Future Decoded, London, Nov 2015 We all know

slide-1
SLIDE 1

Fixing the image problem of the web using machine learning

Chris Heilmann @codepo8, GotoCon Copenhagen, October 2016
slide-2
SLIDE 2

Of innovation and impatience

Chris Heilmann @codepo8, Future Decoded, London, Nov 2015

CHRIS HEILMANN @CODEPO8

slide-3
SLIDE 3

We all know this character, right?

https://en.wikipedia.org/wiki/Mario#Concept_and_creation
slide-4
SLIDE 4

But do you know why it looks like it does?

https://en.wikipedia.org/wiki/Mario#Concept_and_creation
slide-5
SLIDE 5

Red and Blue offered the best contrast to the skin, boots and the game background.

https://en.wikipedia.org/wiki/Mario#Concept_and_creation
slide-6
SLIDE 6

The cap meant there was no need to worry about hair style, eyebrows and forehead.

(There were also not enough pixels for waving hair when falling down a hole)

slide-7
SLIDE 7

The large nose and moustache made it possible to avoid a mouth and facial expressions.

slide-8
SLIDE 8

Design by limitations.

slide-9
SLIDE 9

Everything has its reasons and meaning.

⚙

slide-10
SLIDE 10

🖦→💼→📲

Evolution is happening around us… …and user numbers are shifting.

slide-11
SLIDE 11

T echnology advanced

📲

slide-12
SLIDE 12

Pixels are a side-product

  • f our interactions on

the web.

📹 📺

slide-13
SLIDE 13

So much, that they become a nuisance…

Type Size (kB) Images 1426 Scripts 357 Video 174 Fonts 123 Stylesheets 76 HTML 67 Other 4 Total 2232 kB http://www.httparchive.org/interesting.php#bytesperpage
slide-14
SLIDE 14

Inspirational obesity

🖍 🐙

slide-15
SLIDE 15
  • Wrong file formats
  • Delivering scaled hi-res

images to everybody

  • No automatic conversion and
  • ptimisation steps
  • Hero image instead of text

content

1.4 MB images…

slide-16
SLIDE 16

We need to change that to make the web fast again…

🚁

slide-17
SLIDE 17

The web is much bigger than our little developer world and growth happens

  • utside of it.
slide-18
SLIDE 18 https://www.flickr.com/photos/89306448@N00/334479803 {Guerrilla Futures | Jason Tester}

Surgical solutions: Proxy Browsers and Cloud Services

slide-19
SLIDE 19

Here are a few things you can do.

slide-20
SLIDE 20
  • Huge images for everybody
  • Unoptimised images
  • No alternative content
  • No training or incentive to

add content in CMS

The problems:

slide-21
SLIDE 21
  • Better browsers with responsive

image support

  • Automated, loss-less image
  • ptimisation tools
  • File level access to images to

extract metadata

  • Scripting solutions to offer

alternative content

  • Cloud services with machine

learning APIs for intelligent resizing

  • Machine learning for tagging

Our arsenal:

slide-22
SLIDE 22

Browsers with responsive image support…

slide-23
SLIDE 23 https://www.christianheilmann.com/ 2012/12/19/conditional-loading-of- resources-with-mediaqueries/

Media queries are greedy!

slide-24
SLIDE 24

🖽

<picture> element and srcset

slide-25
SLIDE 25

Support is great!

slide-26
SLIDE 26 https://jakearchibald.com/2015/ anatomy-of-responsive-images/

More info…

slide-27
SLIDE 27 https://dev.windows.com/en-us/microsoft- edge/testdrive/demos/picture/

Live demo…

slide-28
SLIDE 28

Automated tools for lossless image optimisation

slide-29
SLIDE 29 https://imageoptim.com/ PNGOUT, Zopfli, Pngcrush, AdvPNG, extended OptiPNG, JpegOptim, MozJPEG ( jpegtran & jpegrescan), Gifsicle.

ImageOptim

slide-30
SLIDE 30

File-level access to embedded information in images

slide-31
SLIDE 31 http://code.flickr.net/2012/06/01/parsing- exif-client-side-using-javascript-2/

EXIF - hidden gems

slide-32
SLIDE 32 http://code.flickr.net/2012/06/01/parsing- exif-client-side-using-javascript-2/

EXIF - hidden gems

slide-33
SLIDE 33

EXIF - THE HIDDEN GEMS

http://code.flickr.net/2012/06/01/parsing- exif-client-side-using-javascript-2/
slide-34
SLIDE 34 http://removephotodata.com/

Removing EXIF data…

slide-35
SLIDE 35 http://removephotodata.com/image-to-map.html

Remove locations…

slide-36
SLIDE 36 http://www.datapointed.net/2010/01/crayola-crayon-color-chart/

How about providing fallback content?

slide-37
SLIDE 37

Blur-up technique

https://css-tricks.com/the-blur-up-technique- for-loading-background-images/
slide-38
SLIDE 38 https://codepo8.github.io/canvas-images-and-pixels/ http://colorify.rocks/index.html

Counting pixels…

slide-39
SLIDE 39 https://codepo8.github.io/canvas-images-and-pixels/

Counting pixels…

slide-40
SLIDE 40

Colorify.js

http://colorify.rocks
slide-41
SLIDE 41

https://github.com/jwagner/smartcrop.js/

http://lokeshdhakar.com/projects/color-thief/

Color thief…

slide-42
SLIDE 42

SCRIPTING SOLUTIONS FOR ALTERNATIVE CONTENT

Intelligent image resizing

slide-43
SLIDE 43 https://github.com/jwagner/smartcrop.js/

Smartcrop.js

slide-44
SLIDE 44

Intelligent resizing

http://cloudinary.com/blog/ automatically_art_directed_responsive_images
slide-45
SLIDE 45

Intelligent resizing

http://cloudinary.com/blog/ automatically_art_directed_responsive_images
slide-46
SLIDE 46

High contrast tricks

https://blog.imgix.com/2015/10/21/automatic- point-of-interest-cropping-with-imgix.html
slide-47
SLIDE 47

🔯

What about information that isn’t in the image?

slide-48
SLIDE 48

Machine learning and artificial intelligence to the rescue

slide-49
SLIDE 49

Automated alternative text (Facebook)

Image may contain: dog,
  • utdoor and nature
slide-50
SLIDE 50 Image may contain: dog,
  • utdoor and nature
https://code.facebook.com/posts/561187904071636

How?

slide-51
SLIDE 51

Making photos findable…

https://photos.google.com
slide-52
SLIDE 52

Learning from good data…

http://image-net.org/
slide-53
SLIDE 53

Learning from *lots* of data…

http://image-net.org/
slide-54
SLIDE 54

Learning from *lots* of data…

https://github.com/
  • penimages/dataset
slide-55
SLIDE 55

Learning from *lots* of data…

https://github.com/
  • penimages/dataset
slide-56
SLIDE 56

Refining the data with language

https://research.googleblog.com/2016/09/show-and-tell-image-captioning-open.html
slide-57
SLIDE 57

Compare and enhance…

slide-58
SLIDE 58

Detect syntax and mix and match

slide-59
SLIDE 59

Add visual information

slide-60
SLIDE 60

Adding automated content in context…

https://research.googleblog.com/2016/05/aw-so-cute-allo-helps-you-respond-to.html
slide-61
SLIDE 61 http://captionbot.ai

APIs and demos to use yourself…

slide-62
SLIDE 62

Detecting humans…

slide-63
SLIDE 63
slide-64
SLIDE 64
slide-65
SLIDE 65
slide-66
SLIDE 66 Detection Result: JSON: [ { "faceRectangle": { "width": 109, "height": 109, "left": 62, "top": 62 }, "attributes": { "age": 31, "gender": "male", "headPose": { "roll": "2.9", "yaw": "-1.3", "pitch": "0.0" } "faceLandmarks": { "pupilLeft": { "x": "93.6", "y": "88.2" }, "pupilRight": { "x": "138.4", "y": "91.7" }, ...

Detect Faces…

slide-67
SLIDE 67 Verification Result: JSON: [ { "isIdentical":false, "confidence":0.01 } ]

Verify Faces

slide-68
SLIDE 68

Cluster automatically…

slide-69
SLIDE 69

🦅

Empowering people…

slide-70
SLIDE 70 https://www.youtube.com/watch?v=R2mC-NUAmMk
slide-71
SLIDE 71

CHRIS HEILMANN @CODEPO8 CHRISTIANHEILMANN.COM

THANKS!