Responsive Images

Yoav Weiss

BlinkOn - Zurich, May 2014

yoavweiss.github.io/respimg-blinkon-presentation

Who????

• Srcset & Picture implementation

Working on responsive images in my spare time for the last 2 years

A member of the RICG

A Blink & WebKit committer

Prototyped picture in WebKit. Implemented srcset in Blink. Now implementing (the new) picture.

I also been working on front end optimization server side solutions for the last 15 years, and am on a personal vendetta on image bloat on the Web

What is Responsive images?

Efficiently load properly dimensioned images that fit the page's design

Who knows what the responsive images problem is?

Started out as

It started out (or at least got people's attention) as a quality issue.

The solution to that was simple - just send users the big images

"But that's easy"

"Just send the largest possible image"

"And let the browser resize it"

EAZZZZZZY!!!

Well...

This led to BLOAT!!!

Mobile experience became better, but slower

"a responsive site" became a synonym of "a slow site"

72% Serve same resources

guypo.com

Serving the same resources to mobile and desktop hurts performance

On mobile the images are much smaller and some of them are not even displayed

Which resources?

Images - over 61%

httparchive.org

How much can be saved?

Up to™ 72% image data savings

So quality issues became

data plan abuse

And...

Large images?

People demanded a solution

Turned to the mailing lists

Proposals!

Moar proposals!

And the RICG was born

Picture vs. srcset

Turned into picture *with* srcset

The CG came up with the picture proposal, Hixie added srcset to the HTML spec.

A lot of mailing list flame wars. Things got tense.

Over time, the srcset syntax was adopted into the picture syntax, since they covered different use cases

Browsers weren't convinced

Src-N

Then after a long while of little progress

SrcN was proposed by TabAtkins and John Mellor, after a couple of meetups we had with John.

It resolved all the use case, in a single element, but got resistence from some browser people.

Back to picture

The following an IRC chat I had with Simon Pieters, we figured out a way to gather up all the good pieces

from Src-N and wrap it inside something that looks really like the original picture syntax,

only significantly easier to spec, implement and maintain.

TabAtkins then revived the picture spec (basically rewrote most of it), and we've been working on it with Tab and

Simon ever since.

And now

The RICG & whatwg/blink folks are working together on the spec, to make it as awesome as possible. Mozilla is also heavily involved

The implementation in Blink is moving along nicely, and Gecko are happily implementing as well.

If all goes well, we may have the feature sihpping in a few months.

Picture 2.0™

Let's take a closer look at each one of the parts combining the latest spec.

The srcset 'x' part

The oldest part of the spec, goes back to the WHAWG 2012 days

Use case - "Retina images"

Load hi-res images

on hi-res devices

The syntax

<picture>
     <source media="(min-width: 45em)" 
             srcset="large.jpg 1x, large-2x.jpg 2x">
     <source media="(min-width: 18em)" 
             srcset="medium.jpg 1x, medium-2x.jpg 2x">
     <img src="small.jpg" srcset="small-2x.jpg 2x" 
          alt="The president.">
</picture>

Or even

<img src="small-1x.jpg" srcset="small-2x.jpg 2x" 
     alt="The president.">

The <picture> part

Use case - Art direction

Use case #2 - MIME type fallback

Today's Web have become fragmented when it comes to image formats.

We may not like it, but that's the truth :(

Picture will enable us to have client-side mime type fallback if we're using browser specific image formats, just like all other resources which they're type may or may not be supported (e.g. video, fonts).

Note that it doesn't mean that you can't do server-side content negotiation using the accept header, if that's your thing

But client-side may be more accessible to Web developers that can't mess around with their backend, and it has caching advantages.

The syntax

<picture>
     <source media="(min-width: 45em)" srcset="large.jpg">
     <source media="(min-width: 18em)" srcset="medium.jpg">
     <img src="small.jpg" alt="The president.">
</picture>

Here you can see the syntax parts that make up the "picture" part of the spec, namely the picture element and its source children.

Each one of the source children, as well as the img child define an image that fits into a certain responsive breakpoint.

The <img> is not optional

<picture>
     <source media="(min-width: 45em)" srcset="large.jpg">
     <source media="(min-width: 18em)" srcset="medium.jpg">
     <img src="small.jpg" alt="The president.">
</picture>

What's important to understand, and the big difference between this spec and the previous picture version, is that the img element is not optional

It must be there in order for an image to be downloaded and displayed on screen.

picture itself, is very much like a div, with the only difference is that img will look up the resource that it'd pick to download when picture is its parent.

The sizes + srcset 'w' descriptor part

http://ericportis.com/posts/2014/srcset-sizes/

The third part is IMO, the most exciting part.

This is the major innovation that the original src-N proposal brought

It's the most exciting since it applies almost everywhere, once you look at it.

It can be used to save bandwidth in both responsive and adaptive designs and even if hi res displays are not involved.

use case - variable width images

The syntax

<picture>
    <source media="(max-width: 80em)"
            sizes="(max-width: 30em) 100vw, 
             (max-width: 50em) 50vw, 
             calc(33vw - 100px)"
            srcset="pic100.jpg 100w, pic200.jpg 200w, 
                    pic400.jpg 400w, pic800.jpg 800w, pic1600.jpg 1600w, 
                    pic3200.jpg 3200w">
    <img src="otherpic.jpg" alt="The president giving an award.">
</picture>

Or even

<img src="otherpic.jpg" alt="The president giving an award."
     sizes="(max-width: 30em) 100vw, 
         (max-width: 50em) 50vw, 
         calc(33vw - 100px)"
     srcset="pic100.jpg 100w, pic200.jpg 200w, 
             pic400.jpg 400w, pic800.jpg 800w, pic1600.jpg 1600w, 
             pic3200.jpg 3200w">

Browser support

blog.thestudio4.co.uk

Blink - Chrome & Opera

I started out by implementing srcset's x descriptor. It shipped in Chrome/Opera stable a few weeks back.

sizes and the 'w' descriptor in srcset are done. I'll probably aim to ship it in M37

picture work has started and is moving along nicely - Christian is helping with stable state

Firefox

IE

They're showing up on IRC and mailing lists, asking good questions.

WebKit

Implemented srcset's 'x' descriptor

Willing to accept patches behind a compile flag

They're not very enthusiastic about that.

A look under the hood

http://www.newyorkshitty.com/williamsburg/3196

Media Queries evaluation

PreloadScanner requires off-main-thread MQ eval

Getting picture and sizes to wrok requires MQ eval in the preloadScanner

Since the Bison parser and related classes are not thread safe, I had to create an alternative mechanism

MediaQueryTokenizer

Based on eseidel's CSS syntax tokenizer

Implemented the parts needed for MQs and sizes

http://en.numista.com/catalogue/pieces23200.html

I started by taking Eric's tokenizer and adapting it to my needs.

I've deleted the parts that I didn't need and implemented the missing parts that I did

MediaQueryParser

Currently turned on only off-main-thread

Then I've build a parser for MediaQuery syntax that is based on the tokenizer's output

It's currently turned on only off-main-thread (so in the preloader, for 'media' attributes as well as the sizes stuff)

I plan to turn it on for all MQ evaluation soon, since it has better spec compliance, and it'd be easier to add stuff to it (e.g. calc support for MQs, MQ L4).

MediaValues

Used by MediaQueryEvaluator

Cached variant - contains static values

Dynamic variants - refs a LocalFrame

MQ eval no longer depends on RenderStyle

Implements thread safe CSS length computation

SizesAttributeParser

Uses MediaQueryParser as a component

Since 'sizes' syntax is based on CSS, I've used the Tokenizer also for the sizes attribute parser.

SizesCalcParser

Calc syntax => tokens => reverse polish notation => length

Here again, I've used the Tokenizer to get the calc tokens.

Then I got rid of the parens by turning them into reverse Polish notation (Dijkstra's shunting yard algo)

And finally, calculate the output length

HTMLSrcsetParser

Parses the srcset attribute

'w' descriptors converted to DPR (using 'sizes')

Picks the best candidate to load

A stand alone parser for the srcset syntax. Used from both HTMLImageElement and HTMLPreloadScanner

Intrinsic sizing

Both the 'x' descriptor and the 'w' have an impact on the image's intrinsic size.

That means that when the image dimensions are not otherwise defined, the browser will use these values and the image dimensions from the image data itself to determine the display size of the image.

Moar intrinsic sizing

Stable state image loading

Required to avoid double download

Also required for img attribute setting

Worked on by Christian Biesinger (Thanks, Christian! :))

In the HTML spec it is stated that images should start their download only after reaching stable state.

That means that HTMLImageElement is able to know who its parent node is (when set by parser) and avoid a double download

When set by JS, it means that JS running was terminated so that the order in which attributes are set on img doesn't matter

That wasn't how things worked in Blink though

I poked around a little bit with it (added tests and such), and then Christian Biesinger took it upon himself to make sure the spec makes it easier to be compliant and then align Blink's behaviour to the spec.

Possible future improvements

http://earnthis.net/brian-terrills-100-film-favorites-22-back-to-the-future-part-ii/

Client Hints

• Proposed by Ilya Grigorik
• HTTP based content-negotiation solution
• Client sends hints/capabilities
• Server side logic decides on resource

It's a complementary solution that allows for server side image adaptation, on top or instead of client side one

Future doesn't look bright, since Mozilla are opposed to it

Responsive Image Container

• File format approach
• A "layer" per resolution
• Both resolution switching and art-direction

Resolution switching

336x635

106x200

211x400

336x635

Art-direction

770x512

200x200

400x288

770x512

Advantages

• Markup left untouched
• A single file per image
• Better for post-download dimensions changes

No markup

single file

can download diff

disadvantages

• Touches many layers. Will take time
• Decoding performance???
• Fetching mechanism network performance???

complicated

decoding perf?

Fetching mechanism

To sum it up

Thanks!

@yoavweiss on Twitter & GitHub

responsiveimages.org

Slides: yoavweiss.github.io/respimg-blinkon-presentation

Questions?