Responsive Images

Yoav Weiss

Opera - June 2014

yoavweiss.github.io/respimg-opera-workshop

Who????

• Srcset & Picture implementation

Working on responsive images in my spare time for the last 2 years

A member of the RICG

A Blink & WebKit committer

Prototyped picture in WebKit. Implemented srcset in Blink. Implementing picture in Blink and WebKit.

I also been working on front end optimization server side solutions for the last 15 years, and am on a personal vendetta on image bloat on the Web

What is Responsive images?

Efficiently load properly dimensioned images that fit the page's design

Who knows what the responsive images problem is?

<history>

Started out as

It started out (or at least got people's attention) as a quality issue.

The solution to that was simple - just send users the big images

"But that's easy"

"Just send the largest possible image"

"And let the browser resize it"

EAZZZZZZY!!!

Well...

This led to BLOAT!!!

Mobile experience became better, but slower

"a responsive site" became a synonym of "a slow site"

72% Serve same resources

guypo.com

Serving the same resources to mobile and desktop hurts performance

On mobile the images are much smaller and some of them are not even displayed

Which resources?

Images - over 63%

httparchive.org

So quality issues became

data plan abuse

And...

Large images?

People demanded a solution

Turned to the mailing lists

Proposals!

Moar proposals!

And the RICG was born

<picture> proposal

The RICG discussed the issue for a few months and came up with the picture element proposal

srcset proposal

At the same time, Ted O'Connor from Apple proposed the srcset attribute for simple DPR selection

Hixie added it to the spec over the weekend without much public discussion, then tried to retrofit the other use cases into it

A lot of mailing list flame wars. Things got tense.

Picture *and* srcset

Florian Rivoal (from Opera?) proposed to join forces. the (original) srcset syntax was adopted into the picture syntax, since they covered different use cases

Browsers weren't convinced

18 (EIGHTEEN!!!) months later

I prototyped picture in WebKit. Marcos and I Talked at TPAC. Marcos speced the proposal. Use cases doc. FPWD

But nothing substantial happened

Paris!

Marcos and I organized a meetup in Mozilla paris. BruceL stood us up.

Nothing got decided, but it laid the foundations for things to come

Src-N

Then after a long while of little progress

SrcN was proposed by TabAtkins and John Mellor, after a couple of meetups we had with John.

It resolved all the use case, in a single element, but got resistence from some browser people.

Then some more

There was a hecktic time at the end of last year, where proposals were coming in twice a day. But now this is all behind us.

And MOAR

Back to picture

The following an IRC chat I had with Simon Pieters, we figured out a way to gather up all the good pieces

from Src-N and wrap it inside something that looks really like the original picture syntax,

only significantly easier to spec, implement and maintain.

TabAtkins then revived the picture spec (basically rewrote most of it), and we've been working on it with Tab and

Simon ever since.

Browsers were like

Browsers liked it way better

But Blink was like

Blink still resisted on implementation grounds, basically missing infrastructure.

So I got busy

Basically, I started tackling the infrastructure issues that were the cause for Blink's resistence, in order to defuse it

And soon became

And realized that if I'd keep doing that in evenings, it's gonna take a looooong while

So, crowdfunding to the rescue

A group effort

XXXXXXX - Talk about Wilto and Geri and stuff regarding the IndieGoGo thing

The RICG & whatwg/blink folks are working together on the spec, to make it as awesome as possible. Mozilla is also heavily involved

The implementation in Blink and Gecko is almost done.

It went well

And Opera chipped in :)

And patches got landing

~60 patches later

Blink work is done

WebKit is underway

Hopefully shipping in Chrome 38

</history>

<syntax>

Picture 2.0™

Let's take a closer look at each one of the parts combining the latest spec.

The srcset 'x' part

The oldest part of the spec, goes back to the WHAWG 2012 days

Use case - "Retina images"

Load hi-res images

on hi-res devices

The syntax

<img src="1x.jpg" srcset="2x.jpg 2x, 3x.jpg 3x" 
     alt="The president.">

The <picture> part

Use case - Art direction

Use case #2 - MIME type fallback

Today's Web have become fragmented when it comes to image formats.

We may not like it, but that's the truth :(

Picture will enable us to have client-side mime type fallback if we're using browser specific image formats, just like all other resources which they're type may or may not be supported (e.g. video, fonts).

Note that it doesn't mean that you can't do server-side content negotiation using the accept header, if that's your thing

But client-side may be more accessible to Web developers that can't mess around with their backend, and it has caching advantages.

The syntax

<picture>
     <source media="(min-width: 45em)" srcset="large.jpg">
     <source media="(min-width: 18em)" srcset="medium.jpg">
     <img src="small.jpg" alt="The president.">
</picture>

Here you can see the syntax parts that make up the "picture" part of the spec, namely the picture element and its source children.

Each one of the source children, as well as the img child define an image that fits into a certain responsive breakpoint.

The sizes + srcset 'w' descriptor part

http://ericportis.com/posts/2014/srcset-sizes/

The third part is IMO, the most exciting part.

This is the major innovation that the original src-N proposal brought

It's the most exciting since it applies almost everywhere, once you look at it.

It can be used to save bandwidth in both responsive and adaptive designs and even if hi res displays are not involved.

use case - variable width images

The syntax

<img src="otherpic.jpg" alt="The president giving an award."
     sizes="(max-width: 30em) 100vw, 
         (max-width: 50em) 50vw, 
         calc(33vw - 100px)"
     srcset="pic100.jpg 100w, pic200.jpg 200w, 
             pic400.jpg 400w, pic800.jpg 800w, pic1600.jpg 1600w, 
             pic3200.jpg 3200w">

Didn't add an example,so go over this example in length!

</syntax>

<support>

blog.thestudio4.co.uk

Blink - Chrome & Opera

I started out by implementing srcset's x descriptor. It shipped in Chrome/Opera stable a few months back.

sizes and the 'w' descriptor in srcset are done. I'll probably aim to ship it in M37

picture work has started and is moving along nicely - Christian is helping with stable state

Firefox

IE

They're showing up on IRC and mailing lists, asking good questions.

WebKit

Implemented & shipped srcset's 'x' descriptor

Willing to accept patches behind a compile flag

I'm currently working on patches

</support>

<internals>

MediaQueryTokenizer

I started by taking Eric's tokenizer and adapting it to my needs.

I've deleted the parts that I didn't need and implemented the missing parts that I did

MediaQueryParser

Then I've build a parser for MediaQuery syntax that is based on the tokenizer's output

It's currently turned on only off-main-thread (so in the preloader, for 'media' attributes as well as the sizes stuff)

I plan to turn it on for all MQ evaluation soon, since it has better spec compliance, and it'd be easier to add stuff to it (e.g. calc support for MQs, MQ L4).

MediaValues

Used by MediaQueryEvaluator

Cached variant - contains static values

Dynamic variants - refs a LocalFrame

MQ eval no longer depends on RenderStyle

Implements thread safe CSS length computation

Intrinsic sizing

Both the 'x' descriptor and the 'w' have an impact on the image's intrinsic size.

That means that when the image dimensions are not otherwise defined, the browser will use these values and the image dimensions from the image data itself to determine the display size of the image.

Async image loading

Required to avoid double download

Also required for img attribute setting

Worked on by Christian Biesinger (Thanks, Christian! :))

In the HTML spec it is stated that images should start their download only after reaching stable state.

That means that HTMLImageElement is able to know who its parent node is (when set by parser) and avoid a double download

When set by JS, it means that JS running was terminated so that the order in which attributes are set on img doesn't matter

That wasn't how things worked in Blink though

I poked around a little bit with it (added tests and such), and then Christian Biesinger took it upon himself to make sure the spec makes it easier to be compliant and then align Blink's behaviour to the spec.

Future improvements?

http://earnthis.net/brian-terrills-100-film-favorites-22-back-to-the-future-part-ii/

Smarter resource selection

Avoid loading lower quality when high quality is available

Display cached/dataURI images while waiting for the real one

User preference, bandwidth considerations ...

Content embedder selection policy

The resource selection policy in the spec is described as UA specific

One could argue that the control of that should be given to the embedding layer

</internals>

<advice>

To picturefill or

OTOH, picture has a good fallback mechanism built in. Old browsers get the 1x src image

Source order matters

<picture>
     <source media="(min-width: 45em)" srcset="large.jpg">
     <source media="(min-width: 18em)" srcset="medium.jpg">
     <img src="small.jpg" alt="The president.">
</picture>

<picture>
     <source media="(min-width: 18em)" srcset="medium.jpg">
     <source media="(min-width: 45em)" srcset="large.jpg">
     <img src="small.jpg" alt="The president.">
</picture>

Source-size order matters

sizes="(max-width: 30em) 100vw, 
     (max-width: 50em) 50vw, 
     calc(33vw - 100px)"

sizes="(max-width: 50em) 50vw,
     (max-width: 30em) 100vw, 
     calc(33vw - 100px)"

'sizes' default value

<img src="otherpic.jpg" alt="The president giving an award."
     srcset="pic100.jpg 100w, pic200.jpg 200w">

<img src="otherpic.jpg" sizes="100vw"
     alt="The president giving an award."
     srcset="pic100.jpg 100w, pic200.jpg 200w">

Intrinsic sizing

Both the 'x' descriptor and the 'w' have an impact on the image's intrinsic size.

That means that when the image dimensions are not otherwise defined, the browser will use these values and the image dimensions from the image data itself to determine the display size of the image.

Moar intrinsic sizing

@viewport should be inlined!

<style>
    @viewport {
        width: auto;
    }
</style>

src cannot be selected when 'w' is present

<img src="otherpic.jpg" alt="The president giving an award."
     srcset="pic100.jpg 100w, pic200.jpg 200w">

Old 'w' syntax

If you're writing old srcset syntax, you're stating the viewport width, but it's being interpreted as the image's width

Since you didn't add sizes, the browser will assume 100vw.

That means that the resource you think should be picked will be picked (YAY!)

But its intrinsic sizing will be WAY off.

Styling

"Picture is a magical span, nothing more"

- Tab Atkins

As far as styling goes, all the style that should impact the image itself, should still go on the img element.

Picture is simply a wrapper around that img, which as far as styling goes, behaves like a span.

Future compat

Feature detection FTW!

HTMLImageElement.sizes & HTMLPictureElement

The syntax may be extended in the future (e.g. with 'h' descriptors for height-constrained images).

Make sure your polyfill uses feature detection, and backs off when native support is detected.

Which resource was picked?

currentSrc to the rescue

</advice>

<wwic>

You will encounter blog posts that think that this is a huge mistake and their unimplementable idea is the best way forward

Custom MQs to the rescue

Content images need to be in content

CSS override is not really implementable

Not a respimg specific problem

URLs being verbose is a platform wide issue

Server side templates and gzip are doing a good job mitigating that

</wwic>

The path not (yet) taken

Client Hints

Content negotiation based

It's a complementary solution that allows for server side image adaptation, on top or instead of client side one

Future doesn't look bright, since Mozilla are opposed to it

Request headers

CH-DPR

CH-RW

Implicitly relies on src-N's viewport-url

Response headers

DPR

Server confirmation header

Used for intrinsic image sizing

Opt-in only

Not sent on initial HTML request

Responsive Image Container

• File format approach
• A "layer" per resolution
• Both resolution switching and art-direction

Resolution switching

336x635

106x200

211x400

336x635

Art-direction

770x512

200x200

400x288

770x512

Advantages

• Markup left untouched
• A single file per image
• Better for post-download dimensions changes

No markup

single file

can download diff

disadvantages

• Touches many layers. Will take time
• Decoding performance???
• Fetching mechanism network performance???

complicated

decoding perf?

Fetching mechanism

To sum it up

Thanks!

@yoavweiss on Twitter & GitHub

responsiveimages.org

Slides: yoavweiss.github.io/respimg-opera-workshop

Questions?