Cropping images with <canvas>

Created by: Chad Smith

WELCOME TO PITTSBURGH!

My name is Chad Smith, and I am the software development manager for the fornt end group at GiftCards.com!

We developed an application that allows users to upload their photos and have them be printed onto a Giftcard. Yes, even you could have your face be on a VISA gift card.

The talk today will center around how my team and I reshaped our Flash application and used HTML5 to do all the work.

Damn it's hot

I want you to just relax, take a deep breath and enjoy the heat. We should be happy after the frigid winter right?

Who am I?

Introduce yourself

That's my twitter, go use that. I am on there a bit, when I remember.

Please copy down the slides and demo urls, they are also in the footer for the slides

What are we going to talk about?

• Where are phones going?
• Old School vs New School
• HTML5 Components
• Using AngularJS
• Challenges we faced
• Demo

Images are big

Data cost and speed

Average cost per roaming MB $5.02 to $13.98 in Europe.

3G upload average is 350-500Kbps.

Your point?

IT TAKES AT WORST 9 SECONDS AND COSTS THEM $39 TO UPLOAD!

Images are getting bigger, and the data is expensive. You as a developer MUST start to consider the impacts your choices make on your users.

Not only are you helping your users data plan but you also are giving users what they want. Immediate gratification using your app.

Before HTML5

Users' want immediate feedback! We had no feasible way prior to 2010 to do that. HTML5 APIs were not implemented until Chrome 7, FireFox 3.6, IE10 (yes that was in 2012), and Safari 6. Most companies who wanted to provide immediate feedback did it with Flash.

In 2014 we now have HTML5 implemented on all mainstream browsers. That gives us access to FileReader, Canvas, and many other tools.

HTML5 Implementation

HTML5 comes out, with FileReader, canvas, and XMLHttpRequest! We now have a real platform that can compete with Flash for file uploading.

AngularJS

• Multi-page router implementation
• Decoupled directives
• Testable code
• jqLite (it's jQuery Selectorish)
• Consistent framework for development

Our application had multiple steps to conceptually break apart our application into the following parts: Photo upload, Photo Crop, Gift Card information, Greeting Card

During these steps they use decoupled directives that call back to injected parent scope API methods from it's parent scope

These directives takes in configuration options for photo sizes, and other details that are specific to GiftCards

We also had testable directives that would intake photos and provide mocked elements that could then be used to test for jCrop and others. Just recently we have started scratching the surface for what we can do with end to end testing

Nate Peterson earlier today had a great presentation on testing.

FileReader Events

• FileReader.onloadend()
• FileReader.onerror()
• FileReader.onprogress()

FileReader has a couple of methods implemented that we should note. The most notable is onloadend, this event is fired when the reading operation is completed and there's either a success or failure.

FileReader is a read only instance, as soon as the async process is finished the data is gone.

FileReader has a method that will be triggered if there's an error reading the data. Method onerror could happen if the user is attempting to read a file from a cloud drive for instance.

onProgress allows you to respond to a user's image being uploaded. Though it's typically very fast if you have a use-case where the images aren't going to be small (i.e: GIS data) then using progress to let them know it's going is a good thing

FileReader Methods

• readAsDataURL(file)
• abort()
• readAsBinaryString() -- WARNING

FileReader's most notable method that's implemented is readAsDataURL. This method takes the image and will read it into a data URL. That url contains the correct prefixes needed to display onto a webpage.

Abort can be used when a user decides to change their mind. Example usage of this is if you provide a progress bar from the event and the user decides "Meh, no. I don't want that uploaded" they can cancel the reading of the file.

FileReader has gone through a few revisions though! You need to know that anytime you see readAsBinaryString, it's not officially supported. It was implemented by a lot of browser, to then be removed from the W3C specification in 2012.

FileReader Example

elem.find('.img-upload-file-input').on('change', function (evt) {
    var fileReader = new FileReader();
    var file = evt.target.files[0];

    fileReader.onloadend = function (e) {
        var img = new Image();
        img.id = 'pic';
        img.src = e.target.result;

        angular.element('body').append(sourceImage);
    };

    fileReader.readAsDataURL(file);
});

Flipped images are bad

Phones rotate images automatically

How many of you have ever uploaded an image to a website only to have it look like this? That's due to camera phones taking photos. Why does this matter? Take out your phone, look at the button on it. Where you take the photo matters. Flip your phone upside down and take a photo, it looks right side up. It's not. The image itself is upside down and it's relying on the UI to change the rotation.

Metadata to the rescue

Don't feel ... DOWN about it. get it? 311 reference? Oh whatever!

Don't worry! Most (if not all) phones track the position the phone was in when it took the photo. If that's not there then the devices don't know how to rotate it "right side up".

Using EXIF.js

<script src="js/exif/exif.js"></script>

scope.getFileOrientation = function (files) {
    EXIF.getData(files[0], function () {
        var orientation = EXIF.getTag(this, "Orientation");
        if (typeof(orientation) !== 'undefined') {
            scope.rotateAngle = orientation;
        }
    });
};

You are thinking ... that's great Chad. How do we get the orientation? More importantly what can we do to make it right? With some minor code you can flip the images right side up.

Small images aren't an issue

Allow users to upload large or small images!

scope.createScaledImage = function (image) {
    // Scale the image
    var canvas = document.createElement('canvas');
    canvas.width = (image.width > scope.minSize) ? image.width : scope.minSize;
    canvas.height = (image.height > scope.minSize) ? image.height : scope.minSize;

    var ctx = canvas.getContext('2d');
    ctx.fillStyle = 'rgb(255,255,255)';
    ctx.fillRect(0, 0, canvas.width, canvas.height);
    ctx.drawImage(image,
        (canvas.width / 2 - image.width / 2),
        (canvas.height / 2 - image.height / 2)
    );

    return canvas.toDataURL();
};

We didn't like having to restrict our users from being able to upload a certain size of image!

We instead figured out with a little math and canvas we can make their experience so much nicer.

Using JCrop with Canvas

Consistency between browsers

Achieved our desired layout

Used jCrop as our crop because it allowed us to take the coordinates and clone from one canvas to another. Additionally if users were on their IE8 machines at work, and then went to their Chrome machines at home it would look exactly the same.

Used jCrop also because it enabled our use case to put a custom photo on the background layer to show what the end result would look like when it was printed.

Copy between <canvas>

function imageCallback(c) {
    var exportCanvas = document.getElementById('exportCanvas').getContext('2d');
    exportCanvas.fillStyle = 'rgb(255,255,255)';
    exportCanvas.fillRect(0, 0, 1050, 672);
    var w = c.x2 - c.x;
    var h = c.y2 - c.y;
    exportCanvas.drawImage(sourceCanvas, c.x, c.y, w, h, 0, 0, 1050, 672);
}

AngularJS $http service

• Core Angular service
• Facilitates all communication
• Promise based

Angular has a great way of handling everything one would want to do with any self respectable HTTP class. When using HTTP you have mock objects available for testing, and it just works.

$http example

.controller('FinishCtrl', function ($scope, $location, $http) {
    var source = angular.element('#crop-image-1').get(0);
    $http({
        url: 'http://www.yourdomain.com/uploadImage',
        method: "POST",
        data: {'image':source.src},
        headers: {'Content-Type': 'application/x-www-form-urlencoded'}
    }).success(function (data, status, headers, config) {
        $scope.success = data;
    }).error(function (data, status, headers, config) {
        $scope.status = status;
    });
}
            

This example shows how we can use AngularJS to send the base64 encoded image through to the server.

First Release

Android 2.3

Built with old specifications

Export as JPEG? lol. No.

Android 2.3 was built with some very early HTML5 specifications. They didn't take into account that the canvas toDataURL() was able to have a JPEG. It always defaulted to PNG

This browser also didn't transfer from one canvas to another consistently.

We ended up putting a specific AGENT check to not use the canvas crop for Android 2.3

Cellular based iPhone4S+

Resource limitation and image subsampling

The next problem that we came across is that our great customer service representatives were noticing a lot of white images.

It took a lot of testing and working with our customers to isolate it was when they were using an iPhone 4S+, and not on a WIFI network and taking the photo in landscape mode.

Apple wrote a lengthy article explaining it's origins and it's to protect the user from uploading too large of images across the wire, and since it goes to a canvas it's determined as "across the wire"

ImageFile MegaPixel

• Detects and corrects subsampling
• Enabled canvas rotation
• Small library

Knowing the problem with subsampling, we found a class that solved this issue. We found that it takes the canvas, detects subsampling, and then will stretch it back out.

The class also had rotation built into it. Given an orientation from EXIF it would rotate the photo correctly for you

Ability to export the image in a number of ways, either to another canvas or an image

How is subsampling detected?

function detectSubsampling(img) {
    var iw = img.naturalWidth, ih = img.naturalHeight;
    // subsampling may happen over megapixel image
    if (iw * ih > 1024 * 1024) {
        var canvas = document.createElement('canvas');
        canvas.width = canvas.height = 1;
        var ctx = canvas.getContext('2d');
        ctx.drawImage(img, -iw + 1, 0);
        // subsampled image becomes half smaller in rendering size.
        // check alpha channel value to confirm image is
        // covering edge pixel or not.
        // if alpha value is 0 image is not covering, hence subsampled.
        return ctx.getImageData(0, 0, 1, 1).data[3] === 0;
    } else {
        return false;
    }
}

Demo

Let's see this thing in action.

Questions?

THANK YOU!

Chad Smith Software Development Manager - Front-end chad.smith@giftcards.com @hppycoder