Frontend Services

Frontend Services

@kenstruys

- Hello, I'm Ken Struys and I'm an Engineering Manager at Yelp
- Today I'm going to talk about how we're scaling our engineering team Services
- more specifically services created for Frontend Web Application

Yelp's Mission

Connecting people with great local businesses

- Yelp is a app and site that connects people to get local businesses
- Making those connections on a global scale across multiple platforms requires
a large engineering team
- Being able to add additional engineers to the team is a requirement to
accomplish our mission

- Today we're going to talk about monolithic application
- Every company starts as one
- They sneak up very quickly as a problem
- My goal today is to help you recongize the symtoms
- Provide suggestions to help

- So, what are the biggest issues we've seen

Team Growth

- If you're company does well, engineering will grow
- Yelp grew and is still growing
- Our monolith has been a problem

Development

• Write Code/Tests
• Run Build/Tests
• Code Review
• Merge

- At yelp, all development is done in git feature branches

In those git branches engineers:
- Write code and tests for their code
- Run builds and regression tests
- Have their features and code reviewed by peers
- and finally,
- Merge their branch into a branch for deployment

Ship it

- We take a set of developer changes and deploy them together to production
- Until a feature branch ships, the development process doesn't matter
- When a feature ships, that's where engineers actually impact local business

- Engineers feel great every time they see their code reach production
- The higher the frequency you can release code, the more often an engineer will be happy

- Shipping often is so ingrained in our culture,
- we make sure every engineer gets something in production in their first week

- Unfortunately when shipping multiple branches together to production
- In some cases it's nessisary to revert the entire deploy branch
- We've observed it's become more difficult as we've grown

99%

Chance a Branch is Bug Free

- The reason it's become worse over time is really easy to understand
- Lets say you have great engineers, and the probability
- a single engineers branch is bug free is 99%
- Don't get me wrong, the engineers at Yelp as amazing,
- but even our best probably don't get to 99%
- I'm probably being generous

99%⋅99%=98%

2 Bug Free Branches

- When you take two engineers and put them into the same deployment
- The overall probability of success goes down by a small amount

99%30=74%

30 Bug Free Branches

- If you take 30 branches and deploy them
- The overall probability of success starts to get really low

99%n

n>500=0

- This is what's called an exponential decay function

- If you continue to grow, it just gets worse

- Monolith applications fall into this trap and
it goes from a minor annoyance to a serious problem

Monolithic Code Changes

- Another big reason monolith aren't great is,
- making large code migrations is awful for a single engineer
- They deal with merge conflicts
- and they don't ship very often

- If you've ever gone through a migration like bootstrap 2 to 3 in a million line codebase
- you know how awful this can be

- it can often be unclear what you should do.
- In many cases it requires coordination with many teams

yelp-main

• Yelp's monolith
• Mostly Python
• MVC Sites
• Offline batches

- We have a monolith at Yelp
- Mostly python code
- with a handful of different MVC Sites
- Sites for business owners, consumers like yourselves
- and administrative sites
- and scripts that run periodically called batches

yelp-main 2011

• 60 Engineers
• 2 scheduled daily deploys
• 5 - 8 changes per deploy

- When I first joined Yelp, the monoith worked well
- We had around 60 engineers
- We deployed to product twice a day, with 5-8 changes per deployment

yelp-main 2011-2013

• 60 → 200 Engineer
• More products, code, tests
• 3 scheduled daily deploys

- from 2011 to 2013 we grew a lot
- With more engineers, more products, more code and more tests
- More tests can also slow down a deployment if you don't build
- a sophisticated infrastructure to support all those tests
- We managed to get up to 3 deloys to support the growth

Deployments Get Scary

• 30+ changes per deploy
• 50,000 Tests, 200 Selenium Tests
• No one dev knows every error case
• Rollbacks start to become common

- That's around the time when deployments and code changes got scary
- We became more paranoid with tests
- No once can possible know how things could break
- If you were working on search and you're on ads,
- You might break something in their code and not know

- And rollbacks start to become more common

Services Might Help

- Looking at the rest of the industry
- people suggested services could help with the scaling issue

What is a Service?

• Provides a service to other services
• Often done via HTTP, Thrift, protobuff
• Example: The Facebook API

- So what is a service?
- Provides a service to other services
- Communication between services is done with HTTP
- Think something like the Facebook API. Yelp talks to Facebook via and API.

How to Service

• Usually a separate repository
• Own set of tests
• Own deployment
• Backwards compatible
• Share code via Libraries

- Services typically use a separate git repo
- Have a set of test to ensure it's working as expected
- Deploys independently from other services
- Backwards compatiable changes
- Share code between services with libraries

What is a Library?

• A collection of functions/classes/values
• Have their own build/tests
• Libraries can depend on other libraries
• Multiple services can depend on libraries
• Libraries are versioned
• Example: jQuery

- A library is similar to a service but it provides functions, classes, and values
- to services
- Similar to services they have their own set of tests
- A both services and libraries can depend on other libaries
- Libraries are versioned to allow people to use both old and new copies
- jQuery is a great example. You might use jQuery 1.9 and I use 1.8.

- Great, we know what services are
- And if you look at your monolith, you can probably break the code into a couple categories
of things to service
- For us we thought about Routing, Frontends, Backend/Business Logic and Storage

- Now What?...
- If you look this stuff up online
- you mostly find talks where people do this:
- "Well, had this monolith"

???

"It was embarsing"

- And... then we had services
- Wait, what? But how?

Yelp 2013

• Backend Services
• Monolith API
• Everything New → Service

- In 2013, we started with our backend
- If you go down this route, you'll most likely also want to make your monolith
have an API
- Everything new, we obviously a service

- This is where we were going
- Turns out pulling out the backend isn't easy

Backend Service Problems

Availability, Performance, Query Complexity

    def most_recent_review_for_user(user_id)

        business, review = DB.query('
            SELECT *
            FROM review
            JOIN user ON user.id=review.user_id
            JOIN business ON review.business_id=business.id
            WHERE user.id=${user_id}
            ORDER BY review.created
            LIMIT 1;
        ', user_id)

        return business, review

                    

You'll Need Frontend/Product Help

- You'll need help from Frontend Engineers, Product Managers and Designers
to decide how to handle availability

2014

Something Still Isn't Right...

- In 2014, we still felt like something wasn't quite right.

- Let's say we get to this world where the backend is well isolated
- The critical path from User to the Backend Still requires the monolith
- If we want to have use impact, you need a frontend

Frontend Services

HTML Rendering Services

- That's when we decided we need to make Frontend Services
- These services would return HTML instead of JSON
- Users would be able to go from the routing layer directly to a UI outside of the monlith
- And our product teams would be able to iterate extremely quickly on this well isolated
section of a site

Frontend Isn't Easy Either

• Very few talks about Frontend SOA
• Keep Product Consistency
• SOA Education

- Very few engineers talk about Frontend SOA (Service Orient Arch!)
- Unlike backend services, you can't just pick any technology to render JSON
- You'll want to make user transitions between the monolith and services seamless

Where did we start?

• Move fast, fail early
• A page as a unit
• Make it faster next time
• Leverage existing SOA

- We're probably going to make mistakes, make them really early
- We decided to move 1 page at a time from the monolith to a service
- Whatever we move, should make the next things faster to mvoe
- Don't reinvent the wheel, try to use the existing serivce stack

git://services/styleguide

- We decided to use our styleguide as our first page

yelp.com/styleguide

- If you haven't seen it before, check it out.
- It's basically like bootstrap on steroids specifically for Yelp

- It contains a lot of the simple things like different headers

- Also more complicated widgets like all the ways we show users on yelp.com
- Our Product managers, designers and engineers all use our styleguide for documentation
- and creating a common design language
- It also helps our designers to avoid reinventing a existing frontend pattern

git://services/styleguide

• No backend
• English only
• Low Impact
• Renders everything useful

There we a number of reasons to pull the styleguide first
- Absolutely no backend
- It's 1 of the only english only pages on yelp
- If we break it, it's not a disaster
- Renders most of the useful stuff
- Will make the next page fast on the frontend

How do we share code?

• git://yelp-main
• git://services/styleguide

- So we spun up this new service
- We needed a way to share code between the monolith and the new service
- Just like with backend services, libraries are going to be the answer for code sharing

PyPi + Bower

- For backend code we use PyPI
- For frontend code we use Bower
- We run internal versions of both of these package managers inside yelp

    > cat styleguide/bower.json
    {
        "name": "service-styleguide",
        "dependencies": {
            "jquery": "1.8.2",
            "yelp_styleguide": "0.10.3"
        }
    }
                    

- In every service, we include a file called bower.json
- In the case of the styleguide service, it's really simple
- the service is a simple wrapper providing documentation for our library called yelp_styleguide

    $ bower install
    ...
    bower jquery#1.8.2              install jquery#1.8.2
    bower yelp_styleguide#0.10.3    install yelp_styleguide#0.10.3
    ...

    $ tree bower_components/ -L 1
    bower_components/
    ├── jquery
    ├── yelp_styleguide
                    

- When you run bower install, bower downloads all of the packages
- and installs them into a directory called bower_components

yelp_styleguide

• SCSS (underscored)
• JavaScript
• Images/Icons
• Tests
• bower.json

    @import "bower_components/yelp_styleguide/assets/scss/grid_layout";
                    

- That library is now available inside the service code

Releasing Styleguide Service

• Trivial HTTP proxy in monolith
• Delete all code/tests in monolith

Service Tips

- A couple protips that we've found are important for both stability and developers sanity

Semantic Versioning

MAJOR.MINOR.PATCH

- Use semantic versioning on libraries
- The MAJOR version tells you if there's an incompatible API change
- The MINOR version tells you if there was functionality added but nothing backward incompatable
- The PATCH version tells you a bug fix was made

Example

- In the first example jump from one version to another should just fix problems
- The second example you'll have a new API but everything else should continue to work
- The third example you need to make sure you update your code to handle the changes

Semver in Bower

- You can make libraries and services have various restrictions on what versions they use
- For example in the first example we're saying I need jQuery, I don't care what version
- in the second example we're saying, anything from 1.8-1.9 is fine, any patches between are acceptable
- in the third that syntax says anything in the 1 release is fine, minors and patches are good
- in the last example we're being very explicit, we only work with the 2nd patch version of 1.9

Dep Management

Min Versions in Libraries

    > cat yelp_styleguide/bower.json
    {
        "name": "yelp_styleguide",
        "dependencies": {
            "jquery": ">=1.8",
        }
    }
                    

Build/Test Libraries Daily

On Failure:

Pin Deps in Services

    > cat styleguide/bower.json
    {
        "name": "service-styleguide",
        "dependencies": {
            "jquery": "1.8.2",
            "yelp_styleguide": "0.10.3"
        }
    }
                    

We Made Mistakes

Let's make a monolith!

30+ Pages Overnight

Thanks!

Questions?