Done well, Operations enhances the safety, contentment, knowledge and freedom of both the authors and users of the system.

• Design is fundamental
• Each choice you make needs to make life better for the humans involved
• That also leads to better business outcomes, as we'll learn later
• Ultimately, the most scalable, fastest systems are also the ones that are best for the humans invovled, most of the time

• Imagine you were early days at twitter
• The system wasn't human safety critical, in your mind
• Until it became a source of human saftey and communication during countless revolutions

• Happiness is fleeting
• If you are in trouble, contentment helps you make better decisions
• Think about a brutal on call week - if the systems that support you are good, you survive

• The right knowledge, at the right time
• Think about PaaS - its awesome you just git push
• Until you are Rap Genius and heroku changes the router and everything sucks and you don't know why
• Which doesn't make PaaS awful - at a different level of criticality, who cares?

• The Big Web got this right
• Empower individuals to work as they see fit
• Trust them to do the right things
• Build systems that increase the trust needed to allow more freedom

Safety

Contentment

Knowledge

Freedom

• We will come back to these throughout

Being good at Operations

Availability

Efficiency

• Availability: Is the system down? Bring it back up.
• Efficiency:Make the effort required to do work easier.
• The work here is building and maintaining computers, networks, and applications
• So efficiently doing that covers damn near everything

Focus on Availability

Efficiency Follows

• Availability shows where you need to be most efficient now
• It's a virtuous cycle

Availability is everybody's problem

• There is no team that owns availability - other than the company itself
• The problems are too big

• The difference in magnitude matters - days, hours, half hours, minutes, seconds
• To achieve higher levels, everything has to get more precise
• Know your target, and communicate it
• It probably isn't five nines

The M's

• Mean Time To Failure (MTTF) ↑ The average time there is correct behavior
• Mean Time To Diagnose (MTTD) ↓ The average time it takes to diagnose the problem
• Mean Time To Repair (MTTR) ↓ The average time it takes to fix a problem
• Mean Time Between Failures (MTBF) ↑ The average time between failures

• We want to decrease MTTD and MTTR
• And increase MTTF and MTBF

• All systems fail
• Fear of failure is the greatest killer of availability

Slow and ponderous

Fast and nimble

• Online banking is a huge thing for consumer banks
• I met with one that has 5 9's of availability
• They achieved this through changing the website once ever 6 months
• After a torture chamber of hate and pain
• They were not better at diagnose and repair - they were good at MTBF, and lucky
• Contrast that with a more nimble org, who might have more frequent outages (say scheduled maintenance once a week)
• But the system improves week over week
• Raise your hand which one you want!
• It's safer, increases human contentment, is easier to reason about, and frees people up

• You can't fix what you can't see
• Metrics resolution has direct impact on MTTD

• One binary metric - can your users use your stuff
• Money is often a trailing indicator of deeper systemic problems that are hard to see
• I helped run an ad network back in the day, and the hour-by-hour money graph was the fastest way to see if we were letting people run over cap
• Money graph also helps you justify other activity!

Diagnose

Graphing, Trends and Analysis

Use graphs to understand normal behavior.

Graphs taken from Theo Schlossnagle and OmniTI

• Lets say this is puppy.com - the prime source for puppy news
• Nice, easy content day - 70% utilization, smooth peaks and valleys

• The Doge dog beats up the taco bell chihuahua outside the most posh dog park in LA
• Puppy.com has the exclusive video

Diagnose

Graphing, Trends and Analysis

Use graphs to understand abnormal behavior.

Graphs taken from Theo Schlossnagle and OmniTI

• The new york times picks it up, and adds long exposure traffic
• Digg shows up, and it goes to 11
• Happens in 60 seconds!

Auto-Scaling Will Not Save You

• Either you design for this load, or you fail to meet the expectations
• The right answer here is serve puppy.com from behind fast.ly :)

Capacity Planning

• Do this on a regular cadence - monthly, etc.
• Show your R-squared - think of it as a confidence number
• This could be any metric that matters for your system
• This is the number one source of trivially preventable outages

• There is nothing more disrespectful than waking someone up for shit they can't fix
• It's happening.. its happening... again

Repair

Incident Response

• Observe: whats going on
• Orient: put whats going on in context of waht you know about the system, people, and dynamics
• Decide: what to do next
• Act: take action
• Originally for fighter pilots to get inside the heads of the enemy
• A faster loop means success in combat
• This is the same pattern for responding to operations availability issues

• In fighter jets, knowing typical behavior, jets, and culture was crucial
• Rob Pike and Ken Thompson working on a visual language
• Rob typed faster, so he was at the keyboard
• Rob attacked bugs, Ken thought about it
• Ken was orienting better
• Unlike a fighter jet, he had time :)

Repair

Incident Command

The First Responder is the default Incident Commander

There is only ONE Incident Commander.

This isn't always true in real Incident Command, but go with it.

• When it gets bigger than one person can handle, we flip to this
• Knowing we have a Process, and command structure makes it easier to OODA
• And faster loops means faster resolution

• This should be the IC at the end of the incident

Learn

How to run a Post Mortem

• We hold post mortems to learn and improve, not to blame and punish
• Puppys.com went down when Digg linked to the Doge/Chihuaua story
• Story gets posted at 8am PST, NYT picks it up at 8:15am PST, Digg posts at 8:30am PST
• Site goes down at 8:30am, alert at 8:31am, diagnosed at 8:50am, more capacity launched on ec2 at 8:55am, online and resolved at 9:00am PST
• The traffic load overwhelmed mpm worker apache configuration, and exhausted capacity
• People could not watch the doge dog crush the chihuaha, and click ads
• Launched more capacity. Long term remediation is to move static content to a CDN
• We investigated a denial of service and backend database issues before we looked at traffic graphs. Add passive alert on traffic.

Prioritize the outcomes

• The process works because you prioritize the outcomes
• Our remediation steps are the efficiency improvements you want
• If you fail to act, or do other stuff, you're wasting the opportunity

People

Process

Technology

• 3 areas for efficiency, in order or most potential for gains
• Think about Puppy's dot com - if we didn't have the right people, if we didn't have a process for incidents, if we didn't have post mortems, the technology fixes wouldn't make a dent long term

• What is the mission?
• How does your organization intend to fulfill it?
• How do you contribute?
• What are the stakes?
• Knowing your purpose enables you to put decisions in context
• The more context you have, the better your decision will be
• Like a very long OODA loop

• Trust is crucial to effective operations
• Knowing people is crucial to trusting them
• Set up lunch dates
• Talk about your lives
• THIS IS WHERE DEVOPS COMES FROM
• John Allspaw and Paul Hammond are friends

• The way we do our work informs our lives
• Having good lives improves the quality of our work in every dimension
• We are blessed to be the architects of our environment
• Lets back Thay up with data

People

Engaged Workers Rule

• Gallup has been running this study since the 90s
• They have proven the impact engaged workers have is causul
• What other single thing could you possibly do that has a 22% impact on profitability?
• 21% impact on productivity!
• 65% less turnover!
• Or a 41% impact on defects! Happy people care about their work more
• It's the most critical operations efficiency task

• Repetition, Repetition, Repetition
• Training people is like training cats - you gotta be on that

• Pick someone out, insult them gently, then compliment them
• Point out this is what they will remember from this talk, forever

What you can do

• Don't be an Asshole, and fire or shun those who are
• Set clear expectations for others
• Praise people
• Make friends with, and care about your co-workers
• Listen to each other
• Take pride in your work

Process

The way we work is critical to our outcomes

Original Image

Kaizen

Small improvements

Evaluate a process, make it better.

Try using the scientific method:

Kaizen

Anyone can do it

• Continuous Delivery is a good example
• If you are a big, waterfall org with manual testing
• Incrementally moving to CD is going to fail
• You need to blow up the way you work, learn how that feels, and kaizen your way to happiness
• A house built on sand and all that

Original Image

Technology

Systems Design

Understand the requirements

• Big retailers web division, wanted to automate, I wanted to sell software
• Asked how they felt about Cd, said they weren't CD people
• I was like: Me neither! ;)
• They told me their design, said "then we come together and make it work"
• We rebuilt it in that room, much better - not real requirements

Rolling Upgrade

• Traditional web servers behind a load balancer
• Upgrade servers one at a time

Naive way

• This is what you would do if you wrote the steps down!
• And it's whats going to happen in any case
• But linearly implementing these as a script - whoa doggies
• 600 configuration changes to the load balancer!

Autonomous Actors

Promises

Each Autonomous Actor promises to behave a certain way.

Other Actors can verify those promises.

Identify Autonomous Actors

Load Balancers

Promises to route traffic to working app servers

Application Servers

Promises to serve application traffic and publish status

Better way

• Add a service that is smart about the apps status to each server
• Monitor that service with the load balancer
• Upgrade process manages that services response
• Load balancer just blindly routes traffic
• All the questions from the neive implementation can be answered by improvements to the status endpoint

The better solution has fewer interactions.

But it has more pieces.

• We reduced the degree of difficulty in the process
• Increased the number of moving parts
• Safety: Resilient against many more failure modes
• Knowledge Far easier to reason about during Orient in the OODA loop
• Freedom: Pattern adapts to different values of "available" based on service needs
• Contentment Safer>, easier to reason about, and more flexible - that makes everyone content