Devops
- Devops comes from the lived experience of the practitioner
- Making it, in all cases, unique to the practitioner
- We had a lot of overlapping experiences though
- Relationship to failure
- High trust, high interaction relationships with each other
- A penchant for automation
- A tendency towards self reliance
- Obsession with metrics and graphs
- A focus on revenue and availability
Devops and the struggle for definition
- I think the struggle to define devops comes from this nature
- You can look at an individual, and see the 'school' they came from
- If you look at the 10 people I worked with longest, they way they 'do devops' is very, very similar to mine
- Its not surprise - our experiences are very similar
- The further you go from there, the more small differences there are
- In general, the macro trend holds - you can see us all under one umbrella, with many small differences
Gōngfu (功夫)
- 功 (gōng) meaning "work", "achievement", or "merit"
- 夫 (fū) often treated as being a word for "man"
The excellence achieved through long practice in one's skills
Source
Kung fu is like that
- Originally, it did not mean just to practice martial arts
- It referred to the process of ones training - the refinement of body and mind, the learning and perfection of your skills
Wushu (武术)
Basics
Forms
Application
Many styles of Wushu
- Hundreds of different schools
- They vary widely on what they emphasize
- But the broad methods are the same
- Basics - the fundamental internal/external foundation
- Forms - a series of predetermined behaviors that can combine to a continuous way of acting
- Applications - using your Kung fu in the field, and adapting the basics and forms to your situational awareness
Source - http://en.wikipedia.org/wiki/Kung_fu_%28term%29
- When you see someone doing it, and you are broadly ignorant, you know its different than boxing
- When you are someone who knows it well, you can tell the differences right away
- Its because the basics and forms are different
- Both are obviously fighters, but only one is Wushu
- You can have strong kung fu at both boxing and wushu
- Sound familiar?
Devops Kung fu
- A deep well of practical philosophy
- Broad experience, unique to each individual
- Recognizable by the basics and forms
- Practically applied to the situations in your life
- There is a deep well of practical philosophy
- There is a broad base of experience, unique to each individual
- We are recognizably similar
- There is a way to teach people how to do it
- It starts with the basics, and everyone must master them
- It moves to Forms - predetermined behaviors that can be combined into continuous sets of actions
- It ends in Application - actually doing your Kung fu in the field, using what you have learned
- TMTOWTDI is true and honest and important
- However - if you don't know how to do it, you could probably skip the 15+ year journey of difficulty
- And practice the basics, learn the forms, and apply them in your life
We can and often do argue about what it means
- Those arguments might change what overall makes a form
- But they don't change its nature
- We have a common language and palette for expression of our ideas
DevOps is reinventing how we run our businesses
Who practices it?
Everyone
- The application will be different, but the principles hold
- We are not generalists - we are well connected specialists
- CEOs and Executives
- Managers (Cyclops)
- Systems Administrators (Beast)
- Software Developers (Storm)
- Network Administrators (Kitty Pryde)
- Security Professionals (Wolverine)
- Sales Reps (Cannonball)
- Product Managers (Forge)
- All of these roles (and more) should practice our Kung fu
Chef Style DevOps Kung fu
This is my Devops Kung fu
- I call it "Chef Style"
- It comes from my experience
- As a systems administrator
- As a software developer
- As an entrepreneur
- As an executive
- As a human being dedicating my adult life to the internet
- As a father
None of the below endorse this talk, or Chef Style DevOps Kung fu
Mark Burgess
Johnny Cash
John Allspaw
Sammy Hagar
Nathan Haneysmith
Ronnie James Dio
Tom Thomas
Jesse Leach
Christopher Brown
Ben Rockwood
Jesse Robbins
Jez Humble
Barry Crist
Thich Naht Hanh
Mitch Hill
Larry Wall
Paul Edelhertz
Lao Tzu
Soo Choi
Jennifer Dumas
Alex Ethier
Yukihiro Matsumoto
Jeff Hackert
Theo Schlossnagle
This list is not complete, or in order. I ran out of room.
- It draws on many important influences
Foundational Document
- On Github: http://github.com/chef/devops-kungfu
- Join by PR
- We work together to develop our principles, forms, and applications
- Start your own school by forking, removing the names, and changing the content
- This talk is my foundational document of that Kung fu
- Once you have watched it, you can join my school
- We can develop the basics, our forms, and help on application
- Talib Kweli explained to me that artists amplify the movement they see in the world
- Marvin Gaye didn't write What's Goin On and
- then
- the movement happened
- The movement happened, and Marvin had to respond to, and amplify it, with his art
- That is my intent here - I will use my platform to raise up the voices of those who are doing the work
- And we can then leverage it as a platform to help each other
- And to further our aims of having our workplace be a better place
Elements of DevOps
Principles
(Universal)
Forms
(Shared)
Application
(Unique)
- The principles we aspire to (How you recognize DevOps when you see it)
- Wushu calls this the 'basics', but it involves a lot of pushups, and there isn't a lot of pushups in devops
- This includes a definition of DevOps
- The forms we use - the actual things we practice, that try and re-enforce and bolster our principles
- The application - the use of those forms in our actual situation
- These three things are the evolutionary arc of your practice - in the beginning, you learn from others
- As you grow, you will use the forms others have used to learn about the deeper meaning and philosophy
- The more you apply them in practice, the less reliant on the forms you will be - it all just becomes the natural flow
- Leading you, and anyone else, to be where those of us with already strong Kung Fu are - uniquely suited to our task
- (It's not about whether you are good at it - it's about the soul of our kung fu)
Principles
Universally held beliefs; the foundation of all DevOps styles
DevOps
A cultural and professional movement, focused on how we build and operate high velocity organizations, born from the experiences of its practitioners.
- Focused on how we build and operate - it encompasses the details of how we build things, and the details of how we operate them
- High velocity organizations - because we aren't very good at operating slow moving organizations; we were built to go fast
- Born from the experiences of it's practitioners - many of whom were the web innovators - because they were the first ones that had to go fast
- 'Do not be timid about the scope or the influence - we are re-inventing the way we run businesses"
Cultural and professional
-
Cultural like Hip Hop, Heavy Metal, or Otaku
-
Professional like an MC, Lead Guitarist, or Animator
DevOps: A cultural and professional movement, focused on how we build and operate high velocity organizations, born from the experiences of its practitioners.
- Focused on how we build and operate - it encompasses the details of how we build things, and the details of how we operate them
- High velocity organizations - because we aren't very good at operating slow moving organizations; we were built to go fast
- Born from the experiences of it's practitioners - many of whom were the web innovators - because they were the first ones that had to go fast
- 'Do not be timid about the scope or the influence - we are re-inventing the way we run businesses"
Focused on how we build and operate
- Encompasses the details of how we build things
- And how we operate them over time
DevOps: A cultural and professional movement, focused on how we build and operate high velocity organizations, born from the experiences of its practitioners.
- High velocity organizations - because we aren't very good at operating slow moving organizations; we were built to go fast
- Born from the experiences of it's practitioners - many of whom were the web innovators - because they were the first ones that had to go fast
- 'Do not be timid about the scope or the influence - we are re-inventing the way we run businesses"
High velocity organizations
- Our experiences were in building high velocity organizations
- The same principles applied to low velocity organizations bring instability
DevOps: A cultural and professional movement, focused on how we build and operate high velocity organizations, born from the experiences of its practitioners.
- Born from the experiences of it's practitioners - many of whom were the web innovators - because they were the first ones that had to go fast
- 'Do not be timid about the scope or the influence - we are re-inventing the way we run businesses"
Born from the experiences of its practitioners
- Most of whom were web innovators
- The first explorers of DevOps
DevOps: A cultural and professional movement, focused on how we build and operate high velocity organizations, born from the experiences of its practitioners.
- 'Do not be timid about the scope or the influence - we are re-inventing the way we run businesses"
Design for the safety, contentment, knowledge and freedom of both your peers and your customers.
- Design is fundamental
- Each choice you make needs to make life better for the humans involved
- That also leads to better business outcomes, as we'll learn later
- Ultimately, the most scalable, fastest systems are also the ones that are best for the humans involved, most of the time
Safety
- Human safety
- Information safety
- The ability for individuals to act without fear of unintended consequences
Safety is a slider – different systems have different thresholds
- Imagine you were early days at twitter
- The system wasn't human safety critical, in your mind
- Until it became a source of human safety and communication during countless revolutions
Contentment
Contentment is about being satisfied with what you have.
Happiness is not a goal – it’s a by-product of a life well lived
- Eleanor Roosevelt
- Happiness is fleeting
- If you are in trouble, contentment helps you make better decisions
- Think about a brutal on call week - if the systems that support you are good, you survive
Knowledge
Access to knowledge is a leading indicator of social progress.
The goal isn’t to minimize needed knowledge – it's to provide access to the wealth of it, when we need it.
- The right knowledge, at the right time
- Think about PaaS - it's awesome you just git push
- Until you are Rap Genius and heroku changes the router and everything sucks and you don't know why
- Which doesn't make PaaS awful - at a different level of criticality, who cares?
The power or right to act, speak, or think as one wants without hindrance or restraint. – The Internet
We should be empowering ourselves and others to act, speak, and think as they need to with less hindrance.
- The Big Web got this right
- Empower individuals to work as they see fit
- Trust them to do the right things
- Build systems that increase the trust needed to allow more freedom
People
Products
Companies
People -> Product -> Company
- Both the producer and the consumer
- When you do the right thing for the people, the right things happen for the product and the company
We are Lean
- Eliminate non-value-added action (Waste/Muda)
- Pull over Push
- Kaizen (Continuous Improvement)
- Kaikaku (Disruptive Change)
- Small Batch + Experimentation
- Eliminate things that do not add value to the outcome
- The "Pull" of demand aligns process and resources, rather than the "Push" of central planning and prediction
- Always be improving the way you work
- When you need to incorporate things that dramatically alter the way you work, recognize that, and flip the table
- Favor smaller batches and experimentation
- Failure is a learning opportunity
- a status quo event
- not an anomaly
Ubiquitous Workflow Automation
- Makes the easy path flow with our principles
- Make Conway's law work for you
- organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations
Diversity
- is good, both in thought, opinion, experience, and background
- this is sweeping kung fu - you need help
- Voices, stupidity, similarity
- You want harmony, thinking in different ways, bringing their skills to bear
- Good D+D parites are diverse
- We are not generalists - we are highly diverse specialists with high empathy and connection
Forms
Shared between many styles; emphasis varies from style to style
- There are probably more, but these are core to Chef Style
- There are many, so we are going to go fast
- Make it public
- Shoot for one sentence
- Include the people you are trying to help
- Include your product
- Include the change the people you are helping will see
The most enduring and transformative companies use Chef to become fast, efficient, and innovative software driven organizations
- Need a sense of clearly defined purpose thats bigger than the task at hand
- Sources of engagement
- Gallup Q12
- Write it down publicly
- Always refer back to it in every internal communication
- Shoot for one sentence
- Include the people you are trying to help
- Include your product
- Include the change the people you are helping will see
"The most enduring and transformative companies use Chef to become fast, efficient, and innovative software driven organizations"
(Contentment, Knowledge, Freedom)
(People over Product over Company)
(Lean Product Development)
- What brings about good outcomes in your environment
- Write them down, make them public
- Include the principles of DevOps
- Mix in the things that are unique to your industry/problem
We build software for our users; we do what is right for them. (The Right Hard Thing)
- Remember to use active voice
- Its okay to not be perfect! Nobody is!
Empowered teams
- Permission to act
- Paired with the context to make good decisions
- With leaders who care about your purpose
- And share your beliefs
- Empowerment
- Permission to act
- Context to make good decisions
- Access to help, guidance, and more context
- Find leaders who
- Care about your purpose
- Align with your beliefs
- Have leadership as a talent
- Holding of context and caring for your subordinates
- Be available for context and caring at any time - up to senior leaders, down to individuals
Form diverse bonds
- Take people to lunch, or have meetings, with people outside your specialty
- Ask them what they do, and try and understand their problems and perspective
- Legal, Finance, Sales, Marketing, Business Development, Software Development, Systems Administrators, Security Professionals, Product
- Take people to lunch, or have meetings, outside your specialty
- Ask them what they do, and try and understand
- You need to talk to:
- Legal
(Ask them what the role of a lawyer is, and how you can help make any interactions you have smooth)
- Finance
(Ask them what the role of a finance is, what key metrics they watch, and how you can help)
- Sales
(Ask them what their quota is, what do they struggle with at customers, and how can you help them make their number)
- Marketing
(Ask them what message is resonating, why, and how they see the market evolving)
- Business Development
(Ask them what partners are moving the needle, how they think about those arrangements, and whats on the horizon)
- Software Development
(Ask them what they are building, and why it matters, and what they are excited about in the future)
- Systems Administrators
(Ask them about availability, what do they think is the biggest limit to your organizational efficiency)
- Security Professionals
(Ask them how they evaluate your business and technology in terms of risk, how you can help)
- Product
(Ask them what they think the gaps in your product are, and to talk about the last 5 customers they've met)
Build consensus on important decisions
- Use your bonds to circulate plans with stakeholders
- Incorporate their criticism and feedback into the plan
- Present to the group
- Ask for consensus
- Before you try and do a major decision, make sure you know the answer going in
- Have a written down plan, and why you think its important
- Talk to all the key stakeholders, preferably 1:1
- Get their criticism and feedback
- Update your plan, incorporating their feedback and criticism (Make It Theirs)
- Present plan to the group
- Get it done
Strong value propositions
- Pain Killers, not vitamins
- Make something people ♡
- Needs not wants (one customer wants a feature; many customers need a feature)
Make products that have strong value propositions
- Product/Pain Relievers/Gain Creators
- Map to Customer Jobs, Customer Pains, and Customer Gains
- Pain Killers not vitamins
- Make something people LOVE
- Needs not wants (customer wants a feature, customer(s) need features)
- How to identify the opportunities
- Call out customer pain through engagement
- Interview customers to learn about their problems
- Solve the pain you experience yourself
- Write down your ideas for products
What is it? (product)
Who is it for? (target customers)
What problem does it solves and why does it matter? (problem & pain)
How does it work? (pain relievers & gain creators)
What is unique about it? (uniqueness)
What is the value proposition? (value)
How do we make money? (cost & revenue)
Build a roadmap
Start with your vision
Align with customer feedback
Balance innovation with customer needs
Group them into themes, with an outcome for each
Distill those into features, and validate with customers
- Your THEMES should hold
- Your OUTCOMES might hold
- Your FEATURES will change
- Lead with vision
- Align with customer feedback
- Balance innovation with customer needs
- Group them into themes
- Define the outcomes for the themes
- Distill those into features, and validate with customers
- Iterate on features according to vision and feedback
- Put it on a grid
- Focus on themes
- The power of the parenthetical
- Keep it alive through iteration
- Your THEMES should hold - your OUTCOMES might hold, and your FEATURES should shift all the time
Always have delighters
This is a simplified Kano model of product development
Always have delighters
- (The Creating Delight slide)
Build features iteratively
1
2
3
Paint the Mona LisaIncremental
Woman in pastoral settingIterative
Awesome analogy from Jeff Patton
Mona lisa slide
- You don't have debt if nobody is using it
- Don't mistake "doing it right" for doing it "incrementally"
- It's not iterating if you only do it once
- That means deploying these features! It's all connected!
Manage Risk
- Small batches, near term hypothesis
- Validation comes from customers
- Introduce near-term volatility to gain decreased long-term risk
- You are building for high velocity
- So small batches, with near term hypothesis
- Shipping the smallest and simplest thing possible to help
Get closer to our objectives
Get feedback
learn
validate
De-risk
- Experiments and validation with customers
- You introduce near-term volatility to gain decreased long-term failure
- Stylistic difference - there wasn't a way to be declarative about stylistic choices
- There is no reason you have to act drunk when doing drunk boxing
Do not worry about scale
Until you should
Way later than you think
Scaling your company, your software, your kung fu
Solve theory arguments with execution
- Theory is fun for yak shaving
- Reality is complex
- Execution is all that matters
- Weekly
- Anyone with anything to show
- Invite everyone
- Record it and post it internally
- Helps you focus on ITERATIVE vs INCREMENTAL (hard to demo one arm of the mona lisa)
- Teams that aren't demo-ing are probably being INCREMENTAL
Choose languages and tools that fit the job
- We are all polyglots
- Learning new languages and tools is one of the great joys of our industry
- Small batch + iterative development protects you from risk
- Use source code control
- You must have lightweight feature branches
Have a bug database
- Triage and prioritize the bugs
- Be nice to your customers
Continuous Integration
- Always integrate branches to master
- They should be short lived, iterative branches
- Fix the build when it goes red.
The four-eye rule
No change happens without at least four eyes on it
Write tests
- Unit test (a single function)
- Integration tests (multiple classes/units)
- Functional tests (user-oriented, high level, full stack)
- Smoke tests (quickly determine if the system is "working")
- Write one test at a timeS
Martin Fowler
Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time.
One path for change
- The way change moves through your organization is fixed
- Designed to re-enforce your principles and aid flow
- Flexible at the level of execution
- Avoid using words like
- Production
- Staging
- QA
- Those are activities and locations all rolled in to one
- They happen ALL OVER The workflow
Code goes through the same workflow
- Applications are code
- Infrastructure is code
Focus on Availability
- Availability = uptime/(uptime+downtime)
- Focus your efforts on reducing mean time to diagnose and mean time to repair
- Failure is inevitable; it's how you detect and react that matter most
Collect metrics
- From the operating system, network, applications, and your process
- High resolution matters
- Have as few systems as possible
Plan for capacity
- Identify key metrics
- Put them on a graph
- Set a limit
- Plot a trend line
- Expand your time horizon
Only alert on what is actionable
- Get the attention of the right humans
- As few alerts as possible
- Routed to the people who can take action
- Start with the is it up alert
Practice incident response
- Orient is the step that matters most
Practice incident response
- The First Responder is the default Incident Commander
- Decides what to do next
- Coordinates resources
- Communicates status
- Not about rank
- There is only ONE Incident Commander.
Incidents lead to Post Mortems
- Invoke the space: we are here to learn, not to blame
- Describe the incident
- Establish the timeline
- Identify contributing factors
- Describe customer impact
- Describe remediation tasks
- Describe improvement tasks for response process
Use scalable systems design
- Autonomous actors, responsible for themselves
- Making progress toward their goal
- Making clear promises to other actors
- Having those actors evaluate the quality of those promises
- About your code
- About your position
- About your knowledge
Simplicity
Extensibility
Re-use
- Conway's Law “Interpersonal variability, that is the capability and behavior differences between programmers using the same language, tends to account for more differences between programs than a change of the programming language.”
- Make a component as simple as it can be, but no more
- Design for re-use when you can
- CERN and the LXC - one framework for four experiments
- The number is 3 - you don't need a hundred re-use points
- Can you get 1 less than the number of actors
- Make extensible system
- There is a loop that gets you here
- Complicated
- Break down
- Simple
Unique to the individual
- Those are the forms we practice - the components of our DevOps Kung Fu
- Now we need to talk about what it means to take our Kung Fu into the world
Safety in technique
- Remember your principles
- Practice your forms
- Use your discernment
- When you use your core principles, and you use your discernment, you are in the right place
- The process by which we do things is what matters
- It is training and the situation and experience that brings you to the right value
- The next step is application - actually doing all of these things in your organization
- Given a long enough application, you will develop your own style - what works, what doesn't
- Before you go off and fight the whole school
- We have a way for you to practice the application of your forms in a safe way
- Which also happens to be the right way to start bringing devops in to your organization
- Take a business problem
- Bring together all the stakeholders
- Practice your Kung Fu together for 8 weeks, on that problem
- Lets you feel what its like, increases your bonds to each other, is specific to your problem
- Lets you build consensus together
- It ends - so there is low risk. The time box matters.
Stage 1 - Problem Selection
- Small enough to have a meaningful iteration in 8 weeks
- Vertical not Horizontal
Stage 2 - Purpose, Beliefs and Teams
- Write down your purpose
- Write down your beliefs
- Empower the team to act
- Be available for context
Stage 3 - Product Development
- Write down your value proposition
- Build a roadmap (Themes, Outcomes, Features)
- Include delighters
- Simplicity, Extensibility, Re-use
Stage 4 - Iterate on Features
- Iterate on features
- Manage risk through small batches, validated with customers
- Choose languages and tools that fit the job
- Ignore scale
- When arguing about theory, re-focus on execution
- Demo every week
Stage 4 - Iterate on Features
- Use source control
- Have a bug database
- One workflow for change
- Four eyes
Stage 4 - Iterate on Features
- Continuous Integration
- Continuous Delivery
- One test at a time (Unit, Integration, Functional, Smoke)
- Use scalable systems design
Stage 5 - Operate
- Focus on availability
- Collect metrics
- Plan for capacity
- Alert on what is actionable
- Run incident response
- Hold post mortems
Chef Style DevOps Kung Fu
- On Github: http://github.com/chef/devops-kungfu
- Join by PR
- We work together to develop our principles, forms, and applications
- Start your own school by forking, removing the names, and changing the content
Find your own way
- DevOps is real
- I know it by its universal principles
- I practice it by its shared forms
- I apply it to my daily work
- My practice is unique - and so is yours