Devopsdays Austin 2019

My mission: Deconstruct and simplify a complex, multi-layered infrastructure pipeline with no standards, define those standards, create individual tools that conform to them, and address edge cases. Lessons learned from my experience enacting and implementing standards in a large organization.

It is easy to sacrifice standards on the altar of temporary convenience. The solution: Improve the standard where necessary, even when it is hard. However, what do you do when you have _already_ offered sacrifices on the altar of convenience, over and over? Now there is a system laden with technical debt, and in order to move forward, that debt must be paid down.

Come learn about our year-long journey to _write and enact standards_ for our tools (and, by extension, all APIs published by Google), _refactor our tools_ to conform to those standards, _deal with the edge cases_ that we could not or would not standardize, and ultimately **reduce the cost** of our infrastructure pipeline. Additionally, come learn about how I intend to automate myself out of a job in 2020.

UX is driving you crazy, a black throwing off timelines and killing ideas. UX doesn't seem Lean or Agile. Can't we circumvent or exclude these people? This session will explain how the UX process fits into Agile; saves companies money; augments DevOps goals; and increases customer satisfaction.

UX is driving you crazy, a black throwing off timelines and killing ideas. UX doesn't seem Lean or Agile. Can't anybody make wireframes? Can't we circumvent or exclude these people?

DevOps is truly about so much more than how developers connect with IT, how infrastructure is managed, and how frameworks can be improved. It's about recognizing how many teams are truly involved in the software development process and finding better ways to make sure everybody is at the table.

This short Minimum Viable Presentation will explain how the UX process fits into Agile; saves companies money; augments DevOps goals; and increases customer satisfaction.

Learn about CI/CD as a practice and see how easy it is to scale a Jenkins environment with ephemeral build agents in a Kubernetes cluster.

Running containerized, ephemeral build agents in Jenkins allows you to isolate application dependencies and dynamically scale in response to fluctuations in CI/CD workloads, but you need a container orchestration solution or you trade in the management of individual Jenkins agents for management of individual container engines. In this session, you'll learn how to dynamically provision Jenkins agents to run on a Kubernetes cluster and build, test, and deploy applications to Kubernetes using Jenkins' scripted pipeline.

Using my experience as an EMT and time working in ERs and ambulances, places where incident response can be a high-stress environment, I present a high-level framework for beginner and intermediate incident responders that will help them calmly and systematically tackle incidents.

Effective incident response is more than just "get alerted, fix a problem." Having a systematic approach that discourages "tunnel vision," helps responders develop their own intuition, style, and procedures are critical in not only solving problems but also in preventing responder burnout.

Knowing what to monitor, how, and what questions to ask of your internal customers and of your monitoring platform itself can be difficult. Using my experience as an EMT and time working in ERs and ambulances, places where incident response can be a high-stress environment, I present a high-level framework for beginner and intermediate incident responders that will help them calmly and systematically tackle incidents.

I'll cover things like:

Your patient's vital signs
Treating with time
The difference between emergency response and surgery
What triage really means
Treating your patient and not the monitor
Sleeping or dead?
An acronym to help you always ask the right questions

Dealing with any neuro-diversity is difficult and stigmatized, but lessons learned from DevOps and Complex Systems management have helped me improve how I handle situations. These lessons can be applied broadly and shift our thinking to how DevOps extends beyond tech alone.

Mental illness is stigmatized and hard to talk about publicly. Doubly so in the tech industry where we assign much of our personal value to our brains. I want to share my struggles with mental health and the lessons I've learned from Systems operation and DevOps that have helped me get by and frame my struggle in a manageable way.

I want to start a conversation to:

- Help reduce the stigma of Mental Illness by talking about it
- Provide some illustration of things that have helped me, so hopefully they can help you
- Show how the concepts of DevOps have application reaching beyond High Technology
- Spark further conversation about Mental health in tech
- Hopefully provide a new group and community for those for whom mental health is a concern

With metrics, histograms, and traces, we finally have a complete picture of the microservices application that will empower us to not only get a pulse of the overall health of their application, but also drill down into the specific misbehaving areas more accurately and faster than ever.

The modern application landscape is changing fast! A microservices architecture modularizes your application where each service can be built and scaled independently of the other. But with great power comes great responsibility. It is also ever more challenging for developers and operations to not only monitor, but to be able to quickly triage and identify the pain points along with the speed of continuous integration and deployment. In this talk, we are going to explore how utilizing metrics, histograms, and traces can help tackle challenges of modern applications observability riddles.

Scaling is hard, especially when the thing you need to scale is your CI system. At a certain point, conventional approaches break down and the blog posts telling you the five easy steps to solve your problems start disappearing. It's time to assess the problems and find solutions.

When setting up a company for success, one of the least-invested systems is CI infrastructure, and yet with rampant growth and large Engineering departments it inevitably comes to the fore when it cannot scale. Increased build durations, increased queue times, and flakey test systems are symptomatic as this problem compounds. Once this tipping point is reached, it is a scramble to find the bottlenecks and patching and hoping it never happens again. In this talk, we'll go over ways to identify and mitigate in the moment, as well as approaches used to make our CI systems as scalable as our application code.

Security in FaaS isn't what we are used to, but this talk shows you how what we learned in appsec still applies. Using LambHack, which is a vulnerable serverless application written in Go on AWS Lambda using Sparta, we will evaluate how to do security in serverless.

In this talk, we will talk about security strategies and pitfalls in the serverless world. You'll leave with an understanding of how to approach security conversations about serverel

Talk goals:

- How to approach the security concerns in a serverless world.
- Talk about the 'WIP' methodology for serverless security.
- Understand current serverless attacks for things to defend against.
- Learn what different cloud providers (AWS/GKE/Azure/Oracle Cloud) do to protect you in a serverless world.

Is your On-prem K8s blender on fire? Let's figure out how to combine Kubernetes and BYO Infrastructure in a safe way that minimized operator risk. DevOps first, Margaritas after!

Kubernetes is hyped as the end of infrastructure; however, it's still mainly a public cloud only platform. This talk lives that the intersection of the dream of Kubernetes as the perfect platform abstraction and the reality of snow flaked on-premises operations.

We'll explore the requirements inside and outside of Kubernetes core operational needs, like load balancing and certificates, that make on-premises deployments tricky and discuss ways that to provide those services in a constructive, automated way. The goal is to provide a manageable list of considerations for people looking to manage their own Kubernetes deployments in cloud, on-premises or edge.

We'll also include bonus topics that relate to Day-2 ops for Kubernetes including: community cluster automation efforts, Kubernetes on metal, and immutable live-image installs.

While Datadog is not an ideal medium for creating error budget burn rate alerts, it is possible to approximate the SRE Workbook Chapter 5 recommended "Multiwindow, Multi-Burn-Rate Alerts". If you're using Datadog, come by and see the setup.

This talk walks through my setup of Multiwindow, Multi-Burn-Rate Alerts on Datadog (See section 6 of [Chapter 5](https://landing.google.com/sre/workbook/chapters/alerting-on-slos/)). Datadog has a lot of gotcha's, so it is not very straightforward to do. I'll show example Datadog alert (screenshots/slideware most likely) and explain what the various bits do. Leading up to the slideware demo will be an explanation of Multiwindow, Multi-Burn-Rate Alerts, which is likely to be regurgitation of what's in the SRE Workbook. The main takeaway is the details of what to put in the Datadog monitor fields to get what we want out of it. I'm not affiliated with Datadog, but we use it at work and I assume a lot of the audience may too.

While learning how to do this, I didn't find burn-rate alerts intuitive at all and had to spend a few days understanding what the formulas mean. For example, what are the units of "burn rate"? Why do we monitor for burn rate being greater than 7 if we want to be notified about running out of error budget within 48 hours on a 2 week SLO measuring period? The plan is to share all those details in Open Space as various questions come up.

There are a variety of tools available that make it easier to offer and manage third party services (open source and commercial) on Kubernetes. In this conversation we will provide an overview of common service patterns for certain use cases, and open up to a deeper conversation including demos!

Kubernetes runs more than custom applications. Each day there are more third party services being offered with Kubernetes-native deployment options. With tools like Helm, Operator Framework, and Service Catalog available; it can be overwhelming to to differentiate between these options and understand what is the right tool(s) for the job at hand.

In this conversation, we will take a practical approach to compare and contrast different tools and patterns for creating, managing, and consuming 3rd party services on Kubernetes. In the morning 20 minute session, we will identify common service deployment patterns including platform add-ons, shared-instance deployments, dedicated instance deployments, and on-demand services running outside the cluster.

Then in the afternoon we will continue the conversation and discuss the tools that benefit these patterns, considering the needs of both Platform Operators (visibility, security) and Developers (choice and flexibility, ease-of use). Demos included!