加收藏
Speeding up development by Matthew Leventi@mleventi
第1页
Docker at Lyft
Speeding up development Matthew Leventi @mleventi #dockercon
Speeding up development Matthew Leventi @mleventi #dockercon
第2页
Lyft Engineering
第3页
Lyft Engineering
Organization - Rapidly growing headcount - Fluid teams - Everyone does devops
Technology - 50+ microservices - 25 server deploys a day - 2 client pushes a week - Highly available
DockerCon 2015
Organization - Rapidly growing headcount - Fluid teams - Everyone does devops
Technology - 50+ microservices - 25 server deploys a day - 2 client pushes a week - Highly available
DockerCon 2015
第4页
Systems Engineering
Developer Productivity - New developers ship on Day 1 - Seamless team switches - Faster feature development
Operational Stability - everything must scale - nothing goes down
DockerCon 2015
Developer Productivity - New developers ship on Day 1 - Seamless team switches - Faster feature development
Operational Stability - everything must scale - nothing goes down
DockerCon 2015
第5页
Developer Productivity
第6页
Inefficiencies Multiply...
DockerCon 2015
DockerCon 2015
第7页
General Problems
“It doesn’t work on my box!” “I don’t understand how the client got into that state!” “It worked in development!” “How do I get service X to talk to service Y?” “How do I test this feature from the client?” “How do I get started working on a new team?”
DockerCon 2015
“It doesn’t work on my box!” “I don’t understand how the client got into that state!” “It worked in development!” “How do I get service X to talk to service Y?” “How do I test this feature from the client?” “How do I get started working on a new team?”
DockerCon 2015
第8页
Invest in Dev Environments
第9页
In the past...
AWS Dev EC2 Instances - 1 per dev per service NFS syncing for code changes Service discovery through dev config sections Manual task to stay up to date on changes Individual SQS, Dynamo resources per developer Expensive to orchestrate
Vagrant VM Images Hard to run more than 2 on a mac Hard to interface with cloud resources.
DockerCon 2015
AWS Dev EC2 Instances - 1 per dev per service NFS syncing for code changes Service discovery through dev config sections Manual task to stay up to date on changes Individual SQS, Dynamo resources per developer Expensive to orchestrate
Vagrant VM Images Hard to run more than 2 on a mac Hard to interface with cloud resources.
DockerCon 2015
第10页
Development Environment
Devbox: Everyone has the same up to date local environment Onebox: All of lyft, in the cloud, running any combination of builds CI: Cross service integration testing, deploys
DockerCon 2015
Devbox: Everyone has the same up to date local environment Onebox: All of lyft, in the cloud, running any combination of builds CI: Cross service integration testing, deploys
DockerCon 2015
第11页
Devbox
Start a set of services easily:
./service start api dispatch eta
Automatically mount repos into services:
ls . api dispatch eta payments python-sdk
Load and save state snapshots:
./service snap issue519 ./service apply issue519
Open websites locally
./service open api
Build new services locally
./service build new_service_X
DockerCon 2015
Start a set of services easily:
./service start api dispatch eta
Automatically mount repos into services:
ls . api dispatch eta payments python-sdk
Load and save state snapshots:
./service snap issue519 ./service apply issue519
Open websites locally
./service open api
Build new services locally
./service build new_service_X
DockerCon 2015
第12页
Onebox
Every QA engineer has their own environment. No mocking needed for client development. Easy to share state between developers.
DockerCon 2015
Every QA engineer has their own environment. No mocking needed for client development. Easy to share state between developers.
DockerCon 2015
第13页
CI
Every service defines test suites with dependent services. Tests are run per pull request and on master commits. Isolated cross service integration tests.
DockerCon 2015
Every service defines test suites with dependent services. Tests are run per pull request and on master commits. Isolated cross service integration tests.
DockerCon 2015
第14页
How?
第15页
Service Model
- Single fat containers - Stateless - Fixed static ip address - Single “stateful” local
container - Auto detect code changes
DockerCon 2015
- Single fat containers - Stateless - Fixed static ip address - Single “stateful” local
container - Auto detect code changes
DockerCon 2015
第16页
Building a Service Image
- Docker image is a fs snapshot of config management. - Each image has:
- git clone of a central ops codebase - git clone of a service codebase - a salt stack provisioning run. - runit configuration for processes
ID = $(docker run --env SERVICE=api --env SERVICE_SHA=abc --env OPS_SHA=def lyft/base) docker commit $ID api docker push api
No dockerfiles!
DockerCon 2015
- Docker image is a fs snapshot of config management. - Each image has:
- git clone of a central ops codebase - git clone of a service codebase - a salt stack provisioning run. - runit configuration for processes
ID = $(docker run --env SERVICE=api --env SERVICE_SHA=abc --env OPS_SHA=def lyft/base) docker commit $ID api docker push api
No dockerfiles!
DockerCon 2015
第17页
Running a Service Image
- Rerun salt provisioning on new SHAs - Start runit processes - Terminate the container if initial runit checks fail Allows - Developers can easily apply ops modifications - Testing PRs are a matter of changing env variables - Don’t need to wait for an image build, deltas are applied during
runtime - Easy to mount code volumes and trigger changes
DockerCon 2015
- Rerun salt provisioning on new SHAs - Start runit processes - Terminate the container if initial runit checks fail Allows - Developers can easily apply ops modifications - Testing PRs are a matter of changing env variables - Don’t need to wait for an image build, deltas are applied during
runtime - Easy to mount code volumes and trigger changes
DockerCon 2015
第18页
Single Host
DevBox Mac docker host using vmware fusion with shared folders
CI Slave AWS ubuntu docker host for short lived containers
Onebox AWS ubuntu docker host for long lived environments
DockerCon 2015
DevBox Mac docker host using vmware fusion with shared folders
CI Slave AWS ubuntu docker host for short lived containers
Onebox AWS ubuntu docker host for long lived environments
DockerCon 2015
第19页
Managing State
All stateful processes run inside the same container. - Redis - MongoDB - DynamoLocal - SQS Local - Fake Kinesis Standard import/export scripts to S3 tar files. All developers, qa, slaves get their own data environment.
DockerCon 2015
All stateful processes run inside the same container. - Redis - MongoDB - DynamoLocal - SQS Local - Fake Kinesis Standard import/export scripts to S3 tar files. All developers, qa, slaves get their own data environment.
DockerCon 2015
第20页
Demo
第21页
Results
第22页
Results
Productivity Majority of new hires push to production on day one. Feature development is no longer blocked by devops. QA client testing is parallelized.
Stability 99% of deploys are successful. Every PR on every service is integration tested.
DockerCon 2015
Productivity Majority of new hires push to production on day one. Feature development is no longer blocked by devops. QA client testing is parallelized.
Stability 99% of deploys are successful. Every PR on every service is integration tested.
DockerCon 2015
第23页
Lessons Learned
第24页
Lessons Learned
VMWare Fusion can be unstable under load Frequent image downloads take time Bugs in config management can freeze development Easy service creation leads to unnecessary services Approach limits on what can run on a single box Static IP allocation not supported in docker
DockerCon 2015
VMWare Fusion can be unstable under load Frequent image downloads take time Bugs in config management can freeze development Easy service creation leads to unnecessary services Approach limits on what can run on a single box Static IP allocation not supported in docker
DockerCon 2015
第25页
Future
第26页
Future Ideas
Tons of t2.smalls to replace VMWare: - One container per host in the cloud - NFSv4 code syncing - Same static ip private network using libnetwork - docker-machine
Exploring production docker usage: - ETL jobs in docker - Containers to reduce ASG spin up/down times - Containers for atomic deploys
Tons of t2.smalls to replace VMWare: - One container per host in the cloud - NFSv4 code syncing - Same static ip private network using libnetwork - docker-machine
Exploring production docker usage: - ETL jobs in docker - Containers to reduce ASG spin up/down times - Containers for atomic deploys
第27页
Thank you
Matthew Leventi mleventi@lyft.com @mleventi #dockercon
Matthew Leventi mleventi@lyft.com @mleventi #dockercon