Large-Scale Deployments With Pogo At Yahoo!

1. Large-Scale Deployments With Pogo At Yahoo! Mike Schilli YAPC::NA 06/14/2012

2. What’s Pogo? • Yahoo’s deployment system • Highly scalable (thousands of targets) • Highly parallelized rollouts • In active use for several years • Open Source version available (pre-alpha)

3. Pogo in a CI environment

4. What’s “Deployment”? • Run one command on many hosts • Each host “knows” what it’s supposed to look like via a central database • One command brings host into “desired” state • Fetches package updates from edge-server repos

5. Run one command on many hosts • Act on all hosts immediately

6. Run one command on many hosts • Act on all hosts immediately

7. Example Server Farm

8. Pogo Features • Sequences • Parallelizes with constraints • Halts if errors violate constraints • Runs health checks (pre/post hooks) • Command agnostic (rpm, apt-get, custom, etc.)

9. Server Farm Release via Pogo

10. Server Farm Release via Pogo

11. Server Farm Release via Pogo

12. Server Farm Release via Pogo

13. Server Farm Release via Pogo

14. Server Farm Release via Pogo

15. Server Farm Release via Pogo

16. Server Farm Release via Pogo

17. Server Farm Release via Pogo

18. Server Farm Release via Pogo

19. Server Farm Release via Pogo

20. Pogo Configuration

21. Run a Pogo job (tags)

22. Pogo UI – Single Job View

23. Pogo UI Overview

24. Pogo UI – Host View

25. Pogo UI – Start it Up $ perl-Ilib bin/pogo-api pogo-api-28> Listening on 0.0.0.0:7657 $ perl -Ilib bin/pogo-ui -v pogo-ui-22> Listening on port 5000

26. Run a Pogo job (targets)

27. Pogo Configuration

28. State of Pogo • Pre-alpha Open Source version on Github • OSS Pogo will be completed in the coming months

29. Pogo on Github • Developed in the open: • https://github.com/ytoolshed/pogo • Internally used with plugins to adapt to specific data sources

30. CI Tests with travis-ci.org

31. The Gory Details

32. Pogo – User View

33. Pogo Workflow

34. Authentication/Authorization • Web server authentication for submitting jobs • Target authentication via passwords or priv/pub keys • Credentials encrypted with worker pubkey • Credentials never stored persistently (or ZooKeeper)

35. Pogo “Hooks” • Scripts that run before or after Pogo commands • Used to prepare host, perform check health, or other functions • Installed locally on target hosts • rc.d style invocation: run in alphanumeric order

36. Pogo “Hooks” (cont.) • Two types: • pre hook: run before the specified command • post hook: run after the specified command • If any hook script fails: • No further scripts or commands run • Host is marked as failed

37. Pre-hook Examples • Take a host out of rotation • Drain database connections • Check that host is ready to be updated

38. Post-hook Examples • Check whether local webserver is serving expected content • Ping required services • Put host back into rotation

39. Host Failures and Concurrency • If a host fails: • Command returns non-zero • Pre- or post-hook returns non-zero • The failed host counts against the acceptable number/percent of unavailable hosts • Example: • Four hosts in an app can run at a time • One host in the app fails • Only three hosts will be acted on at a time going forward

40. Implementation • AnyEvent Framework • Asynchronous, event-based • Added Object::Event for event pub/sub

41. HTTP Client in AnyEvent

42. HTTP Client in AnyEvent

43. HTTP Client in AnyEvent

44. AnyEvent and Object::Event

45. AnyEvent and Object::Event

46. AnyEvent and Object::Event

47. Queue Processor with AnyEvent and Object::Event

48. AnyEvent bin/pogo-dispatcher

49. Testing AnyEvent Components

50. All Pogo Components