White p ages Practical Experience Converting from Ruby to Reactive

1. Whitepages Practical Experience Converting from Ruby to Reactive John Nestor and Dragos Manolescu {jnestor,dmanolescu}@whitepages.com

2. Disclaimer The following material is subjective and represents the views of the presenters. We hope our views and experience may be of use to others making the transitive from Ruby to Scala, but your mileage may vary.

3. Whitepages Typesafe Case Study • WhitePages Rebuilds Core Parts of Application Stack with Scala and Akka to Improve Scaling. • http://j.mp/wpscala

4. Outline • Background • Technology • People

5. Background

6. About Whitepages • Top web and mobile site for finding phones, people and locations. • 50M unique users per month. • 35M search queries per day. • 70 engineers, mostly in Seattle.

7. Why replace Ruby? Problems with Ruby legacy systems: • High latencies. • High resource consumption: memory and processors. • Poor tolerance of upstream degradation: • Limited concurrency. • Worker starvation when blocking on slow upstream. • Poor connection management and healing.

8. Why replace Ruby with Scala? • Elegant blend of Functional and OO paradigms. • Static type system: • Type inference avoids the excess boilerplate of Java. • Compiler catches many errors. • Runs on the JVM: • Fast. • Near seamless interop with JVMlibraries. • Reasonably mature tooling support. • Actor-based concurrency framework:Akka.

9. Whitepages Reactive Services • Service-oriented-architecture where communication is Thrift or Json over HTTP. • Both latency and throughput are very important. • We require excellent logging and monitoring. • Agility of development, testing, build and deployment provide leverage and competitive advantage.

10. Results Performance

11. Status of Reactive Scala Services • We have been using Scala for a little more than 1.5 years. • We started with 1 development team, we now have 4. • We started with a single back end service. We now have 4 services in productionand another 6 in development. • Our first project had 6 Scala developers, we now have over 20 Scala developers. • We changed not only language but also tools (more automation) and processes (more agile).

12. Future Reactive Work • Transition additional Ruby back-end services to Scala. • Exploring new technologies: • Spark and Mesos. • A messaging system based on Kafka. • A graph store based on Titan/Cassandra. • A deployment and lifecycle-management system based on Dockerthat uses remote actors.

13. Technology

14. Technology Selection GitHub Enterprise IntelliJ Jenkins (for builds) Artifactory Yourkit SCoverage Scrooge (for thrift, from Twitter) Scala Akka Spray SBT – with custom plugins Typesafe Config Scala Test Coda Hale Metrics Persist Json

15. Whitepages Service Framework • Configuration (based on Typesafe Config, with multi-level overrides). • Monitoring (central actor that uses Coda Hale metrics and publishes to Graphite, Nagios and Riemann). • Logging (custom system with central actor, Json log content and real time streaming to external analytics). • Admin REST API. • REST Thrift/JSON Server (based on Spray). • Clients (Spray, JDBC, Redis, SOLR, Riak,… -- advanced error detection/ recovery).

16. Technology Successes • Scala concise syntax. – Higher developer productivity. • Use of async future based code. – Great performance. • Immutable collections and functional programming. – Fewer bugs. • Strong type checking. – Fewer bugs and more maintainable code (But meta-programming becomes more difficult). • Concurrency. – No race conditions or deadlock on complex concurrent system. • Spray. – Great performance and an async API. • SBT. – Able to easily customize to our needs (after steep learning curve). • IntelliJ IDEA. – Scala support is good and keeps getting better. • Typesafe Developer support contract. – Fast response, great answers to complex questions. (We should have gotten it earlier!)

17. Technology Problems -1 • Poor documentation: • SBT is the poster child. • We still often have to read source code of Scala and Akka libraries. • Stable APIs: • Typesafe has made great strides here. • Other components are still troublesome. • We lost many weeks as a result of breaking changes in ScalaTest. • We currently can’t compile a 2.11 version of Kafka because the Zinc Scala compiler is not backward compatible.

18. Technology Problems -2 • Performance analysis of concurrent constructs: • We went with the Typesafe Console. • It never fully worked and then was abandoned. • There are some new tools,but we have not had time to evaluate.

19. Technology Problems -3 • Ecosystem: • Not as mature as the Java ecosystem and often missing Scala specific features. For example, Scala Mock does not support named parameters. • Some needed components were missing: until recently code coverage. • In some cases there were too many options: over 10 Json libraries. • Lots of ScalaGitHub projects. Quality is variable; many become abandoned.

20. Technology Problems -4 • Debugging: • Debuggers work on the Java byte code, that for some Scala constructs is quite convoluted. • Debugging chains of futures is much harder than debugging nested calls. Some kind of tracing is desperately needed. • Actors have lots of power but can be exceptionally hard to debug. We need both better approaches to building debug-able actors and better debugging tools.

21. Technology Problems -5 • Language and Library Issues: • Type erasure is a major flaw in the Scala language. It causes lots of developer confusion and errors. The workarounds are awkward at best. • The lack of type checking on Actor constructors is an odd hole in the type checking and a real annoyance. • Some Scala code that looks simple and intuitive creates major difficulties when a developer needs to understand what is really going on. Implicits are one of many problems here.

22. Technology Overall • The problems mentioned above create real day to day difficulties. • In many cases Typesafe has been actively working to incrementally correct some of these problems. • Since we started using Scala, the number of problems has significantly decreased and we expect that trend to continue. • Overall, Scala/Akka is a great technology for building reactive systems and its benefits far outweigh the problems.

23. People

24. Levels of Scala Expertise • Basic OO features. Easy transition from most other languages including Java and Ruby. • Adding functional. Ruby has functional features that are similar. But knowing when best to use functional versus OO requires experience. • Adding concurrency. Via futures, promises, and actors. A whole new world for Ruby programmers that requires a new way of thinking. • API design. Not all developers need to design reusable APIs, but those that do need a higher level of experience and feature mastery to do it well.

25. Building ScalaDevelopment Teams • The are still not a large number of experienced Scala developers. • Those out there are in considerable demand. • We have followed a mixed strategy of both recruiting Scala developers and also training existing Ruby developers to become Scala developers. • Having an experienced Scala developer greatly accelerated the training of Ruby developers. Note: Using Scala, Akka and other advanced technologies makes Whitepages very attractive to some of the very best developers.

26. Recruiting: Finding Developers • Personal contacts. The best way to find great developers. • Meet-ups and Events • John Nestor is the organizer for the Seattle Scala meet-up. • Whitepages hosts the Scala meet-up (and several others) • Whitepages sponsors many other technology conferences and events. • Job posts: local Seattle and internet Scala focused. • College recruiting events. UW and other top CS schools.

27. Recruiting: Attracting Developers • Strong support from the Whitepages CTO and CEO and other management for Scala and other advanced technologies. • We have a great office (recently completely redone) • We provide developers with the best workstations (top-end MacBook pros). • We have competitive benefits, excellent work-life balance and a collaborative supportive culture.

28. Training: External Resources • Books. Odersky: Programming in Scala, and others • Coursera. Functional programming, reactive programming. • Typesafe training classes. Both Seattle public and in-house. • Typesafe developer support contract and consulting. Particularly valuable if you don’t have in-house Scala expertise. • Conferences. We sent people to Scala Days in NY and Berlin.

29. Training: Internal Resources • Whitepages developers are in a single office and come in most days. • This permits a high degree of personal interactions. • Those interactions enable rapid learning and high code quality. • This would be much harder with multiple offices and/or widely distributed teams.

30. Training: Internal Approaches • Expert mentoring. • Provide quick answers to questions. • Detailed reviews and feedback for Ruby developers initial Scala code. • Code reviews • Improve code quality. • Help those learning Scala. • Develop a shared culture and style. • We use a mixture of group and pair code reviews. • Electronic means like code review tools and pull requests were less effective.

31. Training: Success and Failure • Not everyone made a successful transition from Ruby to Scala. • Next slide give some positive and negative indicators…

32. Training: What to Look For • Desire to learn new technologies. • Desire to learn new ways to think. • Prior functional language experience. • Likes to work closely with others.

33. Scala Features to Avoid • Scala is a very large language. • New developers can easily get lost on the rich feature set. Often there are several different ways of doing something, and they don’t always know how to choose. • We also provide the following list of features for new developers to avoid (or to fully understand before using). • We encourage asking more experienced developers to provide that understanding. • These features are also flags for discussion in code reviews.

34. Scala Features to Avoid Reflection and macros asInstanceOf Top level objects with mutable state Lazy vals Scala enumerations ? (the ask pattern) onComplete (unless body enclosed in try) Nulls Return, break, continue Mutable collections var, while, until Generics Implicits Actors Await

35. We Are Hiring*! *Smart people (not just technologies) are key to our success

36. Thank you!Questions?