Platform Quality APIs

1. Platform Quality APIs Jim des RivièresIBM OTI Labs (Ottawa)jeem@ca.ibm.com

2. Outline of Talk • Platform quality APIs • Long-haul APIs to support open-ended set of clients • Best Practices for API Design • Design advice for good APIs • API Evolution • Changing APIs while keeping existing clients happy • Best Practices for API Development • How to work effectively on components with APIs

3. Topic – Platform quality APIs • Platforms are… • Useful • Platform provides useful services to clients • Open • Platform intended for open-ended set of diverse clients • Stable • Platform provides stability across multiple releases • Growing • Platform evolves from release to release to better meet client needs

4. Successful platforms • Examples of successful platforms • Win32 • Intel x86 • IBM 360 • J2SE • Mac OS • Successful platforms last a long time • They don’t break existing clients – compatible evolution • They keep existing binaries working – binary compatibility

5. Platform quality APIs • APIs just among friends (adjacent components in same release) • Unsupported for arbitrary clients • Don’t need to be stable from release-to-release • You get chance to correct your mistakes • Stark contrast to platform APIs • Platform APIs must support open-ended set of clients and be stable • Once API is released there’s no going back • Breaking changes are bad news • Either extend API in next release • Or provide improved API alongside old API

6. Platform quality APIs are unforgiving • One chance • To tell clients how to use API – the “API specs” • To choose helpful class and method names • To decide between class or interface • Once API is in service • Constrained to work within framework laid down • Repair and improve without breaking clients • Unclear, inadequate, or incorrect API specs are bad news • Confusing to clients • Fixing API specs may make things worse

7. Good specs are key to platform quality APIs • Good APIs needs good specs • Code is not enough (even source code) • API specs • Describe intended behavior of API element • How it’s supposed to work • How clients should behave to ensure they will not be broken by future changes • Client code written to spec • Every reason to believe it will work with any conforming implementation • Implementation code written to spec • Every reason to believe it will support all conforming clients • Best (&only) known recipe for long-term survival • Allows API to endure, evolve, and satisfy clients that build upon it

8. Topic – Best Practices for API Design • Design advice for good APIs

9. Clearly separate API from internals • APIs should be highly visible so that clients can find • Publish API specs • Draw clear distinction between API and non-API • Eclipse non-API packages “internal”; e.g., org.eclipse.ui.internal • API consists of public classes and interfaces in API packages • All interface members • Public and protected members of classes • No “visible” connection from API to non-API • API should be closed story • Implementation details should not leak into API

10. Separate Core from UI • “Core” component • Provides basic infrastructure that can be used in headless manner • Usually provides API • May depend on other core components • Must not depend directly or indirectly on UI components • “UI” component • Provides elements or frameworks for use in GUI • May have API (but often not) • May depend on other UI components • May depend on core components • All inter-component dependencies are via APIs

11. Segregate client API from provider API • Component often provide APIs for different parties • Client APIs • Used by bread-and-butter clients, like a UI • Provider APIs (aka SPIs) • Used only for plugging in new pieces • Keep different kinds of APIs segregated • Client API should be easy for clients • SPI should be flexible, powerful for service provider • Clients shouldn’t have to understand SPI too • Clients shouldn’t be able to meddle in SP affairs

12. Use Java classes and interfaces wisely • API interfaces that clients do not implement are useful • Excellent way to insulate client from implementation internals • Allows there to be multiple implementations • Adding methods in future is non-breaking • API interfaces that clients may implement are inflexible • Adding method breaks binary compatibility • Abstract classes work better for this • API classes that clients do not subclass are useful • Use factory methods and hide constructors • API classes that clients may subclass are useful, but difficult • API frameworks are notoriously hard to specify • Needs additional contracts for subclassers

13. Use Java modifiers to block unintended usage • If API class is not intended to be subclassed by clients • Declare class final, or make constructors package-private • If API class is not intended to be instantiated by clients • Declare constructors private • If API method is not intended to be overridden by client subclasses • Declare method final • If API class is intended to be subclassed but not instantiated by clients • Declare class abstract • Declare constructors protected • Use “soft” injunctions in specs if there’s no way to block • “This interface is not intended to be implemented by clients.”

14. Factory methods are better than constructors • Constructors are limiting as API • All constructors effectively have same name • Cannot have two with same type parameters • public Path(String platformNeutralString) • public Path(String osSpecificString) • Constructors cannot return instance of subclass • Factory methods are more flexible than constructors • Factory method have names • public static Path newPath(String platformSpecificString) • public static Path newOSPath(String osSpecificString) • Factory method can return instance of subclass • Keep constructors hidden (private, package-private, or protected)

15. Methods are better than fields • API fields are very weak • public String hostName; • No way to intercept field accesses either read or write • No lazy init on first read • No possibility to report changes • Non-final fields vulnerable to overwrite • More flexible to expose get/set API methods • public final String getHostName(); • public final void setHostName(String hostName); • Use fields in API only for constants (public static final)

16. Program extra defensively at API boundary • Clients do the strangest things • Usually accidentally, but sometimes intentionally • Check incoming arguments are to spec • Don’t ignore - throw some form of RuntimeException • In Eclipse null argument is illegal unless explicitly allowed by spec • Worry about argument capture • What are consequences if client later modifies object being captured? • Consider copying arrays and collections • Worry about result disclosure • What are consequences if client later modifies object being return? • Consider copying arrays, returning unmodifiable collections • Worry about threading, or reentrancy • Are you holding locks when you issue that callback?

17. Conclusion • No one can really tell you how to design good APIs • Any more than they can really tell you how to write good programs • API design is just a form of high-stakes program design • Talk to other programmers • Ask how they’d approach it • Study successful APIs • APIs are generally available and well-documented • Recommended reading • Effective Java Programming Language Guide, by Josh Blochhttp://java.sun.com/docs/books/effective/ • How to Design a (module) APIhttp://openide.netbeans.org/nonav/tutorial/api-design.html

18. Topic – API Evolution • APIs need to evolve from release to release • Changes to API could invalidate existing clients • Evolve API in compatible ways • Preserve as much value as possible across API changes • Keep existing clients working • Two general considerations • Contract compatibility – Honor existing API contracts • Binary compatibility – Keeping the JVM happy

19. Contract Compatibility Before /** Returns the non-empty list of indices. */ public int[] getIndices(); After /** Returns the list of indices. The list may be empty. */ public int[] getIndices(); Breaks some existing callers int[] d = getIndices(); System.print(d[0]); // possible array index out of bounds However, existing implementers are fine public int[] getIndices() { …; return result; // result is non-empty }

20. Evolving API Contracts • API contracts are expressed in API specs • API contracts promise the client certain things • Clients can play multiple roles – e.g., caller, implementer • Different roles have different contracts • Changes to contracts should not invalidate existing clients

21. Binary Compatibility Before public void register(String key); After public void register(Object key); Existing calls re-compile as expected register(“foo”); // no compile error But existing binaries no longer link register(“foo”); // link error

22. Binary Compatibility DON’Ts for API Elements • Rename a package, class, method, or field • Delete a package, class, method, or field • Decrease visibility (change public to non-public) • Add or delete method parameters • Change type of a method parameter • Add or delete checked exceptions to a method • Change return type of a method • Change type of a field • Change value of a compile-time constant field • Change an instance method to/from a static method • Change an instance field to/from a static field • Change a class to/from an interface • Make a class final (if clients may subclass) • Make a class abstract (if clients may subclass) …

23. Binary Compatibility DOs for API Elements • Add packages, classes, and interfaces • Change body of a method • Do anything you want with non-API elements • Add fields and type members to classes and interfaces • Add methods to classes (if clients cannot subclass) • Add methods to interfaces (if clients cannot implement) • Add non-abstract methods to classes (if clients may implement) • Reorder class and interface member declarations • Change value of a field (if not compile-time constant) • Move a method up to a superclass • Make a final class non-final • Make an abstract class non-abstract • Change name of method formal parameter …

24. Binary Compatibility • Java VM has special rules for binary compatibility • API changes should be binary compatible • Existing clients should continue to work without recompiling • N.B. Java compiler does not detect this kind of breakage • Ref: Evolving Java-based APIshttp://eclipse.org/eclipse/development/java-api-evolution.html

25. Adding Methods to API Interfaces • API interfaces used to hide implementation work well • “/** … This interface is not intended to be implemented by clients */” • Add new methods to API interface • Add corresponding methods to implementing class package org.eclipse.core.resource; /** … This interface is not intended to be implemented by clients */ public interface IWorkspace { … } package org.eclipse.core.internal.resource; class Workspace implements IWorkspace { … } • public boolean isTreeLocked(); // new • public boolean isTreeLocked() {…}

26. Avoid API Interfaces that Clients May Implement • API interfaces that clients may implement are problematic • Adding method breaks binary compatibility • Use API class instead of API interface… • When client may implement • When there is a chance new methods needed in future • N.B. converting interface to class breaks binary compatibility

27. Adding Methods via I*2 Extension Interfaces • If no choice, add new methods in extending API interface • Avoids breaking existing clients that implement package org.eclipse.ui; public interface IActionDelegate { … } // original interface public interface IActionDelegate2 extends IActionDelegate { void dispose(); // new } Usage IActionDelegate d = new IActionDelegate2() {…}; if (d instanceof IActionDelegate2) { IActionDelegate2 d2 = (IActionDelegate2) d; d2.dispose(); // call new method }

28. How to Delete API • API deletion always breaks any existing clients • But replacing API with improved version is usually doable

29. Replacing API Methods • Add replacement API method • Deprecate original method • Ensure original method continues to work package org.eclipse.jdt.core.dom; public class Message { … /** … * @deprecated Use getStartPosition() instead */ public int getSourcePosition() { return getStartPosition(); // forward to new method } public int getStartPosition() { … } } package org.eclipse.jdt.core.dom; public class Message { … /** … * */ public int getSourcePosition() { // rename getStartPosition() … } }

30. API Evolution – Summary • Evolve API in compatible ways • Honor existing API contracts • Observe technical rules for Java binary compatibility • Usually feasible to find way to improve API and keep existing clients working without recompiling • Design APIs with future evolution in mind

31. Topic – Best Practices for API Development • Good APIs don’t just appear overnight • Significant design effort • Good APIs require design iteration • Feedback loop involving clients and implementers • Improve API over time • Components build upon the APIs of other components • Need collaborative working relationship between teams • Some ways to work effectively on components with APIs, based on Eclipse Project experience

32. Before you begin • Have and agree on common API guidelines • Eclipse Naming conventionshttp://dev.eclipse.org/naming.html • How to Use the Eclipse APIhttp://www.eclipse.org/articles/Article-API%20use/eclipse-api-usage-rules.html • Have someone in charge of the API early on • Have well-defined component boundaries and dependencies • core vs. UI

33. Work with API clients • APIs exist to serve the needs of clients • Where are those clients? What do they need? • Important to work with actual clients when designing API • Designing APIs requires feedback from real clients who will use it • Risks crummy API that real clients cannot use • Find a primary client • Ideally: adjacent component, different team, same release • E.g., JDT UI is primary client of JDT Core • Work closely with primary client • Listen to their problems with using API • Watch out for lots of utility classes in client code symptomatic of mismatch between API and what client really needs • Work together to find solutions

34. APIs First • Basic “APIs First” workflow • Work with primary client to decide what API you want/need (design API) • Write API specs • Write API test suites • Implement API (Expect numerous iterations within basic workflow) • Helps ensure APIs are • Ready for clients to use (specs and impl. in place) • Of high quality and stable • Ready to evolve to meet next requirement

35. Don’t reveal API too early • “We shall ship no APIs before its time” (to paraphrase Orson Welles' old wine commercial) • Keep work in internal packages until new API is ready • API specs in place • API test suite • Credible implementation • When ready, release by moving to API package • Keeps everyone’s expectations realistic • Clients • Component owner • Avoids embarrassing recall of unfinished API just before shipping

36. Write comprehensive API unit tests • Unit tests for each API element • Tests written “to spec” • Assumes and tests only what is covered in spec • Plays role of idealized client • Helps you to keep client view of the API • Implementer’s safety net • Catches stupid mistakes before they can screw over clients • Helps working relationship with primary client • Core components • Complete API coverage by automated tests • UI components • Partial coverage by automated tests • May required additional manual tests for visual elements

37. Keep API specs consistent at all times • You write a story. Someone else changes the story. How do you know whether the story still reads well? • Errors and inconsistencies within specs are hard to detect/remove • API spec is more like a book than like code • Maintaining consistency require re-reading • Start with initial set of consistent API specs • Scrupulously maintain consistency as API spec grows/changes • Have a “Keeper of the Specs” • Responsible long-term for maintaining specs and reviewing spec changes • Requires thorough knowledge of API and how it got its spots

38. Keep API tests up to date at all times • Errors and inconsistencies within tests are found immediately • But insufficient test coverage is hard to detect/remove • Start with a set of tests that completely cover API • Scrupulously maintain tests as API spec grows/changes • Can be done as part and parcel of changing API • Or by “Keeper of the Specs” if they keep the tests as well

39. Funnel all external dependencies thru API • Component provides API for arbitrary clients • API exists to tame inter-component coupling • Client components are expected to use API “to spec” • Not depend on behavior not covered in API spec • Not depend on internals • Foolish to make exceptions for close friends • Close friends don’t point out flaws • Sets bad example for others • Most common form of Eclipse component is single plug-in • Internal packages have “.internal.” in name • Plug-ins should not reference internal packages of other plug-ins

40. Focus clients on API • Encourage clients to discuss/propose API changes in terms of existing API • Grounds discussion in reality • API owner ultimately decides how best to address issue • API needs to make sense for all clients • Final decision may differ for what was proposed

41. Tread carefully in vicinity of APIs • Be especially careful when working on code in component with an API • Adding/deleting public method • internal class - no problem • API class - changes API • Be careful with renaming/refactoring tools • Make sure APIs not affected

42. Tag API elements with @since • Scrupulously record which release API element first appeared in • In Javadoc with @since tag • @since 3.1 • Elsewhere in words • During release cycle • Distinguishes old API from new API added during current release cycle • Still flexibility to change (or delete) newly added API • After release has shipped • Help clients targeting release >= N to avoid API added after N

43. Minimize disruptive API changes • Breaking API changes are very disruptive to clients • Non-breaking API changes also cause work for clients • Work required to port to use new/improved APIs • During release cycle • Schedule API work for early in cycle • Whenever possible, preserve API contracts • When not possible, coordinate with clients to arrange cut-over plan • After release has shipped • Evolve APIs preserving API contract and binary compatibility

44. Replace, deprecate, and forward • Add replacement API in non-breaking way • Deprecate old API but keep it working • @deprecated This method has been replaced by {@link #bar()}. • For new API added earlier in release cycle • Negotiate a short fuse on deleting deprecated stuff • Make sure clean up happens well before end of release cycle • Schedule API review & polish before too late to fix • Reviews often uncover inconsistencies • For API added in earlier release • Deprecation warnings inform of better way • API contract and binary compatibility preserved

45. Outgoing API changes for a component • Component team works in HEAD stream • Other component teams compile and run against version as of latest (weekly) integration build • Keep HEAD compiling and working cleanly • Do not release changes that would hose other team members • Re-run all unit tests before releasing • Outgoing API changes require coordination with downstream teams • Post “preview version” of component a few days before I-build • Allows downstream components to coordinate their changes • Use nightly builds to reduce risk for a broken integration build

46. Incoming API changes affecting a component • For dependent components, use version as of latest (weekly) integration build • Team members upgrade to latest I-build each week • Laggards slow the team down • Puts premium on everyone making each I-build better than last • Incoming API changes from upstream teams • Receive “preview version” of component a few days before I-build • Allows component to prepare coordinated changes for next I-build

47. Staging large changes to existing APIs • E.g., JDT Core support for new J2SE 5 language features • Work done in stages, over several months • Coordinated with primary client • Parcels of API changes with stub implementations • throw new RuntimeException(“Not implemented yet”); • Allows clients to write and compile code (but not run/test) • Implementation of API parcels done later • Clients can run/test code as soon as API is implemented

48. API-related Resources • How to Use the Eclipse API, by Jim des Riviereshttp://www.eclipse.org/articles/Article-API%20use/eclipse-api-usage-rules.html • Effective Java Programming Language Guide, by Josh Blochhttp://java.sun.com/docs/books/effective/ • Requirements for Writing Java API Specificationshttp://java.sun.com/products/jdk/javadoc/writingapispecs/index.html • How to Write Doc Comments for the Javadoc Toolhttp://java.sun.com/products/jdk/javadoc/writingdoccomments/index.html • Evolving Java-based APIshttp://eclipse.org/eclipse/development/java-api-evolution.html • Contributing to Eclipse, by Erich Gamma and Kent Beckhttp://www.aw-bc.com/catalog/academic/product/0,4096,0321205758,00.html • Internal Tool (reports cross-plug-in references to internals)http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/jdt-core-home/tools/internal/index.html • How to Design a (module) APIhttp://openide.netbeans.org/nonav/tutorial/api-design.html