Why is API Testing Different?

1. Nikhil Nilakantan Hewlett Packard Corp. (nikhil_nilakantan@hp.com) Ibrahim K. El-Far Florida Institute of Technology (ielfar@acm.org) Why is API Testing Different?

2. Introduction APIs Are Different APIs Are Complex Testing APIs Three Nuts to Crack Additional Issues Exploring an API Testing an API Automation Automation Demo Future Work & Conclusion Q & A Agenda

3. Introduction • Many styles of testing • Many types of applications • APIs constitute a large part of these applications • No Silver Bullets

4. APIs Are Different • Generally misunderstood • Different for many reasons: • Invisible to the human user • Require a knowledge of inner workings • Require considerable programming skills • Tend to be high in complexity

5. APIs Are Complex • The Word Object 9.0 Model contains: • Around 200 classes • Over 200 enumerated types • Most classes have at least 10 methods • The Word 9.0 Document Class contains: • 125 fields, 3 events & 63 methods Not counting the parameters or events for each method we have 125 X 63 = 7875 combinations • Many parameters are complex types or class objects

6. Testing APIs • Testing involves designing of sequences to satisfy test objectives • Requires isolation of specific parameters • Requires a mechanism to evaluate return values How would you test a call with 31 parameters? Function CreateLetterContent (DateFormat As String, IncludeHeaderFooter As Boolean, PageDesign As String, LetterStyle As WdLetterStyle, Letterhead As Boolean, LetterheadLocation As WdLetterheadLocation, LetterheadSize As Single, RecipientName As String, RecipientAddress As String, Salutation As String, SalutationType As WdSalutationType, RecipientReference As String, MailingInstructions As String, AttentionLine As String, Subject As String, CCList As String, ReturnAddress As String, SenderName As String, Closing As String, SenderCompany As String, SenderJobTitle As String, SenderInitials As String, EnclosureNumber As Long, [InfoBlock], [RecipientCode], [RecipientGender], [ReturnAddressShortForm], [SenderCity], [SenderCode], [SenderGender], [SenderReference]) As LetterContent

7. Three Nuts to Crack • Parameter selection • Selecting “interesting” values • Exercising boundary conditions • Parameter combination • Exercising stored data & computation • Separately legal values maybe illegal when used together • Call sequencing • Almost impossible to test all combinations

8. Additional Issues • Inadequate domain knowledge • Poor documentation • Unavailability of source code • Time constraints

9. Exploring an API • Things to do BEFORE you begin testing: • Review documentation • Map the interface • Review source code • Isolate (and deliver) the following artifacts: • Common calls, parameters • Valid/Invalid parameters & return values • Realistic usage scenarios • Utilized resources

10. Testing The API • When is automation desirable? • Do you have enough information about the system under test? • How much effort is required to write the automation? • How expensive is the automation to maintain? • How effective is the automation going to be? • Kinds of automation • Capture-Replay automation • Monkeys • Intelligent test automation

11. Building Automation Good automation should be: • Modular • Flexible (easy to change) • Scalable (easy to extend) • Understandable

12. Automation Too Expensive? If automation is too expensive – Go back to basics! • Assess risky API calls, parameters & values • Perform boundary analysis on parameters, values • Stress boundary conditions • Use combination tests

13. Automation Demo

14. Conclusions • APIs are different • APIs are complex • API testing is NOT an easy problem • Work on API testing is in its fledgling stages

15. Future Work • Defining API test adequacy criteria • Using model-based approach(es) to resolve: • Parameter selection problem • Parameter combination problem • Automated API exploration tools • Comparing effectiveness of testing methodologies with respect to APIs

16. Questions?Comments? Suggestions? Nikhil Nilakantan nikhil_nilakantan@hp.com Ibrahim K. El-Far ielfar@acm.org