Maximizing Software Quality: White Box Testing Essentials
E N D
Presentation Transcript
White Box Testing • Agenda • The need • Benefits • Code review • Unit tests and automation • cppUnit Concepts • Memory leaks • Quantification • Coverage • APIs • Challenges • Best Practices • Don’ts • Metrics • Cyclomatic Complexity
Need for White Box Testing • Better to test anything from outside-in as well as inside-out • Most of the functional and performance issues arise due to bad coding • Ultimately the code matters – hence test the code left right and center • Black box tests are used to check cause and effect without getting into internals • White box tests do the same by getting into the internals of every program • Every developer is by default a white box tester • Medical field requires general physicians and surgeons • Certain problems can be identified and solved only by surgeons • A person may look healthy but internally the person may have high bp and sugar; they are not visible externally • So far, the importance of white box testing is not upto the mark • Product companies definitely need white box testing • If your application must scale to a very large extent, white box tests are inevitable
Black Box Cannot Test These • It takes long time to check memory leaks • Any exceptions that happened but unnoticed or unhandled • Which portion of if condition is evaluated • What path the program took to achieve the end result • For every request what is the memory and cpu time taken • Is there any dead and unused code • Is there any extra code that is not needed • What are the potential breaking points in code • Is the code compliant to best practices • In large business apps, the data paths will be too large to test; it will require senior business analysts; if done thru white box mode, normal programmers can bring out great bugs
Diagnostics tools • IBM Rational Purifyplus • JProbe • Parasoft tools • HP Mercury Diagnostics • Dynatrace • .Net profiler • AppPerfect • Netbeans • AQTime • All tools work on instrumentation and trace options • Some tools help in analyzing system memory space as well • Usually the system cpu executions are privileged, and hence we may not get access to those
Business Benefits • Ensures that the core building blocks are intact, when adding new features and modules • Go to market with confidence on product stability • Provides greater visibility in terms of technical and internal compliance of the product • Vulnerability testing always needs a greater amount of white box testing • Resource utilization reduction is key for marketing. Products that use less memory and bandwidth are the most welcome ones • Keeps business leaders informed on the underlying strengths and weaknesses of the application platform
Understand Design • Honestly speaking, how many of us use proper design before coding? • Can the same happen to real estate industry without a proper floor plan or for pharma industry without proper process and formula for drugs? • Design has 4 parts – UI, database, Interface and Logic design • UI design is key aspect when it comes to usability and compliance • Database design ensures scalability and storage • Interface design is key for plug and play during integration • Logic is important for the business flow of the application • Logic design can be a flow chart or text algorithm or pseudo code • Most of the modelers do have pictorial representation of design • Ideally the code must reflect the translation of pseudo code • Traceability of design elements to code is one of the key areas we need to look into • A change in design may impact several parts of code
Code Review • Most of the times, this is ignored or not done to its true spirit • Code review must have design and code side by side • Code review can save upto 30% of testing cost at a later point in time • Usual problems happen due to cut and paste of code where it is not required • When there is a transition from one developer to another, code goes thru challenges • Quick fixes on big projects cause enormous amount of turbulence on production systems • Ideally senior people must review code • Spend at least 10% of coding time on code review
Code Review Checklist • Just try these 10 critical points in daily life • No hard coding • Ensure loop termination conditions • No object creation inside loops • Close every object that you open • Give comments to every code block • Use database connection only when you need • Remove unused variables and code portions • Follow a consistent naming convention • Try to reduce overloading methods very often • Check SQL or file status soon after every disk operation
Unit Testing • The smallest piece of code that can be independently run and tested is a unit • Unit can be a page or a function or a method within a class or a whole program itself • It is the developer’s responsibility to unit test the code • Unit test must focus on data type of every parameter • Focus on data format of every parameter • Focus on boundary values of every parameter • For any given code, what deliverable do we provide as part of unit testing? • Most of the times it is just the trust on the developer • Developer must spend at least 30% of coding time on unit testing alone
How to draw boundaries • What is the line that demarcates unit and other tests? • One expected output out of a function • No chaining • When to stop unit tests? • When a set of parameter combinations are tested, stop the same • Boundaries and equivalence partitions must be tested before exit
Unit Testing Checklist • Are the naming conventions followed? • Are the comments done properly? • Is there any hard coding? • Test case for custom exceptions. • Test case for system exceptions. • Test case for “body” of an if condition. • Test case for “body” of an else condition. • Test case for every loop termination. • Test case for every recursion termination. • Test case for pointers release (for memory leaks). • Test case for every procedure entry and exit. • Test case for every parameter validation for procedures. • Test case for resource release (Closing DB connections, releasing objects etc.)
Unit Test Matrix • A single test must make a call to a function with some parameters combination and expect return value(s) • If a function has got multiple parameters that are of different types, then combinations of all parameters are possible and it will become a Cartesian product table • As the first step, a matrix needs to be prepared with all such combinations • Then the matrix must be optimized to reflect the practicality of the inputs • Possibilities of return paths must also be documented • The samples may go in 100s but not to worry on • Since automated unit tests run extremely fast without human inputs, 1000s of tests can be executed in minutes
Automate Unit Tests • It is not practically possible to give all combinations of the inputs to programs and run • To reduce time, we need to write a program, that calls the program under test and pass parameters • Ideally the test program must be in the same language as the program being tested • This will help in maintaining the homogeneity of the data type of parameters passed • Since developers know the programming language, it is easy for them to write the test program as well • These are called test suites or test scripts • The investment on the unit test automation is one time as long as changes are minimal • The person who automates test for a program must know the program well
The core of unit automation • Class A • Method m1(p1, p2) • Method m2(p3, p4) • TestClass TA • Testcase tm1 • ta = new Class A • rc = ta.m1(4, 5) • If (rc == exp_result) • Pass • Else • Fail • // destroy ta
Level 1 Automation • Crude method – quick solution for short term projects • Write wrapper callers • Hard code parameters • Cut and paste test code with no hesitation • Spend little time on test creation • Test code can be throw-away soon • Simple logs, more manual result analysis required
Level 2 Automation • Parameters to tests must be separated from test code • You can use csv or xml or any format for test data • Useful to run same test on larger test data sets • When a mixture of customers use the product, the data set also will change largely – this will be helpful in that case • Use your own test logs with more details
Level 3 Automation • Use test tools such as cppUnit or jUnit etc. • Build auto recovery options if tests fail in between • Ensure every test has start-run-cleanup structure • Use this on long term test engagements • When tests are to be reusable, this is the best option • If you know cppUnit, the concepts are exactly the same in nUnit, jUnit etc.
Stubs and Drivers • When the program under test needs some pre-built data or state, we need to mimic the same • This is a short-cut approach, but it serves the purpose • Stub is the one that mimics a called function • Driver is the one that mimics a caller function • When we use stubs and drivers, it is a must that we ensure that the data created by them are destroyed soon after the test
Testing APIs • APIs do exchange of data from one piece to another • APIs are usually consumed by many independent programs • APIs in general are meant to provide a window of information to 3rd parties • One change in API will affect all consumers at once • Most of the current day apps provide APIs • We can send tweets from our application using APIs • We can send emails from Amazon SES thru APIs • API calls must be assumed to be context neutral • Some APIs do require authentication mechanism • APIs fall under external interfaces
Wrapper Tools for APIs • A set of business critical APIs are now being used in all EAI scenarios • FIX, SWIFT, News feeds, HL7, HIPAA, Part 11 standards etc • A variety of commercial tools are available in each category • When messages are sent, these tools validate the structures of XML or EDI against the templates • Also, there are different versions of these APIs exist like html 5 etc.
cppUnit • Write test wrappers in cpp for cpp programs • If you are familiar with cpp programming, you may not require cppUnit ! • There are many versions of cppUnit itself as it is open source – many people modified it for their convenience • You can write your own test library and framework • A set of pre-built classes and methods are already made available to minimize test coding time • Testcase, Testsuite, Assert are the important aspects in cppUnit • Testcase is the basic block – every single test • Testsuite contains multiple tests ( a batch) • Assert is the actual verification point where actual result is compared with expected result
cppUnit • Fixture – known set of objects for test • setUp is the place where we create testcases • tearDown is the place where we remove all created tests • Commonly used asserts • Assert • Assert_message • Assert_equal • Assert_throw • Assert_equal_message
Instrumentation • White box tools and profiler tools use one single concept called Instrumentation • Source code is compiled and built as binary • These tools understand the format of these binary files – where function entry points are made, how they exit, where stack is maintained etc. • Before profiling, they instrument the binary – inject their own log codes in the binary locations • During the execution, the instrumented code is run and every detail is logged using these injected code (such as entered function x, executed loop y, exited function y, destroyed object o etc.) • Using these logs, the tools provide internal details • This is very similar to sending a probe inside our body and doing a scan of that probe
Overheads • When profilers run along with the instrumented programs, it adds load to the machine • This is a necessary overhead we need to take • The performance overheads are negligible when compared to actual object size • The statistics that are collected out of the programs will vary from time to time; but if the variance is beyond 5%, then the tool has real issues • Also, do not run 2 competing tools on the same machine. The effects are unpredictable
Memory Leaks • When a memory pointer is declared and initialized and it is not released, it is a leak • Even in managed code sections, the leaks are there for a time till the garbage collector acts • Leaked memory locations are useless and vulnerable • One program leaking 10 bytes of memory per call – can bring down the system in a day when 1000s of users use the same for 100s of transaction • This is very important when we deal with device drivers and embedded systems
Memory Profiling • This happens at runtime • Profiling is the basic block of performance engineering • Concept of instrumentation and tracing happens • Call graph will tell what is the path the program touched in terms of functions • Statistical profilers talk about memory and cpu • Memory consumed at that instance by variables and functions • Memory released at any given point of time
Memory Quantification • What is the consumption of memory and cpu is quantified • Usual quantification happens in basic unit of measurement • UOM can be KB or machine cycles • How many cpu cycles are spent in this method when this is running • How many cpus cycles are spent to initialize the class
Buffer Overflow • When arrays or memory blocks are declared and we try to access beyond those boundaries it is called overflow • When program variable space is packed, the overflows can cause damage to program counters or other stack areas • When stack is compromised, the result is program abort • Dynamic arrays are also vulnerable to overflow when people do not keep track of the max allocations • Buffer overflow can result on network buffers also when the response buffer is underestimated • Protection of buffer overflow is usually made by providing padding space to every memory space – but this needs to be done against memory optimization
Object Issues • Objects require space and the methods need cpu cycles • When objects are declared and not destroyed, they also cause leaks • Unused objects are vulnerable areas for attack • Closure and nullifying objects are essential for optimized use of memory • Creating a series of objects in a multi user environment can cause spiral problems
Function-wise Quantification • Every function has variables and statements • Parameters and variables take space from symbol table as well as stack • Statements do take cpu cycles based on how many times they get executed • When one function calls another, there is a huge amount of context switch happening at heap and program registry levels • How many times these context switches happen will also determine the complexity of the programs
Test case for coverage • A path is defined as a route thru which a specific functionality is achieved by calling a series of steps • If programs flow from top to bottom, it is easy • The more conditions, the more complex it is • Coverage must ultimately make up to 100% of the tested function • Purecoverage will help in achieving this • Covering statements based on design is the best method • But when design is not in sync with program, we need to apply simple logic to hit every statement that is present in a program
Coverage for conditions • a < b && c > d • In the above c>d is evaluated only when a < b comes to true • a < b || c > d • In the above c>d is evaluated only when a < b comes to false • Though part of condition is not evaluated by compiler as part of optimization, they may be later used in the program • Conditions need not be derived based on business case • Most of the time, the conditions are based on parameter values of the functions to determine the path to be taken
Coverage for blocks • This depends on how many conditions we have in the code • Write one test for every relational operation of the condition • Definitely test db, IO and memory exceptions • Write one condition for every catch block • In case the exception cannot be easily triggered, have conditional compiled statements to raise such exceptions • Most of the time, people ignore exceptions thinking that it will be taken care – remember, it will be caught, but how to treat the exception is up to us
Coverage goals • Coverage must have 100% at the end of all tests for every function • There must be 1 test for every relational operation in conditions • There must be 1 test for every loop termination • There must be 1 test for every exception • There must be 1 test for every test exit point (if method has multiple return points) • There must be one test for successful sql operation • There must be one test for unsuccessful sql operation
Metrics Collection • Average number of unit test cases per page/component/any specific unit in the project • Total number of unit test cases • Total number of unit test cases failed in test pass-1 • Total number of unit test cases failed in test pass-2 • Total number of unit test cases failed in test pass-n • Total number of memory leaks found • Percentage code coverage as per the tool statistics • Execution time for each test run • Total test execution time for all units • Ratio of # of defects found in unit testing to # of defects found in code review • Defect distribution across units (asp related, db related, parameter related etc.)
Metrics Analysis • Trend analysis helps in seeing whether the error rate moves up or down • See pattern of failures and find out the corrective action • Metrics vary a lot from developer to developer; hence find out error rate pattern across developers • Analyze failure pattern and time of failure • It is not easy to catch all errors in one round of testing; hence do more rounds; more often
Configuration Management • Since we develop test code, we need to check in these code sections to VSS or subversion or PVCS • Maintain a uniform x.y version code to all test cases and suites • Check in the test data along with test code as well • Have one single person who can oversee all the CM operations • Never allow people to have local copies of test code
Best Practices • Keep adding more tests every day • More data preparation is the key • Test often - even twice a day • Never ignore warnings from IDEs • Run unit tests on a clean test bed • Review unit test code as well • Run regression unit tests daily • Run unit tests from other developer’s machine
Best Practices • Run unit tests by a different developer • Make unit test code simple • Never try to have if then else in a test code • Do not have loops in a test code • Run critical tests first • Run all unit tests in a single session; never stagger
Don’ts • Do not test entire functionality in one test • One test must satisfy just one condition • Never underestimate any parameter combination • When skype or sony programs can come down just by 1 bug, our program too can come down • If certain validations are performed in other layers, do not do in unit tests • Never run tests on a test machine without clean up
Don’ts • Never miss the null value test for any parameter • Never miss any negative value tests for database connection or query tests • Never miss trying to access an object or record that does not exist, because these are the places where exceptions get triggered • Do not miss to pass a wrong xml structure to web services
Challenges • White box testers must be developers • Hence companies use them to do core development work and ignore writing white box tests • White box tests need tools and sometimes they are a little expensive; hence companies avoid this route • Main challenge comes in not documenting design. Since design is the basis, and if that is not present or incomplete, white box tests do not yield good results • Project planning never considers the time for white box testing; especially in service companies, this is out of scope! • When changes are made to the programs being tested, the changes are not documented well and hence unit tests go out of gear
Challenges • When same developer runs the unit tests, there is a chance to ignore issues – so have developer A testing the code of developer B • Initial unit test automation code takes some time; hence do not expect the ROI on day 1 • Results can be seen only after one or 2 quarters. Many management teams do not have patience to wait for such a long time • People try to implement logic in different layers that they are designated for; instead of coding some logic in Java, they code in stored procedure and call that from java – this makes profiling and coverage difficult
Cyclomatic Complexity • This is to arrive at the testing complexity of code • Nodes, edges are the primary factors • Higher the complexity, higher the number of tests we need • This is based on graph theory • Many physical problems can be solved by graph theory • e.g. traveling salesman theory • M = E - N + 2P • E - number of edges (lines) • N - number of nodes (rounds) • P is the connected components (start and destination points) • The more if conditions and branches, the nodes and edges increase • Call graph of all methods will show the independent connected components
Business Flow Tests • When we do white box tests, we test individual components • Hence it comes to unit tests • Divide flow into multiple components • Test them one by one • Aim at one transaction at a time • Club white box tools with functional test tool • Since business transactions need database setup, ensure you have test stubs that pre-populate records in tables
How to speed-up • Let a senior create a sample test for every function • Let a senior prepare unit test matrix • For a few tests, senior and junior developers must do pair test automation • Then junior takes care of rest of the unit tests for automation • Senior reviews the unit test code • Lead runs all tests every day
Get ready to take blame • Unit tests are code; hence they may also have bugs. Developers may start blaming the tests themselves • Tests may be redundant if not properly reviewed • Sequence of unit tests must be carefully done. Always use bottom up approach on unit tests – test the smallest one first • Run unit tests soon after build is released, and before BVT starts by the black box team
Never Give Up • Teams give up white box tests after a quarter or so • This is because, they do not see visible returns • Remember – these things fail is a feel-good factor on testing; at the same time, these things work is more important as well • Most of the times, great systems are brought down due to missing white box tests and not black box functionality • It may take years to find out, one great bug – but it is essential for big time products • If you are in product company, be patient and white box will pay back
