1. Discussion on Testing Simantics Technical Board Meeting 30.1.2012

2. Organizing Test Suites • Suggestion: new plug-in org.simantics.tests • Contains test suites for different testing activities: • Regression • TDD • Performance • Stress • Composes test suites for the SDK from around the codebase, doesn’t contain test cases • Most suites are automatically tested via Jenkins

3. Testing Activities • Regression • First step is to move all currently existing and passing tests into regression tests • TDD • Development-time, moved into regression when development is complete, manually executed • Performance/Scalability • Focus on performance-critical parts of the SDK right now: databoard, DB, graphics, history • Stress • For uncovering hidden unstabilities, mainly relevant for DB

4. Tools to Incorporate • Squish (finally) • EMMA (code coverage for unit tests) • Possibly static analysis tools

5. Static Analysis • Several (free) tools available for Java [1] that have Eclipse & Jenkins plug-ins : • PMD, FindBugs, Checkstyle • AFAIK, none of us has been actively using any of these with Simantics • Relatively cheap to automate • Need to test the tools first to say anything definitive of usefulness

6. What to Test? • org.simantics.tests: everything that is part of the platform – not products built on top of it (sysdyn, modelica, etc.) • A separate test suite for every product • UI testing • Difficult without a Workbench product • Use movie tutorial ? • Perhaps create the planned diagram editor for it ?

7. Reporting • Jenkins provides unit/squish test reports • Performance testing needs different logging • Instrument all tested code with necessary performance logging (See Eclipse tracing facility) • See about using Apache JMeter (Jenkins plug-in) • For stress tests it is important to have proper logging enabled and relevant data kept for failed tests (databases, etc.)

8. Epics (prioritized) • Create and automate org.simantics.tests • Sysdyn product test suite with UI tests • List most important testing topics • Coverage/Static analysis tools into use • Performance testing plan • How to measure? Relativity of results? Reporting/tools? • Prevent publishing of builds with broken tests & ensure email is delivered about breakage • Try to force developers to react to acute problems

9. Working with tests and failure • Regression tests divided into three suites: • 1st: only intended to contain tests that work. It is important to know when something breaks, and it is easiest to discover this when builds turn from green to red/yellow or vice versa. • 2nd: broken tests that have been moved from 1st if fixing the bug that caused the regression is not possible immediately. • 3rd: broken tests that have no incoming fix for a long time. 2nd->3rd moves done only at the turns of sprints.

10. References • [1] http://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis#Java