A novel approach to automatic extraction of configuration parameters from industrial code

1. A novel approach to automatic extraction of configuration parameters from industrial code Ian Davis, Ric Holt †, Ron Mraz‡ † ‡

2. OWL Computing Technologies Inc.( www.owlcti.com ) • Provides solutions that: • Facilitate ultra-secure unidirectional data transfer across the internet (very secure one-way firewall) • Potential Customers: • Governments • Military and Intelligence • Power plants • Water management, etc. Fact Extraction from Bash in Support of Script Migration

3. The Idea • Two Unix machines in a single box • Write only network card upstream • Read only network card downstream • Internal cable connects two network cards • 10Mb/sec and 155Mb/sec (fiber optics) supported • External connections for: • Incoming and outgoing network cable • Local access to each Unix machine • USB Sockets for each Unix machine Fact Extraction from Bash in Support of Script Migration

4. The Product Line For Personnel For Vehicles Fact Extraction from Bash in Support of Script Migration

5. An Application Fact Extraction from Bash in Support of Script Migration

6. The Unix Software • Supports: • Transparent unidirectional TCP/UDP firewall • SMTP email systems and other protocols • Automatic file mirroring to/from secure machine • All major historian/tracking/logging systems • Bash scripts: • Facilitate software installation • Initiate client/server and other executables • Monitor real time behaviour of system • Stop/restart services, etc Fact Extraction from Bash in Support of Script Migration

7. The Engineering Problem • Customers requirements vary • They must change configuration variable values • This results in Bash scripts being cloned • Why is this problematic • Customers have to understand Bash • Customers risk breaking their scripts • Repairing broken scripts very costly • Owl must support many variants of scripts • Hard to switch from Bash to better languages Fact Extraction from Bash in Support of Script Migration

8. The Issues with Bash • Problems • Antiquated (limited support) • Cryptic language (people dislike it) • Slow (Python can be 6 to 21 times faster) • Memory leaks and other bugs (software failure) • To configure software scripts must be edited • Solution • Refactor all scripts to use external XML files • Port Bash to Python/Ruby etc. Fact Extraction from Bash in Support of Script Migration

9. Benefits of XML configuration file • Supports script migration • Massive consolidation of scripts • Releasing updates to scripts much easier • End users do not need to understand scripts • Intrinsic benefits: • Users know exactly what they can change • They don’t risk breaking the scripts • Configurations documented in the XML files • Can easily compare configurations • XML files can be shared across scripts Fact Extraction from Bash in Support of Script Migration

10. Our Challenge • Mission impossible: • Change ~1,000 scripts to use XML configuration • The largest script ~2,000 lines long • Scripts may not be copied off site • You are only on site for a week • We have no clue what the scripts do: • Don’t understand the contents of the scripts • Don’t know what parameterizes the scripts • Don’t know what parameters users may change • No obvious way of reading an XML file in Bash Fact Extraction from Bash in Support of Script Migration

11. The First Hour • Refactoring task • Manually examine first script • Record how variables are used • Guess which variables customer’s change • Somehow change the script to read XML • Unworkable solution • Very tedious work • Requires constant attention to detail • Task enormous • Deliverable will be far from optimal Fact Extraction from Bash in Support of Script Migration

12. Prowl(Parameterizing Owl) Variable Properties XML FILE BASH SCRIPT Fact Extraction from Bash in Support of Script Migration

13. The Solution • Hack the Bash 4.2 interpreter • Open source C code is easily changed • Continue to source scripts etc • Change interpreter to “action” everything • Remove ability to invoke external programs • Augment variables with usage flags • Output XML file when interpreter finishes Fact Extraction from Bash in Support of Script Migration

14. Dynamic Fact Extraction • y=2; • if (…test…) • then • return • fi • case $x in • …;break;; • 2) …code… • esac x=2; if (…test…) then …statements… else …statements… fi function($x,$y) Fact Extraction from Bash in Support of Script Migration

15. Example XML output • y=10; x=1; y=2; z=$x+$y; exit $(($z)) <prowl> <script1 version="default"> <default> <x>1</x> <y changed>2</y> <z derived returned>1+2</z> </default> </script1> </prowl> Fact Extraction from Bash in Support of Script Migration

16. XML Configuration File(Manually edited) <prowl> <script1 version="default"> <default> <x>10</x> <y>20</y> </default> </script1> </prowl> Look at: • Naming conventions • Documentation • Examine variable usage • Compare cloned scripts • What customers change Fact Extraction from Bash in Support of Script Migration

17. Prowler XML FILE Variable Assignments BASH SCRIPT Fact Extraction from Bash in Support of Script Migration

18. Reading Configuration File • Small C program parses the XML file • Optional script name • Optional version name • Optional external assignments (x=21 y=3) • Prowler outputs: • Bash variable assignment statements • x="10";y="20"; • This string then eval’d within the script Fact Extraction from Bash in Support of Script Migration

19. Just One Extra Line/Script • But where to place this one line ? • To early: • Configuration values later get changed • To late: • Configuration values are used before set • Just right • How to tell ? Fact Extraction from Bash in Support of Script Migration

20. Validate Placement • Run prowl on the refactored Bash script • Execute prowler (and only prowler) • Complain about any variable named by prowler already flagged as USED. • Prowler invoked too late • Complain if any variable named by prowler is later assigned a value. • Prowler invoked too early Fact Extraction from Bash in Support of Script Migration

21. Contributions • Automated analysis of all Bash scripts • Automated construction of XML files • One line solution for parsing XML files • Ability to validate changes applied • Removed a significant road block for OWL • Existing customers were asking for this • Meets new US purchasing requirements Fact Extraction from Bash in Support of Script Migration

22. What Was Learned • Think outside the box • Necessity is the mother of invention • Source code is a valuable resource • It can be exploited in novel ways • Tools that make a good worker • Don’t refactor if a tool can do the work for you • Getting it right saves a lot of time • Getting it wrong isn’t an option Fact Extraction from Bash in Support of Script Migration

23. Conclusions • Software implemented in a week • Only minor changes to Bash C source files • Available as open source software • http://www.swag.uwaterloo.ca/prowl • http://cs.uwaterloo.ca/~ijdavis (CSMR paper) • Easy to port to other Bash environments • [Almost] as robust as Bash • OWL thrilled • Customers happy Fact Extraction from Bash in Support of Script Migration

24. Thank You That is our novel approach to automatic extraction of configuration parameters from industrial code Fact Extraction from Bash in Support of Script Migration