AWIPS Continuous Technology Refresh (CTR) AWIPS Software CTR

1. AWIPS Continuous Technology Refresh (CTR)AWIPS Software CTR T010 TIM Nov. 5, 2008

2. TIM Topics • CLI update (Mark Fegan) – • - textdb work in T010 delivery, Python • - uEngine script runner code complete, script management in progress • Subscription update (trigger equivalent functionality) • Design in progress • Smart Tools "architecture" in AWIPS II • Smart Tool migration discussion      - Things that need to be done            - Approach going forward • Q&A - open discussion

3. Number Description CLTS 1 The AWIPS II Command Line Tool Suite shall preserve the command line interface of the legacy textdb tool CLTS 2 The AWIPS II Command Line Tool Suite shall preserve the command line interface of the legacy handleOUP tool CLTS 3 The AWIPS II Command Line Tool Suite shall include a command line interface to the µEngine script runners Command Line Tool Suite Requirements

4. Command Line Tool Suite Design Concept • This diagram shows the basic interfaces shared by the CLI Suite tools • the user runs the CLI tool from the command line • data is provided via standard input • results are returned via standard output • The CLI tool communicates the request to the appropriate EDEX/Mule service endpoint via HTTP • The service endpoint performs the required actions and returns the result to the CLI tool

5. Command Line Tool Suite Use Case • User enters a command line including operation flags and operands • Tool decodes the command line • Tool reads data from standard input • Tool converts command line and data to Canonical XML message • Tool sends message to EDEX SOA Endpoint • Tool receives response from EDEX SOA Endpoint • Tool decodes message • Tool writes contents of message to standard output • Tool returns exit status to User The User utilizes a Command Line Interface Tool to submit a request to the EDEX Server

6. Command Line Tool Suite Sequence Diagrams • This diagram illustrates the execution basic sequence used by the CLI tools • This execution sequence leads to a set of components/classes that are shared by the various tools

7. Command Line Tool Suite Shared Components • This diagram illustrates the expected component design of a CLI Tool • The CLI library will be a class library providing common functionality • Message: handles conversions to and from XML • read/write provides façade for all I/O operations • CL Parser provides standard command line parser • Comm Link isolates CLI Tool ↔ EDEX SOA Endpoint communications

8. Command Line Tool Shared Library This is the class diagram for the library shared by all components of the CLI Tools suite. It can be available for other modules as well.

9. uEngine CLI Tool Design Concept • This diagram shows the basic interactions for the uEngine tool. • The user • provides commands via command line • provides data (scripts) via STDIN • receives responses via STDOUT • uEngine communicates via HTTP • execution requests are sent to ProductSrv • subscription requests are sent to SubscriptionSrv Standard CLIS interface Proposed subscription endpoint

10. uEngine CLI Tool Design Concept The basic idea is to provide a command line interface for the uEngine that has functionality similar to that provided by the textdb CLI Tool. That tool allows clients, either software or human, to perform standard database tasks; inserts, retrieval, certain table updates, via a simple command structure. The uEngine CLI will allow the user to perform two basic uEngine related functions: 1) Execution of a uEngine script with the results returned 2) Routine maintenance of product subscriptions Use Cases for these two main scenarios are covered on the next few slides.

11. uEngine CLI Tool Use Cases • This diagram illustrates the basic Use Cases for the uEngine CLI Tool • There are two main actors • User: runs uEngine • EDEX: the EDEX server • There are two main Use Cases • Run Script: User uses uEngine to execute a uEngine script • Manage Subscription: User uses uEngine to manage a subscription

12. uEngine CLI Tool Use Case 1 Run Script • User executes the uEngine run script command • uEngine decodes the command line arguments • uEngine reads the script from STDIN • uEngine converts the script into a canonical message • uEngine sends the message to EDEX for processing • EDEX processes the message and returns the result in canonical XML format • uEngine decodes the return message • uEngine writes the result to STDOUT • uEngine returns status code to User The User executes uEngine to submit a script to EDEX. EDEX executes the Scripts and returns the result to User.

13. uEngine CLI Tool Use Case 1

14. uEngine CLI Tool Use Case 2 Manage Subscription • User executes the uEngine manage subscription command • uEngine decodes the command line arguments • uEngine converts command line into a canonical message • uEngine sends the message to EDEX for processing • EDEX processes the message and returns the result in canonical XML format • uEngine decodes the return message • uEngine writes the result to STDOUT • uEngine returns status code to User User executes uEngine to submit a subscription request to EDEX. EDEX executes the request and returns the result to User. Note: This scenario assumes no input from STDIN. One variation would have script contents input from STDIN or a file.

15. uEngine CLI Tool Use Case 2

16. uEngine CLI Tool Use Case 2 • Use case 2, and its variations, depends on implementation of a subscription management endpoint in EDEX • Initially, this use case will be stubbed in in the uEngine CLI Tool

17. uEngine CLI Tool Interface • The basic interface is a combination of command line arguments and standard input • see typical command line (below) • This table describes the command line flags • Only the highlighted options will be initially implemented • When required, the uEngine script is read from standard input • file redirection may also be used • Results are written to standard output; errors are written to standard error • Process return values: • 0 for successful execution • 1 if any error occurs uEngine -r python -s MESSAGE "Hello from Omaha" < data/HelloWorld.txt

18. Subscription Service Design Concept • This diagram shows the existing subscription service • The Subscription Manager is a JVM based singleton • Script management is handled through ProductSrv • Subscriptions are limited to Rhino/JavaScript scripts • Script fulfillment is handled by NotifySrv and AutoBldSrv • scripts are executed based on arrival of a data product • subscription notifications are sent to specific, registered, clients • Scripts generally create products in a standard image format.

19. Subscription Service Design Concept • This CONOPS for subscription was based on the need for CAVE to automatically update the display as data was ingested on EDEX • This proved inadequate for the auto update capability required by CAVE • This CONOPS was replaced in TO6 with CAVE’s auto update facility which is based on data ingest w/o utilizing product creation scripts

20. Subscription Service Design Concept • This design, particularly the (JVM based) singleton implementation of the Subscription Manager) had certain notable problems • Product subscriptions tended to fail on clustered installations about 50% of the time • Orphaned subscriptions exits since CAVE does not cancel subscriptions on exit

21. Subscription Service Design Concept • This diagram shows the revised design of the subscription management functionality • the Subscription Manager is merged into NotifySrv and renamed SubscriptSrv • an EDEX client interacts directly with SubscriptSrv to manage subscriptions • SubscriptSrv interacts with the data store and passes messages to the script runner (AutoBldSrv) • AutoBldSrv will be refactored to allow multiple script runners as well as product and timer based execution

22. Subscription Service Design Concept Subscription Fulfillment • AutoBldSrv is triggered by system • AutoBldSrv obtains subscription list from Subscription Manager • Subscription Manager obtains subscription list from database • AutoBldSrv obtains script runner from uEngine factory • AutoBldSrv triggers the script and returns result to the system 4 3 1 2

23. Subscription Service Design Concept Script Management • EDEX client creates a subscription management request • Subscription request is passed via HTTP to the subscription manager • Subscription manager accesses /updates the subscription tables • Subscription manager prepares and returns a response to the EDEX client 1 2 4 3

24. Subscription Service Design Concept Additional Considerations • Subscription management will be provided via the uEngine CLI tool • provides a simple command line interface • may be supplemented via Python/TK or another GUI interface • Initially, we expect to provide 3 uEngine script runners • a Python script runner based on the existing Python uEngine • a JavaScript script runner based on the existing JavaScript uEngine • a system script runner based roughly on the adapter service • the idea is to allow running external scripts as if from command line

25. Smart Tool Architecture Layers • JEP – Java Embedded Python • Modified for improved performance and numpy support • PythonScript – Java class that wraps JEP calls • SmartToolController (new in TO10)‏ • Java class that extends PythonScript • Manages smart tools in a single python interpreter • SmartToolInterface – python class that assists the SmartToolController with managing and running smart tools • Tool – Java class ported from Tool.py that prepares and executes smart tools through the SmartToolController

26. AWIPS-II Smart Tool Architecture Overview <Python> SmartScript <Python> BaseTool Python wrapper functions (95) Python utility functions (29) <Java> PythonScript <JEPP> Java to cPython Bridge + numPy datatypes cPython + Module libs AWIPS-II interface To Python by Jepp <Java> SmartToolController <Java> SmartInitScript <Java> ProcedureScript <Java> FormatterScript Runs GFE SmartTool python Runs GFE SmartInit python Runs GFE Procedure python Runs text formatter python <Java> SmartUtil +runTool() <Java> DataManager <Java> WEGroupManager +5 functions Central Singleton <Java> IFPClient <Java> ParmManager <Java> ReferenceSetManager <Java> GFESpatialDisplay.. <Java> SampleSetManager +28 functions +18 functions +22 functions +24 functions +40 functions 137 Functions Implemented In Core GFE Code

27. Smart Tool Migration • Imports from Numeric must be changed to import from numpy • Wx tools must return a tuple of (numpy byte array, keys)‏ • It appears that with numpy operations sometimes the array comes back as an array of type float32 (e.g. [0, 1, 0] could come back as [0.0, 1.0, 0.0])‏ • If this occurs, at the bottom of the tool just ensure the wx is still of type byte • wxValues = asarray(wxValues, 'int8')‏ • Any calls on the C objects that did not go through SmartScript will have to be updated to call the correct Java methods

28. Analysis of RNK tools • RNK delivered Omaha 173 smart tools the other week • Analysis (all estimates)‏ • 74 should work with TO9 with minor changes (previous slide)‏ • 36 more should work with TO10 delivery with minor changes • 20 obsolete - Adjust tool makes others obsolete, some are point-based or parm-based tools • 30 incompatible – ISC, hardcoded directory paths, data type not currently supported in AWIPS II • Others unsure on compatibility – e.g. tkinter