1 / 47

AWIPS Continuous Technology Refresh (CTR) T04 Capability Additions to the ADE Architecture

AWIPS Continuous Technology Refresh (CTR) T04 Capability Additions to the ADE Architecture. T04 Capability Concept Descriptions January 2006. Presentation Introduction Focus on new capability T04 added. Top Level Architecture Review CAVE product request flow architecture description

kass
Télécharger la présentation

AWIPS Continuous Technology Refresh (CTR) T04 Capability Additions to the ADE Architecture

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. AWIPS Continuous Technology Refresh (CTR)T04 Capability Additions to the ADE Architecture T04 Capability Concept Descriptions January 2006

  2. Presentation IntroductionFocus on new capability T04 added • Top Level Architecture Review • CAVE product request flow architecture description • Data ingest architecture description • Regression Test Environment Delivered in T04 • Function Additions • Volume Browser with successive disclosure of plot data • Spatial Index and Query Capability + GIS analysis • Product Dissemination • Math Scripting • SBN Radar Ingest • CAVE Drawing and Collaboration • CAVE XY Rendering {Skew-T, Hydro Graph, Initial GFE} • Objective Analysis Pattern {Barnes Single Pass} • Preliminary Conceptual Security Architecture

  3. T04 DVD Delivery Artifacts All the fixed map files and demo data sets in CAVE menus Source in Eclipse Project Structure Linux/Windows Cave Install All the step by step test procedures For all the entries in the RTM Run Time Execution Dir Structure + dependencies Server Side Source in Eclipse Project Structure Complete Regression Test Data Set Time Correlated Metadata Tool to view all the metadata fields and values

  4. Architecture ReviewAWIPS Term Definitions • (SOA) Service Orientated Architecture: concept of system functionality available to clients through network endpoints, usually includes a wide set of end points, I/O Routing, and simple Transforms • State Less SOA Service: the condition where a service is independent of the order of the requests • Design Pattern: describes a problem which occurs over and over and provides the core of the solution to the problem that is reusable over a wide range of situations • Container: is an operating system process that holds the execution objects and provides communication interfaces • Layered Architecture: an approach where the path from client to data is organized into precise layers with defined api’s between layers • MetaData: is data about the ingested data objects constructed to facilitate unique data retrieval

  5. Architecture ReviewAWIPS Definitions Continued • ADE: AWIPS Development Environment, source code to execution framework enterprise development kit including tools • Canonical XML: Well formed XML that follows high level rules, in AWIPS it is dynamically extensible • Technical Reference Architecture: A physical software execution framework • JMS, JMX: Java Messaging System (API), Java Management Extensions • CAVE: Common AWIPS Visualization Environment • SEDA: Serial Event Driven Architecture used for scaling up services and clustering boxes

  6. SOA Framework Concept Review Extensible Reference Architecture – Minimize Coupling Remote Management (JMX) Core Base of Services • Micro Engine • Plug-in Framework • Extensible XML model • Core libraries <<Technical Reference Architecture>> Services +Ingest Data +Request +Index Data +Store Data +Transform +Auto Build Extend to a specific domain • Plug in specific libraries • Plug in data types, transforms +Subscribe +Collaborate <<Technical Reference Architecture>> SOA System Services Satellite Feed Data Types Meta Data Transforms +domain libraries Data Scripts Local Data +mapping libraries Service Interface To Data • Clean separation between data and visualization • Canonical XML data model • Scriptable Interface <<Visualization Framework>> CAVE Vis Plugin 1 +core plug-ins Vis Plugin 2 + libraries Vis Plugin n

  7. Conceptual Architecture Review: Logical Layered Viewpoint Client/Presentation Services • Thin Client Browser Shared Operational Picture CAVE Management Console • Controls/Status Displays Client Applications • Web Applications • • Alerts Native Services Visualization/HMI Infrastructure: Identity Visualization Services Collaboration/Drawing • Network Services • Information Assurance • Workflow Wx Mission Services External Programs ProductSrv AutoBldSrv IngestSrv NotifySrv AdapterSrv PersistSrv IndexSrv GridEditSrv ColaborateSrv AdaptertSrv SubscribeSrv Communications/Networking Services Security Services /Demilitarized Zone (DMZ) Hydro Models Integration Framework LAPS Platform/Infrastructure Services Application Services: • Messaging • Notification • Transformation • Routing • Security System Services: • Operating System • System Management • Access Services Data Services: • Metadata index • Spatial index • Metadata query • Data store • Data access FORTRAN/C/C++ Command Line Programs Computing Infrastructure Platform/Infrastructure Resources Spatial Index Metadata Index Data Persistence Store Data Links

  8. AWIPS ADE High Level System Services –ADE SOA Services as of T04 WAN Visible ADE 1.0 Services JMX Remote Service Management Client Delivered T03 Delivered T04 Delivered T05 Delivered T0{M} Mbean Mbean Mbean Mbean Mbean IngestSrv PersistSrv IndexSrv ProductSrv AdapterSrv CAVE Visualization Client ASCIIIngestSrv FileSystem Lucene – Reverse Idx uEngine Scripts Exec Adapter Data Rendering GridIngestSrv RDBMS via JDBC Spatial R-Tree Grid Diagnostics JNI Adapter CAVE Bundles SBNSatIngestSrv HDF5 persistence Spatial Data Base Dissemination Data Interrogation RadarSrv CAVE Procedures Radar Level II Wx Drawing ProductSrv Wx Warning ProductSrv Radar All Tilts Hydro Visualization Mbean Mbean Mbean Mbean NotifySrv SubscribeSrv AutoBldSrv Collaboration GridEditSrv uEngine Scripts SubscriptionNotify XMPP AlertNotify Enterprise Service Bus – HTTP, JMS, Virtual Memory, File Endpoints Enterprise Service Bus – HTTP, JMS, Virtual Memory, File Endpoints Services Independent of End Points

  9. CAVE Layered View of SOA Service with Data FlowCave Requests Data for Display as a GIS Layer Client/Presentation Layer Native Services Layer ProductSrv Platform/Infrastructure Services Layer Input Message Canonical XML With body containing action commands uEngine Pattern Enterprise Service Bus – HTTP,JMS, VM,… Endpoints termQuery fileIn makeResponse Output Message Canonical XML Response to Client Containing URI references to data generated Data Access Objects Platform/Infrastructure Resources Layer Persistance Respository MetaData Index

  10. Sequence Diagram – Product RequestProductSrv Uses uEngine to get at data Client ProductSrv uEngine launcher Data Respository uEngine Task launch termQuery execute Request CAVE MetaData Index search parseAndRun fileIn Persistance Respository read makeResponse Response

  11. Legend Class Inheritance Class association CAVE Singleton Class <<abstract class>> Task Data flow + Public class method <<Java Class>> Action <<Java Class>> Launcher { } Code Block +addRules() +init() +execute() +parseAndRun() +executeTasks() AbstractMessageSrv <<Java Class>> TermQueryIndex +onCall() +register() +addRules() +init() +execute() <<Java Class>> IndexSearch +getInstance() ProductSrv – High Level Class Structure <<ESB Service>> ProductSrv +receive() {String output = (String) myLaunch.parseAndRun(msg);} Commons Digester {action = DigesterSingleton.parse(reader); action.executeTasks();} lucene {Task task = i.next(); task.init(retObj); retObj = task.execute();}

  12. Layered View of SOA Service with Data FlowIngest of SBN Satellite Data Native Services Layer PersistSrv NotifySrv File Endpoint IndexSrv SBNsatIngestSrv Input Message Byte[ ] lucene Platform/Infrastructure Services Layer Plugin Pattern Enterprise Service Bus – HTTP,JMS, VM,… Endpoints Satellite Imagery Writer Filename Decode Extract MetaData Data Access Objects Platform/Infrastructure Resources Layer Persistence Repository MetaData Index

  13. Sequence Diagram – Satellite IngestExample Ingest Sequence Sat End Point SBNsatIngestSrv PersistSrv NotifySrv IndexSrv PluginFactory File Event Extract Metadata PluginFactory store write Persistance Respository write MetaData Index Metadata Metadata getDataURI createNotifyMessage send

  14. CAVE Conceptual Data Model DesignI/O Formats Follows Existing Standards • Output Formats • Vector: ERSI shape file, SVG • Raster: GeoTIFF,png,jpg • Text: Canonical XML, WMO Bulletins… • VTEC warnings • Input Formats • grib1/2 • GINI • ASCII (WMO, shef, …) • Radar Level II Data Flow Internal Object Model • Request Formats • Canonical XML • Decoded • Persistance • Respository • Flat Files • HDF5 • RDBMS • Ingested • Persistance • Respository • Flat Files • HDF5 • Metadata • Index • Lucene • RDBMS • Raw Storage Model • Transmission Formats • HDF5 model • Decoded Storage Model • RDBMS Schema • HDF5 model • Static data model • MetaData Model • RDBMS Schema • Lucene fields

  15. Internal Object Model Conceptual Data Model DesignCanonical XML: Dynamically Extensible Plugin Decode Input Formats JIBX builds XML response from object model, very high performance (eliminates Object to Schema Impedance mismatch) AF3… Plugin Dynamically Extends Canonical XML by adding Task segments with XML rules JIBX Dynamic XML • Plugin Pattern • Extract Metadata • Decoder • uEngine Task • OutputFormats • Response XML MSG Rule driven parser enables dynamic extension of Action tags in request XML DOM Parser Commons_ Disgestor • Request Formats • Input XML Action MSG Task Task Task

  16. Regression Test: Driven by RTMRequirements Traceability Matrix (Sample entries)

  17. Regression Test EnvironmentWeb Interface running in Tomcat Server Purpose: Simple Light Interface to SOA Services to facilate continous testing during development Performs Catalog Queue to Refresh Menus Allows viewing editing Canonical XML request message / response Copies Regression Test sets to ingest endpoints

  18. Regression Test Structure For Services Independent of CAVE Test Machine (laptop/workstation) Linux / Windows Test Client Web Interface Tomcat Web Server activeMQ JMS Server uEngineWeb.war EDEX Services Libraries Plugins Mule ESB ESB Ingest File Endpoints Regression Test Data Sets

  19. Volume Browser Data Request CapabilityInitial Manual Mechanism to retrieve data • Dynamically Populates Select Boxes Through Catalog Queries • Submits Request to ProductSrv to Allow Light Client Operation Mule ESB Service Container Log Catalog Query ~ 0.6sec Grid, Observations, Radar, Satellite Canonical XML Message Request a Catalog of Metars Note ProductSrv Scaling

  20. Volume Browser Retrieving Point DataSuccessive Disclosure – Two Mechanisms Note: SVG prototype plot model. Full plot model later. D2D Style Disclosure Pan Disclosure Panning dynamically brings in more data

  21. Volume Browser Retrieving Gridded DataTransformed Into Image or Vector Field Dynamic Selection Labels from Data Source Color Map Selection Contour | Image Selection

  22. shapeQuery Spatial Index and Query CapabilityCreates a light weight flexible GIS spatial data base Return List of ICAOs in bounding box GeoTools ChainData List icao station.shp fsl-w88d.shp GOES.shp Grids.shp

  23. uEngine: Product DisseminationLeverage ESB Endpoint capabilities dissemination.xml ftpDissem.xml • Dissemination Test Action Script • Queries for GOES IR data • Decodes raw data into raster • Applies colormap • Writes to disk at ftpout

  24. <<Java abstract Class>> JythonTask <<Java Class>> MathScriptTask <<Java abstract Class>> Task <<Java Class>> JMath +execute() {interp.set(key, grib|image} {interp.exec(script)} +execute() +awndspd(u, v) +amin(a) +amax(a) … uEngine: Math ScriptingUses Jython as a Python Interpreter MathScriptTask XML tasks have the following format: <mathScript using="...">} <input name="..." value="..." />} <input name="..." value="..." />} <param name="..." value="..." />} <param name="..." value="..." />} <![CDATA[} import com.raytheon.edex.uengine.util.jython as utilities} utils = utilities.JMath()} </mathScript>} Generalized scripting capability. Emulates a multi-variable function having the form {@code y = f(a1, a2, ..., an,x1,x2,x3,...,xn)} This allows the processing of the outputs of multiple queries to obtain a single,new query result. +setScript() +utility methods() ChainData grid | Image ChainData List grids | images Jython Transformed grid | Image

  25. uEngine Math ScriptingExtension to general Jython Scripting Get Multiple Grid Records Degrib Multiple Records Pass to script to compute a scalar Apply Color Map Output as Image

  26. Map Library Capability – GeoToolsAWIPS GIS Capabilities: T04 • DataStore and Data Formats • ESRI Shapefile (Read/ Write) • WFS Web Feature Server (OGC) (Read / Write) • PostGIS geometric objects for PostgreSQL • GML Geography Markup Language • GeoTIFF georeferenced TIFF image • Coordinate Transformation (OpenGIS CTS) • Map Projections (Mercator, Transverse Mercator, Lambert Conformal Conic, Albers Equal Area Conic, Sterographic, Orthographic, …) • Math Transform

  27. <<Java Class>> ReprojectImage <<Java abstract Class>> Task <<Java abstract Class>> ShapefileQuery <<Java abstract Class>> Shapefile +execute() {MathTransform …} +execute() +execute() +execute() Map Library Capability – Map reprojectionTransform Ingested Data to Display Projection • GeoTools builds a transformation Matrix that gets reused in transform operations. • The Transformation matrix can be build up through a series of transforms. • Used In Barnes Analysis Pattern to calculate distances. • Degrib Task uses GeoTools to dynamically define projection data (I.e. NCEP hur.) ChainData List icao ChainData List icao GeoTools ReprojectedImage CoordinateReferenceSystem GridGeometry2D BufferedImage Note: Solid warping reprojection techniques are being considered to improve performance.

  28. Map Libraries Through GeoTools + JTSGeoTools: JTS Topology Suite Spatial Operations Java API • Computing Spatial Relationships • Shape File Intersect Demonstration • Distance Tool Demonstration Overlay Operations Buffer Operations Polygonization & Quantization

  29. Map Library Capability – Shape FilesuEngine Creates Shape File from Contoured Grids Normal Grib Query + Degribbing Contouring (!T04) Task Output Contour as a Shape File GeoTools

  30. <<Java Class>> SBNradarIngestSrv <<Java Class>> AbstractMessageSrv +process() +process() Ingest SBN Radar DataSOARadar Ingest Service Listening on End Point Satellite Broadcast Network Ingest Scan Directories Enterprise Service Bus End Point Configuration: Ingest.xml Plugin Pattern {PluginFactory.getPlugin(“RADAR”..)} .decode() .store()

  31. <<Java Class>> SBNradarIngestSrv +process() Ingest SBN Radar Data SOARadar Ingest Service Multi-threaded Note: SBN radar ingest automatically scales to 8 execution threads Mule ESB Service Container Log4j Log File Ingesting 64 dropped in files

  32. Drawing CapabilityImplemented in CAVE Plugin CAVE Menu Interface To Drawing CAVE Drawing Plugin Scalable Vector Graphic SVG map anonnation com.raytheon.viz.drawing plugin.xml Drawing Tool Interface

  33. <<Java Class>> StationaryFrontTool +createFront() {StationaryFront(…) } <<Java Class>> StationaryFront <<Java Class>> AbstractFrontTool <<Java Class>> AbstractDrawingTool <<Java Class>> AbstractModalMapTool <<Java Class>> GLMapEditor +handleMouse{…}() +runTool() +setActiveEditor() +setActiveEditor() +createVisuals() +createAdornments() Drawing CapabilityReuse Lower Level Fx-C Front Drawing Code Fx-C Code OpenGL GeoTools

  34. <<Java Class>> StartCollaborationTool +runTool() Collaboration CapabilityBased on XMPP protocal standards AWIPS Laptop 1 AWIPS Laptop 2 Connecting to a Specific Collaboration wildfire Collaboration Wide Area Network xmpp Jabber Standard Collaboration Protocal CAVE Menu Interface To Collaboration Concurrent CAVE Perspective has Chat Area Supports VoIP through plugin

  35. <<Java Class>> CollaborationManager <<Java Class>> DrawingLayer <<Java Class>> DrawingIO <<Java Class>> Line <<Java Class>> StartCollaborationTool <<Java Class>> DrawingChangeMessage <<Java Class>> CollaborationMessage <<Java Class>> Symbol <<Java Class>> CollaborationObject +buildLineChangeMessage() -marshallMessage() +connect() +sendCollabMessage(msg) +sendChatMessage(msg) +drawLine() {msg = DrawingIO… } +centerPoint +symbolID +setPoints( CoordinateXML[ ]) +getId() +getDate() +runTool() +getObject() +getState() Collaboration CapabilityAccurate GIS referenced Drawing and Symbols <<Java Eclipse Class>> Job +run() JiBX

  36. <<Java Class>> HydroShapefileResource <<Java Class>> ShapefileResource <<Java Class>> HydroParamWindow <<Java Class>> NewHydroEditorAction <<Java Class>> ShowHydroParamAction <<Java Class>> HydroInspectAdapter +handleDoubleClick(x, y) +run() +createContents() +init() +interrogate(Coordinate) +interrogate(Coordinate) +run() XYEditors added in T04 Initial Hydro Graph Display and Gage Select XYEditors: Skew-T, Hydro GFE, … ahps_gage.shp com.raytheon.hydro plugin.xml Hydro Data Repository

  37. <<Java Class>> HydroGraphWindow <<Java Class>> GraphMouseManager <<Java Class>> GLDisplayPane <<Java Class>> Window +createContents() +handleEvent() +setRenderable(graphRsc) +refresh() { OpenGL interactions } { Eclipse jface window } XYEditor: Initial Hydro Graph DisplayT04 uses generated canned data Hydro Data Repository

  38. <<Java Class>> ShowSkewTAction <<Java Class>> SkewTShapefileResource <<Java Class>> EditorInspectAdapter <<Java Class>> OpenSkewTAction <<Java Class>> SkewTWindow +init() +run() +interrogate(Coordinate) +handleDoubleClick(x, y) -retrieveData() +init(), +run() +createContents() XYEditor: Initial Skew-T Log P RenderingStation Selected From Map Display ua_stations.shp com.raytheon.hydro plugin.xml

  39. XYEditor:Initial Skew-T Log P Rendering

  40. <<Java Class>> EditorInspectAdapter <<Java Class>> ShowGFEAction <<Java Interface Class>> IWorkbenchWindow <<Java Class>> GFEResource <<Java Class>> GridDataRepository <<Java Class>> GridDataElement <<Java Class>> LocalStore +getResource() +handle mouse events… +init() +run() +checkout/in() +get/put/read() +getData() [ ] [ ] +getEnvelope() +get/removeData() +set/removeId() XYEditors added in T04 ContinuedInitial GFE editor Capability XYEditors: Skew-T, Hydro GFE, … Concept of Check in/out Editing Float Grid Value with Mouse events and rendering

  41. <<Java abstract Class>> Task <<Java Class>> ObjAnalysisBarnes <<Java Class>> Observation <<Java Class>> ObjAnalysis… <<Java Class>> ObjectiveAnalysis <<Java Class>> ObjectiveAnalysisFactory +execute() +getWorker() uEngine: Objective Analysis PatternBarnes first pass sample implementation +execute() +addRules() +getLatitude() +getLongitude() +getObservation() +doAnalysis()

  42. uEngine: Objective Analysis PatternBarnes first pass sample implementation script Querying + Degrib of a Grib data record (ie TMP) Performs a Spatial Query to get list of ICAOs in Bounding box Retrieves METARs returned by the spatial query

  43. uEngine: Objective Analysis PatternBarnes first pass sample implementation script Decode METARs and Load temperature Run Selected Analysis With parameters as set Apply colormap and convert to GeoTIFF for CAVE display

  44. Security ConceptsDo not change anything significantly • Each installation has perimeter security similar to what exists now we are not changing the security model or C&A • Goal is to be able to use a remote service as if it is local • Access Authentication • Service Authorization • Data Ingest Through Data Diode Type concept • SOA Endpoint Security Through ESB & JMS provider

  45. Security Concepts ContinuedSOA infrastructure has built in capabilities • JMS provider: activeMQ • Support JAAS as a provider for authentication • Default points to a “login.config” on classpath that in turn points to: • user.properties {has: user=password info} • group.properties {has: group=user1,user2… } • User Roles on Queues and Topics: • Read, Write, Admin (create) • Configurable message authorization policy allows each message to be content based authorized by a custom policy • Mule Security: Acegi; JAAS; PGP • Built in configurable security manager is responsible for authenticating requests • Configurable encryption strategy includes transports such as SSL and HTTPS • JAAS security provider (Java Authentication & Authorization Service) – API in since Java 1.4 • Mule is extendable to PGP signed encryption on endpoints for end-to-end communications

  46. CAVE Security ArchitectureData Ingest Concept of Data Diode Low Side LDAD High Side DX Cluster Firewall Initiates All Request And Data Transfers Has No Clue About Other Side Custom Socket Listener Custom Socket Client /clean-inbound/… /inbound/… One Way Interface ESB Endpoint Security Filter HTPPS ESB Endpoint

  47. New Capabilities Being Added in T05Capability Dependency Graph CAVE Data Interrogation

More Related