Enhancing Data Transport Standards: Real-Time Exchange and Reduction in Costs

1. Data Transport Standard Nathan Chitty - PESC Gary Sandler - PESC

2. Data Transport Standard - DTS • DTS uses Internet technologies to facilitate real time data exchange and transaction processing • DTS builds on stable technologies, not specific products • DTS, once implemented, reduces programming and per-transaction costs through standardization

3. DTS Defined • Data Transport Standard is a specification not a product • Established by Postsecondary Education Standards Council (PESC) for exchanging data for: • Inquiries • Reports • Transactions

4. DTS Defined • A specification for an adjunct to or a replacement for existing data transport mechanisms • PGP / GnuPG encryption • SecretAgent w/ Email • FTP and SecureFTP

5. DTS Benefits • A Web Services implementation • Delivery confirmation included – no guessing • All requests get a response • All submissions get an answer of some kind • Facilitates real time data exchange

6. DTS Benefits • Includes automatic data encryption • Uses digital signature standards • Platform independent • Strong authentication with non-repudiation

7. Benefits To “System Providers” • Add value to schools’ systems • Schools want transport added to systems and are generally not concerned with the technologies • Easier to build one transport protocol for all recipients • Just as CommonRecord created the drive to build one XML format

8. Benefits to “Service Providers” • As everyone implements DTS, the need to support other transports will drop • If any school implements DTS, service providers will have to support it • Also provides a single communication infrastructure option for internal systems

9. DTS Specification • Specification covers • Technical interchange rules and processes • Recommended best practices • Technical Specification is the pure Simple Object Access Protocol (SOAP) interface • Implementation Guide is for both .Net and Java reference implementations

10. DTS Specification • Reference implementation examples are available • Specification does not cover • Business rules for transaction processing • Operational oversight, monitoring or escalation

11. Data Transport Issues in Higher Ed • E-mail is not reliable or flexible enough • No guarantee of delivery • No guarantee of order of delivery for sequence dependent data • No automatic confirmation of receipt or facility for retransmit

12. Data Transport Issues in Higher Ed • E-mail is not reliable or flexible enough (continued) • No synchronous response available • Email size limitations

13. Data Transport Issues in Higher Ed • FTP data exchange has own challenges • Possible to overwrite earlier files • No confirmation of receipt • No synchronous response

14. Data Transport Issues in Higher Ed • Encryption today is always separate and subject to its own • Issues • Maintenance • Failures

15. DTS Addresses These Transport Issues • DTS addresses • The confirmation issue with a send-receive protocol – confirmation is built in • The order of delivery problem by actively delivering and receiving the data – no unconfirmed hand-offs

16. DTS Addresses These Transport Issues • DTS addresses • The size problem through data compression • The FTP overwrite problem by not using filenames

17. DTS Addresses These Transport Issues • DTS addresses • The lack of a synchronous response by building in a required synchronous response, even if only for handling status • The encryption issue by using standard HTTPS for encryption – the same technology as for online banking

18. DTS Technical Workgroup • Task: Create a written specification for real-time exchange of data between organizations • Meets business requirements • Standards based • Standard technologies (Java, .Net) • Payload Insensitive • Secure and reliable

19. DTS Technologies • Global XML Web Service Architecture (GXA), generally accepted as the foundation for building Web Services • WSDL (Web Service Definition Language) • SOAP (Simple Object Access Protocol) • WS-I (Web Service Interoperability) • WS-S (Web Service Security)

20. DTS Technologies • WS-Security (Digital Signatures) • Strong authentication with non-repudiation • X.509 encryption keys and certificate authorities • SSL encryption of HTTP streams

21. Anticipated Architectures • Immediate processing • Request and processed Result Response • “Push/Push” deferred processing • Request and Acknowledge Response • Request with Result and Acknowledge Response • “Push/Pull” deferred processing • Request and Acknowledge Response (just send) • Request for Result and Result Response

22. Immediate

23. “Push/Push”

24. Push/Pull”

25. DTS Analogy • DTS is the definition of the “Pipe” and the structure of its contents • The “Pipe” is the internet • The content is SOAP • The end points/junctions are Web Services • The sources are Web Service enabled clients

26. DTS Analogy • DTS defines how others can connect to the “Pipe” already installed • Any connections must have certain “threads” • Any connections must handle two way traffic independent of how the traffic will be used

27. DTS Analogy • By knowing about the pipe and the type of connections, any “plumber” can use his/her own tools to make connections; just so long as the threads match

28. Extending the Analogy • We all have plumbing and fixtures • Very unlikely we all have the same type of fixtures • Yet our water companies still deliver to us all • All our fixtures use (“process”) it • All our drains return it

29. How Did We Do It? • Created basic HelloWorld service and client • Worked interoperable • Added simple Headers to HelloWorld • Was not interoperable • Added complex Header to HelloWorld • Was not interoperable

30. Why SOAP Headers • To answer routing and processing expectations without opening the payload • Remain payload insensitive • Allow extensibility for new processes

31. DTS SOAP Headers • DTSRequestRouting • DTSRequestServiceExpectation • DTSRequestPayloadType • DTSRequestSignature • DTSResponseRouting • DTSResponseAcknowledge • DTSResponsePayloadType • DTSResponseSignature

32. Convoluted Filename vs Header Elements • A [B] <X.Y.Z:M> A = File Type, B = Encrytption, X.Y.Z = key identifier, M = Unique message ID • Encryption unnecessary because using HTTPS • DTSRequestPayloadType = A • DTSRequestRouting • SourceIDSubCode = X, SourceID =Y(.Z) • UUID = M

33. Interop Problem with SOAP Headers • xsi:type attribute in Header elements • Java includes and requires this attribute • .Net does not

34. All about SOAP <soap:Header> <DTSRequestPayloadType xsi:type="DTSRequestPayloadType" xmlns="http://www.datatransportstandard.com"> <value>CRC01Request</value> </DTSRequestPayloadCode>

35. SOAP is the Key • The SOAP transmitted across the wire is of primary importance • Element names • Type attribute • Not Namespace moniker (Java uses one by default, .Net does not) • How you get the correct SOAP is not important

36. SOAP Differences That Do Not Matter Java: <ns1:DTSRequestSignature soapenv:mustUnderstand="0" xsi:type="ns1:DTSRequestSignature" xmlns:ns1="http://www.datatransportstandard.com"> <ns1:value>SignatureValue</ns1:value> </ns1:DTSRequestSignature> .Net: <DTSRequestSignature xsi:type="DTSRequestSignature"xmlns="http://www.datatransportstandard.com"> <value>SignatureValue</value> </DTSRequestSignature>

37. Reference Implementation Architecture • Client Application • Client Core • Service Core • Service Application

38. Client Application • Knows nothing of SOAP or Web Services • Implements Client Core Interface • “Setters” and “Getters” of DTS specific elements • Houses specific business logic

39. Client Core • Knows nothing of business logic • Uses properties set to construct the SOAP • Interface for “setting send” and “getting returned” elements • Handles the communication to Service Core- DTS Specification

40. Service Core • Accepts transmissions from Client Core • Implements Service Application Interface • “Setters” and “Getters” of DTS specific elements • Creates return SOAP • Format return acknowledgement or data from Service Application • Construct SOAP faults

41. Service Core (continued) • Isolated business logic • Examples • Invoke Service Application based on payload • Place payload in “queue”

42. Service Application • Interface for “setting sent” and “getting to be returned” elements • Houses specific business logic • Knows nothing of SOAP or Web Services

43. Connecting the layers

44. Connecting the layers

45. Connecting the layers

46. Connecting the layers

47. Connecting the layers

48. Additional DTS Information • Visit PESC at www.pesc.org • Materials available include • Executive summaries • Specifications • Reference (proof of concept) implementations

49. Adding SOAP Headers • Change WSDL – Still has problems • Create Container Classes • Container Classes require serialization/de-serialization directives

50. Adding SOAP Headers • Augment Service Code • Augment Client Code