Introduction to WebSphere MQ File Transfer Edition

1. Introduction to WebSphere MQ File Transfer Edition

2. WebSphere MQ: Connectivity to, from and within an Enterprise Enterprise Regional Office Sensor e.g. RFID Branch Outlet Retail Store Pervasive Device Refinery Petrol Forecourt Mobile Phone 2 • A Universal Message Bus for access to data wherever it exists to support your business. MQ moves: • Data • Messages • Files • Events • Services • Provides a comprehensive range of Messaging capabilities to support your Business requirements for data integration • Reliability and availability QoS • Messaging integration patterns • Managed File Transfer • SOA foundation • Provides appropriate data accessand data privacy controls to help meet audit and regulatory requirements • MQ 7.5 is GAed on 6/15/2012, single, integrated offering for all messaging functions CSS: F S

3. What problems does Managed File Transfer Solution solve • Reliability • Do you experience lost file, partially written file and duplicate due to inconsistent network or system being not available? • What is the business impact when files aren’t properly transferred? • How much time and energy do you spend trying to figure out what happened when a file went astray? • How much time and effort does it take to recover/restore things after a failure? • Performance • Do you need to transfer large file or many files at the same time? Are you able to meet the SLA? • Every time when a file transfer fails, does it require to be restarted from the beginning? • Security • Do you have any security concerns, such as plain file in transit which contains sensitive data? • How about access control, such as who can transfer what file(s) from source to destination? • Visibility and Traceability • Do you have any visibility of the file transfer status and progress, whether the file is partially sent, completes successfully or not, in a timely fashion? • Do you want to track the entire journey of a file across the enterprise? • Are you able to show in an audit where your last 10 transferred files and documents came from and went? • Automated and Integrated • Do you need various ways to invoke file transfer such as ad-hoc, scheduled, event triggered, or through programming API? • Do you need to integrate file transfer with the rest of your IT infrastructure and business processes?

4. FTP – the lowest common denominator for file transfer FTP is 30 years old, and showing its age. It is no meets business needs Limited Reliability Limited Security • Partially written files (often only detected after substantial process and time is wasted) • Cost of re-transmitting big files (no check-point restart) • Lack of Character Set conversion (file data can be unusable after transfer • Username and password are often in plain text • Lack of Privacy, authentication and encryption often not available Little visibility or traceability Limited Flexibility • No notifications when a transfer fails • No central management and progress monitoring • Cannot track the entire journey of files • All resources have to be available at the same time • ftp Scripts are typically scattered across machines, and require platform specific skills • Often only one ftp transfer can run at a time

5. WebSphere MQ File Transfer Edition Adds managed file transfer capabilities to WebSphere MQ …… WebSphere MQ File Transfer Edition C Y Z A B X

6. WebSphere MQ - A consolidated transport for both files and messages Maximizes the ROI • Traditional approaches to file transfer result in parallel infrastructures • One for files – typically built on FTP • One for application messaging – based on WebSphere MQ, or similar • High degree of duplication in creating and maintaining the two infrastructures • MQ File Transfer Edition reuses the MQ network for managed file transfer and yields: • Operational savings and simplification • Reduced administration effort • Reduced skills requirements and maintenance Application Messaging File Transfers Consolidated Transport for Messages & Files

7. How does WebSphere MQ FTE work? MQ Server Agent QM 2. Receive file data 1. Send file data MQ FTE Agent MQ FTE Agent MQ network MQ network

8. How does WebSphere MQ FTE work?

9. Components of a typical WMQ File Transfer Edition Network • WebSphere MQ • Provides the transport for all communication between agents, commands and the coordination hub • Agents • The endpoints for file transfers • Long running, multi-threaded MQ applications, • Associated with one particular queue manager (V6 or V7) • Efficient transfer protocol for files • Commands • Send instructions to agents and display information about agent configuration • Many implementations of commands: MQ Explorer, Command Line, Scripting, API • Coordination Hub • Stores configuration information about the WMQ FTE network • Collects events for the file transfers • Keeps a historical account of transfers that have taken place

10. Ways to configure and control managed file transfers Graphical Eclipse-based MQ Explorer Command Line Interface Job Control Language (JCL) XML Scripts using Apache Ant

11. Centralized configuration using MQ Explorer Eclipse GUI integrated into WebSphere MQ Explorer • Shows progress of current transfers and outcome of recent transfers • 3rd party and bespoke applications can also subscribe to these events

12. Managed file transfers using MQ Explorer • Graphical user interface that allows: • Operators to create ad-hoc file transfers to satisfy a particular business need • File transfers can: • Specify code page conversion • Group together multiple files into a single transfer • Recursively move directory structures

13. Scheduled transfers • Transfers can be scheduled to take place at a specific time • Scheduled transfers can be used to initiate a managed file transfer during a processing window • Scheduled transfers can repeat: • Every day, week, year etc. • For a specified number of repetitions • Until an end date/time

14. Directory Monitoring • Monitor file system for presence of “trigger files” andstart a file transfer operation • Directory monitoring can be used to integrate with an application which produce files as its output – without needing changes to the existing application • “Trigger files” can be: • Specified using wildcards • Found by recursively searching directory trees • Attributes of file transfer can be inferred from trigger file, for example: • Transfer all files from same directory as trigger file • Select destination agent name from parent directory of trigger file • Name destination file with timestamp of trigger file • Etc. • Supports extension via user exit routines

15. Via the Command Line... • Equivalent function to the GUI is also scriptable from the command line • Administrative commands: • Define and delete agents • Configure agents • List and show details about agents • Start and stop agents • Create and delete scheduled transfers • Create and delete directory monitors • Operational commands: • Start transfers • Cancel transfers • List in-progress transfers

16. … and also Job Control Language (JCL)

17. Scripting • Script together complex sequences of transfer operations which execute conditionally • E.g. If step1 successful do step2 else email administrator • Use scripting to call out to other programs to perform pre/post processing of file data • E.g. Run a program prior to the transfer to generate the file, or after the transfer to process the file • The output from any programs run is captured in the transfer audit information • Based on the Apache Ant language which is: • Open… (so you can get a book on it…) • ..and extensible (so you can use Ant tasks developed elsewhere)

18. Example of a file transfer script • FTE can also start another application to process the file • FTE transports fileto destination • Application writesfile to file system ExistingApplication WMQFTEAgent WMQFTEAgent ExistingApplication *tap* • Agent monitors filesystem, spots arrival of file and based on rules, transfers the file • At destination MQ FTE writes file to file system

19. Other Examples • Transfer one or multiple files from source to destination • Transfer file(s) from source to multiple destinations • Unzip, Combine and Zip files to send from source to destination • Concatenate files from multiple sources into a single file • Split a single file into multiple files and send to different destinations • Validate the file data before file transfer • Send email notification if file transfer fails

20. Integration with event driven applications MQ FTE Helps: Deliver files as message payloads and vice versa Monitor queues and transfer message payloads to files Bridge between batch based applications and event driven applications WebSphere MQ&File Transfer Edition 20

21. The WMQ FTE agent can monitor queues for the arrival of messages, then perform an action, such as transferring the payload from the message to a file Conditions that can be monitored for: Queue not empty Complete group of messages MQ queue monitoring FTE agent can monitor for files arriving… ExistingApplication WMQFTEAgent It can also monitor for messages arriving on a queue… ExistingApplication WMQFTEAgent

22. Example Usage of converting data between files and messages • One file becomes one message One file to one message WMQFTE • The file can be split based on: • Size • Binary delimiter • Regular expression One file to a group of messages WMQFTE • One message becomes one file One message to one file WMQFTE • Optionally, a delimiter can be inserted between each message used to compose the file A group of messages (or all messages on the queue) to one file WMQFTE

23. API for Invoking File Transfer 1. Send request File Transfers can be invoked by ESB as a service ESB 2. Start file transfer 4. Receive file data 3. Send file data MQ FTE Agent MQ Server MQ FTE Agent MQ network MQ network Database Logger Central Management Tool

24. File Broker – integration with ESB ESB File to Message(s) or Message(s) to File conversion 2. Process file data if required such as data transformation MQ FTE Agent 1. Send file data 3. Receive file data MQ Server MQ FTE Agent MQ FTE Agent MQ network MQ network • File Transfers can be initiated via • - Scheduler • - File Directory Monitor • Ad Hoc • Script Database Logger Central Management Tool

25. Web Gateway - Integration with Web • Pain-points: • Difficult to mix human imitated file transfers with existing infrastructure for machine-to-machine managed file transfer • Managed file transfers to zero-install, small-footprint devices • File Transfer Edition Helps: • A RESTful API for exchange files with an WMQ FTE network • Example web 2.0 applications to use as a starting point Web Gateway Agent Agent Agent HTTPS WebSphere MQ HTTP HTTPS Agent Agent

26. Protocol Bridging Agents • Support for transferring files located on FTP and SFTP servers • The source or destination for a transfer can be an FTP or an SFTP server • Enables incremental modernization of FTP-based home-grown solutions • Provides auditability of transfers across FTP/SFTP to central audit log • Ensures reliability of transfers across FTP/SFTP with checkpoint restart • Fully integrated into graphical, command line and XML scripting interfaces • Just looks like another FTE agent… Files exchanged between FTE and FTP/SFTP FTP/SFTPClient FTP/SFTP Agent Agent Agent FTP/SFTPClient ProtocolBridgeAgent FTP/SFTPServer WebSphere MQ FTP/SFTPClient Audit information

27. Security - File Transfer Access Control • WebSphere MQ already provides access control that can be used to prevent unauthorized users from accessing MQ objects (such as queues) • File Transfer Edition extends this to include authorities that relate to file transfer operations (e.g. should this user be able to transfer files from the source system, and to the destination system)

28. Securing file data transfer with SSL and WMQ AMS • WMQ FTE supports transport level encryption using SSL • Data is encrypted before it is sent over a channel and decrypted when it is received WebSphereMQQueueManager WebSphereMQQueueManager svrconn channel sndr/rcvrchannels Agent Agent • When combined with WMQ Advanced Message Security • Allows file data to be encrypted at the source system and only decrypted when it reaches the destination system • Data is secure even when at rest on a queue WebSphereMQQueueManager WebSphereMQQueueManager svrconn channel sndr/rcvrchannels Agent Agent

29. File Transfer Solution Design Considerations • Who and How to invoke file transfer - scheduled, event driven, ad-hoc, or via API? • How to transfer the file reliably and securely - HTTP, FTP, MQ or in combination? • How to access the file data at source and destination – File directory, MQ, HTTP, etc.? • Is any file data processing required such as transformation? • Is there any pre and post file transfer processing steps required at source and destination, such as notification? • How to handle file transfer failure? • Is there any file transfer tasks which need to be executed in sequence? • Is there any integration points with other applications/systems which are not based on files? • What’s monitoring and auditing requirement if any? • Is security a concern? Other considerations • How big is the file, and how often are the file transfers required? • How many systems (source and destination) are involved and what system? Same location or not? • What’s the topology for file transfers, hub/spoke or point to point?

30. Resources • Information Center: • http://publib.boulder.ibm.com/infocenter/wmqfte/v7r0/index.jsp • Redbooks / Redguides / Redpapers: • Getting Started with WebSphere MQ File Transfer Edition V7 • http://www.redbooks.ibm.com/abstracts/sg247760.html • IBM WebSphere MQ File Transfer Edition Solution Overview • http://www.redbooks.ibm.com/abstracts/redp4532.html • Managed File Transfer for SOA using IBM WebSphere MQ File Transfer Edition • http://www.redbooks.ibm.com/abstracts/redp4533.html • B2B Enabled Managed File Transfer using WebSphere DataPower B2B Appliance XB60 and WebSphere MQ File Transfer Edition • http://www.redbooks.ibm.com/abstracts/redp4603.html • IBM Sterling Managed File Transfer Integration and WebSphere Connectivity for a Multi-Enterprise Solution • http://www.redbooks.ibm.com/abstracts/sg247927.htm • Multi-Enterprise File Transfer with WebSphere Connectivity • http://www.redbooks.ibm.com/abstracts/sg247886.htm • Trial Download: • http://www.ibm.com/software/integration/wmq/filetransfer/ • Early Design Program • Interested in participating in the development of future versions of FTE? • Ask your local IBM representative to nominate you for the FTE EDP program

31. Any Questions?

32. Back-up material

33. MQ Advanced Message Security • Is a MQ plug-in, which pre-req MQ • Adds additional security services to WebSphere MQ • Enables secure message transfers at application level • Replaces costly, home-grown solutions that lack management controls • Extra Security capabilities • Policy-based authentication for each application • Protects message contents end-to-end including when residing in queues • Centralized control of fine-grained MQ security policies • Audit logs of data and queue access • No changes needed to applications or queues

34. WMQ vs WMQ AMS 34 • WebSphere MQ • Authentication (local OS for server apps or peer authenticated SSL for client apps) • Authorisation (OAM on distributed, RACF on z/OS) • Integrity (SSL for channels) • Privacy (SSL for channels) • WebSphere MQ AMS • As above, additionally: • Integrity (Digital signature of message content) • Privacy (Message content encryption)

35. Securing file transfer with MQ AMS File data is decrypted and signature verified after transfer File data is encrypted and signed before transfer 2. Receive file data encrypted and signed 1. Send file data encrypted and signed MQ FTE Agent MQ Server MQ FTE Agent MQ network MQ AMS Client MQ network MQ AMS Server MQ AMS Client Database Logger Central Management Tool

36. WebSphere Message Broker Nodes for MQ FTE FTEInput node Build flows that accepts file transfers from the WMQ FTE network FTEOutput node Build flows that are designed to send a file across a WMQ FTE network When WMQ FTE nodes are used in a flow an FTE agent is automatically started in the Message Broker Execution Group Part ofWMB 7.0.0.1 Message Broker Execution Group WMQFTEAgent Message Flow WMQFTEAgent WMQFTEAgent WMQFTEAgent FTEInput FTEOutput 36

37. Integration with Message Broker (1) data transformation, Enrichment, convert file data to message(s), etc. Integrate with non file oriented systems • File Transfers can be initiated via • - Scheduler • - File Directory Monitor • Ad Hoc • Script/API WMB 3. Receive file data Message Flow Web Services, MQ, DB, etc. MQ FTE Agent 2. Process file data if required 1. Send file data 3. Receive file data MQ Server MQ FTE Agent MQ FTE Agent MQ network MQ network Database Logger Central Management Tool

38. Integration with Message Broker (2) WMB File Transfers can be invoked By Message Broker as a service Message Broker MQ FTE Agent 1. Start file transfer 3. Receive file data 2. Send file data MQ FTE Agent MQ Server MQ FTE Agent MQ network MQ network Database Logger Central Management Tool

39. Integration with IBM Sterling Connect:Direct C:DNode • The Connect:Direct Bridge capability supports managed file transfers that span FTE and C:D with a joined up audit trail Agent Agent Agent C:DNode C:DNode C:DBridgeAgent WebSphere MQ C:DNode Reference Trading Partner FTEAudit C:DAudit Inside the FTE audit trail… The audit information for each FTE transferreferences related C:D audit information