1. Wagner Silveira • Microsoft MVP - Azure Creating a Processing Pipeline with Azure Function and AIS

2. About me wsilveiranz wagner.silveira@notetoself.tech https://notetoself.tech Principal Integration Architect Microsoft Azure MVP MCSE Cloud Platform

3. Agenda • Case Study • Initial Solution • Updated Solution • Key Components • A year later – what changed? • Lessons Learned

4. Case Study

5. The solution requirements • Onboarding of external EDI messages • Should allow for different transports • Message need to be Validated, translated, enriched, routed • Control on retries and notification over failed messages • Auditing of initial message • Visibility of where a single message was in the process

6. Logical Design

7. Initial Solution

8. Reality Check • Big Payloads • EDIFACT schema not available • Operational costs • End to end Monitoring

9. Updated Solution

10. How was this different from the initial solution? • Azure Functions • EDIFACT support via .NET Package • Claim check pattern • Dedicated instances • Azure Storage • Payload storage (claim check pattern) • Application Insights • End to End with custom events • Single technology for monitoring and notification

11. Key Components

12. Key Components • Claim check pattern • App Insights Custom Events

13. The claim check pattern • Publishing function separates content from metadata then: • Store content on a safe repository • Publish metadata + storage pointer + Reprocessing count to message engine • Subscribing processes receive metadata and retrieve content when required

14. App Insights custom events • End to end traceability using a Correlation ID • Allow for exceptions to be captured the same way • Leverage Functions support for App Insights • Implemented as a reusable component App Insights

15. Hands on Time?

16. Key Management • Leveraged MSI support where available • SQL Server • Storage • Key vault • Kept keys in key vault where MSI where not supported directly • Service Bus

17. Exception Management & Retrieson Functions • Catch blocks using a notify and throw pattern • Leverage Function SB binding peek-lock pattern • Adjusted Maximum Delivery Count on SB • Messages that failed where delivered to DLQ

18. DLQ Management • Logic Apps polling subscriptions DLQ every 6 hours • Each subscription DLQ could have it’s own logic • Reprocessing count • Resubmission logic • Notification logic • Email notification • Error blob storage

19. One year later…

20. New technology candidates • Integration Services Engine • Azure Durable Functions • Event Grid

21. What about new features • Azure Functions Premium • Improved support for Key Vault • Improved support for MSI • Virtual Network support + Service Endpoints

22. Lessons learned

23. Review the fine print

24. Operational cost is a design consideration

25. Make the best of each technology

26. Think about the big picture

27. In Summary • Weight your technology options • Think about limitations, operational costs • Understand how your solution fit in the big picture • Understand the components roadmap

28. Thank you for attending!