Media Foundation: Supporting Hardware Codecs and Cameras

1. CL34 Media Foundation: Supporting Hardware Codecs and Cameras Stan Pennington Program Manager Microsoft Corporation

2. Agenda • Media Foundation (MF) Hardware Codecs, and Asynchronous Media Foundation Transforms (MFTs) in general • MF video capture support, with focus on UVC 1.1 HD webcams • But first, let’s all get up to speed on MF… MFT MF

3. MF Background: Media Infrastructures in WindowsPerspective • Team focus on unmanaged APIs • Performance concerns for managed data flow components • Non-deterministic finalization of GC • Can expose configuration, control, and status through managed. Legacy and roadmap issues with wrappers • Four infrastructures • Video for Windows (VfW) • DirectShow (DShow) • Windows Media Format SDK (WM FSDK) • MF

4. MF Background: Media Infrastructures in WindowsPerspective (cont’d) • One or both of two development models for ISVs • Applications built on APIs • New data flow components using extensibility points – often driven by media format or DSP needs

5. MF Background: Media Infrastructures in WindowsEvolution 1990 1995 2000 2005 2010 Current Usage ACM Codecs VfW DVD TV DVD, TV (Filter, DMO, ACM, VCM Codecs) DShow WM True Streaming (DMO, ACM Codecs) WM FSDK Windows 7 Key Mainstream Scenarios & New Features (MFT Codecs) MF

6. MF Background: Media Infrastructures in WindowsTypical Microsoft usage in Windows 7 • MF • Playback of non-streaming ASF, MP3, MPEG-4, and selected AVI permutations for digital still cameras • New metadata handling in Shell Property Handlers • Transcoding and new media sharing • DShow • DVD, TV, MPEG-2, MPEG-1, custom I/O, video editing • FSDK • Best compatibility for ASF true streaming • Historic integration of ASF read, write

7. MF Background: Media Infrastructures in WindowsUse by Windows Media Experiences • Information provided to set expectations, subject to change in servicing or major releases • Windows Media Player (WMP) and WMP OCX • Opportunistically use MF, FSDK, or DShow as necessary for best end-user playback experience • MF (+ DShow source hybrid) for Windows 7 transcoding • Windows Media Center (WMC) • WMP OCX with custom presenter - for playback other than TV, DVD • Shell drag-and-drop transcoding for devices (Windows 7 only) • MF (+ DShow source hybrid)

8. MF Background: Media Infrastructures in WindowsUse by Windows Media Experiences (cont’d) • Change: Protected Media Path (PMP) in WMP and OCX • Windows Vista: PMP used for MF even with unprotected content, providing seamless cross-fade in mixed DRM/non-DRM playlists • Windows 7: PMP only used for DRM-protected content in MF

9. MF Background: Media Infrastructures in WindowsUse by Windows Media Experiences: Summary • Cannot assume WMP or WMC extensibility from media infrastructure documentation • Only WMP SDK plug-ins are documented as extensibility points, per http://msdn.microsoft.com/en-us/library/dd564682(VS.85).aspx • WMC extensibility is per http://msdn.microsoft.com/en-us/library/bb895967.aspx • Ask Microsoft for technical clarifications or documentation enhancements

10. MF Background: Why does Microsoft use MF? • Stream objects (IStream) for media input, output • Most connectivity and data flow divorced from streaming components, moved to higher layer • No push-pull component issues • Better potential for dynamic changes • Better control of threading • More constrained component roles and graphs topologies

11. MF Background: MF Features • All COM, C++ • Windows Vista • High-level API: Media Session • Formats: ASF w/ WMV and WMA, MP3 • Windows 7 • Backward compatibility • New high-level APIs: Source Reader, Sink Writer, Transcode API (part of “Fast Transcode” feature) • Formats: + AVI w/ DV and MJPEG, + MPEG-4 container w/ MPEG-4 ASP, H.264/AVC, AAC LC, HE v1, HE v2, key ACM wrapping • Asynchronous MFT and Hardware Codecs • Capture support: Device Proxy

12. TopoEdit = GraphEdt for MF demo

13. Async MFTModel • Sync MFT refresher • DMO enhanced for MF media types and samples • Simplest use is to call ProcessInput, ProcessOutput, repeat until output ready status • Need for adding async model • Inefficient for high level to poll for hardware completion • Async changes • Locked as sync for backward compatibility • Limited interface use until unlocked • Events fired on input needed, output ready, drain complete, marker processed

14. Async MFTHardware Acceleration • Candidates • Video and audio codecs, processors • Why codecs as a new model? • No DXVA support for encode • No efficient hybrid of DXVA decode and non-GPU encode for transcoding

15. Async MFTHardware Acceleration Usage • Generically, any ISV can write, any ISV can use • Subject to any licensing rules, of course • Otherwise, use local registration (MFTRegisterLocal)

16. Async MFTHardware Acceleration Usage (cont’d) • Picked up by high-level MF APIs, no need to code directly to Async MFTs, if: • Overall enable/disable registry keys enabled • MF API opt-in when not using WMP or OCX. Set MF_TOPOLOGY_HARDWARE_MODE attribute on topology • Codec has Trusted Merit if overlapping in-box functionality, and preferred media subtypes match, or fallback subtypes not satisfied by in-box component • Can specify directly in own MF topology (MFTEnumEx)

17. Async MFTHardware Acceleration Usage (cont’d) • For Windows Media Experiences and automatic Media Session selection, must constrain preference • Codec Trusted Merit cryptographic challenge/response • Keys and certificates are managed through Microsoft Hardware Codecs license program

18. Hardware CodecsLicense Program Under the Hood • Only for video codecs, video processors • Preferred over in-box MFT codecs for Windows Media Experiences if properly licensed through wmla@microsoft.com • Only for developer of actual codec core • Two hardware models supported • GPU – vendor writes complete MFT • AVStream – vendor uses Microsoft-provided proxy MFT, or writes complete MFT • Test suite protects preference • Shares test content with Media Performance Test Kit

19. Hardware CodecsLicense Program (cont’d) Under the Hood • Two codec types • Transcoding only • General – for transcoding and playback • Proper MF attributes must be set (see additional resources links) • Directly on vendor MFTs • Through KS property sets for AVStream • AVStream MFTs can use hardware data flow via Hardware Mediums

20. Hardware CodecsWindows Usage Under the Hood • Windows Media Experiences • WMP and Shell leverage the Transcode API for transcoding to device • Note: No sink extensibility • Several events documented for Hardware Codecs in general are optional in WMP usage. 3rd-party applications may be able to make hardware state machines more efficient by using optional events

21. Hardware CodecsMerit Rules Under the Hood • Video media subtypes in scope for Trusted Merit * * Test suite requirements are subject to change per the licensing program ** Existing technology may already provide hardware acceleration

22. Hardware CodecsFast Transcoding Under the Hood • Focused on conversion for portable devices • Based on MF Media Session, implemented in WMP and Shell layers • Non-public DShow wrapper Media Source used for broad source format support, can output compressed or uncompressed

23. Hardware CodecsMedia Session-Based Transcoding Under the Hood Source Resolver Topology & Loader Encoding Profile Management Provided by: Media Session Microsoft Microsoft or ISV Scheme Handler IHV Video Scaling/Conversion, Other Processing Byte Stream Handler File, Network Pipeline Topology Proc File, Network Media Source MFT Decoder MFT Encoder Media Sink Byte Stream Byte Stream or or Custom Input Custom Output or GPU-Based Hardware Codec Model Shown Hardware Acceleration Webcam

24. Hardware Codecs: New Quick Transcoding Sample demo

25. MF CaptureOverview • Device Proxy (DevProxy) – internal object for lighter-weight use of Kernel Streaming (KS) than typical DShow • AVStream driver model supported • USB Video Class (UVC) 1.1 Webcams • Audio capture through WASAPI • Other, but not full tuner capabilities

26. MF CaptureOverview (cont’d) • Picked up by MF high-level APIs through MF Media Source, no need to code directly to DevProxy • Create Media Source and pass in to Source Reader using MFCreateSourceReaderFromMediaSource • Hardware codecs supported by setting MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS attribute • Focus here on UVC capture • 1.1 includes value-add plug-in model for camera vendors • Diagram: http://msdn.microsoft.com/en-us/library/ee349147.aspx

27. MF CaptureUse with Source Reader • Source Reader high-level API benefits • Easy way to obtain uncompressed or compressed samples from media or device • Has constrained extensibility • Async and sync APIs. Async is best • Good model for feeding a composition engine or adding to existing media app design

28. MF CaptureDev Proxy + Source Reader Provided by: Specify URL, Byte Stream, or Media Source Async or Sync Control and Status Microsoft IHV Source Reader Direct access to consume and modify media samples File, Network Media Source Custom Input Webcam Specific Conversions* MFT Decoder * AVStream Hardware Codec Model Shown * Optional Hardware Acceleration

29. MF Capture: SDK Sample demo

30. MF DevelopmentTools • Tracing • Challenges • Async programming • Worker queues for Media Sources • Constrained ETW events • Some Windows Vista WPP events removed for performance optimizations. Windows TMFs not provided • Solution • New MFTrace tool, planned for release in 2010 • Detours hooking of public MF interfaces, adding events • Consolidation of existing events • Usage • Live console output • Redirect to file, process with tool such as TextAnalysisNET • Topology and timestamp graphing cmds

31. MF DevelopmentTools (cont’d) • General • TopoEdit – now aware of hardware-accelerated MFTs

32. Summary • MF high-level APIs in Windows 7 provide integration for: • Hardware-accelerated transforms, including Hardware Codecs • Resultant fast transcoding • Simple AVStream-based audio/video capture, including HD webcams compliant with UVC 1.1 • Development and tracing tools are being improved

33. Call to Action • Evaluate MF for new designs • Consider MF in opportunities for new functionality or improvements in existing designs

34. Additional Resources • MSDN MF documentation in Windows SDK • Top level: http://msdn.microsoft.com/en-us/library/ms694197(VS.85).aspx • Async MFTs: http://msdn.microsoft.com/en-us/library/dd317909(VS.85).aspx • Writing a custom MFT: http://msdn.microsoft.com/en-us/library/dd940441(VS.85).aspx • Source Reader: http://msdn.microsoft.com/en-us/library/dd940436(VS.85).aspx • Video capture: http://msdn.microsoft.com/en-us/library/dd940439(VS.85).aspx • TopoEdit help: http://msdn.microsoft.com/en-us/library/ms694197(VS.85).aspx

35. Additional Resources (cont’d) • MSDN AVStream documentation in Windows DDK • Hardware Codec support in AVStream: http://msdn.microsoft.com/en-us/library/dd568169.aspx • MSDN MF Forum: http://social.msdn.microsoft.com/forums/en-US/mediafoundationdevelopment • Blog: http://blogs.msdn.com/mf

36. YOUR FEEDBACK IS IMPORTANT TO US! Please fill out session evaluation forms online at MicrosoftPDC.com

37. Learn More On Channel 9 • Expand your PDC experience through Channel 9 • Explore videos, hands-on labs, sample code and demos through the new Channel 9 training courses channel9.msdn.com/learn Built by Developers for Developers….