Advanced Image Filtering Techniques: Leveraging GDI and DX Hardware Acceleration

1. Image Filtering Advanced Image filtering with GDI and DX HW accelerations

2. An introduction : domain terms • Bitmap (let’s assume Bitmap == GDI+ Bitmap) System memory portion containing Pixel as color’s byte • DX Surface System or Video memory portion representing a bitmap • Texture Mainly Video memory portion containing a bitmap that will be mapped onto a polygon Image Filtering Advanced

3. Enhanced Image Filter Library • What does a filter do? • What is a filter? • What is a filter chain? A filter is a function from image to image, A filter chain is a functional composition so, it’s a function again. Image Filtering Advanced

4. The speed part 1 : unsafe protected override void filter(Bitmap b, Bitmap[] temp) { BitmapData bmData = b.LockBits(new Rectangle(0, 0, b.Width, b.Height), ImageLockMode.ReadWrite, PixelFormat.Format24bppRgb); int stride = bmData.Stride; System.IntPtr Scan0 = bmData.Scan0; unsafe { byte * p = (byte *)(void *)Scan0; int nOffset = stride - b.Width*3; int nWidth = b.Width * 3; for(int y=0;y<b.Height;++y) { for(int x=0; x < nWidth; ++x ) { p[0] = (byte)(255-p[0]); ++p; } p += nOffset; } } b.UnlockBits(bmData); } } This way is faster than accessing pixels through the GetPixel GDI function Image Filtering Advanced

5. Library structure Image Filtering Advanced

6. [Serializable] public abstract class Filter { private Filter next; protected Filter(Filter f) { next = f; } protected Filter() { next = null; } protected abstract void filter(Bitmap b, Bitmap[] temp); protected abstract int NumCopies { get; } public BaseConf GetConfig(){ return new BaseConf(); } public void Apply(Bitmap b) { int n = this.NumCopies; Filter p = next; while (p != null) { n = Math.Max(n, p.NumCopies); p = p.next; } Bitmap[] tmp = new Bitmap[n]; for (int i = 0; i < n; i++) tmp[i] = new Bitmap(b.Width, b.Height); p = this; do { p.filter(b, tmp); p = p.next; } while (p != null); foreach (Bitmap bm in tmp) { bm.Dispose(); } } } The key for success : the filter base class design Create all the temporary Images before your filter chain goes with its work! Image Filtering Advanced

7. The speed part 2 : ngen • Let’s get native! Run the ngen tool on the filter library assembly to get it compiled once for all. Image Filtering Advanced

8. The speed part 3 : Get HW! • A little trick to achieve the real magic • Let’s pretend : • Remember you gain speed but you get also limitations! Image => plane => polygon Image Filtering Advanced

9. DX Structure • We can use the HW 3D acceleration to boost 2D image filtering using the Programmable Pixel Shader Image Filtering Advanced

10. HSL with DX 9 • Cross Hardware language! • Can contains multiple techniques and passes! • Exposes a large number of common function as lerp and so on Image Filtering Advanced

11. HSL Stupid Example float4 Light(float3 LightDir : TEXCOORD1, uniform float4 LightColor, float2 texcrd : TEXCOORD0, uniform sampler samp) : COLOR { float3 Normal = tex2D(samp,texcrd); return dot((Normal*2 - 1), LightDir)*LightColor; } Image Filtering Advanced

12. The Idea • Create a DX device on a control • Create a 3D plane • Put the image you want to process on the plane as a texture • Use the Pixel Shader Program to make the GPU works for you • Use the Control Graphics class to save your processed image. Image Filtering Advanced

13. A simple but real FX example Image Filtering Advanced

14. And the cons? Here we are . . . • You can use only power of 2 sized bitmap • The display size is the maximum size for the output image . . . But you get a common PC running as it’s got a real expensive (and dedicated) DSP! Image Filtering Advanced

15. The Direct Show way • DShow use a graph from sources to renderes (audio and video) • Along the graph you can have filters • 2 approaches • In place transformations • Not in place transformations • Only C++ (the unmanaged one!) Image Filtering Advanced

16. CTransformFilter::Transform • This filter uses the CTransformInputPin class for its input pin, and the CTransformOutputPin class for its output pin. • Use this base class if you want to try filter and then pass it through the graph • Beware of memory leak! Image Filtering Advanced

17. Example of transform HRESULT CRleFilter::Transform(IMediaSample *pSource, IMediaSample *pDest) { // Get pointers to the underlying buffers. BYTE *pBufferIn, *pBufferOut; hr = pSource->GetPointer(&pBufferIn); if (FAILED(hr)) { return hr; } hr = pDest->GetPointer(&pBufferOut); if (FAILED(hr)) { return hr; } // Process the data. DWORD cbDest = EncodeFrame(pBufferIn, pBufferOut); KASSERT((long)cbDest <= pDest->GetSize()); pDest->SetActualDataLength(cbDest); pDest->SetSyncPoint(TRUE); return S_OK; } Image Filtering Advanced

18. CTransInPlaceFilter::Transform • Transform the input sample in place • Use it for real real-time processing • What do you want more? Image Filtering Advanced

19. The Frame Dispatcer Class If you want to apply filter to a real time camera or to an avi file in C# there’s an utility from the Medialab (university of Pisa) that can help you to get your frames inside a managed application! Image Filtering Advanced

20. References • Enhanced Filter Library and Frame Dispatcer • http://dotnet.di.unipi.it Image Filtering Advanced