Enhanced GPU Algorithms for Efficient Matrix Multiplications

1. CSE 690: GPGPULecture 7: Matrix Multiplications Klaus Mueller Computer Science, Stony Brook University

2. Basic Concept • Triple loop

3. GPU Algorithms • First algorithm: • render a rectangle of size NxN • represent the matrices as NxN textures • each (i,j) is then a fragment • each fragment program is a loop or an unrolled loop -> may get too long • must pull in the same data many times -> poor data reuse, needs bandwidth • makes no use of 4-way RGBA parallelism -> wastes speedup

4. GPU Algorithms • Better algorithm: • use RGBA channels, pack a 2x2 submatrix • use swizzling to facilitate data reuse • swizzling improves fragment code length by factor 2 • may need multiple passes for larger matrices

5. GPU Algorithms • Using multi-texturing • requires l passes

6. GPU Algorithms • Can use RGBA parallelism as well • each texel represents a 2x2 submatrix • use swizzling as usual • needs l/2 passes

7. GPU Algorithms • Instead of a 2x2 submatrix, pack 4x1 column vectors • makes 4-times reuse of texels read from B, but uses texels from A only once

8. GPU Algorithms • Instead of a 2x2 submatrix, pack 4x1 column vectors • 6 fetches are needed for 4 mad’s (mult-add’s) -> 1.5 times more than before • but less rows and columns are accessed per pass -> improves cache hit frequency

9. GPU Algorithms • Originally only compute one product per shader • practically can unroll the loop 3-6 times (compute 3-6 products) • maximal fragment program length is the limit • reduces the number of passes required

10. Reality Check • Would like to compare CPU and GPU efficiencies for GPGPU tasks • The task of matrix multiplication is insightful here • features much data reuse • graphics programs are generally more stream-like and have less data reuse • this may lead to some limitations

11. Platforms • Pentium 4 3Ghz CPU, 512KB L2 cache • 12 GFLOPS peak compute • 44.1GB/sec cache BW • Using sgemm routine from ATLAS package • NVIDIA • GeForce 5900 Ultra • GeForce 6800 Ultra • ATI • Radeon 9800 XT • Radeon X800 XT PE

12. Performance

13. Bandwidth vs. Peak FLOPS

14. Analysis • Currently: • GPUs can fetch 16 floats and perform 16 4-component mad’s per clock • our app fetches 8 floats to perform one 4-component mad -> not enough computations • need more math ops per float fetched (> 8)

15. Analysis • Pentium processors have large L1 caches to boost memory bandwidth (bw) • bw / compute ratio better • main reason for only small performance gain achieved with GPUs

16. Analysis • Pentium processors have large L1 caches to boost memory bandwidth (bw) • bw / compute ratio better • main reason for only small performance gain achieved with GPUs • for matrix multiplications

17. Analysis • Expectations • make sure that there is enough arithmetic per data item fetched • lots of data resuse in the algorithm / task will make the CPU look better • streaming data OK -> they don’t “suffer” from reuse • matrix multiplication is an excellent reality-check example

18. Analysis • What do GPUs need: • bigger caches to enable larger blocks • currently there are enough registers to store a 6x6 submatrix • but currently shaders can only produce a small number of outputs -> limits the amount of blocking • Provide full-floating point accumulator registers • Widen path between texture and register files

19. References • E. Larsen and D. McAllister, “Fast matrix multiplies using graphics hardware,” Supercomputing 2001. • J. Hall, N. Carr and J. Hart, “Cache and bandwidth aware matrix multiplication on the GPU,” Tech Report UIUCDCS-R-2003-2328-1 • K. Fatahalian, J. Sugerman, and P. Hanrahan, “Understanding the efficiency of GPU algorithms for matrix-matrix multiplication,” Graphics Hardware Workshop 2004.