15-211 Fundamental Structures of Computer Science

1. 15-211Fundamental Structuresof Computer Science

2. Data Compression with SVD

3. Data Compression We have studied two important data compression algorithms • Huffman Code • Lemple-Ziv Dictionary Method. • Provides a good introduction to lossless compression. • What if we can compress an image by degrading the image a bit? • We know that lossy compression techniques are used in jpeg and gif compression algorithms • Next we will study a method to do a lossy compression using a matrix decomposition method known as SVD

4. Singular Value Decomposition(SVD) • Suppose A is an mxn matrix • We can find a decomposition of the matrix A such that A = U S VT, where • U and V are orthonormal matrices (I.e. UUT = I and V VT = I, where I-identity matrix • S is a diagonal matrix such that S = diag(s1, s2, s3, … sk, 0,0,…0), where si ‘s are called the singular values of A and k is the rank of A. It is possible to choose U and V such that s1> s1> …. > sk

5. Here is another way of expressing A • A = s1 U1V1T + s2 U2V2T + ….+ sK UKVKT where Ui and Vi are ith column of U and V respectively • Bit of a knowledge about block matrix multiplication will convince you that this sum is indeed equal to A. • So How does this applies to image compression? It is very very interesting • Any image is really an mxn matrix of pixels and in a bitmap color image each pixel is represented by 3-bytes (R,G,B)

6. Here is a look at it

7. Consider this color image This is part of a famous image (Do you know who? Hint: Splay) The image is a 16x16 bitmap image enlarged

8. Here is the Red part of the image

9. Green Part

10. Blue Part

11. The Red matrix representation of the image (16x16 matrix) 173 165 165 165 148 132 123 132 140 156 173 181 181 181 189 173 198 189 189 189 181 165 148 165 165 173 181 198 206 198 181 165 206 206 206 206 198 189 181 181 198 206 206 222 231 214 181 165 231 222 206 198 189 181 181 181 206 222 222 222 231 222 198 181 231 214 189 173 165 165 173 181 181 189 198 222 239 231 206 214 206 189 173 148 148 148 148 165 156 148 165 198 222 231 214 239 181 165 140 123 123 115 115 123 140 148 140 148 165 206 239 247 165 82 66 82 90 82 90 107 123 123 115 132 140 165 198 231 123 198 74 49 57 82 82 99 107 115 115 123 132 132 148 214 239 239 107 82 82 74 90 107 123 115 115 123 115 115 123 198 255 90 74 74 99 74 115 123 132 123 123 115 115 140 165 189 247 99 99 82 90 107 123 123 123 123 123 132 140 156 181 198 247 239 165 132 107 148 140 132 132 123 132 148 140 140 156 214 198 231 165 156 132 156 156 140 140 140 148 148 132 140 156 222 247 239 222 181 181 140 156 140 148 148 148 140 132 156 206 222 214 198 181 181 181 181 173 148 156 148 140 140 165 198 222 239 • So the idea is to apply SVD to this matrix and get a close enough approximation using as fewer columns of U and V as possible. • So for example, if the above matrix has one large dominant singular value, we might be able to get a pretty good approximation using just a single vector from U and V

12. Apply SVD to a simple matrix • Lets take a look at applying the SVD to a smaller matrix • This example will allow us to understand what is going on here A = 120 100 120 100 10 10 10 10 60 60 70 80 150 120 150 150 U = 0.5709 -0.6772 -0.4532 0.1009 0.0516 -0.0005 -0.1539 -0.9867 0.3500 0.7121 -0.5984 0.1113 0.7409 0.1854 0.6425 -0.0615 S = 386.1540 0 0 0 0 20.6541 0 0 0 0 7.5842 0 0 0 0 0.9919 V = 0.5209 -0.5194 0.6004 -0.3137 0.4338 -0.1330 -0.7461 -0.4873 0.5300 -0.1746 -0.1886 0.8081 0.5095 0.8259 0.2176 -0.1049 Note that first eigen value is substantially larger than the others.

13. Form a rank-1 sum A1 = s1 U1 V1T • A1 = 115 96 117 112 10 9 11 10 70 59 72 69 149 124 152 146 • Error Matrix |A - A1| is 5 4 3 12 0 1 1 0 10 1 2 11 1 4 2 4 Error is relatively small with a rank-1 approximation.

14. What have we learnt here? • Perhaps in this case we only need to know one column vector, one row vector, one singular value to get a pretty good approximation to the original image. • So instead of 4x4 = 16 bytes, we can store = 4 + 4 + 1 bytes to get an image fairly “close” to original image (almost 50% savings)

15. What if we do a rank-2 approximation? A2 = s1 U1 V1T + s2 U2 V2T A2 = 122 98 119 100 10 9 11 10 62 57 69 81 147 123 151 149 • Error Matrix |A - A2| 2 2 1 0 0 1 1 0 2 3 1 1 3 3 1 1 • Even smaller error matrix

16. Analysis To get an idea of how close the approximation to the original matrix is, we can calculate: • Mean of Rank1 error matrix = 3.8125 • Mean of Rank2 error matrix = 1.3750 Where mean is the average of the all entries • We really don’t gain much by calculating the rank-2 approximation (why?)

17. Why is that? • If you look at the first singular value it is fairly large compared to others. • Therefore, contribution from the rank-1 sum is very significant compared to the sum of all other rank approximations. • So even if you leave out all other rank sums, you still get a pretty good approximation with just two vectors. So we are up to something here • It is time to look at some samples

18. Some Samples (128x128) Original mage 49K Rank 1 approx 825 bytes

19. Samples ctd… Rank 8 approx 7K Rank 16 approx 13K

20. Some size observations • Note that theoretically the sizes of the compressed images should be • Rank 1 = 54 + (128 + 128 + 1)*3 • Rank 8 = 54 + (128+128+1)*3*8 = 6K • Rank 16 = 54 + (128 + 128 + 1)*3*16 = 12K • Rank 32 = 54 + (128 + 128 + 1)*3*32 = 24K • Rank 64 = 48K (pretty close to the original) Bmp Header U1 V1 + S1bytes/pixel

21. Implementation (compression) 001100100 000110000 1100000100 SVD Compressed file stores U, V and S for the Rank selected for each of the colors R , G and B and header bytes SVD SVD Header bytes COMPRESSION STEP

22. Implementation (decompression) 001100100 000110000 1100000100 Form rank sum Compressed file stores U, V and S for the Rank selected for each of the colors R , G and B and the bmp header Form rank sum Form rank sum Header bytes DECOMPRESSION

23. Adaptive Rank Methods • All popular image compression programs apply compression algorithm to sub-blocks of the image • This exploit the uneven characteristics of the original image • If parts of the image are less complex than the others, then a smaller number of singular values are needed to obtain a "close" approximation

24. Adaptive Rank Methods ctd.. • So instead of picking same fixed rank for each sub-block, we decide how many singular values to pick from each sub-block by looking at the following: • Percent of r values = s1 + s2 + ….+ sr ------------------ s1 + s2 + ….+ sk Where k is the max number of non zero singular values of A.

25. Results of applying the Adaptive Ranking Method • We applied the adaptive ranking method to Danny Sleator. Here are the results. 80% of singular values 26K 50% of singular values 15K Original 49K 10% of singular values 14K

26. Conclusion • As with any other compression technique there are drawbacks to SVD. For eg: we need to store the matrices U and V. So if we have to store more than n2 /(2n+1) singular values, this is not good. • Instead of selecting a fixed block size for each block, is it possible to achieve a better compression using dynamic block sizes? Can this be done by sub-dividing the original image until an optimal block size is found? • Optimize the storage scheme for U and V. Pack as many bits as possible to obtain a greater compression. • We encourage you to explore the idea further.