Locality-constrained Linear Coding for Image Classification

1. Locality-constrained Linear Coding for Image Classification Presenter : Han-Mu Park

2. Locality-constrained Linear Coding for Image Classification, CVPR 2010 Contents • Introduction • Coding methods • Proposed method • Experimental results • Conclusion • References

3. Locality-constrained Linear Coding for Image Classification, CVPR 2010 Introduction • Bag-of-Words (BoW) model • An image is represented as a collection of visual words. • Generally, to represent the collection, histogram of words form is used.

4. Locality-constrained Linear Coding for Image Classification, CVPR 2010 Introduction • General Spatial Pyramid Matching frameworks • Feature extraction • SIFT • HOG • etc • Coding • Vector Quantization • Sparse coding • etc • Pooling • Max pooling • Sum pooling Spatial Pyramid Matching framework [J.Wang2010]

5. Locality-constrained Linear Coding for Image Classification, CVPR 2010 Introduction • General Spatial Pyramid Matching frameworks Spatial Pyramid Matching framework [J.Wang2010]

6. Locality-constrained Linear Coding for Image Classification, CVPR 2010 Coding methods • Coding methods • Vector quantization (VQ) • Sparce coding (SC) • Locality-constrained Linear Coding (LLC) [J.Wang2010]

7. Locality-constrained Linear Coding for Image Classification, CVPR 2010 Coding methods • Vector quantization (VQ) • Hard quantization method • A set of -dimensional local descriptors • Codebook with entries • Objective function • Where is the set of codes for X

8. Locality-constrained Linear Coding for Image Classification, CVPR 2010 Coding methods • Sparse coding (SC) • Soft quantization method • Relaxed the cardinality constraint • Objective function • The roles of sparsity regularization term • Because the codebook is usually over-complete , it is necessary to ensure that the under-determined system has a unique solution. • Sparsity allows the learned representation to capture salient patterns of local descriptors. • The sparse coding can achieve much less quantization error than VQ.

9. Locality-constrained Linear Coding for Image Classification, CVPR 2010 Proposed method • Locality-constrained Linear Coding (LLC) • Replaced the sparsity regularization term with new constraint. • Objective function • : the element-wise multiplication Where

10. Locality-constrained Linear Coding for Image Classification, CVPR 2010 Proposed method • Properties of LLC • Better reconstruction • Because LLC represents each descriptor by using multiple weighted bases (codewords), it has less reconstruction error than VQ. • Local smooth sparsity • Because the regularization term of in SC is not smooth, therefore, SC loses correlations between codes. • Analytical solution • The solution of LLC can be derived analytically by Where

11. Locality-constrained Linear Coding for Image Classification, CVPR 2010 Proposed method • Approximated LLC for fast encoding • The LLC selects the local bases for each descriptor to form a local coordinate system. • To speedup the encoding process, authors used nearest neighbors of as the local bases , and solve a much smaller linear system to get the codes • The reduced computation complexity • , where

12. Locality-constrained Linear Coding for Image Classification, CVPR 2010 Proposed method • Codebook optimization • To improve the accuracy, authors trained the codebook to optimize for LLC codes. • The optimal codebook can be obtained by • This can be solved by using Coordinate Descent method. • However, because the number of training descriptors is usually very large, the huge memory space is needed to solve that problem.

13. Locality-constrained Linear Coding for Image Classification, CVPR 2010 Proposed method • Incremental codebook optimization • First, initialize by using K-means clustering. • Then loop through all the training descriptors to update incrementally. • In each iteration, we take in a single (or a small set of) examples , and solve original objective function to obtain the corresponding LLC codes. [J.Wang2010]

14. Locality-constrained Linear Coding for Image Classification, CVPR 2010 Proposed method • Incremental codebook optimization • Then select bases whose corresponding weights are larger than predefined threshold, and refit without the locality constraint. • The obtained code is used to update the basis in a gradient descent fashion. [J.Wang2010]

15. Locality-constrained Linear Coding for Image Classification, CVPR 2010 Experimental results • Performance of codebook

16. Locality-constrained Linear Coding for Image Classification, CVPR 2010 Experimental results • Performance under different neighbors

17. Locality-constrained Linear Coding for Image Classification, CVPR 2010 Experimental results • Results using Pascal VOC 2007

18. Locality-constrained Linear Coding for Image Classification, CVPR 2010 Conclusion • Contribution • In this paper, the Locality-constrained Linear Coding (LCC) method is proposed • Better reconstruction • Local smooth sparsity • Analytical solution • For speedup, K-nearest neighbors algorithm is used. • To optimize the accuracy, incremental codebook learning is proposed for LCC.

19. Locality-constrained Linear Coding for Image Classification, CVPR 2010 References [1] J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, Y. Gong, “Locality-constrained Linear Coding for Image Classification,” CVPR 2010.

20. Thank you!