Data Fine-tuning

1. Data Fine-tuning Saheb Chhabra PuspitaMajumdar MayankVatsa Richa Singh Saheb Chhabra, PuspitaMajumdar, Mayank Vatsa, Richa Singh Indraprastha Institute of Information Technology Delhi (IIIT-D), India

2. MOTIVATION: COTS • Commercial off-the-shelf (COTS ) • Black-Box • COTS Requirements • Upgradation with time • Retraining of the system MONEY COTS Note: Images are taken from Internet.

3. MOTIVATION: COTS Performance Expectation Decision COTS Input Reality Note: Images are taken from Internet.

4. MOTIVATION Can’t access the model Don’t know the parameters Don’t know the model Note: Images are taken from Internet.

5. What do we have?  Model Hyperparameters Model Access Dataset Output of Model Model Training  Note: Images are taken from Internet.

6. What do we have?  Model Hyperparameters Model Access Dataset Output of Model Model Training  Can we enhance the performance of a black-box system? Note: Images are taken from Internet.

7. Black Box Model • Black Box Model Input Data Bias Weight

8. Black Box Model • Black Box Model Input Data Bias Weight

9. Data Fine-tuning • Data Fine-tuning (DFT) DFT

10. Data Fine-tuning • Data Fine-tuning (DFT) DFT Y-axis Y-axis DFT Class 1 Class 1 Class 2 Class 2 (b) X-axis Pre-trained model’s decision boundary X-axis (a)

11. Model Fine-tuning • Model Fine-tuning MFT

12. Model Fine-tuning vs Data Fine-tuning Y-axis Y-axis Model Fine-tuning Model Fine-tuning Class 1 Class 1 MFT Class 2 Class 2 (b) X-axis (a) X-axis Data Fine-tuning DFT Y-axis Data Fine-tuning Pre-trained model’s decision boundary Class 1 Fine-tuned model’s decision boundary Class 2 (c) X-axis

13. Literature: Adversarial Perturbation [1]. Xie, Cihang, et al. "Mitigating adversarial effects through randomization." arXiv preprint arXiv:1711.01991 (2017). [2]. S. Chhabra, R. Singh, M. Vatsa, G. Gupta, Anonymizing k Facial Attributes via Adversarial Perturbations, IJCAI , 2018,

14. Data Fine-tuning: Challenges • Learn a single perturbation for a given dataset • The visual appearance of the image should be preserved after performing data fine-tuning.

15. Optimization Original Training Set True Labels Number of Images Perturbed Training Set Perturbation Set of Attributes Transform image in range of 0 to 1 Output scores Model Input Enforces the outputs scores towards true labels

16. True Labels Block Diagram Attribute Prediction Minimize Loss Optimize over variable (a) (b)

17. Illustration of Data Fine-tuning Fine tuned Dataset: Dataset: Dataset: Input Image Space Input Image Space Input Image Space Y-axis Y-axis Y-axis Add Perturbation Class 1 Class 2 Class 2 Class 2 Class 1 Class 1 X-axis X-axis X-axis (c) (a) (e) Pre-trained Attribute Prediction Model Attribute Prediction Model Pre-trained Attribute Prediction Model Data fine-tuning Training on Dataset Output Class Scores Output Class Scores Output Class Scores (W+b) (WX+b) (WZ+b) Y-axis Y-axis Y-axis Class 2 Class 1 Class 2 Class 2 Class 1 Class 1 X-axis X-axis X-axis (b) (d) (f)

18. Experiments and Results • Two experiments are performed for Facial attribute classification • Black Box Data Fine-tuning: Intra Dataset • Black Box Data Fine-tuning: Inter Dataset • The proposed algorithm is evaluated on three datasets: CelebA, LFW, and MUCT [1] . Liu, Ziwei, et al. "Deep learning face attributes in the wild." Proceedings of the IEEE International Conference on Computer Vision. 2015. [2] . Huang, Gary B., et al. "Labeled faces in the wild: A database for studying face recognition in unconstrained environments." Workshop on faces in'Real-Life'Images: detection, alignment, and recognition. 2008. [3] . Milborrow, Stephen, John Morkel, and Fred Nicolls. "The MUCT landmarked face database." Pattern Recognition Association of South Africa 201.0 (2010).

19. Black Box Data Fine-tuning: Intra Dataset Table: Classification accuracy (%) before and after Data Fine-tuning for three attributes. Overall increase in the classification accuracies of all the attributes for both the datasets Table: Classification accuracy (%) of before and after Data Fine-tuning for ‘Gender’ attribute. Classification accuracy improved by 1% to 3% for all three datasets

20. Black Box Data Fine-tuning: Intra Dataset Probability Distribution • Smiling Score distribution • Overlapping region among both the classes reduced after Data Fine-tuning Score Figure: Smiling attribute score distribution pertaining to before and after Data Fine-tuning on LFW dataset

21. Black Box Data Fine-tuning: Intra Dataset Smiling Attribute Pale Skin Attribute Bushy Eyebrows Attribute Misclassified Before DFT Smiling Not Smiling Bushy Eyebrows Not Bushy Eyebrows Pale Skin Not Pale Skin Correctly Classified Before DFT Not Smiling Smiling Not Bushy Eyebrows Bushy Eyebrows Not Pale Skin Pale Skin

22. Black Box Data Fine-tuning: Inter Dataset Table: Classification accuracy (%) for other attributes Significant Improvement after Data Fine-tuning Table: Classification accuracy (%) for ‘Gender’ attribute. Classification accuracy increases by atleast 12% and upto 30 after Data Fine-tuning

23. Black Box Data Fine-tuning: Inter Dataset Dataset: LFW Model: CelebA Dataset: CelebA Model: LFW True Positive Rate False Positive Rate

24. Black Box Data Fine-tuning: Inter Dataset Before Data Fine-tuning Dataset: LFW Model trained on: CelebA: Dataset: CelebA Model trained on: LFW: Probability Distribution After Data Fine-tuning Score Figure: Score Distributions pertaining to before and after Data Fine-tuning

25. Model Fine-tuning vs Data Fine-tuning

26. Summary • Proposed a novel concept, Data Fine-tuning for enhancing the performance of black-box models. • Experiments are performed on CelebA, LFW, and MUCT databases. • The proposed concept, Data Fine-tuning uses adversarial perturbation to learn a single noise for a given dataset.

27. Thank you