Evaluating Transfer Learning Approaches for Image information mining applications

1. Evaluating Transfer Learning Approaches for Image information mining applications Surya Durbha*, Roger King, Nicolas Younan, *Indian Institute of Technology(IIT), Bombay Center for Advanced Vehicular Systems (CAVS) Department of Electrical and Computer Eng. Mississippi State University, USA

2. Outline • Background • Image information Mining • Transfer Learning • Knowledge Transfer in EO • Methodology • Results • Summary

3. Image Information Mining Image information mining (IIM) of remote sensing data deals with the retrieval and analysis of image content in an image using various supervised, semi-supervised, and unsupervised classification methods. In IIM applications, the goal is to link the semantic conceptualization of a phenomenon, usually represented by land use/ land cover classes, with lower level image features. The “semantic gap” between the lower level features and higher level conceptual representation is usually reduced using a variety of semantics-based techniques.

4. Understanding Related Knowledge The challenge with human analysis of imagery is that human possesses an information channel that is bandwidth limited. The net result is an inability to cope with the information content in the imagery and any associated co relatable sensors due to the breadth and the quality of the data sources themselves. However, the intuitiveness of the human mind is unparalleled and has the ability to make associations between similar objects which is very hard to replicate in a machine interface.

5. Capability to Make Associations For example, the human mind can look at a region of agriculture, grass land, forest patch, and shrub land to make inferences about the similarity or dissimilarity between various regions The use of prior knowledge and contextual information about these land cover classes in making these distinctions forms a part of the human mind’s approach for pattern recognition Also, there is the ability to transfer knowledge between entities that are similar irrespective of them belonging to the same or interrelated domains.

6. Knowledge Transfer in EO Domains • The active transfer of knowledge between various classes (entities) is vital in many situations in EO domain such as: • Disaster Response • Change detection • Land Use/Land Cover • Thematic information translation from one classification system to another

7. Transfer Learning for Disaster Response • Disaster response scenarios where the knowledge of a particular previous disaster in a closely related domain might be able to help in damage assessment in a new disaster. • For example, understanding the urban classes (buildings, streets, bridges etc) and a post earthquake identification of these classes, should help to provide an insight into how it would look after a closely related disaster (urban warfare, terrorist attacks, floods etc.)

8. Learning new models based on transferring prior knowledge of similar classes between closely related tasks or domains.

9. Transfer Learning for Change Detection Studies Change detection studies use continuously updated information to update the databases and the maps that are produced from them. However, to update these databases with new information pertaining to the current time period, it may not be always possible to obtain a large amount of labeled samples for a supervised classification. In this situation, transferring the already available knowledge from a source task and using it to classify a target task at the current time using only a few labeled samples is useful.

10. Transfer Learning in a Coastal Disaster Scenario This work is focused on applying transfer learning and studying the inter and intra domain knowledge transfer capabilities in a coastal disaster scenario. For this purpose, we adapt a transfer learning methodology based on a modified Weighted Least Squares Support Vector Machines (WLS-SVM) and investigate the recognition rate on small sample sizes of the target classes.

11. Methodology In traditional data mining the assumption is that the training and testing data are in the same distribution, which is not true in several real world situations, especially in the EO domain where there is dynamically changing information, although contextually it is the same entity, but in a slightly different form. For example, in a coastal disaster event, an “agriculture land” could change into a “flooded agriculture land”. Here the data distributions would be different, but the underlying concept is very similar. Semi-supervised methods allow the use of unlabeled data from various spatial databases to augment the paucity of labeled testing (target) data, but they also are constrained by the same data distribution issues.

12. Methodology (Cont.) In this work, we apply and evaluate the transfer method based on transferring instance information (proposed by Tomassi et al,2010, Oraborna et al., 2009) The approach is based on a modified WLS-SVM method to induce transfer between instances and is based on constraining the hyperplanes (LS-SVM) of new category to be close to those of a subset of the previously learned classes. The least squares SVM formulation enables the closed form of Leave one out (LOO) error and can be used for model selection due to its unbiased estimate of the generalization error. This formulation is adapted to constrain a new model to be close to the pre-trained model.

13. Transferring Prior Knowledge • The transfer of prior knowledge between a set of sources classes and a target class could be accomplished in two ways: • Inductive transfer learning in which the knowledge is transferred in situations where the domains could be different or the same but the source and target tasks are different. • Transductive transfer refers to the situations in which the domains are different but the tasks that need to be learned are the same.

14. Approaches • A common approach in transfer learning is the identification of three cases before applying the process; “what to transfer”, “when to transfer”, and “how to transfer” • The first case refers to the kind of entities that are transferred between tasks; these could be grouped into transferring: • Knowledge of Instances • Knowledge of Features • Knowledge of Parameters • Relational Knowledge

15. Transfer Learning in Coastal Disasters Applications In a coastal disaster event such as a hurricane, floods, and other weather events, it is important to provide rapid response. The normal machine learning algorithms require a good amount of training data to develop a reasonable good classification model. However, immediately after a disaster, the data is sparse and only few samples are available and using which it is necessary to train the classification algorithms. To provide rapid response using maps updated by remote sensing imagery, there is a need to be able to build models using small samples. Also, using prior knowledge in such a way as to enhance the learning of a new class would prove to be very useful.

16. Datasets To demonstrate this scenario, we have used the data sets from pre and post Katrina Landsat ETM+ imagery We selected six land cover classes, i.e., agriculture, fallow, flooded agriculture and developed areas, flooded forest areas. The goal is to assess the ability of transfer learning of a target class with samples as small as 8 and also understand the interaction between the classes for their ability to exchange prior knowledge

17. Results Recognition rate for 6 classes (agriculture, fallow, flooded agriculture and developed, flooded forest, forest, water) The case when all the 6 classes (related and unrelated) were used in a LOO approach. It can be seen that both MKT and AKT performed much better than WLS-SVM and was able to provide fairly good recognition rate for small target sample sizes. It is evident that prior knowledge between the categories has helped in better detection.

18. Results Recognition rate for 4 classes (water, forest, agriculture, flooded agriculture developed). The number of classes (categories) have been reduced to 4 (randomly removed 2 classes) to assess the transferability of knowledge, assuming that certain classes could be responsible for negative transfer. However, the results show that the recognition rate is slightly less than for 6 classes (for a sample size of 8). But it is not clear which of the classes have contributed to the reduction in the recognition rate, more simulations and careful selection of the classes might help in a better assessment of these aspects.

19. Summary There is an increasing need to develop methods that work on interdisciplinary and interrelated data to provide a holistic perspective for decision making. So the need for models that are able to adapt, use and transfer knowledge across the domains is important. The adapted approaches for transfer learning in this work have demonstrated their usefulness in practical applications such as disaster recovery. However, more work is needed to understand the specific situations under which transferability of knowledge is possible and also on negative transfer issues.

20. Thank you!