Relative Attributes

Relative Attributes Presenter: ShuaiZheng (Kyle) Supervised by Philip H.S. Torr Author: Devi Parikh (TTI-Chicago) and Kristen Grauman (UT-Austin)

What is visual attributes? • Attributes are properties observable in images that have human-designated names, such as ‘Orange’, ‘striped’, or ‘Furry’.

Learning Binary Attributes • In PASCAL VOC Challenge, we learn to predict binary attributes. (e.g., dog? Or not a dog?) Not a cat dog O. Parkhi, A.VedaldiC.V.Jawahar, A.Zisserman. The Truth About Cats and Dogs. ICCV 2011. Vittorio Ferrari, Andrew Zisserman. Learning Visual Attributes. NIPS 2007.

Problems within Binary Attributes • Given an attribute it is easy to get labelled data on AMT(Amazon Mechanical Turk). • But, where do attributes come from? Can we find a easier way to ask more people rather than experts to tag the images?

Problems within Binary AttributesSome tags are binary while some are relative. Is furry Has four-legs Binary Legs shorter than horses’ Tail longer than donkeys’ Mule relative Has tail

Labeling data

What is relative attributes? • Relative attribute indicates the strength of an attribute in an image with respect to other image rather than simply predicting the presence of an attribute.

Advantages of Relative Attributes • Enhanced human-machine communication • More informative • Natural for humans

Contributions • Propose a model to learn relative attributes • Allow relating images and categories to each other • Learn ranking function for each attribute • Give two novel applications based on the model • Zero-shot learning from attribute comparisons • Automatically generating relative image Marr Prize 2011!

Learning Relative Attributes For each attribute open Supervision is

Learning Relative Attributes Learn a scoring function Image features that best satisfies constraints: Learned parameters

Learning Relative Attributes Max-margin learning to rank formulation 2 1 4 3 6 5 Based on [Joachims 2002] Rank Margin Image Relative Attribute Score

Learning binary attributes v.s. Learning relative attributes

Relative Zero-shot Learning Training: Images from S seencategories and Descriptions ofU unseen categories Need not use all attributes, or all seen categories Testing: Categorize image into one of S+U categories Scarlett Hugh Clive Miley Jared Age: Jared Miley Smiling:

Relative Zero-shot Learning Can predict new classes based on their relationships to existing classes – without training images Age: Scarlett Hugh Clive S Miley Jared Clive Jared Miley Smiling: Smiling Miley J H Age Infer image category using max-likelihood

Automatic Relative Image Description Novel image Density Conventional binary description: not dense Dense: Not dense:

Automatic Relative Image Description Novel image Density less dense than more dense than

Automatic Relative Image Description Novel image Density C C H H H C F H H M F F I F more dense than Highways, less dense than Forests

Datasets Public Figures Face (PubFig) [Kumar 2009] 8 classes, ~800 images, Gist+color 11 attributes: white, chubby, etc. Outdoor Scene Recognition (OSR) [Oliva 2001] 8 classes, ~2700 images, Gist 6 attributes: open, natural, etc. Attributes labeled at category level http://ttic.uchicago.edu/~dparikh/relative.html

Baselines • Zero-shot learning • Binary attributes: Direct Attribute Prediction [Lampert 2009] • Relative attributes via classifier scores • Automatic image-description • Binary attributes – 1 – 2 – 3 + 4 + 5 + 6

Relative Zero-shot Learning Rel. att.(ranker) Rel. att. (classifier) Binary attributes An attribute is more discriminative when used relatively

Automatic Relative Image Description Binary (existing): Not natural Not open Has perspective Relative (ours): More natural than insidecity Less natural than highway More open than street Less open than coast Has more perspective than highway Has less perspective than insidecity

Automatic Relative Image Description • 18 subjects • Test cases: • OSR, 20 PubFig

Traditional Recognition Tiger ??? Dog Chimpanzee Tiger

Attributes-based Recognition Attributes provide a mode of communication between humans and machines! Striped Black White Big Furry White Black Big Striped Yellow Dog Chimpanzee Tiger [Lampert 2009] [Farhadi 2009] [Kumar 2009] [Berg 2010] [Parikh 2010] … Zero-shot learning Describing objects Face verification Attribute discovery Nameable attributes …

Conclusions and Future Work Relative attributes learnt as ranking functions • Natural and accurate zero-shot learning of novel concepts by relating them to existing concepts • Precise image descriptions for human interpretation Attributes-based recognition is an interesting direction for the future object/scenes recognition. Enhanced human-machine communication

Cheers! – ShuaiZheng (Kyle)kylezheng04@gmail.com Created by Tag clouds

Relative Attributes