Data-Free Sketch-Based Image Retrieval

Abhra Chaudhuri1, Ayan Kumar Bhunia2, Yi-Zhe Song2 Anjan Dutta2
1University of Exeter, UK      2Institute for People-Centered AI, University of Surrey, UK
Paper Code

Our proposed Data-Free setting for SBIR does not need a real-world dataset of paired sketches and photos. Using only independently trained, modality-specific classifiers, it can estimate their train set distributions, as well as pair them at class-level for training the sketch and photo encoders.

Video

Poster

 

Abstract

Rising concerns about privacy and anonymity preservation of deep learning models have facilitated research in data-free learning (DFL). For the first time, we identify that for data-scarce tasks like Sketch-Based Image Retrieval (SBIR), where the difficulty in acquiring paired photos and hand-drawn sketches limits data-dependent crossmodal learning algorithms, DFL can prove to be a much more practical paradigm. We thus propose Data-Free (DF)- SBIR, where, unlike existing DFL problems, pre-trained, single-modality classification models have to be leveraged to learn a cross-modal metric-space for retrieval without access to any training data. The widespread availability of pre-trained classification models, along with the difficulty in acquiring paired photo-sketch datasets for SBIR justify the practicality of this setting. We present a methodology for DF-SBIR, which can leverage knowledge from models independently trained to perform classification on photos and sketches. We evaluate our model on the Sketchy, TUBerlin, and QuickDraw benchmarks, designing a variety of baselines based on state-of-the-art DFL literature, and observe that our method surpasses all of them by significant margins. Our method also achieves mAPs competitive with data-dependent approaches, all the while requiring no training data. Implementation is available at https://github.com/abhrac/data-free-sbir.

Method

The photo and sketch estimators reconstruct the train set distributions of their corresponding classifiers, which could then be used to train the downstream encoders. The estimation process is ensured to have semantic consistency and completeness (through adversarial estimation) and class-level, instance-wise correspondence while maintaining modality boundaries.
 

Results

this slowpoke moves
Data-free photo and sketch reconstructions of the Sketchy and TU-Berlin datasets produced by our estimator networks.


this slowpoke moves
Photo and sketch reconstructions without (left) and with (right) the Class-Alignment loss. Each column corresponds to a single reconstruction step using a common input noise vector fed in to the photo and sketch estimators respectively.


this slowpoke moves
Reconstructed photos and sketches of an Apple in the presence and absence of the Modality Guidance loss.


this slowpoke moves
Reconstructions obtained by using Metric-Agnostic Adversarial Estimation, with respective class-scores assigned by the teacher and the student.


this slowpoke moves
DF-SBIR performance of our model when the classifiers (teachers) are trained on only partially overlapping sets of classes.


this slowpoke moves
Variation in mAP@all, as well as reconstruction quality across epochs.


BibTeX

@article{chaudhuri2023dfsbir,
  title={{Data-Free Sketch-Based Image Retrieval}},
  author={Abhra Chaudhuri and Ayan Kumar Bhunia and Yi-Zhe Song and Anjan Dutta},
  booktitle={CVPR},
  year={2023}
}

Last updated: 06 April 2023 | Template Credit: Nerfies