imagenet-r

Dataset Card for ImageNet

Table of Contents

• Dataset Description
  • Dataset Summary
  • Supported Tasks
  • Languages
• Dataset Structure
  • Data Instances
  • Data Fields
  • Data Splits
• Dataset Creation
  • Curation Rationale
  • Source Data
  • Annotations
  • Personal and Sensitive Information
• Considerations for Using the Data
  • Social Impact of Dataset
  • Discussion of Biases
  • Other Known Limitations
• Additional Information
  • Dataset Curators
  • Licensing Information
  • Citation Information
  • Contributions

Dataset Description

• Homepage: https://image-net.org/index.php
• Repository:
• Paper: https://arxiv.org/abs/1409.0575
• Leaderboard: https://paperswithcode.com/sota/image-classification-on-imagenet?tag_filter=171
• Point of Contact: mailto: [email protected]

Dataset Summary

ILSVRC 2012, commonly known as 'ImageNet' is an image dataset organized according to the WordNet hierarchy. Each meaningful concept in WordNet, possibly described by multiple words or word phrases, is called a "synonym set" or "synset". There are more than 100,000 synsets in WordNet, majority of them are nouns (80,000+). ImageNet aims to provide on average 1000 images to illustrate each synset. Images of each concept are quality-controlled and human-annotated.

💡 This dataset provides access to ImageNet (ILSVRC) 2012 which is the most commonly used subset of ImageNet. This dataset spans 1000 object classes and contains 1,281,167 training images, 50,000 validation images and 100,000 test images. The version also has the patch which fixes some of the corrupted test set images already applied. For full ImageNet dataset presented in [2], please check the download section of the main website.

Supported Tasks and Leaderboards

• image-classification: The goal of this task is to classify a given image into one of 1000 ImageNet classes. The leaderboard is available here.

To evaluate the imagenet-classification accuracy on the test split, one must first create an account at https://image-net.org. This account must be approved by the site administrator. After the account is created, one can submit the results to the test server at https://image-net.org/challenges/LSVRC/eval_server.php The submission consists of several ASCII text files corresponding to multiple tasks. The task of interest is "Classification submission (top-5 cls error)". A sample of an exported text file looks like the following:

670 778 794 387 650
217 691 564 909 364
737 369 430 531 124
755 930 755 512 152

The export format is described in full in "readme.txt" within the 2013 development kit available here: https://image-net.org/data/ILSVRC/2013/ILSVRC2013_devkit.tgz. Please see the section entitled "3.3 CLS-LOC submission format". Briefly, the format of the text file is 100,000 lines corresponding to each image in the test split. Each line of integers correspond to the rank-ordered, top 5 predictions for each test image. The integers are 1-indexed corresponding to the line number in the corresponding labels file. See imagenet2012_labels.txt.

Languages

The class labels in the dataset are in English.

Dataset Structure

Data Instances

An example looks like below:

{
  'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=384x512 at 0x276021C5EB8>,
  'label': 23
}

Data Fields

The data instances have the following fields:

• image: A PIL.Image.Image object containing the image. Note that when accessing the image column: dataset[0]["image"] the image file is automatically decoded. Decoding of a large number of image files might take a significant amount of time. Thus it is important to first query the sample index before the "image" column, i.e. dataset[0]["image"] should always be preferred over dataset["image"][0].
• label: an int classification label. -1 for test set as the labels are missing.

The labels are indexed based on a sorted list of synset ids such as n07565083 which we automatically map to original class names. The original dataset is divided into folders based on these synset ids. To get a mapping from original synset names, use the file LOC_synset_mapping.txt available on Kaggle challenge page. You can also use dataset_instance.features["labels"].int2str function to get the class for a particular label index. Also note that, labels for test set are returned as -1 as they are missing.

<details> <summary> Click here to see the full list of ImageNet class labels mapping: </summary> </details>

Data Splits

Dataset Creation

Curation Rationale

The ImageNet project was inspired by two important needs in computer vision research. The first was the need to establish a clear North Star problem in computer vision. While the field enjoyed an abundance of important tasks to work on, from stereo vision to image retrieval, from 3D reconstruction to image segmentation, object categorization was recognized to be one of the most fundamental capabilities of both human and machine vision. Hence there was a growing demand for a high quality object categorization benchmark with clearly established evaluation metrics. Second, there was a critical need for more data to enable more generalizable machine learning methods. Ever since the birth of the digital era and the availability of web-scale data exchanges, researchers in these fields have been working hard to design more and more sophisticated algorithms to index, retrieve, organize and annotate multimedia data. But good research requires good resources. To tackle this problem at scale (think of your growing personal collection of digital images, or videos, or a commercial web search engine’s database), it was critical to provide researchers with a large-scale image database for both training and testing. The convergence of these two intellectual reasons motivated us to build ImageNet.

Source Data

Initial Data Collection and Normalization

Initial data for ImageNet image classification task consists of photographs collected from Flickr and other search engines, manually labeled with the presence of one of 1000 object categories. Constructing ImageNet was an effort to scale up an image classification dataset to cover most nouns in English using tens of millions of manually verified photographs 1. The image classification task of ILSVRC came as a direct extension of this effort. A subset of categories and images was chosen and fixed to provide a standardized benchmark while the rest of ImageNet continued to grow.

Who are the source language producers?

WordNet synsets further quality controlled by human annotators. The images are from Flickr.

Annotations

Annotation process

The annotation process of collecting ImageNet for image classification task is a three step process.

1. Defining the 1000 object categories for the image classification task. These categories have evolved over the years.
2. Collecting the candidate image for these object categories using a search engine.
3. Quality control on the candidate images by using human annotators on Amazon Mechanical Turk (AMT) to make sure the image has the synset it was collected for.

See the section 3.1 in 1 for more details on data collection procedure and 2 for general information on ImageNet.

Who are the annotators?

Images are automatically fetched from an image search engine based on the synsets and filtered using human annotators on Amazon Mechanical Turk. See 1 for more details.

Personal and Sensitive Information

The 1,000 categories selected for this subset contain only 3 people categories (scuba diver, bridegroom, and baseball player) while the full ImageNet contains 2,832 people categories under the person subtree (accounting for roughly 8.3% of the total images). This subset does contain the images of people without their consent. Though, the study in [1] on obfuscating faces of the people in the ImageNet 2012 subset shows that blurring people's faces causes a very minor decrease in accuracy (~0.6%) suggesting that privacy-aware models can be trained on ImageNet. On larger ImageNet, there has been an attempt at filtering and balancing the people subtree in the larger ImageNet.

Considerations for Using the Data

Social Impact of Dataset

The ImageNet dataset has been very crucial in advancement of deep learning technology as being the standard benchmark for the computer vision models. The dataset aims to probe models on their understanding of the objects and has become the de-facto dataset for this purpose. ImageNet is still one of the major datasets on which models are evaluated for their generalization in computer vision capabilities as the field moves towards self-supervised algorithms. Please see the future section in 1 for a discussion on social impact of the dataset.

Discussion of Biases

1. A study of the history of the multiple layers (taxonomy, object classes and labeling) of ImageNet and WordNet in 2019 described how bias is deeply embedded in most classification approaches for of all sorts of images.
2. A study has also shown that ImageNet trained models are biased towards texture rather than shapes which in contrast with how humans do object classification. Increasing the shape bias improves the accuracy and robustness.
3. Another study more potential issues and biases with the ImageNet dataset and provides an alternative benchmark for image classification task. The data collected contains humans without their consent.
4. ImageNet data with face obfuscation is also provided at this link
5. A study on genealogy of ImageNet is can be found at this link about the "norms, values, and assumptions" in ImageNet.
6. See this study on filtering and balancing the distribution of people subtree in the larger complete ImageNet.

Other Known Limitations

1. Since most of the images were collected from internet, keep in mind that some images in ImageNet might be subject to copyrights. See the following papers for more details: [1] [2] [3].

Additional Information

Dataset Curators

Authors of [1] and [2]:

• Olga Russakovsky
• Jia Deng
• Hao Su
• Jonathan Krause
• Sanjeev Satheesh
• Wei Dong
• Richard Socher
• Li-Jia Li
• Kai Li
• Sean Ma
• Zhiheng Huang
• Andrej Karpathy
• Aditya Khosla
• Michael Bernstein
• Alexander C Berg
• Li Fei-Fei

Licensing Information

In exchange for permission to use the ImageNet database (the "Database") at Princeton University and Stanford University, Researcher hereby agrees to the following terms and conditions:

1. Researcher shall use the Database only for non-commercial research and educational purposes.
2. Princeton University and Stanford University make no representations or warranties regarding the Database, including but not limited to warranties of non-infringement or fitness for a particular purpose.
3. Researcher accepts full responsibility for his or her use of the Database and shall defend and indemnify the ImageNet team, Princeton University, and Stanford University, including their employees, Trustees, officers and agents, against any and all claims arising from Researcher's use of the Database, including but not limited to Researcher's use of any copies of copyrighted images that he or she may create from the Database.
4. Researcher may provide research associates and colleagues with access to the Database provided that they first agree to be bound by these terms and conditions.
5. Princeton University and Stanford University reserve the right to terminate Researcher's access to the Database at any time.
6. If Researcher is employed by a for-profit, commercial entity, Researcher's employer shall also be bound by these terms and conditions, and Researcher hereby represents that he or she is fully authorized to enter into this agreement on behalf of such employer.
7. The law of the State of New Jersey shall apply to all disputes under this agreement.

Citation Information

@article{imagenet15russakovsky,
    Author = {Olga Russakovsky and Jia Deng and Hao Su and Jonathan Krause and Sanjeev Satheesh and Sean Ma and Zhiheng Huang and Andrej Karpathy and Aditya Khosla and Michael Bernstein and Alexander C. Berg and Li Fei-Fei},
    Title = { {ImageNet Large Scale Visual Recognition Challenge} },
    Year = {2015},
    journal   = {International Journal of Computer Vision (IJCV)},
    doi = {10.1007/s11263-015-0816-y},
    volume={115},
    number={3},
    pages={211-252}
}

Contributions

Thanks to @apsdehal for adding this dataset.