{"success":true,"data":{"author":"Wikimedia Research (images); Google Research (WIT dataset)","bucketAccessType":"public","bucketName":"wikimedia/wit_base","bucketPrefix":"","category":"Computer Vision","cloudProvider":"huggingface","createdAt":"2026-11-14T14:15:13.270Z","description":"WIT Base is the Wikimedia Research team's release of the image files behind the Wikipedia-based Image Text (WIT) dataset, the multilingual image-text corpus of more than 37 million image-text associations from Wikipedia articles in 108 languages published by Google Research (SIGIR 2021). Google's WIT gives only image names and URLs; Wikimedia released the pixels so that models can be trained directly. This copy contains more than six million images from Wikipedia articles in 100+ languages, resized to 300 px, covering nearly all captioned images in WIT that have a non-null reference description. Images where a person's face covers more than 10% of the frame were excluded for privacy (detected with RetinaFace), as were images that were candidates for deletion on Wikimedia Commons. Each row pairs an image with its reference and attribution descriptions, page and section text and URLs in its language, and was released for the Wikipedia image caption matching Kaggle challenge. Uses: multilingual image captioning, image-text retrieval, contrastive vision-language pretraining. Measured 2026-11-14: 330 Parquet shards in data/, about 308 GB; open with the datasets library (streaming recommended), pandas or pyarrow. Wikimedia releases it under CC BY-SA 4.0; credit Wikimedia and the WIT paper (Srinivasan et al. 2021).","displayTitle":"WIT Base: Wikimedia release of Wikipedia-based Image Text images","id":"3532b740-b1b6-503f-bae3-a55cb346a87d","license":"CC-BY-SA-4.0","licenseEvidenceUrl":"https://huggingface.co/datasets/wikimedia/wit_base","objectCountEstimate":335,"probeStatus":"ok","referenceUrl":"https://arxiv.org/abs/2103.01913","region":"us","schemaCard":{"accessHints":[{"language":"python-datasets","snippet":"from datasets import load_dataset\nds = load_dataset(\"wikimedia/wit_base\")"},{"language":"python-hub","snippet":"from huggingface_hub import snapshot_download\nsnapshot_download(repo_id=\"wikimedia/wit_base\", repo_type=\"dataset\")"}],"caveats":["HuggingFace metadata: author=wikimedia, license:cc-by-sa-4.0, size_categories:100K<n<1M, format:parquet, downloads=8262"],"format":"mixed","generatedAt":"2026-11-14T14:15:20.859Z","layout":{"example":"data/train-00000-of-00330.parquet","pattern":"{repoId}/{filename}"},"mountPath":"/mnt/archive","sampleHeaders":[{"bytes":"50415231150415c4e3800115b0b480014c153c1504120000e2b140f07504570000ffd8ffe000104a4649460001010100b400b40000ffdb00430006040506050406060506070706080a100a0a09090a140e0f0c1017141818171416161a1d251f1a1b231c1616202c20232627292a29191f2d302d283025282928ffdb0043010707070a080a130a0a13281a161a282828282828ae04006cffc000110800e1012c03012100021101031101ffc4001c000002030101af110158040502030607010008ffc400441000020102040404030501c2e8030100010203041100051221061331412251617114328107234291a11572b1c1d13334355273e1246282f0f1164353b2367492c2ffc400190100032e620004000005474405ffc4002f11000202010303030205050101011af4b6fe000102110304213112415113226171c123328191f00514a1b1e162d1ffda000c03010002110311003f00e6b1d3ea654249037b74c150d3c7b01a555b7d37bfe98cecba26d102c003ab7e8b8f9612c08be9dfb601d1735249145aca7536049dce231c05ad70700179a7097661a5079e2e86959e3d48094ff330db086111c0005b2ea71dfa7e98f952f3f2cda46f98201fa5f0ac0faa29d8ff0069641da25f2f7c7c201347f2f263f4ea700c22280a0291280187cc4ef8180894312eaeebd40dcdf0ac745e630a85d82a83b807adb128d58a8b46","encoding":"hex","key":"data/train-00000-of-00330.parquet","truncatedAt":512}],"sampleObjects":["data/train-00000-of-00330.parquet","data/train-00001-of-00330.parquet","data/train-00002-of-00330.parquet","data/train-00003-of-00330.parquet","data/train-00004-of-00330.parquet"],"sidecars":[{"bytes":"---\nannotations_creators:\n- machine-generated\nlanguage_creators:\n- found\nlanguage:\n- af\n- an\n- ar\n- arz\n- ast\n- az\n- azb\n- ba\n- bar\n- be\n- bg\n- bn\n- br\n- bs\n- ca\n- ce\n- ceb\n- ckb\n- cs\n- cv\n- cy\n- da\n- de\n- el\n- en\n- eo\n- es\n- et\n- eu\n- fa\n- fi\n- fil\n- fr\n- fy\n- ga\n- gl\n- hi\n- hr\n- hsb\n- ht\n- hu\n- hy\n- ia\n- id\n- io\n- is\n- it\n- iw\n- ja\n- jv\n- ka\n- kk\n- kn\n- ko\n- la\n- lah\n- lb\n- lmo\n- lt\n- lv\n- mg\n- mk\n- ml\n- mn\n- mr\n- ms\n- my\n- nan\n- nds\n- ne\n- nl\n- nn\n- 'no'\n- nv\n- oc\n- pa\n- pl\n- pt\n- qu\n- ro\n- ru\n- sco\n- si\n- sk\n- sl\n- sq\n- sr\n- sv\n- sw\n- ta\n- te\n- tg\n- th\n- tr\n- tt\n- uk\n- ur\n- uz\n- vec\n- vi\n- vo\n- war\n- xmf\n- yue\n- zh\nlicense:\n- cc-by-sa-4.0\nmultilinguality:\n- multilingual\nsize_categories:\n- 1M<n<10M\nsource_datasets:\n- original\n- extended|wikipedia\ntask_categories:\n- image-to-text\n- text-retrieval\ntask_ids:\n- image-captioning\npaperswithcode_id: wit\npretty_name: Wikipedia-based Image Text\nlanguage_bcp47:\n- af\n- an\n- ar\n- arz\n- ast\n- az\n- azb\n- ba\n- bar\n- be\n- be-tarask\n- bg\n- bn\n- br\n- bs\n- ca\n- ce\n- ceb\n- ckb\n- cs\n- cv\n- cy\n- da\n- de\n- el\n- en\n- eo\n- es\n- et\n- eu\n- fa\n- fi\n- fil\n- fr\n- fy\n- ga\n- gl\n- hi\n- hr\n- hsb\n- ht\n- hu\n- hy\n- ia\n- id\n- io\n- is\n- it\n- iw\n- ja\n- jv\n- ka\n- kk\n- kn\n- ko\n- la\n- lah\n- lb\n- lmo\n- lt\n- lv\n- mg\n- mk\n- ml\n- mn\n- mr\n- ms\n- my\n- nan\n- nds\n- ne\n- nl\n- nn\n- 'no'\n- nv\n- oc\n- pa\n- pl\n- pt\n- qu\n- ro\n- ru\n- sco\n- si\n- sk\n- sl\n- sq\n- sr\n- sr-Latn\n- sv\n- sw\n- ta\n- te\n- tg\n- th\n- tr\n- tt\n- uk\n- ur\n- uz\n- vec\n- vi\n- vo\n- war\n- xmf\n- yue\n- zh\n- zh-TW\ntags:\n- text-image-retrieval\n---\n\n# Dataset Card for WIT\n\n## Table of Contents\n- [Table of Contents](#table-of-contents)\n- [Dataset Description](#dataset-description)\n  - [Dataset Summary](#dataset-summary)\n  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)\n  - [Languages](#languages)\n- [Dataset Structure](#dataset-structure)\n  - [Data Instances](#data-instances)\n  - [Data Fields](#data-fields)\n  - [Data Splits](#data-splits)\n- [Dataset Creation](#dataset-creation)\n  - [Curation Rationale](#curation-rationale)\n  - [Source Data](#source-data)\n  - [Annotations](#annotations)\n  - [Personal and Sensitive Information](#personal-and-sensitive-information)\n- [Considerations for Using the Data](#considerations-for-using-the-data)\n  - [Social Impact of Dataset](#social-impact-of-dataset)\n  - [Discussion of Biases](#discussion-of-biases)\n  - [Other Known Limitations](#other-known-limitations)\n- [Additional Information](#additional-information)\n  - [Dataset Curators](#dataset-curators)\n  - [Licensing Information](#licensing-information)\n  - [Citation Information](#citation-information)\n  - [Contributions](#contributions)\n\n## Dataset Description\n\n- **Homepage:** [WIT homepage](https://github.com/google-research-datasets/wit)\n- **Paper:** [WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning\n](https://arxiv.org/abs/2103.01913)\n- **Leaderboard:** [WIT leaderboard](https://paperswithcode.com/sota/text-image-retrieval-on-wit) and [WIT Kaggle competition](https://www.kaggle.com/competitions/wikipedia-image-caption/leaderboard)\n- **Point of Contact:** [Miriam Redi](mailto:miriam@wikimedia.org)\n\n### Dataset Summary\n\nWikimedia's version of the Wikipedia-based Image Text (WIT) Dataset, a large multimodal multilingual dataset.\n\nFrom the [official blog post](https://techblog.wikimedia.org/2021/09/09/the-wikipedia-image-caption-matching-challenge-and-a-huge-release-of-image-data-for-research/):\n\n> The core training data is taken from the Wikipedia Image-Text (WIT) Dataset, a large curated set of more than 37 million image-text associations extracted from Wikipedia articles in 108 languages that was recently released by Google Research.\n>\n> The WIT dataset offers extremely valuable data about the pieces of text associated with Wikipedia images. However, due to licensing and data volume issues, the Google dataset only provides the image name and corresponding URL for download and not the raw image files.\n>\n> Getting easy access to the image files is crucial for participants to successfully develop competitive models. Therefore, t","encoding":"utf8","key":"README.md","name":"README.md","size":19753,"truncated":true}],"sourceObjectKey":"data/train-00000-of-00330.parquet","starterCell":"from datasets import load_dataset\nds = load_dataset(\"wikimedia/wit_base\")\nprint(ds)","starterSamplePath":"data/train-00000-of-00330.parquet","totalObjects":335},"summary":"WIT Base is the Wikimedia Research team's release of the image files behind the Wikipedia-based Image Text (WIT) dataset, the multilingual image-text corpus of more than 37…","title":"WIT Base: Wikimedia release of Wikipedia-based Image Text images","updatedAt":"2026-11-14T17:48:30.514Z","visibility":"public","websiteUrl":"https://techblog.wikimedia.org/2021/09/09/the-wikipedia-image-caption-matching-challenge-and-a-huge-release-of-image-data-for-research/"}}