{"success":true,"data":{"author":"Cornell University arXiv; mirrored by Hugging Face librarian-bots","bucketAccessType":"public","bucketName":"librarian-bots/arxiv-metadata-snapshot","bucketPrefix":"","category":"Machine Learning & AI","cloudProvider":"huggingface","createdAt":"2026-11-14T14:15:20.121Z","description":"Descriptive metadata for every paper on arXiv, mirrored weekly from the arXiv metadata snapshot that Cornell University publishes (the metadata portion of the Kaggle arXiv dataset). Each row is one e-print with its arXiv id, submitter, author list, title, comments (pages, figures), journal reference, DOI, report number, subject categories, licence, abstract, version history with dates and update date. Full texts are not included; arxiv.org/abs/{id} and arxiv.org/pdf/{id} resolve each paper. Listing data/ (measured 2026-11-14) shows 11 Parquet shards of about 0.17 to 0.28 GB, roughly 2.7 GB in total; the repository's stored history is much larger because each weekly sync adds a revision. The data support scientometrics and science-of-science studies, topic modelling and classification of papers into arXiv categories, author and collaboration analysis, citation-free trend tracking, literature search and recommendation, and building embeddings for retrieval. Load with Hugging Face datasets, pandas, polars or DuckDB. arXiv makes its descriptive metadata available under the CC0 1.0 Public Domain Dedication; paper full texts carry their own licences.","displayTitle":"arXiv Metadata Snapshot (weekly mirror of the Cornell arXiv OAI dataset)","id":"4438a4e7-1990-52c8-aa77-2867a720b974","license":"CC0-1.0","licenseEvidenceUrl":"https://info.arxiv.org/help/api/tou.html","objectCountEstimate":13,"probeStatus":"ok","referenceUrl":"https://www.kaggle.com/datasets/Cornell-University/arxiv","region":"us","schemaCard":{"accessHints":[{"language":"python-datasets","snippet":"from datasets import load_dataset\nds = load_dataset(\"librarian-bots/arxiv-metadata-snapshot\")"},{"language":"python-hub","snippet":"from huggingface_hub import snapshot_download\nsnapshot_download(repo_id=\"librarian-bots/arxiv-metadata-snapshot\", repo_type=\"dataset\")"}],"caveats":["HuggingFace metadata: author=librarian-bots, license:cc0-1.0, size_categories:1M<n<10M, format:parquet, downloads=8371"],"format":"mixed","generatedAt":"2026-11-14T14:15:25.314Z","layout":{"example":"data/train-00000-of-00011.parquet","pattern":"{repoId}/{filename}"},"mountPath":"/mnt/archive","sampleHeaders":[{"bytes":"50415231150415e8a66c15daa4284c159ede071500120000b493363009000000303930352e30303334150d0c34393530010d1c313130362e323638110d14372e33323739051a1c3330352e34323931050d1c3430332e373637360d0d4c392e363234370a000000313530322e3035303536050e203631322e3039323935050e203730312e30333030310d0e18352e3038303630050e1c3830332e3033343009461c3831302e31313533092a1c3930372e30363738091c010e10313238343801380432300138083035340562090e043830092a24323030342e30323737330d2a14372e30393433054620323030382e31333634091c1c3130352e3130393609460031052a043031092a08313130019a0438390546050e08383434059a20323131312e313139360d46010e0c343235340d2a18322e3036333632150e0c31313136251804323201fc083530310946090e043132097e090e04373709701c3230362e31303737111c003701a82918043232010e0832303809701c3230382e3038363909701c3230392e303039380d70018c0831393509700033218808313732110e217a043533295021e70c31333930110e01700431371d0e083539390954003301700836353509a8083330390138003629340033213408313837098c050e0433352d5e050e083738330d3804313125dc1d0e08383533113814312e3132333345220032","encoding":"hex","key":"data/train-00000-of-00011.parquet","truncatedAt":512}],"sampleObjects":["data/train-00000-of-00011.parquet","data/train-00001-of-00011.parquet","data/train-00002-of-00011.parquet","data/train-00003-of-00011.parquet","data/train-00004-of-00011.parquet"],"sidecars":[{"bytes":"---\nlanguage:\n- en\nlicense: cc0-1.0\nsize_categories:\n- 1M<n<10M\ntask_categories:\n- text-generation\n- text-classification\npretty_name: arXiv Metadata Dataset\nconfigs:\n- config_name: default\n  data_files:\n  - split: train\n    path: data/train-*\ndataset_info:\n  features:\n  - name: id\n    dtype: string\n  - name: submitter\n    dtype: string\n  - name: authors\n    dtype: string\n  - name: title\n    dtype: string\n  - name: comments\n    dtype: string\n  - name: journal-ref\n    dtype: string\n  - name: doi\n    dtype: string\n  - name: report-no\n    dtype: string\n  - name: categories\n    dtype: string\n  - name: license\n    dtype: string\n  - name: abstract\n    dtype: string\n  - name: versions\n    list:\n    - name: version\n      dtype: string\n    - name: created\n      dtype: string\n  - name: update_date\n    dtype: timestamp[s]\n  - name: authors_parsed\n    list:\n      list: string\n  splits:\n  - name: train\n    num_bytes: 5088171848\n    num_examples: 3224635\n  download_size: 2700966734\n  dataset_size: 5088171848\ntags:\n- arxiv\n- science\n---\n# Dataset Card for \"arxiv-metadata-oai-snapshot\"\n\n[More Information needed](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)\n\nThis is a mirror of the metadata portion of the arXiv [dataset](https://www.kaggle.com/datasets/Cornell-University/arxiv/versions/147). \n\nThe sync will take place weekly so may fall behind the original datasets slightly if there are more regular updates to the source dataset. \n\n\n## Metadata\n\nThis dataset is a mirror of the original ArXiv data. This dataset contains an entry for each paper, containing:\n\n- id: ArXiv ID (can be used to access the paper, see below)\n- submitter: Who submitted the paper\n- authors: Authors of the paper\n- title: Title of the paper\n- comments: Additional info, such as number of pages and figures\n- journal-ref: Information about the journal the paper was published in\n- doi: [https://www.doi.org](Digital Object Identifier)\n- abstract: The abstract of the paper\n- categories: Categories / tags in the ArXiv system\n- versions: A version history\n\nYou can access each paper directly on ArXiv using these links:\n\n- `https://arxiv.org/abs/{id}`: Page for this paper including its abstract and further links\n- `https://arxiv.org/pdf/{id}`: Direct link to download the PDF\n\n","encoding":"utf8","key":"README.md","name":"README.md","size":2319,"truncated":false}],"sourceObjectKey":"data/train-00000-of-00011.parquet","starterCell":"from datasets import load_dataset\nds = load_dataset(\"librarian-bots/arxiv-metadata-snapshot\")\nprint(ds)","starterSamplePath":"data/train-00000-of-00011.parquet","totalObjects":13},"summary":"Descriptive metadata for every paper on arXiv, mirrored weekly from the arXiv metadata snapshot that Cornell University publishes (the metadata portion of the Kaggle arXiv…","title":"arXiv Metadata Snapshot (weekly mirror of the Cornell arXiv OAI dataset)","updatedAt":"2026-11-14T17:46:07.962Z","visibility":"public","websiteUrl":"https://huggingface.co/datasets/librarian-bots/arxiv-metadata-snapshot"}}