{"success":true,"data":{"author":"Gao Laboratory","bucketAccessType":"public","bucketName":"Gaolaboratory/MSConsensus-100M","bucketPrefix":"","category":"Proteomics","cloudProvider":"huggingface","createdAt":"2026-11-17T14:12:21.745Z","description":"An ML-ready release of MSConsensus from the Gao laboratory: 102,000,000 consensus tandem mass spectra (100M train, 1M validation, 1M test) with peptide labels and precursor charge, drawn from the full 110,209,043-spectrum corpus. The corpus was built from 1.01 PB of public raw mass-spectrometry data in 1,500 PRIDE repositories covering Orbitrap and timsTOF platforms. A Weighted-Score Binning (WSBIN) consensus algorithm produces several consensus spectra per precursor, which suppresses instrument and batch effects so models generalize across vendors. Files are Parquet shards (train-00000-of-00400.parquet and so on, test and validation shards, manifest.json), about 190 GB, readable with pyarrow, polars or Hugging Face datasets. The complete corpus with per-peak annotations, provenance metadata and the random-access msz format is on Harvard Dataverse (doi:10.7910/DVN/NUCT4N), with the MSDatasets loader and a three-task benchmark (spectrum embedding retrieval, fragment intensity prediction, de novo peptide sequencing). Released under CC BY 4.0; every ingested PRIDE submission is CC BY 4.0 or CC0. Cite the MSConsensus paper.","displayTitle":"MSConsensus-100M: 102 million consensus MS/MS spectra from 1,500 PRIDE projects for proteomics machine learning","huggingFaceUrl":"https://huggingface.co/datasets/Gaolaboratory/MSConsensus-100M","id":"163295e6-3ecd-58bf-92d7-23ecc8a1a3d7","license":"CC-BY-4.0","licenseEvidenceUrl":"https://huggingface.co/datasets/Gaolaboratory/MSConsensus-100M","objectCountEstimate":411,"probeStatus":"ok","referenceUrl":"https://doi.org/10.7910/DVN/NUCT4N","region":"us","schemaCard":{"accessHints":[{"language":"python-datasets","snippet":"from datasets import load_dataset\nds = load_dataset(\"Gaolaboratory/MSConsensus-100M\")"},{"language":"python-hub","snippet":"from huggingface_hub import snapshot_download\nsnapshot_download(repo_id=\"Gaolaboratory/MSConsensus-100M\", repo_type=\"dataset\")"}],"caveats":["HuggingFace metadata: author=Gaolaboratory, license:cc-by-4.0, size_categories:100M<n<1B, format:parquet, downloads=760"],"format":"mixed","generatedAt":"2026-11-17T14:23:10.123Z","layout":{"example":"test-00000-of-00004.parquet","pattern":"{repoId}/{filename}"},"mountPath":"/mnt/archive","sampleHeaders":[{"bytes":"50415231150415b8a8800115dcb8704c158e8a20150012000028b52ffd80481c0a1000dcfc0d0e0060e56f2f10a83b6b303353d6ee2233333346555555d53b335a55a26266f6c13585b3767e76d1b6bdb4b4b4b4b4b42c8eb07498d76fd66fbc6fd6604d630dcd1ec6b015c013a229acd6f56435ab16ab458f58ad19c2695a39460b6b339a148bcfbc34f8ec0aca65d92626db1265055b4d5620bd62b1eb278b39c9188cdbc160d346d65e49592bc561ad138db3d230c672e98c9d32b2355bc1d627252fb3188cc58ce4034f381b70e6f0002bacaba20f4c55c5602aa6a6d7d4588f97d475c5ab490cc58ba94cd28b29ccfa42ea96781d8d11be955d31efa3d5eded738dbe7bae3bde3cdd2a8f12d8e5889c221d514fd411dda4378cac73413709177460e7809bec0e1899e580ce05ff931cfc9f981cf7fb13743f1b8beed716bd1fdb93fb3158f3be3fa2ce27a67abe4017e76b83703e0674cea7f5e17b9a20fc495bde9d34cdbc49263f5fa229cf974066f9123004bec4efcc97d428f225227c7c89d6a82bf922ea4abec0b9123fc62ba9f1e34a449c781248e8938071f42480099e44c499275970e04998cc23f913e8486a583a12119a8ea48cd0912c17fd0818c31f0114f12336b47e84e9e5462e316fe4afd08bf84170228f1e4ee44b8f0ff912e443fcf63ea446e087842075218ed80b39117d1046570f02a5fb202688","encoding":"hex","key":"test-00000-of-00004.parquet","truncatedAt":512}],"sampleObjects":["test-00000-of-00004.parquet","test-00001-of-00004.parquet","test-00002-of-00004.parquet","test-00003-of-00004.parquet","train-00000-of-00400.parquet"],"sidecars":[{"bytes":"---\npretty_name: MSConsensus-100M\nlicense: cc-by-4.0\nlanguage:\n  - en\nsize_categories:\n  - 100M<n<1B\ntask_categories:\n  - feature-extraction\n  - other\ntags:\n  - proteomics\n  - mass-spectrometry\n  - tandem-mass-spectrometry\n  - ms-ms\n  - peptide-identification\n  - de-novo-peptide-sequencing\n  - fragment-intensity-prediction\n  - spectral-embedding\n  - batch-effect\n  - consensus-spectra\n  - biology\n  - benchmark\nsource_datasets:\n  - PRIDE\nannotations_creators:\n  - machine-generated\nconfigs:\n  - config_name: default\n    data_files:\n      - split: train\n        path: train-*.parquet\n      - split: validation\n        path: validation-*.parquet\n      - split: test\n        path: test-*.parquet\ndataset_info:\n  features:\n    - name: mz\n      list: float32\n    - name: intensity\n      list: float32\n    - name: charge\n      dtype: int16\n    - name: peptide\n      dtype: string\n  splits:\n    - name: train\n      num_bytes: 209646639529\n      num_examples: 100000000\n    - name: validation\n      num_bytes: 2037239238\n      num_examples: 1000000\n    - name: test\n      num_bytes: 2097093389\n      num_examples: 1000000\n  download_size: 190374435985\n  dataset_size: 213780972156\n---\n\n# MSConsensus\n\n**A hundred-million-scale, batch-effect-suppressed dataset and benchmark for proteomics machine learning.**\n\nMSConsensus holds **110,209,043 consensus MS/MS spectra**. They were built from **1.01 PB** of public raw mass-spectrometry data taken from **1,500 PRIDE repositories** and covering all major Orbitrap and timsTOF platforms. A new consensus algorithm, **Weighted-Score Binning (WSBIN)**, produces *several* consensus spectra per precursor instead of one representative spectrum. This suppresses instrument-, lab- and protocol-specific batch effects while keeping biologically meaningful intensity structure and intra-cluster variance.\n\n> **This repository (`Gaolaboratory/MSConsensus-100M`)** holds a **102,000,000-spectrum** ML-ready release in Parquet: 100M train, 1M validation and 1M test spectra, about 190 GB compressed. Each row is one consensus spectrum with its peptide label and precursor charge. The full 110M corpus, with per-peak annotations, provenance metadata and `msz` files, is on [Harvard Dataverse](https://doi.org/10.7910/DVN/NUCT4N).\n\nThe full MSConsensus release ships with:\n\n- **`msz`**, a binary container with random access. Files are 45% of mzML size and load at 25,378 random-access spectra/s (3.2× HDF5).\n- **MSDatasets**, a drop-in PyTorch / JAX data loader.\n- A **three-task benchmark and public leaderboard** covering spectrum embedding retrieval, fragment-intensity prediction and de novo peptide identification.\n\n| | |\n|---|---|\n| **Primary hosting** | Harvard Dataverse — [doi:10.7910/DVN/NUCT4N](https://doi.org/10.7910/DVN/NUCT4N) |\n| **Leaderboard** | [msconsensus-leaderboard.netlify.app](https://msconsensus-leaderboard.netlify.app) |\n| **Paper** | *MSConsensus: A Hundred-Million-Scale, Batch-Effect–Suppressed Dataset and Benchmark for Proteomics Machine Learning* (under review, NeurIPS 2026) |\n| **License** | CC BY 4.0 |\n\n---\n\n## Dataset Summary\n\nProteomics machine learning faces three linked bottlenecks:\n\n1. **Noise and batch effects.** Models trained on raw PRIDE spectra learn the instrument as easily as they learn the peptide, so cross-vendor generalization fails.\n2. **I/O bottlenecks.** mzML (XML), MGF (text) and HDF5 were designed for analysis pipelines, not for shuffled GPU training.\n3. **Benchmark fragmentation.** Each method ships its own splits, preprocessing and metrics, so results from different papers can't be compared.\n\nMSConsensus addresses all three with one resource: a curated consensus corpus, an ML-native format and loader, and standardized held-out splits with a shared evaluation suite.\n\n### Comparison with existing resources\n\n| Resource | # Spectra | Batch-split | Per-peak annotation | PTM split | Data loader |\n|---|---|---|---|---|---|\n| MassIVE-KB v2 | 5.9 M | partial | ✗ | ✗ | ✗ |\n| NIST Hybrid Libs | 1.3 M | ✗ | partial | partial | ✗ |\n| ProteomeTools | 0.9 M | ✗ | ✓ | ✓ (synth.) | �","encoding":"utf8","key":"README.md","name":"README.md","size":17248,"truncated":true},{"bytes":"{\n  \"dataset\": \"MSConsensus-100M\",\n  \"schema_version\": 1,\n  \"schema\": [\n    {\n      \"name\": \"mz\",\n      \"type\": \"list<item: float>\",\n      \"nullable\": false\n    },\n    {\n      \"name\": \"intensity\",\n      \"type\": \"list<item: float>\",\n      \"nullable\": false\n    },\n    {\n      \"name\": \"charge\",\n      \"type\": \"int16\",\n      \"nullable\": false\n    },\n    {\n      \"name\": \"peptide\",\n      \"type\": \"string\",\n      \"nullable\": false\n    }\n  ],\n  \"seed\": 42,\n  \"source_fingerprint\": {\n    \"digest\": \"0a33479ceadd521aa54ed1f43c6e44ff3ec24d5767ceea3aec6cd61e4bc49a96\",\n    \"files\": [\n      {\n        \"mtime_ns\": 1778014454118378614,\n        \"name\": \"consensus_00.parquet\",\n        \"size\": 2377703127\n      },\n      {\n        \"mtime_ns\": 1778014467993476743,\n        \"name\": \"consensus_01.parquet\",\n        \"size\": 2442558476\n      },\n      {\n        \"mtime_ns\": 1778014481504598530,\n        \"name\": \"consensus_02.parquet\",\n        \"size\": 2376732198\n      },\n      {\n        \"mtime_ns\": 1778014494591747873,\n        \"name\": \"consensus_03.parquet\",\n        \"size\": 2355586484\n      },\n      {\n        \"mtime_ns\": 1778014507546905794,\n        \"name\": \"consensus_04.parquet\",\n        \"size\": 2406253714\n      },\n      {\n        \"mtime_ns\": 1778014520694051240,\n        \"name\": \"consensus_05.parquet\",\n        \"size\": 2374584868\n      },\n      {\n        \"mtime_ns\": 1778014533145241923,\n        \"name\": \"consensus_06.parquet\",\n        \"size\": 2305313191\n      },\n      {\n        \"mtime_ns\": 1778014545912412064,\n        \"name\": \"consensus_07.parquet\",\n        \"size\": 2387581486\n      },\n      {\n        \"mtime_ns\": 1778014559795509670,\n        \"name\": \"consensus_08.parquet\",\n        \"size\": 2432214962\n      },\n      {\n        \"mtime_ns\": 1778014572962653813,\n        \"name\": \"consensus_09.parquet\",\n        \"size\": 2376315975\n      },\n      {\n        \"mtime_ns\": 1778014586141797177,\n        \"name\": \"consensus_10.parquet\",\n        \"size\": 2382222484\n      },\n      {\n        \"mtime_ns\": 1778014599328940021,\n        \"name\": \"consensus_11.parquet\",\n        \"size\": 2354899857\n      },\n      {\n        \"mtime_ns\": 1778014612760067006,\n        \"name\": \"consensus_12.parquet\",\n        \"size\": 2433030071\n      },\n      {\n        \"mtime_ns\": 1778014625055267824,\n        \"name\": \"consensus_13.parquet\",\n        \"size\": 2314833136\n      },\n      {\n        \"mtime_ns\": 1778014637914431985,\n        \"name\": \"consensus_14.parquet\",\n        \"size\": 2374203687\n      },\n      {\n        \"mtime_ns\": 1778014650777595888,\n        \"name\": \"consensus_15.parquet\",\n        \"size\": 2359031125\n      },\n      {\n        \"mtime_ns\": 1778014663120793588,\n        \"name\": \"consensus_16.parquet\",\n        \"size\": 2273034316\n      },\n      {\n        \"mtime_ns\": 1778014676091950471,\n        \"name\": \"consensus_17.parquet\",\n        \"size\": 2427829983\n      },\n      {\n        \"mtime_ns\": 1778014688727129190,\n        \"name\": \"consensus_18.parquet\",\n        \"size\": 2422032469\n      },\n      {\n        \"mtime_ns\": 1778014702378241875,\n        \"name\": \"consensus_19.parquet\",\n        \"size\": 2466983633\n      },\n      {\n        \"mtime_ns\": 1778014715037419038,\n        \"name\": \"consensus_20.parquet\",\n        \"size\": 2319765680\n      },\n      {\n        \"mtime_ns\": 1778014728756527303,\n        \"name\": \"consensus_21.parquet\",\n        \"size\": 2391656612\n      },\n      {\n        \"mtime_ns\": 1778014742523632448,\n        \"name\": \"consensus_22.parquet\",\n        \"size\": 2413226656\n      },\n      {\n        \"mtime_ns\": 1778014755934760800,\n        \"name\": \"consensus_23.parquet\",\n        \"size\": 2394216785\n      },\n      {\n        \"mtime_ns\": 1778014768997912058,\n        \"name\": \"consensus_24.parquet\",\n        \"size\": 2422620491\n      },\n      {\n        \"mtime_ns\": 1778014782393041751,\n        \"name\": \"consensus_25.parquet\",\n        \"size\": 2375132280\n      },\n      {\n        \"mtime_ns\": 1778014795772172482,\n        \"name\": \"consensus_26.parquet\",\n        \"size\": 2369507954\n      },\n      {\n        \"mtime_ns\": 1778014808627337257,\n        \"name\": \"consensus_27.parquet\",\n        \"size\": 2353750777\n      },\n      {\n        \"mtime","encoding":"utf8","key":"manifest.json","name":"manifest.json","size":100122,"truncated":true}],"sourceObjectKey":"test-00000-of-00004.parquet","starterCell":"from datasets import load_dataset\nds = load_dataset(\"Gaolaboratory/MSConsensus-100M\")\nprint(ds)","starterSamplePath":"test-00000-of-00004.parquet","totalObjects":411},"summary":"An ML-ready release of MSConsensus from the Gao laboratory: 102,000,000 consensus tandem mass spectra (100M train, 1M validation, 1M test) with peptide labels and precursor…","title":"MSConsensus-100M: 102 million consensus MS/MS spectra from 1,500 PRIDE projects for proteomics machine learning","updatedAt":"2026-11-17T20:23:07.930Z","visibility":"public","websiteUrl":"https://huggingface.co/datasets/Gaolaboratory/MSConsensus-100M"}}