{"success":true,"data":{"author":"Romero Lab, Duke University","bucketAccessType":"public","bucketName":"RomeroLab-Duke/prism-antibody-data","bucketPrefix":"","category":"Biology","cloudProvider":"huggingface","createdAt":"2026-11-22T01:09:20.802Z","description":"Curated human antibody sequence data used to train PRISM, a germline-aware antibody language model from the Romero Lab at Duke University. Sequences come from the Observed Antibody Space (OAS), human studies only (70 studies, 2021 to 2022 data freeze), passed through quality filters (missing conserved cysteine, heavy fragmentation, non-canonical amino acids), exact deduplication, a somatic mutation filter (more than 3 non-germline mutations in heavy and 2 in light chains), CDR3 and whole-sequence clustering, and ANARCI re-numbering. Two Parquet files hold 66,435,576 unpaired heavy and light chains for pretraining and 763,989 paired VH and VL antibodies for fine-tuning, with amino acid and nucleotide sequences, germline alignments, B-cell type, somatic mutation codes and counts, V and J gene calls, IMGT region masks, cluster IDs and a train/valid/test split column. The OAS download manifest is included; most of the Briney 2019 heavy-chain study was left out for size. Use it for antibody language models, somatic hypermutation and B-cell repertoire analysis. Derived from OAS under CC BY 4.0; cite Olsen et al. 2022, Protein Science, and PRISM.","displayTitle":"PRISM Antibody Training Data: 66 Million Unpaired and 764k Paired Human Antibody Sequences from OAS with Germline Labels (Romero Lab, Duke)","huggingFaceUrl":"https://huggingface.co/datasets/RomeroLab-Duke/prism-antibody-data","id":"631526b8-edff-5de6-9103-c0d91e1aec08","license":"CC-BY-4.0","licenseEvidenceUrl":"https://huggingface.co/datasets/RomeroLab-Duke/prism-antibody-data","objectCountEstimate":5,"probeStatus":"ok","referenceUrl":"https://opig.stats.ox.ac.uk/webapps/oas/","region":"us","schemaCard":{"accessHints":[{"language":"python-datasets","snippet":"from datasets import load_dataset\nds = load_dataset(\"RomeroLab-Duke/prism-antibody-data\")"},{"language":"python-hub","snippet":"from huggingface_hub import snapshot_download\nsnapshot_download(repo_id=\"RomeroLab-Duke/prism-antibody-data\", repo_type=\"dataset\")"}],"caveats":["HuggingFace metadata: author=RomeroLab-Duke, license:cc-by-4.0, size_categories:10M<n<100M, format:parquet, downloads=172"],"format":"mixed","generatedAt":"2026-11-22T01:10:20.831Z","layout":{"example":"paired_anarci_relabeled.parquet","pattern":"{repoId}/{filename}"},"mountPath":"/mnt/archive","sampleHeaders":[{"bytes":"5041523115041590c08d0115bed3474c15e28f01150012000088e046f0b677000000514c514c5145534750474c564b505345544c534c5443545653474753495353525359595747574952515050474b474c45574947534959465347535459594e50534c4b535256544953474454534b4e5146534c4b4c53535654414144544156595943414748534c594e5347575057475147544c5654565353790000004556514c4c45534747474c5651504747534c524c534341415347465446535459414d54575652514150474b474c4557565347495347534747017a74414453564b47524654495352444e534b4e544c594c514d4e534c524145440d7a384b445253545947444e504644595747157d087d0000017d384d564553474747465651504752534c017d0053097d20474746415653575652157d404147494952534541484747545444594141097f285349534b4447534b5449410d7f38544945445447565959435452455346016218595256454641500581005605fe0076217904564c5a7901005921791047594657412e7801144647534d535921970444593d780056397804564b32780118525252534c594805fb0c4c47564c057a007b057a04514c09fb005601fb004709fb004109fb14535359474d482efb002c5654464952594447534e4b445678010048427801305652475050594d4e4441464e4905f9004d057f05f90c49","encoding":"hex","key":"paired_anarci_relabeled.parquet","truncatedAt":512}],"sampleObjects":["bulk_download.sh","paired_anarci_relabeled.parquet","unpaired_anarci_relabeled.parquet"],"sidecars":[{"bytes":"---\nlicense: cc-by-4.0\npretty_name: PRISM Antibody Training Data\nlanguage:\n  - en\ntags:\n  - biology\n  - protein\n  - antibody\n  - immunology\n  - OAS\nsize_categories:\n  - 10M<n<100M\n---\n\n# PRISM Antibody Training Data\n\nTraining data for **PRISM** ([RomeroLab-Duke/prism-antibody](https://huggingface.co/RomeroLab-Duke/prism-antibody)),\na germline/non-germline-aware antibody language model. This repository contains the curated\nunpaired and paired antibody sequence datasets and the exact OAS download manifest used to\nbuild them.\n\n## Files\n\n| File | Rows | Description |\n|------|------|-------------|\n| `unpaired_anarci_relabeled.parquet` | 66,435,576 | Unpaired heavy/light chains (pretraining set) |\n| `paired_anarci_relabeled.parquet` | 763,989 | Paired VH+VL antibodies (fine-tuning set) |\n| `bulk_download.sh` | 14,433 URLs | Exact OAS download manifest used (see caveat below) |\n\nSplits are encoded in the `split` column (`train` / `valid` / `test`).\n\n## Provenance & pipeline\n\n- **Source**: [Observed Antibody Space (OAS)](https://opig.stats.ox.ac.uk/webapps/oas/), human studies only\n  (70 studies, 2021–2022 data freeze).\n- **Pipeline**: download → QC filter (missing conserved cysteine / heavy fragmentation /\n  non-canonical AA) → exact sequence deduplication → **NGL mutation filter** → CDR3 + whole-sequence\n  identity clustering/dedup → ANARCI re-numbering.\n\n## ⚠️ Important caveats (read before reproducing)\n\n1. **This is NOT the full OAS human set.** Most of the largest study, **Briney_2019 heavy**\n   (~1,100 files), was excluded from the unpaired set for download/disk size reasons\n   (these lines are commented out in `bulk_download.sh`, and the loader skips `# wget` lines).\n   This is why the unpaired heavy count is ~58M rather than the full OAS human heavy total.\n2. **NGL (somatic) mutation filter**: sequences are kept only if\n   `num_ngl_muts_hc > 3` (heavy) and `num_ngl_muts_lc > 2` (light). Applied to both sets.\n3. **CDR3 clustering caveat**: the CDR3-identity clustering step used a fixed ANARCI CSV\n   column window that can be shifted by IMGT insertion codes (affects heavy more than light).\n   Treat `*_cdr3_cluster_id` / dedup counts with this in mind.\n\n## Column dictionary\n\n`HEAVY_CHAIN_AA_SEQUENCE`, `LIGHT_CHAIN_AA_SEQUENCE` (AA), `*_NT_SEQUENCE` (nucleotide),\n`*_AA_GERMLINE_ALIGNMENT` (germline-aligned AA), `BType` (B-cell type), `Source_File` (OAS origin),\n`hc_mut_codes` / `lc_mut_codes` (somatic mutation codes), `num_ngl_muts_hc` / `num_ngl_muts_lc` /\n`total_num_ngl_muts` (somatic mutation counts), `*_cdr3_cluster_id` / `*_whole_seq_cluster_id`\n(clustering IDs), `v_gene_*` / `j_gene_*` (V/J gene calls — decode with `gene_vocabulary.json`\nin the model repo), `region_mask_*` (IMGT region masks), `split` (train/valid/test),\n`oas_*_mut_codes` (original OAS mutation codes before ANARCI relabel).\n\n## License & citation\n\nDerived from OAS (CC-BY 4.0). Please cite OAS (Olsen et al. 2022, *Protein Science*) and PRISM.\n\nModel: https://huggingface.co/RomeroLab-Duke/prism-antibody\n","encoding":"utf8","key":"README.md","name":"README.md","size":3043,"truncated":false}],"sourceObjectKey":"paired_anarci_relabeled.parquet","starterCell":"from datasets import load_dataset\nds = load_dataset(\"RomeroLab-Duke/prism-antibody-data\")\nprint(ds)","starterSamplePath":"paired_anarci_relabeled.parquet","totalObjects":5},"summary":"Curated human antibody sequence data used to train PRISM, a germline-aware antibody language model from the Romero Lab at Duke University.","title":"PRISM Antibody Training Data: 66 Million Unpaired and 764k Paired Human Antibody Sequences from OAS with Germline Labels (Romero Lab, Duke)","updatedAt":"2026-11-22T03:39:01.125Z","visibility":"public","websiteUrl":"https://huggingface.co/datasets/RomeroLab-Duke/prism-antibody-data"}}