{"success":true,"data":{"author":"Allen Institute for AI, Semantic Scholar (Shaurya Rohatgi, Sergey Feldman)","bucketAccessType":"public","bucketListable":true,"bucketName":"ai2-s2-research-public","bucketPrefix":"paper_clustering","category":"Natural Language Processing","cloudProvider":"aws","createdAt":"2026-11-21T20:49:29.543Z","description":"S2APLER (Semantic Scholar Agglomeration of Papers with Low Error Rate) is the dataset and model Semantic Scholar uses to cluster records that describe the same scholarly paper, such as a preprint, a journal version and duplicate entries from different sources. The paper_clustering/ folder holds papers.json (about 3.3 GB of paper records with metadata used for pairwise features) and clusters.json (gold clusters of records that are the same work), about 3.1 GiB in total. The reported pairwise model reaches AUROC 0.999 and the final clustering B3 F1 0.9986 on the test set, about a 50 percent error reduction over the previous Semantic Scholar paper clustering. Load with the s2apler Python package from allenai/S2APLER. Some affiliation data comes from the Microsoft Academic Graph. Used for record linkage, deduplication of bibliographic databases and scholarly knowledge graph construction. License: dataset under ODC-BY. Cite the allenai/S2APLER repository (Rohatgi, Feldman et al., Allen Institute for AI).","displayTitle":"S2APLER: Semantic Scholar Paper Clustering Dataset for Deduplicating Scholarly Paper Records (Allen Institute for AI)","fuseMountBrowsable":true,"id":"fad16ac1-cc25-5b62-84dd-42a68c36428f","license":"ODC-By-1.0","licenseEvidenceUrl":"https://github.com/allenai/S2APLER","probeStatus":"ok","region":"us-west-2","schemaCard":{"accessHints":[{"language":"python-anonymous","snippet":"import boto3\nfrom botocore import UNSIGNED\nfrom botocore.client import Config\ns3 = boto3.client('s3', region_name='us-west-2', config=Config(signature_version=UNSIGNED))\ns3.list_objects_v2(Bucket='ai2-s2-research-public', Prefix='paper_clustering/', Delimiter='/')"},{"language":"python-local","snippet":"open('/mnt/archive/clusters.json', 'rb')"}],"caveats":["Bucket has a flat key structure (no prefix folders). Avoid unbounded LIST — page with ContinuationToken or filter by key prefix.","Files are binary — use a domain-specific parser (e.g. metpy / pynexrad / pydicom) rather than text I/O."],"format":"binary","generatedAt":"2026-11-21T20:49:47.240Z","layout":{"example":"paper_clustering/clusters.json","pattern":"{prefix}/{filename}"},"mountPath":"/mnt/archive","sampleHeaders":[{"bytes":"{\"internationalindustrialrelationsass_5872519\": {\"cluster_id\": \"internationalindustrialrelationsass_5872519\", \"sourced_paper_ids\": [2052283648, 3587647857, 3587547877], \"model_version\": -1}, \"internationalindustrialrelationsass_orphans\": {\"cluster_id\": \"internationalindustrialrelationsass_orphans\", \"sourced_paper_ids\": [1535692761615831040, 1613790305, 2008490408, 1452158411, 1431354251, 1462117938, 216923096], \"model_version\": -1}, \"levelsofbacteriafungiandendotoxinin_4831270\": {\"cluster_id\": \"levelsofbact","encoding":"utf8","key":"paper_clustering/clusters.json","truncatedAt":512},{"bytes":"{\"43808411\": {\"authors\": [{\"position\": 0, \"first\": \"Ursula\", \"middle\": [], \"last\": \"Hofer\", \"suffix\": null, \"affiliations\": [], \"email\": null}], \"abstract\": null, \"references\": [], \"source\": \"Medline\", \"doi\": \"10.1038/nrmicro3092\", \"pmid\": 23872943, \"title\": \"Viral evolution: Variation in the gut virome.\", \"year\": 2013, \"venue\": \"Nature reviews. Microbiology\", \"publicationtypes\": [\"LettersAndComments\", \"JournalArticle\"], \"fieldsofstudy\": [\"Medicine\"], \"journal_name\": \"\\n          596\\n        \", \"block\": \"v","encoding":"utf8","key":"paper_clustering/papers.json","truncatedAt":512}],"sampleObjects":["paper_clustering/clusters.json","paper_clustering/papers.json"],"sourceObjectKey":"paper_clustering/clusters.json","starterCell":"with open('/mnt/archive/clusters.json', 'rb') as fh:\n    head = fh.read(1024)\nprint(len(head), head[:64])","starterSamplePath":"clusters.json","totalObjects":"flat keyspace (count unknown from bounded probe)"},"summary":"S2APLER (Semantic Scholar Agglomeration of Papers with Low Error Rate) is the dataset and model Semantic Scholar uses to cluster records that describe the same scholarly paper,…","title":"S2APLER: Semantic Scholar Paper Clustering Dataset for Deduplicating Scholarly Paper Records (Allen Institute for AI)","updatedAt":"2026-11-22T00:44:05.142Z","visibility":"public","websiteUrl":"https://github.com/allenai/S2APLER"}}