{"success":true,"data":{"author":"Michael J. Smith et al. (AstroPT)","bucketAccessType":"public","bucketName":"Smith42/galaxies","bucketPrefix":"","category":"Astronomy & Space","cloudProvider":"huggingface","createdAt":"2026-11-14T01:01:18.839Z","description":"About 8.5 million galaxy cutouts from the DESI Legacy Imaging Surveys DR8, each a 512 by 512 pixel JPEG centred on a galaxy from the Galaxy Zoo DESI catalogue (Walmsley et al. 2023), assembled to train AstroPT, a family of large observation models for astronomy. 98 percent of images form the training set, with 1 percent each for validation and test. Galaxy properties are provided as metadata linked by dr8_id, and pre-computed AstroPT v2 embeddings are available in a companion repository; revision v2.0 joins images and metadata directly. Listing the repository (measured 2026-11-13) shows data/ with 196 Parquet shards (982.5 GB, train and test), with_crops/ with 243 Parquet shards (1,206 GB) and a root metadata.parquet (2.16 GB). Open with Hugging Face datasets (streaming=True is advisable), pyarrow or pandas, decoding the image bytes with PIL. Distributed under CC-BY-SA-4.0 by the uploader over CC-BY-4.0 Legacy Surveys imagery and Galaxy Zoo DESI catalogue; use the Legacy Surveys acknowledgment and cite Dey et al. 2019, Walmsley et al. 2023 and Smith et al. 2024.","displayTitle":"AstroPT Galaxies: 8.5 Million DESI Legacy Survey DR8 Galaxy Cutouts","id":"e0155021-5073-5834-b146-16c394231145","license":"CC-BY-SA-4.0","licenseEvidenceUrl":"https://www.legacysurvey.org/acknowledgment/","objectCountEstimate":442,"probeStatus":"ok","referenceUrl":"https://arxiv.org/abs/2405.14930","region":"us","schemaCard":{"accessHints":[{"language":"python-datasets","snippet":"from datasets import load_dataset\nds = load_dataset(\"Smith42/galaxies\")"},{"language":"python-hub","snippet":"from huggingface_hub import snapshot_download\nsnapshot_download(repo_id=\"Smith42/galaxies\", repo_type=\"dataset\")"}],"caveats":["HuggingFace metadata: author=Smith42, license:cc-by-sa-4.0, size_categories:10M<n<100M, format:parquet, downloads=4172"],"format":"mixed","generatedAt":"2026-11-14T01:01:20.567Z","layout":{"example":"data/test-00000-of-00002.parquet","pattern":"{repoId}/{filename}"},"mountPath":"/mnt/archive","sampleHeaders":[{"bytes":"50415231150415e29cf90a15fa9df90a4c15c8011500120000b1cebc057c7b830100ffd8ffe000104a46494600010100000100010000ffdb0043000302020103e003030304030304050805050404050a070706080c0a0c0c0b0a0b0b0d0e12100d0e110e0b0b1016101113141515150c0f17181614181214151401453c0103040405040509050509140d0b0d14c201006cffc00011080200020003012200021101031101ffc4001f0000010501050100000d015c0102030405060708090a0bffc400b510000201030302040301b6f0b00000017d01020300041105122131410613516107227114328191a1082342b1c11552d1f02433627282090a161718191a25262728292a3435363738393a434445464748494a535455565758595a636465666768696a737475767778797a838485868788898a92939495969798999aa2a3a4a5a6a7a8a9aab2b3b4b5b6b7b8b9bac2c3c4c5c6c7c8c9cad2d3d4d5d6d7d8d9dae1e2e3e4e5e6e7e8e9eaf1f2f3f4f5f6f7f8f9faffc4001f010003010101011dda3ad8004011000201020404030407050404000102770121a81104052131061241510761711322328108144291a1b1c109233352f0156272d10a162434e125f11718191a05da96d90008828384cada0008e2e3e409d908f2f3f40dd8f499fdda000c03010002110311003f00f8026bd90c46063e546e55b01016dc40ddc7","encoding":"hex","key":"data/test-00000-of-00002.parquet","truncatedAt":512}],"sampleObjects":["data/test-00000-of-00002.parquet","data/test-00001-of-00002.parquet","data/train-00000-of-00192.parquet","data/train-00001-of-00192.parquet","data/train-00002-of-00192.parquet"],"sidecars":[{"bytes":"---\nlicense: cc-by-sa-4.0\nsize_categories:\n- 1M<n<10M\ndataset_info:\n- config_name: default\n  features:\n  - name: image\n    dtype: image\n  - name: image_crop\n    dtype: image\n  - name: dr8_id\n    dtype: string\n  - name: galaxy_size\n    dtype: int64\n  splits:\n  - name: test\n    num_bytes: 12057249781.25\n    num_examples: 86471\n  - name: validation\n    num_bytes: 12065699086.25\n    num_examples: 86499\n  - name: train\n    num_bytes: 1181934533243.5\n    num_examples: 8474566\n  download_size: 1206114375284\n  dataset_size: 1206057482111\n- config_name: v1.0\n  features:\n  - name: image\n    dtype: image\n  - name: dr8_id\n    dtype: string\n  splits:\n  - name: train\n    num_bytes: 959387460144.3469\n    num_examples: 8474566\n  - name: test\n    num_bytes: 9785671701.822557\n    num_examples: 86471\n  - name: validation\n    num_bytes: 9798204502.80013\n    num_examples: 86499\n  download_size: 982501453040\n  dataset_size: 978971336348.9696\nconfigs:\n- config_name: default\n  data_files:\n  - split: test\n    path: with_crops/test-*\n  - split: validation\n    path: with_crops/validation-*\n  - split: train\n    path: with_crops/train-*\n- config_name: v1.0\n  data_files:\n  - split: train\n    path: data/train-*\n  - split: test\n    path: data/test-*\n  - split: validation\n    path: data/validation-*\ntags:\n- astronomy\n- huggingscience\n- science\n---\n# Galaxies for training astroPT\n\nHere we have ~8.5 million galaxy cutouts from the [DESI legacy survey DR8](https://www.legacysurvey.org/dr8/description/).\nThe cut outs are 512x512 pixel jpg images centred on the galaxy source.\n\nI've split away 1% of the images into a test set, and 1% into a validation set.\nThe remaining 98% of the images comprise the training set.\n\n## Metadata (galaxy properties)\n\nThere is also accompanying metadata!\nTo combine the metadata with the galaxy images you can do (for example):\n```python\nfrom datasets import load_dataset, concatenate_datasets\n\n# Load the `galaxies' dataset with metadata\ngalaxies = load_dataset(\"Smith42/galaxies\", streaming=True)\nmetadata = load_dataset(\"Smith42/galaxies_metadata\", streaming=True).remove_columns(\"dr8_id\")\ncombined = concatenate_datasets([galaxies['train'], metadata['train']], axis=1)\n```\nOr you can use v2.0 of this dataset to get the metadata direct with no faffing 🤯:\n```python\ngalaxies = load_dataset(\"Smith42/galaxies\", revision=\"v2.0\", streaming=True)\n```\nThe metadata is also available in parquet format in the root dir of this repo. \nYou can link the metadata with the galaxies via their dr8_id.\n\n## Embeddings from the pre-trained family of AstroPTv2 models\n\nSame story with pre-generated embeddings from [AstroPTv2](https://huggingface.co/smith42/astropt_v2.0).\n\nYou can combine the embeddings with galaxy imagery and metadata by doing:\n```python\ngalaxies = load_dataset(\"Smith42/galaxies\", revision=\"v2.0\", streaming=True)\nembs = load_dataset(\"Smith42/galaxies_embeddings\", streaming=True)\ncombined = concatenate_datasets([galaxies['train'], embs['train']], axis=1)\n```\n\n## Useful links\n\nPaper here: [https://arxiv.org/abs/2405.14930](https://arxiv.org/abs/2405.14930)\n\nModels here: [https://huggingface.co/Smith42/astroPT](https://huggingface.co/Smith42/astroPT)\n\nAnd here: [https://huggingface.co/Smith42/astroPT_v2.0](https://huggingface.co/Smith42/astroPT_v2.0)\n\nCode here: [https://github.com/smith42/astroPT](https://github.com/smith42/astroPT)\n\nUpstream catalogue is [on Zenodo](https://zenodo.org/records/8360385) and paper describing the catalogue is available as [Walmsley+2023](https://doi.org/10.1093/mnras/stad2919).\n\nIf you find this dataset useful please consider citing the sources below 🚀🚀:\n\n```\n@article{ref_dey2019,\n\tauthor = {Dey, A. and Schlegel, D. J. and Lang, D. and Blum, R. and Burleigh, K. and Fan, X. and Findlay, J. R. and Finkbeiner, D. and Herrera, D. and Juneau, S. and others},\n\ttitle = {{Overview of the DESI Legacy Imaging Surveys}},\n\tjournal = {Astronomical Journal},\n\tvolume = {157},\n\tnumber = {5},\n\tpages = {168},\n\tyear = {2019},\n\tissn = {1538-3881},\n\tpublisher = {The American Astronomical Society},\n\tdoi = {10.3847/1538-3881/ab","encoding":"utf8","key":"README.md","name":"README.md","size":5000,"truncated":true}],"sourceObjectKey":"data/test-00000-of-00002.parquet","starterCell":"from datasets import load_dataset\nds = load_dataset(\"Smith42/galaxies\")\nprint(ds)","starterSamplePath":"data/test-00000-of-00002.parquet","totalObjects":442},"summary":"About 8.5 million galaxy cutouts from the DESI Legacy Imaging Surveys DR8, each a 512 by 512 pixel JPEG centred on a galaxy from the Galaxy Zoo DESI catalogue (Walmsley et al.","title":"AstroPT Galaxies: 8.5 Million DESI Legacy Survey DR8 Galaxy Cutouts","updatedAt":"2026-11-14T02:31:20.564Z","visibility":"public","websiteUrl":"https://github.com/smith42/astroPT"}}