{"success":true,"data":{"author":"marberi (MasterCosmos BCN, IFAE/PIC), from DESI Collaboration data","bucketAccessType":"public","bucketName":"marberi/desi-dr1-spectra-100k","bucketPrefix":"","category":"Astronomy & Space","cloudProvider":"huggingface","createdAt":"2026-11-21T20:46:01.498Z","description":"A random subset of 100,000 galaxy spectra with redshifts from Data Release 1 (DR1) of the Dark Energy Spectroscopic Instrument (DESI), prepared for the MasterCosmos BCN lecture \"Can you trust your network?\" (IFAE/PIC, Barcelona) for teaching and machine-learning redshift estimation. The parent sample follows Guo and Eriksen (2026): DR1 main-survey HEALPix target-coadded spectra from the dark, bright and backup programs with the three spectrograph arms combined, cut to ZCAT_PRIMARY, good fiber status, ZWARN 0 or 4, SPECTYPE GALAXY, ZERR < 5e-4 and z > 0.01 (about 14.6 million spectra, redshift up to about 1.7). Spectra are normalized by their median flux and stored in spectra.zarr as a float32 array (100000, 2, 7781) of flux and inverse variance on the DESI observed-frame grid from 3600 to 9824 Angstrom in 0.8 Angstrom steps, with bright sky lines masked in the inverse variance. catalog.parquet gives DESI TARGETID, pipeline redshift, spectral type, HEALPix pixel and normalizations. About 5 GB; load with zarr and pandas. Use it for photometric-free spectroscopic redshift networks and galaxy spectra representation learning. DESI data are CC BY 4.0; acknowledge DESI and cite DESI DR1.","displayTitle":"DESI DR1 Galaxy Spectra: 100,000 Normalized Optical Spectra with Redshifts for Machine Learning (Dark Energy Spectroscopic Instrument)","huggingFaceUrl":"https://huggingface.co/datasets/marberi/desi-dr1-spectra-100k","id":"da477f0e-4a48-571e-b2b5-6e9da8aa08ac","license":"CC-BY-4.0","licenseEvidenceUrl":"https://data.desi.lbl.gov/doc/acknowledgments/","objectCountEstimate":14,"probeStatus":"ok","region":"us","schemaCard":{"accessHints":[{"language":"python-datasets","snippet":"from datasets import load_dataset\nds = load_dataset(\"marberi/desi-dr1-spectra-100k\")"},{"language":"python-hub","snippet":"from huggingface_hub import snapshot_download\nsnapshot_download(repo_id=\"marberi/desi-dr1-spectra-100k\", repo_type=\"dataset\")"}],"caveats":["HuggingFace metadata: author=marberi, license:cc-by-4.0, size_categories:100K<n<1M, format:parquet, downloads=187"],"format":"mixed","generatedAt":"2026-11-21T20:48:54.915Z","layout":{"example":"catalog.parquet","pattern":"{repoId}/{filename}"},"mountPath":"/mnt/archive","sampleHeaders":[{"bytes":"50415231150415c0d36115a8fe5a4c15b89a0c1500120000e0e930f48c03a20900dcfecd8c000f0d005f95c98c00cd0500b16dce8c006904c07a62ce8c00fd06408447c98c00680a004b19c98c00570040ee6ace8c006e03800e60c98c00560280c726c98c00f000c02f5bc98c00e80b005c3cc98c000305408a02c98c00b70580f413ce8c002d22007922c98c00400300b86cc98c00490a40f825c98c00da03c0fd23c98c00a30340c972c98c006403800078c98c005e06806224c98c006211407532c98c003b05007ec2c98c00e001407d3cce8c00aa13c0b240c98c001b09009a04ce8c00b204c03354c98c00b70b409d4ec98c00ee04009b57ce8c00440340ea30c98c002404c02b5ec98c00cf1340a060c98c00db0440a04bc98c008b0400ed52c98c00f60c007f49ce8c00ff158076e2c98c005c0900b04bc98c004a0140bd4dc98c001d16404958c98c00ef15401d3fc98c00cf12806f89c98c00d20a003c2ec98c005d0d80f7f8c88c00570d807739c98c005701c0a139ce8c007b14802313c98c00150bc02f2cc98c00c705801aa4c98c00170940b23ace8c005406802b9bc98c00fb15c01240d18c007c110092eac98c00920240242ec98c00ad0c00c7a1c98c00ee00807a30ce8c00790ac0870dc98c00f10500d128ce8c00b60cc01557c98c006d00803033ce8c00ae0c405f93c98c00ac02008026ce8c00c40c","encoding":"hex","key":"catalog.parquet","truncatedAt":512}],"sampleObjects":["catalog.parquet","spectra.zarr/c/0/0/0","spectra.zarr/c/1/0/0","spectra.zarr/c/2/0/0","spectra.zarr/c/3/0/0"],"sidecars":[{"bytes":"---\nlicense: cc-by-4.0\npretty_name: DESI DR1 spectra, 100k galaxies\ntags:\n  - astronomy\n  - astrophysics\n  - spectroscopy\n  - redshift\n  - DESI\n---\n\n# DESI DR1 spectra: 100,000 galaxies for machine learning\n\nA random subset of 100,000 spectra with redshifts from the public Data Release 1\n(DR1) of the Dark Energy Spectroscopic Instrument (DESI), prepared for teaching and\nfor trying out machine-learning methods on real survey data. The spectra are\nnormalized and stored on a common wavelength grid, so they can be fed directly\ninto a neural network.\n\nIt was made for the lecture *Can you trust your network?* in the MasterCosmos BCN\nprogramme (IFAE / PIC, Barcelona). Code and slides:\n[github.com/marberi/can-you-trust-your-network](https://github.com/marberi/can-you-trust-your-network)\n\n## Files\n\n| File | Content |\n|---|---|\n| `catalog.parquet` | One row per spectrum: identifiers, redshift and selection flags |\n| `spectra.zarr` | The spectra, an array of shape (100000, 2, 7781), float32 |\n\nRow `i` of `catalog.parquet` corresponds to `spectra[i]`.\n\n### `catalog.parquet`\n\n| Column | Type | Description |\n|---|---|---|\n| `target_id` | int64 | DESI `TARGETID`, for matching with the original DESI data |\n| `z` | float32 | Redshift from the DESI pipeline |\n| `norm_rest` | float32 | Normalization in a fixed rest-frame wavelength window (as in Melchior et al. 2023); missing when the window falls outside the observed range |\n| `norm_med` | float32 | Median flux over the full observed wavelength range, the normalization applied to the spectra |\n| `hpixid` | int64 | HEALPix pixel of the DESI target-coadded file the spectrum comes from [state nside and ordering] |\n| `spec_type` | string | Spectral type from the DESI pipeline (`SPECTYPE`) |\n| `mask_basic` | bool | [Describe the selection] |\n| `mask_lowz` | bool | [Describe the selection] |\n\n### `spectra.zarr`\n\n- Shape (100000, 2, 7781), float32.\n- Channel 0: flux, normalized by `norm_med`.\n- Channel 1: inverse variance, normalized consistently with the flux. It is set to zero at\n  the wavelengths of bright atmospheric emission lines and at other bad pixels, so use only\n  pixels with inverse variance > 0 when computing losses or statistics.\n- Wavelength grid: the DESI observed-frame grid, 3600 to 9824 Å in steps of 0.8 Å:\n\n  ```python\n  wave = 3600 + 0.8 * np.arange(7781)\n  ```\n\n## How it was made\n\nThe parent sample is the one described in Section 2.1 of Guo & Eriksen (2026):\n\n- DESI DR1 main-survey target-coadded spectra (HEALPix-based) from the `dark`, `bright` and\n  `backup` programs, with the three spectrograph arms combined on one wavelength grid.\n- Quality cuts on the redshift catalogs: `ZCAT_PRIMARY = True`, `COADD_FIBERSTATUS = 0`,\n  `ZWARN = 0` or `4`, `SPECTYPE = 'GALAXY'`, `ZERR < 5e-4` and `z > 0.01`.\n  This gives about 14.6 million spectra, with redshifts up to about 1.7.\n- Bright atmospheric emission lines are masked by setting the inverse variance to zero,\n  following Melchior et al. (2023).\n- Each spectrum is normalized by its median flux over the full observed wavelength range\n  (`norm_med`). Unlike a fixed rest-frame window, this works at all redshifts.\n- 100,000 spectra were drawn at random from this parent sample.\n\nThe redshift distribution has two main peaks, near z ≈ 0.2 and z ≈ 0.85, because DESI\nselects several target classes in different ways. A network trained on this data learns\nthat distribution as its prior.\n\n## Usage\n\n```python\nfrom pathlib import Path\nimport pandas as pd\nimport zarr\nfrom huggingface_hub import snapshot_download\n\ndata_dir = Path(snapshot_download('marberi/desi-dr1-spectra-100k', repo_type='dataset',\n                                  local_dir='data/desi-dr1-spectra-100k'))\ncat = pd.read_parquet(data_dir / 'catalog.parquet').reset_index(drop=True)\nspectra = zarr.open(data_dir / 'spectra.zarr', mode='r')[:]   # (100000, 2, 7781)\n```\n\nThe download is about 5 GB.\n\n## Things to be aware of\n\n- `norm_rest` is computed in a rest-frame window, so it depends on the true redshift.\n  Using it as an input when predicting redshift would leak the answer in","encoding":"utf8","key":"README.md","name":"README.md","size":7191,"truncated":true}],"sourceObjectKey":"catalog.parquet","starterCell":"from datasets import load_dataset\nds = load_dataset(\"marberi/desi-dr1-spectra-100k\")\nprint(ds)","starterSamplePath":"catalog.parquet","totalObjects":14},"summary":"A random subset of 100,000 galaxy spectra with redshifts from Data Release 1 (DR1) of the Dark Energy Spectroscopic Instrument (DESI), prepared for the MasterCosmos BCN lecture…","title":"DESI DR1 Galaxy Spectra: 100,000 Normalized Optical Spectra with Redshifts for Machine Learning (Dark Energy Spectroscopic Instrument)","updatedAt":"2026-11-22T00:47:21.587Z","visibility":"public","websiteUrl":"https://github.com/marberi/can-you-trust-your-network"}}