{"success":true,"data":{"author":"snowsadh","bucketAccessType":"public","bucketName":"snowsadh/goes-omni-electron-flux-forecasting","bucketPrefix":"","category":"Astronomy & Space","cloudProvider":"huggingface","createdAt":"2026-11-21T20:46:01.474Z","description":"A space-weather forecasting dataset that combines cross-calibrated NOAA GOES-14 and GOES-16 measurements of >2 MeV electron flux at geostationary orbit (the outer radiation belt \"killer electrons\" that charge satellites) with NASA/GSFC OMNI solar wind and geomagnetic drivers on a uniform five-minute UTC grid. The scientific-master configuration holds 1,326,459 rows from 2013-05-23 to 2025-12-31 with unscaled source measurements, instrument context, calibration factors and explicit missingness; the ml-ready configuration holds scaled causal features, validity flags, unscaled 30-minute, 6-hour and 12-hour flux targets, and leakage-safe chronological splits (train 2014 to mid-2023, validation to mid-2024, test to April 2025). GOES-14 (EPEAD) and GOES-16 records are spliced in 2017 with MLT-bin cross-calibration; GOES flux ends 2025-04-07. Parquet files load with the Hugging Face datasets library or pandas; DATA_DICTIONARY.md, SOURCES.md and preprocessing_meta.json document fields and provenance. The compiled data are CC BY 4.0 (processing scripts MIT); upstream NOAA and NASA data are public domain.","displayTitle":"GOES and OMNI >2 MeV Geostationary Electron Flux Forecasting Dataset (GOES-14/GOES-16, 2013-2025)","huggingFaceUrl":"https://huggingface.co/datasets/snowsadh/goes-omni-electron-flux-forecasting","id":"dc8e2e9c-bdae-5966-8808-fefdf14a1f31","license":"CC-BY-4.0","licenseEvidenceUrl":"https://huggingface.co/datasets/snowsadh/goes-omni-electron-flux-forecasting","objectCountEstimate":28,"probeStatus":"ok","region":"us","schemaCard":{"accessHints":[{"language":"python-datasets","snippet":"from datasets import load_dataset\nds = load_dataset(\"snowsadh/goes-omni-electron-flux-forecasting\")"},{"language":"python-hub","snippet":"from huggingface_hub import snapshot_download\nsnapshot_download(repo_id=\"snowsadh/goes-omni-electron-flux-forecasting\", repo_type=\"dataset\")"}],"caveats":["HuggingFace metadata: author=snowsadh, license:cc-by-4.0, size_categories:1M<n<10M, format:parquet, downloads=107"],"format":"mixed","generatedAt":"2026-11-21T20:48:54.844Z","layout":{"example":"data/ml-ready/test.parquet","pattern":"{repoId}/{filename}"},"mountPath":"/mnt/archive","sampleHeaders":[{"bytes":"50415231150415e08835159089354c158cd1061500120000b0c41af4ffff9e3d91219b9ecebfb07cd17d94d8e1bff09c4620eedb03c08994a0e19d16babf1367c6adb0c0e4bf80d5d3c99f8ff2bf405d0b06f022e3bf78b005c974fdd2bf7bdf0063e390dfbf9e7bf3e5f43fdabf07fa0400e1e8cbbfb46551ec9b02f4bf715184b3a3f6e0bfda2e297e366ed7bf3482a254ca20cdbf785b18a20c18d1bfcd5f3ddfb5fdb0bfcd2ea314a9e6b9bf3287007be553eebf2c065a0537c4debf40118527089fb7bffab2adf8c203e3bf376e4732a593e5bf942f1f7f2340f3bfec41f9f0dd88d8bf938b28ef5796e5bfd63ff16ff369cabfe402edbc3a32e5bf7cc0295afd82e3bf63d0ddd1a661eabfaddaef41d381d9bf7156ce22148ae0bf109eccd37a91e0bf8115b9cb764bebbf4d1f99b70c7eddbff528de347983dfbf2b7768c85e1595bf32a0fc04cbb4ddbfcbc9d5c3a9cab1bfe925edf69b2de7bf5b855be1b427cbbf488e737bf18ae3bfc29afeacb6dbc3bf3de21dc197c5d1bfb9f43f9add10c5bfc99a04175038bbbffda280d94b93d9bfb1e429ca5f14cfbf797d093d6d17cdbfd1b82cd53925c6bf0d7280dbf975e5bfe186ee56ae9ed6bff21b204fcddbcfbf200385c06288f1bf519cc72d6689d2bfe79d9fb208efe1bf211d54979dd7cebf4f52fabe4b0dc3bf7426750aeda8d2bf629ff959af12e3bf9826","encoding":"hex","key":"data/ml-ready/test.parquet","truncatedAt":512}],"sampleObjects":["CHECKSUMS.sha256","LICENSE-CODE","LICENSE-DATA.md","SOURCES.md","assets/cross_calibration.png"],"sidecars":[{"bytes":"# Changelog\n\n## 1.0.0 — 2026-08-26\n\n- First public Kaggle and Hugging Face release.\n- Adopted the title *GOES–OMNI >2 MeV Electron Flux Forecasting*.\n- Released the compiled data and documentation under CC BY 4.0 and processing\n  code under MIT.\n- Added platform metadata, upstream citation guidance, checksums, and Hugging Face-native\n  train/validation/test files.\n\n## 0.2.0 — 2026-08-26\n\n- Rounded decoded GOES-14 times to exact minutes before resampling; all bins now\n  contain at most five one-minute contributors.\n- Rebuilt GOES-14, GOES-16 orbital merges, cross-calibration, master data, and ML data.\n- Made paths portable and consolidated intermediate GOES tables under `processed/`.\n- Normalized satellite longitude to `[-180, 180)` degrees east.\n- Changed cross-calibration to independently compare like-for-like MLT bins.\n- Corrected `below_g14_noise_floor` to mark flux below 10 in either instrument era.\n- Replaced future backfill with a one-hour causal carry-forward plus train-only median.\n- Added split-boundary target guards and corrected full-day split conditions.\n- Added explicit preprocessing metadata, release audit, dataset card, source record,\n  data dictionary, and declared ML dependencies.\n\n## 0.1.0 — 2026-06-24\n\n- Initial internal GOES–OMNI processing and feature-engineering artifacts.\n","encoding":"utf8","key":"CHANGELOG.md","name":"CHANGELOG.md","size":1326,"truncated":false},{"bytes":"# Data dictionary\n\nThe `time` index in both Parquet tables is a timezone-naive timestamp interpreted\nas UTC. Missing source observations are represented by `NaN` in the master table.\n\n## `processed/master_5min.parquet`\n\n| Column | Type | Unit / values | Meaning |\n|---|---|---|---|\n| `electron_flux_gt2mev` | float | electrons/(cm² s sr) | Cross-calibrated integral electron flux above 2 MeV. |\n| `mlt` | float | hours `[0,24)` | Satellite magnetic/local-time proxy from ephemeris `local_time`. |\n| `sat_lon` | float | degrees east `[-180,180)` | Geographic satellite longitude. |\n| `sat_lat` | float | degrees north | Geographic satellite latitude. |\n| `sat_id` | string | `G14`, `G16` | Contributing GOES satellite. |\n| `correction_factor` | float | dimensionless | MLT-bin multiplier applied to GOES-14; 1 for GOES-16. |\n| `instrument_era` | string | `EPEAD`, `SEISS` | Instrument generation used for the flux value. |\n| `B_mag_avg` | float | nT | Average interplanetary magnetic-field magnitude. |\n| `Bx_gse_gsm` | float | nT | OMNI IMF Bx component; source convention is GSE/GSM. |\n| `By_gsm` | float | nT | OMNI IMF By in GSM. |\n| `Bz_gsm` | float | nT | OMNI IMF Bz in GSM. |\n| `sw_speed` | float | km/s | Solar-wind bulk speed. |\n| `sw_proton_density` | float | cm⁻³ | Solar-wind proton number density. |\n| `sw_proton_temp` | float | K | Solar-wind proton temperature. |\n| `sw_flow_pressure` | float | nPa | Solar-wind flow pressure. |\n| `AE_index` | float | nT | Auroral electrojet index. |\n| `SYM_H` | float | nT | One-minute symmetric disturbance index, averaged to five minutes. |\n| `log10_flux` | float | log10 physical flux | Exact `log10(electron_flux_gt2mev)`. |\n| `below_g14_noise_floor` | bool | true/false | True when flux is below 10, a regime G14 EPEAD could not reliably resolve. |\n\n## `processed/ml_dataset.parquet`\n\nThe following 15 numeric feature columns are **RobustScaler-transformed and\nunitless**. Centers and scales are stored in `processed/preprocessing_meta.json`:\n\n- Current state: `log10_flux`, `sw_speed`, `Bz_gsm`, `SYM_H`, `sw_proton_density`\n- Cyclical local time: `mlt_sin`, `mlt_cos`\n- Lagged state: `log10_flux_lag6h`, `log10_flux_lag12h`,\n  `sw_speed_lag24h`, `sw_speed_lag48h`, `Bz_gsm_lag12h`,\n  `SYM_H_lag12h`, `SYM_H_lag48h`\n- Coupling proxy: `vxBz`, derived before scaling as\n  `sw_speed * max(0, -Bz_gsm) * 1e-3`\n\nUnscaled context columns:\n\n| Column | Values | Meaning |\n|---|---|---|\n| `instrument_era_EPEAD` | 0/1 | Current flux originates from GOES-14 EPEAD. |\n| `instrument_era_SEISS` | 0/1 | Current flux originates from GOES-16 SEISS. |\n| `below_g14_noise_floor` | 0/1 | Current physical flux was below 10 before scaling. |\n\nFor every feature that had missing values before imputation, a matching\n`<feature>_valid` column is included. `1` means the original value was observed;\n`0` means it was causally forward-filled or replaced with a train-only median.\n\nTargets are unscaled physical log-flux values:\n\n| Column | Horizon | Meaning |\n|---|---|---|\n| `target_30min` | +30 minutes | Future `log10_flux`. |\n| `target_6h` | +6 hours | Future `log10_flux`. |\n| `target_12h` | +12 hours | Future `log10_flux`. |\n| `split` | — | `train`, `val`, or `test`; chronological and leakage-guarded. |\n","encoding":"utf8","key":"DATA_DICTIONARY.md","name":"DATA_DICTIONARY.md","size":3250,"truncated":false},{"bytes":"---\nlicense: cc-by-4.0\npretty_name: \"GOES–OMNI >2 MeV Electron Flux Forecasting\"\ntask_categories:\n- time-series-forecasting\n- tabular-regression\ntags:\n- space-weather\n- electron-flux\n- radiation-belts\n- goes\n- omni\n- timeseries\n- tabular\n- parquet\nsize_categories:\n- 1M<n<10M\nconfigs:\n- config_name: ml-ready\n  default: true\n  data_files:\n  - split: train\n    path: data/ml-ready/train.parquet\n  - split: validation\n    path: data/ml-ready/validation.parquet\n  - split: test\n    path: data/ml-ready/test.parquet\n- config_name: scientific-master\n  data_files:\n  - split: full\n    path: data/scientific-master/full.parquet\n---\n\n# GOES–OMNI >2 MeV Electron Flux Forecasting\n\nThis dataset combines cross-calibrated NOAA GOES-14/GOES-16 >2 MeV electron\nflux with NASA/GSFC OMNI solar-wind and geomagnetic drivers on a uniform\nfive-minute UTC grid.\n\nIt provides two configurations:\n\n- **`ml-ready`** (default): scaled causal features, validity flags, unscaled\n  30-minute/6-hour/12-hour targets, and leakage-safe chronological splits.\n- **`scientific-master`**: unscaled source measurements, instrument context,\n  calibration factors, and explicit missingness.\n\n## Load the data\n\n```python\nfrom datasets import load_dataset\n\nml = load_dataset(\n    \"snowsadh/goes-omni-electron-flux-forecasting\",\n    \"ml-ready\",\n)\nmaster = load_dataset(\n    \"snowsadh/goes-omni-electron-flux-forecasting\",\n    \"scientific-master\",\n    split=\"full\",\n)\n```\n\n## Size and coverage\n\n| Configuration / split | Rows | Coverage |\n|---|---:|---|\n| Scientific master | 1,326,459 | 2013-05-23 to 2025-12-31 |\n| ML train | 978,265 | 2014-01-01 to 2023-06-30 11:55 UTC |\n| ML validation | 102,186 | 2023-07-01 to 2024-06-30 11:55 UTC |\n| ML test | 80,612 | 2024-07-01 to 2025-04-07 07:50 UTC |\n\nGOES flux ends on 2025-04-07; later scientific-master rows contain OMNI data\nonly. Every ML target remains inside the same temporal split as its features.\n\n## Processing\n\nGOES-14 one-minute observations are normalized to exact minutes and aggregated\nin log space. GOES-14 and GOES-16 are cross-calibrated in one-hour magnetic\nlocal-time bins over their 2017 overlap. The corrected series is combined with\nfive-minute means of OMNI drivers. ML features use only causal information;\nmissing inputs are carried forward for at most one hour and then replaced with\na training-only median. `RobustScaler` is fitted on the training split only.\n\nFull definitions are in `DATA_DICTIONARY.md`; preprocessing parameters are in\n`preprocessing_meta.json`; source-level provenance is in `SOURCES.md`.\n\n## Intended uses\n\n- Forecasting GEO >2 MeV electron flux 30 minutes, 6 hours, or 12 hours ahead.\n- Space-weather time-series analysis and storm case studies.\n- Multi-instrument calibration and leakage-safe ML demonstrations.\n\n## Limitations\n\n- The data form a GOES-14/GOES-16 time series, not a global radiation-belt map.\n- Satellite longitude and detector geometry change across the 2017 splice.\n- MLT-bin calibration cannot recover values below the GOES-14 EPEAD noise floor.\n- Values below flux 10 are retained and explicitly marked.\n- Imputed ML features must not be interpreted as physical observations.\n- OMNI combines several upstream spacecraft and maps measurements toward\n  Earth's bow-shock nose.\n- No ISRO GRASP data are included, and the series is not validated at Indian\n  GEO longitude.\n\n## License\n\nThe compiled data and documentation are licensed under **CC BY 4.0** to the\nextent the creator holds applicable rights. Processing scripts are MIT-licensed.\nUpstream public-domain/CC0/source-specific terms remain in force; see\n`LICENSE-DATA.md` and `SOURCES.md`.\n","encoding":"utf8","key":"README.md","name":"README.md","size":3629,"truncated":false}],"sourceObjectKey":"data/ml-ready/test.parquet","starterCell":"from datasets import load_dataset\nds = load_dataset(\"snowsadh/goes-omni-electron-flux-forecasting\")\nprint(ds)","starterSamplePath":"assets/cross_calibration.png","totalObjects":28},"summary":"A space-weather forecasting dataset that combines cross-calibrated NOAA GOES-14 and GOES-16 measurements of >2 MeV electron flux at geostationary orbit (the outer radiation belt…","title":"GOES and OMNI >2 MeV Geostationary Electron Flux Forecasting Dataset (GOES-14/GOES-16, 2013-2025)","updatedAt":"2026-11-21T22:55:07.721Z","visibility":"public","websiteUrl":"https://huggingface.co/datasets/snowsadh/goes-omni-electron-flux-forecasting"}}