{"success":true,"data":{"author":"AODN","bucketAccessType":"public","bucketListable":true,"bucketName":"aodn-cloud-optimised","bucketPrefix":"mooring_estuarine_coastal_water_quality_monitoring_realtime_qc.parquet","category":"Environmental Science","cloudProvider":"aws","createdAt":"2026-10-04T00:00:00.000Z","description":"The IMOS Estuarine and Coastal Moorings Facility provides a national, standardised network of estuarine and coastal moorings focused on water quality. The collection brings together IMOS-funded buoys and co-invested monitoring networks, notably the South Australian Harmful Algal Bloom (HAB) coastal monitoring network run by SARDI. Small telemetered buoys (Xylem DB range) are anchored in shallow water and measure pressure, temperature, salinity, chlorophyll-a fluorescence, dissolved oxygen, turbidity and fDOM, with optional extra sensors. Near real-time data at 15 minute intervals are processed by SARDI and automatically quality controlled with IOOS QARTOD tests (location, gross range, spike, rate of change, flat line); every value carries an IOC flag (1 pass, 3 questionable, 4 fail, 9 missing). This store holds the near real-time stream as hive-partitioned parquet: on 2026-10-04 it held 263,012 rows from seven sites (six South Australian aquaculture and gulf sites: American River, Boston Bay, Coffin Bay, St Kilda, Stansbury, Victor Harbor, plus Western Port in Victoria), covering 2025-12-18 to 2026-10-04 and updated several times a day. These are the SA Harmful Algal Bloom coastal monitoring buoys: the near-real-time temperature, salinity, dissolved oxygen, chlorophyll fluorescence and turbidity record at the Adelaide, Yorke Peninsula, Kangaroo Island and Eyre Peninsula sites during the 2025-26 Karenia algal bloom response.","displayTitle":"Moorings - Estuarine and coastal water quality monitoring - Near real time","fuseMountBrowsable":true,"id":"7fa49df9-b523-5738-929e-9134b83db955","license":"http://creativecommons.org/licenses/by/4.0/","objectCountEstimate":7,"probeStatus":"ok","region":"ap-southeast-2","schemaCard":{"accessHints":[{"language":"bash","snippet":"# Task: see the site partitions and the root metadata file (no data read)\naws s3 ls --no-sign-request --region ap-southeast-2 s3://aodn-cloud-optimised/mooring_estuarine_coastal_water_quality_monitoring_realtime_qc.parquet/\n# the whole store is 32 objects / 9 MB; a recursive --summarize listing is fine here\naws s3 ls --no-sign-request --region ap-southeast-2 --recursive --summarize s3://aodn-cloud-optimised/mooring_estuarine_coastal_water_quality_monitoring_realtime_qc.parquet/ | tail -2"},{"language":"python-anonymous","snippet":"# Task: read the schema and the dataset-level metadata (GeoNetwork uuid, title, QC tests) without touching any data file\nimport json, pyarrow.parquet as pq, s3fs\nfs = s3fs.S3FileSystem(anon=True, client_kwargs={'region_name': 'ap-southeast-2'})\nroot = 'aodn-cloud-optimised/mooring_estuarine_coastal_water_quality_monitoring_realtime_qc.parquet'\nwith fs.open(root + '/_common_metadata', 'rb') as f:\n    schema = pq.read_schema(f)\nprint(len(schema.names), 'columns;', [n for n in schema.names if not n.endswith('_quality_control')])\nprint(schema.metadata[b'global_attributes'].decode())            # metadata_uuid, title, quality_control_tests_applied\nprint(schema.metadata[b'CPHL'].decode())                         # per-column attrs: units, long_name, sensor serial, flag meanings"},{"language":"python-anonymous","snippet":"# Task: load one site anonymously (hive partition filter -> only that site's 4-8 files are read) and show its time range\nimport time, pandas as pd, s3fs\nfs = s3fs.S3FileSystem(anon=True, client_kwargs={'region_name': 'ap-southeast-2'})\nroot = 'aodn-cloud-optimised/mooring_estuarine_coastal_water_quality_monitoring_realtime_qc.parquet'   # no 's3://' when passing filesystem=\nt0 = time.time()\ndf = pd.read_parquet(root, filesystem=fs, filters=[('site_code', '==', 'SA-COFFIN-BAY')])\nprint(len(df), 'rows', df.shape[1], 'cols in', round(time.time() - t0, 1), 's;', df.TIME.min(), '->', df.TIME.max())\nprint(df.filename.str.extract(r'_(BKT|CKOSTUZ)_')[0].value_counts().to_dict(), '<- two instrument streams interleaved as separate rows')"},{"language":"python-local","snippet":"# Task: load one site from the mount (hive partition filter -> only that site's 4-8 files are read) and show its time range\nimport time, pandas as pd\nt0 = time.time()\ndf = pd.read_parquet('/mnt/archive', filters=[('site_code', '==', 'SA-COFFIN-BAY')])\nprint(len(df), 'rows', df.shape[1], 'cols in', round(time.time() - t0, 1), 's;', df.TIME.min(), '->', df.TIME.max())\nprint(df.filename.str.extract(r'_(BKT|CKOSTUZ)_')[0].value_counts().to_dict(), '<- two instrument streams interleaved as separate rows')"},{"language":"python-anonymous","snippet":"# Task: decode the three hive partition keys (site_code=<str>, timestamp=<epoch seconds of the year start>, polygon=<WKB hex of a 4x4 degree box>)\nimport struct, pandas as pd, s3fs\nfs = s3fs.S3FileSystem(anon=True, client_kwargs={'region_name': 'ap-southeast-2'})\nroot = 'aodn-cloud-optimised/mooring_estuarine_coastal_water_quality_monitoring_realtime_qc.parquet'\ndef wkb_bbox(hexstr):                       # little-endian WKB Polygon, no shapely needed\n    b = bytes.fromhex(hexstr); n = struct.unpack_from('<I', b, 9)[0]\n    pts = [struct.unpack_from('<dd', b, 13 + 16 * i) for i in range(n)]\n    return (min(p[0] for p in pts), min(p[1] for p in pts), max(p[0] for p in pts), max(p[1] for p in pts))\nfor site_dir in fs.ls(root):\n    if '=' not in site_dir: continue\n    for ts_dir in fs.ls(site_dir):\n        for poly_dir in fs.ls(ts_dir):\n            site = site_dir.split('site_code=')[1]; ts = int(ts_dir.split('timestamp=')[1]); poly = poly_dir.split('polygon=')[1]\n            print(site, pd.Timestamp(ts, unit='s').date(), 'lon/lat bbox', wkb_bbox(poly), len(fs.ls(poly_dir)), 'files')"},{"snippet":"# Task: status board - latest QC-passed temperature, salinity, oxygen and chlorophyll at every site (reads the whole 9 MB store)\nimport pandas as pd\ncols = ['TIME', 'site_code', 'TEMP', 'TEMP_quality_control', 'PSAL', 'PSAL_quality_control', 'DOXY', 'DOXY_quality_control', 'CPHL', 'CPHL_quality_control']\ndf = pd.read_parquet('/mnt/archive', columns=cols)\nrows = []\nfor site, s in df.groupby('site_code', observed=True):\n    r = {'site': site, 'last_obs_utc': s.TIME.max()}\n    for v in ['TEMP', 'PSAL', 'DOXY', 'CPHL']:\n        ok = s[(s[v + '_quality_control'] == 1)].sort_values('TIME')\n        r[v] = round(float(ok[v].iloc[-1]), 2) if len(ok) else None\n    rows.append(r)\nprint(pd.DataFrame(rows).to_string(index=False))","language":"python-local"},{"language":"python-local","snippet":"# Task: QC flag convention is IOC/QARTOD 1 pass, 2 not_evaluated, 3 questionable, 4 fail, 9 missing; show how much each variable loses when you keep only flag 1\nimport pandas as pd\ndf = pd.read_parquet('/mnt/archive', filters=[('site_code', '==', 'SA-ST-KILDA')])\nfor v in ['TEMP', 'PSAL', 'DOXY', 'CPHL', 'TURB', 'FDOM', 'PHYCO', 'DEPTH']:\n    q = df[v + '_quality_control']\n    print(f'{v:6s} measured {int(q.notna().sum()):6d}  pass {int((q == 1).sum()):6d}  questionable {int((q == 3).sum()):6d}  fail {int((q == 4).sum()):5d}')"}],"caveats":["Scale: the store starts 2025-12-18 (SA-ST-KILDA) and the other SA sites begin 2026-01-22 to 2026-05-20. It does NOT cover the 2025 onset of the Karenia mikimotoi bloom (March-November 2025); a '2025-03-01 onward' subset is the whole store. For 2025 conditions use the satellite chlorophyll and SST rows, not this store.","Quirk: two instruments per buoy are stored as SEPARATE rows at the same 15 min cadence, offset by 2-4 minutes. EXO2 sonde rows (filename contains _CKOSTUZ_) carry PSAL, DOXY, DOXS, CPHL, PHYCO, TURB, FDOM, DEPTH; the second fluorometer rows (_BKT_) carry CPHL_PPB, FLU670, FLU690, FLUSI. TEMP is on both. Half the rows have CPHL null (131,787 TEMP-only rows vs 131,222 with both); that is not missing data. Resample per variable rather than dropna across the row.","DO NOT: treat 'realtime_qc' as delayed-mode QC. Every source file is FV00 and the flags come from automated IOOS QARTOD tests (location, gross range, spike, rate of change, flat line, per the _common_metadata global_attributes). Flag 2 (not_evaluated) never occurs; the values are 1, 3, 4, 9. Keep flag == 1 for analysis, but note that doing so drops 25% of CPHL, 55% of FDOM and 73% of PHYCO values (2026-10-04 counts), so a sparse-looking FDOM or phycoerythrin series is a QC artefact, not an outage.","Quirk: deployment handovers duplicate rows. When a buoy is re-deployed the old file keeps extending for about a week, so identical observations appear in two files (Boston Bay: 1,568 rows 2026-09-02 to 09-10; Coffin Bay: 4,772 rows 2026-09-03 to 10-01). drop_duplicates() on the measured columns, or on TIME plus the instrument token, removes them; drop_duplicates(['TIME','filename']) does not.","Quirk: CPHL is fluorescence-INFERRED chlorophyll with a factory Rhodamine calibration (long_name says so), in mg m-3; CPHL_PPB is a second, uranine-calibrated fluorescence in ppb from the other instrument. They are not interchangeable and neither is extracted chlorophyll-a. Small negative CPHL/FDOM values occur and are flagged 3.","Quirk: DEPTH at SA-BOSTON-BAY reads about -10.05 m on 18,533 rows, all flagged 4 (uncalibrated pressure offset); DEPTH is entirely null at SA-COFFIN-BAY and VIC-WESTERN-PORT. Use NOMINAL_DEPTH (0.9 m everywhere) for the sensor depth.","Quirk: hive partition keys are strings in the path: site_code=<SITE>, timestamp=<epoch seconds of 1 January of that year> (1735689600 = 2025, 1767225600 = 2026; yearly here, not monthly), polygon=<WKB hex Polygon of a 4x4 degree lon/lat box>. pandas reads them back as categorical columns; cast site_code with .astype(str) before string ops.","Quirk: columns CPHL_rfu, PHYCOE and PHYCOE_rfu (and their _quality_control twins) exist in the schema but hold no values anywhere on 2026-10-04; they are schema-drift leftovers, do not report them as 'not measured today'.","Quirk: TIME is tz-naive UTC (source filenames carry Z). South Australian local time is UTC+9.5 (ACST) or UTC+10.5 (ACDT), so a 'daily' resample on TIME straddles local midnight by ~10 hours; use df.TIME.dt.tz_localize('UTC').dt.tz_convert('Australia/Adelaide') if local days matter.","Quirk: VIC-WESTERN-PORT is the only non-South-Australian site (IMOS-funded, project = IMOS, 2,351 rows from 2026-09-10) and is outside the SA gulfs bbox; filter site_code.str.startswith('SA-') for the bloom region.","Quirk: with s3fs, pass the path WITHOUT 's3://' when you also pass filesystem=fs (pandas.read_parquet('aodn-cloud-optimised/...parquet', filesystem=fs)); the 's3://' form with an explicit filesystem fails with 'outside base dir'. On the mount, pd.read_parquet('/mnt/archive', filters=[...]) is all that is needed; the root _common_metadata is picked up automatically for the schema.","Scale: the whole store is 9 MB on disk and 97 MB as a pandas frame (263,012 x 49, float32 measurements). Reading everything (28 s anonymously, 2026-10-04) is fine on a 4 GiB pod; one site with a partition filter took 9-17 s. The store is appended several times a day (newest object mtime 2026-10-04 16:49 UTC), so row counts in this card are a floor."],"columns":[{"type":"timestamp[ns], tz-naive UTC; 15 min cadence per instrument, the two instruments at a site are offset by 2-4 min so successive rows alternate ~4 min / ~11 min; range 2025-12-18 to 2026-10-04 13:30","name":"TIME"},{"name":"site_code","type":"string (hive partition key, read back as categorical): SA-AMERICAN-RIVER, SA-BOSTON-BAY, SA-COFFIN-BAY, SA-ST-KILDA, SA-STANSBURY, SA-VICTOR-HARBOR, VIC-WESTERN-PORT"},{"name":"timestamp","type":"int64 hive partition key = epoch seconds of 1 January of the observation year (1735689600 = 2025-01-01, 1767225600 = 2026-01-01); yearly, not monthly, in this store"},{"name":"polygon","type":"string hive partition key: WKB hex (little-endian Polygon) of a 4x4 degree box, e.g. (132,-36)-(136,-32) for Boston Bay and Coffin Bay; four distinct boxes in the store"},{"type":"double, degrees_north / degrees_east, WGS84, nominal mooring position, one constant per site: American River -35.758,137.876; Boston Bay -34.698,135.898; Coffin Bay -34.492,135.382; St Kilda -34.728,138.314; Stansbury -34.930,137.810; Victor Harbor -35.533,138.697; Western Port -38.266,145.275","name":"LATITUDE / LONGITUDE"},{"type":"double, latitude/longitude at most recent GPS fix, carried forward between fixes; present on every row","name":"LATITUDE_GPS / LONGITUDE_GPS"},{"name":"NOMINAL_DEPTH","type":"float, m, positive down, 0.9 at every site"},{"name":"DEPTH","type":"double, m, actual depth from pressure (TEOS-10); EXO2 rows only; QC-passed values 0.41-1.08 m; Boston Bay reports about -10.05 m on 18,533 rows all flagged 4 (pressure offset); null at Coffin Bay and Western Port"},{"name":"TEMP","type":"float, degC, sea_water_temperature; both instruments, every row; QC-passed range 12.6-24.5 (2026-10-04)"},{"name":"PSAL","type":"float, practical salinity (units '1'); EXO2 rows; QC-passed range 33.0-37.3"},{"name":"SPEC_CNDC","type":"float, S m-1, specific conductivity at 25 degC (converted from uS/cm x 1e-4); EXO2 rows"},{"name":"DOXY","type":"float, mg l-1, dissolved oxygen concentration; EXO2 rows; QC-passed 6.5-10.9"},{"type":"float, percent, oxygen saturation; EXO2 rows; QC-passed 91-121","name":"DOXS"},{"name":"CPHL","type":"float, mg m-3, chlorophyll-a INFERRED from fluorescence (factory Rhodamine WT calibration, not extracted pigment); EXO2 rows; QC-passed 0.0-13.2, median 0.77; 32,595 of 131,874 values flagged 3 (questionable), incl. small negative readings"},{"name":"CPHL_RFU","type":"float, relative fluorescence units, raw chlorophyll signal behind CPHL; EXO2 rows"},{"name":"CPHL_PPB","type":"float, ppb (uranine-calibrated) chlorophyll fluorescence from the second fluorometer; BKT rows only; null at VIC-WESTERN-PORT"},{"type":"float, ppb / ppb / dimensionless; 670 nm and 690 nm fluorescence intensity and their spectral shift index FLUSI = C x (FLU690/FLU670) + D; BKT rows only; calibration coefficients in the column metadata","name":"FLU670 / FLU690 / FLUSI"},{"name":"PHYCO / PHYCO_RFU","type":"float, mg m-3 / rfu, phycoerythrin (cyanobacteria pigment) fluorescence; EXO2 rows; 96,391 of 131,874 PHYCO values flagged 3"},{"type":"float, FNU, turbidity; EXO2 rows; QC-passed 1.0-1751.6, median 5.2","name":"TURB"},{"name":"FDOM","type":"float, QSU, fluorescent dissolved organic matter; EXO2 rows; 72,041 of 131,874 flagged 3, 2,178 flagged 9"},{"name":"CPHL_rfu, PHYCOE, PHYCOE_rfu","type":"float columns present in the schema but ENTIRELY NULL on 2026-10-04 (0 of 263,012 rows); their _quality_control columns are all null too"},{"type":"float flag per measured variable: 1 pass, 2 not_evaluated, 3 questionable, 4 fail/bad, 9 missing (IOC Ocean Data Standards 54 vol 3 via IOOS QARTOD); null where the variable is null (other instrument's rows). Observed 2026-10-04 TEMP: 258,107 x1, 398 x3, 4,506 x4, 1 x9; flag 2 never occurs","name":"<VAR>_quality_control (19 columns)"},{"type":"string, source NetCDF name; encodes producer (SARDI or IMOS), instrument stream (BKT or CKOSTUZ), deployment start, site, FV00, deployment id (e.g. SACM-BOS-20260220) and sensor (HAI-AHIW2A-CAD or EXO2-SONDE)","name":"filename"},{"name":"project","type":"string: 'South Australian Coastal Moorings (SACM)' on 260,661 rows, 'Integrated Marine Observing System (IMOS)' on the 2,351 Western Port rows"}],"format":"v2-hybrid","generatedAt":"2026-10-04T00:00:00Z","layout":{"pattern":"hive parquet: /site_code=<SITE>/timestamp=<epoch seconds of 1 Jan of the year>/polygon=<WKB hex of a 4x4 degree lon/lat box>/<SOURCE>_ESTUARINE-COASTAL-MOORINGS_<BKT|CKOSTUZ>_<deploy start UTC>_<SITE>_FV00_<deployment id>_<HAI-AHIW2A-CAD|EXO2-SONDE>.nc-0.parquet; one file per deployment per instrument (24-703 KB each, 31 data files + _common_metadata at the root, no _metadata). Sites 2026-10-04: SA-AMERICAN-RIVER (4 files), SA-BOSTON-BAY (4), SA-COFFIN-BAY (6), SA-ST-KILDA (10, split across timestamp=1735689600 and 1767225600), SA-STANSBURY (4), SA-VICTOR-HARBOR (2), VIC-WESTERN-PORT (1). Deployment files overlap at handovers, so the same observation can appear in two files."},"mountPath":"/mnt/archive","overview":"IMOS Estuarine and Coastal Moorings near real-time water quality (GeoNetwork f3c16fdb-72fc-4b55-bff1-3b5b5722f38f), the SA Harmful Algal Bloom coastal monitoring network buoys processed by SARDI, as a hive-partitioned parquet dataset with a root _common_metadata (no _metadata). Measured 2026-10-04: 32 objects, 9.13 MB, 263,012 rows x 49 columns, seven site_code partitions, each holding 1-8 per-deployment parquet files. Each buoy carries two instruments that land as separate interleaved rows: a YSI EXO2 sonde (CKOSTUZ files: TEMP, PSAL, SPEC_CNDC, DOXY, DOXS, CPHL, CPHL_RFU, PHYCO, PHYCO_RFU, TURB, FDOM, DEPTH) and a second fluorometer package (BKT files: TEMP, CPHL_PPB, FLU670, FLU690, FLUSI), each at 15 minute cadence offset by a few minutes, so TEMP is in every row but CPHL is in half of them (131,222 rows carry both TEMP and CPHL). The data begin 2025-12-18 (SA-ST-KILDA) and the other SA sites start between 2026-01-22 and 2026-05-20; the newest row was written 2026-10-04 13:30 UTC, the day of measurement. Access direction: this is a small table. Read one site with a hive filter (SA-COFFIN-BAY, 38,001 rows, 9.1 s anonymously; SA-BOSTON-BAY 44,261 rows in 16.9 s) or the whole store (263,012 rows, 97 MB in RAM, 28 s anonymously); then resample per site. Do not bother with column pruning for speed, the files are tiny; prune for clarity. TIME is tz-naive UTC (local SA time is UTC+9.5/10.5).","rowCount":"263,012 rows on 2026-10-04 (SA-ST-KILDA 55,738; SA-STANSBURY 48,881; SA-AMERICAN-RIVER 47,541; SA-BOSTON-BAY 44,261; SA-COFFIN-BAY 38,001; SA-VICTOR-HARBOR 26,239; VIC-WESTERN-PORT 2,351); grows daily","sampleHeaders":[{"encoding":"hex","bytes":"5041523115041590a606159ca6064c15c4c9011500120000889303f487c97429a34101189c41a4369c41124e9c414f759c4121939c41d7a39c416fbb9c418dd19c41c4089d41111e9d4131089d4133f99c416ff09c4157ec9c415ef49c413ae99c41cedf9c4100e39c416ff59c41e4f79c411d039d4171559d41be6a9d4149859d419cdc9d41ffec9d410bd29d41b2d79d4122e09d41d6e29d41cae09d41b6db9d4140c19d412fc09d41d0d09d419cc49d4107b69d411cb19d4124b49d41fbcb9d4153b39d41e6969d415c779d41035b9d4193529d418d459d41ba319d41b6109d41971c9d41780b9d4113f29c4169e39c417dcb9c4105c59c419ca79c41e09c9c413a759c41e3709c41dd5e9c41ea5b9c4128449c41fd4d9c41ed479c410a4b9c414e459c41923f9c416a4d9c4193529c41624a9c411a349c4132209c411b2a9c418d289c418a1f9c4148169c414aef9b41cde49b4142ec9b411bd89b41a2d19b41fcc69b41ebc59b41aeb69b4182ad9b4126aa9b41f3ab9b412fa39b41fca99b414da19b41d5959b41f19d9b417c9b9b41baa09b41c39e9b41b1a29b4175ab9b4193a99b41dbbf9b419ace9b4198dd9b41e2e49b41aef59b418bfd9b41c11c9c41d72f9c41fe269c41af259c412d269c41e8309c41863d9c4176379c419d469c4170429c413a589c41a7799c41b77a9c41807d9c418d7f9c4192919c41c5ac","key":"mooring_estuarine_coastal_water_quality_monitoring_realtime_qc.parquet/site_code=SA-AMERICAN-RIVER/timestamp=1767225600/polygon=010300000001000000050000000000000000C0604000000000000042C0000000000040614000000000000042C0000000000040614000000000000040C00000000000C0604000000000000040C00000000000C0604000000000000042C0/SARDI_ESTUARINE-COASTAL-MOORINGS_BKT_20260129T210000Z_SA-AMERICAN-RIVER_FV00_SACM-AMR-20260129_HAI-AHIW2A-CAD.nc-0.parquet","truncatedAt":512},{"key":"mooring_estuarine_coastal_water_quality_monitoring_realtime_qc.parquet/site_code=SA-AMERICAN-RIVER/timestamp=1767225600/polygon=010300000001000000050000000000000000C0604000000000000042C0000000000040614000000000000042C0000000000040614000000000000040C00000000000C0604000000000000040C00000000000C0604000000000000042C0/SARDI_ESTUARINE-COASTAL-MOORINGS_BKT_20260621T232000Z_SA-AMERICAN-RIVER_FV00_SACM-AMR-20260621_HAI-AHIW2A-CAD.nc-0.parquet","encoding":"hex","truncatedAt":512,"bytes":"50415231150415c8bb0415d4bb044c15f28e011500120000e49d02f4e38e1973694151666941539669410ccd694177846941c269694195606941b133694173f46841daac6841f44f68417ff667412ec56741467c6741eb736741745e6741ff78674129b3674119ad6741d5b2674141f16741dcf467416e17684144fa67410dfd674137fd6741b1e16741f9bd67414bc86741c8b5674149a2674182a867413db8674165fc674199476841857c684196956841d3de6841d4f1684184f0684105fa684177f3684102f1684132e668418aab684104736841723368414ed167410c596741f20767416af6664118ec66413fe3664199d866417cd566418ac8664156d466411bbb66414bcd6641fad56641e8f66641082067417385674100e3674191d06841a12d694105a86941b5e0694155f66941ed0d6a41f44f6a41d6906a4187a26a4125926a41e38d6a417f876a41683f6a41df1a6a4162106a41fab369415a47694128f268410a856841912c6841b4b06741691d67415cc96641a7746641a53166410d1a6641e40f6641e3fc6541e3df65417e006641e5f26541540066418e0666410b2466411d206641ee5a6641618e66415d3367413203674109e16741751f6841403069412db26941213c6a41419f6a4154526b41aebb6b41fe606c41249c6c4116306d419d856d41c6f96d41ef206e4109386e414b1f6e41e15d6e4190a0"}],"sampleObjects":["_common_metadata","site_code=SA-BOSTON-BAY/timestamp=1767225600/polygon=01030000000100000005000000000000000080604000000000000042C0000000000000614000000000000042C0000000000000614000000000000040C0000000000080604000000000000040C0000000000080604000000000000042C0/SARDI_ESTUARINE-COASTAL-MOORINGS_CKOSTUZ_20260220T033000Z_SA-BOSTON-BAY_FV00_SACM-BOS-20260220_EXO2-SONDE.nc-0.parquet","site_code=SA-BOSTON-BAY/timestamp=1767225600/polygon=01030000000100000005000000000000000080604000000000000042C0000000000000614000000000000042C0000000000000614000000000000040C0000000000080604000000000000040C0000000000080604000000000000042C0/SARDI_ESTUARINE-COASTAL-MOORINGS_BKT_20260220T033000Z_SA-BOSTON-BAY_FV00_SACM-BOS-20260220_HAI-AHIW2A-CAD.nc-0.parquet","site_code=SA-ST-KILDA/timestamp=1735689600/polygon=01030000000100000005000000000000000000614000000000000042C0000000000080614000000000000042C0000000000080614000000000000040C0000000000000614000000000000040C0000000000000614000000000000042C0/SARDI_ESTUARINE-COASTAL-MOORINGS_CKOSTUZ_20251218T033000Z_SA-ST-KILDA_FV00_SACM-STK-20251218_EXO2-SONDE.nc-0.parquet","site_code=VIC-WESTERN-PORT/timestamp=1767225600/polygon=010300000001000000050000000000000000C0614000000000000044C0000000000040624000000000000044C0000000000040624000000000000042C00000000000C0614000000000000042C00000000000C0614000000000000044C0/IMOS_ESTUARINE-COASTAL-MOORINGS_CKOSTUZ_20260910T020000Z_VIC-WESTERN-PORT_FV00_ECM-WPT-20260910_EXO2-SONDE.nc-0.parquet"],"sampleRows":[{"TIME":"2026-02-21 01:00:00","LATITUDE":-34.698,"NOMINAL_DEPTH":0.9,"DOXY":7.24,"DEPTH":-10.055,"filename":"SARDI_ESTUARINE-COASTAL-MOORINGS_CKOSTUZ_20260220T033000Z_SA-BOSTON-BAY_FV00_SACM-BOS-20260220_EXO2-SONDE.nc","PSAL":36.49,"CPHL_quality_control":3,"TEMP_quality_control":1,"CPHL":-0.19,"site_code":"SA-BOSTON-BAY","LONGITUDE":135.898,"FDOM":-0.3,"TEMP":22.235,"TURB":3.29},{"DOXY":null,"DEPTH":null,"LONGITUDE":135.898,"TEMP":22.182,"FDOM":null,"PSAL":null,"site_code":"SA-BOSTON-BAY","TURB":null,"CPHL_quality_control":null,"TIME":"2026-02-21 01:02:25","LATITUDE":-34.698,"filename":"SARDI_ESTUARINE-COASTAL-MOORINGS_BKT_20260220T033000Z_SA-BOSTON-BAY_FV00_SACM-BOS-20260220_HAI-AHIW2A-CAD.nc","CPHL":null,"NOMINAL_DEPTH":0.9,"TEMP_quality_control":1},{"TEMP":29.719,"LONGITUDE":138.314,"NOMINAL_DEPTH":0.9,"FDOM":null,"site_code":"SA-ST-KILDA","DOXY":null,"TEMP_quality_control":4,"filename":"SARDI_ESTUARINE-COASTAL-MOORINGS_BKT_20251218T033000Z_SA-ST-KILDA_FV00_SACM-STK-20251218_HAI-AHIW2A-CAD.nc","TURB":null,"PSAL":null,"CPHL":null,"CPHL_quality_control":null,"TIME":"2025-12-18 01:17:25","DEPTH":null,"LATITUDE":-34.728}],"sourceObjectKey":"mooring_estuarine_coastal_water_quality_monitoring_realtime_qc.parquet/site_code=SA-AMERICAN-RIVER/timestamp=1767225600/polygon=010300000001000000050000000000000000C0604000000000000042C0000000000040614000000000000042C0000000000040614000000000000040C00000000000C0604000000000000040C00000000000C0604000000000000042C0/SARDI_ESTUARINE-COASTAL-MOORINGS_BKT_20260129T210000Z_SA-AMERICAN-RIVER_FV00_SACM-AMR-20260129_HAI-AHIW2A-CAD.nc-0.parquet","starterCell":"# SA coastal water-quality moorings: daily mean temperature and chlorophyll at one site, 2025-03 onward (QC flag 1 = pass only)\nimport time, pandas as pd\nt0 = time.time()\nsite = 'SA-BOSTON-BAY'   # others: SA-AMERICAN-RIVER, SA-COFFIN-BAY, SA-ST-KILDA, SA-STANSBURY, SA-VICTOR-HARBOR, VIC-WESTERN-PORT\ncols = ['TIME', 'TEMP', 'TEMP_quality_control', 'CPHL', 'CPHL_quality_control']\ndf = pd.read_parquet('/mnt/archive', columns=cols, filters=[('site_code', '==', site)])\ndf = df[df.TIME >= '2025-03-01'].drop_duplicates()        # deployment handovers repeat ~1 week of rows in two files\ntemp = df.loc[df.TEMP_quality_control == 1].set_index('TIME').TEMP.resample('D').mean()\nchl = df.loc[df.CPHL_quality_control == 1].set_index('TIME').CPHL.resample('D').mean()\ndaily = pd.DataFrame({'temp_degC': temp.round(2), 'chl_mg_m3': chl.round(2)}).dropna(how='all')\nprint(site, 'rows', len(df), 'from', df.TIME.min(), 'to', df.TIME.max(), '| read+aggregate', round(time.time() - t0, 1), 's | TIME is UTC')\nprint(daily.to_string(max_rows=24))\n# daily.plot(subplots=True, figsize=(10, 5))   # matplotlib is in the image; chl is fluorescence-inferred, not extracted pigment","starterSamplePath":"site_code=SA-AMERICAN-RIVER/timestamp=1767225600/polygon=010300000001000000050000000000000000C0604000000000000042C0000000000040614000000000000042C0000000000040614000000000000040C00000000000C0604000000000000040C00000000000C0604000000000000042C0/SARDI_ESTUARINE-COASTAL-MOORINGS_BKT_20260129T210000Z_SA-AMERICAN-RIVER_FV00_SACM-AMR-20260129_HAI-AHIW2A-CAD.nc-0.parquet","totalObjects":"32 objects, 9,134,850 bytes (aws s3 ls --recursive --summarize, 2026-10-04)","verifiedRead":{"attempts":[{"family":"parquet","path":"site_code=SA-AMERICAN-RIVER/timestamp=1767225600/polygon=010300000001000000050000000000000000C0604000000000000042C0000000000040614000000000000042C0000000000040614000000000000040C00000000000C0604000000000000040C00000000000C0604000000000000042C0/SARDI_ESTUARINE-COASTAL-MOORINGS_BKT_20260129T210000Z_SA-AMERICAN-RIVER_FV00_SACM-AMR-20260129_HAI-AHIW2A-CAD.nc-0.parquet","seconds":1.31,"ok":true}],"library":"pyarrow","snippet":"import os\nimport pyarrow.parquet as pq\npf = pq.ParquetFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], 'site_code=SA-AMERICAN-RIVER/timestamp=1767225600/polygon=010300000001000000050000000000000000C0604000000000000042C0000000000040614000000000000042C0000000000040614000000000000040C00000000000C0604000000000000040C00000000000C0604000000000000042C0/SARDI_ESTUARINE-COASTAL-MOORINGS_BKT_20260129T210000Z_SA-AMERICAN-RIVER_FV00_SACM-AMR-20260129_HAI-AHIW2A-CAD.nc-0.parquet'))\nprint(pf.metadata)\ndf = pf.read_row_group(0).to_pandas()\ndf.head()","facts":["under 'mooring_estuarine_coastal_water_quality_monitoring_realtime_qc.parquet/': 7 directories (e.g. site_code=SA-AMERICAN-RIVER, site_code=SA-BOSTON-BAY, site_code=SA-COFFIN-BAY, site_code=SA-ST-KILDA); 1 objects named like _common_metadata, 90 KB to 90 KB","under 'mooring_estuarine_coastal_water_quality_monitoring_realtime_qc.parquet/site_code=VIC-WESTERN-PORT/': 1 directories (e.g. timestamp=1767225600)","under 'mooring_estuarine_coastal_water_quality_monitoring_realtime_qc.parquet/site_code=VIC-WESTERN-PORT/timestamp=1767225600/': 1 directories (e.g. polygon=010300000001000000050000000000000000C0614000000000000044C0000000000040624000000000000044C0000000000040624000000000000042C00000000000C0614000000000000042C00000000000C0614000000000000044C0)","under 'mooring_estuarine_coastal_water_quality_monitoring_realtime_qc.parquet/site_code=VIC-WESTERN-PORT/timestamp=1767225600/polygon=010300000001000000050000000000000000C0614000000000000044C0000000000040624000000000000044C0000000000040624000000000000042C00000000000C0614000000000000042C00000000000C0614000000000000044C0/': 1 objects named like IMOS_ESTUARINE-COASTAL-MOORINGS_CKOSTUZ_########T######Z_VIC-WESTERN-PORT_FV##_ECM-WPT-########_EXO#-SONDE.nc-#.parquet, 115 KB to 115 KB"],"image":"531775725781.dkr.ecr.us-east-1.amazonaws.com/scigantic-notebook-geospatial:latest","path":"site_code=SA-AMERICAN-RIVER/timestamp=1767225600/polygon=010300000001000000050000000000000000C0604000000000000042C0000000000040614000000000000042C0000000000040614000000000000040C00000000000C0604000000000000040C00000000000C0604000000000000042C0/SARDI_ESTUARINE-COASTAL-MOORINGS_BKT_20260129T210000Z_SA-AMERICAN-RIVER_FV00_SACM-AMR-20260129_HAI-AHIW2A-CAD.nc-0.parquet","starterCell":{"seconds":18.95,"ok":true},"verifiedAt":"2026-10-05T12:59:26.068Z","summary":"Parquet 13,738 rows, 46 columns, 1 row groups; columns ['TEMP:float', 'TIME:timestamp[ns]', 'FLUSI:float', 'FLU670:float', 'FLU690:float', 'project:string', 'CPHL_PPB:float', 'LATITUDE:double', 'filename:string', 'LONGITUDE:double']","rssDeltaMb":0,"family":"parquet","seconds":1.31,"bytes":523155}},"summary":"The IMOS Estuarine and Coastal Moorings Facility provides a national, standardised network of estuarine and coastal moorings focused on water quality.","title":"Moorings - Estuarine and coastal water quality monitoring - Near real time","updatedAt":"2026-10-07T13:30:43.743Z","visibility":"public","websiteUrl":"https://catalogue-imos.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/f3c16fdb-72fc-4b55-bff1-3b5b5722f38f"}}