{"success":true,"data":{"author":"AODN","bucketAccessType":"public","bucketListable":true,"bucketName":"aodn-cloud-optimised","bucketPrefix":"aggregated_seagrass_nonqc.parquet","category":"Life Sciences","cloudProvider":"aws","createdAt":"2026-10-04T00:00:00.000Z","description":"The Australian Seagrass Aggregated Data Product harmonises disparate seagrass datasets from providers across Australia into a single, standardised resource suitable for ecological assessment, long-term monitoring and habitat modelling. Repeat aerial and satellite remote sensing surveys provide comprehensive spatial coverage for regional and long-term mapping programs; many legacy datasets, including the NSW estuarine macrophytes and the Cockburn Sound historical time series, were derived from purpose-flown aerial photography digitised into habitat polygons with field validation, while recent datasets add UAV orthomosaics, high-resolution multispectral satellite imagery (WorldView-2, IKONOS, QuickBird-2) and PlanetScope imagery to model seagrass distribution, species composition and biomass. Source data were collated from Seamap Australia (IMAS, TropWATER/JCU, CSIRO, AIMS and state agencies such as SA DEWNR, NSW DPE, Vic DEECA and WA DWER) under NESP Marine and Coastal Hub Project 5.9. This store is the cloud-optimised Parquet copy: 3,286,273 long-format presence/absence rows for 29 seagrass taxa from 43 source datasets, 1967 to February 2025, one hive partition per year, 247.5 MiB, each row carrying a WKT footprint, its centroid, a WoRMS name and the source dataset name and metadata link; licensed CC-BY-NC 4.0 because one contributing dataset is non-commercial.","displayTitle":"Seagrass - Australian seagrass occurrence - Aggregated data product (1967 - ongoing)","fuseMountBrowsable":true,"id":"ed1ee487-90e3-5bce-a077-8f3323837918","license":"http://creativecommons.org/licenses/by-nc/4.0/","objectCountEstimate":42,"probeStatus":"ok","region":"ap-southeast-2","schemaCard":{"accessHints":[{"snippet":"# Task: see the 42 year partitions and their sizes (8 KiB to 51.5 MiB); note the negative epoch for 1967\naws s3 ls s3://aodn-cloud-optimised/aggregated_seagrass_nonqc.parquet/ --no-sign-request --region ap-southeast-2\nls /mnt/archive | sort -t= -k2 -n | head","language":"bash"},{"language":"python-local","snippet":"# Task: schema and GeoNetwork uuid from the sidecar, then an exact row count from the file footers (no column data read)\nimport pyarrow.parquet as pq, pyarrow.dataset as ds\nschema = pq.read_schema('/mnt/archive/_common_metadata')\nprint(schema.names)\nprint(schema.metadata[b'global_attributes'])\nprint(schema.field('_temporal_extent').metadata[b'definition'][:120])\ndset = ds.dataset('/mnt/archive', format='parquet', partitioning='hive')   # _common_metadata is skipped automatically\nprint('rows:', dset.count_rows(), 'files:', len(dset.files))"},{"language":"python-anonymous","snippet":"# Task: schema and GeoNetwork uuid from the sidecar, then an exact row count from the file footers (no column data read)\nimport pyarrow.parquet as pq, pyarrow.dataset as ds, s3fs\nfs = s3fs.S3FileSystem(anon=True, client_kwargs={'region_name': 'ap-southeast-2'})\nROOT = 'aodn-cloud-optimised/aggregated_seagrass_nonqc.parquet'\nwith fs.open(f'{ROOT}/_common_metadata', 'rb') as f:\n    schema = pq.read_schema(f)\nprint(schema.names)\nprint(schema.metadata[b'global_attributes'])\ndset = ds.dataset(ROOT, filesystem=fs, format='parquet', partitioning='hive')\nprint('rows:', dset.count_rows(), 'files:', len(dset.files))   # 3,286,273 rows, 42 files; footers only"},{"language":"python-local","snippet":"# Task: ONE year partition WITHOUT the geometry column (2015 = 1420070400: 127,052 rows, ~78 MB in pandas instead of 226 MB)\nimport pyarrow.dataset as ds, pyarrow.compute as pc\ndset = ds.dataset('/mnt/archive', format='parquet', partitioning='hive')\ncols = [c for c in dset.schema.names if c not in ('footprintWKT', 'polygon', 'filename')]\ny2015 = dset.to_table(filter=pc.field('timestamp') == 1420070400, columns=cols).to_pandas()\nprint(len(y2015), y2015.datasetName.value_counts().head())"},{"language":"python-anonymous","snippet":"# Task: ONE year partition WITHOUT the geometry column (2015 = 1420070400: 127,052 rows, ~78 MB in pandas instead of 226 MB)\nimport pyarrow.dataset as ds, pyarrow.compute as pc, s3fs\nfs = s3fs.S3FileSystem(anon=True, client_kwargs={'region_name': 'ap-southeast-2'})\ndset = ds.dataset('aodn-cloud-optimised/aggregated_seagrass_nonqc.parquet', filesystem=fs, format='parquet', partitioning='hive')\ncols = [c for c in dset.schema.names if c not in ('footprintWKT', 'polygon', 'filename')]\ny2015 = dset.to_table(filter=pc.field('timestamp') == 1420070400, columns=cols).to_pandas()\nprint(len(y2015), y2015.datasetName.value_counts().head())"},{"snippet":"# Task: presence rows for one taxon across all years with filter pushdown (Posidonia australis: 1,823 rows from 4 datasets)\nimport pyarrow.dataset as ds, pyarrow.compute as pc\ndset = ds.dataset('/mnt/archive', format='parquet', partitioning='hive')\npos = dset.to_table(filter=(pc.field('scientificName') == 'Posidonia australis') & (pc.field('occurrenceStatus') == 'present'),\n                    columns=['datasetName', 'startYear', 'decimalLatitude', 'decimalLongitude']).to_pandas()\nprint(pos.groupby('datasetName').size())","language":"python-local"},{"snippet":"# Task: geometry for a SMALL subset only (footprintWKT is 1.8 GB over the whole store; one cell can be 13 MB)\nimport pyarrow.dataset as ds, pyarrow.compute as pc\ndset = ds.dataset('/mnt/archive', format='parquet', partitioning='hive')\ngeo = dset.to_table(filter=pc.field('timestamp') == 1735689600,   # 2025: 25 rows\n                    columns=['datasetName', 'scientificName', 'occurrenceStatus', 'footprintWKT']).to_pandas()\nprint(geo.footprintWKT.str[:7].value_counts())              # POINT ( / POLYGON / MULTIPO\n# optional, only if shapely is installed in this image:\n# from shapely import wkt; geo['geom'] = geo.footprintWKT.map(wkt.loads); print(geo.geom.iloc[0].bounds)","language":"python-local"},{"snippet":"# Task: plain pandas read of one small partition when you do not need pyarrow.dataset (1967 Cockburn Sound block, 10,387 rows, 1.5 MiB)\nimport pandas as pd\nold = pd.read_parquet('/mnt/archive/timestamp=-94694400')\nprint(old.datasetName.iloc[0], '|', old.scientificName.unique(), '|', old.occurrenceStatus.value_counts().to_dict())","language":"python-local"}],"caveats":["Scale: DO NOT read this store whole in a 4 GiB pod. footprintWKT alone is 1.84 GB in Arrow (one cell is 13,461,181 characters) and the other 23 columns are 2.79 GB in pandas for 3,286,273 rows. Always pass columns=[...] and a pushdown filter (pc.field('timestamp'), decimalLatitude/decimalLongitude); the row groups are sorted by sourceID then date so dataset and bbox filters prune, taxon filters do not. From outside the region on a ~100-300 KB/s link the SA bbox pushdown took 831 s and a full no-geometry read 838 s (2.41 GB Arrow); inside AWS expect minutes, not seconds, for anything that touches all 42 partitions, and seconds for a single year partition.","DO NOT: expect bloom-period South Australian data. SA rows (lat -36.5..-32, lon 135..139.5) are 129,589 (3.9%), all from 'South Australia State Marine Benthic Habitats (DEWNR)' (123,424; start years 1996-2013) and 'Estuarine Habitats of South Australia' (6,165), latest 2013-05-10, nothing after 2013 and 0 rows since 2025-03-01. The store gives a pre-bloom baseline of mapped seagrass in Spencer Gulf, Gulf St Vincent, Eyre Peninsula and the Coorong, not a 2025-2026 condition.","Departure from canonical: 87% of rows are ABSENCES (occurrenceStatus 'absent' 2,862,901 vs 'present' 423,372). Each mapped feature is expanded into one row per taxon the source scheme could report, so counting rows measures mapping effort x taxon list, not seagrass. Filter occurrenceStatus == 'present' before any extent, richness or trend statement. In SA the 2002-2008 presences are all 'Tracheophyta' (seagrass present, taxon unresolved); species-level presences appear only in the 2013 block.","Quirk: 'Tracheophyta' (taxonRank 'Phylum (Division)', 616,081 rows) means 'seagrass of unknown taxon', from sources that mapped habitat classes rather than species; genus-level rows (642,493) come from 'sp./spp.' labels rolled up to the parent. 193,471 rows carry taxonomicStatus 'unaccepted' names kept as the source reported them (Zostera capricorni, Heterozostera tasmanica); align via scientificNameID (WoRMS AphiaID, int64) if you need current names.","Quirk: the partition key timestamp is epoch seconds of 1 January of startYear and is NEGATIVE for 1967 (-94694400): that partition is the Cockburn Sound (WA) 1967 aerial-photo seagrass map ('Changes in seagrass coverage in Cockburn Sound, Western Australia between 1967 and 1999'), 10,387 rows, all 'Tracheophyta', lat -32.28..-32.14, lon 115.66..115.77, read in 9.0 s. The next partition is 1972 (63072000); 1983 has 22 rows; 2025 has 25 rows (8.4 KiB) despite the GeoNetwork extent ending 2024-12-31.","Quirk: _temporal_extent (date32) is a partitioning convenience built as startYear-(startMonth or 1)-(startDay or 1); the column's own metadata warns against using it for day-sensitive analysis. startDay is NULL on 74% of rows and startMonth on 9%, endYear is filled on only 11,715 rows; most sources know the survey year or month only. pandas returns _temporal_extent as object (datetime.date).","Quirk: organismQuantityType is the literal 'biomass' on every row, but organismQuantity (gdw/m^2) is filled on only 1,533 rows, all from the Hervey Bay / Great Sandy Strait 1998 survey. Do not read organismQuantityType as evidence that biomass exists.","Quirk: decimalLatitude/decimalLongitude are float32 CENTROIDS of footprintWKT (polygons up to whole meadows), so a centroid can fall outside the meadow or on land; the exact shape is in footprintWKT (WKT text in EPSG:4326, not WKB). shapely is not guaranteed in the Life Sciences image: use pandas string ops (startswith 'POINT'/'POLYGON') for triage and only parse WKT for a filtered subset; parsing all 3.3 M geometries is a multi-GB job.","Scale: filter pushdown prunes by PLACE and TIME, not by taxon. Files are sorted by sourceID then date and every mapped feature contributes one row per taxon, so a scientificName or occurrenceStatus filter matches every row group and degenerates into a full no-geometry scan (a Posidonia australis pushdown over S3 was abandoned after 26 min from outside the region; it finishes in 0.1 s on a local copy). Prune with timestamp=<year> or a decimalLatitude/decimalLongitude box first, then filter taxon in pandas.","Quirk: the second hive key polygon=<WKB hex> has one value for all 42 partitions (eastern-hemisphere box) and prunes nothing; pyarrow infers timestamp as int32. Root has _common_metadata only, no _metadata.","Quirk: metadataLink is the literal string 'unknown' for 6 datasets including the largest (GBR seagrass mapping synthesis, 1,139,418 rows) and the Tayaritja and Torres Strait 2016/2017/2020 surveys; one datasetName (Lizard Island) ends in two newline characters, so match datasetName with str.contains or str.strip().","Quirk: licence is CC-BY-NC 4.0 for the aggregate (the record states the sources are a mix of CC-BY, majority, and CC-BY-NC); per the AODN source metadata CSV, 32 sources are CC BY 4.0, 4 CC BY 3.0, 6 'Open Access' (WA DWER) and ONE CC BY-NC 4.0 (NSW Estuarine Macrophytes 2023, 717,685 rows = 22% of the store). Dropping that datasetName leaves CC-BY material, but confirm against each source's own record before commercial use. The kelp sibling store is CC-BY 4.0.","Quirk: no RODA page (https://registry.opendata.aws/roda/aodn_aggregated_seagrass_nonqc/ returned 404 on 2026-10-04) and no dataset DOI on the GeoNetwork record (the RAiD 10.82210/a44c9c1d is the NESP project id). Object dates are 2026-05-08, matching the record's 2026-05 revision; the technical document says the product refreshes on a schedule but no cadence is stated."],"columns":[{"name":"datasetName","type":"string, not null; 43 source dataset titles, e.g. 'South Australia State Marine Benthic Habitats (DEWNR)', 'NSW Estuarine Macrophytes 2023', 'Seagrass mapping synthesis: ... Great Barrier Reef (NESP TWQ 3.2.1 and NESP TWQ 5.4, TropWATER, James Cook University)' (one title carries trailing '\\n\\n': Lizard Island)"},{"name":"sourceID","type":"string, not null; '<source table>.<row>' e.g. 'SeamapAus_SA_state_benthic_habitats_2013.5646'; the primary sort key of the files"},{"name":"metadataLink","type":"string, not null; source record URL (IMAS/eAtlas GeoNetwork, data.wa.gov.au, doi.org) or the literal 'unknown' (6 datasets incl. the 1.14 M-row GBR synthesis)"},{"name":"startYear","type":"int16, not null; 1967 to 2025; equals the partition year on 100% of rows"},{"type":"int8 (float64 in pandas because of nulls), NULL on 297,300 rows; 1-12","name":"startMonth"},{"type":"int8, NULL on 2,437,001 rows (74%); most sources only know the month or year","name":"startDay"},{"name":"endYear / endMonth / endDay","type":"int16 / int8 / int8, filled on only 11,715 / 11,715 / 11,047 rows (multi-year surveys), 1997 to 2023"},{"name":"_temporal_extent","type":"date32[day] (pandas object of datetime.date), not null; startYear-(startMonth or 1)-(startDay or 1); 1967-01-01 to 2025-02-23; the schema metadata itself says: for partitioning only, use startYear/startMonth/startDay for analysis"},{"name":"footprintWKT","type":"string, not null; WKT in EPSG:4326: POINT (1,863,283 rows), POLYGON (1,389,443), MULTIPOLYGON (11,453); median 27 chars, p90 886, max 13.46 M chars; 1.84 GB in Arrow for the whole store"},{"name":"decimalLatitude","type":"float32 (NOT double), EPSG:4326, not null; centroid of footprintWKT; -40.5809 to -9.13895"},{"name":"decimalLongitude","type":"float32, not null; centroid; 114.99439 to 153.60187"},{"name":"h3Index","type":"string, not null; H3 cell at resolution 5 from the centroid, e.g. '85b914c7fffffff'; 1,746 distinct (the technical document's 'resolution 15' sentence is contradicted by the schema metadata and the 15-character cell ids: it is resolution 5)"},{"name":"australianMarineRegionsTags","type":"string, NULL on 358,535 rows; '|'-separated IMCRA / Marine Park / World Heritage Area tags; SA rows carry 'IMCRA:Mesoscale:Name:Spencer Gulf' (37,613), 'North Spencer Gulf' (31,948), 'St Vincent Gulf' (34,943), 'Coorong' (27,322), 'Eyre' (46,368), 'Murat' (20,618) under 'IMCRA:Provincial:Name:Spencer Gulf Shelf Province' (172,506)"},{"type":"string, not null; 29 WoRMS/CAAB-aligned names: Tracheophyta (phylum-level 'seagrass'), 7 genus-level (Ruppia, Zostera, Posidonia, Halophila, Amphibolis ...), 21 species (Halodule uninervis, Halophila ovalis, Syringodium isoetifolium, Posidonia sinuosa, P. australis, Amphibolis antarctica, Zostera muelleri ...)","name":"scientificName"},{"name":"scientificNameID","type":"int64, not null; WoRMS AphiaID (e.g. 374714 Posidonia sinuosa), NOT an LSID string like the kelp sibling"},{"name":"taxonRank","type":"string, not null; Species 2,027,699 / Genus 642,493 / 'Phylum (Division)' 616,081"},{"type":"string, not null; accepted 3,092,802 / unaccepted 193,471 (names such as Zostera capricorni and Heterozostera tasmanica kept as reported by the source)","name":"taxonomicStatus"},{"name":"occurrenceStatus","type":"string, not null; 'absent' 2,862,901 / 'present' 423,372: every taxon the source scheme could report is written out per feature"},{"type":"float32 gdw/m^2 biomass, filled on 1,533 rows only (all from 'Seagrass meadows of Hervey Bay and the Great Sandy Strait ... 1998'), 0 to 51.03","name":"organismQuantity"},{"type":"string; the literal 'biomass' on ALL 3,286,273 rows even where organismQuantity is NULL","name":"organismQuantityType"},{"name":"filename","type":"string; always 'seagrass.parquet'"},{"type":"hive partition key, int32 as inferred by pyarrow; epoch seconds of 1 January of startYear, -94694400 (1967) .. 1735689600 (2025); only exists in the directory name","name":"timestamp"},{"type":"hive partition key, string WKB hex, single value (eastern-hemisphere box) for all partitions","name":"polygon"}],"format":"v2-hybrid","generatedAt":"2026-10-04T00:00:00Z","layout":{"pattern":"Hive-partitioned Parquet directory. Root holds ONE sidecar, _common_metadata (25 KiB: Arrow schema with per-column Darwin Core metadata and global_attributes {metadata_uuid '009a1131-...', dataset_name}); NO _metadata file. Data: timestamp=<epoch seconds of 1 Jan startYear>/polygon=<WKB hex>/seagrass.parquet-0.parquet, 42 partitions from timestamp=-94694400 (1967-01-01, a NEGATIVE epoch that sorts first in listings) and 63072000 (1972) to 1735689600 (2025); one file each, 8.4 KiB (2025) to 51.5 MiB (2004), total 247.5 MiB; the biggest are 2004 51.5 MiB, 2022 29.9 MiB, 2008 24.7 MiB, 2015 21.7 MiB, 2005 13.5 MiB, 2020 13.4 MiB. The polygon key has a single value for all partitions (eastern-hemisphere box). Objects dated 2026-05-08. Files are sorted by sourceID then _temporal_extent, so row groups within a partition cluster by source dataset and the lat/lon statistics make dataset- and bbox-pushdown effective."},"mountPath":"/mnt/archive","overview":"IMOS/NESP MaC 5.9 'Australian Seagrass Occurrence - Aggregated data product' (technical document Seagrass_v1.1, May 2026): 43 Seamap Australia source datasets re-encoded into one long-format Darwin Core table, one row per taxon per mapped feature, carrying BOTH presence and absence. Measured 2026-10-04: 3,286,273 rows, 42 year-partitions 1967..2025, 29 scientificName values (Tracheophyta 616,081 = seagrass of unknown taxon; Ruppia, Zostera, Halodule, Halophila, Posidonia, Amphibolis ...), absent 2,862,901 (87%) vs present 423,372, lat -40.58 to -9.14, lon 114.99 to 153.60; seven datasets hold 93% of rows (GBR synthesis 1,139,418; NSW Estuarine Macrophytes 2023 717,685; Torres Strait/Gulf of Carpentaria 626,724; SA DEWNR benthic habitats 219,720). The access direction is dictated by footprintWKT (POINT 1.86 M rows, POLYGON 1.39 M, MULTIPOLYGON 11 k; one cell is 13.46 M characters): 1.84 GB in Arrow on its own, the other 23 columns 2.79 GB in pandas, so a 4 GiB pod cannot read the store whole; project columns and prune by year partition or lat/lon box. Over s3fs from outside the region on a slow link: count_rows from footers 48 to 123 s, the 2015 partition (127,052 rows) 28 s with geometry (226 MB pandas) or 62 s without (78 MB), the SA bounding box pushdown over all partitions 831 s (129,589 rows), and a full read of the 23 non-geometry columns 838 s (2.41 GB Arrow, 2.79 GB pandas). South Australia: 129,589 rows (3.9%) from DEWNR State Marine Benthic Habitats and Estuarine Habitats of SA, 1996-2013, latest 2013-05-10, 1,846 present rows, nothing after 2013: the bloom period is not in this store.","rowCount":"3,286,273 rows (pyarrow count_rows over all 42 partitions, 2026-10-04); 26 columns in the schema (24 in the files + hive keys timestamp and polygon)","sampleHeaders":[{"truncatedAt":512,"bytes":"504152311504150015024c15001500120000001500151015142c15da6c1510150615061c36da6c000000081c03000000da6c00001504150015024c15001500120000001500151015142c15da6c1510150615061c36da6c000000081c03000000da6c0000150415980115584c150815001200004c300f0000003835613732366437660901191304396209120d131063393933344a13001c6666666666666666150015c01a15801a2c15da6c1510150615061c3600280f383563393933346666666666666666180f3835613732363962666666666666661111000000a00df09a03000000da6c01027140fefacfafbaf2bfeabcbb3faffecaebaff3fabaa8eb2abbfe8baffbd3fbefe2fbffbcfa3bafffcffbebf3bbfbfcbf2ffebecfbbfef3eeeff0ffbfbcee3ebabecebafeb2abeefcff2febff8bfffbf7fbfff8ff3ffcff3fffffcfaebef3bffffcff3fefffdeffabf3aafefdfe020000102c00050100000446000341005a00050500000416000301402000070100001010003201211c400012000301001a0105281e0009010004000110004001151c4116000701000010012cf03c0d01000410104004515400040012000b5100040015055141005510001b054005040401044004001500005001404100010004040050004014000341001a016818000158000501100190a80f010004000054010000400010010010000b1141","key":"aggregated_seagrass_nonqc.parquet/timestamp=-94694400/polygon=01030000000100000005000000000000000000000000000000008056C0000000000080664000000000008056C00000000000806640000000000080564000000000000000000000000000805640000000000000000000000000008056C0/seagrass.parquet-0.parquet","encoding":"hex"}],"sampleObjects":["_common_metadata","timestamp=-94694400/polygon=01030000000100000005000000000000000000000000000000008056C0000000000080664000000000008056C00000000000806640000000000080564000000000000000000000000000805640000000000000000000000000008056C0/seagrass.parquet-0.parquet","timestamp=1072915200/polygon=01030000000100000005000000000000000000000000000000008056C0000000000080664000000000008056C00000000000806640000000000080564000000000000000000000000000805640000000000000000000000000008056C0/seagrass.parquet-0.parquet","timestamp=1420070400/polygon=01030000000100000005000000000000000000000000000000008056C0000000000080664000000000008056C00000000000806640000000000080564000000000000000000000000000805640000000000000000000000000008056C0/seagrass.parquet-0.parquet","timestamp=1735689600/polygon=01030000000100000005000000000000000000000000000000008056C0000000000080664000000000008056C00000000000806640000000000080564000000000000000000000000000805640000000000000000000000000008056C0/seagrass.parquet-0.parquet"],"sampleRows":[{"scientificNameID":374714,"startDay":1,"_temporal_extent":"2008-01-01","timestamp":1199145600,"sourceID":"SeamapAus_SA_state_benthic_habitats_2013.24701","datasetName":"South Australia State Marine Benthic Habitats (DEWNR)","metadataLink":"https://metadata.imas.utas.edu.au/geonetwork/srv/eng/catalog.search#/metadata/cbe02d9e-2450-4364-ad76-c1d27f030943","decimalLongitude":135.425858,"occurrenceStatus":"absent","startYear":2008,"taxonRank":"Species","decimalLatitude":-34.600414,"organismQuantity":null,"h3Index":"85b91133fffffff","organismQuantityType":"biomass","australianMarineRegionsTags":"IMCRA:Provincial:Name:Spencer Gulf Shelf Province|IMCRA:Mesoscale:Water Type:Warm Temperate Waters|IMCRA:Mesoscale:Name:Eyre|IMCRA:Provincial:Water Type:Warm Temperate Waters","scientificName":"Posidonia sinuosa","taxonomicStatus":"accepted","startMonth":1},{"timestamp":1356998400,"datasetName":"South Australia State Marine Benthic Habitats (DEWNR)","startYear":2013,"occurrenceStatus":"present","taxonRank":"Genus","decimalLatitude":-35.791355,"startMonth":5,"startDay":10,"decimalLongitude":137.763016,"scientificName":"Zostera","sourceID":"(2013 block)"},{"startMonth":10,"scientificNameID":208930,"decimalLatitude":-23.78014,"decimalLongitude":151.294357,"h3Index":"859d21d3fffffff","timestamp":1072915200,"occurrenceStatus":"absent","sourceID":"GBR_NESP-TWQ-3.2.1-5.4_JCU_Seagrass_1984-2018_Site-surveys.21712","metadataLink":"unknown","datasetName":"Seagrass mapping synthesis: A resource for coastal management in the Great Barrier Reef (NESP TWQ 3.2.1 and NESP TWQ 5.4, TropWATER, James Cook University)","startYear":2004,"_temporal_extent":"2004-10-01","scientificName":"Halophila ovalis","startDay":null}],"sourceObjectKey":"aggregated_seagrass_nonqc.parquet/timestamp=-94694400/polygon=01030000000100000005000000000000000000000000000000008056C0000000000080664000000000008056C00000000000806640000000000080564000000000000000000000000000805640000000000000000000000000008056C0/seagrass.parquet-0.parquet","starterCell":"# South Australian gulfs (lat -36.5..-32, lon 135..139.5): seagrass PRESENCE rows by year and taxon, with timing\nimport time, pyarrow.dataset as ds, pyarrow.compute as pc\ndset = ds.dataset('/mnt/archive', format='parquet', partitioning='hive')\nt0 = time.time()\nsa = dset.to_table(\n    filter=(pc.field('decimalLatitude') >= -36.5) & (pc.field('decimalLatitude') <= -32.0) &\n           (pc.field('decimalLongitude') >= 135.0) & (pc.field('decimalLongitude') <= 139.5),\n    columns=['datasetName', 'startYear', 'startMonth', 'scientificName', 'occurrenceStatus', 'decimalLatitude', 'decimalLongitude']).to_pandas()\nprint(f'{len(sa):,} SA-gulfs rows (of {dset.count_rows():,} nationally) in {time.time()-t0:.1f}s; datasets: {sa.datasetName.unique().tolist()}')\npresent = sa[sa.occurrenceStatus == 'present']\nprint(present.groupby(['startYear', 'scientificName']).size().unstack(fill_value=0).to_string())\nprint('latest SA record year:', sa.startYear.max(), '| present rows:', len(present), '| rows since 2025-03-01:', int((sa.startYear >= 2025).sum()))\n# measured 2026-10-04 (anonymous s3fs from outside the region, 830.9 s; 0.0 s on a local copy): 129,589 SA rows (2 datasets: DEWNR State Marine Benthic Habitats, Estuarine Habitats of SA), start years 1996-2013,\n# 1,846 present rows (2002-2008 all 'Tracheophyta'; 2013 adds Zostera 191, Posidonia australis 151, Heterozostera 116, Halophila australis 75 ...), 0 rows after 2013","starterSamplePath":"timestamp=-94694400/polygon=01030000000100000005000000000000000000000000000000008056C0000000000080664000000000008056C00000000000806640000000000080564000000000000000000000000000805640000000000000000000000000008056C0/seagrass.parquet-0.parquet","totalObjects":"43 objects, 247.5 MiB (aws s3 ls --summarize, 2026-10-04): 1 x _common_metadata + 42 x seagrass.parquet-0.parquet","verifiedRead":{"bytes":227839,"snippet":"import os\nimport pyarrow.parquet as pq\npf = pq.ParquetFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], 'timestamp=978307200/polygon=01030000000100000005000000000000000000000000000000008056C0000000000080664000000000008056C00000000000806640000000000080564000000000000000000000000000805640000000000000000000000000008056C0/seagrass.parquet-0.parquet'))\nprint(pf.metadata)\ndf = pf.read_row_group(0).to_pandas()\ndf.head()","seconds":1.31,"image":"531775725781.dkr.ecr.us-east-1.amazonaws.com/scigantic-notebook-genomics:latest","path":"timestamp=978307200/polygon=01030000000100000005000000000000000000000000000000008056C0000000000080664000000000008056C00000000000806640000000000080564000000000000000000000000000805640000000000000000000000000008056C0/seagrass.parquet-0.parquet","rssDeltaMb":0,"summary":"Parquet 27,050 rows, 23 columns, 17 row groups; columns ['endDay:int8', 'endYear:int16', 'h3Index:string', 'endMonth:int8', 'sourceID:string', 'startDay:int8', 'startYear:int16', 'taxonRank:string', 'startMonth:int8', 'datasetName:string']","library":"pyarrow","family":"parquet","facts":["under 'aggregated_seagrass_nonqc.parquet/': 42 directories (e.g. timestamp=-94694400, timestamp=1009843200, timestamp=1041379200, timestamp=1072915200); 1 objects named like _common_metadata, 25 KB to 25 KB","under 'aggregated_seagrass_nonqc.parquet/timestamp=978307200/': 1 directories (e.g. polygon=01030000000100000005000000000000000000000000000000008056C0000000000080664000000000008056C00000000000806640000000000080564000000000000000000000000000805640000000000000000000000000008056C0)","under 'aggregated_seagrass_nonqc.parquet/timestamp=978307200/polygon=01030000000100000005000000000000000000000000000000008056C0000000000080664000000000008056C00000000000806640000000000080564000000000000000000000000000805640000000000000000000000000008056C0/': 1 objects named like seagrass.parquet-#.parquet, 222 KB to 222 KB"],"attempts":[{"family":"parquet","path":"timestamp=978307200/polygon=01030000000100000005000000000000000000000000000000008056C0000000000080664000000000008056C00000000000806640000000000080564000000000000000000000000000805640000000000000000000000000008056C0/seagrass.parquet-0.parquet","ok":true,"seconds":1.31}],"starterCell":{"ok":true,"seconds":71.67},"verifiedAt":"2026-10-05T12:59:09.883Z"}},"summary":"The Australian Seagrass Aggregated Data Product harmonises disparate seagrass datasets from providers across Australia into a single, standardised resource suitable for ecological…","title":"Seagrass - Australian seagrass occurrence - Aggregated data product (1967 - ongoing)","updatedAt":"2026-10-07T13:07:53.427Z","visibility":"public","websiteUrl":"https://catalogue-imos.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/009a1131-efc1-4a61-8f90-cf289e7c043d"}}