{"success":true,"data":{"author":"AODN","bucketAccessType":"public","bucketListable":true,"bucketName":"aodn-cloud-optimised","bucketPrefix":"diver_benthic_cover_in_situ_qc.parquet","category":"Life Sciences","cloudProvider":"aws","createdAt":"2026-10-04T00:00:00.000Z","description":"The National Reef Monitoring Network (NRMN) brings together shallow reef surveys conducted around Australia into a centralised database. The IMOS NRMN Sub-Facility collates, cleans, stores and makes this data rapidly available from contributors including Reef Life Survey, Parks Australia, DBCA (Western Australia), DEW (South Australia), DPI (New South Wales), Tasmanian Parks and Wildlife Service and Parks Victoria. This dataset contains the cover of macroalgae and sessile invertebrates scored in situ at Australian Temperate Reef Collaboration (ATRC) sites: quadrats are placed at 10 m spacing along each transect (5 per 50 m transect), and the canopy, mid-storey and substrate layers are assessed in sequence in each single 50-point quadrat by divers skilled in macroalgal identification. This store is the quality-controlled table as hive-partitioned parquet: 489,046 point-count rows (one per species per quadrat) from 14,419 transects at 720 sites in TAS, WA, VIC, SA and NSW, 1992-03 to 2025-03, partitioned by survey year and 20-degree cell.","displayTitle":"National Reef Monitoring Network - Benthic cover data (in situ surveys)","fuseMountBrowsable":true,"id":"32aab664-2cde-583a-be7d-82a7197ee349","license":"http://creativecommons.org/licenses/by/4.0/","objectCountEstimate":34,"probeStatus":"ok","region":"ap-southeast-2","schemaCard":{"layout":{"pattern":"<prefix>/_common_metadata (18 KB, schema + column metadata + global_attributes) and <prefix>/timestamp=<epoch s of 1 Jan>/polygon=<WKB hex of 20x20 deg cell>/ep_m3_isq_public_data.csv-0.parquet. 34 year partitions (1992..2025, no gaps) x 6 polygon cells in use = 95 data files, one file per directory, 11,525 B min / 47,418 B median / 219,688 B max. No _metadata file."},"sourceObjectKey":"diver_benthic_cover_in_situ_qc.parquet/timestamp=1009843200/polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0/ep_m3_isq_public_data.csv-0.parquet","starterCell":"import time, pandas as pd, pyarrow.dataset as ds, s3fs\n# NRMN in-situ benthic cover (ATRC method 3): point counts per species per 50-point quadrat, 5 quadrats per transect\nfs = s3fs.S3FileSystem(anon=True, client_kwargs={'region_name': 'ap-southeast-2'})\nroot = 'aodn-cloud-optimised/diver_benthic_cover_in_situ_qc.parquet'   # in the pod: '/mnt/archive' with no filesystem=\nSA_CELL = ('010300000001000000050000000000000000005E4000000000000044C0000000000080614000000000000044C0'\n           '000000000080614000000000000034C00000000000005E4000000000000034C00000000000005E4000000000000044C0')\nt0 = time.time()\ntbl = ds.dataset(root, filesystem=fs, format='parquet', partitioning='hive').to_table(\n    filter=(ds.field('polygon') == SA_CELL) & (ds.field('timestamp') >= 1672531200))   # 2023-01-01 onward\nsa = tbl.to_pandas().query(\"area == 'South Australia'\").copy()\nprint(f\"SA rows since 2023: {len(sa)} in {time.time()-t0:.1f}s; latest survey {sa['TIME'].max()}; rows since 2025-03-01: {int((sa['TIME'] >= '2025-03-01').sum())}\")\n# `total` is points scored, layers are counted separately, so normalise by the survey's own point total, not by 50\nsa['share_pct'] = sa['total'] / sa.groupby('survey_id')['total'].transform('sum') * 100\ntop = sa.groupby(['site_name', 'species_name'])['share_pct'].sum().groupby(level=0, group_keys=False).nlargest(3)\nprint(top.round(1).to_string())","sampleRows":[{"timestamp":1072915200,"site_code":"GSV168","reporting_name":"Bryozoa spp.","latitude":-35.37,"habitat_groups":null,"phylum":"Bryozoa","program":"ATRC","quadrat":"Q1","total":2,"visibility":"8.0","longitude":136.87,"survey_id":812319571,"area":"South Australia","location":"Yorke Peninsula","TIME":"2004-02-08 00:00:00","report_group":"Bryozoa","ecoregion":"South Australian Gulfs","site_name":"Boulders (East of Bay)","survey_date":"2004-02-08","depth":5.1,"species_name":"Unidentified bryozoans"},{"habitat_groups":"Canopy","survey_id":923400111,"hour":"14:30:00","survey_date":"2020-03-18","program":"FRDC","depth":5.2,"timestamp":1577836800,"site_name":"South East Betsy","site_code":"SBIR26","visibility":"5.0","location":"Tasmania - South East","phylum":"Heterokontophyta","class":"Phaeophyceae","TIME":"2020-03-18 14:30:00","family":"Lessoniaceae","latitude":-43.06,"area":"Tasmania","ecoregion":"Bassian","quadrat":"Q2","total":5,"longitude":147.49,"order":"Laminariales","species_name":"Ecklonia radiata","reporting_name":"Ecklonia radiata"},{"TIME":"2007-06-20 00:00:00","timestamp":1167609600,"site_name":"Deal Island (Garden Cove, mid East Shore)","depth":5.4,"survey_date":"2007-06-20","phylum":"Rhodophyta","quadrat":"Q4","ecoregion":"Cape Howe","total":8,"survey_id":812318434,"longitude":147.33,"class":"Florideophyceae","family":"Pterocladiaceae","program":"ATRC","order":"Gelidiales","location":"Kent Group","site_code":"KG-S14","species_name":"Pterocladiella capillacea","latitude":-39.45,"area":"Tasmania","habitat_groups":"Understorey"}],"overview":"IMOS National Reef Monitoring Network (NRMN) in-situ benthic cover: the ATRC 'method 3' quadrat point counts (Edgar & Barrett methods, 5 x 50-point quadrats per 50 m transect, canopy / understorey / substrate scored as separate layers), collated and QC'd by the IMOS NRMN Sub-Facility and exported from the NRMN endpoint ep_m3_isq_public_data.csv into parquet. Measured 2026-10-04: 489,046 rows x 32 columns, 14,419 surveys (survey_id), 720 sites, 1992-03-10 to 2025-03-14, five states, programs ATRC / Parks Vic / FRDC (Reef Life Survey divers do not run this method, so there are no RLS rows). On disk it is 96 objects, 4.81 MB: _common_metadata plus 95 parquet files in timestamp=<1 Jan epoch>/polygon=<20-degree WKB cell>/ directories of 11 KB to 220 KB each. The whole table reads in 18 s over anonymous S3 into 244 MB of pandas (string columns dominate), so the practical access direction is partition-pruned: one year is one to four files and 2 s; the South Australian subset is 34 files in one polygon cell, 42,104 rows in 17 s. SA data stop at 2023-12-21 (118 gulf sites, 158 SA sites in all), so there is nothing here for the 2025 bloom period itself; the table is the pre-bloom macroalgal baseline.","rowCount":"489,046 rows (exact, full read 2026-10-04)","sampleHeaders":[{"truncatedAt":512,"bytes":"5041523115041580051588054c15501500120000c002f07500c8756b8922060e00c0c970e11e060e00e8deaf744d040e009037858ed4050e00a03733e6cd050e007079ae129d040e00c0713ee051040e0028c445a8f4040e00e0037cdba2040e00a0a2c620f0040e008814b6ce6f060e00e0b3f5d47d060e00e8707cc96a060e00906c02ccf0170e0080575d33f90108f0c58b63c9c4210e001049312779210e00c083453f80210e00e872ec4d86210e00706ba01edb300e004818713a88300e0030d59b7f82300e00a899da47d5300e00603d9950d0300e0040592a5070310e005044e1666b310e0088b56b4775310e0080e2228e35300e0070e274363c300e00804f61c741300e0030fce990be310e00580088e0c2310e00d84b1e3ec7310e0018f55928f3320e0078d3559df8320e0078ef5743cc310e00d057886f09320e00983e45870d320e00f04658d212320e0008a1cfcc00330e150015f20115f8012c15fc3d1510150615061c18080008a1cfcc00330e180800e8deaf744d040e160028080008a1cfcc00330e180800e8deaf744d040e111100000079f07803000000fc3d01066200b00101a00102960203e20104a60205e60106900207d60108ca0109a0020ade020bb4020cf4020df0010eaa020fcc0110b20111e00112fe0113980114bc01156616501764188a0119a2011a7a1b6e1cb4011dc8011ebe011fbc","encoding":"hex","key":"diver_benthic_cover_in_situ_qc.parquet/timestamp=1009843200/polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0/ep_m3_isq_public_data.csv-0.parquet"}],"generatedAt":"2026-10-04T00:00:00Z","verifiedRead":{"rssDeltaMb":2.5,"attempts":[{"family":"parquet","path":"timestamp=978307200/polygon=01030000000100000005000000000000000080614000000000000044C0000000000000644000000000000044C0000000000000644000000000000034C0000000000080614000000000000034C0000000000080614000000000000044C0/ep_m3_isq_public_data.csv-0.parquet","seconds":1.5,"ok":true}],"facts":["under 'diver_benthic_cover_in_situ_qc.parquet/': 34 directories (e.g. timestamp=1009843200, timestamp=1041379200, timestamp=1072915200, timestamp=1104537600); 1 objects named like _common_metadata, 18 KB to 18 KB","under 'diver_benthic_cover_in_situ_qc.parquet/timestamp=978307200/': 3 directories (e.g. polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0, polygon=01030000000100000005000000000000000040604000000000000049C00000000000C0624000000000000049C00000000000C062400000000000003EC000000000004060400000000000003EC0000000000040604000000000000049C0, polygon=01030000000100000005000000000000000080614000000000000044C0000000000000644000000000000044C0000000000000644000000000000034C0000000000080614000000000000034C0000000000080614000000000000044C0)","under 'diver_benthic_cover_in_situ_qc.parquet/timestamp=978307200/polygon=01030000000100000005000000000000000080614000000000000044C0000000000000644000000000000044C0000000000000644000000000000034C0000000000080614000000000000034C0000000000080614000000000000044C0/': 1 objects named like ep_m#_isq_public_data.csv-#.parquet, 18 KB to 18 KB"],"image":"531775725781.dkr.ecr.us-east-1.amazonaws.com/scigantic-notebook-genomics:latest","summary":"Parquet 1,000 rows, 30 columns, 1 row groups; columns ['TIME:timestamp[ns]', 'area:string', 'geom:string', 'hour:string', 'class:string', 'depth:float', 'order:string', 'realm:string', 'total:int32', 'family:string']","snippet":"import os\nimport pyarrow.parquet as pq\npf = pq.ParquetFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], 'timestamp=978307200/polygon=01030000000100000005000000000000000080614000000000000044C0000000000000644000000000000044C0000000000000644000000000000034C0000000000080614000000000000034C0000000000080614000000000000044C0/ep_m3_isq_public_data.csv-0.parquet'))\nprint(pf.metadata)\ndf = pf.read_row_group(0).to_pandas()\ndf.head()","verifiedAt":"2026-10-04T21:07:18.617Z","seconds":1.5,"path":"timestamp=978307200/polygon=01030000000100000005000000000000000080614000000000000044C0000000000000644000000000000044C0000000000000644000000000000034C0000000000080614000000000000034C0000000000080614000000000000044C0/ep_m3_isq_public_data.csv-0.parquet","family":"parquet","starterCell":{"ok":true,"seconds":2.7},"library":"pyarrow","bytes":18036},"format":"v2-hybrid","starterSamplePath":"timestamp=1009843200/polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0/ep_m3_isq_public_data.csv-0.parquet","totalObjects":"96 objects, 4,813,982 bytes (aws s3 ls --summarize 2026-10-04); 95 data files + _common_metadata","columns":[{"name":"TIME","type":"timestamp[ns]; 'Derived timestamp' = survey_date plus hour when hour is known (21.6% of rows have a non-midnight time); range 1992-03-10 to 2025-03-14; no nulls"},{"type":"string; state/territory: Tasmania 293,497 | Western Australia 61,518 | Victoria 54,016 | South Australia 42,104 | New South Wales 37,911 rows","name":"area"},{"type":"string; EWKB hex of the site point with SRID 4326 (prefix 0101000020E6100000); duplicates latitude/longitude","name":"geom"},{"name":"hour","type":"string HH:mm:ss local time; null in 78.2% of rows"},{"name":"class","type":"string; taxonomic class of the scored element (Phaeophyceae, Florideophyceae ...); null 18%"},{"name":"depth","type":"float32 m; transect depth 1.1 to 15.0 m, median 5.3"},{"name":"order","type":"string; taxonomic order; null 19.6%"},{"type":"string; MEOW realm, single value 'Temperate Australasia'","name":"realm"},{"name":"total","type":"int32; number of quadrat points (of 50) that fell on this species in this quadrat; 0 to 322, median 6. Sum over species per (survey_id, quadrat) is NOT 50: median 75, modes 50 and 100, max 391, because canopy, understorey and substrate layers are scored in sequence"},{"name":"family","type":"string; taxonomic family; null 26.8%"},{"type":"string; Rhodophyta, Heterokontophyta, Porifera, Bryozoa ...; 28 nulls","name":"phylum"},{"name":"country","type":"string; single value 'Australia'"},{"type":"string (hive partition key); WKB hex of the 20-degree cell; 6 cells in use; every South Australian row is in the 120E-140E / 40S-20S cell","name":"polygon"},{"name":"program","type":"string; ATRC 437,775 | Parks Vic 43,145 | FRDC 8,126 rows (no RLS rows in this table)"},{"name":"quadrat","type":"string; Q1..Q5 (5 x 50-point quadrats per 50 m transect) plus 28 rows 'No specimen found'"},{"type":"string; single value 'ep_m3_isq_public_data.csv' (the NRMN endpoint export this store was built from)","name":"filename"},{"type":"float32 degrees_north; SITE coordinate rounded to 2 dp, -43.64 to -30.12","name":"latitude"},{"type":"string; 36 regions, e.g. 'Encounter', 'Yorke Peninsula', 'Eyre Peninsula', 'SA - Western', 'Upper Spencer Gulf', 'Kangaroo Island', 'Upper South East (Robe)'","name":"location"},{"type":"string; MEOW ecoregion: Bassian, Cape Howe, Houtman, South Australian Gulfs (30,344 rows), Leeuwin, Western Bassian, Great Australian Bight","name":"ecoregion"},{"type":"float32 degrees_east; site coordinate, 114.94 to 150.84","name":"longitude"},{"name":"site_code","type":"string; 720 distinct; joins diver_site_information_qc.site_code (100% present) and diver_survey_metadata_qc"},{"type":"string; 716 distinct","name":"site_name"},{"name":"survey_id","type":"int32; 14,419 distinct transects; 100% present in diver_survey_metadata_qc.survey_id (unique there); 1 to 112 rows per survey, median 33"},{"type":"int64 (hive partition key); epoch seconds of 1 Jan of the survey year","name":"timestamp"},{"type":"string (not float) m; '9.0', '12.0' ...; null 17%","name":"visibility"},{"name":"survey_date","type":"string YYYY-MM-DD; equals TIME.date() for every row"},{"type":"string; aggregated reporting group ('Sargassum spp.', 'Foliose red algae', 'Sonderopelta/Peyssonnelia'); 88 distinct, null 67%","name":"report_group"},{"name":"species_name","type":"string; scored taxon or substrate: Ecklonia radiata 34,653 | Unidentified algae (crustose coralline) 33,514 | Phyllospora comosa 20,234 | Peyssonnelia spp. (encrusting) 18,558 | Sand 11,430 ..."},{"type":"string; Understorey 209,315 | Canopy 113,781 | Encrusting 83,579 | Substrate 23,382 | Epiphyte 7,035 | Sessile invertebrate 2,888; null 49,066","name":"habitat_groups"},{"type":"string; display name for species_name; 115 nulls","name":"reporting_name"},{"type":"string; survey-level coordinate, null in 97.5% of rows (476,757)","name":"survey_latitude"},{"name":"survey_longitude","type":"string; as survey_latitude"}],"caveats":["Scale: no South Australian rows exist after 2023-12-21 (and nothing anywhere after 2025-03-14, even though the files were rewritten 2026-10-03). The SA-gulfs-since-2025-03 subset is EMPTY: 0 rows. SA coverage is 42,104 rows, 158 sites, 1,009 surveys, 2004-02-05 to 2023-12-21, with dense years 2004-2007, 2009, 2012-2014 and a single 8-site campaign in Dec 2023 (Encounter, Eyre, Kangaroo Island, SA - Western). Use it as the pre-bloom macroalgal baseline, not as bloom monitoring. diver_survey_metadata_qc shows 353 SA surveys since 2024 (latest 2026-05-29), all RLS program, none with method-3 cover rows here.","DO NOT: divide `total` by 50 to get percent cover. `total` is the number of the quadrat's 50 points that hit the species, but layers (canopy, understorey, encrusting, substrate) are scored separately, so per-quadrat totals sum to 0-391 (median 75; only 4.6% of survey x quadrat combinations sum to exactly 50). Normalise by the survey's or quadrat's own point total, or restrict to one habitat_groups layer first.","Quirk: this table holds ATRC / Parks Vic / FRDC surveys only (program column has no 'RLS'); Reef Life Survey divers score benthos from photoquadrats, which live in diver_photoquadrat_score_qc. SA sites surveyed by both programs appear in both tables under the same site_code.","Quirk: `area` is the authoritative state label and is wider than the gulfs bbox: 7,996 SA rows (SA - Western, Upper South East (Robe), part of Eyre Peninsula) fall outside lon 135-139.5 / lat -36.5..-32; the bbox itself holds 34,108 SA rows and nothing from any other state. All 42,104 SA rows sit in ONE polygon partition (120E-140E / 40S-20S), so filter on that cell, then on `area`.","Quirk: latitude/longitude are the SITE coordinates rounded to 2 decimal places (float32, ~1 km); survey_latitude/survey_longitude (string) are populated for only 2.5% of rows. `geom` is the same site point as EWKB hex with SRID 4326.","Quirk: `hour` is null for 78% of rows and TIME is then midnight local-date-as-UTC-naive; `visibility`, `survey_latitude`, `survey_longitude` are strings, not floats. report_group and habitat_groups are null for 67% and 10% of rows respectively, so group on species_name or phylum/class when you need full coverage.","Scale: full table = 95 files, 4.8 MB on S3 but 244 MB in pandas (string columns); 18 s anonymous. One year directory = 1-4 files, 2 s. Reading via the FUSE mount is file-by-file as well; 95 files is fine, but do not os.walk per-row.","Quirk: hive keys are strings in the path. timestamp=<epoch seconds> is the 1 January 00:00 UTC of the row's survey year (34/34 benthic, 21/21 score, 19/19 DwC partitions decoded to Jan-1; TIME.year == partition year for every row), so it is a YEAR partition, not a month. polygon=<WKB hex> is a little-endian WKB Polygon (0103..., 1 ring, 5 points) of a 20 x 20 degree lon/lat cell; decode with struct (or shapely.wkb.loads(bytes.fromhex(h)) where shapely exists).","Quirk: pyarrow's hive discovery types `timestamp` as int64 and `polygon` as string; pandas.read_parquet returns both as category columns. Filter on them with ds.field('timestamp') >= <epoch> and ds.field('polygon') == '<hex>' to prune files; filters on data columns (area, latitude) read every file."],"mountPath":"/mnt/archive","accessHints":[{"language":"bash","snippet":"# Task: see the layout without a recursive walk: root entries, then one year directory\naws s3 ls --no-sign-request --region ap-southeast-2 s3://aodn-cloud-optimised/diver_benthic_cover_in_situ_qc.parquet/\naws s3 ls --no-sign-request --region ap-southeast-2 s3://aodn-cloud-optimised/diver_benthic_cover_in_situ_qc.parquet/timestamp=1704067200/ --recursive\n# in the pod the same tree is under /mnt/archive/  (ls /mnt/archive | head)"},{"language":"python-anonymous","snippet":"# Task: read the schema and dataset identity without touching a data file (_common_metadata at the root)\nimport pyarrow.parquet as pq, s3fs, ast\nfs = s3fs.S3FileSystem(anon=True, client_kwargs={'region_name': 'ap-southeast-2'})\nfor store in ['diver_benthic_cover_in_situ_qc', 'diver_photoquadrat_score_qc', 'diver_photoquadrat_qc_DwC', 'diver_site_information_qc']:\n    schema = pq.read_schema(f'aodn-cloud-optimised/{store}.parquet/_common_metadata', filesystem=fs)\n    meta = {k.decode(): v.decode() for k, v in schema.metadata.items()}\n    print(store, '|', len(schema), 'columns |', ast.literal_eval(meta['global_attributes']))\n    print('   ', ast.literal_eval(meta.get('latitude') or meta.get('decimalLatitude')))"},{"language":"python-local","snippet":"# Task: schema from the mount, then one year filtered by partition key (pandas, no S3 client needed)\nimport pyarrow.parquet as pq, pandas as pd\nschema = pq.read_schema('/mnt/archive/_common_metadata')\nprint(schema.metadata[b'global_attributes'].decode())\n# hive partitions are discovered automatically; timestamp is the 1 Jan epoch of the survey year\ndf = pd.read_parquet('/mnt/archive', filters=[('timestamp', '>=', 1072915200)],    # 2004 onward\n                     columns=['TIME', 'area', 'location', 'site_code', 'survey_id', 'species_name', 'total'])\nsa = df[df['area'] == 'South Australia']\nprint(len(df), 'rows; SA', len(sa), 'latest SA', sa['TIME'].max())"},{"snippet":"# Task: pull every South Australian in-situ benthic cover record (anonymous S3) and report the latest survey dates\nimport time, pandas as pd, pyarrow.dataset as ds, s3fs\nfs = s3fs.S3FileSystem(anon=True, client_kwargs={'region_name': 'ap-southeast-2'})\nroot = 'aodn-cloud-optimised/diver_benthic_cover_in_situ_qc.parquet'\n# hive value of the 120E-140E / 40S-20S cell (WKB polygon hex); every SA row sits in it (measured 2026-10-04)\nSA_CELL = ('010300000001000000050000000000000000005E4000000000000044C0000000000080614000000000000044C0'\n           '000000000080614000000000000034C00000000000005E4000000000000034C00000000000005E4000000000000044C0')\nt0 = time.time()\ndset = ds.dataset(root, filesystem=fs, format='parquet', partitioning='hive')\ntbl = dset.to_table(filter=(ds.field('polygon') == SA_CELL) & (ds.field('timestamp') >= 1072915200),  # 2004-01-01 onward\n                    columns=['TIME', 'area', 'location', 'site_code', 'survey_id', 'quadrat', 'species_name', 'total'])\ndf = tbl.to_pandas()\nsa = df[df['area'] == 'South Australia']\nprint(f\"cell rows {len(df)} in {time.time()-t0:.1f}s; SA rows {len(sa)}, sites {sa['site_code'].nunique()}, \"\n      f\"surveys {sa['survey_id'].nunique()}, latest {sa['TIME'].max()}, since 2025-03-01: {int((sa['TIME'] >= '2025-03-01').sum())}\")\nprint(sa.groupby('location')['TIME'].agg(['min', 'max', 'count']).to_string())","language":"python-anonymous"},{"language":"python-anonymous","snippet":"# Task: join a survey's cover rows to its survey metadata (rugosity, direction, pq flags) via survey_id, and the site register via site_code\nimport pandas as pd, pyarrow.dataset as ds, s3fs\nfs = s3fs.S3FileSystem(anon=True, client_kwargs={'region_name': 'ap-southeast-2'})\nSA_CELL = ('010300000001000000050000000000000000005E4000000000000044C0000000000080614000000000000044'\n           'C0000000000080614000000000000034C00000000000005E4000000000000034C00000000000005E4000000000000044C0')\nflt = (ds.field('polygon') == SA_CELL) & (ds.field('timestamp') >= 1672531200)   # same partition scheme in both tables\ncover = ds.dataset('aodn-cloud-optimised/diver_benthic_cover_in_situ_qc.parquet', filesystem=fs, partitioning='hive').to_table(filter=flt).to_pandas()\nmeta = ds.dataset('aodn-cloud-optimised/diver_survey_metadata_qc.parquet', filesystem=fs, partitioning='hive').to_table(\n    filter=flt, columns=['survey_id', 'direction', 'avg_rugosity', 'max_rugosity', 'has_pq_scores_in_db']).to_pandas()\nsites = pd.read_parquet('s3://aodn-cloud-optimised/diver_site_information_qc.parquet', columns=['site_code', 'programs', 'old_site_codes'],\n                        storage_options={'anon': True, 'client_kwargs': {'region_name': 'ap-southeast-2'}})\nj = cover.merge(meta, on='survey_id', how='left').merge(sites, on='site_code', how='left')\nprint(len(cover), 'cover rows ->', len(j), 'joined;', j['avg_rugosity'].notna().mean().round(3), 'have rugosity (0.0 for the SA 2023 campaign)')"}],"sampleObjects":["_common_metadata","timestamp=1072915200/polygon=010300000001000000050000000000000000005E4000000000000044C0000000000080614000000000000044C0000000000080614000000000000034C00000000000005E4000000000000034C00000000000005E4000000000000044C0/ep_m3_isq_public_data.csv-0.parquet","timestamp=1704067200/polygon=01030000000100000005000000000000000040604000000000000049C00000000000C0624000000000000049C00000000000C062400000000000003EC000000000004060400000000000003EC0000000000040604000000000000049C0/ep_m3_isq_public_data.csv-0.parquet","timestamp=1735689600/polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0/ep_m3_isq_public_data.csv-0.parquet"]},"summary":"The National Reef Monitoring Network (NRMN) brings together shallow reef surveys conducted around Australia into a centralised database.","title":"National Reef Monitoring Network - Benthic cover data (in situ surveys)","updatedAt":"2026-10-07T13:49:44.525Z","visibility":"public","websiteUrl":"https://catalogue-imos.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/ec424e4f-0f55-41a5-a3f2-726bc4541947"}}