{"success":true,"data":{"author":"AODN","bucketAccessType":"public","bucketListable":true,"bucketName":"aodn-cloud-optimised","bucketPrefix":"diver_site_information_qc.parquet","category":"Life Sciences","cloudProvider":"aws","createdAt":"2026-10-04T00:00:00.000Z","description":"The National Reef Monitoring Network (NRMN) brings together shallow reef surveys conducted around Australia into a centralised database. The IMOS NRMN Sub-Facility collates, cleans, stores and makes this data rapidly available from contributors including Reef Life Survey, Parks Australia, DBCA (Western Australia), DEW (South Australia), DPI (New South Wales), Tasmanian Parks and Wildlife Service and Parks Victoria. Reef Life Survey (RLS) and the Australian Temperate Reef Collaboration (ATRC) coordinate surveys of rocky and coral reefs using scientific methods carried out by marine scientists and trained recreational divers. This dataset shows the locations of RLS and ATRC sites on shallow rocky and coral reefs worldwide. Multiple transects (surveys) are usually surveyed within sites on any given date; sites traditionally need a minimum of 200 m separation to be classified as a different site, unless separated by a significant geological feature or reef habitat discontinuity. This store is the site register as parquet: 4,892 sites (one row per unique site_code) in 59 countries with MEOW realm/province/ecoregion, programs and superseded codes, partitioned by 20-degree cell only; it is the lookup every other NRMN table joins to on site_code.","displayTitle":"National Reef Monitoring Network - Site information","fuseMountBrowsable":true,"id":"f0a097bb-4432-50b2-bbf3-0e7e26cfbcf0","license":"http://creativecommons.org/licenses/by/4.0/","objectCountEstimate":93,"probeStatus":"ok","region":"ap-southeast-2","schemaCard":{"rowCount":"4,892 rows (exact, full read 2026-10-04); site_code unique","overview":"The Reef Life Survey / ATRC site register behind the IMOS National Reef Monitoring Network: one row per site_code with its name, 2-dp coordinates, administrative area and 'location' region, Marine Ecoregions of the World realm / province / ecoregion, latitudinal zone, the programs that survey it and any superseded site codes (exported from the NRMN endpoint ep_site_list_public_data.csv). Measured 2026-10-04: 4,892 rows x 16 columns, site_code unique, 59 countries, 153 areas, 221 locations; 100% of diver_benthic_cover_in_situ_qc site codes and 99.1% of diver_photoquadrat_score_qc site codes resolve here. It is the only NRMN store partitioned by polygon alone (no timestamp): 94 objects, 0.63 MB, _common_metadata plus 93 files of 4-25 KB, one per 20-degree cell. The whole register is 1.6 MB in pandas, but reading it means 93 small GETs: 12-50 s anonymous across our runs, 1 s for a single cell file. Read it once per session and keep it. South Australia has 296 sites (227 inside the gulfs bbox lat -36.5..-32 / lon 135..139.5): Encounter (Fleurieu Peninsula) 94, Yorke Peninsula 57, SA - Western 37, Eyre Peninsula 34, Kangaroo Island 28, Upper South East (Robe) 18, Adelaide 15, Upper Spencer Gulf 13.","sampleRows":[{"area":"South Australia","old_site_codes":"ENC13","longitude":138.47,"ecoregion":"South Australian Gulfs","site_name":"Port Stanvac Jetty deep","programs":"RLS","geom":"0101000020E6100000D7A3703D0A4F6140AE47E17A148E41C0","latitude":-35.11,"lat_zone":"Temperate","site_code":"GSV101","location":"Adelaide","realm":"Temperate Australasia","country":"Australia","province":"Southwest Australian Shelf"},{"site_name":"Big_W","country":"Australia","site_code":"GSV64","latitude":-35.68,"longitude":136.91,"province":"Southwest Australian Shelf","realm":"Temperate Australasia","lat_zone":"Temperate","programs":"RLS","old_site_codes":"13KINRM_18, KI13","location":"Kangaroo Island","ecoregion":"South Australian Gulfs","area":"South Australia"},{"old_site_codes":null,"country":"Australia","area":"South Australia","site_code":"EYR24","realm":"Temperate Australasia","programs":"ATRC","ecoregion":"Great Australian Bight","site_name":"Pearson Island Bay NW","location":"Eyre Peninsula","lat_zone":"Temperate"}],"starterCell":"import time, pandas as pd\n# NRMN site register (RLS + ATRC), one row per site_code; the key every other NRMN table joins on\nt0 = time.time()\nsites = pd.read_parquet('s3://aodn-cloud-optimised/diver_site_information_qc.parquet',   # in the pod: '/mnt/archive'\n                        storage_options={'anon': True, 'client_kwargs': {'region_name': 'ap-southeast-2'}})\nsa = sites[sites['area'] == 'South Australia']\ngulfs = sa[sa['latitude'].between(-36.5, -32.0) & sa['longitude'].between(135.0, 139.5)]\nprint(f\"{len(sites)} sites in {time.time()-t0:.1f}s; South Australia {len(sa)}, of which {len(gulfs)} inside the gulfs bbox\")\nprint(sa.groupby(['location', 'programs']).size().unstack(fill_value=0).to_string())\nprint(gulfs[gulfs['location'].isin(['Kangaroo Island', 'Yorke Peninsula', 'Adelaide'])]\n      .sort_values(['location', 'site_code'])[['site_code', 'site_name', 'location', 'latitude', 'longitude', 'programs']].head(12).to_string(index=False))","sampleHeaders":[{"truncatedAt":512,"bytes":"504152311504151215164c150215001200000920050000004d616c74611500151215162c15021510150615061c360028054d616c746118054d616c7461111100000009200200000002010102001504156c15584c15021500120000363c3200000030313031303030303230453605093201004c32443430463632383543384643324635343134301500151215162c15021510150615061c36002832303130313030303032304536313030303030303030303030303030303030324434304636323835433846433246353431343018323031303130303030323045363130303030303030303030303030303030303244343046363238354338464332463534313430111100000009200200000002010102001504153e15424c150215001200001f781b00000054656d706572617465204e6f72746865726e2041746c616e7469631500151215162c15021510150615061c3600281b54656d706572617465204e6f72746865726e2041746c616e746963181b54656d706572617465204e6f72746865726e2041746c616e746963111100000009200200000002010102001504152a152e4c1502150012000015501100000052657075626c6963206f66204d616c74611500151215162c15021510150615061c3600281152657075626c6963206f66204d616c7461181152657075626c6963206f66204d616c746111110000000920","encoding":"hex","key":"diver_site_information_qc.parquet/polygon=010300000001000000050000000000000000000000000000000000344000000000000034400000000000003440000000000000344000000000000044400000000000000000000000000000444000000000000000000000000000003440/ep_site_list_public_data.csv-0.parquet"}],"sampleObjects":["_common_metadata","polygon=010300000001000000050000000000000000005E4000000000000044C0000000000080614000000000000044C0000000000080614000000000000034C00000000000005E4000000000000034C00000000000005E4000000000000044C0/ep_site_list_public_data.csv-0.parquet","polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0/ep_site_list_public_data.csv-0.parquet","polygon=010300000001000000050000000000000000000000000000000000344000000000000034400000000000003440000000000000344000000000000044400000000000000000000000000000444000000000000000000000000000003440/ep_site_list_public_data.csv-0.parquet"],"format":"v2-hybrid","columns":[{"type":"string; state/territory or overseas region, 153 distinct; 'South Australia' 296 sites","name":"area"},{"name":"geom","type":"string; EWKB hex point with SRID 4326 (prefix 0101000020E6100000); same point as latitude/longitude"},{"type":"string; MEOW realm, 12 (Temperate Australasia 1,850 | Central Indo-Pacific 1,414 | Temperate Northern Atlantic 503 | ...)","name":"realm"},{"name":"country","type":"string; 59 countries; 2 nulls"},{"name":"polygon","type":"string (hive key); WKB hex of a 20 x 20 degree cell; 93 cells; SA sites are in the 120E-140E / 40S-20S cell except OTW18 Salmon Hole (lon 140.01) which is in 130E-150E"},{"type":"string; single value 'ep_site_list_public_data.csv'","name":"filename"},{"type":"string; Temperate 2,561 | Tropical 2,290 | Polar 41","name":"lat_zone"},{"type":"float32 degrees_north; 2 dp","name":"latitude"},{"name":"location","type":"string; 221 regions; SA: Encounter 94 | Yorke Peninsula 57 | SA - Western 37 | Eyre Peninsula 34 | Kangaroo Island 28 | Upper South East (Robe) 18 | Adelaide 15 | Upper Spencer Gulf 13"},{"name":"programs","type":"string; comma-joined program list: 'RLS' 4,095 | 'ATRC' 463 | 'ATRC, RLS' 221 | 'ATRC, Parks Vic, RLS' 34 | 'FRDC' 20 | 'RLS, RRH' 15 | 'ATRC, Parks Vic' 11 | 'Parks Vic' 11"},{"name":"province","type":"string; MEOW province, 47 ('Southwest Australian Shelf' for the SA gulfs)"},{"name":"ecoregion","type":"string; MEOW ecoregion, 105 ('South Australian Gulfs', 'Great Australian Bight')"},{"name":"longitude","type":"float32 degrees_east; 2 dp"},{"name":"site_code","type":"string; UNIQUE (4,892 distinct = rows); the join key for every NRMN table; SA prefixes GSV 161, EYR 55, SSG 29, MUR 17, COO 14, NSG 13, OTW 7"},{"name":"site_name","type":"string"},{"type":"string; comma-joined superseded codes ('13KINRM_18, KI13', 'ENC13'); null for 3,165 sites","name":"old_site_codes"}],"accessHints":[{"snippet":"# Task: see the polygon-only layout (93 cells, no year level) and the SA cell's single file\naws s3 ls --no-sign-request --region ap-southeast-2 s3://aodn-cloud-optimised/diver_site_information_qc.parquet/ | head\naws s3 ls --no-sign-request --region ap-southeast-2 \"s3://aodn-cloud-optimised/diver_site_information_qc.parquet/polygon=010300000001000000050000000000000000005E4000000000000044C0000000000080614000000000000044C0000000000080614000000000000034C00000000000005E4000000000000034C00000000000005E4000000000000044C0/\"","language":"bash"},{"language":"python-anonymous","snippet":"# Task: read the schema and dataset identity without touching a data file (_common_metadata at the root)\nimport pyarrow.parquet as pq, s3fs, ast\nfs = s3fs.S3FileSystem(anon=True, client_kwargs={'region_name': 'ap-southeast-2'})\nfor store in ['diver_benthic_cover_in_situ_qc', 'diver_photoquadrat_score_qc', 'diver_photoquadrat_qc_DwC', 'diver_site_information_qc']:\n    schema = pq.read_schema(f'aodn-cloud-optimised/{store}.parquet/_common_metadata', filesystem=fs)\n    meta = {k.decode(): v.decode() for k, v in schema.metadata.items()}\n    print(store, '|', len(schema), 'columns |', ast.literal_eval(meta['global_attributes']))\n    print('   ', ast.literal_eval(meta.get('latitude') or meta.get('decimalLatitude')))"},{"snippet":"# Task: load the whole register from the mount (0.6 MB, 93 files) once, then look up South Australian sites\nimport pandas as pd\nsites = pd.read_parquet('/mnt/archive')\nsa = sites[sites['area'] == 'South Australia']\nprint(len(sites), 'sites;', len(sa), 'in South Australia;', sa['location'].value_counts().to_dict())\nki = sa[sa['location'] == 'Kangaroo Island'][['site_code', 'site_name', 'latitude', 'longitude', 'programs', 'old_site_codes']]\nprint(ki.sort_values('site_code').to_string(index=False))","language":"python-local"},{"snippet":"# Task: list the South Australian NRMN sites and the date each was last surveyed (site table + survey metadata, anonymous S3)\nimport time, pandas as pd, pyarrow.dataset as ds, s3fs\nfs = s3fs.S3FileSystem(anon=True, client_kwargs={'region_name': 'ap-southeast-2'})\nt0 = time.time()\nsites = pd.read_parquet('s3://aodn-cloud-optimised/diver_site_information_qc.parquet',\n                        storage_options={'anon': True, 'client_kwargs': {'region_name': 'ap-southeast-2'}})\nsa = sites[sites['area'] == 'South Australia']\nprint(f\"{len(sites)} sites worldwide in {time.time()-t0:.1f}s; SA: {len(sa)} sites, \"\n      f\"{int((sa.latitude.between(-36.5,-32) & sa.longitude.between(135,139.5)).sum())} inside the gulfs bbox\")\nprint(sa.groupby('location').agg(sites=('site_code', 'size'), programs=('programs', lambda x: ' | '.join(sorted(set(x))))).to_string())\n# survey_metadata is partitioned the same way; take the 120-140E cell from 2024 on\nSA_CELL = ('010300000001000000050000000000000000005E4000000000000044C0000000000080614000000000000044C0'\n           '000000000080614000000000000034C00000000000005E4000000000000034C00000000000005E4000000000000044C0')\nt0 = time.time()\nsm = ds.dataset('aodn-cloud-optimised/diver_survey_metadata_qc.parquet', filesystem=fs, format='parquet', partitioning='hive')\nsurv = sm.to_table(filter=(ds.field('polygon') == SA_CELL) & (ds.field('timestamp') >= 1704067200),\n                   columns=['survey_id', 'site_code', 'survey_date', 'program', 'has_pq_scores_in_db']).to_pandas()\nrecent = surv.merge(sa[['site_code', 'site_name', 'location']], on='site_code')\nprint(f\"SA surveys since 2024-01-01: {len(recent)} ({time.time()-t0:.1f}s); latest {recent['survey_date'].max()}; \"\n      f\"with photoquadrat scores in db: {int(recent['has_pq_scores_in_db'].sum())}\")\nprint(recent.sort_values('survey_date').tail(5)[['survey_date', 'site_code', 'site_name', 'location', 'program']].to_string(index=False))","language":"python-anonymous"},{"language":"python-anonymous","snippet":"# Task: read only the 20-degree cell that holds the SA gulfs (one 24 KB file) instead of all 93\nimport time, pandas as pd, pyarrow.dataset as ds, s3fs\nfs = s3fs.S3FileSystem(anon=True, client_kwargs={'region_name': 'ap-southeast-2'})\nSA_CELL = ('010300000001000000050000000000000000005E4000000000000044C0000000000080614000000000000044'\n           'C0000000000080614000000000000034C00000000000005E4000000000000034C00000000000005E4000000000000044C0')\nt0 = time.time()\ncell = ds.dataset('aodn-cloud-optimised/diver_site_information_qc.parquet', filesystem=fs, format='parquet', partitioning='hive')\\\n         .to_table(filter=ds.field('polygon') == SA_CELL).to_pandas()\nprint(len(cell), 'sites in the 120-140E / 20-40S cell in', round(time.time() - t0, 1), 's;', cell['area'].value_counts().to_dict())"}],"mountPath":"/mnt/archive","layout":{"pattern":"<prefix>/_common_metadata (9.4 KB) and <prefix>/polygon=<WKB hex 20x20 deg cell>/ep_site_list_public_data.csv-0.parquet. 93 polygon partitions, no timestamp level, one file each: 4,382 B min / 5,255 B median / 24,500 B max. No _metadata."},"totalObjects":"94 objects, 630,918 bytes (aws s3 ls --summarize 2026-10-04); 93 data files + _common_metadata","caveats":["Scale: static register, no time axis, so there is no 'since 2025-03' subset; the SA relevance is the key list. South Australia = 296 sites (area column), 227 inside the gulfs bbox; the 69 outside are SA - Western (Ceduna/Head of Bight), Upper South East (Robe) and western Eyre Peninsula. Fleurieu Peninsula sites are filed under location 'Encounter' (94 sites: Carrickalinga, Rapid Head, Myponga, Aldinga, Second Valley, Victor Harbor); metro Gulf St Vincent sites under 'Adelaide' (15, all RLS). No location is named 'Fleurieu', 'Gulf St Vincent' or 'Spencer Gulf' ('Upper Spencer Gulf' and 'Yorke Peninsula' cover the latter).","Quirk: site_code is unique here and is THE join key: 100% of benthic-cover and 99.1% of photoquadrat-score site codes resolve; diver_survey_metadata_qc also carries site_code and old_site_codes. old_site_codes is a comma-joined string of superseded codes (e.g. 'ENC13', '13KINRM_18, KI13'), null for 65% of sites; split on ', ' before matching legacy data.","Quirk: programs is a comma-joined string ('ATRC, RLS'), not a list; use str.contains('RLS') rather than equality. SA: 125 RLS-only, 103 ATRC-only, 68 both. A site being listed does not mean it has data in every table: benthic cover (ATRC method) and photoquadrat scores (RLS) each cover a subset, and SA has had no rows in either since 2023-12 / 2022-02 even though RLS surveys continued to 2026-05-29 (diver_survey_metadata_qc).","Quirk: partitioned by polygon ONLY (93 cells); a site's cell is fixed by its coordinates, so 295 of 296 SA sites are in the 120E-140E / 40S-20S cell and one (OTW18 Salmon Hole, lon 140.01) is in 130E-150E. Reading one cell is a single 24 KB file (1-8 s anonymous); the 120E-140E / 40S-20S cell holds 295 sites and every one of them is area=='South Australia'; the full register is 93 files and latency-bound (12-50 s anonymous in our runs, 1.6 MB in pandas).","Quirk: coordinates are 2 dp (~1 km) float32 and are the same values the other NRMN tables carry as site latitude/longitude; `geom` duplicates them as EWKB hex with SRID 4326. country is null for 2 rows.","Quirk: hive keys are strings in the path. This store has no timestamp level. polygon=<WKB hex> is a little-endian WKB Polygon (0103..., 1 ring, 5 points) of a 20 x 20 degree lon/lat cell; decode with struct (or shapely.wkb.loads(bytes.fromhex(h)) where shapely exists).","Quirk: pyarrow's hive discovery types `polygon` as string; pandas.read_parquet returns it as a category column. Filter on them with ds.field('polygon') == '<hex>' to prune files; filters on data columns (area, latitude) read every file."],"starterSamplePath":"polygon=010300000001000000050000000000000000000000000000000000344000000000000034400000000000003440000000000000344000000000000044400000000000000000000000000000444000000000000000000000000000003440/ep_site_list_public_data.csv-0.parquet","verifiedRead":{"bytes":4519,"summary":"Parquet 1 rows, 15 columns, 1 row groups; columns ['area:string', 'geom:string', 'realm:string', 'country:string', 'filename:string', 'lat_zone:string', 'latitude:float', 'location:string', 'programs:string', 'province:string']","facts":["under 'diver_site_information_qc.parquet/': 93 directories (e.g. polygon=010300000001000000050000000000000000000000000000000000344000000000000034400000000000003440000000000000344000000000000044400000000000000000000000000000444000000000000000000000000000003440, polygon=0103000000010000000500000000000000000000000000000000003E4000000000000034400000000000003E40000000000000344000000000000049400000000000000000000000000000494000000000000000000000000000003E40, polygon=01030000000100000005000000000000000000000000000000000044C0000000000000344000000000000044C0000000000000344000000000000034C0000000000000000000000000000034C0000000000000000000000000000044C0, polygon=010300000001000000050000000000000000000000000000000000494000000000000034400000000000004940000000000000344000000000008051400000000000000000000000000080514000000000000000000000000000004940); 1 objects named like _common_metadata, 9 KB to 9 KB","under 'diver_site_information_qc.parquet/polygon=010300000001000000050000000000000000C067C000000000000034C000000000004065C000000000000034C000000000004065C000000000000000000000000000C067C000000000000000000000000000C067C000000000000034C0/': 1 objects named like ep_site_list_public_data.csv-#.parquet, 4 KB to 4 KB"],"verifiedAt":"2026-10-04T21:10:18.705Z","seconds":0.89,"starterCell":{"seconds":9.29,"ok":true},"attempts":[{"seconds":0.89,"path":"polygon=010300000001000000050000000000000000C067C000000000000034C000000000004065C000000000000034C000000000004065C000000000000000000000000000C067C000000000000000000000000000C067C000000000000034C0/ep_site_list_public_data.csv-0.parquet","ok":true,"family":"parquet"}],"rssDeltaMb":0.1,"family":"parquet","snippet":"import os\nimport pyarrow.parquet as pq\npf = pq.ParquetFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], 'polygon=010300000001000000050000000000000000C067C000000000000034C000000000004065C000000000000034C000000000004065C000000000000000000000000000C067C000000000000000000000000000C067C000000000000034C0/ep_site_list_public_data.csv-0.parquet'))\nprint(pf.metadata)\ndf = pf.read_row_group(0).to_pandas()\ndf.head()","path":"polygon=010300000001000000050000000000000000C067C000000000000034C000000000004065C000000000000034C000000000004065C000000000000000000000000000C067C000000000000000000000000000C067C000000000000034C0/ep_site_list_public_data.csv-0.parquet","library":"pyarrow","image":"531775725781.dkr.ecr.us-east-1.amazonaws.com/scigantic-notebook-genomics:latest"},"sourceObjectKey":"diver_site_information_qc.parquet/polygon=010300000001000000050000000000000000000000000000000000344000000000000034400000000000003440000000000000344000000000000044400000000000000000000000000000444000000000000000000000000000003440/ep_site_list_public_data.csv-0.parquet","generatedAt":"2026-10-04T00:00:00Z"},"summary":"The National Reef Monitoring Network (NRMN) brings together shallow reef surveys conducted around Australia into a centralised database.","title":"National Reef Monitoring Network - Site information","updatedAt":"2026-10-05T09:22:11.173Z","visibility":"public","websiteUrl":"https://catalogue-imos.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/e41efa35-03f0-4dea-be98-2a69e46b510b"}}