{"success":true,"data":{"author":"AODN","bucketAccessType":"public","bucketListable":true,"bucketName":"aodn-cloud-optimised","bucketPrefix":"diver_survey_metadata_qc.parquet","category":"Life Sciences","cloudProvider":"aws","createdAt":"2026-10-04T00:00:00.000Z","description":"The IMOS National Reef Monitoring Network (NRMN) sub-facility brings together shallow reef surveys conducted around Australia into a centralised database, collating, cleaning and publishing data from Reef Life Survey (RLS), the Australian Temperate Reef Collaboration (ATRC), Parks Australia, DBCA (WA), DEW (SA), DPI (NSW), Tasmanian Parks and Wildlife and Parks Victoria. RLS and ATRC coordinate surveys of rocky and coral reefs using standard underwater visual census methods, carried out by marine scientists and trained recreational divers, to establish national marine baselines and support the management of Commonwealth and State marine reserves. This dataset provides the metadata associated with individual 50 m transects surveyed by RLS and ATRC divers and partners, all undertaken using standard methods. ATRC surveys are predominantly collected as a group of 4 x 50 m transects per site; RLS surveys commonly have a minimum of 2 x 50 m transects per site on a date (typically at different depths), but this number varies. It supersedes the RLS-specific 'Survey Locations' collection. This store is the quality-controlled survey list as hive-partitioned parquet: one row per transect survey, 36,974 rows with unique survey_id, 4,892 sites in 59 countries, 1992-03-10 to 2026-07-19, with the methods done, depth, visibility, photoquadrat links and flags; it is the master key for the four NRMN observation tables and the only place the 2026 South Australian surveys currently appear.","displayTitle":"National Reef Monitoring Network - Survey metadata","fuseMountBrowsable":true,"id":"af6e6517-586f-5547-b06e-9f70c9ba0764","license":"http://creativecommons.org/licenses/by/4.0/","objectCountEstimate":35,"probeStatus":"ok","region":"ap-southeast-2","schemaCard":{"accessHints":[{"snippet":"# Task: layout check\naws s3 ls s3://aodn-cloud-optimised/diver_survey_metadata_qc.parquet/ --no-sign-request --region ap-southeast-2 | tail -3\naws s3 ls s3://aodn-cloud-optimised/diver_survey_metadata_qc.parquet/timestamp=1767225600/ --no-sign-request --region ap-southeast-2","language":"bash"},{"language":"python-local","snippet":"# Task: schema and GeoNetwork uuid from _common_metadata\nimport ast, pyarrow.parquet as pq\nschema = pq.read_schema('/mnt/archive/_common_metadata')\nprint(schema.names)\nprint(ast.literal_eval(schema.metadata[b'global_attributes'].decode()))\nprint(ast.literal_eval(schema.metadata[b'methods'].decode()))"},{"language":"python-local","snippet":"# Task: the whole survey list in memory (36,974 rows, ~10 MB without geom/polygon/filename); it is the join key for everything else\nimport pandas as pd\ncols = ['survey_id','site_code','site_name','location','area','country','program','survey_date','hour','depth','visibility','methods','latitude','longitude','pq_zip_url','latest_surveydate_for_site']\nm = pd.read_parquet('/mnt/archive', columns=cols)\nprint(len(m), m['survey_id'].is_unique, m['survey_date'].min(), m['survey_date'].max())\nprint(m['methods'].value_counts().head(5))"},{"language":"python-local","snippet":"# Task: South Australian surveys since the March 2025 bloom, by month and location (pushdown on year + area)\nimport pyarrow.dataset as ds\ndset = ds.dataset('/mnt/archive', format='parquet', partitioning='hive')\nm = dset.to_table(filter=(ds.field('timestamp') >= 1735689600) & (ds.field('area') == 'South Australia') & (ds.field('survey_date') >= '2025-03-01'),\n                  columns=['survey_id','site_code','site_name','location','survey_date','depth','methods','program']).to_pandas()\nprint(len(m), 'surveys', m['site_code'].nunique(), 'sites', m['survey_date'].max())\nprint(m.groupby([m['survey_date'].str[:7], 'location']).size().unstack(fill_value=0))"},{"language":"python-anonymous","snippet":"# Task: same SA subset over anonymous S3 from outside the pod\nimport pyarrow.dataset as ds, pyarrow.fs as pafs\ns3 = pafs.S3FileSystem(anonymous=True, region='ap-southeast-2')\ndset = ds.dataset('aodn-cloud-optimised/diver_survey_metadata_qc.parquet', filesystem=s3, format='parquet', partitioning='hive')\nm = dset.to_table(filter=(ds.field('timestamp') >= 1735689600) & (ds.field('area') == 'South Australia') & (ds.field('survey_date') >= '2025-03-01'),\n                  columns=['survey_id','site_code','site_name','location','survey_date','methods']).to_pandas()\nprint(len(m), m['site_code'].nunique(), m['survey_date'].min(), m['survey_date'].max())\nprint(m.groupby(m['survey_date'].str[:7]).size())"},{"language":"python-anonymous","snippet":"# Task: which of those SA surveys already have published observations, per table (release-lag check; the answer changes as AODN republishes)\nimport pyarrow.dataset as ds, pyarrow.fs as pafs\ns3 = pafs.S3FileSystem(anonymous=True, region='ap-southeast-2')\nB = 'aodn-cloud-optimised/'\nflt = (ds.field('timestamp') >= 1735689600) & (ds.field('area') == 'South Australia') & (ds.field('survey_date') >= '2025-03-01')\nm = ds.dataset(B + 'diver_survey_metadata_qc.parquet', filesystem=s3, format='parquet', partitioning='hive').to_table(filter=flt, columns=['survey_id','survey_date']).to_pandas()\nfor name, store in [('fish M1','diver_reef_fish_abundance_biomass_qc'), ('inverts M2','diver_mobile_macroinvertebrate_abundance_qc'), ('cryptic M2','diver_cryptobenthic_fish_abundance_qc'), ('off-transect M0','diver_off_transect_species_observations_qc')]:\n    ids = ds.dataset(B + store + '.parquet', filesystem=s3, format='parquet', partitioning='hive').to_table(filter=flt, columns=['survey_id']).column('survey_id').unique().to_pylist()\n    m[name] = m['survey_id'].isin(ids)\nprint(m.groupby(m['survey_date'].str[:7])[['fish M1','inverts M2','cryptic M2','off-transect M0']].sum().assign(listed=m.groupby(m['survey_date'].str[:7]).size()))"}],"caveats":["Scale (mount): pyarrow.dataset over /mnt/archive must stat all ~440 files before the first row; measured 244-310 s in a real pod on 2026-10-04 versus 7-20 s through pyarrow's own anonymous S3 client (pafs.S3FileSystem(anonymous=True, region='ap-southeast-2'), no s3fs needed). Prefer the S3 form for whole-dataset filters; the mount is fine for a single known file.","Quirk: hive partition timestamp=<epoch seconds> is 1 January 00:00 UTC of the survey year (35 yearly partitions, 1992 to 2026; every partition value checked). Partition year equals the survey_date year on 100% of rows. Prune with ds.field('timestamp') >= 1735689600 for 2025 onward.","Quirk: the second hive key polygon=<WKB hex> is a 20 x 20 degree cell (93 distinct cells). Cells sit on a 10 degree stride and OVERLAP, so a lat/lon does not map to a unique cell by arithmetic; filter on latitude/longitude and let the engine prune on timestamp. Every SA-gulfs row of this table lives in the one cell lon 120..140, lat -40..-20 (hex starts 0103000000010000000500000000000000005E40...).","Quirk: timestamp and polygon are NOT stored inside the 438 files (all 438 footers read, one identical schema, no partition columns). They only exist as path keys, so a single-file read with pq.read_table(<file>) has no timestamp/polygon column. _common_metadata lists them (timestamp int64), but hive reads return timestamp as int32.","DO NOT: call ds.dataset(...).count_rows() or expect a _metadata file; there is none (root holds only _common_metadata, 16551 bytes, no row count). count_rows fetched every footer and took 396 s on the fish table from outside AWS. Row counts in this card come from reading all footers (55.88 s with 12 threads).","Quirk: geom is a WKB hex POINT with SRID 4326 in (longitude, latitude) order on 100% of distinct points, unlike the four observation tables where the order is swapped to (latitude, longitude). Use the latitude/longitude columns.","Quirk: hour is documented as local time, but 8% of Australian rows carry 21:00 to 05:59 values (survey list: 1,300 of 16,000 non-null Australian hours) while all Spanish rows are 08:00 to 19:59; some Australian entries look like UTC. TIME is simply survey_date + hour (naive datetime, no tz), so filter on survey_date, not TIME.","Quirk: the schema-level key/value metadata in _common_metadata (one entry per column plus global_attributes with the GeoNetwork metadata_uuid) is a Python dict repr with single quotes, not JSON: json.loads fails, ast.literal_eval(value.decode()) works. The per-file footers carry no key/value metadata at all (0 of the files), so read _common_metadata for it.","Quirk: area is the state/territory or sub-national region ('South Australia'); location is the named region within it ('Encounter', 'Yorke Peninsula', 'Adelaide', 'Kangaroo Island', 'Eyre Peninsula', 'SA - Western', 'Upper Spencer Gulf', 'Upper South East (Robe)'). The brief's bbox (lat -36.5..-32, lon 135..139.5) misses the SA rows from Eyre Peninsula, SA - Western and Robe; use area == 'South Australia' for all of SA, and the bbox for the gulfs plus Fleurieu.","Departure from canonical (join contract, confirmed 2026-10-04): survey_id is unique here and every survey_id and site_code in the fish, invertebrate, cryptic-fish and off-transect tables resolves to a row here (100% each); 98.97% of survey_ids here have rows in at least one observation table. methods lists the codes run: '1' matches the fish table on 36,289 surveys (335 list 1 but have no fish rows; 50 fish surveys with method 10 are not listed as 1), '2' matches the invertebrate table on 36,081 (455 listed without rows) and the cryptic-fish table on only 23,239 (13,297 M2 surveys recorded no cryptic fish), '0' matches the off-transect table on 6,338 (90 listed without rows). Codes 3, 4, 5, 6, 7, 12, 13 have no published table in this family.","Quirk (South Australia, measured 2026-10-04): 3,369 surveys (9.11%) have area == 'South Australia' at 296 sites, 2004-02-05 to 2026-05-29; 3,011 are inside the gulfs bbox (the rest are Eyre Peninsula, SA - Western, Robe). Since 2025-03-01: 321 surveys at 76 sites, all RLS: 2025-03 16, 2025-06 27, 2025-11 20, 2025-12 4, 2026-01 88, 2026-02 82, 2026-03 32, 2026-04 8, 2026-05 44; locations Encounter 250, Yorke Peninsula 43, Adelaide 28; 60 sites in 2026 (Gulf St Vincent GSV* and Coorong-side COO* codes: Pullen Island, Port Elliot, Carrickalinga, Normanville, Granite Island, Myponga, Noarlunga, Aldinga, Semaphore, Port Gawler ...). So yes, SA reefs have been surveyed since the bloom, heavily in Jan-May 2026.","Departure from canonical (release lag): of the 254 SA surveys dated Jan-May 2026 that list method 1, only 5 have fish rows, 8 have invertebrate rows, 4 cryptic-fish rows and 1 off-transect row in the sibling tables as of 2026-10-04; all Jan-Apr 2026 SA surveys have zero published observations. This table is ahead of the observation tables; a survey present here with no observations means not yet released, not an empty reef.","Scale: the whole table is 18 MB Arrow / 21 MB pandas (10 MB without geom/polygon/filename), but a full read is 237.64 s from outside AWS because 438 files of median 10 KB are fetched; one year is 33 files in 19.08 s. Read once, keep in memory.","Quirk: this table has no ecoregion, realm or species columns; take ecoregion/realm from the observation tables (they carry them per row) if you need them per survey.","Quirk: direction is uncurated free text (mixed case, trailing spaces, words, two junk values); surface is null on 96% of rows; avg/max_rugosity are null on 94%; survey_latitude/survey_longitude are strings and 95% null. pq_zip_url (rls.tpac.org.au) is present on 32,076 rows but was not fetched to confirm it resolves.","Quirk: latest_surveydate_for_site is True on 10,440 rows for 4,892 sites because several transects share a site's latest date; use it as a filter for 'most recent visit', not as a per-site unique flag."],"columns":[{"type":"timestamp[ns]. timestamp[ns], naive (no tz). Equals survey_date + hour where hour is present, survey_date at 00:00 where hour is null (checked on all rows of the survey list). Not a UTC instant: see the hour caveat.","name":"TIME"},{"type":"string. string. State/territory or sub-national region, e.g. \"South Australia\", \"Tasmania\", \"Catalonia\". 153 distinct values. This is the column to filter SA on.","name":"area"},{"name":"geom","type":"string. string. WKB hex POINT SRID 4326 in (longitude, latitude) order here (the observation tables swap the axes). Redundant with latitude/longitude."},{"type":"string. string \"HH:MM:SS\", documented local time; null on 16,529 rows (45%). 8% of non-null Australian values are 21:00-05:59 (see caveat).","name":"hour"},{"type":"float. float32, metres, transect depth 0 to 42.","name":"depth"},{"name":"country","type":"string. string. 59 distinct; Australia 31,859 surveys (86%). Null on 3."},{"type":"string. string. Comma-separated RLS/ATRC method codes run on this transect, e.g. \"1, 2\" (9,734 surveys), \"1, 2, 3\" (9,417), \"0, 1, 2\" (4,253), \"1, 13, 2\" (2,054); 71 distinct combinations; null on 40. Codes confirmed against the observation tables: 0 = off-transect sightings, 1 = reef fish (M1), 2 = invertebrates + cryptic fish (M2); 3, 4, 5, 6, 7, 10, 12, 13 are not published in this parquet family.","name":"methods"},{"name":"polygon","type":"string. string, hive partition key (not stored in the files). WKB hex of the 20 x 20 degree spatial cell the row is partitioned into."},{"name":"program","type":"string. string. RLS 20,660, ATRC 13,720, Parks Vic 2,430, FRDC 128, RRH 36."},{"name":"surface","type":"string. string. \"Coral\" 1,059, \"Rock\" 419, null on 35,496 (96%)."},{"type":"string. string. Constant \"ep_survey_list_public_data.csv\".","name":"filename"},{"type":"float. float32, degrees_north. Site latitude.","name":"latitude"},{"name":"location","type":"string. string. Named region within the area, e.g. \"Encounter\", \"Yorke Peninsula\", \"Adelaide\", \"Kangaroo Island\". 221 distinct."},{"type":"string. string. Free-text transect heading: N/S/E/W/NE/SW ... but also lowercase, \"N \", \"West\", \"S.W.\", \"Sam G\", \"lengt\"; null on 18,723 (51%). Normalise before use.","name":"direction"},{"name":"longitude","type":"float. float32, degrees_east. Site longitude."},{"name":"site_code","type":"string. string. Site identifier (e.g. \"GSV135\"); 4,892 distinct."},{"name":"site_name","type":"string. string. 4,864 distinct names."},{"type":"int32. int32. UNIQUE per row (36,974 distinct = row count). Primary key; join target for the four observation tables.","name":"survey_id"},{"name":"timestamp","type":"int64. int64 per _common_metadata, read back as int32 by the hive partitioning. Partition key = epoch seconds of 1 January UTC of the survey year."},{"type":"string. string. URL of the photoquadrat zip for the survey, e.g. http://rls.tpac.org.au/pq/923406716/zip/ ; null on 4,898 rows. Not verified to resolve.","name":"pq_zip_url"},{"type":"float. float32, metres; null on 7,420 rows.","name":"visibility"},{"name":"survey_date","type":"string. string \"YYYY-MM-DD\" local date."},{"type":"float. float32, 1 to 4 scale; null on 34,813 rows (94%).","name":"avg_rugosity"},{"name":"max_rugosity","type":"float. float32; null on 34,812 rows."},{"name":"old_site_codes","type":"string. string. Previous codes for the site, comma-separated with stray spaces (\" AJW 43, DL50,3PVPPH,2803\"); null on 11,081."},{"name":"survey_latitude","type":"string. string, null on 35,042 rows (95%)."},{"type":"string. string, as survey_latitude.","name":"survey_longitude"},{"name":"has_pq_scores_in_db","type":"bool. bool. Photoquadrat scores exist in the NRMN database: True 4,972."},{"name":"has_pqs_catalogued_in_db","type":"bool. bool. Photoquadrats catalogued: True 16,258."},{"name":"has_rugosity_scores_in_db","type":"bool. bool. True 2,532."},{"type":"bool. bool. True on 10,440 rows = this is the most recent survey date for the site (several transects share a date, so it is not unique per site).","name":"latest_surveydate_for_site"}],"format":"v2-hybrid","generatedAt":"2026-10-04T00:00:00Z","layout":{"pattern":"Hive-partitioned parquet, depth 3: timestamp=<epoch seconds, 1 Jan UTC of survey year>/polygon=<WKB hex of a 20x20 degree cell>/ep_survey_list_public_data.csv-0.parquet. Root: 35 timestamp= prefixes (1992..2026) plus _common_metadata (16551 B); no _metadata. 438 data files, 1 to 41 per year, 7.6 KB to 29 KB (median 10 KB). Partition columns are not stored in the files."},"mountPath":"/mnt/archive","overview":"IMOS NRMN survey list: the transect-level master table for Reef Life Survey and ATRC underwater visual census, one row per survey_id (unique, 36,974 rows checked), published quality-controlled by AODN as cloud-optimised parquet. It carries where and when each 50 m transect was done (site_code, site_name, location, area, country, lat/lon, survey_date, hour, depth, visibility, direction), which methods were run as a comma-separated string ('1, 2', '0, 1, 2', '1, 2, 3' ...), the program, and photoquadrat/rugosity flags with a pq_zip_url to the images. Measured 2026-10-04: 36,974 surveys, 4,892 sites, 59 countries, 1992-03-10 to 2026-07-19; RLS 20,660, ATRC 13,720, Parks Vic 2,430; every survey_id and site_code in the four observation tables resolves here (100%), and 99% of these surveys have rows in at least one of them. Layout is 438 files (4.9 MB, median 10 KB) under timestamp=<year epoch>/polygon=<20 degree WKB cell>/; the whole table is 18 MB Arrow / 21 MB pandas (237.64 s from outside AWS, file-count bound), the 2025 partition is 33 files / 1,234 rows in 19.08 s, and the SA-gulfs 2025-2026 pushdown returned 321 surveys in 1.17 s after a 17.35 s discovery. Read it whole once; join everything else to it.","rowCount":"36,974 rows = 36,974 unique survey_id (sum of all 438 footers, 2026-10-04); 2025 partition 1,234 rows, 2026 partition 605 rows (to 2026-07-19)","sampleHeaders":[{"bytes":"50415231150415f00a15fa0a4c15ae011500120000b805f4b70200c8756b8922060e00c0c970e11e060e00e8deaf744d040e009037858ed4050e00a03733e6cd050e007079ae129d040e00c0713ee051040e0028c445a8f4040e00e0037cdba2040e00a0a2c620f0040e008814b6ce6f060e00e0b3f5d47d060e00e8707cc96a060e00206749d378110e0070e10e1b38110e002877cc50c9110e00188c153ace110e00c0f58b977f110e00288082b488110e00204dc83d0d170e006827d4ba07170e004087be4050170e00d0eb642b61170e00602ef37c9f170e000044b62a5b170e00906c02ccf0170e0080575d33f9170e00808b63c9c4210e001049312779210e00c083453f80210e00e872ec4d86210e00706ba01edb300e004818713a88300e0030d59b7f82300e00a899da47d5300e00603d9950d0300e0040592a5070310e005044e1666b310e0088b56b4775310e0080e2228e35300e0070e274363c300e00804f61c741300e0030fce990be310e00580088e0c2310e00d84b1e3ec7310e0018f55928f3320e0078d3559df8320e0078ef5743cc310e00d057886f09320e00983e45870d320e00f04658d212320e0008a1cfcc00330e0010b180a3c7440e00e0582fa4cd440e00009325a710450e0078dee45818450e0060a1109ad0470e0068ea77de5d450e0040133e9394470e00b85547e1dc470e00e034421e8f","truncatedAt":512,"key":"diver_survey_metadata_qc.parquet/timestamp=1009843200/polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0/ep_survey_list_public_data.csv-0.parquet","encoding":"hex"}],"sampleObjects":["timestamp=978307200/polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0/ep_survey_list_public_data.csv-0.parquet","timestamp=978307200/polygon=01030000000100000005000000000000000040604000000000000049C00000000000C0624000000000000049C00000000000C062400000000000003EC000000000004060400000000000003EC0000000000040604000000000000049C0/ep_survey_list_public_data.csv-0.parquet","timestamp=978307200/polygon=01030000000100000005000000000000000080614000000000000044C0000000000000644000000000000044C0000000000000644000000000000034C0000000000080614000000000000034C0000000000080614000000000000044C0/ep_survey_list_public_data.csv-0.parquet","timestamp=1735689600/polygon=01030000000100000005000000000000000000000000000000000044C0000000000000344000000000000044C0000000000000344000000000000034C0000000000000000000000000000034C0000000000000000000000000000044C0/ep_survey_list_public_data.csv-0.parquet","timestamp=1735689600/polygon=010300000001000000050000000000000000000000000000000000494000000000000034400000000000004940000000000000344000000000008051400000000000000000000000000080514000000000000000000000000000004940/ep_survey_list_public_data.csv-0.parquet"],"sampleRows":[{"hour":"13:00:00","survey_date":"2026-05-29","survey_id":923406716,"has_pq_scores_in_db":false,"visibility":5,"site_name":"Outside Granite island","location":"Encounter","depth":5.2,"site_code":"GSV11","latitude":-35.57,"pq_zip_url":"http://rls.tpac.org.au/pq/923406716/zip/","methods":"0, 1, 2","program":"RLS","longitude":138.63,"latest_surveydate_for_site":true},{"survey_date":"2026-05-29","pq_zip_url":"http://rls.tpac.org.au/pq/923406717/zip/","site_code":"GSV11","latitude":-35.57,"hour":"13:00:00","methods":"0, 1, 2","survey_id":923406717,"visibility":5,"longitude":138.63,"program":"RLS","location":"Encounter","has_pq_scores_in_db":false,"depth":5.4,"site_name":"Outside Granite island","latest_surveydate_for_site":true},{"longitude":138.61,"has_rugosity_scores_in_db":false,"timestamp":1735689600,"depth":4.2,"methods":"0, 1, 2","visibility":7,"latitude":-35.59,"area":"South Australia","survey_latitude":null,"survey_id":923405656,"has_pqs_catalogued_in_db":true,"TIME":"2025-06-20 11:30:00","surface":null,"location":"Encounter","program":"RLS","direction":"SE","site_code":"GSV17","site_name":"The Bluff","avg_rugosity":null,"pq_zip_url":"http://rls.tpac.org.au/pq/923405656/zip/","max_rugosity":null,"country":"Australia","survey_date":"2025-06-20","old_site_codes":"VH3,3617","survey_longitude":null,"has_pq_scores_in_db":false,"hour":"11:30:00","latest_surveydate_for_site":false}],"sourceObjectKey":"diver_survey_metadata_qc.parquet/timestamp=1009843200/polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0/ep_survey_list_public_data.csv-0.parquet","starterCell":"# South Australian reef surveys since the March 2025 bloom: who surveyed where, and which months are published\nimport time, pyarrow.dataset as ds\nt0 = time.time()\nimport pyarrow.fs as pafs\ntry:   # ~440 small files: anonymous S3 discovers them in seconds; the FUSE mount took 4-5 minutes for the same call in a real pod (2026-10-04)\n    dset = ds.dataset('aodn-cloud-optimised/diver_survey_metadata_qc.parquet', filesystem=pafs.S3FileSystem(anonymous=True, region='ap-southeast-2'), format='parquet', partitioning='hive')\nexcept Exception:\n    dset = ds.dataset('/mnt/archive', format='parquet', partitioning='hive')\nflt = (ds.field('timestamp') >= 1735689600) & (ds.field('area') == 'South Australia') & (ds.field('survey_date') >= '2025-03-01')\nm = dset.to_table(filter=flt, columns=['survey_id','site_code','site_name','location','survey_date','depth','visibility','methods','program','latitude','longitude']).to_pandas()\nprint(f'{len(m)} surveys, {m.site_code.nunique()} sites, {m.survey_date.min()} to {m.survey_date.max()}, {time.time()-t0:.1f} s')\nprint(m.groupby(m['survey_date'].str[:7]).agg(surveys=('survey_id','size'), sites=('site_code','nunique'), mean_depth_m=('depth','mean')).round(1))\nprint(m['location'].value_counts())\nprint(m['methods'].value_counts().head(5))","starterSamplePath":"timestamp=1009843200/polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0/ep_survey_list_public_data.csv-0.parquet","totalObjects":"439 objects, 4,866,001 bytes (4.87 MB) from aws s3 ls --recursive --summarize on 2026-10-04 (files last modified 2026-10-03)","verifiedRead":{"facts":["under 'diver_survey_metadata_qc.parquet/': 35 directories (e.g. timestamp=1009843200, timestamp=1041379200, timestamp=1072915200, timestamp=1104537600); 1 objects named like _common_metadata, 16 KB to 16 KB","under 'diver_survey_metadata_qc.parquet/timestamp=978307200/': 3 directories (e.g. polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0, polygon=01030000000100000005000000000000000040604000000000000049C00000000000C0624000000000000049C00000000000C062400000000000003EC000000000004060400000000000003EC0000000000040604000000000000049C0, polygon=01030000000100000005000000000000000080614000000000000044C0000000000000644000000000000044C0000000000000644000000000000034C0000000000080614000000000000034C0000000000080614000000000000044C0)","under 'diver_survey_metadata_qc.parquet/timestamp=978307200/polygon=01030000000100000005000000000000000080614000000000000044C0000000000000644000000000000044C0000000000000644000000000000034C0000000000080614000000000000034C0000000000080614000000000000044C0/': 1 objects named like ep_survey_list_public_data.csv-#.parquet, 10 KB to 10 KB"],"starterCell":{"ok":true,"seconds":5.4},"path":"_common_metadata","family":"parquet","bytes":16551,"seconds":0.66,"attempts":[{"family":"parquet","seconds":0.66,"path":"_common_metadata","ok":true}],"image":"531775725781.dkr.ecr.us-east-1.amazonaws.com/scigantic-notebook-genomics:latest","verifiedAt":"2026-10-05T12:59:09.607Z","library":"pyarrow","snippet":"import os\nimport pyarrow.parquet as pq\npf = pq.ParquetFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], '_common_metadata'))\nprint(pf.metadata)\ndf = pf.read_row_group(0).to_pandas()\ndf.head()","rssDeltaMb":0.3,"summary":"Parquet 0 rows, 31 columns, 0 row groups; columns ['TIME:timestamp[ns]', 'area:string', 'geom:string', 'hour:string', 'depth:float', 'country:string', 'methods:string', 'polygon:string', 'program:string', 'surface:string']"}},"summary":"The IMOS National Reef Monitoring Network (NRMN) sub-facility brings together shallow reef surveys conducted around Australia into a centralised database, collating, cleaning and…","title":"National Reef Monitoring Network - Survey metadata","updatedAt":"2026-10-07T13:19:18.325Z","visibility":"public","websiteUrl":"https://catalogue-imos.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/997c2e15-b345-438f-afac-49a4ac19be38"}}