{"success":true,"data":{"author":"AODN","bucketAccessType":"public","bucketListable":true,"bucketName":"aodn-cloud-optimised","bucketPrefix":"diver_off_transect_species_observations_qc.parquet","category":"Life Sciences","cloudProvider":"aws","createdAt":"2026-10-04T00:00:00.000Z","description":"The IMOS National Reef Monitoring Network (NRMN) sub-facility brings together shallow reef surveys conducted around Australia into a centralised database, collating, cleaning and publishing data from Reef Life Survey (RLS), the Australian Temperate Reef Collaboration (ATRC), Parks Australia, DBCA (WA), DEW (SA), DPI (NSW), Tasmanian Parks and Wildlife and Parks Victoria. RLS and ATRC coordinate surveys of rocky and coral reefs using standard underwater visual census methods, carried out by marine scientists and trained recreational divers, to establish national marine baselines and support the management of Commonwealth and State marine reserves. This dataset contains unstandardised occurrence records of bony fishes and elasmobranchs (and some invertebrates, turtles and mammals): any sightings by RLS and ATRC divers that were outside the quantitative survey limits of the transect but can be considered as occurring at that site on the date of the survey they are listed with. It supersedes the RLS-specific 'Global reef fish dataset' off-transect records. This store is the quality-controlled Method 0 table as hive-partitioned parquet: 18,571 rows from 6,338 surveys at 2,506 sites, 2006-02-20 to 2026-05-29, presence-style records (median two per survey) rather than densities; 2,074 rows are South Australian.","displayTitle":"National Reef Monitoring Network - Global off-transect species observations","fuseMountBrowsable":true,"id":"3cb491e8-c468-5d0c-8d41-838bade449c0","license":"http://creativecommons.org/licenses/by/4.0/","objectCountEstimate":21,"probeStatus":"ok","region":"ap-southeast-2","schemaCard":{"accessHints":[{"language":"bash","snippet":"# Task: layout check (first partition year is 2006 here, not 1992)\naws s3 ls s3://aodn-cloud-optimised/diver_off_transect_species_observations_qc.parquet/ --no-sign-request --region ap-southeast-2 | head -3\naws s3 ls s3://aodn-cloud-optimised/diver_off_transect_species_observations_qc.parquet/timestamp=1767225600/ --no-sign-request --region ap-southeast-2"},{"snippet":"# Task: schema + GeoNetwork uuid from _common_metadata\nimport ast, pyarrow.parquet as pq\nschema = pq.read_schema('/mnt/archive/_common_metadata')\nprint(schema.names)\nprint(ast.literal_eval(schema.metadata[b'global_attributes'].decode()))","language":"python-local"},{"snippet":"# Task: the whole table, column-pruned (18,571 rows, ~13 MB; 327 files so expect minutes over S3, seconds on a warm mount)\nimport pandas as pd\ndf = pd.read_parquet('/mnt/archive', columns=['survey_id','site_code','site_name','area','location','survey_date','species_name','class','total','size_class'])\nprint(len(df), df['survey_id'].nunique(), df['survey_date'].min(), df['survey_date'].max())\nprint(df['class'].value_counts().head(6))","language":"python-local"},{"snippet":"# Task: SA gulfs since 2025-03: what was seen off-transect (sharks, rays, turtles, unusual fish), pushdown on year + lat/lon\nimport pyarrow.dataset as ds\ndset = ds.dataset('/mnt/archive', format='parquet', partitioning='hive')\nflt = ((ds.field('timestamp') >= 1735689600) & (ds.field('latitude') >= -36.5) & (ds.field('latitude') <= -32.0)\n       & (ds.field('longitude') >= 135.0) & (ds.field('longitude') <= 139.5) & (ds.field('survey_date') >= '2025-03-01'))\nsa = dset.to_table(filter=flt, columns=['survey_id','site_code','site_name','survey_date','species_name','class','total']).to_pandas()\nprint(len(sa), 'rows', sa['survey_id'].nunique(), 'surveys', sa['site_code'].nunique(), 'sites')\nprint(sa.sort_values('survey_date')[['survey_date','site_name','species_name','class','total']].tail(10).to_string(index=False))","language":"python-local"},{"snippet":"# Task: same SA subset over anonymous S3 from outside the pod\nimport pyarrow.dataset as ds, pyarrow.fs as pafs\ns3 = pafs.S3FileSystem(anonymous=True, region='ap-southeast-2')\ndset = ds.dataset('aodn-cloud-optimised/diver_off_transect_species_observations_qc.parquet', filesystem=s3, format='parquet', partitioning='hive')\nflt = ((ds.field('timestamp') >= 1735689600) & (ds.field('latitude') >= -36.5) & (ds.field('latitude') <= -32.0)\n       & (ds.field('longitude') >= 135.0) & (ds.field('longitude') <= 139.5) & (ds.field('survey_date') >= '2025-03-01'))\nsa = dset.to_table(filter=flt, columns=['survey_id','site_code','survey_date','species_name','class','total']).to_pandas()\nprint(len(sa), sa['survey_id'].nunique(), sa['survey_date'].max())\nprint(sa['class'].value_counts())","language":"python-anonymous"},{"snippet":"# Task: presence list for one site across all years (elasmobranchs only), pushdown on site_code; still touches every file\nimport pyarrow.dataset as ds, pyarrow.fs as pafs\ns3 = pafs.S3FileSystem(anonymous=True, region='ap-southeast-2')\ndset = ds.dataset('aodn-cloud-optimised/diver_off_transect_species_observations_qc.parquet', filesystem=s3, format='parquet', partitioning='hive')\ndf = dset.to_table(filter=(ds.field('location') == 'Encounter') & (ds.field('class') == 'Elasmobranchii'), columns=['site_code','site_name','survey_date','species_name','total']).to_pandas()\nprint(df.groupby('species_name')['survey_date'].agg(['count','min','max']).sort_values('count', ascending=False).head(10))","language":"python-anonymous"}],"caveats":["Scale (mount): pyarrow.dataset over /mnt/archive must stat all ~440 files before the first row; measured 244-310 s in a real pod on 2026-10-04 versus 7-20 s through pyarrow's own anonymous S3 client (pafs.S3FileSystem(anonymous=True, region='ap-southeast-2'), no s3fs needed). Prefer the S3 form for whole-dataset filters; the mount is fine for a single known file.","Quirk: hive partition timestamp=<epoch seconds> is 1 January 00:00 UTC of the survey year (21 yearly partitions, 2006 to 2026; every partition value checked). Partition year equals the survey_date year on 100% of rows. Prune with ds.field('timestamp') >= 1735689600 for 2025 onward.","Quirk: the second hive key polygon=<WKB hex> is a 20 x 20 degree cell (83 distinct cells). Cells sit on a 10 degree stride and OVERLAP, so a lat/lon does not map to a unique cell by arithmetic; filter on latitude/longitude and let the engine prune on timestamp. Every SA-gulfs row of this table lives in the one cell lon 120..140, lat -40..-20 (hex starts 0103000000010000000500000000000000005E40...).","Quirk: timestamp and polygon are NOT stored inside the 327 files (all 327 footers read, one identical schema, no partition columns). They only exist as path keys, so a single-file read with pq.read_table(<file>) has no timestamp/polygon column. _common_metadata lists them (timestamp int64), but hive reads return timestamp as int32.","DO NOT: call ds.dataset(...).count_rows() or expect a _metadata file; there is none (root holds only _common_metadata, 19297 bytes, no row count). count_rows fetched every footer and took 396 s on the fish table from outside AWS. Row counts in this card come from reading all footers (52.4 s with 12 threads).","Quirk: geom is a WKB hex POINT with SRID 4326 whose coordinate order is (latitude, longitude), i.e. x=lat, y=lon, on 100% of distinct points, the reverse of the survey list table (which is x=lon, y=lat). Use the latitude/longitude columns instead.","Quirk: hour is documented as local time, but 8% of Australian rows carry 21:00 to 05:59 values (survey list: 1,300 of 16,000 non-null Australian hours) while all Spanish rows are 08:00 to 19:59; some Australian entries look like UTC. TIME is simply survey_date + hour (naive datetime, no tz), so filter on survey_date, not TIME.","Quirk: the schema-level key/value metadata in _common_metadata (one entry per column plus global_attributes with the GeoNetwork metadata_uuid) is a Python dict repr with single quotes, not JSON: json.loads fails, ast.literal_eval(value.decode()) works. The per-file footers carry no key/value metadata at all (0 of the files), so read _common_metadata for it.","Quirk: area is the state/territory or sub-national region ('South Australia'); location is the named region within it ('Encounter', 'Yorke Peninsula', 'Adelaide', 'Kangaroo Island', 'Eyre Peninsula', 'SA - Western', 'Upper Spencer Gulf', 'Upper South East (Robe)'). The brief's bbox (lat -36.5..-32, lon 135..139.5) misses the SA rows from Eyre Peninsula, SA - Western and Robe; use area == 'South Australia' for all of SA, and the bbox for the gulfs plus Fleurieu.","DO NOT: compute densities or abundance trends from this table. Method 0 records are unstandardised sightings outside the transect limits with no survey effort; use them for presence (species seen at a site on a date), and take abundance from the M1 and M2 tables.","Scale: the whole table is 11.5 MB Arrow / 13.2 MB pandas, trivial to hold, but a full read took 212.0 s with pyarrow S3FileSystem from outside AWS because 327 files of median 10 KB are fetched; one year is 30 files in 11.2 s.","Quirk: block is a string column here (int32 in the fish, invertebrate and cryptic tables) and null on 72% of rows; survey_latitude/survey_longitude are strings and 91% null; visibility is a string. Cast before joining to the other tables.","Quirk: partitions start at timestamp=1136073600 (2006), not 1992 like the other four tables, and there are 21 yearly partitions and 83 spatial cells; a filter for years before 2006 returns nothing by construction.","Quirk (South Australia, measured 2026-10-04): 2,074 rows (11.17%) have area == 'South Australia' from 753 surveys at 162 sites, 2008-02-05 to 2026-05-29; 1,938 inside the gulfs bbox. Since 2025-03-01: 82 rows, 34 surveys, 21 sites, all RLS, dates 2025-03-06, 2025-03-07, 2025-03-11, 2025-06-17, 2025-06-18, 2025-06-19, 2025-06-20, 2025-06-21, 2025-11-13, 2025-11-14, 2025-11-15, 2025-11-26, 2025-12-18, 2026-05-29; the only 2026 record so far is 2026-05-29 (Pullen Island).","Departure from canonical (release lag): the survey list has 321 SA surveys since 2025-03-01, 123 of them listing method 0, but only 34 have rows here and none from January-April 2026. Off-transect records are also inherently sparse (median 2 rows per survey; 6,338 surveys with any record out of 36,974 in the survey list).","Quirk: biomass is null on 40% of rows and size_class is 0.0 on 36%; where present, biomass is the row total in grams for `total` individuals, like the fish table. 210 rows duplicate (survey_id, method, block, species_name, size_class)."],"columns":[{"type":"timestamp[ns]. timestamp[ns], naive (no tz). Equals survey_date + hour where hour is present, survey_date at 00:00 where hour is null (checked on all rows of the survey list). Not a UTC instant: see the hour caveat.","name":"TIME"},{"name":"area","type":"string. string. State/territory or sub-national region, e.g. \"South Australia\", \"Tasmania\", \"Catalonia\". 153 distinct values. This is the column to filter SA on."},{"type":"string. string. WKB hex of a POINT with SRID 4326 (prefix 0101000020E6100000). AXIS ORDER IS (latitude, longitude) in this table, the reverse of the survey list table and of the WKB convention. Redundant with latitude/longitude; drop it.","name":"geom"},{"name":"hour","type":"string. string \"HH:MM:SS\"; null on 5,358 rows (29%)."},{"type":"string. string (not int) in this table: null on 13,422 rows (72%), \"0\" 2,852, \"1\" 1,210, \"2\" 1,087. Not meaningful for off-transect records.","name":"block"},{"type":"string. string. 31 classes: Actinopterygii 15,422, Elasmobranchii 1,151, Gastropoda 326, Malacostraca 310, Reptilia 255, Asteroidea 245, Cephalopoda 159 ...; null 129.","name":"class"},{"name":"depth","type":"float. float32, metres, 0.5 to 38."},{"name":"order","type":"string. string. Taxonomic order."},{"type":"string. string. Spalding MEOW realm, e.g. \"Temperate Australasia\". 12 distinct.","name":"realm"},{"name":"total","type":"int32. int32. Individuals sighted; 0 to 10,000; 44 zero rows."},{"type":"string. string. Taxonomic family.","name":"family"},{"type":"int32. int32. Always 0 (M0 off-transect sighting) on all 18,571 rows.","name":"method"},{"type":"string. string. 10 phyla, Chordata 17,144; null 127.","name":"phylum"},{"name":"biomass","type":"float. float32, grams; null on 7,438 rows (40%); 0 to 7,161,824 g. Unstandardised, no effort."},{"name":"country","type":"string. string. 59 distinct; \"Australia\" is 81% of fish rows."},{"name":"polygon","type":"string. string, hive partition key (not stored in the files). WKB hex of the 20 x 20 degree spatial cell the row is partitioned into."},{"name":"program","type":"string. string. RLS 18,435, ATRC 133, RRH 2, FRDC 1."},{"type":"string. string. Source CSV name; a single constant value per table.","name":"filename"},{"name":"latitude","type":"float. float32, degrees_north. Site latitude (-67.57 to 67.44)."},{"name":"location","type":"string. string. Named region within the area, e.g. \"Encounter\", \"Yorke Peninsula\", \"Adelaide\", \"Kangaroo Island\". 221 distinct."},{"name":"ecoregion","type":"string. string. 90 distinct. SA rows: 'South Australian Gulfs' 1,903, 'Western Bassian' 162, 'Great Australian Bight' 9."},{"type":"float. float32, degrees_east.","name":"longitude"},{"name":"site_code","type":"string. string. Site identifier, e.g. \"GSV135\", \"COO18\"; join key to the survey list and the other observation tables."},{"name":"site_name","type":"string. string. Human site name, e.g. \"Pt Noarlunga inside\"."},{"type":"int32. int32. Transect-level survey identifier (1,003,253 to 923,407,054); primary join key, unique in the survey list table.","name":"survey_id"},{"type":"int64. int64 per _common_metadata, read back as int32 by the hive partitioning. Partition key = epoch seconds of 1 January UTC of the survey year.","name":"timestamp"},{"type":"float. float32, cm. 0.0 on 6,763 rows (36%); otherwise RLS bins 2.5 ... 500 plus a few odd values (9, 13.5, 17).","name":"size_class"},{"name":"visibility","type":"string. string, metres; null on 4,425 rows."},{"type":"string. string \"YYYY-MM-DD\", local date. Use this, not TIME, for date filtering.","name":"survey_date"},{"name":"species_name","type":"string. string. 2,278 distinct; sweepers Pempheris multiradiata (367), Brachaluteres jacksonianus (349), Pempheris affinis (266) lead."},{"type":"string. string. Name used for reporting (usually equals species_name).","name":"reporting_name"},{"name":"survey_latitude","type":"string. string, null on 16,963 rows (91%)."},{"name":"survey_longitude","type":"string. string, as survey_latitude."}],"format":"v2-hybrid","generatedAt":"2026-10-04T00:00:00Z","layout":{"pattern":"Hive-partitioned parquet, depth 3: timestamp=<epoch seconds, 1 Jan UTC of survey year>/polygon=<WKB hex of a 20x20 degree cell>/ep_m0_off_transect_sighting_public_data.csv-0.parquet. Root: 21 timestamp= prefixes (2006..2026) plus _common_metadata (19297 B); no _metadata. 327 data files, 1 to 36 per year, 8.5 KB to 38 KB (median 10 KB). Partition columns are not stored in the files."},"mountPath":"/mnt/archive","overview":"Reef Life Survey / ATRC 'Method 0' off-transect sightings published by IMOS NRMN, quality-controlled, in AODN's cloud-optimised parquet layout: opportunistic presence records of species seen at a site on a survey date but outside the 50 m transect's counting limits, so they carry a count and sometimes a size and biomass but no area and no effort. Measured 2026-10-04: 18,571 rows (the smallest NRMN table), 6,338 survey_ids, 2,506 sites, 52 countries, 90 ecoregions, 2006-02-20 to 2026-05-29; RLS 18,435 rows (99%), ATRC 133. Actinopterygii 15,422 rows, Elasmobranchii 1,151, then gastropods, crustaceans, turtles (Reptilia 255), sea stars, cephalopods. Layout is 327 files (3.9 MB, median 10 KB) under timestamp=<year epoch, 2006..2026>/polygon=<20 degree WKB cell>/: the 2025 partition is 30 files, 0.34 MB, 1,153 rows in 11.2 s via pandas/s3fs; the SA-gulfs 2025-2026 pushdown returned 82 rows in 1.26 s after a 6.73 s discovery; the whole table is only 11.5 MB Arrow / 13.2 MB pandas yet took 212.0 s because 327 tiny files are fetched one by one. Read it whole once and keep it in memory; the per-file latency, not the bytes, is the cost.","rowCount":"18,571 rows (sum of all 327 footers, 2026-10-04); 2025 partition 1,153 rows, 2026 partition 186 rows","sampleHeaders":[{"bytes":"504152311504151015144c15021500120000081c000021e8ef7dd30f1500151215162c15021510150615061c1808000021e8ef7dd30f1808000021e8ef7dd30f16002808000021e8ef7dd30f1808000021e8ef7dd30f1111000000092002000000020101020015041526152a4c1502150012000013480f0000004e657720536f7574682057616c65731500151215162c15021510150615061c3600280f4e657720536f7574682057616c6573180f4e657720536f7574682057616c6573111100000009200200000002010102001504156c155a4c15021500120000363c3200000030313031303030303230453605091d0154383033464330304144374133373033444532363334301500151215162c15021510150615061c36002832303130313030303032304536313030303030303030303030303030303830334643303041443741333730334445323633343018323031303130303030323045363130303030303030303030303030303038303346433030414437413337303344453236333430111100000009200200000002010102001504150015024c15001500120000001500150e15122c15021510150615061c36020000000718020000000200001504150015024c15001500120000001500150e15122c15021510150615061c360200000007180200000002000015041518151c4c150215001200000c2c08000000","encoding":"hex","key":"diver_off_transect_species_observations_qc.parquet/timestamp=1136073600/polygon=01030000000100000005000000000000000080614000000000000044C0000000000000644000000000000044C0000000000000644000000000000034C0000000000080614000000000000034C0000000000080614000000000000044C0/ep_m0_off_transect_sighting_public_data.csv-0.parquet","truncatedAt":512}],"sampleObjects":["timestamp=1767225600/polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0/ep_m0_off_transect_sighting_public_data.csv-0.parquet","timestamp=1767225600/polygon=01030000000100000005000000000000000040604000000000000049C00000000000C0624000000000000049C00000000000C062400000000000003EC000000000004060400000000000003EC0000000000040604000000000000049C0/ep_m0_off_transect_sighting_public_data.csv-0.parquet","timestamp=1767225600/polygon=01030000000100000005000000000000000080614000000000000044C0000000000000644000000000000044C0000000000000644000000000000034C0000000000080614000000000000034C0000000000080614000000000000044C0/ep_m0_off_transect_sighting_public_data.csv-0.parquet","timestamp=1735689600/polygon=01030000000100000005000000000000000000000000000000000044C0000000000000344000000000000044C0000000000000344000000000000034C0000000000000000000000000000034C0000000000000000000000000000044C0/ep_m0_off_transect_sighting_public_data.csv-0.parquet","timestamp=1735689600/polygon=010300000001000000050000000000000000000000000000000000494000000000000034400000000000004940000000000000344000000000008051400000000000000000000000000080514000000000000000000000000000004940/ep_m0_off_transect_sighting_public_data.csv-0.parquet"],"sampleRows":[{"class":"Actinopterygii","TIME":"2025-06-20 11:30:00","location":"Encounter","ecoregion":"Western Bassian","visibility":"7.0","family":"Odacidae","order":"Perciformes","total":1,"country":"Australia","reporting_name":"Olisthops cyanomelas","latitude":-35.59,"site_code":"GSV17","survey_latitude":null,"program":"RLS","site_name":"The Bluff","survey_id":923405656,"block":"2","biomass":173.2851,"longitude":138.61,"size_class":25,"hour":"11:30:00","survey_date":"2025-06-20","timestamp":1735689600,"method":0,"depth":4.2,"realm":"Temperate Australasia","phylum":"Chordata","survey_longitude":null,"species_name":"Olisthops cyanomelas","area":"South Australia"},{"survey_date":"2025-06-20","survey_longitude":null,"method":0,"realm":"Temperate Australasia","survey_id":923405657,"visibility":"6.0","size_class":7.5,"species_name":"Enoplosus armatus","ecoregion":"Western Bassian","family":"Enoplosidae","site_code":"GSV17","latitude":-35.59,"program":"RLS","longitude":138.61,"class":"Actinopterygii","country":"Australia","area":"South Australia","hour":"11:30:00","phylum":"Chordata","biomass":3.6914,"site_name":"The Bluff","block":"2","timestamp":1735689600,"total":1,"order":"Perciformes","survey_latitude":null,"reporting_name":"Enoplosus armatus","depth":5.1,"TIME":"2025-06-20 11:30:00","location":"Encounter"},{"TIME":"2026-05-29 10:30:00","method":0,"site_name":"Pullen Island","visibility":"3.0","phylum":"Chordata","block":"0","timestamp":1767225600,"country":"Australia","survey_id":923406700,"survey_date":"2026-05-29","reporting_name":"Pseudogoniistius nigripes","ecoregion":"Western Bassian","realm":"Temperate Australasia","area":"South Australia","class":"Actinopterygii","longitude":138.69,"family":"Latridae","survey_latitude":null,"program":"RLS","survey_longitude":null,"order":"Perciformes","total":2,"latitude":-35.54,"hour":"10:30:00","size_class":25,"species_name":"Pseudogoniistius nigripes","biomass":515.9382,"depth":5.4,"location":"Encounter","site_code":"COO18"}],"sourceObjectKey":"diver_off_transect_species_observations_qc.parquet/timestamp=1136073600/polygon=01030000000100000005000000000000000080614000000000000044C0000000000000644000000000000044C0000000000000644000000000000034C0000000000080614000000000000034C0000000000080614000000000000044C0/ep_m0_off_transect_sighting_public_data.csv-0.parquet","starterCell":"# SA gulfs + Fleurieu off-transect sightings since the March 2025 bloom\nimport time, pyarrow.dataset as ds\nt0 = time.time()\nimport pyarrow.fs as pafs\ntry:   # ~440 small files: anonymous S3 discovers them in seconds; the FUSE mount took 4-5 minutes for the same call in a real pod (2026-10-04)\n    dset = ds.dataset('aodn-cloud-optimised/diver_off_transect_species_observations_qc.parquet', filesystem=pafs.S3FileSystem(anonymous=True, region='ap-southeast-2'), format='parquet', partitioning='hive')\nexcept Exception:\n    dset = ds.dataset('/mnt/archive', format='parquet', partitioning='hive')\nflt = ((ds.field('timestamp') >= 1735689600) & (ds.field('latitude') >= -36.5) & (ds.field('latitude') <= -32.0)\n       & (ds.field('longitude') >= 135.0) & (ds.field('longitude') <= 139.5) & (ds.field('survey_date') >= '2025-03-01'))\nsa = dset.to_table(filter=flt, columns=['survey_id','site_code','site_name','location','survey_date','species_name','class','size_class','total']).to_pandas()\nprint(f'{len(sa)} rows, {sa.survey_id.nunique()} surveys, {sa.site_code.nunique()} sites, {sa.survey_date.min()} to {sa.survey_date.max()}, {time.time()-t0:.1f} s')\nprint(sa.groupby(sa['survey_date'].str[:7])['survey_id'].nunique().rename('surveys_per_month'))\nprint(sa.groupby(['class','species_name'])['total'].sum().sort_values(ascending=False).head(10))","starterSamplePath":"timestamp=1136073600/polygon=01030000000100000005000000000000000080614000000000000044C0000000000000644000000000000044C0000000000000644000000000000034C0000000000080614000000000000034C0000000000080614000000000000044C0/ep_m0_off_transect_sighting_public_data.csv-0.parquet","totalObjects":"328 objects, 3,856,309 bytes (3.86 MB) from aws s3 ls --recursive --summarize on 2026-10-04","verifiedRead":{"library":"pyarrow","family":"parquet","starterCell":{"ok":true,"seconds":5.05},"image":"531775725781.dkr.ecr.us-east-1.amazonaws.com/scigantic-notebook-genomics:latest","path":"_common_metadata","bytes":19297,"snippet":"import os\nimport pyarrow.parquet as pq\npf = pq.ParquetFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], '_common_metadata'))\nprint(pf.metadata)\ndf = pf.read_row_group(0).to_pandas()\ndf.head()","verifiedAt":"2026-10-04T21:31:35.873Z","rssDeltaMb":0.3,"facts":["under 'diver_off_transect_species_observations_qc.parquet/': 21 directories (e.g. timestamp=1136073600, timestamp=1167609600, timestamp=1199145600, timestamp=1230768000); 1 objects named like _common_metadata, 19 KB to 19 KB","under 'diver_off_transect_species_observations_qc.parquet/timestamp=1767225600/': 6 directories (e.g. polygon=0103000000010000000500000000000000000024C00000000000003E4000000000000024400000000000003E400000000000002440000000000000494000000000000024C0000000000000494000000000000024C00000000000003E40, polygon=01030000000100000005000000000000000000594000000000000044C00000000000005E4000000000000044C00000000000005E4000000000000034C0000000000000594000000000000034C0000000000000594000000000000044C0, polygon=010300000001000000050000000000000000005E4000000000000044C0000000000080614000000000000044C0000000000080614000000000000034C00000000000005E4000000000000034C00000000000005E4000000000000044C0, polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0)","under 'diver_off_transect_species_observations_qc.parquet/timestamp=1767225600/polygon=01030000000100000005000000000000000080614000000000000044C0000000000000644000000000000044C0000000000000644000000000000034C0000000000080614000000000000034C0000000000080614000000000000044C0/': 1 objects named like ep_m#_off_transect_sighting_public_data.csv-#.parquet, 15 KB to 15 KB"],"attempts":[{"seconds":0.67,"path":"_common_metadata","ok":true,"family":"parquet"}],"seconds":0.67,"summary":"Parquet 0 rows, 33 columns, 0 row groups; columns ['TIME:timestamp[ns]', 'area:string', 'geom:string', 'hour:string', 'block:string', 'class:string', 'depth:float', 'order:string', 'realm:string', 'total:int32']"}},"summary":"The IMOS National Reef Monitoring Network (NRMN) sub-facility brings together shallow reef surveys conducted around Australia into a centralised database, collating, cleaning and…","title":"National Reef Monitoring Network - Global off-transect species observations","updatedAt":"2026-10-07T13:45:07.975Z","visibility":"public","websiteUrl":"https://catalogue-imos.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/a73907b7-3686-4dcf-8a1d-7efc4e5ffc05"}}