{"success":true,"data":{"author":"AODN","bucketAccessType":"public","bucketListable":true,"bucketName":"aodn-cloud-optimised","bucketPrefix":"diver_reef_fish_abundance_biomass_qc.parquet","category":"Life Sciences","cloudProvider":"aws","createdAt":"2026-10-04T00:00:00.000Z","description":"The IMOS National Reef Monitoring Network (NRMN) sub-facility brings together shallow reef surveys conducted around Australia into a centralised database, collating, cleaning and publishing data from Reef Life Survey (RLS), the Australian Temperate Reef Collaboration (ATRC), Parks Australia, DBCA (WA), DEW (SA), DPI (NSW), Tasmanian Parks and Wildlife and Parks Victoria. RLS and ATRC coordinate surveys of rocky and coral reefs using standard underwater visual census methods, carried out by marine scientists and trained recreational divers, to establish national marine baselines and support the management of Commonwealth and State marine reserves. This dataset contains records of bony fishes and elasmobranchs collected by RLS and ATRC divers along 50 m transects on shallow rocky and coral reefs: abundance for every species recorded within quantitative survey limits (50 x 5 m swathes either side of the transect line, each a 'Block'), with size class and estimated biomass where available. These observations form the Method 1 component of the surveys and supersede the RLS-specific 'Global reef fish dataset'. This store is the quality-controlled M1 table as hive-partitioned parquet: 1,913,820 rows from 36,339 transect surveys at 4,875 sites in 59 countries, 1992-03-10 to 2026-07-19, of which 96,876 rows are South Australian.","displayTitle":"National Reef Monitoring Network - Global reef fish abundance and biomass","fuseMountBrowsable":true,"id":"4de33273-e822-59eb-8841-277d3f6df5ff","license":"http://creativecommons.org/licenses/by/4.0/","objectCountEstimate":35,"probeStatus":"ok","region":"ap-southeast-2","schemaCard":{"accessHints":[{"language":"bash","snippet":"# Task: see the on-disk layout without a recursive walk (root = yearly partitions + _common_metadata)\naws s3 ls s3://aodn-cloud-optimised/diver_reef_fish_abundance_biomass_qc.parquet/ --no-sign-request --region ap-southeast-2\naws s3 ls s3://aodn-cloud-optimised/diver_reef_fish_abundance_biomass_qc.parquet/timestamp=1735689600/ --no-sign-request --region ap-southeast-2 | head -5"},{"language":"python-local","snippet":"# Task: read the schema and the GeoNetwork identity in one request from _common_metadata (fast path, no data read)\nimport ast, pyarrow.parquet as pq\nschema = pq.read_schema('/mnt/archive/_common_metadata')\nprint(schema)\nprint(ast.literal_eval(schema.metadata[b'global_attributes'].decode()))\nprint(ast.literal_eval(schema.metadata[b'biomass'].decode()))   # per-column attrs live here too"},{"snippet":"# Task: one survey year (2025) as a DataFrame; hive keys become columns. 33 files, 1.2 MB, 82,719 rows, ~13 s over S3\nimport pandas as pd\ndf = pd.read_parquet('/mnt/archive/timestamp=1735689600', columns=['survey_id','site_code','site_name','area','location','survey_date','depth','species_name','size_class','total','biomass','block'])\nprint(len(df), df['survey_id'].nunique(), df['survey_date'].min(), df['survey_date'].max())","language":"python-local"},{"language":"python-local","snippet":"# Task: South Australian gulfs since the March 2025 bloom, with partition pruning on year and filter pushdown on lat/lon\nimport pyarrow.dataset as ds\ndset = ds.dataset('/mnt/archive', format='parquet', partitioning='hive')\nflt = ((ds.field('timestamp') >= 1735689600) & (ds.field('latitude') >= -36.5) & (ds.field('latitude') <= -32.0)\n       & (ds.field('longitude') >= 135.0) & (ds.field('longitude') <= 139.5) & (ds.field('survey_date') >= '2025-03-01'))\nsa = dset.to_table(filter=flt, columns=['survey_id','site_code','site_name','location','survey_date','depth','species_name','size_class','total','biomass']).to_pandas()\nprint(len(sa), 'rows', sa['survey_id'].nunique(), 'surveys', sa['site_code'].nunique(), 'sites', sa['survey_date'].max())"},{"snippet":"# Task: the same SA-gulfs-since-2025-03 subset from outside the pod, anonymous S3 (pyarrow's own S3 client, no s3fs needed)\nimport pyarrow.dataset as ds, pyarrow.fs as pafs\ns3 = pafs.S3FileSystem(anonymous=True, region='ap-southeast-2')\ndset = ds.dataset('aodn-cloud-optimised/diver_reef_fish_abundance_biomass_qc.parquet', filesystem=s3, format='parquet', partitioning='hive')\nflt = ((ds.field('timestamp') >= 1735689600) & (ds.field('latitude') >= -36.5) & (ds.field('latitude') <= -32.0)\n       & (ds.field('longitude') >= 135.0) & (ds.field('longitude') <= 139.5) & (ds.field('survey_date') >= '2025-03-01'))\nsa = dset.to_table(filter=flt, columns=['survey_id','site_code','site_name','survey_date','species_name','size_class','total','biomass']).to_pandas()\nprint(len(sa), sa['survey_id'].nunique(), sa['survey_date'].max())\nprint(sa.groupby('species_name')['total'].sum().sort_values(ascending=False).head(10))","language":"python-anonymous"},{"snippet":"# Task: full history of one species, all years, column-pruned (touches all 438 files: expect 2-4 minutes outside AWS, ~400 MB for a core column set)\nimport pyarrow.dataset as ds, pyarrow.fs as pafs\ns3 = pafs.S3FileSystem(anonymous=True, region='ap-southeast-2')\ndset = ds.dataset('aodn-cloud-optimised/diver_reef_fish_abundance_biomass_qc.parquet', filesystem=s3, format='parquet', partitioning='hive')\ntbl = dset.to_table(filter=(ds.field('species_name') == 'Haliotis rubra') | (ds.field('species_name') == 'Notolabrus tetricus'),\n                    columns=['survey_id','site_code','area','survey_date','species_name','size_class','total','biomass'])\ndf = tbl.to_pandas()\nprint(len(df), df.groupby(df['survey_date'].str[:4])['total'].sum().tail(5))","language":"python-anonymous"},{"snippet":"# Task: which SA surveys since 2025-03 in the survey list already have M1 fish rows here (join on survey_id)\nimport pandas as pd, pyarrow.dataset as ds, pyarrow.fs as pafs\ns3 = pafs.S3FileSystem(anonymous=True, region='ap-southeast-2')\nmeta = ds.dataset('aodn-cloud-optimised/diver_survey_metadata_qc.parquet', filesystem=s3, format='parquet', partitioning='hive')\nm = meta.to_table(filter=(ds.field('timestamp') >= 1735689600) & (ds.field('area') == 'South Australia') & (ds.field('survey_date') >= '2025-03-01'),\n                  columns=['survey_id','site_code','site_name','survey_date','methods']).to_pandas()\nfish = ds.dataset('aodn-cloud-optimised/diver_reef_fish_abundance_biomass_qc.parquet', filesystem=s3, format='parquet', partitioning='hive')\nf = fish.to_table(filter=(ds.field('timestamp') >= 1735689600) & (ds.field('area') == 'South Australia'), columns=['survey_id']).to_pandas()\nm['has_fish_rows'] = m['survey_id'].isin(set(f['survey_id']))\nprint(m.groupby(m['survey_date'].str[:7])['has_fish_rows'].agg(['size','sum']))","language":"python-anonymous"}],"caveats":["Scale (mount): pyarrow.dataset over /mnt/archive must stat all ~440 files before the first row; measured 244-310 s in a real pod on 2026-10-04 versus 7-20 s through pyarrow's own anonymous S3 client (pafs.S3FileSystem(anonymous=True, region='ap-southeast-2'), no s3fs needed). Prefer the S3 form for whole-dataset filters; the mount is fine for a single known file.","Quirk: hive partition timestamp=<epoch seconds> is 1 January 00:00 UTC of the survey year (35 yearly partitions, 1992 to 2026; every partition value checked). Partition year equals the survey_date year on 100% of rows. Prune with ds.field('timestamp') >= 1735689600 for 2025 onward.","Quirk: the second hive key polygon=<WKB hex> is a 20 x 20 degree cell (93 distinct cells). Cells sit on a 10 degree stride and OVERLAP, so a lat/lon does not map to a unique cell by arithmetic; filter on latitude/longitude and let the engine prune on timestamp. Every SA-gulfs row of this table lives in the one cell lon 120..140, lat -40..-20 (hex starts 0103000000010000000500000000000000005E40...).","Quirk: timestamp and polygon are NOT stored inside the 438 files (all 438 footers read, one identical schema, no partition columns). They only exist as path keys, so a single-file read with pq.read_table(<file>) has no timestamp/polygon column. _common_metadata lists them (timestamp int64), but hive reads return timestamp as int32.","DO NOT: call ds.dataset(...).count_rows() or expect a _metadata file; there is none (root holds only _common_metadata, 19219 bytes, no row count). count_rows fetched every footer and took 396.43 s on the fish table from outside AWS. Row counts in this card come from reading all footers (54.76 s with 12 threads).","Quirk: geom is a WKB hex POINT with SRID 4326 whose coordinate order is (latitude, longitude), i.e. x=lat, y=lon, on 100% of distinct points, the reverse of the survey list table (which is x=lon, y=lat). Use the latitude/longitude columns instead.","Quirk: hour is documented as local time, but 8% of Australian rows carry 21:00 to 05:59 values (survey list: 1,300 of 16,000 non-null Australian hours) while all Spanish rows are 08:00 to 19:59; some Australian entries look like UTC. TIME is simply survey_date + hour (naive datetime, no tz), so filter on survey_date, not TIME.","Quirk: the schema-level key/value metadata in _common_metadata (one entry per column plus global_attributes with the GeoNetwork metadata_uuid) is a Python dict repr with single quotes, not JSON: json.loads fails, ast.literal_eval(value.decode()) works. The per-file footers carry no key/value metadata at all (0 of the files), so read _common_metadata for it.","Quirk: area is the state/territory or sub-national region ('South Australia'); location is the named region within it ('Encounter', 'Yorke Peninsula', 'Adelaide', 'Kangaroo Island', 'Eyre Peninsula', 'SA - Western', 'Upper Spencer Gulf', 'Upper South East (Robe)'). The brief's bbox (lat -36.5..-32, lon 135..139.5) misses the SA rows from Eyre Peninsula, SA - Western and Robe; use area == 'South Australia' for all of SA, and the bbox for the gulfs plus Fleurieu.","Scale: the whole table is 1.17 GB in Arrow and 1.32 GB in pandas for 1,913,820 rows; polygon alone is 371 MB (a 200-char hex repeated per row), geom 111 MB, filename 56 MB. A 17-column core set is 400 MB. Never pd.read_parquet('/mnt/archive') without columns= on a 4 GiB pod.","Scale: full-table reads are file-count bound (438 files, median 37 KB): pyarrow S3FileSystem to_table 147.86 s, pandas over s3fs 243 s, measured from outside AWS on 2026-10-04. Reading one year is 33 files / 12.79 s.","Quirk (South Australia, measured 2026-10-04): 96,876 rows (5.06%) have area == 'South Australia', from 3,017 surveys at 293 sites, 2004-02-05 to 2026-05-29; the gulfs bbox holds 83,743 of them. Since 2025-03-01 there are 1,149 rows from 71 surveys at 25 sites, all RLS, on 15 dates: 2025-03-06, 2025-03-07, 2025-03-11, 2025-06-17, 2025-06-18, 2025-06-19, 2025-06-20, 2025-06-21, 2025-11-13, 2025-11-14, 2025-11-15, 2025-11-26, 2025-12-18, 2026-05-14, 2026-05-29. By month: 2025-03 16 surveys (Adelaide/Noarlunga/Moana/Horseshoe), 2025-06 26 (Encounter + Yorke Peninsula), 2025-11 20, 2025-12 4 (Outer Harbor), 2026-05 5 (Cape Jervis South 2026-05-14, Pullen Island 2026-05-29).","Departure from canonical (release lag): the survey list table holds 321 SA surveys since 2025-03-01, including 254 in Jan-May 2026 that list method 1, but only 5 of those 2026 surveys have fish rows here (2026-01: 88 listed / 0 with rows; 2026-02: 82 / 0; 2026-03: 32 / 0; 2026-04: 8 / 0; 2026-05: 44 / 5). The post-bloom 2026 Gulf St Vincent / Encounter campaign (60 sites) exists as surveys but its counts were not yet published on 2026-10-04. Join to nrmn-survey-metadata before concluding a site was not surveyed.","Departure from canonical (RLS M1 geometry): a survey is one 50 m transect; M1 counts fish in two 50 x 5 m blocks (block 1, 2), so per-transect density = sum(total) over both blocks / 500 m2, and most analyses also exclude size_class 0 rows. 777 rows duplicate (survey_id, method, block, species_name, size_class), so aggregate with sum rather than assuming one row per key.","Quirk: dtype drift ACROSS the five NRMN tables, not across files: here depth/latitude/longitude/survey_latitude are double and biomass float, while the invertebrate, cryptic-fish, off-transect and survey-list tables use float32 and type survey_latitude/survey_longitude as string. Cast before concatenating tables.","Quirk: survey_latitude/survey_longitude are null on 94% of rows (1,796,571 of 1,913,820); where present they differ from the site latitude by >0.001 degrees on 7.5% of rows. Use latitude/longitude (site position).","Quirk: species_name includes non-fish M1 records (Cephalopoda 1,556 rows, Reptilia 666, Mammalia 309, Scyphozoa/Hydrozoa/ctenophores) whose biomass is always null; filter class in ('Actinopterygii','Teleostei','Elasmobranchii','Actinopteri') for a fish-only census."],"columns":[{"type":"timestamp[ns]. timestamp[ns], naive (no tz). Equals survey_date + hour where hour is present, survey_date at 00:00 where hour is null (checked on all rows of the survey list). Not a UTC instant: see the hour caveat.","name":"TIME"},{"name":"area","type":"string. string. State/territory or sub-national region, e.g. \"South Australia\", \"Tasmania\", \"Catalonia\". 153 distinct values. This is the column to filter SA on."},{"name":"geom","type":"string. string. WKB hex of a POINT with SRID 4326 (prefix 0101000020E6100000). AXIS ORDER IS (latitude, longitude) in this table, the reverse of the survey list table and of the WKB convention. Redundant with latitude/longitude; drop it."},{"name":"hour","type":"string. string \"HH:MM:SS\", documented as local time of sample collection. Null on 32% (fish) to 29% (off-transect) of rows."},{"type":"int32. int32. Which 50 x 5 m swathe of the transect: 1 (974,362 rows) or 2 (939,458 rows); 98% of surveys have both blocks.","name":"block"},{"name":"class","type":"string. string. Actinopterygii 1,892,118 rows, Teleostei 10,078, Elasmobranchii 8,580, Cephalopoda 1,556, Reptilia 666, Mammalia 309, plus jellies; i.e. the M1 census also carries a few non-fish mobile animals."},{"type":"double. double (fish) / float (others), metres. Transect depth, 0 to 42 m.","name":"depth"},{"name":"order","type":"string. string. Taxonomic order."},{"type":"string. string. Spalding MEOW realm, e.g. \"Temperate Australasia\". 12 distinct.","name":"realm"},{"type":"int32. int32. Individuals counted for this species x size class x block; 0 to 800,000 (one row); exactly one zero row.","name":"total"},{"name":"family","type":"string. string. Taxonomic family."},{"name":"method","type":"int32. int32. RLS/ATRC method code: 1 on 1,913,293 rows (M1 fish transect). 527 rows carry method 10 (50 ATRC surveys, Western Australia, 1997 only; the survey list shows methods '10' for them), meaning unverified."},{"type":"string. string. Taxonomic phylum.","name":"phylum"},{"name":"biomass","type":"float. float, grams. Estimated biomass of all `total` individuals in the row (length-weight on size_class): median 24.2 g for total=1 at 10 cm, 269 g for total=10 at 10 cm. Null on 3,564 rows (0.19%), always null for Cephalopoda, Reptilia, Mammalia and jellies. Max 15,810,826 g in a single row."},{"name":"country","type":"string. string. 59 distinct; Australia 1,546,047 rows (81%), then Indonesia, Spain, United States, Panama. Null on 222 rows."},{"type":"string. string, hive partition key (not stored in the files). WKB hex of the 20 x 20 degree spatial cell the row is partitioned into.","name":"polygon"},{"type":"string. string. Contributing program: RLS, ATRC, Parks Vic, FRDC, RRH.","name":"program"},{"name":"filename","type":"string. string. Source CSV name; a single constant value per table."},{"type":"double. double (fish) / float (others), degrees_north WGS84. Site latitude.","name":"latitude"},{"name":"location","type":"string. string. Named region within the area, e.g. \"Encounter\", \"Yorke Peninsula\", \"Adelaide\", \"Kangaroo Island\". 221 distinct."},{"name":"ecoregion","type":"string. string. Spalding MEOW ecoregion, 104 distinct. SA rows fall in 'South Australian Gulfs' (83,272), 'Western Bassian' (7,928) and 'Great Australian Bight' (5,676)."},{"name":"longitude","type":"double. double (fish) / float (others), degrees_east WGS84. Site longitude."},{"name":"site_code","type":"string. string. Site identifier, e.g. \"GSV135\", \"COO18\"; join key to the survey list and the other observation tables."},{"name":"site_name","type":"string. string. Human site name, e.g. \"Pt Noarlunga inside\"."},{"type":"int32. int32. Transect-level survey identifier (1,003,253 to 923,407,054); primary join key, unique in the survey list table.","name":"survey_id"},{"type":"int64. int64 per _common_metadata, read back as int32 by the hive partitioning. Partition key = epoch seconds of 1 January UTC of the survey year.","name":"timestamp"},{"name":"size_class","type":"float. double. RLS size bin in cm: 2.5, 5, 7.5, 10, 12.5, 15, 20, 25, 30, 35, 40, 50, 62.5, 75, 87.5, 100 ... 400 (29 values). 0.0 on 648 rows = not sized."},{"type":"string. string (fish, inverts, off-transect) / float (cryptic), metres. Null on 16% to 24% of rows.","name":"visibility"},{"type":"string. string \"YYYY-MM-DD\", local date. Use this, not TIME, for date filtering.","name":"survey_date"},{"type":"string. string. 3,483 distinct scientific names; Notolabrus tetricus (87,238 rows) is the most frequent.","name":"species_name"},{"name":"reporting_name","type":"string. string. Name used for reporting (usually equals species_name)."},{"name":"survey_latitude","type":"double. double (fish) / float (cryptic) / string (inverts, off-transect). Transect-specific latitude; null on 94% of rows."},{"name":"survey_longitude","type":"double. as survey_latitude."}],"format":"v2-hybrid","generatedAt":"2026-10-04T00:00:00Z","layout":{"pattern":"Hive-partitioned parquet, depth 3: timestamp=<epoch seconds, 1 Jan UTC of survey year>/polygon=<WKB hex of a 20x20 degree cell>/ep_m1_public_data.csv-0.parquet. Root: 35 timestamp= prefixes (1992..2026) plus _common_metadata (19219 B, schema + per-column attrs + global_attributes with the GeoNetwork uuid); no _metadata. 438 data files, one per (year, cell), 1 to 41 per year, 8.9 KB to 384 KB (median 37 KB), 1 to 11 row groups each. Files hold 2 to 41,018 rows. Partition columns are not stored in the files."},"mountPath":"/mnt/archive","overview":"Reef Life Survey / ATRC Method 1 fish census (the RLS 'M1' block-level counts behind Edgar & Stuart-Smith 2014, Sci Data) as published by IMOS NRMN, quality-controlled, in AODN's cloud-optimised parquet layout. Each row is one species x size class x block on one 50 m transect: total count, biomass (g, summed over the count), taxonomy, site, program and region. Measured 2026-10-04: 1,913,820 rows, 36,339 survey_ids, 4,875 sites, 59 countries, 104 ecoregions, 1992-03-10 to 2026-07-19; programs RLS 1,478,993, ATRC 393,577, Parks Vic 38,433. Layout is 438 small files (24.5 MB total, median 37 KB) under timestamp=<year epoch>/polygon=<20 degree WKB cell>/, so the access direction it wants is a year range plus a lat/lon box: the 2025 partition is 33 files, 1.23 MB, 82,719 rows, read in 12.79 s with pandas over s3fs; the SA-gulfs 2025-2026 subset (filter pushdown on timestamp + lat/lon) returned 1,149 rows in 0.81 s after an 7.95 s dataset discovery. A full species history means all 438 files: 147.86 s with pyarrow's S3 filesystem, 396.43 s if you let count_rows do it, and 1.17 GB of Arrow (1.32 GB pandas), 371 MB of which is the repeated polygon hex, so select columns on a 4 GiB pod.","rowCount":"1,913,820 rows (sum of all 438 file footers, 2026-10-04); 2025 partition 82,719 rows, 2026 partition 19,344 rows (year to 2026-07-19)","sampleHeaders":[{"truncatedAt":512,"bytes":"50415231150415f00a15fa0a4c15ae011500120000b805f4b70200c8756b8922060e00c0c970e11e060e00e8deaf744d040e009037858ed4050e00a03733e6cd050e007079ae129d040e00c0713ee051040e0028c445a8f4040e00e0037cdba2040e00a0a2c620f0040e008814b6ce6f060e00e0b3f5d47d060e00e8707cc96a060e00206749d378110e0070e10e1b38110e002877cc50c9110e00188c153ace110e00c0f58b977f110e00288082b488110e00204dc83d0d170e006827d4ba07170e004087be4050170e00d0eb642b61170e00602ef37c9f170e000044b62a5b170e00906c02ccf0170e0080575d33f9170e00808b63c9c4210e001049312779210e00c083453f80210e00e872ec4d86210e00706ba01edb300e004818713a88300e0030d59b7f82300e00a899da47d5300e00603d9950d0300e0040592a5070310e005044e1666b310e0088b56b4775310e0080e2228e35300e0070e274363c300e00804f61c741300e0030fce990be310e00580088e0c2310e00d84b1e3ec7310e0018f55928f3320e0078d3559df8320e0078ef5743cc310e00d057886f09320e00983e45870d320e00f04658d212320e0008a1cfcc00330e0010b180a3c7440e00e0582fa4cd440e00009325a710450e0078dee45818450e0060a1109ad0470e0068ea77de5d450e0040133e9394470e00b85547e1dc470e00e034421e8f","encoding":"hex","key":"diver_reef_fish_abundance_biomass_qc.parquet/timestamp=1009843200/polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0/ep_m1_public_data.csv-0.parquet"}],"sampleObjects":["timestamp=978307200/polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0/ep_m1_public_data.csv-0.parquet","timestamp=978307200/polygon=01030000000100000005000000000000000040604000000000000049C00000000000C0624000000000000049C00000000000C062400000000000003EC000000000004060400000000000003EC0000000000040604000000000000049C0/ep_m1_public_data.csv-0.parquet","timestamp=978307200/polygon=01030000000100000005000000000000000080614000000000000044C0000000000000644000000000000044C0000000000000644000000000000034C0000000000080614000000000000034C0000000000080614000000000000044C0/ep_m1_public_data.csv-0.parquet","timestamp=1735689600/polygon=01030000000100000005000000000000000000000000000000000044C0000000000000344000000000000044C0000000000000344000000000000034C0000000000000000000000000000034C0000000000000000000000000000044C0/ep_m1_public_data.csv-0.parquet","timestamp=1735689600/polygon=010300000001000000050000000000000000000000000000000000494000000000000034400000000000004940000000000000344000000000008051400000000000000000000000000080514000000000000000000000000000004940/ep_m1_public_data.csv-0.parquet"],"sampleRows":[{"latitude":-35.59,"method":1,"biomass":378.4196,"realm":"Temperate Australasia","ecoregion":"Western Bassian","phylum":"Chordata","timestamp":1735689600,"survey_id":923405656,"survey_longitude":null,"size_class":30,"reporting_name":"Pseudogoniistius nigripes","class":"Actinopterygii","country":"Australia","depth":4.2,"survey_date":"2025-06-20","visibility":"7.0","order":"Perciformes","survey_latitude":null,"family":"Cheilodactylidae","location":"Encounter","hour":"11:30:00","site_code":"GSV17","area":"South Australia","species_name":"Pseudogoniistius nigripes","block":1,"total":1,"program":"RLS","longitude":138.61,"site_name":"The Bluff","TIME":"2025-06-20 11:30:00"},{"program":"RLS","TIME":"2025-06-20 11:30:00","species_name":"Pseudogoniistius nigripes","timestamp":1735689600,"biomass":1054.0834,"class":"Actinopterygii","method":1,"ecoregion":"Western Bassian","longitude":138.61,"reporting_name":"Pseudogoniistius nigripes","size_class":35,"family":"Cheilodactylidae","survey_date":"2025-06-20","survey_longitude":null,"hour":"11:30:00","block":1,"order":"Perciformes","latitude":-35.59,"area":"South Australia","visibility":"7.0","realm":"Temperate Australasia","survey_latitude":null,"location":"Encounter","country":"Australia","depth":4.2,"total":2,"phylum":"Chordata","site_name":"The Bluff","survey_id":923405656,"site_code":"GSV17"},{"class":"Actinopterygii","site_name":"Pullen Island","depth":5.3,"block":2,"area":"South Australia","latitude":-35.54,"country":"Australia","longitude":138.69,"realm":"Temperate Australasia","TIME":"2026-05-29 10:30:00","species_name":"Scorpis aequipinnis","visibility":"3.0","program":"RLS","survey_id":923406718,"reporting_name":"Scorpis aequipinnis","survey_date":"2026-05-29","hour":"10:30:00","family":"Scorpididae","location":"Encounter","site_code":"COO18","biomass":76.6714,"ecoregion":"Western Bassian","total":3,"size_class":10,"order":"Perciformes","phylum":"Chordata","survey_longitude":null,"timestamp":1767225600,"method":1,"survey_latitude":null}],"sourceObjectKey":"diver_reef_fish_abundance_biomass_qc.parquet/timestamp=1009843200/polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0/ep_m1_public_data.csv-0.parquet","starterCell":"# South Australian gulfs + Fleurieu (lat -36.5..-32, lon 135..139.5) reef fish census since the March 2025 bloom\nimport time, pyarrow.dataset as ds\nt0 = time.time()\nimport pyarrow.fs as pafs\ntry:   # ~440 small files: anonymous S3 discovers them in seconds; the FUSE mount took 4-5 minutes for the same call in a real pod (2026-10-04)\n    dset = ds.dataset('aodn-cloud-optimised/diver_reef_fish_abundance_biomass_qc.parquet', filesystem=pafs.S3FileSystem(anonymous=True, region='ap-southeast-2'), format='parquet', partitioning='hive')\nexcept Exception:\n    dset = ds.dataset('/mnt/archive', format='parquet', partitioning='hive')\nflt = ((ds.field('timestamp') >= 1735689600) & (ds.field('latitude') >= -36.5) & (ds.field('latitude') <= -32.0)\n       & (ds.field('longitude') >= 135.0) & (ds.field('longitude') <= 139.5) & (ds.field('survey_date') >= '2025-03-01'))\nsa = dset.to_table(filter=flt, columns=['survey_id','site_code','site_name','location','survey_date','depth','species_name','size_class','total','biomass']).to_pandas()\nprint(f'{len(sa)} rows, {sa.survey_id.nunique()} surveys, {sa.site_code.nunique()} sites, {sa.survey_date.min()} to {sa.survey_date.max()}, {time.time()-t0:.1f} s')\nprint(sa.groupby(sa['survey_date'].str[:7])['survey_id'].nunique().rename('surveys_per_month'))\nprint(sa.groupby('species_name')['total'].sum().sort_values(ascending=False).head(8))","starterSamplePath":"timestamp=1009843200/polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0/ep_m1_public_data.csv-0.parquet","totalObjects":"439 objects, 24,471,206 bytes (24.5 MB) from aws s3 ls --recursive --summarize on 2026-10-04 (438 parquet + _common_metadata; all files last modified 2026-10-04, the store is rewritten in place)","verifiedRead":{"summary":"Parquet 0 rows, 33 columns, 0 row groups; columns ['TIME:timestamp[ns]', 'area:string', 'geom:string', 'hour:string', 'block:int32', 'class:string', 'depth:double', 'order:string', 'realm:string', 'total:int32']","facts":["under 'diver_reef_fish_abundance_biomass_qc.parquet/': 35 directories (e.g. timestamp=1009843200, timestamp=1041379200, timestamp=1072915200, timestamp=1104537600); 1 objects named like _common_metadata, 19 KB to 19 KB","under 'diver_reef_fish_abundance_biomass_qc.parquet/timestamp=978307200/': 3 directories (e.g. polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0, polygon=01030000000100000005000000000000000040604000000000000049C00000000000C0624000000000000049C00000000000C062400000000000003EC000000000004060400000000000003EC0000000000040604000000000000049C0, polygon=01030000000100000005000000000000000080614000000000000044C0000000000000644000000000000044C0000000000000644000000000000034C0000000000080614000000000000034C0000000000080614000000000000044C0)","under 'diver_reef_fish_abundance_biomass_qc.parquet/timestamp=978307200/polygon=01030000000100000005000000000000000080614000000000000044C0000000000000644000000000000044C0000000000000644000000000000034C0000000000080614000000000000034C0000000000080614000000000000044C0/': 1 objects named like ep_m#_public_data.csv-#.parquet, 60 KB to 60 KB"],"attempts":[{"ok":true,"family":"parquet","path":"_common_metadata","seconds":0.67}],"library":"pyarrow","starterCell":{"ok":true,"seconds":5.43},"verifiedAt":"2026-10-04T21:31:36.770Z","image":"531775725781.dkr.ecr.us-east-1.amazonaws.com/scigantic-notebook-genomics:latest","seconds":0.67,"rssDeltaMb":0.3,"path":"_common_metadata","bytes":19219,"snippet":"import os\nimport pyarrow.parquet as pq\npf = pq.ParquetFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], '_common_metadata'))\nprint(pf.metadata)\ndf = pf.read_row_group(0).to_pandas()\ndf.head()","family":"parquet"}},"summary":"The IMOS National Reef Monitoring Network (NRMN) sub-facility brings together shallow reef surveys conducted around Australia into a centralised database, collating, cleaning and…","title":"National Reef Monitoring Network - Global reef fish abundance and biomass","updatedAt":"2026-10-07T13:38:12.864Z","visibility":"public","websiteUrl":"https://catalogue-imos.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/b273fafa-03d6-4fc2-9acf-39d8c06581e5"}}