{"success":true,"data":{"author":"AODN","bucketAccessType":"public","bucketListable":true,"bucketName":"aodn-cloud-optimised","bucketPrefix":"diver_mobile_macroinvertebrate_abundance_qc.parquet","category":"Life Sciences","cloudProvider":"aws","createdAt":"2026-10-04T00:00:00.000Z","description":"The IMOS National Reef Monitoring Network (NRMN) sub-facility brings together shallow reef surveys conducted around Australia into a centralised database, collating, cleaning and publishing data from Reef Life Survey (RLS), the Australian Temperate Reef Collaboration (ATRC), Parks Australia, DBCA (WA), DEW (SA), DPI (NSW), Tasmanian Parks and Wildlife and Parks Victoria. RLS and ATRC coordinate surveys of rocky and coral reefs using standard underwater visual census methods, carried out by marine scientists and trained recreational divers, to establish national marine baselines and support the management of Commonwealth and State marine reserves. This dataset contains records of mobile macroinvertebrates collected by RLS and ATRC divers along 50 m transects: abundance for all species recorded within quantitative survey limits (50 x 1 m swathes either side of the transect line, each a 'Block'), with divers searching the reef surface including cracks for sea stars, urchins, gastropods, lobsters and crabs. Recorded concurrently with the cryptobenthic fish observations, these make up the 'Method 2' component; typically one block per transect for ATRC and two for RLS. This store is the quality-controlled M2 invertebrate table as hive-partitioned parquet: 314,801 rows from 36,081 surveys at 4,853 sites, 1992-03-10 to 2026-07-19, 34,861 of them South Australian (abalone Haliotis rubra and the urchin Heliocidaris erythrogramma lead the species list).","displayTitle":"National Reef Monitoring Network - Global mobile macroinvertebrate abundance","fuseMountBrowsable":true,"id":"90c67c2e-600c-5f2f-b7da-7cc5ecafdc90","license":"http://creativecommons.org/licenses/by/4.0/","objectCountEstimate":35,"probeStatus":"ok","region":"ap-southeast-2","schemaCard":{"accessHints":[{"snippet":"# Task: layout check without a recursive walk\naws s3 ls s3://aodn-cloud-optimised/diver_mobile_macroinvertebrate_abundance_qc.parquet/ --no-sign-request --region ap-southeast-2 | tail -4\naws s3 ls s3://aodn-cloud-optimised/diver_mobile_macroinvertebrate_abundance_qc.parquet/timestamp=1767225600/ --no-sign-request --region ap-southeast-2","language":"bash"},{"snippet":"# Task: schema + per-column attributes + GeoNetwork uuid from _common_metadata (one small read)\nimport ast, pyarrow.parquet as pq\nschema = pq.read_schema('/mnt/archive/_common_metadata')\nprint(schema.names)\nprint(ast.literal_eval(schema.metadata[b'global_attributes'].decode()))\nprint(ast.literal_eval(schema.metadata[b'size_class'].decode()))","language":"python-local"},{"snippet":"# Task: one year (2025) of invertebrate counts, column-pruned; 33 files, 0.57 MB, 15,255 rows\nimport pandas as pd\ndf = pd.read_parquet('/mnt/archive/timestamp=1735689600', columns=['survey_id','site_code','area','location','survey_date','block','species_name','size_class','total'])\nprint(len(df), df['survey_id'].nunique(), df['area'].value_counts().head(5))","language":"python-local"},{"language":"python-local","snippet":"# Task: SA gulfs since 2025-03: per-site urchin and abalone totals (block-summed), the first thing a bloom analyst asks for\nimport pyarrow.dataset as ds\ndset = ds.dataset('/mnt/archive', format='parquet', partitioning='hive')\nflt = ((ds.field('timestamp') >= 1735689600) & (ds.field('latitude') >= -36.5) & (ds.field('latitude') <= -32.0)\n       & (ds.field('longitude') >= 135.0) & (ds.field('longitude') <= 139.5) & (ds.field('survey_date') >= '2025-03-01'))\nsa = dset.to_table(filter=flt, columns=['survey_id','site_code','site_name','survey_date','species_name','total']).to_pandas()\ntop = sa.groupby('species_name')['total'].sum().sort_values(ascending=False)\nprint(len(sa), 'rows', sa['survey_id'].nunique(), 'surveys'); print(top.head(10))"},{"language":"python-anonymous","snippet":"# Task: same SA subset from outside the pod over anonymous S3 with pyarrow's S3 client\nimport pyarrow.dataset as ds, pyarrow.fs as pafs\ns3 = pafs.S3FileSystem(anonymous=True, region='ap-southeast-2')\ndset = ds.dataset('aodn-cloud-optimised/diver_mobile_macroinvertebrate_abundance_qc.parquet', filesystem=s3, format='parquet', partitioning='hive')\nflt = ((ds.field('timestamp') >= 1735689600) & (ds.field('latitude') >= -36.5) & (ds.field('latitude') <= -32.0)\n       & (ds.field('longitude') >= 135.0) & (ds.field('longitude') <= 139.5) & (ds.field('survey_date') >= '2025-03-01'))\nsa = dset.to_table(filter=flt, columns=['survey_id','site_code','survey_date','species_name','total']).to_pandas()\nprint(len(sa), sa['survey_id'].nunique(), sa['survey_date'].min(), sa['survey_date'].max())"},{"snippet":"# Task: long-term Encounter (Fleurieu) abalone + urchin series, all years, column-pruned (all 437 files, minutes)\nimport pyarrow.dataset as ds, pyarrow.fs as pafs\ns3 = pafs.S3FileSystem(anonymous=True, region='ap-southeast-2')\ndset = ds.dataset('aodn-cloud-optimised/diver_mobile_macroinvertebrate_abundance_qc.parquet', filesystem=s3, format='parquet', partitioning='hive')\nflt = (ds.field('location') == 'Encounter') & ((ds.field('species_name') == 'Haliotis rubra') | (ds.field('species_name') == 'Heliocidaris erythrogramma'))\ndf = dset.to_table(filter=flt, columns=['survey_id','site_code','survey_date','species_name','total']).to_pandas()\nper_survey = df.groupby(['survey_date','survey_id','species_name'])['total'].sum().reset_index()\nprint(per_survey.groupby([per_survey['survey_date'].str[:4],'species_name'])['total'].mean().unstack().tail(8))","language":"python-anonymous"}],"caveats":["Scale (mount): pyarrow.dataset over /mnt/archive must stat all ~440 files before the first row; measured 244-310 s in a real pod on 2026-10-04 versus 7-20 s through pyarrow's own anonymous S3 client (pafs.S3FileSystem(anonymous=True, region='ap-southeast-2'), no s3fs needed). Prefer the S3 form for whole-dataset filters; the mount is fine for a single known file.","Quirk: hive partition timestamp=<epoch seconds> is 1 January 00:00 UTC of the survey year (35 yearly partitions, 1992 to 2026; every partition value checked). Partition year equals the survey_date year on 100% of rows. Prune with ds.field('timestamp') >= 1735689600 for 2025 onward.","Quirk: the second hive key polygon=<WKB hex> is a 20 x 20 degree cell (93 distinct cells). Cells sit on a 10 degree stride and OVERLAP, so a lat/lon does not map to a unique cell by arithmetic; filter on latitude/longitude and let the engine prune on timestamp. Every SA-gulfs row of this table lives in the one cell lon 120..140, lat -40..-20 (hex starts 0103000000010000000500000000000000005E40...).","Quirk: timestamp and polygon are NOT stored inside the 437 files (all 437 footers read, one identical schema, no partition columns). They only exist as path keys, so a single-file read with pq.read_table(<file>) has no timestamp/polygon column. _common_metadata lists them (timestamp int64), but hive reads return timestamp as int32.","DO NOT: call ds.dataset(...).count_rows() or expect a _metadata file; there is none (root holds only _common_metadata, 19265 bytes, no row count). count_rows fetched every footer and took 396 s on the fish table from outside AWS. Row counts in this card come from reading all footers (77.71 s with 12 threads).","Quirk: geom is a WKB hex POINT with SRID 4326 whose coordinate order is (latitude, longitude), i.e. x=lat, y=lon, on 100% of distinct points, the reverse of the survey list table (which is x=lon, y=lat). Use the latitude/longitude columns instead.","Quirk: hour is documented as local time, but 8% of Australian rows carry 21:00 to 05:59 values (survey list: 1,300 of 16,000 non-null Australian hours) while all Spanish rows are 08:00 to 19:59; some Australian entries look like UTC. TIME is simply survey_date + hour (naive datetime, no tz), so filter on survey_date, not TIME.","Quirk: the schema-level key/value metadata in _common_metadata (one entry per column plus global_attributes with the GeoNetwork metadata_uuid) is a Python dict repr with single quotes, not JSON: json.loads fails, ast.literal_eval(value.decode()) works. The per-file footers carry no key/value metadata at all (0 of the files), so read _common_metadata for it.","Quirk: area is the state/territory or sub-national region ('South Australia'); location is the named region within it ('Encounter', 'Yorke Peninsula', 'Adelaide', 'Kangaroo Island', 'Eyre Peninsula', 'SA - Western', 'Upper Spencer Gulf', 'Upper South East (Robe)'). The brief's bbox (lat -36.5..-32, lon 135..139.5) misses the SA rows from Eyre Peninsula, SA - Western and Robe; use area == 'South Australia' for all of SA, and the bbox for the gulfs plus Fleurieu.","Scale: 314,801 rows is 188 MB Arrow / 217 MB pandas with every column (polygon hex and geom dominate); a full read is 291.69 s with pyarrow S3FileSystem because all 437 files are fetched; one year is 33 files in 13.64 s.","DO NOT: use biomass; it is a string column that is null on every row (0 of 314,801 non-null). Invertebrate biomass is not published.","Quirk: size_class is 0.0 on 76% of rows (240,556): invertebrates are mostly counted, not sized. Sum total across size_class per species rather than dropping zero-size rows as you would for fish.","Quirk: 207 rows are the placeholder species_name 'No species found' with total 0 (a transect block with no invertebrates). Keep them if you need presence/absence by block; drop them for species tables.","Departure from canonical (block effort): ATRC surveys contribute one 50 x 1 m block per transect (mean 1.02 blocks), RLS two (mean 1.94). Normalise by blocks counted (50 m2 each), not by transect, when mixing programs.","Quirk (South Australia, measured 2026-10-04): 34,861 rows (11.07%) have area == 'South Australia' from 3,097 surveys at 296 sites, 2004-02-05 to 2026-05-29; 31,351 are inside the gulfs bbox. Since 2025-03-01: 871 rows, 73 surveys, 26 sites, all RLS, dates 2025-03-06, 2025-03-07, 2025-03-11, 2025-06-17, 2025-06-18, 2025-06-19, 2025-06-20, 2025-06-21, 2025-11-13, 2025-11-14, 2025-11-15, 2025-11-26, 2025-12-18, 2026-05-14, 2026-05-29. 2026 so far: 8 surveys (2026-05-14 and 2026-05-29). 4 of the 871 rows are 'No species found' placeholders.","Departure from canonical (release lag): the survey list holds 321 SA surveys since 2025-03-01 (254 in Jan-May 2026) and 320 of them list method 2, but only 73 have invertebrate rows here and none of the January-April 2026 surveys do (2026-05: 44 listed / 8 with rows). The 2026 post-bloom Gulf St Vincent campaign is surveyed but not yet published in this table as of 2026-10-04.","Quirk: dtype drift across the NRMN tables: here latitude/longitude/depth are float32 and survey_latitude/survey_longitude/biomass/visibility are strings, while the fish table uses double and float. 544 rows duplicate (survey_id, method, block, species_name, size_class); aggregate with sum."],"columns":[{"name":"TIME","type":"timestamp[ns]. timestamp[ns], naive (no tz). Equals survey_date + hour where hour is present, survey_date at 00:00 where hour is null (checked on all rows of the survey list). Not a UTC instant: see the hour caveat."},{"type":"string. string. State/territory or sub-national region, e.g. \"South Australia\", \"Tasmania\", \"Catalonia\". 153 distinct values. This is the column to filter SA on.","name":"area"},{"name":"geom","type":"string. string. WKB hex of a POINT with SRID 4326 (prefix 0101000020E6100000). AXIS ORDER IS (latitude, longitude) in this table, the reverse of the survey list table and of the WKB convention. Redundant with latitude/longitude; drop it."},{"name":"hour","type":"string. string \"HH:MM:SS\", documented as local time of sample collection. Null on 32% (fish) to 29% (off-transect) of rows."},{"type":"int32. int32. 50 x 1 m swathe: 1 (197,141 rows), 2 (117,658), 0 (2 rows). RLS surveys average 1.94 blocks, ATRC 1.02, matching the abstract (ATRC does one block).","name":"block"},{"name":"class","type":"string. string. Gastropoda 99,153 rows, Echinoidea 70,842, Asteroidea 47,633, Malacostraca 39,666, Crinoidea 27,916, Holothuroidea 11,926, Bivalvia 11,656, Ascidiacea 2,275 ... 18 classes; null on 295 rows."},{"type":"float. float32, metres, 0 to 42.","name":"depth"},{"type":"string. string. Taxonomic order.","name":"order"},{"name":"realm","type":"string. string. Spalding MEOW realm, e.g. \"Temperate Australasia\". 12 distinct."},{"name":"total","type":"int32. int32. Individuals counted; 0 to 22,000. 228 zero rows, 207 of them the placeholder species_name \"No species found\"."},{"type":"string. string. Taxonomic family.","name":"family"},{"type":"int32. int32. Always 2 (M2 invertebrate/cryptic search) on all 314,801 rows.","name":"method"},{"name":"phylum","type":"string. string. Echinodermata, Mollusca, Arthropoda, Chordata (ascidians), Platyhelminthes, Cnidaria, Echiura; null on 218 rows."},{"name":"biomass","type":"string. string in this table and NULL on all 314,801 rows. No invertebrate biomass is published; drop the column."},{"type":"string. string. 59 distinct; \"Australia\" is 81% of fish rows.","name":"country"},{"name":"polygon","type":"string. string, hive partition key (not stored in the files). WKB hex of the 20 x 20 degree spatial cell the row is partitioned into."},{"type":"string. string. Contributing program: RLS, ATRC, Parks Vic, FRDC, RRH.","name":"program"},{"name":"filename","type":"string. string. Source CSV name; a single constant value per table."},{"name":"latitude","type":"float. float32, degrees_north. Site latitude (float32 here, double in the fish table)."},{"type":"string. string. Named region within the area, e.g. \"Encounter\", \"Yorke Peninsula\", \"Adelaide\", \"Kangaroo Island\". 221 distinct.","name":"location"},{"name":"ecoregion","type":"string. string. 105 distinct. SA rows: 'South Australian Gulfs' 27,391, 'Western Bassian' 6,162, 'Great Australian Bight' 1,308."},{"name":"longitude","type":"float. float32, degrees_east. Site longitude."},{"name":"site_code","type":"string. string. Site identifier, e.g. \"GSV135\", \"COO18\"; join key to the survey list and the other observation tables."},{"type":"string. string. Human site name, e.g. \"Pt Noarlunga inside\".","name":"site_name"},{"type":"int32. int32. Transect-level survey identifier (1,003,253 to 923,407,054); primary join key, unique in the survey list table.","name":"survey_id"},{"name":"timestamp","type":"int64. int64 per _common_metadata, read back as int32 by the hive partitioning. Partition key = epoch seconds of 1 January UTC of the survey year."},{"name":"size_class","type":"float. float, cm. 0.0 on 240,556 rows (76%): most invertebrates are not sized. Sized rows use the RLS bins 2.5 ... 750."},{"type":"string. string, metres (e.g. \"8.0\"); null on 56,275 rows.","name":"visibility"},{"name":"survey_date","type":"string. string \"YYYY-MM-DD\", local date. Use this, not TIME, for date filtering."},{"name":"species_name","type":"string. string. 2,668 distinct; Haliotis rubra 22,530 rows, Heliocidaris erythrogramma 15,139, Cenolia trichoptera 14,866, Centrostephanus rodgersii 13,522, Jasus edwardsii 7,149. Includes the placeholder \"No species found\"."},{"name":"reporting_name","type":"string. string. Name used for reporting (usually equals species_name)."},{"type":"string. string (not numeric) and null on 296,104 rows (94%). Cast with pd.to_numeric if needed; prefer latitude.","name":"survey_latitude"},{"name":"survey_longitude","type":"string. string, as survey_latitude."}],"format":"v2-hybrid","generatedAt":"2026-10-04T00:00:00Z","layout":{"pattern":"Hive-partitioned parquet, depth 3: timestamp=<epoch seconds, 1 Jan UTC of survey year>/polygon=<WKB hex of a 20x20 degree cell>/ep_m2_inverts_public_data.csv-0.parquet. Root: 35 timestamp= prefixes (1992..2026) plus _common_metadata (19265 B); no _metadata. 437 data files, 1 to 41 per year, 8.6 KB to 86 KB (median 16 KB). Partition columns are not stored in the files."},"mountPath":"/mnt/archive","overview":"Reef Life Survey / ATRC Method 2 mobile macroinvertebrate census (the 'M2 inverts' block counts of the RLS protocol) published by IMOS NRMN, quality-controlled, in AODN's cloud-optimised parquet layout. One row per species x size class x block on a 50 m transect, with count, taxonomy and site. Measured 2026-10-04: 314,801 rows, 36,081 survey_ids, 4,853 sites, 59 countries, 105 ecoregions, 1992-03-10 to 2026-07-19; RLS 209,690 rows, ATRC 89,184, Parks Vic 13,793. Echinodermata 158,418 rows, Mollusca 113,014, Arthropoda 39,741. Layout is 437 files (9.3 MB, median 16 KB) under timestamp=<year epoch>/polygon=<20 degree WKB cell>/, so it wants a year range plus a lat/lon box: the 2025 partition is 33 files, 0.57 MB, 15,255 rows in 13.64 s via pandas/s3fs; the SA-gulfs 2025-2026 pushdown returned 871 rows in 1.05 s after a 8.45 s discovery. The whole table is small enough to hold (188 MB Arrow, 217 MB pandas) but still costs 291.69 s because every one of the 437 files is fetched.","rowCount":"314,801 rows (sum of all 437 footers, 2026-10-04); 2025 partition 15,255 rows, 2026 partition 4,386 rows","sampleHeaders":[{"key":"diver_mobile_macroinvertebrate_abundance_qc.parquet/timestamp=1009843200/polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0/ep_m2_inverts_public_data.csv-0.parquet","truncatedAt":512,"encoding":"hex","bytes":"50415231150415f00a15fa0a4c15ae011500120000b805f4b70200c8756b8922060e00c0c970e11e060e00e8deaf744d040e009037858ed4050e00a03733e6cd050e007079ae129d040e00c0713ee051040e0028c445a8f4040e00e0037cdba2040e00a0a2c620f0040e008814b6ce6f060e00e0b3f5d47d060e00e8707cc96a060e00206749d378110e0070e10e1b38110e002877cc50c9110e00188c153ace110e00c0f58b977f110e00288082b488110e00204dc83d0d170e006827d4ba07170e004087be4050170e00d0eb642b61170e00602ef37c9f170e000044b62a5b170e00906c02ccf0170e0080575d33f9170e00808b63c9c4210e001049312779210e00c083453f80210e00e872ec4d86210e00706ba01edb300e004818713a88300e0030d59b7f82300e00a899da47d5300e00603d9950d0300e0040592a5070310e005044e1666b310e0088b56b4775310e0080e2228e35300e0070e274363c300e00804f61c741300e0030fce990be310e00580088e0c2310e00d84b1e3ec7310e0018f55928f3320e0078d3559df8320e0078ef5743cc310e00d057886f09320e00983e45870d320e00f04658d212320e0008a1cfcc00330e0010b180a3c7440e00e0582fa4cd440e00009325a710450e0078dee45818450e0060a1109ad0470e0068ea77de5d450e0040133e9394470e00b85547e1dc470e00e034421e8f"}],"sampleObjects":["timestamp=978307200/polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0/ep_m2_inverts_public_data.csv-0.parquet","timestamp=978307200/polygon=01030000000100000005000000000000000040604000000000000049C00000000000C0624000000000000049C00000000000C062400000000000003EC000000000004060400000000000003EC0000000000040604000000000000049C0/ep_m2_inverts_public_data.csv-0.parquet","timestamp=978307200/polygon=01030000000100000005000000000000000080614000000000000044C0000000000000644000000000000044C0000000000000644000000000000034C0000000000080614000000000000034C0000000000080614000000000000044C0/ep_m2_inverts_public_data.csv-0.parquet","timestamp=1735689600/polygon=01030000000100000005000000000000000000000000000000000044C0000000000000344000000000000044C0000000000000344000000000000034C0000000000000000000000000000034C0000000000000000000000000000044C0/ep_m2_inverts_public_data.csv-0.parquet","timestamp=1735689600/polygon=010300000001000000050000000000000000000000000000000000494000000000000034400000000000004940000000000000344000000000008051400000000000000000000000000080514000000000000000000000000000004940/ep_m2_inverts_public_data.csv-0.parquet"],"sampleRows":[{"TIME":"2025-06-20 11:30:00","survey_latitude":null,"biomass":null,"total":3,"survey_id":923405656,"program":"RLS","class":"Malacostraca","block":1,"depth":4.2,"survey_longitude":null,"size_class":7.5,"country":"Australia","latitude":-35.59,"site_code":"GSV17","longitude":138.61,"order":"Decapoda","visibility":"7.0","ecoregion":"Western Bassian","phylum":"Arthropoda","realm":"Temperate Australasia","method":2,"species_name":"Jasus edwardsii","reporting_name":"Jasus edwardsii","hour":"11:30:00","location":"Encounter","survey_date":"2025-06-20","site_name":"The Bluff","area":"South Australia","family":"Palinuridae","timestamp":1735689600},{"class":"Malacostraca","hour":"11:30:00","family":"Palinuridae","method":2,"area":"South Australia","survey_longitude":null,"site_name":"The Bluff","latitude":-35.59,"depth":4.2,"total":1,"phylum":"Arthropoda","realm":"Temperate Australasia","location":"Encounter","longitude":138.61,"site_code":"GSV17","visibility":"7.0","timestamp":1735689600,"survey_date":"2025-06-20","block":1,"program":"RLS","size_class":10,"survey_latitude":null,"order":"Decapoda","TIME":"2025-06-20 11:30:00","ecoregion":"Western Bassian","country":"Australia","biomass":null,"reporting_name":"Jasus edwardsii","species_name":"Jasus edwardsii","survey_id":923405656},{"latitude":-35.54,"site_code":"COO18","method":2,"species_name":"Guinusia chabrus","survey_date":"2026-05-29","order":"Decapoda","hour":"10:30:00","size_class":0,"survey_latitude":null,"timestamp":1767225600,"ecoregion":"Western Bassian","total":3,"block":1,"survey_longitude":null,"reporting_name":"Guinusia chabrus","class":"Malacostraca","program":"RLS","survey_id":923406692,"site_name":"Pullen Island","biomass":null,"country":"Australia","location":"Encounter","TIME":"2026-05-29 10:30:00","depth":5.1,"longitude":138.69,"visibility":"3.0","family":"Plagusiidae","phylum":"Arthropoda","area":"South Australia","realm":"Temperate Australasia"}],"sourceObjectKey":"diver_mobile_macroinvertebrate_abundance_qc.parquet/timestamp=1009843200/polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0/ep_m2_inverts_public_data.csv-0.parquet","starterCell":"# SA gulfs + Fleurieu mobile invertebrates since the March 2025 bloom: surveys per month and top species\nimport time, pyarrow.dataset as ds\nt0 = time.time()\nimport pyarrow.fs as pafs\ntry:   # ~440 small files: anonymous S3 discovers them in seconds; the FUSE mount took 4-5 minutes for the same call in a real pod (2026-10-04)\n    dset = ds.dataset('aodn-cloud-optimised/diver_mobile_macroinvertebrate_abundance_qc.parquet', filesystem=pafs.S3FileSystem(anonymous=True, region='ap-southeast-2'), format='parquet', partitioning='hive')\nexcept Exception:\n    dset = ds.dataset('/mnt/archive', format='parquet', partitioning='hive')\nflt = ((ds.field('timestamp') >= 1735689600) & (ds.field('latitude') >= -36.5) & (ds.field('latitude') <= -32.0)\n       & (ds.field('longitude') >= 135.0) & (ds.field('longitude') <= 139.5) & (ds.field('survey_date') >= '2025-03-01'))\nsa = dset.to_table(filter=flt, columns=['survey_id','site_code','site_name','location','survey_date','block','species_name','size_class','total']).to_pandas()\nsa = sa[sa['species_name'] != 'No species found']\nprint(f'{len(sa)} rows, {sa.survey_id.nunique()} surveys, {sa.site_code.nunique()} sites, {sa.survey_date.min()} to {sa.survey_date.max()}, {time.time()-t0:.1f} s')\nprint(sa.groupby(sa['survey_date'].str[:7])['survey_id'].nunique().rename('surveys_per_month'))\nprint(sa.groupby('species_name')['total'].sum().sort_values(ascending=False).head(8))","starterSamplePath":"timestamp=1009843200/polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0/ep_m2_inverts_public_data.csv-0.parquet","totalObjects":"438 objects, 9,265,408 bytes (9.3 MB) from aws s3 ls --recursive --summarize on 2026-10-04","verifiedRead":{"family":"parquet","path":"_common_metadata","image":"531775725781.dkr.ecr.us-east-1.amazonaws.com/scigantic-notebook-genomics:latest","snippet":"import os\nimport pyarrow.parquet as pq\npf = pq.ParquetFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], '_common_metadata'))\nprint(pf.metadata)\ndf = pf.read_row_group(0).to_pandas()\ndf.head()","seconds":0.68,"summary":"Parquet 0 rows, 33 columns, 0 row groups; columns ['TIME:timestamp[ns]', 'area:string', 'geom:string', 'hour:string', 'block:int32', 'class:string', 'depth:float', 'order:string', 'realm:string', 'total:int32']","bytes":19265,"facts":["under 'diver_mobile_macroinvertebrate_abundance_qc.parquet/': 35 directories (e.g. timestamp=1009843200, timestamp=1041379200, timestamp=1072915200, timestamp=1104537600); 1 objects named like _common_metadata, 19 KB to 19 KB","under 'diver_mobile_macroinvertebrate_abundance_qc.parquet/timestamp=978307200/': 3 directories (e.g. polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0, polygon=01030000000100000005000000000000000040604000000000000049C00000000000C0624000000000000049C00000000000C062400000000000003EC000000000004060400000000000003EC0000000000040604000000000000049C0, polygon=01030000000100000005000000000000000080614000000000000044C0000000000000644000000000000044C0000000000000644000000000000034C0000000000080614000000000000034C0000000000080614000000000000044C0)","under 'diver_mobile_macroinvertebrate_abundance_qc.parquet/timestamp=978307200/polygon=01030000000100000005000000000000000080614000000000000044C0000000000000644000000000000044C0000000000000644000000000000034C0000000000080614000000000000034C0000000000080614000000000000044C0/': 1 objects named like ep_m#_inverts_public_data.csv-#.parquet, 15 KB to 15 KB"],"rssDeltaMb":0.2,"verifiedAt":"2026-10-04T21:31:35.875Z","attempts":[{"path":"_common_metadata","seconds":0.68,"ok":true,"family":"parquet"}],"library":"pyarrow","starterCell":{"seconds":5.23,"ok":true}}},"summary":"The IMOS National Reef Monitoring Network (NRMN) sub-facility brings together shallow reef surveys conducted around Australia into a centralised database, collating, cleaning and…","title":"National Reef Monitoring Network - Global mobile macroinvertebrate abundance","updatedAt":"2026-10-07T13:22:09.745Z","visibility":"public","websiteUrl":"https://catalogue-imos.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/48cf3cb9-caa9-4633-9baa-8bba3c4d904a"}}