{"success":true,"data":{"author":"AODN","bucketAccessType":"public","bucketListable":true,"bucketName":"aodn-cloud-optimised","bucketPrefix":"diver_cryptobenthic_fish_abundance_qc.parquet","category":"Life Sciences","cloudProvider":"aws","createdAt":"2026-10-04T00:00:00.000Z","description":"The IMOS National Reef Monitoring Network (NRMN) sub-facility brings together shallow reef surveys conducted around Australia into a centralised database, collating, cleaning and publishing data from Reef Life Survey (RLS), the Australian Temperate Reef Collaboration (ATRC), Parks Australia, DBCA (WA), DEW (SA), DPI (NSW), Tasmanian Parks and Wildlife and Parks Victoria. RLS and ATRC coordinate surveys of rocky and coral reefs using standard underwater visual census methods, carried out by marine scientists and trained recreational divers, to establish national marine baselines and support the management of Commonwealth and State marine reserves. This dataset contains records of cryptobenthic fishes collected by RLS and ATRC divers along 50 m transects: abundance for all species recorded within quantitative survey limits (50 x 1 m swathes either side of the transect line, each a 'Block'), with divers searching the reef surface including cracks for hidden fishes. These observations are recorded concurrently with the macroinvertebrate observations and together make up the 'Method 2' component; typically one block per transect for ATRC and two for RLS. It supersedes the RLS-specific 'Cryptic Fish' collection. This store is the quality-controlled M2 cryptic-fish table as hive-partitioned parquet: 168,994 rows from 23,239 surveys at 4,483 sites, 1992-03-10 to 2026-07-19, with size class and biomass on most rows; 7,587 rows are South Australian.","displayTitle":"National Reef Monitoring Network - Global cryptobenthic fish abundance","fuseMountBrowsable":true,"id":"c6438623-f358-59f3-affe-d0f485d57c26","license":"http://creativecommons.org/licenses/by/4.0/","objectCountEstimate":35,"probeStatus":"ok","region":"ap-southeast-2","schemaCard":{"accessHints":[{"snippet":"# Task: layout check without a recursive walk\naws s3 ls s3://aodn-cloud-optimised/diver_cryptobenthic_fish_abundance_qc.parquet/ --no-sign-request --region ap-southeast-2 | head -3\naws s3 ls s3://aodn-cloud-optimised/diver_cryptobenthic_fish_abundance_qc.parquet/timestamp=1767225600/ --no-sign-request --region ap-southeast-2","language":"bash"},{"language":"python-local","snippet":"# Task: schema and GeoNetwork identity from _common_metadata. NOTE the embedded title is wrong (says off-transect); trust the uuid\nimport ast, pyarrow.parquet as pq\nschema = pq.read_schema('/mnt/archive/_common_metadata')\nprint(schema.names)\nprint(ast.literal_eval(schema.metadata[b'global_attributes'].decode()))"},{"language":"python-local","snippet":"# Task: one year (2025), column-pruned; 33 files, 0.56 MB, 13,587 rows\nimport pandas as pd\ndf = pd.read_parquet('/mnt/archive/timestamp=1735689600', columns=['survey_id','site_code','area','survey_date','block','species_name','size_class','total','biomass'])\nprint(len(df), df['survey_id'].nunique(), df['biomass'].isna().mean().round(3))"},{"language":"python-local","snippet":"# Task: SA gulfs since 2025-03, cryptic fish counts per survey, pushdown on year + lat/lon\nimport pyarrow.dataset as ds\ndset = ds.dataset('/mnt/archive', format='parquet', partitioning='hive')\nflt = ((ds.field('timestamp') >= 1735689600) & (ds.field('latitude') >= -36.5) & (ds.field('latitude') <= -32.0)\n       & (ds.field('longitude') >= 135.0) & (ds.field('longitude') <= 139.5) & (ds.field('survey_date') >= '2025-03-01'))\nsa = dset.to_table(filter=flt, columns=['survey_id','site_code','site_name','survey_date','species_name','total','biomass']).to_pandas()\nprint(len(sa), 'rows', sa['survey_id'].nunique(), 'surveys', sa['site_code'].nunique(), 'sites')\nprint(sa.groupby('species_name')['total'].sum().sort_values(ascending=False).head(8))"},{"language":"python-anonymous","snippet":"# Task: same SA subset over anonymous S3 from outside the pod\nimport pyarrow.dataset as ds, pyarrow.fs as pafs\ns3 = pafs.S3FileSystem(anonymous=True, region='ap-southeast-2')\ndset = ds.dataset('aodn-cloud-optimised/diver_cryptobenthic_fish_abundance_qc.parquet', filesystem=s3, format='parquet', partitioning='hive')\nflt = ((ds.field('timestamp') >= 1735689600) & (ds.field('latitude') >= -36.5) & (ds.field('latitude') <= -32.0)\n       & (ds.field('longitude') >= 135.0) & (ds.field('longitude') <= 139.5) & (ds.field('survey_date') >= '2025-03-01'))\nsa = dset.to_table(filter=flt, columns=['survey_id','site_code','survey_date','species_name','total']).to_pandas()\nprint(len(sa), sa['survey_id'].nunique(), sa['survey_date'].max())"},{"language":"python-anonymous","snippet":"# Task: combine the two M2 halves (cryptic fish + invertebrates) for one SA survey year via survey_id and block\nimport pyarrow.dataset as ds, pyarrow.fs as pafs\ns3 = pafs.S3FileSystem(anonymous=True, region='ap-southeast-2')\nflt = (ds.field('timestamp') == 1735689600) & (ds.field('area') == 'South Australia')\ncols = ['survey_id','site_code','survey_date','block','species_name','total']\ncf = ds.dataset('aodn-cloud-optimised/diver_cryptobenthic_fish_abundance_qc.parquet', filesystem=s3, format='parquet', partitioning='hive').to_table(filter=flt, columns=cols).to_pandas()\ninv = ds.dataset('aodn-cloud-optimised/diver_mobile_macroinvertebrate_abundance_qc.parquet', filesystem=s3, format='parquet', partitioning='hive').to_table(filter=flt, columns=cols).to_pandas()\nprint('cryptic surveys', cf['survey_id'].nunique(), 'invert surveys', inv['survey_id'].nunique(), 'in both', len(set(cf['survey_id']) & set(inv['survey_id'])))"}],"caveats":["Scale (mount): pyarrow.dataset over /mnt/archive must stat all ~440 files before the first row; measured 244-310 s in a real pod on 2026-10-04 versus 7-20 s through pyarrow's own anonymous S3 client (pafs.S3FileSystem(anonymous=True, region='ap-southeast-2'), no s3fs needed). Prefer the S3 form for whole-dataset filters; the mount is fine for a single known file.","Quirk: hive partition timestamp=<epoch seconds> is 1 January 00:00 UTC of the survey year (35 yearly partitions, 1992 to 2026; every partition value checked). Partition year equals the survey_date year on 100% of rows. Prune with ds.field('timestamp') >= 1735689600 for 2025 onward.","Quirk: the second hive key polygon=<WKB hex> is a 20 x 20 degree cell (93 distinct cells). Cells sit on a 10 degree stride and OVERLAP, so a lat/lon does not map to a unique cell by arithmetic; filter on latitude/longitude and let the engine prune on timestamp. Every SA-gulfs row of this table lives in the one cell lon 120..140, lat -40..-20 (hex starts 0103000000010000000500000000000000005E40...).","Quirk: timestamp and polygon are NOT stored inside the 435 files (all 435 footers read, one identical schema, no partition columns). They only exist as path keys, so a single-file read with pq.read_table(<file>) has no timestamp/polygon column. _common_metadata lists them (timestamp int64), but hive reads return timestamp as int32.","DO NOT: call ds.dataset(...).count_rows() or expect a _metadata file; there is none (root holds only _common_metadata, 19294 bytes, no row count). count_rows fetched every footer and took 396 s on the fish table from outside AWS. Row counts in this card come from reading all footers (60.12 s with 12 threads).","Quirk: geom is a WKB hex POINT with SRID 4326 whose coordinate order is (latitude, longitude), i.e. x=lat, y=lon, on 100% of distinct points, the reverse of the survey list table (which is x=lon, y=lat). Use the latitude/longitude columns instead.","Quirk: hour is documented as local time, but 8% of Australian rows carry 21:00 to 05:59 values (survey list: 1,300 of 16,000 non-null Australian hours) while all Spanish rows are 08:00 to 19:59; some Australian entries look like UTC. TIME is simply survey_date + hour (naive datetime, no tz), so filter on survey_date, not TIME.","Quirk: the schema-level key/value metadata in _common_metadata (one entry per column plus global_attributes with the GeoNetwork metadata_uuid) is a Python dict repr with single quotes, not JSON: json.loads fails, ast.literal_eval(value.decode()) works. The per-file footers carry no key/value metadata at all (0 of the files), so read _common_metadata for it.","Quirk: area is the state/territory or sub-national region ('South Australia'); location is the named region within it ('Encounter', 'Yorke Peninsula', 'Adelaide', 'Kangaroo Island', 'Eyre Peninsula', 'SA - Western', 'Upper Spencer Gulf', 'Upper South East (Robe)'). The brief's bbox (lat -36.5..-32, lon 135..139.5) misses the SA rows from Eyre Peninsula, SA - Western and Robe; use area == 'South Australia' for all of SA, and the bbox for the gulfs plus Fleurieu.","DO NOT: trust the title embedded in _common_metadata global_attributes: it reads 'IMOS - National Reef Monitoring Network Sub-Facility - Global off-transect species observations', which is the sibling table's title. The uuid 9efa25cd-4da4-47b5-9385-45e3cbd11705 resolves in GeoNetwork to 'Global cryptobenthic fish abundance' and the file is ep_m2_cryptic_fish_public_data.csv. This card uses the GeoNetwork title.","Scale: 168,994 rows is 103 MB Arrow / 115 MB pandas with every column; a full read is 276.94 s with pyarrow S3FileSystem because all 435 files are fetched; one year is 33 files in 15.57 s.","Departure from canonical (sparsity): only 23,239 of the 36,536 surveys that list method 2 in the survey list have cryptic-fish rows (13,297 M2 surveys recorded none, mostly ATRC: ATRC is 11,000 rows here vs 89,184 in the invertebrate table). Absence of a survey_id here is not evidence of zero cryptic fish unless the survey list shows method 2.","Quirk: biomass is populated on 92% of rows (unlike the invertebrate table, where it is all null) and is the row total in grams for `total` individuals; size_class 0.0 (7.6% of rows) means unsized and those rows usually have null biomass.","Quirk (South Australia, measured 2026-10-04): 7,587 rows (4.49%) have area == 'South Australia' from 2,157 surveys at 264 sites, 2004-02-06 to 2026-05-29; 7,269 inside the gulfs bbox. Since 2025-03-01: 113 rows, 46 surveys, 22 sites, all RLS, dates 2025-03-06, 2025-03-07, 2025-03-11, 2025-06-17, 2025-06-18, 2025-06-19, 2025-06-20, 2025-11-13, 2025-11-14, 2025-11-15, 2025-11-26, 2025-12-18, 2026-05-14, 2026-05-29; 2026 so far 4 surveys (2026-05-14, 2026-05-29).","Departure from canonical (release lag): the survey list holds 321 SA surveys since 2025-03-01 (254 in Jan-May 2026) but only 46 have cryptic-fish rows here and none from January-April 2026 (2026-05: 44 listed / 4 with rows). The 2026 post-bloom campaign is not yet published in this table as of 2026-10-04.","Quirk: dtype drift across the NRMN tables: here visibility and survey_latitude/survey_longitude are float32 (strings in the fish, invertebrate and off-transect tables) and latitude/longitude/depth are float32 (double in the fish table). 211 rows duplicate (survey_id, method, block, species_name, size_class); aggregate with sum."],"columns":[{"name":"TIME","type":"timestamp[ns]. timestamp[ns], naive (no tz). Equals survey_date + hour where hour is present, survey_date at 00:00 where hour is null (checked on all rows of the survey list). Not a UTC instant: see the hour caveat."},{"name":"area","type":"string. string. State/territory or sub-national region, e.g. \"South Australia\", \"Tasmania\", \"Catalonia\". 153 distinct values. This is the column to filter SA on."},{"type":"string. string. WKB hex of a POINT with SRID 4326 (prefix 0101000020E6100000). AXIS ORDER IS (latitude, longitude) in this table, the reverse of the survey list table and of the WKB convention. Redundant with latitude/longitude; drop it.","name":"geom"},{"type":"string. string \"HH:MM:SS\", documented as local time of sample collection. Null on 32% (fish) to 29% (off-transect) of rows.","name":"hour"},{"type":"int32. int32. 50 x 1 m swathe: 1 (92,281 rows) or 2 (76,713).","name":"block"},{"name":"class","type":"string. string. Actinopterygii 161,741, Teleostei 4,484, Chondrichthyes 2,390, Reptilia 81, Actinopteri 80; null 218."},{"type":"float. float32, metres, 0 to 42.","name":"depth"},{"name":"order","type":"string. string. Taxonomic order."},{"type":"string. string. Spalding MEOW realm, e.g. \"Temperate Australasia\". 12 distinct.","name":"realm"},{"name":"total","type":"int32. int32. Individuals counted; 0 to 10,000; 207 zero rows."},{"name":"family","type":"string. string. Taxonomic family."},{"type":"int32. int32. Always 2 (M2) on all 168,994 rows.","name":"method"},{"type":"string. string. Chordata on all non-null rows; null on 218.","name":"phylum"},{"type":"float. float32, grams, for all `total` individuals in the row. Non-null on 155,800 rows (92%); null on 13,194 (7.8%). 0.0007 g to 327,733 g.","name":"biomass"},{"name":"country","type":"string. string. 59 distinct; \"Australia\" is 81% of fish rows."},{"type":"string. string, hive partition key (not stored in the files). WKB hex of the 20 x 20 degree spatial cell the row is partitioned into.","name":"polygon"},{"name":"program","type":"string. string. Contributing program: RLS, ATRC, Parks Vic, FRDC, RRH."},{"name":"filename","type":"string. string. Source CSV name; a single constant value per table."},{"name":"latitude","type":"float. float32, degrees_north. Site latitude."},{"type":"string. string. Named region within the area, e.g. \"Encounter\", \"Yorke Peninsula\", \"Adelaide\", \"Kangaroo Island\". 221 distinct.","name":"location"},{"type":"string. string. 101 distinct. SA rows: 'South Australian Gulfs' 7,101, 'Western Bassian' 337, 'Great Australian Bight' 149.","name":"ecoregion"},{"name":"longitude","type":"float. float32, degrees_east. Site longitude."},{"name":"site_code","type":"string. string. Site identifier, e.g. \"GSV135\", \"COO18\"; join key to the survey list and the other observation tables."},{"name":"site_name","type":"string. string. Human site name, e.g. \"Pt Noarlunga inside\"."},{"name":"survey_id","type":"int32. int32. Transect-level survey identifier (1,003,253 to 923,407,054); primary join key, unique in the survey list table."},{"name":"timestamp","type":"int64. int64 per _common_metadata, read back as int32 by the hive partitioning. Partition key = epoch seconds of 1 January UTC of the survey year."},{"name":"size_class","type":"float. float32, cm. RLS bins 1, 2.5, 5 ... 400; 0.0 on 12,868 rows (7.6%) = not sized."},{"name":"visibility","type":"float. float32, metres (numeric in this table, string in fish/inverts/off-transect); null on 16,043 rows."},{"type":"string. string \"YYYY-MM-DD\", local date. Use this, not TIME, for date filtering.","name":"survey_date"},{"name":"species_name","type":"string. string. 1,565 distinct; Hypoplectrodes maccullochi 6,527 rows, Ostorhinchus limenus 3,066, Cheilodipterus quinquelineatus 3,044, Eviota teresae 2,772, Scorpaena jacksoniensis 2,769."},{"type":"string. string. Name used for reporting (usually equals species_name).","name":"reporting_name"},{"type":"float. float32, null on 159,287 rows (94%).","name":"survey_latitude"},{"name":"survey_longitude","type":"float. float32, as survey_latitude."}],"format":"v2-hybrid","generatedAt":"2026-10-04T00:00:00Z","layout":{"pattern":"Hive-partitioned parquet, depth 3: timestamp=<epoch seconds, 1 Jan UTC of survey year>/polygon=<WKB hex of a 20x20 degree cell>/ep_m2_cryptic_fish_public_data.csv-0.parquet. Root: 35 timestamp= prefixes (1992..2026) plus _common_metadata (19294 B); no _metadata. 435 data files, 1 to 41 per year, 8.1 KB to 68 KB (median 13 KB). Partition columns are not stored in the files."},"mountPath":"/mnt/archive","overview":"Reef Life Survey / ATRC Method 2 cryptobenthic fish census (the 'M2 cryptic fish' half of the invertebrate search: gobies, blennies, triplefins, scorpionfish, cardinalfish) published by IMOS NRMN, quality-controlled, in AODN's cloud-optimised parquet layout. One row per species x size class x block on a 50 m transect with count, biomass (g) and taxonomy. Measured 2026-10-04: 168,994 rows, 23,239 survey_ids (fewer than the other tables: 13,297 M2 surveys recorded no cryptic fish), 4,483 sites, 59 countries, 101 ecoregions, 1992-03-10 to 2026-07-19; RLS 157,381 rows (93%), ATRC 11,000. Layout is 435 files (7.6 MB, median 13 KB) under timestamp=<year epoch>/polygon=<20 degree WKB cell>/: the 2025 partition is 33 files, 0.56 MB, 13,587 rows in 15.57 s via pandas/s3fs; the SA-gulfs 2025-2026 pushdown returned 113 rows in 1.48 s after a 8.14 s discovery; the whole table is 103 MB Arrow / 115 MB pandas and took 276.94 s because all 435 files are fetched. Year-plus-box is the access direction; the per-row cost is trivial.","rowCount":"168,994 rows (sum of all 435 footers, 2026-10-04); 2025 partition 13,587 rows, 2026 partition 2,355 rows","sampleHeaders":[{"bytes":"50415231150415b00215b8024c152615001200009801f09700c8756b8922060e00e8deaf744d040e00a03733e6cd050e007079ae129d040e00c0713ee051040e0028c445a8f4040e00e0037cdba2040e006827d4ba07170e00906c02ccf0170e0080575d33f9170e0030d59b7f82300e0018f55928f3320e00d057886f09320e00e034421e8f470e00e85e074c88470e00883ae034e4620e00106a4b9811640e002885ddcf04650e0038b6840ed0630e15001538153c2c15341510150615061c1808002885ddcf04650e180800e8deaf744d040e16002808002885ddcf04650e180800e8deaf744d040e11110000001c6c0200000034010509200821c418a414731042492dd6dc83510200000015041518151c4c150215001200000c2c08000000566963746f7269611500151215162c15341510150615061c36002808566963746f7269611808566963746f72696111110000000920020000003401013400150415d40c15ac044c151e1500120000aa063c320000003031303130303030323045360509803032393543384643324635323834334330443741333730334430413137363234305636000036150104323601362c383545423531423831453135663600044134095a0c443732330136197a043134663600195c04323501361946043133a63600966c0019fe043537366c0000336a6c001992003405d8194600326a","truncatedAt":512,"encoding":"hex","key":"diver_cryptobenthic_fish_abundance_qc.parquet/timestamp=1009843200/polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0/ep_m2_cryptic_fish_public_data.csv-0.parquet"}],"sampleObjects":["timestamp=978307200/polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0/ep_m2_cryptic_fish_public_data.csv-0.parquet","timestamp=978307200/polygon=01030000000100000005000000000000000040604000000000000049C00000000000C0624000000000000049C00000000000C062400000000000003EC000000000004060400000000000003EC0000000000040604000000000000049C0/ep_m2_cryptic_fish_public_data.csv-0.parquet","timestamp=978307200/polygon=01030000000100000005000000000000000080614000000000000044C0000000000000644000000000000044C0000000000000644000000000000034C0000000000080614000000000000034C0000000000080614000000000000044C0/ep_m2_cryptic_fish_public_data.csv-0.parquet","timestamp=1735689600/polygon=01030000000100000005000000000000000000000000000000000044C0000000000000344000000000000044C0000000000000344000000000000034C0000000000000000000000000000034C0000000000000000000000000000044C0/ep_m2_cryptic_fish_public_data.csv-0.parquet","timestamp=1735689600/polygon=010300000001000000050000000000000000000000000000000000494000000000000034400000000000004940000000000000344000000000008051400000000000000000000000000080514000000000000000000000000000004940/ep_m2_cryptic_fish_public_data.csv-0.parquet"],"sampleRows":[{"phylum":"Chordata","survey_longitude":null,"order":"Perciformes","program":"RLS","longitude":138.61,"block":2,"biomass":3.0896,"size_class":5,"site_name":"The Bluff","TIME":"2025-06-20 11:30:00","method":2,"visibility":7,"hour":"11:30:00","site_code":"GSV17","latitude":-35.59,"class":"Actinopterygii","survey_id":923405656,"species_name":"Pempheris klunzingeri","area":"South Australia","ecoregion":"Western Bassian","depth":4.2,"realm":"Temperate Australasia","survey_date":"2025-06-20","survey_latitude":null,"country":"Australia","reporting_name":"Pempheris klunzingeri","location":"Encounter","family":"Pempheridae","total":1,"timestamp":1735689600},{"class":"Actinopterygii","phylum":"Chordata","country":"Australia","family":"Pempheridae","longitude":138.61,"depth":5.1,"realm":"Temperate Australasia","ecoregion":"Western Bassian","survey_id":923405657,"biomass":9.2687,"total":3,"area":"South Australia","latitude":-35.59,"timestamp":1735689600,"visibility":6,"survey_date":"2025-06-20","TIME":"2025-06-20 11:30:00","location":"Encounter","site_name":"The Bluff","order":"Perciformes","method":2,"hour":"11:30:00","survey_longitude":null,"species_name":"Pempheris multiradiata","site_code":"GSV17","reporting_name":"Pempheris multiradiata","survey_latitude":null,"size_class":5,"block":1,"program":"RLS"},{"depth":5.1,"ecoregion":"Western Bassian","species_name":"Trinorfolkia cristata","timestamp":1767225600,"method":2,"survey_latitude":null,"latitude":-35.54,"location":"Encounter","size_class":5,"survey_id":923406692,"country":"Australia","site_code":"COO18","survey_date":"2026-05-29","TIME":"2026-05-29 10:30:00","realm":"Temperate Australasia","reporting_name":"Trinorfolkia cristata","survey_longitude":null,"site_name":"Pullen Island","phylum":"Chordata","block":2,"biomass":1.4317,"class":"Actinopterygii","total":1,"visibility":3,"family":"Tripterygiidae","longitude":138.69,"hour":"10:30:00","program":"RLS","area":"South Australia","order":"Perciformes"}],"sourceObjectKey":"diver_cryptobenthic_fish_abundance_qc.parquet/timestamp=1009843200/polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0/ep_m2_cryptic_fish_public_data.csv-0.parquet","starterCell":"# SA gulfs + Fleurieu cryptobenthic fishes since the March 2025 bloom\nimport time, pyarrow.dataset as ds\nt0 = time.time()\nimport pyarrow.fs as pafs\ntry:   # ~440 small files: anonymous S3 discovers them in seconds; the FUSE mount took 4-5 minutes for the same call in a real pod (2026-10-04)\n    dset = ds.dataset('aodn-cloud-optimised/diver_cryptobenthic_fish_abundance_qc.parquet', filesystem=pafs.S3FileSystem(anonymous=True, region='ap-southeast-2'), format='parquet', partitioning='hive')\nexcept Exception:\n    dset = ds.dataset('/mnt/archive', format='parquet', partitioning='hive')\nflt = ((ds.field('timestamp') >= 1735689600) & (ds.field('latitude') >= -36.5) & (ds.field('latitude') <= -32.0)\n       & (ds.field('longitude') >= 135.0) & (ds.field('longitude') <= 139.5) & (ds.field('survey_date') >= '2025-03-01'))\nsa = dset.to_table(filter=flt, columns=['survey_id','site_code','site_name','location','survey_date','block','species_name','size_class','total','biomass']).to_pandas()\nprint(f'{len(sa)} rows, {sa.survey_id.nunique()} surveys, {sa.site_code.nunique()} sites, {sa.survey_date.min()} to {sa.survey_date.max()}, {time.time()-t0:.1f} s')\nprint(sa.groupby(sa['survey_date'].str[:7])['survey_id'].nunique().rename('surveys_per_month'))\nprint(sa.groupby('species_name')['total'].sum().sort_values(ascending=False).head(8))","starterSamplePath":"timestamp=1009843200/polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0/ep_m2_cryptic_fish_public_data.csv-0.parquet","totalObjects":"436 objects, 7,576,126 bytes (7.6 MB) from aws s3 ls --recursive --summarize on 2026-10-04","verifiedRead":{"bytes":19294,"family":"parquet","snippet":"import os\nimport pyarrow.parquet as pq\npf = pq.ParquetFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], '_common_metadata'))\nprint(pf.metadata)\ndf = pf.read_row_group(0).to_pandas()\ndf.head()","seconds":0.67,"rssDeltaMb":0,"facts":["under 'diver_cryptobenthic_fish_abundance_qc.parquet/': 35 directories (e.g. timestamp=1009843200, timestamp=1041379200, timestamp=1072915200, timestamp=1104537600); 1 objects named like _common_metadata, 19 KB to 19 KB","under 'diver_cryptobenthic_fish_abundance_qc.parquet/timestamp=978307200/': 3 directories (e.g. polygon=01030000000100000005000000000000000040604000000000000044C00000000000C0624000000000000044C00000000000C0624000000000000034C0000000000040604000000000000034C0000000000040604000000000000044C0, polygon=01030000000100000005000000000000000040604000000000000049C00000000000C0624000000000000049C00000000000C062400000000000003EC000000000004060400000000000003EC0000000000040604000000000000049C0, polygon=01030000000100000005000000000000000080614000000000000044C0000000000000644000000000000044C0000000000000644000000000000034C0000000000080614000000000000034C0000000000080614000000000000044C0)","under 'diver_cryptobenthic_fish_abundance_qc.parquet/timestamp=978307200/polygon=01030000000100000005000000000000000080614000000000000044C0000000000000644000000000000044C0000000000000644000000000000034C0000000000080614000000000000034C0000000000080614000000000000044C0/': 1 objects named like ep_m#_cryptic_fish_public_data.csv-#.parquet, 13 KB to 13 KB"],"verifiedAt":"2026-10-05T12:59:09.819Z","image":"531775725781.dkr.ecr.us-east-1.amazonaws.com/scigantic-notebook-genomics:latest","library":"pyarrow","path":"_common_metadata","starterCell":{"seconds":5.17,"ok":true},"attempts":[{"path":"_common_metadata","seconds":0.67,"ok":true,"family":"parquet"}],"summary":"Parquet 0 rows, 33 columns, 0 row groups; columns ['TIME:timestamp[ns]', 'area:string', 'geom:string', 'hour:string', 'block:int32', 'class:string', 'depth:float', 'order:string', 'realm:string', 'total:int32']"}},"summary":"The IMOS National Reef Monitoring Network (NRMN) sub-facility brings together shallow reef surveys conducted around Australia into a centralised database, collating, cleaning and…","title":"National Reef Monitoring Network - Global cryptobenthic fish abundance","updatedAt":"2026-10-07T13:13:35.417Z","visibility":"public","websiteUrl":"https://catalogue-imos.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/9efa25cd-4da4-47b5-9385-45e3cbd11705"}}