{"success":true,"data":{"author":"Government of South Australia (Department for Environment and Water; PIRSA/SARDI; EPA), mirrored by Scigantic","bucketAccessType":"public","bucketListable":true,"bucketName":"scigantic-sa-open-data","bucketPrefix":"hab-monitoring","category":"Environmental Science","cloudProvider":"aws","createdAt":"2026-10-04T00:00:00.000Z","description":"A weekly, dated mirror of the South Australian Government's Harmful Algal Bloom Water Testing program, the public data behind the SA Algal Bloom Water Testing Open Data Dashboard. Since the Karenia mikimotoi bloom was identified off the Fleurieu Peninsula in March 2025, SARDI (offshore), the EPA (onshore) and ad hoc sampling have counted phytoplankton cells per litre at hundreds of sites: the 21 Adelaide metropolitan beaches and jetties sampled every week (Largs Bay, Grange, Henley, Glenelg, Brighton, Port River and West Lakes among them), plus Gulf St Vincent, Spencer Gulf, the Yorke and Eyre peninsulas, the Fleurieu, Kangaroo Island, the Coorong and the South East, publishing the results only through an ArcGIS feature service with no CSV and no history. Search terms people use: SA algae, algal bloom Adelaide beaches, Karenia mikimotoi, red tide, harmful algal bloom, cells per litre. This archive holds a full parquet copy of that service (94,155 result rows across 858 sites, 2025-03-18 onward, about 650 new rows a week, 623 distinct taxon and summary labels including eight spellings of Karenia), re-pulled every Monday with a snapshot kept for every pull, plus a manifest with upstream edit dates and checksums. The whole dataset is under 1 MB and reads in seconds. No licence is stated by the publisher; the data is Crown copyright, Government of South Australia, mirrored for research use with attribution.","displayTitle":"South Australian Algal Bloom Water Testing - Phytoplankton cell counts - Weekly mirror of the SA Government feature service","fuseMountBrowsable":true,"id":"4459582a-9ed2-59fa-b381-0e2433c46eca","license":"Not stated by the publisher; Crown copyright, Government of South Australia. Mirrored for research use with attribution; no redistribution licence granted.","objectCountEstimate":1,"probeStatus":"ok","referenceUrl":"https://experience.arcgis.com/experience/5f0d6b22301a47bf91d198cabb030670","region":"us-east-1","schemaCard":{"accessHints":[{"snippet":"# Task: see what is here and when it was last pulled (public bucket, no credentials)\naws s3 ls --no-sign-request --region us-east-1 s3://scigantic-sa-open-data/hab-monitoring/ --recursive\naws s3 cp --no-sign-request --region us-east-1 s3://scigantic-sa-open-data/hab-monitoring/MANIFEST.json - | head -40","language":"bash"},{"language":"python-local","snippet":"# Task: Karenia counts at the 21 recurring metropolitan beach sites, one row per site per week (the dashboard's main table)\nimport pandas as pd\ns = pd.read_parquet('/mnt/archive/samples.parquet')\nsites = pd.read_parquet('/mnt/archive/sites.parquet')\nmetro = sites[sites.Region == 'Metropolitan']\nk = s[s.Result_Name.str.contains('Karenia', na=False) & (s.Units == 'Cells/L') & s.Site_Number.isin(metro.SiteNumber)]\nk = k.merge(metro[['SiteNumber', 'SiteName']], left_on='Site_Number', right_on='SiteNumber')\nk['week'] = pd.to_datetime(k.Date_Sample_Collected).dt.to_period('W').dt.start_time\nprint(k.pivot_table(index='week', columns='SiteName', values='Result_Value_Numeric', aggfunc='max').tail(12).to_string())"},{"snippet":"# Task: map the latest count per site (sites.parquet already carries Current_CountCellsL + lon/lat)\nimport pandas as pd, folium\nsites = pd.read_parquet('/mnt/archive/sites.parquet')\nm = folium.Map(location=[-34.9, 137.8], zoom_start=7, tiles='CartoDB positron')\nfor r in sites.dropna(subset=['lon', 'lat']).itertuples():\n    c = r.Current_CountCellsL or 0\n    folium.CircleMarker([r.lat, r.lon], radius=3 + (0 if c <= 0 else min(12, 2 * __import__('math').log10(c))), color='red' if c >= 100000 else 'orange' if c > 0 else 'green', fill=True,\n                        popup=f'{r.SiteName}: {c:,.0f} cells/L on {str(r.Current_Date_)[:10]}').add_to(m)\nm","language":"python-local"},{"language":"python-local","snippet":"# Task: everything detected in one sample, not just Karenia (a 'Full scan' site)\nimport pandas as pd\ns = pd.read_parquet('/mnt/archive/samples.parquet')\none = s[(s.Site_Description == 'Largs Bay Jetty')].sort_values('Date_Sample_Collected')\nlast = one[one.Date_Sample_Collected == one.Date_Sample_Collected.max()]\nprint(last[['Species_Group', 'Result_Name', 'Result_Label']].to_string(index=False))"},{"snippet":"# Task: read the mirror from anywhere, no credentials\nimport pandas as pd\nso = {'anon': True, 'client_kwargs': {'region_name': 'us-east-1'}}\ns = pd.read_parquet('s3://scigantic-sa-open-data/hab-monitoring/samples.parquet', storage_options=so)\nsites = pd.read_parquet('s3://scigantic-sa-open-data/hab-monitoring/sites.parquet', storage_options=so)\nprint(len(s), s.Date_Sample_Collected.max())","language":"python-anonymous"},{"language":"python-anonymous","snippet":"# Task: compare two weekly snapshots to see what the publisher added or changed\nimport pandas as pd\nso = {'anon': True, 'client_kwargs': {'region_name': 'us-east-1'}}\nbase = 's3://scigantic-sa-open-data/hab-monitoring/snapshots/'\na = pd.read_parquet(base + '2026-10-04/samples.parquet', storage_options=so)\nb = pd.read_parquet(base + '<later date>/samples.parquet', storage_options=so)\nnew = b[~b.OBJECTID.isin(a.OBJECTID)]\nprint(len(new), 'new rows;', new.Date_Sample_Collected.min(), 'to', new.Date_Sample_Collected.max())"},{"language":"python-anonymous","snippet":"# Task: the live upstream service, if you need rows newer than the last pull (paged, 2,000 per request, ~48 pages for the whole table)\nimport json, urllib.request, urllib.parse\nB = 'https://services6.arcgis.com/WS2XycMNFieWAsfS/arcgis/rest/services/HarmfulAlgalBloom_MonitoringSites/FeatureServer/1/query?'\nq = urllib.parse.urlencode({'where': \"Date_Sample_Collected >= '2026-09-20'\", 'outFields': '*', 'orderByFields': 'OBJECTID', 'resultRecordCount': 2000, 'f': 'json'})\nd = json.load(urllib.request.urlopen(B + q))\nprint(len(d['features']), 'rows; more pages:', d.get('exceededTransferLimit', False))"}],"caveats":["Departure from canonical: this is a MIRROR, not the publisher's own data product. Upstream is the SA Government's ArcGIS Feature Service behind the Water Testing Open Data Dashboard; there is no official CSV, no API documentation and no history. The mirror re-pulls the full service every Monday 20:30 UTC and MANIFEST.json records pulledAt, the upstream dataLastEditDate (2026-09-28 at the first pull) and every pull's row counts. Check MANIFEST.json before saying what the latest date is.","Licence: NO licence is stated on the ArcGIS item or on algalbloom.sa.gov.au. The data is Crown copyright, Government of South Australia (Department for Environment and Water, PIRSA/SARDI, EPA), published for public use; no redistribution licence has been granted to Scigantic. Attribute 'South Australian Government Algal Bloom Monitoring Program' and do not describe it as open-licensed.","Quirk: one row per sample per RESULT, not per sample. A site-date has a median of 1 row because most sites are 'Karenia counts' only, but 'Full scan' sites carry dozens of taxon rows per sample plus summary rows ('Results Summary', 'Potentially Harmful Species', 'Total Phytoplankton > 5 micrometres', 'Organism - Notes'). Always filter Result_Name before aggregating.","Quirk: Karenia is spelled eight ways in Result_Name: 'Karenia sp.' (4,751), 'Karenia spp.' (4,572), 'Karenia mikimotoi' (1,348), 'Karenia cf. longicanalis' (862), 'Karenia papilionacea' (472), 'Karenia mikimotoi (ant)' (181), 'Karenia mikimotoi (post)' (174), 'Karenia cf. mikimotoi (ant)' (64). Use str.contains('Karenia') for 'any Karenia'; the (ant)/(post) suffixes are antapical/posterior morphology notes from the same count, not extra cells.","Quirk: 13% of rows have Result_Value_Numeric null. 5,164 have Units null and carry text in Result_Value_String (analyst notes); the rest are 'Detected in sample but not observed during cell counts' (6,829) and 'Detected' (2,223) in Result_Label, i.e. presence without a count. Filter Units == 'Cells/L' for numeric work and treat the detected-only rows as presence.","Scale: the bloom peak in this table is 23,000,000 Karenia cells/L (Port River Dock One, 2025-09-08), with the Port River and West Lakes sites holding the top five counts (9.5 to 23 million). Metropolitan beach peaks were 1.4 million in March 2025. By September 2026 the weekly all-site peak was under 6,000 cells/L. A log axis is the only way to plot this series.","Quirk: Site_Number joins sites.SiteNumber for 99.9% of rows, but 27 Karenia rows sit at sites absent from the sites layer, and the samples table has 873 distinct Site_Number values against 858 sites. Use a left join and keep Site_Description as the fallback name. Sites also use two numbering ranges: 1000001.. for program sites, 1xxxx for ad hoc sites.","Quirk: Species_Group is null on 10% of rows and uses 'Dinoflagellate' (singular, 59 rows) next to 'Dinoflagellates' (43,092). Sampling_Type has one value with a trailing space. Monitoring_program mixes 'Regional - X' and 'Ad hoc - X' for the same region; strip the prefix if you want per-region totals.","Quirk: Date_Sample_Collected and Date_Sample_Received are real dates (date32); Date_Analysis and Date_Report are strings upstream and stay strings (often null). Current_Date_ in sites.parquet is a UTC timestamp converted from upstream epoch milliseconds; it is the date of the site's latest count, not the pull date.","DO NOT: rely on OBJECTID as a permanent key across snapshots. It is system-maintained upstream; a publisher reload could renumber. For snapshot diffs it is fine within the same numbering, but match on (Site_Number, Date_Sample_Collected, Result_Name, Test_Name) if OBJECTID ranges jump.","Scale: both files read anonymously in 3.0 s and 1.0 s from outside AWS (0.87 MB + 0.05 MB); samples.parquet is 15 MB in pandas. Nothing here needs filtering at read time. The snapshots/ prefix grows by about 0.92 MB per week.","Related hosted sources for the same bloom: the IMOS Biogeochemical database (Gulf St Vincent Karenia counts from the SARDI/NRS program, 2025-07-03 and 2025-10-17), the AODN AusTemp marine heatwave stores, the AODN chlorophyll-a stores, and the estuarine and coastal water-quality moorings (SA buoys from 2025-12-18)."],"columns":[{"name":"samples.OBJECTID","type":"int64; upstream row id, 1..94155, unique; system-maintained upstream so it may change if the publisher reloads the table"},{"name":"samples.Site_Number","type":"int64; joins sites.SiteNumber (99.9% resolve; 27 Karenia rows at sites not in the sites layer)"},{"name":"samples.Site_Description","type":"string; site name as written on the lab sheet (e.g. 'Mundoo Barrage Upstream'); prefer sites.SiteName after the join"},{"name":"samples.Date_Sample_Collected","type":"date32; 2025-03-18 .. 2026-09-24; the analysis axis"},{"name":"samples.Date_Sample_Received","type":"date32; lab receipt date"},{"type":"string (upstream type is string); often null","name":"samples.Date_Analysis"},{"name":"samples.Date_Report","type":"string 'YYYY-MM-DD' or null; latest 2026-09-25"},{"name":"samples.Monitoring_program","type":"string; Metropolitan 41,816 | Regional - Yorke Peninsula 14,171 | Regional - Eyre Peninsula 10,654 | Regional - Southern Fleurieu 8,232 | Ad hoc - Coorong 5,472 | Regional - Kangaroo Island 4,922 | Offshore 2,773 | Regional - South East 1,850 | Ad hoc - offshore 1,591 | others"},{"type":"string; lab batch file name e.g. '00104054_S-154_1_amended.csv' (973 distinct); groups the results of one lab run, not a test type","name":"samples.Test_Name"},{"name":"samples.Species_Group","type":"string or null (10% null); Dinoflagellates 43,092 | Diatoms 19,631 | Other 12,072 | Cyanoprokaryota | Cryptophytes | Prymnesiophytes | Prasinophytes | Chlorophytes | Chrysophytes | Euglenophyta; 'Dinoflagellate' (singular) on 59 rows"},{"type":"string; taxon or summary label, 623 distinct: 'Karenia sp.' 4,751, 'Karenia spp.' 4,572, 'Karenia mikimotoi' 1,348, 'Karenia cf. longicanalis' 862, 'Karenia papilionacea' 472, 'Karenia mikimotoi (ant)' / '(post)' / 'cf. mikimotoi (ant)' 419; also 'Results Summary', 'Potentially Harmful Species', 'Total Phytoplankton > 5 micrometres', 'Organism - Notes'","name":"samples.Result_Name"},{"name":"samples.Result_Value_String","type":"string or null; free text for non-numeric results (analyst notes such as 'The identification for this organism is tentative only.')"},{"name":"samples.Result_Value_Numeric","type":"double or null (13% null); cells per litre when Units == 'Cells/L'"},{"name":"samples.Units","type":"string; 'Cells/L' on 88,991 rows, null on 5,164 (the text-only rows)"},{"name":"samples.Result_Label","type":"string; the dashboard's display text: '0 Cells/L' 48,525 | 'Detected in sample but not observed during cell counts' 6,829 | '500 Cells/L' | '1,000 Cells/L' | 'Detected' 2,223 ..."},{"type":"int64","name":"sites.OBJECTID"},{"type":"string; e.g. 'Largs Bay Jetty', 'Port River Dock One', 'Grange Jetty'","name":"sites.SiteName"},{"name":"sites.SiteNumber","type":"int64; join key (1000001.. for program sites, 1xxxx for ad hoc)"},{"name":"sites.Longitude / sites.Latitude","type":"double; upstream attribute copies of the point"},{"type":"double; the feature geometry in EPSG:4326 (added by the mirror); 133.60..140.85 E, -38.06..-32.13","name":"sites.lon / sites.lat"},{"type":"string; Ad hoc - offshore 171 | Ad hoc - metropolitan 138 | Offshore 124 | Ad hoc - Yorke Peninsula 104 | Ad hoc - Southern Fleurieu 73 | Ad hoc - Eyre Peninsula 59 | ... | Metropolitan 21 (the recurring beach sites)","name":"sites.Region"},{"type":"string; 'Karenia counts' 664 | 'Quick scan (presence of potentially harmful species)' 113 (one with a trailing space) | 'Karenia counts (through the SA Shellfish Quality Assurance Program)' 52 | 'Full scan (presence of all species)' 18","name":"sites.Sampling_Type"},{"name":"sites.Status","type":"string; 'Sampled' 857, 'Not yet sampled' 1"},{"type":"string or null","name":"sites.Location_Notes"},{"name":"sites.Current_CountCellsL","type":"double; the site's most recent Karenia count as shown on the dashboard"},{"type":"timestamp[ms, UTC]; date of that most recent count (upstream epoch milliseconds)","name":"sites.Current_Date_"}],"format":"v2-hybrid","generatedAt":"2026-10-04T00:00:00Z","layout":{"pattern":"/mnt/archive/  (= s3://scigantic-sa-open-data/hab-monitoring/, us-east-1, anonymous)\n  README.md          provenance, column notes, licence status (1.9 KB)\n  MANIFEST.json      pulledAt, upstream editingInfo dates, per-file rows/bytes/sha256, pull history (4 KB)\n  samples.parquet    table 1 HarmfulAlgalBloom_SampleData, full copy, zstd, 868 KB, 94,155 rows\n  sites.parquet      layer 0 HarmfulAlgalBloom_MonitoringSites, full copy, zstd, 49 KB, 858 rows\n  snapshots/<YYYY-MM-DD>/samples.parquet, sites.parquet   one pair per weekly pull, never deleted (first: 2026-10-04)"},"mountPath":"/mnt/archive","overview":"Scigantic's weekly mirror of the South Australian Government's Harmful Algal Bloom Water Testing program (the 'SA Algal Bloom Water Testing Open Data Dashboard'): phytoplankton cell counts per litre from the SARDI offshore program, the EPA onshore program and ad hoc sampling that began with the March 2025 Karenia mikimotoi bloom in Gulf St Vincent, Spencer Gulf, the Fleurieu, Yorke and Eyre peninsulas, Kangaroo Island, the Coorong and the South East. Upstream is a public ArcGIS Feature Service with no CSV and no history; this mirror re-pulls it in full every Monday 20:30 UTC and keeps a dated snapshot of every pull. Measured 2026-10-04 from the first pull: samples.parquet 94,155 result rows x 15 columns (one row per sample per result, 2025-03-18 to 2026-09-24, 623 distinct Result_Name values, 12,555 Karenia rows), sites.parquet 858 sites x 13 columns with lon/lat (133.6..140.8E, 38.1..32.1S). Both files read anonymously in 3.0 s and 1.0 s (0.87 MB and 0.05 MB zstd); the whole dataset fits in memory, so the access direction is load both files, join on Site_Number = SiteNumber, filter Result_Name. Upstream last edited 2026-09-28; about 650 new rows a week. The bloom's measured peak: 23,000,000 Karenia cells/L at Port River Dock One on 2025-09-08; by September 2026 weekly peaks were under 6,000 cells/L.","rowCount":"samples.parquet 94,155 rows (2026-10-04 pull); sites.parquet 858 rows. Growth about 650 sample rows per week.","sampleHeaders":[{"truncatedAt":512,"encoding":"hex","key":"hab-monitoring/snapshots/2026-10-04/samples.parquet","bytes":"50415231150415b0f95b15f6df0b4c1596bf0b150012000028b52ffd8048587e0b00fc1c025c14040100020003000400050006000700080009000a000b000c000d000e000f0010001100120013001400150016001700180019001a001b001c001d001e001f0020002100220023002400250026002700280029002a002b002c002d002e002f0030003100320033003400350036003700380039003a003b003c003d003e003f0040004100420043004400450046004700480049004a004b004c004d004e004f0050005100520053005400550056005700580059005a005b005c005d005e005f0060006100620063006400650066006700680069006a006b006c006d006e006f0070007100720073007400750076007700780079007a007b007c007d007e007f0080008100820083008400850086008700880089008a008b008c008d008e008f0090009100920093009400950096009700980099009a009b009c009d009e009f00a000a100a200a300a400a500a600a700a800a900aa00ab00ac00ad00ae00af00b000b100b200b300b400b500b600b700b800b900ba00bb00bc00bd00be00bf00c000c100c200c300c400c500c600c700c800c900ca00cb00cc00cd00ce00cf00d000d100d200d300d400d500d600d700d800d900da00db00dc00dd00de00df00e000e100e200e300e400e500e600e700e800e900ea00eb00ec00"},{"bytes":"50415231150415a06b15ba134c15b40d150012000028b52ffd60d0199d2600aa45f80f2c1088cf66c30c0000000000000000000000000000c05f32d1d132c924934c32c924934c32c9242b91524a9902b200d20027011de6f086359c610c5fd8c215a6f084251c61083fd8c10d66f082159c60041fd8c0b0bfee7aebacafae7aeaa89f6e7ae9a48f2e7ae8a07fee79e70316708001ccf9e69a678ef9e596574ef9e492470ef9e38e37cef8e28a278ef8e186174ef8e082070ef8df7ef7cdf7de7ae78df7dd76d74df7dc72c70df7db6eb7bdb6da69a37db6d965933db6d86183ed75d75c6fad75d6585f6d75d5544f2d75d4503fed74d34c2fad74d2481f6d74d1447ff7de9dfb36dc6df75ada59596e2cde9d5d1ddd9c5c1cdc9b5b1bdb9a5a1ada995919d9985818d8975717d7965616d6955515d5945414d4935313d3925212d2915111d1d7dbe56edf5e3bedb3cb1efba10d5d68420f5ad08106f49ffddc673eef59cf79c6f39ded5c673acf59ce7186f39bdddc6636af59cd6946f399cd5c66328f59cc6106f397bddc652e6f59cb59c6f295ad5c652a4f59ca5186f2939ddc64262f59c94946f2918d5c64220f59c84106f28f7ddc631eef58c739c6f18d6d5c631ab378c52a4e318a4f6ce2129378c4220e31883fece1565554535251504f4e4d4c4b4a4948474645445f6f97bbf55e4b3b2b1b7bc850214283040502","truncatedAt":512,"key":"hab-monitoring/snapshots/2026-10-04/sites.parquet","encoding":"hex"}],"sampleObjects":["samples.parquet","sites.parquet","MANIFEST.json","README.md","snapshots/2026-10-04/samples.parquet"],"sampleRows":[{"Monitoring_program":"Ad hoc - Coorong","Site_Number":11884,"Site_Description":"Mundoo Barrage Upstream","Species_Group":null,"Result_Name":"Geosmin-MIB producing BGA - Total","Date_Sample_Collected":"2025-07-30","Result_Label":"256,000 Cells/L","file":"samples.parquet","Units":"Cells/L","OBJECTID":1,"Test_Name":"00104054_S-154_1_amended.csv","Result_Value_Numeric":256000},{"Date_Sample_Collected":"2025-09-08","file":"samples.parquet (highest Karenia count in the table)","SiteName (joined)":"Port River Dock One","Result_Name":"Karenia sp.","Result_Value_Numeric":23000000,"Units":"Cells/L"},{"OBJECTID":1,"Sampling_Type":"Full scan (presence of all species)","Region":"Metropolitan","file":"sites.parquet","SiteNumber":1000001,"Status":"Sampled","Current_CountCellsL":0,"lon":138.4827,"lat":-34.82387,"SiteName":"Largs Bay Jetty","Current_Date_":"2026-09-21T00:00:00Z"}],"sidecars":[{"bytes":"{\n \"bucket\": \"scigantic-sa-open-data\",\n \"region\": \"us-east-1\",\n \"prefix\": \"hab-monitoring\",\n \"pulledAt\": \"2026-10-04T18:49:19+00:00\",\n \"source\": {\n  \"service\": \"https://services6.arcgis.com/WS2XycMNFieWAsfS/arcgis/rest/services/HarmfulAlgalBloom_MonitoringSites/FeatureServer\",\n  \"itemId\": \"e4e217a2206b41b2afc3f34405c5cb3f\",\n  \"dashboard\": \"https://experience.arcgis.com/experience/5f0d6b22301a47bf91d198cabb030670\"\n },\n \"licence\": \"No licence is stated on the ArcGIS item or on algalbloom.sa.gov.au. Crown copyright, Government of South Australia (Department for Environment and Water; PIRSA/SARDI; EPA). Mirrored for research use with attribution; no redistribution licence has been granted.\",\n \"upstream\": {\n  \"serviceItemId\": \"e4e217a2206b41b2afc3f34405c5cb3f\",\n  \"layers\": {\n   \"0\": {\n    \"name\": \"HarmfulAlgalBloom_MonitoringSites\",\n    \"fields\": [\n     [\n      \"OBJECTID\",\n      \"esriFieldTypeOID\"\n     ],\n     [\n      \"SiteName\",\n      \"esriFieldTypeString\"\n     ],\n     [\n      \"SiteNumber\",\n      \"esriFieldTypeInteger\"\n     ],\n     [\n      \"Longitude\",\n      \"esriFieldTypeDouble\"\n     ],\n     [\n      \"Latitude\",\n      \"esriFieldTypeDouble\"\n     ],\n     [\n      \"Region\",\n      \"esriFieldTypeString\"\n     ],\n     [\n      \"Sampling_Type\",\n      \"esriFieldTypeString\"\n     ],\n     [\n      \"Status\",\n      \"esriFieldTypeString\"\n     ],\n     [\n      \"Location_Notes\",\n      \"esriFieldTypeString\"\n     ],\n     [\n      \"Current_CountCellsL\",\n      \"esriFieldTypeInteger\"\n     ],\n     [\n      \"Current_Date_\",\n      \"esriFieldTypeDate\"\n     ]\n    ],\n    \"dataLastEditDate\": \"2026-09-28T01:03:47.103000+00:00\",\n    \"schemaLastEditDate\": \"2026-09-28T01:03:47.103000+00:00\",\n    \"maxRecordCount\": 2000\n   },\n   \"1\": {\n    \"name\": \"HarmfulAlgalBloom_SampleData\",\n    \"fields\": [\n     [\n      \"OBJECTID\",\n      \"esriFieldTypeOID\"\n     ],\n     [\n      \"Site_Number\",\n      \"esriFieldTypeInteger\"\n     ],\n     [\n      \"Site_Description\",\n      \"esriFieldTypeString\"\n     ],\n     [\n      \"Date_Sample_Collected\",\n      \"esriFieldTypeDateOnly\"\n     ],\n     [\n      \"Date_Sample_Received\",\n      \"esriFieldTypeDateOnly\"\n     ],\n     [\n      \"Date_Analysis\",\n      \"esriFieldTypeString\"\n     ],\n     [\n      \"Date_Report\",\n      \"esriFieldTypeString\"\n     ],\n     [\n      \"Monitoring_program\",\n      \"esriFieldTypeString\"\n     ],\n     [\n      \"Test_Name\",\n      \"esriFieldTypeString\"\n     ],\n     [\n      \"Species_Group\",\n      \"esriFieldTypeString\"\n     ],\n     [\n      \"Result_Name\",\n      \"esriFieldTypeString\"\n     ],\n     [\n      \"Result_Value_String\",\n      \"esriFieldTypeString\"\n     ],\n     [\n      \"Result_Value_Numeric\",\n      \"esriFieldTypeDouble\"\n     ],\n     [\n      \"Units\",\n      \"esriFieldTypeString\"\n     ],\n     [\n      \"Result_Label\",\n      \"esriFieldTypeString\"\n     ]\n    ],\n    \"dataLastEditDate\": \"2026-09-28T01:03:47.103000+00:00\",\n    \"schemaLastEditDate\": \"2026-09-28T01:00:59.020000+00:00\",\n    \"maxRecordCount\": 2000\n   }\n  }\n },\n \"files\": {\n  \"sites.parquet\": {\n   \"layer\": 0,\n   \"rows\": 858,\n   \"columns\": [\n    \"OBJECTID\",\n    \"SiteName\",\n    \"SiteNumber\",\n    \"Longitude\",\n    \"Latitude\",\n    \"Region\",\n    \"Sampling_Type\",\n    \"Status\",\n    \"Location_Notes\",\n    \"Current_CountCellsL\",\n    \"Current_Date_\",\n    \"lon\",\n    \"lat\"\n   ],\n   \"bytes\": 48803,\n   \"sha256\": \"d6fa63b3a6e0753db24a540b11343cd3e33bec097cc0b912af42b5050ab3655d\",\n   \"pages\": 1,\n   \"pullSeconds\": 1.2\n  },\n  \"samples.parquet\": {\n   \"layer\": 1,\n   \"rows\": 94155,\n   \"columns\": [\n    \"OBJECTID\",\n    \"Site_Number\",\n    \"Site_Description\",\n    \"Date_Sample_Collected\",\n    \"Date_Sample_Received\",\n    \"Date_Analysis\",\n    \"Date_Report\",\n    \"Monitoring_program\",\n    \"Test_Name\",\n    \"Species_Group\",\n    \"Result_Name\",\n    \"Result_Value_String\",\n    \"Result_Value_Numeric\",\n    \"Units\",\n    \"Result_Label\"\n   ],\n   \"bytes\": 868385,\n   \"sha256\": \"ac91af802c45b3be887e10c7b0b86a64c2455e95b652c26be6f10066bd5a9b71\",\n   \"pages\": 48,\n   \"pullSeconds\": 105.3\n  }\n },\n \"history\": [\n  {\n   \"pulledAt\": \"2026-10-04T18:49:19+00:00\",\n   \"snapshot\": \"2026-10-04\",\n   \"rows\": {\n    \"sites.parquet\": 858,\n    \"samp","truncated":true,"key":"hab-monitoring/MANIFEST.json","name":"MANIFEST.json","encoding":"utf8","size":4129},{"encoding":"utf8","name":"README.md","key":"hab-monitoring/README.md","truncated":false,"bytes":"# South Australian Algal Bloom Water Testing: mirror\n\nA dated copy of the South Australian Government's harmful algal bloom water-testing\nresults (phytoplankton cell counts per litre from the SARDI offshore program, the EPA\nonshore program and ad hoc sampling), as published through the SA Algal Bloom Water\nTesting Open Data Dashboard:\n\n- Dashboard: https://experience.arcgis.com/experience/5f0d6b22301a47bf91d198cabb030670\n- Feature service: https://services6.arcgis.com/WS2XycMNFieWAsfS/arcgis/rest/services/HarmfulAlgalBloom_MonitoringSites/FeatureServer\n- Context: https://www.algalbloom.sa.gov.au/testing-and-monitoring\n\nMirrored by Scigantic (https://scigantic.com). Last pull 2026-10-04T18:49:19+00:00.\n\n## Files\n\n| File | What | Rows |\n|---|---|---|\n| samples.parquet | table 1 HarmfulAlgalBloom_SampleData, one row per (sample, result) | 94,155 |\n| sites.parquet | layer 0 HarmfulAlgalBloom_MonitoringSites, one row per site with lon/lat | 858 |\n| snapshots/YYYY-MM-DD/ | the same two files as pulled on that date, kept forever | |\n| MANIFEST.json | pull time, upstream edit dates, bytes, sha256, pull history | |\n\nColumns keep their upstream names. `Date_Sample_Collected` and `Date_Sample_Received`\nare dates; `Date_Analysis` and `Date_Report` are strings upstream and stay strings.\n`Result_Value_Numeric` is cells per litre when `Units` is 'Cells/L'; about 13% of rows\nhave no numeric value and carry the finding in `Result_Value_String` or `Result_Label`\n(for example 'Detected in sample but not observed during cell counts'). `Site_Number`\njoins to `sites.parquet`.`SiteNumber`.\n\n## Licence\n\nNo licence is stated on the ArcGIS item or on algalbloom.sa.gov.au. Crown copyright, Government of South Australia (Department for Environment and Water; PIRSA/SARDI; EPA). Mirrored for research use with attribution; no redistribution licence has been granted.\n","size":1868}],"sourceObjectKey":"hab-monitoring/snapshots/2026-10-04/samples.parquet","starterCell":"# SA algal bloom water testing: weekly Karenia peak and median across all sites, plus the five highest counts\nimport time, pandas as pd\nt0 = time.time()\ns = pd.read_parquet('/mnt/archive/samples.parquet')\nsites = pd.read_parquet('/mnt/archive/sites.parquet')\nk = s[s.Result_Name.str.contains('Karenia', na=False) & (s.Units == 'Cells/L')]\nk = k.merge(sites[['SiteNumber', 'SiteName', 'Region']], left_on='Site_Number', right_on='SiteNumber', how='left')\nk['week'] = pd.to_datetime(k.Date_Sample_Collected).dt.to_period('W').dt.start_time\nweekly = k.groupby('week').Result_Value_Numeric.agg(peak_cells_L='max', median_cells_L='median', samples='size')\nprint(f'{len(s):,} result rows, {len(k):,} Karenia counts, {s.Date_Sample_Collected.min()} to {s.Date_Sample_Collected.max()}, read {time.time()-t0:.1f}s')\nprint(weekly.tail(8).to_string())\nprint(k.nlargest(5, 'Result_Value_Numeric')[['Date_Sample_Collected', 'SiteName', 'Region', 'Result_Name', 'Result_Value_Numeric']].to_string(index=False))\n# weekly.peak_cells_L.plot(logy=True, title='Karenia, peak cells/L per week, all SA sites')","starterSamplePath":"snapshots/2026-10-04/samples.parquet","totalObjects":"6 objects, 1.84 MB on 2026-10-04 (two current parquet files, one dated snapshot pair, README, MANIFEST); +0.92 MB per weekly snapshot.","verifiedRead":{"attempts":[{"ok":true,"family":"parquet","path":"snapshots/2026-10-04/samples.parquet","seconds":0.08}],"starterCell":{"ok":true,"seconds":1.97},"bytes":868385,"snippet":"import os\nimport pyarrow.parquet as pq\npf = pq.ParquetFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], 'snapshots/2026-10-04/samples.parquet'))\nprint(pf.metadata)\ndf = pf.read_row_group(0).to_pandas()\ndf.head()","family":"parquet","image":"531775725781.dkr.ecr.us-east-1.amazonaws.com/scigantic-notebook-geospatial:latest","verifiedAt":"2026-10-05T12:59:25.938Z","path":"snapshots/2026-10-04/samples.parquet","library":"pyarrow","seconds":0.08,"rssDeltaMb":2.6,"summary":"Parquet 94,155 rows, 15 columns, 1 row groups; columns ['OBJECTID:int64', 'Site_Number:int64', 'Site_Description:large_string', 'Date_Sample_Collected:date32[day]', 'Date_Sample_Received:date32[day]', 'Date_Analysis:large_string', 'Date_Report:large_string', 'Monitoring_program:large_string', 'Test_Name:large_string', 'Species_Group:large_string']","facts":["under 'hab-monitoring/': 1 directories (e.g. snapshots); 4 objects named like MANIFEST.json, 2 KB to 848 KB","under 'hab-monitoring/snapshots/': 1 directories (e.g. 2026-10-04)","under 'hab-monitoring/snapshots/2026-10-04/': 2 objects named like samples.parquet, 48 KB to 848 KB"]}},"summary":"A weekly, dated mirror of the South Australian Government's Harmful Algal Bloom Water Testing program, the public data behind the SA Algal Bloom Water Testing Open Data Dashboard.","title":"South Australian Algal Bloom Water Testing - Phytoplankton cell counts - Weekly mirror of the SA Government feature service","updatedAt":"2026-10-05T12:59:25.938Z","visibility":"public","websiteUrl":"https://www.algalbloom.sa.gov.au/testing-and-monitoring"}}