{"success":true,"data":{"author":"City of Boston (Department of Innovation and Technology)","bucketAccessType":"public","bucketListable":false,"bucketName":"data.boston.gov","category":"Urban & Civic","cloudProvider":"api","createdAt":"2026-08-15T00:00:00.000Z","description":"Analyze Boston is the City of Boston's official open data portal, a CKAN 2.x instance at data.boston.gov publishing 232 datasets from 32 city departments across 1,261 resources. 403 of those resources are loaded into the CKAN DataStore, which means they are queryable over HTTP with SQL rather than only downloadable as files. 216 of the 232 datasets are released under the Open Data Commons Public Domain Dedication and License (PDDL) 1.0; 14 carry no stated license and one is ODbL. This archive is the portal itself rather than a single dataset: launch it to explore the whole catalog, discover which resource holds the series you need, and query it in place. The largest publisher by dataset count is Boston Maps (123 datasets, mostly ArcGIS-backed geospatial layers), followed by the Planning Department (14), Public Works (9), the Boston Police Department (9), and the Department of Innovation and Technology (8). The catalog spans 311 service requests, crime and fire incidents, property assessment rolls, building permits, food inspections, payroll, municipal expenditures, building energy and emissions disclosure, road safety, housing conditions, parcels, and administrative and climate-risk boundaries. Access is entirely anonymous and unauthenticated over three endpoints: package_search and package_show for the catalog, datastore_search for filtered and full-text row reads, and datastore_search_sql for server-side SQL including joins and aggregation. The schema card documents the two behaviours that trip up every first attempt: both datastore endpoints cap a single response at 32,000 rows (the SQL endpoint sets records_truncated: true when it does), and every DataStore column is typed text no matter what it holds, so numeric and date predicates need an explicit cast. Scigantic mounts nothing for this archive; the notebook talks to the portal directly over HTTPS. The same portal is what the OpenContext project (github.com/thealphacubicle/OpenContext) wraps when it exposes municipal CKAN, ArcGIS Hub and Socrata catalogs to AI assistants over MCP.","displayTitle":"Analyze Boston, the City of Boston open data portal (232 datasets)","dockerImageUrl":"531775725781.dkr.ecr.us-east-1.amazonaws.com/scigantic-notebook-minimal:latest","githubUrl":"https://github.com/thealphacubicle/OpenContext","id":"a0bbfa5e-419a-5b7f-84f9-4f6e26e71449","license":"Open Data Commons Public Domain Dedication and License (PDDL) 1.0 for 216 of 232 datasets; 14 unspecified, 1 ODbL","referenceUrl":"https://data.boston.gov/","region":"us-east-1","schemaCard":{"sampleObjects":[],"totalObjects":403,"generatedAt":"2026-08-16T17:39:25.112Z","accessHints":[{"language":"python","snippet":"import io, json, urllib.error, urllib.parse, urllib.request\nimport pandas as pd\n\nCKAN = \"https://data.boston.gov/api/3/action\"\n\ndef ckan(action, **params):\n    \"\"\"Any CKAN action endpoint. Plain urllib works; no auth, no headers.\"\"\"\n    url = f\"{CKAN}/{action}?\" + urllib.parse.urlencode(params)\n    with urllib.request.urlopen(url, timeout=300) as r:\n        return json.load(r)[\"result\"]\n\ndef sql(query):\n    \"\"\"Server-side SQL over the DataStore. Warns when the 32,000-row cap clips.\"\"\"\n    res = ckan(\"datastore_search_sql\", sql=query)\n    if res.get(\"records_truncated\"):\n        print(\"WARNING: clipped at the 32,000-row cap; aggregate or page with OFFSET\")\n    return pd.DataFrame(res[\"records\"])\n"},{"language":"python","snippet":"# Bulk CSV download. pd.read_csv(url) on this portal raises 403: the CKAN link\n# 302s to a presigned S3 URL that the redirect fails on. Resolve it by hand and\n# drop the explicit :443, which is the single change that makes it a 200.\nclass _NoRedirect(urllib.request.HTTPRedirectHandler):\n    def redirect_request(self, *a, **k):\n        return None\n\ndef boston_csv(url, **read_csv_kwargs):\n    try:\n        urllib.request.build_opener(_NoRedirect).open(url, timeout=120)\n        signed = url\n    except urllib.error.HTTPError as e:\n        if e.code not in (301, 302, 303, 307, 308):\n            raise\n        signed = e.headers[\"Location\"].replace(\"s3.amazonaws.com:443\", \"s3.amazonaws.com\")\n    with urllib.request.urlopen(signed, timeout=900) as r:\n        return pd.read_csv(io.BytesIO(r.read()), **read_csv_kwargs)\n"},{"snippet":"# Resolve a resource id by package name rather than hardcoding it, and page\n# past the 32,000-row cap. Verified against a 282,836-row resource.\ndef resources(package_name):\n    pkg = ckan(\"package_show\", id=package_name)\n    return [\n        {\"id\": r[\"id\"], \"name\": r.get(\"name\"), \"file\": r[\"url\"].rsplit(\"/\", 1)[-1],\n         \"format\": r.get(\"format\"), \"live\": bool(r.get(\"datastore_active\"))}\n        for r in pkg[\"resources\"]\n    ]\n\ndef read_all(resource_id, page=32000, **filters):\n    \"\"\"Every row of a DataStore resource, in _id order.\"\"\"\n    out, offset = [], 0\n    while True:\n        res = ckan(\"datastore_search\", resource_id=resource_id,\n                   limit=page, offset=offset, **filters)\n        rows = res[\"records\"]\n        out.extend(rows)\n        if len(rows) < page:\n            return pd.DataFrame(out)\n        offset += page\n","language":"python"}],"caveats":["DO NOT look for a FUSE mount. This archive has cloudProvider \"api\": nothing is mounted, ~/archive and /mnt/archive do not exist, and there are no files on disk to walk. The generic archive preamble you may have been given says otherwise; it is wrong for this row. All data is read over HTTPS from the CKAN API at https://data.boston.gov/api/3/action. The notebook has outbound internet; no credentials, tokens or headers are required for any endpoint below.","DO NOT trust a result set of exactly 32,000 rows. Both datastore_search and datastore_search_sql cap a single response at 32,000 records no matter what LIMIT you write, so a query over a million-row resource returns a full-looking frame that is silently a fragment. datastore_search_sql sets records_truncated: true in the result when it clips; check it on every call. The fixes: aggregate server-side (GROUP BY in SQL returns few enough rows to fit), or page with ORDER BY _id plus OFFSET, which is verified to work past 280,000.","DO NOT do arithmetic or date comparison on a column without casting it. Almost every DataStore column on this portal is typed text regardless of what it holds, so avg(\"latitude\") returns HTTP 409 Conflict rather than a number, and min/max on a date column sorts lexically and returns a confident wrong answer. Cast explicitly in SQL: \"latitude\"::float, \"open_dt\"::timestamp, date_part('year', \"open_dt\"::timestamp). Where a column IS genuinely typed numeric or timestamp it is marked as such in the columns list below, and those are safe to use directly.","SQL AGGREGATES COME BACK AS STRINGS, so arithmetic on the resulting DataFrame silently fails or, worse, concatenates. PostgreSQL `numeric` results are serialized as JSON strings by the CKAN API (sum(\"OVERTIME\") returns '153220443.25', not 153220443.25) while `int`/`bigint` and `double precision` come back as JSON numbers. So count(*) is an int, avg() and sum() over a numeric column are strings, and dividing one such column by another raises \"unsupported operand type(s) for /: 'str' and 'str'\". Run pd.to_numeric on every aggregate column before doing arithmetic on it. Do NOT blanket-coerce every column: ward \"06\", precinct \"0104\" and ZIP \"02128\" are zero-padded identifiers that must stay strings.","A 403 FROM datastore_search_sql MEANS THE SQL TEXT TRIPPED AN EDGE FILTER, NOT THAT YOUR QUERY IS WRONG. Some perfectly valid PostgreSQL is rejected with HTTP 403 Forbidden before it ever reaches the database, so there is no SQL error message to read and retrying the identical query never helps. Verified blocked, deterministically, 5 of 5 attempts each (2026-08-15): agg(...) FILTER (WHERE ...) · extract(<part> FROM ...) · substring(x FROM a FOR b) · trim(BOTH ... FROM ...) · NULLIF(...) · a literal constant comparison such as 1=2. Verified working substitutes for each: sum(CASE WHEN cond THEN 1 ELSE 0 END) for FILTER · date_part('year', ts) and date_part('epoch', ts_a - ts_b) for extract · left(col, n) for substring · CASE WHEN d = 0 THEN NULL ELSE d END for NULLIF. Subqueries, CTEs, LEFT JOIN, IN (...), HAVING, window functions, percentile_cont WITHIN GROUP, the regex operator ~, and casts are all fine. When a query 403s, rewrite it; do not retry it.","DO NOT call pd.read_csv() on a data.boston.gov download URL. It raises HTTP 403. The CKAN download link 302-redirects to a presigned S3 URL on the bucket og-production-open-data-bostonma-892364687672, and the request fails when it is followed from urllib's redirect path, which is what pandas uses. Verified 2026-08-15: pd.read_csv(url) -> 403, while requests.get(url) on the identical URL -> 200. Two fixes, both verified: read through requests (pd.read_csv(io.BytesIO(requests.get(url).content))), or resolve the redirect yourself and strip the explicit \":443\" from the returned Location before fetching it. The bucket itself is private (anonymous ListObjectsV2 and GetObject both return AccessDenied), so there is no S3 path around this: the presigned link is the only way in. See accessHints for a copy-paste helper. None of this applies to the /api/3/action endpoints, which are plain JSON over HTTPS and work with bare urllib.","Resource ids, not names, are the addressing key, and the names are unreliable: CKAN's resource `name` field is null on many Boston resources, and several packages publish generated filenames (tmp4myyj_u8.csv) that carry no year. The ids listed in this card were captured on 2026-08-15. The package NAME is the stable handle, so resolve ids at runtime with GET /api/3/action/package_show?id=<package-name> and match on the resource `name` or `url` rather than hardcoding an index into the resource array, whose order is not guaranteed.","_id is CKAN's synthetic 1-based row number added by the DataStore loader, not a key from the source data. It is the right column to ORDER BY for stable paging and it is meaningless as a join key or an entity id.","package_search returns at most 1000 rows per call and the catalog is 232 datasets, so one call covers it today. Do not assume that stays true; read result[\"count\"] and page on `start` if it exceeds `rows`.","datastore_active: false does NOT mean the resource is unusable, only that it is not SQL-queryable. 858 of the 1,261 resources are in that state and most are perfectly good GeoJSON, Shapefile, XLSX or PDF downloads. Read them as files via the CSV/download helper rather than concluding the data is missing.","A \"dataset\" on this portal is a container, not a table. The 232 packages hold 1,261 resources, and a package routinely mixes a per-year data series, a PDF data dictionary, a Google Doc, and several format mirrors of the same layer. Always inspect the resource list before assuming a package is one table."],"layout":{"pattern":"The Analyze Boston portal itself, not one dataset. CKAN 2.x at https://data.boston.gov, 232 datasets from 32 city departments across 1261 resources, of which 403 are loaded into the CKAN DataStore and therefore queryable with SQL rather than only downloadable.\nFOUR ENDPOINTS DO EVERYTHING. All are anonymous GET, all return JSON, all work with bare urllib.\n  package_search?q=&rows=1000    the whole catalog in one call (pass rows=0 for just the count)\n  package_show?id=<name>         one dataset: notes, license, and every resource with its id,\n                                 format, url and datastore_active flag\n  datastore_search?resource_id=  filtered row reads: filters={\"neighborhood\":\"Roxbury\"},\n                                 q=<full text>, fields=, sort=, limit=, offset=\n  datastore_search_sql?sql=      server-side SQL, including JOINs across resources and\n                                 GROUP BY aggregation. This is the one that scales.\nFORMATS ACROSS THE CATALOG: 384 CSV, 188 HTML, 125 GeoJSON, 120 ArcGIS GeoServices REST API, 118 KML, 105 Shapefile, 73 ZIP, 68 PDF, 47 XLSX. The GeoJSON, KML, Shapefile and ArcGIS entries are overwhelmingly Boston Maps layers, which are mirrors of one ArcGIS Hub item and are best read from the FeatureServer rather than the Hub download API (see the Climate Ready Boston archive for why).\nBIGGEST PUBLISHERS BY DATASET COUNT: Boston Maps 123, Planning Department 14, Public Works 9, Boston Police Department 9, Department of Innovation and Technology 8, Inspectional Services 6, Environment Department 6, DoIT Data & Analytics 5, Archives and Record Management 5, Boston Transportation Department 5.\nLICENSING: 216 of 232 datasets are PDDL 1.0 (public domain dedication), 14 state no license, and 1 is ODbL. Check package_show(...)[\"license_title\"] before redistributing anything from the unspecified 14; absence of a license is not a grant.\nFourteen of the highest-value datasets on this portal are onboarded to Scigantic as their own archives with their own deep cards (311, crime, property assessment, building permits, food inspections, employee earnings, checkbook, BERDO, Vision Zero, fire, BPD FIO, RentSmart, Climate Ready Boston, parcels and boundaries). Use this archive to find the rest.\nSEVEN OF THOSE FOURTEEN ARE NO LONGER READ FROM THIS API. Property assessment, BPD FIO, fire incidents, employee earnings, parcels and boundaries, BERDO and Climate Ready Boston are published annually or have stopped updating, so Scigantic serves them from a mirror at s3://scigantic-boston-open-data/ (raw/ byte-verbatim plus parquet/) that FUSE-mounts at ~/archive with no row cap and no presigned-URL 403. The seven that change daily (311, crime, building permits, food inspections, checkbook, Vision Zero, RentSmart) are deliberately NOT mirrored and still come from this API, because a stale mirror is worse than a slow query. If you are working with one of the mirrored seven, open that archive rather than querying the DataStore through this one.","example":"Find every dataset mentioning \"housing\", then list what is actually queryable in one of them:\n  ckan(\"package_search\", q=\"housing\", rows=50)[\"results\"]\n  [r for r in ckan(\"package_show\", id=\"rentsmart\")[\"resources\"] if r[\"datastore_active\"]]"},"starterCell":"import io, json, urllib.error, urllib.parse, urllib.request\nimport pandas as pd\n\nCKAN = \"https://data.boston.gov/api/3/action\"\n\ndef ckan(action, **params):\n    \"\"\"Any CKAN action endpoint. Plain urllib works; no auth, no headers.\"\"\"\n    url = f\"{CKAN}/{action}?\" + urllib.parse.urlencode(params)\n    with urllib.request.urlopen(url, timeout=300) as r:\n        return json.load(r)[\"result\"]\n\ndef sql(query):\n    \"\"\"Server-side SQL over the DataStore. Warns when the 32,000-row cap clips.\"\"\"\n    res = ckan(\"datastore_search_sql\", sql=query)\n    if res.get(\"records_truncated\"):\n        print(\"WARNING: clipped at the 32,000-row cap; aggregate or page with OFFSET\")\n    return pd.DataFrame(res[\"records\"])\n\n# What is on the portal, and what of it can be queried with SQL.\ncatalog = ckan(\"package_search\", rows=1000)\nrows = []\nfor p in catalog[\"results\"]:\n    live = [r for r in p[\"resources\"] if r.get(\"datastore_active\")]\n    rows.append({\n        \"package\": p[\"name\"],\n        \"title\": p[\"title\"],\n        \"org\": (p.get(\"organization\") or {}).get(\"title\"),\n        \"license\": p.get(\"license_title\"),\n        \"resources\": len(p[\"resources\"]),\n        \"queryable\": len(live),\n    })\ndf = pd.DataFrame(rows)\nprint(f'{len(df)} datasets, {df[\"resources\"].sum()} resources, {df[\"queryable\"].sum()} queryable')\nprint(df.sort_values(\"queryable\", ascending=False).head(15).to_string(index=False))\n","format":"v2-hybrid","rowCount":"232 datasets, 403 SQL-queryable resources"},"summary":"Analyze Boston is the City of Boston's official open data portal, a CKAN 2.x instance at data.boston.gov publishing 232 datasets from 32 city departments across 1,261 resources.","title":"Analyze Boston, the City of Boston open data portal (232 datasets)","updatedAt":"2026-08-16T17:40:35.667Z","visibility":"public","websiteUrl":"https://data.boston.gov/","whiteLabel":"boston"}}