{"success":true,"data":{"author":"City of Boston (Boston Police Department)","bucketAccessType":"public","bucketListable":false,"bucketName":"data.boston.gov","category":"Urban & Civic","cloudProvider":"api","createdAt":"2026-08-15T00:00:00.000Z","description":"Incident reports filed by the Boston Police Department, published as two separate CKAN datasets that together cover July 2012 to the present. The current series (August 2015 onward, sourced from the Mark43 records management system that went live in September 2019) holds 946,325 rows across nine yearly DataStore resources, with 2023 to present consolidated into one 289,421-row resource. The legacy series covers 8 July 2012 to 10 August 2015 with 268,056 rows from the Intergraph system. One row is one reported incident. The current schema carries an incident number, a four-digit offense code with its description and the FBI UCR part (One, Two, Three), the BPD district and reporting area, a shooting flag, the occurrence timestamp broken out into year, month, day of week and hour, the street name, and a latitude and longitude. The legacy schema is different in almost every column name and adds fields the new system dropped: nature code, weapon type, a domestic-incident flag and the shift. Records excluded under Massachusetts General Laws chapter 41 section 98F are withheld, and in the current 2023-to-present resource the OFFENSE_CODE_GROUP and UCR_PART columns are null in all 289,421 rows even though the columns still exist, so any breakdown by crime category has to key off OFFENSE_CODE or OFFENSE_DESCRIPTION rather than the grouping columns. Coordinates are present but 13,208 of those 289,421 rows carry a null or zero latitude, which silently drags any mapped centroid toward the Gulf of Guinea if not filtered. The legacy file stores its timestamp as MM/DD/YYYY HH:MI:SS AM text, so a lexical sort or a plain min/max returns a meaningless answer without a to_timestamp cast. The schema card documents the column mapping between the two systems, the null patterns, and the coordinate filter. Published by the Boston Police Department under PDDL 1.0.","displayTitle":"Boston Crime Incident Reports, 2012 to present","dockerImageUrl":"531775725781.dkr.ecr.us-east-1.amazonaws.com/scigantic-notebook-minimal:latest","id":"523eb363-ad9a-5a33-b034-c79f2989932e","license":"Open Data Commons Public Domain Dedication and License (PDDL) 1.0","referenceUrl":"https://data.boston.gov/dataset/crime-incident-reports-august-2015-to-date-source-new-system","region":"us-east-1","schemaCard":{"layout":{"example":"Top offense descriptions in the current file, with their share of geocoded rows:\n  SELECT \"OFFENSE_DESCRIPTION\", count(*) n,\n         sum(CASE WHEN \"Lat\"::float <> 0 THEN 1 ELSE 0 END) AS geocoded\n  FROM \"b973d8cb-eeb2-4e7e-99da-c92938efc9c0\"\n  GROUP BY 1 ORDER BY n DESC LIMIT 20","pattern":"One row is one reported incident. Two CKAN packages, two record management systems, 1,214,381 rows total.\nCURRENT SERIES (package crime-incident-reports-august-2015-to-date-source-new-system), 946,325 rows in 9 yearly resources plus two empty XLSX stubs. 2023 to present is consolidated into one 289,421-row resource; 2015 to 2022 are one resource per year. Sourced from the Mark43 RMS that went live September 2019 and backfilled to June 2015. 18 columns.\nLEGACY SERIES (package crime-incident-reports-july-2012-august-2015-source-legacy-system), 268,056 rows in one file, verified range 2012-07-08 to 2015-08-10, from the Intergraph RMS. 21 columns, almost none of which share a name with the current schema.\nCOLUMN MAPPING between the two:\n  current                   legacy\n  INCIDENT_NUMBER           COMPNOS\n  OFFENSE_DESCRIPTION       INCIDENT_TYPE_DESCRIPTION\n  OFFENSE_CODE              MAIN_CRIMECODE (different code system, not comparable)\n  DISTRICT                  REPTDISTRICT\n  REPORTING_AREA            REPORTINGAREA\n  OCCURRED_ON_DATE          FROMDATE\n  SHOOTING                  Shooting (\"0\"/\"1\" vs \"Yes\"/\"No\")\n  UCR_PART                  UCRPART (\"Part One\" in legacy; all-null in current)\n  STREET / Lat / Long       STREETNAME / Y / X (legacy X,Y are State Plane, not lat/lon)\n  (none)                    NatureCode, WEAPONTYPE, DOMESTIC, SHIFT, XSTREETNAME\nThe legacy-only columns are the interesting loss: weapon type, the domestic-incident flag and the shift were never carried into Mark43, so any analysis using them ends in August 2015.\nPER-YEAR ROW COUNTS (current): 2015 53,597 | 2016 99,430 | 2017 101,338 | 2018 98,888 | 2019 87,184 | 2020 70,894 | 2021 71,721 | 2022 73,852 | 2023-present 289,421.\nRESOURCES:\n  Crime Incident Reports (August 2015 - To Date) (Source: New System) (data.boston.gov/dataset/crime-incident-reports-august-2015-to-date-source-new-system): 11 DataStore resources of 11 total\n    b973d8cb-eeb2-4e7e-99da-c92938efc9c0     289421 rows  Crime Incident Reports - 2023 to Present\n    313e56df-6d77-49d2-9c49-ee411f10cf58      73852 rows  Crime Incident Reports - 2022\n    f4495ee9-c42c-4019-82c1-d067f07e45d2      71721 rows  Crime Incident Reports - 2021\n    be047094-85fe-4104-a480-4fa3d03f9623      70894 rows  Crime Incident Reports - 2020\n    34e0ae6b-8c94-4998-ae9e-1b51551fe9ba      87184 rows  Crime Incident Reports - 2019\n    e86f8e38-a23c-4c1a-8455-c8f94210a8f1      98888 rows  Crime Incident Reports - 2018\n    64ad0053-842c-459b-9833-ff53d568f2e3     101338 rows  Crime Incident Reports - 2017\n    b6c4e2c3-7b1e-4f4a-b019-bef8c6a0e882      99430 rows  Crime Incident Reports - 2016\n    792031bf-b9bb-467c-b118-fe795befdf00      53597 rows  Crime Incident Reports - 2015\n    9c30453a-fefa-4fe0-b51a-5fc09b0f4655          0 rows  rmscrimeincidentfieldexplanation.xlsx\n    3aeccf51-a231-4555-ba21-74572b4c33d6          0 rows  rmsoffensecodes.xlsx\n  Crime Incident Reports (July 2012 - August 2015) (Source: Legacy System) (data.boston.gov/dataset/crime-incident-reports-july-2012-august-2015-source-legacy-system): 2 DataStore resources of 2 total\n    ba5ed0e2-e901-438c-b2e0-4acfc3c452b9     268056 rows  Crime Incident Reports (July 2012 - August 2015) (Source: Legacy System)\n    b3b7d7f6-2582-45b1-a342-cc14eaa7d14a          0 rows  crimeincidentfieldexplanationjuly-august-2015.xlsx"},"totalObjects":13,"sampleObjects":["b973d8cb-eeb2-4e7e-99da-c92938efc9c0  Crime Incident Reports - 2023 to Present","313e56df-6d77-49d2-9c49-ee411f10cf58  Crime Incident Reports - 2022","f4495ee9-c42c-4019-82c1-d067f07e45d2  Crime Incident Reports - 2021","be047094-85fe-4104-a480-4fa3d03f9623  Crime Incident Reports - 2020","34e0ae6b-8c94-4998-ae9e-1b51551fe9ba  Crime Incident Reports - 2019","e86f8e38-a23c-4c1a-8455-c8f94210a8f1  Crime Incident Reports - 2018","64ad0053-842c-459b-9833-ff53d568f2e3  Crime Incident Reports - 2017","b6c4e2c3-7b1e-4f4a-b019-bef8c6a0e882  Crime Incident Reports - 2016","792031bf-b9bb-467c-b118-fe795befdf00  Crime Incident Reports - 2015","9c30453a-fefa-4fe0-b51a-5fc09b0f4655  rmscrimeincidentfieldexplanation.xlsx","3aeccf51-a231-4555-ba21-74572b4c33d6  rmsoffensecodes.xlsx","ba5ed0e2-e901-438c-b2e0-4acfc3c452b9  Crime Incident Reports (July 2012 - August 2015) (Source: Legacy System)","b3b7d7f6-2582-45b1-a342-cc14eaa7d14a  crimeincidentfieldexplanationjuly-august-2015.xlsx"],"generatedAt":"2026-08-16T17:39:25.112Z","format":"v2-hybrid","rowCount":1214381,"starterCell":"import io, json, urllib.error, urllib.parse, urllib.request\nimport pandas as pd\n\nCKAN = \"https://data.boston.gov/api/3/action\"\n\ndef ckan(action, **params):\n    \"\"\"Any CKAN action endpoint. Plain urllib works; no auth, no headers.\"\"\"\n    url = f\"{CKAN}/{action}?\" + urllib.parse.urlencode(params)\n    with urllib.request.urlopen(url, timeout=300) as r:\n        return json.load(r)[\"result\"]\n\ndef sql(query):\n    \"\"\"Server-side SQL over the DataStore. Warns when the 32,000-row cap clips.\"\"\"\n    res = ckan(\"datastore_search_sql\", sql=query)\n    if res.get(\"records_truncated\"):\n        print(\"WARNING: clipped at the 32,000-row cap; aggregate or page with OFFSET\")\n    return pd.DataFrame(res[\"records\"])\n\nR_CURRENT = \"b973d8cb-eeb2-4e7e-99da-c92938efc9c0\"   # 2023 to present, 289,421 rows\n\n# Hour-of-day profile by district, geocoded rows only. Aggregated server-side.\nprofile = sql(f'''\n    SELECT \"DISTRICT\", \"HOUR\"::int AS hour, count(*) AS incidents\n    FROM \"{R_CURRENT}\"\n    WHERE \"DISTRICT\" IS NOT NULL\n      AND \"Lat\" IS NOT NULL AND \"Lat\"::float <> 0\n    GROUP BY 1, 2\n''')\npivot = profile.pivot(index=\"hour\", columns=\"DISTRICT\", values=\"incidents\").fillna(0)\nprint(pivot.head(24))\nprint(\"\\ntotal geocoded incidents:\", int(profile[\"incidents\"].sum()))\n","columns":[{"name":"_id","type":"int"},{"type":"text","name":"INCIDENT_NUMBER"},{"name":"OFFENSE_CODE","type":"text"},{"type":"text","name":"OFFENSE_CODE_GROUP"},{"name":"OFFENSE_DESCRIPTION","type":"text"},{"name":"DISTRICT","type":"text"},{"name":"REPORTING_AREA","type":"text"},{"name":"SHOOTING","type":"text"},{"type":"text","name":"OCCURRED_ON_DATE"},{"name":"YEAR","type":"text"},{"name":"MONTH","type":"text"},{"type":"text","name":"DAY_OF_WEEK"},{"name":"HOUR","type":"text"},{"name":"UCR_PART","type":"text"},{"name":"STREET","type":"text"},{"name":"Lat","type":"text"},{"name":"Long","type":"text"},{"type":"text","name":"Location"}],"accessHints":[{"language":"python","snippet":"import io, json, urllib.error, urllib.parse, urllib.request\nimport pandas as pd\n\nCKAN = \"https://data.boston.gov/api/3/action\"\n\ndef ckan(action, **params):\n    \"\"\"Any CKAN action endpoint. Plain urllib works; no auth, no headers.\"\"\"\n    url = f\"{CKAN}/{action}?\" + urllib.parse.urlencode(params)\n    with urllib.request.urlopen(url, timeout=300) as r:\n        return json.load(r)[\"result\"]\n\ndef sql(query):\n    \"\"\"Server-side SQL over the DataStore. Warns when the 32,000-row cap clips.\"\"\"\n    res = ckan(\"datastore_search_sql\", sql=query)\n    if res.get(\"records_truncated\"):\n        print(\"WARNING: clipped at the 32,000-row cap; aggregate or page with OFFSET\")\n    return pd.DataFrame(res[\"records\"])\n"},{"snippet":"# Bulk CSV download. pd.read_csv(url) on this portal raises 403: the CKAN link\n# 302s to a presigned S3 URL that the redirect fails on. Resolve it by hand and\n# drop the explicit :443, which is the single change that makes it a 200.\nclass _NoRedirect(urllib.request.HTTPRedirectHandler):\n    def redirect_request(self, *a, **k):\n        return None\n\ndef boston_csv(url, **read_csv_kwargs):\n    try:\n        urllib.request.build_opener(_NoRedirect).open(url, timeout=120)\n        signed = url\n    except urllib.error.HTTPError as e:\n        if e.code not in (301, 302, 303, 307, 308):\n            raise\n        signed = e.headers[\"Location\"].replace(\"s3.amazonaws.com:443\", \"s3.amazonaws.com\")\n    with urllib.request.urlopen(signed, timeout=900) as r:\n        return pd.read_csv(io.BytesIO(r.read()), **read_csv_kwargs)\n","language":"python"},{"snippet":"# Resolve a resource id by package name rather than hardcoding it, and page\n# past the 32,000-row cap. Verified against a 282,836-row resource.\ndef resources(package_name):\n    pkg = ckan(\"package_show\", id=package_name)\n    return [\n        {\"id\": r[\"id\"], \"name\": r.get(\"name\"), \"file\": r[\"url\"].rsplit(\"/\", 1)[-1],\n         \"format\": r.get(\"format\"), \"live\": bool(r.get(\"datastore_active\"))}\n        for r in pkg[\"resources\"]\n    ]\n\ndef read_all(resource_id, page=32000, **filters):\n    \"\"\"Every row of a DataStore resource, in _id order.\"\"\"\n    out, offset = [], 0\n    while True:\n        res = ckan(\"datastore_search\", resource_id=resource_id,\n                   limit=page, offset=offset, **filters)\n        rows = res[\"records\"]\n        out.extend(rows)\n        if len(rows) < page:\n            return pd.DataFrame(out)\n        offset += page\n","language":"python"}],"caveats":["DO NOT look for a FUSE mount. This archive has cloudProvider \"api\": nothing is mounted, ~/archive and /mnt/archive do not exist, and there are no files on disk to walk. The generic archive preamble you may have been given says otherwise; it is wrong for this row. All data is read over HTTPS from the CKAN API at https://data.boston.gov/api/3/action. The notebook has outbound internet; no credentials, tokens or headers are required for any endpoint below.","DO NOT trust a result set of exactly 32,000 rows. Both datastore_search and datastore_search_sql cap a single response at 32,000 records no matter what LIMIT you write, so a query over a million-row resource returns a full-looking frame that is silently a fragment. datastore_search_sql sets records_truncated: true in the result when it clips; check it on every call. The fixes: aggregate server-side (GROUP BY in SQL returns few enough rows to fit), or page with ORDER BY _id plus OFFSET, which is verified to work past 280,000.","DO NOT do arithmetic or date comparison on a column without casting it. Almost every DataStore column on this portal is typed text regardless of what it holds, so avg(\"latitude\") returns HTTP 409 Conflict rather than a number, and min/max on a date column sorts lexically and returns a confident wrong answer. Cast explicitly in SQL: \"latitude\"::float, \"open_dt\"::timestamp, date_part('year', \"open_dt\"::timestamp). Where a column IS genuinely typed numeric or timestamp it is marked as such in the columns list below, and those are safe to use directly.","SQL AGGREGATES COME BACK AS STRINGS, so arithmetic on the resulting DataFrame silently fails or, worse, concatenates. PostgreSQL `numeric` results are serialized as JSON strings by the CKAN API (sum(\"OVERTIME\") returns '153220443.25', not 153220443.25) while `int`/`bigint` and `double precision` come back as JSON numbers. So count(*) is an int, avg() and sum() over a numeric column are strings, and dividing one such column by another raises \"unsupported operand type(s) for /: 'str' and 'str'\". Run pd.to_numeric on every aggregate column before doing arithmetic on it. Do NOT blanket-coerce every column: ward \"06\", precinct \"0104\" and ZIP \"02128\" are zero-padded identifiers that must stay strings.","A 403 FROM datastore_search_sql MEANS THE SQL TEXT TRIPPED AN EDGE FILTER, NOT THAT YOUR QUERY IS WRONG. Some perfectly valid PostgreSQL is rejected with HTTP 403 Forbidden before it ever reaches the database, so there is no SQL error message to read and retrying the identical query never helps. Verified blocked, deterministically, 5 of 5 attempts each (2026-08-15): agg(...) FILTER (WHERE ...) · extract(<part> FROM ...) · substring(x FROM a FOR b) · trim(BOTH ... FROM ...) · NULLIF(...) · a literal constant comparison such as 1=2. Verified working substitutes for each: sum(CASE WHEN cond THEN 1 ELSE 0 END) for FILTER · date_part('year', ts) and date_part('epoch', ts_a - ts_b) for extract · left(col, n) for substring · CASE WHEN d = 0 THEN NULL ELSE d END for NULLIF. Subqueries, CTEs, LEFT JOIN, IN (...), HAVING, window functions, percentile_cont WITHIN GROUP, the regex operator ~, and casts are all fine. When a query 403s, rewrite it; do not retry it.","DO NOT call pd.read_csv() on a data.boston.gov download URL. It raises HTTP 403. The CKAN download link 302-redirects to a presigned S3 URL on the bucket og-production-open-data-bostonma-892364687672, and the request fails when it is followed from urllib's redirect path, which is what pandas uses. Verified 2026-08-15: pd.read_csv(url) -> 403, while requests.get(url) on the identical URL -> 200. Two fixes, both verified: read through requests (pd.read_csv(io.BytesIO(requests.get(url).content))), or resolve the redirect yourself and strip the explicit \":443\" from the returned Location before fetching it. The bucket itself is private (anonymous ListObjectsV2 and GetObject both return AccessDenied), so there is no S3 path around this: the presigned link is the only way in. See accessHints for a copy-paste helper. None of this applies to the /api/3/action endpoints, which are plain JSON over HTTPS and work with bare urllib.","Resource ids, not names, are the addressing key, and the names are unreliable: CKAN's resource `name` field is null on many Boston resources, and several packages publish generated filenames (tmp4myyj_u8.csv) that carry no year. The ids listed in this card were captured on 2026-08-15. The package NAME is the stable handle, so resolve ids at runtime with GET /api/3/action/package_show?id=<package-name> and match on the resource `name` or `url` rather than hardcoding an index into the resource array, whose order is not guaranteed.","_id is CKAN's synthetic 1-based row number added by the DataStore loader, not a key from the source data. It is the right column to ORDER BY for stable paging and it is meaningless as a join key or an entity id.","OFFENSE_CODE_GROUP and UCR_PART are 100% NULL in the current 2023-to-present resource. Measured: 289,421 rows, 0 non-null in either column. The columns still exist, so a groupby on them returns one all-null bucket rather than an error. Categorise on OFFENSE_CODE (the four-digit BPD code) or OFFENSE_DESCRIPTION instead. The pre-2023 yearly resources do carry these columns populated, which is why a naive concat looks fine until 2023.","DO NOT map without filtering the coordinates. 13,208 of the 289,421 rows in the current resource have a null or zero Lat. Zeros are not null, so mean-centroid and heatmap code that only drops nulls pulls the result toward (0, 0) in the Gulf of Guinea. Filter \"Lat\"::float <> 0 AND \"Lat\" IS NOT NULL.","The legacy file stores FROMDATE as MM/DD/YYYY HH:MI:SS AM text. min() and max() on it sort lexically and return \"01/01/2013\" and \"12/31/2014\", which is neither the real first nor the real last record. Cast with to_timestamp(\"FROMDATE\", 'MM/DD/YYYY HH12:MI:SS AM'), which gives the verified true range 2012-07-08 to 2015-08-10.","MAIN_CRIMECODE (legacy, e.g. \"05RB\") and OFFENSE_CODE (current, e.g. \"3115\") are different coding systems. Do not join or union on them, and do not build a single time series of a crime category across the August 2015 boundary without mapping through the description text and saying so.","Incidents excluded under Massachusetts General Laws chapter 41 section 98F are withheld from publication, so this is not a complete census of reports. The 2019 data was also re-extracted after an initial release mixed Intergraph and Mark43 exports; the file served now is the corrected one.","DAY_OF_WEEK is padded to a fixed width (\"Friday   \"). Strip it before grouping or you get one bucket per padding variant."],"sampleRows":[{"__resourceId":"b973d8cb-eeb2-4e7e-99da-c92938efc9c0","SHOOTING":"0","STREET":"FAVRE ST","YEAR":"2023","__schema":"current system (2023 to present)","_id":1,"INCIDENT_NUMBER":"232007173","OFFENSE_CODE_GROUP":null,"Lat":"42.27166144598919","MONTH":"1","REPORTING_AREA":" ","UCR_PART":null,"HOUR":"22","OCCURRED_ON_DATE":"2023-01-27 22:44:00+00","DAY_OF_WEEK":"Friday   ","OFFENSE_DESCRIPTION":"INVESTIGATE PERSON","Location":"(42.27166144598919, -71.09953381947697)","DISTRICT":"B3","OFFENSE_CODE":"3115","Long":"-71.09953381947697"},{"SHIFT":"Last","WEAPONTYPE":"Other","__resourceId":"ba5ed0e2-e901-438c-b2e0-4acfc3c452b9","DOMESTIC":"No","Year":"2012","Location":"(42.34638135, -71.10379454)","INCIDENT_TYPE_DESCRIPTION":"RESIDENTIAL BURGLARY","Month":"7","Y":"2951498.962","XSTREETNAME":null,"FROMDATE":"07/08/2012 06:00:00 AM","STREETNAME":"ABERDEEN ST","REPTDISTRICT":"D4","REPORTINGAREA":"629","MAIN_CRIMECODE":"05RB","DAY_WEEK":"Sunday","_id":1,"NatureCode":"BERPTA","COMPNOS":"120420285","__schema":"legacy system (Jul 2012 to Aug 2015)","Shooting":"No","UCRPART":"Part One","X":"763273.1791"}]},"summary":"Incident reports filed by the Boston Police Department, published as two separate CKAN datasets that together cover July 2012 to the present.","title":"Boston Crime Incident Reports, 2012 to present","updatedAt":"2026-08-16T17:40:34.466Z","visibility":"public","websiteUrl":"https://www.boston.gov/departments/police","whiteLabel":"boston"}}