{"success":true,"data":{"author":"City of Boston (Vision Zero Boston, Boston Transportation Department)","bucketAccessType":"public","bucketListable":false,"bucketName":"data.boston.gov","category":"Urban & Civic","cloudProvider":"api","createdAt":"2026-08-15T00:00:00.000Z","description":"Traffic crashes in Boston that required a public safety response, published as part of the Vision Zero Boston programme: 44,344 crash records from 1 January 2015 to the present, plus a separate 161-record file of traffic fatalities over the same period. Records are compiled by the Department of Innovation and Technology from the city's Computer-Aided Dispatch (911) system and verified as having required a response. One row is one dispatch. Each carries the dispatch timestamp, a mode_type of ped, bike or mv identifying the mode of transport of the most vulnerable person involved (so a car striking a cyclist is coded bike, not mv), a location_type of Street, Intersection or Other, the street name and up to two cross streets, and coordinates in both Massachusetts State Plane (x_cord, y_cord) and WGS84 (lat, long). The dataset is deliberately privacy-limited: it does not indicate injury severity or whether medical transport occurred, which is why the fatality records are published as a separate, much smaller file rather than as a severity flag. Every field carries a per-column description in the DataStore metadata, which is unusual for this portal and is surfaced in the schema card. The 44,344 crash records against 161 fatalities is the ratio that makes this dataset useful: it supports exposure-adjusted risk by mode, corridor and intersection hot-spot identification, temporal patterns by hour and season, and before-and-after evaluation of street redesigns and speed limit changes. The schema card documents the two traps. The lat and long columns are not raw 911 coordinates: the city snaps the reported point to the nearest street centreline and then reprojects, so the coordinates are accurate to the street but not to the exact point of impact, and clustering on them recovers street segments rather than collision points. And when location_type is Intersection the street column is null and the location lives entirely in xstreet1 and xstreet2, so a groupby on street silently discards every intersection crash, which is the majority of the file. Published by Vision Zero Boston and the Boston Transportation Department under PDDL 1.0.","displayTitle":"Boston Vision Zero crash and fatality records, 2015 to present","dockerImageUrl":"531775725781.dkr.ecr.us-east-1.amazonaws.com/scigantic-notebook-minimal:latest","id":"e85a66ab-dbdd-52d3-9100-7055e8700828","license":"Open Data Commons Public Domain Dedication and License (PDDL) 1.0","referenceUrl":"https://data.boston.gov/dataset/vision-zero-crash-records","region":"us-east-1","schemaCard":{"caveats":["DO NOT look for a FUSE mount. This archive has cloudProvider \"api\": nothing is mounted, ~/archive and /mnt/archive do not exist, and there are no files on disk to walk. The generic archive preamble you may have been given says otherwise; it is wrong for this row. All data is read over HTTPS from the CKAN API at https://data.boston.gov/api/3/action. The notebook has outbound internet; no credentials, tokens or headers are required for any endpoint below.","DO NOT trust a result set of exactly 32,000 rows. Both datastore_search and datastore_search_sql cap a single response at 32,000 records no matter what LIMIT you write, so a query over a million-row resource returns a full-looking frame that is silently a fragment. datastore_search_sql sets records_truncated: true in the result when it clips; check it on every call. The fixes: aggregate server-side (GROUP BY in SQL returns few enough rows to fit), or page with ORDER BY _id plus OFFSET, which is verified to work past 280,000.","DO NOT do arithmetic or date comparison on a column without casting it. Almost every DataStore column on this portal is typed text regardless of what it holds, so avg(\"latitude\") returns HTTP 409 Conflict rather than a number, and min/max on a date column sorts lexically and returns a confident wrong answer. Cast explicitly in SQL: \"latitude\"::float, \"open_dt\"::timestamp, date_part('year', \"open_dt\"::timestamp). Where a column IS genuinely typed numeric or timestamp it is marked as such in the columns list below, and those are safe to use directly.","SQL AGGREGATES COME BACK AS STRINGS, so arithmetic on the resulting DataFrame silently fails or, worse, concatenates. PostgreSQL `numeric` results are serialized as JSON strings by the CKAN API (sum(\"OVERTIME\") returns '153220443.25', not 153220443.25) while `int`/`bigint` and `double precision` come back as JSON numbers. So count(*) is an int, avg() and sum() over a numeric column are strings, and dividing one such column by another raises \"unsupported operand type(s) for /: 'str' and 'str'\". Run pd.to_numeric on every aggregate column before doing arithmetic on it. Do NOT blanket-coerce every column: ward \"06\", precinct \"0104\" and ZIP \"02128\" are zero-padded identifiers that must stay strings.","A 403 FROM datastore_search_sql MEANS THE SQL TEXT TRIPPED AN EDGE FILTER, NOT THAT YOUR QUERY IS WRONG. Some perfectly valid PostgreSQL is rejected with HTTP 403 Forbidden before it ever reaches the database, so there is no SQL error message to read and retrying the identical query never helps. Verified blocked, deterministically, 5 of 5 attempts each (2026-08-15): agg(...) FILTER (WHERE ...) · extract(<part> FROM ...) · substring(x FROM a FOR b) · trim(BOTH ... FROM ...) · NULLIF(...) · a literal constant comparison such as 1=2. Verified working substitutes for each: sum(CASE WHEN cond THEN 1 ELSE 0 END) for FILTER · date_part('year', ts) and date_part('epoch', ts_a - ts_b) for extract · left(col, n) for substring · CASE WHEN d = 0 THEN NULL ELSE d END for NULLIF. Subqueries, CTEs, LEFT JOIN, IN (...), HAVING, window functions, percentile_cont WITHIN GROUP, the regex operator ~, and casts are all fine. When a query 403s, rewrite it; do not retry it.","DO NOT call pd.read_csv() on a data.boston.gov download URL. It raises HTTP 403. The CKAN download link 302-redirects to a presigned S3 URL on the bucket og-production-open-data-bostonma-892364687672, and the request fails when it is followed from urllib's redirect path, which is what pandas uses. Verified 2026-08-15: pd.read_csv(url) -> 403, while requests.get(url) on the identical URL -> 200. Two fixes, both verified: read through requests (pd.read_csv(io.BytesIO(requests.get(url).content))), or resolve the redirect yourself and strip the explicit \":443\" from the returned Location before fetching it. The bucket itself is private (anonymous ListObjectsV2 and GetObject both return AccessDenied), so there is no S3 path around this: the presigned link is the only way in. See accessHints for a copy-paste helper. None of this applies to the /api/3/action endpoints, which are plain JSON over HTTPS and work with bare urllib.","Resource ids, not names, are the addressing key, and the names are unreliable: CKAN's resource `name` field is null on many Boston resources, and several packages publish generated filenames (tmp4myyj_u8.csv) that carry no year. The ids listed in this card were captured on 2026-08-15. The package NAME is the stable handle, so resolve ids at runtime with GET /api/3/action/package_show?id=<package-name> and match on the resource `name` or `url` rather than hardcoding an index into the resource array, whose order is not guaranteed.","_id is CKAN's synthetic 1-based row number added by the DataStore loader, not a key from the source data. It is the right column to ORDER BY for stable paging and it is meaningless as a join key or an entity id.","mode_type IS THE MOST VULNERABLE PARTY, NOT THE VEHICLE. A car striking a pedestrian is ped; a car striking a cyclist is bike; mv means no pedestrian or cyclist was involved. Reading it as \"what hit what\" inverts the interpretation of every result.","GROUPING BY street SILENTLY DISCARDS INTERSECTION CRASHES. When location_type is Intersection the street column is null and the location lives entirely in xstreet1 and xstreet2. Intersections are a large share of the file, so a groupby on street quietly analyses only mid-block crashes. Coalesce street with xstreet1, or handle the two location types separately and say which you used.","lat and long ARE SNAPPED TO THE STREET CENTRELINE, per the city's own column description. They are accurate to the street segment and NOT to the point of impact. Clustering on them recovers segments and intersections, not collision points, and a tight spatial buffer around a coordinate is measuring the snap, not the crash.","THERE IS NO SEVERITY OR INJURY COLUMN, by design. Do not infer severity from mode_type, and do not treat the fatality file as a subset you can left-join onto the crash file: they are compiled separately and share no incident identifier.","This is dispatch data, so it counts crashes that got a public safety response. Property-damage crashes handled between drivers, and crashes on state-owned roads within the city, are under-represented. location_type explicitly scopes street to City-owned streets.","x_cord and y_cord are Massachusetts State Plane (feet) and are stored as text with trailing zeros (\"757374.390000000000\"). They are not interchangeable with lat/long and need reprojection, not a rescale."],"totalObjects":2,"starterCell":"import io, json, urllib.error, urllib.parse, urllib.request\nimport pandas as pd\n\nCKAN = \"https://data.boston.gov/api/3/action\"\n\ndef ckan(action, **params):\n    \"\"\"Any CKAN action endpoint. Plain urllib works; no auth, no headers.\"\"\"\n    url = f\"{CKAN}/{action}?\" + urllib.parse.urlencode(params)\n    with urllib.request.urlopen(url, timeout=300) as r:\n        return json.load(r)[\"result\"]\n\ndef sql(query):\n    \"\"\"Server-side SQL over the DataStore. Warns when the 32,000-row cap clips.\"\"\"\n    res = ckan(\"datastore_search_sql\", sql=query)\n    if res.get(\"records_truncated\"):\n        print(\"WARNING: clipped at the 32,000-row cap; aggregate or page with OFFSET\")\n    return pd.DataFrame(res[\"records\"])\n\nCRASHES    = \"e4bfe397-6bfc-49c5-9367-c879fac7401d\"   # 44,344 rows\nFATALITIES = \"92f18923-d4ec-4c17-9405-4e0da63e1d6c\"   # 161 rows\n\n# Crashes by mode and year. mode_type is the MOST VULNERABLE party involved,\n# so \"ped\" means a pedestrian was hit, whatever hit them.\ncrashes = sql(f'''\n    SELECT date_part(\\'year\\', \"dispatch_ts\"::timestamp)::int AS yr,\n           \"mode_type\", count(*) AS crashes\n    FROM \"{CRASHES}\" GROUP BY 1, 2\n''')\ndeaths = sql(f'''\n    SELECT date_part(\\'year\\', \"date_time\"::timestamp)::int AS yr,\n           \"mode_type\", count(*) AS fatalities\n    FROM \"{FATALITIES}\" GROUP BY 1, 2\n''')\nboth = crashes.merge(deaths, on=[\"yr\", \"mode_type\"], how=\"left\").fillna({\"fatalities\": 0})\nprint(both.pivot_table(index=\"yr\", columns=\"mode_type\",\n                       values=[\"crashes\", \"fatalities\"], aggfunc=\"sum\").fillna(0).astype(int))\n","layout":{"pattern":"One row is one 911 dispatch to a traffic crash. Two resources with the SAME 11-column shape apart from the timestamp column name: crash records (44,344 rows, dispatch_ts, verified 2015-01-01 to 2026-07-01) and fatality records (161 rows, date_time, verified 2015-01-22 to 2026-05-15).\nCOLUMNS, and unusually for this portal EVERY ONE CARRIES AN OFFICIAL DESCRIPTION in the DataStore metadata, reproduced here:\n  dispatch_ts / date_time  date and time of 911 emergency services dispatch\n  mode_type      ped, bike or mv. \"Refers to the mode of transport for the most vulnerable\n                 individual involved\", so a car striking a cyclist is coded bike, not mv.\n                 This is the single most misread column in the dataset.\n  location_type  \"street\" if on a City-owned street, \"intersection\" if at a convergence of\n                 two or more, otherwise other\n  street         name of the city-owned street, only when location_type is Street\n  xstreet1       first cross street\n  xstreet2       second cross street\n  x_cord/y_cord  Massachusetts State Plane coordinates\n  lat/long       \"a transformation of snapping the [X/Y] coordinate to the closest street as\n                 well as converting it into WGS84 projection\"\nTHE PRIVACY DESIGN IS DELIBERATE AND SHAPES WHAT YOU CAN ASK. The city states it does not indicate crash severity or whether medical transport occurred, in order to protect the individuals involved. That is why fatalities are a separate 161-row file rather than a flag: there is no severity column to filter on, and injury counts are not derivable.\nRESOURCES:\n  Vision Zero Crash Records (data.boston.gov/dataset/vision-zero-crash-records): 1 DataStore resources of 1 total\n    e4bfe397-6bfc-49c5-9367-c879fac7401d      44344 rows  Vision Zero Crash Records\n  Vision Zero Fatality Records (data.boston.gov/dataset/vision-zero-fatality-records): 1 DataStore resources of 1 total\n    92f18923-d4ec-4c17-9405-4e0da63e1d6c        161 rows  Vision Zero Fatality Records","example":"Crashes by mode and year, the base rate every Vision Zero question starts from:\n  SELECT date_part('year', \"dispatch_ts\"::timestamp)::int AS yr,\n         \"mode_type\", count(*) n\n  FROM \"e4bfe397-6bfc-49c5-9367-c879fac7401d\"\n  GROUP BY 1, 2 ORDER BY 1, 2"},"columns":[{"type":"int","name":"_id"},{"name":"dispatch_ts","type":"text. Date and time of 911 emergency services dispatch"},{"name":"mode_type","type":"text. Type of incident (ped = pedestrian; mv = motor vehicle; bike = bicyclist). Refers to the mode of transport for the most vulnerable individual involved. This means that crashes involving both cyclists and motor vehicles but not pedestrians are coded as cyclist crashes and all other combinations of modes are coded as pedestrian crashes."},{"name":"location_type","type":"text. Locations are coded as \"street\" if the reported location was on a City-owned street; \"intersection\" if the reported location was at the convergence of two or more streets; or \"other\" if the location was not on a City-owned street (i.e. on a highway / off-street bike path / bridge)."},{"name":"street","type":"text. (if location_type = \"Street\") Name of city-owned street as reported in 911 system"},{"name":"xstreet1","type":"text. (if location_type = \"Street\" or \"Intersection\") Name of 1st cross-street"},{"name":"xstreet2","type":"text. (if location_type = \"Street\" or \"Intersection\") Name of 2nd cross-street"},{"type":"text. X coordinate value of location in Massachusetts State Plane projection. ","name":"x_cord"},{"name":"y_cord","type":"text. Y coordinate value of location in Massachusetts State Plane projection. "},{"type":"text. Latitude of location. This field represents a transformation of snapping the Y coordinate to the closest street as well as converting it into WGS84 projection. ","name":"lat"},{"type":"text. Longitude of location. This field represents a transformation of snapping the X coordinate to the closest street as well as converting it into WGS84 projection. ","name":"long"}],"format":"v2-hybrid","generatedAt":"2026-08-16T17:39:25.112Z","sampleRows":[{"dispatch_ts":"2016-09-17 23:59:31+00","y_cord":"2913688.580000000000","_id":1,"xstreet1":"TRUMAN PKWY","lat":"42.24270031686706","long":"-71.12622780252731","__schema":"crash records","x_cord":"757374.390000000000","location_type":"Intersection","__resourceId":"e4bfe397-6bfc-49c5-9367-c879fac7401d","street":null,"mode_type":"mv","xstreet2":"WAKEFIELD AVE"},{"xstreet1":"BROOKLEDGE ST","__schema":"fatality records","date_time":"2021-08-27 23:16:04+00","xstreet2":"SEAVER ST","x_cord":"766700.610000000000","long":"-71.09175104914614","__resourceId":"92f18923-d4ec-4c17-9405-4e0da63e1d6c","lat":"42.31035694234807","_id":1,"y_cord":"2938321.020000000000","mode_type":"mv","street":"HUMBOLDT AVE","location_type":"Street"}],"sampleObjects":["e4bfe397-6bfc-49c5-9367-c879fac7401d  Vision Zero Crash Records","92f18923-d4ec-4c17-9405-4e0da63e1d6c  Vision Zero Fatality Records"],"rowCount":44505,"accessHints":[{"language":"python","snippet":"import io, json, urllib.error, urllib.parse, urllib.request\nimport pandas as pd\n\nCKAN = \"https://data.boston.gov/api/3/action\"\n\ndef ckan(action, **params):\n    \"\"\"Any CKAN action endpoint. Plain urllib works; no auth, no headers.\"\"\"\n    url = f\"{CKAN}/{action}?\" + urllib.parse.urlencode(params)\n    with urllib.request.urlopen(url, timeout=300) as r:\n        return json.load(r)[\"result\"]\n\ndef sql(query):\n    \"\"\"Server-side SQL over the DataStore. Warns when the 32,000-row cap clips.\"\"\"\n    res = ckan(\"datastore_search_sql\", sql=query)\n    if res.get(\"records_truncated\"):\n        print(\"WARNING: clipped at the 32,000-row cap; aggregate or page with OFFSET\")\n    return pd.DataFrame(res[\"records\"])\n"},{"snippet":"# Bulk CSV download. pd.read_csv(url) on this portal raises 403: the CKAN link\n# 302s to a presigned S3 URL that the redirect fails on. Resolve it by hand and\n# drop the explicit :443, which is the single change that makes it a 200.\nclass _NoRedirect(urllib.request.HTTPRedirectHandler):\n    def redirect_request(self, *a, **k):\n        return None\n\ndef boston_csv(url, **read_csv_kwargs):\n    try:\n        urllib.request.build_opener(_NoRedirect).open(url, timeout=120)\n        signed = url\n    except urllib.error.HTTPError as e:\n        if e.code not in (301, 302, 303, 307, 308):\n            raise\n        signed = e.headers[\"Location\"].replace(\"s3.amazonaws.com:443\", \"s3.amazonaws.com\")\n    with urllib.request.urlopen(signed, timeout=900) as r:\n        return pd.read_csv(io.BytesIO(r.read()), **read_csv_kwargs)\n","language":"python"},{"language":"python","snippet":"# Resolve a resource id by package name rather than hardcoding it, and page\n# past the 32,000-row cap. Verified against a 282,836-row resource.\ndef resources(package_name):\n    pkg = ckan(\"package_show\", id=package_name)\n    return [\n        {\"id\": r[\"id\"], \"name\": r.get(\"name\"), \"file\": r[\"url\"].rsplit(\"/\", 1)[-1],\n         \"format\": r.get(\"format\"), \"live\": bool(r.get(\"datastore_active\"))}\n        for r in pkg[\"resources\"]\n    ]\n\ndef read_all(resource_id, page=32000, **filters):\n    \"\"\"Every row of a DataStore resource, in _id order.\"\"\"\n    out, offset = [], 0\n    while True:\n        res = ckan(\"datastore_search\", resource_id=resource_id,\n                   limit=page, offset=offset, **filters)\n        rows = res[\"records\"]\n        out.extend(rows)\n        if len(rows) < page:\n            return pd.DataFrame(out)\n        offset += page\n"}]},"summary":"Traffic crashes in Boston that required a public safety response, published as part of the Vision Zero Boston programme: 44,344 crash records from 1 January 2015 to the present,…","title":"Boston Vision Zero crash and fatality records, 2015 to present","updatedAt":"2026-08-16T17:40:34.756Z","visibility":"public","websiteUrl":"https://www.boston.gov/departments/transportation/vision-zero","whiteLabel":"boston"}}