{"success":true,"data":{"author":"City of Boston (Inspectional Services Department)","bucketAccessType":"public","bucketListable":false,"bucketName":"data.boston.gov","category":"Urban & Civic","cloudProvider":"api","createdAt":"2026-08-15T00:00:00.000Z","description":"Every building permit approved by the City of Boston Inspectional Services Department from 26 September 2006 to the present: 659,934 rows in a single DataStore-loaded resource, refreshed daily. One row is one approved permit. Each carries the permit number, a work-type code (INTEXT, ERECT, EXTREN, RAZE, ELECTRICAL, PLUMBING, and about thirty others), the permit type description (long form, short form, amendment, certificate of occupancy, electrical, plumbing, gas, fire alarm), a free-text description and applicant comments, the applicant name, the declared valuation and total fees, issued and expiration dates, permit status, occupancy type, square footage, the full address with ZIP and ward, the assessing property and parcel identifiers, and coordinates in both Massachusetts State Plane (gpsx, gpsy) and WGS84 (x_longitude, y_latitude). Because the parcel_id and property_id columns join directly to the Property Assessment rolls and to the 2025 parcel geometry, this dataset is the standard instrument for measuring physical investment in the city: construction and renovation intensity by neighborhood over two decades, the pipeline of new housing units, teardown and conversion activity, and the lag between permit issue and completion. Two columns need care and are covered in the schema card. declared_valuation and total_fees are text formatted as currency ('$36,500.00') including negative amendments, so they need the dollar sign and commas stripped before a cast; the declared valuation is what the applicant self-reported at filing, not an appraised cost, and amendment permits restate rather than add to the original. issued_date and expiration_date are the only genuinely typed timestamp columns in the resource, which makes them safe to compare directly, while sq_feet is zero for the large majority of permits rather than null. Published by the Inspectional Services Department under PDDL 1.0.","displayTitle":"Boston Approved Building Permits, 2006 to present","dockerImageUrl":"531775725781.dkr.ecr.us-east-1.amazonaws.com/scigantic-notebook-minimal:latest","id":"0756a907-6919-5e01-b198-3fa99342138d","license":"Open Data Commons Public Domain Dedication and License (PDDL) 1.0","referenceUrl":"https://data.boston.gov/dataset/approved-building-permits","region":"us-east-1","schemaCard":{"rowCount":659934,"format":"v2-hybrid","sampleRows":[{"_id":1,"expiration_date":"2021-07-28T04:00:00","worktype":"INTEXT","applicant":"Patrick Sharkey","__schema":"approved building permits","sq_feet":0,"city":"Boston","description":"Interior/Exterior Work","permitnumber":"A1000569","state":"MA","zip":"02109","gpsx":777000.4679102153,"x_longitude":-71.05292400062602,"y_latitude":42.35919000001041,"parcel_id":303807000,"issued_date":"2021-01-28T16:29:26","status":"Closed","occupancytype":"Mixed","address":"181-183 State ST","ward":3,"property_id":130392,"gpsy":2956234.6530768126,"permittypedescr":"Amendment to a Long Form","__resourceId":"6ddcd912-32a0-43df-9908-63574f8c7e77","total_fees":"$390.00","declared_valuation":"$36,500.00","comments":"This work is to Amend Permit ALT347244. Eliminate construction of two party wall openings. Install new wheelchair lift.;;;;; E- Plans"}],"accessHints":[{"language":"python","snippet":"import io, json, urllib.error, urllib.parse, urllib.request\nimport pandas as pd\n\nCKAN = \"https://data.boston.gov/api/3/action\"\n\ndef ckan(action, **params):\n    \"\"\"Any CKAN action endpoint. Plain urllib works; no auth, no headers.\"\"\"\n    url = f\"{CKAN}/{action}?\" + urllib.parse.urlencode(params)\n    with urllib.request.urlopen(url, timeout=300) as r:\n        return json.load(r)[\"result\"]\n\ndef sql(query):\n    \"\"\"Server-side SQL over the DataStore. Warns when the 32,000-row cap clips.\"\"\"\n    res = ckan(\"datastore_search_sql\", sql=query)\n    if res.get(\"records_truncated\"):\n        print(\"WARNING: clipped at the 32,000-row cap; aggregate or page with OFFSET\")\n    return pd.DataFrame(res[\"records\"])\n"},{"language":"python","snippet":"# Bulk CSV download. pd.read_csv(url) on this portal raises 403: the CKAN link\n# 302s to a presigned S3 URL that the redirect fails on. Resolve it by hand and\n# drop the explicit :443, which is the single change that makes it a 200.\nclass _NoRedirect(urllib.request.HTTPRedirectHandler):\n    def redirect_request(self, *a, **k):\n        return None\n\ndef boston_csv(url, **read_csv_kwargs):\n    try:\n        urllib.request.build_opener(_NoRedirect).open(url, timeout=120)\n        signed = url\n    except urllib.error.HTTPError as e:\n        if e.code not in (301, 302, 303, 307, 308):\n            raise\n        signed = e.headers[\"Location\"].replace(\"s3.amazonaws.com:443\", \"s3.amazonaws.com\")\n    with urllib.request.urlopen(signed, timeout=900) as r:\n        return pd.read_csv(io.BytesIO(r.read()), **read_csv_kwargs)\n"},{"snippet":"# Resolve a resource id by package name rather than hardcoding it, and page\n# past the 32,000-row cap. Verified against a 282,836-row resource.\ndef resources(package_name):\n    pkg = ckan(\"package_show\", id=package_name)\n    return [\n        {\"id\": r[\"id\"], \"name\": r.get(\"name\"), \"file\": r[\"url\"].rsplit(\"/\", 1)[-1],\n         \"format\": r.get(\"format\"), \"live\": bool(r.get(\"datastore_active\"))}\n        for r in pkg[\"resources\"]\n    ]\n\ndef read_all(resource_id, page=32000, **filters):\n    \"\"\"Every row of a DataStore resource, in _id order.\"\"\"\n    out, offset = [], 0\n    while True:\n        res = ckan(\"datastore_search\", resource_id=resource_id,\n                   limit=page, offset=offset, **filters)\n        rows = res[\"records\"]\n        out.extend(rows)\n        if len(rows) < page:\n            return pd.DataFrame(out)\n        offset += page\n","language":"python"}],"starterCell":"import io, json, urllib.error, urllib.parse, urllib.request\nimport pandas as pd\n\nCKAN = \"https://data.boston.gov/api/3/action\"\n\ndef ckan(action, **params):\n    \"\"\"Any CKAN action endpoint. Plain urllib works; no auth, no headers.\"\"\"\n    url = f\"{CKAN}/{action}?\" + urllib.parse.urlencode(params)\n    with urllib.request.urlopen(url, timeout=300) as r:\n        return json.load(r)[\"result\"]\n\ndef sql(query):\n    \"\"\"Server-side SQL over the DataStore. Warns when the 32,000-row cap clips.\"\"\"\n    res = ckan(\"datastore_search_sql\", sql=query)\n    if res.get(\"records_truncated\"):\n        print(\"WARNING: clipped at the 32,000-row cap; aggregate or page with OFFSET\")\n    return pd.DataFrame(res[\"records\"])\n\nPERMITS = \"6ddcd912-32a0-43df-9908-63574f8c7e77\"   # 659,934 rows, 2006 to present\n\n# Twenty years of permitting activity and declared value by work type.\n# issued_date is a real timestamp here, so date_part needs no cast.\nval = \"replace(replace(\\\"declared_valuation\\\", '$', ''), ',', '')::numeric\"\ndf = sql(f'''\n    SELECT date_part(\\'year\\', \"issued_date\")::int AS yr,\n           \"worktype\",\n           count(*) AS permits,\n           sum({val}) AS declared_valuation\n    FROM \"{PERMITS}\"\n    WHERE \"issued_date\" IS NOT NULL\n      AND \"declared_valuation\" IS NOT NULL AND \"declared_valuation\" <> \\'\\'\n    GROUP BY 1, 2\n''')\ntop = df.groupby(\"worktype\")[\"permits\"].sum().nlargest(8).index\nprint(df[df.worktype.isin(top)].pivot_table(\n    index=\"yr\", columns=\"worktype\", values=\"permits\", aggfunc=\"sum\").fillna(0).astype(int))\n","generatedAt":"2026-08-16T17:39:25.112Z","layout":{"example":"Annual declared construction value, cleaned:\n  SELECT date_part('year', \"issued_date\") AS yr, count(*) permits,\n         sum(replace(replace(\"declared_valuation\", '$', ''), ',', '')::numeric) AS declared\n  FROM \"6ddcd912-32a0-43df-9908-63574f8c7e77\"\n  WHERE \"declared_valuation\" IS NOT NULL AND \"declared_valuation\" <> ''\n  GROUP BY 1 ORDER BY 1","pattern":"One row is one approved building permit. 659,934 rows in a single DataStore resource, verified range 2006-09-26 to 2026-08-14 and refreshed daily. Unusually for this portal, the numeric and date columns are GENUINELY TYPED: issued_date and expiration_date are timestamps, and sq_feet, ward, property_id, parcel_id, gpsy, gpsx, y_latitude and x_longitude are numeric, so they need no cast. declared_valuation and total_fees are the exception and are text currency.\nWHAT EACH ROW CARRIES: permitnumber; worktype (a short code such as INTEXT, ERECT, EXTREN, RAZE, INTDEM, EXTDEM, SRVCHG, FSTTRK, SOL, INSUL); permittypedescr (the human form: \"Amendment to a Long Form\", \"Short Form Bldg Permit\", \"Electrical Permit\", \"Plumbing Permit\", \"Certificate of Occupancy\"); description; comments (applicant free text, often carrying the amended permit number); applicant; declared_valuation; total_fees; issued_date; expiration_date; status; occupancytype; sq_feet; address, city, state, zip, ward; property_id and parcel_id; and coordinates in both Massachusetts State Plane (gpsx, gpsy) and WGS84 (x_longitude, y_latitude).\nTHE JOINS ARE THE POINT. parcel_id and property_id are the same identifiers used by the Property Assessment rolls (GIS_ID / PID) and by RentSmart, and parcel_id maps to MAP_PAR_ID in the 2025 parcel geometry. That chain is what turns a permit into an owner, an assessed value, a building age and a polygon on a map.\nRESOURCES:\n  Approved Building Permits (data.boston.gov/dataset/approved-building-permits): 1 DataStore resources of 2 total\n    6ddcd912-32a0-43df-9908-63574f8c7e77     659934 rows  Approved Building Permits"},"columns":[{"type":"int","name":"_id"},{"type":"text","name":"permitnumber"},{"type":"text","name":"worktype"},{"name":"permittypedescr","type":"text"},{"type":"text","name":"description"},{"type":"text","name":"comments"},{"name":"applicant","type":"text"},{"type":"text","name":"declared_valuation"},{"type":"text","name":"total_fees"},{"type":"timestamp","name":"issued_date"},{"type":"timestamp","name":"expiration_date"},{"name":"status","type":"text"},{"type":"text","name":"occupancytype"},{"name":"sq_feet","type":"numeric"},{"type":"text","name":"address"},{"type":"text","name":"city"},{"name":"state","type":"text"},{"type":"text","name":"zip"},{"name":"ward","type":"numeric"},{"type":"numeric","name":"property_id"},{"name":"parcel_id","type":"numeric"},{"type":"numeric","name":"gpsy"},{"name":"gpsx","type":"numeric"},{"name":"y_latitude","type":"numeric"},{"name":"x_longitude","type":"numeric"}],"caveats":["DO NOT look for a FUSE mount. This archive has cloudProvider \"api\": nothing is mounted, ~/archive and /mnt/archive do not exist, and there are no files on disk to walk. The generic archive preamble you may have been given says otherwise; it is wrong for this row. All data is read over HTTPS from the CKAN API at https://data.boston.gov/api/3/action. The notebook has outbound internet; no credentials, tokens or headers are required for any endpoint below.","DO NOT trust a result set of exactly 32,000 rows. Both datastore_search and datastore_search_sql cap a single response at 32,000 records no matter what LIMIT you write, so a query over a million-row resource returns a full-looking frame that is silently a fragment. datastore_search_sql sets records_truncated: true in the result when it clips; check it on every call. The fixes: aggregate server-side (GROUP BY in SQL returns few enough rows to fit), or page with ORDER BY _id plus OFFSET, which is verified to work past 280,000.","DO NOT do arithmetic or date comparison on a column without casting it. Almost every DataStore column on this portal is typed text regardless of what it holds, so avg(\"latitude\") returns HTTP 409 Conflict rather than a number, and min/max on a date column sorts lexically and returns a confident wrong answer. Cast explicitly in SQL: \"latitude\"::float, \"open_dt\"::timestamp, date_part('year', \"open_dt\"::timestamp). Where a column IS genuinely typed numeric or timestamp it is marked as such in the columns list below, and those are safe to use directly.","SQL AGGREGATES COME BACK AS STRINGS, so arithmetic on the resulting DataFrame silently fails or, worse, concatenates. PostgreSQL `numeric` results are serialized as JSON strings by the CKAN API (sum(\"OVERTIME\") returns '153220443.25', not 153220443.25) while `int`/`bigint` and `double precision` come back as JSON numbers. So count(*) is an int, avg() and sum() over a numeric column are strings, and dividing one such column by another raises \"unsupported operand type(s) for /: 'str' and 'str'\". Run pd.to_numeric on every aggregate column before doing arithmetic on it. Do NOT blanket-coerce every column: ward \"06\", precinct \"0104\" and ZIP \"02128\" are zero-padded identifiers that must stay strings.","A 403 FROM datastore_search_sql MEANS THE SQL TEXT TRIPPED AN EDGE FILTER, NOT THAT YOUR QUERY IS WRONG. Some perfectly valid PostgreSQL is rejected with HTTP 403 Forbidden before it ever reaches the database, so there is no SQL error message to read and retrying the identical query never helps. Verified blocked, deterministically, 5 of 5 attempts each (2026-08-15): agg(...) FILTER (WHERE ...) · extract(<part> FROM ...) · substring(x FROM a FOR b) · trim(BOTH ... FROM ...) · NULLIF(...) · a literal constant comparison such as 1=2. Verified working substitutes for each: sum(CASE WHEN cond THEN 1 ELSE 0 END) for FILTER · date_part('year', ts) and date_part('epoch', ts_a - ts_b) for extract · left(col, n) for substring · CASE WHEN d = 0 THEN NULL ELSE d END for NULLIF. Subqueries, CTEs, LEFT JOIN, IN (...), HAVING, window functions, percentile_cont WITHIN GROUP, the regex operator ~, and casts are all fine. When a query 403s, rewrite it; do not retry it.","DO NOT call pd.read_csv() on a data.boston.gov download URL. It raises HTTP 403. The CKAN download link 302-redirects to a presigned S3 URL on the bucket og-production-open-data-bostonma-892364687672, and the request fails when it is followed from urllib's redirect path, which is what pandas uses. Verified 2026-08-15: pd.read_csv(url) -> 403, while requests.get(url) on the identical URL -> 200. Two fixes, both verified: read through requests (pd.read_csv(io.BytesIO(requests.get(url).content))), or resolve the redirect yourself and strip the explicit \":443\" from the returned Location before fetching it. The bucket itself is private (anonymous ListObjectsV2 and GetObject both return AccessDenied), so there is no S3 path around this: the presigned link is the only way in. See accessHints for a copy-paste helper. None of this applies to the /api/3/action endpoints, which are plain JSON over HTTPS and work with bare urllib.","Resource ids, not names, are the addressing key, and the names are unreliable: CKAN's resource `name` field is null on many Boston resources, and several packages publish generated filenames (tmp4myyj_u8.csv) that carry no year. The ids listed in this card were captured on 2026-08-15. The package NAME is the stable handle, so resolve ids at runtime with GET /api/3/action/package_show?id=<package-name> and match on the resource `name` or `url` rather than hardcoding an index into the resource array, whose order is not guaranteed.","_id is CKAN's synthetic 1-based row number added by the DataStore loader, not a key from the source data. It is the right column to ORDER BY for stable paging and it is meaningless as a join key or an entity id.","declared_valuation is text formatted as currency (\"$36,500.00\") and includes NEGATIVE amounts on amendments that reduce scope. Strip \"$\" and \",\" then cast to numeric, and do not filter negatives out as bad data: they are real corrections and dropping them overstates investment.","DECLARED VALUATION IS NOT COST. It is the figure the applicant stated at filing, used to compute the permit fee. It is systematically understated for some work types, is 0 or null for many trade permits, and is never revised to actual cost. Treat it as an indicator of activity, not a construction spend estimate.","AMENDMENTS RESTATE, THEY DO NOT ADD. permittypedescr \"Amendment to a Long Form\" rows reference an earlier permit in the free-text comments field (\"This work is to Amend Permit ALT347244\"). Summing valuation across a permit and its amendments double counts the project. There is no structured parent-permit column; the link exists only in that free text.","sq_feet is 0 rather than null for the large majority of permits, so mean(sq_feet) is dominated by zeros. Filter sq_feet > 0 before any size statistic.","\"Approved\" is the publication threshold, not completion. A permit here was issued; whether the work happened, and when, is not in this dataset. status and expiration_date are the closest proxies and status is a small uncontrolled vocabulary (\"Open\", \"Closed\", \"Issued\").","The dataset covers permits from the current permitting system back to 2006-09-26. Anything described as a Boston building permit before that date is not here."],"sampleObjects":["6ddcd912-32a0-43df-9908-63574f8c7e77  Approved Building Permits"],"totalObjects":1},"summary":"Every building permit approved by the City of Boston Inspectional Services Department from 26 September 2006 to the present: 659,934 rows in a single DataStore-loaded resource,…","title":"Boston Approved Building Permits, 2006 to present","updatedAt":"2026-08-16T17:40:38.368Z","visibility":"public","websiteUrl":"https://www.boston.gov/departments/inspectional-services","whiteLabel":"boston"}}