Heat exposure and cohort change: 2003 and later years¶
Compare a Milan heat-exposure proxy with Italy's annual cohort change, extending the original 2002 analysis through the latest complete common year. This is an exploratory ecological comparison, not a count or causal estimate of heat-related deaths.
The original notebook shifted cohort-change years back by one. That was incorrect: notebook 14's column y already describes the change from 1 January y to 1 January y+1, so it is paired directly with weather during y.
import sys
from pathlib import Path
sys.path.insert(0, str(Path("../scripts").resolve()))
import json
import pandas as pd
import matplotlib.pyplot as plt
from matplotlib.ticker import PercentFormatter
import plotly.express as px
import plotly.io as pio
from weather_source import analysis_period, load_weather
from heatwave_analysis import annual_heat_exposure, align_cohort_changes, correlations, age_sensitivity
pio.renderers.default = "plotly_mimetype+notebook"
pd.options.plotting.backend = "plotly"
# Cached national demographic inputs from notebook 14. No ISTAT request is needed.
dfp = pd.read_csv("../data/pop_by_age_year.csv", index_col=0).rename(columns=int)
dfpcr = pd.read_csv("../data/ccr_by_age_year.csv", index_col=0).rename(columns=int)
start_year, last_year = analysis_period(dfpcr.columns)
assert last_year + 1 in dfp.columns
print(f"Matched complete calendar years: {start_year}–{last_year}")
print(f"Cohort transitions end on 1 January {last_year + 1}")
Milan weather: a consistent ERA5 reanalysis series¶
Daily minimum/maximum 2 m temperatures come from Open-Meteo's Historical Weather API, explicitly selecting models=era5, Celsius and Europe/Rome. These are gridded reanalysis estimates near Milan, not weather-station measurements or national exposure.
No API key is needed for the public noncommercial service. Attribution: ERA5 / ECMWF / Copernicus C3S, distributed by Open-Meteo (CC BY 4.0). This notebook derives monthly means and annual heat-exposure sums from daily data.
Downloads and query/retrieval manifests remain in gitignored data/sources/open_meteo/. Matching same-day queries reuse the cache. The runner fetches six-year blocks with no retries, a 45-second total deadline and a 31.1-second gap after each response. Long date windows count as multiple provider calls, so a multi-decade request is not sent as a single large query. An ordinary notebook execution is offline.
Refresh only this investigation with python scripts/refresh_model.py --heatwaves --refresh from the repository root; omit --refresh to use the local cache.
df, weather_provenance = load_weather(start_year, last_year)
print(f"Complete daily weather records: {len(df):,}; {df.time.min().date()} to {df.time.max().date()}")
print("Returned ERA5 grid:", weather_provenance["queries"][0]["returned_grid"])
print("Retrieved at:", [q["retrieved_at"] for q in weather_provenance["queries"]])
# Publish a derived monthly view, leaving the raw daily download in the ignored cache.
monthly = df.set_index("time")[["temperature_2m_min", "temperature_2m_max"]].resample("MS").mean()
fig = px.line(monthly, labels={"value": "Monthly mean of daily temperature (°C)", "time": "Month"})
fig.update_layout(title="Milan ERA5 grid: monthly means of daily minimum/maximum temperatures", width=900, height=380)
fig.show()
Annual heat-exposure proxy¶
For each complete calendar year, sum max(daily Tmax − 30°C, 0). The unit is °C·days. This measures accumulated heat above a fixed threshold, rather than identifying consecutive-day heatwave events. Years with no days above the threshold remain in the comparison with exposure zero. Incomplete years and missing/duplicated days are rejected.
df_hw = annual_heat_exposure(df, start_year, last_year)
fig = px.bar(df_hw.reset_index(), x="year", y="HeatWaveProxy", hover_data=["days_above_30"])
fig.update_layout(title="Milan ERA5 annual heat exposure above 30°C", xaxis_title="Calendar year",
yaxis_title="Σ max(Tmax − 30°C, 0) (°C·days)", width=900, height=380)
fig.show()
Match weather to the same year's cohort transition¶
Use the unweighted mean cohort change rate (CCR) for destination ages 85–99, as recorded in notebook 14. For destination age a, column y compares residents aged a−1 on 1 January y with residents aged a on 1 January y+1.
Define cohort decline = −mean CCR, so larger positive values mean a larger proportional reduction. This includes deaths, migration and statistical revisions; it must not be labelled as a mortality rate. Every complete weather year is matched directly to its CCR column, with no year offset.
min_age, max_age = 85, 99
dfcorr = align_cohort_changes(df_hw, dfpcr, min_age, max_age)
dfcorr["covid_period"] = dfcorr.year.isin([2020, 2021]).map({True: "2020–2021", False: "Other years"})
display(dfcorr)
fig = px.scatter(dfcorr, x="year", y="cohort_decline", color="HeatWaveProxy", symbol="covid_period",
hover_data=["year", "HeatWaveProxy", "days_above_30"], color_continuous_scale="YlOrRd")
fig.update_layout(title="National cohort decline over time (destination ages 85–99)",
yaxis_title="−mean CCR (includes migration/revisions)", yaxis_tickformat=".1%", width=850, height=420)
fig.show()
fig = px.scatter(dfcorr, x="HeatWaveProxy", y="cohort_decline", color="year", symbol="covid_period",
hover_data=["year", "mean_ccr", "days_above_30"], color_continuous_scale="Viridis")
fig.update_layout(title=f"Milan heat exposure and Italy cohort decline, {start_year}–{last_year}",
xaxis_title="Milan annual heat exposure (°C·days above 30°C)",
yaxis_title="−mean CCR, destination ages 85–99", yaxis_tickformat=".1%", width=850, height=420)
fig.write_html("../images_output/heatwave_cohort_change.html")
fig.show()
# A standalone scientific figure for inspection/sharing.
fig_static, ax = plt.subplots(figsize=(8.5, 5.5), layout="constrained")
points = ax.scatter(dfcorr.HeatWaveProxy, dfcorr.cohort_decline, c=dfcorr.year, cmap="viridis", s=50)
for row in dfcorr.itertuples():
if row.year in {2003, 2020, 2021, 2022, last_year}:
offset = {2003: (-35, -15), 2021: (-25, -17)}.get(row.year, (5, 5))
ax.annotate(str(row.year), (row.HeatWaveProxy, row.cohort_decline), xytext=offset, textcoords="offset points", fontsize=9)
ax.set_xlabel("Milan heat exposure (°C·days above 30°C)")
ax.set_ylabel("Cohort decline: −mean CCR (destination ages 85–99)")
ax.yaxis.set_major_formatter(PercentFormatter(1))
ax.set_title(f"Heat exposure and cohort decline, {start_year}–{last_year}")
ax.grid(alpha=0.2)
fig_static.colorbar(points, ax=ax, label="Calendar year")
fig_static.savefig("../images_output/heatwave_cohort_change.png", dpi=180)
plt.show()
Signed correlations and sensitivity to exceptional years¶
Report signed Spearman correlations: a positive heat/decline coefficient means greater heat exposure accompanies a larger cohort reduction. Keep the sign; the earlier absolute-value plot hid the direction of association.
Compare all complete years with a reference excluding 2020–2021, and repeat after also excluding 2003. Removing exceptional years is only a sensitivity check. Calendar trends and other confounders remain; no causal or statistical-significance claim follows from these coefficients. The 50 overlapping age groups below are exploratory, not independent tests or a basis for selecting a definitive age threshold.
statistics = correlations(dfcorr)
display(statistics)
dfcorrg = age_sensitivity(df_hw, dfpcr)
fig = px.line(dfcorrg, x="minimum_age", y=["heat_exposure", "calendar_year", "heat_without_2003"],
hover_data=["age_group"])
fig.update_layout(title="Age sensitivity of signed Spearman correlations (reference excludes 2020–2021)",
xaxis_title="Minimum destination age (upper age is 99)", yaxis_title="Spearman correlation",
yaxis_range=[-1, 1], legend_title="Compared with cohort decline", width=950, height=430)
fig.write_html("../images_output/heatwave_age_sensitivity.html")
fig.show()
row_2003 = dfcorr.set_index("year").loc[2003]
summary = {
"years": [start_year, last_year], "n_years": len(dfcorr), "daily_weather_records": len(df),
"weather": weather_provenance,
"age_group": {"minimum_destination_age": min_age, "maximum_destination_age": max_age, "weighting": "unweighted mean"},
"alignment": "Weather year y matches CCR column y: 1 January y to 1 January y+1; no year shift",
"heat_metric": "sum(max(daily Tmax Celsius - 30, 0)); units Celsius-days",
"correlations": statistics.to_dict(orient="records"),
"year_2003": {"heat_exposure": float(row_2003.HeatWaveProxy), "cohort_decline": float(row_2003.cohort_decline),
"heat_rank_hottest_first": int(df_hw.HeatWaveProxy.rank(ascending=False, method="min").loc[2003])},
"hottest_proxy_year": int(df_hw.HeatWaveProxy.idxmax()),
"interpretation": "Exploratory ecological association; cohort decline includes migration/revisions and is not mortality",
}
with open("../data/heatwave_analysis_summary.json", "w") as output:
json.dump(summary, output, indent=2, allow_nan=False)
print(f"2003 heat exposure: {summary['year_2003']['heat_exposure']:.1f} °C·days; rank {summary['year_2003']['heat_rank_hottest_first']} of {len(dfcorr)}")
print("Saved methods, source provenance and statistics to data/heatwave_analysis_summary.json")
print("Heat/decline Spearman correlations:", "; ".join(f"{r.sample}: rho={r.heat_vs_decline_spearman:.3f}, n={r.n_years}" for r in statistics.itertuples()))
Conclusions and limitations¶
The current numerical results are shown above and saved in data/heatwave_analysis_summary.json. They use complete common years, the corrected same-year alignment, and signed correlations with explicit sample sizes. The 2003 sensitivity check shows how much the association depends on that exceptional year.
A single reanalysis grid point near Milan is a limited proxy for heat exposure across Italy. National cohort changes combine mortality, migration, measurement and census revisions. Temporal trends, age composition and the pandemic complicate interpretation. This analysis therefore cannot identify how many deaths were caused by heat or establish causation from the scatter plot.
Future work: compare geographically representative temperature series with actual age-specific deaths or death probabilities, account for temporal trends, and assess uncertainty and sensitivity before drawing stronger conclusions.