← Home

Heat exposure and cohort change: 2003 and later years

Compare a Milan heat-exposure proxy with Italy's annual cohort change, extending the original 2002 analysis through the latest complete common year. This is an exploratory ecological comparison, not a count or causal estimate of heat-related deaths.

The original notebook shifted cohort-change years back by one. That was incorrect: notebook 14's column y already describes the change from 1 January y to 1 January y+1, so it is paired directly with weather during y.

In [1]:
import sys
from pathlib import Path
sys.path.insert(0, str(Path("../scripts").resolve()))
import json
import pandas as pd
import matplotlib.pyplot as plt
from matplotlib.ticker import PercentFormatter
import plotly.express as px
import plotly.io as pio
from weather_source import analysis_period, load_weather
from heatwave_analysis import annual_heat_exposure, align_cohort_changes, correlations, age_sensitivity

pio.renderers.default = "plotly_mimetype+notebook"
pd.options.plotting.backend = "plotly"
In [2]:
# Cached national demographic inputs from notebook 14. No ISTAT request is needed.
dfp = pd.read_csv("../data/pop_by_age_year.csv", index_col=0).rename(columns=int)
dfpcr = pd.read_csv("../data/ccr_by_age_year.csv", index_col=0).rename(columns=int)
start_year, last_year = analysis_period(dfpcr.columns)
assert last_year + 1 in dfp.columns
print(f"Matched complete calendar years: {start_year}–{last_year}")
print(f"Cohort transitions end on 1 January {last_year + 1}")
Matched complete calendar years: 2002–2025
Cohort transitions end on 1 January 2026

Milan weather: a consistent ERA5 reanalysis series

Daily minimum/maximum 2 m temperatures come from Open-Meteo's Historical Weather API, explicitly selecting models=era5, Celsius and Europe/Rome. These are gridded reanalysis estimates near Milan, not weather-station measurements or national exposure.

No API key is needed for the public noncommercial service. Attribution: ERA5 / ECMWF / Copernicus C3S, distributed by Open-Meteo (CC BY 4.0). This notebook derives monthly means and annual heat-exposure sums from daily data.

Downloads and query/retrieval manifests remain in gitignored data/sources/open_meteo/. Matching same-day queries reuse the cache. The runner fetches six-year blocks with no retries, a 45-second total deadline and a 31.1-second gap after each response. Long date windows count as multiple provider calls, so a multi-decade request is not sent as a single large query. An ordinary notebook execution is offline.

Refresh only this investigation with python scripts/refresh_model.py --heatwaves --refresh from the repository root; omit --refresh to use the local cache.

In [3]:
df, weather_provenance = load_weather(start_year, last_year)
print(f"Complete daily weather records: {len(df):,}; {df.time.min().date()} to {df.time.max().date()}")
print("Returned ERA5 grid:", weather_provenance["queries"][0]["returned_grid"])
print("Retrieved at:", [q["retrieved_at"] for q in weather_provenance["queries"]])
# Publish a derived monthly view, leaving the raw daily download in the ignored cache.
monthly = df.set_index("time")[["temperature_2m_min", "temperature_2m_max"]].resample("MS").mean()
fig = px.line(monthly, labels={"value": "Monthly mean of daily temperature (°C)", "time": "Month"})
fig.update_layout(title="Milan ERA5 grid: monthly means of daily minimum/maximum temperatures", width=900, height=380)
fig.show()
Complete daily weather records: 8,766; 2002-01-01 to 2025-12-31
Returned ERA5 grid: {'latitude': 45.5, 'longitude': 9.25, 'elevation': 142.0}
Retrieved at: ['2026-11-11T11:08:19.818717+00:00', '2026-11-11T11:08:51.421813+00:00', '2026-11-11T11:09:23.092337+00:00', '2026-11-11T11:09:54.844390+00:00']

Annual heat-exposure proxy

For each complete calendar year, sum max(daily Tmax − 30°C, 0). The unit is °C·days. This measures accumulated heat above a fixed threshold, rather than identifying consecutive-day heatwave events. Years with no days above the threshold remain in the comparison with exposure zero. Incomplete years and missing/duplicated days are rejected.

In [4]:
df_hw = annual_heat_exposure(df, start_year, last_year)
fig = px.bar(df_hw.reset_index(), x="year", y="HeatWaveProxy", hover_data=["days_above_30"])
fig.update_layout(title="Milan ERA5 annual heat exposure above 30°C", xaxis_title="Calendar year",
                  yaxis_title="Σ max(Tmax − 30°C, 0) (°C·days)", width=900, height=380)
fig.show()

Match weather to the same year's cohort transition

Use the unweighted mean cohort change rate (CCR) for destination ages 85–99, as recorded in notebook 14. For destination age a, column y compares residents aged a−1 on 1 January y with residents aged a on 1 January y+1.

Define cohort decline = −mean CCR, so larger positive values mean a larger proportional reduction. This includes deaths, migration and statistical revisions; it must not be labelled as a mortality rate. Every complete weather year is matched directly to its CCR column, with no year offset.

In [5]:
min_age, max_age = 85, 99
dfcorr = align_cohort_changes(df_hw, dfpcr, min_age, max_age)
dfcorr["covid_period"] = dfcorr.year.isin([2020, 2021]).map({True: "2020–2021", False: "Other years"})
display(dfcorr)
year HeatWaveProxy days_above_30 mean_ccr cohort_decline covid_period
0 2002 33.9 17 -0.194923 0.194923 Other years
1 2003 325.5 76 -0.209505 0.209505 Other years
2 2004 89.0 36 -0.185085 0.185085 Other years
3 2005 111.3 42 -0.195627 0.195627 Other years
4 2006 150.4 50 -0.183339 0.183339 Other years
5 2007 98.1 36 -0.184592 0.184592 Other years
6 2008 35.2 26 -0.184668 0.184668 Other years
7 2009 123.3 62 -0.179260 0.179260 Other years
8 2010 72.8 32 -0.171256 0.171256 Other years
9 2011 80.5 35 -0.169289 0.169289 Other years
10 2012 130.0 54 -0.185464 0.185464 Other years
11 2013 93.4 39 -0.177964 0.177964 Other years
12 2014 39.2 13 -0.172303 0.172303 Other years
13 2015 166.6 47 -0.187884 0.187884 Other years
14 2016 30.3 27 -0.172146 0.172146 Other years
15 2017 135.3 57 -0.181120 0.181120 Other years
16 2018 113.8 49 -0.172379 0.172379 Other years
17 2019 132.4 54 -0.174623 0.174623 Other years
18 2020 72.9 40 -0.195722 0.195722 2020–2021
19 2021 75.7 43 -0.182890 0.182890 2020–2021
20 2022 238.6 77 -0.194307 0.194307 Other years
21 2023 152.5 47 -0.176464 0.176464 Other years
22 2024 165.5 57 -0.171055 0.171055 Other years
23 2025 164.7 51 -0.158160 0.158160 Other years
In [6]:
fig = px.scatter(dfcorr, x="year", y="cohort_decline", color="HeatWaveProxy", symbol="covid_period",
                 hover_data=["year", "HeatWaveProxy", "days_above_30"], color_continuous_scale="YlOrRd")
fig.update_layout(title="National cohort decline over time (destination ages 85–99)",
                  yaxis_title="−mean CCR (includes migration/revisions)", yaxis_tickformat=".1%", width=850, height=420)
fig.show()

fig = px.scatter(dfcorr, x="HeatWaveProxy", y="cohort_decline", color="year", symbol="covid_period",
                 hover_data=["year", "mean_ccr", "days_above_30"], color_continuous_scale="Viridis")
fig.update_layout(title=f"Milan heat exposure and Italy cohort decline, {start_year}–{last_year}",
                  xaxis_title="Milan annual heat exposure (°C·days above 30°C)",
                  yaxis_title="−mean CCR, destination ages 85–99", yaxis_tickformat=".1%", width=850, height=420)
fig.write_html("../images_output/heatwave_cohort_change.html")
fig.show()

# A standalone scientific figure for inspection/sharing.
fig_static, ax = plt.subplots(figsize=(8.5, 5.5), layout="constrained")
points = ax.scatter(dfcorr.HeatWaveProxy, dfcorr.cohort_decline, c=dfcorr.year, cmap="viridis", s=50)
for row in dfcorr.itertuples():
    if row.year in {2003, 2020, 2021, 2022, last_year}:
        offset = {2003: (-35, -15), 2021: (-25, -17)}.get(row.year, (5, 5))
        ax.annotate(str(row.year), (row.HeatWaveProxy, row.cohort_decline), xytext=offset, textcoords="offset points", fontsize=9)
ax.set_xlabel("Milan heat exposure (°C·days above 30°C)")
ax.set_ylabel("Cohort decline: −mean CCR (destination ages 85–99)")
ax.yaxis.set_major_formatter(PercentFormatter(1))
ax.set_title(f"Heat exposure and cohort decline, {start_year}–{last_year}")
ax.grid(alpha=0.2)
fig_static.colorbar(points, ax=ax, label="Calendar year")
fig_static.savefig("../images_output/heatwave_cohort_change.png", dpi=180)
plt.show()
No description has been provided for this image

Signed correlations and sensitivity to exceptional years

Report signed Spearman correlations: a positive heat/decline coefficient means greater heat exposure accompanies a larger cohort reduction. Keep the sign; the earlier absolute-value plot hid the direction of association.

Compare all complete years with a reference excluding 2020–2021, and repeat after also excluding 2003. Removing exceptional years is only a sensitivity check. Calendar trends and other confounders remain; no causal or statistical-significance claim follows from these coefficients. The 50 overlapping age groups below are exploratory, not independent tests or a basis for selecting a definitive age threshold.

In [7]:
statistics = correlations(dfcorr)
display(statistics)
dfcorrg = age_sensitivity(df_hw, dfpcr)
fig = px.line(dfcorrg, x="minimum_age", y=["heat_exposure", "calendar_year", "heat_without_2003"],
              hover_data=["age_group"])
fig.update_layout(title="Age sensitivity of signed Spearman correlations (reference excludes 2020–2021)",
                  xaxis_title="Minimum destination age (upper age is 99)", yaxis_title="Spearman correlation",
                  yaxis_range=[-1, 1], legend_title="Compared with cohort decline", width=950, height=430)
fig.write_html("../images_output/heatwave_age_sensitivity.html")
fig.show()

row_2003 = dfcorr.set_index("year").loc[2003]
summary = {
    "years": [start_year, last_year], "n_years": len(dfcorr), "daily_weather_records": len(df),
    "weather": weather_provenance,
    "age_group": {"minimum_destination_age": min_age, "maximum_destination_age": max_age, "weighting": "unweighted mean"},
    "alignment": "Weather year y matches CCR column y: 1 January y to 1 January y+1; no year shift",
    "heat_metric": "sum(max(daily Tmax Celsius - 30, 0)); units Celsius-days",
    "correlations": statistics.to_dict(orient="records"),
    "year_2003": {"heat_exposure": float(row_2003.HeatWaveProxy), "cohort_decline": float(row_2003.cohort_decline),
                  "heat_rank_hottest_first": int(df_hw.HeatWaveProxy.rank(ascending=False, method="min").loc[2003])},
    "hottest_proxy_year": int(df_hw.HeatWaveProxy.idxmax()),
    "interpretation": "Exploratory ecological association; cohort decline includes migration/revisions and is not mortality",
}
with open("../data/heatwave_analysis_summary.json", "w") as output:
    json.dump(summary, output, indent=2, allow_nan=False)
print(f"2003 heat exposure: {summary['year_2003']['heat_exposure']:.1f} °C·days; rank {summary['year_2003']['heat_rank_hottest_first']} of {len(dfcorr)}")
print("Saved methods, source provenance and statistics to data/heatwave_analysis_summary.json")
print("Heat/decline Spearman correlations:", "; ".join(f"{r.sample}: rho={r.heat_vs_decline_spearman:.3f}, n={r.n_years}" for r in statistics.itertuples()))
sample n_years heat_vs_decline_spearman year_vs_decline_spearman
0 All complete years 24 0.106957 -0.457391
1 Exclude 2020–2021 22 0.186900 -0.590062
2 Also exclude 2003 21 0.064935 -0.531169
2003 heat exposure: 325.5 °C·days; rank 1 of 24
Saved methods, source provenance and statistics to data/heatwave_analysis_summary.json
Heat/decline Spearman correlations: All complete years: rho=0.107, n=24; Exclude 2020–2021: rho=0.187, n=22; Also exclude 2003: rho=0.065, n=21

Conclusions and limitations

The current numerical results are shown above and saved in data/heatwave_analysis_summary.json. They use complete common years, the corrected same-year alignment, and signed correlations with explicit sample sizes. The 2003 sensitivity check shows how much the association depends on that exceptional year.

A single reanalysis grid point near Milan is a limited proxy for heat exposure across Italy. National cohort changes combine mortality, migration, measurement and census revisions. Temporal trends, age composition and the pandemic complicate interpretation. This analysis therefore cannot identify how many deaths were caused by heat or establish causation from the scatter plot.

Future work: compare geographically representative temperature series with actual age-specific deaths or death probabilities, account for temporal trends, and assess uncertainty and sensitivity before drawing stronger conclusions.

← Home