← Home

Modelling the future resident population

AIM: extrapolate from the past data, the future resident population in Italy by age distribution, till year 2100.

I'm assuming a model like:

$ pop(a=0,y) = birthprop( pop(a=18:50,y-1), fr_{avg}(y-1) ) $

$ pop(a,y) = p(a-1,y-1) \cdot [1 - dp(a-1, y-1)] + mb(a, y-1) $

$ pop(a=100^{+}, y) = p(a=99,y-1) \cdot [1 - dp(99, y-1)] + p(a=100,y-1) \cdot [1 - dp_{100^+}(a, y-1)] $

Where:

  • $pop(a, y)$ is the resident population of age $a$ in year $y$
  • $birthprop$ is a function returning the number of births given a population in the fertile age range (18-50 yo) and an average fertility rate $fr_{avg}$
  • $dp(a, y)$ is the death probability for age $a$ in year $y$
  • $mb(a, y)$ is the Net Migration Balance, positive for more people immigrating than emigrating
  • $dp_{100^+}(a, y)$ is an effective death probability for people older than 100 years old combined

Moreover:

  • I will make 3 scenarios for the fertility rate (growing, constant, decreasing)
  • I will make 3 scenarios for the Net Migration Balance: no more migration from next year, constant migration, 2x migration from next year. The reference migration balance is the average of the last 20 years.
  • I will combine these 3x3=9 scenarios and see the results
In [1]:
import sys
from pathlib import Path
sys.path.insert(0, str(Path("../scripts").resolve()))

import numpy as np
import pandas as pd
import json
import matplotlib
import matplotlib.pyplot as plt
import plotly.express as px
import plotly.graph_objects as go
import plotly.colors as pc
import plotly.io as pio
from copy import deepcopy
from istat_source import istat_data
import warnings 



pio.renderers.default = 'plotly_mimetype+notebook'
pd.options.plotting.backend = "plotly"
warnings.filterwarnings("ignore", category=FutureWarning)

def get_colors(n, cmap_name="rainbow"):
    """Get colors for px colors_discrete argument, given the number of colors needed, n."""
    cmap = matplotlib.colormaps[cmap_name]
    colors = [cmap(i) for i in np.linspace(0, 1, n)]  # Generate colors
    colors_str = [f"rgba({int(color[0]*250)}, {int(color[1]*250)}, {int(color[2]*250)}, 1.0)" for color in colors]
    return colors_str

# convert default plotly colors to matplotlib
plotly_colors = pc.qualitative.Plotly
matplotlib_colors = []
for color in plotly_colors:
    r, g, b = pc.hex_to_rgb(color)
    matplotlib_colors.append((r/255, g/255, b/255))
In [2]:
# Recover data from previous notebooks
dfp=pd.read_csv("../data/pop_by_age_year.csv", index_col=0).rename(columns=int)
dfdpj=pd.read_csv("../data/deathprob_by_age_year_proj.csv", index_col=0).rename(columns=int)
dfmb = pd.read_csv("../data/average_migration_balance.csv", index_col=0).squeeze()
In [3]:
# Reference years I will be using for the analysis
start_year = dfp.columns[0] # 2002
last_year = dfp.columns[-1] # 2023 (or later if updated)
end_year = 2100

print(f"First-last observation: Jan/{start_year} - Jan/{last_year}")
print(f"First-last prediction:  Jan/{last_year+1} - Jan/{end_year}")
First-last observation: Jan/1952 - Jan/2026
First-last prediction:  Jan/2027 - Jan/2100
In [4]:
# Create coefficients for the migration balance
migration_coeff = {}
for year in range(last_year+1, end_year+1):
    migration_coeff[year] = {
        "double ": 2,
        "average": 1,
        "zero   ": 0,
        #f"↑ 5x in {end_year}": 1 + (5-1) / (end_year - last_year) * (year - last_year), # old scenario
    }
In [5]:
# Values from previous analysis on fertility (see Notebook #40)
fdata = json.load(open("../data/fertility_fitted.json"))
print("Fertility coefficients:")
display(fdata)

def get_newborns_next_year(dfp, year, fertility_rate, fdata):
    """Fom population and fertility rate for a given year,
    get the number of newborns for the NEXT year (fist January of year+1).
    """
    def constrained_sigmoid(x0, k):
        """Get sigmoid function, which is a cumsum of a distribution."""
        x = np.arange(17, 50+1)
        y = 1 / (1 + np.exp(-k * (x - x0)))
        y = y - y[0]
        y = y / y[-1]
        return y
    pop = dfp[year]
    mothers_potential = pop.loc[18:50].to_numpy() * fdata["assumed_women_ratio"]
    x0 = np.polyval(fdata["sigmoid_x0"], year)
    distrib = np.diff(constrained_sigmoid(x0, fdata["sigmoid_k"]))
    newborns = sum(mothers_potential * distrib) * fertility_rate
    return int(newborns)

fertility_rate = {}
m_coeff_scenario_1 = (2 - fdata["last_observed_value"]) / (end_year - fdata["last_observed_year"])
m_coeff_scenario_3 = (0 - fdata["last_observed_value"]) / (end_year - fdata["last_observed_year"])
for year in range(fdata["last_observed_year"], end_year+1):
    fertility_rate[year] = {
        f"↑ 2 in {end_year}": fdata["last_observed_value"] + m_coeff_scenario_1 * (year - fdata["last_observed_year"]), # going to 2
        "eq. to last": fdata["last_observed_value"], # constant
        f"↓ 0 in {end_year}": fdata["last_observed_value"] + m_coeff_scenario_3 * (year - fdata["last_observed_year"]), # going to 0
    }
Fertility coefficients:
{'last_observed_year': 2025,
 'last_observed_value': 1.109729190221021,
 'sigmoid_x0': [0.09617143688493088, -163.08834799890786],
 'sigmoid_k': 0.2963913700353926,
 'assumed_women_ratio': 0.5,
 'last_official_year': 2025,
 'last_official_value': 1.14,
 'calibration': 'Effective rate reproducing age-0 residents on next 1 January; not the official TFR'}

Projection of the resident population till 2100

In [6]:
dfpj_long_all = None
scenarios = [] # trying to sort them by descending end_year population
for migration_scenario in migration_coeff[end_year]:
    for fertility_scenario in fertility_rate[end_year]:
        scenario = f"migr {migration_scenario} & fert {fertility_scenario}"
        scenarios.append(scenario)
        print("Simulating:", scenario)
        dfpj = dfp.copy()
        for year in range(last_year+1, end_year+1):
            dfpj.loc[0, year] = get_newborns_next_year(dfpj, year-1, fertility_rate[year-1][fertility_scenario], fdata)
            dfpj.loc[1:, year] = dfpj.loc[0:99, year-1].values * ( 1 - dfdpj.loc[0:99, year-1]).values
            dfpj.loc[100, year] += dfpj.loc[100, year-1] * (1-0.4)

            # add migration balance
            dfpj.loc[1:80, year] += dfmb.values * migration_coeff[year][migration_scenario]

        dfpj_long = dfpj.reset_index().melt(id_vars="AGE").rename(columns={"variable": "year", "value": "population"})
        if dfpj_long_all is None:
            dfpj_long_all = deepcopy(dfpj_long).rename(columns={"population": scenario})
        else:
            dfpj_long_all[scenario] = dfpj_long["population"]

dfpj_long_all = dfpj_long_all[["year", "AGE"] + scenarios].astype(int) # Integers are ok for years, ages, and population (no need for decimal precision!)
dfpj_long_all.to_csv("../data/pop_by_age_year_proj.csv", index=False)
display(dfpj_long_all)
Simulating: migr double  & fert ↑ 2 in 2100
Simulating: migr double  & fert eq. to last
Simulating: migr double  & fert ↓ 0 in 2100
Simulating: migr average & fert ↑ 2 in 2100
Simulating: migr average & fert eq. to last
Simulating: migr average & fert ↓ 0 in 2100
Simulating: migr zero    & fert ↑ 2 in 2100
Simulating: migr zero    & fert eq. to last
Simulating: migr zero    & fert ↓ 0 in 2100
year AGE migr double  & fert ↑ 2 in 2100 migr double  & fert eq. to last migr double  & fert ↓ 0 in 2100 migr average & fert ↑ 2 in 2100 migr average & fert eq. to last migr average & fert ↓ 0 in 2100 migr zero    & fert ↑ 2 in 2100 migr zero    & fert eq. to last migr zero    & fert ↓ 0 in 2100
0 1952 0 817477 817477 817477 817477 817477 817477 817477 817477 817477
1 1952 1 838469 838469 838469 838469 838469 838469 838469 838469 838469
2 1952 2 851858 851858 851858 851858 851858 851858 851858 851858 851858
3 1952 3 907767 907767 907767 907767 907767 907767 907767 907767 907767
4 1952 4 901277 901277 901277 901277 901277 901277 901277 901277 901277
... ... ... ... ... ... ... ... ... ... ... ...
15044 2100 96 172474 172474 172474 142125 142125 142125 111775 111775 111775
15045 2100 97 134890 134890 134890 111735 111735 111735 88580 88580 88580
15046 2100 98 104452 104452 104452 87237 87237 87237 70022 70022 70022
15047 2100 99 79868 79868 79868 67411 67411 67411 54953 54953 54953
15048 2100 100 139710 139710 139710 120019 120019 120019 100327 100327 100327

15049 rows × 11 columns

In [ ]:
 
In [7]:
fig = px.line(
    dfpj_long_all, 
    x="AGE", 
    y=dfpj_long_all.columns, 
    animation_frame="year",
    markers=False,
)
fig.update_layout(
    xaxis_title="Age",
    yaxis_title="Population by age group",
    yaxis_range=[0, 1e6],
    legend_title="Scenarios",
    margin=dict(l=10, r=10, t=10, b=10),
    width=780,
    height=420,
)
fig.add_vrect(x0=17, x1=50, fillcolor="LightSalmon", opacity=0.5, layer="below", line_width=0)

# Initialize the animation at last_year
nframe_last_year = last_year - start_year
frame_last_year = fig.frames[nframe_last_year]
for extra in frame_last_year.data[len(fig.data):]:
    fig.add_trace(extra)
for i, tr in enumerate(frame_last_year.data):
    payload = tr.to_plotly_json() if hasattr(tr, "to_plotly_json") else dict(tr)
    fig.data[i].update(**payload)
if "sliders" in fig.layout and fig.layout.sliders:
    fig.layout.sliders[0].active = nframe_last_year

fig["layout"].pop("updatemenus") # optional, drop animation buttons
fig.write_html("../images_output/pop_by_age_proj.html", auto_play=False)
fig.show()
In [8]:
# Cached official total-population projections: never fetch implicitly during modelling.
df3 = istat_data(
    resource_id="165_889_DF_DCIS_PREVDEM1_1",
    key={"FREQ": "A", "REF_AREA": "IT", "DATA_TYPE": "JAN", "AGE": "TOTAL", "SEX": "9"},
)
df3p = df3.astype({"TIME_PERIOD": int}).pivot(index="TIME_PERIOD", columns="FORECAST_INTERVAL", values="value").astype(int)
print(f"ISTAT projection base: 1 January {df3p.index.min()}")
display(df3p)
ISTAT projection base: 1 January 2025
FORECAST_INTERVAL PROJLOW50 PROJLOW80 PROJLOW90 PROJMED PROJUPP50 PROJUPP80 PROJUPP90
TIME_PERIOD
2025 58943464 58943464 58943464 58943464 58943464 58943464 58943464
2026 58939689 58937044 58935227 58943005 58946200 58949140 58950649
2027 58907454 58899590 58894053 58917299 58926869 58935662 58940050
2028 58848463 58832939 58821657 58867892 58886771 58904055 58913000
2029 58765626 58739695 58721124 58797505 58828954 58857350 58872069
2030 58659838 58621480 58593276 58707285 58754539 58796185 58818688
2031 58535984 58483057 58442430 58601772 58667423 58725771 58756011
2032 58396648 58326128 58273997 58483951 58570938 58647692 58688706
2033 58244012 58152443 58089024 58355415 58464212 58562448 58614800
2034 58081487 57969744 57888428 58218852 58352120 58475788 58539528
2035 57908189 57775060 57674672 58074876 58236865 58385088 58465824
2036 57726588 57568861 57449772 57924032 58115291 58290637 58390132
2037 57537431 57349467 57211999 57766765 57989727 58193089 58310005
2038 57339326 57121900 56962973 57602783 57861142 58093289 58229884
2039 57133857 56885133 56702817 57431727 57724959 57992589 58149424
2040 56920038 56637011 56429351 57254075 57585794 57885871 58064449
2041 56693715 56374969 56147869 57069408 57441763 57781616 57979262
2042 56461246 56104465 55849261 56876689 57292659 57671963 57890514
2043 56216427 55821564 55540278 56675572 57135595 57554165 57798484
2044 55960168 55522570 55222930 56465629 56970581 57435381 57701337
2045 55693537 55219089 54890818 56246178 56797413 57308791 57597301
2046 55414631 54901965 54545845 56016225 56616475 57175296 57489010
2047 55127398 54564650 54186151 55775408 56430155 57029428 57370683
2048 54824086 54211043 53806526 55522240 56230557 56881702 57238992
2049 54504075 53853136 53419791 55256777 56021816 56714922 57102779
2050 54172736 53475584 53015805 54978808 55793598 56537783 56954480
2051 53826694 53084322 52588333 54687052 55553725 56362485 56790326
2052 53465551 52671472 52145312 54381546 55303597 56162634 56629112
2053 53092317 52245911 51692660 54063926 55039991 55957776 56441638
2054 52705263 51803028 51220474 53734421 54778262 55743686 56248548
2055 52307952 51367556 50739956 53394258 54496804 55518609 56058659
2056 51898514 50905869 50257704 53043967 54204527 55263672 55844656
2057 51490468 50446721 49775974 52685898 53906783 55015929 55646006
2058 51065469 49976406 49252445 52320867 53605163 54767654 55430512
2059 50643749 49497486 48724667 51951136 53290598 54529385 55225806
2060 50206400 49004859 48227967 51579512 52983858 54267234 55010146
2061 49770859 48503625 47679256 51207594 52687876 54015975 54803284
2062 49337140 48039171 47153125 50838178 52390478 53782225 54584012
2063 48912630 47553363 46657535 50473225 52081804 53554023 54372400
2064 48484319 47062651 46157828 50113751 51780781 53317679 54168553
2065 48062186 46577038 45685496 49762705 51485650 53073219 53979795
2066 47666501 46112561 45173755 49421042 51225827 52854052 53807823
2067 47245731 45665485 44648377 49089784 50956308 52668894 53635854
2068 46859153 45199066 44147547 48769660 50716862 52484456 53480580
2069 46470499 44756121 43657227 48461284 50481472 52335913 53323208
2070 46092432 44334597 43181760 48164308 50283700 52164564 53209862
2071 45731190 43901417 42725918 47879955 50055771 52045860 53109361
2072 45397187 43513237 42289443 47608316 49850465 51919843 53010524
2073 45043381 43118962 41836431 47348036 49683094 51823163 52924577
2074 44728887 42719833 41419327 47099160 49517070 51770338 52854775
2075 44424525 42338886 40994353 46860292 49367381 51668739 52820129
2076 44123372 41984348 40565868 46630296 49219764 51583751 52774256
2077 43796952 41584570 40201768 46408267 49073962 51508975 52790584
2078 43481603 41208948 39741272 46193217 48955237 51466418 52778549
2079 43179454 40843297 39356401 45984031 48834070 51400560 52798024
2080 42901845 40493836 38953609 45780862 48704213 51348608 52834478
In [9]:
fig = dfpj_long_all.groupby("year").sum().drop(columns="AGE").plot()
#fig.add_vrect(x0=start_year, x1=last_year, fillcolor="Green", opacity=0.1, layer="below", line_width=0)
fig.update_layout(
    legend_title="Scenarios",
    margin=dict(l=10, r=10, t=20, b=10),
    xaxis_title=None,
    yaxis_title="Total Population",
    width=780,
    height=320,
)
fig.add_trace(go.Scatter(
    x=df3p.index, 
    y=df3p["PROJMED"], 
    mode="lines", 
    name=f"ISTAT base {df3p.index.min()} — central 80% interval",
    line=dict(color="rgba(0,0,0,0.7)")
)).add_trace(go.Scatter(
    x=df3p.index, 
    y=df3p["PROJLOW80"], 
    mode="lines",
    showlegend=False,
    fill=None,
    line=dict(color="rgba(128,128,128,0.3)")
)).add_trace(go.Scatter(
    x=df3p.index, 
    y=df3p["PROJUPP80"], 
    mode="lines",
    showlegend=False,
    fill="tonexty",
    line=dict(color="rgba(128,128,128,0.3)"),
    fillcolor="rgba(128,128,128,0.3)"
))

fig.write_html("../images_output/pop_tot_proj.html")
fig

Conclusions

The nine scenarios are sensitivity assumptions through 2100. The official comparison is ISTAT's base-2025 median and central 80% uncertainty interval through 2080; the model scenarios do not carry probabilities. Summary totals and changes from the previous checked-in model are recorded in data/refresh_validation.json.

The baseline uses residents on 1 January 2026, mortality observations through 2025, and the refitted 2025 effective fertility rate. The migration profile uses only years with both population changes and mortality observations, excludes 2020–2021, and clips negative age-specific residuals as in the original model. Mortality is now aligned to the cohort's age at the start of each year. The open-ended 100+ group retains the original assumed 40% mortality.

Source: ISTAT base-2025 projection release.

← Home