Optimise MixingPools

[kgrantz]

Mixing pools are ~4x (or more) longer than equivalent simulations using CommunityNet - but MixingPools better reflect the input data and desired transmission mechanics.

Options to optimize (thanks to Dan Klein):

• Restricting to infectious and susceptible individuals in compute_transmission could provide a boost. If restricting to infectious agents on the transmission side, the mean could change to a sum divided by len(src).
• Caching: The prototype I shared previously was using AgeGroup, which has caching built in. That's great because I recall the mixing matrices are 20x20... so that's 400 pools! But there are only 20+20 src+dst uids to cache. A better design might do one np.digitize and cache that for the step.
• Update frequency: The AgeGroup class already has caching for the current step, but that might be the place to introduce a larger dt

[kgrantz]

Made a first attempt to decrease update frequency here - ac135e6

But some strange model dynamics so far - big peaks in incidence later in simulations with longer dur_fac (update interval in days):

Occurs in all age groups - lower peak in 0-2y due to susceptible introductions - and regardless of contact exposure rate (exp_fac)

Do have more chronic infections with less frequent updates even before later spikes in incidence. Probably worth running other stochastic iterations/larger pop to confirm this isn't just random.

Do see benefits in model duration moving from per-time step (0 dur) updates, but little difference between 10, 30, and 90d updates

[kgrantz]

Zooming in just on new infections, there is a cyclical pattern with period ~= update frequency. Doesn't seem like there should be enough susceptible build-up or aging between MixingPool updates to affect transmission dynamics, but perhaps there is?

[kgrantz]

Still see similar cyclic peaks with lower birth rate (15 per 1000) - doesn't seem like delaying aging between pools or initiation of transmission for new births is main driver

And perhaps even more strangely, still see same patterns with uniform mixing (Mij=1 for all i/j age class pairs), which I would expect would make mixing pool membership immaterial for transmission risk.

[kgrantz]

Uniform mixing with Mij=10 and tppi=0 - still see cyclical patterns, but lower amplitude, and no peaks later on.

See lower prevalence of active infection in age classes at edge of MixingPool boundaries though - regardless of the MixingPool update frequency.

[kgrantz]

There was an age boundary issue when loading the Prem matrices (see dbf7d97). But still some unexpected dynamics, namely that:

• without natural immunity, incidence over time is lower with longer update durations
• with natural immunity, incidence on average is higher with update frequency>0, but spiky/random

Tppi=0, uniform mixing with degree=10:

Tppi=0.94, uniform mixing with degree=10:

[kgrantz]

Missed that there were overflow errors very occasionally in these lines:

acq = self.eff_contacts[self.dst_uids] * rel_sus[self.dst_uids]
p = beta*disease_beta[0]*trans*acq

Values aren't large enough or NAs/Infs - what seems to be happening is rel_sus is calculated externally to the mixing pools at each time point, so creates an invalid indexing if someone dies but still exists in the MixingPool dst_uids or src_uids.

Here are results as above, but turning off all deaths:

[daniel-klein]

OMG, amazing sleuthing @kgrantz ! So it was both the age matrices and the filtering to alive uids? It's now resolved?

[kgrantz]

Confirming that dynamics are as expected even with deaths added back in.

But the current fix is very slow (slower than original MixingPools updated at every time step...) and smaller relative gains from increasing cache duration. Might be more efficient to remove individual UIDs as they die given deaths are relatively rare (cumulative death rate over 10 yr ~ 8 per 1000 PY, or ~10,000 events in full production run), or to add filtering to get_uids in compute_transmission step.

[kgrantz]

Implemented MixingPools cache in starsim - currently in fix-mp-deaths branch

Primary slow down now is from required state updates with Births (see starsim issue)