Cosmological simulations of the high-redshift galaxy population adopting a variable stellar initial mass function

JWST surveys reveal a greater space density of high-redshift UV-bright galaxies than predicted by conventional galaxy formation models. We present results from a L=100 cMpc cosmological simulation evolved to z=5 with a variation of the COLIBRE galaxy formation model that adopts a density-dependent stellar initial mass function (IMF), such that stellar populations formed from dense gas are born with a top-heavy IMF. Crucially, heavy element and dust yields, and supernova feedback energetics, are self-consistently adjusted to the changing IMF. We model UV/optical emission (including nebular emission) from galaxies and its attenuation by dust. By allowing a significant fraction of high-redshift star formation to proceed with a top-heavy IMF, the rest-frame far-UV luminosities of early galaxies are elevated by up to a factor of 4 with respect to the fiducial COLIBRE L100m6 simulation, which assumes a universal Chabrier IMF. This enables the formation of galaxies with observed brightness up to M UV-20 at z=15 (c.f. M UV-18.5 in the fiducial simulation), illustrating the potential of star formation with a top-heavy IMF to alleviate tensions with JWST data. Later, the boost in far-UV emission is partly offset by attenuation due to increased dust surface densities from i) additional dust grain ejection from core-collapse supernovae and ii) efficient grain growth promoted by more metal-rich interstellar gas. The simulation reproduces the z=5 galaxy stellar mass function and rest-frame optical luminosity function with comparable accuracy to the fiducial simulation, and both simulations exhibit UV continuum slopes that are consistent with JWST observations.