Update: it turns out the problem is partly due to my graphics card
officially being unable to use my monitor's favorite modeline
(unofficially, it's worked fine for years, but nouveau installed
a bugfix that makes it obey the official limits).
Stefan