Bug#947759: Configuration optimizations for the cloud variant
On Wed, Feb 05, 2020 at 04:27:23PM -0500, Noah Meyerhans wrote:
> On Tue, Feb 04, 2020 at 03:48:32PM -0800, Josh Triplett wrote:
> > I would suggest testing on a c5.large. t2 and t3 have shared CPUs, so
> > they have less consistent boot time. c5.large is about the same cost as
> > t3.large, but will have far more consistent performance.
>
> Performance definitely seems to vary quite a bit. Here are a few
> samples from c5.large instances in us-west-2:
>
> 5.5 "generic" kernel:
> $ systemd-analyze
> Startup finished in 4.323s (kernel) + 11.189s (userspace) = 15.513s
> graphical.target reached after 9.865s in userspace
>
> 5.5 "cloud" with optimizations:
> $ systemd-analyze
> Startup finished in 7.273s (kernel) + 21.984s (userspace) = 29.258s
> graphical.target reached after 19.811s in userspace
>
> $ systemd-analyze
> Startup finished in 4.319s (kernel) + 13.684s (userspace) = 18.003s
> graphical.target reached after 12.321s in userspace
>
> $ systemd-analyze
> Startup finished in 3.327s (kernel) + 9.846s (userspace) = 13.174s
> graphical.target reached after 8.831s in userspace
>
> The optimized timings are all taken from the initial boot of newly
> launched instances.
>
> It's certainly possible that things will look better with more data, but
> it's not clear yet that this change is an improvement in all cases.
That's strange. I've gotten very consistent boot-time results from c5
instances, and nowhere near those values. Certainly none of these
changes should introduce anywhere near that much variability in boot
time.
At the moment, I'm focused on the kernel time, not userspace time, since
the latter depends greatly on what you have installed. Would you mind
adding `initcall_debug` to the kernel command line, and providing
(compressed) dmesg logs for a few boots where you're observing that much
variability? That should make it relatively easy to diagnose where the
time is going.
(Also, you may want to make sure your kernel isn't configured to send
all its messages to the (virtual) serial port, which can slow down boot
time.)
> > > I'll post a WIP MR on salsa next. Some cleanup is needed still.
> >
> > Thank you for working on this!
> >
>
> The MR is here: https://salsa.debian.org/kernel-team/linux/merge_requests/206
>
> I'm happy to share binary and/or source .debs and/or AMIs if you'd like
> to run some of your own tests.
>
> > Have you confirmed that with the optimizations, you can boot without
> > needing an initramfs?
>
> I've confirmed that the kernel can boot on c5 and t3 instances without
> an initramfs present at all. However, the Xen backed EC2 instance types
> would also need to statically link (at least) ATA_PIIX, ATA_GENERIC, and
> XEN_BLKDEV_FRONTEND if we wanted to provide kernels that could be
> generally useful in EC2 without an initramfs.
For initramfs-less boot, I would suggest focusing on AWS's new
Nitro-based instances, rather than the Xen-based instances. I don't
think there's value in trying to build in enough drivers to boot without
an initramfs on older cloud hardware as well.
Reply to: