Re: m68k Installer performance and kernel profiling (was: Updated installer images 2020-12-02)
On Sun, 3 Jan 2021, Eero Tamminen wrote:
> On 1/3/21 1:38 AM, Finn Thain wrote:
> > On Sun, 3 Jan 2021, Eero Tamminen wrote:
> > > ...
> > > * While kernel runs init a minute after being
> > > booted [1], installer is *much* slower.
> > >
> > > E.g. after pressing Enter to select another
> > > country, it takes 5-10 mins until installer
> > > presents me with a list from which I can select
> > > Europe
> >
> > Is that a kernel regression or an installer regression?
>
> This is the first time I've run Debian kernel +
> installer in Hatari, so I have no idea whether
> it's a regression. Sorry, I wasn't clear on that!
>
> Based on some profiling I did, slowness is
> completely on the user-space side. [1]
>
If you want to check for a regression in the installer, you can find
various ISO images for m68k going back to 2018 at
https://cdimage.debian.org/cdimage/ports/
>
> > > In cycles, memcpy() uses a bit more compared to
> > > memset(), but memset() still takes most CPU.
> > >
> > > Attached are partial callgraphs showing where
> > > they're most called from. Percentages in the
> > > arrow lines indicate each function node's share
> > > of those calls.
> >
> > If this is a new phenomenon, git bisect may reveal what changed.
>
> I've been testing upstream kernel boot times
> since v5.2, and I don't think there's been any
> significant regression...
>
>
> > Could the memcpy calls be related to 64-bit timekeeping?
>
> ...but if you're interested, I could profile and
> provide similar callgraph for another kernel
> you're interested about. Just point me to kernel
> + initrd + symbols.map packages (or files) you'd
> like me to check.
>
It's not hard to build that yourself, thanks to Debian's cross-compiler
packages. But since I do a lot of these builds, I've built one for you to
try.
https://www.telegraphics.com.au/~fthain/build/linux-m68k-image-4.14.0-multi.tar.gz
https://www.telegraphics.com.au/~fthain/build/vmlinux-4.14.0-multi
MD5 (linux-m68k-image-4.14.0-multi.tar.gz) = fdd15dafc843a86c3e50afd24c864159
MD5 (vmlinux-4.14.0-multi) = 0129bce08219f49a6cdb7b3d21c7afd1
It boots to busybox in aranym but YMMV. It may not work with the debian
installer in hatari. Note that v4.14.213 has fewer bugs but it too may
have performance regressions.
>
> [1] There isn't any way to profile Linux user-
> space in Hatari, except for having single number
> of how much time whole user-space consumes
> compared to kernel.
>
That's quite a useful ratio, I think. If you kept the userspace workload
constant and swapped the kernel release for an older one, you could tell
whether the kernel had regressed. Similarly, you could keep the kernel
build constant and vary the installer release, to look for changes there.
> However, I can get very accurate profiling data
> for all of the kernel side (depending on symbols
> file content), and profiling whole Linux bootup is
> trivial in Hatari.
>
> Profiling kernel usage during specific user-
> space activity would require that user-space
> activity to start & end with call to some kernel
> symbol/syscall that's not otherwise called
> (so that emulator breakpoints can be set to
> start and of the profiling).
>
I guess profiling should start at kernel_execve("/sbin/init", ...)
Reply to: