Re: Bug#605009: serious performance regression with ext4

[Raphael Hertzog <hertzog@debian.org>]

Hi,

adding debian-devel, debian-boot, debian-kernel and Theodore Y. Ts'o in CC
because I'm fed up with this problem. Sorry for the massive crosspost, you
might want to follow up only on -devel and on the bug report.

Some clones/reassign should probably result from this discussion anyway.

On Fri, 26 Nov 2010, Michael Biebl wrote:
> I'm using ext4 with the default mount option and the fs was created with
> the default settings from /etc/mke2fs.conf.
> Since the latest upgrade, performance suffered badly.
> 
> E.g. installing vim-runtime took ~8 secs with 1.15.8.5, now it takes
> ~44secs.
>
> It was suggested that I use the nodelalloc mount option for ext4.
> But that not only takes away some of the benefits of ext4, it also
> requires explicit configuration.
> 
> dpkg should work properly(with good performance) out-of-the-box on ext4.

There's currently no way we're going to fix this.

It all started with report of corrupted (zero-length) files on ext4/ubifs
(see http://bugs.debian.org/430958). We did the right thing to fix this
which is to call fsync() on the fly on each file that dpkg unpacks. That
was introduced in dpkg 1.15.6. (Ubuntu had more than 80 bug reports
related to this and Debian very few because we don't use ext4 by default)

That was disastrous in terms of performance, so we then grouped the
fsync()+rename() calls at the end of the unpack phase before updating the
dpkg database. That can be witnessed in dpkg 1.15.7.

That was ok everywhere except on ext4. For some unknown reasons, with the
nodelalloc option the performance are again reasonable (but we discovered
that only recently). Ubuntu is using ext4 by default and this performance
was not acceptable and we tried to work-around the problem by replacing
all the fsync() by a single sync() on Linux only (because Linux's sync()
is synchronous while it's not necessarily the case on other systems). This
was enabled in dpkg 1.15.7.2 in response to http://bugs.debian.org/578635.

Now using sync() appears to be a bad choice since it resulted in RC bugs
like http://bugs.debian.org/595927 and http://bugs.debian.org/600075 and
was an annoyance for people using dpkg on tmpfs to get super-high speed
(http://bugs.debian.org/588339).

So we reverted that hackish work-around of using sync() and we're back
to the situation where dpkg is very slow on ext4. But we have added a new
option --force-unsafe-io that can be used when data safety is not
important (in a temporary chroot, during initial installation, etc.) as
requested in http://bugs.debian.org/584254.

Where does that leaves us? Here are various options to consider (they are
not necessarily mutually exclusive):

----
1/ This problem and the known work-arounds should be documented in the
release notes.

2/ d-i should be modified to use --force-unsafe-io if it detects a dpkg
version >= 1.15.8.6.

3/ d-i might want to setup the nodelalloc option by default when a
partition is formatted with ext4 and mounted as / or /usr.

4/ Theodore might want to find out why ext4 is behaving so badly under
this usage pattern while ext3 doesn't... i.e. fix
https://bugzilla.kernel.org/show_bug.cgi?id=15910
Or at least suggest another usage pattern which result in the same
guaranty for dpkg but without the poor performance (and sync() is not the
right answer as experience showed us).

Note that doing N x fsync() followed by N x rename() is not giving us any
better result that doing N x (fsync()+rename()).

Note that calling posix_fallocate() before writing the files that are
fsynced() did not help either (Mike Hommey thought it could be a way to
emulate nodelalloc at the dpkg level only but apparently not).

What has not been tried: reordering the fsync() based on FIEMAP
information. Or using fdatasync() instead of fsync() and calling fsync()
on directories.

5/ maybe the debian-kernel team wants to discuss the issue on LKML. Both
for the bad ext4 performances (see above) and the (incorrect?) behaviour
of sync() which is never finishing under important I/O loads (cf
https://bugzilla.kernel.org/show_bug.cgi?id=18632).
----

But right now from the point of view of dpkg maintainers, this bug is a
"wontfix" at our level.

Just to sum up what dpkg --unpack does in 1.15.8.6:
1/ set the package status as half-installed/reinst-required
2/ extract all the new files as *.dpkg-new
3/ for all the unpacked files: fsync(foo.dpkg-new) followed by
   rename(foo.dpkg-new, foo)
4/ set the package status as unpacked

Note that the directories are not fsynced() because we mainly don't care
if the rename is recorded or not, as long as the installed file always has
the content of either the old or the new file. This could be fixed but is
unlikely to help in getting better performances I guess.

The only thing we want to achieve is that:
- when the package is set to unpacked status, all changes have been
  committed
- when the process is abruptly interrupted, we don't leave corrupted files
  around (like unwanted empty files)

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
                      ▶ http://RaphaelHertzog.fr (Français)