Re: Bug#605009: serious performance regression with ext4
On Mon, Nov 29, 2010 at 02:58:16PM +0000, Ian Jackson wrote:
>
> This is the standard way that ordinary files for which reliability was
> important have been updated on Unix for decades. fsync is for files
> which need synchronisation with things external to the computer (or at
> least, external to the volume) - eg, email at final dot.
This is simply not true. And I'm speaking as someone who has been
doing Unix/Linux kernel development since the BSD 4.3 days. (Well,
BSD 4.3+Tahoe days, to be precise.)
fsync() has always been the only guarantee that files would be on
disk. In fact the way BSD worked, there was no guarantee that
rename() would provide any kind of file synchronization primitive;
that's actually something new. No, in the old days, if you really
cared about a file, you would fsync() it. Period. End of paragraph.
It was just that in those days, the main things people cared about
where either source/text files (so the editors of the day would do the
right thing) or e-mail (and no, just for the final delivery; for all
MTA's).
The problem that caused people to get this wrong idea was because (a)
back then Unix machines tended to be more reliable, because they were
run by professionals in machine rooms, very often with UPS's. Also,
(b) people weren't loading craptastic video drivers with buggy
proprietary kernel modules; they may have used proprietary drivers,
but kernels weren't changing all the time, and there was a lot more
careful testing of drivers before they were unloosed onto the world.
Finally (c), ext3 had as an accident of how it provided protection
against old file blocks showing up newly allocated files (something
which BSD 4.3 did __not__ protect against, by the way), had the
behaviour that renaming over a file __usually__ (but not always)
provided atomic guarantees.
(c) was especially unfortunate, because it never applied to all Linux
file systems, just to ext3, and because the fact that it did this was
also responsible for disastrous desktop performance when you had the
combination of large number of streaming writes (i.e., bittorrent,
video ripping/copying, etc.) going on in the background combined with
foreground GUI applications that were fsync-happy() --- i.e., firefox.
Lots of users have complained about the desktop performance problem,
but the reality is we can't really solve that without also taking away
the magic that made (c) happen. Whether you solve it by using
data=writeback and stick with ext3, or switch to ext4, or switch to
XFS, or switch to btrfs --- all of these will solve the desktop
performance problem, but they also leave you vulnerable to file loss
in the case of system crashes and applications that don't use
fsync()/fdatasync().
Hence the fact that all file system developers, whether they were
btrfs developers or XFS developers or ext4 developers, made the joke
at the file system developers summit two years ago, that what the
application programmers really wanted was O_PONY, with the magic pixie
dust. Unfortunately:
http://www.linuxformat.com/files/nopony.jpg
- Ted
Reply to: