Re: [RFC/PATCH 0/4] Re: Bug#605009: serious performance regression with ext4

[Theodore Tso <tytso@MIT.EDU>]

On Nov 29, 2010, at 1:48 AM, Jonathan Nieder wrote:

> 
> Results (on ext4) suggest that patches 1 and 4 matter and the rest are
> within noise.  Timings are rough; sometimes replicates vary by as much
> as a second.  Numbers are cold cache (i.e., after running sync and
> echo 3>.../drop_caches), best of 3, dpkg --install python2.7 and
> python2.7-minimal.
> 
> before:
> 5.73user 1.62system 0:33.84elapsed 21%CPU (0avgtext+0avgdata 89968maxresident)k
> 0inputs+0outputs (0major+46962minor)pagefaults 0swaps
> 
> patch 1 (use SYNC_FILE_RANGE_WRITE):
> 5.64user 1.69system 0:10.47elapsed 69%CPU (0avgtext+0avgdata 90000maxresident)k
> 0inputs+0outputs (0major+46948minor)pagefaults 0swaps
> 
> patch 1+2 (use SYNC_FILE_RANGE_WAIT_BEFORE):
> 5.48user 1.61system 0:10.43elapsed 70%CPU (0avgtext+0avgdata 90000maxresident)k
> 0inputs+0outputs (0major+46958minor)pagefaults 0swaps

So Patch #2 wasn't quite what I talked about doing; patch #2 is adding SYNC_FILE_RANGE_WAIT_BEFORE for each file immediately after writing the file.   So it's the equivalent of:

     extract(a)
     sync_file_range(SYNC_FILE_RANGE_WRITE)
     sync_file_range(SYNC_FILE_RANGE_WAIT_BEFORE)
     extract(b)
     sync_file_range(SYNC_FILE_RANGE_WRITE)
     sync_file_range(SYNC_FILE_RANGE_WAIT_BEFORE)
     extract(b)
     sync_file_range(SYNC_FILE_RANGE_WRITE)
     sync_file_range(SYNC_FILE_RANGE_WAIT_BEFORE)

What I was suggesting was to use a separate for loop in patch #2, like patch #3:

     extract(a)
     sync_file_range(SYNC_FILE_RANGE_WRITE)
     extract(b)
     sync_file_range(SYNC_FILE_RANGE_WRITE)
     extract(b)
     sync_file_range(SYNC_FILE_RANGE_WRITE)

     sync_file_range(a, SYNC_FILE_RANGE_WAIT_BEFORE)
     sync_file_range(b, SYNC_FILE_RANGE_WAIT_BEFORE)
     sync_file_range(c, SYNC_FILE_RANGE_WAIT_BEFORE)

As to why the "voodoo", the idea is to make sure all of the delayed allocation, for all of the files, is completely resolved before the first fsync().    The reason why I suggested doing the WAIT_BEFORE as a separate path was to allow for parallelism in the case where /var/cache/apt/archives is on a different disk spindle than /usr.   By doing this:

     extract(a)
     sync_file_range(SYNC_FILE_RANGE_WRITE)
     sync_file_range(SYNC_FILE_RANGE_WAIT_BEFORE)
     extract(b)
     sync_file_range(SYNC_FILE_RANGE_WRITE)
     sync_file_range(SYNC_FILE_RANGE_WAIT_BEFORE)

.... we make the copying get done in lockstep; that is, we don't start extracting file b until the data blocks for a are done being written.   If /var and /usr were mounted on different floppy disks (yeah, I know) you'd see first one disk light up, and then the other disk light up, back and forth, and it would be slow and horrible.   With the mechanism I suggested, both lights would be on at the same time, since SYNC_FILE_RANGE_WRITE initiates the writeback, but does not block for it to complete.   SYNC_FILE_RANGE_WAIT_BEFORE is what actually blocks.  Does that help to visualize what I was going for?

BTW, if you had opened the file handle in subsequent passes using O_RDONLY|O_NOATIME, the use of fdatasync() instead of fsync() might not have been necessary.   And as far as the comments in patch #4 was concerned, it wasn't a matter of delaying the file modification time update that was my concern; it was avoiding an update of the file access time caused by reopening the file which concerned me.   The reason why I did both in my test program was because (a) I was paranoid, and (b) fdatasync() is standard, where as O_NOATIME is another Linux-specific thing.

Thinking about this some more, though, using O_NOATIME may actually save more time, and may in the end be more important than the use of fdatasync() vs. fsync().  (Although I like doing the last amount of work necessary, and in this case we really don't need to use fsync(); fdatasync() will do.)

-- Ted