[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Speeding up dpkg, a proposal

Yodel again!
On Wednesday 02 March 2011 17.02:11 Marius Vollmer wrote:
> It shows a speed up between factor six and two in our environment (ext4
> on a slowish flash drive) .  I am not sure whether messing with the
> fundamentals of dpkg is worth a factor of two in performance

To not be all negative: read the recent discussion about fsync() and other 
stuff in dpkg (I'm not sure where the discussion happened exactly; it was 
about dpkg becoming extremely slow in some use cases on modern filesystems 
like btrfs and was a short time before the release. Since then, there is an 
option for dpkg:

              unsafe-io:  Do  not  perform  safe  I/O  operations  when
              unpacking.  Currently  this  implies  not performing file
              system syncs before file renames, which is known to cause
              substantial performance degradation on some file systems,
              unfortunately the ones that require the safe I/O  on  the
              first  place  due  to  their unreliable behaviour causing
              zero-length files on abrupt system crashes.

              Note: For ext4, the main offender, consider using instead
              the mount option nodelalloc, which will fix both the per‐
              formance degradation and the data safety issues, the lat‐
              ter  by  making  the  file system not produce zero-length
              files on abrupt system  crashes  with  any  software  not
              doing syncs before atomic renames.

              Warning:  Using  this option might improve performance at
              the cost of losing data, use with care.

So you should compare against dpkg with unsafe-io. Very slightly pre-dating 
this: I often (on btrfs, and when I'm inside development chroots that don't 
matter much) end up wrapping aptitude/apt-get/dpkg with eatmydata and get 
the same benefits. But the tool really deserves its name: a hard reboot 
directly after a package installation will leave a mess behind... (I could 
so far avoid this; I recently had a zero-length initrd after a kernel 
upgrade that *might* have been related. OTOH I did a clean shutdown there so 
it shouldn't have happened...)

Yet another point, for the future: I *think* that btrfs is building an 
interface so applications can directly access btrfs transactions. Which 
would allow to do package upgrades in a btrfs transaction, and since (again, 
I *think* so, I'm not sure) can even be ended without forcing an immediate 
sync (the result is more like a barrier than a sync), this would be a fine 
way to deal with the situation.  At the cost that it's btrfs specific.

-- vbi

BOFH excuse #233:

TCP/IP UDP alarm threshold is set too low.

Attachment: signature.asc
Description: This is a digitally signed message part.

Reply to: