Bug#726731: dump: Huge RAM usage on restore

[John Goerzen <jgoerzen@complete.org>]

Package: dump
Version: 0.4b44-1
Severity: important

Hello,

I am running a restore, and here is output from top:

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND                                                                      
11070 root      20   0 7549m 4.2g  284 D   6.3 53.7 831:32.08 restore                                                                       
The only reason the RSS is so much lower than the VSS is that part of
it has swapped out.

Here's what happens.

On restore, it reads the first 2.7GB or so of the backup, by that
point allocating around 800MB of RAM.  It then spends a long time
creating all the directories in the backup set, and as it does so the
RAM usage gradually increases to many GBs.  Once it starts creating
files, the RAM is up north of 5GB.  As it extracts the dump, the RAM
continues to climb.  I had to move the restore process to a different
machine than the server from which it was made, because that system
had only (!) 4GB RAM.  Extracting over NFS is working, so far, but
this system has 8GB RAM and the restore is only about 2/3 done at this
point.

The dump in question was made from a filesystem containing 1.8TB of
BackupPC data across about 24 million inodes.  BackupPC works with a
hardlink farm, and every backup has a directory skeleton created
(though only the "full" backups have files hardlinked into the storage
pool).

I didn't specifically watch while dump was running, but I would have
noticed if it tried to allocated 8GB RAM.

In addition, restore filled up /tmp because it tried to put 2.8GB
there, causing issues with other programs running on the system.
(Worked around with -T)

I think some people may face a situation where a backup is
unrestoreable because the restore process demands a far beefier system
than the backup process does!


-- System Information:
Debian Release: 7.1
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.2.0-4-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages dump depends on:
ii  e2fslibs      1.42.5-1.1
ii  libblkid1     2.20.1-5.3
ii  libc6         2.13-38
ii  libcomerr2    1.42.5-1.1
ii  libncurses5   5.9-10
ii  libreadline6  6.2+dfsg-0.1
ii  libselinux1   2.1.9-5
ii  libuuid1      2.20.1-5.3
ii  tar           1.26+dfsg-0.1

dump recommends no packages.

dump suggests no packages.

-- no debconf information