Bug#896790: apt on jessie (pre-1.1) closes millions of files on ExecFork (makes Docker slow)

[wjdoekes@osso.nl]

Package: apt
Version: 1.0.9.8.4
Severity: important

Dear Maintainer,

I'm sorry for flagging this as 'important', but in the Docker eco-system
this bug is quite troublesome for everyone doing apt on Jessie.

The issue which is reported (and fixed) here --
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=764204 --
still applies to Jessie:

  Apt ExecFork() tries to close all possible FDs instead of just the
  open ones.

  **In docker builds, this generally defaults to 1 million(!) files**

  So, you'd be looking at this (from 3 to 1024*1024), instead of doing
  apt updates/upgrades.

    [pid 17831] getrlimit(RLIMIT_NOFILE, {rlim_cur=1024*1024, 
rlim_max=1024*1024}) = 0
    [pid 17831] fcntl(148924, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file 
descriptor)

  This caused a simple apt-get update/upgrade run to go from around 15
  secs on Wheezy to a whopping 200 seconds on Jessie.


Apt versions:

  Wheezy runs: apt 0.9.7.9+deb7u7 (fast, closes max. 40 FDs)
  Jessie runs: apt 1.0.9.8.4 (slow, closes max. nofile FDs)
  Stretch runs: apt 1.4.8 (fast, closes exactly open-FD-count)

Fix history:

  Old (broken after 0.9.13), November 2013:
  
https://github.com/Debian/apt/commit/61f954bff040809e7ab57b3adec2fe95339ffb94#diff-44ca32a60c136bb8155104e94ca99e0a

  Better (fixed in 1.0.9.10+), April 2015:
  
https://github.com/Debian/apt/commit/15901516326737a67f2a9af26cd7e434162de019

  Best (fixed in 1.1+), May 2015:
  
https://github.com/Debian/apt/commit/be4d908fb5d56f8a331bb88e878a6fb8d82a77a6


Would you be willing to patch the Jessie Apt versions with the two
latter commits? I could file a patch if you want.


(For the record: workarounds for Docker include setting the nofile limit
in daemon.json {"default-ulimits": {"nofile":"512:1024"}} or passing
--ulimit nofile=512 to 'docker build' (not available for 
docker-compose).)


Cheers,
Walter Doekes
OSSO B.V.



-- Package-specific info:

-- (no /etc/apt/preferences present) --


-- (/etc/apt/sources.list present, but not submitted) --


-- System Information:
Debian Release: 8.9
  APT prefers oldstable
  APT policy: (500, 'oldstable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.4.0-116-generic (SMP w/16 CPU cores)
Locale: LANG=C.UTF-8, LC_CTYPE=C.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages apt depends on:
ii  debian-archive-keyring  2017.5~deb8u1
ii  gnupg                   1.4.18-7+deb8u3
ii  libapt-pkg4.12          1.0.9.8.4
ii  libc6                   2.19-18+deb8u10
ii  libgcc1                 1:4.9.2-10
ii  libstdc++6              4.9.2-10

apt recommends no packages.

Versions of packages apt suggests:
pn  apt-doc                      <none>
pn  aptitude | synaptic | wajig  <none>
ii  dpkg-dev                     1.17.27
ii  python-apt                   0.9.3.12

-- no debconf information