On Wed, 30 Sep 2009 22:44:59 +0200 Simon Richter <sjr@debian.org> wrote: > > When the current grip-cron.sh run finishes (which is now taking > > longer because of the lack of tmpfs), I will want to put a tmpfs > > into /etc/fstab for /org/emdebian/tmpfs and it really needs to be at > > least 512Mb, preferably 1Gb as that is the smallest that I've > > tested so far. > > Hm, it could have been the tmpfs, but I'm not sure gripping packages > is a process that really leads to an active working set of > 500 MB. I've only been able to judge by watching 'df' output during a run but I've seen a few hundred megabytes in tmpfs when processing a single .deb. Java is the main culprit - that came in when we added the build-dependencies for each package in Grip. However, using a tmpfs appears to reduce run time by a factor of five or more - just from superficial estimations. The new logs now contain timestamps at the end of each package processing. View the logs at http://www.emdebian.org/grip/logs/ and look for the Stamp. That should indicate which packages are hogging resources. Most packages should take less than a minute from start to finish (per architecture). I suppose the timestamp could be supplemented by some kind of call to df for the size of the tmpfs immediately before it gets cleared, if that would help. What other figures could be included into the logs for better analysis? > If memory is full, tmpfs just degrades into a non-journalled file > system in swap space, after all. True, but it's better than a ramfs that would just die if out of RAM. > After rebooting with 1.5 GB of RAM, whatever happened on this box > still made it touch swapspace (although only a few MB), so there was > a point where tmpfs contents and nonshared pages together exceeded > that. Sounds somewhat fishy to me. This makes it all the more important to get a buildd.emdebian.org that can be all-but-hidden, have lots more RAM (4Gb) and oodles of disc space but which has only SSH access and no websites etc. My own systems have a 2Gb tmpfs. > Is it normal for grip-cron to take up that much space, or are there > special conditions to trigger it? After all, it should work on a > package-by-package basis IIRC. I think the problem is gripping very large packages that only have a small amount of fluff to be removed. The unpacking, filtering and repacking takes a fair bit of RAM. Any real figures you can obtain would be useful. It also depends whether Hector was building a toolchain at the same time, of course. What were the times of the relevant events? (grip-cron.sh starts at 3pm UTC. With a tmpfs, it was processing over 50 packages for seven architectures in about an hour and a half, including testing migrations. Without a tmpfs, that was closer to 7hrs but there have been other improvements in emdebian-grip-server 2.2.1 that could account for some of that too.) Until the size of the tmpfs is decided, the current runs are without a tmpfs (as it's not mounted) so the current logs can provide some comparison data too. Please wait until grip-cron.sh has finished before mounting the tmpfs again. -- Neil Williams ============= http://www.data-freedom.org/ http://www.linux.codehelp.co.uk/ http://e-mail.is-not-s.ms/
Attachment:
pgpRFn41gQMnl.pgp
Description: PGP signature