[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Bug#510624: ITP: pigz -- Parallel Implementation of GZip

#include <hallo.h>
* Guus Sliepen [Sun, Jan 04 2009, 10:45:23AM]:
> On Sat, Jan 03, 2009 at 09:57:33PM +0100, Eduard Bloch wrote:
> > PS: I plan to hack it a little bit and use syssconf function on Debian
> > systems to determine the real number of CPU cores (#x) since pigz's
> > default value is 8 which is much more than home systems have nowadays,
> > and the performance isn't getting (much) better with a constant number
> > of idle threads, they just consume more memory.
> Although sysconf() can tell you the total number of cores and the number of
> cores that are online in your system, it does not tell you how many cores are
> available for your program. It's better to use sched_getaffinity() to get the
> set of CPU (cores) available to your program. And if you know better than the
> OS, use sched_setaffinity() to bind each thread to it's own core.

Sounds like a plan, but I don't feel very comfortable to do that in the
Debian package. Let me explain why:

 - sched_setaffinity method seems to be Linux specific
 - though it might be beneficial to pinpoint each thread to the same
   CPU, I don't think (*assumption*) that the data resides long enough
   in the CPU cache anyway so we wouldn't win much
 - I have not used it before but I used the sysconf method in portable
   apps (i.e. in cloop-utils), sysconf method is also used in pbzip2
 - it's hard to imagine environments with big difference between
   count(cores) and count(available cores)
 - the code change is minimal, see below

@Marc: please check the -s option, it's called -S in gzip (upper case).
What's the reason for the different case?


--- pigz-2.1.4.orig/pigz.c
+++ pigz-2.1.4/pigz.c
@@ -2795,8 +2795,13 @@
 #ifdef NOTHREAD
     procs = 1;
+    procs=sysconf(_SC_NPROCESSORS_ONLN);
     procs = 8;
+#endif /* _SC_NPROCESSORS_ONLN */
+#endif /* NOTHREAD */
     size = 131072UL;
     rsync = 0;                      /* don't do rsync blocking */
     dict = 1;                       /* initialize dictionary each thread */

Naja, Garbage Collector eben. Holt den Müll sogar vom Himmel.
       (Heise Trollforum über Java in der Flugzeugsteuerung)

Attachment: signature.asc
Description: Digital signature

Reply to: