[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: multi threaded support for xz



On 2016-10-06 23:47:05 [+0200], Guillem Jover wrote:
> Hi!
Hi,

> Please do not request any binNMU, I'm planning on doing a release
> soon, but before that I'll be investigating the effects of the new
> xz-utils package.

didn't plan on my own. Okay, please let me know it seems it did not
build on kfreebsd-*. Will try to fix this…

> > [0] https://ftp-master.debian.org/deferred.html
> 
> (BTW, it's customary when doing NMUs for new upstream versions to use the
> release -0.1 so that we do not take over the maintainer -1 release.)

good point. Thanks, will try to remember.

> On Thu, 2016-10-06 at 08:30:53 +0200, Sebastian Andrzej Siewior wrote:
> > With one CPU you have one block. With multiple CPUs the default block
> > size (as of current xz) is dictionary size * three. So it is
> > reproducible as long as you use one or multiple CPUs.
> > In order to have the same compressed archive with one or multiple CPUs
> > you would need a switch / environment variable to disable the use of
> > multiple CPUs.
> 
> Does this depend on the encoder interface being used? Because dpkg will
> always use the lzma_stream_encoder_mt() call regardless of the number
> of online CPUs compared to xz(1) which changes inerface on single or
> multi-threaded mode. In any case I'll be testing the repoducibility
> of this, and if need be check with xz upstream to get a more clear
> picture (either that or perform some code diving :).

No, not really. You don't specify the block_size parameter in filters so
xz takes care of this. This means:
- one CPU -> one block of all input data
- two or more CPUs -> dict_size times three is the size of each block

This is also what `xz -T1' vs `xz -T2' does (and as I said, -T2 vs -T8
produces the same xz binary). You can see the resulting block sizes and
number of blocks in "xz -lv".

> Thanks,
> Guillem

Sebastian


Reply to: