Re: Parallel GZIP --> speed instead of 2 CPUs running
On Mon, Dec 22, 2008 at 05:13:17PM +0200, Eugene V. Lyubimkin wrote:
> Thomas Preud'homme wrote:
> > The Monday 22 December 2008 13:53:43 Eugene V. Lyubimkin, you wrote :
...
> > I have a core 2 duo and time give me a usage of almost 200% with pbzip2
> >
> > Anyone has better results ?
> Yes, see following:
>
> jackyf@work:~/temp$ time tar -cjf temp1.tar.bz2 debian
>
> real 0m18.830s
> user 0m18.317s
> sys 0m0.268s
> jackyf@work:~/temp$ time bash -c "tar -cf - debian | pbzip2 -c > temp2.tar.bz2"
>
> real 0m10.494s
> user 0m19.557s
> sys 0m0.488s
>
> 10 is much less than 18.
Well.... bzip is slow anyway even with SMP.
The full story is here on my amd64 core2duo system:
Summary: (with -1 option for speed, real time for compression)
Binary files Text file
time(s) size time(s) size
cp 0.060 13166592 0.129 40323584
lzop 0.448 12727856 2.526 28777352
gzip 0.911 12689884 3.526 27281520
pbzip2 2.027 12804577 4.800 25928838
lzma 3.808 12772852 7.636 25926966
bzip2 4.358 12804175 9.938 25919654
lzop beats others with big margin (except cp) and provides decent compression.
Binary files here were compressed already, we gained little by using any
compression tools.
In any case, it is not I/O bound.
If the data is not compressible like this binary data, I see no point of using compression.
So my advise is test it for your data!
Osamu
-------------------
Here is my actual result...
osamu@snoopy:~/Desktop/tmp/zipzip$ zipzip
+ find /home/osamu/Desktop/Downloads -xdev -print0
+ cpio -ov --null -O test0.cpio
/home/osamu/Desktop/Downloads
/home/osamu/Desktop/Downloads/press080917.pdf
/home/osamu/Desktop/Downloads/kouzansousyoPDF.lzh
/home/osamu/Desktop/Downloads/kousoku.pdf
/home/osamu/Desktop/Downloads/06_map.pdf
/home/osamu/Desktop/Downloads/20081012_chuou_setagaya_sokuhou.pdf
/home/osamu/Desktop/Downloads/AoyagiKouzanFontSousyo2.lzh
/home/osamu/Desktop/Downloads/hokuseipanf.pdf
/home/osamu/Desktop/Downloads/04_5sangyou6kankou.pdf
25716 blocks
+ cp test0.cpio test1.cpio
+ cp test0.cpio test9.cpio
+ gzip -1 test1.cpio
real 0m0.911s
user 0m0.756s
sys 0m0.044s
+ gzip -9 test9.cpio
real 0m0.925s
user 0m0.852s
sys 0m0.020s
+ cp test0.cpio test-bz21.cpio
+ cp test0.cpio test-bz29.cpio
+ bzip2 -1 test-bz21.cpio
real 0m4.358s
user 0m3.712s
sys 0m0.168s
+ bzip2 -9 test-bz29.cpio
real 0m3.949s
user 0m3.884s
sys 0m0.064s
+ cp test0.cpio test-pbz21.cpio
+ cp test0.cpio test-pbz29.cpio
+ pbzip2 -1 test-pbz21.cpio
real 0m2.027s
user 0m3.668s
sys 0m0.060s
+ pbzip2 -9 test-pbz29.cpio
real 0m2.439s
user 0m4.548s
sys 0m0.144s
+ cp test0.cpio test-lzma1.cpio
+ cp test0.cpio test-lzma9.cpio
+ lzma -1 test-lzma1.cpio
real 0m3.808s
user 0m3.084s
sys 0m0.092s
+ lzma -9 test-lzma9.cpio
real 0m6.313s
user 0m5.980s
sys 0m0.228s
+ cp test0.cpio test-lzop1.cpio
+ cp test0.cpio test-lzop9.cpio
+ lzop -1 test-lzop1.cpio
real 0m0.448s
user 0m0.336s
sys 0m0.048s
+ lzop -9 test-lzop9.cpio
real 0m2.348s
user 0m2.268s
sys 0m0.048s
osamu@snoopy:~/Desktop/tmp/zipzip$ ls -la
total 163284
drwxr-xr-x 2 osamu osamu 4096 2008-12-23 01:07 .
drwxr-xr-x 5 osamu osamu 4096 2008-12-23 00:52 ..
-rw-r--r-- 1 osamu osamu 13166592 2008-12-23 01:06 test0.cpio
-rw-r--r-- 1 osamu osamu 12689884 2008-12-23 01:06 test1.cpio.gz
-rw-r--r-- 1 osamu osamu 12674359 2008-12-23 01:06 test9.cpio.gz
-rw-r--r-- 1 osamu osamu 12804175 2008-12-23 01:06 test-bz21.cpio.bz2
-rw-r--r-- 1 osamu osamu 12748183 2008-12-23 01:06 test-bz29.cpio.bz2
-rw-r--r-- 1 osamu osamu 12772852 2008-12-23 01:07 test-lzma1.cpio.lzma
-rw-r--r-- 1 osamu osamu 12721542 2008-12-23 01:07 test-lzma9.cpio.lzma
-rw-r--r-- 1 osamu osamu 13166592 2008-12-23 01:07 test-lzop1.cpio
-rw-r--r-- 1 osamu osamu 12727856 2008-12-23 01:07 test-lzop1.cpio.lzo
-rw-r--r-- 1 osamu osamu 13166592 2008-12-23 01:07 test-lzop9.cpio
-rw-r--r-- 1 osamu osamu 12705115 2008-12-23 01:07 test-lzop9.cpio.lzo
-rw-r--r-- 1 osamu osamu 12804577 2008-12-23 01:06 test-pbz21.cpio.bz2
-rw-r--r-- 1 osamu osamu 12748687 2008-12-23 01:06 test-pbz29.cpio.bz2
--------------------------------
osamu@snoopy:~/Desktop/tmp/zizi$ zizi
+ find /home/osamu/Desktop/Checkout/dpkg -xdev -print0
+ cpio -ov --null -O test0.cpio
/home/osamu/Desktop/Checkout/dpkg
/home/osamu/Desktop/Checkout/dpkg/ChangeLog
... (cut out) ...
/home/osamu/Desktop/Checkout/dpkg/README.translators
/home/osamu/Desktop/Checkout/dpkg/configure.ac
/home/osamu/Desktop/Checkout/dpkg/README.api
/home/osamu/Desktop/Checkout/dpkg/NEWS
/home/osamu/Desktop/Checkout/dpkg/triplettable
/home/osamu/Desktop/Checkout/dpkg/Makefile.am
78757 blocks
+ cp test0.cpio test1.cpio
+ cp test0.cpio test9.cpio
+ gzip -1 test1.cpio
real 0m3.526s
user 0m1.860s
sys 0m0.136s
+ gzip -9 test9.cpio
real 0m4.765s
user 0m3.480s
sys 0m0.160s
+ cp test0.cpio test-bz21.cpio
+ cp test0.cpio test-bz29.cpio
+ bzip2 -1 test-bz21.cpio
real 0m9.938s
user 0m8.957s
sys 0m0.116s
+ bzip2 -9 test-bz29.cpio
real 0m10.438s
user 0m10.261s
sys 0m0.116s
+ cp test0.cpio test-pbz21.cpio
+ cp test0.cpio test-pbz29.cpio
+ pbzip2 -1 test-pbz21.cpio
real 0m4.800s
user 0m9.129s
sys 0m0.136s
+ pbzip2 -9 test-pbz29.cpio
real 0m6.115s
user 0m11.689s
sys 0m0.196s
+ cp test0.cpio test-lzma1.cpio
+ cp test0.cpio test-lzma9.cpio
+ lzma -1 test-lzma1.cpio
real 0m7.636s
user 0m7.092s
sys 0m0.160s
+ lzma -9 test-lzma9.cpio
real 0m30.499s
user 0m29.378s
sys 0m0.460s
+ cp test0.cpio test-lzop1.cpio
+ cp test0.cpio test-lzop9.cpio
+ lzop -1 test-lzop1.cpio
real 0m2.526s
user 0m0.820s
sys 0m0.092s
+ lzop -9 test-lzop9.cpio
real 0m9.007s
user 0m8.065s
sys 0m0.152s
osamu@snoopy:~/Desktop/tmp/zizi$ ls -la
total 372256
drwxr-xr-x 2 osamu osamu 4096 2008-12-23 01:23 .
drwxr-xr-x 6 osamu osamu 4096 2008-12-23 01:20 ..
-rw-r--r-- 1 osamu osamu 40323584 2008-12-23 01:21 test0.cpio
-rw-r--r-- 1 osamu osamu 27281520 2008-12-23 01:21 test1.cpio.gz
-rw-r--r-- 1 osamu osamu 26215866 2008-12-23 01:21 test9.cpio.gz
-rw-r--r-- 1 osamu osamu 25919654 2008-12-23 01:21 test-bz21.cpio.bz2
-rw-r--r-- 1 osamu osamu 24659561 2008-12-23 01:21 test-bz29.cpio.bz2
-rw-r--r-- 1 osamu osamu 25926966 2008-12-23 01:22 test-lzma1.cpio.lzma
-rw-r--r-- 1 osamu osamu 23421982 2008-12-23 01:22 test-lzma9.cpio.lzma
-rw-r--r-- 1 osamu osamu 40323584 2008-12-23 01:22 test-lzop1.cpio
-rw-r--r-- 1 osamu osamu 28777352 2008-12-23 01:22 test-lzop1.cpio.lzo
-rw-r--r-- 1 osamu osamu 40323584 2008-12-23 01:22 test-lzop9.cpio
-rw-r--r-- 1 osamu osamu 26933083 2008-12-23 01:22 test-lzop9.cpio.lzo
-rw-r--r-- 1 osamu osamu 25928838 2008-12-23 01:22 test-pbz21.cpio.bz2
-rw-r--r-- 1 osamu osamu 24670171 2008-12-23 01:22 test-pbz29.cpio.bz2
osamu@snoopy:~/Desktop/tmp/zizi$ time cp test0.cpio test0-cp.cpio
real 0m0.129s
user 0m0.008s
sys 0m0.120s
> --
> Eugene V. Lyubimkin aka JackYF, JID: jackyf.devel(maildog)gmail.com
> Ukrainian C++ Developer, Debian Maintainer, APT contributor
>
Reply to: