[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: [KMC + asmlib] KMC Debian package progress



Hi all,

KMC packaging is done.

http://anonscm.debian.org/cgit/debian-med/kmc.git/

Steffen, would you be able to sponsor this package?

Regards,

Jorge

On Mon, Nov 17, 2014 at 8:50 AM, Jorge Sebastião Soares <j.s.soares@gmail.com> wrote:
Hi Marek,

Sorry for the time it took for me to get back to you.
I am at the point that I have the kmc package ready to be submitted.
Just doing final tidy ups and automating man page creation for the two binaries (kmc and kmc_dump).
To do this I use help2man which has been proving great for turning any kind of usage into man pages.
I should be done with this by lunch time today.

At the moment, for the creaiton of the kmc debian package I'm using my github repository.
I hope that's ok for you for the moment.
Once you guys are done with writing the new version of kmc we can point the package to look at your own VCS repository.
Let me know if this ok for now.

On Wed, Nov 12, 2014 at 6:14 PM, Marek Kokot <Marek.Kokot@polsl.pl> wrote:
Hello Jorge,

Hope all is well.
Its fine, thanks. How are you? ;-)


Great!
I'm very well as well. :)
 
Thank you for your tests. I have made a couple of tests on my own and the results was pretty similar to your. As I said it was only couple of tests but on reative big data (human genome about 100 GB gzipped). 

Cool!
 

My question is: Have you used HDD or SSD? I have used HDD, for SSD it is possible that differences in time may be bigger. I will try to test it for SSD.

I was running it on HDD.
Did you run it on SSD already? Did you get any significant improvement?
 

I spoke with Sebastian today and we agree that it is a good option to make compilation without asmlib possible. I will deal with this and I let you know when it is ready.

Awesome!
I'm so glad to hear this.
This will also mean that KMC will be able to be built for other chips other than the intel ones.
This feature will be available only with the new verison of KMC?
 

KMC in current version has also "boost" dependency, but we think to go native, because g++ supports c++11 threads. Other functionalities from "boost" that we use are easy to replace with c++11. It should make compilation time shorter for the ones who hasn't "boost" installed.

I see. It does take a while ot compile, but dependency on Boost is definitely not an issue from where I'm sitting.
The package builds properly with the Debian boost, zlib and libbz2 dependencies.
I do appreciate that it does take a while to compile.
 

In one of earlier e-mails you asked about Agner (asmlib author):
Do you think it would make sense for me to approach him through the Debian Med team?
We would prefer to deal with it on our own.

No problem at all.

 
If there is something I ommited and I didn't answer to any of your questions please let me know.

I think you answered everything. I do appologise for stressing you guys out with these requests, especially at a time when you're concentrating on writing a new release.

Kind regards,

Jorge Soares
 


Od: Jorge Sebastião Soares [j.s.soares@gmail.com]
Wysłano: 11 listopada 2014 18:00
Do: Marek Kokot; Sebastian Deorowicz
DW: Debian Med Project List
Temat: [KMC + asmlib] KMC Debian package progress

Hi guys,

I had to send this again as I used my Sanger email, but that is not subscribed to the Debian Med Mailing list.

So here it goes again (if you can ignore the previous one and respond to this one).

Hope all is well.
Have you given any thought to my proposal of a compile time option that won't use asmlib?

I have included the Debian Med team on this email as they are aware of the packaging of KMC and the whole issue with asmlib.

I have been doing some benchmarking on KMC for the past couple of days.
I have compiled KMC in three ways:

kmc_original - kmc code compiled against the version of asmlib distributed with KMC- alibelf64.a
kmc_native - kmc code compiled against the native OS libraries
kmc_js21 - kmc code compiled against the new version of asmlib, compiled on my machine with my Unix makefile - libaelf64.a

I have also used the executables provided in your website in the benchmark.

kmc_exe

The machine I used for this is a Debian Virtual Machine running on Vagrant.

Here are the architecture details:

vagrant@debian:~$ cat /proc/cpuinfo
processor    : 0
vendor_id    : GenuineIntel
cpu family    : 6
model        : 23
model name    : Intel(R) Core(TM)2 Duo CPU     P8600  @ 2.40GHz
stepping    : 10
microcode    : 0x60b
cpu MHz        : 1426.514
cache size    : 6144 KB
physical id    : 0
siblings    : 1
core id        : 0
cpu cores    : 1
apicid        : 0
initial apicid    : 0
fpu        : yes
fpu_exception    : yes
cpuid level    : 5
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm constant_tsc rep_good nopl pni monitor ssse3 lahf_lm
bogomips    : 2853.02
clflush size    : 64
cache_alignment    : 64
address sizes    : 36 bits physical, 48 bits virtual

To do the benchmark I used a fastq file that has a fair bit of contamination (many different kmers). The file is about 227M in size.

Here are some of the results:

For an average time in seconds over 10 runs for all the differently compiled executables:

[1] kmc_original - 'average_duration' => '41.400'
[2] kmc_native - 'average_duration' => '41.675'
[3] kmc_js21 - 'average_duration' => '41.249'
[4] kmc_exe - 'average_duration' => '44.049'

The cumulative time for 10 runs for all the differently compiled executables:

[5] kmc_original - 'time_taken' => '414 wallclock secs ( 0.26 usr  0.76 sys + 347.94 cusr 61.00 csys = 409.96 CPU) @  0.02/s (n=10)'
[6] kmc_native - 'time_taken' => '412 wallclock secs ( 0.15 usr  0.85 sys + 345.37 cusr 61.17 csys = 407.54 CPU) @  0.02/s (n=10)'
[7] kmc_js21 - 'time_taken' => '423 wallclock secs ( 0.11 usr  0.82 sys + 355.10 cusr 61.95 csys = 417.98 CPU) @  0.02/s (n=10)'
[8] kmc_exe - 'time_taken' => '434 wallclock secs ( 0.06 usr  0.78 sys + 368.63 cusr 60.14 csys = 429.61 CPU) @  0.02/s (n=10)

Note: More detailed results at the end of the email.

From what I can see, [1] generally runs faster than [2] and [3], albeit, only 1 or 2%.

Looking at the cumulative times for a set of 10 runs, the difference between implementations is still small and in this case the native implementation was actually faster. The machine I was doing the benchmark on wasn't fully dedicated to the benchmark.
So there will be slight variation. I would still like to run the benchmark for 100 runs for both methods. I would like to try this over night, just for the 2 main kmc implementations: [1] and [2].
But I'm looking at 23 hours, so essentially one whole day really. I'm not sure I'll do it tonight. Perhaps from Friday to Saturday.  

Anywa, I understand that this performance increase might mean a lot to you, but our group here at the Sanger and Debian can definitely live with the native implementation.

Since the author of asmlib is taking a while to reply, our suggestion would be to package KMC in one of two ways:

1- an implementation on your side which allows KMC to be built without using asmlib; (preferred);

2- using the compilation that does not use asmlib at all (which can be done on my side, as a code patch, at package creation time).

This would make the packaging job slightly easier and faster and would allow me to reach my goal. Packaging the virus assembler written by my colleague here at the Sanger.

If/when Agner (author of KMC) replies we can always package asmlib and state it as a dependency for the KMC package.


Detailed Results:

The benchmarking results were done through a perl script using 2 perl Modules:

[9] Time::HiRes => High resolution alarm, sleep, gettimeofday, interval timers
[10] Benchmark => benchmark running times of Perl code

The options used for the KMC runs are the same as the ones used for the Virus Assembler (IVA) runs:

kmc -k100 -m4 -ci10 -cs100000000 -fq foo.fastq kmc.res bar/

Results with [9] -

Average time in seconds for 10 runs logged under the 'average_duration' attribute.

          'commandline_parameters' => {
                                        'module' => 'time_hires',
                                        'number_of_runs' => 10,
                                        'file_type' => 'fq',
                                        'fastaq_filename' => '12950_1#10_1.fastq'
                                      },
         'kmc_exe' => {
                         'cmd' => '../kmc_exe/kmc -k100 -m4 -ci10 -cs100000000 -fq 12950_1#10_1.fastq ke_out.res /home/vagrant/build/kmc_bin/perl_profiler/ke_analysis/',
                         'output_filename' => 'ke_out.res',
                         'analysis_dir' => '/home/vagrant/build/kmc_bin/perl_profiler/ke_analysis/',
                         'kmc_exe' => '../kmc_exe/kmc',
                         'duration' => '46.303',
                         'average_duration' => '44.049'
                       },
          'kmc_native' => {
                                    'analysis_dir' => '/home/vagrant/build/kmc_bin/perl_profiler/kn_analysis/',
                                    'output_filename' => 'kn_out.res',
                                    'average_duration' => '41.675',
                                    'kmc_exe' => '../kmc_native/kmc',
                                    'duration' => '41.688',
                                    'cmd' => '../kmc_native/kmc -k100 -m4 -ci10 -cs100000000 -fq 12950_1#10_1.fastq kn_out.res /home/vagrant/build/kmc_bin/perl_profiler/kn_analysis/'
                                  },
          'kmc_js21' => {
                          'cmd' => '../kmc_js21/kmc -k100 -m4 -ci10 -cs100000000 -fq 12950_1#10_1.fastq kj_out.res /home/vagrant/build/kmc_bin/perl_profiler/kj_analysis/',
                          'output_filename' => 'kj_out.res',
                          'analysis_dir' => '/home/vagrant/build/kmc_bin/perl_profiler/kj_analysis/',
                          'kmc_exe' => '../kmc_js21/kmc',
                          'duration' => '40.714',
                          'average_duration' => '41.249'
                        },
          'kmc_original' => {
                              'output_filename' => 'kk_out.res',
                              'analysis_dir' => '/home/vagrant/build/kmc_bin/perl_profiler/kk_analysis/',
                              'duration' => '42.670',
                              'kmc_exe' => '../kmc_kmc/kmc',
                              'average_duration' => '41.400',
                              'cmd' => '../kmc_kmc/kmc -k100 -m4 -ci10 -cs100000000 -fq 12950_1#10_1.fastq kk_out.res /home/vagrant/build/kmc_bin/perl_profiler/kk_analysis/'
                            },
        };


The times are consistent with the times reported by the KMC instances when they run on verbose mode.

Results with [10] -

Cumulative time in seconds logged under the 'time_taken' attribute.

          'commandline_parameters' => {
                                        'number_of_runs' => 10,
                                        'fastaq_filename' => '12950_1#10_1.fastq',
                                        'file_type' => 'fq',
                                        'module' => 'bench'
                                      },
            'kmc_original' => {
                              'output_filename' => 'kk_out.res',
                              'time_taken' => '414 wallclock secs ( 0.26 usr  0.76 sys + 347.94 cusr 61.00 csys = 409.96 CPU) @  0.02/s (n=10)',
                              'kmc_exe' => '../kmc_kmc/kmc',
                              'analysis_dir' => '/home/vagrant/build/kmc_bin/perl_profiler/kk_analysis/',
                              'cmd' => '../kmc_kmc/kmc -k100 -m4 -ci10 -cs100000000 -fq 12950_1#10_1.fastq kk_out.res /home/vagrant/build/kmc_bin/perl_profiler/kk_analysis/',
                              'duration' => ''
                            },
          'kmc_native' => {
                                    'duration' => '',
                                    'cmd' => '../kmc_native/kmc -k100 -m4 -ci10 -cs100000000 -fq 12950_1#10_1.fastq kn_out.res /home/vagrant/build/kmc_bin/perl_profiler/kn_analysis/',
                                    'analysis_dir' => '/home/vagrant/build/kmc_bin/perl_profiler/kn_analysis/',
                                    'kmc_exe' => '../kmc_native/kmc',
                                    'time_taken' => '412 wallclock secs ( 0.15 usr  0.85 sys + 345.37 cusr 61.17 csys = 407.54 CPU) @  0.02/s (n=10)',
                                    'output_filename' => 'kn_out.res'
                                  },
          'kmc_js21' => {
                          'kmc_exe' => '../kmc_js21/kmc',
                          'time_taken' => '423 wallclock secs ( 0.11 usr  0.82 sys + 355.10 cusr 61.95 csys = 417.98 CPU) @  0.02/s (n=10)',
                          'output_filename' => 'kj_out.res',
                          'duration' => '',
                          'cmd' => '../kmc_js21/kmc -k100 -m4 -ci10 -cs100000000 -fq 12950_1#10_1.fastq kj_out.res /home/vagrant/build/kmc_bin/perl_profiler/kj_analysis/',
                          'analysis_dir' => '/home/vagrant/build/kmc_bin/perl_profiler/kj_analysis/'
                        },
          'kmc_exe' => {
                         'duration' => '',
                         'cmd' => '../kmc_exe/kmc -k100 -m4 -ci10 -cs100000000 -fq 12950_1#10_1.fastq ke_out.res /home/vagrant/build/kmc_bin/perl_profiler/ke_analysis/',
                         'analysis_dir' => '/home/vagrant/build/kmc_bin/perl_profiler/ke_analysis/',
                         'kmc_exe' => '../kmc_exe/kmc',
                         'time_taken' => '434 wallclock secs ( 0.06 usr  0.78 sys + 368.63 cusr 60.14 csys = 429.61 CPU) @  0.02/s (n=10)',
                         'output_filename' => 'ke_out.res'
                       }
        };


Let me know what you think.

Kind regards,

Jorge



Reply to: