Re: Debian packaging of phylophlan and using vsearch instead of usearch

[Andreas Tille <andreas@an3as.eu>]

Hi Francesco,

On Tue, May 31, 2016 at 06:07:00PM +0200, Francesco Asnicar wrote:
> the usearch version for PhyloPhlAn is "usearch5.2.32". The newest versions
> of usearch are supported in a new version of PhyloPhlAn that is currently
> under development, so we cannot refer to this as stable.

Thanks for the helpful explanation.  This enables me to find at least
for local installations some clue.
 
> About the errors you got from the "guide" example, did you run all the
> steps in order? Because if you only run "step3.sh", there would be missing
> files if the previous steps have not been executed.

I *think* I did so, but I'm not sure.  Since it works now we might
consider this issue settled. :-)
 
> About the errors you got from the "internal_labels" example, instead, I
> found a couple of missing taxa in the tree and a couple of wrong
> annotations. I now fixed the example and updated the repository. Many
> thanks for pointing out this problem.

You are welcome.  I consider this a feature of Debian packaging to get
another pair of eyeballs onto a piece of code.
 
> About the real error you got from the "phylo_small" example, I looked into
> it, but here the example is working just fine. Could it be related to the
> matplotlib version? You mentioned the matplotlib warning, but I couldn't
> reproduce it. I am currently using the 1.3.1 version of matplotlib, which
> one are you using?

On the machine I was doing the test I'm running Debian testing which has
python-matplotlib version 1.5.1.
 
> Instead of including PyPhlAn as sub-repository with GraPhlAn, I included
> the necessary classes directly in GraPhlAn. I also added a tag ("1.1",
> https://bitbucket.org/nsegata/graphlan/commits/tag/1.1) to the latest
> commit in Bitbucket.

That's very helpful.  Since I did not needed to package any
preconditions I was able to upload a package to the Debian new queue[1]
where all new packages need to go first for inspection of Debian
ftpmaster.

Regarding the examples I turned running them into a test suite which
Debian tries to provide as Continuous Integration tests.  You might like
to have a look at the test script[2] (which assumes that the examples
are installed to /usr/share/doc/graphlan/examples).  I have attached the
output of this script when running on my machine.  As you can see there
are two issues:

 1. I needed to skip those tests that are calling export2graphlan
    (saliva_microbiome/PIPELINE.sh and other PIPELINE.sh scripts).  I
    wonder where I could get export2graphlan to be able to run all tests.
 2. metaref_fams/run.sh has some extended output also containing strings
    like
     Error: No tree file found: mr1.xml

Please let me know how to deal with these to issues.
 
> Let us know if you have any other questions.

Not really a question but a hint:  I used help2man to create some simple
manpages from the output when specifying '-h'.  Feel free to download
these[3] and add them to your source distribution.

Kind regards and thanks for your cooperation

    Andreas.

[1] https://ftp-master.debian.org/new/graphlan_1.1-1.html
[2] https://anonscm.debian.org/viewvc/debian-med/trunk/packages/graphlan/trunk/debian/tests/run-unit-test?revision=22059&view=markup
[3] https://anonscm.debian.org/viewvc/debian-med/trunk/packages/graphlan/trunk/debian/
 
> On Fri, May 27, 2016 at 3:48 PM, Andreas Tille <andreas@fam-tille.de> wrote:
> 
> > Sorry for answering my own mail.  When trying the examples of graphlan I
> > noticed that some files are missing:
> >
> >
> > graphlan/examples/guide $ ./step3.sh
> > Error: No tree file found: guide_2.xml
> > Error: No tree file found: guide_3.xml
> > Error: No tree file found: guide_3.xml
> >
> > graphlan/examples/guide $ ./step4.sh
> > Error: No tree file found: guide_3.xml
> > Error: No tree file found: guide_4.xml
> > Error: No tree file found: guide_4.xml
> >
> >
> > I'm not sure about this, please check:
> >
> > graphlan/examples/internal_labels $ ./step4.sh
> > Classes not implemented for external annotations
> > Classes not implemented for external annotations
> > ...
> >
> >
> > I also observed two times warnings from matplotlib saying:
> >
> >    /usr/lib/python2.7/dist-packages/matplotlib/projections/polar.py:149:
> > RuntimeWarning: invalid value encountered in true_divide
> >
> >
> > A real error occures with
> >
> > graphlan/examples/phylo_small $ ./run.sh
> > Traceback (most recent call last):
> >   File "/usr/bin/graphlan", line 70, in <module>
> >     external_legends = args['external_legends'])
> >   File "/usr/share/graphlan/src/graphlan_lib.py", line 1062, in draw
> >     format = out_format,
> >   File "/usr/lib/python2.7/dist-packages/matplotlib/pyplot.py", line 577,
> > in savefig
> >     res = fig.savefig(*args, **kwargs)
> >   File "/usr/lib/python2.7/dist-packages/matplotlib/figure.py", line 1470,
> > in savefig
> >     self.canvas.print_figure(*args, **kwargs)
> >   File "/usr/lib/python2.7/dist-packages/matplotlib/backend_bases.py",
> > line 2141, in print_figure
> >     **kwargs)
> >   File
> > "/usr/lib/python2.7/dist-packages/matplotlib/backends/backend_agg.py", line
> > 521, in print_png
> >     FigureCanvasAgg.draw(self)
> >   File
> > "/usr/lib/python2.7/dist-packages/matplotlib/backends/backend_agg.py", line
> > 469, in draw
> >     self.figure.draw(self.renderer)
> >   File "/usr/lib/python2.7/dist-packages/matplotlib/artist.py", line 59,
> > in draw_wrapper
> >     draw(artist, renderer, *args, **kwargs)
> >   File "/usr/lib/python2.7/dist-packages/matplotlib/figure.py", line 1079,
> > in draw
> >     func(*args)
> >   File "/usr/lib/python2.7/dist-packages/matplotlib/artist.py", line 59,
> > in draw_wrapper
> >     draw(artist, renderer, *args, **kwargs)
> >   File "/usr/lib/python2.7/dist-packages/matplotlib/axes/_base.py", line
> > 2092, in draw
> >     a.draw(renderer)
> >   File "/usr/lib/python2.7/dist-packages/matplotlib/artist.py", line 59,
> > in draw_wrapper
> >     draw(artist, renderer, *args, **kwargs)
> >   File "/usr/lib/python2.7/dist-packages/matplotlib/patches.py", line 462,
> > in draw
> >     gc.set_alpha(self._alpha)
> >   File "/usr/lib/python2.7/dist-packages/matplotlib/backend_bases.py",
> > line 923, in set_alpha
> >     self.set_foreground(self._orig_color)
> >   File "/usr/lib/python2.7/dist-packages/matplotlib/backend_bases.py",
> > line 987, in set_foreground
> >     self._rgb = colors.colorConverter.to_rgba(fg, self._alpha)
> >   File "/usr/lib/python2.7/dist-packages/matplotlib/colors.py", line 376,
> > in to_rgba
> >     'to_rgba: Invalid rgba arg "%s"\n%s' % (str(arg), exc))
> > ValueError: to_rgba: Invalid rgba arg "(0.0, 0.0, 1.0, 1.0175609756098)"
> > number in rbga sequence outside 0-1 range
> >
> >
> > BTW, a conversion to Python 3 for graphlan as well as pyphlan would be
> > nice.
> >
> > Kind regards
> >
> >      Andreas.
> >
> > On Fri, May 27, 2016 at 03:32:37PM +0200, Andreas Tille wrote:
> > > Hi Nicola,
> > >
> > > On Wed, May 25, 2016 at 07:56:08AM +0000, Nicola Segata wrote:
> > > > Hi Adreas,
> > > >   we recently updated PhyloPhlAn to support other usearch versions. I'm
> > > > CC'ing Francesco (the lead on the project) which can be more accurate
> > on
> > > > which usearch version we need for the last version of PhyloPhlAn.
> > >
> > > It would be nice to get some confirmation about the usearch version.
> > >
> > > > Regarding other software I think it would be fantastic to have
> > MetaPhlAn2
> > > > as a Debian package. It's here:
> > https://bitbucket.org/biobakery/metaphlan2 .
> > > > It is widely used and stable (although we are further and actively
> > > > developing it) and published a couple of times in Nat Methods.
> > >
> > > I had a look into this.  It contains quite large binary data chunks.
> > > Before I can seriously attempt to package this I would need to know
> > > where and how theses data are obtained from, how to possibly change the
> > > data (if this makes sense at all - if not why not) and what the
> > > copyright of the data would be.  Otherwise Debian ftpmaster will refuse
> > > the package to be uploaded.
> > >
> > > > Another software we developed that would be great to have as a Debian
> > > > package is GraPhlAn (https://bitbucket.org/nsegata/graphlan/wiki/Home)
> > > > (Described here https://peerj.com/articles/1029/). It's a package to
> > > > produce nice graphical representation of phylogenetic trees.
> > >
> > > That's a really nice one. :-)  I injected packaging code into our
> > > packaging version control system.  However, when doing so I noticed that
> > > I need to package pyphlan[1] first.  I committed the needed packaging
> > > code into our packaging Git and can confirm that the packaging of
> > > graphlan works together with this precondition.  Before I can upload
> > > pyphlan to the Debian mirror it would be great to have at least a
> > > minimum three lines description, what the package is actually doing.  I
> > > found something in the code that there are functions you would like to
> > > see in Biopython (did you contacted Biopython authors about this - I
> > > learned they are very responsive and open for enhancements) but I have
> > > no real clue what the code is doing - except that its a precondition for
> > > graphlan.  So please provide some sensible description.
> > >
> > > Another nice thing would to add a version tag to bitbucket.  This would
> > > enable me to find a proper Debian package version which is currently
> > > simply the date of the youngest file inside the download archive.
> > >
> > > > Importantly, these two packages don't use any proprietary software so
> > it
> > > > should be easier to package them than PhyloPhlAn.
> > >
> > > You are right but as I said above the binary data need some explanation.
> > >
> > > > Let us know if you have any question and if we can help you somehow!
> > >
> > > Thanks a lot for your nice hints.  We like to have this kind of
> > > cooperation with upstream software authors.
> > >
> > > Kind regards
> > >
> > >        Andreas.
> > >
> > >
> > > [1] https://bitbucket.org/nsegata/pyphlan
> > > [2] https://anonscm.debian.org/cgit/debian-med/python-pyphlan.git
> > >
> > > --
> > > http://fam-tille.de
> >
> > --
> > http://fam-tille.de
> >

-- 
http://fam-tille.de

 $ sh run-unit-test 
Running test script ./IBD_biogeography/run.sh .
Running test script ./HMP_tree_simple/run.sh .
Running test script ./phylo_small/run.sh .
Test script ./saliva_microbiome/PIPELINE.sh requires export2graphlan which is not available.
Running test script ./HMP_tree/run.sh .
Running test script ./guide/step0.sh .
Running test script ./guide/step1.sh .
Running test script ./guide/step2.sh .
Running test script ./guide/step3.sh .
Running test script ./guide/step4.sh .
Running test script ./DisappearingMicrobiome/run.sh .
Running test script ./metaref_fams/run.sh .
4
Lactobacillus
set([u'Blautia', u'Thermoactinomycetaceae', u'Lactobacillales', u'Ruminococcaceae', u'Thermoanaerobacterales', u'Bacillale', u'Helcococcus', u'Bacill', u'Clostridiale', u'Peptoniphilus', u'Thermoactinomyces', u'Ureibacillus', u'Planococcaceae', u'Bacillales_incertae_sedis', u'Abiotrophia', u'Veillonellaceae', u'Carnobacterium', u'Kurthia', u'Brevibacillus', u'Finegoldia', u'Sarcina', u'Alkaliphilus', u'Staphylococcus', u'Acidaminococcus', u'Clostridi', u'Granulicatella', u'Bacillus', u'Clostridium', u'Incertae_Sedis_XI', u'Tumebacillus', u'Clostridiales', u'Incertae_Sedis_XIII', u'Mitsuokella', u'Planomicrobium', u'Subdoligranulum', u'Hydrogenoanaerobacterium', u'Gallicola', u'Trichococcus', u'Clostridiaceae', u'Bacillales', u'Alkalibacterium', u'Oscillibacter', u'Selenomonas', u'AA', u'Bacillaceae', u'Exiguobacterium', u'Anaerococcus', u'Jeotgalicoccus', u'Megasphaera', u'Anaeroglobus', u'Dialister', u'Sporosarcina', None, u'Succiniclasticum', u'Papillibacter', u'Thermoanaerobacterium', u'Staphylococcaceae', u'Gemella', u'Thermicanus', u'Clostridia', u'Megamonas', u'Oxobacter', u'Veillonella', u'Geobacillus', u'Eremococcus', u'Aerococcaceae', u'Carnobacteriaceae', u'Paenibacillaceae', u'Incertae_Sedis_XIV', u'Anaerovorax', u'Rummeliibacillus', u'unclassified', u'Parvimonas', u'Macrococcus', u'Thermoanaerobacteraceae', u'Desemzia', u'Mogibacterium', u'Ammoniphilus', u'Aerococcus', u'Bacilli', u'Marinilactibacillus', u'Anoxybacillus', u'Dolosigranulum', u'Planifilum'])
['Bacilli.Lactobacillales.Lactobacillaceae.Lactobacillus', 'ring_shape', '1', 'v']
Classes not implemented for external annotations
Error: No tree file found: mr1.xml
Error: No tree file found: mr1.xml
Warning: label "Y" has a default gray background because no color is found for the corresponding "annotation"
Warning: label "Y1" has a default gray background because no color is found for the corresponding "annotation"
Warning: label "Y2" has a default gray background because no color is found for the corresponding "annotation"
Warning: label "Y3" has a default gray background because no color is found for the corresponding "annotation"
Warning: label "Y" has a default gray background because no color is found for the corresponding "annotation"
Warning: label "Y1" has a default gray background because no color is found for the corresponding "annotation"
Warning: label "Y2" has a default gray background because no color is found for the corresponding "annotation"
Warning: label "Y3" has a default gray background because no color is found for the corresponding "annotation"
Test script ./vaginal_microbiota/PIPELINE.sh requires export2graphlan which is not available.
Test script ./hmp_metahit/PIPELINE.sh requires export2graphlan which is not available.
Running test script ./kegg/run.sh .
Running test script ./gut_microbiome/run.sh .
Running test script ./simple/run.sh .
Test script ./hmp_metahit_functional/PIPELINE.sh requires export2graphlan which is not available.
Running test script ./archaea/run.sh .
Running test script ./PhyloPhlAn/run.sh .
Running test script ./internal_labels/step0.sh .
Running test script ./internal_labels/step1.sh .
Running test script ./internal_labels/step2.sh .
Running test script ./internal_labels/step3.sh .
Running test script ./internal_labels/step4.sh .