Re: Help to run test suite for python-pysam needed (does not find libhts) (Was: 01/01: Reenable check to ask for help on Debian Python list)

[Charles Plessy <plessy@debian.org>]

Hi Andreas,

I tried to build from the latest upstream sources by a) cloning the master
branch from https://github.com/pysam-developers/pysam and then copying the
debian directory from our Git-managed source package.

I had to disable the patch use_external_htslib.patch: setting HTSLIB_MODE to
"external" in setup.py makes that script crash.  It looks like the "external"
mode was superseded by passing HTSLIB_LIBRARY_DIR  HTSLIB_INCLUDE_DIR
environment variables.

Then, trying to build the binary package I reproduced the problem of missing
the faidx_fetch_nseq symbol.  When uploading libhts1 version 1.1 I actually
noticed that the symbol was dropped (thanks to the libhts1.symbols file), but
hoped that the reason why the upstream authors did not bump the soname of their
library after that backwards-incompatible change was that they knew it was not
problematic.  As it turns out, pysam needed it…  This said, given that the the
HTSlib 1.0 was only out for a few weeks, I think that the best is to move
forward by updating pysam to the HTSlib 1.1.  This is done easily with
s/faidx_fetch_nseq/faidx_nseq/.

Then, the package builds but some tests fail after trying to access a remote
file.

	make[2]: Leaving directory '/tmp/pysam/.pybuild/pythonX.Y_2.7/build/tests/pysam_data'
	...............................................................................................................................can't resolve 127.0.0.1:9/: Servname not supported for ai_socktype
	[E::hts_open] fail to open file 'http://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/phase3_EX_or_LC_only_alignment/data/HG00104/alignment/HG00104.chrom11.ILLUMINA.bwa.GBR.low_coverage.20130415.bam'
	E.........................building data files
	starting tests
	................can't resolve 127.0.0.1:9/: Servname not supported for ai_socktype
	[E::hts_open] fail to open file 'http://genserv.anat.ox.ac.uk/downloads/pysam/test/ex1.bam'
	Ecan't resolve 127.0.0.1:9/: Servname not supported for ai_socktype
	[E::hts_open] fail to open file 'http://genserv.anat.ox.ac.uk/downloads/pysam/test/ex1.bam'
	Ecan't resolve 127.0.0.1:9/: Servname not supported for ai_socktype
	[E::hts_open] fail to open file 'http://genserv.anat.ox.ac.uk/downloads/pysam/test/ex1.bam'
	E......[E::hts_open] fail to open file 'missing_file'
	...........
	======================================================================
	ERROR: testRead (__main__.TestHeader1000Genomes)
	----------------------------------------------------------------------
	Traceback (most recent call last):
	  File "./pysam_test.py", line 981, in testRead
	    f = pysam.Samfile(self.bamfile, "rb")
	  File "calignmentfile.pyx", line 319, in pysam.calignmentfile.AlignmentFile.__cinit__ (pysam/calignmentfile.c:4971)
	  File "calignmentfile.pyx", line 467, in pysam.calignmentfile.AlignmentFile._open (pysam/calignmentfile.c:6699)
	ValueError: could not open file (mode='rb') - is it SAM/BAM format?

	======================================================================
	ERROR: testFetch (__main__.TestRemoteFileHTTP)
	----------------------------------------------------------------------
	Traceback (most recent call last):
	  File "./pysam_test.py", line 1613, in testFetch
	    samfile = pysam.Samfile(self.url, "rb")
	  File "calignmentfile.pyx", line 319, in pysam.calignmentfile.AlignmentFile.__cinit__ (pysam/calignmentfile.c:4971)
	  File "calignmentfile.pyx", line 467, in pysam.calignmentfile.AlignmentFile._open (pysam/calignmentfile.c:6699)
	ValueError: could not open file (mode='rb') - is it SAM/BAM format?

	======================================================================
	ERROR: testFetchAll (__main__.TestRemoteFileHTTP)
	----------------------------------------------------------------------
	Traceback (most recent call last):
	  File "./pysam_test.py", line 1623, in testFetchAll
	    samfile = pysam.Samfile(self.url, "rb")
	  File "calignmentfile.pyx", line 319, in pysam.calignmentfile.AlignmentFile.__cinit__ (pysam/calignmentfile.c:4971)
	  File "calignmentfile.pyx", line 467, in pysam.calignmentfile.AlignmentFile._open (pysam/calignmentfile.c:6699)
	ValueError: could not open file (mode='rb') - is it SAM/BAM format?

	======================================================================
	ERROR: testView (__main__.TestRemoteFileHTTP)
	----------------------------------------------------------------------
	Traceback (most recent call last):
	  File "./pysam_test.py", line 1609, in testView
	    result = pysam.view(self.url, self.region)
	  File "/tmp/pysam/.pybuild/pythonX.Y_2.7/build/pysam/__init__.py", line 69, in __call__
	    (retval, "\n".join(stderr)))
	SamtoolsError: 'csamtools returned with error 1: '

	----------------------------------------------------------------------
	Ran 189 tests in 10.148s

	FAILED (errors=4)
	E: pybuild pybuild:256: test: plugin custom failed with: exit code=1: set -e; \
			     cp -a /tmp/pysam/tests /tmp/pysam/.pybuild/pythonX.Y_2.7/build/tests ; \
			     cd /tmp/pysam/.pybuild/pythonX.Y_2.7/build/tests && HTSLIB_LIBRARY_DIR=/usr/lib/x86_64-linux-gnu HTSLIB_INCLUDE_DIR=/usr/include PYTHONPATH=/tmp/pysam/.pybuild/pythonX.Y_2.7/build python2.7 ./pysam_test.py \
						  && HTSLIB_LIBRARY_DIR=/usr/lib/x86_64-linux-gnu HTSLIB_INCLUDE_DIR=/usr/include PYTHONPATH=/tmp/pysam/.pybuild/pythonX.Y_2.7/build python2.7 ./tabix_test.py 
	dh_auto_test: pybuild --test -i python{version} -p 2.7 --test --system=custom --test-args=set -e; \
			     cp -a /tmp/pysam/tests {build_dir}/tests ; \
			     cd {build_dir}/tests && HTSLIB_LIBRARY_DIR=/usr/lib/x86_64-linux-gnu HTSLIB_INCLUDE_DIR=/usr/include PYTHONPATH={build_dir} {interpreter} ./pysam_test.py \
						  && HTSLIB_LIBRARY_DIR=/usr/lib/x86_64-linux-gnu HTSLIB_INCLUDE_DIR=/usr/include PYTHONPATH={build_dir} {interpreter} ./tabix_test.py  --dir . returned exit code 13


I note that upstream committed changes to the test suite no later than
yesterday.  Maybe the best is to wait and see a week or two ?  Or to ask
him about his plans for the next pysam release ?

Have a nice week-end,

Charles

-- 
Charles Plessy
Debian Med packaging team,
http://www.debian.org/devel/debian-med
Tsurumi, Kanagawa, Japan