Re: Trying to build python3 version of scikit-learn, python3-* pkgs come out empty
I would recommend to look into rules of pandas package where I with
Julian's initiative have somewhat "harmonized" (although it is
still quite ugly) building both python2 and python3 with moving .so to
-lib packages .
I know that MDP's author is also eager to get python3-sklearn package so
it would be great if you join the effort... changes -- better send as a
PR against git://github.com/yarikoptic/scikit-learn.git -- would be
easier to review/merge
cheers!
On Tue, 04 Nov 2014, Zack Weinberg wrote:
> The scikit-learn packaging only builds python2 packages, even though
> upstream does support python3 (this is bug #730532). I happen to need
> scikit with python3 so I tried to update the packaging using the
> instructions at https://wiki.debian.org/Python/Pybuild , but I
> consistently get a python3-sklearn package containing only
> /usr/share/doc. Can anyone suggest what I might have done wrong? In
> the build log, it appears that pybuild did build both v2 and v3
> versions of the software, but then only v2 got copied into appropriate
> subdirectories of debian/ to be picked up by dpkg-deb.
> I'm attaching my modified debian/control and debian/rules. I already
> know that commenting out the override_dh_python2 block broke something
> *else* -- that's not the immediate problem.
> zw
> [-- octet-filter file type: "ASCII text" --]
> Source: scikit-learn
> Section: python
> Priority: optional
> Maintainer: NeuroDebian Team <team@neuro.debian.net>
> Uploaders: Yaroslav Halchenko <debian@onerussian.com>, Michael Hanke <michael.hanke@gmail.com>
> Build-Depends: debhelper (>= 8), dh-autoreconf, dh-python,
> python-all-dev (>= 2.6.6-3~),
> python3-all-dev (>= 3.4),
> cython (>= 0.12~),
> python-setuptools,
> python3-setuptools,
> python-numpy, python-scipy,
> python3-numpy, python3-scipy,
> libsvm-dev (>= 2.84.0),
> python-nose, python-matplotlib, python-joblib (>= 0.4.5),
> python3-nose, python3-matplotlib, python3-joblib (>= 0.4.5),
> gfortran, libatlas3-base, libatlas-dev, libblas-dev, liblapack-dev
> Build-Depends-Indep:
> python-sphinx (>= 1.0), python-imaging, graphviz,
> Standards-Version: 3.9.4
> X-Python-Version: >= 2.6
> X-Python3-Version: >= 3.4
> Homepage: http://scikit-learn.sourceforge.net
> Vcs-Browser: http://github.com/yarikoptic/scikit-learn
> Vcs-Git: git://github.com/yarikoptic/scikit-learn.git
> Package: python-sklearn
> Replaces: python-scikits-learn (<< 0.9~)
> Breaks: python-scikits-learn (<< 0.9~)
> Architecture: all
> Depends: ${misc:Depends}, ${python:Depends}, python-numpy, python-scipy, python-sklearn-lib(>= ${source:Version}),
> python-joblib (>= 0.4.5),
> Provides: ${python:Provides}
> Recommends: python-nose, python-matplotlib
> Suggests: python-dap, python-scikits-optimization, python-sklearn-doc, ipython
> Enhances: python-mvpa2, python-mdp
> Description: Python modules for machine learning and data mining
> scikit-learn is a collection of Python modules relevant to
> machine/statistical learning and data mining. Non-exhaustive list of
> included functionality:
> - Gaussian Mixture Models
> - Manifold learning
> - kNN
> - SVM (via LIBSVM)
> .
> This is the Python 2 version.
> Package: python3-sklearn
> Architecture: all
> Depends: ${misc:Depends}, ${python3:Depends}, python3-numpy, python3-scipy, python3-sklearn-lib(>= ${source:Version}),
> python3-joblib (>= 0.4.5),
> Provides: ${python3:Provides}
> Recommends: python3-nose, python3-matplotlib
> Suggests: python3-dap, python3-scikits-optimization, python3-sklearn-doc, ipython
> Enhances: python3-mvpa2, python3-mdp
> Description: Python modules for machine learning and data mining
> scikit-learn is a collection of Python modules relevant to
> machine/statistical learning and data mining. Non-exhaustive list of
> included functionality:
> - Gaussian Mixture Models
> - Manifold learning
> - kNN
> - SVM (via LIBSVM)
> .
> This is the Python 3 version.
> Package: python-sklearn-doc
> Conflicts: python-scikits-learn-doc
> Replaces: python-scikits-learn-doc
> Architecture: all
> Section: doc
> Depends: ${misc:Depends}, libjs-jquery
> Suggests: python-sklearn, python3-sklearn
> Description: documentation and examples for scikit-learn
> This package contains documentation and example scripts for
> python-sklearn.
> Package: python-sklearn-lib
> Conflicts: python-scikits-learn-lib
> Replaces: python-scikits-learn-lib
> Architecture: any
> Depends: ${misc:Depends}, ${shlibs:Depends}, ${python:Depends}, python-numpy
> Provides: ${python:Provides}
> Description: low-level implementations and bindings for scikit-learn
> This is an add-on package for python-sklearn. It provides
> low-level implementations and custom Python bindings for the LIBSVM
> library.
> .
> This is the Python 2 version.
> Package: python3-sklearn-lib
> Architecture: any
> Depends: ${misc:Depends}, ${shlibs:Depends}, ${python3:Depends}, python3-numpy
> Provides: ${python3:Provides}
> Description: low-level implementations and bindings for scikit-learn
> This is an add-on package for python3-sklearn. It provides
> low-level implementations and custom Python bindings for the LIBSVM
> library.
> .
> This is the Python 3 version.
> Package: python-scikits-learn
> Depends: python-sklearn, ${misc:Depends}, ${python:Depends}
> Architecture: all
> Section: oldlibs
> Description: transitional compatibility package for scikits.learn -> sklearn migration
> Provides old namespace (scikits.learn) and could be removed if
> dependent code migrated to use sklearn for clarity of the namespace.
> .
> This is the Python 2 version.
> Package: python3-scikits-learn
> Depends: python3-sklearn, ${misc:Depends}, ${python3:Depends}
> Architecture: all
> Section: oldlibs
> Description: transitional compatibility package for scikits.learn -> sklearn migration
> Provides old namespace (scikits.learn) and could be removed if
> dependent code migrated to use sklearn for clarity of the namespace.
> .
> This is the Python 3 version.
> [-- octet-filter file type: "HTML document, ASCII text" --]
> #!/usr/bin/make -f
> # -*- mode: makefile; coding: utf-8 -*-
> PACKAGE_NAME := python-sklearn
> PACKAGE_ROOT_DIR := debian/${PACKAGE_NAME}
> export PYBUILD_NAME := sklearn
> PYVERS := $(shell pyversions -vr)
> PYVER := $(shell pyversions -vd)
> #SPHINX_VER := $(shell python -c 'from sphinx import __version__; print __version__')
> #NODOC = $(shell dpkg --compare-versions $(SPHINX_VER) lt 1.1.3 && echo "nodoc" || echo "")
> #export DEB_BUILD_OPTIONS += $(NODOC)
> export DEB_BUILD_OPTIONS = nodoc nocheck
> CYTHON_VER := $(shell dpkg -l cython 2>/dev/null | awk '/^ii/{print $$3;}' || echo 0)
> RECONF = $(shell dpkg --compare-versions $(CYTHON_VER) ge 0.19 && echo ",autoreconf" || echo "")
> # Some tests are known to fail randomly so need to be excluded ATM
> NOSEARGS :=
> # --exclude='test_(sparse_svc_clone_with_callable_kernel|spectral_clustering_sparse|ovr_fit_predict)'
> # Mega rule
> %:
> : # Explicit build system to avoid use of all-in-1 Makefile
> dh $@ --buildsystem=pybuild --with python2,python3,$(RECONF)
> cythonize:
> @echo "D: removing previously generated by Cython sources"
> find -iname *.c | xargs grep -l 'Generated by Cython' | xargs rm
> make cython
> override_dh_autoreconf:
> dh_autoreconf debian/rules -- cythonize
> override_dh_clean:
> rm -rf build doc/_build doc/auto_examples *-stamp *.egg-info sklearn/datasets/__config__.py
> dh_clean
> override_dh_auto_install: ${PYVERS:%=python-install%} ${PYVERS:%=python-test%}
> # Per Python version logic -- install, test, move .so into -lib
> python-install%:
> python$* setup.py install --install-layout=deb --root=$(PACKAGE_ROOT_DIR)
> python-test%: python-install%
> ifeq (,$(filter nocheck,$(DEB_BUILD_OPTIONS)))
> : # Run unittests here against installed scikits.learn
> export PYTHONPATH=`/bin/ls -d $$PWD/$(PACKAGE_ROOT_DIR)/usr/lib/python$*/*/`; \
> export MPLCONFIGDIR=$(CURDIR)/build HOME=$(CURDIR)/build; \
> export JOBLIB_MULTIPROCESSING=0; \
> cd build/; python$* /usr/bin/nosetests -s -v $(NOSEARGS) sklearn && \
> { python$* /usr/bin/nosetests -v --with-doctest --doctest-extension=rst ../doc/ || : ; } # doctests are failing ATM
> else
> : # Skip unittests due to nocheck
> endif
> override_dh_installdocs:
> : # Build Documentation using installed scikits.learn
> ifeq (,$(filter nodoc,$(DEB_BUILD_OPTIONS)))
> ifneq (,$(findstring -a,$(DH_INTERNAL_OPTIONS)))
> : # not building documentation in -a
> else
> export PYTHONPATH=`/bin/ls -d $$PWD/$(PACKAGE_ROOT_DIR)/usr/lib/python$(PYVER)/*`; \
> export MPLCONFIGDIR=$(CURDIR)/build HOME=$(CURDIR)/build; \
> cd doc; $(MAKE) html
> endif
> else
> mkdir -p doc/_build/html
> echo "<HTML><BODY><P>sklearn documentation was not built.<P>Please visit <A HREF='http://scikit-learn.org'>http://scikit-learn.org</A></BODY></HTML>" >| doc/_build/html/index.html
> endif
> : # Use jquery from Debian package, so prune shipped one
> -rm doc/_build/html/_static/jquery.js
> dh_installdocs -A AUTHORS.rst README*.rst
> override_dh_installchangelogs:
> dh_installchangelogs doc/whats_new.rst
> ## move binary libraries into -lib
> #override_dh_python2:
> # : # Move platform-specific libraries into -lib
> # set -e; \
> # for lib in $$(find $(PACKAGE_ROOT_DIR)/usr -name '*.so'); do \
> # sdir=$$(dirname $$lib) ; \
> # tdir=$(PACKAGE_ROOT_DIR)-lib/$${sdir#*$(PACKAGE_NAME)/} ; \
> # mkdir -p $$tdir ; \
> # echo "Moving '$$lib' into '$$tdir'." ; \
> # mv $$lib $$tdir ; \
> # done
> # if [ -x /usr/bin/dh_numpy ]; then dh_numpy -ppython-sklearn-lib; fi
> # : # Prune scikits/__init__.py to avoid conflicts
> # find debian -wholename \*scikits/__init__.py -delete
> # : # Move scikits. space into a compatibility package
> # set -e; \
> # find debian -wholename \*scikits -type d | while read skd; do \
> # skbd=$$(dirname $$skd); \
> # skbd_=$$(echo $$skbd | sed -e 's/sklearn/scikits-learn/g'); \
> # mkdir -p $$skbd_; \
> # mv $$skd $$skbd_; \
> # done
> # dh_python2
> #override_dh_python3:
> # : # Move platform-specific libraries into -lib
> # set -e; \
> # for lib in $$(find $(PACKAGE_ROOT_DIR)/usr -name '*.so'); do \
> # sdir=$$(dirname $$lib) ; \
> # tdir=$(PACKAGE_ROOT_DIR)-lib/$${sdir#*$(PACKAGE_NAME)/} ; \
> # mkdir -p $$tdir ; \
> # echo "Moving '$$lib' into '$$tdir'." ; \
> # mv $$lib $$tdir ; \
> # done
> # if [ -x /usr/bin/dh_numpy ]; then dh_numpy -ppython-sklearn-lib; fi
> # : # Prune scikits/__init__.py to avoid conflicts
> # find debian -wholename \*scikits/__init__.py -delete
> # : # Move scikits. space into a compatibility package
> # set -e; \
> # find debian -wholename \*scikits -type d | while read skd; do \
> # skbd=$$(dirname $$skd); \
> # skbd_=$$(echo $$skbd | sed -e 's/sklearn/scikits-learn/g'); \
> # mkdir -p $$skbd_; \
> # mv $$skd $$skbd_; \
> # done
> # dh_python3
> # immediately useable documentation and exemplar scripts/data
> override_dh_compress:
> dh_compress -X.py -X.html -X.pdf -X.css -X.jpg -X.txt -X.js -X.json -X.rtc -Xobjects.inv
> ## to prepare next Debian upstream source tarball
> dfsg: dfsg-releases
> dfsg-%:
> git checkout dfsg
> -git merge --no-commit $*
> -git rm -rf sklearn/externals/joblib
> # -git rm -f ./sklearn/svm/src/libsvm/svm.*
> git commit -m "Merge $* into DFSG (pruning externals: joblib)" -a
> git checkout debian
--
Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Research Scientist, Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik
Reply to: