Sage 5.9.beta1 and debian : progress report
Hi,
on a debian box:
- install all packages mentioned in spkg_to_dpkg.py with the right
version -- some are available from unstable or experimental but others
require building them from the debian git repositories (and see what is
left to do)
- unpack sage-5.9.tar.gz somewhere
- change debian_pruner.py's "sagedir=" line to point to where you
unpacked sage sources
- run debian_pruner.py to do magic symlinking
- type 'make' in sage's directory
- interrupt the build when it's in the doc (it doesn't do anything
anyway: it's very broken)
- kill the remaining processes (I did mention building the doc was
broken...)
(at that point you should have a quite useable sage)
- apply the patches from http://trac.sagemath.org/sage_trac/ticket/11334
(those will very likely end up in a sage version soon) and the patch
from http://trac.sagemath.org/sage_trac/ticket/14365 (that one won't
find its way in directly but I hope the test will disappear) ;
- run sage -b to rebuild the just-patched files
- run spkg/pipestatus "./sage -tp --sagenb --long devel/sage/doc/common
devel/sage/doc/[a-z][a-z] devel/sage/sage 2>&1" "tee -a ptestlong.log"
to run the sage test suite ("make ptestlong" will try to compile the doc
and get stuck, so really use that line instead)
then the failing tests are like this:
sage -t --long
devel/sage/sage/schemes/elliptic_curves/ell_rational_field.py # 1
doctest fails (apparently some computation is supposed to fail, but
doesn't!)
sage -t --long devel/sage/sage/interfaces/expect.py # 1 doctest fails
(strange prompt issue)
sage -t --long devel/sage/sage/misc/sagedoc.py # 7 doctests fail
(because the doc hasn't been built)
sage -t --long devel/sage/sage/doctest/forker.py # 1 doctest fails
(because we fail in cremona.py!)
sage -t --long devel/sage/sage/geometry/lattice_polytope.py # 2
doctests fail (because they expect palp to fail in dimension 7 and
debian's palp goes to 11)
sage -t --long devel/sage/sage/tests/cmdline.py # 2 doctests fail (two
cases of getting a blank line instead of an error refusing to run doctests)
sage -t --long devel/sage/sage/doctest/control.py # 1 doctest fails
(doctest returns a blank line instead of an error)
sage -t --long devel/sage/sage/calculus/test_sympy.py # 1 doctest fails
(display difference, the result is correct)
sage -t --long devel/sage/sage/symbolic/constants.py # 1 doctest fails
(sympy.nan == NaN is supposed to be True...)
sage -t --long devel/sage/sage/interfaces/r.py # 4 doctests fail
(trivial display issue, the results are good!)
sage -t --long devel/sage/sage/misc/sage_extension.py # 1 doctest fails
(because we don't patch ipython to parse %time magic command)
sage -t --long devel/sage/sage/interacts/debugger.py # 2 doctests fail
(pexpect difference, perhaps due to our not having the same paths?)
sage -t --long devel/sage/doc/en/faq/faq-usage.rst # 1 doctest fails
(numpy-related?)
sage -t --long devel/sage/sage/plot/plot3d/implicit_surface.pyx # 2
doctests fail (some numpy methods seem to return True/None instead of
True/False!)
sage -t --long devel/sage/sage/numerical/mip.pyx # 1 doctest fails
(because our glpk version number isn't the same)
sage -t --long devel/sage/sage/interfaces/jmoldata.py # 1 doctest fails
(it tries to run testjava.sh which comes from a patch to jmol)
sage -t --long devel/sage/sage/databases/cremona.py # 5 doctests fail
(because we don't seem to have the same database ordering or some such :-/)
sage -t --long devel/sage/sage/interfaces/genus2reduction.py # 1
doctest fails (exception raised)
notice that if you run this non-locally, you'll probably also see those:
sage -t --long devel/sage/sage/gsl/probability_distribution.pyx # 4
doctests failed (TclError: no display name and no $DISPLAY environment
variable [and its consequences] -- they all pass with a display!)
sage -t --long devel/sage/sage/plot/plot.py # 20 doctests fail
(TclError: no display name and no $DISPLAY environment variable [and its
consequences] -- they all pass with a display!)
All in all, I would say there are still a few things to package in
debian but as you can see things are taking shape: it's now possible to
build sage with most of its spkg replaced by system packages.
Those scripts will not help build a proper sage in debian, but at least
they allow us to see the problems in advance, and as you can see from
http://wiki.debian.org/DebianScience/Sage , I already reported quite a
few issues so hopefully dust will settle.
Snark on #debian-science
#!/usr/bin/python
# -*- coding: utf-8 -*-
import io,os,shutil,subprocess
from apt import *
from spkg_to_dpkg import spkg_deps
do_linking = True # if False, print the colors for the dependencies dot file
verbose = False # if True, print what is done
deb_host_multiarch = subprocess.check_output(['dpkg-architecture', '-qDEB_HOST_MULTIARCH'])[:-1]
# FIXME: I wrote code to find the version of the spkg&deb, but that
# code is not used to check we have the correct versions!
#
# It won't be trivial to code for at least three separate reasons :
#
# - each upstream can have its own numbering scheme ;
#
# - each spkg will have its ".p314" appended, when it's not a
# prepended "cropped"... or something else ;
#
# - each deb will also have modifiers like "-314build314"... or
# - something else.
#
# Wasn't it the beatles who sang "I need love" ?
sagedir = u'/home/jpuydt/sage-5.9.beta1'
cache = Cache() # that line can take long...
def build_spkg_versions():
result = dict()
for filename in os.listdir(os.path.join (sagedir, u'spkg', u'standard')):
if filename[-4:] == u'spkg': # use -4 here because there's u'deps'!
name_and_version = filename[:-5]
version = u'-'.join(name_and_version.split(u'-')[1:])
name = name_and_version.split(u'-')[0]
result[name]=version
return result
spkg_versions = build_spkg_versions()
def spkg_version(name):
return spkg_versions[name]
def dpkg_version(package):
return package.installed.version
def correct_version(s_version, d_version):
# FIXME
return True
def remove_file(name):
os.remove(name)
def create_directory(name):
try:
os.makedirs(name)
if verbose:
print (u'create_directory(%s)' % name)
except:
pass
def symlink_file(source, target):
try:
os.symlink(source, target)
if verbose:
print (u'symlink(%s, %s)' % (source, target))
except:
pass
def touch_file(name):
try:
open(name, 'a')
if verbose:
print(u'touch_file(%s)' % name)
except:
pass
def symlink_directory(source, target):
create_directory(target)
for name in os.listdir(source):
next_source=os.path.join(source, name)
next_target=os.path.join(target, name)
if os.path.isdir(next_source):
symlink_directory(next_source, next_target)
else:
symlink_file(next_source, next_target)
def replace_symlink_with_copy (filename):
real = os.path.realpath (filename)
remove_file(filename)
shutil.copy (real, filename)
def replace_pkgconfig_files_with_copies ():
path = os.path.join (sagedir, u'local', u'lib', u'pkgconfig')
for name in os.listdir(path):
filename = os.path.join (path, name)
if os.path.splitext(filename)[1] == '.pc':
replace_symlink_with_copy (filename)
def conway_polynomials_workarounds():
src = os.path.join (sagedir, u'local', u'share', u'sagemath', 'conway_polynomials')
dest = os.path.join (sagedir, u'local', u'share', 'conway_polynomials')
symlink_directory(src, dest)
def elliptic_curves_workarounds():
src = os.path.join (sagedir, u'local', u'share', u'sagemath', 'ellcurves')
dest = os.path.join (sagedir, u'local', u'share', 'ellcurves')
symlink_directory(src, dest)
src = os.path.join (sagedir, u'local', u'share', u'sagemath', 'cremona')
dest = os.path.join (sagedir, u'local', u'share', 'cremona')
symlink_directory(src, dest)
def graphs_workarounds():
src = os.path.join (sagedir, u'local', u'share', u'sagemath', 'graphs')
dest = os.path.join (sagedir, u'local', u'share', 'graphs')
symlink_directory(src, dest)
def palp_workarounds():
# we should probably patch sage to use what is in debian
directory = os.path.join(sagedir, u'local', u'bin')
for dim in [4, 5, 6, 11]:
for executable in [u'poly', u'class', u'cws', u'nef', u'mori']:
src=os.path.join(directory, u'%s.x' % executable)
dst=os.path.join(directory, u'%s-%dd.x' % (executable, dim))
symlink_file(src, dst)
def polybori_workarounds():
# This is debian bts' #702056
# also related:
# http://trac.sagemath.org/sage_trac/ticket/13989
#
directory = os.path.join(sagedir, u'local', u'lib', deb_host_multiarch)
symlink_file (os.path.join (directory, u'libpolybori-0.8.so'), os.path.join (directory, u'libpolybori.so'))
symlink_file (os.path.join (directory, u'libpolybori_groebner-0.8.so'), os.path.join (directory, u'libpolybori_groebner.so'))
directory = os.path.join (sagedir, u'local', u'share', u'polybori')
filename = os.path.join (directory, u'flags.conf')
create_directory (directory)
touch_file (filename)
filename = os.path.join (sagedir, u'local', u'include', u'polybori.h')
fle = open(filename, 'w')
fle.write('#include <polybori/polybori.h>')
fle.close()
def polytopes_workarounds():
src = os.path.join (sagedir, u'local', u'share', u'sagemath', 'reflexive_polytopes')
dest = os.path.join (sagedir, u'local', u'share', 'reflexive_polytopes')
symlink_directory(src, dest)
def python_workarounds():
directory = os.path.join (sagedir, u'local', u'bin')
symlink_file (os.path.join(directory, u'python2.7'), os.path.join(directory, u'python'))
directory = os.path.join (sagedir, u'local', u'lib')
symlink_file (os.path.join(directory, u'python2.7'), os.path.join(directory, u'python'))
directory = os.path.join (sagedir, u'local', u'lib', u'python2.7')
symlink_file (os.path.join(directory, u'dist-packages'), os.path.join(directory, u'site-packages'))
directory = os.path.join (sagedir, u'local', u'lib', u'python2.7')
symlink_directory (os.path.join(u'/usr', u'lib', u'python2.7', u'dist-packages'), os.path.join(directory, u'dist-packages'))
def setuptools_workarounds():
directory = os.path.join (sagedir, u'local', u'bin')
remove_file(os.path.join (directory, u'easy_install'))
remove_file(os.path.join (directory, u'easy_install-2.7'))
remove_file(os.path.join (directory, u'easy_install-2.6'))
workarounds = {
u'libpolybori-dev': polybori_workarounds,
u'palp': palp_workarounds,
u'python2.7': python_workarounds,
u'python-setuptools': setuptools_workarounds,
u'sagemath-database-conway-polynomials': conway_polynomials_workarounds,
u'sagemath-database-elliptic-curves': elliptic_curves_workarounds,
u'sagemath-database-graphs': graphs_workarounds,
u'sagemath-database-polytopes': polytopes_workarounds
}
def do_special_thing_if_needed (package):
if workarounds.has_key(package.name):
workarounds[package.name]()
print ('Did something special for %s' % package.name)
def prune_spkg(name, version):
filename = os.path.join (sagedir, u'spkg', u'installed', '%s-%s' % (name, version))
stream = io.open(filename, 'w')
stream.write(u'The SYSTEM package was used!\n')
stream.close()
def stow_dpkg(package):
if verbose:
print ('stow_dpkg(%s)' % package.name)
excludes = [u'', u'/.', u'/usr']
for name in package.installed_files:
if name not in excludes:
target_name = os.path.join(sagedir, u'local', os.path.relpath(name, u'/usr'))
if os.path.isdir(name):
create_directory (target_name)
else:
symlink_file (name, target_name)
do_special_thing_if_needed (package)
def loop_on_spkgs():
create_directory (os.path.join (sagedir, u'spkg', u'installed'))
directory = os.path.join (sagedir, u'local')
create_directory(directory)
symlink_file (directory, os.path.join(directory, u'local')) # that one is baffling!
for (spkg_name, dpkg_names) in spkg_deps.items():
prunable = True
for dpkg_name in dpkg_names:
try:
dpkg = cache[dpkg_name]
if dpkg.installed == None:
prunable = False
print ("Can't prune %s: missing package %s" % (spkg_name, dpkg_name))
except KeyError:
prunable = False
if dpkg_name == u'unsupported':
if do_linking:
print ("Can't prune %s: unsupported" % spkg_name)
else:
print (""" %s [color="red"];""" % spkg_name.upper())
else:
if do_linking:
print ("Can't prune %s: %s not found" % (spkg_name, dpkg_name))
else:
print (""" %s [color="orange"];""" % spkg_name.upper())
if not prunable:
break
if prunable:
for dpkg_name in dpkg_names:
dpkg = cache[dpkg_name]
stow_dpkg (dpkg)
if do_linking:
prune_spkg (spkg_name, spkg_version(spkg_name))
print ("Pruning %s" % spkg_name)
else:
print (""" %s [color="green"];""" % spkg_name.upper())
def main():
loop_on_spkgs()
replace_pkgconfig_files_with_copies()
if __name__ == '__main__':
main()
#!/usr/bin/python
# -*- coding: utf-8 -*-
# for each spkg name, a list of debian packages which will cover what
# it provides
#
# with an empty list if nothing is needed (!), and a u'unsupported' if
# we don't know yet what we need (or it doesn't exist... or we know
# we're really outdated)
spkg_deps = {
u'atlas' : [u'libatlas-dev', u'libatlas-base-dev', u'libatlas3gf-base', u'libatlas3-base'],
u'blas' : [u'libblas-dev', u'libblas3gf'],
u'boehm_gc': [u'libgc-dev', u'libgc1c2'],
u'boost_cropped' : [u'libboost-dev'],
u'cddlib': [u'unsupported', u'libcdd-dev', u'libcdd0d'], # it's in debian experimental but sage patches it too much and adds cdd_both_reps and cdd_both_reps_gmp executables
u'cephes' : [], # unneeded: the sage spkg only installs something on freeBSD
u'cliquer': [u'unsupported', u'libcliquer-dev', u'libcliquer1'], # debian bts #703263 and http://trac.sagemath.org/sage_trac/ticket/14349
u'conway_polynomials' : [u'sagemath-database-conway-polynomials'], # packaged in the debian-science git
u'cvxopt': [u'python-cvxopt'],
u'cython' : [u'cython'], # debian experimental
u'docutils' : [u'python-docutils'],
u'ecl' : [u'unsupported', u'ecl'], # if we use debian's package, then build the maxima spkg, it fails : they ask ecl where it's lib dir is... and it's not world-writable -- that will disappear when we'll have a ecl-enabled maxima
u'eclib' : [u'eclib-tools', 'libec0', 'libec-dev'],
u'ecm' : [u'libecm-dev', u'libecm0', u'gmp-ecm'],
u'elliptic_curves' : [u'sagemath-database-elliptic-curves'], # packaged in the debian science git
u'extcode' : [u'unsupported'], # should disappear with the git transition
u'fflas_ffpack' : [u'fflas-ffpack'],
u'flint': [u'libflint-1.52', u'libflint-dev'], # packaged in the debian science git
u'flintqs': [u'unsupported'],
u'freetype': [u'libfreetype6-dev', u'libfreetype6'],
u'gap' : [u'unsupported', u'gap', u'gap-core', u'gap-dev', u'gap-libs', u'gap-prim-groups', u'gap-small-groups', u'gap-trans-groups', u'gap-character-tables', u'gap-table-of-marks'], # debian has too old: #677668
u'gcc' : [u'gfortran'],
u'gd' : [u'libgd2-xpm-dev', u'libgd2-xpm'],
u'gdmodule' : [u'python-gd'],
u'genus2reduction' : [u'genus2reduction'],
u'gfan' : [u'gfan'], # debian experimental
u'givaro' : [u'libgivaro-dev', u'libgivaro1'],
u'glpk' : [u'python-glpk', u'libglpk-dev', u'libglpk0'],
u'graphs' : [u'sagemath-database-graphs'], # packaged in the debian science git
u'gsl': [u'libgsl0-dev', u'libgsl0ldbl'],
u'iconv' : [], # it's in the libc
u'iml' : [u'libiml-dev', u'libiml0'],
u'ipython': [u'ipython'],
u'jinja2' : [u'python-jinja2'],
u'jmol' : [u'jmol'],
u'lapack' : [u'liblapack-dev', u'liblapack3gf'],
u'lcalc': [u'unsupported', u'lcalc'], # debian doesn't ship L.h, which sage wants (debian bts #702901)
u'libfplll' : [u'unsupported', u'libfplll-dev', u'libfplll0'], # debian has latest, but sage is late http://trac.sagemath.org/sage_trac/ticket/12835
u'libgap' : [u'unsupported'],
u'libm4ri' : [u'libm4ri-0.0.20120613', u'libm4ri-dev'], # debian experimental
u'libm4rie' : [ u'libm4rie-0.0.20120613', u'libm4rie-dev'], # debian-science git
u'libpng' : [u'libpng12-dev', u'libpng12-0'],
u'linbox' : [u'liblinbox1', u'liblinbox-dev', u'liblinboxsage1', u'liblinboxsage-dev'], # debian-science git
u'lrcalc': [u'liblrcalc0', u'liblrcalc-dev', u'lrcalc'], # debian-science git
u'matplotlib' : [u'python-matplotlib'],
u'maxima' : [u'unsupported', u'maxima'], # needs to be compiled with ecl support (debian bts #702623)
u'mercurial' : [u'mercurial'],
u'mpc': [u'unsupported'],
u'mpfi' : [u'libmpfi-dev', u'libmpfi0'],
u'mpfr' : [u'libmpfr-dev', u'libmpfr4'],
u'mpir' : [u'unsupported'], # in fact, debian will try not to use it
u'mpmath' : [u'python-mpmath'],
u'networkx' : [u'python-networkx'],
u'ntl' : [u'libntl-dev', u'libntl0'],
u'numpy': [u'python-numpy'], # sage is late (http://trac.sagemath.org/sage_trac/ticket/11334), which makes doctests fail
u'palp' : [u'palp'], # debian-science git
u'pari' : [u'unsupported', u'libpari-dev', u'pari-gp'], # debian bts #703116 and http://trac.sagemath.org/sage_trac/ticket/14346
u'patch' : [u'patch'],
u'pexpect' : [u'python-pexpect'],
u'pil' : [u'python-imaging'],
u'polybori' : [u'python-polybori', u'libpolybori-dev', u'libpolybori-groebner-dev'], # debian experimental
u'polytopes_db' : [u'sagemath-database-polytopes'], # packaged in the debian-science git
u'ppl' : [u'libppl0.11-dev', u'libppl9'],
u'pycrypto' : [u'python-crypto'],
u'pygments' : [u'python-pygments'],
u'pynac' : [u'libpynac5', 'libpynac-dev'], # debian-science git
u'python' : [u'python2.7-dev', u'python2.7', u'python2.7-minimal', u'python-tk'],
u'r' : [u'r-base-dev', u'r-base-core', u'r-cran-mass', u'r-cran-lattice', u'r-cran-matrix', u'r-cran-nlme', u'r-cran-survival', u'r-cran-boot', u'r-cran-cluster', u'r-cran-codetools', u'r-cran-foreign', u'r-cran-kernsmooth', u'r-cran-rpart', u'r-cran-class', u'r-cran-nnet', u'r-cran-spatial', u'r-cran-mgcv'],
u'ratpoints' : [u'libratpoints0', u'libratpoints-dev'], # debian-science git
u'readline' : [u'libreadline6-dev', u'libreadline6'],
u'rpy2' : [u'python-rpy2'],
u'rubiks' : [u'unsupported'],
u'sage' : [u'unsupported'],
u'sagenb' : [u'unsupported', u'python-speaklater', u'flask-openid'], # python-speaklater and flask-openid are packaged in python-modules
u'sage_root' : [u'unsupported'],
u'sage_scripts' : [u'unsupported'],
u'sagetex' : [u'unsupported'],
u'scipy' : [u'python-scipy'],
u'scons' : [u'scons'],
u'setuptools' : [u'python-setuptools', u'python-pkg-resources'],
u'singular' : [u'unsupported', u'singular', u'libsingular-dev', u'libsingular-3-1-6'], # debian-science git, but sage is still on 3-1-5 and it's incompatible
u'sphinx' : [u'python-sphinx'],
u'sqlalchemy' : [u'python-sqlalchemy'],
u'sqlite' : [u'libsqlite3-dev', u'libsqlite3-0', u'sqlite3'],
u'symmetrica' : [u'libsymmetrica-dev', u'libsymmetrica-2.0'],
u'sympow' : [u'sympow'],
u'sympy' : [u'python-sympy'],
u'tachyon' : [u'tachyon'],
u'termcap' : [], # not needed
u'zlib' : [u'zlib1g-dev', u'zlib1g'],
u'zn_poly' : [u'libzn-poly-dev', u'libzn-poly-0.9'],
}
Reply to: