Bug#689606: unblock: mdp/3.3-1
Package: release.debian.org
Severity: normal
User: release.debian.org@packages.debian.org
Usertags: unblock
Please unblock package mdp
Dear Release Team,
mdp/3.3-1 fixes two FTBFS bugs -- #687408 and #689027 -- and enables
full use of python-sklearn 0.11.0 version now in wheezy -- only
versions up to 0.10 were supported until now -- see #689028.
All changes in the attached debdiff are related to the above
mentioned problems. The relatively long diff for the file CHANGES
can be safely ignored, most of those changes are already present in
mdp/3.2+git78-g7db3c50 currently in wheezy. A number of hunks
are whitespace-only changes introduced by the new python-mode in
emacs: upstream didn't feel like rebasing a public github repo just
to expunge them.
The package features a quite extensive unittests battery which is
run at build time. It passes successfully even across different
Debian and Ubuntu releases (tested on the NeuroDebian buildbots).
This gives me enough confidence to ask for the package to be
released with wheezy.
Please allow mdp/3.3-1 in testing.
Thanks,
Tiziano Zito
unblock mdp/3.3-1
-- System Information:
Debian Release: wheezy/sid
APT prefers unstable
APT policy: (500, 'unstable'), (101, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 3.2.0-3-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
diff -Nru mdp-3.2+git78-g7db3c50/CHANGES mdp-3.3/CHANGES
--- mdp-3.2+git78-g7db3c50/CHANGES 2012-04-05 16:01:52.000000000 +0200
+++ mdp-3.3/CHANGES 2012-09-28 14:48:39.000000000 +0200
@@ -1,3 +1,148 @@
+MDP-3.3:
+2012-09-19: FIX: fix error in automatic testing for MultinomialNB.
+2012-09-19: ERF: make sklearn nodes automatic testing more robust The previous
+ solution was actually ignoring the special definitions in NODES.
+
+2012-09-19: FIX: disable pp support if server can not start Being able to
+ import pp is not enough to be sure that pp works. For example in a
+ low memory situation, the following can happen:
+
+ >>> import pp
+ >>> server = pp.Server()
+ [...] OSError: [Errno 12] Cannot allocate memory
+
+ This fix just disables pp support if the server can not be
+ started.
+
+2012-06-18: FIX: Fix wrapping of sklearn 0.11 classifiers
+2012-04-17: FIX: make test_SFA2Node even more robust
+2012-04-16: FIX: make FastICANode test more robust
+2012-04-16: FIX: make test_SFA2Node more robust
+2012-04-05: FIX: fix pp_tests when run multiple times. pp tests were failing
+ when run twice in a row. hugly work-around, but it seems to
+ work...
+
+2012-04-05: FIX: fixed broken test_reload. test_reload was failing when called
+ twice in a row.
+
+2012-04-05: FIX: fix random seed tests. The tests were failing when called
+ twice in a row:
+ >>> import mdp
+ >>> mdp.test()
+ >>> mdp.test() the first call was working, the second one was
+ giving failures.
+
+2012-04-01: ERF: added tests for learning of bias parameters
+2012-03-26: FIX: replace third remaing test for pp_monkeypatch_dirname
+ Hopefully this will fix test suite failures.
+
+2012-03-22: FIX: Decrease the noise level in the DiscreteHopfieldClassifier.
+2012-03-22: FIX: honor MDP_DISABLE_SHOGUN env variable
+2012-03-19: FIX: fix left-over directories from testing pp. I do not know why,
+ but this simple change fixes the leftover directories problem when
+ testig with python-pp and pp monkey-patching. It should have
+ worked even as it was before, but apparently some race condition
+ happens.
+
+2012-03-06: FIX: fix determinant of random rotation matrix determinant sign
+ was wrong if dimensions of rotation matrix were odd. Thanks to
+ Philip DeBoer. Actual Fix.
+
+2012-03-06: FIX: fix determinant of random rotation matrix determinant sign
+ was wrong if dimensions of rotation matrix were odd. Thanks to
+ Philip DeBoer. Failing Test.
+
+2012-02-13: ENH: remove duplicated and overly verbose code
+2012-02-13: FIX: remove FastICA stabilization from tests
+2012-02-13: FIX: remove unused parameter stabilization in FastICA.
+2012-01-19: NEW: added new sklearn algorithms wrapping We now wrap 93 sklearn
+ algorithms!
+
+2012-01-19: FIX: fix another imcompatibility with sklearn 0.10 Although
+ EllipticEnvelop is derived from sklearn.base.ClassifierMixin, it
+ is not a classifier. It is actually a "predictor". Added a check
+ in the sklearn wrappers.
+
+2012-01-19: FIX: fix sklearn wrappers for version 0.10 New sklearn version
+ introduces classifiers and predictors mixin classes that do not
+ have a 'fit' method. Typical error:
+
+ AttributeError: type object 'OutlierDetectionMixin' has no
+ attribute 'fit'
+
+ Just added a check that the method is really present before
+ wrapping.
+
+2012-01-12: FIX: fix failing test for no eigenvalues left problem Check that
+ PCANode now raises the right exception.
+
+2012-01-12: FIX: add useful exception in case of no eigenvalues left. Check
+ for the condition explained in
+ b0810d72ce11925e1db6204c3a20bdfc77741a82 and raise a nicely
+ formatted exception:
+
+ Traceback:
+ [...] File ".../mdp/nodes/pca_nodes.py", line 223, in
+ _stop_training
+ ' var_abs=%e!'%self.var_abs) NodeException: No eigenvalues
+ larger than var_abs=1.000000e-15!
+
+2012-01-12: OTH: added failing test for no eigenvalues left problem. When
+ PCANode is set to use SVD and automatic dimensionality reduction
+ it may happen that after removing directions corresponding to
+ eigenvalues smaller than var_abs (1e-12 default), nothing is left.
+ This happens for example if the data is a matrix of (almost)
+ zeros. The error looks like this: Traceback (most recent call
+ last):
+ [...] File ".../mdp/nodes/pca_nodes.py", line 220, in
+ _stop_training
+ d = d[ d / d.max() > self.var_rel ] ValueError: zero-size array
+ to ufunc.reduce without identity
+
+2012-01-03: FIX: old joblib breaks imports from sklearn.decomposition
+ >>> import sklearn.decomposition Traceback (most recent call
+ last):
+ File "<stdin>", line 1, in <module>
+ File
+ "/usr/lib/pymodules/python2.6/sklearn/decomposition/__init__.py",
+ line 8, in <module>
+ from .sparse_pca import SparsePCA, MiniBatchSparsePCA
+ File
+ "/usr/lib/pymodules/python2.6/sklearn/decomposition/sparse_pca.py",
+ line 10, in <module>
+ from .dict_learning import dict_learning, dict_learning_online
+ File
+ "/usr/lib/pymodules/python2.6/sklearn/decomposition/dict_learning.py",
+ line 17, in <module>
+ from ..externals.joblib import Parallel, delayed, cpu_count
+ ImportError: cannot import name cpu_count
+ >>> joblib.__version__
+ '0.4.3'
+
+2012-01-02: FIX: py3k compatibility for reload reload() is moved to
+ imp.reload().
+
+ It also seems that Python 3 behaves slightly differently wrt.
+ reloads. For some reason, mdp.configuration is not imported
+ properly on reload. Let's just not remove the name from the
+ namespace, as this is the easiest fix.
+
+2011-12-23: ERF: added a failing test for reload MDP does not work with
+ reload. See https://github.com/mdp-toolkit/mdp-toolkit/issues/1
+ for details.
+
+ Thanks to Yaroslav Halchenko for reporting it.
+
+2011-12-18: FIX: Removed the leftover _global_message_emitter attribute (as
+ reported by Tiziano and Hannah).
+2011-12-09: FIX: fix wrong sorting of eigenvectors in degenerate case for SVD.
+ Thanks to Yaroslav Halchenko for reporting and fixing!
+
+2011-12-09: NEW: added failing test for wrong sorting of SVDs
+2011-10-24: FIX: updated sklearn wrappers to silence warnings for version >=
+ 0.9
+
+-------------------------------------------------------------------------------
MDP-3.2:
2011-10-24: FIX: do not complain when failing to remove temporary directory.
The error warning breaks doctests. On top of that, I do not think
diff -Nru mdp-3.2+git78-g7db3c50/COPYRIGHT mdp-3.3/COPYRIGHT
--- mdp-3.2+git78-g7db3c50/COPYRIGHT 2012-04-05 16:01:52.000000000 +0200
+++ mdp-3.3/COPYRIGHT 2012-09-28 14:48:39.000000000 +0200
@@ -1,7 +1,7 @@
This file is part of Modular toolkit for Data Processing (MDP).
All the code in this package is distributed under the following conditions:
-Copyright (c) 2003-2011, MDP Developers <mdp-toolkit-devel@lists.sourceforge.net>
+Copyright (c) 2003-2012, MDP Developers <mdp-toolkit-devel@lists.sourceforge.net>
All rights reserved.
diff -Nru mdp-3.2+git78-g7db3c50/debian/changelog mdp-3.3/debian/changelog
--- mdp-3.2+git78-g7db3c50/debian/changelog 2012-04-16 04:06:33.000000000 +0200
+++ mdp-3.3/debian/changelog 2012-09-30 09:09:35.000000000 +0200
@@ -1,3 +1,18 @@
+mdp (3.3-1) unstable; urgency=low
+
+ * New upstream bug-fix release
+ (list only changes not already in Debian)
+ - Make random failing SFA2Node tests more robust
+ - Make random failing FastICANode tests more robust
+ - Fix sklearn wrappers for version 0.11 (sid) and 0.12 (experimental)
+ (Closes: #689028)
+ - Disable pp support if pp server can not be started
+ (Closes: #689027)
+ * Fix seed in built-time testing of Debian package (Closes: #687408)
+ * Upload sponsored by Yaroslav Halchenko
+
+ -- Tiziano Zito <opossumnano@gmail.com> Wed, 26 Sep 2012 10:28:55 +0200
+
mdp (3.2+git78-g7db3c50-3) unstable; urgency=low
* New upstream git snapshot
diff -Nru mdp-3.2+git78-g7db3c50/debian/rules mdp-3.3/debian/rules
--- mdp-3.2+git78-g7db3c50/debian/rules 2012-04-14 14:19:53.000000000 +0200
+++ mdp-3.3/debian/rules 2012-09-28 17:26:53.000000000 +0200
@@ -1,6 +1,8 @@
#!/usr/bin/make -f
#DH_VERBOSE=1
+RSEED:=1029384756
+
PYTHON2:=$(shell pyversions -vr)
# Check if python3-numpy is available
@@ -32,7 +34,7 @@
ifeq (,$(filter nocheck,$(DEB_BUILD_OPTIONS)))
set -xe && for v in $(PYTHONALL); do \
LOC=$(INSTALLPATH)$(call py_libdir_sh, $$v); \
- python$$v $$LOC/mdp/test/run_tests.py $$LOC/mdp $$LOC/bimdp; \
+ python$$v $$LOC/mdp/test/run_tests.py --seed $(RSEED) $$LOC/mdp $$LOC/bimdp; \
done
: # Prune compiled code which could have been generated during testing
find \( -name __pycache__ -o -name \*.pyc -o -name \*.egg-info \) -delete
diff -Nru mdp-3.2+git78-g7db3c50/mdp/__init__.py mdp-3.3/mdp/__init__.py
--- mdp-3.2+git78-g7db3c50/mdp/__init__.py 2012-04-05 16:01:52.000000000 +0200
+++ mdp-3.3/mdp/__init__.py 2012-09-28 14:48:39.000000000 +0200
@@ -116,7 +116,7 @@
__version__ = '3.3'
__revision__ = configuration.get_git_revision()
__authors__ = 'MDP Developers'
-__copyright__ = '(c) 2003-2011 mdp-toolkit-devel@lists.sourceforge.net'
+__copyright__ = '(c) 2003-2012 mdp-toolkit-devel@lists.sourceforge.net'
__license__ = 'BSD License, see COPYRIGHT'
__contact__ = 'mdp-toolkit-users@lists.sourceforge.net'
__homepage__ = 'http://mdp-toolkit.sourceforge.net'
diff -Nru mdp-3.2+git78-g7db3c50/mdp/configuration.py mdp-3.3/mdp/configuration.py
--- mdp-3.2+git78-g7db3c50/mdp/configuration.py 2012-04-05 16:01:52.000000000 +0200
+++ mdp-3.3/mdp/configuration.py 2012-09-28 14:48:39.000000000 +0200
@@ -296,8 +296,8 @@
# read error from hijacked stdout
error = capture.getvalue()
- mdp._pp_needs_monkeypatching = 'ImportError' in error
-
+ mdp._pp_needs_monkeypatching = 'ImportError' in error
+
return mdp._pp_needs_monkeypatching
def set_configuration():
@@ -327,16 +327,27 @@
if os.getenv('MDP_DISABLE_PARALLEL_PYTHON'):
config.ExternalDepFailed('parallel_python', 'disabled')
else:
- if _pp_needs_monkeypatching():
- if os.getenv('MDP_DISABLE_MONKEYPATCH_PP'):
- config.ExternalDepFailed('parallel_python', pp.version +
- ' broken on Debian')
- else:
- config.ExternalDepFound('parallel_python', pp.version +
- '-monkey-patched')
- config.pp_monkeypatch_dirname = tempfile.gettempdir()
+ # even if we can import pp, starting the server may still fail
+ # for example with:
+ # OSError: [Errno 12] Cannot allocate memory
+ try:
+ server = pp.Server()
+ server.destroy()
+ except Exception, exc:
+ # no idea what exception the pp server may raise
+ # we need to catch all here...
+ config.ExternalDepFailed('parallel_python', exc)
else:
- config.ExternalDepFound('parallel_python', pp.version)
+ if _pp_needs_monkeypatching():
+ if os.getenv('MDP_DISABLE_MONKEYPATCH_PP'):
+ config.ExternalDepFailed('parallel_python', pp.version +
+ ' broken on Debian')
+ else:
+ config.ExternalDepFound('parallel_python', pp.version +
+ '-monkey-patched')
+ config.pp_monkeypatch_dirname = tempfile.gettempdir()
+ else:
+ config.ExternalDepFound('parallel_python', pp.version)
# shogun
try:
@@ -404,7 +415,7 @@
config.ExternalDepFailed('sklearn', exc)
except AttributeError, exc:
config.ExternalDepFailed('sklearn', exc)
- else:
+ else:
if os.getenv('MDP_DISABLE_SKLEARN'):
config.ExternalDepFailed('sklearn', 'disabled')
elif _version_too_old(version, (0,6)):
diff -Nru mdp-3.2+git78-g7db3c50/mdp/nodes/scikits_nodes.py mdp-3.3/mdp/nodes/scikits_nodes.py
--- mdp-3.2+git78-g7db3c50/mdp/nodes/scikits_nodes.py 2012-04-05 16:01:52.000000000 +0200
+++ mdp-3.3/mdp/nodes/scikits_nodes.py 2012-09-28 14:48:39.000000000 +0200
@@ -37,7 +37,7 @@
'feature_selection.rfe', 'feature_extraction.image',
'feature_extraction.text', 'pipelines', 'pls',
'gaussian_process', 'qda']
-else:
+elif _version_too_old(sklearn.__version__, (0, 11)):
# from release 0.9 cross_val becomes cross_validation and hmm is deprecated
scikits_modules = ['svm', 'linear_model', 'naive_bayes', 'neighbors',
'mixture', 'cluster', 'decomposition', 'lda',
@@ -46,6 +46,15 @@
'feature_extraction.text', 'pipelines', 'pls',
'gaussian_process', 'qda', 'ensemble', 'manifold',
'metrics', 'preprocessing', 'tree']
+else:
+ scikits_modules = ['svm', 'linear_model', 'naive_bayes', 'neighbors',
+ 'mixture', 'cluster', 'decomposition', 'lda',
+ 'covariance', 'cross_validation', 'grid_search',
+ 'feature_selection', 'feature_extraction',
+ 'pipeline', 'pls', 'gaussian_process', 'qda',
+ 'ensemble', 'manifold', 'metrics', 'preprocessing',
+ 'semi_supervised', 'tree', 'hmm']
+
for name in scikits_modules:
# not all modules may be available due to missing dependencies
@@ -440,11 +449,13 @@
"""NEED DOCSTRING."""
name = scikits_class.__name__
- if (name[:4] == 'Base' or name == 'LinearModel'):
+ if (name[:4] == 'Base' or name == 'LinearModel'
+ or name.startswith('EllipticEnvelop')
+ or name.startswith('ForestClassifier')):
return
if issubclass(scikits_class, sklearn.base.ClassifierMixin) and \
- hasattr(scikits_class, 'fit') and not hasattr(scikits_class, 'predict'):
+ hasattr(scikits_class, 'fit'):
nodes_list.append(wrap_scikits_classifier(scikits_class))
# Some (abstract) transformers do not implement fit.
elif hasattr(scikits_class, 'transform') and hasattr(scikits_class, 'fit'):
diff -Nru mdp-3.2+git78-g7db3c50/mdp/test/test_SFA2Node.py mdp-3.3/mdp/test/test_SFA2Node.py
--- mdp-3.2+git78-g7db3c50/mdp/test/test_SFA2Node.py 2012-04-05 16:01:52.000000000 +0200
+++ mdp-3.3/mdp/test/test_SFA2Node.py 2012-09-28 14:48:39.000000000 +0200
@@ -17,11 +17,11 @@
correlation = mult(des_mat[:-1,:].T,
numx.take(out[:-1,:], (0,2), axis=1))/(dim-2)
assert_array_almost_equal(abs(correlation),
- numx.eye(2), decimal-3)
+ numx.eye(2), 3)
for nr in xrange(sfa.output_dim):
qform = sfa.get_quadratic_form(nr)
outq = qform.apply(mat)
- assert_array_almost_equal(outq, out[:,nr], decimal)
+ assert_array_almost_equal(outq, out[:,nr], decimal-1)
sfa = mdp.nodes.SFANode(output_dim = 2)
sfa.train(mat)
@@ -29,7 +29,7 @@
assert out.shape[1]==2, 'Wrong output_dim'
correlation = mult(des_mat[:-1,:1].T,out[:-1,:1])/(dim-2)
assert_array_almost_equal(abs(correlation),
- numx.eye(1), decimal-3)
+ numx.eye(1), 3)
def test_range_argument():
node = mdp.nodes.SFA2Node()
diff -Nru mdp-3.2+git78-g7db3c50/mdp/test/test_fastica.py mdp-3.3/mdp/test/test_fastica.py
--- mdp-3.2+git78-g7db3c50/mdp/test/test_fastica.py 2012-04-05 16:01:52.000000000 +0200
+++ mdp-3.3/mdp/test/test_fastica.py 2012-09-28 14:48:39.000000000 +0200
@@ -47,8 +47,8 @@
else:
rand_func = uniform
- # try two times just to clear failures due to randomness
- for exc in (Exception, ()):
+ # try three times just to clear failures due to randomness
+ for exc in (Exception, Exception, ()):
try:
ica = mdp.nodes.FastICANode(limit=10**(-decimal),**parms)
ica2 = ica.copy()
diff -Nru mdp-3.2+git78-g7db3c50/mdp/test/test_nodes_generic.py mdp-3.3/mdp/test/test_nodes_generic.py
--- mdp-3.2+git78-g7db3c50/mdp/test/test_nodes_generic.py 2012-04-05 16:01:52.000000000 +0200
+++ mdp-3.3/mdp/test/test_nodes_generic.py 2012-09-28 14:48:39.000000000 +0200
@@ -38,10 +38,13 @@
for row in range(x.shape[0]):
yield x[numx.newaxis,row,:]
return _Iter()
-
+
def _contrib_get_random_mix():
return get_random_mix(type='d', mat_dim=(100, 3))[2]
+def _positive_get_random_mix():
+ return abs(get_random_mix()[2])
+
def _train_if_necessary(inp, node, sup_arg_gen):
if node.is_trainable():
while True:
@@ -99,7 +102,7 @@
`inp_arg_gen=...a call to get_random_mix('d')`
Used to construct the `inp` data argument used for training and
- execution. It can be an iterable.
+ execution. It can be an iterable.
`sup_arg_gen=None`
A function taking a single argument (`inp`)
@@ -198,7 +201,7 @@
# case 1: output dim set in the constructor
node = klass(output_dim=output_dim, *args)
_test(node)
-
+
# case 2: output_dim set explicitly
node = klass(*args)
node.output_dim = output_dim
@@ -221,7 +224,7 @@
node.output_dim = output_dim
py.test.raises(InconsistentDimException,
'node.input_dim = inp.shape[1]')
-
+
# check that output_dim is set to whatever the output dim is
node = klass(*args)
_train_if_necessary(inp, node, sup_arg_gen)
@@ -348,31 +351,23 @@
dict(klass='LibSVMClassifier',
sup_arg_gen=_rand_labels_array,
init_args=["LINEAR","C_SVC"]),
- dict(klass='NeighborsScikitsNode',
- sup_arg_gen=_rand_1d)
+ dict(klass='MultinomialNBScikitsLearnNode',
+ inp_arg_gen=_positive_get_random_mix,
+ sup_arg_gen=_rand_labels),
+ dict(klass='NeighborsScikitsLearnNode',
+ sup_arg_gen=_rand_1d),
]
-EXCLUDE_NODES = [nodes.ICANode]
+# LabelSpreadingScikitsLearnNode is broken in sklearn version 0.11
+# It works fine in version 0.12
+EXCLUDE_NODES = ['ICANode', 'LabelSpreadingScikitsLearnNode']
-if config.has_sklearn:
- # XXX
- # remove all non classifier nodes from the scikits nodes
- # they do not have a common API that would allow
- # automatic testing
- # XXX
- for node_name in mdp.nodes.__dict__:
- node = mdp.nodes.__dict__[node_name]
- if inspect.isclass(node) and node_name.endswith('ScikitsLearnNode'):
- if issubclass(node, ClassifierNode):
- NODES.append(dict(klass=node_name,
- sup_arg_gen=_rand_labels))
- else:
- EXCLUDE_NODES.append(node)
def generate_nodes_list(nodes_dicts):
nodes_list = []
# append nodes with additional arguments or supervised if they exist
visited = []
+ excluded = []
for dct in nodes_dicts:
klass = dct['klass']
if type(klass) is str:
@@ -381,8 +376,33 @@
# transform class name into class (needed by automatic tests)
klass = getattr(nodes, klass)
dct['klass'] = klass
- nodes_list.append(dct)
- visited.append(klass)
+ # only append to list if the node is present in MDP
+ # in case some of the nodes in NODES are optional
+ if hasattr(nodes, klass.__name__):
+ nodes_list.append(dct)
+ visited.append(klass)
+ for node_name in EXCLUDE_NODES:
+ if hasattr(nodes, node_name):
+ excluded.append(getattr(nodes, node_name))
+ # append sklearn nodes if supported
+ # XXX
+ # remove all non classifier nodes from the scikits nodes
+ # they do not have a common API that would allow
+ # automatic testing
+ # XXX
+ for node_name in mdp.nodes.__dict__:
+ node = mdp.nodes.__dict__[node_name]
+ if (inspect.isclass(node)
+ and node_name.endswith('ScikitsLearnNode')
+ and (node not in visited)
+ and (node not in excluded)):
+ if issubclass(node, ClassifierNode):
+ nodes_list.append(dict(klass=node,
+ sup_arg_gen=_rand_labels))
+ visited.append(node)
+ else:
+ excluded.append(node)
+
# append all other nodes in mdp.nodes
for attr in dir(nodes):
if attr[0] == '_':
@@ -391,7 +411,7 @@
if (inspect.isclass(attr)
and issubclass(attr, mdp.Node)
and attr not in visited
- and attr not in EXCLUDE_NODES):
+ and attr not in excluded):
nodes_list.append(attr)
return nodes_list
Reply to: