Re: ANTs & Data file license in http://slicer.kitware.com/midas3/
On Thu, 27 Feb 2014, brian avants wrote:
> i havent followed the whole discussion but agree that it would be nice to
> separate our testing data entirely from the source, as yarik suggests, at
> least for ANTs.
> i have been struggling a bit with testing lately, specifically, testing
> combinations of parameters on combinations of data. � is there any
> new/better approach to this?
do you mean like a good/fancy testDriver which would be given a
routine, data, expected output to compare?
indeed it would be nice to have some easy/standardized way. I am only
familiar somewhat with Torsten's cmtk way
http://anonscm.debian.org/gitweb/?p=pkg-exppsy/cmtk.git;a=blob;hb=HEAD;f=testing/apps/appsTestDriver.sh.in
where the driver is the actual collection of tests to be ran.
In Python world there is more or less now standardized way to run parametric
tests (the same test used on a set of input parameters), and in PyMVPA we still
use our home-brewed @sweepargs decorator [1] which allows to explore the
combinations of the parameters for the test
[1] http://github.com/PyMVPA/PyMVPA/blob/HEAD/mvpa2/testing/sweep.py
Also, although may be a bit early to relate, Michael Hanke initiated testkraut
https://testkraut.readthedocs.org/en/latest/ validation platform, which
although targeting fingerprinting (state/versions of libraries
etc)/validation of complete analysis pipelines, could be brought in for
parametric testing of specific tools. Its runtime fingerprinting feature could
actually become very beneficial for bisecting sporadic failures across
different environments and/or upgrades.
> � if i am going to put effort into this, then
> would like to improve the testing situation, in general, if possible.
I would also be interested to hear about a good approach for testing,
especially in neuroimaging projects relying on cmake. We are (very) slowly
working on introducing some public testing for AFNI so we could QA it at
build-time.
--
Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Senior Research Associate, Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik
Reply to: