Re: packaging which supports multiple versions of an app on the same host
Hi Scott,
Let me share my view, which is probably shared by others at Debian-Med
and NeuroDebian project, where reproducibility questions come up
from time to time even though it diverges a bit from the original
question on versioned packages
So - if the goal is reproducible results over time, the only way to
achieve it 100% with versioned packages is to build them also without
dynamic linking to any of the 3rd-party libraries you might use, and
never rebuild again unless you could guarantee all the same versions of
those 3rd party software, thus boiling down to versioning EVERY package
in the system -- unacceptable burden imho...
if the goal is to make major still officially supported series of
software available simultaneously (e.g. what we do in pymvpa --
supporting stable 0.4 series, and pushing forward 2.0 series), then
versioned packages imho is a reasonable way to achieve that and you
could look at those examples we have already (e.g. pymvpa/pymvpa2,
fsl-4.1 etc). But for this we don't want to spit out versioned package
for every minor release.
Coming back to reproducible research -- various solutions already exist
superior imho to the burden of maintaining a universe of versioned
packages:
- run final analysis in a virtual machine, take a snapshot
- file system snapshot (where supported, e.g. btrfs?)
- snapshot.debian.org:
gather versions for all packages installed on the system (or
stracing and only recording those of relevance) and then regenerating
complete environment using snapshot.debian.org. could be even simpler
if your system is up to date for a given debian release -- just
remember the date and list of packages (and configuration settings if
they require any)
it would be cool if eventually we automated this -- e.g. enter the
shell which traces/records information on the packages which have been
used, and then provides a protocol which could be seeded later on to
reincarnate such environment
+ given the name and version of a software of interest, you can
already reincarnate such full environment with a single debootstrap
command pointing to snapshot.debian.org for the relevant date.
with help of schroot you could run those scripts inside chroot pretty
much as natively as running on the main system thus mimicking
"versioned packages"
- there also was a tool to create a virtual machine image from current
debian installation -- can't recall its name...
- yet another alternative -- cdepack
http://www.stanford.edu/~pgbovine/cde.html
which would wrap all related to execution of a specific
script/software into a contained directory which could be ran on any
other linux system (btw -- it is packaged, I just need final tune
ups/review and will upload into Debian).
probably there are even more solutions
Best regards,
Yaroslav
> Has Debian-Med encountered the need for this sort of thing from others from the scientific community?
> Do you have recommendations for how best to approach the problem in a way which will help us, and let us contribute back to you, most effectively?
> BACKGROUND:
> The Genome Institute is attempting to shift to using formal packaging for internal software distribution across our compute cluster. All of our blades run Ubuntu Lucid now, so we have an interest in getting Debian packages for all of the popular bioinformatics tools we use, or helping to make them. Our biggest difficulty is that our pipelines expect to produce reproducible results over time, and as such only call "versioned executables". We currently custom compile everything, and install each version next-to each other. The actual executable in the PATH is something like "cufflinks1.2.0", which prepends to the PATH where that code exists, and executes the "cufflinks" binary in that directory.
> In many cases the apps we are packaging are apps which you have already packaged, and we only need to change things for the per-version packaging. In other cases you have an older version of the code. In others, you have not packaged it at all.
> Our current trajectory is to hand-repackage whatever we depend on making the version number part of the package name, and formalizing the above shell-wrapper strategy. For our first few attempts we use etc-alternaives to manage a symlink with the generic app name. Upgrades will not happen as a user might expect, since each version is its own package, but we tentatively planned to make a meta-package with a name like "cufflinks-versioned" which depends on a regularly changing "cufflinksN.N.N" package, and would create the common user experience for anyone who did not request a specific version.
> Thanks in advance for your advice,
> Scott
--
=------------------------------------------------------------------=
Keep in touch www.onerussian.com
Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic
Reply to: