Re: packaging which supports multiple versions of an app on the same host
I see. I'll take a look at pymvpa/pymvpa2. Thanks!
This is handled on Linux with libraries so nicely by default. Version numbers in the file names, symlinks where the version number loses resolution or is gone entirely, the ability to be precise about which version you want or to not care and get the latest.
So if we need to package something which already has Debian packaging, but just not in a multiversion way, we can make a fork and do that.
When we package something which is newer than is provided in Debian-Med, we'll start with the packaging for the older version and update it as necessary. Is there a good way to feed that back to this team?
And when the software is not packaged at all for Debian, and we make a package for it, is there a good avenue to get that to this team so they have less work to do, and could integrate it into your distribution?
On Feb 11, 2012, at 4:18 PM, Yaroslav Halchenko wrote:
> Hi Scott,
>
> Let me share my view, which is probably shared by others at Debian-Med
> and NeuroDebian project, where reproducibility questions come up
> from time to time even though it diverges a bit from the original
> question on versioned packages
>
> So - if the goal is reproducible results over time, the only way to
> achieve it 100% with versioned packages is to build them also without
> dynamic linking to any of the 3rd-party libraries you might use, and
> never rebuild again unless you could guarantee all the same versions of
> those 3rd party software, thus boiling down to versioning EVERY package
> in the system -- unacceptable burden imho...
>
> if the goal is to make major still officially supported series of
> software available simultaneously (e.g. what we do in pymvpa --
> supporting stable 0.4 series, and pushing forward 2.0 series), then
> versioned packages imho is a reasonable way to achieve that and you
> could look at those examples we have already (e.g. pymvpa/pymvpa2,
> fsl-4.1 etc). But for this we don't want to spit out versioned package
> for every minor release.
>
> Coming back to reproducible research -- various solutions already exist
> superior imho to the burden of maintaining a universe of versioned
> packages:
>
> - run final analysis in a virtual machine, take a snapshot
>
> - file system snapshot (where supported, e.g. btrfs?)
>
> - snapshot.debian.org:
> gather versions for all packages installed on the system (or
> stracing and only recording those of relevance) and then regenerating
> complete environment using snapshot.debian.org. could be even simpler
> if your system is up to date for a given debian release -- just
> remember the date and list of packages (and configuration settings if
> they require any)
>
> it would be cool if eventually we automated this -- e.g. enter the
> shell which traces/records information on the packages which have been
> used, and then provides a protocol which could be seeded later on to
> reincarnate such environment
>
> + given the name and version of a software of interest, you can
> already reincarnate such full environment with a single debootstrap
> command pointing to snapshot.debian.org for the relevant date.
>
> with help of schroot you could run those scripts inside chroot pretty
> much as natively as running on the main system thus mimicking
> "versioned packages"
>
> - there also was a tool to create a virtual machine image from current
> debian installation -- can't recall its name...
>
> - yet another alternative -- cdepack
> http://www.stanford.edu/~pgbovine/cde.html
> which would wrap all related to execution of a specific
> script/software into a contained directory which could be ran on any
> other linux system (btw -- it is packaged, I just need final tune
> ups/review and will upload into Debian).
>
> probably there are even more solutions
>
> Best regards,
> Yaroslav
>
>> Has Debian-Med encountered the need for this sort of thing from others from the scientific community?
>
>> Do you have recommendations for how best to approach the problem in a way which will help us, and let us contribute back to you, most effectively?
>
>
>> BACKGROUND:
>
>> The Genome Institute is attempting to shift to using formal packaging for internal software distribution across our compute cluster. All of our blades run Ubuntu Lucid now, so we have an interest in getting Debian packages for all of the popular bioinformatics tools we use, or helping to make them. Our biggest difficulty is that our pipelines expect to produce reproducible results over time, and as such only call "versioned executables". We currently custom compile everything, and install each version next-to each other. The actual executable in the PATH is something like "cufflinks1.2.0", which prepends to the PATH where that code exists, and executes the "cufflinks" binary in that directory.
>
>> In many cases the apps we are packaging are apps which you have already packaged, and we only need to change things for the per-version packaging. In other cases you have an older version of the code. In others, you have not packaged it at all.
>
>> Our current trajectory is to hand-repackage whatever we depend on making the version number part of the package name, and formalizing the above shell-wrapper strategy. For our first few attempts we use etc-alternaives to manage a symlink with the generic app name. Upgrades will not happen as a user might expect, since each version is its own package, but we tentatively planned to make a meta-package with a name like "cufflinks-versioned" which depends on a regularly changing "cufflinksN.N.N" package, and would create the common user experience for anyone who did not request a specific version.
>
>> Thanks in advance for your advice,
>> Scott
> --
> =------------------------------------------------------------------=
> Keep in touch www.onerussian.com
> Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic
Reply to: