Re: Urgent call to BioPerl users (Was: Bug#848236: src:gbrowse: Fails to build from source since bioperl upgrade has broken libbio-graphics-perl)
Hi all,
In the short term, I’m actually listed as co-maint on Bio-Graphics and BioPerl-Run. I can work on a new release for BioPerl-Run that address those specific issues (as well as other missing dependencies I found), but Bio-Graphics has some additional versioning problems with CPAN that need to be addressed (which may take more time). I have a test release out for it though, can you have a look? It seems to be picked up via cpanm now.
https://metacpan.org/release/Bio-Graphics
Now on to BioPerl-Run…
Re: BioPerl upgrades, this is a bit long-winded and maybe tl;dr, but the discussion of splitting out modules (specifically, ones that have become a maintenance or dependency burden) has been on-going for many years now, dating back to even before release 1.6. The core developers had long recognized the need to split out problematic code, as the size of the distribution has become problematic, particularly if short-term bug fixes need to be made available (primarily as the distribution release requires all tests passing, which with the ever-moving target of remote database changes, perl versions, etc can be challenging). We have addressed many of these issues (in particular only running network tests when requested, testing modules with ‘recommended’ dependencies only when those are installed, etc), but the long-term problems with a monolithic distribution remain.
Also, as 1.7.0 was the upcoming release, and since we had traditionally used odd-numbered releases somewhat like a well-tested but more malleable ‘developer’ release series, I decided to use this as a series where the code and API were expected to be stable but there may be changes regarding how modules are distributed (including whether we should keep them in the ‘core’ distribition, which bioperl-live arguably is considered to be). I should also point out, there has been prior precedent for this very seamlessly working w/o problems; Bio::Graphics in particular was once part of bioperl, and Bio::EUtilities, Bio::FeatureIO, Bio::Biblio (Thanks to Carnë), and others have all been released separately to CPAN.
However, that didn’t mean we expected this initial success would continue. We also knew that one of the key problems we have is how we should test for reverse dependency issues; I have a Github Issue on the need for a system to determine the fallout from this, but this hasn’t been put in place yet. Due to the urgency of issues Carnë mentioned re: NCBI HTTPS updates we pressed forward with a release. BioPerl 1.7.0 was the intended to be the initial test case where this was implemented on a somewhat more accelerated basis but focusing on modules that had very few rev deps, splitting out Bio::Coordinate (old and possibly redundant) and Bio::SearchIO::blastxml (XML dependencies). Bio::Tools::Run::WrapperBase was also moved to its namesake distribution (bioperl-run), which had long been planned. Beyond some initial problems with CPAN indexing of module versions (resulting in the release of 1.7.1) very few issues have been reported on Github nor on the mail lists.
I also suspected that, despite our best intents, we would not catch everything, and that turned out to be true (though it happened much later than anticipated). I did an initial search for rdeps with Bio::Coordinate modules and found nothing; as it turns out, that was b/c Bio::Graphics didn’t list it as a required module (it listed Bio::Root::Root and Bio::Root::Version, but not the direct dependency). This, as well as a missing CGI dependency, were fixed in the above CPAN release and so hopefully should be addressed. So, even having a system in place would not have worked in this case.
Finally, as Andreas noted, an interesting issue is that nothing had been reported until now, even though RT exists and we note our use of Github Issues (and we publicly mentioned the release back in Sept). My opinion is this is less due to lack of use of the main distribution; though it is unquestionably lower, I do still see questions about bioperl. I simply think fewer people are asking questions on the bioperl mail lists (they go to BioStar) or even taking time to report problems, and they don’t appear until a system like Debian’s (which checks for reverse dependencies) catches these issues. Also, a very common task is to ‘force-install’ bioperl, Bio-Graphics, and bioperl-run, which completely bypasses any tests. Not much we can do there.
Hope this clarifies things a little.
Thanks,
chris
On 12/16/16, 6:06 AM, "Andreas Tille" <andreas@an3as.eu> wrote:
Hi Carnë,
On Fri, Dec 16, 2016 at 11:38:02AM +0000, Carnë Draug wrote:
> >> The cause is most probably that libbio-graphics-perl does not work
> >> together with bioperl 1.7.1 (see also #848105). I'm now becoming aware
> >> that with the naive upgrade to bioperl 1.7.1 we have triggered problems
> >> that should have solved inside a transition via experimental. My
> >> proposed course of action is as follows:
> >>
> >> 1. Revert the version bump of bioperl and upload the old version
> >> with an epoch (1:1.6.924-6).
> >> 2. Upload 1.7.1-2 to experimental
> >> 3. Solve all issues with BioPerl 1.7.x after Stretch release.
> >>
> >> Does anybody have a better plan?
> >
> > Hi,
> >
> > this is a very *urgent* call to all BioPerl users to raise their
> > opinion. To verify what else might be broken by the new BioPerl I
> > tested its rdepends. I stumbled upon bioperl-run which is making
> > heavy use of
> >
> > /usr/share/perl5/Bio/Tools/Run/WrapperBase.pm
> >
> > which is also removed from latest BioPerl. So bioperl-run is broken as
> > well and even if we would manage to solve the issue with gbrowse I'm
> > afraid we are opening a can of worms when keeping bioperl 1.7.1. So
> > this is a serious and urgent call either for help or to confirm my plan
> > I mentioned below to revert bioperl to the old and tested version.
> >
> > If I do not hear anything I'll do the revert on Saturday or latest
> > Sunday.
>
> Bioperl is a massive collection of perl modules that grew unmaintainable.
> The proposed solution (upstream) for this was that bioperl would be split
> into smaller more manageble distributions.
I confirm that this is most probably a sensible plan by bioperl upstream.
> For example, there is now
> bio-biblio, bio-eutilities, and bio-coordinates which were previously part
> of bioperl itself. The core bioperl is now the bioperl-live distribution.
I guess we could document this in README.Debian since I personally do
not see a reason to rename the Debian package from bioperl to
bioperl-live. Do you think keeping the old name would be confusing for
insiders?
> This may cause issues for other packages in Debian because in Debian they
> are dependent on bioperl package instead of being dependent on a specific
> perl module.
Yes, obviously - that situation has triggered my urgent call. :-)
> Even more trouble because the new perl distributions are not
> yet packaged in Debian at all.
Without having checked I do not think that packaging these modules would
create severe problems.
> Another issue is that some of them are not
> even officially released.
>
> There is at least one very important fix on the new bioperl which fixes
> the connection to the NCBI servers (although the fix is simple enough
> that could be backported).
A backport of this fix would be helpful if we really decide to revert
the upgrade.
> Other reason in favour of upgrading is that
> bioperl-live is tricky to install and the main dependency of the other
> bioperl distributions. Any user wanting to install the other packages
> could probably handle their installation from CPAN but would be nice if
> Debian supplied bioperl-live as a foundation for that.
So we now have two possible options to proceede:
My original proposal:
1. Revert the version bump of bioperl and upload the old version
with an epoch (1:1.6.924-6).
2. Upload 1.7.1-2 to experimental
3. Solve all issues with BioPerl 1.7.x after Stretch release.
4. Backport 1.7.x to Stretch once issues are solved.
Keeping bioperl 1.7.1 and rather drop other packages:
1. Keep bioperl 1.7.1
2. Drop gbrowse and bioperl-run from Stretch
3. Check all rdepends of bioperl whether they work with 1.7.1:
$ apt-cache rdepends bioperl | grep "^ " | sort | uniq | grep -v -e alien-hunter -e libbio-perl-perl -e med-
arb
bioperl-run
gbrowse
iva
libbio-graphics-perl
libchado-perl
libgo-perl
libtfbs-perl
ncbi-epcr
predictprotein
roary
velvetoptimiser
What we definitely know is that bioperl-run, gbrowse and
libbio-graphics-perl are failing. I've just checked the
successfull build of alien-hunter and added a successfull
autopkgtest.
4. Package as many as the needed tools as we might be able to.
I admit I'm personally unbiased what option to choose. Based on
popcon it might make sense do keep BioPerl up to date and live
with the fact that other packages will not make it into Stretch
(and might be backported later). We should include in our
considerations that realistically only one week for action is
left (to get new packages in).
Any opinions what way to go?
Kind regards
Andreas.
PS: What I'm personally wondering: If we have so many BioPerl
users that are relying on up to date BioPerl why did they
wait until I as a non-BioPerl user by chance stumbled upon
the new version. Why is the culture of filing bug reports
"New version available" that badly established?
--
http://fam-tille.de
Reply to: