Hi all, For the record, Christoph and I had the following discussion on IRC today. Paul <elbrus> Myon: I am starting to wonder, are all those PostgreSQL-11 based packages broken with the postgresql-common update, or just their autopkgtests <elbrus> if the latter is the case, I don't mind ignoring it all if you file RC bugs against them complaining about their autopkgtest being not robust against version bumps of PostgreSQL in postgresql-common <Myon> elbrus: it's just the tests, because the wrong set of PostgreSQL versions is tested <elbrus> please file the bugs and I'll ignore the failures <Myon> (except for pg-repack which is a real problem) <Myon> you want 20+ RC bugs? <elbrus> yes <Myon> which will fix themselves after 1 day when the packages transition? <elbrus> autopkgtests are clearly broken <elbrus> ehm, sorry, we are thinking differently -*- elbrus stares at https://sources.debian.org/src/hypopg/1.1.2-1/debian/tests/control/ <elbrus> that says postgresql-contrib-11 <Myon> the problem will disappear once pg-common is in testing <elbrus> so why can't autopkgtest (the program) not figure it out? <Myon> elbrus: ok that's really broken, but that's not the normal case <elbrus> ok, so I checked the wrong example :( <elbrus> can you please explain then why all those autopkgtest hint at regressions? <Myon> sorry I'm only starting to grasp the full picture, in the past this was no problem at all and the transition went without RT intervention -*- elbrus doesn't know how this PostgreSQL world is set-upped <Myon> the new autopkgtests (which I appreciate) really made this 2 orders of magnitude more complicated <elbrus> Myon: that everything went smooth in the past doesn't mean there are no (versioning) bugs <elbrus> autopkgtest is catching a lot of missing versioned dependencies <elbrus> which often aren't a big problem, but technically still missing <Myon> right <Myon> the "normal" case is Test-Depends: postgresql-server-dev-all, and the tests then iterate over the set of versions coded into postgresql-common <elbrus> so I think this just needs debugging and next time we'll have everything go smoothly again <Myon> this set of versions is wrong for the "other" version at the moment <elbrus> so what is missing? <elbrus> from unstable in testing for autopkgtest <Myon> the problem is that at the moment we have a static debian/tests/control which doesn't know about the list of versions <elbrus> because both postgresql-11 and -12 are in testing <Myon> it assums that the list of versions are in sync, and then everything works <Myon> no the list is hardcoded in postgresql-common <elbrus> I still don't see any issue <Myon> https://sources.debian.org/src/postgresql-common/209/debian/supported-versions/#L103 <elbrus> in your description <Myon> the testing postgresql-common wants to test all modules for PG11 <Myon> so the PG12 modules in unstable can't migrate because the testing test tries to test them for PG11 which fails <Myon> the unstable pg-common tests everything for PG12 so it can't migrate because modules in testing are still PG11 <Myon> so they need to migrate in parallel, but there's no dependency that enforces this <Myon> the packages themselves are fine, no matter what the pg-common version is <elbrus> which packages are so called modules (I can't see that from the name) <Myon> they just fail to test <Myon> everything building something postgresql-*-foo <elbrus> with * being 11 or 12? <Myon> which is basically everything we are talking about now, as the rest isn't version-dependant <Myon> elbrus: yes <elbrus> does postgresql-common do anything more than telling autopkgtest which version to test? <Myon> the rest is already done (mostly I think) <elbrus> I guess it tells which PostgreSQL is the default? <Myon> elbrus: it does that, and builds debian/control from debian/control.in using that list of versions <elbrus> you're not allowed to build debian/control during build, so you're meaning something else I hope <Myon> (fwiw, when I say "list of versions", that list has only one element in Debian, but can have more) <Myon> elbrus: it will FTBFS when the debian/control isn't up to date <elbrus> right <Myon> that's why we do all these no-op "souceful" uploads to update the list of packages built <Myon> "sourceful binnmu" uploads <elbrus> and get an new binary package in return <Myon> yes <elbrus> wouldn't it be possible to not embed the postgresql version in? <Myon> that new binary package works fine, but it can only be tested in an environment that has the right version config <elbrus> or you want all this to be co-installable while a new version is being deployed <Myon> we have two packages that do that (postgresql-q3c and pgsphere), but the problem there is, once a new PostgreSQL version is supported, existing users loose support for their version if they upgrade, and that's not nice <elbrus> so you would need an XS-Breaks-Autopkgtest field <elbrus> not even true <Myon> more Depends than Breaks <elbrus> so your modules should update the debian/tests/control file with the right version on postgresql-common <Myon> I put a change into the 2nd-last pg-common upload that makes the set of versions tested depend on the set of module packages actually installed, but that didn't quite work out because it needs additional stuff we don't provide in the necessary form yet <elbrus> it's a versioned test dependency as I get it? <Myon> elbrus: yes that would be one way out <elbrus> except that would mean that the actual test code is not allowed to change in that upload <elbrus> as britney doesn't inspect the test dependencies yet <elbrus> but at least the test for the module would <elbrus> pass <elbrus> and it can migrate to testing <elbrus> and then the -common package can migrate just after that <Myon> the detail that killed my idea is that for testing, we need slightly more than what the module dependencies dictate <elbrus> once it succesfully tests <Myon> in theory, postgresql-12-foo depends on postgresql-12, and that's all what's required at run time <elbrus> ack <elbrus> so if the -12- autopkgtest depends on the "12" -common package, it needs to say so <Myon> but to run the tests, we also need pg_config and pgxs.mk from PG12, but that's only in postgresql-server-dev-12 (which postgresql-server-dev-all depends on) <Myon> so we also need to move these two to postgresql-12 (or to a new micro package which postgresql-12 and postgresql-server-dev-12 pull in) <elbrus> is that already in testing? <Myon> that doesn't exist yet <elbrus> postgresql-server-dev-12 and postgresql-server-dev-all depending on it I mean <Myon> that brings us back to the #1 problem, postgresql-server-dev-all in unstable depends on 12, and on 11 in testing <Myon> "depend on the "12" -common package" isn't quite that easy <Myon> because we can't say ">= 210" or the like <elbrus> maybe I am wrong, but Python first adds a new version before removing the old version. Would that help in this case as well? <Myon> maybe I need to do something like postgresql-server-dev-all Provides: postgresql-test-interface-12 <elbrus> >= 208 <Myon> elbrus: the whole thing is meant to work in other environments as well where 208 might support other versions (9.4 .. 12 in apt.postgresql.org at the moment) <Myon> also, I hate to put >= 208 on all modules because the modules really need something lower only at runtime (and it would spoil the "re-upload with no changes idea") <Myon> elbrus: hypopg is actually an example of a package where it should already work because it explicitly test-depends on the correct PostgreSQL version <Myon> elbrus: the problem there is that upstream isn't ready for PG12 yet <Myon> so yes, hypopg needs an RC bug (which I'll file now) <elbrus> Myon: I meant in debian/tests/control <Myon> elbrus: ? <elbrus> [12:10:07] <Myon> also, I hate to put >= 208 on all modules because the modules really need something lower only at runtime (and it would spoil the "re-upload with no changes idea") <Myon> ah <elbrus> I understand from you that only autopkgtest is broken <elbrus> so that seems like *a* fix <Myon> right <Myon> well so far we got away with a static debian/tests/control for most packages, but the logic to generate it from debian/tests/control.in is there (and is used in a few packages) <Myon> ... yes that sounds like the correct point to attach that lever <elbrus> Myon: let me try to summarize <elbrus> 1) pg-repack/1.4.5-2 is broken <elbrus> 2) all other regressions in the postgresql-common transition are due to mismatch of postgresql-common and modules in testing during the autopkgtest runs? <elbrus> 3) packages that "need a new upstream version" work correctly with PG 11 as long that is in testing <elbrus> 3) PG 11 will remain in testing until reverse dependencies have upgraded or are removed <elbrus> s/3)/4)/ in the last line <elbrus> Myon: did I remove cstore-fdw and wal2json too early then (can I let them migrate again?) <Myon> elbrus: 1..4) full ack <Myon> cstore, wal2json: yeah you can probably let them back in, they will need a new version eventually that will hopefully Just Transition once pg-common in testing is at 12 <Myon> elbrus: fwiw, pg-repack needs a postgresql-12 upload which I was holding back because 12.1 is scheduled for next Thursday anyway <elbrus> Myon: also these haskell packages are falls positives in the ben match, right? -*- elbrus got hopelessly confused by those <Myon> I haven't yet looked, tbh <Myon> you mean the two at the bottom of https://release.debian.org/transitions/html/postgresql-12.html ? <Myon> plus the other at the top <elbrus> others, but yes <elbrus> everything with no cross or tick mark in amd64 <Myon> I have no idea, probably false positives from the ben regexps <Myon> the list different dependencies on https://packages.debian.org/sid/libghc-text-postgresql-prof is frightening <Myon> elbrus: are any RC bugs needed now? <elbrus> Myon: yes, I had that experience this morning as well <elbrus> please work on an improved scheme for autopkgtesting postgresql modules <Myon> yes <elbrus> no need for an RC bug now, but please don't forget <elbrus> for next time :) <Myon> I'll open that bug now -*- elbrus is triggering combinations <elbrus> Myon: the "fix" discussed above will not help for modules that don't have a new upstream version, correct? <elbrus> as they need testing with the old version of PG <Myon> a module is always compiled for a specific PostgreSQL version <elbrus> I understand that now <Myon> ideally the "fix" would need no changes in the module packages, so could be in the pg-common or server infrastructure only <Myon> so maybe we could it have work for the old packages as well <elbrus> so, e.g pglogical isn't updated yet <elbrus> testing with the new PG-common fails <Myon> pglogical (which has a few r-deps, fwiw) is a special beast :( <elbrus> bad example then <elbrus> pick random other module <Myon> upstream is trying to get money out of not timely shipping the open source variant <elbrus> ideally your/our solution for the whole situation is for the autopkgtest to do the right thing <Myon> of course, yes <elbrus> e.g. maybe implement the skipable restriction and exit 77 if the test detects that the PG version has been bumped? <elbrus> with respect to the version it's compiled for <Myon> oh, interesting <elbrus> that would work in *both* directions <Myon> but I think it should be possible to get this done completely properly <elbrus> o, sure <elbrus> but there <Myon> but maybe that would be an interesting thing to do now <elbrus> but there's more internal stuff involved that I am aware of (and then that I care about) <Myon> but I guess that needs updating all packages to add "skipable", in which case it cases more busywork :( <elbrus> for now, sure, more busywork <elbrus> so, I suggest to carefully lay out your options and pick the best one (for some value of best) <elbrus> Myon: by the way, can I fully quote our discussion here into the transition bug? <Myon> my preference is to make the existing package dependencies self-sufficient, e.g. make postgresql-11-foo's dependency on postgresql-11 pull in everything that is needed for testing <Myon> that is in most cases, pg_config, and the pgxs.mk snippet <Myon> I *think* that should be enough <Myon> the bad part being we will possibly only detect bugs during the next transition <Myon> elbrus: yes, please do so, will be helpful <Myon> sorry for the extra work, this was unexpected <elbrus> np, we're ironing out bugs <Myon> but it will lead to better quality, yes <elbrus> indeed <Myon> I just have to avoid making postgresql-XX pull in all of postgresql-server-dev-XX or the other way round <Myon> #944457 [zwiebelbot] Debian#944457: test-dependencies need to be explicit - https://bugs.debian.org/944457 <elbrus> ack <Myon> as an interim hack I could make postgresql-server-dev-all depend on -11 and -12, but that would only fix some "easy" packages <Myon> and only paper over the problem <Myon> elbrus: I think the most effective solution at this point would be to simply force postgresql-common into testing <Myon> this will take care of all packages that already have new versions, and the others need new versions anyway <elbrus> Myon: *force* has a special meaning in migration, I am checking if I can do with less <Myon> telling britney to test more packages in parallel would be good as well, but will probably not take care of everything (as not everything is ready) <Myon> but this is of course your take
Attachment:
signature.asc
Description: OpenPGP digital signature