Re: Age of built packages to be part of the jessie release?
On 25 November 2014 at 20:33, Manuel A. Fernandez Montecelo
<manuel.montezelo@gmail.com> wrote:
> 2014-11-25 11:14 Dimitri John Ledkov:
>>
>> On 24 November 2014 at 18:56, Manuel A. Fernandez Montecelo
>> <manuel.montezelo@gmail.com> wrote:
>>>
>>>
>>> Would it make sense to trigger rebuilds (or binNMUs, or actions to the
>>> same
>>> effect) for all packages in a given architecture when there are important
>>> changes in the toolchain, e.g. the default GCC bumps to a new major
>>> version
>>> [1]?
>>>
>>> For me it makes a lot of sense to have most of the archive rebuilt
>>> at that time, and these newly compiled packages migrated at some
>>> point to testing to be present in the next release.
>>>
>>> With the current method it seems that we only check if it FTBFS,
>>> and packages only get recompiled if there are new uploads or
>>> binNMUs. But actually recompiling and moving it to the archive
>>> will reveal changes in runtime behaviour, and thus bugs might be
>>> caught/reported/hopefully-fixed sooner. In some occasions there
>>> might even be some bugs fixed (if the compiler of years ago was
>>> buggy) or performance improvements.
>>>
>>> I do not know if there are significant drawbacks, for example if it
>>> would be very taxing to have everything built in a short period of
>>> time (especially for less powerful architectures) -- not only for
>>> hardware, but delays to other packages who are uploaded at the same
>>> time. Or if there are other fundamental problems that I did not
>>> consider.
>>>
>>> In summary, if there's a cheap (esp. in humanpower) way to achieve
>>> this, I think that it would be worth doing it.
>>
>>
>> Part of OBS (open build service) there is a tool called
>> "build-compare" which does end comparison to see if things are really
>> different, it does analyse disassembly to see if things are different.
>> E.g. I've been fiddling with a toolchain and e.g. enabling AVX by
>> default did show me all binaries that gained / changed their
>> instructions because of that and build were kept/publish as modified
>> instead of discarding them as "identical" to previous.
>>
>> Imho it would make sense to rebuild locally binaries which are the
>> same since stable, and compare generated maintainer scripts and/or run
>> the binaries through build-compare to see if there are worthwhile
>> differences. And if there are, request release team to binNMU those.
>> (Or do no change uploads, in case of changes in arch:all packages)
>
>
>
> I found these interesting links. First one a bit old but explaining
> OBS, build-compare and the openSUSE approach. Second one is a sample
> of people caring about reproducible builds, which is somewhat related.
>
> https://news.opensuse.org/2009/02/05/more-efficient-factory-development/
>
> https://blogs.kde.org/2013/06/19/really-source-code-software
>
>
>> From the first link, Debian seems to mostly follow "1) Trigger builds
>
> manually" (uploads or binNMUs), while openSUSE follows "2) Full
> Automated Build Triggering" to build packages whenever their
> dependencies change.
>
> It seems that we already have some infrastructure in place or which
> could be adapted, with jenkins.debian.net and the CI -- at least they
> detect if there are problems building packages with the new
> dependencies. openSUSE link explains that by getting rid of
> timestamps in built packages, they get more accurate results, effort
> that Lunar and Holger (and more people?) are doing as part of
> Reproducible Builds:
>
> https://wiki.debian.org/ReproducibleBuilds
>
> And about build-compare, the openSUSE article says what you point out,
> that "The openSUSE Build Service will just drop the build result if
> the new build are essentially the same and calculate the build
> dependencies based on the old build result". I think that it's
> accurate because it only checks if the binary is different, and the
> openSUSE folks (at least at that time) triggered builds in that case.
>
>
> Doing what you say, if I understood it correctly, would require
> significant human intervention, and if not that, at least a lot of
> bandwith, local storage and computing power, I think?
>
not really. filtering out and looking only that have not been rebuild
since stable in jessie will minimise amount of packages (e.g. all of
haskell, texlive, openoffice, kernels, etc have been rebuild). locally
rebuilding a package and comparing versus old one will need only
minimal local storage, or computing power (it doesn't matter how long
it will take to rebuild/compare) and once reports are published they
can be made public for others to analyse.
Note i'm not proposing to do the full OBS thing and rebuild / trigger
rebuilds like crazy. Just a one-off rebuild of old debian packages and
comparison using the build-compare tool.
> One possibility to minimise extra work would be to do that as part of
> the mass builds to detect FTBFS which happens ~twice a year, an extra
i had the impression that debian is rebuild more than twice a year,
given the cloud credits we had donated to us. It's simply not done in
the archive, but outside.
> analysis step of "this package does not FTBFS, but would benefit from
> being rebuilt with this toolchain versions". After the analysis, it
> would be a matter of gathering the list of packages in a single list,
> and ask Release Team (or similar) to schedule the builds. Or extract
> the list at regular intervals from jenkins or CI instance, if they can
> add this extra step of analysies.
>
>
> To me, what openSUSE guys do seems to be optimal, but I don't know how
> hard would be to integrate that in our existing processes and who
> would volunteer to do the work (I might help if there are more people
> involved, but cannot commit right now to new big efforts).
>
I don't believe it is optimal, but mostly waste of build power and
increases chances of missbuilts and propagation of such errors.
> On the other hand, triggering rebuilds by demand every now and then
> (e.g. when important bits of the toolchain changes, 2-4 times in
> between releases at most, so also ~twice a year), and setting the
> priority low when requesting packages being built again (so they don't
> make new uploads to "starve"), might perhaps be a very cheap solution,
> in terms of humanpower and changes needed in the infrastructure, and
> will get us 80% of the way there ("there" being: have all "arch:!all"
> packages built regularly and quite recently).
>
nah, we don't need to pointlessly rebuild things in the archive if we
don't know it will change/improve things. And it minimises bandwidth
for mirrors / updates.
--
Regards,
Dimitri.
Reply to: