Re: when will we finally throw away binary uploads (Re: Please upgrade your build environment when you are affected by transition
On 2/14/14, Sam Hartman <firstname.lastname@example.org> wrote:
> All rants aside, I believe there's a fairly wide agreement that we
> should throw away binaries from builds.
I'd encourage something slightly different and then I'd expand on it a bit.
I think it would be useful to have an historical archive of each
binary, as uploaded by a developer or produced by a build server. In
the event that a binary is imperfect, storing a copy of each binary
may be useful for later analysis. This could help us spot bugs for a
compiler specific problem (such as optimizing something out of the
binary that was otherwise in the source) or when the binary is simply
malicious (such as a backdoor inserted by a compiler or build
I'd say that we might consider *using* that uploaded binary as an
assertion by a given developer. In general it is the expected correct
output binary and it is signed by that developer.
As we move into the deterministic/reproducible world, the archive will
serve an interesting purpose. Each developer would upload their
respective binary and signature per architecture for a given source
package. Their output may be represented as GnuPG signatures over
hashes of source code packages and binary packages. In theory, we
could have an assertion from another developer with another upload and
Even though we should have matching hashes, in some cases, we may not
learn that we have mismatching hashes for hours or days. Thus keeping
a copy of all the built binary parts is useful for later analysis.
This will allow us to verify that the sources correspond to a specific
binary output, we could also ensure that very important packages
require a threshold of signatures before the binaries are consider
Those that upload a binary should not be able to delete the uploaded
binary - thus when a build server or developer builds a *different*
version, we'll be able to inspect *all files* for differences. So for
each developer uploading, we'd keep a copy. We could discard identical
binaries and ensure that a copy is only kept in the long run when
there are no strange events.
If a developer is compromised, we're able to detect it and ensure that
the developer's system cannot be used to erase evidence of such a
compromise. If the build system is compromised, it will be useful to
download the binary from the archive and to inspect it. Similarly the
build system may not erase items from the archive.
This could add to the overall security of the process and to begin to
give us some semblance of a review process for binaries served to
Until something like that happens, I think we should probably not
*ship* the developer built binaries to users. We will however ship
some binary parts to the user and hopefully those will be built by the
build servers, as well as archived somewhere for all time, in case
we're already compromised.
( Append only data store ideas like Chisel may have a non-evil use, hooray! )
> I seem to recall ftp-master sending out mail to debian-devel-announce
> describing the steps along that process a while ago.
> I think it's fine to ask where that project is, and to volunteer
> resources to help.
> However, I'd hope we're all agreed that we need to get there.
I've been reading the deterministic build page and I think it could
use some further discussions.
All the best,