[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Proposed New Incoming System



-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

Below are details of the proposed new incoming system.  This should
have been done a long time ago but in any event the ongoing move
towards crypto-in-main pushed it forward.  The code is 99% (re)written
and being tested locally.  As with the original pools implementation,
the plan is to implement it on non-US first and then ftp-master when
non-US is settled and working.

******** NOTE **********

!! DO NOT UPLOAD CRYPTO TO FTP-MASTER !!

This does NOT mean you can upload crypto yet.  This is just one step
in the crypto-in-main transition.  When the requisite legal hoops have
been jumped through, there will be an announcement but until then
current practice (and policy) applies, i.e. no crypto or crypto
dependents in main.

!! DO NOT UPLOAD CRYPTO TO FTP-MASTER !!

******** NOTE **********

- -------------------------------------------------------------------------------

                     Proposed New Incoming System
		     ============================

This document outlines the proposed new system for handling the
Incoming directories on ftp-master and non-US.

The present:
- ------------

  o incoming is a world writable directory

  o incoming is available to all through http://incoming.debian.org/

  o incoming is processed once a day by dinstall

  o uploads in incoming must have been there > 24 hours before they
    are REJECTed.  If they are processed before that and have problems
    they will be SKIPped (with no notification to the maintainer and/or
    uploader).

The proposed future:
- --------------------

  o There will be 4 incoming directories:

     @ "unchecked"  - where uploads from Queue Daemons and maintainers
		      initially go

     @ "install"    - where installable packages stay until the daily
                      dinstall run

     @ "new"	    - where NEW packages (and their dependents[1]) requiring
		      human processing go after being automatically
		      checked by dinstall.

     @ "byhand"	    - where BYHAND packages (and their dependents[1])
                      requiring human intervention go after being
                      automatically checked by dinstall.

    In addition there will be 3 support directories:

     @ "reject"	    - where rejected uploads go

     @ "done"	    - where the .changes files for packages that have been
		      installed go.

     @ "holding"    - a temporary working area for dinstall to hold
		      packages while checking them.

  o Packages in 'unchecked' are automatically checked every 15 minutes
    and are either: REJECT, ACCEPT (i.e. -> 'install'), NEW or BYHAND.

  o Only 'unchecked' is locally world-writeable.  The others are all,
    of course, locally world-readable but only 'install' and 'byhand'
    are publicly visible on http://incoming.debian.org/

  o 'install' and 'byhand' are made available to the auto-builders so
     they can build out of them.

  o 'install' is processed once a day as before.

  o list notification and bug closures are changed to be done for
    ACCEPTs, not INSTALLs. Mail is sent only to the maintainer/uploader
    on INSTALL.
    [Rationale: this reduces the load both on our list server and our
     BTS server; it also gives people better notice of uploads to
     avoid duplication of work especially, for example, in the case of NMUs]
    [NB: see [3] for clarifications of when ACCEPT/INSTALL mails are sent]

Why:
- ----

  o Security (no more replaceable file races)
  o Integrity (new http://i.d.o contains only signed (+installable) uploads[2])
  o Needed for crypto-in-main integration
  o Allows safe auto-building out of incoming
  o Allows previously-prohibitively-expensive checks to be added to dinstall
  o Much faster feedback on packages; no more 48 hour waits before
    finding out your package has been REJECTed.

What breaks:
- ------------

  o uploads of large packages directly to incoming over a slow link
    can lead to bogus rejections.

    * solution: Ensure the .changes file is the last file to be
                uploaded (dput and dupload already do this) or upload
                to a temporary directory then mv them into place with
                ssh.

  o people who upload packages but then want to retract or replace the
    upload.

    * solution: mostly "Don't do that then"; i.e. test your uploads
      properly.  Uploads can still be replaced, simply by uploading a
      higher versioned replacement.  Total retraction is harder but
      usually only relevant for NEW packages.

================================================================================

[1] For versions of dependents meaning: binaries compiled from the
    source of BYHAND or NEW uploads.  Due to katie's fascist
    source-must-exist checking, these binaries must be held back until
    the BYHAND/NEW uploads are processed.

[2] When this was initially written there was still at least one
    upload queue which will accept unsigned uploads from any
    source. [I've since discover it's been deactivated, but not, AFAIK
    because it allowed unsigned uploads.]

[3]
             --> reject
            /
           /
unchecked  ---------------------------[*]-------> install ------[+]------> pool
           \               ^   ^
            |             /   /
            |-->   new  --   /
            |       |[4]    /
            |       V      /
            |--> byhand --/

[*] is ACCEPT and when list notification and bug closure occurs
[+] is INSTALL and when maintainer/uploader notification occurs

[4] This is a corner case, included for completeness, ignore
    it. [Boring details: NEW trumps BYHAND, so it's possible for a
    upload with both BYHAND and NEW components to go from 'unchecked'
    -> 'new' -> 'byhand' -> 'install']

- -------------------------------------------------------------------------------

- -- 
James
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: Processed by Mailcrypt 3.5.6 <http://mailcrypt.sourceforge.net/>

iEYEARECAAYFAjxlsUgACgkQgD/uEicUG7A6YwCgoLNsIxw+pJAJMHt6+4DJZK5Q
dW8AnRHSqjc3RCvzCeS5mt6+evGrbhUc
=tvs+
-----END PGP SIGNATURE-----



Reply to: