[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bits from the Debian Continuous Integration project

It's been almost 2 years since the Debian Continuous Integration project
has been launched, and it has proven to be a useful resource for the
development of Debian.


This is an update on the latest developments.

Infrastructure upgrade

Back in early 2014 when Debian CI was launched, there were less than 200
source packages with declared test suite metadata, and using a single
worker machine polling the archive for updates and running tests
sequentially in an infinite loop ("the simplest thing that could
possibly work") was OK-ish.

Then our community started an incredible, slow and persistent effort to
prepare source packages for automated testing, and we now have almost
5,000 of them. The original, over-simplistic design had to be replaced.


The effort of transforming debci in a distributed system was started by
Martin Pitt, who did an huge amount of work. In the latest months I was
able to complete that work, to a point where I am confident in letting
it run (mostly) unatended. We also had lots of contributions to the web
UI from Brandon Fairchild, who was a GSOC intern in 2014, and continues
to contribute to this date.

All this work culminated in the migration from a single-worker model to
a master/workers setup, currently with 10 worker nodes. On busy periods
all of those worker nodes will go on for days with full utilization, but
even then the turnaround between package upload and a test run is now _a
lot_ faster than it used to.

Debian members can inspect the resource usage on those systems, as well
as the length of the processing queue, by browsing to the corresponding
munin instance (requires authentication via a SSL client certificated
issued by sso.debian.org).


The system is currenly being hosted on a Amazon EC2 account sponsored by

The setup is fully automated and reproducible. It is not fully (or at
all) documented yet, but those interested should feel free to get in
touch on IRC (OFTC, #debci)


Testing backend changed from schroot to lxc

Together with the infrastructure updates, we also switched to using lxc
instead of schroot as backend. Most test suites should not be affected
by this, but the default lxc settings might cause some very specific
issues in a few packages. See for example #806542 ("liblinux-prctl-perl:
autopkgtest failures: seccomp, capbset")


Adding support for KVM is also in the plans, and we will get to that at
some point.

Learn more

If you want to learn more on how you can add tests for your package, a
good first start is the debci online documentation (which is also
available locally if you install `debci`).


You might also be interested in watching the live tutorial that has been
presented at Debconf 15 earlier this year, full of tips and real
examples from the archive.

  (469 MB)

  It would be _awesome_ if someone wanted to transcribe that into a text
  tutorial ;-)

How to get involved

There are a few ways you can contribute:

- autodep8

  if you are knowledgeable on a subset of packages that are very similar
  and can have their tests executed in a similar way, such as "$Language
  libraries", you might consider writing a test metadata generator so
  that each package does not need to declare a debian/tests/control file
  explicitly, requiring only The `Testsuite:` header in debian/control.


  Ruby and Perl are already covered, and there is initial support for
  NodeJS. Adding support for new types of packages is very easy.

  If you manage to add support for your favorite language, please get in
  touch so we can discuss whitelisting the relavant packages in
  ci.debian.net so that they will get their tests executed even before
  being uploaded with the proper `Testsuite:` control field.

- autopkgtest

  autopkgtest is an responsible for actually running your tests, and you can
  use it to reproduce test runs locally.


- debci

  debci is the system running in ci.debian.net (version 1.0, currently
  in testing, is _exactly_ what is running up there, minus a version
  number and a changelog entry).


  It can also be used to have private clones of ci.debian.net, e.g. for
  derivatives or internal Debian-related development. See for example:


Getting in touch

For maintainer queries and general discussion:

 mailing list: debian-qa@lists.debian.org
 IRC: #debian-qa on OFTC. Feel free to highlight `terceiro`

For the development of debci/autopkgtest/autodep8

 mailing list: autopkgtest-devel@lists.alioth.debian.org
 IRC: #debci on OFTC

Antonio Terceiro <terceiro@debian.org>

Attachment: signature.asc
Description: PGP signature

Reply to: