[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Report from the DSA Team Sprint 2013-06


We just finished a very productive Debian System Administration team
sprint in LinuxHotel, Essen, Germany.  All six of the current DSA
members (Faidon, Luca, Martin, Peter, Stephen, Tollef) and our recruit
(Hector) were present.  This was the first time that all of us have
met in person as whole team.  We would like to thank the University of
British Columbia (specifically Electrical & Computer Engineering and
Information Technology) for their generous donation that more than
offsets the cost this sprint, and LinuxHotel for hosting us at their
open source rates.

The primary goals of the meeting were (1) to review the previous
year's action items, (2) to refresh the Five Year Plan for Debian's
Infrastructure, (3) to work on mail routing and (4) discuss a plethora
of other business.

Status report on items from last year's post[lists:dsa-oslo]:
o) Hosting & Virtualization:
   A very significant and very welcome contribution of physical (a
   rack-full of equipment) and virtual assets (co-location and
   bandwidth) from Bytemark has allowed us to accelerate some of our
   virtualization plans and, more importantly, handle our ongoing
   storage challenge.[www:bm-don]

   At this point, we consider bytemark, grnet, man-da and ubcece to be
   our primary data centers and we continue to make progress in
   migrating services from physical to virtual machines at these data
   centers using the 'ganeti' toolsuite. We recently migrated the
   majority of kvm based virtual machines to ganeti.

   Over the past year, we replaced equipment at man-da, primarily, and
   moved several core services on to virtual machines (eg. master,
   mail relays, the BTS).

   Over the next year, we plan to replace equipment at GRNET, and move
   the remaining services there and at other locations onto virtual
   machines where appropriate.

o) Content Delivery Network:
   Last year we noted that several of the end-user facing services
   that Debian provides just consist of static data served from
   web-servers.  Very often that only was a single machine and thus a
   single point of failure.  We have worked on deploying a content
   delivery network for static data and are now serving
   mozilla.debian.net, planet.debian.org, www.debian.org,
   bits.debian.org, news.debian.net, backports.debian.org and
   ftp-master.metadata.debian.org off a set of machines all over the
   world.  We seem to have reached the primary goal of providing
   machine and hosting redundancy.  More CDN related ideas and
   experiments are mentioned further down (in this year's bullet

o) Single-Sign-On:
   With the help of alioth admins, we now could, in theory,
   authenticate alioth users via Debian's SSO server, in addition to
   all the debian.org people in our LDAP.  This will open
   opportunities for several web services to give even broader access.
   We are in progress of deploying this to some of our web based
   services.  Stay tuned.

o) Disaster Recovery:
   Over the last year we have deployed bacula and are starting to make
   full backups of more of our systems.  We are still far away from
   having complete backups of everything but we're getting there.  We
   discussed how to extend the backup space available at our primary
   backup storage host, beethoven.  Since the system can still take a
   couple SATA disks we probably will look into purchasing these
   before we consider adding external storage.

o) User and Group Management:
   Last year we estimated the number of active shell accounts to be on
   the order of 50.000 over all users/hosts.  We still would like to
   disable unused accounts as described in last year's summary mail
   but nothing has happened to actually implement that.  Help welcome.

[www:bm-don] http://www.debian.org/News/2013/20130404
[lists:dsa-oslo] https://lists.debian.org/debian-project/2012/03/msg00032.html

A selection of even more things that we discussed:
o) CDNs redux:
   Debian currently has multiple content delivery networks.  The most
   obvious one is the archive mirror network.  Second, we have the
   'static cdn' network described above which is used by
   mozilla.debian.net, planet, www and more.  Third, we have the
   geolocation-aware security mirror network.  During the sprint, we
   experimented using a third-party CDN for the security mirror
   network and for the Debian website to determine whether it could be
   a viable option for Debian.  Specifically, we examined what
   integration challenges we might face should we desire to move in
   that direction.

   The experiment showed that we can use a CDN for the http side of
   the security network, but our DNS structure is giving us some
   problems.  Most CDNs use a CNAME record to point users at the
   closest node using techniques such as anycast DNS or GeoDNS.  Our
   challenge is that the DNS name security.debian.org is used for
   several different services: email, http and rsync being the most
   important ones (there is also ftp, but one might consider retiring
   that service, now).  CNAME records and MX records for the same
   label are incompatible, so pointing security.debian.org directly at
   a CDN is not viable in our environment and we need to research a
   mechanism that satisfies our requirements; we are in discussion
   with CDN operators.  Finally, rsync is normally not supported by
   CDNs and we will need to decide on a way forward there, as well.

   In the previous century, open source projects such as Debian were
   very reliant on the good-will of hosting providers (back then,
   primarily universities) to provide worldwide distribution of ISOs,
   etc.  In 2013, we have many options available to us including
   third-party content delivery network operators. As previously
   mentioned, the breadth of infrastructure currently under active
   management is very broad.  It is time to consider letting go of
   some things that others are now able to provide very capably so
   that we can focus on newer, novel initiatives.

   To that end, our preliminary inquiries indicate that multiple
   content delivery network operators have expressed interest in
   sponsoring this for us and, if we can find a good technical
   solution to the challenges we discovered, leveraging third-party
   CDN networks is something that we would like to explore further
   with affected teams.

o) Configuration Management:
   The DSA team uses several tools to help them maintain and monitor
   Debian's infrastructure and to keep track of what systems there
   are, how to access them if things go wrong, when to purchase new
   warranty contracts and so on.  Currently information is kept in a
   large number of different and not cross-referenced systems,
   including the puppet git repository, LDAP, the nagios
   configuration, our password database and a spreadsheet file.

   The distributed nature of this setup makes it difficult to get a
   good, consistent overview and to give other teams like the
   auditor/asset tracking folks the information they require to do
   their job.

   We agreed upon one desired solution:  We would like to have one
   location, presumably a git repository, that has all the information
   we have about a VM or piece of hardware.  Parts of that data would
   have to be encrypted to privileged information, but the majority of
   it should be available publicly.  Systems like LDAP, our nagios and
   parts of the puppet configuration should then get the data they
   need from this new single source of truth.  This is going to be a
   lot of work and it will probably take a long time to get there.  If
   you would like to help please contact us.

o) Points of Contact:
   There are four main points of contact for the debian admin team and
   their current guidelines:
   - dsa@debian.org: Anything that contains private or privileged
     information and should not be made public should go to this
     address.  This new alias replaces the old debian-admin@debian.org
     alias, as the old one caused too much confusion in the past.
   - debian-admin@lists.debian.org: A mailing list that all members of
     the DSA team are on.  There may be other people on that list as
     well, including local admins, other Debian Developers and
     interested third parties.  We will make subscribing to this list
     possible for all DDs, though the technical details are yet to be
     worked out - we have a plan, but no code (RT#4501).  Archives of
     new posts should be available on master, readable by the Debian
     group eventually (RT#4502).
   - request tracker: No changes here.  If you have anything that
     can't or probably won't be dealt with immediately or a request
     where we want a paper trail, this is the place to make your
     request to.  [wiki:rt] has some information on Debian's RT.  We
     would like to deploy SSO authentication (with user management
     from LDAP?) on our RT instance at one point.  Help welcome.
   - The #debian-admin IRC channel on OFTC.  This is our primary work
     channel.  It might be appropriate for a quick question.  You're
     welcome to idle there, but please keep off-topic chatter to a

   While we are at it we would like to remind you of the
   debian-infrastructure-announce mailinglist[lists:dia].  Changes or
   updates that affect our infrastructure and services are posted

   [wiki:rt] http://wiki.debian.org/rt.debian.org
   [lists:dia] https://lists.debian.org/debian-infrastructure-announce/

o) Team Meetings:
   We were, once again, positively surprised about how much work can
   get done during a face-to-face meeting.  We thought we'd try to
   keep the momentum by scheduling monthly meeting on IRC.  We have no
   idea how that will work out, but it's worth a shot.

o) Monitoring:
   Monitoring and trending is a large part of how we keep track of
   things.  As such it was problematic that munin on squeeze was not
   able to keep up with our infrastructure's growth.  We discussed
   some alternatives but most of them require non-trivial amount of
   time investment to get them going.  Also, munin2 as shipped with
   wheezy, while somewhat unusable for us in 7.0, has been fixed with
   7.1 and generally seems to work.  It's not perfect but we appear to
   get good support from upstream so we'll probably stick with it for
   now.  [munin]

   [munin] https://dsa-guest@munin.debian.org/

o) Archive Qualification:
   Concerns were raised about some of our porting and building
   infrastructure.  We plan to contact d-port and d-release about this
   in the coming weeks. [RT#4503]

o) Service Guidelines:
   We tried to come up with some guidelines on how services should be
   developed for and deployed on debian.org infrastructure.  We will
   attempt to summarize in the near future. [RT#4504]

o) Experimental VMs:
   We plan to set up a blade at our hosting at Bytemark with
   Openstack.  Lucas raised the question of having a self-service
   provisioning service for virtual machines available to Debian
   Developers.  Lucas, in a mail to the team, mused that the high bar
   of access to virtual machines was impeding development of tools
   that help the distribution, and we think that if this is indeed the
   case, lowering the bar would be a good thing.  As it's not clear
   what the level of abandoned experiments will be, we are planning to
   initially make this a very disposable service - deletion of VMs
   after N amount of time and that sort of thing.  Exact details are
   not set in stone.

o) Hostnames:
   A bikeshedding discussion was had about naming of hosts.  It's
   apparent that everybody in the team is overwhelmed by the sheer
   number of hosts we have to the point that remembering what a
   particular host does or what host a particular service runs on
   becomes impossible.  Several ideas were floated but we did not come
   to an agreement on what color to paint this particular shed.
   Apparently mauve stripes on orange-green checkered background is
   not generally appreciated.

o) Mail Infrastructure:
   A lot of work was invested in preparing for the eventual decoupling
   of @master.debian.org from @debian.org mail.  The goal is to make
   @debian.org mail be handled by the front-end MXs so that there no
   longer is a single point of failure for most trivial mail
   forwarding.  Expect a mail with more details soon. [RT#3720]

o) Account Management Tools:
   Preparations for deploying new 'ud' (our account management
   software) have been made. The changes to the user-faced mail
   gateway still need to documented. Also ongoing task is the quality
   assurance of the generated account data to not break existing

your Debian System Administration Team

Attachment: signature.asc
Description: Digital signature

Reply to: