Comrades!
We just finished a very productive Debian System Administration team
sprint in LinuxHotel, Essen, Germany. All six of the current DSA
members (Faidon, Luca, Martin, Peter, Stephen, Tollef) and our recruit
(Hector) were present. This was the first time that all of us have
met in person as whole team. We would like to thank the University of
British Columbia (specifically Electrical & Computer Engineering and
Information Technology) for their generous donation that more than
offsets the cost this sprint, and LinuxHotel for hosting us at their
open source rates.
The primary goals of the meeting were (1) to review the previous
year's action items, (2) to refresh the Five Year Plan for Debian's
Infrastructure, (3) to work on mail routing and (4) discuss a plethora
of other business.
Status report on items from last year's post[lists:dsa-oslo]:
-------------------------------------------------------------
o) Hosting & Virtualization:
A very significant and very welcome contribution of physical (a
rack-full of equipment) and virtual assets (co-location and
bandwidth) from Bytemark has allowed us to accelerate some of our
virtualization plans and, more importantly, handle our ongoing
storage challenge.[www:bm-don]
At this point, we consider bytemark, grnet, man-da and ubcece to be
our primary data centers and we continue to make progress in
migrating services from physical to virtual machines at these data
centers using the 'ganeti' toolsuite. We recently migrated the
majority of kvm based virtual machines to ganeti.
Over the past year, we replaced equipment at man-da, primarily, and
moved several core services on to virtual machines (eg. master,
mail relays, the BTS).
Over the next year, we plan to replace equipment at GRNET, and move
the remaining services there and at other locations onto virtual
machines where appropriate.
o) Content Delivery Network:
Last year we noted that several of the end-user facing services
that Debian provides just consist of static data served from
web-servers. Very often that only was a single machine and thus a
single point of failure. We have worked on deploying a content
delivery network for static data and are now serving
mozilla.debian.net, planet.debian.org, www.debian.org,
bits.debian.org, news.debian.net, backports.debian.org and
ftp-master.metadata.debian.org off a set of machines all over the
world. We seem to have reached the primary goal of providing
machine and hosting redundancy. More CDN related ideas and
experiments are mentioned further down (in this year's bullet
points).
o) Single-Sign-On:
With the help of alioth admins, we now could, in theory,
authenticate alioth users via Debian's SSO server, in addition to
all the debian.org people in our LDAP. This will open
opportunities for several web services to give even broader access.
We are in progress of deploying this to some of our web based
services. Stay tuned.
o) Disaster Recovery:
Over the last year we have deployed bacula and are starting to make
full backups of more of our systems. We are still far away from
having complete backups of everything but we're getting there. We
discussed how to extend the backup space available at our primary
backup storage host, beethoven. Since the system can still take a
couple SATA disks we probably will look into purchasing these
before we consider adding external storage.
o) User and Group Management:
Last year we estimated the number of active shell accounts to be on
the order of 50.000 over all users/hosts. We still would like to
disable unused accounts as described in last year's summary mail
but nothing has happened to actually implement that. Help welcome.
[www:bm-don] http://www.debian.org/News/2013/20130404
[lists:dsa-oslo] https://lists.debian.org/debian-project/2012/03/msg00032.html
A selection of even more things that we discussed:
--------------------------------------------------
o) CDNs redux:
Debian currently has multiple content delivery networks. The most
obvious one is the archive mirror network. Second, we have the
'static cdn' network described above which is used by
mozilla.debian.net, planet, www and more. Third, we have the
geolocation-aware security mirror network. During the sprint, we
experimented using a third-party CDN for the security mirror
network and for the Debian website to determine whether it could be
a viable option for Debian. Specifically, we examined what
integration challenges we might face should we desire to move in
that direction.
The experiment showed that we can use a CDN for the http side of
the security network, but our DNS structure is giving us some
problems. Most CDNs use a CNAME record to point users at the
closest node using techniques such as anycast DNS or GeoDNS. Our
challenge is that the DNS name security.debian.org is used for
several different services: email, http and rsync being the most
important ones (there is also ftp, but one might consider retiring
that service, now). CNAME records and MX records for the same
label are incompatible, so pointing security.debian.org directly at
a CDN is not viable in our environment and we need to research a
mechanism that satisfies our requirements; we are in discussion
with CDN operators. Finally, rsync is normally not supported by
CDNs and we will need to decide on a way forward there, as well.
In the previous century, open source projects such as Debian were
very reliant on the good-will of hosting providers (back then,
primarily universities) to provide worldwide distribution of ISOs,
etc. In 2013, we have many options available to us including
third-party content delivery network operators. As previously
mentioned, the breadth of infrastructure currently under active
management is very broad. It is time to consider letting go of
some things that others are now able to provide very capably so
that we can focus on newer, novel initiatives.
To that end, our preliminary inquiries indicate that multiple
content delivery network operators have expressed interest in
sponsoring this for us and, if we can find a good technical
solution to the challenges we discovered, leveraging third-party
CDN networks is something that we would like to explore further
with affected teams.
o) Configuration Management:
The DSA team uses several tools to help them maintain and monitor
Debian's infrastructure and to keep track of what systems there
are, how to access them if things go wrong, when to purchase new
warranty contracts and so on. Currently information is kept in a
large number of different and not cross-referenced systems,
including the puppet git repository, LDAP, the nagios
configuration, our password database and a spreadsheet file.
The distributed nature of this setup makes it difficult to get a
good, consistent overview and to give other teams like the
auditor/asset tracking folks the information they require to do
their job.
We agreed upon one desired solution: We would like to have one
location, presumably a git repository, that has all the information
we have about a VM or piece of hardware. Parts of that data would
have to be encrypted to privileged information, but the majority of
it should be available publicly. Systems like LDAP, our nagios and
parts of the puppet configuration should then get the data they
need from this new single source of truth. This is going to be a
lot of work and it will probably take a long time to get there. If
you would like to help please contact us.
o) Points of Contact:
There are four main points of contact for the debian admin team and
their current guidelines:
- dsa@debian.org: Anything that contains private or privileged
information and should not be made public should go to this
address. This new alias replaces the old debian-admin@debian.org
alias, as the old one caused too much confusion in the past.
- debian-admin@lists.debian.org: A mailing list that all members of
the DSA team are on. There may be other people on that list as
well, including local admins, other Debian Developers and
interested third parties. We will make subscribing to this list
possible for all DDs, though the technical details are yet to be
worked out - we have a plan, but no code (RT#4501). Archives of
new posts should be available on master, readable by the Debian
group eventually (RT#4502).
- request tracker: No changes here. If you have anything that
can't or probably won't be dealt with immediately or a request
where we want a paper trail, this is the place to make your
request to. [wiki:rt] has some information on Debian's RT. We
would like to deploy SSO authentication (with user management
from LDAP?) on our RT instance at one point. Help welcome.
- The #debian-admin IRC channel on OFTC. This is our primary work
channel. It might be appropriate for a quick question. You're
welcome to idle there, but please keep off-topic chatter to a
minimum.
While we are at it we would like to remind you of the
debian-infrastructure-announce mailinglist[lists:dia]. Changes or
updates that affect our infrastructure and services are posted
there.
[wiki:rt] http://wiki.debian.org/rt.debian.org
[lists:dia] https://lists.debian.org/debian-infrastructure-announce/
o) Team Meetings:
We were, once again, positively surprised about how much work can
get done during a face-to-face meeting. We thought we'd try to
keep the momentum by scheduling monthly meeting on IRC. We have no
idea how that will work out, but it's worth a shot.
o) Monitoring:
Monitoring and trending is a large part of how we keep track of
things. As such it was problematic that munin on squeeze was not
able to keep up with our infrastructure's growth. We discussed
some alternatives but most of them require non-trivial amount of
time investment to get them going. Also, munin2 as shipped with
wheezy, while somewhat unusable for us in 7.0, has been fixed with
7.1 and generally seems to work. It's not perfect but we appear to
get good support from upstream so we'll probably stick with it for
now. [munin]
[munin] https://dsa-guest@munin.debian.org/
o) Archive Qualification:
Concerns were raised about some of our porting and building
infrastructure. We plan to contact d-port and d-release about this
in the coming weeks. [RT#4503]
o) Service Guidelines:
We tried to come up with some guidelines on how services should be
developed for and deployed on debian.org infrastructure. We will
attempt to summarize in the near future. [RT#4504]
o) Experimental VMs:
We plan to set up a blade at our hosting at Bytemark with
Openstack. Lucas raised the question of having a self-service
provisioning service for virtual machines available to Debian
Developers. Lucas, in a mail to the team, mused that the high bar
of access to virtual machines was impeding development of tools
that help the distribution, and we think that if this is indeed the
case, lowering the bar would be a good thing. As it's not clear
what the level of abandoned experiments will be, we are planning to
initially make this a very disposable service - deletion of VMs
after N amount of time and that sort of thing. Exact details are
not set in stone.
o) Hostnames:
A bikeshedding discussion was had about naming of hosts. It's
apparent that everybody in the team is overwhelmed by the sheer
number of hosts we have to the point that remembering what a
particular host does or what host a particular service runs on
becomes impossible. Several ideas were floated but we did not come
to an agreement on what color to paint this particular shed.
Apparently mauve stripes on orange-green checkered background is
not generally appreciated.
o) Mail Infrastructure:
A lot of work was invested in preparing for the eventual decoupling
of @master.debian.org from @debian.org mail. The goal is to make
@debian.org mail be handled by the front-end MXs so that there no
longer is a single point of failure for most trivial mail
forwarding. Expect a mail with more details soon. [RT#3720]
o) Account Management Tools:
Preparations for deploying new 'ud' (our account management
software) have been made. The changes to the user-faced mail
gateway still need to documented. Also ongoing task is the quality
assurance of the generated account data to not break existing
accounts.
Cheers,
your Debian System Administration Team
Attachment:
signature.asc
Description: Digital signature