Report from DSA Team Sprint in Oslo
Comrades!
We just finished a very productive Debian System Administration team
sprint in Oslo, Norway. Five of the six current DSA members were
present. We would like to thank Varnish Software for hosting us and
providing food and drink. In addition, our thanks to the many donors
[0] whose contributions have permitted the project to subsidize
transportation and lodging.
The goals of the meeting were to develop a long term plan for Debian's
infrastructure, to review the current set of machines we maintain and
the services we provide, and to formulate some policies and procedures
regarding account and group management.
In addition, we took the opportunity to coordinate issues which have
been outstanding for a long time. This was the first time some of us
met in person.
[0]: http://www.debian.org/donations
Hardware & Sponsorships
=======================
Historically, Debian's hardware requirements have been met through the
generous donation of new and used equipment by individuals and
organizations. This is no longer true and the consequence of this is
that we must find alternate means by which to refresh our
infrastructure. This is becoming an acute issue as the majority of our
machines are now very old and long out of warranty; they are sometimes
quite constrained and/or are starting to fail.
We have developed a Five Year Plan [1] whereby all service-hosting
hardware should be under warranty at any given time. As part of this,
we are planning a five year refresh cycle and the goal is then for no
hardware to be more than five years old. Although our work on the Five
Year Plan has focussed on hardware hosting services such as ftp-master,
lists, wiki, etc., the Debian Project Leader has asked us to augment the
Plan to address buildd/porter hardware.
That said, an ongoing concern for DSA is our ability to source hardware
for our various architectures. We will work with the various teams to
identify commercial sources for hardware and to define the
supportability of the various architectures. The Debian System
Administration team is keen to ensure that we can maintain the Project's
ability to target our operating system for various architectures. This
means understanding and mitigating the risk presented by "un-sourceable"
architectures.
A clear outcome of our work on the Five Year Plan is an understanding
that hardware has now become one of the biggest expense categories for
Debian. If we are to be successful in implementing a five-year refresh
cycle, we will need improve our philanthropic support. We need to
revamp and probably merge our sponsorship and donations pages. We need
to support both targeted campaigns addressing specific goals (a
particular piece of hardware, say) as well as "annual giving". We need
to provide donors with visibility into how we have used funds donated to
date and with information regarding our future needs. We propose that
Debian requires something like the FreeBSD Foundation's sponsorship page
[2] and we are prepared to take a significant role in addressing this
requirement.
[1]: http://en.wikipedia.org/wiki/Five-Year_Plans_for_the_National_Economy_of_the_Soviet_Union
[2]: http://freebsdfoundation.org/donate/sponsors.shtml
Hosting & Virtualization
========================
In terms of hosting, Debian has been fortunate to have had the ability
to host equipment with many different partners. At times, we have had
hardware at more than 50 different locations! In recent years, we have
started to reduce the number of locations, concentrating our equipment
with a smaller set of hosting partners.
Virtualization has come of age and we want to move most services into
virtual machines, thus reducing the overall count of physical machines.
VMs are easier to move between physical hosts and give us service
separation without too much administrative or computational overhead.
Another advantage of virtualization is it makes it easier to increase
the availability of services. However, we recognize that there are
services that will continue to need dedicated hardware due to resource
requirements and, as such, we will not be able to move everything into
VMs.
Therefore, we will continue our consolidation efforts, focusing on 3-5
hosting locations. Some services have specific requirements as to where
they can or should be hosted and we will, of course, consider such
requirements when making any plans or decisions. For instance,
ftp-master currently needs to be hosted in the US.
As we further build out our VM infrastructure, we will attempt to
address requests for new 'machines' through the deployment of a virtual
machine rather than dedicated hardware.
Service Changes
===============
CDN: There are quite a few services which essentially provide static
content but which are currently either not mirrored or mirrored on their
own special mirror network. Examples include planet.d.o and www.d.o.
We would like to consolidate this into what is essentially a Debian
content delivery network [3] where we have one central host to which
services would submit their static content and from which it is then
replicated to a set of mirror nodes around the world. The main archive
is not intended to be part of this mirror network.
SSO: We have been working on deploying a web-based single-sign-on
solution. This will allow all users in the Debian LDAP to authenticate
to various web services using a single password. It is already used by
nm.debian.org and we are working with other teams to enable SSO for
their respective services. There are still some technical issues
outstanding as well as a lack of documentation. That said, if you run a
service which could benefit from this, please get in touch.
Authorization is currently out of scope for the SSO service and needs to
be provided be service in question. It is not out of the question to
extend this service to other authentication providers such as Alioth in
the future.
DR: We have worked with service owners to mitigate disaster risks by
redundantly deploying services across our geographically diverse hosting
locations. Nevertheless, there continue to be a number of non-redundant
services that represent single points of failure. While we can
virtualize many of these services, the ability to perform bare-metal
recovery of service-dedicated hardware or of VM-hosting hardware is
becoming a requirement. Critical portions (/etc and /srv, say) of
debian.org machines are backed up using a venerable tool (da-backup) but
it does not provide for bare-metal recovery. As a result, we have
decided to switch to bacula.
[3]: not to be confused with cdn.debian.net
User and Group Management
=========================
Debian has, approximately, 50 000 shell accounts [4]. We believe most
of these are unused and would therefore like to disable those that are.
The goal is to reduce the our exposure and not to take away anybody's
privileges. We will monitor shell account activity in order to identify
and disable shell accounts that have been unused for a significant
amount of time (months). We will extend ud-ldap to allow users to
easily and quickly re-enable their shell accounts.
Similarly, there is currently no mechanism which ensures that people
only have the group memberships which they are using. We would like to
implement a system which will require users to periodically confirm
their group memberships. Like the shell accounts, the goal is to reduce
our exposure, not to take away anybody's privileges.
[4] sum of accounts across all debian.org machines
Thanks for reading this far. We appreciate the opportunity to serve.
If you have any question, please don't hesitate to ask, either replying
to this mail on -project or contacting us at
debian-admin@lists.debian.org.
Regards,
Luca Filipozzi on behalf of the Debian System Administration Team
--
Luca Filipozzi, Member
Debian System Administration Team
Reply to: