Re: Report from the DSA Team Sprint 2013-06
On Fri, Jun 21, 2013 at 2:07 PM, Peter Palfrader <weasel@debian.org> wrote:
> Comrades!
>
> We just finished a very productive Debian System Administration team
> sprint in LinuxHotel, Essen, Germany. All six of the current DSA
> members (Faidon, Luca, Martin, Peter, Stephen, Tollef) and our recruit
> (Hector) were present. This was the first time that all of us have
> met in person as whole team. We would like to thank the University of
> British Columbia (specifically Electrical & Computer Engineering and
> Information Technology) for their generous donation that more than
> offsets the cost this sprint, and LinuxHotel for hosting us at their
> open source rates.
>
> The primary goals of the meeting were (1) to review the previous
> year's action items, (2) to refresh the Five Year Plan for Debian's
> Infrastructure, (3) to work on mail routing and (4) discuss a plethora
> of other business.
Wow what an epic update!! Thanks a lot for all this work! (A couple
questions below.)
> Status report on items from last year's post[lists:dsa-oslo]:
> -------------------------------------------------------------
> o) Hosting & Virtualization:
> A very significant and very welcome contribution of physical (a
> rack-full of equipment) and virtual assets (co-location and
> bandwidth) from Bytemark has allowed us to accelerate some of our
> virtualization plans and, more importantly, handle our ongoing
> storage challenge.[www:bm-don]
>
> At this point, we consider bytemark, grnet, man-da and ubcece to be
> our primary data centers and we continue to make progress in
> migrating services from physical to virtual machines at these data
> centers using the 'ganeti' toolsuite. We recently migrated the
> majority of kvm based virtual machines to ganeti.
>
> Over the past year, we replaced equipment at man-da, primarily, and
> moved several core services on to virtual machines (eg. master,
> mail relays, the BTS).
>
> Over the next year, we plan to replace equipment at GRNET, and move
> the remaining services there and at other locations onto virtual
> machines where appropriate.
>
> o) Content Delivery Network:
> Last year we noted that several of the end-user facing services
> that Debian provides just consist of static data served from
> web-servers. Very often that only was a single machine and thus a
> single point of failure. We have worked on deploying a content
> delivery network for static data and are now serving
> mozilla.debian.net, planet.debian.org, www.debian.org,
> bits.debian.org, news.debian.net, backports.debian.org and
> ftp-master.metadata.debian.org off a set of machines all over the
> world. We seem to have reached the primary goal of providing
> machine and hosting redundancy. More CDN related ideas and
> experiments are mentioned further down (in this year's bullet
> points).
>
> o) Single-Sign-On:
> With the help of alioth admins, we now could, in theory,
> authenticate alioth users via Debian's SSO server, in addition to
> all the debian.org people in our LDAP. This will open
> opportunities for several web services to give even broader access.
> We are in progress of deploying this to some of our web based
> services. Stay tuned.
>
> o) Disaster Recovery:
> Over the last year we have deployed bacula and are starting to make
> full backups of more of our systems. We are still far away from
> having complete backups of everything but we're getting there. We
> discussed how to extend the backup space available at our primary
> backup storage host, beethoven. Since the system can still take a
> couple SATA disks we probably will look into purchasing these
> before we consider adding external storage.
>
> o) User and Group Management:
> Last year we estimated the number of active shell accounts to be on
> the order of 50.000 over all users/hosts. We still would like to
> disable unused accounts as described in last year's summary mail
> but nothing has happened to actually implement that. Help welcome.
What kind of skills are required to help with this?
> [www:bm-don] http://www.debian.org/News/2013/20130404
> [lists:dsa-oslo] https://lists.debian.org/debian-project/2012/03/msg00032.html
>
>
> A selection of even more things that we discussed:
> --------------------------------------------------
> o) CDNs redux:
> Debian currently has multiple content delivery networks. The most
> obvious one is the archive mirror network. Second, we have the
> 'static cdn' network described above which is used by
> mozilla.debian.net, planet, www and more. Third, we have the
> geolocation-aware security mirror network. During the sprint, we
> experimented using a third-party CDN for the security mirror
> network and for the Debian website to determine whether it could be
> a viable option for Debian. Specifically, we examined what
> integration challenges we might face should we desire to move in
> that direction.
>
> The experiment showed that we can use a CDN for the http side of
> the security network, but our DNS structure is giving us some
> problems. Most CDNs use a CNAME record to point users at the
> closest node using techniques such as anycast DNS or GeoDNS. Our
> challenge is that the DNS name security.debian.org is used for
> several different services: email, http and rsync being the most
> important ones (there is also ftp, but one might consider retiring
> that service, now). CNAME records and MX records for the same
> label are incompatible, so pointing security.debian.org directly at
> a CDN is not viable in our environment and we need to research a
> mechanism that satisfies our requirements; we are in discussion
> with CDN operators. Finally, rsync is normally not supported by
> CDNs and we will need to decide on a way forward there, as well.
>
> In the previous century, open source projects such as Debian were
> very reliant on the good-will of hosting providers (back then,
> primarily universities) to provide worldwide distribution of ISOs,
> etc. In 2013, we have many options available to us including
> third-party content delivery network operators. As previously
> mentioned, the breadth of infrastructure currently under active
> management is very broad. It is time to consider letting go of
> some things that others are now able to provide very capably so
> that we can focus on newer, novel initiatives.
>
> To that end, our preliminary inquiries indicate that multiple
> content delivery network operators have expressed interest in
> sponsoring this for us and, if we can find a good technical
> solution to the challenges we discovered, leveraging third-party
> CDN networks is something that we would like to explore further
> with affected teams.
>
> o) Configuration Management:
> The DSA team uses several tools to help them maintain and monitor
> Debian's infrastructure and to keep track of what systems there
> are, how to access them if things go wrong, when to purchase new
> warranty contracts and so on. Currently information is kept in a
> large number of different and not cross-referenced systems,
> including the puppet git repository, LDAP, the nagios
> configuration, our password database and a spreadsheet file.
>
> The distributed nature of this setup makes it difficult to get a
> good, consistent overview and to give other teams like the
> auditor/asset tracking folks the information they require to do
> their job.
>
> We agreed upon one desired solution: We would like to have one
> location, presumably a git repository, that has all the information
> we have about a VM or piece of hardware. Parts of that data would
> have to be encrypted to privileged information, but the majority of
> it should be available publicly. Systems like LDAP, our nagios and
> parts of the puppet configuration should then get the data they
> need from this new single source of truth. This is going to be a
> lot of work and it will probably take a long time to get there. If
> you would like to help please contact us.
I am involved with a project called "theforeman" which might be useful
as it integrates tightly with Puppet. http://theforeman.org/ It's
currently not part of Debian, but the project does build nightly debs,
and there is a general will to get forman into Debian. (PPAMAIN will
be a real big help here, since it's a fairly fast moving project and
having multiple version support within a single stable lifetime would
be strongly desired.)
Although it also can handle full provisioning of baremetal, VMs, and
cloud instances, I think the big win will be its puppet integration.
It provides an inventory service (central database of facts),
customizable metadata, puppet ENC facilities, support for multiple
Datacenters, RBAC, option LDAP integration for authentication, a rich
query/search interface, and robust puppet reporting capabilities. (The
Foreman team impressed the audience last PuppetConf when they
announced that they were the first ENC to support parameterized
classes.)
Another nice thing about Foreman is that it supports all versions of
Puppet going back to Puppet 0.24.4.
It's also very automatable since it has a REST API and CLI.
Bonus for me is that even though RedHat is now funding much of the
ongoing development, there is no single copyright holder, and it is
licensed GPLv3+. (So clearly DFSG compliant.)
I'm not sure if a tool like this fits into your plans for a "single
source of truth" since you mentioned git. But I strongly encourage you
to consider it, and would be happy to answer any questions you might
have, or discuss offline in IRC (bgupta@oftc).
> o) Points of Contact:
> There are four main points of contact for the debian admin team and
> their current guidelines:
> - dsa@debian.org: Anything that contains private or privileged
> information and should not be made public should go to this
> address. This new alias replaces the old debian-admin@debian.org
> alias, as the old one caused too much confusion in the past.
> - debian-admin@lists.debian.org: A mailing list that all members of
> the DSA team are on. There may be other people on that list as
> well, including local admins, other Debian Developers and
> interested third parties. We will make subscribing to this list
> possible for all DDs, though the technical details are yet to be
> worked out - we have a plan, but no code (RT#4501). Archives of
> new posts should be available on master, readable by the Debian
> group eventually (RT#4502).
> - request tracker: No changes here. If you have anything that
> can't or probably won't be dealt with immediately or a request
> where we want a paper trail, this is the place to make your
> request to. [wiki:rt] has some information on Debian's RT. We
> would like to deploy SSO authentication (with user management
> from LDAP?) on our RT instance at one point. Help welcome.
> - The #debian-admin IRC channel on OFTC. This is our primary work
> channel. It might be appropriate for a quick question. You're
> welcome to idle there, but please keep off-topic chatter to a
> minimum.
>
> While we are at it we would like to remind you of the
> debian-infrastructure-announce mailinglist[lists:dia]. Changes or
> updates that affect our infrastructure and services are posted
> there.
>
> [wiki:rt] http://wiki.debian.org/rt.debian.org
> [lists:dia] https://lists.debian.org/debian-infrastructure-announce/
>
> o) Team Meetings:
> We were, once again, positively surprised about how much work can
> get done during a face-to-face meeting. We thought we'd try to
> keep the momentum by scheduling monthly meeting on IRC. We have no
> idea how that will work out, but it's worth a shot.
>
> o) Monitoring:
> Monitoring and trending is a large part of how we keep track of
> things. As such it was problematic that munin on squeeze was not
> able to keep up with our infrastructure's growth. We discussed
> some alternatives but most of them require non-trivial amount of
> time investment to get them going. Also, munin2 as shipped with
> wheezy, while somewhat unusable for us in 7.0, has been fixed with
> 7.1 and generally seems to work. It's not perfect but we appear to
> get good support from upstream so we'll probably stick with it for
> now. [munin]
>
> [munin] https://dsa-guest@munin.debian.org/
>
> o) Archive Qualification:
> Concerns were raised about some of our porting and building
> infrastructure. We plan to contact d-port and d-release about this
> in the coming weeks. [RT#4503]
>
> o) Service Guidelines:
> We tried to come up with some guidelines on how services should be
> developed for and deployed on debian.org infrastructure. We will
> attempt to summarize in the near future. [RT#4504]
>
> o) Experimental VMs:
> We plan to set up a blade at our hosting at Bytemark with
> Openstack. Lucas raised the question of having a self-service
> provisioning service for virtual machines available to Debian
> Developers. Lucas, in a mail to the team, mused that the high bar
> of access to virtual machines was impeding development of tools
> that help the distribution, and we think that if this is indeed the
> case, lowering the bar would be a good thing. As it's not clear
> what the level of abandoned experiments will be, we are planning to
> initially make this a very disposable service - deletion of VMs
> after N amount of time and that sort of thing. Exact details are
> not set in stone.
>
> o) Hostnames:
> A bikeshedding discussion was had about naming of hosts. It's
> apparent that everybody in the team is overwhelmed by the sheer
> number of hosts we have to the point that remembering what a
> particular host does or what host a particular service runs on
> becomes impossible. Several ideas were floated but we did not come
> to an agreement on what color to paint this particular shed.
> Apparently mauve stripes on orange-green checkered background is
> not generally appreciated.
>
> o) Mail Infrastructure:
> A lot of work was invested in preparing for the eventual decoupling
> of @master.debian.org from @debian.org mail. The goal is to make
> @debian.org mail be handled by the front-end MXs so that there no
> longer is a single point of failure for most trivial mail
> forwarding. Expect a mail with more details soon. [RT#3720]
>
> o) Account Management Tools:
> Preparations for deploying new 'ud' (our account management
> software) have been made. The changes to the user-faced mail
> gateway still need to documented. Also ongoing task is the quality
> assurance of the generated account data to not break existing
> accounts.
>
> Cheers,
> your Debian System Administration Team
Reply to: