Re: Possibly moving Debian services to a CDN
]] Lucas Nussbaum
> Thanks a lot for this status update. I'm very much in favor of exploring
> ways to make the Debian infrastructure easier to manage, and using
> a CDN sounds like a great way to do so. It's great that things worked
> out with Fastly (any plans for a more public announcement?).
I'd like to move some other bits over and get some of the technical
hurdles worked out first. Amongst those are the problems of ensuring
that downstream caches see a consistent view, whether we're talking
about sites such as planet (where the plan is to pull in images to be
served from the planet.d.o domain rather than from the blog posters
domain) or backports. The considerations that apply to backports
would also apply if/when we want to push the main archive through.
In short, the problem is as follows:
We have a static-master with the master copy of the web site. This tree
gets synced to a bunch of mirrors (currently three). In some cases, a
mirror might be unreachable, be down or otherwise not be updateable. If
it's not updated, we want to ensure that mirror is not used until it's
up-to-date.
How we solve this problem is going to differ by CDN, but common to all
of them is we need tight control over timing to ensure users never
encounter out-of-sync mirrors. I have some ideas how to do this for
Varnish-based CDNs, but I'm not sure how to solve it for some of the
other CDNs, so we'll need to talk to them.
> However, in [1], I raised one main non-technical concern that is not
> mentioned in your mail: I fear that, by moving to CDNs without ensuring
> that there are a sufficient number of CDN providers willing and able to
> support Debian, we could end up in a lock-in situation with a specific
> CDN provider (after all, there are not so many of them, and even a
> smaller number could be able to deal with our technical requirements).
>
> [1] https://lists.debian.org/debian-project/2013/10/msg00074.html
You're just mirroring what we talked about in
https://lists.debian.org/debian-project/2013/10/msg00029.html there.
We're so far just reducing latency for users, those sites were served
purely by the static mirror hosts up until recently and we had no load
problems there, so if we want to, we could easily pull back to using our
own infrastructure again.
> Of course, as long as we have the infrastructure to go back to the old
> way of doing things, it is not a big problem. So I'm not worried at the
> moment. But one of the end goals of using CDN is to reduce the number of
> Debian PoP (have Debian machines in a fewer number of datacenters, to
> make them easier to manage). Once we do that, it will be very hard to go
> back.
We're not going to reduce the number of POPs significantly by using a
CDN, and it's not the goal either.
https://lists.debian.org/debian-project/2013/06/msg00164.html talks more
about the initial motivations for using a CDN.
> Have you been trying to reach out to other CDN providers about
> supporting Debian? I know of discussions with Amazon CloudFront, but I
> remember some technical blockers?
I'd like to get the services working well with one CDN before I start
engaging others.
> Could the DPL be of some help to you in that process?
I talked with James Bromberger at LCA, so contact is already established
there. We're also talking with at least one other CDN, so I don't think
we need any help in that area right now. We'll let you know.
--
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are
Reply to: