[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Summary of the Cloud Images BoF at DC16

[ Please note the cross-post and Reply-To ]

Hi folks,

As promised, here's a quick summary of what was discussed at the Cloud
Images BoF session in Cape Town.

Thanks to the awesome efforts of our video team, the session is
already online [1]. I've taken a copy of the Gobby notes too,
alongside my small set of slides for the session. [2]

We spoke about the state of cloud images in Debian, and how we'd like
to progress with creating *official* Debian images.

Stuff we're currently doing

   * Debian images in many clouds with lots of variation
   * Major cloud platforms have custom Debian images
   * Lots of variation, no “official” images

Stuff we should be doing

 * We want to have official images produced by Debian:
   + for better support
   + because users cannot necessarily trust the 3rd party additions
   + to protect the Debian “name”
   + because they are a trusted starting point
     - for users who do not care to build their own custom image (yet?), and
     - for users to base a custom image on.

 * We absolutely do *not* want to prevent people making and using the
   custom images. Debian is Free Software, and we *want* people to do
   whatever they want and need with Debian.

Official image requirements

 * For an image to be *official*, it must be produced by Debian people
   on Debian machines. That should be possible for all the cloud
   platforms, we believe.

 * The contents of an official image are important. Clearly, it must
   contain things only from Debian main. If drivers or platform
   packages are not in main, that is a problem. Virtualbox extensions
   could be a problem here. Other images could be described as "based
   on" Debian, maybe, but that doesn't mean much. Are we happy for
   people to describe their images as "Debian", but not "official

 * Is there a way to check an image loaded remotely, after
   installation? That depends on the cloud platforms - each has
   different features in this area. We'd like for users to be able to
   validate that what they have *is* a Debian image.

 * What image(s) should we provide?

   Thinking of two main options thus far:
   + A simple stable image
   + An image with some backports included (backported kernel,
     backported cloud platform tools). Maybe called "Jessie Refresh"
     or similar? It's common for cloud platform folks to want updates
     for a faster development cycle.
   Each has its pros and cons. The most important thing is that
   whatever image(s) we produce, we must label clearly so users
   understand what they're getting. Could look at these as similar to
   Blends, maybe.

 * cloud-init:
   + Should we include cloud-init? (A package that does the basic
     setup during image boot, to configure your image correctly for
     the cloud platform you're running on). Most people want this, but
     it seems not everybody. Official images should probably go this
   + Does any cloud platform require contrib or non-free bits for
     cloud-init? Most not, but apparently at least Rackspace needs a
     specific agent which is not free and it's difficult to even get
     hold of a copy. There are ways to do things better here, they're
     just not using those methods.

 * It would be nice to not have to provide *lots* of different images,
   one per cloud platform - that's painful. However, there are some
   issues with that plan. Apparently cloud-init can be painfully slow
   if things are not configured specifically to use the right data
   sources for a specific platform. e.g. enabling the EC2-style data
   source (as wanted for AWS images) will cause a 90s timeout for
   users on any other platforms. :-( Need more investigation?

 * Question: could we use d-i preseeding here? We're not using d-i,
   we're building and running complete images so that's not helpful.

 * Security updates:
   + Images need to be regenerated whenever any of the included
     packages are updated. We're not doing that yet, but should be.
     Should images be configured to automatically do a full upgrade on
     first boot? Could be controlled by cloud-init - different people
     have different requirements here.

   + Should unattended-upgrades be installed in official images by
     default? Makes sense for the platform providers, but lots of
     users may not want this - e.g. if you have a long-running cloud
     instance running infrastructure like a database server, you're
     not going to want that to be restarted without careful

     This is a wider question anyway - we should be thinking
     about this for all Debian installations, maybe. But it's a
     question that cloud people want to try and solve now... Another
     issue - if you configure your image to do updates at a specific
     time, what happens when 1,000+ machines in a large cloud all try
     to upgrade simultaneously?

 * QA
   + Official images need good QA
   + Need automation so we can validate our images sensibly. Tt the
     moment we don't have much testing going on at all, and that's

 * Plans
   + Suggested a timeline for building official images, etc. That's
     not been followed at all so far. We want to get some real images
     building and testing, but we've not made any progress so far.

Proposed Sprint

Zach Marano from Google has offered to host a cloud sprint in Seattle
later in the year, to help solve some of our issues. Seattle makes
sense as a location - the three biggest cloud platforms all have their
major development teams based there. We want everybody with an
interest in this area to be able to take part in the discussions and
work sessions - we want to get people working together to improve
things, particularly the official Debian images. More details coming
soon on the cloud list.


 * For *unofficial* images, for them to be called Debian, what should
   we demand of people? Document the changes that have been made, at
   the very least. We want to protect our users, and also Debian's
 * Once we have standard official images, it should become clearer.
 * The best people are already doing the right things here.
 * Suggestion that people should use bootstrap-vz and make the
   manifest available, as an example. People should be saying what
   tooling they used, and what config -> reproducible, trustable
   images. Maybe a single git tree showing all modifications, but
   that's difficult unless everybody's using the same tools.


 * The plethora of tools here does not serve users well.
 * Riku's session at Debconf15 counted image generating tools and got
   into double digits. Everybody starts off scratching their own itch,
   solving a simple problem. Over time we need better answers and good
   flexible tools to make things work better.
 * Maybe in time we'll only bless images as official if they're using
   the "right" tool. Don't want to have the argument *now* about which
   tools are best, but this is definitely something that should be
   discussed at the sprint. A good honest comparison of the features
   of the various tools would be awesome!

Near future

Come up with a minimal policy of what an image needs to look like, for
 * measure existing images against that policy to call these
 * ensure the tool can add the preferred packages for custom images.
 * create a trusted place from where to start.

What packages should be in our cloud images?

There's not been very much discussion about this so far on the cloud
list, and we should fix this. Lots of variation possible:

 * screen vs tmux and other alternative packages
 * There's been some worry that the vagrant images are including too
   much, e.g. Gnome and KDE libraries being pulled in due to broken
   dependencies (pinentry in particular)
 * Most people want a *minimal* installation. Tiny images are
   preferred with nothing installed extra by default. Let tools like
   ansible or puppet add the extra pieces where they're wanted. Every
   extra 1% multiplies up with each new install. Some people are
   paying per MB here.
 * If some folks want some less-minimal images, they should build
   their own images with their own selection of nifty tools. There's a
   lot of personal choice here and a lot of people won't agree on the
   "small list" of extras that should be included.
 * It's not just about image size. Look at the security footprint -
   complexity becomes problematic. Cloning and stacking images is a
   common task that the cloud providers tend to make easy.
 * DSA runs cloud images too on a Ganeti installation too. Should they
   be "official" images? Up to them. :-)
 * Suggestion that the minimal image should just be enough to run apt,
   plus *maybe* openssh-server. ssh is maybe *not* entirely required
   (could be pulled in via cloud-init, or alternatively removed) but
   most people are happy to add it, it seems.

Misc discussion points

 * If we go with a single image, the cloud providers may need to be
   able to pull in extra packages on demand if it helps the user.
 * Should an image look like a d-i installed system? e.g. should it
   use grub rather than syslinux? There have been problems with
   syslinux, apparently. There's a good point that we might want to
   provide as close as possible to a "standard" Debian for cloud
   images here to not surprise users. But should that include nanp
   etc.? Would we consider an image official if it used sysvinit
   instead of systemd?
 * The policy for "official" could/should not lock down what we
   *choose* to produce, necessarily. Let's not be too prescriptive
 * Make sure that our tooling is easy for users to use to create their
   own tweaked images. There's a difference in usage here. Many people
   will expect to create their own images with their own changes if
   they're going to deply a large cloud. In opposition to that, a lot
   of people *won't* know what to do to make their own images, or
   won't care and will want to use our officially provided images as a
   known-stable, known-working system that will do what they
   need. We'll be providing a standard image as a trusted place for
   people to start.

Next steps

 * Organise the sprint - the DPL is happy to approve funding for this
   as/when we ask him
 * Need to define the official image policy soon, due to the Stretch
   freeze. People want clear rules here - join in the discussion on
   the cloud list.

[1] http://meetings-archive.debian.net/pub/debian-meetings/2016/debconf16/Cloud_Images_BoF.webm
[2] http://www.einval.com/~steve/talks/Debconf16-cloud-images-bof/

Steve McIntyre, Cambridge, UK.                                steve@einval.com
"Since phone messaging became popular, the young generation has lost the
 ability to read or write anything that is longer than one hundred and sixty
 characters."  -- Ignatios Souvatzis

Attachment: signature.asc
Description: Digital signature

Reply to: