Re: Status GCE images and trademark (was Please let's not talk about "clouds")
Hi David and Jimmy,
On Thu, 16 May 2013, Jimmy Kaplowitz wrote:
> All of these things will need to be a multidirectional discussion with
> various participants having their areas of familiarity and expertise,
> and other areas where listening to other perspectives and needs will
> be most productive. My individual role is somewhat in the middle here,
> trying to help bridge gaps as well as doing some of the technical
> work. I'm very happy that my colleagues want to do the right thing for
> both Google's customers and Debian.
"do the right thing" is indeed a worthwhile motivator ;) I am all for
it too, so please take my comments/clarifications bellow in that perspective:
> > From Google's perspective, there are a few issues we don't yet know
> > how to solve:
> > 1) As our online services change (rapidly), it's important to get
> > our customers new versions of our tools quickly and easily. It's
> > terrible for customers to get an N-month-old-version of our tools in a
> > Debian-6 or Debian-7 repo, and try to use that. Our current solution
> > to that problem is to build and push images every month or so. We'd
> > *love* a mechanism wherein we can push new versions of our tools to
> > official debian archives every month or two...
none of the tools in question is in any official Debian stable release
(e.g. nor in 6 AKA oldstable/squeeze, neither in 7 AKA
stable/wheezy), and cannot be ever uploaded there -- that train is gone.
So for the next 1-3 years, until next stable Debian release comes out
you should not worry that users of official Debian repositories would
get some stale version. Packages could be uploaded to Debian unstable
(or experimental) only and nearly as often as you like (and your Debian
"sponsor", e.g. Jimmy would be capable to allocate time for) -- e.g.
could be many times a day and the official archive is updated twice a
day I believe. Such packages could even be forbidden from
entering testing, to which packages migrate usually after 10 days (or
could even be shorter) if no grave unfixed bugs present, in case you
really want to have a short leash there (but I do not think it is
necessary).
So altogether I really do not see ANY problem with rapid development of
your tools and their availability in Debian archives. Moreover presence
in the archives with automatic ways to update (instead of "go to website
and download a new version") sounds like the only sensible
approach here.
The only way I see for older versions being dumped upon users within
upcoming 1-2 years would be when Ubuntu (or some other smaller
derivative) picks up such a package from Debian and includes in
their release... but I guess we could deal with that one way (keep them
in experimental) or another (just talk and express the reason not to
include into non-rolling releases).
> > 2) gsutil has an array of c++ and python dependencies that are
> > available in Debian-6, but not with high-enough versions of those
> > dependencies.
once again: we would not even be able to upload to the official
Debian 6, nor Debian 7 -- those are released already, nothing could be
added (besides to backports. repository). And if versioned dependencies
are "good enough" in Debian sid ATM -- there is no problem. If they
are outdated -- please say so, we will work to fix that ;)
> > We have a binary-only debian package that installs all
> > the stuff necessary here, but it's unclear to us whether this is "good
> > enough" for inclusion in official Debian repos. Our .tar.gz version
> > takes these dependencies and includes them essentially "statically
> > linked" in the gsutil installation directory, which is a little
> > unconventional as well.
yes -- it would not work for 'official' Debian binary packages --
but that is not necessary (as per above).
As for backport builds for older releases there could be multiple ways
to resolve the conundrum, and in any case you might end up creating your
own APT repository to provide those binary packages built for different
releases. That is similar to what we are doing in NeuroDebian
(http://neuro.debian.net) where we do provide backport builds of all
packages we also upload to Debian proper. And we strive to not waste
effort in maintaining two copies of the same package. Quite often,
indeed system-provided libraries are outdated and it would have been
detrimental to the integrity of the underlying distribution users use if
we simply provided "fresh" builds of a new library thus replacing some
older but stable version. So far we adhered to two approaches:
1. E.g. cmtk package & mxml-- as long licenses permit I do not strip
needed 3rd party libraries in source distribution --
http://anonscm.debian.org/gitweb/?p=pkg-exppsy/cmtk.git;a=tree;hb=HEAD
Utilities/mxml ships sources (not binaries) which are built/linked
against ONLY on those systems lacking up-to-date version. But within
Debian (i.e. while uploading to sid) I assure that it builds/uses
system-wide installed/maintained versions, so those sources are not
used. This allows us to build cmtk across wide range of Debian and
Ubuntu releases built from a single source package while still
fulfilling Debian policy:
http://neuro.debian.net/pkgs/cmtk.html . So, pretty much, you could
stay with what you are doing and probably shipping those 3rd party
sources inside (or only for backported packages), or #2:
2. E.g. psychtoolbox-3 package & glew 1.9 -- we have glew 1.9 only in
Debian experimental... So I have backported glew as of 1.9 and provided
it from neurodebian as versioned binary packages (i.e. libglew1.9 and
libglew1.9-dev). libglew1.9-dev conflicts with original libglew-dev but
we do not need to have them both present on the same system while
building psychtoolbox-3 and libglew1.9 natively coexists with
libglew1.7. When 1.9 comes to proper Debian/Ubuntu releases those
versions would superseed my backported ones and everyone should stay
happy. So -- now I still build psychtoolbox-3 from the same sources for
Debian proper and NeuroDebian backports:
http://neuro.debian.net/pkgs/octave-psychtoolbox-3.html
... so the point is -- if you would be needing backport builds -- it
should be possible while keeping packaging still acceptable for Debian.
> > 3) Really, focusing on gsutil and gcutil are a little narrow-minded.
> > What we at Google *really* want, is a good understanding of the
> > process and ideal mechanism to cook those artifacts that integrates
> > with Google build tools and with Debian's release tools. Then we can
> > standardize all our Google properties and use this mechanism, and make
> > Debian and Google "tier one" partners for releasing software.
Sounds like a sound goal, but from my "user" point of view -- "I do not
care" ;) I am just told by you that I need to use gcutil to use GCE --
but it is not accessible for me on Debian (click/download/adjust PATH is
not "accessible" in my terms). And that is what I am trying to
address here. And hopefully by the time of next stable Debian things
become more clear to aim for the holy mighty artifacts ;)
> > 4) More deeply, beyond just those tools, there are several bugs
> > we've found in Debian OS, which lead to highly degraded network
> > performance, and what looks to be massive CPU starvation in cloud
> > environments. We want to fix this problem in the Debian-6 and
> > Debian-7 images we provide but the process to do that is still unclear
> > to us.
Let us know if you need any help/guidance -- I would first talk to
corresponding Debian folks (e.g. kernel team). But this is a side topic
from the 1-2 (shipping tools you currently ask users to use).
> > Really, our goal here is to do The Right Thing. The problem is that
> > from our perspective, it's a discussion that is going to take some
> > time. But we're eager to have that discussion.
Here you go -- we could discuss... or may be we could simply start
cooking ;) Do you see any particular (technical) problem with
gcutil source distribution and its goodness of fit for distribution in
Debian sid (i.e. always rolling, uploads could be multiple times a day
etc)?
--
Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Senior Research Associate, Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik
Reply to: