Re: Status GCE images and trademark (was Please let's not talk about "clouds")
That said, the bulk of Google Cloud just wants to ship completely 100%
opensource code here. It shouldn't be terrible to get it right, and
as a posh + proper citizen of jessie. But I'm afraid I'm not sure I
see the path completely yet. Especially on the "how to build the
packages for these things, when versionitis starts setting in" front.
-david
On Thu, May 16, 2013 at 12:42 PM, David McWherter <cache@google.com> wrote:
> I think keeping us in unstable and backports is totally reasonable for
> the time being. Especially if that helps us with some of the
> binary-only packaging we may need to deal with. Jimmy indicated there
> may be quality-control issues with .debs produced by existing Google
> tools, and by FPM, that might still make that hard to deal with, but
> that could be fixed in time too.
>
> I realize that from your user point of view, non-deb is
> non-accessible. And I want to solve that problem too, but the simple
> fact is that 90% of Google Compute customer base doesn't have that
> problem. The biggest issue for me, and for them, I think, is that
> 'apt-get upgrade gcutil' doesn't work for them. Which would also
> solve your problem, I think :)
>
> I think packaging 'gcutil' as a debian that installs cleanly would be
> relatively easy, and it's on our plate (of course we've all been
> overworked with other things getting ready for Google IO). I think a
> "binary" deb is most useful for users. I'm actually not sure what
> differentiates a "binary" deb from a "source" deb if the tool is 100%
> python. Our gcutil would statically link in a bunch of other packages
> it needs. 'gsutil' i think is still a harder story. We're not 100%
> confident that the .deb that we create works correctly on both debian6
> and debian7 because of the crazy version mismatchy problems.
>
> -david
>
> On Thu, May 16, 2013 at 12:08 PM, Yaroslav Halchenko
> <debian@onerussian.com> wrote:
>> Hi David and Jimmy,
>>
>> On Thu, 16 May 2013, Jimmy Kaplowitz wrote:
>>
>>> All of these things will need to be a multidirectional discussion with
>>> various participants having their areas of familiarity and expertise,
>>> and other areas where listening to other perspectives and needs will
>>> be most productive. My individual role is somewhat in the middle here,
>>> trying to help bridge gaps as well as doing some of the technical
>>> work. I'm very happy that my colleagues want to do the right thing for
>>> both Google's customers and Debian.
>>
>> "do the right thing" is indeed a worthwhile motivator ;) I am all for
>> it too, so please take my comments/clarifications bellow in that perspective:
>>
>>> > From Google's perspective, there are a few issues we don't yet know
>>> > how to solve:
>>
>>> > 1) As our online services change (rapidly), it's important to get
>>> > our customers new versions of our tools quickly and easily. It's
>>> > terrible for customers to get an N-month-old-version of our tools in a
>>> > Debian-6 or Debian-7 repo, and try to use that. Our current solution
>>> > to that problem is to build and push images every month or so. We'd
>>> > *love* a mechanism wherein we can push new versions of our tools to
>>> > official debian archives every month or two...
>>
>> none of the tools in question is in any official Debian stable release
>> (e.g. nor in 6 AKA oldstable/squeeze, neither in 7 AKA
>> stable/wheezy), and cannot be ever uploaded there -- that train is gone.
>>
>> So for the next 1-3 years, until next stable Debian release comes out
>> you should not worry that users of official Debian repositories would
>> get some stale version. Packages could be uploaded to Debian unstable
>> (or experimental) only and nearly as often as you like (and your Debian
>> "sponsor", e.g. Jimmy would be capable to allocate time for) -- e.g.
>> could be many times a day and the official archive is updated twice a
>> day I believe. Such packages could even be forbidden from
>> entering testing, to which packages migrate usually after 10 days (or
>> could even be shorter) if no grave unfixed bugs present, in case you
>> really want to have a short leash there (but I do not think it is
>> necessary).
>>
>> So altogether I really do not see ANY problem with rapid development of
>> your tools and their availability in Debian archives. Moreover presence
>> in the archives with automatic ways to update (instead of "go to website
>> and download a new version") sounds like the only sensible
>> approach here.
>>
>> The only way I see for older versions being dumped upon users within
>> upcoming 1-2 years would be when Ubuntu (or some other smaller
>> derivative) picks up such a package from Debian and includes in
>> their release... but I guess we could deal with that one way (keep them
>> in experimental) or another (just talk and express the reason not to
>> include into non-rolling releases).
>>
>>
>>> > 2) gsutil has an array of c++ and python dependencies that are
>>> > available in Debian-6, but not with high-enough versions of those
>>> > dependencies.
>>
>> once again: we would not even be able to upload to the official
>> Debian 6, nor Debian 7 -- those are released already, nothing could be
>> added (besides to backports. repository). And if versioned dependencies
>> are "good enough" in Debian sid ATM -- there is no problem. If they
>> are outdated -- please say so, we will work to fix that ;)
>>
>>> > We have a binary-only debian package that installs all
>>> > the stuff necessary here, but it's unclear to us whether this is "good
>>> > enough" for inclusion in official Debian repos. Our .tar.gz version
>>> > takes these dependencies and includes them essentially "statically
>>> > linked" in the gsutil installation directory, which is a little
>>> > unconventional as well.
>>
>> yes -- it would not work for 'official' Debian binary packages --
>> but that is not necessary (as per above).
>>
>> As for backport builds for older releases there could be multiple ways
>> to resolve the conundrum, and in any case you might end up creating your
>> own APT repository to provide those binary packages built for different
>> releases. That is similar to what we are doing in NeuroDebian
>> (http://neuro.debian.net) where we do provide backport builds of all
>> packages we also upload to Debian proper. And we strive to not waste
>> effort in maintaining two copies of the same package. Quite often,
>> indeed system-provided libraries are outdated and it would have been
>> detrimental to the integrity of the underlying distribution users use if
>> we simply provided "fresh" builds of a new library thus replacing some
>> older but stable version. So far we adhered to two approaches:
>>
>> 1. E.g. cmtk package & mxml-- as long licenses permit I do not strip
>> needed 3rd party libraries in source distribution --
>> http://anonscm.debian.org/gitweb/?p=pkg-exppsy/cmtk.git;a=tree;hb=HEAD
>> Utilities/mxml ships sources (not binaries) which are built/linked
>> against ONLY on those systems lacking up-to-date version. But within
>> Debian (i.e. while uploading to sid) I assure that it builds/uses
>> system-wide installed/maintained versions, so those sources are not
>> used. This allows us to build cmtk across wide range of Debian and
>> Ubuntu releases built from a single source package while still
>> fulfilling Debian policy:
>> http://neuro.debian.net/pkgs/cmtk.html . So, pretty much, you could
>> stay with what you are doing and probably shipping those 3rd party
>> sources inside (or only for backported packages), or #2:
>>
>> 2. E.g. psychtoolbox-3 package & glew 1.9 -- we have glew 1.9 only in
>> Debian experimental... So I have backported glew as of 1.9 and provided
>> it from neurodebian as versioned binary packages (i.e. libglew1.9 and
>> libglew1.9-dev). libglew1.9-dev conflicts with original libglew-dev but
>> we do not need to have them both present on the same system while
>> building psychtoolbox-3 and libglew1.9 natively coexists with
>> libglew1.7. When 1.9 comes to proper Debian/Ubuntu releases those
>> versions would superseed my backported ones and everyone should stay
>> happy. So -- now I still build psychtoolbox-3 from the same sources for
>> Debian proper and NeuroDebian backports:
>> http://neuro.debian.net/pkgs/octave-psychtoolbox-3.html
>>
>> ... so the point is -- if you would be needing backport builds -- it
>> should be possible while keeping packaging still acceptable for Debian.
>>
>>> > 3) Really, focusing on gsutil and gcutil are a little narrow-minded.
>>> > What we at Google *really* want, is a good understanding of the
>>> > process and ideal mechanism to cook those artifacts that integrates
>>> > with Google build tools and with Debian's release tools. Then we can
>>> > standardize all our Google properties and use this mechanism, and make
>>> > Debian and Google "tier one" partners for releasing software.
>>
>> Sounds like a sound goal, but from my "user" point of view -- "I do not
>> care" ;) I am just told by you that I need to use gcutil to use GCE --
>> but it is not accessible for me on Debian (click/download/adjust PATH is
>> not "accessible" in my terms). And that is what I am trying to
>> address here. And hopefully by the time of next stable Debian things
>> become more clear to aim for the holy mighty artifacts ;)
>>
>>> > 4) More deeply, beyond just those tools, there are several bugs
>>> > we've found in Debian OS, which lead to highly degraded network
>>> > performance, and what looks to be massive CPU starvation in cloud
>>> > environments. We want to fix this problem in the Debian-6 and
>>> > Debian-7 images we provide but the process to do that is still unclear
>>> > to us.
>>
>> Let us know if you need any help/guidance -- I would first talk to
>> corresponding Debian folks (e.g. kernel team). But this is a side topic
>> from the 1-2 (shipping tools you currently ask users to use).
>>
>>> > Really, our goal here is to do The Right Thing. The problem is that
>>> > from our perspective, it's a discussion that is going to take some
>>> > time. But we're eager to have that discussion.
>>
>> Here you go -- we could discuss... or may be we could simply start
>> cooking ;) Do you see any particular (technical) problem with
>> gcutil source distribution and its goodness of fit for distribution in
>> Debian sid (i.e. always rolling, uploads could be multiple times a day
>> etc)?
>>
>> --
>> Yaroslav O. Halchenko, Ph.D.
>> http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
>> Senior Research Associate, Psychological and Brain Sciences Dept.
>> Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
>> Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
>> WWW: http://www.linkedin.com/in/yarik
Reply to: