On Jan 22, 2014 10:53 PM, "olivier sallou" <olivier.sallou@gmail.com> wrote:
>
>
>
>
> 2014/1/22 Jimmy Kaplowitz <jkaplowitz@google.com>
>>
>> I'm fine with Olivier doing this, but if access to KVM is a testing/coding bottleneck for you, we can certainly give you no-cost access to GCE to help work on build-debian-cloud, as we did with Tomasz. :) Offer stands for anyone helping, including Olivier.
>
> I am fine for pure KVM testing, I manage a local private cloud running on kvm (and OpenNebula). However I cannot test pure GCE related stuff but by my own expenses.
> If we could get a free GCE access for GCE provider testing and testing KVM in an other environment than mine, it would help .
Great. Yes, I realized you had regular KVM setup, but it sounds like Anders doesn't. Both of you are welcome to use the free GCE access for the purpose of developing or testing a GCE build-debian-cloud provider and the KVM provider on which it will presumably depend.
>> We've made a shared low-quota project intended for Debian image hacking, at Google's expense, and it's very much KVM-based.
>>
>> Send me your Google account info off-list if interested.
>
>
> What info do you need ? (google account email?)
Yeah, that's it. Any Gmail account, non-Gmail Google account, or (depending on domain administrator settings) Google Apps account should work.
If you want me to add the Gmail account you're using on this list, that's fine. I'm traveling today but will set you up soon and then send you the details off-list.
- Jimmy
> Thanks
>
> Olivier
>>
>> - Jimmy
>>
>> On Jan 22, 2014 9:13 AM, "Anders Ingemann" <anders@ingemann.de> wrote:
>>>
>>> On 22 January 2014 16:02, olivier sallou <olivier.sallou@gmail.com> wrote:
>>>>
>>>>
>>>>
>>>>
>>>> 2014/1/22 olivier sallou <olivier.sallou@gmail.com>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2014/1/22 Anders Ingemann <anders@ingemann.de>
>>>>>>
>>>>>> On 22 January 2014 11:41, olivier sallou <olivier.sallou@gmail.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2014/1/22 Anders Ingemann <anders@ingemann.de>
>>>>>>>>
>>>>>>>> On 22 January 2014 10:43, olivier sallou <olivier.sallou@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2014/1/22 Anders Ingemann <anders@ingemann.de>
>>>>>>>>>>
>>>>>>>>>> On 22 January 2014 08:23, olivier sallou <olivier.sallou@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2014/1/22 Jimmy Kaplowitz <jkaplowitz@google.com>
>>>>>>>>>>>>
>>>>>>>>>>>> Neat. Yeah, a GCE image is simply a raw bootable disk named disk.raw, compressed in a gzipped GNU-format tarball with a filename ending in .tar.gz. We also encourage creating disk.raw as a sparse file and using GNU tar's S flag to minimize image size and add time. The tarball is necessary but hopefully the code is general enough to handle that. Certain bits of our image snapshotting tool gcimagebundle expect everything aside from the bootloader to be in a single MSDOS partition number 1.
>>>>>>>>>>>
>>>>>>>>>>> I think this is the case for virtualbox provider.
>>>>>>>>>>>>
>>>>>>>>>>>> We make various tweaks, such as installing the various integration software I've mentioned before, pointing to our Debian mirror, and (with Tomasz's pull request into the google GitHub fork) the host machine's NTP server, etc. Nothing that changes the essence.
>>>>>>>>>>>
>>>>>>>>>>> Some of those (setting up a mirror, installing packages) can be done:
>>>>>>>>>>> 1) with plugins (but as it is a requirement for you this is not the best choice)
>>>>>>>>>>> 2) in your plugin task, use the library part that manage package install etc.. if you look at cloud-init plugin task for example, you will see how to add a source (here debian backports) and packages to the image (here "sudo").
>>>>>>>>>>>
>>>>>>>>>>> The rest of the plugin task can focus on copying some files in the image, setting ntp etc...
>>>>>>>>>>>>
>>>>>>>>>>>> The strategy you suggest sounds worth trying indeed.
>>>>>>>>>>>>
>>>>>>>>>>>> - Jimmy
>>>>>>>>>>>>
>>>>>>>>>>>> On Jan 21, 2014 10:32 PM, "olivier sallou" <olivier.sallou@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2014/1/21 Tomasz Rybak <tomasz.rybak@post.pl>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As Jimmy wrote in his email from 2014-01-14, I started
>>>>>>>>>>>>>> looking at GCE-related parts of build-debian-cloud.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I assume that the course for now is changing the scripts
>>>>>>>>>>>>>> to work in Python, similar to what's being done with EC2
>>>>>>>>>>>>>> and VirtualBox parts. Should we branch from andsens/python
>>>>>>>>>>>>>> and work on it, or do something else? Also, who'll create
>>>>>>>>>>>>>> the main branch (GCE-python-WIP?), into which we would
>>>>>>>>>>>>>> pull proposed changes? I think the best solution would be
>>>>>>>>>>>>>> to create such branch in repository
>>>>>>>>>>>>>> https://github.com/google/build-debian-cloud
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As for the work to do, I think we'll need to:
>>>>>>>>>>>>>> 1. change gce file to proper manifest
>>>>>>>>>>>>>> 2. move tasks from tasks/gce to providers/gce and
>>>>>>>>>>>>>> rewrite them in Python
>>>>>>>>>>>>>> 3. integrate cloud-init when appropriate
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> If the base image requirement is a raw image file and GCE only adds startup/management scripts for boot etc... you may only develop a plugin and use VirtualBox provider which is in fact a quite generic one (not only virtualbox).
>>>>>>>>>>>>>
>>>>>>>>>>>>> I personnaly use VirtualBox provider for my KVM machines and use the opennebula plugin for the OpenNebula contextualization (will be modified soon to use cloud-init too).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Then, for GCE, it would be, for the user, only a matter of user VirtualBox provider (raw format) and activating the GCE plugin.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Olivier
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This is rough idea, and I have not touched
>>>>>>>>>>>>>> packaging of
>>>>>>>>>>>>>> https://github.com/GoogleCloudPlatform/compute-image-packages
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Have I missed something? I assume we need to have
>>>>>>>>>>>>>> more detailed plan of moving to Python so anyone
>>>>>>>>>>>>>> can see what is to be done and volunteer to some
>>>>>>>>>>>>>> tasks ;-) For now I just want to start discussion
>>>>>>>>>>>>>> to see what I forgot about.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best regards.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Tomasz Rybak GPG/PGP key ID: 2AD5 9860
>>>>>>>>>>>>>> Fingerprint A481 824E 7DD3 9C0E C40A 488E C654 FB33 2AD5 9860
>>>>>>>>>>>>>> http://member.acm.org/~tomaszrybak
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>>
>>>>>>>>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>>>>>>>>>>>
>>>>>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>>>>>>>>>>
>>>>>>>>>> Heh, it's great to see others are able to figure out my framework already, without documentation. I should really get on with writing it though... :-)
>>>>>>>>>>
>>>>>>>>>> > If the base image requirement is a raw image file and GCE only adds startup/management scripts for boot etc... you may only develop a plugin and use VirtualBox provider which is in fact a quite generic one (not only virtualbox).
>>>>>>>>>>
>>>>>>>>>> I would strongly suggest to refrain from doing that. If the VirtualBox provider is indeed very generic, things should be abstracted into task sets. I have not abstracted much of it until now, since I wasn't aware of the commonalities between providers (given that there are only to - or three now with kvm). If You add GCE as a separate provider, I can take a look at it and create some new tasksets that should make the task resolving a bit easier.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> So, do you think we should create a KVM provider ? (which would be 99% equivalent to VirtualBox for the moment but would preserve from virtualbox modifications)
>>>>>>>>>
>>>>>>>>> Olivier
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The advantages of having a provider rather than a plugin are manifold.
>>>>>>>>>> Plugins tasks will be resolved *after* the provider tasks, meaning they will be able to remove some provider tasks if they do something more specific.
>>>>>>>>>> You can enforce plugin compatibility in the manifest schemas by looking at the provider string, having a plugin look at what other plugins are loaded is just messy.
>>>>>>>>>> Disadvantages of creating a provider as a plugin:
>>>>>>>>>> Any changes you make because the virtualbox provider doesn't quite fit will become hard to understand - one provider adding a task and the plugin removing that same task... you might end up with a tasklist that is completely different from virtualbox (once you are done).
>>>>>>>>>> Changes to the virtualbox provider will now have to be carefully made because other providers suddenly rely on it
>>>>>>>>>> You will have a hard time adding special things to the manifest since the vbox provider applies its own manifest schema
>>>>>>>>>> > In your plugin task, use the library part that manage package install etc.. if you look at cloud-init plugin task for example, you will see how to add a source (here debian backports) and packages to the image (here "sudo").
>>>>>>>>>> What he said! The package API can save you a lot of trouble. Btw, you can add trusted keyrings to apt through the manifest as well.
>>>>>>>>>>
>>>>>>>>>> Anders
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>>>>>>>>>
>>>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>>>>>>>>
>>>>>>>>
>>>>>>>> > So, do you think we should create a KVM provider?
>>>>>>>> I do. Surely things like guest additions/guest tools are different. It would also allow use to make a proper vagrant box for it.
>>>>>>>> The big challenge is making some proper taskset abstractions. It should be possible to create a minimal resolve_tasks() function if we code the tasksets properly (seeing how vbox and kvm are very similar), this would be very useful for future providers as well (e.g. gce or hp-cloud).
>>>>>>>
>>>>>>> For the current code, the only difference between VB and KVM is KVM have no guest-additions and expect raw format instead of vdi.
>>>>
>>>> Do you want me to create the kvm provider and make a pull request, or do you plan to do it?
>>>>
>>>> Olivier
>>>>>>>
>>>>>>>
>>>>>>> Olivier
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>>>>>>>
>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>>>>>>
>>>>>>
>>>>>> Alright, but something like the virtio drivers would be useful, right?
>>>>>
>>>>>
>>>>> I first planned to create a plugin for this above vb provider, but if we have a kvm provider it would make sense in this case to create a specific optional task in kvm provider.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>>>>>
>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>>>>
>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>>>
>>>
>>> ATM I am not able to test in kvm so you can go ahead and create the provider.
>
>
>
>
> --
>
> gpg key id: 4096R/326D8438 (keyring.debian.org)
>
> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438