Re: What is to be done?

On 11 May 2014 13:57, Anders Ingemann <anders@ingemann.de> wrote:

On 11 May 2014 13:24, Tomasz Rybak <tomasz.rybak@post.pl> wrote:

Hello.
This is rather different email; I'd like to see some vision
of work (or mission?) in debian-cloud.

After finishing porting GCE to bootstrap-vz I'm trying to decide
what to do next. I do not have main itch to scratch regarding cloud
as I do not deal with cloud on daily basis.

I still want to use cloud to test my GPGPU packages (the main
reason I got interested in debian-cloud) but IIRC
there are still problem with using Debian on cg1.* and g2.*
Is someone looking at it, or should I try to see what is missing?

When looking at code I can see that we are calling apt-get, apt-key,
and so on to deal with installing packages. Would it make more sense
to use python-apt instead, and not parse /etc/apt/sources.list
ourselves?

I'm also thinking about adding few "cloudy" plugins - e.g. sending
SNS notification when image is ready, not requiring providing
AWS key and secret in manifest but using ones on ~/.boto file,
or even adding support to profiles from ~/.boto.

Or - maybe look at documentation?

Basically I'm looking for something to do ;-) The most important
is using cloud to test my packages, but I'm not sure if it's doable
without help from Amazon - or at least knowing why images are not
starting, or having logs similar to ones that we have from GCE
serial console.

Best regards.

--
Tomasz Rybak GPG/PGP key ID: 2AD5 9860
Fingerprint A481 824E 7DD3 9C0E C40A 488E C654 FB33 2AD5 9860
http://member.acm.org/~tomaszrybak

> I still want to use cloud to test my GPGPU packages (the main reason I got interested in debian-cloud) but IIRC there are still problem with using Debian on cg1.* and g2.*

> Is someone looking at it, or should I try to see what is missing?

Yes please! GPU instances have been on my todo list for quite a while, but other stuff has always gotten in the way, It would be awesome if you could take a look at it.

> When looking at code I can see that we are calling apt-get, apt-key, and so on to deal with installing packages.
> Would it make more sense to use python-apt instead, and not parse /etc/apt/sources.list ourselves?

I... I didn't know that existed. It looks awesome and would save a lot of code. At first glance I'd say this definitely makes sense!

> I'm also thinking about adding few "cloudy" plugins - e.g. sending SNS notification when image is ready, not requiring providing
> AWS key and secret in manifest but using ones on ~/.boto file, or even adding support to profiles from ~/.boto.

I think it depends on the use case whether someone already has a boto config set up.
If you think it solves a problem that you are not the only one having I'll merge PR.

Dhananjay raised an interesting issue about copying images, which sparked a longer train of thought for me about a publishing plugin by setting access rights etc.

I think it would help streamline image generation for quite a few users that need the images across regions or for different accounts.

Also: GPG signing of images would be an awesome feature especially for official debian images. You can sign an entire device, but I am not sure whether the signature breaks when you create a snapshot of an EBS volume and then verify it using a volume created from that snapshot.

> Or - maybe look at documentation?

The documentation that is missing is mostly for stuff in common/. I'd be grateful for any help in that area, though I am not sure how much work is left, all of base/ is done (it would be nice to have it completed though :-) ).

I am not sure whether it makes sense to document each task, since each one has a description and is often very small - meaning it's almost easier to just read the code.

> Basically I'm looking for something to do ;-)
> The most important is using cloud to test my packages, but I'm not sure if it's doable without help from Amazon -

> or at least knowing why images are not starting, or having logs similar to ones that we have from GCE serial console.

I feel your pain. I have been trying to get S3 HVM images up and running, a minute after I launch an instance it shuts itself down again, without any message...

You do know that there is the "get system log" option in the management console though, right? That gives you the output of the startup process.

What I could really use some help with is getting a few more Xs in my booting table. This is what it looks like right now:

X = working

_ = cannot work (grub on unpartitioned volumes)
! = does not work yet
? = untested

boot\type none msdos gpt

vbox --------------------
grub _ X X
ext X X X

ebs----------------------
pvgrub X X X
grub _ ! !
ext X X !
s3-----------------------

pvgrub X ! ?
grub _ ! !
ext ! ? ?

So in essence the following boot combinations need to be fixed:

S3:
PVM MSDOS+GPT

HVM (everything)
EBS:

HVM MSDOS+GPT (for grub)
HVM GPT (extlinux)

There is of course always the issues list on github.

Of special interest is probably the issue about ec2 optimized networking. I have coded the task, but I never got around to testing it. There's also a problem with the drivers needing to be compiled, installing build-essential requires a lot of space so we need to uninstall it afterwards. Compiling it outside of the chroot would require the system to be identical, yadda yadda and all that jazz. There's definitely some work to be done in that department.

So, as you can see, there is plenty of stuff to sink your teeth into. Take your pick!

Kind regards

Anders Ingemann

About the grub booting on HVM: Mike Christopher actually submitted a pull request that I haven't merged, because I wasn't sure whether it would break other combinations (which I think it will). You can probably get some inspiration from there to figure some things out.