Re: Moving GCE scripts to Python

On 30 January 2014 23:05, Jimmy Kaplowitz <jkaplowitz@google.com> wrote:

Hi Tomaz,

This is exciting! How are you trying to ssh in? Does gcutil ssh <instance> not work? Also, if you configure Linux to use ttyS0 as output - see tasks/gce/30-grub in the master branch - you can get useful debugging information via gcutil getserialportoutput <instance>. If you configure GRUB the same way (a buggy version of this is also in master's tasks/gce/30-grub), you can even use this to debug booting problems.

Thanks! I'll take a closer look and try your code as soon as I get a chance.

- Jimmy

On Thu, Jan 30, 2014 at 1:27 PM, Tomasz Rybak <tomasz.rybak@post.pl> wrote:

Please ignore fact that I'm responding to myself...

I've been working on changing GCE-building scripts
to use new Python code.
Today I managed to build image, and uploaded code
to branch gce-python in my GitHub repository:
https://github.com/rybaktomasz/build-debian-cloud

I was able to upload image to GCE and run it,
but I was not able to ssh-in to running machine.
It might be something simple (as me forgetting
to configure SSH) or larger problem - e.g. grub
configuration or partitioning.

There are still some tasks to do, when compared
to old GCE code:
1. adding preferences to use kernel from backports
2. cleaning - e.g. removing Google repository
3. uploading image - currently done by 95-register-image
4. fixing apt sources - not all mirror addresses
are used in all files

In new Tasks I called tar and apt directly - it might be better
to use Python modules.

Small feedback regarding current state of affairs.
The most difficult was deciding whether to use existing Tasks
or writing new ones - e.g. for adding multi-lines backports.
Old model was straightforward - it was running shell scripts in
order governed by their (numbered) names. Now we have phases
and dependencies; they add more flexibility (as ability
to skip some steps depending on manifest) but require
completely different mental model.

IMO the most important part of documentation would be graph
of Tasks, their dependencies, and sample order of execution.
This way reader will be able to get "the feeling" of system's
inner workings.

Second biggest problem was with the time it takes to build
image. On my system it takes about 15 minutes to build
image, so agile experimentation is hard ;-)
It forced me to try to come with correct solution
faster, so I'm not complaining to much here.

I'll try to convert more of the old GCE scripts
soon, but first want to have image working. If someone
sees mistakes - even the most obvious ones - please let
me know.

Best regards.

--
Tomasz Rybak GPG/PGP key ID: 2AD5 9860
Fingerprint A481 824E 7DD3 9C0E C40A 488E C654 FB33 2AD5 9860
http://member.acm.org/~tomaszrybak

Hey Tomasz

Sounds very promising. A few things:

> IMO the most important part of documentation would be graph of Tasks, their dependencies, and sample order of execution.

> This way reader will be able to get "the feeling" of system's inner workings.

I am working of a d3.js representation of that, it is working, but I have taken it offline until I have the rest of the documentation working. I'll probably publish the documentation this weekend (along with a proper release of the python branch under a new name: bootstrap-vz).

> Second biggest problem was with the time it takes to build image.

> On my system it takes about 15 minutes to build image, so agile experimentation is hard ;-)

I presetn to you: The prebootstrapped plugin. It takes a snapshot of your volume right after debootstrap and apt-get install has run. When you define a path to an image/snapshot in the manifest it will remove all bootstrapping tasks and pick up from there, saving you ~13 minutes out of 15. Without an image path in the manifest it will create a snapshot right after apt-get install.

That right there is one of the reasons I need to get the documentation up and running, you could've saved hours if you knew this beforehand :-)