That may be a possibility, though I'd like to keep all utilities free.On 18 September 2013 12:28, olivier sallou <olivier.sallou@gmail.com> wrote:
>
>
>
> 2013/9/18 Anders Ingemann <anders@ingemann.de>
>>
>> On 18 September 2013 08:31, olivier sallou <olivier.sallou@gmail.com>
>> wrote:
>> >
>> >
>> >
>> > 2013/9/17 Anders Ingemann <anders@ingemann.de>
>> >>
>> >> On 17 September 2013 15:46, olivier sallou <olivier.sallou@gmail.com>
>> >> wrote:
>> >> >
>> >> >
>> >> >
>> >> > 2013/9/17 Anders Ingemann <anders@ingemann.de>
>> >> >>
>> >> >> > The virtualbox provider in python branch worked fine at booting
>> >> >> > the
>> >> >> > first bootable disk with grub installed via the loopback.
>> >> >> Honestly, I was never able to get it to work. I think it failed
>> >> >> exactly because it was a loopback device. Can you send a manifest
>> >> >> that
>> >> >> reproduces your setup? Maybe I can work my way up from there.
>> >> >> > did you use grub2 on host machine ? grub1 fails to install over
>> >> >> > loopback
>> >> >> > device and lead to boot error (can't find grub).
>> >> >> Exactly, which is why I am using dmsetup to fake a real hdd, so grub
>> >> >> installs without a hitch :-).
>> >> >
>> >> >
>> >> > I am using grub2 on my computer, it manages correctly the install
>> >> > over
>> >> > loopback.
>> >> > Being able to use grub1 would be perfect, but I do not know how to
>> >> > make
>> >> > it
>> >> > worked. I tried many things to get grub working, but it only worked
>> >> > when
>> >> > I
>> >> > used grub2 :-(
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> > did you look also at your generated device.map and grub.cfg files
>> >> >> > ?
>> >> >> Yes, everything seems fine, I'll have another look though (the
>> >> >> config
>> >> >> is in the pastebin I provided)
>> >> >>
>> >> >> > I know that while trying to setup grub stuff etc... for kvm and
>> >> >> > virtualbox I had issues with auto-generated grub.cfg using
>> >> >> > loopback
>> >> >> > devices
>> >> >> > instead of using disk devices (see loopback keywork in grub.cfg).
>> >> >> Yes, I circumvent that by making the fake hdd, although maybe there
>> >> >> is
>> >> >> some leftover in the grub.cfg... I'll check it out.
>> >> >>
>> >> >> > I also used disk device (/dev/sda for example) instead of disk
>> >> >> > uuids
>> >> >> > in
>> >> >> > boot menu setup of grub (disk uuids may not be the same when
>> >> >> > booting
>> >> >> > on your
>> >> >> > final host).
>> >> >> Actually. They will be exactly the same, I even have problems when
>> >> >> attaching two disks created by the same snapshot because VBox
>> >> >> doesn't
>> >> >> like duplicate UUIDs. At the very least the partition UUIDs will be
>> >> >> the same (which is what I use in fstab).
>> >> >>
>> >> >> Anders
>> >> >>
>> >> >>
>> >> >> On 17 September 2013 08:13, olivier sallou
>> >> >> <olivier.sallou@gmail.com>
>> >> >> wrote:
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > 2013/9/16 Anders Ingemann <anders@ingemann.de>
>> >> >> >>
>> >> >> >> Hey guys
>> >> >> >>
>> >> >> >> I posted to help-grub@gnu.org, asking for help to figure out why
>> >> >> >> my
>> >> >> >> VirtualBox image is stuck at "GRUB loading.". I thought that it
>> >> >> >> might
>> >> >> >> be more relevant over there, but maybe you guys have some ideas
>> >> >> >> as
>> >> >> >> well.
>> >> >> >
>> >> >> > did you look also at your generated device.map and grub.cfg files
>> >> >> > ?
>> >> >> >
>> >> >> > I know that while trying to setup grub stuff etc... for kvm and
>> >> >> > virtualbox I
>> >> >> > had issues with auto-generated grub.cfg using loopback devices
>> >> >> > instead
>> >> >> > of
>> >> >> > using disk devices (see loopback keywork in grub.cfg).
>> >> >> >
>> >> >> > I also used disk device (/dev/sda for example) instead of disk
>> >> >> > uuids
>> >> >> > in
>> >> >> > boot
>> >> >> > menu setup of grub (disk uuids may not be the same when booting on
>> >> >> > your
>> >> >> > final host).
>> >> >> >
>> >> >> > Olivier
>> >> >> >
>> >> >> >>
>> >> >> >> Here's the link to the mailing list:
>> >> >> >> http://lists.gnu.org/archive/html/help-grub/2013-09/threads.html
>> >> >> >> (not
>> >> >> >> a direct link, mailman seems to take its time indexing my post)
>> >> >> >> ...and here's what I wrote
>> >> >> >> ---------------------------
>> >> >> >> Hello everybody
>> >> >> >>
>> >> >> >> I am developing the debian bootstrapper "build-debian-cloud"
>> >> >> >> (https://github.com/andsens/build-debian-cloud) and am having
>> >> >> >> some
>> >> >> >> trouble getting grub to boot my debian installation.
>> >> >> >> Since I bootstrap from a host system (with chroot etc.) I had to
>> >> >> >> use
>> >> >> >> some workarounds to get grub installed onto a loopback device
>> >> >> >>
>> >> >> >> (http://ebroder.net/2009/08/04/installing-grub-onto-a-disk-image/).
>> >> >> >>
>> >> >> >> The setup consists of a vdi image partitioned with GPT into boot,
>> >> >> >> root, swap (in that order).
>> >> >> >> I am unable to get past the "GRUB loading." message when booting
>> >> >> >> the
>> >> >> >> image in VirtualBox.
>> >> >> >>
>> >> >> >> I have a hard time figuring out what is wrong with the setup,
>> >> >> >> especially because the scenario is a bit off the beaten path
>> >> >> >> (mounting
>> >> >> >> vdi as an network block device, bootstrapping, using dmsetup to
>> >> >> >> fake
>> >> >> >> a
>> >> >> >> real hdd etc.). So the usual "just run BootRepair" or "reinstall
>> >> >> >> grub"
>> >> >> >> won't really help to create a stable bootstrapping process.
>> >> >> >> I have run Ubuntus Boot Repair system to check if anything was
>> >> >> >> wrong,
>> >> >> >> but I can't seem to find anything. The output is here:
>> >> >> >> http://paste.ubuntu.com/6115737/
>> >> >> >>
>> >> >> >> The setup can be fully reproduced by cloning my repo and running
>> >> >> >> `./build-debian-cloud manifests/virtualbox.manifest.json` (only
>> >> >> >> tested
>> >> >> >> on debian wheezy).
>> >> >> >> Simply create a new virtual machine in VBox, attach the resulting
>> >> >> >> image and boot it.
>> >> >> >>
>> >> >> >> I would appreciate any help you can offer
>> >> >> >> Anders Ingemann
>> >> >> >> ---------------------------
>> >> >> >>
>> >> >> >>
>> >> >> >> --
>> >> >> >> To UNSUBSCRIBE, email to debian-cloud-request@lists.debian.org
>> >> >> >> with a subject of "unsubscribe". Trouble? Contact
>> >> >> >> listmaster@lists.debian.org
>> >> >> >> Archive:
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> [🔎] CAMcOGXFiLv2+WQ3fXe87wGv6ihs8eTfQBejqb4Vuof99OnrzZg@mail.gmail.com" target="_blank">http://lists.debian.org/[🔎] CAMcOGXFiLv2+WQ3fXe87wGv6ihs8eTfQBejqb4Vuof99OnrzZg@mail.gmail.com
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> >
>> >> >> > gpg key id: 4096R/326D8438 (keyring.debian.org)
>> >> >> >
>> >> >> > Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D
>> >> >> > 8438
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> >
>> >> > gpg key id: 4096R/326D8438 (keyring.debian.org)
>> >> >
>> >> > Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>> >>
>> >> OK, having done a bit of research, I feel like I need to clear some
>> >> stuff up, so we avoid any further confusion. I *think* I am getting
>> >> this right, please correct me if I am wrong.
>> >>
>> >> > I am using grub2 on my computer, it manages correctly the install
>> >> > over
>> >> > loopback.
>> >> > Being able to use grub1 would be perfect, but I do not know how to
>> >> > make
>> >> > it worked. I tried many things to get grub working, but it only
>> >> > worked when
>> >> > I used grub2 :-(
>> >> This is the main point that got me confused. The bootstrapper doesn't
>> >> install grub1 on virtualbox and never has. It has always been grub2,
>> >> debian switched to grub2 quite a while ago (since squeeze:
>> >> https://wiki.debian.org/Grub). I am a bit puzzled as to how you
>> >> managed to install the bootloader since grub2 gets all confused about
>> >> loopback devices, which is why I haven't been able to install grub2
>> >> with neither your virtualbox version nor mine (yet :-) ).
>> >> Are you sure you didn't accidentally test the wrong image after you
>> >> bootstrapped?
>> >
>> >
>> > Sure, I am currently working on a vagrant provider, which is an extend
>> > of
>> > virtualbox provider and it works fine in virtualbox (at least for vbox
>> > part,
>> > not vagrant...).
>> >
>> > I had a quick look, and the fact is I get it worked because I use grub2
>> > 2.0.x from sid. Previous versions (1.98, 1.99) did not manage correctly
>> > indeed the loopback interface. The version > 2.0 manages it correctly.
>> >
>> > If you are on ubuntu or wheezy,... you get older grub2 release and it
>> > does
>> > not work...
>> > v2.x should be in testing but it seems some bugs prevent migration to
>> > testing.
>> >>
>> >>
>> >> There is one exception to the use of grub1. We need it on EC2 because
>> >> of the paravirtualized (PV) nature of the system. Here the grub
>> >> bootloader is not actually installed to the volume, but rather
>> >> launched from outside the box. That special version of grub then looks
>> >> into the volume and parses the menu.lst file.
>> >> menu.lst is the way grub1 figured out where the inital ram disk
>> >> (initrd) was. The EC2 PV bootloader hasn't been updated yet and can
>> >> therefore not parse the newer grub2 grub.cfg file.
>> >> To that end we create a special grub2 configuration generation script
>> >> which mimicks the old menu.lst layout. This config is then outputted
>> >> by `update-grub` to /boot/grub/grub.cfg, which my bootloader symlinks
>> >> to /boot/grub/menu.lst where PV-Grub (http://wiki.xen.org/wiki/PvGrub)
>> >> can find it.
>> >> Bonus info: This is what Charles Plessy has been working on with the
>> >> "pv-grub-menu" package. Once that is working and packaged we don't the
>> >> special grub2 config files any longer, because the tool generates that
>> >> file.
>> >>
>> >> Now... on virtualbox _we don't need that workaround_. VirtualBox
>> >> generates a full environment for grub to actually load. Meaning vbox
>> >> couldn't care less about what the grub.cfg or menu.lst looks like.
>> >>
>> >> OK, so that was a bit of a tangent. I also want to report back what I
>> >> have figured out so far.
>> >>
>> >> I am fairly certain that my grub.cfg is wrong, it has a `set
>> >> root='/dev/mapper/vdb'`, which surely is not what grub sees at boot,
>> >> mostly because '/dev/mapper' is only used by kpartx and lvm. Then
>> >> again, what would root be set to when you boot from an lvm partitioned
>> >> volume???
>> >> When I change that line to `set root='/dev/sda1'` it still does not
>> >> work. But at this point I figured out that my partitioning may be a
>> >> bit wrong, the boot partition started at sector 0 instead of the usual
>> >> 1MiB mark. Also, I used MB for partition sizes not MiB, meaning my
>> >> 1024MB volume suddenly grew to 1074MB.
>> >>
>> >> There is another issue about grub not being able to figure out where
>> >> the partitions are located on the volume. I am pretty sure however
>> >> that this can be fixed with a proper device.map (btw. the device.map
>> >> file is not read at boot, it is used by the grub tools to figure out
>> >> how to make the grub.cfg file work).
>> >>
>> >> I believe grub should map hd0 as /dev/sda at boot, however even when I
>> >> try to set root to (hd0,gpt1) it still does not work.
>> >>
>> >> Again, if anybody finds an error with what I have written, please
>> >> chime in, I would appreciate it.
>> >>
>> >>
>> >> Anders Ingemann
>> >
>> >
>> >
>> >
>> > --
>> >
>> > gpg key id: 4096R/326D8438 (keyring.debian.org)
>> >
>> > Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>>
>> > Sure, I am currently working on a vagrant provider
>> Aw maan, I was looking forward to doing that myself :-)
>
>
> Vagrant provider almost work, I have a few updates to do but first tests are
> fine.
>
> However Vagrant needs VirtualBox on host (not image) to *pack* the image in
> a Vagrant box. VirtualBox is in contrib, not free (and contrib backports for
> wheezy).
>
> Should we require virtualbox and launch commands to create the box, even if
> in contrib, or should we generate the vmdk disk and give instructions to the
> user to create the pack box ? (I have a shell script for this for the
> moment)
>
>
>
>>
>> Guess I'll just work on my other plugin idea then: Minimizing the
>> image size by e.g. preventing apt from writing the apt-cache to the
>> bootstrap volume (meaning the vdi won't be expanded).
>
>
> For the moment (what I coded), the virtualbox provider create a raw disk,
> and is converted to vdi only at the end of the process. So vdi will be the
> expanded size, not a compact one.
>>
>>
>> > I had a quick look, and the fact is I get it worked because I use grub2
>> > 2.0.x from sid. Previous versions (1.98, 1.99) did not manage correctly
>> > indeed the loopback interface. The version > 2.0 manages it correctly.
>> Oooh, ok. That explains it!
>>
>> > v2.x should be in testing but it seems some bugs prevent migration to
>> > testing.
>> No matter, we'll just handle it depending on the grub version once
>> it's in debian testing. It'll cut down on the weird hackery I am doing
>> now :-)
>
>
> I do not know however when it gonna move to testing. Issue is related to
> dependencies, so it could last long... :-(
>
> Olivier
>
>
>
> --
>
> gpg key id: 4096R/326D8438 (keyring.debian.org)
>
> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
> Should we require virtualbox and launch commands to create the box, even if in contrib,
Are you sure we can't package with qemu tools only?
No, definitely not. The entire philosophy for this bootstrapper is to
> or should we generate the vmdk disk and give instructions to the user to create the pack box ? (I have a shell script for this for the moment)
avoid exactly those scenarios, no matter how convoluted a packaging
process may be, you can always script it. Once you require manual
steps the iterable nature of the tool goes away. A great advantage of
this tool is that you can modify the script/add plugins, bootstrap,
test and then repeat the whole process without any cumbersome manual
steps. Also, because of the manifest everything is reproducible,
manual steps remove that feature.
In the WIP branch I use qemu-nbd to bootstrap onto a proper vdi volume :-)
> For the moment (what I coded), the virtualbox provider create a raw disk, and is converted to vdi only at the end of the process. So vdi will be the expanded size, not a compact one.
> I had a quick look, and the fact is I get it worked because I use grub2 2.0.x from sid. Previous versions (1.98, 1.99) did not manage correctly indeed the loopback interface. The version > 2.0 manages it correctly.
Oooh, ok. That explains it!
> I do not know however when it gonna move to testing. Issue is related to dependencies, so it could last long... :-(Doesn't matter, the hack is scripted now, so it's not really an issue
any longer.
p.s.: You forgot to reply to the mailing list btw.
gpg key id: 4096R/326D8438 (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438