On Thu, 31 Mar 2016 19:18:23 +0200 Thomas Goirand <zigo@debian.org> wrote: > Hi Steve, > > On 03/30/2016 02:55 AM, Steve McIntyre wrote: > > IME it's much easier to automate using VMs if possible, but I'm open > > to being convinced otherwise. > > Sure, but how do I get a new VM for it? Should I write a mail to > dsa@ ? One of the benefits of lava.debian.net is that the machine running the VM can be local to the tester with the results being published through lava.debian.net. That's useful for some tests but tends to lead to that machine only being available when the local tester is available. I'm currently looking at adding a VM host box and that would be available to everyone using lava.debian.net. Bare metal boxes require a lot more infrastructure to automate (remotely automated power control for one) but are the life blood of most of the current LAVA instances. > Not sure if you want more insights here, but I'll give them (as maybe, > others will happily read). Feel free to skip if you have no time. > > On 03/30/2016 02:55 AM, Steve McIntyre wrote: > > We could do something like that quite readily, I hope. Neil Williams > > has already started setting up a Debian instance of LAVA [1] which > > is designed to do exactly this kind of thing! See lave.debian.net if > > you're interested. > > > > [1] https://wiki.linaro.org/LAVA > > I was at his presentation of LAVA in Portland. Things have moved on in the meantime, I've submitted a request for another LAVA talk in Cape Town. > As long as I get a new > reseted Debian on which I can ssh to, then I can do the rest and > install OpenStack on it, and test a Debian image. Yes. Your LAVA job would download the image to use for the base VM, add some support scripts that include files from git repositories which you maintain, boot that VM and give you multiple automated SSH connections into it. Each connection then runs it's own test scripts. (In the documentation, this method is termed Secondary Connections.) Once inside the VM, the test writer has a free hand. (If you wanted to reboot the VM that would need to be handled by LAVA rather than simply killing the VM.) Note: interactive SSH sessions (hacking sessions) are supported too but require some network configuration to allow you to access the machine. With a local box that is trivial, for a hosted box it requires permission to modify the firewall. > Otherwise, a VM > that can be rebooted (and reinstalled from scratch?) with some kind > of API will do. Maybe LAVA can provide such an API? Using LAVA to build images tends to not make the best use of LAVA - Jenkins is usually better suited to running the debootstrap wrapper you choose for your image build. Rebooting a VM needs to be done in the test job rather than by scripts in the test itself as LAVA is monitoring the VM to determine whether the job has gone into a tailspin and needs to be killed. > Here's an example of a test run of OpenStack and its functional test > suite, called Tempest: > https://mitaka-jessie.pkgs.mirantis.com/job/openstack-tempest-ci/64/consoleFull > > As you can see, this is a test using Jessie, and using the *next* > version of OpenStack (called Mitaka), to be released in a few days, > and currently living in Experimental. How many of the dependencies live in the base VM image is up to the test writer but it would make tests much faster if the build scripts created a VM image with things like mysql-server-core pre-installed. (Speed is a useful goal if the tests are to be commit-based to be able to get a result back to the developer within a usable timeframe.) This is a similar job used within the LAVA software team for our own test suite: https://staging.validation.linaro.org/scheduler/job/143951/complete_log?debug=on§ion=test The results of the job can be seen here: https://staging.validation.linaro.org/results/143951/lava-server-pipeline-unit-tests This is one of the longer jobs we use as running this test takes a fair amount of preparation. The test took 18 minutes. We also run local tests using VM images with the base packages pre-installed. This is a test on an ARMv8 board using secondary connections: https://staging.validation.linaro.org/scheduler/job/141505.0 (It doesn't launch a VM, it provides multiple ssh connections to a bare metal device running a user-specified NFS root filesystem.) This is the kind of test that just cannot be done in a build-based system or single VM. One thing I will mention, if a particular device is not available on lava.debian.net, it can be added. In the meantime, check one of the other instances of LAVA - if that is validation.linaro.org or staging.validation.linaro.org, there is a process by which community members can submit jobs to those instances. https://staging.validation.linaro.org/static/docs/v2/development.html#register-with-linaro-as-a-community-contributor Linaro is also looking at bare-metal ARMv8 testing of cloud-type systems but that involves a large investment in hardware and infrastructure. > I'll have to setup something using Stretch instead, and whatever > version of OpenStack that will be in it (maybe Mitaka, maybe the next > one called Newton). It shouldn't be too hard to do in Stretch, > provided there's no new added bugs, which I don't think there is, as > I can already run all unit tests at build time (this env. will only > run functional tests). > > This currently uses the Cirros image, because it's smaller (a few > megs), so it's faster to test OpenStack. In the context of testing > the Debian image, I could switch to the Debian image: it's only 1 or > 2 lines to change, as this was what I was doing to begin with. LAVA is in the midst of a large migration from LAVA V1 to LAVA V2. lava.debian.net only runs V2 test jobs and tracks the packages as uploaded to jessie-backports. Now is the right time to provide input of your use-cases, your Test Plans and what you want to do with a system which can provide automated access to VMs and bare metal. Steve has the role of coordinating lots of use cases from teams in Linaro and from the community. So talk to us, take a look at what can currently be done and help us improve the documentation. Let's expand tests within Debian from in-build tests and package-based tests to tests which are not constrained within a single system and not constrained by pre-configured base configuration. -- Neil Williams ============= http://www.linux.codehelp.co.uk/
Attachment:
pgpM3Tlkm6ejO.pgp
Description: OpenPGP digital signature