Re: AWS EC2 Jessie preparations: Deprecating i386. Deprecating PVM.
On Mon, Nov 3, 2014 at 11:49 AM, James Bromberger <james@rcpt.to> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hello all,
>
> We're fast approaching the big freeze[1], so I wanted to start to summarise
> up the status of the EC2 AMIs.
>
> Thanks to Eric Evans, Python-Boto 2.33 hit testing in the last few days.
> This represents a huge step forward from Wheezy's current 2.3.0, bringing
> not only many more services, but one of my favourites, support for IAM Roles
> for EC2 instances. There is only one newer upstream release of boto (2.34),
> which unfortunately contains support for the latest AWS Region in Frankfurt
> [3], but with a freeze due now its not going to make it.
>
>
> The Jessie images I have been creating [4] have worked well for me.
> Improvements over Wheezy includes:
> - - correct usage of grub (Jessie's grub is not as broken as the 1.99 we
> have in Wheezy).
> - - SR-IOV (Enhanced Networking, low jitter), through the Jessie kernel
> 3.16-3 includes sr-iov 2.12.1 (run "modinfo ixgbevf" and look for Version),
> which works but is below the recommended 2.14.2 [5]
> - - support for multiple ENIs via DHCP
> - - support for sub-interfaces via a DHCP script (and metdata server info)
> - - AWS CLI (package awscli) version 1.4.2 (upstream is 1.5.4; differences
> are here [6]) is installed
Great work James!
> With the rise of HVM replacing PVM in the AWS environment, and every new
> instance providing more and more memory, I'm going to suggest a few things.
>
>
> First, many providers are now signalling the end of i386 images, and I am
> going to propose that we do similar for Jessie. Wheezy has supported
> multiarch[7], so i386 binaries can run on an amd64 image.
Agreed on dropping 32-bit images. They haven't been worth the trouble for some
time.
> Second, with all current-generation instance types being HVM capable, I am
> going to suggest we no longer generate Paravirtualisation images, . The
> current-generation instance types are generally cheaper and/or more resource
> (memory, CPU) than the previous-generation PVM-only instances, so users
> pretty quickly want only HVM amd64 across their entire set of instances.
There are still reasons to run the older generation instances, so I'd
be hesitant to completely drop them before AWS announces the
retirement of the HW. That said, I don't know how much work is
involved in building/maintaining them. Nor do I have inside
information on when support for the 1st generation HW will be
retired.
> This will reduce the number of images we're creating: HVM EBS backed, and
> HVM S3 backed. I suspect that simplifying this choice makes life easier for
> our users. At this stage, we haven't generated any HVM S3 backed Jessie
> images, mainly because euca2ools has a few bugs [8], and ec2-api-tools'
> ec2-bundle-image doesn't work on Jessie (as it requires a binary called
> 'grub', and that doesnt exist in Jessie as its all grub-* in Grub 2.xx on
> Jessie).
I think that as far as our users go, there is a strong and healthy
debate on what
is best, simplicity or choice. However, if it's not a lot more work to maintain
the PVM images, I'd argue that there is tangible benefit to allowing people to
utilize all available instance-types. Not doing so would put us at a competitive
disadvantage.
> Jessie (HVM, amd64, EBS) test images have been put into all AWS Regions
> (North Virginia, Northern California, Oregon, Ireland, Frankfurt, Singapore,
> Tokyo, Sydney, Singapore, and the isolated/separated Regions of Beijing and
> US Gov Cloud).
Great!
> So what's missing?
> * cfn-init, cfn-signal, cfn-hub tools for CloudFormation [9] would be one on
> my wish list.
Agreed. If I'm not mistaken they were released with a DFSG compliant license,
and packaging should be straightforward for an experienced DD?
(Apache 2.0).
> * Base image with the non-free nvidia drivers pre-installed for the g2 (GPU)
> instance type[10] - I have tested this by manually installing the
> drivers[11] using nvidia-kernel-dkms but not scripted it or produced Debian
> blessed images already containing this.
I'd think it would be concerned if we released these as "official", especially
if the drivers can be installed after instance instantiation.
> * Having a working bundle-image for S3 images for Jessie.
Ack.
> Does anyone here have any other requests?
>
>
> Now some numbers for you. Debian's AMIs are shared to users/customers in the
> EC2 environment in two ways:
> 1) From the Debian AMI Account(s) directly, what AWS's called "Community
> shared", and are the AMI IDs that I publish here. There are three accounts:
> Beijing, GovCloud, and "Standard" rest-of-world. Any DD or DM that wants
> access into the account to have a poke around is welcome to - just ask me
> off list.
> 2) The AMIs generated in US-East-1 are duplicated into the AWS Marketplace
> by the Amazon Marketplace team, and in parallel gets distributed (with
> separate AMI IDs).
>
>
> I don't have numbers on usage from 1), but from 2) I can tell you there are
> over 11,000 subscribers (customers, not instances). Furthermore, as some on
> this list have indicated, others in the EC2 environment generate their
> Community-Shared AMIs from these base Debian images, so the end number of
> instances is probably significant.
>
>
>
> On a personal note, I no longer work for AWS, and as always, patches and
> assistance is welcomed.
Good luck in future ventures!!
> James
>
>
>
>
> [1] https://lists.debian.org/debian-devel-announce/2014/09/msg00002.html
> [2]
> http://blogs.aws.amazon.com/security/post/Tx3M0IFB5XBOCQX/Granting-Permission-to-Launch-EC2-Instances-with-IAM-Roles-PassRole-Permission
> [3] http://docs.pythonboto.org/en/latest/releasenotes/v2.34.0.html
> [4] https://wiki.debian.org/Cloud/AmazonEC2Image/Jessie
> [5]
> http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html
> [6] https://github.com/aws/aws-cli/blob/develop/CHANGELOG.rst
> [7] https://wiki.debian.org/Multiarch
> [8] https://eucalyptus.atlassian.net/browse/TOOLS-294
> [9]
> http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-signal.html
> [10]
> http://aws.amazon.com/about-aws/whats-new/2013/11/04/announcing-new-amazon-ec2-gpu-instance-type/
> [11] https://wiki.debian.org/NvidiaGraphicsDrivers
> - --
> /Mobile:/ +61 422 166 708, /Email:/ james_AT_rcpt.to
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (MingW32)
>
> iQIcBAEBAgAGBQJUV7IxAAoJEK7IKHSdhcU8CYQQAJEcxldEsa/RExStNHN0Irbx
> DHPSQ/ftDUh/h/wGfCwn3kfiP4ArvqccRxOF+sQJXrg2vqaWN/h+nupUW8vcDJWN
> cTH0G92PMLqogoxxP/6a9kJoK0Q9BTjmi3HTiPKwWhM8dePBoECK4CxaNeRSNsF3
> uzVUx0Atm4p9FO69wabqEG8q//TI/iswkfZqvGwXmTKubgH29WlGQtINkmNJMgHq
> GTeyWc8WisK9b6hIYlqDNRQRkDAjEIQSp61FJFgQhDBtnKPDu2LRvcDD5JAy4bdD
> FPcDvvXJk9m0aEutN5nYn0XpRi0IQgpdqbLKr8u2oDl/nnU8y76gV10t4+uvukJW
> 0S0e5+KqlLul0zX/aXsOSyp3njIOlLQ6gLw3/w7jgtCmcM9I+mRqEcRb6jbEovzk
> RIsegSrLMd7r25SRrlBiZLxzW9XfjYldzfZlRLH6objWcYcT6wvsFLDMWmyA9Sqd
> K0EKTULldGgizVzVt+VyyfHwBIJ0oBNljjFdvAqIAXdTDSVJF9v/5/uK52vzSj7J
> Ws/7ZIu1B6ehalzq1sg55b05DUXLhTYOI1dl49kCKN68Gz+QCiIO2jwVwDuai7fG
> ssaq95noshmTqwLTOLKGKh5NYTpQe/j6Sc63wOsxeQ4lcr4MTo8wGW008O/7p+IQ
> 11A9rNB8Yyne24ZX0KJu
> =MUQg
> -----END PGP SIGNATURE-----
>
Reply to: