Re: AWS EC2: Debian 7.4 AMIs now available

[James Bromberger <james@rcpt.to>]

Hi all,

Sorry for the delay in my response - been caught up teaching and just jumped across Australia today.

The 1GiB is perhaps a little too small - especially on an m1.small host with no other instance store disks. 10GiB is listed as the maximum for S3 baked instances... but my rushed testing since Sunday found that the bundling failed for this using our scripts (ie, I updated the manifest file).

if you're using the S3 backed image on anything larger than an m1.small then you may have additional ephemeral/instance-store disks that you can map (on instance launch), format and mount to their full size, but the root disk is still limited to being with the 10 GiB upper bound.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ComponentsAMIs.html#storage-for-the-root-device

If you're finding IO times on EBS are an issue, reach out to support for them to have a look; otherwise, have a look at Provisioned IOPs. It's worth noting that after a certain amount of I/O, turning on PIOPs can be cheaper than not having it turned on - so if you're doing that level of I/O its worth looking closer at if disk IO is the issue. Another option you have is to use EBS for your root file system, but use (a set of) instance store disks for your application to serve from. You may also want to RAID 1 mirror multiple instance store disks together to guard against failure of individual instance store disks (and should any fail, schedule the instance to be terminated and re-provisioned at your convenience).

Creating your own AMI should be as easy as checking out from github (https://github.com/andsens/bootstrap-vz), setting your environment variables up with a secret key, accesss key - and for an S3 backed AMi you currently also need a certificate (see IAM) - edit the manifest file (manifests/ec2-s3-....manifest) to change the size of the disk, and then run bootstrap-vz manifests/ec2-s3-....manifest)


  James

On 11/02/2014 1:01 AM, Alistair Prestidge wrote:

Thanks for that list its pretty much what I thought you would say. Glad I am not missing something with EBS volumes.

I am for the time being going to perceiver with the instance store backed AMI's and give it a go.

However I just tried to increase the size of the root partition myself for ami-80d322f7 but for some reason I cannot create an my own AMI from this ami 

the option in the aws console is greyed out any thoughts?

Also I was wondering how easy is it for me to build my own AMI using all your amazing scripts? ;)

Where do I start? I cannot find a great deal of documentation on how to create AMI from scratch.

Or I can find a lot that does not really make a lot of sense to me.

Is there a dummies guide somewhere that you could recommend?

Thanks

On 10 February 2014 08:13, Brian Gupta <brian.gupta@brandorr.com> wrote:

On Mon, Feb 10, 2014 at 2:38 AM, Alistair Prestidge
<alistair.prestidge@googlemail.com> wrote:
> Hi brain thanks for your reply.
>
> You say in almost all cases you would recommend esb backed device's. Can I
> ask why? Iam looking to migrate to instance store from ebs mainly for io
> reliability issues.
>
> I am having io issues with ebs backed instances the io can be very erratic
> and cause my sites to have very slow response times from time to time.
>
> I have currently put the code and logs on an ephemeral partition and the
> sites are much more stable.
>
> I have no reason to keep any code or data that is on the server after reboot
> or termination.
>
> Thanks

Well you may actually have a use case that justifies instance-store,
if cost is a
big concern. Typically I have found EBS more pragmatic for the
following reasons:

1) Can base infrastructure off of a single AMI, and use Block Device
parameters [1]
     to change the characteristics of the resulting root volume. (i.e.
- one can specify
     block device size and performance characteristics.)
2) One is not limited to a 10GB root volume. (1TB is EBS current limit.)
3) In the case one needs variant AMIs, it is generally much simpler
with EBS root
    volumes, since the process is "snapshot" based, and controllable
at the API layer.
4) One gets many of the benefits of a Storage Area Network, at
reasonable prices.
     (Snapshots, volume cloning, etc.)
5) Allows me to prototype on t1.micros, which don't support instance
storage, and
    bring them up as a different instance type for production.
6) Data persistence in the face of HW failure.

I like you have also observed that an instance-store root volume has
more consistent
performance than a standard EBS volume. However, the introduction of Provisioned
IOPS [2] EBS volumes has largely changed this for root volumes, as I
can now crank up
the disk performance, but it can get expensive. (Basically you can pay
extra and get an
SLA on your disk performance).

Also do be aware that EBS volumes have a first-access penalty, that
can be addressed
by prewarming your volume. [3]

Bear in mind you can use an EBS root volume, and still take advantage
of instance-
storage for non-root partitions. (For all instance types except
t1.micros, which come with
no instance storage.)

Currently I believe the best deal on AWS instances for smallish fast
storage, might be the m3
class of instances, where one of the ephemeral (instance-store) block
devices is backed by
local SSD. [4] (Whether instance-store or EBS root)

Cheers,
Brian

[1] - http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html
[2] - http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-io-characteristics.html
[3] - http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-prewarm.html
[4] - http://aws.typepad.com/aws/2014/01/aws-update-new-m3-features-reduced-ebs-prices-reduced-s3-prices.html

--

-----------------------------------------

Alistair Prestidge
alistair.prestidge@gmail.com