Re: Debian snapshot.debian.org replica on the Amazon cloud?

[James Bromberger <james@rcpt.to>]

On 5/12/2013 1:54 AM, Peter Palfrader wrote:

I wonder if we can somehow, somewhere tag files that we got from non-US
or archive.d.o (which also covers non-US) and no other tree.  Somebody
would have to write code for that.

Do you have the source of where the files originated from somewhere? We can add arbitrary metadata as headers to objects if we want to. That metadata tag gets served back as headers when you get/head the object (file).

I wonder if we can replicate to a running postgres instance.  If not, we
might have to feed it individually, importing the dumps that the master
produces.  Thoughts?

A dump from the current master would be a good start. What size are they
(is it the 2.1 GB file I saw in there)? Peter, would you like the
credentials for this DB (also in US-East right now)? If so, can you give
me an IPv4 you'll be accessing it from?

I'm not sure I can make use of DB access right now, thanks.  When we
still had a mirror at UBC, we used postgresql's DB replication feature
to keep that mirror in sync.  Is that an option with this instance?

Not right now - the replication that is supported is currently wholly within the AWS environment - Multi-AZ is the feature, synchronous block level replication from host in one cluster of data-centers (Availability Zone - AZ), to a standby host in the second AZ.

OTOH, we may not necessarily need a DB at amazon.  It should certainly
be possible to seperate backend hosts from frontend from database hosts.

Absolutely - completely your choice.


See attached for an untested example to 'restore' a given file that has been archived; this is not a working example, just an initial sketch. The user credentials I sent you (offlist, naturally) for data ingest to S3 does not have access to call this restore at this time; we'll do a separate user with only access to restore (and not ingest) that can be in this script - but the concept is easy:
* We'll default to restoring a file to live storage for 14 days (after which the duplicate in lice storage is automatically removed)
* We'll limit to doing 100 file restores per day from archive


Feel free to edit parameters as above - but hopefully it shows you how this plugs together.

  James
(Going to bed now - 12:30am now here at AWST+0800)

--
Mobile: +61 422 166 708, Email: james_AT_rcpt.to

#!/usr/bin/python
# vi: ft=python
restore_time_days = 14
restores_per_day = 100
bucket_name = aws.snapshot.debian.org
simpledb_table = snapshot.debian.org

import datetime
import boto.sdb
from boto.s3.connection import S3Connection
s3_conn = S3Connection('<aws access key>', '<aws secret key>')
bucket = s3_conn.create_bucket(bucket_name)
from boto.s3.key import Key
sdb_conn.create_domain(simpledb_table)


def check_file_is_archived(file_name):
  if bucket.list(prefix=file_name).storage_class = "GLACIER"
    return 1
  return 0

def check_file_already_being_restored(file_name);
  if bucket.get_key(file_name).ongoing_restore
    return 1
  return 0

def check_daily_restores():
  today = datetime.datetime.today()
  tomorrow = datetime.date.today() + datetime.timedelta(days=1)
  sdb_domain = sdb_conn.get_domain(simpledb_table)
  todays_sdb_data = sdb_domain.get_item(today.day)
  return True if todays_sbd_data.count < restores_per_day

def restore_file(file_name, days):
  key = bucket.get_key(file_name)
  key.restore(days)  

def update_daily_restores():
  today = datetime.datetime.today()
  sdb_domain = sdb_conn.get_domain(simpledb_table)
  todays_sdb_data = sdb_domain.get_item(today.day)
  if todays_sdb_data.today != today
    todays_sdb_data.count=1
    todays_sdb_data.today = today
  else
    todays_sdb_data.count++
  todays_sdb_data.save()

def process(file_name): 
  if (not check_file_is_archived(file_name))
    return "File is not achived."
  else if (check_file_already_being_restored(file_name))
    return "File is already being restored - please wait 3 - 5 hours from the inital restore"
  else if not check_daily_restores
    return "Too many restores have been done today - come back tomorrow"
  else
    restore_file(file_name, restore_time_days)
    update_daily_restores()
    return "Your file has been scheduled for restore - please try and access it in 3 - 5 hours"

def main():
	# If this is web accessible, probably drop some HTML in here...
	print process()