|
On 21/11/2013 12:37 AM, Peter Palfrader
wrote:
Be aware that snapshot includes the old debian-non-US archive, which contains things we might not be able to export from the US, once it's there. As such, the data should only be outside of the US if we want to link to it freely. Hm I'd forgotten those bad old days! ;) Where is this source now? Would you prefer to keep it outside of the US? I'm only syncing the first folder "00" (53k files, 84 GB) right now. I've set a 4 day 'archive to Glacier' policy on this bucket. All objects we transfer in will go from the standard S3 (live) to archive storage after 4 days of being ingested. This is tunable; but we want older files (less likely to be recalled) to be in the cheaper tier of storage. On demand we can initiate a pullback from Archive of those files (3-5 hours to complete) - which brings back a copy of the file into "live S3" (using the Reduced Redundancy tier of storage for this duplicate live copy) for a number of days.It's nice to have the backup, but I'm unsure how to use a backend that has that amount of latency. Ideas welcome. My thoughts was this would eventually go out to a 1 year policy - but in the short term, since everything is ingesting with a date of now() that means none of it would be in the archive tier. Perhaps even a 2 year policy - such that we reduce the number of times we're doing a recall form archive. * While this first sync is happening, we can look at the Postgres database. AWS announced last week that the managed database environment, RDS (Relational Database Service) now supports Postgres as the 4th engine - with PG 9.3.1. I have brought up one of these instances in US-East ready for this database (snapshot-prod.cjaijq7ayn5u.us-east-1.rds.amazonaws.com:5432).I wonder if we can replicate to a running postgres instance. If not, we might have to feed it individually, importing the dumps that the master produces. Thoughts? A dump from the current master would be a good start. What size are they (is it the 2.1 GB file I saw in there)? Peter, would you like the credentials for this DB (also in US-East right now)? If so, can you give me an IPv4 you'll be accessing it from? James --
Mobile: +61 422 166 708, Email: james_AT_rcpt.to |