Re: Geographically Diverse file-replication
On 3 Aug 2009, at 15:55, lists@truthisfreedom.org.uk wrote:
I'm working on something at the moment where I will require file
replication across geographically-diverse networks.
Initially this will probably be multiple sites within the UK,
however it is hoped that it will expand to global datacentres.
We're currently using NetApp 6080's to serve up the files and I'm
wondering if anyone can recommend a good way of syncing the data
(nearly 5 terabytes) so that updates made in one location are
distributed to all the others in the shortest time possible.
Hi, --
We have built stuff like this before and done it differently depending
on
- the user facing service that is going to be in front of this
infrastructure,
- how often the files change,
- and how you are going to originate the content.
The common mechanism for our customers is that they want to serve up
files via HTTP for web services of some kind, they want to use anycast
or dns trickery (read: hackery!) to approximate the location of the
users, and the files might change, say, once an hour, but they might
be created at any time.
We have done this by sitting reverse proxies in locations close to
their users, which are configured to have an understanding of where
content is originated. They then fetch and cache files on these
devices close to end users.
This is a DIY content delivery network (CDN) of sorts. Sometimes our
customers pass this to established CDNs like Limelight or Akamai,
sometimes they do not because they have a strongly defined geographic
scope of users and building their own is cheaper. Sometimes they want
to build their own because they like to use open source tools.
This advice is not useful if you are not shoving HTTP service in front
of these files of course. :-) The advantage of proxying this content
is that you only need to replicate the data which is in use.
Feel free to reply on or off list or give me a phone call.
--
Regards, Andy Davidson +44 (0)20 7993 1700 www.netsumo.com
NetSumo Ltd, Specialist networks consultancy for ISPs, Whitelabel
24/7 NOC
/* Opinions are my own and & may not constitute policy of any org I
work for */
Reply to: