Re: SSL for screenshots.debian.net?
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 12.02.2014 21:21, Peter Palfrader wrote:
> On Wed, 12 Feb 2014, Christoph Haas wrote:
>
>>> Have you given any consideration into making this service
>>> mirrorable? I.e. have one master instance through which updates
>>> are added, and more hosts that can then serve the content to
>>> end-users?
>>
>> Yes, I have. The challenge would mainly be the storage backends.
>> PostgreSQL has been my favorite database. But replication was an
>> issue as all mirrors would have to be able to write and send
>> their updates to other servers. Besides the actual image (PNG)
>> blobs would have to be copied, too. So I wondered if I have to
>> implement replication on an application layer.
>
>> Simple mirrors would work in a master/slave fashion. But that is
>> not the concept of using a common screenshot database for
>> multiple distributions.
>
> I was thinking of redirecting all udpates to
> https://screenshots-master.debian.org/, but serve the "normal"
> queries from https://screenshots.debian.org/. The -master is on a
> single host, the screenshots site could be replicated, i.e. a round
> robin of several machines.
>
> If you use pg and a filesystem as your storage backends, we can
> replicate the database pretty much instantly - the non-master
> clients will only need read-only access. The filesystem tree could
> just be replicated using rsync - for bugs.d.o we use a inotify
> based rsyncing strategy that runs an rsync over new/changed files
> within the minute they happen.
Thanks for sharing your ideas. Sending updates to a single server is
indeed an interesting idea. Timely updates are not really an issue. If
someone uploads a screenshot it does not have to visible instantly.
(That's why I even considered MongoDB because the application does not
require ACID.) It's indeed a simple mechanism that is easy to
understand and operate. Just as you said… "SPOF" instantly popped up
in my brain. :)
I will try to make the implementation open enough to either use
pg/rsync replication or some other mean like application-based
replication. The only decent alternative to pg seems to be Cassandra
and I'm not sure I want to ride that horse.
As you don't mind providing a service for other free distributions as
well the distribution issue surely won't a problem at this stage.
Getting a successor version up and running needs a higher priority
than winning the nobel price for the fanciest replication. :) As long
as it can be implemented later without throwing everything away. The
current application is based on a deprecated Python framework and I
just hope that I get it up and running on Wheezy for the next months -
currently it runs on Squeeze.
> This is, of course, not fully replicated as the -master hist is
> still a SPOF. But if that goes down we only lose the ability to
> add new screenshots, not the ability to serve them. Also, it seems
> like it ought to be relatively simple and straight forward, but
> then I don't know the intricate details of screenshots.d.n.
Oh, the application is nothing magical really. It gets Packages.bz2
files from various DEB-based distributions (currently
Ubuntu-integration is just a dirty hack that needs a proper
implementation) and imports the metadata into a pg database. Users can
then browse the site for screenshots by names and debtags (although
for some reason the debtags support seems increasingly broken and I
haven't found the cause yet) and upload screenshots if they are
missing. And of course it features HTTP-based APIs for
packages.debian.org and packages.ubuntu.com as well for Synaptic and
the Ubuntu Software Center application with some smart logic in the
backend for dealing with different versions and distributions.
I will keep that basic concept in the rewrite. However I will use an
actively maintained Ruby framework based on Rack so that the operation
should be easy for the next 5+ years. And the major feature is support
for multiple distributions to get more screenshots. Also I'm taking
better care of performance issues like delivering static assets and
caching. Other social features like commenting on packages are second
class. For the curious… planning takes place here:
https://trello.com/b/hRpf5XUQ/projekt-debshots
Actually I'm still surprised of the success. screenshots.debian.net
was a fun project and now delivers 40 screenshots/sec. So it's worth
taking it serious in my opinion. It's very motivating seeing your
support for the idea. Thanks.
…Christoph
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlL73YAACgkQCV53xXnMZYb5pgCglj11mDu1/qlI24CIrjt5YVUJ
/d8AoL5C2II5nK67dxxMzlUdwcjR7cLl
=UlJb
-----END PGP SIGNATURE-----
Reply to: