Re: GSoC project: fedmsg for the Debian infrastructure
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On 25/04/13 18:07, Simon Chopin wrote:
> Quoting Daniel Pocock (2013-04-25 17:34:03)
>> ZeroMQ is a very lightweight solution - it is brokerless (like
>> multicast) so won't necessarily support the requirement for
>> durable subscriptions (keeping messages queued up for clients
>> that are disconnected)
>> http://www.zeromq.org/topics:requirements-for-reliability
> As I mentionned in my original email (granted, without expanding
> much on it), I plan to implement a mechanism to solve this very
> problem by keeping the messages sent on the emitter side and
> resending them on demand.
That seems like extra work to do something that can already be done in
a standard way, and it also seems like it is just transferring the
scalability problems from the brokers to the message sources.
>> Some things that are worth looking at:
>>
>> - do we want to use an AMQP broker? In theory, this is an open
>> standard like SMTP: the clients and brokers are interchangeable
>
> This solution has been already examined by the Fedora folks, and
> rejected by lack of scalability WRT the broker. This sort of
> question is actually why I think it is a good idea to use fedmsg
> because its authors have needs that are quite similar to ours, and
> virtually the same use cases.
Red Hat promotes a number of messaging solutions, I've used several of
these things commercially - they publish a very interesting roadmap[1]:
- - HornetQ "new ultra high performance enterprise grade messaging"
- - MRG Messaging (based on Qpid) "Exploits Linux-specific optimizations
for performance"
- - Fuse MQ (based on ActiveMQ)
- - JBoss XQ Messaging
and I'm surprised that they ruled them all out and went for ZeroMQ
OpenStack is working with RabbitMQ, it is based on a broker paradigm too
My own feeling is that brokers do scale to some extent: if reliable
delivery is a requirement, then you just have to buy the right
hardware to run the broker at scale.
>> - as an alternative, could we use something like SIP or XMPP as
>> a messaging platform? Then people can receive messages in their
>> chat client. In this case it doesn't appear to be the optimal
>> solution, but these protocols are quite good for systems that are
>> very widely distributed over public networks.
> Correct me if I'm wrong, but isn't SIP specific to VoIP? I don't
> know much about XMPP outside of its application in chatting.
In practice, that is how they are used, but they are interchangeable
for both purposes. (and please see [2], feedback needed!)
SIP supports methods such as MESSAGE, SUBSCRIBE and NOTIFY which could
provide some very interesting ways to distribute data to
intermittently connected developers in all the odd locations where we
choose to have our workstations.
These protocols may not be at the core of the solution, they could
just be an external interface.
On 25/04/13 19:13, Nicolas Dandrimont wrote:
> * Daniel Pocock <daniel@pocock.com.au> [2013-04-25 17:34:03
> +0200]:
>> - then again, there are plenty of examples of systems like Apache
>> Camel that can take a message from a traditional broker and
>> deliver it to
just
>> about anything: SIP, XMPP, email, IRC channel, syslog, Nagios +
>> 200 other possible destinations:
>> http://camel.apache.org/components.html and it can use Java or
>> XML to describe various filters and transforms
>
> Those are interesting suggestions, I actually had a look at Camel
when it was
> suggested to us on another thread, but we will keep going on with
fedmsg.
I didn't necessarily suggest that Camel would replace fedmsg. I have
used it in much larger environments where it has been the core
component for mediating messaging workflows, running multiple
instances in parallel off a HA message broker. As such, I'm confident
it could scale well enough to replace fedmsg, but that is not what I'm
arguing - I'm just suggesting that it is a useful tool. fedmsg could
well be the core of the solution, and a couple of Camel instances
could be set up to implement custom workflows and deliver messages (or
derivatives of the messages) over transports that fedmsg doesn't
natively support.
> I wanted Simon to start this thread to gather ideas: I know some
people, mostly
> in DSA, have thought about such a system for some time, so this
> was
more of a
> heads-up than wanting to reconsider the project from scratch. It
landed on
> -devel because, well, we couldn't find a more appropriate list :)
>
> To get back to your remark, it would definitely be possible to
> build a fedmsg-to-xmpp filtering bridge, if there is interest, but
> that's
not really
> the scope of this GSoC project.
I agree the GSoC scope needs to be contained, but it would also be
good to have a wider architecture in mind and understand how things
fit together, so please don't feel I'm hijacking that - I just saw the
request for comments and was keen to help.
fedmsg may well be the best way forward, but it would be interesting
to know if we can make the messaging platform interchangeable, perhaps
to support the durable subscriptions. There is one project,
OpenMAMA[3], that aims to allow applications to use interchangeable
middleware for messaging, it is an official Linux Foundation project
and it is designed for low-latency market data, so in theory it would
be better than having a direct dependency on 0MQ. That could well be
a GSoC project on it's own.
And just to add to the list of messaging APIs, I came across
Disruptor[4] the other day
1.
http://www.redhat.com/promo/jboss_integration_week_sessions/pdf/MessagingRoadmap_10-16-12.pdf
2. http://datatracker.ietf.org/doc/draft-ivov-xmpp-cusax/
3. http://www.openmama.org/
4. http://lmax-exchange.github.io/disruptor/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQIcBAEBCAAGBQJReXxVAAoJEOm1uwJp1aqDLS8QAJumdY7d9YYJfXTvfevJp9au
vqKy3dpx9hAS6ke/oRHuAGWR29yikuyNvPxjbPsgX13thlIoz1IMWZBI7h9F4U32
5xYShTrTsSSBu1t6Xmud3AIZppOcHTn4I7/Cft0+nzaPGX3W8ygmVl+q5TucbZVC
qva9XklpU8b6zNNd/VYHCaw5hKqsi5YTcD2zpdhWbab7icmiZiHGAkiq9hjnhL5X
BxLk7Y6f7mlR1vay/IK6I6a4ShbbX9XQXD95sDVF7ufwlE0vxN30YCFJjhy0nY14
/0tfw9YTjktnsSuq7rs4A8ru8447KS22anTXQBMCVsBuOyi09tCFGSqdMDjR60lM
0+VvMbvzbHRAIJxJp3GXi0xANqAnL3V17FbCr+pLsJi1BPl6ml9q0sJNGLJ892zE
FYiLZcgsiVoumlE61dMa4wtbqa4jw76Vr00d8Y8xjlP7FVUiv8FmS/QDiUaG5fxH
Vk5aUBcRSLzEM21nVMDpYrjb/rqQk94indV7ek1lKrVJKr9H9LY8fx4ckwVmDR/k
c012M7tmTv8t1KNBMMiRidupgvGMz84tvuzRQRlRyslgmmUNH2kEYwpDnFnn6dcZ
2jYz3q46ANw6Y091aNDlJMtU5K/qc6HZx5ptuEuNDyQGj+ZwZBG2OleG00TBr2yS
HxyHPe0YmRizKja+Wi0o
=nu96
-----END PGP SIGNATURE-----
Reply to: