Cometd: The Long Tail of Bad Puns

Thanks to a premature Ajaxian mention there’s some real wind in the sails of Cometd, a primordial little Comet server and protocol project that I’m lucky enough to be working on. The goal of the project is to produce a content-level protocol for publish/subscribe event notification down to browsers. We need a name for the protocol (as distinct from the code), so if you have ideas, we’re all ears.

Along with the protocol, we’re producing multiple server implementations and at least one JavaScript client library.

Unlike some predecessors, however, a couple of neat features should help to make Cometd clients and servers resilient in the face of browser and network stupidity. The first of these features is “transport negotiation”. At the core of it, there’s a realization that browsers very rarely all do the same thing the same way…especially when you’re out pushing on the lightly tested bits like we are. We get around this by having the client and server advertise what they want to speak over, and if they can come to some agreement, the conversation starts. Since the protocol is so simple, this generally means that servers can implement almost every type of transport with exactly the same code. The Twisted Python server does exactly this. This is very interesting because it allows us to implement most of the known Comet techniques with very little overhead. In the last 2 days I’ve implemented 3 different styles (forever-frame, multi-part-mime, and long-polling) which cover most of the modern browsers with at least one transport method. Each method exhibits different bugs, browser compatibility headaches, and performance characteristics, but being able to swap out one technique cheaply for another is going to let us experiment that much faster with how best to implement Comet systems.

If there’s a secret sauce to Cometd it’s that both of the initial implementations are being based on async network responder frameworks. On the Perl side, Perlbal is doing the heavy lifting while the Python variant relies on Twisted. Obviously there’s a lot more to these things than picking a good library, but it sure helps. There’s even been some discussion of how Java, PHP, and Erlang implementations could be made to scale using similar techniques.

Hopefully we’ll get a new name for the protocol bits of Cometd soon (and a draft of the spec shortly thereafter) and once the Python server does topic and client pruning I’ll try to get some demos hooked up. After all, it’s the sexy collaboration that sells this stuff.

Update: seems I forgot to mention that in addition to the mailing list, the Cometd project also has one of them newfangled “blog” thingers.

3 Comments

  1. grumpY!
    Posted July 19, 2006 at 9:50 am | Permalink

    interesting to see this move along, i read your slides at cometd.com. looking at the problem space you want to address with this proposal (presuming your suggested problem domains are realistic), i am wondering why there is such a pressing need to wedge this into the web stack when it appears that architectures such as xmpp appear to solve the problem in a more graceful manner, particularly with regards to the carrier protocol, for which i continue to believe http is poorly suited to solve the problems you appear to be tackling, particularly for sites that must manage high rates of traffic.

    a common response to these points i hear is that everything must reside in the browser, that new clients are to be discouraged…but this is old news, IM clients and now voip clients are nearly ubiquitous, there doesn’t seem to be great resistance to one more piece of software.

    in any case i will be following the progress of this, seems interesting.

  2. Posted July 19, 2006 at 12:13 pm | Permalink

    hey grumpY!,

    It’s funny, I always want to say “hey grumpY!!” to express genuine excitement at having someone smart and thoughtful commenting on my blog, but the double exclamation marks just feels weird. But I digress.

    We’ve been having some discussions about how much of XMPP we can “borrow”, potentially with an eye toward peering with an XMPP server to route messages. Our designs so far have been JSON-centric since it’s so much easier to work with and since it can be directly embedded in the output stream for a lot of the output scenarios we care about. But I think we’re amenable to everything up-to-and-including XMPP-in-JSON.

    HTTP might be a bad solution to these problems, but XMPP isn’t something that’s chunkwise digestible by the browsers we have today (at least not on anything but IE) and I’m not willing to accept that browsers aren’t an appropriate endpoint just yet. Users might not be resistant to another piece of software, but do you *really* want something else running on your desktop? Hell, I run Adium for IM specifically because it *reduces* the number of these apps I need to run. I kinda think this is like “convergence devices” in that people want the same capabilities in one place as opposed to being spread out, but not enough to want sucky versions of it. Anyway, Comet and cometd aren’t strictly about IM. It’s the obvious first application, but I don’t think it’s the whole story. It sure isn’t the only thing we used it for at Jot and I’m not sure that we’re using it for chat in any capacity at SitePen.

    If you accept the premise that having push to browsers is useful at some level, something has to sit in the middle even if you just want an XMPP passthrough. The XHTML extensions to XMPP sure aren’t it. Hopefully Cometd clients and servers will provide just enough structure to allow those kinds of passthroughs to get written in a portable way. If we build just that — low-latency server push passthroughs to the browser — I’ll happily call Cometd a success.

    Regards

  3. grumpY!
    Posted July 19, 2006 at 12:50 pm | Permalink

    i think your comments are sound – it is worthwhile finding out where the browser hits the wall and your efforts will certainly bring that to light. i also agree that if possible, stay in the browser. if anything, this model will act as a bridge to the type of client i believe we will be dealing with by 2010-2015 – something like secondlife that introduces tens of thousands of threads into an evnironment that it highly graphical and exceedingly concurrent (not only in terms of user perception, but true network provisioning).