Infrequently Noted

Alex Russell on browsers, standards, and the process of progress.

Cometd: The Long Tail of Bad Puns

Thanks to a premature Ajaxian mention there's some real wind in the sails of Cometd, a primordial little Comet server and protocol project that I'm lucky enough to be working on. The goal of the project is to produce a content-level protocol for publish/subscribe event notification down to browsers. We need a name for the protocol (as distinct from the code), so if you have ideas, we're all ears.

Along with the protocol, we're producing multiple server implementations and at least one JavaScript client library.

Unlike some predecessors, however, a couple of neat features should help to make Cometd clients and servers resilient in the face of browser and network stupidity. The first of these features is "transport negotiation". At the core of it, there's a realization that browsers very rarely all do the same thing the same way...especially when you're out pushing on the lightly tested bits like we are. We get around this by having the client and server advertise what they want to speak over, and if they can come to some agreement, the conversation starts. Since the protocol is so simple, this generally means that servers can implement almost every type of transport with exactly the same code. The Twisted Python server does exactly this. This is very interesting because it allows us to implement most of the known Comet techniques with very little overhead. In the last 2 days I've implemented 3 different styles (forever-frame, multi-part-mime, and long-polling) which cover most of the modern browsers with at least one transport method. Each method exhibits different bugs, browser compatibility headaches, and performance characteristics, but being able to swap out one technique cheaply for another is going to let us experiment that much faster with how best to implement Comet systems.

If there's a secret sauce to Cometd it's that both of the initial implementations are being based on async network responder frameworks. On the Perl side, Perlbal is doing the heavy lifting while the Python variant relies on Twisted. Obviously there's a lot more to these things than picking a good library, but it sure helps. There's even been some discussion of how Java, PHP, and Erlang implementations could be made to scale using similar techniques.

Hopefully we'll get a new name for the protocol bits of Cometd soon (and a draft of the spec shortly thereafter) and once the Python server does topic and client pruning I'll try to get some demos hooked up. After all, it's the sexy collaboration that sells this stuff.

Update: seems I forgot to mention that in addition to the mailing list, the Cometd project also has one of them newfangled "blog" thingers.


It's been too long since I've written a non-trivial amount of Python and for the last couple of days I've been spending some time to look around and see what got better and what still sucks in Python land. The good news is that a lot of things have gotten better, including the language itself. Sadly, Python is still missing a clean way to represent enclosed scope, which seems totally out of kilter for a language that is otherwise so dynamic. Sure, you return a named throw-away function from another function in order to generate a closure, but you can't create an anonymous variant of the same thing. Also, Python's lambda function is still crippled, to my great chagrin. JavaScript's function and Ruby's blocks still have Python by the balls on this one.

Also, I'm not sold on decorators. The burgeoning catastrophe that is Java 5 annotations should be an object lesson, but I'm afraid good taste hasn't prevailed in either language yet. But whatever quibbles I might have with various syntactic forms in modern Python, it's still the best thing going. Idiomatic Python remains readable, audit-able, and fast.

Here's the short list of the toolchain I'm assembling for an upcoming project:

While the greenlet project looks like an interesting alternative to Twisted, it doesn't look "baked" enough to handle most of the situations I'll be relying on Twisted for. In much the same way that library depth is the only reason I keep inflicting Java on myself, protocol implementation breadth and depth are the reasons that Twisted probably has very long legs, despite the kludgyness of requiring factory classes every time you want to sneeze.

I'm tremendously excited about lxml. It implements the ElementTree API but extends it with all kinds of XPath and XSLT goodies that make doing XML in Python much less painful than the cluster-fuck that is 4suite and PyXML. Our long XML processing nightmare may finally be over, no surprise, thanks to libxml/libxslt.

Is there anything that I've totally missed in the last 2+ years of Python development that I shouldn't be going without?


Somewhere there must be written an engineering mantra that states something along the lines of "knowing that a thing can be done liberates the mind to do it".

Is there a name for this "law" of software engineering?

Phobos builds available

I've mentioned Phobos before, but until now you couldn't get code. Seems that builds are now available for the adventurous to play with.


So I'm procrastinating (I mean...erm....thinking) on some code for a talk I'm giving in a couple of days, and as usual, I wind up doing mental gymnastics on every topic but the one at hand when I'm staring down a deadline. This time, I can't get my head off of mobile apps.

At a conference a couple of weeks ago, I got to see a version of KHTML on a phone, and for the first time in a long while my interest in doing mobile app development has been rekindled. The Series60 Python port was good, but I think that this several orders of magnitude more important.

Back when J2ME as something new, I got to spending a summer trying to get phones to do enough crypto to pass themselves off as smart cards. Now, this shouldn't be hard, if only because most phones include actual smart cards. Problem is that the carriers force device manufacturers to seal off access to all of the interesting hardware on cell phones from apps running on them. We couldn't get access to the real crypto co-processors already embedded on the phones. These closed platform antics of carriers and operators are in part to enforce monopolistic control of the user base, in part to exert cartel-like control over the handset manufacturers, and in part to enforce monopolistic control of the upstart application and content authors who they need so desperately but can't bring themselves to treat as anything other parasites.

They're the phone company, after all.

While I can't say much more than that about the project, I can say that we were able to get SSL working from a phone in 2001 without hardware or OS-level software access. Which only served to teach me how utterly evil telcos are. With the proof-of-concept project done, it came time to ask ourselves how we'd get our app into the hands (literally) of people that could benefit from it. The stunningly bad answer was "you don't". At least not without lawyers, businessfolk, and a small mountain of cash. There are two major "platforms" for building apps that get delivered to phones: BREW and J2ME. Both suck.

Carriers who are sufficiently in bed with Qualcomm to have built CDMA networks are that much more likely to be enforcing BREW as their app delivery "solution". The system works something like this:

Now, admittedly, BREW is worst-case-scenario for mobile app deployment, but things are hardly much better over in J2ME-land. MIDP 1.0? MIDP 2.0? CLDC? KJava? How in the hell is anyone supposed to target a "platform" when there are so many permutations and versions of these castrated "profiles" with vendor-specific extensions hanging off the edges all over the place? It's not like Java is a forgiving environment when the right API isn't available at the right time. That wasn't part of the spec. Well, ok, to be fair, it wasn't part of any of the myriad applicable specs, not that you'd be able to find that out easily. And that's not even addressing getting the app onto the phone if the device supports some variety of J2ME. Carriers that ship J2ME-enabled handsets aren't somehow more enlightened when it comes to demanding their pound of proverbial flesh. They just don't have a unified system with which to extract it.

And that, in a nutshell, is why doing mobile phone development is the software equivalent of the life of Job. Just when you think it got bad, you look over and your cattle are dying of famine and the APIs you were counting on have withered on the vine of carrier apathy and manufacturer helplessness.

Which is why the web is our trojan horse (again). This is gonna sound like old news, but remember when there was this entrenched monopoly that controlled what kinds of apps could succeed (or be bought and then re-introduced) and which would be crushed under the fist of the almighty marketing dollar and vaporware announcement? The web routed around some part of that damage. Open Source did too, but it depends much more heavily on the concept of a liberated hardware platform, which is why RMS is fighting so darned hard to keep "trusted computing" out of the bootstrap of commodity computing devices via GPLv3. I don't agree with RMS most of the time, but I think his fundamental understanding of the problem regarding closed hardware and Free Software is correct.

Which is why the web is doubly important to mobile devices. Instead of being inert data, it turns out that the web has all of the characteristics of a programming environment that can succeed in the face of device foibles, carrier antics, and handset manufacturer impotence. WEP failed and the heir apparent is the web as we know it today.

If we can get the UI figured out.

I've been noodling a bit on the problem of "stackable" interfaces for mobile devices. It seems to me that for as good as the KHTML demo I saw was, it lacks the ability to easily build spatial or temporal associations with a piece of data in a page. The S60 KHTML makes it easy to relate pages to each other, but it's hard to scan a page for important information to dig into if everything is "below the fold" or if the page is zoomed out so far that you can really only make out visual landmarks. What we need is a way to promote things that are essentially "headlines" of the page into that spot in a remixed "folded" view of the page. This stack or list of "headlines" might represent Augment's outline view or a stack of cards with a fast "flip" function, but the content has to be the regular-old-web. It's one applications where I see micro-formats or a Dojo-style extra-attribute syntax as being essential. Instead of air-headed &lt:div> tags to separate things out such that CSS can have it's pretty way with the content, it would be much better if sections of a page which contained related data could be marked as such. The developer incentive to do this would be that page sections that do this would be "hoisted" into the stack whereas those that don't could be ignored by a browser (or script) that creates this sort of UI. This way, advertising content could be placed in the stack along with regular content, keeping everyone happy and still giving users a better search-and-drill experience than they currently have.

All while reducing carriers to the glorified purveyors of bandwidth that they should have been relegated to years ago. The web is our end-run. Our fulcrum.

I suspect we haven't even begun to see the start of the browser wars that will actually matter for most of the world's population. Add some limited local storage to a remixable, stackable web interface, throw in bluetooth or proximity networking and you have village-level networking. Negroponte's immoral gamble is likely to fail at the hands of good-old worse-is-better web tech. And the best part is that it starts with kicking the telcos in the groin here at home.

Either that, or I need to start getting more sleep.

Older Posts

Newer Posts