I’ve been concerned a bit that it may appear as though Joe joining SitePen may carry with it the perception that the Dojo Foundation is an arm of SitePen in some way or that DWR will now need to become Dojo-centric. Luckily neither is the case, although reading assurances to that effect on this blog should be taken with a grain of salt. The Foundation has an open door for deserving projects which need a good legal umbrella and don’t want a lot of process or formality, and we’ve extended personal invitations to many non-Dojo-centric projects over the years to join (including direct competitors).

For anyone still worried about the DWR/DF arrangement, working it backwards from both perspectives should yield some comfort. It does very little good for Dojo to be shoved down someone’s throat by the choice of some orthogonal tool in the same way that it would be tremendously foolhardy for the Foundation to lose its independence in any way. Neither would be very meritocratic and in particular the independence of the Foundation and its track record of providing a level playing field is most of what it has going for it. To that end, putting DWR at the Foundation and not under the getahead legal entity should help to make the distinction between commercial interest and community development even clearer than it previously was.

An umbrella organization to help support projects in meeting their goals is the under-appreciated bit of what separates successful stand-alone projects from projects which can’t escape the yolk of either a closed development process or a sponsoring company that just can’t let go of assumed control or brand affinity. Having your work backed by a legal entity is essential if you’re not going to pick a GPL-ish license, but having that entity be a brand-neutral known quantity helps get projects adopted by organizations which have both lawyers and some experience with OSS. For component software like DWR, Cometd, and Dojo that’s nearly all of the users which an OSS license alone won’t convince. The independent nature of the Foundation also isolates users from the employment decisions of key contributors. Through the Foundation, Dojo’s licensing has survived wholly in-tact through changes in employment status of nearly all of its most prolific contributors.

Having “external” committers, a reasonable process for minting new ones, and a peaceful atmosphere where developers can build trust along with software isn’t rocket science but it does require some amount of prioritization of those concerns over personal and corporate directives. We’re not perfect at this at the Dojo Foundation, and we haven’t yet encountered the wrenching attempts at “brand drafting” which Apache deals with on a regular basis, but we’re committed to keeping the process as hands-off as we can and working to ensure that the umbrella doesn’t imply more than it really provides.

Our door is open. All that we require of new projects is that all committers on the project sign a CLA, that the lineage of the code be “clean” (within reason), that the project community is relatively healthy, and that you can convince the existing committers that yours is worthy of Foundation support. It has always been the intent of the Foundation to host projects that have nothing to do whatsoever with Dojo and it’s my sincere hope that the DWR announcement drives this home, but we’re not going to leave it to chance. More on this in a forthcoming post.

Until then, my congrats again to Joe and the entire DWR community. Thanks for giving the Foundation the opportunity to make good on the trust you’ve invested in us.

Michael Carter (of Orbited) walks us through the details of getting the beautiful htmlfile hack to work. This is new and novel information which is useful to all implementers of Comet clients and servers, so hats off to him on his sleuthing and persistence.

Lastly, Joe Walker leads off with as post that does one of the clearest jobs of explaining why Comet is inevitable that I’ve seen.

It should be noted, first, that these issues are designed to be small, (relatively) easily implemented point solutions to accute problems. They are intentionally not on the scale of “add a new tag for…” some feature or “standardize and implement XBL” or even “make renderers pluggable”. While those would all be good, the current pace of browser progress makes me think they’re beyond the pale.

This list also tries to avoid vendor-specific issues (of which there are a pile, and many of them may be much more pressing on a per-browser basis than the below issues). Lastly, I’m also not asking for standardization of these things in short order (although it’s clearly preferable). We DHTML hackers are hearty folk. We’ll take whatever they give us, but we could deliver much, much better developer and end-user experiences if only the browser vendors would all give us:

• Event Opacity: we need a way to tell browsers that for some nodes in the z-index stack that events should be passed through to their background. For instance, when implementing drag-and-drop, Ajax libraries have a stark choice: attempt to calculate the locations of all drop targets to enable dragging of the source with offset (expensive hitbox calculations, ahoy!) or move the drag shadow with an offset from the cursor (efficient and what we do in 0.9, but visually unsatisfying). Yes, yes, there are WHATWG specs for drag-and-drop and various browser vendors have implemented them to some extent for some time, but what I’m asking for here is much more constrained: a single API extension that can help us deliver better experiences until all the browsers get DnD right. There are also lots of other use cases where visual decorations should be able to defer their events to underlying elements (think drop shadows, etc.)
• Long-Lived Connections: this was on my IE7 list, but it’s still a problem nearly everywhere. The basic issue is that we can’t implement Comet reliably because if two tabs both keep a long-lived connection to the same server, no other connections can open up to that server, meaning that normal Ajax (and even style changes) will be blocked. Very often this means that an app will appear to be locked up. One solution is to provide a way to specify in an HTTP header that a connection will be long lived or to provide pages a way to request more concurrent connections be made available to a particular server (on a per-tab basis, and with a hard limit, of course). If feasible, it might just suffice to just break the global lock connection limits and make them per-tab in general. Whatever the solution, we need ways to be able to have multiple tabs each able to create Comet connections without worrying.
• Expose [DontEnum] To Library Authors: The contortions that Ajax toolkit vendors go through to keep iteration over JavaScript objects and primitives coherent is, quite simply, insane. Much of Dojo, in particular, is designed around this problem. Giving us a way to say that a property is [DontEnum], before ES4 lands, would go a long way toward alleviating this back pressure.
• Fast LiveCollection -> Array Transforms: That many DOM apis return live collections is a bug, but it need not be fatal. Browser vendors could start to provide a simple toArray() method on these live collections to provide a way to “fix” them in place.
• Provided A Blessed Cache For Ajax Libraries: Long story short, Ajax libraries are going to be here for a while. The idea that a browser is going to be so good that it will remove the need for a JS library simply doesn’t hold water any more, and even if one browser was that good, the other browsers wouldn’t be and none of them would be pervasive enough given current upgrade trends. We need to be able to better support our own patches to the browsing platform, and we need the browser vendors to get on board and realize that Ajax toolkits aren’t a threat. We can’t be…we don’t have enough leverage. With all of that being true, it’s high time for the browser vendors to provide Ajax toolkit vendors a way to specify a canonical URL or hash scheme which would bypass the network entirely, cross-site. This is something of an extension to the CDN concept for Ajax toolkits, but would go a long way to fundamentally changing the way Ajax toolkits evolve. Instead of fretting about how much 10K on the wire is going to degrade the user experience, we can focus on delivering better and better tool sets, even behind the firewall or offline (where CDN usage isn’t feasible).
  
  Obviously, this one is going to require some vendor coordination, but it’s the kind of thing where if one vendor does it (well), everyone else should follow quickly without much risk. The Open Ajax Alliance could even function as a body to provide a list of toolkits and hashes to browser vendors should they demure from the task themselves. Lastly, before the flames start rolling in on this topic, I should note that I’m proposing this with some hesitation. Who are we (the Ajax toolkits) to suggest that our content deserves a more “blessed” cache position than site content? I’ve been wrestling with this for a long time, but now believe that we don’t have much of a choice. This solution is good for everyone and while it has the potential to create an uneven playing filed, I think that can be handled at an organizational level.
• Mutation Events: The browsers already know when a new item is added to a DOM, why can’t they tell us, the poor toolkit/framework authors? I’m not going to suggest in this list that browser vendors should fully figure out HTML tag subclassing since it will generally require architecture changes for the least capable browsers (*cough* IE *cough*). Instead, point solutions like mutation events everywhere will go a long way to allowing us to further band-aid their brokenness and allow us to more seamlessly upgrade content while we wait for the new-tag cavalry to arrive.
• onLayoutComplete: onDomReady doesn’t cut it. Toolkits that want to avoid FOUC when applying behaviors and progressive enhancement to pages are currently attempting to get into the page rendering stream as early as possible. The problem is that for anything that needs to manage layout of widgets on the page, we need to know the dimensions, and that also must mean that CSS has been applied and any initial flow computations have been completed. Obviously, there are issues with progressive rendering of a page, but generally speaking I beleive toolkits are looking to browser vendors for a semantic that is roughly equivalent to “after onDomReady, but potentially before all images have finished loading, inform us when the layout and geometry have stabilized.”
• HttpOnly cookies: There’s a lot wrong with WebAppSec these days, and the traditional trust boundaries are constantly under attack. Worse, none of the browser vendors seem to feel it’s their responsibility to figure out cross-domain or JS sandboxing. This infuriating state of affairs leads directly to my next item, but the minimum any browser vendor should be required to do is to implement HttpOnly cookies. It’s no silver bullet, but it’s another tool in the toolbox and one we need badly.
• Bundle Gears: Until it’s primary APIs are put through the standardization process and introduced in browsers natively, any browser that includes Flash support should, out of good-faith respect for the Open Source and Open Web nature of its intent, bundle Google Gears as soon as it’s stable. A commitment to do so in the future will suffice until that time.
• Standardize on the Firebug API’s: Firebug provides great debugging and performance profiling APIs. These need to be built in so that we can stop shipping Firebug Lite around the net ad-infinitum (as we do in Dojo 0.9). Being able to have built in timing tic/tock apis and a UI to view it would be a hugely useful. This falls far short of other proposals that have been floated for unified debugging APIs and protocols, but again represents the least vendors can do to alleviate the pain.

So that’s my list…what’s on yours? What am I forgetting? And how should we organize to ask the vendors for these in a way that will really stick?

• SitePen begins to offer Dojo Training to the public! Signup here. Since we employ a huge percentage of the committer base and fund significant new development on the toolkit you’ll know you’re getting it “from the horse’s mouth”.
• Shane O’Sullivan of IBM has made a first public release of his GUI Dojo build tool.
• The second part in the SitePen performance blog series is up (part 1).
• Dojo 0.4.1 blew past the 100K download mark (now at ~180K) and continues its march to becoming the most successful Dojo release ever
• And not to tease too much, but there’s stuff coming down the pike that I’m tremendously excited about. More news on that after 3D2.

Multipart is attractive because it provides a way of avoiding TCP set-up and tear-down for each and every event across the channel. While it’s not significant overhead (comparatively), being able to also reduce the number of HTTP header blocks sent can also help out when it comes to wringing latency out of the system. The code indicates that multipart is supported on Safari and Mozilla, but while events are indeed delivered at the right times on Safari, you can’t get at the payload until the connection closes completely. Not useful.

Things were looking better on Firefox and it was the preferred transport type in the Dojo client, but I think that’s going to have to change. Sadly, it seems we can’t actually tell when a multipart connection has failed. In “normal” XHR requests, the 200 HTTP status code plus a “finished” readystate indicates that the contents of the request can be read and control handed back. In the multipart case, each successful block fires of a load handler and resets the readystate. That means that the combination of readystate and and status can’t be used to differentiate between block success and connection success. Making matters worse, server-side connection failure doesn’t fire any kind of readystatechange handler, and even if it did, it doesn’t appear to be possible to determine if the connection is closed from any of the public properties on the object.

So, OK, what about falling back to a timer that restarts the connection every N seconds for good measure? This might work in cases like failover where a lag of 10 to 30 seconds might be acceptable but not for normal operation. Should events be flow regularly, it might never be necessary to hit this “backstop”. Not great, but I gave it a try, only to discover that Firefox won’t give you responseText of an XHR request if the connection is marked as multipart but the response isn’t a 200 and wrapped in a multipart block. Since we’re trying to use HTTP status codes correctly and keep the server internals from needing to fork significantly for each pluggable transport, it’s something of a step backward to need this kind of hand-holding.

I’d still like to support the multipart transport type, but until at least one of the implementations becomes rational for use from the XHR object, I think I’m going to just be commenting this transport out in cometd.js. Like XHR itself, it’s one to mark down for resurrection a year or two from now. At least we still have good enough options in the mean time.

Bayeux is a JSON-based protocol for clients to register interest in events and for servers to deliver them in a more timely fashion than Ajax-based polling allows. The goals of the Bayeux spec so far have been to:

• make event delivery fast
• keep it simple
• provide extension points in the protocol

As a result we’ve variously rejected plans that involved implementing some subset of XMPP in JSON or actually tunneling XMPP over some other carrier. JSON libraries are available almost everywhere and you can’t beat the speed of eval for message deserialization on the client. It keeps things fast, simple, and pluggable. Authentication and authorization, while being given spots in the protocol, have also been left entirely to implementations to negotiate. The result has been that implementations have gotten off the ground very fast. Since the protocol doesn’t specify any guarantees around durability, entirely ephemeral event busses can be as conformant as MOM-style guaranteed-delivery systems.

So what, then is Cometd and how does it relate to this Bayeux specification? The short answer is that Cometd is a project to provide several reference implementations of Bayeux and to evolve the spec until it’s good enough for some other group to take over maintenance. Bayeux and Cometd are two complimentary parts of a plan to tackle the complexity around building and deploying comet apps, and we’re hoping for many independent Bayeux implementations outside of the Cometd project. For more interactive apps to finally take off, we need a set of portable concepts about how this kind of message passing should be handled in much the same ways that Ajax could be thought of as “RESTian web services for browsers”. Today lots of folks are thinking about event busses under the buzzword banner of SOA, but hopefully in the same way that REST was the lighter but easier variant of SOAP for doing request-response web services, Bayeux can be the less-sophisticated (but easier to use) alternative to exotic schemes for connecting MQSeries and TIBCO to browsers.

In other news, the Cometd project has been accepted into the Dojo Foundation. With multiple demonstrated implementations of the pre-0.1 Bayeux spec, we’ve got lots of momentum around solving the “last mile” problem for events to browser. Work on the spec has been bursty but I think that’s contributed to keeping it simple. Right now we’re tackling topic/channel globbing, event timeouts, and multiple event delivery semantics. Perhaps the most exciting thing to Bayeux to me, though, is that implementations of it lend themselves to extension and experimentation with implementing new styles of Comet transport mechanisms. The current Bayeux client in Dojo supports 4 different transport types with more on the way.

I think it’s going to be interesting to see how much the ability of platforms and languages to handle Comet workloads impairs or enhances their chances. Can Ruby and PHP pull off something that’ll allow them to scale? Will Twisted Python, Erlang, or full-stack Perl finally have their days in the webdev sun? Will great implementations in Java and C# hold the dynamic languages at bay?

I can’t wait to find out.

“Cross domain Ajax” (see JSONP) hasn’t taken off the way that I’d hoped, perhaps in part because of a lack of necessity. It might just be easier to set up a permissive proxy to do requests to services hosted on other domains. Moore’s Law discounts distributed coordination in favor of localized transformation. But whatever the reason, there hasn’t been huge uptake, even with beautiful abstractions like Dojo’s JSON-RPC facilities (see the Yahoo services example). Intensely useful, but not everyone is on board just yet. Hopefully Joseph Smarr‘s talk on the topic at OSCON will help turn some heads.

But there’s one scenario where it’s a total no-brainer: cross-domain Comet. Normally when you set up a Comet server, it either has to sit in front of the application server or have requests passed off to it from the main web server. In the case where the app server dishes off the requests, it’s potentially still “involved” in the process if it acts like a proxy. Unless your front-end web server is also built with event-based-IO in it’s core, this can really really hurt. Luckily, there’s hope on the horizon. Sun even announced that they’re baking Comet support directly into their app server. But until the web tier collectively upgrades, we need some other way to simplify the configuration of apps+Comet servers. Until now, help has come in the form of iframes plus document.domain setting. This lets you put the Comet server on a sub-domain and still pass event payloads back-and-forth between parent document and the iframe. This is how the tried-and-true mod_pubsub client operates, but it’s brittle. Browsers get picky when you try to set document.domain and things don’t line up exactly the right way. Add Safari’s non-deterministic iframe behaviors to the mix and it’s a recipe for sleepless nights and begging forgiveness from the ops folks when you need just one more DNS change and server restart.

We can do better.

The solution to this that Juggernaut employs is to use the built-in cross-domain capabilities of the Flash player and it’s proprietary crossdomain.xml scheme. This isn’t a bad solution, but having Flash turned off isn’t unheard of. The performance-across-the-flash-boundary problems are mostly licked these days (thanks dojo.flash!), but the complexity with getting it all set up correctly puts it in the position of “Plan B” if something better comes along. And we just might have something better in the form of <script src="...">.

By building on top of the Dojo ScriptSrcIO infrastructure it was trivial to make a JSONP transport for Cometd. We just adapt the long-poll style of Comet but make the initial handshake messages JSONP-clued. That in place, the rest works just like “normal” long-polling, except that the client can connect from *any domain*. The configuration problem just dropped from “make sure everything cooperates at the systems and network level” to “make sure that the client and server speak the same protocol”…and I like solutions that put the problem in software and not external configuration or human effort. The checked-in server-side code shows how to handle it.

So what’s the downside? The first downside is that the latency characteristics for cross-domain Comet using this technique are the same as long-polling. Not bad, mind you, but not as optimal as iframe, multipart-mime, or Flash-based solutions when you have large numbers of messages being thrown at the client. Additionally, it’s more difficult to know when a connection “times out” or in some other way fails, but I think these are tractable. Having a Plan A and a Plan B for cross-domain Comet and a server that can speak both styles via pluggable transport wrappers is exciting to me. By lowering the configuration barrier, I think we’ll start to see a significant uptick in Comet adoption.

Next time: Why we need a standard protocol for this stuff, and what we’re doing about that in Cometd.

PS: as usual, big thanks go out to my employer SitePen who is funding this research and making it available under liberal Open Source licenses.

Along with the protocol, we’re producing multiple server implementations and at least one JavaScript client library.

Unlike some predecessors, however, a couple of neat features should help to make Cometd clients and servers resilient in the face of browser and network stupidity. The first of these features is “transport negotiation”. At the core of it, there’s a realization that browsers very rarely all do the same thing the same way…especially when you’re out pushing on the lightly tested bits like we are. We get around this by having the client and server advertise what they want to speak over, and if they can come to some agreement, the conversation starts. Since the protocol is so simple, this generally means that servers can implement almost every type of transport with exactly the same code. The Twisted Python server does exactly this. This is very interesting because it allows us to implement most of the known Comet techniques with very little overhead. In the last 2 days I’ve implemented 3 different styles (forever-frame, multi-part-mime, and long-polling) which cover most of the modern browsers with at least one transport method. Each method exhibits different bugs, browser compatibility headaches, and performance characteristics, but being able to swap out one technique cheaply for another is going to let us experiment that much faster with how best to implement Comet systems.

If there’s a secret sauce to Cometd it’s that both of the initial implementations are being based on async network responder frameworks. On the Perl side, Perlbal is doing the heavy lifting while the Python variant relies on Twisted. Obviously there’s a lot more to these things than picking a good library, but it sure helps. There’s even been some discussion of how Java, PHP, and Erlang implementations could be made to scale using similar techniques.

Hopefully we’ll get a new name for the protocol bits of Cometd soon (and a draft of the spec shortly thereafter) and once the Python server does topic and client pruning I’ll try to get some demos hooked up. After all, it’s the sexy collaboration that sells this stuff.

Update: seems I forgot to mention that in addition to the mailing list, the Cometd project also has one of them newfangled “blog” thingers.