Infrequently Noted

Here are the slides from my talk today at FOSDEM. Looking forward to seeing you there!

I'm sitting at home feverishly working on new Wiki features in preparation for my ass being out of the proverbial hot seat during FOSDEM, and what shows up in my well-worn maildir? The shipment notification for my new MacBook Pro.

I use my laptop for nearly everything and I've been frustrated with the speed for a while. The macworld keynote wasn't even over when, with some fellow uber-dorks from work, I trundled down University Ave. to see if I could get my hands on one of these things. 2 cores, each faster than my current CPU? Yes, please. Now. Freaking right now!!!

Needless to say, that didn't work. Who knew that they Apple stores don't get shipments of stuff until it actually ships? We all skulked back to work laughing nervously at ourselves, all kind of thankful that we didn't have to explain 3+ thousand dollars of new hardware to our respective S/O's. Not that I'd let a little setback like that stop me.

As soon as the Apple Store (online variant) come gasping back to life, my order was in. RAM: maxed. Hard drive: spindle speed, baby! I think I might have actually signed away my vote in the next presidential election in the process. I sure as hell wouldn't have noticed.

So it's finally shipping. I should be ecstatic, right? Kinda.

Problem is that I'm also an unrepentant Unix dork. This little Powerbook has so many configuration tweaks and external utilities installed that transferring to another system will probably take me the better part of a week. Last OS update, my mail client alone took two days to resurrect. All of this means I can't really trust new hardware enough to take it to conferences yet, even if it does arrive before I leave.

It's a cock tease in the form of a laptop.

Aaron says I need to work on my slug sentences, and I agree. On reflection, I suck at writing almost any kind of sentence. I guess I shouldn't expect slugs to be different.

For a couple of years I've been akwardly saying things like "and I would suspect that JavaScript is the most successful scripting language ever" or "I don't really have any hard numbers, but JavaScript seems incredibly successful as a language". Note all the hedging.

What I would like to know is this: on what axis is JavaScript not the most successful scripting language of all time? Can some other language beat JS in:

• implementations of a ratified language spec (scheme? lisp? do they even "count"?)
• interpreters deployed (Forth is in every Apple and SUN bios...can that possibly beat the distribution of IE, Firefox, and Safari?)
• lines of code written for said language

Perhaps KSH and/or bash would win? I'm curious to know if anyone can think up other strong contenders.

I've uploaded updated renderings (pdf or png) of the document I use to keep track of the non-trivial, public DHTML toolkit efforts that have occurred over the years.

Interestingly, the difference between this version and previous ones is that many companies are starting to either release or talk about tools that they had quietly built in-house years ago. Also, Google and Yahoo have been on something of a hiring binge. Yahoo seems to be dedicating more (visible) resources to their responsive UI cause than Google. The secrecy difference between the two cultures might explain the delta, but I'm convinced that's not the whole story.

Also interesting is that many of the commercial DHTML toolkit vendors have been able to effectively keep mum about who is working on their products. The TIBCO's, Backbase's, and Bindows of the world must be paying their people astoundingly well for them to keep out of eyesight of the increasingly frenzied recruiters who are banging down the doors of every competent JavaScript hacker I know.

I will admit to not having kept this document as up-to-date as it should be. The proliferation of toolkits over the last year has been pretty astounding, and investigating each one to find out if it's just another crappy 10-line XMLHTTP wrapper hasn't been on the top of my list of things to do. So in true open-source style, your help is requested! If you have corrections or new information, please either mail them to me or submit a patch to the graphviz source file. The format is straightforward and easy to figure out.

So the new GTalk interface in GMail is pretty rad. Congrats to Dan and the rest of the team that made it "go".

The talk feature is cool not just from a UI perspective as the code is also chock full of little gems. I'm kind of a dork about low-latency data transport to the browser. HTTP wasn't meant to be used this way...so of course I'm interested! Ever since Joyce got me involved in the rewrite of mod_pubsub I've had my eye on the various ways that servers can push data to browsers and the kinds of technology that will prevent a server that's doing this from melting down (hellooooooooo Twisted). Using just what's available to the browser, it's possible to have the server push data encapsulated in <script> blocks and rely on a progressive rendering behavior that every modern browser implements to dispatch events in near real-time (compared to full page refresh or polling delay). There are a mountain of browser quirks that of course play into this process. The least desirable of these to the user are the "phantom click" and the "throbber of doom" that afflict IE users.

When a page (or an iframe it hosts) is loading content, your browser usually shows some sort of "I'm working" indicator. In the bottom "taskbar" there is usually some sort of progress meter. In the upper right (on IE) the "throbber" will continue to animate until the work is done. Of course in the scenario I'm describing the sent page is never done. The whole point is that the server keeps the connection open. Combine this with the IE behavior of producing a "click" like sound when an iframe is navigated to a different URL, and you've got a pretty poor user experience.

But couldn't you do something with XMLHTTP? Short answer: yes, but not as portably and it won't get you around IE's 2-connection limit either so there's not much of a win. For the long answer, see my talk at ETech or wait for me to post the slides. At the end of the day, the hidden <iframe> hack scales best and is the most portable. Especially if you can lick the UX problems.

Which Google has.

How? By cleverly abusing another safe-for-scripting ActiveX control in IE. Here's the basic structure of the hack:

  // we were served from child.example.com but
  // have already set document.domain to example.com
  var currentDomain = "http://exmaple.com/";
  var dataStreamUrl = currentDomain+"path/to/server.cgi";
  var transferDoc = new ActiveXObject("htmlfile"); // !?!
  // make sure it's really scriptable
  transferDoc.open();
  transferDoc.write("<html>");
  transferDoc.write("<script>document.domain='"+currentDomain+"';</script>");
  transferDoc.write("</html>");
  transferDoc.close();
  // set the iframe up to call the server for data
  var ifrDiv = transferDoc.createElement("div");
  transferDoc.appendChild(ifrDiv);
  // start communicating
  ifrDiv.innerHTML = "<iframe src='"+dataStreamUrl+"'></iframe>";

This is the kind of fundamental technique that is critical to making the next generation of interactive experiences a reality. Server tools like mod_pubsub and LivePage (and perhaps even JMS buses) are starting to come into their own and the benefits of event-driven IO are starting to become well understood by server-side devs. It's only a matter of time before server-push data hits an inflection point in the same way that background single-request/single-response data transfer did with Ajax. Dojo will, of course, have infrastructure to support this kind of thing when the borader developer community is ready (most if it is already in place).

From long and painful experience and amazingly deep respect, I take my hat off and bow to whoever it was on the GMail/GTalk team that figured this out. It's a hell of a hack. It's no wonder that Google has been able to attract and develop the best DHTML hackers in the world.

Update: so just to be very clear, I worked on the rewrite of the mod_pubsub client. The server rewrite was handled by some folks who are much smarter than I am.