Infrequently Noted

Alex Russell on browsers, standards, and the process of progress.

Dojo Needs Your Projector (and room, and network, and...)! (updated)

The votes have just come in, and the next set of Dojo Developer Days will be in the San Francisco Bay Area Feb 7-8 or 8-9, but as of now we don't have a venue.

In years past, IBM and AOL have graciously hosted these events, provided network connections and projectors, and generally made us feel at home.

This is where you come in! If you're working at a Bay Area company who is using or otherwise benefits from the work we're doing in Dojo and can spare a room for 20-30 people (with working network) for a couple of days, this is a great chance for you to meet the community of folks building the toolkit, put faces to names, and do your bit to help ensure that Dojo continues to succeed.

If you can spare some space on either of those sets of dates (Feb 7-8 or 8-9), please send me mail.

Update: Our friends at Google and Mozilla have both generously offered their spaces, and so it looks the Feb DDD will be in Mt View. I'll update this post once more once dates and locations are final. Thanks Google/Mozilla!

Update 2: Thanks to Google, Dojo Dev Day will be Feb 7-8th at Google's Mt. View campus. The agenda is being constructed here, and we can use all the help and feedback we can get on what to discuss, particularly if you plan on attending. It's always hard trying to address every concern at these meetings, so now's the time to make your voice heard! We'll be posting logistics details and the RSVP address to that page in the next couple of days.

This DDD is going to rock! Can't wait to see everyone there.

How IE Mangles The Design Of JavaScript Libraries

A lot of hyperbole gets thrown around about how painful IE 6 and 7 make the world of JS development, and so I thought I'd do a bit of cataloging to help those using Dojo understand why it's built the way it is and indeed, why all JS widget libraries suffer similar design warts. I know the good folks up in Redmond are working hard at delivering something better, but the fact of the matter remains that until they outline when we're going to get it (and the version after) and how it's going to be distributed, IE 8 only serves to make the current situation look as shabby as it really is. Here are but 5 examples of how IE makes your toolkit of choice less elegant than it probably should be.

  1. Array's Can't Be Usefully Subclassed (test case)

    At first blush, this seems wrong. You can use the Array base-class as the prototypal delegate for any user-defined class you wish. Methods are correctly delegated to and hash-style indexes work fine. Almost everything works right...except when you try to use the built-in array manipulation methods like push, pop, and shift. They dutifully change the internal contents of the subclass instance's indexed attributes, but they don't manipulate the length property. This means that while you can use for(var x in list){ ... style iteration, you can't do anything *aside from key iteration* to know how many items are in the array. Obviously, one could try to wrap the intrinsic functions and detect how they manipulate the length property, but then you've ruined their [DontEnum] status and now they end up in the iterable surface area of instances. Ugg.

    Arrays without a working length property are nearly useless, and JScript mangles the design of toolkits as a result.

    So how do we get dojo.NodeList to be a "real" array with extra methods then?

    As you might expect, it's a giant hack. When you use the "new" keyword with the dojo.NodeList function, you expect that the system will create a new instance and do it's normal "stamp with a constructor" business. Instead, we resort to creating (and populating) a regular Array instance and "NodeList-ifying" it by copying named attributes from the class prototype into the instance as member properties. The "constructor" function then explicitly return a new object, bypassing the "new" keyword's create/stamp machinery, at which point the return of the new operator becomes our explicit return and not the object which it would have otherwise implicitly returned.

    In Dojo 0.9 we had used an even more aggressively hackish workaround for IE which involved creating a sub-document and mapping its interpreter's intrinsic Array class into the parent document at a different name. Both are slow for different reasons but we eventually switched to the create-and-mix-in style because some popup blockers were interfering with the old method.

    Lest you think that Dojo is a dirty, dirty toolkit for doing this kind of thing, consider the janktastic "it's not really an array" thing that JQuery resorts to instead. By giving up all "[]" index operations, JQuery manually maintains it's internal length property by re-implementing all of push, pop, etc. functions. This has the benefit of allowing prototypal delegation to work of pre-existing instances when new features are added to the base prototype, but at the expense of no longer being able to think about a dense list of things as an array. Dojo's approach is painful, but so are all the alternatives today.

    I think it's safe to say that both toolkits would subclass Array directly to save code, were it a reasonable thing to do.

  2. Where Art Thou Getters/Setters?

    As JavaScript toolkits get pushed out of their current workhorse tasks of plastering over JavaScript and DOM implementation gaffes by positive browser progress and Moore's Law, they increasingly take on application construction tasks (e.g., As the toolkits have approached these tasks, we've collectively started to hit some very serious usability limitations due, in large part, to JScript's lack of progress.

    Toolkits like Dojo have widget systems because HTML just hasn't kept up. That means that these toolkits have a responsibility to keep as many of the positive aspects of the "native" web as they can. From CSS customization to accessibility all the way through implementing declarative creation and DOM-ish JavaScript APIs, the better a job a toolkit can do in making the abstraction feel more solid, the better the toolkit is. Widgets are essentially a way to "subclass HTML elements".

    In many places, DOM allows you to affect the behavior of the visible document using "setter"-style syntax. For example:

    document.getElementById("foo").style.border = "5px dotted black";

    Custom widget classes can have the same behavior on every browser except IE.

    This means, of course, that JavaScript toolkits can't really implement the behavior, backing JavaScript programmers up against a wall when they design their tools. Instead of providing the natural property-oriented behavior, it forces class authors to write getSomeProperty/setSomeProperty method pairs on their classes should they want to do anything when values are gotten or set. The resulting code feels a lot more like Java than JavaScript, which is usually a sign that something is horribly wrong in a browser.

    As bad as this problem is for visual widgets, it's worse for data binding systems. API's like would be designed in fundamentally different ways if getters and setters were available everywhere. Instead of the rigamarole of making users fetch an item reference and then fetch attribute values using the opaque item handle and the property name, we'd just set up getters and setters on the properties themselves and defer the implementation of fetching those values down to the data store. Further, assigning a linkage between a query or store and a widget which represents it could be as simple as assigning a property to the widget object.

    So are workarounds to this possible? I think they are, and I'm testing some of them out for use in Dojo 1.1 right now. I'll post more about them should they pan out. Every avenue which looks potentially workable right now involves gigantic hacks which also deeply constrain API designs. Fundamentally, this problem can't be solved without good language-level support.

    It's perhaps folly to assume that this will be addressed in IE 8, but given the enormous back-pressure of nearly every JavaScript toolkit author demanding this feature and the embarrassment of every other browser beating them to the punch, I have some hope that we could see getters and setters for JScript in the near future. It won't matter much, though, unless the JScript team ships their new engine to all IE versions when they release IE 8. Not bloody likely.

  3. Performance

    Kudos are in order to the JScript team for fixing their long-b0rken GC heuristic and pushing it out to everyone...but it's the tip of the iceberg.

    Performance is one of those areas where differences in implementations can tightly circumscribe what's possible despite exacting spec conformance. On this front, JScript's raw VM-level execution time leaves a lot to be desired, but the true travesties really show up when you hit the DOM for computed style information or try to do anything reasonably complicated that involves string operations.

    Most non-trivial blocks of JS code today rely on innerHTML to bootstrap some new chunk of DOM in response to user action due in large part to the cross-browser speed and size advantages of innerHTML vs. raw DOM methods for equivalent DOM structures. This reality pushes IE's string performance woes to the fore as more and more client-side systems push far enough to hit the new "wall".

    Similarly, getting computed box model calculations out of IE is not for the faint of optimization foo. When we profiled Dojo's widgets to plan our attack for 0.9 and 1.0, we noted very quickly that getting box-model data out of the browser for any element is hugely costly on every browser, but on IE the cost was not just big... it was enormous. Our best guess right now is that the properties on the currentStyle property are re-calculated when they're requested from script and not cached in the bound object when layout happens. The resulting performance penalty requires that developers nearly never manage layout in code, severely constraining the layouts which are attempted by toolkits like Dojo.

    Across the board, from DOM performance to raw JScript execution speed, IE is a dog, and the odds are good that whatever toolkit you're using spends a lot of time working around that reality.

  4. Doctype Switching

    Doctype switching to toggle box-model behavior is perhaps the single most limiting implementation error in IE. Saner browsers allow you to use a CSS property to affect the layout model in use in a particular section of a document. This makes tons of sense in a templated world where most of the markup your system generates starts and ends in the "news hole". Today, that covers nearly everyone. A quick line count in any HTML document shows that doctypes are a scarce resource whose scarcity is made problematic when it's semantics are overloading. I've been on product teams where the idea of changing the doctype would require months to recover from. That kind of cost related to what should be simply a markup dialect change (not a formatting policy change) implies strongly that the doctype is a terribly brittle way to control several independent concerns.

    Instead of giving devs fine-grained layout system control, IE makes it all-or-nothing. The global flag approach backs toolkit developers into doing script-based layout calculations or "just throw it in another div" solutions where we'd really rather not. Both are slow and both may be required since it's completely impractical to dictate to users which doctype they'll be using. While any app may be able to be disciplined enough to not care, toolkit developers must work everywhere. Hilarity ensues.

    I fear this is going to get even worse with IE8 as the IE team looks to implement some of HTML 5 and hopefully many of CSS 2.1's clarifications. The sooner they abandon the global switch, the better...but I'll wager it's pain they just don't feel. Building a browser is a very different pursuit from building portable apps to run inside it.

  5. HTC's Can't Be Inlined (Even With Hacks)

    Modern browsers have built-in widget systems. On IE, it's HTCs + Viewlink and on Firefox it's XBL. Even a cursory reading through the docks for both is enough to illuminate the gigantic overlap. Alas, no one is yelling at them to standardize and the result is a terrible mess in which both sub-optimal formats limp along with nearly zero Open Web usage.

    So why do I single out IE for whipping here when XBL is just as lame and similarly b0rken with regards to single-file embedding? Well, on Mozilla, you have a lot more "outs". I strongly suspect that you can use "data:" urls to generate and evaluate component definitions for FF, which would enable compiling down from a single (more sane) format in the running page environment. IE prevents any such useful code-loading approaches, meaning you have to generate files on disk in their b0rken-ass format in order to be able to use them. Given how far we've gotten with non-builtin widget systems, it's pretty clear that toolkit authors aren't going to contort themselves into requiring a build step that splats files all over disk just so that we can give IE and Mozilla different views of the same component description. Instead, we all limp along on our own class hierarchies without any of the benefits of element subclassing, getter/setter handling, and inline (scoped) method description that these browser-provided systems would allow. It's kind of pitiful, really.

    IE8 may include strong "data:" URL support given that it's needed to pass Acid2, but I'm not holding my breath. I strongly suspect that HTC's are a dark, unloved corner of the Trident codebase that none of the current engineering team are really fired up about fixing (which they could have done by just allowing Data Islands to contain HTC definitions...but I digress). The takeaway here is that we probably shouldn't even need JS toolkits to build widget systems, but it's too late now. We're an abstraction short and a decade late and now the Flex frankenstein is beating HTML at it's own game.

In a vacuum of feature data or builds to work with, it's prudent to assume that IE 8's DOM and JavaScript implementations will continue to warp attempts at building useful, idiomatic JavaScript libraries to ease the problems that HTML + CSS aren't effectively solving. So next time you wonder why your toolkit of choice is built the way it is or why it's even necessary, just remember that in many cases they are protecting you from a decade or more of bad decision making.

From that perspective, they're worth every penny.

Kris Zyp Joins SitePen

Some time back I posted about an opening in SitePen's R&D "group". The responses I got back were astounding, and I couldn't be happier that Kris Zyp is joining me to help work on things that we feel are important to the future of the open web. This is yet another high point in our recent string of great hires. SitePen has never been a better place to work, and that's saying something.

The W3C Cannot Save Us

Things are finally moving over in CSS-land. On the positive side CSS column layouts are looking pretty nice, having dropped their dependency on the the janktastic "advanced" layout module and there's some initial movement on improving the CSS-OM.

But all is not well, nor has it been for a long, long time. No work on hbox or vbox or mixins/inheritance. There's also no indication that the WG has taken stabs exposing the expressive power that Microsoft exposed with CSS expressions.

More importantly, Andy Clarke is pretty disgusted by what he sees of the process and participants and so, apparently, is David Baron. Anyone who has met David in person will probably understand how much of a big deal this really is. It signals the effective end of the CSS WG as we (don't) know it. Rebuilding credibility in the WG is going to be much, much harder now that Mozilla's representative has effectively given up on the closed-door process. The working group's secret cabal style of operation is imploding. It was inevitable, but the timing is still a surprise.

But why was it inevitable? And should we take Andy's suggestion seriously and expect a re-chartered WG to do better? After thinking about it for a while, I think the answer is that we can't expect any standards body to do what is being asked of the CSS WG; namely to invent the future by committee. It's particularly unreasonable to expect progress when deployed browsers provide so much momentum that the safest thing to do is to solve the little problems and ignore the large ones. This is also a well-worn stalling tactic used by vendors playing for time as to marginalize the standards process or solidify their lead such that it becomes a de-facto standard. This is a good reason to reject closed working groups, but the web standards to which advocates now cling were developed in those same closed systems. Closed clearly can't be all bad nor can it be a primary cause for the lack of progress. We seem to be hugely confused and conflicted about the relationship between standards, vendors, and how we get to a better future.

Before I lay out all the forces involved, let me first say what everyone knows but few truly accept:

In order for the future to be better by a large amount, it must be different by a large amount.

I think that statement alone is enough to indict Opera's anti-trust actions as stupid and ill-considered. But we should also recognize that it forms the basis of Opera's grievances. We should all be pissed off that the discussion today hinges on how we will get MSIE to improve by slight degrees (or should we expect more?). Opera could have done better, though, by shipping Gears and working with Google to make it a host for pluggable Opera. Too bad Opera has prioritized proving a point over actually improving the situation. But I digress.

So what was so different about the late 90's that it allowed a closed process to make huge gains in a short order while we can't even get basic architectural issues addressed in a timely fashion today?

The first major reason is that web developers in the 90's were looking forward, not backward. I remember being excited about getting the chance to use new features and not caring who gave them to us. As a community, web developers hadn't collectively "picked sides". I think the market as a whole still has that essential optimism built in, it's just that the more that self-identified web developers focus on how standards compliant things are (or aren't), the more we lose the sense that it can get better. So platforms not tied to standards race ahead. Why is anyone surprised that Adobe has essentially kicked HTML's ass with Flex or that Microsoft feels it can do the same with Silverlight in a couple of revs? If you buy either Adobe or MS's lines about how these platforms aren't there to replace HTML, I'd like to sell you some prime property in the Pacific.

But, I hear you yelling, it sucked to build things back then! It was hard! We didn't know what bits of the technology we could use portably, and the W3C saved us from that mess!

Oh really?

Try teaching a good programmer without a web background to build anything reasonably sophisticated with web technologies today. Doing so will teach you painfully, embarrassingly that there are huge tracts of the HTML, CSS, and DOM specs that you simply can't use. IE's bugs combined with its market share conspire to ensure that's true, and we wouldn't get off the hook should IE 8 magically transform into a perfect reference implementation. Mozilla, Opera, and Safari all have their own warts as we get to the edges of what's theoretically possible with current specs. And that's not even taking into account how utterly wrong, broken, and silent the specs are in several key areas. Even if the specs were great, we'd still be gated by the adoption of new renderers.

It's clear then that vendors in the market are the ones who deploy new technologies which improve the situation. The W3C has the authority to standardize things, but vendors have all the power when it comes to actually making those things available. Ten years ago, we counted on vendors introducing new and awesome things into the wild and then we yelled at them to go standardize. It mostly worked. Today, we yell at the standards bodies to introduce new and awesome things and yell at the browser vendors to implement them, regardless of how unrealistic they may be. It doesn't work. See the problem?

Andy is probably wrong to suggest that a new working group (no matter the structure) can succeed in fixing the impasses of the existing CSS WG if only because no working group at the W3C has the power to effectively and definitively introduce new things into the market. Only browser vendors can do that and no amount of buy-in at the W3C can force vendors to implement. The last decade is proof enough of that. We need to wise up to this key point: standards bodies are downstream of implementations, and that's the only time when they work well.

Until we get some great new (non-standard) CSS features out Mozilla, Opera, and IE nothing will get better to the extent that we will again be optimistic about the future (Safari earns a pass). The size of the improvements they deliver in the future are directly tied to our expectations of how different the future will be. Only when there are large and divergent ideas of how to proceed expressed through competing, successful implementations will standardization really work to whatever extent that it can reasonably be expected to.

Let that sink in a bit. To get a better future, not only do we need a return to "the browser wars", we need to applaud and use the hell out of "non-standard" features until such time as there's a standard to cover equivalent functionality. Non-standard features are the future, and suggesting that they are somehow "bad" is to work against your own self-interest.

Web developers everywhere need to start burning their standards advocacy literature and start telling their browser vendors to give them the new shiny. Do we want things to work the same everywhere? Of course, but we've got plenty of proof to suggest that only healthy browser competition is going to get us there. Restructuring the CSS WG or expecting IE8 to be "fully standards compliant" is a fools game.

Put simply, Zeldman is hurting you and only you can make it stop. Neither the CSS WG nor the HTML 5 WG nor, indeed, any W3C working group can define the future. They can only round off the sharp edges once the future becomes the past and that's all we should ever expect of them. As much as they tell us (and themselves) that they can, and as much as they really would like to, the W3C cannot save us.

Update: the implosion continues apace. Hixie outs Microsoft's rep as stalling on embeddable fonts and runs up the flag on why he thinks it's just not working in general. Interestingly, one of the MS reps has seemingly seconded dbarons "lets do this in public" sentiment as Hixie smacks down Bert Bos's leadership of the CSS WG. It's easy to get caught up in the horse race on this, but I want to again suggest that none of this will get better until the browsers start taking risks. Hopefully openness and transparency will allow the world to judge the actions of the WG until then without removing the ability for the WG to function when the cavalry does arrive.

The Non-Relational DB Strikes Back!

When I started at Jot, one of the things I fell most in love with about the platform was the way that application developers on the system never, ever had to think about "the database". You just had nodes (JS objects which could serialize themselves to XML) and nodes had properties. Setting a property on the node persisted it and created a new version of the node.

Instead of thinking about how you were going to represent your problem in objects and then figuring out some way to map them to an RDBMS, you just started figuring out your problem in code, letting the normal development cycle iterations of what a "thing" is continue without stopping after every change to babysit the database or perform some sort of lame (or brittle) "migration". Recently I've been excited to see this kind of work start to evolve in the Django world while aspects of the non-relational data store have been finding more mindshare through projects like CouchDB and ERlang's built-in Mnesia persistence layer, although they all still feel relatively primitive in comparison to the "experimentation is free" environment that Jot offered. Sometimes folks ask my why I don't get into RoR, and every time I look into it again I'm alway struck how....backward it is. Hopefully the rumored gemstone port to ruby will plug up some of the remaining conceptual leaks that the RDBMS addiction has tortured the RoR and DJango app development process with.

Adding to the non-RDBMS data storage action is the announcement by Amazon of their SimpleDB service. It shares some of the best features of the Jot model (easy key/value setting, no schema, query anything) but doesn't yet seem to have the ability to version individual records. Even if SimpleDB doesn't do it, I expect it to pop up in another form somewhere else soon.

I'm tremendously excited about these sorts of services and data stores. It's been clear for some time that most data storage tasks for even departmental applications are main-memory tasks. It'll be interesting to see how the language environments respond to these changes. Microsoft's LINQ integration into .NET languages is the first major stab in this direction, and I expect the next up-and-coming language will probably develop something similar in order to one-up Java and Ruby by making "schema evolution" look more like adding properties to an object or a class prototype (in JS parlance).

Hopefully soon all of this work will soon yield a web framework for general consumption that will show the rest of the world what Jot got dead right: that when your data and your program can evolve in harmony and without friction or risk, you are truly liberated. When storage is free (and it nearly is), "screwing up" should mean starting a fire in your data center. Everything else is just a version rollback.

Older Posts

Newer Posts