Having considered the question at some length, it seems to me the answer must now be “no“.

Consider IndexedDB. It’s not implemented in any browser yet, but some hard-working sould did the arduous work to specify a second synchronous version of its API which is meant to be available only in Workers where it can’t lock up the UI thread. That spec work was probably done because it was thought that a synchronous API would be nicer to use than the async version. As a result, the API is now double the size, but only in some contexts. I came across this while attempting to rework the IDB API to use Futures in order to improve usability in a backwards-compatible way.

So why not the sync version? At least 2 reasons:

• The async version doesn’t have to be thorny as the current IDB API is. The current IDB API has its issues: it uses events for one-time operations (not 0-through-N times operations). To do so it has to create non-obvious control flow contracts to give developers a place to set up listeners. Another legacy of using events is that delivered values are always wrapped in event objects and must be dug out with boilerplate. Some API conveniences can be added, and some things in the API are implicit that perhaps shouldn’t be (e.g. transactions). All of these things can be repaired in a backwards-compatible way that improves async IDB to the point of being nearly as nice to use as sync IDB. Futures can help, as can some reworking of optional arguments.
• There’s very little a worker will do that is really long running and truly synchronous. Even things that look like they’re a good fit at first blush break down quickly. Just think of the user experience: who wants their laptop fan spinning up, their lap getting warm, and their precious battery indicator slipping towards a powerless coma without being able to either find what is causing it or, better, some UI that communicates what work is being done on their behalf and a way to cancel it if necessary. Getting work off the main thread is what allows apps to do work for users without locking up those UI indicators, but to provide any ability to interrupt the task or broadcast progress, it must yield to the main event loop.

It’s this second concern that I think it truly fatal to the cause of sync worker APIs: assuming they work and are popular, they will create a world in which it’s necessary to put limits on their overall running time…limits that will be circumvented by breaking up the work into smaller chunks and dealing with it asynchronously inside the worker. Likewise, anyone building a serious app that’s trying to do the right thing by the user will factor their worker’s tasks into small enough chunks that they can both service “stop” messages and distribute progress notifications to the UI. There might be scenarios where such messages aren’t necessary and where users aren’t coveting CPUs and batteries…where sending SIGHUP doesn’t matter. But the intersection of those scenarios and the client-side web seems mostly to be a happy accident: your code might not have encountered enough data to create the problems. Yet.

This is particularly clear in the IDB cases: upgrading, iterating over, and updating hundreds of thousands of items of data is the sort of thing that will take a while, and is likely in response to some application semantic the user cares about: synchronizing mail, migrating to a faster schema layout in the process of some upgrade, etc. A blind for loop is asking for trouble. All of this might work fine in a dev environment with a (small) staging set of data…but it’s recipe for disaster when power-users with tons of data encounter it. What then? If the APIs an app depends on are all synchronous, it’s a huge boulder to roll up a hill to provide notifications, chunk up work, and refactor around async-ish patterns that chunk work up. If the work was async in the first place, the burden is much lower. So even apps that aren’t Doing It Right (TM) are likely to reap some benefit down the line from thinking in terms of async first.

There are other arguments that you can field against these sorts of APIs, particularly ones that double-up API surface area, but it doesn’t seem to me that they’re necessary. The person attempting to justify synchronous worker APIs who provides a good argument for ergonomics and learnability still has all their work ahead of them: they must show that these APIs are not harmful to the user experience. After all, Workers were added to the platform as a way of improving UX (by moving work off the main thread). And I fear they cannot do so without violating core JS semantics.

So let’s pour one out for our sync API dreams: we’re gonna miss you, control flow integration. But not for too long. Generators, iterators, and yield will see you avenged.

May the deities we’ve invented forgive us for the tripe we’re about to sell each other as “news”.

What’s bound to be missing in most of this coverage is what’s plainly said, if not in so many words, in the official blog post: going faster matters.

Not (just) code execution, but cycle times: how long does it take you to build a thing you can try out, poke at, improve, or demolish? We mere humans do better when we have directness of action. This is what Bret Victor points us towards — the inevitable constraints of our ape-derived brains. Directness of action matters, and when you’re swimming through build files for dozens of platforms you don’t work on, that’s a step away from directness. When you’re working to fix or prevent regressions you can’t test against, that’s a step away. When compiles and checkouts take too long, that’s a step away. When landing a patch in both WebKit and Chromium stretches into a multi-day dance of flags, stub implementations, and dep-rolls, that’s many steps away. And each step hurts by a more-than-constant factor.

This hit home for me when I got my first workstation refresh. I’d been working on Chrome on Windows for nearly a year in preparation for the Chrome Frame release, and all the while I’d been hesitant to ask for one of the shiny new boxes that the systems people were peddling like good-for-you-crack — who the hell was I to ask for new hardware? They just gave me this shiny many-core thing a year ago, after all. And I had a linux box besides. And a 30″ monitor. What sort of unthankful bastard asks for more? Besides, as the junior member of the team, surely somebody else should get the allocation first.

Months later they gave me one anyway. Not ungrateful, I viewed the new system with trepidation: it’d take a while to set up and I was in the middle of a marathon weekend debugging session over a crazy-tastic re-entracy bug in a GCF interaction with urlmon.dll that was blocking the GCF launch. If there was a wrong time to change horses, surely this was it. At some point it dawned that 5-10 minute link times provided enough time to start staging/configuring at the shiny i7 box.

A couple of hours later the old box was still force-heating the eerily dark, silent, 80-degree floor of the SF office — it wasn’t until a couple of weeks later that I mastered the after-hours A/C — when my new, even hotter workstation had an OS, a checkout, compiler, and WinDBG + cargo-culted symserver config. One build on the new box and I was hooked.

5-10 minute links went to 1-2…and less in many cases because I could now enable incremental linking! And HT really worked on the i7′s, cutting build times further. Hot damn! In what felt like no-time at all, my drudgery turned to sleuthing/debugging bliss (if there is such a thing). I could make code changes, compile them, and be working with the results in less time than it took to make coffee. Being able to make changes and then feel them near-instantly turned the tide, keeping me in the loop longer, letting me explore faster, and making me less afraid to change things for fear of the time it would take to roll back to a previous state. It wasn’t the webdev nirvana of ctrl-r, but it was so liberating that it nearly felt that way. What had been a week-long investigation was wrapped up in a day. The launch was un-blocked (at least by that bug) and the world seemed new.

The difference was directness.

The same story repeats itself over and over again throughout the history of Chrome: shared-library builds, ever-faster workstations, trybots and then faster trybots, gyp (instead of Scons), many different forms of distributed builds, make builds for gyp (courtesy of Evan Martin), clang, and of course ninja (also Evan…dude’s a frickin hero). Did I mention faster workstations? They’ve made all the same sort of liberating difference. Truly and honestly, in ways I cannot describe to someone who has not felt the difference between ctrl-r development and the traditional Visual Studio build of a massive project, these are the things that change your life for the better when you’re lashed to the mast of a massive C++ behemoth.

If there is wisdom in the Chrome team, it is that these projects are not only recognized as important, but the very best engineers volunteer to take them on. They seem thankless, but Chrome is an environment that rewards this sort of group-adaptive behavior: the highest good you can do as an engineer is to make your fellow engineers more productive.

And that’s what you’re missing from everything else you’re reading about this announcement today. To make a better platform faster, you must be able to iterate faster. Steps away from that are steps away from a better platform. Today’s WebKit defeats that imperative in ways large and small. It’s not anybody’s fault, but it does need to change. And changing it will allow us to iterate faster, working through the annealing process that takes a good idea from drawing board to API to refined feature. We’ve always enjoyed this freedom in the Chromey bits of Chrome, and unleashing Chrome’s Web Platform team will deliver the same sorts of benefits to the web platform that faster iteration and cycle times have enabled at the application level in Chrome.

Why couldn’t those cycle-time-improving changes happen inside WebKit? After all, much work has happened in the past 4 years (often by Googlers) to improve the directness of WebKit work: EWS bots, better code review flow, improved scripts and tools for managing checkins, the commit queue itself. The results have been impressive and have enabled huge growth and adoption by porters. WebKit now supports multiple multi-process architecture designs, something like a half-dozen network stack plug-ins, and similar diversity at every point where the engine calls back to outside systems for low-level implementation (GPU, network, storage, databases, fonts…you name it). The community is now committed to enabling porters, and due to WebKit’s low-ish level of abstraction each new port raises the tax paid by every other port. As James Robinson has observed, this diversity creates an ongoing drag when the dependencies are intertwined with core APIs in such a way that they can bite you every time you go to make a change. The Content API boundary is Blink’s higher-level “embedding” layer and encapsulates all of those concerns, enabling much cleaner lines of sight through the codebase and the removal of abstractions that seek only to triangulate between opaque constraints of other ports. Blink gives developers much more assurance that when they change something, it’s only affecting the things they think it’s affecting. Moving without fear is the secret of all good programming. Putting your team in a position to move with more surety and less fear is hugely enabling.

Yes, there are losses. Separating ourselves from a community of hugely talented people who have worked with us for years to build a web engine is not easy. The decision was wrenching. We’ll miss their insight, intelligence, and experience. In all honesty, we may have paid too high a price for too long because of this desire to stay close to WebKit. But whatever the “right” timing may have been, the good that will come from this outweighs the ill in my mind.

Others will cover better than I can how this won’t affect your day-to-day experience of WebKit-derived browser testing, or how it won’t change the feature-set of Chrome over-night, or how the new feature governance process is more open and transparent. But the most important thing is that we’ll all be going faster, either directly via Blink-embedding browsers or via benchmarks and standards conformance shaming. You won’t feel it overnight, but it’s the sort of change in model that enables concrete changes in architecture and performance and that is something to cheer about — change is the predicate for positive change, after all.

After doing much reading of TAG meeting minutes, f2f notes, issues, delivered products, and findings I’ve come to a sobering conclusion: the TAG isn’t focused on eliminating the biggest sources of developer pain today. Now, you can argue that this might not be their job, but I won’t agree. It’s the TAGs job to help ensure the health of the platform, both for publishers and search engines, but also for authors. And the web as a platform is in some real trouble today.

There doesn’t seem to be a sense of urgency in the TAG about the corrosive effects of poor design and integration on the overall usability and health of the system. The Web to the TAG, as I can understand it through the meeting minutes and notes, is a collection of documents that represent internally-referential collections of data which are are linked to other documents, not a series of applications that are attempting ever more impressive feats of richness on a platform that stymies them every time you hit one of the seams. In reality it is (aspirationally) both things but the very real tensions between them don’t appear in the TAG’s work, leading me to believe that it doesn’t comprehend the latter aspect of web development today and what that means for the future health and viability of the platform.

I drone on and on and on about layering because explaining each bit of the platform in terms of the one below it relieves much of the tension created by disconnected declarative forms and APIs. This matters because in today’s web when you go ever so slightly off the path paved by a spec’s use-cases, the drop-off is impossibly steep, and the only way to keep from risking life-threatening abstraction level transitions is to flood the entire canyon with JavaScript and hope you can still swim in the resulting inland sea of complexity. This is what our biggest, “best” webapps do today, relying on ENORMOUS piles of JavaScript that largely serve to re-create what browsers already do in the hopes of marginally extending that capability. It’s simply nuts, but the TAG doesn’t seem to acknowledge the threat this poses to everything it holds dear: linking, declarative forms, data…it’s all about to be lost beneath the waves, and because the TAG doesn’t understand the growing importance of JS, it seemingly doesn’t see the threat. Declarative forms disappear first beneath imperatively-delivered complexity; lingua-franca APIs next. Without ways of getting what your app needs while keeping one foot on the declarative path, app developers do what they must; declarative data and relationships become “nice to haves” not “the way you do it”. Layering provides easy steps between the levels of abstraction, avoiding the need to re-create what the platform was already doing for you along with whatever custom thing you need — and it’s not the TAG’s current agenda.

If elected, I will work to fix that. The TAG is the right group to formulate and articulate a theory of good layering in the web platform’s architecture and it’s the only group at the W3C whose mission is to help spec authors wrestle with large-scale design and integration problems like this. My background is in apps, and JS, and browsers, and I work at one of the few places deeply invested in ensuring that we maintain a healthy, declarative web for the future. I care tremendously about the viability of the largely-declarative web. Through my work with Dimitry Glazkov and many others on Web Components I’ve done as much as anyone in the last decade to help build a bridge between the JS and declarative worlds. Dimitry and I created and led the team here at Google that have put Shadow DOM, CSS Variables, Custom Elements, Mutation Observers (and Object.observe) into specs and on the platform agenda, all with the explicit goal of creating better layering; explaining the magic in today’s platform and drawing connections between the bits that had none before. And I think we need to keep building more of those bridges, but it’s hard when W3C culture views that agenda with suspicion. Why would any WG concern itself with integration with specs outside its charter? It’s the TAGs job to inject that global perspective. I believe the TAG should pursue the following work as a way of filling its charter:

• Getting reconnected to web developers: today’s TAG isn’t composed of web developers (Sir Tim excepted) and the general level of familiarity on the committee with the pressing issues in web development seems low. As a member I’ll press to ensure that at least one of the face-to-face meetings each year overlaps with an industry web development conference (not an academic symposium). Having the TAG simply go and listen, and perhaps answer questions in such a forum, would do much to illuminate the gulf in understanding. I also support Marco’s agenda of direct developer outreach (G+, AMA, IRC, etc.)
• Forming and expressing an opinion about idiomatic JS APIs: the W3C is the body that specifies JavaScript’s largest, most important standard library yet it doesn’t seem to treat that task with any care. Poor integration with JS types, idioms, library conventions, and calling conventions are the calling card of W3C APIs. The detrimental effects are pervasive and corrosive. The TAG should produce a guide on designing idiomatic JS APIs for WGs to use. Additionally, the TAG should use its spec review perch to call out poorly designed JS APIs as such and propose changes that improve full-stack integration. Lastly, the TAG should be open to facilitating (or teaching!) classes on how to design good APIs for the web. It’s a skill that can be learned like any other, and the TAG has a unique responsibility towards ensuring that the next generation of APIs is better than the last.
• Bridge building to TC39: I’ve served on ECMA TC39 — the standards body for JavaScript — since 2007 and in that time I’ve seen how the dysfunctional relationship with the W3C enables terrible API design. Needless to say, the results haven’t been great. For a small taste, go look at some IndexedDB samples. There are dozens of terrible DOM APIs (from the perspective of JS) and JS fails to help describe interfaces well enough (from the perspective of DOM). The impasse has even been codified. For the sake of the platform and the developers who must use it, we need to do better. I will push to ensure that at least one of the face-to-face meetings of TC39 each year overlaps with a TAG meeting. I’m hopeful that with my fellow TC39 delegate (Yehuda Katz) and the maintainer of DOM (Anne van Kesteren) also on the TAG, we can make serious progress on this front, actively building a culture of understanding between TC39 and the W3C and furthering the common goal of a web that’s competitive and well integrated.
• Advocate for layering: it’s the TAGs natural role to advocate for coherence in the platform, not only at the markup level, but also inside the app runtime bits that take up so very much of the time in W3C’s most active WGs. If the TAG doesn’t do this, asking the important questions at the right time and working to show the benefits of collaboration and cross-API thinking, who will? To combat this, I propose that the TAG take as an open issue the lack of coherence in the client side platform and work to identify the largest developer pain-points in findings that work to set an agenda for WGs in the future. There is no hope for a well-integrated, layered platform without every WG accepting some responsibility for the usability and layering commons, and if elected I will work to ensure the TAG is a tireless advocate for that commons.

If that sounds like meaningful progress to you, I’d appreciate your organization’s vote; along with your consideration to vote for my fellow reformers: Yehuda Katz, Anne van Kesteren, Peter Linss, and Marcos Cáceres. AC reps for each organization can vote here and have 4 votes to allocate in this election. Voting closes near the end of the month, and it’s also holiday season, so if you work at a member organization and aren’t the AC rep, please, find out who that person in your organization is and make sure they vote. The TAG can’t fix the web or the W3C, but I believe that with the right people involved it can do a lot more to help the well-intentioned people who are hard at work in the WGs to build in smarter ways that pay all of us back in the long run.

“That’s just a browser caching problem.”

Look for this to be deployed along standards-wonk classics such as “our job should just be to provide low-level primitives”, “we don’t know enough to solve this”, and “people will just write tiny libraries to provide that”.

“It’s a caching problem” is a common refrain of those who don’t build apps and aren’t judged on their latency. And it’s transparently bullshit in the web context. It relies on you forgetting — for just that split second while it sounds possible to duck the work at hand — that we’re talking about a platform whose primary use-case is sending content across a narrow, high-latency pipe. If you work in web standards and don’t acknowledge this constraint, you’re a menace to others.

Recent history alone should be enough to invalidate the caching argument; remember Applets? Yeah, Java had lots of problems on the client side, but what you saw with client-side Java were the assumptions of server-side engineers (who get to control things like their deployment VM versions, their hardware and spindle speeds, etc.) imported to distributed code environments where you weren’t pulling from a fast local disk in those critical moments when you first introduced your site/app/whatever to the users. Instead, you saw the piles upon piles of JARs that get created when you assume that that next byte is nearly free, that disk is only 10ms away, and that cold startups are the rare case. It worked out predictably and Java-like systems succeed on the client when their cultural assumptions do align with deployment constraints — Android and iOS are great examples. Their mediated install processes see to it.

Back out here on the wolly web, caching is something that’s under user control. It must be for privacy reasons, and we know from long experience that users clear their caches. The “cache” they can’t clear is the baseline set of functionality the platform provides — i.e., the built in stuff on all runtimes developer cares about…which is what specs can effect (obliquely and with some delay).

By the time you’re having a serious discussion about adding a thing to a spec among participants who aren’t obvious bozos, you can bet your sweet ass that the reason it was brought up in the first place is a clear and obvious replication of effort among existing users bordering on the ubiquitous — often in libraries they use. Saying that someone can write a library rises no higher than mere truism, and saying that a standards body shouldn’t provide some part of the common features found among those existing libraries because caching will make the cost of those libraries moot is ignorance or standards-jujitsu in an attempt to get a particular proposal shelved for some other reason.

As bad as the above is, consider the (historically prevalent) use of this argument regarding “why we don’t need no stinking” markup features — just “fix” browser caching and “give me a low-level thing and I’ll build [rounded-corners, gradients, new layout mechanisms, CSS3d xforms, etc.] myself”. The subtle bias towards JavaScript and away from a declarative web is one of the worst, most insidious biases of the already enfranchised upper-class of JavaScript-savvy web developers, perpetuating a two-tier web in which “real engineers” can do better every year but wherein those same expressiveness and performance gains aren’t transmitted to folks without CS degrees (yes, I’m looking straight at you, WebGL). You almost want to give it to them, though, so they’ll finally come to terms with how wrong they really are about the ability for caching to be “fixed”.

We’ve spent a decade on this — remember that we’re all using CDNs, pulling our JS libraries from edge-cached servers with stable URLs for optimal LRU behavior, running minifiers, setting far-forward expires, etc. And that’s just what webdevs are doing: meanwhile browser vendors have been working day and night to increase the sizes of caches, ensure full caches where possible, and implement sometimes crazy specs that promise to help. It’s not for lack of trying, but we still don’t collectively know how to “fix” caching.

Think libraries are free? Show me how you’ll make ‘em free and I’ll start taking you seriously. Until then, bozo bit flipped.

“It should have exactly the same API as library X”

Similar arguments include “it must be pollyfillable”, etc.

Why is this bullshit? Not because it represents an aversion to being caught in an implementation/deployment dead zone — that’s a serious concern. Nobody wants a better world dangling out there just beyond reach, waiting for Vendor X to pick it up and implement or for Version Y of Old Browser to die so that 90% of your clients can acces the feature. That’s where libraries provide great value and will continue to, no matter what the eventual feature looks like. Remember, the dominant libraries in use by developers don’t have Firefox, Chrome, IE, and Opera-specific versions that you serve up based on client. The platonic ideal is 180-degrees in the other direction: one code base, feature detection, polyfills, progressive enhancement — basically anything within (and often well outside of) reason to keep from serving differential content. So your library is going to have all those versions anyway until all of your clients have the new-spec version. Optimizing based on existing API design because “now we can polyfill it without extra effort!” misunderstands both the role of spec designers and of libraries and serves users very poorly indeed.

Where this argument becomes truly inadmissible, though, is when it seeks to define the problem to be “what our API already does”. Turns out that the language you can design features with when all you have is JavaScript and HTML/CSS patterns is…well…the JS, HTML, and CSS you have today. Powerful yes. Expressive? Hrm. If your job, however, is to evolve an integrated platform, taking on a constraint predicated on the current form of your systems is nuts. There are some hard constraints like that (backwards compatibility), but adopting some form of hack as the “blessed” way of doing something without looking around and going “how can we do this better given that we have more powerful and expressive language to solve the problem with?” is nothing but lost opportunity.

Yes, a standards group might look around and go “nope, we can’t do better than that polyfill/library” and adopt the solution wholesale. That’s not a bad thing. What is bad, though, is advocacy about far-future solutions to current problems based solely on the idea that some library totally nailed it and we shouldn’t be asking for anything more — e.g.: Microdata cribbed from RDFa and Microformats when what you really wanted was Web Components. Rote recital of existing practice is predictably weak sauce that robs the platform of its generative spark. We should all be hoping for enhancements that make the future web stronger than today’s, and you only find those opportunities by taking off the blinders and giving yourself free reign to do things that libraries, polyfills, and hack just can’t.

What to do about the developers and users caught in the crossfire? Well, we can advocate for degradable, polyfill-friendly designs, but there are limits to that. The most hopeful, powerful way to make the pain disappear is to ensure that more of our users year-over-year are on browsers/platforms that auto-update and won’t ever be structurally left-behind again. And yes, that’s something that every web developer should be working towards; prompting users to update to current-version auto-upgrading — “evergreen”, if you will — browsers. Remember: you get the web you ask your users for.

“Just tell us the use cases”

This is the one I’m most sick of hearing from the mouths of standards wonks and authors. What they’re trying to say is some well-meaning combination of:

• If you tell us exactly how you’re doing whatever it is you’re accomplishing today, we’ll run the risk of just adopting some API that was designed with some bogus constraints (see above).
• We need to communicate to a lot of people involved in this effort why they should care, ’cause putting stuff into browsers is a shitload of work.

What it often winds up doing, however, is serving as a way for a standards body or author to shut up unproductive folks; folks who aren’t willing to do the work of helping them come to enlightenment about the architecture of the current solution, the constraints it was built under, and the deficiencies it leaves you with. Or it can be used to avoid that process entirely — which is where we’re getting towards bullshit territory. Taken to the extreme (and it too often is) the “use-cases, not details” approach infantilizes standards body participants, setting a bar too high for the folks who are crying out for help and setting the bar too low for the standards body because, inevitably, the tortured text of the “use cases” is an over-terse substitue for a process, a way of building, and an architectural approach. Use-cases (as you see them for HTML and CSS in particular) become architecture-free statements, no more informative than a line plucked at random from Henry V. Suggestive, yes. Useful? Nope. The very idea of building standards without a shared concept of architecture is bogus, and standards participants need to stop hiding behind this language. If you don’t understand something, say so. If many people work hard to explain it and can’t, ask for a sample application or snippet to hack on so you can do a mind-meld with their code and can ask questions about it in context.

Yes, having the use-cases matters, but nobody should be allowed to pretend that they’re a substitue for understanding the challenges of an architecture.

While I’ve only covered a few of the very worst spec-building tropes, it’s by no means the end of the rhetorical shit list. Perhaps this will be a continuing series — although I’m sure this post will offend enough folks to make me think twice about it. If we put an end to just these, though, a lot of good could be accomplished. Here’s to hoping.

”every year we’re throwing more and more JS on top of the web”

The way things are going in my world, we are looking at replacing the web with JS, not simply layering. At a certain point you look at it all and say “why bother”. Browsers render the DOM not markup. They parse markup. Just cut out the middle man and send out DOM – in the form of JS constructs.

The second part is to stop generating this markup which then must be parsed on a server at the other end of a slow and high latency prone communication channel. Instead send small compact instructions to the browser/client that tells it how to render and when to render. Later you send over the data, when it’s needed…

This is a clear distillation of what scares me about the road we’re headed down because for each layer you throw out and decide to re-build in JS, you end up only doing what you must, and that’s often a deadline-driven must. Accessibility went to hell? Latency isn’t great? Doesn’t work at all without script? Can’t be searched? When you use built-ins, those things are more-or-less taken care of. When we make them optional by seizing the reigns with script, not only do we wind up playing them off against each other (which matters more, a11y or latency?) we often find that developers ignore the bits that aren’t flashy. Think a11y on the web isn’t great now? Just wait ’till it’s all JS driven.

It doesn’t have to be this way. When we get Model Driven Views into the browser we’ll have the powerful “just send data and template it on the client side” system everyone’s looking for but without threatening the searchability, a11y, and fallback behaviors that make the web so great. And this is indicative of a particularly strong property of markup: it’s about relationships. “This thing references that thing over there and does something with it” is hard for a search engine to tease out if it’s hidden in code, but if you put it in markup, well, you’ve got a future web that continues to be great for users, the current crop of developers, and whoever builds and uses systems constructed on top of it all later. That last group, BTW, is you if you use a search engine.

But it wasn’t all clarity and light in the comments. Austin Cheney commented on the last post to say:

This article seems to misunderstand the intention of these technologies. HTML is a data structure and nothing more. JavaScript is an interpreted language whose interpreter is supplied with many of the most common HTML parsers. That is as deep as that relationship goes and has little or nothing to do with DOM.

…It would be safe to say that DOM was created because of JavaScript, but standard DOM has little or nothing to do with JavaScript explicitly. Since the release of standard DOM it can be said that DOM is the primary means by which XML/HTML is parsed suggesting an intention to serve as a parse model more than a JavaScript helper.

Types in DOM have little or nothing to do with types in JavaScript. There is absolutely no relationship here and there shouldn’t be…You cannot claim to understand the design intentions around DOM without experience working on either parsers or schema language design, but its operation and design have little or nothing to do with JavaScript. JavaScript is just an interconnecting technology like Java and this is specifically addressed in the specification in Appendix G and H respectively.

And, after I tried to make the case that noting how it is today is no replacement for a vision for how it should be, Austin responds:

The problem with today’s web is that it is so focused on empowering the people that it is forgetting the technology along the way. One school of thought suggests the people would be better empowered if their world were less abstract, cost radically less to build and maintain, and is generally more expressive. One way to achieve such objectives is alter where costs exist in the current software life cycle of the web. If, for instance, the majority of costs were moved from maintenance to initial build then it could be argued that more time is spent being creative instead of maintaining.

I have found that when working in HTML alone that I save incredible amounts of time when I develop only in XHTML 1.1, because the browser tells you where your errors are. … Challenges are removed and costs are reduced by pushing the largest cost challenges to the front of development.

… The typical wisdom is that people need to be empowered. If you would not think this way in your strategic software planning then why would it make sense to think this way about strategic application of web technologies? …

This might all sound very rational on one level, but a few points need to be made:

• If we’re not building technology for people, WTF are we doing with these CPU cycles exactly?
• Speed of iteration is a key enabler of progress in any technology stack I’ve ever worked in
• Strictness fails in the wild

I think Austin’s point about moving costs from maintenance to build is supposed to suggest that if we were only more strict about things, we’d have less expensive maintenance of systems, but it’s not clear to me that this has anything to do with strictness. My observation from building systems is that this has a lot more to do with being able to build modular, isolated systems that compose well. Combine that with systems that let you iterate fast, and you can grow very large things that can evolve in response to user needs without turning into spaghetti quite so quickly. Yes, the web isn’t great for that today, but strictness is orthogonal. Nothing about Web Components demands strictness to make maintainability infinitely better.

And the last point isn’t news. Postel’s Law isn’t a plea about what you, dear software designer, should be doing, it’s an insightful clue into the economics of systems at scale. XML tried being strict and it didn’t work. Not even for RSS. Mark Pilgrim’s famously heroic attempts at building a reliable feed parser match the war stories I’ve heard out of the builders of every large RSS system I’ve ever talked to. It’s not that it’s a nice idea to be forgiving about what you accept, it’s that there’s no way around it if you want scale. What Austin has done is the classic bait-and-switch: he has rhetorically substituted what works in his organization (and works well!) for what’s good for the whole world, or even what’s plausible. I see this common logical error in many a standards adherent/advocate. They imagine some world in which it’s possible to be strict about what you accept. I think that world might be possible, but the population would need to be less than the size of a small city. Such a population would never have ever created any of the technology we have, and real-world laws would be how we’d adjudicate disputes. As soon as your systems and contracts deal with orders of magnitude more people, it pays to be reliable. You’ll win if you do and lose if you don’t. It’s inescapable. So lets banish this sort of small-town thinking to the mental backwaters where it belongs and get on with building things for everyone. After all, this is about people. Helping sentient beings achieve their goals in ways that are both plausible and effective.

If helping people is not what this is about, I want out.

This is why I get so exercised about WebIDL and the way it breaks the mental model of JS’s “it’s just extensible objects and callable functions”. It’s also why my discussions with folks at last year’s TPAC were so bleakly depressing. I’ve been meaning to write about TPAC ever since it happened, but the time and context never presented themselves. Now that I got some of my words out about layering in the platform, the time seems right.

Let me start by trying to present the argument I heard from multiple sources, most likely from (in my feeble memory) Anne van Kestern Jonas Sicking(?):

ECMAScript is not fully self-describing. Chapter 8 drives a hole right through the semantics, allowing host objects to whatever they want and more to the point, there’s no way in JS to describe e.g. list accessor semantics. You can’t subclass an Array in JS meaningfully. JS doesn’t follow it’s own rules, so why should we? DOM is just host objects and all of DOM, therefore, is Chapter 8 territory.

Brain asploded.

Consider the disconnect: they’re not saying “oh, it sure would be nice if our types played better with JS”, they’re saying “you and what army are gonna make us?” Remember, WebIDL isn’t just a shorthand for describing JavaScript classes, it’s an entirely parallel type hierarchy.

Many of the Chapter 8 properties and operations are still in the realm of magic from JS today, and we’re working to open more of them up over time by giving them API — in particular I’m hopeful about Allen Wirfs-Brock’s work on making array accessors something that we can treat as a protocol — but it’s magic that DOM is appealing to and even specifying itself in terms of. Put this in the back of your brain: DOM’s authors have declared that they can and will do magic.

Ok, that’s regrettable, but you can sort of understand where it comes from. Browsers are largely C/C++ enterprises and DOM started in most of the successful runtimes as an FFI call from JS to an underlying set of objects which are owned by C/C++. The truth of the document’s state was not owned by the JS heap, meaning every API you expose is a conversation with a C++ object, not a call into a fellow JS traveler, and this has profound implications. While we have one type for strings in JS, your C++ side might have bstr, cstring, wstring, std::string, and/or some variant of string16.

JS, likewise, has Number while C++ has char, short int, int, long int, float, double, long double, long long int…you get the idea. If you’ve got storage, C++ has about 12 names for it. Don’t even get me started on Array.

It’s natural, then, for DOM to just make up it’s own types so long as its raison d’être is to front for C++ and not to be a standard library for JS. Not because it’s malicious, but because that’s just what one does in C++. Can’t count on a particular platform/endianness/compiler/stdlib? Macro that baby into submission. WTF, indeed.

This is the same dynamic that gives rise to the tussle over constructable constructors. To recap, there is no way in JS to create a function which cannot have new on the left-hand-side. Yes, that might return something other than an instance of the function-object on the right-hand side. It might even throw an exception or do something entirely non-sensical, but because function is a JavaScript concept and because all JS classes are just functions, the idea of an unconstructable constructor is entirely alien. It’s not that you shouldn’t do it…the moment to have an opinion about that particular question never arises in JS. That’s not true if you’re using magic to front for a C/C++ object graph, though. You can have that moment of introspection, and you can choose to say “no, JS is wrong”. And they do, over and over.

What we’re witnessing here isn’t “right” or “wrong”-ness. It’s entirely conflicting world views that wind up in tension because from the perspective of some implementations and all spec authors, the world looks like this:

Not to go all Jeff Foxworthy on you, but if this looks reasonable to you, you might be a browser developer. In this worldview, JS is just a growth protruding from the side of an otherwise healthy platform. But that’s not how webdevs think of it. True or not, this is the mental model of someone scripting the browser:

The parser, DOM, and rendering system are browser-provided, but they’re just JS libraries in some sense. With <canvas>‘s 2D and 3D contexts, we’re even punching all the way up to the rendering stack with JS, and it gets ever-more awkward the more our implementations look like the first diagram and not the second.

To get from parser to DOM in the layered world, you have to describe your objects as JS objects. This is the disconnect. Today’s spec hackers don’t think of their task as the work of describing the imperative bits of the platform in the platform’s imperative language. Instead, their mental model (when it includes JS at all) pushes it to the side as a mere consumer in an ecosystem that it is not a coherent part of. No wonder they’re unwilling to deploy the magic they hold dear to help get to better platform layering; it’s just not something that would ever occur to them.

Luckily, at least on the implementation side, this is changing. Mozilla’s work on dom.js is but one of several projects looking to move the source of truth for the rendering system out of the C++ heap and into the JS heap. Practice is moving on. It’s time for us to get our ritual lined up with the new reality.

Which brings me too David Flanagan who last fall asked to read my manifesto on how the platform should be structured. Here it is, then:

The network is our bottleneck and markup is our lingua-franca. To deny these facts is to design for failure. Because the network is our bottleneck, there is incredible power in growing the platform to cover our common use cases. To the extent possible, we should attempt to grow the platform through markup first, since markup provides the most value to the largest set of people and provides a coherent way to expose APIs via DOM.

Markup begets JS objects via a parser. DOM, therefore, is merely the largest built-in JS library.

Any place where you cannot draw a line from browser-provided behavior from a tag to the JS API which describes it is magical. The job of Web API designers is first to introduce new power through markup and second to banish magic, replacing it with understanding. There may continue to be things which exist outside of our understanding, but that is a challenge to be met by cataloging and describing them in our language, not an excuse for why we cannot or should not.

The ground below our feet is moving and alignment throughout the platform, while not inevitable, is clearly desirable and absolutely doable in a portable and interoperable way. Time, then, to start making Chapter 8 excuses in the service of being more idiomatic and better layered. Not less and worse.

I had a chance last week to share some of my thinking here to an unlikely audience at EclipseCon, a wonderful experience for which my thanks go to Mike Milinkovich and Ian Skerrett for being crazy enough to invite a “web guy” to give a talk.

One of the points I tried (and perhaps failed) to make in the talk was that in every platform that’s truly a platform it’s important to have a stable conceptual model of what’s “down there”. For Java that’s not the language, it’s the JVM. For the web…well…um. Yes, it bottoms out at C/C++, but that’s mostly observable through spooky action at a distance. The expressive capacity of C/C++ show up as limitations and mismatches in web specs all the time, but the essential semantics — C/C++ is just words in memory that you can do whatever you please with — are safely hidden away behind APIs and declarative forms that are unfailingly high-level. Until they aren’t. And you can forget about composition most of the time.

For a flavor of this, I always turn back to Joel Webber’s question to me several years ago: why can’t I over-ride the rendering of a border around an HTML element?

It’s a fair question and one I wrote off too quickly the first time he posed it. We have <canvas> which lets us draw lines however we like, so why can’t we override the path painting for borders? Why isn’t it just a method you implement like in Flex or Silverlight?

Put another way: there are some low level APIs in the web that suggest that such power should be in the hands of us mortals. When using a low-level thing, you pay-as-you-go since lower-level things need more code (latency and complexity)…but that’s a choice. Today’s web is often mysteriously devoid of the sort of sane layering, forcing you to re-build parallel systems to what’s already in the browser to get a job done. You can’t just subclass the right thing or plug into the right lifecycle method most of the time. Want a <canvas>? Fine. There you go. Want a <span>? Hot <span>s coming up! But don’t go getting any big ideas about using the drawing APIs from <canvas> to render your <span>. Both are magic in their own right and for no reason other than that’s the way it has always been.

The craziest part in all of this is that JavaScript does exist in the web so you can strictly speaking do whatever you want. Goodness knows that when the platform fails us today, we’re all-too-willing to just throw JS at it. It’s crazy, in this context then, that spec authors seem to be trying to uphold a golden principle: JavaScript doesn’t exist. Writing it out of the story allows you to just claim that your bit of the system is magic and that it doesn’t need an exposed lifecycle and plug-in architecture. New things can just be bolted onto the magic, no layering required. It’s magical turtles all the way down.

You can see why people who think in terms of VM’s and machine words might find this a bit ahem limiting.

But how much should we “web people” care about what they think? After all, “real programmers” have been predicting the imminent death of this toy browser thing for so long that I’m forgetting exactly when the hate took its various turns through the 7 stages; “Applets will save us from this insanity!”…”Ajax is a hack”…”just put a compiler in front of it and treat it as the dumbest assembler ever” (which is at least acceptance, of a sort). The web continues to succeed in spite of all of of this. So why bother with the gnashing of teeth?

Thanks to Steve Souders, I have an answer: every year we’re throwing more and more JS on top of the web, dooming our best intended semantic thoughts to suffocation in the Turing tar pit. Inexorably, and until we find a way to send less code down the wire, us is them, and more so every day.

Let that picture sink in: at 180KB of JS on average, script isn’t some helper that gives meaning to pages in the breech, it is the meaning of the page. Dress it up all you like, but that’s where this is going.

Don’t think 180KB of JS is a lot? Remember, that’s transfer size which accounts for gzipping, not total JS size. Oy. And in most cases that’s more than 3x the size of the HTML being served (both for the page and for whatever iframes it embeds). And that’s not all; it’s worse for many sites which should know better. Check out those loading “filmstrip” views for gawker, techcrunch, and the NYT. You might be scrolling down, looking at the graphs, and thinking to yourself “looks like Flash is the big ticket item…”, and while that’s true in absolute terms, Flash isn’t what’s blocking page loads. JS is.

And what for? What’s all that code doing, anyway?

It’s there for three reasons: first, to clean up the messes that browser vendors aren’t willing or able to clean up for themselves; second, to provide an API that becomes the new platform, and lastly to provide the app-specific stuff you are trying to get across. Only the last one is strictly valuable. You’re not including JQuery, Backbone, Prototype or Dojo into your pages just because you like the API (if you are, stop it). You’re doing it because the combination of API and even behavior across browsers makes them the bedrock. They are the new lisp of application construction; the common language upon which you and your small team can agree; just don’t expect anyone else to be able to pick up your variant without squinting hard.

This is as untenable as it is dangerous. It was this realization that set me and Dimitri Glazkov off to build a team to do something about it more than a year and a half ago. The results are showing up now in the form of Web Components and Shadow DOM, Mutation Observers as plumbing for Model Driven View, and a host of new CSS capabilities and JavaScript language expressiveness wins. If that sounds like a huge pile of seemingly un-related work, let me walk back to one of the motivating questions and then I’ll fast forward to the approach:

What would it mean to be able to subclass an HTML Element?

We observed that most of what the current libraries and frameworks are doing is just trying to create their own “widgets” and that most of these new UI controls had a semantic they’d like to describe in a pretty high-level way, an implementation for drawing the current state, and the need to parent other widgets or live in a hierarchy of widgets.

Heeeeeyyyyyy….wait a minute…that sounds a lot like what HTML does! And you even have HTML controls which generate extra elements for visual styling but which you can’t access from script. This, BTW, is what you want when building your own controls. Think the bullets of list items or the sliders generated by <input type="range">. There are even these handy (non-constructable!?!) constructors for the superclasses in JS already.

So what would you need access to in order to plug into that existing system? And how should it be described? This, by the way, is the danger zone. Right about this point in the logical chain most folks tend to fall back to what they know best: C++ hacker? Give ‘em a crappy C++-inspired high-level-ish JS API that will make the people yelling loudest stop beating you up. Declarative guy? Force everyone to describe their components as separate documents and…yeah. XUL. You get the idea. JavaScript person? Demand the lowest level API and as much unwarranted power as possible and pretend you don’t need the browser. JS libraries are the “fuck it, we’ll do it live!” of the web.

None of these are satisfying. Certainly not if what we want is a platform of the sort you might consider using “naked”. And if your “platform” always needs the same shims here and polyfills there, let me be candid: it ain’t no platform. It’s some timber and bolts out of which you can make a nice weekend DIY project of building a real platform.

So we need to do better.

What does better look like?

Better is layered. Better is being able to just replace what you need, to plug in your own bits to a whole that supports that instead of making you re-create everything above any layer you want to shim something into. This is why mutable root prototypes in JS and object mutability in general are such cherished and loathed properties of the web. It is great power. It’s just a pity we need it so often. Any plan for making things better that’s predicated on telling people “oh, just go pile more of your own parallel systems on top of a platform that already does 90% of what you need but which won’t open up the API for it” is DOOMED.

Thus began a archaeology project, one which has differed in scope and approach from most of the recently added web capabilities I can think of, not because it’s high-level or low-level, but because it is layered. New high-level capabilities are added, but instead of then poking a hole nearly all the way down to C++ when we want a low-level thing, the approach is to look at the high-level thing and say:

How would we describe what it’s doing at the next level down in an API that we could expose?

This is the reason low-level-only API proposals drive me nuts. New stuff in the platform tends to be driven by scenarios. You want to do a thing, that thing probably has some UI (probably browser provided), and might invoke something security sensitive. If you start designing at the lowest level, throwing a C++ API over the wall, you’ve turned off any opportunity or incentive to layer well. Just tell everyone to use the very fine JS API, after all. Why should anyone want more? (hint: graph above). Instead of opening doors, though, it’s mostly burden. Everything you have to do from script is expensive and slow and prone to all sorts of visual and accessibility problems by default. If the browser can provide common UI and interaction for the scenario, isn’t that better most of the time? Just imagine how much easier it would be to build an app if the initial cut at location information had been <input type="location"> instead of the Geolocation API we have now. True, that input element would need lots of configuration flags and, eventually, a fine-grained API…if only there were a way to put an API onto an HTML element type…hrm…

In contrast, if we go declarative-only we get a lot of the web platform today. Fine at first but horrible to work with over time, prone to attracting API barnacles to fill perceived gaps, and never quite enough. The need for that API keeps coming back to haunt us. We’re gonna need both sides, markup and imperative, sooner or later. A framework for thinking about what that might look like seems in order. Our adventure in excavation with Web Components has largely been a success, not because we’re looking to “kernalize the browser” in JS — good or bad, that’s an idea with serious reality-hostile properties as soon as you add a network — but because when you force yourself to think about what’s already down there as an API designer, you start making connections, finding the bits that are latent in the platform and should be used to explain more of the high level things in terms of fewer, more powerful primitives at the next layer down. This isn’t a manifesto for writing the whole world in JS; it’s a reasonable and practical approach for how to succeed by starting high and working backwards from the 80% use-case to something that eventually has most of the flexibility and power that high-end users crave.

The concrete steps are:

1. Introduce new platform capabilities with high-level, declarative forms. I.e., invent new tags and attributes. DOM gives you an API for free when you do it that way. Everyone’s a winner.
2. When the thing you want feels like something that’s already “down there” somewhere, try to explain the bits that already exist in markup in terms of a lower-level JS or markup primitive. If you can’t do that or you think your new API has no connection to markup, go back to step 1 and start again.
3. When it feels like you’re inventing new language primitives in DOM just to get around JS language limitations, extend the language, not the API

On the web, JavaScript is what’s down there. When it’s not, we’re doing it wrong. It has taken me a very long time to understand why the Java community puts such a high premium on the “pure java” label, and fundamentally what it says to others in the community is “I appealed to no gods and no magic in the construction of this, merely physics”. That’s a Good Thing (TM), and the sort of property that proper platforms should embody to the greatest extent possible.

And this brings me to my final point. C/C++ might be what’s “down there” for web browsers, but that’s also been true of Java. What separates the web and Java, however, is that the Java community sees their imperative abstraction that keeps them from having to think about memory correctness (the JVM) as an asset and many “web people” think of JS as pure liability. I argue that because of the “you’re gonna need both sides” dynamic, trying to write JS out of the picture is a dumb as it is doomed to fail. JavaScript is what’s “down there” for the web. The web has an imperative backbone and we’re never going to expose C/C++ ABI for it, which means JS is our imperative successor. The archaeological dig which is adding features like Web Components is providing more power to JS by the day and if we do this right and describe each bit as a layer with an API that the one above builds on, we can see pretty clearly how the logical regress of the “you must use JS to implement the whole browser” isn’t insane. JS itself is implemented as C/C++, so there’s always room for the mother tongue and of course many of the APIs that we interact with from JS must be C/C++; you can’t write it out of the story — but that doesn’t mean we need to design our APIs there or throw bad design decisions over the wall for someone else to clean up. It is high time we started designing low-level stuff for the web in idiomatic JS (not IDL), start describing the various plug-in points for what they are. We can provide power from our imperative abstraction to and through our declarative layer in ways that make both high and low-level users of the web platform more able to interoperate, build on each other’s work, and deliver better experiences at reasonable cost. That’s the difference between a minefield and a platform. Only one of them is reasonable to build on.

The trash truck just came by which means it’s 6AM here in almost-sunny London. WordPress is likewise telling me that I’m dangerously out of column-inches, so I guess I’ll see if I can’t get a last hour or two of sleep before the weekend is truly over. The arguments here may not be well presented, and they are subtle, but layering matters. We don’t have enough of it and when done well, it can be a powerful tool in ending the battle between imperative and declarative. I’ll make the case some other time for why custom element names are a good idea, but consider it in the layered context: if I could subclass HTMLElement from JavaScript in the natural way, why can’t I just put a new tag name in the map the parser is using to create instances of all the other element types? Aside from the agreement about the names, what makes the built-in elements so special, anyway?

Cognitive dissonance, ahoy! You’re welcome ;-)

Note: this post has evolved in the several days since its initial posting, thanks largely to feedback from Annie Sullivan and Steve Souders. But it’s not their fault. I promise.

Your Moment Of Zen

The backdrop to this debate is that CSS is absolutely the worst, least productive part of the web platform. Apps teams at Google are fond of citing the meme that “CSS is good for documents, but not for apps”. I push back on this, noting the myriad ways in which CSS is abysmal for documents. That isn’t to minimize the pain it causes when building apps, it’s just that the common wisdom is that CSS surely must be fantastic for somebody else. Once we find that person, I’ll be sure to let you know. In the mean time we should contemplate how very, very far behind the web platform is in making it delightful to build the sorts of things that are work-a-day in native environments.

But it’s worse than simply being under-powered: CSS has the weakest escape hatches to imperative code and demands the most world-view contortion to understand its various layout modes and their interactions. Imagining a more congruent system isn’t hard — there are many in the literature, might I humbly suggest that now might be a good time to read Badros & Borning? — and even CSS could be repaired were we able to iterate quickly enough. Things have been moving faster lately, but fast enough to catch up with the yawning gap in platform capabilities? We’ll come back to the speed/scale of change later.

For now, consider that the debate (as captured by Nate Vaughn) is about the retrospective implications of the few things that have already gotten better for some set of developers in some situations. That this sort of meaningful progress (corners, gradients, animations, transitions, flexing boxes, etc.) is rare makes the debate all the more bone-chilling to me. We finally got a few of the long list of things we’ve been needing for a decade or more, and now because the future is unevenly distributed, we’re about to blow up the process that enabled even that modicum of progress? How is it we’re extrapolating causality about engine usage from this unevenness, anyhow? None of this is obvious to me. The credible possibility of losing prefixes as a way to launch-and-iterate is mind-exploding when you realize that the salient competition isn’t other browsers, it’s other platforms. Either the proponents of squatting other vendor’s prefixes haven’t thought this through very hard or they’re bad strategists on behalf of the web as a platform…not to mention their own products. The analysis that we’re being asked to accept rests on an entire series of poor arguments. Lets start with the…

Uncomfortable Assumptions

In an interview out yesterday with Eric Meyer, Tantek Çelik of Mozilla tried to present this debate as a question of barriers to the adoption of non-WebKit based browsers, specifically Firefox Mobile. Opera has made a similar case. What they ommit is that the only platforms where they can credibly ship such browsers are Android and S60 (a dead end). That’s a large (and growing) chunk of the world’s handsets — good news for me, as I now work on Chrome for Android here in London — but for whatever reason it appears that iOS users surf a whole lot more.

Let that sink in: on the devices that are the source of most mobile web traffic today, it’s not even possible to install a browser based on a different engine, at least not without a proxy architecture like the one used in the excellent Opera Mini or Amazon’s Silk. iOS and Windows Phone are both locked-down platforms that come with only one choice of engine (if not browser shell). When folks from the vendors who want to appropriate others’ prefixes talk about “not being able to compete”, remember that competition isn’t even an option for the most active users of mobile browsers. And it’s prefixes that are keeping us down? We must go deeper.

The tension comes into focus when we talk in terms of conversion, retention, and attrition. Conversions are users who, if able, switch to a new product from an old one. Retention is a measure of how many of those users continue to use the product after some period of time. Today (and since Windows first included a browser), the single largest factor in the conversion of users to new browsers is distribution with an OS. This is the entire rationale behind the EU’s browser choice UI, mandated on first start of new Windows installs. Attrition is the rate at which users stop choosing to use your product day after day, and for most desktop installed software, attrition is shockingly high after 3 to 6 months. The attrition rate is usually measured by cohorts over time; users who installed on the same day/week/month to measure what % of that group continue to use the product over increasingly long periods of time. The rate of decay falls, but the overall attrition invariably continues to rise. You might not get un-installed, but that doesn’t mean you’ll still be the go-to icon on the user’s home screen or desktop. Eventually every device is recycled, wiped, or left for history in a drafty warehouse and along with it, your software. A key factor in getting attrition under control for Chrome has been evangelism to improve site compatibility, e.g. “I’m not using your browser because my bank website doesn’t work with it”. That argument — that site compatibility is key to ensuring a competitive landscape for what otherwise are substitutes — puts the entire thing in some perspective. Attrition isn’t the same thing as conversion, and conversion is driven primarily by integrations and advertising. Implicit in the arguments by Tantek and others is a sub-argument of the form:

Our product would have measurably more users if sites were more compatible.

Thanks to what we know about what drives conversions, in the short run this is simply false. Long term, what invariably gives you more users is starting with more users. The set of things that are most effective at convincing users to swap browsers, even for a day, include: advertising, word-of-mouth, a superior product, distribution/bundling, and developer advocacy. Depressingly, only one of those involves actually being a better product, and the prerequisite for all of them is the ability to switch (thanks, Android team!). There’s a similar dynamic at work when doing advocacy to web developers: if you’re nowhere in their browser stats, they’re adding support for a standard or worse a second prefix in order to do service to a cause, not because it’s measurably good for them. Clearly, that’s going to be somewhat less effective. Where, then, is the multi-million dollar advertising campaign for Fennec? The carrier and OEM rev-share deals for bundling on new Android phones? Hmm. To hear Tantek et. al. tell it, non-WebKit-based browsers would be prevalent on mobile if only it weren’t for those damned browser prefixes causing users of other browsers to receive different experiences! Also, those kids and that damned dog.

Over the long haul compatibility can have a dramatic effect on the rate of attrition by changing the slope of the curve — which, remember, is a rate with decay and not a single % — but it begs the next uncomfortable question: what do we mean by “compatibility” here? What sorts of incompatibility cause attrition? Is it content that looks slightly worse but still essentially works (think grey PNG backgrounds on IE6) or does it simply turn you away, not allow you to play in any way, and generally fails (think the ActiveX dependencies of yore)?

Inaccessible or Ugly?

Eric was good enough to call out what I view as a key point in this debate: what sort of “broken” are we talking about? Tantek responded with a link to side-by-side screenshots of various properties rendered on Chrome for Android’s Beta and current Fennec. In some of these cases we may be looking at Fennec bugs. WordPress.com serves the same content to Fennec which seems to bork what float: left; means. That, or some media query is preventing the main blocks from being floated; it’s hard to tell which from a quick view-source:. For the versions of google.* included in the list, the front end is simply serving the desktop version to Fennec which makes the wonky button rendering even stranger. Is there room to improve what gets sent to Fennec? You bet, but that’s not what’s being argued in the main. Ask yourself this: is what you see on that page worth destroying the prefix system for? ‘Cause that’s what the advocates of prefix-squatting would have you believe. In effect, they’re suggesting that nothing will cause developers to test on non-pervasive engines, a deeply fascinating assertion. Even if we accept it, it doesn’t point clearly to a single tactic to resolve the tension. It certainly doesn’t argue most strongly for prefix-squatting.

An important point Eric failed to follow up on was Tantek’s assertion that Mozilla will be cloaking user-agent strings. Does he imagine that the only thing that might be cause someone to send different content is CSS support? API support for things like touch events differs, the performance characteristics of device classes and browsers vary wildly, and application developers are keen to deliver known-good, focused experiences. The endless saga of position: fixed; as plumbed by Google teams and others is a story of competing constraints: browser vendors optimize for content, content fights back. Repeat. What does Mozilla imagine is going to happen here? Maintained content will react to the browser usage of end-users (and as we’ve covered, compat != conversions). Unmaintained content, well, that’s what fallback is all about. And bad things deserve to lose. Assuming that your browser is 100% compatible with developer expectations and testing if you only switch the UA and implement a few prefixes is NPAPI-level crazy all over again, and it’s entirely avoidable. Tantek and Brendan, of all people, should be able to reason that far. I guess they’ll find out soon enough — although we will have all been made poorer for it.

Now, what about the related argument that Mozilla & Co. are only going to be doing this for properties which are “stable” (nevermind their actual standardization status)? The argument says that because something hasn’t changed in another vendor’s prefixed version in a while, it must be safe to squat on. Not only is this (again) incredibly short-sighted, it says that instead of forcing a standard over the line and clobbering both developers and other vendors with the dreaded label of “proprietary” (the usual and effective tactic in this situation), they’re instead willing to claim ownership and therefore blame for the spread of this soon-to-be-proprietary stuff, all the while punting on having an independent opinion about how the features should be structured and giving up on the standards process…and all for what?

Product vs. Platform

Perhaps there wasn’t space in Tantek’s interview with Eric, but both of them chose not to be introspective about the causes of WebKits use in so many mobile browsers, with Tantek merely flagging the use of a single engine by multiple products as “a warning sign.” But a warning of what, exactly? Eric didn’t challenge him on this point, but I sorely wish he had. Why did Safari, the Android Browser, Chrome, Silk, Black Berry, and many others all pick WebKit as the basis for their mobile browsers?

WebKit isn’t a browser. It’s just not. To make a browser based on WebKit, one might bring along at least the following bits of infrastructure which WebKit treats as bits to be plugged in:

• Networking
• Caches of some sort
• Graphics rendering
• A build system
• POSIX or other platform plumbing

And that’s a minimum. Competitive ports tend to include WebSQL, LocalStorage, and Indexed DB back-ends, video codecs, 3D infrastructure (deeply non-trivial), perhaps an alternative JavaScript engine (V8 or other), and alternative/additional image decoders (e.g., WebP). All of that is in addition to needing to create your own UI for bookmarking, navigation, etc. WebKit is an engine, not a fully-functioning vehicle. Therein may lay some of the difference in the relative success of the WebKit and Gecko on mobile to date. Want to build a Gecko-based browser? Great, first clone the entire Firefox codebase from Mercurial, then layer your stuff on top/around. Oh, wait, things might not cleanly be factored to allow you to plug in your own X, Y, or Z? Your builds take forever? Welcome to life in the Mozilla universe where your product is always second fiddle to Firefox. Now, that’s not by way of criticism, mind you. It is entirely reasonable for a product like Firefox not to pay coordination costs with other vendors/users of their code. God knows the overhead over here in WebKit land of trying to keep the menagerie of ports building against all changes is downright daunting. Mozilla (the organization) has made choices that prioritized the success of their product (Firefox) over their codebase (Gecko). WebKit was structured as platform only (no product), both forcing enormous costs onto every port while also freeing them from swimming upstream against a single product’s imperatives in the codebase.

What we’re witnessing isn’t open vs. closed, it’s differences in initial cost of adoption. In JS terms, it’s jQuery (focused core, plugins for everything else) vs. Sencha or Dojo (kitchen sink). Entirely different target users, and both will find their place. Nobody should be shocked to see smaller, more focused bits of code with good plugin characteristics spreading as the basis for new projects. The Mozilla Foundation wants to help prevent monoculture? In addition to making the Firefox product a success, there are concrete engineering things they can do to make Gecko more attractive to the next embedder, Firefox-branded or not. I haven’t heard of progress or prioritization along those lines, but I’m out of the loop. Perhaps such an effort is underway, if so, I applaud it. Whatever the future for Gecko, Product success isn’t related to platform success as a first approximation. Having a good, portable, pluggable core increases the odds of success in evolutionary terms, but it’s absolutely not determinant; see MSIE.

Speaking of IE…I respect those guys a lot, but the logical leap they’re asking us to swallow is that the reason people return Windows Mobile phones is that some CSS doesn’t work. That’s what attrition means on a platform where they’re the only native runtime. Data would change my mind, but it’s a hell of a lot to accept without proof.

The Time Component

Lets take a step back and consider Tantek’s claim that Mozilla has gotten very little traction in evangelizing multi-prefix or prefix-free development for the past year: Firefox for Android has been available since Oct. 2010 and stable for just 6 months. Opera Mobile on Android has been stable for just over a year. IE 9 (the only IE for mobile you could ever seriously consider not serving fallback experiences to) only appeared with Windows Phone 7.5 (aka “Mango”), shipped to users an entire 6 months ago.

And we expect webdevs to have updated all their (maintained) content? Never mind the tenuous correlation between the sorts of soft incompatibilities we’re seeing at the hands of CSS and user attrition; the argument that even this lesser form of harm hasn’t been blunted by evangelism appears suspect. Taking the incompatibilities seriously, I can quickly think of several other measures which are preferable to destroying the positive incentives towards standardization the prefix system creates (from least to most drastic):

• Continued evangelism to web developers with particular focus on major sites
• Political pressure on browser vendors to start dropping prefixes (i.e., we’d all be equally disadvantaged until users pick up the standard version)
• UA spoofing without prefix squatting
• Blacklists to trigger alternative identity (UA/prefixes) on a subset of sites

All of these are less blow-up-the-world than what MSFT, Mozilla, and Opera are proposing. It’s not even an exhaustive list. I’m sure you can think of more. Why these have either been not considered or dismissed remains a mystery.

It’s More Complicated Than That

In all of this, we’re faced with an existential question: what right do web developers have to shoot themselves in the foot? Is there a limit to that right? What sort of damage do we allow when some aspect of the system fails or falls out of kilter for some period of time? It’s a question with interesting parallels in other walks of life (for a flavor, substitute “web developers” with “banks”).

Can we show active harm to other browsers from the use of prefixes? The data is at best unclear. Arguing that any harm rises to a level that would justifies destroying the prefixes system entirely is rash. I argued many of the reasons for this in my last post, but lets assume in our mental model that developers respond to incentives in some measure. If, concurrently with achieving as-yet un-managed distribution, Mozilla et. al. implement others’ prefixes, what should we expect developers to do in response? After all, they will have reduced whatever tension might have been created by content that “looked wonky” and, where standards exist, will have reduced the incentive to switch to the standard version.

Now lets play the model one more turn of the wheel forward too: assume that Chrome or Safari (or both!) act in good faith and contemplate removing the -webkit-* prefix support for standardized features at a quick clip…and Mozilla doesn’t. You see how this quickly leads to a Mexican standoff: web developers won’t stop using prefixed versions because those are the way you get 100% support (thanks to Mozilla’s support for them); vendors won’t un-prefix things because others who squat their prefixes will then have a compatibility advantage; and nobody will be keen to add new things behind prefixes because they can no longer be assumed to be experiments that can change. Lose, lose, lose.

Some on the other side of the debate are keen to cite game theory as a support for their course of action, but the only conclusion I can draw is that their analysis must be predicated on a set of user and developer motivations that are entirely unlike the observable world we inhabit.

A Call To Reason, Not Arms

Based on a better understanding of the landscape, what should the various parties do to make the world better for themselves now and in the long run and for the web as a platform?

• Web Devs: first, do no harm; test in multiple runtimes, pointedly including a “fallback”. Then enhance with prefixes. Do not apologize for giving some (or even many) of your users a better experience. That, after all, is your job. But know this: prefixed properties are not supported, will go away, and when something you didn’t test the fallback for falls over, it’s your fault.
• Browser Vendors: invest in advocacy and distribution enhancing moves for your product before threatening to blow up effective standards policies. If you’re going to implement a prefixed version, please have a different opinion or push to ram a standard through to Recommendation ASAP. Incompatible right-hand-sides help developers understand that things are still evolving. DO NOT squat on prefixes. It’s both relative ineffective and will make developer’s lives harder when they want to legitimately move to standard or support your prefixes.
• Vendor CSS WG Reps: get it through your heads, you’re behind. It’s not quaint and it’s not excusable. The platform needs more powerful CSS features, and stat. It’s long past time to start stealing good ideas from the pre-processors. Appeals to a lack of manpower to implement must never block others and shouldn’t block standardization, so please stop making them. If you care about the platform’s success, let those who are able and willing to take risks do so.
• The CSS WG (as a whole): get the lead out. It’s not exclusively the W3C’s fault that things are slow, but the current MTTR (Mean Time To Recommendation) is still glacial. It is unreasonable to expect vendors to drop prefix support immediately upon standardization, but the W3C has a role here to advocate for quick sunsetting. Daniel Glazman is, as ever, right on most of this, but more can be done to streamline the process post CR.
• The WebKit Project: Add build flags to allow WebKit-based products to enable/disable vendor prefix support independently.
• Chrome/Safari/Other-prefix-supporting-browsers: Sunset prefixes as soon as is practicable post-standardization. Similarly, don’t ship prefixed features you’re not willing to be on the hook for via your reps to the CSS WG. Disabling them may be painful, but it’s the only good-faith thing to do.

Endnotes

I’ve left a lot out of this post, but it’s too long already. I do truly hope it’s the last I write on prefixes because, as I said up front, we have much bigger fish to fry. Stat. Prefixes do work, they’re essential to delivering a future platform that can compete, and yes, they should be torn down at the same pace they’re erected.

A few things that folks have asked about as tangents to this debate:

• It’s never a good thing for there to be homogeneity in the experimentation phase. The explicit goal of the prefixes system is to enable diversity of early opinion and fast coalescing around the best answer, thereby enabling the writing of a standard which is likely to need less revising and iteration. Diversity provides some value, the market tests the alternatives, and we deliver the most value we can over time through the standard version. It has always been thus, but prefixes make it less risky…assuming we don’t start stepping on everyone else’s toes.
• If the reasoning behind prefixes is to set up and tear down large-scale experiments, iterate, and collect feedback then Lea’s -prefix-free approach and PPK’s -alpha-*/-beta-* proposals are equally counter-productive and should be avoided at all costs. Making prefixes less painful to use reduces the natural incentives for migrating to a standard while blindly assuming the same right-hand for a future standard version as we have for some prefixed versions is plainly idiotic. What were they thinking?
• @-vendor-unlock is only slightly smarter, but in every possible way inferior to CSS Mixins. Would that the WG spent as much time on Mixins as they have on this prefix kerfuffle.
• Yes, I was in Paris when the CSS WG F2F was happening. No, I wasn’t at the meetings. Duty (Chrome for Android) called.
• If you’ve read this far, congrats. You may be the only one. I’ve been assured by CSS WG delegates that nobody cares what I think, which statistically seems to just be rounding down by a tiny bit. Fair enough.

Update: Michael Mullany of Sencha adds epicly good, epicly long context about what causes developers to target UAs and what the incentives that’ll change their minds about supporting a given browser really are.

Thanks to Frances Berriman, Jake Archibald, Tony Gentilcore, and Tab Atkins for reviewing earlier versions of this post. Errors of fact and form are all mine, however. Updated occasionally for clarity and to remove typos.

For a while now I’ve been hearing the meme resurface from CSS standards folks and a few implementers that “vendor prefixes have failed”. I’d assumed this was either a (bad) joke or that it was one of those things that web developers would scoff at loudly enough to turn the meme recessive. I was wrong.

Henri Sivonen, Mozilla hacker extrordinare, has made the case directly and at length. Daniel Glazman, co-chair of the CSS WG posted a point-by-point response. If you have the patience, you should read both.

Lost in the debate between “browser people” and “spec people” is the the essential nature of what has happened with prefixes: they worked. From the perspective of a web developer, any first approximation of the history of vendor prefixes are pure win, even if only a fraction of the value that has been delivered behind them is attributable to prefixes un-blocking vendors from taking risks and shipping early.

Daniel’s rebuttal to Henri gets a lot of things right, but he gives in on an essential point; by agreeing with Henri that vendor prefixes are “hurting web authors” he wites off the benefits that they’ve delivered — namely the ability of vendors to get things out to devs in a provisional way that has good fallback and future-proofing properties and the ability for devs to build with/for the future in an opt-in, degradable way.

Rounded corners, gradients, animations, flex box, etc. are all design and experience enablers that developers have been able to take advantage of while waiting for the standards dust to settle, and thanks to W3C process, it takes a LONG time to to settle. Yes, that has some costs associated with it. Henri is very worried that browsers that aren’t keeping up quickly will be “left behind” by webdevs who use only one vendor’s prefix. But surely that’s a lesser harm than not getting new features and not having the ability to iterate. And it provides incentive for following browsers to try to make a standard happen. What’s not to love? More to the point, I just don’t believe that this is a serious problem in practice. What front-ender in 2011 doesn’t test on at least two browsers? Yes, yes, i’m sure such a retrograde creature exists, but they were going to be making non-portable content regardless of prefixes. Assuming you’re testing fallback at all (e.g., by testing on more than one browser), prefixes not appearing on some browser are just the fallback case. CSS FTW! Webdevs who don’t test on more than one browser…well, they’re the ones hanging the noose around the neck of their own apps. Vendor prefixes no more enable this stupidity than the existence of the User-Agent header. Compatibility is a joint responsibility and the best each side (browser, webdev) can hope of the other is some respect and some competence. Cherry picking egregious examples and claiming “it’s hurting the web” seems, at a minimum, premature.

And how did we think we’d get a good standard, anyway? By sitting in a room in a conference center more often and thinking about it harder? Waiting on a handfull of early adopters to try something out in a tech demo and never stress it in practice? That’s not a market test (see: XHTML2), it doesn’t expose developers to the opportunities and tradeoffs that come with a new feature, and doesn’t do anything to address the inevitable need to integrate feedback at some point.

Yes, we could go with Henri’s suggestion that the first person to ship wins by default, never iterate on any designs, and avoid any/all first-mover disadvantage situations, but who among the browser vendors is perfect? And what would the predictable consequences be? I can only assume that Henri thinks that we’ll end up in a situation where vendors coordinate with the CSS WG early to add new stuff, will design things more-or-less in the open, and will only ship features to stable (no flag) when they’re sure of their design. That could happen at the limit, but I doubt it. Instead, the already fraught process of adding new features to the platform will be attempted by even fewer engineers. Who wants the responsibility for having to be perfect lest you screw the web over entirely? Fuck that noise, I’m gonna go work on a new database back-end or tune something to go faster. Browsers are made by smart people who have a choice of things to be working on, and any time you see a new platform feature, it probably came about as the result of an engineer taking a risk. Many times the engineers in a position to take those risks don’t have a great sense for what good, idiomatic web platform features might be designed, so they’ll need to tweak/iterate based on feedback. And feedback is painfully hard to extract from webdevs unless you’ve made something available in a tangible way such that they can use it and discover the limitations. Shipping things only to dev is perhaps a good idea for other aspects of the platform where we can’t count on CSS’s forgiving parsing behavior (the basis for prefixes). Syntax changes for JS and CSS seem like good examples. But for features that are primarily new CSS properties? Oy. Making the stakes even higher, reducing the ability to get feedback and iterate isn’t going to lead to a harmonious world of good, fast standards creation. It’s going to predictably reduce the amount of new awesome that shows up in the platform.

Prefixes are an enabler in allowing the necessary process of use, iteration, and consensus building to take place. Want fewer messes? There’s an easy way to achieve that: try less stuff, add fewer features, and make each one more risky to add. That’s Henri’s prescription, wether or not he knows it, and the predictable result is a lower rate of progress — advocating this sort of thing is much worse for the web and for developers than any of the harm that either Henri or Daniel perceive.

Which brings me to Henri’s undifferentiated view of harm. His post doesn’t acknowledge the good being done by prefixed implementations — I get the sense he doesn’t build apps with this stuff or it’d be obvious how valuable prefixed implementations are for work-a-day web app building — instead focusing on how various aspects of the process of prefixed competition can be negative. So what? Everything worth having costs something. Saying that things “hurt the web” or “hurt web developers” without talking in terms of relative harm is just throwing up a rhetorical smoke screen to hide behind. If you focus only on the costs but write the benefits out of the story of course the conclusion will be negative. In many cases, the costs that Henri points out are correctly aligned with getting to a better world: having to type things out many times sucks, creating demand among webdevs for there to be a single, standardized winner. Having multiple implementations in your engine sucks, creating demand from vendors to settle the question and get the standards-based solution out to users quickly. Those are good incentives, driven by prices that are low but add up over time in ways that encourage a good outcome: a single standard implemented widely.

And as Sylvain Galineau pointed out, what looks like pure cost to one party might be huge value to another. I think there’s a lot of that going on here, and we shouldn’t let it go un-contextualized. The things that Henri sees as down-sides are the predictable, relatively minor, costs inherent in a process that allows us to make progress faster and distribute the benefits quickly, all while minimizing the harm. That he’s not paying the price for not having features available to build with doesn’t mean those opportunity costs aren’t real and aren’t being borne by webdevs every day. Being able to kill table and image based hacks for rounded corners is providing HUGE value, well ahead of the spec. Same for gradients, transitions, and all the rest. Calling prefixed implementations in the wild a bad thing needs to argue that the harm is greater than all of that value. I don’t think Henri could make that case, nor has he tried.

I think the thing that most shocks me about Henri’s point of view is that he’s arguing against a process when in fact the motivating examples (transforms, gradients) have been sub-optimal in exactly the better-than-before ways we might have hoped for! Gradients, for example, saw a lot of changes and browsers had different ideas about what the syntax should be. Yes, it’s harder to get a consistent result when you’re trying to triangulate multiple competing syntaxes, but we got to use this stuff, get our hands dirty, and get most of the benefits of the feature while the dust settled. Huzzah! This is exactly> the way a functioning market figures out what’s good! Prefixes help developers understand that stuff can and will change, and they clear the way for competition of ideas without burdening the eventual feature’s users with legacy bagage tied to a single identifier.

So what about the argument that there might be content that doesn’t (quickly?) adopt the non-prefixed version, or that vendors can’t remove their prefixed implementations because content depends on it?

To the first, I say: show me a world where 90+% of users have browsers support the standard feature and I’ll show you a world in which nobody (functionally) continues to include prefixes. That process is gated in part by the WG’s ability to agree to a spec, and here I think there’s real opportunity for the CSS WG to go faster. The glacial pace of CSS WG in getting things to a final, ratified spec is in part due to amazingly drawn-out W3C process, and in part a cultural decision on the part of the WG members to go slow. My view is that they should be questioning both of these and working to change them, not blaming prefixes for whatever messes are created in the interim.

As for removing prefixes, this is about vendors just doing it, and quickly. But the definition of “quickly” matters here. My view is that vendors should be given at least as long as it took to get a standard finalized from the introduction of their prefixed version for the removal process to be complete. So if Opera adds an amazing feature behind a -o- prefix in early 2012 and the standard is finalized in 2014, the deprecation and eventual removal should be expected to take 2 years (2016). This has the nice symmetry of incentives that punish the WG for going slow (want to kill prefix impls? get the standard done) while allowing the vendors who took the biggest risks to provide the softest landings for their users. And it doesn’t require that we simply go all-in on the first person’s design to ship. Yes, there will be mounting pressure to get something done, but that’s good too!

The standards process needs to lag implementations, which means that we need spaces for implementations to lead in. CSS vendor prefixes are one of the few shining examples of this working in practice. It’s short-term thinking in the extreme to either flag the costs associated with them as either justifying their removal or even suggesting that the costs are too high.

And webdevs, always be skeptical when someone working on an implementation or a spec tells you that something is “hurting the web” when your experience tells you otherwise. The process of progress needs more ways to effectively gauge webdev interest, collect feedback, and test ideas. Not fewer or narrower channels.

Which it is not.

Before I get to why, let me first say some nice, well-earned things about WebIDL: first, it has helped us out of the ad-hoc IDL sludge that used to be how APIs for JavaScript have been exposed in the past. It has shaved off many sharp edges and is giving spec authors a single dialect in which to write their API descriptions. From a browser perspective, this is a Very Good Thing (TM). Next, the draft in question contains some wonderful changes from the status quo, particularly the addition of a sane prototype to all WebIDL-specified objects.

That all sounds good, so what’s missing?

In a word, constructors.

Well, a lot more than that, but I’d settle for constructors. Functionally speaking, it boils down to the fact that WebIDL makes spec authors do extra work to make something like this sane:

new HTMLDivElement();

Why doesn’t this work today? Funny story…see, HTML defines HTMLDivElement as a regular WebIDL interface. WebIDL doesn’t really have the notion of concrete classes, just interfaces with and without constructors. Since the HTML spec is just doing what most specs will do — adding the smallest IDL you can get away with — the JS usability of this API is left in limbo; neither clearly HTML5′s responsibility nor WebIDL’s.

So what should a contentious version of HTML5 do? One answer is to specify a constructor, turning the IDL from this:

interface HTMLDivElement : HTMLElement {};

to this:

[Constructor]
interface HTMLDivElement : HTMLElement {};

Repeat ad-infinitum for each and every interface that should be constructable in every single spec that browser vendors ever implement. Don’t miss any! And please make sure that all your spec editors are on-board with good JS APIs as a goal! As it stands today, WebIDL doesn’t even force most spec authors to consider the question “do I need a constructor here?” — spoiler: yes — let alone the obvious follow-ups like “what arguments should one take?”.

The obvious better answer here is to flip the default on interfaces, causing them to generate constructors by default unless turned off with [NoConstructor] attributes or specified as partial interfaces (i.e., mixins or traits).

Cameron McCormack who is heading up the WebIDL effort tweeted in response to my exasperation that:

I think a “W3C Web API design guidelines” document would be a perfect place for such a recommendation.

For serious? Such a document might be useful (and I’m working on something that might pass as a first draft), but what’s the argument against flipping the default here? This isn’t a dissent on the facts of the situation: most WebIDL “interfaces” that are exposed to JS are things that could be easily new‘d up to useful ends. Most specs flub this in spectacular style. Most spec authors seem entirely ignorant of the problem and the design language of WebIDL continues to lead down a primrose copy-and-paste path that has little overlap with sanity. So why punt the decision? And why did it take and act of coordination with TC39 to get the prototype thing fixed?

And Why Are We Having This Discussion Anyway?

WebIDL, for all of its virtues, is deeply confused.

If you’re reading any of the stuff in the HTML5 spec that’s describing its API this way, it’s hard to see how it would have any sane relationship to JavaScript. Sure, you could argue that there might be other languages that matter, other languages for which you’d need to be able to generate some API, but none of them rise to anything like the importance of JavaScript. It is the programming language of the web, so if WebIDL has any animating force at all, it’s JS. Then there’s the “accident of history” aspect. Early DOM was specified as a form of IDL in part because there was some expectation that other languages would need to consume it and IDL was how C++ hackers (who still make up the entire set of people working on browser engines) are/were comfortable in describing their FFIs thanks to the legacy of COM/CORBA. Hilarious examples of multi-language-ism still persist in the WebIDL spec for no apparent reason whatsoever, warping the entire design around the altar of an ideal that is either quixotic or vestigial depending on which argument you give more weight.

Since the debate was re-kindled thanks to a debate at a TC39 meeting in July, I’ve been on the receiving end of more than one webdev’s rant about DOM’s JS incoherence, generally taking the form:

Why the *#!*?^@$ isn’t DOM just #@!*@ing specified in JavaScript?

To be honest, I have no answer aside from pointing to the IDL history, the fact that browser hackers don’t tend to write JS so don’t feel the pain, and noting that WebIDL is better in some important ways. Certainly these interfaces could be specified in a subset of JS with annotations for where native behavior is required. But their larger lament has merit too: seamlessness with JS is the bar WebIDL should be judged by. I.e. does it help spec authors do the right thing by JS devs? Or does it lead them down paths that make their generated APIs stick out like sore thumbs, full of anti-social/alien behavior such that you can’t think of them as “regular” JS?

Yes, constructors are only one minor step toward reaching this aspiration, but the fact that WebIDL has gotten to last-call without a reasonable solution to them speak volumes. If WebIDL isn’t animated by the need of JS developers, it would be good if that could be loudly called out somewhere so that the community can have the spirited debate that this point demands. If it is, can we please get on discussing how best to ensure that most “interfaces” generate constructors and stop punting?

Either way, WebIDL isn’t done yet.

Update: It occurred to me, as part of the discussion in the comments, that the provision against new with a class or type of any type is completely non-sensical in JS, as is the lack of call() and apply() methods on them. Idiomatic subclassing requires that the mixin-style be available, which uses ClassName.call(this). This is what you’d do with things that are “virtual” or “partial” interfaces if you’re describing them in actual JS. And there’s no real problem with folks new-ing them up. Doesn’t happen, doesn’t matter. Strong restrictions against it are, to quote Andrew DuPont, anti-social. Long story short: there is absolutely no reason whatsoever to disable construction on any WebIDL interface. It’s deeply non-sensical from the JavaScript perspective.

Making the rounds is an accidentally leaked early draft of notes from a meeting last year that discusses both Dart and JavaScript. I work on many web platform-related things at Google, including serving as a representative to TC39, the body that standardizes the JavaScript language. I wasn’t at the meetings having previously committed to presenting at FFJS, but my views were represented by others and my name is on the document. As I said, though, it was a draft and doesn’t reflect either the reality of what has happened in the meantime or even the decisions that were taken as a result. And it certainly doesn’t reflect my personal views.

So what’s the deal with Google and JavaScript?

Simply stated, Google is absolutely committed to making JavaScript better, and we’re pushing hard to make it happen.

Erik Arvidsson, Mark Miller, Waldemar Horwat, Andreas Rossberg, Nebojša Ćirić, Mark Davis, Jungshik Shin and I attend TC39 meetings, work on implementations, and try to push JS forward in good faith. And boy, does it need a push.

Erik and I have specifically been working to focus the TC39 agenda away from syntax-free, semantics-only APIs (Object.defineProperty, anyone?) which might be good for tools, compilers, and frameworks but which are hard for day-to-day use.

Through Traceur and other efforts we’ve been socializing the idea that the one thing the committee can exclusively do — and should do more of — is to carve out syntax for commonly exercised semantics. Seemingly small things like the class keyword as sugar for the constructor function pattern, or a shorter syntax for functions are big improvements, if only because it’s TC39 that’s making them. Syntax can end battles over common patterns and help you say what you mean, and JS is overdue for some of this. Larger but more subtle things, like agreement to use pragmas as a way to enable the process of progress in a compatible web, are even more exciting to me. Even proposals that haven’t made it through like Scoped Object Extensions and deferred functions have been fought for by us because we desperately want JavaScript to get better. We’re big fans of much of the work coming out of Mozilla for the same reason — in particular Dave Hermann’s excellent modules proposal. Together, these are going to do much to help give modern JS some “backbone” in the near term. I’m proud of what we’ve accomplished in TC39 over the last year, and while I hoped for more, the result is an ES.next that looks like it’ll embody many of the things I feel are currently missing. The day-to-day practice of writing JS is going to change dramatically for the better when ES.next arrives.

But it’s not the end — not by a long shot. Classes will give us a humane, interoperable inheritance syntax, but it leaves composition unaddressed by syntax. I’m hopeful that we bless traits in future versions, removing the use of inheritance in most cases. Similarly, I think we can find a way to repair “this” binding foot-guns with softly-bound “this”. Repairing the shared-prototypes issue, either through DOM or through something like Scoped Object Extensions, can and should be done. And once we have all of this, the stage will be set for a flexible, advanced type system that does not need to be all-or nothing and does not need to be hobbled by the ghost of C++/Java’s inflexible nominal-only types. That’s the dream, and we’re not shying away from it.

It’s hard to square this sort of wild enthusiasm for “raw” JavaScript with what’s in the leaked memo, and I can only beg for some amount of understanding. As committed and enthusiastic as I am about the prospects for JavaScript, others are just as enthused about Dart. Google is big, can do many things at once, and often isn’t of one mind. What we do agree on is that we’re trying to make things better the best we know how. Anyone who watches Google long enough should anticipate that we often have different ideas about what that means. For my part, then, consider me and my team to be committed JS partisans for as long as we think we can make a difference.

Reality Check

There are risks, of course. TC39 is long on seasoned language design skill and short on webdev experience, meaning that many things that Erik and I may take for granted as pressing problems need to be explained, sometimes to an incredulous audience. The flip side risk is that naïve solutions may have better alternatives that seasoned language hands can quickly spot and that simple answers have non-obvious risks or preclude movement in other important areas later. It’s good, then, that the committee is working well and is taking appeals to developer productivity seriously.

Whatever you might think about programming languages for the browser, let me assure you of one thing: your problem isn’t the language. Not really, anyway. We’ve made good progress in the last year repairing some of the seams between JS and DOM, and Cameron McCormack has helped us drive a new version of WebIDL that correctly explains DOM as a reasonable prototype chain. But it’s only the beginning. The DOM is in terrible shape, and not due to implementation differences. The malign neglect of IDL-addled designs, the ghosts of dead-end XML experiments, and endless cruft will plague any language that sidles up to it. Until we get a real Component Model, a better CSS OM, and some sort of a pragma for DOM that allows us to fix DOM’s abhorrent JS bindings, we’ll continue to be hostage to C++ APIs inartfully wired up to an incredibly dynamic language. And that’s to say nothing of the pressing need for better CSS, animations, and a built-in data-binding/templating system. When the platform doesn’t provide it, today, we get it from JavaScript. But that’s not where the solutions always belong.

Let me put it another way: when you find yourself thinking “man, JavaScript sucks,” remember that it’s only painful in large quantities. And why do you need so much of it? ‘Cause the DOM, CSS, and HTML standards are letting you down. Any language wired up to the browser today is subject to the same fate, and the insane reality that these things are specified under different roofs in processes that aren’t subject to the popular will of web developers. Python doesn’t have it’s DOM APIs decided by the W3C, they borrow the idiomatic ElementTree API from within their own community. WebIDL is an artifact of a different time that has a tenuous relationship to idiomatic JavaScript, the CSS-OM barely exists, and DOM apologists are doing more harm than any of JavaScript’s warts ever have. We can fix these things, of course, and here at Google we’re trying – in good faith – to work in standards to make them better. But the bottom line is that the language isn’t the problem. I repeat: the language isn’t the problem, the platform is.

The only thing that’s going to replace the web as universal platform is the next version of the web. Those of us working on Chrome believe that to the core and feel a deep urge to make things better faster. We might not always agree on the “how,” but we all believe that we can’t do it alone.

It’s good to see MSFT belatedly trying to put IE6 out to pasture, but what about IE 7? Or 8? Lets take stock of where we really are and where we’re likely to be in the next couple of years. First, remember that there’s no IE 9 for Windows XP — an OS that’s currently the most popular in the world — and no matter what happens with IE 6, IE 8 is the end of the upgrade road for XP. Unless you think half of the world’s computers will be replaced/upgraded in the next couple of years, it seems likely that IE 8 will be with us for the foreseeable future.

And what about the folks who do get IE 9? Well, so far, there’s nothing to make me believe that the uptake rate will be anything better than the IE 8 transition; a process which has taken 2 years to give ~30% of the market the latest version. If anything, we should expect that rate to be retarded somewhat by the XP hurdle.

MSFT’s browser replacement rates bear understanding because they’re the most popular and suffer from the longest half-lives. That is to say, the time it takes for an old version of IE to decay in the wild is much, much higher than for other browsers. Some part of this is surely due to sheer market share, but not all of it. The XP hurdle, for instance, is a form of structural drag on uptake rates — a flaw that browsers that aren’t tightly tied to OSes don’t suffer from. For web developers, I dare say that half-life of popular browsers matters much, much more than the current or trending market share since it’s predictive of our potential for browser improvement in the near future. It’s one thing to get the new shiny, but how long will it take you to install it? If the shiny is old and dingy by the time it’s in place, what good is that? It’s this lens that makes browser market share stats interesting to me; i.e., what percentage of the web’s users will get the new features soonest? ‘Cause those are the folks we can start building super compelling content for.

The average half-life of the majority of browsers in the wild also gates the rate of progress in standards. When the process is working well, bugs in browsers or pre-standards implementations of features aren’t a permanent features of the landscape. Instead, they’re the understandable and inevitable result of a process that prioritizes implementation experience and iteration over raw compliance with an academic spec that may or may not actually get it right on the first go ’round. But that iterative, feedback-rich process only works when browsers iterate quickly and web developers can target the future without thinking so hard about the past, else progress simply turns into something to resent and distrust. That’s good for no one, and a shorter half-life is the key to making progress more than just a spec-tease.

I’m personally hopeful that when IE 9 is finally RTM’d, that it includes some provisions for shortening its life expectancy in the ways that Chrome and Firefox have through aggressive auto-updating. Getting IE 9 out to the world will be a good thing, but only if it happens quickly and if IE 10 can follow it even faster.

There’s obviously a lot more to talk about at the browser panel — Chrome 10 just launched with Crankshaft, for instance — but the fact that nearly every Chrome user will have those improvements this week and that if you’re building a Chrome Web Store app, you’ll get to target those improvements nearly instantly seems like the biggest, most interesting change from where we were just a couple of years ago.

Wait, what? Why does Alex Russell hate feature testing, kittens, and cute fuzzy ducklings?

I don’t. Paul warned me that my approach isn’t going to be popular at first glance, but hear me out. My assumptions are as follows:

• Working is better than busted
• Fast is better than slow
• No browser vendor changes the web-facing features in a given version. Evar. Does not happen

If you buy those, then I think we can all get some satisfaction by retracing our steps and asking, seriously, what is the point of feature testing?

Ok, I’ll go first: feature testing is motivated by a desire not to be busted, particularly in the face of new versions of UA’s which will (hopefully) improve standards support and reduce the need for hacks in the first place. Sensible enough. Why should users wait for a new version of your library just ’cause a new browser was released or because you didn’t test on some version of something.

Extra bonus: if you don’t mind running them every time, you can write just the feature test and your work is done now and in the future! Awesome! Except some of us do mind. Yes, things are now resilient in the face of new UA’s and new versions of old ones, but only on the back of testing for everything you need ever time you load a library on a page. Slowly. Veeerrrrry slooowly.

Paul suggested that some library could use something like Local Storage to cache the results of these tests locally, but this hardly seems like an answer. First, what if the user upgrades their browser? Guess you have to cache and check against the UA string anyway. And what about the cost of going to storage? Paul reported that these tests can be wicked expensive to run at all, on the order of 30ms for the full suite (which you hopefully won’t hit…but sheesh). Reported worst-case for has.js is even worse. But apparently going to Local Storage is also expensive. And we’re still running all these tests in the fast path the first time anyway. If we think that they’re so expensive that we want to cache the results, why don’t we think they’re so expensive that we don’t want to run them in the common case?

Now for a modest proposal: feature tests should only ever be run when you don’t know what UA you’re running in.

Feature testing libraries should contain pre-built caches — the kind that come with the library, not the kind that get built on the client — but they should only be consulted for UA versions that you know you know. If we assume that behavior for UA/version combination never changes, we’ve got ourselves a get-out-of-jail free card. Libraries can have O(1) behavior in the common case and in the situations where feature testing would keep you from being busted, you’re still not busted.

So what’s the cost to this? Frankly, given the size of some of the feature tests I’ve seen, it’s going to be pretty minimal vs. the bloat the feature tests add. All performance work is always a tradeoff, but if your library thinks it’s important not to break and to be fast, then I don’t see many alternatives. New versions of libraries can continue to update the caches and tests as necessary, keeping the majority of users fast, while at the same time keeping things working in hostile or unknown environments.

Anyhow, if you’re a library author or maintainer, please please please consider the costs of feature tests, particularly the sort that mangle DOM and or read-back computed layout values. Going slow hurts users, hurts the web, and hurts the culture of performance that’s so critical to keeping the platform a viable contender for the next generation of apps. We owe it to users to go faster.

A quick aside: I hesitated writing this for the same reasons that Paul cautioned me about how unpopular this was going to be: there’s a whole lot of know-nothing advocacy that’s still happening in the JS/webdev/design world these days, and it annoys me to no end. I’m not sure how our community got so religious and fact-disoriented, but it has got to stop. If you read this and your takeaway was “Alex Russell is against feature testing”, then you’re part of the problem. Think of it like a feature test for bogosity. Did you pass? If so, congrats, and thanks for being part of the bits of the JavaScript universe that I like.

How bad is it? The next time you read some tech journalist write about how some new browser version is just around the corner and how it’ll make everything better, remember that:

• 50+% of Windows users are on XP a year after Windows 7 shipped and 3.5 years since Vista shipped
• Windows XP will be supported until 2014, giving organizations on XP extra breathing room to limp along on IE 6-8
• There will be no IE 9 for Windows XP
• After something like 4 years of MSFT urging customers in the strongest possible language to get off of IE 6, it still has 16% of the market, and it’s not falling nearly fast enough. At the current 0.75%/month dropoff, we’re looking at 20+ more months of IE 6. At this rate, kids born today will be walking and maybe even talking by the time we can write IE 6 into the history books.

The goal here isn’t to drive you to drink, but to call out the dichotomy between when users get features and when developers can address features. For users, things tend to get better as soon as you pick up a new browser. Developers have to wait until every user makes that sort of choice. Said differently, users can benefit from the bleeding edge but developers are beholden to the late adopters.

We should cut tech journalists and users a little slack, though; in a world without adoption friction, the launch of a new feature would translate directly into the sort of “now you can build sites and apps with awesome thing X”. The instinct to believe that’s how it should be is spot on. But that’s observably not how the world works. Don’t believe the hype. New browsers alone haven’t fixed the problem so far, so what makes us think they will in future? No vendor wants to talk about their last version, but for web developers stuck in the trenches, that all there is to talk about. That’s where the pain is, after all, and counting on browser upgrades to fix the problem quickly isn’t working well enough. Chrome’s aggressive auto-update feature is changing the way things will work in future, but for now we’re still stuck in the slow-upgrade dynamic.

We need a Plan B.

First, my position: ceteris paribus, view-source was necessary (but not sufficient) to make HTML the dominant application platform of our times. I also hold that it is under attack — not least of all from within — and that losing view-source poses a significant danger to the overall health of the web.

That’s a lot to hang on the shoulders of a relatively innocent design decision, and I don’t mean to imply that any system that has a view-source like feature will become dominant. But I do argue that it helps, particularly when coupled with complementary features like reliable parsing, semantic-ish markup, and plain-text content. Perhaps it’s moving the goal line a bit, but when I talk about the importance of view-source, I’m more often than not discussing these properties together.

To understand the importance of view-source, consider how people learn. Some evidence exists that even trained software engineers chose to work with copy-and-pasted example code. Participants in the linked study even expressed guilt over the copy-paste-tweak method of learning, but guilt didn’t change the dynamic: a blank slate and abstract documentation doesn’t facilitate learning nearly as well as poking at an example and feeling out the edges by doing. View-source provides a powerful catalyst to creating a culture of shared learning and learning-by-doing, which in turn helps formulate a mental model of the relationship between input and output faster. Web developers get started by taking some code, pasting it into a file, saving, loading it in a browser and hitting ctrl-r. Web developers switch between editor and browser between even the most minor changes. This is a stark contrast with technologies that impose a compilation step where the process of seeing what was done requires an intermediate step. In other words, immediacy of output helps build an understanding of how the system will behave, and ctrl-r becomes a seductive and productive way for developers to accelerate their learning in the copy-paste-tweak loop. The only required equipment is a text editor and a web browser, tools that are free and work together instantly. That is to say, there’s no waiting between when you save the file to disk and when you can view the results. It’s just a ctrl-r away.

With that hyper-productive workflow as the background, view-source helps turn the entire web into a giant learning lab, and one that’s remarkably resilient to error and experimentation. See an interesting technique or layout? No one can tell you “no” to figuring out how it was done. Copy some of it, paste it into your document, and you’ll get something out the other side. Browsers recovering from errors gracefully create a welcome learning environment, free of the inadequacy that a compile failure tends to evoke. You can see what went wrong as often as not. The evolutionary advantages of reliable parsing have helped to ensure that strict XML content comprises roughly none of the web, a decade after it was recognized as “better” by world+dog. Even the most sophisticated (or broken) content is inspectable at the layout level and tools like Firebug and the Web Inspector accelerate the copy-paste-tweak cycle by inspecting dynamic content and allowing live changes without reloads, even on pages you don’t “own”. The predicate to these incredibly powerful tools is the textual, interpreted nature of HTML. There’s much more to say about this, but lets instead turn to the platform’s relative weaknesses as a way of understanding how view-source is easily omitted from competing technologies.

The first, and most obvious, downside to the open-by-default nature of the web is that it encourages multiple renderers. Combined with the ambiguities of reliable parsing and semantics that leave room for interpretation, it’s no wonder that web developers struggle through incompatibilities. In a world where individual users each need to be convinced to upgrade to the newest version of even a single renderer, differences only in version can wreak havoc in the development process. Things that work in one place may not look exactly the same in another. This is both a strength and a weakness for the platform, but at the level of sophisticated applications, it’s squarely a liability. Next, ambiguities in interpretation and semantics mean that the project of creating tooling for the platform is significantly more complex. If only one viewer is prevalent (for whatever reason), then tools only need to consume and generate code that understands the constraints, quirks, and performance of a single runtime. Alternate forms of this simplification include only allowing code (not markup) so as to eliminate parsing ambiguity. The code-not-markup approach yields a potentially more flexible platform and one that can begin to execute content more quickly (as Flash does). These advantages, taken together, can create an incredibly productive environment for experts in the tools that generate content: no output ambiguity, better performance, and tools that can deliver true WYSIWYG authoring. These tools can sidestep the ctrl-r cycle entirely.

But wait, I hear you shout, It’s possible to do code-only, toolable, full fidelity development in JavaScript! Tools like GWT and Cappuccino generate code that generates UI, ensuring that only those who can write code or have tools that can will participate; removing the potential value of view-source for those apps. But lets be honest: view source is nearly never locally beneficial. I can hardly count the number of times I’ve seen the “how do I hide my code?” question from a web n00b who (rightly or wrongly) imagines there’s value in it. For GWT the fact that the output is an HTML DOM that’s styled with CSS is as much annoyance as benefit. The big upside is that browsers are the dominant platform and you don’t have to convince users to install some new runtime.

Similarly Flex, Laszlo, GWT’s UI Binder, and Silverlight have discovered the value in markup as a simple declarative way for developers to understand the hierarchical relationships between components, but they correspond to completely unambiguous definitions of components they rely on compiled code — not reliably parsed markup — for final delivery of the UI. These tight contracts turn into an evolutionary straightjacket. Great if you’re shipping compiled code down the wire that can meet the contract, but death if those tags and attributes are designed to live for long periods of time or across multiple implementations. You might be able to bolt view-source into the output, but it’ll always be optional and ad-hoc, features that work against it being pervasive. Put another way, the markup versions of these systems are leaky abstractions on the precise, code-centric system that under-girds both the authoring and runtime environments. This code-centric bias is incredibly powerful for toolmakers and “real” developers, but it cuts out others entirely; namely those who won’t “learn to program” or who want to build tools that inject content into the delivery format.

Whatever the strengths of code-based UI systems, they throw web crawlers for a loop. Today, most search engines deal best with text-based formats, and those search engines help make content more valuable in aggregate than it is on its own. Perhaps it’s inevitable that crawlers and search engines will need to execute code in order to understand the value of content, but I remain unconvinced. As a thought experiment, consider a web constructed entirely of Flash content. Given that Flash bytecode lacks a standard, semantic way to denote a relationship between bits of Flash content, what parts of the web wouldn’t have been built? What bits of your work would you do differently? What would the process be? There’s an alternate path forward that suggests that we can upgrade the coarse semantics of the web to deal with ever-more-sophisticated content requirements. Or put another way, use the features of today’s toolkits and code generators as a TODO list for markup driven features. But the jury is still out on the viability that approach; the same dynamic that makes multiple renderers possible ensures that getting them to move in a coordinated way is much harder than the unilateral feature roadmap that plugin vendors enjoy. HTML 5 and CSS 3 work is restarting those efforts, but only time will tell if we can put down the code and pick markup back up as a means to express ourselves.

I’ve glossed over a lot of details here, and I haven’t discussed implications for the server side of a non-text format as our lingua-franca, nor have I dug into the evolution metaphor. Many of the arguments are likewise conditional on economic assumptions. There’s lots of discussion yet to have, so if you’ve got links to concrete research in either direction or have an experience that bears on the debate post in the comments! Hopefully my fellow panelists will respond in blog form and I’ll update this post when they do.