Infrequently Noted

Alex Russell on browsers, standards, and the process of progress.

App Engine: Most Of The Stuff I Want, None Of The Stuff I Don't

I'm not sure who or how, but I got an invite to yesterday's Camp Fire One event at Google where they announced their new App Engine platform. The event itself was small-ish, with lots of interesting people invited (both from Google and not). I had no idea ahead of time what the announcement would be, and frankly I forgot all about the event until the day of. I'm glad I remembered (note to future self: next time, pack gloves and a hat).

As they started talking about the platform and what's part of it (and what's not), I couldn't escape escape the feeling that they'd gotten it right. It is absolutely the case that for most apps, scaling requires some amount of re-architecting. Systems like Rails are built with such a dependence (intentional or not) on the features of relational data stores that they quickly hit bottlenecks because frameworks aren't keeping developers out of the gutter. This is nearly the same lesson that the security community collectively came to when it started to beat the average developer about the head regarding the awesome power of defaults. What systems do and don't do for you "cheaply" defines their character, and in many systems those choices aren't made consciously, or if they are, they don't have the benefit of a different perspective which might de-emphasize certain traits. Call it libertarian paternalism, choice architecture, or pure condescension, but whatever it is, systems and platforms today are in the explicit business of making some things easier than others.

As the presenters showed how to make an app quickly, I knew I was looking at something I hadn't seen the likes of since Jot. We all know that Big Table is a column-oriented data-store, but since most people are still stuck on the likes of MySQL, it's hard to appreciate how liberating it is not having to think about how adding properties to models will affect a schema. The way App Engine is constructed lets the data store layer provide something called expando models. These models give us the kind of flexibility that I've only ever enjoyed before inside of Jot. Want a property? Just add it. No migration, no schema version...just data finding a happy home, and as your app's skeleton "solidifies" and you figure out which properties really do need to be faster, you can migrate that data into fixed properties with indexes and types and all that jazz. It's like a gradual type system, only for data stores. It's agility nirvana for application development.

Speaking of application development nirvana, they also had the good sense to start with a great language (Python) and a great webapp framework (Django) as the basis for the new system. For all the Rubyists and Java heads out there who are surely crying bloody murder, I suggest that they just try it. Seriously. Django's template system is freaking sweet, and Python (despite it's lambda-related warts) is close enough to being executable pseudo-code as to still hold the second place in my toolbox of languages.

There's a lot which I'm sure others will (and have) covered about how App Engine is going to change the game for startups and players like Amazon, but I think that if someone else had launched this system it would still survive on its merits alone. The only wrinkle in the whole thing will be seeing what's done about pricing over the long haul. It really shouldn't be hard for Google to beat S3 on price, and I'm sure there will still be a market for EC2 for non-traditional tasks, but fundamentally I think App Engine has all the makings of something really, truly better than the current (assumed) stack.

After more than a year of mourning the effective loss of Jot as a platform, writing apps on the server just got interesting for me again, and that may be the highest praise I can offer and framework or platform.

Dojo 1.x: Industrial Strength Open Web Tech

...just ask AOL. James Burke just pinged me to let me know that AOL Webmail has been updated with a slew of new features built using Dojo 1.0. A quick inspection of the app shows all kinds of great stuff including tons of custom widgets as well as extensive use of the Grid. To keep loading of the app quick, AOL is using custom builds to pull Dojo from a CDN and a number of application-specific layers in order to defer loading of code until it's needed. Congrats to the AOL Mail team! It's inspiring to see a site that does billions of page-views a month using Dojo so effectively.

Dojo 1.x is now powering the UI of the world's largest mapping service, one of the world's largest email services, the most useful personal information service anywhere, and the front-end of my favorite RSS reader.

The proof of Dojo 1.x's quality really is in the pudding. Congrats again to the AOL Webmail team!

Update: Travis Vachon also just sent mail to let me know that OSAF's Cosmo project has also released their 0.14.0 version which has also made the upgrade to Dojo 1.0. Congrats, Cosmologists!

Dojo 1.1: Some Awesome For You App

I could go on for a long, long time about what's great in Dojo 1.1...but I'll spare you most of that. James, Pete, Dylan, and the release notes can give you a strong sense of why Dojo 1.1 is the most polished, fastest, and easiest-to-use release of Dojo we've ever done. For the impatient, you can already start using it from the CDN without downloading anything.

I should mention a couple of Core features from 1.1 that might otherwise go overlooked, though. The first is a lack of visible change. Dojo Core and Dijit from 1.1 are fully backwards compatible with 1.0. We promised that the fundamental incompatibility between 0.4.x and 0.9.x+ was a one-time change, and Dojo 1.1 keeps that promise. We've already had reports during the RC cycle of people swapping out Dojo 1.1 for 1.0 without any changes to their apps. It takes dedication and discipline across the entire team to achieve that kind of API stability while still taking risks to deliver better features, reliability, and performance.

Next up, we updated the animation APIs significantly. In Dojo 1.0, we moved to a unified timing loop for animations which helped to significantly improve the performance of Dojo animations. In 1.1, we've again improved the performance of the animation system but have also added some great syntactic sugar to the already powerful APIs which we expose. As always, start and end coordinates for an animation can be values or functions which return calculated starting and ending positions. Now, though, you can elide away the { end: 30 } structure and just provide a value. This lets us go from:

dojo.animateProperty({
    node: "thinger",
    duration: 500,
    properties: {
        width: { end: 500 },
        height: { end: 500 }
    }
}).play();

To just:

dojo.animateProperty({
    node: "thinger",
    duration: 500,
    properties: { width: 500 , height: 500 }
}).play();

But to get even more terse, we'll need a different API structure that doesn't require so much explicit argument packing. Dojo 1.1 adds just such an API:

dojo.anim("thinger", { width: 500, height: 500 }, 500);

Note that in addition to being able to drop a lot of stuff out of the function call, dojo.anim also doesn't require that we call .play() on its returned animation object since in the common case you'll just want to play right away.

Next up, Dojo now has a unified dojo.xhr() function that covers a lot of bases and gives you a single entry point into the various bits of XHR goodness contained in dojo.js.

As I've noted elsewhere, Dojo is also now supporting querySelectorAll on the browsers that support it sanely. Dojo's CSS engine has always been fast, and by keeping our query syntax to just what CSS provides, we've avoided getting ourselves into a situation where we'll always need to be shipping such a query engine down the wire. Sooner or later, dojo.query() will become nothing more than a call into querySelectorAll plus some syntactic sugar on the returned array. Best yet, API won't change and you can get the speedup on the browsers that support it now, knowing full-well that things will only get faster and smaller in the future. An investment in a toolkit that is pays attention to the evolution of the web is already paying dividends for Dojo users.

Lastly (for now), there have been some fun additions to the API of dojo.NodeList, the thing that's returned out of every call to dojo.query(). dojo.attr() and dojo.anim() are now available on groups of nodes. For instance, we can make a group of elements tab-focusable on browsers that support it and then draw some attention to them:

dojo.require("dojo.NodeList-fx");

dojo.query("#nav > .focusable"). attr("tabIndex", 0). style("border", "1px solid transparent"). anim({ "borderColor": { start: "yellow", end: "white" } });

There's also a new empty() method on node lists which makes removing children simpler, and the instantiate() method which helps you create instances of a class for every item in the list. Lets say you want to encapsulate some behavior in a class. You could use the hotness that is dojo.behavior or you could use the traditional combination of the Dijit base classes plus the Core parser system. Or you could strike out on your own, particularly for implementing something like a microformat handling library:

// define a Card class which we can create instances of
dojo.declare("mf.Card", null, {
    constructor: function(props, node){
        this.setAddrs(dojo.query("> .adr", node));
        this.setName(dojo.query(".fn, .n", node));
        // ...
    },
    name: null,
    addrs: [],
    tels: [],
    emails: [],
    // ...
});

dojo.query(".vcard").instantiate(mf.Card);

Taken together, these extensions make dojo.query significantly more powerful. There's also a growing set of extensions to node lists, such as those for easy templating of generated content via dojox.dtl. We've also taken care to ensure that anywhere that Dojo returns you an array of nodes, it'll probably be handing you a dojo.NodeList. That means you can treat these arrays as regular, intrinsic Array instances, or you can use these chainable methods for added node-manipulating power.

An Aside

The team that put together Dojo 1.1 is really, truly, amazing and it shows in the end product.

When it became clear that we had a responsibility to the a11y of Dojo application, they built it and shipped solutions. While others still aren't thinking about localization, they built it and shipped solutions to right-to-left as well as string handling (including optimizing those processes for deployment). When it became clear that the CSS file structure for building themes was too hard, they fixed it...in a 100% backwards compatible way. When Dojo was "dinged" for poor documentation, they did what they knew was the right thing and kept working on what is now the world's best JavaScript documentation tool.

What's most impressive to me about the Dojo team, though, is that this isn't a centrally planned, in-house operation by a giant company. There isn't a product road-map other than what's decided at the Developer Day events and in the weekly IRC meetings. Dojo is the collective work of volunteers, committers, and companies that understand that building things together in a truly open project is better for everyone. Having the source of a product be available under an OSS license gives you certain rights in an instantaneous sense, but for those advantages are usually just advantages for people already "inside" of a project. To really convert on the benefits of openness, the process of developing the product needs to be just as open as the license. Open Source business models all tacitly acknowledge that most pithy of truisms: the people on a project are its greatest asset. If new people can't join or "get in", then the project is effectively closed, and the benefits of openness can't be fully realized. Users win when advocates for their needs can emerge from more places, developers win when they can get involved and make a real difference, and a project benefits its community is really that: a group of people who show up because they want to make things better. I'm proud to be working with just such a dedicated group on Dojo.

During the 1.1 release cycle, Wolfram Kriesing and Nikolai Onken earned their committer stripes. They didn't start making Dojo better because they happened to work somewhere or because their company sells Dojo...they joined the project the way most of us did: they saw a place where they could contribute and because Dojo is run as an open project, they were able to have a huge impact. Nikolai's artwork and theme wrangling are largely responsible for Dojo 1.1 being the most attractive Dojo ever and Wolfram's code contributions have made interfacing services with dojo.data stores easier than ever before.

We've got an ambitious set of improvements slated for 1.2, and I'm confident that while it always looks slightly chaotic, the advantages of working in the open with whoever shows up will continue to pay off for everyone. If you've been a Dojo user and have thought about contributing, there's never been a better time to come find us on IRC or the forums and help to shape the future of the toolkit. You can be part of this amazing team too. It's as simple as that, really.

I can't wait to work with you to make our corner of the Open Web better.

Progress Is N+1

Not sure how I got there, but I just stumbled over this bit of dark humor at Joel Spolsky's expense, and in reading it I was reminded of a discussion last Friday where I voiced my frustration that as much as IE 8 looks to be a good point release, we know next to nothing about it's intentions with regards to ship dates or auto-update deployment. I'm not talking exact dates or firm plans here, just "within the next N months" or "we'll wait N (plus or minus 3) months to put it on Windows Update". Knowing those things fill in the missing bits of making any plans around IE. No plan is complete without a "who", a "how", and a "when". Right now we've got the first two bits (ish), but the third is a mystery....which means we don't have a collective plan.

By the transitive property of IE being the monopoly market-share browser, we can clearly state that we have no idea when the Open Web will be revved. This is based solely on the IE team's lack of commitments. This is a terrible result, and one which I think the frenzy over IE 8's features has obscured.

Joel's article and some of Mark's commenters make it obvious that there's now a gap in the expectations of some devs about what it means to be a web developer versus what it meant 4 or 5 years ago. Back then we all assumed that browsers would get better, or at least different. There was bitter complaining about how incompatible everything was and how horrid it was to have to update everything...but underlying that discontent was the assumption that the web would keep changing. Web developers have to a great extent lost this assumption and I see a lot of the acrimonious discussions about new browser features in this light. There is great fear on the part of the web development community that progress will cost something, and they're right. So long as we're only talking about one revision, the cost looks new and surprising, but if we were to start talking about how we'd keep revving the web, those costs could be assumed by all parties again. It's for this reason that I posit that the most important thing about IE 8 won't be any of its features...it'll be that it ships soon-ish and goes out to auto-update (if that is indeed the plan, which we don't know).

The analogies that Joel brings up about standardization are perhaps valid if you take a snapshot in time of features vs. conformance. What happens in the browser world, though, is that things where were only marginally possible with old features become the norm via new features. New tags or CSS rules get added which make what was hard simple, if less flexible. In that process, we find that we need strong agreement on the behavior of those "mainline" things, while it's perfectly OK for browser vendors to disagree about the fringes. Those fringes, after all, are were new things should be getting developed and introduced for market testing. It's only when new things don't get introduced and that broad agreement isn't forged and "full" standards conformance comes into the fore. We need 100% standards conformance for all the nit-picky corner cases when those are all we've got in the way of "new" platform capabilities. Joel's view of the world doesn't take into account healthy evolution and improvement based on real competition. Competition makes suppliers responsive to customers and gives a real path for evolution. Standards make suppliers responsive to proxy bodies who are easily distracted.

IE and all browsers are as much a platforms as applications, but browser vendors get themselves into strange positions with regards to the platform bits of their products. Since browsers are positively differentiated by their chrome and not their engines, you can think of renderers and script engines and all the things that webdevs care about as costs of doing business for a browser-producing organization. Insofar as they make money, they do it on chrome, not rendering. Of course, getting into the good graces of web developers is great for a browser maker, but getting in front of users is the only real metric of browser success and to get there one only needs a renderer that's on par with the others. You can't positively differentiate a browser by making it more standards compliant or even by introducing new and awesome non-standard features. Those things can have strategic value, but they can never stand on their own.

Which brings us back to IE being a platform. The bits that webdevs care about must change in order for the web to get better, and today webdevs are platform customers without a commitment from their biggest supplier to ship another version beyond the one they're working on now. If this were any other sort of platform, this would never ever fly. Code-in-escrow would be demanded along with a roadmap, or at a minimum a commitment to an N+1 version in a reasonable time frame. But webdevs don't have that leverage by virtue of the disintermediation that browser economics now demand.

So as webdevs, we must be canny enough to find a way to "better" which doesn't put all of our eggs in any particular basket. Every browser that we depend on either needs an open development process or it needs to have a public plan for N+1. The goal is to ensure that the market knows that there is momentum and a vehicle+timeline for progress. When that's not possible or available, it becomes incumbent on us to support alternate schemes to rev the web faster. Google Gears is our best hope for this right now, and at the same time as we're encouraging browser venders to do the right thing, we should also be championing the small, brave Open Source team that is bringing us a viable Plan B. Every webdev should be building Gear-specific features into their app today, not because Gears is the only way to get something in an app, but because in lieu of a real roadmap from Microsoft, Gears is our best chance at getting great things into the fabric of the web faster. If the IE team doesn't produce a roadmap, we'll be staring down a long flush-out cycle to replace it with other browsers. The genius of Gears is that it can augment existing browsers instead of replacing them wholesale. Gears targets the platform/product split and gives us a platform story even when we're neglected by the browser vendors.

Gears has an open product development process, an auto-upgrade plan, and a plan for N+1.

At this point in the webs evolution, I'm glad to see browser vendors competing and I still feel like that's our best long-term hope. But we've been left at the altar before, and the IE team isn't giving us lots of reasons to trust them as platform vendors (not as product vendors). For once, we have an open, viable Plan B.

Gears is real, bankable progress.

Updates and Dojo 1.1