Infrequently Noted

Alex Russell on browsers, standards, and the process of progress.

The Pursuit of Appiness

TL;DR: App stores exist to provide overpowered and easily abused native app platforms a halo of safety. Pre-publication gates were valuable when better answers weren't available, but commentators should update their priors to account for hardware and software progress of the past 13 years. Policies built for a different era don't make sense today, and we no longer need to accept sweeping restrictions in the name of safety. The idea that preventing browser innovation is pro-user is particularly risible, leading to entirely avoidable catch-22 scenarios for developers and users. If we're going to get to better outcomes from stores on OSes without store diversity or side-loading, it's worth re-grounding our understanding of stores.

Contemporary debates about app stores start from premises that could use review. This post reviews these from a technical perspective. For the business side, this post by Ben Thompson is useful context.

...Only Try To Realize The Truth: There Is No "App"

OSes differ widely in the details of how "apps" are built and delivered. The differences point to a core truth: technically speaking, being "an app" is merely meeting a set of arbitrary and changeable OS conventions.

The closer one looks, the less a definition of "appiness" can be pinned down to specific technologies. Even "toy" apps with identical functionality are very different under the covers when built on each OSes preferred stack. Why are iOS apps "direct metal" binaries while Android apps use Java runtimes? Is one of these more (or less) "an app"? The lack of portability highlights the absence of clear technical underpinnings for what it means to "be an app".

To be sure, there are platonic technical ideals of applications as judged by any OS's framework team[1], but real-world needs and behaviour universally betray these ideals.[2] Native platforms always provide ways to dynamically load code and content from the network, both natively or via webview and choice of code platform is not dispositive in identifying if an experience is "an app".

The solid-land definition "an app" is best understood through UI requirements to communicate "appiness" to end-users: metadata to identify the publisher and provide app branding, plus bootstrap code to render initial screens. Store vendors might want to deliver the full user experience through the binary they vend from their stores, but that's not how things work in practice.

Industry commentators often write as though the transition to mobile rigidly aligned OSes with specific technology platforms. At a technical level, this is not correct.

Since day one, iOS has supported app affordances for websites with proprietary metadata, and Android has similar features. "Appiness" is an illusion, an arbitrary line drawn to include programs built on some platforms but not others, and all of today's OSes allow multiple platforms to expose "real apps."

Platforms, while plural, are not equal.

Apple could unveil iOS with support for web apps without the need for an app store because web applications are safe by default. Circa '07, those web apps were notably slower than native code could be. The world didn't yet have pervasive multi-process browser sandboxing, JS JITs, or metal-speed binary isolation. Their arrival, e.g. via WASM, has opened up entirely new classes of applications on a platform (the web) that demands memory safety and secure-by-default behavior as a legacy of 90s browser battles.

In '07, safety implied an unacceptable performance hit on slow single-core devices with 128MiB of RAM. Many applications couldn't be delivered if strict protection was required using the tools of the day. This challenging situation gave rise to app stores: OS vendors understood that fast-enough-for-games needed dangerous trade-offs. How could the bad Windows security experience be prevented? A priori restraint on publishing allowed vendors to set and enforce policy quickly.

Fast forward a decade, and both the software and hardware situations have changed dramatically. New low-end devices are 4-to-8 core, 2GHz systems with 2GiB of RAM. Sandboxing is now universal (tho quality may differ) and we collectively got savvy about running code quickly while retaining memory safety. The wisdom of runtime permission grants (the web's model) has come to be widely accepted.

For applications that need peak compute performance, the effective tax rate of excellent runtime security is now in the 5-10% range, rather than the order of magnitude cost a decade ago. Safety is within our budget, assuming platforms don't make exotic and dangerous APIs available to all programs — more on that in a second.

So that's apps: they're whatever the OS says they are, that definition can change at any moment, and both reasonably constrained and superpowered programs can "be apps." Portability is very much a possibility if only OSes deign to allow it[3].

What Is An "App Store" Technically?

App stores, as we know them today, are a fusion of several functions:

  1. Security screens to prevent malign developer behavior on overpowered native platforms
  2. Discovery mechanisms to help users find content, e.g., search
  3. App distribution tools; i.e., a CDN for binaries
  4. Low-friction payment clearing houses

Because app stores rose to prominence on platforms that could not afford good safety using runtime guards, security is the core value proposition of a store. When working well — efficiently applying unobjectionable policy — store screening is invisible to everyone but developers. Other roles played by stores may have value to users but primarily serve to build a proprietary moat around commodity (or lightly differentiable) proprietary, vertically integrated platforms.

Amazon and search engines demonstrate that neither payment nor discovery requires stores. If users are aware of security in app stores, it's in the breach, e.g., the semi-occasional article noting how a lousy app got through. The desired effect of pre-screening is to ensure that everything discovered in the store is safe to try, despite being built on native platforms that dole out over-broad permissions. It's exceedingly difficult to convince folks to use devices heavily if apps can brick the device or drain ones bank account.[4]

The Stratechery piece linked at the top notes how important this is:

It is essential to note that this forward integration has had huge benefits for everyone involved. While Apple pretends like the Internet never existed as a distribution channel, the truth is it was a channel that wasn’t great for a lot of users: people were scared to install apps, convinced they would mess up their computers, get ripped off, or accidentally install a virus.

Benedict Evans adopts similar framing:

Specifically, Apple tried to solve three kinds of problem.

  • Putting apps in a sandbox, where they can only do things that Apple allows and cannot ask (or persuade, or trick) the user for permission to do ‘dangerous’ things, means that apps become completely safe. A horoscope app can’t break your computer, or silt it up, or run your battery down, or watch your web browser and steal your bank details.

  • An app store is a much better way to distribute software. Users don’t have to mess around with installers and file management to put a program onto their computer - they just press ‘Get’. If you (or your customers) were technical this didn’t seem like a problem, but for everyone else with 15 copies of the installer in their download folder, baffled at what to do next, this was a huge step forward.

  • Asking for a credit card to buy an app online created both a friction barrier and a safety barrier - ‘can I trust this company with my card?’ Apple added frictionless, safe payment.

These takes highlight the value of safety but skip right past the late desktop experience: we didn't primarily achieve security on Windows by erecting software distribution choke points. Instead, we got protection by deciding not to download (unsafe) apps, moving computing to a safe-by-default platform: the web.[5]

In contrast with the web, native OSes have historically blessed all programs with far too much ambient authority. They offer over-broad access to predictably terrifying misfeatures (per-device identifiers, long-running background code execution and location reporting, full access to contacts, the ability to read all SMS messages, pervasive access to clipboard state, etc.).

Native platform generosity with capabilities ensures that every native app is a vast ocean of abuse potential. Pervasive "SDK" misbehavior converts this potential to dollars. Data available to SDK developers make fears about web privacy look quaint. Because native apps are not portable or standards-based, the usual web-era solutions of switching browsers or installing an extension are unavailable. To accept any part of the native app ecosystem proposition is to buy the whole thing.

Don't like the consequences? The cost of changing one's mind is now starts at several hundred dollars. The moat of proprietary APIs and distribution is sold on utility and protection, but it's primary roles is protect margins. A lack of app and data interoperability prevents users from leaving. The web, meanwhile, extends security as far as the link can take you. Moats aren't necessities when security is assured.

This situation are improving glacially, often at the pace of hardware replacement.[6] The slog towards mobile OSes adopting a more web-ish contract with developers hints at the interests of OS vendors. Native platforms haven't reset the developer contract to require safety because they recall what happened to Windows. When the web got to parity in enough areas, interest in the proprietary platform receded to specialist niches (e.g., AAA gaming, CAD). Portable, safe apps returned massive benefits to users who no longer needed to LARP as systems administrators.

The web's safety transformed software distribution so thoroughly that many stopped thinking of applications as such. Nothing is as terrifying for a vertically integrated hardware vendor as the thought that developers might leave their walled garden, leading users to a lower-tax-rate jurisdiction.

Safety enables low friction, and the residual costs in data and acquisition of apps through stores indicate not security, but the inherent risk of the proprietary platforms they guard. Windows, Android, and ChromeOS allow web apps in their stores, so long as the apps conform to content policies. Today, developers can reach users of all these devices on their native app stores from a single codebase. Only one vendor prevents developers from meeting user needs with open technology.

This prejudice against open, unowned, safe alternatives exists to enable differentiated integrations. Why? To preserve the ability to charge much more over the device's life than differences in hardware quality might justify. The downsides now receiving scrutiny are a direct consequence of this strategic tax on users and developers. Restrictions, once justifiably required to maintain safety, are now antique. What developer would knowingly accept such a one-sided offer if made today?

Compound Disinterest

Two critical facts about Apple's store policies bear repeating:

  1. Apple explicitly prevents web content within their store
  2. Apple prevents browser makers from shipping better engines in alternative iOS browsers[7]

It is uniquely perverse that these policies ensure the web on iOS can only ever be as competent, safe, and attractive to developers as Apple allows.

Were Apple producing a demonstrably modern and capable browser, it might not be a crisis, but Apple's browser and the engine they force others to use is years behind.

The continuing choice to under-invest in Safari and WebKit has an ecosystem-wide impact, despite iOS's modest reach. iOS developers who attempt to take Apple up on their generous offer to reach users through the web are directly harmed. The low rate of progress in Apple's browsers all but guarantees that any developer who tries will fail.

But that's not the biggest impact. Because the web's advantage is portability, web developers view features not available "everywhere" as entirely unavailable. The lowest common denominator sets a cap on developer ambitions, and Apple knows it. App Store policies taht restrict meaningful browser competition ensure the cap is set low so as to prevent real competition with Apple's proprietary tools and distribution channel.

Apple doesn't need a majority of web usage to come from browsers without critical features to keep capabilities from being perceived skeptically; they only need to keep them out of the hands of, say, 10% of users. So much the better if users who cannot perceive or experience the web delivering great experiences are wealthy developers and users. Money talks.

Thirteen years of disinterest in a competitive mobile web by Apple has produced a crisis for the web just as we are free of hardware shackles. The limits legitimated the architecture of app store control are gone, but the rules have not changed. The web was a lifeboat away from native apps for Windows XP users. That lifeboat won't come for iPhone owners because Apple won't allow it. Legitimately putting user's interests first is good marketing, but terrible for rent extraction.

The Mobile Web: MIA

Arguments against expanding web capabilities to reach minimum-mobile-viability sometimes come wrapped in pro-privacy language from the very parties that have flogged anti-privacy native app platforms[8]. These claims are suspect. Detractors (perhaps unintentionally) conflate wide and narrow grants of permission. The idea that a browser may grant some sites a capability is, of course, not equivalent to giving superpowers to all sites with low friction. Nobody's proposing to make native's mistakes.

Regardless, consistent misrepresentations are effective in stoking fear. Native apps are so dangerous they require app store gatekeepers, after all. Without looking closely at the details, it's easy to believe that expansions of web capability to be similarly broad. Thoughtful, limited expansions of heavily cabined capabilities take time and effort to explain. The nuances of careful security design are common casualties in FUD-slinging fights.

A cynic might suggest that these misrepresentations deflect questions about the shocking foreclosure of opportunities for the web that Apple has engineered — and benefits from directly.

Apple's defenders offer contradictory arguments:

  1. Browsers are essential to modern operating systems, and so iOS includes a good browser. To remain "a good browser," it continually adds and markets new features.
  2. Browsers are wildly unsafe because they load untrusted code and, therefore, they must use only Apple's (safe?) engine.[9]
  3. App stores ensure superpowered apps are trustworthy because Apple approves all the code.
  4. Apple doesn't allow web apps into the store as they might change at runtime to subvert policy.

Taken in isolation, the last two present a simple, coherent solution: ban browsers, including Safari. Also, webviews and other systems for dynamically pushing code to the client (e.g., React Native). No ads that aren't pre-packaged with the binary, thanks. No web content — you never can tell where bits if of a web page might be coming from, after all! This solution also aligns with current anti-web store policies.

The first two points are coherent if one ignores the last decade of software and hardware progress. They also tell the story of iOS's history: 13 years ago, smartphones were niche, and access to the web corpus was important. The web was a critical bridge in the era before native mobile-first development. Strong browser sandboxing was only then becoming A Thing (TM), and resource limits delayed mobile browsers from providing multi-process protection. OS rootings in the low-memory, 1-2 core-count era perhaps confirmed Cupertino's suspicions.[10]

Time, software, and hardware reality have all moved forward. The most recent weaksauce served to justify this aging policy has been the need for JIT in modern browsers. Browser vendors might be happy to live under JIT-less rules should it come to that — the tech is widely available — and yet alternative engines remain banned. Why?

Whatever the reason, in 2020, it isn't security.

Deadweight Losses

Undoubtedly, web engines face elevated risks from untrusted, third-party content. Mitigating these risks has played a defining role in evolving browsers for more than twenty years.

Browsers compete on security, privacy, and other sensitive aspects of the user's experience. That competition has driven two decades of incredible security innovation without resorting to prior-restraint gatekeeping. The webkit-wrapper browsers on iOS are just as superpowered as other apps. They can easily subvert user intent and leak sensitive information (e.g., bank passwords). Trustworthiness is not down to Apple's engine restrictions; the success of secure browsers on every OS shows this to be the case. These browsers provide more power to users and developers while maintaining an even more aggressive security posture than any browser on iOS is allowed to implement.

But let's say the idea of "drive-by" web content accessing expanded capabilities unsettled Apple, so much so that they wanted to set a policy to prevent powerful features from being available to websites in mere tabs. Could they? Certainly. Apple's Add to Home Screen feature puts installed sites into little jails distinct from a user's primary browsing context. Apple could trivially enforce a policy that unlocked powerful features only upon completion of an installation ceremony, safe in the knowledge that they will not bleed out into "regular" web use.[11] We designed web APIs to accommodate a diversity of views about exposing features, and developers guard against variable support already.

Further, it does not take imagination to revoke features should they be abused by a site. Safari, like all recent-vintage browsers, includes an extensible, list-based abuse mitigation system.

The preclusion of better iOS browsers hinges on opacity. Control drives from developer policy rather than restrictions visible to users, rendering browser switching moot. Browser makers can either lend their brands to products they do not think of as "real browsers" or be cut off from the world's wealthiest users. No other OS forces this choice. But the drama is tucked behind the curtains, as is Apple's preference.

Most folks aren't technical enough to understand they can never leave Hotel Cupertino, and none of the alternative browsers they download even tell them they're trapped. A more capable, more competitive web might allow users to move away from native apps, leaving Apple without the ability to hold developers over the barrel. Who precludes this possibility? Apple, naturally.

A lack of real browser competition coupled with a trailing-edge engine creates deadweight losses, and these losses aren't constrained to the web. Apple's creative reliance on outdated policy explicitly favors the proprietary over the open, harming consumers along the way.

Suppose Apple wishes us to believe their app store policies fair, while simultaneously claiming the web is an outlet for categories of apps they do not wish to host in their store. In that case, our pundit class should at least query why Apple will (uniquely) neither allow nor build a credible, modern version of the web for iOS users.

  1. The platonic ideal of "an app" is invariably built using the system's preferred UI toolkit and constructed to be self-contained. These apps tend to be bigger than toys, but smaller than what you might attempt as an end developer; think the system-provided calculator. ↩︎

  2. Bits downloaded from the store seldom constrain app behavior, protests by the store's shopkeepers notwithstanding. Before the arrival of Android Dynamic Apps, many top apps used slightly-shady techniques to dynamically load JARs at runtime, bypassing DEX and signing restrictions. It's a testament to the prevalence of the need to compose experiences that this is now officially supported dynamically. ↩︎

  3. Portability is a frequent trade-off in the capability/safety discussion. Java and Kotlin Android apps are safer and more portable by default than NDK binaries, which need to worry about endianness and other low-level system properties.

    The price paid for portability is a hit to theoretical maximum performance. The same is true of the web vs. Java apps, tho in all cases, performance is contingent on the use-case and where platform engineers have poured engineer-years into optimisation. ↩︎

  4. Not draining other's bank accounts may be something of an exercise to the reader. ↩︎

  5. Skipping over the web and its success is a curious blind-spot for such thoughtful tech observers. Depressingly, it may signal that the web has been so thoroughly excluded from the land of the relevant by Apple that it isn't top, middle, or even bottom-of-mind. ↩︎

  6. Worldwide sales of iOS and the high-end Android devices that get yearly OS updates are less than a third of total global handset volume. Reading this on an iPhone? Congrats! You're a global 15%-er. ↩︎

  7. Section 2.5.6 is the "Hotel Cupertino" clause: you can pick any browser you choose, but you can never leave Apple's system-provided WebKit. ↩︎

  8. Mozilla's current objections are, by contrast, confused to my ears but they are at least not transparently disingenuous. ↩︎

  9. Following this argument, Apple is justified in (uniquely) restricting the ability of other vendors to deliver more features to the web (even though adding new features is part of what it means to be "a good browser"). Perversely, this is also meant to justify restrictions that improve the security of both new and existing features in ways that might make Safari look bad. I wonder what year it will be when Safari supports Cross-Origin Isolation or Site Isolation on any OS?

    Why does it matter? Presumably, iOS will disallow WASM threading until that far-off day. ↩︎

  10. For all of Apple's fears about web content and Safari's security track record, Apple's restrictions on competing web engines mean that nobody else has been allowed to bring stronger sandboxing tech to the iOS party.

    Chrome, for one, would very much like to improve this situation the way it has on MacOS but is not allowed to. It's unclear how this helps Apple's users. ↩︎

  11. webviews can plumb arbitrary native capabilities into web content they host, providing isolated-storage with expanded powers. The presence of webview-powered Cordova apps in Apple's App Store raises many questions in a world where Apple neither expands Safari's capabilities to meet web developer needs nor allows other browsers to do so in their stead. ↩︎

Why Browser Security UI Isn't Specified

Update: my colleague Emily Stark has just written an essential-reading post on this topic. We probably need to write something about all of the non-prompt controls that Chrome has put in place behind the patterns discussed in this post, but Emily's post really drives home the case for flexibility with practical examples.

From time to time, working groups (particularly at the W3C) are asked to adopt requirements on browser-presented UI (pixels drawn outside or over-top the page content) in the normative text of a specification. For example, someone might advocate for text like along the lines of "before proceeding to Step 5, conforming User Agents MUST request user consent". Less frequently, advocates will push for the inclusion of specific text to be presented, or in the most extreme cases, specific UI layouts.

The most intense of these debates occur in the context of security-sensitive moments: warning users of potential risks, requesting permission to access sensitive features, etc. Thankfully, these efforts usually fail in the face of bristling browser feedback.

What's going on here? Don't browsers care about providing consistent UI? Don't they want the best possible security?

The answer to both questions is an emphatic yes. Counter-intuitively, it is because browsers care so much about security, privacy, and consistency (in that order) that we err on the side of rejecting constraints on security UI. In the modern era, that is paired with API design patterns that maximize flexibility about these moments.

Bit Shifter Priorities

Web Standards are voluntary. The force that most powerfully compels their adoption is competition, rather than regulation. In practice, this means that vendors of integrated browsers[1] have final say over what their browsers do. Those who ship the bits, rather than those who write the spec text, are ultimately responsible for the properties of the subsequent experience. This is the basis for functioning browser competition.[2] Those who shift the bits call the shots.

This is an inherent property of modern browsers. Vendors participate in standards processes not because they need anyone else to tell them what to do, and not because they are somehow subject to the dictates of standards bodies, but rather to learn from developers and find agreement with competitors in a problem space where compatibility returns outsized gains.

Web Standards can be understood as liberally licensed, meticulously ventilated documentation about the detailed workings of a system that implementers agree to potentially be wrong about together.

Normative restrictions in a well-built spec should only cover aspects of a design where benefits of consistency and compatibility reliably outweigh the costs to getting some detail wrong.[3] That is, if an API is found to be wanting in some way, the specified bits of the design are limited to aspects vendors can take time to address through an iterative, deliberative process.

Of course, there are classes of errors that vendors are not willing to be wrong about for the time it takes to forge new consensus. Chief among them are security vulnerabilities. Pervasive, abusive behaviour is not far behind. There are dozens of instances over the decades of browsers unilaterally breaking spec'd behaviours in the interest of guarding users. UI prescriptions in specifications that intersect with these concerns will be blithely ignored by vendors looking to protect users and their own reputations. This flows directly from the frequently-cited priority of constituencies.

Normative text about properties that are neither developer exposed nor are likely to be upheld in the breech create long-term liabilities. Over time, predictable conformance breakage — either from incident response or iterative exploration toward better solutions — has the effect of putting asterisks over otherwise-implementable specifications. This creates credibility problems for specs, their editors, working groups that back them, reviewers who do not flag unenforceable text, and SDOs that ratify toothless language. In a world of voluntary standards, reputation matters, and bad spec language can create a lasting negative impression.

Hitchen's Razor

One of the strongest reasons to avoid making MUST (or even SHOULD)-strength statements about what browsers present is the vast depth of our collective ignorance.

Security UX is a relatively young field, and the set of organisations funding research into it is relatively small. We know from practice that understandability of systems is a major challenge and that users are poor at judging risk. This often causes those concerned about specific risks to demand prompts or user confirmation, but evidence also teaches that we risk "dialog blindness" through repeated re-confirmations.

The most successful examples of guiding users to safety in recent years involve a great deal of trial-and-error and follow-up research. Our best tools, collectively, are studies of variations and watching behaviour at scale. There are no models that I'm aware of that consistently maximize user understanding of choice over time while reducing decision burdens.

Consequently, as API designers, we owe it to our UXR peers to maximize their ability to iterate on potential presentations in service of those dual goals. Failing to acknowledge our own ignorance is a surefire way to paint ourselves into a bad corner.

If the case for a specific UI treatment on the basis of reckons is suspect, the case for consistency between browsers is doubly so. There is some evidence that a not-inconsiderable subset of desktop users may encounter multiple browsers in a week, but it is not a majority. Consistency between them should not then be a more important goal than internal consistency within a single browser's UI surfaces and/or consistency with underlying OS affordances (which also differ). Unlike the APIs that might invoke them, these UIs are not developer defined or exposed, removing the usual argument for consistency in a web standard.

Frustratingly, intense demands are often paired with generalized statements of the form "I think..." and "users don't know X...", without associated research. These reckons can take up a great deal of time when they should fail-fast against Hitchen's Razor, particularly given the browser community's history of iteration on these surfaces. Should future proposals for normative browser security surfaces come with peer-reviewed, wide-scale UXR attached then, perhaps, it may be worth revisiting blanket push-back.

Until then, and given our vast lack of understanding, it seems prudent to design defensively while expanding each spec's Privacy and Security Considerations sections to exhaustively enumerate risks that UX researchers will be asked to navigate. Designing for maximum iteration potential in this area has been, thus far, the best way to be least-cumulatively-wrong.[4]

  1. Integrated browsers are those that ship an engine implementation along with browser UI. "WebView browsers" are an example of non-integrated browsers; they are able to define browser UI, can't change the behaviour of the entire API surface. They are hostage in some measure to the system-provided WebView for consequential choices that affect both users and developers. Browser vendors are hopping mad at Apple about iOS because — unlike every other commodity OS, including ChromeOS — it restricts integrated browsers to a single (first) party, forcing all other vendors to implicitly market Apple's (sub-standard) idea of what a browser can be through WebView-browsers with their brands...or forfeit the right to reach iOS users entirely. This undermines the competition that is central to progress and should be viewed by all users (and particularly web developers) as an intolerable distortion. If there is anything good to be said for browser engine diversity, it is that it spurs competition, sparking improvement across the board. Undercutting competition is, therefore, striking directly at the heart of the reason to want multiple engines in the first place. ↩︎

  2. End-to-end vendor responsibility in integrated browsers — and the inability to make better decisions about consequential choices within WebView-based browsers — also explains why the Hotel Cupertino clause is a slow-moving disaster for progress, depressing the rate of innovation on the web overall. ↩︎

  3. For a live example, consider the HTML5 History API. It's a hot mess. But it's an overwhelmingly compatible mess. As we dig into potential solutions, none of them involve breaking changes to the existing APIs and, if breaking changes were proposed, would only be done so on the basis of close study (e.g. via usage statistics), potentially spanning multiple quarters. The plan to deprecate of Web Components V0 APIs, for example, is in its second year of execution, with a lifeline for deprecation extending until at least next year. We'd expect breaking changes to the HTML5 History API to require a multiple on that timeline given relative use. Therefore, proposed fixes are largely constrained to compatible extensions and new, parallel API surfaces that can be incrementally adopted by developers over a span of years. By the same token, we have been willing to subvert the behavior of this same API in breaking ways to prevent abuse of users on relatively tight timelines, and without consensus. Security and abuse issues are a different class of concern and reliably trump compatibility. ↩︎

  4. Non-normative notes and external "explainer" documents can help supplement the core text and provide examples of ideas for implementers to keep in mind. If a spec must contain some text that tries to talk about browser UI, this is the way to do it. ↩︎

Platform Adjacency Theory

TL;DR: Does it matter if the web platform adds new capabilities? And if it should, which ones? The web is a meta-platform. Like other meta-platforms the web thrives or declines to the extent it can accomplish the lion's share of the things we expect most computers to do. Adjacency to the current set of capabilities provides a disciplined way to think about where to invest next when working to stave off irrelevance. If distribution of runtimes is blocked, competition falters, and adjacent capabilities can go un-addressed. Ecosystem decline, and eventual collapse, follows. Apple and Mozilla posturing about risks from new capabilities is belied by accepted extant native platform risks. For different reasons, they are both working to cast the web in amber. We will all have lost a future worth wanting should they succeed.

Chromium's Project Fugu 🐡 is an open collaboration between Google, Microsoft, Intel, Samsung, and others to safely bring capabilities to the web which native developers have come to expect. Henrik's piece on our work is particularly lucid about the concrete value of new, seemingly-abstract features. Allowing some subset of trusted web apps to access the serial ports (with user/admin consent) isn't academic; it's helping Henrik deliver health care at lower cost. Suz Hinton's talks on WebUSB show the through-line: as the web gets more capable, whole classes of software suddenly benefit from frictionless deployment, increased safety, better privacy, and more meaningful user controls. After all, those are properties that browsers compete on, benefiting users in the process.

We started down this path with Project Fizz in '14. That (mobile-first) work added PWA installability, Push Notifications, Web Bluetooth, WebUSB, and Background Sync. Fugu 🐡 expands our remit to desktop and new classes of apps via powerful APIs — everything from codecs to windowing controls to font access to Serial and HID devices — a list that might seem like a semi-random grab bag. In retrospect, and without context, the features we delivered in Fizz may also appear random.[1] Expansions of the web's power can seem by turns scary, capricious, or a distraction from the real problems developers face.[2]

We've faced our fair share of push-back, often from other browser engines, usually shrouded in privacy rhetoric.[3] So, why do this? Why push through the sorts of snide comments Henrik highlighted? Why risk so much on new features when there's a never-ending list of things that need to be fixed with the existing set?

The answer lies in what I have come to understand as Platform Adjacency Theory.

Our Random Walk

Folks who claim "the web doesn't need that!" invariably point to successes brought by things that the web already does, as though it is a logically coherent set.

But is it?

Why does it make sense for a browser to speak only HTTP, MIME-encode a fixed set of data types with corresponding (inextensible!) form elements, decode (but not always encode) a subset of popular media formats, and marry that to a constraint-solver for laying out text and drawing boxes?[4] Oh, and it should be imperatively programmed with Lisp-in-C's-clothing.


Recall that browsers arose atop platforms that universally provided a host of services by the late 90's:

The web wasn't the only meta-platform in this era. Meta-platforms exist to abstract over their hosts, providing a uniform surface API to create portability. Portability reduces single-platform lock-in, often (but not always) buttressed by open standards option-value. Because meta-platforms intermediate, they have the ability to provide ecosystem benefits that are difficult for underlying platforms to replicate, e.g. lower distribution friction, streaming content execution, faster platform updates, and better security. These traits manifest as faster development, iterative deployment, and lower cost of ownership for businesses and developers. To convert these theoretical benefits to real gains, meta-platforms implement runtimes for each underlying OS. Runtimes need wide distribution to make meta-platforms attractive to developers. This matters because it is the corpus of attractive content uniquely available through a meta-platform that convinces users to adopt its runtime, and no platform developer can afford to build enough content to be compelling on their own.

A few meta-platforms achieved wide distribution by the turn of the century, earning a chance at handling tasks previously requiring native Win32 binaries. Shockwave and Flash leaned into scripting, graphics, and multimedia. Distribution in the Windows XP base image ensured Flash was a part of desktop computing for more than a decade afterwards. Huge reach, plus the licensing of On2's video codecs, made Flash the default video delivery mechanism for many years.

Java arrived with a focus on structured programming (including threads), GUI controls, networking, device access, and graphics. Distribution was competitive, with multiple implementations (some tied to browsers) sometimes complicating the platform for developers.[5] As with other meta-platforms of the time, early browser plug-ins provided avenues for runtime and content distribution.[6] Java found some client-side niches, particularly in enterprise. Absent compelling content and attentive security and performance focus, it soon retreated to the server.[7]

Since the end of the first browser wars, most of the world's consumer computers retained baseline 90's features (CD drives and printers excepted) while adding:

Some legacy devices lack some of these features, however, all commodity operating systems support them and the plurality of devices sold have them. The set of things "most computers do" started larger than what the early web supported (particularly without plug-ins) and continues to accelerate away.

The Overlapping Competitions

Every computing platform is locked in overlapping competitions. The first is for developer time and attention. The set of successful applications not built to your platform is a bug to the mind of every competent platform strategist (malware excepted). It is a problem for OSes when developers do not use their APIs directly, but instead reach for portability layers atop the native surface. Developers deliver content. Compelling content and services draw users. If the weight of these experiences is compelling enough, it can even convince potential users to acquire your device or runtime to get access. The resulting install base of a platform and it's capabilities are an asset in convincing the next developer to build for a platform.[9] This plays out in a loop over time, either adding or shedding momentum from a platform. Winning the second competition (install base growth) in each iteration makes it ever easier to draw developer attention — assuming the platform remains competitively capable.

Teams make roughly exclusive choices about platforms for each project. Maximizing user-experience quality and reach from the smallest possible investment is how developers justify outsized rewards for easily learned skills. Meta-platforms enter this calculus with a deficit to maximum capability and performance but (hopefully) gains to reach. When an application's essential needs can be handled effectively while providing a competitive enough user experience, meta-platforms can win the business. In situations where key capabilities in a domain are missing from a meta-platform, applications simply won't appear there. This results in higher (per-OS) development, acquisition, and maintenance costs for the ecosystem (often centralizing benefits to fewer, capital-rich players) and leading to other deadweight losses.[10]

Consider this straw-person conceptual framing:

Mind the gap.

Our theoretical meta-platform starts at the left (Time 0) with a relatively complete set of features versus what most computers can do, but substantially less than the leading edge, and perhaps trailing what all commodity operating systems provide by some amount. There will be some apps that can't be attempted, but most can. Developers are making decisions under uncertainty at every point, and the longer a platform lives and the more successful it is, the less risk to developers in betting on it. Assuming wide distribution for our meta-platform, it's a good bet for most developers across the majority of the graph's width.

In each quantum of time, hardware and OS vendors press ahead, adding features. Some "exotic" features become commonplace, widely supported across the deployed fleet of devices. As OS and hardware deployed base integrate these components, the set of what most computers can do is expanded. This is often determined by hardware integration and device replacement rates. The fates of these advances occupy the majority of tech press (and, sadly, developer) attention thanks to marketing from vendors looking to differentiate at the high-end (where margins both exist and require justification).[11]

Toward the right-hand side of our chart, the meta-platform that once looked like a good bet is suddenly legacy technology. By failing to keep up with what most computers can do, or even increase at a constant rate such that the height of the relevance gap remains constant, developers making a choice might want to re-use knowledge they've gained from earlier cycles betting on the meta-platform, but will likely need to route around the suddenly-stagnant legacy system.

But why should it feel sudden when, in reality, the trend lines are nearly constant over time?

Consider the re-development cycles of products and services. If teams re-evaluate their stack choices only at the start, middle, and end of the chart our meta-platform will appear to go from excitingly competent, to mildly concerning (but still more than capable), to a career and product risk. Each of these moments is experienced stochastically in our model, with the final moment presenting a previously un-hedged risk. Developer FOMO can be self-fulfilling; exoduses from platforms aren't so much the result of analysis so much as collective and personal ego defenestration. The Twitter bios of any 10 randomly picked programmers will confirm the identitarian aspects of platforms and their communities. Breaks with these identities are sharp and painful, leading to tribal flight. Assuming the capability gap doesn't close, developers moving their attention seals the fate of our straw-person meta-platform.

It doesn't have to be this way, though. There's no technical reason why, with continued investment, meta-platforms can't integrate new features. As the set expands, use cases that were previously the exclusive purview of native (single-OS) apps can transition to the meta-platform, gaining whatever benefits come with its model.

Missing alt text

If our theoretical team encountered the platform evolution in this chart at the moments we discussed earlier, it would still be a good bet at the right-hand side of the chart, even including sizeable stagnant periods.

Katamari Capability

The web's benefits of frictionless deployment, universal access, streaming execution, collaboration, security, and user choice are model advantages for both content and services. Together, they make software better.

Software moves to the web (when not prevented by poor browsers and/or unconscionable policy) because it's a better version of computing:

It has been really surprising to me how few frontend folks are investing deeply in sync & collaboration tech.

But not every application can make the leap. Consider the set of features the circa-2000 web could handle:

Missing alt text

Starting from a text-only document format in 1990, the web added features at a furious pace in the first decade, but not in a principled way. Whatever the motivations of early browser hackers, some use-cases were very near to the original needs of academic information sharing, news delivery among them:[12]

Missing alt text

The system was capable enough by 1995 to support an Ajax-y email, but the experiences weren't a tight fit with user expectations of the time.[13]

Missing alt text

The Jobs To Be Done of an email client are:

Only the last two proved challenging once Ajax was possible across browsers, sparking desktop email's inexorable move to the web. Some things never got ironed out. Web-based clients still can't talk IMAP or POP directly, and the look-and-feel of web clients isn't 1:1 with any OS. Nice-to-have features didn't impact adoption much.[15] The web's model advantages more than made up for cosmetic deficiencies when security, cost, and ease-of-access were factored in. Being just capable enough unlocks massive value through model shifts.

Other use-cases weren't addressable with early aughts tech. WebEx and Skype and Tandberg and many other video conferencing solutions emerged on native and didn't cross over to the web until relatively recently. They needed what early browsers could do, but much, much more as well. Rendering text is important (think login screens), but there's no product without low-latency, adaptive codecs, networking, and camera/microphone access.

Missing alt text

The surface area of needs swamped the capabilities of the early web.

Between the end of the first browser wars and today, we added adjacent capabilities to unlock core use-cases in this area. Fast-forward to modern browsers, and Zoom, WebEx, Google Meet, Jitsi, and even Skype work well (and most safely) on the web.

Missing alt text

We still haven't covered every possible feature of the native versions of these tools, but enough is there to enable a massive shift way from desktop-native apps; a still-unfinished transition that I expect to last at least another half decade:

Missing alt text

This is the core loop at the heart of platform adjacency theory. The next set of important use-cases for a platform to tackle often isn't simply an improvement on what it can already do.[16]

Growing a platform's success requires unlocking use-cases not already serviced. That mean finding needs that combine things your platform is already good at with a small number of missing capabilities, as big-bang investments into new areas rarely create the sort of feedback loop and partner engagement that can help you make the dozens of critical course corrections along the way.

This is playing out now with video conferencing, screen-sharing, and streamed gaming (e.g. Stadia and GeForce NOW) which combine various forms of input capture (mouse, keyboard, gamepad, and HID) with codec, networking, and window management improvements to displace heavily-used native apps.

Productivity apps have a similar texture, missing access to fonts, files, and high-end compute...but not for long. Google Earth, SketchUp, Figma, Photopea, and many more are starting to explore what's possible in anticipation of even richer features.

For platform makers, adjacency is both a roadmap and a caution flag. Instead of directly going after AAA games, we should instead find segments of gaming that can be mostly unlocked by what we've already got plus a small number of critical additions that unlock the key experience features. Same with IDEs and developer tools. Same with utilities. Instead of adding direct support for hardware features available on less than 1% of deployed devices and supported by a single OS, we should focus on features already within the set of things most computers can do that enjoy wide OS support.

The time provided to learn and course-correct is a secondary benefit to adjacency-based incremental addition. New APIs always have issues, and ironing out the kinks, removing performance cliffs, and understanding the next set of adjacent needs happens best when talking with bought-in developers who are succeeding (if trailblazing) in your newly-opened space.

The Committee To Cast The Web In Amber

There is a contingent of browser vendors today who do not wish to expand the web platform to cover adjacent use-cases or meaningfully close the relevance gap that the shift to mobile has opened.

Apple is "deciding not to implement" a long list of features for reasons that boil down to some variant of "it's hard and we don't want to try". This is, of course, bad for users and developers because it creates a larger market for substantially less secure, substantially less mediated, and less privacy-preserving native software.[17][18][19]

Going back to our stagnant meta-platform straw-person, what result should we expect?

Missing alt text

At any short-run moment, there won't be much change. Remember, developers don't feel trends as a smooth set of tiny increments. The "lumpy" nature of platform choices makes short-run gap growth less of a concern, particularly when there have historically been fast catch-up phases between stagnant periods.

Further, the status quo does not change quickly, and the web is one of a handful of generational computing platforms. But there have been others that have fallen into relative disuse, and once the smell of death hangs over a platform and its use declines below a fuzzy threshold, precipitous ecosystem collapse follows, reducing the platform's carrying capacity going forward. This doom-loop effect precludes even medium-term recovery to growth. Importantly, collapse isn't the same thing as extinction; mainframes hardly went away, but the set of use-cases primarily addressed by Fortan and COBOL programmers has been fixed in a narrow band for a long, long time. Once the ecosystem collapses, return to growth is prevented by market perception as much as technical capacity. Nobody buys a car they can't imagine themselves driving.[20]

It takes a lot to push a vibrant ecosystem to collapse. Technical ecosystems are complex systems, with layers of resilience built-in. Unfortunately, much has already been done to speed the web's irrelevance.

Tools like Cordova and Electron are manifestations of a relevance gap so large, so painful that developers are willing to abandon one (or more) core benefits of web deployment in return for (presumed) necessary discovery and capability access parity.

Claims that adjacent APIs are rarely used are (generously) confused.[21] Low use of an individual API says nothing about the size of the gap it contributes to or the cumulative impact of un-addressable use-cases growing without bound. Such claims are also wildly out of step with the demonstrated needs of developers and the breadth of capabilities available to native app developers.

Developers building on meta-platforms are in competition with each other, but also services built directly to underlying operating system APIs. Apple is claiming (simultaneously) that native app developers should continue to enjoy a nearly unbounded advantages about commodity capabilities and that meaningful engine competition on iOS isn't worth wanting.[22] Never mind that a competition can demonstrate how common capabilities can be safely provided by browsers willing to risk their reputations and market position to pioneer new protections.[23]

Impressively, these arguments are made with a straight face against a backdrop of more than a decade to come up with workable alternatives.[24]

What developers should reasonably expect from viable meta-platforms is not instant access to every single feature of high-margin, low-volume devices. However, steady integration of things most computers can do into the safer, standards-based, open, universally available meta-platforms should be the norm. Something is fundamentally broken in a market when a participant prevents meta-platforms from closing the commodity capability gap. iOS is prevalent enough in discrete, wealthy geographies like the US and the UK for Apple's anti-browser-choice policies to distort the market worldwide. This distortion is more, not less, impactful for its prevalance among the wealthy. Folks with money for thousand dollar phones shape technology decision making and purchasing. If the web can't be relevant to them, it's a dead letter.

Web developers do not (and should not) expect zero-friction availability from every web page to the entire suite of sensors attached to every computer.[25] It wouldn't serve anyone, and it's not what we're providing through Project Fugu. Instead, we are doing the work to explore the fuzzy and complex space of user consent, usage indicators, revokability, and recall — the exact space that operating systems are iterating on too, but with worse results (on average). We seek at every moment to build upon whatever baseline level of safety that operating systems provide and do better along these axes, the way the web historically has with other sensitive capabilities.

Competition is key to unlocking new, creative approaches to this set of problems and the next. Without competition, and paired to structural under-investment by both Apple and Mozilla, we know what a growing relevance gap will do; it's just a question of time.

Every organism cast in amber is, after all, deceased.

Thanks to Josh Bell, Jonathan Bingham, Henrik Joreteg, Frances Berriman, Jeffery Yasskin, and Sam Dutton for their thoughtful feedback on drafts of this post. Remaining errors persist because I produce them at a rate their combined powers cannot surmount.

  1. Both Fizz and Fugu heavily prioritised the needs of engaged partners who communicated clearly what would unblock them in moving their best experiences to the web. Our seemingly random walk has been anything but. ↩︎

  2. "The real problems" are invariably defined in terms of incremental improvements to better serve developers already on the platform. In a situation where your platform supports most of computing, that's not wrong.

    The web, however, is not in that situation, thanks to mobile, which is now most of computing. Focusing exclusively or primarily on current developers when you're the underdog is, therefore, a critical strategic error. Unlocking what's needed to bring computing to the web on mobile is the only way to stave off irrelevance. ↩︎

  3. Apple and Mozilla's privacy argument requires specialist knowledge to assess, which might be why they picked it.

    Why isn't it a privacy risk to add all these new scary sounding features to the web?

    First, know that no browser is currently in a position to make meaningful anti-fingerprinting claims of any sort. Apple's early attempts backfired, ironically creating an unclearble "supercookie". Side channels are hard to defeat. To prevent effective fingerprinting we must ensure that no more than 31 or 32 bits of potential identifying information are available. Above that threshold, the jig is up.

    The use of IP networks, and the lack of pervasive onion-routing (think Tor) uses up substantially all of this space for most users today. Variance added by screen resolutions, assistive technologies (including system font sizes), and availability of hardware acceleration for many operations (video decoding, graphics operations, cryptography, etc.) ensures that nearly all web users are above the threshold even with the entire set of currently marketed privacy protections in place.

    What is left, then, is a set of wicked problems: we must work to prevent expansion of ambiently available additional information while trying to remediate existing issues, while at the same time acknowledging that solutions to the current set fingerprinting surface-area issues are going to be applicable to new features too. Existing geolocation, WebGL, and directory upload APIs have substantially the same problems as presented by the new crop of features.

    There is, of course a simple solution: a browser that doesn't do very much.

    Nobody is making or marketing that browser today (tho they easily could), and the counterfactual in which most web use moves to it is also a world where computing moves off the web and onto fundamentally less secure and less privacy-preserving platforms. Assuming we don't want that, we must find solutions to privacy that can be incrementally adopted. That's the thinking behind Chrome's work to (for the first time) write down a real threat model for general-purpose browsing and the Privacy Sandbox project. These aren't the only potential approaches, of course. Many alternatives are possible for exposing new capabilities in constrained ways to make fingerprinting less effective and/or more transparent.

    For instance, browsers could chose to expose new device APIs only to installed PWAs with separate storage, or could chose to create UIs with increasing intensity of push-back as more sites request overlapping access, or even outright cap the number of sites that a user can "bless" with a permission. This is not an exhaustive list of potential fixes. The problem is hard, pre-existing, and has many possible solutions.

    Vendors that ship the existing set of potentially problematic features while refusing to acknowledge the symmetry of the new and old capabilities — and their solutions — are not squaring with developers or users. We hope to do better through Privacy Budgets, Trust Tokens, and First-Party Sets., but as with all claims about general-purpose browser fingerprinting, we're in the early stages.

    Whatever solutions emerge will absolutely constrain Fugu APIs as well as legacy capabilities. It is non sequitur to claim privacy or fingerprinting as the basis to avoid adding features. Making the claim, however, indicates either worrisome confusion about how to effectively protect user privacy or is a telling admission of problematic under-investment. ↩︎

  4. But never LaTeX quality layout (regardless of how fast CPUs get). ↩︎

  5. It didn't have to be that way. Sun's justified suspicions of MSFT cemented Java 1.1 in the ecosystem much longer than was natural. A sad story with a bad ending. ↩︎

  6. By the time Microsoft tried the same tactics to introduce Silverlight — including paying for the rights to exclusive distribution for the '08 Summer Olympic Games — users and developers had become wary of plug-ins. A tarnished developer reputation from browser-war shenanigans surely didn't help. ↩︎

  7. Java was later plucked from server obscurity by Android's Dalvik VM which made client-side friendly trade-offs that Sun had persistently resisted; memory is precious and start-up time really does matter. ↩︎

  8. Neural net acceleration will soon be on the list. ↩︎

  9. Conversely, a perceived lack of users or capabilities is a risk to developers considering supporting a platform. ↩︎

  10. Building things separately for each OS supported has linear effects on technical performance, but creates non-linear impairments to organisational efficiency.

    Experiences might be faster because they're custom-written, but not often enough to justify the adjacent costs. From coordinating design, to iteration speed, to launch coordination, to merging metrics feedback, the slowdowns incurred at the business level can let competitors into your OODA loop unless you're product's key benefit is maximum performance on the specific silicon in question. Meta-platforms flourish (when allowed by platform players) because they meaningfully reduce the amount of time bad decisions go uncorrected. ↩︎

  11. One way to conceive of meta-platforms and open standards is that, like Open Source, they help to reduce the pricing power of incumbents regarding commodity features that just happen to be wrapped up in proprietary APIs.

    Dropping the price of long-ago developed features distributed at zero marginal cost is a form of pricing pressure. This works against rent extraction and forces competitors to plow economic windfalls into new development to justify continued high prices. To the extent it is the economy's job to accelerate the rate at which technology and scientific breakthroughs become widely-shared benefits in society, this is intensely desirable. ↩︎

  12. While not strictly necessary for advertising, things really took off when <iframe> and document.write() made ad placement dynamic. I submit, however, that news would likely have moved to the web in a fully-static, forms-and-cookies-and-images world too. ↩︎

  13. "The web is for email" is a recurring theme to this day. ↩︎

  14. Reliable offline email has been a constant struggle, starting with Gears, continuing with AppCache, and culminating in Service Workers. Only took a decade! ↩︎

  15. Arguments from consistency with the OS tend to overlook the end-state: rendering and form inputs could become consistent by getting full-fidelity with native....or they could achieve internal consistency across sites. It's really a question of which platform users spent most of their time in. If that platform is the browser, OS fidelity drops in importance for users. ↩︎

  16. If iteration can unlock outsized value, you'll know it by investigating the performance of the best-constructed apps on your platform. My experience from a decade of platform work, though, suggests that efficiency gains are most often returned to developer comfort rather than user experience. What can be built is often the same at the start and end of an optimisation journey unless the gains are at least an order of magnitude. You're usually better off tackling latency with aggressive performance budgeting. ↩︎

  17. You can take a browser vendor at face value on claimed benefits to not addressing the relevance gap the minute they ship a browser that does not allow native binaries to be downloaded or app stores linked to from the web. Until then, this represents, at best, confusion about the mission and, at worst, distraction from user-hostile inaction. ↩︎

  18. "deciding not to implement" is language that chiefly serves to deflect critique of world's first trillion-dollar company for structural under-investment in their browser engine combined with policies that explicitly prevent meaningful browser choice on iOS. This keeps others from picking up the pieces the way Firefox did on Windows in the early '00s.

    The game becomes clear when zooming out to include missing or years-late capabilities that aren't covered by the too-small "privacy" fig-leaf, including (but very much not limited to):

    ...among others. ↩︎

  19. Mozilla's position is, at best, confused. At worst, it is misdirection. For opaque reasons, the Mozilla Foundation is declining to invest sufficiently in engine development (pdf) whilst simultaneously failing to challenge Apple's anti-competition stance on mobile engine choice.

    This is a 180-degree turnaround from the energy and mission that powered the original Mozilla Manifesto and Firefox project. Declining to compete on features while giving air-cover to a competitor that won't let you stand on even footing goes against everything the web community was led to believe Mozilla stands for.

    The web and the world need a vibrant Mozilla, one that is pushing against unfairness and expanding what's possible on open platforms. A Mozilla that won't answer the call of it's manifesto is not succeeding, even on its own terms. ↩︎

  20. The market mismatch between COBOL skills and needs is a textbook refutation of the efficient market hypothesis.

    Demographics being what they are, we should have a small over-supply of COBOL programmers in 2020, rather than a crash effort to find willing pensioners or train-up new grads. Tribal identity about software systems often leaves good money on the table. ↩︎

  21. Less generously, claims of low individual API use are deployed as chaff to convince non-participants to the debate that one "side" is engaged in special-pleading and that there's no cause for legitimate concern.

    Were that the case, however, hardware vendors would begin to withdraw these sensors and capabilities to save on the almighty Bill of Materials (BOM) that directly determines OEMs profitability. If an OEM is making this case whilst shipping these sensors and supporting them with OS-level APIs and controls, caveat emptor. ↩︎

  22. Mozilla, Google, Opera, and many others have failed to make a public issue over the lack of effective browser choice in iOS. I'm disappointed by the entire browser community on this front, but hope that perhaps by banding together — and with developer support — we might coordinate a message to users that helps inform them understand how they're being denied access to the best version of the web on iOS. This is worth fighting for, including taking hits in user perception in the short run. ↩︎

  23. One particularly disingenuous argument brought about by Apple's defenders is that they couldn't have made ITP (the unintentional supercookie API) happen if meaningful browser choice had been allowed on iOS.

    This is a surprising claim, given that it was successfully introduced on Safari for MacOS.

    More perniciously, it also presumes that Apple is incapable of making an iOS browser that users will actually want. I have more faith in the quality of Apple engineering than this, having seen them best larger teams over and over in open competition. Why do Apple's nominal defenders have such little confidence? ↩︎

  24. If anything, we're excruciatingly late to unlock these capabilities in Project Fugu. One would have expected a device vendor with total control of hardware, OS, and browser to have either come to the conclusion that no third party should have access to them, or that there are conditions (which can be re-created on the web by inspection) under which it's safe.

    There isn't a middle ground, only an indefensible status quo. ↩︎

  25. Arguments about why the web can't have these things in a-priori limited, heavily mediated ways...but that they're ok in native apps...must account for the ways that native developers — including the SDKs they pervasively invite into their binaries — count on unfettered access. It's transparently false that native platform prevent fingerprinting through these mechanism; indeed, they go further and directly provide stable, rarely-cycled, on-by-default device IDs to all callers.

    Those IDs might help provide semantic transparency and improved user control in a world where OS vendors clamp down on direct fingerprinting or begin to audit SDK behaviour.

    But that's not happening.

    Curious. ↩︎

A Faster Blog, Faster

Update: These patches have now been merged into Eleventy and will be part of the upcoming 1.0 release. My thanks to everyone who tried them out and provided feedback. If you'd like to run with these patches before 1.0 is released, install Eleventy from current source via npm install 11ty/eleventy

Moving this site to Eleventy and Netlify has been a great way to improve the performance of the deployed end-product, which was reason enough to make the switch.

During the port from Wordpress I've also been pleasantly surprised at how quick the build process is, particularly given the ways build-time work has grown as I've added features like comment imports. Avoiding breakage of URLs that were previously supported by WP has necessitated multiple paginations across the full set of posts, for example to generate year indexes and numbered pages. Each of these additional steps creates several hundred more files, and the fact that everything continues to take less than 10 seconds given my newbie 11ty and nunjucks skills has been great. A quick build also makes it easier to verify changes when deploying.

10 seconds feels great for a full build, but it's not ideal in a ctrl-r development loop. Perhaps there's a better solution, but I've been iterating on content and templates by firing up a linux terminal on my 2017 Pixelbook and starting the built-in 11ty filewatcher and browser sync tools. This monitors the filesystem and fully re-builds every time a file changes. There's work in progress to make incremental builds possible for 11ty, but until then, the speed of a full build is the speed at which changes are visible.

So can we go faster?

Investigating this turned into a fun little project that has helped me learn loads about 11ty and has reacquainted me Node's performance tools.[1]

Skipping to the punchline, my blog builds 35% faster and Frances' is north of 40% faster. Trimmed build output from "before":

slightlyoff@:~/projects/infrequently$ npm run build
Writing build/feed/feed.xml from ./_posts/feed.njk.
Writing build/page/128/index.html from ./_posts/
Copied 1189 files / Wrote 1175 files in 7.83 seconds (6.7ms each, v0.11.0)

...and after...

Writing build/page/128/index.html from ./_posts/
Copied 1189 files / Wrote 1175 files in 5.75 seconds (4.9ms each, v0.11.0)

The changes themselves aren't particularly clever. Eleventy is I/O heavy, and for correctness sake, it didn't do much caching. The caches added by these changes are likely error prone and may bloat build memory for large sites. If you've got a big 11ty project, I'd appreciate help testing them.

To do that, replace Eleventy in your project with[2]:

$ cd project-directory
$ npm install slightlyoff/eleventy#faster-builds

Then re-build your project and, if you can, note down before/after timings and potential correctness issues w/ content. Issues (or success stories) very much appreciated in the PR conversation. Details about build hardware, OS, and template configuration are particularly useful.

It's been fun getting this far, and while I expect fully incremental builds to make this work obsolete at development time, build speedups still help deployment through Netlify or other cloud-build systems.

  1. The addition of Node debugging in Chrome Devtools, combined with good integration of console.profile/console.profileEnd, plus --debug-brk made many things easier. ↩︎

  2. A previous version of this post included long-winded sibling-directory installs, but Mathias Bynens dropped some NPM knowledge on me, simplifying things significantly. ↩︎

New Blog, Who Dis?

For the past few months, I've been on-again-off-again working to migrate this site from WordPress hosted on a linux VPS to 11ty ("eleven-tea") hosted on Netlify.

I'd avoided the move for sundry reasons that boiled down to inertia, but a recent weekend spent on maintainance — not to mention the treadmill of re-testing to ensure WP hadn't inadvertantly bloated pages via plugin "upgrade" — finally wore me down.

This is a strictly lateral move in terms of functionality, and writing importers and ensuring that urls didn't break was an adventure I'm not keen to revisit. Most of the complexity has been a consequence of this blog's sFTP-era Blogger → hosted Blogger → html-flavoured-WordPress → markdown-flavoured-WordPress → markdown-in-git transitions. I might write up the process at some point and perhaps publish the import script that consumed so much time, but this "first post!!!!" will have to suffice for now.

The transition has offered me the chance to fix niggling performance issues and opens the door to more and better in future. It has also been nice to have something else to plink away at (besides work) whilst the weeks in isolation flow anonymously, seemingly faster every day.

Simpler times, Tokyo.
Simpler times, Tokyo.

Older Posts

Newer Posts