Infrequently Noted

Alex Russell on browsers, standards, and the process of progress.

Cache and Prizes

Serious Platforms Don't Play Favourites

If you work on a browser, you will often hear remarks like, Why don't you just put [popular framework] in the browser?

This is a good question — or at least it illuminates how browser teams think about tradeoffs. Spoiler: it's gnarly.

Before we get into it, let's make the subtext of the proposal explicit:

None of this holds.

The best available proxy data also suggests that shared caches would have a minimal positive effect on performance. So, it's an idea that probably won't work the way anyone wants it to, likely can't do much good, and might do a lot of harm.[1]

Understanding why requires more context, and this post is an attempt to capture the considerations I've been outlining on whiteboards for more than a decade.

Trust Falls

Every technical challenge in this design space is paired with an even more daunting governance problem. Like other successful platforms, the web operates on a finely tuned understanding of risk and trust, and deprecations are particularly explosive.[2] As we will see, pressure to remove some libraries to make space for others will be intense.

That's the first governance challenge: who will adjudicate removals? Browser vendors? And if so, under what rules? The security implications alone mean that whomever manages removals will need to be quick, fair, and accurate. Thar be dragons.

Removal also creates a performance cliff for content that assumes cache availability. Vendors have a powerful incentive to "not break the web ". Since cache sizes will be limited (more on that in a second), the predictable outcome will be slower library innovation. Privileging a set of legacy frameworks will create disincentives to adopt modern platform features and reduce the potential value of modern libraries. Introducing these incentives would be an obvious error for a team building a platform.

Entropic Forces

An unstated, but ironclad, requirement of shared caches is that they are uniform, complete, and common.

Browsers now understand the classic shared HTTP cache behaviour as a privacy bug.

Cache timing is a fingerprinting vector that is fixed via cache partitioning by origin. In the modern cache-partitioned world, if a user visits, which fetches, then visits, which displays the same image, it will be requested again. Partitioning by origin ensures that can't observe any details about that user's browsing history. De-duplication can prevent multiple copies on disk, but we're out of luck on reducing network costs.

A shared library cache that isn't a fingerprinting vector must work like one big zip file: all or nothing. If a resource is in that bundle — and enough users have the identical bundle — using a resource from it on both and won't leak information about the user's browsing history.

Suppose a user has only downloaded part of the cache. A browser couldn't use resources from it, lest missing resources (leaked through a timing side channel) uniquely identify the user. These privacy requirements put challenging constraints on the design and pratical effectiveness of a shared cache.

Variations on a Theme

But the completeness constraint is far from our only design challenge. When somebody asks, Why don't you just put [popular framework] in the browser?, the thoughtful browser engineer's first response is often which versions?

This is not a dodge.

The question of precaching JavaScript libraries has been around since the Prototype days. It's such a perennial idea that vendors have looked into the real-world distribution of library use. One available data source is Google's hosted JS libraries service.

Last we looked, jQuery (still the most popular JS tool by a large margin) showed usage almost evenly split between five or six leading versions, with a long tail that slowly tapers. This reconfirms observations from the HTTP Archive, published by Steve Souders in 2013.

TL;DR?: Many variants of the same framework will occur in the cache's limited space. Because a few "unpopular" legacy tools are large, heavily used, and exhibit flat distribution of use among their top versions, the breadth of a shared cache will be much smaller than folks anticipate.

The web is not centrally managed, and sites depend on many different versions of many different libraries. Browsers are unable to rely on semantic versioning or trust file names because returned resources must contain the exact code that developers request. If browsers provide a similar-but-slightly-different file, things break in mysterious ways, and "don't break the web" is Job #1 for browsers.

Plausible designs must avoid keying on URLs or filenames. Another sort of opt-in is required because:

Subresource Integrity (SRI) to the rescue! SRI lets developers add a hash of the resource they're expecting as a security precaution, but we could re-use these assertions as a safe cache key. Sadly, relatively few sites deploy SRI today, with growth driven by a single source (Shopify), meaning developers aren't generally adopting it for their first-party dependencies.

It turns out this idea has been circulating for a while. Late in the drafting of this post, it was pointed out to me by Frederik Braun that Brad Hill had considered the downsides of a site-populated SRI addressable cache back in 2016. Since that time, SRI adoption has yet to reach a point where it can meaningfully impact cache hit rates. Without pervasive SRI, it's difficult to justify the potential download size hit of a shared cache, not to mention the ongoing required updates.

The size of the proposed cache matters to vendors because browser binary size increases negatively impact adoption. The bigger the download, the less likely users are to switch browser. The graphs could not be more explicit: downloads and installs fail as binaries grow.

Browser teams aggressively track and manage browser download size, and scarce engineering resources are spent to develop increasingly exotic mechanisms to distribute only the code a user needs. They even design custom compression algorithms to make incremental patch downloads smaller. That's how much wire size matters. Shared caches must make sense within this value system.

Back of the Napkin

So what's a reasonable ballpark for a maximum shared cache? Consider:

The last point is crucial because it will cement the cache's contents for years, creating yet another governance quagmire. The space available for cache growth will be what's left for the marginal user after accounting for increases in the browser binary. Most years, that will be nothing. Browser engineers won't give JS bundles breathing room they could use to win users.

Given these constraints, it's impossible to imagine a download budget larger than 20 MiB for a cache. It would also be optional (not bundled with the browser binary) and perhaps fetched on the first run (if resources are available). Having worked on browsers for more than a decade, I think it would be shocking if a browser team agreed to more than 10 MiB for a feature like this, especially if it won't dramatically speed up the majority of existing websites. That is unlikely given the need for SRI annotations or the unbecoming practice of speeding up high-traffic sites, but not others. These considerations put tremendous downward pressure on the prospective size budget for a cache.

20 MiB over the wire expands to no more than 100 MiB of JS with aggressive Brotli compression. A more likely 5 MiB (or less) wire size budget provides something like ~25 MiB of on-disk size. Assuming no source maps, this may seem like a lot, but recall that we need to include many versions of popular libraries. A dozen versions of jQuery, jQuery UI, and Moment (all very common) burn through this space in a hurry.

One challenge for fair inclusion is the tension between site-count-weighting and traffic-weighting. Crawler-based tools (like the HTTP Archive's Almanac and BuiltWith) give a distorted picture. A script on a massive site can account for more traffic than many thousands of long-tail sites. Which way should the policy go? Should a cache favour code that occurs in many sites (the long tail), or the code that has the most potential to improve the median page load?

Thar be more dragons.

Today's Common Libraries Will Be Underrepresented

Anyone who has worked on a modern JavaScript front end is familiar with the terrifyingly long toolchains that dominate the landscape today. Putting aside my low opinion of the high costs and slow results, these tools all share a crucial feature: code motion.

Modern toolchains ensure libraries are transformed and don't resemble the files that arrive on disk from npm. From simple concatenation to the most sophisticated tree-shaking, the days of downloading library.min.js and SFTP-ing one's dependencies to the server are gone. A side effect of this change has been a reduction in the commonality of site artifacts. Even if many sites depend on identical versions of a library, their output will mix that code with bits from other tools in ways that defeat matching hashes.

Advocates for caches suggest this is solvable, but it is not — at least in any reasonable timeframe or in a way that is compatible with other pressures outlined here. Frameworks built in an era that presumes transpilers and npm may never qualify for inclusion in a viable shared cache.

Common Sense

Because of privacy concerns, caches will be disabled for a non-trivial number of users. Many folks won't have enough disk space to include a uniform and complete version, and the cache file will be garbage collected under disk pressure for others.

Users at the "tail" tend not to get updates to their software as often as users in the "head" and "torso" of the distribution. Multiple factors contribute, including the pain of updates on slow networks and devices, systems that are out of disk space, and harmful interactions with AV software. One upshot is that browsers will need to delete or disable shared caches for users in this state so they don't become fingerprinting vectors. A simple policy would be to remove caches from service after a particular date, creating a performance cliff that disproportionately harms those on the slowest devices.

First, Do No Harm

Code distributed to every user is a tax on end-user resources, and because caches must be uniform, complete, and common, they will impose a regressive tax. The most enfranchised users with the most high-performance devices and networks will feel their initial (and ongoing) download impacted the least. In contrast, users on the margins will pay a relatively higher price for any expansions of the cache over time and any differential updates to it.

Induced demand is real, and it widens inequality, rather than shrinking it.

If a shared cache is to do good, it must do good for those in the tail and on the margins, not make things worse for them.

For all of the JS ecosystem's assertions that modern systems are fast enough to support the additional overhead of expensive parallel data structures, slow diffing algorithms, and unbounded abstractions, nearly all computing growth has occurred over the past decade at the low end. For most of today's users, the JS community's shared assumptions about compute and bandwidth abundance have been consistently wrong.

The dream of a global, shared, user-subsidised cache springs from the same mistaken analysis about the whys and wherefores of client-side computing. Perhaps one's data centre is post-scarcity, and maybe the client side will be too, someday. But that day is not today. And it won't be this year either.

Cache Back

Speaking of governance, consider that a shared cache would suddenly create disincentives to adopt anything but the last generation of "winner" libraries. Instead of fostering innovation, the deck would be stacked against new and innovative tools that best use the underlying platform. This is a double disincentive. Developers using JS libraries would suffer an additional real-world cost whenever they pick a more modern option, and browser vendors would feel less pressure to integrate new features into the platform.

As a strategy to improve performance for users, significant questions remain unanswered. Meanwhile, such a cache poses a clear and present danger to the cause of platform progress. The only winners are the makers of soon-to-be obsolete legacy frameworks. No wonder they're asking for it.

Own Goals

At this point, it seems helpful to step back and consider that the question of resource caching may have different constituencies with different needs:

For self-evident security and privacy reasons, browser vendors will be the ones to define the eventual contents of a shared cache and distribute it. Therefore, it will be browser imperatives that drive its payload. This will lead many to be surprised and disappointed at the contents of a fair, shared global cache.

First, to do best by users, the cache will likely be skewed away from the tools of engaged developers building new projects on the latest framework because the latest versions of libraries will lack the deployed base to qualify for inclusion. Expect legacy versions of jQuery and Prototype to find a much established place in the cache than anything popular in "State Of" surveys.

Next, because it will be browser vendors that manage the distribution of the libraries, they are on the hook for creating and distributing derivative works. What does this mean? In short, a copyright and patent minefield. Consider the scripts most likely to qualify based on use: code for embedding media players, analytics scripts, and ad network bootstraps. These aren't the scripts that most people think of when they propose that "[b]rowsers should ship with the top 1 GiB of npm built-in", but they are the sorts of code that will have the highest cache hit ratios.

Plus, they're also copyrighted and unlikely to be open-source, creating new headaches.

Browsers build platform features through web standards, not because it's hard to agree on technical details (although it is), but because vendors need the legal protections that liberally licensed standards provide.[4] These protections are the only things standing between hordes of lawyers and the wallets of folks who build on the web platform. Even OSS isn't enough to guarantee the sorts of patent license and pooling that Standards Development Organisations (SDOs) like the W3C and IETF provide.

A reasonable response would be to have caches constrain themselves to non-copyleft OSS, rendering them less effective.

And, so we hit bedrock. If the goal isn't to make things better for users at the margins, why bother? Serving only the enfranchised isn't what the web is about. Proposals that externalise governance and administrative costs are also unattractive to browser makers. Without a credible plan for deprecation and removal, why wade into this space? It's an obvious trap.

"That's Just Standardisation With Extra Steps!"

Another way to make libraries smaller is to upgrade the platform. New platform features usually let developers remove code, which can reduce costs. This is in the back of browser engineers minds when asked, "Why don't you just put Library X in the browser?".

A workable cache proposal features the same problems as standards development:

Why build this new tool when the existing ones are likely to be as effective, if not more so?

Compression is a helpful lens for thinking about caches and platform APIs. The things that platforms integrate into the de facto computing base are nouns and verbs. As terms become common, they no longer need to be explained every time.

Requests to embed libraries into the web's shared computing base is a desire to increase their compression ratio.

Shared precaching is an inefficient way to accomplish the goal. If we can identify common patterns being downloaded frequently, we can modify the platform to include standardised versions. Either way, developers need to account for situations when the native implementations aren't available (polyfilling).

Given that a shared cache system will look nothing like the dreams of those who propose them, it's helpful to instead ask why browser vendors are moving so slowly to improve the DOM, CSS, data idioms, and many other core areas of the platform in response to the needs expressed by libraries.

Thoughtful browser engineers are right to push back on shared caches, but the slow pace of progress in the DOM (at the hands of Apple's under-funding of the Safari team) has been a collective disaster. If we can't have unicorns, we should at least be getting faster horses.

Is There a Version That Could Work?

Perhaps, but it's unlikely to resemble anything that web developers want. First, let's re-stipulate the constraints previously outlined:

To be maximally effective, we might want a cache to trigger for sites that haven't opted-in via SRI. A bloom filter could elide SRI annotations for high-traffic files, but this presents additional governance and operational challenges.

Only resources served as public and immutable (as observed by a trusted crawler) can possibly be auto-substituted. A browser that is cavalier enough to attempt to auto-substitute under other conditions deserves all of the predictable security vulnerabilities it will create.

An auto-substitution URL list will take space, and must also be uniform, complete, and common for privacy reasons. This means that the list itself is competing for space with libraries. This creates real favouritism challenges.

A cache designed to do the most good will need mechanisms to work against induced demand. Many policies could achieve this, but the simplest might be for vendors to disable caches for developers and users on high-end machines. We might also imagine adding scaled randomness to cache hits: the faster the box, the more often a cache hit will silently fail.

Such policies won't help users stuck at the tail of the distribution, but might add a pinch of balance to a system that could further slow progress on the web platform.

A workable cache will also need a new governance body within an existing OSS project or SDO. The composition of such a working group will be fraught. Rules that ensure representation by web developers (not framework authors) and browser vendors can be postulated, but governance will remain challenging. How the interests of security researchers and users on the margins are represented are open problems.

So, could we add a cache? If all the challenges and constraints outlined above are satisfied, maybe. But it's not where I'd recommend anyone who wants to drive the web forward invest their time — particularly if they don't relish chairing a new, highly contentious working group.

Thanks to Eric Lawrence, Laurie Voss, Fred K. Schott, Frederik Braun, and Addy Osmani for their insightful comments on drafts of this post.

  1. My assessment of the potential upside of this sort of cache is generally negative, but in the interest of fairness, I should outline some ways in which pre-caching scripts could enable them to be sped up:

    • Scripts downloaded this way can be bytecode cached on the device (at the cost of some CPU burn), but this will put downward pressure on both the size of the cache (as bytecode takes 4-8× the disk space of script text) and on cache growth (time spent optimizing potentially unused scripts is a waste).
    • The benefits of download time scale with the age of the script loading technique. For example, using a script from a third-party CDN requires DNS, TCP, TLS, and HTTP handshaking to a new server, all of which can be shortcut. The oldest sites are the most likely to use this pattern, but are also most likely to be unmaintained.
  2. Case in point: it was news last year when the Blink API owners[5] approved a plan to deprecate and remove the long-standing alert(), confirm(), and prompt() methods from within cross-origin <iframe>s. Not all <iframe>s would be affected, and top-level documents would continue to function normally. The proposal was scapular — narrowly tailored to address user abuse while reducing collateral damage.

    The change was also shepherded with care and caution. The idea was floated in 2017, aired in concrete form for more than a year, and our friends at Mozilla spoke warmly of it. WebKit even implemented the change. This deprecation built broad consensus and was cautiously introduced.

    It blew up anyway.[6]

    Overnight, influential web developers — including voices that regularly dismiss the prudential concerns of platform engineers — became experts in platform evolution, security UX, nested event loops in multi-process applications, Reverse Origin Trials, histograms, and Chromium's metrics. More helpfully, collaboration with affected enterprise sites is improving the level of collective understanding about the risks. Changes are now on hold until the project regains confidence through this additional data collection.

    This episode and others like it reveal that developers expect platforms to be conservative. Their trust in browsers comes from the expectation that the surface they program to will not change, particularly regarding existing and deployed code.

    And these folks weren't wrong. It is the responsibility of platform stewards to maintain stability. There's even a helpful market incentive attached: browsers that don't render all sites don't have many users. The fast way to lose users is to break sites, and in a competitive market, that means losing share. The compounding effect is for platform maintainers to develop a (sometimes unhelpful) belief that moving glacially is good per se.

    A more durable lesson is that, like a diamond, anything added to the developer-accessible surface of a successful platform may not be valuable — but it is forever.[7] ↩︎

  3. With the advent of H/2 connection re-use and partitioned caches, hosting third-party libraries has become an anti-pattern. It's always faster to host files from your server, which means a shared cache shouldn't encourage users to centralise on standard URLs for hosted libraries, lest they make the performance of the web even worse when the cache is unavailable. ↩︎

  4. For an accessible introduction to the necessity of SDOs and recent history of modern technical standard development, I highly recommend Open Standards and the Digital Age by Andrew L. Russell (no relation). ↩︎

  5. Your humble narrator serves as a Blink API OWNER and deserves his share of the blame for the too-hasty deprecation of alert(), confirm(), and prompt().

    In Blink, the buck stops with us, not the folks proposing changes, and this was a case where we should have known that our lingering "enterprise blindness" [6:1] in the numbers merited even more caution, despite the extraordinary care taken by the team. ↩︎

  6. Responsible browser projects used to shoot from the hip when removing features, which often led to them never doing it due to the unpredictability of widespread site breakage.

    Thankfully, this is no longer the case, thanks to the introduction of anonymised feature instrumentation and metrics. These data sets are critical to modern browser teams, powering everything from global views of feature use to site-level performance reporting and efforts like Core Web Vitals.

    One persistent problem has been what I've come to think of as "enterprise blindness".

    In the typical consumer scenario, users are prompted to opt-in to metrics and crash reporting on the first run. Even if only a relatively small set of users participate, the Law of Large Numbers ensures our understanding of these histograms is representative across the billions of pages out there.

    Enterprises, meanwhile, tend to roll out software for their users and simultaneously push down policy configurations to machines that disable metrics reporting. The result is that enterprise users and the web applications they frequent are dramatically under-reported in the public stats.

    Given the data at hand, the team deprecating cross-origin <iframe> prompts was being responsible. It's just that the data had a blind spot, one whose size has been maddeningly difficult to quantify. ↩︎ ↩︎

  7. Forever, give or take half a decade. ↩︎

Towards a Unified Theory of Web Performance

Note: This post first ran as part of Sergey Chernyshev and Stoyan Stefanov's indispensible annual series. It's being reposted here for completeness, but if you care about web performance, make sure to check out the whole series and get subscribed to their RSS feed to avoid missing any of next year's posts.

In a recent Perf Planet Advent Calendar post, Tanner Hodges asked for what many folks who work in the space would like for the holidays: a unified theory of web performance.

I propose four key ingredients:

  • Definition: What is "performance" beyond page speed? What, in particular, is "web performance"?
  • Purpose: What is web performance trying to accomplish as a discipline? What are its goals?
  • Principles: What fundamental truths are guiding the discipline and moving it forward?
  • Practice: What does it look like to work on web performance? How do we do it?

This is a tall order!

A baseline theory, doctrine, and practicum represent months of work. While I don't have that sort of space and time at the moment, the web performance community continues to produce incredible training materials, and I trust we'll be able to connect theory to pracice once we roughly agree on what web performance is and what it's for.

This Is for Everyone

Missing alt text
Tim Berners-Lee tweets that 'This is for everyone' at the 2012 Olympic Games opening ceremony using the NeXT computer he used to build the first browser and web server.
Tim Berners-Lee tweets that 'This is for everyone' at the 2012 Olympic Games opening ceremony using the NeXT computer he used to build the first browser and web server.

Embedded in the term "web performance" is the web, and the web is for humans.

That assertion might start an argument in the wrong crowd, but 30+ years into our journey, attempts to promote a different first-order constituency are considered failures, as the Core Platform Loop predicts. The web ecosystem grows or contracts with its ability to reach people and meet their needs with high safely and low friction.

Taking "this is for everyone" seriously, aspirational goals for web performance emerge. To the marginal user, performance is the difference between access and exclusion.

The mission of web performance is to expand access to information and services.

Page Load Isn't Special

It may seem that web performance comprises two disciplines:

  1. Optimising page load
  2. Optimising post-load interactions

The tools of performance investigators in each discipline overlap to some degree but generally feel like separate concerns. The metrics that we report against implicitly cleave these into different "camps", leaving us thinking about pre- and post-load as distinct universes.

But what if they aren't?

Consider the humble webmail client.

Here are two renderings of the same Gmail inbox in different architectural styles: one based on Ajax, and the other on "basic" HTML:

The Ajax version of Gmail with two messages
The Ajax version of Gmail loads 4.8MiB of resources, including 3.8MiB of JavaScript to load an inbox containing two messages.

The same inbox in Gmail's 'simple HTML' mode
The 'basic' HTML version of Gmail loads in 23KiB, including 1.3KiB of JavaScript.

The difference in weight between the two architectures is interesting, but what we should focus on is the per interaction loop. Typing in the address bar, hitting Enter, and becoming ready to handle the next input is effectively the same interaction in both versions. One of these is better, and it isn't the experience of the "modern" style.

These steps inform a general description of the interaction loop:

  1. The system is ready to receive input.
  2. Input is received and processed.
  3. Progress indicators are displayed.
  4. Work starts; progress indicators update.
  5. Work completes; output is displayed.
  6. The system is ready to receive input.

Tradeoffs In Depth

Consider the next step of our journey, opening the first message. The Ajax version leaves most of the UI in place, whereas the HTML version performs a full page reload. Regardless of architecture, Gmail needs to send an HTTP request to the server and update some HTML when the server replies. The chief effect of the architectural difference is to shift the distribution of latency within the loop.

Some folks frame performance as a competition between Team Local (steps 2 & 3) and Team Server (steps 1 & 4). Today's web architecture debates (e.g. SPA vs. MPA) embody this tension.

Team Local values heap state because updating a few kilobytes of state in memory can, in theory, involve less work to return to interactivity (step 5) while improving the experience of steps 2 and 3.

Intuitively, modifying a DOM subtree should generate less CPU load and need less network traffic than tearing down the entire contents of a document, asking the server to compose a new one, and then parsing/rendering it along with all of its subresources. Successive HTML documents tend to be highly repetitive, after all, with headers, footers, and shared elements continually re-created from source when navigating between pages.

But is this intuitive understanding correct? And what about the other benefits of avoiding full page refreshes, like the ability to animate smoothly between states?

Herein lies our collective anxiety about front-end architectures: traversing networks is always fraught, and so we want to avoid it being jarring. However, the costs to deliver client-side logic that can cushion the experience from the network latency remain stubbornly high. Improving latency for one scenario often degrades it for another. Despite partisan protests, there are no silver bullets; only complex tradeoffs that must be grounded in real-world contexts — in other words, engineering.

As a community, we aren't very good at naming or talking about the distributional effects of these impacts. Performance engineers have a fluency in histograms and percentiles that the broader engineering community could benefit from as a lens for thinking about the impacts of design choices.

Given the last decade of growth in JavaScript payloads, it's worth resetting our foundational understanding of these relative costs. Here, for instance, are the network costs of transitioning from the inbox view of Gmail to a message:

Missing alt text
Displaying the first message requires 82KiB of network traffic in the Ajax version of Gmail, half of which are images embedded in the message.
Displaying the first message requires 82KiB of network traffic in the Ajax version of Gmail, half of which are images embedded in the message.

Displaying a message in the 'basic' HTML version requires a full page refresh.
Displaying a message in the 'basic' HTML version requires a full page refresh.

Despite fully reloading the page, the HTML version of Gmail consumes fewer network resources (~70KiB) and takes less overall time to return to interaction.
Despite fully reloading the page, the HTML version of Gmail consumes fewer network resources (~70KiB) and takes less overall time to return to interaction.

Objections to the comparison are legion.

First, not all interactions within an email client modify such a large portion of the document. Some UI actions may be lighter in the Ajax version, especially if they operate exclusively in the client-side state. Second, while avoiding a full-page refresh, steps 2, 3, and 4 in our interaction loop can be communicated with greater confidence and in a less jarring way. Lastly, by avoiding an entire back-and-forth with the server for all UI states, it's possible to add complex features — like chat and keyboard accelerators — in a way that doesn't incur loss of context or focus.

The deeper an app's session length and the larger the number of "fiddly" interactions a user may perform, the more attractive a large up-front bundle can be to hide future latency.

This insight gives rise to a second foundational goal for web performance:

We expand access by reducing latency and variance across all interactions in a user's session to more reliably return the system to an interactive state.

For sites with low interaction depths and short sessions, this implies that web performance engineering might remove as much JavaScript and client-side logic as possible. For other, richer apps, performance engineers might add precisely this sort of payload to reduce session-depth-amortised latency and variance. The tradeoff is contextual and informed by data and business goals.

No silver bullets, only engineering.

Medians Don't Matter

Not all improvements are equal. To understand impacts, we must learn to think in terms of distributions.

Our goal is to minimise latency and variance in the interactivity loop... but for whom? Going back to our first principle, we understand that performance is the predicate to access. This points us in the right direction. Performance engineers across the computing industry have learned the hard way that the sneaky, customer-impactful latency is waaaaay out in the tail of our distributions. Many teams have reported making performance better at the tail only to see their numbers get worse upon shipping improvements. Why? Fewer bouncing users. That is, more users who get far enough into the experience for the system to boot up in order to report that things are slow (previously, those users wouldn't even get that far).

Tail latency is paramount. Doing better for users at the median might not have a big impact on users one or two sigmas out, whereas improving latency and variance for users at the 75th percentile ("P75") and higher tend to make things better for everyone.

As web performance engineers, we work to improve the tail of the distribution (P75+) because that is how we make systems accessible, reliable, and equitable.

A Unified Theory

And so we have the three parts of a uniform mission, or theory, of web performance:

Perhaps a better writer can find a pithier way to encapsulate these values.

However they're formulated, these principles are the north star of my performance consulting. They explain tensions in architecture and performance tradeoffs. They also focus teams more productively on marginal users, which helps to direct investigations and remediation work. When we focus on getting back to interactive for the least enfranchised, the rest tends to work itself out.

Minimum Standards for iOS Browser Competition

There has been a recent flurry of regulatory, legislative, and courtroom activity regarding mobile OSes and their app stores. One emergent theme is Apple's shocking disregard for the spirit of legal findings it views as adverse to its interests.

Take, for instance, Apple's insistence that it would take "months" to support the addition of links to external payment providers from within apps, nevermind it had months of notice. There is a case to be made that formulating policy and constructing a comissions system takes time, but this is ridiculous.

Just sit with Apple's claim a moment. Cupertino is saying that it will take additional months to allow other people to add links within their own apps.

Or consider South Korea's law, passed last August, that elicited months of stonewalling by Google and Apple until, at last, an exasperated KCC started speaking a language Cupertino understands: fines. Having run the clock for half a year, Apple has started making noises that indicate a willingness to potentially allow alternative payment systems at some point in the future.

Never fear, though; there's no chance that grudging "compliance" will abide the plain-language meaning of the law. Apple is fully committed to predatory delay and sneering, legalistic, inch-measuring conformance. Notably, it signaled it feels entitled to skim equivalent revenues to it's monopoly rents for payments paid through third-party processors in both Korea and the Netherlands. If regulators are going to bring this to heel, they will need to start assuming bad-faith.

Dutch regulators have been engaged in a narrowly-focused enquiry into the billing practices of dating apps, and they came to remarkably similar conclusions: Apple's pracctices are flagarantly anti-competitive.

In a malign sort of consistency, Apple responded in the most obtuse, geographically constrained, and difficult-to-use solution possible. Developers would need to submit a separate, country-specific version of their app, only available from the Dutch version of the App Store, and present users with pejorative messaging. Heaven help users attempting to navigate such a mess.

This "solution" was poorly received. Perhaps learning from the KCC's experience, Dutch regulators moved to impose fines more quickly, but perhaps misjudged how little 50 million EUR is to a firm that makes 600× as much in profit per quarter. It certainly hasn't modulated Apple's duplicitous audacity.

Cupertino's latest proposed alternative to its 30% revenue cut will charge developers that use external payment processors a 27% fee which, after the usual 3% credit card processing fee... well, you can do the math.

Regulators in the low countries are rightly incensed, but Apple's disrespect for the rule of law isn't going to be reformed by slowly deployed half-measures, one vertical at a time. Structural change is necessary, and the web could bring that change if it is unshackled from Apple's slipshod browser engine.

A Floor for Functional Browser Choice

With all of this as context, we should seriously consider how companies this shameless will behave if required to facilitate genuine browser choice. What are the policy and technical requirements regulators should set to ensure fairness? How can the lines be drawn so delay and obfuscation aren't used to scuttle capable competitors? How can regulators anticipate and get ahead of brazenly bad-faith actions, not only by Apple, but Google and Facebook as well?

Geographic Games

One oligopolist response has been to change anti-competitive behaviour only within small markets, e.g. making changes only to the "dating" category of apps and only within the Netherlands. Apple and Google calculate that they can avoid the Brussels Effect by making a patchwork set of market-specific changes. A goal of this strategy is to confuse users and make life hard for rebellious developers by tightly drawing "fixes" around the letter of the law in each jurisdiction.

While technically meeting legal requirements, these systems will be so hard to use that residents and businesses blame regulators, rather than store proprietors for additional day-to-day inconvenience. Because they're implemented in software, this market well-poisoning is cost-free for Apple. It also buys bad-faith actors months of delay on substantive change while they negotiate with regulators.

Could regulators include language that stipulates how market fairness requirements cannot be met with country-specific versions of apps or capabilities? This quickly hits jurisdictional boundaries, likely triggering years of court appeals. This is undesirable as delay is the ne'er do well's friend.

Regulators generally have scope over commerce within their territorial borders. Multilateral treaty organisations like the the WTO have supranational jurisdiction but no appetite or the treaty scope to tackle firm-level competition issues. They focus instead on tariffs and "dumping" practices that privilege one nation's industries over another, as those are the sorts of disputes national laws cannot address.

A More Durable Approach

Effective regulation needs market-opening technologies that function without constant oversight. The lines drawn around undermining this technology should be so clear, and the consequences for stepping over them so painful, that even Apple, Google, and Facebook dare not go near them.

When regulators adopt similar (if not identical) regulations they increase the costs to bad actors of country-specific gamesmanship. Regulators that "harmonise" their interventions multiply the chances of compliance, creating a "Brussels of the Willing".

A competitive landscape for web browsers should be part of any compliance framework because:

For the web to perform these essential market functions on mobile, regulation must disrupt the status quo and facilitate competition from within. This also provides a solution to user safety and security concerns that pervasive sideloading may raise, as browsers are aggressively sandboxed. Meanginful choice, coupled with powerful browsers, can deliver better outcomes:

Table Stakes

Discussion of sideloading and alternative app stores often elides requirements that regulators should put in place to create competition amongst capable browsers. I have previously proposed a set of minimal interventions to ensure meaningful user choice. To restate the broadest points:

iOS Specifics

Because Apple's iOS is egregiously closed to genuine browser competition, regulators should pay specific attention to the capabilities that vendors that port engines will need. They should also ensure other capabilities are made available by default; the presumption for browser developers must be open access. Apple has shown itself to be a serial bad-faith actor regarding competition and browser choice, so while an enumeration of these features may seem pedantic, it sadly also seems necessary.

Today, Apple's Safari browser enjoys privileged access to certain APIs necessary for any competing browser vendor that wants to match Safari's features. Only Safari can:

As a general rule, competing browsers must also be allowed access to all private and undocumented APIs that are used by Safari, as well as iOS entitlements granted to other applications.

Regulators must also ensure capabilities are not prohibited or removed from browsers by secret agreements that Cupertino forces developers to sign. Further (and it's a shame this has to be said), Apple must not be allowed to comply with these terms by further neutering Safari's already industry-trailing feature set.

Apple must also be required to allow browsers with alternative engines to be procured directly through its App Store. It is easy to predict a world of country-specific sideloading regulations, with Apple attempting to blunt the impact of competitive browsers by continuing to banish them from their "legit" discovery surface.

Web browsers must also be allowed to implement the Web Payments API without being forced to use Apple Pay as the only back end. Apple must further be enjoined from requiring specific UI treatments that subvert these flows and prejudice users away from open payment systems.

Lastly, Apple must not be allowed to publish new versions of browsers through an arbitrary and capricious "review" process. Regulators must demand that Apple be forced to publish new browser versions and, if it objects to features within them, file a request for regulatory arbitration of the dispute post publication. Apple has long demonstrated it cannot be trusted with the benefit of the doubt in this area, and allowing updates to flow quickly is critical to ensuring users of the web remain safe.

Only within the contours of this sort of regime can ongoing enforcement of negotiated policy proceed in good faith.

A Week to Define the Web for Decades

If you live or do business in the UK or the US, what you do in the next seven days could define the web for decades to come. By filing public comments with UK regulators and US legislators this week, you can change the course of mobile computing more than at any other time in the past decade. Read on for why this moment matters and how to seize the day.

By way of background, regulators in much of the anglophone world (and beyond) spent much of 2021 investigating the state of the mobile ecosystem.

This is important because Apple has succeeded in neutering the web's potential through brazenly anti-competitive practices and obfuscation. Facebook and Google, meanwhile, have undermined user agency in browser choice for fun and profit.

I kid.

It was all for profit:

Public statements from leading authorities who have looked into this behaviour leave a distinct impression of being unimpressed. Here's the unflappably measured UK Competition and Markets Authority (CMA) weighing in last month:

Apple and Google have developed a vice-like grip over how we use mobile phones and we're concerned that it's causing millions of people across the UK to lose out.

The CMA's 400+ page interim report (plus an additional ~200 pages of detailed appendices) didn't make the waves it deserved when it was released near the winter holidays.[1] That's a shame as the report is by turns scathing and detailed, particularly in its proposed remedies, all of which would have a profoundly positive impact on you, me, and anyone else who uses the mobile web:

The report sets out a range of actions that could be taken to address these issues, including:

  • Making it easier for users to switch between iOS and Android phones when they want to replace their device without losing functionality or data.
  • Making it easier to install apps through methods other than the App Store or Play Store, including so-called "web apps".
  • Enabling all apps to give users a choice of how they pay in-app for things like game credits or subscriptions, rather than being tied to Apple's and Google's payment systems.
  • Making it easier for users to choose alternatives to Apple and Google for services like browsers, in particular by making sure they can easily set which browser they have as default.

This is shockingly blunt language from a regulatory body:

Our market study has provisionally found that:

❌ People aren’t seeing the full benefit of innovative new products and services such as cloud #gaming and web #apps.


Our provisional findings also suggest:

💷 customers could be facing higher prices than they would in a more competitive market.


The report demonstrates that the CMA understands the anti-competitive browser and browser-engine landscape too. Its findings are no less direct than the summary:

Impact of the WebKit restriction

As a result of the WebKit restriction, there is no competition in browser engines on iOS and Apple effectively dictates the features that browsers on iOS can offer[.]

The CMA has outlined its next steps and is requesting comment until February 7th, 2022.

Apple, in particular, has demonstrated that it is a bad actor with regards to competition law. This post could easily contain nothing but a rundown of fruity skulduggery; that's how brazen Cupertino's anti-competitive practices have become. Suffice to say, Apple sees being fined 5M EUR per week over entirely reasonable requests a "cost of doing business." Big respect-for-the-rule-of-law vibes.

But this sort of thing isn't going to last. Regulators don't like being taken for a ride.

...Meanwhile in Washington

On this side of the pond, things are getting serious. In just the past two weeks:

We're even getting gauzy coverage of pro-regulatory senators. It's quite the moment, and indicates dawning awareness of these blatantly anti-competitive practices.

This Is About More Than Browsers

It's tempting to think of browser choice and app store regulation as wholly separate concerns, but neither the web nor apps exist in a vacuum. As the desktop web becomes ever-more powerful on every OS, even the most sophisticated app developers gain more choice in how they reach users.

Unleashing true choice and competition in mobile browsers won't only help web developers and users, it will level the playing field more broadly. Native app developers that feel trapped in abusive policy regimes will suddenly have real alternatives. This, in turn, will put pricing pressure on app store owners that extract egregious rents today.

Web apps and PWAs compete with app stores for distribution, lowering the price to develop and deliver competitive experiences. This allows a larger pool of developers and businesses to "play".

App store "review" and its capricious policy interpretations have always been tragicomic, but true competition is needed to drive the point home. Businesses are forced into the app store, requiring they spend huge amounts to re-build features multiple times. Users risk unsafe native app platforms when the much-safer web could easily handle many day-to-day tasks. We're only stuck in this morass because it helps Google and Apple build proprietary moats that raise switching costs and allow them to extort rents from indie developers and hapless users.

A better future for mobile computing is possible when the web is unshackled, and that will only happen when competition has teeth.

What You Can Do

This is the last week to lodge comment by email with the UK's CMA regarding the findings of its interim report. Anyone who does business in the UK and cares about mobile browser choice should send comments, both as an individual and through corporate counsel.

For US residents, the speed at which legislation on this front is moving through Congress suggests that this is the moment for a well-timed email or, more preferably, call to your elected senator.

If you live or do business in the US or the UK, this week matters.

Whichever geography you submit comment to, please note that regulators and legislators have specific remits and will care more or less depending on the salience of your input to their particular goals. To maximize your impact, consider including the following points in your comments:

Leaving your contact information for follow-up and verification never hurts either.

It's been 761 weeks since Apple began the destruction of mobile browser choice, knowingly coating its preference for a "native" device experience (in web-friendly garb) at the expense of the mobile web. This is the week you can do something about it.

Carpe diem.

  1. While the tech press may have been asleep at the wheel, Bruce Lawson covered the report's release. Read his post for a better sense of the content without needing to wade through 600+ pages. ↩︎

Washed Up

The term 'web3' is an attempted whitewash, and it isn't working
Photo by Jarrod Reed

The rhetorical "web3" land-grab by various VCs, their shills, and folks genuinely confused about legal jurisdiction may appear to be a done deal.

VCs planted the flag with sufficient force and cash (of dubious origin) to cause even sceptical outlets to report on it as though "web3" is a real thing.

Which it is not — at least not in any useful sense:

Thank god someone finally solved the problem of not being able to pay money to pretend you own a jpg

Technologies marketed under the "web3" umbrella are generally not fit for purpose, unless that purpose is to mislead:

Out of curiosity I dug into how NFT's actually reference the media you're "buying" and my eyebrows are now orbiting the moon

Short version:

The NFT token you bought either points to a URL on the internet, or an IPFS hash. In most circumstances it references an IPFS gateway on the internet run by the startup you bought the NFT from.

Oh, and that URL is not the media. That URL is a JSON metadata file

Here's an example. This artwork is by Beeple and sold via Nifty:

The NFT token is for this JSON file hosted directly on Nifty's servers:

Image from Tweet

THAT file refers to the actual media you just "bought". Which in this case is hosted via a @cloudinary CDN, served by Nifty's servers again.

So if Nifty goes bust, your token is now worthless. It refers to nothing. This can't be changed.

"But you said some use IPFS!"

Let's look at the $65m Beeple, sold by Christies. Fancy.

That NFT token refers directly to an IPFS hash ( We can take that IPFS hash and fetch the JSON metadata using a public gateway:

Image from Tweet

So, well done for referring to IPFS - it references the specific file rather than a URL that might break!

...however the metadata links to ""

This is an IPFS gateway run by, the NFT-minting startup.

Who will go bust one day

It's all going about as well as one might expect.

What has perhaps earned proponents of JSON-files-that-point-to-JPGs less scorn, however, are attempts to affiliate their technologies with the web when, in fact, the two are technically unrelated by design. The politics of blockchain proponents have led them to explicitly reject the foundational protocols and technical underpinnings of the web. "web3" tech can't be an evolution of the web because it was designed to stand apart.

What is the web proper?

Cribbing from Dieter Bohn's definition, the web is the set of HTML documents (and subresources they reference) currently reachable via links.

To be on the web is to be linked to and linked from — very literally, connected by edges in a great graph. And the core of that connection? DNS, the "Domain Name System" that allows servers running at opaque and forgettable IP addresses to be found at friendlier names, like

DNS underpins URLs. URLs and links make the web possible. Without these indirections, the web would never have escaped the lab.

These systems matter because the web is for humans, and humans have feeble wetware that doesn't operate naturally on long strings of semi-random numbers and characters. This matters to claims of decentralisation because, underneath DNS, the systems that delivered this very page you're reading to your screen are, in fact, distributed and decentralised.

Naming is centralising.

"web3" partisans often cast a return to nameless, unmemorable addresses as a revolution when their systems rely on either the same centralising mechanisms or seek to re-create them under new (less transparent, equally rent-seeking) management. As a technical matter, browsers are capable of implementing content-addressed networking, thanks to Web Packages, without doing violence to the web's gaurantees of safety in the process. Still, it turns out demand for services of this sort hasn't been great, in part, because of legitimate privacy concerns.

"web3" proponents variously dismiss and (sometimes) claim to solve privacy concerns, but the technical reality is less hospitable: content-addressed data must be either fully public or rely on obscurity.

Accessing "web3"-hosted files is less private because the architecture of decentralisation choosen by "web3" systems eschews mechanisms that build trust in the transport layer. A fully public, immutable ledger of content, offered by servers you don't control and can't attribute or verify, over links you can't trust, is hardly a recipe for privacy. One could imagine blockchain-based solutions to some of these problems, but this isn't the focus of "web3" boosters today.

Without DNS-backed systems like TLS there's little guarantee that content consumption will prevent tracking by parties even more unknowable than in the "web 2.0" mess that "web3" advocates decry.

Hanlon's Razor demands we treat these errors and omissions as sincere, if misguided.

What's less excusable is an appropriation of the term "web" concerning (but not limited to):

Despite forceful assertions that these systems represent the next evolution of "the web", they technically have no connection to it.

This takes doing! The web is vastly capable, and browsers today are in the business of providing access to nearly every significant subsystem of modern commodity computers. If "web3" were truly an evolution of the web, surely there would be some technical linkage... and yet.

Having rejected the foundational protocols of the web, these systems sail a parallel plane, connecting only over "bridges" and "gateways" which, in turn, give those who run the gateways incredible centralised power.

Browsers aren't going to engineer this stuff into the web's liberally licensed core because the cryptocurrency community hasn't done the necessary licensing work. Intricate toil is required to make concrete proposals that might close these gaps and demonstrate competent governance, and some of it is possible. But the community waving the red shirt of "web3" isn't showing up and isn't doing that work.

What this amounts to, then, is web-washing.

The term "web3" is a transparent attempt to associate technologies diametrically opposed to the web with its success; an effort to launder the reputation of systems that have most effectively served as vehicles for money laundering, fraud, and the acceleration of ransomware using the good name of a system that I help maintain.

Perhaps this play to appropriate the value of the web is what it smells like: a desperate move by bag-holders to lure in a new tranche of suckers, allowing them to clear speculative positions. Or perhaps it's honest confusion. Technically speaking, whatever it is, it isn't the web or any iteration of it.

The worst versions of this stuff use profligate, world-burning designs that represent a threat to the species. There's work happening in some communities to address those challenges, and that's good (if overdue). Even so, if every technology jockeying for a spot under the "web3" banner evolves beyond proof-of-work blockchains, these systems will still not be part of the web because they were designed not to be.

That could change. Durable links could be forged, but I see little work in that direction today. For instance, systems like IPFS could be made to host Web Packages which would (at least for public content) create a web-centric reason to integrate the protocol into browsers. Until that sort of work is done, folks using the "web3" coinage unironically are either grifters or dupes. Have pity, but don't let this nonsense slide.

"web3" ain't the web, and the VCs talking their own book don't get the last word, no matter how much dirty money they throw at it.

Older Posts

Newer Posts