Infrequently Noted

Alex Russell on browsers, standards, and the process of progress.

Hobson's Browser

How Apple, Facebook, and Google Broke the Mobile Browser Market by Silently Undermining User Choice

Update: This post was turned into a talk for State of The Browser in October 2021; you can watch the recording here.

Update, The Second: Welp, I was wrong. I assumed that Facebook PMs and engineers were smart. Of course they were going to get found out modifying content via In-App Browsers, just as this post warned they could. It's long past time for Google and Apple to act to curb this abuse via App Store policy, and regulators interested in gatekeeper shenanigans should take notice.

The post has also been updated for readability and to more clearly outline the previously-embedded per-page opt-out proposal.


At first glance, the market for mobile browsers looks roughly functional. The 85% global-share OS (Android) has historically facilitated browser choice and diversity in browser engines. Engine diversity is essential, as it is the mechanism that causes competition to deliver better performance, capability, privacy, security, and user controls. More on that when we get to iOS.

Tech pundits and policymakers are generally older and wealthier than the median user and likely formed expectations of browsers on the desktop. They may, therefore, tend to think about mobile browser competition through the lends of desktop browsing. To recap:

Each point highlights a different aspect of ecosystem health. Together, these properties show how functioning markets work: clear and meaningful user choice creates competitive pressure that improves products over time. Users select higher quality products in the dimensions they care about most, driving progress.

The mobile ecosystem appears to retain these properties, but the resemblance is only skin deep. Understanding how mobile OSes undermine browser choice requires an understanding of OS and browser technology. It's no wonder that few commenters are connecting the dots.[2]

How bad is the situation? It may surprise you to learn that until late last year only Safari could be default browser on iOS. It may further disorient to know that competitors are still prevented from using their own browser engines.

Meanwhile, the #2 and #3 sources of web traffic on Android — Google's search box and Facebook's native apps — do not respect browser choice. Users can have any browser with any engine they like, but it's unlikely to be used. The Play Store is little more than a Potemkin Village of browser choice; a vibrant facade to hide the rot.

Registering to handle link taps is only half the battle. For a browser to be the user's agent, it must also receive navigations. Google's Search App and Facebook's apps undermine these choices in slightly different ways.[3] This defangs the privacy and security choices made through browsers. Developers suffer higher costs when they cannot escape Google, Facebook, and Apple's walled gardens or effectively reach users through the web.

Web engineers frequently refer to browsers as "User Agents", a nod to their unique role in giving users the final say over how the web is experienced. A silent erosion of browser choice has transferred power away from users, depositing it with dominant platforms and apps. To understand how this sell-out happened under our noses, literally, let's look at how mobile and desktop differ.

The Baseline Scenario

The predominant desktop situation is straightforward:


Browsers handle links, and non-browsers defer loading http and https URLs to the system, which in turn invokes the user's default browser. This flow is what gives links utility. If the players involved (OSes, browsers, or referring apps) violate aspects of the contract, user choice in browsers has less effect.

"What, then, is a 'browser'?" you might ask? I've got a long blog post brewing on this, but jumping to the end, an operable definition is:

A browser is an application that can register with an OS to handle http and https navigations by default.

On Android this is expressed via manifest intent filters and the BROWSABLE category. iOS gained browser support in late 2020 — a dozen years late — via an Entitlement.[4] Windows and other Desktop OSes have similar (if less tidy) mechanisms.

No matter how an OS facilitates browser choice, it's this ability to replace the default handler for links that defines browsers. How often links lead users to their browser defines the meaningfulness of this choice.

Modern browsers like Chrome and Samsung Internet support a long list of features that make web apps powerful and keep users safe. Both pass all eighteen feature tests in Thomas Steiner's excellent 🕵️ PWA Feature Detector

Android's "In-App Browser" Problem(s)

Mobile browsers started in a remarkably resource-constrained environment. First-generation iOS and Android smartphones were slow single-core, memory-impoverished affairs, leading mobile OSes to adopt heuristics for killing background apps to reclaim memory. This helped ensure the whole system remained responsive.

But background task killing created problems for link-heavy apps. Launching the browser placed linking apps in the background and browser UI didn't provide affordances for returning to referring applications. This reduced the probability users would return, hurting engagement.

Being put in the background also increased the likelihood of a linking app being killed.[5] It can take seconds to re-start the original app and restore UI state, an experience that gets worse on low-end devices that are most likely to evict apps in the first place.

To compensate, engagement-thirsty apps began including "In-App Browsers" ("IABs") to keep links from bouncing users to browsers. Contrary to any plain-language understanding of "a browser", these IABs cannot be installed as browsers, even where OSes enabled browser choice. Instead, they load content referred by their hosting native app in system-provided WebViews.

The benefits to apps that adopt WebView-based IABs are numerous:

In the unlikely scenario users are happy for browsers to forget their saved passwords, login state, privacy preferences, extensions, and accessibility settings, this could, in theory, be a win-win. In practice it is a hidden, ecosystem-wide tax.

IABs are hard to spot unless you know exactly what you're looking for, and the controls to disable them are consistently buried in hard-to-find UI. The cumulative result is that tapping links generally feels broken.[7]

1...2...3...Party Time!

WebViews are the source of much confusion in debates around apps and browser choice. Thankfully, the situation is only complicated rather than complex.

There are two dimensions in play:


So, a browser can be WebView-based, and so can an IAB. But neither has to be.

What is a WebView? To quote the Android documentation, a WebView is...:

A View that displays web pages. ... In most cases, we recommend using a standard web browser, like Chrome, to deliver content to the user.

WebViews have a long history in mobile OSes, filling several roles:

The use of WebViews in non-browser apps is appropriate for first and second-party content. Here, apps are either rendering their own web content or the content can be expected to know about the limits imposed by the WebView implementation. Instead of breaking content, WebViews rendering first and second party content can help apps deliver better experiences without additional privacy and security concerns.

All bets are off regarding WebViews and third-party content. Remember, WebViews are not browsers.

WebViews support core features for rendering web content, along with hooks that allow embedders to "light up" permission-based APIs (e.g., webcam access). Making a full browser out of a WebView requires a lot of additional UI and glue code.

Features that need extra care to support include:

Few (if any) WebView browsers implement all of these features, even when underlying system WebViews provide the right hooks.

The situation is even more acute in WebView IABs, where features are often broken even when they appear to be available to developers. Debugging content in IAB franken-browsers is challenging, and web developers are often blind to the volume of traffic they generate, meaning they may not even understand how broken their experiences are.

How can that be? Web developers are accustomed to real browsers and industry standard tools, analytics, and feature dashboards do break out or highlight IABs. The biggest IAB promulgators (Facebook, Pinterest, Snap, etc.) are complicit, investing nothing in clarifying the situation.

Neither users nor developers understand Facebook, Pinterest, or Google Go as browsers. If they did, they would be livid at the poor quality and broken feature set. WebView IABs strip users of choice, and technical limits they impose prevent web developers from any recourse to real browsers.


No documentation is available for third-party web developers from any of the largest WebView IAB makers. This scandalous free-riding is shady, but not surprising. It is, however, all the more egregious for the subtlety and scale of breakage.

Thanks to IAB shenanigans, Facebook is the third largest Android "browser"-maker. If it employs a single developer relations engineer or doc writer to cover these issues, I'm unaware of it. Meanwhile, forums are full of melancholy posts recounting myriad ways these submarine renderers break features that work in other browsers.

Update (Oct '21): How feature-deprived are WebView IABs? Several months after this post was published, and with no apparent irony, Facebook dropped support for WebViews as a login mechanism to Facebook itself. That's right, Facebook's own app is now an unsupported browser for the purposes of logging in to Facebook.

"Facebook Mobile Browser" relies on the system WebView built from the same Chromium revision as the installed copy of Chrome. Despite the common code lineage and exceedingly low cost to Facebook to develop, it fails to support half of the most meaningful PWA features, cutting third-party web developers off at the knees.

WebView IAB makers have been given "the first 80%" of a browser. Development and distribution of critical components is also subsidised by OS vendors. Despite these considerable advantages, WebView IABs universally fail to keep up their end of the bargain.

First-party developers can collaborate with their IAB colleagues to build custom access to any feature they need.

Likewise, second-party developers expect less and their content will not appear to be broken — ads are generally not given broad feature access.

But third-party developers? They are helpless to understand why an otherwise browser-presenting environment is subtly, yet profoundly, broken.

There are still users browsing with a Chrome 37 engine (7 years ago), not because they don't update their browsers but because it's Facebook Mobile Browser on Android 5 using a webview. Facebook does NOT honor user browser choice leaving that user with an old engine. +

Image from Tweet

These same app publishers request (and heavily use) features within real browsers they do not enable for others, even when spotted the bulk of the work. Perhaps browser and platform vendors should consider denying these apps access to capabilities they undermine for others.

WebView IAB User Harms

The consequences of WebView IABs on developers are noteworthy, but it's the impacts on users that inspire confusion and rage.

Consider again the desktop reference scenario:


Clicking links takes users to their browser, assuming they are not already in a browsesr. If a link from an email application points to example.com, previous login state and passwords are not forgotten. Saved addresses and payment information are readily available, speeding up checkout flows. Most importantly, accessibility settings and privacy preferences are consistently applied.

Facebook's IAB features predictably dismal privacy, security, and accessibility settings. Disabling the IAB is <a href='https://twitter.com/slightlylate/status/1167548118876901376' target='_new'>Kafkaesque journey one must embark anew for each Facebook-made app on each device.</a>
Facebook's IAB features predictably dismal privacy, security, and accessibility settings. Disabling the IAB is Kafkaesque journey one must embark anew for each Facebook-made app on each device.

By contrast, WebView IABs fracture state, storing it in silos within each application. This creates a continuous partial amnesia, where privacy settings, accessibility options, passwords, logins, and app state are frequently lost.

The resulting confusion doesn't hurt apps that foist WebView IABs on unsuspecting users and developers. The costs are borne by publishers and users, harming the larger web ecosystem. IABs are, in this understanding, a negative externality.

Does anyone expect anything one does on a website loaded from a link within Facebook, Instagram, or Google Go can be monitored by those apps? That passwords can be collected? That all sites you visit can be tracked?[8]

To be clear, there's no record of these apps using this extraordinary access in overtly hostile ways, but even the unintended side-effects reduce user control over data and security. (August '22) Facebook has been caught red-handed absuing this power to track users within their IAB browser without explicit consent. Sanctions, App Store policies, and opt-out mechanisms are overdue.

The WebView IAB sleight of hand is to act as a browser when users least expect it, but never to cop to the privacy implications of silently undermining user choice.

CCT: A New Hope?


As libraries emerged to facilitate the construction of WebView IABs, OS and browser vendors belated became aware that users were becoming confused and that web publishers were anguished about the way that social media apps broke login state.

To address this challenge, Apple introduced SFSafariViewController ("SFSVC")[9] and Google followed suit with Chrome Custom Tabs protocol ("CCT"). Both systems let native apps to skip the work of building WebView IABs and, instead, provide an OS-wide mechanism for invoking the user's default browser over top of a native app.

Like WebView IABs, CCT and SFSVC address background eviction and lost app state. However, because they invoke the user's actual browser, they also prevent user confusion. They also provide the complete set of features supported by the user's default browser, improving business outcomes for web publishers.

These solutions come at the cost of flexibility for app developers who lose access to snoop on page content, read network traffic, or inject custom behavior. Frustratingly, no OS or App Store mandate their use for IAB needs. More on this shortly.

CCT working as intended from the Twitter native app. Samsung Internet is set as the default browser and loads links within the app. Important developer-facing features work and privacy settings are respected.

Et Tu, Google?

CCT sounds pretty great, huh?

Well, it is. At least in the default configuration. Despite the clunky inclusion of "Chrome" in the name, the CCT library and protocol are browser-agnostic. A well-behaved CCT-invoking-app (e.g., Twitter for Android) will open URLs in the CCT-provided IAB-alike UI via Firefox, Brave, Samsung Internet, Edge, or Chrome if they are the system default browser.

That is unless the native app overrides the default behaviour and invokes a specific browser.

@slightlylate I recently was talking to my Dad about the Web and asked what browser he uses and he showed me what he does:
He searches for the Web site in the Google search widget and then just uses the results page Chrome tab as his entire browser.
His default browser is not set to Chrome.

Who would do this, you might ask? None other than Google's own Search App; you know the one that comes on every reputable Android device via the ubiquitous home screen search widget.

AGSA's homescreen widget; the text box that launched two billion phones. Links followed from search results always load in Chrome via CCT, regardless which browser users have set as default.
AGSA's homescreen widget; the text box that launched two billion phones. Links followed from search results always load in Chrome via CCT, regardless which browser users have set as default.

Known as the "Android Google Search App" ("AGSA", or "AGA"), this humble text input is the source of a truly shocking amount of web traffic; traffic that all goes to Chrome, no matter the user's choice of browser.

Early on, there were justifiable reasons to hard-code Chrome. Before support for CCT was widespread, some browsers exhibited showstopper bugs.

Fast-forward to 2021 and those bugs are long gone, but the hard-coding persists. Today, the primary effect is to distort the market for browsers and undermine user choice. This subverts privacy and makes it hard for alternative browsers to compete on a level playing field.

This is admittedly better than the wholesale neutering of important features by WebView IABs, but when users change browsers, continuious partial amnesia on the web gets worse. A Hobson's Choice of browser.

<b>'Powered By Chrome'</b>: Google's Search App disregarding browser choice on a system with Samsung Internet set as the default browser.
'Powered By Chrome': Google's Search App disregarding browser choice on a system with Samsung Internet set as the default browser.

WebLayer: New Frontiers In User Confusion

Google can (and should) revert to CCT's default behavior which respects user choice. Since AGSA uses CCT to load web pages rather than a WebView, this would be a nearly trivial code change. CCT's core design is sound and has enormous potential if made mandatory in place of WebView IABs. The Android and Play teams could mandate better behavior in IABs to improve user privacy.

There's reason to worry that this is unlikely.

Instead of addressing frequent developer requests for features in the CCT library, the Chrome team has invested heavily in the "WebLayer" project. You can think of WebLayer like a WebView-with-batteries-included, repairing issues related to missing features but continuing to fracture state and user choice.

There is a weakly positive case for WebLayer. For folks making browsers, WebLayer dramatically reduces the amount of custom glue code needed to light up adavanced features. In the context of IABs, however, WebLayer looks set to entrench user-hostile patterns even further.

Subversion of choice is a dispiriting trend in search apps. Stealing traffic without any effort to honestly earn a spot as the user's preferred browser is, at best, uncouth and adopting WebLayer will not meaningfully improve the user experience or privacy of these amnesiac browsing experiences.

Google Go, the Google app for iOS, and Microsoft's Bing app for Android all capture outbound links in WebView IABs, subverting browser choice and rubbishing features for developers. If there's any mercy, it's that their low use limits the impact on the ecosystem.

Google Go's WebView IAB is just as broken as Facebook's. As the default Search app on Android Go devices, it creates new challenges for the web in emerging markets.

Google and Apple could prevent this bad behavior through App Store policies and technical changes. They have the chance to lead, to show they aren't user-hostile, and remove a permission structure for lousy behaviour that less scrupulous players exploit. More on that in a moment.

iOS's Outsized, Malign Influence

Imagine if automakers could only use one government-mandated engine model across all cars and trucks.

Different tires and upholstery only go so far. With the wrong engine, many jobs cannot be done, rendering whole classes of vehicles pointless. If the mandated engine were particularly polluting, choosing a different model would have little effect on emissions.

That's the situation iOS creates regarding browsers today. The only recourse it to buy a phone running a different OS.

iOS matters because wealthy users carry iPhones. It's really as simple as that. Even when Apple's products fail to gain a numerical majority of users in a market, the margin contribution of iOS users can dominate all other business considerations.

Bosses, board memebers, and tech reviewers all live in the iOS ecosystem. Because Apple prevents better web engines anywhere on its platform, browser choice is hollow.

Apple has deigned to allow "browsers" in its App Store since 2012. Those apps could not be browsers in a meaningful sense because they could not replace Safari as the default handler for http/https links.

The decade-long charade of choice without effect finally ended with the release of iOS 14.2 in late 2020, bringing iOS into line with every other significant OS in supporting alternative browsers.[10]

But Apple has taken care to ensure that choice is only skin deep. Browsers on Windows, Linux, ChromeOS, Android, and MacOS can be Integrated Browsers, including their own competing engines. iOS, meanwhile, restricts browsers to shells over the system-provided WebView.

Unlike WebView browsers on other OSes, Apple locks down these components in ways that prevent competition in additional areas, including restrictions on network stacks that block improved performance, new protocols, or increased privacy. These restrictions make some sense in the context of WebView IABs, but extending them to browsers only serves to deflect pressure from Apple to improve their browser.

Perhaps it would be reasonable for iOS to foreclose competition from integrated browsers if it also kept other native apps from accessing powerful features. Such policies would represent a different view of what computing should be. However, Apple is happy to provide a wide variety of scary features to unsafe native applications, so long as they comply with the coercive terms of its App Store.

Powerful browsers present a threat to the fundamentals of Apple and Google's whale-based, dopamine fueled, "casual" gaming monetisation rackets.

Unlike other native apps, browsers are principally concerned with user safety. A safe-by-default, capable platform with low-friction discovery could obviate the root justification for app stores: that they keep over-powered native apps in check.

Apple forestalls this threat by keeping the web on iOS from feature parity. Outlawing true browser choice leaves only Apple's own, farcially under-powered, Safari/WebKit browser/engine...and there's precious little that other WebView browsers can do to improve the situation at a deep level.[11]

Web developers are understandably livid:

Seeing a Web App I worked on used by *Apple* to justify that the Web is a viable platform on iOS is bullshit

The Web can be an ideal place to build apps but Apple is consistently dragging their heals on implementing the Web APIs that would allow them to compete with native apps twitter.com/stopsatgreen/status/1389593307219701760


In addition, by refusing to let any other Web browser engines run on iOS. They are preventing any other browser filling in the feature gap. Truly holding back Web Apps on iOS.


I have defended Apple's choice to restrict web browsers on their platform before and I still do but they can't have their cake and eat it to.

They should not hold back Web Apps with one hand and then turn around and say that Web Apps can compete with native apps.

Developer anger only hints at the underlying structural rot. 25+ years of real browser competition has driven waves of improvements in security, capability, and performance. Competition has been so effective that browsers now represent most computing time on OSes with meaningful browser choice.

Hollowing out choice while starving Safari and WebKit of resources managed to put the genie back in the bottle. Privacy, security, performance, and feature evolution all suffer when the competition is less vibrant — and that's how Apple likes it.

Mark(et)d Impacts

A vexing issue for commentators regarding Apple's behaviour in this area is that of "market definition". What observers should understand is that, in the market for browsers, the costs that a browser vendor can inflict on web developers extend far beyond the market penetration for their specific product.

A typical (but misleading) shorthand for this is "standards conformanc". While Apple's engine falls woefully short on support for basic standards, that isn't even the beginning of the negative impacts.[12] Because the web is an open, interoperable platform, web developers build sites to reach the vast majority of browsers from a single codebase.

When browsers with more than ~10% share fail to add a feature or exhibit nasty bugs, developers must spend more to work around these limitations. When important APIs go missing, entire classes of content may simply be viewed as unworkable.

The cost of these capability gaps is steep. When the web cannot deliver experiences that native apps can (a very long list), businesses must build entirely different apps using Apple's proprietary tools. These apps, not coincidentally, can only be distributed via Apple's high-tax App Store.

A lack of meaningful choice in browsers leads directly to higher costs for users and developers across the mobile ecosystem even for folks that don't use Apple's products. Apple's norm-eroding policies have created a permission struture for bad actors like Facebook. Apple's leadership in the race to the bottom has inspired a burgeoning field of fast-followers.

Browser choice is not unrelated to other objectionable App Store policies. Keeping the web from competing is part and parcel of an architecture of control that tilts commerce into coercive, centralising stores, even though safer, browser-based alternatives would otherwise be possible.

Small Changes to Restore Choice

Here's a quick summary of the systems and variations we've seen thus far, as well as their impacts on user choice:

System Respects Choice Notes
Integrated Browsers Yes Maximizes impact of choice
WebView Browsers Yes Reduces diversity in engines; problematic when the only option (iOS).
WebView IABs No Undermines user choice, reduces engine diversity, and directly harms developers through lower monetisation and feature availability (e.g., Facebook, Google Go).
Chrome Custom Tabs (CCT) Partial WebView IABs replacement, preserves choice by default (e.g. Twitter). Problematic when configured to ignore user preferences (e.g. AGA).
WebLayer No Like WebView with better feature support. Beneficial when used in place of WebViews for browsers. Problematic when used as a replacement for WebView IABs.
SFSafariViewController Partial Similar to CCT in spirit, but fails to support multiple browsers.

Proposals to repair the situation must centre on the effectiveness of browser choice.

Some policymakers have suggested browser choice ballots, but these will not be effective if user choice is undermined no matter which browser they choose. Interventions that encourage brand-level choice cannot have a positive effect until the deeper positive impacts of choice are assured.

Thankfully, repairing the integrity of browser choice in the mobile ecosystem can be accomplished with relatively small interventions. We only need to ensure that integrated browsers are universally available and that when third-party content is displayed, user choice of browser is respected.

Android

Repairing the IAB situation will likely require multiple steps, given the extreme delay in new Android OS revisions gaining a foothold in the market. Thankfully, many fixes don't need OS updates:

Future releases of Android should bolster these improvements by creating system-wide opt-out of WebView and WebLayer IABs.

Play policy enforcement of rules regarding CCT, WebView, and WebLayer respect for user and developer choice will also be necessary. Such enforcement is not challenging for Google, given its existing binary analysis infrastructure.

Together, these small changes can redress the worst anti-web, anti-user, anti-developer, and anti-choice behaviour of Google and Facebook regarding Android browsers, putting users back in control of their data and privacy along the way.

iOS

iOS begins from a more troubling baseline but with somewhat better IAB policies. What's undermining user choice there require deeper, OS-level fixes, including:

Allowing integrated browsers will require updates to Apple's App Store policies to clarify that alternative engines are permitted in the context of com.apple.developer.web-browser entitled applications.

Don't WebView Me Bro!

The proposal for a header to allow sites to demand CCT/SFSVC instead of a WebView IAB may seem complex, but it is technically straightforward and can be implemented very quickly.

Websites would include a tag (or the equivalent HTTP header) in top-level pages like this:

  <meta http-equiv="Content-Security-Policy"
content="frame-ancestors 'system-default'">

OS vendors would update their system WebViews to respect this tag and invoke CCT if encountered in a top-level document. This is compatible with the existing ecosystem, as no first-party content (help pages) or second-party integration (ad network) would send these headers, existing apps would not need to be updated. Websites could incrementally add the hint and benefit from the new behavior.

Android's WebView component auto-updates with Chrome, ensuring huge reach for such a fix in a short time. iOS updates are fused to OS upgrades, but iOS users tend to upgrade quickly. The net effect is that we should expect such a policy to begin to have a large, positive effect in less than 6 months.

What about apps that try to subvert the default behavior? App store policies can be easily formulated to punish this sort of poor behavior. There's a great deal of evidence that these policies work, at least for the "head" of an app catalog, and would surely condition Facebook's behavior.

For Markets To Work, Choice Must Matter

The mobile web is a pale shadow of its potential because the vehicle of progress that has delivered consistent gains for two decades has silently been eroded to benefit native app platforms and developers. These attacks on the commons have at their core a shared disrespect for the sanctity of user choice, substituting the agenda of app and OS developers for mediation by a user's champion.

This power inversion has been as corrosive as it has been silent, but it is not too late. OSes and app developers that wish to take responsibility can start today to repair their own rotten, choice-undermining behaviour and put users back in control of their browsing, their data, and their digital lives.

The ball's in your court, platforms.

Deepest thanks to Eric Lawrence and Kevin Marks for their thoughtful feedback on drafts of this post.


  1. Windows 10, for example includes several features (taskbar search box, lock screen links) that disrespect a user's choice of default browser. This sort of shortcut-taking has a long and discouraging history, but until relatively recently was viewed as out of bounds. Mobile has shifted the Overton Window.

    A decade of norm degradation by mobile OSes has made these tie-ins less exceptional, creating a permission structure for bad behaviour. The Hobbesian logic of might-makes-right is escalatory and hermetic; a bad act in one turn ensures two in the next. Dark patterns and choice subversion also work against differentation through quality. Why bother when you can just tweak the rules of the game?

    Fixing mobile won't be sufficient to unwind desktop's dark patterns, but that's no reason to delay. Giving users real choice on their most personal devices will help to reset expectations of Sillicon Valley PMs and managers. They were clever enough to read the rules when it allowed cheating, and they'll cotton on quickly once it doesn't, assuming the consequences are severe enough. ↩︎

  2. It's unclear why Mozilla is MIA. Why is it not making noise about the situation? Mozilla has had a front-row seat to the downsides of degraded user choice; not being able to bring Gecko to iOS directly harms the competitiveness of Firefox, and the link-stealing behaviour of Facebook, Google Search, and iOS's defaults policy (until late 2020) materially harmed Mozilla's products. So why are they silent?

    It seems plausible that the Firefox OS experience has so thoroughly burned management that they feel work on mobile is not worth the risk, even if constrained to jawboning or blog posts.

    If any organisation can credibly, independently connect the dots, it should be the Mozilla Foundation. One hopes they do. ↩︎

  3. The history, competitive pressures, and norms of Android app developers caused many smaller apps to capture clicks (and user data), failing to send navigations onward.

    A shortlist of top apps that do so would include:

    • Facebook Messenger
    • Instagram
    • Pinterest
    • Snapchat
    • Microsoft Bing Search

    Some apps that previously abused WebViews for IABs in the pre-CCT era did better, though, switching to CCT when it became available &emdash; notably Twitter. ↩︎

  4. Defining "a browser" as an application that can be set by a user to handle links by default may sit uncomfortably with some folks, as it means that the "browsers" that were avialable in the iOS App store (including a "Chrome" branded product from 2012) didn't count. This is the definition working as intended.

    Even ignoring Apple's ongoing anti-competitive and anti-web behaviour regarding engine choice, the presence of web browsing apps that couldn't be installed as the default wasn't a meaningful choice. Potempkin villiages of browser fronts served Apple well, but didn't do much to aid users, developers, or the web ecosystem.

    Indeed, not all applications that can load web pages are browsers. Only apps that can become the user's agent are. Being the user's agent means being able to reliable assist users, sand off the harmful aspects of sites, and assist users in getting jobs done with data previously entrusted to it.

    Without the ability to catch all navigations sent to the OS, users who downloaded these programs suffered frequent amnesia. User preferences were only respected if users started browsing from within a specific app. Incidental navigations, however, were subject to Apple's monopoly on link handling and whatever choices Safari projected. This is is anti-user, anti-developer, and anti-competitive. ↩︎

  5. Problems related to background task killing are avoided by building a web apps instead of a native apps, as browsers themselves tend not to get backgrounded as often.

    Developers tried this path for a while but quickly found themselves at an impossible feature disadvantage. Lack of Push Notifications alone proved to be a business-defining disadvantage, and Apple's App Store policies explicitly forbid web apps in their store.

    To be discovered, and to retain access business-critical features, mobile platforms forced all serious developers into app stores. A strong insinuation that things would not go well for them in app stores if they used web technologies (via private channels, naturally) reliably accompanied this Sophie's choice.

    iOS and Android played these games in mobile's early days to dig a moat of exclusive apps. Exclusives create friction in the form of switching costs. Nobody wants a devices that doesn't "do" all the stuff their current devices can.

    Platform owners also know the cost of re-developing proprietary apps for each OS creates an investment cliff. When independent software vendors invest heavily in their proprietary systems, it becomes less likely they can deliver quality experiences on their competitor's system, particularly if the code can't be shared.

    App developers only have so many hours in the day, and it costs enormous amounts, both initially and in an ongoing way, to re-build features for each additional platform. The web is a portable platform, and portability is a bug that duopolists want to squash, as it randomises their game of divide-and-counquer.

    Apples's combination of browser engine neglect, feature gap maintainence, and app store policies against web participation — explicit and implied — proved effective.

    In time, rent extraction from a very narrow class of social games and the users addicted to them grew into a multi-billion dollar gambling business that the duopolists have no intention of allowing the web to disrupt.

    Android and iOS may not have been intentional attacks on open computing, but their current form makes them a threat to its future. Regulators will need to act decisively to restore true browser choice so that the web can contest application portability for mobile OSes the way it has transformed desktop investments. ↩︎

  6. Lots of folks have covered the harms social media firms caused by the relentless pursuit of "north star" metrics. There's little I can add.

    I can, however, confirm that some uncharitable takes are directionally correct. You can't cannot engage with engineers and PMs from these organisations for a years without learning their team's values.

    "Make number go up"-OKRs have absolutely created a set of ecosystem castastrophes because these firms (and FB in particular) do not measure or manage for ecosystem health.

    Change is possible, but it will not come from within.

    Browser choice might not seem high up on the long list of anti-competitive ills of modern tech products, but the current "regulatory moment" is a chance to put in structural fixes. It would be a missed opportunity not to put users back in control of their digital lives and attenuate unfettered data collection while the iron is hot. Real browser choice is a predicate for different futures, so we have to guard it zealously. ↩︎

  7. Social apps strip-mining ecosystems they didn't build while deflecting responsibility?

    Heaven forfend! ↩︎

  8. Facebook engineers have noted that the FB IAB is important in fighting bad behaviour on their social network. We should take these claims at face value.

    Having done so, several further questions present themselves:

    • Why, then, is this system not opt-in? Presumably Facebook can convince a representative subset of users to enable it while preserving browser choice for the vast majority.
    • Why is CCT not invoked for low risk origins?
    • Why is Facebook not publicly attempting to improve CCT and SFSVC in ways that can meets its needs, given they may be required to move to SFSafariViewController for iOS
    • Why is this not a game-over problem for Facebook's desktop website?
    • If it's necessary to keep users within a browser that Facebook owns end-to-end, why not simply allow Facebook's native apps to be browsers.

    Becoming a "real" browser is a simple Android manifest change that would put FB back into line with the norms of the web community, allowing FB's differentiated features to compete for browsing time on the up-and-up. Not doing so suggests they have something to hide.

    The need for more information to protect users may be real, but undermining choice for all is a remedy that, at least with the information that's public thus far, seems very tough to justify. ↩︎

  9. iOS didn't support browser choice at the time of SFSafariViewController's introduction and appeared only to have acquiesced to minimal (and initially broken) browser choice under regulatory duress. It's not surprising that Apple hasn't updated SFSafariViewController to work with other default browsers, but it needs to be fixed.

    Will they? Doubtful, at least not until someone makes serious, sustained noise. Goodness knows there's a lot on the backlog, and they're chronically short-staffed (by choice). ↩︎

  10. Yes, even ChromeOS supports changing the default browser, complete with engine choice! ↩︎

  11. The supine position of browser makers regarding Apple's anti-competitive prohibition on integrated iOS browsers is vexing.

    Perhaps it's Great Power calculations or a myopic focus on desktop, but none of the major browser vendors has publicly challenged these rules or the easily-debunked arguments offered to support them.

    To recap, Apple has argued its anti-competitive policies against integrated browsers are necessary because Just-In-Time (JIT) compilers are unsafe. Like other vendors, Apple mitigates the issues with JITs by creating sandboxed processes to run them in. Today, Apple restricts both the ability to create sandboxed processes, as well as the ability to implement a JIT. For competing browsers to credibly port their engines, they'll need both.

    JITs are central to modern JavaScript engines but are not strictly necessary in integrated browsers. Disallowing non-JITing alternative engines on this basis is nonsensical.

    Commenters parroting Apple's line tend not to understand browser architecture. Any modern browser can suffer attacks against the privileged "parent" process, JIT or not. These "sandbox escapes" are not less likely for the mandated use of WebKit; indeed, by failing to expose APIs for sandboxed process creation, Apple prevents others from bringing stronger protections to users. iOS's security track record, patch velocity, and update latency frankly stink.

    But Apple's right to worry about engine security. iOS is frequently exploited via WebKit, and you'd be warry too if those were your priors. But that doesn't make the restriction coherent or justifiable. Other vendors don't under-invest in the security of their engines the way Apple has, and Apple management surely know this.

    It's backwards to under-invest while simultaneously preventing more secure, more capable browsers that can protect users better. Apple's multi-year delay in shipping Site Isolation should indicate just how unserious these arguments are.

    User security will be meaningfully improved when Apple is forced to allow integrated browser competitions on their OS. Perhaps this can be gated by a policy that requires "Apple-standard or better" patch velocity. Such a policy would not be hard to formulate, and the ability of competing browsers to iterate without full OS updates would meaningfully improve patch rates versus today's OS-update-locked cadence for WebKit.

    Some commenters claim that browsers might begin to provide features that some users deem (without evidence) unnecessary or unsafe if alternative engines were allowed. These claims are doubly misinformed.

    Alternative WebView browsers can already add features, and those features are subject to exactly the sorts of attacks that comments posulate. That is, the "bad" future is actually the status quo that Apple have engineered.

    There's no security or privacy benefit in forcing browser vendors to re-build these features with contorted, one-off tools on top of WebViews. Indeed, bringing integrated engines to iOS would prevent whole classes of security issues that arise from these lightly analysed hacks. This isn't theoretical; extensions built in this way have been a frequent source of bugs in iOS WebView browsers for years.

    Securing a single codebase is easier than analysing a multiplicty of platform-specific variants. Engine choice will improve security, in part, by focusing limited security reviewer and fuzzing time on fewer attack vectors. Of course, a functioning market for browsers will still allow users to pick an under-powered, less secure, slower-updating, feature-light browser, just as they can today; Safari, for example.

    Misdirection about JITs serves to distract from iOS's deeper restrictions that harm security. Capable integrated browsers will need access to a suite of undocumented APIs and capabilities Apple currently reserves to Safari, including the inability to create processes, set tighter sandboxing boundaries, and efficiently decode alternative media formats. Opening these APIs to competing integrated browsers will pave the way to safer, faster, more capable computing for iPhone owners.

    Others have argued on Apple's behalf that if engine competition were allowed, Chromium's (Open Source) Blink engine would become ubiquitous on iOS, depriving the ecosystem of diversity in engines. This argument is seemingly offered with a straight face to defend the very policies that have prevented effective engine diversity to date. Mozilla ported Gecko twice, but was never allowed to bring its benefits to iOS users. In addition to being self-defeating regarding engine choice, this fear also seems to ignore the best available comparison points. Safari is the default browser for MacOS and has maintained a healthy 40-50% share for many years, despite healthy competition from other integrated browsers (Chrome, Firefox, Opera, Edge, etc.). Such an outcome is at least as likely on iOS.

    Sitting under all of these arguments are, I suspect, more salient concerns to Apple's executives to resist increasing RAM in the iPhone's Bill of Materials. In the coerced status quo, Apple can drive device margins by provisioning relatively little in the way of (expensive) RAM components while still supporting multitasking. A vital aspect of this penny-pinching is to maximise sharing of "code pages" between programs. If alternative browsers suddenly began bringing their engines, code page sharing would not be as effective, requiring more RAM in Apple's devices to provide good multitasking experiences. More RAM could help deliver increased safety and choice to users, but would negatively impact Apple's bottom line.

    Undermining user choice in browsers has, in this way, returned significant benefits — to AAPL shareholders, anyway. ↩︎

  12. Browser engineers have an outsized ability in standards bodies to deny new features and designs the ability to become standards in the first place. This leads to a Catch-22 that is easy to spot once you know to look for it, but casual observers are often unacquainted with the way feature development on the web works.

    In a nutshell, features are often shipped by browsers ahead of final, formal web standards process ratification. This isn't to say they're low-quality or that they don't have good specifications and tests, it's just that they aren't standards (yet).

    Specifications are documents that describe the working of a system. Some specifications are ratified by Standards Development Organisations (SDOs) like the World Wide Web Consortium (W3C) or Internet Engineering Task Force (IETF). At the end of a long process, those specifications become "web standards". Thanks to wide implementation and unambiguous IP licensing, standards can increase market confidence and adoption of designs. But no feature begins life as a standard, and for most of it's early years in the market, it will not be a formal, adopted standard. This process can go quickly or slowly, depending on the enthusiasm of other participants in an SDO...and here lies the rub: by not engaging in early, open development, or by raising spurious objections at Working Group formation time, vendors can prevent early designs that solve important problems from ever becoming standards.

    Market testing of designs ("running code" in IETF-speak) is essential for progress, and pejorative claims that a feature in this state is "proprietary" is misleading. Open development and high-quality design work are undertaken with the intent to standardise, not to retain proprietary control. Claims that open, standards-track features in this state are "proprietary" bleeds into active deception when invoked by vendors who aren't proposing alternatives, no participating in the effort.

    Withholding engagement, then claiming that someone else is proceeding unilaterally — when your input would remove the stain — is a rhetorical Möbius strip. ↩︎

Git Worktrees Step-By-Step

Git Worktrees appear to solve a set of challenges I encounter when working on this blog:

  1. Maintenance branches for 11ty and other dependencies come and go with some frequency.
  2. Writing new posts on parallel branches isn't fluid when switching frequently.
  3. If I incidentally mix some build upgrades into a content PR, it can be difficult to extract and re-apply if developed in a single checkout.

Worktrees hold the promise of parallel working branch directories without separate backing checkouts. Tutorials I've found seemed to elide some critical steps, or required deeper Git knowledge than I suspect is common (I certainly didn't have it!).

After squinting at man pages for more time than I'd care to admit and making many mistakes along the way, here is a short recipe for setting up worktrees for a blog repo that, in theory, already exists at github.com/example/workit:

##
# Make a directory to hold a branches, including main
##

$ cd /projects/
$ mkdir workit
$ cd workit
$ pwd
# /projects/workit

##
# Next, make a "bare" checkout into `.bare/`
##

$ git clone --bare git@github.com:example/workit.git .bare
# Cloning into bare repository '.bare'...
# remote: Enumerating objects: 19601, done.
# remote: Counting objects: 100% (1146/1146), done.
# ...

##
# Tell Git that's where the goodies are via a `.git`
# file that points to it
##

$ echo "gitdir: ./.bare" > .git

##
# *Update* (2021-09-18): OPTIONAL
#
# If your repo is going to make use of Git LFS, at
# this point you should stop and edit `.bare/config`
# so that the `[remote "origin"]` section reads as:
#
# [remote "origin"]
# url = git@github.com:example/workit.git
# fetch = +refs/heads/*:refs/remotes/origin/*
#
# This ensures that new worktrees do not attempt to
# re-upload every resource on first push.
##

##
# Now we can use worktrees.
#
# Start by checking out main; will fetch repo history
# and may therefore be slow.
##

$ git worktree add main
# Preparing worktree (checking out 'main')
# ...
# Filtering content: 100% (1226/1226), 331.65 MiB | 1.17 MiB/s, done.
# HEAD is now at e74bc877 do stuff, also things

##
# From here on out, adding new branches will be fast
##

$ git worktree add test
# Preparing worktree (new branch 'test')
# Checking out files: 100% (2216/2216), done.
# HEAD is now at e74bc877 do stuff, also things

##
# Our directory structure should now look like
##

$ ls -la
# total 4
# drwxr-xr-x 1 slightlyoff eng 38 Jul 7 23:11 .
# drwxr-xr-x 1 slightlyoff eng 964 Jul 7 23:04 ..
# drwxr-xr-x 1 slightlyoff eng 144 Jul 7 23:05 .bare
# -rw-r--r-- 1 slightlyoff eng 16 Jul 7 23:05 .git
# drwxr-xr-x 1 slightlyoff eng 340 Jul 7 23:11 main
# drwxr-xr-x 1 slightlyoff eng 340 Jul 7 23:05 test

##
# We can work in `test` and `main` independently now
##

$ cd test
$ cat "yo" > test.txt
$ git add test.txt
$ git commit -m "1, 2, 3..." test.txt
# [test 2e3f30b9] 1, 2, 3...
# 1 file changed, 1 insertion(+)
# create mode 100644 test.txt

$ git push --set-upstream origin test
# ...

Thankfully, commands like git worktree list and git worktree remove are relatively WYSIWYG by comparison to the initial setup.

Perhaps everyone else understands .git file syntax and how it works with --bare checkouts, but I didn't. Hopefully some end-to-end exposition can help drive adoption of this incredibly useful feature.

Older Posts

Newer Posts