Infrequently Noted

Alex Russell on browsers, standards, and the process of progress.

CRM 114 on OS X

A quick note to my future self on getting CRM 114 to build and install on OS X.

First, download the latest tarball to a suitable location (/tmp will do). Explode the tarball and cd into the TRE library directory inside of it, currently tre-0.7.4. Next, run:

sudo ./configure --enable-static && make && make install

Once TRE is installed, run man agrep and marvel at the wonder that is agrep. Holy crap is that cool.

Next, edit the main CRM 114 Makefile. Comment out the line in the that reads:

LDFLAGS += -static

On OS X, dynamic library lookup is preferred and I wasn't able to get static linking working anyway. Next, uncomment these lines:

CFLAGS += -I/usr/local/include
LDFLAGS += -L/usr/local/lib

But make sure that this line is still commented out:

#LIBS += -litnl -liconv

Otherwise you'll be on a wild goose chase to find a package that includes a dynamic library for GNU gettext. Luckily, the entropy.ch php packages have such a beast, but to avoid more build path mucking than is absolutely necessaray, just make sure that -lintl isn't in your GCC calls.

The last change is to modify the line that reads:

-lm -ltre -o crm114_tre

to omit the "-lm" flag. It should then read simply:

-ltre -o crm114_tre

At this point, it's safe to build with:

sudo make clean && make && make install

Huzzah!

Update: A couple of final snags, aside from the various setup bits and bobs that aren't automated). In order to actually process my spam/ham folders, it was necessary to patch crm114_config.h and rebuild. The substitution was:

//   default size of the data window: 8 megabytes.
// #define DEFAULT_DATA_WINDOW  8388608
#define DEFAULT_DATA_WINDOW 16777216

which ups the processing window for messages significantly. Also, it was necessaray, as per the comment in the file, to split up the first line of mailreaver.crm into 2 lines, like this:

#!/usr/bin/crm
#    -( spam good cache dontstore stats_only outbound undo verbose maxprio minprio delprio)

Reason #39 That Nothing Gets Written For Phones (Yet)

Moore's Law can imply a lot of different things depending on what you place as your primary priority. If it's raw compute power with no concern for power use or heat dissipation, you get huge boots every couple of years. If you're after more compute on the same power budget, just wait a while. Predicting what we'll be using in a couple of years is a related-rates problem where the backstop conditions are "any trend that can't continue won't" and whatever market forces drive the non-CPU constraints. Datacenters hit a wall in terms of power use and heat dissipation, and the result is that Intel, AMD, and Sun now are plastering the Financial District with competing billboards touting their FLOPS-per-watt in lieu of GHz ratings.

I suspect that we're in for some similar reckonings in the mobile device world, but the constraints are very different from what the PC world has been up against. For one, the overall power density in phone/PDA batteries is increasing, but only at about something between 9 and 18 percent a year depending on who you read. Competing for this meager power budget are:

So the assumption that we'll get more raw CPU power for PIM-style apps and web browsing in next year's model is related as much to the design goals of the industry as it is to any particular technology trend. But the big barrier for the use of the mobile web is, and has been, the UI. And I don't just mean the layout of mobile browsers or the way pages get formatted. No, I mean the way people actually punch alphanumeric characters into these things. The available options all revolve around form factor. Clamshell == numberpad + 3tap. PDA-ish == thumb-board, widscreen == on-screen keyboard. All of these options put at least one set of characters at a tremendous speed dis-advantage. The biggest common (fast) input element is some sort of directional input system with a center-click button. It seems like a minor quibble in the pantheon of problems with getting the web onto phones in a meaningful way, but I think it creates a real problem for the "flatness" of the internet. In the same way that there's huge pressure to get a short domain name for startups, I can easily envision a scenario where there's a bidding war to get in the top page of links on the carrier's (non-resettable?) mobile browser homepage. And carriers don't leave money on the table. They hate open markets almost as much as they hate you.

So where does that put us? If there's good news, it's that eventually carriers will learn to sell bandwidth at a reasonable price ('cause that's how they'll keep customers) and beef up their minimum-spec processor and browser requirements, but I don't think it'll happen for a couple of years. First, the UI issues are going to require something like ubiquitous thumbpads or something more drastic (chording?) to turn these devices into anything but low-end digital cameras with super-slow upload facilities. Secondly, screen density improvements and the attendant bandwidth chewing properties of bitmap graphics are likely to keep CPU/SOC spare power in check until most phones sprout dedicated graphics co-processors (and if they already have, then they're just competing for power). Once screen density, contrast, color depth, and touch sensitivity start to settle down, we'll probably be left with a sudden improvement in processing budget, but it's unclear how long that'll get soaked up by piggier OSes before trickling down to applications.

The last hurdle for making the mobile web a reality is bandwidth and latency. Luckily, bandwidth will probably get figured out. Latency, OTOH, won't. We'll just have to learn to with that one. Handset turnover rates combined with improvements in carrier networks give me some hope that by the time we have realistic browsers on phones we'll be sucking down content over relatively wide pipes. Of course, when the carriers have the "a ha!" moment around fixed-price mobile data is the big unknown. There's more money to be made if they give up a little control, but that goes against everything they know and believe.

Guess I should wrap this up, but next time, I'll detail my experience with getting smartphone emulators installed, the SNAFU that started me down this entire train of thought in the first place.

Something's Always Wrong

Like a sled dog who would gladly run himself to death, I seem not to be able to kick the habit of over-working, especially not on things that I find interesting. If my interests and efforts were mostly aligned at Jot, they're now identical vectors at SitePen. My full-time job is now what I once did as a hobby, and my hobbies are all the interesting little technical pursuits I've put off in the last couple of years. Add a large dose of Lutheran guilt about doing the right things the right way and a seeming inability to say "no" to people, and sleep seems like a particularly quaint anachronism.

Were it not for Jennifer, the woman I love madly, I think I'd be clinically insane by now. I'm good at last-minute panic. Things like "did we turn off the stove?" and "maybe I should double-check when that flight is leaving...". Jennifer, on the the other hand, is the organizer. Instead of leaving important details for the last minute, she ensures that big things happen at all. Months in advance, she'll be asking "so [insert name of band] is coming, do you want to see them?", and thanks to that (and a little "where did we put the tickets?" from me) we've been able to see Buddy Guy, BB King, and Toad The Wet Sprocket this summer.

Toad was a special treat since they've been broken up for years. Despite it carbon-dating my highschool years, I've got something of a soft spot in my iTunes playlist for old Toad stuff. Even new "Ok Go" doesn't make me as happy cycling through my playlist during these endless hours at the terminal. While their set the other night was pretty heavily scripted, they still sounded as good as any of the studio recordings. Thanks to Jennifer, I got out of the house for some celebration after turning my brain to mush in preparation for a talk at LinuxWorld and I finally got to see a band I've been listening to heavily for nearly a decade.

No matter what you may think of the music, it's hard to imagine a better gift.

Sweet, Blessed Bandwidth

For a while now there have been flyers in the elevator of the building where we live advertising network service through the 10baseT network that was presciently added to the original design of the place. It helps that the building is fairly new, having been constructed in the wake of the '89 quake and the destruction of the Emabarcadero Freeway. At some point in the late 90's internet service had been provided to every unit via a shared T1 line, but that died about the same time that pets.com's sock puppet hung it up, or went back into the drawer, or wherever it is that sock puppets go in the afterlife. Thankfully everything old is new again and the service has been resumed by a new company, albiet with a fatter pipe (T3).

Contrasting VVD's installation process with SBC's is a study in how a small company can beat the living crap out of entrenched players. Of course, the local phone and cable monopolies don't usually tend to sent an exceedingly high bar for customer service, but when you can select (online) what time you'd like the install (to within the hour) and the guy shows up on time, the phone company should start to worry. A lot.

When I lived in Madison, I happily gave my business to TDS instead of the local telco monopoly, and I was always impressed with their service and general geek-friendlyness. It seems that VVD is a bandwidth provider in the same mold. After the initial "install" (lighting up an ethernet port) I found myself staring down some weird speed issues. In my haste to configure the Airport for the new service instead of the previous PPPoE setup, I neglected to reset the ethernet duplex settings. Halarity ensued while I tried to figure out why the OS X boxes (which don't support Selective ACK) saw roughtly 1/10 the upload bandwidth of the linux boxen (which do). Some dinner and a whack-the-forehead moment later and we've now got a network connection that's faster than anything I've used on a daily basis since college.

A better, synchronous connection and the ability to stop paying monopoly rent for bandwidth...now that's what I call progress.

Cometd, Bayeux, and Why They're Different

Ajax has been stupendously successful in capturing the imaginations of webdevs in part because of the backlog of demand for better interactivity in browser-based apps, but also because it's stone-simple to implement. One of the biggest problems facing the adoption of Comet is that it's, by definition, not as simple. It's usually not possible to take ye-old-RESTian HTTP endpoint and suddenly imbue your app with realtime event delivery using it unless you want your servers to fall over. The thread and process pooling models common to most web serving environments usually guarantees this will be true. Add to that the complexity of figuring out what browsers will support what kinds of janky hacks to make event delivery work and you've got a recipe for abysmal adoption. That complexity is why we started work on Bayeux.

Bayeux is a JSON-based protocol for clients to register interest in events and for servers to deliver them in a more timely fashion than Ajax-based polling allows. The goals of the Bayeux spec so far have been to:

As a result we've variously rejected plans that involved implementing some subset of XMPP in JSON or actually tunneling XMPP over some other carrier. JSON libraries are available almost everywhere and you can't beat the speed of eval for message deserialization on the client. It keeps things fast, simple, and pluggable. Authentication and authorization, while being given spots in the protocol, have also been left entirely to implementations to negotiate. The result has been that implementations have gotten off the ground very fast. Since the protocol doesn't specify any guarantees around durability, entirely ephemeral event busses can be as conformant as MOM-style guaranteed-delivery systems.

So what, then is Cometd and how does it relate to this Bayeux specification? The short answer is that Cometd is a project to provide several reference implementations of Bayeux and to evolve the spec until it's good enough for some other group to take over maintenance. Bayeux and Cometd are two complimentary parts of a plan to tackle the complexity around building and deploying comet apps, and we're hoping for many independent Bayeux implementations outside of the Cometd project. For more interactive apps to finally take off, we need a set of portable concepts about how this kind of message passing should be handled in much the same ways that Ajax could be thought of as "RESTian web services for browsers". Today lots of folks are thinking about event busses under the buzzword banner of SOA, but hopefully in the same way that REST was the lighter but easier variant of SOAP for doing request-response web services, Bayeux can be the less-sophisticated (but easier to use) alternative to exotic schemes for connecting MQSeries and TIBCO to browsers.

Older Posts

Newer Posts