Comments for The Non-Relational DB Strikes Back!
And my personal opinion is that Zope partly failed due to this, the same issue Smalltalk always had :-) People just do not want to put their data (and their code!) in some binary blob.
Kris: Perservere is really pretty close, but doesn't have a query language. Pure orthogonal persistence gets some some of the way there, but you need more than that to really make apps sing.
Helge: ZODB suffered, primarily, from difficulty integrating (and a weird build process that hampered adoption). I was also brutally slow and required "rebuilding" pretty frequently. I had great hopes for it at one point, but it appears to be an artifact of history now.
I also don't know what you mean by it being "brutally slow". In what context?
Finally I don't know at all by what you mean by rebuilding it frequently. Do you mean packing it to remove older transactions? The equivalent operation, I understand, needs to be done by the typical RDB as well to clean up space on the filesystem.
That's not to say the ZODB doesn't have its problems, I just have a hard time actually recognizing the problems you describe. One of the problems is indeed one described by Helge: often people don't want to commit their whole "world" into a single database (especially not code, though that hasn't been the primary use of the ZODB for a long time). And having a powerful query language like SQL around is sometimes nice.
One difference between the ZODB and something like CouchDB is that CouchDB focuses on usage over the network, out-of-process. The ZODB is in-process (though one can spread it over multiple processes using ZEO). Both approaches have separate advantages and drawbacks.
"when your data and your program can evolve in harmony and without friction or risk, you are truly liberated"
sounds like an OODB to me.
I read your post with great interest. My company, AppJet (appjet.com), has a hosted server-side JavaScript framework for building web apps from your browser. We share your philosophy about storage frameworks; sometimes people ask us why we didn't just use RoR and/or SQL, and the reasons in your post are why.
Our persistent storage system is based on JavaScript objects -- you can persist an arbitrary graph of JavaScript objects that you create to be "storable". You can also have "StorableCollections" of objects, which support filtered and sorted views.
We just launched a few days ago to let people start playing around with our system, so things are pretty basic, but we'd be interested in your feedback. We want AppJet to be a dream environment for quickly putting web apps together, and we're willing to write our own software (framework, storage system, execution engine) when we feel existing components won't cut it.
-- David
However, I do think ZODB is closer to your description of world dominating non-relational data stores than may be apparent at first blush. From what I remember, and it was way way back around the turn of the century in fact, ZODB was plenty fast and I didn't have to rebuild it on a daily basis. You have to purge old transactions once in a while but that could be automated.
Even in record serialization to XML, you'd still need to "compact" the store once in a while, or at least one should look into it. ZODB just provides that out of the box.
I'm not advocating ZODB, but I think it may be more advanced than the process you describe here. Something closer to the "jot" process may be the old "pickling" paradigm that Pythons of yore used to do. Not sure if anyone still uses pickling, but that was the object serialization that they also used about half a decade ago.
So I really really wanted ZODB to be "the one" but when I had last looked at it deeply it just wasn't. When I last used it there was some serious difficulty in integrating it non-Zope environments and ensuring that it's performance was acceptable. I gave up on it after following the project for nearly a year. It seems to have moved forward significantly in the interim (thanks for the poke on that Martjin). I will look into it more deeply.
I like Python a lot (although the lack of fixed lambdas and reasonable closure syntax has really soured me in the last couple of years). Hopefully someone can convince Guido that these kinds of features deserve some support deeper in the system.
Regards
One more point on Jot: I'm not free to discuss it's serialization strategy at more length than the old API documentation (http://developer.jot.com/WikiHome/DevDocToc) used to describe and to say that the system exposed serialization through endpoints to multiple formats. The platform never exposed compacting or any other type of cleanup or indexing to users. It's hard to see how ZODB has evolved in this direction since it's wiki appears to be down. It's not clear where the evolutions which had at one point been slated for ZODB4 ever went to.
Lastly, the query language turns out to be important. Jot used a modified version of XPath with the ability to easily call into JavaScript functions to perform higher-order matching, and this worked very well. I don't like XPath per sae, but having a compact way to say "get me stuff that matches this rule from here" is useful. Since the data's hierarchy usually represents the relationships between things in the system, there wasn't much of the ad-hoc tree walking stuff that ZODB kinda backed you into (at least the last time I spent serious time with it, which was years ago).
Regards
Since you left Jot, I developed several serious apps using their API, including a mini-ERP. (Finally got reasonably good at Dojo on that last one...)
It ain't perfect, but I still haven't seen anything better. The query features of the api are good, but the full integrated server-side Javascript means you can do almost anything.
And having a very nice Wiki surrounding your business app makes it even better.
I'm hoping it all comes back to life again. But if you ever see anything open-source that comes close, please post something on your blog.
Alex Russell recently wrote about the advantages of non-relational (dynamic object) databases for application development. The flexibility of using dynamic persisted objects that behave like dynamic OOP objects provides a great level of agi...