Zope: A Crisis of Faith. Coming this March to TNT (TNT! TNT!)

For the first time since spring of 1997, I find myself having a crisis of faith regarding Zope. I’m not sure it remembers where it came from. I’m not sure it knows its own future. It’s suffering from a crazy identity crisis. And there seems to be a core effort to take it in directions that I really wish it wouldn’t go. Is it a good tool? Oh hell yeah. Ever since I wrote my first serious Bobo web application, spending a spring day salvaging a nowhere-going Python socket server / Java client application by putting a web interface on it, I’ve made my living off of Zope and the technologies and systems that preceded it.

Back in 1996, if not earlier, Jim Fulton wrote ‘Bobo’. The legend goes that right after being hired by Digital Creations, he was flown out to give a talk about Python CGI / web programming, which he had never really done. Jim is a very smart man, and the legend (as I remember hearing it) is that he found CGI programming to be a terrible experience. He sketched out the concepts for Bobo on the flight back, and changed many lives forever.

At the time, CGI was the dominant web programming way. FastCGI and other tools were available to work around CGI’s shortcomings, but working with any of them required changing core code. One of the concepts of Bobo was to abstract out the “object publishing” process: Bobo dealt with the protocol/interface specifics. You just wrote regular Python code following a couple of simple rules, and you had a web application. Bobo was, in essence, an ORB (Object Request Broker). It turned an HTTP Request into an object call, typically following the path based on the URL:

    /greyhounds/betty_joan/editForm
       |
       |
       V
    greyhounds['betty_joan'].editForm()

For example. You could customize traversal on any object along the path, and in essence get any kind of system you want. The default behavior was just to follow Python attribute or ‘getitem’ access. To just start getting something up on the web, you could just use ‘Bobo’ and what you already had, more or less.

Bobo also took care of another problem that plagued CGI programmers. I started doing Python CGI programming not long after picking up the language in 1996, and I found that I spent more of my time trying to get data out of the request and getting it ready to pass into my program than I did doing my little programs themselves. It was very frustrating. Most of the errors would occur in the input / output phases of the CGI scripts I was writing. If I had a field like ‘races_won’ that I expected to come in as an integer, I had to convert it myself (at that time in Python, one had to use string.atoi(), int() wouldn’t parse strings), or deal with it not being in the request, and so on. Multiply this up for larger forms, and it was quite the annoyance. Form libraries didn’t really exist at the time either. So I had a script expecting an integer to come in, and I had all this code in front of it to make sure that (a) something came in, and (b) it was an integer. It wasn’t a hard thing to do, just frustrating. Since Bobo was about turning web requests into Python object calls, it had a way of marshalling the data in the form. With Bobo, I could have races_won:int as the name of a text field, and have a form action of ‘setRacesWon’. On a GET, the URL and resulting object call might look like this:

    /greyhounds/betty_joan/setRacesWon?races_won:int=8
      |
      |
      V
    greyhounds['betty_joan'].setRacesWon(races_won=8)

This was more impressive with larger inputs, like lists and (later on) records. And especially impressive when stacked up, so preferred_powerball_numbers:list:int would always bring in a list, whether there was one value submitted or many. This was all to shorten the space between the web and your own Python code.

Bobo also featured an easy entry point. Actually, it was a little tricky how you’d use the cgi module publisher (you’d basically do a symbolic link in your CGI directory to cgi_module_publisher, which was the thin wrapper that would publish your application by looking up the module name (packages didn’t really exist then either) based on the symbolic link). But! In your module, you could provide a couple of entry points: an object named bobo_application which would be the root of the application and where traversal would start from, or a dictionary called web_objects, with names (keys) mapping to objects/methods to publish. That was easy! That was awesome. In the application that I wrote that Spring 1997 day, I did that to make my front end. I already had this application written that searched a selectable list of different Glimpse text indexes and returned objects pointing to the found file, as well as matching line (and surrounding lines) information. All I really had to do was:

1. Make a page with a text field and some checkboxes.


2. Have that call my search method. My search method returned the HTML with the results.


3. Load and display the matched files.

For my situation, it was the view/controller layer straight up. I didn’t need another model layer: I already had a model of custom Python objects that bridged to this third party data.

So I made a DocumentTemplate for the search form, and had a function in my application module that ran the search by going through the selected search targets, based on the input. And I could publish it with the following in the top level of the module:

    def glimpseSearch(searchfor='', engines=[]):
        """ Search for the submitted text across engines """ 
        #.... python code

    def getFile(path=''):
        """ Get the published file from the relative path """ 
        #.... python code

    searchform = DocumentTemplate('searchform.dtml')

    web_objects = {
        'index_html': searchform,
        'search': glimpseSearch,
        'getfile': getFile,
    }

That was about it. I got a quick search form and quick results template together, and spent the first half of my day working on the search method and discarding some of the old socket-server specific code. Then I spent the second half of the day making the HTML look pretty. I still look at that and go “now there’s a starting point that I could teach somebody”. Zope has long lost that easy starting point. There are many many many starting point options. And even Zope 3, which is supposed to be stripped down and simpler in its concepts, cannot be so easily explained.

I was very impressed by that. I had my own “web framework” going at the time. Python’s always made it easy to write your own, be it 1996 or 2006. But after taking this death march project I had been working on and giving it new life by putting it on the web in a matter of hours, I completely abandoned my own foolish ideas and continued to build on Bobo.

At the time, there was a full “back to front” stack available for Bobo, but you could use all of these parts completely on their own with no dependency on the others:

1. bobo, aka the python object publisher: publishing objects on the web, marshalling requests. Oh, and it offered security too. Could be used on its own as the foundation for building a web application. Generally stayed out of your way otherwise.


2. DocumentTemplate: also known as DTML. Before Principia/Zope, DTML was really quite a nice template language. It did not have any Python expressions at all, and thus could not be used for programming. It had a lot of built in features for displaying and formatting data, especially when working with sequences. It was purely, purely, purely about inserting data into a string, and was quite nice to use at the time.


3. BoboPOS: This eventually became the ZODB. This was a persistent Python object storage. When used with Bobo, the Bobo publisher would start a transaction at the beginning of a web request, and commit it at the end if no errors were raised. I only used it lightly, but loved it when I did. Prior to this, I always had the problem (during development) of having code blow up, leaving me with partially written data that I’d have to clean by hand while also fixing my bugs.

You could use these individually, or all together, or in any combination.

The nice thing was – it was so easy! At least, after learning a couple of relatively simple concepts. There was an easy entry point to basically say ‘for this name, call this function’ or ‘start traversing this object’ and start writing a web application. The job of the FCGI Publisher, or CGI Publisher, was to find the name of the module to publish and pass the module into the publisher core, and that would look for these entry points.

Zope is just a Bobo application. Seriously! Zope 2 is still published as a ‘bobo_application’ object in a module. It just happens to be an object that wraps a ZODB instance. But at its heart, this is all it is.

But being able to so easily start writing a dumb simple web application based on other Python code you might have is not so obvious any more. Bobo got swallowed up into Zope 2. The core pieces are theoretically available again in Zope 3 as independently usable pieces, particularly zope.publisher. And there’s been a zope.bobo effort to restore that ease of use. But the effort hasn’t had much work. And I don’t contribute code to much these days.

I’m a bad person in this respect. I just bitch. But I’ve got a job, a dog, and art projects I’m neglecting that I shouldn’t be.

Anyways, I’ve been in discussions lately advocating a product strategy that offers up the core of the Zope 3 architecture. No object database. Just the component architecture (a pretty simple system on its own) and zope.publisher, which is the descendent of Bobo. Something that could be used to make a web application. Or not. But something that was separate, documented, exciting, etc. Something where you could actually do the “20 minute wiki” tutorial without spending the 20 minutes talking about the zope management interface, about skins, about different development options, about zope.conf and site.zcml files.

I’ve also been advocating a “Zope without all of Zope” release. One that has the Zope Object Database (it’s a better solution than relational databases for many web-targeted projects, and no translation layer like an object-relational mapper), and many of the items that are in the Zope 3 download you can get right now. But it wouldn’t have the “Zope Management Interface” that Zope 3 has. It wouldn’t have the half-assed content objects that are like weaker and harder to use and understand versions of their Zope 2 counterparts (things like page templates, python scripts, SQL scripts that are stored in the ZODB). They’re a terrible distraction. They’re hard to explain. I’ve seen very very smart Zope developers get confused about what to do and whether to use those things. And for writing custom applications, they’re completely unnecessary.

There’s a lot that I love in Zope 3. But there are also just enough frustrating things that I’m running into that, combined with the continued debating over whether or not to rename Zope 3 or what Zope 2 features should come in or god knows what, is having me looking at one of the Ruby on Rails mantras longingly:

Beauty Leads To Happiness

I’m still not as happy as I want to be. And I nearly came unglued when confronted by someone (someone I respect too) advocating a single “everything and the kitchen sink” distribution. I don’t care if that’s available. But I just want a couple of pipes and a wrench. If I have to keep running with this analogy by talking about all of the walls I need to knock down in order to understand why the values for the ‘select’ widget aren’t being saved properly, I’m just not sure this is a house I can stay in.

Ruby On Rails sticks to a strong opinion. Django and Turbogears in the Python world do too. They have a message that they all seem to believe in and love.

And I’m starting to realize that there’s never been a single Zope message that everyone has been able to rally behind. Is this why Zope 3 seems timid to even say it exists? It’s such a good architecture! There are some GREAT tools shipping with it (zope.formlib – the best form library I’ve ever worked with and it absolutely kills anything else I’ve seen out in the Python community), and some GREAT tools available for it (if you fish through Subversion repositories. zc.table looks like it’s the equivalent of ‘formlib’ for doing comprehensive and intelligent data grids of many styles, and I hope to take a close look at it very soon now; ‘hurry’ collects many useful little tools. It’s also pretty easy to start writing your own reusable supporting toolkit, so easy that you don’t even realize it’s happening). It has a lot of potential.

But it really feels like Zope doesn’t know where it wants to go. There’s an existing system to support, Zope 2, that has a lot of good work and maturity behind it. There’s a new system that – my gods – has the potential to just kick ass. And I think it could kick more ass if we could start breaking it out and making it simpler, smaller, and easier to use for developers while still providing larger options for those that like the complete stack or the larger feature set of Zope 2.

But… I don’t know which way it’s going. And there are still a lot of frustrating pieces in the code. And while some people are doing valiant work simplifying some core concepts, the meaning of it all if we can’t talk about it is beyond me.

I still have a hard time knowing how best to start a new ‘application’ in Zope 2 or 3. There are too many options, with too much work to go along with each one. And there are plenty of documents out there that will argue one way as being the best, and one will argue a completely different way as being the best.

And I see the quick start tutorials that other projects have and go “that makes sense” and as they mature, I really start to wonder if this Zope boat is the one I want to stay on if it can’t stick to a direction, communicate its intent, and (if it does pick a direction) heads back towards the land of more features that I’ve been struggling to get away from.