Agility versus Agility in Zope Development

Paul Everitt, one of the original founders of what is now Zope Corporation who is now doing a thousand different things across Europe, has a good post about 'agile content development', in reference to a CMS Watch post about a Mark Baker presentation (PDF) about this issue. Paul makes some interesting notes and reminders about some of the power of "classic Zope" through-the-web development, including a particular case where a fed-up Vignette user taught himself just enough through-the-web (TTW) Zope to be "agile" and made a replacement system for the big expensive Vignette setup in a matter of weeks.

Most seasoned Python and hardcore Zope developers cringe at the thought of managing Zope 1 and 2's through the web code. It presents many challenges for software configuration management, version control, editing, and so on. But for many users, there was a lot of flexibility in what Zope 1 and 2 offer here: free form schema management and development (using folder and document properties or ZClasses), changes instantly reflected throughout the system, ability to tweak individual scripts / handlers / templates, and so on. It turns the ZODB into more of an 'image' (in Smalltalk sense) than being just a plain database.

An interesting example of how people have learned and applied these techniques rapidly is something that a co-worker did a few years ago for one of our major customers. This customer could manage all of their free-form content via FTP, using some clever tricks we implemented. But for structured content - entries that had specialized fields - they managed that through web forms. Before Zope Page Templates, there were a couple of core dynamic content types in Zope - 'DTML Methods' and 'DTML Documents'. DTML Methods had no properties of their own - they were meant to act in the context of their container (usually a regular Zope folder). DTML Documents had their own property sheets, and the dynamic content acted in the context of the document. With 'Properties' on most Zope objects that supported them, you could add and remove property fields free-form, and marshal them to a particular data type. The default Zope property management screen would render and parse appropriately. This is long before any intelligent widget / form framework existed for Zope. It was just something that Zope did. So what my co-worker did for this customer was to use this properties system to make special objects. They were all DTML Documents, but the customer didn't know that. A property on the folder described what properties and types to add to new documents, and then they were rendered into editable fields by just asking for the properties. It's something that I wouldn't have even thought to do as a more seasoned Zope developer. But it was a plan that worked and served this customer well.

I think that maybe this is why I'm not so impressed when I see the "scaffolding in action!" screen cast for Rails or its wanna-bes. With classic Zope, even hurried and unfamiliar developers seemed to find ways to implement things like that, and without ever having to touch SQL. And for all of the faults that through-the-web development (particularly on the code side) may have, it was nice to be able to make quick fixes and patches from almost anywhere. There is something interesting about environments where objects (data) and code live in the same big ol' database. Smalltalk users and people who have used Lisp Machines often speak of their love of the seamless environment.

But let's face it - Zope 2 is great. But it's also weird and sloppy. It tries to cater to too many audiences, and there are people who have great love for certain aspects that other people loathe. ZClasses, which came the closest to allowing actual object and class development through the web, have little love from many 'power' users. Yet in the early days of Zope 3 development when features and plans were being discussed, there were many Zope 2 users who stood up in its defense: "It's so easy for me to make custom schemas through the web and make new objects for my users without having to program!" came up from a few different people. I don't even understand them! Through-the-web coding and data management in Zope 2 is a very fuzzy system. There often aren't any clear boundaries, and sustained application development in this model often gets difficult as time and size progress. The CMF (Content Management Framework), upon which Plone is built had some interesting ideas, some of which allowed for some powerful and strange customization. One of these was the 'Factory Method Content Type' (I think that was the name). Or 'Scriptable Content Type'? Anyways, with it you could take a basic base class/object type, like a Document, and really customize what happened to it at creation time. It was another way of providing this sort of strange 'agile content development' (from an implementer side) by allowing programmatic customization. Tres Seaver, one of the core CMF developers, once talked about a prototype concept, allowing any object in the Zope system to become a prototype.

An interesting concept of classic Zope, especially in the Principia (pre-Zope 2) days, was that Folders were customizable objects. They could have any number of custom attributes (properties) set on them, and have custom code (DTML Methods, initially) written for them. If you could grasp this concept, it could really be quite powerful for more than a few situations.

We still have one customer whose application is written in the old style - folders full of scripts and templates - templates, code, and configuration in the ZODB with business data in an RDBMS. We separate the scripts from the templates as much as possible, and it's a fairly manageable site. It's one that I sometimes cringe at having to deal with. But it's serving our customer just fine. While this customer is not much of a developer, he has often taught himself ways around the system when he wants to do something new (and small). We do all the major development. But it's often an interesting surprise to see how he's organically grown the parts he has access to.

Since late summer, we've shifted a lot of our focus to Zope 3. We've been building our own content management system on top of it, largely from scratch. Its initial customers are four sister companies, one with unique content management requirements, the other three with basic requirements. Zope 3 makes it easy to write and structure good frameworks and toolkits, and initialize them in a specialized order (much better than Zope 2's magical Products package initialization). We were able to structure our code into layers, from the big horizontal CMS layer to a narrower layer that provided tools that worked for all of these (and other potentially related) customers to individual packages providing direct customized components and views. By the time we got to the fourth customer's site, we were able to turn it around in a week, including graphic design and content entry. I think that's one of our fastest turn-arounds. Agile? You bet. And as the core improves, it's easy to get those updates to our customers, since it's properly managed Python code.

Now the ZODB is a database exclusively for content/data. We have no code in it anywhere. Those features aren't really there in Zope 3 right now, and I don't miss it... much. Unlike the myriad of potential audiences and uses that one could get out of a stock Zope 2 configuration, Zope 3 remains focused on developers at this point. I think that's a good thing. Even if one were to ignore the zope.app package, which provides most of the Zope application server functionality, there is a very impressive base. If someone wanted to make a version that had through-the-web classes and code, many of the base components that are necessary are there. If someone wanted to do a prototype based system, they could do it. I believe that with Zope 3 we are going to see more specialized applications released and built that can target different audiences, hopefully without having to make the core Zope systems have to cater to everyone.

Work has been done to allow Python code to exist in the ZODB as modules in Zope 3, with tools to allow checking code in and out from the file system. It's still incomplete, it seems, and I don't know if it's going anywhere. The file system checkin/checkout would allow one to use traditional tools (editors, version control) while also being able to take advantage of having code running in an image - dynamic updates, distribution (using tools like ZEO or Zope Replication Services), and so on.

Why might this be nice? Watch this QuickTime movie of Seaside halos. Seaside, a continuations based web framework written in Smalltalk, shows off many of the benefits of running code out of an image - live debugging, instant changes, and more. What's most interesting to me, however, are the "Halos". When they're toggled to be on, many elements of the web page get wrapped in boxes with an extra toolbar that allows live inspection and manipulation. Need to change the code that's rendering the shopping cart? Click on the halo button and you'll be taken directly to the method that's rendering it. All other code seems to be available for manipulation as well.

That's the closest I've been reminded of Zope 1 or 2. A dynamic running object environment on the web with live code in the database engine. The key difference is that Seaside's code editor still fits into how one might expect Smalltalk to behave, providing a web version of the System Browser. And I imagine that the Seaside Code Browser treats all code and objects the same. With Zope, file system code (regular Python code) is naturally off limits. It's also loaded and reloaded differently - updating a Python Script through the web has immediate repercussions. Updating something on the file system requires a manual product refresh or server restart.

Ugh. This is long and rambling and I'm not sure if there's really any point to it all - just some thoughts. Paul's comments reminded me of what we were getting away with in Zope back in the late nineties, before PHP had even started to make much of an impact. Zope 1 and 2 has always been a strange beast, quite unique from all other servers and frameworks. It's been very successful, yet also has a lot of people who hate it. Those of us who have used it for so many years still don't know how to explain it, and may even have two or three very different development styles for it depending on mood and the problem to be solved. Zope 3 is so refreshing - I get to take the core concepts that I've been using since 1997 but get a clean and free architecture to work with. I feel I have more freedom to make Zope 3 "my own". I could implement 'halos' for our content system. Or maybe start offering prototype based content objects. It's wide open.