apk zipalign

After ensuring some existing code would work under Zope 3.2 (still in beta testing at this point), I've been allowed this week to start work on a small internal (for now) project. I decided to take this opportunity to start looking at some new Zope 3.2 features. And while my experience with any of these is severely limited at this point, I want to mention them because they're all very interesting.

Content Providers, Viewlets, and Viewlet Managers

Two new top level packages in the Zope 3.2 distribution are zope.contentprovider and zope.viewlet. They're related, with the components in zope.viewlet building on zope.contentprovider.

What are these, and why are they important? In short, content providers are a simple construct that can be used to modularize web pages. Zope Page Templates have provided a decent way to modularize pages, using slots and macros to allow writing into multiple page parts more easily than a basic "insert header / body body body / insert footer" style template system. But it doesn't benefit from the flexibility that the component architecture can provide. As I've covered recently, A view in Zope is a multi-adapter, which discriminates against context (the object being viewed) and the request (HTTP request with skin information, FTP requests, and so on). A content provider adds to that, discriminating against the view as well. It's a smaller view used for that bigger view. More common terms for this are things like 'portlet'. There are some pages that you have, I'm sure, that you look at and you can see clear boxes that you are rendering in the common template, in included files, or being forced together a bit clumsily. You might be thinking to yourself "There's a lot of logic involved in rendering those navigation menus. I've got this great component architecture behind me, and I'm pulling that into a page with a 'var' and hoping it's there? Agh. And I don't want that navigation to show up like that in all of my shopping cart views... I need a different set of information over there during checkout..."

It appears to me that this is what content providers (and the viewlets built on top of them) are meant to solve. Based on JSR 168 - Portlet Specification, a content provider must implement only two methods: update() and render(). update() is the chance that the provider has to respond to the request and change state. It's always called first. Then render() is called to, um, render. On its own, it doesn't seem impressive. But its simplicity belies its power. zope.contentprovider doesn't provide an implementation, only some Interface specifications and a TALES namespace for using content provider components in Zope Page Template documents using the registered name. The component architecture will deliver the right component for the page being rendered. One has to look at zope.viewlet for base implementations.

zope.viewlet provides a common content provider in the form of a ViewletManager. The ViewletManager is probably the most common content provider one might call directly in a template. Viewlets are then registered with a ViewletManager, which will filter, sort, update, and render each Viewlet. A viewlet adds a forth discrimination - the viewlet manager. It all still builds on the component architecture - it's still multi-adapters underneath it all. So how might one use it? One example is the good old "side column" part of a page. With viewlets, one might register this as a viewlet manager with an interface name like ISideColumn or IAdditionalInformation. Then, one might provide a small viewlet for ISideColumn for all pages that shows the title, created date, modified date. Let's then say that you install a system that can find related content based on the tags or text of the current object. You could write a new viewlet that lists those pages in the ISideColumn - but only on pages that implement 'ITaggable'. The original template or viewlet manager don't have to be modified to render this, or to have any logic in them to discriminate against 'ITaggable'. The component architecture does the bulk of the work to find viewlets that can be drawn in a particular manager, and a particular viewlet manager implementation may do its own sorting and filtering of those viewlets - the base manager class provided in Zope 3.2, for example, filters out viewlets that the current user can't see.

What does this mean? What are the benefits of content providers / viewlets? I'm only speculating, based on very little personal experience so far, but I'm impressed with the simplicity of the interface. So here's a bulleted list of where I expect to benefit from this, and where I think others could too:

• Easier to specify page parts for current and future development. Build the page out of ViewletManagers, which are backed by Zope Interfaces, and then provide page parts against those. Navigation sections, promotional image displayers, menu items, footer links, sidebars, main areas. You can find these parts and formally specify them, and then provide the viewlets to flow into them.


• More intelligence in small parts. Viewlets may be simple or complex, but basically they can now have the same great (or small) intelligence that full view components have had. You know how it is - the middle left column of the page is expected to provide a search form, a navigation menu listing local pages, and a couple of small banner ads. Now all of those pieces can be more easily coded as separate components, but all in the same manner. This is possible in Zope 3.1, but content providers make this a lot easier.


• From a single code base, provide unique viewlets for the home page, section fronts, and common templates. This is another common little thing - lets say you have to have to add a hidden field to the little search form box used on every page, and the one on the front page needs a pull-down menu to allow discriminating results against different parts of the site. Normally, this little form is a small bit of code that's just managed in the templates. No big deal, there's only three - but they're all slightly different. You can't just copy and paste without paying attention to the unique requirements for each scenario. Putting that search form into a viewlet, you'd have access to the code used in each. Maybe you're using the same in each one. You can now make a specialized viewlet for the home page that adds the drop-down menu, while keeping it all tied together on the search page. You can also ensure that the 'search' sidebar widget does NOT draw on the search results page, or provide a more advanced search sidebar to filter results.


• Make a viewlet manager to handle HEAD links and easily add extra javascript library or CSS loading to pages that need them. I've commonly made page template 'slots' to allow me to load or add page-specific javascript or style sheets to individual templates. Using a viewlet manager for these, you can ensure that 'foolib.js' is there for every IFooListing based page.


• Use Page Templates, use another template language, render HTML directly - it doesn't matter. The interface is so simple, you no longer have to wonder about whether to use __call__ or __str__ or render as the best way to render a little component. 'update()' and 'render()' are all you need to worry about. Yet they can all plug into a larger page with ease, regardless of how that page is rendered. There are niceties for ZPT to make using content providers easy, but one could easily make a big "standard template" without a template language at all - just call and render a few principal ViewletManagers.


• Caching. There hasn't been a formal way of being able to cache fragments of pages. There's decent caching support (with more pluggable) in Zope 3, but no easy way of being able to apply them to parts of a page. With viewlets, it should be easier to put in caching. With more intelligence put into a viewlet, it could manage its own cached data or output. More effectively, one could put this in at the Viewlet Manager level without having to effect individual viewlets. The spec for content providers is simple enough that this could be plugged in a lot more easily and as needed.


• Based on simple standard. It's impressive to see that this is based on JSR 168. When I first heard this, some time ago, I flinched. JSR specifications are from a world that I just don't understand anymore, it seems. Yet this one is nice and simple and effective. It also should quiet some not-invented-here criticisms of Zope 3. At the same time, it has been specified and implemented in a way that fits with Zope 3. When I first looked at content providers and viewlets, I flinched then too. But as I started to use them, I realized that they didn't differ much from what I had already learned about views, widgets, and other multi-adapters. I've made 'views' with the express purpose of being rendered as part of a page, but they haven't followed a common interface. There are many other elements I see now that can fit into this system and ease use.

zope.formlib

The other addition I wanted to cover, briefly, is zope.formlib. Formlib is a general toolkit and API for doing intelligent web form development in Python. It's not a widget library - most of those are provided by the widgets in zope.app.form.browser, but could come from anywhere. It's not a validation framework - that's provided by zope.schema based fields (which work with Zope Interfaces). It does use both of those, and provides a lot of extra features to collect all of the input, apply the validation or collect errors, do invariant based validation ('end date must be later than start date'), dispatch to different responsive actions, and more.

It's a toolkit capable of building powerful base classes and individual uses for forms that are simple (just a few fields) or very complex. There were some form views provided in Zope 3.0 and 3.1 that do similar, but they weren't so easy to use and understand and didn't handle complex forms. Custom field and widget combinations required a lot of custom work, and so did dispatching to multiple actions based on success, failure, and so on. Lastly, the older (non-formlib) techniques seemed to prefer being built via the Zope 3 component configuration language when it really required Python to get the kind of flexibility that form development requires.

In wrap-up, formlib is promising because it shows a preference for using Python and some of the recent features such as decorators over giving everything over to configuration based automation. Content Providers are a simple specification for modularizing web pages into manageable components, with viewlets and viewlet managers providing a reasonable base implementation to work with. The dual layers provide a boundary between a basic spec and small set of tools (the content provider interface, a TALES expression for use in zope page templates) that have broad usage and more specialized providers, built from real-world usage and previous art and examples like JSR 168.

My previous post showed how Interfaces and Adapters provide the heart of the Zope Component Architecture, which beats at the center of the Zope 3 framework and application server, and is also pumping new life into Zope 2. The other chief piece of the Zope Component Architecture are Utilities. Interfaces allow coding to a specification, which can be as detailed or as general as one chooses. Adapters allow for other objects to fit that specification. Utilities are more general purpose tools that match a specification. The rest of the core 'zope' packages enhance or build on these facets, and the 'zope.app' package provides the Zope application server which is the most akin to the Zope that people know and love or hate.

Yesterday I showed how a very simple web publishing system could be made using just the basic pieces of the component architecture. Today I want to augment yesterday's example by showing how the component architecture provides "the one way to do it".

Kevin Dangoor, the primary developer behind the Turbogears project, wrote a great post today about What Turbogears Is Not. In it he discusses how choices are made about what goes into Turbogears, which prides itself on not being new technology but a combination of already offered Python toolkits. Kevin goes into coverage "one way to do it" by mentioning how the Turbogears widgets system uses Kid, the primary template system used for Turbogears. It's the default and chosen template system, so why not take advantage and say "this is how we do widgets. period," right?

Anyways, it got me thinking about what I said yesterday about the Zope Component Architecture and how everything flows through it at some point. 'Multi-adapters', especially, have become powerful. Most commonly, they're used to provide views for objects. Yesterday showed a couple of custom views built to display simple and detailed information about racing greyhounds. Today I want to show how widgets work.

First off - I did a little bit of code cleanup of the module and use zope.publisher.interfaces.IRequest and zope.publisher.browser.TestRequest instead of defining my own dummy request.

import zope.component
import zope.interface
import zope.publisher.browser
import zope.publisher.interfaces
import zope.schema

I'm not going to show the entire module again, but I want to bring back IGreyhoundData. This interface specifies what we expect any greyhound data providing object to give us:

class IGreyhoundData(zope.interface.Interface):
    """
    zope.schema provides a way of defining the properties and their bounderies
    as they make up a component. It integrates with the interface system and
    can be used for documentation, validation, data retrieval and setting, and
    more. The fields define 'get' and 'set' methods, among others, useful for
    querying and setting data on an object without having to have intimate
    knowledge of the object's structure. Useful for web forms, but also for
    database storage and retrieval.
    """
    name = zope.schema.TextLine(title=u"The Greyhound's Name")
    raceName = zope.schema.TextLine(title=u"The name on the race card.")
    fastestTime = zope.schema.Float(title=u"Fastest race time (in seconds).")
    trackLength = zope.schema.Int(title=u"The length of the track, in yards, of the fastest time.")
    retired = zope.schema.Bool(title=u"Is the greyhound still racing?")
    adopted = zope.schema.Bool(title=u"Has the greyhound been adopted?")

Lastly, I created a 'simpleConfigure' function in the top level of the module that registers the main views and utilities with the component architecture. This configure option can be run without any requirements aside from the 'zope' Python packages being in the Python path.

def basicConfigure():
    """ 
    Register the views and utilities with the component architecture. This
    could have been done in the top-level of the module, but deferring until
    now is a little bit cleaner and allows other configuration options to be
    made.
    """
    greyhounds = GreyhoundFinder()
    zope.component.provideUtility(greyhounds, name='greyhound')
    zope.component.provideAdapter(RacerView, name='view')
    zope.component.provideAdapter(DetailsView, name='detail')

All of the provided components in this setup also provide their own information about what interfaces they in turn provide and adapt. As a refresher, 'greyhounds' is an IDataFinder utility, used by the publishing system to look up the "model" object. The two adapters are multi adapters for IGreyhoundData and zope.publisher.interfaces.IRequest. They provide 'ISimpleView', which has a 'render()' method to render the view to the response stream. I want to reiterate that this has no dependencies on anything that the world thinks of as Zope. This is all in a single module that could very easily be published via CGI without ZODB, transactions, long-running-anything, security restrictions, through the web programming... None of it.. This just uses the heart.

However, showcasing the widgets used by Zope 3 does require a full Zope setup. Technically, I could configure the needed widgets myself and show how that's done, but that's beyond the point of this exercise. I chiefly want to show how the Widgets flow through the same heart. There's just one way to do it. To make a rudimentary edit form for a Greyhound in the little system I have set up, we create another class to provide the edit form view. I try to document everything that's going on. Take note of the use of zope.component.getMultiAdapter() to get the widget.

class EditForm(object):
    """ Requires zope.app.form widgets to be configured. """
    zope.interface.implements(ISimpleView)
    zope.component.adapts(IGreyhoundData, zope.publisher.interfaces.IRequest)

    def __init__(self, context, request):
        self.context = context   # aka, the racer
        self.request = request

    def render(self):
        """ 
        Goes through the schema fields and looks up the IInputWidget adapter
        configured for the field and request type. 'zope.app.form' is imported
        here to avoid placing burden on the rest of this module, since this
        code requires the widget configuration to have occurred. Just importing
        'zope' does not provide this, but running a Zope application server
        does.
        """
        from zope.component import getMultiAdapter
        from zope.app.form.interfaces import IInputWidget
        racer = self.context
        request = self.request
        out = []
        for name, field in zope.schema.getFieldsInOrder(IGreyhoundData):
            value = field.get(racer)

            # Just like getting a full 'view' to render, widgets are also
            # provided as multi adapters, adapting an object (in this case, a
            # field) and the request to an IInputWidget implementation. A
            # display-only widget would use IDisplayWidget.
            widget = getMultiAdapter((field, request), IInputWidget)
            widget.setRenderedValue(value)

            # A widget has a label, which it often gets from the field's title.
            # To render the widget, just call it.
            out.append('%s: %s' % (widget.label, widget()))

        return '\n'.join(out)

The first few lines of the render() method just help establish some shorter names, and also defer loading zope.app.form so it's not required by this whole module - although in theory it wouldn't harm anything if it were imported at the top. The zope.schema introspection machinery is used again to walk through the fields of an Interface object and get the fields. We get the object's current value by going through the field. Then we get the widget. The Zope Component Architecture comes into play once again as we try to adapt both the field and request object to an IInputWidget. The value is set on the widget based on the object's value, and we add it to the output.

How is that widget made? Does it use templates? Just write out strings in Python directly? It doesn't matter. The IInputWidget interface is all we care about here. Form submission and validation goes through the widgets and fields too, but that's not important for right now. What is important and interesting is how adapters are used here to deal with widgets, and they're completely independent from the schema. Using the 'schema' objects, one could write completely different adapters to do object-relational mapping and provide converters to read/write SQL data based on the object specification, while still using that same specification to render web forms or write out dynamic XML-RPC or JSON request responses. sqlformatter = getMultiAdapter((field, databaseConnection), ISQLFormatter) could easily be used. As a technical aside - with how multi-adapters and interfaces work, there could be generic formatters for all data types but a specific datetime formatter could be provided for a SQLite or Oracle connection. Multi-adapter lookup will match the most specific adapter based on the specifications of the objects passed in. See what I mean about everything flowing through the same heart?

It's time to put our new view into play. I added an extra configuration method that would set up the basic components and add in the edit form:

def configureWithWidgets():
    """ 
    Establishes basicConfigure and also enables the edit form which uses the
    zope.app.form system for widgets. Requires a configured Zope app server
    setup - at least with the zope.app.form widgets configured to provide
    adapters.
    """
    basicConfigure()
    zope.component.provideAdapter(EditForm, name='edit')

And with how our 'simplePublish' system is set up (the only change since yesterdays code is the use of zope.publisher.browser.TestRequest instead of the self-defined simple request), we now have edit actions:


>>> simple.configureWithWidgets()

>>> print simple.simplePublish('greyhound/edit/1')

The Greyhound's Name: <input class="textType" id="field.name" name="field.name" size="20" type="text" value="Betty Joan" />

The name on the race card.: <input class="textType" id="field.raceName" name="field.raceName" size="20" type="text" value="Beauty" />

Fastest race time (in seconds).: <input class="textType" id="field.fastestTime" name="field.fastestTime" size="10" type="text" value="31.22" />

The length of the track, in yards, of the fastest time.: <input class="textType" id="field.trackLength" name="field.trackLength" size="10" type="text" value="503" />

Is the greyhound still racing?: <input class="hiddenType" id="field.retired.used" name="field.retired.used" type="hidden" value="" /> <input class="checkboxType" checked="checked" id="field.retired" name="field.retired" type="checkbox" value="on" />

Has the greyhound been adopted?: <input class="hiddenType" id="field.adopted.used" name="field.adopted.used" type="hidden" value="" /> <input class="checkboxType" checked="checked" id="field.adopted" name="field.adopted" type="checkbox" value="on" />



We've done amazing work in the past few months with Zope 3. Compared to the black magic of Zope 2 and so many other frameworks I look at, it's quite refreshing. It's not always easy, but it's been fun while also being quite trustworthy. Even without a deep understanding of all that's available, my very small company and our even smaller team of developers have achieved incredible results: a from scratch rewrite of one of our oldest and most heavily used customer's web sites, three other customers with solutions built on top of the same stack written and extracted for that earlier customer; a one week turnaround (including graphic design) of one of those other customer's web sites; sharing of benefits of refactoring of code from the customer specific layers of the stack to more general layers; an 'admin' user interface far more capable and complete than just about anything we've delivered in the past; all developed faster than ever before; and with a clean architecture despite only the lightest of planning.

What makes all this possible? The Zope Component Architecture. The Zope Component Architecture is a pretty lightweight system, especially since Zope 3.1 which simplified the component architecture that shipped with Zope 3.0 into a simple vocabulary. There are only a couple of core Python packages that provide the heart of the component architecture. And everything essentially flow through the heart.

The core packages that provide the Zope Component Architecture are zope.interface and zope.component. zope.interface provides a way of describing and querying specifications. Similar in some ways to Interface objects in Java or in systems like CORBA and ILU, zope.interface Interfaces can be used to describe objects, policies, and APIs separate from their implementation. By writing against the interface, Zope 3 takes advantage of one of the key advantages of dynamic languages like Python - Duck Typing. Duck Typing is a way of saying "if it quacks like a duck, then that's good enough for me!" Just putting the zope libraries on the Python path - configuring and setting up nothing else - here's how this works:

>>> from zope.interface import implements,  providedBy, Interface
>>> class IQuacksLikeADuck(Interface):
...     def quack():
...         """ Returns a quacking sound. """
... 
>>> class Platypus(object):
...     implements(IQuacksLikeADuck)
...  
...     def quack(self):
...         return "In my world, a platypus goes 'quack!'"
... 
>>> ben = Platypus()
>>> IQuacksLikeADuck.providedBy(ben)
True
>>> print ben.quack()
In my world, a platypus goes 'quack!'

Now, the normal and generally allowable Python way of Duck Typing is to do something like if hasattr(ben, 'quack'): ben.quack(). But that sort of 'hasattr' code only goes so far. Zope 2 is littered with skeletons like if getattr(obj, '_isPrincipiaFolderish', 0):.... As systems grow, developers forget those old names. Or objects just grow heavy with odd little attributes that say they have not only one thing about them that makes them folderish, but many. And if its folderish, then maybe we'll draw a tree for it. But if '_isPrincipiaFolderish' is really an important attribute - where does that get documented? Who knows it's there? Who knows what its values may be. Can it be callable? Zope 2 actually has attributes like this - elements that started out as simple attributes, and then grew to be callable for situations that needed to do computation. And then grew to be callable with parameters for really specific situations. But finding out about all of these combinations and their meaning was not always easy to document or even infer. Recognizing this, an Interfaces package was made for Zope 2 but was initially used only for documentation. But Interfaces were not used as core elements, and there was little impetus to keep an interface spec up to date when it wasn't actually used in code. But in Zope 3 and in new Zope 2 work, that changes. Interfaces are core members. It means that not only are most important things documented, but that it's really more important for an object providing a certain interface to just provide that interface - and not how its implemented. I don't care how it quacks, as long as it quacks. Quacking is all that I require. Interfaces allow one to formalize duck typing, without really adding too much weight to their program. In fact, with interfaces being a sort of public interface, I find it easiest now to look at a packages 'interfaces.py' module first when looking at new code. How a specific implementation of a security policy doesn't really matter to me at first - I want to know how I interact with the security policy. So interfaces become useful for documentation without having to generate or wade through documentation generated for every single function and class and method in a particular module.

Of course, interfaces on their own are still not all that great. The second part offered by zope.interface is adaptation. Sticking with the quacking example, let's add a hunter into the mix. He wants to go duck hunting. He's going to need to attract ducks by quacking like them.

>>> class Hunter(object):
...     def __init__(self, name):
...             self.name = name
... 
>>> tom = Hunter('Tom')
>>> IQuacksLikeADuck.providedBy(tom)
False

What we need here is an adapter. This is where zope.component and zope.interface really collaborate. By default, most adapters in Zope adapt from one interface to another. But you can also adapt a class to an interface, which is what we'll do here since we didn't make an 'IHunter' or 'IPerson' or 'IQuackless' interface.

>>> from zope.component import adapts
>>> class DuckCall(object):
...     implements(IQuacksLikeADuck)
...     adapts(Hunter)
...  
...     def __init__(self, hunter):
...         # Adapters are passed in their adaptee as first argument
...         self.hunter = hunter                                 
...  
...     def quack(self):
...         return self.hunter.name + ' quacks with a duck call'
... 
>>> zope.component.provideAdapter(DuckCall)

So here, we say that instances of DuckCall implement the IQuackLikeADuck interface, and that this particular class adapts Hunter objects. The next line registers the adapter with the global component architecture adapter registry. This is the easiest version of the call - a factory (in this case a class) is passed in which declares what it adapts and provides directly. Now there's a duck call available for hunters. How do we use it? There are a couple of ways. One is to use zope.component.queryAdapter or getAdapter - the only difference being that queryAdapter returns 'None' or a user supplied default if the adaptation can't happen, while getAdapter raises an exception. But the interesting way is to use the interface directly, passing in the object:

>>> IQuacksLikeADuck(tom)
<__main__.DuckCall object at 0x67cad0>

Like with 'queryAdapter', a second argument can be passed in as a default if the object cannot be adapted, otherwise an exception is raised. What this form allows for is code that works with objects by interface, and it allows for other code to provide expected interfaces for existing objects. Lets add to the wildlife. First, there's the hunters dog:

>>> class IrishWolfhound(object):
...     size = 'HUGE'
...     def __init__(self, name):
...             self.name = name
... 
>>> squeekers = IrishWolfhound('Squeekers')

And a duck. Since the duck provides 'quack' directly, we can separately say that instances of the class provide the IQuacksLikeADuck interface, or we could even say that a particular instance provides it. This shows how we can separately say that the class implements it.

>>> class Duck(object):
...     def quack(self):
...             return "The best quack is a duck quack."
... 
>>> hunted = Duck()
>>> IQuacksLikeADuck.providedBy(hunted)
False
>>> from zope.interface import classImplements
>>> classImplements(Duck, IQuacksLikeADuck)
>>> IQuacksLikeADuck.providedBy(hunted)
True

And it should be noted that if an object provides an interface itself, then the adapter lookup will return the object itself. So now that we have quite a little menagerie. Lets see what some simple tickler code might look like that would take a list of objects and try to make them quack.

>>> def tickler(*args):
...     " Goes through the provided arguments and tries to make them quack "   
...     for potentialQuacker in args:
...         quacker = IQuacksLikeADuck(potentialQuacker, None)
...         if quacker is None:
...             print "Could not quack: %r" % potentialQuacker
...         else:
...             print quacker.quack()
... 
>>> tickler(ben, tom, squeekers, hunted)
In my world, a platypus goes 'quack!'
Tom quacks with a duck call
Could not quack: <__main__.IrishWolfhound object at 0x698bd0>
The best quack is a duck quack.

Of course, this is a fiendishly simple showcase. But it's a powerful building block, and is one of the core concepts of Zope 3. Just about everything flows through interfaces and adapter registries. A powerful extension of the 'adapter' concept is the 'multi-adapter'. The multi-adapter situation most commonly used is the Browser View. A Browser View is made by adapting an object, like the Duck, and another object, like an HTTP Request, to something that gets rendered to a web browser via HTTP. Fundamentally, the Component Architecture merely provides the registries for loosely-coupled objects to work together, and it removes a lot of the magic.

In something purely theoretical, lets remove all of Zope as people know it (the object database, the publisher that traverses URLs to return objects, and so on and so on. Using the component architecture, lets do an extremely lightweight 'rails'-ish thing. This code makes a fake little database with just tuples of data, as might be returned by a basic dbapi adapter. It defines a Utility to find the 'model' from this database. A utility is a new concept. In quick, utilities are other abstract notions, used loosely. One can define and install different utilities, and the code that uses the utility has no hard dependencies on the utility object. Examples include RDB connections - 'get me an IRDBConnection with the name "racers"'; caches; and more. Sometimes all utilities for a particular interface are queried, sometimes you just want an individual one. The zope.component.provideUtility call establishes the utility. In this code we have only one and we name it 'greyhound'. That name is used later in the 'simplePublish()' function to find the utility by combination of its name and published interface. Again - this code was tested by ensuring that my Zope 3.1 installations lib/python directory was in the Python path and that's it.

import zope.component
import zope.interface
import zope.schema
class IGreyhoundData(zope.interface.Interface):
    """
    zope.schema provides a way of defining the properties and their bounderies
    as they make up a component. It integrates with the interface system and
    can be used for documentation, validation, data retrieval and setting, 
    and more. The fields define 'get' and 'set' methods, among others, useful
    for querying and setting data on an object without having to have intimate
    knowledge of the object's structure. Useful for web forms, but also for
    database storage and retrieval.
    """
    name = zope.schema.TextLine(title=u"The Greyhound's Name")
    raceName = zope.schema.TextLine(title=u"The name on the race card.")
    fastestTime = zope.schema.Float(title=u"Fastest race time (in seconds).")
    trackLength = zope.schema.Int(title=u"The length of the track, in yards, of the fastest time.")
    retired = zope.schema.Bool(title=u"False if the greyhound is still racing.")
    adopted = zope.schema.Bool(title=u"True if the greyhound has a home after the track.")

class IDataFinder(zope.interface.Interface):
    def find(oid):
        """ Returns an object that matches this object id. Raises KeyError if not found. oid should be an integer. """

# Define a fake database
greyhounddb = {
  1: ('Betty Joan', 'Beauty', 31.22, 503, True, True),
  2: ('Dazzling', 'Dazzling Pet', 32.00, 503, True, False),
}

class Phantom(object): 
    """ Phantoms can be anything... """
    pass
 
class DataFinder(object):
    zope.interface.implements(IDataFinder)
    schema = None
    factory = None
    db = None
    
    def find(self, oid):
        """ 
        Calls the fake database and uses the schema system to map fields
        from the tuple returned to a new Phantom object, and then ensures that 
        the phantom is marked to provide the schema in 'self.schema'
        """
        raw = self.db.get(oid)
        newobj = self.factory()
        for idx, name in enumerate(zope.schema.getFieldNamesInOrder(self.schema)):
            field = self.schema[name]
            value = raw[idx]
            field.set(newobj, value)
        zope.interface.directlyProvides(newobj, self.schema)
        return newobj

# Define a basic data finder
class GreyhoundFinder(DataFinder):
    schema = IGreyhoundData
    factory = Phantom
    db = greyhounddb

greyhounds = GreyhoundFinder()
zope.component.provideUtility(greyhounds, name='greyhound')

class ISimpleRequest(zope.interface.Interface):
    headers = zope.interface.Attribute('Mapping of request headers')

class SimpleRequest(object):
    zope.interface.implements(ISimpleRequest)
    def __init__(self, **headers):
        self.headers = {}
        self.headers.update(headers)

class IView(zope.interface.Interface):
    """ Simple view interface. """
    def render():
        """ Returns a rendered string of this view. """

class RacerView(object):
    zope.interface.implements(IView)
    
    def __init__(self, context, request):
        self.context = context   # aka, the racer
        self.request = request
        
    def render(self):
        return "Name: %s; Fastest Time: %0.2f" % (self.context.name, self.context.fastestTime)
zope.component.provideAdapter(RacerView, (IGreyhoundData, ISimpleRequest), name='view')

detail_template = """Name: %(name)s
Racing Name: %(raceName)s
Fastest Time: %(fastestTime)0.2f for length %(trackLength)d
Retired: %(retired)s
Adopted: %(adopted)s
"""

class DetailsView(object):
    zope.interface.implements(IView)
    
    def __init__(self, context, request):
        self.context = context   # aka, the racer
        self.request = request

    def render(self):
        racer = self.context
        mapping = dict([ 
            (name, field.get(racer)) 
            for name, field in zope.schema.getFieldsInOrder(IGreyhoundData) 
            ])
        return detail_template % mapping
zope.component.provideAdapter(DetailsView, (IGreyhoundData, ISimpleRequest), name='detail')
        
def simplePublish(path):
    """ 
    Breaks 'path' up into three components:
    
    - ``name``: used to look up a data finder utility
    - ``viewname``: used to query a multi-adapter for the found object and simple
      http request
    - ``oid``: used to look up an object out of the finder
    
    from ``name/viewname/oid``
    
    returns the rendered view.
    
    A very very simple showcase of how the zope component architecture might
    be used to register and find model accessors and views in a web publishing
    system besides the traditional Zope and 'zope.app' setup.
    """
    request = SimpleRequest(ignored='hi')
    getter, viewname, oid = path.split('/')
    oid = int(oid)
    
    # This gets an 'IDataFinder' utility with the name in the 'getter' value.
    # A greyhound getter that queries a real database could have been installed
    # instead - it doesn't impact this code.
    model = zope.component.getUtility(IDataFinder, name=getter).find(oid)
    
    # Now that we have the model object and the fake request, find something
    # that adapts both of them to a single object. Since the request is fake
    # and the views are simple, we don't really show how the request object
    # is used here. But this is basically where the web environment and model
    # environment would actually meet - only in the multi-adapter. Other than
    # that, both environments are oblivious to each other. It's only the zope
    # component architecture registries that know what's in them, and they're
    # pretty agnostic.
    view = zope.component.getMultiAdapter((model, request), name=viewname)
    return view.render()

def test():
    print "Publishing ``greyhound/view/1``"
    print simplePublish('greyhound/view/1')
    print
    print "Publishing ``greyhound/view/2``"
    print simplePublish('greyhound/view/2')
    print
    print "Publishing ``greyhound/detail/1``"
    print simplePublish('greyhound/detail/1')
    print
    print "Publishing ``greyhound/detail/2``"
    print simplePublish('greyhound/detail/2')
    
if __name__ == '__main__':
    test()

And when run:

Desktop> python zopetest.py
Publishing ``greyhound/view/1``
Name: Betty Joan; Fastest Time: 31.22

Publishing ``greyhound/view/2``
Name: Dazzling; Fastest Time: 32.00

Publishing ``greyhound/detail/1``
Name: Betty Joan
Racing Name: Beauty
Fastest Time: 31.22 for length 503
Retired: True
Adopted: True


Publishing ``greyhound/detail/2``
Name: Dazzling
Racing Name: Dazzling Pet
Fastest Time: 32.00 for length 503
Retired: True
Adopted: False

Paul Everitt, one of the original founders of what is now Zope Corporation who is now doing a thousand different things across Europe, has a good post about 'agile content development', in reference to a CMS Watch post about a Mark Baker presentation (PDF) about this issue. Paul makes some interesting notes and reminders about some of the power of "classic Zope" through-the-web development, including a particular case where a fed-up Vignette user taught himself just enough through-the-web (TTW) Zope to be "agile" and made a replacement system for the big expensive Vignette setup in a matter of weeks.

Most seasoned Python and hardcore Zope developers cringe at the thought of managing Zope 1 and 2's through the web code. It presents many challenges for software configuration management, version control, editing, and so on. But for many users, there was a lot of flexibility in what Zope 1 and 2 offer here: free form schema management and development (using folder and document properties or ZClasses), changes instantly reflected throughout the system, ability to tweak individual scripts / handlers / templates, and so on. It turns the ZODB into more of an 'image' (in Smalltalk sense) than being just a plain database.

An interesting example of how people have learned and applied these techniques rapidly is something that a co-worker did a few years ago for one of our major customers. This customer could manage all of their free-form content via FTP, using some clever tricks we implemented. But for structured content - entries that had specialized fields - they managed that through web forms. Before Zope Page Templates, there were a couple of core dynamic content types in Zope - 'DTML Methods' and 'DTML Documents'. DTML Methods had no properties of their own - they were meant to act in the context of their container (usually a regular Zope folder). DTML Documents had their own property sheets, and the dynamic content acted in the context of the document. With 'Properties' on most Zope objects that supported them, you could add and remove property fields free-form, and marshal them to a particular data type. The default Zope property management screen would render and parse appropriately. This is long before any intelligent widget / form framework existed for Zope. It was just something that Zope did. So what my co-worker did for this customer was to use this properties system to make special objects. They were all DTML Documents, but the customer didn't know that. A property on the folder described what properties and types to add to new documents, and then they were rendered into editable fields by just asking for the properties. It's something that I wouldn't have even thought to do as a more seasoned Zope developer. But it was a plan that worked and served this customer well.

I think that maybe this is why I'm not so impressed when I see the "scaffolding in action!" screen cast for Rails or its wanna-bes. With classic Zope, even hurried and unfamiliar developers seemed to find ways to implement things like that, and without ever having to touch SQL. And for all of the faults that through-the-web development (particularly on the code side) may have, it was nice to be able to make quick fixes and patches from almost anywhere. There is something interesting about environments where objects (data) and code live in the same big ol' database. Smalltalk users and people who have used Lisp Machines often speak of their love of the seamless environment.

But let's face it - Zope 2 is great. But it's also weird and sloppy. It tries to cater to too many audiences, and there are people who have great love for certain aspects that other people loathe. ZClasses, which came the closest to allowing actual object and class development through the web, have little love from many 'power' users. Yet in the early days of Zope 3 development when features and plans were being discussed, there were many Zope 2 users who stood up in its defense: "It's so easy for me to make custom schemas through the web and make new objects for my users without having to program!" came up from a few different people. I don't even understand them! Through-the-web coding and data management in Zope 2 is a very fuzzy system. There often aren't any clear boundaries, and sustained application development in this model often gets difficult as time and size progress. The CMF (Content Management Framework), upon which Plone is built had some interesting ideas, some of which allowed for some powerful and strange customization. One of these was the 'Factory Method Content Type' (I think that was the name). Or 'Scriptable Content Type'? Anyways, with it you could take a basic base class/object type, like a Document, and really customize what happened to it at creation time. It was another way of providing this sort of strange 'agile content development' (from an implementer side) by allowing programmatic customization. Tres Seaver, one of the core CMF developers, once talked about a prototype concept, allowing any object in the Zope system to become a prototype.

An interesting concept of classic Zope, especially in the Principia (pre-Zope 2) days, was that Folders were customizable objects. They could have any number of custom attributes (properties) set on them, and have custom code (DTML Methods, initially) written for them. If you could grasp this concept, it could really be quite powerful for more than a few situations.

We still have one customer whose application is written in the old style - folders full of scripts and templates - templates, code, and configuration in the ZODB with business data in an RDBMS. We separate the scripts from the templates as much as possible, and it's a fairly manageable site. It's one that I sometimes cringe at having to deal with. But it's serving our customer just fine. While this customer is not much of a developer, he has often taught himself ways around the system when he wants to do something new (and small). We do all the major development. But it's often an interesting surprise to see how he's organically grown the parts he has access to.

Since late summer, we've shifted a lot of our focus to Zope 3. We've been building our own content management system on top of it, largely from scratch. Its initial customers are four sister companies, one with unique content management requirements, the other three with basic requirements. Zope 3 makes it easy to write and structure good frameworks and toolkits, and initialize them in a specialized order (much better than Zope 2's magical Products package initialization). We were able to structure our code into layers, from the big horizontal CMS layer to a narrower layer that provided tools that worked for all of these (and other potentially related) customers to individual packages providing direct customized components and views. By the time we got to the fourth customer's site, we were able to turn it around in a week, including graphic design and content entry. I think that's one of our fastest turn-arounds. Agile? You bet. And as the core improves, it's easy to get those updates to our customers, since it's properly managed Python code.

Now the ZODB is a database exclusively for content/data. We have no code in it anywhere. Those features aren't really there in Zope 3 right now, and I don't miss it... much. Unlike the myriad of potential audiences and uses that one could get out of a stock Zope 2 configuration, Zope 3 remains focused on developers at this point. I think that's a good thing. Even if one were to ignore the zope.app package, which provides most of the Zope application server functionality, there is a very impressive base. If someone wanted to make a version that had through-the-web classes and code, many of the base components that are necessary are there. If someone wanted to do a prototype based system, they could do it. I believe that with Zope 3 we are going to see more specialized applications released and built that can target different audiences, hopefully without having to make the core Zope systems have to cater to everyone.

Work has been done to allow Python code to exist in the ZODB as modules in Zope 3, with tools to allow checking code in and out from the file system. It's still incomplete, it seems, and I don't know if it's going anywhere. The file system checkin/checkout would allow one to use traditional tools (editors, version control) while also being able to take advantage of having code running in an image - dynamic updates, distribution (using tools like ZEO or Zope Replication Services), and so on.

Why might this be nice? Watch this QuickTime movie of Seaside halos. Seaside, a continuations based web framework written in Smalltalk, shows off many of the benefits of running code out of an image - live debugging, instant changes, and more. What's most interesting to me, however, are the "Halos". When they're toggled to be on, many elements of the web page get wrapped in boxes with an extra toolbar that allows live inspection and manipulation. Need to change the code that's rendering the shopping cart? Click on the halo button and you'll be taken directly to the method that's rendering it. All other code seems to be available for manipulation as well.

That's the closest I've been reminded of Zope 1 or 2. A dynamic running object environment on the web with live code in the database engine. The key difference is that Seaside's code editor still fits into how one might expect Smalltalk to behave, providing a web version of the System Browser. And I imagine that the Seaside Code Browser treats all code and objects the same. With Zope, file system code (regular Python code) is naturally off limits. It's also loaded and reloaded differently - updating a Python Script through the web has immediate repercussions. Updating something on the file system requires a manual product refresh or server restart.

Ugh. This is long and rambling and I'm not sure if there's really any point to it all - just some thoughts. Paul's comments reminded me of what we were getting away with in Zope back in the late nineties, before PHP had even started to make much of an impact. Zope 1 and 2 has always been a strange beast, quite unique from all other servers and frameworks. It's been very successful, yet also has a lot of people who hate it. Those of us who have used it for so many years still don't know how to explain it, and may even have two or three very different development styles for it depending on mood and the problem to be solved. Zope 3 is so refreshing - I get to take the core concepts that I've been using since 1997 but get a clean and free architecture to work with. I feel I have more freedom to make Zope 3 "my own". I could implement 'halos' for our content system. Or maybe start offering prototype based content objects. It's wide open.

Yesterday I touched on using Tinderbox as it's become a core part of my Getting Things Done workflow. In particular, I became enamored with the Map View yesterday as I had to plan out a tricky little project. While working in the map view, where I was able to think things out spatially instead of hierarchically, the items I created were still regular Tinderbox items and were wired into my GTD agents. When I went to work on this project today, I was back in hierarchical mode, checking things off as I got them done.

Ryan Holcomb's GTD template uses color to distinguish items as they move through the flow, with these colors being set automatically by the Tinderbox agents - yellow for 'on hold', orange for 'complete', bright green for 'due today', red for 'overdue' (for items marked with a due date). A benefit of this is how it appears in the map view. Even when items are children of items, they show up in a kindof thumbnail view at the current level. Thus today, checking in on the map view of my project yielded this:

This shows me that I still have one item to do for the 'Sharing View' (in the box on the right), and one item on hold in the other box. Very nice.

Over recent weeks, I've been trying (as I often do) to get and stay on top of things. I think I may have found a system that works. I've said this many times before, but this time may be different.

It started a couple of months ago when Kinkless GTD showed up on the scene. Kinkless GTD, also known as KGTD, is an implementation of David Allen's Getting Things Done system as an OmniOutliner document with supporting AppleScripts. It's an excellent showcase for what can be done with a well-scriptable application. OmniOutliner (and the pro version which KGTD uses) is fast, nimble, simple, but still quite powerful. It's been one of the best Mac OS X applications, and one I've owned since the first release.

Kinkless GTD had some great features that made maintaining my list of things to do easy: combination of projects and contexts (projects are a collection of actions, contexts is where those actions take place). Actions can be entered in either the project-oriented or action-oriented view and get synchronized between the two lists (different sections of the outline). Next actions are distinctly highlighted. Archival of completed actions is a single click action, easily hiding old actions and information. All of this, and more, done by taking advantages of some of the simple yet flexible features of OmniOutliner and its scripting dictionary.

At the same time, there were some aspects of KGTD that didn't quite work for me. Upgrading wasn't always easy and it's still in active development. I stopped upgrading after 0.69 (it's at 0.75 as I write this). In order to keep the lists in sync, deletion and completion required using special buttons that could be installed on the toolbar that ran applescripts to make sure the version on the other list was removed or cleared too. Changing the text of an item on one list would cause the old version to re-appear (sometimes twice) after synchronization. There's good reasons for why this happened - the script just copied items from the lists (projects, actions, and 'next actions' - the single next action for any project). It was hard for me to break habit from regular OmniOutliner usage - using native widgets to mark 'done', etc. The number columns in the document started growing rapidly, which cut back on room to dedicate to the actual text of the action. Configuring the toolbar with the new buttons caused all documents to pick up those buttons, whether they're KGTD or not. This is a Mac OS X detail. So while the toolbar made working with the KGTD document easy, it was useless for other OmniOutliner documents.

So while I was really impressed with KGTD, the shortcomings (which are admittedly minor) bothered me enough to start looking at something bigger: Tinderbox.

Tinderbox is a strange beast. I've looked at it before and have downloaded the demo often, but I couldn't quite understand it well enough to justify the cost involved. Tinderbox is really a sort of free-form hierarchical / hypertext database for notes. What attracted me this time is that it too has some strong templates for implementing a system for Getting Things Done - Ryan Holcomb's "GTD Improved" and "GTD Lite" templates, available from the Tinderbox File Exchange.

Tinderbox features prototypes, extensible attributes (that don't eat into column views), Rules and OnAdd actions for automatically setting attributes for children of certain elements, and most importantly - agents. Agents are queries and actions that can be run automatically or manually. For Holcomb's template, they do a lot of what KGTD's synchronization script does - bring items from the project lists to contextual action lists. Unlike KGTD, they're not copies - only aliases. No manual 'sync' action needs to be run - as you mark things off as completed, for example, they disappear from the action lists. Holcomb's template is not under heavy development, so there's no upgrade worries. The way that it works is done using standard Tinderbox features, and it was pretty easy for me to start customizing some behaviors and defining new agents and prototypes without even having read much of the Tinderbox manual. It didn't take long for it to start feeling like my own system.

In Kinkless GTD I stayed in project mode most of the time, in a fashion I was used to working with outliners for task lists. The contexts didn't impact me that much, even though it was KGTD that showed me why the context lists were valuable. With Tinderbox it's easier to separate planning from doing - to go through and plan out the list of next actions, their contexts, notes, etc, and then look at the '@office' agent view and start churning through the list. As that list is churned through and items are marked completed, they disappear from that list. The original item is colored orange so that when seen in other views there's a visual cue that the item is completed. The orange coloring is done by an agent. Other agents modify visual cues based on whether an item is on hold, due soon, or past due.

So this is all really cool. It's a hierarchical active database. The agents, prototypes, and ability to tweak 'OnAdd' actions bring the active parts without need for heavy scripting - just learning the query language. In some ways, this might sound like it's entering into Frontier territory. But Tinderbox is not a development environment. While it uses agents and prototypes, and can house/use AppleScript, it's a very different beast than even old versions of Frontier. Tinderbox is very much focused on maintaining projects, notes, and plans, and has many different ways of viewing and interacting with your data. And today, two weeks after finally purchasing it, I finally used one of its most powerful features: map view.

In map view, you work with visual representations of notes, appearing as boxes with the title displayed. You can add extra adornments (items only seen in map view) to group items together in a non-containment way. You can resize notes and drop other notes inside them to turn those notes into children. And perhaps most importantly, you can easily create links between items in this view. While I'm not a fan of hardcore mind-mapping, I do like the basic concepts. And sometimes it just helps to think visually. With Tinderbox, you can have many different windows open on the same data. So today, I opened one of my projects up into a map view window and started re-thinking what needed to be done in order to get things moving. I'd avoided the map view up until now, even though there's a note in Holcomb's document that says it's great for brainstorming. And boy did that prove right. Normally I think very well in outlines (and thinking rapidly in outline mode is still where OmniOutliner excels over Tinderbox), but I just couldn't do it today. Thumbing back through a small notebook I've been using on recent projects, I came across a mind-map style pair of graphs and thought "maybe that's what I need."

Working in the map view of Tinderbox enabled me to see the items I'd already thought of and I could start playing with them spatially. Using adornments to make visual groupings, I was able to start segmenting off the tasks involved. And before long, I had a document that looked something like the following. Note: The following is a quick example I made up tonight - keep your eyes off my project!

Since this was in my Getting-Things-Done document, all of the items in my new visual plan were still picking up prototypes and attributes and responding to the agents already defined. I initially worried that using map view would work best with separate documents, so I was especially happy to see it work like it did. In any of the outline views, I could still do basic prioritizing by reordering items, and this did not effect the layout on the map.

Outline View

Of course, this is a hierarchical system, and that works even in map view. Notes that contain other notes can appear like this in map view. When zoomed in on any level, you can see a faint grey outline that represents the boundaries on the parent view so you can get an idea of what the thumbnailed view will look like.

Finally, agents can be written to collect interesting items. So while working in map view, properties can be set that affect the agents and you can quickly get another view of your data. This, again, is from my example document. Holcomb's GTD template has plenty of great agents that I've found useful for working in action-mode, wherein its important to just keep an eye on just enough of the road ahead to be prepared while being comfortable knowing that all other plans are in the system and don't have to be worried about.

Agent Example

Overall, I'm impressed with Tinderbox. Being able to sort-of free think in map view and still have all of those items flow into my daily work-management system is great. The application is a bit strange, but manages to be lightweight and flexible. A blank document imposes no structural inhibitions - you can keep a document as flat and boring as you like, without having to "program down" complicated defaults. At the same time, an active user community and file exchange provide a lot of insights into the program. The price is a little steep, but after using it it's starting to feel like a true professional application - that is, an application that really helps me get my work done in a way that I want.

Finally, I'm working on using Tinderbox for managing actions and planning for both my personal and professional life. I'm working on using DEVONthink to organize and manage reference materials - artifacts that need to survive beyond the day to day morass of planning and thinking and working. So far, it's working out pretty well. In the past, I tried using DEVONthink in the planning process as well as reference, and it just didn't fit the bill so well. Separating these systems has taken away the open loop of wondering where to deal with big chunks of text or random note-taking versus where to deal with planning and acting on that plan.

I've been reminded, in some excellent email exchanges, that there is no real site for Zope 3. There is dev.zope.org/Zope3 - the Zope 3 development Wiki, but not much else. I think that it may be due, in part, to the fact that Zope X3.0.0 had the 'X' in the name and wasn't quite taken seriously as a final release? I don't know. But now that Zope 3.1.0 is released and the 'X' (which previously stood for experimental) has been dropped - Zope 3 has truly arrived. However, although there are at least two books available covering Zope 3 development and usage, there isn't much in the way of informative and "quick start" web sites right now. Having not the time to make one myself, nor wishing to request someone else whose time is also limited to do so, I want to just point to a couple of Zope 3 Quick Starts that are showing up.

Benji York's Quick Start

Benji York has started compiling this quick start. It is not based on Zope 3.1, but rather on a pre-release version of Zope 3.2 (due this winter) from Subversion with instructions to check out. Personally, I prefer using release versions which use a more familiar "configure; make; make install" process. But setting up aside, Benji's document is helpful and comprehensive and also showcases some Zope 3.2 features such as zope.formlib.

My personal complaint - or just a note - about Benji's piece is that he takes the world's stupidest example application, Hello World!, and shows a verbose Zope 3 solution for a situation that is quite simple. So people may look at it and go "It takes all of that to do "Hello World!?!?" when that is not the point. The point is showing the various aspects of Zope 3 development - using zope.schema, persistence, and configuration via ZCML.

Zope 3 in 30 Minutes

This one showcases a bookmark application which runs and works in the default Zope skin (but should work in most skins that build on the default layer). Um, don't worry about skins. What's important here is that this is an application with containers and objects and testing. In its current version, the author is short on explanatory text, but it is full of working code. There is a fair amount of ZCML to look at, but what sort of domain specific languages could or should be used for configuration is another debate.

Both of the above are young drafts, but they do provide a couple of decent places for getting started without a book.

Some other resources:

Simple Todo

The Simple Todo Application is a series of posts that I wrote on Industrie Toulouse which has now been properly archived. The series went up to a usable point, but still could have gone farther. I also don't have some screen captures properly re-linked yet and am unsure when I might get around to that. But still! There it is. And I know that this content should be reworked and re-released online soon by another party.

World Cookery

The site supporting Philipp von Weitershausen's book, Web Component Development with Zope 3. The site is likely to expand soon with more useful information. But along with supporting the book, it contains downloads of source code from the book, and also of the code that runs the site.

Z3 Base

A collection of Zope 3 related software. There is a great trove in the subversion archives. This is where Five came from. Five allows Zope 3 software to be used in Zope 2. It also houses sqlos which is a SQLObject Support package for Zope 3, which allows SQLObject objects to fit into the greater Zope 3 application architecture. I don't know how well that's been maintained, but it is definitely an interesting project to look at.

Z3ECM - Content Management for Zope 3

This site has blogs, documentation, proposals, movies, prototypes, and more, for building a high class content management platform on top of the Zope 3 framework. There is a lot of information and ideas floating around here, and the animations are very impressive.

At long last, I have finally gotten around to archiving my old weblog, Industrie Toulouse. It is moved to euc.cx/toulouse, aka Toulouse Archives. This houses my blogging activity from 2002 to early 2005. Not all images may work, and there is no search, but all other links seem to work fine.

Griddle Noise will continue to be the home for new posts and surliness for the time being.

Sincerely,

A post came to my attention today covering "Best of breed controllers for MVC Web Frameworks." It covers, again, toolkits like Ruby On Rails, Django, CherryPy, and so on. The author initially treated this as a recent phenomenon of the last few years, but he was quickly corrected by Philip Eby in the first comment.

Publishing objects on a nice URL is nothing new, and one of the best toolkits to do this sprang up in 1996 - Bobo, aka, The Python Object Publisher. In the comments, some of the legends and myths are covered and dispelled concerning Bobo's origins. My former boss and all-around-great-guy Paul Everitt mentions in the comments (and on his on weblog):

I remember thinking last year, after all this time, most packages haven’t caught up to what Jim had on his laptop after the plane ride back. So many systems seem more like the “Gateway” in CGI, than systems that adopt the object model of the web.

A few days ago I started writing a post on my own about my experiences with this back in the mid-nineties. Now that the history of Bobo has been covered elsewhere, I can just tell of my experiences with it prior to my joining Digital Creations in mid 1997.

The problem of "numerous web frameworks" is nothing new for Python. In 1996 I had crafted my own, taking some (very light) inspiration from the great WebObjects. I was aware of this tool "Bobo" but I didn't quite grasp its significance at the time. When I published the first version of my tool, netstijl, to the Python web-sig, Paul responded with a "yeah, we've done that too" reply.

A few months later, at a rarely-paying job high on the east benches of Salt Lake City, I was growing frustrated with an application I was working on. My responsibility was writing a socket server that queried closed-caption text feed archives from television, and returning responses to a Java Applet Client which would show the results and also pull down still images associated with those results. This was before server side Java, RMI, etc. My part of the code was written in Python, and was a good model-controller separation. But the Java client development was coming along atrociously slow. So sometime in early spring, 1997, I decided one day that I was going to turn my work into a web application and not care about the applet.

I turned to Bobo for my work. By the end of the day, the conversion was done. I worked on a new set of controllers for a few hours in the morning, had lunch, and spent the rest of the day making the HTML look pretty.

One Day was all that took.

Over the next couple of weeks, new features were added. The biggest of these was the "advanced search" which used HTML Frames and "multipart/x-mixed-replace" to push data to the client, such as the images that were shown next to the search results. All of this on my model from the socket server implementation, but now published via Bobo and my controllers and templates. In short time, I had done what we were having trouble accomplishing with the Java Applet, but now using just HTML, Bobo and Python 1.3 and 1.4.

Ever since that spring, I have been using that same basic toolkit. In September 1997 I joined Digital Creations as 'Principia 1.0' was being developed, and worked applications built on Principia and Zope (which was the open source Principia) there for the next four years before coming back to Utah to be poor (wink) as both on my own and with a couple of different companies I've continued to develop and deploy Zope based applications. I've done just about every different kind of Zope development imaginable. DTML Scripting, ZClasses (Briefly), ZPatterns (a Philip Eby invention), CMF, scripts and templates over RDBMS based data (often still published along a readable URL!), 'Product based', building our own frameworks over Zope, and through gateways to our own data management frameworks that had little knowledge of Zope but could communicate through a new model layer and take advantage of Zope for managing not only views and web publishing, but for managing transactions.

Other things that made Bobo cool were some of the little features that you didn't necessarily need, but were useful when you did - security and transactions. Security has always been a feature of Bobo, Principia, and Zope. Bobo also came with a transaction manager that - to this day - starts a transaction at the beginning of a request and commits if the response is successful and aborts if there's an error. With it, I never had to worry about half-written data (so long as I was using a persistent database manager), or data being written in one database and not the other... Unless of course that database was pre-InnoDB MySQL or an LDAP directory (and I even made a couple of valiant efforts to do transactional LDAP writing, at least within Zope. They never quite worked properly though).

I remember coming across this feature in 1997, before I joined Digital Creations, and how happy it made me. Web development was still very tough at the time, and I still bore many scars from CGI programming where I'd often have half-written data before an exception happened and I didn't get a full file made. When I started using BoboPOS, the persistence system that became the ZODB (Zope Object Database), such problems magically disappeared!

Bobo was even doing REST-ish communications back in the day. There was a 'bobo Client' library which basically allowed for RPC / Corba-ish communication, but using HTTP as the backbone. Again, this was long before XML, XML-RPC, SOAP, webMethods, ICE, and so on. I used it many times to script batch updates that I needed to make to an application server. I don't have any code in front of me, but basically it looked like this:

john = boboclient.Object('http://www.example.com/people/johndoe')
john.setJobTitle(jobtitle='Vice President, Software Engineering')

On the 'Object(...)' call, you could set up authentication parameters, what HTTP method to use, etc. And basically all that the bobo client stuff would do is turn the method call, 'setJobTitle' into a GET or POST to the url with the method name attached. As a GET, it would look like this:

http://www.example.com/people/johndoe/setJobTitle?jobtitle=Vice%20President....

Bobo on the server would then turn that into a traversal path and method call, like:

root()['people']['johndoe'].setJobTitle(jobtitle='Vice President....')

And all that John Doe's class might have to do is something like this:

class Person(...):
    def setJobTitle(self, jobtitle):
        """ Update the job title for this person """
        self._jobTitle = jobtitle

That's a very simple example, but not really that far of a cry from what I still do today. In fact, using Ajax and Zope 3 has been extremely easy because of things like this, and I have some MochiKit based helpers that let me construct Javascript client objects to server side 'browser views' (which are also being controllers in this case) to do operations that are very similar to the simple example above.

It's now been about nine years since I started programming Python. For almost all of that time, I've developed for the web. And for all but the first seven or eight moths, I've used Bobo or one of its descendants. Now I'm using Zope 3, which is the completely re-architected Zope framework and application server. And I'm amazed at what we've done in the past month and a half on it. We've watched so many frameworks and concepts come and go, go into vogue and then out. All this time, Bobo has been sitting at the heart of Zope and even larger systems built on top of Zope (CMF, Plone, CPS, to name a few), doing the simple job of publishing objects on the web. How complex and overburdened those objects may have gotten in Zope 2 is a different debate. For as long as Django has been alive as an internal project before bursting its way onto the public scene recently, Zope and the CMF have been quietly and steadily managing many large television and newspaper sites around the world.

On fads... I remember fondly when it seemed of utmost importance to make Perl work in or with Zope. And then it was Java - "yeah, but could it work with J2EE / JSP?" and early PHP. But I believe that the core Bobo ideas are still extremely valid. Because although Zope 3 is a new and clean architecture compared to the jungle of Zope 2, the principal ideas (and even some of the very core code) seem to have changed very little, even as other trends have come along for the rest of the Python community to try to duplicate... I never understood servlets in Java. And the Python implementations were even harder to understand. Bobo... I got it.

Throughout the years, I've been fascinated by compound document technology. I still pine for OpenDoc. Most of my work has been on the web, and every few jobs I seem to come into doing content management and every time I do, it's seldom as simple as "entering some text on a page".

I have proposals, diagrams, and outlines going back to my previous major employer. But most of my memory of getting to really deal with the problem stems from my time when I came back to Utah. I continued to use the Zope platforms through various stages of minor and self employment before landing in my current situation (where Zope is still used). Three generations of working with building compound documents come to mind.

Generation 1 - CMF Based, OpenDoc Influenced, Dynamic Structure

We had a customer that provided us with the basic structural layout of the site that they needed to manage. They needed to be able to add new pages, and those pages needed to include images, lists, etc, as well as text. I decided this was a great time to try my compound document ideas. Taking a page from OpenDoc, I decided that a document was a very basic element that was just composed of parts. In fact, the Document basically held a 'root container' part. Container parts held other parts, such as text, images, and so on. Parts could, theoretically, be quite complex. Container parts would manage sub-part creation, access, and ordering. In implementation, I believe they derived from Zope's ObjectManager base class(es). For this customer, they could create a new document and then just add parts to it. They could only edit one part at a time, but its editor would display in-place with the rest of the document. Parts could also be re-ordered. Images could be dropped in between paragraphs and told to flow on the right, left, or center. The content editor did not have to type in any code to refer to images. Navigation to other documents was also managed by another drop-in part. Most folders have index pages that contain a navigation part (listing sibling and one-folder-down content) and a text part to give a folder table of contents some meaning.

This was a pretty good system. Unfortunately, it was developed at a time when the CMF (Content Management Framework) had stalled out, it seemed, with releases. It's since picked up. But we had basically a munged version on our hands.

We were able to take advantage of much of what the CMF offered - indexing, workflow, user management, and a nice separation of UI code from content (which can sometimes be tricky with Zope 2 development). I still felt very frustrated with the experience though. It felt like I had to do too much work to get the CMF to get "out of the way" where I needed it to in order to satisfy my customer's needs. While the CMF paved the way for Zope 3 by providing some much needed new concepts to Zope development - replaceable components, service/policy objects that were easy to look up and use while also being configurable locally, the concept of "skins" to build layered and swappable user interfaces, the ability for many objects to participate on behalf of a basic content object so that a document could just be a document - it always felt quite heavy to me. Starting out with it from 'scratch' without the default content management application implementation was overwhelming. Hiding much of the default implementation was tedious. And to me - it's actually quite a nice system. It was often just too heavy for our customers budgets or our own time constraints.

But that system that I built - which was for a non profit literacy group - is still in use today and has more content than I ever envisioned. My own struggles with getting my implementation going put aside, it's not a bad system and I'm glad that it still sees use long after most of those involved with the launch of the project have since moved on. Later, when other small companies and organizations came to us with need for simple content management, I felt that a different approach was needed.

Generation 2 - Pure Zope, 'CMF Lite' in a way, still OpenDoc based, fixed structure

For the second generation it was decided that we needed something that was more directly under our control and understanding. It didn't need to solve all problems at first, but it should be flexible for growth. I had recently gotten a lot of benefit of implementing and deploying D4 (Dynamic, Declarative, Data-Driven) architectures on Zope, whereby model designs were used to dynamically build user interfaces, provide validation, and so on (things that are increasingly taken for granted today, but seemed to take a long time to come around to). Most of our D4 solutions at that point were against relational databases, with one application tied to LDAP. I wanted to take what we had learned and applying to these content documents. We had customers coming to us now that didn't have such free form content, but instead had pages that held more common structure. I wanted it to be easy to define this content and store it in the ZODB while having my code avoid direct use of the Zope 2 framework if possible. The new solution consisted of:

Parts

Still the chunks that documents were made of - parts were separate but complete small content items. A part could be text, an image, a headline, or any kind of structured record (contact information, links and descriptions about last years festival, etc).

Policies

There were policies for folders and for documents. This is where a lot of the pluggable and extensible architecture came from. Folder policies would set things like "what is the default document to show for this folder?" or "what kind of documents may be added here, and how many?". They also provided APIs for listing and organizing content. An events folder might plug in an event listing policy that would sort events on their starting time. Like CMF Tools, policies had interfaces and common names, and you could replace them at will. Unlike CMF Tools, folder policies were kept in a special containing object on folders. There were no global site settings (although I've been wanting to add this feature). Documents got their policies out of a document policy manager.

Document Schema

A document schema basically put the parts together that would make up a particular document type. A 'contact information' page might have a text part, a picture part, and an address part. The schema was just another policy on the document type. The schema policy was also responsible for getting form data out, validating it, and saving it.

Simple Events

The individual policies for a folder or document could respond to events that were passed on from the containing policy manager. The Schema policy would respond to a 'document added' event and populate the document with empty parts based on the schema definition.

Skins, and composable interfaces

Another common policy was the 'views policy', which said which template/view objects to use. These objects were given common names, like 'edit', and could be configured differently for different skins. The skins tool we used was written by Chris McDonough and was basically lifted directly out of the CMF. Skins are built into the CMF and Zope 3. A skin is composed of layers. View objects are assigned to these different layers, and basically the view object is looked up in each layer until it is found. Thus you can have both an 'admin' skin and a 'public' skin. In each case, objects are traversed to directly (this is Zope after all). But how they'll be rendered and controlled depends on the skin that's placed on top of them. It's not a great explanation, I know, but it's a pretty significant concept.

Basically, what I had managed to do was mimic a lot of functionality that I liked about Zope 3 (which was still deep in pre-release development at that time). It's a successful system - I was even able to integrate it into our e-commerce solution which is a very different styled application. It even reuses a fair amount of code from the first generation, since many of the internals (documents and parts) were CMF agnostic.

The big difference between the second generation and the first was the move away from free-form documents. The system could allow them with some other policies plugged in. But the advantage of the rigid documents is that we could design certain pages more easily because we knew exactly what slots were going to be on there, and what slots might be on there. The customer still has flexibility with the content that they can enter, but the pages are crisper than they might be if all we could to to render it is loop over the parts and tell them to draw themselves.

Generation 3 - Zope 3 based. No OpenDoc heritage. Lightweight, in less time

The main reason that I'm writing this entry is that I didn't realize until today that I had basically redone much of the work from Generation 2 in a customer application we're developing in Zope 3. The concept of parts are gone. Generation 2 schema was defined through the web. Generation 3 uses Zope 3 concepts. Instead of parts, Fields are used - even for complex objects. A page is defined through its Interface, a very strong Zope 3 concept. And that interface's schema is then used to build edit forms and has provided other exciting benefits as well. For example, I can take any of my content objects and find all of their fields that have text in them that can be used for indexing. The field in question might be simple - just a line or block of text. Or it might be complex - like a table with a title, a caption, and rows of cells. In fact - it's this table field that I wrote that made me realize that I had already made a more complicated 'part' than I had ever made for either of the previous generations. And it wasn't until this evening as I was showing progress to my boss that I realized that this set of work - much of it done just in the last three work days - pretty much blows away the 2nd generation.

Zope 3 delivers a lot of functionality that I can then build my own application stack on top of. It's very rich, like the CMF. But at the same time, it doesn't feel like it gets in the way nearly as much. It's still a rich and complex system - you can't really write a "20 minute wiki demo and video from scratch" on top of it without also having to explain the default user interface or skin configuration or URL namespaces. But at the same time, many of these concepts are (relatively) easy to deploy when setting up a new application. This one that I'm working on I've been at for less than a week now. And it's pretty much caught up to our first 'generation 2' site which took a few weeks, mostly because the Generation 2 compound document system was being written along side with it. This 3rd generation implementation is still missing some key elements that I'd like to have implemented, but I don't think they'll be that big of a concern. And the UI is much more impressive than anything we've done in the past. Thanks to MochiKit and much of the work done for the customer prior to this, we have a nice dynamic / AJAX based UI with sortable folder content listings (in javascript). Drag and drop content reordering. Inline renaming and retitling. In the past couple of days I've written a complex Table field for Zope 3 and gave it an interface that allows for dynamic Javascript editing (adding / removing rows - the column definitions are fixed). In less than an hour today I wrote a Textile field (basically a plain descendent of a text field), whose widget uses AJAX to allow for in page previewing of the Textile output.

And all of this is within an architecture that I understand. Granted - there are a lot of Zope 3 concepts that one has to learn, and there are many that I haven't learned yet. But even my own "Generation 2" system I had trouble understanding in comparison to what we've done on Zope 3. The deep dark caverns of the "Generation 2" schema system - which is quite powerful - are scary places for even me, the author, to go.

What impresses me most about Zope 3 is how easy it is to build ones own layer of toolkits / frameworks on top if it all. A common stack we have, which came together within the first week or two of using Zope 3, is this:

• Customer Specific - skins, components, utilities specific to this customers needs. May build on the next layer.


• Similar Customers - content and utility components that are common across a certain set of customers, which is the situation we're in right now with this first batch of Zope 3 work.


• Our CMS / Framework - Provides the management UI, common components, configurations, etc, that are common across many sites that we expect to build.


• zope.app - The Zope 3 application framework. Provides many of the basic components for a full Zope 3 application - containers, utilities, apis, etc. A pretty beefy layer.


• Core Zope 3 systems (zope.interface, zope.schema, zope.component, persistence, transaction, etc). These are the basic layers underneath Zope. One could, in theory, use many of these outside of 'zope.app' to build a different application server that did not depend on or provide much of what zope.app requires. There is a 'bobo' branch, for example, which builds on just a few of these core things. Using the zope.bobo, one could in theory write a 20 minute Wiki or 20 minute todo-list in a manner similar to what Rails or TurboGears shows off.

Along with all of this, there are other python libraries, including internal ones, and Zope 3 based libraries like Hurry, used as needed.

What surprises me most of all is how quickly this latest solution came together, almost without my seeing it. Nice.

A cool photoset on flickr: "Apple's Front Pages Along The Years". Going back to 1996, it includes a couple of captures from the Amelio years. I remember that site. Apple at that time was quite large and sprawling and many sites were off on their own subdomain with their own look and feel. There was a trove of information out there on projects like the Dylan Programming Language as well as OpenDoc, Cyberdog, and a browser plug-in called "HotSauce" which could fly through MCF - Meta Content Format - data in 3D. MCF would go on to eventually become RDF.

The photoset includes NeXT's page at the time of the Apple-NeXT merger, which was a very exciting day for me. That merger literally set fire to the company. All of the excess was trimmed, and eventually Apple started the "Think Different" campaign and launched a new web site in 1997. By the time the iMac was announced (with the campaign of "Pro. Go. Whoa."), Apple's home page has stayed pretty much the same in the time since then. There's the big picture, the "hot news" scroller, and now four cells underneath (there used to be just three). It's interesting to see such consistency on a major corporate web site.