Fun with Declarative Python. Er, "Domain Specific Languages". Er, whatever.

This weekend I had the luxury of free time for what feels like the first time in a good while. After some discussions on the Zope 3 mailing lists this weekend I decided that I wanted to try some new development options in the application I’m working on. Normally I do my development on a server at the office and sometimes work from home via SSH, but I wanted to set up a good local copy on my desktop so I could try working more in the TextMate editor, and keep the number of connections to work to a minimum.

This led to me thinking about how to get all of the supporting libraries for my app checked out from our own repository and a couple of third party subversion repositories. A goal of mine is to start getting our release management under control, so this line of thought led to me starting work on a kind of packaging tool. I doubt it will ever be public – it’s not meant to compete with eggs or zpkg. I needed something simple so I could check out these five packages.

I first thought about how I was going to define the map – what needs to be checked out, from where, to where, what revision / tag, etc. Should it be YAML? XML? ZConfig? My own syntax? I looked at YAML first, but it looks like to use it best it needs to go through C bindings. I thought about XML, but I’ve never gotten so insanely comfortable with XML parsing that I could come up with something quickly. As I looked at the other options (including YAML again), the translation layer between some text file in some language and native (and potentially rich) Python types seemed too big, at least for me to want to deal with at the time.

Something that’s been in my head a lot lately is the content of the Snakes and Rubies presentations about Django and Rails. One thing that David Heinemeier Hansson mentions a lot about the design of Ruby on Rails is that Ruby makes it easy to do “domain specific languages” within ruby, and Rails takes advantage of that by using a fairly natural syntax in elements like Active Record models that creates a lot of methods on the fly. I believe that in Ruby you can fairly easily interact with the class while inside of the class. In Python, this isn’t that easy. Well, it’s not easily exposed. But it can be done, albeit with tricks. I’ve seen this in Zope 3 – there’s been a move away from this style of programming:

  class IFooContained(IContained):
      __parent__ = zope.schema.Field(
          constraint=ContainerTypesConstraint(IFooLib))

  class Foo(object):
      __implements__ = (IFoo, IFooContained)
      __used_for__ = IBar

To this style:

  class IFooContained(IContained):
      containers(IFooLib)

  class Foo(object):
      implements(IFoo, IFooContained)
      adapts(IBar)

And with metaclasses, decorators, descriptors, and so on, it’s certainly possible to do more domain specific language style things within Python while not straying far from being “Pythonic” (for all of the mystical meanings of that word).

So I decided to give it a try: using the Zope 3 component architecture (but none of the Zope 3 app server or web specific parts), could I write something that let me define a release map in Python that was largely declarative but showed structure? Could I still have it validate? Could I have it turn certain keyword arguments from a regular string into a templated string without the map being filled with things like template("${src}/bar"). Could I have those templates allow for namespaces so I could easily access environment variables like ${env:CVSROOT} ? It turns out, the answer is yes. It took me a while to get it all together, including a couple of start-overs to simplify things. I wrote a Python file with a couple of classes / declarations that was my model for what I wanted and worked from there. It changed slightly throughout the day and night, but at the end of it, I have something like this, and it’s working:

from mypkg.api import *

class DemoApp(Registrations):
    zope = Subversion('svn://svn.zope.org/repos/main')[
        source('zc.catalog',
            repository='Sandbox/zc/catalog',
            target='zc/catalog',
            revision='37377',
            #post=command(tools.touch, 'zc/__init__.py'),
            ),
        ]

    codespeak = Subversion('http://codespeak.net/svn/z3')[
        source('hurry',
            repository='hurry/trunk/src/hurry',
            target='hurry',
            revision='18433',
            ),
        ]

    personal = CVS('${env:CVSROOT}')[
        source('demolib', repository='Support/demolib', target='demolib'),
        source('demoapp', repository='Applications/demoapp', target='demo'),
        ]

if __name__ == '__main__':
    main(DemoApp)

There are a few things going on behind the scenes to support this so that even a lot of the front end code in main(registrations) is really quite straightforward. A factory method style design pattern combined with use of zope.interface and zope.schema to validate and/or convert values seen above into what’s wanted inside the system (like turning ’${env:CVSROOT}’ into a templated string) takes care of the parsing problem while helping keep the map code clean. A little bit of metaclass-ish intelligence turns Registrations based classes into easily accessible data structures. A couple of simple adapters help preserve an easy interface on how some items find each other or get formatted, and keep the registration information separate from the objects that are ultimately created from the registrations, and when the system is finished a few more will provide the full command line interface options and the relationship between those commands and what’s produced by the map above.

My thoughts after this? I don’t know. The project is half done and may still change. In general, I like it. Even without falling back on meta-classes, I’m trying to apply this pattern in many places, to use intelligent data structures, contracts, and collaborating components to do more things declaratively where applicable. The Zope Component Architecture makes this very nice – define a contract/interface like IRepository, and then start providing adapters for it to provide things like command line formatting, log querying, etc; or to parse or format schema fields for a plain file representation.