apk zipalign

By the way, the crisis of faith mentioned in my last post may be minor. There is a lot about Zope 3 that I like. There’s a lot about classic Zope that I’ve liked. But I feel frustrated. That frustration is felt on more than one level too. I’ll get to the positives soon enough.

Again, I’ll mention that I’m not necessarily a good person. I haven’t contributed any actual code for more than a couple of years now. I don’t even run checkouts from Subversion. I must deliver results to my customers on as much trusted, supported, and released software as possible. I try to keep up on the general discussions on the mailing lists, but can only keep up on a few usually.

But I do have a passion for this. I want beauty, I want happiness, I want Python, I want Zope. I want others to want this and have it. But if I’m having frustrations, what are they going to feel?

Gripe 1, Message message message. There is no message.

Well, if there’s a message, good luck finding it. Now, a lot of people say “Python has a lot of web frameworks”. Well, I wouldn’t call every Python programmer’s one-off a real “framework”. There are a lot of options. But there are only two others that have a lot of buzz and momentum behind them, are usable today as complete options, and look like they might actually make it to a 1.0 release and beyond. Let’s look at what they offer (keeping in mind of course that it may change after the time of this writing):

TurboGears has a nice site with a narrative text and clear links to what’s important. You can quickly learn what Turbogears is about, and can quickly see tutorials that deliver something usable and understandable. I think what I like best about the current version of the TurboGears site is that the text is readable. It’s prose, but full of links. It’s not just bullet points (we’ll get to this in a minute).

Django also has a great site. The main focus is on some great introductory text (again, with links) about where Django came from. Elsewhere on the page, in nicely and subtly separated section, are links and text (with links!) about the essentials of the framework, documentation, showcase sites, and weblog entries about the project itself. The weblog side is small (title, description only). It shows that there is movement and momentum in the project, but it doesn’t overwhelm the page as the central piece of information. The newspaper mentality of the core Django developers shows through here. Like the front page of any good newspaper or news site (emphasis on the ‘good’ part), there are headlines and columns and information that opens up into inner pages, and it all seems pretty appropriately weighted.

And while it’s not a Python framework, a quick visit to the framework that has made the dynamic language set in vogue again, Ruby on Rails. It’s opinionated. It’s got big buttons – “Get Excited” – promotional screencasts, presentations, and more; “Get Started” – the download link (big clear ‘1.0’ with a release date underneath the big shiny button); “Get Better” – API references, books, documentation, etc, for both Rails and the Ruby language; “Get Involved” – community resources.

And what about Zope 3? Zope.org does have a nice introductory paragraph. Personally, I find it a little hard to focus on with a noisy lists and lists and lists of links design. Beyond the introductory paragraph, there’s not much real text on the page. “Zope 3” is buried down in the “Developers” section in the list of links on the left. On a 1024×768 screen, it may or may not even make it “above the fold”.

The Zope 3 page is full of many sins. It’s basically the development wiki for the project. Quick – can you spot the download link? Visit the ComponentArchitectureOverview page. Is it up to date with how things have worked since Zope 3.1? Back to the front page – is there anything on there your eyes are actually happy to land on and spend a few minutes soaking in?

The sites for TurboGears, Django, and Ruby on Rails all have wikis, but they’re not the prominent interface. They’re the backup resource. I believe they also all use Trac. The Zope 3 project is older than Trac. But I often wish it would find a way to move to it. Trac’s unification of subversion browsing, bug tracking, milestone managing, and more is a great feature. I love the turbogears roadmap which is just a Trac generated page. “what’s going on with the 0.9 release? Oh, there it is.” The Zope 3 Roadmap? – Nothing. It’s a good attempt at providing useful information.

Some systems provide proposal tracking, where proposals are documents with metadata that are attached to a milestone and release. Granted, this information is only as useful as people keep it up to date. But man, I wish I knew what is coming in Zope 3.3, or might be coming, and the status of it. A wiki can’t get me that information. It’s a terrible interface for information like that. How can I get excited? Excited enough to actually get involved and contribute? Excited enough to say “well, Zope 3.2 didn’t quite do what I wanted, but if that new thing is coming in 3.3 and it will be released this summer, I’ll definitely come back and check it out then.”

But can you get any of that information out of the Zope 3 Proposals page? No. What about the zope 3 subprojects page? Nope. What about the projects that have been discussed (or sometimes just mentioned in passing) on the mailing lists? And why should they go in the Wiki anyways? It’s hard to navigate. It’s hard to query. It’s hard to make information stand out. It’s hard to group associated tickets, subversion branches, etc in right with it.

Of course, I’m biased against wikis in general. With very few exceptions, I’m screaming and cursing within the first few links into any wiki. And contributing / editing is just as bad, as most of the “structured texts” of the world tend to really bite. The question marks that show up unwanted in text, the FunNyCapitAliZation (especially of words like ZoPe and PyThon) that can show up makes me cringe, and makes any text very hard to read as my brain has to stop and take the word apart and put it back together. Augh. Frustrating.

The thing is: the whole Zope 3 community knows this. There’s no good web site that gets people excited, shows motion, offers easy and obvious download(s), offers introductory tutorials, or takes a stand and says “Zope 3 Kicks Ass”.

And, well, there is no message. There’s no stand. Debates are still going on about whether to give Zope a new name. Whether to use silly release/code names (a terrible concept, I think). Whether Zope 3 means the core set of technologies, whether it means the full stack and half assed content pieces that ship with the current Zope 3 download (which basically provides the “Zope 3 Application Server”), or whether it means something on par with Zope 2 and offers a future for Zope 2 and Zope 3 as we may (or may not) know them today.

So no one can say “this is what Zope means”. They don’t know. I don’t know. And we can’t agree. Zope 2 offered so many ways of working with it that it was hard to agree. “You can develop your web site like this… Or you could do it like this… Or you could do it like this…”. Not that there is anything wrong with that kind of flexibility. But it’s ultimately too many features, too many options, too many different ways of doing things that aren’t compatible. And we suffer from this in Zope 3 already. Not as badly as in Zope 2. But … I don’t know. I really don’t.

I don’t know why whenever I start a new project / application I’m not excited. I always feel like I’m having to copy / load / configure / tweak / copy quite a bit, still, to get started. It doesn’t take that long, and I’m sure I can extract it out into a usable starting kit framework. But even for me, after all of these months working with Zope 3 and years of working with Bobo/Principia/Zope 1 and 2, it’s hard knowing the best way to start. There are “easy” features we could use. For immediate results, we have a couple of different options. But none of them grow easily into a real application.

Having those is a distraction. There’s a user set that I’m sure they’re useful for. But as I mentioned in the last post, I’d love not only the core simple concepts of Bobo (Python object publishing). I’d love one that was stripped of these distractions, but was otherwise Zope as we know it: easy container/containment relationships. Pure persistent Python content objects. But most of all, it’d be one that could be installed and used with a tutorial that didn’t have to spend time explaining “this ZMI thing” or “that i18n file add menu item”. The slate would be blank, the possibilities wide open. Concepts that don’t apply to the situation don’t need to be learned. For me, things that don’t concern me don’t need to be pushed out of the way. With this tool, you’d still be signing up for certain aspects of the Zope experience: the object database (or maybe a really well integrated object-relational mapper), etc. But you wouldn’t be signing up for the bigger set of tools that build on that. Put that in another release.

And a message. We need a message. We need to prove that message in code examples, tutorials, etc. The fact that it seems to be so hard to even come up with this message is not good. And after a night of wrestling with decoding ZCML directives and vocabulary lookup and binding to write a test to prove (and then fix) a bug in my code – many many many many lines of setup support in a test harness (setup support that acts nothing like it does when used in the application environment). The bug fix? 3 lines of Python plus 1 line of adapter registration. That seemed very disproportionate. And as I grumpily sat at work the next day, the lines from David Heinemeier Hansson’s presentation about Ruby on Rails at Snakes and Rubies kept running through my mind (especially as I started reading their PDF book, Getting Real): Beauty leads to Happiness, Happiness leads to Productivity. I’ve spent just enough time wrestling with things that I shouldn’t this past week for that to be very appealing.

For the first time since spring of 1997, I find myself having a crisis of faith regarding Zope. I’m not sure it remembers where it came from. I’m not sure it knows its own future. It’s suffering from a crazy identity crisis. And there seems to be a core effort to take it in directions that I really wish it wouldn’t go. Is it a good tool? Oh hell yeah. Ever since I wrote my first serious Bobo web application, spending a spring day salvaging a nowhere-going Python socket server / Java client application by putting a web interface on it, I’ve made my living off of Zope and the technologies and systems that preceded it.

Back in 1996, if not earlier, Jim Fulton wrote ‘Bobo’. The legend goes that right after being hired by Digital Creations, he was flown out to give a talk about Python CGI / web programming, which he had never really done. Jim is a very smart man, and the legend (as I remember hearing it) is that he found CGI programming to be a terrible experience. He sketched out the concepts for Bobo on the flight back, and changed many lives forever.

At the time, CGI was the dominant web programming way. FastCGI and other tools were available to work around CGI’s shortcomings, but working with any of them required changing core code. One of the concepts of Bobo was to abstract out the “object publishing” process: Bobo dealt with the protocol/interface specifics. You just wrote regular Python code following a couple of simple rules, and you had a web application. Bobo was, in essence, an ORB (Object Request Broker). It turned an HTTP Request into an object call, typically following the path based on the URL:

    /greyhounds/betty_joan/editForm
       |
       |
       V
    greyhounds['betty_joan'].editForm()

For example. You could customize traversal on any object along the path, and in essence get any kind of system you want. The default behavior was just to follow Python attribute or ‘getitem’ access. To just start getting something up on the web, you could just use ‘Bobo’ and what you already had, more or less.

Bobo also took care of another problem that plagued CGI programmers. I started doing Python CGI programming not long after picking up the language in 1996, and I found that I spent more of my time trying to get data out of the request and getting it ready to pass into my program than I did doing my little programs themselves. It was very frustrating. Most of the errors would occur in the input / output phases of the CGI scripts I was writing. If I had a field like ‘races_won’ that I expected to come in as an integer, I had to convert it myself (at that time in Python, one had to use string.atoi(), int() wouldn’t parse strings), or deal with it not being in the request, and so on. Multiply this up for larger forms, and it was quite the annoyance. Form libraries didn’t really exist at the time either. So I had a script expecting an integer to come in, and I had all this code in front of it to make sure that (a) something came in, and (b) it was an integer. It wasn’t a hard thing to do, just frustrating. Since Bobo was about turning web requests into Python object calls, it had a way of marshalling the data in the form. With Bobo, I could have races_won:int as the name of a text field, and have a form action of ‘setRacesWon’. On a GET, the URL and resulting object call might look like this:

    /greyhounds/betty_joan/setRacesWon?races_won:int=8
      |
      |
      V
    greyhounds['betty_joan'].setRacesWon(races_won=8)

This was more impressive with larger inputs, like lists and (later on) records. And especially impressive when stacked up, so preferred_powerball_numbers:list:int would always bring in a list, whether there was one value submitted or many. This was all to shorten the space between the web and your own Python code.

Bobo also featured an easy entry point. Actually, it was a little tricky how you’d use the cgi module publisher (you’d basically do a symbolic link in your CGI directory to cgi_module_publisher, which was the thin wrapper that would publish your application by looking up the module name (packages didn’t really exist then either) based on the symbolic link). But! In your module, you could provide a couple of entry points: an object named bobo_application which would be the root of the application and where traversal would start from, or a dictionary called web_objects, with names (keys) mapping to objects/methods to publish. That was easy! That was awesome. In the application that I wrote that Spring 1997 day, I did that to make my front end. I already had this application written that searched a selectable list of different Glimpse text indexes and returned objects pointing to the found file, as well as matching line (and surrounding lines) information. All I really had to do was:

1. Make a page with a text field and some checkboxes.


2. Have that call my search method. My search method returned the HTML with the results.


3. Load and display the matched files.

For my situation, it was the view/controller layer straight up. I didn’t need another model layer: I already had a model of custom Python objects that bridged to this third party data.

So I made a DocumentTemplate for the search form, and had a function in my application module that ran the search by going through the selected search targets, based on the input. And I could publish it with the following in the top level of the module:

    def glimpseSearch(searchfor='', engines=[]):
        """ Search for the submitted text across engines """ 
        #.... python code

    def getFile(path=''):
        """ Get the published file from the relative path """ 
        #.... python code

    searchform = DocumentTemplate('searchform.dtml')

    web_objects = {
        'index_html': searchform,
        'search': glimpseSearch,
        'getfile': getFile,
    }

That was about it. I got a quick search form and quick results template together, and spent the first half of my day working on the search method and discarding some of the old socket-server specific code. Then I spent the second half of the day making the HTML look pretty. I still look at that and go “now there’s a starting point that I could teach somebody”. Zope has long lost that easy starting point. There are many many many starting point options. And even Zope 3, which is supposed to be stripped down and simpler in its concepts, cannot be so easily explained.

I was very impressed by that. I had my own “web framework” going at the time. Python’s always made it easy to write your own, be it 1996 or 2006. But after taking this death march project I had been working on and giving it new life by putting it on the web in a matter of hours, I completely abandoned my own foolish ideas and continued to build on Bobo.

At the time, there was a full “back to front” stack available for Bobo, but you could use all of these parts completely on their own with no dependency on the others:

1. bobo, aka the python object publisher: publishing objects on the web, marshalling requests. Oh, and it offered security too. Could be used on its own as the foundation for building a web application. Generally stayed out of your way otherwise.


2. DocumentTemplate: also known as DTML. Before Principia/Zope, DTML was really quite a nice template language. It did not have any Python expressions at all, and thus could not be used for programming. It had a lot of built in features for displaying and formatting data, especially when working with sequences. It was purely, purely, purely about inserting data into a string, and was quite nice to use at the time.


3. BoboPOS: This eventually became the ZODB. This was a persistent Python object storage. When used with Bobo, the Bobo publisher would start a transaction at the beginning of a web request, and commit it at the end if no errors were raised. I only used it lightly, but loved it when I did. Prior to this, I always had the problem (during development) of having code blow up, leaving me with partially written data that I’d have to clean by hand while also fixing my bugs.

You could use these individually, or all together, or in any combination.

The nice thing was – it was so easy! At least, after learning a couple of relatively simple concepts. There was an easy entry point to basically say ‘for this name, call this function’ or ‘start traversing this object’ and start writing a web application. The job of the FCGI Publisher, or CGI Publisher, was to find the name of the module to publish and pass the module into the publisher core, and that would look for these entry points.

Zope is just a Bobo application. Seriously! Zope 2 is still published as a ‘bobo_application’ object in a module. It just happens to be an object that wraps a ZODB instance. But at its heart, this is all it is.

But being able to so easily start writing a dumb simple web application based on other Python code you might have is not so obvious any more. Bobo got swallowed up into Zope 2. The core pieces are theoretically available again in Zope 3 as independently usable pieces, particularly zope.publisher. And there’s been a zope.bobo effort to restore that ease of use. But the effort hasn’t had much work. And I don’t contribute code to much these days.

I’m a bad person in this respect. I just bitch. But I’ve got a job, a dog, and art projects I’m neglecting that I shouldn’t be.

Anyways, I’ve been in discussions lately advocating a product strategy that offers up the core of the Zope 3 architecture. No object database. Just the component architecture (a pretty simple system on its own) and zope.publisher, which is the descendent of Bobo. Something that could be used to make a web application. Or not. But something that was separate, documented, exciting, etc. Something where you could actually do the “20 minute wiki” tutorial without spending the 20 minutes talking about the zope management interface, about skins, about different development options, about zope.conf and site.zcml files.

I’ve also been advocating a “Zope without all of Zope” release. One that has the Zope Object Database (it’s a better solution than relational databases for many web-targeted projects, and no translation layer like an object-relational mapper), and many of the items that are in the Zope 3 download you can get right now. But it wouldn’t have the “Zope Management Interface” that Zope 3 has. It wouldn’t have the half-assed content objects that are like weaker and harder to use and understand versions of their Zope 2 counterparts (things like page templates, python scripts, SQL scripts that are stored in the ZODB). They’re a terrible distraction. They’re hard to explain. I’ve seen very very smart Zope developers get confused about what to do and whether to use those things. And for writing custom applications, they’re completely unnecessary.

There’s a lot that I love in Zope 3. But there are also just enough frustrating things that I’m running into that, combined with the continued debating over whether or not to rename Zope 3 or what Zope 2 features should come in or god knows what, is having me looking at one of the Ruby on Rails mantras longingly:

Beauty Leads To Happiness

I’m still not as happy as I want to be. And I nearly came unglued when confronted by someone (someone I respect too) advocating a single “everything and the kitchen sink” distribution. I don’t care if that’s available. But I just want a couple of pipes and a wrench. If I have to keep running with this analogy by talking about all of the walls I need to knock down in order to understand why the values for the ‘select’ widget aren’t being saved properly, I’m just not sure this is a house I can stay in.

Ruby On Rails sticks to a strong opinion. Django and Turbogears in the Python world do too. They have a message that they all seem to believe in and love.

And I’m starting to realize that there’s never been a single Zope message that everyone has been able to rally behind. Is this why Zope 3 seems timid to even say it exists? It’s such a good architecture! There are some GREAT tools shipping with it (zope.formlib – the best form library I’ve ever worked with and it absolutely kills anything else I’ve seen out in the Python community), and some GREAT tools available for it (if you fish through Subversion repositories. zc.table looks like it’s the equivalent of ‘formlib’ for doing comprehensive and intelligent data grids of many styles, and I hope to take a close look at it very soon now; ‘hurry’ collects many useful little tools. It’s also pretty easy to start writing your own reusable supporting toolkit, so easy that you don’t even realize it’s happening). It has a lot of potential.

But it really feels like Zope doesn’t know where it wants to go. There’s an existing system to support, Zope 2, that has a lot of good work and maturity behind it. There’s a new system that – my gods – has the potential to just kick ass. And I think it could kick more ass if we could start breaking it out and making it simpler, smaller, and easier to use for developers while still providing larger options for those that like the complete stack or the larger feature set of Zope 2.

But… I don’t know which way it’s going. And there are still a lot of frustrating pieces in the code. And while some people are doing valiant work simplifying some core concepts, the meaning of it all if we can’t talk about it is beyond me.

I still have a hard time knowing how best to start a new ‘application’ in Zope 2 or 3. There are too many options, with too much work to go along with each one. And there are plenty of documents out there that will argue one way as being the best, and one will argue a completely different way as being the best.

And I see the quick start tutorials that other projects have and go “that makes sense” and as they mature, I really start to wonder if this Zope boat is the one I want to stay on if it can’t stick to a direction, communicate its intent, and (if it does pick a direction) heads back towards the land of more features that I’ve been struggling to get away from.

I’m getting ready to work on a large Zope 2 system that we’ve developed in house over the last few years. This system is a bit of a framework in its own right, with some inspirations coming from the “Component Architecture” influencing bits of the CMF and also from Zope 3 itself (while Zope 3 was being developed). This internal framework has long since moved from code managed through the web and into Python based products. It’s fairly modular, and we’ve even been able to bridge a couple of very different systems together using some relatively smart architectures.

I was looking at some code I wrote for this system. This code is comparatively recent and lightweight compared to what is in the core. I noticed something as I was looking at it though – I felt really claustrophobic. It’s not that it’s bad Python code or anything, really. But it was just claustrophobic. Methods were too long. Lines were too long! There were too many strings, strangely enough…. There was just something off. And again, I must say that it’s not bad code. It’s not being too complex, too tricky, too verbose, nor is it riddled with shortcuts. There was just a gut feel that didn’t sit quite right with me.

I decided to do a quick comparison against some of the recent code I’ve written for Zope 3 based applications. I decided to check against some of the more hairy “written under deadline” code at that, and I tried to find comparable situations (doing custom adds / edits). The verdict? The Zope 3 targeted code did not feel so claustrophobic! At all.

Now, maybe it has something to do with the older code feeling more alien and the the new code feeling more fresh.

But I think it has to do with something else that I’ve noticed about Zope 3.

I think it’s easier to chunk code up into smaller units of work in Zope 3. I think it’s easier to refactor by doing common refactorings like “Extract Method” and not have to pass as much information around. It’s easier to look up adapters and utilities that provide certain functionality instead of having all of that functionality heaped onto an object through a large inheritance tree.

And it seems like it’s really easy to abstract out common concepts into helpful tools and utilities, which grow into ones own helpful module, toolkit, or framework. At least, it’s easy in comparison to Zope 2.

I certainly know that I’ve been able to easily extract out helpful tools from common actions and update my Zope 3 based code to use those abstractions. And every time I do it, my code gets tighter and more powerful and then taken for granted since we no longer have to worry about it.

For example, a few months ago I worked hard on making a nice “undo” interface. Many ZODB storages can undo actions at the database layer. As I saw applications like GMail provide a nice “Message discarded (Undo Discard)” message for certain actions, I thought “I’d like to provide that, and hey! the ZODB can already undo things for me so I don’t have to track them myself!” So I went to work on providing a nice undo message. I ran into some snags along the way (related to things like how I was using sessions to send page messages to the user), but soon got it all buttoned down nice and neat. I was able to wrap it up in a nice API and handle deletes like this:

        name = zapi.getName(item)
        message="Deleted %s" % name

        del self.context[name]
        undomsg = cmsapi.undomessage(self, message)
        cmsapi.flash(self, undomsg)

After I checked this work in, I also added it to our oldest Zope 3 code, a content management system, in just a couple of lines, and got rid of an annoying Javascript alert box in the process. And then I promptly forgot about it until I was watching a co-worker using the CMS yesterday and saw the “undo delete” message. And watched it work flawlessly, with even the undo being undoable.

The point of this? Zope 3 eases refactoring and restructuring. I’m still amazed at how quickly our content management system came together, and how easily it was to separate out the stack into increasingly vertical levels. We almost didn’t even have to think about it. During post-mortems after major releases, consolidating code that had been copied and pasted into three different customer libraries into a single unit, without breaking our customer’s sites, was almost too easy. And on more than one occasion, just after such a restructuring had happened, a feature request would come along that was easy to apply due to that recent restructuring. “We need to make all of these things cacheable.” “Wow, a week ago that would have been four times the effort had we not boiled all of this stuff down into a core package and updated everyone already.”

And I think that’s the difference I notice between the code I was looking at today from our well-factored Zope 2 efforts compared to our more recent Zope 3 ones: we’ve been able to get rid of a lot of the line noise in our Zope 3 code, and much of the good Zope 3 code (like ‘formlib’) does a good job of clearing out line noise already. Fascinating.

Today I added to my iPod collection with the iPod Shuffle. I actually wanted to get it last night, but a late day at work caused me to miss the store being open. A couple of other important errands got bumped back too. The plan was – get iPod Shuffle, go skiing. Since here is something nice and simple that can (theoretically) be operated with gloves. Or even, perhaps, by just hitting the middle of your chest. At some point soon I’ll have to test this theory for real. But in any case – I’m liking the iPod Shuffle conceptually right now. I have a playlist that I’m listening to quite heavily these days that just barely fits in the shuffle’s space. It has better battery life than my aging 3rd Gen iPod. It probably responds to cold environments better too. And it’s so tiny!

One problem. The Shuffle is geared towards USB 2.0. USB2 is only a feature on very recent Macs (within the last year or two). My main iTunes library is at the office, which is an older machine (dual 867 G4 tower). It’s not that old of a machine. But, no USB2. So I can fill it via USB1.1 at 12MBPS, and holy hell – slow.

I’ve been considering replacing my aging 3rd Gen main iPod with a new video model if I see a good price (refurbished, etc). It’s only been lightweight consideration so far. Today, however, I found out that the video (aka 5th generation) iPods and iPod Nano will not work with Firewire at all. They can charge via firewire, but need USB to be recognized by a computer.

It looks like it might take close to an hour for this 1GB shuffle to fill. Some of the slowness may be related to an extra option I have set, “Encode higher bit rate songs to 128KBPS for this iPod” to squeeze space out of it. But still – this is ridiculously slow.

I can’t imagine trying to fill a 30 or 60 GB iPod over USB 1.1. Yet if I were to get one and still expect to use it with the music library I have on my office machine, that’s the only option I have (short of buying a USB 2 card). This is a strange direction for Apple to take. I mean – I guess it is obvious that USB 2 has equalled or beaten Firewire in many markets and is used heavily in the ‘Wintel’ PC crowd. But still – to not offer support for Firewire at all? If I were still using my old home iMac G3, I’d be furious. There’s no easy upgrade option for it or for many fairly recent (2-5) year old Macs. I find that very strange.

This weekend I had the luxury of free time for what feels like the first time in a good while. After some discussions on the Zope 3 mailing lists this weekend I decided that I wanted to try some new development options in the application I’m working on. Normally I do my development on a server at the office and sometimes work from home via SSH, but I wanted to set up a good local copy on my desktop so I could try working more in the TextMate editor, and keep the number of connections to work to a minimum.

This led to me thinking about how to get all of the supporting libraries for my app checked out from our own repository and a couple of third party subversion repositories. A goal of mine is to start getting our release management under control, so this line of thought led to me starting work on a kind of packaging tool. I doubt it will ever be public – it’s not meant to compete with eggs or zpkg. I needed something simple so I could check out these five packages.

I first thought about how I was going to define the map – what needs to be checked out, from where, to where, what revision / tag, etc. Should it be YAML? XML? ZConfig? My own syntax? I looked at YAML first, but it looks like to use it best it needs to go through C bindings. I thought about XML, but I’ve never gotten so insanely comfortable with XML parsing that I could come up with something quickly. As I looked at the other options (including YAML again), the translation layer between some text file in some language and native (and potentially rich) Python types seemed too big, at least for me to want to deal with at the time.

Something that’s been in my head a lot lately is the content of the Snakes and Rubies presentations about Django and Rails. One thing that David Heinemeier Hansson mentions a lot about the design of Ruby on Rails is that Ruby makes it easy to do “domain specific languages” within ruby, and Rails takes advantage of that by using a fairly natural syntax in elements like Active Record models that creates a lot of methods on the fly. I believe that in Ruby you can fairly easily interact with the class while inside of the class. In Python, this isn’t that easy. Well, it’s not easily exposed. But it can be done, albeit with tricks. I’ve seen this in Zope 3 – there’s been a move away from this style of programming:

  class IFooContained(IContained):
      __parent__ = zope.schema.Field(
          constraint=ContainerTypesConstraint(IFooLib))

  class Foo(object):
      __implements__ = (IFoo, IFooContained)
      __used_for__ = IBar

To this style:

  class IFooContained(IContained):
      containers(IFooLib)

  class Foo(object):
      implements(IFoo, IFooContained)
      adapts(IBar)

And with metaclasses, decorators, descriptors, and so on, it’s certainly possible to do more domain specific language style things within Python while not straying far from being “Pythonic” (for all of the mystical meanings of that word).

So I decided to give it a try: using the Zope 3 component architecture (but none of the Zope 3 app server or web specific parts), could I write something that let me define a release map in Python that was largely declarative but showed structure? Could I still have it validate? Could I have it turn certain keyword arguments from a regular string into a templated string without the map being filled with things like template("${src}/bar"). Could I have those templates allow for namespaces so I could easily access environment variables like ${env:CVSROOT} ? It turns out, the answer is yes. It took me a while to get it all together, including a couple of start-overs to simplify things. I wrote a Python file with a couple of classes / declarations that was my model for what I wanted and worked from there. It changed slightly throughout the day and night, but at the end of it, I have something like this, and it’s working:

from mypkg.api import *

class DemoApp(Registrations):
    zope = Subversion('svn://svn.zope.org/repos/main')[
        source('zc.catalog',
            repository='Sandbox/zc/catalog',
            target='zc/catalog',
            revision='37377',
            #post=command(tools.touch, 'zc/__init__.py'),
            ),
        ]

    codespeak = Subversion('http://codespeak.net/svn/z3')[
        source('hurry',
            repository='hurry/trunk/src/hurry',
            target='hurry',
            revision='18433',
            ),
        ]

    personal = CVS('${env:CVSROOT}')[
        source('demolib', repository='Support/demolib', target='demolib'),
        source('demoapp', repository='Applications/demoapp', target='demo'),
        ]

if __name__ == '__main__':
    main(DemoApp)

There are a few things going on behind the scenes to support this so that even a lot of the front end code in main(registrations) is really quite straightforward. A factory method style design pattern combined with use of zope.interface and zope.schema to validate and/or convert values seen above into what’s wanted inside the system (like turning ’${env:CVSROOT}’ into a templated string) takes care of the parsing problem while helping keep the map code clean. A little bit of metaclass-ish intelligence turns Registrations based classes into easily accessible data structures. A couple of simple adapters help preserve an easy interface on how some items find each other or get formatted, and keep the registration information separate from the objects that are ultimately created from the registrations, and when the system is finished a few more will provide the full command line interface options and the relationship between those commands and what’s produced by the map above.

My thoughts after this? I don’t know. The project is half done and may still change. In general, I like it. Even without falling back on meta-classes, I’m trying to apply this pattern in many places, to use intelligent data structures, contracts, and collaborating components to do more things declaratively where applicable. The Zope Component Architecture makes this very nice – define a contract/interface like IRepository, and then start providing adapters for it to provide things like command line formatting, log querying, etc; or to parse or format schema fields for a plain file representation.

I’ve had my eye, idly, on the TextMate editor for Mac OS X for some time. I don’t do much editing on my desktop – most development is done on a development server and all the glory of VIM (I admit it – I converted from EMacs a couple of years ago and have been generally happy every since). But I’ve been tinkering a bit more with things at both home and office and I’m always looking for things to make my day better, if possible.

For what little desktop text editing I’ve done in recent months, BBEdit’s free little brother TextWrangler has been decent for me: code coloring, decent editing support, but none of the automation or contextual intelligence that BBEdit might have. Having used BBEdit since “back in the day” (1996, perhaps earlier), I’m still fairly comfortable with it but am not comfortable using it for long periods. The classic Mac OS feel still bleeds through, and I’m still not sure how I feel about how TextWrangler implements “tabs”. I like the single window interface option, but switching between open files is unpredictable and seems based on recent access rather than the order they appear. There may be a setting somewhere to change this behavior, but I haven’t really cared enough. For how little I need to use it, it’s worth the free price.

TextMate, on the other hand, is a fully native Mac OS X editor built on the Cocoa framework that descended from NeXTStep. Cocoa has long had extremely nice text editing options that all Cocoa based applications pick up automatically. I forget that people still whine about lack of spell checking in web browser text areas. Why? Because Safari’s textareas use Cocoa’s text editing, and all Cocoa text areas pick up on the spell check options automatically. All Cocoa text editing also supports some basic EMacs navigation commands – control-A and control-E for beginning / end of line, and even the Emacs style cut and yank (control-k and control-Y) which operates independent of the normal system clipboard. Good for quickly moving lines around. So while all applications can stylize things differently, there is a lot of very powerful and flexible text editing / viewing / manipulating features built in that work the same whether you’re editing an element in OmniOutliner or setting notes on an iCal event or using TextEdit – the free plain and rich text editor that seems to serve as a showcase for all that the text system in Cocoa can do. For example, when I upgraded to Mac OS X 10.4 (or maybe it was back with 10.3?) that there were named style options in TextEdit, which came in handy right away for some small documents I was working on. When I opened up Aquaminds NoteTaker after doing the upgrade, I saw that the same named styles system was available in there as well. And it appears that in the Mac OS X 10.4.4 update, released recently, that the text system has learned a new trick: rectangular selection. I can clip some code from the Python interpreter running in Terminal and paste it into just about anything – Safari, TextEdit, TextMate, Stickies even!, and by holding down the ‘option’ key (the mouse takes on a crosshair cursor) I can select and cut the leading characters of the interpeter (the >>> and the ...).

So – Cocoa’s text editing system is cool. Applications can all do their own thing, but almost all of them offer the same great editing features that are far and beyond what you see in other major operating environments. What does this have to do with TextMate?

I found out today that there’s a command in TextMate to install a Edit in TextMate… command. It installs an Input Manager which is a way for other tools to augment the text input system. On my desktop, I have a very old “platform completed input” manager, which I never really used. It was released in the early Mac OS X days, I don’t know by whom. Besides, auto-completion is now a built in feature of Mac OS X text input. I always thought it was a cool example of how the system could be extended. TextMate plugs in using this system, adding a new command and shortcut to the Edit menu. So right now, I fired up Blogger (which doesn’t support ‘rich text’ editing in Safari) and hit control-command-E and boom! TextMate!

TextMate is also a fascinating example of Mac / Unix integration. It’s a very good looking application. Text, of course, renders beautifully. And it’s incredibly extensible. It’s not a super-programmatic environment unto itself like Emacs. Instead it uses environment variables, shell commands, and launches into different programming languages (Python, Ruby, Perl, Shell Scripting) to do operations on text. So when I fired this entry up out of Safari, I can write away here in Textile. There’s a command to convert it to textile which just runs through Textile.rb. And then, just for fun, I made a command that uses html2text.py (which is in the Markdown bundle for TextMate), and I can turn the Textile source into Markdown, which is done through Python. I can convert it back to HTML from the Markdown bundle which uses Markdown.pl, which is (obviously) Perl. I’m impressed with how nicely this all flows together.

Why is it so exciting though? Good question. I’m about to do some hard work over the next couple of weeks on a Zope 2 based product, which has a lot of its major cough “logic” moved into disk based Python components but has some pretty hairy templates and a thin layer of Python Scripts in the ZODB. Most ZODB development is done through the web, although there are FTP and WebDAV options (TextWrangler supports direct FTP editing). There are advantages and disadvantages to Zope’s through-the-web capabilities and this isn’t a debate over those. I’m just looking forward to being able to use Zope’s admin interface (the ZMI) to go to the template I need to edit, hit command-control-E, and be able to edit it in a full featured native 100% Mac OS X editor.

For the last couple of weeks (last few days of the old year, first few days of the new), I've been working on a small application written in Zope 3. This time it's a Knowledge Base application. A simple setup, in general - add articles to a container, organized by tags (keywords). Some of it has been easy, some of it has been a bit trying. It's taken a while because I've been using this as an opportunity to explore some of the new features of Zope 3.2, which was released in final form recently, as well as some available components that I haven't been able to take advantage of until now.

Good: ZODB. The ZODB is still the best option for persistence. With Zope 3, separating out the persistent core model objects is a cleaner and more obvious design solution than it was in Zope 2. Unlike an object-relational mapping, the ZODB is pure python persistence and there's no translation needed from database values and Python values. Where does this really stand out? It's most obvious in container / contained relationships, which is the most common type of one-to-one mapping. Hierarchies are easily represented, and most containers are used just like any Python mapping (dictionary-style) object. But where else does the ZODB show its strength? When using Python values like tuples. 'keywords' can just be a tuple of strings. There's no need to create a lookup table to map an item in one row to one or more keywords. foo.tags = ('silly', 'example') saves as Python, loads as Python.

Mixed: SetIndexes. The only way to get these great plug-in indexes for Zope 3's "catalog" system, which provides indexing support to Zope 3 applications for rapid querying of content data, is to check them out from Subversion. The indexes that come with Zope 3 are barely-there, it seems. But they do their job competently. The zc.catalog code provides new indexes that are easier to query and provides some new features. The best one - SetIndex. With 'tags' set up as a SetIndex in the catalog, searching for AnyOf tags for an iterable value, like ('silly', 'example'), quickly returns all items whose tags (which can be adapted to) have either of those words in their set. The SetIndex works great. I hope these indexes can find their way into the core or into a more available release.

Mostly Good: Viewlets. Generally, I like viewlets for designing applications like this. Existing documentation is very developer heavy. But at least that documentation exists and I can (generally) understand it. Viewlets make it possible, at the very least, to componentize web pages. Knowing the best practices for how to use them or their managers is a little vague right now. In general, my experience with Viewlets was really good. They were handy for handling menus, little boxes on the side (like 'search' or 'related tags'). They can be applied to only certain content items, or even certain views, or even more constrained within that. They make it easier to pull a lot of "display logic" out of master templates and into Python classes with specific responsibilities. They make it easier to pull a web page (or different pages) apart into a specification that says "this is the header viewlet manager, this is the sidebar. Register for the ISidebarManager to put a viewlet in there". Then a new viewlet for the sidebar, such as a search box, can be written and registered for the ISidebarManager months later and should slide in easily, using the component architecture instead of brute-force templating or vague conventions.

Good: Annotations. "Annotations" are how other components and systems can add specialized data to a content object without interfering with that object's data. One of the nice things about relational databases is the way that tables can easily be made to "relate" to other tables. In a RDBMS content management system, a lot of the Dublin Core metadata spec would probably be kept in a dublincore table with columns for the different elements, and all content items might have a "dublincore_id" column to facilitate joining. In Zope 3, Annotations provide similar one-to-one mappings. This isn't contained data, per se. It's extra info for an object. Annotations are kept in their own namespaces, and everything that uses annotations maintains its own namespace. All that a content object has to do is declare support for an annotatable interface, the most common being "IAttributeAnnotatable" which stores the root annotations for that content object on a special attribute (__annotatable__ or something like that). The nice thing is that all of these internals are kept hidden. The component architecture is used to access the annotations. A ZODB based object can easily support this extra attribute, but an object from an object-relational mapper or some other data source might not. A different annotator could be provided for those. Client code shouldn't have to worry about how the annotations are stored (or even if annotations are how the code is being stored).

Good: Formlib. zope.formlib is a new feature of Zope 3.2. It's a more powerful and generalized form system than the one that's in zope.app.form. Note that neither of these form libraries are responsible for field specifications nor widgets. It does provide a lot of useful functions, classes, and decorators for getting and validating input from widgets against a set of field specifications, for rendering widgets, for handling errors (including invariants), and handling actions (different "submit" buttons, how to handle input errors if 'Preview' is clicked instead of 'Save'). Nicely, it puts a lot of this power into Python classes and not configuration. It allows for, I suppose, beautiful programming. My limited experiences with formlib have been more rewarding than my experiences with zope.app.form.

Frustrating, but good I guess: Custom Traversal. Zope, going back to Bobo, has always been a strong proponent of "nice URLs". In Zope, you traverse to objects to publish them. This has always been the core concept of the system. And you could always provide some custom traversal, but the APIs for doing it in the past were vague and increasingly forgotten. There were a couple of options available to me for doing these tag URLs, where the tags in the URL were just to be used to construct a catalog query. One option involved just getting all of the names past a certain point and chopping them off of the traversal stack, ending traversal at the root of the 'tags' search. This was easiest, but not really fitting with the object publishing style of Zope, since the 'tags/foo/bar' wasn't publishing an object at 'bar', but at 'tags'. I replaced this first implementation with one that actually published objects for each tag in the URL. This wasn't necessarily easy, at least not when compared to things like Django's URL Dispatch. But now that it's in place, I like it better. Perhaps I'm just too used to Bobo's concept of object publishing. The biggest frustration was that there are a few traversing APIs and adapters to choose from, but each does different things, and finding the one to use for URLs specifically was not obvious. Even more frustrating - I think I just realized a more Pythonic way to write my tag traverser (basically to allow 'getitem' use, so that ...tags/foo/bar/baz really turns into Python (or could be done in Python with) ..tags['foo']['bar']['baz'].

Bad: This Post. It's past midnight on Sunday. I'm posting this primarily because I've been promising to write something about my recent experiences. They've been mostly good. I'm having a lot of fun with Zope 3. There are some frustrations from time to time, and it sure could use a good site housing tips and tricks and recipes and articles like "New in Zope 3.2 - formlib. What it means to you as a form developer". I haven't given very much back to the community, but if time permits (which it seldom does), I'd like to provide more things like that in a better atmosphere than the one provided by this blog. I make no promises. Time is a rare commodity between personal, professional, and artistic obligations. And NFL playoffs and ski days.