apk zipalign

This weekend I had the luxury of free time for what feels like the first time in a good while. After some discussions on the Zope 3 mailing lists this weekend I decided that I wanted to try some new development options in the application I’m working on. Normally I do my development on a server at the office and sometimes work from home via SSH, but I wanted to set up a good local copy on my desktop so I could try working more in the TextMate editor, and keep the number of connections to work to a minimum.

This led to me thinking about how to get all of the supporting libraries for my app checked out from our own repository and a couple of third party subversion repositories. A goal of mine is to start getting our release management under control, so this line of thought led to me starting work on a kind of packaging tool. I doubt it will ever be public – it’s not meant to compete with eggs or zpkg. I needed something simple so I could check out these five packages.

I first thought about how I was going to define the map – what needs to be checked out, from where, to where, what revision / tag, etc. Should it be YAML? XML? ZConfig? My own syntax? I looked at YAML first, but it looks like to use it best it needs to go through C bindings. I thought about XML, but I’ve never gotten so insanely comfortable with XML parsing that I could come up with something quickly. As I looked at the other options (including YAML again), the translation layer between some text file in some language and native (and potentially rich) Python types seemed too big, at least for me to want to deal with at the time.

Something that’s been in my head a lot lately is the content of the Snakes and Rubies presentations about Django and Rails. One thing that David Heinemeier Hansson mentions a lot about the design of Ruby on Rails is that Ruby makes it easy to do “domain specific languages” within ruby, and Rails takes advantage of that by using a fairly natural syntax in elements like Active Record models that creates a lot of methods on the fly. I believe that in Ruby you can fairly easily interact with the class while inside of the class. In Python, this isn’t that easy. Well, it’s not easily exposed. But it can be done, albeit with tricks. I’ve seen this in Zope 3 – there’s been a move away from this style of programming:

  class IFooContained(IContained):
      __parent__ = zope.schema.Field(
          constraint=ContainerTypesConstraint(IFooLib))

  class Foo(object):
      __implements__ = (IFoo, IFooContained)
      __used_for__ = IBar

To this style:

  class IFooContained(IContained):
      containers(IFooLib)

  class Foo(object):
      implements(IFoo, IFooContained)
      adapts(IBar)

And with metaclasses, decorators, descriptors, and so on, it’s certainly possible to do more domain specific language style things within Python while not straying far from being “Pythonic” (for all of the mystical meanings of that word).

So I decided to give it a try: using the Zope 3 component architecture (but none of the Zope 3 app server or web specific parts), could I write something that let me define a release map in Python that was largely declarative but showed structure? Could I still have it validate? Could I have it turn certain keyword arguments from a regular string into a templated string without the map being filled with things like template("${src}/bar"). Could I have those templates allow for namespaces so I could easily access environment variables like ${env:CVSROOT} ? It turns out, the answer is yes. It took me a while to get it all together, including a couple of start-overs to simplify things. I wrote a Python file with a couple of classes / declarations that was my model for what I wanted and worked from there. It changed slightly throughout the day and night, but at the end of it, I have something like this, and it’s working:

from mypkg.api import *

class DemoApp(Registrations):
    zope = Subversion('svn://svn.zope.org/repos/main')[
        source('zc.catalog',
            repository='Sandbox/zc/catalog',
            target='zc/catalog',
            revision='37377',
            #post=command(tools.touch, 'zc/__init__.py'),
            ),
        ]

    codespeak = Subversion('http://codespeak.net/svn/z3')[
        source('hurry',
            repository='hurry/trunk/src/hurry',
            target='hurry',
            revision='18433',
            ),
        ]

    personal = CVS('${env:CVSROOT}')[
        source('demolib', repository='Support/demolib', target='demolib'),
        source('demoapp', repository='Applications/demoapp', target='demo'),
        ]

if __name__ == '__main__':
    main(DemoApp)

There are a few things going on behind the scenes to support this so that even a lot of the front end code in main(registrations) is really quite straightforward. A factory method style design pattern combined with use of zope.interface and zope.schema to validate and/or convert values seen above into what’s wanted inside the system (like turning ’${env:CVSROOT}’ into a templated string) takes care of the parsing problem while helping keep the map code clean. A little bit of metaclass-ish intelligence turns Registrations based classes into easily accessible data structures. A couple of simple adapters help preserve an easy interface on how some items find each other or get formatted, and keep the registration information separate from the objects that are ultimately created from the registrations, and when the system is finished a few more will provide the full command line interface options and the relationship between those commands and what’s produced by the map above.

My thoughts after this? I don’t know. The project is half done and may still change. In general, I like it. Even without falling back on meta-classes, I’m trying to apply this pattern in many places, to use intelligent data structures, contracts, and collaborating components to do more things declaratively where applicable. The Zope Component Architecture makes this very nice – define a contract/interface like IRepository, and then start providing adapters for it to provide things like command line formatting, log querying, etc; or to parse or format schema fields for a plain file representation.

I’ve had my eye, idly, on the TextMate editor for Mac OS X for some time. I don’t do much editing on my desktop – most development is done on a development server and all the glory of VIM (I admit it – I converted from EMacs a couple of years ago and have been generally happy every since). But I’ve been tinkering a bit more with things at both home and office and I’m always looking for things to make my day better, if possible.

For what little desktop text editing I’ve done in recent months, BBEdit’s free little brother TextWrangler has been decent for me: code coloring, decent editing support, but none of the automation or contextual intelligence that BBEdit might have. Having used BBEdit since “back in the day” (1996, perhaps earlier), I’m still fairly comfortable with it but am not comfortable using it for long periods. The classic Mac OS feel still bleeds through, and I’m still not sure how I feel about how TextWrangler implements “tabs”. I like the single window interface option, but switching between open files is unpredictable and seems based on recent access rather than the order they appear. There may be a setting somewhere to change this behavior, but I haven’t really cared enough. For how little I need to use it, it’s worth the free price.

TextMate, on the other hand, is a fully native Mac OS X editor built on the Cocoa framework that descended from NeXTStep. Cocoa has long had extremely nice text editing options that all Cocoa based applications pick up automatically. I forget that people still whine about lack of spell checking in web browser text areas. Why? Because Safari’s textareas use Cocoa’s text editing, and all Cocoa text areas pick up on the spell check options automatically. All Cocoa text editing also supports some basic EMacs navigation commands – control-A and control-E for beginning / end of line, and even the Emacs style cut and yank (control-k and control-Y) which operates independent of the normal system clipboard. Good for quickly moving lines around. So while all applications can stylize things differently, there is a lot of very powerful and flexible text editing / viewing / manipulating features built in that work the same whether you’re editing an element in OmniOutliner or setting notes on an iCal event or using TextEdit – the free plain and rich text editor that seems to serve as a showcase for all that the text system in Cocoa can do. For example, when I upgraded to Mac OS X 10.4 (or maybe it was back with 10.3?) that there were named style options in TextEdit, which came in handy right away for some small documents I was working on. When I opened up Aquaminds NoteTaker after doing the upgrade, I saw that the same named styles system was available in there as well. And it appears that in the Mac OS X 10.4.4 update, released recently, that the text system has learned a new trick: rectangular selection. I can clip some code from the Python interpreter running in Terminal and paste it into just about anything – Safari, TextEdit, TextMate, Stickies even!, and by holding down the ‘option’ key (the mouse takes on a crosshair cursor) I can select and cut the leading characters of the interpeter (the >>> and the ...).

So – Cocoa’s text editing system is cool. Applications can all do their own thing, but almost all of them offer the same great editing features that are far and beyond what you see in other major operating environments. What does this have to do with TextMate?

I found out today that there’s a command in TextMate to install a Edit in TextMate… command. It installs an Input Manager which is a way for other tools to augment the text input system. On my desktop, I have a very old “platform completed input” manager, which I never really used. It was released in the early Mac OS X days, I don’t know by whom. Besides, auto-completion is now a built in feature of Mac OS X text input. I always thought it was a cool example of how the system could be extended. TextMate plugs in using this system, adding a new command and shortcut to the Edit menu. So right now, I fired up Blogger (which doesn’t support ‘rich text’ editing in Safari) and hit control-command-E and boom! TextMate!

TextMate is also a fascinating example of Mac / Unix integration. It’s a very good looking application. Text, of course, renders beautifully. And it’s incredibly extensible. It’s not a super-programmatic environment unto itself like Emacs. Instead it uses environment variables, shell commands, and launches into different programming languages (Python, Ruby, Perl, Shell Scripting) to do operations on text. So when I fired this entry up out of Safari, I can write away here in Textile. There’s a command to convert it to textile which just runs through Textile.rb. And then, just for fun, I made a command that uses html2text.py (which is in the Markdown bundle for TextMate), and I can turn the Textile source into Markdown, which is done through Python. I can convert it back to HTML from the Markdown bundle which uses Markdown.pl, which is (obviously) Perl. I’m impressed with how nicely this all flows together.

Why is it so exciting though? Good question. I’m about to do some hard work over the next couple of weeks on a Zope 2 based product, which has a lot of its major cough “logic” moved into disk based Python components but has some pretty hairy templates and a thin layer of Python Scripts in the ZODB. Most ZODB development is done through the web, although there are FTP and WebDAV options (TextWrangler supports direct FTP editing). There are advantages and disadvantages to Zope’s through-the-web capabilities and this isn’t a debate over those. I’m just looking forward to being able to use Zope’s admin interface (the ZMI) to go to the template I need to edit, hit command-control-E, and be able to edit it in a full featured native 100% Mac OS X editor.

For the last couple of weeks (last few days of the old year, first few days of the new), I've been working on a small application written in Zope 3. This time it's a Knowledge Base application. A simple setup, in general - add articles to a container, organized by tags (keywords). Some of it has been easy, some of it has been a bit trying. It's taken a while because I've been using this as an opportunity to explore some of the new features of Zope 3.2, which was released in final form recently, as well as some available components that I haven't been able to take advantage of until now.

Good: ZODB. The ZODB is still the best option for persistence. With Zope 3, separating out the persistent core model objects is a cleaner and more obvious design solution than it was in Zope 2. Unlike an object-relational mapping, the ZODB is pure python persistence and there's no translation needed from database values and Python values. Where does this really stand out? It's most obvious in container / contained relationships, which is the most common type of one-to-one mapping. Hierarchies are easily represented, and most containers are used just like any Python mapping (dictionary-style) object. But where else does the ZODB show its strength? When using Python values like tuples. 'keywords' can just be a tuple of strings. There's no need to create a lookup table to map an item in one row to one or more keywords. foo.tags = ('silly', 'example') saves as Python, loads as Python.

Mixed: SetIndexes. The only way to get these great plug-in indexes for Zope 3's "catalog" system, which provides indexing support to Zope 3 applications for rapid querying of content data, is to check them out from Subversion. The indexes that come with Zope 3 are barely-there, it seems. But they do their job competently. The zc.catalog code provides new indexes that are easier to query and provides some new features. The best one - SetIndex. With 'tags' set up as a SetIndex in the catalog, searching for AnyOf tags for an iterable value, like ('silly', 'example'), quickly returns all items whose tags (which can be adapted to) have either of those words in their set. The SetIndex works great. I hope these indexes can find their way into the core or into a more available release.

Mostly Good: Viewlets. Generally, I like viewlets for designing applications like this. Existing documentation is very developer heavy. But at least that documentation exists and I can (generally) understand it. Viewlets make it possible, at the very least, to componentize web pages. Knowing the best practices for how to use them or their managers is a little vague right now. In general, my experience with Viewlets was really good. They were handy for handling menus, little boxes on the side (like 'search' or 'related tags'). They can be applied to only certain content items, or even certain views, or even more constrained within that. They make it easier to pull a lot of "display logic" out of master templates and into Python classes with specific responsibilities. They make it easier to pull a web page (or different pages) apart into a specification that says "this is the header viewlet manager, this is the sidebar. Register for the ISidebarManager to put a viewlet in there". Then a new viewlet for the sidebar, such as a search box, can be written and registered for the ISidebarManager months later and should slide in easily, using the component architecture instead of brute-force templating or vague conventions.

Good: Annotations. "Annotations" are how other components and systems can add specialized data to a content object without interfering with that object's data. One of the nice things about relational databases is the way that tables can easily be made to "relate" to other tables. In a RDBMS content management system, a lot of the Dublin Core metadata spec would probably be kept in a dublincore table with columns for the different elements, and all content items might have a "dublincore_id" column to facilitate joining. In Zope 3, Annotations provide similar one-to-one mappings. This isn't contained data, per se. It's extra info for an object. Annotations are kept in their own namespaces, and everything that uses annotations maintains its own namespace. All that a content object has to do is declare support for an annotatable interface, the most common being "IAttributeAnnotatable" which stores the root annotations for that content object on a special attribute (__annotatable__ or something like that). The nice thing is that all of these internals are kept hidden. The component architecture is used to access the annotations. A ZODB based object can easily support this extra attribute, but an object from an object-relational mapper or some other data source might not. A different annotator could be provided for those. Client code shouldn't have to worry about how the annotations are stored (or even if annotations are how the code is being stored).

Good: Formlib. zope.formlib is a new feature of Zope 3.2. It's a more powerful and generalized form system than the one that's in zope.app.form. Note that neither of these form libraries are responsible for field specifications nor widgets. It does provide a lot of useful functions, classes, and decorators for getting and validating input from widgets against a set of field specifications, for rendering widgets, for handling errors (including invariants), and handling actions (different "submit" buttons, how to handle input errors if 'Preview' is clicked instead of 'Save'). Nicely, it puts a lot of this power into Python classes and not configuration. It allows for, I suppose, beautiful programming. My limited experiences with formlib have been more rewarding than my experiences with zope.app.form.

Frustrating, but good I guess: Custom Traversal. Zope, going back to Bobo, has always been a strong proponent of "nice URLs". In Zope, you traverse to objects to publish them. This has always been the core concept of the system. And you could always provide some custom traversal, but the APIs for doing it in the past were vague and increasingly forgotten. There were a couple of options available to me for doing these tag URLs, where the tags in the URL were just to be used to construct a catalog query. One option involved just getting all of the names past a certain point and chopping them off of the traversal stack, ending traversal at the root of the 'tags' search. This was easiest, but not really fitting with the object publishing style of Zope, since the 'tags/foo/bar' wasn't publishing an object at 'bar', but at 'tags'. I replaced this first implementation with one that actually published objects for each tag in the URL. This wasn't necessarily easy, at least not when compared to things like Django's URL Dispatch. But now that it's in place, I like it better. Perhaps I'm just too used to Bobo's concept of object publishing. The biggest frustration was that there are a few traversing APIs and adapters to choose from, but each does different things, and finding the one to use for URLs specifically was not obvious. Even more frustrating - I think I just realized a more Pythonic way to write my tag traverser (basically to allow 'getitem' use, so that ...tags/foo/bar/baz really turns into Python (or could be done in Python with) ..tags['foo']['bar']['baz'].

Bad: This Post. It's past midnight on Sunday. I'm posting this primarily because I've been promising to write something about my recent experiences. They've been mostly good. I'm having a lot of fun with Zope 3. There are some frustrations from time to time, and it sure could use a good site housing tips and tricks and recipes and articles like "New in Zope 3.2 - formlib. What it means to you as a form developer". I haven't given very much back to the community, but if time permits (which it seldom does), I'd like to provide more things like that in a better atmosphere than the one provided by this blog. I make no promises. Time is a rare commodity between personal, professional, and artistic obligations. And NFL playoffs and ski days.