apk zipalign

A couple of months ago, we started work on a major rewrite of one of our core sites. What started as a small and specialized e-commerce service a few years ago has grown mightily over those same years. The architecture, however, has hit a limit - especially when compared to the large push for what we want to do (and can do) next. The current site works great and handles its load and duties very well. But there are small annoyances internally with some things, and there is some growth and market potential that it cannot satisfy.

The first version of the site’s primary work was done in less than a month. The main focus of that version was the visitor’s experience. The original HTML mockups drove a lot of the site, and the implementation was primarily done in Zope 2, through the web, using templates, scripts, and SQL Methods. It was a nice enough system for the time - we were very fast with it, we could tweak elements in each other’s offices, etc. It ended with an all-night debug-and-deploy session, wrought with all sorts of strange problems and angst and fun. But it worked. We continued to work on it over the next few weeks, of course, filling in some major holes that couldn’t be filled in initial development and responding to site usage.

The biggest problem with that implementation was the administration side. Since we were the only users of the administration screens, we punted on that issue. There was no form validation. All of the database CRUD statements were entered manually. Zope’s SQL Methods made this a bit easier, but their main optimization is for read-queries. Any new field required numerous code updates all over the place. The whole business was a little messy back then - requiring us to print out certain reports and delivering them to our providers on time-constrained schedules.

The site was never empty, but it got perilously close a few times during the lifetime of version 1. But there was enough interest to keep it going. With eyes on expanding and getting new providers, with providing new services (including physical goods), and with running the business side of things better, we went into version 2.

As I saw all of the HTML forms and database statements I’d have to change just to accommodate the expanding and changing data requirements, I started looking at options for easing that pain. I ultimately rolled my own system focused primarily on the common CRUD operations. There was no object-relational mapping solution at the time that I liked. A problem, at that time, was that we had many queries in place that would not work in the object-relational tools available.

What I really wanted was an architecture that:

1. Helped me get important / core business and administration logic out of the ZODB. We wanted to have as much of the core software on disk and under source control as possible.


2. Dynamic form and SQL generation for common tasks (primarily in the administration section). I wanted this so that adding new fields to the database schema required only one or two changes.


3. Still allowed us to use Zope’s SQL Methods for generating complex queries.


4. Could be introduced in a way that the public application might not even be aware: instead of a folder full of Python scripts in the ZODB, there’d be a single persistent instance of a class that had many of those scripts as methods.

The resulting framework was essentially a service layer comprised of very rough Table Data Gateways. You couldn’t load an object out and do a jeff.hair = 'bleached' and have it save. The get/create/save statements basically took a dictionary (hash table) of values to save and would flush it out to the database, after harvesting some information. It was expected that the gateways would have the data prepared before passing it off to the lowest level interface, handle_op, which would perform the requested operation.

When dealing with fairly set data, such as a form, this didn’t really matter. I cared about loading and binding the data to the widgets when a page was loaded, and then parsing/validating/converting that data on save. For that, this system worked like a champ. For some other pieces of business logic, it did OK. It wasn’t great, but it was better than what we had before - especially because it didn’t require specialized update SQL to be written for those situations where just a couple of values needed to be changed.

That site has served us fine, and now the offerings on the site are quite full (many many pages). The content providers have access to their items and reports, so we no longer have to run around town delivering reports and lists.

Both versions were deployed, in full, and then continuously massaged over time. There was no ‘beta’. Some features sneaked in after the deployment deadlines. Some specialized parts of the system were overhauled a couple of times to deal with scaling issues - ones that we could never have predicted.

We’ve kept laundry lists over the years of the things we wanted to do, but couldn’t under the first two designs. Some of these came from dealing with some hacks pulled together for a couple of special customers, some came from wishful thinking, some came from complicated set-ups required to make certain rare items available for purchase in the existing system. In a getting real sense, this was OK, but not great. It limited our ability to start working with larger and more specialized providers.

Architectures have continued to improve over the past couple of years as well. When Zope 3, version 3.1 specifically, came out, we were able to do a major rewrite of a content system for one of our oldest customers. We were even able to pick up additional related customers as a result. We’ve delivered other solutions on top of Zope 3 and the ZODB (Zope Object Database) that have been both impressive and fun to work on, although sometimes Zope 3 can still cause me to go off in a huge screaming match.

I had hoped (and still wish) that for this particular site’s version 3 implementation we could ditch the relational database completely and just use the ZODB. Instead of classic Zope ZODB usage, where scripts and templates fill up the database, Zope 3 makes it much easier to keep that stuff OUT, and then easier to keep real data in. This is one of the nicer parts about working with the Zope 3 / ZODB stack: Python is just Python is just Python. There’s no translation layer, no tricks, trying to fake inheritance. There’s no real worry about translating statements into a query language - Python is the query language (with tools like the Catalog providing application level indexes for larger queries). I had an application that I was working on in Zope 3 that I thought would be a good testbed for writing as a Rails app - until I realized that Active Record’s inheritance model could never match my object model, at least as of Rails 1.0. I also played a little bit with Python’s Turbogears and SQLObject stacks, and they had me feeling pissed and frustrated within seconds. No offense to their authors, but they just did NOT work for me.

But the decision was made that for this site, we would continue with using a relational database. Well, I knew the strengths and limitations of our current systems. And I admit, I was envious of Ruby on Rails. In fact, I even suggested that we ditch Zope for this project and use Rails instead! But it was feared that there was no time on the schedule to learn Ruby. We did, however, agree on wanting a real object-relational system, or at least something more object oriented than what we used for Version 2. Version 3 has richer business logic requirements, and I’m happiest when that code can look as clean and natural as possible. I also wanted it to be easy to define new pages and views and to have to go through less guess-and-hope work than Zope 3 (as of 3.2) typically requires.

So we’ve birthed yet another in-house framework because there’s little out there that’s satisfactory. The one thing that is satisfactory is SQLAlchemy. I built a base storage framework on top of that which allowed us to use SQL Alchemy fairly transparently from within Zope. Fields and properties now let us define and use relationships fairly transparently. Items get bound to their context, are easily traversable, and can even masquerade as Zope containers without interfering with SQLAlchemy’s on system of managing work. We also birthed a base web framework that provides some useful base classes for constructing the kinds of pages and sub-views that we need with ease. These base classes corral some core Zope 3 features and some other in-house features together to ease our web development.

It’s all pretty cool, for what it is. But it’s going so slowly that I can’t help but wonder if we would have been better off working in Rails. I still haven’t found anything in the Python world that I’d leave Zope for, but Zope 3 continues to jump between being insanely cool and powerful and flexible, and being incredibly frustrating and aggravating. That I’ve been able to pull many of the tricks I’ve already been able to pull is a testament to the better parts of its design.

Even with all of my tricks and base classes, it can sometimes still take an entire morning just to get what seems like a simple page together and rendering. On the other hand, once you get pieces in place they’re pretty sturdy. There’s low likelihood of accidentally overriding a critical method name or a view/page that applies to a different context or interface layer.

But as it’s been going so seemingly slow, this development, I start to wonder why it feels that way. I really was trying to give myself a system where I could enjoy the benefits of Zope 3’s Component Architecture with an intelligent database / object-relational mapper while also enjoying the benefits of Getting Real. “Getting Real” isn’t anything really new to me (or many people). It just helps give you the OK to say “no”. Martin Fowler’s book, Refactoring, made me realize that bad smells are OK now, so long as they’re cleaned up later. In the interest of getting something done, it’s hard to go for purity. Or in the case of Python’s “import this”:

Special cases aren’t special enough to break the rules.

Although practicality beats purity.

So, what is the problem with me then?

I think it comes down to a very special case: the major rewrite. This is not a new application for us, nor is it a moderate upgrade or maintenance release. We’ve acknowledged that the existing architecture has issues when it comes to certain growth paths. To this point, we’ve also been good in saying “no” to those growth paths. But business situations and opportunities have changed.

One problem with the rewrite is the knowledge that it’s already been done. “We already did this three years ago, why can’t you show me it working in the new system already?” There senses of worry, fear, excitement all change. This is further exaggerated by the fact that these kind of rewrites are often architectural, perhaps deeply architectural. All of those agile decisions made in earlier versions no longer apply, since many of those decisions may have led to the limitations you’re trying to overcome. Those decisions were right at the time, and have been right for the past few years. But now you realize that being able to do more promotions, discount options, special offers, are among the things you really want to do with the cart. You also need to handle more delivery options instead of the two you’ve used for the past two years. You fight and push and think and tango with the cart, just focusing on shipping, with coupons in the back of the mind. Part of you thinks “I just want to add a shipping column / attribute here and be done with it, and we’ll expand it later…. Oh wait… This IS later… Oh yeah, we need to support more options… Oh yeah, this is one of the growth areas we’re focusing on. Crap!”

It’s an interesting struggle now, those two sides in my mind. In practical terms, I just want to add an extra column or attribute and be done. But I know in a week it’ll already be pushed to its limits. Better to apologize for lack of screens and work this the hell out now. Nothing fancy - keep the interfaces and collaborators simple, and expand on them if needed when more data presents itself.

Last night I floated like a ghost past the strip club, under the freeway, over the railroad tracks, past the big electric sign and refrigeration companies and worn out corners until I got to the super-center. I needed milk, inner tubes for my bike, and dog treats. Of course, my cart started filling up with much more than just that. And in the middle of my 11pm shopping excursion I stumbled across the music/video section. As I stared (half-angrily) at the so-called “top sellers” in the country department (it’s where my cart stopped), I remembered coming across a new Johnny Cash collection on the iTunes Music Store.

So I now have Personal File. It’s a collection of recordings Johnny made (circa 1973-1980) in his studio. It’s just Johnny and his guitar: the same kind of intimate, personal, and raw sound that made the first American Recordings album such a standout. Some of these pieces are recognizable, such as When it’s Springtime in Alaska, It’s 40 Below and Jim, I Wore a Tie Today. Many songs have stories attached to them - why Johnny remembers it, why he wrote it, why he sings it, etc. It was a style seen on some of his commercial albums from that time, although those were accompanied by a larger band and sound.

The second disc of Personal File hasn’t grabbed me much. It’s fairly religious, and while I respect Johnny Cash’s religious strength, I have little interest in it. There are some nice songs on that disc though. But the first disc is the good one. It’s from a good time in Johnny’s life - settling into his life with June, his drug days behind him. The voice is powerful, most of the songs quite rich. A very enjoyable listen.

I’m still offline at my new home. Curiously, this hasn’t been a huge deal for me, so far. I’m getting home so late (relatively), and then still have to walk the dog (if she stayed home that day), feed her, feed myself, etc. Other factors have been eating up bits of my evenings as well: cleaning out the old place (finally done as of yesterday), getting some finishing-touch little furnishings (one at a time since I don’t drive), and yet-more organizing and layout.

The situation with Comcast has just pissed me off enough that I haven’t wanted to call back to find out what’s next. Part of this is due to the inherent stress of workdays recently. I don’t want to add to the pile with long calls and situations where no one really seems to know what’s going on. But I know I have to, soon. Tomorrow.

I have not had any major beefs with Comcast (and AT&T before them) up to this point. And the whole situation may not be their fault, but rather the fault of bad information and the odd nature of this particular move. But as someone who has been a good customer for so long, I do feel a little insulted that someone hasn’t jumped up and said “hey! you know what, you’re a valued customer and we want to keep you so I’ll figure this out personally.” Not that I expect the average customer service person to care that much, but maybe I could be passed on up to someone who does.

This is a big thing for me right now. If they don’t seem to care, why should I? And if I don’t care, why aren’t I looking for alternatives? I still don’t really want to deal with phone companies and DSL, but I know it’s an option. And as far as television goes: I’ve long known that I need to cut back. As much as I’ve loved HBO and its original programming, I haven’t even noticed that I’ve missed a couple of weeks of the Sopranos now. I get a handful of channels off my antenna. I’m starting to think that maybe an option like USDTV could work too. Dish/Satellite is out. I hate those things and don’t really have anywhere to put the damn dish anyways.

But it comes down to this: I was told what the builder was told. “When the first person in wants cable, we just have to connect it to the building. It’s there and ready to go otherwise.” But now even the landlord / builder are getting the run-around.

This service is supposedly “right there, almost ready to go”. Why can’t I just get it turned on already?

Anyways - thank goodness for Apple’s iDisk service (as part of their .Mac offerings). iDisk can be synchronized and worked with off-line. So I can at least keep my iBook up to date with my plans/actions list and a few other key documents, and I can update source code to work from home by bringing the iBook into work and using CVS. This helps with the offline mode - knowing that some information, including work stuff (useful in these busy times) can be brought with me even if there’s no connection. This is the downside to otherwise wonderful services like Backpack: except for emailing myself a copy of pages, I’m cut off from any information I may be collecting in a system like that.

This is something that I’ve been concerned about ever since Microsoft first started promoting “internet everywhere” inside of Windows. I don’t use Windows, so it didn’t concern me much (but it’s been interesting to see how their strategy effectively made the security problem they wrestle with today). But I always wondered: “what happens if there’s no internet? if the network’s down?”. The Dashboard in Mac OS X 10.4 is effectively useless on my iMac at home now. Not that I’d expect anything different: how can it update its backpack widget and all of those weather feeds with nothing to feed it? It’s just sad though, seeing that computer that is normally so great become so useless without a connection. I can’t upload photos, can’t update my web site, can’t collaborate with musicians, can’t shop for art supplies, can’t check the train schedule, can’t get new documentation, can’t get new music from the iTunes Music Store, can’t manage the Netflix queue that I’m finally going to set up… All of these things, taken for granted. And now, thanks to a stressful and confusing mess, I don’t know when I’ll be able to take them for granted again.

"But I don't have $100 and I can't come by with a truck for two weeks..." No piano for you! Life sucks; get a helmet.

I really enjoy forgetting. When I first come to a place, I notice all the little details. I notice the way the sky looks. The color of white paper. The way people walk. Doorknobs. Everything. Then I get used to the place and I don't notice those things anymore. So only by forgetting can I see the place again as it really is. - David Byrne as Narrator, True Stories (1986)

This is an AppleScript I cobbled together real quick to post to this weblog quickly from QuickSilver, using MarsEdit to handle the posting. I don’t claim to have any real in-depth knowledge of AppleScript, and I doubt this will win any “prettiest” or “best” code awards. I just thought it might be helpful to others.

    using terms from application "Quicksilver" 
        on process text ThisClipping
            my SendToMars(ThisClipping)
        end process text
    end using terms from

    on SendToMars(IncomingString)
        set mdcommand to "/opt/local/bin/Markdown.pl " 
        set webtext to do shell script "echo " & quoted form of IncomingString & " | " & mdcommand
        tell application "MarsEdit" 
            make new post window
            tell post window 1
                set body to webtext
                send post to weblog
            end tell
        end tell
    end SendToMars

Any text formatter could be used here – I went with Markdown for this situation as it seems to be the best fit for the tiny little text window you get with QuickSilver.

To install, I put this in Library/Application Support/Quicksilver/Actions inside my home folder under the name Post to Griddle. Any text typed can then be posted immediately. Your mileage may vary, depending on your MarsEdit configuration.

This is a test, of sorts, to see how easy it is to write an applescript to send a post to GriddleNoise via MarsEdit.

It's an attempt to see how easy it might be to do some tumblelog style posts using QuickSilver as the entry pad.