apk zipalign

As a result of a peculiar chain of events, we’ve been stuck with a compiled Flash file requiring data off of a web service. The old database is gone. The old server is gone. We have been unable to get much information about the data required. We don’t have the Flash source. All we have is a single ColdFusion Component that is used a web service via WSDL and SOAP.

I’ve never used ColdFusion. Fortunately, Allaire, er, Macromedia, er, Adobe makes demos available. ColdFusion even comes with a developer version that can live beyond the typical 30 days. So using that, I was able to re-generate the WSDL. I didn’t understand it, but I had it. Then I installed a demo of Flash Professional. I made a quick document and put some web service links in it by stumbling around, and watched the HTTP/SOAP communication between Flash and Apache/ColdFusion.

I tried looking at the specs for SOAP, and recoiled. Then I naively thought I could emulate the response as a Zope Page Template emitting XML, looping over the little bit of data that the Flash client wanted. But then I realized that the SOAP style in use was RPC style, and the ‘method’ was buried in the posted XML, so I couldn’t even easily answer the incoming requests.

Since there’s no standard SOAP toolkit available for Python, Zope 3 only holds a skeleton off which one can hang a SOAP handling REQUEST/RESPONSE pair. I’m too dumb, or perhaps just too busy, to wrestle with XML parsing myself, and I quickly realized that I needed some kind of toolkit. The data that I saw in the WSDL and SOAP communications appeared far too complicated for me to read and write on my own. Again I tried going to the specs, and again I ran rapidly away as I couldn’t even figure out how the WSDL connected to the SOAP requests and responses.

Now, obviously this stuff must work out for somebody. When we first approached this problem, the advice that we got was along the lines “just put up an XML gateway.” Um, what? I then realized that the tools were probably doing everything and the developers with whom we were communicating never gave much thought to what was going on. Data format? There is no data format - just put up an XML gateway!

I can’t fault anyone for this, per se. We all get abstracted away from something. Many web developers plug along happily without knowing too many details about HTTP. Even when one knows quite a bit about the more advanced or nuanced details of HTTP, one’s not likely to know much about what’s going on at the TCP/IP layer.

While stumbling around Flash Professional, I was a wee bit impressed with how quickly I could make a web service connector, wire it to a table widget, write a quick button-click handler, and could see the results in the table. The ColdFusion Component had its WSDL generated by merely adding ?wsdl to the end of the URL. Since both tools are maintained by Macromedia, er, Adobe, the SOAP messages actually seem to work together.

But my gods, it sucks.

I have been so spoiled by HTML and Python, and by Javascript, CSS, Zope, etc. There’s no end to the value of the web browser’s “View Source” command. With Python, I can read the source of almost any library and figure out its innards if I can’t get answers from the documentation. CSS and JavaScript are equally open to public viewing. One can easily send plain text as results of HTTP calls - I think that if we had designed the system we’re wresting with right now, we would have served some kind of delimited text that any tool could read - even human eyes. This week I realized just how hard it is to make a simple SOAP request by hand, leaving me to use a TCP watcher/proxy to test requests and responses.

Using ZSI from the Python Web Services project, I was able to mash together a basic SOAP Publisher for Zope 3. It doesn’t deal with multipart messages, WS-Addressing or whatever the hell that is, “document/literal”, or even Request argument processing. Well, that’s made available on the Parsed Soap instance from ZSI that’s put on the request. The handlers - the remote procedures / “controllers” / “views” / whatever you want to call them, still have to do quite a bit of work themselves, at least with the Parsed Soap message. This is because, as far as I can tell, one can’t do much without using explicitly build custom types, generated from tools like wsdl2python. Ahhh, this takes me back to all of the crap I had to deal with in ILU/CORBA - baroque generated code that one couldn’t live without, nor was likely to understand.

Even after all of that, I had to do a lot of ZSI typecode wrangling in said generated code before I even began to get results like what was generated from ColdFusion. And these are simple calls, really - no request arguments, and the returns are just lists of dictionaries. I stand amazed at just how much work is involved with getting these little details right, even with a toolkit.

Even with that toolkit’s aid, the status line in FireFox would read “Waiting for data from…” This was puzzling, as I thought had finally mangled a seemingly proper Map and Array typecode / values together from the available tools. I started to worry - is there something at the HTTP level that SOAP is doing differently? Does it want chunked results? Is it not closing the connections? I started crawling deep in the networking guts of Zope and Twisted, trying different transfer encodings, verifying that the content-length header matched the content length in the response … nothing.

Then, on a whim, I decided to try dimensioning my array. ZSI’s array, by default, doesn’t seem to handle specifying some kind of variable length in the array type. So it was generating xsd:anyType[]. The ColdFusion results had xsd:anyType[n] with n being the length of the array. I finally got this working by dynamically altering half of a tuple on a typecode that descended from the strange Result type/typecode that I was building. I’m not even sure if this is thread safe, come to think of it. But by finally getting that dimension in there, Flash started to respond again.

Ugh. Remember, The S Stands For Simple.

That kind of development - closed, compiled, heavily dependent on tools like Flash that work on few desktops and cost a good chunk of money - is just so foreign to me. I’m no die-hard open source zealot: I love a lot of the commercial tools and applications for Mac OS X, and would rather use it than any of the “free desktops”. But when it comes to web development, I’m used to dynamic languages and source code that is often readily available or shareable without the need of heavy tools. This is (usually) code that I can read, which can teach me new techniques by real implementation and not vague “hello world” examples followed by nightmare specs. I’m used to the layers below that being either so lightweight or established that I don’t have to think about them much. I’m grateful that I still seldom have to think in XML. I’m glad that I have next to none of the endless Java acronyms memorized. Sure, I guess there’s a place for them all, but I’m happy that I’ve been able to make a living for the past ten years without all of that. And I’ll be happier still when this particular problem is behind us.

My David Lynch Coffee arrived today. I won't get a chance to try it until tomorrow. But damn! I'm excited.

So, my recent problems with Zope 3 that were making me want to resume smoking? Yeah, I’m a smoker again. Two months clean. The stress just got to be too much.

I reported my problems to the community, with long and detailed explanations and explorations and pleas for help. The responses? Very few. Far fewer than I would have hoped. I had to scrap upgrading a customer to Zope 3.3 after losing a day of work trying to upgrade their database, unsuccessfully. Back to 3.1 they go. And now maybe we’ll forget about this for another year and have an even harder time upgrading next year.

Combined with another day lost chasing an obscure bug (moral of the story - sometimes saying that you implement the interface you implement can be bad, at least in cases where people do identity checks against self when attempting to retrieve keys out of a system), I’m pretty pissed. I really want to leave it all behind. I can’t. But right now, I’d like to. Even when Zope 3 is working, there are some common activities that seem to take way too long to accomplish. I feel like I’ve been spending a vast chunk of my time in recent months focusing on building tools and classes on top of Zope 3 that could bring at least some of what I see and like in Rails and Pylons while taking advantage of Zope 3’s strengths.

I think Zope 3 can be good for CMS type applications. And even for big business and public applications, it can be a good system. The so-called “MVC” stack of the new breed of systems bugs me. I have mixed feelings on templates these days. I think that with good objects, one can generate simple HTML out of Python (or any other programming language) which can then be styled up with CSS. It’s not appropriate for all situations, but it can be a great tool.

But, I must admit, I like what I’ve seen of Pylons. I installed it yesterday and ran through a couple of the tutorials (not even getting to the Wiki tutorial). Last night, I decided to convert the Basic To-do Application that I wrote up last year to Pylons, using SQLAlchemy as the storage mechanism. Fortunately I had some recent (and inspired, if I do say so) SQLAlchemy work done for Zope 3 whose ideas I was able to borrow to get the basic Alchemy session and engine management working in my application. I had just a single table and thus a single model class (keys: ‘id’, ‘description’, and ‘done’).

I decided to go for the same minimum of functionality that I had gone for in the Basic To-do scenario: have a single page that lists completed and incomplete items, with a small form to add new items to the system.

I got this done - including the writing of the simple SQLAlchemy bridge - in about an hour or hour and a half. No time wasted writing ZCML. No time wasted turning off the ZMI. It was generally quite easy. I made a few faltering steps along the way, but not many.

I think Pylons may be the sleeper hit out of all of the Python “web frameworks”. TurboGears got to be overwhelming, immediately. I think that I’ve just never been a fan of its core components. No offense to their authors. Django is also a bit overwhelming. It’s more like Zope: more of a complete environment. That can be great for many uses, but it can be a hindrance for others. I think Django is a terrific system from what I’ve seen of it, but it’s too big for me. I have one big system in my life already.

I have a bit more admiration for the core bits of Pylons. While I’m not a big fan of its syntax, I do think that Myghty (a port of Mason) has a substantial amount to offer beyond just being a templating language: namely lots of caching options and intelligence. SkunkWeb was the only other Python web system that I’ve seen that seems to make caching such a priority. The alternate templating languages one can use in Pylons - Kid and Cheetah - are not at all appealing to me.

WebHelpers are alright as a collection of useful functions for use in templates and controllers. I’ve had a blast taking the helpers concept and implementing it in our new Zope 3 based sites. It makes a big difference. With a good base set of helpers that are easy to expand on, it becomes much more desirable to use Python expressions straight up in templates. When I saw the ERB code in many Rails projects, I was impressed - it didn’t look like the typical mess of common PHP / Classic ASP style development; nor did it have to create yet-another-language on top of its base with hard to learn bind / call semantics. One doesn’t have to translate::

<% h.text_field(name='description', size=35, maxlength=100) %>

into “what Python or template or other thing does that there code call?” compared to::

<f:text name=quest label="Question:" />

The latter coming from Spyce, from an entry on the Spyced Blog.

Actually, I was starting to want to use Spyce in my Zope projects in spaces where we had Helpers doing many common tasks (like seen above) and ZPT was way way way too verbose:

<input tal:replace="structure python:view.h.text_field(...)" />

But Spyce has grown up into a richer system, where all I really want is a good basic Python Server Pages style system. But I’ve never come across one that I’ve liked: something small, simple, fast, easy to bind, meant to work with others (ie - not meant to be standalone pages with big blocks of Python, but rather to work with a view/controller class and helpers).

Back to what I like about Pylons - the last two are Routes, which does the URL dispatch (writing maps like in Rails and Django); and the aggressive use of WSGI / Paste. These seem to be bringing the Component Architecture dream of Zope 3 closer to reality as there are additional filters and middleware one can put around one’s application. Paste provides all sorts of cool features. There seem to be a lot of options for interchangeability.

I’m hoping that Zope 3’s “eggification” will make it easy to write Zope 3 style applications without having to use Zope 3. There are a lot of great libraries and toolkits in the heart of Zope 3 whose liberation would be most welcome. Even without that, I hope to look into what might happen if Zope 3’s libraries were just added to the Python path so that a Pylons application might be written up, using Zope 3’s concept of Views (Multi-adapters) and Transactions with objects pulled from SQLAlchemy without all of Zope’s security and nonsense.

There’s a great set of lines delivered by Lloyd Bridges in Airplane:

• Looks like I picked the wrong week to quit drinking.


• Looks like I picked the wrong week to quit smoking.


• Looks like I picked the wrong week to quit sniffing glue.


• Looks like I picked the wrong week to quit amphetamines.

Sometimes I love Zope 3. Other times, it makes me start to sound like McCroskey. I haven’t smoked now in two months, but I just smashed my head into another foolish and crappy scenario in Zope 3 (after discovering an incredibly crappy scenario earlier this week). Too stressed and confused to think clearly and engineer / hack my way around this latest situation, I just want to grab a fresh pack of Lucky Strikes (unfiltered, of course).

Waaa.

Ian Bicking wrote a Zope 3 Critique a couple of weeks or so ago, partially in response to a critique by Jean-Marc Orliaguet. I’ve wanted to write a response of my own, but time has not allowed it. Now, I’d like to use a few posts to post my own responses to a few items.

The first point I’d like to address comes under headline 3, The importance of standards:

I personally don’t care much about namespaces. Is zope.formlib any more “right” than zope_formlib? Is it any more hierarchical? I don’t think so. I have one namespace package, but it’s kind of a historical accident. I wouldn’t (and don’t) put any more libraries under that namespace. Each library should stand on its own, so why create these hierarchies? People read too much into those namespaces, which is happening right here.

Ugh. zope.formlib is far more “right” than zope_formlib. Granted, this is largely opinion. But man, do I love namespaces. zope_formlib looks like one of those crap-ass hacks and tricks used by the litany of languages who have no namespaces and have to fake it.

First off - I hate the underscore, and believe it should never enter package / module names. Mixed case package/module names are also evil. PEP 8 recommends this:

Modules should have short, lowercase names, without underscores.
…
Like modules, Python packages should have short, all-lowercase names, without underscores.

While I don’t follow PEP 8 too religiously, I’ve found my life to be generally better by trying to stay close to it. I believe that short lowercase names without underscores are a nice constraint, requiring one to think a little bit about what a module/package will entail and how it will be used. If it’s a module that will be reused frequently, it’s good to have a name that’s easy to remember and type.

At least we don’t see as much of the following any more:

from DocumentTemplate.DocumentTemplate import DocumentTemplate

ugh.

Finally, Zope 3 is a large system. I make fairly large systems using many of its pieces, some more than others. Some of those are quite nice to use from an import standpoint when one just imports the sub-package:

from zope import component, interface

from interfaces import IBar

class IFoo(interface.Interface):
    def newbar():
        """ returns 'ha!' """

class BarAdapter(object):
    interface.implements(IFoo)
    component.adapts(IBar)

    ...

That’s a trivial and nearly pointless example, but it’s a pattern I find myself using more and more. I’d be a much less happy mac if I had to do the following:

import zope_component
import zope_interface
import sqlalchemy_sql

Ugh. Why not just introduce crap like ‘require’ and ‘use’ and leave this modularism behind?

There has been one problem, historically, with Python’s packages: unlike Perl, Ruby, Java, etc, the namespaces in Python are based exclusively on filenames and file system hierarchies. That has meant that a “namespace package” like zope.* would have to include everything. It couldn’t be distributed and used as smaller chunks easily, whereas if this were Java the packages/modules would use the ‘package’ keyword:

package org.zope.component;

But now SetupTools allows one to break these pieces up and distribute them separately while still getting the organizational and conceptual benefits of big hierarchical namespaces.

I know that I haven’t posted much lately. I do have a couple of long draft pieces saved that I haven’t had time to finish. I don’t know if I’ll get back to them, but I’ll try.

I’ve been looking at a lot of different things recently, looking for new inspirations, tools, and/or chances to widen my understanding. I find myself growing increasingly fond of a couple of things in particular: Seaside on the web development side, and Smalltalk, Scheme/LISP, Io, and Javascript (particularly with the Prototype.js library) on the programming language side.

Seaside interests me on many levels. I’ve been staring at it with a lustful eye for some time now. There is definitely a place for heavily stateful web apps. Seaside and WebObjects make this a part of their core, where a developer can make an application in a style that is more like a desktop application: less dealing with sessions, freedom from the typical request-response cycle, etc.

But Seaside is more than continuations and callbacks. I love how you generate HTML in it: NO TEMPLATES!!! In a post titled “Turtles All The Way Down”, Ramon Leon writes:

Seaside lets me work in Smalltalk, at every level, all the way from programmatic generation of the HTML and Javascript to configuration of the application. No HTML, XML, XSLT, or SQL is necessary to build a web application, just simple, pure Smalltalk. Programming’s never been more fun!

I’ve made some in-house tools that get us closer to that goal when using Zope 3 and Python. I’ve started to hate template languages again. Well - I don’t mind using them for layout intensive work. But as I get more proficient with CSS, even that is changing.

[Edit: Moved side rant about Zope and Views and Templates out... This isn't the post for it]

It still stare lustfully at Seaside code, especially things like this very basic example:

renderContentOn: html
    html heading: count.
    html anchor callback: [self increase]; text: '++'.
    html space.
    html anchor callback: [self decrease]; text: '--'.

Ooo. Two links are made with anonymous functions that call self increase and self decrease respectively. There’s no “absolute URL”-ness going on. There’s no “how do I make it so that increase can be called through the web?”. No exposure decorator. No seven lines of XML code each for ‘increase’ and ‘decrease’ to be made into separate views. And no template. Just a solid binding between a link on a form and real code.

Seaside is a testament to the power of Smalltalk. People think that Seaside is all about continuations, and if they get just that part down, they’ve made something to compete. But it’s more than that: Seaside is a tool for building web applications in pretty much the same way one makes a desktop application. It’s not the tool or style for every job, but it definitely has its place.

In a recent bit of work we've done in Zope 3, I would have loved to have something like Seaside, especially for the checkout process. I've tried to engineer that process as well as I could, with extra preconditions and redirectors on pages to ensure that all desired information is supplied (ie - so that one can't jump from step 1 to step 4). With Seaside I could have encompassed the in a WATask component, expressing the checkout workflow as normal Python code: while shippingAddressRequired and (not order.shippingAddress): yield self.askForShippingAddressForm() (or something like that). The shopper could never accidentally (or purposefully) get past that point without supplying the right information. Doing this with preconditions, sessions, hidden form fields, cookies, etc, is all possible. But it can be a lot more code overhead.

Back to the other interests listed, really quickly: I’ve used LISP on very rare occasions (for customizing EMacs, basically, back in the day). I’ve never quite understood the appeal that LISP / Scheme have for some people until recently. While I enjoy and will probably continue to work in general purpose languages like Python, I’ve started to grow fond of these languages that are built on very simple precepts. Smalltalk has practically no syntax: it’s all messages. And I really like that Squeak is written in itself - there’s just something impressive about that. On a related topic, I checked on the progress of PyPy and was very impressed. Like Squeak, which has a small subset-of-smalltalk-to-C translator for generating the core pieces from Smalltalk itself, PyPy is Python written in Python, with translators to lower-level languages.

Regarding Scheme, I found a Scheme translator for Smalltalk and have spent a small amount of time playing with it this weekend. I came away impressed with how easy it is to parse and process Scheme, and how easy it is to implement it and its primitives in quite easily readable Smalltalk. The “everything is a list” (‘car’ and ‘cdr’) power finally came through to me as I browsed through the Smalltalk code to see how things that are keywords and operators in other languages (‘+’, ‘if’, etc) were implemented by just using car and cdr (the head and remainder of any list/pair, on which all things are based).

I’ve also become very interested in prototype-based programming. With prototype based programming, one has Objects but without classes and metaclasses and on and on. I’ve known about prototype based programming for some time, but I’ve only recently started to appreciate it. It’s radically simple. Looking around at Prototype based languages, I came across the relative newcomer Io. Io has no keywords - which, again, I find quite interesting and provocative. And powerful. It’s message based, which I like, and it has no classes. You just clone something to make something else. A simple object example from the sample code page:

Account := Object clone do(
    balance := 0
    deposit  := method(v, balance = balance + v)
    withdraw := method(v, balance = balance - v)
    show := method(writeln("Account balance: $", balance))
)

myAccount := Account clone
myAccount show
"Depositing $10\n" print
myAccount deposit(10)
myAccount show

Oh yeah - and the minimalist in me swoons over its web site.

There it is - a scattering of a portion of the thoughts that occupy my brain when I have the chance to take a breath at work.

I’ve surprised myself by starting to enjoy doing browser-side development in CSS and JavaScript more than I enjoy the server side. Prototype, Scriptaculous, LowPro, and a comfortably growing in-house library have gone a long way towards helping with this shift, along with finally getting down into the guts of CSS and learning about the box model. Tools like FireBug and the Web Developer Toolbar for Firefox have helped out tremendously, as well as native Mac OS X tools like XyleScope and direct linking to the CSS specs from within TextMate (very helpful when wanting to learn the real details about any given property).