SOAP. Web Services. Flash. Waaa.

As a result of a peculiar chain of events, we’ve been stuck with a compiled Flash file requiring data off of a web service. The old database is gone. The old server is gone. We have been unable to get much information about the data required. We don’t have the Flash source. All we have is a single ColdFusion Component that is used a web service via WSDL and SOAP.

I’ve never used ColdFusion. Fortunately, Allaire, er, Macromedia, er, Adobe makes demos available. ColdFusion even comes with a developer version that can live beyond the typical 30 days. So using that, I was able to re-generate the WSDL. I didn’t understand it, but I had it. Then I installed a demo of Flash Professional. I made a quick document and put some web service links in it by stumbling around, and watched the HTTP/SOAP communication between Flash and Apache/ColdFusion.

I tried looking at the specs for SOAP, and recoiled. Then I naively thought I could emulate the response as a Zope Page Template emitting XML, looping over the little bit of data that the Flash client wanted. But then I realized that the SOAP style in use was RPC style, and the ‘method’ was buried in the posted XML, so I couldn’t even easily answer the incoming requests.

Since there’s no standard SOAP toolkit available for Python, Zope 3 only holds a skeleton off which one can hang a SOAP handling REQUEST/RESPONSE pair. I’m too dumb, or perhaps just too busy, to wrestle with XML parsing myself, and I quickly realized that I needed some kind of toolkit. The data that I saw in the WSDL and SOAP communications appeared far too complicated for me to read and write on my own. Again I tried going to the specs, and again I ran rapidly away as I couldn’t even figure out how the WSDL connected to the SOAP requests and responses.

Now, obviously this stuff must work out for somebody. When we first approached this problem, the advice that we got was along the lines “just put up an XML gateway.” Um, what? I then realized that the tools were probably doing everything and the developers with whom we were communicating never gave much thought to what was going on. Data format? There is no data format - just put up an XML gateway!

I can’t fault anyone for this, per se. We all get abstracted away from something. Many web developers plug along happily without knowing too many details about HTTP. Even when one knows quite a bit about the more advanced or nuanced details of HTTP, one’s not likely to know much about what’s going on at the TCP/IP layer.

While stumbling around Flash Professional, I was a wee bit impressed with how quickly I could make a web service connector, wire it to a table widget, write a quick button-click handler, and could see the results in the table. The ColdFusion Component had its WSDL generated by merely adding ?wsdl to the end of the URL. Since both tools are maintained by Macromedia, er, Adobe, the SOAP messages actually seem to work together.

But my gods, it sucks.

I have been so spoiled by HTML and Python, and by Javascript, CSS, Zope, etc. There’s no end to the value of the web browser’s “View Source” command. With Python, I can read the source of almost any library and figure out its innards if I can’t get answers from the documentation. CSS and JavaScript are equally open to public viewing. One can easily send plain text as results of HTTP calls - I think that if we had designed the system we’re wresting with right now, we would have served some kind of delimited text that any tool could read - even human eyes. This week I realized just how hard it is to make a simple SOAP request by hand, leaving me to use a TCP watcher/proxy to test requests and responses.

Using ZSI from the Python Web Services project, I was able to mash together a basic SOAP Publisher for Zope 3. It doesn’t deal with multipart messages, WS-Addressing or whatever the hell that is, “document/literal”, or even Request argument processing. Well, that’s made available on the Parsed Soap instance from ZSI that’s put on the request. The handlers - the remote procedures / “controllers” / “views” / whatever you want to call them, still have to do quite a bit of work themselves, at least with the Parsed Soap message. This is because, as far as I can tell, one can’t do much without using explicitly build custom types, generated from tools like wsdl2python. Ahhh, this takes me back to all of the crap I had to deal with in ILU/CORBA - baroque generated code that one couldn’t live without, nor was likely to understand.

Even after all of that, I had to do a lot of ZSI typecode wrangling in said generated code before I even began to get results like what was generated from ColdFusion. And these are simple calls, really - no request arguments, and the returns are just lists of dictionaries. I stand amazed at just how much work is involved with getting these little details right, even with a toolkit.

Even with that toolkit’s aid, the status line in FireFox would read “Waiting for data from…” This was puzzling, as I thought had finally mangled a seemingly proper Map and Array typecode / values together from the available tools. I started to worry - is there something at the HTTP level that SOAP is doing differently? Does it want chunked results? Is it not closing the connections? I started crawling deep in the networking guts of Zope and Twisted, trying different transfer encodings, verifying that the content-length header matched the content length in the response … nothing.

Then, on a whim, I decided to try dimensioning my array. ZSI’s array, by default, doesn’t seem to handle specifying some kind of variable length in the array type. So it was generating xsd:anyType[]. The ColdFusion results had xsd:anyType[n] with n being the length of the array. I finally got this working by dynamically altering half of a tuple on a typecode that descended from the strange Result type/typecode that I was building. I’m not even sure if this is thread safe, come to think of it. But by finally getting that dimension in there, Flash started to respond again.

Ugh. Remember, The S Stands For Simple.

That kind of development - closed, compiled, heavily dependent on tools like Flash that work on few desktops and cost a good chunk of money - is just so foreign to me. I’m no die-hard open source zealot: I love a lot of the commercial tools and applications for Mac OS X, and would rather use it than any of the “free desktops”. But when it comes to web development, I’m used to dynamic languages and source code that is often readily available or shareable without the need of heavy tools. This is (usually) code that I can read, which can teach me new techniques by real implementation and not vague “hello world” examples followed by nightmare specs. I’m used to the layers below that being either so lightweight or established that I don’t have to think about them much. I’m grateful that I still seldom have to think in XML. I’m glad that I have next to none of the endless Java acronyms memorized. Sure, I guess there’s a place for them all, but I’m happy that I’ve been able to make a living for the past ten years without all of that. And I’ll be happier still when this particular problem is behind us.