Dynamic browser-based interfaces are nontrivial, in the sense that your math teacher might talk about things being nontrivial. That's fancy talk for being really hard.

You get plenty of widgets to play with. Creating them and slapping them up on the screen is fairly easy. Wiring the widgets up to javascript code is actually quite trivial.

The trick is when you want to make something intelligent happen.

Consider the humble case where you don't have a value yet because you haven't yet asked the server for that value. Suppose, now, that you want to display that value. Well, you want to let the user know that you've made a request, and you're waiting for a response before the real result can be displayed.

There are several things that could happen. The server could barf, in which case you should display a usable error message, where usable means that it gets the problem fixed quickly. The server could fail to respond in any reasonable time, which would bore the user and be grounds for the program to appologize. The user could get bored more quickly, and decide to cancel the request and work without it. After doing so, the user may later wish to restart the request in hopes of getting a quicker response, perhaps after plugging the network cable back in. And finally, the networking layer might report some other kind of error, which should be communicated (politely) to the user.

What we have here is a rather complex protocol for interaction among the user and various subsystems. Protocols veritably scream out for state machines. I want a state machine.

Now I'm a big fan of reusable program components. I figure that if a state machine is so generic an idea, then we should be able to program with them where it makes sense.

Javascript doesn't come with a built-in state machine system, but of course any competent computer scientist can create one with relative ease. In fact, he or she may easily create a profusion of different, reusable, unlikely-to-actually-be-reused state machine engines for each time that a problem arises which calls out for the idea of a finite state machine. There's a jillion ways to write a state machine, and they all take time.

So when do we get to the part where the magic happens?

I've been experimenting with a tree-oriented permissions assignment system. The idea is straightforward. For each branch of the tree, a user can have either all, none, or some of the permissions within that branch. Leaf nodes just need a simple yes or no. Assuming we name all the nodes of the tree according to something like a pathname, it becomes possible to represent user rights as a set of permission paths beneath which access should be granted.

I want the permission system to work as independently as possible from the applications which it controls. However, those applications might have permission structures which change with time, and I don't want to have to update the database because of a change in the permission structure of an application. A new permission might come about because of an added feature, or it might exist because of something someone did that doesn't even touch the database. For example, right now on the client site, there's a feature that relies on files in a certain folder. Each time the client adds a file to this folder, there's another subcategory of privilege that springs into existence. I should not have to duplicate the information in some database table. There is a single point of truth, and in this case, that point is not in the database.

How, then, can the permission management system access the permission structure of the applications that it plugs into?

The basic shape of the problem is that of collecting and collating configuration information from widely scattered parts of a system. Might this be the same gluey substance that weighs on all large systems? What happens the next time some feature spreads tentacles everywhere? How do I keep a growing system from becoming little more than a tangled mass of configuration tentacles?

So that's one hassle. The next, of course, is on the client side.

Client: Installed. Mostly.
ok. It's done. Mostly. My main client now has a new version of her big labor-saving software package installed and configured. The only problem is that there are bugs.

It's only natural that there would be bugs. Of course bugs happen when you go on long hiatuses fairly often. Long ago, I committed never to leave the site in a permanently damaged condition, so I'm going to fix these bugs.

Now here's what I love about my client: We talk about the bugs rationally. She gives me screenshots and tells me what she expected and what actually happened. What she did. What other people did. And somehow, it's all always relevant. She doesn't pretend to know what's going on under the hood, so she doesn't try to solve the problem and make it worse. She just explains what's going on, with full faith that I'll quickly make it right, which I do.

If I didn't mention, the project is a web site. It's not just any old web site, though. She manages the web systems for a school. Way back years ago, our intrepid web designer (she) got a cool job at a private (and really good) religious school in a place where the cows really do come home. With Dreamweaver in hand, she ran the site as well as she knew how, but was constantly backlogged with requests to evolve the web site.

My relationship with this client has always been symbiotic. I heard through a mutual frient that she was having trouble with the workload, so I approached her with a simple request to see her way of working. Without criticising, I simply observed for a while, and then said that I would come up with something which I thought she would like. She happened to be in the midst of a site re-design at the time, so it was the perfect opportunity to insert some new technology.

Soon afterward, I gave her a demonstration of a little tool I had come up with. I matched her web site pixel for pixel, but I did it within a relatively sophisticated content management framework. She was thrilled with the system, mainly because of two things:

1. It eliminated all of her multiple-maintenance headaches, as now every page had consistent menus and links and such.
2. It eliminated HTML completely, in favor not of a graphical editor, but rather of a super-simplified markup that any monkey could learn.

Between these two characteristics, she essentially never wrote another line of HTML. She will occasionally twiddle the menus, which are raw HTML chunks, but for the most part she can delegate responsibility out to the people who know what they want to publish, and those people are unable to screw the site up.

Her workload vanished overnight.

You can be sure that she became a faithful customer. In fact, she was so happy with the new system that she grew a 20-page site to well over 600 pages in the space of a few years. You can't manage that without automated tools, and I was there to provide them.

Naturally, with her new-found powers, my client also found new and creative ways to be frustrated. We never could have predicted in advance what would later seem the limiting factor in getting her home early on Fridays. Over time, we've talked about and implemented a few new features, and each time it makes lots of people in her organization really happy to have hired her.

If there's a moral to the story, I think it's this: You can only really know a client's needs when you've walked a few dozen miles in her mocassins. The next best thing is to develop a solid trusting relationship. Communicate openly and respectfully, and completely accept each other. Recognize that you are ignorant of her needs, and seek genuinely to understand. And never suffer a bug to live.

JSON, JSONT, and my take on them

JSON is cool stuff. Basically, there's no faster way to feed structured data to your browser. The arrangement is simple. The browser is really good at interpretting javascript, so we arrange to send it a literal javascript object in source code form. On the browser, it's an eval() and two parenthesis to a perfectly formed data structure. Naturally, we can save lots of server effort by delivering json to the browser as much as possible, and offloading the rendering gunk into javascript.

Maybe you can tell I'm headed into AJAX territory. Well, actually, I'm collecting parts for a framework I intend to call "Borax". So far, it's quite small, but I want to let it grow organically and see how far it gets.

So anyway, some genious by the name of Stefan Goessner came up with the brilliant idea to let you write a rules-engine based, essentially, on css-like selectors for parts of a potentially complicated JSON object. (Actually, that term is reduntant. JSO should read "JavaScript Object".) The rules are written in JSON, so he called this tactic jsonT, reminiscent of XSLT, which uses XML to transform XML.

If what you want is a document at the end, then jsonT is great. I've used it and liked it. I cannot recommend it highly enough for the generally simple case of turning a JSO into some text or HTML which you will then stuff into a document via something like the innerHTML property of a DOM node.

JsonT has also given me a few ideas.

I'd like to separate the object-tree-walking behavior, the rule matching behavior, and the rule execution behavior. Here's why:

The way jsonT is currently written, your basic default result is a text document. If you wish instead to produce a tree of DOM nodes based on the semantic content of your JSO, then all that string conversion and concatenation in the standard jsonT function gets in the way. It's hard to develop toward your goal incrementally. In this case, we want to change how subtrees are combined in general, and that means hacking up jsonT to produce a version that doesn't constantly lean toward strings.

Note that much of the behavior remains the same in both versions. If I had the parts in a separate and composable form, I could construct a new part that let me create DOM trees with less incremental effort. Of course, getting there is rather difficult in the short run. The jsonT code is rather dense, and I'm not really that great with JS as it is.

We'll see what happens.

A rant on prototype.js, and how to compile code.

I had gotten the hint that prototype.js is not the only game in town, so I looked up some stuff on this whole AJAX mess. Yes, it's pretty cool to combine all these technologies together. No, Sam Stephenson is not a saint. He is, however, quite a programmer.

From reading his prototype.js code, it's clear to me that he's spent quite a lot of time and effort cleaning up after browser bugs and incompatibilities. On the other hand, he also expresses a political agenda in this code, and one with which I disagree. No, it's not the MIT license. Actually, I'm happy that he used that one, because it means that I can copy the parts I like and gut out the rest.

"What's the agenda," you may ask.

It seems that our dear Sam has become so infatuated with Ruby's style of OO programming and block-passing that he undertook to recreate the same concepts, but with javascript syntax. There's nothing wrong with a little experimentation, but I think that trying to wrend language concepts at this level is misguided.

Sure, Javascript lets you pass functions around as parameters, and call them, and so forth. Sure, you can write anonymous functions, and place them anywhere in code as a value of the first order. Unfortunately, playing such games as he does with the syntax doesn't actually buy you much. Compared with a language like Smalltalk or Ruby where creating a callable block is about three or four characters, JS requires quite a bit more visual noise to create an anonymous function. There's more ceremony to it. It looks bulkier.

Maybe you want to write in a functional style. I've experimented with the same ideas in PHP. People who understand the general ideas involved will follow your code with difficulty because of the linguistic dissonance that comes from using a language built around Algol principles. You can't, for example, in PHP, create a partial (or curried) function without all the syntax of a whole function, or at least a fairly explicit function call which has to be explained in detail to the uninitiated.

Yes, I think Haskell is a great language. Yes, Lisp can do these things, but I don't plan to use Lisp in the near future. These are languages with support from the most basic foundations for composing parts of functions to express greater and more powerful wholes.

PHP and JS don't operate that way, by nature.

In Javascript, we're given a completely different worldview. Everything is not inherently a recursive algorithm which the compiler mysteriously changes into an iterative implementation when it's both possible and correctly incanted by the programmer. The basic idea is that people can express a set of steps easily, but trying to explain a process in terms of functions is more difficult.

Get that. Algol-style languages are about process. Lisp-style languages are about function. Both are valid perspectives, and each will yield great productivity for people who apply them well. When you apply the wrong technique, it's like trying to pound a nail using a screwdriver. You might get somewhere, but it's a lot more effort than it's worth.

Way back in the heyday of the structured-programming debates, someone wrote a book by the title of "Structured Programming". (Fancy that.) I don't recall the author, but I remember reading the book. In this day and age, so much of the results are taken for gospel that much of the thought process leading up to current structured programming practice seems forgotten in the noise and hype about the greatest new language.

Now, before you get your hackles up about the second-most-ancient programming paradigm being so old, please understand that all object-oriented programs are structured. Structured programming is just the idea that common themes found in the practice of programming may be factored away from the expression of any particular algorithm and replaced by something equivalent, more concise, and less error-prone. Structured programming (SP) is absolutely as relevant today as it was when the idea was first formalized.

You have to look at the development of the SP idea within it's historical context, though, to get a good understanding of why we are where we are today.

Early on, people who had been used to working with process flowcharts and dramatically smaller data processing tasks noticed that the most common overarching patterns were the loops, conditional branches, multiplexing, and function calls that now commonly populate reference manuals for Algol-family languages like C, Basic, Pascal, Java, Javascript, PHP, and so forth. We typically think of these as control structures. That's because the people who created and named them were out to solve problems of control. They didn't need to worry about higher-level concepts yet, because:

• control is the most basic quality of a process, and,
• these languages are based on expressing process.
  

There was one major thing that Lisp and its ilk did completely differently. Actually expressing the discrete steps of a process in Lisp is rather difficult, but you can do it once and trivially pass in executable code as a parameter to how the process should work. In short, you can (with some difficulty) make your own control structures in Lisp. If you know enough about how the Lisp interpretter (or compiler) works, then your control structures might even be computationally efficient.

Ruby is quite a nice language because you can create your own control structures fairly easily, and yet it's a process language at root, so it's easy for people to understand what's going on. In truth, there are limitations, though. If you want the really pretty syntax that Ruby offers for custom control structures, then in fact you're limited to structures that simply wrap a block of code. You can't, for example, create a C-style for(;;) loop from scratch purely with Ruby's syntax for custom control structures. On the other hand, I have little doubt that Lisp offers such ability, even if the actual syntax might not be so familiar to us process-description folks.

Sam is trying, with prototype.js, to create his own control structures. Javascript can be pushed in this direction, but it's an uncomfortable push. You face weird limitations and awkward syntax.

What you probably really want is more like a macro preprocessor that turns uber-Javascript into executable Javascript, and uber-PHP into executable PHP. You express your higher-level concepts for code organization in this other language, and then let the computer do the harder work of converting this smaller, easier, and less-error-prone version into something that the browser can actually run.

Some would say that you've re-created Lisp, or Scheme, or any one of its variants.

Imagine this sort of thing:

To reduce a sequence, given an initial value and a function of two values:

1. If the seqence is empty, return the initial value.
2. Remove the first element from the sequence and call it "head".
   
3. Apply the given function to the initial value and the head, and take the result to be the new initial value.
   
4. Go back to step one.
   

As you can see, it's not hard to express a totally general process using completely procedural English. The correctness of the algorithm should be self-evident, and you don't need a strong understanding of recursion to write the program correctly the first and only time. From then out, if your macro language allows you to compose definitions, you can say things like:

To get the sum of a sequence, give the initial value of zero and the function of addition to the reduction procedure. Return the result.

Presumably, your macro processor would be able to write all the code necessary in the target language to produce the desired result. In short, it would produce a function for summing sequences of numbers, and that function would work, and we would absolutely not care how or why it worked.

That is called compiling a program. The only difference here is that we're using something other than machine code for the target language.

Ok, rant over with.

Anyway, I've got deep dark plans on this subject for much later on.

Why did I start this blog?

Well sir, it's actually fairly simple. I wanted a centralized project notebook that I can hit from the internet. I wanted to be able to record my random thoughts. I want to be legendary, even if only in my own mind.

A blog might not be the world's greatest tool for this, but it should work, and require much less work on my part.

Furthermore, one of my pet "blow-up-the-universe" projects involves internet programming, so I figured I would harness some of that internet to shepherd the project along. We'll see.

Here, I intend to introduce myself, and generally get started.

For those who don't know me, I'm 25 years into a programming lifestyle that started when I was about five years old. Naturally, nobody could teach a five year old to program, so I had to do that part myself. Just as naturally, I didn't do the same kind of job that a formal teacher would have done. I just sort of played around on the Applesoft interpretter and went bananas when I found out about machine language.

I tried to major in CS in college, which should have been easy after so much hobby programming, but what actually happened was something else. I had not developed the qualities it took to get through certain courses of study, so eventually I stopped going to college, got some jobs in tech support, married, and kept on programming on the side.

All of this was well, but I had always wanted to be a real programmer. You know, the kind that makes money at it. One day, I found my opportunity. I was hired.

About four years later, I concluded that the life of a programmer was not for me. I joined the military instead, and now I have an outstanding career path with lots of opportunity. I work with a number of very talented and sharp people saving the world one day at a time. Oh, and did I mention the benefits? They're phenomenal.

I did not, however, give up programming.

You see, programming computers inspires in me a sense of magic, of absolute power that none can fully master lest it master the practitioner. I want to form worlds, shape consequences, and generally make cool things happen. I learned to do that on the computer.

These days, I divide my computer time between a handful of far-out blue-sky projects and one significant client who I seem to have completely won over. Actually, I'm no longer completely sure if she's with me because of satisfaction or a sense of vendor lock-in, but I have started a "pay-what-you-wish" campaign with her. Of course, it's difficult to get it past her accounting department, because I'm supposed to send invoices. I figure that eventually, we'll find a good solution. The fact is that I'm happy enough to keep working on the site, and she keeps coming up with ideas. It's a hobby for me now, a pleasant distraction from my real job, about which I shall speak no further.