Is it time for a new computer language?

It might be. Or not. But here are my thoughts:

Let's compare Perl and Python for a few moments. No right-minded person will try to tell you that Perl is somehow beautiful, but Python is clearly beautiful. It's also broken for just that reason.

Perl has flaws. Python has deeper ones, and they stem from the very fact that Python is so beautiful.

Take regular expressions.

• In Python, you give an ordinary string to the regex compiler, and you get back an object. That object can do things like search for or replace matches in some target string.

• In Perl, you have special grammar that lets you talk about regular expressions and the things they do with strings.

You might claim that Python is better because it's more consistent. English is not consistent, and most of us do pretty well with it. English is better than Python. (More people use it.)

Let's take another look. How do you get the regular expression into the system?

• In Python, you must first escape anything that has special meaning to the regex compiler, and then you must escape anything that has special meaning to the string literal quoting system. That means you must type four backslashes to match a single one in the target string. That's confusing and hard to read.

• In Perl, you quote once for the regex mini-language, and that's it.

True, regular expressions are a special case of a mini-language embedded into Perl. But in reality, they're so fundamental to the things you do with Perl that it just makes sense for regular expressions to be part of the base grammar of the language.

That's where my plan starts to diverge.

Really, there's a whole category of languages for matching strings. Globs are one such mini-language, familiar to shell users in the form of * and ? marks for selecting a group of files that match a pattern.

Therefore:

• There's a good reason to have grammar that introduces mini-languages directly into the syntax of the surrounding language.
• It should be possible to define a mini-language in terms of the available grammar features.

In fact, I would go beyond this to say that it's better for the language to provide grammar, but let the programmer extend what's meant by the various grammatical forms. This is like human languages. The grammar stays the same, though whole vocabularies may exist within specialized domains.

In a typical programming language, the grammatical elements you get to extend are extremely limited. Python's handling of regular expressions demonstrates that regular expressions were added into Python in the same manner as other extensions. That means that you get the same inadequate grammar for regexes in Python as you do for every other object oriented extension. What's really going on is that you want to bind input (a string) to a subprogram in a domain-specific language which happens to be called "Regular Expression". There are tons of times when this concept would be useful, but you just can't do it in any programming language I know of.

Some claim that Scheme lets you do something like this. The classic example is the finite state machine. Given a syntax definition, you can write up a state machine in scheme that looks a lot like what you might write on the chalkboard. The problem is that there really aren't very many applications for this. Given one really good way to write down FSMs in your favorite progamming language, it's done deal, right?

Well, Scheme is nice that way, but can you create a regular expression compiler with this mechanism? Perhaps, but it will be ugly as heck. That's because you don't actually have full control over the design of the embedded language.

It's time we took a step back from our programming systems and asked what ways we want to be able to extend languages, and not just add modules. The proper extensible language, for example, might be able to treat hash-table lookup as a control structure: either you find what you're looking for, in which case you execute the "if part", or you don't, in which case you execute the "else part". Here again, Scheme and Lisp can already do this particular trick, but their grammars are so spare that they don't give any context clues.

Real human languages put a certain amount of redundancy in the grammar. Each language does it differently, but they all have it. That's why C is easier to read that Lisp. C has redundancy in the grammar. Code is surrounded by braces, and virtually nothing else is. Parenthesis are used either for ordering algebraic operations, or else for invoking functions, or else in certain other grammatical forms that are not confusing. On the other hand, Lisp puts parenthesis everywhere, and it uses them (and nothing else) for everything. Therefore, at first glance, it's hard to tell from looking at the code what says what, especially if you're new to the vocabulary in use.


Let's look at the operations people do with their programming systems to figure out what grammatical forms to provide. Let's establish a concise core, and provide powerful ways of extending the system in terms of grammatical forms, rather than only function calls. Let us provide a standard set of modules that provide the vocabulary for various problem domains. And let us not forget that we need to be able to let people build domain-specific languages into the whole system, and have them appear nearly as if they were designed in from the start.

That's a tall order, but I think it might make as much difference as high-level programming languages did. Why? Mainly because it would make programs easier to read as well as write.

In a future article, I might describe the design of a grammar meant for extension.