version proto

      character paul
        image paul.png
      end

      character angie
        image angie.png
      end

      angie says I thought you were going to bed!
      paul say But I really wanted to try and write a comic markup language prototype.
      angie says What? What the heck is wrong with you? Go to sleep.
      paul says Just give me a half hour.
      angie says Ok, fine, but only a half hour. Crazy person.

only resulted in the following output:

Unrecognized command at line 3: "character"

Ah well, there’s always tomorrow…

While playing around with Nitrogen, which is a great project by the way (more in a later blog post, I guarantee it), I noticed that I had to restart the server to re-load modules as I changed them (or I could manually, reload them, I know, I know..).

Since I was using Mochiweb as my backend, and I’m used to the development version of Mochiweb auto-reloading modules as you recompile, I wanted the same functionality here.

Turns out it’s as simple as changing the default start.sh from

#!/bin/sh
cd `dirname $0`
 
echo Starting Nitrogen.
erl \
	-name nitrogen@localhost \
	-pa ./ebin -pa ./include \
	-s make all \
	-eval "application:start(appname)"

to this

#!/bin/sh
cd `dirname $0`
 
echo Starting Nitrogen.
erl \
	-name nitrogen@localhost \
	-pa ./ebin -pa ./include \
	-s make all \
	-s reloader \
	-eval "application:start(appname)"

… and bam, modules now reload as you recompile them.

It seemed to me that it needn’t be as complex as it was, so I decided to try and write my own, and here’s what I came up with:

function growTextArea(e) {
    if (!this.rows || this.rows < 1) this.rows = 1;
    while (this.clientHeight >= this.scrollHeight && this.rows > 1) this.rows -= 1;
    while (this.clientHeight < this.scrollHeight) this.rows += 1;
}

A working example, embedded using jQuery:

$('#example-resizing-textarea')
    .keyup(growTextArea).keyup();

Type some text in here:

Compatibility

This works for me in Firefox, IE6, IE7, Chrome and Chromium, Safari (for Windows, testing in a Windows VM along with the IEs and Chrome), and Opera 9.62. As you may have noticed I’ve avoided using any JS-library or browser-specific code, so this should work equally well with jQuery, Prototype, or with no library at all.

IE actually gives incorrect values for scrollHeight, but that is easily made up for by using “this.clientHeight >= this.scrollHeight” in the first while loop rather than “this.clientHeight == this.scrollHeight”

Minimum and Maximum rows

If you want to set a minimum and maximum height, it takes a little more work:

function sizeTextArea(min, max) {
    return function(e) {
        if (!this.rows || this.rows < min) this.rows = min;
        while ((this.clientHeight >= this.scrollHeight 
                && this.rows > 1 && this.rows <= max) 
               || this.rows > max) this.rows -= 1;
        while ((this.clientHeight < this.scrollHeight 
                || this.rows < min) 
               && this.rows < max) this.rows += 1;
        if (this.rows == max
            && this.clientHeight < this.scrollHeight) this.style.overflow = 'auto';
        else this.style.overflow = 'hidden';
    }
}

Another example, limited to having between 4 and 8 rows:

var sizer48 = sizeTextArea(4, 8);
$('#example-resizing-textarea')
    .keyup(sizer48).keyup();

Type some text in here, too:

Possible modifications could include using pixel sizes instead of changing the “row” attribute and animating the resizing.

This could easily be packaged into a plugin for jQuery or Prototype, and I might just do that, but right now it’s getting late, so I’m going to sleep instead.

For some reason, Apache had been dying on me every few days.

I finally looked in my error.log file, and saw the output of wget, downloading a file called k.c from http://66.90.103.116/k.c (they’ve taken it down since, sometime within the last couple of hours. Perhaps they noticed my poking around at their server). Anyway, somehow they were getting in through apache somewhere, downloading this file, compiling it, and running it.

The code itself (a derivative of kaiten.c logs into an IRC server and watches for commands on which servers to attack or commands to run.

I eventually located the point of entry as a bug in the version of Roundcube webmail I was running. I was running a really old version of it (something like version 0.1 beta, I think), and all it took was an apt-get install of the latest version and the security hole is gone.

I’m going to add the news feeds of any software I’m using on this server to my RSS reader in the hopes that I will find out about such security holes (or at least software updates) a bit sooner in the future. I’ll also try to make it a habit to do a apt-get upgrade more often as well.

EDIT: I’ve set up cron-apt as suggested, so that should help me keep things up to date.

It’s handy for those times where I think of something to do while working on a project, but I am already working on something. With this I can just append it to the end of my queue for that project and forget about it until later.

The code can be gotten from bitbucket, if you are interested.

You can use it like so:

Push items to the top of a queue:

$ q todo push 'do that one thing'
do that one thing
$ q todo push 'do that other thing'
do that other thing

See what’s at the top of a queue:

$ q todo
do that other thing

Pop items from the top of a queue

$ q todo pop
do that one thing

Append items to the end of a queue:

$ q todo append 'do some more things'
do that one thing

The help q prints out if you call it with no parameters:

$ ./q.py 
usage: ./q.py queuename [command [params]]
  commands:
    push       Add one or more lines to the queue
    all        Show all the items in the queue
    append     Add one or more lines to the end of the queue
    pop        Remove the top item from the specified queue and push it onto the .done queue for that file
    next       (default) Return the top item in the queue

Other commands I plan on adding include “rot” to swap the current top item with the next item (or with a numerical parameter to move it even further down).

I’ve been experimenting around with Mercurial, and since that was also written in Python, I was wondering what it would take to rewrite this as a hg plugin. It could be handy to use something like this to manage a todo file. It could add some output to show what todo items were marked as done for each commit and perhaps also do a pre-commit hook where it pulls any new lines starting with “TODO:” and adds them to the todo file.

I must say that I agree with Ben Ramsey (see “arguments agains” above) in suggesting we use rel=”alternate shorter” instead.

I also like the idea that Chris Shiflett had of using a HTTP header and a HEAD request to make it so you neither have to retrieve the entire requested page nor parse any HTML. I’d stick with Ben’s suggestion, however, and make the header something like “X-Alternate-Shorter:”, rather than “X-Rev-Canonical”. What’s the harm in calling it something that actually makes sense?

The idea of using HTTP HEAD requests to solve the problem inspired me to come up with a more immediate solution to one of the problems introduced by using url shortening services: uncertainty about where a URL leads.

This problem can be solved on the client side, which requires no work on the part of Twitter (meaning this is more likely to be put into use sooner).

Since most URL shortening services use an HTTP redirect to do their job, all it takes is a HEAD request to the tiny URL in question, and then a look at whatever “Location:” header is returned to see what the real URL is. In fact, you don’t even really need to do a HEAD request in most cases, since most URL shortening services don’t return any body, since they are just redirecting you anyway.

Read on for more information and implementations of an untinyurl function in various languages.

There’s actually already a site online that offers the service of un-shortening URLs for you at UnTinyURL.com, but I wouldn’t suggest using that in any sort of automated system, and it’s of limited usefulness since you don’t really want to have to go to this site just to see what site you’re about to go to. Most people will just click a link, even if it means they might get RickRolled.

For those comfortable with the commandline, a simple curl call can give you the same basic info:

curl -I http://tinyurl.com/c8f5bz
HTTP/1.1 301 Moved Permanently
X-Powered-By: PHP/5.2.9
Location: http://blog.paulbonser.com/2009/04/11/untiny-that-url/
Content-type: text/html
Date: Sun, 12 Apr 2009 01:26:08 GMT
Server: TinyURL/1.6

Toss in a grep and an awk, and you get your URL in a single line, perfect if you’re handing shortened URLs in a shell script for some reason:

$ curl -s -I http://tinyurl.com/c8f5bz | grep Location | awk '{print $2}'
http://blog.paulbonser.com/2009/04/11/untiny-that-url/

Here’s untinyurl in Python:

import httplib
import urlparse
 
def untinyurl(tinyurl):
    url = urlparse.urlsplit(tinyurl)
    req = urlparse.urlunsplit(('', '', url.path, url.query, url.fragment))
    con = httplib.HTTPConnection(url.netloc)
    try:
        con.request('HEAD', req)
    except:
        return None
    response = con.getresponse()
    return response.getheader('Location', None)

And here’s a version in PHP. It’s a bit longer and uglier than the Python version because I’m using the low-level fsockopen function to do my HTTP request rather than using cUrl or the HTTP extension. The reason I did this is because every PHP install will have fsockopen, whereas not every install will have cUrl or the HTTP extension.

<?php
 
function untinyurl($tinyurl) 
{
    $url = parse_url($tinyurl);
    $host = $url['host'];
    $port = isset($url['port']) ? $url['port'] : 80;
    $query = isset($url['query']) ? '?' . $url['query'] : '';
    $fragment = isset($url['fragment']) ? '#' . $url['fragment'] : '';
 
    $sock = @fsockopen($host, $port);
    if (!$sock) return $tinyurl;
 
    $url = $url['path'] . $query . $fragment;
    $request = "HEAD {$url} HTTP/1.0\r\nHost: {$host}\r\nConnection: Close\r\n\r\n";
 
    fwrite($sock, $request);
    $response = '';
    while (!feof($sock)) {
        $response .= fgets($sock, 128);
    }
    $lines = explode("\r\n", $response);
    foreach ($lines as $line) {
        if (strpos(strtolower($line), 'location:') === 0) {
            list(, $location) = explode(':', $line, 2);
            return ltrim($location);
        }
    }
    return $tinyurl;
}

I’m not too familiar with Ruby, but after poking around for a little bit, I came up with this Ruby version. Holy crap, Ruby, that was easy and short!

require 'net/http'
require 'uri'
def untinyurl(tinyurl)
  Net::HTTP.get_response(URI.parse(tinyurl))['location'] or tinyurl
rescue
  tinyurl
end

And one more, and Erlang implementation (that <SEMI> is supposed to be a semicolon, but something is wrong with the syntax highlighter Erlang plugin). Be sure you call “inets:start()” before calling this function.

-module(untinyurl).
-export([untinyurl/1]).
untinyurl(TinyUrl) ->
    case http:request(head, {TinyUrl, []}, [{autoredirect, false}], []) of
        {ok, {_Status, Headers, _Body}} -> proplists:get_value("location", Headers, TinyUrl);
        _ -> TinyUrl
    end.

Interesting how the Erlang and Ruby implementations look pretty similar.

I’ve made the source code available at GitHub. If you would like to contribute an untinyurl implementation in another language or have a bug-fix or suggestion for an improvement of one of the implementations I have so far, either email me, send me a pull request on GitHub, or post a comment here.

Basically, a small PHP script was put on the remote server which took a couple of GET parameters and set some cookies based on their values. The script then output a 1×1 transparent GIF. A PHP script on the local server generated an IMG tag which linked to this image and set the parameters based on the COOKIES on the local server.

This process also had to happen in the reverse direction, so I had to send the script to developers on the other side. I wanted to keep it as simple as possible, so I put the actual image contents in the PHP file as a base64 encoded string. I used the GIMP to generate the smallest transparent GIF I could manage so there wouldn’t be too huge of a nasty string in the file. I came up with something like the following:

<?php
setcookie('foo', $_GET['foo']);
header('Content-Type: image/gif');
echo base64_decode('R0lGODlhAQABAIABAP///wAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==');
?>

Remembering this got me to wondering, how small could you make a GIF? The file generated by the GIMP was only 43 bytes, but it seemed to be that you should be able to make a file which is representing a single pixel a bit smaller than that.

So, with equal parts of determination and derangement, I set about finding out.

Though of somewhat dubious usefulness, I managed to generate a perfectly valid GIF of only 26 bytes in length, which has the potential to display completely differently in various different software.

Read on to see how I found my way to this point.

Starting point: the GIF89a specification

The file generated by the GIMP, as I said, was 43 bytes long. You can find it here: tinytrans.gif

To understand what was taking up all those bytes, I read through the GIF89a Spec. The file tinytrans.gif above consists of the following sections:

• Header, 6 bytes: Consists of the bytes “GIF” and the version number, which is usually “89a”.

• Logical Screen Descriptor, 7 bytes: Without going into too much detail, this section of the file indicates the following:

  • The file is 1×1 pixels in size
  • There is a global color table
  • There are 2 colors in the global color table, the second one should be used as the background color

• Global Color Table, 6 bytes: Consists of 3 bytes per color, a byte for red, green, and blue, respectively. In our file, the first color is white an the second color is black.

• Graphic Control Extension, 8 bytes: Used to indicate that the second color in the color table should be treated as transparent (can also be used for animation parameters, but isn’t in this file)

• Image Descriptor, 10 bytes: A GIF file can actually contain multiple “images” within it, which keeps you from having to specify image data for parts of the image which have the same color as the background color. Each image block has a position and size within the overall image size. In the above file, the position is 0,0 and the size is 1×1.

• Image Data, 5 bytes: One LZW-encoded block of image data. It takes 5 bytes to represent the single pixel the image has in it. The compression algorithm wasn’t designed to compress a single byte very well.

• GIF Trailer, 1 byte: A single byte with a hex value of 3B (”;” in ASCII) indicates the end of the GIF.

Making a smaller GIF

Based on the required structures for a transparent GIF, it turns out that 43 bytes is pretty close to as small as you can get.

But, I managed to figure out one trick to make it a bit smaller. It’s mentioned in the standard that it is optional to have a global color table. Of course, it’s undefined as to what happens when you make a GIF without a color table at all (there are also optional local color tables per image block).

When you have a color table index defined as transparent, however, GIF decoders don’t seem to care that there isn’t actually a color table.

So I changed the logical screen descriptor to indicate there was no global color table and removed the table itself, saving a total of six bytes, bringing the file size down to a mere 37 bytes.

Here it is in all its glory: handtinytrans.gif

Interestingly enough, Wordpress gave me a lovely list of error messages of GD complaining that this isn’t a valid GIF file, despite the fact that Firefox and the GIMP both open and display (is it “displayed” when it’s transparent?) the file just fine.

So there you go, the tiniest transparent GIF possible (if you can make one smaller, let me know).

But of course, there’s room still for the tiniest GIF ever.

Even smaller, no longer transparent

To make it even smaller, I looked to the biggest remaining “optional” block in the image, the graphic control extension. If you don’t need transparency, this block is no longer needed, and that’s another 8 bytes you can take away.

Sadly, I wanted a whilte image, and I couldn’t get it to come out white without adding the global color table back in, so I only managed to trim 2 bytes off of the file.

Still, 35 bytes is indeed smaller: handtinywhite.gif

The tiniest possible GIF

If I remove the constraint of having the image be white (or any color I choose), I can make it even smaller, though admittedly this becomes much less useful.

I removed both the global image table and the graphic control extension, and then no matter what color the pixel originally was, it comes out black. From there, I took it a step further, because the standard never says you have to actually set any pixels in the image, I removed the LZW encoded image data. I left the required lzw code-size header and block terminator, because without those Firefox and the GIMP both complained it wasn’t a valid GIF, and then I had it.

The tiniest GIF ever, 26 bytes: handtinyblack.gif

Despite the fact that this is called “handtinyblack.gif”, black is just the color that the GIMP chooses to display when there’s no global color table. It’s stated in the spec “If no color table is available at all, the decoder is free to use a system color table or a table of its own.”

This means that any GIF decoder is free to pick whatever color it wants for this image. Opening this image in Firefox at various different times, it has show up as shades of blue, green and brown, as well as transparent if set as a background image.

So I’ve started writing my own. So far I’ve got the first necessary step complete: HTML-based discovery.

I’m starting with version 1.1, simply because it is shorter and requires less (I don’t want to implement the XRI or Yadis protocols just yet).

It turns out that mochiweb comes with an HTML parser, so I used that, since I’m using mochiweb for my application. The parsed HTML comes back as a series of nested tuples of the format {<<”tag”>>, Attributes, Children}, where “tag” is the tagname (the root will be <<”html”>>, for example), Attributes is a proplist of that tag’s attributes, and Children is a list of more tuples of the same format and/or binaries with the contents of text nodes. Everything is represented as binaries, so I use those directly rather than converting between strings and binaries.

Here’s the code which finds the link tags with rel=”openid.server” and rel=”openid.delegate” (if it is there):

get_openid_server(Identifier) ->
    NormalizedIdentifier = normalize_identifier(Identifier),
    case http:request(NormalizedIdentifier) of
        {ok, {_Status, _Headers, Body}} ->
            HtmlTokens = mochiweb_html:parse(Body),
            find_openid_tags(HtmlTokens);
        _ ->
            {error, http_error}
    end.
 
normalize_identifier(Ident = "http://" ++ _Rest) ->
    Ident;
normalize_identifier(Ident) ->
    "http://" ++ Ident.
 
find_openid_tags(HtmlTokens) ->
    case find_tag(<<"head">>, [HtmlTokens]) of
        {<<"head">>, _Attrs, Children, _Rest} ->
            case find_tag_with_attr(<<"link">>, {<<"rel">>, <<"openid.server">>}, Children) of
                not_found ->
                    {error, openid_server_not_found};
                ServerAttrs ->
                    Server = proplists:get_value(<<"href">>, ServerAttrs),
                    case find_tag_with_attr(<<"link">>, {<<"rel">>, <<"openid.delegate">>}, Children) of
                        not_found ->
                            [{server, Server}];
                        DelegateAttrs ->
                            Delegate = proplists:get_value(<<"href">>, DelegateAttrs),
                            [{server, Server}, {delegate, Delegate}]
                    end
            end;
        not_found ->
            {error, no_head_tag}
    end.
 
find_tag(_TagName, []) ->
    not_found;
find_tag(TagName, [{TagName, Attributes, Children} | Rest]) ->
    {TagName, Attributes, Children, Rest};
find_tag(TagName, [{_OtherTag, _Attributes, Children} | Rest]) ->
    find_tag(TagName, Children ++ Rest);
find_tag(TagName, [_Other | Rest]) ->
    find_tag(TagName, Rest).
 
find_tag_with_attr(_TagName, {_AttrKey, _AttrVal}, []) ->
    not_found;
find_tag_with_attr(TagName, Attr = {AttrKey, AttrVal}, Tags) ->
    case find_tag(TagName, Tags) of
        not_found ->
            not_found;
        {TagName, Attributes, Children, Rest} ->
            case proplists:get_value(AttrKey, Attributes) of
                AttrVal ->
                    Attributes;
                _ -> 
                    find_tag_with_attr(TagName, Attr, Children ++ Rest) 
            end
    end.

(the <PIPE>s above should be “|”s and the <SEMI>s should be “;”s. Not sure why the syntax highlighter is doing that to them…

Assuming the above code is put into a module called “openid”, you get the following:

1> openid:get_openid_server("blog.paulbonser.com")
[{server,<<"http://www.livejournal.com/openid/server.bml">>},
 {delegate,<<"http://misterpib.livejournal.com/">>}]
2>

As I said, this is the first step. Hopefully I’ll have some time very soon to get on with the next couple of steps, and then I’ll be done.

Don’t complain if it doesn’t conform, because it’s not a fixed standard yet.

Have you ever dreamed of a language which uses strictly prefix (a.k.a. polish, Łukasiewicz) notation?

No? Well, I can’t say I’m surprised. Lisp is often called a prefix notation language, but I’ll let you in on a secret, it’s not purely prefix notation. It uses another notation you’ve probably never heard of: outfix notation.

Outfix notation?

I’d say I made outfix notation up, but I found a reference to it on abstractmath.org, so I at least have something to back this claim up with. Basically, the parentheses are a function which says, “put these items into a list.”

Of course, Lisp uses lists for everything, so you can hardly call it a prefix notation language any more.

Real prefix notation

Now, how about making a real prefix notation language? A real prefix notation language needs no parentheses because it knows how many arguments each function takes, so it can simply pull in the next two expressions following the function name.

A real prefix notation language is a piece of cake to implement, as long as every function has a fixed arity and that arity is known at compile time. Of course, then how do we represent things such as lists with varying amounts of items. How do we pass a variable number of arguments to a function?

The same way Lisp does, we use a list.

Using lists without list syntax

Of course, we have to encode our list a bit differently. For example, take the following simple Lisp snippet:

(setq numbers '(1 2 3))

The set function takes a fixed number of arguments, so that’s not a problem. The list at the end, how do we do that? Well, if we assume we have standard lisp functions, we do this:

setq numbers cons 1 cons 2 cons 3 nil

Now that wouldn’t look half bad if it weren’t for the conses of doom at the end there. So what can we do? Well, Lisp has these handy things called macros. Compile-time functions, basically.

Compile time functions

Of course, since we don’t know ahead of time how many items are going to be in the list. Lisp hands macros a list of the arguments to the macro based on what was in the parentheses with the macro name, but once again, we don’t have (or really want, honestly. I’m trying to do something new here!) the parentheses, so this is where we must start deviating from Lisp a bit.

setq numbers list 1 2 3 .

What’s happening here, is that list is a compile-time function which simply requests items from the reader (the thing that takes characters and interprets them into symbols, numbers, and strings) until it gets back the ‘.’ symbol. If it’s anything else, it just has to tell the compiler to compile it as it would a normal expression. This means that we can call functions as we please. The difference between this an a “normal” outfix operator is that it is controlled by the code itself, rather than built into the compiler. The idea is that you could define your own compile-time functions to extend or change the “syntax” as you see fit.

For example, if you think the above list function is a bit too wordy:

def-compile ( : 
  list-while /= setq token reader-next
                   ')
    compile token

Assuming “list-while” is a function which will build a list with the value of each iteration, this code does the following:

• Define a compile-time function bound to the symbol “(” which does the following:
  • Read the next token
  • If the token is not “)”, have the compiler compile the token as usual, returning the result
  • If the token is “)”, then we are done.

With this new compile time function, we can write nested lists as easily as we can in lisp:

setq nested-numbers ( 1 2 ( 3 4 5 ( 6 7 ) ( 8 ) ) )

Well, that’s almost as easy as lisp, there’s one problem: those spaces are mandatory. Since “(” and “)” are just plain old symbols, the whole thing would be read as one symbol if we took the spaces out. This can be solved by modifying the reader to recognize “(” and “)” as complete symbols.

Reprogramming the reader

Without going into too many specifics of how the reader is implemented, making it possible to leave out the spaces around parentheses would be something like the following:

push-reader (
  expr "(" ( add-char output-symbol 
    push-reader (
      expr ")" ( add-char output-symbol pop-reader )
      symbol ")" ( output-symbol ) ) )

This makes the reader treat a single “(” as a symbol when it’s expecting an expression, and also pushes a new reader state including “)” onto the reader stack. This is popped off when a “)” is found. This means that “)” will stop being special once the one matching the first “(” is reached. This shows a bit of the power of the reader implementation, allowing the syntax to be extended for just a portion of the compilation.

For example, something like the following wouldn’t be too hard to implement using this system:

setq json-snippet json {
  "a": [1, 2, 3], 
  "b": ["x", "y", "z"], 
  "foo": "bar"
 }

It would simply have to modify the reader to recognize “{”, “}”, “[", "]“, “,”, and “:” as separate symbols with or without whitespace, and add functions for “{” and “[” which also take the commas and colons into account.

No longer lisp, but lisp-like

At this point what we’ve got probably can’t be called “a Lisp” anymore.

There are some things that this language will need that Lisp doesn’t. One thing is that it will need to keep track of functions as they are defined in the code, noting how many arguments they take so it can handle them being called later on. When a function is passed as an argument to another function, it will need to be called using something along the lines of funcall or be given a manual arity hint if you want to call it like other functions.

The “everything is a list” feature of Lisp is kind of lost, though the internal representation could end up being basically the same.

This language doesn’t exist … yet

This is something which is more concept in my brain and on paper at the moment. I’ve done some prototype implementations of the reader, some prototype implementations of the prefix notation function calling, and some prototype compile-time functions (implemented in the host language, however).

What’s left now is to put the whole thing together into a system in which every part is well defined and accessible from within the language itself.

I’m also thinking that I’d like to build this thing from the ground up with an underlying object-oriented system. Not class-oriented, mind you, just objects. Something prototype-based I’m thinking, like JavaScript, Io, Pepsi, and so on. In the end a user-mutable object-functional language is what I think I’ll end up with.

Any comments, suggestions, ridicules, etc… would be well appreciated. Also, if anyone has seen anything like this already in existence from which I could draw inspiration and ideas, please let me know. I’ve already looked at Cola/Coke/Pepsi and gotten some good ideas from there.