Names in a traditional file system are made unique based on a full path to the file, but most people think of a file name as just the final component. This would then cause a problem with the move to Insight, as a file could appear in multiple directories, and its only distinguishing feature would be the final component of its path. This is counter-intuitive and can cause all sorts of problems.

Consider makefiles, for example. They rely on a standard named file (Makefile) appearing at various levels in the hierarchy in order to work. Obviously, you would want different makefiles at different levels and in different projects, but Insight as it stands has no way to handle this.

I then started thinking about what makes a file unique. In the end, I came up with two things: name and content. This covers the makefile case (same name, different content) as well as the backup case (same content, different name). It then occurred to me that, in the general case, all you need to distinguish a file is its content, and then actually finding it can all be left up to metadata.

If files are then thought of as containers for data that happen to have a unique internal identifier (which never needs to be exposed to the user, although it can be accessed as, say, the file's inode number) then the idea of a content-based file manager comes into play. These examples work best with visual media, particularly images, but there is no reason in principle that this could not be extended.

Imagine searching for a file. You know it's a photo, but you have a large collection of them. With a digital camera and a large-capacity memory card, who needs to ever delete a photo? We'll assume that you've dilligently tagged the photos with metadata as you've imported them from the camera, through some easy batch process.

On the tagging point: a lot can be taken from the metadata stored by the camera (date/time, resolution, orientation, black and white/colour, perhaps GPS co-ordinates) and with the right tools, more can be inferred (auto-tagging faces, buildings, perhaps recognising common events like football matches, converting GPS co-ordinates to places, ...). As time goes on, people will need to do less and less manual tagging.

Anyway, back to the file manager. You know you are after a picture or a set of pictures. Normal thought processes will probably follow a path similar to: "Yeah, I wanted to show dad those photos from that holiday in Paris that we had two months ago. I think he'd particularly like the ones we got of the Louvre, as well as the ones with me in, of course." I've highlighted various key words that can be translated directly to metadata searches. Notice how these all involve a narrowing down of the query.

To convert these to filters, we then have:

• type: photo
• holiday
• location: Paris
• date: two months ago
• at least one of:
  • location: Louvre
  • person: Me

This could also be represented by a query:

type:photo AND holiday AND location:Paris AND date:-2m
AND (location:Louvre OR person:me)

Breaking it down in this way feels fairly technical and wordy, however. I'd much prefer a visual view.

Imagine a black field, speckled with points of light representing your photos:

You filter by "holiday", and (because it learns based on previous searches) it then groups by location. The ones which have been filtered out fade into nothing, and the photos group into labelled blobs and enlarge slightly:

You filter by date, and as you drag the slider, irrelevant items fade away and relevant ones enlarge:

Then you add the final filters and set the photos up for viewing, perhaps as a slideshow... and you're done!

Pretty neat, I think.

Apache module

The first thing I have in mind is to create an Apache 2.2 module that uses the APR (Apache Portable Runtime) to make configuration file templating easy.

On my server, there are a lot of very similar Apache configuration files. At the moment, they are semi-automatically generated by running a Perl script that reads the configuration from a database and then creates the output files as required. This is all well and good, but changing anything requires running the Perl script, regenerating all of the configuration files, then prodding Apache to reload its configuration. There has to be a better way, especially one that doesn't require elevated permissions like that.

Enter my proposed solution: mod_sqltemplate. This module will add a few extra settings and (at least) two extra sections to the standard Apache configuration. The propsed features are:

• SQLTemplate_DBDriver «driver» – sets the APR driver to be used
• SQLTemplate_DBOpts «option-string» – options for the driver (e.g. database name, username, password...)
• SQLTemplate_Caching «on|off» – enable/disable configuration caching
• SQLTemplate_CacheDir «path» – directory in which to save cached configurations
• <SQLRepeat «query»> – new section, as described below
• <SQLCatSet «sep» «query»> – new section, as described below

SQLRepeat

The SQLRepeat section is probably going to be the most-used new directive. It provides templating functionality similar to mod_macro, although the template variables are taken from the column names of the query rather than being explicitly specified. The section will then be repeated for each row of the resultset, substitutions occurring as required.

Perhaps an example would help:

<SQLRepeat "SELECT id,hostname,domain FROM apache_hosts">
<VirtualHost *:80>
  DocumentRoot /var/www/${hostname}.${domain}
  ServerName ${hostname}.${domain}
  # ...
</VirtualHost>
</SQLRepeat>

For this example, I've avoided the configuration directives for clarity, assuming they are specified elsewhere. Supposing the database table contains simply:

Then this will expand to be equivalent to:

<VirtualHost *:80>
  DocumentRoot /var/www/www.example.com
  ServerName www.example.com
  # ...
</VirtualHost>
<VirtualHost *:80>
  DocumentRoot /var/www/test.example.com
  ServerName test.example.com
  # ...
</VirtualHost>
<VirtualHost *:80>
  DocumentRoot /var/www/yourhost.somewhere.local
  ServerName yourhost.somewhere.local
  # ...
</VirtualHost>

And so on. These sections can be nested, so if you wanted to store a list of aliases as well:

You could use something like this:

<SQLRepeat "SELECT id,hostname,domain FROM apache_hosts">
<VirtualHost *:80>
  DocumentRoot /var/www/${hostname}.${domain}
  ServerName ${hostname}.${domain}
  # ...
  <SQLRepeat "SELECT alias,target FROM apache_aliases WHERE hostid=?" ${id}>
    Alias "${alias}" "${target}"
  </SQLRepeat>
</VirtualHost>
</SQLRepeat>

which would then expand to:

<VirtualHost *:80>
  DocumentRoot /var/www/www.example.com
  ServerName www.example.com
  # ...
    Alias "/icons" "/var/www/_share/icons"
    Alias "/example" "/var/www/_share/foo"
</VirtualHost>
<VirtualHost *:80>
  DocumentRoot /var/www/test.example.com
  ServerName test.example.com
  # ...
</VirtualHost>
<VirtualHost *:80>
  DocumentRoot /var/www/yourhost.somewhere.local
  ServerName yourhost.somewhere.local
  # ...
    Alias "/bar" "/var/www/_share/foo"
</VirtualHost>

So far, so good. Something to note is that the variable names above did not conflict. If we had included "id" in the list of columns for the inner SQLRepeat, then included ${id}, the module should use the innermost scope. Disambiguation should be obtained by table alias prefixing (e.g. ${apache_hosts.id} or ${apache_aliases.id}). Full variable names should always be specified for clarity. This also shows that if the query returns no results, the repeating section is not included, as you would expect. Also note that queries that use variable substitution should use the prepared statement format, with variables specified afterwards. Variables will be included directly, so anything that might contain spaces must be quoted.

However, there are some directives like ServerAlias that must be specified once with a list of space-separated values. Enter SQLCatSet...

SQLCatSet

This acts in a similar way to SQLRepeat, but it concatenates all of the results for each field before substituting, and it includes its content at most once. If there are no results for the query, then it will not include its content. For example, with a server aliases table like:

You could use something like this:

<SQLRepeat "SELECT id,hostname,domain FROM apache_hosts">
<VirtualHost *:80>
  DocumentRoot /var/www/${hostname}.${domain}
  ServerName ${hostname}.${domain}
  <SQLCatSet " " "SELECT alias FROM apache_server_aliases WHERE hostid=?" ${id}>
    ServerAlias ${alias}
  </SQLCatSet>
  # ...
  <SQLRepeat "SELECT alias,target FROM apache_aliases WHERE hostid=?" ${id}>
    Alias "${alias}" "${target}"
  </SQLRepeat>
</VirtualHost>
</SQLRepeat>

which would then expand to:

<VirtualHost *:80>
  DocumentRoot /var/www/www.example.com
  ServerName www.example.com
  # ...
    Alias "/icons" "/var/www/_share/icons"
    Alias "/example" "/var/www/_share/foo"
</VirtualHost>
<VirtualHost *:80>
  DocumentRoot /var/www/test.example.com
  ServerName test.example.com
    ServerAlias test2.example.com test.example.net
  # ...
</VirtualHost>
<VirtualHost *:80>
  DocumentRoot /var/www/yourhost.somewhere.local
  ServerName yourhost.somewhere.local
    ServerAlias *.somewhere.local
  # ...
    Alias "/bar" "/var/www/_share/foo"
</VirtualHost>

This would seem to cover the major functionality needed for automatically generating configurations in a really flexible way, although I'm sure more will probably be needed at some point... like SQLIf, perhaps.

PHP Framework

It occurs to me that having a framework's code in PHP itself could mean that it is quite slow. It would be interesting to put together a framework in C and embedded as a module, to see if there are any memory/speed improvements possible. It might just be a case of implementing an ORM (Object Relational Model) system to begin with. Obviously, this idea is nowhere near as fully-formed!