Insight braindump
Well this is going to be a very random and unstructured post, but I just wanted to note down a few thoughts and ideas before I forget about them... and I'll probably update this a couple of times before finally giving in and making it a page. This is an evolving post.
Insight: Implementation Thoughts
So, I've been reading the book about BFS (the file system of BeOS), which shares some ideas with mine, although it still adheres to the idea of a hierarchical filesystem. It goes into some depth, and although I've only reached partway through chapter 6, I have found it to be a very useful resource, particularly with a brief overview of the heuristics of search optimisations.
Anyway, the key thing recently is that I've started to think about how to implement the data structures for DBFS that deal with storing the attributes, and I feel that I have come across a rather neat solution. Whether it is efficient or fast is another matter entirely, but that is definitely not the point of this project; I just want to prove it is possible. If performance isn't too bad, then so much the better.
Insight: some thoughts
Apologies to anyone this confuses, but this post is just a collection of random thoughts about DBFS (now called Insight); what it might do, how it might be implemented... all sorts of things. Some of these will be taken from The Book of the Project, and some have developed as I do more research.
Beginning research
So I've begun my research with Hans Reiser's The Naming System Venture whitepaper, which is now no longer available online. Unfortunately the Wayback Machine doesn't provide the diagrams, so unless I can find anyone with a mirror of it (so far unsuccessful) then I shall just have to do without them. Incidentally, I am also writing it up again myself, so I will make a PDF available here.
The more I read and re-read this paper, the clearer some of my own ideas become, and it seems to me that my initial intuition that I would have to write my own database system seems more valid. The only thing is that I had been thinking in terms of a relational database, as that is all that I have really had experience. I have plans for multiple indexes (or indices, whichever plural you prefer) linking into the data, with particular optimisations for finding distinct items.
I have also found an interesting project by a student called Onne Gorter at the University of Twente, in the Netherlands, written in August 2004. He aimed to create a database filesystem in O'Caml, integrated fairly tightly with KDE. He took quite a different approach to me, which is refreshing and reassuring, but he also took my planned name. Now I need a better one... but he has a useful bibliography, which may indicate some further items to read.
Finally, I have also briefly investigated Google's BigTable system, and that also has some interesting ideas. I shall have to consider this further. Maybe later I will write a post explaining some of the things in my DBFS journal, a.k.a. The Book Of The Project.
And still the research continues.
Masters thesis
Well, the initial individual project allocations have been published, and it looks like I've definitely had my project confirmed. As far as I know, it was going to happen anyway, but it's nice to have official confirmation now, so I can start working on it.
What is my project? Well, it was my own proposal, and the title is Towards a Database Filesystem, which may give you a clue. The aim of this project is to build a database filesystem that can store various elements of metadata about a file, allowing you to categorise and organise files differently depending on how you're feeling or how you want to find your data.