The iptables tool is wonderful, especially if you're suddenly getting a lot of traffic that you don't want. Recently, I've been seeing a message in the logs, warning "ip_conntrack: table full, dropping packet."

"WTF? How can the connection tracking table be full? I'm not using connection tracking..."

It turns out that rules that use the "match" plugin and check a connection's state start tracking that connection, just in case you ever want to match against ESTABLISHED or RELATED states. Let me explain with an example.

Some firewalls are set up with a rule like the following:

-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -m comment --comment "Allow packets from existing connections through" -A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT -m comment --comment "Allow new connections to HTTPD" -A INPUT -j DROP -m comment --comment "Drop everything else on the floor"

This, however, has a problem: once you get a LOT of simultaneous connections, it will fill the connection tracking table and your firewall may start behaving erratically. This can be handled in one of two ways.

If you really need to match against new connections, replace

-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT

with:

-A INPUT -m tcp -p tcp --syn --dport 80 -j LOG

Don't forget that the rule above will only cover new connections. Most of the time, however, you just want to accept all traffic on the port, new or otherwise:

-A INPUT -m tcp -p tcp --dport 80 -j ACCEPT

Easy when you know how. And why.

I've had some experience with the LaTeX document typesetting system before, having used it quite a lot during university for my final project's reports and presentation. I used the beamer package before, but this time I found its styles too restrictive. The three major presentation packages that I'm aware of are prosper (which I believe is quite old now), beamer, and powerdot. After some investigation (looking primarily at features and visual styles), I settled for powerdot. I will explain the basics of putting a presentation together in the rest of this post, but be warned: it assumes some basic familiarity with LaTeX, and I don't explain everything!

Starting out

The first steps when creating a slideshow are different to a regular document. In standard LaTeX, you open the document and just start writing, adding part, chapter and section headings as necessary. Creating a presentation is necessarily different. Here is a skeleton presentation:

\documentclass[style=paintings]{powerdot}

% Presentation metadata

\title{My First Powerdot Presentation}

\author{Joe Bloggs}

\date{8th November 2010}



\begin{document}



  % title slide

  \maketitle



  % section: title takes up full slide

  \section{First section}



  % text slide:

  \begin{slide}{Slide Title}

    \begin{itemize}

      \item This is an item

      \item This is also an item

      \item Finally, a third item in my list

    \end{itemize}

  \end{slide}



\end{document}

Doesn't look too bad, right? The presentation is contained within the document, and each slide is contained within a slide environment. The \section command now builds a standalone slide, as does the initial \maketitle.

Almost anything that you can include in a normal document can be used on a slide, including tables and images, and tricks like using \vspace{\stretch{1}} at the beginning and end of the slide to vertically centre the content.

Progressive disclosure

Powerdot allows you to show only parts of a slide to your audience at a time, to emphasise each point (for example), or to better illustrate the flow through a diagram. This can be accomplished in two ways: the simple \pause command, or the more complex and more powerful \onslide.

The simple \pause command takes no arguments, and simply stops anything in the slide after that command from appearing, as if content display had been "paused" at that point. It is important to note that everything following the command is typeset, in order to work out the space it would have taken up. This ensures that your slide content doesn't shift around as you go through.

\begin{slide}{Progressive Disclosure}

  \begin{itemize}

    \item This is my first item

    \item This is the second item, after which we wait \pause

    \item The third item appears on the page following the second \pause

    \item And this fourth and final item appears on the page after the third

  \end{itemize}

\end{slide}

Note that the command can be used multiple times, and it will work just as you might expect. If you want to do something more complex (such as displaying things out of order, displaying something over multiple frames, or scoping a change to a subset of the frames) you will need to use \onslide.

This command is much more powerful, and takes two arguments: the frames on which the contents should be displayed, and the contents themselves. The example above could be rewritten:

\begin{slide}{Progressive Disclosure}

  \begin{itemize}

    \item This is my first item

    \item This is the second item, after which we wait

    \onslide{2-}{\item The third item appears on the page following the second}

    \onslide{3-}{\item And this fourth and final item appears on the page after the third}

  \end{itemize}

\end{slide}

But we could also (in theory) choose to do something more bizarre:

\begin{slide}{Progressive Disclosure}

  \begin{itemize}

    \onslide{4-}{\item This is my first item, and the last to appear}

    \onslide{3-}{\item This is the second item, although technically the third}

    \onslide{2-}{\item The third item appears on the page following the fourth}

    \item This fourth and final item is actually the first that is visible

  \end{itemize}

\end{slide}

The \onslide command has three variations. The first version, as shown above, allocates space for items that have not yet been displayed. If you add a star to the command before its arguments (\onslide*) then no space will be allocated. In the example above, this means that the new bullet points would appear to push the existing items downwards. If you add a plus to the command (\onslide+) then the contents will always be typeset, but a different colour will be used for the item on the specified slide(s).

I have found \onslide* to be particularly useful for progressive disclosure of diagrams, where not displaying an item at all can be very important.

Graphics and diagrams

It is important to remember that LaTeX itself will only accept graphics in EPS format via the \includegraphics command from the graphicx package. pdfTeX will allow other formats if used directly (including PNG) but something like pstricks may then become unusable. For my presentation, I used a combination of pstricks diagrams and included EPS images.

Building diagrams in pstricks gave me a major advantage that I had not foreseen: the ability to define and then re-use a picture. I used this to good effect with the diagram of the OAuth "dance", showing the data flow between the User, Consumer, and Provider. When it came to the code samples for the implementation, I kept a miniature of this diagram on each slide to remind the viewer which stages of the dance were covered by the code on the current slide. This would not have been possible to do with OpenOffice (as far as I'm aware), and it saved me a great deal of time when I made changes to the original diagram.

If you have never used pstricks, then I highly recommend it. Although it may seem weird to specify co-ordinates in a text file rather than drawing the image graphically, it is extremely powerful and I found that I soon got used to it. It is worth remembering that some programs, like Inkscape, can actually save a vector image as a set of pstricks macros.

Code samples

Something that's often important when building a technical presentation is including one or more code samples. It's usually nice if they are syntax-highlighted in one way or another, to make it easier to read at a glance. By the way, it is generally advised that code samples in a presentation should be dark-coloured text on a light background rather than the other way round.

When it comes to using code samples in powerdot presentations, there are two things to be aware of. Firstly, choosing the typesetting package. I have always used the listings package for code samples. It has support for a large number of languages (as well as defining your own), and it is very configurable. You can even define short-cut environments for re-using options if you are working on a multiple-language presentation, for example.

The second thing to remember when adding listings to the presentation is that you need to inform the powerdot package that a slide has special "verbatim" content. This is done with an optional argument to the slide, method. This has three possible values (normal, direct, and file), but I have only found the first two to work reliably.

If the slide contains verbatim content, you must use either the direct or file rendering method, otherwise you will get an error. The file method can easily get confused (in my experience) and so I used direct. The downside is that you cannot use the \pause or \onslide commands in a direct-rendered slide; they will be silently ignored.

\begin{slide}[method=direct]{Hello World in C}

  \begin{lstlisting}[language=c,gobble=4]

    #include <stdio.h>



    int main(int c, char **v) {

      fprintf(stdout, "Hello, world!\n");

      return 0;

    }

  \end{lstlisting}

\end{slide}

Choosing and customising your style

The visual style of your presentation can make a difference to how it is received. Dark text on a very light background is usually considered to be the most readable. The powerdot package has a number of built-in styles (pp. 21-25), and I eventually settled on a modified version of paintings with the Charon palette. Creating your own style is not difficult, and you have a large amount of control over how it looks. I won't go into it here, as it is already well covered by the manual.

And finally...

This is all very much a work in progress. This post was based upon my first experience of writing a presentation in LaTeX with powerdot. As time goes on, I hope to build upon and standardise a visual style for both personal and company presentations. I am also prepared to send the LaTeX sources of my OAuth presentation to anyone who is interested, so please let me know either in the comments below or contact me by email or Twitter.

For me, the biggest advantage of LaTeX is that content comes first -- it's much easier to write a presentation without obsessing over how it looks until the very end. The better-defined your visual style, the less you have to worry about the design, and the more time you can spend making your presentation great!

First up, aliases. I have a few useful shortcuts defined, as well as some nice ways of displaying a repository's history. I've aliased many of the common commands to two-letter versions which evoke the similar Subversion command aliases:

[alias]

  st = status

  di = diff

  co = checkout

  ci = commit

  br = branch

  sta = stash

  graph = log --decorate --oneline --graph --branches --date-order

  lg = log --graph --pretty=format:'%C(yellow)%h%Creset -%C(yellow)%d%Creset %s %C(green)(%cr)%Creset %Cred<%an>%Creset' --abbrev-commit --date=relative --date-order

  today = !git log --author=$(git config user.email) --since=yesterday

The last alias (git today) reminds you what you've done today. The previous two aliases provide a nice graphical view of history in the console. git graph displays all history of all local branches in date order rather than topological order. This means that commits on different branches may be interleaved, rather than grouping commits on different branches. git lg displays a little more information (including author and relative timestamps), and allows me to pick which branches I want to see. The examples below are all taken from the Jerity repository.

The second tip is repository URL rewriting. This tackles the annoying problem of having submodules in projects that you want to be able to commit to, while making sure other people only see/use the read-only URLs. Git will very helpfully rewrite URLs from one form to another, so by having:

[url "ssh://git@github.com/dingram/"]

  pushInsteadOf = git://github.com/dingram/

in my configuration, I can git clone git://github.com/dingram/repository and when I push, it will automatically push to git@github.com:dingram/repository instead. This can be verified using git remote -v. The only thing to beware of is the format: the replacement URL goes in the section header, and the pattern to be replaced appears in the section itself.

Quick start

For those who just want the commands and don't want to wait:

git clone user@hostname.tld:/path/to/git-svn.git git-svn-clone

cd git-svn-clone

git svn init -s svn://hostname.tld/path/to/svn

git config svn.authorsfile $(basename $(pwd))/git-authors

rsync -avpP user@hostname.tld:/path/to/git-svn.git/svn .git/

git update-ref refs/remotes/trunk origin/master

git svn fetch --all && git svn rebase

Full details

So,  what does all this do? Let's look at it in stages.

1. Clone the existing git-svn repository, to get the history and avoid hitting the Subversion server any more than we need to (git clone)
2. Set up the Subversion integration information (git svn init)
3. Set up the translation between Subversion authors (bare username) and git authors (name + email address) (git config svn.authorsfile)
4. Copy the Subversion branch metadata from the original git-svn repository
5. Create the local "trunk" reference that git-svn will require
6. Fetch any new revisions from Subversion (and update the revision map between git and Subversion), and then make sure that "master" points at the head of the Subversion trunk

Bonus tip

If you have a number of git branches, it can be a pain to keep rebasing them to the latest trunk (as Subversion doesn't like non-linear history). This little script may help:

This will automatically rebase all of the branches onto master, although it won't handle sub-branches very well.

It was still surprising that the bulk of what I wanted to do can be done in about 16 lines of code. That's right: sixteen.

The plan was to create an edit-in-place system that would degrade nicely in browsers with JavaScript disabled. This isn't necessarily as elegant as it could be and it doesn't yet handle all input types, but that's what version one is for! So without further ado, I present the code:

$.fn.in_place_edit = function() {

  this.each(function() {

    var $eip = $('<span></span>');

    var $el=$(this).hide().after($eip);

    $eip.text($el.val());

    $eip.click(function(){ $eip.hide(); $el.data('origval', $el.val()).show().focus().select(); });

    $el.blur(function(){

      if($el.is('select')) $el.data('origval', $el.val());

      $eip.text($el.data('origval')).show(); $el.hide().val($el.data('origval'));

    });

    $el.keydown(function(e){

      if (e.which==27) $el.val($el.data('origval')).blur();

      if (e.which==13||e.which==9) {

        e.preventDefault();

        $el.data('origval', $el.val()).blur();

      }

    });

  });

};

Usage:

$(function(){ $('input, select').in_place_edit(); });

So, what does all this do? First of all, we create a new function on the main jQuery object, called in_place_edit. This then iterates over elements matching the selector. It hides the input and creates a placeholder <span> element with the input's value. Then we simply set up event handlers to deal with the placeholder being clicked, the input losing focus, and various keypresses (enter/tab to accept, esc to cancel). To see it in action, I've put together a rough-and-ready demo.

Planned future enhancements:

• Checkbox input types (display using the labels?)
• Radio input types
• File input types (if possible)
• Textareas
• Any other types...

Maybe further updates later on!

The background

I recently received a new hard drive, and so I excitedly partitioned and formatted it, extended my LVM VGs and merrily moved my LVs completely across to the new drive using pvmove. So far, so wonderful -- I was now ready to rebuild my other two drives so that only critical things were covered by RAID and everything else was just in a large VG that spanned multiple drives. I now had three drives, all Samsung SpinPoint:

1. 500GB  (sdb)
2. 1TB (sda)
3. 1.5TB (sdc, and brand-new)

The partition setup I wanted:

• 48MiB RAID-1 across all drives, for /boot (already set up across sda and sdb)
• 4.8GiB RAID-1 across all drives, for /(already set up across sda and sdb)
• 50GiB RAID-1 across all drives, for  /usr, /opt, /var
• 50GiB RAID-5 across all drives, for /home (giving 100GiB usable space)
• The rest of each drive divided into 50/100GiB partitions, to be spread among my large data VG, containing films, music, backups, and other large data.

The disaster

I moved all of the existing LVs to the new drive, which had plenty of space. Next, I sorted out the partitions for the two older drives, and initialised the RAIDs for my sys and safe VGs:

I then expanded the VGs and started to copy data across. I began with the safe VG that contained /home, as I wanted to be sure it was indeed safe. I have all of the photos I've ever taken, my email backups and archives, and various private keys and other important data, as well as things that could be replaced due to duplication elsewhere (including working copies of most of my projects). Being LVM, I could move the LVs around without needing to unmount them or reboot into a LiveCD:

Then, disaster struck:  pvmove spat a set of I/O errors at me... but kept running! I wasn't paying attention (as moving 40GiB of data takes quite a while) and so I had just left it to work away.

It was only after it had completed and I started getting I/O and "permission denied" errors while trying to access my home directory that I first realised anything had gone wrong. After some frantic searching on the Internet, it seemed that I had to wave goodbye to all of my personal data, including those bits that weren't backed up (though I've learned my lesson about backups now!).

I was determined not to give in to data loss, and so I gave myself a few recovery options:

1. Create an image of the partition where the data used to reside.
2. Try to recover pictures (the most critical thing for me) via foremost, which is a non-destructive operation.
3. Try to rebuild the file system using reiserfsck --rebuild-tree.

During my investigations and poking around, I discovered that there was another option: I could try to rebuild the VG from an old copy of its metadata.

Introduction to recovery

It turns out that many LVM utilities store a backup of the VG's metadata before and after performing various operations, so as long as your /etc/lvm directory is fine, you have a fighting chance. (Actually, it is theoretically possible to retrieve old versions of the metadata from the LVM partition itself, but this gets much more complex and I won't deal with it here.)

The reason for this is that pvmove works by creating a temporary target LV, mirroring data from the existing LV to the temporary target (making checkpoints along the way), and only removing the original once the mirroring has completed. This operation can then be interrupted and resumed at any time without loss of data. This also has the advantage that a full copy of the data on the original LV is still available; only the metadata explaining where it begins and ends has changed.

To access the old versions of this metadata, investigate the contents of your /etc/lvm/archive directory as root. You should find one or more files of the form volgroup_nnnnn.vg, where volgroup is the name of the relevant VG and nnnnn is a number. These files contain a backup of the volume group's metadata, such as which partitions it resides on and where it's located. The contents may look something like this:

I went through the backups for my safe volume group, and found the one before I extended the volume group to the (now corrupt) /dev/md3, which is actually the example given above.

Recovering the volume group

It was at this point that I rebooted and held my breath. I brought the system into runlevel 1 (single-user mode) and logged in as root. Out of curiosity I ran some SMART tests on the drives -- sda was reporting no partitions, and internal testing eventually reported a persistent read error at about 80% into the drive. It looked like that drive was toast, which would explain why writing to the RAID-5 had failed.

After verifying that it was a hardware issue, I decided to try to point the VG at its previous location (just /dev/sdc1), thereby getting it to forget that the home LV was on the now-inaccessible RAID-5. This could be accomplished by the magic of vgcfgrestore:

After the mount succeeded, I began to copy all of the data to my large partition, where it should be safe. I also copied the major bits I didn't want to lose to a USB flash drive as well, just to be certain. Once that was done, I took a deep breath and rebooted again, this time back to the full system.

Clearing up afterwards

Fortunately, everything worked fine -- it was as if nothing had ever gone wrong! All that was left now was to rebuild the broken RAID section.

I decided for safety that I would re-initialise the safe VG's RAID as RAID-1 rather than RAID-5, so that any two of the drives could fail without losing my precious data:

What I hadn't anticipated was that this would rebuild the RAID-1 using the data from the previous RAID-5... including the LVM descriptors and corrupt content. Once the RAID was online, I ran pvscan and received a warning that the VG metadata was inconsistent. For one heart-stopping moment, it also appeared that LVM thought my home partition was on the corrupt RAID! I quickly stopped the array and thought again.

After some further investigation, I discovered that the first 256 sectors of a partition are used by LVM to hold various information (including multiple copies of the VG metadata). All I had to do to fix this was to destroy that information:

Once this was done, I could run pvcreate /dev/md3 and proceed to extend the VG and pvmove data as before, without worrying that sda would give out on me and cause everything to fail again.

I hope this helps someone!

There seems to be a complete lack of cables designed for people with a single hearing aid, so one of my upcoming projects will be to convert a Y-cable that can serve two hearing aids into a joint DAI/earphone cable instead.

DAI connector pinout

The DAI connector has the following pinout:

• short/thick pin: ground
• long/thin pin: VDD (apparently -- doesn't seem to actually be connected to anything on a 3.5mm cable)
• short/thin pin: signal

Remember that a standard 3.5mm audio plug has the following pinout:

• tip: left signal
• inner sheath: right signal
• outer sheath: ground

Next stage: connecting an earphone to the plug on one side of the cable, to see if there are any impedance issues, although the cable I am using (an fmGenie one) is a very low-impedance cable because I have an Oticon hearing aid. Measured with a multimeter, it seems to be around 0Ω, so I think that's good enough for me. I believe attenuated cables have a 330kΩ resistor in series with each signal line (according to Neil Ferguson's bluetooth hearing aid hack page).

My ultimate goal is to get hold of a DAI jack that I can solder directly to an old earphone, so that I can just plug the spare DAI cable into the earphone directly, almost as if it were a hearing aid itself. Unfortunately, it seems that it is impossible to get hold of such things. According to Connevans, DAI jacks are only custom-made for hearing aid shoes. After some discussion, it seems like I should be able to make one myself though (with some help!) so we shall have to see how that turns out.

Names in a traditional file system are made unique based on a full path to the file, but most people think of a file name as just the final component. This would then cause a problem with the move to Insight, as a file could appear in multiple directories, and its only distinguishing feature would be the final component of its path. This is counter-intuitive and can cause all sorts of problems.

Consider makefiles, for example. They rely on a standard named file (Makefile) appearing at various levels in the hierarchy in order to work. Obviously, you would want different makefiles at different levels and in different projects, but Insight as it stands has no way to handle this.

I then started thinking about what makes a file unique. In the end, I came up with two things: name and content. This covers the makefile case (same name, different content) as well as the backup case (same content, different name). It then occurred to me that, in the general case, all you need to distinguish a file is its content, and then actually finding it can all be left up to metadata.

If files are then thought of as containers for data that happen to have a unique internal identifier (which never needs to be exposed to the user, although it can be accessed as, say, the file's inode number) then the idea of a content-based file manager comes into play. These examples work best with visual media, particularly images, but there is no reason in principle that this could not be extended.

Imagine searching for a file. You know it's a photo, but you have a large collection of them. With a digital camera and a large-capacity memory card, who needs to ever delete a photo? We'll assume that you've dilligently tagged the photos with metadata as you've imported them from the camera, through some easy batch process.

On the tagging point: a lot can be taken from the metadata stored by the camera (date/time, resolution, orientation, black and white/colour, perhaps GPS co-ordinates) and with the right tools, more can be inferred (auto-tagging faces, buildings, perhaps recognising common events like football matches, converting GPS co-ordinates to places, ...). As time goes on, people will need to do less and less manual tagging.

Anyway, back to the file manager. You know you are after a picture or a set of pictures. Normal thought processes will probably follow a path similar to: "Yeah, I wanted to show dad those photos from that holiday in Paris that we had two months ago. I think he'd particularly like the ones we got of the Louvre, as well as the ones with me in, of course." I've highlighted various key words that can be translated directly to metadata searches. Notice how these all involve a narrowing down of the query.

To convert these to filters, we then have:

• type: photo
• holiday
• location: Paris
• date: two months ago
• at least one of:
  • location: Louvre
  • person: Me

This could also be represented by a query:

type:photo AND holiday AND location:Paris AND date:-2m
AND (location:Louvre OR person:me)

Breaking it down in this way feels fairly technical and wordy, however. I'd much prefer a visual view.

Imagine a black field, speckled with points of light representing your photos:

You filter by "holiday", and (because it learns based on previous searches) it then groups by location. The ones which have been filtered out fade into nothing, and the photos group into labelled blobs and enlarge slightly:

You filter by date, and as you drag the slider, irrelevant items fade away and relevant ones enlarge:

Then you add the final filters and set the photos up for viewing, perhaps as a slideshow... and you're done!

Pretty neat, I think.

The host server partitions were LVM-formatted volumes on top of hardware RAID-1, and each of the virtual hosts were partitioned with LVM too, internally. This means that the restore process will not be at all trivial. With just a complete image of the host system, I would need to restore (deep breath) files on a partition on an LVM logical volume (inside a volume group, on a physical volume) in a VMWare hard disk stored on LVM (logical volume inside volume group of physical volumes) inside a disk image that is itself a file on a disk. How very convoluted.

I used a Gentoo system to restore the data, although any Linux system with the appropriate packages should be able to do it. No searches turned up information on doing all of this, and I had to come up with some of it myself, so I thought I would document the process.

Just to illustrate the complexity of the data layout:

Requirements

The machine that is performing the restore requires:

• Linux
• LVM2
• VMWare Server
• losetup
• NBD (Network Block Device) kernel driver
• file system drivers (if applicable)

Procedure

Boot the dead server (vm-host) using a Linux LiveCD and copy its partitions to files on another drive. If you don't know which partitions the LVM was stored across, you can either copy all of them, or use <tt>pvscan</tt> (possibly after running <tt>vgscan</tt>) to work out which belong to the appropriate volume group. This dump may take some time.

Attach the drive with the partition images to the rescue machine (rescue-host), edit /etc/lvm/lvm.conf on rescue-host, and modify the filter line to allow LVM to scan loopback and network block devices. You'll probably want to change this back afterwards, so remember what it was.

Set up the partition images as loop devices:

Scan for and activate the volume group:

Find the logical volume with the data we're after (LogVol06 in volume group VolGroup00 in this case) and mount it:

Copy the VMWare hard disk files out of the LVM mount, because we know that the volume group stored on them will conflict with the existing one:

Check the partition table on the virtual hard disk:

This will give you something like:

The second partition (with type 8E) is the one we wanted in this case, but it's LVM-formatted. This means we can't use vmware-mount because an LVM partition cannot be mounted normally.

We need to unmount the currently-active volume group on the loopback devices (i.e. the LVM from vm-host):

Now you need a dedicated terminal to mount the LVM partition from the virtual hard disk, as this process must stay running in order to access the virtual drive:

If you have more than one virtual partition as part of the LVM, you'll need to have one terminal for each, and select partition numbers and NBD device numbers as appropriate.
Now find the volume group again:

Find the appropriate logical volume and mount it:

Do whatever you need to do with the data, then unmount it all:

And that's it! I hope this helps someone other than me...