The iptables tool is wonderful, especially if you're suddenly getting a lot of traffic that you don't want. Recently, I've been seeing a message in the logs, warning "ip_conntrack: table full, dropping packet."

"WTF? How can the connection tracking table be full? I'm not using connection tracking..."

It turns out that rules that use the "match" plugin and check a connection's state start tracking that connection, just in case you ever want to match against ESTABLISHED or RELATED states. Let me explain with an example.

Some firewalls are set up with a rule like the following:

-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -m comment --comment "Allow packets from existing connections through" -A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT -m comment --comment "Allow new connections to HTTPD" -A INPUT -j DROP -m comment --comment "Drop everything else on the floor"

This, however, has a problem: once you get a LOT of simultaneous connections, it will fill the connection tracking table and your firewall may start behaving erratically. This can be handled in one of two ways.

If you really need to match against new connections, replace

-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT

with:

-A INPUT -m tcp -p tcp --syn --dport 80 -j LOG

Don't forget that the rule above will only cover new connections. Most of the time, however, you just want to accept all traffic on the port, new or otherwise:

-A INPUT -m tcp -p tcp --dport 80 -j ACCEPT

Easy when you know how. And why.

The Idea

The basic idea is that we have a sorted list of items that may have some duplicates, say:

{A0, A1, B0, C0, C1, E0, F0, F1, F2, G0, H0, H1}

and we want to remove all of the duplicates, so we get two sets:

{A0, B0, C0, E0, F0, G0, H0} {A1, C1, F1, F2, H1}

Note that it doesn't matter which of the duplicates ends up in which set, just that the items in the first are all unique and that we don't lose any. Also, A₀ and A₁ have the same value but the subscript will help to distinguish exactly which of the As I'm talking about. Now we have the problem, let's look at some solutions.

Lossless solution - additional lists

So the first solution uses two temporary lists, unique and duplicate. We start with two pointers into the original list: p and q. We set p to point to the start of the list, and q to point to the element after p. We then copy the item p points at to our unique list. Then, while we still have more items to examine:

• If the items pointed to by p and q have the same value, then we copy the item q points at to the duplicate list and advance p and q.
• If they are different, we copy the item at q to our unique list, and advance p and q.

We keep doing this until we get to the end of the array, and then copy the unique list to the start of the array, the duplicate list after that (both overwriting the previous contents) and set the number of unique items to the length of the unique

list.

Lossy solution - one list

We can do a similar thing in-place, although this will result in data loss. The essential idea is:

• We start in the same way: p pointing to the first element, and q to the second
• While q has the same value as p, advance q along the list
• When they are different, advance p, copy q to p, and advance q
• Once q goes past the end of the list, we're done, and the offset of p is the number of unique items.

Now this is quite simple, and only requires one pass through the list, but it results in the loss of information, which is unacceptable in this case.

Lossless solution - one list

For this solution, we can also do it with one pass through the list and two pointers. The basic intuition is that we're going through the array, accumulating a rotating block of stuff that we don't want. At any given point:

• Everything between the start and p (not inclusive) is the unique list (so far)
• Everything between p and q (not inclusive) is an unwanted duplicate
• Everything between q (inclusive) and the end is yet to be processed.

At the end, we return the number of unique items at the start of the list; everything above that is a duplicate.

For integers

Here is some pseudo-Java that performs my algorithm for a set of integers.

int set_uniq(int set[]) {

  int count = set.length();

  int tmp, p=0, q=1;

  while (p < set.length && q < set.length) {

    while (p < set.length && set[p] != set[q]) {

      p = p + 1;

      if (q < set.length) {

        if (p < q) {

          tmp = set[p];

          set[p] = set[q];

          set[q] = tmp;

        }

        q = q + 1;

      }

      else if (count > p) {

        count = p;

      }

    }

    while (q < max && set[p] == set[q]) {

      q = q + 1;

    }

  }

  // q hit the end and is still the same as p

  if (count > p)

    count = p + 1;



  return count;

}

The general code

The arguments to the general set_uniq() function are very similar to the arguments to standard qsort():

• The (sorted) array to work on
• The number of items in the array
• The size of each item in the array
• A comparator function that returns a negative integer if the first argument is less than the second, zero if they are equal, and a positive integer if the first argument is greater than the second.

So, without further ado, my code:

int set_uniq(void *set,

    size_t count,

    size_t elem_size,

    int (*cmp)(const void*, const void*)) {

  void *p, *q, *max;

  void *tmp=calloc(1, elem_size); /* TODO: check for failure */

  p=set;

  q=set+elem_size;

  max=set+(count*elem_size);

  while (p< p-set)

      count = (p-set)/elem_size;

    }

    while (q p-set)

    count = 1 + (p-set)/elem_size;

  ifree(tmp);

  return count;

}

Conclusion

This code is released under a Creative Commons licence (see below). A similar idea will work well for other solutions, like set difference. I'll post those later if anyone really cares. Also, there is a better way of swapping the items by using the fact that:

a ^= b;

b ^= a;

a ^= b;

swaps a and b without needing a temporary variable. If you just go along the data in 32-bit chunks, performing those XOR operations to swap the data. This actually works out to be 30% or so faster (if I remember correctly). Finally - general code like this is always going to be slower than code that's been specialised for a particular purpose. Calling this on integers, for example, will be much slower than coding an integer-specific version. I should also point out that you can just overwrite integers, as you don't need to free them