The Idea

The basic idea is that we have a sorted list of items that may have some duplicates, say:

{A0, A1, B0, C0, C1, E0, F0, F1, F2, G0, H0, H1}

and we want to remove all of the duplicates, so we get two sets:

{A0, B0, C0, E0, F0, G0, H0} {A1, C1, F1, F2, H1}

Note that it doesn't matter which of the duplicates ends up in which set, just that the items in the first are all unique and that we don't lose any. Also, A₀ and A₁ have the same value but the subscript will help to distinguish exactly which of the As I'm talking about. Now we have the problem, let's look at some solutions.

Lossless solution - additional lists

So the first solution uses two temporary lists, unique and duplicate. We start with two pointers into the original list: p and q. We set p to point to the start of the list, and q to point to the element after p. We then copy the item p points at to our unique list. Then, while we still have more items to examine:

• If the items pointed to by p and q have the same value, then we copy the item q points at to the duplicate list and advance p and q.
• If they are different, we copy the item at q to our unique list, and advance p and q.

We keep doing this until we get to the end of the array, and then copy the unique list to the start of the array, the duplicate list after that (both overwriting the previous contents) and set the number of unique items to the length of the unique

list.

Lossy solution - one list

We can do a similar thing in-place, although this will result in data loss. The essential idea is:

• We start in the same way: p pointing to the first element, and q to the second
• While q has the same value as p, advance q along the list
• When they are different, advance p, copy q to p, and advance q
• Once q goes past the end of the list, we're done, and the offset of p is the number of unique items.

Now this is quite simple, and only requires one pass through the list, but it results in the loss of information, which is unacceptable in this case.

Lossless solution - one list

For this solution, we can also do it with one pass through the list and two pointers. The basic intuition is that we're going through the array, accumulating a rotating block of stuff that we don't want. At any given point:

• Everything between the start and p (not inclusive) is the unique list (so far)
• Everything between p and q (not inclusive) is an unwanted duplicate
• Everything between q (inclusive) and the end is yet to be processed.

At the end, we return the number of unique items at the start of the list; everything above that is a duplicate.

For integers

Here is some pseudo-Java that performs my algorithm for a set of integers.

int set_uniq(int set[]) {

  int count = set.length();

  int tmp, p=0, q=1;

  while (p < set.length && q < set.length) {

    while (p < set.length && set[p] != set[q]) {

      p = p + 1;

      if (q < set.length) {

        if (p < q) {

          tmp = set[p];

          set[p] = set[q];

          set[q] = tmp;

        }

        q = q + 1;

      }

      else if (count > p) {

        count = p;

      }

    }

    while (q < max && set[p] == set[q]) {

      q = q + 1;

    }

  }

  // q hit the end and is still the same as p

  if (count > p)

    count = p + 1;



  return count;

}

The general code

The arguments to the general set_uniq() function are very similar to the arguments to standard qsort():

• The (sorted) array to work on
• The number of items in the array
• The size of each item in the array
• A comparator function that returns a negative integer if the first argument is less than the second, zero if they are equal, and a positive integer if the first argument is greater than the second.

So, without further ado, my code:

int set_uniq(void *set,

    size_t count,

    size_t elem_size,

    int (*cmp)(const void*, const void*)) {

  void *p, *q, *max;

  void *tmp=calloc(1, elem_size); /* TODO: check for failure */

  p=set;

  q=set+elem_size;

  max=set+(count*elem_size);

  while (p< p-set)

      count = (p-set)/elem_size;

    }

    while (q p-set)

    count = 1 + (p-set)/elem_size;

  ifree(tmp);

  return count;

}

Conclusion

This code is released under a Creative Commons licence (see below). A similar idea will work well for other solutions, like set difference. I'll post those later if anyone really cares. Also, there is a better way of swapping the items by using the fact that:

a ^= b;

b ^= a;

a ^= b;

swaps a and b without needing a temporary variable. If you just go along the data in 32-bit chunks, performing those XOR operations to swap the data. This actually works out to be 30% or so faster (if I remember correctly). Finally - general code like this is always going to be slower than code that's been specialised for a particular purpose. Calling this on integers, for example, will be much slower than coding an integer-specific version. I should also point out that you can just overwrite integers, as you don't need to free them

There has been some recent controversy over Facebook's Terms and Conditions: specifically, it seems that these Terms imply that they own any Content that you post to the site. When I first heard about this on 1st October 2007, I thought that it couldn't be right. A company couldn't just bury that in their Terms and Conditions… could they?

I decided to go through it, and having assauged my own doubts, I felt it was worth explaining in detail to anyone who is interested. Hi to anyone visiting from a group set up about this.

Disclaimer

I am not a lawyer. This is my interpretation of this policy, intended to help inform people about what exactly is going on. These are entirely my views and do not represent the views of Imperial College, its Computing Department, or any other persons or organisations, mentioned or implied.

This page is accurate as of Tuesday 2^nd October, 2007.

So what does it say?

The full content of the policy is below (taken at 23:44 BST on 1^st October, 2007):

User Content Posted on the Site

You are solely responsible for the photos, profiles, messages, notes, text, information, music, video, advertisements, listings, and other content that you upload, publish or display (hereinafter, "post") on or through the Service or the Site, or transmit to or share with other users (collectively the "User Content"). You may not post, transmit, or share User Content on the Site or Service that you did not create or that you do not have permission to post. You understand and agree that the Company may, but is not obligated to, review the Site and may delete or remove (without notice) any Site Content or User Content in its sole discretion, for any reason or no reason, including without limitation User Content that in the sole judgment of the Company violates this Agreement or the Facebook Code of Conduct, or which might be offensive, illegal, or that might violate the rights, harm, or threaten the safety of users or others. You are solely responsible at your sole cost and expense for creating backup copies and replacing any User Content you post or store on the Site or provide to the Company.

When you post User Content to the Site, you authorize and direct us to make such copies thereof as we deem necessary in order to facilitate the posting and storage of the User Content on the Site. By posting User Content to any part of the Site, you automatically grant, and you represent and warrant that you have the right to grant, to the Company an irrevocable, perpetual, non-exclusive, transferable, fully paid, worldwide license (with the right to sublicense) to use, copy, publicly perform, publicly display, reformat, translate, excerpt (in whole or in part) and distribute such User Content for any purpose on or in connection with the Site or the promotion thereof, to prepare derivative works of, or incorporate into other works, such User Content, and to grant and authorize sublicenses of the foregoing. You may remove your User Content from the Site at any time. If you choose to remove your User Content, the license granted above will automatically expire, however you acknowledge that the Company may retain archived copies of your User Content.

Let's take this apart

OK, so let's take this apart almost sentence by sentence and see what it gives us, and what exactly they say.

You are solely responsible for the photos, profiles, messages, notes, text, information, music, video, advertisements, listings, and other content that you upload, publish or display (hereinafter, "post") on or through the Service or the Site, or transmit to or share with other users (collectively the "User Content").

This appears to be quite straightforward. They are saying (in a rather long-winded way) that anything you post to the site or share with other users is called “User Content” and is your sole responsibility. Therefore if there is some legal trouble, it is you the poster who will get burned.

You may not post, transmit, or share User Content on the Site or Service that you did not create or that you do not have permission to post.

This is also a fair point: if you neither own it nor have permission to share it, then you should not post it on Facebook. You could start arguing about fair use under copyright law here, but we will stay away from that for now as that is not the issue we are discussing.

You understand and agree that the Company may, but is not obligated to, review the Site and may delete or remove (without notice) any Site Content or User Content in its sole discretion, for any reason or no reason, including without limitation User Content that in the sole judgment of the Company violates this Agreement or the Facebook Code of Conduct, or which might be offensive, illegal, or that might violate the rights, harm, or threaten the safety of users or others.

Wow, this is quite lengthy. This boils down to self-policing of Facebook. The Company may review the site (but it doesn't have to) and can decide that some Content should be removed. It doesn't have to supply a reason, but it will remove Content that violates the Terms of Use, the Code of Conduct, or any Content which is offensive, illegal, threatening, etc.

You are solely responsible at your sole cost and expense for creating backup copies and replacing any User Content you post or store on the Site or provide to the Company.

Although Facebook may take backups, this is for their own internal reasons and they should not be relied upon. If you put anything crucial here, make sure you have a copy yourself; you can't go crying to them if you lose it. They may try to help, but don't count on it.

When you post User Content to the Site, you authorize and direct us to make such copies thereof as we deem necessary in order to facilitate the posting and storage of the User Content on the Site.

This section is there to cover themselves against the fact that the files you upload may be copied between servers internally, for performance reasons. This next part has caused the controversy, and so merits careful and detailed examination.

By posting User Content to any part of the Site, you automatically grant, and you represent and warrant that you have the right to grant, to the Company

If you post Content to the Site, you imply that you have permission to do this, and that you may grant permission to others.

an irrevocable, perpetual, non-exclusive, transferable, fully paid, worldwide license (with the right to sublicense)

A licence which:

1. cannot be revoked
2. lasts without a time limit
3. is not exclusive to the Company (i.e. you can grant use of the Content to other people)
4. may be transferred from the Company to another individual or organisation
5. does not require the Company to pay you
6. can be used worldwide without restriction (as Facebook is a global website)

You also grant the ability for Facebook to sublicence this Content under the above terms at most, but potentially under more restrictive terms.

to use, copy, publicly perform, publicly display, reformat, translate, excerpt (in whole or in part) and distribute such User Content

A list of all the things they wish to be able to do with this Content if they so choose.

for any purpose on or in connection with the Site or the promotion thereof, to prepare derivative works of, or incorporate into other works, such User Content, and to grant and authorize sublicenses of the foregoing.

A re-iteration of their permission to sublicence the Content, preceded by other things they may wish to do. For example, if Facebook wished to produce a series of adverts, they would be perfectly within their rights to use screenshots of the site with user photos or other details in those screenshots – that is covered by this part of the sentence, as a derivative work.

You may remove your User Content from the Site at any time. If you choose to remove your User Content, the license granted above will automatically expire, however you acknowledge that the Company may retain archived copies of your User Content.

Finally, Facebook offer you protection: if you do not want your Content used in this way, or you choose to remove it for any reason of your own, you immediately invalidate their licence to use the Content. However, they may still have backup copies of the Content which it would be infeasible to track down and eliminate.

Conclusion

While Facebook may be overstepping the mark slightly with requesting the right to sublicence your Content, I still feel the Terms are reasonable. They certainly make no mention of transfer of ownership, but neither do they make clear that ownership stays with the original creator. In their position, I would amend the terms to make this absolutely clear, and I would reconsider the sublicensing clause (or at least provide an explanation for it).

If anyone would like to make any comments about this page, please feel free to drop me a message on Facebook!