Weird utf-8 bug in QuickLook: it’s the EA

A while ago, I noticed a weird bug affecting the way QuickLook on Leopard showed characters with french accents, while being careful to save in UTF-8 from TextMate:

TM’s on left, QL on the right. Unix deities seemed to confirm that TM was not to blame:

$ file test.txt 
test.txt: UTF-8 Unicode text
$ cat test.txt
é ç ë

On the other hand, opening test.txt with TextEdit gave the same result as QuickLook — messed up characters. If I fixed the characters in TextEdit and saved, the display of this particular file was always correct from then on (even if I modified it with TM afterwards). Weird.

Apart from a grand total of one (1) related and unanswered thread at Apple Discussions, a Google search for “Quicklook utf8″ or “Quicklook unicode” turned out nothing – so at first it seemed like there were only two people on the entire planet affected by this bug (well, three). By being a little more creative with my keywords, however, this post by Nico Weber on a vi-related thread turned up:

Indeed, if you check the two files with `ls -l`, you’ll see that the
file written by TextEdit has an “@” (that means it has an extended
attribute). Now, if you do `ls -l@` or `xattr -l filename`, you’ll see
that the TextEdit file has the com.apple.TextEncoding attribute set:

Macintosh-2:b nico$ xattr -l texteditfile.txt
com.apple.TextEncoding: UTF-8;134217984

The file written by MacVim does not have this attribute. If you add it
(`xattr -w com.apple.TextEncoding ‘UTF-8;134217984′ macvimfile.txt`),
the file shows up correctly in Quicklook (and in TextEdit too; it
didn’t do that before)

Applying the `xattr` command on test.txt did the trick: non-ASCII UTF-8 characters now show up fine in QL.

So what gives? It seems pretty harsh to me to demand extended attributes just to have the encoding right, especially when all 3-rd party programs are handling the matter perfectly fine without them, thank you very much. I don’t feel like applying xattr on all my text files either. I also don’t understand why the issue is not more widespread, i.e. why nobody talks about it. I feel like maybe this bug only appears with a specific combination of tools and locales? Should TextMate set the relevant attributes? I’m puzzled.