Thursday, 15 May 2003

SpamBayes and Mail's Junk filter ::

Via Jon Udell, Bayesian Nets, Latent Semantics, Despamming and other speculations: a fascinating (well, at least for programmers and mathematicians ;-) comparison of how Bayesian nets differ from latent semantic analysis. While I don’t believe Apple has said anything about how Mail’s junk filter works, it does seem to use some form of latent semantic analysis. I’ve noticed that it doesn’t catch spam which doesn’t contain many recognizable words (e.g. base64-encoded and all-HTML messages). I do seem to have trained SpamBayes enough for it to be catching such spams, however. The combination of Mail with only a minimally-trained SpamBayes seems to be catching almost all of my spam now, and no false positives yet...