bogofilter ate 3/5
Quoted 1 line · On Wed, 6 Sep 2006 07:00:44 +0200 Willy Tarreau wrote:
spam has never been a real problem for me on LKML
I spend a lot of time deleting spam from the mailing lists I archive. The kernel list gets a lot of spam--it's just not as obvious as it is on some lists because of the large amount of legit traffic. Some of the slower lists get nothing but spam for weeks at a time.
While bogofilter may not be the answer, doing something may be quite desirable.
Quoted 9 lines · Rick Ellis
Quoted 1 line · On Wed, 6 Sep 2006 07:00:44 +0200 Willy Tarreau wrote:
spam has never been a real problem for me on LKML
I spend a lot of time deleting spam from the mailing lists I archive. The kernel list gets a lot of spam--it's just not as obvious as it is on some lists because of the large amount of legit traffic. Some of the slower lists get nothing but spam for weeks at a time.
agreed, but I still think that losing some legitimate emails is worse than getting 5% spam.
Quoted 2 lines
While bogofilter may not be the answer, doing something may be quite desirable.
OK, but doing something could simply consist in adding a header that anyone is free to filter on or not.
Regards,
Willy
Quoted 11 lines · Willy Tarreau
Quoted 9 lines · Rick Ellis
Quoted 1 line · On Wed, 6 Sep 2006 07:00:44 +0200 Willy Tarreau wrote:
spam has never been a real problem for me on LKML
I spend a lot of time deleting spam from the mailing lists I archive. The kernel list gets a lot of spam--it's just not as obvious as it is on some lists because of the large amount of legit traffic. Some of the slower lists get nothing but spam for weeks at a time.
Quoted 8 lines
agreed, but I still think that losing some legitimate emails is worse than getting 5% spam.
Quoted 2 lines
While bogofilter may not be the answer, doing something may be quite desirable.
OK, but doing something could simply consist in adding a header that anyone is free to filter on or not.
BTW, I forgot to say that I fear that maintaining the bogofilter in the long term will be supplementary work for Matti, with few chances of 100% success with happy readers.
Regards,
Willy
Quoted 2 lines
OK, but doing something could simply consist in adding a header that anyone is free to filter on or not.
The problem with that is the post gets no indication that his mail has been filtered. The way it works now is the rejection happens at SMTP time and that causes the poster to see the problem. If people filtered on a header, you'd never know why you weren't getting a response.
Quoted 9 lines · On Wed, 6 Sep 2006, [email protected] wrote:
Quoted 2 lines
OK, but doing something could simply consist in adding a header that anyone is free to filter on or not.
The problem with that is the post gets no indication that his mail has been filtered. The way it works now is the rejection happens at SMTP time and that causes the poster to see the problem. If people filtered on a header, you'd never know why you weren't getting a response.
How about this:
1. Incoming mail from subscribers is accepted
2. Incoming mail to honeypot addresses is trained as SPAM
3. Incoming mail from non-subscribers is marked with X-Bogofilter:
4. A handy Perl script subscribes to lkml, and for any message it gets with an X-Bogofilter: SPAM header, it sends a notification (rate-limited) to the message sender explaining that his message will be filtered as SPAM by some recipients, and inviting him to contact postmaster to resolve the issue, and additionally letting him know that notification is rate-limited and there is a website he can check to see the SUBJECTs of all messages filtered as SPAM on lkml (say for the last week or two) if he wants to try and correct the problem himself.
Thanks,
Chase
Quoted 1 line · Chase Venters
1. Incoming mail from subscribers is accepted
How do you know if the sender is really a subscriber?
Quoted 3 lines
4. A handy Perl script subscribes to lkml, and for any message it gets with an X-Bogofilter: SPAM header, it sends a notification (rate-limited) to the message sender
How do you know who the sender really is? IMHO bouncing anything (especially spam) after SMTP OK is worse than the spam itself.
Signature · 1 line
Krzysztof Halasa
Quoted 5 lines · Krzysztof Halasa
Quoted 1 line · Chase Venters
1. Incoming mail from subscribers is accepted
How do you know if the sender is really a subscriber?
You can check the From: or envelope sender against the subscriber database. Forgery isn't a concern because we're not trying to stop forgery with this method. Subscribers subscribing one address and sending from another is also not a problem since a lookup failure just means you get to ride through the bogofilter. Note as well that #4 is a separate program; this lookup is likely done by the mailing list software.
#1 should significantly reduce the load on the bogofilter (not sure if that matters though).
Quoted 7 lines
Quoted 3 lines
4. A handy Perl script subscribes to lkml, and for any message it gets with an X-Bogofilter: SPAM header, it sends a notification (rate-limited) to the message sender
How do you know who the sender really is? IMHO bouncing anything (especially spam) after SMTP OK is worse than the spam itself.
The perl script behaves as an optional autoresponder. Autoresponders would respond to spam as well (well, unless you put a spam filter in front of them, but I assume that many don't).
Also note that a number of people (myself included, at work anyway) have perl scripts that respond to all incoming mail and require a reply cookie from original envelope senders. We do it because it almost entirely prevents spam from arriving in our inboxes (I say almost because there is the occasional spammer that doesn't forge their sender address and has some kind of autoresponder behind it). I had to do this for my work account to stop the hundreds of messages I was getting each day after a co-worker "pranked me" by signing me up for all that crap.
Thanks,
Chase
Quoted 3 lines · Chase Venters
You can check the From: or envelope sender against the subscriber database. Forgery isn't a concern because we're not trying to stop forgery with this method.
That's the first problem.
Quoted 3 lines
The perl script behaves as an optional autoresponder. Autoresponders would respond to spam as well (well, unless you put a spam filter in front of them, but I assume that many don't).
Yep. Sending their "responses" to innocent people, instead of spam senders. That's what many "antivirus" do.
Quoted 4 lines
Also note that a number of people (myself included, at work anyway) have perl scripts that respond to all incoming mail and require a reply cookie from original envelope senders. We do it because it almost entirely prevents spam from arriving in our inboxes
Sure. Don't you think is also prevents a lot of legitimate mail? Hope that all addresses you send mail to are automatically added to a white list? (I'm especially annoyed with people asking me for something, and then my answer bounces with "click somewhere" response).
Signature · 1 line
Krzysztof Halasa
Quoted 8 lines · Krzysztof Halasa
Quoted 3 lines · Chase Venters
You can check the From: or envelope sender against the subscriber database. Forgery isn't a concern because we're not trying to stop forgery with this method.
That's the first problem.
The problem with trying to stop forgery is that there is not yet a foolproof or reasonably foolproof method of doing so. All available options require cooperation from other MTAs, MUAs, users or administrators; hence, they're only good at increasing 'HAM' score.
It is also at least partially unapplicable to LKML since this list deliberately allows anyone to post.
Quoted 12 lines
Quoted 3 lines
The perl script behaves as an optional autoresponder. Autoresponders would respond to spam as well (well, unless you put a spam filter in front of them, but I assume that many don't).
Yep. Sending their "responses" to innocent people, instead of spam senders. That's what many "antivirus" do.
Quoted 4 lines
Also note that a number of people (myself included, at work anyway) have perl scripts that respond to all incoming mail and require a reply cookie from original envelope senders. We do it because it almost entirely prevents spam from arriving in our inboxes
Quoted 6 lines
Sure. Don't you think is also prevents a lot of legitimate mail? Hope that all addresses you send mail to are automatically added to a white list? (I'm especially annoyed with people asking me for something, and then my answer bounces with "click somewhere" response).
Yeah, I whitelist on that address as well. And if someone does respond to the challenge (all it takes is reply/send, as I've used a signed VERP as the Reply-To), they get whitelisted. If the challenge bounces, it bounces to the envelope sender (a different signed VERP) which increases their blacklist score by 1. Get a blacklist score of 3 and I'll silently ignore your mail for 6 months.
Thanks,
Chase
Quoted 2 lines · Chase Venters
The problem with trying to stop forgery is that there is not yet a foolproof or reasonably foolproof method of doing so.
The first important point here is that vger rejects mail in SMTP DATA phase, thus making the forgery irrelevant WRT such bounces. Second, being on the list isn't enough for the message to go through, it has to pass the filters as everyone else.
Third, while I think "normal" autoresponders (vacation etc.) are perfectly reasonable (not in mailing list context of course), ones which by design respond mostly to forged addresses (do you think antivirus and antispam qualify?) are aimed at the wrong, innocent person.
Signature · 1 line
Krzysztof Halasa
Quoted 10 lines · Krzysztof Halasa
Quoted 2 lines · Chase Venters
The problem with trying to stop forgery is that there is not yet a foolproof or reasonably foolproof method of doing so.
The first important point here is that vger rejects mail in SMTP DATA phase, thus making the forgery irrelevant WRT such bounces. Second, being on the list isn't enough for the message to go through, it has to pass the filters as everyone else.
Fair enough. The discussion drifted with the introduction of the SpamCop idea that bounce messages are evil. As you have shown, it is possible in many instances to refuse mail at the door; however, it's pretty unfeasible in many environments...
Quoted 7 lines
Third, while I think "normal" autoresponders (vacation etc.) are perfectly reasonable (not in mailing list context of course), ones which by design respond mostly to forged addresses (do you think antivirus and antispam qualify?) are aimed at the wrong, innocent person.
Well, this is an interesting point applicable to my specific suggestion. You are right that such a script would generate a lot of misdirected mail. In my experience in dealing with this issue, it's usually e-mail viruses that spoof legitimate addresses: most of the spam I see comes from junk addresses at junk domains. So it's not like you're bugging a *human*.
In any case, this specific idea was just a brainstorm for how we could use X-Bogofilter: headers to allow LKML subscribers to make their own filtering decision about the messages. Another idea is to divert junk mail to a second list which is for people that want to be real sure they don't miss patches.
A third choice is obviously to leave the problem alone and let bogofilter eat our patches ;)
Whichever route is taken, though, it looks like we might have to switch away from Majordomo soon, because SpamCop and Stuart MacDonald are going to bring an end to programs that automatically respond to mail.
Thanks,
Chase
Quoted 2 lines · Chase Venters
A third choice is obviously to leave the problem alone and let bogofilter eat our patches ;)
Actually I think Matti will just fix the filter and all such problems will go away.
Signature · 1 line
Krzysztof Halasa
Quoted 7 lines · Krzysztof Halasa
To: Chase Venters <[email protected]>
Cc: [email protected], [email protected] (Willy Tarreau),
[email protected]
Subject: Re: bogofilter ate 3/5
From: Krzysztof Halasa <[email protected]>
Date: Fri, 08 Sep 2006 00:58:02 +0200
Quoted 7 lines
Quoted 2 lines · Chase Venters
A third choice is obviously to leave the problem alone and let bogofilter eat our patches ;)
Actually I think Matti will just fix the filter and all such problems will go away.
Yes. Ever so slowly, but eventually.
I have this Feynemanish approach on problem solving. Gather data, think a lot, try something...
Quoted 2 lines
Signature · 1 line
Krzysztof Halasa
/Matti Aarnio
From: On Behalf Of Chase Venters
Quoted 3 lines
You can check the From: or envelope sender against the subscriber database. Forgery isn't a concern because we're not trying to stop forgery with this method. Subscribers subscribing one address
Forgery is always a concern...
Quoted 5 lines
The perl script behaves as an optional autoresponder. Autoresponders would
respond to spam as well (well, unless you put a spam filter in front of
them, but I assume that many don't).
..because autoresponders are always replying to forged addresses:
http://www.spamcop.net/fom-serve/cache/329.html
Quoted 7 lines
Also note that a number of people (myself included, at work anyway) have
perl scripts that respond to all incoming mail and require a reply cookie from original
envelope senders. We do it because it almost entirely prevents spam from
arriving in our inboxes (I say almost because there is the occasional
Autoresponder by another name, see above URL.
..Stu
Quoted 24 lines · Stuart MacDonald
From: On Behalf Of Chase Venters
Quoted 3 lines
You can check the From: or envelope sender against the subscriber database. Forgery isn't a concern because we're not trying to stop forgery with this method. Subscribers subscribing one address
Forgery is always a concern...
Quoted 5 lines
The perl script behaves as an optional autoresponder. Autoresponders would
respond to spam as well (well, unless you put a spam filter in front of
them, but I assume that many don't)...because autoresponders are always replying to forged addresses:
http://www.spamcop.net/fom-serve/cache/329.htmlQuoted 7 lines
Also note that a number of people (myself included, at work anyway) have
perl scripts that respond to all incoming mail and require a reply cookie from original
envelope senders. We do it because it almost entirely prevents spam from
arriving in our inboxes (I say almost because there is the occasional
Quoted 1 line
Autoresponder by another name, see above URL.
Fortunately, the bulk of bulk mail I receive these days is forged but not forged from legitimate users. To give you an example from my daily log (which is e-mailed to me so I can keep an eye on the insanity):
2006-09-06T06:25:11 -- Challenged 'Beliefnet Daily Inspiration <[email protected]>' 2006-09-06T06:40:23 -- Challenged '"[email protected]" <[email protected]>' 2006-09-06T09:56:13 -- Challenged '"LexingtonLawBringsYou" <[email protected]>' 2006-09-06T12:25:34 -- Challenged '"OFFER CONFIRMATION." <[email protected]>' 2006-09-06T12:30:39 -- Challenged '"Rate Alert!" <[email protected]>' 2006-09-06T12:57:54 -- Challenged '"Rate Alert!" <[email protected]>' 2006-09-06T12:57:56 -- Challenged '"OFFER CONFIRMATION." <[email protected]>' 2006-09-06T13:08:02 -- Challenged '"PlatinumRewardsClubEmailOffers" <[email protected]>' 2006-09-06T13:34:18 -- Challenged '"CellPhoneGiveawaysNetDeals" <[email protected]>' 2006-09-06T13:39:23 -- Challenged '"Barber" <[email protected]>' 2006-09-06T13:59:36 -- Challenged '"Barber" <[email protected]>' 2006-09-06T14:08:44 -- Challenged '"LifeScript Healthy Advantage" <[email protected]>' 2006-09-06T14:27:00 -- Challenged 'FS Report <[email protected]>' 2006-09-06T14:46:12 -- Challenged '"OFFER_C0NFIRMATI0N!" <[email protected]>' 2006-09-06T15:07:26 -- Challenged '"Maureen&Team" <[email protected]>' 2006-09-06T15:07:27 -- Delivered message from 'Sune Kloppenborg Jeppesen <[email protected]>' (whitelist) 2006-09-06T15:09:30 -- Challenged '"BHG.com Kitchen"<[email protected]>' 2006-09-06T15:11:40 -- Challenged '"1 2 3 I n k Jets" <[email protected]>'
If these challenges bounce (_many_ of them do), the box and host end up on the blacklist.
Quoted 3 lines
..Stu
Thanks,
Chase
Quoted 24 lines · Stuart MacDonald
From: On Behalf Of Chase Venters
Quoted 3 lines
You can check the From: or envelope sender against the subscriber database. Forgery isn't a concern because we're not trying to stop forgery with this method. Subscribers subscribing one address
Forgery is always a concern...
Quoted 5 lines
The perl script behaves as an optional autoresponder. Autoresponders would
respond to spam as well (well, unless you put a spam filter in front of
them, but I assume that many don't)...because autoresponders are always replying to forged addresses:
http://www.spamcop.net/fom-serve/cache/329.htmlQuoted 7 lines
Also note that a number of people (myself included, at work anyway) have
perl scripts that respond to all incoming mail and require a reply cookie from original
envelope senders. We do it because it almost entirely prevents spam from
arriving in our inboxes (I say almost because there is the occasional
Quoted 1 line
Autoresponder by another name, see above URL.
I should also point out that common and regular mailing list software already often behaves as an autoresponder, and it is completely reasonable! Suppose that you send a message to a mailing list that is subscriber-only. What usually happens? You get mail back saying that your message has been queued for moderator review!
Naturally, such a system suffers from the same problems described by the Spamcop page you linked. But it is unreasonable to ask list managers not to respond to unknown traffic, because sending a message to a list and having it silently disappear is unacceptable.
Now, I'm sure there are some people that don't run mailing lists that would love to call this behavior 'bad'. But there are also people who would like to rewrite the rules for Internet mail (see: SPF and the problem with mail forwarders, and their so-called 'solution'). Since Internet mail was designed in a vacuum where these modern problems don't exist, we're all forced to work around them in unusual ways. I find it highly ironic that spam blocker services tell you not to use certain techniques (autoresponders, bounce messages) that are not only commonplace, but precedented and even mandated by RFC on the grounds that they may cause you to be blocked. Then they move on to criticize anti-spam techniques that fall in these domains with one of their subpoints saying 'they can cause you to miss legitimate mail!'
Guess what: so does indiscriminately blocking people whose sites don't bow down to your unreasonable demands, especially when their behavior (say, sending bounce messages) is described in the official protocol documentation.
Taking away auto-responders is like taking away hair gel from airline passengers: a gross overreaction that diminishes the quality of service for everyone.
Quoted 2 lines
..Stu
Thanks,
Chase
From: Chase Venters [mailto:[email protected]]
Quoted 10 lines
highly ironic that spam blocker services tell you not to use certain techniques (autoresponders, bounce messages) that are not only commonplace, but precedented and even mandated by RFC on the grounds that
they may cause you to be blocked. Then they move on to criticize anti-spam
techniques that fall in these domains with one of their subpoints saying
'they can cause you to miss legitimate mail!'
Quoted 6 lines
Guess what: so does indiscriminately blocking people whose sites don't bow
down to your unreasonable demands, especially when their behavior (say,
sending bounce messages) is described in the official protocol documentation.
I will assume you are referring to SpamCop. Their service does not behave the way you think it does:
http://forum.spamcop.net/forums/index.php?autocom=custom&page=whatis
Read the paragraph: "SpamCop works exactly like the credit reporting agencies..."
..Stu
Quoted 18 lines · Stuart MacDonald
From: Chase Venters [mailto:[email protected]]
Quoted 10 lines
highly ironic that spam blocker services tell you not to use certain techniques (autoresponders, bounce messages) that are not only commonplace, but precedented and even mandated by RFC on the grounds that
they may cause you to be blocked. Then they move on to criticize anti-spam
techniques that fall in these domains with one of their subpoints saying
'they can cause you to miss legitimate mail!'Quoted 6 lines
Guess what: so does indiscriminately blocking people whose sites don't bow
down to your unreasonable demands, especially when their behavior (say,
sending bounce messages) is described in the official protocol documentation.
Quoted 6 lines
I will assume you are referring to SpamCop. Their service does not behave the way you think it does:
http://forum.spamcop.net/forums/index.php?autocom=custom&page=whatis
Read the paragraph: "SpamCop works exactly like the credit reporting agencies..."
What are you implying - that SpamCop doesn't make decisions about who to block and who to not block for third parties? Their weasel wording comparing themselves to credit reporting is pretty weak and doesn't stand
a chance of obscuring their purpose, methods or results from anyone with half
a clue. But let's pretend that it did for just long enough that I can point out that because of the coercive nature of credit agency decisions, they are expected to behave reasonably as well. Saying, "but the creditors are making the decisions and we're just providing data!" is no excuse for credit reporting agencies to, say, give low credit scores to people who live in a certain part of town, or practice a certain religion, or happen to have skin of a certain color.
I will strongly criticize any service that purports to label senders of automatic responses as senders of unsolicited mail. The response sent by an autoresponder (be it a deferral, bounce, vacation or mailing list manager message) is most definitely solicited. The fact that Internet mail is broken enough to allow terribly easy envelope forgery does not change this fact. Trying to defer responsibility for a misdirected automatic response to the program or party using the program that sent it is like trying to blame gun manufacturers for specific instances of murder; it's absurd and it misses the point.
Whether or not SpamCop servers actually drop any messages is irrelevant when they are providing purportedly reputable data to third parties who use it to decide whether or not to drop messages themselves. The sad fact is that most postmasters just aren't educated enough about these issues to see through the glossy marketing materials RBLs and other services use to promote their wares.
And on the specific issue of autoresponders, I think a reasonable compromise is to support DomainKeys. That way if a sender is irritated that they are receiving automatic responses from messages they didn't send, they can personally take action to invalidate the forgery.
But mark my words: Asking hosts to stop sending bounce messages or automatic responses is insane and contrary to over a decade of established postmaster precedent.
Quoted 3 lines
..Stu
Thanks,
Chase
From: Chase Venters [mailto:[email protected]]
Quoted 3 lines
What are you implying - that SpamCop doesn't make decisions about who to
block and who to not block for third parties? Their weasel wording
It's not an implication, it's a fact.
Quoted 3 lines
I will strongly criticize any service that purports to label senders of
automatic responses as senders of unsolicited mail. The
What would you call it then when I receive a bounce/etc that is in reponse to a message someone else sent? I certainly never solicited that.
Perhaps you could send me your snail mail address; I'll solicit some junk mail but put your address down. But don't call it junk mail when you receive it, because it was solicited!
Quoted 5 lines
And on the specific issue of autoresponders, I think a reasonable compromise is to support DomainKeys. That way if a sender is irritated
that they are receiving automatic responses from messages they didn't send, they can personally take action to invalidate the forgery.
IMO one should never have to receive "automatic responses from messages they didn't send".
Quoted 4 lines
But mark my words: Asking hosts to stop sending bounce messages or automatic responses is insane and contrary to over a decade of established
postmaster precedent.
Things change.
..Stu
Quoted 13 lines · Chase Venters
Quoted 8 lines · On Wed, 6 Sep 2006, [email protected] wrote:
Quoted 2 lines
OK, but doing something could simply consist in adding a header that anyone is free to filter on or not.
The problem with that is the post gets no indication that his mail has been filtered. The way it works now is the rejection happens at SMTP time and that causes the poster to see the problem. If people filtered on a header, you'd never know why you weren't getting a response.
How about this:
Quoted 11 lines
1. Incoming mail from subscribers is accepted
2. Incoming mail to honeypot addresses is trained as SPAM
3. Incoming mail from non-subscribers is marked with X-Bogofilter:
4. A handy Perl script subscribes to lkml, and for any message it gets with an X-Bogofilter: SPAM header, it sends a notification (rate-limited) to the message sender explaining that his message will be filtered as SPAM by some recipients, and inviting him to contact postmaster to resolve the issue, and additionally letting him know that notification is rate-limited and there is a website he can check to see the SUBJECTs of all messages filtered as SPAM on lkml (say for the last week or two) if he wants to try and correct the problem himself.
Actually...
At front-door the bogofilter analyzes message for spam-signature and does classification. It reports the class to SMTP receiver's content policy controller, that usually chooses to believe it.
A number of recipient addresses are considered "filter free" and they do always get messages no matter what BF or other content filters consider the email.
The SMTP receiver also recognizes diffs in texts (but only when unquoted) and exempts them of BF rulings (which sometimes do bite on diffs.)
Honeypots train BF for SPAM.
Majordomo has tons of 'TABOO' filters and any match at them trains BF also for SPAM.
Any message that was successfully accepted to any list is trained as HAM.
I am considering teaching the system to recognize VGER's lists specially, and to verify that sender is in the list, or at possible extra 'posters' list. (Did I create one, or did I plan only ?) Anyway, if MAIL FROM address is at either, then message is again exempted of Bogofilter.
Presently majordomo and front-end SMTP are not in any direct contact, and that kind of policy decissions are not made at SMTP input time. They are done silently a bit latter.
I can move most of the Majordomo policy rules to front-end policy controller -- it is perl, after all :-)
Quoted 2 lines
Thanks,
Chase
/Matti Aarnio -- one of <[email protected]>
Quoted 8 lines · On Wed, Sep 06, 2006 at 11:56:28AM -0700, [email protected] wrote:
Quoted 2 lines
OK, but doing something could simply consist in adding a header that anyone is free to filter on or not.
The problem with that is the post gets no indication that his mail has been filtered. The way it works now is the rejection happens at SMTP time and that causes the poster to see the problem. If people filtered on a header, you'd never know why you weren't getting a response.
Valid point.
Regards,
Willy