Skip to contentSkip to conversation

mlmmj and spf

mlmmj theo borm 4 participants 13 messages May 11, 2012

lore
Share & tools
lore

theo borm

author
#0 lore raw

Dear list members,

We operate a small, closed, moderated mailing list that recently stopped working for a large part of its subscribers. The organization of which these subscribers are a member maintains an SPF record which denies access to all servers except a named few, which seems to be the cause of these problems.

As a work-around I set mlmmj to use a different from address in the "From:" header. This solution is, however, plainly bad as it removes the original sender from the headers. I have seen other lists use "Sender:" header, but results are a mixed bag. With strict SPF checking of the "From:" header in place these mails also don't pass.

What would be the correct solution?

any help, pointers greatly appreciated.

regards, Theo

#1 lore raw
Quoted 15 lines · theo borm

Dear list members,

We operate a small, closed, moderated mailing list that recently stopped working for a large part of its subscribers. The organization of which these subscribers are a member maintains an SPF record which denies access to all servers except a named few, which seems to be the
cause of these problems.

As a work-around I set mlmmj to use a different from address in the "From:" header. This solution is, however, plainly bad as it removes the original sender from the headers. I have seen other lists use "Sender:" header, but results are a mixed bag. With strict SPF checking of the "From:" header in place these mails also don't pass.

Quoted 5 lines

What would be the correct solution?

any help, pointers greatly appreciated.

regards, Theo

You have 2 options:

- get your mailserver in the SPF records for the domains for which you send mail (probably not a valid option, depending on the number of domains and the willingness of the DNS admins for those)
- make it so that the SMTP header "mail from" is set to the original sender, the "From:" part is in fact only data, it is the SMTP "mail from" that's being checked AFAIK.

I can't check it from here, so I can't say if rewriting just the SMTP "mail from" is possible in mlmmj.

Franky

author
#2 lore raw
In reply to theo borm

Thanks

I already contacted the organization and they replied: your server is not under our control, so therefore we will not put it in our SPF records (...). Check of the actual spf records reveals that microsoft, 2 /16 subnets, a few ISPs and a couple of others are already in there. Must be a very powerful organization to control all these.

Mlmmj as-is doesn't appear to have a way to *rewrite* headers. I can delete and insert headers no problem, but nothing in the way of rewriting or variable substitution on headers.

I just had a look at the sources. Hacking support for this into mlmmj-process.c/do_all_the_voodoo_here.c looks doable.

regards, Theo

do_all_the_voodoo_here

Quoted 48 lines · Marc MAURICE

Hello Theo,

The facts that the organization put this SPF means that the organization doesn't want to allow mail comming from @organization.org to be sent by external, non whitelisted servers.

Either ask the organization to add your server to the SPF, if possible. Or install the mlmmj on a server already whitelisted (probably not possible).

Otherwise the only solution I see is rewriting the From header as you said. You can always add the original sender in a header, or in the mail body (is it possible to add the sender in the body with mlmmj?)

From my little knowledge...

Regards,
Marc

Quoted 23 lines · theo borm

Dear list members,

We operate a small, closed, moderated mailing list that recently stopped working for a large part of its subscribers. The organization of which these subscribers are a member maintains an SPF record which denies access to all servers except a named few, which seems to be the cause of these problems.

As a work-around I set mlmmj to use a different from address in the "From:" header. This solution is, however, plainly bad as it removes the original sender from the headers. I have seen other lists use "Sender:" header, but results are a mixed bag. With strict SPF checking of the "From:" header in place these mails also don't pass.

What would be the correct solution?

any help, pointers greatly appreciated.

regards, Theo

#3 lore raw
In reply to theo borm
Quoted 6 lines · theo borm

We operate a small, closed, moderated mailing list that recently stopped working for a large part of its subscribers. The organization of which these subscribers are a member maintains an SPF record which denies access to all servers except a named few, which seems to be the cause of these problems.

It sounds like their SPF implementation is broken.

Quoted 5 lines

As a work-around I set mlmmj to use a different from address in the "From:" header. This solution is, however, plainly bad as it removes the original sender from the headers. I have seen other lists use "Sender:" header, but results are a mixed bag. With strict SPF checking of the "From:" header in place these mails also don't pass.

SPF checking should be done on the enevelope FROM address, not the From:
header contained inside the mail.

When mlmmj send out mails to the list subscribers the envelope from looks something like this:
[email protected]

So the SPF checking is done against the domain that the list is running on.

The correct cause of action would be to get the organization with broken SPF checking to fix that.

Signature · 1 line
Christian Laursen
#4 lore raw
In reply to theo borm

I second what Christian says. It sounds like the SPF checking is broken. It should be against the envelope-from. Checking against the "From:" header is wrong and bad for a number of reasons.

If they want to validate the "From:" and/or other headers, they should use DKIM. It will actually be effective and more efficient for validating headers.

I think it's pretty unlikely I would accept a change to Mlmmj to work around a buggy SPF implementation.

Cheers,

Ben.

Quoted 28 lines · Christian Laursen
Quoted 6 lines · theo borm

We operate a small, closed, moderated mailing list that recently stopped working for a large part of its subscribers. The organization of which these subscribers are a member maintains an SPF record which denies access to all servers except a named few, which seems to be the cause of these problems.

It sounds like their SPF implementation is broken.

Quoted 5 lines

As a work-around I set mlmmj to use a different from address in the "From:" header. This solution is, however, plainly bad as it removes the original sender from the headers. I have seen other lists use "Sender:" header, but results are a mixed bag. With strict SPF checking of the "From:" header in place these mails also don't pass.

SPF checking should be done on the enevelope FROM address, not the From: header contained inside the mail.

When mlmmj send out mails to the list subscribers the envelope from looks something like this:
[email protected]

So the SPF checking is done against the domain that the list is running on.

The correct cause of action would be to get the organization with broken SPF checking to fix that.

#5 lore raw
In reply to theo borm
Quoted 3 lines · Ben Schmidt

I second what Christian says. It sounds like the SPF checking is broken. It should be against the envelope-from. Checking against the "From:" header is wrong and bad for a number of reasons.

yes, I forgot that mlmmj already implements verp for the envelope from (or "mail from" as I simplisticly called it) :-)
So I agree here as well.

Franky

author
#6 lore raw
In reply to theo borm
Quoted 3 lines · Ben Schmidt

I think it's pretty unlikely I would accept a change to Mlmmj to work around a buggy SPF implementation.

Hi Ben,

I would be the last to suggest a patch to work around a Sender-Id misconfiguration. ;-)

However, a patch allowing slightly more advanced header manipulations (say regexp match/replace, between delheader and customheader), would /that/ be welcomed?

regards,

Theo

author
#7 lore raw
In reply to theo borm

Hi Christian,

Thanks for the answer.

Quoted 9 lines · Christian Laursen
Quoted 6 lines · theo borm

We operate a small, closed, moderated mailing list that recently stopped working for a large part of its subscribers. The organization of which these subscribers are a member maintains an SPF record which denies access to all servers except a named few, which seems to be the cause of these problems.

It sounds like their SPF implementation is broken.

microsoft?

Quoted 8 lines
Quoted 6 lines

As a work-around I set mlmmj to use a different from address in the "From:" header. This solution is, however, plainly bad as it removes the original sender from the headers. I have seen other lists use "Sender:" header, but results are a mixed bag. With strict SPF checking of the "From:" header in place these mails also don't pass.

SPF checking should be done on the enevelope FROM address, not the From: header contained inside the mail.

http://www.openspf.org/SPF_vs_Sender_ID summarizes this nicely:

<quote>
How will /Sender ID/ implementations violating the /SPF/ specification affect me?

If you have published an |v=spf1| policy to protect the use of your domain in the MAIL FROM and HELO addresses, /Sender ID/ implementations that apply your policy to /PRA/ (per RFC 4406) will reject your mail if you use your domain in the "|From|" (or generally /PRA/) header field while sending from (MAIL FROM) another system. </quote>

organization has an |v=spf1| policy in place. Mail is outsourced to microsoft, which uses sender ID.

It's the receiving server which has to implement spf/sender-ID, so delivery is erratic to say the least.

Quoted 3 lines

When mlmmj send out mails to the list subscribers the envelope from looks something like this:
[email protected]

listowner.tld doesn't have an spf record.

this is not the problem. problem is the "From" field/

Quoted 5 lines

So the SPF checking is done against the domain that the list is running on.

The correct cause of action would be to get the organization with broken SPF checking to fix that.

I (and others) have told them a few times to no avail.

regards, Theo

#8 lore raw
In reply to theo borm
Quoted 3 lines · theo borm
Quoted 1 line

It sounds like their SPF implementation is broken.

microsoft?

Well, you mentioned SPF in your original mail, but MS indeed uses Sender ID :-)

Quoted 5 lines

organization has an v=spf1 policy in place. Mail is outsourced to microsoft, which uses sender ID.

It's the receiving server which has to implement spf/sender-ID, so delivery is erratic to say the least.

Well, not the receiving end, but the one doing the DNS records. So try to convince the DNS owners for those domains to change their "spf2.0/mfrom,pra" to just "spf2.0/mfrom"

Quoted 4 lines

I (and others) have told them a few times to no avail.

regards, Theo

Of course MS won't listen :-) But if the DNS owners are reasonable ...

Franky

author
#9 lore raw
In reply to theo borm
Quoted 16 lines · Franky Van Liedekerke
Quoted 15 lines · theo borm

Hi Franky,

Thanks for the answer.

There are two types of "from"'s: the envelope sender as specified in the SMTP communication with the "mail from" command, and the one embedded in the mail data itself using the "From" header. Either can be checked with SPF and the microsoft variant (Sender ID), and both have a different way of interpreting these info.
Normall the envelope sender is used by mlmmj to implement "VERP", which allows it to efficiently detect which email addresses bounce and take appropriate action. It is also normally not shown to end-users, so it is not much use here. Besides, I don't want to break VERP processing.

Quoted 7 lines
Quoted 1 line

regards, Theo

yes, I forgot about the VERP ...
Let me brush off my SPF knowledge again, I thought there was an exception for mailing lists and the Return-Path header or so ...

Franky

Hi Franky,

Apologies for accidentally taking this off-list....

As it stands, broken or misconfigured sender-id / spf implementations/interpretations are being used. I doubt if reliance on headers that can just as easily be faked to define exceptions would be more reliable.

regards, Theo

author
#10 lore raw
In reply to theo borm
Quoted 8 lines · Franky Van Liedekerke
Quoted 3 lines · theo borm
Quoted 1 line

It sounds like their SPF implementation is broken.

microsoft?

Well, you mentioned SPF in your original mail, but MS indeed uses Sender ID :-)

sorry for the confusion.

Quoted 10 lines
Quoted 6 lines

organization has an v=spf1 policy in place. Mail is outsourced to microsoft, which uses sender ID.

It's the receiving server which has to implement spf/sender-ID, so delivery is erratic to say the least.

Well, not the receiving end, but the one doing the DNS records. So try to convince the DNS owners for those domains to change their "spf2.0/mfrom,pra" to just "spf2.0/mfrom"

It's microsoft who recommends using v=spf1 record as-if it is a spf2.0/mfrom,pra record. It is the receiving end (microsioft) using a sender ID implementation that is configured to interpret it this way. It is the DNS owner who *might* be able to circumvent this stupidity by using a spf2.0/mfrom record instead. But it may even be reasonable for the DNS owner to refuse this on the fundamental grounds that they shouldn't be fixing other peoples' problem.

Quoted 7 lines
Quoted 5 lines

I (and others) have told them a few times to no avail.

regards, Theo

Of course MS won't listen :-) But if the DNS owners are reasonable ...

I have warned the DNS owners ~ 2 years ago. Their response was that "they would discuss this with the "software vendor" (Wietse Venema), but that I should not expect a resolution". Until recently they operated their own mail servers, which applied the spf record correctly. Recently they have outsourced their mail to microsoft (externally their only MX is mail.messaging.microsoft.com), and that is when the trouble started.

Of course there has been trouble before, but these were "fringe users" on providers that used sender id, and the problem was simply solved by instructing them to use gmail or whatever. Now I have to tell the majority of list users to go to gmail because their IT department is what?.

well. It's a nuisance.

cheers, Theo

#11 lore raw
In reply to theo borm
Quoted 13 lines

http://www.openspf.org/SPF_vs_Sender_ID summarizes this nicely:

<quote>
How will /Sender ID/ implementations violating the /SPF/ specification affect me?

If you have published an |v=spf1| policy to protect the use of your domain in the MAIL FROM and HELO addresses, /Sender ID/ implementations that apply your policy to /PRA/ (per RFC 4406) will reject your mail if you use your domain in the "|From|" (or generally /PRA/) header field while sending from (MAIL FROM) another system. </quote>

Quoted 5 lines

organization has an |v=spf1| policy in place. Mail is outsourced to microsoft, which uses sender ID.

It's the receiving server which has to implement spf/sender-ID, so delivery is erratic to say the least.

Yeah. This kind of stuff is a PITA.

As well as the solution by publishing an spf2.0 record, the document you referenced suggests using the "Sender:" header. I can't really remember, but did you say you already tried that and it didn't help?

Quoted 7 lines
Quoted 3 lines

When mlmmj send out mails to the list subscribers the envelope from looks something like this:
[email protected]

listowner.tld doesn't have an spf record.

this is not the problem. problem is the "From" field/

Hmm. Well, when an implementation is broken, who knows how broken? Maybe the combination of a "Sender:" header and a published SPF record for that domain (which will also be the domain of the envelope-from) will help...if it can be done without too much hassle....

Ben.

#12 lore raw
In reply to theo borm
Quoted 11 lines · theo borm
Quoted 3 lines · Ben Schmidt

I think it's pretty unlikely I would accept a change to Mlmmj to work around a buggy SPF implementation.

I would be the last to suggest a patch to work around a Sender-Id misconfiguration. ;-)

However, a patch allowing slightly more advanced header manipulations (say regexp match/replace, between delheader and customheader), would /that/ be welcomed?

Hi, Theo,

Yes, I'd quite probably accept such a change, provided it is kept simple, and particularly if some specifics are floated past the mailing list prior to preparing a patch, so people have a chance to raise any issues.

There are also some pending changes to header stuff in the bug tracker, so it would be good to at least keep those in mind, or even implement them along with the new code (probably in separate patches, but nonetheless, the work is related and might be good to be implemented together).

Cheers,

Ben.