misc moderation patches

Here are some patches mostly related to moderating emails for discussion. I currently admin some lists with mailman, but would like to move to mlmmj.

mlmmj-X-List-Administrivia_header.patch

This patch adds two headers, an "X-List_administrivia: Yes" header for a few of the administrative messages that mlmmj sends out, and a header "X-mlmmj-mod-msg-id:" with the random string that is used as the argument for the 'moderate' command. The former is used similarly by mailman, and I filter based on it; I don't strictly need it if I have the latter header. The latter is really useful for writing scripts to moderate multiple messages at a time, rather than requiring them to parse out the Reply-To: header.

If it's deemed worthwhile to keep the administrivia header, there's likely more locations that could use it, or it should just be added automatically in prepstdreply(). Another alternative would be to add these headers in the listtexts.

mlmmj-multiple_moderation_args.patch

This patch adds the ability to moderate multiple messages by sending mlmmj a single message with multiple cookies addressed in the form

listname+moderate-COOKIE[-COOKIE...]@domain.tld

(This is one of my gripes with mailman, that in order to reject multiple spam messages by email, my script has to send a message for each message to reject, rather than just one message per list.) I left the moderation of subscriptions alone.

mlmmj-reject_command.patch

This adds a 'reject' command to support rejecting messages for moderation, rather than accepting them as the 'moderate' command does. It supports rejecting multiple messages by taking messages in the form

listname+reject-COOKIE[-COOKIE...]@domain.tld

I didn't add support for rejecting subscriptions; that should probably be fixed.

mlmmj-add_moderator_command.patch

This patch converts the From: address (currently "listname-moderators") in moderation messages to be a real address supported by mlmmj and adds the corresponding 'moderators' command. I'm not sure proper envelope filtering is done here, though.

As these patches touch similar areas of code, they are expected to be applied in the order above.

Thanks.

Steve Beattie
SUSE Labs, Novell Inc. 
<[email protected]>
http://NxNW.org/~steve/

---
 src/mlmmj-process.c |   11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

Index: b/src/mlmmj-process.c
===================================================================
@@ -79,6 +79,7 @@ void newmoderated(const char *listdir, c
 	int queuefd, moderatorsfd, mailfd;
 	size_t count = 0;
 	char *maildata[4] = { "moderateaddr", NULL, "moderators", NULL };
+	char *adminhdrs = "X-List-Administrivia: Yes";
 #if 0
 	printf("mailfilename = [%s], mailbasename = [%s]\n", mailfilename,
 			                                     mailbasename);

@@ -114,6 +115,7 @@ void newmoderated(const char *listdir, c
 
 	maildata[1] = replyto;
 	maildata[3] = moderators;
+	adminhdrs = concatstr(3, adminhdrs, "\nX-mlmmj-mod-msg-id: ", mailbasename);
 
 	from = concatstr(4, listname, listdelim, "owner@", listfqdn);
 	to = concatstr(3, listname, "-moderators@", listfqdn); /* FIXME JFA: Should this be converted? Why, why not? */

@@ -123,7 +125,7 @@ void newmoderated(const char *listdir, c
 	myfree(listfqdn);
 
 	queuefilename = prepstdreply(listdir, "moderation", "$listowner$",
-				     to, replyto, 2, maildata, NULL);
+				     to, replyto, 2, maildata, adminhdrs);
 
 	if((queuefd = open(queuefilename, O_WRONLY|O_APPEND)) < 0) {
 		log_error(LOG_ARGS, "Could not open '%s'", queuefilename);

@@ -347,6 +349,7 @@ int main(int argc, char **argv)
 	char *queuefilename, *recipextra = NULL, *owner = NULL;
 	char *maxmailsizestr;
 	char *maildata[4] = { "posteraddr", NULL, "maxmailsize", NULL };
+	char *adminhdr = "X-List-Administrivia: Yes";
 	char *envstr, *efrom;
 	struct stat st;
 	uid_t uid;

@@ -657,7 +660,7 @@ int main(int argc, char **argv)
 			queuefilename = prepstdreply(listdir,
 					"maxmailsize", "$listowner$",
 					fromemails.emaillist[0],
-					NULL, 2, maildata, NULL);
+					NULL, 2, maildata, adminhdr);
 			MY_ASSERT(queuefilename)
 			myfree(listdelim);
 			myfree(listname);

@@ -768,7 +771,7 @@ int main(int argc, char **argv)
 				     listfqdn);
 		queuefilename = prepstdreply(listdir, "notintocc",
 					"$listowner$", fromemails.emaillist[0],
-					     NULL, 0, NULL, NULL);
+					     NULL, 0, NULL, adminhdr);
 		MY_ASSERT(queuefilename)
 		myfree(listdelim);
 		myfree(listname);

@@ -829,7 +832,7 @@ int main(int argc, char **argv)
 					"bounces-help@", listfqdn);
 			queuefilename = prepstdreply(listdir, "subonlypost",
 					"$listowner$", fromemails.emaillist[0],
-						     NULL, 1, maildata, NULL);
+						     NULL, 1, maildata, adminhdr);
 			MY_ASSERT(queuefilename)
 			myfree(listaddr);
 			myfree(listdelim);

(Note: some mail clients like mutt have hard coded limits on the length of a To: address; e.g. mutt has a limit of 256 characters which it will truncate if the address is longer than that.)

---
 src/listcontrol.c |   95 +++++++++++++++++++++++++++++++++++++++---------------
 1 file changed, 69 insertions(+), 26 deletions(-)

---
 src/listcontrol.c |   26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)

Index: b/src/listcontrol.c
===================================================================
@@ -62,6 +62,7 @@ enum ctrl_e {
 	CTRL_CONFUNSUB,
 	CTRL_BOUNCES,
 	CTRL_MODERATE,
+	CTRL_REJECT,
 	CTRL_HELP,
 	CTRL_FAQ,
 	CTRL_GET,

@@ -92,6 +93,7 @@ static struct ctrl_command ctrl_commands
 	{ "confunsub",          1 },
 	{ "bounces",            1 },
 	{ "moderate",           1 },
+	{ "reject",             1 },
 	{ "help",               0 },
 	{ "faq",                0 },
 	{ "get",                1 },

@@ -620,8 +622,6 @@ int listcontrol(struct email_container *
 
 	/* listname+moderate-COOKIE[-COOKIE...]@domain.tld */
 	case CTRL_MODERATE:
-		/* TODO Add accept/reject parameter to moderate */
-
 		/* Subscriber moderation */
 		if(strncmp(param, "subscribe", 9) == 0) {
 			moderatefilename = concatstr(3, listdir, "/moderation/", param);

@@ -674,6 +674,28 @@ int listcontrol(struct email_container *
 		exit(rc);
 		break;
 
+	/* listname+reject-COOKIE[-COOKIE...]@domain.tld */
+	case CTRL_REJECT:
+		params = splitparam(param, '-');
+		for (i = 0; i < params->count; i++) {
+			moderatefilename = concatstr(3, listdir, "/moderation/", params->strs[i]);
+
+			if (stat(moderatefilename, &stbuf) < 0) {
+				/* no mail to moderate */
+				errno = 0;
+				log_error(LOG_ARGS, "A reject request was"
+					" sent with a mismatching cookie."
+					" Ignoring mail");
+			} else {
+				unlink(moderatefilename);
+				log_oper(listdir, OPLOGFNAME, "%s rejected %s",
+					fromemails->emaillist[0], moderatefilename);
+			}
+			myfree(moderatefilename);
+		}
+		exit(EXIT_SUCCESS);
+		break;
+
 	/* [email protected] */
 	case CTRL_HELP:
 		if(!strchr(fromemails->emaillist[0], '@')) {

---
 src/listcontrol.c   |   29 +++++++++++++++++++++++++++--
 src/mlmmj-process.c |    2 +-
 src/mlmmj-sub.c     |    2 +-
 3 files changed, 29 insertions(+), 4 deletions(-)

Index: b/src/mlmmj-sub.c
===================================================================
@@ -131,7 +131,7 @@ void moderate_sub(const char *listdir, c
 	modfilebase = mybasename(modfilename);
 
 	from = concatstr(4, listname, listdelim, "owner@", listfqdn);
-	to = concatstr(3, listname, "-moderators@", listfqdn);
+	to = concatstr(4, listname, listdelim, "moderators@", listfqdn);
 	replyto = concatstr(6, listname, listdelim, "moderate-", modfilebase,
 			"@", listfqdn);
 	myfree(modfilebase);

Index: b/src/mlmmj-process.c
===================================================================
@@ -118,7 +118,7 @@ void newmoderated(const char *listdir, c
 	adminhdrs = concatstr(3, adminhdrs, "\nX-mlmmj-mod-msg-id: ", mailbasename);
 
 	from = concatstr(4, listname, listdelim, "owner@", listfqdn);
-	to = concatstr(3, listname, "-moderators@", listfqdn); /* FIXME JFA: Should this be converted? Why, why not? */
+	to = concatstr(4, listname, listdelim, "moderators@", listfqdn); /* FIXME JFA: Should this be converted? Why, why not? */
 
 	myfree(listdelim);
 	myfree(listname);

Index: b/src/listcontrol.c
===================================================================
@@ -35,6 +35,7 @@
 #include "listcontrol.h"
 #include "find_email_adr.h"
 #include "getlistdelim.h"
+#include "getlistaddr.h"
 #include "strgen.h"
 #include "send_help.h"
 #include "send_list.h"

@@ -62,6 +63,7 @@ enum ctrl_e {
 	CTRL_CONFUNSUB,
 	CTRL_BOUNCES,
 	CTRL_MODERATE,
+	CTRL_MODERATORS,
 	CTRL_REJECT,
 	CTRL_HELP,
 	CTRL_FAQ,

@@ -93,6 +95,7 @@ static struct ctrl_command ctrl_commands
 	{ "confunsub",          1 },
 	{ "bounces",            1 },
 	{ "moderate",           1 },
+	{ "moderators",         0 },
 	{ "reject",             1 },
 	{ "help",               0 },
 	{ "faq",                0 },

@@ -137,6 +140,7 @@ int listcontrol(struct email_container *
 	char *bouncenr, *tmpstr;
 	char *param = NULL, *conffilename, *moderatefilename;
 	char *c, *archivefilename, *sendfilename;
+	char *listfqdn, *listname, *listaddr = getlistaddr(listdir), *listdelim;
 	const char *subswitch;
 	struct stat stbuf;
 	int closedlist, nosubconfirm, tmpfd, noget, i, closedlistsub,

@@ -216,8 +220,9 @@ int listcontrol(struct email_container *
 		return -1;
 	}
 
-	/* We only need the control mail when bouncing, to save bounced msg */
-	if(ctrl != CTRL_BOUNCES)
+	/* We only need the control mail when bouncing, to save
+	   bounced msg, and when sending to moderators */
+	if(ctrl != CTRL_BOUNCES && ctrl != CTRL_MODERATORS)
 		unlink(mailname);
 
 	switch (ctrl) {

@@ -696,6 +701,26 @@ int listcontrol(struct email_container *
 		exit(EXIT_SUCCESS);
 		break;
 
+	/* [email protected] */
+	case CTRL_MODERATORS:
+		listfqdn = genlistfqdn(listaddr);
+		listname = genlistname(listaddr);
+		listdelim = getlistdelim(listdir);
+		tmpstr = concatstr(4, listname, listdelim, "owner@", listfqdn);
+		log_oper(listdir, OPLOGFNAME, "mlmmj-recieve: sending"
+				" mail from %s to moderators",
+				fromemails->emaillist[0]);
+		execlp(mlmmjsend, mlmmjsend,
+				"-l", "2",
+				"-L", listdir,
+				"-m", mailname,
+				"-F", tmpstr,
+				"-a", (char *)NULL);
+		log_error(LOG_ARGS, "execlp() of '%s' failed",
+					mlmmjsend);
+		exit(EXIT_FAILURE);
+		break;
+
 	/* [email protected] */
 	case CTRL_HELP:
 		if(!strchr(fromemails->emaillist[0], '@')) {

Hi Steve,

Welcome, and thanks for contributing.

I've looked at your patches, and they look clean. I have a few comments. See below.

We're in the RC-period for 1.2.15 (feedback please!), so your patches will not be merged to CVS right now. Please remind me of them when 1.2.15 is out :)

[snip]

Yes, please do that.

[snip]

Could you add that as well?

[snip]

Yes, please. Also, please send out a reject mail. Otherwise it's a discard - not reject.

[snip]

Envelope filtering?

Morten

Morten K. Poulsen <[email protected]>
http://www.afdelingp.dk/

Yes, I was aware of the RC-release which is why I was posting for comment rather than inclusion. The patches I posted were against 1.2.15-RC1. Unfortunately, the timing failed to work out that I didn't get to these before the RC1 release; timing's never been my strong point. :-)

The only problems I've run into with the RC have been due to bogons introduced by myself, but I barely scratch the surface in terms of using the functionality of mlmmj. Others will need to comment on the current RC.

Sure, will do. I've pushed a quilt series up to
http://www.nxnw.org/~steve/warez/mlmmj/ which is where I'll update the patches to incorporate your feedback.

Fixed in the quilt series.

Possibly. The multiple cookie parsing splits on dashes, and I figured it would break on email addresses containing dashes as well, which is why I avoided adding support for multiple addresses. I'm not sure how exactly to format the reply commands in a way that will be easily parsable; ideas?

Okay.

Doh, yes, of course. Though would a discard command be accepted as well? The vast majority of moderation that I do is of spam rather than content-based, where the From: address has been forged; returning a rejected message would just add to the spam problem and not do anything useful. But for a content moderated list, it makes sense to send out a reject message; however, presumably in that situation a moderator would like to be able to send a custom message to explain why the message was rejected. Perhaps just returning the body of the moderation message to the original sender?

It's entirely probable I am confused. I was looking at the code in src/mlmmj-process.c that handles the 'owner' command:

   569                          /* strip envelope from before resending */
   570                          delheaders->count = 0;
   571                          delheaders->strs = NULL;
   572                          delheaders->strs = myrealloc(delheaders->strs,

573 (delheaders->count+3) * sizeof(char *));
574 delheaders->strs[delheaders->count++] =
575 mystrdup("From ");
576 delheaders->strs[delheaders->count++] =
577 mystrdup("Return-Path:");
578 delheaders->strs[delheaders->count] = NULL;

and (a) wasn't sure why this command was handled here rather than in the listcontrol(), and (b) assumed the 'moderators' command would need to do something similar. But SMTP envelopes are an area that I don't quite grok, and pointers to clues are welcome.

Thanks.

Steve Beattie
SUSE Labs, Novell Inc. 
<[email protected]>
http://NxNW.org/~steve/

Hi Steve,

:-)

Ok. Anybody else? Does the thing even compile on various systems?

It works for me on:
- Debian GNU/Linux 3.1
- FreeBSD 6.2
- OpenBSD 4.1

$ telnet www.nxnw.org 80
Trying 208.151.246.43...
telnet: connect to address 208.151.246.43: Connection timed out

(And no, I *don't* browse with telnet ;-)

[snip]

Cool.

But subscription moderation does not contain the (new) subscribers address in the argument. It is something like [email protected]

I am not sure why mmj reused the "moderate" keyword, and added the magic "subscribe" before the cookie. That might change in mlmmj-2.0.0 ;-)

Discard would be OK. Both would be perfect.

Yes, that is a good idea.

Oh yes, this code just removes "Return-Path: " and "From " headers, which contain the SMTP envelope from. Those headers are only useful when a mail is delivered (locally) and saved in a file. They should never be present inside the SMTP DATA. That is why they are being stripped here.

I am not sure:

    /* Why is this here, and not in listcontrol() ?  -- mortenp 20060409 */

mmj?

I think it is safe to move it.

Yes.

http://www.xkcd.com/c138.html ;-)

An example of a SMTP session might look like this:

# banner
S: 220 example.com SMTP server ready

# hello
C: HELO sender.test
S: 250 example.com

# envelope from
C: MAIL FROM:<[email protected]>
S: 250 OK

# envelope to
C: RCPT TO:<[email protected]>
S: 250 OK

# the mail itself (headers + body)
C: DATA
S: 354 Start mail input; end with <CRLF>.<CRLF>
C: From: Joe User <[email protected]>
C: To: Somebody <[email protected]>
C: Subject: blah blah blah
C:
C: mail body here
C: .
S: 250 OK, queued as 29381984

# end
C: QUIT
S: 221 Bye

The envelope addresses are used for mail (and bounce) routing. The addresses in the DATA section are only used by user agents (and some *very* broken transfer agents).

When delivering the mail to its final destination, the delivery agent usually adds the envelope addresses to the DATA section. They are often added as "Delivered-To: " (envelope to) and "Return-Path: " or "From " (note! no colon) (envelope from).

RFC 821, RFC 1869 and RFC 2821 are good references.

Morten

Morten K. Poulsen <[email protected]>
http://www.afdelingp.dk/

Hi Christoph,

No, not yet. MMJ reminded me about them a few weeks ago.

I just started a new job one month ago (Ad: PointZero Hosting), and I am currently commuting 5 hours a day, so probably not. But if there is a slow day at work any time soon, I will merge Steve's patches and roll a RC.

Does anybody else have patches, which I've forgotten about?

Morten

Dear Morten,

I sent you several months ago improved function recipient_extra that should work correct on any MTAs (especially with qmail) with any maillist aliases and with any commands.
Would be good if you accept it.

Best regards,
 Alexander

Yes, I remember it. I rejected it because it introduced a number of problems, without fully solving the issue (using mlmmj lists with the delimiter in their names with vpopmail).

Morten

Dear Morten,

Yes, you rejected small changes to if clauses. But later I sent the whole function that don't needed these changes. It doesn't affect old possibilities. It just adds new ones.
Could you please give me an example with a problem? Thanks.

Best regards,
 Alexander

They weren't applied because there were some issues you raised that I still need to address. Despite being laid off from Novell, I haven't had time to get back to them, but I'll try to work on them later this week. My apologies for not getting to it earlier.

Steve Beattie
<[email protected]>
http://NxNW.org/~steve/

Hi Steve,

Ah, ok. I had forgotten about that.

I am sorry to hear that.

No problem. I will wait for your patches, before rolling a RC.

Morten

And it breaks in some cases.

$ echo 'Delivered-To: test-list+' >/tmp/mail
$ ./mlmmj-process -L /home/mopo/mlmmj-test/ -m /tmp/mail
$ echo $?
1
$ tail -n 1 /var/log/mail.err 
Nov 12 22:38:34 mopo-laptop ./mlmmj-process[13111]: mlmmj-process.c:327:
malloc(-134578623) failed! Bailing out!: Cannot allocate memory

Morten