PATCHES: richer listtext

PATCH Ben Schmidt 1 participant 1 message Feb 25, 2010

Share & tools

Ben Schmidt

author

2010-02-25 11:44 UTC #0 lore raw

Hi, all,

Given there has been positive interest, and there will be minimal/no disruption to mlmmj's operation, I have implemented the richer listtext features I wrote about earlier.

I'm sending these patches through now, though they have had only minimal testing, particularly as Robin Johnson has volunteered to help test.

mlmmj-listtext-headers.patch
- Allows arbitrary headers to be included at the beginning of listtexts.
- All the headers in the listtext will be included in the mails mlmmj sends.
- Mlmmj standard headers of the same name as those in the listtext will be omitted.
- Substitutions using the $whatever$ mechanism can be used.
- \uNNNN escaping is supported in the headers, as with the rest of the listtext.
- Automatic =?utf-8?q?...?= quoting is done for (and only for) the
Subject: header.
- Also includes a couple of bugfixes related to sending digests.
- This is compatible with current behaviour.
- The interface to prepstdreply() is changed, however; there is no longer a customheaders argument. It was never used, and since this patch allows custom headers to be included in listtext, is not really necessary as a function argument. Incorporating it in the implementation would have been more trouble than value.

mlmmj-origmail-count.patch
- Alters the way $originalmail$ works; it doesn't buffer the entire $originalmail$ substitution in memory, but does it a line at a time.
- $originalmail$ must be first on a line, optionally preceded by whitespace. The whitespace is prepended to each line of the mail that is included.
- The rest of the line following $originalmail$ is ignored.
- $originalmailNNN$ can be used, where NNN is a number of lines to include. Use a large number such as 1000000000 to include the whole mail (but be aware of integer overflow so don't go too big!).
- If NNN is omitted, the default is 100.
- To get current behaviour, a space must be prepended to the lines currently containing $originalmail$ in the listtexts. Apart from that, this is compatible with current behaviour.
- Again, there is an interface change. substitute() and substitute_one() no longer take the original mail filename, and cannot be used to do $originalmail$ substitution. This was never used except by prepstdreply() which now incorporates that substitution itself.

mlmmj-subject-substitute.patch
- Allows $subject$ to be used for the subject of the mail being moderated (or denied, etc.).
- Compatible.

mlmmj-random-substitute.patch
- Allows $random0$ through $random5$ to be used as distinct random strings.
- Compatible.

mlmmj-efficient-random.patch
- Makes random number generation more efficient by only seeding the generator once.
- Compatible, though behaviour will change slightly.

mlmmj-fixed-length-random.patch
- Makes the random strings produced always the same length rather the smaller random numbers producing shorter strings which could be problematic.
- Compatible, though behaviour will change slightly.

I still plan to include proper documentation (in the patches or as a separate patch), but for testing purposes, this email should suffice.

The moderation listtext I have tested with is attached.

I believe the patches will not all apply cleanly unless you have already applied these patches:

- patch-mlmmj-1.2.17-origmail.diff ([email protected])
- mlmmj-notifymod.patch ([email protected])

(I've attached them for convenience, since one of them doesn't seem to have made it into the web archive, and the +get trick hasn't worked for me yet, though I suspect the latter problem is just a delay.)

Nor will all apply cleanly unless you apply them in order.

If you have trouble, let me know. It's possible another of my patches I have applied affects this, though I don't think so.

The following is a rough list of things that should be tested that I made as I implemented:

- digests with and without thread summary work
- Content-Transfer-Encoding headers correctly output for both digest and listtext
- multiple headers of same type in listtext are preserved
- \uNNNN escaping works
- number of lines of $originalmail$ correctly honoured
- when larger than mail length
- when smaller than mail length
- when equal to mail length
- number of lines of $originalmail$ handled as 100 when not given
- whitespace preceding $originalmail$ correctly prepended to every line of included mail
- content after $originalmail$ ignored (no crash)
- six random values are distinct, but the same when reused in the same listtext
- subject substitution
- including when original subject contains utf8
- headerless listtext (warnings output)
- with and without blank line
- invalid header in listtext OK (no crash)

Help testing will be particularly appreciated. Code review and other feedback also welcome.

Ben.

include/prepstdreply.h +1 -1

diff -r 6e97ed95f9a8 include/prepstdreply.h
@@ -32,6 +32,6 @@
 int open_listtext(const char *listdir, const char *filename);
 char *prepstdreply(const char *listdir, const char *filename, const char *from,
 		const char *to, const char *replyto, size_t tokencount,
-		char **data, char *customheaders, const char *mailname);
+		char **data, const char *mailname);
 
 #endif /* PREPSTDREPLY_H */

src/mlmmj-bounce.c +1 -1

diff -r 6e97ed95f9a8 src/mlmmj-bounce.c
@@ -146,7 +146,7 @@
 
 	maildata[1] = indexstr;
 	queuefilename = prepstdreply(listdir, "bounce-probe", "$listowner$",
-					myaddr, NULL, 1, maildata, NULL, NULL);
+					myaddr, NULL, 1, maildata, NULL);
 	MY_ASSERT(queuefilename);
 	myfree(indexstr);

src/mlmmj-process.c +6 -6

diff -r 6e97ed95f9a8 src/mlmmj-process.c
@@ -133,7 +133,7 @@
 	myfree(listfqdn);
 
 	queuefilename = prepstdreply(listdir, "moderation", "$listowner$",
-				     to, replyto, 2, maildata, NULL,
+				     to, replyto, 2, maildata,
 				     mailfilename);
 
 	/* we might need to exec more than one mlmmj-send */

@@ -176,7 +176,7 @@
 
 	queuefilename = prepstdreply(listdir, "moderation-poster",
 				     "$listowner$", efromsender,
-				     NULL, 1, maildata+2, NULL, mailfilename);
+				     NULL, 1, maildata+2, mailfilename);
 
 	execlp(mlmmjsend, mlmmjsend,
 			"-l", "1",

@@ -696,7 +696,7 @@
 			queuefilename = prepstdreply(listdir,
 					"maxmailsize", "$listowner$",
 					fromemails.emaillist[0],
-					NULL, 2, maildata, NULL, donemailname);
+					NULL, 2, maildata, donemailname);
 			MY_ASSERT(queuefilename)
 			myfree(listdelim);
 			myfree(listname);

@@ -811,7 +811,7 @@
 				     listfqdn);
 		queuefilename = prepstdreply(listdir, "notintocc",
 					"$listowner$", fromemails.emaillist[0],
-					     NULL, 0, NULL, NULL, donemailname);
+					     NULL, 0, NULL, donemailname);
 		MY_ASSERT(queuefilename)
 		myfree(listdelim);
 		myfree(listname);

@@ -872,7 +872,7 @@
 					"bounces-help@", listfqdn);
 			queuefilename = prepstdreply(listdir, "subonlypost",
 					"$listowner$", fromemails.emaillist[0],
-						     NULL, 1, maildata, NULL, donemailname);
+						     NULL, 1, maildata, donemailname);
 			MY_ASSERT(queuefilename)
 			myfree(listaddr);
 			myfree(listdelim);

@@ -926,7 +926,7 @@
 			queuefilename = prepstdreply(listdir, "access",
 							"$listowner$",
 							fromemails.emaillist[0],
-						     NULL, 0, NULL, NULL, donemailname);
+						     NULL, 0, NULL, donemailname);
 			MY_ASSERT(queuefilename)
 			myfree(listaddr);
 			myfree(listdelim);

src/mlmmj-sub.c +6 -6

diff -r 6e97ed95f9a8 src/mlmmj-sub.c
@@ -147,7 +147,7 @@
 	maildata[5] = moderators;
 
 	queuefilename = prepstdreply(listdir, "submod-moderator",
-				"$listowner$", to, replyto, 3, maildata, NULL, NULL);
+				"$listowner$", to, replyto, 3, maildata, NULL);
 	
 	myfree(maildata[1]);

@@ -189,7 +189,7 @@
 
 	from = concatstr(4, listname, listdelim, "bounces-help@", listfqdn);
 	queuefilename = prepstdreply(listdir, "submod-requester", "$listowner$",
-					subaddr, NULL, 0, NULL, NULL, NULL);
+					subaddr, NULL, 0, NULL, NULL);
 	
 	myfree(listname);
 	myfree(listfqdn);

@@ -282,7 +282,7 @@
 	}
 
 	queuefilename = prepstdreply(listdir, listtext, "$helpaddr$",
-				     subaddr, NULL, 0, NULL, NULL, NULL);
+				     subaddr, NULL, 0, NULL, NULL);
 	MY_ASSERT(queuefilename);
 	myfree(listtext);

@@ -329,7 +329,7 @@
 	}
 
 	queuefilename = prepstdreply(listdir, listtext, "$listowner$",
-				"$listowner$", NULL, 1, maildata, NULL, NULL);
+				"$listowner$", NULL, 1, maildata, NULL);
 	MY_ASSERT(queuefilename)
 	myfree(listtext);
 	myfree(maildata[1]);

@@ -416,7 +416,7 @@
 	maildata[3] = mystrdup(confirmaddr);
 
 	queuefilename = prepstdreply(listdir, listtext, "$helpaddr$", subaddr,
-				     confirmaddr, 2, maildata, NULL, NULL);
+				     confirmaddr, 2, maildata, NULL);
 
 	myfree(maildata[1]);
 	myfree(maildata[3]);

@@ -469,7 +469,7 @@
 	myfree(listdelim);
 
 	queuefilename = prepstdreply(listdir, "sub-subscribed", "$helpaddr$",
-				     subaddr, NULL, 0, NULL, NULL, NULL);
+				     subaddr, NULL, 0, NULL, NULL);
 	MY_ASSERT(queuefilename);
 
 	myfree(listaddr);

src/mlmmj-unsub.c +4 -4

diff -r 6e97ed95f9a8 src/mlmmj-unsub.c
@@ -76,7 +76,7 @@
 	}
 
 	queuefilename = prepstdreply(listdir, listtext, "$helpaddr$",
-				     subaddr, NULL, 0, NULL, NULL, NULL);
+				     subaddr, NULL, 0, NULL, NULL);
 	MY_ASSERT(queuefilename);
 	myfree(listtext);

@@ -124,7 +124,7 @@
 	}
 	
 	queuefilename = prepstdreply(listdir, listtext, "$listowner$",
-				     "$listowner$", NULL, 1, maildata, NULL, NULL);
+				     "$listowner$", NULL, 1, maildata, NULL);
 	MY_ASSERT(queuefilename);
 	myfree(listtext);
 	myfree(maildata[1]);

@@ -213,7 +213,7 @@
 	maildata[3] = mystrdup(confirmaddr);
 
 	queuefilename = prepstdreply(listdir, listtext, "$helpaddr$", subaddr,
-				     confirmaddr, 2, maildata, NULL, NULL);
+				     confirmaddr, 2, maildata, NULL);
 
 	myfree(maildata[1]);
 	myfree(maildata[3]);

@@ -305,7 +305,7 @@
 	myfree(listdelim);
 
 	queuefilename = prepstdreply(listdir, "unsub-notsubscribed",
-				     "$helpaddr$", subaddr, NULL, 0, NULL, NULL, NULL);
+				     "$helpaddr$", subaddr, NULL, 0, NULL, NULL);
 	MY_ASSERT(queuefilename);
 
 	myfree(listaddr);

src/prepstdreply.c +155 ~~-79~~

272 lines not on this page · raw message

src/send_digest.c +3 -1

diff -r 6e97ed95f9a8 src/send_digest.c
@@ -321,7 +321,7 @@
 
 		tmp = concatstr(3, "\n--", boundary,
 				"\nContent-Type: text/plain; charset=UTF-8"
-				"\nContent-Encoding: 8bit"
+				"\nContent-Transfer-Encoding: 8bit"
 				"\n\n");
 		if (writen(fd, tmp, strlen(tmp)) == -1) {
 			log_error(LOG_ARGS, "Could not write digest text/plain"

@@ -376,6 +376,8 @@
 		}
 
 		close(txtfd);
+	} else if (txtfd > 0) {
+		close(txtfd);
 	}
 
 	myfree(line);

src/send_help.c +1 -1

diff -r 6e97ed95f9a8 src/send_help.c
@@ -57,7 +57,7 @@
 	myfree(listdelim);
 
 	queuefilename = prepstdreply(listdir, textfile, "$listowner$",
-					emailaddr, NULL, 0, NULL, NULL, NULL);
+					emailaddr, NULL, 0, NULL, NULL);
 	if(queuefilename == NULL) {
 		log_error(LOG_ARGS, "Could not prepare %s mail", name);
 		exit(EXIT_FAILURE);

src/send_list.c +1 -1

diff -r 6e97ed95f9a8 src/send_list.c
@@ -99,7 +99,7 @@
 	myfree(listdelim);
 
 	queuefilename = prepstdreply(listdir, "listsubs", "$listowner$",
-					emailaddr, NULL, 0, NULL, NULL, NULL);
+					emailaddr, NULL, 0, NULL, NULL);
 	if(queuefilename == NULL) {
 		log_error(LOG_ARGS, "Could not prepare sub list mail");
 		exit(EXIT_FAILURE);

include/prepstdreply.h +2 -3

diff -r b4d1fe2b5b93 include/prepstdreply.h
@@ -25,10 +25,9 @@
 #define PREPSTDREPLY_H
 
 char *substitute(const char *line, const char *listaddr, const char *listdelim,
-		size_t datacount, char **data, const char* mailname);
+		size_t datacount, char **data);
 char *substitute_one(const char *line, const char *listaddr,
-		const char *listdelim, size_t datacount, char **data,
-		const char* mailname);
+		const char *listdelim, size_t datacount, char **data);
 int open_listtext(const char *listdir, const char *filename);
 char *prepstdreply(const char *listdir, const char *filename, const char *from,
 		const char *to, const char *replyto, size_t tokencount,

src/prepstdreply.c +68 ~~-42~~

172 lines not on this page · raw message

src/send_digest.c +2 -2

diff -r b4d1fe2b5b93 src/send_digest.c
@@ -263,7 +263,7 @@
 		utfsub = unistr_escaped_to_utf8(line + 9);
 	}
 
-	utfsub2 = substitute(utfsub, listaddr, listdelim, 5, subst_data, NULL);
+	utfsub2 = substitute(utfsub, listaddr, listdelim, 5, subst_data);
 	subject = unistr_utf8_to_header(utfsub2);
 	myfree(utfsub);
 	myfree(utfsub2);

@@ -362,7 +362,7 @@
 				myfree(line);
 
 				tmp = substitute(utfline, listaddr, listdelim,
-						5, subst_data, NULL);
+						5, subst_data);
 				myfree(utfline);
 
 				if(writen(fd, tmp, strlen(tmp)) < 0) {

src/mlmmj-process.c +32 ~~-13~~

148 lines not on this page · raw message

src/prepstdreply.c +46 ~~-31~~

170 lines not on this page · raw message

src/random-int.c +4 -0

diff -r b443b4df35ff -r 631a406a220a src/random-int.c
@@ -30,10 +30,13 @@
 
 int random_int()
 {
+	static int init = 0;
 	unsigned int seed;
 	int devrandom;
 	unsigned char ch;
 
+	if (init) return rand();
+
 	seed = (unsigned int)time(NULL);
 
 	devrandom = open("/dev/urandom", O_RDONLY);

@@ -53,6 +56,7 @@
 	}
 
 	srand(seed);
+	init = 1;
 
 	return rand();
 }

src/strgen.c +2 -2

diff -r 3478ff663b92 -r 4daaf7c66e1e src/strgen.c
@@ -40,10 +40,10 @@
 
 char *random_str()
 {
-	size_t len = 128;
+	size_t len = 17;
 	char *dest = mymalloc(len);
 
-	snprintf(dest, len, "%x%x", random_int(), random_int());
+	snprintf(dest, len, "%08x%08x", random_int(), random_int());
 
 	return dest;
 }

Subject: Moderation needed: $subject$
MIME-Version: 1.0
Content-Type: multipart/mixed;
boundary="=_$random1$_="
Content-Transfer-Encoding: 8bit

--=_$random1$_=
Content-Type: text/plain; charset=utf8
Content-Transfer-Encoding: 8bit

Hello,

Someone has submitted a message for moderation to $listaddr$. It is attached.

To accept it send a message to:

$moderateaddr$

Your mailer probably automatically replies to this address, when you hit the reply button. If you don't want it sent to the list, simply ignore this message.

The following moderators have received this mail:

$moderators$

--=_$random1$_=
Content-Type: message/rfc822
Content-Transfer-Encoding: 8bit
Content-Disposition: attachment; filename="message.eml"

$originalmail1000000000$
--=_$random1$_=--

Index: src/prepstdreply.c
===================================================================
RCS file: /home/mmj/MMJCVS/mlmmj/src/prepstdreply.c,v retrieving revision 1.28

src/prepstdreply.c +1 -2

diff -u -r1.28 prepstdreply.c
@@ -145,10 +145,9 @@
 				value = concatstr(3, value, " ", str);
 				if(tmp)
 					myfree(tmp);
+				myfree(str);
 				count++;
 			}
-			if(str)
-				myfree(str);
 			close(mailfd);
 		}else{
 			log_error(LOG_ARGS, "Could not substitute $originalmail$ (mailname == %s)",mailname);

TUNABLES +5 -0

diff -r 2918436fa450 TUNABLES
@@ -103,6 +103,11 @@
    If this file is present, the owner(s) will get a mail with the address of
    someone sub/unsubscribing to a mailinglist.
 
+ ��� notifymod			(boolean)
+
+   If this file is present, the poster (based on the envelope from) will
+   get a mail when their post is being moderation.
+
  ��� digestinterval		(normal)
 
    This file specifies how many seconds will pass before the next digest is

contrib/web/perl-admin/conf/tunables.pl +5 -0

diff -r 2918436fa450 contrib/web/perl-admin/conf/tunables.pl
@@ -100,6 +100,11 @@
 			  "Notify subscribers",
 			  "If this option is set, the owner(s) will get a mail with the address of someone sub/unsubscribing to a mailinglist.");
 
+mlmmj_boolean("notifymod",
+			  "Notify moderation",
+			  "If this option is set, the poster (based on the envelope from) will ".
+			  "get a mail when their post is being moderation.");
+
 mlmmj_string("digestinterval",
 			 "Digest interval",
 			 "This option specifies how many seconds will pass before the ".

contrib/web/php-admin/conf/tunables.pl +5 -0

diff -r 2918436fa450 contrib/web/php-admin/conf/tunables.pl
@@ -100,6 +100,11 @@
 			  "Notify subscribers",
 			  "If this option is set, the owner(s) will get a mail with the address of someone sub/unsubscribing to a mailinglist.");
 
+mlmmj_boolean("notifymod",
+			  "Notify moderation",
+			  "If this option is set, the poster (based on the envelope from) will ".
+			  "get a mail when their post is being moderation.");
+
 mlmmj_string("digestinterval",
 			 "Digest interval",
 			 "This option specifies how many seconds will pass before the ".

listtexts/en/moderation-poster new file +11 -0

diff -r 2918436fa450 listtexts/en/moderation-poster
@@ -0,0 +1,11 @@
+Subject: Post waiting for approval
+
+Hi, this is the mlmmj program managing the mailinglist
+
+$listaddr$
+
+This list is configured to have moderated posts, the post has been queued
+for approval.
+
+--- Below this line are the first 100 lines of the message up for moderation --->
+$originalmail$

src/mlmmj-process.c +52 ~~-16~~

84 lines not on this page · raw message