Skip to contentSkip to conversation

List archive howto?

mlmmj Morgan Gangwere 6 participants 9 messages Aug 18, 2010 – Oct 11, 2010

lore
Share & tools
lore
#0 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Howdy y'all

I'm kinda curious as to what is used for the archiving of the mailing list?

I've got my own list set up and I'm pouring through the documentation looking for what actually generates the html. Any help?

- --

Morgan Gangwere

Quoted 2 lines
Quoted 1 line

Why?

Because it breaks the logical flow of conversation, plus makes

messages unreadable.

Quoted 1 line
Quoted 1 line
Quoted 1 line

Top-Posting is evil.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)

iQIcBAEBAgAGBQJMaziCAAoJEEURiCSotvJDQmwP/jHrz4H2siLqZdsg4ZXvIz2A xC9H2iaynQ5hLGHwbsS8ontRKoodD69Z33NtWRdtLtmoAEQ4LoT2goZtDwmj9/VB cS1QpjSdkAaqoqeqL0iXx6yY6SvMZwigD1d/GCn3/VousqvOublR5F0VzNozAU5+ lPn6ty1EaP+wK1p9sktMWhN5DgF+KUNzDbc/v9Y6A75njp5SJgVfD3j20KBdkfw2 6B1ipnnAZu0kZLS1lEOj5CZYwQ4WaOQaL0F4lPYWUyOxRcE4zm+qi+XLiinacm2p nTZS3RBP3XguRKYTbyKcxvRVHxUCMQlH5Lhduowdeecsgj6csaDawKuD68viDfsE qIH+sjhn+eVwlMcEl1id1Mu5NDzH7S7cDDPDv2Kiz40NIQn0g2PqxPr7ibjsvWvg 7HJmQ4zfbEE/Wyfl45MgpsYhiq4wlVKFJnGgZTHyGaI33KVWIpSfkXOZvBo2iNzi bGv7bAbUyfLUwcoLI/r7m/2iCuN9obXinP+Px7aYJ/34EXIzYdrHjr/BiQYWwEhU wCU39EvN173Dv6FIiUUPySnGgOmubNzVXvcCVcovfw7tHv6VwR8WS06rVwDKW3Gk Rr29D7ksrjwN9i4lmQwuM2mB0xJrr9Yp6AXYIq0pjNhnwzy+m1/zW66Znf4p8q7G B6Ki/clGUYP+7OMtP5Tu
=PB0G
-----END PGP SIGNATURE-----

#1 lore raw
Quoted 1 line · On Wednesday 18 August 2010 03:33:54 Morgan Gangwere wrote:

Howdy y'all

Hi,

Quoted 4 lines

I'm kinda curious as to what is used for the archiving of the mailing list?

I've got my own list set up and I'm pouring through the documentation looking for what actually generates the html. Any help?

I've created mlmmj-webarchiver, you can get it here.

http://git.cynapses.org/projects/mlmmj-webarchiver.git/

Maybe somone wants to link it on the website. The output looks like this:

http://www.libssh.org/archive/

-- andreas

#2 lore raw
In reply to Morgan Gangwere
Quoted 3 lines · Morgan Gangwere

Howdy y'all

I'm kinda curious as to what is used for the archiving of the mailing list?

We're using hypermail to create the webarchive. I have also made my own script to control the use of hypermail.

Quoted 3 lines

I've got my own list set up and I'm pouring through the documentation looking for what actually generates the html. Any help?

I've attached my script, it is a cli php script

I also have a script to use it which is cleaning up after hypermail:
#!/bin/bash
/usr/local/bin/wwwmmm
find /var/www/mlmmj.org/archive -type d -exec chmod ug=rwx,o=x {} \; find /var/www/mlmmj.org/archive -type f -exec chmod ug=rw,o=r {} \;

--Wolf

#3 lore raw
In reply to Morgan Gangwere

I've drafted some documentation about this. It is here:

http://mlmmj.org/docs/readme-archives/

Comments?

Wolf, perhaps at some stage you'd like to tidy up your scripts, and document their dependencies and installation (e.g. with cron?) in a README, and I could whack them in contrib?

You're welcome to do the same if you like, Andreas, though if you prefer to keep your stuff in its existent web home, I reckon that's got its advantages, too.

Thanks to both of you for contributing to Mlmmj in this way.

Smiles,

Ben.

#4 lore raw
In reply to Morgan Gangwere
Quoted 6 lines · Morgan Gangwere

Howdy y'all

I'm kinda curious as to what is used for the archiving of the mailing list?

I've got my own list set up and I'm pouring through the documentation looking for what actually generates the html. Any help?

For Gentoo, we use mhonarc, with some minor modifications.

One of the key modifications was ultimately caused by the message numbering issue in threads.

Specifically, what URL should exist if a message arrives out of sequence due to a delay. Some archive tools renumber the urls, others keep an index based on some part of the mail. Both of these caused some degree of problems for us, and we had to come up with an alternative.

Specifically, when the message is being received, we add two headers:
X-Archives-Salt: ${UUID}
then we hash the entire message file, including the new header, and generate:
X-Archives-Hash: ${SHA1}

This gets used in the filename for the individual message files we write to disk:
http://archives.gentoo.org/gentoo-scm/msg_c0f2f8f123f85bb8b664827b4a1dcb09.xml
It's consistent regardless of how many times we have to rebuild a message archive.

We've got nearly 588k messages in the archive, and now the next problem is starting to be the storage/access scalability of some of the larger lists. Also need to work on better parallelization for building archives for each list.

Largest lists, by mails:
72808 gentoo-dev
97619 gentoo-user
283036 gentoo-commits

Footnote:
Yes, we generate XML, that's to integrate with our templating, but you could generate HTML even more easily.

Signature · 4 lines
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : [email protected]
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85
#5 lore raw
In reply to Morgan Gangwere

I'm afraid you've completely lost me there, Robin, possibly because I know basically nothing about mhonarc. The concepts sound great, though.

A question, though.... Is there anything specific and useful in your stuff reply that I should be working into the documentation about archive solutions? Or is this a complex solution, or not detailed enough, and best left out and revealed if someone finds the other solutions aren't adequate for them?

Cheers,

Ben.

Quoted 43 lines · Robin H. Johnson
Quoted 6 lines · Morgan Gangwere

Howdy y'all

I'm kinda curious as to what is used for the archiving of the mailing list?

I've got my own list set up and I'm pouring through the documentation looking for what actually generates the html. Any help?

For Gentoo, we use mhonarc, with some minor modifications.

One of the key modifications was ultimately caused by the message numbering issue in threads.

Specifically, what URL should exist if a message arrives out of sequence due to a delay. Some archive tools renumber the urls, others keep an index based on some part of the mail. Both of these caused some degree of problems for us, and we had to come up with an alternative.

Specifically, when the message is being received, we add two headers:
X-Archives-Salt: ${UUID}
then we hash the entire message file, including the new header, and generate:
X-Archives-Hash: ${SHA1}

This gets used in the filename for the individual message files we write to disk:
http://archives.gentoo.org/gentoo-scm/msg_c0f2f8f123f85bb8b664827b4a1dcb09.xml
It's consistent regardless of how many times we have to rebuild a message archive.

We've got nearly 588k messages in the archive, and now the next problem is starting to be the storage/access scalability of some of the larger lists. Also need to work on better parallelization for building archives for each list.

Largest lists, by mails:
72808 gentoo-dev
97619 gentoo-user
283036 gentoo-commits

Footnote:
Yes, we generate XML, that's to integrate with our templating, but you could generate HTML even more easily.

#6 lore raw
In reply to Morgan Gangwere
Quoted 8 lines · Ben Schmidt

I'm afraid you've completely lost me there, Robin, possibly because I know basically nothing about mhonarc. The concepts sound great, though.

A question, though.... Is there anything specific and useful in your stuff reply that I should be working into the documentation about archive solutions? Or is this a complex solution, or not detailed enough, and best left out and revealed if someone finds the other solutions aren't adequate for them?

Mhonarc itself is fairly simple to setup.

The URLs based on SHA1 help with larger lists in case you have to rebuild the archive only.

Signature · 4 lines
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : [email protected]
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85
#7 lore raw
In reply to Morgan Gangwere
Quoted 9 lines · Robin H. Johnson
Quoted 8 lines · Ben Schmidt

I'm afraid you've completely lost me there, Robin, possibly because I know basically nothing about mhonarc. The concepts sound great, though.

A question, though.... Is there anything specific and useful in your stuff reply that I should be working into the documentation about archive solutions? Or is this a complex solution, or not detailed enough, and best left out and revealed if someone finds the other solutions aren't adequate for them?

Quoted 4 lines

Mhonarc itself is fairly simple to setup.

The URLs based on SHA1 help with larger lists in case you have to rebuild the archive only.

Is this something that if I mention it in the docs a mhonarc user would know what to do to make it happen? If not, are you able to flesh it out a bit for me so I can include it?

Cheers,

Ben.

#8 lore raw
In reply to Morgan Gangwere

Hi Robin,

2010/10/3 Robin H. Johnson <[email protected]>:

Quoted 10 lines · Morgan Gangwere

For Gentoo, we use mhonarc, with some minor modifications.

One of the key modifications was ultimately caused by the message numbering issue in threads.

Specifically, what URL should exist if a message arrives out of sequence due to a delay. Some archive tools renumber the urls, others keep an index based on some part of the mail. Both of these caused some degree of problems for us, and we had to come up with an alternative.

can you share how you did thas? Would be interested to deploy something similar on our machines as well. ;)

Thanks,
Florian