infra: add the transient download mechanism
PATCH v2 1/1 infra: add the transient download mechanism
People have been hollering for a long time, asking for a way to be able to trigger a build that will always get the current HEAD (or whatever it is called outside of git) of a branch, whatever the conditions, so that they can trigger automated builds (e.g. in a CI or whatever) to test their greatest and latest changes, by pushing again and again ad libitum on the same branch, without having the need to update the sha1 in the .mk of the affected package.
We introduce the concept of a transient download, which means that Buildroot will never consider any already downloaded archive, and so will re-download it (but only if the package has not already been downloaded for the current build).
A package declares its download as transient with:
FOO_TRANSIENT_DOWNLOAD = YES
Since the check is done in the download wrapper, we have no TOCTOU race in case two builds would attempt the same transient download: the archive is only replaced atomically as usual.
So, if the package uses a branch as version, the branch's HEAD at that very moment will be re-downloaded.
Obviously, such builds are not reproducible.
This also has the pitfall that two builds in parallel may get slightly different content for the same branch, and the first build could end up using the download of the second build:
build-1 build-2
download
| download
| |
save to dl-dir |
[...] save to dl-dir
extract
Also, as noticed by Thomas, legal-info of build-1 is not correct when executed after build-2 downloaded the archive. But since such CI builds are expected to be done during development, legal-info is usually not that important in that case (and even then, such packages are probably proprietary packages that are probably "FOO_REDISTRIBUTE = NO" already anyway).
Furthermore, even with a single build, it might not get what it expects:
developer CI
git push branch
| trigger build-1
git push branch |
download
In that case the build in the CI gets the code of the second push, which might or might not be the expected behaviour.
We would also want to always print the "Downloading foo" message for transient packages, while keeping it silent for the already-downloaded packages. But that would make the code a bit more complex. We just simplify it by always displaying that message, even if there is actually nothing to download (this actually makes the download step more like the others, btw: if there is nothing to do to configure, we are still printing the "Configuring foo", "Building foo" messages, etc...).
Finally, by design, we can't check the hashes for a transient package.
For people who are aiming at their feet, we're now providing them with a loaded gun. ;-]
---
Changes v1 -> v2:
- typoes in commit log
Changes RFC -> v1:
- only be transient with 'YES', not with 'NO' (Thomas)
- add blurb about legal-info (Thomas)
- do not check hash for transient downloads
- rename variable: DOWNLOAD_TRANSIENT -> TRANSIENT_DOWNLOAD
- typoes fixed (and probably others added)
- always print "Downloading"
- rework the commit log
--- docs/manual/adding-packages-generic.txt | 27 ++++++++++++++++++++++--- package/pkg-download.mk | 1 + package/pkg-generic.mk | 20 +++++++++++------- support/download/dl-wrapper | 8 +++++--- 4 files changed, 43 insertions(+), 13 deletions(-)
docs/manual/adding-packages-generic.txt +24 -3
@@ -206,12 +206,14 @@ information is (assuming the package name is +libfoo+) : ** a tag for a git tree +LIBFOO_VERSION = v0.1.2+ + .Note: -Using a branch name as +FOO_VERSION+ is not supported, because it does -not and can not work as people would expect it should: +Using a branch name as +FOO_VERSION+, although technically possible, +is highly discouraged, because it does not and can not work as people +would expect it should: + 1. due to local caching, Buildroot will not re-fetch the repository, so people who expect to be able to follow the remote repository - would be quite surprised and disappointed; + would be quite surprised and disappointed (but see + +LIBFOO_TRANSIENT_DOWNLOAD+, later); 2. because two builds can never be perfectly simultaneous, and because the remote repository may get new commits on the branch anytime, two users, using the same Buildroot tree and building the same
@@ -342,6 +344,25 @@ not and can not work as people would expect it should: submodules when they contain bundled libraries, in which case we prefer to use those libraries from their own package. +* +LIBFOO_TRANSIENT_DOWNLOAD+ can be set to +YES+ or +NO+ (the default). + If set to +YES+, the download for that package will be attempted at + every build from scratch; any already downloaded archive is ignored + as if missing. This may help when a branch is specified in + +LIBFOO_VERSION+, and the head/tip of the branch is to be built, like + a CI pipeline would need for example. This still suffers from the + other issues listed above about using branches, though, the most + obvious being the *lack of reproducibility*. Besides, it has its own + pitfalls that one must be aware of as well: + ** when two builds are running in parallel on the same machine, there + is a TOCTOU race for each build to download and extract the archive, + so each build may not get what it downloaded, but what the other + build did download; + ** as a consequence, legal-info is not reliable when a package is + transient; + ** if a branch is pushed to an automated build bot (in a CI for + example), and then re-pushed to while the build is still in + progress, the build may get the wrong version of the branch. + * +LIBFOO_STRIP_COMPONENTS+ is the number of leading components (directories) that tar must strip from file names on extraction. The tarball for most packages has one leading component named
package/pkg-download.mk +1 -0
@@ -108,6 +108,7 @@ define DOWNLOAD -n '$($(2)_BASENAME_RAW)' \ -N '$($(2)_RAWNAME)' \ -o '$($(2)_DL_DIR)/$(notdir $(1))' \ + $(if $(filter YES,$($(2)_TRANSIENT_DOWNLOAD)),-F) \ $(if $($(2)_GIT_SUBMODULES),-r) \ $(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \ $(QUIET) \
package/pkg-generic.mk +13 -7
@@ -144,13 +144,7 @@ $(BUILD_DIR)/%/.stamp_downloaded: @$(call step_start,download) $(call prepare-per-package-directory,$($(PKG)_FINAL_DOWNLOAD_DEPENDENCIES)) $(foreach hook,$($(PKG)_PRE_DOWNLOAD_HOOKS),$(call $(hook))$(sep)) -# Only show the download message if it isn't already downloaded - $(Q)for p in $($(PKG)_ALL_DOWNLOADS); do \ - if test ! -e $($(PKG)_DL_DIR)/`basename $$p` ; then \ - $(call MESSAGE,"Downloading") ; \ - break ; \ - fi ; \ - done + @$(call MESSAGE,"Downloading") $(foreach p,$($(PKG)_ALL_DOWNLOADS),$(call DOWNLOAD,$(p),$(PKG))$(sep)) $(foreach hook,$($(PKG)_POST_DOWNLOAD_HOOKS),$(call $(hook))$(sep)) $(Q)mkdir -p $(@D)
@@ -563,6 +557,18 @@ ifndef $(2)_DL_OPTS endif endif +ifndef $(2)_TRANSIENT_DOWNLOAD + ifdef $(3)_TRANSIENT_DOWNLOAD + $(2)_TRANSIENT_DOWNLOAD = $$($(3)_TRANSIENT_DOWNLOAD) + else + $(2)_TRANSIENT_DOWNLOAD = NO + endif +endif + +ifeq ($$($(2)_TRANSIENT_DOWNLOAD),YES) +BR_NO_CHECK_HASH_FOR += $$($(2)_SOURCE) +endif + ifneq ($$(filter bzr cvs hg,$$($(2)_SITE_METHOD)),) BR_NO_CHECK_HASH_FOR += $$($(2)_SOURCE) endif
support/download/dl-wrapper +5 -3
@@ -21,11 +21,12 @@ export BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e" main() { local OPT OPTARG - local backend output hfile recurse quiet rc + local backend output hfile recurse quiet rc force local -a uris # Parse our options; anything after '--' is for the backend - while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do + force=false + while getopts ":c:d:D:o:n:N:H:rf:u:qF" OPT; do case "${OPT}" in c) cset="${OPTARG}";; d) dl_dir="${OPTARG}";;
@@ -38,6 +39,7 @@ main() { f) filename="${OPTARG}";; u) uris+=( "${OPTARG}" );; q) quiet="-q";; + F) force=true;; :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";; \?) error "unknown option '%s'\n" "${OPTARG}";; esac
@@ -67,7 +69,7 @@ main() { # - matches all its hashes: do not download it again and exit promptly # - fails at least one of its hashes: force a re-download # - there's no hash (but a .hash file): consider it a hard error - if [ -e "${output}" ]; then + if ! ${force} && [ -e "${output}" ]; then if support/download/check-hash ${quiet} "${hfile}" "${output}" "${output##*/}"; then exit 0 elif [ ${?} -ne 2 ]; then
Signature · 1 line
2.25.1
PATCH v2 1/1 infra: add the transient download mechanism
Hello,
Quoted 7 lines · On Wed, 30 Sep 2020 19:30:30 +0200 "Yann E. MORIN" <[email protected]>…
People have been hollering for a long time, asking for a way to be able to trigger a build that will always get the current HEAD (or whatever it is called outside of git) of a branch, whatever the conditions, so that they can trigger automated builds (e.g. in a CI or whatever) to test their greatest and latest changes, by pushing again and again ad libitum on the same branch, without having the need to update the sha1 in the .mk of the affected package.
[...]
I'm all happy with the implementation, but I still disagree on one bit, which is present in both the commit log and the documentation, so I'll comment on the documentation.
docs/manual/adding-packages-generic.txt not and can not work as people would expect it should: · line 359
Quoted 6 lines
docs/manual/adding-packages-generic.txt +6
-0+ obvious being the *lack of reproducibility*. Besides, it has its own + pitfalls that one must be aware of as well: + ** when two builds are running in parallel on the same machine, there + is a TOCTOU race for each build to download and extract the archive, + so each build may not get what it downloaded, but what the other + build did download;
This is not a pitfall, it is what we expect from the feature. The feature is "grab" whatever is currently available on the Git repository for that package. It is pretty clear that it is not guarantee that we grab what existed in the Git repository at the moment the user hits "Enter" after typing the make command. There is no way we could atomically snapshot all Git trees of all packages to ensure that what we build is a consistent view of what existed at the moment the build was started.
So I think this pitfall is needlessly worrying, and does not actually point to an actual problem. Or more precisely, the problem you're pointing is *exactly* what the transient download mechanism aims at doing.
So to me, the wording of your pitfall is a bit as if the wget(1) man page had an entry like this:
BUGS
WARNING: wget might download data from remote servers to your local machine!!
docs/manual/adding-packages-generic.txt not and can not work as people would expect it should: · line 361
Quoted 2 lines
docs/manual/adding-packages-generic.txt +2
-0+ ** as a consequence, legal-info is not reliable when a package is + transient;
*That* I believe is the real pitfall.
docs/manual/adding-packages-generic.txt not and can not work as people would expect it should: · line 364
Quoted 3 lines
docs/manual/adding-packages-generic.txt +3
-0+ ** if a branch is pushed to an automated build bot (in a CI for + example), and then re-pushed to while the build is still in + progress, the build may get the wrong version of the branch.
This is also *NOT* a pitfall, it is what we ask the transient download mechanism to do.
Once this is fixed, I'll be happy to apply the patch!
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH v2 1/1 infra: add the transient download mechanism
Thomas, All,
On 2020-09-30 21:45 +0200, Thomas Petazzoni spake thusly:
Quoted 2 lines
On Wed, 30 Sep 2020 19:30:30 +0200
"Yann E. MORIN" <[email protected]> wrote:
[--SNIP--]
Quoted 16 lines
docs/manual/adding-packages-generic.txt not and can not work as people would expect it should: · line 359
Quoted 6 lines
docs/manual/adding-packages-generic.txt +6
-0+ obvious being the *lack of reproducibility*. Besides, it has its own + pitfalls that one must be aware of as well: + ** when two builds are running in parallel on the same machine, there + is a TOCTOU race for each build to download and extract the archive, + so each build may not get what it downloaded, but what the other + build did download;This is not a pitfall, it is what we expect from the feature. The feature is "grab" whatever is currently available on the Git repository for that package. It is pretty clear that it is not guarantee that we grab what existed in the Git repository at the moment the user hits "Enter" after typing the make command. There is no way we could atomically snapshot all Git trees of all packages to ensure that what we build is a consistent view of what existed at the moment the build was started.
Quoted 4 lines
So I think this pitfall is needlessly worrying, and does not actually point to an actual problem. Or more precisely, the problem you're pointing is *exactly* what the transient download mechanism aims at doing.
I think it is important to stress that, though, because people can be very confused but the outcome, one way or the other.
For example, let's assume that I push my branch, that triggers a pipeline in the CI, the pipeline uses my branch as expected. Then I repush a modified branch, the new pipeline uses the new HEAD of the branch, and again this is as expected..
But now consider the following scenario:
- I push my branch, a pipeline is triggered;
- before the pipeline has a chance to git-fetch, I do a quick change
and repush, a second pipeline is triggered;
- the first pipeline uses the code from the second push.
However, my change was intentional, and I was expecting the CI to test the two changes, and I wanted to have the results of both to decide which to keep.
This for example, is very interesting because the CI can do tests I can't always do locally, like reproducibility test for performance, for example, or test on hardware I don't have at hand (when working from home for example), and so on.
Thus my case is borked. Yeah, I could do two branches, but the fact is that if I wait "just enough" between the pushes, it works, while if I don't wait "just enough" it does not work.
Conversely, if I notice that I forgot to commit part of the changes, and amend my commit and repush, I don't care what the first pipeline gets; I only care about the second.
So, this is a pitfall: behaviours that may or may not match your expectations, depending on what your expectations are, because this mechanism caters to both antagonist cases where expectations are opposite one from the other.
[--SNIP--]
Quoted 3 lines
docs/manual/adding-packages-generic.txt not and can not work as people would expect it should: · line 361
Quoted 2 lines
docs/manual/adding-packages-generic.txt +2
-0+ ** as a consequence, legal-info is not reliable when a package is + transient;*That* I believe is the real pitfall.
Not, it is a consequence of the pitfall.
Quoted 8 lines
docs/manual/adding-packages-generic.txt not and can not work as people would expect it should: · line 364
Quoted 3 lines
docs/manual/adding-packages-generic.txt +3
-0+ ** if a branch is pushed to an automated build bot (in a CI for + example), and then re-pushed to while the build is still in + progress, the build may get the wrong version of the branch.This is also *NOT* a pitfall, it is what we ask the transient download mechanism to do.
Once this is fixed, I'll be happy to apply the patch!
Guess we'll have to refer to higher-level hierarchy to take a decision. ;-)
Thanks for the comments. :-)
Regards,
Yann E. MORIN.
Quoted 7 lines
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v2 1/1 infra: add the transient download mechanism
Quoted 28 lines · On Wed, 30 Sep 2020 22:13:03 +0200 "Yann E. MORIN" <[email protected]>…
I think it is important to stress that, though, because people can be very confused but the outcome, one way or the other.
For example, let's assume that I push my branch, that triggers a pipeline in the CI, the pipeline uses my branch as expected. Then I repush a modified branch, the new pipeline uses the new HEAD of the branch, and again this is as expected..
But now consider the following scenario:
- I push my branch, a pipeline is triggered;
- before the pipeline has a chance to git-fetch, I do a quick changeand repush, a second pipeline is triggered;- the first pipeline uses the code from the second push.
However, my change was intentional, and I was expecting the CI to test the two changes, and I wanted to have the results of both to decide which to keep.
This for example, is very interesting because the CI can do tests I can't always do locally, like reproducibility test for performance, for example, or test on hardware I don't have at hand (when working from home for example), and so on.
Thus my case is borked. Yeah, I could do two branches, but the fact is that if I wait "just enough" between the pushes, it works, while if I don't wait "just enough" it does not work.
Quoted 8 lines
Conversely, if I notice that I forgot to commit part of the changes, and amend my commit and repush, I don't care what the first pipeline gets; I only care about the second.
So, this is a pitfall: behaviours that may or may not match your expectations, depending on what your expectations are, because this mechanism caters to both antagonist cases where expectations are opposite one from the other.
I understand what you're describing here, and I understand that it indeed should be explained. Then, it shouldn't be presented as a "pitfall", but really as an explanation of the consequences.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH v2 1/1 infra: add the transient download mechanism
Thomas, All,
On 2020-09-30 22:31 +0200, Thomas Petazzoni spake thusly:
Quoted 4 lines
Quoted 2 lines · On Wed, 30 Sep 2020 22:13:03 +0200 "Yann E. MORIN" <[email protected]>…
I think it is important to stress that, though, because people can be very confused but the outcome, one way or the other.
[--SNIP--]
Quoted 3 lines
I understand what you're describing here, and I understand that it indeed should be explained. Then, it shouldn't be presented as a "pitfall", but really as an explanation of the consequences.
So I thought again about this, and I stand by my position.
I do not want to play down the issues attached to that (mis)feature, and I still think this is a pitfall. So, I am not going to change the wording.
Feel free to grab the patch and adapt if you feel strongly about it.
Thanks! :-)
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v2 1/1 infra: add the transient download mechanism
In reply to Yann E. MORINQuoted 30 lines · Yann E. MORIN
Thomas, All,
On 2020-09-30 21:45 +0200, Thomas Petazzoni spake thusly:
Quoted 2 lines
On Wed, 30 Sep 2020 19:30:30 +0200
"Yann E. MORIN" <[email protected]> wrote:[--SNIP--]
Quoted 16 lines
docs/manual/adding-packages-generic.txt not and can not work as people would expect it should: · line 359
Quoted 6 lines
docs/manual/adding-packages-generic.txt +6
-0+ obvious being the *lack of reproducibility*. Besides, it has its own + pitfalls that one must be aware of as well: + ** when two builds are running in parallel on the same machine, there + is a TOCTOU race for each build to download and extract the archive, + so each build may not get what it downloaded, but what the other + build did download;This is not a pitfall, it is what we expect from the feature. The feature is "grab" whatever is currently available on the Git repository for that package. It is pretty clear that it is not guarantee that we grab what existed in the Git repository at the moment the user hits "Enter" after typing the make command. There is no way we could atomically snapshot all Git trees of all packages to ensure that what we build is a consistent view of what existed at the moment the build was started.
Quoted 4 lines
So I think this pitfall is needlessly worrying, and does not actually point to an actual problem. Or more precisely, the problem you're pointing is *exactly* what the transient download mechanism aims at doing.
I think it is important to stress that, though, because people can be very confused but the outcome, one way or the other.
Quoted 8 lines
For example, let's assume that I push my branch, that triggers a pipeline in the CI, the pipeline uses my branch as expected. Then I repush a modified branch, the new pipeline uses the new HEAD of the branch, and again this is as expected..
But now consider the following scenario:
- I push my branch, a pipeline is triggered;
Here your reasoning is wrong: this feature should *NOT* be used for triggering
a build on push to some repo. For that use case, OVERRIDE_SRCDIR is much more suitable, because that *will* use the exact sources that correspond to the trigger.
What this feature is meant for is to do actual continuous integration, i.e.
people are developing code in different repos, and a regularly triggered (or continuous) build brings all these repos together. It sounds insane to do something like that, but it is the way that some projects work [1].
Regards,
Arnout
[1] https://abseil.io/about/philosophy#we-recommend-that-you-choose-to-live-at-head
Quoted 57 lines
- before the pipeline has a chance to git-fetch, I do a quick change
and repush, a second pipeline is triggered;- the first pipeline uses the code from the second push.
However, my change was intentional, and I was expecting the CI to test the two changes, and I wanted to have the results of both to decide which to keep.
This for example, is very interesting because the CI can do tests I can't always do locally, like reproducibility test for performance, for example, or test on hardware I don't have at hand (when working from home for example), and so on.
Thus my case is borked. Yeah, I could do two branches, but the fact is that if I wait "just enough" between the pushes, it works, while if I don't wait "just enough" it does not work.
Conversely, if I notice that I forgot to commit part of the changes, and amend my commit and repush, I don't care what the first pipeline gets; I only care about the second.
So, this is a pitfall: behaviours that may or may not match your expectations, depending on what your expectations are, because this mechanism caters to both antagonist cases where expectations are opposite one from the other.
[--SNIP--]
Quoted 3 lines
docs/manual/adding-packages-generic.txt not and can not work as people would expect it should: · line 361
Quoted 2 lines
docs/manual/adding-packages-generic.txt +2
-0+ ** as a consequence, legal-info is not reliable when a package is + transient;*That* I believe is the real pitfall.
Not, it is a consequence of the pitfall.
Quoted 8 lines
docs/manual/adding-packages-generic.txt not and can not work as people would expect it should: · line 364
Quoted 3 lines
docs/manual/adding-packages-generic.txt +3
-0+ ** if a branch is pushed to an automated build bot (in a CI for + example), and then re-pushed to while the build is still in + progress, the build may get the wrong version of the branch.This is also *NOT* a pitfall, it is what we ask the transient download mechanism to do.
Once this is fixed, I'll be happy to apply the patch!
Guess we'll have to refer to higher-level hierarchy to take a decision. ;-)
Thanks for the comments. :-)
Regards,
Yann E. MORIN.Quoted 7 lines
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH v2 1/1 infra: add the transient download mechanism
Arnout, All,
On 2020-10-06 20:43 +0200, Arnout Vandecappelle spake thusly:
Quoted 11 lines · Yann E. MORIN
Quoted 10 lines
On 2020-09-30 21:45 +0200, Thomas Petazzoni spake thusly:
Quoted 2 lines
On Wed, 30 Sep 2020 19:30:30 +0200
"Yann E. MORIN" <[email protected]> wrote:[--SNIP--]
Quoted 6 lines
Quoted 6 lines
docs/manual/adding-packages-generic.txt +6
-0+ obvious being the *lack of reproducibility*. Besides, it has its own + pitfalls that one must be aware of as well: + ** when two builds are running in parallel on the same machine, there + is a TOCTOU race for each build to download and extract the archive, + so each build may not get what it downloaded, but what the other + build did download;
[--SNIP--]
Quoted 5 lines
Quoted 2 lines
But now consider the following scenario:
- I push my branch, a pipeline is triggered;Here your reasoning is wrong: this feature should *NOT* be used for triggering
a build on push to some repo. For that use case,OVERRIDE_SRCDIRis much more suitable, because that *will* use the exact sources that correspond to the trigger.
But OVERRIDE_SRCDIR can *also* be used for the CI part, so why are we even looking at this transient thing, then?
When transient is available, people are going to use it, even for cases it was been designed for. And the use-case I describe is really one such (un)expected scenario transient is made for.
Quoted 4 lines
What this feature is meant for is to do actual continuous integration, i.e.
people are developing code in different repos, and a regularly triggered (or continuous) build brings all these repos together. It sounds insane to do something like that, but it is the way that some projects work [1].
Insane what people can come up with once they have some feature at hand! ;-)
But I am no longer interested in pushing this (mis)feature anyway. Remember I am initially opposed to it to begin with...
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'