Skip to contentSkip to conversation

ignore UTF-8 byte order mark

PATCH Superseded Heiko Hund 2 participants 9 messages Jan 22, 2013 – Feb 6, 2013

lore
Share & tools
lore

Heiko Hund

author
#0 lore raw

PATCH 1/1 ignore UTF-8 byte order mark

Currently openvpn exits when reading configuration from a file that has an UTF-8 byte order mark (EF BB BF) at its start.

While it is useless to put a BOM into UTF-8 encoded files, the Unicode standard [1] permits it (on page 36):

"Use of a BOM is neither required nor recommended for UTF-8, but may be encountered in contexts where UTF-8 data is converted from other encoding forms that use a BOM or where the BOM is used as a UTF-8 signature."

Notepad.exe, the default text editor on Windows, always puts the BOM into UTF-8 encoded files when saving them. Others may do as well.

Just ignoring the UTF-8 BOM will make config files with UTF-8 BOM readable.

[1] http://www.unicode.org/versions/Unicode5.0.0/ch02.pdf

---
 src/openvpn/options.c |    3 +++
 1 file changed, 3 insertions(+)
src/openvpn/options.c +3 -0
index 99f70ad..46f6f26 100644
@@ -3773,6 +3773,9 @@ read_config_file (struct options *options,
 	    {
 	      CLEAR (p);
 	      ++line_num;
+              /* Ignore UTF-8 BOM at start of stream */
+              if (line_num == 1 && strncmp (line, "\xEF\xBB\xBF", 3) == 0)
+                memmove (line, line + 3, strlen (line) - 2);
 	      if (parse_line (line, p, SIZE (p), file, line_num, msglevel, &options->gc))
 		{
 		  bypass_doubledash (&p[0]);
Signature · 1 line
tg: (77fb72e..) t/0005/ignore_utf8_bom (depends on: t/0004/utf8_compatibility)
#1 lore raw
Quoted 30 lines · Heiko Hund

Currently openvpn exits when reading configuration from a file that has an UTF-8 byte order mark (EF BB BF) at its start.

While it is useless to put a BOM into UTF-8 encoded files, the Unicode standard [1] permits it (on page 36):

"Use of a BOM is neither required nor recommended for UTF-8, but may be encountered in contexts where UTF-8 data is converted from other encoding forms that use a BOM or where the BOM is used as a UTF-8 signature."

Notepad.exe, the default text editor on Windows, always puts the BOM into UTF-8 encoded files when saving them. Others may do as well.

Just ignoring the UTF-8 BOM will make config files with UTF-8 BOM readable.

[1] http://www.unicode.org/versions/Unicode5.0.0/ch02.pdf

---
 src/openvpn/options.c |    3 +++
 1 file changed, 3 insertions(+)
src/openvpn/options.c +0 -0
index 99f70ad..46f6f26 100644

src/openvpn/options.c read_config_file (struct options *options, · line 3779

Quoted 7 lines
src/openvpn/options.c +3 -0
@@ -3773,6 +3773,9 @@ read_config_file (struct options *options,
 	    {
 	      CLEAR (p);
 	      ++line_num;
+              /* Ignore UTF-8 BOM at start of stream */
+              if (line_num == 1 && strncmp (line, "\xEF\xBB\xBF", 3) == 0)
+                memmove (line, line + 3, strlen (line) - 2);
 	      if (parse_line (line, p, SIZE (p), file, line_num, msglevel, &options->gc))

Wouldn't this approach solve it in an easier way?

int offset = 0;
if (line_num == 1 && strncmp (line, "\xEF\xBB\xBF", 3) == 0)
offset = 3

if (parse_line (line+offset, p, SIZE (p), file, line_num ....

Should be safer as you don't need care about proper NULL termination of the buffer.

Signature · 3 lines
kind regards,

David Sommerseth
author
#2 lore raw
Quoted 10 lines · On Saturday 02 February 2013 17:47:40 David Sommerseth wrote:

Wouldn't this approach solve it in an easier way?

int offset = 0;
if (line_num == 1 && strncmp (line, "\xEF\xBB\xBF", 3) == 0)
offset = 3

if (parse_line (line+offset, p, SIZE (p), file, line_num ....

Should be safer as you don't need care about proper NULL termination of the buffer.

Although fgets(3) is supposed to always \0 terminate the buffer I like your approach better, as it's more lightweight. Will send an updated version.

Heiko

author
#3 lore raw

PATCH v2 1/1 ignore UTF-8 byte order mark

In reply to Heiko Hund

Currently openvpn exits when reading configuration from a file that has an UTF-8 byte order mark (EF BB BF) at its start.

While it is useless to put a BOM into UTF-8 encoded files, the Unicode standard [1] permits it (on page 36):

"Use of a BOM is neither required nor recommended for UTF-8, but may be encountered in contexts where UTF-8 data is converted from other encoding forms that use a BOM or where the BOM is used as a UTF-8 signature."

Notepad.exe, the default text editor on Windows, always puts the BOM into UTF-8 encoded files when saving them. Others may do as well.

Just ignoring the UTF-8 BOM will make config files with UTF-8 BOM readable.

[1] http://www.unicode.org/versions/Unicode5.0.0/ch02.pdf

---
 src/openvpn/options.c |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
src/openvpn/options.c +5 -1
index 8ca41a3..c52b699 100644
@@ -3773,7 +3773,11 @@ read_config_file (struct options *options,
 	    {
 	      CLEAR (p);
 	      ++line_num;
-	      if (parse_line (line, p, SIZE (p), file, line_num, msglevel, &options->gc))
+              /* Ignore UTF-8 BOM at start of stream */
+              int offset = 0;
+              if (line_num == 1 && strncmp (line, "\xEF\xBB\xBF", 3) == 0)
+                offset = 3;
+	      if (parse_line (line + offset, p, SIZE (p), file, line_num, msglevel, &options->gc))
 		{
 		  bypass_doubledash (&p[0]);
 		  check_inline_file_via_fp (fp, p, &options->gc);
Signature · 1 line
1.7.9.5
#4 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 25 lines · Heiko Hund

Currently openvpn exits when reading configuration from a file that has an UTF-8 byte order mark (EF BB BF) at its start.

While it is useless to put a BOM into UTF-8 encoded files, the Unicode standard [1] permits it (on page 36):

"Use of a BOM is neither required nor recommended for UTF-8, but may be encountered in contexts where UTF-8 data is converted from other encoding forms that use a BOM or where the BOM is used as a UTF-8 signature."

Notepad.exe, the default text editor on Windows, always puts the BOM into UTF-8 encoded files when saving them. Others may do as well.

Just ignoring the UTF-8 BOM will make config files with UTF-8 BOM readable.

[1] http://www.unicode.org/versions/Unicode5.0.0/ch02.pdf

src/openvpn/options.c | 6 +++++- 1 file changed, 5
insertions(+), 1 deletion(-)

src/openvpn/options.c index renamed from src/openvpn/options.c +0 -0
Quoted 6 lines

8ca41a3..c52b699 100644 --- a/src/openvpn/options.c +++ b/src/openvpn/options.c @@ -3773,7 +3773,11 @@ read_config_file
(struct options *options, { CLEAR (p); ++line_num; - if
(parse_line (line, p, SIZE (p), file, line_num, msglevel,
&options->gc)) + /* Ignore UTF-8 BOM at start of
stream */ + int offset = 0;

Some compilers will complain with int being declared here. Moving it up to before the CLEAR (p); line is probably going to work better ...

- --
kind regards,

David Sommerseth
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAlEOd/EACgkQDC186MBRfrr9+wCghuv++EJkTlCAb3p0YoLzSkGv b1YAn3PKYmvfaIZzaYsu3NAXl6m2Dutz
=Fyps
-----END PGP SIGNATURE-----

#5 lore raw
In reply to Heiko Hund
Quoted 29 lines · Heiko Hund

Currently openvpn exits when reading configuration from a file that has an UTF-8 byte order mark (EF BB BF) at its start.

While it is useless to put a BOM into UTF-8 encoded files, the Unicode standard [1] permits it (on page 36):

"Use of a BOM is neither required nor recommended for UTF-8, but may be encountered in contexts where UTF-8 data is converted from other encoding forms that use a BOM or where the BOM is used as a UTF-8 signature."

Notepad.exe, the default text editor on Windows, always puts the BOM into UTF-8 encoded files when saving them. Others may do as well.

Just ignoring the UTF-8 BOM will make config files with UTF-8 BOM readable.

[1] http://www.unicode.org/versions/Unicode5.0.0/ch02.pdf

---
 src/openvpn/options.c |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
src/openvpn/options.c +0 -0
index 8ca41a3..c52b699 100644

src/openvpn/options.c read_config_file (struct options *options, · line 3779

Quoted 8 lines
src/openvpn/options.c +4 -1
@@ -3773,7 +3773,11 @@ read_config_file (struct options *options,
 	    {
 	      CLEAR (p);
 	      ++line_num;
-	      if (parse_line (line, p, SIZE (p), file, line_num, msglevel, &options->gc))
+              /* Ignore UTF-8 BOM at start of stream */
+              int offset = 0;
+              if (line_num == 1 && strncmp (line, "\xEF\xBB\xBF", 3) == 0)
+                offset = 3;

Lets try again without word-wrap ....

Some compilers will complain with int offset being declared "down" here. Moving it up to before the CLEAR (p); line is probably going to work better ...

Signature · 3 lines
kind regards,

David Sommerseth
author
#6 lore raw

PATCH v3 1/1 ignore UTF-8 byte order mark

In reply to Heiko Hund

Currently openvpn exits when reading configuration from a file that has an UTF-8 byte order mark (EF BB BF) at its start.

While it is useless to put a BOM into UTF-8 encoded files, the Unicode standard [1] permits it (on page 36):

"Use of a BOM is neither required nor recommended for UTF-8, but may be encountered in contexts where UTF-8 data is converted from other encoding forms that use a BOM or where the BOM is used as a UTF-8 signature."

Notepad.exe, the default text editor on Windows, always puts the BOM into UTF-8 encoded files when saving them. Others may do as well.

Just ignoring the UTF-8 BOM will make config files with UTF-8 BOM readable.

[1] http://www.unicode.org/versions/Unicode5.0.0/ch02.pdf

---
 src/openvpn/options.c |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
src/openvpn/options.c +5 -1
index 8ca41a3..ac05a29 100644
@@ -3771,9 +3771,13 @@ read_config_file (struct options *options,
 	  line_num = 0;
 	  while (fgets(line, sizeof (line), fp))
 	    {
+              int offset = 0;
 	      CLEAR (p);
 	      ++line_num;
-	      if (parse_line (line, p, SIZE (p), file, line_num, msglevel, &options->gc))
+              /* Ignore UTF-8 BOM at start of stream */
+              if (line_num == 1 && strncmp (line, "\xEF\xBB\xBF", 3) == 0)
+                offset = 3;
+	      if (parse_line (line + offset, p, SIZE (p), file, line_num, msglevel, &options->gc))
 		{
 		  bypass_doubledash (&p[0]);
 		  check_inline_file_via_fp (fp, p, &options->gc);
Signature · 1 line
1.7.10.4
#7 lore raw

Your patch has been applied to the master branch.

commit 6e6f55f4ba5deda5649679a13e4e323e07b3e661

Author: Heiko Hund
Date:   Mon Feb 4 11:39:25 2013 +0000
     Ignore UTF-8 byte order mark

     Signed-off-by: Heiko Hund <[email protected]...>
     Acked-by: David Sommerseth <[email protected]...>
     Message-Id: [email protected]...
     URL: http://article.gmane.org/gmane.network.openvpn.devel/7342
     Signed-off-by: David Sommerseth <[email protected]...>

--
kind regards,

David Sommerseth

#8 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Forgot to add this was also applied to release/2.3 commit a8fe779b04cc5bf698a1bf5bed4b0189126c3a38

Quoted 5 lines · David Sommerseth

Your patch has been applied to the master branch.

commit 6e6f55f4ba5deda5649679a13e4e323e07b3e661 Author: Heiko Hund
Date: Mon Feb 4 11:39:25 2013 +0000

Quoted 8 lines

Ignore UTF-8 byte order mark

Sommerseth <[email protected]...> Message-Id:
[email protected]... URL:
http://article.gmane.org/gmane.network.openvpn.devel/7342

- --
kind regards,

David Sommerseth
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAlESqzAACgkQDC186MBRfrrKygCfTqg7qDJDo/f0rlKln8Qi4bbJ 2yIAoJeuU9Wt2GSrKGDHgbi0FFIZ2TZQ
=PUer
-----END PGP SIGNATURE-----