New: mkusers script bash errors

https://bugs.busybox.net/show_bug.cgi?id=9456

Bug ID: 9456
Summary: mkusers script bash errors
Product: buildroot
Version: unspecified
Hardware: All
OS: Linux
Status: NEW
Severity: normal
Priority: P5
Component: Other
Assignee: unassigned at buildroot.uclibc.org
Reporter: georgebrmz at oss3d.com
CC: buildroot at uclibc.org
Target Milestone: ---

This happens with the latest git as of 2016/11/28:

/buildroot/support/scripts/mkusers: line 417: [: -ge: unary operator expected /buildroot/support/scripts/mkusers: line 426: [: -eq: unary operator expected

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #1 from Peter Korsgaard <[email protected]> --- (In reply to George Y. from comment #0)

What is your .config? Have you added any custom packages? It sounds like wrong values are passed to mkusers

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #2 from George Y. <[email protected]> --- Created attachment 6811
--> https://bugs.busybox.net/attachment.cgi?id=6811&action=edit
Buildroot config

The config is attached. Nothing specifically different.

By the way, the build process does not stop. It continues and ignores those errors, so if it happens for you as well, you'll only see it from the messages, but the building will continue and will give an impression of success.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #3 from Arnout Vandecappelle <[email protected]> --- Paul__ on IRC reported the same issue. I spent some time trying to debug it, but I wasn't able to reproduce with the same users settings.

The only thing that changed in that area is pseudo, as far as I can see. However, on 2016/11/28, pseudo was already reverted. That said, the .config file that is attached to this bug says it is from 2016.11-rc2 so I'm not entirely sure if the revert is already included. Please try again with 2016.11-rc3 - if it does work there, then we have identified the culprit.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #4 from George Y. <[email protected]> ---
Please disregard the version number in the config file, I'm using the same config for a while without letting buildroot upgrade it, and I'm always doing a clean git clone for the master branch. I'm also on OpenSuSE distribution, if that matters.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #5 from Thomas Petazzoni <[email protected]> --- George, do you still see the issue with the latest Buildroot 2017.02-rc1 ?

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

Thomas Petazzoni <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

             Status|NEW                         |RESOLVED
         Resolution|---                         |NEEDINFO

--- Comment #6 from Thomas Petazzoni <[email protected]> --- No feedback from George since end of November, so I'll close the bug as NEEDINFO. George, if you still see the problem with the latest Buildroot version, do not hesitate to reopen this bug with more details.

Thanks!

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #7 from George Y. <[email protected]> ---
We are not making buildroot builds too often. I'll attempt a build soon, and if it still happens, I'll reopen the bug. Thanks!

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #8 from paulstewartis at gmail.com <[email protected]> --- (In reply to Thomas Petazzoni from comment #5)

Hello is Paul__ from the irc channel.

The issue is not resolved for me with 2017.02-rc3. It is still reporting the errors. Because the problem occurs with the first line, my workaround is to add a dummy user before the real users. My user table looks like this:

dummy -1 users -1 * - - - Dummy user for mkusers script bug paul 1000 users 100 =mypassword /home/paul /bin/sh ssh-user git 1001 users 100 =gitpassword /home/git /usr/bin/git-shell ssh-user

If you remove the first line, it will not create the paul user. The passwd file will include the standard users followed by my git user and sshd (because the openssh package is included in my buildroot .config).

This is what it looks like if you omit the dummy line above:

root:x:0:0:root:/root:/bin/sh
daemon:x:1:1:daemon:/usr/sbin:/bin/false
bin:x:2:2:bin:/bin:/bin/false
sys:x:3:3:sys:/dev:/bin/false
sync:x:4:100:sync:/bin:/bin/sync
mail:x:8:8:mail:/var/spool/mail:/bin/false
www-data:x:33:33:www-data:/var/www:/bin/false
operator:x:37:37:Operator:/var:/bin/false
nobody:x:65534:65534:nobody:/home:/bin/false
git:x:1001:100::/home/git:/usr/bin/git-shell
sshd:x:1000:1000:SSH drop priv user:/:/bin/false

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

Yann E. MORIN <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

                 CC|                            |yann.morin.1998 at free.fr
             Status|RESOLVED                    |REOPENED
         Resolution|NEEDINFO                    |---

--- Comment #9 from Yann E. MORIN <[email protected]> --- (In reply to paulstewartis at gmail.com from comment #8)

OK, let's see what I can do with that users table. Thanks for the use-case.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #10 from Yann E. MORIN <[email protected]> --- (In reply to paulstewartis at gmail.com from comment #8)

Still no luck in trying to reproduce the issue with your users table (minus the dummy user) and the following defconfig:

BR2_arm=y
BR2_cortex_a7=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_INIT_NONE=y
BR2_SYSTEM_BIN_SH_NONE=y
BR2_ROOTFS_USERS_TABLES="/home/ymorin/users.list"
# BR2_PACKAGE_BUSYBOX is not set

Here is the /etc/passwd the gets me:

root:x:0:0:root:/root:/bin/sh
daemon:x:1:1:daemon:/usr/sbin:/bin/false
bin:x:2:2:bin:/bin:/bin/false
sys:x:3:3:sys:/dev:/bin/false
sync:x:4:100:sync:/bin:/bin/sync
mail:x:8:8:mail:/var/spool/mail:/bin/false
www-data:x:33:33:www-data:/var/www:/bin/false
operator:x:37:37:Operator:/var:/bin/false
nobody:x:65534:65534:nobody:/home:/bin/false
paul:x:1000:100::/home/paul:/bin/sh
git:x:1001:100::/home/git:/usr/bin/git-shell

So as you can see, both the paul and git users are there.

What is your bash version:

/usr/bin/env bash --version

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #11 from paulstewartis at gmail.com <[email protected]> --- (In reply to Yann E. MORIN from comment #10)

paul at linux:~> /usr/bin/env bash --version
GNU bash, version 4.3.42(1)-release (x86_64-suse-linux-gnu) Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. paul at linux:~>

I can post my .config if you want too.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #12 from Yann E. MORIN <[email protected]> --- (In reply to paulstewartis at gmail.com from comment #11)

OK, bash is not too old ( I have 4.3.46).

Wild guess: your users table file does not have CR-LF endings, does it?

I have absolutely no clue what's going on... :-/

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #13 from paulstewartis at gmail.com <[email protected]> --- (In reply to Yann E. MORIN from comment #12)

"Wild guess: your users table file does not have CR-LF endings, does it?"

LF only. I tried both by the way. No difference either way.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #14 from Arnout Vandecappelle <[email protected]> --- Perhaps there is some mojibake in the users file? Can you attach it as binary?

The bash errors make no sense at all, because exactly the same read commands and LINES array is used a bit higher up in the mkusers script. Except of course if one of the spaces is not a real space. Since mkusers is executed with LANG=C, it will most likely not split on any of the > U+007F whitespace, and you end up with e.g. uid = 1000\xc2\xa0users.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #15 from Paul Stewart <[email protected]> --- Created attachment 6906
--> https://bugs.busybox.net/attachment.cgi?id=6906&action=edit
User table for paul buildroot config

Here is the actual file. The only difference from what I pasted before is the passwords. I have tried different, simple, short passwords and it did not make a difference. I will change these passwords, of course, now that they are posted on the internet. :-)

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #16 from Paul Stewart <[email protected]> --- (In reply to Arnout Vandecappelle from comment #14)

Looking at the Hexdump, it looks like 0x20 for the spaces:

00000000  64 75 6d 6d 79 20 2d 31  20 75 73 65 72 73 20 2d  |dummy -1 users -|
00000010  31 20 2a 20 2d 20 2d 20  2d 20 44 75 6d 6d 79 20  |1 * - - - Dummy |
<snip>
00000110  72 2f 62 69 6e 2f 67 69  74 2d 73 68 65 6c 6c 20  |r/bin/git-shell |
00000120  73 73 68 2d 75 73 65 72  0a                       |ssh-user.|

00000129

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #17 from Yann E. MORIN <[email protected]> --- (In reply to Paul Stewart from comment #15)

So I downloaded that file and uses it:

- as-is, and it creates the three users: dummy, paul and git;
- without the dummy user, and it creates paul and git.

I'll try to instrument the script and attach a patch so that you can report the output. Give me a moment...

Regards,
Yann E. MORIN.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #18 from Yann E. MORIN <[email protected]> --- Created attachment 6911
--> https://bugs.busybox.net/attachment.cgi?id=6911&action=edit
Debug mkusers

Paul,

Could you test with the attched patch and report the result, please?

Here is what I got:

line='paul 1000 users 100 =21248f7aac088184a7c9dd3a3cd8bd1bafecc5718174e9025bab7053756a013a /home/paul /bin/sh ssh-user'
username='paul'

  uid     ='1000'
  group   ='users'
  gid     ='100'
  paswd   ='=21248f7aac088184a7c9dd3a3cd8bd1bafecc5718174e9025bab7053756a013a'
  home    ='/home/paul'
  shell   ='/bin/sh'
  groups  ='ssh-user'

comment =''
line='git 1001 users 100 =de5d4f76680ec42ff304682eebeaec1c4756e3d5de3e4ce6db747cb223d6037c /home/git /usr/bin/git-shell ssh-user'
username='git'

  uid     ='1001'
  group   ='users'
  gid     ='100'
  paswd   ='=de5d4f76680ec42ff304682eebeaec1c4756e3d5de3e4ce6db747cb223d6037c'
  home    ='/home/git'
  shell   ='/usr/bin/git-shell'
  groups  ='ssh-user'

comment =''

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #19 from Paul Stewart <[email protected]> --- (In reply to Yann E. MORIN from comment #18)

line='paul 1000 users 100 =21248f7aac088184a7c9dd3a3cd8bd1bafecc5718174e9025bab7053756a013a /home/paul /bin/sh ssh-user'
username='paul'

  uid     ='1000'
  group   ='users'
  gid     ='100'
  paswd   ='=21248f7aac088184a7c9dd3a3cd8bd1bafecc5718174e9025bab7053756a013a'
  home    ='/home/paul'
  shell   ='/bin/sh'
  groups  ='ssh-user'

comment =''
line='git 1001 users 100 =de5d4f76680ec42ff304682eebeaec1c4756e3d5de3e4ce6db747cb223d6037c /home/git /usr/bin/git-shell ssh-user'
username='git'

  uid     ='1001'
  group   ='users'
  gid     ='100'
  paswd   ='=de5d4f76680ec42ff304682eebeaec1c4756e3d5de3e4ce6db747cb223d6037c'
  home    ='/home/git'
  shell   ='/usr/bin/git-shell'
  groups  ='ssh-user'
  comment =''
line='sshd -1 sshd -1 * - - - SSH drop priv user'
  username='sshd'
  uid     ='-1'
  group   ='sshd'
  gid     ='-1'
  paswd   ='*'
  home    ='-'
  shell   ='-'
  groups  ='-'

comment ='SSH drop priv user'

This is immediately followed by:
/home/paul/src/buildroot/support/scripts/mkusers: line 407: [: -eq: unary operator expected /home/paul/src/buildroot/support/scripts/mkusers: line 432: [: -ge: unary operator expected /home/paul/src/buildroot/support/scripts/mkusers: line 441: [: -eq: unary operator expected

And the paul user is not created.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #20 from Paul Stewart <[email protected]> --- I created a default rpi 2 config and set it to use my user table. And I get the same issue. So it isn't my .config file neither.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

Yann E. MORIN <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

   Attachment #6911|0                           |1
        is obsolete|                            |
           Assignee|unassigned at buildroot.uclibc |yann.morin.1998 at free.fr
                   |.org                        |

--- Comment #21 from Yann E. MORIN <[email protected]> --- Created attachment 6916
--> https://bugs.busybox.net/attachment.cgi?id=6916&action=edit
Debug mkusers - paranoid

(In reply to Paul Stewart from comment #19)

Damn, that's uterly weird...

OK, so let's be a little bit more paranoiac. Can you test with the attached patch, please?

The expected output is described in that patch, too.

Regards,
Yann E. MORIN.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #22 from Paul Stewart <[email protected]> --- Created attachment 6921
--> https://bugs.busybox.net/attachment.cgi?id=6921&action=edit
Output from mkusers script with debug paranoid patch.

Attached output of paranoid patched mkusers.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #23 from Arnout Vandecappelle <[email protected]> --- D'oh!

From the bash man page:

LINES Used by the select compound command to determine the column length for
printing selection lists. Automatically set if the checkwinsize option is enabled or in an interactive shell upon receipt of a
SIGWINCH.

Your bash is receiving a SIGWINCH from your terminal and updates the LINES variable when it generates some output...

Yann, please rename LINES to a non-conflicting variable :-)

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #24 from Yann E. MORIN <[email protected]> --- (In reply to Arnout Vandecappelle from comment #23)

I'll do the change, for sanity sake.

However:
- the checkwinsize option is disabled by default in a non interactive shell,
- this is not interactive shell, so SIGWINCH should be ignored.

I'll paste a patch in a moment...

Regards,
Yann E. MORIN.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

Yann E. MORIN <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

   Attachment #6916|0                           |1
        is obsolete|                            |
   Attachment #6921|0                           |1
        is obsolete|                            |

--- Comment #25 from Yann E. MORIN <[email protected]> --- Created attachment 6926
--> https://bugs.busybox.net/attachment.cgi?id=6926&action=edit
Rename variable, paranoid debug

Paul,

Could you please test this patch and report the output, please?

For the record, the output here contains:

checkwinsize off

Regards,
Yann E. MORIN.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #26 from Paul Stewart <[email protected]> --- (In reply to Yann E. MORIN from comment #25)

That produced the correct output. The passwd, group and shadow files were all created properly. Yay! Will post the output file momentarily.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #27 from Paul Stewart <[email protected]> --- Created attachment 6931
--> https://bugs.busybox.net/attachment.cgi?id=6931&action=edit
Rename variable, paranoid debug output file

Here is the (successful) output from the renamed variable patch.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #28 from Yann E. MORIN <[email protected]> --- (In reply to Paul Stewart from comment #27)

Thanks a lot for all the testing you did! :-)

This is really weird that your shell does have checkwinsize on. It should be off by default in a non-interactive shell, and shell script are running in non-interactive shells...

Anyway... I just sent the patch:

    http://lists.busybox.net/pipermail/buildroot/2017-March/185755.html

Thanks again!

Regards,
Yann E. MORIN.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #29 from Paul Stewart <[email protected]> --- (In reply to Yann E. MORIN from comment #28)

You're welcome. Thank you for the fix! Will this go into a 2017.02.1 bugfix or will it wait for the 2017.05 release?

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

--- Comment #30 from Yann E. MORIN <[email protected]> --- (In reply to Paul Stewart from comment #29)

Since we decided to have 2017.02 be our first long-term maintainance branch, I believe this will go in, yes.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9456

Thomas Petazzoni <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

         Resolution|---                         |FIXED
             Status|REOPENED                    |RESOLVED

--- Comment #31 from Thomas Petazzoni <[email protected]> --- Fixed by
https://git.buildroot.org/buildroot/commit/?id=00d34e8a6f378653a384c66d68f9a65e13b8034f.
Thanks Paul, Arnout and Yann for the investigation!

You are receiving this mail because:
You are on the CC list for the bug.