Fix to fcntl guts to ignore flags that can be ORed into cmd

I previously submitted this patch to the OE-core mailing list, and was directed over here.

From c29b7aecfe671013897119b1dee2731900439285 Mon Sep 17 00:00:00 2001
From: Will Page <[email protected]>
Date: Fri, 15 Sep 2017 13:07:44 -0700
Subject: [pseudo][PATCH] Fix to fcntl guts to ignore flags that can be ORed
into cmd

The fcntl guts switch on "cmd" parameter to identify the fcntl command being issued, but isn't aware of the file creation flags that can be ORed in.

This change masks out the flags from the command only for the switch statement so that it can correctly determine whether it needs to pass "lock" or "arg". When real_fcntl() is called, the original value is still passed on. This corrects an issue observed using pseudo on modern linux desktops resulting in "pseudo: unknown fcntl argument 1030, assuming long argument." diagnostics in the output. Decimal 1030 is 0x0406, which translates to O_NOCTTY | F_SETLK, and should call "rc = real_fcntl(fd, cmd, lock);", but instead falls through to the default case instead calling "rc = real_fcntl(fd, cmd, arg);".

Tested the change using perftest - without this patch, the issue is trivially reproducible on my Ubuntu Xenial system, and after patching it emitted no "unknown fcntl argument" diagnostics.

---
 ports/linux/guts/fcntl.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

index 639fd24..d278a8c 100644
@@ -8,6 +8,10 @@
  */
  	long arg;
 	int save_errno;
+	/* some bits can be ORed into the cmd should be explicitly ignored
+	 * see fcntl documentation on F_SETFL */
+	int o_mode_mask = (O_RDONLY | O_WRONLY | O_RDWR) |
+		(O_CREAT | O_EXCL | O_NOCTTY | O_TRUNC);
 
 	/* we don't know whether we need lock or arg; grab both, which
 	 * should be safe enough on Linuxy systems. */

@@ -15,7 +19,7 @@
 	arg = va_arg(ap, long);
 	va_end(ap);
 
-	switch (cmd) {
+	switch (cmd & ~(o_mode_mask)) {
 	case F_DUPFD:
 #ifdef F_DUPFD_CLOEXEC
 	case F_DUPFD_CLOEXEC:

2.7.4