Initial support for keycodemapdb GIT submodule

This patch series is an update to:

v1: https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg02047.html
v2: https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg02471.html
v3: https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg02517.html
v4: https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg02708.html
v5: https://lists.gnu.org/archive/html/qemu-devel/2017-09/msg02950.html
v6: https://lists.gnu.org/archive/html/qemu-devel/2017-09/msg07673.html
v7: https://lists.gnu.org/archive/html/qemu-devel/2017-09/msg07740.html

The keycodemap project[1] provides a database mapping between many different keysym/keycode/scancode sets, along with a tool to generate mapping/lookup tables in various programming languages. It is already used by GTK-VNC, SPICE-GTK and libvirt.

In this v6 posting I have dropped most of the conversion to keycodemapdb across the devices / UI frontends. This only converts the core input APIs, and wires up the git submodule build system mechanics. The remaining patches from the v5 posting will be submitted separately, once this initial conversion is mergable.

Changed in v8:

- Fix error checking in archive-source.sh changes (Eric)
- Use -e instead of -d to check for .git (Eric)
- Misc typos (Eric)
- Use && instead of ; in make shell rules (Eric)
- Fully quote $source_path usage (Eric)
- Fix position of trap statement for cleanup (Eric)
- Added missing dependnacy for Makefile (Patchew)
- Moved build rules from ui/Makefile.objs into Makefile because we must not have those rules defined in the target specific Makefiles

Changed in v7:

- subdir-dtc and ui/input-keymap-* make rules must depend on the submodule status file to ensure build ordering (patchew)
- Use 'git stash' to ensure archive-source.sh keeps non-committed changes for docker/vm testing (Fam)

Changed in v6:

- Switched back to using a git submodule
- Wire up Makefile rules so that git submodule is automatically refreshed when needed prior to build.
- Fix source tarball creation for docker/vm tests wrt submodules

Changed in v5:

- Don't try to initialize git submodule at all
- Store generate keymap files in GIT

Changed in v4:

- Run submodule update in source_dir for vpath builds (patchew)
- Force submodule update in docker rules in case they are run without configure (patchew)

Changed in v3:

- Ensure docker builds pull in keycodemapdb submodule (patchew)
- Add compat with py26 for RHEL-6 in keycodemapdb tools (patchew)
- Initialize submodule in configure script (patchew)

Changed in v2:

- Change filename pattern to 'ui/input-keymap-$SRC-to-$DST.c' and map names 'qemu_input_map_$SRC_to_$DST' (Eric)
- Fix typos (Eric)
- Drop changes to InputKeyEvent struct (Eric)
- Fix VPATH build (patchew)
- Fix code style errors (patchew)

[1] https://gitlab.com/keycodemap/keycodemapdb/

Daniel P. Berrange (6):
build: automatically handle GIT submodule checkout for dtc
docker: don't rely on submodules existing in the main checkout
ui: add keycodemapdb repository as a GIT submodule
ui: convert common input code to keycodemapdb
ui: convert key events to QKeyCodes immediately
ui: don't export qemu_input_event_new_key

 .gitignore                |   2 +
 .gitmodules               |   3 +
 MAINTAINERS               |   6 +
 Makefile                  |  46 ++++++-
 configure                 |  52 +++++---
 include/ui/input.h        |  12 +-
 scripts/archive-source.sh |  34 ++++-
 scripts/git-submodule.sh  |  38 ++++++
 ui/input-keymap.c         | 326 +++-------------------------------------------
 ui/input.c                |  26 ++--
 ui/keycodemapdb           |   1 +
 11 files changed, 198 insertions(+), 348 deletions(-)
 create mode 100755 scripts/git-submodule.sh
 create mode 160000 ui/keycodemapdb

2.13.5

Currently if DTC is required by configure and not available in the host OS install, we exit with an error message telling the user to checkout a git submodule or install the library.

This introduces automatic handling of the git submodule checkout process and enables it for dtc. This only runs if building from GIT, so users of release tarballs still need the system library install. The current state of the git checkout is stashed in .git-submodule-status, and a helper program is used to determine if this state matches the desired submodule state. A dependency against 'Makefile' ensures that the submodule state is refreshed at the start of the build process

---
 .gitignore               |  1 +
 MAINTAINERS              |  6 ++++++
 Makefile                 | 25 ++++++++++++++++++++++++-
 configure                | 46 ++++++++++++++++++++++++++--------------------
 scripts/git-submodule.sh | 38 ++++++++++++++++++++++++++++++++++++++
 5 files changed, 95 insertions(+), 21 deletions(-)
 create mode 100755 scripts/git-submodule.sh

index 40acfcb9e2..06bf972fc3 100644
@@ -112,6 +112,7 @@
 /docs/version.texi
 *.tps
 .stgit-*
+.git-submodule-status
 cscope.*
 tags
 TAGS

index 932443df41..7708bbb21d 100644
@@ -1947,3 +1947,9 @@ M: Daniel P. Berrange <[email protected]>
 S: Odd Fixes
 F: docs/devel/build-system.txt
 
+Build System
+------------
+GIT submodules
+M: Daniel P. Berrange <[email protected]>
+S: Odd Fixes
+F: scripts/git-submodule.sh

index 2be61fcf1c..c00ed7141a 100644
@@ -14,6 +14,29 @@ ifneq ($(wildcard config-host.mak),)
 all:
 include config-host.mak
 
+git-submodule-update:
+
+.PHONY: git-submodule-update
+
+ifeq (0,$(MAKELEVEL))
+  git_module_status := $(shell \
+    cd '$(SRC_PATH)' && \
+    ./scripts/git-submodule.sh status $(GIT_SUBMODULES); \
+    echo $$?; \
+  )
+
+ifeq (1,$(git_module_status))
+git-submodule-update:
+	$(call quiet-command, \
+          (cd $(SRC_PATH) && ./scripts/git-submodule.sh update $(GIT_SUBMODULES)), \
+          "GIT","$(GIT_SUBMODULES)")
+endif
+endif
+
+.git-submodule-status: git-submodule-update
+
+Makefile: .git-submodule-status
+
 # Check that we're not trying to do an out-of-tree build from
 # a tree that's been used for an in-tree build.
 ifneq ($(realpath $(SRC_PATH)),$(realpath .))

@@ -329,7 +352,7 @@ DTC_MAKE_ARGS=-I$(SRC_PATH)/dtc VPATH=$(SRC_PATH)/dtc -C dtc V="$(V)" LIBFDT_src
 DTC_CFLAGS=$(CFLAGS) $(QEMU_CFLAGS)
 DTC_CPPFLAGS=-I$(BUILD_DIR)/dtc -I$(SRC_PATH)/dtc -I$(SRC_PATH)/dtc/libfdt
 
-subdir-dtc:dtc/libfdt dtc/tests
+subdir-dtc: .git-submodule-status dtc/libfdt dtc/tests
 	$(call quiet-command,$(MAKE) $(DTC_MAKE_ARGS) CPPFLAGS="$(DTC_CPPFLAGS)" CFLAGS="$(DTC_CFLAGS)" LDFLAGS="$(LDFLAGS)" ARFLAGS="$(ARFLAGS)" CC="$(CC)" AR="$(AR)" LD="$(LD)" $(SUBDIR_MAKEFLAGS) libfdt/libfdt.a,)
 
 dtc/%:

new file mode 100755
index 0000000000..9f73f0e6e3
@@ -0,0 +1,38 @@
+#!/bin/bash
+#
+# This code is licensed under the GPL version 2 or later.  See
+# the COPYING file in the top-level directory.
+
+set -e
+
+substat=".git-submodule-status"
+
+command=$1
+shift
+modules="$@"
+
+if test -z "$modules"
+then
+    test -e $substat || touch $substat
+    exit 0
+fi
+
+if ! test -e ".git"
+then
+    echo "$0: unexpectedly called with submodules but no git checkout exists"
+    exit 1
+fi
+
+case "$command" in
+status)
+    test -f "$substat" || exit 1
+    trap "rm -f ${substat}.tmp" EXIT
+    git submodule status $modules > "${substat}.tmp"
+    diff "${substat}" "${substat}.tmp" >/dev/null
+    exit $?
+    ;;
+update)
+    git submodule update --init $modules 1>/dev/null 2>&1
+    git submodule status $modules > "${substat}"
+    ;;
+esac

2.13.5

When building the tarball to pass into the docker/vm test image, the code relies on the git submodules being checked out in the main checkout.

ie if the developer has not run 'git submodule update --init dtc' many of the docker tests will fail due to the libfdt package not being present in the test images. Patchew manually checks out the dtc submodule in the main git checkout, but this is a bad idea.

When running tests we want to have a predictable set of submodules included in the source that's tested. The build environment is completely independent of the developers host OS, so the submodules the developer has checked out should not be considered relevant for the tests.

This changes the archive-source.sh script so that it clones the current git checkout into a temporary directory, checks out a fixed set of submodules, builds the tarball and finally removes the temporary git clone.

---
 scripts/archive-source.sh | 34 ++++++++++++++++++++++++++++------
 1 file changed, 28 insertions(+), 6 deletions(-)

index c4e7d98f4d..4029de7b20 100755
@@ -18,15 +18,37 @@ if test $# -lt 1; then
     error "Usage: $0 <output tarball>"
 fi
 
-tar_file="$1"
-list_file="$1.list"
-submodules=$(git submodule foreach --recursive --quiet 'echo $name')
+tar_file=`realpath "$1"`
+list_file="${tar_file}.list"
+vroot_dir="${tar_file}.vroot"
 
-if test $? -ne 0; then
-    error "git submodule command failed"
+# We want a predictable list of submodules for builds, that is
+# independent of what the developer currently has initialized
+# in their checkout, because the build environment is completely
+# different to the host OS.
+submodules="dtc"
+
+trap "status=$?; rm -rf \"$list_file\" \"$vroot_dir\"; exit \$status" 0 1 2 3 15
+
+if git diff-index --quiet HEAD -- &>/dev/null
+then
+    HEAD=HEAD
+else
+    HEAD=`git stash create`
 fi
+git clone --shared . "$vroot_dir"
+test $? -ne 0 && error "failed to clone into '$vroot_dir'"
+
+cd "$vroot_dir"
+test $? -ne 0 && error "failed to change into '$vroot_dir'"
+
+git checkout $HEAD
+test $? -ne 0 && error "failed to checkout $HEAD revision"
 
-trap "status=$?; rm -f \"$list_file\"; exit \$status" 0 1 2 3 15
+for sm in $submodules; do
+    git submodule update --init $sm
+    test $? -ne 0 && error "failed to init submodule $sm"
+done
 
 if test -n "$submodules"; then
     {

2.13.5

I feel like we've just gone around in one big circle here. The whole point of passing the tarball into the docker build in the first place was to avoid messing around with checkouts and the like. Is it not possible to get what we want direct from git?

What I've implemented here gets what we want straight from git, with minimal performance overhead because entire .git is shared with hard links.

Regards,
Daniel

|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

The https://gitlab.com/keycodemap/keycodemapdb/ repo contains a data file mapping between all the different scancode/keycode/keysym sets that are known, and a tool to auto-generate lookup tables for different combinations.

It is used by GTK-VNC, SPICE-GTK and libvirt for mapping keys. Using it in QEMU will let us replace many hand written lookup tables with auto-generated tables from a master data source, reducing bugs. Adding new QKeyCodes will now only require the master table to be updated, all ~20 other tables will be automatically updated to follow.

---
 .gitignore                |  1 +
 .gitmodules               |  3 +++
 Makefile                  | 18 ++++++++++++++++++
 configure                 |  8 +++++++-
 scripts/archive-source.sh |  2 +-
 ui/keycodemapdb           |  1 +
 6 files changed, 31 insertions(+), 2 deletions(-)
 create mode 160000 ui/keycodemapdb

index 06bf972fc3..1eada71afd 100644
@@ -14,6 +14,7 @@
 /trace/generated-tcg-tracers.h
 /ui/shader/texture-blit-frag.h
 /ui/shader/texture-blit-vert.h
+/ui/input-keymap-*.c
 *-timestamp
 /*-softmmu
 /*-darwin-user

index 84c54cdc49..f3bbc01f82 100644
@@ -34,3 +34,6 @@
 [submodule "roms/QemuMacDrivers"]
 	path = roms/QemuMacDrivers
 	url = git://git.qemu.org/QemuMacDrivers.git
+[submodule "ui/keycodemapdb"]
+	path = ui/keycodemapdb
+	url = https://gitlab.com/keycodemap/keycodemapdb.git

index c00ed7141a..8eb6c9085e 100644
@@ -214,6 +214,24 @@ trace-dtrace-root.h: trace-dtrace-root.dtrace
 
 trace-dtrace-root.o: trace-dtrace-root.dtrace
 
+KEYCODEMAP_GEN = $(SRC_PATH)/ui/keycodemapdb/tools/keymap-gen
+KEYCODEMAP_CSV = $(SRC_PATH)/ui/keycodemapdb/data/keymaps.csv
+
+KEYCODEMAP_FILES = \
+		 $(NULL)
+
+GENERATED_FILES += $(KEYCODEMAP_FILES)
+
+ui/input-keymap-%.c: $(KEYCODEMAP_GEN) $(KEYCODEMAP_CSV) $(SRC_PATH)/ui/Makefile.objs .git-submodule-status
+	$(call quiet-command,\
+	    src=$$(echo $@ | sed -E -e "s,^ui/input-keymap-(.+)-to-(.+)\.c$$,\1,") && \
+	    dst=$$(echo $@ | sed -E -e "s,^ui/input-keymap-(.+)-to-(.+)\.c$$,\2,") && \
+	    $(PYTHON) $(KEYCODEMAP_GEN) \
+	          --lang glib2 \
+	          --varname qemu_input_map_$${src}_to_$${dst} \
+	          code-map $(KEYCODEMAP_CSV) $${src} $${dst} \
+	        > $@ || rm $@, "GEN", "$@")
+
 # Don't try to regenerate Makefile or configure
 # We don't generate any of them
 Makefile: ;

index b324e057f1..eb420abc47 100755
@@ -264,7 +264,13 @@ cc_i386=i386-pc-linux-gnu-gcc
 libs_qga=""
 debug_info="yes"
 stack_protector=""
-git_submodules=""
+
+if test -e "$source_path/.git"
+then
+    git_submodules="ui/keycodemapdb"
+else
+    git_submodules=""
+fi
 
 # Don't accept a target_list environment variable.
 unset target_list

index 4029de7b20..4e63774f9a 100755
@@ -26,7 +26,7 @@ vroot_dir="${tar_file}.vroot"
 # independent of what the developer currently has initialized
 # in their checkout, because the build environment is completely
 # different to the host OS.
-submodules="dtc"
+submodules="dtc ui/keycodemapdb"
 
 trap "status=$?; rm -rf \"$list_file\" \"$vroot_dir\"; exit \$status" 0 1 2 3 15
 

new file mode 160000
index 0000000000..56ce5650d2
@@ -0,0 +1 @@
+Subproject commit 56ce5650d2c6ea216b4580df44b9a6dd3bc92c3b

2.13.5

Replace the number_to_qcode, qcode_to_number and linux_to_qcode tables with automatically generated tables.

Missing entries in linux_to_qcode now fixed:

KEY_LINEFEED -> Q_KEY_CODE_LF
KEY_KPEQUAL -> Q_KEY_CODE_KP_EQUALS
KEY_COMPOSE -> Q_KEY_CODE_COMPOSE
KEY_AGAIN -> Q_KEY_CODE_AGAIN
KEY_PROPS -> Q_KEY_CODE_PROPS
KEY_UNDO -> Q_KEY_CODE_UNDO
KEY_FRONT -> Q_KEY_CODE_FRONT
KEY_COPY -> Q_KEY_CODE_COPY
KEY_OPEN -> Q_KEY_CODE_OPEN
KEY_PASTE -> Q_KEY_CODE_PASTE
KEY_CUT -> Q_KEY_CODE_CUT
KEY_HELP -> Q_KEY_CODE_HELP
KEY_MEDIA -> Q_KEY_CODE_MEDIASELECT

In addition, some fixes:

- KEY_PLAYPAUSE now maps to Q_KEY_CODE_AUDIOPLAY, instead of KEY_PLAYCD. KEY_PLAYPAUSE is defined across almost all scancodes sets, while KEY_PLAYCD only appears in AT set1, so the former is a more useful mapping.

Missing entries in qcode_to_number now fixed:

Q_KEY_CODE_AGAIN -> 0x85
Q_KEY_CODE_PROPS -> 0x86
Q_KEY_CODE_UNDO -> 0x87
Q_KEY_CODE_FRONT -> 0x8c
Q_KEY_CODE_COPY -> 0xf8
Q_KEY_CODE_OPEN -> 0x64
Q_KEY_CODE_PASTE -> 0x65
Q_KEY_CODE_CUT -> 0xbc
Q_KEY_CODE_LF -> 0x5b
Q_KEY_CODE_HELP -> 0xf5
Q_KEY_CODE_COMPOSE -> 0xdd
Q_KEY_CODE_KP_EQUALS -> 0x59
Q_KEY_CODE_MEDIASELECT -> 0xed

In addition, some fixes:

- Q_KEY_CODE_MENU was incorrectly mapped to the compose scancode (0xdd) and is now mapped to 0x9e
- Q_KEY_CODE_FIND was mapped to 0xe065 (Search) instead of to 0xe041 (Find)
- Q_KEY_CODE_HIRAGANA was mapped to 0x70 (Katakanahiragana) instead of of 0x77 (Hirigana)
- Q_KEY_CODE_PRINT was mapped to 0xb7 which is not a defined scan code in AT set 1, it is now mapped to 0x54 (sysrq)

---
 Makefile           |   3 +
 include/ui/input.h |  11 +-
 ui/input-keymap.c  | 326 +++--------------------------------------------------
 3 files changed, 28 insertions(+), 312 deletions(-)

index 8eb6c9085e..64cb116bf7 100644
@@ -218,6 +218,9 @@ KEYCODEMAP_GEN = $(SRC_PATH)/ui/keycodemapdb/tools/keymap-gen
 KEYCODEMAP_CSV = $(SRC_PATH)/ui/keycodemapdb/data/keymaps.csv
 
 KEYCODEMAP_FILES = \
+		 ui/input-keymap-linux-to-qcode.c \
+		 ui/input-keymap-qcode-to-qnum.c \
+		 ui/input-keymap-qnum-to-qcode.c \
 		 $(NULL)
 
 GENERATED_FILES += $(KEYCODEMAP_FILES)

index c488585def..479cc46cfc 100644
@@ -43,7 +43,7 @@ void qemu_input_event_send_key(QemuConsole *src, KeyValue *key, bool down);
 void qemu_input_event_send_key_number(QemuConsole *src, int num, bool down);
 void qemu_input_event_send_key_qcode(QemuConsole *src, QKeyCode q, bool down);
 void qemu_input_event_send_key_delay(uint32_t delay_ms);
-int qemu_input_key_number_to_qcode(uint8_t nr);
+int qemu_input_key_number_to_qcode(unsigned int nr);
 int qemu_input_key_value_to_number(const KeyValue *value);
 int qemu_input_key_value_to_qcode(const KeyValue *value);
 int qemu_input_key_value_to_scancode(const KeyValue *value, bool down,

@@ -69,4 +69,13 @@ void qemu_input_check_mode_change(void);
 void qemu_add_mouse_mode_change_notifier(Notifier *notify);
 void qemu_remove_mouse_mode_change_notifier(Notifier *notify);
 
+extern const guint qemu_input_map_linux_to_qcode_len;
+extern const guint16 qemu_input_map_linux_to_qcode[];
+
+extern const guint qemu_input_map_qcode_to_qnum_len;
+extern const guint16 qemu_input_map_qcode_to_qnum[];
+
+extern const guint qemu_input_map_qnum_to_qcode_len;
+extern const guint16 qemu_input_map_qnum_to_qcode[];
+
 #endif /* INPUT_H */

2.13.5

Always use QKeyCode in the InputKeyEvent struct, by converting key numbers to QKeyCode at the time the event is created. This allows the code processing / consuming key events to assume QKeyCode is used. The only place we accept a key number in the InputKeyEvent struct is with QMP commands sent by the user.

---
 ui/input.c | 24 +++++++++++++++++-------
 1 file changed, 17 insertions(+), 7 deletions(-)

index 3422d4a8ef..4b241aa823 100644
@@ -157,9 +157,16 @@ void qmp_input_send_event(bool has_device, const char *device,
     }
 
     for (e = events; e != NULL; e = e->next) {
-        InputEvent *event = e->value;
-
-        qemu_input_event_send(con, event);
+        InputEvent *evt = e->value;
+
+        if (evt->type == INPUT_EVENT_KIND_KEY &&
+            evt->u.key.data->key->type == KEY_VALUE_KIND_NUMBER) {
+            KeyValue *key = evt->u.key.data->key;
+            QKeyCode code = qemu_input_key_number_to_qcode(key->u.qcode.data);
+            qemu_input_event_send_key_qcode(con, code, evt->u.key.data->down);
+        } else {
+            qemu_input_event_send(con, evt);
+        }
     }
 
     qemu_input_event_sync();

@@ -341,6 +348,11 @@ void qemu_input_event_send_impl(QemuConsole *src, InputEvent *evt)
 
 void qemu_input_event_send(QemuConsole *src, InputEvent *evt)
 {
+    /* Expect all parts of QEMU to send events with QCodes exclusively.
+     * Key numbers are only supported as end-user input via QMP */
+    assert(!(evt->type == INPUT_EVENT_KIND_KEY &&
+             evt->u.key.data->key->type == KEY_VALUE_KIND_NUMBER));
+
     if (!runstate_is_running() && !runstate_check(RUN_STATE_SUSPENDED)) {
         return;
     }

@@ -400,10 +412,8 @@ void qemu_input_event_send_key(QemuConsole *src, KeyValue *key, bool down)
 
 void qemu_input_event_send_key_number(QemuConsole *src, int num, bool down)
 {
-    KeyValue *key = g_new0(KeyValue, 1);
-    key->type = KEY_VALUE_KIND_NUMBER;
-    key->u.number.data = num;
-    qemu_input_event_send_key(src, key, down);
+    QKeyCode code = qemu_input_key_number_to_qcode(num);
+    qemu_input_event_send_key_qcode(src, code, down);
 }
 
 void qemu_input_event_send_key_qcode(QemuConsole *src, QKeyCode q, bool down)

2.13.5

All public code should use qemu_input_event_send_key* functions instead of creating an event directly.

---
 include/ui/input.h | 1 -
 ui/input.c         | 2 +-
 2 files changed, 1 insertion(+), 2 deletions(-)

index 479cc46cfc..f8cee43f65 100644
@@ -38,7 +38,6 @@ void qemu_input_event_send_impl(QemuConsole *src, InputEvent *evt);
 void qemu_input_event_sync(void);
 void qemu_input_event_sync_impl(void);
 
-InputEvent *qemu_input_event_new_key(KeyValue *key, bool down);
 void qemu_input_event_send_key(QemuConsole *src, KeyValue *key, bool down);
 void qemu_input_event_send_key_number(QemuConsole *src, int num, bool down);
 void qemu_input_event_send_key_qcode(QemuConsole *src, QKeyCode q, bool down);

index 4b241aa823..290b47354a 100644
@@ -386,7 +386,7 @@ void qemu_input_event_sync(void)
     replay_input_sync_event();
 }
 
-InputEvent *qemu_input_event_new_key(KeyValue *key, bool down)
+static InputEvent *qemu_input_event_new_key(KeyValue *key, bool down)
 {
     InputEvent *evt = g_new0(InputEvent, 1);
     evt->u.key.data = g_new0(InputKeyEvent, 1);

2.13.5

FYI, in case people were wondering, patchew successfully passed all tests. So it looks like 8th time lucky for getting submodules working correctly unless someone can break it again.....

http://patchew.org/QEMU/[email protected]/

Regards,
Daniel

|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|