OCaml fixes for Xen 4.17
v3 cover OCaml fixes for Xen 4.17
These are the patches that I have outstanding for Xen 4.17. I have included a reason why I'm requesting them to be included in 4.17 after the --- line in each individual patch, see also a summary below.
For convenience the patches are also available in a git repo:
``` git remote add edwintorok https://github.com/edwintorok/xen.git git fetch edwintorok private/edvint/for-4.17 git log -p origin/master..private/edvint/for-4.17 ```
And viewable with a browser too:
https://github.com/edwintorok/xen/compare/private/edvint/for-4.17
* 3 patches related to OCaml 5 support
https://patchwork.kernel.org/project/xen-devel/list/?series=680975
These have already been posted to the list previously, but not yet committed to master (I probably didn't use the correct subject and CC line for patches meant for 4.17, I think I've fixed that now)
* Makefile.rules followup
Also part of https://patchwork.kernel.org/project/xen-devel/list/?series=680975
these address some review feedback that I received after patches got committed
* oxenstored live update bugfixes
Testing of oxenstored live update has revealed some bugs (some of which got discovered on the C side too and fixed during one of the previous XSAs, but unfortunately none of that discussion is public, and we've ended up rediscovering the issue in the OCaml implementation too, which reminded me of the XSA discussions at the time).
This brings the OCaml live update handling of event channels closer to the C xenstored version.
It also fixes a few more bugs regarding logging and exception handling during live update, and during out of memory situations (theoretical now after XSA-326 fix).
* a bugfix for a xenctrl binding
Xen returns uninitialized data as part of a paging op domctl when a domain is dying. Workaround in the C stub by always initializing the domctl arguments to detect this.
Xen fix in hypervisor side will be done separately, but even then having this is useful defensive coding.
This is a 9 year old bug that still happens today, I've encountered it while testing this very series, hence the inclusion here.
I expect most of these to be straight forward bugfixes, the only one slightly controversial might be the indentation one: changing tabs to spaces to match Xen coding style.
I was unsure whether to include it here,
but I think it is best to have it in 4.17 to simplify future (security) backports from master to 4.17, and avoid having to deal with whitespace issues all the time when writing patches.
The code here used a style that was different from Xen's, and also different from every other piece of code that I work on, and OCaml indentation tools also only support spaces, not tabs, so there really is no reason to keep the code as is (initially I thought it uses tabs to follow Xen style, but after reading CODING_STYLE I realized that is not true). It is very easy to verify that the patch changes nothing with git diff -w, or git log -p -1.
Edwin Török (15):
tools/ocaml/libs/eventchn: do not leak event channels and OCaml 5.0
compat
tools/ocaml/libs/xc: OCaml 5.0 compatibility tools/ocaml/libs/{xb, mmap}: use Data_abstract_val wrapper tools/ocaml/xenstored/Makefile: use ocamldep -sort for linking order tools/ocaml/Makefile.rules: do not run ocamldep on distclean tools/ocaml/Makefile.rules: hide -include on *clean CODING_STYLE(tools/ocaml): add 'make format' and remove tabs tools/ocaml/libs/evtchn: add xenevtchn_fdopen bindings tools/ocaml/xenstored/store.ml: fix build error tools/ocaml/xenstored: keep eventchn FD open across live update tools/ocaml/xenstored: do not rebind event channels after live update tools/ocaml/xenstored: log live update issues at warning level tools/ocaml/xenstored: set uncaught exception handler tools/ocaml/xenstored/syslog_stubs.c: avoid potential NULL dereference tools/ocaml/libs/xc: fix use of uninitialized memory in
shadow_allocation_get
tools/ocaml/Makefile | 5 + tools/ocaml/Makefile.rules | 4 +- tools/ocaml/libs/eventchn/xeneventchn.ml | 11 +- tools/ocaml/libs/eventchn/xeneventchn.mli | 14 +- tools/ocaml/libs/eventchn/xeneventchn_stubs.c | 199 +- tools/ocaml/libs/mmap/mmap_stubs.h | 9 +- tools/ocaml/libs/mmap/xenmmap.ml | 2 +- tools/ocaml/libs/mmap/xenmmap.mli | 4 +- tools/ocaml/libs/mmap/xenmmap_stubs.c | 114 +- tools/ocaml/libs/xb/op.ml | 76 +- tools/ocaml/libs/xb/packet.ml | 30 +- tools/ocaml/libs/xb/partial.ml | 48 +- tools/ocaml/libs/xb/xb.ml | 422 ++-- tools/ocaml/libs/xb/xb.mli | 106 +- tools/ocaml/libs/xb/xenbus_stubs.c | 50 +- tools/ocaml/libs/xb/xs_ring.ml | 28 +- tools/ocaml/libs/xb/xs_ring_stubs.c | 216 +- tools/ocaml/libs/xc/abi-check | 2 +- tools/ocaml/libs/xc/xenctrl.ml | 330 +-- tools/ocaml/libs/xc/xenctrl.mli | 12 +- tools/ocaml/libs/xc/xenctrl_stubs.c | 1428 ++++++------ tools/ocaml/libs/xentoollog/caml_xentoollog.h | 6 +- .../ocaml/libs/xentoollog/xentoollog_stubs.c | 196 +- tools/ocaml/libs/xl/xenlight_stubs.c | 2022 ++++++++--------- tools/ocaml/libs/xs/queueop.ml | 48 +- tools/ocaml/libs/xs/xs.ml | 220 +- tools/ocaml/libs/xs/xs.mli | 46 +- tools/ocaml/libs/xs/xsraw.ml | 300 +-- tools/ocaml/libs/xs/xst.ml | 76 +- tools/ocaml/libs/xs/xst.mli | 20 +- tools/ocaml/test/dmesg.ml | 26 +- tools/ocaml/test/list_domains.ml | 4 +- tools/ocaml/test/raise_exception.ml | 4 +- tools/ocaml/test/xtl.ml | 28 +- tools/ocaml/xenstored/Makefile | 6 +- tools/ocaml/xenstored/config.ml | 156 +- tools/ocaml/xenstored/connection.ml | 594 ++--- tools/ocaml/xenstored/connections.ml | 304 +-- tools/ocaml/xenstored/define.ml | 6 +- tools/ocaml/xenstored/disk.ml | 218 +- tools/ocaml/xenstored/domain.ml | 104 +- tools/ocaml/xenstored/domains.ml | 320 +-- tools/ocaml/xenstored/event.ml | 12 +- tools/ocaml/xenstored/history.ml | 62 +- tools/ocaml/xenstored/logging.ml | 467 ++-- tools/ocaml/xenstored/packet.ml | 20 +- tools/ocaml/xenstored/parse_arg.ml | 106 +- tools/ocaml/xenstored/perms.ml | 216 +- tools/ocaml/xenstored/poll.ml | 68 +- tools/ocaml/xenstored/poll.mli | 4 +- tools/ocaml/xenstored/process.ml | 1212 +++++----- tools/ocaml/xenstored/quota.ml | 74 +- tools/ocaml/xenstored/select_stubs.c | 62 +- tools/ocaml/xenstored/stdext.ml | 190 +- tools/ocaml/xenstored/store.ml | 752 +++--- tools/ocaml/xenstored/symbol.ml | 2 +- tools/ocaml/xenstored/syslog.ml | 48 +- tools/ocaml/xenstored/syslog_stubs.c | 33 +- tools/ocaml/xenstored/systemd_stubs.c | 10 +- tools/ocaml/xenstored/transaction.ml | 352 +-- tools/ocaml/xenstored/trie.ml | 222 +- tools/ocaml/xenstored/trie.mli | 22 +- tools/ocaml/xenstored/utils.ml | 146 +- tools/ocaml/xenstored/xenstored.ml | 1051 ++++----- 64 files changed, 6557 insertions(+), 6388 deletions(-)
base-commit: e61a78981364925a43c9cc24dc77b62ff7b93c9f
Signature · 1 line
2.34.1
PATCH v3 1/15 tools/ocaml/libs/eventchn: do not leak event channels and OCaml 5.0 compat
Add a finalizer on the event channel value, so that it calls xenevtchn_close when the value would be GCed.
In practice oxenstored seems to be the only user of this, and it creates a single global event channel only,
but freeing this could still be useful when run with OCAMLRUNPARAM=c
The code was previously casting a C pointer to an OCaml value, which should be avoided: OCaml 5.0 won't support it. (all "naked" C pointers must be wrapped inside an OCaml value,
either an Abstract tag, or Nativeint, see the manual
https://ocaml.org/manual/intfc.html#ss:c-outside-head)
---
Reason for inclusion in 4.17:
- Fixes a runtime bug with upcoming OCaml 5.0 (already in beta)
Changes since v2:
- added Acked-by line
- add 4.17 reason
--- tools/ocaml/libs/eventchn/xeneventchn_stubs.c | 29 +++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-)
tools/ocaml/libs/eventchn/xeneventchn_stubs.c +27 -2
@@ -33,7 +33,30 @@#include <caml/fail.h> #include <caml/signals.h> -#define _H (__h) ((xenevtchn_handle *)(__h)) +/* We want to close the event channel when it is no longer in use, +which can only be done safely with a finalizer. +Event channels are typically long lived, so we don't need tighter control over resource deallocation. +Use a custom block +*/ + +/* Access the xenevtchn_t* part of the OCaml custom block */ +#define _H (__h) (*((xenevtchn_handle**)Data_custom_val(__h))) + +static void stub_evtchn_finalize (value v) +{ +/* docs say to not use any CAMLparam* macros here */ +xenevtchn_close (_H (v)); +} + +static struct custom_operations xenevtchn_ops = { +"xenevtchn" , + stub_evtchn_finalize, + custom_compare_default,/* raises Failure, cannot compare */ + custom_hash_default,/* ignored */ + custom_serialize_default,/* raises Failure, can't serialize */ + custom_deserialize_default,/* raises Failure, can't deserialize */ + custom_compare_ext_default/* raises Failure */ +}; CAMLprim value stub_eventchn_init(void) {
@@ -48,7 +71,9 @@ CAMLprim value stub_eventchn_init(void)if (xce ==NULL )caml_failwith ("open failed" ); - result = (value )xce; +/* contains file descriptors, trigger full GC at least every 128 allocations */ + result =caml_alloc_custom (&xenevtchn_ops,sizeof (xce),0 ,1 ); +_H (result) = xce;CAMLreturn (result); }
Signature · 1 line
2.34.1
PATCH v3 2/15 tools/ocaml/libs/xc: OCaml 5.0 compatibility
In reply to Edwin TörökFollow the manual to avoid naked pointers:
https://v2.ocaml.org/manual/intfc.html#ss:c-outside-head
No functional change, except on OCaml 5.0 where it is a bugfix.
---
Reason for inclusion in 4.17:
- bugfix for upcoming OCaml 5.0 compiler (already in beta)
Changes since v2:
- add Acked-by line
--- tools/ocaml/libs/mmap/mmap_stubs.h | 5 +++++ tools/ocaml/libs/xc/xenctrl_stubs.c | 11 ++++++----- 2 files changed, 11 insertions(+), 5 deletions(-)
tools/ocaml/libs/mmap/mmap_stubs.h +5 -0
@@ -30,4 +30,9 @@ struct mmap_interface int len; }; +/* for compatibility with OCaml 4.02.3 */ +#ifndef Data_abstract_val +#define Data_abstract_val (v) ((void*) Op_val(v)) +#endif +#endif
tools/ocaml/libs/xc/xenctrl_stubs.c +6 -5
@@ -37,7 +37,7 @@#include "mmap_stubs.h" -#define _H (__h) ((xc_interface *)(__h)) +#define _H (__h) *((xc_interface **) Data_abstract_val(__h))#define _D (__d) ((uint32_t)Int_val(__d))#ifndef Val_none
@@ -70,14 +70,15 @@ static void Noreturn failwith_xc(xc_interface *xch) CAMLprim value stub_xc_interface_open(void) {CAMLparam0 (); -xc_interface *xch; +CAMLlocal1 (result); + result =caml_alloc (1 , Abstract_tag);/* Don't assert XC_OPENFLAG_NON_REENTRANT because these bindings * do not prevent re-entrancy to libxc */ - xch =xc_interface_open (NULL ,NULL ,0 ); -if (xch ==NULL ) +_H (result) =xc_interface_open (NULL ,NULL ,0 ); +if (_H (result) ==NULL )failwith_xc (NULL ); -CAMLreturn ((value )xch); +CAMLreturn (result); }
Signature · 1 line
2.34.1
PATCH v3 3/15 tools/ocaml/libs/{xb, mmap}: use Data_abstract_val wrapper
In reply to Edwin TörökThis is not strictly necessary since it is essentially a no-op
currently: a cast to void* and value*, even in OCaml 5.0.
However it does make it clearer that what we have here is not a regular OCaml value, but one allocated with Abstract_tag or Custom_tag, and follows the example from the manual more closely:
https://v2.ocaml.org/manual/intfc.html#ss:c-outside-head
It also makes it clearer that these modules have been reviewed for compat with OCaml 5.0.
No functional change.
---
Reason for inclusion in 4.17:
- make code follow best practice for upcoming OCaml 5.0 compiler (already in beta)
Changes since v2:
- add Acked-by line
--- tools/ocaml/libs/mmap/xenmmap_stubs.c | 2 +- tools/ocaml/libs/xb/xs_ring_stubs.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
tools/ocaml/libs/mmap/xenmmap_stubs.c +1 -1
@@ -28,7 +28,7 @@#include <caml/fail.h> #include <caml/callback.h> -#define Intf_val (a) ((struct mmap_interface *) a) +#define Intf_val (a) ((struct mmap_interface *) Data_abstract_val(a)) static int mmap_interface_init(struct mmap_interface *intf, int fd, int pflag, int mflag,
tools/ocaml/libs/xb/xs_ring_stubs.c +1 -1
@@ -35,7 +35,7 @@#include <sys/mman.h> #include "mmap_stubs.h" -#define GET_C_STRUCT (a) ((struct mmap_interface *) a) +#define GET_C_STRUCT (a) ((struct mmap_interface *) Data_abstract_val(a))/* * Bytes_val has been introduced by Ocaml 4.06.1. So define our own version
Signature · 1 line
2.34.1
PATCH v3 4/15 tools/ocaml/xenstored/Makefile: use ocamldep -sort for linking order
In reply to Edwin TörökA better solution is being worked on for master,
but for now use ocamldep -sort instead of a manually established link order.
The manually established link order will be wrong when (security) patches introduce new dependencies between files that would require changing the link order.
If dune was used as a build system this wouldn't be a problem, but we can't use Dune yet due to OSSTest, which is stuck on Debian oldstable.
No functional change.
---
Reason for inclusion in 4.17:
Avoids having to put this patch as a prerequisite into a security update. Earlier versions of XSA-326 needed this, and although latest version didn't we don't know whether it might be needed again in the future or not.
Changes since v2:
- new in v3 (was previously emailed to security team though)
--- tools/ocaml/xenstored/Makefile | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
tools/ocaml/xenstored/Makefile +5 -1
@@ -72,7 +72,11 @@ XENSTOREDLIBS = \ PROGRAMS = oxenstored oxenstored_LIBS = $(XENSTOREDLIBS) -oxenstored_OBJS = $(OBJS) +# use ocamldep to figure out link order, otherwise the Makefile would have +# to be continously adjusted for security patches that introduce new +# dependencies between files +oxenstored_MLSORTED = $(shell $(OCAMLDEP) -sort $(OBJS:=.ml)) +oxenstored_OBJS = $(oxenstored_MLSORTED:.ml=) OCAML_PROGRAM = oxenstored
Signature · 1 line
2.34.1
Re: [PATCH for-4.17 v3 04/15] tools/ocaml/xenstored/Makefile: use ocamldep -sort for linking order
On 8 Nov 2022, at 15:33, Edwin Török <[email protected]<mailto:[email protected]>> wrote:
A better solution is being worked on for master,
but for now use ocamldep -sort instead of a manually established link order.
The manually established link order will be wrong when (security) patches introduce new dependencies between files that would require changing the link order.
If dune was used as a build system this wouldn't be a problem, but we can't use Dune yet due to OSSTest, which is stuck on Debian oldstable.
No functional change.
---
Reason for inclusion in 4.17:
Avoids having to put this patch as a prerequisite into a security update. Earlier versions of XSA-326 needed this, and although latest version didn't we don't know whether it might be needed again in the future or not.
Changes since v2:
- new in v3 (was previously emailed to security team though)
---tools/ocaml/xenstored/Makefile | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
PATCH v3 5/15 tools/ocaml/Makefile.rules: do not run ocamldep on distclean
In reply to Edwin Török---
Reason for inclusion in 4.17:
- addresses review comments received post commit
Changes since v2:
- add Fixes line
- add 4.17 reason
--- tools/ocaml/Makefile.rules | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/ocaml/Makefile.rules +1 -1
@@ -44,7 +44,7 @@ META: META.in ALL_OCAML_OBJ_SOURCES=$(addsuffix .ml, $(ALL_OCAML_OBJS)) -ifneq ($(MAKECMDGOALS),clean) +ifeq (,$(findstring clean,$(MAKECMDGOALS))) .ocamldep.make: $(ALL_OCAML_OBJ_SOURCES) Makefile $(OCAML_TOPLEVEL)/Makefile.rules $(call quiet-command, $(OCAMLDEP) $(ALL_OCAML_OBJ_SOURCES) *.mli $o,MLDEP,) endif
Signature · 1 line
2.34.1
Re: [PATCH for-4.17 v3 05/15] tools/ocaml/Makefile.rules: do not run ocamldep on distclean
On 8 Nov 2022, at 15:33, Edwin Török <[email protected]<mailto:[email protected]>> wrote:
---
Reason for inclusion in 4.17:
- addresses review comments received post commit
Changes since v2:
- add Fixes line
- add 4.17 reason
---tools/ocaml/Makefile.rules | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
PATCH v3 6/15 tools/ocaml/Makefile.rules: hide -include on *clean
In reply to Edwin Török---
Reason for inclusion in 4.17:
- addresses review comments received post commit
Changes since v2:
- add Fixes line
- add 4.17 reason
--- tools/ocaml/Makefile.rules | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tools/ocaml/Makefile.rules +2 -2
@@ -44,10 +44,8 @@ META: META.in ALL_OCAML_OBJ_SOURCES=$(addsuffix .ml, $(ALL_OCAML_OBJS)) -ifeq (,$(findstring clean,$(MAKECMDGOALS))) .ocamldep.make: $(ALL_OCAML_OBJ_SOURCES) Makefile $(OCAML_TOPLEVEL)/Makefile.rules $(call quiet-command, $(OCAMLDEP) $(ALL_OCAML_OBJ_SOURCES) *.mli $o,MLDEP,) -endif clean: $(CLEAN_HOOKS) $(Q)rm -f .*.d *.o *.so *.a *.cmo *.cmi *.cma *.cmx *.cmxa *.annot *.spot *.spit $(LIBS) $(PROGRAMS) $(GENERATED_FILES) .ocamldep.make META
@@ -94,7 +92,9 @@ define C_PROGRAM_template $(call quiet-command, $(CC) $(LDFLAGS) -o $$@ $$+,BIN,$$@) endef +ifeq (,$(findstring clean,$(MAKECMDGOALS))) -include .ocamldep.make +endif $(foreach lib,$(OCAML_LIBRARY),$(eval $(call OCAML_LIBRARY_template,$(lib)))) $(foreach lib,$(OCAML_NOC_LIBRARY),$(eval $(call OCAML_NOC_LIBRARY_template,$(lib))))
Signature · 1 line
2.34.1
Re: [PATCH for-4.17 v3 06/15] tools/ocaml/Makefile.rules: hide -include on *clean
On 8 Nov 2022, at 15:33, Edwin Török <[email protected]<mailto:[email protected]>> wrote:
---
Reason for inclusion in 4.17:
- addresses review comments received post commit
Changes since v2:
- add Fixes line
- add 4.17 reason
---tools/ocaml/Makefile.rules | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
PATCH v3 7/15 CODING_STYLE(tools/ocaml): add 'make format' and remove tabs
In reply to Edwin TörökSee CODING_STYLE: Xen uses spaces, not tabs.
* OCaml code:
Using ocp-indent for now to just make minimal modifications in tabs vs spaces and get the right indentation.
We can introduce ocamlformat later.
* C stubs:
just replace tabs with spaces now, using indent or clang-format would change code too much for 4.17.
This avoids perpetuating a formatting style that is inconsistent with the rest of Xen, and that makes preparing and submitting patches more difficult (OCaml indentation tools usually only support spaces, not tabs).
Contains a bugfix for abi-check script to handle the change in the amount of whitespace.
No functional change.
--
Reason for inclusion in 4.17:
- makes it easier to backport changes from master to 4.17
- avoid perpetuating a different coding style (I thought tabs were mandated by Xen, and was about to fix up my editor config to match when I realized Xen already mandates the use of spaces)
- should make submitting patches for OCaml easier (OCaml indentation tools know only about spaces, so I either can't use them, or have to manually adjust indentation every time I submit a patch)
- it can be verified that the only change here is the Makefile change for the new rule, 'git log -p -1 -w' should be otherwise empty
Changes since v2:
- new in v3
--- tools/ocaml/Makefile | 5 + tools/ocaml/libs/eventchn/xeneventchn_stubs.c | 170 +- tools/ocaml/libs/mmap/mmap_stubs.h | 4 +- tools/ocaml/libs/mmap/xenmmap.ml | 2 +- tools/ocaml/libs/mmap/xenmmap.mli | 4 +- tools/ocaml/libs/mmap/xenmmap_stubs.c | 112 +- tools/ocaml/libs/xb/op.ml | 76 +- tools/ocaml/libs/xb/packet.ml | 30 +- tools/ocaml/libs/xb/partial.ml | 48 +- tools/ocaml/libs/xb/xb.ml | 422 ++-- tools/ocaml/libs/xb/xb.mli | 106 +- tools/ocaml/libs/xb/xenbus_stubs.c | 50 +- tools/ocaml/libs/xb/xs_ring.ml | 28 +- tools/ocaml/libs/xb/xs_ring_stubs.c | 214 +- tools/ocaml/libs/xc/abi-check | 2 +- tools/ocaml/libs/xc/xenctrl.ml | 330 +-- tools/ocaml/libs/xc/xenctrl.mli | 12 +- tools/ocaml/libs/xc/xenctrl_stubs.c | 1424 ++++++------ tools/ocaml/libs/xentoollog/caml_xentoollog.h | 6 +- .../ocaml/libs/xentoollog/xentoollog_stubs.c | 196 +- tools/ocaml/libs/xl/xenlight_stubs.c | 2022 ++++++++--------- tools/ocaml/libs/xs/queueop.ml | 48 +- tools/ocaml/libs/xs/xs.ml | 220 +- tools/ocaml/libs/xs/xs.mli | 46 +- tools/ocaml/libs/xs/xsraw.ml | 300 +-- tools/ocaml/libs/xs/xst.ml | 76 +- tools/ocaml/libs/xs/xst.mli | 20 +- tools/ocaml/test/dmesg.ml | 26 +- tools/ocaml/test/list_domains.ml | 4 +- tools/ocaml/test/raise_exception.ml | 4 +- tools/ocaml/test/xtl.ml | 28 +- tools/ocaml/xenstored/config.ml | 156 +- tools/ocaml/xenstored/connection.ml | 594 ++--- tools/ocaml/xenstored/connections.ml | 304 +-- tools/ocaml/xenstored/define.ml | 6 +- tools/ocaml/xenstored/disk.ml | 218 +- tools/ocaml/xenstored/domain.ml | 100 +- tools/ocaml/xenstored/domains.ml | 310 +-- tools/ocaml/xenstored/event.ml | 4 +- tools/ocaml/xenstored/history.ml | 62 +- tools/ocaml/xenstored/logging.ml | 434 ++-- tools/ocaml/xenstored/packet.ml | 20 +- tools/ocaml/xenstored/parse_arg.ml | 106 +- tools/ocaml/xenstored/perms.ml | 216 +- tools/ocaml/xenstored/poll.ml | 68 +- tools/ocaml/xenstored/poll.mli | 4 +- tools/ocaml/xenstored/process.ml | 1212 +++++----- tools/ocaml/xenstored/quota.ml | 74 +- tools/ocaml/xenstored/select_stubs.c | 62 +- tools/ocaml/xenstored/stdext.ml | 190 +- tools/ocaml/xenstored/store.ml | 752 +++--- tools/ocaml/xenstored/symbol.ml | 2 +- tools/ocaml/xenstored/syslog.ml | 48 +- tools/ocaml/xenstored/syslog_stubs.c | 30 +- tools/ocaml/xenstored/systemd_stubs.c | 10 +- tools/ocaml/xenstored/transaction.ml | 352 +-- tools/ocaml/xenstored/trie.ml | 222 +- tools/ocaml/xenstored/trie.mli | 22 +- tools/ocaml/xenstored/utils.ml | 146 +- tools/ocaml/xenstored/xenstored.ml | 1018 ++++----- 60 files changed, 6391 insertions(+), 6386 deletions(-)
tools/ocaml/Makefile +5 -0
@@ -34,3 +34,8 @@ build-tools-oxenstored: $(MAKE) -s -C libs/xb $(MAKE) -s -C libs/xc $(MAKE) -C xenstored + +.PHONY: format +format: + git ls-files '*.ml' '*.mli' | xargs -n1 ocp-indent -i + git ls-files '*.c' '*.h' | xargs -n1 sed -ie 's/\t/ /g'
tools/ocaml/libs/eventchn/xeneventchn_stubs.c +85 -85
234 lines not on this page · raw message
tools/ocaml/libs/mmap/mmap_stubs.h +2 -2
@@ -26,8 +26,8 @@struct mmap_interface { -void *addr; -int len; +void *addr; +int len; };/* for compatibility with OCaml 4.02.3 */
tools/ocaml/libs/mmap/xenmmap.ml +1 -1
@@ -21,7 +21,7 @@ type mmap_map_flag = SHARED | PRIVATE (* mmap: fd -> prot_flag -> map_flag -> length -> offset -> interface *) external mmap: Unix.file_descr -> mmap_prot_flag -> mmap_map_flag - -> int -> int -> mmap_interface = "stub_mmap_init" + -> int -> int -> mmap_interface = "stub_mmap_init" external unmap: mmap_interface -> unit = "stub_mmap_final" (* read: interface -> start -> length -> data *) external read: mmap_interface -> int -> int -> string = "stub_mmap_read"
tools/ocaml/libs/mmap/xenmmap.mli +2 -2
@@ -19,10 +19,10 @@ type mmap_prot_flag = RDONLY | WRONLY | RDWR type mmap_map_flag = SHARED | PRIVATE external mmap : Unix.file_descr -> mmap_prot_flag -> mmap_map_flag -> int -> int - -> mmap_interface = "stub_mmap_init" + -> mmap_interface = "stub_mmap_init" external unmap : mmap_interface -> unit = "stub_mmap_final" external read : mmap_interface -> int -> int -> string = "stub_mmap_read" external write : mmap_interface -> string -> int -> int -> unit - = "stub_mmap_write" + = "stub_mmap_write" external getpagesize : unit -> int = "stub_mmap_getpagesize"
tools/ocaml/libs/mmap/xenmmap_stubs.c +56 -56
153 lines not on this page · raw message
tools/ocaml/libs/xb/op.ml +38 -38
88 lines not on this page · raw message
tools/ocaml/libs/xb/packet.ml +15 -15
@@ -15,12 +15,12 @@ *) type t = -{ - tid: int; - rid: int; - ty: Op.operation; - data: string; -} + { + tid: int; + rid: int; + ty: Op.operation; + data: string; + } exception Error of string exception DataError of string
@@ -30,21 +30,21 @@ external string_of_header: int -> int -> int -> int -> string = "stub_string_of_ let create tid rid ty data = { tid = tid; rid = rid; ty = ty; data = data; } let of_partialpkt ppkt = - create ppkt.Partial.tid ppkt.Partial.rid ppkt.Partial.ty (Buffer.contents ppkt.Partial.buf) + create ppkt.Partial.tid ppkt.Partial.rid ppkt.Partial.ty (Buffer.contents ppkt.Partial.buf) let to_string pkt = - let header = string_of_header pkt.tid pkt.rid (Op.to_cval pkt.ty) (String.length pkt.data) in - header ^ pkt.data + let header = string_of_header pkt.tid pkt.rid (Op.to_cval pkt.ty) (String.length pkt.data) in + header ^ pkt.data let unpack pkt = - pkt.tid, pkt.rid, pkt.ty, pkt.data + pkt.tid, pkt.rid, pkt.ty, pkt.data let get_tid pkt = pkt.tid let get_ty pkt = pkt.ty let get_data pkt = - let l = String.length pkt.data in - if l > 0 && pkt.data.[l - 1] = '\000' then - String.sub pkt.data 0 (l - 1) - else - pkt.data + let l = String.length pkt.data in + if l > 0 && pkt.data.[l - 1] = '\000' then + String.sub pkt.data 0 (l - 1) + else + pkt.data let get_rid pkt = pkt.rid
tools/ocaml/libs/xb/partial.ml +24 -24
63 lines not on this page · raw message
tools/ocaml/libs/xb/xb.ml +211 -211
524 lines not on this page · raw message
tools/ocaml/libs/xb/xb.mli +53 -53
111 lines not on this page · raw message
tools/ocaml/libs/xb/xenbus_stubs.c +25 -25
65 lines not on this page · raw message
tools/ocaml/libs/xb/xs_ring.ml +14 -14
@@ -15,14 +15,14 @@ *) module Server_feature = struct - type t = - | Reconnection + type t = + | Reconnection end module Server_features = Set.Make(struct - type t = Server_feature.t - let compare = compare -end) + type t = Server_feature.t + let compare = compare + end) external read: Xenmmap.mmap_interface -> bytes -> int -> int = "ml_interface_read" external write: Xenmmap.mmap_interface -> bytes -> int -> int = "ml_interface_write"
@@ -31,18 +31,18 @@ external _internal_set_server_features: Xenmmap.mmap_interface -> int -> unit = external _internal_get_server_features: Xenmmap.mmap_interface -> int = "ml_interface_get_server_features" [@@noalloc] let write_substring mmap buff len = - write mmap (Bytes.unsafe_of_string buff) len + write mmap (Bytes.unsafe_of_string buff) len let get_server_features mmap = - (* NB only one feature currently defined above *) - let x = _internal_get_server_features mmap in - if x = 0 - then Server_features.empty - else Server_features.singleton Server_feature.Reconnection + (* NB only one feature currently defined above *) + let x = _internal_get_server_features mmap in + if x = 0 + then Server_features.empty + else Server_features.singleton Server_feature.Reconnection let set_server_features mmap set = - (* NB only one feature currently defined above *) - let x = if set = Server_features.empty then 0 else 1 in - _internal_set_server_features mmap x + (* NB only one feature currently defined above *) + let x = if set = Server_features.empty then 0 else 1 in + _internal_set_server_features mmap x external close: Xenmmap.mmap_interface -> unit = "ml_interface_close" [@@noalloc]
tools/ocaml/libs/xb/xs_ring_stubs.c +107 -107
260 lines not on this page · raw message
tools/ocaml/libs/xc/abi-check +1 -1
@@ -39,7 +39,7 @@ while (<C_FILE>) { } } else { $cline++; - m{^\s+/\* \s+ ! \s+ (.*?) \s* \*/\s*$}x or + m{^\s*/\*\s+! \s+ (.*?) \s* \*/\s*$}x or die "at line $cline of annotation, did not expect $_ ?"; my @vals = split /\s+/, $1; if ($cline == 1 && !@vals) {
tools/ocaml/libs/xc/xenctrl.ml +165 -165
443 lines not on this page · raw message
tools/ocaml/libs/xc/xenctrl.mli +6 -6
@@ -187,11 +187,11 @@ external domain_getinfo : handle -> domid -> domaininfo external domain_get_vcpuinfo : handle -> int -> int -> vcpuinfo = "stub_xc_vcpu_getinfo" external domain_ioport_permission: handle -> domid -> int -> int -> bool -> unit - = "stub_xc_domain_ioport_permission" + = "stub_xc_domain_ioport_permission" external domain_iomem_permission: handle -> domid -> nativeint -> nativeint -> bool -> unit - = "stub_xc_domain_iomem_permission" + = "stub_xc_domain_iomem_permission" external domain_irq_permission: handle -> domid -> int -> bool -> unit - = "stub_xc_domain_irq_permission" + = "stub_xc_domain_irq_permission" external vcpu_affinity_set : handle -> domid -> int -> bool array -> unit = "stub_xc_vcpu_setaffinity" external vcpu_affinity_get : handle -> domid -> int -> bool array
@@ -226,11 +226,11 @@ external map_foreign_range : = "stub_map_foreign_range" external domain_assign_device: handle -> domid -> (int * int * int * int) -> unit - = "stub_xc_domain_assign_device" + = "stub_xc_domain_assign_device" external domain_deassign_device: handle -> domid -> (int * int * int * int) -> unit - = "stub_xc_domain_deassign_device" + = "stub_xc_domain_deassign_device" external domain_test_assign_device: handle -> domid -> (int * int * int * int) -> bool - = "stub_xc_domain_test_assign_device" + = "stub_xc_domain_test_assign_device" external version : handle -> version = "stub_xc_version_version" external version_compile_info : handle -> compile_info
tools/ocaml/libs/xc/xenctrl_stubs.c +712 -712
890 lines not on this page · raw message
tools/ocaml/libs/xentoollog/caml_xentoollog.h +3 -3
Lines of this file are not shown. View the raw message.
tools/ocaml/libs/xentoollog/xentoollog_stubs.c +98 -98
196 lines not on this page · raw message
tools/ocaml/libs/xl/xenlight_stubs.c +1011 -1011
2,022 lines not on this page · raw message
tools/ocaml/libs/xs/queueop.ml +24 -24
48 lines not on this page · raw message
tools/ocaml/libs/xs/xs.ml +110 -110
220 lines not on this page · raw message
tools/ocaml/libs/xs/xs.mli +23 -23
46 lines not on this page · raw message
tools/ocaml/libs/xs/xsraw.ml +150 -150
300 lines not on this page · raw message
tools/ocaml/libs/xs/xst.ml +38 -38
76 lines not on this page · raw message
tools/ocaml/xenstored/config.ml +78 -78
156 lines not on this page · raw message
tools/ocaml/xenstored/connection.ml +297 -297
594 lines not on this page · raw message
tools/ocaml/xenstored/connections.ml +152 -152
304 lines not on this page · raw message
tools/ocaml/xenstored/disk.ml +109 -109
218 lines not on this page · raw message
tools/ocaml/xenstored/domain.ml +50 -50
100 lines not on this page · raw message
tools/ocaml/xenstored/domains.ml +155 -155
310 lines not on this page · raw message
tools/ocaml/xenstored/history.ml +31 -31
62 lines not on this page · raw message
tools/ocaml/xenstored/logging.ml +217 -217
434 lines not on this page · raw message
tools/ocaml/xenstored/parse_arg.ml +53 -53
106 lines not on this page · raw message
tools/ocaml/xenstored/perms.ml +108 -108
216 lines not on this page · raw message
tools/ocaml/xenstored/poll.ml +34 -34
68 lines not on this page · raw message
tools/ocaml/xenstored/process.ml +606 -606
1,212 lines not on this page · raw message
tools/ocaml/xenstored/quota.ml +37 -37
74 lines not on this page · raw message
tools/ocaml/xenstored/select_stubs.c +31 -31
62 lines not on this page · raw message
tools/ocaml/xenstored/stdext.ml +95 -95
190 lines not on this page · raw message
tools/ocaml/xenstored/store.ml +376 -376
752 lines not on this page · raw message
tools/ocaml/xenstored/syslog.ml +24 -24
48 lines not on this page · raw message
tools/ocaml/xenstored/transaction.ml +176 -176
352 lines not on this page · raw message
tools/ocaml/xenstored/trie.ml +111 -111
222 lines not on this page · raw message
tools/ocaml/xenstored/utils.ml +73 -73
146 lines not on this page · raw message
tools/ocaml/xenstored/xenstored.ml +509 -509
1,018 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.34.1
Part of the diff is not shown here. View the raw message.
Re: [PATCH for-4.17 v3 07/15] CODING_STYLE(tools/ocaml): add 'make format' and remove tabs
Hi,
Quoted 20 lines · Edwin Török
See
CODING_STYLE: Xen uses spaces, not tabs.* OCaml code:
Using
ocp-indentfor now to just make minimal modifications in tabs vs spaces and get the right indentation.
We can introduceocamlformatlater.* C stubs:
just replace tabs with spaces now, using
indentorclang-formatwould change code too much for 4.17.This avoids perpetuating a formatting style that is inconsistent with the rest of Xen, and that makes preparing and submitting patches more difficult (OCaml indentation tools usually only support spaces, not tabs).
Contains a bugfix for
abi-checkscript to handle the change in the amount of whitespace.
Quoted 7 lines
No functional change.
Signed-off-by Edwin Török <[email protected]>--
Reason for inclusion in 4.17:
- makes it easier to backport changes from master to 4.17
Right, but you will have the problem when backporting to 4.16 and older. So the overhead will always be there for a couple of years.
Quoted 8 lines
- avoid perpetuating a different coding style (I thought tabs were mandated by Xen, and was about to fix up my editor config to match when I realized Xen already mandates the use of spaces)
- should make submitting patches for OCaml easier (OCaml indentation tools know only about spaces, so I either can't use them, or have to manually adjust indentation every time I submit a patch)
- it can be verified that the only change here is theMakefilechange for the new rule, 'git log -p -1 -w' should be otherwise empty
While I understand the goal and support, this seems to be a bit too late to do it in Xen 4.17 (we are only a couple of weeks away). At this stage of the release we should only do bug fix.
This is clearly only a comesmetic change and there I would argue this should be deferred to 4.18. That said the last call is from the RM.
Cheers,
Signature · 1 line
Julien Grall
Re: [PATCH for-4.17 v3 07/15] CODING_STYLE(tools/ocaml): add 'make format' and remove tabs
Quoted 26 lines · Julien Grall
Hi,
Quoted 14 lines · Edwin Török
See
CODING_STYLE: Xen uses spaces, not tabs.
* OCaml code:
Usingocp-indentfor now to just make minimal modifications in tabs vs spaces and get the right indentation.
We can introduceocamlformatlater.
* C stubs:
just replace tabs with spaces now, usingindentorclang-formatwould change code too much for 4.17.
This avoids perpetuating a formatting style that is inconsistent with the rest of Xen, and that makes preparing and submitting patches more difficult (OCaml indentation tools usually only support spaces, not tabs). Contains a bugfix forabi-checkscript to handle the change in the amount of whitespace.
No functional change.Quoted 4 lines
Signed-off-by Edwin Török <[email protected]>--
Reason for inclusion in 4.17:
- makes it easier to backport changes from master to 4.17
Quoted 1 line
Right, but you will have the problem when backporting to 4.16 and older. So the overhead will always be there for a couple of years.
There will always be more than one Xen release in support, which means we'd never be able to fix this. At some point we need to decide to go ahead and just do it (which may or may not be 4.17)
The whitespace fixup needs to happen somewhere anyway, there are a couple of options:
* use git rebase/cherry-pick -Xignore-space-change to avoid backport failing to apply
* do a reformat on 4.16 branch too (backport the couple of lines of Makefile change and run 'make format' and commit the result)
* include the reformat as a prerequisite of the next security patch (I'd prefer not to, hence including it in 4.17)
* do the whitespace fixup prior to submission, turning a correctly formatted patch with spaces into an incorrectly formatted one with tabs. I wouldn't have minded doing this if the coding style said tabs, I could've run my reformat tool locally and fix it up with 'sed' prior to submission. However this seems unnecessary overhead if doing this works actively against the existing coding style.
However if we put this into 4.17 and there are no new security issues discovered before 4.16 goes out of support then we'll be in a much better position for 4.18 onward. There is still quite a lot of work outstanding in testing of oxenstored (updating my fuzzer for one), so I can't really predict at this point whether there are more security bugs in oxenstored or not, so maybe we need to take this decision at 4.18 time? Not sure.
Quoted 11 lines
Quoted 9 lines
- avoid perpetuating a different coding style (I thought tabs were mandated by Xen, and was about to fix up my editor config to match when I realized Xen already mandates the use of spaces)
- should make submitting patches for OCaml easier (OCaml indentation tools know only about spaces, so I either can't use them, or have to manually adjust indentation every time I submit a patch)
- it can be verified that the only change here is theMakefilechange for the new rule, 'git log -p -1 -w' should be otherwise emptyWhile I understand the goal and support, this seems to be a bit too late to do it in Xen 4.17 (we are only a couple of weeks away). At this stage of the release we should only do bug fix.
I think it can be fairly easily proven that there is no functional change by rerunning the make format command manually, and by looking at the diff with ignore whitespace as suggested above. I understand the reluctance in including it (which is why I was not sure whether to post it in the first place), but I think it might be beneficial to do it. There is a large backlog of work in oxenstored that got piled up during the past couple of years of XSA work, and it'd be a lot easier to update and upstream those if we wouldn't have to worry about indentation at all.
Usually patches on LCM and security branches are avoided to reduce the risk of breaking anything, but a reindentation patch should not really break anything (well other than the abi-check script in the build, but I fixed that to accept both ways).
One alternative would be that I add another step after reformat that runs sed and turns spaces back into tabs for now, and that way I can still run 'make format' at each step while preparing patches for master, or 4.17 or security patches and get something consistent, and that minimizes other whitespace changes, but it wouldn't completely eliminate them (e.g. there are pieces of code that are just wrongly indented, so there'd be at least a diff to fix all that).
Quoted 2 lines
This is clearly only a comesmetic change and there I would argue this should be deferred to 4.18. That said the last call is from the RM.
Best regards,
--Edwin
Quoted 8 lines
Cheers,
Signature · 1 line
Julien Grall
Re: [PATCH for-4.17 v3 07/15] CODING_STYLE(tools/ocaml): add 'make format' and remove tabs
Quoted 26 lines · Edwin Torok
Quoted 26 lines · Julien Grall
Hi,
Quoted 18 lines · Edwin Török
See
CODING_STYLE: Xen uses spaces, not tabs.
* OCaml code:
Usingocp-indentfor now to just make minimal modifications in tabs vs spaces and get the right indentation.
We can introduceocamlformatlater.
* C stubs:
just replace tabs with spaces now, usingindentorclang-formatwould change code too much for 4.17.
This avoids perpetuating a formatting style that is inconsistent with the rest of Xen, and that makes preparing and submitting patches more difficult (OCaml indentation tools usually only support spaces, not tabs). Contains a bugfix forabi-checkscript to handle the change in the amount of whitespace.
No functional change.Signed-off-by Edwin Török <[email protected]>--
Reason for inclusion in 4.17:
- makes it easier to backport changes from master to 4.17
Quoted 4 lines
Quoted 1 line
Right, but you will have the problem when backporting to 4.16 and older. So the overhead will always be there for a couple of years.
There will always be more than one Xen release in support, which means we'd never be able to fix this.
Note that I haven't said this should never be done. This just need to be correctly timed. Doing it in the middle of a deep freeze doesn't look right to me.
[...]
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
- avoid perpetuating a different coding style (I thought tabs were mandated by Xen, and was about to fix up my editor config to match when I realized Xen already mandates the use of spaces)
- should make submitting patches for OCaml easier (OCaml indentation tools know only about spaces, so I either can't use them, or have to manually adjust indentation every time I submit a patch)
- it can be verified that the only change here is theMakefilechange for the new rule, 'git log -p -1 -w' should be otherwise empty
Quoted 3 lines
Quoted 1 line
While I understand the goal and support, this seems to be a bit too late to do it in Xen 4.17 (we are only a couple of weeks away). At this stage of the release we should only do bug fix.
I think it can be fairly easily proven that there is no functional change by rerunning the make format command manually, and by looking at the diff with ignore whitespace as suggested above.
That's not really the point here. The point is that if we start to allow large coding style change (whether automatic or manual) very late in the release then it will be hard to reject it in the future.
In fact we already have guidelines for that. If you look at [1], only bug fixes should be done past the code freeze (23rd September).
As I wrote before, this patch only seem to be a cosmetic/quality improvement. IOW this is not a bug fix and would not qualify for 4.17.
Quoted 3 lines
I understand the reluctance in including it (which is why I was not sure whether to post it in the first place), but I think it might be beneficial to do it. There is a large backlog of work in oxenstored that got piled up during the past couple of years of XSA work, and it'd be a lot easier to update and upstream those if we wouldn't have to worry about indentation at all.
This is an argument for including this patch in Xen 4.18. As I wrote above, I am not against that.
Quoted 2 lines
Usually patches on LCM and security branches are avoided to reduce the risk of breaking anything, but a reindentation patch should not really break anything (well other than the abi-check script in the build, but I fixed that to accept both ways).
What does LCM stands for?
Quoted 2 lines
One alternative would be that I add another step after reformat that runs sed and turns spaces back into tabs for now, and that way I can still run 'make format' at each step while preparing patches for master, or 4.17 or security patches and get something consistent, and that minimizes other whitespace changes, but it wouldn't completely eliminate them (e.g. there are pieces of code that are just wrongly indented, so there'd be at least a diff to fix all that).
I would view this as a feature. Which again doesn't qualify for Xen 4.17 release. This doesn't mean the patch couldn't be backported afterwards.
Cheers,
[1] AS8PR08MB7991145C8063D6939AFFED8F92829@AS8PR08MB7991.eurprd08.prod.outlook.com
Quoted 14 lines
Best regards,
--EdwinQuoted 8 lines
Cheers,
Signature · 1 line
Julien Grall
Signature · 1 line
Julien Grall
Re: [PATCH for-4.17 v3 07/15] CODING_STYLE(tools/ocaml): add 'make format' and remove tabs
Quoted 60 lines · Julien Grall
Quoted 24 lines · Edwin Torok
Quoted 24 lines · Julien Grall
Hi,
Quoted 18 lines · Edwin Török
See
CODING_STYLE: Xen uses spaces, not tabs.
* OCaml code:
Usingocp-indentfor now to just make minimal modifications in tabs vs spaces and get the right indentation.
We can introduceocamlformatlater.
* C stubs:
just replace tabs with spaces now, usingindentorclang-formatwould change code too much for 4.17.
This avoids perpetuating a formatting style that is inconsistent with the rest of Xen, and that makes preparing and submitting patches more difficult (OCaml indentation tools usually only support spaces, not tabs). Contains a bugfix forabi-checkscript to handle the change in the amount of whitespace.
No functional change.Signed-off-by Edwin Török <[email protected]>--
Reason for inclusion in 4.17:
- makes it easier to backport changes from master to 4.17Quoted 2 lines
Quoted 1 line
Right, but you will have the problem when backporting to 4.16 and older. So the overhead will always be there for a couple of years.
There will always be more than one Xen release in support, which means we'd never be able to fix this.
Note that I haven't said this should never be done. This just need to be correctly timed. Doing it in the middle of a deep freeze doesn't look right to me.
[...]
Quoted 11 lines
Quoted 10 lines
Quoted 8 lines
- avoid perpetuating a different coding style (I thought tabs were mandated by Xen, and was about to fix up my editor config to match when I realized Xen already mandates the use of spaces)
- should make submitting patches for OCaml easier (OCaml indentation tools know only about spaces, so I either can't use them, or have to manually adjust indentation every time I submit a patch)
- it can be verified that the only change here is theMakefilechange for the new rule, 'git log -p -1 -w' should be otherwise emptyWhile I understand the goal and support, this seems to be a bit too late to do it in Xen 4.17 (we are only a couple of weeks away). At this stage of the release we should only do bug fix.
I think it can be fairly easily proven that there is no functional change by rerunning the make format command manually, and by looking at the diff with ignore whitespace as suggested above.
That's not really the point here. The point is that if we start to allow large coding style change (whether automatic or manual) very late in the release then it will be hard to reject it in the future.
In fact we already have guidelines for that. If you look at [1], only bug fixes should be done past the code freeze (23rd September).
As I wrote before, this patch only seem to be a cosmetic/quality improvement. IOW this is not a bug fix and would not qualify for 4.17.
Quoted 3 lines
I understand the reluctance in including it (which is why I was not sure whether to post it in the first place), but I think it might be beneficial to do it. There is a large backlog of work in oxenstored that got piled up during the past couple of years of XSA work, and it'd be a lot easier to update and upstream those if we wouldn't have to worry about indentation at all.
Quoted 5 lines
This is an argument for including this patch in Xen 4.18. As I wrote above, I am not against that.
Quoted 1 line
Usually patches on LCM and security branches are avoided to reduce the risk of breaking anything, but a reindentation patch should not really break anything (well other than the abi-check script in the build, but I fixed that to accept both ways).
What does LCM stands for?
LCM is what we internally call the long term release branch where we backport fixes (i.e. 4.16 is an LCM branch after it is released). I think it'd be equivalent to the stable branch of Xen/Linux, I should've used that terminology. (I think it is an abbreviation of software lifecycle management)
Quoted 4 lines
Quoted 2 lines
One alternative would be that I add another step after reformat that runs sed and turns spaces back into tabs for now, and that way I can still run 'make format' at each step while preparing patches for master, or 4.17 or security patches and get something consistent, and that minimizes other whitespace changes, but it wouldn't completely eliminate them (e.g. there are pieces of code that are just wrongly indented, so there'd be at least a diff to fix all that).
I would view this as a feature. Which again doesn't qualify for Xen 4.17 release. This doesn't mean the patch couldn't be backported afterwards.
Ah ok, if this can be backported to 4.17.1, or applied for 4.18 that might work too. (I just thought the rules around backport would be even stricter than what can go into a release) I'll try to find a short term workaround for the spaces vs tabs issues meanwhile.
Quoted 4 lines
Cheers,
[1] AS8PR08MB7991145C8063D6939AFFED8F92829@AS8PR08MB7991.eurprd08.prod.outlook.com
Hmm I thought that is my Outlook rewriting the link, but the archive at lore.kernel.org seems to have this mangled URL as well which I cannot open. Could you send it in such a way that it is not encoded when being sent (e.g. base64 encode it...)
Quoted 12 lines
Quoted 9 lines
Best regards,
--EdwinQuoted 6 lines
Cheers,
Signature · 1 line
Julien GrallSignature · 1 line
Julien Grall
RE: [PATCH for-4.17 v3 07/15] CODING_STYLE(tools/ocaml): add 'make format' and remove tabs
Hi Edwin,
Quoted 10 lines
-----Original Message-----
Quoted 1 line
[1]
[email protected] .prod.outlook.com
Hmm I thought that is my Outlook rewriting the link, but the archive at lore.kernel.org seems to have this mangled URL as well which I cannot open. Could you send it in such a way that it is not encoded when being sent (e.g. base64 encode it...)
Appending "https://lore.kernel.org/xen-devel/" before the link from Julien works for me. But I think the link is just the release schedule :)
Kind regards,
Henry
Re: [PATCH for-4.17 v3 07/15] CODING_STYLE(tools/ocaml): add 'make format' and remove tabs
Quoted 12 lines · Henry Wang
Hi Edwin,
Quoted 6 lines
-----Original Message-----
Quoted 1 line
[1]
[email protected] .prod.outlook.com
Quoted 7 lines
Quoted 4 lines
Hmm I thought that is my Outlook rewriting the link, but the archive at lore.kernel.org seems to have this mangled URL as well which I cannot open. Could you send it in such a way that it is not encoded when being sent (e.g. base64 encode it...)
Appending "https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.kernel.org%2Fxen-devel%2F&data=05%7C01%7Cedvin.torok%40citrix.com%7C1c7455639df84f970eab08dac2012fce%7C335836de42ef43a2b145348c2ee9ca5b%7C0%7C0%7C638035607613905983%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=l0oQ0815ndwWt4buGasPYYmVV3FdnrC%2F4wKrIoLqki8%3D&reserved=0" before the link from Julien works for me. But I think the link is just the release schedule :)
Thanks, this link works indeed (it is an Outlook message ID after all, not one of Outlook's link rewrites):
https://lore.kernel.org/xen-devel/AS8PR08MB7991145C8063D6939AFFED8F92829@AS8PR08MB7991.eurprd08.prod.outlook.com/
Best regards,
--Edwin
RE: [PATCH for-4.17 v3 07/15] CODING_STYLE(tools/ocaml): add 'make format' and remove tabs
In reply to Julien GrallHi Julien,
Quoted 10 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Subject: Re: [PATCH for-4.17 v3 07/15]CODING_STYLE(tools/ocaml): add 'make format' and remove tabs
While I understand the goal and support, this seems to be a bit too late to do it in Xen 4.17 (we are only a couple of weeks away). At this stage of the release we should only do bug fix.This is clearly only a comesmetic change and there I would argue this should be deferred to 4.18. That said the last call is from the RM.
I agree with your point. I think maybe defer the patch to 4.18 is better, given the deep freeze state we are currently in.
Kind regards,
Henry
Quoted 5 lines
Cheers,
--
Julien Grall
Re: [PATCH for-4.17 v3 07/15] CODING_STYLE(tools/ocaml): add 'make format' and remove tabs
Quoted 14 lines · Henry Wang
Hi Julien,
Quoted 8 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Subject: Re: [PATCH for-4.17 v3 07/15]CODING_STYLE(tools/ocaml): add 'make format' and remove tabs
While I understand the goal and support, this seems to be a bit too late to do it in Xen 4.17 (we are only a couple of weeks away). At this stage of the release we should only do bug fix.
Quoted 5 lines
Quoted 2 lines
This is clearly only a comesmetic change and there I would argue this should be deferred to 4.18. That said the last call is from the RM.
I agree with your point. I think maybe defer the patch to 4.18 is better, given the deep freeze state we are currently in.
On the other hand here is a bug I just spotted when looking at indentation changes (or rather reading the code after the indentation change), and there are probably more:
```
if not (Domain.get_io_credit dom > 0) then
debug "Looking up domid %d" (Domain.get_id dom);
- let con = Connections.find_domain cons (Domain.get_id dom) in
- if not (Connection.has_more_work con) then (
- Process.do_output store cons domains con;
- Process.do_input store cons domains con;
- if Connection.has_more_work con then
- (* Previously thought as no work, but detect some after scan (as
- processing a new message involves multiple steps.) It's very
- likely to be a "lazy" client, bump its credit. It could be false
- positive though (due to time window), but it's no harm to give a
- domain extra credit. *)
- let n = 32 + 2 * (Domains.number domains) in
- info "found lazy domain %d, credit %d" (Domain.get_id dom) n;
- Domain.set_io_credit ~n dom
- ) in
+ let con = Connections.find_domain cons (Domain.get_id dom) in
+ if not (Connection.has_more_work con) then (
+ Process.do_output store cons domains con;
+ Process.do_input store cons domains con;
+ if Connection.has_more_work con then
+ (* Previously thought as no work, but detect some after scan (as
+ processing a new message involves multiple steps.) It's very
+ likely to be a "lazy" client, bump its credit. It could be false
+ positive though (due to time window), but it's no harm to give a
+ domain extra credit. *)
+ let n = 32 + 2 * (Domains.number domains) in
+ info "found lazy domain %d, credit %d" (Domain.get_id dom) n;
+ Domain.set_io_credit ~n dom
+ ) in
```Notice how all that code "seems" to be inside the if unless you read really closely, but in fact it isn't, just the debug statement is.
Which means whenever I reviewed this code (to look for performance or security bugs) I've been reading it wrong the same way the original author got it wrong when indenting it. In this case the original author being me, as I've introduced this bug in 42f0581a91d4340ae66768a29fd779f83415bdfe back in 2021, where prior to the change in that commit indentation was correct, but the patch added the 'debug' line in the wrong place (before the let instead of after it, and had I had my usual tools available to indent the file correctly this problem would've been detected and corrected before commiting the bug into the codebase...
And was probably a side-effect of trying not to reindent the code to reduce the patch size for the security fix, and by doing so introducing an actual functional bug )
(And I've recently fixed a similar bug elsewhere in XAPI, in which case I wasn't the original author of such a bug)
Indentation can't really be trusted to humans :)
(It means that even if a domain already has IO credit we still scan its ring for more work)
So some indentation changes will probably come in as bugfixes for 4.17.1 (well maybe not reindenting the whole file, just the problematic region of code/function).
Best regards,
--Edwin
Re: [PATCH for-4.17 v3 07/15] CODING_STYLE(tools/ocaml): add 'make format' and remove tabs
Quoted 18 lines · Edwin Torok
Quoted 15 lines · Henry Wang
Hi Julien,
Quoted 8 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Subject: Re: [PATCH for-4.17 v3 07/15]CODING_STYLE(tools/ocaml): add 'make format' and remove tabs
While I understand the goal and support, this seems to be a bit too late to do it in Xen 4.17 (we are only a couple of weeks away). At this stage of the release we should only do bug fix.
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
This is clearly only a comesmetic change and there I would argue this should be deferred to 4.18. That said the last call is from the RM.
I agree with your point. I think maybe defer the patch to 4.18 is better, given the deep freeze state we are currently in.
Indentation can't really be trusted to humans :)
It might be better to consider oxenstored unsupported in 4.17 at this point and try again for 4.17.1 or 4.18, it is probably too late to fix up all the bugs that I keep finding in it in a way in which it can be (security) supported, although I thought the goal of a release candidate is to try and find bugs and fix them before release.
But doing that while also trying to keep whitespace and indentation changes out of the patches (or trying to keep the patches small), is more likely to introduce more bugs into it at this point rather than fix things.
I was not able to do or send any of these patches sooner because the XSA-326 and XSA-115/etc. work in the past years has prevented any other kind of work from being sent (it'd have made rebasing the XSA series more difficult, and/or reveal the XSA as part of various other changes and commits), it is unfortunate that there is a release quite so close after an oxenstored XSA
I tried fixing up the rest of the series to not depend on this patch, but without an actual 'make format' rule to give it consistent indentation
it is just too error-prone or risky to fix it up at this point (I don't really mind whether indentation is tabs vs spaces, it is the inconsistency and non-automated nature of it that is the problem),
e.g. it is quite easy to accidentally duplicate code, or get the code in the wrong scope, or introduce bugs like the one I just mentioned before when a new statement gets inserted and the code seemingly looks ok but its semantics is entirely changed (and that is hidden by the inconsistent/non-automated indentation).
Perhaps by trying to merge some of the changes/fixes from https://github.com/mirage/ocaml-xenstore and getting that production ready, which is a much better codebase to start from than
the current one in the Xen tree.:
* it has actual unit tests (which I tried to introduce as part of a security fix for the intree version of oxenstored but got repeatedly discouraged from including it to not make the security fix too large)
* it was not vulnerable XSA-353
* it has fixed part of XSA-326 together with a unit test nearly 10 years before Xen project rediscovered the same bug and realized it is a security bug in the in-tree version: https://github.com/mirage/ocaml-xenstore/commit/21e96654c27c01cf52e5d7aabc5ee53e07f2cbb7
* (of course mirage version has never been used in production so it is entirely likely it has *different* bugs)
* in-tree Xen is difficult to contribute to:
* it has a broken build system that keeps throwing 'inconsistent assumptions'
* it has inconsistent and as we can see sometimes wrong indentation
* patches get posted to the mailing list and sometimes lost (e.g. https://patchwork.kernel.org/project/xen-devel/list/?series=339731&archive=both is still not committed), and I'm fairly sure I've seen an ack somewhere, but patchwork can't find it now
So I think oxenstored in 4.18+ will likely be sufficiently different than 4.17 that direct backports won't be possible anyway (indentation changes or not), so having this indentation patch in 4.17 wouldn't really help much. The disadvantage is that we lose all the bugfixes in the patch series after the indentation change, but if we consider oxenstored unsupported in 4.17 that may not matter.
I can resend this patch series for master without a 'for-4.17 tag' and keep doing development there.
Best regards,
--Edwin
Re: [PATCH for-4.17 v3 07/15] CODING_STYLE(tools/ocaml): add 'make format' and remove tabs
Quoted 29 lines · Edwin Török
Quoted 26 lines · Edwin Torok
Quoted 20 lines · Henry Wang
Hi Julien,
Quoted 10 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Subject: Re: [PATCH for-4.17 v3 07/15]CODING_STYLE(tools/ocaml): add 'make format' and remove tabs
While I understand the goal and support, this seems to be a bit too late to do it in Xen 4.17 (we are only a couple of weeks away). At this stage of the release we should only do bug fix.This is clearly only a comesmetic change and there I would argue this should be deferred to 4.18. That said the last call is from the RM.
I agree with your point. I think maybe defer the patch to 4.18 is better, given the deep freeze state we are currently in.
Quoted 5 lines
Quoted 2 lines
Indentation can't really be trusted to humans :)
It might be better to consider oxenstored unsupported in 4.17 at this point and try again for 4.17.1 or 4.18
Ah I see that 'Live Update' for oxenstored is already marked as 'not functional' (I indeed remember seeing a patch to that effect), so looks like nothing else needs to be done here for 4.17.
https://xenbits.xen.org/docs/unstable/support-matrix.html
I can try fixing that up for master, 4.17.1 or 4.18, whenever that opens.
Best regards,
--Edwin
Re: [PATCH for-4.17 v3 07/15] CODING_STYLE(tools/ocaml): add 'make format' and remove tabs
In reply to Henry WangQuoted 13 lines · Henry Wang
Quoted 10 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Subject: Re: [PATCH for-4.17 v3 07/15]CODING_STYLE(tools/ocaml): add 'make format' and remove tabs
While I understand the goal and support, this seems to be a bit too late to do it in Xen 4.17 (we are only a couple of weeks away). At this stage of the release we should only do bug fix.
Quoted 5 lines
Quoted 2 lines
This is clearly only a comesmetic change and there I would argue this should be deferred to 4.18. That said the last call is from the RM.
I agree with your point. I think maybe defer the patch to 4.18 is better, given the deep freeze state we are currently in.
I disagree. This is an automated change that can be verified to not add functional changes. Edvin has demonstrated that wrong indentation has mislead reviewers in the past and caused bugs. Nobody except Edvin has contributed to the affected code in years and thus it is not a burden on the project outside the OCaml part. I suggest to accept this.
— C
RE: [PATCH for-4.17 v3 07/15] CODING_STYLE(tools/ocaml): add 'make format' and remove tabs
Hi Christian,
Quoted 11 lines
-----Original Message-----
From: Christian Lindig <[email protected]>
Subject: Re: [PATCH for-4.17 v3 07/15]CODING_STYLE(tools/ocaml): add 'make format' and remove tabsQuoted 7 lines
Quoted 6 lines
While I understand the goal and support, this seems to be a bit too late to do it in Xen 4.17 (we are only a couple of weeks away). At this stage of the release we should only do bug fix.
This is clearly only a comesmetic change and there I would argue this should be deferred to 4.18. That said the last call is from the RM.
Quoted 8 lines
Quoted 2 lines
I agree with your point. I think maybe defer the patch to 4.18 is better, given the deep freeze state we are currently in.
I disagree. This is an automated change that can be verified to not add functional changes. Edvin has demonstrated that wrong indentation has mislead reviewers in the past and caused bugs. Nobody except Edvin has contributed to the affected code in years and thus it is not a burden on the project outside the OCaml part. I suggest to accept this.
I understand points from you, Edwin and Julien, but I think in the earlier discussion in this thread, Julien has provided an argument [1] which I do think is a valid reason to defer this patch a little bit.
But since you are the only maintainer of the Ocaml code, so if you strongly insist this patch should be included for the release and there would not be any more explicit objections from others in the next couple of days, I think I will provide my release-ack for the purpose of respecting opinions from the maintainer. Hope this solution should be acceptable to you.
[1] https://lore.kernel.org/xen-devel/[email protected]/
Kind regards,
Henry
Quoted 5 lines
— C
Re: [PATCH for-4.17 v3 07/15] CODING_STYLE(tools/ocaml): add 'make format' and remove tabs
Quoted 30 lines · Henry Wang
Hi Christian,
Quoted 11 lines
-----Original Message-----
From: Christian Lindig <[email protected]>
Subject: Re: [PATCH for-4.17 v3 07/15]CODING_STYLE(tools/ocaml): add 'make format' and remove tabsQuoted 7 lines
Quoted 6 lines
While I understand the goal and support, this seems to be a bit too late to do it in Xen 4.17 (we are only a couple of weeks away). At this stage of the release we should only do bug fix.
This is clearly only a comesmetic change and there I would argue this should be deferred to 4.18. That said the last call is from the RM.
Quoted 8 lines
Quoted 2 lines
I agree with your point. I think maybe defer the patch to 4.18 is better, given the deep freeze state we are currently in.
I disagree. This is an automated change that can be verified to not add functional changes. Edvin has demonstrated that wrong indentation has mislead reviewers in the past and caused bugs. Nobody except Edvin has contributed to the affected code in years and thus it is not a burden on the project outside the OCaml part. I suggest to accept this.
I understand points from you, Edwin and Julien, but I think in the earlier discussion in this thread, Julien has provided an argument [1] which I do think is a valid reason to defer this patch a little bit.
Quoted 5 lines
But since you are the only maintainer of the Ocaml code, so if you strongly insist this patch should be included for the release and there would not be any more explicit objections from others in the next couple of days, I think I will provide my release-ack for the purpose of respecting opinions from the maintainer. Hope this solution should be acceptable to you.
Thanks Henry. I think the argument here is the balance between maintaining a policy against late large changes and improving the quality and the reliability of future patches by using more automation. I agree that large functional changes and any change that can’t be verified should be avoided but I don’t think this case is one. However, I am fine deferring the patch based on an agreed policy if we can make it a priority to get in in soon. For me this is part of improving the OCaml code base and project quality by using more automation in formatting and the build system that lowers the barrier for contributors such that they don’t have to worry about procedural aspects like tabs, spaces, indentation, or build systems.
— C
RE: [PATCH for-4.17 v3 07/15] CODING_STYLE(tools/ocaml): add 'make format' and remove tabs
Hi Christian,
Quoted 39 lines
-----Original Message-----
From: Christian Lindig <[email protected]>
Subject: Re: [PATCH for-4.17 v3 07/15]CODING_STYLE(tools/ocaml): add 'make format' and remove tabsQuoted 22 lines · Henry Wang
Hi Christian,
Quoted 18 lines
-----Original Message-----
From: Christian Lindig <[email protected]>
Subject: Re: [PATCH for-4.17 v3 07/15]CODING_STYLE(tools/ocaml): add 'make format' and remove tabsQuoted 9 lines
Quoted 6 lines
While I understand the goal and support, this seems to be a bit too late to do it in Xen 4.17 (we are only a couple of weeks away). At this stage of the release we should only do bug fix.
This is clearly only a comesmetic change and there I would argue this should be deferred to 4.18. That said the last call is from the RM.
I agree with your point. I think maybe defer the patch to 4.18 is better, given the deep freeze state we are currently in.
I disagree. This is an automated change that can be verified to not add functional changes. Edvin has demonstrated that wrong indentation has mislead reviewers in the past and caused bugs. Nobody except Edvin has contributed to the affected code in years and thus it is not a burden on
the
Quoted 11 lines
Quoted 1 line
project outside the OCaml part. I suggest to accept this.
I understand points from you, Edwin and Julien, but I think in the earlier discussion in this thread, Julien has provided an argument [1] which I do think is a valid reason to defer this patch a little bit.
But since you are the only maintainer of the Ocaml code, so if you strongly insist this patch should be included for the release and there would not be any more explicit objections from others in the next couple of days, I think I will provide my release-ack for the purpose of respecting opinions from the maintainer. Hope this solution should be acceptable to you.
Quoted 7 lines
Thanks Henry. I think the argument here is the balance between maintaining a policy against late large changes and improving the quality and the reliability of future patches by using more automation. I agree that large functional changes and any change that can’t be verified should be avoided but I don’t think this case is one. However,
I am fine deferring the patch based on an agreed policy if we can make it a priority to get in in soon.
Thanks for your understanding. I will take a note of this patch and try to ping committers to commit this patch as soon as the staging tree gets unfrozen after the release. In this way I think your concerns in...
Quoted 5 lines
For me this is part of improving the OCaml code
base and project quality by using more automation in formatting and the build system that lowers the barrier for contributors such that they don’t have to worry about procedural aspects like tabs, spaces, indentation, or build systems.
...here would be minimized.
I do understand your points and frustration. Thanks again for your understanding.
Kind regards,
Henry
Quoted 2 lines
— C
PATCH v3 8/15 tools/ocaml/libs/evtchn: add xenevtchn_fdopen bindings
In reply to Edwin Török---
Reason for inclusion in 4.17:
- needed for a bugfix in a followup commit
Changes since v2:
- new in v3
--- tools/ocaml/libs/eventchn/xeneventchn.ml | 6 +++- tools/ocaml/libs/eventchn/xeneventchn.mli | 9 +++++- tools/ocaml/libs/eventchn/xeneventchn_stubs.c | 28 +++++++++++++++++-- 3 files changed, 38 insertions(+), 5 deletions(-)
tools/ocaml/libs/eventchn/xeneventchn.ml +5 -1
@@ -16,7 +16,11 @@ type handle -external init: unit -> handle = "stub_eventchn_init" +external init_cloexec: bool -> handle = "stub_eventchn_init" + +let init ?(cloexec=true) () = init_cloexec cloexec + +external fdopen: Unix.file_descr -> handle = "stub_eventchn_fdopen" external fd: handle -> Unix.file_descr = "stub_eventchn_fd" type t = int
tools/ocaml/libs/eventchn/xeneventchn.mli +8 -1
@@ -43,7 +43,14 @@ val to_int: t -> int val of_int: int -> t -val init: unit -> handle +val init: ?cloexec:bool -> unit -> handle +(** [init ?cloexec ()] + Return an initialised event channel interface. + The default is to close the underlying file descriptor + on [execve], which can be overriden with [~cloexec:false]. + On error it will throw a Failure exception. *) + +val fdopen: Unix.file_descr -> handle (** Return an initialised event channel interface. On error it will throw a Failure exception. *)
tools/ocaml/libs/eventchn/xeneventchn_stubs.c +25 -3
@@ -58,14 +58,36 @@ static struct custom_operations xenevtchn_ops = {custom_compare_ext_default /* raises Failure */ }; -CAMLprim value stub_eventchn_init(void) +CAMLprim value stub_eventchn_init(value cloexec) { -CAMLparam0 (); +CAMLparam1 (cloexec);CAMLlocal1 (result);xenevtchn_handle *xce;caml_enter_blocking_section (); - xce =xenevtchn_open (NULL ,0 ); + xce =xenevtchn_open (NULL ,Bool_val (cloexec) ?0 :XENEVTCHN_NO_CLOEXEC ); +caml_leave_blocking_section (); + +if ( xce ==NULL ) +caml_failwith ("open failed" ); + +/* contains file descriptors, trigger full GC at least every 128 +* allocations +*/ + result =caml_alloc_custom (&xenevtchn_ops,sizeof (xce),0 ,1 ); +_H (result) = xce; +CAMLreturn (result); +} + +CAMLprim value stub_eventchn_fdopen(value fdval) +{ +CAMLparam1 (fdval); +CAMLlocal1 (result); +xenevtchn_handle *xce; + +caml_enter_blocking_section (); +/* having any flags here would raise EINVAL */ + xce =xenevtchn_fdopen (NULL ,Int_val (fdval),0 );caml_leave_blocking_section ();if (xce ==NULL )
Signature · 1 line
2.34.1
Re: [PATCH for-4.17 v3 08/15] tools/ocaml/libs/evtchn: add xenevtchn_fdopen bindings
On 8 Nov 2022, at 15:34, Edwin Török <[email protected]<mailto:[email protected]>> wrote:
---
Reason for inclusion in 4.17:
- needed for a bugfix in a followup commit
Changes since v2:
- new in v3
---
tools/ocaml/libs/eventchn/xeneventchn.ml | 6 +++- tools/ocaml/libs/eventchn/xeneventchn.mli | 9 +++++- tools/ocaml/libs/eventchn/xeneventchn_stubs.c | 28 +++++++++++++++++-- 3 files changed, 38 insertions(+), 5 deletions(-)
PATCH v3 9/15 tools/ocaml/xenstored/store.ml: fix build error
In reply to Edwin TörökBuilding with Dune in release mode fails with:
``` File "ocaml/xenstored/store.ml", line 464, characters 13-32: Warning 18: this type-based record disambiguation is not principal. File "ocaml/xenstored/store.ml", line 1: Error: Some fatal warnings were triggered (1 occurrences) ```
This is a warning to help keep the code futureproof, quoting from its documentation:
Quoted 7 lines
Check information path during type-checking, to make sure that all types are derived in a principal way. When using labelled arguments and/or polymorphic methods, this flag is required to ensure future versions of the compiler will be able to infer types correctly, even if internal algorithms change. All programs accepted in -principal mode are also accepted in the default mode with equivalent types, but different binary signatures, and this may slow down type checking; yet it is a good idea to use it once before publishing source code.
---
Reason for inclusion in 4.17:
- fixes a build error in a previous commit that is already in master
Changes since v2:
- new in v3
--- tools/ocaml/xenstored/store.ml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/ocaml/xenstored/store.ml +1 -1
@@ -461,7 +461,7 @@ let reset_permissions store domid = | Some perms -> if perms <> node.perms then Logging.debug "store|node" "Changed permissions for node %s" (Node.get_name node); - Some { node with perms } + Some { node with Node.perms } ) store.root type ops = {
Signature · 1 line
2.34.1
RE: [PATCH for-4.17 v3 09/15] tools/ocaml/xenstored/store.ml: fix build error
Hi Edwin,
Quoted 22 lines
-----Original Message-----
From: Edwin Török <[email protected]>
Subject: [PATCH for-4.17 v3 09/15]tools/ocaml/xenstored/store.ml: fix build errorBuilding with Dune in release mode fails with:
``` File "ocaml/xenstored/store.ml", line 464, characters 13-32: Warning 18: this type-based record disambiguation is not principal. File "ocaml/xenstored/store.ml", line 1: Error: Some fatal warnings were triggered (1 occurrences) ```This is a warning to help keep the code futureproof, quoting from its documentation:
Quoted 1 line
Check information path during type-checking, to make sure that all types
are
Quoted 1 line
derived in a principal way. When using labelled arguments and/or
polymorphic
Quoted 3 lines
methods, this flag is required to ensure future versions of the compiler will be able to infer types correctly, even if internal algorithms change. All programs accepted in -principal mode are also accepted in the default
Quoted 7 lines
mode with
Quoted 1 line
equivalent types, but different binary signatures, and this may slow down
type
Quoted 1 line
checking; yet it is a good idea to use it once before publishing source code.
Fixes db471408edd46 "tools/ocaml/xenstored: Fix quota bypass on domainshutdown"
Nit: The format of this "Fixes:" tag might need to be fixed?
Quoted 5 lines
Signed-off-by Edwin Török <[email protected]>---
Reason for inclusion in 4.17:
- fixes a build error in a previous commit that is already in master
Yes, given this is a simple enough patch:
Kind regards,
Henry
Re: [PATCH for-4.17 v3 09/15] tools/ocaml/xenstored/store.ml: fix build error
Quoted 33 lines · Henry Wang
Quoted 22 lines
-----Original Message-----
From: Edwin Török <[email protected]>
Subject: [PATCH for-4.17 v3 09/15]tools/ocaml/xenstored/store.ml: fix build errorBuilding with Dune in release mode fails with:
``` File "ocaml/xenstored/store.ml", line 464, characters 13-32: Warning 18: this type-based record disambiguation is not principal. File "ocaml/xenstored/store.ml", line 1: Error: Some fatal warnings were triggered (1 occurrences) ```This is a warning to help keep the code futureproof, quoting from its documentation:
Quoted 1 line
Check information path during type-checking, to make sure that all types
are
Quoted 1 line
derived in a principal way. When using labelled arguments and/or
polymorphic
Quoted 3 lines
methods, this flag is required to ensure future versions of the compiler will be able to infer types correctly, even if internal algorithms change. All programs accepted in -principal mode are also accepted in the default
Quoted 7 lines
mode with
Quoted 1 line
equivalent types, but different binary signatures, and this may slow down
type
Quoted 1 line
checking; yet it is a good idea to use it once before publishing source code.
Fixes db471408edd46 "tools/ocaml/xenstored: Fix quota bypass on domainshutdown"
Nit: The format of this "Fixes:" tag might need to be fixed?
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Signed-off-by Edwin Török <[email protected]>---
Reason for inclusion in 4.17:
- fixes a build error in a previous commit that is already in masterYes, given this is a simple enough patch:
Release-acked-by Henry Wang <[email protected]>
Afaics this patch was previously posted in isolation, and it was already release-acked. What's lacking there is a 2nd maintainer's ack or a proper R-b. When it now is patch 9 in a series, it isn't really obvious whether this could also be committed in isolation (it looks like it does, but a clear statement to this effect would have been beneficial).
Jan
Re: [PATCH for-4.17 v3 09/15] tools/ocaml/xenstored/store.ml: fix build error
Quoted 45 lines · Jan Beulich
Quoted 40 lines · Henry Wang
Quoted 22 lines
-----Original Message-----
From: Edwin Török <[email protected]>
Subject: [PATCH for-4.17 v3 09/15]tools/ocaml/xenstored/store.ml: fix build errorBuilding with Dune in release mode fails with:
``` File "ocaml/xenstored/store.ml", line 464, characters 13-32: Warning 18: this type-based record disambiguation is not principal. File "ocaml/xenstored/store.ml", line 1: Error: Some fatal warnings were triggered (1 occurrences) ```This is a warning to help keep the code futureproof, quoting from its documentation:
Quoted 1 line
Check information path during type-checking, to make sure that all types
are
Quoted 1 line
derived in a principal way. When using labelled arguments and/or
polymorphic
Quoted 3 lines
methods, this flag is required to ensure future versions of the compiler will be able to infer types correctly, even if internal algorithms change. All programs accepted in -principal mode are also accepted in the default
Quoted 7 lines
mode with
Quoted 1 line
equivalent types, but different binary signatures, and this may slow down
type
Quoted 1 line
checking; yet it is a good idea to use it once before publishing source code.
Fixes db471408edd46 "tools/ocaml/xenstored: Fix quota bypass on domainshutdown"
Nit: The format of this "Fixes:" tag might need to be fixed?
Quoted 5 lines
Signed-off-by Edwin Török <[email protected]>---
Reason for inclusion in 4.17:
- fixes a build error in a previous commit that is already in masterYes, given this is a simple enough patch:
Quoted 9 lines
Quoted 1 line
Release-acked-by Henry Wang <[email protected]>Afaics this patch was previously posted in isolation, and it was already release-acked. What's lacking there is a 2nd maintainer's ack or a proper R-b. When it now is patch 9 in a series, it isn't really obvious whether this could also be committed in isolation (it looks like it does, but a clear statement to this effect would have been beneficial).
You're right it already has both acks, it just hasn't been commited yet: https://patchwork.kernel.org/project/xen-devel/patch/5a453393dad1de8286fe5db16504d3db2906eef8.1667500970.git.edvin.torok@citrix.com/
I've added the acks now to my github branch, so next time I resend the series it should be there.
It can be applied independently, I've rebased and moved it at the beginning of the series (there was just some whitespace to fix up)
If it helps here is the commit in isolation that could be cherry-picked onto master:
https://github.com/edwintorok/xen/commit/da88b438e03da36212d07d24d67ab151ae287f4e
Best regards,
--Edwin
Re: [PATCH for-4.17 v3 09/15] tools/ocaml/xenstored/store.ml: fix build error
Quoted 56 lines · Edwin Torok
Quoted 45 lines · Jan Beulich
Quoted 40 lines · Henry Wang
Quoted 22 lines
-----Original Message-----
From: Edwin Török <[email protected]>
Subject: [PATCH for-4.17 v3 09/15]tools/ocaml/xenstored/store.ml: fix build errorBuilding with Dune in release mode fails with:
``` File "ocaml/xenstored/store.ml", line 464, characters 13-32: Warning 18: this type-based record disambiguation is not principal. File "ocaml/xenstored/store.ml", line 1: Error: Some fatal warnings were triggered (1 occurrences) ```This is a warning to help keep the code futureproof, quoting from its documentation:
Quoted 1 line
Check information path during type-checking, to make sure that all types
are
Quoted 1 line
derived in a principal way. When using labelled arguments and/or
polymorphic
Quoted 3 lines
methods, this flag is required to ensure future versions of the compiler will be able to infer types correctly, even if internal algorithms change. All programs accepted in -principal mode are also accepted in the default
Quoted 7 lines
mode with
Quoted 1 line
equivalent types, but different binary signatures, and this may slow down
type
Quoted 1 line
checking; yet it is a good idea to use it once before publishing source code.
Fixes db471408edd46 "tools/ocaml/xenstored: Fix quota bypass on domainshutdown"
Nit: The format of this "Fixes:" tag might need to be fixed?
Quoted 5 lines
Signed-off-by Edwin Török <[email protected]>---
Reason for inclusion in 4.17:
- fixes a build error in a previous commit that is already in masterYes, given this is a simple enough patch:
Quoted 9 lines
Quoted 1 line
Release-acked-by Henry Wang <[email protected]>Afaics this patch was previously posted in isolation, and it was already release-acked. What's lacking there is a 2nd maintainer's ack or a proper R-b. When it now is patch 9 in a series, it isn't really obvious whether this could also be committed in isolation (it looks like it does, but a clear statement to this effect would have been beneficial).
Quoted 1 line
You're right it already has both acks, it just hasn't been commited yet:
Oh, that's only because I overlooked Christian's ack. Will commit this now.
Jan
Re: [PATCH for-4.17 v3 09/15] tools/ocaml/xenstored/store.ml: fix build error
Quoted 57 lines · Jan Beulich
Quoted 56 lines · Edwin Torok
Quoted 54 lines · Jan Beulich
Quoted 33 lines · Henry Wang
Quoted 22 lines
-----Original Message-----
From: Edwin Török <[email protected]>
Subject: [PATCH for-4.17 v3 09/15]tools/ocaml/xenstored/store.ml: fix build errorBuilding with Dune in release mode fails with:
``` File "ocaml/xenstored/store.ml", line 464, characters 13-32: Warning 18: this type-based record disambiguation is not principal. File "ocaml/xenstored/store.ml", line 1: Error: Some fatal warnings were triggered (1 occurrences) ```This is a warning to help keep the code futureproof, quoting from its documentation:
Quoted 1 line
Check information path during type-checking, to make sure that all types
are
Quoted 1 line
derived in a principal way. When using labelled arguments and/or
polymorphic
Quoted 3 lines
methods, this flag is required to ensure future versions of the compiler will be able to infer types correctly, even if internal algorithms change. All programs accepted in -principal mode are also accepted in the default
Quoted 7 lines
mode with
Quoted 1 line
equivalent types, but different binary signatures, and this may slow down
type
Quoted 1 line
checking; yet it is a good idea to use it once before publishing source code.
Fixes db471408edd46 "tools/ocaml/xenstored: Fix quota bypass on domainshutdown"
Nit: The format of this "Fixes:" tag might need to be fixed?
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Signed-off-by Edwin Török <[email protected]>---
Reason for inclusion in 4.17:
- fixes a build error in a previous commit that is already in masterYes, given this is a simple enough patch:
Release-acked-by Henry Wang <[email protected]>Afaics this patch was previously posted in isolation, and it was already release-acked. What's lacking there is a 2nd maintainer's ack or a proper R-b. When it now is patch 9 in a series, it isn't really obvious whether this could also be committed in isolation (it looks like it does, but a clear statement to this effect would have been beneficial).
Quoted 3 lines
Quoted 1 line
You're right it already has both acks, it just hasn't been commited yet:
Oh, that's only because I overlooked Christian's ack. Will commit this now.
But, sigh, I had to fix up the patch: Even the one submitted standalone used space indentation when the file in the tree uses hard tabs. And even if I had wanted to pull from your github tree I would have had to fix up at least the Fixes: tag. So I ended up hand-editing indentation
...
Jan
Re: [PATCH for-4.17 v3 09/15] tools/ocaml/xenstored/store.ml: fix build error
Quoted 62 lines · Jan Beulich
Quoted 57 lines · Jan Beulich
Quoted 56 lines · Edwin Torok
Quoted 54 lines · Jan Beulich
Quoted 33 lines · Henry Wang
Quoted 22 lines
-----Original Message-----
From: Edwin Török <[email protected]>
Subject: [PATCH for-4.17 v3 09/15]tools/ocaml/xenstored/store.ml: fix build errorBuilding with Dune in release mode fails with:
``` File "ocaml/xenstored/store.ml", line 464, characters 13-32: Warning 18: this type-based record disambiguation is not principal. File "ocaml/xenstored/store.ml", line 1: Error: Some fatal warnings were triggered (1 occurrences) ```This is a warning to help keep the code futureproof, quoting from its documentation:
Quoted 1 line
Check information path during type-checking, to make sure that all types
are
Quoted 1 line
derived in a principal way. When using labelled arguments and/or
polymorphic
Quoted 3 lines
methods, this flag is required to ensure future versions of the compiler will be able to infer types correctly, even if internal algorithms change. All programs accepted in -principal mode are also accepted in the default
Quoted 7 lines
mode with
Quoted 1 line
equivalent types, but different binary signatures, and this may slow down
type
Quoted 1 line
checking; yet it is a good idea to use it once before publishing source code.
Fixes db471408edd46 "tools/ocaml/xenstored: Fix quota bypass on domainshutdown"
Nit: The format of this "Fixes:" tag might need to be fixed?
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Signed-off-by Edwin Török <[email protected]>---
Reason for inclusion in 4.17:
- fixes a build error in a previous commit that is already in masterYes, given this is a simple enough patch:
Release-acked-by Henry Wang <[email protected]>Afaics this patch was previously posted in isolation, and it was already release-acked. What's lacking there is a 2nd maintainer's ack or a proper R-b. When it now is patch 9 in a series, it isn't really obvious whether this could also be committed in isolation (it looks like it does, but a clear statement to this effect would have been beneficial).
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
You're right it already has both acks, it just hasn't been commited yet:
Oh, that's only because I overlooked Christian's ack. Will commit this now.
But, sigh, I had to fix up the patch: Even the one submitted standalone used space indentation when the file in the tree uses hard tabs. And even if I had wanted to pull from your github tree I would have had to fix up at least the Fixes: tag.
I thought I fixed it (the missing '('), but the format of the Fixes: line was not documented in
https://wiki.xenproject.org/wiki/Submitting_Xen_Project_Patches (which is the canonical resource I use when sending patches) and I just tried to guess the format based on the Fixes: entries I found in master (some of which have or less characters in the git hash)
I see there is a https://xenbits.xen.org/docs/unstable/process/sending-patches.html which has some more useful details,
including a way to automatically generate the Fixes: line using a git config, which is very useful and I'll use that in the future instead of crafting it by hand (which is what I've been doing so far).
I've edited the wiki to include this information now.
Quoted 1 line
So I ended up hand-editing indentation
Thanks,
--Edwin
Re: [PATCH for-4.17 v3 09/15] tools/ocaml/xenstored/store.ml: fix build error
In reply to Edwin TörökOn 8 Nov 2022, at 15:34, Edwin Török <[email protected]<mailto:[email protected]>> wrote:
Building with Dune in release mode fails with:
``` File "ocaml/xenstored/store.ml", line 464, characters 13-32: Warning 18: this type-based record disambiguation is not principal. File "ocaml/xenstored/store.ml", line 1: Error: Some fatal warnings were triggered (1 occurrences) ```
This is a warning to help keep the code futureproof, quoting from its documentation:
Check information path during type-checking, to make sure that all types are derived in a principal way. When using labelled arguments and/or polymorphic methods, this flag is required to ensure future versions of the compiler will be able to infer types correctly, even if internal algorithms change. All programs accepted in -principal mode are also accepted in the default mode with equivalent types, but different binary signatures, and this may slow down type checking; yet it is a good idea to use it once before publishing source code.
---
Reason for inclusion in 4.17:
- fixes a build error in a previous commit that is already in master
Changes since v2:
- new in v3
---tools/ocaml/xenstored/store.ml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
PATCH v3 10/15 tools/ocaml/xenstored: keep eventchn FD open across live update
In reply to Edwin TörökIt has been discovered that the Windows PV drivers rebind their local port of the xenstore evtchn post migrate,
which changes the eventchn port from e.g.
the default of 3 (nVCPUs+1) to 1 without oxenstored knowing about it. And oxenstored goes and tries to rebind port 3, which is already in use by something else.
The guest also writes to HVMPARAM, however there is a race condition between it resetting the eventchn port and writing to HVMPARAM, thus it is not possible to atomically obtain the guest's eventchn port for xenstore.
It is possible to recover such a guest manually from a failed live update by running introduceDomain from Dom0 with the correct port number obtain using 'xen-vmdebug <domid> hvmparam'.
There is also a small race condition window between closing the evtchn FD and reopening it where guest drivers would get an unbound evtchn port error from Xen when trying to talk to xenstore during live update.
Avoid this by trying to keep the eventchn file descriptor open across live updates, using the new xenevtchn_fdopen API, similar to how C xenstored works.
However an old enough oxenstored won't have kept the evtchn fd open and won't have saved it in the update stream, so handle that case by reopening the evtchn as a fallback.
A followup commit will avoid rebind the guest remote port (we cannot rely on the remote port staying the same, e.g. the windows PV drivers change it)
---
Reason for inclusion in 4.17:
- fixes live update in oxenstored, making future security updates easier if the base code already has this patch
Changes since v2:
- new in v3
--- tools/ocaml/xenstored/domains.ml | 1 + tools/ocaml/xenstored/event.ml | 7 ++- tools/ocaml/xenstored/xenstored.ml | 71 +++++++++++++++++++++--------- 3 files changed, 56 insertions(+), 23 deletions(-)
tools/ocaml/xenstored/domains.ml +1 -0
@@ -56,6 +56,7 @@ let exist doms id = Hashtbl.mem doms.table id let find doms id = Hashtbl.find doms.table id let number doms = Hashtbl.length doms.table let iter doms fct = Hashtbl.iter (fun _ b -> fct b) doms.table +let eventchn doms = doms.eventchn let rec is_empty_queue q = Queue.is_empty q ||
tools/ocaml/xenstored/event.ml +6 -1
@@ -20,7 +20,12 @@ type t = { mutable virq_port: Xeneventchn.t option; } -let init () = { handle = Xeneventchn.init (); virq_port = None; } +let init ?fd () = + let handle = match fd with + | None -> Xeneventchn.init ~cloexec:false () + | Some fd -> Xeneventchn.fdopen fd + in + { handle; virq_port = None } let fd eventchn = Xeneventchn.fd eventchn.handle let bind_dom_exc_virq eventchn = eventchn.virq_port <- Some (Xeneventchn.bind_dom_exc_virq eventchn.handle) let bind_interdomain eventchn domid port = Xeneventchn.bind_interdomain eventchn.handle domid port
tools/ocaml/xenstored/xenstored.ml +49 -22
171 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH for-4.17 v3 10/15] tools/ocaml/xenstored: keep eventchn FD open across live update
Quoted 22 lines · Edwin Török
Avoid this by trying to keep the eventchn file descriptor open across live updates, using the new
xenevtchn_fdopenAPI, similar to how C xenstored works.However an old enough oxenstored won't have kept the evtchn fd open and won't have saved it in the update stream, so handle that case by reopening the evtchn as a fallback.
A followup commit will avoid rebind the guest remote port (we cannot rely on the remote port staying the same, e.g. the windows PV drivers change it)
Signed-off-by Edwin Török <[email protected]>---
Reason for inclusion in 4.17:
- fixes live update in oxenstored, making future security updates easier if the base code already has this patch
Quoted 7 lines
Changes since v2:
- new in v3
---tools/ocaml/xenstored/domains.ml | 1 + tools/ocaml/xenstored/event.ml | 7 ++- tools/ocaml/xenstored/xenstored.ml | 71 +++++++++++++++++++++--------- 3 files changed, 56 insertions(+), 23 deletions(-)
tools/ocaml/xenstored/xenstored.ml module DB = struct · line 213
Quoted 13 lines
tools/ocaml/xenstored/xenstored.ml +13
-0+ let require_doms () = + match !doms with + | None -> + let missing_eventchnfd = !eventchnfd = None in + if missing_eventchnfd then + warn "No event channel file descriptor available in dump!"; + let eventchn = Event.init ?fd:!eventchnfd () in + let domains = createdoms eventchn in + if missing_eventchnfd then + Event.bind_dom_exc_virq eventchn; + doms := Some domains; + domains + | Some d -> d
I like to use the “when” clause in these situations in order to keep the match cases flat even at the cost of a slight code duplication. But it’s a matter of taste.
| None when !eventchnfd = None ->
Or
match !doms, !eventchnfd with
| None, None ->
|
PATCH v3 11/15 tools/ocaml/xenstored: do not rebind event channels after live update
In reply to Edwin TörökSee explanation in previous commit.
This introduces a new field into the live update stream to retain both ports, and handles the missing value in a backward compatible way.
---
Reason for inclusion 4.17:
- fixes a bug in oxenstored live update, needed to make live updates with future 4.17 security fixes possible/more reliable
Changes since v2:
- new in v3
--- tools/ocaml/libs/eventchn/xeneventchn.ml | 5 +++++ tools/ocaml/libs/eventchn/xeneventchn.mli | 5 +++++ tools/ocaml/xenstored/domain.ml | 6 +++++- tools/ocaml/xenstored/domains.ml | 13 +++++++++---- tools/ocaml/xenstored/event.ml | 1 + tools/ocaml/xenstored/xenstored.ml | 11 +++++++---- 6 files changed, 32 insertions(+), 9 deletions(-)
tools/ocaml/libs/eventchn/xeneventchn.ml +5 -0
@@ -43,6 +43,11 @@ type virq_t = external notify: handle -> int -> unit = "stub_eventchn_notify" external bind_interdomain: handle -> int -> int -> int = "stub_eventchn_bind_interdomain" + +let restore_interdomain handle _domid _remote_port local_port = + notify handle local_port; + local_port + external bind_virq: handle -> virq_t -> int = "stub_eventchn_bind_virq" let bind_dom_exc_virq handle = bind_virq handle Dom_exc external unbind: handle -> int -> unit = "stub_eventchn_unbind"
tools/ocaml/libs/eventchn/xeneventchn.mli +5 -0
@@ -68,6 +68,11 @@ val bind_interdomain : handle -> int -> int -> t channel connected to domid:remote_port. On error it will throw a Failure exception. *) +val restore_interdomain : handle -> int -> int -> int -> t +(** [restore_interdomain h domid remote_port local_port] returns a local event + channel connected to domid:remote_port. On error it will + throw a Failure exception. *) + val bind_dom_exc_virq : handle -> t (** Binds a local event channel to the VIRQ_DOM_EXC (domain exception VIRQ). On error it will throw a Failure
tools/ocaml/xenstored/domain.ml +5 -1
@@ -61,7 +61,7 @@ let string_of_port = function | Some x -> string_of_int (Xeneventchn.to_int x) let dump d chan = - fprintf chan "dom,%d,%nd,%d\n" d.id d.mfn d.remote_port + fprintf chan "dom,%d,%nd,%d,%s\n" d.id d.mfn d.remote_port (string_of_port d.port) let notify dom = match dom.port with | None ->
@@ -77,6 +77,10 @@ let bind_interdomain dom = dom.port <- Some (Event.bind_interdomain dom.eventchn dom.id dom.remote_port); debug "bound domain %d remote port %d to local port %s" dom.id dom.remote_port (string_of_port dom.port) +let restore_interdomain dom localport = + assert (dom.port = None); + dom.port <- Some (Event.restore_interdomain dom.eventchn dom.id dom.remote_port localport); + debug "restored interdomain %d remote port %d to local port %s" dom.id dom.remote_port (string_of_port dom.port) let close dom = debug "domain %d unbound port %s" dom.id (string_of_port dom.port);
tools/ocaml/xenstored/domains.ml +9 -4
@@ -123,17 +123,22 @@ let cleanup doms = let resume _doms _domid = () -let create doms domid mfn port = +let maybe_bind_interdomain restore_localport dom = + match restore_localport with + | None -> Domain.bind_interdomain dom + | Some p -> Domain.restore_interdomain dom p + +let create doms domid mfn ?restore_localport port = let interface = Xenctrl.map_foreign_range xc domid (Xenmmap.getpagesize()) mfn in let dom = Domain.make domid mfn port interface doms.eventchn in Hashtbl.add doms.table domid dom; - Domain.bind_interdomain dom; + maybe_bind_interdomain restore_localport dom; dom let xenstored_kva = ref "" let xenstored_port = ref "" -let create0 doms = +let create0 ?restore_localport doms = let port, interface = ( let port = Utils.read_file_single_integer !xenstored_port
@@ -147,7 +152,7 @@ let create0 doms = in let dom = Domain.make 0 Nativeint.zero port interface doms.eventchn in Hashtbl.add doms.table 0 dom; - Domain.bind_interdomain dom; + maybe_bind_interdomain restore_localport dom; Domain.notify dom; dom
tools/ocaml/xenstored/event.ml +1 -0
@@ -29,6 +29,7 @@ let init ?fd () = let fd eventchn = Xeneventchn.fd eventchn.handle let bind_dom_exc_virq eventchn = eventchn.virq_port <- Some (Xeneventchn.bind_dom_exc_virq eventchn.handle) let bind_interdomain eventchn domid port = Xeneventchn.bind_interdomain eventchn.handle domid port +let restore_interdomain eventchn domid port local_port = Xeneventchn.restore_interdomain eventchn.handle domid port local_port let unbind eventchn port = Xeneventchn.unbind eventchn.handle port let notify eventchn port = Xeneventchn.notify eventchn.handle port let pending eventchn = Xeneventchn.pending eventchn.handle
tools/ocaml/xenstored/xenstored.ml +7 -4
@@ -169,10 +169,13 @@ module DB = struct event_f ~eventfd | "socket" :: fd :: [] -> socket_f ~fd:(int_of_string fd) - | "dom" :: domid :: mfn :: port :: []-> + | "dom" :: domid :: mfn :: port :: rest -> domain_f (int_of_string domid) (Nativeint.of_string mfn) (int_of_string port) + (match rest with + | [] -> None (* backward compat: old version didn't have it *) + | localport :: _ -> Some (int_of_string localport)) | "watch" :: domid :: path :: token :: [] -> watch_f (int_of_string domid) (unhexify path) (unhexify token)
@@ -232,13 +235,13 @@ module DB = struct else warn "Ignoring invalid socket FD %d" fd in - let domain_f domid mfn port = + let domain_f domid mfn port restore_localport = let doms = require_doms () in let ndom = if domid > 0 then - Domains.create doms domid mfn port + Domains.create doms domid mfn ?restore_localport port else - Domains.create0 doms + Domains.create0 ?restore_localport doms in Connections.add_domain cons ndom; in
Signature · 1 line
2.34.1
PATCH v3 12/15 tools/ocaml/xenstored: log live update issues at warning level
In reply to Edwin TörökDuring live update oxenstored tries a best effort approach to recover as many domains and information as possible even if it encounters errors restoring some domains.
This defensive approach has already proven useful to recover domains after a bug in oxenstored.
However the default log level is warning, which means that problems during live update were not logged at all, unless you've changes oxenstored.conf prior to live-update.
---
Reason for inclusion in 4.17:
- makes live update issues debuggable (live update can be used to apply security updates to 4.17)
Changes since v2:
- new in v3
--- tools/ocaml/xenstored/xenstored.ml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tools/ocaml/xenstored/xenstored.ml +2 -2
@@ -184,9 +184,9 @@ module DB = struct (Perms.Node.of_string (unhexify perms ^ "\000")) (unhexify value) | _ -> - info "restoring: ignoring unknown line: %s" line + warn "restoring: ignoring unknown line: %s" line with exn -> - info "restoring: ignoring unknown line: %s (exception: %s)" + warn "restoring: ignoring unknown line: %s (exception: %s)" line (Printexc.to_string exn); () with End_of_file ->
Signature · 1 line
2.34.1
Re: [PATCH for-4.17 v3 12/15] tools/ocaml/xenstored: log live update issues at warning level
On 8 Nov 2022, at 15:34, Edwin Török <[email protected]<mailto:[email protected]>> wrote:
However the default log level is warning, which means that problems during live update were not logged at all, unless you've changes oxenstored.conf prior to live-update.
---
Reason for inclusion in 4.17:
- makes live update issues debuggable (live update can be used to apply security updates to 4.17)
Changes since v2:
- new in v3
---tools/ocaml/xenstored/xenstored.ml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
PATCH v3 13/15 tools/ocaml/xenstored: set uncaught exception handler
In reply to Edwin TörökHelps debug fatal errors during live update
Previously this would've just gone to /dev/null, because:
* daemonize reopens stderr as /dev/null
* systemd redirects stderr to /dev/null too
Previously the only way to debug this was to manually run oxenstored with --no-fork, but when you have a fatal error and oxenstored just disappears you'd want to know why.
There has been at least one observed instance of a bug where oxenstored just disappeared inexplicably (it was believed due to an OOM exception).
---
Reason for inclusion in 4.17:
- avoids losing crucial information during a fatal error (e.g. during live update)
Changes since v2:
- new in v3
--- tools/ocaml/xenstored/logging.ml | 33 ++++++++++++++++++++++++++++++ tools/ocaml/xenstored/xenstored.ml | 3 ++- 2 files changed, 35 insertions(+), 1 deletion(-)
tools/ocaml/xenstored/logging.ml +33 -0
@@ -342,3 +342,36 @@ let xb_answer ~tid ~con ~ty data = let watch_not_fired ~con perms path = let data = Printf.sprintf "EPERM perms=[%s] path=%s" perms path in access_logging ~tid:0 ~con ~data Watch_not_fired ~level:Info + +let print_flush msg = + prerr_endline msg; + flush stderr + +let msg_of exn bt = + Printf.sprintf "Fatal exception: %s\n%s\n" (Printexc.to_string exn) + (Printexc.raw_backtrace_to_string bt) + +let fallback_exception_handler exn bt = + (* stderr goes to /dev/null, so use the logger where possible, + but always print to stderr too, in case everything else fails, + e.g. this can be used to debug with --no-fork + + this function should try not to raise exceptions, but if it does + the ocaml runtime should still print the exception, both the original, + and the one from this function, but to stderr this time + *) + let msg = msg_of exn bt in + print_flush msg; + (* See Printexc.set_uncaught_exception_handler, need to flush, + so has to call stop and flush *) + match !xenstored_logger with + | Some l -> error "xenstored-fallback" "%s" msg; l.stop () + | None -> + (* Too early, no logger set yet. + We normally try to use the configured logger so we don't flood syslog + during development for example, or if the user has a file set + *) + try Syslog.log Syslog.Daemon Syslog.Err msg + with e -> + let bt = Printexc.get_raw_backtrace () in + print_flush @@ msg_of e bt
tools/ocaml/xenstored/xenstored.ml +2 -1
@@ -357,7 +357,8 @@ let tweak_gc () = Gc.set { (Gc.get ()) with Gc.max_overhead = !Define.gc_max_overhead } -let _ = +let () = + Printexc.set_uncaught_exception_handler Logging.fallback_exception_handler; let cf = do_argv in let pidfile = if Sys.file_exists (config_filename cf) then
Signature · 1 line
2.34.1
Re: [PATCH for-4.17 v3 13/15] tools/ocaml/xenstored: set uncaught exception handler
Quoted 16 lines · Edwin Török
Helps debug fatal errors during live update
Previously this would've just gone to
/dev/null, because:
* daemonize reopens stderr as/dev/null
* systemd redirects stderr to/dev/nulltooPreviously the only way to debug this was to manually run oxenstored with --no-fork, but when you have a fatal error and oxenstored just disappears you'd want to know why.
There has been at least one observed instance of a bug where oxenstored just disappeared inexplicably (it was believed due to an OOM exception).
Quoted 5 lines
Signed-off-by Edwin Török <[email protected]>---
Reason for inclusion in 4.17:
- avoids losing crucial information during a fatal error (e.g. during live update)
tools/ocaml/xenstored/logging.ml let xb_answer ~tid ~con ~ty data = · line 348
Quoted 3 lines
tools/ocaml/xenstored/logging.ml +3
-0+let print_flush msg = + prerr_endline msg; + flush stderr
The flush is not required because prerr_endline does it implicitly.
PATCH v3 14/15 tools/ocaml/xenstored/syslog_stubs.c: avoid potential NULL dereference
In reply to Edwin TörökIf we are out of memory then strdup may return NULL, and passing NULL to syslog may cause a crash.
Avoid this by using caml_stat_strdup which will raise an OCaml out of memory exception instead.
This then needs to be paired with caml_stat_free.
---
Reason for inclusion in 4.17:
- fixes a bug in out of memory situations
Changes since v2:
- new in v3
--- tools/ocaml/xenstored/syslog_stubs.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
tools/ocaml/xenstored/syslog_stubs.c +5 -2
@@ -14,6 +14,7 @@#include <syslog.h> #include <string.h> +#include <caml/fail.h> #include <caml/mlvalues.h> #include <caml/memory.h> #include <caml/alloc.h>
@@ -35,14 +36,16 @@ static int __syslog_facility_table[] = {value stub_syslog (value facility,value level,value msg) {CAMLparam3 (facility, level, msg); -const char *c_msg =strdup (String_val (msg)); +char *c_msg =strdup (String_val (msg));int c_facility = __syslog_facility_table[Int_val (facility)] | __syslog_level_table[Int_val (level)]; +if ( !c_msg ) +caml_raise_out_of_memory ();caml_enter_blocking_section ();syslog (c_facility,"%s" , c_msg);caml_leave_blocking_section (); -free ((void *)c_msg); +free (c_msg);CAMLreturn (Val_unit); }
Signature · 1 line
2.34.1
Re: [PATCH for-4.17 v3 14/15] tools/ocaml/xenstored/syslog_stubs.c: avoid potential NULL dereference
Quoted 31 lines · Edwin Török
If we are out of memory then strdup may return NULL, and passing NULL to syslog may cause a crash.
Avoid this by using
caml_stat_strdupwhich will raise an OCaml out of memory exception instead.
This then needs to be paired withcaml_stat_free.Signed-off-by Edwin Török <[email protected]>---
Reason for inclusion in 4.17:
- fixes a bug in out of memory situationsChanges since v2:
- new in v3--- tools/ocaml/xenstored/syslog_stubs.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)tools/ocaml/xenstored/syslog_stubs.c +1
-0@@ -14,6 +14,7 @@#include <syslog.h> #include <string.h> +#include <caml/fail.h> #include <caml/mlvalues.h> #include <caml/memory.h> #include <caml/alloc.h>
tools/ocaml/xenstored/syslog_stubs.c static int __syslog_facility_table[] = { · line 39
Quoted 5 lines
tools/ocaml/xenstored/syslog_stubs.c +1
-1@@ -35,14 +36,16 @@ static int __syslog_facility_table[] = {value stub_syslog (value facility,value level,value msg) {CAMLparam3 (facility, level, msg); -const char *c_msg =strdup (String_val (msg)); +char *c_msg =strdup (String_val (msg));
This change seems to be unrelated with the goal of the commit. IMHO, this should be done in a separate patch.
The minimum would be to mention in the commit message.
Cheers,
Signature · 1 line
Julien Grall
Re: [PATCH for-4.17 v3 14/15] tools/ocaml/xenstored/syslog_stubs.c: avoid potential NULL dereference
Quoted 41 lines · Julien Grall
Quoted 26 lines · Edwin Török
If we are out of memory then strdup may return NULL, and passing NULL to syslog may cause a crash.
Avoid this by usingcaml_stat_strdupwhich will raise an OCaml out of memory exception instead.
This then needs to be paired withcaml_stat_free.Signed-off-by Edwin Török <[email protected]>---
Reason for inclusion in 4.17:
- fixes a bug in out of memory situations
Changes since v2:
- new in v3--- tools/ocaml/xenstored/syslog_stubs.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)tools/ocaml/xenstored/syslog_stubs.c +1
-0@@ -14,6 +14,7 @@#include <syslog.h> #include <string.h> +#include <caml/fail.h> #include <caml/mlvalues.h> #include <caml/memory.h> #include <caml/alloc.h> tools/ocaml/xenstored/syslog_stubs.c static int __syslog_facility_table[] = { · line 39
Quoted 5 lines
tools/ocaml/xenstored/syslog_stubs.c +1
-1@@ -35,14 +36,16 @@ static int __syslog_facility_table[] = {value stub_syslog (value facility,value level,value msg) {CAMLparam3 (facility, level, msg); -const char *c_msg =strdup (String_val (msg)); +char *c_msg =strdup (String_val (msg));This change seems to be unrelated with the goal of the commit. IMHO, this should be done in a separate patch.
Quoted 1 line
The minimum would be to mention in the commit message.
That is to avoid freeing 'const char*' (there is a typecast below). I'll mention it.
Quoted 5 lines
Cheers,
Signature · 1 line
Julien Grall
Re: [PATCH for-4.17 v3 14/15] tools/ocaml/xenstored/syslog_stubs.c: avoid potential NULL dereference
In reply to Edwin TörökOn 8 Nov 2022, at 15:34, Edwin Török <[email protected]<mailto:[email protected]>> wrote:
If we are out of memory then strdup may return NULL, and passing NULL to syslog may cause a crash.
Avoid this by using caml_stat_strdup which will raise an OCaml out of memory exception instead.
This then needs to be paired with caml_stat_free.
---
Reason for inclusion in 4.17:
- fixes a bug in out of memory situations
Changes since v2:
- new in v3
---tools/ocaml/xenstored/syslog_stubs.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
PATCH v3 15/15 tools/ocaml/libs/xc: fix use of uninitialized memory in shadow_allocation_get
In reply to Edwin TörökIt has been noticed in 2013 that shadow allocation sometimes returns the wrong value, which got worked around by adding a limit to the shadow multiplier of 1000 and ignoring the value from Xen in that case to avoid a shadow multiplier causing a VM to request 6PB of memory for example:
https://github.com/xapi-project/xen-api/pull/1215/commits/be55a8c30b41d1cd7596fc100ab1cfd3539f74eb
However that is just a workaround, and I've just reproduced this by killing a VM mid migration, which resulted in a shadow multiplier of 629.42, rendering the VM unbootable even after a host reboot.
The real bug is in Xen: when a VM is dying it will return '0' for paging op domctls and log a message at info level
'Ignoring paging op on dying domain', which leaves the 'mb' parameter uninitialized upon return from the domctl.
The binding also doesn't initialize the 'c->mb' parameter (it is meant to be used only when setting, not when querying the allocation), which results in the VM getting a shadow allocation (and thus multiplier) set based on what value happened to be currently on the stack.
Explicitly initialize the value passed to the domctl, and detect the uninitialized case (shadow allocation of 0), and raise an exception in that case. The exception will cause xenopsd to skip setting the shadow multiplier.
Note that the behaviour of Xen here is inconsistent between x86 and ARM:
ARM would return EINVAL when it gets a paging op on a dying domain, and X86-64 would return 0 with possibly uninitialized data.
It might be desirable to change the x86 path in the hypervisor to return EINVAL, although that would require more testing in case it breaks somethig.
But the bindings should be defensive anyway against bugs like this.
---
Reason for inclusion in 4.17:
- fixes a long-standing (>9y old) bug that is still happening today
Changes since v2:
- new in v3
--- tools/ocaml/libs/xc/xenctrl_stubs.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
tools/ocaml/libs/xc/xenctrl_stubs.c +4 -1
@@ -1019,7 +1019,7 @@ CAMLprim value stub_shadow_allocation_get(value xch, value domid) {CAMLparam2 (xch, domid);CAMLlocal1 (mb); -unsigned int c_mb; +unsigned int c_mb =0 ;int ret;caml_enter_blocking_section ();
@@ -1029,6 +1029,9 @@ CAMLprim value stub_shadow_allocation_get(value xch, value domid)caml_leave_blocking_section ();if (ret !=0 )failwith_xc (_H (xch)); +if ( !c_mb ) +caml_failwith ("domctl returned uninitialized data for shadow " +"allocation, dying domain?" ); mb =Val_int (c_mb);CAMLreturn (mb);
Signature · 1 line
2.34.1
Re: [PATCH for-4.17 v3 15/15] tools/ocaml/libs/xc: fix use of uninitialized memory in shadow_allocation_get
Quoted 11 lines · Edwin Török
It has been noticed in 2013 that shadow allocation sometimes returns the wrong value, which got worked around by adding a limit to the shadow multiplier of 1000 and ignoring the value from Xen in that case to avoid a shadow multiplier causing a VM to request 6PB of memory for example:
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fxapi-project%2Fxen-api%2Fpull%2F1215%2Fcommits%2Fbe55a8c30b41d1cd7596fc100ab1cfd3539f74eb&data=05%7C01%7Cedvin.torok%40citrix.com%7C54fa199055674737536f08dac19f7026%7C335836de42ef43a2b145348c2ee9ca5b%7C0%7C0%7C638035187781870066%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=l3%2BSDQinoqZ9CZvWsAcXgFl5vEbJf7hjVzBPLKoVYp4%3D&reserved=0However that is just a workaround, and I've just reproduced this by killing a VM mid migration, which resulted in a shadow multiplier of 629.42, rendering the VM unbootable even after a host reboot.
After some more discussion it looks like this is getting fixed in the hypervisor, so this workaround wouldn't be needed, might want to hold off on this patch until the domctl discussion is settled at:
https://lore.kernel.org/xen-devel/[email protected]/
Best regards,
--Edwin
Quoted 61 lines
The real bug is in Xen: when a VM is dying it will return '0' for paging op domctls and log a message at info level
'Ignoring paging op on dying domain', which leaves the 'mb' parameter uninitialized upon return from the domctl.The binding also doesn't initialize the '
c->mb' parameter (it is meant to be used only when setting, not when querying the allocation), which results in the VM getting a shadow allocation (and thus multiplier) set based on what value happened to be currently on the stack.Explicitly initialize the value passed to the domctl, and detect the uninitialized case (shadow allocation of 0), and raise an exception in that case. The exception will cause xenopsd to skip setting the shadow multiplier.
Note that the behaviour of Xen here is inconsistent between x86 and ARM:
ARM would return EINVAL when it gets a paging op on a dying domain, and X86-64 would return 0 with possibly uninitialized data.It might be desirable to change the x86 path in the hypervisor to return EINVAL, although that would require more testing in case it breaks somethig.
But the bindings should be defensive anyway against bugs like this.Signed-off-by Edwin Török <[email protected]>---
Reason for inclusion in 4.17:
- fixes a long-standing (>9y old) bug that is still happening todayChanges since v2:
- new in v3
---tools/ocaml/libs/xc/xenctrl_stubs.c| 5 ++++-1 file changed, 4 insertions(+), 1 deletion(-)tools/ocaml/libs/xc/xenctrl_stubs.c +4
-1@@ -1019,7 +1019,7 @@ CAMLprim value stub_shadow_allocation_get(value xch, value domid) {CAMLparam2 (xch, domid);CAMLlocal1 (mb); -unsigned int c_mb; +unsigned int c_mb =0 ;int ret;caml_enter_blocking_section ();@@ -1029,6 +1029,9 @@ CAMLprim value stub_shadow_allocation_get(value xch, value domid)caml_leave_blocking_section ();if (ret !=0 )failwith_xc (_H (xch)); +if ( !c_mb ) +caml_failwith ("domctl returned uninitialized data for shadow " +"allocation, dying domain?" ); mb =Val_int (c_mb);CAMLreturn (mb);Signature · 1 line
2.34.1
Summary: Re: [PATCH for-4.17 v3 00/15] OCaml fixes for Xen 4.17
In reply to Edwin TörökNothing here is critical enough to go into 4.17 at this juncture.
Various notes/observations from having spent a day trying to untangle things.
1) Patches 5/6 are a single bugfix and need merging. Except there was also an error when taking feedback from the list, and the net result regresses the original optimisation. I have a fix sorted in my local queue.
2) The indentation fix (not attached to this series) should scope the logic, not delete a debug line which was presumably added for a good reason. I've got a fix to this effect in my local queue, and we can discuss the pros/cons of the approach in due course.
3) Patch 1, evtchn Ocaml 5.0 compat, is still missing some corrections which I gave on earlier postings. I've fixed it up locally in my queue.
I also notice, while reviewing the whole, that stub_eventchn_init() passes NULL as a logger, which has the side effect of libxenevtchn instantiating a default logger which takes control of stdout/stderr. Without starting the fight over toxic library behaviour yet again, it occurs to me in the context of Patch 13, uncaught exception handler, that in oxenstored, any logging from the C level needs to end up elsewhere.
While we do have ocaml bindings for xentoollog, nothing uses it, and none of the other libraries (save xl, which isn't used) have a way of passing the Ocaml Xentoollog down. This probably wants rethinking, one way or another.
4) Patches 2/3. All these libraries have far worse problems than evtchn, because they can easily use-after-free. They all need to be Custom with a finaliser.
5) Patch 4. The commit message says "A better solution is being worked on for master", but this is master. Also, it's not a prerequisite for a security fix; merely something to make a developers life easier.
6) The re-indent patch. Policies of when to do it aside, having tried using it, the format adjustment is incomplete (running ocp-indent gets me deltas in files I haven't touched), and there needs to be some .gitignore changes.
That said, it is usually frowned upon to have logic depending on being in a git tree. This was perhaps a bigger deal back when we used hg by default and mirrored into multiple SCMs, but it's still expected not to rely on this.
7) Patch 8, evtchn fdopen, is two separate patches. One adding fdopen, and one adding a NOCLOEXEC argument to the existing init.
They want splitting in two. fdopen() ought to pass flags so we don't have to break the ABI again when there is a flag needing passing, and cloexec probably shouldn't be a boolean. We should either pass a raw int32, or a list-of-enums like we do in the xenctrl stubs. Also, this patch has inherited errors from patch 1.
9) Patches 8 thru 15 need to be the other side of the intent patch, because they need backporting to branches which will never get it. This is why bugfixes always go at the head of a patch series, and improvements at the tail.
10) Patch 12 talks about default log levels, but that's bogus reasoning. The messages should be warnings because they non-fatal exceptional cases.
11) Patch 14 talks about using caml_stat_strdup(), but doesn't.
~Andrew
Re: Summary: Re: [PATCH for-4.17 v3 00/15] OCaml fixes for Xen 4.17
Quoted 5 lines · Andrew Cooper
Nothing here is critical enough to go into 4.17 at this juncture.
Agreed
Quoted 3 lines
Various notes/observations from having spent a day trying to untangle things.
Thanks.
Quoted 9 lines
1) Patches 5/6 are a single bugfix and need merging. Except there was also an error when taking feedback from the list, and the net result regresses the original optimisation. I have a fix sorted in my local queue.
2) The indentation fix (not attached to this series) should scope the logic, not delete a debug line which was presumably added for a good reason. I've got a fix to this effect in my local queue, and we can discuss the pros/cons of the approach in due course.
I deleted the debug line to avoid reindenting code, which was frowned upon. Either way it is fine for me.
Quoted 11 lines
3) Patch 1, evtchn Ocaml 5.0 compat, is still missing some corrections which I gave on earlier postings. I've fixed it up locally in my queue.
I also notice, while reviewing the whole, that
stub_eventchn_init()passes NULL as a logger, which has the side effect of libxenevtchn instantiating a default logger which takes control of stdout/stderr. Without starting the fight over toxic library behaviour yet again, it occurs to me in the context of Patch 13, uncaught exception handler, that in oxenstored, any logging from the C level needs to end up elsewhere.
Quoted 4 lines
While we do have ocaml bindings for xentoollog, nothing uses it, and none of the other libraries (save xl, which isn't used) have a way of passing the Ocaml Xentoollog down. This probably wants rethinking, one way or another.
We should probably start by reviewing the xentoollog bindings, if they never got used they're probably buggy. I think Pau might have some xentoollog bindings though.
Quoted 4 lines
4) Patches 2/3. All these libraries have far worse problems than evtchn, because they can easily use-after-free. They all need to be Custom with a finaliser.
Indeed, the bindings aren't very safe, and that should be fixed separately. I've got some patches somewhere to stop the mmap bindings from segfaulting on invalid data, but I lost track whether that got commited or still in one of my local branches.
Quoted 4 lines
5) Patch 4. The commit message says "A better solution is being worked on for master", but this is master. Also, it's not a prerequisite for a security fix; merely something to make a developers life easier.
It is to avoid having to add Makefile changes in each security patch commit (potentially). Perhaps the commit message can be changed to say this is the minimal change to unbreak the build system, and a more comprehensive solution is being worked on (using Dune, or dune generated makefiles).
Quoted 4 lines
6) The re-indent patch. Policies of when to do it aside, having tried using it, the format adjustment is incomplete (running ocp-indent gets me deltas in files I haven't touched),
Perhaps we need to use the strict_comments setting, I think it tries to leave comments untouched, but that also means the outcome depends on the starting state.
And I hope it doesn't depend on ocp-indent version.
Quoted 7 lines
and there needs to be some
.gitignore changes.That said, it is usually frowned upon to have logic depending on being in a git tree. This was perhaps a bigger deal back when we used hg by default and mirrored into multiple SCMs, but it's still expected not to rely on this.
We could use 'find' instead.
Quoted 6 lines
7) Patch 8, evtchn fdopen, is two separate patches. One adding fdopen, and one adding a NOCLOEXEC argument to the existing init.
They want splitting in two.
fdopen()ought to pass flags so we don't have to break the ABI again
The ABI changes everytime a new variant is added (the interface hash will change, and you need to rebuild/relink), so using a single flag variant doesn't avoid ABI changes like it would in C.
Quoted 2 lines
when there is a flag needing passing, and
cloexec probably shouldn't be a boolean.
The preferred way to bind CLOEXEC in OCaml is to use a boolean, see Unix.html <https://v2.ocaml.org/api/Unix.html>, in particular `val socket : ?cloexec:bool ->socket_domain -> socket_type -> int -> file_descr` Perhaps I should've spelled this out in the commit message.
Quoted 6 lines
We should either pass a raw
int32, or a list-of-enums like we do in the xenctrl stubs. Also, this patch has inherited errors from patch 1.9) Patches 8 thru 15 need to be the other side of the intent patch, because they need backporting to branches which will never get it.
This was done on purpose like this to ensure that indentation is backported in some way,
because the lack of indentation has previously broken backports/rebases (see the debug line introduced in the wrong place in the live update patch). Without a comprehensive testsuite (which is being worked on, but not ready yet) it is then impossible to tell whether a backport is correct or not, even if it compiles, it may have some things in the wrong place, and wrong indentation just makes reviewing those more difficult.
Otherwise I have to keep making changes with the wrong indentation or avoid indentation changes in patches, which has previously introduced bugs. It is extra work, and all it does is decrease the quality of the end result and confuse both patch authors and reviewers. Furthermore the indentation commit on its own is separate and can be proven to have no functional changes if you view the diff ignoring whitespace.
We first need to make oxenstored suitable to be developed on, which means starting at the basics, fixing up indentation, build systems (all the bugs in the bindings you pointed out), etc.
I tried my best to avoid making changes like this within the XSA fix (which only contributed to lengthening the time to develop it), but now that we're no longer constrained by XSA rules we should fix things the right way.
Keeping the status quo just for the sake of it only discourages contribution to oxenstored.
If it helps we can consider all past versions of oxenstored unsupported (by me) and support only master going forward, and once we have master in a reasonable shape we can decide what and how to backport, and which versions to reinstate support on. That would avoid placing artificial constraints on master development. I intend to change master substantially enough that most backports will be impossible unless you take an almost entirely new version of oxenstored.
In fact releases of oxenstored shouldn't be tied to a particular hypervisor version: there should be a single long term supported stable branch of oxenstored and a master branch. (I understand that is not possible yet due to the use of the 2 unstable xenctrl APIs, one of which has an outstanding patch series to remove the dependency) The current situation only creates the illusion of support for the backported versions, because there is no (comprehensive) testsuite to check that those backports work, and XenServer only tests internally 4.13 and latest master, anything inbetween is technically untested, and may be more buggy than just running one version.
Quoted 3 lines
This
is why bugfixes always go at the head of a patch series, and improvements at the tail.
I agree that is how it should generally be, but that means improvements can never be done because we always keep finding more and more bugs as we do improvements, and then do a lot of risky rebases to get patches moved ahead, and without minimal things like automated tools to keep at least indentation consistent the end result is a mess that cannot be trusted.
Quoted 6 lines
10) Patch 12 talks about default log levels, but that's bogus reasoning. The messages should be warnings because they non-fatal exceptional cases.
11) Patch 14 talks about using
caml_stat_strdup(), but doesn't.
The commit message for this is fixed on my github branch: https://github.com/edwintorok/xen/commit/dc893a079fd7cf2a9bb8ed03cca3d249a53e3c44
It initially had that function, but it is not available in 4.02.3
Thanks for helping sort out the patch series.
Best regards,
--Edwin