RFC: Cleanup xdiff and begin its rustification

In order to facilitate the quick adoption of am/xdiff-hash-tweak, this round drops the changes to hashing in xdiff and instead modifies another part of xdiff.

A high level overview of v3:

* patch 1: add a policy for using Rust (brian's patch, with a small tweak)
* patch 2: introduce Rust to the codebase
* patches 3-5: adapt CI (github workflows) to build Git with Rust
* patch 6: introduce the ivec type
* patches 7-14: xdiff code cleanup in preparation for translating to Rust
* patch 15: translate a C function into Rust and call it from C

I'm particularly interested in what folks think of the new ivec type for sharing data across the language barrier. Thoughts?

Build results for these changes:
https://github.com/git/git/actions/runs/17170212383

Links to older versions, which focused on hashing in xdiff:

* v1:
https://lore.kernel.org/git/[email protected]/
* v2:
https://lore.kernel.org/git/[email protected]/

Ezekiel Newren (13):
xdiff: introduce rust
github workflows: install rust
github workflows: upload Cargo.lock
ivec: create a vector type that is interoperable between C and Rust xdiff/xprepare: remove superfluous forward declarations
xdiff: delete unnecessary fields from xrecord_t and xdfile_t
xdiff: make fields of xrecord_t Rust friendly
xdiff: use one definition for freeing xdfile_t
xdiff: replace chastore with an ivec in xdfile_t
xdiff: delete nrec field from xdfile_t
xdiff: delete recs field from xdfile_t
xdiff: make xdfile_t more rust friendly
xdiff: implement xdl_trim_ends() in Rust

Johannes Schindelin (1):
win+Meson: do allow linking with the Rust-built xdiff

brian m. carlson (1):
doc: add a policy for using Rust

 .github/workflows/main.yml                    |  89 +++-
 .gitignore                                    |   3 +
 Documentation/Makefile                        |   1 +
 Documentation/technical/platform-support.adoc |   2 +
 Documentation/technical/rust-support.adoc     | 142 ++++++
 Makefile                                      |  69 ++-
 build_rust.sh                                 |  57 +++
 ci/install-dependencies.sh                    |  14 +-
 ci/install-rust-toolchain.sh                  |  30 ++
 ci/install-rustup.sh                          |  25 +
 ci/lib.sh                                     |   1 +
 ci/make-test-artifacts.sh                     |   9 +
 ci/run-build-and-tests.sh                     |  13 +
 config.mak.uname                              |   4 +
 git-compat-util.h                             |  17 +
 interop/ivec.c                                | 151 ++++++
 interop/ivec.h                                |  52 ++
 meson.build                                   |  54 +-
 rust/Cargo.toml                               |   6 +
 rust/interop/Cargo.toml                       |  14 +
 rust/interop/src/ivec.rs                      | 462 ++++++++++++++++++
 rust/interop/src/lib.rs                       |  10 +
 rust/xdiff/Cargo.toml                         |  15 +
 rust/xdiff/src/lib.rs                         |  15 +
 rust/xdiff/src/xprepare.rs                    |  27 +
 rust/xdiff/src/xtypes.rs                      |  19 +
 xdiff/xdiffi.c                                |  60 +--
 xdiff/xdiffi.h                                |   8 +-
 xdiff/xemit.c                                 |  24 +-
 xdiff/xhistogram.c                            |   2 +-
 xdiff/xmerge.c                                |  72 +--
 xdiff/xpatience.c                             |  16 +-
 xdiff/xprepare.c                              | 271 ++++------
 xdiff/xtypes.h                                |  27 +-
 xdiff/xutils.c                                |  12 +-
 35 files changed, 1474 insertions(+), 319 deletions(-)
 create mode 100644 Documentation/technical/rust-support.adoc
 create mode 100755 build_rust.sh
 create mode 100755 ci/install-rust-toolchain.sh
 create mode 100755 ci/install-rustup.sh
 create mode 100644 interop/ivec.c
 create mode 100644 interop/ivec.h
 create mode 100644 rust/Cargo.toml
 create mode 100644 rust/interop/Cargo.toml
 create mode 100644 rust/interop/src/ivec.rs
 create mode 100644 rust/interop/src/lib.rs
 create mode 100644 rust/xdiff/Cargo.toml
 create mode 100644 rust/xdiff/src/lib.rs
 create mode 100644 rust/xdiff/src/xprepare.rs
 create mode 100644 rust/xdiff/src/xtypes.rs

base-commit: 16bd9f20a403117f2e0d9bcda6c6e621d3763e77
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1980%2Fezekielnewren%2Fxdiff_rust_speedup-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1980/ezekielnewren/xdiff_rust_speedup-v3
Pull-Request: https://github.com/git/git/pull/1980

Range-diff vs v2:

1: 75dfb40ead3 ! 1: 6d065f550fe doc: add a policy for using Rust
@@ Commit message
contributors.

+ [en: Added some comments about types, and changed the recommondations
+ about cbindgen, bindgen, rustix, libc.]

## Documentation/Makefile ##
@@ Documentation/technical/platform-support.adoc: meet the following minimum requir

* Has active security support (taking security releases of dependencies, etc)

-+* Supports Rust and the toolchain version specified in link:rust-support.txt[]. ++* Supports Rust and the toolchain version specified in link:rust-support.adoc[].
+
These requirements are a starting point, and not sufficient on their own for the Git community to be enthusiastic about supporting your platform. Maintainers of
@@ Documentation/technical/rust-support.adoc (new)
+caused by porting a binary wholesale from one language to another that might +introduce bugs.
+
-+We will use the bindgen and cbindgen crates for handling C-compatible -+bindings and the rustix crate for POSIX-compatible interfaces. The libc -+crate, which is used by rustix, does not expose safe interfaces and does not -+handle differences between platforms, such as differing 64-bit stat call -+names, and so is less desirable as a target than rustix. We may still choose -+to use it in some cases if rustix does not offer suitable interfaces. ++Crates like libc or rustix define types like c_long, but in ways that are not ++safe across platforms.
++From https://docs.rs/rustix/latest/rustix/ffi/type.c_long.html:
++
++ This type will always be i32 or i64. Most notably, many Linux-based
++ systems assume an i64, but Windows assumes i32. The C standard technically
++ only requires that this type be a signed integer that is at least 32 bits
++ and at least the size of an int, although in practice, no system would
++ have a long that is neither an i32 nor i64.
++
++Also, note that other locations, such as ++https://docs.rs/libc/latest/libc/type.c_long.html, just hardcode c_long as i64 ++even though C may mean i32 on some platforms.
++
++As such, using the c_long type would give us portability issues, and ++perpetuate some of the bugs git has faced across platforms. Avoid using C's ++types (long, unsigned, char, etc.), and switch to unambiguous types (e.g. i32 ++or i64) before trying to make C and Rust interoperate.
++
++Crates like libc and rustix may have also traditionally aided interoperability ++with older versions of Rust (e.g. when worrying about stat[64] system calls), ++but the Rust standard library in newer versions of Rust handle these concerns ++in a platform agnostic way. There may arise cases where we need to consider ++these crates, but for now we omit them.
++
++Tools like bindgen and cbindgen create C-styled unsafe Rust code rather than ++idiomatic Rust; where possible, we prefer to switch to idiomatic Rust. Any ++standard C library functions that are needed can be manually wrapped on the ++Rust side.
+
+Rust upstream releases every six weeks and only supports the latest stable +release. While it is nice that upstream is active, we would like our software

  2:  7709e5eddba <  -:  ----------- xdiff: introduce rust
  8:  7dc241e6682 !  2:  03939951256 github workflows: install rust
     @@ Metadata
      Author: Ezekiel Newren <[email protected]>
      
       ## Commit message ##
     -    github workflows: install rust
     +    xdiff: introduce rust
      
     -    Since we have introduced rust, it needs to be installed for the
     -    continuous integration build targets. Create an install script
     -    (build_rust.sh) that needs to be run as the same user that builds git.
     -    Because of the limitations of meson, create build_rust.sh which makes
     -    it easy to centralize how rust is built between meson and make.
     +    Upcoming patches will simplify xdiff, while also porting parts of it to
     +    Rust. In preparation, add some stubs and setup the Rust build. For now,
     +    it is easier to let cargo build rust and have make or meson merely link
     +    against the static library that cargo builds. In line with ongoing
     +    libification efforts, use multiple crates to allow more modularity on
     +    the Rust side. xdiff is the crate that this series will focus on, but
     +    we also introduce the interop crate for future patch series.
      
     -    There are 2 interesting decisions worth calling out in this commit:
     -
     -    * The 'output' field of custom_target() does not allow specifying a
     -      file nested inside the build directory. Thus create build_rust.sh to
     -      build rust with all of its parameters and then moves libxdiff.a to
     -      the root of the build directory.
     -
     -    * Install curl, to facilitate the rustup install script.
     +    In order to facilitate interoperability between C and Rust, introduce C
     +    definitions for Rust primitive types in git-compat-util.h.
      
          Signed-off-by: Ezekiel Newren <[email protected]>
      
     - ## .github/workflows/main.yml ##
     -@@ .github/workflows/main.yml: on: [push, pull_request]
     - 
     - env:
     -   DEVELOPER: 1
     -+  RUST_VERSION: 1.87.0
     - 
     - # If more than one workflow run is triggered for the very same commit hash
     - # (which happens when multiple branches pointing to the same commit), only
     + ## .gitignore ##
     +@@ .gitignore: Release/
     + /contrib/buildsystems/out
     + /contrib/libgit-rs/target
     + /contrib/libgit-sys/target
     ++/.idea/
     ++/rust/target/
     ++/rust/Cargo.lock
      
       ## Makefile ##
      @@ Makefile: TEST_SHELL_PATH = $(SHELL_PATH)
     @@ Makefile: TEST_SHELL_PATH = $(SHELL_PATH)
      +
      +EXTLIBS =
      +
     - ifeq ($(DEBUG), 1)
     --RUST_LIB = rust/target/debug/libxdiff.a
     ++ifeq ($(DEBUG), 1)
      +  RUST_BUILD_MODE = debug
     - else
     --RUST_LIB = rust/target/release/libxdiff.a
     ++else
      +  RUST_BUILD_MODE = release
      +endif
      +
     @@ Makefile: TEST_SHELL_PATH = $(SHELL_PATH)
      +UNAME_S := $(shell uname -s)
      +ifeq ($(UNAME_S),Linux)
      +  EXTLIBS += -ldl
     - endif
     ++endif
      +
       REFTABLE_LIB = reftable/libreftable.a
       
     @@ Makefile: UNIT_TEST_OBJS += $(UNIT_TEST_DIR)/lib-reftable.o
       # xdiff and reftable libs may in turn depend on what is in libgit.a
       GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(LIB_FILE)
      -EXTLIBS =
     - 
     --GITLIBS += $(RUST_LIB)
     ++
       
       GIT_USER_AGENT = git/$(GIT_VERSION)
       
     @@ Makefile: $(REMOTE_CURL_ALIASES): $(REMOTE_CURL_PRIMARY)
       	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \
       		$(filter %.o,$^) $(LIBS)
       
     +@@ Makefile: $(LIB_FILE): $(LIB_OBJS)
     + $(XDIFF_LIB): $(XDIFF_OBJS)
     + 	$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^
     + 
     ++
     + $(REFTABLE_LIB): $(REFTABLE_OBJS)
     + 	$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^
     + 
      @@ Makefile: perf: all
       
       t/helper/test-tool$X: $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS)) $(UNIT_TEST_DIR)/test-lib.o
     @@ Makefile: perf: all
       	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(filter %.a,$^) $(LIBS)
       
       check-sha1:: t/helper/test-tool$X
     +@@ Makefile: cocciclean:
     + 	$(RM) -r .build/contrib/coccinelle
     + 	$(RM) contrib/coccinelle/*.cocci.patch
     + 
     +-clean: profile-clean coverage-clean cocciclean
     ++rustclean:
     ++	cd rust && cargo clean
     ++
     ++clean: profile-clean coverage-clean cocciclean rustclean
     + 	$(RM) -r .build $(UNIT_TEST_BIN)
     + 	$(RM) GIT-TEST-SUITES
     + 	$(RM) po/git.pot po/git-core.pot
      @@ Makefile: FUZZ_CXXFLAGS ?= $(ALL_CFLAGS)
       .PHONY: fuzz-all
       fuzz-all: $(FUZZ_PROGRAMS)
     @@ build_rust.sh (new)
      @@
      +#!/bin/sh
      +
     -+if [ -z "$CARGO_HOME" ]; then
     -+  export CARGO_HOME=$HOME/.cargo
     -+  echo >&2 "::warning:: CARGO_HOME is not set"
     -+fi
     -+echo "CARGO_HOME=$CARGO_HOME"
      +
     -+rustc -vV
     -+cargo --version
     ++rustc -vV || exit $?
     ++cargo --version || exit $?
      +
      +dir_git_root=${0%/*}
      +dir_build=$1
     -+rust_target=$2
     ++rust_build_profile=$2
      +crate=$3
      +
      +dir_rust=$dir_git_root/rust
     @@ build_rust.sh (new)
      +  exit 1
      +fi
      +
     -+if [ "$rust_target" = "" ]; then
     -+  echo "did not specify the rust_target"
     ++if [ "$rust_build_profile" = "" ]; then
     ++  echo "did not specify the rust_build_profile"
      +  exit 1
      +fi
      +
     -+if [ "$rust_target" = "release" ]; then
     ++if [ "$rust_build_profile" = "release" ]; then
      +  rust_args="--release"
     -+  export RUSTFLAGS='-Aunused_imports -Adead_code'
     -+elif [ "$rust_target" = "debug" ]; then
     ++  export RUSTFLAGS=''
     ++elif [ "$rust_build_profile" = "debug" ]; then
      +  rust_args=""
     -+  export RUSTFLAGS='-Aunused_imports -Adead_code -C debuginfo=2 -C opt-level=1 -C force-frame-pointers=yes'
     ++  export RUSTFLAGS='-C debuginfo=2 -C opt-level=1 -C force-frame-pointers=yes'
      +else
     -+  echo "illegal rust_target value $rust_target"
     ++  echo "illegal rust_build_profile value $rust_build_profile"
      +  exit 1
      +fi
      +
     -+cd $dir_rust && cargo clean && pwd && cargo build -p $crate $rust_args; cd ..
     ++cd $dir_rust && cargo clean && pwd && cargo build -p $crate $rust_args; cd $dir_git_root
      +
      +libfile="lib${crate}.a"
     ++if rustup show active-toolchain | grep windows-msvc; then
     ++  libfile="${crate}.lib"
     ++fi
      +dst=$dir_build/$libfile
      +
      +if [ "$dir_git_root" != "$dir_build" ]; then
     -+  src=$dir_rust/target/$rust_target/$libfile
     ++  src=$dir_rust/target/$rust_build_profile/$libfile
      +  if [ ! -f $src ]; then
     -+    echo >&2 "::error:: cannot find path of static library"
     ++    echo >&2 "::error:: cannot find path of static library $src is not a file or does not exist"
      +    exit 5
      +  fi
      +
     @@ build_rust.sh (new)
      +  mv $src $dst
      +fi
      
     - ## ci/install-dependencies.sh ##
     -@@ ci/install-dependencies.sh: fi
     - 
     - case "$distro" in
     - alpine-*)
     --	apk add --update shadow sudo meson ninja-build gcc libc-dev curl-dev openssl-dev expat-dev gettext \
     -+	apk add --update shadow sudo meson ninja-build gcc libc-dev curl curl-dev openssl-dev expat-dev gettext \
     - 		zlib-ng-dev pcre2-dev python3 musl-libintl perl-utils ncurses \
     - 		apache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \
     - 		bash cvs gnupg perl-cgi perl-dbd-sqlite perl-io-tty >/dev/null
     - 	;;
     - fedora-*|almalinux-*)
     - 	dnf -yq update >/dev/null &&
     --	dnf -yq install shadow-utils sudo make gcc findutils diffutils perl python3 gawk gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null
     -+	dnf -yq install shadow-utils sudo make gcc findutils diffutils perl python3 gawk gettext zlib-devel expat-devel openssl-devel curl curl-devel pcre2-devel >/dev/null
     - 	;;
     - ubuntu-*|i386/ubuntu-*|debian-*)
     - 	# Required so that apt doesn't wait for user input on certain packages.
     -@@ ci/install-dependencies.sh: ubuntu-*|i386/ubuntu-*|debian-*)
     - 	sudo apt-get -q update
     - 	sudo apt-get -q -y install \
     - 		$LANGUAGES apache2 cvs cvsps git gnupg $SVN \
     --		make libssl-dev libcurl4-openssl-dev libexpat-dev wget sudo default-jre \
     --		tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl \
     -+		make libssl-dev curl libcurl4-openssl-dev libexpat-dev wget sudo default-jre \
     -+		tcl tk gettext zlib1g zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl \
     - 		libemail-valid-perl libio-pty-perl libio-socket-ssl-perl libnet-smtp-ssl-perl libdbd-sqlite3-perl libcgi-pm-perl \
     - 		libsecret-1-dev libpcre2-dev meson ninja-build pkg-config \
     - 		${CC_PACKAGE:-${CC:-gcc}} $PYTHON_PACKAGE
     -@@ ci/install-dependencies.sh: ClangFormat)
     - 	;;
     - StaticAnalysis)
     - 	sudo apt-get -q update
     --	sudo apt-get -q -y install coccinelle libcurl4-openssl-dev libssl-dev \
     -+	sudo apt-get -q -y install coccinelle curl libcurl4-openssl-dev libssl-dev \
     - 		libexpat-dev gettext make
     - 	;;
     - sparse)
     - 	sudo apt-get -q update -q
     --	sudo apt-get -q -y install libssl-dev libcurl4-openssl-dev \
     --		libexpat-dev gettext zlib1g-dev sparse
     -+	sudo apt-get -q -y install libssl-dev curl libcurl4-openssl-dev \
     -+		libexpat-dev gettext zlib1g zlib1g-dev sparse
     - 	;;
     - Documentation)
     - 	sudo apt-get -q update
     + ## git-compat-util.h ##
     +@@ git-compat-util.h: static inline int is_xplatform_dir_sep(int c)
     + #include "compat/msvc.h"
     + #endif
     + 
     ++/* rust types */
     ++typedef uint8_t   u8;
     ++typedef uint16_t  u16;
     ++typedef uint32_t  u32;
     ++typedef uint64_t  u64;
     ++
     ++typedef int8_t    i8;
     ++typedef int16_t   i16;
     ++typedef int32_t   i32;
     ++typedef int64_t   i64;
     ++
     ++typedef float     f32;
     ++typedef double    f64;
     ++
     ++typedef size_t    usize;
     ++typedef ptrdiff_t isize;
     ++
     + /* used on Mac OS X */
     + #ifdef PRECOMPOSE_UNICODE
     + #include "compat/precompose_utf8.h"
      
     - ## ci/install-rust.sh (new) ##
     -@@
     -+#!/bin/sh
     -+
     -+if [ "$(id -u)" -eq 0 ]; then
     -+  echo >&2 "::warning:: installing rust as root"
     -+fi
     -+
     -+if [ "$CARGO_HOME" = "" ]; then
     -+  echo >&2 "::warning:: CARGO_HOME is not set"
     -+  export CARGO_HOME=$HOME/.cargo
     -+fi
     -+
     -+export RUSTUP_HOME=$CARGO_HOME
     -+
     -+if [ "$RUST_VERSION" = "" ]; then
     -+  echo >&2 "::error:: RUST_VERSION is not set"
     -+  exit 2
     -+fi
     -+
     -+## install rustup
     -+curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --default-toolchain none -y
     -+if [ ! -f $CARGO_HOME/env ]; then
     -+  echo "PATH=$CARGO_HOME/bin:\$PATH" > $CARGO_HOME/env
     -+fi
     -+## install a specific version of rust
     -+if [ "$BITNESS" = "32" ]; then
     -+  $CARGO_HOME/bin/rustup set default-host i686-unknown-linux-gnu || exit $?
     -+  $CARGO_HOME/bin/rustup install $RUST_VERSION || exit $?
     -+  $CARGO_HOME/bin/rustup default --force-non-host $RUST_VERSION || exit $?
     -+else
     -+  $CARGO_HOME/bin/rustup default $RUST_VERSION || exit $?
     -+fi
     -+
     -+. $CARGO_HOME/env
     -
     - ## ci/lib.sh ##
     -@@
     - # Library of functions shared by all CI scripts
     - 
     -+
     -+export BITNESS="64"
     -+if command -v getconf >/dev/null && [ "$(getconf LONG_BIT 2>/dev/null)" = "32" ]; then
     -+  export BITNESS="32"
     -+fi
     -+echo "BITNESS=$BITNESS"
     -+
     -+
     - if test true = "$GITHUB_ACTIONS"
     - then
     - 	begin_group () {
     -
     - ## ci/make-test-artifacts.sh ##
     -@@ ci/make-test-artifacts.sh: mkdir -p "$1" # in case ci/lib.sh decides to quit early
     - 
     - . ${0%/*}/lib.sh
     - 
     -+## install rust per user rather than system wide
     -+. ${0%/*}/install-rust.sh
     -+
     - group Build make artifacts-tar ARTIFACTS_DIRECTORY="$1"
     - 
     -+if [ -d "$CARGO_HOME" ]; then
     -+  rm -rf $CARGO_HOME
     -+fi
     -+
     - check_unignored_build_artifacts
     -
     - ## ci/run-build-and-tests.sh ##
     -@@
     - 
     - . ${0%/*}/lib.sh
     + ## meson.build ##
     +@@ meson.build: version_gen_environment.set('GIT_DATE', get_option('build_date'))
     + version_gen_environment.set('GIT_USER_AGENT', get_option('user_agent'))
     + version_gen_environment.set('GIT_VERSION', get_option('version'))
       
     -+## install rust per user rather than system wide
     -+. ${0%/*}/install-rust.sh
     ++if get_option('optimization') in ['2', '3', 's', 'z']
     ++  rust_build_profile = 'release'
     ++else
     ++  rust_build_profile = 'debug'
     ++endif
      +
     -+rustc -vV
     -+cargo --version || exit $?
     ++# Run `rustup show active-toolchain` and capture output
     ++rustup_out = run_command('rustup', 'show', 'active-toolchain',
     ++                         check: true).stdout().strip()
     ++
     ++rust_crates = ['xdiff']
     ++rust_builds = []
     ++
     ++foreach crate : rust_crates
     ++  if rustup_out.contains('windows-msvc')
     ++    libfile = crate + '.lib'
     ++  else
     ++    libfile = 'lib' + crate + '.a'
     ++  endif
     ++
     ++  rust_builds += custom_target(
     ++    'rust_build_'+crate,
     ++    output: libfile,
     ++    build_by_default: true,
     ++    build_always_stale: true,
     ++    command: [
     ++      meson.project_source_root() / 'build_rust.sh',
     ++      meson.current_build_dir(), rust_build_profile, crate,
     ++    ],
     ++    install: false,
     ++  )
     ++endforeach
      +
     - run_tests=t
     - 
     - case "$jobname" in
     -@@ ci/run-build-and-tests.sh: case "$jobname" in
     - 	;;
     - esac
     - 
     -+if [ -d "$CARGO_HOME" ]; then
     -+  rm -rf $CARGO_HOME
     -+fi
      +
     - check_unignored_build_artifacts
     - save_good_tree
     -
     - ## meson.build ##
     -@@ meson.build: else
     -   rustflags = '-Aunused_imports -Adead_code -C debuginfo=2 -C opt-level=1 -C force-frame-pointers=yes'
     - endif
     - 
     --
     --rust_leaf = custom_target('rust_leaf',
     -+rust_build_xdiff = custom_target('rust_build_xdiff',
     -   output: 'libxdiff.a',
     -   build_by_default: true,
     -   build_always_stale: true,
     --  command: ['cargo', 'build',
     --            '--manifest-path', meson.project_source_root() / 'rust/Cargo.toml'
     --  ] + rust_args,
     --  env: {
     --    'RUSTFLAGS': rustflags,
     --  },
     -+  command: [
     -+    meson.project_source_root() / 'build_rust.sh',
     -+    meson.current_build_dir(), rust_target, 'xdiff',
     -+  ],
     -   install: false,
     - )
     - 
     --rust_xdiff_dep = declare_dependency(
     --  link_args: ['-L' + meson.project_source_root() / 'rust/target' / rust_target, '-lxdiff'],
     --#  include_directories: include_directories('xdiff/include'),  # Adjust if you expose headers
     --)
     --
     --
       compiler = meson.get_compiler('c')
       
       libgit_sources = [
      @@ meson.build: version_def_h = custom_target(
     - )
       libgit_sources += version_def_h
       
     --libgit_dependencies += rust_xdiff_dep
     --
       libgit = declare_dependency(
      -  link_with: static_library('git',
      -    sources: libgit_sources,
     @@ meson.build: version_def_h = custom_target(
      +      dependencies: libgit_dependencies,
      +      include_directories: libgit_include_directories,
      +    ),
     -+    rust_build_xdiff,
     -+  ],
     ++  ] + rust_builds,
         compile_args: libgit_c_args,
         dependencies: libgit_dependencies,
         include_directories: libgit_include_directories,
     +
     + ## rust/Cargo.toml (new) ##
     +@@
     ++[workspace]
     ++members = [
     ++    "xdiff",
     ++    "interop",
     ++]
     ++resolver = "2"
     +
     + ## rust/interop/Cargo.toml (new) ##
     +@@
     ++[package]
     ++name = "interop"
     ++version = "0.1.0"
     ++edition = "2021"
     ++
     ++[lib]
     ++name = "interop"
     ++path = "src/lib.rs"
     ++## staticlib to generate xdiff.a for use by gcc
     ++## cdylib (optional) to generate xdiff.so for use by gcc
     ++## rlib is required by the rust unit tests
     ++crate-type = ["staticlib", "rlib"]
     ++
     ++[dependencies]
     +
     + ## rust/interop/src/lib.rs (new) ##
     +
     + ## rust/xdiff/Cargo.toml (new) ##
     +@@
     ++[package]
     ++name = "xdiff"
     ++version = "0.1.0"
     ++edition = "2021"
     ++
     ++[lib]
     ++name = "xdiff"
     ++path = "src/lib.rs"
     ++## staticlib to generate xdiff.a for use by gcc
     ++## cdylib (optional) to generate xdiff.so for use by gcc
     ++## rlib is required by the rust unit tests
     ++crate-type = ["staticlib", "rlib"]
     ++
     ++[dependencies]
     ++interop = { path = "../interop" }
     +
     + ## rust/xdiff/src/lib.rs (new) ##

12: fffdb326710 ! 3: a98d9e4d21b github workflows: define rust versions and targets in the same place
@@ Metadata
Author: Ezekiel Newren <[email protected]>

## Commit message ##
- github workflows: define rust versions and targets in the same place
+ github workflows: install rust

- Consolidate the Rust toolchain definitions in main.yaml. Prefer using
- actions-rs/toolchain@v1 where possible, but for docker targets use
- a script to install the Rust toolchain. Four overrides are used in
+ Prefer using actions-rs/toolchain@v1 where possible to install rustup,
+ but for docker targets use a script to install rustup. Consolidate the
+ Rust toolchain definitions in main.yaml. Use install-rust-toolchain.sh
+ to ensure the correct toolchain is used. Five overrides are used in
main.yaml:

* On Windows: Rust didn't resolve the bcrypt library on Windows correctly until version 1.78.0. Also since rustup mis-identifies the Rust toolchain, the Rust target triple must be set to
- x86_64-pc-windows-gnu.
+ x86_64-pc-windows-gnu for make (win build), and
+ x86_64-pc-windows-msvc for meson (win+Meson build).
* On musl: libc differences, such as ftruncate64 vs ftruncate, were not accounted for until Rust version 1.72.0. No older version of Rust will work on musl for our needs.
* In a 32-bit docker container running on a 64-bit host, we need to override the Rust target triple. This is because rustup asks the kernel for the bitness of the system and it says 64, even though
- the container will only run 32-bit. This also allows us to remove
- the BITNESS environment variable in ci/lib.sh.
+ the container is 32-bit. This also allows us to remove the
+ BITNESS environment variable in ci/lib.sh.

+ The logic for selecting library names was initially provided in a patch
+ from Johannes, but was reworked and squashed into this commit.
+
+ Helped-by: Johannes Schindelin <[email protected]>

## .github/workflows/main.yml ##
-@@ .github/workflows/main.yml: on: [push, pull_request]
-
- env:
- DEVELOPER: 1
-- RUST_VERSION: 1.87.0
-
- # If more than one workflow run is triggered for the very same commit hash
- # (which happens when multiple branches pointing to the same commit), only
@@ .github/workflows/main.yml: jobs:
outputs:
enabled: ${{ steps.check-ref.outputs.enabled }}${{ steps.skip-if-redundant.outputs.enabled }}
@@ .github/workflows/main.yml: jobs:
+ rust_version_windows: 1.78.0
+ rust_version_musl: 1.72.0
+ ## the rust target is inferred by rustup unless specified
-+ rust_target_windows: x86_64-pc-windows-gnu
++ rust_target_windows_make: x86_64-pc-windows-gnu
++ rust_target_windows_meson: x86_64-pc-windows-msvc
+ rust_target_32bit_linux: i686-unknown-linux-gnu
steps:
- name: try to clone ci-config branch
run: |
@@ .github/workflows/main.yml: jobs:
- /c/Program\ Files/Git/mingw64/bin/curl -Lo libuserenv.a \
- https://github.com/git-for-windows/git-sdk-64/raw/HEAD/mingw64/lib/libuserenv.a
- }
-+ - name: Install Rust
+ needs: ci-config
+ if: needs.ci-config.outputs.enabled == 'yes'
+ runs-on: windows-latest
++ env:
++ CARGO_HOME: "/c/Users/runneradmin/.cargo"
+ concurrency:
+ group: windows-build-${{ github.ref }}
+ cancel-in-progress: ${{ needs.ci-config.outputs.skip_concurrent == 'yes' }}
+ steps:
+ - uses: actions/checkout@v4
+ - uses: git-for-windows/setup-git-for-windows-sdk@v1
++ - name: Install rustup via github actions
+ uses: actions-rs/toolchain@v1
+ with:
-+ toolchain: ${{ needs.ci-config.outputs.rust_version_windows }}
-+ target: ${{ needs.ci-config.outputs.rust_target_windows }}
++ toolchain: stable
+ profile: minimal
-+ override: true
++ override: false
++ - name: Install Rust toolchain
++ shell: bash
++ env:
++ RUST_VERSION: ${{ needs.ci-config.outputs.rust_version_windows }}
++ RUST_TARGET: ${{ needs.ci-config.outputs.rust_target_windows_make }}
++ run: ci/install-rust-toolchain.sh
- name: build
shell: bash
env:
- HOME: ${{runner.workspace}}
- NO_PERL: 1
-+ CARGO_HOME: "/c/Users/runneradmin/.cargo"
- run: . /etc/profile && ci/make-test-artifacts.sh artifacts
- - name: zip up tracked files
- run: git archive -o artifacts/tracked.tar.gz HEAD
@@ .github/workflows/main.yml: jobs:
+ needs: ci-config
+ if: needs.ci-config.outputs.enabled == 'yes'
+ runs-on: windows-latest
++ env:
++ CARGO_HOME: "/c/Users/runneradmin/.cargo"
+ concurrency:
+ group: windows-meson-build-${{ github.ref }}
+ cancel-in-progress: ${{ needs.ci-config.outputs.skip_concurrent == 'yes' }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
-+ - name: Install Rust
++ - name: Install rustup via github actions
+ uses: actions-rs/toolchain@v1
+ with:
-+ toolchain: ${{ needs.ci-config.outputs.rust_version_windows }}
-+ target: ${{ needs.ci-config.outputs.rust_target_windows }}
++ toolchain: stable
+ profile: minimal
-+ override: true
++ override: false
++ - name: Install Rust toolchain
++ shell: bash
++ env:
++ RUST_VERSION: ${{ needs.ci-config.outputs.rust_version_windows }}
++ RUST_TARGET: ${{ needs.ci-config.outputs.rust_target_windows_meson }}
++ run: ci/install-rust-toolchain.sh
- name: Set up dependencies
shell: pwsh
run: pip install meson ninja
@@ .github/workflows/main.yml: jobs:
+ jobname: ${{matrix.vector.jobname}}
+ CI_JOB_IMAGE: ${{matrix.vector.pool}}
+ TEST_OUTPUT_DIRECTORY: ${{github.workspace}}/t
++ CARGO_HOME: "/Users/runner/.cargo"
+ runs-on: ${{matrix.vector.pool}}
steps:
- uses: actions/checkout@v4
- run: ci/install-dependencies.sh
-+ - name: Install Rust
+- - run: ci/run-build-and-tests.sh
++ - name: Install rustup via github actions
+ uses: actions-rs/toolchain@v1
+ with:
-+ toolchain: ${{ needs.ci-config.outputs.rust_version_minimum }}
++ toolchain: stable
+ profile: minimal
-+ override: true
- - run: ci/run-build-and-tests.sh
++ override: false
++ - name: Install Rust toolchain
++ shell: bash
++ env:
++ RUST_VERSION: ${{ needs.ci-config.outputs.rust_version_minimum }}
++ run: ci/install-rust-toolchain.sh
++ - name: Run build and tests
++ run: ci/run-build-and-tests.sh
- name: print test failures
if: failure() && env.FAILED_TEST_ARTIFACTS != ''
+ run: ci/print-test-failures.sh
@@ .github/workflows/main.yml: jobs:
cc: gcc
- jobname: linux-musl-meson
@@ .github/workflows/main.yml: jobs:
CI_JOB_IMAGE: ${{matrix.vector.image}}
+ CI_IS_DOCKER: "true"
CUSTOM_PATH: /custom
-+ RUST_VERSION: ${{ matrix.vector.rust_version_override || needs.ci-config.outputs.rust_version_minimum }}
-+ RUST_TARGET: ${{ matrix.vector.rust_target_override || '' }}
+ CARGO_HOME: /home/builder/.cargo
runs-on: ubuntu-latest
container: ${{matrix.vector.image}}
steps:
+@@ .github/workflows/main.yml: jobs:
+ - run: ci/install-dependencies.sh
+ - run: useradd builder --create-home
+ - run: chown -R builder .
++ - name: Install rustup via script
++ run: sudo --preserve-env --set-home --user=builder ci/install-rustup.sh
++ - name: Install Rust toolchain
++ env:
++ RUST_VERSION: ${{ matrix.vector.rust_version_override || needs.ci-config.outputs.rust_version_minimum }}
++ RUST_TARGET: ${{ matrix.vector.rust_target_override || '' }}
++ run: sudo --preserve-env --set-home --user=builder ci/install-rust-toolchain.sh
+ - run: sudo --preserve-env --set-home --user=builder ci/run-build-and-tests.sh
+ - name: print test failures
+ if: failure() && env.FAILED_TEST_ARTIFACTS != ''

- ## build_rust.sh ##
-@@
- #!/bin/sh
-
--if [ -z "$CARGO_HOME" ]; then
-- export CARGO_HOME=$HOME/.cargo
-- echo >&2 "::warning:: CARGO_HOME is not set" --fi
--echo "CARGO_HOME=$CARGO_HOME"
+ ## ci/install-dependencies.sh ##
+@@ ci/install-dependencies.sh: fi

--rustc -vV
--cargo --version
+ case "$distro" in
+ alpine-*)
+- apk add --update shadow sudo meson ninja-build gcc libc-dev curl-dev openssl-dev expat-dev gettext \
++ apk add --update shadow sudo meson ninja-build gcc libc-dev curl curl-dev openssl-dev expat-dev gettext \
+ zlib-ng-dev pcre2-dev python3 musl-libintl perl-utils ncurses \
+ apache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \
+ bash cvs gnupg perl-cgi perl-dbd-sqlite perl-io-tty >/dev/null
+ ;;
+ fedora-*|almalinux-*)
+ dnf -yq update >/dev/null &&
+- dnf -yq install shadow-utils sudo make gcc findutils diffutils perl python3 gawk gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null
++ dnf -yq install shadow-utils sudo make gcc findutils diffutils perl python3 gawk gettext zlib-devel expat-devel openssl-devel curl curl-devel pcre2-devel >/dev/null
+ ;;
+ ubuntu-*|i386/ubuntu-*|debian-*)
+ # Required so that apt doesn't wait for user input on certain packages.
+@@ ci/install-dependencies.sh: ubuntu-*|i386/ubuntu-*|debian-*)
+ sudo apt-get -q update
+ sudo apt-get -q -y install \
+ $LANGUAGES apache2 cvs cvsps git gnupg $SVN \
+- make libssl-dev libcurl4-openssl-dev libexpat-dev wget sudo default-jre \
+- tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl \
++ make libssl-dev curl libcurl4-openssl-dev libexpat-dev wget sudo default-jre \
++ tcl tk gettext zlib1g zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl \
+ libemail-valid-perl libio-pty-perl libio-socket-ssl-perl libnet-smtp-ssl-perl libdbd-sqlite3-perl libcgi-pm-perl \
+ libsecret-1-dev libpcre2-dev meson ninja-build pkg-config \
+ ${CC_PACKAGE:-${CC:-gcc}} $PYTHON_PACKAGE
+@@ ci/install-dependencies.sh: ClangFormat)
+ ;;
+ StaticAnalysis)
+ sudo apt-get -q update
+- sudo apt-get -q -y install coccinelle libcurl4-openssl-dev libssl-dev \
++ sudo apt-get -q -y install coccinelle curl libcurl4-openssl-dev libssl-dev \
+ libexpat-dev gettext make
+ ;;
+ sparse)
+ sudo apt-get -q update -q
+- sudo apt-get -q -y install libssl-dev libcurl4-openssl-dev \
+- libexpat-dev gettext zlib1g-dev sparse
++ sudo apt-get -q -y install libssl-dev curl libcurl4-openssl-dev \
++ libexpat-dev gettext zlib1g zlib1g-dev sparse
+ ;;
+ Documentation)
+ sudo apt-get -q update
+
+ ## ci/install-rust-toolchain.sh (new) ##
+@@
++#!/bin/sh
++
++if [ "$CARGO_HOME" = "" ]; then
++ echo >&2 "::error:: CARGO_HOME is not set"
++ exit 2
++fi
++export PATH="$CARGO_HOME/bin:$PATH"
++rustup -vV || exit $?
++
++## Enforce the correct Rust toolchain
++rustup override unset || true
++
++## install a specific version of rust
++if [ "$RUST_TARGET" != "" ]; then
++ rustup default --force-non-host "$RUST_VERSION-$RUST_TARGET" || exit $? ++else
++ rustup default "$RUST_VERSION" || exit $?
++fi
++
+rustc -vV || exit $?
-+cargo --version || exit $?
-
- dir_git_root=${0%/*}
- dir_build=$1
-@@ build_rust.sh: dst=$dir_build/$libfile
- if [ "$dir_git_root" != "$dir_build" ]; then
- src=$dir_rust/target/$rust_target/$libfile
- if [ ! -f $src ]; then
-- echo >&2 "::error:: cannot find path of static library"
-+ echo >&2 "::error:: cannot find path of static library $src is not a file or does not exist"
- exit 5
- fi
-
++
++RE_RUST_TARGET="$RUST_TARGET"
++if [ "$RUST_TARGET" = "" ]; then
++ RE_RUST_TARGET="[^ ]+"
++fi
++
++if ! rustup show active-toolchain | grep -E "^$RUST_VERSION-$RE_RUST_TARGET \(default\)$"; then ++ echo >&2 "::error:: wrong Rust toolchain, active-toolchain: $(rustup show active-toolchain)" ++ exit 3
++fi

- ## ci/install-rust.sh (mode change 100644 => 100755) ##
+ ## ci/install-rustup.sh (new) ##
@@
- #!/bin/sh
-
++#!/bin/sh
++
+## github workflows actions-rs/toolchain@v1 doesn't work for docker
+## targets. This script should only be used if the ci pipeline
+## doesn't support installing rust on a particular target.
+
- if [ "$(id -u)" -eq 0 ]; then
- echo >&2 "::warning:: installing rust as root"
- fi
-
--if [ "$CARGO_HOME" = "" ]; then
-- echo >&2 "::warning:: CARGO_HOME is not set"
-- export CARGO_HOME=$HOME/.cargo
--fi
--
--export RUSTUP_HOME=$CARGO_HOME
--
- if [ "$RUST_VERSION" = "" ]; then
- echo >&2 "::error:: RUST_VERSION is not set"
-+ exit 1
++if [ "$(id -u)" -eq 0 ]; then
++ echo >&2 "::warning:: installing rust as root"
+fi
+
+if [ "$CARGO_HOME" = "" ]; then
+ echo >&2 "::error:: CARGO_HOME is not set"
- exit 2
- fi
-
++ exit 2
++fi
++
+export RUSTUP_HOME=$CARGO_HOME
+
- ## install rustup
- curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --default-toolchain none -y
- if [ ! -f $CARGO_HOME/env ]; then
- echo "PATH=$CARGO_HOME/bin:\$PATH" > $CARGO_HOME/env
- fi
++## install rustup
++curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --default-toolchain none -y ++if [ ! -f $CARGO_HOME/env ]; then
++ echo "PATH=$CARGO_HOME/bin:\$PATH" > $CARGO_HOME/env
++fi
+. $CARGO_HOME/env
+
- ## install a specific version of rust
--if [ "$BITNESS" = "32" ]; then
-- $CARGO_HOME/bin/rustup set default-host i686-unknown-linux-gnu || exit $?
-- $CARGO_HOME/bin/rustup install $RUST_VERSION || exit $?
-- $CARGO_HOME/bin/rustup default --force-non-host $RUST_VERSION || exit $?
-+if [ "$RUST_TARGET" != "" ]; then
-+ rustup default --force-non-host "$RUST_VERSION-$RUST_TARGET" || exit $?
- else
-- $CARGO_HOME/bin/rustup default $RUST_VERSION || exit $?
-- if [ "$CI_OS_NAME" = "windows" ]; then
-- $CARGO_HOME/bin/rustup target add x86_64-pc-windows-gnu || exit $?
-- fi
-+ rustup default "$RUST_VERSION" || exit $?
- fi
-
--. $CARGO_HOME/env
-+rustc -vV || exit $?
++rustup -vV

## ci/lib.sh ##
@@
# Library of functions shared by all CI scripts

-
--export BITNESS="64"
--if command -v getconf >/dev/null && [ "$(getconf LONG_BIT 2>/dev/null)" = "32" ]; then -- export BITNESS="32"
--fi
--echo "BITNESS=$BITNESS"
--
--
++
if test true = "$GITHUB_ACTIONS"
then
begin_group () {
@@ ci/make-test-artifacts.sh: mkdir -p "$1" # in case ci/lib.sh decides to quit ear

. ${0%/*}/lib.sh

--## install rust per user rather than system wide --. ${0%/*}/install-rust.sh
-+if [ -z "$CARGO_HOME" ]; then
++## ensure rustup is in the PATH variable
++if [ "$CARGO_HOME" = "" ]; then
+ echo >&2 "::error:: CARGO_HOME is not set"
-+ exit 1
++ exit 2
+fi
-
--group Build make artifacts-tar ARTIFACTS_DIRECTORY="$1"
+export PATH="$CARGO_HOME/bin:$PATH"
-
--if [ -d "$CARGO_HOME" ]; then
-- rm -rf $CARGO_HOME
--fi
-+group Build make artifacts-tar ARTIFACTS_DIRECTORY="$1" ++
++rustc -vV
++
+ group Build make artifacts-tar ARTIFACTS_DIRECTORY="$1"

check_unignored_build_artifacts

@@ ci/run-build-and-tests.sh

. ${0%/*}/lib.sh

--## install rust per user rather than system wide
--. ${0%/*}/install-rust.sh
-+## actions-rs/toolchain@v1 doesn't work for docker targets. -+if [ "$CI_IS_DOCKER" = "true" ]; then
-+ . ${0%/*}/install-rust.sh
++## ensure rustup is in the PATH variable
++if [ "$CARGO_HOME" = "" ]; then
++ echo >&2 "::error:: CARGO_HOME is not set"
++ exit 2
+fi
-
--rustc -vV
++. $CARGO_HOME/env
++
+rustc -vV || exit $?
- cargo --version || exit $?
-
++
run_tests=t
-
- ## meson.build ##
-@@ meson.build: rust_build_xdiff = custom_target('rust_build_xdiff',
- meson.project_source_root() / 'build_rust.sh',
- meson.current_build_dir(), rust_target, 'xdiff',
- ],
-+ env: script_environment,
- install: false,
- )

+ case "$jobname" in
+@@ ci/run-build-and-tests.sh: case "$jobname" in
+ ;;
+ esac
+
++if [ -d "$CARGO_HOME" ]; then
++ rm -rf $CARGO_HOME
++fi
++
+ check_unignored_build_artifacts
+ save_good_tree
11: 382067a09e3 ! 4: 0d2b39c3e03 win+Meson: do allow linking with the Rust-built xdiff
@@ .github/workflows/main.yml: jobs:
+ /c/Program\ Files/Git/mingw64/bin/curl -Lo libuserenv.a \
+ https://github.com/git-for-windows/git-sdk-64/raw/HEAD/mingw64/lib/libuserenv.a
+ }
- - name: build
- shell: bash
- env:
+ - name: Install rustup via github actions
+ uses: actions-rs/toolchain@v1
+ with:

## config.mak.uname ##
@@ config.mak.uname: ifeq ($(uname_S),MINGW)
-
- export CARGO_BUILD_TARGET
- RUST_TARGET_DIR = rust/target/$(CARGO_BUILD_TARGET)/$(RUST_BUILD_MODE)
+ COMPAT_CFLAGS += -D_USE_32BIT_TIME_T
+ BASIC_LDFLAGS += -Wl,--large-address-aware
+ endif
++
+ # Unfortunately now needed because of Rust
+ EXTLIBS += -luserenv
-
++
CC = gcc
COMPAT_CFLAGS += -D__USE_MINGW_ANSI_STDIO=0 -DDETECT_MSYS_TTY \
+ -fstack-protector-strong

## meson.build ##
@@ meson.build: elif host_machine.system() == 'windows'

 13:  44784f0d672 =  5:  e65488ab993 github workflows: upload Cargo.lock
  -:  ----------- >  6:  db5d22b1887 ivec: create a vector type that is interoperable between C and Rust
  3:  56c96d35554 =  7:  d4bed954632 xdiff/xprepare: remove superfluous forward declarations
  4:  ebec3689dce =  8:  7c68ce5349c xdiff: delete unnecessary fields from xrecord_t and xdfile_t
  5:  769d1a5b9d2 =  9:  e516ccc8c0a xdiff: make fields of xrecord_t Rust friendly
  6:  87623495994 ! 10:  21bfb9f0883 xdiff: separate parsing lines from hashing them
     @@ Metadata
      Author: Ezekiel Newren <[email protected]>
      
       ## Commit message ##
     -    xdiff: separate parsing lines from hashing them
     +    xdiff: use one definition for freeing xdfile_t
      
     -    We want to use xxhash for faster hashing. To facilitate that
     -    and to simplify the code. Separate the concerns of parsing
     -    and hashing into discrete steps. This makes swapping the hash
     -    function much easier. Since xdl_hash_record() both parses and
     -    hashses lines, this requires some slight code restructuring.
     +    Simplify xdl_prepare_ctx() by using xdl_free_ctx() instead of using
     +    local variables with hand rolled memory management.
      
          Signed-off-by: Ezekiel Newren <[email protected]>
      
     @@ xdiff/xprepare.c: static int xdl_classify_record(unsigned int pass, xdlclassifie
       }
       
       
     -+static void xdl_parse_lines(mmfile_t *mf, long narec, xdfile_t *xdf) {
     -+	u8 const* ptr = (u8 const*) mf->ptr;
     -+	usize len = (usize) mf->size;
     -+
     -+	xdf->recs = NULL;
     -+	xdf->nrec = 0;
     -+	XDL_ALLOC_ARRAY(xdf->recs, narec);
     -+
     -+	while (len > 0) {
     -+		xrecord_t *rec = NULL;
     -+		usize length;
     -+		u8 const* result = memchr(ptr, '\n', len);
     -+		if (result) {
     -+			length = result - ptr + 1;
     -+		} else {
     -+			length = len;
     -+		}
     -+		if (XDL_ALLOC_GROW(xdf->recs, xdf->nrec + 1, narec))
     -+			die("XDL_ALLOC_GROW failed");
     -+		rec = xdl_cha_alloc(&xdf->rcha);
     -+		rec->ptr = ptr;
     -+		rec->size = length;
     -+		rec->ha = 0;
     -+		xdf->recs[xdf->nrec++] = rec;
     -+		ptr += length;
     -+		len -= length;
     -+	}
     -+
     ++static void xdl_free_ctx(xdfile_t *xdf) {
     ++	xdl_free(xdf->rindex);
     ++	xdl_free(xdf->rchg - 1);
     ++	xdl_free(xdf->ha);
     ++	xdl_free(xdf->recs);
     ++	xdl_cha_free(&xdf->rcha);
      +}
      +
      +
       static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_t const *xpp,
       			   xdlclassifier_t *cf, xdfile_t *xdf) {
      -	long nrec, bsize;
     --	unsigned long hav;
     --	char const *blk, *cur, *top, *prev;
     --	xrecord_t *crec;
     ++	long bsize;
     + 	unsigned long hav;
     + 	char const *blk, *cur, *top, *prev;
     + 	xrecord_t *crec;
      -	xrecord_t **recs;
     - 	unsigned long *ha;
     - 	char *rchg;
     - 	long *rindex;
     -@@ xdiff/xprepare.c: static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_
     - 	ha = NULL;
     - 	rindex = NULL;
     - 	rchg = NULL;
     +-	unsigned long *ha;
     +-	char *rchg;
     +-	long *rindex;
     + 
     +-	ha = NULL;
     +-	rindex = NULL;
     +-	rchg = NULL;
      -	recs = NULL;
     ++	xdf->ha = NULL;
     ++	xdf->rindex = NULL;
     ++	xdf->rchg = NULL;
     ++	xdf->recs = NULL;
     ++	xdf->nrec = 0;
       
       	if (xdl_cha_init(&xdf->rcha, sizeof(xrecord_t), narec / 4 + 1) < 0)
       		goto abort;
      -	if (!XDL_ALLOC_ARRAY(recs, narec))
     --		goto abort;
     ++	if (!XDL_ALLOC_ARRAY(xdf->recs, narec))
     + 		goto abort;
       
      -	nrec = 0;
     --	if ((cur = blk = xdl_mmfile_first(mf, &bsize))) {
     --		for (top = blk + bsize; cur < top; ) {
     --			prev = cur;
     --			hav = xdl_hash_record(&cur, top, xpp->flags);
     + 	if ((cur = blk = xdl_mmfile_first(mf, &bsize))) {
     + 		for (top = blk + bsize; cur < top; ) {
     + 			prev = cur;
     + 			hav = xdl_hash_record(&cur, top, xpp->flags);
      -			if (XDL_ALLOC_GROW(recs, nrec + 1, narec))
     --				goto abort;
     --			if (!(crec = xdl_cha_alloc(&xdf->rcha)))
     --				goto abort;
     --			crec->ptr = (u8 const*) prev;
     --			crec->size = (long) (cur - prev);
     --			crec->ha = hav;
     ++			if (XDL_ALLOC_GROW(xdf->recs, xdf->nrec + 1, narec))
     + 				goto abort;
     + 			if (!(crec = xdl_cha_alloc(&xdf->rcha)))
     + 				goto abort;
     + 			crec->ptr = (u8 const*) prev;
     + 			crec->size = (long) (cur - prev);
     + 			crec->ha = hav;
      -			recs[nrec++] = crec;
     --			if (xdl_classify_record(pass, cf, crec) < 0)
     --				goto abort;
     --		}
     -+	xdl_parse_lines(mf, narec, xdf);
     -+
     -+	for (usize i = 0; i < (usize) xdf->nrec; i++) {
     -+		xrecord_t *rec = xdf->recs[i];
     -+		char const* dump = (char const*) rec->ptr;
     -+		rec->ha = xdl_hash_record(&dump, (char const*) (rec->ptr + rec->size), xpp->flags);
     -+		xdl_classify_record(pass, cf, rec);
     ++			xdf->recs[xdf->nrec++] = crec;
     + 			if (xdl_classify_record(pass, cf, crec) < 0)
     + 				goto abort;
     + 		}
       	}
       
      -	if (!XDL_CALLOC_ARRAY(rchg, nrec + 2))
     -+
     -+	if (!XDL_CALLOC_ARRAY(rchg, xdf->nrec + 2))
     ++	if (!XDL_CALLOC_ARRAY(xdf->rchg, xdf->nrec + 2))
       		goto abort;
       
       	if ((XDF_DIFF_ALG(xpp->flags) != XDF_PATIENCE_DIFF) &&
       	    (XDF_DIFF_ALG(xpp->flags) != XDF_HISTOGRAM_DIFF)) {
      -		if (!XDL_ALLOC_ARRAY(rindex, nrec + 1))
     -+		if (!XDL_ALLOC_ARRAY(rindex, xdf->nrec + 1))
     ++		if (!XDL_ALLOC_ARRAY(xdf->rindex, xdf->nrec + 1))
       			goto abort;
      -		if (!XDL_ALLOC_ARRAY(ha, nrec + 1))
     -+		if (!XDL_ALLOC_ARRAY(ha, xdf->nrec + 1))
     ++		if (!XDL_ALLOC_ARRAY(xdf->ha, xdf->nrec + 1))
       			goto abort;
       	}
       
      -	xdf->nrec = nrec;
      -	xdf->recs = recs;
     - 	xdf->rchg = rchg + 1;
     - 	xdf->rindex = rindex;
     +-	xdf->rchg = rchg + 1;
     +-	xdf->rindex = rindex;
     ++	xdf->rchg += 1;
       	xdf->nreff = 0;
     - 	xdf->ha = ha;
     +-	xdf->ha = ha;
       	xdf->dstart = 0;
      -	xdf->dend = nrec - 1;
      +	xdf->dend = xdf->nrec - 1;
       
       	return 0;
       
     -@@ xdiff/xprepare.c: abort:
     - 	xdl_free(ha);
     - 	xdl_free(rindex);
     - 	xdl_free(rchg);
     + abort:
     +-	xdl_free(ha);
     +-	xdl_free(rindex);
     +-	xdl_free(rchg);
      -	xdl_free(recs);
     -+	xdl_free(xdf->recs);
     - 	xdl_cha_free(&xdf->rcha);
     +-	xdl_cha_free(&xdf->rcha);
     ++	xdl_free_ctx(xdf);
       	return -1;
       }
     + 
     + 
     +-static void xdl_free_ctx(xdfile_t *xdf) {
     +-	xdl_free(xdf->rindex);
     +-	xdl_free(xdf->rchg - 1);
     +-	xdl_free(xdf->ha);
     +-	xdl_free(xdf->recs);
     +-	xdl_cha_free(&xdf->rcha);
     +-}
     +-
     +-
     + void xdl_free_env(xdfenv_t *xe) {
     + 
     + 	xdl_free_ctx(&xe->xdf2);
  7:  d74fd4ef67a <  -:  ----------- xdiff: conditionally use Rust's implementation of xxhash
  9:  96041a10d54 <  -:  ----------- Do support Windows again after requiring Rust
 10:  1194de3f39c <  -:  ----------- win+Meson: allow for xdiff to be compiled with MSVC
 14:  f20efdff7aa <  -:  ----------- xdiff: implement a white space iterator in Rust
  -:  ----------- > 11:  6ce0e252b38 xdiff: replace chastore with an ivec in xdfile_t
  -:  ----------- > 12:  0cfc6cf26b7 xdiff: delete nrec field from xdfile_t
  -:  ----------- > 13:  cf0387d851c xdiff: delete recs field from xdfile_t
  -:  ----------- > 14:  ea699135f95 xdiff: make xdfile_t more rust friendly
 15:  c8d41173274 ! 15:  b18544b74f3 xdiff: create line_hash() and line_equal()
     @@ Metadata
      Author: Ezekiel Newren <[email protected]>
      
       ## Commit message ##
     -    xdiff: create line_hash() and line_equal()
     +    xdiff: implement xdl_trim_ends() in Rust
      
     -    These functions use the whitespace iterator, when applicable, to hash,
     -    and compare lines.
     +    Replace the C implementation of xdl_trim_ends() with a Rust
     +    implementation.
      
          Signed-off-by: Ezekiel Newren <[email protected]>
      
       ## rust/xdiff/src/lib.rs ##
      @@
     -+use std::hash::Hasher;
     -+use xxhash_rust::xxh3::Xxh3Default;
     -+use crate::xutils::*;
     -+
     - pub mod xutils;
     ++pub mod xprepare;
     ++pub mod xtypes;
       
     - pub const XDF_IGNORE_WHITESPACE: u64 = 1 << 1;
     -@@ rust/xdiff/src/lib.rs: unsafe extern "C" fn xxh3_64(ptr: *const u8, size: usize) -> u64 {
     -     let slice = std::slice::from_raw_parts(ptr, size);
     -     xxhash_rust::xxh3::xxh3_64(slice)
     - }
     ++use crate::xprepare::trim_ends;
     ++use crate::xtypes::xdfile;
      +
      +#[no_mangle]
     -+unsafe extern "C" fn xdl_line_hash(ptr: *const u8, size: usize, flags: u64) -> u64 {
     -+    let line = std::slice::from_raw_parts(ptr, size);
     -+
     -+    line_hash(line, flags)
     -+}
     ++unsafe extern "C" fn xdl_trim_ends(xdf1: *mut xdfile, xdf2: *mut xdfile) -> i32 {
     ++    let xdf1 = xdf1.as_mut().expect("null pointer");
     ++    let xdf2 = xdf2.as_mut().expect("null pointer");
      +
     -+#[no_mangle]
     -+unsafe extern "C" fn xdl_line_equal(lhs: *const u8, lhs_len: usize, rhs: *const u8, rhs_len: usize, flags: u64) -> bool {
     -+    let lhs_line = std::slice::from_raw_parts(lhs, lhs_len);
     -+    let rhs_line = std::slice::from_raw_parts(rhs, rhs_len);
     ++    trim_ends(xdf1, xdf2);
      +
     -+    line_equal(lhs_line, rhs_line, flags)
     ++    0
      +}
      
     - ## rust/xdiff/src/xutils.rs ##
     + ## rust/xdiff/src/xprepare.rs (new) ##
      @@
     - use crate::*;
     -+use xxhash_rust::xxh3::xxh3_64;
     - 
     - pub(crate) fn xdl_isspace(v: u8) -> bool {
     -     match v {
     -@@ rust/xdiff/src/xutils.rs: where
     -     run_option0.is_none() && run_option1.is_none()
     - }
     - 
     ++use crate::xtypes::xdfile;
      +
     -+pub fn line_hash(line: &[u8], flags: u64) -> u64 {
     -+    if (flags & XDF_WHITESPACE_FLAGS) == 0 {
     -+        return xxh3_64(line);
     -+    }
     ++///
     ++/// Early trim initial and terminal matching records.
     ++///
     ++pub(crate) fn trim_ends(xdf1: &mut xdfile, xdf2: &mut xdfile) {
     ++    let mut lim = std::cmp::min(xdf1.record.len(), xdf2.record.len());
      +
     -+    let mut hasher = Xxh3Default::new();
     -+    for chunk in WhitespaceIter::new(line, flags) {
     -+        hasher.update(chunk);
     ++    for i in 0..lim {
     ++        if xdf1.record[i].ha != xdf2.record[i].ha {
     ++            xdf1.dstart = i as isize;
     ++            xdf2.dstart = i as isize;
     ++            lim -= i;
     ++            break;
     ++        }
      +    }
      +
     -+    hasher.finish()
     -+}
     -+
     -+
     -+pub fn line_equal(lhs: &[u8], rhs: &[u8], flags: u64) -> bool {
     -+    if (flags & XDF_WHITESPACE_FLAGS) == 0 {
     -+        return lhs == rhs;
     ++    for i in 0..lim {
     ++        let f1i = xdf1.record.len() - 1 - i;
     ++        let f2i = xdf2.record.len() - 1 - i;
     ++        if xdf1.record[f1i].ha != xdf2.record[f2i].ha {
     ++            xdf1.dend = f1i as isize;
     ++            xdf2.dend = f2i as isize;
     ++            break;
     ++        }
      +    }
     -+
     -+    let lhs_it = WhitespaceIter::new(lhs, flags);
     -+    let rhs_it = WhitespaceIter::new(rhs, flags);
     -+
     -+    chunked_iter_equal(lhs_it, rhs_it)
      +}
     +
     + ## rust/xdiff/src/xtypes.rs (new) ##
     +@@
     ++use interop::ivec::IVec;
      +
     ++#[repr(C)]
     ++pub(crate) struct xrecord {
     ++    pub(crate) ptr: *const u8,
     ++    pub(crate) size: usize,
     ++    pub(crate) ha: u64,
     ++}
      +
     - #[cfg(test)]
     - mod tests {
     -     use crate::*;
     ++#[repr(C)]
     ++pub(crate) struct xdfile {
     ++    pub(crate) record: IVec<xrecord>,
     ++    pub(crate) dstart: isize,
     ++    pub(crate) dend: isize,
     ++    pub(crate) rchg: *mut u8,
     ++    pub(crate) rindex: *mut usize,
     ++    pub(crate) nreff: usize,
     ++    pub(crate) ha: *mut u64,
     ++}
     +
     + ## xdiff/xprepare.c ##
     +@@ xdiff/xprepare.c: static int xdl_cleanup_records(xdlclassifier_t *cf, xdfile_t *xdf1, xdfile_t *xd
     + }
     + 
     + 
     +-/*
     +- * Early trim initial and terminal matching records.
     +- */
     +-static int xdl_trim_ends(xdfile_t *xdf1, xdfile_t *xdf2) {
     +-	long i, lim;
     +-	xrecord_t *recs1, *recs2;
     +-
     +-	recs1 = xdf1->record.ptr;
     +-	recs2 = xdf2->record.ptr;
     +-	for (i = 0, lim = XDL_MIN(xdf1->record.length, xdf2->record.length); i < lim;
     +-	     i++, recs1++, recs2++)
     +-		if (recs1->ha != recs2->ha)
     +-			break;
     +-
     +-	xdf1->dstart = xdf2->dstart = i;
     +-
     +-	recs1 = xdf1->record.ptr + xdf1->record.length - 1;
     +-	recs2 = xdf2->record.ptr + xdf2->record.length - 1;
     +-	for (lim -= i, i = 0; i < lim; i++, recs1--, recs2--)
     +-		if (recs1->ha != recs2->ha)
     +-			break;
     +-
     +-	xdf1->dend = xdf1->record.length - i - 1;
     +-	xdf2->dend = xdf2->record.length - i - 1;
     +-
     +-	return 0;
     +-}
     ++extern i32 xdl_trim_ends(xdfile_t *xdf1, xdfile_t *xdf2);
     + 
     + 
     + static int xdl_optimize_ctxs(xdlclassifier_t *cf, xdfile_t *xdf1, xdfile_t *xdf2) {
 16:  f7829c55871 <  -:  ----------- xdiff: optimize case where --ignore-cr-at-eol is the only whitespace flag
 17:  395609aff4b <  -:  ----------- xdiff: use rust's version of whitespace processing

gitgitgadget