Skip to contentSkip to conversation

Windows basic memory management

RFC PATCH Superseded Dmitry Kozlyuk 13 participants 218 messages Mar 30, 2020 – Jun 16, 2020 Page 1 of 3

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Windows basic memory management

This RFC implements basic MM with the following features:

* Hugepages are dynamically allocated in user-mode.
* IOVA is always PA, obtained through kernel-mode driver.
* No 32-bit support (presumably not demanded).
* No multi-process support. Note that without --in-memory EAL MM will create files in current working (runtime) directory.
* No-huge mode for testing with no IOVA available. IOVA could be obtained using Address Windowing Extensions, but it is doubtfully demanded.

Roadmap for Windows [1] proposes that memory management (MM) should be implemented in basic and advanced stages:

1. Basic MM must be sufficient for core libraries and network PMDs:
2. Advanced MM could address features missing from basic stage.

Advanced memory management discussion is out of scope of this RFC. Windows community calls suggest is will be focused on security and IOMMU support. I will post a separate thread with background and suggestions. Cc'ing Dmitry Malloy (MESHCHANINOV) nevertheless.

Because netUIO is not yet committed to dpdk-kmods, the first commit introduces a new simple driver, virt2phys. It will almost certainly become a part of netUIO once it is available for patches. Until then, it must be installed according to documentation provided with the patch. User-mode code locates the driver interface by GUID, so transition from virt2phys to netUIO should not require changes to DPDK.

Hugepages allocation on Windows requires some privilege setup. Please refer to documentation provided in the "initialize hugepage info" patch.

New EAL public functions for memory mapping are introduced. Their implementation for Linux and FreeBSD is identical. The upcoming patch series reorganizing EAL directories will help fixing that [2].

Windows MM duplicates quite a lot of code from Linux EAL:

* eal_memalloc_alloc_seg_bulk
* eal_memalloc_free_seg_bulk
* calc_num_pages_per_socket
* rte_eal_hugepage_init

Need input if it should be left as-is to evolve independently, or some "common to memory hot-plug" code should be factored out. This duplication may be reduced naturally when advanced MM is implemented.

Notes on checkpatch warnings:

* No space after comma / no space before closing parent in macros--- definitely a false-positive, unclear how to suppress this.

* Issues from imported BSD code---probably should be ignored?

* Checkpatch is not run against dpdk-kmods (Windows drivers).

[1]: http://core.dpdk.org/roadmap/windows/
[2]: https://patchwork.dpdk.org/project/dpdk/list/?series=9070

Dmitry Kozlyuk (8):
eal/windows: do not expose private EAL facilities eal/windows: improve CPU and NUMA node detection eal/windows: initialize hugepage info
eal: introduce internal wrappers for file operations
eal: introduce memory management wrappers
eal/windows: fix rte_page_sizes with Clang on Windows eal/windows: replace sys/queue.h with a complete one from FreeBSD eal/windows: implement basic memory management

 config/meson.build                            |   12 +-
 doc/guides/windows_gsg/build_dpdk.rst         |   20 -
 doc/guides/windows_gsg/index.rst              |    1 +
 doc/guides/windows_gsg/run_apps.rst           |   47 +
 lib/librte_eal/common/eal_common_fbarray.c    |   57 +-
 lib/librte_eal/common/eal_common_memory.c     |   50 +-
 lib/librte_eal/common/eal_private.h           |  116 +-
 lib/librte_eal/common/include/rte_memory.h    |   69 +
 lib/librte_eal/common/malloc_heap.c           |    1 +
 lib/librte_eal/freebsd/eal/eal.c              |   40 +
 lib/librte_eal/freebsd/eal/eal_memory.c       |  118 +-
 lib/librte_eal/linux/eal/eal.c                |   40 +
 lib/librte_eal/linux/eal/eal_memory.c         |  117 ++
 lib/librte_eal/meson.build                    |    4 +
 lib/librte_eal/rte_eal_exports.def            |  119 ++
 lib/librte_eal/rte_eal_version.map            |    4 +
 lib/librte_eal/windows/eal/eal.c              |  152 +++
 lib/librte_eal/windows/eal/eal_hugepages.c    |  108 ++
 lib/librte_eal/windows/eal/eal_lcore.c        |  187 ++-
 lib/librte_eal/windows/eal/eal_memalloc.c     |  423 ++++++
 lib/librte_eal/windows/eal/eal_memory.c       | 1166 +++++++++++++++++
 lib/librte_eal/windows/eal/eal_mp.c           |  103 ++
 lib/librte_eal/windows/eal/eal_thread.c       |    1 +
 lib/librte_eal/windows/eal/eal_windows.h      |  113 ++
 lib/librte_eal/windows/eal/include/pthread.h  |    2 +
 lib/librte_eal/windows/eal/include/rte_os.h   |   48 +-
 .../windows/eal/include/rte_virt2phys.h       |   34 +
 .../windows/eal/include/rte_windows.h         |   43 +
 .../windows/eal/include/sys/queue.h           |  663 +++++++++-
 lib/librte_eal/windows/eal/include/unistd.h   |    3 +
 lib/librte_eal/windows/eal/meson.build        |   15 +
 31 files changed, 3626 insertions(+), 250 deletions(-)
 create mode 100644 doc/guides/windows_gsg/run_apps.rst
 create mode 100644 lib/librte_eal/windows/eal/eal_hugepages.c
 create mode 100644 lib/librte_eal/windows/eal/eal_memalloc.c
 create mode 100644 lib/librte_eal/windows/eal/eal_memory.c
 create mode 100644 lib/librte_eal/windows/eal/eal_mp.c
 create mode 100644 lib/librte_eal/windows/eal/eal_windows.h
 create mode 100644 lib/librte_eal/windows/eal/include/rte_virt2phys.h
 create mode 100644 lib/librte_eal/windows/eal/include/rte_windows.h
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/1 virt2phys: virtual to physical address translator for Windows

This patch is for dpdk-kmods tree.

This driver supports Windows EAL memory management by translating current process virtual addresses to physical addresses (IOVA). Standalone virt2phys allows using DPDK without PMD and provides a reference implementation. UIO drivers might also implement virt2phys interface, thus rendering this separate driver unneeded.

---
 windows/README.rst                          |  79 +++++++
 windows/virt2phys/virt2phys.c               | 129 +++++++++++
 windows/virt2phys/virt2phys.h               |  34 +++
 windows/virt2phys/virt2phys.inf             |  85 ++++++++
 windows/virt2phys/virt2phys.sln             |  27 +++
 windows/virt2phys/virt2phys.vcxproj         | 228 ++++++++++++++++++++
 windows/virt2phys/virt2phys.vcxproj.filters |  36 ++++
 7 files changed, 618 insertions(+)
 create mode 100644 windows/README.rst
 create mode 100755 windows/virt2phys/virt2phys.c
 create mode 100755 windows/virt2phys/virt2phys.h
 create mode 100755 windows/virt2phys/virt2phys.inf
 create mode 100755 windows/virt2phys/virt2phys.sln
 create mode 100755 windows/virt2phys/virt2phys.vcxproj
 create mode 100755 windows/virt2phys/virt2phys.vcxproj.filters
windows/README.rst new file +79 -0
new file mode 100644
index 0000000..84506fa
@@ -0,0 +1,79 @@
+Developing Windows Drivers
+==========================
+
+Prerequisites
+-------------
+
+Building Windows Drivers is only possible on Windows.
+
+1. Visual Studio 2019 Community or Professional Edition
+2. Windows Driver Kit (WDK) for Windows 10, version 1903
+
+Follow the official instructions to obtain all of the above:
+https://docs.microsoft.com/en-us/windows-hardware/drivers/download-the-wdk
+
+
+Build the Drivers
+-----------------
+
+Build from Visual Studio
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Open a solution (``*.sln``) with Visual Studio and build it (Ctrl+Shift+B).
+
+
+Build from Command-Line
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Run "Developer Command Prompt for VS 2019" from the Start menu.
+
+Navigate to the solution directory (with ``*.sln``), then run:
+
+    msbuild
+
+To build a particular combination of configuration and platform:
+
+    msbuild -p:Configuration=Debug;Platform=x64
+
+
+Install the Drivers
+-------------------
+
+Disable Driver Signature Enforcement
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+By default Windows prohibits installing and loading drivers without digital
+signature obtained from Microsoft (read more: `Driver Signing`_).
+For development signature enforcement may be disabled as follows.
+
+In Elevated Command Prompt:
+
+    bcdedit -set loadoptions DISABLE_INTEGRITY_CHECKS
+    bcdedit -set TESTSIGNING ON
+    shutdown -r -t 0
+
+Upon reboot, an overlay message should appear on the desktop informing
+that Windows is in test mode, which means it allows loading unsigned drivers.
+
+.. Driver Signing: https://docs.microsoft.com/en-us/windows-hardware/drivers/install/driver-signing
+
+
+Install, List, and Uninstall Drivers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Driver package is by default located in a subdirectory of its source tree,
+e.g. ``x64\Debug\virt2phys\virt2phys`` (note two levels of ``virt2phys``).
+
+To install the driver and bind associated devices to it:
+
+    pnputil /add-driver x64\Debug\virt2phys\virt2phys\virt2phys.inf /install
+
+A graphical confirmation to load an unsigned driver will still appear.
+
+To list installed drivers:
+
+    pnputil /enum-drivers
+
+To remove the driver package and to uninstall its devices:
+
+    pnputil /delete-drive oem2.inf /install
windows/virt2phys/virt2phys.c new file +129 -0

129 lines not on this page · raw message

windows/virt2phys/virt2phys.h new file +34 -0
new file mode 100755
index 0000000..4bb2b4a
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2020 Dmitry Kozlyuk
+ */
+
+/**
+ * @file virt2phys driver interface
+ */
+
+/**
+ * Driver device interface GUID {539c2135-793a-4926-afec-d3a1b61bbc8a}.
+ */
+DEFINE_GUID(GUID_DEVINTERFACE_VIRT2PHYS,
+	0x539c2135, 0x793a, 0x4926,
+	0xaf, 0xec, 0xd3, 0xa1, 0xb6, 0x1b, 0xbc, 0x8a);
+
+/**
+ * Driver device type for IO control codes.
+ */
+#define VIRT2PHYS_DEVTYPE 0x8000
+
+/**
+ * Translate a valid non-paged virtual address to a physical address.
+ *
+ * Note: A physical address zero (0) is reported if input address
+ * is paged out or not mapped. However, if input is a valid mapping
+ * of I/O port 0x0000, output is also zero. There is no way
+ * to distinguish between these cases by return value only.
+ *
+ * Input: a non-paged virtual address (PVOID).
+ *
+ * Output: the corresponding physical address (LARGE_INTEGER).
+ */
+#define IOCTL_VIRT2PHYS_TRANSLATE CTL_CODE( \
+	VIRT2PHYS_DEVTYPE, 0x800, METHOD_BUFFERED, FILE_ANY_ACCESS)
windows/virt2phys/virt2phys.inf new file +85 -0

85 lines not on this page · raw message

windows/virt2phys/virt2phys.sln new file +27 -0
new file mode 100755
index 0000000..0f5ecdc
@@ -0,0 +1,27 @@
+
+Microsoft Visual Studio Solution File, Format Version 12.00
+# Visual Studio Version 16
+VisualStudioVersion = 16.0.29613.14
+MinimumVisualStudioVersion = 10.0.40219.1
+Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "virt2phys", "virt2phys.vcxproj", "{0EEF826B-9391-43A8-A722-BDD6F6115137}"
+EndProject
+Global
+	GlobalSection(SolutionConfigurationPlatforms) = preSolution
+		Debug|x64 = Debug|x64
+		Release|x64 = Release|x64
+	EndGlobalSection
+	GlobalSection(ProjectConfigurationPlatforms) = postSolution
+		{0EEF826B-9391-43A8-A722-BDD6F6115137}.Debug|x64.ActiveCfg = Debug|x64
+		{0EEF826B-9391-43A8-A722-BDD6F6115137}.Debug|x64.Build.0 = Debug|x64
+		{0EEF826B-9391-43A8-A722-BDD6F6115137}.Debug|x64.Deploy.0 = Debug|x64
+		{0EEF826B-9391-43A8-A722-BDD6F6115137}.Release|x64.ActiveCfg = Release|x64
+		{0EEF826B-9391-43A8-A722-BDD6F6115137}.Release|x64.Build.0 = Release|x64
+		{0EEF826B-9391-43A8-A722-BDD6F6115137}.Release|x64.Deploy.0 = Release|x64
+	EndGlobalSection
+	GlobalSection(SolutionProperties) = preSolution
+		HideSolutionNode = FALSE
+	EndGlobalSection
+	GlobalSection(ExtensibilityGlobals) = postSolution
+		SolutionGuid = {845012FB-4471-4A12-A1C4-FF7E05C40E8E}
+	EndGlobalSection
+EndGlobal
windows/virt2phys/virt2phys.vcxproj new file +228 -0

229 lines not on this page · raw message

windows/virt2phys/virt2phys.vcxproj.filters new file +36 -0
new file mode 100755
index 0000000..0fe65fc
@@ -0,0 +1,36 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+  <ItemGroup>
+    <Filter Include="Source Files">
+      <UniqueIdentifier>{4FC737F1-C7A5-4376-A066-2A32D752A2FF}</UniqueIdentifier>
+      <Extensions>cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx</Extensions>
+    </Filter>
+    <Filter Include="Header Files">
+      <UniqueIdentifier>{93995380-89BD-4b04-88EB-625FBE52EBFB}</UniqueIdentifier>
+      <Extensions>h;hpp;hxx;hm;inl;inc;xsd</Extensions>
+    </Filter>
+    <Filter Include="Resource Files">
+      <UniqueIdentifier>{67DA6AB6-F800-4c08-8B7A-83BB121AAD01}</UniqueIdentifier>
+      <Extensions>rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;png;wav;mfcribbon-ms</Extensions>
+    </Filter>
+    <Filter Include="Driver Files">
+      <UniqueIdentifier>{8E41214B-6785-4CFE-B992-037D68949A14}</UniqueIdentifier>
+      <Extensions>inf;inv;inx;mof;mc;</Extensions>
+    </Filter>
+  </ItemGroup>
+  <ItemGroup>
+    <Inf Include="virt2phys.inf">
+      <Filter>Driver Files</Filter>
+    </Inf>
+  </ItemGroup>
+  <ItemGroup>
+    <ClInclude Include="virt2phys.h">
+      <Filter>Header Files</Filter>
+    </ClInclude>
+  </ItemGroup>
+  <ItemGroup>
+    <ClCompile Include="virt2phys.c">
+      <Filter>Source Files</Filter>
+    </ClCompile>
+  </ItemGroup>
+</Project>
Signature · 1 line
2.25.1
#2 lore raw

Re: [dpdk-dev] [PATCH 1/1] virt2phys: virtual to physical address translator for Windows

Quoted 33 lines · Dmitry Kozlyuk

This patch is for dpdk-kmods tree.

This driver supports Windows EAL memory management by translating current process virtual addresses to physical addresses (IOVA). Standalone virt2phys allows using DPDK without PMD and provides a reference implementation. UIO drivers might also implement virt2phys interface, thus rendering this separate driver unneeded.

---
 windows/README.rst                          |  79 +++++++
 windows/virt2phys/virt2phys.c               | 129 +++++++++++
 windows/virt2phys/virt2phys.h               |  34 +++
 windows/virt2phys/virt2phys.inf             |  85 ++++++++
 windows/virt2phys/virt2phys.sln             |  27 +++
 windows/virt2phys/virt2phys.vcxproj         | 228 ++++++++++++++++++++
 windows/virt2phys/virt2phys.vcxproj.filters |  36 ++++
 7 files changed, 618 insertions(+)
 create mode 100644 windows/README.rst
 create mode 100755 windows/virt2phys/virt2phys.c
 create mode 100755 windows/virt2phys/virt2phys.h
 create mode 100755 windows/virt2phys/virt2phys.inf
 create mode 100755 windows/virt2phys/virt2phys.sln
 create mode 100755 windows/virt2phys/virt2phys.vcxproj
 create mode 100755 windows/virt2phys/virt2phys.vcxproj.filters
windows/README.rst new file +0 -0
new file mode 100644
index 0000000..84506fa

windows/README.rst line 10

Quoted 10 lines
windows/README.rst new file +10 -0
@@ -0,0 +1,79 @@
+Developing Windows Drivers
+==========================
+
+Prerequisites
+-------------
+
+Building Windows Drivers is only possible on Windows.
+
+1. Visual Studio 2019 Community or Professional Edition
+2. Windows Driver Kit (WDK) for Windows 10, version 1903

Looks like WDK is licence is "Proprietary commercial software".

I am just wondering, Can we package this driver in binary form for end-users to whose scope
is to not develop Windows DPDK but fixing any issues reported by build CI for Window due
to generic EAL patches or so.

Quoted 4 lines
windows/README.rst new file +4 -0
+
+Follow the official instructions to obtain all of the above:
+https://docs.microsoft.com/en-us/windows-hardware/drivers/download-the-wdk
+
#3 lore raw

Re: [dpdk-dev] [PATCH 1/1] virt2phys: virtual to physical address translator for Windows

Quoted 13 lines

windows/README.rst line 10

Quoted 10 lines
windows/README.rst new file +10 -0
+Developing Windows Drivers
+==========================
+
+Prerequisites
+-------------
+
+Building Windows Drivers is only possible on Windows.
+
+1. Visual Studio 2019 Community or Professional Edition
+2. Windows Driver Kit (WDK) for Windows 10, version 1903  

Looks like WDK is licence is "Proprietary commercial software".

Quoted 5 lines

I am just wondering, Can we package this driver in binary form for end-users to whose scope
is to not develop Windows DPDK but fixing any issues reported by build CI for Window due
to generic EAL patches or so.

As cover letter says, this driver should not be required in the end. For netUIO, Harini announced at Windows community call 2020-03-14 that it will be freely available in binary form hosted by Microsoft. In general, publishing driver binaries is OK, take virtio for example:
https://www.linux-kvm.org/page/WindowsGuestDrivers/Download_Drivers
Driver signing is a different matter, but CI probably doesn't require it.

Signature · 1 line
Dmitry Kozlyuk
#4 lore raw

Re: [dpdk-dev] [PATCH 1/1] virt2phys: virtual to physical address translator for Windows

In reply to Dmitry Kozlyuk
Quoted 10 lines · Dmitry Kozlyuk

This patch is for dpdk-kmods tree.

This driver supports Windows EAL memory management by translating current process virtual addresses to physical addresses (IOVA). Standalone virt2phys allows using DPDK without PMD and provides a reference implementation. UIO drivers might also implement virt2phys interface, thus rendering this separate driver unneeded.

---

<Snip!>

windows/virt2phys/virt2phys.c line 82

Quoted 9 lines
windows/virt2phys/virt2phys.c new file +9 -0
+
+_Use_decl_annotations_
+VOID
+virt2phys_device_EvtIoInCallerContext(
+	IN WDFDEVICE device, IN WDFREQUEST request)
+{
+	WDF_REQUEST_PARAMETERS params;
+	ULONG code;
+	PVOID *virt;

Should this be PVOID virt; (instead of PVOID *virt)? If so, changes will be required to parameters passed in to WdfRequestRetrieveInputBuffer() call.

Quoted 25 lines
windows/virt2phys/virt2phys.c new file +25 -0
+	PHYSICAL_ADDRESS *phys;
+	size_t size;
+	NTSTATUS status;
+
+	UNREFERENCED_PARAMETER(device);
+
+	PAGED_CODE();
+
+	WDF_REQUEST_PARAMETERS_INIT(&params);
+	WdfRequestGetParameters(request, &params);
+
+	if (params.Type != WdfRequestTypeDeviceControl) {
+		KdPrint(("bogus request type=%u\n", params.Type));
+		WdfRequestComplete(request, STATUS_NOT_SUPPORTED);
+		return;
+	}
+
+	code = params.Parameters.DeviceIoControl.IoControlCode;
+	if (code != IOCTL_VIRT2PHYS_TRANSLATE) {
+		KdPrint(("bogus IO control code=%lu\n", code));
+		WdfRequestComplete(request, STATUS_NOT_SUPPORTED);
+		return;
+	}
+
+	status = WdfRequestRetrieveInputBuffer(

windows/virt2phys/virt2phys.c line 117

Quoted 10 lines
windows/virt2phys/virt2phys.c new file +10 -0
+			request, sizeof(*virt), (PVOID *)&virt, &size);
+	if (!NT_SUCCESS(status)) {
+		KdPrint(("WdfRequestRetrieveInputBuffer() failed, "
+			"status=%08x\n", status));
+		WdfRequestComplete(request, status);
+		return;
+	}
+
+	status = WdfRequestRetrieveOutputBuffer(
+		request, sizeof(*phys), &phys, &size);

Better to put a (PVOID *)typecast for &phys here:

	status = WdfRequestRetrieveOutputBuffer(
		request, sizeof(*phys), (PVOID *)&phys, &size);

windows/virt2phys/virt2phys.c line 129

Quoted 12 lines
windows/virt2phys/virt2phys.c new file +12 -0
+	if (!NT_SUCCESS(status)) {
+		KdPrint(("WdfRequestRetrieveOutputBuffer() failed, "
+			"status=%08x\n", status));
+		WdfRequestComplete(request, status);
+		return;
+	}
+
+	*phys = MmGetPhysicalAddress(*virt);
+
+	WdfRequestCompleteWithInformation(
+		request, STATUS_SUCCESS, sizeof(*phys));
+}

<Snip!>

Co-installers are no longer required (and discouraged) as per Microsoft. So you can remove the lines indicated below from the .inf file.

Quoted 10 lines
windows/virt2phys/virt2phys.inf new file +4 -0
new file mode 100755
index 0000000..e8adaac
@@ -0,0 +1,85 @@
+; SPDX-License-Identifier: BSD-3-Clause
+; Copyright (c) 2020 Dmitry Kozlyuk
+
+[Version]

windows/virt2phys/virt2phys.inf line 14

Quoted 10 lines
windows/virt2phys/virt2phys.inf new file +10 -0
+Signature = "$WINDOWS NT$"
+Class = %ClassName%
+ClassGuid = {78A1C341-4539-11d3-B88D-00C04FAD5171}
+Provider = %ManufacturerName%
+CatalogFile = virt2phys.cat
+DriverVer =
+
+[DestinationDirs]
+DefaultDestDir = 12
+virt2phys_Device_CoInstaller_CopyFiles = 11

Remove this line

Quoted 6 lines
windows/virt2phys/virt2phys.inf new file +6 -0
+
+; ================= Class section =====================
+
+[ClassInstall32]
+Addreg = virt2phys_ClassReg
+

windows/virt2phys/virt2phys.inf line 30

Quoted 10 lines
windows/virt2phys/virt2phys.inf new file +10 -0
+[virt2phys_ClassReg]
+HKR,,,0,%ClassName%
+HKR,,Icon,,-5
+
+[SourceDisksNames]
+1 = %DiskName%,,,""
+
+[SourceDisksFiles]
+virt2phys.sys  = 1,,
+WdfCoInstaller$KMDFCOINSTALLERVERSION$.dll = 1

Remove this line

Quoted 19 lines
windows/virt2phys/virt2phys.inf new file +19 -0
+
+;*****************************************
+; Install Section
+;*****************************************
+
+[Manufacturer]
+%ManufacturerName%=Standard,NT$ARCH$
+
+[Standard.NT$ARCH$]
+%virt2phys.DeviceDesc%=virt2phys_Device, Root\virt2phys
+
+[virt2phys_Device.NT]
+CopyFiles = Drivers_Dir
+
+[Drivers_Dir]
+virt2phys.sys
+
+;-------------- Service installation
+[virt2phys_Device.NT.Services]

windows/virt2phys/virt2phys.inf line 59

Quoted 10 lines
windows/virt2phys/virt2phys.inf new file +10 -0
+AddService = virt2phys,%SPSVCINST_ASSOCSERVICE%, virt2phys_Service_Inst
+
+; -------------- virt2phys driver install sections
+[virt2phys_Service_Inst]
+DisplayName    = %virt2phys.SVCDESC%
+ServiceType    = 1 ; SERVICE_KERNEL_DRIVER
+StartType      = 3 ; SERVICE_DEMAND_START
+ErrorControl   = 1 ; SERVICE_ERROR_NORMAL
+ServiceBinary  = %12%\virt2phys.sys
+

Remove entire co-installer section below

Quoted 4 lines
windows/virt2phys/virt2phys.inf new file +4 -0
+;
+;--- virt2phys_Device Coinstaller installation ------
+;
+

windows/virt2phys/virt2phys.inf line 73

Quoted 10 lines
windows/virt2phys/virt2phys.inf new file +10 -0
+[virt2phys_Device.NT.CoInstallers]
+AddReg = virt2phys_Device_CoInstaller_AddReg
+CopyFiles = virt2phys_Device_CoInstaller_CopyFiles
+
+[virt2phys_Device_CoInstaller_AddReg]
+HKR,,CoInstallers32,0x00010000, "WdfCoInstaller$KMDFCOINSTALLERVERSION$.dll,WdfCoInstaller"
+
+[virt2phys_Device_CoInstaller_CopyFiles]
+WdfCoInstaller$KMDFCOINSTALLERVERSION$.dll
+

Remove up to here

windows/virt2phys/virt2phys.inf line 85

Quoted 12 lines
windows/virt2phys/virt2phys.inf new file +12 -0
+[virt2phys_Device.NT.Wdf]
+KmdfService = virt2phys, virt2phys_wdfsect
+[virt2phys_wdfsect]
+KmdfLibraryVersion = $KMDFVERSION$
+
+[Strings]
+SPSVCINST_ASSOCSERVICE = 0x00000002
+ManufacturerName = "Dmitry Kozlyuk"
+ClassName = "Kernel bypass"
+DiskName = "virt2phys Installation Disk"
+virt2phys.DeviceDesc = "Virtual to physical address translator"
+virt2phys.SVCDESC = "virt2phys Service"

<Snip!>

ranjit m.

#5 lore raw

Re: [dpdk-dev] [PATCH 1/1] virt2phys: virtual to physical address translator for Windows

Quoted 5 lines
Quoted 5 lines
windows/virt2phys/virt2phys.c new file +5 -0
+
+_Use_decl_annotations_
+VOID
+virt2phys_device_EvtIoInCallerContext(
+	IN WDFDEVICE device, IN WDFREQUEST request)
Quoted 8 lines

windows/virt2phys/virt2phys.c line 82

Quoted 4 lines
windows/virt2phys/virt2phys.c new file +4 -0
+{
+	WDF_REQUEST_PARAMETERS params;
+	ULONG code;
+	PVOID *virt;  

Should this be PVOID virt; (instead of PVOID *virt)? If so, changes will be required to parameters passed in to WdfRequestRetrieveInputBuffer() call.

This should be PVOID *virt (pointer to an untyped pointer). User-mode passes a virtual address as a PVOID value, WdfRequestRetrieveInputBuffer() fills virt with the address of that parameter, so that *virt is the virtual address user-mode wants to translate into a physical one.

Quoted 5 lines
Quoted 5 lines
windows/virt2phys/virt2phys.c new file +4 -0
+	PHYSICAL_ADDRESS *phys;
+	size_t size;
+	NTSTATUS status;
+

[snip]

Quoted 7 lines

windows/virt2phys/virt2phys.c line 117

Quoted 3 lines
windows/virt2phys/virt2phys.c new file +3 -0
+
+	status = WdfRequestRetrieveOutputBuffer(
+		request, sizeof(*phys), &phys, &size);  

Better to put a (PVOID *)typecast for &phys here:

	status = WdfRequestRetrieveOutputBuffer(
		request, sizeof(*phys), (PVOID *)&phys, &size); 

What do you mean? Without a typecast the built-in static analyzer emits a warning (and all warnings are treated as errors for a driver):

virt2phys.c(108,46): error C2220: the following warning is treated as an error virt2phys.c(108,46): warning C4047: 'function': 'PVOID *' differs in levels of indirection from 'PVOID **' virt2phys.c(108,46): warning C4022: 'WdfRequestRetrieveInputBuffer': pointer mismatch for actual parameter 3

Quoted 15 lines

windows/virt2phys/virt2phys.c line 129

Quoted 12 lines
windows/virt2phys/virt2phys.c new file +12 -0
+	if (!NT_SUCCESS(status)) {
+		KdPrint(("WdfRequestRetrieveOutputBuffer() failed, "
+			"status=%08x\n", status));
+		WdfRequestComplete(request, status);
+		return;
+	}
+
+	*phys = MmGetPhysicalAddress(*virt);
+
+	WdfRequestCompleteWithInformation(
+		request, STATUS_SUCCESS, sizeof(*phys));
+}  

<Snip!>

Quoted 2 lines

Co-installers are no longer required (and discouraged) as per Microsoft. So you can remove the lines indicated below from the .inf file.

Thanks, will remove in v2. They were generated by WDK solution template.

Signature · 1 line
Dmitry Kozlyuk
#6 lore raw

Re: [dpdk-dev] [PATCH 1/1] virt2phys: virtual to physical address translator for Windows

Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
windows/virt2phys/virt2phys.c new file +4 -0
+	PHYSICAL_ADDRESS *phys;
+	size_t size;
+	NTSTATUS status;
+  

[snip]

Quoted 7 lines

windows/virt2phys/virt2phys.c line 117

Quoted 3 lines
windows/virt2phys/virt2phys.c new file +3 -0
+
+	status = WdfRequestRetrieveOutputBuffer(
+		request, sizeof(*phys), &phys, &size);    

Better to put a (PVOID *)typecast for &phys here:

	status = WdfRequestRetrieveOutputBuffer(
		request, sizeof(*phys), (PVOID *)&phys, &size);   
Quoted 7 lines

What do you mean? Without a typecast the built-in static analyzer emits a warning (and all warnings are treated as errors for a driver):

virt2phys.c(108,46): error C2220: the following warning is treated as an error virt2phys.c(108,46): warning C4047: 'function': 'PVOID *' differs in levels of indirection from 'PVOID **' virt2phys.c(108,46): warning C4022: 'WdfRequestRetrieveInputBuffer': pointer mismatch for actual parameter 3

Never mind, I thought you were referring to the existing typecast in WdfRequestRetrieveInputBuffer(). Thanks for the suggestion, will add in v2.

Signature · 1 line
Dmitry Kozlyuk
#7 lore raw

Re: [dpdk-dev] [PATCH 1/1] virt2phys: virtual to physical address translator for Windows

In reply to Dmitry Kozlyuk
Quoted 14 lines · Dmitry Kozlyuk
Quoted 5 lines
Quoted 5 lines
windows/virt2phys/virt2phys.c new file +5 -0
+
+_Use_decl_annotations_
+VOID
+virt2phys_device_EvtIoInCallerContext(
+	IN WDFDEVICE device, IN WDFREQUEST request)
Quoted 8 lines

windows/virt2phys/virt2phys.c line 82

Quoted 4 lines
windows/virt2phys/virt2phys.c new file +4 -0
+{
+	WDF_REQUEST_PARAMETERS params;
+	ULONG code;
+	PVOID *virt;

Should this be PVOID virt; (instead of PVOID *virt)? If so, changes will be required to parameters passed in to WdfRequestRetrieveInputBuffer() call.

Quoted 5 lines

This should be PVOID *virt (pointer to an untyped pointer). User-mode passes a virtual address as a PVOID value, WdfRequestRetrieveInputBuffer() fills virt with the address of that parameter, so that *virt is the virtual address user-mode wants to translate into a physical one.

Makes sense. Thanks for the explanation, Dmitry.

Quoted 36 lines
Quoted 5 lines
Quoted 5 lines
windows/virt2phys/virt2phys.c new file +4 -0
+	PHYSICAL_ADDRESS *phys;
+	size_t size;
+	NTSTATUS status;
+

[snip]

Quoted 7 lines

windows/virt2phys/virt2phys.c line 117

Quoted 3 lines
windows/virt2phys/virt2phys.c new file +3 -0
+
+	status = WdfRequestRetrieveOutputBuffer(
+		request, sizeof(*phys), &phys, &size);

Better to put a (PVOID *)typecast for &phys here:

	status = WdfRequestRetrieveOutputBuffer(
		request, sizeof(*phys), (PVOID *)&phys, &size);

What do you mean? Without a typecast the built-in static analyzer emits a warning (and all warnings are treated as errors for a driver):

virt2phys.c(108,46): error C2220: the following warning is treated as an error virt2phys.c(108,46): warning C4047: 'function': 'PVOID *' differs in levels of indirection from 'PVOID **' virt2phys.c(108,46): warning C4022: 'WdfRequestRetrieveInputBuffer': pointer mismatch for actual parameter 3

Quoted 15 lines

windows/virt2phys/virt2phys.c line 129

Quoted 12 lines
windows/virt2phys/virt2phys.c new file +12 -0
+	if (!NT_SUCCESS(status)) {
+		KdPrint(("WdfRequestRetrieveOutputBuffer() failed, "
+			"status=%08x\n", status));
+		WdfRequestComplete(request, status);
+		return;
+	}
+
+	*phys = MmGetPhysicalAddress(*virt);
+
+	WdfRequestCompleteWithInformation(
+		request, STATUS_SUCCESS, sizeof(*phys));
+}

<Snip!>

Quoted 5 lines
Quoted 2 lines

Co-installers are no longer required (and discouraged) as per Microsoft. So you can remove the lines indicated below from the .inf file.

Thanks, will remove in v2. They were generated by WDK solution template.

I see the above changes in v2. Thanks!

ranjit m.

#8 lore raw

RFC PATCH 2/9 eal/windows: do not expose private EAL facilities

In reply to Dmitry Kozlyuk

The goal of rte_os.h is to mitigate OS differences for EAL users. In Windows EAL, rte_os.h did excessive things:

1. It included platform SDK headers (windows.h, etc). Those files are huge, require specific inclusion order, and are generally unused by the code including rte_os.h. Declarations from platform SDK may break otherwise platform-independent code, e.g. min, max, ERROR.

2. It included pthread.h, which is clearly not always required.

3. It defined functions private to Windows EAL.

Reorganize Windows EAL includes in the following way:

1. Create rte_windows.h to properly import Windows-specific facilities. Primary users are bus drivers, tests, and external applications.

2. Remove platform SDK includes from rte_os.h to prevent breaking otherwise portable code by including rte_os.h on Windows. Copy necessary definitions to avoid including those headers.

3. Remove pthread.h include from rte_os.h.

4. Move declarations private to Windows EAL into eal_windows.h.

Fixes 428eb983f5f7 ("eal: add OS specific header file")
---
 lib/librte_eal/windows/eal/eal.c              |  2 +
 lib/librte_eal/windows/eal/eal_lcore.c        |  2 +
 lib/librte_eal/windows/eal/eal_thread.c       |  1 +
 lib/librte_eal/windows/eal/eal_windows.h      | 29 ++++++++++++
 lib/librte_eal/windows/eal/include/pthread.h  |  2 +
 lib/librte_eal/windows/eal/include/rte_os.h   | 44 ++++++-------------
 .../windows/eal/include/rte_windows.h         | 41 +++++++++++++++++
 lib/librte_eal/windows/eal/meson.build        |  1 +
 8 files changed, 91 insertions(+), 31 deletions(-)
 create mode 100644 lib/librte_eal/windows/eal/eal_windows.h
 create mode 100644 lib/librte_eal/windows/eal/include/rte_windows.h
lib/librte_eal/windows/eal/eal.c +2 -0
index e4b50df3b..2cf7a04ef 100644
@@ -18,6 +18,8 @@
 #include <eal_options.h>
 #include <eal_private.h>
 
+#include "eal_windows.h"
+
  /* Allow the application to print its usage message too if set */
 static rte_usage_hook_t	rte_application_usage_hook;
 
lib/librte_eal/windows/eal/eal_lcore.c +2 -0
index b3a6c63af..82ee45413 100644
@@ -2,12 +2,14 @@
  * Copyright(c) 2019 Intel Corporation
  */
 
+#include <pthread.h>
 #include <stdint.h>
 
 #include <rte_common.h>
 
 #include "eal_private.h"
 #include "eal_thread.h"
+#include "eal_windows.h"
 
 /* global data structure that contains the CPU map */
 static struct _wcpu_map {
lib/librte_eal/windows/eal/eal_thread.c +1 -0
index 9e4bbaa08..e149199a6 100644
@@ -14,6 +14,7 @@
 #include <eal_thread.h>
 
 #include "eal_private.h"
+#include "eal_windows.h"
 
 RTE_DEFINE_PER_LCORE(unsigned int, _lcore_id) = LCORE_ID_ANY;
 RTE_DEFINE_PER_LCORE(unsigned int, _socket_id) = (unsigned int)SOCKET_ID_ANY;
lib/librte_eal/windows/eal/eal_windows.h new file +29 -0
new file mode 100644
index 000000000..fadd676b2
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2020 Dmitry Kozlyuk
+ */
+
+#ifndef _EAL_WINDOWS_H_
+#define _EAL_WINDOWS_H_
+
+/**
+ * @file Facilities private to Windows EAL
+ */
+
+#include <rte_windows.h>
+
+/**
+ * Create a map of processors and cores on the system.
+ */
+void eal_create_cpu_map(void);
+
+/**
+ * Create a thread.
+ *
+ * @param thread
+ *   The location to store the thread id if successful.
+ * @return
+ *   0 for success, -1 if the thread is not created.
+ */
+int eal_thread_create(pthread_t *thread);
+
+#endif /* _EAL_WINDOWS_H_ */
lib/librte_eal/windows/eal/include/pthread.h +2 -0
index b9dd18e56..cfd53f0b8 100644
@@ -5,6 +5,8 @@
 #ifndef _PTHREAD_H_
 #define _PTHREAD_H_
 
+#include <stdint.h>
+
 /**
  * This file is required to support the common code in eal_common_proc.c,
  * eal_common_thread.c and common\include\rte_per_lcore.h as Microsoft libc
lib/librte_eal/windows/eal/include/rte_os.h +13 -31

75 lines not on this page · raw message

lib/librte_eal/windows/eal/include/rte_windows.h new file +41 -0

41 lines not on this page · raw message

lib/librte_eal/windows/eal/meson.build +1 -0
index 2a062c365..21cd84459 100644
@@ -6,6 +6,7 @@ eal_inc += include_directories('include')
 env_objs = []
 env_headers = files(
 	'include/rte_os.h',
+	'include/rte_windows.h',
 )
 common_sources = files(
 	'../../common/eal_common_bus.c',
Signature · 1 line
2.25.1
#9 lore raw

RFC PATCH 3/9 eal/windows: improve CPU and NUMA node detection

In reply to Dmitry Kozlyuk

1. Map CPU cores to their respective NUMA nodes as reported by system.
2. Support systems with more than 64 cores (multiple processor groups).
3. Fix magic constants, styling issues, and compiler warnings.
4. Add EAL private function to map DPDK socket ID to NUMA node number.

Fixes 53ffd9f080fc ("eal/windows: add minimum viable code")
---
 lib/librte_eal/windows/eal/eal_lcore.c   | 185 ++++++++++++++---------
 lib/librte_eal/windows/eal/eal_windows.h |  10 ++
 2 files changed, 124 insertions(+), 71 deletions(-)
lib/librte_eal/windows/eal/eal_lcore.c +114 -71
index 82ee45413..c11f37de3 100644
@@ -3,103 +3,146 @@
  */
 
 #include <pthread.h>
+#include <stdbool.h>
 #include <stdint.h>
 
 #include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_lcore.h>
+#include <rte_os.h>
 
 #include "eal_private.h"
 #include "eal_thread.h"
 #include "eal_windows.h"
 
-/* global data structure that contains the CPU map */
-static struct _wcpu_map {
-	unsigned int total_procs;
-	unsigned int proc_sockets;
-	unsigned int proc_cores;
-	unsigned int reserved;
-	struct _win_lcore_map {
-		uint8_t socket_id;
-		uint8_t core_id;
-	} wlcore_map[RTE_MAX_LCORE];
-} wcpu_map = { 0 };
-
-/*
- * Create a map of all processors and associated cores on the system
- */
+/** Number of logical processors (cores) in a processor group (32 or 64). */
+#define EAL_PROCESSOR_GROUP_SIZE (sizeof(KAFFINITY) * CHAR_BIT)
+
+struct lcore_map {
+	uint8_t socket_id;
+	uint8_t core_id;
+};
+
+struct socket_map {
+	uint16_t node_id;
+};
+
+struct cpu_map {
+	unsigned int socket_count;
+	unsigned int lcore_count;
+	struct lcore_map lcores[RTE_MAX_LCORE];
+	struct socket_map sockets[RTE_MAX_NUMA_NODES];
+};
+
+static struct cpu_map cpu_map = { 0 };
+
 void
-eal_create_cpu_map()
+eal_create_cpu_map(void)
 {
-	wcpu_map.total_procs =
-		GetActiveProcessorCount(ALL_PROCESSOR_GROUPS);
-
-	LOGICAL_PROCESSOR_RELATIONSHIP lprocRel;
-	DWORD lprocInfoSize = 0;
-	BOOL ht_enabled = FALSE;
-
-	/* First get the processor package information */
-	lprocRel = RelationProcessorPackage;
-	/* Determine the size of buffer we need (pass NULL) */
-	GetLogicalProcessorInformationEx(lprocRel, NULL, &lprocInfoSize);
-	wcpu_map.proc_sockets = lprocInfoSize / 48;
-
-	lprocInfoSize = 0;
-	/* Next get the processor core information */
-	lprocRel = RelationProcessorCore;
-	GetLogicalProcessorInformationEx(lprocRel, NULL, &lprocInfoSize);
-	wcpu_map.proc_cores = lprocInfoSize / 48;
-
-	if (wcpu_map.total_procs > wcpu_map.proc_cores)
-		ht_enabled = TRUE;
-
-	/* Distribute the socket and core ids appropriately
-	 * across the logical cores. For now, split the cores
-	 * equally across the sockets.
-	 */
-	unsigned int lcore = 0;
-	for (unsigned int socket = 0; socket <
-			wcpu_map.proc_sockets; ++socket) {
-		for (unsigned int core = 0;
-			core < (wcpu_map.proc_cores / wcpu_map.proc_sockets);
-			++core) {
-			wcpu_map.wlcore_map[lcore]
-					.socket_id = socket;
-			wcpu_map.wlcore_map[lcore]
-					.core_id = core;
-			lcore++;
-			if (ht_enabled) {
-				wcpu_map.wlcore_map[lcore]
-					.socket_id = socket;
-				wcpu_map.wlcore_map[lcore]
-					.core_id = core;
-				lcore++;
+	SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *infos, *info;
+	DWORD infos_size;
+	bool full = false;
+
+	infos_size = 0;
+	if (!GetLogicalProcessorInformationEx(
+			RelationNumaNode, NULL, &infos_size)) {
+		DWORD error = GetLastError();
+		if (error != ERROR_INSUFFICIENT_BUFFER) {
+			rte_panic("cannot get NUMA node info size, error %lu",
+				GetLastError());
+		}
+	}
+
+	infos = malloc(infos_size);
+	if (infos == NULL) {
+		rte_panic("cannot allocate memory for NUMA node information");
+		return;
+	}
+
+	if (!GetLogicalProcessorInformationEx(
+			RelationNumaNode, infos, &infos_size)) {
+		rte_panic("cannot get NUMA node information, error %lu",
+			GetLastError());
+	}
+
+	info = infos;
+	while ((uint8_t *)info - (uint8_t *)infos < infos_size) {
+		unsigned int node_id = info->NumaNode.NodeNumber;
+		GROUP_AFFINITY *cores = &info->NumaNode.GroupMask;
+		struct lcore_map *lcore;
+		unsigned int i, socket_id;
+
+		/* NUMA node may be reported multiple times if it includes
+		 * cores from different processor groups, e. g. 80 cores
+		 * of a physical processor comprise one NUMA node, but two
+		 * processor groups, because group size is limited by 32/64.
+		 */
+		for (socket_id = 0; socket_id < cpu_map.socket_count;
+		    socket_id++) {
+			if (cpu_map.sockets[socket_id].node_id == node_id)
+				break;
+		}
+
+		if (socket_id == cpu_map.socket_count) {
+			if (socket_id == RTE_DIM(cpu_map.sockets)) {
+				full = true;
+				goto exit;
 			}
+
+			cpu_map.sockets[socket_id].node_id = node_id;
+			cpu_map.socket_count++;
+		}
+
+		for (i = 0; i < EAL_PROCESSOR_GROUP_SIZE; i++) {
+			if ((cores->Mask & ((KAFFINITY)1 << i)) == 0)
+				continue;
+
+			if (cpu_map.lcore_count == RTE_DIM(cpu_map.lcores)) {
+				full = true;
+				goto exit;
+			}
+
+			lcore = &cpu_map.lcores[cpu_map.lcore_count];
+			lcore->socket_id = socket_id;
+			lcore->core_id =
+				cores->Group * EAL_PROCESSOR_GROUP_SIZE + i;
+			cpu_map.lcore_count++;
 		}
+
+		info = (SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *)(
+			(uint8_t *)info + info->Size);
+	}
+
+exit:
+	if (full) {
+		/* RTE_LOG() is not yet available, but this is important. */
+		fprintf(stderr, "Enumerated maximum of %u NUMA nodes and %u cores\n",
+			cpu_map.socket_count, cpu_map.lcore_count);
 	}
+
+	free(infos);
 }
 
-/*
- * Check if a cpu is present by the presence of the cpu information for it
- */
 int
 eal_cpu_detected(unsigned int lcore_id)
 {
-	return (lcore_id < wcpu_map.total_procs);
+	return lcore_id < cpu_map.lcore_count;
 }
 
-/*
- * Get CPU socket id for a logical core
- */
 unsigned
 eal_cpu_socket_id(unsigned int lcore_id)
 {
-	return wcpu_map.wlcore_map[lcore_id].socket_id;
+	return cpu_map.lcores[lcore_id].socket_id;
 }
 
-/*
- * Get CPU socket id (NUMA node) for a logical core
- */
 unsigned
 eal_cpu_core_id(unsigned int lcore_id)
 {
-	return wcpu_map.wlcore_map[lcore_id].core_id;
+	return cpu_map.lcores[lcore_id].core_id;
+}
+
+unsigned int
+eal_socket_numa_node(unsigned int socket_id)
+{
+	return cpu_map.sockets[socket_id].node_id;
 }
lib/librte_eal/windows/eal/eal_windows.h +10 -0
index fadd676b2..390d2fd66 100644
@@ -26,4 +26,14 @@ void eal_create_cpu_map(void);
  */
 int eal_thread_create(pthread_t *thread);
 
+/**
+ * Get system NUMA node number for a socket ID.
+ *
+ * @param socket_id
+ *  Valid EAL socket ID.
+ * @return
+ *  NUMA node number to use with Win32 API.
+ */
+unsigned int eal_socket_numa_node(unsigned int socket_id);
+
 #endif /* _EAL_WINDOWS_H_ */
Signature · 1 line
2.25.1
#10 lore raw

RFC PATCH 4/9 eal/windows: initialize hugepage info

In reply to Dmitry Kozlyuk

Add hugepages discovery ("large pages" in Windows terminology) and update documentation for required privilege setup.

---
 config/meson.build                         |   2 +
 doc/guides/windows_gsg/build_dpdk.rst      |  20 ----
 doc/guides/windows_gsg/index.rst           |   1 +
 doc/guides/windows_gsg/run_apps.rst        |  47 +++++++++
 lib/librte_eal/windows/eal/eal.c           |  14 +++
 lib/librte_eal/windows/eal/eal_hugepages.c | 108 +++++++++++++++++++++
 lib/librte_eal/windows/eal/meson.build     |   1 +
 7 files changed, 173 insertions(+), 20 deletions(-)
 create mode 100644 doc/guides/windows_gsg/run_apps.rst
 create mode 100644 lib/librte_eal/windows/eal/eal_hugepages.c
config/meson.build +2 -0
index abedd76f2..73cf69814 100644
@@ -263,6 +263,8 @@ if is_windows
 	if cc.get_id() == 'gcc'
 		add_project_arguments('-D__USE_MINGW_ANSI_STDIO', language: 'c')
 	endif
+
+	add_project_link_arguments('-ladvapi32', language: 'c')
 endif
 
 if get_option('b_lto')
doc/guides/windows_gsg/build_dpdk.rst +0 -20
index d46e84e3f..650483e3b 100644
@@ -111,23 +111,3 @@ Depending on the distribution, paths in this file may need adjustments.
 
     meson --cross-file config/x86/meson_mingw.txt -Dexamples=helloworld build
     ninja -C build
-
-
-Run the helloworld example
-==========================
-
-Navigate to the examples in the build directory and run `dpdk-helloworld.exe`.
-
-.. code-block:: console
-
-    cd C:\Users\me\dpdk\build\examples
-    dpdk-helloworld.exe
-    hello from core 1
-    hello from core 3
-    hello from core 0
-    hello from core 2
-
-Note for MinGW-w64: applications are linked to ``libwinpthread-1.dll``
-by default. To run the example, either add toolchain executables directory
-to the PATH or copy the library to the working directory.
-Alternatively, static linking may be used (mind the LGPLv2.1 license).
doc/guides/windows_gsg/index.rst +1 -0
index d9b7990a8..e94593572 100644
@@ -12,3 +12,4 @@ Getting Started Guide for Windows
 
     intro
     build_dpdk
+    run_apps
doc/guides/windows_gsg/run_apps.rst new file +47 -0

47 lines not on this page · raw message

lib/librte_eal/windows/eal/eal.c +14 -0
index 2cf7a04ef..a84b6147a 100644
@@ -18,8 +18,11 @@
 #include <eal_options.h>
 #include <eal_private.h>
 
+#include "eal_hugepages.h"
 #include "eal_windows.h"
 
+#define MEMSIZE_IF_NO_HUGE_PAGE (64ULL * 1024ULL * 1024ULL)
+
  /* Allow the application to print its usage message too if set */
 static rte_usage_hook_t	rte_application_usage_hook;
 
@@ -242,6 +245,17 @@ rte_eal_init(int argc, char **argv)
 	if (fctret < 0)
 		exit(1);
 
+	if (!internal_config.no_hugetlbfs && (eal_hugepage_info_init() < 0)) {
+		rte_eal_init_alert("Cannot get hugepage information.");
+		rte_errno = EACCES;
+		return -1;
+	}
+
+	if (internal_config.memory == 0 && !internal_config.force_sockets) {
+		if (internal_config.no_hugetlbfs)
+			internal_config.memory = MEMSIZE_IF_NO_HUGE_PAGE;
+	}
+
 	eal_thread_init_master(rte_config.master_lcore);
 
 	RTE_LCORE_FOREACH_SLAVE(i) {
lib/librte_eal/windows/eal/eal_hugepages.c new file +108 -0

108 lines not on this page · raw message

lib/librte_eal/windows/eal/meson.build +1 -0
index 21cd84459..8b407c9ae 100644
@@ -22,6 +22,7 @@ common_sources = files(
 )
 env_sources = files('eal.c',
 	'eal_debug.c',
+	'eal_hugepages.c',
 	'eal_lcore.c',
 	'eal_thread.c',
 	'getopt.c',
Signature · 1 line
2.25.1
#11 lore raw

RFC PATCH 5/9 eal: introduce internal wrappers for file operations

In reply to Dmitry Kozlyuk

EAL common code uses file locking and truncation. Introduce OS-independent wrapeprs in order to support both POSIX and Windows:

* eal_file_lock: lock or unlock an open file.
* eal_file_truncate: enforce a given size for an open file.

Wrappers follow POSIX semantics, but interface is not POSIX, so that it can be made more clean, e.g. by not mixing locking operation and behaviour on conflict.

WIP2

---
 lib/librte_eal/common/eal_private.h | 45 ++++++++++++++++
 lib/librte_eal/freebsd/eal/eal.c    | 40 ++++++++++++++
 lib/librte_eal/linux/eal/eal.c      | 40 ++++++++++++++
 lib/librte_eal/windows/eal/eal.c    | 83 +++++++++++++++++++++++++++++
 4 files changed, 208 insertions(+)
lib/librte_eal/common/eal_private.h +45 -0
index ddcfbe2e4..0130571e8 100644
@@ -443,4 +443,49 @@ rte_option_usage(void);
 uint64_t
 eal_get_baseaddr(void);
 
+/** File locking operation. */
+enum rte_flock_op {
+	RTE_FLOCK_SHARED,    /**< Acquire a shared lock. */
+	RTE_FLOCK_EXCLUSIVE, /**< Acquire an exclusive lock. */
+	RTE_FLOCK_UNLOCK     /**< Release a previously taken lock. */
+};
+
+/** Behavior on file locking conflict. */
+enum rte_flock_mode {
+	RTE_FLOCK_WAIT,  /**< Wait until the file gets unlocked to lock it. */
+	RTE_FLOCK_RETURN /**< Return immediately if the file is locked. */
+};
+
+/**
+ * Lock or unlock the file.
+ *
+ * On failure @code rte_errno @endcode is set to the error code
+ * specified by POSIX flock(3) description.
+ *
+ * @param fd
+ *  Opened file descriptor.
+ * @param op
+ *  Operation to perform.
+ * @param mode
+ *  Behavior on conflict.
+ * @return
+ *  0 on success, (-1) on failure.
+ */
+int eal_file_lock(int fd, enum rte_flock_op op, enum rte_flock_mode mode);
+
+/**
+ * Truncate or extend the file to the specified size.
+ *
+ * On failure @code rte_errno @endcode is set to the error code
+ * specified by POSIX ftruncate(3) description.
+ *
+ * @param fd
+ *  Opened file descriptor.
+ * @param size
+ *  Desired file size.
+ * @return
+ *  0 on success, (-1) on failure.
+ */
+int eal_file_truncate(int fd, ssize_t size);
+
 #endif /* _EAL_PRIVATE_H_ */
lib/librte_eal/freebsd/eal/eal.c +40 -0
index 6ae37e7e6..4bbcc7ab7 100644
@@ -697,6 +697,46 @@ static void rte_eal_init_alert(const char *msg)
 	RTE_LOG(ERR, EAL, "%s\n", msg);
 }
 
+int
+eal_file_truncate(int fd, ssize_t size)
+{
+	int ret;
+
+	ret = ftruncate(fd, size);
+	if (ret)
+		rte_errno = errno;
+
+	return ret;
+}
+
+int
+eal_file_lock(int fd, enum rte_flock_op op, enum rte_flock_mode mode)
+{
+	int sys_flags = 0;
+	int ret;
+
+	if (mode == RTE_FLOCK_RETURN)
+		sys_flags |= LOCK_NB;
+
+	switch (op) {
+	case RTE_FLOCK_EXCLUSIVE:
+		sys_flags |= LOCK_EX;
+		break;
+	case RTE_FLOCK_SHARED:
+		sys_flags |= LOCK_SH;
+		break;
+	case RTE_FLOCK_UNLOCK:
+		sys_flags |= LOCK_UN;
+		break;
+	}
+
+	ret = flock(fd, sys_flags);
+	if (ret)
+		rte_errno = errno;
+
+	return ret;
+}
+
 /* Launch threads, called at application init(). */
 int
 rte_eal_init(int argc, char **argv)
lib/librte_eal/linux/eal/eal.c +40 -0
index 9530ee55f..d75c162c1 100644
@@ -956,6 +956,46 @@ is_iommu_enabled(void)
 	return n > 2;
 }
 
+int
+eal_file_truncate(int fd, ssize_t size)
+{
+	int ret;
+
+	ret = ftruncate(fd, size);
+	if (ret)
+		rte_errno = errno;
+
+	return ret;
+}
+
+int
+eal_file_lock(int fd, enum rte_flock_op op, enum rte_flock_mode mode)
+{
+	int sys_flags = 0;
+	int ret;
+
+	if (mode == RTE_FLOCK_RETURN)
+		sys_flags |= LOCK_NB;
+
+	switch (op) {
+	case RTE_FLOCK_EXCLUSIVE:
+		sys_flags |= LOCK_EX;
+		break;
+	case RTE_FLOCK_SHARED:
+		sys_flags |= LOCK_SH;
+		break;
+	case RTE_FLOCK_UNLOCK:
+		sys_flags |= LOCK_UN;
+		break;
+	}
+
+	ret = flock(fd, sys_flags);
+	if (ret)
+		rte_errno = errno;
+
+	return ret;
+}
+
 /* Launch threads, called at application init(). */
 int
 rte_eal_init(int argc, char **argv)
lib/librte_eal/windows/eal/eal.c +83 -0

89 lines not on this page · raw message

Signature · 1 line
2.25.1
#12 lore raw

Re: [dpdk-dev] [RFC PATCH 5/9] eal: introduce internal wrappers for file operations

Quoted 31 lines · Dmitry Kozlyuk

EAL common code uses file locking and truncation. Introduce OS-independent wrapeprs in order to support both POSIX and Windows:

* eal_file_lock: lock or unlock an open file.
* eal_file_truncate: enforce a given size for an open file.

Wrappers follow POSIX semantics, but interface is not POSIX, so that it can be made more clean, e.g. by not mixing locking operation and behaviour on conflict.

WIP2

---
 lib/librte_eal/common/eal_private.h | 45 ++++++++++++++++
 lib/librte_eal/freebsd/eal/eal.c    | 40 ++++++++++++++
 lib/librte_eal/linux/eal/eal.c      | 40 ++++++++++++++
 lib/librte_eal/windows/eal/eal.c    | 83 +++++++++++++++++++++++++++++
 4 files changed, 208 insertions(+)
lib/librte_eal/common/eal_private.h +2 -0
index ddcfbe2e4..0130571e8 100644
@@ -443,4 +443,49 @@ rte_option_usage(void);
 uint64_t
 eal_get_baseaddr(void);
 
+/** File locking operation. */
+enum rte_flock_op {

lib/librte_eal/common/eal_private.h rte_option_usage(void); · line 457

Quoted 10 lines
lib/librte_eal/common/eal_private.h +10 -0
+       RTE_FLOCK_SHARED,    /**< Acquire a shared lock. */
+       RTE_FLOCK_EXCLUSIVE, /**< Acquire an exclusive lock. */
+       RTE_FLOCK_UNLOCK     /**< Release a previously taken lock. */
+};
+
+/** Behavior on file locking conflict. */
+enum rte_flock_mode {
+       RTE_FLOCK_WAIT,  /**< Wait until the file gets unlocked to lock it. */
+       RTE_FLOCK_RETURN /**< Return immediately if the file is locked. */
+};

Avoid using RTE_ for internal symbols. IMO, EAL_FLOCK_* would be enough for some something defined in eal_private.h.

#13 lore raw

RFC PATCH 6/9 eal: introduce memory management wrappers

In reply to Dmitry Kozlyuk

System meory management is implemented differently for POSIX and Windows. Introduce wrapper functions for operations used across DPDK:

* rte_mem_map()
Create memory mapping for a regular file or a page file (swap). This supports mapping to a reserved memory region even on Windows.

* rte_mem_unmap()
Remove mapping created with rte_mem_map().

* rte_get_page_size()
Obtain default system page size.

* rte_mem_lock()
Make arbitrary-sized memory region non-swappable.

Wrappers follow POSIX semantics limited to DPDK tasks, but their signatures deliberately differ from POSIX ones to be more safe and expressive.

---
 config/meson.build                         |  10 +-
 lib/librte_eal/common/eal_private.h        |  71 +++-
 lib/librte_eal/common/include/rte_memory.h |  63 +++
 lib/librte_eal/freebsd/eal/eal_memory.c    | 117 ++++++
 lib/librte_eal/linux/eal/eal_memory.c      | 117 ++++++
 lib/librte_eal/rte_eal_exports.def         |   4 +
 lib/librte_eal/rte_eal_version.map         |   4 +
 lib/librte_eal/windows/eal/eal.c           |   6 +
 lib/librte_eal/windows/eal/eal_memory.c    | 433 +++++++++++++++++++++
 lib/librte_eal/windows/eal/eal_windows.h   |  51 +++
 lib/librte_eal/windows/eal/meson.build     |   1 +
 11 files changed, 871 insertions(+), 6 deletions(-)
 create mode 100644 lib/librte_eal/windows/eal/eal_memory.c
config/meson.build +8 -2
index 73cf69814..295425742 100644
@@ -256,14 +256,20 @@ if is_freebsd
 endif
 
 if is_windows
-	# Minimum supported API is Windows 7.
-	add_project_arguments('-D_WIN32_WINNT=0x0601', language: 'c')
+	# VirtualAlloc2() is available since Windows 10 / Server 2016.
+	add_project_arguments('-D_WIN32_WINNT=0x0A00', language: 'c')
 
 	# Use MinGW-w64 stdio, because DPDK assumes ANSI-compliant formatting.
 	if cc.get_id() == 'gcc'
 		add_project_arguments('-D__USE_MINGW_ANSI_STDIO', language: 'c')
 	endif
 
+	# Contrary to docs, VirtualAlloc2() is exported by mincore.lib
+	# in Windows SDK, while MinGW exports it by advapi32.a.
+	if is_ms_linker
+		add_project_link_arguments('-lmincore', language: 'c')
+	endif
+
 	add_project_link_arguments('-ladvapi32', language: 'c')
 endif
 
lib/librte_eal/common/eal_private.h +67 -4

99 lines not on this page · raw message

lib/librte_eal/common/include/rte_memory.h +63 -0

69 lines not on this page · raw message

lib/librte_eal/freebsd/eal/eal_memory.c +117 -0

120 lines not on this page · raw message

lib/librte_eal/linux/eal/eal_memory.c +117 -0

120 lines not on this page · raw message

lib/librte_eal/rte_eal_exports.def +4 -0
index 12a6c79d6..bacf9a107 100644
@@ -5,5 +5,9 @@ EXPORTS
 	rte_eal_mp_remote_launch
 	rte_eal_mp_wait_lcore
 	rte_eal_remote_launch
+	rte_get_page_size
 	rte_log
+	rte_mem_lock
+	rte_mem_map
+	rte_mem_unmap
 	rte_vlog
lib/librte_eal/rte_eal_version.map +4 -0
index f9ede5b41..07128898f 100644
@@ -337,5 +337,9 @@ EXPERIMENTAL {
 	rte_thread_is_intr;
 
 	# added in 20.05
+	rte_get_page_size;
 	rte_log_can_log;
+	rte_mem_lock;
+	rte_mem_map;
+	rte_mem_unmap;
 };
lib/librte_eal/windows/eal/eal.c +6 -0
index 4932185ec..98afd8a68 100644
@@ -339,6 +339,12 @@ rte_eal_init(int argc, char **argv)
 			internal_config.memory = MEMSIZE_IF_NO_HUGE_PAGE;
 	}
 
+	if (eal_mem_win32api_init() < 0) {
+		rte_eal_init_alert("Cannot access Win32 memory management");
+		rte_errno = ENOTSUP;
+		return -1;
+	}
+
 	eal_thread_init_master(rte_config.master_lcore);
 
 	RTE_LCORE_FOREACH_SLAVE(i) {
lib/librte_eal/windows/eal/eal_memory.c new file +433 -0

433 lines not on this page · raw message

lib/librte_eal/windows/eal/eal_windows.h +51 -0

55 lines not on this page · raw message

lib/librte_eal/windows/eal/meson.build +1 -0
index 8b407c9ae..7b0f03f0a 100644
@@ -24,6 +24,7 @@ env_sources = files('eal.c',
 	'eal_debug.c',
 	'eal_hugepages.c',
 	'eal_lcore.c',
+	'eal_memory.c',
 	'eal_thread.c',
 	'getopt.c',
 )
Signature · 1 line
2.25.1
#14 lore raw

Re: [dpdk-dev] [RFC PATCH 6/9] eal: introduce memory management wrappers

30/03/2020 06:10, Dmitry Kozlyuk:

lib/librte_eal/common/include/rte_memory.h struct rte_memseg_list { · line 149

Quoted 12 lines
lib/librte_eal/common/include/rte_memory.h +12 -0
+/**
+ * Lock region in physical memory and prevent it from swapping.
+ *
+ * @param virt
+ *   The virtual address.
+ * @param size
+ *   Size of the region.
+ * @return
+ *   0 on success, negative on error.
+ */
+__rte_experimental
+int rte_mem_lock(const void *virt, size_t size);

Please refer to the similar rte_mem_lock_page in the above doxygen comment. You could use the syntax @see.

#15 lore raw

RFC PATCH 7/9 eal/windows: fix rte_page_sizes with Clang on Windows

In reply to Dmitry Kozlyuk

Clang on Windows follows MS ABI where enum values are signed 32-bit, Enum rte_page_size has members valued beyond 2^32. EAL cannot use -fno-ms-compatibility because its code is OS-dependent. The only option is to define these values outside enum, but this prohibits using -fstrict-enums. Another consequence is that enum rte_page_size cannot be used to hold page size, because on Windows it won't be able to hold all possible values.

---
 lib/librte_eal/common/include/rte_memory.h | 6 ++++++
 lib/librte_eal/meson.build                 | 1 +
 2 files changed, 7 insertions(+)
lib/librte_eal/common/include/rte_memory.h +6 -0
index 1742fde9a..5b0e2d8b5 100644
@@ -34,8 +34,14 @@ enum rte_page_sizes {
 	RTE_PGSIZE_256M  = 1ULL << 28,
 	RTE_PGSIZE_512M  = 1ULL << 29,
 	RTE_PGSIZE_1G    = 1ULL << 30,
+/* Work around Clang on Windows being limited to 32-bit underlying type. */
+#if !defined(RTE_TOOLCHAIN_CLANG) || !defined(RTE_EXEC_ENV_WINDOWS)
 	RTE_PGSIZE_4G    = 1ULL << 32,
 	RTE_PGSIZE_16G   = 1ULL << 34,
+#else
+#define RTE_PGSIZE_4G  (1ULL << 32)
+#define RTE_PGSIZE_16G (1ULL << 34)
+#endif
 };
 
 #define SOCKET_ID_ANY -1                    /**< Any NUMA socket. */
lib/librte_eal/meson.build +1 -0
index 1730d603f..ec80bd6be 100644
@@ -23,6 +23,7 @@ endif
 if cc.has_header('getopt.h')
 	cflags += ['-DHAVE_GETOPT_H', '-DHAVE_GETOPT', '-DHAVE_GETOPT_LONG']
 endif
+cflags += '-fno-strict-enums'
 sources = common_sources + env_sources
 objs = common_objs + env_objs
 headers = common_headers + env_headers
Signature · 1 line
2.25.1
#16 lore raw

RFC PATCH 8/9 eal/windows: replace sys/queue.h with a complete one from FreeBSD

In reply to Dmitry Kozlyuk

Limited version imported previously lacks at least SLIST macros. Import a complete file from FreeBSD, since its license exception is already approved by Technical Board.

---
 .../windows/eal/include/sys/queue.h           | 663 ++++++++++++++++--
 1 file changed, 601 insertions(+), 62 deletions(-)
lib/librte_eal/windows/eal/include/sys/queue.h +601 -62
index a65949a78..9756bee6f 100644
@@ -8,7 +8,36 @@
 #define	_SYS_QUEUE_H_
 
 /*
- * This file defines tail queues.
+ * This file defines four types of data structures: singly-linked lists,
+ * singly-linked tail queues, lists and tail queues.
+ *
+ * A singly-linked list is headed by a single forward pointer. The elements
+ * are singly linked for minimum space and pointer manipulation overhead at
+ * the expense of O(n) removal for arbitrary elements. New elements can be
+ * added to the list after an existing element or at the head of the list.
+ * Elements being removed from the head of the list should use the explicit
+ * macro for this purpose for optimum efficiency. A singly-linked list may
+ * only be traversed in the forward direction.  Singly-linked lists are ideal
+ * for applications with large datasets and few or no removals or for
+ * implementing a LIFO queue.
+ *
+ * A singly-linked tail queue is headed by a pair of pointers, one to the
+ * head of the list and the other to the tail of the list. The elements are
+ * singly linked for minimum space and pointer manipulation overhead at the
+ * expense of O(n) removal for arbitrary elements. New elements can be added
+ * to the list after an existing element, at the head of the list, or at the
+ * end of the list. Elements being removed from the head of the tail queue
+ * should use the explicit macro for this purpose for optimum efficiency.
+ * A singly-linked tail queue may only be traversed in the forward direction.
+ * Singly-linked tail queues are ideal for applications with large datasets
+ * and few or no removals or for implementing a FIFO queue.
+ *
+ * A list is headed by a single forward pointer (or an array of forward
+ * pointers for a hash table header). The elements are doubly linked
+ * so that an arbitrary element can be removed without a need to
+ * traverse the list. New elements can be added to the list before
+ * or after an existing element or at the head of the list. A list
+ * may be traversed in either direction.
  *
  * A tail queue is headed by a pair of pointers, one to the head of the
  * list and the other to the tail of the list. The elements are doubly
@@ -17,65 +46,93 @@
  * after an existing element, at the head of the list, or at the end of
  * the list. A tail queue may be traversed in either direction.
  *
+ * For details on the use of these macros, see the queue(3) manual page.
+ *
  * Below is a summary of implemented functions where:
  *  +  means the macro is available
  *  -  means the macro is not available
  *  s  means the macro is available but is slow (runs in O(n) time)
  *
- *				TAILQ
- * _HEAD			+
- * _CLASS_HEAD			+
- * _HEAD_INITIALIZER		+
- * _ENTRY			+
- * _CLASS_ENTRY			+
- * _INIT			+
- * _EMPTY			+
- * _FIRST			+
- * _NEXT			+
- * _PREV			+
- * _LAST			+
- * _LAST_FAST			+
- * _FOREACH			+
- * _FOREACH_FROM		+
- * _FOREACH_SAFE		+
- * _FOREACH_FROM_SAFE		+
- * _FOREACH_REVERSE		+
- * _FOREACH_REVERSE_FROM	+
- * _FOREACH_REVERSE_SAFE	+
- * _FOREACH_REVERSE_FROM_SAFE	+
- * _INSERT_HEAD			+
- * _INSERT_BEFORE		+
- * _INSERT_AFTER		+
- * _INSERT_TAIL			+
- * _CONCAT			+
- * _REMOVE_AFTER		-
- * _REMOVE_HEAD			-
- * _REMOVE			+
- * _SWAP			+
+ *				SLIST	LIST	STAILQ	TAILQ
+ * _HEAD			+	+	+	+
+ * _CLASS_HEAD			+	+	+	+
+ * _HEAD_INITIALIZER		+	+	+	+
+ * _ENTRY			+	+	+	+
+ * _CLASS_ENTRY			+	+	+	+
+ * _INIT			+	+	+	+
+ * _EMPTY			+	+	+	+
+ * _FIRST			+	+	+	+
+ * _NEXT			+	+	+	+
+ * _PREV			-	+	-	+
+ * _LAST			-	-	+	+
+ * _LAST_FAST			-	-	-	+
+ * _FOREACH			+	+	+	+
+ * _FOREACH_FROM		+	+	+	+
+ * _FOREACH_SAFE		+	+	+	+
+ * _FOREACH_FROM_SAFE		+	+	+	+
+ * _FOREACH_REVERSE		-	-	-	+
+ * _FOREACH_REVERSE_FROM	-	-	-	+
+ * _FOREACH_REVERSE_SAFE	-	-	-	+
+ * _FOREACH_REVERSE_FROM_SAFE	-	-	-	+
+ * _INSERT_HEAD			+	+	+	+
+ * _INSERT_BEFORE		-	+	-	+
+ * _INSERT_AFTER		+	+	+	+
+ * _INSERT_TAIL			-	-	+	+
+ * _CONCAT			s	s	+	+
+ * _REMOVE_AFTER		+	-	+	-
+ * _REMOVE_HEAD			+	-	+	-
+ * _REMOVE			s	+	s	+
+ * _SWAP			+	+	+	+
  *
  */
-
-#ifdef __cplusplus
-extern "C" {
+#ifdef QUEUE_MACRO_DEBUG
+#warn Use QUEUE_MACRO_DEBUG_TRACE and/or QUEUE_MACRO_DEBUG_TRASH
+#define	QUEUE_MACRO_DEBUG_TRACE
+#define	QUEUE_MACRO_DEBUG_TRASH
 #endif
 
-/*
- * List definitions.
- */
-#define	LIST_HEAD(name, type)						\
-struct name {								\
-	struct type *lh_first;	/* first element */			\
-}
+#ifdef QUEUE_MACRO_DEBUG_TRACE
+/* Store the last 2 places the queue element or head was altered */
+struct qm_trace {
+	unsigned long	 lastline;
+	unsigned long	 prevline;
+	const char	*lastfile;
+	const char	*prevfile;
+};
+
+#define	TRACEBUF	struct qm_trace trace;
+#define	TRACEBUF_INITIALIZER	{ __LINE__, 0, __FILE__, NULL } ,
+
+#define	QMD_TRACE_HEAD(head) do {					\
+	(head)->trace.prevline = (head)->trace.lastline;		\
+	(head)->trace.prevfile = (head)->trace.lastfile;		\
+	(head)->trace.lastline = __LINE__;				\
+	(head)->trace.lastfile = __FILE__;				\
+} while (0)
 
+#define	QMD_TRACE_ELEM(elem) do {					\
+	(elem)->trace.prevline = (elem)->trace.lastline;		\
+	(elem)->trace.prevfile = (elem)->trace.lastfile;		\
+	(elem)->trace.lastline = __LINE__;				\
+	(elem)->trace.lastfile = __FILE__;				\
+} while (0)
+
+#else	/* !QUEUE_MACRO_DEBUG_TRACE */
 #define	QMD_TRACE_ELEM(elem)
 #define	QMD_TRACE_HEAD(head)
 #define	TRACEBUF
 #define	TRACEBUF_INITIALIZER
+#endif	/* QUEUE_MACRO_DEBUG_TRACE */
 
+#ifdef QUEUE_MACRO_DEBUG_TRASH
+#define	QMD_SAVELINK(name, link)	void **name = (void *)&(link)
+#define	TRASHIT(x)		do {(x) = (void *)-1;} while (0)
+#define	QMD_IS_TRASHED(x)	((x) == (void *)(intptr_t)-1)
+#else	/* !QUEUE_MACRO_DEBUG_TRASH */
+#define	QMD_SAVELINK(name, link)
 #define	TRASHIT(x)
 #define	QMD_IS_TRASHED(x)	0
-
-#define	QMD_SAVELINK(name, link)
+#endif	/* QUEUE_MACRO_DEBUG_TRASH */
 
 #ifdef __cplusplus
 /*
@@ -86,6 +143,445 @@ struct name {								\
 #define	QUEUE_TYPEOF(type) struct type
 #endif
 
+/*
+ * Singly-linked List declarations.
+ */
+#define	SLIST_HEAD(name, type)						\
+struct name {								\
+	struct type *slh_first;	/* first element */			\
+}
+
+#define	SLIST_CLASS_HEAD(name, type)					\
+struct name {								\
+	class type *slh_first;	/* first element */			\
+}
+
+#define	SLIST_HEAD_INITIALIZER(head)					\
+	{ NULL }
+
+#define	SLIST_ENTRY(type)						\
+struct {								\
+	struct type *sle_next;	/* next element */			\
+}
+
+#define	SLIST_CLASS_ENTRY(type)						\
+struct {								\
+	class type *sle_next;		/* next element */		\
+}
+
+/*
+ * Singly-linked List functions.
+ */
+#if (defined(_KERNEL) && defined(INVARIANTS))
+#define	QMD_SLIST_CHECK_PREVPTR(prevp, elm) do {			\
+	if (*(prevp) != (elm))						\
+		panic("Bad prevptr *(%p) == %p != %p",			\
+		    (prevp), *(prevp), (elm));				\
+} while (0)
+#else
+#define	QMD_SLIST_CHECK_PREVPTR(prevp, elm)
+#endif
+
+#define SLIST_CONCAT(head1, head2, type, field) do {			\
+	QUEUE_TYPEOF(type) *curelm = SLIST_FIRST(head1);		\
+	if (curelm == NULL) {						\
+		if ((SLIST_FIRST(head1) = SLIST_FIRST(head2)) != NULL)	\
+			SLIST_INIT(head2);				\
+	} else if (SLIST_FIRST(head2) != NULL) {			\
+		while (SLIST_NEXT(curelm, field) != NULL)		\
+			curelm = SLIST_NEXT(curelm, field);		\
+		SLIST_NEXT(curelm, field) = SLIST_FIRST(head2);		\
+		SLIST_INIT(head2);					\
+	}								\
+} while (0)
+
+#define	SLIST_EMPTY(head)	((head)->slh_first == NULL)
+
+#define	SLIST_FIRST(head)	((head)->slh_first)
+
+#define	SLIST_FOREACH(var, head, field)					\
+	for ((var) = SLIST_FIRST((head));				\
+	    (var);							\
+	    (var) = SLIST_NEXT((var), field))
+
+#define	SLIST_FOREACH_FROM(var, head, field)				\
+	for ((var) = ((var) ? (var) : SLIST_FIRST((head)));		\
+	    (var);							\
+	    (var) = SLIST_NEXT((var), field))
+
+#define	SLIST_FOREACH_SAFE(var, head, field, tvar)			\
+	for ((var) = SLIST_FIRST((head));				\
+	    (var) && ((tvar) = SLIST_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	SLIST_FOREACH_FROM_SAFE(var, head, field, tvar)			\
+	for ((var) = ((var) ? (var) : SLIST_FIRST((head)));		\
+	    (var) && ((tvar) = SLIST_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	SLIST_FOREACH_PREVPTR(var, varp, head, field)			\
+	for ((varp) = &SLIST_FIRST((head));				\
+	    ((var) = *(varp)) != NULL;					\
+	    (varp) = &SLIST_NEXT((var), field))
+
+#define	SLIST_INIT(head) do {						\
+	SLIST_FIRST((head)) = NULL;					\
+} while (0)
+
+#define	SLIST_INSERT_AFTER(slistelm, elm, field) do {			\
+	SLIST_NEXT((elm), field) = SLIST_NEXT((slistelm), field);	\
+	SLIST_NEXT((slistelm), field) = (elm);				\
+} while (0)
+
+#define	SLIST_INSERT_HEAD(head, elm, field) do {			\
+	SLIST_NEXT((elm), field) = SLIST_FIRST((head));			\
+	SLIST_FIRST((head)) = (elm);					\
+} while (0)
+
+#define	SLIST_NEXT(elm, field)	((elm)->field.sle_next)
+
+#define	SLIST_REMOVE(head, elm, type, field) do {			\
+	QMD_SAVELINK(oldnext, (elm)->field.sle_next);			\
+	if (SLIST_FIRST((head)) == (elm)) {				\
+		SLIST_REMOVE_HEAD((head), field);			\
+	}								\
+	else {								\
+		QUEUE_TYPEOF(type) *curelm = SLIST_FIRST(head);		\
+		while (SLIST_NEXT(curelm, field) != (elm))		\
+			curelm = SLIST_NEXT(curelm, field);		\
+		SLIST_REMOVE_AFTER(curelm, field);			\
+	}								\
+	TRASHIT(*oldnext);						\
+} while (0)
+
+#define SLIST_REMOVE_AFTER(elm, field) do {				\
+	SLIST_NEXT(elm, field) =					\
+	    SLIST_NEXT(SLIST_NEXT(elm, field), field);			\
+} while (0)
+
+#define	SLIST_REMOVE_HEAD(head, field) do {				\
+	SLIST_FIRST((head)) = SLIST_NEXT(SLIST_FIRST((head)), field);	\
+} while (0)
+
+#define	SLIST_REMOVE_PREVPTR(prevp, elm, field) do {			\
+	QMD_SLIST_CHECK_PREVPTR(prevp, elm);				\
+	*(prevp) = SLIST_NEXT(elm, field);				\
+	TRASHIT((elm)->field.sle_next);					\
+} while (0)
+
+#define SLIST_SWAP(head1, head2, type) do {				\
+	QUEUE_TYPEOF(type) *swap_first = SLIST_FIRST(head1);		\
+	SLIST_FIRST(head1) = SLIST_FIRST(head2);			\
+	SLIST_FIRST(head2) = swap_first;				\
+} while (0)
+
+/*
+ * Singly-linked Tail queue declarations.
+ */
+#define	STAILQ_HEAD(name, type)						\
+struct name {								\
+	struct type *stqh_first;/* first element */			\
+	struct type **stqh_last;/* addr of last next element */		\
+}
+
+#define	STAILQ_CLASS_HEAD(name, type)					\
+struct name {								\
+	class type *stqh_first;	/* first element */			\
+	class type **stqh_last;	/* addr of last next element */		\
+}
+
+#define	STAILQ_HEAD_INITIALIZER(head)					\
+	{ NULL, &(head).stqh_first }
+
+#define	STAILQ_ENTRY(type)						\
+struct {								\
+	struct type *stqe_next;	/* next element */			\
+}
+
+#define	STAILQ_CLASS_ENTRY(type)					\
+struct {								\
+	class type *stqe_next;	/* next element */			\
+}
+
+/*
+ * Singly-linked Tail queue functions.
+ */
+#define	STAILQ_CONCAT(head1, head2) do {				\
+	if (!STAILQ_EMPTY((head2))) {					\
+		*(head1)->stqh_last = (head2)->stqh_first;		\
+		(head1)->stqh_last = (head2)->stqh_last;		\
+		STAILQ_INIT((head2));					\
+	}								\
+} while (0)
+
+#define	STAILQ_EMPTY(head)	((head)->stqh_first == NULL)
+
+#define	STAILQ_FIRST(head)	((head)->stqh_first)
+
+#define	STAILQ_FOREACH(var, head, field)				\
+	for((var) = STAILQ_FIRST((head));				\
+	   (var);							\
+	   (var) = STAILQ_NEXT((var), field))
+
+#define	STAILQ_FOREACH_FROM(var, head, field)				\
+	for ((var) = ((var) ? (var) : STAILQ_FIRST((head)));		\
+	   (var);							\
+	   (var) = STAILQ_NEXT((var), field))
+
+#define	STAILQ_FOREACH_SAFE(var, head, field, tvar)			\
+	for ((var) = STAILQ_FIRST((head));				\
+	    (var) && ((tvar) = STAILQ_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	STAILQ_FOREACH_FROM_SAFE(var, head, field, tvar)		\
+	for ((var) = ((var) ? (var) : STAILQ_FIRST((head)));		\
+	    (var) && ((tvar) = STAILQ_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	STAILQ_INIT(head) do {						\
+	STAILQ_FIRST((head)) = NULL;					\
+	(head)->stqh_last = &STAILQ_FIRST((head));			\
+} while (0)
+
+#define	STAILQ_INSERT_AFTER(head, tqelm, elm, field) do {		\
+	if ((STAILQ_NEXT((elm), field) = STAILQ_NEXT((tqelm), field)) == NULL)\
+		(head)->stqh_last = &STAILQ_NEXT((elm), field);		\
+	STAILQ_NEXT((tqelm), field) = (elm);				\
+} while (0)
+
+#define	STAILQ_INSERT_HEAD(head, elm, field) do {			\
+	if ((STAILQ_NEXT((elm), field) = STAILQ_FIRST((head))) == NULL)	\
+		(head)->stqh_last = &STAILQ_NEXT((elm), field);		\
+	STAILQ_FIRST((head)) = (elm);					\
+} while (0)
+
+#define	STAILQ_INSERT_TAIL(head, elm, field) do {			\
+	STAILQ_NEXT((elm), field) = NULL;				\
+	*(head)->stqh_last = (elm);					\
+	(head)->stqh_last = &STAILQ_NEXT((elm), field);			\
+} while (0)
+
+#define	STAILQ_LAST(head, type, field)				\
+	(STAILQ_EMPTY((head)) ? NULL :				\
+	    __containerof((head)->stqh_last,			\
+	    QUEUE_TYPEOF(type), field.stqe_next))
+
+#define	STAILQ_NEXT(elm, field)	((elm)->field.stqe_next)
+
+#define	STAILQ_REMOVE(head, elm, type, field) do {			\
+	QMD_SAVELINK(oldnext, (elm)->field.stqe_next);			\
+	if (STAILQ_FIRST((head)) == (elm)) {				\
+		STAILQ_REMOVE_HEAD((head), field);			\
+	}								\
+	else {								\
+		QUEUE_TYPEOF(type) *curelm = STAILQ_FIRST(head);	\
+		while (STAILQ_NEXT(curelm, field) != (elm))		\
+			curelm = STAILQ_NEXT(curelm, field);		\
+		STAILQ_REMOVE_AFTER(head, curelm, field);		\
+	}								\
+	TRASHIT(*oldnext);						\
+} while (0)
+
+#define STAILQ_REMOVE_AFTER(head, elm, field) do {			\
+	if ((STAILQ_NEXT(elm, field) =					\
+	     STAILQ_NEXT(STAILQ_NEXT(elm, field), field)) == NULL)	\
+		(head)->stqh_last = &STAILQ_NEXT((elm), field);		\
+} while (0)
+
+#define	STAILQ_REMOVE_HEAD(head, field) do {				\
+	if ((STAILQ_FIRST((head)) =					\
+	     STAILQ_NEXT(STAILQ_FIRST((head)), field)) == NULL)		\
+		(head)->stqh_last = &STAILQ_FIRST((head));		\
+} while (0)
+
+#define STAILQ_SWAP(head1, head2, type) do {				\
+	QUEUE_TYPEOF(type) *swap_first = STAILQ_FIRST(head1);		\
+	QUEUE_TYPEOF(type) **swap_last = (head1)->stqh_last;		\
+	STAILQ_FIRST(head1) = STAILQ_FIRST(head2);			\
+	(head1)->stqh_last = (head2)->stqh_last;			\
+	STAILQ_FIRST(head2) = swap_first;				\
+	(head2)->stqh_last = swap_last;					\
+	if (STAILQ_EMPTY(head1))					\
+		(head1)->stqh_last = &STAILQ_FIRST(head1);		\
+	if (STAILQ_EMPTY(head2))					\
+		(head2)->stqh_last = &STAILQ_FIRST(head2);		\
+} while (0)
+
+
+/*
+ * List declarations.
+ */
+#define	LIST_HEAD(name, type)						\
+struct name {								\
+	struct type *lh_first;	/* first element */			\
+}
+
+#define	LIST_CLASS_HEAD(name, type)					\
+struct name {								\
+	class type *lh_first;	/* first element */			\
+}
+
+#define	LIST_HEAD_INITIALIZER(head)					\
+	{ NULL }
+
+#define	LIST_ENTRY(type)						\
+struct {								\
+	struct type *le_next;	/* next element */			\
+	struct type **le_prev;	/* address of previous next element */	\
+}
+
+#define	LIST_CLASS_ENTRY(type)						\
+struct {								\
+	class type *le_next;	/* next element */			\
+	class type **le_prev;	/* address of previous next element */	\
+}
+
+/*
+ * List functions.
+ */
+
+#if (defined(_KERNEL) && defined(INVARIANTS))
+/*
+ * QMD_LIST_CHECK_HEAD(LIST_HEAD *head, LIST_ENTRY NAME)
+ *
+ * If the list is non-empty, validates that the first element of the list
+ * points back at 'head.'
+ */
+#define	QMD_LIST_CHECK_HEAD(head, field) do {				\
+	if (LIST_FIRST((head)) != NULL &&				\
+	    LIST_FIRST((head))->field.le_prev !=			\
+	     &LIST_FIRST((head)))					\
+		panic("Bad list head %p first->prev != head", (head));	\
+} while (0)
+
+/*
+ * QMD_LIST_CHECK_NEXT(TYPE *elm, LIST_ENTRY NAME)
+ *
+ * If an element follows 'elm' in the list, validates that the next element
+ * points back at 'elm.'
+ */
+#define	QMD_LIST_CHECK_NEXT(elm, field) do {				\
+	if (LIST_NEXT((elm), field) != NULL &&				\
+	    LIST_NEXT((elm), field)->field.le_prev !=			\
+	     &((elm)->field.le_next))					\
+	     	panic("Bad link elm %p next->prev != elm", (elm));	\
+} while (0)
+
+/*
+ * QMD_LIST_CHECK_PREV(TYPE *elm, LIST_ENTRY NAME)
+ *
+ * Validates that the previous element (or head of the list) points to 'elm.'
+ */
+#define	QMD_LIST_CHECK_PREV(elm, field) do {				\
+	if (*(elm)->field.le_prev != (elm))				\
+		panic("Bad link elm %p prev->next != elm", (elm));	\
+} while (0)
+#else
+#define	QMD_LIST_CHECK_HEAD(head, field)
+#define	QMD_LIST_CHECK_NEXT(elm, field)
+#define	QMD_LIST_CHECK_PREV(elm, field)
+#endif /* (_KERNEL && INVARIANTS) */
+
+#define LIST_CONCAT(head1, head2, type, field) do {			      \
+	QUEUE_TYPEOF(type) *curelm = LIST_FIRST(head1);			      \
+	if (curelm == NULL) {						      \
+		if ((LIST_FIRST(head1) = LIST_FIRST(head2)) != NULL) {	      \
+			LIST_FIRST(head2)->field.le_prev =		      \
+			    &LIST_FIRST((head1));			      \
+			LIST_INIT(head2);				      \
+		}							      \
+	} else if (LIST_FIRST(head2) != NULL) {				      \
+		while (LIST_NEXT(curelm, field) != NULL)		      \
+			curelm = LIST_NEXT(curelm, field);		      \
+		LIST_NEXT(curelm, field) = LIST_FIRST(head2);		      \
+		LIST_FIRST(head2)->field.le_prev = &LIST_NEXT(curelm, field); \
+		LIST_INIT(head2);					      \
+	}								      \
+} while (0)
+
+#define	LIST_EMPTY(head)	((head)->lh_first == NULL)
+
+#define	LIST_FIRST(head)	((head)->lh_first)
+
+#define	LIST_FOREACH(var, head, field)					\
+	for ((var) = LIST_FIRST((head));				\
+	    (var);							\
+	    (var) = LIST_NEXT((var), field))
+
+#define	LIST_FOREACH_FROM(var, head, field)				\
+	for ((var) = ((var) ? (var) : LIST_FIRST((head)));		\
+	    (var);							\
+	    (var) = LIST_NEXT((var), field))
+
+#define	LIST_FOREACH_SAFE(var, head, field, tvar)			\
+	for ((var) = LIST_FIRST((head));				\
+	    (var) && ((tvar) = LIST_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	LIST_FOREACH_FROM_SAFE(var, head, field, tvar)			\
+	for ((var) = ((var) ? (var) : LIST_FIRST((head)));		\
+	    (var) && ((tvar) = LIST_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	LIST_INIT(head) do {						\
+	LIST_FIRST((head)) = NULL;					\
+} while (0)
+
+#define	LIST_INSERT_AFTER(listelm, elm, field) do {			\
+	QMD_LIST_CHECK_NEXT(listelm, field);				\
+	if ((LIST_NEXT((elm), field) = LIST_NEXT((listelm), field)) != NULL)\
+		LIST_NEXT((listelm), field)->field.le_prev =		\
+		    &LIST_NEXT((elm), field);				\
+	LIST_NEXT((listelm), field) = (elm);				\
+	(elm)->field.le_prev = &LIST_NEXT((listelm), field);		\
+} while (0)
+
+#define	LIST_INSERT_BEFORE(listelm, elm, field) do {			\
+	QMD_LIST_CHECK_PREV(listelm, field);				\
+	(elm)->field.le_prev = (listelm)->field.le_prev;		\
+	LIST_NEXT((elm), field) = (listelm);				\
+	*(listelm)->field.le_prev = (elm);				\
+	(listelm)->field.le_prev = &LIST_NEXT((elm), field);		\
+} while (0)
+
+#define	LIST_INSERT_HEAD(head, elm, field) do {				\
+	QMD_LIST_CHECK_HEAD((head), field);				\
+	if ((LIST_NEXT((elm), field) = LIST_FIRST((head))) != NULL)	\
+		LIST_FIRST((head))->field.le_prev = &LIST_NEXT((elm), field);\
+	LIST_FIRST((head)) = (elm);					\
+	(elm)->field.le_prev = &LIST_FIRST((head));			\
+} while (0)
+
+#define	LIST_NEXT(elm, field)	((elm)->field.le_next)
+
+#define	LIST_PREV(elm, head, type, field)			\
+	((elm)->field.le_prev == &LIST_FIRST((head)) ? NULL :	\
+	    __containerof((elm)->field.le_prev,			\
+	    QUEUE_TYPEOF(type), field.le_next))
+
+#define	LIST_REMOVE(elm, field) do {					\
+	QMD_SAVELINK(oldnext, (elm)->field.le_next);			\
+	QMD_SAVELINK(oldprev, (elm)->field.le_prev);			\
+	QMD_LIST_CHECK_NEXT(elm, field);				\
+	QMD_LIST_CHECK_PREV(elm, field);				\
+	if (LIST_NEXT((elm), field) != NULL)				\
+		LIST_NEXT((elm), field)->field.le_prev = 		\
+		    (elm)->field.le_prev;				\
+	*(elm)->field.le_prev = LIST_NEXT((elm), field);		\
+	TRASHIT(*oldnext);						\
+	TRASHIT(*oldprev);						\
+} while (0)
+
+#define LIST_SWAP(head1, head2, type, field) do {			\
+	QUEUE_TYPEOF(type) *swap_tmp = LIST_FIRST(head1);		\
+	LIST_FIRST((head1)) = LIST_FIRST((head2));			\
+	LIST_FIRST((head2)) = swap_tmp;					\
+	if ((swap_tmp = LIST_FIRST((head1))) != NULL)			\
+		swap_tmp->field.le_prev = &LIST_FIRST((head1));		\
+	if ((swap_tmp = LIST_FIRST((head2))) != NULL)			\
+		swap_tmp->field.le_prev = &LIST_FIRST((head2));		\
+} while (0)
+
 /*
  * Tail queue declarations.
  */
@@ -123,10 +619,58 @@ struct {								\
 /*
  * Tail queue functions.
  */
+#if (defined(_KERNEL) && defined(INVARIANTS))
+/*
+ * QMD_TAILQ_CHECK_HEAD(TAILQ_HEAD *head, TAILQ_ENTRY NAME)
+ *
+ * If the tailq is non-empty, validates that the first element of the tailq
+ * points back at 'head.'
+ */
+#define	QMD_TAILQ_CHECK_HEAD(head, field) do {				\
+	if (!TAILQ_EMPTY(head) &&					\
+	    TAILQ_FIRST((head))->field.tqe_prev !=			\
+	     &TAILQ_FIRST((head)))					\
+		panic("Bad tailq head %p first->prev != head", (head));	\
+} while (0)
+
+/*
+ * QMD_TAILQ_CHECK_TAIL(TAILQ_HEAD *head, TAILQ_ENTRY NAME)
+ *
+ * Validates that the tail of the tailq is a pointer to pointer to NULL.
+ */
+#define	QMD_TAILQ_CHECK_TAIL(head, field) do {				\
+	if (*(head)->tqh_last != NULL)					\
+	    	panic("Bad tailq NEXT(%p->tqh_last) != NULL", (head)); 	\
+} while (0)
+
+/*
+ * QMD_TAILQ_CHECK_NEXT(TYPE *elm, TAILQ_ENTRY NAME)
+ *
+ * If an element follows 'elm' in the tailq, validates that the next element
+ * points back at 'elm.'
+ */
+#define	QMD_TAILQ_CHECK_NEXT(elm, field) do {				\
+	if (TAILQ_NEXT((elm), field) != NULL &&				\
+	    TAILQ_NEXT((elm), field)->field.tqe_prev !=			\
+	     &((elm)->field.tqe_next))					\
+		panic("Bad link elm %p next->prev != elm", (elm));	\
+} while (0)
+
+/*
+ * QMD_TAILQ_CHECK_PREV(TYPE *elm, TAILQ_ENTRY NAME)
+ *
+ * Validates that the previous element (or head of the tailq) points to 'elm.'
+ */
+#define	QMD_TAILQ_CHECK_PREV(elm, field) do {				\
+	if (*(elm)->field.tqe_prev != (elm))				\
+		panic("Bad link elm %p prev->next != elm", (elm));	\
+} while (0)
+#else
 #define	QMD_TAILQ_CHECK_HEAD(head, field)
 #define	QMD_TAILQ_CHECK_TAIL(head, headname)
 #define	QMD_TAILQ_CHECK_NEXT(elm, field)
 #define	QMD_TAILQ_CHECK_PREV(elm, field)
+#endif /* (_KERNEL && INVARIANTS) */
 
 #define	TAILQ_CONCAT(head1, head2, field) do {				\
 	if (!TAILQ_EMPTY(head2)) {					\
@@ -191,9 +735,8 @@ struct {								\
 
 #define	TAILQ_INSERT_AFTER(head, listelm, elm, field) do {		\
 	QMD_TAILQ_CHECK_NEXT(listelm, field);				\
-	TAILQ_NEXT((elm), field) = TAILQ_NEXT((listelm), field);	\
-	if (TAILQ_NEXT((listelm), field) != NULL)			\
-		TAILQ_NEXT((elm), field)->field.tqe_prev =		\
+	if ((TAILQ_NEXT((elm), field) = TAILQ_NEXT((listelm), field)) != NULL)\
+		TAILQ_NEXT((elm), field)->field.tqe_prev = 		\
 		    &TAILQ_NEXT((elm), field);				\
 	else {								\
 		(head)->tqh_last = &TAILQ_NEXT((elm), field);		\
@@ -217,8 +760,7 @@ struct {								\
 
 #define	TAILQ_INSERT_HEAD(head, elm, field) do {			\
 	QMD_TAILQ_CHECK_HEAD(head, field);				\
-	TAILQ_NEXT((elm), field) = TAILQ_FIRST((head));			\
-	if (TAILQ_FIRST((head)) != NULL)				\
+	if ((TAILQ_NEXT((elm), field) = TAILQ_FIRST((head))) != NULL)	\
 		TAILQ_FIRST((head))->field.tqe_prev =			\
 		    &TAILQ_NEXT((elm), field);				\
 	else								\
@@ -250,21 +792,24 @@ struct {								\
  * you may want to prefetch the last data element.
  */
 #define	TAILQ_LAST_FAST(head, type, field)			\
-	(TAILQ_EMPTY(head) ? NULL : __containerof((head)->tqh_last,	\
-	QUEUE_TYPEOF(type), field.tqe_next))
+    (TAILQ_EMPTY(head) ? NULL : __containerof((head)->tqh_last, QUEUE_TYPEOF(type), field.tqe_next))
 
 #define	TAILQ_NEXT(elm, field) ((elm)->field.tqe_next)
 
 #define	TAILQ_PREV(elm, headname, field)				\
 	(*(((struct headname *)((elm)->field.tqe_prev))->tqh_last))
 
+#define	TAILQ_PREV_FAST(elm, head, type, field)				\
+    ((elm)->field.tqe_prev == &(head)->tqh_first ? NULL :		\
+     __containerof((elm)->field.tqe_prev, QUEUE_TYPEOF(type), field.tqe_next))
+
 #define	TAILQ_REMOVE(head, elm, field) do {				\
 	QMD_SAVELINK(oldnext, (elm)->field.tqe_next);			\
 	QMD_SAVELINK(oldprev, (elm)->field.tqe_prev);			\
 	QMD_TAILQ_CHECK_NEXT(elm, field);				\
 	QMD_TAILQ_CHECK_PREV(elm, field);				\
 	if ((TAILQ_NEXT((elm), field)) != NULL)				\
-		TAILQ_NEXT((elm), field)->field.tqe_prev =		\
+		TAILQ_NEXT((elm), field)->field.tqe_prev = 		\
 		    (elm)->field.tqe_prev;				\
 	else {								\
 		(head)->tqh_last = (elm)->field.tqe_prev;		\
@@ -277,26 +822,20 @@ struct {								\
 } while (0)
 
 #define TAILQ_SWAP(head1, head2, type, field) do {			\
-	QUEUE_TYPEOF(type) * swap_first = (head1)->tqh_first;		\
-	QUEUE_TYPEOF(type) * *swap_last = (head1)->tqh_last;		\
+	QUEUE_TYPEOF(type) *swap_first = (head1)->tqh_first;		\
+	QUEUE_TYPEOF(type) **swap_last = (head1)->tqh_last;		\
 	(head1)->tqh_first = (head2)->tqh_first;			\
 	(head1)->tqh_last = (head2)->tqh_last;				\
 	(head2)->tqh_first = swap_first;				\
 	(head2)->tqh_last = swap_last;					\
-	swap_first = (head1)->tqh_first;				\
-	if (swap_first != NULL)						\
+	if ((swap_first = (head1)->tqh_first) != NULL)			\
 		swap_first->field.tqe_prev = &(head1)->tqh_first;	\
 	else								\
 		(head1)->tqh_last = &(head1)->tqh_first;		\
-	swap_first = (head2)->tqh_first;				\
-	if (swap_first != NULL)			\
+	if ((swap_first = (head2)->tqh_first) != NULL)			\
 		swap_first->field.tqe_prev = &(head2)->tqh_first;	\
 	else								\
 		(head2)->tqh_last = &(head2)->tqh_first;		\
 } while (0)
 
-#ifdef __cplusplus
-}
-#endif
-
-#endif /* _SYS_QUEUE_H_ */
+#endif /* !_SYS_QUEUE_H_ */
Signature · 1 line
2.25.1
#17 lore raw

RFC PATCH 9/9 eal/windows: implement basic memory management

In reply to Dmitry Kozlyuk

Basic memory management supports core libraries and PMDs operating in IOVA as PA mode. It uses a kernel-mode driver to obtain IOVAs of hugepages allocated from user-mode.

---
 config/meson.build                            |   2 +-
 lib/librte_eal/common/eal_common_fbarray.c    |  57 +-
 lib/librte_eal/common/eal_common_memory.c     |  50 +-
 lib/librte_eal/common/malloc_heap.c           |   1 +
 lib/librte_eal/freebsd/eal/eal_memory.c       |   1 -
 lib/librte_eal/linux/eal/eal_memory.c         |   2 +-
 lib/librte_eal/meson.build                    |   5 +-
 lib/librte_eal/rte_eal_exports.def            | 119 ++-
 lib/librte_eal/windows/eal/eal.c              |  47 ++
 lib/librte_eal/windows/eal/eal_memalloc.c     | 423 ++++++++++
 lib/librte_eal/windows/eal/eal_memory.c       | 735 +++++++++++++++++-
 lib/librte_eal/windows/eal/eal_mp.c           | 103 +++
 lib/librte_eal/windows/eal/eal_windows.h      |  23 +
 lib/librte_eal/windows/eal/include/rte_os.h   |   4 +
 .../windows/eal/include/rte_virt2phys.h       |  34 +
 .../windows/eal/include/rte_windows.h         |   2 +
 lib/librte_eal/windows/eal/include/unistd.h   |   3 +
 lib/librte_eal/windows/eal/meson.build        |  12 +
 18 files changed, 1557 insertions(+), 66 deletions(-)
 create mode 100644 lib/librte_eal/windows/eal/eal_memalloc.c
 create mode 100644 lib/librte_eal/windows/eal/eal_mp.c
 create mode 100644 lib/librte_eal/windows/eal/include/rte_virt2phys.h
config/meson.build +1 -1
index 295425742..49d208006 100644
@@ -270,7 +270,7 @@ if is_windows
 		add_project_link_arguments('-lmincore', language: 'c')
 	endif
 
-	add_project_link_arguments('-ladvapi32', language: 'c')
+	add_project_link_arguments('-ladvapi32', '-lsetupapi', language: 'c')
 endif
 
 if get_option('b_lto')
lib/librte_eal/common/eal_common_fbarray.c +26 -31

168 lines not on this page · raw message

lib/librte_eal/common/eal_common_memory.c +22 -28

118 lines not on this page · raw message

lib/librte_eal/common/malloc_heap.c +1 -0
index 842eb9de7..6534c895c 100644
@@ -729,6 +729,7 @@ malloc_heap_alloc(const char *type, size_t size, int socket_arg,
 		if (ret != NULL)
 			return ret;
 	}
+
 	return NULL;
 }
 
lib/librte_eal/freebsd/eal/eal_memory.c +0 -1
index bcceba636..b38cc380b 100644
@@ -393,7 +393,6 @@ alloc_va_space(struct rte_memseg_list *msl)
 	return 0;
 }
 
-
 static int
 memseg_primary_init(void)
 {
lib/librte_eal/linux/eal/eal_memory.c +1 -1
index 72205580a..1d24579b5 100644
@@ -2518,7 +2518,7 @@ eal_mem_reserve(void *requested_addr, size_t size,
 	if (flags & RTE_RESERVE_EXACT_ADDRESS)
 		sys_flags |= MAP_FIXED;
 
-	return mem_map(requested_addr, size, PROT_READ, sys_flags, -1, 0);
+	return mem_map(requested_addr, size, PROT_NONE, sys_flags, -1, 0);
 }
 
 void *
lib/librte_eal/meson.build +4 -1
index ec80bd6be..6a6ec0ddb 100644
@@ -10,6 +10,7 @@ subdir('common') # defines common_sources, common_objs, etc.
 # Now do OS/exec-env specific settings, including building kernel modules
 # The <exec-env>/eal/meson.build file should define env_sources, etc.
 dpdk_conf.set('RTE_EXEC_ENV_' + exec_env.to_upper(), 1)
+dpdk_conf.set10('RTE_EAL_NUMA_AWARE_HUGEPAGES', true)
 subdir(exec_env + '/eal')
 
 allow_experimental_apis = true
@@ -23,7 +24,9 @@ endif
 if cc.has_header('getopt.h')
 	cflags += ['-DHAVE_GETOPT_H', '-DHAVE_GETOPT', '-DHAVE_GETOPT_LONG']
 endif
-cflags += '-fno-strict-enums'
+if cc.get_id() == 'clang'
+	cflags += '-fno-strict-enums'
+endif
 sources = common_sources + env_sources
 objs = common_objs + env_objs
 headers = common_headers + env_headers
lib/librte_eal/rte_eal_exports.def +117 -2

130 lines not on this page · raw message

lib/librte_eal/windows/eal/eal.c +47 -0

59 lines not on this page · raw message

lib/librte_eal/windows/eal/eal_memalloc.c new file +423 -0

423 lines not on this page · raw message

lib/librte_eal/windows/eal/eal_memory.c +734 -1

767 lines not on this page · raw message

lib/librte_eal/windows/eal/eal_mp.c new file +103 -0

103 lines not on this page · raw message

lib/librte_eal/windows/eal/eal_windows.h +23 -0
index 002d7e4a5..4b504a023 100644
@@ -9,8 +9,24 @@
  * @file Facilities private to Windows EAL
  */
 
+#include <rte_errno.h>
 #include <rte_windows.h>
 
+/**
+ * Log current function as not implemented and set rte_errno.
+ */
+#define EAL_LOG_NOT_IMPLEMENTED() \
+	do { \
+		RTE_LOG(DEBUG, EAL, "%s() is not implemented\n", __func__); \
+		rte_errno = ENOTSUP; \
+	} while (0)
+
+/**
+ * Log current function as a stub.
+ */
+#define EAL_LOG_STUB() \
+	RTE_LOG(DEBUG, EAL, "Windows: %s() is a stub\n", __func__)
+
 /**
  * Create a map of processors and cores on the system.
  */
@@ -36,6 +52,13 @@ int eal_thread_create(pthread_t *thread);
  */
 unsigned int eal_socket_numa_node(unsigned int socket_id);
 
+/**
+ * Open virt2phys driver interface device.
+ *
+ * @return 0 on success, (-1) on failure.
+ */
+int eal_mem_virt2iova_init(void);
+
 /**
  * Locate Win32 memory management routines in system libraries.
  *
lib/librte_eal/windows/eal/include/rte_os.h +4 -0
index 510e39e03..62805a307 100644
@@ -36,6 +36,10 @@ extern "C" {
 
 #define strncasecmp(s1, s2, count)        _strnicmp(s1, s2, count)
 
+#define open _open
+#define close _close
+#define unlink _unlink
+
 /* cpu_set macros implementation */
 #define RTE_CPU_AND(dst, src1, src2) CPU_AND(dst, src1, src2)
 #define RTE_CPU_OR(dst, src1, src2) CPU_OR(dst, src1, src2)
lib/librte_eal/windows/eal/include/rte_virt2phys.h new file +34 -0
new file mode 100644
index 000000000..4bb2b4aaf
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2020 Dmitry Kozlyuk
+ */
+
+/**
+ * @file virt2phys driver interface
+ */
+
+/**
+ * Driver device interface GUID {539c2135-793a-4926-afec-d3a1b61bbc8a}.
+ */
+DEFINE_GUID(GUID_DEVINTERFACE_VIRT2PHYS,
+	0x539c2135, 0x793a, 0x4926,
+	0xaf, 0xec, 0xd3, 0xa1, 0xb6, 0x1b, 0xbc, 0x8a);
+
+/**
+ * Driver device type for IO control codes.
+ */
+#define VIRT2PHYS_DEVTYPE 0x8000
+
+/**
+ * Translate a valid non-paged virtual address to a physical address.
+ *
+ * Note: A physical address zero (0) is reported if input address
+ * is paged out or not mapped. However, if input is a valid mapping
+ * of I/O port 0x0000, output is also zero. There is no way
+ * to distinguish between these cases by return value only.
+ *
+ * Input: a non-paged virtual address (PVOID).
+ *
+ * Output: the corresponding physical address (LARGE_INTEGER).
+ */
+#define IOCTL_VIRT2PHYS_TRANSLATE CTL_CODE( \
+	VIRT2PHYS_DEVTYPE, 0x800, METHOD_BUFFERED, FILE_ANY_ACCESS)
lib/librte_eal/windows/eal/include/rte_windows.h +2 -0
index ed6e4c148..899ed7d87 100644
@@ -23,6 +23,8 @@
 
 #include <basetsd.h>
 #include <psapi.h>
+#include <setupapi.h>
+#include <winioctl.h>
 
 /* Have GUIDs defined. */
 #ifndef INITGUID
lib/librte_eal/windows/eal/include/unistd.h +3 -0
index 757b7f3c5..6b33005b2 100644
@@ -9,4 +9,7 @@
  * as Microsoft libc does not contain unistd.h. This may be removed
  * in future releases.
  */
+
+#include <io.h>
+
 #endif /* _UNISTD_H_ */
lib/librte_eal/windows/eal/meson.build +12 -0
index 7b0f03f0a..519bb5a3b 100644
@@ -7,24 +7,36 @@ env_objs = []
 env_headers = files(
 	'include/rte_os.h',
 	'include/rte_windows.h',
+	'include/rte_virt2phys.h',
 )
 common_sources = files(
 	'../../common/eal_common_bus.c',
 	'../../common/eal_common_class.c',
 	'../../common/eal_common_devargs.c',
 	'../../common/eal_common_errno.c',
+	'../../common/eal_common_fbarray.c',
 	'../../common/eal_common_launch.c',
 	'../../common/eal_common_lcore.c',
 	'../../common/eal_common_log.c',
+	'../../common/eal_common_mcfg.c',
+	'../../common/eal_common_memalloc.c',
+	'../../common/eal_common_memory.c',
+	'../../common/eal_common_memzone.c',
 	'../../common/eal_common_options.c',
+	'../../common/eal_common_tailqs.c',
 	'../../common/eal_common_thread.c',
+	'../../common/malloc_elem.c',
+	'../../common/malloc_heap.c',
+	'../../common/rte_malloc.c',
 	'../../common/rte_option.c',
 )
 env_sources = files('eal.c',
 	'eal_debug.c',
 	'eal_hugepages.c',
 	'eal_lcore.c',
+	'eal_memalloc.c',
 	'eal_memory.c',
+	'eal_mp.c',
 	'eal_thread.c',
 	'getopt.c',
 )
Signature · 1 line
2.25.1
#18 lore raw

v2 cover eal: Windows basic memory management

In reply to Dmitry Kozlyuk

This patchset implements basic MM with the following features:

* Hugepages are dynamically allocated in user-mode.
* Only 2MB hugepages are supported.
* IOVA is always PA, obtained through kernel-mode driver.
* No 32-bit support (presumably not demanded).
* Ni multi-process support (it is forcefully disabled).
* No-huge mode for testing with IOVA unavailable.

The first commit introduces a new kernel-mode driver, virt2phys. It translates user-mode virtual addresses into physical addresses. On Windows community call 2020-04-01 it was decided this driver can be used for now, later netUIO may pick up its code/interface or not.

New EAL public functions for memory mapping are introduced to mitigate OS differences in DPDK libraries and applications:

* rte_mem_map
* rte_mem_unmap
* rte_mem_lock

To support common MM routines, internal wrappers for low-level memory reservation and file management are introduced. These changes affect Linux and FreeBSD EAL. Shared code is placed unded /unix/ subdirectory (suggested by Thomas).

Also, entire <sys/queue.h> is imported from FreeBSD, replacing existing partial import. There is already a license exception for this file.

Windows MM duplicates quite a lot of code from Linux EAL:

* eal_memalloc_alloc_seg_bulk
* eal_memalloc_free_seg_bulk
* calc_num_pages_per_socket
* rte_eal_hugepage_init

Perhaps this should be left as-is until Windows MM evolves into having some specific requirements for these parts.

Notes on checkpatch warnings:

* No space after comma / no space before closing parent in macros--- definitely a false-positive, unclear how to suppress this.

* Issues from imported BSD code---probably should be ignored?

* Checkpatch is not run against dpdk-kmods (Windows drivers).

---

There were comments from Mellanox and Microsoft outside of this mailing list. Still missing input from Anatoly Burakov.

v2:

* Rebase on ToT. Move all new code shared between Linux and FreeBSD to /unix/ subdirectory, also factor out some existing code there.
* Improve description of Clang issue with rte_page_sizes on Windows. Restore -fstrict-enum for EAL. Check running, not target compiler.
* Use EAL prefix for private facilities instead if RTE.
* Improve documentation comments for new functions.
* Remove co-installer for virt2phys. Add a typecast for clarity.
* Document virt2phys in user guide, improve its own README.
* Explicitly and forcefully disable multi-process.

Dmitry Kozlyuk (9):
eal/windows: do not expose private EAL facilities eal/windows: improve CPU and NUMA node detection eal/windows: initialize hugepage info
eal: introduce internal wrappers for file operations
eal: introduce memory management wrappers
eal: extract common code for memseg list initialization eal/windows: fix rte_page_sizes with Clang on Windows eal/windows: replace sys/queue.h with a complete one from FreeBSD eal/windows: implement basic memory management

 config/meson.build                            |   12 +-
 doc/guides/windows_gsg/build_dpdk.rst         |   20 -
 doc/guides/windows_gsg/index.rst              |    1 +
 doc/guides/windows_gsg/run_apps.rst           |   77 ++
 lib/librte_eal/common/eal_common_fbarray.c    |   57 +-
 lib/librte_eal/common/eal_common_memory.c     |  104 +-
 lib/librte_eal/common/eal_private.h           |  134 +-
 lib/librte_eal/common/malloc_heap.c           |    1 +
 lib/librte_eal/common/meson.build             |    9 +
 lib/librte_eal/freebsd/eal_memory.c           |   55 +-
 lib/librte_eal/include/rte_memory.h           |   74 ++
 lib/librte_eal/linux/eal_memory.c             |   68 +-
 lib/librte_eal/meson.build                    |    5 +
 lib/librte_eal/rte_eal_exports.def            |  119 ++
 lib/librte_eal/rte_eal_version.map            |    4 +
 lib/librte_eal/unix/eal.c                     |   47 +
 lib/librte_eal/unix/eal_memory.c              |  112 ++
 lib/librte_eal/unix/meson.build               |    7 +
 lib/librte_eal/windows/eal.c                  |  160 +++
 lib/librte_eal/windows/eal_hugepages.c        |  108 ++
 lib/librte_eal/windows/eal_lcore.c            |  187 +--
 lib/librte_eal/windows/eal_memalloc.c         |  423 ++++++
 lib/librte_eal/windows/eal_memory.c           | 1133 +++++++++++++++++
 lib/librte_eal/windows/eal_mp.c               |  103 ++
 lib/librte_eal/windows/eal_thread.c           |    1 +
 lib/librte_eal/windows/eal_windows.h          |  129 ++
 lib/librte_eal/windows/include/meson.build    |    2 +
 lib/librte_eal/windows/include/pthread.h      |    2 +
 lib/librte_eal/windows/include/rte_os.h       |   48 +-
 .../windows/include/rte_virt2phys.h           |   34 +
 lib/librte_eal/windows/include/rte_windows.h  |   43 +
 lib/librte_eal/windows/include/sys/queue.h    |  663 +++++++++-
 lib/librte_eal/windows/include/unistd.h       |    3 +
 lib/librte_eal/windows/meson.build            |    4 +
 34 files changed, 3599 insertions(+), 350 deletions(-)
 create mode 100644 doc/guides/windows_gsg/run_apps.rst
 create mode 100644 lib/librte_eal/unix/eal.c
 create mode 100644 lib/librte_eal/unix/eal_memory.c
 create mode 100644 lib/librte_eal/unix/meson.build
 create mode 100644 lib/librte_eal/windows/eal_hugepages.c
 create mode 100644 lib/librte_eal/windows/eal_memalloc.c
 create mode 100644 lib/librte_eal/windows/eal_memory.c
 create mode 100644 lib/librte_eal/windows/eal_mp.c
 create mode 100644 lib/librte_eal/windows/eal_windows.h
 create mode 100644 lib/librte_eal/windows/include/rte_virt2phys.h
 create mode 100644 lib/librte_eal/windows/include/rte_windows.h
Signature · 1 line
2.25.1
#19 lore raw

PATCH v2 1/1 virt2phys: virtual to physical address translator for Windows

This driver supports Windows EAL memory management by translating current process virtual addresses to physical addresses (IOVA). Standalone virt2phys allows using DPDK without PMD and provides a reference implementation.

---

Note: this patch is for dpdk-kmods tree.

 windows/README.rst                          |  92 ++++++++
 windows/virt2phys/virt2phys.c               | 129 +++++++++++
 windows/virt2phys/virt2phys.h               |  34 +++
 windows/virt2phys/virt2phys.inf             |  64 ++++++
 windows/virt2phys/virt2phys.sln             |  27 +++
 windows/virt2phys/virt2phys.vcxproj         | 228 ++++++++++++++++++++
 windows/virt2phys/virt2phys.vcxproj.filters |  36 ++++
 7 files changed, 610 insertions(+)
 create mode 100644 windows/README.rst
 create mode 100755 windows/virt2phys/virt2phys.c
 create mode 100755 windows/virt2phys/virt2phys.h
 create mode 100755 windows/virt2phys/virt2phys.inf
 create mode 100755 windows/virt2phys/virt2phys.sln
 create mode 100755 windows/virt2phys/virt2phys.vcxproj
 create mode 100755 windows/virt2phys/virt2phys.vcxproj.filters
windows/README.rst new file +92 -0
new file mode 100644
index 0000000..e30d4dc
@@ -0,0 +1,92 @@
+Developing Windows Drivers
+==========================
+
+Prerequisites
+-------------
+
+Building Windows Drivers is only possible on Windows.
+
+1. Visual Studio 2019 Community or Professional Edition
+2. Windows Driver Kit (WDK) for Windows 10, version 1903
+
+Follow the official instructions to obtain all of the above:
+https://docs.microsoft.com/en-us/windows-hardware/drivers/download-the-wdk
+
+
+Build the Drivers
+-----------------
+
+Build from Visual Studio
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Open a solution (``*.sln``) with Visual Studio and build it (Ctrl+Shift+B).
+
+
+Build from Command-Line
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Run *Developer Command Prompt for VS 2019* from the Start menu.
+
+Navigate to the solution directory (with ``*.sln``), then run:
+
+.. code-block:: console
+
+    msbuild
+
+To build a particular combination of configuration and platform:
+
+.. code-block:: console
+
+    msbuild -p:Configuration=Debug;Platform=x64
+
+
+Install the Drivers
+-------------------
+
+Disable Driver Signature Enforcement
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+By default Windows prohibits installing and loading drivers without `digital
+signature`_ obtained from Microsoft. For development signature enforcement may
+be disabled as follows.
+
+In Elevated Command Prompt (from this point, sufficient privileges are
+assumed):
+
+.. code-block:: console
+
+    bcdedit -set loadoptions DISABLE_INTEGRITY_CHECKS
+    bcdedit -set TESTSIGNING ON
+    shutdown -r -t 0
+
+Upon reboot, an overlay message should appear on the desktop informing
+that Windows is in test mode, which means it allows loading unsigned drivers.
+
+.. _digital signature: https://docs.microsoft.com/en-us/windows-hardware/drivers/install/driver-signing
+
+
+Install, List, and Remove Drivers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Driver package is by default located in a subdirectory of its source tree,
+e.g. ``x64\Debug\virt2phys\virt2phys`` (note two levels of ``virt2phys``).
+
+To install the driver and bind associated devices to it:
+
+.. code-block:: console
+
+    pnputil /add-driver x64\Debug\virt2phys\virt2phys\virt2phys.inf /install
+
+A graphical confirmation to load an unsigned driver will still appear.
+
+To list installed drivers:
+
+.. code-block:: console
+
+    pnputil /enum-drivers
+
+To remove the driver package and to uninstall its devices:
+
+.. code-block:: console
+
+    pnputil /delete-driver oem2.inf /uninstall
windows/virt2phys/virt2phys.c new file +129 -0

129 lines not on this page · raw message

windows/virt2phys/virt2phys.h new file +34 -0
new file mode 100755
index 0000000..4bb2b4a
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2020 Dmitry Kozlyuk
+ */
+
+/**
+ * @file virt2phys driver interface
+ */
+
+/**
+ * Driver device interface GUID {539c2135-793a-4926-afec-d3a1b61bbc8a}.
+ */
+DEFINE_GUID(GUID_DEVINTERFACE_VIRT2PHYS,
+	0x539c2135, 0x793a, 0x4926,
+	0xaf, 0xec, 0xd3, 0xa1, 0xb6, 0x1b, 0xbc, 0x8a);
+
+/**
+ * Driver device type for IO control codes.
+ */
+#define VIRT2PHYS_DEVTYPE 0x8000
+
+/**
+ * Translate a valid non-paged virtual address to a physical address.
+ *
+ * Note: A physical address zero (0) is reported if input address
+ * is paged out or not mapped. However, if input is a valid mapping
+ * of I/O port 0x0000, output is also zero. There is no way
+ * to distinguish between these cases by return value only.
+ *
+ * Input: a non-paged virtual address (PVOID).
+ *
+ * Output: the corresponding physical address (LARGE_INTEGER).
+ */
+#define IOCTL_VIRT2PHYS_TRANSLATE CTL_CODE( \
+	VIRT2PHYS_DEVTYPE, 0x800, METHOD_BUFFERED, FILE_ANY_ACCESS)
windows/virt2phys/virt2phys.inf new file +64 -0

64 lines not on this page · raw message

windows/virt2phys/virt2phys.sln new file +27 -0
new file mode 100755
index 0000000..0f5ecdc
@@ -0,0 +1,27 @@
+
+Microsoft Visual Studio Solution File, Format Version 12.00
+# Visual Studio Version 16
+VisualStudioVersion = 16.0.29613.14
+MinimumVisualStudioVersion = 10.0.40219.1
+Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "virt2phys", "virt2phys.vcxproj", "{0EEF826B-9391-43A8-A722-BDD6F6115137}"
+EndProject
+Global
+	GlobalSection(SolutionConfigurationPlatforms) = preSolution
+		Debug|x64 = Debug|x64
+		Release|x64 = Release|x64
+	EndGlobalSection
+	GlobalSection(ProjectConfigurationPlatforms) = postSolution
+		{0EEF826B-9391-43A8-A722-BDD6F6115137}.Debug|x64.ActiveCfg = Debug|x64
+		{0EEF826B-9391-43A8-A722-BDD6F6115137}.Debug|x64.Build.0 = Debug|x64
+		{0EEF826B-9391-43A8-A722-BDD6F6115137}.Debug|x64.Deploy.0 = Debug|x64
+		{0EEF826B-9391-43A8-A722-BDD6F6115137}.Release|x64.ActiveCfg = Release|x64
+		{0EEF826B-9391-43A8-A722-BDD6F6115137}.Release|x64.Build.0 = Release|x64
+		{0EEF826B-9391-43A8-A722-BDD6F6115137}.Release|x64.Deploy.0 = Release|x64
+	EndGlobalSection
+	GlobalSection(SolutionProperties) = preSolution
+		HideSolutionNode = FALSE
+	EndGlobalSection
+	GlobalSection(ExtensibilityGlobals) = postSolution
+		SolutionGuid = {845012FB-4471-4A12-A1C4-FF7E05C40E8E}
+	EndGlobalSection
+EndGlobal
windows/virt2phys/virt2phys.vcxproj new file +228 -0

229 lines not on this page · raw message

windows/virt2phys/virt2phys.vcxproj.filters new file +36 -0
new file mode 100755
index 0000000..0fe65fc
@@ -0,0 +1,36 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+  <ItemGroup>
+    <Filter Include="Source Files">
+      <UniqueIdentifier>{4FC737F1-C7A5-4376-A066-2A32D752A2FF}</UniqueIdentifier>
+      <Extensions>cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx</Extensions>
+    </Filter>
+    <Filter Include="Header Files">
+      <UniqueIdentifier>{93995380-89BD-4b04-88EB-625FBE52EBFB}</UniqueIdentifier>
+      <Extensions>h;hpp;hxx;hm;inl;inc;xsd</Extensions>
+    </Filter>
+    <Filter Include="Resource Files">
+      <UniqueIdentifier>{67DA6AB6-F800-4c08-8B7A-83BB121AAD01}</UniqueIdentifier>
+      <Extensions>rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;png;wav;mfcribbon-ms</Extensions>
+    </Filter>
+    <Filter Include="Driver Files">
+      <UniqueIdentifier>{8E41214B-6785-4CFE-B992-037D68949A14}</UniqueIdentifier>
+      <Extensions>inf;inv;inx;mof;mc;</Extensions>
+    </Filter>
+  </ItemGroup>
+  <ItemGroup>
+    <Inf Include="virt2phys.inf">
+      <Filter>Driver Files</Filter>
+    </Inf>
+  </ItemGroup>
+  <ItemGroup>
+    <ClInclude Include="virt2phys.h">
+      <Filter>Header Files</Filter>
+    </ClInclude>
+  </ItemGroup>
+  <ItemGroup>
+    <ClCompile Include="virt2phys.c">
+      <Filter>Source Files</Filter>
+    </ClCompile>
+  </ItemGroup>
+</Project>
Signature · 1 line
2.25.1
Reviewed-by Acked-by on v2 1/1
#20 lore raw

Re: [dpdk-dev] [PATCH v2 1/1] virt2phys: virtual to physical address translator for Windows

Quoted 6 lines · Dmitry Kozlyuk

This driver supports Windows EAL memory management by translating current process virtual addresses to physical addresses (IOVA). Standalone virt2phys allows using DPDK without PMD and provides a reference implementation.

#21 lore raw

PATCH v2 2/10 eal/windows: do not expose private EAL facilities

In reply to Dmitry Kozlyuk

The goal of rte_os.h is to mitigate OS differences for EAL users. In Windows EAL, rte_os.h did excessive things:

1. It included platform SDK headers (windows.h, etc). Those files are huge, require specific inclusion order, and are generally unused by the code including rte_os.h. Declarations from platform SDK may break otherwise platform-independent code, e.g. min, max, ERROR.

2. It included pthread.h, which is clearly not always required.

3. It defined functions private to Windows EAL.

Reorganize Windows EAL includes in the following way:

1. Create rte_windows.h to properly import Windows-specific facilities. Primary users are bus drivers, tests, and external applications.

2. Remove platform SDK includes from rte_os.h to prevent breaking otherwise portable code by including rte_os.h on Windows. Copy necessary definitions to avoid including those headers.

3. Remove pthread.h include from rte_os.h.

4. Move declarations private to Windows EAL into eal_windows.h.

Fixes 428eb983f5f7 ("eal: add OS specific header file")
---
 lib/librte_eal/windows/eal.c                 |  2 +
 lib/librte_eal/windows/eal_lcore.c           |  2 +
 lib/librte_eal/windows/eal_thread.c          |  1 +
 lib/librte_eal/windows/eal_windows.h         | 29 +++++++++++++
 lib/librte_eal/windows/include/meson.build   |  1 +
 lib/librte_eal/windows/include/pthread.h     |  2 +
 lib/librte_eal/windows/include/rte_os.h      | 44 ++++++--------------
 lib/librte_eal/windows/include/rte_windows.h | 41 ++++++++++++++++++
 8 files changed, 91 insertions(+), 31 deletions(-)
 create mode 100644 lib/librte_eal/windows/eal_windows.h
 create mode 100644 lib/librte_eal/windows/include/rte_windows.h
lib/librte_eal/windows/eal.c +2 -0
index e4b50df3b..2cf7a04ef 100644
@@ -18,6 +18,8 @@
 #include <eal_options.h>
 #include <eal_private.h>
 
+#include "eal_windows.h"
+
  /* Allow the application to print its usage message too if set */
 static rte_usage_hook_t	rte_application_usage_hook;
 
lib/librte_eal/windows/eal_lcore.c +2 -0
index b3a6c63af..82ee45413 100644
@@ -2,12 +2,14 @@
  * Copyright(c) 2019 Intel Corporation
  */
 
+#include <pthread.h>
 #include <stdint.h>
 
 #include <rte_common.h>
 
 #include "eal_private.h"
 #include "eal_thread.h"
+#include "eal_windows.h"
 
 /* global data structure that contains the CPU map */
 static struct _wcpu_map {
lib/librte_eal/windows/eal_thread.c +1 -0
index 9e4bbaa08..e149199a6 100644
@@ -14,6 +14,7 @@
 #include <eal_thread.h>
 
 #include "eal_private.h"
+#include "eal_windows.h"
 
 RTE_DEFINE_PER_LCORE(unsigned int, _lcore_id) = LCORE_ID_ANY;
 RTE_DEFINE_PER_LCORE(unsigned int, _socket_id) = (unsigned int)SOCKET_ID_ANY;
lib/librte_eal/windows/eal_windows.h new file +29 -0
new file mode 100644
index 000000000..fadd676b2
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2020 Dmitry Kozlyuk
+ */
+
+#ifndef _EAL_WINDOWS_H_
+#define _EAL_WINDOWS_H_
+
+/**
+ * @file Facilities private to Windows EAL
+ */
+
+#include <rte_windows.h>
+
+/**
+ * Create a map of processors and cores on the system.
+ */
+void eal_create_cpu_map(void);
+
+/**
+ * Create a thread.
+ *
+ * @param thread
+ *   The location to store the thread id if successful.
+ * @return
+ *   0 for success, -1 if the thread is not created.
+ */
+int eal_thread_create(pthread_t *thread);
+
+#endif /* _EAL_WINDOWS_H_ */
lib/librte_eal/windows/include/meson.build +1 -0
index 7d18dd52f..5fb1962ac 100644
@@ -5,4 +5,5 @@ includes += include_directories('.')
 
 headers += files(
         'rte_os.h',
+        'rte_windows.h',
 )
lib/librte_eal/windows/include/pthread.h +2 -0
index b9dd18e56..cfd53f0b8 100644
@@ -5,6 +5,8 @@
 #ifndef _PTHREAD_H_
 #define _PTHREAD_H_
 
+#include <stdint.h>
+
 /**
  * This file is required to support the common code in eal_common_proc.c,
  * eal_common_thread.c and common\include\rte_per_lcore.h as Microsoft libc
lib/librte_eal/windows/include/rte_os.h +13 -31

75 lines not on this page · raw message

lib/librte_eal/windows/include/rte_windows.h new file +41 -0

41 lines not on this page · raw message

Signature · 1 line
2.25.1
#22 lore raw

PATCH v2 3/10 eal/windows: improve CPU and NUMA node detection

In reply to Dmitry Kozlyuk

1. Map CPU cores to their respective NUMA nodes as reported by system.
2. Support systems with more than 64 cores (multiple processor groups).
3. Fix magic constants, styling issues, and compiler warnings.
4. Add EAL private function to map DPDK socket ID to NUMA node number.

Fixes 53ffd9f080fc ("eal/windows: add minimum viable code")
---
 lib/librte_eal/windows/eal_lcore.c   | 185 +++++++++++++++++----------
 lib/librte_eal/windows/eal_windows.h |  10 ++
 2 files changed, 124 insertions(+), 71 deletions(-)
lib/librte_eal/windows/eal_lcore.c +114 -71
index 82ee45413..9d931d50a 100644
@@ -3,103 +3,146 @@
  */
 
 #include <pthread.h>
+#include <stdbool.h>
 #include <stdint.h>
 
 #include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_lcore.h>
+#include <rte_os.h>
 
 #include "eal_private.h"
 #include "eal_thread.h"
 #include "eal_windows.h"
 
-/* global data structure that contains the CPU map */
-static struct _wcpu_map {
-	unsigned int total_procs;
-	unsigned int proc_sockets;
-	unsigned int proc_cores;
-	unsigned int reserved;
-	struct _win_lcore_map {
-		uint8_t socket_id;
-		uint8_t core_id;
-	} wlcore_map[RTE_MAX_LCORE];
-} wcpu_map = { 0 };
-
-/*
- * Create a map of all processors and associated cores on the system
- */
+/** Number of logical processors (cores) in a processor group (32 or 64). */
+#define EAL_PROCESSOR_GROUP_SIZE (sizeof(KAFFINITY) * CHAR_BIT)
+
+struct lcore_map {
+	uint8_t socket_id;
+	uint8_t core_id;
+};
+
+struct socket_map {
+	uint16_t node_id;
+};
+
+struct cpu_map {
+	unsigned int socket_count;
+	unsigned int lcore_count;
+	struct lcore_map lcores[RTE_MAX_LCORE];
+	struct socket_map sockets[RTE_MAX_NUMA_NODES];
+};
+
+static struct cpu_map cpu_map = { 0 };
+
 void
-eal_create_cpu_map()
+eal_create_cpu_map(void)
 {
-	wcpu_map.total_procs =
-		GetActiveProcessorCount(ALL_PROCESSOR_GROUPS);
-
-	LOGICAL_PROCESSOR_RELATIONSHIP lprocRel;
-	DWORD lprocInfoSize = 0;
-	BOOL ht_enabled = FALSE;
-
-	/* First get the processor package information */
-	lprocRel = RelationProcessorPackage;
-	/* Determine the size of buffer we need (pass NULL) */
-	GetLogicalProcessorInformationEx(lprocRel, NULL, &lprocInfoSize);
-	wcpu_map.proc_sockets = lprocInfoSize / 48;
-
-	lprocInfoSize = 0;
-	/* Next get the processor core information */
-	lprocRel = RelationProcessorCore;
-	GetLogicalProcessorInformationEx(lprocRel, NULL, &lprocInfoSize);
-	wcpu_map.proc_cores = lprocInfoSize / 48;
-
-	if (wcpu_map.total_procs > wcpu_map.proc_cores)
-		ht_enabled = TRUE;
-
-	/* Distribute the socket and core ids appropriately
-	 * across the logical cores. For now, split the cores
-	 * equally across the sockets.
-	 */
-	unsigned int lcore = 0;
-	for (unsigned int socket = 0; socket <
-			wcpu_map.proc_sockets; ++socket) {
-		for (unsigned int core = 0;
-			core < (wcpu_map.proc_cores / wcpu_map.proc_sockets);
-			++core) {
-			wcpu_map.wlcore_map[lcore]
-					.socket_id = socket;
-			wcpu_map.wlcore_map[lcore]
-					.core_id = core;
-			lcore++;
-			if (ht_enabled) {
-				wcpu_map.wlcore_map[lcore]
-					.socket_id = socket;
-				wcpu_map.wlcore_map[lcore]
-					.core_id = core;
-				lcore++;
+	SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *infos, *info;
+	DWORD infos_size;
+	bool full = false;
+
+	infos_size = 0;
+	if (!GetLogicalProcessorInformationEx(
+			RelationNumaNode, NULL, &infos_size)) {
+		DWORD error = GetLastError();
+		if (error != ERROR_INSUFFICIENT_BUFFER) {
+			rte_panic("cannot get NUMA node info size, error %lu",
+				GetLastError());
+		}
+	}
+
+	infos = malloc(infos_size);
+	if (infos == NULL) {
+		rte_panic("cannot allocate memory for NUMA node information");
+		return;
+	}
+
+	if (!GetLogicalProcessorInformationEx(
+			RelationNumaNode, infos, &infos_size)) {
+		rte_panic("cannot get NUMA node information, error %lu",
+			GetLastError());
+	}
+
+	info = infos;
+	while ((uint8_t *)info - (uint8_t *)infos < infos_size) {
+		unsigned int node_id = info->NumaNode.NodeNumber;
+		GROUP_AFFINITY *cores = &info->NumaNode.GroupMask;
+		struct lcore_map *lcore;
+		unsigned int i, socket_id;
+
+		/* NUMA node may be reported multiple times if it includes
+		 * cores from different processor groups, e. g. 80 cores
+		 * of a physical processor comprise one NUMA node, but two
+		 * processor groups, because group size is limited by 32/64.
+		 */
+		for (socket_id = 0; socket_id < cpu_map.socket_count;
+		    socket_id++) {
+			if (cpu_map.sockets[socket_id].node_id == node_id)
+				break;
+		}
+
+		if (socket_id == cpu_map.socket_count) {
+			if (socket_id == RTE_DIM(cpu_map.sockets)) {
+				full = true;
+				goto exit;
 			}
+
+			cpu_map.sockets[socket_id].node_id = node_id;
+			cpu_map.socket_count++;
+		}
+
+		for (i = 0; i < EAL_PROCESSOR_GROUP_SIZE; i++) {
+			if ((cores->Mask & ((KAFFINITY)1 << i)) == 0)
+				continue;
+
+			if (cpu_map.lcore_count == RTE_DIM(cpu_map.lcores)) {
+				full = true;
+				goto exit;
+			}
+
+			lcore = &cpu_map.lcores[cpu_map.lcore_count];
+			lcore->socket_id = socket_id;
+			lcore->core_id =
+				cores->Group * EAL_PROCESSOR_GROUP_SIZE + i;
+			cpu_map.lcore_count++;
 		}
+
+		info = (SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *)(
+			(uint8_t *)info + info->Size);
+	}
+
+exit:
+	if (full) {
+		/* RTE_LOG() may not be available, but this is important. */
+		fprintf(stderr, "Enumerated maximum of %u NUMA nodes and %u cores\n",
+			cpu_map.socket_count, cpu_map.lcore_count);
 	}
+
+	free(infos);
 }
 
-/*
- * Check if a cpu is present by the presence of the cpu information for it
- */
 int
 eal_cpu_detected(unsigned int lcore_id)
 {
-	return (lcore_id < wcpu_map.total_procs);
+	return lcore_id < cpu_map.lcore_count;
 }
 
-/*
- * Get CPU socket id for a logical core
- */
 unsigned
 eal_cpu_socket_id(unsigned int lcore_id)
 {
-	return wcpu_map.wlcore_map[lcore_id].socket_id;
+	return cpu_map.lcores[lcore_id].socket_id;
 }
 
-/*
- * Get CPU socket id (NUMA node) for a logical core
- */
 unsigned
 eal_cpu_core_id(unsigned int lcore_id)
 {
-	return wcpu_map.wlcore_map[lcore_id].core_id;
+	return cpu_map.lcores[lcore_id].core_id;
+}
+
+unsigned int
+eal_socket_numa_node(unsigned int socket_id)
+{
+	return cpu_map.sockets[socket_id].node_id;
 }
lib/librte_eal/windows/eal_windows.h +10 -0
index fadd676b2..390d2fd66 100644
@@ -26,4 +26,14 @@ void eal_create_cpu_map(void);
  */
 int eal_thread_create(pthread_t *thread);
 
+/**
+ * Get system NUMA node number for a socket ID.
+ *
+ * @param socket_id
+ *  Valid EAL socket ID.
+ * @return
+ *  NUMA node number to use with Win32 API.
+ */
+unsigned int eal_socket_numa_node(unsigned int socket_id);
+
 #endif /* _EAL_WINDOWS_H_ */
Signature · 1 line
2.25.1
#23 lore raw

PATCH v2 4/10 eal/windows: initialize hugepage info

In reply to Dmitry Kozlyuk

Add hugepages discovery ("large pages" in Windows terminology) and update documentation for required privilege setup.

---
 config/meson.build                     |   2 +
 doc/guides/windows_gsg/build_dpdk.rst  |  20 -----
 doc/guides/windows_gsg/index.rst       |   1 +
 doc/guides/windows_gsg/run_apps.rst    |  47 +++++++++++
 lib/librte_eal/windows/eal.c           |  14 ++++
 lib/librte_eal/windows/eal_hugepages.c | 108 +++++++++++++++++++++++++
 lib/librte_eal/windows/meson.build     |   1 +
 7 files changed, 173 insertions(+), 20 deletions(-)
 create mode 100644 doc/guides/windows_gsg/run_apps.rst
 create mode 100644 lib/librte_eal/windows/eal_hugepages.c
config/meson.build +2 -0
index 58421342b..4607655d9 100644
@@ -263,6 +263,8 @@ if is_windows
 	if cc.get_id() == 'gcc'
 		add_project_arguments('-D__USE_MINGW_ANSI_STDIO', language: 'c')
 	endif
+
+	add_project_link_arguments('-ladvapi32', language: 'c')
 endif
 
 if get_option('b_lto')
doc/guides/windows_gsg/build_dpdk.rst +0 -20
index d46e84e3f..650483e3b 100644
@@ -111,23 +111,3 @@ Depending on the distribution, paths in this file may need adjustments.
 
     meson --cross-file config/x86/meson_mingw.txt -Dexamples=helloworld build
     ninja -C build
-
-
-Run the helloworld example
-==========================
-
-Navigate to the examples in the build directory and run `dpdk-helloworld.exe`.
-
-.. code-block:: console
-
-    cd C:\Users\me\dpdk\build\examples
-    dpdk-helloworld.exe
-    hello from core 1
-    hello from core 3
-    hello from core 0
-    hello from core 2
-
-Note for MinGW-w64: applications are linked to ``libwinpthread-1.dll``
-by default. To run the example, either add toolchain executables directory
-to the PATH or copy the library to the working directory.
-Alternatively, static linking may be used (mind the LGPLv2.1 license).
doc/guides/windows_gsg/index.rst +1 -0
index d9b7990a8..e94593572 100644
@@ -12,3 +12,4 @@ Getting Started Guide for Windows
 
     intro
     build_dpdk
+    run_apps
doc/guides/windows_gsg/run_apps.rst new file +47 -0

47 lines not on this page · raw message

lib/librte_eal/windows/eal.c +14 -0
index 2cf7a04ef..63461f51a 100644
@@ -18,8 +18,11 @@
 #include <eal_options.h>
 #include <eal_private.h>
 
+#include "eal_hugepages.h"
 #include "eal_windows.h"
 
+#define MEMSIZE_IF_NO_HUGE_PAGE (64ULL * 1024ULL * 1024ULL)
+
  /* Allow the application to print its usage message too if set */
 static rte_usage_hook_t	rte_application_usage_hook;
 
@@ -242,6 +245,17 @@ rte_eal_init(int argc, char **argv)
 	if (fctret < 0)
 		exit(1);
 
+	if (!internal_config.no_hugetlbfs && (eal_hugepage_info_init() < 0)) {
+		rte_eal_init_alert("Cannot get hugepage information");
+		rte_errno = EACCES;
+		return -1;
+	}
+
+	if (internal_config.memory == 0 && !internal_config.force_sockets) {
+		if (internal_config.no_hugetlbfs)
+			internal_config.memory = MEMSIZE_IF_NO_HUGE_PAGE;
+	}
+
 	eal_thread_init_master(rte_config.master_lcore);
 
 	RTE_LCORE_FOREACH_SLAVE(i) {
lib/librte_eal/windows/eal_hugepages.c new file +108 -0

108 lines not on this page · raw message

lib/librte_eal/windows/meson.build +1 -0
index 09dd4ab2f..5f118bfe2 100644
@@ -6,6 +6,7 @@ subdir('include')
 sources += files(
 	'eal.c',
 	'eal_debug.c',
+	'eal_hugepages.c',
 	'eal_lcore.c',
 	'eal_thread.c',
 	'getopt.c',
Signature · 1 line
2.25.1
#24 lore raw

PATCH v2 5/10 eal: introduce internal wrappers for file operations

In reply to Dmitry Kozlyuk

EAL common code uses file locking and truncation. Introduce OS-independent wrappers in order to support both Linux/FreeBSD and Windows:

* eal_file_lock: lock or unlock an open file.
* eal_file_truncate: enforce a given size for an open file.

Wrappers follow POSIX semantics, but interface is not POSIX, so that it can be made more clean, e.g. by not mixing locking operation and behaviour on conflict.

Implementation for Linux and FreeBSD is placed in "unix" subdirectory, which is intended for common code between the two. Files should be named after the ones from which the code is factored in OS subdirectory.

---
 lib/librte_eal/common/eal_private.h | 45 ++++++++++++++++
 lib/librte_eal/meson.build          |  4 ++
 lib/librte_eal/unix/eal.c           | 47 ++++++++++++++++
 lib/librte_eal/unix/meson.build     |  6 +++
 lib/librte_eal/windows/eal.c        | 83 +++++++++++++++++++++++++++++
 5 files changed, 185 insertions(+)
 create mode 100644 lib/librte_eal/unix/eal.c
 create mode 100644 lib/librte_eal/unix/meson.build
lib/librte_eal/common/eal_private.h +45 -0
index ddcfbe2e4..65d61ff13 100644
@@ -443,4 +443,49 @@ rte_option_usage(void);
 uint64_t
 eal_get_baseaddr(void);
 
+/** File locking operation. */
+enum eal_flock_op {
+	EAL_FLOCK_SHARED,    /**< Acquire a shared lock. */
+	EAL_FLOCK_EXCLUSIVE, /**< Acquire an exclusive lock. */
+	EAL_FLOCK_UNLOCK     /**< Release a previously taken lock. */
+};
+
+/** Behavior on file locking conflict. */
+enum eal_flock_mode {
+	EAL_FLOCK_WAIT,  /**< Wait until the file gets unlocked to lock it. */
+	EAL_FLOCK_RETURN /**< Return immediately if the file is locked. */
+};
+
+/**
+ * Lock or unlock the file.
+ *
+ * On failure @code rte_errno @endcode is set to the error code
+ * specified by POSIX flock(3) description.
+ *
+ * @param fd
+ *  Opened file descriptor.
+ * @param op
+ *  Operation to perform.
+ * @param mode
+ *  Behavior on conflict.
+ * @return
+ *  0 on success, (-1) on failure.
+ */
+int eal_file_lock(int fd, enum eal_flock_op op, enum eal_flock_mode mode);
+
+/**
+ * Truncate or extend the file to the specified size.
+ *
+ * On failure @code rte_errno @endcode is set to the error code
+ * specified by POSIX ftruncate(3) description.
+ *
+ * @param fd
+ *  Opened file descriptor.
+ * @param size
+ *  Desired file size.
+ * @return
+ *  0 on success, (-1) on failure.
+ */
+int eal_file_truncate(int fd, ssize_t size);
+
 #endif /* _EAL_PRIVATE_H_ */
lib/librte_eal/meson.build +4 -0
index 9d219a0e6..1f89efb88 100644
@@ -6,6 +6,10 @@ subdir('include')
 
 subdir('common')
 
+if not is_windows
+	subdir('unix')
+endif
+
 dpdk_conf.set('RTE_EXEC_ENV_' + exec_env.to_upper(), 1)
 subdir(exec_env)
 
lib/librte_eal/unix/eal.c new file +47 -0

47 lines not on this page · raw message

lib/librte_eal/unix/meson.build new file +6 -0
new file mode 100644
index 000000000..13564838e
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020 Dmitry Kozlyuk
+
+sources += files(
+	'eal.c',
+)
lib/librte_eal/windows/eal.c +83 -0

89 lines not on this page · raw message

Signature · 1 line
2.25.1
#25 lore raw

PATCH v2 6/10 eal: introduce memory management wrappers

In reply to Dmitry Kozlyuk

System meory management is implemented differently for POSIX and Windows. Introduce wrapper functions for operations used across DPDK:

* rte_mem_map()
Create memory mapping for a regular file or a page file (swap). This supports mapping to a reserved memory region even on Windows.

* rte_mem_unmap()
Remove mapping created with rte_mem_map().

* rte_get_page_size()
Obtain default system page size.

* rte_mem_lock()
Make arbitrary-sized memory region non-swappable.

Wrappers follow POSIX semantics limited to DPDK tasks, but their signatures deliberately differ from POSIX ones to be more safe and expressive.

---
 config/meson.build                   |  10 +-
 lib/librte_eal/common/eal_private.h  |  51 +++-
 lib/librte_eal/include/rte_memory.h  |  68 +++++
 lib/librte_eal/rte_eal_exports.def   |   4 +
 lib/librte_eal/rte_eal_version.map   |   4 +
 lib/librte_eal/unix/eal_memory.c     | 112 +++++++
 lib/librte_eal/unix/meson.build      |   1 +
 lib/librte_eal/windows/eal.c         |   6 +
 lib/librte_eal/windows/eal_memory.c  | 433 +++++++++++++++++++++++++++
 lib/librte_eal/windows/eal_windows.h |  67 +++++
 lib/librte_eal/windows/meson.build   |   1 +
 11 files changed, 753 insertions(+), 4 deletions(-)
 create mode 100644 lib/librte_eal/unix/eal_memory.c
 create mode 100644 lib/librte_eal/windows/eal_memory.c
config/meson.build +8 -2
index 4607655d9..bceb5ef7b 100644
@@ -256,14 +256,20 @@ if is_freebsd
 endif
 
 if is_windows
-	# Minimum supported API is Windows 7.
-	add_project_arguments('-D_WIN32_WINNT=0x0601', language: 'c')
+	# VirtualAlloc2() is available since Windows 10 / Server 2016.
+	add_project_arguments('-D_WIN32_WINNT=0x0A00', language: 'c')
 
 	# Use MinGW-w64 stdio, because DPDK assumes ANSI-compliant formatting.
 	if cc.get_id() == 'gcc'
 		add_project_arguments('-D__USE_MINGW_ANSI_STDIO', language: 'c')
 	endif
 
+	# Contrary to docs, VirtualAlloc2() is exported by mincore.lib
+	# in Windows SDK, while MinGW exports it by advapi32.a.
+	if is_ms_linker
+		add_project_link_arguments('-lmincore', language: 'c')
+	endif
+
 	add_project_link_arguments('-ladvapi32', language: 'c')
 endif
 
lib/librte_eal/common/eal_private.h +49 -2

73 lines not on this page · raw message

lib/librte_eal/include/rte_memory.h +68 -0

74 lines not on this page · raw message

lib/librte_eal/rte_eal_exports.def +4 -0
index 12a6c79d6..bacf9a107 100644
@@ -5,5 +5,9 @@ EXPORTS
 	rte_eal_mp_remote_launch
 	rte_eal_mp_wait_lcore
 	rte_eal_remote_launch
+	rte_get_page_size
 	rte_log
+	rte_mem_lock
+	rte_mem_map
+	rte_mem_unmap
 	rte_vlog
lib/librte_eal/rte_eal_version.map +4 -0
index f9ede5b41..07128898f 100644
@@ -337,5 +337,9 @@ EXPERIMENTAL {
 	rte_thread_is_intr;
 
 	# added in 20.05
+	rte_get_page_size;
 	rte_log_can_log;
+	rte_mem_lock;
+	rte_mem_map;
+	rte_mem_unmap;
 };
lib/librte_eal/unix/eal_memory.c new file +112 -0

112 lines not on this page · raw message

lib/librte_eal/unix/meson.build +1 -0
index 13564838e..50c019a56 100644
@@ -3,4 +3,5 @@
 
 sources += files(
 	'eal.c',
+	'eal_memory.c',
 )
lib/librte_eal/windows/eal.c +6 -0
index 9dba895e7..cf55b56da 100644
@@ -339,6 +339,12 @@ rte_eal_init(int argc, char **argv)
 			internal_config.memory = MEMSIZE_IF_NO_HUGE_PAGE;
 	}
 
+	if (eal_mem_win32api_init() < 0) {
+		rte_eal_init_alert("Cannot access Win32 memory management");
+		rte_errno = ENOTSUP;
+		return -1;
+	}
+
 	eal_thread_init_master(rte_config.master_lcore);
 
 	RTE_LCORE_FOREACH_SLAVE(i) {
lib/librte_eal/windows/eal_memory.c new file +433 -0

433 lines not on this page · raw message

lib/librte_eal/windows/eal_windows.h +67 -0

71 lines not on this page · raw message

lib/librte_eal/windows/meson.build +1 -0
index 5f118bfe2..81d3ee095 100644
@@ -8,6 +8,7 @@ sources += files(
 	'eal_debug.c',
 	'eal_hugepages.c',
 	'eal_lcore.c',
+	'eal_memory.c',
 	'eal_thread.c',
 	'getopt.c',
 )
Signature · 1 line
2.25.1
#26 lore raw

Re: [dpdk-dev] [PATCH v2 06/10] eal: introduce memory management wrappers

Quoted 15 lines

Subject: [PATCH v2 06/10] eal: introduce memory management wrappers

System meory management is implemented differently for POSIX and Windows. Introduce wrapper functions for operations used across DPDK:

* rte_mem_map()
Create memory mapping for a regular file or a page file (swap). This supports mapping to a reserved memory region even on Windows.

* rte_mem_unmap()
Remove mapping created with rte_mem_map().

* rte_get_page_size()
Obtain default system page size.

Quoted 8 lines

* rte_mem_lock()
Make arbitrary-sized memory region non-swappable.

Wrappers follow POSIX semantics limited to DPDK tasks, but their signatures deliberately differ from POSIX ones to be more safe and expressive.

---

<Snip!>

Quoted 19 lines
(quoted diff) +2 -0
+void *
+rte_mem_map(void *requested_addr, size_t size, enum rte_mem_prot

prot,

lib/librte_eal/unix/eal_memory.c new file +16 -0
+	enum rte_map_flags flags, int fd, size_t offset) {
+	int sys_prot = 0;
+	int sys_flags = 0;
+
+	sys_prot = mem_rte_to_sys_prot(prot);
+
+	if (flags & RTE_MAP_SHARED)
+		sys_flags |= MAP_SHARED;
+	if (flags & RTE_MAP_ANONYMOUS)
+		sys_flags |= MAP_ANONYMOUS;
+	if (flags & RTE_MAP_PRIVATE)
+		sys_flags |= MAP_PRIVATE;
+	if (flags & RTE_MAP_FIXED)
+		sys_flags |= MAP_FIXED;
+
+	return mem_map(requested_addr, size, sys_prot, sys_flags, fd,
Quoted 2 lines

offset);
+}

Looks like there is a different behavior in the Windows and Unix implementation of rte_mem_map in case of memory mapping failure, in Windows the function returns NULL however in Unix the return value is the MAP_FAILED value of mmap which is ((void *)-1)

#27 lore raw

PATCH v2 7/10 eal: extract common code for memseg list initialization

In reply to Dmitry Kozlyuk

All supported OS create memory segment lists (MSL) and reserve VA space for them in a nearly identical way. Move common code into EAL private functions to reduce duplication.

---
 lib/librte_eal/common/eal_common_memory.c | 54 ++++++++++++++++++
 lib/librte_eal/common/eal_private.h       | 34 ++++++++++++
 lib/librte_eal/freebsd/eal_memory.c       | 54 +++---------------
 lib/librte_eal/linux/eal_memory.c         | 68 +++++------------------
 4 files changed, 110 insertions(+), 100 deletions(-)
lib/librte_eal/common/eal_common_memory.c +54 -0
index cc7d54e0c..d9764681a 100644
@@ -25,6 +25,7 @@
 #include "eal_private.h"
 #include "eal_internal_cfg.h"
 #include "eal_memcfg.h"
+#include "eal_options.h"
 #include "malloc_heap.h"
 
 /*
@@ -182,6 +183,59 @@ eal_get_virtual_area(void *requested_addr, size_t *size,
 	return aligned_addr;
 }
 
+int
+eal_reserve_memseg_list(struct rte_memseg_list *msl,
+		enum eal_mem_reserve_flags flags)
+{
+	uint64_t page_sz;
+	size_t mem_sz;
+	void *addr;
+
+	page_sz = msl->page_sz;
+	mem_sz = page_sz * msl->memseg_arr.len;
+
+	addr = eal_get_virtual_area(msl->base_va, &mem_sz, page_sz, 0, flags);
+	if (addr == NULL) {
+		if (rte_errno == EADDRNOTAVAIL)
+			RTE_LOG(ERR, EAL, "Cannot reserve %llu bytes at [%p] - "
+				"please use '--" OPT_BASE_VIRTADDR "' option\n",
+				(unsigned long long)mem_sz, msl->base_va);
+		else
+			RTE_LOG(ERR, EAL, "Cannot reserve memory\n");
+		return -1;
+	}
+	msl->base_va = addr;
+	msl->len = mem_sz;
+
+	return 0;
+}
+
+int
+eal_alloc_memseg_list(struct rte_memseg_list *msl, uint64_t page_sz,
+		int n_segs, int socket_id, int type_msl_idx, bool heap)
+{
+	char name[RTE_FBARRAY_NAME_LEN];
+
+	snprintf(name, sizeof(name), MEMSEG_LIST_FMT, page_sz >> 10, socket_id,
+		 type_msl_idx);
+	if (rte_fbarray_init(&msl->memseg_arr, name, n_segs,
+			sizeof(struct rte_memseg))) {
+		RTE_LOG(ERR, EAL, "Cannot allocate memseg list: %s\n",
+			rte_strerror(rte_errno));
+		return -1;
+	}
+
+	msl->page_sz = page_sz;
+	msl->socket_id = socket_id;
+	msl->base_va = NULL;
+	msl->heap = heap;
+
+	RTE_LOG(DEBUG, EAL, "Memseg list allocated: 0x%zxkB at socket %i\n",
+			(size_t)page_sz >> 10, socket_id);
+
+	return 0;
+}
+
 static struct rte_memseg *
 virt2memseg(const void *addr, const struct rte_memseg_list *msl)
 {
lib/librte_eal/common/eal_private.h +34 -0
index 1e89338f2..76938e379 100644
@@ -246,6 +246,40 @@ void *
 eal_get_virtual_area(void *requested_addr, size_t *size, size_t page_sz,
 	int flags, int mmap_flags);
 
+/**
+ * Reserve VA space for a memory segment list.
+ *
+ * @param msl
+ *  Memory segment list with page size defined.
+ * @param flags
+ *  Extra memory reservation flags. Can be 0 if unnecessary.
+ * @return
+ *  0 on success, (-1) on failure and rte_errno is set.
+ */
+int
+eal_reserve_memseg_list(struct rte_memseg_list *msl,
+	enum eal_mem_reserve_flags flags);
+
+/**
+ * Initialize a memory segment list with its backing storage.
+ *
+ * @param msl
+ *  Memory segment list to be filled.
+ * @param page_sz
+ *  Size of segment pages in the MSL.
+ * @param n_segs
+ *  Number of segments.
+ * @param socket_id
+ *  Socket ID. Must not be SOCKET_ID_ANY.
+ * @param type_msl_idx
+ *  Index of the MSL among other MSLs of the same socket and page size.
+ * @param heap
+ *  Mark MSL as pointing to a heap.
+ */
+int
+eal_alloc_memseg_list(struct rte_memseg_list *msl, uint64_t page_sz,
+	int n_segs, int socket_id, int type_msl_idx, bool heap);
+
 /**
  * Get cpu core_id.
  *
lib/librte_eal/freebsd/eal_memory.c +8 -46

76 lines not on this page · raw message

lib/librte_eal/linux/eal_memory.c +14 -54

129 lines not on this page · raw message

Signature · 1 line
2.25.1
#28 lore raw

PATCH v2 8/10 eal/windows: fix rte_page_sizes with Clang on Windows

In reply to Dmitry Kozlyuk

Clang on Windows follows MS ABI where enum values are limited to 2^31-1. Enum rte_page_size has members valued above this limit, which get wrapped to zero, resulting in compilation error (duplicate values in enum). Using MS ABI is mandatory for Windows EAL to call Win32 APIs.

Define these values outside of the enum for Clang on Windows only. This does not affect runtime, because Windows doesn't run on machines with 4GiB and 16GiB hugepages.

---
 lib/librte_eal/include/rte_memory.h | 6 ++++++
 1 file changed, 6 insertions(+)
lib/librte_eal/include/rte_memory.h +6 -0
index 1b7c3e5df..3ec673f51 100644
@@ -34,8 +34,14 @@ enum rte_page_sizes {
 	RTE_PGSIZE_256M  = 1ULL << 28,
 	RTE_PGSIZE_512M  = 1ULL << 29,
 	RTE_PGSIZE_1G    = 1ULL << 30,
+/* Work around Clang on Windows being limited to 32-bit underlying type. */
+#if !defined(RTE_CC_CLANG) || !defined(RTE_EXEC_ENV_WINDOWS)
 	RTE_PGSIZE_4G    = 1ULL << 32,
 	RTE_PGSIZE_16G   = 1ULL << 34,
+#else
+#define RTE_PGSIZE_4G  (1ULL << 32)
+#define RTE_PGSIZE_16G (1ULL << 34)
+#endif
 };
 
 #define SOCKET_ID_ANY -1                    /**< Any NUMA socket. */
Signature · 1 line
2.25.1
#29 lore raw

PATCH v2 9/10 eal/windows: replace sys/queue.h with a complete one from FreeBSD

In reply to Dmitry Kozlyuk

Limited version imported previously lacks at least SLIST macros. Import a complete file from FreeBSD, since its license exception is already approved by Technical Board.

---
 lib/librte_eal/windows/include/sys/queue.h | 663 +++++++++++++++++++--
 1 file changed, 601 insertions(+), 62 deletions(-)
lib/librte_eal/windows/include/sys/queue.h +601 -62
index a65949a78..9756bee6f 100644
@@ -8,7 +8,36 @@
 #define	_SYS_QUEUE_H_
 
 /*
- * This file defines tail queues.
+ * This file defines four types of data structures: singly-linked lists,
+ * singly-linked tail queues, lists and tail queues.
+ *
+ * A singly-linked list is headed by a single forward pointer. The elements
+ * are singly linked for minimum space and pointer manipulation overhead at
+ * the expense of O(n) removal for arbitrary elements. New elements can be
+ * added to the list after an existing element or at the head of the list.
+ * Elements being removed from the head of the list should use the explicit
+ * macro for this purpose for optimum efficiency. A singly-linked list may
+ * only be traversed in the forward direction.  Singly-linked lists are ideal
+ * for applications with large datasets and few or no removals or for
+ * implementing a LIFO queue.
+ *
+ * A singly-linked tail queue is headed by a pair of pointers, one to the
+ * head of the list and the other to the tail of the list. The elements are
+ * singly linked for minimum space and pointer manipulation overhead at the
+ * expense of O(n) removal for arbitrary elements. New elements can be added
+ * to the list after an existing element, at the head of the list, or at the
+ * end of the list. Elements being removed from the head of the tail queue
+ * should use the explicit macro for this purpose for optimum efficiency.
+ * A singly-linked tail queue may only be traversed in the forward direction.
+ * Singly-linked tail queues are ideal for applications with large datasets
+ * and few or no removals or for implementing a FIFO queue.
+ *
+ * A list is headed by a single forward pointer (or an array of forward
+ * pointers for a hash table header). The elements are doubly linked
+ * so that an arbitrary element can be removed without a need to
+ * traverse the list. New elements can be added to the list before
+ * or after an existing element or at the head of the list. A list
+ * may be traversed in either direction.
  *
  * A tail queue is headed by a pair of pointers, one to the head of the
  * list and the other to the tail of the list. The elements are doubly
@@ -17,65 +46,93 @@
  * after an existing element, at the head of the list, or at the end of
  * the list. A tail queue may be traversed in either direction.
  *
+ * For details on the use of these macros, see the queue(3) manual page.
+ *
  * Below is a summary of implemented functions where:
  *  +  means the macro is available
  *  -  means the macro is not available
  *  s  means the macro is available but is slow (runs in O(n) time)
  *
- *				TAILQ
- * _HEAD			+
- * _CLASS_HEAD			+
- * _HEAD_INITIALIZER		+
- * _ENTRY			+
- * _CLASS_ENTRY			+
- * _INIT			+
- * _EMPTY			+
- * _FIRST			+
- * _NEXT			+
- * _PREV			+
- * _LAST			+
- * _LAST_FAST			+
- * _FOREACH			+
- * _FOREACH_FROM		+
- * _FOREACH_SAFE		+
- * _FOREACH_FROM_SAFE		+
- * _FOREACH_REVERSE		+
- * _FOREACH_REVERSE_FROM	+
- * _FOREACH_REVERSE_SAFE	+
- * _FOREACH_REVERSE_FROM_SAFE	+
- * _INSERT_HEAD			+
- * _INSERT_BEFORE		+
- * _INSERT_AFTER		+
- * _INSERT_TAIL			+
- * _CONCAT			+
- * _REMOVE_AFTER		-
- * _REMOVE_HEAD			-
- * _REMOVE			+
- * _SWAP			+
+ *				SLIST	LIST	STAILQ	TAILQ
+ * _HEAD			+	+	+	+
+ * _CLASS_HEAD			+	+	+	+
+ * _HEAD_INITIALIZER		+	+	+	+
+ * _ENTRY			+	+	+	+
+ * _CLASS_ENTRY			+	+	+	+
+ * _INIT			+	+	+	+
+ * _EMPTY			+	+	+	+
+ * _FIRST			+	+	+	+
+ * _NEXT			+	+	+	+
+ * _PREV			-	+	-	+
+ * _LAST			-	-	+	+
+ * _LAST_FAST			-	-	-	+
+ * _FOREACH			+	+	+	+
+ * _FOREACH_FROM		+	+	+	+
+ * _FOREACH_SAFE		+	+	+	+
+ * _FOREACH_FROM_SAFE		+	+	+	+
+ * _FOREACH_REVERSE		-	-	-	+
+ * _FOREACH_REVERSE_FROM	-	-	-	+
+ * _FOREACH_REVERSE_SAFE	-	-	-	+
+ * _FOREACH_REVERSE_FROM_SAFE	-	-	-	+
+ * _INSERT_HEAD			+	+	+	+
+ * _INSERT_BEFORE		-	+	-	+
+ * _INSERT_AFTER		+	+	+	+
+ * _INSERT_TAIL			-	-	+	+
+ * _CONCAT			s	s	+	+
+ * _REMOVE_AFTER		+	-	+	-
+ * _REMOVE_HEAD			+	-	+	-
+ * _REMOVE			s	+	s	+
+ * _SWAP			+	+	+	+
  *
  */
-
-#ifdef __cplusplus
-extern "C" {
+#ifdef QUEUE_MACRO_DEBUG
+#warn Use QUEUE_MACRO_DEBUG_TRACE and/or QUEUE_MACRO_DEBUG_TRASH
+#define	QUEUE_MACRO_DEBUG_TRACE
+#define	QUEUE_MACRO_DEBUG_TRASH
 #endif
 
-/*
- * List definitions.
- */
-#define	LIST_HEAD(name, type)						\
-struct name {								\
-	struct type *lh_first;	/* first element */			\
-}
+#ifdef QUEUE_MACRO_DEBUG_TRACE
+/* Store the last 2 places the queue element or head was altered */
+struct qm_trace {
+	unsigned long	 lastline;
+	unsigned long	 prevline;
+	const char	*lastfile;
+	const char	*prevfile;
+};
+
+#define	TRACEBUF	struct qm_trace trace;
+#define	TRACEBUF_INITIALIZER	{ __LINE__, 0, __FILE__, NULL } ,
+
+#define	QMD_TRACE_HEAD(head) do {					\
+	(head)->trace.prevline = (head)->trace.lastline;		\
+	(head)->trace.prevfile = (head)->trace.lastfile;		\
+	(head)->trace.lastline = __LINE__;				\
+	(head)->trace.lastfile = __FILE__;				\
+} while (0)
 
+#define	QMD_TRACE_ELEM(elem) do {					\
+	(elem)->trace.prevline = (elem)->trace.lastline;		\
+	(elem)->trace.prevfile = (elem)->trace.lastfile;		\
+	(elem)->trace.lastline = __LINE__;				\
+	(elem)->trace.lastfile = __FILE__;				\
+} while (0)
+
+#else	/* !QUEUE_MACRO_DEBUG_TRACE */
 #define	QMD_TRACE_ELEM(elem)
 #define	QMD_TRACE_HEAD(head)
 #define	TRACEBUF
 #define	TRACEBUF_INITIALIZER
+#endif	/* QUEUE_MACRO_DEBUG_TRACE */
 
+#ifdef QUEUE_MACRO_DEBUG_TRASH
+#define	QMD_SAVELINK(name, link)	void **name = (void *)&(link)
+#define	TRASHIT(x)		do {(x) = (void *)-1;} while (0)
+#define	QMD_IS_TRASHED(x)	((x) == (void *)(intptr_t)-1)
+#else	/* !QUEUE_MACRO_DEBUG_TRASH */
+#define	QMD_SAVELINK(name, link)
 #define	TRASHIT(x)
 #define	QMD_IS_TRASHED(x)	0
-
-#define	QMD_SAVELINK(name, link)
+#endif	/* QUEUE_MACRO_DEBUG_TRASH */
 
 #ifdef __cplusplus
 /*
@@ -86,6 +143,445 @@ struct name {								\
 #define	QUEUE_TYPEOF(type) struct type
 #endif
 
+/*
+ * Singly-linked List declarations.
+ */
+#define	SLIST_HEAD(name, type)						\
+struct name {								\
+	struct type *slh_first;	/* first element */			\
+}
+
+#define	SLIST_CLASS_HEAD(name, type)					\
+struct name {								\
+	class type *slh_first;	/* first element */			\
+}
+
+#define	SLIST_HEAD_INITIALIZER(head)					\
+	{ NULL }
+
+#define	SLIST_ENTRY(type)						\
+struct {								\
+	struct type *sle_next;	/* next element */			\
+}
+
+#define	SLIST_CLASS_ENTRY(type)						\
+struct {								\
+	class type *sle_next;		/* next element */		\
+}
+
+/*
+ * Singly-linked List functions.
+ */
+#if (defined(_KERNEL) && defined(INVARIANTS))
+#define	QMD_SLIST_CHECK_PREVPTR(prevp, elm) do {			\
+	if (*(prevp) != (elm))						\
+		panic("Bad prevptr *(%p) == %p != %p",			\
+		    (prevp), *(prevp), (elm));				\
+} while (0)
+#else
+#define	QMD_SLIST_CHECK_PREVPTR(prevp, elm)
+#endif
+
+#define SLIST_CONCAT(head1, head2, type, field) do {			\
+	QUEUE_TYPEOF(type) *curelm = SLIST_FIRST(head1);		\
+	if (curelm == NULL) {						\
+		if ((SLIST_FIRST(head1) = SLIST_FIRST(head2)) != NULL)	\
+			SLIST_INIT(head2);				\
+	} else if (SLIST_FIRST(head2) != NULL) {			\
+		while (SLIST_NEXT(curelm, field) != NULL)		\
+			curelm = SLIST_NEXT(curelm, field);		\
+		SLIST_NEXT(curelm, field) = SLIST_FIRST(head2);		\
+		SLIST_INIT(head2);					\
+	}								\
+} while (0)
+
+#define	SLIST_EMPTY(head)	((head)->slh_first == NULL)
+
+#define	SLIST_FIRST(head)	((head)->slh_first)
+
+#define	SLIST_FOREACH(var, head, field)					\
+	for ((var) = SLIST_FIRST((head));				\
+	    (var);							\
+	    (var) = SLIST_NEXT((var), field))
+
+#define	SLIST_FOREACH_FROM(var, head, field)				\
+	for ((var) = ((var) ? (var) : SLIST_FIRST((head)));		\
+	    (var);							\
+	    (var) = SLIST_NEXT((var), field))
+
+#define	SLIST_FOREACH_SAFE(var, head, field, tvar)			\
+	for ((var) = SLIST_FIRST((head));				\
+	    (var) && ((tvar) = SLIST_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	SLIST_FOREACH_FROM_SAFE(var, head, field, tvar)			\
+	for ((var) = ((var) ? (var) : SLIST_FIRST((head)));		\
+	    (var) && ((tvar) = SLIST_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	SLIST_FOREACH_PREVPTR(var, varp, head, field)			\
+	for ((varp) = &SLIST_FIRST((head));				\
+	    ((var) = *(varp)) != NULL;					\
+	    (varp) = &SLIST_NEXT((var), field))
+
+#define	SLIST_INIT(head) do {						\
+	SLIST_FIRST((head)) = NULL;					\
+} while (0)
+
+#define	SLIST_INSERT_AFTER(slistelm, elm, field) do {			\
+	SLIST_NEXT((elm), field) = SLIST_NEXT((slistelm), field);	\
+	SLIST_NEXT((slistelm), field) = (elm);				\
+} while (0)
+
+#define	SLIST_INSERT_HEAD(head, elm, field) do {			\
+	SLIST_NEXT((elm), field) = SLIST_FIRST((head));			\
+	SLIST_FIRST((head)) = (elm);					\
+} while (0)
+
+#define	SLIST_NEXT(elm, field)	((elm)->field.sle_next)
+
+#define	SLIST_REMOVE(head, elm, type, field) do {			\
+	QMD_SAVELINK(oldnext, (elm)->field.sle_next);			\
+	if (SLIST_FIRST((head)) == (elm)) {				\
+		SLIST_REMOVE_HEAD((head), field);			\
+	}								\
+	else {								\
+		QUEUE_TYPEOF(type) *curelm = SLIST_FIRST(head);		\
+		while (SLIST_NEXT(curelm, field) != (elm))		\
+			curelm = SLIST_NEXT(curelm, field);		\
+		SLIST_REMOVE_AFTER(curelm, field);			\
+	}								\
+	TRASHIT(*oldnext);						\
+} while (0)
+
+#define SLIST_REMOVE_AFTER(elm, field) do {				\
+	SLIST_NEXT(elm, field) =					\
+	    SLIST_NEXT(SLIST_NEXT(elm, field), field);			\
+} while (0)
+
+#define	SLIST_REMOVE_HEAD(head, field) do {				\
+	SLIST_FIRST((head)) = SLIST_NEXT(SLIST_FIRST((head)), field);	\
+} while (0)
+
+#define	SLIST_REMOVE_PREVPTR(prevp, elm, field) do {			\
+	QMD_SLIST_CHECK_PREVPTR(prevp, elm);				\
+	*(prevp) = SLIST_NEXT(elm, field);				\
+	TRASHIT((elm)->field.sle_next);					\
+} while (0)
+
+#define SLIST_SWAP(head1, head2, type) do {				\
+	QUEUE_TYPEOF(type) *swap_first = SLIST_FIRST(head1);		\
+	SLIST_FIRST(head1) = SLIST_FIRST(head2);			\
+	SLIST_FIRST(head2) = swap_first;				\
+} while (0)
+
+/*
+ * Singly-linked Tail queue declarations.
+ */
+#define	STAILQ_HEAD(name, type)						\
+struct name {								\
+	struct type *stqh_first;/* first element */			\
+	struct type **stqh_last;/* addr of last next element */		\
+}
+
+#define	STAILQ_CLASS_HEAD(name, type)					\
+struct name {								\
+	class type *stqh_first;	/* first element */			\
+	class type **stqh_last;	/* addr of last next element */		\
+}
+
+#define	STAILQ_HEAD_INITIALIZER(head)					\
+	{ NULL, &(head).stqh_first }
+
+#define	STAILQ_ENTRY(type)						\
+struct {								\
+	struct type *stqe_next;	/* next element */			\
+}
+
+#define	STAILQ_CLASS_ENTRY(type)					\
+struct {								\
+	class type *stqe_next;	/* next element */			\
+}
+
+/*
+ * Singly-linked Tail queue functions.
+ */
+#define	STAILQ_CONCAT(head1, head2) do {				\
+	if (!STAILQ_EMPTY((head2))) {					\
+		*(head1)->stqh_last = (head2)->stqh_first;		\
+		(head1)->stqh_last = (head2)->stqh_last;		\
+		STAILQ_INIT((head2));					\
+	}								\
+} while (0)
+
+#define	STAILQ_EMPTY(head)	((head)->stqh_first == NULL)
+
+#define	STAILQ_FIRST(head)	((head)->stqh_first)
+
+#define	STAILQ_FOREACH(var, head, field)				\
+	for((var) = STAILQ_FIRST((head));				\
+	   (var);							\
+	   (var) = STAILQ_NEXT((var), field))
+
+#define	STAILQ_FOREACH_FROM(var, head, field)				\
+	for ((var) = ((var) ? (var) : STAILQ_FIRST((head)));		\
+	   (var);							\
+	   (var) = STAILQ_NEXT((var), field))
+
+#define	STAILQ_FOREACH_SAFE(var, head, field, tvar)			\
+	for ((var) = STAILQ_FIRST((head));				\
+	    (var) && ((tvar) = STAILQ_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	STAILQ_FOREACH_FROM_SAFE(var, head, field, tvar)		\
+	for ((var) = ((var) ? (var) : STAILQ_FIRST((head)));		\
+	    (var) && ((tvar) = STAILQ_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	STAILQ_INIT(head) do {						\
+	STAILQ_FIRST((head)) = NULL;					\
+	(head)->stqh_last = &STAILQ_FIRST((head));			\
+} while (0)
+
+#define	STAILQ_INSERT_AFTER(head, tqelm, elm, field) do {		\
+	if ((STAILQ_NEXT((elm), field) = STAILQ_NEXT((tqelm), field)) == NULL)\
+		(head)->stqh_last = &STAILQ_NEXT((elm), field);		\
+	STAILQ_NEXT((tqelm), field) = (elm);				\
+} while (0)
+
+#define	STAILQ_INSERT_HEAD(head, elm, field) do {			\
+	if ((STAILQ_NEXT((elm), field) = STAILQ_FIRST((head))) == NULL)	\
+		(head)->stqh_last = &STAILQ_NEXT((elm), field);		\
+	STAILQ_FIRST((head)) = (elm);					\
+} while (0)
+
+#define	STAILQ_INSERT_TAIL(head, elm, field) do {			\
+	STAILQ_NEXT((elm), field) = NULL;				\
+	*(head)->stqh_last = (elm);					\
+	(head)->stqh_last = &STAILQ_NEXT((elm), field);			\
+} while (0)
+
+#define	STAILQ_LAST(head, type, field)				\
+	(STAILQ_EMPTY((head)) ? NULL :				\
+	    __containerof((head)->stqh_last,			\
+	    QUEUE_TYPEOF(type), field.stqe_next))
+
+#define	STAILQ_NEXT(elm, field)	((elm)->field.stqe_next)
+
+#define	STAILQ_REMOVE(head, elm, type, field) do {			\
+	QMD_SAVELINK(oldnext, (elm)->field.stqe_next);			\
+	if (STAILQ_FIRST((head)) == (elm)) {				\
+		STAILQ_REMOVE_HEAD((head), field);			\
+	}								\
+	else {								\
+		QUEUE_TYPEOF(type) *curelm = STAILQ_FIRST(head);	\
+		while (STAILQ_NEXT(curelm, field) != (elm))		\
+			curelm = STAILQ_NEXT(curelm, field);		\
+		STAILQ_REMOVE_AFTER(head, curelm, field);		\
+	}								\
+	TRASHIT(*oldnext);						\
+} while (0)
+
+#define STAILQ_REMOVE_AFTER(head, elm, field) do {			\
+	if ((STAILQ_NEXT(elm, field) =					\
+	     STAILQ_NEXT(STAILQ_NEXT(elm, field), field)) == NULL)	\
+		(head)->stqh_last = &STAILQ_NEXT((elm), field);		\
+} while (0)
+
+#define	STAILQ_REMOVE_HEAD(head, field) do {				\
+	if ((STAILQ_FIRST((head)) =					\
+	     STAILQ_NEXT(STAILQ_FIRST((head)), field)) == NULL)		\
+		(head)->stqh_last = &STAILQ_FIRST((head));		\
+} while (0)
+
+#define STAILQ_SWAP(head1, head2, type) do {				\
+	QUEUE_TYPEOF(type) *swap_first = STAILQ_FIRST(head1);		\
+	QUEUE_TYPEOF(type) **swap_last = (head1)->stqh_last;		\
+	STAILQ_FIRST(head1) = STAILQ_FIRST(head2);			\
+	(head1)->stqh_last = (head2)->stqh_last;			\
+	STAILQ_FIRST(head2) = swap_first;				\
+	(head2)->stqh_last = swap_last;					\
+	if (STAILQ_EMPTY(head1))					\
+		(head1)->stqh_last = &STAILQ_FIRST(head1);		\
+	if (STAILQ_EMPTY(head2))					\
+		(head2)->stqh_last = &STAILQ_FIRST(head2);		\
+} while (0)
+
+
+/*
+ * List declarations.
+ */
+#define	LIST_HEAD(name, type)						\
+struct name {								\
+	struct type *lh_first;	/* first element */			\
+}
+
+#define	LIST_CLASS_HEAD(name, type)					\
+struct name {								\
+	class type *lh_first;	/* first element */			\
+}
+
+#define	LIST_HEAD_INITIALIZER(head)					\
+	{ NULL }
+
+#define	LIST_ENTRY(type)						\
+struct {								\
+	struct type *le_next;	/* next element */			\
+	struct type **le_prev;	/* address of previous next element */	\
+}
+
+#define	LIST_CLASS_ENTRY(type)						\
+struct {								\
+	class type *le_next;	/* next element */			\
+	class type **le_prev;	/* address of previous next element */	\
+}
+
+/*
+ * List functions.
+ */
+
+#if (defined(_KERNEL) && defined(INVARIANTS))
+/*
+ * QMD_LIST_CHECK_HEAD(LIST_HEAD *head, LIST_ENTRY NAME)
+ *
+ * If the list is non-empty, validates that the first element of the list
+ * points back at 'head.'
+ */
+#define	QMD_LIST_CHECK_HEAD(head, field) do {				\
+	if (LIST_FIRST((head)) != NULL &&				\
+	    LIST_FIRST((head))->field.le_prev !=			\
+	     &LIST_FIRST((head)))					\
+		panic("Bad list head %p first->prev != head", (head));	\
+} while (0)
+
+/*
+ * QMD_LIST_CHECK_NEXT(TYPE *elm, LIST_ENTRY NAME)
+ *
+ * If an element follows 'elm' in the list, validates that the next element
+ * points back at 'elm.'
+ */
+#define	QMD_LIST_CHECK_NEXT(elm, field) do {				\
+	if (LIST_NEXT((elm), field) != NULL &&				\
+	    LIST_NEXT((elm), field)->field.le_prev !=			\
+	     &((elm)->field.le_next))					\
+	     	panic("Bad link elm %p next->prev != elm", (elm));	\
+} while (0)
+
+/*
+ * QMD_LIST_CHECK_PREV(TYPE *elm, LIST_ENTRY NAME)
+ *
+ * Validates that the previous element (or head of the list) points to 'elm.'
+ */
+#define	QMD_LIST_CHECK_PREV(elm, field) do {				\
+	if (*(elm)->field.le_prev != (elm))				\
+		panic("Bad link elm %p prev->next != elm", (elm));	\
+} while (0)
+#else
+#define	QMD_LIST_CHECK_HEAD(head, field)
+#define	QMD_LIST_CHECK_NEXT(elm, field)
+#define	QMD_LIST_CHECK_PREV(elm, field)
+#endif /* (_KERNEL && INVARIANTS) */
+
+#define LIST_CONCAT(head1, head2, type, field) do {			      \
+	QUEUE_TYPEOF(type) *curelm = LIST_FIRST(head1);			      \
+	if (curelm == NULL) {						      \
+		if ((LIST_FIRST(head1) = LIST_FIRST(head2)) != NULL) {	      \
+			LIST_FIRST(head2)->field.le_prev =		      \
+			    &LIST_FIRST((head1));			      \
+			LIST_INIT(head2);				      \
+		}							      \
+	} else if (LIST_FIRST(head2) != NULL) {				      \
+		while (LIST_NEXT(curelm, field) != NULL)		      \
+			curelm = LIST_NEXT(curelm, field);		      \
+		LIST_NEXT(curelm, field) = LIST_FIRST(head2);		      \
+		LIST_FIRST(head2)->field.le_prev = &LIST_NEXT(curelm, field); \
+		LIST_INIT(head2);					      \
+	}								      \
+} while (0)
+
+#define	LIST_EMPTY(head)	((head)->lh_first == NULL)
+
+#define	LIST_FIRST(head)	((head)->lh_first)
+
+#define	LIST_FOREACH(var, head, field)					\
+	for ((var) = LIST_FIRST((head));				\
+	    (var);							\
+	    (var) = LIST_NEXT((var), field))
+
+#define	LIST_FOREACH_FROM(var, head, field)				\
+	for ((var) = ((var) ? (var) : LIST_FIRST((head)));		\
+	    (var);							\
+	    (var) = LIST_NEXT((var), field))
+
+#define	LIST_FOREACH_SAFE(var, head, field, tvar)			\
+	for ((var) = LIST_FIRST((head));				\
+	    (var) && ((tvar) = LIST_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	LIST_FOREACH_FROM_SAFE(var, head, field, tvar)			\
+	for ((var) = ((var) ? (var) : LIST_FIRST((head)));		\
+	    (var) && ((tvar) = LIST_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	LIST_INIT(head) do {						\
+	LIST_FIRST((head)) = NULL;					\
+} while (0)
+
+#define	LIST_INSERT_AFTER(listelm, elm, field) do {			\
+	QMD_LIST_CHECK_NEXT(listelm, field);				\
+	if ((LIST_NEXT((elm), field) = LIST_NEXT((listelm), field)) != NULL)\
+		LIST_NEXT((listelm), field)->field.le_prev =		\
+		    &LIST_NEXT((elm), field);				\
+	LIST_NEXT((listelm), field) = (elm);				\
+	(elm)->field.le_prev = &LIST_NEXT((listelm), field);		\
+} while (0)
+
+#define	LIST_INSERT_BEFORE(listelm, elm, field) do {			\
+	QMD_LIST_CHECK_PREV(listelm, field);				\
+	(elm)->field.le_prev = (listelm)->field.le_prev;		\
+	LIST_NEXT((elm), field) = (listelm);				\
+	*(listelm)->field.le_prev = (elm);				\
+	(listelm)->field.le_prev = &LIST_NEXT((elm), field);		\
+} while (0)
+
+#define	LIST_INSERT_HEAD(head, elm, field) do {				\
+	QMD_LIST_CHECK_HEAD((head), field);				\
+	if ((LIST_NEXT((elm), field) = LIST_FIRST((head))) != NULL)	\
+		LIST_FIRST((head))->field.le_prev = &LIST_NEXT((elm), field);\
+	LIST_FIRST((head)) = (elm);					\
+	(elm)->field.le_prev = &LIST_FIRST((head));			\
+} while (0)
+
+#define	LIST_NEXT(elm, field)	((elm)->field.le_next)
+
+#define	LIST_PREV(elm, head, type, field)			\
+	((elm)->field.le_prev == &LIST_FIRST((head)) ? NULL :	\
+	    __containerof((elm)->field.le_prev,			\
+	    QUEUE_TYPEOF(type), field.le_next))
+
+#define	LIST_REMOVE(elm, field) do {					\
+	QMD_SAVELINK(oldnext, (elm)->field.le_next);			\
+	QMD_SAVELINK(oldprev, (elm)->field.le_prev);			\
+	QMD_LIST_CHECK_NEXT(elm, field);				\
+	QMD_LIST_CHECK_PREV(elm, field);				\
+	if (LIST_NEXT((elm), field) != NULL)				\
+		LIST_NEXT((elm), field)->field.le_prev = 		\
+		    (elm)->field.le_prev;				\
+	*(elm)->field.le_prev = LIST_NEXT((elm), field);		\
+	TRASHIT(*oldnext);						\
+	TRASHIT(*oldprev);						\
+} while (0)
+
+#define LIST_SWAP(head1, head2, type, field) do {			\
+	QUEUE_TYPEOF(type) *swap_tmp = LIST_FIRST(head1);		\
+	LIST_FIRST((head1)) = LIST_FIRST((head2));			\
+	LIST_FIRST((head2)) = swap_tmp;					\
+	if ((swap_tmp = LIST_FIRST((head1))) != NULL)			\
+		swap_tmp->field.le_prev = &LIST_FIRST((head1));		\
+	if ((swap_tmp = LIST_FIRST((head2))) != NULL)			\
+		swap_tmp->field.le_prev = &LIST_FIRST((head2));		\
+} while (0)
+
 /*
  * Tail queue declarations.
  */
@@ -123,10 +619,58 @@ struct {								\
 /*
  * Tail queue functions.
  */
+#if (defined(_KERNEL) && defined(INVARIANTS))
+/*
+ * QMD_TAILQ_CHECK_HEAD(TAILQ_HEAD *head, TAILQ_ENTRY NAME)
+ *
+ * If the tailq is non-empty, validates that the first element of the tailq
+ * points back at 'head.'
+ */
+#define	QMD_TAILQ_CHECK_HEAD(head, field) do {				\
+	if (!TAILQ_EMPTY(head) &&					\
+	    TAILQ_FIRST((head))->field.tqe_prev !=			\
+	     &TAILQ_FIRST((head)))					\
+		panic("Bad tailq head %p first->prev != head", (head));	\
+} while (0)
+
+/*
+ * QMD_TAILQ_CHECK_TAIL(TAILQ_HEAD *head, TAILQ_ENTRY NAME)
+ *
+ * Validates that the tail of the tailq is a pointer to pointer to NULL.
+ */
+#define	QMD_TAILQ_CHECK_TAIL(head, field) do {				\
+	if (*(head)->tqh_last != NULL)					\
+	    	panic("Bad tailq NEXT(%p->tqh_last) != NULL", (head)); 	\
+} while (0)
+
+/*
+ * QMD_TAILQ_CHECK_NEXT(TYPE *elm, TAILQ_ENTRY NAME)
+ *
+ * If an element follows 'elm' in the tailq, validates that the next element
+ * points back at 'elm.'
+ */
+#define	QMD_TAILQ_CHECK_NEXT(elm, field) do {				\
+	if (TAILQ_NEXT((elm), field) != NULL &&				\
+	    TAILQ_NEXT((elm), field)->field.tqe_prev !=			\
+	     &((elm)->field.tqe_next))					\
+		panic("Bad link elm %p next->prev != elm", (elm));	\
+} while (0)
+
+/*
+ * QMD_TAILQ_CHECK_PREV(TYPE *elm, TAILQ_ENTRY NAME)
+ *
+ * Validates that the previous element (or head of the tailq) points to 'elm.'
+ */
+#define	QMD_TAILQ_CHECK_PREV(elm, field) do {				\
+	if (*(elm)->field.tqe_prev != (elm))				\
+		panic("Bad link elm %p prev->next != elm", (elm));	\
+} while (0)
+#else
 #define	QMD_TAILQ_CHECK_HEAD(head, field)
 #define	QMD_TAILQ_CHECK_TAIL(head, headname)
 #define	QMD_TAILQ_CHECK_NEXT(elm, field)
 #define	QMD_TAILQ_CHECK_PREV(elm, field)
+#endif /* (_KERNEL && INVARIANTS) */
 
 #define	TAILQ_CONCAT(head1, head2, field) do {				\
 	if (!TAILQ_EMPTY(head2)) {					\
@@ -191,9 +735,8 @@ struct {								\
 
 #define	TAILQ_INSERT_AFTER(head, listelm, elm, field) do {		\
 	QMD_TAILQ_CHECK_NEXT(listelm, field);				\
-	TAILQ_NEXT((elm), field) = TAILQ_NEXT((listelm), field);	\
-	if (TAILQ_NEXT((listelm), field) != NULL)			\
-		TAILQ_NEXT((elm), field)->field.tqe_prev =		\
+	if ((TAILQ_NEXT((elm), field) = TAILQ_NEXT((listelm), field)) != NULL)\
+		TAILQ_NEXT((elm), field)->field.tqe_prev = 		\
 		    &TAILQ_NEXT((elm), field);				\
 	else {								\
 		(head)->tqh_last = &TAILQ_NEXT((elm), field);		\
@@ -217,8 +760,7 @@ struct {								\
 
 #define	TAILQ_INSERT_HEAD(head, elm, field) do {			\
 	QMD_TAILQ_CHECK_HEAD(head, field);				\
-	TAILQ_NEXT((elm), field) = TAILQ_FIRST((head));			\
-	if (TAILQ_FIRST((head)) != NULL)				\
+	if ((TAILQ_NEXT((elm), field) = TAILQ_FIRST((head))) != NULL)	\
 		TAILQ_FIRST((head))->field.tqe_prev =			\
 		    &TAILQ_NEXT((elm), field);				\
 	else								\
@@ -250,21 +792,24 @@ struct {								\
  * you may want to prefetch the last data element.
  */
 #define	TAILQ_LAST_FAST(head, type, field)			\
-	(TAILQ_EMPTY(head) ? NULL : __containerof((head)->tqh_last,	\
-	QUEUE_TYPEOF(type), field.tqe_next))
+    (TAILQ_EMPTY(head) ? NULL : __containerof((head)->tqh_last, QUEUE_TYPEOF(type), field.tqe_next))
 
 #define	TAILQ_NEXT(elm, field) ((elm)->field.tqe_next)
 
 #define	TAILQ_PREV(elm, headname, field)				\
 	(*(((struct headname *)((elm)->field.tqe_prev))->tqh_last))
 
+#define	TAILQ_PREV_FAST(elm, head, type, field)				\
+    ((elm)->field.tqe_prev == &(head)->tqh_first ? NULL :		\
+     __containerof((elm)->field.tqe_prev, QUEUE_TYPEOF(type), field.tqe_next))
+
 #define	TAILQ_REMOVE(head, elm, field) do {				\
 	QMD_SAVELINK(oldnext, (elm)->field.tqe_next);			\
 	QMD_SAVELINK(oldprev, (elm)->field.tqe_prev);			\
 	QMD_TAILQ_CHECK_NEXT(elm, field);				\
 	QMD_TAILQ_CHECK_PREV(elm, field);				\
 	if ((TAILQ_NEXT((elm), field)) != NULL)				\
-		TAILQ_NEXT((elm), field)->field.tqe_prev =		\
+		TAILQ_NEXT((elm), field)->field.tqe_prev = 		\
 		    (elm)->field.tqe_prev;				\
 	else {								\
 		(head)->tqh_last = (elm)->field.tqe_prev;		\
@@ -277,26 +822,20 @@ struct {								\
 } while (0)
 
 #define TAILQ_SWAP(head1, head2, type, field) do {			\
-	QUEUE_TYPEOF(type) * swap_first = (head1)->tqh_first;		\
-	QUEUE_TYPEOF(type) * *swap_last = (head1)->tqh_last;		\
+	QUEUE_TYPEOF(type) *swap_first = (head1)->tqh_first;		\
+	QUEUE_TYPEOF(type) **swap_last = (head1)->tqh_last;		\
 	(head1)->tqh_first = (head2)->tqh_first;			\
 	(head1)->tqh_last = (head2)->tqh_last;				\
 	(head2)->tqh_first = swap_first;				\
 	(head2)->tqh_last = swap_last;					\
-	swap_first = (head1)->tqh_first;				\
-	if (swap_first != NULL)						\
+	if ((swap_first = (head1)->tqh_first) != NULL)			\
 		swap_first->field.tqe_prev = &(head1)->tqh_first;	\
 	else								\
 		(head1)->tqh_last = &(head1)->tqh_first;		\
-	swap_first = (head2)->tqh_first;				\
-	if (swap_first != NULL)			\
+	if ((swap_first = (head2)->tqh_first) != NULL)			\
 		swap_first->field.tqe_prev = &(head2)->tqh_first;	\
 	else								\
 		(head2)->tqh_last = &(head2)->tqh_first;		\
 } while (0)
 
-#ifdef __cplusplus
-}
-#endif
-
-#endif /* _SYS_QUEUE_H_ */
+#endif /* !_SYS_QUEUE_H_ */
Signature · 1 line
2.25.1
#30 lore raw

PATCH v2 10/10 eal/windows: implement basic memory management

In reply to Dmitry Kozlyuk

Basic memory management supports core libraries and PMDs operating in IOVA as PA mode. It uses a kernel-mode driver, virt2phys, to obtain IOVAs of hugepages allocated from user-mode.

---
 config/meson.build                            |   2 +-
 doc/guides/windows_gsg/run_apps.rst           |  30 +
 lib/librte_eal/common/eal_common_fbarray.c    |  57 +-
 lib/librte_eal/common/eal_common_memory.c     |  50 +-
 lib/librte_eal/common/eal_private.h           |   6 +-
 lib/librte_eal/common/malloc_heap.c           |   1 +
 lib/librte_eal/common/meson.build             |   9 +
 lib/librte_eal/freebsd/eal_memory.c           |   1 -
 lib/librte_eal/meson.build                    |   1 +
 lib/librte_eal/rte_eal_exports.def            | 119 ++-
 lib/librte_eal/windows/eal.c                  |  55 ++
 lib/librte_eal/windows/eal_memalloc.c         | 423 +++++++++++
 lib/librte_eal/windows/eal_memory.c           | 702 +++++++++++++++++-
 lib/librte_eal/windows/eal_mp.c               | 103 +++
 lib/librte_eal/windows/eal_windows.h          |  23 +
 lib/librte_eal/windows/include/meson.build    |   1 +
 lib/librte_eal/windows/include/rte_os.h       |   4 +
 .../windows/include/rte_virt2phys.h           |  34 +
 lib/librte_eal/windows/include/rte_windows.h  |   2 +
 lib/librte_eal/windows/include/unistd.h       |   3 +
 lib/librte_eal/windows/meson.build            |   2 +
 21 files changed, 1561 insertions(+), 67 deletions(-)
 create mode 100644 lib/librte_eal/windows/eal_memalloc.c
 create mode 100644 lib/librte_eal/windows/eal_mp.c
 create mode 100644 lib/librte_eal/windows/include/rte_virt2phys.h
config/meson.build +1 -1
index bceb5ef7b..7b8baa788 100644
@@ -270,7 +270,7 @@ if is_windows
 		add_project_link_arguments('-lmincore', language: 'c')
 	endif
 
-	add_project_link_arguments('-ladvapi32', language: 'c')
+	add_project_link_arguments('-ladvapi32', '-lsetupapi', language: 'c')
 endif
 
 if get_option('b_lto')
doc/guides/windows_gsg/run_apps.rst +30 -0
index 21ac7f6c1..e858cf8c1 100644
@@ -27,6 +27,36 @@ See `Large-Page Support`_ in MSDN for details.
 .. _Large-page Support: https://docs.microsoft.com/en-us/windows/win32/memory/large-page-support
 
 
+Load virt2phys Driver
+---------------------
+
+Access to physical addresses is provided by a kernel-mode driver, virt2phys.
+It is mandatory at least for using hardware PMDs, but may also be required
+for mempools.
+
+This driver is not signed, so signature checking must be disabled to load it.
+Refer to documentation in ``dpdk-kmods`` repository for details on system
+setup, driver build and installation.
+
+Compiled package, consisting of ``virt2phys.inf``, ``virt2phys.cat``,
+and ``virt2phys.sys``, is installed as follows (from Elevated Command Line):
+
+.. code-block:: console
+
+    pnputil /add-driver virt2phys.inf /install
+
+When loaded successfully, the driver is shown in *Device Manager* as *Virtual
+to physical address translator* device under *Kernel bypass* category.
+
+If DPDK is unable to communicate with the driver, a warning is printed
+on initialization (debug-level logs provide more details):
+
+.. code-block:: text
+
+    EAL: Cannot open virt2phys driver interface
+
+
+
 Run the ``helloworld`` Example
 ------------------------------
 
lib/librte_eal/common/eal_common_fbarray.c +26 -31

168 lines not on this page · raw message

lib/librte_eal/common/eal_common_memory.c +22 -28

118 lines not on this page · raw message

lib/librte_eal/common/eal_private.h +3 -3
index 76938e379..59ac41916 100644
@@ -226,8 +226,8 @@ enum eal_mem_reserve_flags {
  *   Page size on which to align requested virtual area.
  * @param flags
  *   EAL_VIRTUAL_AREA_* flags.
- * @param mmap_flags
- *   Extra flags passed directly to mmap().
+ * @param reserve_flags
+ *   Extra flags passed directly to rte_mem_reserve().
  *
  * @return
  *   Virtual area address if successful.
@@ -244,7 +244,7 @@ enum eal_mem_reserve_flags {
 /**< immediately unmap reserved virtual area. */
 void *
 eal_get_virtual_area(void *requested_addr, size_t *size, size_t page_sz,
-	int flags, int mmap_flags);
+	int flags, enum eal_mem_reserve_flags reserve_flags);
 
 /**
  * Reserve VA space for a memory segment list.
lib/librte_eal/common/malloc_heap.c +1 -0
index 842eb9de7..6534c895c 100644
@@ -729,6 +729,7 @@ malloc_heap_alloc(const char *type, size_t size, int socket_arg,
 		if (ret != NULL)
 			return ret;
 	}
+
 	return NULL;
 }
 
lib/librte_eal/common/meson.build +9 -0
index 02d9280cc..6dcdcc890 100644
@@ -9,11 +9,20 @@ if is_windows
 		'eal_common_class.c',
 		'eal_common_devargs.c',
 		'eal_common_errno.c',
+		'eal_common_fbarray.c',
 		'eal_common_launch.c',
 		'eal_common_lcore.c',
 		'eal_common_log.c',
+		'eal_common_mcfg.c',
+		'eal_common_memalloc.c',
+		'eal_common_memory.c',
+		'eal_common_memzone.c',
 		'eal_common_options.c',
+		'eal_common_tailqs.c',
 		'eal_common_thread.c',
+		'malloc_elem.c',
+		'malloc_heap.c',
+		'rte_malloc.c',
 		'rte_option.c',
 	)
 	subdir_done()
lib/librte_eal/freebsd/eal_memory.c +0 -1
index 5174f9cd0..99bf6ec9e 100644
@@ -355,7 +355,6 @@ memseg_list_reserve(struct rte_memseg_list *msl)
 	return eal_reserve_memseg_list(msl, flags);
 }
 
-
 static int
 memseg_primary_init(void)
 {
lib/librte_eal/meson.build +1 -0
index 1f89efb88..1d750f003 100644
@@ -11,6 +11,7 @@ if not is_windows
 endif
 
 dpdk_conf.set('RTE_EXEC_ENV_' + exec_env.to_upper(), 1)
+dpdk_conf.set10('RTE_EAL_NUMA_AWARE_HUGEPAGES', true)
 subdir(exec_env)
 
 subdir(arch_subdir)
lib/librte_eal/rte_eal_exports.def +117 -2

130 lines not on this page · raw message

lib/librte_eal/windows/eal.c +55 -0

73 lines not on this page · raw message

lib/librte_eal/windows/eal_memalloc.c new file +423 -0

423 lines not on this page · raw message

lib/librte_eal/windows/eal_memory.c +701 -1

734 lines not on this page · raw message

lib/librte_eal/windows/eal_mp.c new file +103 -0

103 lines not on this page · raw message

lib/librte_eal/windows/eal_windows.h +23 -0
index b202a1aa5..083ab8b93 100644
@@ -9,8 +9,24 @@
  * @file Facilities private to Windows EAL
  */
 
+#include <rte_errno.h>
 #include <rte_windows.h>
 
+/**
+ * Log current function as not implemented and set rte_errno.
+ */
+#define EAL_LOG_NOT_IMPLEMENTED() \
+	do { \
+		RTE_LOG(DEBUG, EAL, "%s() is not implemented\n", __func__); \
+		rte_errno = ENOTSUP; \
+	} while (0)
+
+/**
+ * Log current function as a stub.
+ */
+#define EAL_LOG_STUB() \
+	RTE_LOG(DEBUG, EAL, "Windows: %s() is a stub\n", __func__)
+
 /**
  * Create a map of processors and cores on the system.
  */
@@ -36,6 +52,13 @@ int eal_thread_create(pthread_t *thread);
  */
 unsigned int eal_socket_numa_node(unsigned int socket_id);
 
+/**
+ * Open virt2phys driver interface device.
+ *
+ * @return 0 on success, (-1) on failure.
+ */
+int eal_mem_virt2iova_init(void);
+
 /**
  * Locate Win32 memory management routines in system libraries.
  *
lib/librte_eal/windows/include/meson.build +1 -0
index 5fb1962ac..b3534b025 100644
@@ -5,5 +5,6 @@ includes += include_directories('.')
 
 headers += files(
         'rte_os.h',
+        'rte_virt2phys.h',
         'rte_windows.h',
 )
lib/librte_eal/windows/include/rte_os.h +4 -0
index 510e39e03..62805a307 100644
@@ -36,6 +36,10 @@ extern "C" {
 
 #define strncasecmp(s1, s2, count)        _strnicmp(s1, s2, count)
 
+#define open _open
+#define close _close
+#define unlink _unlink
+
 /* cpu_set macros implementation */
 #define RTE_CPU_AND(dst, src1, src2) CPU_AND(dst, src1, src2)
 #define RTE_CPU_OR(dst, src1, src2) CPU_OR(dst, src1, src2)
lib/librte_eal/windows/include/rte_virt2phys.h new file +34 -0
new file mode 100644
index 000000000..4bb2b4aaf
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2020 Dmitry Kozlyuk
+ */
+
+/**
+ * @file virt2phys driver interface
+ */
+
+/**
+ * Driver device interface GUID {539c2135-793a-4926-afec-d3a1b61bbc8a}.
+ */
+DEFINE_GUID(GUID_DEVINTERFACE_VIRT2PHYS,
+	0x539c2135, 0x793a, 0x4926,
+	0xaf, 0xec, 0xd3, 0xa1, 0xb6, 0x1b, 0xbc, 0x8a);
+
+/**
+ * Driver device type for IO control codes.
+ */
+#define VIRT2PHYS_DEVTYPE 0x8000
+
+/**
+ * Translate a valid non-paged virtual address to a physical address.
+ *
+ * Note: A physical address zero (0) is reported if input address
+ * is paged out or not mapped. However, if input is a valid mapping
+ * of I/O port 0x0000, output is also zero. There is no way
+ * to distinguish between these cases by return value only.
+ *
+ * Input: a non-paged virtual address (PVOID).
+ *
+ * Output: the corresponding physical address (LARGE_INTEGER).
+ */
+#define IOCTL_VIRT2PHYS_TRANSLATE CTL_CODE( \
+	VIRT2PHYS_DEVTYPE, 0x800, METHOD_BUFFERED, FILE_ANY_ACCESS)
lib/librte_eal/windows/include/rte_windows.h +2 -0
index ed6e4c148..899ed7d87 100644
@@ -23,6 +23,8 @@
 
 #include <basetsd.h>
 #include <psapi.h>
+#include <setupapi.h>
+#include <winioctl.h>
 
 /* Have GUIDs defined. */
 #ifndef INITGUID
lib/librte_eal/windows/include/unistd.h +3 -0
index 757b7f3c5..6b33005b2 100644
@@ -9,4 +9,7 @@
  * as Microsoft libc does not contain unistd.h. This may be removed
  * in future releases.
  */
+
+#include <io.h>
+
 #endif /* _UNISTD_H_ */
lib/librte_eal/windows/meson.build +2 -0
index 81d3ee095..2f4fa91a9 100644
@@ -8,7 +8,9 @@ sources += files(
 	'eal_debug.c',
 	'eal_hugepages.c',
 	'eal_lcore.c',
+	'eal_memalloc.c',
 	'eal_memory.c',
+	'eal_mp.c',
 	'eal_thread.c',
 	'getopt.c',
 )
Signature · 1 line
2.25.1
#31 lore raw

Re: [dpdk-dev] [PATCH v2 10/10] eal/windows: implement basic memory management

Quoted 767 lines · Dmitry Kozlyuk

Basic memory management supports core libraries and PMDs operating in IOVA as PA mode. It uses a kernel-mode driver, virt2phys, to obtain IOVAs of hugepages allocated from user-mode.

---
 config/meson.build                            |   2 +-
 doc/guides/windows_gsg/run_apps.rst           |  30 +
 lib/librte_eal/common/eal_common_fbarray.c    |  57 +-
 lib/librte_eal/common/eal_common_memory.c     |  50 +-
 lib/librte_eal/common/eal_private.h           |   6 +-
 lib/librte_eal/common/malloc_heap.c           |   1 +
 lib/librte_eal/common/meson.build             |   9 +
 lib/librte_eal/freebsd/eal_memory.c           |   1 -
 lib/librte_eal/meson.build                    |   1 +
 lib/librte_eal/rte_eal_exports.def            | 119 ++-
 lib/librte_eal/windows/eal.c                  |  55 ++
 lib/librte_eal/windows/eal_memalloc.c         | 423 +++++++++++
 lib/librte_eal/windows/eal_memory.c           | 702 +++++++++++++++++-
 lib/librte_eal/windows/eal_mp.c               | 103 +++
 lib/librte_eal/windows/eal_windows.h          |  23 +
 lib/librte_eal/windows/include/meson.build    |   1 +
 lib/librte_eal/windows/include/rte_os.h       |   4 +
 .../windows/include/rte_virt2phys.h           |  34 +
 lib/librte_eal/windows/include/rte_windows.h  |   2 +
 lib/librte_eal/windows/include/unistd.h       |   3 +
 lib/librte_eal/windows/meson.build            |   2 +
 21 files changed, 1561 insertions(+), 67 deletions(-)
 create mode 100644 lib/librte_eal/windows/eal_memalloc.c
 create mode 100644 lib/librte_eal/windows/eal_mp.c
 create mode 100644 lib/librte_eal/windows/include/rte_virt2phys.h
config/meson.build +1 -1
index bceb5ef7b..7b8baa788 100644
@@ -270,7 +270,7 @@ if is_windows
 		add_project_link_arguments('-lmincore', language: 'c')
 	endif
 
-	add_project_link_arguments('-ladvapi32', language: 'c')
+	add_project_link_arguments('-ladvapi32', '-lsetupapi', language: 'c')
 endif
 
 if get_option('b_lto')
doc/guides/windows_gsg/run_apps.rst +30 -0
index 21ac7f6c1..e858cf8c1 100644
@@ -27,6 +27,36 @@ See `Large-Page Support`_ in MSDN for details.
 .. _Large-page Support: https://docs.microsoft.com/en-us/windows/win32/memory/large-page-support
 
 
+Load virt2phys Driver
+---------------------
+
+Access to physical addresses is provided by a kernel-mode driver, virt2phys.
+It is mandatory at least for using hardware PMDs, but may also be required
+for mempools.
+
+This driver is not signed, so signature checking must be disabled to load it.
+Refer to documentation in ``dpdk-kmods`` repository for details on system
+setup, driver build and installation.
+
+Compiled package, consisting of ``virt2phys.inf``, ``virt2phys.cat``,
+and ``virt2phys.sys``, is installed as follows (from Elevated Command Line):
+
+.. code-block:: console
+
+    pnputil /add-driver virt2phys.inf /install
+
+When loaded successfully, the driver is shown in *Device Manager* as *Virtual
+to physical address translator* device under *Kernel bypass* category.
+
+If DPDK is unable to communicate with the driver, a warning is printed
+on initialization (debug-level logs provide more details):
+
+.. code-block:: text
+
+    EAL: Cannot open virt2phys driver interface
+
+
+
 Run the ``helloworld`` Example
 ------------------------------
 
lib/librte_eal/common/eal_common_fbarray.c +26 -31
index 1312f936b..236db9cb7 100644
@@ -5,15 +5,15 @@
 #include <fcntl.h>
 #include <inttypes.h>
 #include <limits.h>
-#include <sys/mman.h>
 #include <stdint.h>
 #include <errno.h>
-#include <sys/file.h>
 #include <string.h>
+#include <unistd.h>
 
 #include <rte_common.h>
-#include <rte_log.h>
 #include <rte_errno.h>
+#include <rte_log.h>
+#include <rte_memory.h>
 #include <rte_spinlock.h>
 #include <rte_tailq.h>
 
@@ -85,19 +85,16 @@ resize_and_map(int fd, void *addr, size_t len)
 	char path[PATH_MAX];
 	void *map_addr;
 
-	if (ftruncate(fd, len)) {
+	if (eal_file_truncate(fd, len)) {
 		RTE_LOG(ERR, EAL, "Cannot truncate %s\n", path);
 		/* pass errno up the chain */
 		rte_errno = errno;
 		return -1;
 	}
 
-	map_addr = mmap(addr, len, PROT_READ | PROT_WRITE,
-			MAP_SHARED | MAP_FIXED, fd, 0);
+	map_addr = rte_mem_map(addr, len, RTE_PROT_READ | RTE_PROT_WRITE,
+			RTE_MAP_SHARED | RTE_MAP_FIXED, fd, 0);
 	if (map_addr != addr) {
-		RTE_LOG(ERR, EAL, "mmap() failed: %s\n", strerror(errno));
-		/* pass errno up the chain */
-		rte_errno = errno;
 		return -1;
 	}
 	return 0;
@@ -735,7 +732,7 @@ rte_fbarray_init(struct rte_fbarray *arr, const char *name, unsigned int len,
 		return -1;
 	}
 
-	page_sz = sysconf(_SC_PAGESIZE);
+	page_sz = rte_get_page_size();
 	if (page_sz == (size_t)-1) {
 		free(ma);
 		return -1;
@@ -756,9 +753,12 @@ rte_fbarray_init(struct rte_fbarray *arr, const char *name, unsigned int len,
 
 	if (internal_config.no_shconf) {
 		/* remap virtual area as writable */
-		void *new_data = mmap(data, mmap_len, PROT_READ | PROT_WRITE,
-				MAP_FIXED | MAP_PRIVATE | MAP_ANONYMOUS, fd, 0);
-		if (new_data == MAP_FAILED) {
+		void *new_data = rte_mem_map(
+			data, mmap_len,
+			RTE_PROT_READ | RTE_PROT_WRITE,
+			RTE_MAP_FIXED | RTE_MAP_PRIVATE | RTE_MAP_ANONYMOUS,
+			fd, 0);
+		if (new_data == NULL) {
 			RTE_LOG(DEBUG, EAL, "%s(): couldn't remap anonymous memory: %s\n",
 					__func__, strerror(errno));
 			goto fail;
@@ -778,7 +778,8 @@ rte_fbarray_init(struct rte_fbarray *arr, const char *name, unsigned int len,
 					__func__, path, strerror(errno));
 			rte_errno = errno;
 			goto fail;
-		} else if (flock(fd, LOCK_EX | LOCK_NB)) {
+		} else if (eal_file_lock(
+				fd, EAL_FLOCK_EXCLUSIVE, EAL_FLOCK_RETURN)) {
 			RTE_LOG(DEBUG, EAL, "%s(): couldn't lock %s: %s\n",
 					__func__, path, strerror(errno));
 			rte_errno = EBUSY;
@@ -789,10 +790,8 @@ rte_fbarray_init(struct rte_fbarray *arr, const char *name, unsigned int len,
 		 * still attach to it, but no other process could reinitialize
 		 * it.
 		 */
-		if (flock(fd, LOCK_SH | LOCK_NB)) {
-			rte_errno = errno;
+		if (eal_file_lock(fd, EAL_FLOCK_SHARED, EAL_FLOCK_RETURN))
 			goto fail;
-		}
 
 		if (resize_and_map(fd, data, mmap_len))
 			goto fail;
@@ -824,7 +823,7 @@ rte_fbarray_init(struct rte_fbarray *arr, const char *name, unsigned int len,
 	return 0;
 fail:
 	if (data)
-		munmap(data, mmap_len);
+		rte_mem_unmap(data, mmap_len);
 	if (fd >= 0)
 		close(fd);
 	free(ma);
@@ -862,7 +861,7 @@ rte_fbarray_attach(struct rte_fbarray *arr)
 		return -1;
 	}
 
-	page_sz = sysconf(_SC_PAGESIZE);
+	page_sz = rte_get_page_size();
 	if (page_sz == (size_t)-1) {
 		free(ma);
 		return -1;
@@ -895,10 +894,8 @@ rte_fbarray_attach(struct rte_fbarray *arr)
 	}
 
 	/* lock the file, to let others know we're using it */
-	if (flock(fd, LOCK_SH | LOCK_NB)) {
-		rte_errno = errno;
+	if (eal_file_lock(fd, EAL_FLOCK_SHARED, EAL_FLOCK_RETURN))
 		goto fail;
-	}
 
 	if (resize_and_map(fd, data, mmap_len))
 		goto fail;
@@ -916,7 +913,7 @@ rte_fbarray_attach(struct rte_fbarray *arr)
 	return 0;
 fail:
 	if (data)
-		munmap(data, mmap_len);
+		rte_mem_unmap(data, mmap_len);
 	if (fd >= 0)
 		close(fd);
 	free(ma);
@@ -944,8 +941,7 @@ rte_fbarray_detach(struct rte_fbarray *arr)
 	 * really do anything about it, things will blow up either way.
 	 */
 
-	size_t page_sz = sysconf(_SC_PAGESIZE);
-
+	size_t page_sz = rte_get_page_size();
 	if (page_sz == (size_t)-1)
 		return -1;
 
@@ -964,7 +960,7 @@ rte_fbarray_detach(struct rte_fbarray *arr)
 		goto out;
 	}
 
-	munmap(arr->data, mmap_len);
+	rte_mem_unmap(arr->data, mmap_len);
 
 	/* area is unmapped, close fd and remove the tailq entry */
 	if (tmp->fd >= 0)
@@ -999,8 +995,7 @@ rte_fbarray_destroy(struct rte_fbarray *arr)
 	 * really do anything about it, things will blow up either way.
 	 */
 
-	size_t page_sz = sysconf(_SC_PAGESIZE);
-
+	size_t page_sz = rte_get_page_size();
 	if (page_sz == (size_t)-1)
 		return -1;
 
@@ -1025,7 +1020,7 @@ rte_fbarray_destroy(struct rte_fbarray *arr)
 		 * has been detached by all other processes
 		 */
 		fd = tmp->fd;
-		if (flock(fd, LOCK_EX | LOCK_NB)) {
+		if (eal_file_lock(fd, EAL_FLOCK_EXCLUSIVE, EAL_FLOCK_RETURN)) {
 			RTE_LOG(DEBUG, EAL, "Cannot destroy fbarray - another process is using it\n");
 			rte_errno = EBUSY;
 			ret = -1;
@@ -1042,14 +1037,14 @@ rte_fbarray_destroy(struct rte_fbarray *arr)
 			 * we're still holding an exclusive lock, so drop it to
 			 * shared.
 			 */
-			flock(fd, LOCK_SH | LOCK_NB);
+			eal_file_lock(fd, EAL_FLOCK_SHARED, EAL_FLOCK_RETURN);
 
 			ret = -1;
 			goto out;
 		}
 		close(fd);
 	}
-	munmap(arr->data, mmap_len);
+	rte_mem_unmap(arr->data, mmap_len);
 
 	/* area is unmapped, remove the tailq entry */
 	TAILQ_REMOVE(&mem_area_tailq, tmp, next);
lib/librte_eal/common/eal_common_memory.c +22 -28
index d9764681a..5c3cf1f75 100644
@@ -11,7 +11,6 @@
 #include <string.h>
 #include <unistd.h>
 #include <inttypes.h>
-#include <sys/mman.h>
 #include <sys/queue.h>
 
 #include <rte_fbarray.h>
@@ -44,7 +43,7 @@ static uint64_t system_page_sz;
 #define MAX_MMAP_WITH_DEFINED_ADDR_TRIES 5
 void *
 eal_get_virtual_area(void *requested_addr, size_t *size,
-		size_t page_sz, int flags, int mmap_flags)
+	size_t page_sz, int flags, enum eal_mem_reserve_flags reserve_flags)
 {
 	bool addr_is_hint, allow_shrink, unmap, no_align;
 	uint64_t map_sz;
@@ -52,9 +51,7 @@ eal_get_virtual_area(void *requested_addr, size_t *size,
 	uint8_t try = 0;
 
 	if (system_page_sz == 0)
-		system_page_sz = sysconf(_SC_PAGESIZE);
-
-	mmap_flags |= MAP_PRIVATE | MAP_ANONYMOUS;
+		system_page_sz = rte_get_page_size();
 
 	RTE_LOG(DEBUG, EAL, "Ask a virtual area of 0x%zx bytes\n", *size);
 
@@ -98,24 +95,24 @@ eal_get_virtual_area(void *requested_addr, size_t *size,
 			return NULL;
 		}
 
-		mapped_addr = mmap(requested_addr, (size_t)map_sz, PROT_NONE,
-				mmap_flags, -1, 0);
-		if (mapped_addr == MAP_FAILED && allow_shrink)
-			*size -= page_sz;
+		mapped_addr = eal_mem_reserve(
+			requested_addr, (size_t)map_sz, reserve_flags);
+		if ((mapped_addr == NULL) && allow_shrink)
+			size -= page_sz;
 
-		if (mapped_addr != MAP_FAILED && addr_is_hint &&
-		    mapped_addr != requested_addr) {
+		if ((mapped_addr != NULL) && addr_is_hint &&
+				(mapped_addr != requested_addr)) {
 			try++;
 			next_baseaddr = RTE_PTR_ADD(next_baseaddr, page_sz);
 			if (try <= MAX_MMAP_WITH_DEFINED_ADDR_TRIES) {
 				/* hint was not used. Try with another offset */
-				munmap(mapped_addr, map_sz);
-				mapped_addr = MAP_FAILED;
+				eal_mem_free(mapped_addr, *size);
+				mapped_addr = NULL;
 				requested_addr = next_baseaddr;
 			}
 		}
 	} while ((allow_shrink || addr_is_hint) &&
-		 mapped_addr == MAP_FAILED && *size > 0);
+		(mapped_addr == NULL) && (*size > 0));
 
 	/* align resulting address - if map failed, we will ignore the value
 	 * anyway, so no need to add additional checks.
@@ -125,20 +122,17 @@ eal_get_virtual_area(void *requested_addr, size_t *size,
 
 	if (*size == 0) {
 		RTE_LOG(ERR, EAL, "Cannot get a virtual area of any size: %s\n",
-			strerror(errno));
-		rte_errno = errno;
+			strerror(rte_errno));
 		return NULL;
-	} else if (mapped_addr == MAP_FAILED) {
+	} else if (mapped_addr == NULL) {
 		RTE_LOG(ERR, EAL, "Cannot get a virtual area: %s\n",
-			strerror(errno));
-		/* pass errno up the call chain */
-		rte_errno = errno;
+			strerror(rte_errno));
 		return NULL;
 	} else if (requested_addr != NULL && !addr_is_hint &&
 			aligned_addr != requested_addr) {
 		RTE_LOG(ERR, EAL, "Cannot get a virtual area at requested address: %p (got %p)\n",
 			requested_addr, aligned_addr);
-		munmap(mapped_addr, map_sz);
+		eal_mem_free(mapped_addr, map_sz);
 		rte_errno = EADDRNOTAVAIL;
 		return NULL;
 	} else if (requested_addr != NULL && addr_is_hint &&
@@ -154,7 +148,7 @@ eal_get_virtual_area(void *requested_addr, size_t *size,
 		aligned_addr, *size);
 
 	if (unmap) {
-		munmap(mapped_addr, map_sz);
+		eal_mem_free(mapped_addr, map_sz);
 	} else if (!no_align) {
 		void *map_end, *aligned_end;
 		size_t before_len, after_len;
@@ -172,12 +166,12 @@ eal_get_virtual_area(void *requested_addr, size_t *size,
 		/* unmap space before aligned mmap address */
 		before_len = RTE_PTR_DIFF(aligned_addr, mapped_addr);
 		if (before_len > 0)
-			munmap(mapped_addr, before_len);
+			eal_mem_free(mapped_addr, before_len);
 
 		/* unmap space after aligned end mmap address */
 		after_len = RTE_PTR_DIFF(map_end, aligned_end);
 		if (after_len > 0)
-			munmap(aligned_end, after_len);
+			eal_mem_free(aligned_end, after_len);
 	}
 
 	return aligned_addr;
@@ -586,10 +580,10 @@ rte_eal_memdevice_init(void)
 int
 rte_mem_lock_page(const void *virt)
 {
-	unsigned long virtual = (unsigned long)virt;
-	int page_size = getpagesize();
-	unsigned long aligned = (virtual & ~(page_size - 1));
-	return mlock((void *)aligned, page_size);
+	uintptr_t virtual = (uintptr_t)virt;
+	int page_size = rte_get_page_size();
+	uintptr_t aligned = (virtual & ~(page_size - 1));
+	return rte_mem_lock((void *)aligned, page_size);
 }
 
 int
lib/librte_eal/common/eal_private.h +3 -3
index 76938e379..59ac41916 100644
@@ -226,8 +226,8 @@ enum eal_mem_reserve_flags {
  *   Page size on which to align requested virtual area.
  * @param flags
  *   EAL_VIRTUAL_AREA_* flags.
- * @param mmap_flags
- *   Extra flags passed directly to mmap().
+ * @param reserve_flags
+ *   Extra flags passed directly to rte_mem_reserve().
  *
  * @return
  *   Virtual area address if successful.
@@ -244,7 +244,7 @@ enum eal_mem_reserve_flags {
 /**< immediately unmap reserved virtual area. */
 void *
 eal_get_virtual_area(void *requested_addr, size_t *size, size_t page_sz,
-	int flags, int mmap_flags);
+	int flags, enum eal_mem_reserve_flags reserve_flags);
 
 /**
  * Reserve VA space for a memory segment list.
lib/librte_eal/common/malloc_heap.c +1 -0
index 842eb9de7..6534c895c 100644
@@ -729,6 +729,7 @@ malloc_heap_alloc(const char *type, size_t size, int socket_arg,
 		if (ret != NULL)
 			return ret;
 	}
+
 	return NULL;
 }
 
lib/librte_eal/common/meson.build +9 -0
index 02d9280cc..6dcdcc890 100644
@@ -9,11 +9,20 @@ if is_windows
 		'eal_common_class.c',
 		'eal_common_devargs.c',
 		'eal_common_errno.c',
+		'eal_common_fbarray.c',
 		'eal_common_launch.c',
 		'eal_common_lcore.c',
 		'eal_common_log.c',
+		'eal_common_mcfg.c',
+		'eal_common_memalloc.c',
+		'eal_common_memory.c',
+		'eal_common_memzone.c',
 		'eal_common_options.c',
+		'eal_common_tailqs.c',
 		'eal_common_thread.c',
+		'malloc_elem.c',
+		'malloc_heap.c',
+		'rte_malloc.c',
 		'rte_option.c',
 	)
 	subdir_done()
lib/librte_eal/freebsd/eal_memory.c +0 -1
index 5174f9cd0..99bf6ec9e 100644
@@ -355,7 +355,6 @@ memseg_list_reserve(struct rte_memseg_list *msl)
 	return eal_reserve_memseg_list(msl, flags);
 }
 
-
 static int
 memseg_primary_init(void)
 {
lib/librte_eal/meson.build +1 -0
index 1f89efb88..1d750f003 100644
@@ -11,6 +11,7 @@ if not is_windows
 endif
 
 dpdk_conf.set('RTE_EXEC_ENV_' + exec_env.to_upper(), 1)
+dpdk_conf.set10('RTE_EAL_NUMA_AWARE_HUGEPAGES', true)
 subdir(exec_env)
 
 subdir(arch_subdir)
lib/librte_eal/rte_eal_exports.def +117 -2
index bacf9a107..854b83bcd 100644
@@ -1,13 +1,128 @@
 EXPORTS
 	__rte_panic
+	rte_calloc
+	rte_calloc_socket
 	rte_eal_get_configuration
+	rte_eal_has_hugepages
 	rte_eal_init
+	rte_eal_iova_mode
 	rte_eal_mp_remote_launch
 	rte_eal_mp_wait_lcore
+	rte_eal_process_type
 	rte_eal_remote_launch
-	rte_get_page_size
+	rte_eal_tailq_lookup
+	rte_eal_tailq_register
+	rte_eal_using_phys_addrs
+	rte_free
 	rte_log
+	rte_malloc
+	rte_malloc_dump_stats
+	rte_malloc_get_socket_stats
+	rte_malloc_set_limit
+	rte_malloc_socket
+	rte_malloc_validate
+	rte_malloc_virt2iova
+	rte_mcfg_mem_read_lock
+	rte_mcfg_mem_read_unlock
+	rte_mcfg_mem_write_lock
+	rte_mcfg_mem_write_unlock
+	rte_mcfg_mempool_read_lock
+	rte_mcfg_mempool_read_unlock
+	rte_mcfg_mempool_write_lock
+	rte_mcfg_mempool_write_unlock
+	rte_mcfg_tailq_read_lock
+	rte_mcfg_tailq_read_unlock
+	rte_mcfg_tailq_write_lock
+	rte_mcfg_tailq_write_unlock
+	rte_mem_lock_page
+	rte_mem_virt2iova
+	rte_mem_virt2phy
+	rte_memory_get_nchannel
+	rte_memory_get_nrank
+	rte_memzone_dump
+	rte_memzone_free
+	rte_memzone_lookup
+	rte_memzone_reserve
+	rte_memzone_reserve_aligned
+	rte_memzone_reserve_bounded
+	rte_memzone_walk
+	rte_vlog
+	rte_realloc
+	rte_zmalloc
+	rte_zmalloc_socket
+
+	rte_mp_action_register
+	rte_mp_action_unregister
+	rte_mp_reply
+	rte_mp_sendmsg
+
+	rte_fbarray_attach
+	rte_fbarray_destroy
+	rte_fbarray_detach
+	rte_fbarray_dump_metadata
+	rte_fbarray_find_contig_free
+	rte_fbarray_find_contig_used
+	rte_fbarray_find_idx
+	rte_fbarray_find_next_free
+	rte_fbarray_find_next_n_free
+	rte_fbarray_find_next_n_used
+	rte_fbarray_find_next_used
+	rte_fbarray_get
+	rte_fbarray_init
+	rte_fbarray_is_used
+	rte_fbarray_set_free
+	rte_fbarray_set_used
+	rte_malloc_dump_heaps
+	rte_mem_alloc_validator_register
+	rte_mem_alloc_validator_unregister
+	rte_mem_check_dma_mask
+	rte_mem_event_callback_register
+	rte_mem_event_callback_unregister
+	rte_mem_iova2virt
+	rte_mem_virt2memseg
+	rte_mem_virt2memseg_list
+	rte_memseg_contig_walk
+	rte_memseg_list_walk
+	rte_memseg_walk
+	rte_mp_request_async
+	rte_mp_request_sync
+
+	rte_fbarray_find_prev_free
+	rte_fbarray_find_prev_n_free
+	rte_fbarray_find_prev_n_used
+	rte_fbarray_find_prev_used
+	rte_fbarray_find_rev_contig_free
+	rte_fbarray_find_rev_contig_used
+	rte_memseg_contig_walk_thread_unsafe
+	rte_memseg_list_walk_thread_unsafe
+	rte_memseg_walk_thread_unsafe
+
+	rte_malloc_heap_create
+	rte_malloc_heap_destroy
+	rte_malloc_heap_get_socket
+	rte_malloc_heap_memory_add
+	rte_malloc_heap_memory_attach
+	rte_malloc_heap_memory_detach
+	rte_malloc_heap_memory_remove
+	rte_malloc_heap_socket_is_external
+	rte_mem_check_dma_mask_thread_unsafe
+	rte_mem_set_dma_mask
+	rte_memseg_get_fd
+	rte_memseg_get_fd_offset
+	rte_memseg_get_fd_offset_thread_unsafe
+	rte_memseg_get_fd_thread_unsafe
+
+	rte_extmem_attach
+	rte_extmem_detach
+	rte_extmem_register
+	rte_extmem_unregister
+
+	rte_fbarray_find_biggest_free
+	rte_fbarray_find_biggest_used
+	rte_fbarray_find_rev_biggest_free
+	rte_fbarray_find_rev_biggest_used
+
+	rte_get_page_size
 	rte_mem_lock
 	rte_mem_map
 	rte_mem_unmap
-	rte_vlog
lib/librte_eal/windows/eal.c +55 -0
index cf55b56da..38f17f09c 100644
@@ -93,6 +93,24 @@ eal_proc_type_detect(void)
 	return ptype;
 }
 
+enum rte_proc_type_t
+rte_eal_process_type(void)
+{
+	return rte_config.process_type;
+}
+
+int
+rte_eal_has_hugepages(void)
+{
+	return !internal_config.no_hugetlbfs;
+}
+
+enum rte_iova_mode
+rte_eal_iova_mode(void)
+{
+	return rte_config.iova_mode;
+}
+
 /* display usage */
 static void
 eal_usage(const char *prgname)
@@ -328,6 +346,13 @@ rte_eal_init(int argc, char **argv)
 	if (fctret < 0)
 		exit(1);
 
+	/* Prevent creation of shared memory files. */
+	if (internal_config.no_shconf == 0) {
+		RTE_LOG(WARNING, EAL, "Multi-process support is requested, "
+			"but not available.\n");
+		internal_config.no_shconf = 1;
+	}
+
 	if (!internal_config.no_hugetlbfs && (eal_hugepage_info_init() < 0)) {
 		rte_eal_init_alert("Cannot get hugepage information");
 		rte_errno = EACCES;
@@ -345,6 +370,36 @@ rte_eal_init(int argc, char **argv)
 		return -1;
 	}
 
+	if (eal_mem_virt2iova_init() < 0) {
+		/* Non-fatal error if physical addresses are not required. */
+		RTE_LOG(WARNING, EAL, "Cannot access virt2phys driver, "
+			"PA will not be available\n");
+	}
+
+	if (rte_eal_memzone_init() < 0) {
+		rte_eal_init_alert("Cannot init memzone");
+		rte_errno = ENODEV;
+		return -1;
+	}
+
+	if (rte_eal_memory_init() < 0) {
+		rte_eal_init_alert("Cannot init memory");
+		rte_errno = ENOMEM;
+		return -1;
+	}
+
+	if (rte_eal_malloc_heap_init() < 0) {
+		rte_eal_init_alert("Cannot init malloc heap");
+		rte_errno = ENODEV;
+		return -1;
+	}
+
+	if (rte_eal_tailqs_init() < 0) {
+		rte_eal_init_alert("Cannot init tail queues for objects");
+		rte_errno = EFAULT;
+		return -1;
+	}
+
 	eal_thread_init_master(rte_config.master_lcore);
 
 	RTE_LCORE_FOREACH_SLAVE(i) {
lib/librte_eal/windows/eal_memalloc.c new file +58 -0
new file mode 100644
index 000000000..c7c3cf8df
@@ -0,0 +1,423 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2020 Dmitry Kozlyuk
+ */
+
+#include <rte_errno.h>
+#include <rte_os.h>
+#include <rte_windows.h>
+
+#include "eal_internal_cfg.h"
+#include "eal_memalloc.h"
+#include "eal_memcfg.h"
+#include "eal_private.h"
+#include "eal_windows.h"
+
+int
+eal_memalloc_get_seg_fd(int list_idx, int seg_idx)
+{
+	/* Hugepages have no assiciated files in Windows. */
+	RTE_SET_USED(list_idx);
+	RTE_SET_USED(seg_idx);
+	EAL_LOG_NOT_IMPLEMENTED();
+	return -1;
+}
+
+int
+eal_memalloc_get_seg_fd_offset(int list_idx, int seg_idx, size_t *offset)
+{
+	/* Hugepages have no assiciated files in Windows. */
+	RTE_SET_USED(list_idx);
+	RTE_SET_USED(seg_idx);
+	RTE_SET_USED(offset);
+	EAL_LOG_NOT_IMPLEMENTED();
+	return -1;
+}
+
+static int
+alloc_seg(struct rte_memseg *ms, void *requested_addr, int socket_id,
+	struct hugepage_info *hi)
+{
+	HANDLE current_process;
+	unsigned int numa_node;
+	size_t alloc_sz;
+	void *addr;
+	rte_iova_t iova = RTE_BAD_IOVA;
+	PSAPI_WORKING_SET_EX_INFORMATION info;
+	PSAPI_WORKING_SET_EX_BLOCK *page;
+
+	if (ms->len > 0) {
+		/* If a segment is already allocated as needed, return it. */
+		if ((ms->addr == requested_addr) &&
+			(ms->socket_id == socket_id) &&
+			(ms->hugepage_sz == hi->hugepage_sz)) {
+			return 0;
+		}
+
+		/* Bugcheck, should not happen. */
+		RTE_LOG(DEBUG, EAL, "Attempted to reallocate segment %p "
+			"(size %zu) on socket %d", ms->addr,

lib/librte_eal/windows/eal_memalloc.c line 68

Quoted 10 lines
lib/librte_eal/windows/eal_memalloc.c new file +10 -0
+			ms->len, ms->socket_id);
+		return -1;
+	}
+
+	current_process = GetCurrentProcess();
+	numa_node = eal_socket_numa_node(socket_id);
+	alloc_sz = hi->hugepage_sz;
+
+	if (requested_addr == NULL) {
+		/* Request a new chunk of memory and enforce address hint. */

Does requested_addr being NULL means that no hint was provided? It also looks like eal_mem_alloc_socket ignores the address hint anyway and just calls VirtualAllocExNuma with NULL for lpAddress. Maybe remove the second part of the comment.

lib/librte_eal/windows/eal_memalloc.c line 76

Quoted 8 lines
lib/librte_eal/windows/eal_memalloc.c new file +8 -0
+		addr = eal_mem_alloc_socket(alloc_sz, socket_id);
+		if (addr == NULL) {
+			RTE_LOG(DEBUG, EAL, "Cannot allocate %zu bytes "
+				"on socket %d\n", alloc_sz, socket_id);
+			return -1;
+		}
+
+		if (addr != requested_addr) {

requested_addr is NULL on this branch and we confirmed with the previous 'if' that addr is not NULL. Should this branch be removed, since requested_addr is NULL, so there is no hint provided?

lib/librte_eal/windows/eal_memalloc.c line 87

Quoted 11 lines
lib/librte_eal/windows/eal_memalloc.c new file +11 -0
+			RTE_LOG(DEBUG, EAL, "Address hint %p not respected, "
+				"got %p\n", requested_addr, addr);
+			goto error;
+		}
+	} else {
+		/* Requested address is already reserved, commit memory. */
+		addr = eal_mem_commit(requested_addr, alloc_sz, socket_id);
+		if (addr == NULL) {
+			RTE_LOG(DEBUG, EAL, "Cannot commit reserved memory %p "
+				"(size %zu)\n", requested_addr, alloc_sz);
+			goto error;

Execution jumps to 'error' with an invalid addr, so it will try to call eal_mem_decommit with NULL as parameter. Instead of 'goto error', maybe we should return here.

Quoted 1,338 lines
lib/librte_eal/windows/eal_memalloc.c new file +336 -0
+		}
+	}
+
+	/* Force OS to allocate a physical page and select a NUMA node.
+	 * Hugepages are not pageable in Windows, so there's no race
+	 * for physical address.
+	 */
+	*(volatile int *)addr = *(volatile int *)addr;
+
+	/* Only try to obtain IOVA if it's available, so that applications
+	 * that do not need IOVA can use this allocator.
+	 */
+	if (rte_eal_using_phys_addrs()) {
+		iova = rte_mem_virt2iova(addr);
+		if (iova == RTE_BAD_IOVA) {
+			RTE_LOG(DEBUG, EAL,
+				"Cannot get IOVA of allocated segment\n");
+			goto error;
+		}
+	}
+
+	/* Only "Ex" function can handle hugepages. */
+	info.VirtualAddress = addr;
+	if (!QueryWorkingSetEx(current_process, &info, sizeof(info))) {
+		RTE_LOG_WIN32_ERR("QueryWorkingSetEx()");
+		goto error;
+	}
+
+	page = &info.VirtualAttributes;
+	if (!page->Valid || !page->LargePage) {
+		RTE_LOG(DEBUG, EAL, "Got regular page instead of hugepage\n");
+		goto error;
+	}
+	if (page->Node != numa_node) {
+		RTE_LOG(DEBUG, EAL,
+			"NUMA node hint %u (socket %d) not respected, got %u\n",
+			numa_node, socket_id, page->Node);
+		goto error;
+	}
+
+	ms->addr = addr;
+	ms->hugepage_sz = hi->hugepage_sz;
+	ms->len = alloc_sz;
+	ms->nchannel = rte_memory_get_nchannel();
+	ms->nrank = rte_memory_get_nrank();
+	ms->iova = iova;
+	ms->socket_id = socket_id;
+
+	return 0;
+
+error:
+	/* Only jump here when `addr` and `alloc_sz` are valid. */
+	eal_mem_decommit(addr, alloc_sz);
+	return -1;
+}
+
+static int
+free_seg(struct rte_memseg *ms)
+{
+	if (eal_mem_decommit(ms->addr, ms->len))
+		return -1;
+
+	/* Must clear the segment, because alloc_seg() inspects it. */
+	memset(ms, 0, sizeof(*ms));
+	return 0;
+}
+
+struct alloc_walk_param {
+	struct hugepage_info *hi;
+	struct rte_memseg **ms;
+	size_t page_sz;
+	unsigned int segs_allocated;
+	unsigned int n_segs;
+	int socket;
+	bool exact;
+};
+
+static int
+alloc_seg_walk(const struct rte_memseg_list *msl, void *arg)
+{
+	struct rte_mem_config *mcfg = rte_eal_get_configuration()->mem_config;
+	struct alloc_walk_param *wa = arg;
+	struct rte_memseg_list *cur_msl;
+	size_t page_sz;
+	int cur_idx, start_idx, j;
+	unsigned int msl_idx, need, i;
+
+	if (msl->page_sz != wa->page_sz)
+		return 0;
+	if (msl->socket_id != wa->socket)
+		return 0;
+
+	page_sz = (size_t)msl->page_sz;
+
+	msl_idx = msl - mcfg->memsegs;
+	cur_msl = &mcfg->memsegs[msl_idx];
+
+	need = wa->n_segs;
+
+	/* try finding space in memseg list */
+	if (wa->exact) {
+		/* if we require exact number of pages in a list, find them */
+		cur_idx = rte_fbarray_find_next_n_free(
+			&cur_msl->memseg_arr, 0, need);
+		if (cur_idx < 0)
+			return 0;
+		start_idx = cur_idx;
+	} else {
+		int cur_len;
+
+		/* we don't require exact number of pages, so we're going to go
+		 * for best-effort allocation. that means finding the biggest
+		 * unused block, and going with that.
+		 */
+		cur_idx = rte_fbarray_find_biggest_free(
+			&cur_msl->memseg_arr, 0);
+		if (cur_idx < 0)
+			return 0;
+		start_idx = cur_idx;
+		/* adjust the size to possibly be smaller than original
+		 * request, but do not allow it to be bigger.
+		 */
+		cur_len = rte_fbarray_find_contig_free(
+			&cur_msl->memseg_arr, cur_idx);
+		need = RTE_MIN(need, (unsigned int)cur_len);
+	}
+
+	for (i = 0; i < need; i++, cur_idx++) {
+		struct rte_memseg *cur;
+		void *map_addr;
+
+		cur = rte_fbarray_get(&cur_msl->memseg_arr, cur_idx);
+		map_addr = RTE_PTR_ADD(cur_msl->base_va, cur_idx * page_sz);
+
+		if (alloc_seg(cur, map_addr, wa->socket, wa->hi)) {
+			RTE_LOG(DEBUG, EAL, "attempted to allocate %i segments, "
+				"but only %i were allocated\n", need, i);
+
+			/* if exact number wasn't requested, stop */
+			if (!wa->exact)
+				goto out;
+
+			/* clean up */
+			for (j = start_idx; j < cur_idx; j++) {
+				struct rte_memseg *tmp;
+				struct rte_fbarray *arr = &cur_msl->memseg_arr;
+
+				tmp = rte_fbarray_get(arr, j);
+				rte_fbarray_set_free(arr, j);
+
+				if (free_seg(tmp))
+					RTE_LOG(DEBUG, EAL, "Cannot free page\n");
+			}
+			/* clear the list */
+			if (wa->ms)
+				memset(wa->ms, 0, sizeof(*wa->ms) * wa->n_segs);
+
+			return -1;
+		}
+		if (wa->ms)
+			wa->ms[i] = cur;
+
+		rte_fbarray_set_used(&cur_msl->memseg_arr, cur_idx);
+	}
+
+out:
+	wa->segs_allocated = i;
+	if (i > 0)
+		cur_msl->version++;
+
+	/* if we didn't allocate any segments, move on to the next list */
+	return i > 0;
+}
+
+struct free_walk_param {
+	struct hugepage_info *hi;
+	struct rte_memseg *ms;
+};
+static int
+free_seg_walk(const struct rte_memseg_list *msl, void *arg)
+{
+	struct rte_mem_config *mcfg = rte_eal_get_configuration()->mem_config;
+	struct rte_memseg_list *found_msl;
+	struct free_walk_param *wa = arg;
+	uintptr_t start_addr, end_addr;
+	int msl_idx, seg_idx, ret;
+
+	start_addr = (uintptr_t) msl->base_va;
+	end_addr = start_addr + msl->len;
+
+	if ((uintptr_t)wa->ms->addr < start_addr ||
+		(uintptr_t)wa->ms->addr >= end_addr)
+		return 0;
+
+	msl_idx = msl - mcfg->memsegs;
+	seg_idx = RTE_PTR_DIFF(wa->ms->addr, start_addr) / msl->page_sz;
+
+	/* msl is const */
+	found_msl = &mcfg->memsegs[msl_idx];
+	found_msl->version++;
+
+	rte_fbarray_set_free(&found_msl->memseg_arr, seg_idx);
+
+	ret = free_seg(wa->ms);
+
+	return (ret < 0) ? (-1) : 1;
+}
+
+int
+eal_memalloc_alloc_seg_bulk(struct rte_memseg **ms, int n_segs,
+		size_t page_sz, int socket, bool exact)
+{
+	unsigned int i;
+	int ret = -1;
+	struct alloc_walk_param wa;
+	struct hugepage_info *hi = NULL;
+
+	if (internal_config.legacy_mem) {
+		RTE_LOG(ERR, EAL, "dynamic allocation not supported in legacy mode\n");
+		return -ENOTSUP;
+	}
+
+	for (i = 0; i < internal_config.num_hugepage_sizes; i++) {
+		struct hugepage_info *hpi = &internal_config.hugepage_info[i];
+		if (page_sz == hpi->hugepage_sz) {
+			hi = hpi;
+			break;
+		}
+	}
+	if (!hi) {
+		RTE_LOG(ERR, EAL, "cannot find relevant hugepage_info entry\n");
+		return -1;
+	}
+
+	memset(&wa, 0, sizeof(wa));
+	wa.exact = exact;
+	wa.hi = hi;
+	wa.ms = ms;
+	wa.n_segs = n_segs;
+	wa.page_sz = page_sz;
+	wa.socket = socket;
+	wa.segs_allocated = 0;
+
+	/* memalloc is locked, so it's safe to use thread-unsafe version */
+	ret = rte_memseg_list_walk_thread_unsafe(alloc_seg_walk, &wa);
+	if (ret == 0) {
+		RTE_LOG(ERR, EAL, "cannot find suitable memseg_list\n");
+		ret = -1;
+	} else if (ret > 0) {
+		ret = (int)wa.segs_allocated;
+	}
+
+	return ret;
+}
+
+struct rte_memseg *
+eal_memalloc_alloc_seg(size_t page_sz, int socket)
+{
+	struct rte_memseg *ms = NULL;
+	eal_memalloc_alloc_seg_bulk(&ms, 1, page_sz, socket, true);
+	return ms;
+}
+
+int
+eal_memalloc_free_seg_bulk(struct rte_memseg **ms, int n_segs)
+{
+	int seg, ret = 0;
+
+	/* dynamic free not supported in legacy mode */
+	if (internal_config.legacy_mem)
+		return -1;
+
+	for (seg = 0; seg < n_segs; seg++) {
+		struct rte_memseg *cur = ms[seg];
+		struct hugepage_info *hi = NULL;
+		struct free_walk_param wa;
+		size_t i;
+		int walk_res;
+
+		/* if this page is marked as unfreeable, fail */
+		if (cur->flags & RTE_MEMSEG_FLAG_DO_NOT_FREE) {
+			RTE_LOG(DEBUG, EAL, "Page is not allowed to be freed\n");
+			ret = -1;
+			continue;
+		}
+
+		memset(&wa, 0, sizeof(wa));
+
+		for (i = 0; i < RTE_DIM(internal_config.hugepage_info);
+				i++) {
+			hi = &internal_config.hugepage_info[i];
+			if (cur->hugepage_sz == hi->hugepage_sz)
+				break;
+		}
+		if (i == RTE_DIM(internal_config.hugepage_info)) {
+			RTE_LOG(ERR, EAL, "Can't find relevant hugepage_info entry\n");
+			ret = -1;
+			continue;
+		}
+
+		wa.ms = cur;
+		wa.hi = hi;
+
+		/* memalloc is locked, so it's safe to use thread-unsafe version
+		 */
+		walk_res = rte_memseg_list_walk_thread_unsafe(free_seg_walk,
+				&wa);
+		if (walk_res == 1)
+			continue;
+		if (walk_res == 0)
+			RTE_LOG(ERR, EAL, "Couldn't find memseg list\n");
+		ret = -1;
+	}
+	return ret;
+}
+
+int
+eal_memalloc_free_seg(struct rte_memseg *ms)
+{
+	return eal_memalloc_free_seg_bulk(&ms, 1);
+}
+
+int
+eal_memalloc_sync_with_primary(void)
+{
+	/* No multi-process support. */
+	EAL_LOG_NOT_IMPLEMENTED();
+	return -1;
+}
+
+int
+eal_memalloc_init(void)
+{
+	/* No action required. */
+	return 0;
+}
lib/librte_eal/windows/eal_memory.c +701 -1
index 59606d84c..a9a35b7dc 100644
@@ -1,11 +1,23 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2010-2014 Intel Corporation (functions from Linux EAL)
+ * Copyright (c) 2020 Dmitry Kozlyuk (Windows specifics)
+ */
+
+#include <inttypes.h>
 #include <io.h>
 
 #include <rte_errno.h>
 #include <rte_memory.h>
 
+#include "eal_internal_cfg.h"
+#include "eal_memalloc.h"
+#include "eal_memcfg.h"
+#include "eal_options.h"
 #include "eal_private.h"
 #include "eal_windows.h"
 
+#include <rte_virt2phys.h>
+
 /* MinGW-w64 headers lack VirtualAlloc2() in some distributions.
  * Provide a copy of definitions and code to load it dynamically.
  * Note: definitions are copied verbatim from Microsoft documentation
@@ -120,6 +132,119 @@ eal_mem_win32api_init(void)
 
 #endif /* no VirtualAlloc2() */
 
+static HANDLE virt2phys_device = INVALID_HANDLE_VALUE;
+
+int
+eal_mem_virt2iova_init(void)
+{
+	HDEVINFO list = INVALID_HANDLE_VALUE;
+	SP_DEVICE_INTERFACE_DATA ifdata;
+	SP_DEVICE_INTERFACE_DETAIL_DATA *detail = NULL;
+	DWORD detail_size;
+	int ret = -1;
+
+	list = SetupDiGetClassDevs(
+		&GUID_DEVINTERFACE_VIRT2PHYS, NULL, NULL,
+		DIGCF_DEVICEINTERFACE | DIGCF_PRESENT);
+	if (list == INVALID_HANDLE_VALUE) {
+		RTE_LOG_WIN32_ERR("SetupDiGetClassDevs()");
+		goto exit;
+	}
+
+	ifdata.cbSize = sizeof(ifdata);
+	if (!SetupDiEnumDeviceInterfaces(
+		list, NULL, &GUID_DEVINTERFACE_VIRT2PHYS, 0, &ifdata)) {
+		RTE_LOG_WIN32_ERR("SetupDiEnumDeviceInterfaces()");
+		goto exit;
+	}
+
+	if (!SetupDiGetDeviceInterfaceDetail(
+		list, &ifdata, NULL, 0, &detail_size, NULL)) {
+		if (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {
+			RTE_LOG_WIN32_ERR(
+				"SetupDiGetDeviceInterfaceDetail(probe)");
+			goto exit;
+		}
+	}
+
+	detail = malloc(detail_size);
+	if (detail == NULL) {
+		RTE_LOG(ERR, EAL, "Cannot allocate virt2phys "
+			"device interface detail data\n");
+		goto exit;
+	}
+
+	detail->cbSize = sizeof(*detail);
+	if (!SetupDiGetDeviceInterfaceDetail(
+		list, &ifdata, detail, detail_size, NULL, NULL)) {
+		RTE_LOG_WIN32_ERR("SetupDiGetDeviceInterfaceDetail(read)");
+		goto exit;
+	}
+
+	RTE_LOG(DEBUG, EAL, "Found virt2phys device: %s\n", detail->DevicePath);
+
+	virt2phys_device = CreateFile(
+		detail->DevicePath, 0, 0, NULL, OPEN_EXISTING, 0, NULL);
+	if (virt2phys_device == INVALID_HANDLE_VALUE) {
+		RTE_LOG_WIN32_ERR("CreateFile()");
+		goto exit;
+	}
+
+	/* Indicate success. */
+	ret = 0;
+
+exit:
+	if (detail != NULL)
+		free(detail);
+	if (list != INVALID_HANDLE_VALUE)
+		SetupDiDestroyDeviceInfoList(list);
+
+	return ret;
+}
+
+phys_addr_t
+rte_mem_virt2phy(const void *virt)
+{
+	LARGE_INTEGER phys;
+	DWORD bytes_returned;
+
+	if (virt2phys_device == INVALID_HANDLE_VALUE)
+		return RTE_BAD_PHYS_ADDR;
+
+	if (!DeviceIoControl(
+			virt2phys_device, IOCTL_VIRT2PHYS_TRANSLATE,
+			&virt, sizeof(virt), &phys, sizeof(phys),
+			&bytes_returned, NULL)) {
+		RTE_LOG_WIN32_ERR("DeviceIoControl(IOCTL_VIRT2PHYS_TRANSLATE)");
+		return RTE_BAD_PHYS_ADDR;
+	}
+
+	return phys.QuadPart;
+}
+
+/* Windows currently only supports IOVA as PA. */
+rte_iova_t
+rte_mem_virt2iova(const void *virt)
+{
+	phys_addr_t phys;
+
+	if (virt2phys_device == INVALID_HANDLE_VALUE)
+		return RTE_BAD_IOVA;
+
+	phys = rte_mem_virt2phy(virt);
+	if (phys == RTE_BAD_PHYS_ADDR)
+		return RTE_BAD_IOVA;
+
+	return (rte_iova_t)phys;
+}
+
+/* Always using physical addresses under Windows if they can be obtained. */
+int
+rte_eal_using_phys_addrs(void)
+{
+	return virt2phys_device != INVALID_HANDLE_VALUE;
+}
+
 /* Approximate error mapping from VirtualAlloc2() to POSIX mmap(3). */
 static int
 win32_alloc_error_to_errno(DWORD code)
@@ -360,7 +485,7 @@ rte_mem_map(void *requested_addr, size_t size, enum rte_mem_prot prot,
 		return NULL;
 	}
 
-	/* TODO: there is a race for the requested_addr between mem_free()
+	/* There is a race for the requested_addr between mem_free()
 	 * and MapViewOfFileEx(). MapViewOfFile3() that can replace a reserved
 	 * region with a mapping in a single operation, but it does not support
 	 * private mappings.
@@ -410,6 +535,16 @@ rte_mem_unmap(void *virt, size_t size)
 	return 0;
 }
 
+uint64_t
+eal_get_baseaddr(void)
+{
+	/* Windows strategy for memory allocation is undocumented.
+	 * Returning 0 here effectively disables address guessing
+	 * unless user provides an address hint.
+	 */
+	return 0;
+}
+
 int
 rte_get_page_size(void)
 {
@@ -431,3 +566,568 @@ rte_mem_lock(const void *virt, size_t size)
 
 	return 0;
 }
+
+static int
+memseg_list_alloc(struct rte_memseg_list *msl, uint64_t page_sz,
+		int n_segs, int socket_id, int type_msl_idx)
+{
+	return eal_alloc_memseg_list(
+		msl, page_sz, n_segs, socket_id, type_msl_idx, true);
+}
+
+static int
+memseg_list_reserve(struct rte_memseg_list *msl)
+{
+	return eal_reserve_memseg_list(msl, 0);
+}
+
+/*
+ * Remaining code in this file largely duplicates Linux EAL.
+ * Although Windows EAL supports only one hugepage size currently,
+ * code structure and comments are preserved so that changes may be
+ * easily ported until duplication is removed.
+ */
+
+static int
+memseg_primary_init(void)
+{
+	struct rte_mem_config *mcfg = rte_eal_get_configuration()->mem_config;
+	struct memtype {
+		uint64_t page_sz;
+		int socket_id;
+	} *memtypes = NULL;
+	int i, hpi_idx, msl_idx, ret = -1; /* fail unless told to succeed */
+	struct rte_memseg_list *msl;
+	uint64_t max_mem, max_mem_per_type;
+	unsigned int max_seglists_per_type;
+	unsigned int n_memtypes, cur_type;
+
+	/* no-huge does not need this at all */
+	if (internal_config.no_hugetlbfs)
+		return 0;
+
+	/*
+	 * figuring out amount of memory we're going to have is a long and very
+	 * involved process. the basic element we're operating with is a memory
+	 * type, defined as a combination of NUMA node ID and page size (so that
+	 * e.g. 2 sockets with 2 page sizes yield 4 memory types in total).
+	 *
+	 * deciding amount of memory going towards each memory type is a
+	 * balancing act between maximum segments per type, maximum memory per
+	 * type, and number of detected NUMA nodes. the goal is to make sure
+	 * each memory type gets at least one memseg list.
+	 *
+	 * the total amount of memory is limited by RTE_MAX_MEM_MB value.
+	 *
+	 * the total amount of memory per type is limited by either
+	 * RTE_MAX_MEM_MB_PER_TYPE, or by RTE_MAX_MEM_MB divided by the number
+	 * of detected NUMA nodes. additionally, maximum number of segments per
+	 * type is also limited by RTE_MAX_MEMSEG_PER_TYPE. this is because for
+	 * smaller page sizes, it can take hundreds of thousands of segments to
+	 * reach the above specified per-type memory limits.
+	 *
+	 * additionally, each type may have multiple memseg lists associated
+	 * with it, each limited by either RTE_MAX_MEM_MB_PER_LIST for bigger
+	 * page sizes, or RTE_MAX_MEMSEG_PER_LIST segments for smaller ones.
+	 *
+	 * the number of memseg lists per type is decided based on the above
+	 * limits, and also taking number of detected NUMA nodes, to make sure
+	 * that we don't run out of memseg lists before we populate all NUMA
+	 * nodes with memory.
+	 *
+	 * we do this in three stages. first, we collect the number of types.
+	 * then, we figure out memory constraints and populate the list of
+	 * would-be memseg lists. then, we go ahead and allocate the memseg
+	 * lists.
+	 */
+
+	/* create space for mem types */
+	n_memtypes = internal_config.num_hugepage_sizes * rte_socket_count();
+	memtypes = calloc(n_memtypes, sizeof(*memtypes));
+	if (memtypes == NULL) {
+		RTE_LOG(ERR, EAL, "Cannot allocate space for memory types\n");
+		return -1;
+	}
+
+	/* populate mem types */
+	cur_type = 0;
+	for (hpi_idx = 0; hpi_idx < (int) internal_config.num_hugepage_sizes;
+			hpi_idx++) {
+		struct hugepage_info *hpi;
+		uint64_t hugepage_sz;
+
+		hpi = &internal_config.hugepage_info[hpi_idx];
+		hugepage_sz = hpi->hugepage_sz;
+
+		for (i = 0; i < (int) rte_socket_count(); i++, cur_type++) {
+			int socket_id = rte_socket_id_by_idx(i);
+
+			memtypes[cur_type].page_sz = hugepage_sz;
+			memtypes[cur_type].socket_id = socket_id;
+
+			RTE_LOG(DEBUG, EAL, "Detected memory type: "
+				"socket_id:%u hugepage_sz:%" PRIu64 "\n",
+				socket_id, hugepage_sz);
+		}
+	}
+	/* number of memtypes could have been lower due to no NUMA support */
+	n_memtypes = cur_type;
+
+	/* set up limits for types */
+	max_mem = (uint64_t)RTE_MAX_MEM_MB << 20;
+	max_mem_per_type = RTE_MIN((uint64_t)RTE_MAX_MEM_MB_PER_TYPE << 20,
+			max_mem / n_memtypes);
+
+	/*
+	 * limit maximum number of segment lists per type to ensure there's
+	 * space for memseg lists for all NUMA nodes with all page sizes
+	 */
+	max_seglists_per_type = RTE_MAX_MEMSEG_LISTS / n_memtypes;
+
+	if (max_seglists_per_type == 0) {
+		RTE_LOG(ERR, EAL, "Cannot accommodate all memory types, please increase %s\n",
+			RTE_STR(CONFIG_RTE_MAX_MEMSEG_LISTS));
+		goto out;
+	}
+
+	/* go through all mem types and create segment lists */
+	msl_idx = 0;
+	for (cur_type = 0; cur_type < n_memtypes; cur_type++) {
+		unsigned int cur_seglist, n_seglists, n_segs;
+		unsigned int max_segs_per_type, max_segs_per_list;
+		struct memtype *type = &memtypes[cur_type];
+		uint64_t max_mem_per_list, pagesz;
+		int socket_id;
+
+		pagesz = type->page_sz;
+		socket_id = type->socket_id;
+
+		/*
+		 * we need to create segment lists for this type. we must take
+		 * into account the following things:
+		 *
+		 * 1. total amount of memory we can use for this memory type
+		 * 2. total amount of memory per memseg list allowed
+		 * 3. number of segments needed to fit the amount of memory
+		 * 4. number of segments allowed per type
+		 * 5. number of segments allowed per memseg list
+		 * 6. number of memseg lists we are allowed to take up
+		 */
+
+		/* calculate how much segments we will need in total */
+		max_segs_per_type = max_mem_per_type / pagesz;
+		/* limit number of segments to maximum allowed per type */
+		max_segs_per_type = RTE_MIN(max_segs_per_type,
+				(unsigned int)RTE_MAX_MEMSEG_PER_TYPE);
+		/* limit number of segments to maximum allowed per list */
+		max_segs_per_list = RTE_MIN(max_segs_per_type,
+				(unsigned int)RTE_MAX_MEMSEG_PER_LIST);
+
+		/* calculate how much memory we can have per segment list */
+		max_mem_per_list = RTE_MIN(max_segs_per_list * pagesz,
+				(uint64_t)RTE_MAX_MEM_MB_PER_LIST << 20);
+
+		/* calculate how many segments each segment list will have */
+		n_segs = RTE_MIN(max_segs_per_list, max_mem_per_list / pagesz);
+
+		/* calculate how many segment lists we can have */
+		n_seglists = RTE_MIN(max_segs_per_type / n_segs,
+				max_mem_per_type / max_mem_per_list);
+
+		/* limit number of segment lists according to our maximum */
+		n_seglists = RTE_MIN(n_seglists, max_seglists_per_type);
+
+		RTE_LOG(DEBUG, EAL, "Creating %i segment lists: "
+				"n_segs:%i socket_id:%i hugepage_sz:%" PRIu64 "\n",
+			n_seglists, n_segs, socket_id, pagesz);
+
+		/* create all segment lists */
+		for (cur_seglist = 0; cur_seglist < n_seglists; cur_seglist++) {
+			if (msl_idx >= RTE_MAX_MEMSEG_LISTS) {
+				RTE_LOG(ERR, EAL,
+					"No more space in memseg lists, please increase %s\n",
+					RTE_STR(CONFIG_RTE_MAX_MEMSEG_LISTS));
+				goto out;
+			}
+			msl = &mcfg->memsegs[msl_idx++];
+
+			if (memseg_list_alloc(msl, pagesz, n_segs,
+					socket_id, cur_seglist))
+				goto out;
+
+			if (memseg_list_reserve(msl)) {
+				RTE_LOG(ERR, EAL, "Cannot allocate VA space for memseg list\n");
+				goto out;
+			}
+		}
+	}
+	/* we're successful */
+	ret = 0;
+out:
+	free(memtypes);
+	return ret;
+}
+
+static int
+memseg_secondary_init(void)
+{
+	EAL_LOG_NOT_IMPLEMENTED();
+	return -1;
+}
+
+int
+rte_eal_memseg_init(void)
+{
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return memseg_primary_init();
+	return memseg_secondary_init();
+}
+
+static inline uint64_t
+get_socket_mem_size(int socket)
+{
+	uint64_t size = 0;
+	unsigned int i;
+
+	for (i = 0; i < internal_config.num_hugepage_sizes; i++) {
+		struct hugepage_info *hpi = &internal_config.hugepage_info[i];
+		size += hpi->hugepage_sz * hpi->num_pages[socket];
+	}
+
+	return size;
+}
+
+static int
+calc_num_pages_per_socket(uint64_t *memory,
+		struct hugepage_info *hp_info,
+		struct hugepage_info *hp_used,
+		unsigned int num_hp_info)
+{
+	unsigned int socket, j, i = 0;
+	unsigned int requested, available;
+	int total_num_pages = 0;
+	uint64_t remaining_mem, cur_mem;
+	uint64_t total_mem = internal_config.memory;
+
+	if (num_hp_info == 0)
+		return -1;
+
+	/* if specific memory amounts per socket weren't requested */
+	if (internal_config.force_sockets == 0) {
+		size_t total_size;
+		int cpu_per_socket[RTE_MAX_NUMA_NODES];
+		size_t default_size;
+		unsigned int lcore_id;
+
+		/* Compute number of cores per socket */
+		memset(cpu_per_socket, 0, sizeof(cpu_per_socket));
+		RTE_LCORE_FOREACH(lcore_id) {
+			cpu_per_socket[rte_lcore_to_socket_id(lcore_id)]++;
+		}
+
+		/*
+		 * Automatically spread requested memory amongst detected
+		 * sockets according to number of cores from cpu mask present
+		 * on each socket.
+		 */
+		total_size = internal_config.memory;
+		for (socket = 0; socket < RTE_MAX_NUMA_NODES && total_size != 0;
+				socket++) {
+
+			/* Set memory amount per socket */
+			default_size = internal_config.memory *
+				cpu_per_socket[socket] / rte_lcore_count();
+
+			/* Limit to maximum available memory on socket */
+			default_size = RTE_MIN(
+				default_size, get_socket_mem_size(socket));
+
+			/* Update sizes */
+			memory[socket] = default_size;
+			total_size -= default_size;
+		}
+
+		/*
+		 * If some memory is remaining, try to allocate it by getting
+		 * all available memory from sockets, one after the other.
+		 */
+		for (socket = 0; socket < RTE_MAX_NUMA_NODES && total_size != 0;
+				socket++) {
+			/* take whatever is available */
+			default_size = RTE_MIN(
+				get_socket_mem_size(socket) - memory[socket],
+				total_size);
+
+			/* Update sizes */
+			memory[socket] += default_size;
+			total_size -= default_size;
+		}
+	}
+
+	for (socket = 0; socket < RTE_MAX_NUMA_NODES && total_mem != 0;
+			socket++) {
+		/* skips if the memory on specific socket wasn't requested */
+		for (i = 0; i < num_hp_info && memory[socket] != 0; i++) {
+			strncpy(hp_used[i].hugedir, hp_info[i].hugedir,
+				sizeof(hp_used[i].hugedir));
+			hp_used[i].num_pages[socket] = RTE_MIN(
+					memory[socket] / hp_info[i].hugepage_sz,
+					hp_info[i].num_pages[socket]);
+
+			cur_mem = hp_used[i].num_pages[socket] *
+					hp_used[i].hugepage_sz;
+
+			memory[socket] -= cur_mem;
+			total_mem -= cur_mem;
+
+			total_num_pages += hp_used[i].num_pages[socket];
+
+			/* check if we have met all memory requests */
+			if (memory[socket] == 0)
+				break;
+
+			/* Check if we have any more pages left at this size,
+			 * if so, move on to next size.
+			 */
+			if (hp_used[i].num_pages[socket] ==
+					hp_info[i].num_pages[socket])
+				continue;
+
+			/* At this point we know that there are more pages
+			 * available that are bigger than the memory we want,
+			 * so lets see if we can get enough from other page
+			 * sizes.
+			 */
+			remaining_mem = 0;
+			for (j = i+1; j < num_hp_info; j++)
+				remaining_mem += hp_info[j].hugepage_sz *
+				hp_info[j].num_pages[socket];
+
+			/* Is there enough other memory?
+			 * If not, allocate another page and quit.
+			 */
+			if (remaining_mem < memory[socket]) {
+				cur_mem = RTE_MIN(
+					memory[socket], hp_info[i].hugepage_sz);
+				memory[socket] -= cur_mem;
+				total_mem -= cur_mem;
+				hp_used[i].num_pages[socket]++;
+				total_num_pages++;
+				break; /* we are done with this socket*/
+			}
+		}
+		/* if we didn't satisfy all memory requirements per socket */
+		if (memory[socket] > 0 &&
+				internal_config.socket_mem[socket] != 0) {
+			/* to prevent icc errors */
+			requested = (unsigned int)(
+				internal_config.socket_mem[socket] / 0x100000);
+			available = requested -
+				((unsigned int)(memory[socket] / 0x100000));
+			RTE_LOG(ERR, EAL, "Not enough memory available on "
+				"socket %u! Requested: %uMB, available: %uMB\n",
+				socket, requested, available);
+			return -1;
+		}
+	}
+
+	/* if we didn't satisfy total memory requirements */
+	if (total_mem > 0) {
+		requested = (unsigned int) (internal_config.memory / 0x100000);
+		available = requested - (unsigned int) (total_mem / 0x100000);
+		RTE_LOG(ERR, EAL, "Not enough memory available! "
+			"Requested: %uMB, available: %uMB\n",
+			requested, available);
+		return -1;
+	}
+	return total_num_pages;
+}
+
+/* Limit is checked by validator itself, nothing left to analyze.*/
+static int
+limits_callback(int socket_id, size_t cur_limit, size_t new_len)
+{
+	RTE_SET_USED(socket_id);
+	RTE_SET_USED(cur_limit);
+	RTE_SET_USED(new_len);
+	return -1;
+}
+
+static int
+eal_hugepage_init(void)
+{
+	struct hugepage_info used_hp[MAX_HUGEPAGE_SIZES];
+	uint64_t memory[RTE_MAX_NUMA_NODES];
+	int hp_sz_idx, socket_id;
+
+	memset(used_hp, 0, sizeof(used_hp));
+
+	for (hp_sz_idx = 0;
+			hp_sz_idx < (int) internal_config.num_hugepage_sizes;
+			hp_sz_idx++) {
+		/* also initialize used_hp hugepage sizes in used_hp */
+		struct hugepage_info *hpi;
+		hpi = &internal_config.hugepage_info[hp_sz_idx];
+		used_hp[hp_sz_idx].hugepage_sz = hpi->hugepage_sz;
+	}
+
+	/* make a copy of socket_mem, needed for balanced allocation. */
+	for (socket_id = 0; socket_id < RTE_MAX_NUMA_NODES; socket_id++)
+		memory[socket_id] = internal_config.socket_mem[socket_id];
+
+	/* calculate final number of pages */
+	if (calc_num_pages_per_socket(memory,
+			internal_config.hugepage_info, used_hp,
+			internal_config.num_hugepage_sizes) < 0)
+		return -1;
+
+	for (hp_sz_idx = 0;
+			hp_sz_idx < (int)internal_config.num_hugepage_sizes;
+			hp_sz_idx++) {
+		for (socket_id = 0; socket_id < RTE_MAX_NUMA_NODES;
+				socket_id++) {
+			struct rte_memseg **pages;
+			struct hugepage_info *hpi = &used_hp[hp_sz_idx];
+			unsigned int num_pages = hpi->num_pages[socket_id];
+			unsigned int num_pages_alloc;
+
+			if (num_pages == 0)
+				continue;
+
+			RTE_LOG(DEBUG, EAL,
+				"Allocating %u pages of size %" PRIu64 "M on socket %i\n",
+				num_pages, hpi->hugepage_sz >> 20, socket_id);
+
+			/* we may not be able to allocate all pages in one go,
+			 * because we break up our memory map into multiple
+			 * memseg lists. therefore, try allocating multiple
+			 * times and see if we can get the desired number of
+			 * pages from multiple allocations.
+			 */
+
+			num_pages_alloc = 0;
+			do {
+				int i, cur_pages, needed;
+
+				needed = num_pages - num_pages_alloc;
+
+				pages = malloc(sizeof(*pages) * needed);
+
+				/* do not request exact number of pages */
+				cur_pages = eal_memalloc_alloc_seg_bulk(pages,
+						needed, hpi->hugepage_sz,
+						socket_id, false);
+				if (cur_pages <= 0) {
+					free(pages);
+					return -1;
+				}
+
+				/* mark preallocated pages as unfreeable */
+				for (i = 0; i < cur_pages; i++) {
+					struct rte_memseg *ms = pages[i];
+					ms->flags |=
+						RTE_MEMSEG_FLAG_DO_NOT_FREE;
+				}
+				free(pages);
+
+				num_pages_alloc += cur_pages;
+			} while (num_pages_alloc != num_pages);
+		}
+	}
+	/* if socket limits were specified, set them */
+	if (internal_config.force_socket_limits) {
+		unsigned int i;
+		for (i = 0; i < RTE_MAX_NUMA_NODES; i++) {
+			uint64_t limit = internal_config.socket_limit[i];
+			if (limit == 0)
+				continue;
+			if (rte_mem_alloc_validator_register("socket-limit",
+					limits_callback, i, limit))
+				RTE_LOG(ERR, EAL, "Failed to register socket "
+					"limits validator callback\n");
+		}
+	}
+	return 0;
+}
+
+static int
+eal_nohuge_init(void)
+{
+	struct rte_mem_config *mcfg;
+	struct rte_memseg_list *msl;
+	int n_segs, cur_seg;
+	uint64_t page_sz;
+	void *addr;
+	struct rte_fbarray *arr;
+	struct rte_memseg *ms;
+
+	mcfg = rte_eal_get_configuration()->mem_config;
+
+	/* nohuge mode is legacy mode */
+	internal_config.legacy_mem = 1;
+
+	/* create a memseg list */
+	msl = &mcfg->memsegs[0];
+
+	page_sz = RTE_PGSIZE_4K;
+	n_segs = internal_config.memory / page_sz;
+
+	if (rte_fbarray_init(&msl->memseg_arr, "nohugemem", n_segs,
+		sizeof(struct rte_memseg))) {
+		RTE_LOG(ERR, EAL, "Cannot allocate memseg list\n");
+		return -1;
+	}
+
+	addr = eal_mem_alloc(internal_config.memory, 0);
+	if (addr == NULL) {
+		RTE_LOG(ERR, EAL, "Cannot allocate %zu bytes",
+		internal_config.memory);
+		return -1;
+	}
+
+	msl->base_va = addr;
+	msl->page_sz = page_sz;
+	msl->socket_id = 0;
+	msl->len = internal_config.memory;
+	msl->heap = 1;
+
+	/* populate memsegs. each memseg is one page long */
+	for (cur_seg = 0; cur_seg < n_segs; cur_seg++) {
+		arr = &msl->memseg_arr;
+
+		ms = rte_fbarray_get(arr, cur_seg);
+		ms->iova = RTE_BAD_IOVA;
+		ms->addr = addr;
+		ms->hugepage_sz = page_sz;
+		ms->socket_id = 0;
+		ms->len = page_sz;
+
+		rte_fbarray_set_used(arr, cur_seg);
+
+		addr = RTE_PTR_ADD(addr, (size_t)page_sz);
+	}
+
+	if (mcfg->dma_maskbits &&
+		rte_mem_check_dma_mask_thread_unsafe(mcfg->dma_maskbits)) {
+		RTE_LOG(ERR, EAL,
+			"%s(): couldn't allocate memory due to IOVA "
+			"exceeding limits of current DMA mask.\n", __func__);
+		return -1;
+	}
+
+	return 0;
+}
+
+int
+rte_eal_hugepage_init(void)
+{
+	return internal_config.no_hugetlbfs ?
+		eal_nohuge_init() : eal_hugepage_init();
+}
+
+int
+rte_eal_hugepage_attach(void)
+{
+	EAL_LOG_NOT_IMPLEMENTED();
+	return -1;
+}
lib/librte_eal/windows/eal_mp.c new file +103 -0
new file mode 100644
index 000000000..16a5e8ba0
@@ -0,0 +1,103 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2020 Dmitry Kozlyuk
+ */
+
+/**
+ * @file Multiprocess support stubs
+ *
+ * Stubs must log an error until implemented. If success is required
+ * for non-multiprocess operation, stub must log a warning and a comment
+ * must document what requires success emulation.
+ */
+
+#include <rte_eal.h>
+#include <rte_errno.h>
+
+#include "eal_private.h"
+#include "eal_windows.h"
+#include "malloc_mp.h"
+
+void
+rte_mp_channel_cleanup(void)
+{
+	EAL_LOG_NOT_IMPLEMENTED();
+}
+
+int
+rte_mp_action_register(const char *name, rte_mp_t action)
+{
+	RTE_SET_USED(name);
+	RTE_SET_USED(action);
+	EAL_LOG_NOT_IMPLEMENTED();
+	return -1;
+}
+
+void
+rte_mp_action_unregister(const char *name)
+{
+	RTE_SET_USED(name);
+	EAL_LOG_NOT_IMPLEMENTED();
+}
+
+int
+rte_mp_sendmsg(struct rte_mp_msg *msg)
+{
+	RTE_SET_USED(msg);
+	EAL_LOG_NOT_IMPLEMENTED();
+	return -1;
+}
+
+int
+rte_mp_request_sync(struct rte_mp_msg *req, struct rte_mp_reply *reply,
+	const struct timespec *ts)
+{
+	RTE_SET_USED(req);
+	RTE_SET_USED(reply);
+	RTE_SET_USED(ts);
+	EAL_LOG_NOT_IMPLEMENTED();
+	return -1;
+}
+
+int
+rte_mp_request_async(struct rte_mp_msg *req, const struct timespec *ts,
+		rte_mp_async_reply_t clb)
+{
+	RTE_SET_USED(req);
+	RTE_SET_USED(ts);
+	RTE_SET_USED(clb);
+	EAL_LOG_NOT_IMPLEMENTED();
+	return -1;
+}
+
+int
+rte_mp_reply(struct rte_mp_msg *msg, const char *peer)
+{
+	RTE_SET_USED(msg);
+	RTE_SET_USED(peer);
+	EAL_LOG_NOT_IMPLEMENTED();
+	return -1;
+}
+
+int
+register_mp_requests(void)
+{
+	/* Non-stub function succeeds if multi-process is not supported. */
+	EAL_LOG_STUB();
+	return 0;
+}
+
+int
+request_to_primary(struct malloc_mp_req *req)
+{
+	RTE_SET_USED(req);
+	EAL_LOG_NOT_IMPLEMENTED();
+	return -1;
+}
+
+int
+request_sync(void)
+{
+	/* Common memory allocator depends on this function success. */
+	EAL_LOG_STUB();
+	return 0;
+}
lib/librte_eal/windows/eal_windows.h +23 -0
index b202a1aa5..083ab8b93 100644
@@ -9,8 +9,24 @@
  * @file Facilities private to Windows EAL
  */
 
+#include <rte_errno.h>
 #include <rte_windows.h>
 
+/**
+ * Log current function as not implemented and set rte_errno.
+ */
+#define EAL_LOG_NOT_IMPLEMENTED() \
+	do { \
+		RTE_LOG(DEBUG, EAL, "%s() is not implemented\n", __func__); \
+		rte_errno = ENOTSUP; \
+	} while (0)
+
+/**
+ * Log current function as a stub.
+ */
+#define EAL_LOG_STUB() \
+	RTE_LOG(DEBUG, EAL, "Windows: %s() is a stub\n", __func__)
+
 /**
  * Create a map of processors and cores on the system.
  */
@@ -36,6 +52,13 @@ int eal_thread_create(pthread_t *thread);
  */
 unsigned int eal_socket_numa_node(unsigned int socket_id);
 
+/**
+ * Open virt2phys driver interface device.
+ *
+ * @return 0 on success, (-1) on failure.
+ */
+int eal_mem_virt2iova_init(void);
+
 /**
  * Locate Win32 memory management routines in system libraries.
  *
lib/librte_eal/windows/include/meson.build +1 -0
index 5fb1962ac..b3534b025 100644
@@ -5,5 +5,6 @@ includes += include_directories('.')
 
 headers += files(
         'rte_os.h',
+        'rte_virt2phys.h',
         'rte_windows.h',
 )
lib/librte_eal/windows/include/rte_os.h +4 -0
index 510e39e03..62805a307 100644
@@ -36,6 +36,10 @@ extern "C" {
 
 #define strncasecmp(s1, s2, count)        _strnicmp(s1, s2, count)
 
+#define open _open
+#define close _close
+#define unlink _unlink
+
 /* cpu_set macros implementation */
 #define RTE_CPU_AND(dst, src1, src2) CPU_AND(dst, src1, src2)
 #define RTE_CPU_OR(dst, src1, src2) CPU_OR(dst, src1, src2)
lib/librte_eal/windows/include/rte_virt2phys.h new file +34 -0
new file mode 100644
index 000000000..4bb2b4aaf
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2020 Dmitry Kozlyuk
+ */
+
+/**
+ * @file virt2phys driver interface
+ */
+
+/**
+ * Driver device interface GUID {539c2135-793a-4926-afec-d3a1b61bbc8a}.
+ */
+DEFINE_GUID(GUID_DEVINTERFACE_VIRT2PHYS,
+	0x539c2135, 0x793a, 0x4926,
+	0xaf, 0xec, 0xd3, 0xa1, 0xb6, 0x1b, 0xbc, 0x8a);
+
+/**
+ * Driver device type for IO control codes.
+ */
+#define VIRT2PHYS_DEVTYPE 0x8000
+
+/**
+ * Translate a valid non-paged virtual address to a physical address.
+ *
+ * Note: A physical address zero (0) is reported if input address
+ * is paged out or not mapped. However, if input is a valid mapping
+ * of I/O port 0x0000, output is also zero. There is no way
+ * to distinguish between these cases by return value only.
+ *
+ * Input: a non-paged virtual address (PVOID).
+ *
+ * Output: the corresponding physical address (LARGE_INTEGER).
+ */
+#define IOCTL_VIRT2PHYS_TRANSLATE CTL_CODE( \
+	VIRT2PHYS_DEVTYPE, 0x800, METHOD_BUFFERED, FILE_ANY_ACCESS)
lib/librte_eal/windows/include/rte_windows.h +2 -0
index ed6e4c148..899ed7d87 100644
@@ -23,6 +23,8 @@
 
 #include <basetsd.h>
 #include <psapi.h>
+#include <setupapi.h>
+#include <winioctl.h>
 
 /* Have GUIDs defined. */
 #ifndef INITGUID
lib/librte_eal/windows/include/unistd.h +3 -0
index 757b7f3c5..6b33005b2 100644
@@ -9,4 +9,7 @@
  * as Microsoft libc does not contain unistd.h. This may be removed
  * in future releases.
  */
+
+#include <io.h>
+
 #endif /* _UNISTD_H_ */
lib/librte_eal/windows/meson.build +2 -0
index 81d3ee095..2f4fa91a9 100644
@@ -8,7 +8,9 @@ sources += files(
 	'eal_debug.c',
 	'eal_hugepages.c',
 	'eal_lcore.c',
+	'eal_memalloc.c',
 	'eal_memory.c',
+	'eal_mp.c',
 	'eal_thread.c',
 	'getopt.c',
 )
Signature · 1 line
2.25.1
#32 lore raw

Re: [dpdk-dev] [PATCH v2 10/10] eal/windows: implement basic memory management

[snip]

Quoted 29 lines
Quoted 29 lines
lib/librte_eal/windows/eal_memalloc.c new file +29 -0
+static int
+alloc_seg(struct rte_memseg *ms, void *requested_addr, int socket_id,
+	struct hugepage_info *hi)
+{
+	HANDLE current_process;
+	unsigned int numa_node;
+	size_t alloc_sz;
+	void *addr;
+	rte_iova_t iova = RTE_BAD_IOVA;
+	PSAPI_WORKING_SET_EX_INFORMATION info;
+	PSAPI_WORKING_SET_EX_BLOCK *page;
+
+	if (ms->len > 0) {
+		/* If a segment is already allocated as needed, return it. */
+		if ((ms->addr == requested_addr) &&
+			(ms->socket_id == socket_id) &&
+			(ms->hugepage_sz == hi->hugepage_sz)) {
+			return 0;
+		}
+
+		/* Bugcheck, should not happen. */
+		RTE_LOG(DEBUG, EAL, "Attempted to reallocate segment %p "
+			"(size %zu) on socket %d", ms->addr,
+			ms->len, ms->socket_id);
+		return -1;
+	}
+
+	current_process = GetCurrentProcess();
+	numa_node = eal_socket_numa_node(socket_id);
Quoted 8 lines

lib/librte_eal/windows/eal_memalloc.c line 68

Quoted 4 lines
lib/librte_eal/windows/eal_memalloc.c new file +4 -0
+	alloc_sz = hi->hugepage_sz;
+
+	if (requested_addr == NULL) {
+		/* Request a new chunk of memory and enforce address hint. */  

Does requested_addr being NULL means that no hint was provided? It also looks like eal_mem_alloc_socket ignores the address hint anyway and just calls VirtualAllocExNuma with NULL for lpAddress. Maybe remove the second part of the comment.

Correct, also see below.

Quoted 12 lines

lib/librte_eal/windows/eal_memalloc.c line 76

Quoted 9 lines
lib/librte_eal/windows/eal_memalloc.c new file +8 -0
+		addr = eal_mem_alloc_socket(alloc_sz, socket_id);
+		if (addr == NULL) {
+			RTE_LOG(DEBUG, EAL, "Cannot allocate %zu bytes "
+				"on socket %d\n", alloc_sz, socket_id);
+			return -1;
+		}
+
+		if (addr != requested_addr) {  

requested_addr is NULL on this branch and we confirmed with the previous 'if' that addr is not NULL. Should this branch be removed, since requested_addr is NULL, so there is no hint provided?

Good catch, this check doesn't belong here. In fact, apart from an allocation failure, address hint might not be respected iff the hint is not aligned to hugepage boundary, which would be an allocator bug (use of an incorrect MSL).

Quoted 6 lines
Quoted 6 lines
lib/librte_eal/windows/eal_memalloc.c new file +6 -0
+			RTE_LOG(DEBUG, EAL, "Address hint %p not respected, "
+				"got %p\n", requested_addr, addr);
+			goto error;
+		}
+	} else {
+		/* Requested address is already reserved, commit memory. */
Quoted 8 lines

lib/librte_eal/windows/eal_memalloc.c line 87

Quoted 5 lines
lib/librte_eal/windows/eal_memalloc.c new file +5 -0
+		addr = eal_mem_commit(requested_addr, alloc_sz, socket_id);
+		if (addr == NULL) {
+			RTE_LOG(DEBUG, EAL, "Cannot commit reserved memory %p "
+				"(size %zu)\n", requested_addr, alloc_sz);
+			goto error;  

Execution jumps to 'error' with an invalid addr, so it will try to call eal_mem_decommit with NULL as parameter. Instead of 'goto error', maybe we should return here.

Will fix in v3.

I'd like to also clarify the following. On Windows, eal_mem_commit() is not
atomic: it may first split a reserved region, then commit it. If splitting succeeds and commit fails, the reserved region remains split into parts. eal_mem_decommit() does not coalesce these parts intentionally: they're hugepage-sized, so the next eal_mem_commit() on the same region just doesn't have to split it.

Quoted 56 lines
Quoted 56 lines
lib/librte_eal/windows/eal_memalloc.c new file +55 -0
+		}
+	}
+
+	/* Force OS to allocate a physical page and select a NUMA node.
+	 * Hugepages are not pageable in Windows, so there's no race
+	 * for physical address.
+	 */
+	*(volatile int *)addr = *(volatile int *)addr;
+
+	/* Only try to obtain IOVA if it's available, so that applications
+	 * that do not need IOVA can use this allocator.
+	 */
+	if (rte_eal_using_phys_addrs()) {
+		iova = rte_mem_virt2iova(addr);
+		if (iova == RTE_BAD_IOVA) {
+			RTE_LOG(DEBUG, EAL,
+				"Cannot get IOVA of allocated segment\n");
+			goto error;
+		}
+	}
+
+	/* Only "Ex" function can handle hugepages. */
+	info.VirtualAddress = addr;
+	if (!QueryWorkingSetEx(current_process, &info, sizeof(info))) {
+		RTE_LOG_WIN32_ERR("QueryWorkingSetEx()");
+		goto error;
+	}
+
+	page = &info.VirtualAttributes;
+	if (!page->Valid || !page->LargePage) {
+		RTE_LOG(DEBUG, EAL, "Got regular page instead of hugepage\n");
+		goto error;
+	}
+	if (page->Node != numa_node) {
+		RTE_LOG(DEBUG, EAL,
+			"NUMA node hint %u (socket %d) not respected, got %u\n",
+			numa_node, socket_id, page->Node);
+		goto error;
+	}
+
+	ms->addr = addr;
+	ms->hugepage_sz = hi->hugepage_sz;
+	ms->len = alloc_sz;
+	ms->nchannel = rte_memory_get_nchannel();
+	ms->nrank = rte_memory_get_nrank();
+	ms->iova = iova;
+	ms->socket_id = socket_id;
+
+	return 0;
+
+error:
+	/* Only jump here when `addr` and `alloc_sz` are valid. */
+	eal_mem_decommit(addr, alloc_sz);
+	return -1;
+}

[snip]

Signature · 1 line
Dmitry Kozlyuk
#33 lore raw

v3 cover Windows basic memory management

In reply to Dmitry Kozlyuk

Note: no changes in cover letter since v2.

This patchset implements basic MM with the following features:

* Hugepages are dynamically allocated in user-mode.
* Only 2MB hugepages are supported.
* IOVA is always PA, obtained through kernel-mode driver.
* No 32-bit support (presumably not demanded).
* Ni multi-process support (it is forcefully disabled).
* No-huge mode for testing with IOVA unavailable.

The first commit introduces a new kernel-mode driver, virt2phys. It translates user-mode virtual addresses into physical addresses. On Windows community call 2020-04-01 it was decided this driver can be used for now, later netUIO may pick up its code/interface or not.

New EAL public functions for memory mapping are introduced to mitigate OS differences in DPDK libraries and applications:

* rte_mem_map
* rte_mem_unmap
* rte_mem_lock

To support common MM routines, internal wrappers for low-level memory reservation and file management are introduced. These changes affect Linux and FreeBSD EAL. Shared code is placed unded /unix/ subdirectory (suggested by Thomas).

Also, entire <sys/queue.h> is imported from FreeBSD, replacing existing partial import. There is already a license exception for this file.

Windows MM duplicates quite a lot of code from Linux EAL:

* eal_memalloc_alloc_seg_bulk
* eal_memalloc_free_seg_bulk
* calc_num_pages_per_socket
* rte_eal_hugepage_init

Perhaps this should be left as-is until Windows MM evolves into having some specific requirements for these parts.

Notes on checkpatch warnings:

* No space after comma / no space before closing parent in macros--- definitely a false-positive, unclear how to suppress this.

* Issues from imported BSD code---probably should be ignored?

* Checkpatch is not run against dpdk-kmods (Windows drivers).

---

v3:

* Fix Linux build on and aarch64 and 32-bit x86 (reported by CI).
* Fix logic and error handling while allocating segments.
* Fix Unix rte_mem_map(): return NULL on failure.
* Fix some checkpatch.sh issues:
* Do not return positive errno, use DWORD for GetLastError().
* Make dpdk-kmods source files non-executable.
* Improve GSG for Windows Server (suggested by Ranjit Menon).

v2:

* Rebase on ToT. Move all new code shared between Linux and FreeBSD to /unix/ subdirectory, also factor out some existing code there.
* Improve description of Clang issue with rte_page_sizes on Windows. Restore -fstrict-enum for EAL. Check running, not target compiler.
* Use EAL prefix for private facilities instead if RTE.
* Improve documentation comments for new functions.
* Remove co-installer for virt2phys. Add a typecast for clarity.
* Document virt2phys in user guide, improve its own README.
* Explicitly and forcefully disable multi-process.

Dmitry Kozlyuk (9):
eal/windows: do not expose private EAL facilities eal/windows: improve CPU and NUMA node detection eal/windows: initialize hugepage info
eal: introduce internal wrappers for file operations
eal: introduce memory management wrappers
eal: extract common code for memseg list initialization eal/windows: fix rte_page_sizes with Clang on Windows eal/windows: replace sys/queue.h with a complete one from FreeBSD eal/windows: implement basic memory management

 config/meson.build                            |   12 +-
 doc/guides/windows_gsg/build_dpdk.rst         |   20 -
 doc/guides/windows_gsg/index.rst              |    1 +
 doc/guides/windows_gsg/run_apps.rst           |   84 ++
 lib/librte_eal/common/eal_common_fbarray.c    |   57 +-
 lib/librte_eal/common/eal_common_memory.c     |  104 +-
 lib/librte_eal/common/eal_private.h           |  134 +-
 lib/librte_eal/common/malloc_heap.c           |    1 +
 lib/librte_eal/common/meson.build             |    9 +
 lib/librte_eal/freebsd/eal_memory.c           |   55 +-
 lib/librte_eal/include/rte_memory.h           |   74 ++
 lib/librte_eal/linux/eal_memory.c             |   68 +-
 lib/librte_eal/meson.build                    |    4 +
 lib/librte_eal/rte_eal_exports.def            |  119 ++
 lib/librte_eal/rte_eal_version.map            |    4 +
 lib/librte_eal/unix/eal.c                     |   47 +
 lib/librte_eal/unix/eal_memory.c              |  113 ++
 lib/librte_eal/unix/meson.build               |    7 +
 lib/librte_eal/windows/eal.c                  |  160 +++
 lib/librte_eal/windows/eal_hugepages.c        |  108 ++
 lib/librte_eal/windows/eal_lcore.c            |  187 ++-
 lib/librte_eal/windows/eal_memalloc.c         |  418 ++++++
 lib/librte_eal/windows/eal_memory.c           | 1141 +++++++++++++++++
 lib/librte_eal/windows/eal_mp.c               |  103 ++
 lib/librte_eal/windows/eal_thread.c           |    1 +
 lib/librte_eal/windows/eal_windows.h          |  129 ++
 lib/librte_eal/windows/include/meson.build    |    2 +
 lib/librte_eal/windows/include/pthread.h      |    2 +
 lib/librte_eal/windows/include/rte_os.h       |   48 +-
 .../windows/include/rte_virt2phys.h           |   34 +
 lib/librte_eal/windows/include/rte_windows.h  |   43 +
 lib/librte_eal/windows/include/sys/queue.h    |  663 +++++++++-
 lib/librte_eal/windows/include/unistd.h       |    3 +
 lib/librte_eal/windows/meson.build            |    6 +
 34 files changed, 3611 insertions(+), 350 deletions(-)
 create mode 100644 doc/guides/windows_gsg/run_apps.rst
 create mode 100644 lib/librte_eal/unix/eal.c
 create mode 100644 lib/librte_eal/unix/eal_memory.c
 create mode 100644 lib/librte_eal/unix/meson.build
 create mode 100644 lib/librte_eal/windows/eal_hugepages.c
 create mode 100644 lib/librte_eal/windows/eal_memalloc.c
 create mode 100644 lib/librte_eal/windows/eal_memory.c
 create mode 100644 lib/librte_eal/windows/eal_mp.c
 create mode 100644 lib/librte_eal/windows/eal_windows.h
 create mode 100644 lib/librte_eal/windows/include/rte_virt2phys.h
 create mode 100644 lib/librte_eal/windows/include/rte_windows.h
Signature · 1 line
2.25.1
#34 lore raw

PATCH v3 1/1 virt2phys: virtual to physical address translator for Windows

This driver supports Windows EAL memory management by translating current process virtual addresses to physical addresses (IOVA). Standalone virt2phys allows using DPDK without PMD and provides a reference implementation.

---
 windows/README.rst                          | 103 +++++++++
 windows/virt2phys/virt2phys.c               | 129 +++++++++++
 windows/virt2phys/virt2phys.h               |  34 +++
 windows/virt2phys/virt2phys.inf             |  64 ++++++
 windows/virt2phys/virt2phys.sln             |  27 +++
 windows/virt2phys/virt2phys.vcxproj         | 228 ++++++++++++++++++++
 windows/virt2phys/virt2phys.vcxproj.filters |  36 ++++
 7 files changed, 621 insertions(+)
 create mode 100644 windows/README.rst
 create mode 100644 windows/virt2phys/virt2phys.c
 create mode 100644 windows/virt2phys/virt2phys.h
 create mode 100644 windows/virt2phys/virt2phys.inf
 create mode 100644 windows/virt2phys/virt2phys.sln
 create mode 100644 windows/virt2phys/virt2phys.vcxproj
 create mode 100644 windows/virt2phys/virt2phys.vcxproj.filters
windows/README.rst new file +103 -0
new file mode 100644
index 0000000..45a1d80
@@ -0,0 +1,103 @@
+Developing Windows Drivers
+==========================
+
+Prerequisites
+-------------
+
+Building Windows Drivers is only possible on Windows.
+
+1. Visual Studio 2019 Community or Professional Edition
+2. Windows Driver Kit (WDK) for Windows 10, version 1903
+
+Follow the official instructions to obtain all of the above:
+https://docs.microsoft.com/en-us/windows-hardware/drivers/download-the-wdk
+
+
+Build the Drivers
+-----------------
+
+Build from Visual Studio
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Open a solution (``*.sln``) with Visual Studio and build it (Ctrl+Shift+B).
+
+
+Build from Command-Line
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Run *Developer Command Prompt for VS 2019* from the Start menu.
+
+Navigate to the solution directory (with ``*.sln``), then run:
+
+.. code-block:: console
+
+    msbuild
+
+To build a particular combination of configuration and platform:
+
+.. code-block:: console
+
+    msbuild -p:Configuration=Debug;Platform=x64
+
+
+Install the Drivers
+-------------------
+
+Disable Driver Signature Enforcement
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+By default Windows prohibits installing and loading drivers without `digital
+signature`_ obtained from Microsoft. For development signature enforcement may
+be disabled as follows.
+
+In Elevated Command Prompt (from this point, sufficient privileges are
+assumed):
+
+.. code-block:: console
+
+    bcdedit -set loadoptions DISABLE_INTEGRITY_CHECKS
+    bcdedit -set TESTSIGNING ON
+    shutdown -r -t 0
+
+Upon reboot, an overlay message should appear on the desktop informing
+that Windows is in test mode, which means it allows loading unsigned drivers.
+
+.. _digital signature: https://docs.microsoft.com/en-us/windows-hardware/drivers/install/driver-signing
+
+Install, List, and Remove Drivers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Driver package is by default located in a subdirectory of its source tree,
+e.g. ``x64\Debug\virt2phys\virt2phys`` (note two levels of ``virt2phys``).
+
+To install the driver and bind associated devices to it:
+
+.. code-block:: console
+
+    pnputil /add-driver x64\Debug\virt2phys\virt2phys\virt2phys.inf /install
+
+A graphical confirmation to load an unsigned driver will still appear.
+
+On Windows Server additional steps are required if the driver uses a custom
+setup class:
+
+1. From "Device Manager", "Action" menu, select "Add legacy hardware".
+2. It will launch the "Add Hardware Wizard". Click "Next".
+3. Select second option "Install the hardware that I manually select
+   from a list (Advanced)".
+4. On the next screen, locate the driver device class.
+5. Select it, and click "Next".
+6. The previously installed drivers will now be installed for
+   the appropriate devices (software devices will be created).
+
+To list installed drivers:
+
+.. code-block:: console
+
+    pnputil /enum-drivers
+
+To remove the driver package and to uninstall its devices:
+
+.. code-block:: console
+
+    pnputil /delete-driver oem2.inf /uninstall
windows/virt2phys/virt2phys.c new file +129 -0

129 lines not on this page · raw message

windows/virt2phys/virt2phys.h new file +34 -0
new file mode 100644
index 0000000..4bb2b4a
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2020 Dmitry Kozlyuk
+ */
+
+/**
+ * @file virt2phys driver interface
+ */
+
+/**
+ * Driver device interface GUID {539c2135-793a-4926-afec-d3a1b61bbc8a}.
+ */
+DEFINE_GUID(GUID_DEVINTERFACE_VIRT2PHYS,
+	0x539c2135, 0x793a, 0x4926,
+	0xaf, 0xec, 0xd3, 0xa1, 0xb6, 0x1b, 0xbc, 0x8a);
+
+/**
+ * Driver device type for IO control codes.
+ */
+#define VIRT2PHYS_DEVTYPE 0x8000
+
+/**
+ * Translate a valid non-paged virtual address to a physical address.
+ *
+ * Note: A physical address zero (0) is reported if input address
+ * is paged out or not mapped. However, if input is a valid mapping
+ * of I/O port 0x0000, output is also zero. There is no way
+ * to distinguish between these cases by return value only.
+ *
+ * Input: a non-paged virtual address (PVOID).
+ *
+ * Output: the corresponding physical address (LARGE_INTEGER).
+ */
+#define IOCTL_VIRT2PHYS_TRANSLATE CTL_CODE( \
+	VIRT2PHYS_DEVTYPE, 0x800, METHOD_BUFFERED, FILE_ANY_ACCESS)
windows/virt2phys/virt2phys.inf new file +64 -0

64 lines not on this page · raw message

windows/virt2phys/virt2phys.sln new file +27 -0
new file mode 100644
index 0000000..0f5ecdc
@@ -0,0 +1,27 @@
+
+Microsoft Visual Studio Solution File, Format Version 12.00
+# Visual Studio Version 16
+VisualStudioVersion = 16.0.29613.14
+MinimumVisualStudioVersion = 10.0.40219.1
+Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "virt2phys", "virt2phys.vcxproj", "{0EEF826B-9391-43A8-A722-BDD6F6115137}"
+EndProject
+Global
+	GlobalSection(SolutionConfigurationPlatforms) = preSolution
+		Debug|x64 = Debug|x64
+		Release|x64 = Release|x64
+	EndGlobalSection
+	GlobalSection(ProjectConfigurationPlatforms) = postSolution
+		{0EEF826B-9391-43A8-A722-BDD6F6115137}.Debug|x64.ActiveCfg = Debug|x64
+		{0EEF826B-9391-43A8-A722-BDD6F6115137}.Debug|x64.Build.0 = Debug|x64
+		{0EEF826B-9391-43A8-A722-BDD6F6115137}.Debug|x64.Deploy.0 = Debug|x64
+		{0EEF826B-9391-43A8-A722-BDD6F6115137}.Release|x64.ActiveCfg = Release|x64
+		{0EEF826B-9391-43A8-A722-BDD6F6115137}.Release|x64.Build.0 = Release|x64
+		{0EEF826B-9391-43A8-A722-BDD6F6115137}.Release|x64.Deploy.0 = Release|x64
+	EndGlobalSection
+	GlobalSection(SolutionProperties) = preSolution
+		HideSolutionNode = FALSE
+	EndGlobalSection
+	GlobalSection(ExtensibilityGlobals) = postSolution
+		SolutionGuid = {845012FB-4471-4A12-A1C4-FF7E05C40E8E}
+	EndGlobalSection
+EndGlobal
windows/virt2phys/virt2phys.vcxproj new file +228 -0

229 lines not on this page · raw message

windows/virt2phys/virt2phys.vcxproj.filters new file +36 -0
new file mode 100644
index 0000000..0fe65fc
@@ -0,0 +1,36 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+  <ItemGroup>
+    <Filter Include="Source Files">
+      <UniqueIdentifier>{4FC737F1-C7A5-4376-A066-2A32D752A2FF}</UniqueIdentifier>
+      <Extensions>cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx</Extensions>
+    </Filter>
+    <Filter Include="Header Files">
+      <UniqueIdentifier>{93995380-89BD-4b04-88EB-625FBE52EBFB}</UniqueIdentifier>
+      <Extensions>h;hpp;hxx;hm;inl;inc;xsd</Extensions>
+    </Filter>
+    <Filter Include="Resource Files">
+      <UniqueIdentifier>{67DA6AB6-F800-4c08-8B7A-83BB121AAD01}</UniqueIdentifier>
+      <Extensions>rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;png;wav;mfcribbon-ms</Extensions>
+    </Filter>
+    <Filter Include="Driver Files">
+      <UniqueIdentifier>{8E41214B-6785-4CFE-B992-037D68949A14}</UniqueIdentifier>
+      <Extensions>inf;inv;inx;mof;mc;</Extensions>
+    </Filter>
+  </ItemGroup>
+  <ItemGroup>
+    <Inf Include="virt2phys.inf">
+      <Filter>Driver Files</Filter>
+    </Inf>
+  </ItemGroup>
+  <ItemGroup>
+    <ClInclude Include="virt2phys.h">
+      <Filter>Header Files</Filter>
+    </ClInclude>
+  </ItemGroup>
+  <ItemGroup>
+    <ClCompile Include="virt2phys.c">
+      <Filter>Source Files</Filter>
+    </ClCompile>
+  </ItemGroup>
+</Project>
Signature · 1 line
2.25.1
Reviewed-by Acked-by on v3 1/1
#35 lore raw

Re: [dpdk-dev] [PATCH v3 1/1] virt2phys: virtual to physical address translator for Windows

Quoted 16 lines · Dmitry Kozlyuk

This driver supports Windows EAL memory management by translating current process virtual addresses to physical addresses (IOVA). Standalone virt2phys allows using DPDK without PMD and provides a reference implementation.

---
  windows/README.rst                          | 103 +++++++++
  windows/virt2phys/virt2phys.c               | 129 +++++++++++
  windows/virt2phys/virt2phys.h               |  34 +++
  windows/virt2phys/virt2phys.inf             |  64 ++++++
  windows/virt2phys/virt2phys.sln             |  27 +++
  windows/virt2phys/virt2phys.vcxproj         | 228 ++++++++++++++++++++
  windows/virt2phys/virt2phys.vcxproj.filters |  36 ++++
  7 files changed, 621 insertions(+)
Quoted 8 lines

create mode 100644 windows/README.rst
create mode 100644 windows/virt2phys/virt2phys.c create mode 100644 windows/virt2phys/virt2phys.h create mode 100644 windows/virt2phys/virt2phys.inf create mode 100644 windows/virt2phys/virt2phys.sln create mode 100644 windows/virt2phys/virt2phys.vcxproj create mode 100644 windows/virt2phys/virt2phys.vcxproj.filters

#36 lore raw

Re: [dpdk-dev] [PATCH v3 1/1] virt2phys: virtual to physical address translator for Windows

15/04/2020 01:35, Ranjit Menon:

Quoted 11 lines · Dmitry Kozlyuk
Quoted 7 lines

This driver supports Windows EAL memory management by translating current process virtual addresses to physical addresses (IOVA). Standalone virt2phys allows using DPDK without PMD and provides a reference implementation.

Applied in dpdk-kmods, thanks.

This is the very first kernel module in this repository:

	http://git.dpdk.org/dpdk-kmods/
#37 lore raw

Re: [dpdk-dev] [PATCH v3 1/1] virt2phys: virtual to physical address translator for Windows

In reply to Dmitry Kozlyuk

Hi Dmitry,
When trying to test the series under Linux (RedHat 7.4, x86_64-native-linuxapp-gcc) I get compilation errors. I am using make. It seems that there are calls from common which are not compiled under linux.

x86_64-native-linuxapp-gcc/lib/librte_eal.a(eal_common_memory.o): In function `eal_get_virtual_area':
eal_common_memory.c:(.text+0x27e): undefined reference to `eal_mem_reserve'

[dpdk-branches]$ find lib -name '*.c' | xargs grep eal_mem_reserve
lib/librte_eal/common/eal_common_memory.c: mapped_addr = eal_mem_reserve(
lib/librte_eal/windows/eal_memory.c:eal_mem_reserve(void *requested_addr, size_t size, lib/librte_eal/unix/eal_memory.c:eal_mem_reserve(void *requested_addr, size_t size,

Does it succeed for you?
If not - can you please resend the series such that it compiles under Linux with Makefile?

Quoted 687 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of Dmitry Kozlyuk
Sent: Tuesday, April 14, 2020 10:44 PM
To: [email protected]
Cc: Dmitry Malloy (MESHCHANINOV) <[email protected]>; Narcisa Ana Maria Vasile <[email protected]>; Fady Bader <[email protected]>; Tal Shnaiderman <[email protected]>; Dmitry Kozlyuk <[email protected]>; Ranjit Menon <[email protected]>
Subject: [dpdk-dev] [PATCH v3 1/1] virt2phys: virtual to physical address translator for Windows

This driver supports Windows EAL memory management by translating current process virtual addresses to physical addresses (IOVA). Standalone virt2phys allows using DPDK without PMD and provides a reference implementation.

---
 windows/README.rst                          | 103 +++++++++
 windows/virt2phys/virt2phys.c               | 129 +++++++++++
 windows/virt2phys/virt2phys.h               |  34 +++
 windows/virt2phys/virt2phys.inf             |  64 ++++++
 windows/virt2phys/virt2phys.sln             |  27 +++
 windows/virt2phys/virt2phys.vcxproj         | 228 ++++++++++++++++++++
 windows/virt2phys/virt2phys.vcxproj.filters |  36 ++++
 7 files changed, 621 insertions(+)
 create mode 100644 windows/README.rst
 create mode 100644 windows/virt2phys/virt2phys.c  create mode 100644

windows/virt2phys/virt2phys.h create mode 100644 windows/virt2phys/virt2phys.inf create mode 100644 windows/virt2phys/virt2phys.sln create mode 100644 windows/virt2phys/virt2phys.vcxproj

 create mode 100644 windows/virt2phys/virt2phys.vcxproj.filters
windows/README.rst new file mode renamed from windows/README.rst +0 -0

100644 index 0000000..45a1d80

windows/README.rst new file +13 -0
@@ -0,0 +1,103 @@
+Developing Windows Drivers
+==========================
+
+Prerequisites
+-------------
+
+Building Windows Drivers is only possible on Windows.
+
+1. Visual Studio 2019 Community or Professional Edition 2. Windows
+Driver Kit (WDK) for Windows 10, version 1903
+
+Follow the official instructions to obtain all of the above:
+https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs

. +microsoft.com%2Fen-us%2Fwindows-hardware%2Fdrivers%2Fdownload- the-wdk&a +mp;data=02%7C01%7Cophirmu%40mellanox.com%7C86da431f2b904dca3 d8208d7e0ac +4cd0%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C637224902 939680479&am +p;sdata=RSZXmo%2FewOpl9VCCg4BIgYUFvpreZzqX1X%2Bcl9vNRjU%3D&a mp;reserved
+=0

windows/README.rst new file +50 -0
+
+
+Build the Drivers
+-----------------
+
+Build from Visual Studio
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Open a solution (``*.sln``) with Visual Studio and build it (Ctrl+Shift+B).
+
+
+Build from Command-Line
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Run *Developer Command Prompt for VS 2019* from the Start menu.
+
+Navigate to the solution directory (with ``*.sln``), then run:
+
+.. code-block:: console
+
+    msbuild
+
+To build a particular combination of configuration and platform:
+
+.. code-block:: console
+
+    msbuild -p:Configuration=Debug;Platform=x64
+
+
+Install the Drivers
+-------------------
+
+Disable Driver Signature Enforcement
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+By default Windows prohibits installing and loading drivers without
+`digital signature`_ obtained from Microsoft. For development signature
+enforcement may be disabled as follows.
+
+In Elevated Command Prompt (from this point, sufficient privileges are
+assumed):
+
+.. code-block:: console
+
+    bcdedit -set loadoptions DISABLE_INTEGRITY_CHECKS
+    bcdedit -set TESTSIGNING ON
+    shutdown -r -t 0
+
+Upon reboot, an overlay message should appear on the desktop informing
+that Windows is in test mode, which means it allows loading unsigned

drivers.

windows/README.rst new file +3 -0
+
+.. _digital signature:
+https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs

.
+microsoft.com%2Fen-us%2Fwindows- hardware%2Fdrivers%2Finstall%2Fdriver-s +igning&amp;data=02%7C01%7Cophirmu%40mellanox.com%7C86da431f2b 904dca3d82 +08d7e0ac4cd0%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C63 72249029396 +85478&amp;sdata=tLVdijiQ56xwSDW0zVsVE3TnIpuMPTLd9DBtS3Bnf4Q%3 D&amp;rese

windows/README.rst new file +7 -0
+rved=0
+
+Install, List, and Remove Drivers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Driver package is by default located in a subdirectory of its source
+tree, e.g. ``x64\Debug\virt2phys\virt2phys`` (note two levels of

``virt2phys``).

windows/README.rst new file +33 -0
+
+To install the driver and bind associated devices to it:
+
+.. code-block:: console
+
+    pnputil /add-driver x64\Debug\virt2phys\virt2phys\virt2phys.inf
+ /install
+
+A graphical confirmation to load an unsigned driver will still appear.
+
+On Windows Server additional steps are required if the driver uses a
+custom setup class:
+
+1. From "Device Manager", "Action" menu, select "Add legacy hardware".
+2. It will launch the "Add Hardware Wizard". Click "Next".
+3. Select second option "Install the hardware that I manually select
+   from a list (Advanced)".
+4. On the next screen, locate the driver device class.
+5. Select it, and click "Next".
+6. The previously installed drivers will now be installed for
+   the appropriate devices (software devices will be created).
+
+To list installed drivers:
+
+.. code-block:: console
+
+    pnputil /enum-drivers
+
+To remove the driver package and to uninstall its devices:
+
+.. code-block:: console
+
+    pnputil /delete-driver oem2.inf /uninstall
windows/virt2phys/virt2phys.c new file +14 -0
new file mode 100644 index 0000000..e157e9c
@@ -0,0 +1,129 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Dmitry Kozlyuk
+ */
+
+#include <ntddk.h>
+#include <wdf.h>
+#include <wdmsec.h>
+#include <initguid.h>
+
+#include "virt2phys.h"
+
+DRIVER_INITIALIZE DriverEntry;
+EVT_WDF_DRIVER_DEVICE_ADD virt2phys_driver_EvtDeviceAdd;
+EVT_WDF_IO_IN_CALLER_CONTEXT

virt2phys_device_EvtIoInCallerContext;

windows/virt2phys/virt2phys.c new file +4 -0
+
+NTSTATUS
+DriverEntry(
+	IN PDRIVER_OBJECT driver_object, IN PUNICODE_STRING

registry_path) {

windows/virt2phys/virt2phys.c new file +7 -0
+	WDF_DRIVER_CONFIG config;
+	WDF_OBJECT_ATTRIBUTES attributes;
+	NTSTATUS status;
+
+	PAGED_CODE();
+
+	WDF_DRIVER_CONFIG_INIT(&config,

virt2phys_driver_EvtDeviceAdd);

windows/virt2phys/virt2phys.c new file +100 -0
+	WDF_OBJECT_ATTRIBUTES_INIT(&attributes);
+	status = WdfDriverCreate(
+			driver_object, registry_path,
+			&attributes, &config, WDF_NO_HANDLE);
+	if (!NT_SUCCESS(status)) {
+		KdPrint(("WdfDriverCreate() failed, status=%08x\n", status));
+	}
+
+	return status;
+}
+
+_Use_decl_annotations_
+NTSTATUS
+virt2phys_driver_EvtDeviceAdd(
+	WDFDRIVER driver, PWDFDEVICE_INIT init) {
+	WDF_OBJECT_ATTRIBUTES attributes;
+	WDFDEVICE device;
+	NTSTATUS status;
+
+	UNREFERENCED_PARAMETER(driver);
+
+	PAGED_CODE();
+
+	WdfDeviceInitSetIoType(
+		init, WdfDeviceIoNeither);
+	WdfDeviceInitSetIoInCallerContextCallback(
+		init, virt2phys_device_EvtIoInCallerContext);
+
+	WDF_OBJECT_ATTRIBUTES_INIT(&attributes);
+
+	status = WdfDeviceCreate(&init, &attributes, &device);
+	if (!NT_SUCCESS(status)) {
+		KdPrint(("WdfDeviceCreate() failed, status=%08x\n", status));
+		return status;
+	}
+
+	status = WdfDeviceCreateDeviceInterface(
+			device, &GUID_DEVINTERFACE_VIRT2PHYS, NULL);
+	if (!NT_SUCCESS(status)) {
+		KdPrint(("WdfDeviceCreateDeviceInterface() failed, "
+			"status=%08x\n", status));
+		return status;
+	}
+
+	return STATUS_SUCCESS;
+}
+
+_Use_decl_annotations_
+VOID
+virt2phys_device_EvtIoInCallerContext(
+	IN WDFDEVICE device, IN WDFREQUEST request) {
+	WDF_REQUEST_PARAMETERS params;
+	ULONG code;
+	PVOID *virt;
+	PHYSICAL_ADDRESS *phys;
+	size_t size;
+	NTSTATUS status;
+
+	UNREFERENCED_PARAMETER(device);
+
+	PAGED_CODE();
+
+	WDF_REQUEST_PARAMETERS_INIT(&params);
+	WdfRequestGetParameters(request, &params);
+
+	if (params.Type != WdfRequestTypeDeviceControl) {
+		KdPrint(("bogus request type=%u\n", params.Type));
+		WdfRequestComplete(request, STATUS_NOT_SUPPORTED);
+		return;
+	}
+
+	code = params.Parameters.DeviceIoControl.IoControlCode;
+	if (code != IOCTL_VIRT2PHYS_TRANSLATE) {
+		KdPrint(("bogus IO control code=%lu\n", code));
+		WdfRequestComplete(request, STATUS_NOT_SUPPORTED);
+		return;
+	}
+
+	status = WdfRequestRetrieveInputBuffer(
+			request, sizeof(*virt), (PVOID *)&virt, &size);
+	if (!NT_SUCCESS(status)) {
+		KdPrint(("WdfRequestRetrieveInputBuffer() failed, "
+			"status=%08x\n", status));
+		WdfRequestComplete(request, status);
+		return;
+	}
+
+	status = WdfRequestRetrieveOutputBuffer(
+		request, sizeof(*phys), (PVOID *)&phys, &size);
+	if (!NT_SUCCESS(status)) {
+		KdPrint(("WdfRequestRetrieveOutputBuffer() failed, "
+			"status=%08x\n", status));
+		WdfRequestComplete(request, status);
+		return;
+	}
+
+	*phys = MmGetPhysicalAddress(*virt);
+
+	WdfRequestCompleteWithInformation(
+		request, STATUS_SUCCESS, sizeof(*phys)); }
windows/virt2phys/virt2phys.h new file +34 -0
new file mode 100644 index 0000000..4bb2b4a
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2020 Dmitry Kozlyuk
+ */
+
+/**
+ * @file virt2phys driver interface
+ */
+
+/**
+ * Driver device interface GUID {539c2135-793a-4926-afec-d3a1b61bbc8a}.
+ */
+DEFINE_GUID(GUID_DEVINTERFACE_VIRT2PHYS,
+	0x539c2135, 0x793a, 0x4926,
+	0xaf, 0xec, 0xd3, 0xa1, 0xb6, 0x1b, 0xbc, 0x8a);
+
+/**
+ * Driver device type for IO control codes.
+ */
+#define VIRT2PHYS_DEVTYPE 0x8000
+
+/**
+ * Translate a valid non-paged virtual address to a physical address.
+ *
+ * Note: A physical address zero (0) is reported if input address
+ * is paged out or not mapped. However, if input is a valid mapping
+ * of I/O port 0x0000, output is also zero. There is no way
+ * to distinguish between these cases by return value only.
+ *
+ * Input: a non-paged virtual address (PVOID).
+ *
+ * Output: the corresponding physical address (LARGE_INTEGER).
+ */
+#define IOCTL_VIRT2PHYS_TRANSLATE CTL_CODE( \
+	VIRT2PHYS_DEVTYPE, 0x800, METHOD_BUFFERED,

FILE_ANY_ACCESS)

(quoted diff) +0 -0

b/windows/virt2phys/virt2phys.inf new file mode 100644 index 0000000..e35765e

windows/virt2phys/virt2phys.inf new file +48 -0
@@ -0,0 +1,64 @@
+; SPDX-License-Identifier: BSD-3-Clause ; Copyright (c) 2020 Dmitry
+Kozlyuk
+
+[Version]
+Signature = "$WINDOWS NT$"
+Class = %ClassName%
+ClassGuid = {78A1C341-4539-11d3-B88D-00C04FAD5171}
+Provider = %ManufacturerName%
+CatalogFile = virt2phys.cat
+DriverVer =
+
+[DestinationDirs]
+DefaultDestDir = 12
+
+; ================= Class section =====================
+
+[ClassInstall32]
+Addreg = virt2phys_ClassReg
+
+[virt2phys_ClassReg]
+HKR,,,0,%ClassName%
+HKR,,Icon,,-5
+
+[SourceDisksNames]
+1 = %DiskName%,,,""
+
+[SourceDisksFiles]
+virt2phys.sys  = 1,,
+
+;*****************************************
+; Install Section
+;*****************************************
+
+[Manufacturer]
+%ManufacturerName%=Standard,NT$ARCH$
+
+[Standard.NT$ARCH$]
+%virt2phys.DeviceDesc%=virt2phys_Device, Root\virt2phys
+
+[virt2phys_Device.NT]
+CopyFiles = Drivers_Dir
+
+[Drivers_Dir]
+virt2phys.sys
+
+;-------------- Service installation
+[virt2phys_Device.NT.Services]
+AddService = virt2phys,%SPSVCINST_ASSOCSERVICE%,

virt2phys_Service_Inst

windows/virt2phys/virt2phys.inf new file +16 -0
+
+; -------------- virt2phys driver install sections
+[virt2phys_Service_Inst]
+DisplayName    = %virt2phys.SVCDESC%
+ServiceType    = 1 ; SERVICE_KERNEL_DRIVER
+StartType      = 3 ; SERVICE_DEMAND_START
+ErrorControl   = 1 ; SERVICE_ERROR_NORMAL
+ServiceBinary  = %12%\virt2phys.sys
+
+[Strings]
+SPSVCINST_ASSOCSERVICE = 0x00000002
+ManufacturerName = "Dmitry Kozlyuk"
+ClassName = "Kernel bypass"
+DiskName = "virt2phys Installation Disk"
+virt2phys.DeviceDesc = "Virtual to physical address translator"
+virt2phys.SVCDESC = "virt2phys Service"
(quoted diff) +0 -0

b/windows/virt2phys/virt2phys.sln new file mode 100644 index 0000000..0f5ecdc

windows/virt2phys/virt2phys.sln new file +1 -0
@@ -0,0 +1,27 @@
++Microsoft Visual Studio Solution File, Format Version 12.00+# Visual Studio

Version 16+VisualStudioVersion = 16.0.29613.14+MinimumVisualStudioVersion = 10.0.40219.1+Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "virt2phys", "virt2phys.vcxproj", "{0EEF826B-9391-43A8-A722- BDD6F6115137}"+EndProject+Global+

	GlobalSection(SolutionConfigurationPlatforms) = preSolution+
	Debug|x64 = Debug|x64+		Release|x64 = Release|x64+
	EndGlobalSection+
	GlobalSection(ProjectConfigurationPlatforms) = postSolution+
	{0EEF826B-9391-43A8-A722-BDD6F6115137}.Debug|x64.ActiveCfg =
Debug|x64+		{0EEF826B-9391-43A8-A722-
BDD6F6115137}.Debug|x64.Build.0 = Debug|x64+		{0EEF826B-
9391-43A8-A722-BDD6F6115137}.Debug|x64.Deploy.0 = Debug|x64+
	{0EEF826B-9391-43A8-A722-BDD6F6115137}.Release|x64.ActiveCfg

= Release|x64+ {0EEF826B-9391-43A8-A722-
BDD6F6115137}.Release|x64.Build.0 = Release|x64+ {0EEF826B-
9391-43A8-A722-BDD6F6115137}.Release|x64.Deploy.0 = Release|x64+
EndGlobalSection+ GlobalSection(SolutionProperties) =
preSolution+ HideSolutionNode = FALSE+ EndGlobalSection+

	GlobalSection(ExtensibilityGlobals) = postSolution+
	SolutionGuid = {845012FB-4471-4A12-A1C4-FF7E05C40E8E}+
	EndGlobalSection+EndGlobaldiff --git

a/windows/virt2phys/virt2phys.vcxproj
b/windows/virt2phys/virt2phys.vcxproj
new file mode 100644
index 0000000..fa51916

windows/virt2phys/virt2phys.vcxproj new file +1 -0
@@ -0,0 +1,228 @@
+<?xml version="1.0" encoding="utf-8"?>+<Project DefaultTargets="Build"

ToolsVersion="12.0" xmlns="https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2 Fschemas.microsoft.com%2Fdeveloper%2Fmsbuild%2F2003&amp;data=02% 7C01%7Cophirmu%40mellanox.com%7C86da431f2b904dca3d8208d7e0ac4c d0%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C637224902939 685478&amp;sdata=N7HPDma1qrXV6sobOS0tblhsgaEUePD8lXqhnyXD1sU% 3D&amp;reserved=0">+ <ItemGroup Label="ProjectConfigurations">+ <ProjectConfiguration Include="Debug|Win32">+
<Configuration>Debug</Configuration>+ <Platform>Win32</Platform>+
</ProjectConfiguration>+ <ProjectConfiguration
Include="Release|Win32">+ <Configuration>Release</Configuration>+
<Platform>Win32</Platform>+ </ProjectConfiguration>+
<ProjectConfiguration Include="Debug|x64">+
<Configuration>Debug</Configuration>+ <Platform>x64</Platform>+
</ProjectConfiguration>+ <ProjectConfiguration Include="Release|x64">+
<Configuration>Release</Configuration>+ <Platform>x64</Platform>+
</ProjectConfiguration>+ <ProjectConfiguration Include="Debug|ARM">+
<Configuration>Debug</Configuration>+ <Platform>ARM</Platform>+
</ProjectConfiguration>+ <ProjectConfiguration Include="Release|ARM">+
<Configuration>Release</Configuration>+ <Platform>ARM</Platform>+
</ProjectConfiguration>+ <ProjectConfiguration
Include="Debug|ARM64">+ <Configuration>Debug</Configuration>+
<Platform>ARM64</Platform>+ </ProjectConfiguration>+
<ProjectConfiguration Include="Release|ARM64">+ <Configuration>Release</Configuration>+
<Platform>ARM64</Platform>+ </ProjectConfiguration>+ </ItemGroup>+

<ItemGroup>+    <ClCompile Include="virt2phys.c" />+  </ItemGroup>+
<ItemGroup>+    <ClInclude Include="virt2phys.h" />+  </ItemGroup>+
<ItemGroup>+    <Inf Include="virt2phys.inf" />+  </ItemGroup>+

<PropertyGroup Label="Globals">+ <ProjectGuid>{0EEF826B-9391-43A8-
A722-BDD6F6115137}</ProjectGuid>+ <TemplateGuid>{497e31cb-056b-
4f31-abb8-447fd55ee5a5}</TemplateGuid>+ <TargetFrameworkVersion>v4.5</TargetFrameworkVersion>+ <MinimumVisualStudioVersion>12.0</MinimumVisualStudioVersion>+
<Configuration>Debug</Configuration>+ <Platform
Condition="'$(Platform)' == ''">Win32</Platform>+ <RootNamespace>virt2phys</RootNamespace>+ </PropertyGroup>+ <Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />+ <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'"
Label="Configuration">+ <TargetVersion>Windows10</TargetVersion>+
<UseDebugLibraries>true</UseDebugLibraries>+ <PlatformToolset>WindowsKernelModeDriver10.0</PlatformToolset>+ <ConfigurationType>Driver</ConfigurationType>+ <DriverType>KMDF</DriverType>+ <DriverTargetPlatform>Universal</DriverTargetPlatform>+ </PropertyGroup>+ <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'"
Label="Configuration">+ <TargetVersion>Windows10</TargetVersion>+
<UseDebugLibraries>false</UseDebugLibraries>+ <PlatformToolset>WindowsKernelModeDriver10.0</PlatformToolset>+ <ConfigurationType>Driver</ConfigurationType>+ <DriverType>KMDF</DriverType>+ <DriverTargetPlatform>Universal</DriverTargetPlatform>+ </PropertyGroup>+ <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'"
Label="Configuration">+ <TargetVersion>Windows10</TargetVersion>+
<UseDebugLibraries>true</UseDebugLibraries>+ <PlatformToolset>WindowsKernelModeDriver10.0</PlatformToolset>+ <ConfigurationType>Driver</ConfigurationType>+ <DriverType>KMDF</DriverType>+ <DriverTargetPlatform>Universal</DriverTargetPlatform>+ </PropertyGroup>+ <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'"
Label="Configuration">+ <TargetVersion>Windows10</TargetVersion>+
<UseDebugLibraries>false</UseDebugLibraries>+ <PlatformToolset>WindowsKernelModeDriver10.0</PlatformToolset>+ <ConfigurationType>Driver</ConfigurationType>+ <DriverType>KMDF</DriverType>+ <DriverTargetPlatform>Universal</DriverTargetPlatform>+ </PropertyGroup>+ <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|ARM'"
Label="Configuration">+ <TargetVersion>Windows10</TargetVersion>+
<UseDebugLibraries>true</UseDebugLibraries>+ <PlatformToolset>WindowsKernelModeDriver10.0</PlatformToolset>+ <ConfigurationType>Driver</ConfigurationType>+ <DriverType>KMDF</DriverType>+ <DriverTargetPlatform>Universal</DriverTargetPlatform>+ </PropertyGroup>+ <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|ARM'"
Label="Configuration">+ <TargetVersion>Windows10</TargetVersion>+
<UseDebugLibraries>false</UseDebugLibraries>+ <PlatformToolset>WindowsKernelModeDriver10.0</PlatformToolset>+ <ConfigurationType>Driver</ConfigurationType>+ <DriverType>KMDF</DriverType>+ <DriverTargetPlatform>Universal</DriverTargetPlatform>+ </PropertyGroup>+ <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|ARM64'"
Label="Configuration">+ <TargetVersion>Windows10</TargetVersion>+
<UseDebugLibraries>true</UseDebugLibraries>+ <PlatformToolset>WindowsKernelModeDriver10.0</PlatformToolset>+ <ConfigurationType>Driver</ConfigurationType>+ <DriverType>KMDF</DriverType>+ <DriverTargetPlatform>Universal</DriverTargetPlatform>+ </PropertyGroup>+ <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|ARM64'"
Label="Configuration">+ <TargetVersion>Windows10</TargetVersion>+
<UseDebugLibraries>false</UseDebugLibraries>+ <PlatformToolset>WindowsKernelModeDriver10.0</PlatformToolset>+ <ConfigurationType>Driver</ConfigurationType>+ <DriverType>KMDF</DriverType>+ <DriverTargetPlatform>Universal</DriverTargetPlatform>+ </PropertyGroup>+ <Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />+ <ImportGroup Label="ExtensionSettings">+ </ImportGroup>+ <ImportGroup
Label="PropertySheets">+ <Import
Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />+ </ImportGroup>+ <PropertyGroup Label="UserMacros" />+ <PropertyGroup />+ <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">+ <DebuggerFlavor>DbgengKernelDebugger</DebuggerFlavor>+ </PropertyGroup>+ <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">+ <DebuggerFlavor>DbgengKernelDebugger</DebuggerFlavor>+ </PropertyGroup>+ <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">+ <DebuggerFlavor>DbgengKernelDebugger</DebuggerFlavor>+ </PropertyGroup>+ <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">+ <DebuggerFlavor>DbgengKernelDebugger</DebuggerFlavor>+ </PropertyGroup>+ <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|ARM'">+ <DebuggerFlavor>DbgengKernelDebugger</DebuggerFlavor>+ </PropertyGroup>+ <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|ARM'">+ <DebuggerFlavor>DbgengKernelDebugger</DebuggerFlavor>+ </PropertyGroup>+ <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|ARM64'">+ <DebuggerFlavor>DbgengKernelDebugger</DebuggerFlavor>+ </PropertyGroup>+ <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|ARM64'">+ <DebuggerFlavor>DbgengKernelDebugger</DebuggerFlavor>+ </PropertyGroup>+ <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">+
<ClCompile>+ <WppEnabled>true</WppEnabled>+
<WppRecorderEnabled>true</WppRecorderEnabled>+ <WppScanConfigurationData Condition="'%(ClCompile.ScanConfigurationData)' == ''">trace.h</WppScanConfigurationData>+
<WppKernelMode>true</WppKernelMode>+ </ClCompile>+
</ItemDefinitionGroup>+ <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">+
<ClCompile>+ <WppEnabled>true</WppEnabled>+
<WppRecorderEnabled>true</WppRecorderEnabled>+ <WppScanConfigurationData Condition="'%(ClCompile.ScanConfigurationData)' == ''">trace.h</WppScanConfigurationData>+
<WppKernelMode>true</WppKernelMode>+ </ClCompile>+
</ItemDefinitionGroup>+ <ItemDefinitionGroup
Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">+ <ClCompile>+
<WppEnabled>false</WppEnabled>+ <WppRecorderEnabled>true</WppRecorderEnabled>+ <WppScanConfigurationData Condition="'%(ClCompile.ScanConfigurationData)' == ''">trace.h</WppScanConfigurationData>+
<WppKernelMode>true</WppKernelMode>+ </ClCompile>+ <Link>+
<AdditionalDependencies>$(DDK_LIB_PATH)wdmsec.lib;%(AdditionalDepend
encies)</AdditionalDependencies>+ </Link>+ <Inf>+
<TimeStamp>0.1</TimeStamp>+ </Inf>+ </ItemDefinitionGroup>+
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">+
<ClCompile>+ <WppEnabled>true</WppEnabled>+
<WppRecorderEnabled>true</WppRecorderEnabled>+ <WppScanConfigurationData Condition="'%(ClCompile.ScanConfigurationData)' == ''">trace.h</WppScanConfigurationData>+
<WppKernelMode>true</WppKernelMode>+ </ClCompile>+
</ItemDefinitionGroup>+ <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|ARM'">+
<ClCompile>+ <WppEnabled>true</WppEnabled>+
<WppRecorderEnabled>true</WppRecorderEnabled>+ <WppScanConfigurationData Condition="'%(ClCompile.ScanConfigurationData)' == ''">trace.h</WppScanConfigurationData>+
<WppKernelMode>true</WppKernelMode>+ </ClCompile>+
</ItemDefinitionGroup>+ <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|ARM'">+
<ClCompile>+ <WppEnabled>true</WppEnabled>+
<WppRecorderEnabled>true</WppRecorderEnabled>+ <WppScanConfigurationData Condition="'%(ClCompile.ScanConfigurationData)' == ''">trace.h</WppScanConfigurationData>+
<WppKernelMode>true</WppKernelMode>+ </ClCompile>+
</ItemDefinitionGroup>+ <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|ARM64'">+
<ClCompile>+ <WppEnabled>true</WppEnabled>+
<WppRecorderEnabled>true</WppRecorderEnabled>+ <WppScanConfigurationData Condition="'%(ClCompile.ScanConfigurationData)' == ''">trace.h</WppScanConfigurationData>+
<WppKernelMode>true</WppKernelMode>+ </ClCompile>+
</ItemDefinitionGroup>+ <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|ARM64'">+
<ClCompile>+ <WppEnabled>true</WppEnabled>+
<WppRecorderEnabled>true</WppRecorderEnabled>+ <WppScanConfigurationData Condition="'%(ClCompile.ScanConfigurationData)' == ''">trace.h</WppScanConfigurationData>+
<WppKernelMode>true</WppKernelMode>+ </ClCompile>+
</ItemDefinitionGroup>+ <ItemGroup>+ <FilesToPackage
Include="$(TargetPath)" />+ </ItemGroup>+ <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />+ <ImportGroup Label="ExtensionTargets">+ </ImportGroup>+</Project>
\ No newline at end of file

windows/virt2phys/virt2phys.vcxproj.filters new file +1 -0
new file mode 100644
index 0000000..0fe65fc
@@ -0,0 +1,36 @@
+<?xml version="1.0" encoding="utf-8"?>+<Project ToolsVersion="4.0"

xmlns="https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2 Fschemas.microsoft.com%2Fdeveloper%2Fmsbuild%2F2003&amp;data=02% 7C01%7Cophirmu%40mellanox.com%7C86da431f2b904dca3d8208d7e0ac4c d0%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C637224902939 685478&amp;sdata=N7HPDma1qrXV6sobOS0tblhsgaEUePD8lXqhnyXD1sU%
3D&amp;reserved=0">+ <ItemGroup>+ <Filter Include="Source Files">+
<UniqueIdentifier>{4FC737F1-C7A5-4376-A066- 2A32D752A2FF}</UniqueIdentifier>+ <Extensions>cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx</Extensions>+
</Filter>+ <Filter Include="Header Files">+
<UniqueIdentifier>{93995380-89BD-4b04-88EB- 625FBE52EBFB}</UniqueIdentifier>+
<Extensions>h;hpp;hxx;hm;inl;inc;xsd</Extensions>+ </Filter>+ <Filter
Include="Resource Files">+ <UniqueIdentifier>{67DA6AB6-F800-4c08-
8B7A-83BB121AAD01}</UniqueIdentifier>+ <Extensions>rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;pn
g;wav;mfcribbon-ms</Extensions>+ </Filter>+ <Filter Include="Driver
Files">+ <UniqueIdentifier>{8E41214B-6785-4CFE-B992-
037D68949A14}</UniqueIdentifier>+
<Extensions>inf;inv;inx;mof;mc;</Extensions>+ </Filter>+ </ItemGroup>+
<ItemGroup>+ <Inf Include="virt2phys.inf">+ <Filter>Driver
Files</Filter>+ </Inf>+ </ItemGroup>+ <ItemGroup>+ <ClInclude
Include="virt2phys.h">+ <Filter>Header Files</Filter>+ </ClInclude>+
</ItemGroup>+ <ItemGroup>+ <ClCompile Include="virt2phys.c">+
<Filter>Source Files</Filter>+ </ClCompile>+ </ItemGroup>+</Project>--
2.25.1

#38 lore raw

PATCH v3 2/10 eal/windows: do not expose private EAL facilities

In reply to Dmitry Kozlyuk

The goal of rte_os.h is to mitigate OS differences for EAL users. In Windows EAL, rte_os.h did excessive things:

1. It included platform SDK headers (windows.h, etc). Those files are huge, require specific inclusion order, and are generally unused by the code including rte_os.h. Declarations from platform SDK may break otherwise platform-independent code, e.g. min, max, ERROR.

2. It included pthread.h, which is clearly not always required.

3. It defined functions private to Windows EAL.

Reorganize Windows EAL includes in the following way:

1. Create rte_windows.h to properly import Windows-specific facilities. Primary users are bus drivers, tests, and external applications.

2. Remove platform SDK includes from rte_os.h to prevent breaking otherwise portable code by including rte_os.h on Windows. Copy necessary definitions to avoid including those headers.

3. Remove pthread.h include from rte_os.h.

4. Move declarations private to Windows EAL into eal_windows.h.

Fixes 428eb983f5f7 ("eal: add OS specific header file")
---
 lib/librte_eal/windows/eal.c                 |  2 +
 lib/librte_eal/windows/eal_lcore.c           |  2 +
 lib/librte_eal/windows/eal_thread.c          |  1 +
 lib/librte_eal/windows/eal_windows.h         | 29 +++++++++++++
 lib/librte_eal/windows/include/meson.build   |  1 +
 lib/librte_eal/windows/include/pthread.h     |  2 +
 lib/librte_eal/windows/include/rte_os.h      | 44 ++++++--------------
 lib/librte_eal/windows/include/rte_windows.h | 41 ++++++++++++++++++
 8 files changed, 91 insertions(+), 31 deletions(-)
 create mode 100644 lib/librte_eal/windows/eal_windows.h
 create mode 100644 lib/librte_eal/windows/include/rte_windows.h
lib/librte_eal/windows/eal.c +2 -0
index e4b50df3b..2cf7a04ef 100644
@@ -18,6 +18,8 @@
 #include <eal_options.h>
 #include <eal_private.h>
 
+#include "eal_windows.h"
+
  /* Allow the application to print its usage message too if set */
 static rte_usage_hook_t	rte_application_usage_hook;
 
lib/librte_eal/windows/eal_lcore.c +2 -0
index b3a6c63af..82ee45413 100644
@@ -2,12 +2,14 @@
  * Copyright(c) 2019 Intel Corporation
  */
 
+#include <pthread.h>
 #include <stdint.h>
 
 #include <rte_common.h>
 
 #include "eal_private.h"
 #include "eal_thread.h"
+#include "eal_windows.h"
 
 /* global data structure that contains the CPU map */
 static struct _wcpu_map {
lib/librte_eal/windows/eal_thread.c +1 -0
index 9e4bbaa08..e149199a6 100644
@@ -14,6 +14,7 @@
 #include <eal_thread.h>
 
 #include "eal_private.h"
+#include "eal_windows.h"
 
 RTE_DEFINE_PER_LCORE(unsigned int, _lcore_id) = LCORE_ID_ANY;
 RTE_DEFINE_PER_LCORE(unsigned int, _socket_id) = (unsigned int)SOCKET_ID_ANY;
lib/librte_eal/windows/eal_windows.h new file +29 -0
new file mode 100644
index 000000000..fadd676b2
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2020 Dmitry Kozlyuk
+ */
+
+#ifndef _EAL_WINDOWS_H_
+#define _EAL_WINDOWS_H_
+
+/**
+ * @file Facilities private to Windows EAL
+ */
+
+#include <rte_windows.h>
+
+/**
+ * Create a map of processors and cores on the system.
+ */
+void eal_create_cpu_map(void);
+
+/**
+ * Create a thread.
+ *
+ * @param thread
+ *   The location to store the thread id if successful.
+ * @return
+ *   0 for success, -1 if the thread is not created.
+ */
+int eal_thread_create(pthread_t *thread);
+
+#endif /* _EAL_WINDOWS_H_ */
lib/librte_eal/windows/include/meson.build +1 -0
index 7d18dd52f..5fb1962ac 100644
@@ -5,4 +5,5 @@ includes += include_directories('.')
 
 headers += files(
         'rte_os.h',
+        'rte_windows.h',
 )
lib/librte_eal/windows/include/pthread.h +2 -0
index b9dd18e56..cfd53f0b8 100644
@@ -5,6 +5,8 @@
 #ifndef _PTHREAD_H_
 #define _PTHREAD_H_
 
+#include <stdint.h>
+
 /**
  * This file is required to support the common code in eal_common_proc.c,
  * eal_common_thread.c and common\include\rte_per_lcore.h as Microsoft libc
lib/librte_eal/windows/include/rte_os.h +13 -31

75 lines not on this page · raw message

lib/librte_eal/windows/include/rte_windows.h new file +41 -0

41 lines not on this page · raw message

Signature · 1 line
2.25.1
#39 lore raw

Re: [dpdk-dev] [PATCH v3 02/10] eal/windows: do not expose private EAL facilities

14/04/2020 21:44, Dmitry Kozlyuk:

Quoted 21 lines

The goal of rte_os.h is to mitigate OS differences for EAL users. In Windows EAL, rte_os.h did excessive things:

1. It included platform SDK headers (windows.h, etc). Those files are huge, require specific inclusion order, and are generally unused by the code including rte_os.h. Declarations from platform SDK may break otherwise platform-independent code, e.g. min, max, ERROR.

2. It included pthread.h, which is clearly not always required.

3. It defined functions private to Windows EAL.

Reorganize Windows EAL includes in the following way:

1. Create rte_windows.h to properly import Windows-specific facilities. Primary users are bus drivers, tests, and external applications.

2. Remove platform SDK includes from rte_os.h to prevent breaking otherwise portable code by including rte_os.h on Windows. Copy necessary definitions to avoid including those headers.

Quoted 7 lines

3. Remove pthread.h include from rte_os.h.

4. Move declarations private to Windows EAL into eal_windows.h.

Fixes 428eb983f5f7 ("eal: add OS specific header file")

Applied as a separate patch because it is needed for a patch fixing compilation on Windows.

#40 lore raw

PATCH v3 3/10 eal/windows: improve CPU and NUMA node detection

In reply to Dmitry Kozlyuk

1. Map CPU cores to their respective NUMA nodes as reported by system.
2. Support systems with more than 64 cores (multiple processor groups).
3. Fix magic constants, styling issues, and compiler warnings.
4. Add EAL private function to map DPDK socket ID to NUMA node number.

Fixes 53ffd9f080fc ("eal/windows: add minimum viable code")
---
 lib/librte_eal/windows/eal_lcore.c   | 185 +++++++++++++++++----------
 lib/librte_eal/windows/eal_windows.h |  10 ++
 2 files changed, 124 insertions(+), 71 deletions(-)
lib/librte_eal/windows/eal_lcore.c +114 -71
index 82ee45413..9d931d50a 100644
@@ -3,103 +3,146 @@
  */
 
 #include <pthread.h>
+#include <stdbool.h>
 #include <stdint.h>
 
 #include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_lcore.h>
+#include <rte_os.h>
 
 #include "eal_private.h"
 #include "eal_thread.h"
 #include "eal_windows.h"
 
-/* global data structure that contains the CPU map */
-static struct _wcpu_map {
-	unsigned int total_procs;
-	unsigned int proc_sockets;
-	unsigned int proc_cores;
-	unsigned int reserved;
-	struct _win_lcore_map {
-		uint8_t socket_id;
-		uint8_t core_id;
-	} wlcore_map[RTE_MAX_LCORE];
-} wcpu_map = { 0 };
-
-/*
- * Create a map of all processors and associated cores on the system
- */
+/** Number of logical processors (cores) in a processor group (32 or 64). */
+#define EAL_PROCESSOR_GROUP_SIZE (sizeof(KAFFINITY) * CHAR_BIT)
+
+struct lcore_map {
+	uint8_t socket_id;
+	uint8_t core_id;
+};
+
+struct socket_map {
+	uint16_t node_id;
+};
+
+struct cpu_map {
+	unsigned int socket_count;
+	unsigned int lcore_count;
+	struct lcore_map lcores[RTE_MAX_LCORE];
+	struct socket_map sockets[RTE_MAX_NUMA_NODES];
+};
+
+static struct cpu_map cpu_map = { 0 };
+
 void
-eal_create_cpu_map()
+eal_create_cpu_map(void)
 {
-	wcpu_map.total_procs =
-		GetActiveProcessorCount(ALL_PROCESSOR_GROUPS);
-
-	LOGICAL_PROCESSOR_RELATIONSHIP lprocRel;
-	DWORD lprocInfoSize = 0;
-	BOOL ht_enabled = FALSE;
-
-	/* First get the processor package information */
-	lprocRel = RelationProcessorPackage;
-	/* Determine the size of buffer we need (pass NULL) */
-	GetLogicalProcessorInformationEx(lprocRel, NULL, &lprocInfoSize);
-	wcpu_map.proc_sockets = lprocInfoSize / 48;
-
-	lprocInfoSize = 0;
-	/* Next get the processor core information */
-	lprocRel = RelationProcessorCore;
-	GetLogicalProcessorInformationEx(lprocRel, NULL, &lprocInfoSize);
-	wcpu_map.proc_cores = lprocInfoSize / 48;
-
-	if (wcpu_map.total_procs > wcpu_map.proc_cores)
-		ht_enabled = TRUE;
-
-	/* Distribute the socket and core ids appropriately
-	 * across the logical cores. For now, split the cores
-	 * equally across the sockets.
-	 */
-	unsigned int lcore = 0;
-	for (unsigned int socket = 0; socket <
-			wcpu_map.proc_sockets; ++socket) {
-		for (unsigned int core = 0;
-			core < (wcpu_map.proc_cores / wcpu_map.proc_sockets);
-			++core) {
-			wcpu_map.wlcore_map[lcore]
-					.socket_id = socket;
-			wcpu_map.wlcore_map[lcore]
-					.core_id = core;
-			lcore++;
-			if (ht_enabled) {
-				wcpu_map.wlcore_map[lcore]
-					.socket_id = socket;
-				wcpu_map.wlcore_map[lcore]
-					.core_id = core;
-				lcore++;
+	SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *infos, *info;
+	DWORD infos_size;
+	bool full = false;
+
+	infos_size = 0;
+	if (!GetLogicalProcessorInformationEx(
+			RelationNumaNode, NULL, &infos_size)) {
+		DWORD error = GetLastError();
+		if (error != ERROR_INSUFFICIENT_BUFFER) {
+			rte_panic("cannot get NUMA node info size, error %lu",
+				GetLastError());
+		}
+	}
+
+	infos = malloc(infos_size);
+	if (infos == NULL) {
+		rte_panic("cannot allocate memory for NUMA node information");
+		return;
+	}
+
+	if (!GetLogicalProcessorInformationEx(
+			RelationNumaNode, infos, &infos_size)) {
+		rte_panic("cannot get NUMA node information, error %lu",
+			GetLastError());
+	}
+
+	info = infos;
+	while ((uint8_t *)info - (uint8_t *)infos < infos_size) {
+		unsigned int node_id = info->NumaNode.NodeNumber;
+		GROUP_AFFINITY *cores = &info->NumaNode.GroupMask;
+		struct lcore_map *lcore;
+		unsigned int i, socket_id;
+
+		/* NUMA node may be reported multiple times if it includes
+		 * cores from different processor groups, e. g. 80 cores
+		 * of a physical processor comprise one NUMA node, but two
+		 * processor groups, because group size is limited by 32/64.
+		 */
+		for (socket_id = 0; socket_id < cpu_map.socket_count;
+		    socket_id++) {
+			if (cpu_map.sockets[socket_id].node_id == node_id)
+				break;
+		}
+
+		if (socket_id == cpu_map.socket_count) {
+			if (socket_id == RTE_DIM(cpu_map.sockets)) {
+				full = true;
+				goto exit;
 			}
+
+			cpu_map.sockets[socket_id].node_id = node_id;
+			cpu_map.socket_count++;
+		}
+
+		for (i = 0; i < EAL_PROCESSOR_GROUP_SIZE; i++) {
+			if ((cores->Mask & ((KAFFINITY)1 << i)) == 0)
+				continue;
+
+			if (cpu_map.lcore_count == RTE_DIM(cpu_map.lcores)) {
+				full = true;
+				goto exit;
+			}
+
+			lcore = &cpu_map.lcores[cpu_map.lcore_count];
+			lcore->socket_id = socket_id;
+			lcore->core_id =
+				cores->Group * EAL_PROCESSOR_GROUP_SIZE + i;
+			cpu_map.lcore_count++;
 		}
+
+		info = (SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *)(
+			(uint8_t *)info + info->Size);
+	}
+
+exit:
+	if (full) {
+		/* RTE_LOG() may not be available, but this is important. */
+		fprintf(stderr, "Enumerated maximum of %u NUMA nodes and %u cores\n",
+			cpu_map.socket_count, cpu_map.lcore_count);
 	}
+
+	free(infos);
 }
 
-/*
- * Check if a cpu is present by the presence of the cpu information for it
- */
 int
 eal_cpu_detected(unsigned int lcore_id)
 {
-	return (lcore_id < wcpu_map.total_procs);
+	return lcore_id < cpu_map.lcore_count;
 }
 
-/*
- * Get CPU socket id for a logical core
- */
 unsigned
 eal_cpu_socket_id(unsigned int lcore_id)
 {
-	return wcpu_map.wlcore_map[lcore_id].socket_id;
+	return cpu_map.lcores[lcore_id].socket_id;
 }
 
-/*
- * Get CPU socket id (NUMA node) for a logical core
- */
 unsigned
 eal_cpu_core_id(unsigned int lcore_id)
 {
-	return wcpu_map.wlcore_map[lcore_id].core_id;
+	return cpu_map.lcores[lcore_id].core_id;
+}
+
+unsigned int
+eal_socket_numa_node(unsigned int socket_id)
+{
+	return cpu_map.sockets[socket_id].node_id;
 }
lib/librte_eal/windows/eal_windows.h +10 -0
index fadd676b2..390d2fd66 100644
@@ -26,4 +26,14 @@ void eal_create_cpu_map(void);
  */
 int eal_thread_create(pthread_t *thread);
 
+/**
+ * Get system NUMA node number for a socket ID.
+ *
+ * @param socket_id
+ *  Valid EAL socket ID.
+ * @return
+ *  NUMA node number to use with Win32 API.
+ */
+unsigned int eal_socket_numa_node(unsigned int socket_id);
+
 #endif /* _EAL_WINDOWS_H_ */
Signature · 1 line
2.25.1
#41 lore raw

PATCH v3 4/10 eal/windows: initialize hugepage info

In reply to Dmitry Kozlyuk

Add hugepages discovery ("large pages" in Windows terminology) and update documentation for required privilege setup.

---
 config/meson.build                     |   2 +
 doc/guides/windows_gsg/build_dpdk.rst  |  20 -----
 doc/guides/windows_gsg/index.rst       |   1 +
 doc/guides/windows_gsg/run_apps.rst    |  47 +++++++++++
 lib/librte_eal/windows/eal.c           |  14 ++++
 lib/librte_eal/windows/eal_hugepages.c | 108 +++++++++++++++++++++++++
 lib/librte_eal/windows/meson.build     |   1 +
 7 files changed, 173 insertions(+), 20 deletions(-)
 create mode 100644 doc/guides/windows_gsg/run_apps.rst
 create mode 100644 lib/librte_eal/windows/eal_hugepages.c
config/meson.build +2 -0
index 58421342b..4607655d9 100644
@@ -263,6 +263,8 @@ if is_windows
 	if cc.get_id() == 'gcc'
 		add_project_arguments('-D__USE_MINGW_ANSI_STDIO', language: 'c')
 	endif
+
+	add_project_link_arguments('-ladvapi32', language: 'c')
 endif
 
 if get_option('b_lto')
doc/guides/windows_gsg/build_dpdk.rst +0 -20
index d46e84e3f..650483e3b 100644
@@ -111,23 +111,3 @@ Depending on the distribution, paths in this file may need adjustments.
 
     meson --cross-file config/x86/meson_mingw.txt -Dexamples=helloworld build
     ninja -C build
-
-
-Run the helloworld example
-==========================
-
-Navigate to the examples in the build directory and run `dpdk-helloworld.exe`.
-
-.. code-block:: console
-
-    cd C:\Users\me\dpdk\build\examples
-    dpdk-helloworld.exe
-    hello from core 1
-    hello from core 3
-    hello from core 0
-    hello from core 2
-
-Note for MinGW-w64: applications are linked to ``libwinpthread-1.dll``
-by default. To run the example, either add toolchain executables directory
-to the PATH or copy the library to the working directory.
-Alternatively, static linking may be used (mind the LGPLv2.1 license).
doc/guides/windows_gsg/index.rst +1 -0
index d9b7990a8..e94593572 100644
@@ -12,3 +12,4 @@ Getting Started Guide for Windows
 
     intro
     build_dpdk
+    run_apps
doc/guides/windows_gsg/run_apps.rst new file +47 -0

47 lines not on this page · raw message

lib/librte_eal/windows/eal.c +14 -0
index 2cf7a04ef..63461f51a 100644
@@ -18,8 +18,11 @@
 #include <eal_options.h>
 #include <eal_private.h>
 
+#include "eal_hugepages.h"
 #include "eal_windows.h"
 
+#define MEMSIZE_IF_NO_HUGE_PAGE (64ULL * 1024ULL * 1024ULL)
+
  /* Allow the application to print its usage message too if set */
 static rte_usage_hook_t	rte_application_usage_hook;
 
@@ -242,6 +245,17 @@ rte_eal_init(int argc, char **argv)
 	if (fctret < 0)
 		exit(1);
 
+	if (!internal_config.no_hugetlbfs && (eal_hugepage_info_init() < 0)) {
+		rte_eal_init_alert("Cannot get hugepage information");
+		rte_errno = EACCES;
+		return -1;
+	}
+
+	if (internal_config.memory == 0 && !internal_config.force_sockets) {
+		if (internal_config.no_hugetlbfs)
+			internal_config.memory = MEMSIZE_IF_NO_HUGE_PAGE;
+	}
+
 	eal_thread_init_master(rte_config.master_lcore);
 
 	RTE_LCORE_FOREACH_SLAVE(i) {
lib/librte_eal/windows/eal_hugepages.c new file +108 -0

108 lines not on this page · raw message

lib/librte_eal/windows/meson.build +1 -0
index 09dd4ab2f..5f118bfe2 100644
@@ -6,6 +6,7 @@ subdir('include')
 sources += files(
 	'eal.c',
 	'eal_debug.c',
+	'eal_hugepages.c',
 	'eal_lcore.c',
 	'eal_thread.c',
 	'getopt.c',
Signature · 1 line
2.25.1
#42 lore raw

PATCH v3 5/10 eal: introduce internal wrappers for file operations

In reply to Dmitry Kozlyuk

EAL common code uses file locking and truncation. Introduce OS-independent wrappers in order to support both Linux/FreeBSD and Windows:

* eal_file_lock: lock or unlock an open file.
* eal_file_truncate: enforce a given size for an open file.

Wrappers follow POSIX semantics, but interface is not POSIX, so that it can be made more clean, e.g. by not mixing locking operation and behaviour on conflict.

Implementation for Linux and FreeBSD is placed in "unix" subdirectory, which is intended for common code between the two. Files should be named after the ones from which the code is factored in OS subdirectory.

---
 lib/librte_eal/common/eal_private.h | 45 ++++++++++++++++
 lib/librte_eal/meson.build          |  4 ++
 lib/librte_eal/unix/eal.c           | 47 ++++++++++++++++
 lib/librte_eal/unix/meson.build     |  6 +++
 lib/librte_eal/windows/eal.c        | 83 +++++++++++++++++++++++++++++
 5 files changed, 185 insertions(+)
 create mode 100644 lib/librte_eal/unix/eal.c
 create mode 100644 lib/librte_eal/unix/meson.build
lib/librte_eal/common/eal_private.h +45 -0
index ddcfbe2e4..65d61ff13 100644
@@ -443,4 +443,49 @@ rte_option_usage(void);
 uint64_t
 eal_get_baseaddr(void);
 
+/** File locking operation. */
+enum eal_flock_op {
+	EAL_FLOCK_SHARED,    /**< Acquire a shared lock. */
+	EAL_FLOCK_EXCLUSIVE, /**< Acquire an exclusive lock. */
+	EAL_FLOCK_UNLOCK     /**< Release a previously taken lock. */
+};
+
+/** Behavior on file locking conflict. */
+enum eal_flock_mode {
+	EAL_FLOCK_WAIT,  /**< Wait until the file gets unlocked to lock it. */
+	EAL_FLOCK_RETURN /**< Return immediately if the file is locked. */
+};
+
+/**
+ * Lock or unlock the file.
+ *
+ * On failure @code rte_errno @endcode is set to the error code
+ * specified by POSIX flock(3) description.
+ *
+ * @param fd
+ *  Opened file descriptor.
+ * @param op
+ *  Operation to perform.
+ * @param mode
+ *  Behavior on conflict.
+ * @return
+ *  0 on success, (-1) on failure.
+ */
+int eal_file_lock(int fd, enum eal_flock_op op, enum eal_flock_mode mode);
+
+/**
+ * Truncate or extend the file to the specified size.
+ *
+ * On failure @code rte_errno @endcode is set to the error code
+ * specified by POSIX ftruncate(3) description.
+ *
+ * @param fd
+ *  Opened file descriptor.
+ * @param size
+ *  Desired file size.
+ * @return
+ *  0 on success, (-1) on failure.
+ */
+int eal_file_truncate(int fd, ssize_t size);
+
 #endif /* _EAL_PRIVATE_H_ */
lib/librte_eal/meson.build +4 -0
index 9d219a0e6..1f89efb88 100644
@@ -6,6 +6,10 @@ subdir('include')
 
 subdir('common')
 
+if not is_windows
+	subdir('unix')
+endif
+
 dpdk_conf.set('RTE_EXEC_ENV_' + exec_env.to_upper(), 1)
 subdir(exec_env)
 
lib/librte_eal/unix/eal.c new file +47 -0

47 lines not on this page · raw message

lib/librte_eal/unix/meson.build new file +6 -0
new file mode 100644
index 000000000..13564838e
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020 Dmitry Kozlyuk
+
+sources += files(
+	'eal.c',
+)
lib/librte_eal/windows/eal.c +83 -0

89 lines not on this page · raw message

Signature · 1 line
2.25.1
#43 lore raw

Re: [dpdk-dev] [PATCH v3 05/10] eal: introduce internal wrappers for file operations

Thanks for starting the new directory for Unix common implementation.

14/04/2020 21:44, Dmitry Kozlyuk:

lib/librte_eal/unix/eal.c

Quoted 2 lines
lib/librte_eal/unix/eal.c new file +0 -0

Please take care of not creating a new file without adding SPDX tag and copyright owner.

Quoted 11 lines
lib/librte_eal/unix/eal.c new file +10 -0
@@ -0,0 +1,47 @@
+#include <sys/file.h>
+#include <sys/mman.h>
+#include <unistd.h>
+
+#include <rte_errno.h>
+
+#include "eal_private.h"
+
+int
+eal_file_truncate(int fd, ssize_t size)

[...]

lib/librte_eal/unix/eal.c

Quoted 2 lines
lib/librte_eal/unix/eal.c new file +2 -0
+int
+eal_file_lock(int fd, enum eal_flock_op op, enum eal_flock_mode mode)

+1
Adding new abstractions is the way to go in my opinion. Thanks

#44 lore raw

Re: [dpdk-dev] [PATCH v3 05/10] eal: introduce internal wrappers for file operations

In reply to Dmitry Kozlyuk
Quoted 36 lines · Dmitry Kozlyuk

EAL common code uses file locking and truncation. Introduce OS-independent wrappers in order to support both Linux/FreeBSD and Windows:

* eal_file_lock: lock or unlock an open file.
* eal_file_truncate: enforce a given size for an open file.

Wrappers follow POSIX semantics, but interface is not POSIX, so that it can be made more clean, e.g. by not mixing locking operation and behaviour on conflict.

Implementation for Linux and FreeBSD is placed in "unix" subdirectory, which is intended for common code between the two. Files should be named after the ones from which the code is factored in OS subdirectory.

---
  lib/librte_eal/common/eal_private.h | 45 ++++++++++++++++
  lib/librte_eal/meson.build          |  4 ++
  lib/librte_eal/unix/eal.c           | 47 ++++++++++++++++
  lib/librte_eal/unix/meson.build     |  6 +++
  lib/librte_eal/windows/eal.c        | 83 +++++++++++++++++++++++++++++
  5 files changed, 185 insertions(+)

create mode 100644 lib/librte_eal/unix/eal.c create mode 100644 lib/librte_eal/unix/meson.build

lib/librte_eal/common/eal_private.h +2 -0
index ddcfbe2e4..65d61ff13 100644
@@ -443,4 +443,49 @@ rte_option_usage(void);
  uint64_t
  eal_get_baseaddr(void);
 
+/** File locking operation. */
+enum eal_flock_op {

lib/librte_eal/common/eal_private.h rte_option_usage(void); · line 457

Quoted 10 lines
lib/librte_eal/common/eal_private.h +10 -0
+	EAL_FLOCK_SHARED,    /**< Acquire a shared lock. */
+	EAL_FLOCK_EXCLUSIVE, /**< Acquire an exclusive lock. */
+	EAL_FLOCK_UNLOCK     /**< Release a previously taken lock. */
+};
+
+/** Behavior on file locking conflict. */
+enum eal_flock_mode {
+	EAL_FLOCK_WAIT,  /**< Wait until the file gets unlocked to lock it. */
+	EAL_FLOCK_RETURN /**< Return immediately if the file is locked. */
+};

Nitpicking, but why not blocking/unblocking? The terminology seems pretty standard.

Signature · 2 lines
Thanks,
Anatoly
#45 lore raw

Re: [dpdk-dev] [PATCH v3 05/10] eal: introduce internal wrappers for file operations

On 2020-04-17 13:24 GMT+0100 Burakov, Anatoly wrote:
[snip]

Quoted 8 lines

lib/librte_eal/common/eal_private.h rte_option_usage(void); · line 457

Quoted 5 lines
lib/librte_eal/common/eal_private.h +5 -0
+/** Behavior on file locking conflict. */
+enum eal_flock_mode {
+	EAL_FLOCK_WAIT,  /**< Wait until the file gets unlocked to lock it. */
+	EAL_FLOCK_RETURN /**< Return immediately if the file is locked. */
+};  

Nitpicking, but why not blocking/unblocking? The terminology seems pretty standard.

On Windows, LockFileEx() call may be non-blocking, but still configured to fail if the lock is already taken. To avoid confusion, these names reflect what the behavior will be instead of how it will be achieved.

Signature · 1 line
Dmitry Kozlyuk
#46 lore raw

PATCH v3 6/10 eal: introduce memory management wrappers

In reply to Dmitry Kozlyuk

System meory management is implemented differently for POSIX and Windows. Introduce wrapper functions for operations used across DPDK:

* rte_mem_map()
Create memory mapping for a regular file or a page file (swap). This supports mapping to a reserved memory region even on Windows.

* rte_mem_unmap()
Remove mapping created with rte_mem_map().

* rte_get_page_size()
Obtain default system page size.

* rte_mem_lock()
Make arbitrary-sized memory region non-swappable.

Wrappers follow POSIX semantics limited to DPDK tasks, but their signatures deliberately differ from POSIX ones to be more safe and expressive.

---
 config/meson.build                   |  10 +-
 lib/librte_eal/common/eal_private.h  |  51 +++-
 lib/librte_eal/include/rte_memory.h  |  68 +++++
 lib/librte_eal/rte_eal_exports.def   |   4 +
 lib/librte_eal/rte_eal_version.map   |   4 +
 lib/librte_eal/unix/eal_memory.c     | 113 +++++++
 lib/librte_eal/unix/meson.build      |   1 +
 lib/librte_eal/windows/eal.c         |   6 +
 lib/librte_eal/windows/eal_memory.c  | 437 +++++++++++++++++++++++++++
 lib/librte_eal/windows/eal_windows.h |  67 ++++
 lib/librte_eal/windows/meson.build   |   1 +
 11 files changed, 758 insertions(+), 4 deletions(-)
 create mode 100644 lib/librte_eal/unix/eal_memory.c
 create mode 100644 lib/librte_eal/windows/eal_memory.c
config/meson.build +8 -2
index 4607655d9..bceb5ef7b 100644
@@ -256,14 +256,20 @@ if is_freebsd
 endif
 
 if is_windows
-	# Minimum supported API is Windows 7.
-	add_project_arguments('-D_WIN32_WINNT=0x0601', language: 'c')
+	# VirtualAlloc2() is available since Windows 10 / Server 2016.
+	add_project_arguments('-D_WIN32_WINNT=0x0A00', language: 'c')
 
 	# Use MinGW-w64 stdio, because DPDK assumes ANSI-compliant formatting.
 	if cc.get_id() == 'gcc'
 		add_project_arguments('-D__USE_MINGW_ANSI_STDIO', language: 'c')
 	endif
 
+	# Contrary to docs, VirtualAlloc2() is exported by mincore.lib
+	# in Windows SDK, while MinGW exports it by advapi32.a.
+	if is_ms_linker
+		add_project_link_arguments('-lmincore', language: 'c')
+	endif
+
 	add_project_link_arguments('-ladvapi32', language: 'c')
 endif
 
lib/librte_eal/common/eal_private.h +49 -2

73 lines not on this page · raw message

lib/librte_eal/include/rte_memory.h +68 -0

74 lines not on this page · raw message

lib/librte_eal/rte_eal_exports.def +4 -0
index 12a6c79d6..bacf9a107 100644
@@ -5,5 +5,9 @@ EXPORTS
 	rte_eal_mp_remote_launch
 	rte_eal_mp_wait_lcore
 	rte_eal_remote_launch
+	rte_get_page_size
 	rte_log
+	rte_mem_lock
+	rte_mem_map
+	rte_mem_unmap
 	rte_vlog
lib/librte_eal/rte_eal_version.map +4 -0
index f9ede5b41..07128898f 100644
@@ -337,5 +337,9 @@ EXPERIMENTAL {
 	rte_thread_is_intr;
 
 	# added in 20.05
+	rte_get_page_size;
 	rte_log_can_log;
+	rte_mem_lock;
+	rte_mem_map;
+	rte_mem_unmap;
 };
lib/librte_eal/unix/eal_memory.c new file +113 -0

113 lines not on this page · raw message

lib/librte_eal/unix/meson.build +1 -0
index 13564838e..50c019a56 100644
@@ -3,4 +3,5 @@
 
 sources += files(
 	'eal.c',
+	'eal_memory.c',
 )
lib/librte_eal/windows/eal.c +6 -0
index 9dba895e7..cf55b56da 100644
@@ -339,6 +339,12 @@ rte_eal_init(int argc, char **argv)
 			internal_config.memory = MEMSIZE_IF_NO_HUGE_PAGE;
 	}
 
+	if (eal_mem_win32api_init() < 0) {
+		rte_eal_init_alert("Cannot access Win32 memory management");
+		rte_errno = ENOTSUP;
+		return -1;
+	}
+
 	eal_thread_init_master(rte_config.master_lcore);
 
 	RTE_LCORE_FOREACH_SLAVE(i) {
lib/librte_eal/windows/eal_memory.c new file +437 -0

437 lines not on this page · raw message

lib/librte_eal/windows/eal_windows.h +67 -0

71 lines not on this page · raw message

lib/librte_eal/windows/meson.build +1 -0
index 5f118bfe2..81d3ee095 100644
@@ -8,6 +8,7 @@ sources += files(
 	'eal_debug.c',
 	'eal_hugepages.c',
 	'eal_lcore.c',
+	'eal_memory.c',
 	'eal_thread.c',
 	'getopt.c',
 )
Signature · 1 line
2.25.1
#47 lore raw

Re: [dpdk-dev] [PATCH v3 06/10] eal: introduce memory management wrappers

14/04/2020 21:44, Dmitry Kozlyuk:

Quoted 1 line

System meory management is implemented differently for POSIX and

meory -> memory

Quoted 20 lines

Windows. Introduce wrapper functions for operations used across DPDK:

* rte_mem_map()
Create memory mapping for a regular file or a page file (swap). This supports mapping to a reserved memory region even on Windows.

* rte_mem_unmap()
Remove mapping created with rte_mem_map().

* rte_get_page_size()
Obtain default system page size.

* rte_mem_lock()
Make arbitrary-sized memory region non-swappable.

Wrappers follow POSIX semantics limited to DPDK tasks, but their signatures deliberately differ from POSIX ones to be more safe and expressive.

[...]

lib/librte_eal/common/eal_private.h int rte_eal_alarm_init(void); · line 214

Quoted 9 lines
lib/librte_eal/common/eal_private.h +9 -0
+/**
+ * Memory reservation flags.
+ */
+enum eal_mem_reserve_flags {
+	/**< Reserve hugepages (support may be limited or missing). */
+	EAL_RESERVE_HUGEPAGES = 1 << 0,
+	/**< Fail if requested address is not available. */
+	EAL_RESERVE_EXACT_ADDRESS = 1 << 1
+};

Maybe more context is needed to understand the meaning of these flags. [...]

lib/librte_eal/common/eal_private.h int rte_eal_check_module(const char *module_name); · line 247

Quoted 4 lines
lib/librte_eal/common/eal_private.h +2 -2
-eal_get_virtual_area(void *requested_addr, size_t *size,
-		size_t page_sz, int flags, int mmap_flags);
+eal_get_virtual_area(void *requested_addr, size_t *size, size_t page_sz,
+	int flags, int mmap_flags);

Is there any change here?

[...]

Quoted 1 line

+ * If @code virt @endcode and @code size @endcode describe a part of the

I am not sure about using @code.
It makes reading from source harder.
Is there a real benefit?

[...]

lib/librte_eal/include/rte_memory.h struct rte_memseg_list { · line 95

Quoted 8 lines
lib/librte_eal/include/rte_memory.h +8 -0
+/**
+ * Memory protection flags.
+ */
+enum rte_mem_prot {
+	RTE_PROT_READ = 1 << 0,   /**< Read access. */
+	RTE_PROT_WRITE = 1 << 1,   /**< Write access. */
+	RTE_PROT_EXECUTE = 1 << 2 /**< Code execution. */
+};

Alignment of comments would look better :-)

lib/librte_eal/include/rte_memory.h struct rte_memseg_list { · line 103

Quoted 8 lines
lib/librte_eal/include/rte_memory.h +8 -0
+
+/**
+ * Memory mapping additional flags.
+ *
+ * In Linux and FreeBSD, each flag is semantically equivalent
+ * to OS-specific mmap(3) flag with the same or similar name.
+ * In Windows, POSIX and MAP_ANONYMOUS semantics are followed.
+ */

I don't understand this comment.
The flags and meanings are the same no matter the OS, right?

Quoted 17 lines
lib/librte_eal/include/rte_memory.h +17 -0
+enum rte_map_flags {
+	/** Changes of mapped memory are visible to other processes. */
+	RTE_MAP_SHARED = 1 << 0,
+	/** Mapping is not backed by a regular file. */
+	RTE_MAP_ANONYMOUS = 1 << 1,
+	/** Copy-on-write mapping, changes are invisible to other processes. */
+	RTE_MAP_PRIVATE = 1 << 2,
+	/** Fail if requested address cannot be taken. */
+	RTE_MAP_FIXED = 1 << 3
+};
+
+/**
+ * OS-independent implementation of POSIX mmap(3)
+ * with MAP_ANONYMOUS Linux/FreeBSD extension.
+ */
+__rte_experimental
+void *rte_mem_map(void *requested_addr, size_t size, enum rte_mem_prot prot,

lib/librte_eal/include/rte_memory.h struct rte_memseg_list { · line 130

Quoted 10 lines
lib/librte_eal/include/rte_memory.h +10 -0
+	enum rte_map_flags flags, int fd, size_t offset);
+
+/**
+ * OS-independent implementation of POSIX munmap(3).
+ */
+__rte_experimental
+int rte_mem_unmap(void *virt, size_t size);
+
+/**
+ * Get system page size. This function never failes.

failes -> fails

Quoted 24 lines

+ *
+ * @return
+ * Positive page size in bytes.
+ */

lib/librte_eal/include/rte_memory.h +20 -0
+__rte_experimental
+int rte_get_page_size(void);
+
+/**
+ * Lock region in physical memory and prevent it from swapping.
+ *
+ * @param virt
+ *   The virtual address.
+ * @param size
+ *   Size of the region.
+ * @return
+ *   0 on success, negative on error.
+ *
+ * @note Implementations may require @p virt and @p size to be multiples
+ *       of system page size.
+ * @see rte_get_page_size()
+ * @see rte_mem_lock_page()
+ */
+__rte_experimental
+int rte_mem_lock(const void *virt, size_t size);

[...]

lib/librte_eal/unix/eal_memory.c

Quoted 2 lines
lib/librte_eal/unix/eal_memory.c new file +0 -0

License and copyright missing.

Quoted 8 lines
lib/librte_eal/unix/eal_memory.c new file +7 -0
@@ -0,0 +1,113 @@
+#include <string.h>
+#include <sys/mman.h>
+#include <unistd.h>
+
+#include <rte_errno.h>
+#include <rte_log.h>
+#include <rte_memory.h>

lib/librte_eal/unix/eal_memory.c line 17

Quoted 10 lines
lib/librte_eal/unix/eal_memory.c new file +10 -0
+
+#include "eal_private.h"
+
+static void *
+mem_map(void *requested_addr, size_t size, int prot, int flags,
+	int fd, size_t offset)
+{
+	void *virt = mmap(requested_addr, size, prot, flags, fd, offset);
+	if (virt == MAP_FAILED) {
+		RTE_LOG(ERR, EAL,

Not sure it should be a log level so high.
We could imagine checking a memory map.
What about INFO level?
The real error log will be made by the caller.

Quoted 5 lines
lib/librte_eal/unix/eal_memory.c new file +5 -0
+			"Cannot mmap(%p, 0x%zx, 0x%x, 0x%x, %d, 0x%zx): %s\n",
+			requested_addr, size, prot, flags, fd, offset,
+			strerror(errno));
+		rte_errno = errno;
+		return NULL;

[...]

lib/librte_eal/unix/eal_memory.c line 48

Quoted 10 lines
lib/librte_eal/unix/eal_memory.c new file +10 -0
+void *
+eal_mem_reserve(void *requested_addr, size_t size,
+	enum eal_mem_reserve_flags flags)
+{
+	int sys_flags = MAP_PRIVATE | MAP_ANONYMOUS;
+
+#ifdef MAP_HUGETLB
+	if (flags & EAL_RESERVE_HUGEPAGES)
+		sys_flags |= MAP_HUGETLB;
+#endif

If MAP_HUGETLB is false, and flags contain EAL_RESERVE_HUGEPAGES, I think an error should be returned.

Quoted 5 lines
lib/librte_eal/unix/eal_memory.c new file +5 -0
+	if (flags & EAL_RESERVE_EXACT_ADDRESS)
+		sys_flags |= MAP_FIXED;
+
+	return mem_map(requested_addr, size, PROT_NONE, sys_flags, -1, 0);
+}

[...]

lib/librte_eal/unix/eal_memory.c line 113

Quoted 11 lines
lib/librte_eal/unix/eal_memory.c new file +11 -0
+int
+rte_get_page_size(void)
+{
+	return getpagesize();
+}
+
+int
+rte_mem_lock(const void *virt, size_t size)
+{
+	return mlock(virt, size);
+}

Why don't you replace existing code with these new functions?

#48 lore raw

Re: [dpdk-dev] [PATCH v3 06/10] eal: introduce memory management wrappers

Answering the questions. All snipped comments will be fixed in v4.

Quoted 12 lines

[...]

lib/librte_eal/common/eal_private.h int rte_eal_alarm_init(void); · line 214

Quoted 9 lines
lib/librte_eal/common/eal_private.h +9 -0
+/**
+ * Memory reservation flags.
+ */
+enum eal_mem_reserve_flags {
+	/**< Reserve hugepages (support may be limited or missing). */
+	EAL_RESERVE_HUGEPAGES = 1 << 0,
+	/**< Fail if requested address is not available. */
+	EAL_RESERVE_EXACT_ADDRESS = 1 << 1
+};  

Maybe more context is needed to understand the meaning of these flags.

Will extend the comment in v4. It's basically MAP_HUGE and MAP_FIXED.

Quoted 7 lines

[...]

lib/librte_eal/common/eal_private.h int rte_eal_check_module(const char *module_name); · line 247

Quoted 4 lines
lib/librte_eal/common/eal_private.h +2 -2
-eal_get_virtual_area(void *requested_addr, size_t *size,
-		size_t page_sz, int flags, int mmap_flags);
+eal_get_virtual_area(void *requested_addr, size_t *size, size_t page_sz,
+	int flags, int mmap_flags);  

Is there any change here?

No, will fix this artifact.

Quoted 6 lines

[...]

Quoted 1 line

+ * If @code virt @endcode and @code size @endcode describe a part of the

I am not sure about using @code.
It makes reading from source harder. Is there a real benefit?

It should be either @p or no markup (as in the rest of the comments), @code is indeed inappropriate.

Quoted 11 lines

lib/librte_eal/include/rte_memory.h struct rte_memseg_list { · line 103

Quoted 8 lines
lib/librte_eal/include/rte_memory.h +8 -0
+
+/**
+ * Memory mapping additional flags.
+ *
+ * In Linux and FreeBSD, each flag is semantically equivalent
+ * to OS-specific mmap(3) flag with the same or similar name.
+ * In Windows, POSIX and MAP_ANONYMOUS semantics are followed.
+ */  

I don't understand this comment.
The flags and meanings are the same no matter the OS, right?

Correct. MAP_ANONYMOUS is not POSIX so I mentioned it explicitly. I'll try to come up with better wording.

Quoted 13 lines

lib/librte_eal/unix/eal_memory.c line 17

Quoted 7 lines
lib/librte_eal/unix/eal_memory.c new file +7 -0
+static void *
+mem_map(void *requested_addr, size_t size, int prot, int flags,
+	int fd, size_t offset)
+{
+	void *virt = mmap(requested_addr, size, prot, flags, fd, offset);
+	if (virt == MAP_FAILED) {
+		RTE_LOG(ERR, EAL,  

Not sure it should be a log level so high. We could imagine checking a memory map. What about INFO level?
The real error log will be made by the caller.

Quoted 5 lines

lib/librte_eal/unix/eal_memory.c line 22

Quoted 5 lines
lib/librte_eal/unix/eal_memory.c new file +5 -0
+			"Cannot mmap(%p, 0x%zx, 0x%x, 0x%x, %d, 0x%zx): %s\n",
+			requested_addr, size, prot, flags, fd, offset,
+			strerror(errno));
+		rte_errno = errno;
+		return NULL;  

The same level is used now in places from which this code is extracted:
lib/librte_eal/common/{eal_common_fbarray.c:97,eal_common_memory:131}, see also lib/librte_pci/rte_pci.c:144. To my understanding, DEBUG is used to log implementation-specific details like these OS API calls, so I'll change level to that.

Quoted 14 lines

[...]

lib/librte_eal/unix/eal_memory.c line 113

Quoted 11 lines
lib/librte_eal/unix/eal_memory.c new file +11 -0
+int
+rte_get_page_size(void)
+{
+	return getpagesize();
+}
+
+int
+rte_mem_lock(const void *virt, size_t size)
+{
+	return mlock(virt, size);
+}  

Why don't you replace existing code with these new functions?

In this patchset I tried to touch existing code as little as possible, at least I'd like to limit the scope to EAL. Libraries and drivers using Unix functions directly will fail to compile when enabled on Windows, but patches will be trivial. I propose replacing calls in EAL in v4.

Signature · 1 line
Dmitry Kozlyuk
#49 lore raw

Re: [dpdk-dev] [PATCH v3 06/10] eal: introduce memory management wrappers

In reply to Dmitry Kozlyuk
Quoted 22 lines · Dmitry Kozlyuk

System meory management is implemented differently for POSIX and Windows. Introduce wrapper functions for operations used across DPDK:

* rte_mem_map()
Create memory mapping for a regular file or a page file (swap). This supports mapping to a reserved memory region even on Windows.

* rte_mem_unmap()
Remove mapping created with rte_mem_map().

* rte_get_page_size()
Obtain default system page size.

* rte_mem_lock()
Make arbitrary-sized memory region non-swappable.

Wrappers follow POSIX semantics limited to DPDK tasks, but their signatures deliberately differ from POSIX ones to be more safe and expressive.

---

<snip>

lib/librte_eal/common/eal_private.h int rte_eal_alarm_init(void); · line 213

Quoted 8 lines
lib/librte_eal/common/eal_private.h +8 -0
+/**
+ * Memory reservation flags.
+ */
+enum eal_mem_reserve_flags {
+	/**< Reserve hugepages (support may be limited or missing). */
+	EAL_RESERVE_HUGEPAGES = 1 << 0,
+	/**< Fail if requested address is not available. */
+	EAL_RESERVE_EXACT_ADDRESS = 1 << 1

I *really* don't like this terminology.

In Linux et al., MAP_FIXED is not just "reserve at this exact address". MAP_FIXED is actually fairly dangerous if you don't know what you're doing, because it will unconditionally unmap any previously mapped memory. Also, to my knowledge, a call to MAP_FIXED cannot fail unless something went very wrong - it will *not* "fail if requested address is not available". We basically use MAP_FIXED because we have already mapped that area with MAP_ANONYMOUS previously, so we can guarantee that it's safe to call MAP_FIXED.

I would greatly prefer if this was named to better reflect the above. EAL_FORCE_RESERVE perhaps? The comment also needs to be adjusted.

Quoted 15 lines
lib/librte_eal/common/eal_private.h +4 -2
+};
+
  /**
   * Get virtual area of specified size from the OS.
   *
@@ -232,8 +243,8 @@ int rte_eal_check_module(const char *module_name);
  #define EAL_VIRTUAL_AREA_UNMAP (1 << 2)
  /**< immediately unmap reserved virtual area. */
  void *
-eal_get_virtual_area(void *requested_addr, size_t *size,
-		size_t page_sz, int flags, int mmap_flags);
+eal_get_virtual_area(void *requested_addr, size_t *size, size_t page_sz,
+	int flags, int mmap_flags);
 
  /**

<snip>

Quoted 9 lines
lib/librte_eal/common/eal_private.h +8 -0
+/**
+ * Reserve a region of virtual memory.
+ *
+ * Use eal_mem_free() to free reserved memory.
+ *
+ * @param requested_addr
+ *  A desired reservation address. The system may not respect it.
+ *  NULL means the address will be chosen by the system.

lib/librte_eal/common/eal_private.h int eal_file_lock(int fd, enum eal_flock_op op, enum eal_flock_mode mode); · line 519

Quoted 10 lines
lib/librte_eal/common/eal_private.h +10 -0
+ * @param size
+ *  Reservation size. Must be a multiple of system page size.
+ * @param flags
+ *  Reservation options.
+ * @returns
+ *  Starting address of the reserved area on success, NULL on failure.
+ *  Callers must not access this memory until remapping it.
+ */
+void *eal_mem_reserve(void *requested_addr, size_t size,
+	enum eal_mem_reserve_flags flags);

This seems fairly suspect to me. I know that technically enum is an int, but semantically, IIRC an enum value should always contain exactly one value - you can't use an enum value like a set of flags.

Quoted 12 lines
lib/librte_eal/common/eal_private.h +12 -0
+
+/**
+ * Free memory obtained by eal_mem_reserve() or eal_mem_alloc().
+ *
+ * If @code virt @endcode and @code size @endcode describe a part of the
+ * reserved region, only this part of the region is freed (accurately
+ * up to the system page size). If @code virt @endcode points to allocated
+ * memory, @code size @endcode must match the one specified on allocation.
+ * The behavior is undefined if the memory pointed by @code virt @endcode
+ * is obtained from another source than listed above.
+ *
+ * @param virt

<snip>

Quoted 6 lines
lib/librte_eal/include/rte_memory.h +6 -0
+/**
+ * Memory mapping additional flags.
+ *
+ * In Linux and FreeBSD, each flag is semantically equivalent
+ * to OS-specific mmap(3) flag with the same or similar name.
+ * In Windows, POSIX and MAP_ANONYMOUS semantics are followed.

lib/librte_eal/include/rte_memory.h struct rte_memseg_list { · line 112

Quoted 10 lines
lib/librte_eal/include/rte_memory.h +10 -0
+ */
+enum rte_map_flags {
+	/** Changes of mapped memory are visible to other processes. */
+	RTE_MAP_SHARED = 1 << 0,
+	/** Mapping is not backed by a regular file. */
+	RTE_MAP_ANONYMOUS = 1 << 1,
+	/** Copy-on-write mapping, changes are invisible to other processes. */
+	RTE_MAP_PRIVATE = 1 << 2,
+	/** Fail if requested address cannot be taken. */
+	RTE_MAP_FIXED = 1 << 3

Again, MAP_FIXED does not behave the way you describe. See above comments.

Quoted 14 lines
lib/librte_eal/include/rte_memory.h +14 -0
+};
+
+/**
+ * OS-independent implementation of POSIX mmap(3)
+ * with MAP_ANONYMOUS Linux/FreeBSD extension.
+ */
+__rte_experimental
+void *rte_mem_map(void *requested_addr, size_t size, enum rte_mem_prot prot,
+	enum rte_map_flags flags, int fd, size_t offset);
+
+/**
+ * OS-independent implementation of POSIX munmap(3).
+ */
+__rte_experimental

lib/librte_eal/include/rte_memory.h struct rte_memseg_list { · line 136

Quoted 10 lines
lib/librte_eal/include/rte_memory.h +10 -0
+int rte_mem_unmap(void *virt, size_t size);
+
+/**
+ * Get system page size. This function never failes.
+ *
+ * @return
+ *   Positive page size in bytes.
+ */
+__rte_experimental
+int rte_get_page_size(void);

uint32_t? or maybe uint64_t?

Quoted 20 lines
lib/librte_eal/include/rte_memory.h +19 -0
+
+/**
+ * Lock region in physical memory and prevent it from swapping.
+ *
+ * @param virt
+ *   The virtual address.
+ * @param size
+ *   Size of the region.
+ * @return
+ *   0 on success, negative on error.
+ *
+ * @note Implementations may require @p virt and @p size to be multiples
+ *       of system page size.
+ * @see rte_get_page_size()
+ * @see rte_mem_lock_page()
+ */
+__rte_experimental
+int rte_mem_lock(const void *virt, size_t size);
+
  /**
Signature · 2 lines
Thanks,
Anatoly
#50 lore raw

Re: [dpdk-dev] [PATCH v3 06/10] eal: introduce memory management wrappers

In reply to Dmitry Kozlyuk
Quoted 30 lines

System meory management is implemented differently for POSIX and Windows. Introduce wrapper functions for operations used across DPDK:

* rte_mem_map()
Create memory mapping for a regular file or a page file (swap). This supports mapping to a reserved memory region even on Windows.

* rte_mem_unmap()
Remove mapping created with rte_mem_map().

* rte_get_page_size()
Obtain default system page size.

* rte_mem_lock()
Make arbitrary-sized memory region non-swappable.

Wrappers follow POSIX semantics limited to DPDK tasks, but their signatures deliberately differ from POSIX ones to be more safe and expressive.

---
 config/meson.build                   |  10 +-
 lib/librte_eal/common/eal_private.h  |  51 +++-
lib/librte_eal/include/rte_memory.h  |  68 +++++
 lib/librte_eal/rte_eal_exports.def   |   4 +
 lib/librte_eal/rte_eal_version.map   |   4 +
 lib/librte_eal/unix/eal_memory.c     | 113 +++++++
 lib/librte_eal/unix/meson.build      |   1 +
 lib/librte_eal/windows/eal.c         |   6 +
 lib/librte_eal/windows/eal_memory.c  | 437
Quoted 1 line

+++++++++++++++++++++++++++ lib/librte_eal/windows/eal_windows.h |

<Snip!>

Quoted 7 lines
lib/librte_eal/windows/eal_memory.c new file +7 -0
+eal_mem_win32api_init(void)
+{
+	static const char library_name[] = "kernelbase.dll";
+	static const char function[] = "VirtualAlloc2";
+
+	OSVERSIONINFO info;
+	HMODULE library = NULL;

lib/librte_eal/windows/eal_memory.c line 83

Quoted 10 lines
lib/librte_eal/windows/eal_memory.c new file +10 -0
+	int ret = 0;
+
+	/* Already done. */
+	if (VirtualAlloc2 != NULL)
+		return 0;
+
+	/* IsWindows10OrGreater() may also be unavailable. */
+	memset(&info, 0, sizeof(info));
+	info.dwOSVersionInfoSize = sizeof(info);
+	GetVersionEx(&info);

I'd remove the GetVersionEx check entirely and add the comments regarding OS dependency to the RTE_LOG Of the LoadLibraryA failure below, GetVersionEx returns the Windows 8 OS version on newer servers

Also, it looks like not all Win2016 servers versions support VirtualAlloc2, I'm using Microsoft Windows Server 2016 Datacenter Version 10.0.14393 and LoadLibraryA failed to load VirtualAlloc2.

Quoted 17 lines
(quoted diff) +2 -0
+	/* Checking for Windows 10+ will also detect Windows Server 2016+.
+	 * Do not abort, because Windows may report false version

depending

lib/librte_eal/windows/eal_memory.c new file +4 -0
+	 * on executable manifest, compatibility mode, etc.
+	 */
+	if (info.dwMajorVersion < 10)
+		RTE_LOG(DEBUG, EAL, "Windows 10+ or Windows Server

2016+ "

lib/librte_eal/windows/eal_memory.c new file +5 -0
+			"is required for advanced memory features\n");
+
+	library = LoadLibraryA(library_name);
+	if (library == NULL) {
+		RTE_LOG_WIN32_ERR("LoadLibraryA(\"%s\")",

library_name);

lib/librte_eal/windows/eal.c +3 -0
+		return -1;
+	}
+
#51 lore raw

Re: [dpdk-dev] [PATCH v3 06/10] eal: introduce memory management wrappers

Quoted 2 lines

I'd remove the GetVersionEx check entirely and add the comments regarding OS dependency to the RTE_LOG Of the LoadLibraryA failure below, GetVersionEx returns the Windows 8 OS version on newer servers

Agreed, will do in v4.

Quoted 1 line

Also, it looks like not all Win2016 servers versions support VirtualAlloc2, I'm using Microsoft Windows Server 2016 Datacenter Version 10.0.14393 and LoadLibraryA failed to load VirtualAlloc2.

I confirm this. Documentation states Windows Server 2016 is supported, but it is at least partially incorrect, see comments in meson.build and GitHub issue:

https://github.com/MicrosoftDocs/feedback/issues/1129

How would you estimate Server 2016 support importance? Server 2019 and Windows 10 are known to work.

Signature · 1 line
Dmitry Kozlyuk
#52 lore raw

Re: [dpdk-dev] [PATCH v3 06/10] eal: introduce memory management wrappers

In reply to Dmitry Kozlyuk
Quoted 13 lines · Dmitry Kozlyuk

System meory management is implemented differently for POSIX and Windows. Introduce wrapper functions for operations used across DPDK:

* rte_mem_map()
Create memory mapping for a regular file or a page file (swap). This supports mapping to a reserved memory region even on Windows.

* rte_mem_unmap()
Remove mapping created with rte_mem_map().

* rte_get_page_size()
Obtain default system page size.

Quoted 8 lines

* rte_mem_lock()
Make arbitrary-sized memory region non-swappable.

Wrappers follow POSIX semantics limited to DPDK tasks, but their signatures deliberately differ from POSIX ones to be more safe and expressive.

<Snip!>

Quoted 117 lines
lib/librte_eal/windows/eal_memory.c new file +111 -0
new file mode 100644
index 000000000..5697187ce
@@ -0,0 +1,437 @@
+#include <io.h>
+
+#include <rte_errno.h>
+#include <rte_memory.h>
+
+#include "eal_private.h"
+#include "eal_windows.h"
+
+/* MinGW-w64 headers lack VirtualAlloc2() in some distributions.
+ * Provide a copy of definitions and code to load it dynamically.
+ * Note: definitions are copied verbatim from Microsoft documentation
+ * and don't follow DPDK code style.
+ */
+#ifndef MEM_PRESERVE_PLACEHOLDER
+
+/* https://docs.microsoft.com/en-us/windows/win32/api/winnt/ne-winnt-mem_extended_parameter_type */
+typedef enum MEM_EXTENDED_PARAMETER_TYPE {
+	MemExtendedParameterInvalidType,
+	MemExtendedParameterAddressRequirements,
+	MemExtendedParameterNumaNode,
+	MemExtendedParameterPartitionHandle,
+	MemExtendedParameterMax,
+	MemExtendedParameterUserPhysicalHandle,
+	MemExtendedParameterAttributeFlags
+} *PMEM_EXTENDED_PARAMETER_TYPE;
+
+#define MEM_EXTENDED_PARAMETER_TYPE_BITS 4
+
+/* https://docs.microsoft.com/en-us/windows/win32/api/winnt/ns-winnt-mem_extended_parameter */
+typedef struct MEM_EXTENDED_PARAMETER {
+	struct {
+		DWORD64 Type : MEM_EXTENDED_PARAMETER_TYPE_BITS;
+		DWORD64 Reserved : 64 - MEM_EXTENDED_PARAMETER_TYPE_BITS;
+	} DUMMYSTRUCTNAME;
+	union {
+		DWORD64 ULong64;
+		PVOID   Pointer;
+		SIZE_T  Size;
+		HANDLE  Handle;
+		DWORD   ULong;
+	} DUMMYUNIONNAME;
+} MEM_EXTENDED_PARAMETER, *PMEM_EXTENDED_PARAMETER;
+
+/* https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-virtualalloc2 */
+typedef PVOID (*VirtualAlloc2_type)(
+	HANDLE                 Process,
+	PVOID                  BaseAddress,
+	SIZE_T                 Size,
+	ULONG                  AllocationType,
+	ULONG                  PageProtection,
+	MEM_EXTENDED_PARAMETER *ExtendedParameters,
+	ULONG                  ParameterCount
+);
+
+/* VirtualAlloc2() flags. */
+#define MEM_COALESCE_PLACEHOLDERS 0x00000001
+#define MEM_PRESERVE_PLACEHOLDER  0x00000002
+#define MEM_REPLACE_PLACEHOLDER   0x00004000
+#define MEM_RESERVE_PLACEHOLDER   0x00040000
+
+/* Named exactly as the function, so that user code does not depend
+ * on it being found at compile time or dynamically.
+ */
+static VirtualAlloc2_type VirtualAlloc2;
+
+int
+eal_mem_win32api_init(void)
+{
+	static const char library_name[] = "kernelbase.dll";
+	static const char function[] = "VirtualAlloc2";
+
+	OSVERSIONINFO info;
+	HMODULE library = NULL;
+	int ret = 0;
+
+	/* Already done. */
+	if (VirtualAlloc2 != NULL)
+		return 0;
+
+	/* IsWindows10OrGreater() may also be unavailable. */
+	memset(&info, 0, sizeof(info));
+	info.dwOSVersionInfoSize = sizeof(info);
+	GetVersionEx(&info);
+
+	/* Checking for Windows 10+ will also detect Windows Server 2016+.
+	 * Do not abort, because Windows may report false version depending
+	 * on executable manifest, compatibility mode, etc.
+	 */
+	if (info.dwMajorVersion < 10)
+		RTE_LOG(DEBUG, EAL, "Windows 10+ or Windows Server 2016+ "
+			"is required for advanced memory features\n");
+
+	library = LoadLibraryA(library_name);
+	if (library == NULL) {
+		RTE_LOG_WIN32_ERR("LoadLibraryA(\"%s\")", library_name);
+		return -1;
+	}
+
+	VirtualAlloc2 = (VirtualAlloc2_type)(
+		(void *)GetProcAddress(library, function));
+	if (VirtualAlloc2 == NULL) {
+		RTE_LOG_WIN32_ERR("GetProcAddress(\"%s\", \"%s\")\n",
+			library_name, function);
+		ret = -1;
+	}
+
+	FreeLibrary(library);
+
+	return ret;
+}
+

lib/librte_eal/windows/eal_memory.c line 121

Quoted 10 lines
lib/librte_eal/windows/eal_memory.c new file +10 -0
+#else
+
+/* Stub in case VirtualAlloc2() is provided by the compiler. */
+int
+eal_mem_win32api_init(void)
+{
+	return 0;
+}
+
+#endif /* no VirtualAlloc2() */

Can you fix this comment to match the #ifndef definition above? BTW...Why use MEM_PRESERVE_PLACEHOLDER (which is also defined within the block?)

ranjit m.

#53 lore raw

Re: [dpdk-dev] [PATCH v3 06/10] eal: introduce memory management wrappers

In reply to Dmitry Kozlyuk

<Snip!>

lib/librte_eal/windows/eal_memory.c

Quoted 6 lines · Dmitry Kozlyuk
lib/librte_eal/windows/eal_memory.c new file +0 -0
new file mode 100644
index 000000000..5697187ce
@@ -0,0 +1,437 @@

<Snip!>

lib/librte_eal/windows/eal_memory.c line 171

Quoted 3 lines
lib/librte_eal/windows/eal_memory.c new file +3 -0
+
+	if ((flags & EAL_RESERVE_EXACT_ADDRESS) && (virt != requested_addr)) {
+		if (!VirtualFree(virt, 0, MEM_RELEASE))

Microsoft documentation suggests that we use VirtualFreeEx() to free memory allocated by VirtualAlloc2(). VirtualFreeEx() would require the handle to the current process that was passed into VirtualAlloc2()

There are 6 other such occurrences in this file.

ranjit m.

#54 lore raw

PATCH v3 7/10 eal: extract common code for memseg list initialization

In reply to Dmitry Kozlyuk

All supported OS create memory segment lists (MSL) and reserve VA space for them in a nearly identical way. Move common code into EAL private functions to reduce duplication.

---
 lib/librte_eal/common/eal_common_memory.c | 54 ++++++++++++++++++
 lib/librte_eal/common/eal_private.h       | 34 ++++++++++++
 lib/librte_eal/freebsd/eal_memory.c       | 54 +++---------------
 lib/librte_eal/linux/eal_memory.c         | 68 +++++------------------
 4 files changed, 110 insertions(+), 100 deletions(-)
lib/librte_eal/common/eal_common_memory.c +54 -0
index cc7d54e0c..d9764681a 100644
@@ -25,6 +25,7 @@
 #include "eal_private.h"
 #include "eal_internal_cfg.h"
 #include "eal_memcfg.h"
+#include "eal_options.h"
 #include "malloc_heap.h"
 
 /*
@@ -182,6 +183,59 @@ eal_get_virtual_area(void *requested_addr, size_t *size,
 	return aligned_addr;
 }
 
+int
+eal_reserve_memseg_list(struct rte_memseg_list *msl,
+		enum eal_mem_reserve_flags flags)
+{
+	uint64_t page_sz;
+	size_t mem_sz;
+	void *addr;
+
+	page_sz = msl->page_sz;
+	mem_sz = page_sz * msl->memseg_arr.len;
+
+	addr = eal_get_virtual_area(msl->base_va, &mem_sz, page_sz, 0, flags);
+	if (addr == NULL) {
+		if (rte_errno == EADDRNOTAVAIL)
+			RTE_LOG(ERR, EAL, "Cannot reserve %llu bytes at [%p] - "
+				"please use '--" OPT_BASE_VIRTADDR "' option\n",
+				(unsigned long long)mem_sz, msl->base_va);
+		else
+			RTE_LOG(ERR, EAL, "Cannot reserve memory\n");
+		return -1;
+	}
+	msl->base_va = addr;
+	msl->len = mem_sz;
+
+	return 0;
+}
+
+int
+eal_alloc_memseg_list(struct rte_memseg_list *msl, uint64_t page_sz,
+		int n_segs, int socket_id, int type_msl_idx, bool heap)
+{
+	char name[RTE_FBARRAY_NAME_LEN];
+
+	snprintf(name, sizeof(name), MEMSEG_LIST_FMT, page_sz >> 10, socket_id,
+		 type_msl_idx);
+	if (rte_fbarray_init(&msl->memseg_arr, name, n_segs,
+			sizeof(struct rte_memseg))) {
+		RTE_LOG(ERR, EAL, "Cannot allocate memseg list: %s\n",
+			rte_strerror(rte_errno));
+		return -1;
+	}
+
+	msl->page_sz = page_sz;
+	msl->socket_id = socket_id;
+	msl->base_va = NULL;
+	msl->heap = heap;
+
+	RTE_LOG(DEBUG, EAL, "Memseg list allocated: 0x%zxkB at socket %i\n",
+			(size_t)page_sz >> 10, socket_id);
+
+	return 0;
+}
+
 static struct rte_memseg *
 virt2memseg(const void *addr, const struct rte_memseg_list *msl)
 {
lib/librte_eal/common/eal_private.h +34 -0
index 1e89338f2..76938e379 100644
@@ -246,6 +246,40 @@ void *
 eal_get_virtual_area(void *requested_addr, size_t *size, size_t page_sz,
 	int flags, int mmap_flags);
 
+/**
+ * Reserve VA space for a memory segment list.
+ *
+ * @param msl
+ *  Memory segment list with page size defined.
+ * @param flags
+ *  Extra memory reservation flags. Can be 0 if unnecessary.
+ * @return
+ *  0 on success, (-1) on failure and rte_errno is set.
+ */
+int
+eal_reserve_memseg_list(struct rte_memseg_list *msl,
+	enum eal_mem_reserve_flags flags);
+
+/**
+ * Initialize a memory segment list with its backing storage.
+ *
+ * @param msl
+ *  Memory segment list to be filled.
+ * @param page_sz
+ *  Size of segment pages in the MSL.
+ * @param n_segs
+ *  Number of segments.
+ * @param socket_id
+ *  Socket ID. Must not be SOCKET_ID_ANY.
+ * @param type_msl_idx
+ *  Index of the MSL among other MSLs of the same socket and page size.
+ * @param heap
+ *  Mark MSL as pointing to a heap.
+ */
+int
+eal_alloc_memseg_list(struct rte_memseg_list *msl, uint64_t page_sz,
+	int n_segs, int socket_id, int type_msl_idx, bool heap);
+
 /**
  * Get cpu core_id.
  *
lib/librte_eal/freebsd/eal_memory.c +8 -46

76 lines not on this page · raw message

lib/librte_eal/linux/eal_memory.c +14 -54

129 lines not on this page · raw message

Signature · 1 line
2.25.1
#55 lore raw

Re: [dpdk-dev] [PATCH v3 07/10] eal: extract common code for memseg list initialization

14/04/2020 21:44, Dmitry Kozlyuk:

Quoted 11 lines

All supported OS create memory segment lists (MSL) and reserve VA space for them in a nearly identical way. Move common code into EAL private functions to reduce duplication.

---
 lib/librte_eal/common/eal_common_memory.c | 54 ++++++++++++++++++
 lib/librte_eal/common/eal_private.h       | 34 ++++++++++++
 lib/librte_eal/freebsd/eal_memory.c       | 54 +++---------------
 lib/librte_eal/linux/eal_memory.c         | 68 +++++------------------
 4 files changed, 110 insertions(+), 100 deletions(-)

Didn't review this change, but thanks for doing such cleanup.

#56 lore raw

Re: [dpdk-dev] [PATCH v3 07/10] eal: extract common code for memseg list initialization

In reply to Dmitry Kozlyuk
Quoted 25 lines · Dmitry Kozlyuk

All supported OS create memory segment lists (MSL) and reserve VA space for them in a nearly identical way. Move common code into EAL private functions to reduce duplication.

---
  lib/librte_eal/common/eal_common_memory.c | 54 ++++++++++++++++++
  lib/librte_eal/common/eal_private.h       | 34 ++++++++++++
  lib/librte_eal/freebsd/eal_memory.c       | 54 +++---------------
  lib/librte_eal/linux/eal_memory.c         | 68 +++++------------------
  4 files changed, 110 insertions(+), 100 deletions(-)
lib/librte_eal/common/eal_common_memory.c +1 -0
index cc7d54e0c..d9764681a 100644
@@ -25,6 +25,7 @@
  #include "eal_private.h"
  #include "eal_internal_cfg.h"
  #include "eal_memcfg.h"
+#include "eal_options.h"
  #include "malloc_heap.h"
 
  /*

lib/librte_eal/common/eal_common_memory.c eal_get_virtual_area(void *requested_addr, size_t *size, · line 188

Quoted 6 lines
lib/librte_eal/common/eal_common_memory.c +3 -0
@@ -182,6 +183,59 @@ eal_get_virtual_area(void *requested_addr, size_t *size,
  	return aligned_addr;
  }
 
+int
+eal_reserve_memseg_list(struct rte_memseg_list *msl,
+		enum eal_mem_reserve_flags flags)

This and other similar places in this and other patches: i don't think using enums for this purpose (i.e. to hold multiple values at once) is good practice. I would suggest replacing with int.

Also, i don't think "eal_reserve_memseg_list" is a particularly good or descriptive name (and neither is "eal_alloc_memseg_list" for that matter). Suggestion: "eal_memseg_list_create" (or "_init") and "eal_memseg_list_alloc"?

Quoted 32 lines
lib/librte_eal/common/eal_common_memory.c +32 -0
+{
+	uint64_t page_sz;
+	size_t mem_sz;
+	void *addr;
+
+	page_sz = msl->page_sz;
+	mem_sz = page_sz * msl->memseg_arr.len;
+
+	addr = eal_get_virtual_area(msl->base_va, &mem_sz, page_sz, 0, flags);
+	if (addr == NULL) {
+		if (rte_errno == EADDRNOTAVAIL)
+			RTE_LOG(ERR, EAL, "Cannot reserve %llu bytes at [%p] - "
+				"please use '--" OPT_BASE_VIRTADDR "' option\n",
+				(unsigned long long)mem_sz, msl->base_va);
+		else
+			RTE_LOG(ERR, EAL, "Cannot reserve memory\n");
+		return -1;
+	}
+	msl->base_va = addr;
+	msl->len = mem_sz;
+
+	return 0;
+}
+
+int
+eal_alloc_memseg_list(struct rte_memseg_list *msl, uint64_t page_sz,
+		int n_segs, int socket_id, int type_msl_idx, bool heap)
+{
+	char name[RTE_FBARRAY_NAME_LEN];
+
+	snprintf(name, sizeof(name), MEMSEG_LIST_FMT, page_sz >> 10, socket_id,
+		 type_msl_idx);

lib/librte_eal/common/eal_common_memory.c eal_get_virtual_area(void *requested_addr, size_t *size, · line 230

Quoted 10 lines
lib/librte_eal/common/eal_common_memory.c +10 -0
+	if (rte_fbarray_init(&msl->memseg_arr, name, n_segs,
+			sizeof(struct rte_memseg))) {
+		RTE_LOG(ERR, EAL, "Cannot allocate memseg list: %s\n",
+			rte_strerror(rte_errno));
+		return -1;
+	}
+
+	msl->page_sz = page_sz;
+	msl->socket_id = socket_id;
+	msl->base_va = NULL;

It probably needs to be documented that eal_alloc_memseg_list must be called before eal_reserve_memseg_list.

Quoted 11 lines
lib/librte_eal/common/eal_common_memory.c +8 -0
+	msl->heap = heap;
+
+	RTE_LOG(DEBUG, EAL, "Memseg list allocated: 0x%zxkB at socket %i\n",
+			(size_t)page_sz >> 10, socket_id);
+
+	return 0;
+}
+
  static struct rte_memseg *
  virt2memseg(const void *addr, const struct rte_memseg_list *msl)
  {
Signature · 2 lines
Thanks,
Anatoly
#57 lore raw

PATCH v3 8/10 eal/windows: fix rte_page_sizes with Clang on Windows

In reply to Dmitry Kozlyuk

Clang on Windows follows MS ABI where enum values are limited to 2^31-1. Enum rte_page_size has members valued above this limit, which get wrapped to zero, resulting in compilation error (duplicate values in enum). Using MS ABI is mandatory for Windows EAL to call Win32 APIs.

Define these values outside of the enum for Clang on Windows only. This does not affect runtime, because Windows doesn't run on machines with 4GiB and 16GiB hugepages.

---
 lib/librte_eal/include/rte_memory.h | 6 ++++++
 1 file changed, 6 insertions(+)
lib/librte_eal/include/rte_memory.h +6 -0
index 1b7c3e5df..3ec673f51 100644
@@ -34,8 +34,14 @@ enum rte_page_sizes {
 	RTE_PGSIZE_256M  = 1ULL << 28,
 	RTE_PGSIZE_512M  = 1ULL << 29,
 	RTE_PGSIZE_1G    = 1ULL << 30,
+/* Work around Clang on Windows being limited to 32-bit underlying type. */
+#if !defined(RTE_CC_CLANG) || !defined(RTE_EXEC_ENV_WINDOWS)
 	RTE_PGSIZE_4G    = 1ULL << 32,
 	RTE_PGSIZE_16G   = 1ULL << 34,
+#else
+#define RTE_PGSIZE_4G  (1ULL << 32)
+#define RTE_PGSIZE_16G (1ULL << 34)
+#endif
 };
 
 #define SOCKET_ID_ANY -1                    /**< Any NUMA socket. */
Signature · 1 line
2.25.1
#58 lore raw

Re: [dpdk-dev] [PATCH v3 08/10] eal/windows: fix rte_page_sizes with Clang on Windows

Quoted 20 lines · Dmitry Kozlyuk

Clang on Windows follows MS ABI where enum values are limited to 2^31-1. Enum rte_page_size has members valued above this limit, which get wrapped to zero, resulting in compilation error (duplicate values in enum). Using MS ABI is mandatory for Windows EAL to call Win32 APIs.

Define these values outside of the enum for Clang on Windows only. This does not affect runtime, because Windows doesn't run on machines with 4GiB and 16GiB hugepages.

---
 lib/librte_eal/include/rte_memory.h | 6 ++++++
 1 file changed, 6 insertions(+)
lib/librte_eal/include/rte_memory.h +0 -0
index 1b7c3e5df..3ec673f51 100644

lib/librte_eal/include/rte_memory.h enum rte_page_sizes { · line 37

Quoted 4 lines
lib/librte_eal/include/rte_memory.h +1 -0
@@ -34,8 +34,14 @@ enum rte_page_sizes {
        RTE_PGSIZE_256M  = 1ULL << 28,
        RTE_PGSIZE_512M  = 1ULL << 29,
        RTE_PGSIZE_1G    = 1ULL << 30,
+/* Work around Clang on Windows being limited to 32-bit underlying type. */

It does look like "enum rte_page_sizes" NOT used as enum anywhere.

[master][dpdk.org] $ grep -ri "enum rte_page_sizes" lib/ lib/librte_eal/include/rte_memory.h:enum rte_page_sizes {

Why not remove this workaround and define all items as #define to avoid below ifdef clutter.

lib/librte_eal/include/rte_memory.h enum rte_page_sizes { · line 38

Quoted 1 line
lib/librte_eal/include/rte_memory.h +1 -0
+#if !defined(RTE_CC_CLANG) || !defined(RTE_EXEC_ENV_WINDOWS)

See above.

Quoted 12 lines
lib/librte_eal/include/rte_memory.h +4 -1
        RTE_PGSIZE_4G    = 1ULL << 32,
        RTE_PGSIZE_16G   = 1ULL << 34,
+#else
+#define RTE_PGSIZE_4G  (1ULL << 32)
+#define RTE_PGSIZE_16G (1ULL << 34)
+#endif
 };
 
 #define SOCKET_ID_ANY -1                    /**< Any NUMA socket. */
--

2.25.1

#59 lore raw

Re: [dpdk-dev] [PATCH v3 08/10] eal/windows: fix rte_page_sizes with Clang on Windows

Quoted 46 lines · Dmitry Kozlyuk
Quoted 20 lines

Clang on Windows follows MS ABI where enum values are limited to 2^31-1. Enum rte_page_size has members valued above this limit, which get wrapped to zero, resulting in compilation error (duplicate values in enum). Using MS ABI is mandatory for Windows EAL to call Win32 APIs.

Define these values outside of the enum for Clang on Windows only. This does not affect runtime, because Windows doesn't run on machines with 4GiB and 16GiB hugepages.

---
 lib/librte_eal/include/rte_memory.h | 6 ++++++
 1 file changed, 6 insertions(+)
lib/librte_eal/include/rte_memory.h +0 -0
index 1b7c3e5df..3ec673f51 100644

lib/librte_eal/include/rte_memory.h enum rte_page_sizes { · line 37

Quoted 4 lines
lib/librte_eal/include/rte_memory.h +1 -0
@@ -34,8 +34,14 @@ enum rte_page_sizes {
        RTE_PGSIZE_256M  = 1ULL << 28,
        RTE_PGSIZE_512M  = 1ULL << 29,
        RTE_PGSIZE_1G    = 1ULL << 30,
+/* Work around Clang on Windows being limited to 32-bit underlying type. */  

It does look like "enum rte_page_sizes" NOT used as enum anywhere.

[master][dpdk.org] $ grep -ri "enum rte_page_sizes" lib/ lib/librte_eal/include/rte_memory.h:enum rte_page_sizes {

Why not remove this workaround and define all items as #define to avoid below ifdef clutter.

lib/librte_eal/include/rte_memory.h enum rte_page_sizes { · line 38

Quoted 1 line
lib/librte_eal/include/rte_memory.h +1 -0
+#if !defined(RTE_CC_CLANG) || !defined(RTE_EXEC_ENV_WINDOWS)  

See above.

Quoted 8 lines
lib/librte_eal/include/rte_memory.h +4 -0
        RTE_PGSIZE_4G    = 1ULL << 32,
        RTE_PGSIZE_16G   = 1ULL << 34,
+#else
+#define RTE_PGSIZE_4G  (1ULL << 32)
+#define RTE_PGSIZE_16G (1ULL << 34)
+#endif
 };
 
Quoted 4 lines
Quoted 4 lines
lib/librte_eal/include/rte_memory.h +0 -1
 #define SOCKET_ID_ANY -1                    /**< Any NUMA socket. */
--

2.25.1

This is a public header and removing enum rte_page_sizes will break API. Moving members out of enum while keeping enum itself might break compilation because of integer constants being converted to enum (with -Werror).

Signature · 1 line
Dmitry Kozlyuk
#60 lore raw

Re: [dpdk-dev] [PATCH v3 08/10] eal/windows: fix rte_page_sizes with Clang on Windows

Quoted 47 lines · Dmitry Kozlyuk
Quoted 47 lines · Dmitry Kozlyuk
Quoted 20 lines

Clang on Windows follows MS ABI where enum values are limited to 2^31-1. Enum rte_page_size has members valued above this limit, which get wrapped to zero, resulting in compilation error (duplicate values in enum). Using MS ABI is mandatory for Windows EAL to call Win32 APIs.

Define these values outside of the enum for Clang on Windows only. This does not affect runtime, because Windows doesn't run on machines with 4GiB and 16GiB hugepages.

---
 lib/librte_eal/include/rte_memory.h | 6 ++++++
 1 file changed, 6 insertions(+)
lib/librte_eal/include/rte_memory.h +0 -0
index 1b7c3e5df..3ec673f51 100644

lib/librte_eal/include/rte_memory.h enum rte_page_sizes { · line 37

Quoted 4 lines
lib/librte_eal/include/rte_memory.h +1 -0
@@ -34,8 +34,14 @@ enum rte_page_sizes {
        RTE_PGSIZE_256M  = 1ULL << 28,
        RTE_PGSIZE_512M  = 1ULL << 29,
        RTE_PGSIZE_1G    = 1ULL << 30,
+/* Work around Clang on Windows being limited to 32-bit underlying type. */

It does look like "enum rte_page_sizes" NOT used as enum anywhere.

[master][dpdk.org] $ grep -ri "enum rte_page_sizes" lib/ lib/librte_eal/include/rte_memory.h:enum rte_page_sizes {

Why not remove this workaround and define all items as #define to avoid below ifdef clutter.

lib/librte_eal/include/rte_memory.h enum rte_page_sizes { · line 38

Quoted 1 line
lib/librte_eal/include/rte_memory.h +1 -0
+#if !defined(RTE_CC_CLANG) || !defined(RTE_EXEC_ENV_WINDOWS)

See above.

Quoted 8 lines
lib/librte_eal/include/rte_memory.h +4 -0
        RTE_PGSIZE_4G    = 1ULL << 32,
        RTE_PGSIZE_16G   = 1ULL << 34,
+#else
+#define RTE_PGSIZE_4G  (1ULL << 32)
+#define RTE_PGSIZE_16G (1ULL << 34)
+#endif
 };
 
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
lib/librte_eal/include/rte_memory.h +0 -1
 #define SOCKET_ID_ANY -1                    /**< Any NUMA socket. */
--

2.25.1

This is a public header and removing enum rte_page_sizes will break API. Moving members out of enum while keeping enum itself might break compilation because of integer constants being converted to enum (with -Werror).

If none of the public API is using this enum then I think, we may not need to make this enum as public.
Since it has ULL, I believe both cases(enum or define), it will be treated as unsigned long long. ie. NO ABI breakage.

Quoted 3 lines

--
Dmitry Kozlyuk

#61 lore raw

Re: [dpdk-dev] [PATCH v3 08/10] eal/windows: fix rte_page_sizes with Clang on Windows

Quoted 59 lines · Dmitry Kozlyuk
Quoted 47 lines
Quoted 47 lines · Dmitry Kozlyuk
Quoted 20 lines

Clang on Windows follows MS ABI where enum values are limited to 2^31-1. Enum rte_page_size has members valued above this limit, which get wrapped to zero, resulting in compilation error (duplicate values in enum). Using MS ABI is mandatory for Windows EAL to call Win32 APIs.

Define these values outside of the enum for Clang on Windows only. This does not affect runtime, because Windows doesn't run on machines with 4GiB and 16GiB hugepages.

---
 lib/librte_eal/include/rte_memory.h | 6 ++++++
 1 file changed, 6 insertions(+)
lib/librte_eal/include/rte_memory.h +0 -0
index 1b7c3e5df..3ec673f51 100644

lib/librte_eal/include/rte_memory.h enum rte_page_sizes { · line 37

Quoted 4 lines
lib/librte_eal/include/rte_memory.h +1 -0
@@ -34,8 +34,14 @@ enum rte_page_sizes {
        RTE_PGSIZE_256M  = 1ULL << 28,
        RTE_PGSIZE_512M  = 1ULL << 29,
        RTE_PGSIZE_1G    = 1ULL << 30,
+/* Work around Clang on Windows being limited to 32-bit underlying type. */  

It does look like "enum rte_page_sizes" NOT used as enum anywhere.

[master][dpdk.org] $ grep -ri "enum rte_page_sizes" lib/ lib/librte_eal/include/rte_memory.h:enum rte_page_sizes {

Why not remove this workaround and define all items as #define to avoid below ifdef clutter.

lib/librte_eal/include/rte_memory.h enum rte_page_sizes { · line 38

Quoted 1 line
lib/librte_eal/include/rte_memory.h +1 -0
+#if !defined(RTE_CC_CLANG) || !defined(RTE_EXEC_ENV_WINDOWS)  

See above.

Quoted 8 lines
lib/librte_eal/include/rte_memory.h +4 -0
        RTE_PGSIZE_4G    = 1ULL << 32,
        RTE_PGSIZE_16G   = 1ULL << 34,
+#else
+#define RTE_PGSIZE_4G  (1ULL << 32)
+#define RTE_PGSIZE_16G (1ULL << 34)
+#endif
 };
 
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
lib/librte_eal/include/rte_memory.h +0 -1
 #define SOCKET_ID_ANY -1                    /**< Any NUMA socket. */
--

2.25.1

This is a public header and removing enum rte_page_sizes will break API. Moving members out of enum while keeping enum itself might break compilation because of integer constants being converted to enum (with -Werror).

Quoted 2 lines

If none of the public API is using this enum then I think, we may not need to make this enum as public.

Agreed.

Quoted 2 lines

Since it has ULL, I believe both cases(enum or define), it will be treated as unsigned long long. ie. NO ABI breakage.

I was talking about API only (compile-time compatibility). Getting rid of #ifdef and workarounds sounds right, we'll just need a notice in release notes.

Signature · 1 line
Dmitry Kozlyuk
#62 lore raw

Re: [dpdk-dev] [PATCH v3 08/10] eal/windows: fix rte_page_sizes with Clang on Windows

Quoted 63 lines · Dmitry Kozlyuk
Quoted 60 lines · Dmitry Kozlyuk
Quoted 47 lines
Quoted 47 lines · Dmitry Kozlyuk
Quoted 20 lines

Clang on Windows follows MS ABI where enum values are limited to 2^31-1. Enum rte_page_size has members valued above this limit, which get wrapped to zero, resulting in compilation error (duplicate values in enum). Using MS ABI is mandatory for Windows EAL to call Win32 APIs.

Define these values outside of the enum for Clang on Windows only. This does not affect runtime, because Windows doesn't run on machines with 4GiB and 16GiB hugepages.

---
 lib/librte_eal/include/rte_memory.h | 6 ++++++
 1 file changed, 6 insertions(+)
lib/librte_eal/include/rte_memory.h +0 -0
index 1b7c3e5df..3ec673f51 100644

lib/librte_eal/include/rte_memory.h enum rte_page_sizes { · line 37

Quoted 4 lines
lib/librte_eal/include/rte_memory.h +1 -0
@@ -34,8 +34,14 @@ enum rte_page_sizes {
        RTE_PGSIZE_256M  = 1ULL << 28,
        RTE_PGSIZE_512M  = 1ULL << 29,
        RTE_PGSIZE_1G    = 1ULL << 30,
+/* Work around Clang on Windows being limited to 32-bit underlying type. */

It does look like "enum rte_page_sizes" NOT used as enum anywhere.

[master][dpdk.org] $ grep -ri "enum rte_page_sizes" lib/ lib/librte_eal/include/rte_memory.h:enum rte_page_sizes {

Why not remove this workaround and define all items as #define to avoid below ifdef clutter.

lib/librte_eal/include/rte_memory.h enum rte_page_sizes { · line 38

Quoted 1 line
lib/librte_eal/include/rte_memory.h +1 -0
+#if !defined(RTE_CC_CLANG) || !defined(RTE_EXEC_ENV_WINDOWS)

See above.

Quoted 8 lines
lib/librte_eal/include/rte_memory.h +4 -0
        RTE_PGSIZE_4G    = 1ULL << 32,
        RTE_PGSIZE_16G   = 1ULL << 34,
+#else
+#define RTE_PGSIZE_4G  (1ULL << 32)
+#define RTE_PGSIZE_16G (1ULL << 34)
+#endif
 };
 
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
lib/librte_eal/include/rte_memory.h +0 -1
 #define SOCKET_ID_ANY -1                    /**< Any NUMA socket. */
--

2.25.1

This is a public header and removing enum rte_page_sizes will break API. Moving members out of enum while keeping enum itself might break compilation because of integer constants being converted to enum (with -Werror).

Quoted 2 lines

If none of the public API is using this enum then I think, we may not need to make this enum as public.

Quoted 8 lines

Agreed.

Quoted 2 lines

Since it has ULL, I believe both cases(enum or define), it will be treated as unsigned long long. ie. NO ABI breakage.

I was talking about API only (compile-time compatibility). Getting rid of #ifdef and workarounds sounds right, we'll just need a notice in release notes.

Good to check ./devtools/check-abi.sh for any ABI breakage.

Quoted 3 lines

--
Dmitry Kozlyuk

#63 lore raw

Re: [dpdk-dev] [PATCH v3 08/10] eal/windows: fix rte_page_sizes with Clang on Windows

Quoted 66 lines · Jerin Jacob
Quoted 65 lines · Dmitry Kozlyuk
Quoted 60 lines · Dmitry Kozlyuk
Quoted 47 lines
Quoted 47 lines · Dmitry Kozlyuk
Quoted 20 lines

Clang on Windows follows MS ABI where enum values are limited to 2^31-1. Enum rte_page_size has members valued above this limit, which get wrapped to zero, resulting in compilation error (duplicate values in enum). Using MS ABI is mandatory for Windows EAL to call Win32 APIs.

Define these values outside of the enum for Clang on Windows only. This does not affect runtime, because Windows doesn't run on machines with 4GiB and 16GiB hugepages.

---
 lib/librte_eal/include/rte_memory.h | 6 ++++++
 1 file changed, 6 insertions(+)
lib/librte_eal/include/rte_memory.h +0 -0
index 1b7c3e5df..3ec673f51 100644

lib/librte_eal/include/rte_memory.h enum rte_page_sizes { · line 37

Quoted 4 lines
lib/librte_eal/include/rte_memory.h +1 -0
@@ -34,8 +34,14 @@ enum rte_page_sizes {
        RTE_PGSIZE_256M  = 1ULL << 28,
        RTE_PGSIZE_512M  = 1ULL << 29,
        RTE_PGSIZE_1G    = 1ULL << 30,
+/* Work around Clang on Windows being limited to 32-bit underlying type. */

It does look like "enum rte_page_sizes" NOT used as enum anywhere.

[master][dpdk.org] $ grep -ri "enum rte_page_sizes" lib/ lib/librte_eal/include/rte_memory.h:enum rte_page_sizes {

Why not remove this workaround and define all items as #define to avoid below ifdef clutter.

lib/librte_eal/include/rte_memory.h enum rte_page_sizes { · line 38

Quoted 1 line
lib/librte_eal/include/rte_memory.h +1 -0
+#if !defined(RTE_CC_CLANG) || !defined(RTE_EXEC_ENV_WINDOWS)

See above.

Quoted 8 lines
lib/librte_eal/include/rte_memory.h +4 -0
        RTE_PGSIZE_4G    = 1ULL << 32,
        RTE_PGSIZE_16G   = 1ULL << 34,
+#else
+#define RTE_PGSIZE_4G  (1ULL << 32)
+#define RTE_PGSIZE_16G (1ULL << 34)
+#endif
 };
 
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
lib/librte_eal/include/rte_memory.h +0 -1
 #define SOCKET_ID_ANY -1                    /**< Any NUMA socket. */
--

2.25.1

This is a public header and removing enum rte_page_sizes will break API. Moving members out of enum while keeping enum itself might break compilation because of integer constants being converted to enum (with -Werror).

Quoted 2 lines

If none of the public API is using this enum then I think, we may not need to make this enum as public.

Agreed.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Since it has ULL, I believe both cases(enum or define), it will be treated as unsigned long long. ie. NO ABI breakage.

I was talking about API only (compile-time compatibility). Getting rid of #ifdef and workarounds sounds right, we'll just need a notice in release notes.

Good to check ./devtools/check-abi.sh for any ABI breakage.

or something like this to cover all the bases. DPDK_ABI_REF_DIR=/build/dpdk/reference/ DPDK_ABI_REF_VERSION=v20.02 ./devtools/test-meson-builds.sh

Quoted 3 lines
Quoted 3 lines

--
Dmitry Kozlyuk

#64 lore raw

PATCH v3 9/10 eal/windows: replace sys/queue.h with a complete one from FreeBSD

In reply to Dmitry Kozlyuk

Limited version imported previously lacks at least SLIST macros. Import a complete file from FreeBSD, since its license exception is already approved by Technical Board.

---
 lib/librte_eal/windows/include/sys/queue.h | 663 +++++++++++++++++++--
 1 file changed, 601 insertions(+), 62 deletions(-)
lib/librte_eal/windows/include/sys/queue.h +601 -62
index a65949a78..9756bee6f 100644
@@ -8,7 +8,36 @@
 #define	_SYS_QUEUE_H_
 
 /*
- * This file defines tail queues.
+ * This file defines four types of data structures: singly-linked lists,
+ * singly-linked tail queues, lists and tail queues.
+ *
+ * A singly-linked list is headed by a single forward pointer. The elements
+ * are singly linked for minimum space and pointer manipulation overhead at
+ * the expense of O(n) removal for arbitrary elements. New elements can be
+ * added to the list after an existing element or at the head of the list.
+ * Elements being removed from the head of the list should use the explicit
+ * macro for this purpose for optimum efficiency. A singly-linked list may
+ * only be traversed in the forward direction.  Singly-linked lists are ideal
+ * for applications with large datasets and few or no removals or for
+ * implementing a LIFO queue.
+ *
+ * A singly-linked tail queue is headed by a pair of pointers, one to the
+ * head of the list and the other to the tail of the list. The elements are
+ * singly linked for minimum space and pointer manipulation overhead at the
+ * expense of O(n) removal for arbitrary elements. New elements can be added
+ * to the list after an existing element, at the head of the list, or at the
+ * end of the list. Elements being removed from the head of the tail queue
+ * should use the explicit macro for this purpose for optimum efficiency.
+ * A singly-linked tail queue may only be traversed in the forward direction.
+ * Singly-linked tail queues are ideal for applications with large datasets
+ * and few or no removals or for implementing a FIFO queue.
+ *
+ * A list is headed by a single forward pointer (or an array of forward
+ * pointers for a hash table header). The elements are doubly linked
+ * so that an arbitrary element can be removed without a need to
+ * traverse the list. New elements can be added to the list before
+ * or after an existing element or at the head of the list. A list
+ * may be traversed in either direction.
  *
  * A tail queue is headed by a pair of pointers, one to the head of the
  * list and the other to the tail of the list. The elements are doubly
@@ -17,65 +46,93 @@
  * after an existing element, at the head of the list, or at the end of
  * the list. A tail queue may be traversed in either direction.
  *
+ * For details on the use of these macros, see the queue(3) manual page.
+ *
  * Below is a summary of implemented functions where:
  *  +  means the macro is available
  *  -  means the macro is not available
  *  s  means the macro is available but is slow (runs in O(n) time)
  *
- *				TAILQ
- * _HEAD			+
- * _CLASS_HEAD			+
- * _HEAD_INITIALIZER		+
- * _ENTRY			+
- * _CLASS_ENTRY			+
- * _INIT			+
- * _EMPTY			+
- * _FIRST			+
- * _NEXT			+
- * _PREV			+
- * _LAST			+
- * _LAST_FAST			+
- * _FOREACH			+
- * _FOREACH_FROM		+
- * _FOREACH_SAFE		+
- * _FOREACH_FROM_SAFE		+
- * _FOREACH_REVERSE		+
- * _FOREACH_REVERSE_FROM	+
- * _FOREACH_REVERSE_SAFE	+
- * _FOREACH_REVERSE_FROM_SAFE	+
- * _INSERT_HEAD			+
- * _INSERT_BEFORE		+
- * _INSERT_AFTER		+
- * _INSERT_TAIL			+
- * _CONCAT			+
- * _REMOVE_AFTER		-
- * _REMOVE_HEAD			-
- * _REMOVE			+
- * _SWAP			+
+ *				SLIST	LIST	STAILQ	TAILQ
+ * _HEAD			+	+	+	+
+ * _CLASS_HEAD			+	+	+	+
+ * _HEAD_INITIALIZER		+	+	+	+
+ * _ENTRY			+	+	+	+
+ * _CLASS_ENTRY			+	+	+	+
+ * _INIT			+	+	+	+
+ * _EMPTY			+	+	+	+
+ * _FIRST			+	+	+	+
+ * _NEXT			+	+	+	+
+ * _PREV			-	+	-	+
+ * _LAST			-	-	+	+
+ * _LAST_FAST			-	-	-	+
+ * _FOREACH			+	+	+	+
+ * _FOREACH_FROM		+	+	+	+
+ * _FOREACH_SAFE		+	+	+	+
+ * _FOREACH_FROM_SAFE		+	+	+	+
+ * _FOREACH_REVERSE		-	-	-	+
+ * _FOREACH_REVERSE_FROM	-	-	-	+
+ * _FOREACH_REVERSE_SAFE	-	-	-	+
+ * _FOREACH_REVERSE_FROM_SAFE	-	-	-	+
+ * _INSERT_HEAD			+	+	+	+
+ * _INSERT_BEFORE		-	+	-	+
+ * _INSERT_AFTER		+	+	+	+
+ * _INSERT_TAIL			-	-	+	+
+ * _CONCAT			s	s	+	+
+ * _REMOVE_AFTER		+	-	+	-
+ * _REMOVE_HEAD			+	-	+	-
+ * _REMOVE			s	+	s	+
+ * _SWAP			+	+	+	+
  *
  */
-
-#ifdef __cplusplus
-extern "C" {
+#ifdef QUEUE_MACRO_DEBUG
+#warn Use QUEUE_MACRO_DEBUG_TRACE and/or QUEUE_MACRO_DEBUG_TRASH
+#define	QUEUE_MACRO_DEBUG_TRACE
+#define	QUEUE_MACRO_DEBUG_TRASH
 #endif
 
-/*
- * List definitions.
- */
-#define	LIST_HEAD(name, type)						\
-struct name {								\
-	struct type *lh_first;	/* first element */			\
-}
+#ifdef QUEUE_MACRO_DEBUG_TRACE
+/* Store the last 2 places the queue element or head was altered */
+struct qm_trace {
+	unsigned long	 lastline;
+	unsigned long	 prevline;
+	const char	*lastfile;
+	const char	*prevfile;
+};
+
+#define	TRACEBUF	struct qm_trace trace;
+#define	TRACEBUF_INITIALIZER	{ __LINE__, 0, __FILE__, NULL } ,
+
+#define	QMD_TRACE_HEAD(head) do {					\
+	(head)->trace.prevline = (head)->trace.lastline;		\
+	(head)->trace.prevfile = (head)->trace.lastfile;		\
+	(head)->trace.lastline = __LINE__;				\
+	(head)->trace.lastfile = __FILE__;				\
+} while (0)
 
+#define	QMD_TRACE_ELEM(elem) do {					\
+	(elem)->trace.prevline = (elem)->trace.lastline;		\
+	(elem)->trace.prevfile = (elem)->trace.lastfile;		\
+	(elem)->trace.lastline = __LINE__;				\
+	(elem)->trace.lastfile = __FILE__;				\
+} while (0)
+
+#else	/* !QUEUE_MACRO_DEBUG_TRACE */
 #define	QMD_TRACE_ELEM(elem)
 #define	QMD_TRACE_HEAD(head)
 #define	TRACEBUF
 #define	TRACEBUF_INITIALIZER
+#endif	/* QUEUE_MACRO_DEBUG_TRACE */
 
+#ifdef QUEUE_MACRO_DEBUG_TRASH
+#define	QMD_SAVELINK(name, link)	void **name = (void *)&(link)
+#define	TRASHIT(x)		do {(x) = (void *)-1;} while (0)
+#define	QMD_IS_TRASHED(x)	((x) == (void *)(intptr_t)-1)
+#else	/* !QUEUE_MACRO_DEBUG_TRASH */
+#define	QMD_SAVELINK(name, link)
 #define	TRASHIT(x)
 #define	QMD_IS_TRASHED(x)	0
-
-#define	QMD_SAVELINK(name, link)
+#endif	/* QUEUE_MACRO_DEBUG_TRASH */
 
 #ifdef __cplusplus
 /*
@@ -86,6 +143,445 @@ struct name {								\
 #define	QUEUE_TYPEOF(type) struct type
 #endif
 
+/*
+ * Singly-linked List declarations.
+ */
+#define	SLIST_HEAD(name, type)						\
+struct name {								\
+	struct type *slh_first;	/* first element */			\
+}
+
+#define	SLIST_CLASS_HEAD(name, type)					\
+struct name {								\
+	class type *slh_first;	/* first element */			\
+}
+
+#define	SLIST_HEAD_INITIALIZER(head)					\
+	{ NULL }
+
+#define	SLIST_ENTRY(type)						\
+struct {								\
+	struct type *sle_next;	/* next element */			\
+}
+
+#define	SLIST_CLASS_ENTRY(type)						\
+struct {								\
+	class type *sle_next;		/* next element */		\
+}
+
+/*
+ * Singly-linked List functions.
+ */
+#if (defined(_KERNEL) && defined(INVARIANTS))
+#define	QMD_SLIST_CHECK_PREVPTR(prevp, elm) do {			\
+	if (*(prevp) != (elm))						\
+		panic("Bad prevptr *(%p) == %p != %p",			\
+		    (prevp), *(prevp), (elm));				\
+} while (0)
+#else
+#define	QMD_SLIST_CHECK_PREVPTR(prevp, elm)
+#endif
+
+#define SLIST_CONCAT(head1, head2, type, field) do {			\
+	QUEUE_TYPEOF(type) *curelm = SLIST_FIRST(head1);		\
+	if (curelm == NULL) {						\
+		if ((SLIST_FIRST(head1) = SLIST_FIRST(head2)) != NULL)	\
+			SLIST_INIT(head2);				\
+	} else if (SLIST_FIRST(head2) != NULL) {			\
+		while (SLIST_NEXT(curelm, field) != NULL)		\
+			curelm = SLIST_NEXT(curelm, field);		\
+		SLIST_NEXT(curelm, field) = SLIST_FIRST(head2);		\
+		SLIST_INIT(head2);					\
+	}								\
+} while (0)
+
+#define	SLIST_EMPTY(head)	((head)->slh_first == NULL)
+
+#define	SLIST_FIRST(head)	((head)->slh_first)
+
+#define	SLIST_FOREACH(var, head, field)					\
+	for ((var) = SLIST_FIRST((head));				\
+	    (var);							\
+	    (var) = SLIST_NEXT((var), field))
+
+#define	SLIST_FOREACH_FROM(var, head, field)				\
+	for ((var) = ((var) ? (var) : SLIST_FIRST((head)));		\
+	    (var);							\
+	    (var) = SLIST_NEXT((var), field))
+
+#define	SLIST_FOREACH_SAFE(var, head, field, tvar)			\
+	for ((var) = SLIST_FIRST((head));				\
+	    (var) && ((tvar) = SLIST_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	SLIST_FOREACH_FROM_SAFE(var, head, field, tvar)			\
+	for ((var) = ((var) ? (var) : SLIST_FIRST((head)));		\
+	    (var) && ((tvar) = SLIST_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	SLIST_FOREACH_PREVPTR(var, varp, head, field)			\
+	for ((varp) = &SLIST_FIRST((head));				\
+	    ((var) = *(varp)) != NULL;					\
+	    (varp) = &SLIST_NEXT((var), field))
+
+#define	SLIST_INIT(head) do {						\
+	SLIST_FIRST((head)) = NULL;					\
+} while (0)
+
+#define	SLIST_INSERT_AFTER(slistelm, elm, field) do {			\
+	SLIST_NEXT((elm), field) = SLIST_NEXT((slistelm), field);	\
+	SLIST_NEXT((slistelm), field) = (elm);				\
+} while (0)
+
+#define	SLIST_INSERT_HEAD(head, elm, field) do {			\
+	SLIST_NEXT((elm), field) = SLIST_FIRST((head));			\
+	SLIST_FIRST((head)) = (elm);					\
+} while (0)
+
+#define	SLIST_NEXT(elm, field)	((elm)->field.sle_next)
+
+#define	SLIST_REMOVE(head, elm, type, field) do {			\
+	QMD_SAVELINK(oldnext, (elm)->field.sle_next);			\
+	if (SLIST_FIRST((head)) == (elm)) {				\
+		SLIST_REMOVE_HEAD((head), field);			\
+	}								\
+	else {								\
+		QUEUE_TYPEOF(type) *curelm = SLIST_FIRST(head);		\
+		while (SLIST_NEXT(curelm, field) != (elm))		\
+			curelm = SLIST_NEXT(curelm, field);		\
+		SLIST_REMOVE_AFTER(curelm, field);			\
+	}								\
+	TRASHIT(*oldnext);						\
+} while (0)
+
+#define SLIST_REMOVE_AFTER(elm, field) do {				\
+	SLIST_NEXT(elm, field) =					\
+	    SLIST_NEXT(SLIST_NEXT(elm, field), field);			\
+} while (0)
+
+#define	SLIST_REMOVE_HEAD(head, field) do {				\
+	SLIST_FIRST((head)) = SLIST_NEXT(SLIST_FIRST((head)), field);	\
+} while (0)
+
+#define	SLIST_REMOVE_PREVPTR(prevp, elm, field) do {			\
+	QMD_SLIST_CHECK_PREVPTR(prevp, elm);				\
+	*(prevp) = SLIST_NEXT(elm, field);				\
+	TRASHIT((elm)->field.sle_next);					\
+} while (0)
+
+#define SLIST_SWAP(head1, head2, type) do {				\
+	QUEUE_TYPEOF(type) *swap_first = SLIST_FIRST(head1);		\
+	SLIST_FIRST(head1) = SLIST_FIRST(head2);			\
+	SLIST_FIRST(head2) = swap_first;				\
+} while (0)
+
+/*
+ * Singly-linked Tail queue declarations.
+ */
+#define	STAILQ_HEAD(name, type)						\
+struct name {								\
+	struct type *stqh_first;/* first element */			\
+	struct type **stqh_last;/* addr of last next element */		\
+}
+
+#define	STAILQ_CLASS_HEAD(name, type)					\
+struct name {								\
+	class type *stqh_first;	/* first element */			\
+	class type **stqh_last;	/* addr of last next element */		\
+}
+
+#define	STAILQ_HEAD_INITIALIZER(head)					\
+	{ NULL, &(head).stqh_first }
+
+#define	STAILQ_ENTRY(type)						\
+struct {								\
+	struct type *stqe_next;	/* next element */			\
+}
+
+#define	STAILQ_CLASS_ENTRY(type)					\
+struct {								\
+	class type *stqe_next;	/* next element */			\
+}
+
+/*
+ * Singly-linked Tail queue functions.
+ */
+#define	STAILQ_CONCAT(head1, head2) do {				\
+	if (!STAILQ_EMPTY((head2))) {					\
+		*(head1)->stqh_last = (head2)->stqh_first;		\
+		(head1)->stqh_last = (head2)->stqh_last;		\
+		STAILQ_INIT((head2));					\
+	}								\
+} while (0)
+
+#define	STAILQ_EMPTY(head)	((head)->stqh_first == NULL)
+
+#define	STAILQ_FIRST(head)	((head)->stqh_first)
+
+#define	STAILQ_FOREACH(var, head, field)				\
+	for((var) = STAILQ_FIRST((head));				\
+	   (var);							\
+	   (var) = STAILQ_NEXT((var), field))
+
+#define	STAILQ_FOREACH_FROM(var, head, field)				\
+	for ((var) = ((var) ? (var) : STAILQ_FIRST((head)));		\
+	   (var);							\
+	   (var) = STAILQ_NEXT((var), field))
+
+#define	STAILQ_FOREACH_SAFE(var, head, field, tvar)			\
+	for ((var) = STAILQ_FIRST((head));				\
+	    (var) && ((tvar) = STAILQ_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	STAILQ_FOREACH_FROM_SAFE(var, head, field, tvar)		\
+	for ((var) = ((var) ? (var) : STAILQ_FIRST((head)));		\
+	    (var) && ((tvar) = STAILQ_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	STAILQ_INIT(head) do {						\
+	STAILQ_FIRST((head)) = NULL;					\
+	(head)->stqh_last = &STAILQ_FIRST((head));			\
+} while (0)
+
+#define	STAILQ_INSERT_AFTER(head, tqelm, elm, field) do {		\
+	if ((STAILQ_NEXT((elm), field) = STAILQ_NEXT((tqelm), field)) == NULL)\
+		(head)->stqh_last = &STAILQ_NEXT((elm), field);		\
+	STAILQ_NEXT((tqelm), field) = (elm);				\
+} while (0)
+
+#define	STAILQ_INSERT_HEAD(head, elm, field) do {			\
+	if ((STAILQ_NEXT((elm), field) = STAILQ_FIRST((head))) == NULL)	\
+		(head)->stqh_last = &STAILQ_NEXT((elm), field);		\
+	STAILQ_FIRST((head)) = (elm);					\
+} while (0)
+
+#define	STAILQ_INSERT_TAIL(head, elm, field) do {			\
+	STAILQ_NEXT((elm), field) = NULL;				\
+	*(head)->stqh_last = (elm);					\
+	(head)->stqh_last = &STAILQ_NEXT((elm), field);			\
+} while (0)
+
+#define	STAILQ_LAST(head, type, field)				\
+	(STAILQ_EMPTY((head)) ? NULL :				\
+	    __containerof((head)->stqh_last,			\
+	    QUEUE_TYPEOF(type), field.stqe_next))
+
+#define	STAILQ_NEXT(elm, field)	((elm)->field.stqe_next)
+
+#define	STAILQ_REMOVE(head, elm, type, field) do {			\
+	QMD_SAVELINK(oldnext, (elm)->field.stqe_next);			\
+	if (STAILQ_FIRST((head)) == (elm)) {				\
+		STAILQ_REMOVE_HEAD((head), field);			\
+	}								\
+	else {								\
+		QUEUE_TYPEOF(type) *curelm = STAILQ_FIRST(head);	\
+		while (STAILQ_NEXT(curelm, field) != (elm))		\
+			curelm = STAILQ_NEXT(curelm, field);		\
+		STAILQ_REMOVE_AFTER(head, curelm, field);		\
+	}								\
+	TRASHIT(*oldnext);						\
+} while (0)
+
+#define STAILQ_REMOVE_AFTER(head, elm, field) do {			\
+	if ((STAILQ_NEXT(elm, field) =					\
+	     STAILQ_NEXT(STAILQ_NEXT(elm, field), field)) == NULL)	\
+		(head)->stqh_last = &STAILQ_NEXT((elm), field);		\
+} while (0)
+
+#define	STAILQ_REMOVE_HEAD(head, field) do {				\
+	if ((STAILQ_FIRST((head)) =					\
+	     STAILQ_NEXT(STAILQ_FIRST((head)), field)) == NULL)		\
+		(head)->stqh_last = &STAILQ_FIRST((head));		\
+} while (0)
+
+#define STAILQ_SWAP(head1, head2, type) do {				\
+	QUEUE_TYPEOF(type) *swap_first = STAILQ_FIRST(head1);		\
+	QUEUE_TYPEOF(type) **swap_last = (head1)->stqh_last;		\
+	STAILQ_FIRST(head1) = STAILQ_FIRST(head2);			\
+	(head1)->stqh_last = (head2)->stqh_last;			\
+	STAILQ_FIRST(head2) = swap_first;				\
+	(head2)->stqh_last = swap_last;					\
+	if (STAILQ_EMPTY(head1))					\
+		(head1)->stqh_last = &STAILQ_FIRST(head1);		\
+	if (STAILQ_EMPTY(head2))					\
+		(head2)->stqh_last = &STAILQ_FIRST(head2);		\
+} while (0)
+
+
+/*
+ * List declarations.
+ */
+#define	LIST_HEAD(name, type)						\
+struct name {								\
+	struct type *lh_first;	/* first element */			\
+}
+
+#define	LIST_CLASS_HEAD(name, type)					\
+struct name {								\
+	class type *lh_first;	/* first element */			\
+}
+
+#define	LIST_HEAD_INITIALIZER(head)					\
+	{ NULL }
+
+#define	LIST_ENTRY(type)						\
+struct {								\
+	struct type *le_next;	/* next element */			\
+	struct type **le_prev;	/* address of previous next element */	\
+}
+
+#define	LIST_CLASS_ENTRY(type)						\
+struct {								\
+	class type *le_next;	/* next element */			\
+	class type **le_prev;	/* address of previous next element */	\
+}
+
+/*
+ * List functions.
+ */
+
+#if (defined(_KERNEL) && defined(INVARIANTS))
+/*
+ * QMD_LIST_CHECK_HEAD(LIST_HEAD *head, LIST_ENTRY NAME)
+ *
+ * If the list is non-empty, validates that the first element of the list
+ * points back at 'head.'
+ */
+#define	QMD_LIST_CHECK_HEAD(head, field) do {				\
+	if (LIST_FIRST((head)) != NULL &&				\
+	    LIST_FIRST((head))->field.le_prev !=			\
+	     &LIST_FIRST((head)))					\
+		panic("Bad list head %p first->prev != head", (head));	\
+} while (0)
+
+/*
+ * QMD_LIST_CHECK_NEXT(TYPE *elm, LIST_ENTRY NAME)
+ *
+ * If an element follows 'elm' in the list, validates that the next element
+ * points back at 'elm.'
+ */
+#define	QMD_LIST_CHECK_NEXT(elm, field) do {				\
+	if (LIST_NEXT((elm), field) != NULL &&				\
+	    LIST_NEXT((elm), field)->field.le_prev !=			\
+	     &((elm)->field.le_next))					\
+	     	panic("Bad link elm %p next->prev != elm", (elm));	\
+} while (0)
+
+/*
+ * QMD_LIST_CHECK_PREV(TYPE *elm, LIST_ENTRY NAME)
+ *
+ * Validates that the previous element (or head of the list) points to 'elm.'
+ */
+#define	QMD_LIST_CHECK_PREV(elm, field) do {				\
+	if (*(elm)->field.le_prev != (elm))				\
+		panic("Bad link elm %p prev->next != elm", (elm));	\
+} while (0)
+#else
+#define	QMD_LIST_CHECK_HEAD(head, field)
+#define	QMD_LIST_CHECK_NEXT(elm, field)
+#define	QMD_LIST_CHECK_PREV(elm, field)
+#endif /* (_KERNEL && INVARIANTS) */
+
+#define LIST_CONCAT(head1, head2, type, field) do {			      \
+	QUEUE_TYPEOF(type) *curelm = LIST_FIRST(head1);			      \
+	if (curelm == NULL) {						      \
+		if ((LIST_FIRST(head1) = LIST_FIRST(head2)) != NULL) {	      \
+			LIST_FIRST(head2)->field.le_prev =		      \
+			    &LIST_FIRST((head1));			      \
+			LIST_INIT(head2);				      \
+		}							      \
+	} else if (LIST_FIRST(head2) != NULL) {				      \
+		while (LIST_NEXT(curelm, field) != NULL)		      \
+			curelm = LIST_NEXT(curelm, field);		      \
+		LIST_NEXT(curelm, field) = LIST_FIRST(head2);		      \
+		LIST_FIRST(head2)->field.le_prev = &LIST_NEXT(curelm, field); \
+		LIST_INIT(head2);					      \
+	}								      \
+} while (0)
+
+#define	LIST_EMPTY(head)	((head)->lh_first == NULL)
+
+#define	LIST_FIRST(head)	((head)->lh_first)
+
+#define	LIST_FOREACH(var, head, field)					\
+	for ((var) = LIST_FIRST((head));				\
+	    (var);							\
+	    (var) = LIST_NEXT((var), field))
+
+#define	LIST_FOREACH_FROM(var, head, field)				\
+	for ((var) = ((var) ? (var) : LIST_FIRST((head)));		\
+	    (var);							\
+	    (var) = LIST_NEXT((var), field))
+
+#define	LIST_FOREACH_SAFE(var, head, field, tvar)			\
+	for ((var) = LIST_FIRST((head));				\
+	    (var) && ((tvar) = LIST_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	LIST_FOREACH_FROM_SAFE(var, head, field, tvar)			\
+	for ((var) = ((var) ? (var) : LIST_FIRST((head)));		\
+	    (var) && ((tvar) = LIST_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	LIST_INIT(head) do {						\
+	LIST_FIRST((head)) = NULL;					\
+} while (0)
+
+#define	LIST_INSERT_AFTER(listelm, elm, field) do {			\
+	QMD_LIST_CHECK_NEXT(listelm, field);				\
+	if ((LIST_NEXT((elm), field) = LIST_NEXT((listelm), field)) != NULL)\
+		LIST_NEXT((listelm), field)->field.le_prev =		\
+		    &LIST_NEXT((elm), field);				\
+	LIST_NEXT((listelm), field) = (elm);				\
+	(elm)->field.le_prev = &LIST_NEXT((listelm), field);		\
+} while (0)
+
+#define	LIST_INSERT_BEFORE(listelm, elm, field) do {			\
+	QMD_LIST_CHECK_PREV(listelm, field);				\
+	(elm)->field.le_prev = (listelm)->field.le_prev;		\
+	LIST_NEXT((elm), field) = (listelm);				\
+	*(listelm)->field.le_prev = (elm);				\
+	(listelm)->field.le_prev = &LIST_NEXT((elm), field);		\
+} while (0)
+
+#define	LIST_INSERT_HEAD(head, elm, field) do {				\
+	QMD_LIST_CHECK_HEAD((head), field);				\
+	if ((LIST_NEXT((elm), field) = LIST_FIRST((head))) != NULL)	\
+		LIST_FIRST((head))->field.le_prev = &LIST_NEXT((elm), field);\
+	LIST_FIRST((head)) = (elm);					\
+	(elm)->field.le_prev = &LIST_FIRST((head));			\
+} while (0)
+
+#define	LIST_NEXT(elm, field)	((elm)->field.le_next)
+
+#define	LIST_PREV(elm, head, type, field)			\
+	((elm)->field.le_prev == &LIST_FIRST((head)) ? NULL :	\
+	    __containerof((elm)->field.le_prev,			\
+	    QUEUE_TYPEOF(type), field.le_next))
+
+#define	LIST_REMOVE(elm, field) do {					\
+	QMD_SAVELINK(oldnext, (elm)->field.le_next);			\
+	QMD_SAVELINK(oldprev, (elm)->field.le_prev);			\
+	QMD_LIST_CHECK_NEXT(elm, field);				\
+	QMD_LIST_CHECK_PREV(elm, field);				\
+	if (LIST_NEXT((elm), field) != NULL)				\
+		LIST_NEXT((elm), field)->field.le_prev = 		\
+		    (elm)->field.le_prev;				\
+	*(elm)->field.le_prev = LIST_NEXT((elm), field);		\
+	TRASHIT(*oldnext);						\
+	TRASHIT(*oldprev);						\
+} while (0)
+
+#define LIST_SWAP(head1, head2, type, field) do {			\
+	QUEUE_TYPEOF(type) *swap_tmp = LIST_FIRST(head1);		\
+	LIST_FIRST((head1)) = LIST_FIRST((head2));			\
+	LIST_FIRST((head2)) = swap_tmp;					\
+	if ((swap_tmp = LIST_FIRST((head1))) != NULL)			\
+		swap_tmp->field.le_prev = &LIST_FIRST((head1));		\
+	if ((swap_tmp = LIST_FIRST((head2))) != NULL)			\
+		swap_tmp->field.le_prev = &LIST_FIRST((head2));		\
+} while (0)
+
 /*
  * Tail queue declarations.
  */
@@ -123,10 +619,58 @@ struct {								\
 /*
  * Tail queue functions.
  */
+#if (defined(_KERNEL) && defined(INVARIANTS))
+/*
+ * QMD_TAILQ_CHECK_HEAD(TAILQ_HEAD *head, TAILQ_ENTRY NAME)
+ *
+ * If the tailq is non-empty, validates that the first element of the tailq
+ * points back at 'head.'
+ */
+#define	QMD_TAILQ_CHECK_HEAD(head, field) do {				\
+	if (!TAILQ_EMPTY(head) &&					\
+	    TAILQ_FIRST((head))->field.tqe_prev !=			\
+	     &TAILQ_FIRST((head)))					\
+		panic("Bad tailq head %p first->prev != head", (head));	\
+} while (0)
+
+/*
+ * QMD_TAILQ_CHECK_TAIL(TAILQ_HEAD *head, TAILQ_ENTRY NAME)
+ *
+ * Validates that the tail of the tailq is a pointer to pointer to NULL.
+ */
+#define	QMD_TAILQ_CHECK_TAIL(head, field) do {				\
+	if (*(head)->tqh_last != NULL)					\
+	    	panic("Bad tailq NEXT(%p->tqh_last) != NULL", (head)); 	\
+} while (0)
+
+/*
+ * QMD_TAILQ_CHECK_NEXT(TYPE *elm, TAILQ_ENTRY NAME)
+ *
+ * If an element follows 'elm' in the tailq, validates that the next element
+ * points back at 'elm.'
+ */
+#define	QMD_TAILQ_CHECK_NEXT(elm, field) do {				\
+	if (TAILQ_NEXT((elm), field) != NULL &&				\
+	    TAILQ_NEXT((elm), field)->field.tqe_prev !=			\
+	     &((elm)->field.tqe_next))					\
+		panic("Bad link elm %p next->prev != elm", (elm));	\
+} while (0)
+
+/*
+ * QMD_TAILQ_CHECK_PREV(TYPE *elm, TAILQ_ENTRY NAME)
+ *
+ * Validates that the previous element (or head of the tailq) points to 'elm.'
+ */
+#define	QMD_TAILQ_CHECK_PREV(elm, field) do {				\
+	if (*(elm)->field.tqe_prev != (elm))				\
+		panic("Bad link elm %p prev->next != elm", (elm));	\
+} while (0)
+#else
 #define	QMD_TAILQ_CHECK_HEAD(head, field)
 #define	QMD_TAILQ_CHECK_TAIL(head, headname)
 #define	QMD_TAILQ_CHECK_NEXT(elm, field)
 #define	QMD_TAILQ_CHECK_PREV(elm, field)
+#endif /* (_KERNEL && INVARIANTS) */
 
 #define	TAILQ_CONCAT(head1, head2, field) do {				\
 	if (!TAILQ_EMPTY(head2)) {					\
@@ -191,9 +735,8 @@ struct {								\
 
 #define	TAILQ_INSERT_AFTER(head, listelm, elm, field) do {		\
 	QMD_TAILQ_CHECK_NEXT(listelm, field);				\
-	TAILQ_NEXT((elm), field) = TAILQ_NEXT((listelm), field);	\
-	if (TAILQ_NEXT((listelm), field) != NULL)			\
-		TAILQ_NEXT((elm), field)->field.tqe_prev =		\
+	if ((TAILQ_NEXT((elm), field) = TAILQ_NEXT((listelm), field)) != NULL)\
+		TAILQ_NEXT((elm), field)->field.tqe_prev = 		\
 		    &TAILQ_NEXT((elm), field);				\
 	else {								\
 		(head)->tqh_last = &TAILQ_NEXT((elm), field);		\
@@ -217,8 +760,7 @@ struct {								\
 
 #define	TAILQ_INSERT_HEAD(head, elm, field) do {			\
 	QMD_TAILQ_CHECK_HEAD(head, field);				\
-	TAILQ_NEXT((elm), field) = TAILQ_FIRST((head));			\
-	if (TAILQ_FIRST((head)) != NULL)				\
+	if ((TAILQ_NEXT((elm), field) = TAILQ_FIRST((head))) != NULL)	\
 		TAILQ_FIRST((head))->field.tqe_prev =			\
 		    &TAILQ_NEXT((elm), field);				\
 	else								\
@@ -250,21 +792,24 @@ struct {								\
  * you may want to prefetch the last data element.
  */
 #define	TAILQ_LAST_FAST(head, type, field)			\
-	(TAILQ_EMPTY(head) ? NULL : __containerof((head)->tqh_last,	\
-	QUEUE_TYPEOF(type), field.tqe_next))
+    (TAILQ_EMPTY(head) ? NULL : __containerof((head)->tqh_last, QUEUE_TYPEOF(type), field.tqe_next))
 
 #define	TAILQ_NEXT(elm, field) ((elm)->field.tqe_next)
 
 #define	TAILQ_PREV(elm, headname, field)				\
 	(*(((struct headname *)((elm)->field.tqe_prev))->tqh_last))
 
+#define	TAILQ_PREV_FAST(elm, head, type, field)				\
+    ((elm)->field.tqe_prev == &(head)->tqh_first ? NULL :		\
+     __containerof((elm)->field.tqe_prev, QUEUE_TYPEOF(type), field.tqe_next))
+
 #define	TAILQ_REMOVE(head, elm, field) do {				\
 	QMD_SAVELINK(oldnext, (elm)->field.tqe_next);			\
 	QMD_SAVELINK(oldprev, (elm)->field.tqe_prev);			\
 	QMD_TAILQ_CHECK_NEXT(elm, field);				\
 	QMD_TAILQ_CHECK_PREV(elm, field);				\
 	if ((TAILQ_NEXT((elm), field)) != NULL)				\
-		TAILQ_NEXT((elm), field)->field.tqe_prev =		\
+		TAILQ_NEXT((elm), field)->field.tqe_prev = 		\
 		    (elm)->field.tqe_prev;				\
 	else {								\
 		(head)->tqh_last = (elm)->field.tqe_prev;		\
@@ -277,26 +822,20 @@ struct {								\
 } while (0)
 
 #define TAILQ_SWAP(head1, head2, type, field) do {			\
-	QUEUE_TYPEOF(type) * swap_first = (head1)->tqh_first;		\
-	QUEUE_TYPEOF(type) * *swap_last = (head1)->tqh_last;		\
+	QUEUE_TYPEOF(type) *swap_first = (head1)->tqh_first;		\
+	QUEUE_TYPEOF(type) **swap_last = (head1)->tqh_last;		\
 	(head1)->tqh_first = (head2)->tqh_first;			\
 	(head1)->tqh_last = (head2)->tqh_last;				\
 	(head2)->tqh_first = swap_first;				\
 	(head2)->tqh_last = swap_last;					\
-	swap_first = (head1)->tqh_first;				\
-	if (swap_first != NULL)						\
+	if ((swap_first = (head1)->tqh_first) != NULL)			\
 		swap_first->field.tqe_prev = &(head1)->tqh_first;	\
 	else								\
 		(head1)->tqh_last = &(head1)->tqh_first;		\
-	swap_first = (head2)->tqh_first;				\
-	if (swap_first != NULL)			\
+	if ((swap_first = (head2)->tqh_first) != NULL)			\
 		swap_first->field.tqe_prev = &(head2)->tqh_first;	\
 	else								\
 		(head2)->tqh_last = &(head2)->tqh_first;		\
 } while (0)
 
-#ifdef __cplusplus
-}
-#endif
-
-#endif /* _SYS_QUEUE_H_ */
+#endif /* !_SYS_QUEUE_H_ */
Signature · 1 line
2.25.1
#65 lore raw

PATCH v3 10/10 eal/windows: implement basic memory management

In reply to Dmitry Kozlyuk

Basic memory management supports core libraries and PMDs operating in IOVA as PA mode. It uses a kernel-mode driver, virt2phys, to obtain IOVAs of hugepages allocated from user-mode.

---
 config/meson.build                            |   2 +-
 doc/guides/windows_gsg/run_apps.rst           |  43 +-
 lib/librte_eal/common/eal_common_fbarray.c    |  57 +-
 lib/librte_eal/common/eal_common_memory.c     |  50 +-
 lib/librte_eal/common/eal_private.h           |   6 +-
 lib/librte_eal/common/malloc_heap.c           |   1 +
 lib/librte_eal/common/meson.build             |   9 +
 lib/librte_eal/freebsd/eal_memory.c           |   1 -
 lib/librte_eal/rte_eal_exports.def            | 119 ++-
 lib/librte_eal/windows/eal.c                  |  55 ++
 lib/librte_eal/windows/eal_memalloc.c         | 418 +++++++++++
 lib/librte_eal/windows/eal_memory.c           | 706 +++++++++++++++++-
 lib/librte_eal/windows/eal_mp.c               | 103 +++
 lib/librte_eal/windows/eal_windows.h          |  23 +
 lib/librte_eal/windows/include/meson.build    |   1 +
 lib/librte_eal/windows/include/rte_os.h       |   4 +
 .../windows/include/rte_virt2phys.h           |  34 +
 lib/librte_eal/windows/include/rte_windows.h  |   2 +
 lib/librte_eal/windows/include/unistd.h       |   3 +
 lib/librte_eal/windows/meson.build            |   4 +
 20 files changed, 1571 insertions(+), 70 deletions(-)
 create mode 100644 lib/librte_eal/windows/eal_memalloc.c
 create mode 100644 lib/librte_eal/windows/eal_mp.c
 create mode 100644 lib/librte_eal/windows/include/rte_virt2phys.h
config/meson.build +1 -1
index bceb5ef7b..7b8baa788 100644
@@ -270,7 +270,7 @@ if is_windows
 		add_project_link_arguments('-lmincore', language: 'c')
 	endif
 
-	add_project_link_arguments('-ladvapi32', language: 'c')
+	add_project_link_arguments('-ladvapi32', '-lsetupapi', language: 'c')
 endif
 
 if get_option('b_lto')
doc/guides/windows_gsg/run_apps.rst +40 -3

57 lines not on this page · raw message

lib/librte_eal/common/eal_common_fbarray.c +26 -31

168 lines not on this page · raw message

lib/librte_eal/common/eal_common_memory.c +22 -28

118 lines not on this page · raw message

lib/librte_eal/common/eal_private.h +3 -3
index 76938e379..59ac41916 100644
@@ -226,8 +226,8 @@ enum eal_mem_reserve_flags {
  *   Page size on which to align requested virtual area.
  * @param flags
  *   EAL_VIRTUAL_AREA_* flags.
- * @param mmap_flags
- *   Extra flags passed directly to mmap().
+ * @param reserve_flags
+ *   Extra flags passed directly to rte_mem_reserve().
  *
  * @return
  *   Virtual area address if successful.
@@ -244,7 +244,7 @@ enum eal_mem_reserve_flags {
 /**< immediately unmap reserved virtual area. */
 void *
 eal_get_virtual_area(void *requested_addr, size_t *size, size_t page_sz,
-	int flags, int mmap_flags);
+	int flags, enum eal_mem_reserve_flags reserve_flags);
 
 /**
  * Reserve VA space for a memory segment list.
lib/librte_eal/common/malloc_heap.c +1 -0
index 842eb9de7..6534c895c 100644
@@ -729,6 +729,7 @@ malloc_heap_alloc(const char *type, size_t size, int socket_arg,
 		if (ret != NULL)
 			return ret;
 	}
+
 	return NULL;
 }
 
lib/librte_eal/common/meson.build +9 -0
index 02d9280cc..6dcdcc890 100644
@@ -9,11 +9,20 @@ if is_windows
 		'eal_common_class.c',
 		'eal_common_devargs.c',
 		'eal_common_errno.c',
+		'eal_common_fbarray.c',
 		'eal_common_launch.c',
 		'eal_common_lcore.c',
 		'eal_common_log.c',
+		'eal_common_mcfg.c',
+		'eal_common_memalloc.c',
+		'eal_common_memory.c',
+		'eal_common_memzone.c',
 		'eal_common_options.c',
+		'eal_common_tailqs.c',
 		'eal_common_thread.c',
+		'malloc_elem.c',
+		'malloc_heap.c',
+		'rte_malloc.c',
 		'rte_option.c',
 	)
 	subdir_done()
lib/librte_eal/freebsd/eal_memory.c +0 -1
index 5174f9cd0..99bf6ec9e 100644
@@ -355,7 +355,6 @@ memseg_list_reserve(struct rte_memseg_list *msl)
 	return eal_reserve_memseg_list(msl, flags);
 }
 
-
 static int
 memseg_primary_init(void)
 {
lib/librte_eal/rte_eal_exports.def +117 -2

130 lines not on this page · raw message

lib/librte_eal/windows/eal.c +55 -0

73 lines not on this page · raw message

lib/librte_eal/windows/eal_memalloc.c new file +418 -0

418 lines not on this page · raw message

lib/librte_eal/windows/eal_memory.c +705 -1

744 lines not on this page · raw message

lib/librte_eal/windows/eal_mp.c new file +103 -0

103 lines not on this page · raw message

lib/librte_eal/windows/eal_windows.h +23 -0
index b202a1aa5..083ab8b93 100644
@@ -9,8 +9,24 @@
  * @file Facilities private to Windows EAL
  */
 
+#include <rte_errno.h>
 #include <rte_windows.h>
 
+/**
+ * Log current function as not implemented and set rte_errno.
+ */
+#define EAL_LOG_NOT_IMPLEMENTED() \
+	do { \
+		RTE_LOG(DEBUG, EAL, "%s() is not implemented\n", __func__); \
+		rte_errno = ENOTSUP; \
+	} while (0)
+
+/**
+ * Log current function as a stub.
+ */
+#define EAL_LOG_STUB() \
+	RTE_LOG(DEBUG, EAL, "Windows: %s() is a stub\n", __func__)
+
 /**
  * Create a map of processors and cores on the system.
  */
@@ -36,6 +52,13 @@ int eal_thread_create(pthread_t *thread);
  */
 unsigned int eal_socket_numa_node(unsigned int socket_id);
 
+/**
+ * Open virt2phys driver interface device.
+ *
+ * @return 0 on success, (-1) on failure.
+ */
+int eal_mem_virt2iova_init(void);
+
 /**
  * Locate Win32 memory management routines in system libraries.
  *
lib/librte_eal/windows/include/meson.build +1 -0
index 5fb1962ac..b3534b025 100644
@@ -5,5 +5,6 @@ includes += include_directories('.')
 
 headers += files(
         'rte_os.h',
+        'rte_virt2phys.h',
         'rte_windows.h',
 )
lib/librte_eal/windows/include/rte_os.h +4 -0
index 510e39e03..62805a307 100644
@@ -36,6 +36,10 @@ extern "C" {
 
 #define strncasecmp(s1, s2, count)        _strnicmp(s1, s2, count)
 
+#define open _open
+#define close _close
+#define unlink _unlink
+
 /* cpu_set macros implementation */
 #define RTE_CPU_AND(dst, src1, src2) CPU_AND(dst, src1, src2)
 #define RTE_CPU_OR(dst, src1, src2) CPU_OR(dst, src1, src2)
lib/librte_eal/windows/include/rte_virt2phys.h new file +34 -0
new file mode 100644
index 000000000..4bb2b4aaf
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2020 Dmitry Kozlyuk
+ */
+
+/**
+ * @file virt2phys driver interface
+ */
+
+/**
+ * Driver device interface GUID {539c2135-793a-4926-afec-d3a1b61bbc8a}.
+ */
+DEFINE_GUID(GUID_DEVINTERFACE_VIRT2PHYS,
+	0x539c2135, 0x793a, 0x4926,
+	0xaf, 0xec, 0xd3, 0xa1, 0xb6, 0x1b, 0xbc, 0x8a);
+
+/**
+ * Driver device type for IO control codes.
+ */
+#define VIRT2PHYS_DEVTYPE 0x8000
+
+/**
+ * Translate a valid non-paged virtual address to a physical address.
+ *
+ * Note: A physical address zero (0) is reported if input address
+ * is paged out or not mapped. However, if input is a valid mapping
+ * of I/O port 0x0000, output is also zero. There is no way
+ * to distinguish between these cases by return value only.
+ *
+ * Input: a non-paged virtual address (PVOID).
+ *
+ * Output: the corresponding physical address (LARGE_INTEGER).
+ */
+#define IOCTL_VIRT2PHYS_TRANSLATE CTL_CODE( \
+	VIRT2PHYS_DEVTYPE, 0x800, METHOD_BUFFERED, FILE_ANY_ACCESS)
lib/librte_eal/windows/include/rte_windows.h +2 -0
index ed6e4c148..899ed7d87 100644
@@ -23,6 +23,8 @@
 
 #include <basetsd.h>
 #include <psapi.h>
+#include <setupapi.h>
+#include <winioctl.h>
 
 /* Have GUIDs defined. */
 #ifndef INITGUID
lib/librte_eal/windows/include/unistd.h +3 -0
index 757b7f3c5..6b33005b2 100644
@@ -9,4 +9,7 @@
  * as Microsoft libc does not contain unistd.h. This may be removed
  * in future releases.
  */
+
+#include <io.h>
+
 #endif /* _UNISTD_H_ */
lib/librte_eal/windows/meson.build +4 -0
index 81d3ee095..0bd56cd8f 100644
@@ -8,7 +8,11 @@ sources += files(
 	'eal_debug.c',
 	'eal_hugepages.c',
 	'eal_lcore.c',
+	'eal_memalloc.c',
 	'eal_memory.c',
+	'eal_mp.c',
 	'eal_thread.c',
 	'getopt.c',
 )
+
+dpdk_conf.set10('RTE_EAL_NUMA_AWARE_HUGEPAGES', true)
Signature · 1 line
2.25.1
#66 lore raw

Re: [dpdk-dev] [PATCH v3 10/10] eal/windows: implement basic memory management

Quoted 6 lines · Dmitry Kozlyuk

Basic memory management supports core libraries and PMDs operating in IOVA as PA mode. It uses a kernel-mode driver, virt2phys, to obtain IOVAs of hugepages allocated from user-mode.

Quoted 8 lines
---
 config/meson.build                            |   2 +-
 doc/guides/windows_gsg/run_apps.rst           |  43 +-
 lib/librte_eal/common/eal_common_fbarray.c    |  57 +-
 lib/librte_eal/common/eal_common_memory.c     |  50 +-
 lib/librte_eal/common/eal_private.h           |   6 +-
 lib/librte_eal/common/malloc_heap.c           |   1 +
 lib/librte_eal/common/meson.build             |   9 +

IMO, We should split the lib/librte_eal/common/e and ib/librte_eal/windows/ as separate
patch for better review and align the patch title accordingly.

Quoted 17 lines
 lib/librte_eal/freebsd/eal_memory.c           |   1 -
 lib/librte_eal/rte_eal_exports.def            | 119 ++-
 lib/librte_eal/windows/eal.c                  |  55 ++
 lib/librte_eal/windows/eal_memalloc.c         | 418 +++++++++++
 lib/librte_eal/windows/eal_memory.c           | 706 +++++++++++++++++-
 lib/librte_eal/windows/eal_mp.c               | 103 +++
 lib/librte_eal/windows/eal_windows.h          |  23 +
 lib/librte_eal/windows/include/meson.build    |   1 +
 lib/librte_eal/windows/include/rte_os.h       |   4 +
 .../windows/include/rte_virt2phys.h           |  34 +
 lib/librte_eal/windows/include/rte_windows.h  |   2 +
 lib/librte_eal/windows/include/unistd.h       |   3 +
 lib/librte_eal/windows/meson.build            |   4 +
 20 files changed, 1571 insertions(+), 70 deletions(-)
 create mode 100644 lib/librte_eal/windows/eal_memalloc.c
 create mode 100644 lib/librte_eal/windows/eal_mp.c
 create mode 100644 lib/librte_eal/windows/include/rte_virt2phys.h
#67 lore raw

Re: [dpdk-dev] [PATCH v3 10/10] eal/windows: implement basic memory management

In reply to Dmitry Kozlyuk
Quoted 6 lines · Dmitry Kozlyuk

Basic memory management supports core libraries and PMDs operating in IOVA as PA mode. It uses a kernel-mode driver, virt2phys, to obtain IOVAs of hugepages allocated from user-mode.

---

<Snip!>

Quoted 30 lines
lib/librte_eal/windows/include/rte_virt2phys.h new file +24 -0
new file mode 100644
index 000000000..4bb2b4aaf
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2020 Dmitry Kozlyuk
+ */
+
+/**
+ * @file virt2phys driver interface
+ */
+
+/**
+ * Driver device interface GUID {539c2135-793a-4926-afec-d3a1b61bbc8a}.
+ */
+DEFINE_GUID(GUID_DEVINTERFACE_VIRT2PHYS,
+	0x539c2135, 0x793a, 0x4926,
+	0xaf, 0xec, 0xd3, 0xa1, 0xb6, 0x1b, 0xbc, 0x8a);
+
+/**
+ * Driver device type for IO control codes.
+ */
+#define VIRT2PHYS_DEVTYPE 0x8000
+
+/**
+ * Translate a valid non-paged virtual address to a physical address.
+ *
+ * Note: A physical address zero (0) is reported if input address

lib/librte_eal/windows/include/rte_virt2phys.h line 34

Quoted 10 lines
lib/librte_eal/windows/include/rte_virt2phys.h new file +10 -0
+ * is paged out or not mapped. However, if input is a valid mapping
+ * of I/O port 0x0000, output is also zero. There is no way
+ * to distinguish between these cases by return value only.
+ *
+ * Input: a non-paged virtual address (PVOID).
+ *
+ * Output: the corresponding physical address (LARGE_INTEGER).
+ */
+#define IOCTL_VIRT2PHYS_TRANSLATE CTL_CODE( \
+	VIRT2PHYS_DEVTYPE, 0x800, METHOD_BUFFERED, FILE_ANY_ACCESS)

This file is a duplicate of: <kmods>: windows/virt2phys/virt2phys.h This is by design, since it documents the driver interface.

So, to prevent the two files going out-of-sync, should we:
1. Make the two filenames the same?
2. Add comments to both files referencing each other and the need to keep them both in sync?
3. Do both (1) and (2)?

This will also be an issue for the upcoming Windows netuio kernel driver and I reckon this could be an issue for Linux kernel modules too.

Thoughts?

ranjit m.

#68 lore raw

Re: [dpdk-dev] [PATCH v3 10/10] eal/windows: implement basic memory management

On 2020-04-16 11:34 GMT-0700 Ranjit Menon wrote:
[snip]

Quoted 2 lines
Quoted 2 lines
lib/librte_eal/windows/include/rte_virt2phys.h new file +0 -0

[snip]

Quoted 10 lines

This file is a duplicate of: <kmods>: windows/virt2phys/virt2phys.h This is by design, since it documents the driver interface.

So, to prevent the two files going out-of-sync, should we:
1. Make the two filenames the same?
2. Add comments to both files referencing each other and the need to keep them both in sync?
3. Do both (1) and (2)?

Quoted 4 lines

This will also be an issue for the upcoming Windows netuio kernel driver and I reckon this could be an issue for Linux kernel modules too.

Thoughts?

I guess it won't be much of an issue in practice. When you edit driver interface you're already considering user-mode and kernel interaction, by definition of that interface, so you can't forget about the counterpart.

I'd be more worried about ABI, because memory corruption in driver will crash the system, being hard to debug. Can we leverage abigail for these interfaces?

Signature · 1 line
Dmitry Kozlyuk
Tested-by on the v3 series
#69 lore raw
In reply to Dmitry Kozlyuk
Quoted 124 lines · Dmitry Kozlyuk

Note: no changes in cover letter since v2.

This patchset implements basic MM with the following features:

* Hugepages are dynamically allocated in user-mode.
* Only 2MB hugepages are supported.
* IOVA is always PA, obtained through kernel-mode driver.
* No 32-bit support (presumably not demanded).
* Ni multi-process support (it is forcefully disabled).
* No-huge mode for testing with IOVA unavailable.

The first commit introduces a new kernel-mode driver, virt2phys. It translates user-mode virtual addresses into physical addresses. On Windows community call 2020-04-01 it was decided this driver can be used for now, later netUIO may pick up its code/interface or not.

New EAL public functions for memory mapping are introduced to mitigate OS differences in DPDK libraries and applications:

* rte_mem_map
* rte_mem_unmap
* rte_mem_lock

To support common MM routines, internal wrappers for low-level memory reservation and file management are introduced. These changes affect Linux and FreeBSD EAL. Shared code is placed unded /unix/ subdirectory (suggested by Thomas).

Also, entire <sys/queue.h> is imported from FreeBSD, replacing existing partial import. There is already a license exception for this file.

Windows MM duplicates quite a lot of code from Linux EAL:

* eal_memalloc_alloc_seg_bulk
* eal_memalloc_free_seg_bulk
* calc_num_pages_per_socket
* rte_eal_hugepage_init

Perhaps this should be left as-is until Windows MM evolves into having some specific requirements for these parts.

Notes on checkpatch warnings:

* No space after comma / no space before closing parent in macros--- definitely a false-positive, unclear how to suppress this.

* Issues from imported BSD code---probably should be ignored?

* Checkpatch is not run against dpdk-kmods (Windows drivers).

---

v3:

* Fix Linux build on and aarch64 and 32-bit x86 (reported by CI).
* Fix logic and error handling while allocating segments.
* Fix Unix rte_mem_map(): return NULL on failure.
* Fix some checkpatch.sh issues:
* Do not return positive errno, use DWORD for GetLastError().
* Make dpdk-kmods source files non-executable.
* Improve GSG for Windows Server (suggested by Ranjit Menon).

v2:

* Rebase on ToT. Move all new code shared between Linux and FreeBSD to /unix/ subdirectory, also factor out some existing code there.
* Improve description of Clang issue with rte_page_sizes on Windows. Restore -fstrict-enum for EAL. Check running, not target compiler.
* Use EAL prefix for private facilities instead if RTE.
* Improve documentation comments for new functions.
* Remove co-installer for virt2phys. Add a typecast for clarity.
* Document virt2phys in user guide, improve its own README.
* Explicitly and forcefully disable multi-process.

Dmitry Kozlyuk (9):
eal/windows: do not expose private EAL facilities eal/windows: improve CPU and NUMA node detection eal/windows: initialize hugepage info
eal: introduce internal wrappers for file operations
eal: introduce memory management wrappers
eal: extract common code for memseg list initialization eal/windows: fix rte_page_sizes with Clang on Windows eal/windows: replace sys/queue.h with a complete one from FreeBSD eal/windows: implement basic memory management

  config/meson.build                            |   12 +-
  doc/guides/windows_gsg/build_dpdk.rst         |   20 -
  doc/guides/windows_gsg/index.rst              |    1 +
  doc/guides/windows_gsg/run_apps.rst           |   84 ++
  lib/librte_eal/common/eal_common_fbarray.c    |   57 +-
  lib/librte_eal/common/eal_common_memory.c     |  104 +-
  lib/librte_eal/common/eal_private.h           |  134 +-
  lib/librte_eal/common/malloc_heap.c           |    1 +
  lib/librte_eal/common/meson.build             |    9 +
  lib/librte_eal/freebsd/eal_memory.c           |   55 +-
  lib/librte_eal/include/rte_memory.h           |   74 ++
  lib/librte_eal/linux/eal_memory.c             |   68 +-
  lib/librte_eal/meson.build                    |    4 +
  lib/librte_eal/rte_eal_exports.def            |  119 ++
  lib/librte_eal/rte_eal_version.map            |    4 +
  lib/librte_eal/unix/eal.c                     |   47 +
  lib/librte_eal/unix/eal_memory.c              |  113 ++
  lib/librte_eal/unix/meson.build               |    7 +
  lib/librte_eal/windows/eal.c                  |  160 +++
  lib/librte_eal/windows/eal_hugepages.c        |  108 ++
  lib/librte_eal/windows/eal_lcore.c            |  187 ++-
  lib/librte_eal/windows/eal_memalloc.c         |  418 ++++++
  lib/librte_eal/windows/eal_memory.c           | 1141 +++++++++++++++++
  lib/librte_eal/windows/eal_mp.c               |  103 ++
  lib/librte_eal/windows/eal_thread.c           |    1 +
  lib/librte_eal/windows/eal_windows.h          |  129 ++
  lib/librte_eal/windows/include/meson.build    |    2 +
  lib/librte_eal/windows/include/pthread.h      |    2 +
  lib/librte_eal/windows/include/rte_os.h       |   48 +-
  .../windows/include/rte_virt2phys.h           |   34 +
  lib/librte_eal/windows/include/rte_windows.h  |   43 +
  lib/librte_eal/windows/include/sys/queue.h    |  663 +++++++++-
  lib/librte_eal/windows/include/unistd.h       |    3 +
  lib/librte_eal/windows/meson.build            |    6 +
  34 files changed, 3611 insertions(+), 350 deletions(-)
Quoted 12 lines

create mode 100644 doc/guides/windows_gsg/run_apps.rst create mode 100644 lib/librte_eal/unix/eal.c
create mode 100644 lib/librte_eal/unix/eal_memory.c create mode 100644 lib/librte_eal/unix/meson.build create mode 100644 lib/librte_eal/windows/eal_hugepages.c create mode 100644 lib/librte_eal/windows/eal_memalloc.c create mode 100644 lib/librte_eal/windows/eal_memory.c create mode 100644 lib/librte_eal/windows/eal_mp.c create mode 100644 lib/librte_eal/windows/eal_windows.h create mode 100644 lib/librte_eal/windows/include/rte_virt2phys.h create mode 100644 lib/librte_eal/windows/include/rte_windows.h

#70 lore raw

v4 cover Windows basic memory management

In reply to Dmitry Kozlyuk

Note: symbols and release notes updated for v20.05, despite this patch
now targeting v20.08, because proper sections don't yet exist.

This patchset implements basic MM with the following features:

* Hugepages are dynamically allocated in user-mode.
* Only 2MB hugepages are supported.
* IOVA is always PA, obtained through kernel-mode driver.
* No 32-bit support (presumably not demanded).
* Ni multi-process support (it is forcefully disabled).
* No-huge mode for testing with IOVA unavailable.

New EAL public functions for memory mapping are introduced to mitigate OS differences in DPDK libraries and applications:

* rte_mem_map
* rte_mem_unmap
* rte_mem_lock
* rte_get_page_size

To support common MM routines, internal wrappers for low-level memory reservation and file management are introduced. These changes affect Linux and FreeBSD EAL. Shared code is placed unded /unix/ subdirectory (suggested by Thomas).

Also, entire <sys/queue.h> is imported from FreeBSD, replacing existing partial import. There is already a license exception for this file.

Windows MM duplicates quite a lot of code from Linux EAL:

* eal_memalloc_alloc_seg_bulk
* eal_memalloc_free_seg_bulk
* calc_num_pages_per_socket
* rte_eal_hugepage_init

Perhaps this should be left as-is until Windows MM evolves into having some specific requirements for these parts.

Notes on checkpatch warnings:

* No space after comma / no space before closing parent in macros--- definitely a false-positive, unclear how to suppress this.
* Issues from imported BSD code---probably should be ignored?

---

v4:

* Rebase on ToT, drop patches merged into master.
* Rearrange patches to split Windows code (Jerin).
* Fix Linux and FreeBSD build with make (Ophir).
* Use int instead of enum to hold a set of flags (Anatoly).
* Rename eal_mem_reserve items and fix their description (Anatoly).
* Add eal_mem_set_dump() wrapper around madvise (Anatoly).
* Don't claim Windows Server 2016 support due to lack of API (Tal).
* Replace enum rte_page_sizes with a set of #defines (Jerin).
* Fix documentation, SPDX tags, logging (Thomas).

v3:

* Fix Linux build on and aarch64 and 32-bit x86 (reported by CI).
* Fix logic and error handling while allocating segments.
* Fix Unix rte_mem_map(): return NULL on failure.
* Fix some checkpatch.sh issues:
* Do not return positive errno, use DWORD for GetLastError().
* Make dpdk-kmods source files non-executable.
* Improve GSG for Windows Server (suggested by Ranjit Menon).

v2:

* Rebase on ToT. Move all new code shared between Linux and FreeBSD to /unix/ subdirectory, also factor out some existing code there.
* Improve description of Clang issue with rte_page_sizes on Windows. Restore -fstrict-enum for EAL. Check running, not target compiler.
* Use EAL prefix for private facilities instead if RTE.
* Improve documentation comments for new functions.
* Remove co-installer for virt2phys. Add a typecast for clarity.
* Document virt2phys in user guide, improve its own README.
* Explicitly and forcefully disable multi-process.

Dmitry Kozlyuk (8):
eal: replace rte_page_sizes with a set of constants
eal: introduce internal wrappers for file operations
eal: introduce memory management wrappers
eal: extract common code for memseg list initialization eal/windows: replace sys/queue.h with a complete one from FreeBSD eal/windows: improve CPU and NUMA node detection eal/windows: initialize hugepage info
eal/windows: implement basic memory management

 config/meson.build                            |   12 +-
 doc/guides/rel_notes/release_20_05.rst        |    2 +
 doc/guides/windows_gsg/build_dpdk.rst         |   20 -
 doc/guides/windows_gsg/index.rst              |    1 +
 doc/guides/windows_gsg/run_apps.rst           |   95 ++
 lib/librte_eal/common/eal_common_fbarray.c    |   56 +-
 lib/librte_eal/common/eal_common_memory.c     |  116 +-
 lib/librte_eal/common/eal_common_memzone.c    |    7 +
 lib/librte_eal/common/eal_private.h           |  155 ++-
 lib/librte_eal/common/meson.build             |   10 +
 lib/librte_eal/common/rte_malloc.c            |    9 +
 lib/librte_eal/freebsd/Makefile               |    5 +
 lib/librte_eal/freebsd/eal_memory.c           |   57 +-
 lib/librte_eal/include/rte_memory.h           |  109 +-
 lib/librte_eal/linux/Makefile                 |    5 +
 lib/librte_eal/linux/eal_memalloc.c           |    5 +-
 lib/librte_eal/linux/eal_memory.c             |   68 +-
 lib/librte_eal/meson.build                    |    4 +
 lib/librte_eal/rte_eal_exports.def            |  119 ++
 lib/librte_eal/rte_eal_version.map            |    4 +
 lib/librte_eal/unix/eal_unix.c                |   51 +
 lib/librte_eal/unix/eal_unix_memory.c         |  144 ++
 lib/librte_eal/unix/meson.build               |    7 +
 lib/librte_eal/windows/eal.c                  |  158 +++
 lib/librte_eal/windows/eal_hugepages.c        |  108 ++
 lib/librte_eal/windows/eal_lcore.c            |  185 ++-
 lib/librte_eal/windows/eal_memalloc.c         |  418 ++++++
 lib/librte_eal/windows/eal_memory.c           | 1155 +++++++++++++++++
 lib/librte_eal/windows/eal_mp.c               |  103 ++
 lib/librte_eal/windows/eal_windows.h          |  100 ++
 lib/librte_eal/windows/include/meson.build    |    1 +
 lib/librte_eal/windows/include/rte_os.h       |    4 +
 .../windows/include/rte_virt2phys.h           |   34 +
 lib/librte_eal/windows/include/rte_windows.h  |    2 +
 lib/librte_eal/windows/include/sys/queue.h    |  663 +++++++++-
 lib/librte_eal/windows/include/unistd.h       |    3 +
 lib/librte_eal/windows/meson.build            |    6 +
 37 files changed, 3655 insertions(+), 346 deletions(-)
 create mode 100644 doc/guides/windows_gsg/run_apps.rst
 create mode 100644 lib/librte_eal/unix/eal_unix.c
 create mode 100644 lib/librte_eal/unix/eal_unix_memory.c
 create mode 100644 lib/librte_eal/unix/meson.build
 create mode 100644 lib/librte_eal/windows/eal_hugepages.c
 create mode 100644 lib/librte_eal/windows/eal_memalloc.c
 create mode 100644 lib/librte_eal/windows/eal_memory.c
 create mode 100644 lib/librte_eal/windows/eal_mp.c
 create mode 100644 lib/librte_eal/windows/include/rte_virt2phys.h
Signature · 1 line
2.25.1
#71 lore raw

PATCH v4 1/8 eal: replace rte_page_sizes with a set of constants

Clang on Windows follows MS ABI where enum values are limited to 2^31-1. Enum rte_page_sizes has members valued above this limit, which get wrapped to zero, resulting in compilation error (duplicate values in enum). Using MS ABI is mandatory for Windows EAL to call Win32 APIs.

Remove rte_page_sizes and replace its values with #define's. This enumeration is not used in public API, so there's no ABI breakage. Document API change in release notes.

---
 doc/guides/rel_notes/release_20_05.rst |  2 ++
 lib/librte_eal/include/rte_memory.h    | 23 ++++++++++-------------
 2 files changed, 12 insertions(+), 13 deletions(-)
doc/guides/rel_notes/release_20_05.rst +2 -0
index b124c3f28..76ba59220 100644
@@ -225,6 +225,8 @@ Removed Items
    Also, make sure to start the actual text at the margin.
    =========================================================
 
+* ``enum rte_page_sizes`` is removed, ``RTE_PGSIZE_*`` constants are kept.
+
 
 API Changes
 -----------
lib/librte_eal/include/rte_memory.h +10 -13
index 3d8d0bd69..65374d53a 100644
@@ -24,19 +24,16 @@ extern "C" {
 #include <rte_config.h>
 #include <rte_fbarray.h>
 
-__extension__
-enum rte_page_sizes {
-	RTE_PGSIZE_4K    = 1ULL << 12,
-	RTE_PGSIZE_64K   = 1ULL << 16,
-	RTE_PGSIZE_256K  = 1ULL << 18,
-	RTE_PGSIZE_2M    = 1ULL << 21,
-	RTE_PGSIZE_16M   = 1ULL << 24,
-	RTE_PGSIZE_256M  = 1ULL << 28,
-	RTE_PGSIZE_512M  = 1ULL << 29,
-	RTE_PGSIZE_1G    = 1ULL << 30,
-	RTE_PGSIZE_4G    = 1ULL << 32,
-	RTE_PGSIZE_16G   = 1ULL << 34,
-};
+#define RTE_PGSIZE_4K   (1ULL << 12)
+#define RTE_PGSIZE_64K  (1ULL << 16)
+#define RTE_PGSIZE_256K (1ULL << 18)
+#define RTE_PGSIZE_2M   (1ULL << 21)
+#define RTE_PGSIZE_16M  (1ULL << 24)
+#define RTE_PGSIZE_256M (1ULL << 28)
+#define RTE_PGSIZE_512M (1ULL << 29)
+#define RTE_PGSIZE_1G   (1ULL << 30)
+#define RTE_PGSIZE_4G   (1ULL << 32)
+#define RTE_PGSIZE_16G  (1ULL << 34)
 
 #define SOCKET_ID_ANY -1                    /**< Any NUMA socket. */
 
Signature · 1 line
2.25.1
#72 lore raw

PATCH v4 2/8 eal: introduce internal wrappers for file operations

In reply to Dmitry Kozlyuk

EAL common code uses file locking and truncation. Introduce OS-independent wrappers in order to support both Linux/FreeBSD and Windows:

* eal_file_lock: lock or unlock an open file.
* eal_file_truncate: enforce a given size for an open file.

Wrappers follow POSIX semantics, but interface is not POSIX, so that it can be made more clean, e.g. by not mixing locking operation and behaviour on conflict.

Implementation for Linux and FreeBSD is placed in "unix" subdirectory, which is intended for common code between the two. Files should be named after the ones from which the code is factored in OS subdirectory.

---
 lib/librte_eal/common/eal_common_fbarray.c | 19 ++++----
 lib/librte_eal/common/eal_private.h        | 45 +++++++++++++++++++
 lib/librte_eal/freebsd/Makefile            |  4 ++
 lib/librte_eal/linux/Makefile              |  4 ++
 lib/librte_eal/meson.build                 |  4 ++
 lib/librte_eal/unix/eal_unix.c             | 51 ++++++++++++++++++++++
 lib/librte_eal/unix/meson.build            |  6 +++
 7 files changed, 122 insertions(+), 11 deletions(-)
 create mode 100644 lib/librte_eal/unix/eal_unix.c
 create mode 100644 lib/librte_eal/unix/meson.build
lib/librte_eal/common/eal_common_fbarray.c +8 -11
index 4f8f1af73..1e55757ca 100644
@@ -8,8 +8,8 @@
 #include <sys/mman.h>
 #include <stdint.h>
 #include <errno.h>
-#include <sys/file.h>
 #include <string.h>
+#include <unistd.h>
 
 #include <rte_common.h>
 #include <rte_log.h>
@@ -85,7 +85,7 @@ resize_and_map(int fd, void *addr, size_t len)
 	char path[PATH_MAX];
 	void *map_addr;
 
-	if (ftruncate(fd, len)) {
+	if (eal_file_truncate(fd, len)) {
 		RTE_LOG(ERR, EAL, "Cannot truncate %s\n", path);
 		/* pass errno up the chain */
 		rte_errno = errno;
@@ -778,7 +778,8 @@ rte_fbarray_init(struct rte_fbarray *arr, const char *name, unsigned int len,
 					__func__, path, strerror(errno));
 			rte_errno = errno;
 			goto fail;
-		} else if (flock(fd, LOCK_EX | LOCK_NB)) {
+		} else if (eal_file_lock(
+				fd, EAL_FLOCK_EXCLUSIVE, EAL_FLOCK_RETURN)) {
 			RTE_LOG(DEBUG, EAL, "%s(): couldn't lock %s: %s\n",
 					__func__, path, strerror(errno));
 			rte_errno = EBUSY;
@@ -789,10 +790,8 @@ rte_fbarray_init(struct rte_fbarray *arr, const char *name, unsigned int len,
 		 * still attach to it, but no other process could reinitialize
 		 * it.
 		 */
-		if (flock(fd, LOCK_SH | LOCK_NB)) {
-			rte_errno = errno;
+		if (eal_file_lock(fd, EAL_FLOCK_SHARED, EAL_FLOCK_RETURN))
 			goto fail;
-		}
 
 		if (resize_and_map(fd, data, mmap_len))
 			goto fail;
@@ -895,10 +894,8 @@ rte_fbarray_attach(struct rte_fbarray *arr)
 	}
 
 	/* lock the file, to let others know we're using it */
-	if (flock(fd, LOCK_SH | LOCK_NB)) {
-		rte_errno = errno;
+	if (eal_file_lock(fd, EAL_FLOCK_SHARED, EAL_FLOCK_RETURN))
 		goto fail;
-	}
 
 	if (resize_and_map(fd, data, mmap_len))
 		goto fail;
@@ -1025,7 +1022,7 @@ rte_fbarray_destroy(struct rte_fbarray *arr)
 		 * has been detached by all other processes
 		 */
 		fd = tmp->fd;
-		if (flock(fd, LOCK_EX | LOCK_NB)) {
+		if (eal_file_lock(fd, EAL_FLOCK_EXCLUSIVE, EAL_FLOCK_RETURN)) {
 			RTE_LOG(DEBUG, EAL, "Cannot destroy fbarray - another process is using it\n");
 			rte_errno = EBUSY;
 			ret = -1;
@@ -1042,7 +1039,7 @@ rte_fbarray_destroy(struct rte_fbarray *arr)
 			 * we're still holding an exclusive lock, so drop it to
 			 * shared.
 			 */
-			flock(fd, LOCK_SH | LOCK_NB);
+			eal_file_lock(fd, EAL_FLOCK_SHARED, EAL_FLOCK_RETURN);
 
 			ret = -1;
 			goto out;
lib/librte_eal/common/eal_private.h +45 -0

49 lines not on this page · raw message

lib/librte_eal/freebsd/Makefile +4 -0
index a8400f20a..a26c455c7 100644
@@ -7,6 +7,7 @@ LIB = librte_eal.a
 
 ARCH_DIR ?= $(RTE_ARCH)
 VPATH += $(RTE_SDK)/lib/librte_eal/$(ARCH_DIR)
+VPATH += $(RTE_SDK)/lib/librte_eal/unix
 VPATH += $(RTE_SDK)/lib/librte_eal/common
 
 CFLAGS += -I$(SRCDIR)/include
@@ -74,6 +75,9 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_service.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_random.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_reciprocal.c
 
+# from unix dir
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_unix.c
+
 # from arch dir
 SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_cpuflags.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_hypervisor.c
lib/librte_eal/linux/Makefile +4 -0
index a77eb1757..fa41f00bf 100644
@@ -7,6 +7,7 @@ LIB = librte_eal.a
 
 ARCH_DIR ?= $(RTE_ARCH)
 VPATH += $(RTE_SDK)/lib/librte_eal/$(ARCH_DIR)
+VPATH += $(RTE_SDK)/lib/librte_eal/unix
 VPATH += $(RTE_SDK)/lib/librte_eal/common
 
 CFLAGS += -I$(SRCDIR)/include
@@ -81,6 +82,9 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_service.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_random.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_reciprocal.c
 
+# from unix dir
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_unix.c
+
 # from arch dir
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_cpuflags.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_hypervisor.c
lib/librte_eal/meson.build +4 -0
index 0267c3b9d..98c97dd07 100644
@@ -6,6 +6,10 @@ subdir('include')
 
 subdir('common')
 
+if not is_windows
+	subdir('unix')
+endif
+
 dpdk_conf.set('RTE_EXEC_ENV_' + exec_env.to_upper(), 1)
 subdir(exec_env)
 
lib/librte_eal/unix/eal_unix.c new file +51 -0

51 lines not on this page · raw message

lib/librte_eal/unix/meson.build new file +6 -0
new file mode 100644
index 000000000..cfa1b4ef9
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020 Dmitry Kozlyuk
+
+sources += files(
+	'eal_unix.c',
+)
Signature · 1 line
2.25.1
Acked-by on v4 2/8
#73 lore raw

Re: [dpdk-dev] [PATCH v4 2/8] eal: introduce internal wrappers for file operations

Quoted 17 lines · Dmitry Kozlyuk

EAL common code uses file locking and truncation. Introduce OS-independent wrappers in order to support both Linux/FreeBSD and Windows:

* eal_file_lock: lock or unlock an open file.
* eal_file_truncate: enforce a given size for an open file.

Wrappers follow POSIX semantics, but interface is not POSIX, so that it can be made more clean, e.g. by not mixing locking operation and behaviour on conflict.

Implementation for Linux and FreeBSD is placed in "unix" subdirectory, which is intended for common code between the two. Files should be named after the ones from which the code is factored in OS subdirectory.

---

<snip>

lib/librte_eal/common/eal_common_fbarray.c resize_and_map(int fd, void *addr, size_t len) · line 91

Quoted 12 lines
  
  #include <rte_common.h>
  #include <rte_log.h>
lib/librte_eal/common/eal_common_fbarray.c +1 -1
@@ -85,7 +85,7 @@ resize_and_map(int fd, void *addr, size_t len)
  	char path[PATH_MAX];
  	void *map_addr;
 
-	if (ftruncate(fd, len)) {
+	if (eal_file_truncate(fd, len)) {
  		RTE_LOG(ERR, EAL, "Cannot truncate %s\n", path);
  		/* pass errno up the chain */
  		rte_errno = errno;

eal_file_truncate sets rte_errno, so no need to pass it up the chain any more.

Otherwise,

Signature · 2 lines
Thanks,
Anatoly
#74 lore raw

PATCH v4 3/8 eal: introduce memory management wrappers

In reply to Dmitry Kozlyuk

Introduce OS-independent wrappers for memory management operations used across DPDK and specifically in common code of EAL:

* rte_mem_map()
* rte_mem_unmap()
* rte_get_page_size()
* rte_mem_lock()

Windows uses different APIs for memory mapping and reservation, while Unices reserve memory by mapping it. Introduce EAL private functions to support memory reservation in common code:

* eal_mem_reserve()
* eal_mem_free()
* eal_mem_set_dump()

Wrappers follow POSIX semantics limited to DPDK tasks, but their signatures deliberately differ from POSIX ones to be more safe and expressive.

---
 lib/librte_eal/common/eal_common_fbarray.c |  37 +++---
 lib/librte_eal/common/eal_common_memory.c  |  62 ++++-----
 lib/librte_eal/common/eal_private.h        |  74 ++++++++++-
 lib/librte_eal/freebsd/Makefile            |   1 +
 lib/librte_eal/include/rte_memory.h        |  86 ++++++++++++
 lib/librte_eal/linux/Makefile              |   1 +
 lib/librte_eal/linux/eal_memalloc.c        |   5 +-
 lib/librte_eal/rte_eal_version.map         |   4 +
 lib/librte_eal/unix/eal_unix_memory.c      | 144 +++++++++++++++++++++
 lib/librte_eal/unix/meson.build            |   1 +
 10 files changed, 350 insertions(+), 65 deletions(-)
 create mode 100644 lib/librte_eal/unix/eal_unix_memory.c
lib/librte_eal/common/eal_common_fbarray.c +17 -20
index 1e55757ca..b3b6c8521 100644
@@ -5,15 +5,15 @@
 #include <fcntl.h>
 #include <inttypes.h>
 #include <limits.h>
-#include <sys/mman.h>
 #include <stdint.h>
 #include <errno.h>
 #include <string.h>
 #include <unistd.h>
 
 #include <rte_common.h>
-#include <rte_log.h>
 #include <rte_errno.h>
+#include <rte_log.h>
+#include <rte_memory.h>
 #include <rte_spinlock.h>
 #include <rte_tailq.h>
 
@@ -92,12 +92,9 @@ resize_and_map(int fd, void *addr, size_t len)
 		return -1;
 	}
 
-	map_addr = mmap(addr, len, PROT_READ | PROT_WRITE,
-			MAP_SHARED | MAP_FIXED, fd, 0);
+	map_addr = rte_mem_map(addr, len, RTE_PROT_READ | RTE_PROT_WRITE,
+			RTE_MAP_SHARED | RTE_MAP_FORCE_ADDRESS, fd, 0);
 	if (map_addr != addr) {
-		RTE_LOG(ERR, EAL, "mmap() failed: %s\n", strerror(errno));
-		/* pass errno up the chain */
-		rte_errno = errno;
 		return -1;
 	}
 	return 0;
@@ -735,7 +732,7 @@ rte_fbarray_init(struct rte_fbarray *arr, const char *name, unsigned int len,
 		return -1;
 	}
 
-	page_sz = sysconf(_SC_PAGESIZE);
+	page_sz = rte_get_page_size();
 	if (page_sz == (size_t)-1) {
 		free(ma);
 		return -1;
@@ -756,9 +753,11 @@ rte_fbarray_init(struct rte_fbarray *arr, const char *name, unsigned int len,
 
 	if (internal_config.no_shconf) {
 		/* remap virtual area as writable */
-		void *new_data = mmap(data, mmap_len, PROT_READ | PROT_WRITE,
-				MAP_FIXED | MAP_PRIVATE | MAP_ANONYMOUS, fd, 0);
-		if (new_data == MAP_FAILED) {
+		static const int flags = RTE_MAP_FORCE_ADDRESS |
+			RTE_MAP_PRIVATE | RTE_MAP_ANONYMOUS;
+		void *new_data = rte_mem_map(data, mmap_len,
+			RTE_PROT_READ | RTE_PROT_WRITE, flags, fd, 0);
+		if (new_data == NULL) {
 			RTE_LOG(DEBUG, EAL, "%s(): couldn't remap anonymous memory: %s\n",
 					__func__, strerror(errno));
 			goto fail;
@@ -823,7 +822,7 @@ rte_fbarray_init(struct rte_fbarray *arr, const char *name, unsigned int len,
 	return 0;
 fail:
 	if (data)
-		munmap(data, mmap_len);
+		rte_mem_unmap(data, mmap_len);
 	if (fd >= 0)
 		close(fd);
 	free(ma);
@@ -861,7 +860,7 @@ rte_fbarray_attach(struct rte_fbarray *arr)
 		return -1;
 	}
 
-	page_sz = sysconf(_SC_PAGESIZE);
+	page_sz = rte_get_page_size();
 	if (page_sz == (size_t)-1) {
 		free(ma);
 		return -1;
@@ -913,7 +912,7 @@ rte_fbarray_attach(struct rte_fbarray *arr)
 	return 0;
 fail:
 	if (data)
-		munmap(data, mmap_len);
+		rte_mem_unmap(data, mmap_len);
 	if (fd >= 0)
 		close(fd);
 	free(ma);
@@ -941,8 +940,7 @@ rte_fbarray_detach(struct rte_fbarray *arr)
 	 * really do anything about it, things will blow up either way.
 	 */
 
-	size_t page_sz = sysconf(_SC_PAGESIZE);
-
+	size_t page_sz = rte_get_page_size();
 	if (page_sz == (size_t)-1)
 		return -1;
 
@@ -961,7 +959,7 @@ rte_fbarray_detach(struct rte_fbarray *arr)
 		goto out;
 	}
 
-	munmap(arr->data, mmap_len);
+	rte_mem_unmap(arr->data, mmap_len);
 
 	/* area is unmapped, close fd and remove the tailq entry */
 	if (tmp->fd >= 0)
@@ -996,8 +994,7 @@ rte_fbarray_destroy(struct rte_fbarray *arr)
 	 * really do anything about it, things will blow up either way.
 	 */
 
-	size_t page_sz = sysconf(_SC_PAGESIZE);
-
+	size_t page_sz = rte_get_page_size();
 	if (page_sz == (size_t)-1)
 		return -1;
 
@@ -1046,7 +1043,7 @@ rte_fbarray_destroy(struct rte_fbarray *arr)
 		}
 		close(fd);
 	}
-	munmap(arr->data, mmap_len);
+	rte_mem_unmap(arr->data, mmap_len);
 
 	/* area is unmapped, remove the tailq entry */
 	TAILQ_REMOVE(&mem_area_tailq, tmp, next);
lib/librte_eal/common/eal_common_memory.c +23 -39

137 lines not on this page · raw message

lib/librte_eal/common/eal_private.h +71 -3

102 lines not on this page · raw message

lib/librte_eal/freebsd/Makefile +1 -0
index a26c455c7..647dfd0f2 100644
@@ -77,6 +77,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_reciprocal.c
 
 # from unix dir
 SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_unix.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_unix_memory.c
 
 # from arch dir
 SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_cpuflags.c
lib/librte_eal/include/rte_memory.h +86 -0

92 lines not on this page · raw message

lib/librte_eal/linux/Makefile +1 -0
index fa41f00bf..06428f0de 100644
@@ -84,6 +84,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_reciprocal.c
 
 # from unix dir
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_unix.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_unix_memory.c
 
 # from arch dir
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_cpuflags.c
lib/librte_eal/linux/eal_memalloc.c +2 -3
index 2c717f8bd..bf29b83c6 100644
@@ -630,7 +630,7 @@ alloc_seg(struct rte_memseg *ms, void *addr, int socket_id,
 mapped:
 	munmap(addr, alloc_sz);
 unmapped:
-	flags = MAP_FIXED;
+	flags = EAL_RESERVE_FORCE_ADDRESS;
 	new_addr = eal_get_virtual_area(addr, &alloc_sz, alloc_sz, 0, flags);
 	if (new_addr != addr) {
 		if (new_addr != NULL)
@@ -687,8 +687,7 @@ free_seg(struct rte_memseg *ms, struct hugepage_info *hi,
 		return -1;
 	}
 
-	if (madvise(ms->addr, ms->len, MADV_DONTDUMP) != 0)
-		RTE_LOG(DEBUG, EAL, "madvise failed: %s\n", strerror(errno));
+	eal_mem_set_dump(ms->addr, ms->len, false);
 
 	exit_early = false;
 
lib/librte_eal/rte_eal_version.map +4 -0
index 6088e7f6c..5d6d3a8c6 100644
@@ -373,7 +373,11 @@ EXPERIMENTAL {
 	__rte_trace_point_register;
 	per_lcore_trace_mem;
 	per_lcore_trace_point_sz;
+	rte_get_page_size;
 	rte_log_can_log;
+	rte_mem_lock;
+	rte_mem_map;
+	rte_mem_unmap;
 	rte_thread_getname;
 	rte_trace_dump;
 	rte_trace_is_enabled;
lib/librte_eal/unix/eal_unix_memory.c new file +144 -0

144 lines not on this page · raw message

lib/librte_eal/unix/meson.build +1 -0
index cfa1b4ef9..5734f26ad 100644
@@ -3,4 +3,5 @@
 
 sources += files(
 	'eal_unix.c',
+	'eal_unix_memory.c',
 )
Signature · 1 line
2.25.1
#75 lore raw

Re: [dpdk-dev] [PATCH v4 3/8] eal: introduce memory management wrappers

Quoted 22 lines · Dmitry Kozlyuk

Introduce OS-independent wrappers for memory management operations used across DPDK and specifically in common code of EAL:

* rte_mem_map()
* rte_mem_unmap()
* rte_get_page_size()
* rte_mem_lock()

Windows uses different APIs for memory mapping and reservation, while Unices reserve memory by mapping it. Introduce EAL private functions to support memory reservation in common code:

* eal_mem_reserve()
* eal_mem_free()
* eal_mem_set_dump()

Wrappers follow POSIX semantics limited to DPDK tasks, but their signatures deliberately differ from POSIX ones to be more safe and expressive.

---

<snip>

Quoted 4 lines

RTE_LOG(DEBUG, EAL, "Ask a virtual area of 0x%zx bytes\n", *size);

lib/librte_eal/common/eal_common_memory.c +0 -0
@@ -105,24 +94,24 @@ eal_get_virtual_area(void *requested_addr, size_t *size,
  			return NULL;

lib/librte_eal/common/eal_common_memory.c eal_get_virtual_area(void *requested_addr, size_t *size, · line 100

Quoted 10 lines
lib/librte_eal/common/eal_common_memory.c +4 -4
  		}
 
-		mapped_addr = mmap(requested_addr, (size_t)map_sz, PROT_NONE,
-				mmap_flags, -1, 0);
-		if (mapped_addr == MAP_FAILED && allow_shrink)
-			*size -= page_sz;
+		mapped_addr = eal_mem_reserve(
+			requested_addr, (size_t)map_sz, reserve_flags);
+		if ((mapped_addr == NULL) && allow_shrink)
+			size -= page_sz;

Should be *size -= page_sz, size is a pointer in this case.

lib/librte_eal/common/eal_common_memory.c eal_get_virtual_area(void *requested_addr, size_t *size, · line 108

Quoted 12 lines
lib/librte_eal/common/eal_common_memory.c +3 -4
-		if (mapped_addr != MAP_FAILED && addr_is_hint &&
-		    mapped_addr != requested_addr) {
+		if ((mapped_addr != NULL) && addr_is_hint &&
+				(mapped_addr != requested_addr)) {
  			try++;
  			next_baseaddr = RTE_PTR_ADD(next_baseaddr, page_sz);
  			if (try <= MAX_MMAP_WITH_DEFINED_ADDR_TRIES) {
  				/* hint was not used. Try with another offset */
-				munmap(mapped_addr, map_sz);
-				mapped_addr = MAP_FAILED;
+				eal_mem_free(mapped_addr, *size);

Why change map_sz to *size?

Quoted 8 lines
lib/librte_eal/common/eal_common_memory.c +2 -1
+				mapped_addr = NULL;
  				requested_addr = next_baseaddr;
  			}
  		}
  	} while ((allow_shrink || addr_is_hint) &&
-		 mapped_addr == MAP_FAILED && *size > 0);
+		(mapped_addr == NULL) && (*size > 0));
 

<snip>

lib/librte_eal/common/eal_common_memory.c rte_eal_memdevice_init(void) · line 536

Quoted 11 lines
lib/librte_eal/common/eal_common_memory.c +3 -4
@@ -547,10 +531,10 @@ rte_eal_memdevice_init(void)
  int
  rte_mem_lock_page(const void *virt)
  {
-	unsigned long virtual = (unsigned long)virt;
-	int page_size = getpagesize();
-	unsigned long aligned = (virtual & ~(page_size - 1));
-	return mlock((void *)aligned, page_size);
+	uintptr_t virtual = (uintptr_t)virt;
+	int page_size = rte_get_page_size();
+	uintptr_t aligned = (virtual & ~(page_size - 1));

Might as well fix to use macros? e.g.

size_t pagesz = rte_get_page_size();
return rte_mem_lock(RTE_PTR_ALIGN(virt, pagesz), pagesz);

(also, note that rte_get_page_size() returns size_t, not int)

Quoted 10 lines
lib/librte_eal/common/eal_common_memory.c +1 -0
+	return rte_mem_lock((void *)aligned, page_size);
  }
 
  int
lib/librte_eal/common/eal_private.h +0 -0
index 3aafd892f..67ca83e47 100644
@@ -11,6 +11,7 @@
 

<snip>

Quoted 8 lines

+ * Reservation size. Must be a multiple of system page size.
+ * @param flags
+ * Reservation options, a combination of eal_mem_reserve_flags.
+ * @returns
+ * Starting address of the reserved area on success, NULL on failure.
+ * Callers must not access this memory until remapping it.
+ */
+void *eal_mem_reserve(void *requested_addr, size_t size, int flags);

Should we also require requested_addr to be page-aligned?

Also, here and in other added API's, nitpick but our coding style guide (and the code style in this file) suggests that return value should be on a separate line, e.g.

void *
eal_mem_reserve(...);

Quoted 11 lines
lib/librte_eal/common/eal_private.h +11 -0
+
+/**
+ * Free memory obtained by eal_mem_reserve() or eal_mem_alloc().
+ *
+ * If *virt* and *size* describe a part of the reserved region,
+ * only this part of the region is freed (accurately up to the system
+ * page size). If *virt* points to allocated memory, *size* must match
+ * the one specified on allocation. The behavior is undefined
+ * if the memory pointed by *virt* is obtained from another source
+ * than listed above.
+ *

<snip>

lib/librte_eal/common/eal_private.h

Quoted 6 lines
lib/librte_eal/common/eal_private.h +6 -0
+}
+
+static int
+mem_rte_to_sys_prot(int prot)
+{
+	int sys_prot = 0;

Maybe set it to PROT_NONE to make it more obvious?

Quoted 6 lines
lib/librte_eal/common/eal_private.h +6 -0
+
+	if (prot & RTE_PROT_READ)
+		sys_prot |= PROT_READ;
+	if (prot & RTE_PROT_WRITE)
+		sys_prot |= PROT_WRITE;
+	if (prot & RTE_PROT_EXECUTE)

lib/librte_eal/common/eal_private.h

Quoted 10 lines
lib/librte_eal/common/eal_private.h +10 -0
+		sys_prot |= PROT_EXEC;
+
+	return sys_prot;
+}
+
+void *
+rte_mem_map(void *requested_addr, size_t size, int prot, int flags,
+	int fd, size_t offset)
+{
+	int sys_prot = 0;

Not necessary to initialize sys_prot (and it's counter-productive as compiler warning about uninitialized usage is a *good thing*!).

Quoted 16 lines
lib/librte_eal/common/eal_private.h +16 -0
+	int sys_flags = 0;
+
+	sys_prot = mem_rte_to_sys_prot(prot);
+
+	if (flags & RTE_MAP_SHARED)
+		sys_flags |= MAP_SHARED;
+	if (flags & RTE_MAP_ANONYMOUS)
+		sys_flags |= MAP_ANONYMOUS;
+	if (flags & RTE_MAP_PRIVATE)
+		sys_flags |= MAP_PRIVATE;
+	if (flags & RTE_MAP_FORCE_ADDRESS)
+		sys_flags |= MAP_FIXED;
+
+	return mem_map(requested_addr, size, sys_prot, sys_flags, fd, offset);
+}
+

lib/librte_eal/common/eal_private.h

Quoted 10 lines
lib/librte_eal/common/eal_private.h +10 -0
+int
+rte_mem_unmap(void *virt, size_t size)
+{
+	return mem_unmap(virt, size);
+}
+
+size_t
+rte_get_page_size(void)
+{
+	return sysconf(_SC_PAGESIZE);

Can we perhaps cache this value?

lib/librte_eal/common/eal_private.h

Quoted 6 lines
lib/librte_eal/common/eal_private.h +6 -0
+}
+
+int
+rte_mem_lock(const void *virt, size_t size)
+{
+	return mlock(virt, size);

This call can fail. It should pass errno as rte_errno as well, just like all other calls from this family.

Also, if the implementation "may require" page alignment, how about requiring it unconditionally?

Quoted 12 lines
lib/librte_eal/common/eal_private.h +1 -0
+}
lib/librte_eal/unix/meson.build +1 -0
index cfa1b4ef9..5734f26ad 100644
@@ -3,4 +3,5 @@
 
  sources += files(
  	'eal_unix.c',
+	'eal_unix_memory.c',
  )
Signature · 2 lines
Thanks,
Anatoly
#76 lore raw

Re: [dpdk-dev] [PATCH v4 3/8] eal: introduce memory management wrappers

Quoted 14 lines · Anatoly
Quoted 11 lines
(quoted diff) +0 -0
@@ -547,10 +531,10 @@ rte_eal_memdevice_init(void)

  int
  rte_mem_lock_page(const void *virt)
  {
-    unsigned long virtual = (unsigned long)virt; -    int page_size = getpagesize();
-    unsigned long aligned = (virtual & ~(page_size - 1)); -    return mlock((void *)aligned, page_size);
+    uintptr_t virtual = (uintptr_t)virt;
+    int page_size = rte_get_page_size();
+    uintptr_t aligned = (virtual & ~(page_size - 1));

Might as well fix to use macros? e.g.

Quoted 4 lines

size_t pagesz = rte_get_page_size();
return rte_mem_lock(RTE_PTR_ALIGN(virt, pagesz), pagesz);

(also, note that rte_get_page_size() returns size_t, not int)

Apologies, this should've been RTE_PTR_ALIGN_FLOOR(virt, pagesz)

Signature · 2 lines
Thanks,
Anatoly
#77 lore raw

Re: [dpdk-dev] [PATCH v4 3/8] eal: introduce memory management wrappers

In reply to Burakov, Anatoly

Thanks for pointing out the errors, see some comments inline.

Quoted 1 line · Anatoly

On 29-Apr-20 12:50 AM, Dmitry Kozlyuk wrote:

<snip>

Quoted 10 lines
Quoted 8 lines

+ * Reservation size. Must be a multiple of system page size.
+ * @param flags
+ * Reservation options, a combination of eal_mem_reserve_flags.
+ * @returns
+ * Starting address of the reserved area on success, NULL on failure.
+ * Callers must not access this memory until remapping it.
+ */
+void *eal_mem_reserve(void *requested_addr, size_t size, int flags);

Should we also require requested_addr to be page-aligned?

Yes.

Quoted 6 lines

Also, here and in other added API's, nitpick but our coding style guide (and the code style in this file) suggests that return value should be on a separate line, e.g.

void *
eal_mem_reserve(...);

Will follow your advice in v5 to keep the style within this file consistent. However, DPDK Coding Style explicitly says:

Unlike function definitions, the function prototypes do not need to place the function return type on a separate line.

[snip]

Quoted 11 lines

lib/librte_eal/unix/eal_unix_memory.c line 143

Quoted 5 lines
lib/librte_eal/unix/eal_unix_memory.c new file +5 -0
+
+int
+rte_mem_lock(const void *virt, size_t size)
+{
+	return mlock(virt, size);  

This call can fail. It should pass errno as rte_errno as well, just like all other calls from this family.

Also, if the implementation "may require" page alignment, how about requiring it unconditionally?

IMO even better to document this function as locking all pages crossed by the address region. This would save address checking/alignment at call site and all implementations work this way. Locking memory implies paging system.

Signature · 1 line
Dmtiry Kozlyuk
#78 lore raw

Re: [dpdk-dev] [PATCH v4 3/8] eal: introduce memory management wrappers

Quoted 41 lines · Dmitry Kozlyuk

Thanks for pointing out the errors, see some comments inline.

Quoted 1 line · Anatoly

On 29-Apr-20 12:50 AM, Dmitry Kozlyuk wrote:

<snip>

Quoted 10 lines
Quoted 8 lines

+ * Reservation size. Must be a multiple of system page size.
+ * @param flags
+ * Reservation options, a combination of eal_mem_reserve_flags.
+ * @returns
+ * Starting address of the reserved area on success, NULL on failure.
+ * Callers must not access this memory until remapping it.
+ */
+void *eal_mem_reserve(void *requested_addr, size_t size, int flags);

Should we also require requested_addr to be page-aligned?

Yes.

Quoted 6 lines

Also, here and in other added API's, nitpick but our coding style guide (and the code style in this file) suggests that return value should be on a separate line, e.g.

void *
eal_mem_reserve(...);

Will follow your advice in v5 to keep the style within this file consistent. However, DPDK Coding Style explicitly says:

Unlike function definitions, the function prototypes do not need to place the function return type on a separate line.

[snip]

Quoted 9 lines

lib/librte_eal/unix/eal_unix_memory.c line 143

Quoted 5 lines
lib/librte_eal/unix/eal_unix_memory.c new file +5 -0
+
+int
+rte_mem_lock(const void *virt, size_t size)
+{
+	return mlock(virt, size);

This call can fail. It should pass errno as rte_errno as well, just like all other calls from this family.

Quoted 7 lines
Quoted 2 lines

Also, if the implementation "may require" page alignment, how about requiring it unconditionally?

IMO even better to document this function as locking all pages crossed by the address region. This would save address checking/alignment at call site and all implementations work this way. Locking memory implies paging system.

I don't think any other external API we provide does automagic pointer alignment, so i'm not sure if it indeed would be better to have it align automatically. It's also better from the standpoint of not silently allowing seemingly invalid arguments. So, i would lean on the side of requiring alignment, but not doing it ourselves.

Signature · 2 lines
Thanks,
Anatoly
#79 lore raw

PATCH v4 4/8 eal: extract common code for memseg list initialization

In reply to Dmitry Kozlyuk

All supported OS create memory segment lists (MSL) and reserve VA space for them in a nearly identical way. Move common code into EAL private functions to reduce duplication.

---
 lib/librte_eal/common/eal_common_memory.c | 54 ++++++++++++++++++
 lib/librte_eal/common/eal_private.h       | 36 ++++++++++++
 lib/librte_eal/freebsd/eal_memory.c       | 57 +++----------------
 lib/librte_eal/linux/eal_memory.c         | 68 +++++------------------
 4 files changed, 113 insertions(+), 102 deletions(-)
lib/librte_eal/common/eal_common_memory.c +54 -0
index 1196a8037..56eff0acb 100644
@@ -24,6 +24,7 @@
 #include "eal_private.h"
 #include "eal_internal_cfg.h"
 #include "eal_memcfg.h"
+#include "eal_options.h"
 #include "malloc_heap.h"
 
 /*
@@ -181,6 +182,59 @@ eal_get_virtual_area(void *requested_addr, size_t *size,
 	return aligned_addr;
 }
 
+int
+eal_memseg_list_init(struct rte_memseg_list *msl, uint64_t page_sz,
+		int n_segs, int socket_id, int type_msl_idx, bool heap)
+{
+	char name[RTE_FBARRAY_NAME_LEN];
+
+	snprintf(name, sizeof(name), MEMSEG_LIST_FMT, page_sz >> 10, socket_id,
+		 type_msl_idx);
+	if (rte_fbarray_init(&msl->memseg_arr, name, n_segs,
+			sizeof(struct rte_memseg))) {
+		RTE_LOG(ERR, EAL, "Cannot allocate memseg list: %s\n",
+			rte_strerror(rte_errno));
+		return -1;
+	}
+
+	msl->page_sz = page_sz;
+	msl->socket_id = socket_id;
+	msl->base_va = NULL;
+	msl->heap = heap;
+
+	RTE_LOG(DEBUG, EAL, "Memseg list allocated: 0x%zxkB at socket %i\n",
+			(size_t)page_sz >> 10, socket_id);
+
+	return 0;
+}
+
+int
+eal_memseg_list_alloc(struct rte_memseg_list *msl, int reserve_flags)
+{
+	uint64_t page_sz;
+	size_t mem_sz;
+	void *addr;
+
+	page_sz = msl->page_sz;
+	mem_sz = page_sz * msl->memseg_arr.len;
+
+	addr = eal_get_virtual_area(
+		msl->base_va, &mem_sz, page_sz, 0, reserve_flags);
+	if (addr == NULL) {
+		if (rte_errno == EADDRNOTAVAIL)
+			RTE_LOG(ERR, EAL, "Cannot reserve %llu bytes at [%p] - "
+				"please use '--" OPT_BASE_VIRTADDR "' option\n",
+				(unsigned long long)mem_sz, msl->base_va);
+		else
+			RTE_LOG(ERR, EAL, "Cannot reserve memory\n");
+		return -1;
+	}
+	msl->base_va = addr;
+	msl->len = mem_sz;
+
+	return 0;
+}
+
 static struct rte_memseg *
 virt2memseg(const void *addr, const struct rte_memseg_list *msl)
 {
lib/librte_eal/common/eal_private.h +36 -0
index 67ca83e47..4a28274ec 100644
@@ -254,6 +254,42 @@ void *
 eal_get_virtual_area(void *requested_addr, size_t *size,
 		size_t page_sz, int flags, int reserve_flags);
 
+/**
+ * Initialize a memory segment list and create its backing storage.
+ *
+ * @param msl
+ *  Memory segment list to be filled.
+ * @param page_sz
+ *  Size of segment pages in the MSL.
+ * @param n_segs
+ *  Number of segments.
+ * @param socket_id
+ *  Socket ID. Must not be SOCKET_ID_ANY.
+ * @param type_msl_idx
+ *  Index of the MSL among other MSLs of the same socket and page size.
+ * @param heap
+ *  Mark MSL as pointing to a heap.
+ * @return
+ *  0 on success, (-1) on failure and rte_errno is set.
+ */
+int
+eal_memseg_list_init(struct rte_memseg_list *msl, uint64_t page_sz,
+	int n_segs, int socket_id, int type_msl_idx, bool heap);
+
+/**
+ * Reserve VA space for a memory segment list
+ * previously initialized with eal_memseg_list_init().
+ *
+ * @param msl
+ *  Memory segment list with page size defined.
+ * @param reserve_flags
+ *  Extra memory reservation flags. Can be 0 if unnecessary.
+ * @return
+ *  0 on success, (-1) on failure and rte_errno is set.
+ */
+int
+eal_memseg_list_alloc(struct rte_memseg_list *msl, int reserve_flags);
+
 /**
  * Get cpu core_id.
  *
lib/librte_eal/freebsd/eal_memory.c +9 -48

94 lines not on this page · raw message

lib/librte_eal/linux/eal_memory.c +14 -54

129 lines not on this page · raw message

Signature · 1 line
2.25.1
#80 lore raw

Re: [dpdk-dev] [PATCH v4 4/8] eal: extract common code for memseg list initialization

Quoted 6 lines · Dmitry Kozlyuk

All supported OS create memory segment lists (MSL) and reserve VA space for them in a nearly identical way. Move common code into EAL private functions to reduce duplication.

---

Would it be possible to extract all similar code and make it use eal_memseg_list_init? For example, in FreeBSD eal_memory.c in nohuge mode (or in Linux legacy mem) the code remains untouched. I wonder if it can be ported to using this function.

Quoted 34 lines
  lib/librte_eal/common/eal_common_memory.c | 54 ++++++++++++++++++
  lib/librte_eal/common/eal_private.h       | 36 ++++++++++++
  lib/librte_eal/freebsd/eal_memory.c       | 57 +++----------------
  lib/librte_eal/linux/eal_memory.c         | 68 +++++------------------
  4 files changed, 113 insertions(+), 102 deletions(-)
lib/librte_eal/common/eal_common_memory.c +13 -0
index 1196a8037..56eff0acb 100644
@@ -24,6 +24,7 @@
  #include "eal_private.h"
  #include "eal_internal_cfg.h"
  #include "eal_memcfg.h"
+#include "eal_options.h"
  #include "malloc_heap.h"
 
  /*
@@ -181,6 +182,59 @@ eal_get_virtual_area(void *requested_addr, size_t *size,
  	return aligned_addr;
  }
 
+int
+eal_memseg_list_init(struct rte_memseg_list *msl, uint64_t page_sz,
+		int n_segs, int socket_id, int type_msl_idx, bool heap)
+{
+	char name[RTE_FBARRAY_NAME_LEN];
+
+	snprintf(name, sizeof(name), MEMSEG_LIST_FMT, page_sz >> 10, socket_id,
+		 type_msl_idx);
+	if (rte_fbarray_init(&msl->memseg_arr, name, n_segs,
+			sizeof(struct rte_memseg))) {
+		RTE_LOG(ERR, EAL, "Cannot allocate memseg list: %s\n",
+			rte_strerror(rte_errno));

lib/librte_eal/common/eal_common_memory.c eal_get_virtual_area(void *requested_addr, size_t *size, · line 206

Quoted 10 lines
lib/librte_eal/common/eal_common_memory.c +10 -0
+		return -1;
+	}
+
+	msl->page_sz = page_sz;
+	msl->socket_id = socket_id;
+	msl->base_va = NULL;
+	msl->heap = heap;
+
+	RTE_LOG(DEBUG, EAL, "Memseg list allocated: 0x%zxkB at socket %i\n",
+			(size_t)page_sz >> 10, socket_id);

The log message looks odd here. Maybe change it to indicate that the kB value is the page size, not the size of the memory?

Quoted 17 lines
lib/librte_eal/common/eal_common_memory.c +17 -0
+
+	return 0;
+}
+
+int
+eal_memseg_list_alloc(struct rte_memseg_list *msl, int reserve_flags)
+{
+	uint64_t page_sz;
+	size_t mem_sz;
+	void *addr;
+
+	page_sz = msl->page_sz;
+	mem_sz = page_sz * msl->memseg_arr.len;
+
+	addr = eal_get_virtual_area(
+		msl->base_va, &mem_sz, page_sz, 0, reserve_flags);
+	if (addr == NULL) {

lib/librte_eal/common/eal_common_memory.c eal_get_virtual_area(void *requested_addr, size_t *size, · line 233

Quoted 10 lines
lib/librte_eal/common/eal_common_memory.c +10 -0
+		if (rte_errno == EADDRNOTAVAIL)
+			RTE_LOG(ERR, EAL, "Cannot reserve %llu bytes at [%p] - "
+				"please use '--" OPT_BASE_VIRTADDR "' option\n",
+				(unsigned long long)mem_sz, msl->base_va);
+		else
+			RTE_LOG(ERR, EAL, "Cannot reserve memory\n");
+		return -1;
+	}
+	msl->base_va = addr;
+	msl->len = mem_sz;

Perhaps add a log message saying that space was allocated for this memseg list?

Signature · 2 lines
Thanks,
Anatoly