Skip to contentSkip to conversation

One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

DISCUSSION Sinan Kaya 5 participants 25 messages Jan 18, 2018 – Feb 5, 2018 6 messages not in the archive PCI

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): SH2PR01MB106A1E21DEB5FE3FFB3D61C83E90@SH2PR01MB106.CHNPR01.prod.partner.outlook.cn

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): SHAPR01MB173A5EA1677C2138CB528F2FEE90@SHAPR01MB173.CHNPR01.prod.partner.outlook.cn

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): CABhMZUU0643U-qVc9ymA+1PMZToSLFm2dg8-cu=iQ2LGw2Pi8Q@mail.gmail.com

A message that is not in the archive (sent off-list or not yet received): SHAPR01MB173A36104635A8BFF9A83E1FEE80@SHAPR01MB173.CHNPR01.prod.partner.outlook.cn

Sinan Kaya

author
#6 lore raw

+Linux-PCI.

Please use mailing list email syntax moving forward. (inline and 75 characters per line)

Quoted 9 lines · Ron Yuan

Bjorn,
I believe no one in current CC list would against be cc'd in larger group.

Sinan,
we do understand that BIOS have full control of MRRS. But the problem is once kernel boot with pcie_bus_perf, MRRS setting would be overwritten by the code we complained. Btw, the reason we need to use pcie_bus_perf is because Intel white paper indicated system should use pcie_bus_perf to support NVMe SSD hot plug, specifically for a correct MPS to be set.

So I think at the moment back to the code, we should
1. when we don’t agree on what's the best value to set, at least remove the code MRRS=MPS, leave MRRS to default value instead of changing it incorrectly. The setting is for one device, hence the device should be responsible for itself, choosing a suitable value as default.

As I indicated below, MRRS is not a value that a PCIe endpoint device can choose. It depends on the root complex silicon. Some chip internal bus may handle 4k writes poorly; whereas, another one can handle it better.

Also, according to the spec; the default value of MRRS is 512 not 4096.

Coming back to why MRRS=MPS? I think this is because of a isochronous device statement in the spec.

"Software must limit the Max_Payload_Size for each path that supports isochronous to meet the isochronous latency. For example, all traffic flowing on a path from an isochronous capable device to the Root Complex should be limited to packets that do not exceed the Max_Payload_Size required to meet the isochronous latency requirements."

This is again a hefty price to pay for some specific PCIE endpoint type that not everybody cares about.

Quoted 4 lines

2. try to get a common understanding what is the best performance for MRRS, maybe try largest MRRS supported by the system/device, or have additional command line input.

Item 1 is something we should do as first step.
For item 2, we need bigger group of experienced PCIe engineers to discuss.

We need to understand that MPS and MRRS have much broader impact. All silicon is different.

I don't think there is a common solution to this and my preference is for these parameters to be specified along with the perf option. Then, we can adjust things accordingly.

Let's see what others think about this.

Quoted 3 lines

Here is one link for your reference:
https://www.xilinx.com/support/documentation/white_papers/wp350.pdf

Sinan

Quoted 106 lines

Thanks!
Ron
-----Original Message-----
From: Bjorn Helgaas [mailto:[email protected]]
Sent: 2018年1月17日 22:27
To: Sinan Kaya <[email protected]>
Cc: Ron Yuan <[email protected]>; Bo Chen <[email protected]>; William Huang <[email protected]>; Fengming Wu <[email protected]>; Bjorn Helgaas <[email protected]>; Jason Jiang <[email protected]>; Radjendirane Codandaramane <[email protected]>; Ramyakanth Edupuganti <[email protected]>; William Cheng <[email protected]>; Kim Helper (khelper) <[email protected]>
Subject: Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

I'd be happy to participate in this discussion, but I'd prefer to include the [email protected] mailing list because that is public, archived, and searchable. It also includes people who have worked on MPS and MRRs in the past. If anyone doesn't want to be cc'd on that list, let me know and I'll remove you from the cc list before I respond.

Quoted 94 lines · Sinan Kaya
Quoted 12 lines · Ron Yuan

The code we refer to is
static void pcie_write_mrrs(struct pci_dev *dev) { ...
/* For Max performance, the MRRS must be set to the largest supported
* value. However, it cannot be configured larger than the MPS the
* device or the bus can support. This should already be properly
* configured by a prior call to pcie_write_mps.
*/
mrrs = pcie_get_mps(dev);
...
}

Quoted 3 lines

Here the statement " However, it cannot be configured larger than the MPS the device or the bus can support." is obviously incorrect. A simple against statement is from PCIe spec, which define default MPS to 128B, and *default MRRS to 512B*. From HW understanding, MRRS is multiple of MPS in the system, it must be no less than MPS.

Yeah, I mentioned it in my presentation too. MPS is not equal to MRRS. The only requirement is for MRRS to be a multiple of MPS.

Quoted 6 lines

Look at the code of Pcie_bus_safe/Pcie_bus_perf/Pcie_bus_peer2peer, it is not handling MRRS correctly. Peer2peer mode should actually use the smallest 128B for MRRS, which
is also discussed in https://patchwork.kernel.org/patch/8747521/
Safe mode may be better match MRRS to MDS for safety reason.
Perf mode should set MRRS to largest accepted value, 4k for example.

I don't disagree.

Choosing 4k is problematic. It could be 1k for one person and 2k for another person.We can have some configuration around this via kernel command line if Bjorn agrees.

Quoted 5 lines

Your reply says some hw logic may have poorer performance with larger MRRS sounds new to me, some HW engineer may have better look. Then in this case, "leave device recommended MRRS untouched" is more appropriate for performance purpose.
Currently I can see a lot of device is using 256B MPS, and force downgrade default 4k/512B MRRS to 256B is actually getting poorer performance.

Most BIOS allows you to choose the MPS and MRRS settings. If your BIOS does, then I recommend you to use that option at this moment for tuning your system along with the safe boot option.

Quoted 42 lines

Thanks again for your time!

Ron
Beijing Memblaze Technology

-----Original Message-----
From: Sinan Kaya [mailto:[email protected]]
Sent: 2018年1月17日 12:21
To: Bo Chen <[email protected]>
Cc: Ron Yuan <[email protected]>; William Huang <[email protected]>; Fengming Wu <[email protected]>; Bjorn Helgaas <[email protected]>
Subject: Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

+Bjorn,

Hi Bo,

Quoted 4 lines · Bo Chen

Hi Sinan,

We went through linux pci related code found that nvme device has different performance when kernel boot with pcie_bus_safe or pcie_bus_perf.
For details, when boot with pcie_bus_safe (eg: nvme device A: MaxPayload 128B, MaxReadReq 4096B), after pci bus init and enumerate, device A devCtl remains 128B for MaxPayload and 4096B for MaxReadReq.

pcie_bus_safe option is used to balance the maximum payload size setting across the bus. The goal is to make sure that maximum payload size is consistent. That's why, it is called safe. It also honors the maximum read request size setting done by the BIOS. It does not touch the MRRS setting.

Quoted 5 lines

While boot with pcie_bus_perf, we found pcie_write_mrrs set the read request size to the same with MPS. For nvme device A, it means both MaxPayload and MaxReadReq are set to 128B so that we observed it has limited device performance in read req.

Quoted 1 line

From our point of view on nvme device, the setting of pcie_bus_perf has lower performance compared with setting with pcie_bus_safe.

Is there any consideration for pcie_bus_perf to set MaxReadReq the same value with pcie_bus_safe? Is there any risk when we boot with pcie_bus_perf for nvme device? Thank you very much!

/*
* If using the "performance" PCIe config, we clamp the
* read rq size to the max packet size to prevent the
* host bridge generating requests larger than we can
* cope with
*/

I found this comment in the code that seems to match the behavior you are seeing.

Quoted 8 lines

Problem with maximum read request size is that there is no one-size-fits-all solution. It depends on the bus-fabric of the silicon you are using. An MRRS of 4k might behave slower than 1k. 1k might behave better than 512 bytes.

The recommendation is for these settings to be adjustable in BIOS and tweak it according to the application.

I was planning to make a change in this area but I was warned by our hardware engineers that performance does not always scale with MRRS setting.

Sinan

--
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.

Signature · 3 lines
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
#7 lore raw
Quoted 33 lines · Sinan Kaya
Quoted 7 lines · Ron Yuan

we do understand that BIOS have full control of MRRS. But the problem is once kernel boot with pcie_bus_perf, MRRS setting would be overwritten by the code we complained. Btw, the reason we need to use pcie_bus_perf is because Intel white paper indicated system should use pcie_bus_perf to support NVMe SSD hot plug, specifically for a correct MPS to be set.

Quoted 6 lines

So I think at the moment back to the code, we should
1. when we don’t agree on what's the best value to set, at least remove the code MRRS=MPS, leave MRRS to default value instead of changing it incorrectly. The setting is for one device, hence the device should be responsible for itself, choosing a suitable value as default.

As I indicated below, MRRS is not a value that a PCIe endpoint device can choose. It depends on the root complex silicon. Some chip internal bus may handle 4k writes poorly; whereas, another one can handle it better.

Also, according to the spec; the default value of MRRS is 512 not 4096.

Coming back to why MRRS=MPS? I think this is because of a isochronous device statement in the spec.

"Software must limit the Max_Payload_Size for each path that supports isochronous to meet the isochronous latency. For example, all traffic flowing on a path from an isochronous capable device to the Root Complex should be limited to packets that do not exceed the Max_Payload_Size required to meet the isochronous latency requirements."

Quoted 2 lines

This is again a hefty price to pay for some specific PCIE endpoint type that not everybody cares about.

In PCIE_BUS_PERFORMANCE mode, pcie_write_mrrs() does try to set MRRS=MPS, but the reason is not to support isochronous. The reason is to allow us to use larger MPS settings than we otherwise would.

Consider a switch leading to an endpoint that supports only MPS=128. The simplest approach would be to configure every device in the fabric with MPS=128. That guarantees the endpoint will never receive a TLP with a payload larger than 128 bytes.

Here's my understanding of how PCIE_BUS_PERFORMANCE works:

There are two ways an endpoint may receive TLPs with data payloads:
(1) Memory Write Requests that target the endpoint, and (2) Completions with Data in response to Memory Read Requests generated by the endpoint.

In PCIE_BUS_PERFORMANCE mode, we assume Memory Write Requests are not an issue because:

- We assume a CPU Memory Write Request is never larger than MPS (128 bytes in this case). This is fairly safe because CPUs generally can't write more than a cache line in one request, and most CPUs have cache lines of 128 bytes or less.

- We assume there's no peer-to-peer DMA, so other devices in the fabric will never send Memory Write Requests to the endpoint, so we don't need to limit their MPS settings.

That leaves Completions. We limit the size of Completions by limiting MRRS. If we set the endpoint's MRRS to its MPS (128 in this case), it will never request more than MPS bytes at a time, so it will never receive a Completion with more than MPS bytes.

Therefore, we may be able to configure other devices in the fabric with MPS larger than 128, which may benefit those devices.

Quoted 39 lines
Quoted 6 lines

2. try to get a common understanding what is the best performance for MRRS, maybe try largest MRRS supported by the system/device, or have additional command line input.

Item 1 is something we should do as first step. For item 2, we need bigger group of experienced PCIe engineers to discuss.

We need to understand that MPS and MRRS have much broader impact. All silicon is different.

I don't think there is a common solution to this and my preference is for these parameters to be specified along with the perf option. Then, we can adjust things accordingly.

Let's see what others think about this.

Quoted 23 lines

Here is one link for your reference:
https://www.xilinx.com/support/documentation/white_papers/wp350.pdf

Quoted 19 lines · Sinan Kaya
Quoted 17 lines · Ron Yuan

The code we refer to is
static void pcie_write_mrrs(struct pci_dev *dev) { ...
/* For Max performance, the MRRS must be set to the largest supported
* value. However, it cannot be configured larger than the MPS the
* device or the bus can support. This should already be properly
* configured by a prior call to pcie_write_mps.
*/
mrrs = pcie_get_mps(dev);
...
}

Here the statement " However, it cannot be configured larger than the MPS the device or the bus can support." is obviously incorrect. A simple against statement is from PCIe spec, which define default MPS to 128B, and *default MRRS to 512B*. From HW understanding, MRRS is multiple of MPS in the system, it must be no less than MPS.

Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

Yeah, I mentioned it in my presentation too. MPS is not equal to MRRS. The only requirement is for MRRS to be a multiple of MPS.

From a hardware point of view, I do not think there is any requirement

that MRRS be related to MPS. PCIe r4.0, sec 7.5.3.4 does not mention any constraints on MRRS.

Quoted 1 line

From sec 2.2.2, MPS sets the maximum TLP payload size for a function:

- A function never generates a TLP larger than its MPS
- A function treats received TLPs larger than its MPS as malformed

MPS applies to TLPs with data payloads. These are either Memory Write Requests that carry data directly, or Completions with Data in response to a Memory Read Request.

MRRS sets the maximum Read Request size a function will generate. A Memory Read Request itself contains no data payload and MPS doesn't apply to the request.

The completer of the Read Request will generate Completion TLPs carrying the requested data. The payload size of these Completion TLPs is determined by the completer's MPS setting, and it is software's responsibility to ensure that the completer's MPS is smaller than or equal to the requester's MPS.

The requester's MRRS indirectly affects the payload size because the completer will not generate a payload larger than the read request size.

Consider the case where a function has MPS=256 and MRRS=1024: if the function generates a 1024-byte read request, it should receive 4 completions, each with a 256-byte data payload.

Or, consider the case where MPS=1024 and MRRS=256: if the function generates a 256-byte read request, it should receive 1 completion with a 256-byte data payload.

In this case, the function *could* receive a Memory Write Request with a 1024-byte data payload. CPUs normally write at most a cacheline, which is probably smaller than 1024 bytes, so these large writes would most likely be peer-to-peer DMA writes.

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=b03e7495a862

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

Look at the code of Pcie_bus_safe/Pcie_bus_perf/Pcie_bus_peer2peer, it is not handling MRRS correctly. Peer2peer mode should actually use the smallest 128B for MRRS, which is also discussed in
https://patchwork.kernel.org/patch/8747521/
Safe mode may be better match MRRS to MDS for safety reason. Perf mode should set MRRS to largest accepted value, 4k for example.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

I don't disagree.

Choosing 4k is problematic. It could be 1k for one person and 2k for another person. We can have some configuration around this via kernel command line if Bjorn agrees.

In peer-to-peer mode, we currently set MPS=128 for all devices and we don't touch MRRS. Why do you think we should set MRRS=128?

As far as I can tell, that is not required by spec. Limiting MRRS may prevent a device from monopolizing bandwidth, and that may be worthwhile to do, but it is not specifically a peer-to-peer issue and I don't want to artificially tie it to MPS.

I try to avoid command-line parameters because they're hard for users and distros to deal with. In this case particularly, the issues are hard to understand so I want to untangle them a bit before we add more knobs.

Quoted 52 lines
Quoted 52 lines
Quoted 52 lines
Quoted 7 lines

Your reply says some hw logic may have poorer performance with larger MRRS sounds new to me, some HW engineer may have better look. Then in this case, "leave device recommended MRRS untouched" is more appropriate for performance purpose. Currently I can see a lot of device is using 256B MPS, and force downgrade default 4k/512B MRRS to 256B is actually getting poorer performance.

Most BIOS allows you to choose the MPS and MRRS settings. If your BIOS does, then I recommend you to use that option at this moment for tuning your system along with the safe boot option.

Quoted 40 lines

Thanks again for your time!

Ron
Beijing Memblaze Technology

-----Original Message-----
From: Sinan Kaya [mailto:[email protected]]
Sent: 2018年1月17日 12:21
To: Bo Chen <[email protected]>
Cc: Ron Yuan <[email protected]>; William Huang <[email protected]>; Fengming Wu <[email protected]>; Bjorn Helgaas <[email protected]>
Subject: Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

+Bjorn,

Hi Bo,

Quoted 8 lines · Bo Chen

Hi Sinan,

We went through linux pci related code found that nvme device has different performance when kernel boot with pcie_bus_safe or pcie_bus_perf. For details, when boot with pcie_bus_safe (eg: nvme device A: MaxPayload 128B, MaxReadReq 4096B), after pci bus init and enumerate, device A devCtl remains 128B for MaxPayload and 4096B for MaxReadReq.

pcie_bus_safe option is used to balance the maximum payload size setting across the bus. The goal is to make sure that maximum payload size is consistent. That's why, it is called safe. It also honors the maximum read request size setting done by the BIOS. It does not touch the MRRS setting.

Quoted 5 lines

While boot with pcie_bus_perf, we found pcie_write_mrrs set the read request size to the same with MPS. For nvme device A, it means both MaxPayload and MaxReadReq are set to 128B so that we observed it has limited device performance in read req.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

From our point of view on nvme device, the setting of pcie_bus_perf has lower performance compared with setting with pcie_bus_safe.

Is there any consideration for pcie_bus_perf to set MaxReadReq the same value with pcie_bus_safe? Is there any risk when we boot with pcie_bus_perf for nvme device?

Correct me if I'm wrong:

- The NVMe device has a maximum MPS of 128
- The NVMe device supports MRRS of 4096
- In SAFE mode, Linux sets MPS=128 and doesn't touch MRRS, so the BIOS setting of MRRS=4096 remains
- In PERFORMANCE mode, Linux sets MPS=128 and MRRS=128

In SAFE mode, if Linux sets MPS=128, it means there's some device that doesn't support MPS larger than 128. That's obvious in this case because the NVMe device only supports MPS=128.

In PERFORMANCE mode, Linux tries to keep larger MPS settings above the NVMe endpoint and sets MRRS as described above to make this safe. However, in this case, we apparently can't use larger MPS settings anywhere else, so all we did is reduce MRRS, which reduces your performance.

We should be able to be smarter about this, but I don't have a patch to propose. Maybe we can work on this during the next cycle.

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

/*
* If using the "performance" PCIe config, we clamp the
* read rq size to the max packet size to prevent the
* host bridge generating requests larger than we can
* cope with
*/

I found this comment in the code that seems to match the behavior you are seeing.

Problem with maximum read request size is that there is no one-size-fits-all solution. It depends on the bus-fabric of the silicon you are using. An MRRS of 4k might behave slower than 1k. 1k might behave better than 512 bytes.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

The recommendation is for these settings to be adjustable in BIOS and tweak it according to the application.

I was planning to make a change in this area but I was warned by our hardware engineers that performance does not always scale with MRRS setting.

Bjorn

author
#8 lore raw
Quoted 34 lines · Bjorn Helgaas
Quoted 33 lines · Sinan Kaya
Quoted 7 lines · Ron Yuan

we do understand that BIOS have full control of MRRS. But the problem is once kernel boot with pcie_bus_perf, MRRS setting would be overwritten by the code we complained. Btw, the reason we need to use pcie_bus_perf is because Intel white paper indicated system should use pcie_bus_perf to support NVMe SSD hot plug, specifically for a correct MPS to be set.

Quoted 6 lines

So I think at the moment back to the code, we should
1. when we don’t agree on what's the best value to set, at least remove the code MRRS=MPS, leave MRRS to default value instead of changing it incorrectly. The setting is for one device, hence the device should be responsible for itself, choosing a suitable value as default.

As I indicated below, MRRS is not a value that a PCIe endpoint device can choose. It depends on the root complex silicon. Some chip internal bus may handle 4k writes poorly; whereas, another one can handle it better.

Also, according to the spec; the default value of MRRS is 512 not 4096.

Coming back to why MRRS=MPS? I think this is because of a isochronous device statement in the spec.

"Software must limit the Max_Payload_Size for each path that supports isochronous to meet the isochronous latency. For example, all traffic flowing on a path from an isochronous capable device to the Root Complex should be limited to packets that do not exceed the Max_Payload_Size required to meet the isochronous latency requirements."

Quoted 7 lines
Quoted 2 lines

This is again a hefty price to pay for some specific PCIE endpoint type that not everybody cares about.

In PCIE_BUS_PERFORMANCE mode, pcie_write_mrrs() does try to set MRRS=MPS, but the reason is not to support isochronous. The reason is to allow us to use larger MPS settings than we otherwise would.

OK. I didn't know the history behind the change. I was looking at MPS and MRRS references in the spec.

Quoted 12 lines

Consider a switch leading to an endpoint that supports only MPS=128. The simplest approach would be to configure every device in the fabric with MPS=128. That guarantees the endpoint will never receive a TLP with a payload larger than 128 bytes.

Here's my understanding of how PCIE_BUS_PERFORMANCE works:

There are two ways an endpoint may receive TLPs with data payloads:
(1) Memory Write Requests that target the endpoint, and (2) Completions with Data in response to Memory Read Requests generated by the endpoint.

Quoted 8 lines

In PCIE_BUS_PERFORMANCE mode, we assume Memory Write Requests are not an issue because:

- We assume a CPU Memory Write Request is never larger than MPS (128 bytes in this case). This is fairly safe because CPUs generally can't write more than a cache line in one request, and most CPUs have cache lines of 128 bytes or less.

Fair assumption.

Quoted 4 lines

- We assume there's no peer-to-peer DMA, so other devices in the fabric will never send Memory Write Requests to the endpoint, so we don't need to limit their MPS settings.

Quoted 8 lines

That leaves Completions. We limit the size of Completions by limiting MRRS. If we set the endpoint's MRRS to its MPS (128 in this case), it will never request more than MPS bytes at a time, so it will never receive a Completion with more than MPS bytes.

Therefore, we may be able to configure other devices in the fabric with MPS larger than 128, which may benefit those devices.

This is still problematic. One application may be doing a lot of writes compared to reads. We prefer maximizing endpoint write performance compared to read performance by reducing the MRRS setting.

Quoted 50 lines
Quoted 39 lines
Quoted 6 lines

2. try to get a common understanding what is the best performance for MRRS, maybe try largest MRRS supported by the system/device, or have additional command line input.

Item 1 is something we should do as first step. For item 2, we need bigger group of experienced PCIe engineers to discuss.

We need to understand that MPS and MRRS have much broader impact. All silicon is different.

I don't think there is a common solution to this and my preference is for these parameters to be specified along with the perf option. Then, we can adjust things accordingly.

Let's see what others think about this.

Quoted 23 lines

Here is one link for your reference:
https://www.xilinx.com/support/documentation/white_papers/wp350.pdf

Quoted 19 lines · Sinan Kaya
Quoted 17 lines · Ron Yuan

The code we refer to is
static void pcie_write_mrrs(struct pci_dev *dev) { ...
/* For Max performance, the MRRS must be set to the largest supported
* value. However, it cannot be configured larger than the MPS the
* device or the bus can support. This should already be properly
* configured by a prior call to pcie_write_mps.
*/
mrrs = pcie_get_mps(dev);
...
}

Here the statement " However, it cannot be configured larger than the MPS the device or the bus can support." is obviously incorrect. A simple against statement is from PCIe spec, which define default MPS to 128B, and *default MRRS to 512B*. From HW understanding, MRRS is multiple of MPS in the system, it must be no less than MPS.

Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

Yeah, I mentioned it in my presentation too. MPS is not equal to MRRS. The only requirement is for MRRS to be a multiple of MPS.

From a hardware point of view, I do not think there is any requirement

that MRRS be related to MPS. PCIe r4.0, sec 7.5.3.4 does not mention any constraints on MRRS.

Quoted 1 line

From sec 2.2.2, MPS sets the maximum TLP payload size for a function:

- A function never generates a TLP larger than its MPS
- A function treats received TLPs larger than its MPS as malformed

Quoted 6 lines

MPS applies to TLPs with data payloads. These are either Memory Write Requests that carry data directly, or Completions with Data in response to a Memory Read Request>
MRRS sets the maximum Read Request size a function will generate. A Memory Read Request itself contains no data payload and MPS doesn't apply to the request.

Agreed, MRRS has an indirect effect on the completions. It determines the number of completions that endpoint will receive. As the ratio of MPS/MRRS gets closer to 0, the overhead becomes more significant.

Quoted 15 lines

The completer of the Read Request will generate Completion TLPs carrying the requested data. The payload size of these Completion TLPs is determined by the completer's MPS setting, and it is software's responsibility to ensure that the completer's MPS is smaller than or equal to the requester's MPS.

The requester's MRRS indirectly affects the payload size because the completer will not generate a payload larger than the read request size.

Consider the case where a function has MPS=256 and MRRS=1024: if the function generates a 1024-byte read request, it should receive 4 completions, each with a 256-byte data payload.

Quoted 8 lines

Or, consider the case where MPS=1024 and MRRS=256: if the function generates a 256-byte read request, it should receive 1 completion with a 256-byte data payload.

In this case, the function *could* receive a Memory Write Request with a 1024-byte data payload. CPUs normally write at most a cacheline, which is probably smaller than 1024 bytes, so these large writes would most likely be peer-to-peer DMA writes.

Why do you think that CPU is involved in fetching a memory read response to a read request?

If an endpoint tries to read 4k from system memory, this would be typically done by SOC internal fabric not by the CPU. How fast a fabric can feed 1k to the endpoint depends on the capability of the fabric. Some fabric might do better with 1k, others could be better with 256 bytes.

CPU would be only involved if it was doing a memory write to the endpoint by itself. Then, mostly cache line or CPU write instruction size is in effect.

Quoted 26 lines

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=b03e7495a862

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

Look at the code of Pcie_bus_safe/Pcie_bus_perf/Pcie_bus_peer2peer, it is not handling MRRS correctly. Peer2peer mode should actually use the smallest 128B for MRRS, which is also discussed in
https://patchwork.kernel.org/patch/8747521/
Safe mode may be better match MRRS to MDS for safety reason. Perf mode should set MRRS to largest accepted value, 4k for example.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

I don't disagree.

Choosing 4k is problematic. It could be 1k for one person and 2k for another person. We can have some configuration around this via kernel command line if Bjorn agrees.

In peer-to-peer mode, we currently set MPS=128 for all devices and we don't touch MRRS. Why do you think we should set MRRS=128?

As far as I can tell, that is not required by spec. Limiting MRRS may prevent a device from monopolizing bandwidth, and that may be worthwhile to do, but it is not specifically a peer-to-peer issue and I don't want to artificially tie it to MPS.

Quoted 4 lines

I try to avoid command-line parameters because they're hard for users and distros to deal with. In this case particularly, the issues are hard to understand so I want to untangle them a bit before we add more knobs.

OK.

Quoted 73 lines
Quoted 53 lines
Quoted 53 lines
Quoted 53 lines
Quoted 8 lines

Your reply says some hw logic may have poorer performance with larger MRRS sounds new to me, some HW engineer may have better look. Then in this case, "leave device recommended MRRS untouched" is more appropriate for performance purpose. Currently I can see a lot of device is using 256B MPS, and force downgrade default 4k/512B MRRS to 256B is actually getting poorer performance.

Most BIOS allows you to choose the MPS and MRRS settings. If your BIOS does, then I recommend you to use that option at this moment for tuning your system along with the safe boot option.

Quoted 40 lines

Thanks again for your time!

Ron
Beijing Memblaze Technology

-----Original Message-----
From: Sinan Kaya [mailto:[email protected]]
Sent: 2018年1月17日 12:21
To: Bo Chen <[email protected]>
Cc: Ron Yuan <[email protected]>; William Huang <[email protected]>; Fengming Wu <[email protected]>; Bjorn Helgaas <[email protected]>
Subject: Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

+Bjorn,

Hi Bo,

Quoted 8 lines · Bo Chen

Hi Sinan,

We went through linux pci related code found that nvme device has different performance when kernel boot with pcie_bus_safe or pcie_bus_perf. For details, when boot with pcie_bus_safe (eg: nvme device A: MaxPayload 128B, MaxReadReq 4096B), after pci bus init and enumerate, device A devCtl remains 128B for MaxPayload and 4096B for MaxReadReq.

pcie_bus_safe option is used to balance the maximum payload size setting across the bus. The goal is to make sure that maximum payload size is consistent. That's why, it is called safe. It also honors the maximum read request size setting done by the BIOS. It does not touch the MRRS setting.

Quoted 5 lines

While boot with pcie_bus_perf, we found pcie_write_mrrs set the read request size to the same with MPS. For nvme device A, it means both MaxPayload and MaxReadReq are set to 128B so that we observed it has limited device performance in read req.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

From our point of view on nvme device, the setting of pcie_bus_perf has lower performance compared with setting with pcie_bus_safe.

Is there any consideration for pcie_bus_perf to set MaxReadReq the same value with pcie_bus_safe? Is there any risk when we boot with pcie_bus_perf for nvme device?

Correct me if I'm wrong:

- The NVMe device has a maximum MPS of 128
- The NVMe device supports MRRS of 4096
- In SAFE mode, Linux sets MPS=128 and doesn't touch MRRS, so the BIOS setting of MRRS=4096 remains
- In PERFORMANCE mode, Linux sets MPS=128 and MRRS=128

In SAFE mode, if Linux sets MPS=128, it means there's some device that doesn't support MPS larger than 128. That's obvious in this case because the NVMe device only supports MPS=128.

Quoted 9 lines

In PERFORMANCE mode, Linux tries to keep larger MPS settings above the NVMe endpoint and sets MRRS as described above to make this safe. However, in this case, we apparently can't use larger MPS settings anywhere else, so all we did is reduce MRRS, which reduces your performance.

We should be able to be smarter about this, but I don't have a patch to propose. Maybe we can work on this during the next cycle.

Sure, I was planning to work on this once the CRS change gets merged. I was doing feasibility analysis in the meantime.

Quoted 24 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

/*
* If using the "performance" PCIe config, we clamp the
* read rq size to the max packet size to prevent the
* host bridge generating requests larger than we can
* cope with
*/

I found this comment in the code that seems to match the behavior you are seeing.

Problem with maximum read request size is that there is no one-size-fits-all solution. It depends on the bus-fabric of the silicon you are using. An MRRS of 4k might behave slower than 1k. 1k might behave better than 512 bytes.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

The recommendation is for these settings to be adjustable in BIOS and tweak it according to the application.

I was planning to make a change in this area but I was warned by our hardware engineers that performance does not always scale with MRRS setting.

Bjorn

Signature · 3 lines
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
author
#9 lore raw
Quoted 6 lines · Sinan Kaya
Quoted 4 lines

In this case, the function *could* receive a Memory Write Request with a 1024-byte data payload. CPUs normally write at most a cacheline, which is probably smaller than 1024 bytes, so these large writes would most likely be peer-to-peer DMA writes.

Why do you think that CPU is involved in fetching a memory read response to a read request?

Never mind. I realized you started talking about Memory Writes after the completions in the above paragraph now. Another source for memory write could be an SOC internal DMA memcpy engine as well as a peer-to-peer access.

Quoted 5 lines

If an endpoint tries to read 4k from system memory, this would be typically done by SOC internal fabric not by the CPU. How fast a fabric can feed 1k to the endpoint depends on the capability of the fabric. Some fabric might do better with 1k, others could be better with 256 bytes.

Signature · 3 lines
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
#10 lore raw
In reply to Sinan Kaya
Quoted 32 lines · Sinan Kaya
Quoted 12 lines · Bjorn Helgaas

Consider a switch leading to an endpoint that supports only MPS=128. The simplest approach would be to configure every device in the fabric with MPS=128. That guarantees the endpoint will never receive a TLP with a payload larger than 128 bytes.

Here's my understanding of how PCIE_BUS_PERFORMANCE works:

There are two ways an endpoint may receive TLPs with data payloads:
(1) Memory Write Requests that target the endpoint, and (2) Completions with Data in response to Memory Read Requests generated by the endpoint.

Quoted 7 lines

In PCIE_BUS_PERFORMANCE mode, we assume Memory Write Requests are not an issue because:

- We assume a CPU Memory Write Request is never larger than MPS (128 bytes in this case). This is fairly safe because CPUs generally can't write more than a cache line in one request, and most CPUs have cache lines of 128 bytes or less.

Fair assumption.

Quoted 9 lines

- We assume there's no peer-to-peer DMA, so other devices in the fabric will never send Memory Write Requests to the endpoint, so we don't need to limit their MPS settings.

That leaves Completions. We limit the size of Completions by limiting MRRS. If we set the endpoint's MRRS to its MPS (128 in this case), it will never request more than MPS bytes at a time, so it will never receive a Completion with more than MPS bytes.

Quoted 7 lines
Quoted 2 lines

Therefore, we may be able to configure other devices in the fabric with MPS larger than 128, which may benefit those devices.

This is still problematic. One application may be doing a lot of writes compared to reads. We prefer maximizing endpoint write performance compared to read performance by reducing the MRRS setting.

Help me understand exactly what is problematic. No matter what your read/write mix is, a single device in isolation should get the best performance with both MPS and MRRS at the highest possible settings.

Reducing MPS may be necessary if there are several devices in the hierarchy and one requires a smaller MPS than the others. That obviously reduces the maximum read and write performance.

Reducing the MRRS may be useful to prevent one device from hogging a link, but of course, it reduces read performance for that device because we need more read requests.

author
#11 lore raw
Quoted 11 lines · Bjorn Helgaas
Quoted 7 lines
Quoted 7 lines

That leaves Completions. We limit the size of Completions by limiting MRRS. If we set the endpoint's MRRS to its MPS (128 in this case), it will never request more than MPS bytes at a time, so it will never receive a Completion with more than MPS bytes.

Therefore, we may be able to configure other devices in the fabric with MPS larger than 128, which may benefit those devices.

Help me understand exactly what is problematic. No matter what your read/write mix is, a single device in isolation should get the best performance with both MPS and MRRS at the highest possible settings.

The performance approach is trying to maximize MPS while reducing MRRS value to MPS value. Meaning improving write performance while trading off read performance.

Quoted 9 lines

Reducing MPS may be necessary if there are several devices in the hierarchy and one requires a smaller MPS than the others. That obviously reduces the maximum read and write performance.

Reducing the MRRS may be useful to prevent one device from hogging a link, but of course, it reduces read performance for that device because we need more read requests.

Maybe, a picture could help.

			root (MPS=256)
			 |
		 ------------------
		/		   \
	   bridge0 (MPS=256)      bridge1 (MPS=128)
	    /				\
	   EP0 (MPS=256)		EP1 (MPS=128)

If I understood this right, code allows the configuration above with the performance mode so that MPS doesn't have to be uniform across the tree.

It just needs to be consistent between the root port and endpoints.

Why are we reducing MRRS in this case?

Are we assuming that root bus cannot handle more than 256 bytes and bridge1 would be starved while root bus is passing the completions to bridge0?

If yes, the answer to the assumption depends on the architecture. One silicon could serve any completions in any speed.

It depends on how root bus was implemented. If it is a real one or an emulated one. I think most silicon emulate root bus.

Signature · 3 lines
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
#12 lore raw
Quoted 10 lines · Sinan Kaya
Quoted 8 lines · Bjorn Helgaas
Quoted 8 lines
Quoted 8 lines

That leaves Completions. We limit the size of Completions by limiting MRRS. If we set the endpoint's MRRS to its MPS (128 in this case), it will never request more than MPS bytes at a time, so it will never receive a Completion with more than MPS bytes.

Therefore, we may be able to configure other devices in the fabric with MPS larger than 128, which may benefit those devices.

Quoted 8 lines
Quoted 4 lines

Help me understand exactly what is problematic. No matter what your read/write mix is, a single device in isolation should get the best performance with both MPS and MRRS at the highest possible settings.

The performance approach is trying to maximize MPS while reducing MRRS value to MPS value. Meaning improving write performance while trading off read performance.

Right. The intent of the PERFORMANCE mode is exactly to maximize MPS (which maximizes write performance) by reducing MRRS in some cases (which reduces read performance).

Quoted 4 lines · You had
Quoted 4 lines
Quoted 4 lines

This is still problematic. One application may be doing a lot of writes compared to reads. We prefer maximizing endpoint write performance compared to read performance by reducing the MRRS setting.

so I thought you had an issue with this, and I was trying to understand what you wanted instead.

Quoted 18 lines
Quoted 7 lines

Reducing MPS may be necessary if there are several devices in the hierarchy and one requires a smaller MPS than the others. That obviously reduces the maximum read and write performance.

Reducing the MRRS may be useful to prevent one device from hogging a link, but of course, it reduces read performance for that device because we need more read requests.

Maybe, a picture could help.

               root (MPS=256)
                 |
         ------------------
        /                  \
   bridge0 (MPS=256)      bridge1 (MPS=128)
      /                       \
    EP0 (MPS=256)            EP1 (MPS=128)
Quoted 3 lines

If I understood this right, code allows the configuration above with the performance mode so that MPS doesn't have to be uniform across the tree.

Yes. In PERFORMANCE mode, we will set EP1's MRRS=128 and EP0's MRRS=256, just as you show.

Quoted 1 line

It just needs to be consistent between the root port and endpoints.

No, it doesn't need to be consistent. In PERFORMANCE mode, we'll set the root's MPS=256 and EP1's MPS=128.

(I'm not actually 100% convinced that the PERFORMANCE mode approach of reducing MRRS is safe, necessary, and maintainable. I suspect that in many of the interesting cases, the device we care about is the only one below a Root Port, and we can get the performance we need by maximizing MPS and MRRS for that Root Port and its children, independent of the rest of the system.)

Quoted 1 line

Why are we reducing MRRS in this case?

We have to set EP1's MRRS=128 so it will never receive a completion larger than 128 bytes. If we set EP1's MRRS=256, it could receive 256-byte TLPs, which it would treat as malformed. (We also assume no peer-to-peer DMA that targets EP1.)

Quoted 3 lines

Are we assuming that root bus cannot handle more than 256 bytes and bridge1 would be starved while root bus is passing the completions to bridge0?

We don't have to assume. Every device tells us via Dev Cap what size TLPs it can handle. In your example, I assume the root's Dev Cap tells us it supports 256-byte TLPs.

PERFORMANCE mode reduces MRRS not because of a starvation issue, but because reducing EP1's MRRS allows EP0 to use a larger MPS.

author
#13 lore raw
Quoted 33 lines · Bjorn Helgaas
Quoted 20 lines · Sinan Kaya
Quoted 7 lines · Bjorn Helgaas

Reducing MPS may be necessary if there are several devices in the hierarchy and one requires a smaller MPS than the others. That obviously reduces the maximum read and write performance.

Reducing the MRRS may be useful to prevent one device from hogging a link, but of course, it reduces read performance for that device because we need more read requests.

Maybe, a picture could help.

               root (MPS=256)
                 |
         ------------------
        /                  \
   bridge0 (MPS=256)      bridge1 (MPS=128)
      /                       \
    EP0 (MPS=256)            EP1 (MPS=128)
Quoted 3 lines

If I understood this right, code allows the configuration above with the performance mode so that MPS doesn't have to be uniform across the tree.

Yes. In PERFORMANCE mode, we will set EP1's MRRS=128 and EP0's MRRS=256, just as you show.

Quoted 1 line

It just needs to be consistent between the root port and endpoints.

No, it doesn't need to be consistent. In PERFORMANCE mode, we'll set the root's MPS=256 and EP1's MPS=128.

Quoted 6 lines

(I'm not actually 100% convinced that the PERFORMANCE mode approach of reducing MRRS is safe, necessary, and maintainable. I suspect that in many of the interesting cases, the device we care about is the only one below a Root Port, and we can get the performance we need by maximizing MPS and MRRS for that Root Port and its children, independent of the rest of the system.)

Maybe, I started seeing more and more NVMe devices behind a switch every day. That's why, I'm concerned.

Quoted 7 lines
Quoted 2 lines

Why are we reducing MRRS in this case?

We have to set EP1's MRRS=128 so it will never receive a completion larger than 128 bytes. If we set EP1's MRRS=256, it could receive 256-byte TLPs, which it would treat as malformed. (We also assume no peer-to-peer DMA that targets EP1.)

What if we were to keep root port MPS as 128? and not touch the BIOS configured MRRS (4k) ?

Everybody should be happy, right?

I know there is a rule to check the completions against MPS. Root port could generate transactions that is a multiple of 128 bytes for reads.

Is there any rule against checking incoming writes?

Quoted 12 lines
Quoted 4 lines

Are we assuming that root bus cannot handle more than 256 bytes and bridge1 would be starved while root bus is passing the completions to bridge0?

We don't have to assume. Every device tells us via Dev Cap what size TLPs it can handle. In your example, I assume the root's Dev Cap tells us it supports 256-byte TLPs.

PERFORMANCE mode reduces MRRS not because of a starvation issue, but because reducing EP1's MRRS allows EP0 to use a larger MPS.

Signature · 3 lines
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
#14 lore raw
Quoted 41 lines · Sinan Kaya
Quoted 33 lines · Bjorn Helgaas
Quoted 20 lines · Sinan Kaya
Quoted 7 lines · Bjorn Helgaas

Reducing MPS may be necessary if there are several devices in the hierarchy and one requires a smaller MPS than the others. That obviously reduces the maximum read and write performance.

Reducing the MRRS may be useful to prevent one device from hogging a link, but of course, it reduces read performance for that device because we need more read requests.

Maybe, a picture could help.

               root (MPS=256)
                 |
         ------------------
        /                  \
   bridge0 (MPS=256)      bridge1 (MPS=128)
      /                       \
    EP0 (MPS=256)            EP1 (MPS=128)
Quoted 3 lines

If I understood this right, code allows the configuration above with the performance mode so that MPS doesn't have to be uniform across the tree.

Yes. In PERFORMANCE mode, we will set EP1's MRRS=128 and EP0's MRRS=256, just as you show.

Quoted 1 line

It just needs to be consistent between the root port and endpoints.

No, it doesn't need to be consistent. In PERFORMANCE mode, we'll set the root's MPS=256 and EP1's MPS=128.

Quoted 6 lines

(I'm not actually 100% convinced that the PERFORMANCE mode approach of reducing MRRS is safe, necessary, and maintainable. I suspect that in many of the interesting cases, the device we care about is the only one below a Root Port, and we can get the performance we need by maximizing MPS and MRRS for that Root Port and its children, independent of the rest of the system.)

Quoted 2 lines

Maybe, I started seeing more and more NVMe devices behind a switch every day. That's why, I'm concerned.

Are there a mix of devices that support large MPS and those that only support a smaller MPS behind the same switch?

I don't have any actual data about topologies of interest. I'm just guessing that high-performance devices will often have their own root port, without being mixed in with low-speed devices. It's OK to have switches in the path; it's the mixing high-speed with low-speed devices that causes the problems.

Quoted 11 lines
Quoted 6 lines
Quoted 1 line

Why are we reducing MRRS in this case?

We have to set EP1's MRRS=128 so it will never receive a completion larger than 128 bytes. If we set EP1's MRRS=256, it could receive 256-byte TLPs, which it would treat as malformed. (We also assume no peer-to-peer DMA that targets EP1.)

What if we were to keep root port MPS as 128? and not touch the BIOS configured MRRS (4k) ?

Everybody should be happy, right?

No. In your picture (which helps a lot, thank you!), if you set the root's MPS=128, EP1 will be fine no matter what its MRRS is because the entire path has MPS=128.

But if EP0's MRRS is larger than 128, it's in trouble because it may issue a 256-byte DMA write, which the root port will treat as malformed. The root port, as the receiver of that Memory Write Request, is required to check the TLP against the MPS in its Device Control (not Device Capability) register (this is in PCIe r4.0, sec 2.2.2).

Quoted 1 line

I know there is a rule to check the completions against MPS.

Sec 2.2.2 says a receiver must check "data payloads". I think that includes both Completions and Memory Writes.

Quoted 2 lines

Root port could generate transactions that is a multiple of 128 bytes for reads.

If the root port generates a 256-byte Memory Read request to EP1, that's fine because EP1 will only respond with 128-byte completions. If it sends that 256-byte Memory Read to EP0, we have a problem because EP0 may generate a 256-byte completion, which will cause an error if the root port has MPS=128.

Quoted 1 line

Is there any rule against checking incoming writes?

Sec 2.2.2 says a receiver *must* check the payload size of incoming Memory Write requests.

author
#15 lore raw
Quoted 42 lines · Bjorn Helgaas
Quoted 41 lines · Sinan Kaya
Quoted 39 lines · Bjorn Helgaas
Quoted 20 lines · Sinan Kaya
Quoted 7 lines · Bjorn Helgaas

Reducing MPS may be necessary if there are several devices in the hierarchy and one requires a smaller MPS than the others. That obviously reduces the maximum read and write performance.

Reducing the MRRS may be useful to prevent one device from hogging a link, but of course, it reduces read performance for that device because we need more read requests.

Maybe, a picture could help.

               root (MPS=256)
                 |
         ------------------
        /                  \
   bridge0 (MPS=256)      bridge1 (MPS=128)
      /                       \
    EP0 (MPS=256)            EP1 (MPS=128)
Quoted 3 lines

If I understood this right, code allows the configuration above with the performance mode so that MPS doesn't have to be uniform across the tree.

Yes. In PERFORMANCE mode, we will set EP1's MRRS=128 and EP0's MRRS=256, just as you show.

Quoted 1 line

It just needs to be consistent between the root port and endpoints.

No, it doesn't need to be consistent. In PERFORMANCE mode, we'll set the root's MPS=256 and EP1's MPS=128.

(I'm not actually 100% convinced that the PERFORMANCE mode approach of reducing MRRS is safe, necessary, and maintainable. I suspect that in many of the interesting cases, the device we care about is the only one below a Root Port, and we can get the performance we need by maximizing MPS and MRRS for that Root Port and its children, independent of the rest of the system.)

Quoted 5 lines
Quoted 2 lines

Maybe, I started seeing more and more NVMe devices behind a switch every day. That's why, I'm concerned.

Are there a mix of devices that support large MPS and those that only support a smaller MPS behind the same switch?

I guess we should understand the environment from Ron Yuan. Can we see lspci -vvv output of your system?

We could maybe detect coexistence of slow and fast device condition and put some suggestions like moving the card to another slot as a best effort solution.

Quoted 6 lines

I don't have any actual data about topologies of interest. I'm just guessing that high-performance devices will often have their own root port, without being mixed in with low-speed devices. It's OK to have switches in the path; it's the mixing high-speed with low-speed devices that causes the problems.

Agreed. Mine was just a general rant. I was curious if we could make it better.

Quoted 17 lines
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

Why are we reducing MRRS in this case?

We have to set EP1's MRRS=128 so it will never receive a completion larger than 128 bytes. If we set EP1's MRRS=256, it could receive 256-byte TLPs, which it would treat as malformed. (We also assume no peer-to-peer DMA that targets EP1.)

Quoted 4 lines

What if we were to keep root port MPS as 128? and not touch the BIOS configured MRRS (4k) ?

Everybody should be happy, right?

No. In your picture (which helps a lot, thank you!), if you set the root's MPS=128, EP1 will be fine no matter what its MRRS is because the entire path has MPS=128.

Quoted 6 lines

But if EP0's MRRS is larger than 128, it's in trouble because it may issue a 256-byte DMA write, which the root port will treat as malformed. The root port, as the receiver of that Memory Write Request, is required to check the TLP against the MPS in its Device Control (not Device Capability) register (this is in PCIe r4.0, sec 2.2.2).

Yeah, this breaks my theory.

Quoted 20 lines
Quoted 2 lines

I know there is a rule to check the completions against MPS.

Sec 2.2.2 says a receiver must check "data payloads". I think that includes both Completions and Memory Writes.

Quoted 2 lines

Root port could generate transactions that is a multiple of 128 bytes for reads.

If the root port generates a 256-byte Memory Read request to EP1, that's fine because EP1 will only respond with 128-byte completions. If it sends that 256-byte Memory Read to EP0, we have a problem because EP0 may generate a 256-byte completion, which will cause an error if the root port has MPS=128.

Quoted 1 line

Is there any rule against checking incoming writes?

Sec 2.2.2 says a receiver *must* check the payload size of incoming Memory Write requests.

Signature · 3 lines
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
#16 lore raw

Bjorn and Sinan,
Thanks for the discussion.

Quoted 2 lines

PERFORMANCE mode reduces MRRS not because of a starvation issue, but because reducing EP1's MRRS allows EP0 to use a larger MPS.

Looks like this case is talking about EP1 requests data directly from EP0, using MRRS to control the return data payload, while still keeping the traffic from EP0 to RC in 256B. This is reasonable.
However, is this common in the real world? We can see kernel driver usually control the whole system, set MPS well to align all EPs. Only in rare case where different EP has no other sync method, then setting MRRS for EP1 can provide more "safe" for itself. This does not match with "perf" name, so I proposed earlier move the MRRS setting to "safe" mode to make it clear.

In my situation, data center customer is using many NVMe SSDs, they care most about best performance from each SSD, while there is no out of control EP to EP access.

I will provide a full lspci log tomorrow. First attach the screen shoot in "safe" mode for different devices. You can see MPS are set to 256B while MRRS default remain differently. In Perf mode, all MRRS would be limited to 256B and loss some performance.

Thanks
Ron

-----Original Message-----
From: Sinan Kaya [mailto:[email protected]]
Sent: 2018年1月23日 10:27
To: Bjorn Helgaas <[email protected]>
Cc: Ron Yuan <[email protected]>; Bjorn Helgaas <[email protected]>; Bo Chen <[email protected]>; William Huang <[email protected]>; Fengming Wu <[email protected]>; Jason Jiang <[email protected]>; Radjendirane Codandaramane <[email protected]>; Ramyakanth Edupuganti <[email protected]>; William Cheng <[email protected]>; Kim Helper (khelper) <[email protected]>; Linux PCI <[email protected]>
Subject: Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

Quoted 42 lines · Bjorn Helgaas
Quoted 41 lines · Sinan Kaya
Quoted 39 lines · Bjorn Helgaas
Quoted 20 lines · Sinan Kaya
Quoted 7 lines · Bjorn Helgaas

Reducing MPS may be necessary if there are several devices in the hierarchy and one requires a smaller MPS than the others. That obviously reduces the maximum read and write performance.

Reducing the MRRS may be useful to prevent one device from hogging a link, but of course, it reduces read performance for that device because we need more read requests.

Maybe, a picture could help.

               root (MPS=256)
                 |
         ------------------
        /                  \
   bridge0 (MPS=256)      bridge1 (MPS=128)
      /                       \
    EP0 (MPS=256)            EP1 (MPS=128)
Quoted 3 lines

If I understood this right, code allows the configuration above with the performance mode so that MPS doesn't have to be uniform across the tree.

Yes. In PERFORMANCE mode, we will set EP1's MRRS=128 and EP0's MRRS=256, just as you show.

Quoted 1 line

It just needs to be consistent between the root port and endpoints.

No, it doesn't need to be consistent. In PERFORMANCE mode, we'll set the root's MPS=256 and EP1's MPS=128.

(I'm not actually 100% convinced that the PERFORMANCE mode approach of reducing MRRS is safe, necessary, and maintainable. I suspect that in many of the interesting cases, the device we care about is the only one below a Root Port, and we can get the performance we need by maximizing MPS and MRRS for that Root Port and its children, independent of the rest of the system.)

Quoted 5 lines
Quoted 2 lines

Maybe, I started seeing more and more NVMe devices behind a switch every day. That's why, I'm concerned.

Are there a mix of devices that support large MPS and those that only support a smaller MPS behind the same switch?

I guess we should understand the environment from Ron Yuan. Can we see lspci -vvv output of your system?

We could maybe detect coexistence of slow and fast device condition and put some suggestions like moving the card to another slot as a best effort solution.

Quoted 6 lines

I don't have any actual data about topologies of interest. I'm just guessing that high-performance devices will often have their own root port, without being mixed in with low-speed devices. It's OK to have switches in the path; it's the mixing high-speed with low-speed devices that causes the problems.

Agreed. Mine was just a general rant. I was curious if we could make it better.

Quoted 17 lines
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

Why are we reducing MRRS in this case?

We have to set EP1's MRRS=128 so it will never receive a completion larger than 128 bytes. If we set EP1's MRRS=256, it could receive 256-byte TLPs, which it would treat as malformed. (We also assume no peer-to-peer DMA that targets EP1.)

Quoted 4 lines

What if we were to keep root port MPS as 128? and not touch the BIOS configured MRRS (4k) ?

Everybody should be happy, right?

No. In your picture (which helps a lot, thank you!), if you set the root's MPS=128, EP1 will be fine no matter what its MRRS is because the entire path has MPS=128.

Quoted 6 lines

But if EP0's MRRS is larger than 128, it's in trouble because it may issue a 256-byte DMA write, which the root port will treat as malformed. The root port, as the receiver of that Memory Write Request, is required to check the TLP against the MPS in its Device Control (not Device Capability) register (this is in PCIe r4.0, sec 2.2.2).

Yeah, this breaks my theory.

Quoted 15 lines
Quoted 2 lines

I know there is a rule to check the completions against MPS.

Sec 2.2.2 says a receiver must check "data payloads". I think that includes both Completions and Memory Writes.

Quoted 2 lines

Root port could generate transactions that is a multiple of 128 bytes for reads.

If the root port generates a 256-byte Memory Read request to EP1, that's fine because EP1 will only respond with 128-byte completions. If it sends that 256-byte Memory Read to EP0, we have a problem because EP0 may generate a 256-byte completion, which will cause an error if the root port has MPS=128.

Quoted 5 lines
Quoted 1 line

Is there any rule against checking incoming writes?

Sec 2.2.2 says a receiver *must* check the payload size of incoming Memory Write requests.

--
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.

#17 lore raw

Just got the log, see attachment. Kernel is under "perf" mode. SSD and Ethernet controller are both set to 256B MRRS.

-----Original Message-----
From: Ron Yuan
Sent: 2018年1月23日 21:26
To: 'Sinan Kaya' <[email protected]>; Bjorn Helgaas <[email protected]>
Cc: Bjorn Helgaas <[email protected]>; Bo Chen <[email protected]>; William Huang <[email protected]>; Fengming Wu <[email protected]>; Jason Jiang <[email protected]>; Radjendirane Codandaramane <[email protected]>; Ramyakanth Edupuganti <[email protected]>; William Cheng <[email protected]>; Kim Helper (khelper) <[email protected]>; Linux PCI <[email protected]>
Subject: RE: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

Bjorn and Sinan,
Thanks for the discussion.

Quoted 2 lines

PERFORMANCE mode reduces MRRS not because of a starvation issue, but because reducing EP1's MRRS allows EP0 to use a larger MPS.

Looks like this case is talking about EP1 requests data directly from EP0, using MRRS to control the return data payload, while still keeping the traffic from EP0 to RC in 256B. This is reasonable.
However, is this common in the real world? We can see kernel driver usually control the whole system, set MPS well to align all EPs. Only in rare case where different EP has no other sync method, then setting MRRS for EP1 can provide more "safe" for itself. This does not match with "perf" name, so I proposed earlier move the MRRS setting to "safe" mode to make it clear.

In my situation, data center customer is using many NVMe SSDs, they care most about best performance from each SSD, while there is no out of control EP to EP access.

I will provide a full lspci log tomorrow. First attach the screen shoot in "safe" mode for different devices. You can see MPS are set to 256B while MRRS default remain differently. In Perf mode, all MRRS would be limited to 256B and loss some performance.

Thanks
Ron

-----Original Message-----
From: Sinan Kaya [mailto:[email protected]]
Sent: 2018年1月23日 10:27
To: Bjorn Helgaas <[email protected]>
Cc: Ron Yuan <[email protected]>; Bjorn Helgaas <[email protected]>; Bo Chen <[email protected]>; William Huang <[email protected]>; Fengming Wu <[email protected]>; Jason Jiang <[email protected]>; Radjendirane Codandaramane <[email protected]>; Ramyakanth Edupuganti <[email protected]>; William Cheng <[email protected]>; Kim Helper (khelper) <[email protected]>; Linux PCI <[email protected]>
Subject: Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

Quoted 42 lines · Bjorn Helgaas
Quoted 41 lines · Sinan Kaya
Quoted 39 lines · Bjorn Helgaas
Quoted 20 lines · Sinan Kaya
Quoted 7 lines · Bjorn Helgaas

Reducing MPS may be necessary if there are several devices in the hierarchy and one requires a smaller MPS than the others. That obviously reduces the maximum read and write performance.

Reducing the MRRS may be useful to prevent one device from hogging a link, but of course, it reduces read performance for that device because we need more read requests.

Maybe, a picture could help.

               root (MPS=256)
                 |
         ------------------
        /                  \
   bridge0 (MPS=256)      bridge1 (MPS=128)
      /                       \
    EP0 (MPS=256)            EP1 (MPS=128)
Quoted 3 lines

If I understood this right, code allows the configuration above with the performance mode so that MPS doesn't have to be uniform across the tree.

Yes. In PERFORMANCE mode, we will set EP1's MRRS=128 and EP0's MRRS=256, just as you show.

Quoted 1 line

It just needs to be consistent between the root port and endpoints.

No, it doesn't need to be consistent. In PERFORMANCE mode, we'll set the root's MPS=256 and EP1's MPS=128.

(I'm not actually 100% convinced that the PERFORMANCE mode approach of reducing MRRS is safe, necessary, and maintainable. I suspect that in many of the interesting cases, the device we care about is the only one below a Root Port, and we can get the performance we need by maximizing MPS and MRRS for that Root Port and its children, independent of the rest of the system.)

Quoted 5 lines
Quoted 2 lines

Maybe, I started seeing more and more NVMe devices behind a switch every day. That's why, I'm concerned.

Are there a mix of devices that support large MPS and those that only support a smaller MPS behind the same switch?

I guess we should understand the environment from Ron Yuan. Can we see lspci -vvv output of your system?

We could maybe detect coexistence of slow and fast device condition and put some suggestions like moving the card to another slot as a best effort solution.

Quoted 6 lines

I don't have any actual data about topologies of interest. I'm just guessing that high-performance devices will often have their own root port, without being mixed in with low-speed devices. It's OK to have switches in the path; it's the mixing high-speed with low-speed devices that causes the problems.

Agreed. Mine was just a general rant. I was curious if we could make it better.

Quoted 17 lines
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

Why are we reducing MRRS in this case?

We have to set EP1's MRRS=128 so it will never receive a completion larger than 128 bytes. If we set EP1's MRRS=256, it could receive 256-byte TLPs, which it would treat as malformed. (We also assume no peer-to-peer DMA that targets EP1.)

Quoted 4 lines

What if we were to keep root port MPS as 128? and not touch the BIOS configured MRRS (4k) ?

Everybody should be happy, right?

No. In your picture (which helps a lot, thank you!), if you set the root's MPS=128, EP1 will be fine no matter what its MRRS is because the entire path has MPS=128.

Quoted 6 lines

But if EP0's MRRS is larger than 128, it's in trouble because it may issue a 256-byte DMA write, which the root port will treat as malformed. The root port, as the receiver of that Memory Write Request, is required to check the TLP against the MPS in its Device Control (not Device Capability) register (this is in PCIe r4.0, sec 2.2.2).

Yeah, this breaks my theory.

Quoted 15 lines
Quoted 2 lines

I know there is a rule to check the completions against MPS.

Sec 2.2.2 says a receiver must check "data payloads". I think that includes both Completions and Memory Writes.

Quoted 2 lines

Root port could generate transactions that is a multiple of 128 bytes for reads.

If the root port generates a 256-byte Memory Read request to EP1, that's fine because EP1 will only respond with 128-byte completions. If it sends that 256-byte Memory Read to EP0, we have a problem because EP0 may generate a 256-byte completion, which will cause an error if the root port has MPS=128.

Quoted 5 lines
Quoted 1 line

Is there any rule against checking incoming writes?

Sec 2.2.2 says a receiver *must* check the payload size of incoming Memory Write requests.

--
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.

00:00.0 Host bridge: Intel Corporation Device 2020 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 0
Capabilities: [90] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM not supported, Exit Latency L0s <512ns, L1 <16us
ClockPM- Surprise+ LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna- CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
Capabilities: [144 v1] Vendor Specific Information: ID=0004 Rev=1 Len=03c <?>
Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
Capabilities: [250 v1] #19
Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>

00:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 11
Region 0: Memory at 383ffff2c000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [80] MSI-X: Enable- Count=1 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

00:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 10
Region 0: Memory at 383ffff28000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [80] MSI-X: Enable- Count=1 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

00:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin C routed to IRQ 11
Region 0: Memory at 383ffff24000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [80] MSI-X: Enable- Count=1 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

00:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin D routed to IRQ 11
Region 0: Memory at 383ffff20000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [80] MSI-X: Enable- Count=1 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

00:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 11
Region 0: Memory at 383ffff1c000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [80] MSI-X: Enable- Count=1 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

00:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 10
Region 0: Memory at 383ffff18000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [80] MSI-X: Enable- Count=1 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

00:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin C routed to IRQ 11
Region 0: Memory at 383ffff14000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [80] MSI-X: Enable- Count=1 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

00:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin D routed to IRQ 11
Region 0: Memory at 383ffff10000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [80] MSI-X: Enable- Count=1 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

00:05.0 System peripheral: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

00:05.2 System peripheral: Intel Corporation Device 2025 (rev 04)
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

00:05.4 PIC: Intel Corporation Device 2026 (rev 04) (prog-if 20 [IO(X)-APIC])
Subsystem: Intel Corporation Device 2026
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Region 0: Memory at 9d20a000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [44] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

00:08.0 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

00:08.1 Performance counters: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:08.2 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

00:11.0 Unassigned class [ff00]: Intel Corporation Device a1ec (rev 08)
Subsystem: Intel Corporation Device 7270
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Capabilities: [80] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

00:11.5 SATA controller: Intel Corporation Lewisburg SSATA Controller [AHCI mode] (rev 08) (prog-if 01 [AHCI 1.0])

	Subsystem: Intel Corporation Device 7270
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 47
	Region 0: Memory at 9d206000 (32-bit, non-prefetchable) [size=8K]
	Region 1: Memory at 9d209000 (32-bit, non-prefetchable) [size=256]
	Region 2: I/O ports at 3070 [size=8]
	Region 3: I/O ports at 3060 [size=4]
	Region 4: I/O ports at 3020 [size=32]
	Region 5: Memory at 9d180000 (32-bit, non-prefetchable) [size=512K]
	Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fee00318  Data: 0000
	Capabilities: [70] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [a8] SATA HBA v1.0 BAR4 Offset=00000004
	Kernel driver in use: ahci

00:14.0 USB controller: Intel Corporation Lewisburg USB 3.0 xHCI Controller (rev 08) (prog-if 30 [XHCI])
Subsystem: Intel Corporation Device 7270
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 46
Region 0: Memory at 383ffff00000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [70] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
Address: 00000000fee002f8 Data: 0000
Kernel driver in use: xhci_hcd

00:14.2 Signal processing controller: Intel Corporation Device a1b1 (rev 08)
Subsystem: Intel Corporation Device 7270
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin C routed to IRQ 11
Region 0: Memory at 383ffff33000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
Address: 00000000 Data: 0000

00:16.0 Communication controller: Intel Corporation Lewisburg CSME: HECI #1 (rev 08)
Subsystem: Intel Corporation Device 7270
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 11
Region 0: Memory at 383ffff32000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [8c] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000

00:16.1 Communication controller: Intel Corporation Lewisburg CSME: HECI #2 (rev 08)
Subsystem: Intel Corporation Device 7270
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 11
Region 0: Memory at 383ffff31000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [8c] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000

00:16.4 Communication controller: Intel Corporation Lewisburg CSME: HECI #3 (rev 08)
Subsystem: Intel Corporation Device 7270
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 11
Region 0: Memory at 383ffff30000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [8c] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000

00:17.0 SATA controller: Intel Corporation Lewisburg SATA Controller [AHCI mode] (rev 08) (prog-if 01 [AHCI 1.0])

	Subsystem: Intel Corporation Device 7270
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 48
	Region 0: Memory at 9d204000 (32-bit, non-prefetchable) [size=8K]
	Region 1: Memory at 9d208000 (32-bit, non-prefetchable) [size=256]
	Region 2: I/O ports at 3050 [size=8]
	Region 3: I/O ports at 3040 [size=4]
	Region 4: I/O ports at 3000 [size=32]
	Region 5: Memory at 9d100000 (32-bit, non-prefetchable) [size=512K]
	Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fee00338  Data: 0000
	Capabilities: [70] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [a8] SATA HBA v1.0 BAR4 Offset=00000004
	Kernel driver in use: ahci

00:1c.0 PCI bridge: Intel Corporation Lewisburg PCI Express Root Port #1 (rev f8) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00001000-00001fff
Memory behind bridge: 90000000-901fffff
Prefetchable memory behind bridge: 0000380000000000-00003800001fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #1, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
ClockPM- Surprise- LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
Slot #0, PowerLimit 0.000W; Interlock- NoCompl+
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt- HPIrq+ LinkChg+
Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF Via WAKE# ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
Address: fee002b8 Data: 0000
Capabilities: [90] Subsystem: Intel Corporation Device 7270
Capabilities: [a0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Kernel driver in use: pcieport

00:1c.4 PCI bridge: Intel Corporation Lewisburg PCI Express Root Port #5 (rev f8) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=02, subordinate=03, sec-latency=0
I/O behind bridge: 00002000-00002fff
Memory behind bridge: 9c000000-9d0fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #5, Speed 8GT/s, Width x1, ASPM not supported, Exit Latency L0s <1us, L1 <16us
ClockPM- Surprise- LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #0, PowerLimit 10.000W; Interlock- NoCompl+
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF Not Supported ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
Address: fee002d8 Data: 0000
Capabilities: [90] Subsystem: Intel Corporation Device 7270
Capabilities: [a0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Capabilities: [140 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [220 v1] #19
Kernel driver in use: pcieport

00:1f.0 ISA bridge: Intel Corporation Lewisburg LPC Controller (rev 08)
Subsystem: Intel Corporation Device 7270
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0

00:1f.2 Memory controller: Intel Corporation Lewisburg PMC (rev 08)
Subsystem: Intel Corporation Device 7270
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Region 0: Memory at 9d200000 (32-bit, non-prefetchable) [size=16K]

00:1f.4 SMBus: Intel Corporation Lewisburg SMBus (rev 08)
Subsystem: Intel Corporation Device 7270
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 16
Region 0: Memory at 380000200000 (64-bit, non-prefetchable) [size=256]
Region 4: I/O ports at 0780 [size=32]
Kernel driver in use: i801_smbus

00:1f.5 Serial bus controller [0c80]: Intel Corporation Lewisburg SPI Controller (rev 08)
Subsystem: Intel Corporation Device 7270
Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Region 0: Memory at fe010000 (32-bit, non-prefetchable) [size=4K]

02:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=02, secondary=03, subordinate=03, sec-latency=32
I/O behind bridge: 00002000-00002fff
Memory behind bridge: 9c000000-9d0fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [78] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [80] Express (v2) PCI-Express to PCI/PCI-X Bridge, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ BrConfRtry-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <2us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [c0] Subsystem: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge
Capabilities: [100 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01
Status: NegoPending- InProgress-
Capabilities: [800 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-

03:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41) (prog-if 00 [VGA controller])
Subsystem: ASPEED Technology, Inc. ASPEED Graphics Family
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 16
Region 0: Memory at 9c000000 (32-bit, non-prefetchable) [size=16M]
Region 1: Memory at 9d000000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at 2000 [size=128]
Expansion ROM at <unassigned> [disabled]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/2 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Kernel driver in use: ast

17:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1A (rev 04) (prog-if 00 [Normal decode])
Physical Slot: 0
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=17, secondary=18, subordinate=19, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: aad00000-aaefffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Subsystem: Intel Corporation Device 0000
Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
Address: fee00038 Data: 0000
Masking: 00000002 Pending: 00000000
Capabilities: [90] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #1, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <512ns, L1 <16us
ClockPM- Surprise+ LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
Capabilities: [110 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [148 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
Capabilities: [250 v1] #19
Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
Kernel driver in use: pcieport

17:02.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1C (rev 04) (prog-if 00 [Normal decode])
Physical Slot: 1
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=17, secondary=1a, subordinate=1a, sec-latency=0
I/O behind bridge: 00004000-00004fff
Memory behind bridge: 9d800000-9d9fffff
Prefetchable memory behind bridge: 0000384000000000-00003840001fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Subsystem: Intel Corporation Device 0000
Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
Address: fee00058 Data: 0000
Masking: 00000002 Pending: 00000000
Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #3, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <512ns, L1 <16us
ClockPM- Surprise+ LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
Slot #1, PowerLimit 0.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
Control: AttnInd Off, PwrInd Off, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
Capabilities: [110 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [148 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
Capabilities: [250 v1] #19
Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
Kernel driver in use: pcieport

17:05.0 System peripheral: Intel Corporation Device 2034 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

17:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

17:05.4 PIC: Intel Corporation Device 2036 (rev 04) (prog-if 20 [IO(X)-APIC])
Subsystem: Intel Corporation Device 2036
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Region 0: Memory at aaf00000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [44] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

17:08.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:08.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:08.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:08.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:08.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:08.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:08.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:08.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:09.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:09.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:09.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:09.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:09.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:09.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:09.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:09.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0a.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0a.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0a.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0a.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0a.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0a.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0a.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0a.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0b.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0b.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0b.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0b.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0e.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0e.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0e.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0e.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0e.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0e.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0e.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0e.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0f.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0f.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0f.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0f.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0f.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0f.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0f.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0f.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:10.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:10.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:10.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:10.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:10.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:10.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:10.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:10.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:11.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:11.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:11.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:11.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:1d.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:1d.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:1d.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:1d.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:1e.0 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:1e.1 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:1e.2 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:1e.3 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:1e.4 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:1e.5 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:1e.6 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

18:00.0 PCI bridge: Intel Corporation Device 37c0 (rev 08) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Region 0: Memory at aae00000 (64-bit, non-prefetchable) [size=128K]
Bus: primary=18, secondary=19, subordinate=19, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- Expansion ROM at aad00000 [disabled] [size=1M]
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Express (v2) Upstream Port, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 256 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <1us, L1 <32us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; Disabled- CommClk+
ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [80] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [88] Subsystem: Device beef:dead
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [200 v1] #19
Kernel driver in use: pcieport

3a:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1A (rev 04) (prog-if 00 [Normal decode])
Physical Slot: 2
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=3a, secondary=3b, subordinate=3b, sec-latency=0
I/O behind bridge: 00006000-00006fff
Memory behind bridge: b8500000-b86fffff
Prefetchable memory behind bridge: 000038bffa000000-000038bffeffffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Subsystem: Intel Corporation Device 0000
Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
Address: fee00038 Data: 0000
Masking: 00000002 Pending: 00000000
Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #5, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <512ns, L1 <16us
ClockPM- Surprise+ LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
Slot #2, PowerLimit 0.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
Control: AttnInd Off, PwrInd Off, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
Capabilities: [110 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [148 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
Capabilities: [250 v1] #19
Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
Kernel driver in use: pcieport

3a:02.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1C (rev 04) (prog-if 00 [Normal decode])
Physical Slot: 3
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=3a, secondary=3c, subordinate=3c, sec-latency=0
I/O behind bridge: 00007000-00007fff
Memory behind bridge: ab000000-ab1fffff
Prefetchable memory behind bridge: 0000388000000000-00003880001fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Subsystem: Intel Corporation Device 0000
Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
Address: fee00058 Data: 0000
Masking: 00000002 Pending: 00000000
Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #7, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <512ns, L1 <16us
ClockPM- Surprise+ LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
Slot #3, PowerLimit 0.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
Control: AttnInd Off, PwrInd Off, Power+ Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
Capabilities: [110 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [148 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
Capabilities: [250 v1] #19
Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
Kernel driver in use: pcieport

3a:05.0 System peripheral: Intel Corporation Device 2034 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

3a:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

3a:05.4 PIC: Intel Corporation Device 2036 (rev 04) (prog-if 20 [IO(X)-APIC])
Subsystem: Intel Corporation Device 2036
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Region 0: Memory at b8700000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [44] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

3a:08.0 System peripheral: Intel Corporation Device 2066 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:09.0 System peripheral: Intel Corporation Device 2066 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0a.0 System peripheral: Intel Corporation Device 2040 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0a.1 System peripheral: Intel Corporation Device 2041 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0a.2 System peripheral: Intel Corporation Device 2042 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0a.3 System peripheral: Intel Corporation Device 2043 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0a.4 System peripheral: Intel Corporation Device 2044 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0a.5 System peripheral: Intel Corporation Device 2045 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0a.6 System peripheral: Intel Corporation Device 2046 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0a.7 System peripheral: Intel Corporation Device 2047 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0b.0 System peripheral: Intel Corporation Device 2048 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0b.1 System peripheral: Intel Corporation Device 2049 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0b.2 System peripheral: Intel Corporation Device 204a (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0b.3 System peripheral: Intel Corporation Device 204b (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0c.0 System peripheral: Intel Corporation Device 2040 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0c.1 System peripheral: Intel Corporation Device 2041 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0c.2 System peripheral: Intel Corporation Device 2042 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0c.3 System peripheral: Intel Corporation Device 2043 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0c.4 System peripheral: Intel Corporation Device 2044 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0c.5 System peripheral: Intel Corporation Device 2045 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0c.6 System peripheral: Intel Corporation Device 2046 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0c.7 System peripheral: Intel Corporation Device 2047 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0d.0 System peripheral: Intel Corporation Device 2048 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0d.1 System peripheral: Intel Corporation Device 2049 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0d.2 System peripheral: Intel Corporation Device 204a (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0d.3 System peripheral: Intel Corporation Device 204b (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3b:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
Subsystem: Mellanox Technologies Device 0021
Physical Slot: 2-1
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 51
Region 0: Memory at 38bffc000000 (64-bit, prefetchable) [size=32M]
Expansion ROM at b8600000 [disabled] [size=1M]
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 256 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported, Exit Latency L0s unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [48] Vital Product Data
Product Name: CX4421A- ConnectX-4 LX SFP28
Read-only fields:
[PN] Part number: MCX4421A-ACQN
[EC] Engineering changes: A6
[SN] Serial number: MT1719X02671
[V0] Vendor specific: PCIeGen3 x8
[RV] Reserved: checksum good, 0 byte(s) reserved
End
Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [c0] Vendor Specific Information: Len=18 <?>
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot-,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 04, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
IOVSta: Migration-
Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00 VF offset: 2, stride: 1, Device ID: 1016
Supported Page Size: 000007ff, System Page Size: 00000001
Region 0: Memory at 000038bffe800000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [1c0 v1] #19
Capabilities: [230 v1] Access Control Services
ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Kernel driver in use: mlx5_core

3b:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
Subsystem: Mellanox Technologies Device 0021
Physical Slot: 2-1
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 168
Region 0: Memory at 38bffa000000 (64-bit, prefetchable) [size=32M]
Expansion ROM at b8500000 [disabled] [size=1M]
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 256 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported, Exit Latency L0s unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [48] Vital Product Data
Product Name: CX4421A- ConnectX-4 LX SFP28
Read-only fields:
[PN] Part number: MCX4421A-ACQN
[EC] Engineering changes: A6
[SN] Serial number: MT1719X02671
[V0] Vendor specific: PCIeGen3 x8
[RV] Reserved: checksum good, 0 byte(s) reserved
End
Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [c0] Vendor Specific Information: Len=18 <?>
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot-,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 04, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00 VF offset: 9, stride: 1, Device ID: 1016
Supported Page Size: 000007ff, System Page Size: 00000001
Region 0: Memory at 000038bffe000000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [230 v1] Access Control Services
ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Kernel driver in use: mlx5_core

5d:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1A (rev 04) (prog-if 00 [Normal decode])
Physical Slot: 4
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=5d, secondary=5e, subordinate=5e, sec-latency=0
I/O behind bridge: 00008000-00008fff
Memory behind bridge: b8800000-b89fffff
Prefetchable memory behind bridge: 000038c000000000-000038c0001fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Subsystem: Intel Corporation Device 0000
Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
Address: fee00038 Data: 0000
Masking: 00000002 Pending: 00000000
Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #9, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <512ns, L1 <16us
ClockPM- Surprise+ LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
Slot #4, PowerLimit 0.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
Control: AttnInd Off, PwrInd Off, Power+ Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
Capabilities: [110 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [148 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
Capabilities: [250 v1] #19
Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
Kernel driver in use: pcieport

5d:02.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1C (rev 04) (prog-if 00 [Normal decode])
Physical Slot: 6
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=5d, secondary=5f, subordinate=5f, sec-latency=0
I/O behind bridge: 00009000-00009fff
Memory behind bridge: b8a00000-b8bfffff
Prefetchable memory behind bridge: 000038c000200000-000038c0003fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Subsystem: Intel Corporation Device 0000
Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
Address: fee00058 Data: 0000
Masking: 00000002 Pending: 00000000
Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #11, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <512ns, L1 <16us
ClockPM- Surprise+ LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
Slot #6, PowerLimit 0.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
Control: AttnInd Off, PwrInd Off, Power+ Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
Capabilities: [110 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [148 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
Capabilities: [250 v1] #19
Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
Kernel driver in use: pcieport

5d:05.0 System peripheral: Intel Corporation Device 2034 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

5d:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

5d:05.4 PIC: Intel Corporation Device 2036 (rev 04) (prog-if 20 [IO(X)-APIC])
Subsystem: Intel Corporation Device 2036
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Region 0: Memory at c5f00000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [44] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

5d:0e.0 Performance counters: Intel Corporation Device 2058 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

5d:0e.1 System peripheral: Intel Corporation Device 2059 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
Capabilities: [100 v0] Vendor Specific Information: ID=0001 Rev=0 Len=0f0 <?>

5d:0f.0 Performance counters: Intel Corporation Device 2058 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

5d:0f.1 System peripheral: Intel Corporation Device 2059 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
Capabilities: [100 v0] Vendor Specific Information: ID=0001 Rev=0 Len=0f0 <?>

5d:10.0 Performance counters: Intel Corporation Device 2058 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

5d:10.1 System peripheral: Intel Corporation Device 2059 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
Capabilities: [100 v0] Vendor Specific Information: ID=0001 Rev=0 Len=0f0 <?>

5d:12.0 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

5d:12.1 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

5d:12.2 System peripheral: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

5d:12.4 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

5d:12.5 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

5d:15.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

5d:16.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

5d:16.4 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

5d:17.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

80:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 11
Region 0: Memory at 393ffff1c000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [80] MSI-X: Enable- Count=1 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

80:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 10
Region 0: Memory at 393ffff18000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [80] MSI-X: Enable- Count=1 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

80:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin C routed to IRQ 11
Region 0: Memory at 393ffff14000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [80] MSI-X: Enable- Count=1 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

80:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin D routed to IRQ 11
Region 0: Memory at 393ffff10000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [80] MSI-X: Enable- Count=1 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

80:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 11
Region 0: Memory at 393ffff0c000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [80] MSI-X: Enable- Count=1 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

80:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 10
Region 0: Memory at 393ffff08000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [80] MSI-X: Enable- Count=1 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

80:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin C routed to IRQ 11
Region 0: Memory at 393ffff04000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [80] MSI-X: Enable- Count=1 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

80:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin D routed to IRQ 11
Region 0: Memory at 393ffff00000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [80] MSI-X: Enable- Count=1 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

80:05.0 System peripheral: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

80:05.2 System peripheral: Intel Corporation Device 2025 (rev 04)
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

80:05.4 PIC: Intel Corporation Device 2026 (rev 04) (prog-if 20 [IO(X)-APIC])
Subsystem: Intel Corporation Device 2026
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Region 0: Memory at d3700000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [44] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

80:08.0 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

80:08.1 Performance counters: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

80:08.2 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

85:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1A (rev 04) (prog-if 00 [Normal decode])
Physical Slot: 8
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=85, secondary=86, subordinate=86, sec-latency=0
I/O behind bridge: 0000c000-0000cfff
Memory behind bridge: d3800000-d39fffff
Prefetchable memory behind bridge: 0000394000000000-00003940001fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Subsystem: Intel Corporation Device 0000
Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
Address: fee00038 Data: 0000
Masking: 00000002 Pending: 00000000
Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #1, Speed 8GT/s, Width x16, ASPM L1, Exit Latency L0s <512ns, L1 <16us
ClockPM- Surprise+ LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
Slot #8, PowerLimit 0.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
Control: AttnInd Off, PwrInd Off, Power+ Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
Capabilities: [110 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [148 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
Capabilities: [250 v1] #19
Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
Kernel driver in use: pcieport

85:05.0 System peripheral: Intel Corporation Device 2034 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

85:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

85:05.4 PIC: Intel Corporation Device 2036 (rev 04) (prog-if 20 [IO(X)-APIC])
Subsystem: Intel Corporation Device 2036
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Region 0: Memory at e0f00000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [44] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

85:08.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:08.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:08.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:08.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:08.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:08.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:08.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:08.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:09.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:09.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:09.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:09.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:09.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:09.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:09.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:09.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0a.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0a.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0a.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0a.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0a.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0a.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0a.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0a.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0b.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0b.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0b.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0b.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0e.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0e.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0e.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0e.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0e.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0e.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0e.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0e.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0f.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0f.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0f.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0f.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0f.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0f.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0f.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0f.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:10.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:10.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:10.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:10.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:10.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:10.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:10.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:10.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:11.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:11.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:11.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:11.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:1d.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:1d.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:1d.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:1d.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:1e.0 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:1e.1 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:1e.2 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:1e.3 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:1e.4 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:1e.5 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:1e.6 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

ae:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1A (rev 04) (prog-if 00 [Normal decode])
Physical Slot: 12
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=ae, secondary=af, subordinate=af, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: ee600000-ee6fffff
Prefetchable memory behind bridge: 0000398000000000-00003980001fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Subsystem: Intel Corporation Device 0000
Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
Address: fee00038 Data: 0000
Masking: 00000002 Pending: 00000000
Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #5, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s <512ns, L1 <16us
ClockPM- Surprise+ LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
SltCap: AttnBtn- PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
Slot #12, PowerLimit 0.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
Control: AttnInd Off, PwrInd Off, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
Capabilities: [110 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [148 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
Capabilities: [250 v1] #19
Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
Kernel driver in use: pcieport

ae:01.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1B (rev 04) (prog-if 00 [Normal decode])
Physical Slot: 13
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=ae, secondary=b0, subordinate=b0, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: ee500000-ee5fffff
Prefetchable memory behind bridge: 0000398000200000-00003980003fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Subsystem: Intel Corporation Device 0000
Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
Address: fee00058 Data: 0000
Masking: 00000002 Pending: 00000000
Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #6, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s <512ns, L1 <16us
ClockPM- Surprise+ LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
SltCap: AttnBtn- PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
Slot #13, PowerLimit 0.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
Control: AttnInd Off, PwrInd Off, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
Capabilities: [110 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [148 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
Capabilities: [250 v1] #19
Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
Kernel driver in use: pcieport

ae:02.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1C (rev 04) (prog-if 00 [Normal decode])
Physical Slot: 14
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=ae, secondary=b1, subordinate=b1, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: e1000000-e11fffff
Prefetchable memory behind bridge: 0000398000400000-00003980005fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Subsystem: Intel Corporation Device 0000
Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
Address: fee00078 Data: 0000
Masking: 00000002 Pending: 00000000
Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #7, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s <512ns, L1 <16us
ClockPM- Surprise+ LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
Slot #14, PowerLimit 0.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
Control: AttnInd Off, PwrInd Off, Power+ Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
Capabilities: [110 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [148 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
Capabilities: [250 v1] #19
Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
Kernel driver in use: pcieport

ae:03.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1D (rev 04) (prog-if 00 [Normal decode])
Physical Slot: 15
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=ae, secondary=b2, subordinate=b2, sec-latency=0
I/O behind bridge: 0000e000-0000efff
Memory behind bridge: e1200000-e13fffff
Prefetchable memory behind bridge: 0000398000600000-00003980007fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Subsystem: Intel Corporation Device 0000
Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
Address: fee00098 Data: 0000
Masking: 00000002 Pending: 00000000
Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #8, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s <512ns, L1 <16us
ClockPM- Surprise+ LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
Slot #15, PowerLimit 0.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
Control: AttnInd Off, PwrInd Off, Power+ Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
Capabilities: [110 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [148 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
Capabilities: [250 v1] #19
Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
Kernel driver in use: pcieport

ae:05.0 System peripheral: Intel Corporation Device 2034 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

ae:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

ae:05.4 PIC: Intel Corporation Device 2036 (rev 04) (prog-if 20 [IO(X)-APIC])
Subsystem: Intel Corporation Device 2036
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Region 0: Memory at ee700000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [44] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

ae:08.0 System peripheral: Intel Corporation Device 2066 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:09.0 System peripheral: Intel Corporation Device 2066 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0a.0 System peripheral: Intel Corporation Device 2040 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0a.1 System peripheral: Intel Corporation Device 2041 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0a.2 System peripheral: Intel Corporation Device 2042 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0a.3 System peripheral: Intel Corporation Device 2043 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0a.4 System peripheral: Intel Corporation Device 2044 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0a.5 System peripheral: Intel Corporation Device 2045 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0a.6 System peripheral: Intel Corporation Device 2046 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0a.7 System peripheral: Intel Corporation Device 2047 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0b.0 System peripheral: Intel Corporation Device 2048 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0b.1 System peripheral: Intel Corporation Device 2049 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0b.2 System peripheral: Intel Corporation Device 204a (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0b.3 System peripheral: Intel Corporation Device 204b (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0c.0 System peripheral: Intel Corporation Device 2040 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0c.1 System peripheral: Intel Corporation Device 2041 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0c.2 System peripheral: Intel Corporation Device 2042 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0c.3 System peripheral: Intel Corporation Device 2043 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0c.4 System peripheral: Intel Corporation Device 2044 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0c.5 System peripheral: Intel Corporation Device 2045 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0c.6 System peripheral: Intel Corporation Device 2046 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0c.7 System peripheral: Intel Corporation Device 2047 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0d.0 System peripheral: Intel Corporation Device 2048 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0d.1 System peripheral: Intel Corporation Device 2049 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0d.2 System peripheral: Intel Corporation Device 204a (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0d.3 System peripheral: Intel Corporation Device 204b (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

af:00.0 Non-Volatile memory controller: Device 1c5f:0550 (rev 02) (prog-if 02 [NVM Express])
Subsystem: Device 1c5f:0552
Physical Slot: 12-1
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 49
Region 0: Memory at ee610000 (64-bit, non-prefetchable) [size=16K]
Expansion ROM at ee600000 [disabled] [size=64K]
Capabilities: [c0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [c8] MSI: Enable- Count=1/32 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000001
Capabilities: [e0] MSI-X: Enable+ Count=129 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-
MaxPayload 256 bytes, MaxReadReq 256 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM not supported, Exit Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [40] Vendor Specific Information: Len=24 <?>
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [190 v1] #19
Capabilities: [150 v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?> Kernel driver in use: nvme

b0:00.0 Non-Volatile memory controller: Device 1c5f:0550 (rev 02) (prog-if 02 [NVM Express])
Subsystem: Device 1c5f:0552
Physical Slot: 13-1
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 50
Region 0: Memory at ee510000 (64-bit, non-prefetchable) [size=16K]
Expansion ROM at ee500000 [disabled] [size=64K]
Capabilities: [c0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [c8] MSI: Enable- Count=1/32 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000001
Capabilities: [e0] MSI-X: Enable+ Count=129 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-
MaxPayload 256 bytes, MaxReadReq 256 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM not supported, Exit Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [40] Vendor Specific Information: Len=24 <?>
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [190 v1] #19
Capabilities: [150 v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?> Kernel driver in use: nvme

d7:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1A (rev 04) (prog-if 00 [Normal decode])
Physical Slot: 16
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=d7, secondary=d8, subordinate=d8, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: ee800000-ee9fffff
Prefetchable memory behind bridge: 000039c000000000-000039c0001fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Subsystem: Intel Corporation Device 0000
Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
Address: fee00038 Data: 0000
Masking: 00000002 Pending: 00000000
Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #9, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <512ns, L1 <16us
ClockPM- Surprise+ LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
Slot #16, PowerLimit 0.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
Control: AttnInd Off, PwrInd Off, Power+ Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
Capabilities: [110 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [148 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
Capabilities: [250 v1] #19
Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
Kernel driver in use: pcieport

d7:02.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1C (rev 04) (prog-if 00 [Normal decode])
Physical Slot: 18
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=d7, secondary=d9, subordinate=d9, sec-latency=0
I/O behind bridge: 0000f000-0000ffff
Memory behind bridge: eea00000-eebfffff
Prefetchable memory behind bridge: 000039c000200000-000039c0003fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Subsystem: Intel Corporation Device 0000
Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
Address: fee00058 Data: 0000
Masking: 00000002 Pending: 00000000
Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #11, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <512ns, L1 <16us
ClockPM- Surprise+ LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
Slot #18, PowerLimit 0.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
Control: AttnInd Off, PwrInd Off, Power+ Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
Capabilities: [110 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [148 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
Capabilities: [250 v1] #19
Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
Kernel driver in use: pcieport

d7:05.0 System peripheral: Intel Corporation Device 2034 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

d7:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

d7:05.4 PIC: Intel Corporation Device 2036 (rev 04) (prog-if 20 [IO(X)-APIC])
Subsystem: Intel Corporation Device 2036
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Region 0: Memory at fbf00000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [44] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

d7:0e.0 Performance counters: Intel Corporation Device 2058 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

d7:0e.1 System peripheral: Intel Corporation Device 2059 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
Capabilities: [100 v0] Vendor Specific Information: ID=0001 Rev=0 Len=0f0 <?>

d7:0f.0 Performance counters: Intel Corporation Device 2058 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

d7:0f.1 System peripheral: Intel Corporation Device 2059 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
Capabilities: [100 v0] Vendor Specific Information: ID=0001 Rev=0 Len=0f0 <?>

d7:10.0 Performance counters: Intel Corporation Device 2058 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

d7:10.1 System peripheral: Intel Corporation Device 2059 (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
Capabilities: [100 v0] Vendor Specific Information: ID=0001 Rev=0 Len=0f0 <?>

d7:12.0 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

d7:12.1 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

d7:12.2 System peripheral: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

d7:12.4 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

d7:12.5 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

d7:15.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

d7:16.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

d7:16.4 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

d7:17.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

#18 lore raw
Quoted 2 lines · Ron Yuan

Just got the log, see attachment. Kernel is under "perf" mode. SSD and Ethernet controller are both set to 256B MRRS.

Here's what I see from lspci:

  3a:00.0 Root Port to 3b     MPS_cap=256 MPS=256 MRRS=128
  3b:00.0 NIC Endpoint        MPS_cap=512 MPS=256 MRRS=256
  3b:00.1 NIC Endpoint        MPS_cap=512 MPS=256 MRRS=256

Here, the NICs support up to MPS=512 but the Root Port only supports MPS=256. We must set MPS=256 for the NICs. Otherwise, the NICs could do 512-byte DMA writes to system memory, and the Root Port would treat those TLPs as malformed.

There is no need to limit MRRS because there are no other devices under this Root Port. We can't tell from lspci what the maximum MRRS is, but if the NICs support MRRS=4096, we could use that.

  ae:00.0 Root Port to af     MPS_cap=256 MPS=256 MRRS=128
  ae:01.0 Root Port to b0     MPS_cap=256 MPS=256 MRRS=128
  af:00.0 SSD Endpoint        MPS_cap=256 MPS=256 MRRS=256
  b0:00.0 SSD Endpoint        MPS_cap=256 MPS=256 MRRS=256

Everything here supports MPS=256, so that's what we should use.

As with the NICs, there's no need to limit MRRS here. We don't know what MRRS the endpoints support, but PERFORMANCE mode is being unnecessarily conservative when it limits MRRS to the MPS (256 in this case).

It's not as simple as just removing the "set MRRS=MPS" part because we do rely on that in some topologies. There are interesting topologies where we *don't* need it, like both of the ones above, and we need to make Linux smart enough to recognize them.

#19 lore raw
Quoted 31 lines · Bjorn Helgaas
Quoted 2 lines · Ron Yuan

Just got the log, see attachment. Kernel is under "perf" mode. SSD and Ethernet controller are both set to 256B MRRS.

Here's what I see from lspci:

  3a:00.0 Root Port to 3b     MPS_cap=256 MPS=256 MRRS=128
  3b:00.0 NIC Endpoint        MPS_cap=512 MPS=256 MRRS=256
  3b:00.1 NIC Endpoint        MPS_cap=512 MPS=256 MRRS=256

Here, the NICs support up to MPS=512 but the Root Port only supports MPS=256. We must set MPS=256 for the NICs. Otherwise, the NICs could do 512-byte DMA writes to system memory, and the Root Port would treat those TLPs as malformed.

There is no need to limit MRRS because there are no other devices under this Root Port. We can't tell from lspci what the maximum MRRS is, but if the NICs support MRRS=4096, we could use that.

  ae:00.0 Root Port to af     MPS_cap=256 MPS=256 MRRS=128
  ae:01.0 Root Port to b0     MPS_cap=256 MPS=256 MRRS=128
  af:00.0 SSD Endpoint        MPS_cap=256 MPS=256 MRRS=256
  b0:00.0 SSD Endpoint        MPS_cap=256 MPS=256 MRRS=256

Everything here supports MPS=256, so that's what we should use.

As with the NICs, there's no need to limit MRRS here. We don't know what MRRS the endpoints support, but PERFORMANCE mode is being unnecessarily conservative when it limits MRRS to the MPS (256 in this case).

Quoted 4 lines

It's not as simple as just removing the "set MRRS=MPS" part because we do rely on that in some topologies. There are interesting topologies where we *don't* need it, like both of the ones above, and we need to make Linux smart enough to recognize them.

The mailing list dropped Ron's email with the lspci log because it was a multipart base64-encoded message. The lists accept plain text only (see http://vger.kernel.org/majordomo-info.html).

Here's Ron's message:

Quoted 2 lines

Just got the log, see attachment. Kernel is under "perf" mode. SSD and Ethernet controller are both set to 256B MRRS.

And here are the relevant parts of the lspci.log that he attached (I didn't include it all because it's over 100K and the mailing list wouldn't accept that either):

3a:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1A (rev 04) (prog-if 00 [Normal decode])
Physical Slot: 2
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=3a, secondary=3b, subordinate=3b, sec-latency=0
I/O behind bridge: 00006000-00006fff
Memory behind bridge: b8500000-b86fffff
Prefetchable memory behind bridge: 000038bffa000000-000038bffeffffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Subsystem: Intel Corporation Device 0000
Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
Address: fee00038 Data: 0000
Masking: 00000002 Pending: 00000000
Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #5, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <512ns, L1 <16us
ClockPM- Surprise+ LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
Slot #2, PowerLimit 0.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
Control: AttnInd Off, PwrInd Off, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
Capabilities: [110 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [148 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
Capabilities: [250 v1] #19
Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
Kernel driver in use: pcieport

3b:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
Subsystem: Mellanox Technologies Device 0021
Physical Slot: 2-1
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 51
Region 0: Memory at 38bffc000000 (64-bit, prefetchable) [size=32M]
Expansion ROM at b8600000 [disabled] [size=1M]
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 256 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported, Exit Latency L0s unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [48] Vital Product Data
Product Name: CX4421A- ConnectX-4 LX SFP28
Read-only fields:
[PN] Part number: MCX4421A-ACQN
[EC] Engineering changes: A6
[SN] Serial number: MT1719X02671
[V0] Vendor specific: PCIeGen3 x8
[RV] Reserved: checksum good, 0 byte(s) reserved
End
Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [c0] Vendor Specific Information: Len=18 <?>
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot-,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 04, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
IOVSta: Migration-
Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00 VF offset: 2, stride: 1, Device ID: 1016
Supported Page Size: 000007ff, System Page Size: 00000001
Region 0: Memory at 000038bffe800000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [1c0 v1] #19
Capabilities: [230 v1] Access Control Services
ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Kernel driver in use: mlx5_core

3b:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
Subsystem: Mellanox Technologies Device 0021
Physical Slot: 2-1
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 168
Region 0: Memory at 38bffa000000 (64-bit, prefetchable) [size=32M]
Expansion ROM at b8500000 [disabled] [size=1M]
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 256 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported, Exit Latency L0s unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [48] Vital Product Data
Product Name: CX4421A- ConnectX-4 LX SFP28
Read-only fields:
[PN] Part number: MCX4421A-ACQN
[EC] Engineering changes: A6
[SN] Serial number: MT1719X02671
[V0] Vendor specific: PCIeGen3 x8
[RV] Reserved: checksum good, 0 byte(s) reserved
End
Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [c0] Vendor Specific Information: Len=18 <?>
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot-,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 04, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00 VF offset: 9, stride: 1, Device ID: 1016
Supported Page Size: 000007ff, System Page Size: 00000001
Region 0: Memory at 000038bffe000000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [230 v1] Access Control Services
ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Kernel driver in use: mlx5_core

ae:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1A (rev 04) (prog-if 00 [Normal decode])
Physical Slot: 12
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=ae, secondary=af, subordinate=af, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: ee600000-ee6fffff
Prefetchable memory behind bridge: 0000398000000000-00003980001fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Subsystem: Intel Corporation Device 0000
Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
Address: fee00038 Data: 0000
Masking: 00000002 Pending: 00000000
Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #5, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s <512ns, L1 <16us
ClockPM- Surprise+ LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
SltCap: AttnBtn- PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
Slot #12, PowerLimit 0.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
Control: AttnInd Off, PwrInd Off, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
Capabilities: [110 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [148 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
Capabilities: [250 v1] #19
Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
Kernel driver in use: pcieport

ae:01.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1B (rev 04) (prog-if 00 [Normal decode])
Physical Slot: 13
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=ae, secondary=b0, subordinate=b0, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: ee500000-ee5fffff
Prefetchable memory behind bridge: 0000398000200000-00003980003fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Subsystem: Intel Corporation Device 0000
Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
Address: fee00058 Data: 0000
Masking: 00000002 Pending: 00000000
Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #6, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s <512ns, L1 <16us
ClockPM- Surprise+ LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
SltCap: AttnBtn- PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
Slot #13, PowerLimit 0.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
Control: AttnInd Off, PwrInd Off, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
Capabilities: [110 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [148 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
Capabilities: [250 v1] #19
Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
Kernel driver in use: pcieport

af:00.0 Non-Volatile memory controller: Device 1c5f:0550 (rev 02) (prog-if 02 [NVM Express])
Subsystem: Device 1c5f:0552
Physical Slot: 12-1
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 49
Region 0: Memory at ee610000 (64-bit, non-prefetchable) [size=16K]
Expansion ROM at ee600000 [disabled] [size=64K]
Capabilities: [c0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [c8] MSI: Enable- Count=1/32 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000001
Capabilities: [e0] MSI-X: Enable+ Count=129 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-
MaxPayload 256 bytes, MaxReadReq 256 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM not supported, Exit Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [40] Vendor Specific Information: Len=24 <?>
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [190 v1] #19
Capabilities: [150 v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?> Kernel driver in use: nvme

b0:00.0 Non-Volatile memory controller: Device 1c5f:0550 (rev 02) (prog-if 02 [NVM Express])
Subsystem: Device 1c5f:0552
Physical Slot: 13-1
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 50
Region 0: Memory at ee510000 (64-bit, non-prefetchable) [size=16K]
Expansion ROM at ee500000 [disabled] [size=64K]
Capabilities: [c0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [c8] MSI: Enable- Count=1/32 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000001
Capabilities: [e0] MSI-X: Enable+ Count=129 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-
MaxPayload 256 bytes, MaxReadReq 256 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM not supported, Exit Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [40] Vendor Specific Information: Len=24 <?>
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [190 v1] #19
Capabilities: [150 v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?> Kernel driver in use: nvme

#20 lore raw
In reply to Ron Yuan

On Tue, Jan 23, 2018 at 01:25:56PM +0000, Ron Yuan wrote:

I'm reproducing Sinan's picture here so we can see what you're talking about:

Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
               root (MPS=256)
                 |
         ------------------
        /                  \
   bridge0 (MPS=256)      bridge1 (MPS=128)
      /                       \
    EP0 (MPS=256)            EP1 (MPS=128)
Quoted 6 lines
Quoted 2 lines

PERFORMANCE mode reduces MRRS not because of a starvation issue, but because reducing EP1's MRRS allows EP0 to use a larger MPS.

Looks like this case is talking about EP1 requests data directly from EP0, using MRRS to control the return data payload, while still keeping the traffic from EP0 to RC in 256B.

No, this is not talking about EP1 requesting data from EP0. That would be peer-to-peer DMA, and PERFORMANCE mode explicitly assumes there is no peer-to-peer DMA. It reduces MRRS to allow EP0 to use a larger MPS.

We must guarantee that no device receives a TLP larger than its MPS setting. The simple and obvious configuration is to set MPS=128 for everything in Sinan's picture. That works correctly but limits EP0's performance.

What PERFORMANCE mode does is set MPS as shown in the picture and set EP1's MRRS=128. We're assuming no peer-to-peer DMA, but of course EP1 may still need to do DMA reads from system memory, and setting its MRRS=128 means those reads will be of 128 bytes or less.

If we set EP1's MRRS=256, it could do a 256-byte DMA read from system memory, the root port could send a 256-byte completion, and bridge1 would treat that as a malformed TLP because its MPS=128.

#21 lore raw

Hi Bjorne,

Ceiling the MRRS to the MPS value in order to guarantee the interoperabilit= y in pcie_bus_perf mode does not make sense. A device can make a memrd requ= est according to the MRRS setting (which can be higher than its MPS), but t= he completer has to respect the MPS and send completions accordingly. As an=
example, system can configure MPS=3D128B and MRRS=3D4K, where an endpoint =
can a make 4K MemRd request, but the completer has to send completions as 1= 28B TLPs, by respecting the MPS setting. MRRS does not force a device to us= e higher MPS value than it is configured to.

Another factor that need to be considered for storage devices is that suppo= rt of T10 Protection Information (DIF). For every 512B or 4KB, a 8B PI is c= omputed and inserted or verified, which require the 512B of data to arrive = in sequence. If the MRRS is < 512B, this might pose out of order completion= s to the storage device, if the EP has to submit multiple outstanding read = requests in order to achieve higher performance. This would be a challenge = for the storage endpoints that process the T10 PI inline with the transfer,=
now they have to store and process the 512B sectors once they receive all =
the TLPs for that sector.

So, it is better to decouple the MRRS and MPS in pcie_bus_perf mode. Like s= tated earlier in the thread, provide an option to configure MRRS separately=
in pcie_bus_perf mode.

Regards,
Radj.

-----Original Message-----
From: Bjorn Helgaas [mailto:[email protected]]=20
Sent: Tuesday, January 23, 2018 6:39 AM
To: Ron Yuan <[email protected]>
Cc: Sinan Kaya <[email protected]>; Bjorn Helgaas <[email protected]>;=
Bo Chen <[email protected]>; William Huang <[email protected]>=
; Fengming Wu <[email protected]>; Jason Jiang <jason.jiang@microsem= i.com>; Radjendirane Codandaramane <[email protected]>; Ra= myakanth Edupuganti <[email protected]>; William Cheng <w= [email protected]>; Kim Helper (khelper) <[email protected]>; Lin= ux PCI <[email protected]>
Subject: Re: One Question About PCIe BUS Config Type with pcie_bus_safe or = pcie_bus_perf On NVMe Device

EXTERNAL EMAIL

On Tue, Jan 23, 2018 at 01:25:56PM +0000, Ron Yuan wrote:

I'm reproducing Sinan's picture here so we can see what you're talking about:

Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
               root (MPS=3D256)
                 |
         ------------------
        /                  \
   bridge0 (MPS=3D256)      bridge1 (MPS=3D128)
      /                       \
    EP0 (MPS=3D256)            EP1 (MPS=3D128)
Quoted 6 lines
Quoted 2 lines

PERFORMANCE mode reduces MRRS not because of a starvation issue, but=20 because reducing EP1's MRRS allows EP0 to use a larger MPS.

Looks like this case is talking about EP1 requests data directly from=20 EP0, using MRRS to control the return data payload, while still=20 keeping the traffic from EP0 to RC in 256B.

No, this is not talking about EP1 requesting data from EP0. That would be = peer-to-peer DMA, and PERFORMANCE mode explicitly assumes there is no peer-= to-peer DMA. It reduces MRRS to allow EP0 to use a larger MPS.

We must guarantee that no device receives a TLP larger than its MPS setting= . The simple and obvious configuration is to set MPS=3D128 for everything = in Sinan's picture. That works correctly but limits EP0's performance.

What PERFORMANCE mode does is set MPS as shown in the picture and set EP1's=
MRRS=3D128. We're assuming no peer-to-peer DMA, but of course EP1 may sti=
ll need to do DMA reads from system memory, and setting its MRRS=3D128 means those reads will be of 128 bytes or less.

If we set EP1's MRRS=3D256, it could do a 256-byte DMA read from system mem= ory, the root port could send a 256-byte completion, and bridge1 would trea= t that as a malformed TLP because its MPS=3D128.

#22 lore raw

Radjendirane,

I've struggled with a response to your latest posting all night. I don't want to come off as offensive or terse as happens all too often on linux mail lists, all that does is shut things down unnecessarly without relaying any information that is being sought.

Bjorn is just one person trying to keep up with this entire list. On this particular topic he has taken considerable time explaining, and answering peoples specific questions, as to how Linux currently handles PCIe MPS and MRRS settings. As such, it was quite surprising to see your latest posting as the majority of the content had already
been covered - the exact same questions/points - in great detail.

We should respect Bjorn's time as much as possible and "do our homework"; in this specific case, take the time to read the entire thread, carefully, as that would have circumvented this awkward and frustrating situation. If there are still questions on covered topics, then ask the questions at that point (i.e. use proper response techniques to preserve context for all; don't just repeat a question much later in the thread that has already been discussed).

Again, I'm not trying to shut you down.

As Sinan pointed out in the thread's inception:
"Please use mailing list email syntax moving forward. (inline and 75 characters per line)".

Quoted 1 line · Radjendirane Codandaramane

Hi Bjorne,

Bjorn

Quoted 2 lines

Ceiling the MRRS to the MPS value in order to guarantee the interoperabil=

ity in pcie_bus_perf mode does not make sense. A device can make a memrd re= quest according to the MRRS setting (which can be higher than its MPS), but=
the completer has to respect the MPS and send completions accordingly. As =
an example, system can configure MPS=3D128B and MRRS=3D4K, where an endpoin= t can a make 4K MemRd request, but the completer has to send completions as=
128B TLPs, by respecting the MPS setting. MRRS does not force a device to =
use higher MPS value than it is configured to.

This was covered by the very first topic in Bjorn's first reply within the thread...

Quoted 2 lines

Another factor that need to be considered for storage devices is that sup=

port of T10 Protection Information (DIF). For every 512B or 4KB, a 8B PI is=
computed and inserted or verified, which require the 512B of data to arriv=
e in sequence. If the MRRS is < 512B, this might pose out of order completi= ons to the storage device, if the EP has to submit multiple outstanding rea= d requests in order to achieve higher performance. This would be a challeng= e for the storage endpoints that process the T10 PI inline with the transfe= r, now they have to store and process the 512B sectors once they receive al= l the TLPs for that sector.

The "T10" aspects are new and I've not heard about them before. On the surface they seem to be storage device specific. If that is indeed the case then there seems to be some mixing of two, distinctly different, things. PCIe TLPs, MPS, MRRS, ..., are all PCIe defined items that are covered by its specification. Expecting that T10 specifics can be intermixed within PCIe's protocol doesn't make any sense and sounds much more like something that will have to be taken care of at the controller's
level. Perhaps I'm way off base here, we'll have to hear more about this
to come to some understanding.

Quoted 2 lines

So, it is better to decouple the MRRS and MPS in pcie_bus_perf mode. Like=

stated earlier in the thread, provide an option to configure MRRS separate=
ly in pcie_bus_perf mode.

This also has been brought up twice already, and covered in the prior responses...

Quoted 3 lines

Regards,
Radj.

#23 lore raw

Qmpvcm4sIA0KDQpJIGdvIGJhY2sgcmVhZGluZyB5b3VyIGVhcmxpZXIgcmVwbHkgY2FyZWZ1bGx5 IGFuZCBmaW5hbGx5IHVuZGVyc3RhbmQgdGhhdCB5b3VyIGV4YW1wbGUgdGhlIFJQIG1heSByZXBs eSAyNTZCIFRMUCBiZWNhdXNlIG9mIGl0cyBvd24gbGFyZ2VyIE1QUyBzZXR0aW5nLiBUaGlzIGlz IGVhc2lseSBvdmVybG9va2VkIHNpbmNlIHVzdWFsbHkgTVBTIGlzIHN5bmMgb24gdGhlIHdob2xl IHBhdGggYXQgdGhlIGJlZ2lubmluZywgSSBhbG1vc3QgdGFrZSBpdCBmb3IgZ3JhbnRlZC4gVGhh bmsgeW91ISAgDQoNCkhlcmUgaXMgbXkgdW5kZXJzdGFuZGluZzogDQpJZiBNUFMgaXMgc2FtZSBv biBhIHBhdGgsIE1SUlMgaXMgbm90IG5lY2Vzc2FyeSB0byBiZSBwYXJ0IG9mIE1QUyBjb250cm9s LiAtPiB3aGljaCBoZWxwIG1lIGluIE5WTWUgY2FzZSwgTVBTPTI1NkIgYXJlIHN1cHBvcnRlZCBm b3IgYm90aCBSUCBhbmQgRVAuDQpPbmx5IGlmIE1QUyBpcyBub3QgdGhlIHNhbWUgb24gYSBwYXRo IChpbiB0aGUgZXhhbXBsZSBSUCBhbmQgRVAxKSwgTVJSUyBzaG91bGQgYmUgc2V0IHRvIHNhbWUg YXMgTVBTIHRvIGF2b2lkIHJlY2VpdmluZyBsYXJnZXIgY29tcGxldGlvbiBUTFAgdGhhbiBpdHMg TVBTLiBQZXJoYXBzIHRoZSBjb2RlIGNhbiBjaGVjayBhbGwgdXBzdHJlYW0gZGV2aWNlcycgTVBT IGFuZCBkZXRlcm1pbmUgaWYgaXQgbmVlZHMgdG8gYXBwbHkgYSBNUlJTIGxpbWl0YXRpb24uIA0K DQpCdHcsIEkgYWxzbyBmb3VuZCBmb2xsb3dpbmcgc3RhdGVtZW50IGZyb20gUENJRSBzcGVjLCB3 aGljaCBtaWdodCBiZSB0aGUgb3JpZ2luYWwgcmVhc29uIG9mIHRoZSBjb2RlIGF0IHRoZSBiZWdp bm5pbmcuIA0KUGFnZSA1MDAsIFNlY3Rpb24gNi4zLjQuMSBSdWxlcyBmb3IgU29mdHdhcmUgQ29u ZmlndXJhdGlvbg0KIlNvZnR3YXJlIG11c3Qgc2V0IE1heF9SZWFkX1JlcXVlc3RfU2l6ZSBvZiBh biBpc29jaHJvbm91cy1jb25maWd1cmVkIGRldmljZSB3aXRoIGEgdmFsdWUgdGhhdCBkb2VzIG5v dCBleGNlZWQgdGhlIE1heF9QYXlsb2FkX1NpemUgc2V0IGZvciB0aGUgZGV2aWNlLiINCg0KDQpS b24NCg0KLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCkZyb206IE15cm9uIFN0b3dlIFttYWls dG86bXlyb24uc3Rvd2VAZ21haWwuY29tXSANClNlbnQ6IDIwMTjlubQx5pyIMjXml6UgMDozMA0K VG86IFJhZGplbmRpcmFuZSBDb2RhbmRhcmFtYW5lIDxyYWRqZW5kaXJhbmUuY29kYW5kYUBtaWNy b3NlbWkuY29tPg0KQ2M6IEJqb3JuIEhlbGdhYXMgPGhlbGdhYXNAa2VybmVsLm9yZz47IFJvbiBZ dWFuIDxyb24ueXVhbkBtZW1ibGF6ZS5jb20+OyBTaW5hbiBLYXlhIDxva2F5YUBjb2RlYXVyb3Jh Lm9yZz47IEJqb3JuIEhlbGdhYXMgPGJoZWxnYWFzQGdvb2dsZS5jb20+OyBCbyBDaGVuIDxiby5j aGVuQG1lbWJsYXplLmNvbT47IFdpbGxpYW0gSHVhbmcgPHdpbGxpYW0uaHVhbmdAbWVtYmxhemUu Y29tPjsgRmVuZ21pbmcgV3UgPGZlbmdtaW5nLnd1QG1lbWJsYXplLmNvbT47IEphc29uIEppYW5n IDxqYXNvbi5qaWFuZ0BtaWNyb3NlbWkuY29tPjsgUmFteWFrYW50aCBFZHVwdWdhbnRpIDxSYW15 YWthbnRoLkVkdXB1Z2FudGlAbWljcm9zZW1pLmNvbT47IFdpbGxpYW0gQ2hlbmcgPHdpbGxpYW0u Y2hlbmdAbWljcm9zZW1pLmNvbT47IEtpbSBIZWxwZXIgKGtoZWxwZXIpIDxraGVscGVyQG1pY3Jv bi5jb20+OyBMaW51eCBQQ0kgPGxpbnV4LXBjaUB2Z2VyLmtlcm5lbC5vcmc+DQpTdWJqZWN0OiBS ZTogT25lIFF1ZXN0aW9uIEFib3V0IFBDSWUgQlVTIENvbmZpZyBUeXBlIHdpdGggcGNpZV9idXNf c2FmZSBvciBwY2llX2J1c19wZXJmIE9uIE5WTWUgRGV2aWNlDQoNClJhZGplbmRpcmFuZSwNCg0K SSd2ZSBzdHJ1Z2dsZWQgd2l0aCBhIHJlc3BvbnNlIHRvIHlvdXIgbGF0ZXN0IHBvc3RpbmcgYWxs IG5pZ2h0LiAgSSBkb24ndCB3YW50IHRvIGNvbWUgb2ZmIGFzIG9mZmVuc2l2ZSBvciB0ZXJzZSBh cyBoYXBwZW5zIGFsbCB0b28gb2Z0ZW4gb24gbGludXggbWFpbCBsaXN0cywgYWxsIHRoYXQgZG9l cyBpcyBzaHV0IHRoaW5ncyBkb3duIHVubmVjZXNzYXJseSB3aXRob3V0IHJlbGF5aW5nIGFueSBp bmZvcm1hdGlvbiB0aGF0IGlzIGJlaW5nIHNvdWdodC4NCg0KQmpvcm4gaXMganVzdCBvbmUgcGVy c29uIHRyeWluZyB0byBrZWVwIHVwIHdpdGggdGhpcyBlbnRpcmUgbGlzdC4gIE9uIHRoaXMgcGFy dGljdWxhciB0b3BpYyBoZSBoYXMgdGFrZW4gY29uc2lkZXJhYmxlIHRpbWUgZXhwbGFpbmluZywg YW5kIGFuc3dlcmluZyBwZW9wbGVzIHNwZWNpZmljIHF1ZXN0aW9ucywgYXMgdG8gaG93IExpbnV4 IGN1cnJlbnRseSBoYW5kbGVzIFBDSWUgTVBTIGFuZCBNUlJTIHNldHRpbmdzLiAgQXMgc3VjaCwg aXQgd2FzIHF1aXRlIHN1cnByaXNpbmcgdG8gc2VlIHlvdXIgbGF0ZXN0IHBvc3RpbmcgYXMgdGhl IG1ham9yaXR5IG9mIHRoZSBjb250ZW50IGhhZCBhbHJlYWR5IGJlZW4gY292ZXJlZCAtIHRoZSBl eGFjdCBzYW1lIHF1ZXN0aW9ucy9wb2ludHMgLSBpbiBncmVhdCBkZXRhaWwuDQoNCldlIHNob3Vs ZCByZXNwZWN0IEJqb3JuJ3MgdGltZSBhcyBtdWNoIGFzIHBvc3NpYmxlIGFuZCAiZG8gb3VyIGhv bWV3b3JrIjsgaW4gdGhpcyBzcGVjaWZpYyBjYXNlLCB0YWtlIHRoZSB0aW1lIHRvIHJlYWQgdGhl IGVudGlyZSB0aHJlYWQsIGNhcmVmdWxseSwgYXMgdGhhdCB3b3VsZCBoYXZlIGNpcmN1bXZlbnRl ZCB0aGlzIGF3a3dhcmQgYW5kIGZydXN0cmF0aW5nIHNpdHVhdGlvbi4gIElmIHRoZXJlIGFyZSBz dGlsbCBxdWVzdGlvbnMgb24gY292ZXJlZCB0b3BpY3MsIHRoZW4gYXNrIHRoZSBxdWVzdGlvbnMg YXQgdGhhdCBwb2ludCAoaS5lLiB1c2UgcHJvcGVyIHJlc3BvbnNlIHRlY2huaXF1ZXMgdG8gcHJl c2VydmUgY29udGV4dCBmb3IgYWxsOyBkb24ndCBqdXN0IHJlcGVhdCBhIHF1ZXN0aW9uIG11Y2gg bGF0ZXIgaW4gdGhlIHRocmVhZCB0aGF0IGhhcyBhbHJlYWR5IGJlZW4gZGlzY3Vzc2VkKS4NCg0K QWdhaW4sIEknbSBub3QgdHJ5aW5nIHRvIHNodXQgeW91IGRvd24uDQoNCg0KQXMgU2luYW4gcG9p bnRlZCBvdXQgaW4gdGhlIHRocmVhZCdzIGluY2VwdGlvbjoNCiAgIlBsZWFzZSB1c2UgbWFpbGlu ZyBsaXN0IGVtYWlsIHN5bnRheCBtb3ZpbmcgZm9yd2FyZC4gKGlubGluZSBhbmQgNzUNCiAgY2hh cmFjdGVycyBwZXIgbGluZSkiLg0KDQpPbiBUdWUsIEphbiAyMywgMjAxOCBhdCA0OjUwIFBNLCBS YWRqZW5kaXJhbmUgQ29kYW5kYXJhbWFuZSA8cmFkamVuZGlyYW5lLmNvZGFuZGFAbWljcm9zZW1p LmNvbT4gd3JvdGU6DQo+IEhpIEJqb3JuZSwNCg0KQmpvcm4NCg0KPg0KPiBDZWlsaW5nIHRoZSBN UlJTIHRvIHRoZSBNUFMgdmFsdWUgaW4gb3JkZXIgdG8gZ3VhcmFudGVlIHRoZSBpbnRlcm9wZXJh YmlsaXR5IGluIHBjaWVfYnVzX3BlcmYgbW9kZSBkb2VzIG5vdCBtYWtlIHNlbnNlLiBBIGRldmlj ZSBjYW4gbWFrZSBhIG1lbXJkIHJlcXVlc3QgYWNjb3JkaW5nIHRvIHRoZSBNUlJTIHNldHRpbmcg KHdoaWNoIGNhbiBiZSBoaWdoZXIgdGhhbiBpdHMgTVBTKSwgYnV0IHRoZSBjb21wbGV0ZXIgaGFz IHRvIHJlc3BlY3QgdGhlIE1QUyBhbmQgc2VuZCBjb21wbGV0aW9ucyBhY2NvcmRpbmdseS4gQXMg YW4gZXhhbXBsZSwgc3lzdGVtIGNhbiBjb25maWd1cmUgTVBTPTEyOEIgYW5kIE1SUlM9NEssIHdo ZXJlIGFuIGVuZHBvaW50IGNhbiBhIG1ha2UgNEsgTWVtUmQgcmVxdWVzdCwgYnV0IHRoZSBjb21w bGV0ZXIgaGFzIHRvIHNlbmQgY29tcGxldGlvbnMgYXMgMTI4QiBUTFBzLCBieSByZXNwZWN0aW5n IHRoZSBNUFMgc2V0dGluZy4gTVJSUyBkb2VzIG5vdCBmb3JjZSBhIGRldmljZSB0byB1c2UgaGln aGVyIE1QUyB2YWx1ZSB0aGFuIGl0IGlzIGNvbmZpZ3VyZWQgdG8uDQoNClRoaXMgd2FzIGNvdmVy ZWQgYnkgdGhlIHZlcnkgZmlyc3QgdG9waWMgaW4gQmpvcm4ncyBmaXJzdCByZXBseSB3aXRoaW4g dGhlIHRocmVhZC4uLg0KDQo+DQo+IEFub3RoZXIgZmFjdG9yIHRoYXQgbmVlZCB0byBiZSBjb25z aWRlcmVkIGZvciBzdG9yYWdlIGRldmljZXMgaXMgdGhhdCBzdXBwb3J0IG9mIFQxMCBQcm90ZWN0 aW9uIEluZm9ybWF0aW9uIChESUYpLiBGb3IgZXZlcnkgNTEyQiBvciA0S0IsIGEgOEIgUEkgaXMg Y29tcHV0ZWQgYW5kIGluc2VydGVkIG9yIHZlcmlmaWVkLCB3aGljaCByZXF1aXJlIHRoZSA1MTJC IG9mIGRhdGEgdG8gYXJyaXZlIGluIHNlcXVlbmNlLiBJZiB0aGUgTVJSUyBpcyA8IDUxMkIsIHRo aXMgbWlnaHQgcG9zZSBvdXQgb2Ygb3JkZXIgY29tcGxldGlvbnMgdG8gdGhlIHN0b3JhZ2UgZGV2 aWNlLCBpZiB0aGUgRVAgaGFzIHRvIHN1Ym1pdCBtdWx0aXBsZSBvdXRzdGFuZGluZyByZWFkIHJl cXVlc3RzIGluIG9yZGVyIHRvIGFjaGlldmUgaGlnaGVyIHBlcmZvcm1hbmNlLiBUaGlzIHdvdWxk IGJlIGEgY2hhbGxlbmdlIGZvciB0aGUgc3RvcmFnZSBlbmRwb2ludHMgdGhhdCBwcm9jZXNzIHRo ZSBUMTAgUEkgaW5saW5lIHdpdGggdGhlIHRyYW5zZmVyLCBub3cgdGhleSBoYXZlIHRvIHN0b3Jl IGFuZCBwcm9jZXNzIHRoZSA1MTJCIHNlY3RvcnMgb25jZSB0aGV5IHJlY2VpdmUgYWxsIHRoZSBU TFBzIGZvciB0aGF0IHNlY3Rvci4NCg0KVGhlICJUMTAiIGFzcGVjdHMgYXJlIG5ldyBhbmQgSSd2 ZSBub3QgaGVhcmQgYWJvdXQgdGhlbSBiZWZvcmUuICBPbiB0aGUgc3VyZmFjZSB0aGV5IHNlZW0g dG8gYmUgc3RvcmFnZSBkZXZpY2Ugc3BlY2lmaWMuICBJZiB0aGF0IGlzIGluZGVlZCB0aGUgY2Fz ZSB0aGVuIHRoZXJlIHNlZW1zIHRvIGJlIHNvbWUgbWl4aW5nIG9mIHR3bywgZGlzdGluY3RseSBk aWZmZXJlbnQsIHRoaW5ncy4gIFBDSWUgVExQcywgTVBTLCBNUlJTLCAuLi4sIGFyZSBhbGwgUENJ ZSBkZWZpbmVkIGl0ZW1zIHRoYXQgYXJlIGNvdmVyZWQgYnkgaXRzIHNwZWNpZmljYXRpb24uICBF eHBlY3RpbmcgdGhhdCBUMTAgc3BlY2lmaWNzIGNhbiBiZSBpbnRlcm1peGVkIHdpdGhpbiBQQ0ll J3MgcHJvdG9jb2wgZG9lc24ndCBtYWtlIGFueSBzZW5zZSBhbmQgc291bmRzIG11Y2ggbW9yZSBs aWtlIHNvbWV0aGluZyB0aGF0IHdpbGwgaGF2ZSB0byBiZSB0YWtlbiBjYXJlIG9mIGF0IHRoZSBj b250cm9sbGVyJ3MNCmxldmVsLiAgIFBlcmhhcHMgSSdtICB3YXkgb2ZmIGJhc2UgaGVyZSwgd2Un bGwgaGF2ZSB0byBoZWFyIG1vcmUgYWJvdXQgdGhpcw0KdG8gY29tZSB0byBzb21lIHVuZGVyc3Rh bmRpbmcuDQoNCj4NCj4gU28sIGl0IGlzIGJldHRlciB0byBkZWNvdXBsZSB0aGUgTVJSUyBhbmQg TVBTIGluIHBjaWVfYnVzX3BlcmYgbW9kZS4gTGlrZSBzdGF0ZWQgZWFybGllciBpbiB0aGUgdGhy ZWFkLCBwcm92aWRlIGFuIG9wdGlvbiB0byBjb25maWd1cmUgTVJSUyBzZXBhcmF0ZWx5IGluIHBj aWVfYnVzX3BlcmYgbW9kZS4NCg0KVGhpcyBhbHNvIGhhcyBiZWVuIGJyb3VnaHQgdXAgdHdpY2Ug YWxyZWFkeSwgYW5kIGNvdmVyZWQgaW4gdGhlIHByaW9yIHJlc3BvbnNlcy4uLg0KDQo+DQo+IFJl Z2FyZHMsDQo+IFJhZGouDQo=

#24 lore raw
In reply to Radjendirane Codandaramane
Quoted 1 line · Radjendirane Codandaramane

Hi Bjorne,

s/Bjorne/Bjorn/

Also, the convention on Linux mailing lists is to use interleaved reply style [1] because it makes it easier to follow the conversation.

Quoted 2 lines

Ceiling the MRRS to the MPS value in order to guarantee the interoperability in pcie_bus_perf mode does not make sense.

I disagree with this part. The Linux pcie_bus_perf mode isn't my favorite approach, but given the assumptions it makes, it does make sense. I laid out the explanation in [2]; if you disagree with something there, please be specific about what's wrong.

Quoted 7 lines

A device can make a memrd request according to the MRRS setting (which can be higher than its MPS), but the completer has to respect the MPS and send completions accordingly. As an example, system can configure MPS=128B and MRRS=4K, where an endpoint can a make 4K MemRd request, but the completer has to send completions as 128B TLPs, by respecting the MPS setting. MRRS does not force a device to use higher MPS value than it is configured to.

This part is certainly true. I used a very similar example in [2]:

Consider the case where a function has MPS=256 and MRRS=1024: if the function generates a 1024-byte read request, it should receive 4 completions, each with a 256-byte data payload.

Quoted 4 lines

Another factor that need to be considered for storage devices is that support of T10 Protection Information (DIF). For every 512B or 4KB, a 8B PI is computed and inserted or verified, which require the 512B of data to arrive in sequence.

MPS and MRRS are parts of the PCIe TLP protocol. I'm not aware of any connection between the PCIe protocol and the T10 Protection Information/DIF stuff. I assume the T10/DIF stuff is content being carried by PCIe, and PCIe doesn't care at all about the content it carries. Can you clarify what the connection is here?

Quoted 3 lines

If the MRRS is < 512B, this might pose out of order completions to the storage device, if the EP has to submit multiple outstanding read requests in order to achieve higher performance.

If I understand correctly, this is exactly the situation described in the implementation note in PCIe r4.0, sec 2.4.1: completions for a single memory read request are always returned in order (Table 2-39, entry D5b), but completions for multiple read requests may be returned in any order because completions may pass each other (entry D5a). The note doesn't mention this, but I think even the multiple outstanding read requests themselves can pass each other (entry B3).

So I agree that if the function generates multiple outstanding read requests, the completions for those requests may come back out of order. For example, if the function generates req-A and req-B, completions for req-A will be in order and completions for req-B will be in order, but it might receive cmpl-B1, cmpl-A1, cmpl-B2, cmpl-B3, cmpl-A2, cmpl-A3, cmpl-A4, cmpl-B4.

This reordering may happen no matter what the MRRS setting is. The only way to avoid it is to avoid multiple outstanding read requests.

Quoted 3 lines

This would be a challenge for the storage endpoints that process the T10 PI inline with the transfer, now they have to store and process the 512B sectors once they receive all the TLPs for that sector.

Are you saying these endpoints depend on MRRS >= 512 for correct operation?

I don't think think that would be legal, per sec 7.5.3.4, which says "the Function must not generate Read Requests with a size exceeding the set [MRRS] value." That doesn't leave the function any wiggle room -- if the OS programs MRRS=128, the function has to deal with it. There's no provision for the function to say "I can only support MRRS > X".

But you probably mean the endpoints will function correctly even with MRRS=128, but performance will be poor. That's certainly a concern for all devices; reducing MRRS increases overhead, so we should use the largest possible MRRS, subject to the constraints of any MPS/MRRS design (like pcie_bus_perf) and concerns about bandwidth allocation (see the implementation note in sec 7.5.3.4).

Quoted 3 lines

So, it is better to decouple the MRRS and MPS in pcie_bus_perf mode. Like stated earlier in the thread, provide an option to configure MRRS separately in pcie_bus_perf mode.

If you read [2] again, it will be obvious why the Linux pcie_bus_perf mode requires the use of MRRS in addition to MPS. At the very end of that email and in [3], I pointed out that the current Linux implementation is unnecessarily conservative in some cases, and there is room for fixing that, which should improve performance in some cases. But given the design of pcie_bus_perf, it's impossible to completely decouple MRRS and MPS in that mode.

[1] https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
[2] https://lkml.kernel.org/r/[email protected]
[3] https://lkml.kernel.org/r/[email protected]

#25 lore raw

SGksIEkgd291bGQgbGlrZSB0byBwcm92aWRlIG1vcmUgaW5mb3JtYXRpb24ganVzdCBmb3IgYW55 b25lIHdobyBtaWdodCBiZSBpbnRlcmVzdGVkLiANCldlIG1vZGlmeSBGVyB0byBzaW11bGF0ZSBh IE1QUyAxMjggY2FwYWJpbGl0eSBTU0QsIGFuZCBleHBlcmltZW50IHdpdGggZGlmZmVyZW50IHBj aXNfYnVzIG1vZGUgb24gRGVsbCBSNzMwWEQsIGhlbmNlIHdlIGNhbiBoYXZlIGEgYmV0dGVyIGxv b2sgYXQgdGhlIHdob2xlIHBpY3R1cmUuDQoNCkZpcnN0LCBjb2xkIGJvb3Qgd2l0aCBzaW5nbGUg U1NEIA0KZGV2aWNlOiAgU2xvdCAoQyAyNTYpICAgICBTU0QgKEMgMTI4QikJU2xvdCAoQyAyNTZC KSAgIFNTRCAoQyAyNTZCKQkNCiAgICAgICAgTVBTICBNUlJTICAgIE1QUyAgTVJSUyAgIE1QUyBN UlJTICAgICAgIE1QUyBNUlJTIA0KTm9ybWFsCTEyOAkxMjgJICAgICAxMjgJNDA5NiANCk5vcm1h bAkgICAgICAgICAgIAkgICAgICAgICAgICAgICAJMjU2CTEyOAkgICAgICAgMjU2CTQwOTYNClBl cmYJICAgIDI1NgkxMjgJICAgICAxMjgJMTI4CSAgIA0KUGVyZgkgICAgICAgICAgCSAgICAgICAg ICAgIAkgICAgICAgIDI1NgkxMjggICAgICAgIDI1NgkyNTYNClNhZmUgICAgCTEyOAkxMjgJICAg ICAxMjgJNDA5NgkNClNhZmUJICAgICAgICAgIAkgICAgICAgICAgICAJICAgICAgICAyNTYJMTI4 CSAgICAgICAyNTYJNDA5Ng0KDQpUaGVuIGNvbGQgYm9vdCB3aXRoIHR3byBkZXZpY2VzLCBkaWZm ZXJlbnQgTVBTIGNhcGFiaWxpdHmjrCBib3RoIHNsb3RzIGFyZSBkaXJlY3RseSBjb25uZWN0IHRv IENQVQ0KZGV2aWNlOiAgc2xvdCAoQyAyNTYpICAgICBTU0QgKEMgMTI4QikJICAgU2xvdCAoQyAy NTZCKSAgICAgU1NEIChDIDI1NkIpCQ0KICAgICAgICBNUFMgIE1SUlMgICAgTVBTICBNUlJTICAg ICAgTVBTIE1SUlMgICAgICAgTVBTIE1SUlMgICAgICANCk5vcm1hbAkxMjgJMTI4CSAgICAgMTI4 CTQwOTYJMjU2ICAxMjggICAgICAgMjU2ICA0MDk2ICAgIA0KUGVyZiAgICAgMjU2CTEyOAkgICAg IDEyOAkxMjgJICAgIDI1NiAgMTI4ICAgICAgIDI1NiAgMjU2ICAgIA0KU2FmZSAgCTEyOAkxMjgJ ICAgICAxMjgJNDA5NgkyNTYgIDEyOCAgICAgICAyNTYgIDQwOTYgICANCg0KRmluYWxseSwgdG8g bWF0Y2ggU2luYW4ncyBleGFtcGxlLCB3ZSB1c2UgYSBQQ0llIHN3aXRjaCBmb3IgdHdvIFUuMiBT U0Q6DQpcLVswMDAwOjAwXS0rLTAwLjANCiAgICAgICAgICAgICArLTAxLjAtWzAzXS0tLS0wMC4w DQogICAgICAgICAgICAgKy0wMi4wLVswNF0tLQ0KICAgICAgICAgICAgICstMDMuMC1bMDJdLS0r LTAwLjANCiAgICAgICAgICAgICB8ICAgICAgICAgICAgXC0wMC4xDQogICAgICAgICAgICAgKy0w My4xLVswMV0tLSstMDAuMA0KICAgICAgICAgICAgIHwgICAgICAgICAgICBcLTAwLjENCiAgICAg ICAgICAgICArLTAzLjItWzA1LTBhXS0tLS0wMC4wLVswNi0wYV0tLSstMDQuMC1bMDddLS0NCiAg ICAgICAgICAgICB8ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICstMDUuMC1bMDhdLS0t LTAwLjAgICAgLT4gY29ubmVjdCBhIDI1NkIgc3NkDQogICAgICAgICAgICAgfCAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICArLTA2LjAtWzA5XS0tLS0wMC4wICAgIC0+IGNvbm5lY3QgYSAx MjhCIHNzZA0KDQoJCTAwLjAzLjIgKEMgMjU2KSAgMDU6MDAuMCAoQzUxMikJMDY6MDUuMCAoQzUx MikJICAwODowMC4wIChTU0QgQzI1NqOpMDY6MDYuMCAoQzUxMikJMDk6MDAuMCAoU1NEIEMxMjgp CQ0KCQlNUFMJTVJSUwkgTVBTCU1SUlMJTVBTCU1SUlMJIE1QUwlNUlJTCU1QUyBNUlJTCU1QUwlN UlJTDQpOb3JtYWwJMTI4CTEyOAkgICAgIDEyOAkxMjgJICAgIDEyOAkxMjgJICAgICAxMjgJNDA5 NgkxMjgJMTI4CSAgICAxMjgJNDA5Ng0KUGVyZgkJMjU2CTEyOCAgICAgIDI1NgkxMjgJICAgIDI1 NgkxMjgJICAgICAyNTYJMjU2CSAgICAyNTYJMTI4CSAgICAxMjgJMTI4DQpTYWZlCQkxMjgJMTI4 CSAgICAgMTI4CTEyOAkgICAgMTI4CTEyOAkgICAgIDEyOAk0MDk2CTEyOAkxMjgJICAgIDEyOAk0 MDk2DQoNCkkgdGhpbmsgZnJvbSBhYm92ZSBleGFtcGxlczoNCjEuIHBlcmYgbW9kZSBpcyBtb3Zp bmcgZGV2aWNlcyB0byAyNTYgTVBTIGFzIGl0IGNhbi4NCjIuIHNhZmUgbW9kZSBpcyBzZXR0aW5n IHRvIDEyOCBNUFMgDQozLiBwZXJmIG1vZGUgc2V0IE1SUlM9TVBTIGlzIGEgQ09SUkVDVCBjYWxs IGZvciBkZXZpY2Ugd2l0aCBNUFNDIGxvd2VyIHRoYW4gaXRzIHBhcmVudHMuDQo0LiBwZXJmIG1v ZGUgc2V0IE1SUlM9TVBTIGlzIG5vdCBuZWNlc3NhcnkgZm9yIGEgZGV2aWNlIHdpdGggU0FNRSBN UFNDIGFzIGl0cyBwYXJlbnRzPyANCjUuIGl0IGlzIGFuIGludGVyZXN0ZWQgcG9pbnQgdG8gbWUg dGhhdCBzbG90L3N3aXRjaC9yb290IE1SUlMgYXJlIGFsd2F5cyBzZXQgdG8gMTI4QiwgSSBoYXZl IG5vdCBmb3VuZCBvdXQgd2h5Lg0KDQpBZ2FpbiwgdGhhbmtzIGZvciBldmVyeW9uZSdzIHRpbWUg b24gdGhpcyBzdWJqZWN0LiBXZSBoYXZlIGxlYXJudCBhIGxvdC4NCg0KUm9uDQoNCi0tLS0tT3Jp Z2luYWwgTWVzc2FnZS0tLS0tDQpGcm9tOiBCam9ybiBIZWxnYWFzIFttYWlsdG86aGVsZ2Fhc0Br ZXJuZWwub3JnXSANClNlbnQ6IDIwMTjE6jHUwjI1yNUgMjowMQ0KVG86IFJhZGplbmRpcmFuZSBD b2RhbmRhcmFtYW5lIDxyYWRqZW5kaXJhbmUuY29kYW5kYUBtaWNyb3NlbWkuY29tPg0KQ2M6IFJv biBZdWFuIDxyb24ueXVhbkBtZW1ibGF6ZS5jb20+OyBTaW5hbiBLYXlhIDxva2F5YUBjb2RlYXVy b3JhLm9yZz47IEJqb3JuIEhlbGdhYXMgPGJoZWxnYWFzQGdvb2dsZS5jb20+OyBCbyBDaGVuIDxi by5jaGVuQG1lbWJsYXplLmNvbT47IFdpbGxpYW0gSHVhbmcgPHdpbGxpYW0uaHVhbmdAbWVtYmxh emUuY29tPjsgRmVuZ21pbmcgV3UgPGZlbmdtaW5nLnd1QG1lbWJsYXplLmNvbT47IEphc29uIEpp YW5nIDxqYXNvbi5qaWFuZ0BtaWNyb3NlbWkuY29tPjsgUmFteWFrYW50aCBFZHVwdWdhbnRpIDxS YW15YWthbnRoLkVkdXB1Z2FudGlAbWljcm9zZW1pLmNvbT47IFdpbGxpYW0gQ2hlbmcgPHdpbGxp YW0uY2hlbmdAbWljcm9zZW1pLmNvbT47IEtpbSBIZWxwZXIgKGtoZWxwZXIpIDxraGVscGVyQG1p Y3Jvbi5jb20+OyBMaW51eCBQQ0kgPGxpbnV4LXBjaUB2Z2VyLmtlcm5lbC5vcmc+DQpTdWJqZWN0 OiBSZTogT25lIFF1ZXN0aW9uIEFib3V0IFBDSWUgQlVTIENvbmZpZyBUeXBlIHdpdGggcGNpZV9i dXNfc2FmZSBvciBwY2llX2J1c19wZXJmIE9uIE5WTWUgRGV2aWNlDQoNCk9uIFR1ZSwgSmFuIDIz LCAyMDE4IGF0IDExOjUwOjMwUE0gKzAwMDAsIFJhZGplbmRpcmFuZSBDb2RhbmRhcmFtYW5lIHdy b3RlOg0KPiBIaSBCam9ybmUsDQoNCnMvQmpvcm5lL0Jqb3JuLw0KDQpBbHNvLCB0aGUgY29udmVu dGlvbiBvbiBMaW51eCBtYWlsaW5nIGxpc3RzIGlzIHRvIHVzZSBpbnRlcmxlYXZlZCByZXBseSBz dHlsZSBbMV0gYmVjYXVzZSBpdCBtYWtlcyBpdCBlYXNpZXIgdG8gZm9sbG93IHRoZSBjb252ZXJz YXRpb24uDQoNCj4gQ2VpbGluZyB0aGUgTVJSUyB0byB0aGUgTVBTIHZhbHVlIGluIG9yZGVyIHRv IGd1YXJhbnRlZSB0aGUgDQo+IGludGVyb3BlcmFiaWxpdHkgaW4gcGNpZV9idXNfcGVyZiBtb2Rl IGRvZXMgbm90IG1ha2Ugc2Vuc2UuDQoNCkkgZGlzYWdyZWUgd2l0aCB0aGlzIHBhcnQuICBUaGUg TGludXggcGNpZV9idXNfcGVyZiBtb2RlIGlzbid0IG15IGZhdm9yaXRlIGFwcHJvYWNoLCBidXQg Z2l2ZW4gdGhlIGFzc3VtcHRpb25zIGl0IG1ha2VzLCBpdCBkb2VzIG1ha2Ugc2Vuc2UuICBJIGxh aWQgb3V0IHRoZSBleHBsYW5hdGlvbiBpbiBbMl07IGlmIHlvdSBkaXNhZ3JlZSB3aXRoIHNvbWV0 aGluZyB0aGVyZSwgcGxlYXNlIGJlIHNwZWNpZmljIGFib3V0IHdoYXQncyB3cm9uZy4NCg0KPiBB IGRldmljZSBjYW4gbWFrZSBhIG1lbXJkIHJlcXVlc3QgYWNjb3JkaW5nIHRvIHRoZSBNUlJTIHNl dHRpbmcgKHdoaWNoIA0KPiBjYW4gYmUgaGlnaGVyIHRoYW4gaXRzIE1QUyksIGJ1dCB0aGUgY29t cGxldGVyIGhhcyB0byByZXNwZWN0IHRoZSBNUFMgDQo+IGFuZCBzZW5kIGNvbXBsZXRpb25zIGFj Y29yZGluZ2x5LiBBcyBhbiBleGFtcGxlLCBzeXN0ZW0gY2FuIGNvbmZpZ3VyZSANCj4gTVBTPTEy OEIgYW5kIE1SUlM9NEssIHdoZXJlIGFuIGVuZHBvaW50IGNhbiBhIG1ha2UgNEsgTWVtUmQgcmVx dWVzdCwgDQo+IGJ1dCB0aGUgY29tcGxldGVyIGhhcyB0byBzZW5kIGNvbXBsZXRpb25zIGFzIDEy OEIgVExQcywgYnkgcmVzcGVjdGluZyANCj4gdGhlIE1QUyBzZXR0aW5nLiBNUlJTIGRvZXMgbm90 IGZvcmNlIGEgZGV2aWNlIHRvIHVzZSBoaWdoZXIgTVBTIHZhbHVlIA0KPiB0aGFuIGl0IGlzIGNv bmZpZ3VyZWQgdG8uDQoNClRoaXMgcGFydCBpcyBjZXJ0YWlubHkgdHJ1ZS4gIEkgdXNlZCBhIHZl cnkgc2ltaWxhciBleGFtcGxlIGluIFsyXToNCg0KICBDb25zaWRlciB0aGUgY2FzZSB3aGVyZSBh IGZ1bmN0aW9uIGhhcyBNUFM9MjU2IGFuZCBNUlJTPTEwMjQ6IGlmIHRoZQ0KICBmdW5jdGlvbiBn ZW5lcmF0ZXMgYSAxMDI0LWJ5dGUgcmVhZCByZXF1ZXN0LCBpdCBzaG91bGQgcmVjZWl2ZSA0DQog IGNvbXBsZXRpb25zLCBlYWNoIHdpdGggYSAyNTYtYnl0ZSBkYXRhIHBheWxvYWQuDQoNCj4gQW5v dGhlciBmYWN0b3IgdGhhdCBuZWVkIHRvIGJlIGNvbnNpZGVyZWQgZm9yIHN0b3JhZ2UgZGV2aWNl cyBpcyB0aGF0IA0KPiBzdXBwb3J0IG9mIFQxMCBQcm90ZWN0aW9uIEluZm9ybWF0aW9uIChESUYp LiBGb3IgZXZlcnkgNTEyQiBvciA0S0IsIGEgDQo+IDhCIFBJIGlzIGNvbXB1dGVkIGFuZCBpbnNl cnRlZCBvciB2ZXJpZmllZCwgd2hpY2ggcmVxdWlyZSB0aGUgNTEyQiBvZiANCj4gZGF0YSB0byBh cnJpdmUgaW4gc2VxdWVuY2UuDQoNCk1QUyBhbmQgTVJSUyBhcmUgcGFydHMgb2YgdGhlIFBDSWUg VExQIHByb3RvY29sLiAgSSdtIG5vdCBhd2FyZSBvZiBhbnkgY29ubmVjdGlvbiBiZXR3ZWVuIHRo ZSBQQ0llIHByb3RvY29sIGFuZCB0aGUgVDEwIFByb3RlY3Rpb24gSW5mb3JtYXRpb24vRElGIHN0 dWZmLiAgSSBhc3N1bWUgdGhlIFQxMC9ESUYgc3R1ZmYgaXMgY29udGVudCBiZWluZyBjYXJyaWVk IGJ5IFBDSWUsIGFuZCBQQ0llIGRvZXNuJ3QgY2FyZSBhdCBhbGwgYWJvdXQgdGhlIGNvbnRlbnQg aXQgY2Fycmllcy4gIENhbiB5b3UgY2xhcmlmeSB3aGF0IHRoZSBjb25uZWN0aW9uIGlzIGhlcmU/ DQoNCj4gSWYgdGhlIE1SUlMgaXMgPCA1MTJCLCB0aGlzIG1pZ2h0IHBvc2Ugb3V0IG9mIG9yZGVy IGNvbXBsZXRpb25zIHRvIHRoZSANCj4gc3RvcmFnZSBkZXZpY2UsIGlmIHRoZSBFUCBoYXMgdG8g c3VibWl0IG11bHRpcGxlIG91dHN0YW5kaW5nIHJlYWQgDQo+IHJlcXVlc3RzIGluIG9yZGVyIHRv IGFjaGlldmUgaGlnaGVyIHBlcmZvcm1hbmNlLg0KDQpJZiBJIHVuZGVyc3RhbmQgY29ycmVjdGx5 LCB0aGlzIGlzIGV4YWN0bHkgdGhlIHNpdHVhdGlvbiBkZXNjcmliZWQgaW4gdGhlIGltcGxlbWVu dGF0aW9uIG5vdGUgaW4gUENJZSByNC4wLCBzZWMgMi40LjE6IGNvbXBsZXRpb25zIGZvciBhIHNp bmdsZSBtZW1vcnkgcmVhZCByZXF1ZXN0IGFyZSBhbHdheXMgcmV0dXJuZWQgaW4gb3JkZXIgKFRh YmxlIDItMzksIGVudHJ5IEQ1YiksIGJ1dCBjb21wbGV0aW9ucyBmb3IgbXVsdGlwbGUgcmVhZCBy ZXF1ZXN0cyBtYXkgYmUgcmV0dXJuZWQgaW4gYW55IG9yZGVyIGJlY2F1c2UgY29tcGxldGlvbnMg bWF5IHBhc3MgZWFjaCBvdGhlciAoZW50cnkgRDVhKS4gIFRoZSBub3RlIGRvZXNuJ3QgbWVudGlv biB0aGlzLCBidXQgSSB0aGluayBldmVuIHRoZSBtdWx0aXBsZSBvdXRzdGFuZGluZyByZWFkIHJl cXVlc3RzIHRoZW1zZWx2ZXMgY2FuIHBhc3MgZWFjaCBvdGhlciAoZW50cnkgQjMpLg0KDQpTbyBJ IGFncmVlIHRoYXQgaWYgdGhlIGZ1bmN0aW9uIGdlbmVyYXRlcyBtdWx0aXBsZSBvdXRzdGFuZGlu ZyByZWFkIHJlcXVlc3RzLCB0aGUgY29tcGxldGlvbnMgZm9yIHRob3NlIHJlcXVlc3RzIG1heSBj b21lIGJhY2sgb3V0IG9mIG9yZGVyLiAgRm9yIGV4YW1wbGUsIGlmIHRoZSBmdW5jdGlvbiBnZW5l cmF0ZXMgcmVxLUEgYW5kIHJlcS1CLCBjb21wbGV0aW9ucyBmb3IgcmVxLUEgd2lsbCBiZSBpbiBv cmRlciBhbmQgY29tcGxldGlvbnMgZm9yIHJlcS1CIHdpbGwgYmUgaW4gb3JkZXIsIGJ1dCBpdCBt aWdodCByZWNlaXZlIGNtcGwtQjEsIGNtcGwtQTEsIGNtcGwtQjIsIGNtcGwtQjMsIGNtcGwtQTIs IGNtcGwtQTMsIGNtcGwtQTQsIGNtcGwtQjQuDQoNClRoaXMgcmVvcmRlcmluZyBtYXkgaGFwcGVu IG5vIG1hdHRlciB3aGF0IHRoZSBNUlJTIHNldHRpbmcgaXMuICBUaGUgb25seSB3YXkgdG8gYXZv aWQgaXQgaXMgdG8gYXZvaWQgbXVsdGlwbGUgb3V0c3RhbmRpbmcgcmVhZCByZXF1ZXN0cy4NCg0K PiBUaGlzIHdvdWxkIGJlIGEgY2hhbGxlbmdlIGZvciB0aGUgc3RvcmFnZSBlbmRwb2ludHMgdGhh dCBwcm9jZXNzIHRoZQ0KPiBUMTAgUEkgaW5saW5lIHdpdGggdGhlIHRyYW5zZmVyLCBub3cgdGhl eSBoYXZlIHRvIHN0b3JlIGFuZCBwcm9jZXNzIA0KPiB0aGUgNTEyQiBzZWN0b3JzIG9uY2UgdGhl eSByZWNlaXZlIGFsbCB0aGUgVExQcyBmb3IgdGhhdCBzZWN0b3IuDQoNCkFyZSB5b3Ugc2F5aW5n IHRoZXNlIGVuZHBvaW50cyBkZXBlbmQgb24gTVJSUyA+PSA1MTIgZm9yIGNvcnJlY3Qgb3BlcmF0 aW9uPw0KDQpJIGRvbid0IHRoaW5rIHRoaW5rIHRoYXQgd291bGQgYmUgbGVnYWwsIHBlciBzZWMg Ny41LjMuNCwgd2hpY2ggc2F5cyAidGhlIEZ1bmN0aW9uIG11c3Qgbm90IGdlbmVyYXRlIFJlYWQg UmVxdWVzdHMgd2l0aCBhIHNpemUgZXhjZWVkaW5nIHRoZSBzZXQgW01SUlNdIHZhbHVlLiIgIFRo YXQgZG9lc24ndCBsZWF2ZSB0aGUgZnVuY3Rpb24gYW55IHdpZ2dsZSByb29tIC0tIGlmIHRoZSBP UyBwcm9ncmFtcyBNUlJTPTEyOCwgdGhlIGZ1bmN0aW9uIGhhcyB0byBkZWFsIHdpdGggaXQuDQpU aGVyZSdzIG5vIHByb3Zpc2lvbiBmb3IgdGhlIGZ1bmN0aW9uIHRvIHNheSAiSSBjYW4gb25seSBz dXBwb3J0IE1SUlMgPiBYIi4NCg0KQnV0IHlvdSBwcm9iYWJseSBtZWFuIHRoZSBlbmRwb2ludHMg d2lsbCBmdW5jdGlvbiBjb3JyZWN0bHkgZXZlbiB3aXRoIE1SUlM9MTI4LCBidXQgcGVyZm9ybWFu Y2Ugd2lsbCBiZSBwb29yLiAgVGhhdCdzIGNlcnRhaW5seSBhIGNvbmNlcm4gZm9yIGFsbCBkZXZp Y2VzOyByZWR1Y2luZyBNUlJTIGluY3JlYXNlcyBvdmVyaGVhZCwgc28gd2Ugc2hvdWxkIHVzZSB0 aGUgbGFyZ2VzdCBwb3NzaWJsZSBNUlJTLCBzdWJqZWN0IHRvIHRoZSBjb25zdHJhaW50cyBvZiBh bnkgTVBTL01SUlMgZGVzaWduIChsaWtlIHBjaWVfYnVzX3BlcmYpIGFuZCBjb25jZXJucyBhYm91 dCBiYW5kd2lkdGggYWxsb2NhdGlvbiAoc2VlIHRoZSBpbXBsZW1lbnRhdGlvbiBub3RlIGluIHNl YyA3LjUuMy40KS4NCg0KPiBTbywgaXQgaXMgYmV0dGVyIHRvIGRlY291cGxlIHRoZSBNUlJTIGFu ZCBNUFMgaW4gcGNpZV9idXNfcGVyZiBtb2RlLg0KPiBMaWtlIHN0YXRlZCBlYXJsaWVyIGluIHRo ZSB0aHJlYWQsIHByb3ZpZGUgYW4gb3B0aW9uIHRvIGNvbmZpZ3VyZSBNUlJTIA0KPiBzZXBhcmF0 ZWx5IGluIHBjaWVfYnVzX3BlcmYgbW9kZS4NCg0KSWYgeW91IHJlYWQgWzJdIGFnYWluLCBpdCB3 aWxsIGJlIG9idmlvdXMgd2h5IHRoZSBMaW51eCBwY2llX2J1c19wZXJmIG1vZGUgcmVxdWlyZXMg dGhlIHVzZSBvZiBNUlJTIGluIGFkZGl0aW9uIHRvIE1QUy4gIEF0IHRoZSB2ZXJ5IGVuZCBvZiB0 aGF0IGVtYWlsIGFuZCBpbiBbM10sIEkgcG9pbnRlZCBvdXQgdGhhdCB0aGUgY3VycmVudCBMaW51 eCBpbXBsZW1lbnRhdGlvbiBpcyB1bm5lY2Vzc2FyaWx5IGNvbnNlcnZhdGl2ZSBpbiBzb21lIGNh c2VzLCBhbmQgdGhlcmUgaXMgcm9vbSBmb3IgZml4aW5nIHRoYXQsIHdoaWNoIHNob3VsZCBpbXBy b3ZlIHBlcmZvcm1hbmNlIGluIHNvbWUgY2FzZXMuICBCdXQgZ2l2ZW4gdGhlIGRlc2lnbiBvZiBw Y2llX2J1c19wZXJmLCBpdCdzIGltcG9zc2libGUgdG8gY29tcGxldGVseSBkZWNvdXBsZSBNUlJT IGFuZCBNUFMgaW4gdGhhdCBtb2RlLg0KDQpbMV0gaHR0cHM6Ly9lbi53aWtpcGVkaWEub3JnL3dp a2kvUG9zdGluZ19zdHlsZSNJbnRlcmxlYXZlZF9zdHlsZQ0KWzJdIGh0dHBzOi8vbGttbC5rZXJu ZWwub3JnL3IvMjAxODAxMTkyMDUxNTMuR0IxNjA2MThAYmhlbGdhYXMtZ2xhcHRvcC5yb2FtLmNv cnAuZ29vZ2xlLmNvbQ0KWzNdIGh0dHBzOi8vbGttbC5rZXJuZWwub3JnL3IvMjAxODAxMjMxNzQ4 MjEuR0Y1MzE3QGJoZWxnYWFzLWdsYXB0b3Aucm9hbS5jb3JwLmdvb2dsZS5jb20NCg==

#26 lore raw
Quoted 1 line · Ron Yuan

Hi, I would like to provide more information just for anyone who might be=

interested.

Quoted 1 line

We modify FW to simulate a MPS 128 capability SSD, and experiment with di=

fferent pcis_bus mode on Dell R730XD, hence we can have a better look at th= e whole picture.

Quoted 6 lines

First, cold boot with single SSD
device: Slot (C 256) SSD (C 128B) Slot (C 256B) SSD (C 256B)
MPS MRRS MPS MRRS MPS MRRS MPS MRRS
Normal 128 128 128 4096
Normal 256 128 256 4=

096

Quoted 2 lines

Perf 256 128 128 128
Perf 256 128 256 2=

56

Quoted 2 lines

Safe 128 128 128 4096
Safe 256 128 25=

6 4096

Quoted 2 lines

Then cold boot with two devices, different MPS capability=EF=BC=8C both s=

lots are directly connect to CPU

Quoted 4 lines

device: slot (C 256) SSD (C 128B) Slot (C 256B) SSD (C 256B)
MPS MRRS MPS MRRS MPS MRRS MPS MRRS

Normal  128     128          128        4096    256  128       256  4096
Perf     256    128          128        128         256  128       256  2=

56

Quoted 4 lines

Safe 128 128 128 4096 256 128 256 4096

Finally, to match Sinan's example, we use a PCIe switch for two U.2 SSD:
\-[0000:00]-+-00.0

Quoted 8 lines
             +-01.0-[03]----00.0
             +-02.0-[04]--
             +-03.0-[02]--+-00.0
             |            \-00.1
             +-03.1-[01]--+-00.0
             |            \-00.1
             +-03.2-[05-0a]----00.0-[06-0a]--+-04.0-[07]--
             |                               +-05.0-[08]----00.0    -> co=

nnect a 256B ssd

Quoted 1 line
             |                               +-06.0-[09]----00.0    -> co=

nnect a 128B ssd

Quoted 2 lines
                00.03.2 (C 256)  05:00.0 (C512) 06:05.0 (C512)    08:00.0=

(SSD C256=EF=BC=8906:06.0 (C512) 09:00.0 (SSD C128)

Quoted 1 line

MPS MRRS MPS MRRS MPS MRRS MPS M=

RRS MPS MRRS MPS MRRS

Quoted 1 line

Normal 128 128 128 128 128 128 128 =

       4096    128     128         128 4096
> Perf            256     128      256    128         256 128          256 =
       256         256 128         128 128
Quoted 1 line

Safe 128 128 128 128 128 128 =

    128        4096    128     128         128 4096
Quoted 5 lines

I think from above examples:
1. perf mode is moving devices to 256 MPS as it can.
2. safe mode is setting to 128 MPS
3. perf mode set MRRS=3DMPS is a CORRECT call for device with MPSC lower =

than its parents.

Quoted 1 line

4. perf mode set MRRS=3DMPS is not necessary for a device with SAME MPSC =

as its parents?

Quoted 1 line

5. it is an interested point to me that slot/switch/root MRRS are always =

set to 128B, I have not found out why.

In Sinan's original posting, a reference to
https://www.xilinx.com/support/documentation/white_papers/wp350.pdf
was provided. When I read that paper and got to the "Read Completion Boundary" section I thought to myself: "If RCB can only be 64 or 128 bytes then what's the point of MPS (or MRRS) as all TLP completions would be limited to 64 or 128 bytes? (see also the paper's 'Read Completions with the RCB Set to 64 Bytes' figure)". I brought this up to a colleague and they surmised that possibly only _lower end_ (a.k.a. lazy) chipset implementations would truly have RCB limited sized completions; higher end chipsets would of course have to comply with RCB when communicating with the memory controller but could then aggregate data into larger MPS (or MRRS) sized TLP completion packets. Perhaps this might explain why you always saw slot/switch/root values set at 128B?

Quoted 4 lines

Again, thanks for everyone's time on this subject. We have learnt a lot.

Ron

author
#27 lore raw
Quoted 6 lines · Myron Stowe
Quoted 6 lines

I think from above examples:
1. perf mode is moving devices to 256 MPS as it can.
2. safe mode is setting to 128 MPS
3. perf mode set MRRS=MPS is a CORRECT call for device with MPSC lower than its parents.
4. perf mode set MRRS=MPS is not necessary for a device with SAME MPSC as its parents?
5. it is an interested point to me that slot/switch/root MRRS are always set to 128B, I have not found out why.

Quoted 15 lines

In Sinan's original posting, a reference to
https://www.xilinx.com/support/documentation/white_papers/wp350.pdf
was provided. When I read that paper and got to the "Read Completion Boundary" section I thought to myself: "If RCB can only be 64 or 128 bytes then what's the point of MPS (or MRRS) as all TLP completions would be limited to 64 or 128 bytes? (see also the paper's 'Read Completions with the RCB Set to 64 Bytes' figure)". I brought this up to a colleague and they surmised that possibly only _lower end_ (a.k.a. lazy) chipset implementations would truly have RCB limited sized completions; higher end chipsets would of course have to comply with RCB when communicating with the memory controller but could then aggregate data into larger MPS (or MRRS) sized TLP completion packets. Perhaps this might explain why you always saw slot/switch/root values set at 128B?

I have looked at that paper before. It is plain wrong. Read Completion Boundary is about the alignment of addresses that an endpoint is sending in memory read packets according to the spec. It has nothing to do with the packet size.

The limitation on MRRS could be that device (switch) doesn't support it. A device is allowed to reject an MRRS set value.

Signature · 3 lines
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
#28 lore raw
Quoted 23 lines · Sinan Kaya
Quoted 7 lines · Myron Stowe
Quoted 6 lines

I think from above examples:
1. perf mode is moving devices to 256 MPS as it can.
2. safe mode is setting to 128 MPS
3. perf mode set MRRS=MPS is a CORRECT call for device with MPSC lower than its parents.
4. perf mode set MRRS=MPS is not necessary for a device with SAME MPSC as its parents?
5. it is an interested point to me that slot/switch/root MRRS are always set to 128B, I have not found out why.

Quoted 14 lines

In Sinan's original posting, a reference to
https://www.xilinx.com/support/documentation/white_papers/wp350.pdf
was provided. When I read that paper and got to the "Read Completion Boundary" section I thought to myself: "If RCB can only be 64 or 128 bytes then what's the point of MPS (or MRRS) as all TLP completions would be limited to 64 or 128 bytes? (see also the paper's 'Read Completions with the RCB Set to 64 Bytes' figure)". I brought this up to a colleague and they surmised that possibly only _lower end_ (a.k.a. lazy) chipset implementations would truly have RCB limited sized completions; higher end chipsets would of course have to comply with RCB when communicating with the memory controller but could then aggregate data into larger MPS (or MRRS) sized TLP completion packets. Perhaps this might explain why you always saw slot/switch/root values set at 128B?

Quoted 4 lines

I have looked at that paper before. It is plain wrong. Read Completion Boundary is about the alignment of addresses that an endpoint is sending in memory read packets according to the spec. It has nothing to do with the packet size.

I hope this isn't being too pedantic, but I think RCB applies to the *completer*, not the requester. Typically an endpoint generates a memory read and a root port supplies data to complete it.

I didn't see anything in the "Read Completion Boundary" section of the paper that I thought *contradicted* the spec. I did think this was speculation:

Typically, most root complexes set the RCB at 64 bytes and return data in 64-byte completions instead of what might be allowed by the MPS.

That would be legal per spec, but I don't have direct knowledge of whether root complexes actually do that. I tend to doubt it because it seems like people do observe performance differences based on the MPS setting.

I agree that the "Read Transaction Throughput" section is inaccurate when it says:

The size of the completion is determined by a completer's read completion boundary.

RCB may limit the size of the first completion to less than MPS, but subsequent completions (except the final one) only need to be a multiple of RCB.

RCB determines the boundaries at which a completer is allowed to end a TLP (PCIe r4.0, sec 2.3.1.1 has the rules). Let's work through an example:

MPS=128
RCB=64

Endpoint generates a 256-byte read request for addresses 0x2-0x101

In the absence of RCB, a root port could satisfy this request with two completions:

completion 0: bytes 0x2- 0x81 (128 bytes) <-- illegal
completion 0: bytes 0x82-0x101 (128 bytes)

But with RCB, the first completion is illegal because it doesn't satisfy the entire request and it doesn't end on a multiple of RCB. Because of RCB the completer must respond with at least three completions. The best it can do is this:

  completion 0: bytes   0x2- 0x7f  (126 bytes)
  completion 1: bytes  0x80- 0xff  (128 bytes)
  completion 2: bytes 0x100-0x101  (2 bytes)

It is *allowed* but not required to respond with more completions, e.g.,

  completion 0: bytes   0x2- 0x3f  (62 bytes)
  completion 1: bytes  0x40- 0x7f  (64 bytes)
  completion 2: bytes  0x80- 0xbf  (64 bytes)
  completion 3: bytes  0xc0- 0xff  (64 bytes)
  completion 4: bytes 0x100-0x101  (2 bytes)
author
#29 lore raw
Quoted 80 lines · Bjorn Helgaas
Quoted 23 lines · Sinan Kaya
Quoted 7 lines · Myron Stowe
Quoted 6 lines

I think from above examples:
1. perf mode is moving devices to 256 MPS as it can.
2. safe mode is setting to 128 MPS
3. perf mode set MRRS=MPS is a CORRECT call for device with MPSC lower than its parents.
4. perf mode set MRRS=MPS is not necessary for a device with SAME MPSC as its parents?
5. it is an interested point to me that slot/switch/root MRRS are always set to 128B, I have not found out why.

Quoted 14 lines

In Sinan's original posting, a reference to
https://www.xilinx.com/support/documentation/white_papers/wp350.pdf
was provided. When I read that paper and got to the "Read Completion Boundary" section I thought to myself: "If RCB can only be 64 or 128 bytes then what's the point of MPS (or MRRS) as all TLP completions would be limited to 64 or 128 bytes? (see also the paper's 'Read Completions with the RCB Set to 64 Bytes' figure)". I brought this up to a colleague and they surmised that possibly only _lower end_ (a.k.a. lazy) chipset implementations would truly have RCB limited sized completions; higher end chipsets would of course have to comply with RCB when communicating with the memory controller but could then aggregate data into larger MPS (or MRRS) sized TLP completion packets. Perhaps this might explain why you always saw slot/switch/root values set at 128B?

Quoted 4 lines

I have looked at that paper before. It is plain wrong. Read Completion Boundary is about the alignment of addresses that an endpoint is sending in memory read packets according to the spec. It has nothing to do with the packet size.

I hope this isn't being too pedantic, but I think RCB applies to the *completer*, not the requester. Typically an endpoint generates a memory read and a root port supplies data to complete it.

I didn't see anything in the "Read Completion Boundary" section of the paper that I thought *contradicted* the spec. I did think this was speculation:

Typically, most root complexes set the RCB at 64 bytes and return data in 64-byte completions instead of what might be allowed by the MPS.

That would be legal per spec, but I don't have direct knowledge of whether root complexes actually do that. I tend to doubt it because it seems like people do observe performance differences based on the MPS setting.

I agree that the "Read Transaction Throughput" section is inaccurate when it says:

The size of the completion is determined by a completer's read completion boundary.

RCB may limit the size of the first completion to less than MPS, but subsequent completions (except the final one) only need to be a multiple of RCB.

RCB determines the boundaries at which a completer is allowed to end a TLP (PCIe r4.0, sec 2.3.1.1 has the rules). Let's work through an example:

MPS=128
RCB=64

Endpoint generates a 256-byte read request for addresses 0x2-0x101

In the absence of RCB, a root port could satisfy this request with two completions:

completion 0: bytes 0x2- 0x81 (128 bytes) <-- illegal
completion 0: bytes 0x82-0x101 (128 bytes)

But with RCB, the first completion is illegal because it doesn't satisfy the entire request and it doesn't end on a multiple of RCB. Because of RCB the completer must respond with at least three completions. The best it can do is this:

  completion 0: bytes   0x2- 0x7f  (126 bytes)
  completion 1: bytes  0x80- 0xff  (128 bytes)
  completion 2: bytes 0x100-0x101  (2 bytes)
Quoted 9 lines

It is *allowed* but not required to respond with more completions, e.g.,

  completion 0: bytes   0x2- 0x3f  (62 bytes)
  completion 1: bytes  0x40- 0x7f  (64 bytes)
  completion 2: bytes  0x80- 0xbf  (64 bytes)
  completion 3: bytes  0xc0- 0xff  (64 bytes)
  completion 4: bytes 0x100-0x101  (2 bytes)

Very good summary. Thanks for doing a detailed analysis.

Signature · 3 lines
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
author
#30 lore raw
Quoted 4 lines · Sinan Kaya

I think from above examples:
1. perf mode is moving devices to 256 MPS as it can.
2. safe mode is setting to 128 MPS
3. perf mode set MRRS=MPS is a CORRECT call for device with MPSC lower than its parents.

Thinking more about this...

The only way for a transaction of size that's greater than MRRS can happen if there is a device specific driver attached that sends commands to the device and device issues a large sized memory read transaction.

Let's say that we find the slow device in tree.

If no driver is attached to a slow device in the tree and assume it is safe and do not reduce MRRS?

Quoted 2 lines

4. perf mode set MRRS=MPS is not necessary for a device with SAME MPSC as its parents?
5. it is an interested point to me that slot/switch/root MRRS are always set to 128B, I have not found out why.

Signature · 3 lines
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.