Memory API
The current memory APIs (cpu_register_io_memory, cpu_register_physical_memory) suffer from a number of drawbacks:
- lack of centralized bookkeeping
- a cpu_register_physical_memory() that overlaps an existing region
will overwrite the preexisting region; but a following cpu_unregister_physical_memory() will not restore things to status quo ante.
- coalescing and dirty logging are all cleared on unregistering, so
the client has to re-issue them when re-registering
- lots of opaques
- no nesting
- if a region has multiple subregions that need different handling
(different callbacks, RAM interspersed with MMIO) then client code has to deal with that manually
- we can't interpose code between a device and global memory handling
To fix that, I propose an new API to replace the existing one:
#ifndef MEMORY_H
#define MEMORY_H
typedef struct MemoryRegionOps MemoryRegionOps;
typedef struct MemoryRegion MemoryRegion;
typedef uint32_t (*MemoryReadFunc)(MemoryRegion *mr, target_phys_addr_t addr); typedef void (*MemoryWriteFunc)(MemoryRegion *mr, target_phys_addr_t addr,
uint32_t data);
struct MemoryRegionOps {
MemoryReadFunc readb, readw, readl;
MemoryWriteFunc writeb, writew, writel;
};struct MemoryRegion {
const MemoryRegionOps *ops;
target_phys_addr_t size;
target_phys_addr_t addr;
};void memory_region_init(MemoryRegion *mr,
target_phys_addr_t size);
void memory_region_init_io(MemoryRegion *mr,
const MemoryRegionOps *ops, target_phys_addr_t size);
void memory_region_init_ram(MemoryRegion *mr,
target_phys_addr_t size);
void memory_region_init_ram_ptr(MemoryRegion *mr,
target_phys_addr_t size,
void *ptr);void memory_region_destroy(MemoryRegion *mr);
void memory_region_set_offset(MemoryRegion *mr, target_phys_addr_t offset); void memory_region_set_log(MemoryRegion *mr, bool log);
void memory_region_clear_coalescing(MemoryRegion *mr);
void memory_region_add_coalescing(MemoryRegion *mr,
target_phys_addr_t offset,
target_phys_addr_t size);void memory_region_add_subregion(MemoryRegion *mr,
target_phys_addr_t offset,
MemoryRegion *subregion);
void memory_region_del_subregion(MemoryRegion *mr,
target_phys_addr_t offset,
MemoryRegion *subregion);void cpu_register_memory_region(MemoryRegion *mr, target_phys_addr_t addr); void cpu_unregister_memory_region(MemoryRegion *mr);
#endif
The API is nested: you can define, say, a PCI BAR containing RAM and MMIO, and give it to the PCI subsystem. PCI can then enable/disable the BAR and move it to different addresses without calling any callbacks; the client code can enable or disable logging or coalescing without caring if the BAR is mapped or not. For example:
MemoryRegion mr, mr_mmio, mr_ram;
memory_region_init(&mr); memory_region_init_io(&mr_mmio, &mmio_ops, 0x1000); memory_region_init_ram(&mr_ram, 0x100000); memory_region_add_subregion(&mr, 0, &mr_ram); memory_region_add_subregion(&mr, 0x10000, &mr_io); memory_region_add_coalescing(&mr_ram, 0, 0x100000); pci_register_bar(&pci_dev, 0, &mr);
at this point the PCI subsystem knows everything about the BAR and can enable or disable it, or move it around, without further help from the device code. On the other hand, the device code can change logging or coalescing, or even change the structure of the region, without caring about whether the region is currently registered or not.
If we can agree on the API, then I think the way forward is to implement it in terms of the old API, change over all devices, then fold the old API into the new one.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 8 lines · Avi Kivity
The current memory APIs (
cpu_register_io_memory,cpu_register_physical_memory) suffer from a number of drawbacks:- lack of centralized bookkeeping
- acpu_register_physical_memory()that overlaps an existing region
will overwrite the preexisting region; but a followingcpu_unregister_physical_memory()will not restore things to status quo ante.
Restoring is not the problem. The problem is that the current API deletes or truncates regions implicitly by overwriting. That makes tracking the layout hard, and it is also error-prone as the device that accidentally overlaps with some other device won't receive a notification of this potential conflict.
Such implicite truncation or deletion must be avoided in a new API, forcing the users to explicitly reference an existing region when dropping or modifying it. But your API goes in the right direction.
Quoted 8 lines
- coalescing and dirty logging are all cleared on unregistering, so
the client has to re-issue them when re-registering
- lots of opaques
- no nesting
- if a region has multiple subregions that need different handling
(different callbacks, RAM interspersed with MMIO) then client code has to deal with that manually
- we can't interpose code between a device and global memory handling
I would add another drawback:
- Inability to identify the origin of a region accesses and handle them differently based on the source.
That is at least problematic for the x86 APIC which is CPU local. Our current way do deal with it is, well, very creative and falls to dust if a guest actually tries to remap the APIC.
However, I'm unsure if that can easily be addressed. As long as only x86 is affected, it's tricky to ask for a big infrastructure to handle this special case. Maybe there some other use cases, don't know.
Quoted 20 lines
To fix that, I propose an new API to replace the existing one:
#ifndef
MEMORY_H
#defineMEMORY_Htypedef struct MemoryRegionOps MemoryRegionOps;
typedef struct MemoryRegion MemoryRegion;typedef
uint32_t(*MemoryReadFunc)(MemoryRegion *mr,target_phys_addr_taddr);
typedef void (*MemoryWriteFunc)(MemoryRegion *mr,target_phys_addr_taddr,uint32_t data);struct MemoryRegionOps { MemoryReadFunc readb, readw, readl; MemoryWriteFunc writeb, writew, writel; };
Quoted 8 lines
struct MemoryRegion { const MemoryRegionOps *ops; target_phys_addr_t size; target_phys_addr_t addr; };void memory_region_init(MemoryRegion *mr,
target_phys_addr_t size);
What use case would this abstract region cover?
Quoted 16 lines
void memory_region_init_io(MemoryRegion *mr,
const MemoryRegionOps *ops,target_phys_addr_tsize);
void memory_region_init_ram(MemoryRegion *mr,target_phys_addr_t size); void memory_region_init_ram_ptr(MemoryRegion *mr, target_phys_addr_t size, void *ptr);void
memory_region_destroy(MemoryRegion *mr);
voidmemory_region_set_offset(MemoryRegion *mr, target_phys_addr_t offset); voidmemory_region_set_log(MemoryRegion *mr, bool log);
voidmemory_region_clear_coalescing(MemoryRegion *mr);
void memory_region_add_coalescing(MemoryRegion *mr,target_phys_addr_t offset, target_phys_addr_t size);
Quoted 8 lines
void memory_region_add_subregion(MemoryRegion *mr,
target_phys_addr_t offset, MemoryRegion *subregion); void memory_region_del_subregion(MemoryRegion *mr, target_phys_addr_t offset, MemoryRegion *subregion);void
cpu_register_memory_region(MemoryRegion *mr, target_phys_addr_t addr);
This could create overlaps. I would suggest to reject them, so we need a return code.
Quoted 9 lines
void
cpu_unregister_memory_region(MemoryRegion *mr);#endif
The API is nested: you can define, say, a PCI BAR containing RAM and MMIO, and give it to the PCI subsystem. PCI can then enable/disable the BAR and move it to different addresses without calling any callbacks; the client code can enable or disable logging or coalescing without caring if the BAR is mapped or not. For example:
Interesting feature.
Quoted 17 lines
MemoryRegion mr,
mr_mmio,mr_ram;memory_region_init(&mr); memory_region_init_io(&mr_mmio, &mmio_ops, 0x1000); memory_region_init_ram(&mr_ram, 0x100000); memory_region_add_subregion(&mr, 0, &mr_ram); memory_region_add_subregion(&mr, 0x10000, &mr_io); memory_region_add_coalescing(&mr_ram, 0, 0x100000); pci_register_bar(&pci_dev, 0, &mr);at this point the PCI subsystem knows everything about the BAR and can enable or disable it, or move it around, without further help from the device code. On the other hand, the device code can change logging or coalescing, or even change the structure of the region, without caring about whether the region is currently registered or not.
Quoted 3 lines
If we can agree on the API, then I think the way forward is to implement it in terms of the old API, change over all devices, then fold the old API into the new one.
There are more aspects that should be clarified before moving forward:
- How to maintain memory regions internally?
- Get rid of wasteful PhysPageDesc at this chance?
- How to hook into the region maintenance (CPUPhysMemoryClient, listening vs. filtering or modifying changes)? How to simplify memory clients this way?
BTW, any old API should be removed ASAP once the new one demonstrated its feasibility. IMHO, we can afford carrying yet another set of legacy interface around.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 16 lines · Jan Kiszka
Quoted 8 lines · Avi Kivity
The current memory APIs (
cpu_register_io_memory,cpu_register_physical_memory) suffer from a number of drawbacks:- lack of centralized bookkeeping
- acpu_register_physical_memory()that overlaps an existing region
will overwrite the preexisting region; but a followingcpu_unregister_physical_memory()will not restore things to status quo ante.Restoring is not the problem. The problem is that the current API deletes or truncates regions implicitly by overwriting. That makes tracking the layout hard, and it is also error-prone as the device that accidentally overlaps with some other device won't receive a notification of this potential conflict.
Quoted 3 lines
Such implicite truncation or deletion must be avoided in a new API, forcing the users to explicitly reference an existing region when dropping or modifying it. But your API goes in the right direction.
It is avoided. The unregistering/deleting APIs do not take a range, just an object. This implies that the range is stored in the object.
The initial implementation will probably have the same problem as it will still be backed by the phys_desc array, but we'll have all of the information in MemoryRegions (we can have a single MemoryRegion that is all of memory) so we can avoid it with a better information.
Quoted 15 lines
Quoted 9 lines
- coalescing and dirty logging are all cleared on unregistering, so
the client has to re-issue them when re-registering
- lots of opaques
- no nesting
- if a region has multiple subregions that need different handling
(different callbacks, RAM interspersed with MMIO) then client code has to deal with that manually
- we can't interpose code between a device and global memory handlingI would add another drawback:
- Inability to identify the origin of a region accesses and handle them differently based on the source.
Quoted 7 lines
That is at least problematic for the x86 APIC which is CPU local. Our current way do deal with it is, well, very creative and falls to dust if a guest actually tries to remap the APIC.
However, I'm unsure if that can easily be addressed. As long as only x86 is affected, it's tricky to ask for a big infrastructure to handle this special case. Maybe there some other use cases, don't know.
We could implement it with a per-cpu MemoryRegion, with each cpu's MemoryRegion populated by a different APIC sub-region.
Quoted 27 lines
Quoted 27 lines
To fix that, I propose an new API to replace the existing one:
#ifndef
MEMORY_H
#defineMEMORY_Htypedef struct MemoryRegionOps MemoryRegionOps;
typedef struct MemoryRegion MemoryRegion;typedef
uint32_t(*MemoryReadFunc)(MemoryRegion *mr,target_phys_addr_taddr);
typedef void (*MemoryWriteFunc)(MemoryRegion *mr,target_phys_addr_taddr,uint32_t data);struct MemoryRegionOps { MemoryReadFunc readb, readw, readl; MemoryWriteFunc writeb, writew, writel;};
struct MemoryRegion { const MemoryRegionOps *ops; target_phys_addr_t size; target_phys_addr_t addr;};
Quoted 4 lines
Quoted 2 lines
void memory_region_init(MemoryRegion *mr,
target_phys_addr_t size);What use case would this abstract region cover?
An empty container, fill it with memory_region_add_subregion().
Quoted 24 lines
Quoted 24 lines
void memory_region_init_io(MemoryRegion *mr,
const MemoryRegionOps *ops,target_phys_addr_tsize);
void memory_region_init_ram(MemoryRegion *mr,target_phys_addr_t size); void memory_region_init_ram_ptr(MemoryRegion *mr, target_phys_addr_t size, void *ptr); void memory_region_destroy(MemoryRegion *mr); void memory_region_set_offset(MemoryRegion *mr, target_phys_addr_t offset); void memory_region_set_log(MemoryRegion *mr, bool log); void memory_region_clear_coalescing(MemoryRegion *mr); void memory_region_add_coalescing(MemoryRegion *mr, target_phys_addr_t offset, target_phys_addr_t size);void memory_region_add_subregion(MemoryRegion *mr,
target_phys_addr_t offset, MemoryRegion *subregion); void memory_region_del_subregion(MemoryRegion *mr, target_phys_addr_t offset, MemoryRegion *subregion);
Quoted 4 lines
Quoted 1 line
void
cpu_register_memory_region(MemoryRegion *mr, target_phys_addr_t addr);This could create overlaps. I would suggest to reject them, so we need a return code.
There is nothing we can do with a return code. You can't fail an mmio that causes overlapping physical memory map.
Quoted 2 lines
Quoted 2 lines
void
cpu_unregister_memory_region(MemoryRegion *mr);
Instead, we need cpu_unregister_memory_region() to restore any previously hidden ranges.
Quoted 28 lines
Quoted 8 lines
#endif
The API is nested: you can define, say, a PCI BAR containing RAM and MMIO, and give it to the PCI subsystem. PCI can then enable/disable the BAR and move it to different addresses without calling any callbacks; the client code can enable or disable logging or coalescing without caring if the BAR is mapped or not. For example:
Interesting feature.
Quoted 17 lines
MemoryRegion mr, mr_mmio, mr_ram; memory_region_init(&mr); memory_region_init_io(&mr_mmio,&mmio_ops, 0x1000); memory_region_init_ram(&mr_ram, 0x100000); memory_region_add_subregion(&mr, 0,&mr_ram); memory_region_add_subregion(&mr, 0x10000,&mr_io); memory_region_add_coalescing(&mr_ram, 0, 0x100000); pci_register_bar(&pci_dev, 0,&mr);at this point the PCI subsystem knows everything about the BAR and can enable or disable it, or move it around, without further help from the device code. On the other hand, the device code can change logging or coalescing, or even change the structure of the region, without caring about whether the region is currently registered or not.
Quoted 6 lines
Quoted 3 lines
If we can agree on the API, then I think the way forward is to implement it in terms of the old API, change over all devices, then fold the old API into the new one.
There are more aspects that should be clarified before moving forward:
- How to maintain memory regions internally?
Not sure what you mean by the question, but my plan was to have the client responsible for allocating the objects (and later use container_of() in the callbacks - note there are no void *s any longer).
Quoted 1 line
- Get rid of wasteful PhysPageDesc at this chance?
That's the intent, but not at this chance, rather later on. But I want the API to be compatible with the goal so we don't have to touch all devices again.
Quoted 3 lines
- How to hook into the region maintenance (CPUPhysMemoryClient, listening vs. filtering or modifying changes)? How to simplify memory clients this way?
I'd leave things as is, at least for the beginning. CPUPhysMemoryClient is global in nature, whereas MemoryRegion is local (offsets are relative to the containing region).
Quoted 4 lines
BTW, any old API should be removed ASAP once the new one demonstrated its feasibility. IMHO, we can afford carrying yet another set of legacy interface around.
That's the plan. New API implemented on top of old API, convert all devices, fold old API into new API.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 9 lines · Avi Kivity
Quoted 9 lines
I would add another drawback:
- Inability to identify the origin of a region accesses and handle them differently based on the source.
That is at least problematic for the x86 APIC which is CPU local. Our current way do deal with it is, well, very creative and falls to dust if a guest actually tries to remap the APIC.
Quoted 6 lines
Quoted 3 lines
However, I'm unsure if that can easily be addressed. As long as only x86 is affected, it's tricky to ask for a big infrastructure to handle this special case. Maybe there some other use cases, don't know.
We could implement it with a per-cpu MemoryRegion, with each cpu's MemoryRegion populated by a different APIC sub-region.
The tricky part is wiring this up efficiently for TCG, ie. in QEMU's softmmu. I played with passing the issuing CPUState (or NULL for devices) down the MMIO handler chain. Not totally beautiful as decentralized dispatching was still required, but at least only moderately invasive. Maybe your API allows for cleaning up the management and dispatching part, need to rethink...
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
To fix that, I propose an new API to replace the existing one:
#ifndef
MEMORY_H
#defineMEMORY_Htypedef struct MemoryRegionOps MemoryRegionOps;
typedef struct MemoryRegion MemoryRegion;typedef
uint32_t(*MemoryReadFunc)(MemoryRegion *mr,target_phys_addr_taddr);
typedef void (*MemoryWriteFunc)(MemoryRegion *mr,target_phys_addr_taddr,uint32_t data);struct MemoryRegionOps { MemoryReadFunc readb, readw, readl; MemoryWriteFunc writeb, writew, writel;};
struct MemoryRegion { const MemoryRegionOps *ops; target_phys_addr_t size; target_phys_addr_t addr;};
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
void memory_region_init(MemoryRegion *mr,
target_phys_addr_t size);What use case would this abstract region cover?
An empty container, fill it with
memory_region_add_subregion().
Yeah, of course.
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
void memory_region_init_io(MemoryRegion *mr,
const MemoryRegionOps *ops,target_phys_addr_tsize);
void memory_region_init_ram(MemoryRegion *mr,target_phys_addr_t size); void memory_region_init_ram_ptr(MemoryRegion *mr, target_phys_addr_t size, void *ptr); void memory_region_destroy(MemoryRegion *mr); void memory_region_set_offset(MemoryRegion *mr, target_phys_addr_t offset); void memory_region_set_log(MemoryRegion *mr, bool log); void memory_region_clear_coalescing(MemoryRegion *mr); void memory_region_add_coalescing(MemoryRegion *mr, target_phys_addr_t offset, target_phys_addr_t size);void memory_region_add_subregion(MemoryRegion *mr,
target_phys_addr_t offset, MemoryRegion *subregion); void memory_region_del_subregion(MemoryRegion *mr, target_phys_addr_t offset, MemoryRegion *subregion);
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
void
cpu_register_memory_region(MemoryRegion *mr, target_phys_addr_t addr);This could create overlaps. I would suggest to reject them, so we need a return code.
There is nothing we can do with a return code. You can't fail an mmio that causes overlapping physical memory map.
We must fail such requests to make progress with the API. That may happen either on caller side or in cpu_register_memory_region itself (hwerror). Otherwise the new API will just be a shiny new facade for on old and still fragile building.
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
void
cpu_unregister_memory_region(MemoryRegion *mr);Instead, we need
cpu_unregister_memory_region()to restore any previously hidden ranges.
I disagree. Both approaches, rejecting overlaps or restoring them, imply subtle semantical changes that exiting device models have to deal with. We can't use any of both without some review and conversion work. So better head for the clearer and, thus, cleaner approach.
Quoted 36 lines
Quoted 29 lines
Quoted 9 lines
#endif
The API is nested: you can define, say, a PCI BAR containing RAM and MMIO, and give it to the PCI subsystem. PCI can then enable/disable the BAR and move it to different addresses without calling any callbacks; the client code can enable or disable logging or coalescing without caring if the BAR is mapped or not. For example:
Interesting feature.
Quoted 17 lines
MemoryRegion mr, mr_mmio, mr_ram; memory_region_init(&mr); memory_region_init_io(&mr_mmio,&mmio_ops, 0x1000); memory_region_init_ram(&mr_ram, 0x100000); memory_region_add_subregion(&mr, 0,&mr_ram); memory_region_add_subregion(&mr, 0x10000,&mr_io); memory_region_add_coalescing(&mr_ram, 0, 0x100000); pci_register_bar(&pci_dev, 0,&mr);at this point the PCI subsystem knows everything about the BAR and can enable or disable it, or move it around, without further help from the device code. On the other hand, the device code can change logging or coalescing, or even change the structure of the region, without caring about whether the region is currently registered or not.
Quoted 6 lines
Quoted 3 lines
If we can agree on the API, then I think the way forward is to implement it in terms of the old API, change over all devices, then fold the old API into the new one.
There are more aspects that should be clarified before moving forward:
- How to maintain memory regions internally?
Quoted 7 lines
Not sure what you mean by the question, but my plan was to have the client responsible for allocating the objects (and later use
container_of()in the callbacks - note there are no void *s any longer).Quoted 1 line
- Get rid of wasteful PhysPageDesc at this chance?
That's the intent, but not at this chance, rather later on.
The features you expose to the users somehow have to be mapped on data structures internally. Those need to support both the registration/deregistration as well as the lookup efficiently. By postponing that internal design to the point when we already switched to facade, the risk arises that a suboptimal interface was exposed and conversion was done in vain.
Quoted 3 lines
But I want
the API to be compatible with the goal so we don't have to touch all devices again.
We can't perform any proper change in the area without touching all users, some a bit more, some only minimally.
Quoted 8 lines
Quoted 4 lines
- How to hook into the region maintenance (CPUPhysMemoryClient, listening vs. filtering or modifying changes)? How to simplify memory clients this way?
I'd leave things as is, at least for the beginning. CPUPhysMemoryClient is global in nature, whereas MemoryRegion is local (offsets are relative to the containing region).
See [1]: We really need to get rid of slot management on CPUPhysMemoryClient side. Your API provides a perfect opportunity to establish the infrastructure of slot tracking at a central place. We can then switch from reporting cpu_registering_memory events to reporting coalesced changes to slots, those slot that also the core uses. So a new CPUPhysMemoryClient API needs to be considered in this API change as well - or we change twice in the end.
Jan
[1] http://thread.gmane.org/gmane.comp.emulators.qemu/102893
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 8 lines · Jan Kiszka
Quoted 2 lines · Avi Kivity
There is nothing we can do with a return code. You can't fail an mmio that causes overlapping physical memory map.
We must fail such requests to make progress with the API. That may happen either on caller side or in
cpu_register_memory_regionitself (hwerror). Otherwise the new API will just be a shiny new facade for on old and still fragile building.
If we don't allow overlapping regions, then how do you implement things like "on startup board maps ROM into lower addresses over top of devices, but later it is unmapped and you can see the underlying devices" ? (You can't currently do this AFAIK, and it would be nice if the new API supported it.)
-- PMM
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Jan Kiszka
Quoted 2 lines · Avi Kivity
There is nothing we can do with a return code. You can't fail an mmio that causes overlapping physical memory map.
We must fail such requests to make progress with the API. That may happen either on caller side or in
cpu_register_memory_regionitself (hwerror). Otherwise the new API will just be a shiny new facade for on old and still fragile building.
Quoted 5 lines
If we don't allow overlapping regions, then how do you implement things like "on startup board maps ROM into lower addresses over top of devices, but later it is unmapped and you can see the underlying devices" ? (You can't currently do this AFAIK, and it would be nice if the new API supported it.)
Right, we can't do this properly, and that's why the attempt if the i440fx chipset model is so horribly broken ATM.
Just allowing overlapping does not solve this problem either. It does not specify what region is on top and what is below (even worse if multiple regions overlap at the place).
We need some managing instance here, and that is e.g. the chipset that provide control over the overlap in reality. It could hook up a PhysMemClient to receive and redirect updates to subregions, or only allow to register them in disabled state.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 20 lines · Jan Kiszka
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Jan Kiszka
Quoted 2 lines · Avi Kivity
There is nothing we can do with a return code. You can't fail an mmio that causes overlapping physical memory map.
We must fail such requests to make progress with the API. That may happen either on caller side or in
cpu_register_memory_regionitself (hwerror). Otherwise the new API will just be a shiny new facade for on old and still fragile building.Quoted 5 lines
If we don't allow overlapping regions, then how do you implement things like "on startup board maps ROM into lower addresses over top of devices, but later it is unmapped and you can see the underlying devices" ? (You can't currently do this AFAIK, and it would be nice if the new API supported it.)
Right, we can't do this properly, and that's why the attempt if the i440fx chipset model is so horribly broken ATM.
Quoted 8 lines
Just allowing overlapping does not solve this problem either. It does not specify what region is on top and what is below (even worse if multiple regions overlap at the place).
We need some managing instance here, and that is e.g. the chipset that provide control over the overlap in reality. It could hook up a PhysMemClient to receive and redirect updates to subregions, or only allow to register them in disabled state.
I think that gets ugly pretty fast. The way this works IRL is that all I/O dispatches pass through the chipset. You literally need something as simple as:
static void i440fx_io_intercept(void *opaque, uint64_t addr, uint32_t value, int size, MemRegion *next)
{
I440FX *s = opaque;
if (range_overlaps(addr, size, PAM_REGION)) {
...
} else {
dispatch_io(next, addr, value, size);
}
}There's no need for an explicit intercept mechanism if you make multiple levels have their own dispatch tables and register progressively larger regions. In fact....
You really don't need to register 90% of the time. In the case of a PC with i440fx, it's really quite simple:
if an I/O is to the APIC page,
it's handled by the APIC
elif the I/O is in ROM regions:
if PAM RE or WE
redirect to RAM appropriately
else:
send to ROMs
elif the I/O is in the PCI windows:
send to the PCI bus
else:
send to the PIIX3
For x86, you could easily completely skip the explicit registration and just have a direct dispatch to the i440fx and implement something slightly more fancy than the above.
And I think this is true for most other types of boards too.
Regards,
Anthony Liguori
Quoted 3 lines
Jan
Quoted 50 lines · Anthony Liguori
Quoted 20 lines · Jan Kiszka
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Jan Kiszka
Quoted 2 lines · Avi Kivity
There is nothing we can do with a return code. You can't fail an mmio that causes overlapping physical memory map.
We must fail such requests to make progress with the API. That may happen either on caller side or in
cpu_register_memory_regionitself (hwerror). Otherwise the new API will just be a shiny new facade for on old and still fragile building.Quoted 5 lines
If we don't allow overlapping regions, then how do you implement things like "on startup board maps ROM into lower addresses over top of devices, but later it is unmapped and you can see the underlying devices" ? (You can't currently do this AFAIK, and it would be nice if the new API supported it.)
Right, we can't do this properly, and that's why the attempt if the i440fx chipset model is so horribly broken ATM.
Quoted 8 lines
Just allowing overlapping does not solve this problem either. It does not specify what region is on top and what is below (even worse if multiple regions overlap at the place).
We need some managing instance here, and that is e.g. the chipset that provide control over the overlap in reality. It could hook up a PhysMemClient to receive and redirect updates to subregions, or only allow to register them in disabled state.
I think that gets ugly pretty fast. The way this works IRL is that all I/O dispatches pass through the chipset. You literally need something as simple as:
static void
i440fx_io_intercept(void *opaque, uint64_t addr, uint32_t value, int size, MemRegion *next){ I440FX *s = opaque; if (range_overlaps(addr, size, PAM_REGION)) { ... } else { dispatch_io(next, addr, value, size); } }There's no need for an explicit intercept mechanism if you make multiple levels have their own dispatch tables and register progressively larger regions. In fact....
Quoted 5 lines
You really don't need to register 90% of the time. In the case of a PC with i440fx, it's really quite simple:
if an I/O is to the APIC page,
it's handled by the APIC
That's not that simple. We need to tell apart:
- if a cpu issued the request, and which one => forward to APIC
- if the range was addressed by a device (PCI or other system bus devices) => forward to MSI or other MMIO handlers
Quoted 10 lines
elif the I/O is in ROM regions:
if PAM RE or WE
redirect to RAM appropriately
else:
send to ROMs
elif the I/O is in the PCI windows:
send to the PCI bus
else:
send to the PIIX3
Quoted 5 lines
For x86, you could easily completely skip the explicit registration and just have a direct dispatch to the i440fx and implement something slightly more fancy than the above.
And I think this is true for most other types of boards too.
This all melts down that we need to stop accepting memory region mappings from everywhere at core level, but properly dispatch them up the device tree. A device should register against its bus which can then forward or handle the mapping directly. And that handling requires a central tool box to avoid reinventing wheels.
That's a worthwhile change, though a much bigger one than I was originally hoping to get away with.
Jan
Quoted 10 lines · Jan Kiszka
Quoted 6 lines · Anthony Liguori
On 05/18/2011 10:30 AM, Jan Kiszka wrote:
You really don't need to register 90% of the time. In the case of a PC with i440fx, it's really quite simple:if an I/O is to the APIC page,
it's handled by the APICThat's not that simple. We need to tell apart:
- if a cpu issued the request, and which one => forward to APIC
Right, but what I'm saying is that this logic lives in kvm-all.c:kvm_run():case EXIT_MMIO.
Obviously for TCG, it's a bit more complicated but this should be handled way before there's any kind of general dispatch.
Quoted 2 lines
- if the range was addressed by a device (PCI or other system bus devices) => forward to MSI or other MMIO handlers
The same is true for MSI. Other MMIO handlers can be handled as appropriate. For instance, once an I/O is sent to the PCI bus, you can walk each PCI device's BAR list to figure out which device owns the I/O event.
For ISA, it's a little trickier since ISA doesn't do positive decoding.
You need each ISA device to declare what I/O addresses it handles.
Quoted 17 lines
Quoted 11 lines
elif the I/O is in ROM regions:
if PAM RE or WE
redirect to RAM appropriately
else:
send to ROMs
elif the I/O is in the PCI windows:
send to the PCI bus
else:
send to the PIIX3Quoted 5 lines
For x86, you could easily completely skip the explicit registration and just have a direct dispatch to the i440fx and implement something slightly more fancy than the above.
And I think this is true for most other types of boards too.
Quoted 8 lines
This all melts down that we need to stop accepting memory region mappings from everywhere at core level, but properly dispatch them up the device tree. A device should register against its bus which can then forward or handle the mapping directly. And that handling requires a central tool box to avoid reinventing wheels.
That's a worthwhile change, though a much bigger one than I was originally hoping to get away with.
Agreed and as long as we've got the right long term vision in mind, we don't have to do everything up front.
Regards,
Anthony Liguori
Quoted 2 lines
Jan
Quoted 21 lines · Anthony Liguori
Quoted 10 lines · Jan Kiszka
Quoted 6 lines · Anthony Liguori
On 05/18/2011 10:30 AM, Jan Kiszka wrote:
You really don't need to register 90% of the time. In the case of a PC with i440fx, it's really quite simple:if an I/O is to the APIC page,
it's handled by the APICThat's not that simple. We need to tell apart:
- if a cpu issued the request, and which one => forward to APICRight, but what I'm saying is that this logic lives in kvm-all.c:kvm_run():case
EXIT_MMIO.Obviously for TCG, it's a bit more complicated but this should be handled way before there's any kind of general dispatch.
Quoted 2 lines
- if the range was addressed by a device (PCI or other system bus devices) => forward to MSI or other MMIO handlers
Quoted 7 lines
The same is true for MSI. Other MMIO handlers can be handled as appropriate. For instance, once an I/O is sent to the PCI bus, you can walk each PCI device's BAR list to figure out which device owns the I/O event.
For ISA, it's a little trickier since ISA doesn't do positive decoding.
You need each ISA device to declare what I/O addresses it handles.
Do we only need to handle CPU based I/O with this API? I would think we would be layering memory regions and implementing them as a hierarchy that reflects the actual hardware layout we're emulating. An access from an I/O device may get a different translation to memory than a CPU (IOMMU). We also might have a system with two VGA devices that both register 0xa0000 with a switch in the chipset that decides which one sees the accesses, just as real hardware does. ISTR a presentation at one of the first KVM forums from you that talked about this type of model. Thanks,
Alex
Quoted 30 lines · Alex Williamson
Quoted 21 lines · Anthony Liguori
Quoted 10 lines · Jan Kiszka
Quoted 6 lines · Anthony Liguori
On 05/18/2011 10:30 AM, Jan Kiszka wrote:
You really don't need to register 90% of the time. In the case of a PC with i440fx, it's really quite simple:if an I/O is to the APIC page,
it's handled by the APICThat's not that simple. We need to tell apart:
- if a cpu issued the request, and which one => forward to APICRight, but what I'm saying is that this logic lives in kvm-all.c:kvm_run():case
EXIT_MMIO.Obviously for TCG, it's a bit more complicated but this should be handled way before there's any kind of general dispatch.
Quoted 2 lines
- if the range was addressed by a device (PCI or other system bus devices) => forward to MSI or other MMIO handlers
Quoted 7 lines
The same is true for MSI. Other MMIO handlers can be handled as appropriate. For instance, once an I/O is sent to the PCI bus, you can walk each PCI device's BAR list to figure out which device owns the I/O event.
For ISA, it's a little trickier since ISA doesn't do positive decoding.
You need each ISA device to declare what I/O addresses it handles.
Quoted 9 lines
Do we only need to handle CPU based I/O with this API? I would think we would be layering memory regions and implementing them as a hierarchy that reflects the actual hardware layout we're emulating. An access from an I/O device may get a different translation to memory than a CPU (IOMMU). We also might have a system with two VGA devices that both register
0xa0000with a switch in the chipset that decides which one sees the accesses, just as real hardware does. ISTR a presentation at one of the first KVM forums from you that talked about this type of model. Thanks,
IIUC, that switch is present in every PCI bridge. It can forward legacy VGA I/O request to its devices or ignore them.
Jan
Quoted 41 lines · Jan Kiszka
Quoted 30 lines · Alex Williamson
Quoted 21 lines · Anthony Liguori
Quoted 10 lines · Jan Kiszka
Quoted 6 lines · Anthony Liguori
On 05/18/2011 10:30 AM, Jan Kiszka wrote:
You really don't need to register 90% of the time. In the case of a PC with i440fx, it's really quite simple:if an I/O is to the APIC page,
it's handled by the APICThat's not that simple. We need to tell apart:
- if a cpu issued the request, and which one => forward to APICRight, but what I'm saying is that this logic lives in kvm-all.c:kvm_run():case
EXIT_MMIO.Obviously for TCG, it's a bit more complicated but this should be handled way before there's any kind of general dispatch.
Quoted 2 lines
- if the range was addressed by a device (PCI or other system bus devices) => forward to MSI or other MMIO handlers
Quoted 7 lines
The same is true for MSI. Other MMIO handlers can be handled as appropriate. For instance, once an I/O is sent to the PCI bus, you can walk each PCI device's BAR list to figure out which device owns the I/O event.
For ISA, it's a little trickier since ISA doesn't do positive decoding.
You need each ISA device to declare what I/O addresses it handles.Quoted 9 lines
Do we only need to handle CPU based I/O with this API? I would think we would be layering memory regions and implementing them as a hierarchy that reflects the actual hardware layout we're emulating. An access from an I/O device may get a different translation to memory than a CPU (IOMMU). We also might have a system with two VGA devices that both register
0xa0000with a switch in the chipset that decides which one sees the accesses, just as real hardware does. ISTR a presentation at one of the first KVM forums from you that talked about this type of model. Thanks,
Quoted 2 lines
IIUC, that switch is present in every PCI bridge. It can forward legacy VGA I/O request to its devices or ignore them.
Right, I just thought it was a good example of how we both need to have decode done in a hierarchy that reflects actual hardware and how we need to support overlapping ranges. Thanks,
Alex
Quoted 9 lines · Anthony Liguori
Quoted 8 lines · Jan Kiszka
Quoted 6 lines · Anthony Liguori
On 05/18/2011 10:30 AM, Jan Kiszka wrote:
You really don't need to register 90% of the time. In the case of a PC with i440fx, it's really quite simple:if an I/O is to the APIC page,
it's handled by the APIC
Quoted 8 lines
Quoted 2 lines
That's not that simple. We need to tell apart:
- if a cpu issued the request, and which one => forward to APICRight, but what I'm saying is that this logic lives in kvm-all.c:kvm_run():case
EXIT_MMIO.Obviously for TCG, it's a bit more complicated but this should be handled way before there's any kind of general dispatch.
Hmm, checking again, I think the APIC should not show up here at all. We really need to filter it out very early at CPU level, i.e. when creating the iotlb (or when dispatching a KVM EXIT_MMIO). It's cpu local, nothing the chipset will ever see.
I really wonder now why I dropped the idea of handling per-cpu regions as a special case in tlb_set_page. It looks trivial, could even be done with a linear per-cpu list before looking at any chipset mappings.
Jan
Quoted 19 lines · Jan Kiszka
Quoted 9 lines · Anthony Liguori
Quoted 8 lines · Jan Kiszka
Quoted 6 lines · Anthony Liguori
On 05/18/2011 10:30 AM, Jan Kiszka wrote:
You really don't need to register 90% of the time. In the case of a PC with i440fx, it's really quite simple:if an I/O is to the APIC page,
it's handled by the APICQuoted 8 lines
Quoted 2 lines
That's not that simple. We need to tell apart:
- if a cpu issued the request, and which one => forward to APICRight, but what I'm saying is that this logic lives in kvm-all.c:kvm_run():case
EXIT_MMIO.Obviously for TCG, it's a bit more complicated but this should be handled way before there's any kind of general dispatch.
Quoted 8 lines
Hmm, checking again, I think the APIC should not show up here at all. We really need to filter it out very early at CPU level, i.e. when creating the iotlb (or when dispatching a KVM
EXIT_MMIO). It's cpu local, nothing the chipset will ever see.I really wonder now why I dropped the idea of handling per-cpu regions as a special case in
tlb_set_page. It looks trivial, could even be done with a linear per-cpu list before looking at any chipset mappings.
Yup, I agree this is the right way to do it.
Regards,
Anthony Liguori
Quoted 3 lines
Jan
Quoted 5 lines · Jan Kiszka
Quoted 2 lines
if an I/O is to the APIC page,
it's handled by the APICThat's not that simple. We need to tell apart:
- if a cpu issued the request, and which one => forward to APIC
And cpu mode may affect where access is forwarded to. If cpu is in SMM mode access to frame buffer may be forwarded to a memory (depends on chipset configuration).
Quoted 3 lines
- if the range was addressed by a device (PCI or other system bus devices) => forward to MSI or other MMIO handlers
--
Gleb.
Quoted 9 lines · Gleb Natapov
Quoted 5 lines · Jan Kiszka
Quoted 2 lines
if an I/O is to the APIC page,
it's handled by the APICThat's not that simple. We need to tell apart:
- if a cpu issued the request, and which one => forward to APICAnd cpu mode may affect where access is forwarded to. If cpu is in SMM mode access to frame buffer may be forwarded to a memory (depends on chipset configuration).
So we have a second use case for CPU-local I/O regions?
I wonder if only a single CPU can enter SMM or if all have to. Right now only the first CPU can switch to that mode, and that affects the behaviour of the chipset /wrt SMRAM mapping. Is that another hack?
Jan
Quoted 7 lines · Jan Kiszka
Quoted 6 lines
Quoted 3 lines
That's not that simple. We need to tell apart:
- if a cpu issued the request, and which one => forward to APICAnd cpu mode may affect where access is forwarded to. If cpu is in SMM mode access to frame buffer may be forwarded to a memory (depends on chipset configuration).
Quoted 5 lines
So we have a second use case for CPU-local I/O regions?
I wonder if only a single CPU can enter SMM or if all have to. Right now only the first CPU can switch to that mode, and that affects the behaviour of the chipset /wrt SMRAM mapping. Is that another hack?
It's a hack. SMM is a per-cpu setting. Effectively it's another address pin - it changes the meaning of (potentially) all addresses.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 8 lines · Avi Kivity
Quoted 7 lines · Jan Kiszka
Quoted 6 lines
Quoted 3 lines
That's not that simple. We need to tell apart:
- if a cpu issued the request, and which one => forward to APICAnd cpu mode may affect where access is forwarded to. If cpu is in SMM mode access to frame buffer may be forwarded to a memory (depends on chipset configuration).
Quoted 8 lines
Quoted 5 lines
So we have a second use case for CPU-local I/O regions?
I wonder if only a single CPU can enter SMM or if all have to. Right now only the first CPU can switch to that mode, and that affects the behaviour of the chipset /wrt SMRAM mapping. Is that another hack?
It's a hack. SMM is a per-cpu setting. Effectively it's another address pin - it changes the meaning of (potentially) all addresses.
Hrm, this may be splitting hairs, but the chipset enables SMM globally.
The processor can enable it by raising a pin but there is only a
single pin.
It may raise/lower the pin on every load/store, but from the chipset's perspective, it's a global setting (at least in the i440fx).
Regards,
Anthony Liguori
Quoted 1 line
Quoted 7 lines · Jan Kiszka
Quoted 6 lines · Gleb Natapov
Quoted 5 lines · Jan Kiszka
Quoted 2 lines
if an I/O is to the APIC page,
it's handled by the APICThat's not that simple. We need to tell apart:
- if a cpu issued the request, and which one => forward to APIC
Quoted 7 lines
Quoted 3 lines
And cpu mode may affect where access is forwarded to. If cpu is in SMM mode access to frame buffer may be forwarded to a memory (depends on chipset configuration).
So we have a second use case for CPU-local I/O regions?
I wonder if only a single CPU can enter SMM or if all have to.
For the i440fx, it's a chipset register (not a per-CPU register).
Maybe it's per-CPU on more modern chipsets... I'm not really sure.
Regards,
Anthony Liguori
Quoted 8 lines · Anthony Liguori
Quoted 3 lines
So we have a second use case for CPU-local I/O regions?
I wonder if only a single CPU can enter SMM or if all have to.
For the i440fx, it's a chipset register (not a per-CPU register).
Maybe it's per-CPU on more modern chipsets... I'm not really sure.
IIRC the chipset register is only used during setup, so you can have access to this memory without entering SMM mode. Once you are in SMM mode, you don't need to touch anything.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 12 lines · Anthony Liguori
Quoted 11 lines · Jan Kiszka
Quoted 9 lines · Gleb Natapov
Quoted 5 lines · Jan Kiszka
Quoted 2 lines
if an I/O is to the APIC page,
it's handled by the APICThat's not that simple. We need to tell apart:
- if a cpu issued the request, and which one => forward to APICAnd cpu mode may affect where access is forwarded to. If cpu is in SMM mode access to frame buffer may be forwarded to a memory (depends on chipset configuration).
Quoted 5 lines
Quoted 3 lines
So we have a second use case for CPU-local I/O regions?
I wonder if only a single CPU can enter SMM or if all have to.
For the i440fx, it's a chipset register (not a per-CPU register).
There are two sources: the chipset register and the mode of the first CPU. Both things were apparently incorrectly merged into the minimalistic i440fx model.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 15 lines · Jan Kiszka
Quoted 14 lines · Anthony Liguori
Quoted 13 lines · Jan Kiszka
Quoted 9 lines · Gleb Natapov
Quoted 5 lines · Jan Kiszka
Quoted 2 lines
if an I/O is to the APIC page,
it's handled by the APICThat's not that simple. We need to tell apart:
- if a cpu issued the request, and which one => forward to APICAnd cpu mode may affect where access is forwarded to. If cpu is in SMM mode access to frame buffer may be forwarded to a memory (depends on chipset configuration).
So we have a second use case for CPU-local I/O regions?
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
I wonder if only a single CPU can enter SMM or if all have to.
For the i440fx, it's a chipset register (not a per-CPU register).
There are two sources: the chipset register and the mode of the first CPU. Both things were apparently incorrectly merged into the minimalistic i440fx model.
Right, the chipset register is mainly used to program the contents of SMM.
There is a single access pin that has effectively the same semantics as setting the chipset register.
It's not a per-CPU setting--that's the point. You can't have one CPU reading SMM memory at the exactly same time as accessing VGA.
But I guess you can never have two simultaneous accesses anyway so perhaps it's splitting hairs :-)
Regards,
Anthony Liguori
Quoted 2 lines
Jan
Quoted 26 lines · Anthony Liguori
Quoted 15 lines · Jan Kiszka
Quoted 14 lines · Anthony Liguori
Quoted 13 lines · Jan Kiszka
Quoted 9 lines · Gleb Natapov
Quoted 5 lines · Jan Kiszka
Quoted 2 lines
if an I/O is to the APIC page,
it's handled by the APICThat's not that simple. We need to tell apart:
- if a cpu issued the request, and which one => forward to APICAnd cpu mode may affect where access is forwarded to. If cpu is in SMM mode access to frame buffer may be forwarded to a memory (depends on chipset configuration).
So we have a second use case for CPU-local I/O regions?
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
I wonder if only a single CPU can enter SMM or if all have to.
For the i440fx, it's a chipset register (not a per-CPU register).
There are two sources: the chipset register and the mode of the first CPU. Both things were apparently incorrectly merged into the minimalistic i440fx model.
Right, the chipset register is mainly used to program the contents of SMM.
Quoted 8 lines
There is a single access pin that has effectively the same semantics as setting the chipset register.
It's not a per-CPU setting--that's the point. You can't have one CPU reading SMM memory at the exactly same time as accessing VGA.
But I guess you can never have two simultaneous accesses anyway so perhaps it's splitting hairs :-)
Not sure. If one CPU enters SMM, it must be able to read SMRAM content, independently of the second CPU - unless there is some magic to stop that CPU in the meantime.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 8 lines · Anthony Liguori
Right, the chipset register is mainly used to program the contents of SMM.
There is a single access pin that has effectively the same semantics as setting the chipset register.
It's not a per-CPU setting--that's the point. You can't have one CPU reading SMM memory at the exactly same time as accessing VGA.
But I guess you can never have two simultaneous accesses anyway so perhaps it's splitting hairs :-)
Exactly - it just works.
btw, a way to implement it would be to have two memory maps, one for SMM and one for non-SMM, and select between them based on the CPU mode.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 8 lines · Avi Kivity
Quoted 7 lines · Anthony Liguori
Right, the chipset register is mainly used to program the contents of SMM.
There is a single access pin that has effectively the same semantics as setting the chipset register.
Quoted 7 lines
Quoted 5 lines
It's not a per-CPU setting--that's the point. You can't have one CPU reading SMM memory at the exactly same time as accessing VGA.
But I guess you can never have two simultaneous accesses anyway so perhaps it's splitting hairs :-)
Exactly - it just works.
Well, not really.
kvm.ko has a global mapping of RAM regions and currently only allows code execution from RAM.
This means the only way for QEMU to enable SMM support is to program the global RAM regions table to enable allow RAM access for the VGA region.
The problem with this is that it's perfectly conceivable to have CPU 0 in SMM mode while CPU 1 is doing MMIO to the VGA planar.
The same problem exists with PAM. It would be much easier to implement PAM correctly in QEMU if it were possible to execute code via MMIO as we could just mark the BIOS memory as non-RAM and deal with the dispatch ourselves.
Would it be fundamentally hard to support this in KVM? I guess you would need to put the VCPU in single step mode and maintain a page to copy the results into.
Regards,
Anthony Liguori
Quoted 4 lines
btw, a way to implement it would be to have two memory maps, one for SMM and one for non-SMM, and select between them based on the CPU mode.
Quoted 25 lines · Anthony Liguori
Quoted 8 lines · Avi Kivity
Quoted 7 lines · Anthony Liguori
Right, the chipset register is mainly used to program the contents of SMM.
There is a single access pin that has effectively the same semantics as setting the chipset register.
Quoted 7 lines
Quoted 5 lines
It's not a per-CPU setting--that's the point. You can't have one CPU reading SMM memory at the exactly same time as accessing VGA.
But I guess you can never have two simultaneous accesses anyway so perhaps it's splitting hairs :-)
Exactly - it just works.
Well, not really.
kvm.ko has a global mapping of RAM regions and currently only allows code execution from RAM.
This means the only way for QEMU to enable SMM support is to program the global RAM regions table to enable allow RAM access for the VGA region.
Quoted 7 lines
The problem with this is that it's perfectly conceivable to have CPU 0 in SMM mode while CPU 1 is doing MMIO to the VGA planar.
The same problem exists with PAM. It would be much easier to implement PAM correctly in QEMU if it were possible to execute code via MMIO as we could just mark the BIOS memory as non-RAM and deal with the dispatch ourselves.
If we already have to change KVM (I guess we have to), let's better add per-CPU memory slot support. That will allow to switch between VGA and SMRAM without costly dispatching. At this chance, I think we also need some support for half-MMIO (MMIO on write, RAM on read) for proper flash support.
Jan
Quoted 17 lines · Jan Kiszka
Quoted 8 lines · Anthony Liguori
Well, not really.
kvm.ko has a global mapping of RAM regions and currently only allows code execution from RAM.
This means the only way for QEMU to enable SMM support is to program the global RAM regions table to enable allow RAM access for the VGA region.
Quoted 7 lines
The problem with this is that it's perfectly conceivable to have CPU 0 in SMM mode while CPU 1 is doing MMIO to the VGA planar.
The same problem exists with PAM. It would be much easier to implement PAM correctly in QEMU if it were possible to execute code via MMIO as we could just mark the BIOS memory as non-RAM and deal with the dispatch ourselves.
Quoted 5 lines
If we already have to change KVM (I guess we have to), let's better add per-CPU memory slot support. That will allow to switch between VGA and SMRAM without costly dispatching. At this chance, I think we also need some support for half-MMIO (MMIO on write, RAM on read) for proper flash support.
This is needed for PAM too.
But RAM isn't mapped per-CPU so this is at best an optimization. You can (and do) execute instructions out of non-RAM memory though. I think if we lifted this restriction in KVM, it would allow us to handle SMRAM/PAM in a more thorough way.
Regards,
Anthony Liguori
Quoted 3 lines
Jan
Quoted 18 lines · Anthony Liguori
Quoted 17 lines · Jan Kiszka
Quoted 15 lines · Anthony Liguori
Well, not really.
kvm.ko has a global mapping of RAM regions and currently only allows code execution from RAM.
This means the only way for QEMU to enable SMM support is to program the global RAM regions table to enable allow RAM access for the VGA region.
The problem with this is that it's perfectly conceivable to have CPU 0 in SMM mode while CPU 1 is doing MMIO to the VGA planar.
The same problem exists with PAM. It would be much easier to implement PAM correctly in QEMU if it were possible to execute code via MMIO as we could just mark the BIOS memory as non-RAM and deal with the dispatch ourselves.
Quoted 7 lines
Quoted 5 lines
If we already have to change KVM (I guess we have to), let's better add per-CPU memory slot support. That will allow to switch between VGA and SMRAM without costly dispatching. At this chance, I think we also need some support for half-MMIO (MMIO on write, RAM on read) for proper flash support.
This is needed for PAM too.
Yeah, right, there were some to-do comments I strictly ignored
Quoted 2 lines
But RAM isn't mapped per-CPU so this is at best an optimization.
SMRAM is mapped per CPU, depending on the execution mode. That is the point.
Quoted 4 lines
You
can (and do) execute instructions out of non-RAM memory though. I think if we lifted this restriction in KVM, it would allow us to handle SMRAM/PAM in a more thorough way.
I do not disagree that having such feature would be nice for certain corner cases. But executing code by jumping to user space on every instruction fetch, maybe just to dispatch between SMRAM and normal MMIO on a per-CPU base, will give you horrible performance. That can at best be an intermediate step, though I bet it's better to address both at roughly the same time.
Jan
Quoted 17 lines · Anthony Liguori
Quoted 8 lines · Avi Kivity
Quoted 7 lines · Anthony Liguori
Right, the chipset register is mainly used to program the contents of SMM.
There is a single access pin that has effectively the same semantics as setting the chipset register.
Quoted 7 lines
Quoted 5 lines
It's not a per-CPU setting--that's the point. You can't have one CPU reading SMM memory at the exactly same time as accessing VGA.
But I guess you can never have two simultaneous accesses anyway so perhaps it's splitting hairs :-)
Exactly - it just works.
Quoted 7 lines
Well, not really.
kvm.ko has a global mapping of RAM regions and currently only allows code execution from RAM.
This means the only way for QEMU to enable SMM support is to program the global RAM regions table to enable allow RAM access for the VGA region.
The problem with this is that it's perfectly conceivable to have CPU 0 in SMM mode while CPU 1 is doing MMIO to the VGA planar.
kvm needs updates to support SMM; I already outlined them several months ago.
Quoted 2 lines
The same problem exists with PAM.
PAM is a completely different problem. The changes are global and fit kvm slot management.
Quoted 3 lines
It would be much easier to implement PAM correctly in QEMU if it were possible to execute code via MMIO as we could just mark the BIOS memory as non-RAM and deal with the dispatch ourselves.
Would it be fundamentally hard to support this in KVM? I guess you would need to put the VCPU in single step mode and maintain a page to copy the results into.
You need to emulate everything. We're probably not far from that. However there may be a significant performance loss.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 5 lines · Jan Kiszka
Quoted 4 lines · Gleb Natapov
Quoted 3 lines · Jan Kiszka
Quoted 2 lines
if an I/O is to the APIC page,
it's handled by the APIC
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
That's not that simple. We need to tell apart:
- if a cpu issued the request, and which one => forward to APICAnd cpu mode may affect where access is forwarded to. If cpu is in SMM mode access to frame buffer may be forwarded to a memory (depends on chipset configuration).
So we have a second use case for CPU-local I/O regions?
SuperSparc MXCC (memory cache controller) should be CPU specific. Currently we handle this for accesses via ASI, but the registers could be mapped with MMU and then the ASI-less access would not be handled.
Another case would be the cache-as-ram mode for some x86 CPUs, which Coreboot people would like to see IIRC.
Quoted 2 lines · Blue Swirl
Another case would be the cache-as-ram mode for some x86 CPUs, which Coreboot people would like to see IIRC.
That's probably best handled as a cache emulation layer, as this is not associated with any specific address range.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 46 lines · Anthony Liguori
Quoted 20 lines · Jan Kiszka
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Jan Kiszka
Quoted 2 lines · Avi Kivity
There is nothing we can do with a return code. You can't fail an mmio that causes overlapping physical memory map.
We must fail such requests to make progress with the API. That may happen either on caller side or in
cpu_register_memory_regionitself (hwerror). Otherwise the new API will just be a shiny new facade for on old and still fragile building.Quoted 5 lines
If we don't allow overlapping regions, then how do you implement things like "on startup board maps ROM into lower addresses over top of devices, but later it is unmapped and you can see the underlying devices" ? (You can't currently do this AFAIK, and it would be nice if the new API supported it.)
Right, we can't do this properly, and that's why the attempt if the i440fx chipset model is so horribly broken ATM.
Quoted 8 lines
Just allowing overlapping does not solve this problem either. It does not specify what region is on top and what is below (even worse if multiple regions overlap at the place).
We need some managing instance here, and that is e.g. the chipset that provide control over the overlap in reality. It could hook up a PhysMemClient to receive and redirect updates to subregions, or only allow to register them in disabled state.
I think that gets ugly pretty fast. The way this works IRL is that all I/O dispatches pass through the chipset. You literally need something as simple as:
static void
i440fx_io_intercept(void *opaque, uint64_t addr, uint32_t value, int size, MemRegion *next){ I440FX *s = opaque; if (range_overlaps(addr, size, PAM_REGION)) { ... } else { dispatch_io(next, addr, value, size); } }
Quoted 3 lines
There's no need for an explicit intercept mechanism if you make multiple levels have their own dispatch tables and register progressively larger regions. In fact....
Actually, things are a bit more complicated: This layer has to properly adopt the coalescing properties of underlying regions or we cause performance regressions to VGA emulation. That means it has to register dispatching slots of the corresponding size and set the coalescing flag accordingly. And it likely need to adjust them as the regions below change.
IOW, I don't think we get away with that simple approach above but still require to track mapping via a PhysMemClient. But we should be able to avoid filtering by adding overlapping regions with higher prio.
Jan
Quoted 18 lines · Jan Kiszka
Quoted 13 lines · Anthony Liguori
On 05/18/2011 10:30 AM, Jan Kiszka wrote:
static voidi440fx_io_intercept(void *opaque, uint64_t addr, uint32_t value, int size, MemRegion *next){ I440FX *s = opaque; if (range_overlaps(addr, size, PAM_REGION)) { ... } else { dispatch_io(next, addr, value, size); } }Quoted 3 lines
There's no need for an explicit intercept mechanism if you make multiple levels have their own dispatch tables and register progressively larger regions. In fact....
Quoted 5 lines
Actually, things are a bit more complicated: This layer has to properly adopt the coalescing properties of underlying regions or we cause performance regressions to VGA emulation. That means it has to register dispatching slots of the corresponding size and set the coalescing flag accordingly. And it likely need to adjust them as the regions below change.
As I mentioned in another thread, I don't think we want to "design" coalescing into the API. Coalescing is something that breaks through abstractions layers and is really just a hack.
Regards,
Anthony Liguori
Quoted 6 lines
IOW, I don't think we get away with that simple approach above but still require to track mapping via a PhysMemClient. But we should be able to avoid filtering by adding overlapping regions with higher prio.
Jan
Quoted 26 lines · Anthony Liguori
Quoted 18 lines · Jan Kiszka
Quoted 13 lines · Anthony Liguori
On 05/18/2011 10:30 AM, Jan Kiszka wrote:
static voidi440fx_io_intercept(void *opaque, uint64_t addr, uint32_t value, int size, MemRegion *next){ I440FX *s = opaque; if (range_overlaps(addr, size, PAM_REGION)) { ... } else { dispatch_io(next, addr, value, size); } }Quoted 3 lines
There's no need for an explicit intercept mechanism if you make multiple levels have their own dispatch tables and register progressively larger regions. In fact....
Quoted 6 lines
Actually, things are a bit more complicated: This layer has to properly adopt the coalescing properties of underlying regions or we cause performance regressions to VGA emulation. That means it has to register dispatching slots of the corresponding size and set the coalescing flag accordingly. And it likely need to adjust them as the regions below change.
Quoted 3 lines
As I mentioned in another thread, I don't think we want to "design" coalescing into the API. Coalescing is something that breaks through abstractions layers and is really just a hack.
We depend on it for speed, and even if it's nothing to be found in real HW, we need to design it in as this example demonstrates. The pain of not doing it, specifically when we go for hierarchical management, will be much higher.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 8 lines · Anthony Liguori
Quoted 5 lines
Actually, things are a bit more complicated: This layer has to properly adopt the coalescing properties of underlying regions or we cause performance regressions to VGA emulation. That means it has to register dispatching slots of the corresponding size and set the coalescing flag accordingly. And it likely need to adjust them as the regions below change.
As I mentioned in another thread, I don't think we want to "design" coalescing into the API. Coalescing is something that breaks through abstractions layers and is really just a hack.
It's impossible not to design it into the API. The layer which wants to do coalescing (the device) has no idea if and where its memory is mapped.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 9 lines · Avi Kivity
Quoted 8 lines · Anthony Liguori
Quoted 6 lines
Actually, things are a bit more complicated: This layer has to properly adopt the coalescing properties of underlying regions or we cause performance regressions to VGA emulation. That means it has to register dispatching slots of the corresponding size and set the coalescing flag accordingly. And it likely need to adjust them as the regions below change.
Quoted 6 lines
Quoted 3 lines
As I mentioned in another thread, I don't think we want to "design" coalescing into the API. Coalescing is something that breaks through abstractions layers and is really just a hack.
It's impossible not to design it into the API. The layer which wants to do coalescing (the device) has no idea if and where its memory is mapped.
There's two places coalescing currently matters: VGA and PCI devices. Since VGA is just a special PCI device, let's just focus on PCI devices.
The PCI bus knows exactly where the memory is mapped. Yes, if you have complex IOMMU hierarchies it doesn't, but this is my point. We don't need to design coalesced IO to support these sort of complex hierarchies.
Regards,
Anthony Liguori
Quoted 14 lines · Anthony Liguori
Quoted 13 lines · Avi Kivity
Quoted 11 lines · Anthony Liguori
Quoted 6 lines
Actually, things are a bit more complicated: This layer has to properly adopt the coalescing properties of underlying regions or we cause performance regressions to VGA emulation. That means it has to register dispatching slots of the corresponding size and set the coalescing flag accordingly. And it likely need to adjust them as the regions below change.
As I mentioned in another thread, I don't think we want to "design" coalescing into the API. Coalescing is something that breaks through abstractions layers and is really just a hack.
Quoted 5 lines
Quoted 2 lines
It's impossible not to design it into the API. The layer which wants to do coalescing (the device) has no idea if and where its memory is mapped.
There's two places coalescing currently matters: VGA and PCI devices. Since VGA is just a special PCI device, let's just focus on PCI devices.
Every frame buffer device, PCI or not, benefits from it. Don't focus on PCI or x86.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 6 lines · Jan Kiszka
Quoted 3 lines
There's two places coalescing currently matters: VGA and PCI devices. Since VGA is just a special PCI device, let's just focus on PCI devices.
Every frame buffer device, PCI or not, benefits from it. Don't focus on PCI or x86.
Actually, only framebuffers that can't be mapped as RAM benefit from it. The only one I know of is VGA in planar mode (but there may be others).
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 14 lines · Anthony Liguori
Quoted 13 lines · Avi Kivity
Quoted 11 lines · Anthony Liguori
Quoted 6 lines
Actually, things are a bit more complicated: This layer has to properly adopt the coalescing properties of underlying regions or we cause performance regressions to VGA emulation. That means it has to register dispatching slots of the corresponding size and set the coalescing flag accordingly. And it likely need to adjust them as the regions below change.
As I mentioned in another thread, I don't think we want to "design" coalescing into the API. Coalescing is something that breaks through abstractions layers and is really just a hack.
Quoted 6 lines
Quoted 2 lines
It's impossible not to design it into the API. The layer which wants to do coalescing (the device) has no idea if and where its memory is mapped.
There's two places coalescing currently matters: VGA and PCI devices. Since VGA is just a special PCI device, let's just focus on PCI devices.
The PCI bus knows exactly where the memory is mapped.
The PCI bus doesn't know about coalescing. Only the device does. Look at e1000 for an example. Cirrus also enables/disables coalescing dynamically.
Quoted 2 lines
Yes, if you have complex IOMMU hierarchies it doesn't, but this is my point. We don't need to design coalesced IO to support these sort of complex hierarchies.
Are we aiming at our own feet again?
Look at the API, it adds three functions that non-users need not care about.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 47 lines · Anthony Liguori
Quoted 20 lines · Jan Kiszka
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Jan Kiszka
Quoted 2 lines · Avi Kivity
There is nothing we can do with a return code. You can't fail an mmio that causes overlapping physical memory map.
We must fail such requests to make progress with the API. That may happen either on caller side or in
cpu_register_memory_regionitself (hwerror). Otherwise the new API will just be a shiny new facade for on old and still fragile building.Quoted 5 lines
If we don't allow overlapping regions, then how do you implement things like "on startup board maps ROM into lower addresses over top of devices, but later it is unmapped and you can see the underlying devices" ? (You can't currently do this AFAIK, and it would be nice if the new API supported it.)
Right, we can't do this properly, and that's why the attempt if the i440fx chipset model is so horribly broken ATM.
Quoted 8 lines
Just allowing overlapping does not solve this problem either. It does not specify what region is on top and what is below (even worse if multiple regions overlap at the place).
We need some managing instance here, and that is e.g. the chipset that provide control over the overlap in reality. It could hook up a PhysMemClient to receive and redirect updates to subregions, or only allow to register them in disabled state.
I think that gets ugly pretty fast. The way this works IRL is that all I/O dispatches pass through the chipset. You literally need something as simple as:
static void
i440fx_io_intercept(void *opaque, uint64_t addr, uint32_t value, int size, MemRegion *next){ I440FX *s = opaque; if (range_overlaps(addr, size, PAM_REGION)) { ... } else { dispatch_io(next, addr, value, size); } }There's no need for an explicit intercept mechanism if you make multiple levels have their own dispatch tables and register progressively larger regions. In fact....
You really don't need to register 90% of the time. In the case of a PC with i440fx, it's really quite simple:
Quoted 12 lines
if an I/O is to the APIC page,
it's handled by the APIC
elif the I/O is in ROM regions:
if PAM RE or WE
redirect to RAM appropriately
else:
send to ROMs
elif the I/O is in the PCI windows:
send to the PCI bus
else:
send to the PIIX3
Memory regions do the exact same thing, except at registration time. The nested if/elif/else tree is captured in the nested region tree.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 17 lines · Jan Kiszka
Quoted 9 lines · Avi Kivity
Quoted 9 lines
I would add another drawback:
- Inability to identify the origin of a region accesses and handle them differently based on the source.
That is at least problematic for the x86 APIC which is CPU local. Our current way do deal with it is, well, very creative and falls to dust if a guest actually tries to remap the APIC.
Quoted 6 lines
Quoted 3 lines
However, I'm unsure if that can easily be addressed. As long as only x86 is affected, it's tricky to ask for a big infrastructure to handle this special case. Maybe there some other use cases, don't know.
We could implement it with a per-cpu MemoryRegion, with each cpu's MemoryRegion populated by a different APIC sub-region.
Quoted 6 lines
The tricky part is wiring this up efficiently for TCG, ie. in QEMU's softmmu. I played with passing the issuing CPUState (or NULL for devices) down the MMIO handler chain. Not totally beautiful as decentralized dispatching was still required, but at least only moderately invasive. Maybe your API allows for cleaning up the management and dispatching part, need to rethink...
My suggestion is opposite - have a different MemoryRegion for each (e.g. CPUState::memory). Then the TLBs will resolve to a different ram_addr_t for the same physical address, for the local APIC range.
Quoted 12 lines
Quoted 3 lines
There is nothing we can do with a return code. You can't fail an mmio that causes overlapping physical memory map.
We must fail such requests to make progress with the API. That may happen either on caller side or in
cpu_register_memory_regionitself (hwerror). Otherwise the new API will just be a shiny new facade for on old and still fragile building.Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
void cpu_unregister_memory_region(MemoryRegion *mr);
Quoted 7 lines
Quoted 2 lines
Instead, we need
cpu_unregister_memory_region()to restore any previously hidden ranges.I disagree. Both approaches, rejecting overlaps or restoring them, imply subtle semantical changes that exiting device models have to deal with. We can't use any of both without some review and conversion work. So better head for the clearer and, thus, cleaner approach.
We need to head for the more hardware-like approach. What happens when you program overlapping BARs? I imagine the result is implementation-defined, but ends up with one region decoded in preference to the other. There is simply no way to reject an overlapping mapping.
Quoted 10 lines
Quoted 3 lines
Quoted 1 line
- Get rid of wasteful PhysPageDesc at this chance?
That's the intent, but not at this chance, rather later on.
The features you expose to the users somehow have to be mapped on data structures internally. Those need to support both the registration/deregistration as well as the lookup efficiently. By postponing that internal design to the point when we already switched to facade, the risk arises that a suboptimal interface was exposed and conversion was done in vain.
That is true, and one of the reasons this is posted as an [RFC] (I usually prefer [PATCH]). However, I don't see a way to change the internals without changing the API.
Quoted 6 lines
Quoted 3 lines
But I want
the API to be compatible with the goal so we don't have to touch all devices again.We can't perform any proper change in the area without touching all users, some a bit more, some only minimally.
Right. I want to touch the devices *once*, to covert them to this API, then work on the internals.
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines
- How to hook into the region maintenance (CPUPhysMemoryClient, listening vs. filtering or modifying changes)? How to simplify memory clients this way?
I'd leave things as is, at least for the beginning. CPUPhysMemoryClient is global in nature, whereas MemoryRegion is local (offsets are relative to the containing region).
Quoted 7 lines
See [1]: We really need to get rid of slot management on CPUPhysMemoryClient side. Your API provides a perfect opportunity to establish the infrastructure of slot tracking at a central place. We can then switch from reporting
cpu_registering_memoryevents to reporting coalesced changes to slots, those slot that also the core uses. So a new CPUPhysMemoryClient API needs to be considered in this API change as well - or we change twice in the end.
The kvm memory slots (and hopefully future qemu memory slots) are a flattened view of the MemoryRegion tree. There is no 1:1 mapping.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 25 lines · Avi Kivity
Quoted 17 lines · Jan Kiszka
Quoted 9 lines · Avi Kivity
Quoted 9 lines
I would add another drawback:
- Inability to identify the origin of a region accesses and handle them differently based on the source.
That is at least problematic for the x86 APIC which is CPU local. Our current way do deal with it is, well, very creative and falls to dust if a guest actually tries to remap the APIC.
Quoted 6 lines
Quoted 3 lines
However, I'm unsure if that can easily be addressed. As long as only x86 is affected, it's tricky to ask for a big infrastructure to handle this special case. Maybe there some other use cases, don't know.
We could implement it with a per-cpu MemoryRegion, with each cpu's MemoryRegion populated by a different APIC sub-region.
Quoted 6 lines
The tricky part is wiring this up efficiently for TCG, ie. in QEMU's softmmu. I played with passing the issuing CPUState (or NULL for devices) down the MMIO handler chain. Not totally beautiful as decentralized dispatching was still required, but at least only moderately invasive. Maybe your API allows for cleaning up the management and dispatching part, need to rethink...
Quoted 3 lines
My suggestion is opposite - have a different MemoryRegion for each (e.g.
CPUState::memory). Then the TLBs will resolve to a differentram_addr_tfor the same physical address, for the local APIC range.
Need to recheck, but I think I dropped that idea due to invasiveness of the implementation. Not a very good argument, but without a clear picture how useful this per-cpu registration is beyond x86, that was more important.
Quoted 21 lines
Quoted 13 lines
Quoted 4 lines
There is nothing we can do with a return code. You can't fail an mmio that causes overlapping physical memory map.
We must fail such requests to make progress with the API. That may happen either on caller side or in
cpu_register_memory_regionitself (hwerror). Otherwise the new API will just be a shiny new facade for on old and still fragile building.Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
void cpu_unregister_memory_region(MemoryRegion *mr);Quoted 7 lines
Quoted 2 lines
Instead, we need
cpu_unregister_memory_region()to restore any previously hidden ranges.I disagree. Both approaches, rejecting overlaps or restoring them, imply subtle semantical changes that exiting device models have to deal with. We can't use any of both without some review and conversion work. So better head for the clearer and, thus, cleaner approach.
Quoted 5 lines
We need to head for the more hardware-like approach. What happens when you program overlapping BARs? I imagine the result is implementation-defined, but ends up with one region decoded in preference to the other. There is simply no way to reject an overlapping mapping.
But there is also now simple way to allow them. At least not without exposing control about their ordering AND allowing to hook up managing code (e.g. of the PCI bridge or the chipset) that controls registrations.
...
Quoted 10 lines
Quoted 7 lines
See [1]: We really need to get rid of slot management on CPUPhysMemoryClient side. Your API provides a perfect opportunity to establish the infrastructure of slot tracking at a central place. We can then switch from reporting
cpu_registering_memoryevents to reporting coalesced changes to slots, those slot that also the core uses. So a new CPUPhysMemoryClient API needs to be considered in this API change as well - or we change twice in the end.The kvm memory slots (and hopefully future qemu memory slots) are a flattened view of the MemoryRegion tree. There is no 1:1 mapping.
We need a flatted view of your memory regions during runtime as well. No major difference here. If we share that view with PhysMemClients, they can drop most of their creative slot tracking algorithms, focusing on the real differences.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 10 lines · Jan Kiszka
Quoted 6 lines
We need to head for the more hardware-like approach. What happens when you program overlapping BARs? I imagine the result is implementation-defined, but ends up with one region decoded in preference to the other. There is simply no way to reject an overlapping mapping.
But there is also now simple way to allow them. At least not without exposing control about their ordering AND allowing to hook up managing code (e.g. of the PCI bridge or the chipset) that controls registrations.
What about memory_region_add_subregion(..., int priority) as I suggested in another message?
Regarding bridges, every registration request flows through them so they already have full control.
Quoted 9 lines
...
Quoted 8 lines
Quoted 7 lines
See [1]: We really need to get rid of slot management on CPUPhysMemoryClient side. Your API provides a perfect opportunity to establish the infrastructure of slot tracking at a central place. We can then switch from reporting
cpu_registering_memoryevents to reporting coalesced changes to slots, those slot that also the core uses. So a new CPUPhysMemoryClient API needs to be considered in this API change as well - or we change twice in the end.
Quoted 7 lines
Quoted 2 lines
The kvm memory slots (and hopefully future qemu memory slots) are a flattened view of the MemoryRegion tree. There is no 1:1 mapping.
We need a flatted view of your memory regions during runtime as well. No major difference here. If we share that view with PhysMemClients, they can drop most of their creative slot tracking algorithms, focusing on the real differences.
We'll definitely have a flattened view (phys_desc is such a flattened view, hopefully we'll have a better one).
We can basically run a tree walk on each change, emitting ranges in order and sending them to PhysMemClients.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 8 lines · Avi Kivity
Quoted 7 lines · Jan Kiszka
Quoted 6 lines
We need to head for the more hardware-like approach. What happens when you program overlapping BARs? I imagine the result is implementation-defined, but ends up with one region decoded in preference to the other. There is simply no way to reject an overlapping mapping.
Quoted 6 lines
Quoted 3 lines
But there is also now simple way to allow them. At least not without exposing control about their ordering AND allowing to hook up managing code (e.g. of the PCI bridge or the chipset) that controls registrations.
What about
memory_region_add_subregion(..., int priority)as I suggested in another message?
That's fine, but also requires a change how, or better where devices register their regions.
Quoted 3 lines
Regarding bridges, every registration request flows through them so they already have full control.
Not everything is PCI, we also have ISA e.g. If we were able to route such requests also through a hierarchy of abstract bridges, then even better.
Quoted 13 lines
Quoted 13 lines
...
Quoted 10 lines
Quoted 7 lines
See [1]: We really need to get rid of slot management on CPUPhysMemoryClient side. Your API provides a perfect opportunity to establish the infrastructure of slot tracking at a central place. We can then switch from reporting
cpu_registering_memoryevents to reporting coalesced changes to slots, those slot that also the core uses. So a new CPUPhysMemoryClient API needs to be considered in this API change as well - or we change twice in the end.The kvm memory slots (and hopefully future qemu memory slots) are a flattened view of the MemoryRegion tree. There is no 1:1 mapping.
Quoted 7 lines
Quoted 4 lines
We need a flatted view of your memory regions during runtime as well. No major difference here. If we share that view with PhysMemClients, they can drop most of their creative slot tracking algorithms, focusing on the real differences.
We'll definitely have a flattened view (
phys_descis such a flattened view, hopefully we'll have a better one).
phys_desc is not exportable. If we try (and we do from time to time...), we end up with more slots than clients like kvm will ever be able to handle.
Quoted 3 lines
We can basically run a tree walk on each change, emitting ranges in order and sending them to PhysMemClients.
I'm specifically thinking of fully trackable slot updates. The clients should not have to maintain the flat layout. They should just receive updates in the form of slot X added/modified/removed. For now, this magic happens multiple times in the clients, and that is very bad.
Given that not only memory clients need that view but that ever TLB miss (in TCG mode) requires to identify the effective slot as well, it might be worth preparing a runtime structure at registration time that supports this efficiently - but this time without wasting memory.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 13 lines · Jan Kiszka
Quoted 12 lines · Avi Kivity
Quoted 10 lines · Jan Kiszka
Quoted 6 lines
We need to head for the more hardware-like approach. What happens when you program overlapping BARs? I imagine the result is implementation-defined, but ends up with one region decoded in preference to the other. There is simply no way to reject an overlapping mapping.
But there is also now simple way to allow them. At least not without exposing control about their ordering AND allowing to hook up managing code (e.g. of the PCI bridge or the chipset) that controls registrations.
Quoted 5 lines
Quoted 2 lines
What about
memory_region_add_subregion(..., int priority)as I suggested in another message?That's fine, but also requires a change how, or better where devices register their regions.
Lost you - please elaborate.
Quoted 7 lines
Quoted 3 lines
Regarding bridges, every registration request flows through them so they already have full control.
Not everything is PCI, we also have ISA e.g. If we were able to route such requests also through a hierarchy of abstract bridges, then even better.
Yes, it's a tree of nested MemoryRegions.
Quoted 5 lines
Quoted 2 lines
We'll definitely have a flattened view (
phys_descis such a flattened view, hopefully we'll have a better one).
phys_descis not exportable. If we try (and we do from time to time...), we end up with more slots than clients like kvm will ever be able to handle.
If we coalesce ajacent phys_descs we end up with a minimal representation. Of course that's not the most efficient implementation (a tree walk is better).
Quoted 8 lines
Quoted 3 lines
We can basically run a tree walk on each change, emitting ranges in order and sending them to PhysMemClients.
I'm specifically thinking of fully trackable slot updates. The clients should not have to maintain the flat layout. They should just receive updates in the form of slot X added/modified/removed. For now, this magic happens multiple times in the clients, and that is very bad.
Slots don't have any meaning. You can have a RAM region which is overlaid by a smaller mmio region -> the RAM slot is split into two.
We should just send clients a list of ranges, and they can associate them with slots themselves.
Quoted 4 lines
Given that not only memory clients need that view but that ever TLB miss (in TCG mode) requires to identify the effective slot as well, it might be worth preparing a runtime structure at registration time that supports this efficiently - but this time without wasting memory.
Yes. Won't be easy though. Perhaps a perfect hash table for small regions and a sorted-by-size array for large regions.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 14 lines · Avi Kivity
Quoted 13 lines · Jan Kiszka
Quoted 12 lines · Avi Kivity
Quoted 10 lines · Jan Kiszka
Quoted 6 lines
We need to head for the more hardware-like approach. What happens when you program overlapping BARs? I imagine the result is implementation-defined, but ends up with one region decoded in preference to the other. There is simply no way to reject an overlapping mapping.
But there is also now simple way to allow them. At least not without exposing control about their ordering AND allowing to hook up managing code (e.g. of the PCI bridge or the chipset) that controls registrations.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
What about
memory_region_add_subregion(..., int priority)as I suggested in another message?That's fine, but also requires a change how, or better where devices register their regions.
Lost you - please elaborate.
Devices currently register against the core, that's nothing your API automatically changes (though it lays the foundation to do so). But devices should rather register against the corresponding bus. The bus (ie. the device managing it) could then forward the request, stick it into a subregion, or have it for dinner.
However, we are yet in troubles if we want to change that because devices can only be on one bus - at least so far.
...
Quoted 10 lines
Quoted 4 lines
I'm specifically thinking of fully trackable slot updates. The clients should not have to maintain the flat layout. They should just receive updates in the form of slot X added/modified/removed. For now, this magic happens multiple times in the clients, and that is very bad.
Slots don't have any meaning. You can have a RAM region which is overlaid by a smaller mmio region -> the RAM slot is split into two.
We should just send clients a list of ranges, and they can associate them with slots themselves.
And that association logic should be as simple as matching a unique range ID against an existing slot (for updates and deletions) or adding a new slot for a new range and storing the ID. Anything else will not allow to simplify the existing code bases noticeably. That's my point.
Quoted 8 lines
Quoted 5 lines
Given that not only memory clients need that view but that ever TLB miss (in TCG mode) requires to identify the effective slot as well, it might be worth preparing a runtime structure at registration time that supports this efficiently - but this time without wasting memory.
Yes. Won't be easy though. Perhaps a perfect hash table for small regions and a sorted-by-size array for large regions.
Keep in mind that TCG will be our benchmark for these changes as it requires much more accesses than KVM. We must avoid designing the new infrastructure with exclusive focus on KVM (and also x86).
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 11 lines · Jan Kiszka
Quoted 5 lines
Quoted 3 lines
That's fine, but also requires a change how, or better where devices register their regions.
Lost you - please elaborate.
Devices currently register against the core, that's nothing your API automatically changes (though it lays the foundation to do so). But devices should rather register against the corresponding bus. The bus (ie. the device managing it) could then forward the request, stick it into a subregion, or have it for dinner.
Yes. We'd change pci_register_bar() to accept a MemoryRegion.
Quoted 2 lines
However, we are yet in troubles if we want to change that because devices can only be on one bus - at least so far.
Nothing prohibits a device from calling pci_register_bar() for one region and some other API for another.
btw, another motivation for this API is for dual ISA/PCI devices. This way most of the work is bus agnostic, with just the actual registration being bus specific.
Quoted 9 lines
...
Quoted 8 lines
Quoted 4 lines
I'm specifically thinking of fully trackable slot updates. The clients should not have to maintain the flat layout. They should just receive updates in the form of slot X added/modified/removed. For now, this magic happens multiple times in the clients, and that is very bad.
Slots don't have any meaning. You can have a RAM region which is overlaid by a smaller mmio region -> the RAM slot is split into two.
Quoted 7 lines
Quoted 2 lines
We should just send clients a list of ranges, and they can associate them with slots themselves.
And that association logic should be as simple as matching a unique range ID against an existing slot (for updates and deletions) or adding a new slot for a new range and storing the ID. Anything else will not allow to simplify the existing code bases noticeably. That's my point.
We won't have a natural ID. But I'll see if I can have a library calculate the minimum difference between the previous layout and current layout. Should not be too hard.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Given that not only memory clients need that view but that ever TLB miss (in TCG mode) requires to identify the effective slot as well, it might be worth preparing a runtime structure at registration time that supports this efficiently - but this time without wasting memory.
Quoted 6 lines
Quoted 2 lines
Yes. Won't be easy though. Perhaps a perfect hash table for small regions and a sorted-by-size array for large regions.
Keep in mind that TCG will be our benchmark for these changes as it requires much more accesses than KVM. We must avoid designing the new infrastructure with exclusive focus on KVM (and also x86).
I think TCG will not be affected much since it stores ram_addr_t's in its TLBs, so the lookup is only done on a TLB miss. In fact there's a change for a slight win since we'll probably be able to keep the new layout in cache compared to phys_desc.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 11 lines · Avi Kivity
Quoted 11 lines
Quoted 10 lines
Quoted 4 lines
I'm specifically thinking of fully trackable slot updates. The clients should not have to maintain the flat layout. They should just receive updates in the form of slot X added/modified/removed. For now, this magic happens multiple times in the clients, and that is very bad.
Slots don't have any meaning. You can have a RAM region which is overlaid by a smaller mmio region -> the RAM slot is split into two.
We should just send clients a list of ranges, and they can associate them with slots themselves.
Quoted 6 lines
Quoted 4 lines
And that association logic should be as simple as matching a unique range ID against an existing slot (for updates and deletions) or adding a new slot for a new range and storing the ID. Anything else will not allow to simplify the existing code bases noticeably. That's my point.
We won't have a natural ID.
The address of the data structure describing a region could be such a thing. Provided, of course, we prepare a flatted view ahead of time, not on the fly.
Quoted 3 lines
But I'll see if I can have a library
calculate the minimum difference between the previous layout and current layout. Should not be too hard.
We need exact match on the client side with the old range. E.g. if you put a new region over an existing one, effectively splitting it into two halves, the core has to
- shrink the existing range to form the first half
- register two new ranges to reflect the rest
On unregistering of that overlap, we need the reverse. So all the client has to do then is to decide if it is interested in some range, and then store it internally with some additional data (and process it, of course). No more merging, no more overlap detection and splitting up at client level.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 17 lines · Jan Kiszka
Quoted 16 lines · Avi Kivity
Quoted 15 lines
Quoted 10 lines
Quoted 4 lines
I'm specifically thinking of fully trackable slot updates. The clients should not have to maintain the flat layout. They should just receive updates in the form of slot X added/modified/removed. For now, this magic happens multiple times in the clients, and that is very bad.
Slots don't have any meaning. You can have a RAM region which is
overlaid by a smaller mmio region -> the RAM slot is split into two.We should just send clients a list of ranges, and they can associate them with slots themselves.
And that association logic should be as simple as matching a unique range ID against an existing slot (for updates and deletions) or adding a new slot for a new range and storing the ID. Anything else will not allow to simplify the existing code bases noticeably. That's my point.
Quoted 5 lines
Quoted 1 line
We won't have a natural ID.
The address of the data structure describing a region could be such a thing. Provided, of course, we prepare a flatted view ahead of time, not on the fly.
It will change as soon as the memory map changes.
Quoted 10 lines
Quoted 3 lines
But I'll see if I can have a library
calculate the minimum difference between the previous layout and current layout. Should not be too hard.We need exact match on the client side with the old range. E.g. if you put a new region over an existing one, effectively splitting it into two halves, the core has to
- shrink the existing range to form the first half
- register two new ranges to reflect the rest
Quoted 5 lines
On unregistering of that overlap, we need the reverse. So all the client has to do then is to decide if it is interested in some range, and then store it internally with some additional data (and process it, of course). No more merging, no more overlap detection and splitting up at client level.
Right. We do a symmetric set difference between the old and new maps and let the client know what has changed.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 18 lines · Avi Kivity
Quoted 17 lines · Jan Kiszka
Quoted 16 lines · Avi Kivity
Quoted 15 lines
Quoted 10 lines
Quoted 4 lines
I'm specifically thinking of fully trackable slot updates. The clients should not have to maintain the flat layout. They should just receive updates in the form of slot X added/modified/removed. For now, this magic happens multiple times in the clients, and that is very bad.
Slots don't have any meaning. You can have a RAM region which is
overlaid by a smaller mmio region -> the RAM slot is split into two.We should just send clients a list of ranges, and they can associate them with slots themselves.
And that association logic should be as simple as matching a unique range ID against an existing slot (for updates and deletions) or adding a new slot for a new range and storing the ID. Anything else will not allow to simplify the existing code bases noticeably. That's my point.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
We won't have a natural ID.
The address of the data structure describing a region could be such a thing. Provided, of course, we prepare a flatted view ahead of time, not on the fly.
It will change as soon as the memory map changes.
...which is supposed to be properly reported to the client beforehand.
Quoted 11 lines
Quoted 11 lines
Quoted 4 lines
But I'll see if I can have a library
calculate the minimum difference between the previous layout and current layout. Should not be too hard.We need exact match on the client side with the old range. E.g. if you put a new region over an existing one, effectively splitting it into two halves, the core has to
- shrink the existing range to form the first half
- register two new ranges to reflect the rest
Quoted 8 lines
Quoted 5 lines
On unregistering of that overlap, we need the reverse. So all the client has to do then is to decide if it is interested in some range, and then store it internally with some additional data (and process it, of course). No more merging, no more overlap detection and splitting up at client level.
Right. We do a symmetric set difference between the old and new maps and let the client know what has changed.
That would be fine as well. Then the internal representation could be anything, though I bet a flattened form would have its advantages.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 1 line · Avi Kivity
Yes. We'd change
pci_register_bar()to accept a MemoryRegion.
Surely this detail would be hidden on the pci_dev->bus?
Quoted 4 lines
Quoted 2 lines
However, we are yet in troubles if we want to change that because devices can only be on one bus - at least so far.
Nothing prohibits a device from calling
pci_register_bar()for one region and some other API for another.
Sure, but the majority of PCI devices are plain pci, and that sort of complexity should be hidden by default.
r~
Quoted 4 lines · Richard Henderson
Quoted 1 line · Avi Kivity
Yes. We'd change
pci_register_bar()to accept a MemoryRegion.Surely this detail would be hidden on the
pci_dev->bus?
Not sure what you mean.
The reason I want pci_register_bar() to accept a memory region is that some BARs are composed of multiple subregions, for example cirrus has a RAM framebuffer an an mmio region in one BAR. So the device describes the region relative to its start point and hands it off to the pci subsystem, which can then enable or disable the region, and place it anywhere in the bus address space it likes.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
However, we are yet in troubles if we want to change that because devices can only be on one bus - at least so far.
Nothing prohibits a device from calling
pci_register_bar()for one region and some other API for another.Sure, but the majority of PCI devices are plain pci, and that sort of complexity should be hidden by default.
There is pci_register_bar_simple().
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 8 lines · Avi Kivity
Quoted 7 lines · Jan Kiszka
Quoted 6 lines
We need to head for the more hardware-like approach. What happens when you program overlapping BARs? I imagine the result is implementation-defined, but ends up with one region decoded in preference to the other. There is simply no way to reject an overlapping mapping.
Quoted 6 lines
Quoted 3 lines
But there is also now simple way to allow them. At least not without exposing control about their ordering AND allowing to hook up managing code (e.g. of the PCI bridge or the chipset) that controls registrations.
What about
memory_region_add_subregion(..., int priority)as I suggested in another message?
Haven't saw another message yet, but how caller knows about priority?
--
Gleb.
Quoted 9 lines · Gleb Natapov
Quoted 8 lines · Avi Kivity
Quoted 7 lines · Jan Kiszka
Quoted 6 lines
We need to head for the more hardware-like approach. What happens when you program overlapping BARs? I imagine the result is implementation-defined, but ends up with one region decoded in preference to the other. There is simply no way to reject an overlapping mapping.
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
But there is also now simple way to allow them. At least not without exposing control about their ordering AND allowing to hook up managing code (e.g. of the PCI bridge or the chipset) that controls registrations.
What about
memory_region_add_subregion(..., int priority)as I suggested in another message?Haven't saw another message yet, but how caller knows about priority?
The caller is emulating some hub or router and should decide on priority like real hardware.
For example, piix gives higher priority to the vga window over RAM.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 14 lines · Avi Kivity
Quoted 13 lines · Gleb Natapov
Quoted 12 lines · Avi Kivity
Quoted 10 lines · Jan Kiszka
Quoted 6 lines
We need to head for the more hardware-like approach. What happens when you program overlapping BARs? I imagine the result is implementation-defined, but ends up with one region decoded in preference to the other. There is simply no way to reject an overlapping mapping.
But there is also now simple way to allow them. At least not without exposing control about their ordering AND allowing to hook up managing code (e.g. of the PCI bridge or the chipset) that controls registrations.
Quoted 9 lines
Quoted 3 lines
Quoted 2 lines
What about
memory_region_add_subregion(..., int priority)as I suggested in another message?Haven't saw another message yet, but how caller knows about priority?
The caller is emulating some hub or router and should decide on priority like real hardware.
For example, piix gives higher priority to the vga window over RAM.
Hmm, but if a caller of the memory_region_add_subregion() function is a device itself how does it know about chipset priorities. All it wants to tell to the system is that it is ready to handle mmio access in this phys range, but chipset may decide to forward those accesses elsewhere.
--
Gleb.
Quoted 22 lines · Gleb Natapov
Quoted 21 lines · Avi Kivity
Quoted 9 lines · Gleb Natapov
Quoted 8 lines · Avi Kivity
Quoted 7 lines · Jan Kiszka
Quoted 6 lines
We need to head for the more hardware-like approach. What happens when you program overlapping BARs? I imagine the result is implementation-defined, but ends up with one region decoded in preference to the other. There is simply no way to reject an overlapping mapping.
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
But there is also now simple way to allow them. At least not without exposing control about their ordering AND allowing to hook up managing code (e.g. of the PCI bridge or the chipset) that controls registrations.
What about
memory_region_add_subregion(..., int priority)as I suggested in another message?Haven't saw another message yet, but how caller knows about priority?
The caller is emulating some hub or router and should decide on priority like real hardware.
Quoted 6 lines
Quoted 2 lines
For example, piix gives higher priority to the vga window over RAM.
Hmm, but if a caller of the
memory_region_add_subregion()function is a device itself how does it know about chipset priorities. All it wants to tell to the system is that it is ready to handle mmio access in this phys range, but chipset may decide to forward those accesses elsewhere.
In this case the device would call a chipset function, passing the memory region as a parameter, and the chipset would call m_r_add_subregion(). Alternatively, the chipset can instantiate the device (if it is an embedded one) and call m_r_add_subregion() itself.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 25 lines · Avi Kivity
Quoted 24 lines · Gleb Natapov
Quoted 23 lines · Avi Kivity
Quoted 9 lines · Gleb Natapov
Quoted 8 lines · Avi Kivity
Quoted 7 lines · Jan Kiszka
Quoted 6 lines
We need to head for the more hardware-like approach. What happens when you program overlapping BARs? I imagine the result is implementation-defined, but ends up with one region decoded in preference to the other. There is simply no way to reject an overlapping mapping.
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
But there is also now simple way to allow them. At least not without exposing control about their ordering AND allowing to hook up managing code (e.g. of the PCI bridge or the chipset) that controls registrations.
What about
memory_region_add_subregion(..., int priority)as I suggested in another message?Haven't saw another message yet, but how caller knows about priority?
The caller is emulating some hub or router and should decide on priority like real hardware.
For example, piix gives higher priority to the vga window over RAM.
Quoted 8 lines
Quoted 4 lines
Hmm, but if a caller of the
memory_region_add_subregion()function is a device itself how does it know about chipset priorities. All it wants to tell to the system is that it is ready to handle mmio access in this phys range, but chipset may decide to forward those accesses elsewhere.In this case the device would call a chipset function, passing the memory region as a parameter, and the chipset would call
m_r_add_subregion().
But then chipset can resolve all overlapping by itself and register only regions that are actually accessible by a guest software. Also there are devices that on some architectures are accessed through a chipset and on other they resides directly on a system bus. If they will need to call different memory registration api depending on how they are instantiated the code can become messy.
Quoted 4 lines
Alternatively, the chipset can instantiate the
device (if it is an embedded one) and callm_r_add_subregion()itself.
--
Gleb.
Quoted 34 lines · Gleb Natapov
Quoted 25 lines · Avi Kivity
Quoted 24 lines · Gleb Natapov
Quoted 23 lines · Avi Kivity
Quoted 9 lines · Gleb Natapov
Quoted 8 lines · Avi Kivity
Quoted 7 lines · Jan Kiszka
Quoted 6 lines
We need to head for the more hardware-like approach. What happens when you program overlapping BARs? I imagine the result is implementation-defined, but ends up with one region decoded in preference to the other. There is simply no way to reject an overlapping mapping.
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
But there is also now simple way to allow them. At least not without exposing control about their ordering AND allowing to hook up managing code (e.g. of the PCI bridge or the chipset) that controls registrations.
What about
memory_region_add_subregion(..., int priority)as I suggested in another message?Haven't saw another message yet, but how caller knows about priority?
The caller is emulating some hub or router and should decide on priority like real hardware.
For example, piix gives higher priority to the vga window over RAM.
Quoted 8 lines
Quoted 4 lines
Hmm, but if a caller of the
memory_region_add_subregion()function is a device itself how does it know about chipset priorities. All it wants to tell to the system is that it is ready to handle mmio access in this phys range, but chipset may decide to forward those accesses elsewhere.In this case the device would call a chipset function, passing the memory region as a parameter, and the chipset would call
m_r_add_subregion().
Quoted 6 lines
But then chipset can resolve all overlapping by itself and register only regions that are actually accessible by a guest software. Also there are devices that on some architectures are accessed through a chipset and on other they resides directly on a system bus. If they will need to call different memory registration api depending on how they are instantiated the code can become messy.
Devices shall register their regions with the bus. Every device is on some bus, so that's not a problem. And we can then provide registration handlers at bus level that either implement specific logic or just forward the request to the next hierarchy level (default handler).
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 42 lines · Jan Kiszka
Quoted 34 lines · Gleb Natapov
Quoted 25 lines · Avi Kivity
Quoted 24 lines · Gleb Natapov
Quoted 23 lines · Avi Kivity
Quoted 9 lines · Gleb Natapov
Quoted 8 lines · Avi Kivity
Quoted 7 lines · Jan Kiszka
Quoted 6 lines
We need to head for the more hardware-like approach. What happens when you program overlapping BARs? I imagine the result is implementation-defined, but ends up with one region decoded in preference to the other. There is simply no way to reject an overlapping mapping.
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
But there is also now simple way to allow them. At least not without exposing control about their ordering AND allowing to hook up managing code (e.g. of the PCI bridge or the chipset) that controls registrations.
What about
memory_region_add_subregion(..., int priority)as I suggested in another message?Haven't saw another message yet, but how caller knows about priority?
The caller is emulating some hub or router and should decide on priority like real hardware.
For example, piix gives higher priority to the vga window over RAM.
Quoted 8 lines
Quoted 4 lines
Hmm, but if a caller of the
memory_region_add_subregion()function is a device itself how does it know about chipset priorities. All it wants to tell to the system is that it is ready to handle mmio access in this phys range, but chipset may decide to forward those accesses elsewhere.In this case the device would call a chipset function, passing the memory region as a parameter, and the chipset would call
m_r_add_subregion().Quoted 6 lines
But then chipset can resolve all overlapping by itself and register only regions that are actually accessible by a guest software. Also there are devices that on some architectures are accessed through a chipset and on other they resides directly on a system bus. If they will need to call different memory registration api depending on how they are instantiated the code can become messy.
Quoted 5 lines
Devices shall register their regions with the bus. Every device is on some bus, so that's not a problem. And we can then provide registration handlers at bus level that either implement specific logic or just forward the request to the next hierarchy level (default handler).
Yes, I agree with that. I just don't see the need for "priority" parameter in this model.
--
Gleb.
Quoted 8 lines · Gleb Natapov
Quoted 6 lines
Devices shall register their regions with the bus. Every device is on some bus, so that's not a problem. And we can then provide registration handlers at bus level that either implement specific logic or just forward the request to the next hierarchy level (default handler).
Yes, I agree with that. I just don't see the need for "priority" parameter in this model.
Priority allows you to register RAM from 0-EOM and overlay it with the ROM and VGA windows as necessary. It also allows PCI to override RAM (or vice versa, however we decide).
Sure, you can let the caller chop the various regions manually in the first case, but it's just extra work that can be done in common code. And it cannot be done at all for the second case, if RAM overrides PCI (the PCI bus doesn't know how to chop BARs).
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 10 lines · Avi Kivity
Quoted 8 lines · Gleb Natapov
Quoted 6 lines
Devices shall register their regions with the bus. Every device is on some bus, so that's not a problem. And we can then provide registration handlers at bus level that either implement specific logic or just forward the request to the next hierarchy level (default handler).
Yes, I agree with that. I just don't see the need for "priority" parameter in this model.
Quoted 9 lines
Priority allows you to register RAM from 0-EOM and overlay it with the ROM and VGA windows as necessary. It also allows PCI to override RAM (or vice versa, however we decide).
Sure, you can let the caller chop the various regions manually in the first case, but it's just extra work that can be done in common code. And it cannot be done at all for the second case, if RAM overrides PCI (the PCI bus doesn't know how to chop BARs).
In the model described by Jan no device or bus (except system bus) will call memory API directly. Instead each device will call memory registration function of its bus and the bus will call memory registration function of a device that provides the bus and so on up to the system bus (maps nicely to qdev!). Each level knows by itself (and only it really knows!) how to solve the overlapping, but it can't provide meaningful priority to the layer above since only layer above knows what are relative priority between siblings (e.g only chipset knows that it will forward access to particular address to RAM and not PCI, no matter that PCI bus asked for priority 100 and RAM asked for 0).
So it boils down to this: since chopping will have to happen at each level of the hierarchy I do not see why the lowest level of the hierarchy should be made special.
--
Gleb.
Quoted 43 lines · Gleb Natapov
Quoted 42 lines · Jan Kiszka
Quoted 40 lines · Gleb Natapov
Quoted 25 lines · Avi Kivity
Quoted 24 lines · Gleb Natapov
Quoted 23 lines · Avi Kivity
Quoted 9 lines · Gleb Natapov
Quoted 8 lines · Avi Kivity
Quoted 7 lines · Jan Kiszka
Quoted 6 lines
We need to head for the more hardware-like approach. What happens when you program overlapping BARs? I imagine the result is implementation-defined, but ends up with one region decoded in preference to the other. There is simply no way to reject an overlapping mapping.
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
But there is also now simple way to allow them. At least not without exposing control about their ordering AND allowing to hook up managing code (e.g. of the PCI bridge or the chipset) that controls registrations.
What about
memory_region_add_subregion(..., int priority)as I suggested in another message?Haven't saw another message yet, but how caller knows about priority?
The caller is emulating some hub or router and should decide on priority like real hardware.
For example, piix gives higher priority to the vga window over RAM.
Quoted 8 lines
Quoted 4 lines
Hmm, but if a caller of the
memory_region_add_subregion()function is a device itself how does it know about chipset priorities. All it wants to tell to the system is that it is ready to handle mmio access in this phys range, but chipset may decide to forward those accesses elsewhere.In this case the device would call a chipset function, passing the memory region as a parameter, and the chipset would call
m_r_add_subregion().But then chipset can resolve all overlapping by itself and register only regions that are actually accessible by a guest software. Also there are devices that on some architectures are accessed through a chipset and on other they resides directly on a system bus. If they will need to call different memory registration api depending on how they are instantiated the code can become messy.
Quoted 7 lines
Quoted 5 lines
Devices shall register their regions with the bus. Every device is on some bus, so that's not a problem. And we can then provide registration handlers at bus level that either implement specific logic or just forward the request to the next hierarchy level (default handler).
Yes, I agree with that. I just don't see the need for "priority" parameter in this model.
"Controlled" overlapping can indeed simplify the implementation of a dispatcher that wants to flip between putting some region logically over an existing, possibly complex layout and disabling this overlay. For that purpose, the overlay requires a high prio than what is below (using a default priority).
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 6 lines · Gleb Natapov
Quoted 4 lines
In this case the device would call a chipset function, passing the memory region as a parameter, and the chipset would call
m_r_add_subregion().But then chipset can resolve all overlapping by itself and register only regions that are actually accessible by a guest software.
Sure it can (and it does now), but it's hard. This API centralizes the logic, leaving the devices/chipsets to specify what they want.
For a PC, we have at least two such cases, the ISA bus and the PCI bus.
Quoted 5 lines
Also there are
devices that on some architectures are accessed through a chipset and on other they resides directly on a system bus. If they will need to call different memory registration api depending on how they are instantiated the code can become messy.
An example is ne2000-isa and ne2000-pci. There's no getting around some glue logic, but I think this API minimizes it (you can declare everything about your memory region in common code, the only thing that is different is registration).
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 19 lines · Avi Kivity
Quoted 6 lines · Gleb Natapov
Quoted 4 lines
In this case the device would call a chipset function, passing the memory region as a parameter, and the chipset would call
m_r_add_subregion().But then chipset can resolve all overlapping by itself and register only regions that are actually accessible by a guest software.
Sure it can (and it does now), but it's hard. This API centralizes the logic, leaving the devices/chipsets to specify what they want.
For a PC, we have at least two such cases, the ISA bus and the PCI bus.
Quoted 5 lines
Also there are
devices that on some architectures are accessed through a chipset and on other they resides directly on a system bus. If they will need to call different memory registration api depending on how they are instantiated the code can become messy.
Quoted 4 lines
An example is ne2000-isa and ne2000-pci. There's no getting around some glue logic, but I think this API minimizes it (you can declare everything about your memory region in common code, the only thing that is different is registration).
If devices register against the corresponding qbus (I expect we'll have multiple ones in the future), not even that need to be different.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 8 lines · Jan Kiszka
Quoted 5 lines
An example is ne2000-isa and ne2000-pci. There's no getting around some glue logic, but I think this API minimizes it (you can declare everything about your memory region in common code, the only thing that is different is registration).
If devices register against the corresponding qbus (I expect we'll have multiple ones in the future), not even that need to be different.
Eventually we may make the memory API a sub-API of qdev. I don't want to start with that however, the change is large enough already.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 7 lines · Avi Kivity
Quoted 6 lines · Jan Kiszka
Quoted 5 lines
An example is ne2000-isa and ne2000-pci. There's no getting around some glue logic, but I think this API minimizes it (you can declare everything about your memory region in common code, the only thing that is different is registration).
Quoted 5 lines
Quoted 2 lines
If devices register against the corresponding qbus (I expect we'll have multiple ones in the future), not even that need to be different.
Eventually we may make the memory API a sub-API of qdev. I don't want to start with that however, the change is large enough already.
Touching all devices again at that point to change the way they register regions may not be the best approach. I would try to consolidate the refactoring work that affects the majority of device models.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 7 lines · Jan Kiszka
Quoted 3 lines
Eventually we may make the memory API a sub-API of qdev. I don't want to start with that however, the change is large enough already.
Touching all devices again at that point to change the way they register regions may not be the best approach. I would try to consolidate the refactoring work that affects the majority of device models.
The risk is that the entire work will be stalled if it requires too much effort.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 11 lines · Avi Kivity
Quoted 7 lines · Jan Kiszka
Quoted 3 lines
Eventually we may make the memory API a sub-API of qdev. I don't want to start with that however, the change is large enough already.
Touching all devices again at that point to change the way they register regions may not be the best approach. I would try to consolidate the refactoring work that affects the majority of device models.
The risk is that the entire work will be stalled if it requires too much effort.
Then we could still switch one gear down, converting at least some exemplary devices completely to demonstrate that the API changes fit into the big picture.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 10 lines · Jan Kiszka
Quoted 9 lines · Avi Kivity
Quoted 7 lines · Jan Kiszka
Quoted 3 lines
Eventually we may make the memory API a sub-API of qdev. I don't want to start with that however, the change is large enough already.
Touching all devices again at that point to change the way they register regions may not be the best approach. I would try to consolidate the refactoring work that affects the majority of device models.
Quoted 6 lines
Quoted 2 lines
The risk is that the entire work will be stalled if it requires too much effort.
Then we could still switch one gear down, converting at least some exemplary devices completely to demonstrate that the API changes fit into the big picture.
My plan is:
- post RFC v1 with updated API in patch form
- RFC v2 with implementation + significant fraction of PC devices coverted
- PATCH v3 with full conversion an elimination of the old API
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 18 lines · Avi Kivity
Quoted 10 lines · Jan Kiszka
Quoted 9 lines · Avi Kivity
Quoted 7 lines · Jan Kiszka
Quoted 3 lines
Eventually we may make the memory API a sub-API of qdev. I don't want to start with that however, the change is large enough already.
Touching all devices again at that point to change the way they register regions may not be the best approach. I would try to consolidate the refactoring work that affects the majority of device models.
Quoted 6 lines
Quoted 2 lines
The risk is that the entire work will be stalled if it requires too much effort.
Then we could still switch one gear down, converting at least some exemplary devices completely to demonstrate that the API changes fit into the big picture.
Quoted 5 lines
My plan is:
- post RFC v1 with updated API in patch form
- RFC v2 with implementation + significant fraction of PC devices coverted
- PATCH v3 with full conversion an elimination of the old API
And when introducing hierarchical registration, we will have to go through all of this once again. Plus the API may have to be changed again if it does not fulfill all requirements of the hierarchical region management. And we have no proof that it allows an efficient core implementation.
Before touching any device at all, what about building the infrastructure to manage and address memory regions hierarchically via qdev first? That could be started on a green field, then applied to the PC architecture, and finally rolled out for all.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 25 lines · Jan Kiszka
Quoted 18 lines · Avi Kivity
Quoted 10 lines · Jan Kiszka
Quoted 9 lines · Avi Kivity
Quoted 7 lines · Jan Kiszka
Quoted 3 lines
Eventually we may make the memory API a sub-API of qdev. I don't want to start with that however, the change is large enough already.
Touching all devices again at that point to change the way they register regions may not be the best approach. I would try to consolidate the refactoring work that affects the majority of device models.
Quoted 6 lines
Quoted 2 lines
The risk is that the entire work will be stalled if it requires too much effort.
Then we could still switch one gear down, converting at least some exemplary devices completely to demonstrate that the API changes fit into the big picture.
Quoted 5 lines
My plan is:
- post RFC v1 with updated API in patch form
- RFC v2 with implementation + significant fraction of PC devices coverted
- PATCH v3 with full conversion an elimination of the old API
Quoted 5 lines
And when introducing hierarchical registration, we will have to go through all of this once again. Plus the API may have to be changed again if it does not fulfill all requirements of the hierarchical region management. And we have no proof that it allows an efficient core implementation.
This API *is* hierarchical registration. v2 will (hopefully) prove that it can be done efficiently.
What is missing is to make qdev and this API the same hierarchy.
Quoted 4 lines
Before touching any device at all, what about building the infrastructure to manage and address memory regions hierarchically via qdev first? That could be started on a green field, then applied to the PC architecture, and finally rolled out for all.
I feel a lot less comfortable about it since it introduces more variables. It also means a full conversion is impossible.
While doing it in one pass reduces the total effort, it increases the risk IMO.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 26 lines · Avi Kivity
Quoted 25 lines · Jan Kiszka
Quoted 23 lines · Avi Kivity
Quoted 10 lines · Jan Kiszka
Quoted 9 lines · Avi Kivity
Quoted 7 lines · Jan Kiszka
Quoted 3 lines
Eventually we may make the memory API a sub-API of qdev. I don't want to start with that however, the change is large enough already.
Touching all devices again at that point to change the way they register regions may not be the best approach. I would try to consolidate the refactoring work that affects the majority of device models.
Quoted 6 lines
Quoted 2 lines
The risk is that the entire work will be stalled if it requires too much effort.
Then we could still switch one gear down, converting at least some exemplary devices completely to demonstrate that the API changes fit into the big picture.
My plan is:
- post RFC v1 with updated API in patch form
- RFC v2 with implementation + significant fraction of PC devices coverted
- PATCH v3 with full conversion an elimination of the old API
Quoted 8 lines
Quoted 5 lines
And when introducing hierarchical registration, we will have to go through all of this once again. Plus the API may have to be changed again if it does not fulfill all requirements of the hierarchical region management. And we have no proof that it allows an efficient core implementation.
This API *is* hierarchical registration. v2 will (hopefully) prove that it can be done efficiently.
It may supports it, but most users will not use it like this. That comes with consequent qdev integration. PCI is just an exception here as it already provides some instantiation services.
Quoted 8 lines
What is missing is to make qdev and this API the same hierarchy.
Quoted 4 lines
Before touching any device at all, what about building the infrastructure to manage and address memory regions hierarchically via qdev first? That could be started on a green field, then applied to the PC architecture, and finally rolled out for all.
Quoted 5 lines
I feel a lot less comfortable about it since it introduces more variables. It also means a full conversion is impossible.
While doing it in one pass reduces the total effort, it increases the risk IMO.
If we are sure we can reasonably evolve from this conversion level and it will still fit the final model, then we may take that route. If you want to go that way, OK, let's see what v2 will bring. We should use the meantime and try to further develop the long-term APIs.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 4 lines · Jan Kiszka
Quoted 4 lines
I feel a lot less comfortable about it since it introduces more variables. It also means a full conversion is impossible.
Quoted 8 lines
Quoted 2 lines
While doing it in one pass reduces the total effort, it increases the risk IMO.
If we are sure we can reasonably evolve from this conversion level and it will still fit the final model, then we may take that route. If you want to go that way, OK, let's see what v2 will bring. We should use the meantime and try to further develop the long-term APIs.
Yes. This way people who are familiar with qdev and non-x86 can look at the API and the partial conversion and see if something is missing.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 11 lines · Avi Kivity
Quoted 7 lines · Jan Kiszka
On 2011-05-19 15:07, Avi Kivity wrote:
And when introducing hierarchical registration, we will have to go through all of this once again. Plus the API may have to be changed again if it does not fulfill all requirements of the hierarchical region management. And we have no proof that it allows an efficient core implementation.
This API *is* hierarchical registration. v2 will (hopefully) prove that it can be done efficiently.
We also need hierarchical dispatch. Priorities are just a weak attempt to emulate hierarchical dispatch but I don't think there's an improvement over a single dispatch table.
Hierarchical dispatch is simpler. You just need a simple list at each bus.
I don't see a strong need to tie anything to qdev here FWIW.
Regards,
Anthony Liguori
Quoted 10 lines · Anthony Liguori
Quoted 9 lines · Avi Kivity
Quoted 7 lines · Jan Kiszka
On 2011-05-19 15:07, Avi Kivity wrote:
And when introducing hierarchical registration, we will have to go through all of this once again. Plus the API may have to be changed again if it does not fulfill all requirements of the hierarchical region management. And we have no proof that it allows an efficient core implementation.
Quoted 7 lines
Quoted 2 lines
This API *is* hierarchical registration. v2 will (hopefully) prove that it can be done efficiently.
We also need hierarchical dispatch. Priorities are just a weak attempt to emulate hierarchical dispatch but I don't think there's an improvement over a single dispatch table.
Hierarchical dispatch is simpler. You just need a simple list at each bus.
The API itself says nothing about whether the hierarchy is evaluated at run-time or registration time. We could easily have the implementation walk the memory hierarchy to dispatch an mmio.
However, RAM cannot be dispatched this way (we need to resolve which ranges are RAM when the regions are registered, not accessed) so a data structure that contains all of the information is mandatory.
Since the first implementation will use the existing cpu_physical_register_memory() API as a back-end, we'll end up with a flattened dispatch model (which I think is the right thing anyway).
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 19 lines · Avi Kivity
Quoted 18 lines · Anthony Liguori
Quoted 10 lines · Avi Kivity
Quoted 8 lines · Jan Kiszka
On 2011-05-19 15:07, Avi Kivity wrote:
And when introducing hierarchical registration, we will have to go through all of this once again. Plus the API may have to be changed again if it does not fulfill all requirements of the hierarchical region
management. And we have no proof that it allows an efficient core implementation.Quoted 2 lines
This API *is* hierarchical registration. v2 will (hopefully) prove that it can be done efficiently.
We also need hierarchical dispatch. Priorities are just a weak attempt to emulate hierarchical dispatch but I don't think there's an improvement over a single dispatch table.
Quoted 6 lines
Quoted 3 lines
Hierarchical dispatch is simpler. You just need a simple list at each bus.
The API itself says nothing about whether the hierarchy is evaluated at run-time or registration time.
Except for priorities.
If you've got a hierarchy like:
- CPU:0
- i440fx:1
- PIIX3:2
- ISA:3
- DeviceA
- PCI:2
- DeviceB
In your model, the default priorities are as shown, but nothing stops DeviceB from registering with a priority of 0 which means it can intercept accesses that would normally go to the i440fx.
This is impossible in a hierarchical dispatch model. There is no setting that a PCI device can use to trap accesses that the i440fx would normally take.
I don't mind if we don't have hierarchical dispatch to start with, but priorities are fundamentally broken.
Quoted 6 lines
We could easily have the implementation
walk the memory hierarchy to dispatch an mmio.However, RAM cannot be dispatched this way (we need to resolve which ranges are RAM when the regions are registered, not accessed) so a data structure that contains all of the information is mandatory.
There is only one device that is capable of affecting the view of RAM--the i440fx PMC. The reason is simple, the i440fx is the thing that sends a request from a CPU either to a DIMM or to some device. It doesn't know which device it goes to and it doesn't care.
That's where the RAM mapping lives. It doesn't matter how the PCI I/O window is split up. You don't need that information to know where RAM is.
Regards,
Anthony Liguori
Quoted 39 lines · Anthony Liguori
Quoted 19 lines · Avi Kivity
Quoted 18 lines · Anthony Liguori
Quoted 10 lines · Avi Kivity
Quoted 8 lines · Jan Kiszka
On 2011-05-19 15:07, Avi Kivity wrote:
And when introducing hierarchical registration, we will have to go through all of this once again. Plus the API may have to be changed again if it does not fulfill all requirements of the hierarchical region
management. And we have no proof that it allows an efficient core implementation.Quoted 2 lines
This API *is* hierarchical registration. v2 will (hopefully) prove that it can be done efficiently.
We also need hierarchical dispatch. Priorities are just a weak attempt to emulate hierarchical dispatch but I don't think there's an improvement over a single dispatch table.
Quoted 6 lines
Quoted 3 lines
Hierarchical dispatch is simpler. You just need a simple list at each bus.
The API itself says nothing about whether the hierarchy is evaluated at run-time or registration time.
Except for priorities.
If you've got a hierarchy like:
- CPU:0
- i440fx:1
- PIIX3:2
- ISA:3
- DeviceA
- PCI:2
- DeviceB
Quoted 3 lines
In your model, the default priorities are as shown, but nothing stops DeviceB from registering with a priority of 0 which means it can intercept accesses that would normally go to the i440fx.
Priorities would be local, so the normal tree would look like this:
- CPU:0
- i440fx:0
- PIIX3:0
- DeviceA
- PCI-DeviceB:0
If the i440fx would like to map something different over DeviceA (or parts of it), it would create a region of prio 1 or higher.
The same would happen at CPU-level with SMRAM when SMM is entered.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 46 lines · Jan Kiszka
Quoted 39 lines · Anthony Liguori
Quoted 19 lines · Avi Kivity
Quoted 18 lines · Anthony Liguori
Quoted 10 lines · Avi Kivity
Quoted 8 lines · Jan Kiszka
On 2011-05-19 15:07, Avi Kivity wrote:
And when introducing hierarchical registration, we will have to go through all of this once again. Plus the API may have to be changed again if it does not fulfill all requirements of the hierarchical region
management. And we have no proof that it allows an efficient core implementation.Quoted 2 lines
This API *is* hierarchical registration. v2 will (hopefully) prove that it can be done efficiently.
We also need hierarchical dispatch. Priorities are just a weak attempt to emulate hierarchical dispatch but I don't think there's an improvement over a single dispatch table.
Quoted 6 lines
Quoted 3 lines
Hierarchical dispatch is simpler. You just need a simple list at each bus.
The API itself says nothing about whether the hierarchy is evaluated at run-time or registration time.
Except for priorities.
If you've got a hierarchy like:
- CPU:0
- i440fx:1
- PIIX3:2
- ISA:3
- DeviceA
- PCI:2
- DeviceBQuoted 3 lines
In your model, the default priorities are as shown, but nothing stops DeviceB from registering with a priority of 0 which means it can intercept accesses that would normally go to the i440fx.
Priorities would be local, so the normal tree would look like this:
Quoted 8 lines
- CPU:0
- i440fx:0
- PIIX3:0
- DeviceA
- PCI-DeviceB:0If the i440fx would like to map something different over DeviceA (or parts of it), it would create a region of prio 1 or higher.
If it's local, then you need a local dispatch table, no?
Regards,
Anthony Liguori
Quoted 4 lines
The same would happen at CPU-level with SMRAM when SMM is entered.
Jan
Quoted 53 lines · Anthony Liguori
Quoted 52 lines · Jan Kiszka
Quoted 39 lines · Anthony Liguori
Quoted 19 lines · Avi Kivity
Quoted 18 lines · Anthony Liguori
Quoted 10 lines · Avi Kivity
Quoted 8 lines · Jan Kiszka
On 2011-05-19 15:07, Avi Kivity wrote:
And when introducing hierarchical registration, we will have to go through all of this once again. Plus the API may have to be changed again if it does not fulfill all requirements of the hierarchical region
management. And we have no proof that it allows an efficient core implementation.Quoted 2 lines
This API *is* hierarchical registration. v2 will (hopefully) prove that it can be done efficiently.
We also need hierarchical dispatch. Priorities are just a weak attempt to emulate hierarchical dispatch but I don't think there's an improvement over a single dispatch table.
Quoted 6 lines
Quoted 3 lines
Hierarchical dispatch is simpler. You just need a simple list at each bus.
The API itself says nothing about whether the hierarchy is evaluated at run-time or registration time.
Except for priorities.
If you've got a hierarchy like:
- CPU:0
- i440fx:1
- PIIX3:2
- ISA:3
- DeviceA
- PCI:2
- DeviceBQuoted 3 lines
In your model, the default priorities are as shown, but nothing stops DeviceB from registering with a priority of 0 which means it can intercept accesses that would normally go to the i440fx.
Priorities would be local, so the normal tree would look like this:
- CPU:0
- i440fx:0
- PIIX3:0
- DeviceA
- PCI-DeviceB:0
Quoted 4 lines
Quoted 2 lines
If the i440fx would like to map something different over DeviceA (or parts of it), it would create a region of prio 1 or higher.
If it's local, then you need a local dispatch table, no?
Not working for the coalescing reason pointed out before. It's also more handy to rely on the core to do the proper dispatching then write your own logic over and over again. The core has to deal with overlapping anyway.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 9 lines · Jan Kiszka
Quoted 9 lines
Quoted 9 lines
Priorities would be local, so the normal tree would look like this:
- CPU:0
- i440fx:0
- PIIX3:0
- DeviceA
- PCI-DeviceB:0
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
If the i440fx would like to map something different over DeviceA (or parts of it), it would create a region of prio 1 or higher.
If it's local, then you need a local dispatch table, no?
Not working for the coalescing reason pointed out before.
+ RAM (including framebuffers)
Quoted 3 lines
It's also more
handy to rely on the core to do the proper dispatching then write your own logic over and over again. The core has to deal with overlapping anyway.
Plus, to reiterate, if you have the information you can calculate a flattened global dispatch table taking into account all offsets and priorities.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 11 lines · Jan Kiszka
Quoted 10 lines · Anthony Liguori
Quoted 9 lines · Jan Kiszka
On 2011-05-19 16:15, Anthony Liguori wrote:
Priorities would be local, so the normal tree would look like this:- CPU:0
- i440fx:0
- PIIX3:0
- DeviceA
- PCI-DeviceB:0
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
If the i440fx would like to map something different over DeviceA (or parts of it), it would create a region of prio 1 or higher.
If it's local, then you need a local dispatch table, no?
Not working for the coalescing reason pointed out before. It's also more handy to rely on the core to do the proper dispatching then write your own logic over and over again. The core has to deal with overlapping anyway.
So.... do you do:
isa_register_region(ISABus *bus, MemoryRegion *mr, int priority)
{
chipset_register_region(bus->chipset, mr, priority + 1);
}I don't really understand how you can fold everything into one table and not allow devices to override their parents using priorities.
Regards,
Anthony Liguori
Quoted 3 lines
Jan
Quoted 9 lines · Anthony Liguori
So.... do you do:
isa_register_region(ISABus *bus, MemoryRegion *mr, int priority){ chipset_register_region(bus->chipset, mr, priority + 1); }I don't really understand how you can fold everything into one table and not allow devices to override their parents using priorities.
Think of how a window manager folds windows with priorities onto a flat framebuffer.
You do a depth-first walk of the tree. For each child list, you iterate it from the lowest to highest priority, allowing later subregions override earlier subregions.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 13 lines · Avi Kivity
Quoted 11 lines · Anthony Liguori
So.... do you do:
isa_register_region(ISABus *bus, MemoryRegion *mr, int priority){ chipset_register_region(bus->chipset, mr, priority + 1); }I don't really understand how you can fold everything into one table and not allow devices to override their parents using priorities.
Quoted 7 lines
Think of how a window manager folds windows with priorities onto a flat framebuffer.
You do a depth-first walk of the tree. For each child list, you iterate it from the lowest to highest priority, allowing later subregions override earlier subregions.
And how you set those priorities in a sensible way? Why two device on a PCI bus will want to register their memory region with anything but highest priority? And if you let PCI subsystem to assign priorities how it will coordinate with ISA subsystem/memory controller what priorities to assign to get meaningful system?
--
Gleb.
Quoted 21 lines · Gleb Natapov
Quoted 13 lines · Avi Kivity
Quoted 11 lines · Anthony Liguori
So.... do you do:
isa_register_region(ISABus *bus, MemoryRegion *mr, int priority){ chipset_register_region(bus->chipset, mr, priority + 1); }I don't really understand how you can fold everything into one table and not allow devices to override their parents using priorities.
Quoted 7 lines
Think of how a window manager folds windows with priorities onto a flat framebuffer.
You do a depth-first walk of the tree. For each child list, you iterate it from the lowest to highest priority, allowing later subregions override earlier subregions.
Quoted 5 lines
And how you set those priorities in a sensible way? Why two device on a PCI bus will want to register their memory region with anything but highest priority? And if you let PCI subsystem to assign priorities how it will coordinate with ISA subsystem/memory controller what priorities to assign to get meaningful system?
Priorities >default will only be used for explicit overlays, e.g. RAM over MMIO in PAM regions. Non-default priorities won't be assigned to normal PCI bars or any other device's region.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 28 lines · Jan Kiszka
Quoted 21 lines · Gleb Natapov
Quoted 13 lines · Avi Kivity
Quoted 11 lines · Anthony Liguori
So.... do you do:
isa_register_region(ISABus *bus, MemoryRegion *mr, int priority){ chipset_register_region(bus->chipset, mr, priority + 1); }I don't really understand how you can fold everything into one table and not allow devices to override their parents using priorities.
Quoted 7 lines
Think of how a window manager folds windows with priorities onto a flat framebuffer.
You do a depth-first walk of the tree. For each child list, you iterate it from the lowest to highest priority, allowing later subregions override earlier subregions.
Quoted 5 lines
And how you set those priorities in a sensible way? Why two device on a PCI bus will want to register their memory region with anything but highest priority? And if you let PCI subsystem to assign priorities how it will coordinate with ISA subsystem/memory controller what priorities to assign to get meaningful system?
Quoted 4 lines
Priorities >default will only be used for explicit overlays, e.g. RAM over MMIO in PAM regions. Non-default priorities won't be assigned to normal PCI bars or any other device's region.
That does not explain who and how assign those priorities in globally meaningful way.
--
Gleb.
Quoted 29 lines · Gleb Natapov
Quoted 28 lines · Jan Kiszka
Quoted 26 lines · Gleb Natapov
Quoted 13 lines · Avi Kivity
Quoted 11 lines · Anthony Liguori
So.... do you do:
isa_register_region(ISABus *bus, MemoryRegion *mr, int priority){ chipset_register_region(bus->chipset, mr, priority + 1); }I don't really understand how you can fold everything into one table and not allow devices to override their parents using priorities.
Quoted 7 lines
Think of how a window manager folds windows with priorities onto a flat framebuffer.
You do a depth-first walk of the tree. For each child list, you iterate it from the lowest to highest priority, allowing later subregions override earlier subregions.
And how you set those priorities in a sensible way? Why two device on a PCI bus will want to register their memory region with anything but highest priority? And if you let PCI subsystem to assign priorities how it will coordinate with ISA subsystem/memory controller what priorities to assign to get meaningful system?
Quoted 6 lines
Quoted 4 lines
Priorities >default will only be used for explicit overlays, e.g. RAM over MMIO in PAM regions. Non-default priorities won't be assigned to normal PCI bars or any other device's region.
That does not explain who and how assign those priorities in globally meaningful way.
There are no global priorities. Priorities are only used inside each level of the memory region hierarchy to generate a resulting, flattened view for the next higher level. At that level, everything imported from below has the default prio again, ie. the lowest one.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 34 lines · Jan Kiszka
Quoted 33 lines · Gleb Natapov
Quoted 32 lines · Jan Kiszka
Quoted 21 lines · Gleb Natapov
Quoted 13 lines · Avi Kivity
Quoted 11 lines · Anthony Liguori
So.... do you do:
isa_register_region(ISABus *bus, MemoryRegion *mr, int priority){ chipset_register_region(bus->chipset, mr, priority + 1); }I don't really understand how you can fold everything into one table and not allow devices to override their parents using priorities.
Quoted 7 lines
Think of how a window manager folds windows with priorities onto a flat framebuffer.
You do a depth-first walk of the tree. For each child list, you iterate it from the lowest to highest priority, allowing later subregions override earlier subregions.
Quoted 5 lines
And how you set those priorities in a sensible way? Why two device on a PCI bus will want to register their memory region with anything but highest priority? And if you let PCI subsystem to assign priorities how it will coordinate with ISA subsystem/memory controller what priorities to assign to get meaningful system?
Priorities>default will only be used for explicit overlays, e.g. RAM over MMIO in PAM regions. Non-default priorities won't be assigned to normal PCI bars or any other device's region.
Quoted 7 lines
Quoted 2 lines
That does not explain who and how assign those priorities in globally meaningful way.
There are no global priorities. Priorities are only used inside each level of the memory region hierarchy to generate a resulting, flattened view for the next higher level. At that level, everything imported from below has the default prio again, ie. the lowest one.
Then SMM is impossible.
Why do we need priorities at all? There should be no overlap at each level in the hierarchy.
If you have overlapping BARs, the PCI bus will always send the request to a single device based on something that's implementation specific. This works because each PCI device advertises the BAR locations and sizes in it's config space.
To dispatch a request, the PCI bus will walk the config space to find a match. If you remove something that was previously causing an overlap, it'll the other device will now get the I/O requests.
To model this correctly, you need to let the PCI bus decide how to dispatch I/O requests (again, you need hierarchical dispatch).
In the absence of this, the PCI bus needs to look at all of the devices, figure out the flat mapping, and register it. When a device is added or removed, it needs to recalculate the flat mapping and register it.
There is no need to have centralized logic to decide this.
Regards,
Anthony Liguori
Quoted 3 lines
Jan
Quoted 38 lines · Anthony Liguori
Quoted 37 lines · Jan Kiszka
Quoted 35 lines · Gleb Natapov
Quoted 28 lines · Jan Kiszka
Quoted 21 lines · Gleb Natapov
Quoted 13 lines · Avi Kivity
Quoted 11 lines · Anthony Liguori
So.... do you do:
isa_register_region(ISABus *bus, MemoryRegion *mr, int priority){ chipset_register_region(bus->chipset, mr, priority + 1); }I don't really understand how you can fold everything into one table and not allow devices to override their parents using priorities.
Quoted 7 lines
Think of how a window manager folds windows with priorities onto a flat framebuffer.
You do a depth-first walk of the tree. For each child list, you iterate it from the lowest to highest priority, allowing later subregions override earlier subregions.
Quoted 5 lines
And how you set those priorities in a sensible way? Why two device on a PCI bus will want to register their memory region with anything but highest priority? And if you let PCI subsystem to assign priorities how it will coordinate with ISA subsystem/memory controller what priorities to assign to get meaningful system?
Quoted 4 lines
Priorities>default will only be used for explicit overlays, e.g. RAM over MMIO in PAM regions. Non-default priorities won't be assigned to normal PCI bars or any other device's region.
That does not explain who and how assign those priorities in globally meaningful way.
Quoted 6 lines
Quoted 4 lines
There are no global priorities. Priorities are only used inside each level of the memory region hierarchy to generate a resulting, flattened view for the next higher level. At that level, everything imported from below has the default prio again, ie. the lowest one.
Then SMM is impossible.
For sure it is. The CPU and the chipset, each at their mapping level, create a corresponding RAM region and register it with higher prio at the SMRAM start address (CPU and chipset will need to exchange that address or otherwise coordinate the mapping information - the price for per-CPU SMRAM).
Quoted 8 lines
Why do we need priorities at all? There should be no overlap at each level in the hierarchy.
If you have overlapping BARs, the PCI bus will always send the request to a single device based on something that's implementation specific. This works because each PCI device advertises the BAR locations and sizes in it's config space.
That's not a use case for priorities at all. Priorities are useful for PAM and SMRAM-like scenarios.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 44 lines · Jan Kiszka
Quoted 43 lines · Anthony Liguori
Quoted 41 lines · Jan Kiszka
Quoted 29 lines · Gleb Natapov
Quoted 28 lines · Jan Kiszka
Quoted 26 lines · Gleb Natapov
Quoted 13 lines · Avi Kivity
Quoted 11 lines · Anthony Liguori
So.... do you do:
isa_register_region(ISABus *bus, MemoryRegion *mr, int priority){ chipset_register_region(bus->chipset, mr, priority + 1); }I don't really understand how you can fold everything into one table and not allow devices to override their parents using priorities.
Quoted 7 lines
Think of how a window manager folds windows with priorities onto a flat framebuffer.
You do a depth-first walk of the tree. For each child list, you iterate it from the lowest to highest priority, allowing later subregions override earlier subregions.
And how you set those priorities in a sensible way? Why two device on a PCI bus will want to register their memory region with anything but highest priority? And if you let PCI subsystem to assign priorities how it will coordinate with ISA subsystem/memory controller what priorities to assign to get meaningful system?
Quoted 6 lines
Quoted 4 lines
Priorities>default will only be used for explicit overlays, e.g. RAM over MMIO in PAM regions. Non-default priorities won't be assigned to normal PCI bars or any other device's region.
That does not explain who and how assign those priorities in globally meaningful way.
There are no global priorities. Priorities are only used inside each level of the memory region hierarchy to generate a resulting, flattened view for the next higher level. At that level, everything imported from below has the default prio again, ie. the lowest one.
Quoted 8 lines
Quoted 1 line
Then SMM is impossible.
For sure it is. The CPU and the chipset, each at their mapping level, create a corresponding RAM region and register it with higher prio at the SMRAM start address (CPU and chipset will need to exchange that address or otherwise coordinate the mapping information - the price for per-CPU SMRAM).
So to get priorities right two components need to know a priori about overlap and coordinate the priorities?
Quoted 4 lines
Quoted 4 lines
Why do we need priorities at all? There should be no overlap at each level in the hierarchy.
Quoted 8 lines
Quoted 4 lines
If you have overlapping BARs, the PCI bus will always send the request to a single device based on something that's implementation specific. This works because each PCI device advertises the BAR locations and sizes in it's config space.
That's not a use case for priorities at all. Priorities are useful for PAM and SMRAM-like scenarios.
It looks like you are talking about very shallow model were overlap may happen only in chipset and chipset directly controls all of the physical address space. But we need to have solution for more complex topologies where PAM/SMRAM like scenarios may happen on each level. You are dismissing PCI as an example because all memory regions there are of the same priority, but this is just the special case of more generic scenario. Why this is not the "use case for priorities"?
--
Gleb.
Quoted 47 lines · Gleb Natapov
Quoted 46 lines · Jan Kiszka
Quoted 44 lines · Anthony Liguori
Quoted 34 lines · Jan Kiszka
Quoted 33 lines · Gleb Natapov
Quoted 32 lines · Jan Kiszka
Quoted 21 lines · Gleb Natapov
Quoted 13 lines · Avi Kivity
Quoted 11 lines · Anthony Liguori
So.... do you do:
isa_register_region(ISABus *bus, MemoryRegion *mr, int priority){ chipset_register_region(bus->chipset, mr, priority + 1); }I don't really understand how you can fold everything into one table and not allow devices to override their parents using priorities.
Quoted 7 lines
Think of how a window manager folds windows with priorities onto a flat framebuffer.
You do a depth-first walk of the tree. For each child list, you iterate it from the lowest to highest priority, allowing later subregions override earlier subregions.
Quoted 5 lines
And how you set those priorities in a sensible way? Why two device on a PCI bus will want to register their memory region with anything but highest priority? And if you let PCI subsystem to assign priorities how it will coordinate with ISA subsystem/memory controller what priorities to assign to get meaningful system?
Priorities>default will only be used for explicit overlays, e.g. RAM over MMIO in PAM regions. Non-default priorities won't be assigned to normal PCI bars or any other device's region.
Quoted 7 lines
Quoted 2 lines
That does not explain who and how assign those priorities in globally meaningful way.
There are no global priorities. Priorities are only used inside each level of the memory region hierarchy to generate a resulting, flattened view for the next higher level. At that level, everything imported from below has the default prio again, ie. the lowest one.
Then SMM is impossible.
Quoted 8 lines
Quoted 6 lines
For sure it is. The CPU and the chipset, each at their mapping level, create a corresponding RAM region and register it with higher prio at the SMRAM start address (CPU and chipset will need to exchange that address or otherwise coordinate the mapping information - the price for per-CPU SMRAM).
So to get priorities right two components need to know a priori about overlap and coordinate the priorities?
Nope, the integrator, i.e. the bridge (an abstract one, please) needs to know that it registers possibly overlapping regions. It declares that some region is allowed to overlap by using the corresponding service, optionally providing a priority > default in order get a well-define ordering.
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Why do we need priorities at all? There should be no overlap at each level in the hierarchy.
If you have overlapping BARs, the PCI bus will always send the request to a single device based on something that's implementation specific. This works because each PCI device advertises the BAR locations and sizes in it's config space.
Quoted 7 lines
Quoted 3 lines
That's not a use case for priorities at all. Priorities are useful for PAM and SMRAM-like scenarios.
It looks like you are talking about very shallow model were overlap may happen only in chipset and chipset directly controls all of the physical address space. But we need to have solution for more complex topologies where PAM/SMRAM like scenarios may happen on each level.
That's precisely my goal. PAM/SMRAM is just one example for such overlays at any bridge level, not just the chipset.
Quoted 4 lines
You are dismissing
PCI as an example because all memory regions there are of the same priority, but this is just the special case of more generic scenario. Why this is not the "use case for priorities"?
Because we know that PCI bars can overlap and are allowed to, and that this will generate some random result. So we don't need to worry about assigning priorities, we just need to declare overlaps valid.
Jan
Quoted 8 lines · Jan Kiszka
Quoted 5 lines
If you have overlapping BARs, the PCI bus will always send the request to a single device based on something that's implementation specific. This works because each PCI device advertises the BAR locations and sizes in it's config space.
That's not a use case for priorities at all. Priorities are useful for PAM and SMRAM-like scenarios.
Correct. Priorities are also useful to decide if BARs hide RAM or vice-versa (determined by the PCI container's priority vs. the RAM container priorities, not individual BARs' priorities).
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 8 lines · Anthony Liguori
Quoted 4 lines
There are no global priorities. Priorities are only used inside each level of the memory region hierarchy to generate a resulting, flattened view for the next higher level. At that level, everything imported from below has the default prio again, ie. the lowest one.
Then SMM is impossible.
It doesn't follow.
Quoted 1 line
Why do we need priorities at all? There should be no overlap at each level in the hierarchy.
Of course there is overlap. PCI BARs overlap each other, the VGA windows and ROM overlap RAM.
Quoted 2 lines
If you have overlapping BARs, the PCI bus will always send the request to a single device based on something that's implementation specific. This works because each PCI device advertises the BAR locations and sizes in it's config space.
BARs in general don't need priority, except we need to decide if BARs overlap RAM of vice-versa.
Quoted 2 lines
To dispatch a request, the PCI bus will walk the config space to find a match. If you remove something that was previously causing an overlap, it'll the other device will now get the I/O requests.
That's what *exactl* what priority means. Which device is in front, and which is in the back.
Quoted 2 lines
To model this correctly, you need to let the PCI bus decide how to dispatch I/O requests (again, you need hierarchical dispatch).
And again, this API gives you hierarchical dispatch, with the addition that some of it is done at registration time so we can prepare the RAM slots.
Quoted 2 lines
In the absence of this, the PCI bus needs to look at all of the devices, figure out the flat mapping, and register it. When a device is added or removed, it needs to recalculate the flat mapping and register it.
However we do this, we need to look at all devices.
Quoted 3 lines
There is no need to have centralized logic to decide this.
I think you're completely missing the point of my proposal.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 10 lines · Avi Kivity
Quoted 8 lines · Anthony Liguori
Quoted 4 lines
There are no global priorities. Priorities are only used inside each level of the memory region hierarchy to generate a resulting, flattened view for the next higher level. At that level, everything imported from below has the default prio again, ie. the lowest one.
Then SMM is impossible.
Quoted 7 lines
It doesn't follow.
Quoted 2 lines
Why do we need priorities at all? There should be no overlap at each level in the hierarchy.
Of course there is overlap. PCI BARs overlap each other, the VGA windows and ROM overlap RAM.
Here's what I'm still struggling with:
If children normally overlap their parents, but child priorities are always less than their parents, then what's the benefit of having anything more than two priorities settings.
As far as I can understand it, a priority of 0 means "let children windows overlap" whereas a priority of 1 means "don't let children windows overlap".
Is there a use-case for a priority above 1 and if so, what does it mean?
Quoted 9 lines
Quoted 4 lines
If you have overlapping BARs, the PCI bus will always send the request to a single device based on something that's implementation specific. This works because each PCI device advertises the BAR locations and sizes in it's config space.
BARs in general don't need priority, except we need to decide if BARs overlap RAM of vice-versa.
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
To dispatch a request, the PCI bus will walk the config space to find a match. If you remove something that was previously causing an overlap, it'll the other device will now get the I/O requests.
That's what *exactl* what priority means. Which device is in front, and which is in the back.
Why not use registration order to resolve this type of conflict? What are the use cases to use priorities where registration order wouldn't be adequate?
Quoted 4 lines
Quoted 2 lines
There is no need to have centralized logic to decide this.
I think you're completely missing the point of my proposal.
I'm struggling to find the mental model for priorities. I may just be dense here but the analogy of transparent window ordering isn't helping me.
Regards,
Anthony Liguori
Quoted 1 line · Anthony Liguori
Is there a use-case for a priority above 1 and if so, what does it mean?
In a modern northbridge mmconfig has priority over external access, and other internal registers (apic for instance) have priority over mmconfig. Bios sometimes map mmconfig is a zone colliding with something else but where it doesn't really care because the whole range is not needed.
OG.
Quoted 8 lines · Olivier Galibert
Quoted 1 line · Anthony Liguori
Is there a use-case for a priority above 1 and if so, what does it mean?
In a modern northbridge mmconfig has priority over external access, and other internal registers (apic for instance) have priority over mmconfig. Bios sometimes map mmconfig is a zone colliding with something else but where it doesn't really care because the whole range is not needed.
But priority and registration order are roughly equivalent though for things like this, no?
And the question is, "how do I control what order to register in?" is equivalent to the question, "how do I determine which region gets which priority?".
Both have an answer of, something has to understand all of the regions that may overlap and assign an explicit order/priority.
Regards,
Anthony Liguori
Quoted 4 lines
OG.
Quoted 11 lines · Anthony Liguori
Quoted 2 lines
Of course there is overlap. PCI BARs overlap each other, the VGA windows and ROM overlap RAM.
Here's what I'm still struggling with:
If children normally overlap their parents, but child priorities are always less than their parents, then what's the benefit of having anything more than two priorities settings.
As far as I can understand it, a priority of 0 means "let children windows overlap" whereas a priority of 1 means "don't let children windows overlap".
Is there a use-case for a priority above 1 and if so, what does it mean?
Children always overlap their parents. Priorities are among children of the same parent.
I expect there won't be a use case for priority 2, but that's not because of some inherent property of the design; the hardware designers simply haven't got around to designing such whacky hardware.
Note that a container is transparent, so layering several containers on top of each other simply generates a flattening of the tree. You can have an opaque container by having a lowest-priority child that covers the entire address space.
Quoted 4 lines
Quoted 2 lines
That's what *exactl* what priority means. Which device is in front, and which is in the back.
Why not use registration order to resolve this type of conflict? What are the use cases to use priorities where registration order wouldn't be adequate?
Registration order is not something you want in a declarative API. There is a non-priority equivalent, and that is to unregister (_del_subregion) all subregions in the contended region, except the one you want.
That doesn't work for PCI, since the "contended region" isn't known. You want RAM to hide PCI BARs (or perhaps vice versa), you need to tell the system which one takes precedence.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
There is no need to have centralized logic to decide this.
I think you're completely missing the point of my proposal.
I'm struggling to find the mental model for priorities. I may just be dense here but the analogy of transparent window ordering isn't helping me.
If you like, you can think of "priority" as "explicit registration order". The advantage is that is works dynamically, if a region is unregistered and re-registered, that doesn't mess up your registration order.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 34 lines · Jan Kiszka
Quoted 33 lines · Gleb Natapov
Quoted 32 lines · Jan Kiszka
Quoted 21 lines · Gleb Natapov
Quoted 13 lines · Avi Kivity
Quoted 11 lines · Anthony Liguori
So.... do you do:
isa_register_region(ISABus *bus, MemoryRegion *mr, int priority){ chipset_register_region(bus->chipset, mr, priority + 1); }I don't really understand how you can fold everything into one table and not allow devices to override their parents using priorities.
Quoted 7 lines
Think of how a window manager folds windows with priorities onto a flat framebuffer.
You do a depth-first walk of the tree. For each child list, you iterate it from the lowest to highest priority, allowing later subregions override earlier subregions.
Quoted 5 lines
And how you set those priorities in a sensible way? Why two device on a PCI bus will want to register their memory region with anything but highest priority? And if you let PCI subsystem to assign priorities how it will coordinate with ISA subsystem/memory controller what priorities to assign to get meaningful system?
Priorities >default will only be used for explicit overlays, e.g. RAM over MMIO in PAM regions. Non-default priorities won't be assigned to normal PCI bars or any other device's region.
Quoted 8 lines
Quoted 2 lines
That does not explain who and how assign those priorities in globally meaningful way.
There are no global priorities. Priorities are only used inside each level of the memory region hierarchy to generate a resulting, flattened view for the next higher level. At that level, everything imported from below has the default prio again, ie. the lowest one.
Ah, so you are advocating for filtering on each level then. Because highest level (the one that actually uses memory API) will never see two regions with different priorities since layer bellow will flatten the memory layout. So why do you need priorities if this is the case? The layer that does flattening is the layer that assign priorities anyway.
--
Gleb.
Quoted 43 lines · Gleb Natapov
Quoted 34 lines · Jan Kiszka
Quoted 33 lines · Gleb Natapov
Quoted 32 lines · Jan Kiszka
Quoted 21 lines · Gleb Natapov
Quoted 13 lines · Avi Kivity
Quoted 11 lines · Anthony Liguori
So.... do you do:
isa_register_region(ISABus *bus, MemoryRegion *mr, int priority){ chipset_register_region(bus->chipset, mr, priority + 1); }I don't really understand how you can fold everything into one table and not allow devices to override their parents using priorities.
Quoted 7 lines
Think of how a window manager folds windows with priorities onto a flat framebuffer.
You do a depth-first walk of the tree. For each child list, you iterate it from the lowest to highest priority, allowing later subregions override earlier subregions.
Quoted 5 lines
And how you set those priorities in a sensible way? Why two device on a PCI bus will want to register their memory region with anything but highest priority? And if you let PCI subsystem to assign priorities how it will coordinate with ISA subsystem/memory controller what priorities to assign to get meaningful system?
Priorities >default will only be used for explicit overlays, e.g. RAM over MMIO in PAM regions. Non-default priorities won't be assigned to normal PCI bars or any other device's region.
Quoted 8 lines
Quoted 2 lines
That does not explain who and how assign those priorities in globally meaningful way.
There are no global priorities. Priorities are only used inside each level of the memory region hierarchy to generate a resulting, flattened view for the next higher level. At that level, everything imported from below has the default prio again, ie. the lowest one.
Quoted 6 lines
Ah, so you are advocating for filtering on each level then. Because highest level (the one that actually uses memory API) will never see two regions with different priorities since layer bellow will flatten the memory layout. So why do you need priorities if this is the case? The layer that does flattening is the layer that assign priorities anyway.
See my reply to Anthony.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 13 lines · Avi Kivity
Quoted 11 lines · Anthony Liguori
So.... do you do:
isa_register_region(ISABus *bus, MemoryRegion *mr, int priority){ chipset_register_region(bus->chipset, mr, priority + 1); }I don't really understand how you can fold everything into one table and not allow devices to override their parents using priorities.
Quoted 7 lines
Think of how a window manager folds windows with priorities onto a flat framebuffer.
You do a depth-first walk of the tree. For each child list, you iterate it from the lowest to highest priority, allowing later subregions override earlier subregions.
I do not think that window manager is a good analogy. Window can overlap with only its siblings. In our memory tree each final node may overlap with any other node in the tree.
--
Gleb.
Quoted 11 lines · Gleb Natapov
Quoted 7 lines
Think of how a window manager folds windows with priorities onto a flat framebuffer.
You do a depth-first walk of the tree. For each child list, you iterate it from the lowest to highest priority, allowing later subregions override earlier subregions.
I do not think that window manager is a good analogy. Window can overlap with only its siblings. In our memory tree each final node may overlap with any other node in the tree.
Transparent windows.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 8 lines · Avi Kivity
Quoted 7 lines · Gleb Natapov
Quoted 7 lines
Think of how a window manager folds windows with priorities onto a flat framebuffer.
You do a depth-first walk of the tree. For each child list, you iterate it from the lowest to highest priority, allowing later subregions override earlier subregions.
Quoted 7 lines
Quoted 4 lines
I do not think that window manager is a good analogy. Window can overlap with only its siblings. In our memory tree each final node may overlap with any other node in the tree.
Transparent windows.
No, still not that. Think about child windows that resides outside of its parent windows on screen. In our memory region terms think about PCI BAR is registered to overlap with RAM at address 0x1000 for instance. PCI BAR memory region and RAM memory region are on very different branches of the global tree.
--
Gleb.
Quoted 14 lines · Gleb Natapov
Quoted 13 lines · Avi Kivity
Quoted 11 lines · Gleb Natapov
Quoted 7 lines
Think of how a window manager folds windows with priorities onto a flat framebuffer.
You do a depth-first walk of the tree. For each child list, you iterate it from the lowest to highest priority, allowing later subregions override earlier subregions.
I do not think that window manager is a good analogy. Window can overlap with only its siblings. In our memory tree each final node may overlap with any other node in the tree.
Quoted 7 lines
Quoted 2 lines
Transparent windows.
No, still not that. Think about child windows that resides outside of its parent windows on screen. In our memory region terms think about PCI BAR is registered to overlap with RAM at address
0x1000for instance. PCI BAR memory region and RAM memory region are on very different branches of the global tree.
Right. But what's the problem with that?
Which one takes precedence is determined by the priorities of the RAM subregion vs. the PCI bus subregion.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 17 lines · Avi Kivity
Quoted 16 lines · Gleb Natapov
Quoted 15 lines · Avi Kivity
Quoted 11 lines · Gleb Natapov
Quoted 7 lines
Think of how a window manager folds windows with priorities onto a flat framebuffer.
You do a depth-first walk of the tree. For each child list, you iterate it from the lowest to highest priority, allowing later subregions override earlier subregions.
I do not think that window manager is a good analogy. Window can overlap with only its siblings. In our memory tree each final node may overlap with any other node in the tree.
Transparent windows.
Quoted 8 lines
Quoted 5 lines
No, still not that. Think about child windows that resides outside of its parent windows on screen. In our memory region terms think about PCI BAR is registered to overlap with RAM at address
0x1000for instance. PCI BAR memory region and RAM memory region are on very different branches of the global tree.Right. But what's the problem with that?
None, unless you want to make PCI BAR visible at address 0x1000 (like what will happen today) the case above.
Quoted 3 lines
Which one takes precedence is determined by the priorities of the RAM subregion vs. the PCI bus subregion.
Yes, and that is why PCI subsystem or platform code can't directly uses memory API to register PCI memory region/RAM memory region respectively, because they wouldn't know what priorities to specify. Only chipset code knows, so the RAM/PCI memory registration should go through chipset code, but even chipset code doesn't know everything. It knows nothing about cpu local memory regions for instance, so all registrations should go through system bus in the end. Is this how API suppose to be used?
--
Gleb.
Quoted 24 lines · Gleb Natapov
Quoted 23 lines · Avi Kivity
Quoted 21 lines · Gleb Natapov
Quoted 8 lines · Avi Kivity
Quoted 7 lines · Gleb Natapov
Quoted 7 lines
Think of how a window manager folds windows with priorities onto a flat framebuffer.
You do a depth-first walk of the tree. For each child list, you iterate it from the lowest to highest priority, allowing later subregions override earlier subregions.
Quoted 7 lines
Quoted 4 lines
I do not think that window manager is a good analogy. Window can overlap with only its siblings. In our memory tree each final node may overlap with any other node in the tree.
Transparent windows.
No, still not that. Think about child windows that resides outside of its parent windows on screen. In our memory region terms think about PCI BAR is registered to overlap with RAM at address
0x1000for instance. PCI BAR memory region and RAM memory region are on very different branches of the global tree.
Quoted 4 lines
Quoted 2 lines
Right. But what's the problem with that?
None, unless you want to make PCI BAR visible at address
0x1000(like what will happen today) the case above.
There is no problem. If the PCI bus priority is higher than RAM priority, then PCI BARs will override RAM.
Quoted 8 lines
Quoted 3 lines
Which one takes precedence is determined by the priorities of the RAM subregion vs. the PCI bus subregion.
Yes, and that is why PCI subsystem or platform code can't directly uses memory API to register PCI memory region/RAM memory region respectively, because they wouldn't know what priorities to specify. Only chipset code knows, so the RAM/PCI memory registration should go through chipset code,
Correct. Chipset code creates RAM and PCI regions, and gives the PCI region to the PCI bus. Devices give BAR subregions to the PCI bus. The PCI bus does the registration.
Quoted 3 lines
but even chipset code doesn't know everything. It knows nothing
about cpu local memory regions for instance, so all registrations should go through system bus in the end. Is this how API suppose to be used?
Yes. Every point where a decision is made on how to route memory accesses is a modelled as a container node.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 25 lines · Avi Kivity
Quoted 24 lines · Gleb Natapov
Quoted 23 lines · Avi Kivity
Quoted 21 lines · Gleb Natapov
Quoted 8 lines · Avi Kivity
Quoted 7 lines · Gleb Natapov
Quoted 7 lines
Think of how a window manager folds windows with priorities onto a flat framebuffer.
You do a depth-first walk of the tree. For each child list, you iterate it from the lowest to highest priority, allowing later subregions override earlier subregions.
Quoted 7 lines
Quoted 4 lines
I do not think that window manager is a good analogy. Window can overlap with only its siblings. In our memory tree each final node may overlap with any other node in the tree.
Transparent windows.
No, still not that. Think about child windows that resides outside of its parent windows on screen. In our memory region terms think about PCI BAR is registered to overlap with RAM at address
0x1000for instance. PCI BAR memory region and RAM memory region are on very different branches of the global tree.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Right. But what's the problem with that?
None, unless you want to make PCI BAR visible at address
0x1000(like what will happen today) the case above.There is no problem. If the PCI bus priority is higher than RAM priority, then PCI BARs will override RAM.
So if memory region has no subregion that covers part of the region lower prio region is used? Now the same with the pictures:
-- root
-- PCI 0x00000 - 0x2ffff (prio 0)
-- BAR A 0x1000 - 0x1fff
-- BAR B 0x20000 - 0x20fff -- RAM 0x00000 - 0x1ffff (prio 1)
In the tree above at address 0x0 PCI has higher priority, but since there is no subregion at this range next prio region is used instead (RAM). Is this correct? If yes how core knows that container is transparent like that (RAM container is not)?
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Which one takes precedence is determined by the priorities of the RAM subregion vs. the PCI bus subregion.
Yes, and that is why PCI subsystem or platform code can't directly uses memory API to register PCI memory region/RAM memory region respectively, because they wouldn't know what priorities to specify. Only chipset code knows, so the RAM/PCI memory registration should go through chipset code,
Quoted 3 lines
Correct. Chipset code creates RAM and PCI regions, and gives the PCI region to the PCI bus. Devices give BAR subregions to the PCI bus. The PCI bus does the registration.
OK. What happens if device tries to create subregion outside of PCI region provided by chipset to PCI bus?
Quoted 8 lines
Quoted 4 lines
but even chipset code doesn't know everything. It knows nothing
about cpu local memory regions for instance, so all registrations should go through system bus in the end. Is this how API suppose to be used?Yes. Every point where a decision is made on how to route memory accesses is a modelled as a container node.
Excellent. I would argue that this is exactly the point where an overlapping can be resolved too :)
--
Gleb.
Quoted 6 lines · Gleb Natapov
Quoted 3 lines
There is no problem. If the PCI bus priority is higher than RAM priority, then PCI BARs will override RAM.
So if memory region has no subregion that covers part of the region lower prio region is used? Now the same with the pictures:
Quoted 7 lines
-- root
-- PCI0x00000-0x2ffff(prio 0)-- BAR A 0x1000 - 0x1fff -- BAR B 0x20000 - 0x20fff-- RAM
0x00000-0x1ffff(prio 1)In the tree above at address
0x0PCI has higher priority,
In a previous mail, I asserted that 1 > 0. Under this assumption, PCI has lower priority.
Quoted 3 lines
but since
there is no subregion at this range next prio region is used instead (RAM). Is this correct?
Assuming pci prio = 1 and ram prio = 0, yes.
Quoted 2 lines
If yes how core knows that container is
transparent like that (RAM container is not)?
All containers are transparent. Only RAM and MMIO regions are not.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Which one takes precedence is determined by the priorities of the RAM subregion vs. the PCI bus subregion.
Yes, and that is why PCI subsystem or platform code can't directly uses memory API to register PCI memory region/RAM memory region respectively, because they wouldn't know what priorities to specify. Only chipset code knows, so the RAM/PCI memory registration should go through chipset code,
Quoted 5 lines
Quoted 3 lines
Correct. Chipset code creates RAM and PCI regions, and gives the PCI region to the PCI bus. Devices give BAR subregions to the PCI bus. The PCI bus does the registration.
OK. What happens if device tries to create subregion outside of PCI region provided by chipset to PCI bus?
What does real hardware do?
The region API has an offset and size, and clips anything outside that. However PCI may be more complicated due to 64-bit BARs, we may need an extra API to control clipping.
Quoted 10 lines
Quoted 8 lines
Quoted 4 lines
but even chipset code doesn't know everything. It knows nothing
about cpu local memory regions for instance, so all registrations should go through system bus in the end. Is this how API suppose to be used?Yes. Every point where a decision is made on how to route memory accesses is a modelled as a container node.
Excellent. I would argue that this is exactly the point where an overlapping can be resolved too :)
It is.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 12 lines · Avi Kivity
Quoted 10 lines · Anthony Liguori
So.... do you do:
isa_register_region(ISABus *bus, MemoryRegion *mr, int priority){chipset_register_region(bus->chipset, mr, priority + 1); }I don't really understand how you can fold everything into one table and not allow devices to override their parents using priorities.
Quoted 7 lines
Think of how a window manager folds windows with priorities onto a flat framebuffer.
You do a depth-first walk of the tree. For each child list, you iterate it from the lowest to highest priority, allowing later subregions override earlier subregions.
Okay, but this doesn't explain how you'll let RAM override the VGA mapping since RAM is not represented in the same child list as VGA (RAM is a child of the PMC whereas VGA is a child of ISA/PCI, both of which are at least one level removed from the PMC).
Regards,
Anthony Liguori
Quoted 21 lines · Anthony Liguori
Quoted 12 lines · Avi Kivity
Quoted 10 lines · Anthony Liguori
So.... do you do:
isa_register_region(ISABus *bus, MemoryRegion *mr, int priority){chipset_register_region(bus->chipset, mr, priority + 1); }I don't really understand how you can fold everything into one table and not allow devices to override their parents using priorities.
Quoted 7 lines
Think of how a window manager folds windows with priorities onto a flat framebuffer.
You do a depth-first walk of the tree. For each child list, you iterate it from the lowest to highest priority, allowing later subregions override earlier subregions.
Quoted 4 lines
Okay, but this doesn't explain how you'll let RAM override the VGA mapping since RAM is not represented in the same child list as VGA (RAM is a child of the PMC whereas VGA is a child of ISA/PCI, both of which are at least one level removed from the PMC).
You can always create a new memory region with higher priority, pointing to the RAM window you want to have above VGA. That's what we do today as well, just with different effects on the internal representation.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 22 lines · Jan Kiszka
Quoted 21 lines · Anthony Liguori
Quoted 19 lines · Avi Kivity
Quoted 10 lines · Anthony Liguori
So.... do you do:
isa_register_region(ISABus *bus, MemoryRegion *mr, int priority){chipset_register_region(bus->chipset, mr, priority + 1); }I don't really understand how you can fold everything into one table and not allow devices to override their parents using priorities.
Think of how a window manager folds windows with priorities onto a flat framebuffer.
You do a depth-first walk of the tree. For each child list, you iterate it from the lowest to highest priority, allowing later subregions override earlier subregions.
Quoted 8 lines
Quoted 4 lines
Okay, but this doesn't explain how you'll let RAM override the VGA mapping since RAM is not represented in the same child list as VGA (RAM is a child of the PMC whereas VGA is a child of ISA/PCI, both of which are at least one level removed from the PMC).
You can always create a new memory region with higher priority, pointing to the RAM window you want to have above VGA. That's what we do today as well, just with different effects on the internal representation.
But then we're no better than we are today. I thought the whole point of this thread of discussion was to allow overlapping I/O regions to be handled in a better way than we do today?
Regards,
Anthony Liguori
Quoted 3 lines
Jan
Quoted 28 lines · Anthony Liguori
Quoted 27 lines · Jan Kiszka
Quoted 25 lines · Anthony Liguori
Quoted 12 lines · Avi Kivity
Quoted 10 lines · Anthony Liguori
So.... do you do:
isa_register_region(ISABus *bus, MemoryRegion *mr, int priority){chipset_register_region(bus->chipset, mr, priority + 1); }I don't really understand how you can fold everything into one table and not allow devices to override their parents using priorities.
Quoted 7 lines
Think of how a window manager folds windows with priorities onto a flat framebuffer.
You do a depth-first walk of the tree. For each child list, you iterate it from the lowest to highest priority, allowing later subregions override earlier subregions.
Okay, but this doesn't explain how you'll let RAM override the VGA mapping since RAM is not represented in the same child list as VGA (RAM is a child of the PMC whereas VGA is a child of ISA/PCI, both of which are at least one level removed from the PMC).
Quoted 7 lines
Quoted 3 lines
You can always create a new memory region with higher priority, pointing to the RAM window you want to have above VGA. That's what we do today as well, just with different effects on the internal representation.
But then we're no better than we are today. I thought the whole point of this thread of discussion was to allow overlapping I/O regions to be handled in a better way than we do today?
We would be much better than today. Today some broken magic is applied to account for the fact that overlapping registrations destroy the information below. In the future, that will be preserved, and removing the overlap will restore previous mappings - or even those which were modified in the meantime.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 6 lines · Anthony Liguori
Quoted 3 lines
You can always create a new memory region with higher priority, pointing to the RAM window you want to have above VGA. That's what we do today as well, just with different effects on the internal representation.
But then we're no better than we are today. I thought the whole point of this thread of discussion was to allow overlapping I/O regions to be handled in a better way than we do today?
It is, and the goal is achieved. Right now the code saves the old contents in isa_page_descs. With the new approach it calls memory_region_del_subregion() and the previous contents magically appear (or new contents if they changed in the meanwhile).
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 10 lines · Anthony Liguori
Quoted 7 lines
Think of how a window manager folds windows with priorities onto a flat framebuffer.
You do a depth-first walk of the tree. For each child list, you iterate it from the lowest to highest priority, allowing later subregions override earlier subregions.
Okay, but this doesn't explain how you'll let RAM override the VGA mapping since RAM is not represented in the same child list as VGA (RAM is a child of the PMC whereas VGA is a child of ISA/PCI, both of which are at least one level removed from the PMC).
VGA will override RAM.
Memory controller
|
+-- RAM container (prio 0)
|
+-- PCI container (prio 1)
|
+--- vga windowSignature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 17 lines · Avi Kivity
Quoted 9 lines · Anthony Liguori
Quoted 7 lines
Think of how a window manager folds windows with priorities onto a flat framebuffer.
You do a depth-first walk of the tree. For each child list, you iterate it from the lowest to highest priority, allowing later subregions override earlier subregions.
Quoted 4 lines
Okay, but this doesn't explain how you'll let RAM override the VGA mapping since RAM is not represented in the same child list as VGA (RAM is a child of the PMC whereas VGA is a child of ISA/PCI, both of which are at least one level removed from the PMC).
VGA will override RAM.
Quoted 7 lines
Memory controller
|+-- RAM container (prio 0) | +-- PCI container (prio 1) | +--- vga window
Unless the RAM controller increases it's priority, right? That's how you would implement SMM, by doing priority++?
But if you have:
Memory controller
|
+-- RAM container (prio 0) | +-- PCI container (prio 1) | +-- PCI-X container (prio 2) | +--- vga window
Now you need to do priority = 3?
Jan had mentioned previously about registering a new temporary window. I assume the registration always gets highest_priority++, or do you have to explicitly specify that PCI container gets priority=1?
Regards,
Anthony Liguori
Quoted 2 lines
Quoted 41 lines · Anthony Liguori
Quoted 17 lines · Avi Kivity
Quoted 9 lines · Anthony Liguori
Quoted 7 lines
Think of how a window manager folds windows with priorities onto a flat framebuffer.
You do a depth-first walk of the tree. For each child list, you iterate it from the lowest to highest priority, allowing later subregions override earlier subregions.
Quoted 4 lines
Okay, but this doesn't explain how you'll let RAM override the VGA mapping since RAM is not represented in the same child list as VGA (RAM is a child of the PMC whereas VGA is a child of ISA/PCI, both of which are at least one level removed from the PMC).
VGA will override RAM.
Quoted 7 lines
Memory controller
|+-- RAM container (prio 0) | +-- PCI container (prio 1) | +--- vga windowUnless the RAM controller increases it's priority, right? That's how you would implement SMM, by doing priority++?
But if you have:
Memory controller
|+-- RAM container (prio 0) | +-- PCI container (prio 1) | +-- PCI-X container (prio 2) | +--- vga window
Quoted 5 lines
Now you need to do priority = 3?
Jan had mentioned previously about registering a new temporary window. I assume the registration always gets highest_priority++, or do you have to explicitly specify that PCI container gets priority=1?
The latter.
And I really prefer to have this explicit over deriving the priority from the registration order. That's way too fragile/unhandy. If you decide to replace a region of lower priority later on, you need to reregister everything at that level.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 11 lines · Jan Kiszka
Quoted 4 lines
Jan had mentioned previously about registering a new temporary window. I assume the registration always gets highest_priority++, or do you have to explicitly specify that PCI container gets priority=1?
The latter.
And I really prefer to have this explicit over deriving the priority from the registration order. That's way too fragile/unhandy. If you decide to replace a region of lower priority later on, you need to reregister everything at that level.
Exactly.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 7 lines · Avi Kivity
Quoted 6 lines · Jan Kiszka
Quoted 3 lines
Jan had mentioned previously about registering a new temporary window. I assume the registration always gets highest_priority++, or do you
have
Quoted 1 line
to explicitly specify that PCI container gets priority=1?
Quoted 8 lines
Quoted 6 lines
The latter.
And I really prefer to have this explicit over deriving the priority from the registration order. That's way too fragile/unhandy. If you decide to replace a region of lower priority later on, you need to reregister everything at that level.
Exactly.
So priority is equivalent to registration order except it's explicit?
So this is more palatable to me but I do prefer registration order since it promises less.
But that said, this is a relatively minor feature IMHO given that I now understand the scope so I'm not going to object strongly.
Regards,
Anthony Liguori
Quoted 13 lines · Anthony Liguori
Except for priorities.
If you've got a hierarchy like:
- CPU:0
- i440fx:1
- PIIX3:2
- ISA:3
- DeviceA
- PCI:2
- DeviceB
Quoted 1 line
In your model, the default priorities are as shown, but nothing stops DeviceB from registering with a priority of 0 which means it can intercept accesses that would normally go to the i440fx.
Priorities are only within the parent region. DeviceB's priorities have no effect whatsoever since it is an only child.
(and btw the intent is to have higher priorities hide lower priorities).
The model is that of a nested window system with transparent windows.
Quoted 4 lines
This is impossible in a hierarchical dispatch model. There is no setting that a PCI device can use to trap accesses that the i440fx would normally take.
I don't mind if we don't have hierarchical dispatch to start with, but priorities are fundamentally broken.
Are not.
Quoted 11 lines
Quoted 7 lines
We could easily have the implementation
walk the memory hierarchy to dispatch an mmio.However, RAM cannot be dispatched this way (we need to resolve which ranges are RAM when the regions are registered, not accessed) so a data structure that contains all of the information is mandatory.
There is only one device that is capable of affecting the view of RAM--the i440fx PMC. The reason is simple, the i440fx is the thing that sends a request from a CPU either to a DIMM or to some device. It doesn't know which device it goes to and it doesn't care.
That's where the RAM mapping lives. It doesn't matter how the PCI I/O window is split up. You don't need that information to know where RAM is.
There is also RAM in the framebuffer and the vga windows. It moves around.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 16 lines · Avi Kivity
Quoted 15 lines · Gleb Natapov
Quoted 3 lines · Avi Kivity
Quoted 2 lines · Jan Kiszka
Quoted 2 lines
We need to head for the more hardware-like approach. What happens
when
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
you program overlapping BARs? I imagine the result is implementation-defined, but ends up with one region decoded in preference to the other. There is simply no way to reject an overlapping mapping.
But there is also now simple way to allow them. At least not without exposing control about their ordering AND allowing to hook up managing code (e.g. of the PCI bridge or the chipset) that controls
registrations.
Quoted 1 line
Quoted 8 lines
Quoted 3 lines
Quoted 2 lines
What about
memory_region_add_subregion(..., int priority)as I suggested in another message?Haven't saw another message yet, but how caller knows about priority?
The caller is emulating some hub or router and should decide on priority like real hardware.
For example, piix gives higher priority to the vga window over RAM.
Well......
The i440fx may direct VGA accesses to RAM depending on the SMM registers. By the time the PIIX gets the I/O request, we're past the memory controller.
This is my biggest concern about this whole notion of "priority". These sort of issues are not dealt with by a simple z-ordering. There is logic in each component that may be arbitrarily complex.
We're going to end up having to dynamically change the "priority" based how registers are programmed. But priorities are relative so it's unclear to me how the I440FX would prioritize RAM over dispatch to PIIX (for VGA, for instance).
Regards,
Anthony Liguori
Quoted 32 lines · Anthony Liguori
Quoted 16 lines · Avi Kivity
Quoted 15 lines · Gleb Natapov
Quoted 3 lines · Avi Kivity
Quoted 2 lines · Jan Kiszka
Quoted 2 lines
We need to head for the more hardware-like approach. What happens
when
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
you program overlapping BARs? I imagine the result is implementation-defined, but ends up with one region decoded in preference to the other. There is simply no way to reject an overlapping mapping.
But there is also now simple way to allow them. At least not without exposing control about their ordering AND allowing to hook up managing code (e.g. of the PCI bridge or the chipset) that controls
registrations.
Quoted 1 line
Quoted 8 lines
Quoted 3 lines
Quoted 2 lines
What about
memory_region_add_subregion(..., int priority)as I suggested in another message?Haven't saw another message yet, but how caller knows about priority?
The caller is emulating some hub or router and should decide on priority like real hardware.
For example, piix gives higher priority to the vga window over RAM.
Well......
The i440fx may direct VGA accesses to RAM depending on the SMM registers. By the time the PIIX gets the I/O request, we're past the memory controller.
Quoted 8 lines
This is my biggest concern about this whole notion of "priority". These sort of issues are not dealt with by a simple z-ordering. There is logic in each component that may be arbitrarily complex.
We're going to end up having to dynamically change the "priority" based how registers are programmed. But priorities are relative so it's unclear to me how the I440FX would prioritize RAM over dispatch to PIIX (for VGA, for instance).
But creating an extra RAM window region with higher priority than the underlying mappings.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 11 lines · Jan Kiszka
Quoted 10 lines · Anthony Liguori
Well......
The i440fx may direct VGA accesses to RAM depending on the SMM registers. By the time the PIIX gets the I/O request, we're past the memory controller.
This is my biggest concern about this whole notion of "priority". These sort of issues are not dealt with by a simple z-ordering. There is logic in each component that may be arbitrarily complex.
Quoted 7 lines
Quoted 4 lines
We're going to end up having to dynamically change the "priority" based how registers are programmed. But priorities are relative so it's unclear to me how the I440FX would prioritize RAM over dispatch to PIIX (for VGA, for instance).
But creating an extra RAM window region with higher priority than the underlying mappings.
But the i440fx doesn't register the VGA region. The PIIX3 (ISA bus) does, so how does it know what the priority of that mapping is?
Regards,
Anthony Liguori
Quoted 17 lines · Anthony Liguori
Quoted 16 lines · Jan Kiszka
Quoted 14 lines · Anthony Liguori
Well......
The i440fx may direct VGA accesses to RAM depending on the SMM registers. By the time the PIIX gets the I/O request, we're past the memory controller.
This is my biggest concern about this whole notion of "priority". These sort of issues are not dealt with by a simple z-ordering. There is logic in each component that may be arbitrarily complex.
We're going to end up having to dynamically change the "priority" based how registers are programmed. But priorities are relative so it's unclear to me how the I440FX would prioritize RAM over dispatch to PIIX (for VGA, for instance).
Quoted 5 lines
Quoted 2 lines
But creating an extra RAM window region with higher priority than the underlying mappings.
But the i440fx doesn't register the VGA region. The PIIX3 (ISA bus) does, so how does it know what the priority of that mapping is?
Everything imported from "below" is of default priority for a bridge. So it just has to add 1 to that prio value (or more if it needs to support more layers).
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 3 lines · Anthony Liguori
But the i440fx doesn't register the VGA region. The PIIX3 (ISA bus) does, so how does it know what the priority of that mapping is?
The PCI bridge also has a say, no?
(and it would be a VGA region over memory, not the other way around).
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 7 lines · Avi Kivity
Quoted 4 lines · Anthony Liguori
But the i440fx doesn't register the VGA region. The PIIX3 (ISA bus) does, so how does it know what the priority of that mapping is?
The PCI bridge also has a say, no?
For legacy VGA memory? That's a good question. I've always assumed that legacy VGA memory is handled directly in the chipset by redirecting writes to the first VGA adapter it encounters (which usually happens to be the builtin one these days).
I'm not sure it's possible to have a VGA device behind a bridge that also handles legacy VGA memory because the bridge pretty clearly can only have BARs within a certain region of memory (based on the bridge's config space).
Regards,
Anthony Liguori
Quoted 3 lines
(and it would be a VGA region over memory, not the other way around).
Quoted 7 lines · Anthony Liguori
Quoted 6 lines · Avi Kivity
Quoted 4 lines · Anthony Liguori
But the i440fx doesn't register the VGA region. The PIIX3 (ISA bus) does, so how does it know what the priority of that mapping is?
Quoted 6 lines
Quoted 1 line
The PCI bridge also has a say, no?
For legacy VGA memory? That's a good question. I've always assumed that legacy VGA memory is handled directly in the chipset by redirecting writes to the first VGA adapter it encounters (which usually happens to be the builtin one these days).
Nope. It's well defined in the PCI specs that every PCI-PCI bridge can (or have to? need to check) control the flow of legacy VGA to its downstream devices.
Quoted 5 lines
I'm not sure it's possible to have a VGA device behind a bridge that also handles legacy VGA memory because the bridge pretty clearly can only have BARs within a certain region of memory (based on the bridge's config space).
That's part of my notebook PCI tree, I bet you have something similar:
\-[0000:00]-+-00.0 Intel Corporation Core Processor DRAM Controller
+-01.0-[01]--+-00.0 nVidia Corporation GT216 [Quadro FX 880M]
| \-00.1 nVidia Corporation High Definition Audio ControllerSo even this single, though not built-in VGA adapter is behind a bridge.
And if you look closer, you can find:
00:01.0 PCI bridge: Intel Corporation Core Processor PCI Express x16 Root Port (rev 02) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00002000-00002fff
Memory behind bridge: cc000000-cdefffff
Prefetchable memory behind bridge: 00000000ce000000-00000000dfffffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
...
Note that 'VGA+' in BridgeCtl. It allows the NVIDIA adapter to handle legacy VGA.
Jan
Quoted 6 lines · Anthony Liguori
The i440fx may direct VGA accesses to RAM depending on the SMM registers. By the time the PIIX gets the I/O request, we're past the memory controller.
This is my biggest concern about this whole notion of "priority". These sort of issues are not dealt with by a simple z-ordering. There is logic in each component that may be arbitrarily complex.
We're going to end up having to dynamically change the "priority" based how registers are programmed. But priorities are relative so it's unclear to me how the I440FX would prioritize RAM over dispatch to PIIX (for VGA, for instance).
You can change priorities by removing the region and re-adding it with a different priority. In practice I don't think this is ever necessary; we'll have fixed priorities and dynamic addition and removal.
For the per-cpu SMM case the only reasonable solution I see is a per-cpu memory map (= root region).
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 9 lines · Gleb Natapov
Quoted 8 lines · Avi Kivity
Quoted 7 lines · Jan Kiszka
Quoted 6 lines
We need to head for the more hardware-like approach. What happens when you program overlapping BARs? I imagine the result is implementation-defined, but ends up with one region decoded in preference to the other. There is simply no way to reject an overlapping mapping.
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
But there is also now simple way to allow them. At least not without exposing control about their ordering AND allowing to hook up managing code (e.g. of the PCI bridge or the chipset) that controls registrations.
What about
memory_region_add_subregion(..., int priority)as I suggested in another message?Haven't saw another message yet, but how caller knows about priority?
I dont think it does, overlap behaviour should be decided by nodes in the bus path from master to slave. I think Anthonys call chain approach makes sense, but it doesn't mean that one cant create a "flat" cached representation of it for fast acceses. The bus (interconnects & bridges) must be able to invalidate it when configuration changes though (leading to a new walk through the call chain and creation of an updated cached "flat" state).
Some other comments:
On the CPU local aspect, I think it is increasingly common in the embedded space to see local busses with CPU local peripherals in addition to the "system" bus with "global" peripherals.
Another thing that was discussed was the ability for devices to know who is accessing them, I think this is uncommon but still it does exist. IIRC an example is the MIPS GIC has some internal decoding based on which CPU is accessing it (but there are other examples). Would be nice to be able to handle it but the accesor info should definitely not be passed to devices by default, it'll only lead to hacks when not strictly necessary...
Overall I like the initiaive and look forward to see code and testing it.
Cheers
Quoted 3 lines · Edgar E. Iglesias
On the CPU local aspect, I think it is increasingly common in the embedded space to see local busses with CPU local peripherals in addition to the "system" bus with "global" peripherals.
Yes: newer ARM cores have per-CPU builtin peripherals (timers and the like). The interrupt controller also typically has some registers which are per-CPU and some which are global. At the moment these are rather weirdly placed outside the CPU and it's the obligation of the board model to instantiate them.
(Cache modelling would also require a bit more care about the distinction between which core made a memory request and where per-core peripherals live in the cache hierarchy.)
The other sort-of-per-core thing is that for TrustZone (which we don't currently support but might want to) if a core is in 'secure' mode it can potentially see a completely different memory map. [In hardware this works by the secure/nonsecure bit being passed around with memory transactions and devices or fabric behaving differently depending on its value.]
It would also be nice if the APIs supported more heterogenous setups (for instance a VersatileExpress model where we model a quad core A9 main CPU and also the M3 microcontroller that does system control and has a completely different view of the world.)
Quoted 3 lines
Another thing that was discussed was the ability for devices to know who is accessing them, I think this is uncommon but still it does exist.
Yes, again the ARM GIC is an example. At the momemnt this is implemented by the GIC implementation looking at cpu_single_env->cpu_index.
Debug setups also sometimes have magic peripherals that behave differently depending on who is accessing them.
-- PMM
Quoted 10 lines · Avi Kivity
Quoted 6 lines
The tricky part is wiring this up efficiently for TCG, ie. in QEMU's softmmu. I played with passing the issuing CPUState (or NULL for devices) down the MMIO handler chain. Not totally beautiful as decentralized dispatching was still required, but at least only moderately invasive. Maybe your API allows for cleaning up the management and dispatching part, need to rethink...
My suggestion is opposite - have a different MemoryRegion for each (e.g.
CPUState::memory). Then the TLBs will resolve to a differentram_addr_tfor the same physical address, for the local APIC range.
I don't understand the different ram_addr_t part.
The TLB should dispatch to a per-CPU dispatch table. The per-CPU should dispatch almost everything to a global dispatch table.
The global dispatch table is the chipset (Northbridge/Southbridge).
The chipset can then dispatch to individual busses which can then further dispatch as appropriate.
Overlapping regions can be handled differently at each level. For instance, if a PCI device registers an IO region to the same location as the APIC, the APIC always wins because the PCI bus will never see the access.
You cannot do this properly with a single dispatch table because the behavior depends on where in the hierarchy the I/O is being handled.
Regards,
Anthony Liguori
Quoted 8 lines · Anthony Liguori
Quoted 7 lines · Avi Kivity
Quoted 6 lines
The tricky part is wiring this up efficiently for TCG, ie. in QEMU's softmmu. I played with passing the issuing CPUState (or NULL for devices) down the MMIO handler chain. Not totally beautiful as decentralized dispatching was still required, but at least only moderately invasive. Maybe your API allows for cleaning up the management and dispatching part, need to rethink...
Quoted 6 lines
Quoted 3 lines
My suggestion is opposite - have a different MemoryRegion for each (e.g.
CPUState::memory). Then the TLBs will resolve to a differentram_addr_tfor the same physical address, for the local APIC range.I don't understand the different
ram_addr_tpart.
The TLBs map a virtual address to a ram_addr_t.
Quoted 8 lines
The TLB should dispatch to a per-CPU dispatch table. The per-CPU should dispatch almost everything to a global dispatch table.
The global dispatch table is the chipset (Northbridge/Southbridge).
The chipset can then dispatch to individual busses which can then further dispatch as appropriate.
Overlapping regions can be handled differently at each level. For instance, if a PCI device registers an IO region to the same location as the APIC, the APIC always wins because the PCI bus will never see the access.
That's inefficient, since you always have to traverse the hierarchy.
Quoted 1 line
You cannot do this properly with a single dispatch table because the behavior depends on where in the hierarchy the I/O is being handled.
You can. When you have a TLB miss, you walk the memory hierarchy (which is per-cpu) and end up with a ram_addr_t which is stowed in the TLB entry. Further accesses dispatch via this ram_addr_t, without taking the cpu into consideration (the TLB is, after all, already per-cpu).
Since each APIC will have its own ram_addr_t, we don't need per-cpu dispatch.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 9 lines · Avi Kivity
Quoted 8 lines · Anthony Liguori
Quoted 7 lines · Avi Kivity
Quoted 6 lines
The tricky part is wiring this up efficiently for TCG, ie. in QEMU's softmmu. I played with passing the issuing CPUState (or NULL for devices) down the MMIO handler chain. Not totally beautiful as decentralized dispatching was still required, but at least only moderately invasive. Maybe your API allows for cleaning up the management and dispatching part, need to rethink...
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
My suggestion is opposite - have a different MemoryRegion for each (e.g.
CPUState::memory). Then the TLBs will resolve to a differentram_addr_tfor the same physical address, for the local APIC range.I don't understand the different
ram_addr_tpart.The TLBs map a virtual address to a
ram_addr_t.
It actually maps virtual address to host virtual addresses. Virtual addresses that map to I/O memory never get stored in the TLB.
You don't need separate I/O registration addresses in order to do per-CPU dispatch provided that you route the dispatch routines through the CPUs first.
Quoted 7 lines
Quoted 5 lines
Overlapping regions can be handled differently at each level. For instance, if a PCI device registers an IO region to the same location as the APIC, the APIC always wins because the PCI bus will never see the access.
That's inefficient, since you always have to traverse the hierarchy.
Is efficiency really a problem here? Besides, I don't think that's really correct. You're adding at most 2-3 extra function pointer invocations. I don't think you can really call that inefficient.
Quoted 6 lines
Quoted 2 lines
You cannot do this properly with a single dispatch table because the behavior depends on where in the hierarchy the I/O is being handled.
You can. When you have a TLB miss, you walk the memory hierarchy (which is per-cpu) and end up with a
ram_addr_twhich is stowed in the TLB entry.
I think we're overloading the term TLB. Are you referring to l1_phys_map as the TLB because I thought Jan was referring to the actual emulated TLB that TCG uses?
Quoted 5 lines
Further accesses dispatch via this
ram_addr_t, without taking the cpu into consideration (the TLB is, after all, already per-cpu).Since each APIC will have its own
ram_addr_t, we don't need per-cpu dispatch.
You need to have per-CPU l1_phys_maps which would result in quite a lot of additional memory overhead.
Regards,
Anthony Liguori
Quoted 4 lines · Anthony Liguori
Quoted 1 line
The TLBs map a virtual address to a
ram_addr_t.It actually maps virtual address to host virtual addresses. Virtual addresses that map to I/O memory never get stored in the TLB.
They are stored. Look at glue(io_read, SUFFIX) and its caller for example.
Quoted 2 lines
You don't need separate I/O registration addresses in order to do per-CPU dispatch provided that you route the dispatch routines through the CPUs first.
Right, but you pay for an extra lookup which always fails.
Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
Overlapping regions can be handled differently at each level. For instance, if a PCI device registers an IO region to the same location as the APIC, the APIC always wins because the PCI bus will never see the access.
That's inefficient, since you always have to traverse the hierarchy.
Is efficiency really a problem here? Besides, I don't think that's really correct. You're adding at most 2-3 extra function pointer invocations. I don't think you can really call that inefficient.
Well, you'll need something special for SMM as well. Per-cpu memory map solves all that neatly.
Quoted 9 lines
Quoted 7 lines
Quoted 3 lines
You cannot do this properly with a single dispatch table because the behavior depends on where in the hierarchy the I/O is being handled.
You can. When you have a TLB miss, you walk the memory hierarchy (which is per-cpu) and end up with a
ram_addr_twhich is stowed in the TLB entry.I think we're overloading the term TLB. Are you referring to
l1_phys_mapas the TLB because I thought Jan was referring to the actual emulated TLB that TCG uses?
env->iotlb. For some reason I though this was folded into enc->tlb_table, but it isn't (should it be? saves a lookup).
Quoted 8 lines
Quoted 6 lines
Further accesses dispatch via this
ram_addr_t, without taking the cpu into consideration (the TLB is, after all, already per-cpu).Since each APIC will have its own
ram_addr_t, we don't need per-cpu dispatch.You need to have per-CPU
l1_phys_mapswhich would result in quite a lot of additional memory overhead.
This is predicated on a better lookup method, yes.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 22 lines · Anthony Liguori
Quoted 10 lines · Avi Kivity
Quoted 6 lines
The tricky part is wiring this up efficiently for TCG, ie. in QEMU's softmmu. I played with passing the issuing CPUState (or NULL for devices) down the MMIO handler chain. Not totally beautiful as decentralized dispatching was still required, but at least only moderately invasive. Maybe your API allows for cleaning up the management and dispatching part, need to rethink...
My suggestion is opposite - have a different MemoryRegion for each (e.g.
CPUState::memory). Then the TLBs will resolve to a differentram_addr_tfor the same physical address, for the local APIC range.I don't understand the different
ram_addr_tpart.The TLB should dispatch to a per-CPU dispatch table. The per-CPU should dispatch almost everything to a global dispatch table.
The global dispatch table is the chipset (Northbridge/Southbridge).
The chipset can then dispatch to individual busses which can then further dispatch as appropriate.
Quoted 7 lines
Overlapping regions can be handled differently at each level. For instance, if a PCI device registers an IO region to the same location as the APIC, the APIC always wins because the PCI bus will never see the access.
You cannot do this properly with a single dispatch table because the behavior depends on where in the hierarchy the I/O is being handled.
Ah, now I remember why I did not follow that path: Not invasiveness, but performance concerns. I assume TLB refills have their share in TCG performance, and adding another lookup layer, probably for every target, will be measurable. I was wondering if that is worth the, granted, cleaner design.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 9 lines · Jan Kiszka
Quoted 3 lines
You cannot do this properly with a single dispatch table because the behavior depends on where in the hierarchy the I/O is being handled.
Ah, now I remember why I did not follow that path: Not invasiveness, but performance concerns. I assume TLB refills have their share in TCG performance, and adding another lookup layer, probably for every target, will be measurable. I was wondering if that is worth the, granted, cleaner design.
We can have a per-cpu hash table and flattened slots list, though that seems wasteful. I agree that a tree walk is too expensive for a tlb miss.
We'll start however with the existing phys_desc, so no performance concerns, and no per-cpu APIC address either (btw it is broken in kvm as well, and hard to fix - we don't want per-cpu page tables).
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 29 lines · Jan Kiszka
Quoted 8 lines · Avi Kivity
The current memory APIs (
cpu_register_io_memory,cpu_register_physical_memory) suffer from a number of drawbacks:- lack of centralized bookkeeping
- acpu_register_physical_memory()that overlaps an existing region
will overwrite the preexisting region; but a followingcpu_unregister_physical_memory()will not restore things to status quo ante.Restoring is not the problem. The problem is that the current API deletes or truncates regions implicitly by overwriting. That makes tracking the layout hard, and it is also error-prone as the device that accidentally overlaps with some other device won't receive a notification of this potential conflict.
Such implicite truncation or deletion must be avoided in a new API, forcing the users to explicitly reference an existing region when dropping or modifying it. But your API goes in the right direction.
Quoted 8 lines
- coalescing and dirty logging are all cleared on unregistering, so
the client has to re-issue them when re-registering
- lots of opaques
- no nesting
- if a region has multiple subregions that need different handling
(different callbacks, RAM interspersed with MMIO) then client code has to deal with that manually
- we can't interpose code between a device and global memory handling
Quoted 8 lines
I would add another drawback:
- Inability to identify the origin of a region accesses and handle them differently based on the source.
That is at least problematic for the x86 APIC which is CPU local. Our current way do deal with it is, well, very creative and falls to dust if a guest actually tries to remap the APIC.
This is about registration. Right now you can only register IO intercepts in the chipset, not on a per-CPU basis. We could just as easily have:
CPUState {
MemoryRegion apic_region;
};per_cpu_register_memory(env, &env->apic_region);
To make that work.
We need to revamp registration. We should be able to register at the following levels:
1) per-CPU (new API)
2) chipset (effectively the current cpu_register_physical_memory)
3) per-BUS (pci_register_bar())
Regards,
Anthony Liguori
Quoted 42 lines · Anthony Liguori
Quoted 32 lines · Jan Kiszka
Quoted 8 lines · Avi Kivity
The current memory APIs (
cpu_register_io_memory,cpu_register_physical_memory) suffer from a number of drawbacks:- lack of centralized bookkeeping
- acpu_register_physical_memory()that overlaps an existing region
will overwrite the preexisting region; but a followingcpu_unregister_physical_memory()will not restore things to status quo ante.Restoring is not the problem. The problem is that the current API deletes or truncates regions implicitly by overwriting. That makes tracking the layout hard, and it is also error-prone as the device that accidentally overlaps with some other device won't receive a notification of this potential conflict.
Such implicite truncation or deletion must be avoided in a new API, forcing the users to explicitly reference an existing region when dropping or modifying it. But your API goes in the right direction.
Quoted 9 lines
- coalescing and dirty logging are all cleared on unregistering, so
the client has to re-issue them when re-registering
- lots of opaques
- no nesting
- if a region has multiple subregions that need different handling
(different callbacks, RAM interspersed with MMIO) then client code has to deal with that manually
- we can't interpose code between a device and global memory
handlingI would add another drawback:
Quoted 8 lines
- Inability to identify the origin of a region accesses and handle
them
differently based on the source.That is at least problematic for the x86 APIC which is CPU local.
Our
current way do deal with it is, well, very creative and falls to dust if a guest actually tries to remap the APIC.
Quoted 8 lines
This is about registration. Right now you can only register IO intercepts in the chipset, not on a per-CPU basis. We could just as easily have:
CPUState { MemoryRegion apic_region; };
per_cpu_register_memory(env, &env->apic_region);
Right. Or all memory per-cpu, with two sub-regions:
- global memory
- overlaid apic memory
for this, we need to have well defined semantics for overlap (perhaps a priority argument to memory_region_add_subregion).
Quoted 9 lines
To make that work.
We need to revamp registration. We should be able to register at the following levels:
1) per-CPU (new API)
2) chipset (effectively the current
cpu_register_physical_memory)3) per-BUS (
pci_register_bar())
The important thing is that all of these take a MemoryRegion argument, so we don't have to duplicate the coalescing, logging, and other APIs.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 10 lines · Avi Kivity
Quoted 8 lines
This is about registration. Right now you can only register IO intercepts in the chipset, not on a per-CPU basis. We could just as easily have:
CPUState { MemoryRegion apic_region; };
per_cpu_register_memory(env, &env->apic_region);
Quoted 6 lines
Right. Or all memory per-cpu, with two sub-regions:
- global memory
- overlaid apic memoryfor this, we need to have well defined semantics for overlap (perhaps a priority argument to
memory_region_add_subregion).
Or even
cpu_memory_region
|
+-- global memory map (prio 0) | | | +-- RAM (prio 0) | | | +-- PCI (prio 1) | +-- SMM memory (if active, prio 1) | +-- APIC memory (if active, prio 2)
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 23 lines · Avi Kivity
Quoted 12 lines · Avi Kivity
Quoted 10 lines
This is about registration. Right now you can only register IO intercepts in the chipset, not on a per-CPU basis. We could just as easily have:
CPUState { MemoryRegion apic_region; };
per_cpu_register_memory(env, &env->apic_region);Quoted 7 lines
Right. Or all memory per-cpu, with two sub-regions:
- global memory
- overlaid apic memoryfor this, we need to have well defined semantics for overlap (perhaps a priority argument to
memory_region_add_subregion).Or even
Quoted 7 lines
cpu_memory_region
|+-- global memory map (prio 0) | | | +-- RAM (prio 0) | | | +-- PCI (prio 1)
It depends on the chipset and its configuration (via PAM e.g.) in what region which takes precedence. Fixed prios do not help here.
Quoted 4 lines
|
+-- SMM memory (if active, prio 1) | +-- APIC memory (if active, prio 2)
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 32 lines · Jan Kiszka
Quoted 23 lines · Avi Kivity
Quoted 12 lines · Avi Kivity
Quoted 10 lines
This is about registration. Right now you can only register IO intercepts in the chipset, not on a per-CPU basis. We could just as easily have:
CPUState { MemoryRegion apic_region;};
per_cpu_register_memory(env,&env->apic_region);Quoted 7 lines
Right. Or all memory per-cpu, with two sub-regions:
- global memory
- overlaid apic memoryfor this, we need to have well defined semantics for overlap (perhaps a priority argument to
memory_region_add_subregion).Or even
Quoted 7 lines
cpu_memory_region| +-- global memory map (prio 0) | | | +-- RAM (prio 0) | | | +-- PCI (prio 1)
Quoted 2 lines
It depends on the chipset and its configuration (via PAM e.g.) in what region which takes precedence. Fixed prios do not help here.
The priorities are determined by the device code.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 33 lines · Avi Kivity
Quoted 32 lines · Jan Kiszka
Quoted 30 lines · Avi Kivity
Quoted 12 lines · Avi Kivity
Quoted 10 lines
This is about registration. Right now you can only register IO intercepts in the chipset, not on a per-CPU basis. We could just as easily have:
CPUState {
MemoryRegionapic_region;
};
per_cpu_register_memory(env,&env->apic_region);Quoted 7 lines
Right. Or all memory per-cpu, with two sub-regions:
- global memory
- overlaid apic memoryfor this, we need to have well defined semantics for overlap (perhaps a priority argument to
memory_region_add_subregion).Or even
cpu_memory_region
|+-- global memory map (prio 0) | | | +-- RAM (prio 0) | | | +-- PCI (prio 1)
Quoted 2 lines
Quoted 2 lines
It depends on the chipset and its configuration (via PAM e.g.) in what region which takes precedence. Fixed prios do not help here.
It's really layering.
To implement PAM in a robust way, you need a certain set of memory accesses to first flow through the chipset before going to the next location with the ability to intercept.
We do something rather weird today by changing registrations first saving the current registrations. It would be much more elegant to just intercept the I/O requests and redirect accordingly.
Regards,
Anthony Liguori
Quoted 3 lines
The priorities are determined by the device code.
Quoted 39 lines · Anthony Liguori
Quoted 33 lines · Avi Kivity
Quoted 32 lines · Jan Kiszka
Quoted 30 lines · Avi Kivity
Quoted 12 lines · Avi Kivity
Quoted 10 lines
This is about registration. Right now you can only register IO intercepts in the chipset, not on a per-CPU basis. We could just as easily have:
CPUState {
MemoryRegionapic_region;
};
per_cpu_register_memory(env,&env->apic_region);Quoted 7 lines
Right. Or all memory per-cpu, with two sub-regions:
- global memory
- overlaid apic memoryfor this, we need to have well defined semantics for overlap (perhaps a priority argument to
memory_region_add_subregion).Or even
cpu_memory_region
|+-- global memory map (prio 0) | | | +-- RAM (prio 0) | | | +-- PCI (prio 1)Quoted 2 lines
Quoted 2 lines
It depends on the chipset and its configuration (via PAM e.g.) in what region which takes precedence. Fixed prios do not help here.
It's really layering.
Quoted 7 lines
To implement PAM in a robust way, you need a certain set of memory accesses to first flow through the chipset before going to the next location with the ability to intercept.
We do something rather weird today by changing registrations first saving the current registrations. It would be much more elegant to just intercept the I/O requests and redirect accordingly.
That's what I implemented already, though using the current API with some tweaks (filtering PhysMemClient) and then facing massive slot fragmentation problems on KVM side.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 23 lines · Avi Kivity
The current memory APIs (
cpu_register_io_memory,cpu_register_physical_memory) suffer from a number of drawbacks:- lack of centralized bookkeeping
- acpu_register_physical_memory()that overlaps an existing region will overwrite the preexisting region; but a followingcpu_unregister_physical_memory()will not restore things to status quo ante.
- coalescing and dirty logging are all cleared on unregistering, so the client has to re-issue them when re-registering
- lots of opaques
- no nesting
- if a region has multiple subregions that need different handling (different callbacks, RAM interspersed with MMIO) then client code has to deal with that manually
- we can't interpose code between a device and global memory handlingTo fix that, I propose an new API to replace the existing one:
#ifndef
MEMORY_H
#defineMEMORY_H
Quoted 7 lines
typedef struct MemoryRegionOps MemoryRegionOps;
typedef struct MemoryRegion MemoryRegion;typedef
uint32_t(*MemoryReadFunc)(MemoryRegion *mr,target_phys_addr_taddr);
typedef void (*MemoryWriteFunc)(MemoryRegion *mr,target_phys_addr_taddr,uint32_tdata);
The API should be 64-bit and needs to have a size associated with it.
Quoted 4 lines
struct MemoryRegionOps {
MemoryReadFunc readb, readw, readl;
MemoryWriteFunc writeb, writew, writel;
};
I'm not a fan of having per-access type function pointers.
Quoted 5 lines
struct MemoryRegion {
const MemoryRegionOps *ops;target_phys_addr_tsize;target_phys_addr_taddr;
};
Should include a flags argument for future expansion.
Quoted 3 lines
void memory_region_init(MemoryRegion *mr,
target_phys_addr_tsize);
What is this used for? It's not clear to me.
Quoted 3 lines
void memory_region_init_io(MemoryRegion *mr,
const MemoryRegionOps *ops,target_phys_addr_tsize);
How does one test a MemoryRegion to determine it's type?
Quoted 7 lines
void
memory_region_init_ram(MemoryRegion *mr, target_phys_addr_t size);
void memory_region_init_ram_ptr(MemoryRegion *mr,target_phys_addr_tsize,
void *ptr);
voidmemory_region_destroy(MemoryRegion *mr);
voidmemory_region_set_offset(MemoryRegion *mr, target_phys_addr_t offset);
What's "offset" mean?
Quoted 5 lines
void
memory_region_set_log(MemoryRegion *mr, bool log); voidmemory_region_clear_coalescing(MemoryRegion *mr); void memory_region_add_coalescing(MemoryRegion *mr,target_phys_addr_toffset,target_phys_addr_tsize);
I don't think it's worth while to try to fit coalescing into this API. It's a KVM specific hack. I think it's fine to be a hacked on API.
Quoted 35 lines
void memory_region_add_subregion(MemoryRegion *mr,
target_phys_addr_toffset,
MemoryRegion *subregion);
void memory_region_del_subregion(MemoryRegion *mr,target_phys_addr_toffset,
MemoryRegion *subregion);void
cpu_register_memory_region(MemoryRegion *mr, target_phys_addr_t addr); voidcpu_unregister_memory_region(MemoryRegion *mr);#endif
The API is nested: you can define, say, a PCI BAR containing RAM and MMIO, and give it to the PCI subsystem. PCI can then enable/disable the BAR and move it to different addresses without calling any callbacks; the client code can enable or disable logging or coalescing without caring if the BAR is mapped or not. For example:
MemoryRegion mr,
mr_mmio,mr_ram;
memory_region_init(&mr);memory_region_init_io(&mr_mmio, &mmio_ops, 0x1000);memory_region_init_ram(&mr_ram, 0x100000);memory_region_add_subregion(&mr, 0, &mr_ram);memory_region_add_subregion(&mr, 0x10000, &mr_io);memory_region_add_coalescing(&mr_ram, 0, 0x100000);pci_register_bar(&pci_dev, 0, &mr);at this point the PCI subsystem knows everything about the BAR and can enable or disable it, or move it around, without further help from the device code. On the other hand, the device code can change logging or coalescing, or even change the structure of the region, without caring about whether the region is currently registered or not.
Quoted 3 lines
If we can agree on the API, then I think the way forward is to implement it in terms of the old API, change over all devices, then fold the old API into the new one.
I think it pretty much makes sense to me. It might be worth while to allow memory region definitions to be static. For instance:
MemoryRegion e1000_bar = { ... };
Regards,
Anthony Liguori
Quoted 27 lines · Anthony Liguori
Quoted 26 lines · Avi Kivity
The current memory APIs (
cpu_register_io_memory,cpu_register_physical_memory) suffer from a number of drawbacks:- lack of centralized bookkeeping
- acpu_register_physical_memory()that overlaps an existing region will overwrite the preexisting region; but a followingcpu_unregister_physical_memory()will not restore things to status quo ante.
- coalescing and dirty logging are all cleared on unregistering, so the client has to re-issue them when re-registering
- lots of opaques
- no nesting
- if a region has multiple subregions that need different handling (different callbacks, RAM interspersed with MMIO) then client code has to deal with that manually
- we can't interpose code between a device and global memory handlingTo fix that, I propose an new API to replace the existing one:
#ifndef
MEMORY_H
#defineMEMORY_Htypedef struct MemoryRegionOps MemoryRegionOps;
typedef struct MemoryRegion MemoryRegion;
Quoted 7 lines
Quoted 4 lines
typedef
uint32_t(*MemoryReadFunc)(MemoryRegion *mr,target_phys_addr_taddr);
typedef void (*MemoryWriteFunc)(MemoryRegion *mr,target_phys_addr_taddr,uint32_tdata);The API should be 64-bit and needs to have a size associated with it.
That will make conversion hard. Maybe we can have a native with-size callback, and a helper that converts to the traditional form. Otherwise the effort to fully covert the tree increases dramatically.
Quoted 5 lines
Quoted 3 lines
void memory_region_init(MemoryRegion *mr,
target_phys_addr_tsize);What is this used for? It's not clear to me.
Empty container, fill with memory_region_add_subregion() to taste.
Quoted 6 lines
Quoted 4 lines
void memory_region_init_io(MemoryRegion *mr,
const MemoryRegionOps *ops,target_phys_addr_tsize);How does one test a MemoryRegion to determine it's type?
One doesn't. Of course the low-level will need to do this (via a private API that isn't exposed to devices).
Quoted 9 lines
Quoted 7 lines
void
memory_region_init_ram(MemoryRegion *mr, target_phys_addr_t size);
void memory_region_init_ram_ptr(MemoryRegion *mr,target_phys_addr_tsize,
void *ptr);
voidmemory_region_destroy(MemoryRegion *mr);
voidmemory_region_set_offset(MemoryRegion *mr, target_phys_addr_t offset);What's "offset" mean?
It's the equivalent of cpu_register_physical_memory_offset().
Note the intent is to have addresses always be relative to the innermost container. Perhaps we can get away without offset.
Quoted 8 lines
Quoted 6 lines
void
memory_region_set_log(MemoryRegion *mr, bool log); voidmemory_region_clear_coalescing(MemoryRegion *mr); void memory_region_add_coalescing(MemoryRegion *mr,target_phys_addr_toffset,target_phys_addr_tsize);I don't think it's worth while to try to fit coalescing into this API. It's a KVM specific hack. I think it's fine to be a hacked on API.
The problem is that only the device knows about coalescing, while the region can be mapped, unmapped, or moved without device knowledge. So if a PCI is unmapped and then remapped (possibly at a different address) we need to tell kvm about it, but there is no device callback involved.
Quoted 4 lines
I think it pretty much makes sense to me. It might be worth while to allow memory region definitions to be static. For instance:
MemoryRegion
e1000_bar= { ... };
It doesn't work in general, since multiple instances of e1000 will each need its own e1000_bar. So e1000_bar will be a member of the e1000 device state structure.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 9 lines · Avi Kivity
Quoted 9 lines
Quoted 8 lines
void
memory_region_init_ram(MemoryRegion *mr, target_phys_addr_t size);
void memory_region_init_ram_ptr(MemoryRegion *mr,target_phys_addr_tsize,
void *ptr);
voidmemory_region_destroy(MemoryRegion *mr);
voidmemory_region_set_offset(MemoryRegion *mr, target_phys_addr_t offset);
Quoted 6 lines
Quoted 1 line
What's "offset" mean?
It's the equivalent of
cpu_register_physical_memory_offset().Note the intent is to have addresses always be relative to the innermost container. Perhaps we can get away without offset.
Offset is supposed to support the transition of devices that expect absolute I/O addresses in their callbacks to those that are fine with relative ones (based on the region start). This API change is a good chance to finally get rid of the former group.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
void
memory_region_set_log(MemoryRegion *mr, bool log); voidmemory_region_clear_coalescing(MemoryRegion *mr); void memory_region_add_coalescing(MemoryRegion *mr,target_phys_addr_toffset,target_phys_addr_tsize);
Quoted 7 lines
Quoted 2 lines
I don't think it's worth while to try to fit coalescing into this API. It's a KVM specific hack. I think it's fine to be a hacked on API.
The problem is that only the device knows about coalescing, while the region can be mapped, unmapped, or moved without device knowledge. So if a PCI is unmapped and then remapped (possibly at a different address) we need to tell kvm about it, but there is no device callback involved.
Doesn't Xen use coalescing as well, or could use? It looks like a generic optimization feature for hypervisors that want to build on top of QEMU.
Jan
Am 18.05.2011 17:08, schrieb Anthony Liguori:
Quoted 33 lines · Avi Kivity
Quoted 23 lines
The current memory APIs (
cpu_register_io_memory,cpu_register_physical_memory) suffer from a number of drawbacks:- lack of centralized bookkeeping
- acpu_register_physical_memory()that overlaps an existing region will overwrite the preexisting region; but a followingcpu_unregister_physical_memory()will not restore things to status quo ante.
- coalescing and dirty logging are all cleared on unregistering, so the client has to re-issue them when re-registering
- lots of opaques
- no nesting
- if a region has multiple subregions that need different handling (different callbacks, RAM interspersed with MMIO) then client code has to deal with that manually
- we can't interpose code between a device and global memory handlingTo fix that, I propose an new API to replace the existing one:
#ifndef
MEMORY_H
#defineMEMORY_HQuoted 7 lines
typedef struct MemoryRegionOps MemoryRegionOps;
typedef struct MemoryRegion MemoryRegion;typedef
uint32_t(*MemoryReadFunc)(MemoryRegion *mr,target_phys_addr_taddr);
typedef void (*MemoryWriteFunc)(MemoryRegion *mr,target_phys_addr_taddr,uint32_tdata);
Quoted 8 lines
The API should be 64-bit and needs to have a size associated with it.
Quoted 4 lines
struct MemoryRegionOps {
MemoryReadFunc readb, readw, readl;
MemoryWriteFunc writeb, writew, writel;
};I'm not a fan of having per-access type function pointers.
Do you think of something like these declaration:
typedef uint64_t (*MemoryReadFunc)(MemoryRegion *mr, target_phys_addr_t addr, size_t size); typedef void (*MemoryWriteFunc)(MemoryRegion *mr, target_phys_addr_t addr, uint64_t data, size_tsize);
For 32 bit host / target, this would mean some unnecessary overhead.
What about passing values by address:
typedef void (*MemoryReadFunc)(MemoryRegion *mr, target_phys_addr_t addr, void *data, size_t size); typedef void (*MemoryWriteFunc)(MemoryRegion *mr, target_phys_addr_t addr, const void *data, size_t size);
If we keep per-access type function pointers, they should use individual prototypes for the different access types:
typedef uint8_t (*MemoryReadbFunc)(MemoryRegion *mr, target_phys_addr_t addr); typedef uint16_t (*MemoryReadwFunc)(MemoryRegion *mr, target_phys_addr_t addr); typedef uint32_t (*MemoryReadlFunc)(MemoryRegion *mr, target_phys_addr_t addr); typedef uint64_t (*MemoryReadllFunc)(MemoryRegion *mr, target_phys_addr_t addr);
...
Regards,
Stefan W.
Quoted 9 lines · Stefan Weil
Quoted 1 line
I'm not a fan of having per-access type function pointers.
Do you think of something like these declaration:
typedef
uint64_t(*MemoryReadFunc)(MemoryRegion *mr,target_phys_addr_taddr,size_tsize); typedef void (*MemoryWriteFunc)(MemoryRegion *mr,target_phys_addr_taddr,uint64_tdata,size_tsize);For 32 bit host / target, this would mean some unnecessary overhead.
Frankly, the overhead is pretty low. I think we can neglect it.
Quoted 5 lines
What about passing values by address:
typedef void (*MemoryReadFunc)(MemoryRegion *mr,
target_phys_addr_taddr, void *data,size_tsize); typedef void (*MemoryWriteFunc)(MemoryRegion *mr,target_phys_addr_taddr, const void *data,size_tsize);
Those void *s will be quite annoying. Especially on hosts which don't allow misaligned data, you'll never know how to reference them.
Quoted 9 lines
If we keep per-access type function pointers, they should use individual prototypes for the different access types:
typedef
uint8_t(*MemoryReadbFunc)(MemoryRegion *mr,target_phys_addr_taddr); typedefuint16_t(*MemoryReadwFunc)(MemoryRegion *mr,target_phys_addr_taddr); typedefuint32_t(*MemoryReadlFunc)(MemoryRegion *mr,target_phys_addr_taddr); typedefuint64_t(*MemoryReadllFunc)(MemoryRegion *mr,target_phys_addr_taddr);
...
I prefer having size as an argument.
Something else I though about:
void memory_region_set_access_sizes(MemoryRegion *mr, int min, int max);
if, for example, min=2 and max=4, then byte accesses will be emulated as word accesses (RMW for writes) and quad accesses will be emulated as two long accesses. So a device that emulates 32-bit registers can set min=max=4 and get all the other sizes for free.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 10 lines · Avi Kivity
Quoted 8 lines
If we keep per-access type function pointers, they should use individual prototypes for the different access types:
typedef
uint8_t(*MemoryReadbFunc)(MemoryRegion *mr,target_phys_addr_taddr); typedefuint16_t(*MemoryReadwFunc)(MemoryRegion *mr,target_phys_addr_taddr); typedefuint32_t(*MemoryReadlFunc)(MemoryRegion *mr,target_phys_addr_taddr); typedefuint64_t(*MemoryReadllFunc)(MemoryRegion *mr,target_phys_addr_taddr);
...I prefer having size as an argument.
The one thing that makes having these function pointers split apart nice is that it makes it easy to set policy for what different sized reads do.
E.g. for devices for which only 4 byte reads are defined, you only fill in readl, and let the other sizes cause a machine-check.
Alternately, for devices for which the fundamental size is 1 byte, but which does handle larger reads in a more-or-less memory-like fashion, you can fill in a common readw_via_readb function that does the composition for you, and without having to have code for that scattered through every device.
r~
Quoted 6 lines · Richard Henderson
Quoted 2 lines
I prefer having size as an argument.
The one thing that makes having these function pointers split apart nice is that it makes it easy to set policy for what different sized reads do.
Quoted 8 lines
E.g. for devices for which only 4 byte reads are defined, you only fill in readl, and let the other sizes cause a machine-check.
Alternately, for devices for which the fundamental size is 1 byte, but which does handle larger reads in a more-or-less memory-like fashion, you can fill in a common
readw_via_readbfunction that does the composition for you, and without having to have code for that scattered through every device.
I plan to centralize this logic in the memory API. You'll just declare how your device behaves.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 1 line · Avi Kivity
I plan to centralize this logic in the memory API. You'll just declare how your device behaves.
Excellent. Those were my only two concerns then.
r~
Quoted 3 lines · Avi Kivity
void
cpu_register_memory_region(MemoryRegion *mr, target_phys_addr_t addr); voidcpu_unregister_memory_region(MemoryRegion *mr);
These two can probably implemented as:
extern MemoryRegion memory_map;
void cpu_register_memory_region(MemoryRegion *mr,target_phys_addr_t addr)
{
memory_region_add_subregion(&memory_map, addr, mr);
}
etc.Eventually when everything is in a bus relationship they'll just go away.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 4 lines · Avi Kivity
struct MemoryRegionOps {
MemoryReadFunc readb, readw, readl;
MemoryWriteFunc writeb, writew, writel;
};Look back to last May for a discussion between myself and Paul Brook on this subject. That started with me merely wanting to expand the interface to support 8-byte reads/writes, and he wanting a fairly substantial reorganization.
I'm not married to Paul's total reorg, but please include readq/writeq support in any reorganization in this area.
r~
Quoted 10 lines · Richard Henderson
Quoted 4 lines · Avi Kivity
struct MemoryRegionOps { MemoryReadFunc readb, readw, readl; MemoryWriteFunc writeb, writew, writel;};
Look back to last May for a discussion between myself and Paul Brook on this subject. That started with me merely wanting to expand the interface to support 8-byte reads/writes, and he wanting a fairly substantial reorganization.
I wasn't able to find this conversation, sorry.
Quoted 2 lines
I'm not married to Paul's total reorg, but please include readq/writeq support in any reorganization in this area.
Certainly. Since I'm going with Anthony's single callback plus size, it will come naturally.
Signature · 1 line
error compiling committee.c: too many arguments to function
Copying kvm@.
Quoted 83 lines · Avi Kivity
The current memory APIs (
cpu_register_io_memory,cpu_register_physical_memory) suffer from a number of drawbacks:- lack of centralized bookkeeping
- acpu_register_physical_memory()that overlaps an existing region
will overwrite the preexisting region; but a followingcpu_unregister_physical_memory()will not restore things to status quo ante.
- coalescing and dirty logging are all cleared on unregistering, so
the client has to re-issue them when re-registering
- lots of opaques
- no nesting
- if a region has multiple subregions that need different handling
(different callbacks, RAM interspersed with MMIO) then client code has to deal with that manually
- we can't interpose code between a device and global memory handlingTo fix that, I propose an new API to replace the existing one:
#ifndef
MEMORY_H
#defineMEMORY_Htypedef struct MemoryRegionOps MemoryRegionOps;
typedef struct MemoryRegion MemoryRegion;typedef
uint32_t(*MemoryReadFunc)(MemoryRegion *mr,target_phys_addr_taddr); typedef void (*MemoryWriteFunc)(MemoryRegion *mr,target_phys_addr_taddr,uint32_t data);struct MemoryRegionOps { MemoryReadFunc readb, readw, readl; MemoryWriteFunc writeb, writew, writel; };struct MemoryRegion { const MemoryRegionOps *ops; target_phys_addr_t size; target_phys_addr_t addr; };void memory_region_init(MemoryRegion *mr,
target_phys_addr_t size);void memory_region_init_io(MemoryRegion *mr,
const MemoryRegionOps *ops,target_phys_addr_tsize);
void memory_region_init_ram(MemoryRegion *mr,target_phys_addr_t size); void memory_region_init_ram_ptr(MemoryRegion *mr, target_phys_addr_t size, void *ptr);void
memory_region_destroy(MemoryRegion *mr);
voidmemory_region_set_offset(MemoryRegion *mr, target_phys_addr_t offset); voidmemory_region_set_log(MemoryRegion *mr, bool log);
voidmemory_region_clear_coalescing(MemoryRegion *mr);
void memory_region_add_coalescing(MemoryRegion *mr,target_phys_addr_t offset, target_phys_addr_t size);void memory_region_add_subregion(MemoryRegion *mr,
target_phys_addr_t offset, MemoryRegion *subregion); void memory_region_del_subregion(MemoryRegion *mr, target_phys_addr_t offset, MemoryRegion *subregion);void
cpu_register_memory_region(MemoryRegion *mr, target_phys_addr_t addr); voidcpu_unregister_memory_region(MemoryRegion *mr);#endif
The API is nested: you can define, say, a PCI BAR containing RAM and MMIO, and give it to the PCI subsystem. PCI can then enable/disable the BAR and move it to different addresses without calling any callbacks; the client code can enable or disable logging or coalescing without caring if the BAR is mapped or not. For example:
MemoryRegion mr,
mr_mmio,mr_ram;memory_region_init(&mr); memory_region_init_io(&mr_mmio, &mmio_ops, 0x1000); memory_region_init_ram(&mr_ram, 0x100000); memory_region_add_subregion(&mr, 0, &mr_ram); memory_region_add_subregion(&mr, 0x10000, &mr_io); memory_region_add_coalescing(&mr_ram, 0, 0x100000); pci_register_bar(&pci_dev, 0, &mr);at this point the PCI subsystem knows everything about the BAR and can enable or disable it, or move it around, without further help from the device code. On the other hand, the device code can change logging or coalescing, or even change the structure of the region, without caring about whether the region is currently registered or not.
If we can agree on the API, then I think the way forward is to implement it in terms of the old API, change over all devices, then fold the old API into the new one.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 1 line · Avi Kivity
void
cpu_register_memory_region(MemoryRegion *mr, target_phys_addr_t addr);
OK, let's allow overlapping, but make it explicit:
void cpu_register_memory_region_overlap(MemoryRegion *mr,
target_phys_addr_t addr,
int priority);We need that ordering, so we need an interface. Regions registered via cpu_register_memory_region must not overlap with existing one or we will throw an hwerror. And they shall get a low default priority.
Jan
Quoted 5 lines · Jan Kiszka
Quoted 1 line · Avi Kivity
void
cpu_register_memory_region(MemoryRegion *mr, target_phys_addr_t addr);OK, let's allow overlapping, but make it explicit:
Quoted 8 lines
void cpu_register_memory_region_overlap(MemoryRegion *mr,
target_phys_addr_t addr, int priority);We need that ordering, so we need an interface. Regions registered via
cpu_register_memory_regionmust not overlap with existing one or we will throw an hwerror. And they shall get a low default priority.
PCI BARs can overlap with anything. So any region can overlap with any other region.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 11 lines · Avi Kivity
Quoted 10 lines · Jan Kiszka
Quoted 1 line · Avi Kivity
void cpu_register_memory_region(MemoryRegion *mr,
target_phys_addr_taddr);OK, let's allow overlapping, but make it explicit:
void cpu_register_memory_region_overlap(MemoryRegion *mr,
target_phys_addr_t addr, int priority);
Quoted 7 lines
Quoted 4 lines
We need that ordering, so we need an interface. Regions registered via
cpu_register_memory_regionmust not overlap with existing one or we will throw an hwerror. And they shall get a low default priority.PCI BARs can overlap with anything. So any region can overlap with any other region.
I know, but that result is unspecified anyway. The user (guest OS) can't expect any reasonable result. We rather need priorities for useful overlapping.
Jan
Quoted 17 lines · Jan Kiszka
Quoted 16 lines · Avi Kivity
Quoted 14 lines · Jan Kiszka
Quoted 1 line · Avi Kivity
void cpu_register_memory_region(MemoryRegion *mr,
target_phys_addr_taddr);OK, let's allow overlapping, but make it explicit:
void cpu_register_memory_region_overlap(MemoryRegion *mr,
target_phys_addr_t addr, int priority);We need that ordering, so we need an interface. Regions registered via
cpu_register_memory_regionmust not overlap with existing one or we will throw an hwerror. And they shall get a low default priority.
Quoted 6 lines
Quoted 2 lines
PCI BARs can overlap with anything. So any region can overlap with any other region.
I know, but that result is unspecified anyway. The user (guest OS) can't expect any reasonable result. We rather need priorities for useful overlapping.
Unspecified doesn't mean abort. It means we need to specify something (which translates to: we get to pick the priorities).
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 22 lines · Avi Kivity
Quoted 21 lines · Jan Kiszka
Quoted 5 lines · Avi Kivity
Quoted 4 lines · Jan Kiszka
Quoted 1 line · Avi Kivity
void cpu_register_memory_region(MemoryRegion *mr,
target_phys_addr_taddr);Quoted 7 lines
Quoted 7 lines
OK, let's allow overlapping, but make it explicit:
void cpu_register_memory_region_overlap(MemoryRegion *mr,
target_phys_addr_t addr, int priority);We need that ordering, so we need an interface. Regions registered
via
Quoted 1 line
Quoted 1 line
cpu_register_memory_regionmust not overlap with existing one orwe will
Quoted 4 lines
Quoted 2 lines
throw an hwerror. And they shall get a low default priority.
PCI BARs can overlap with anything. So any region can overlap with
any
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
other region.
I know, but that result is unspecified anyway. The user (guest OS) can't expect any reasonable result. We rather need priorities for useful overlapping.
Unspecified doesn't mean abort. It means we need to specify something (which translates to: we get to pick the priorities).
Of course, PCI bars would have to be registered via cpu_register_memory_region_overlap, just specifying the default priority. Here we know that overlapping can happen and is not a bug in the board emulation. I want to avoid that such use cases make overlapping generally legal, papering over real bugs.
Jan
Quoted 9 lines · Jan Kiszka
Quoted 3 lines
Unspecified doesn't mean abort. It means we need to specify something (which translates to: we get to pick the priorities).
Of course, PCI bars would have to be registered via
cpu_register_memory_region_overlap, just specifying the default priority. Here we know that overlapping can happen and is not a bug in the board emulation. I want to avoid that such use cases make overlapping generally legal, papering over real bugs.
But those are the majority of regions. There are a few extra RAM and ROM and fixed function regions, but these are hardly likely to clash.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 5 lines · Avi Kivity
Quoted 4 lines · Jan Kiszka
Quoted 3 lines
Unspecified doesn't mean abort. It means we need to specify something (which translates to: we get to pick the priorities).
Quoted 8 lines
Quoted 5 lines
Of course, PCI bars would have to be registered via
cpu_register_memory_region_overlap, just specifying the default priority. Here we know that overlapping can happen and is not a bug in the board emulation. I want to avoid that such use cases make overlapping generally legal, papering over real bugs.But those are the majority of regions. There are a few extra RAM and ROM and fixed function regions, but these are hardly likely to clash.
You are probably only thinking about x86, which does not provide the majority of QEMU devices.
Jan
Quoted 12 lines · Jan Kiszka
Quoted 11 lines · Avi Kivity
Quoted 9 lines · Jan Kiszka
Quoted 3 lines
Unspecified doesn't mean abort. It means we need to specify something (which translates to: we get to pick the priorities).
Of course, PCI bars would have to be registered via
cpu_register_memory_region_overlap, just specifying the default priority. Here we know that overlapping can happen and is not a bug in the board emulation. I want to avoid that such use cases make overlapping generally legal, papering over real bugs.
Quoted 5 lines
Quoted 2 lines
But those are the majority of regions. There are a few extra RAM and ROM and fixed function regions, but these are hardly likely to clash.
You are probably only thinking about x86, which does not provide the majority of QEMU devices.
The next version of the API will include an overlap property.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 8 lines · Jan Kiszka
Quoted 1 line · Avi Kivity
void
cpu_register_memory_region(MemoryRegion *mr, target_phys_addr_t addr);OK, let's allow overlapping, but make it explicit:
void cpu_register_memory_region_overlap(MemoryRegion *mr,
target_phys_addr_t addr, int priority);
The device doesn't actually know how overlapping is handled. This is based on the bus hierarchy.
Regards,
Anthony Liguori
Quoted 6 lines
We need that ordering, so we need an interface. Regions registered via
cpu_register_memory_regionmust not overlap with existing one or we will throw an hwerror. And they shall get a low default priority.Jan
Quoted 5 lines · Anthony Liguori
Quoted 4 lines · Jan Kiszka
Quoted 2 lines · Avi Kivity
void
cpu_register_memory_region(MemoryRegion *mr, target_phys_addr_t addr);
Quoted 8 lines
Quoted 5 lines
OK, let's allow overlapping, but make it explicit:
void cpu_register_memory_region_overlap(MemoryRegion *mr,
target_phys_addr_t addr, int priority);The device doesn't actually know how overlapping is handled. This is based on the bus hierarchy.
Devices don't register their regions, buses do.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 5 lines · Jan Kiszka
Quoted 2 lines
The device doesn't actually know how overlapping is handled. This is based on the bus hierarchy.
Devices don't register their regions, buses do.
Exactly. Devices may register sub-regions to describe complex BARs, but most will just create a region and hand it off to their bus.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 8 lines · Jan Kiszka
Quoted 7 lines · Anthony Liguori
Quoted 6 lines · Jan Kiszka
Quoted 2 lines · Avi Kivity
void
cpu_register_memory_region(MemoryRegion *mr, target_phys_addr_t addr);OK, let's allow overlapping, but make it explicit:
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
void cpu_register_memory_region_overlap(MemoryRegion *mr,
target_phys_addr_t addr, int priority);The device doesn't actually know how overlapping is handled. This is based on the bus hierarchy.
Devices don't register their regions, buses do.
Today PCI device may register region that overlaps with any other registered memory region without even knowing it. Guest can write any RAM address into PCI BAR and this RAM address will be come mmio are. More interesting is what happens when guest reprogram PCI BAR to other address
- the RAM that was at the previous address just disappears. Obviously this is crazy behaviour, but the question is how do we want to handle
it? One option is to disallow such overlapping registration, another is to restore RAM mapping after PCI BAR is reprogrammed. If we chose second one the PCI will not know that _overlap() should be called.
Another example may be APIC region and PCI. They overlap, but neither CPU nor PCI knows about it.
--
Gleb.
Quoted 13 lines · Gleb Natapov
Quoted 12 lines · Jan Kiszka
Quoted 11 lines · Anthony Liguori
Quoted 9 lines · Jan Kiszka
Quoted 2 lines · Avi Kivity
void
cpu_register_memory_region(MemoryRegion *mr, target_phys_addr_t addr);OK, let's allow overlapping, but make it explicit:
void cpu_register_memory_region_overlap(MemoryRegion *mr,
target_phys_addr_t addr, int priority);
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
The device doesn't actually know how overlapping is handled. This is based on the bus hierarchy.
Devices don't register their regions, buses do.
Today PCI device may register region that overlaps with any other registered memory region without even knowing it. Guest can write any RAM address into PCI BAR and this RAM address will be come mmio are.
Right, but this is not how a real machine works.
With the exception of the few regions that the chipset treats specially, RAM accesses don't get a chance to be intercepted by the PCI bus.
Quoted 4 lines
More
interesting is what happens when guest reprogram PCI BAR to other address
- the RAM that was at the previous address just disappears. Obviously this is crazy behaviour, but the question is how do we want to handle
The CPU should continue to access RAM at this address. It's unclear to me what the right behavior is for device-to-device I/O but I'm pretty certain it doesn't matter.
Quoted 6 lines
it? One option is to disallow such overlapping registration, another is to restore RAM mapping after PCI BAR is reprogrammed. If we chose second one the PCI will not know that
_overlap()should be called.Another example may be APIC region and PCI. They overlap, but neither CPU nor PCI knows about it.
And APIC always wins when accesses are coming from the CPU.
Regards,
Anthony Liguori
Quoted 4 lines
--
Gleb.
Quoted 17 lines · Anthony Liguori
Quoted 16 lines · Gleb Natapov
Quoted 15 lines · Jan Kiszka
Quoted 13 lines · Anthony Liguori
Quoted 9 lines · Jan Kiszka
Quoted 2 lines · Avi Kivity
void
cpu_register_memory_region(MemoryRegion *mr, target_phys_addr_t addr);OK, let's allow overlapping, but make it explicit:
void cpu_register_memory_region_overlap(MemoryRegion *mr,
target_phys_addr_t addr, int priority);The device doesn't actually know how overlapping is handled. This is based on the bus hierarchy.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Devices don't register their regions, buses do.
Today PCI device may register region that overlaps with any other registered memory region without even knowing it. Guest can write any RAM address into PCI BAR and this RAM address will be come mmio are.
Right, but this is not how a real machine works.
Very likely.
Quoted 10 lines
With the exception of the few regions that the chipset treats specially, RAM accesses don't get a chance to be intercepted by the PCI bus.
Quoted 4 lines
More
interesting is what happens when guest reprogram PCI BAR to other address
- the RAM that was at the previous address just disappears. Obviously this is crazy behaviour, but the question is how do we want to handleThe CPU should continue to access RAM at this address.
I think it should continue using it even when PCI BAR is still mapped to the RAM address.
Quoted 4 lines
It's unclear
to me what the right behavior is for device-to-device I/O but I'm pretty certain it doesn't matter.
For PC may be. But IIRC this thread already had request to support different memory view from different devices.
Quoted 9 lines
Quoted 6 lines
it? One option is to disallow such overlapping registration, another is to restore RAM mapping after PCI BAR is reprogrammed. If we chose second one the PCI will not know that
_overlap()should be called.Another example may be APIC region and PCI. They overlap, but neither CPU nor PCI knows about it.
And APIC always wins when accesses are coming from the CPU.
Of course, my question is how proposed API handles this.
--
Gleb.
Quoted 6 lines · Gleb Natapov
Quoted 5 lines · Anthony Liguori
On 05/19/2011 12:39 PM, Gleb Natapov wrote:
With the exception of the few regions that the chipset treats specially, RAM accesses don't get a chance to be intercepted by the PCI bus.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
More
interesting is what happens when guest reprogram PCI BAR to other address
- the RAM that was at the previous address just disappears. Obviously this is crazy behaviour, but the question is how do we want to handleThe CPU should continue to access RAM at this address.
I think it should continue using it even when PCI BAR is still mapped to the RAM address.
Agreed. Minus PAM and SMM, RAM is always RAM. When a CPU accesses it, it's always the same.
This is not the case for device accesses to RAM but this is probably a separate topic.
Quoted 6 lines
Quoted 4 lines
It's unclear
to me what the right behavior is for device-to-device I/O but I'm pretty certain it doesn't matter.For PC may be. But IIRC this thread already had request to support different memory view from different devices.
All platforms have in common the fact that I/O is dispatched in a hierarchical fashion. The reason is quite simple--physics only allow so many point-to-point connections between components. So I/O access is always going to fan out. The components closer to the source of the I/O access are always able to overrule the lower components.
Regards,
Anthony Liguori
Quoted 17 lines
Quoted 9 lines
Quoted 6 lines
it? One option is to disallow such overlapping registration, another is to restore RAM mapping after PCI BAR is reprogrammed. If we chose second one the PCI will not know that
_overlap()should be called.Another example may be APIC region and PCI. They overlap, but neither CPU nor PCI knows about it.
And APIC always wins when accesses are coming from the CPU.
Of course, my question is how proposed API handles this.
--
Gleb.
Quoted 18 lines · Gleb Natapov
Quoted 8 lines · Jan Kiszka
Quoted 7 lines · Anthony Liguori
Quoted 6 lines · Jan Kiszka
Quoted 2 lines · Avi Kivity
void
cpu_register_memory_region(MemoryRegion *mr, target_phys_addr_t addr);OK, let's allow overlapping, but make it explicit:
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
void cpu_register_memory_region_overlap(MemoryRegion *mr,
target_phys_addr_t addr, int priority);The device doesn't actually know how overlapping is handled. This is based on the bus hierarchy.
Devices don't register their regions, buses do.
Quoted 9 lines
Today PCI device may register region that overlaps with any other registered memory region without even knowing it. Guest can write any RAM address into PCI BAR and this RAM address will be come mmio are. More interesting is what happens when guest reprogram PCI BAR to other address
- the RAM that was at the previous address just disappears. Obviously this is crazy behaviour, but the question is how do we want to handle
it? One option is to disallow such overlapping registration, another is to restore RAM mapping after PCI BAR is reprogrammed. If we chose second one the PCI will not know that_overlap()should be called.
BARs may overlap with other BARs or with RAM. That's well-known, so PCI bridged need to register their regions with the _overlap variant unconditionally. In contrast to the current PhysPageDesc mechanism, the new region management will not cause any harm to overlapping regions so that they can "recover" when the overlap is gone.
Quoted 3 lines
Another example may be APIC region and PCI. They overlap, but neither CPU nor PCI knows about it.
And they do not need to. The APIC regions will be managed by the per-CPU region management, reusing the tool box we need for all bridges. It will register the APIC page with a priority higher than the default one, thus overriding everything that comes from the host bridge. I think that reflects pretty well real machine behaviour.
Jan
Quoted 29 lines · Jan Kiszka
Quoted 18 lines · Gleb Natapov
Quoted 8 lines · Jan Kiszka
Quoted 7 lines · Anthony Liguori
Quoted 6 lines · Jan Kiszka
Quoted 2 lines · Avi Kivity
void
cpu_register_memory_region(MemoryRegion *mr, target_phys_addr_t addr);OK, let's allow overlapping, but make it explicit:
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
void cpu_register_memory_region_overlap(MemoryRegion *mr,
target_phys_addr_t addr, int priority);The device doesn't actually know how overlapping is handled. This is based on the bus hierarchy.
Devices don't register their regions, buses do.
Quoted 9 lines
Today PCI device may register region that overlaps with any other registered memory region without even knowing it. Guest can write any RAM address into PCI BAR and this RAM address will be come mmio are. More interesting is what happens when guest reprogram PCI BAR to other address
- the RAM that was at the previous address just disappears. Obviously this is crazy behaviour, but the question is how do we want to handle
it? One option is to disallow such overlapping registration, another is to restore RAM mapping after PCI BAR is reprogrammed. If we chose second one the PCI will not know that_overlap()should be called.
Quoted 3 lines
BARs may overlap with other BARs or with RAM. That's well-known, so PCI bridged need to register their regions with the
_overlapvariant unconditionally. In contrast to the current PhysPageDesc mechanism, the
With what priority? If it needs to call _overlap unconditionally why not always call _overlap and drop not _overlap variant?
Quoted 4 lines
new region management will not cause any harm to overlapping regions so that they can "recover" when the overlap is gone.
Quoted 1 line
Quoted 9 lines
Quoted 2 lines
Another example may be APIC region and PCI. They overlap, but neither CPU nor PCI knows about it.
And they do not need to. The APIC regions will be managed by the per-CPU region management, reusing the tool box we need for all bridges. It will register the APIC page with a priority higher than the default one, thus overriding everything that comes from the host bridge. I think that reflects pretty well real machine behaviour.
What is "higher"? How does it know that priority is high enough? I thought, from reading other replies, that priorities are meaningful only on the same hierarchy level (which kinda make sense), but now you are saying that you will override PCI address from another part of the topology?
--
Gleb.
Quoted 30 lines · Gleb Natapov
Quoted 29 lines · Jan Kiszka
Quoted 27 lines · Gleb Natapov
Quoted 8 lines · Jan Kiszka
Quoted 7 lines · Anthony Liguori
Quoted 6 lines · Jan Kiszka
Quoted 2 lines · Avi Kivity
void
cpu_register_memory_region(MemoryRegion *mr, target_phys_addr_t addr);OK, let's allow overlapping, but make it explicit:
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
void cpu_register_memory_region_overlap(MemoryRegion *mr,
target_phys_addr_t addr, int priority);The device doesn't actually know how overlapping is handled. This is based on the bus hierarchy.
Devices don't register their regions, buses do.
Today PCI device may register region that overlaps with any other registered memory region without even knowing it. Guest can write any RAM address into PCI BAR and this RAM address will be come mmio are. More interesting is what happens when guest reprogram PCI BAR to other address
- the RAM that was at the previous address just disappears. Obviously this is crazy behaviour, but the question is how do we want to handle
it? One option is to disallow such overlapping registration, another is to restore RAM mapping after PCI BAR is reprogrammed. If we chose second one the PCI will not know that_overlap()should be called.
Quoted 5 lines
Quoted 3 lines
BARs may overlap with other BARs or with RAM. That's well-known, so PCI bridged need to register their regions with the
_overlapvariant unconditionally. In contrast to the current PhysPageDesc mechanism, theWith what priority? If it needs to call
_overlapunconditionally why not always call_overlapand drop not_overlapvariant?
Because we should catch accidental overlaps in all those non PCI devices with hard-wired addressing. That's a bug in the device/machine model and should be reported as such by QEMU.
Quoted 8 lines
Quoted 8 lines
new region management will not cause any harm to overlapping regions so that they can "recover" when the overlap is gone.
Quoted 3 lines
Another example may be APIC region and PCI. They overlap, but neither CPU nor PCI knows about it.
Quoted 7 lines
Quoted 6 lines
And they do not need to. The APIC regions will be managed by the per-CPU region management, reusing the tool box we need for all bridges. It will register the APIC page with a priority higher than the default one, thus overriding everything that comes from the host bridge. I think that reflects pretty well real machine behaviour.
What is "higher"? How does it know that priority is high enough?
Because no one else manages priorities at a specific hierarchy level. There is only one.
Quoted 5 lines
I
thought, from reading other replies, that priorities are meaningful only on the same hierarchy level (which kinda make sense), but now you are saying that you will override PCI address from another part of the topology?
Everything from below in the hierarchy is fed in with default priority, the lowest one. So to let some region created at this level override those regions, just pick default+1. If you want to create more overlay levels (can't imagine a good scenario, though), pick default+1..default+n. It's really that simple.
Jan
Quoted 34 lines · Jan Kiszka
Quoted 33 lines · Gleb Natapov
Quoted 32 lines · Jan Kiszka
Quoted 18 lines · Gleb Natapov
Quoted 8 lines · Jan Kiszka
Quoted 7 lines · Anthony Liguori
Quoted 6 lines · Jan Kiszka
Quoted 2 lines · Avi Kivity
void
cpu_register_memory_region(MemoryRegion *mr, target_phys_addr_t addr);OK, let's allow overlapping, but make it explicit:
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
void cpu_register_memory_region_overlap(MemoryRegion *mr,
target_phys_addr_t addr, int priority);The device doesn't actually know how overlapping is handled. This is based on the bus hierarchy.
Devices don't register their regions, buses do.
Quoted 9 lines
Today PCI device may register region that overlaps with any other registered memory region without even knowing it. Guest can write any RAM address into PCI BAR and this RAM address will be come mmio are. More interesting is what happens when guest reprogram PCI BAR to other address
- the RAM that was at the previous address just disappears. Obviously this is crazy behaviour, but the question is how do we want to handle
it? One option is to disallow such overlapping registration, another is to restore RAM mapping after PCI BAR is reprogrammed. If we chose second one the PCI will not know that_overlap()should be called.BARs may overlap with other BARs or with RAM. That's well-known, so PCI bridged need to register their regions with the
_overlapvariant unconditionally. In contrast to the current PhysPageDesc mechanism, the
Quoted 6 lines
Quoted 2 lines
With what priority? If it needs to call
_overlapunconditionally why not always call_overlapand drop not_overlapvariant?Because we should catch accidental overlaps in all those non PCI devices with hard-wired addressing. That's a bug in the device/machine model and should be reported as such by QEMU.
Why should we complicate API to catch unlikely errors? If you want to debug that add capability to dump memory map from the monitor.
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
new region management will not cause any harm to overlapping regions so that they can "recover" when the overlap is gone.
Quoted 3 lines
Another example may be APIC region and PCI. They overlap, but neither CPU nor PCI knows about it.
And they do not need to. The APIC regions will be managed by the per-CPU region management, reusing the tool box we need for all bridges. It will register the APIC page with a priority higher than the default one, thus overriding everything that comes from the host bridge. I think that reflects pretty well real machine behaviour.
Quoted 5 lines
Quoted 1 line
What is "higher"? How does it know that priority is high enough?
Because no one else manages priorities at a specific hierarchy level. There is only one.
PCI and CPU are on different hierarchy levels. PCI is under the PIIX and CPU is on a system BUS.
Quoted 6 lines
Quoted 5 lines
I
thought, from reading other replies, that priorities are meaningful only on the same hierarchy level (which kinda make sense), but now you are saying that you will override PCI address from another part of the topology?
Quoted 6 lines
Everything from below in the hierarchy is fed in with default priority, the lowest one. So to let some region created at this level override those regions, just pick default+1. If you want to create more overlay levels (can't imagine a good scenario, though), pick default+1..default+n. It's really that simple.
Except that PCI and CPU are not on the same level.
--
Gleb.
Quoted 35 lines · Gleb Natapov
Quoted 34 lines · Jan Kiszka
Quoted 33 lines · Gleb Natapov
Quoted 32 lines · Jan Kiszka
Quoted 18 lines · Gleb Natapov
Quoted 8 lines · Jan Kiszka
Quoted 7 lines · Anthony Liguori
Quoted 6 lines · Jan Kiszka
Quoted 2 lines · Avi Kivity
void
cpu_register_memory_region(MemoryRegion *mr, target_phys_addr_t addr);OK, let's allow overlapping, but make it explicit:
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
void cpu_register_memory_region_overlap(MemoryRegion *mr,
target_phys_addr_t addr, int priority);The device doesn't actually know how overlapping is handled. This is based on the bus hierarchy.
Devices don't register their regions, buses do.
Quoted 9 lines
Today PCI device may register region that overlaps with any other registered memory region without even knowing it. Guest can write any RAM address into PCI BAR and this RAM address will be come mmio are. More interesting is what happens when guest reprogram PCI BAR to other address
- the RAM that was at the previous address just disappears. Obviously this is crazy behaviour, but the question is how do we want to handle
it? One option is to disallow such overlapping registration, another is to restore RAM mapping after PCI BAR is reprogrammed. If we chose second one the PCI will not know that_overlap()should be called.BARs may overlap with other BARs or with RAM. That's well-known, so PCI bridged need to register their regions with the
_overlapvariant unconditionally. In contrast to the current PhysPageDesc mechanism, the
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
With what priority? If it needs to call
_overlapunconditionally why not always call_overlapand drop not_overlapvariant?Because we should catch accidental overlaps in all those non PCI devices with hard-wired addressing. That's a bug in the device/machine model and should be reported as such by QEMU.
Why should we complicate API to catch unlikely errors? If you want to debug that add capability to dump memory map from the monitor.
Because we need to switch tons of code that so far saw a fairly different reaction of the core to overlapping regions.
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
new region management will not cause any harm to overlapping regions so that they can "recover" when the overlap is gone.
Quoted 3 lines
Another example may be APIC region and PCI. They overlap, but neither CPU nor PCI knows about it.
And they do not need to. The APIC regions will be managed by the per-CPU region management, reusing the tool box we need for all bridges. It will register the APIC page with a priority higher than the default one, thus overriding everything that comes from the host bridge. I think that reflects pretty well real machine behaviour.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
What is "higher"? How does it know that priority is high enough?
Because no one else manages priorities at a specific hierarchy level. There is only one.
PCI and CPU are on different hierarchy levels. PCI is under the PIIX and CPU is on a system BUS.
The priority for the APIC mapping will be applied at CPU level, of course. So it will override everything, not just PCI.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
I
thought, from reading other replies, that priorities are meaningful only on the same hierarchy level (which kinda make sense), but now you are saying that you will override PCI address from another part of the topology?
Quoted 7 lines
Quoted 6 lines
Everything from below in the hierarchy is fed in with default priority, the lowest one. So to let some region created at this level override those regions, just pick default+1. If you want to create more overlay levels (can't imagine a good scenario, though), pick default+1..default+n. It's really that simple.
Except that PCI and CPU are not on the same level.
See above.
Jan
Quoted 39 lines · Jan Kiszka
Quoted 38 lines · Gleb Natapov
Quoted 37 lines · Jan Kiszka
Quoted 35 lines · Gleb Natapov
Quoted 29 lines · Jan Kiszka
Quoted 18 lines · Gleb Natapov
Quoted 8 lines · Jan Kiszka
Quoted 7 lines · Anthony Liguori
Quoted 6 lines · Jan Kiszka
Quoted 2 lines · Avi Kivity
void
cpu_register_memory_region(MemoryRegion *mr, target_phys_addr_t addr);OK, let's allow overlapping, but make it explicit:
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
void cpu_register_memory_region_overlap(MemoryRegion *mr,
target_phys_addr_t addr, int priority);The device doesn't actually know how overlapping is handled. This is based on the bus hierarchy.
Devices don't register their regions, buses do.
Quoted 9 lines
Today PCI device may register region that overlaps with any other registered memory region without even knowing it. Guest can write any RAM address into PCI BAR and this RAM address will be come mmio are. More interesting is what happens when guest reprogram PCI BAR to other address
- the RAM that was at the previous address just disappears. Obviously this is crazy behaviour, but the question is how do we want to handle
it? One option is to disallow such overlapping registration, another is to restore RAM mapping after PCI BAR is reprogrammed. If we chose second one the PCI will not know that_overlap()should be called.Quoted 3 lines
BARs may overlap with other BARs or with RAM. That's well-known, so PCI bridged need to register their regions with the
_overlapvariant unconditionally. In contrast to the current PhysPageDesc mechanism, theWith what priority? If it needs to call
_overlapunconditionally why not always call_overlapand drop not_overlapvariant?
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
Because we should catch accidental overlaps in all those non PCI devices with hard-wired addressing. That's a bug in the device/machine model and should be reported as such by QEMU.
Why should we complicate API to catch unlikely errors? If you want to debug that add capability to dump memory map from the monitor.
Because we need to switch tons of code that so far saw a fairly different reaction of the core to overlapping regions.
How so? Today if there is accidental overlap device will not function properly. With new API it will be the same.
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 7 lines
new region management will not cause any harm to overlapping regions so that they can "recover" when the overlap is gone.
Quoted 1 line
Quoted 9 lines
Quoted 2 lines
Another example may be APIC region and PCI. They overlap, but neither CPU nor PCI knows about it.
And they do not need to. The APIC regions will be managed by the per-CPU region management, reusing the tool box we need for all bridges. It will register the APIC page with a priority higher than the default one, thus overriding everything that comes from the host bridge. I think that reflects pretty well real machine behaviour.
What is "higher"? How does it know that priority is high enough?
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
Because no one else manages priorities at a specific hierarchy level. There is only one.
PCI and CPU are on different hierarchy levels. PCI is under the PIIX and CPU is on a system BUS.
The priority for the APIC mapping will be applied at CPU level, of course. So it will override everything, not just PCI.
So you do not need explicit priority because the place in hierarchy implicitly provides you with one.
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 6 lines
I
thought, from reading other replies, that priorities are meaningful only on the same hierarchy level (which kinda make sense), but now you are saying that you will override PCI address from another part of the topology?Everything from below in the hierarchy is fed in with default priority, the lowest one. So to let some region created at this level override those regions, just pick default+1. If you want to create more overlay levels (can't imagine a good scenario, though), pick default+1..default+n. It's really that simple.
Quoted 6 lines
Quoted 1 line
Except that PCI and CPU are not on the same level.
See above.
Jan
--
Gleb.
Quoted 43 lines · Gleb Natapov
Quoted 42 lines · Jan Kiszka
Quoted 41 lines · Gleb Natapov
Quoted 40 lines · Jan Kiszka
Quoted 30 lines · Gleb Natapov
Quoted 29 lines · Jan Kiszka
Quoted 27 lines · Gleb Natapov
Quoted 8 lines · Jan Kiszka
Quoted 7 lines · Anthony Liguori
Quoted 6 lines · Jan Kiszka
Quoted 2 lines · Avi Kivity
void
cpu_register_memory_region(MemoryRegion *mr, target_phys_addr_t addr);OK, let's allow overlapping, but make it explicit:
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
void cpu_register_memory_region_overlap(MemoryRegion *mr,
target_phys_addr_t addr, int priority);The device doesn't actually know how overlapping is handled. This is based on the bus hierarchy.
Devices don't register their regions, buses do.
Today PCI device may register region that overlaps with any other registered memory region without even knowing it. Guest can write any RAM address into PCI BAR and this RAM address will be come mmio are. More interesting is what happens when guest reprogram PCI BAR to other address
- the RAM that was at the previous address just disappears. Obviously this is crazy behaviour, but the question is how do we want to handle
it? One option is to disallow such overlapping registration, another is to restore RAM mapping after PCI BAR is reprogrammed. If we chose second one the PCI will not know that_overlap()should be called.Quoted 5 lines
Quoted 3 lines
BARs may overlap with other BARs or with RAM. That's well-known, so PCI bridged need to register their regions with the
_overlapvariant unconditionally. In contrast to the current PhysPageDesc mechanism, theWith what priority? If it needs to call
_overlapunconditionally why not always call_overlapand drop not_overlapvariant?Because we should catch accidental overlaps in all those non PCI devices with hard-wired addressing. That's a bug in the device/machine model and should be reported as such by QEMU.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Why should we complicate API to catch unlikely errors? If you want to debug that add capability to dump memory map from the monitor.
Because we need to switch tons of code that so far saw a fairly different reaction of the core to overlapping regions.
How so? Today if there is accidental overlap device will not function properly. With new API it will be the same.
I rather expect subtle differences as overlapping registration changes existing regions, in the future those will recover.
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 11 lines
Quoted 11 lines
new region management will not cause any harm to overlapping regions so that they can "recover" when the overlap is gone.
Quoted 3 lines
Another example may be APIC region and PCI. They overlap, but neither CPU nor PCI knows about it.
Quoted 7 lines
Quoted 6 lines
And they do not need to. The APIC regions will be managed by the per-CPU region management, reusing the tool box we need for all bridges. It will register the APIC page with a priority higher than the default one, thus overriding everything that comes from the host bridge. I think that reflects pretty well real machine behaviour.
What is "higher"? How does it know that priority is high enough?
Because no one else manages priorities at a specific hierarchy level. There is only one.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
PCI and CPU are on different hierarchy levels. PCI is under the PIIX and CPU is on a system BUS.
The priority for the APIC mapping will be applied at CPU level, of course. So it will override everything, not just PCI.
So you do not need explicit priority because the place in hierarchy implicitly provides you with one.
Yes. Alternatively, you could add a prio offset to all mappings when climbing one level up, provided that offset is smaller than the prio range locally available to each level.
Jan
Quoted 3 lines · Jan Kiszka
Alternatively, you could add a prio offset to all mappings when climbing one level up, provided that offset is smaller than the prio range locally available to each level.
Err, forget that, wrong analogy. It's more a round up after flattening the view.
Jan
Quoted 6 lines · Jan Kiszka
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Because we should catch accidental overlaps in all those non PCI devices with hard-wired addressing. That's a bug in the device/machine model and should be reported as such by QEMU.
Why should we complicate API to catch unlikely errors? If you want to debug that add capability to dump memory map from the monitor.
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
Because we need to switch tons of code that so far saw a fairly different reaction of the core to overlapping regions.
How so? Today if there is accidental overlap device will not function properly. With new API it will be the same.
I rather expect subtle differences as overlapping registration changes existing regions, in the future those will recover.
Where do you expect the differences will come from? Conversion to the new API shouldn't change the order of the registration and if the last registration will override previous one the end result should be the same as we have today.
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
new region management will not cause any harm to overlapping regions so that they can "recover" when the overlap is gone.
Quoted 3 lines
Another example may be APIC region and PCI. They overlap, but neither CPU nor PCI knows about it.
And they do not need to. The APIC regions will be managed by the per-CPU region management, reusing the tool box we need for all bridges. It will register the APIC page with a priority higher than the default one, thus overriding everything that comes from the host bridge. I think that reflects pretty well real machine behaviour.
Quoted 5 lines
Quoted 1 line
What is "higher"? How does it know that priority is high enough?
Because no one else manages priorities at a specific hierarchy level. There is only one.
PCI and CPU are on different hierarchy levels. PCI is under the PIIX and CPU is on a system BUS.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
The priority for the APIC mapping will be applied at CPU level, of course. So it will override everything, not just PCI.
So you do not need explicit priority because the place in hierarchy implicitly provides you with one.
Yes.
OK :) So you agree that we can do without priorities :)
Quoted 4 lines
Alternatively, you could add a prio offset to all mappings when
climbing one level up, provided that offset is smaller than the prio range locally available to each level.
Then a memory region final priority will depend on a tree height. If two disjointed tree branches of different height will claim the same memory region the higher one will have higher priority. I think this priority management is a can of worms.
Only the lowest level (aka system bus) will use memory API directly. PCI device will call PCI subsystem. PCI subsystem, instead of assigning arbitrary priorities to all overlappings, may just resolve them and pass flattened view to the chipset. Chipset in turn will look for overlappings between PCI memory areas and RAM/ISA/other memory areas that are outside of PCI windows and resolve all those passing the flattened view to system bus where APIC/PCI conflict will be resolved and finally memory API will be used to create memory map. In such a model I do not see the need for priorities. All overlappings are resolved in the most logical place, the one that has the best knowledge about how to resolve the conflict. The will be no code duplication. Overlapping resolution code will be in separate library used by all layers.
--
Gleb.
Quoted 13 lines · Gleb Natapov
Quoted 12 lines · Jan Kiszka
Quoted 11 lines
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
Because we should catch accidental overlaps in all those non PCI devices with hard-wired addressing. That's a bug in the device/machine model and should be reported as such by QEMU.
Why should we complicate API to catch unlikely errors? If you want to debug that add capability to dump memory map from the monitor.
Because we need to switch tons of code that so far saw a fairly different reaction of the core to overlapping regions.
How so? Today if there is accidental overlap device will not function properly. With new API it will be the same.
Quoted 7 lines
Quoted 3 lines
I rather expect subtle differences as overlapping registration changes existing regions, in the future those will recover.
Where do you expect the differences will come from? Conversion to the new API shouldn't change the order of the registration and if the last registration will override previous one the end result should be the same as we have today.
A) Removing regions will change significantly. So far this is done by setting a region to IO_MEM_UNASSIGNED, keeping truncation. With the new API that will be a true removal which will additionally restore hidden regions.
B) Uncontrolled overlapping is a bug that should be caught by the core, and a new API is a perfect chance to do this.
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
new region management will not cause any harm to overlapping regions so that they can "recover" when the overlap is gone.
Quoted 3 lines
Another example may be APIC region and PCI. They overlap, but neither CPU nor PCI knows about it.
And they do not need to. The APIC regions will be managed by the per-CPU region management, reusing the tool box we need for all bridges. It will register the APIC page with a priority higher than the default one, thus overriding everything that comes from the host bridge. I think that reflects pretty well real machine behaviour.
Quoted 5 lines
Quoted 1 line
What is "higher"? How does it know that priority is high enough?
Because no one else manages priorities at a specific hierarchy level. There is only one.
PCI and CPU are on different hierarchy levels. PCI is under the PIIX and CPU is on a system BUS.
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
The priority for the APIC mapping will be applied at CPU level, of course. So it will override everything, not just PCI.
So you do not need explicit priority because the place in hierarchy implicitly provides you with one.
Yes.
OK :) So you agree that we can do without priorities :)
Nope, see below how your own example depends on them.
Quoted 9 lines
Quoted 5 lines
Alternatively, you could add a prio offset to all mappings when
climbing one level up, provided that offset is smaller than the prio range locally available to each level.Then a memory region final priority will depend on a tree height. If two disjointed tree branches of different height will claim the same memory region the higher one will have higher priority. I think this priority management is a can of worms.
It is not as it remains a pure local thing and helps implementing the sketched scenarios. Believe, I tried to fix PAM/SMRAM already.
Quoted 2 lines
Only the lowest level (aka system bus) will use memory API directly.
Not necessarily. It depends on how much added value buses like PCI or ISA or whatever can offer for managing I/O regions. For some purposes, it may as well be fine to just call the memory_* service directly and pass the result of some operation to the bus API later on.
Quoted 3 lines
PCI
device will call PCI subsystem. PCI subsystem, instead of assigning arbitrary priorities to all overlappings,
Again: PCI will _not_ assign arbitrary priorities but only MEMORY_REGION_DEFAULT_PRIORITY, likely 0.
Quoted 10 lines
may just resolve them and pass
flattened view to the chipset. Chipset in turn will look for overlappings between PCI memory areas and RAM/ISA/other memory areas that are outside of PCI windows and resolve all those passing the flattened view to system bus where APIC/PCI conflict will be resolved and finally memory API will be used to create memory map. In such a model I do not see the need for priorities. All overlappings are resolved in the most logical place, the one that has the best knowledge about how to resolve the conflict. The will be no code duplication. Overlapping resolution code will be in separate library used by all layers.
That does not specify how the PCI bridge or the chipset will tell that overlapping resolution lib _how_ overlapping regions shall be translated into a flat representation. And precisely here come priorities into play. It is the way to tell that lib either "region A shall override region B" if A has higher prio or "if region A and B overlap, do whatever you want" if both have the same prio.
Jan
Quoted 22 lines · Jan Kiszka
Quoted 13 lines · Gleb Natapov
Quoted 12 lines · Jan Kiszka
Quoted 11 lines
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
Because we should catch accidental overlaps in all those non PCI devices with hard-wired addressing. That's a bug in the device/machine model and should be reported as such by QEMU.
Why should we complicate API to catch unlikely errors? If you want to debug that add capability to dump memory map from the monitor.
Because we need to switch tons of code that so far saw a fairly different reaction of the core to overlapping regions.
How so? Today if there is accidental overlap device will not function properly. With new API it will be the same.
Quoted 7 lines
Quoted 3 lines
I rather expect subtle differences as overlapping registration changes existing regions, in the future those will recover.
Where do you expect the differences will come from? Conversion to the new API shouldn't change the order of the registration and if the last registration will override previous one the end result should be the same as we have today.
Quoted 5 lines
A) Removing regions will change significantly. So far this is done by setting a region to
IO_MEM_UNASSIGNED, keeping truncation. With the new API that will be a true removal which will additionally restore hidden regions.
And what problem do you expect may arise from that? Currently accessing such region after unassign will result in undefined behaviour, so this code is non working today, you can't make it worse.
Quoted 3 lines
B) Uncontrolled overlapping is a bug that should be caught by the core, and a new API is a perfect chance to do this.
Well, this will indeed introduce the difference in behaviour :) The guest that ran before will abort now. Are you actually aware of any such overlaps in the current code base?
But if priorities are gona stay why not fail if two regions with the same priority overlap? If that happens it means that the memory creation didn't pass the point where conflict should have been resolved (by assigning different priorities) and this means that overlap is unintentional, no?
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
new region management will not cause any harm to overlapping regions so that they can "recover" when the overlap is gone.
Quoted 3 lines
Another example may be APIC region and PCI. They overlap, but neither CPU nor PCI knows about it.
And they do not need to. The APIC regions will be managed by the per-CPU region management, reusing the tool box we need for all bridges. It will register the APIC page with a priority higher than the default one, thus overriding everything that comes from the host bridge. I think that reflects pretty well real machine behaviour.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
What is "higher"? How does it know that priority is high enough?
Because no one else manages priorities at a specific hierarchy level. There is only one.
PCI and CPU are on different hierarchy levels. PCI is under the PIIX and CPU is on a system BUS.
The priority for the APIC mapping will be applied at CPU level, of course. So it will override everything, not just PCI.
Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
So you do not need explicit priority because the place in hierarchy implicitly provides you with one.
Yes.
OK :) So you agree that we can do without priorities :)
Nope, see below how your own example depends on them.
It depends on them in very defined way. Only layer that knows exactly what is going on defines priorities. The priorities do not leak on any other level or global database. It is different from propagating priority from PCI BAR to core memory API.
I am starting to see how you can represent all this local decisions as priority numbers and then travel this weighted tree to find what memory region should be accessed (memory registration _has_ to be hierarchical for that to work in meaningful way). I still don't see why it is better than flattening the tree in the point of conflict.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Alternatively, you could add a prio offset to all mappings when
climbing one level up, provided that offset is smaller than the prio range locally available to each level.
Quoted 7 lines
Quoted 4 lines
Then a memory region final priority will depend on a tree height. If two disjointed tree branches of different height will claim the same memory region the higher one will have higher priority. I think this priority management is a can of worms.
It is not as it remains a pure local thing and helps implementing the sketched scenarios. Believe, I tried to fix PAM/SMRAM already.
If it remains local thing then I misunderstand what do you mean by "could add a prio offset to all mappings when climbing one level up". Doesn't sound like local things to me any more.
What problem did you have with PAM except low number of KVM slots btw?
Quoted 8 lines
Quoted 3 lines
Only the lowest level (aka system bus) will use memory API directly.
Not necessarily. It depends on how much added value buses like PCI or ISA or whatever can offer for managing I/O regions. For some purposes, it may as well be fine to just call the memory_* service directly and pass the result of some operation to the bus API later on.
Depend on what memory_* service you are talking about. Just creating unattached memory region is OK. But if two independent pieces of code want to map two different memory regions into the same phys address I do not see who will resolve the conflict.
Quoted 7 lines
Quoted 4 lines
PCI
device will call PCI subsystem. PCI subsystem, instead of assigning arbitrary priorities to all overlappings,Again: PCI will _not_ assign arbitrary priorities but only
MEMORY_REGION_DEFAULT_PRIORITY, likely 0.
That is as arbitrary as it can get. Just assigning MEMORY_REGION_DEFAULT_PRIORITY/2^0xfff will work equally well, so what is not arbitrary about that number?
BTW why wouldn't PCI layer assign different priorities to overlapping regions to let the core know which one should be actually available? Why leave this decision to the core if it is clearly belongs to PCI?
Quoted 12 lines
Quoted 11 lines
may just resolve them and pass
flattened view to the chipset. Chipset in turn will look for overlappings between PCI memory areas and RAM/ISA/other memory areas that are outside of PCI windows and resolve all those passing the flattened view to system bus where APIC/PCI conflict will be resolved and finally memory API will be used to create memory map. In such a model I do not see the need for priorities. All overlappings are resolved in the most logical place, the one that has the best knowledge about how to resolve the conflict. The will be no code duplication. Overlapping resolution code will be in separate library used by all layers.
Quoted 7 lines
That does not specify how the PCI bridge or the chipset will tell that overlapping resolution lib _how_ overlapping regions shall be translated into a flat representation. And precisely here come priorities into play. It is the way to tell that lib either "region A shall override region B" if A has higher prio or "if region A and B overlap, do whatever you want" if both have the same prio.
Yep! And the question is why shouldn't this be done on the level that knows most about the conflict but propagated to the core. I am not arguing that priorities do not exists! Obviously they are. I am questioning the usefulness of priorities be part of memory core API.
--
Gleb.
Quoted 10 lines · Gleb Natapov
Quoted 6 lines
A) Removing regions will change significantly. So far this is done by setting a region to
IO_MEM_UNASSIGNED, keeping truncation. With the new API that will be a true removal which will additionally restore hidden regions.And what problem do you expect may arise from that? Currently accessing such region after unassign will result in undefined behaviour, so this code is non working today, you can't make it worse.
If the conversion were perfect then yes. However there is a possibility that the conversion will not be perfect.
It's also good to have to have the code document its intentions. If you see _overlap() you know there is dynamic address decoding going on, or something clever.
Quoted 6 lines
Quoted 3 lines
B) Uncontrolled overlapping is a bug that should be caught by the core, and a new API is a perfect chance to do this.
Well, this will indeed introduce the difference in behaviour :) The guest that ran before will abort now. Are you actually aware of any such overlaps in the current code base?
Put a BAR over another BAR, then unmap it.
Quoted 5 lines
But if priorities are gona stay why not fail if two regions with the same priority overlap? If that happens it means that the memory creation didn't pass the point where conflict should have been resolved (by assigning different priorities) and this means that overlap is unintentional, no?
It may be intentional, as in the case of PCI, or PAM (though you can do PAM without priorities, by removing all but one of the subregions in the area).
Quoted 4 lines
I am starting to see how you can represent all this local decisions as priority numbers and then travel this weighted tree to find what memory region should be accessed (memory registration _has_ to be hierarchical for that to work in meaningful way).
Priorities don't travel up the tree. They're used to resolve local conflicts *only*.
Quoted 2 lines
I still don't see why it is better
than flattening the tree in the point of conflict.
How do you decide which subregion wins?
Quoted 8 lines
Quoted 4 lines
Not necessarily. It depends on how much added value buses like PCI or ISA or whatever can offer for managing I/O regions. For some purposes, it may as well be fine to just call the memory_* service directly and pass the result of some operation to the bus API later on.
Depend on what memory_* service you are talking about. Just creating unattached memory region is OK. But if two independent pieces of code want to map two different memory regions into the same phys address I do not see who will resolve the conflict.
They have to ask the bus to _add_subregion(). Only the bus knows about the priorities (or the bus can ask them to create the subregions).
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines
PCI
device will call PCI subsystem. PCI subsystem, instead of assigning arbitrary priorities to all overlappings,Again: PCI will _not_ assign arbitrary priorities but only
MEMORY_REGION_DEFAULT_PRIORITY, likely 0.That is as arbitrary as it can get. Just assigning MEMORY_REGION_DEFAULT_PRIORITY/2^0xfff will work equally well, so what is not arbitrary about that number?
That's just splitting hairs. Array indexes start from zero, an arbitrary but convenient number.
Quoted 3 lines
BTW why wouldn't PCI layer assign different priorities to overlapping regions to let the core know which one should be actually available? Why leave this decision to the core if it is clearly belongs to PCI?
You mean overlapping BARs? If PCI wants BAR 1 to override BAR 2, then it can indicate it with priorities. If it doesn't want to, it can use the same priority for all regions.
Quoted 8 lines
Quoted 8 lines
That does not specify how the PCI bridge or the chipset will tell that overlapping resolution lib _how_ overlapping regions shall be translated into a flat representation. And precisely here come priorities into play. It is the way to tell that lib either "region A shall override region B" if A has higher prio or "if region A and B overlap, do whatever you want" if both have the same prio.
Quoted 5 lines
Yep! And the question is why shouldn't this be done on the level that knows most about the conflict but propagated to the core. I am not arguing that priorities do not exists! Obviously they are. I am questioning the usefulness of priorities be part of memory core API.
The chipset knows about the priorities. How to communicate them to the core?
- at runtime, with hierarchical dispatch of ->read() and ->write(): slow, and doesn't work at all for RAM.
- using registration order: fragile
- using priorities
We need to get the information out of the chipset and into the core, so the core can make use of it (like flattening the tree to produce kvm slots).
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 19 lines · Avi Kivity
Quoted 10 lines · Gleb Natapov
Quoted 6 lines
A) Removing regions will change significantly. So far this is done by setting a region to
IO_MEM_UNASSIGNED, keeping truncation. With the new API that will be a true removal which will additionally restore hidden regions.And what problem do you expect may arise from that? Currently accessing such region after unassign will result in undefined behaviour, so this code is non working today, you can't make it worse.
If the conversion were perfect then yes. However there is a possibility that the conversion will not be perfect.
It's also good to have to have the code document its intentions. If you see
_overlap()you know there is dynamic address decoding going on, or something clever.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
B) Uncontrolled overlapping is a bug that should be caught by the core, and a new API is a perfect chance to do this.
Well, this will indeed introduce the difference in behaviour :) The guest that ran before will abort now. Are you actually aware of any such overlaps in the current code base?
Put a BAR over another BAR, then unmap it.
_overlap will not help with that. PCI BARs can overlap, so _overlap will be used to register them. You do not what to abort qemu when guest configure overlapping PCI BARs don't you?
Quoted 10 lines
Quoted 5 lines
But if priorities are gona stay why not fail if two regions with the same priority overlap? If that happens it means that the memory creation didn't pass the point where conflict should have been resolved (by assigning different priorities) and this means that overlap is unintentional, no?
It may be intentional, as in the case of PCI, or PAM (though you can do PAM without priorities, by removing all but one of the subregions in the area).
When two PCI BARs overlap, somebody somewhere has to decide which one of them will be visible. You can register both of them with same priority and let the core to decide this at the time it calculates flattened view or you can assign different priorities at PCI layer. In the later case you do not need _overlap API.
Quoted 7 lines
Quoted 4 lines
I am starting to see how you can represent all this local decisions as priority numbers and then travel this weighted tree to find what memory region should be accessed (memory registration _has_ to be hierarchical for that to work in meaningful way).
Priorities don't travel up the tree. They're used to resolve local conflicts *only*.
Yes, that what I mean by "traveling weighted tree". At each node you look at local priorities to decide where to move.
Quoted 5 lines
Quoted 3 lines
I still don't see why it is better
than flattening the tree in the point of conflict.How do you decide which subregion wins?
The same way you decide what priority you assign to each region. You do that at the point where you know the priorities. Suppose chipset has RAM from 0x0000 to 0x1fff and PCI from 0x2000 0x2fff and PCI layer tires to map a BAR from 0x1500 to 0x3500. Since chipset code knows that RAM has higher priority and it knows where PCI windows ends it can create two memory regions from that. One is RAM from 0x0000 to 0x1fff another PCI BAR from 0x2000 to 0x2fff. How your tree will look like in this case BTW?
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Not necessarily. It depends on how much added value buses like PCI or ISA or whatever can offer for managing I/O regions. For some purposes, it may as well be fine to just call the memory_* service directly and pass the result of some operation to the bus API later on.
Quoted 8 lines
Quoted 4 lines
Depend on what memory_* service you are talking about. Just creating unattached memory region is OK. But if two independent pieces of code want to map two different memory regions into the same phys address I do not see who will resolve the conflict.
They have to ask the bus to
_add_subregion(). Only the bus knows about the priorities (or the bus can ask them to create the subregions).
Yes, that what I wrote and Jan responded to. Buses all the way down.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
PCI
device will call PCI subsystem. PCI subsystem, instead of assigning arbitrary priorities to all overlappings,
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Again: PCI will _not_ assign arbitrary priorities but only
MEMORY_REGION_DEFAULT_PRIORITY, likely 0.That is as arbitrary as it can get. Just assigning MEMORY_REGION_DEFAULT_PRIORITY/2^0xfff will work equally well, so what is not arbitrary about that number?
That's just splitting hairs. Array indexes start from zero, an arbitrary but convenient number.
The point is that instead of assigning priorities PCI subsystem may resolve the conflict before passing registration to a chipset.
Quoted 23 lines
Quoted 4 lines
BTW why wouldn't PCI layer assign different priorities to overlapping regions to let the core know which one should be actually available? Why leave this decision to the core if it is clearly belongs to PCI?
You mean overlapping BARs? If PCI wants BAR 1 to override BAR 2, then it can indicate it with priorities. If it doesn't want to, it can use the same priority for all regions.
Quoted 8 lines
Quoted 8 lines
That does not specify how the PCI bridge or the chipset will tell that overlapping resolution lib _how_ overlapping regions shall be translated into a flat representation. And precisely here come priorities into play. It is the way to tell that lib either "region A shall override region B" if A has higher prio or "if region A and B overlap, do whatever you want" if both have the same prio.
Quoted 5 lines
Yep! And the question is why shouldn't this be done on the level that knows most about the conflict but propagated to the core. I am not arguing that priorities do not exists! Obviously they are. I am questioning the usefulness of priorities be part of memory core API.
Quoted 8 lines
The chipset knows about the priorities. How to communicate them to the core?
- at runtime, with hierarchical dispatch of ->read() and ->write():
slow, and doesn't work at all for RAM.
- using registration order: fragile
- using priorities
- by resolving overlapping and registering flattened list with the core. (See example above).
Quoted 7 lines
We need to get the information out of the chipset and into the core, so the core can make use of it (like flattening the tree to produce kvm slots).
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
--
Gleb.
Quoted 10 lines · Gleb Natapov
Quoted 8 lines
The chipset knows about the priorities. How to communicate them to the core?
- at runtime, with hierarchical dispatch of ->read() and ->write():
slow, and doesn't work at all for RAM.
- using registration order: fragile
- using priorities- by resolving overlapping and registering flattened list with the core. (See example above).
[Registration would happens with the help of the core against the next higher layer.]
To do this, you need to
- open-code the resolution logic at every level (very bad idea)
- provide library services to obtain a flattened representation
Please try to specify such an API without any parameters that are priority-like.
Jan
Quoted 15 lines · Jan Kiszka
Quoted 10 lines · Gleb Natapov
Quoted 8 lines
The chipset knows about the priorities. How to communicate them to the core?
- at runtime, with hierarchical dispatch of ->read() and ->write():
slow, and doesn't work at all for RAM.
- using registration order: fragile
- using priorities- by resolving overlapping and registering flattened list with the core. (See example above).
[Registration would happens with the help of the core against the next higher layer.]
Quoted 6 lines
To do this, you need to
- open-code the resolution logic at every level (very bad idea)
- provide library services to obtain a flattened representationPlease try to specify such an API without any parameters that are priority-like.
Another way of saying the same thing: having the chipset code resolve conflicts, and having the chipset code assign priorities, are equivalent. But having priorities allows flattening to take place without further involvement of the chipset code.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 20 lines · Avi Kivity
Quoted 19 lines · Jan Kiszka
Quoted 10 lines · Gleb Natapov
Quoted 8 lines
The chipset knows about the priorities. How to communicate them to the core?
- at runtime, with hierarchical dispatch of ->read() and ->write():
slow, and doesn't work at all for RAM.
- using registration order: fragile
- using priorities- by resolving overlapping and registering flattened list with the core. (See example above).
[Registration would happens with the help of the core against the next higher layer.]
To do this, you need to
- open-code the resolution logic at every level (very bad idea)
- provide library services to obtain a flattened representation
Quoted 8 lines
Quoted 2 lines
Please try to specify such an API without any parameters that are priority-like.
Another way of saying the same thing: having the chipset code resolve conflicts, and having the chipset code assign priorities, are equivalent. But having priorities allows flattening to take place without further involvement of the chipset code.
Yes, that's basically sums it up. My main concern with API was that it was not clear how two independent pieces of code would know how to assign priorities. But if all registrations happen hierarchically there will be no such "independent pieces of code".
--
Gleb.
Quoted 27 lines · Gleb Natapov
Quoted 26 lines · Avi Kivity
Quoted 10 lines · Gleb Natapov
Quoted 6 lines
A) Removing regions will change significantly. So far this is done by setting a region to
IO_MEM_UNASSIGNED, keeping truncation. With the new API that will be a true removal which will additionally restore hidden regions.And what problem do you expect may arise from that? Currently accessing such region after unassign will result in undefined behaviour, so this code is non working today, you can't make it worse.
If the conversion were perfect then yes. However there is a possibility that the conversion will not be perfect.
It's also good to have to have the code document its intentions. If you see
_overlap()you know there is dynamic address decoding going on, or something clever.Quoted 6 lines
Quoted 3 lines
B) Uncontrolled overlapping is a bug that should be caught by the core, and a new API is a perfect chance to do this.
Well, this will indeed introduce the difference in behaviour :) The guest that ran before will abort now. Are you actually aware of any such overlaps in the current code base?
Quoted 5 lines
Quoted 2 lines
Put a BAR over another BAR, then unmap it.
_overlapwill not help with that. PCI BARs can overlap, so_overlapwill be used to register them. You do not what to abort qemu when guest configure overlapping PCI BARs don't you?
I'd rather guests have no way to abort qemu, except by explicit agreement... even if they program BARs randomly or do anything else. Right now my virtual server provider won't let me run my own kernels because they are paranoid that a non-approved kernel might crash KVM. Which is reasonable. Even so, it's possible to reprogram BARs from guest userspace.
Hot-adding devices, including ones with MMIO or IO addresses that overlap another existing device, shouldn't make qemu abort either. Perhaps disable the device, perhaps respond with an error, that's all.
Even then, if hot-adding some ISA device overlaps an existing PCI BAR, it would be preferable if the devices (probably both of them) simply didn't receive any bus cycles until the BARs were moved elsewhere, maybe triggered PCI bus errors or MCEs or something like that, rather than introducing never-tested-in-practice management-visible state such as a "disabled" or "refused" device.
I don't know if qemu has devices like this, but many real ISA devices have software-configurable IO, MMI and IRQ settings (ISAPNP) - it's not just PCI.
I thoroughly approve of the plan to keep track of overlapping regions so that adding/removing them has no side effect. When they conflict at equal priorities I suggest a good behaviour would be:
- No access to the underlying device
- MCE interrupt or equivalent, signalling a bus error
Then the order of registration doesn't make any difference, which is good.
-- Jamie
Quoted 5 lines · Gleb Natapov
Quoted 4 lines
BARs may overlap with other BARs or with RAM. That's well-known, so PCI bridged need to register their regions with the
_overlapvariant unconditionally. In contrast to the current PhysPageDesc mechanism, theWith what priority?
It doesn't matter, since the spec doesn't define priorities among PCI BARs.
Quoted 2 lines
If it needs to call
_overlapunconditionally why not always call_overlapand drop not_overlapvariant?
Other uses need non-overlapping registration.
Quoted 8 lines
Quoted 7 lines
And they do not need to. The APIC regions will be managed by the per-CPU region management, reusing the tool box we need for all bridges. It will register the APIC page with a priority higher than the default one, thus overriding everything that comes from the host bridge. I think that reflects pretty well real machine behaviour.
What is "higher"? How does it know that priority is high enough?
It is well known that 1 > 0, for example.
Quoted 5 lines
I
thought, from reading other replies, that priorities are meaningful only on the same hierarchy level (which kinda make sense), but now you are saying that you will override PCI address from another part of the topology?
-- per-cpu memory
|
+--- apic page (prio 1)
|
+--- global memory (prio 0)Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 9 lines · Avi Kivity
Quoted 5 lines · Gleb Natapov
Quoted 4 lines
BARs may overlap with other BARs or with RAM. That's well-known, so PCI bridged need to register their regions with the
_overlapvariant unconditionally. In contrast to the current PhysPageDesc mechanism, theWith what priority?
It doesn't matter, since the spec doesn't define priorities among PCI BARs.
And among PCI BAR and memory (the case the question above referred too).
Quoted 4 lines
Quoted 2 lines
If it needs to call
_overlapunconditionally why not always call_overlapand drop not_overlapvariant?Other uses need non-overlapping registration.
And who prohibit them from creating one?
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
And they do not need to. The APIC regions will be managed by the per-CPU region management, reusing the tool box we need for all bridges. It will register the APIC page with a priority higher than the default one, thus overriding everything that comes from the host bridge. I think that reflects pretty well real machine behaviour.
Quoted 4 lines
Quoted 1 line
What is "higher"? How does it know that priority is high enough?
It is well known that 1 > 0, for example.
That is if you have global scale. In the case I am asking about you do not. Even if PCI will register memory region that overlaps APIC address with priority 1000 APIC memory region should still be able to override it even with priority 0. Voila 1000 < 0? Where is your sarcasm now? :)
But Jan already answered this one. Actually what really matters is the place of the node in a topology, not priority. But then for all of this to make sense registration has to be hierarchical.
Quoted 6 lines
Quoted 5 lines
I
thought, from reading other replies, that priorities are meaningful only on the same hierarchy level (which kinda make sense), but now you are saying that you will override PCI address from another part of the topology?
Quoted 9 lines
-- per-cpu memory
| +--- apic page (prio 1) | +--- global memory (prio 0)Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
--
Gleb.
Quoted 11 lines · Gleb Natapov
Quoted 9 lines · Avi Kivity
Quoted 5 lines · Gleb Natapov
Quoted 4 lines
BARs may overlap with other BARs or with RAM. That's well-known, so PCI bridged need to register their regions with the
_overlapvariant unconditionally. In contrast to the current PhysPageDesc mechanism, theWith what priority?
It doesn't matter, since the spec doesn't define priorities among PCI BARs.
And among PCI BAR and memory (the case the question above referred too).
One of them gets priority 0, the other gets priority 1. Depending on who should win according to the spec.
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
If it needs to call
_overlapunconditionally why not always call_overlapand drop not_overlapvariant?Other uses need non-overlapping registration.
And who prohibit them from creating one?
Nothing.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
And they do not need to. The APIC regions will be managed by the per-CPU region management, reusing the tool box we need for all bridges. It will register the APIC page with a priority higher than the default one, thus overriding everything that comes from the host bridge. I think that reflects pretty well real machine behaviour.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
What is "higher"? How does it know that priority is high enough?
It is well known that 1> 0, for example.
That is if you have global scale. In the case I am asking about you do not. Even if PCI will register memory region that overlaps APIC address with priority 1000 APIC memory region should still be able to override it even with priority 0. Voila 1000< 0? Where is your sarcasm now? :)
This can be done in two ways:
1. Assign APIC the highest priority. Priority is not determined by the guest, but by qemu. Problem solved.
2. Use a hierarchy:
root
|
+-- RAM (prio 0) | +-- PCI (prio 1, lots of children) | +-- APIC (prio 2)
Nothing under the PCI subregion can obscure the APIC.
Quoted 3 lines
But Jan already answered this one. Actually what really matters is the place of the node in a topology, not priority.
You need priority for the children of the same parent.
Quoted 2 lines
But then for all of this
to make sense registration has to be hierarchical.
Well, that's the whole point.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.