How to expose various BMC chip controls ?
Hi Folks !
I would like to discuss something we need to solve for BMC chips before we start implementing a solution that you'll reject upstream :-)
So quick recap: the BMC chip is the management controller of your server, typically some kind of specialized ARM SoC which controls a variety of things ranging from power supply, fans, inventory, flash, to having even a built-in GPU connected to the main server processor via PCIe etc...
As part of the OpenBMC project, we have been contributing (along with others from Google etc...) a bunch of upstream drivers to support the BMC chips from Aspeed which are quite popular. There's work in progress to also upstream other vendor(s) chip support.
For most things, we can write appropriate drivers.
However, there's a range of things for which there is no good fit for dedicated per-function kernel drivers in the BMC:
The BMC chips have a variety of control bits and pieces (registers) that affect the host in all sorts of ways. A few semi-random examples:
- Scratch registers that the host BIOS can read via LPC that contain
system specific details of interest to the BIOS itself. The content/format is specific to a given BIOS <-> BMC userspace pair.
- Similar ones related to the built-in VGA (the one the host sees on
PCIe, it doesn't appear as a device on the BMC side). Used to pass some wiring information to the VGA BIOS and the Linux driver among others.
- Bits controlling which parts of the BMC the host can access remotely
via either LPC or PCIe. (Visibility of the VGA to the host, of the BMC PCIe system-controller device, opening of various windows from the host into the BMC address space etc...)
- ... and a few more
Those things tend to be fairly system specific. Meaning that what's written in them (or read from them) and when it's written in these registers should be under userspace control on the BMC. There isn't necessarily many registers that need to be exposed that way, here too, which specific ones is rather system specific.
But /dev/mem isn't a solution :-)
So one simple idea I had, you tell me if it can fly, is to have some driver for the SoC as a whole (thinking of syscon here...) expose a selection of these things via sysfs.
The list being potentially system specific, it would be provided to the kernel via the device-tree (binding tbd) where we would effectively provide a list of names, register offset, start bit, bitfield size.
Any better idea ? It's a fairly simple problem, and the above solution would be very little code, but I just don't want us to go down a rabbit hole if some of you have religious objections to some of it :)
Cheers,
Ben.
Quoted 51 lines · Benjamin Herrenschmidt
Hi Folks !
I would like to discuss something we need to solve for BMC chips before we start implementing a solution that you'll reject upstream :-)
So quick recap: the BMC chip is the management controller of your server, typically some kind of specialized ARM SoC which controls a variety of things ranging from power supply, fans, inventory, flash, to having even a built-in GPU connected to the main server processor via PCIe etc...
As part of the OpenBMC project, we have been contributing (along with others from Google etc...) a bunch of upstream drivers to support the BMC chips from Aspeed which are quite popular. There's work in progress to also upstream other vendor(s) chip support.
For most things, we can write appropriate drivers.
However, there's a range of things for which there is no good fit for dedicated per-function kernel drivers in the BMC:
The BMC chips have a variety of control bits and pieces (registers) that affect the host in all sorts of ways. A few semi-random examples:
- Scratch registers that the host BIOS can read via LPC that contain
system specific details of interest to the BIOS itself. The content/format is specific to a given BIOS <-> BMC userspace pair.- Similar ones related to the built-in VGA (the one the host sees on
PCIe, it doesn't appear as a device on the BMC side). Used to pass some wiring information to the VGA BIOS and the Linux driver among others.- Bits controlling which parts of the BMC the host can access remotely
via either LPC or PCIe. (Visibility of the VGA to the host, of the BMC PCIe system-controller device, opening of various windows from the host into the BMC address space etc...)- ... and a few more
Those things tend to be fairly system specific. Meaning that what's written in them (or read from them) and when it's written in these registers should be under userspace control on the BMC. There isn't necessarily many registers that need to be exposed that way, here too, which specific ones is rather system specific.
But
/dev/memisn't a solution :-)So one simple idea I had, you tell me if it can fly, is to have some driver for the SoC as a whole (thinking of syscon here...) expose a selection of these things via sysfs.
Quoted 7 lines
The list being potentially system specific, it would be provided to the kernel via the device-tree (binding tbd) where we would effectively provide a list of names, register offset, start bit, bitfield size.
Any better idea ? It's a fairly simple problem, and the above solution would be very little code, but I just don't want us to go down a rabbit hole if some of you have religious objections to some of it :)
I think the hard part is finding the right balance: traditionally, these kinds of systems used ad-hoc interfaces like /dev/mem or random kernel interfaces to do both very generic and very non-generic thigns.
The initial reaction you get when posting a patch that adds an ad-hoc interface will (should?) always be a question of "can this be done using a generic interface?", but that doesn't mean that the answer is always "yes", we just need to come up with ideas of how that interface would look like and figure out together if it's actually worth it or not.
In the cases where an ad-hoc interface is needed, I can see two options: we can stick something in drivers/soc that exposes it in the least ugly way, completely specific to that particular SoC, or we can create a new subsystem from "random BMC slave-side interfaces" and have it maintained by you all (leave me out of the picture, except if you are looking for ideas).
In any case, I think it makes most sense to discuss new interfaces between the people working on the competing BMC implementations, mainly to see if some functionality can be abstracted in a common way or not. I believe you are already working with the Nuvoton BMC people, but I don't know much about the others. Is the Emulex/Broadcom/Aspeed "Pilot" BMC series used anywhere that you know of? It seems that Renesas and TI are no longer active in this space, and that the big server vendors are either not running Linux (e.g. HP) on their BMC or they have their own software stack on the generic third-party BMCs (e.g. Dell). I think Huawei also has their own BMC hardware, but no idea what they run on it. Do you know of any others?
Arnd
Quoted 8 lines · Arnd Bergmann
Quoted 3 lines
Any better idea ? It's a fairly simple problem, and the above solution would be very little code, but I just don't want us to go down a rabbit hole if some of you have religious objections to some of it :)
I think the hard part is finding the right balance: traditionally, these kinds of systems used ad-hoc interfaces like
/dev/memor random kernel interfaces to do both very generic and very non-generic thigns.
Right. This is always the question, and for a bunch of things we *have* used either generic interfaces or specialized drivers. There's an open ended issue with our mailbox driver since we don't use the mailbox framework, which we need to resolve, but otherwise, we are keen on using existing interfaces when one exists.
Quoted 6 lines
The initial reaction you get when posting a patch that adds an ad-hoc interface will (should?) always be a question of "can this be done using a generic interface?", but that doesn't mean that the answer is always "yes", we just need to come up with ideas of how that interface would look like and figure out together if it's actually worth it or not.
So I listed a few examples, I think Joel and Andrew can come up with a few mores. These things, mostly because they control host-visible things rather than BMC-visible things tend to simply not fit in existing interfaces.
They are also sufficiently platform *and* BMC type specific that they wouldn't fit into some kind of generic "BMC controls" driver either I suspect.
Basically, what we need a set of named "knobs" that the BMC userspace code can tweak based on policies set by a given system vendor and the interactions between their BMC implementation (including OpenBMC variant) and their BIOS/Firmware.
Quoted 6 lines
In the cases where an ad-hoc interface is needed, I can see two options: we can stick something in
drivers/socthat exposes it in the least ugly way, completely specific to that particular SoC, or we can create a new subsystem from "random BMC slave-side interfaces" and have it maintained by you all (leave me out of the picture, except if you are looking for ideas).
I don't think there enough commonality in these things to create a "BMC slave interface" class that is anything other than a bunch of sysfs files.
That's why I ended up with my personal conclusion/preference of just having the syscon "driver" for the SoC just expose a table of a dozen or so such knobs, and use the DT to describe them to abstract a bit the difference between SoC models/revisions and how they are wired up in a given platform.
Quoted 4 lines
In any case, I think it makes most sense to discuss new interfaces between the people working on the competing BMC implementations, mainly to see if some functionality can be abstracted in a common way or not.
So we have some ideas of what competing BMC *chip* vendors do since OpenBMC people communicates with other vendors.
We know a lot less about BMC SW stacks, but those are a rat nest of closed source & GPL stuff that doesn't get upstream (or even released sometimes ... ahem ...). Having had access to some vendor code in the past, I can tell you that you really don't want to look at what's going on in there ;-)
Now, lets be clear, these is really for a few (we hope maybe a handful or two as described in my examples) knobs that really don't fit anywhere else. By their very nature they are very specific to a given BMC HW implementation.
Quoted 2 lines
I believe you are already working with the Nuvoton BMC
people, but I don't know much about the others.
Nuvoton and Aspeed own the market as far as I know these days...
Quoted 3 lines
Is the
Emulex/Broadcom/Aspeed "Pilot" BMC series used anywhere that you know of?
I don't know.
Quoted 6 lines
It seems that Renesas and TI are no longer
active in this space, and that the big server vendors are either not running Linux (e.g. HP) on their BMC or they have their own software stack on the generic third-party BMCs (e.g. Dell). I think Huawei also has their own BMC hardware, but no idea what they run on it. Do you know of any others?
Nope. That said, I don't expect much commonality for this specific problem.
Cheers,
Ben.
Quoted 1 line
Arnd
Quoted 13 lines · Benjamin Herrenschmidt
Quoted 6 lines · Arnd Bergmann
In the cases where an ad-hoc interface is needed, I can see two options: we can stick something in
drivers/socthat exposes it in the least ugly way, completely specific to that particular SoC, or we can create a new subsystem from "random BMC slave-side interfaces" and have it maintained by you all (leave me out of the picture, except if you are looking for ideas).I don't think there enough commonality in these things to create a "BMC slave interface" class that is anything other than a bunch of sysfs files.
Quoted 5 lines
That's why I ended up with my personal conclusion/preference of just having the syscon "driver" for the SoC just expose a table of a dozen or so such knobs, and use the DT to describe them to abstract a bit the difference between SoC models/revisions and how they are wired up in a given platform.
The idea with having a subsystem for it would be to group stuff together in one directory and have that owned by people that care about it, even if there is little commonality between the things it's used for.
That might be more useful than having it in drivers/misc/ or drivers/soc/, especially when it comes to recognizing things that are common between different BMC hardware implementations or (host) firmware stacks that were initially thought to be more specific.
Quoted 5 lines
Quoted 4 lines
In any case, I think it makes most sense to discuss new interfaces between the people working on the competing BMC implementations, mainly to see if some functionality can be abstracted in a common way or not.
Quoted 8 lines
So we have some ideas of what competing BMC *chip* vendors do since OpenBMC people communicates with other vendors.
We know a lot less about BMC SW stacks, but those are a rat nest of closed source & GPL stuff that doesn't get upstream (or even released sometimes ... ahem ...). Having had access to some vendor code in the past, I can tell you that you really don't want to look at what's going on in there ;-)
Sure, I really only meant the hardware capabilities here.
Arnd
Quoted 20 lines · Arnd Bergmann
Quoted 12 lines · Benjamin Herrenschmidt
Quoted 6 lines · Arnd Bergmann
In the cases where an ad-hoc interface is needed, I can see two options: we can stick something in
drivers/socthat exposes it in the least ugly way, completely specific to that particular SoC, or we can create a new subsystem from "random BMC slave-side interfaces" and have it maintained by you all (leave me out of the picture, except if you are looking for ideas).I don't think there enough commonality in these things to create a "BMC slave interface" class that is anything other than a bunch of sysfs files.
Quoted 5 lines
That's why I ended up with my personal conclusion/preference of just having the syscon "driver" for the SoC just expose a table of a dozen or so such knobs, and use the DT to describe them to abstract a bit the difference between SoC models/revisions and how they are wired up in a given platform.
Quoted 3 lines
The idea with having a subsystem for it would be to group stuff together in one directory and have that owned by people that care about it, even if there is little commonality between the things it's used for.
Ok, so we could use a subsystem as a grouping method in sysfs, that does make some amount of sense ;-)
That said, fundamentally, it is going to be exposed by whatever creates the syscon for the SoC, so yes, a "SoC" driver is probably the right thing to instanciate that thing, as those knobs are very much SoC specific. Though I do want to keep the actual definition DT based at least for Aspeed.
As for finding the commonalities, the connundrum here is that we juts don't have enough experience of platforms with different kinds of BMCs to do that (provided there's anything to find).
So we can wait a few years until we do and keep out-of-tree crap drivers, or just solve the immediate problem in a semi-reasonable way, and maybe make things evolve as we get more SoC's into the picture.
My issue right now is that we need products out of the door (and not just we IBM, but everybody using OpenBMC) and we are just carrying various horrid out of tree hacks to do so, and that needs urgent fixing.
The main worry I have with my own idea here is that it's too easily abused for things that shouldn't get in there. But I can't find a way around that, we'll just have to be vigilant I suppose...
Cheers,
Ben.