Skip to contentSkip to conversation

your mail

DISCUSSION Jason Gunthorpe 3 participants 3 messages Jan 21, 2015 – Jan 29, 2015 2 messages not in the archive

lore
Share & tools
lore
#2 lore raw
In reply to a message not in the archive

[unfutzd the cc a bit, sorry]

Quoted 7 lines · atull
Quoted 4 lines

If we consider a Zynq, for instance, there are a number of clock nets that the CPU drives into the FPGA fabric. These nets are controlled by the kernel CLK framework. So, before we program the FPGA bitstream the clocks must be setup properly.

It's pretty normal for drivers to find out what their clocks are from the DT and enable them.

Sure, but the clocks are bitfile specific, and not related to programming. Some bitfiles may not require CPU clocks at all.

Quoted 8 lines

Yes the DT overlay can specify:
* clock info
* firmware file name if user is doing it that way
* fpga manager - specific info
* compatiblity string specifies what type of fpga it is
* which fpga this image should go into
* fpga/processor bridges to enable
* driver(s) info that is dependent on the above

All sounds reasonable

Quoted 11 lines
Quoted 10 lines

Today in our Zynq systems we have the bootloader preconfigure everything for what we are trying to do - but that is specific to the particular FPGA we are expecting to run, and eg, I expect if we ran a kernel using the Zynq clk framework there would be problems with it mangling the configuration.

So there would have to be some kind of sequence where the DT is loaded, the zynq specific FPGA programmer does its pre setup, then the request_firmarw/fpga_program_fw loads the bitstream and another pass for a zynq specific post setup and completion handshake?

Quoted 8 lines

fpga-mgr.c has the concept that each different FPGA family will likely need its own way of doing these 3 steps:
* write_init (prepare fgpa for receiving configuration information)
* write (write configuration info to the fpga)
* write_complete (done writing, put fpga into running mode)

There are callbacks into the manufacturor/fpga family specific lower level driver to do these things (as part of the "fpga_manager_ops"

I think the missing bit here is that there are bitfile specific things as well.

The functions above are fine for a generic manufacturer bitfile loader, ie Xilinx GPIO twiddling, Altera JTAG, Zynq DMA, etc.

But wrappered around that should be another set of functions that are bitfile specific.

Like Zynq-PL-boot-protocol-v1 - which deasserts a reset line and waits for the PL to signal back that it has completed reset.

Or jgg-boot-protocol-v1 which monitors the configuration GPIOs for a specific ready pattern..

Or ...

All of those procedures depend on the bitfile to implement something.

Quoted 11 lines
Quoted 3 lines

The DT needs to specify not only the bitstream programming HW to use but this ancillary programming protocol. There are many ways to do a out of reset and completion handshake on Zynq, for instance.

Currently the lower level driver supports only one preferred method of programming. I guess we could add an enumerated DT property to select programming protocol. It would have to be manufacturor specific. Alternatively it could be encoded into the compatibility string if that makes sense.

From a DT perspective I'd expect it to look something like:

soc {

  // This is the 'how to program a bitstream'
  fpga-bitstream0: zynq_pl_dma 
  {
     compatible = "xilinx,zynq,pl,dma";
     regs = <..>
  }

  fpga: ..
  {
     // This is 'what is in the bitstream'
     boot-protocol = "xilinx,zynq,protocol1";
     compatible = "jgg,fpga-foo-bar";
     manager = @fpga-bitstream0
     clocks = ..
     clock-frequency = ...
     zynq_axi_gp0
     {
      // Settings for a CPU to FPGA AXI bridge
       axi setting 1 = ...
       [...]
     }

}
}

I could also see integrating with the regulator framework as well to power up FPGA specific controllable power supplies.

Quoted 7 lines
Quoted 2 lines

And then user space would need to have control points between each of these steps.

We could have two options, configurable from the ioctl:
* When the DT is loaded, do everything
* Even when the DT is loaded, wait for further instructions from ioctl or

Quoted 8 lines

Freewheeling flow:
* Tell ioctl that we are in freewheeling mode
* Load DT overlay

Tightly controlled flow:
* Tell ioctl that we are doing things stepwise
* Load DT overlay
* Use ioctl to step through getting the fpga loaded and known to be happy

I think you've certainly got the idea!

Thinking it through some more, if the kernel DT tells the fpga-mgr that it is

  boot-protocol = "xilinx,zynq,protocol1","jgg,foo-bar";

Then the kernel should refuse to start it if it doesn't know how to do both 'xilinx,zynq,protocol1' and 'jgg,foo-bar'.

Thus the user space ioctl interface becomes more of how to implement a boot protocol helper in userspace? With the proper locking - while the helper is working the FPGA cannot be messed with..

Jason

#3 lore raw
Quoted 5 lines

The functions above are fine for a generic manufacturer bitfile loader, ie Xilinx GPIO twiddling, Altera JTAG, Zynq DMA, etc.

But wrappered around that should be another set of functions that are bitfile specific.

And also a transport layer. You can have the same FPGA with the same loader protocol off multiple different bus types (from USB to on CPU die and all the way between).

Alan

#4 lore raw
In reply to Jason Gunthorpe
Quoted 8 lines · Jason Gunthorpe

[unfutzd the cc a bit, sorry]

Quoted 5 lines · atull
Quoted 4 lines

If we consider a Zynq, for instance, there are a number of clock nets that the CPU drives into the FPGA fabric. These nets are controlled by the kernel CLK framework. So, before we program the FPGA bitstream the clocks must be setup properly.

Quoted 5 lines
Quoted 2 lines

It's pretty normal for drivers to find out what their clocks are from the DT and enable them.

Sure, but the clocks are bitfile specific, and not related to programming. Some bitfiles may not require CPU clocks at all.

The bitfile specific clocks are the clocks that are turned on by the device driver for that chunk of the bitfile. So those clocks can be specified the same way as clocks are specied in the DT.

Quoted 81 lines
Quoted 9 lines

Yes the DT overlay can specify:
* clock info
* firmware file name if user is doing it that way
* fpga manager - specific info
* compatiblity string specifies what type of fpga it is
* which fpga this image should go into
* fpga/processor bridges to enable
* driver(s) info that is dependent on the above

All sounds reasonable

Quoted 11 lines
Quoted 10 lines

Today in our Zynq systems we have the bootloader preconfigure everything for what we are trying to do - but that is specific to the particular FPGA we are expecting to run, and eg, I expect if we ran a kernel using the Zynq clk framework there would be problems with it mangling the configuration.

So there would have to be some kind of sequence where the DT is loaded, the zynq specific FPGA programmer does its pre setup, then the request_firmarw/fpga_program_fw loads the bitstream and another pass for a zynq specific post setup and completion handshake?

Quoted 8 lines

fpga-mgr.c has the concept that each different FPGA family will likely need its own way of doing these 3 steps:
* write_init (prepare fgpa for receiving configuration information)
* write (write configuration info to the fpga)
* write_complete (done writing, put fpga into running mode)

There are callbacks into the manufacturor/fpga family specific lower level driver to do these things (as part of the "fpga_manager_ops"

I think the missing bit here is that there are bitfile specific things as well.

The functions above are fine for a generic manufacturer bitfile loader, ie Xilinx GPIO twiddling, Altera JTAG, Zynq DMA, etc.

But wrappered around that should be another set of functions that are bitfile specific.

Like Zynq-PL-boot-protocol-v1 - which deasserts a reset line and waits for the PL to signal back that it has completed reset.

Or jgg-boot-protocol-v1 which monitors the configuration GPIOs for a specific ready pattern..

Or ...

All of those procedures depend on the bitfile to implement something.

Quoted 9 lines
Quoted 3 lines

The DT needs to specify not only the bitstream programming HW to use but this ancillary programming protocol. There are many ways to do a out of reset and completion handshake on Zynq, for instance.

Currently the lower level driver supports only one preferred method of programming. I guess we could add an enumerated DT property to select programming protocol. It would have to be manufacturor specific. Alternatively it could be encoded into the compatibility string if that makes sense.

From a DT perspective I'd expect it to look something like:

soc {

  // This is the 'how to program a bitstream'
  fpga-bitstream0: zynq_pl_dma 
  {
     compatible = "xilinx,zynq,pl,dma";
     regs = <..>
  }

  fpga: ..
  {
     // This is 'what is in the bitstream'
     boot-protocol = "xilinx,zynq,protocol1";
     compatible = "jgg,fpga-foo-bar";
     manager = @fpga-bitstream0
     clocks = ..
     clock-frequency = ...
Quoted 8 lines
     zynq_axi_gp0
     {
      // Settings for a CPU to FPGA AXI bridge
       axi setting 1 = ...
       [...]
     }

}
}

That's good. It also needs to specify the driver(s) for the hardware that the bitstream will instantiate.

Quoted 36 lines

I could also see integrating with the regulator framework as well to power up FPGA specific controllable power supplies.

Quoted 7 lines
Quoted 2 lines

And then user space would need to have control points between each of these steps.

We could have two options, configurable from the ioctl:
* When the DT is loaded, do everything
* Even when the DT is loaded, wait for further instructions from ioctl or

Quoted 8 lines

Freewheeling flow:
* Tell ioctl that we are in freewheeling mode
* Load DT overlay

Tightly controlled flow:
* Tell ioctl that we are doing things stepwise
* Load DT overlay
* Use ioctl to step through getting the fpga loaded and known to be happy

I think you've certainly got the idea!

Thinking it through some more, if the kernel DT tells the fpga-mgr that it is

  boot-protocol = "xilinx,zynq,protocol1","jgg,foo-bar";

Then the kernel should refuse to start it if it doesn't know how to do both 'xilinx,zynq,protocol1' and 'jgg,foo-bar'.

Thus the user space ioctl interface becomes more of how to implement a boot protocol helper in userspace? With the proper locking - while the helper is working the FPGA cannot be messed with..

Jason