habanalabs: all FD must be closed before removing device
PATCH 1/3 habanalabs: all FD must be closed before removing device
This patch fixes a bug in the implementation of the function that removes the device.
The bug can happen when the device is removed but not the driver itself (e.g. remove by the OS due to PCI freeze in Power architecture).
In that case, there maybe open users that are calling IOCTLs while the device is removed. This is a possible race condition that the driver must handle. Otherwise, a kernel panic may occur.
This race is prevented in the hard-reset flow, because the driver makes sure the users are closed before continuing with the hard-reset. This race can not occur when the driver itself is removed because the OS makes sure all the file descriptors are closed.
The fix is to make sure the open users close their file descriptors and if they don't (after a certain amount of time), the driver sends them a SIGKILL, because the remove of the device can't be stopped.
The patch re-uses the same code that is called from the hard-reset flow.
--- drivers/misc/habanalabs/device.c | 32 +++++++++++++++++++++++++++----- 1 file changed, 27 insertions(+), 5 deletions(-)
drivers/misc/habanalabs/device.c +27 -5
@@ -513,11 +513,8 @@ int hl_device_resume(struct hl_device *hdev)return rc; } -static void hl_device_hard_reset_pending (struct work_struct *work) +static void device_kill_open_processes (struct hl_device *hdev) { -struct hl_device_reset_work *device_reset_work = - container_of(work, struct hl_device_reset_work, reset_work); -struct hl_device *hdev = device_reset_work->hdev;u16 pending_total, pending_cnt;struct task_struct *task =NULL ;
@@ -552,6 +549,12 @@ static void hl_device_hard_reset_pending(struct work_struct *work) } } +/* We killed the open users, but because the driver cleans up after the +* user contexts are closed (e.g. mmu mappings), we need to wait again +* to make sure the cleaning phase is finished before continuing with +* the reset +*/ + pending_cnt = pending_total;while ((atomic_read (&hdev->fd_open_cnt)) && (pending_cnt)) {
@@ -567,6 +570,16 @@ static void hl_device_hard_reset_pending(struct work_struct *work)mutex_unlock (&hdev->fd_open_cnt_lock); +} + +static void device_hard_reset_pending (struct work_struct *work) +{ +struct hl_device_reset_work *device_reset_work = + container_of(work, struct hl_device_reset_work, reset_work); +struct hl_device *hdev = device_reset_work->hdev; + +device_kill_open_processes (hdev); +hl_device_reset (hdev, true, true);kfree (device_reset_work);
@@ -650,7 +663,7 @@ int hl_device_reset(struct hl_device *hdev, bool hard_reset,* from a dedicated work */ INIT_WORK (&device_reset_work->reset_work, - hl_device_hard_reset_pending); + device_hard_reset_pending); device_reset_work->hdev = hdev;schedule_work (&device_reset_work->reset_work);
@@ -1049,6 +1062,15 @@ void hl_device_fini(struct hl_device *hdev)/* Mark device as disabled */ hdev->disabled = true; +/* +* Flush anyone that is inside the critical section of enqueue +* jobs to the H/W +*/ + hdev->asic_funcs->hw_queues_lock (hdev); + hdev->asic_funcs->hw_queues_unlock (hdev); + +device_kill_open_processes (hdev); +hl_hwmon_fini (hdev);device_late_fini (hdev);
Signature · 1 line
2.17.1
PATCH 2/3 habanalabs: improve IOCTLs behavior when disabled or reset
This patch makes some improvement in how IOCTLs behave when the device is disabled or under reset.
The new code checks, at the start of every IOCTL, if the device is disabled or in reset. If so, it prints an appropriate kernel message and returns -EBUSY to user-space.
In addition, the code modifies the location of where the hard_reset_pending flag is being set or cleared:
1. It is now cleared immediately after the reset *tear-down* flow is finished but before the re-initialization flow begins.
2. It is being set in the remove function of the device, to make the behavior the same with the hard-reset flow
There are two exceptions to the disable or in reset check:
1. The HL_INFO_DEVICE_STATUS opcode in the INFO IOCTL. This opcode allows the user to inquire about the status of the device, whether it is operational, in reset or malfunction (disabled). If the driver will block this IOCTL, the user won't be able to retrieve the status in case of malfunction or in reset.
2. The WAIT_FOR_CS IOCTL. This IOCTL allows the user to inquire about the status of a CS. We want to allow the user to continue to do so, even if we started a soft-reset process because it will allow the user to get the correct error code for each CS he submitted.
--- drivers/misc/habanalabs/command_buffer.c | 7 +++++++ drivers/misc/habanalabs/device.c | 9 +++++---- drivers/misc/habanalabs/habanalabs_ioctl.c | 3 ++- drivers/misc/habanalabs/memory.c | 3 ++- 4 files changed, 16 insertions(+), 6 deletions(-)
drivers/misc/habanalabs/command_buffer.c +7 -0
@@ -214,6 +214,13 @@ int hl_cb_ioctl(struct hl_fpriv *hpriv, void *data)u64 handle;int rc; +if (hl_device_disabled_or_in_reset (hdev)) { +dev_warn_ratelimited (hdev->dev, +"Device is %s. Can't execute CB IOCTL\n" , +atomic_read (&hdev->in_reset) ?"in_reset" :"disabled" ); +return -EBUSY ; + } +switch (args->in.op) {case HL_CB_OP_CREATE : rc = hl_cb_create(hdev, &hpriv->cb_mgr, args->in.cb_size,
drivers/misc/habanalabs/device.c +5 -4
@@ -641,6 +641,8 @@ int hl_device_reset(struct hl_device *hdev, bool hard_reset,if ((hard_reset) && (!from_hard_reset_thread)) {struct hl_device_reset_work *device_reset_work; + hdev->hard_reset_pending = true; +if (!hdev->pdev) {dev_err (hdev->dev,"Reset action is NOT supported in simulator\n" );
@@ -648,8 +650,6 @@ int hl_device_reset(struct hl_device *hdev, bool hard_reset, goto out_err; } - hdev->hard_reset_pending = true; - device_reset_work =kzalloc (sizeof (*device_reset_work),GFP_ATOMIC );if (!device_reset_work) {
@@ -718,6 +718,7 @@ int hl_device_reset(struct hl_device *hdev, bool hard_reset,if (hard_reset) { hdev->device_cpu_disabled = false; + hdev->hard_reset_pending = false;if (hdev->kernel_ctx) { dev_crit(hdev->dev,
@@ -779,8 +780,6 @@ int hl_device_reset(struct hl_device *hdev, bool hard_reset, }hl_set_max_power (hdev, hdev->max_power); - - hdev->hard_reset_pending = false; }else { rc = hdev->asic_funcs->soft_reset_late_init (hdev);if (rc) {
@@ -1069,6 +1068,8 @@ void hl_device_fini(struct hl_device *hdev) hdev->asic_funcs->hw_queues_lock (hdev); hdev->asic_funcs->hw_queues_unlock (hdev); + hdev->hard_reset_pending = true; +device_kill_open_processes (hdev);hl_hwmon_fini (hdev);
drivers/misc/habanalabs/habanalabs_ioctl.c +2 -1
@@ -207,7 +207,8 @@ static int hl_info_ioctl(struct hl_fpriv *hpriv, void *data)if (hl_device_disabled_or_in_reset (hdev)) {dev_warn_ratelimited (hdev->dev, -"Device is disabled or in reset. Can't execute INFO IOCTL\n" ); +"Device is %s. Can't execute INFO IOCTL\n" , +atomic_read (&hdev->in_reset) ?"in_reset" :"disabled" );return -EBUSY ; }
drivers/misc/habanalabs/memory.c +2 -1
@@ -1159,7 +1159,8 @@ int hl_mem_ioctl(struct hl_fpriv *hpriv, void *data)if (hl_device_disabled_or_in_reset (hdev)) {dev_warn_ratelimited (hdev->dev, -"Device is disabled or in reset. Can't execute memory IOCTL\n" ); +"Device is %s. Can't execute MEMORY IOCTL\n" , +atomic_read (&hdev->in_reset) ?"in_reset" :"disabled" );return -EBUSY ; }
Signature · 1 line
2.17.1
PATCH 3/3 habanalabs: prevent device PTE read/write during hard-reset
In reply to Oded GabbayDuring hard-reset, contexts are closed as part of the tear-down process. After a context is closed, the driver cleans up the page tables of that context in the device's DRAM. This action is both dangerous and unnecessary.
It is unnecessary, because the device is going through a hard-reset, which means the device's DRAM contents are no longer valid and the device's MMU is being reset.
It is dangerous, because if the hard-reset came as a result of a PCI freeze, this action may cause the entire host machine to hang.
Therefore, prevent all device PTE updates when a hard-reset operation is pending.
--- drivers/misc/habanalabs/goya/goya.c | 6 ++++++ 1 file changed, 6 insertions(+)
drivers/misc/habanalabs/goya/goya.c +6 -0
@@ -4057,6 +4057,9 @@ static u64 goya_read_pte(struct hl_device *hdev, u64 addr) {struct goya_device *goya = hdev->asic_specific; +if (hdev->hard_reset_pending) +return U64_MAX ; +return readq (hdev->pcie_bar[DDR_BAR_ID ] + (addr - goya->ddr_bar_cur_addr)); }
@@ -4065,6 +4068,9 @@ static void goya_write_pte(struct hl_device *hdev, u64 addr, u64 val) {struct goya_device *goya = hdev->asic_specific; +if (hdev->hard_reset_pending) +return ; +writeq (val, hdev->pcie_bar[DDR_BAR_ID ] + (addr - goya->ddr_bar_cur_addr)); }
Signature · 1 line
2.17.1