habanalabs: support info queries by multiple processes
v3 cover habanalabs: support info queries by multiple processes
v3 introduces a few fixes and changes the control char device's minor number to be 1 above that of the main device.
Original text:
Today, the driver allows only a single user application (the deep-learning application) to perform queries of the device's stats, information, idle state and more.
This is a serious limitation in data centers, where there are multiple system/monitorining applications that want to continuously retrieve that information, while allowing the deep-learning application to perform work on the device.
This patch-set allows unlimited number of user applications to perform the above queries (by calling the INFO IOCTL), while the deep-learning application is running.
This is done by creating an additional char device per ASIC, that is dedicated to information retrieval only (allows only to call the INFO IOCTL). This method will maintain backward compatibility with existing userspace applications.
- Patches 1-4 makes small improvements to existing code.
- Patch 5 removes the accounting of the number of open file-descriptors and replace it with tracking of the driver's internal file private data strcuture.
- Patch 6 is a pre-requisite to creating the two char devices
- Patch 7 introduce the additional char device
Thanks,
Oded
Oded Gabbay (7):
habanalabs: add handle field to context structure
habanalabs: kill user process after CS rollback
habanalabs: show the process context dram usage
habanalabs: rename user_ctx as compute_ctx
habanalabs: maintain a list of file private data objects
habanalabs: change device_setup_cdev() to be more generic
habanalabs: create two char devices per ASIC
drivers/misc/habanalabs/context.c | 38 +-- drivers/misc/habanalabs/debugfs.c | 4 +- drivers/misc/habanalabs/device.c | 255 ++++++++++++--------- drivers/misc/habanalabs/goya/goya_hwmgr.c | 25 +- drivers/misc/habanalabs/habanalabs.h | 49 ++-- drivers/misc/habanalabs/habanalabs_drv.c | 165 ++++++++----- drivers/misc/habanalabs/habanalabs_ioctl.c | 104 ++++++--- 7 files changed, 403 insertions(+), 237 deletions(-)
Signature · 1 line
2.17.1
PATCH v3 1/7 habanalabs: add handle field to context structure
This patch adds a field to the context's structure that will hold a unique handle for the context.
This will be needed when the user will create the context.
--- drivers/misc/habanalabs/context.c | 27 ++++++++++++++++----------- drivers/misc/habanalabs/habanalabs.h | 2 ++ 2 files changed, 18 insertions(+), 11 deletions(-)
drivers/misc/habanalabs/context.c +16 -11
@@ -67,9 +67,20 @@ int hl_ctx_create(struct hl_device *hdev, struct hl_fpriv *hpriv) goto out_err; } +mutex_lock (&mgr->ctx_lock); + rc =idr_alloc (&mgr->ctx_handles, ctx,1 ,0 ,GFP_KERNEL ); +mutex_unlock (&mgr->ctx_lock); + +if (rc <0 ) { +dev_err (hdev->dev,"Failed to allocate IDR for a new CTX\n" ); + goto free_ctx; + } + + ctx->handle = rc; + rc =hl_ctx_init (hdev, ctx, false);if (rc) - goto free_ctx; + goto remove_from_idr;hl_hpriv_get (hpriv); ctx->hpriv = hpriv;
@@ -78,18 +89,12 @@ int hl_ctx_create(struct hl_device *hdev, struct hl_fpriv *hpriv) hpriv->ctx = ctx; hdev->user_ctx = ctx; -mutex_lock (&mgr->ctx_lock); - rc =idr_alloc (&mgr->ctx_handles, ctx,1 ,0 ,GFP_KERNEL ); -mutex_unlock (&mgr->ctx_lock); - -if (rc <0 ) { -dev_err (hdev->dev,"Failed to allocate IDR for a new CTX\n" ); -hl_ctx_free (hdev, ctx); - goto out_err; - } -return 0 ; +remove_from_idr: +mutex_lock (&mgr->ctx_lock); +idr_remove (&mgr->ctx_handles, ctx->handle); +mutex_unlock (&mgr->ctx_lock); free_ctx:kfree (ctx); out_err:
drivers/misc/habanalabs/habanalabs.h +2 -0
@@ -631,6 +631,7 @@ struct hl_va_range {* execution phase before the context switch phase * has finished. * @asid: context's unique address space ID in the device's MMU. +* @handle: context's opaque handle for user */ struct hl_ctx { DECLARE_HASHTABLE(mem_hash, MEM_HASH_TABLE_BITS);
@@ -652,6 +653,7 @@ struct hl_ctx {atomic_t thread_ctx_switch_token;u32 thread_ctx_switch_wait_token;u32 asid; +u32 handle; };/**
Signature · 1 line
2.17.1
PATCH v3 2/7 habanalabs: kill user process after CS rollback
In reply to Oded GabbayThis patch calls the kill user process function after we rollback the in-flight CSs. This is because the user process can't be closed while there are open CSs. Therefore, there is no point of sending it a SIGKILL before we do the rollback CS part.
--- drivers/misc/habanalabs/device.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-)
drivers/misc/habanalabs/device.c +13 -4
@@ -630,8 +630,6 @@ static void device_hard_reset_pending(struct work_struct *work) container_of(work, struct hl_device_reset_work, reset_work);struct hl_device *hdev = device_reset_work->hdev; -device_kill_open_processes (hdev); -hl_device_reset (hdev, true, true);kfree (device_reset_work);
@@ -736,6 +734,13 @@ int hl_device_reset(struct hl_device *hdev, bool hard_reset,/* Go over all the queues, release all CS and their jobs */ hl_cs_rollback_all (hdev); +/* Kill processes here after CS rollback. This is because the process +* can't really exit until all its CSs are done, which is what we +* do in cs rollback +*/ +if (from_hard_reset_thread) +device_kill_open_processes (hdev); +/* Release kernel context */ if ((hard_reset) && (hl_ctx_put (hdev->kernel_ctx) ==1 )) hdev->kernel_ctx =NULL ;
@@ -1130,8 +1135,6 @@ void hl_device_fini(struct hl_device *hdev) hdev->hard_reset_pending = true; -device_kill_open_processes (hdev); -hl_hwmon_fini (hdev);device_late_fini (hdev);
@@ -1150,6 +1153,12 @@ void hl_device_fini(struct hl_device *hdev)/* Go over all the queues, release all CS and their jobs */ hl_cs_rollback_all (hdev); +/* Kill processes here after CS rollback. This is because the process +* can't really exit until all its CSs are done, which is what we +* do in cs rollback +*/ +device_kill_open_processes (hdev); +hl_cb_pool_fini (hdev);/* Release kernel context */
Signature · 1 line
2.17.1
PATCH v3 3/7 habanalabs: show the process context dram usage
In reply to Oded GabbayWhen the user query the dram usage of a context, show it the dram usage of its context, not the user context that is currently running on the device.
This has no effect right now as we only have a single process and a single context, but this makes the code more ready for multiple process support.
--- drivers/misc/habanalabs/habanalabs_ioctl.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
drivers/misc/habanalabs/habanalabs_ioctl.c +7 -4
@@ -89,8 +89,9 @@ static int hw_events_info(struct hl_device *hdev, struct hl_info_args *args)return copy_to_user (out, arr,min (max_size, size)) ? -EFAULT :0 ; } -static int dram_usage_info (struct hl_device *hdev,struct hl_info_args *args) +static int dram_usage_info (struct hl_fpriv *hpriv,struct hl_info_args *args) { +struct hl_device *hdev = hpriv->hdev;struct hl_info_dram_usage dram_usage = {0 };u32 max_size = args->return_size;void __user *out = (void __user *) (uintptr_t ) args->return_pointer;
@@ -104,7 +105,9 @@ static int dram_usage_info(struct hl_device *hdev, struct hl_info_args *args) prop->dram_base_address); dram_usage.dram_free_mem = (prop->dram_size - dram_kmd_size) -atomic64_read (&hdev->dram_used_mem); - dram_usage.ctx_dram_mem =atomic64_read (&hdev->user_ctx->dram_phys_mem); +if (hpriv->ctx) + dram_usage.ctx_dram_mem = +atomic64_read (&hpriv->ctx->dram_phys_mem);return copy_to_user (out, &dram_usage,min ((size_t ) max_size,sizeof (dram_usage))) ? -EFAULT :0 ;
@@ -218,7 +221,7 @@ static int hl_info_ioctl(struct hl_fpriv *hpriv, void *data)break ;case HL_INFO_DRAM_USAGE : - rc =dram_usage_info (hdev, args); + rc =dram_usage_info (hpriv, args);break ;case HL_INFO_HW_IDLE :
@@ -321,7 +324,7 @@ long hl_ioctl(struct file *filep, unsigned int cmd, unsigned long arg) cmd = ioctl->cmd; }else {dev_err (hdev->dev,"invalid ioctl: pid=%d, nr=0x%02x\n" , -task_pid_nr (current), nr); +task_pid_nr (current), nr);return -ENOTTY ; }
Signature · 1 line
2.17.1
PATCH v3 4/7 habanalabs: rename user_ctx as compute_ctx
In reply to Oded GabbayThis patch renames the "user_ctx" field in the device structure to "compute_ctx". This better reflects the meaning of this context.
In addition, we also check in the ctx_fini() that the debug mode should be disabled only if the context being destroyed is the compute context. This has no effect right now as we only have a single process and a single context, but this makes the code more ready for multiple process support.
--- drivers/misc/habanalabs/context.c | 13 ++++++++----- drivers/misc/habanalabs/debugfs.c | 4 ++-- drivers/misc/habanalabs/device.c | 14 +++++++------- drivers/misc/habanalabs/habanalabs.h | 9 ++++----- 4 files changed, 21 insertions(+), 19 deletions(-)
drivers/misc/habanalabs/context.c +8 -5
@@ -26,12 +26,13 @@ static void hl_ctx_fini(struct hl_ctx *ctx)dma_fence_put (ctx->cs_pending[i]);if (ctx->asid !=HL_KERNEL_ASID_ID ) { -/* -* The engines are stopped as there is no executing CS, but the +/* The engines are stopped as there is no executing CS, but the * Coresight might be still working by accessing addresses * related to the stopped engines. Hence stop it explicitly. +* Stop only if this is the compute context, as there can be +* only one compute context */ -if (hdev->in_debug) +if ((hdev->in_debug) && (hdev->compute_ctx == ctx))hl_device_set_debug_mode (hdev, false);hl_vm_ctx_fini (ctx);
@@ -85,9 +86,11 @@ int hl_ctx_create(struct hl_device *hdev, struct hl_fpriv *hpriv)hl_hpriv_get (hpriv); ctx->hpriv = hpriv; -/* TODO: remove for multiple contexts */ +/* TODO: remove for multiple contexts per process */ hpriv->ctx = ctx; - hdev->user_ctx = ctx; + +/* TODO: remove the following line for multiple process support */ + hdev->compute_ctx = ctx;return 0 ;
drivers/misc/habanalabs/debugfs.c +2 -2
@@ -370,7 +370,7 @@ static int mmu_show(struct seq_file *s, void *data)if (dev_entry->mmu_asid ==HL_KERNEL_ASID_ID ) ctx = hdev->kernel_ctx;else - ctx = hdev->user_ctx; + ctx = hdev->compute_ctx;if (!ctx) {dev_err (hdev->dev,"no ctx available\n" );
@@ -533,7 +533,7 @@ static bool hl_is_device_va(struct hl_device *hdev, u64 addr)static int device_va_to_pa (struct hl_device *hdev,u64 virt_addr,u64 *phys_addr) { -struct hl_ctx *ctx = hdev->user_ctx; +struct hl_ctx *ctx = hdev->compute_ctx;u64 hop_addr, hop_pte_addr, hop_pte;u64 offset_mask =HOP4_MASK |OFFSET_MASK ;int rc =0 ;
drivers/misc/habanalabs/device.c +7 -7
@@ -59,7 +59,7 @@ static void hpriv_release(struct kref *ref)atomic_dec (&hdev->fd_open_cnt);/* This allows a new user context to open the device */ - hdev->user_ctx =NULL ; + hdev->compute_ctx =NULL ; }void hl_hpriv_get (struct hl_fpriv *hpriv)
@@ -590,7 +590,7 @@ static void device_kill_open_processes(struct hl_device *hdev) }if (atomic_read (&hdev->fd_open_cnt)) { - task =get_pid_task (hdev->user_ctx->hpriv->taskpid, + task =get_pid_task (hdev->compute_ctx->hpriv->taskpid,PIDTYPE_PID );if (task) {dev_info (hdev->dev,"Killing user processes\n" );
@@ -760,9 +760,9 @@ int hl_device_reset(struct hl_device *hdev, bool hard_reset,hl_cq_reset (hdev, &hdev->completion_queue[i]);/* Make sure the context switch phase will run again */ -if (hdev->user_ctx) { -atomic_set (&hdev->user_ctx->thread_ctx_switch_token,1 ); - hdev->user_ctx->thread_ctx_switch_wait_token =0 ; +if (hdev->compute_ctx) { +atomic_set (&hdev->compute_ctx->thread_ctx_switch_token,1 ); + hdev->compute_ctx->thread_ctx_switch_wait_token =0 ; }/* Finished tear-down, starting to re-initialize */
@@ -793,7 +793,7 @@ int hl_device_reset(struct hl_device *hdev, bool hard_reset, goto out_err; } - hdev->user_ctx =NULL ; + hdev->compute_ctx =NULL ; rc =hl_ctx_init (hdev, hdev->kernel_ctx, true);if (rc) {
@@ -970,7 +970,7 @@ int hl_device_init(struct hl_device *hdev, struct class *hclass) goto mmu_fini; } - hdev->user_ctx =NULL ; + hdev->compute_ctx =NULL ; rc =hl_ctx_init (hdev, hdev->kernel_ctx, true);if (rc) {
drivers/misc/habanalabs/habanalabs.h +4 -5
@@ -905,7 +905,7 @@ struct hl_debug_params {* @hdev: habanalabs device structure. * @filp: pointer to the given file structure. * @taskpid: current process ID. -* @ctx: current executing context. +* @ctx: current executing context. TODO: remove for multiple ctx per process * @ctx_mgr: context manager to handle multiple context for this FD. * @cb_mgr: command buffer manager to handle multiple buffers for this FD. * @debugfs_list: list of relevant ASIC debugfs.
@@ -916,7 +916,7 @@ struct hl_fpriv {struct hl_device *hdev;struct file *filp;struct pid *taskpid; -struct hl_ctx *ctx;/* TODO: remove for multiple ctx */ +struct hl_ctx *ctx;struct hl_ctx_mgr ctx_mgr;struct hl_cb_mgr cb_mgr;struct list_head debugfs_list;
@@ -1182,7 +1182,7 @@ struct hl_device_reset_work {* @hl_debugfs: device's debugfs manager. * @cb_pool: list of preallocated CBs. * @cb_pool_lock: protects the CB pool. -* @user_ctx: current user context executing. +* @compute_ctx: current compute context executing. * @dram_used_mem: current DRAM memory consumption. * @timeout_jiffies: device CS timeout value. * @max_power: the max power of the device, as configured by the sysadmin. This
@@ -1259,8 +1259,7 @@ struct hl_device {struct list_head cb_pool;spinlock_t cb_pool_lock; -/* TODO: remove user_ctx for multiple process support */ -struct hl_ctx *user_ctx; +struct hl_ctx *compute_ctx;atomic64_t dram_used_mem;u64 timeout_jiffies;
Signature · 1 line
2.17.1
PATCH v3 5/7 habanalabs: maintain a list of file private data objects
In reply to Oded GabbayThis patch adds a new list to the driver's device structure. The list will keep the file private data structures that the driver creates when a user process opens the device.
This change is needed because it is useless to try to count how many FD are open. Instead, track our own private data structure per open file and once it is released, remove it from the list. As long as the list is not empty, it means we have a user that can do something with our device.
---
Changes in v3:
- fix deadlock when changing pm mode to manual
drivers/misc/habanalabs/device.c | 132 ++++++++++------------ drivers/misc/habanalabs/goya/goya_hwmgr.c | 25 ++-- drivers/misc/habanalabs/habanalabs.h | 24 ++-- drivers/misc/habanalabs/habanalabs_drv.c | 78 +++++++------ 4 files changed, 127 insertions(+), 132 deletions(-)
drivers/misc/habanalabs/device.c +61 -71
@@ -42,10 +42,12 @@ static void hpriv_release(struct kref *ref) {struct hl_fpriv *hpriv;struct hl_device *hdev; +struct hl_ctx *ctx; hpriv = container_of(ref, struct hl_fpriv, refcount); hdev = hpriv->hdev; + ctx = hpriv->ctx;put_pid (hpriv->taskpid);
@@ -53,13 +55,12 @@ static void hpriv_release(struct kref *ref)mutex_destroy (&hpriv->restore_phase_mutex); -kfree (hpriv); - -/* Now the FD is really closed */ -atomic_dec (&hdev->fd_open_cnt); - -/* This allows a new user context to open the device */ +mutex_lock (&hdev->fpriv_list_lock); +list_del (&hpriv->dev_node); hdev->compute_ctx =NULL ; +mutex_unlock (&hdev->fpriv_list_lock); + +kfree (hpriv); }void hl_hpriv_get (struct hl_fpriv *hpriv)
@@ -229,14 +230,14 @@ static int device_early_init(struct hl_device *hdev)hl_cb_mgr_init (&hdev->kernel_cb_mgr); -mutex_init (&hdev->fd_open_cnt_lock);mutex_init (&hdev->send_cpu_message_lock);mutex_init (&hdev->debug_lock);mutex_init (&hdev->mmu_cache_lock);INIT_LIST_HEAD (&hdev->hw_queues_mirror_list);spin_lock_init (&hdev->hw_queues_mirror_lock); +INIT_LIST_HEAD (&hdev->fpriv_list); +mutex_init (&hdev->fpriv_list_lock);atomic_set (&hdev->in_reset,0 ); -atomic_set (&hdev->fd_open_cnt,0 );atomic_set (&hdev->cs_active_cnt,0 );return 0 ;
@@ -266,6 +267,8 @@ static void device_early_fini(struct hl_device *hdev)mutex_destroy (&hdev->debug_lock);mutex_destroy (&hdev->send_cpu_message_lock); +mutex_destroy (&hdev->fpriv_list_lock); +hl_cb_mgr_fini (hdev, &hdev->kernel_cb_mgr);kfree (hdev->hl_chip_info);
@@ -277,8 +280,6 @@ static void device_early_fini(struct hl_device *hdev)if (hdev->asic_funcs->early_fini) hdev->asic_funcs->early_fini (hdev); - -mutex_destroy (&hdev->fd_open_cnt_lock); }static void set_freq_to_low_job (struct work_struct *work)
@@ -286,9 +287,13 @@ static void set_freq_to_low_job(struct work_struct *work) struct hl_device *hdev = container_of(work, struct hl_device, work_freq.work); -if (atomic_read (&hdev->fd_open_cnt) ==0 ) +mutex_lock (&hdev->fpriv_list_lock); + +if (!hdev->compute_ctx)hl_device_set_frequency (hdev,PLL_LOW ); +mutex_unlock (&hdev->fpriv_list_lock); +schedule_delayed_work (&hdev->work_freq,usecs_to_jiffies (HL_PLL_LOW_JOB_FREQ_USEC )); }
@@ -338,7 +343,7 @@ static int device_late_init(struct hl_device *hdev) hdev->high_pll = hdev->asic_prop.high_pll;/* force setting to low frequency */ -atomic_set (&hdev->curr_pll_profile,PLL_LOW ); + hdev->curr_pll_profile =PLL_LOW ;if (hdev->pm_mng_profile ==PM_AUTO ) hdev->asic_funcs->set_pll_profile (hdev,PLL_LOW );
@@ -387,38 +392,26 @@ static void device_late_fini(struct hl_device *hdev)* @hdev: pointer to habanalabs device structure * @freq: the new frequency value * -* Change the frequency if needed. -* We allose to set PLL to low only if there is no user process -* Returns 0 if no change was done, otherwise returns 1; +* Change the frequency if needed. This function has no protection against +* concurrency, therefore it is assumed that the calling function has protected +* itself against the case of calling this function from multiple threads with +* different values +* +* Returns 0 if no change was done, otherwise returns 1 */ int hl_device_set_frequency (struct hl_device *hdev,enum hl_pll_frequency freq) { -enum hl_pll_frequency old_freq = - (freq ==PLL_HIGH ) ?PLL_LOW :PLL_HIGH ; -int ret; - -if (hdev->pm_mng_profile ==PM_MANUAL ) +if ((hdev->pm_mng_profile ==PM_MANUAL ) || + (hdev->curr_pll_profile == freq))return 0 ; - ret =atomic_cmpxchg (&hdev->curr_pll_profile, old_freq, freq); -if (ret == freq) -return 0 ; - -/* -* in case we want to lower frequency, check if device is not -* opened. We must have a check here to workaround race condition with -* hl_device_open -*/ -if ((freq ==PLL_LOW ) && (atomic_read (&hdev->fd_open_cnt) >0 )) { -atomic_set (&hdev->curr_pll_profile,PLL_HIGH ); -return 0 ; - } -dev_dbg (hdev->dev,"Changing device frequency to %s\n" , freq ==PLL_HIGH ?"high" :"low" ); hdev->asic_funcs->set_pll_profile (hdev, freq); + hdev->curr_pll_profile = freq; +return 1 ; }
@@ -449,19 +442,8 @@ int hl_device_set_debug_mode(struct hl_device *hdev, bool enable) goto out; } -mutex_lock (&hdev->fd_open_cnt_lock); - -if (atomic_read (&hdev->fd_open_cnt) >1 ) { -dev_err (hdev->dev, -"Failed to enable debug mode. More then a single user is using the device\n" ); - rc = -EPERM ; - goto unlock_fd_open_lock; - } - hdev->in_debug =1 ; -unlock_fd_open_lock: -mutex_unlock (&hdev->fd_open_cnt_lock); out:mutex_unlock (&hdev->debug_lock);
@@ -568,6 +550,7 @@ int hl_device_resume(struct hl_device *hdev)static void device_kill_open_processes (struct hl_device *hdev) {u16 pending_total, pending_cnt; +struct hl_fpriv *hpriv;struct task_struct *task =NULL ;if (hdev->pldm)
@@ -575,32 +558,30 @@ static void device_kill_open_processes(struct hl_device *hdev)else pending_total =HL_PENDING_RESET_PER_SEC ; - pending_cnt = pending_total; - -/* Flush all processes that are inside hl_open */ -mutex_lock (&hdev->fd_open_cnt_lock); - -while ((atomic_read (&hdev->fd_open_cnt)) && (pending_cnt)) { - - pending_cnt--; - -dev_info (hdev->dev, -"Can't HARD reset, waiting for user to close FD\n" ); +/* Giving time for user to close FD, and for processes that are inside +* hl_device_open to finish +*/ +if (!list_empty (&hdev->fpriv_list))ssleep (1 ); - } -if (atomic_read (&hdev->fd_open_cnt)) { - task =get_pid_task (hdev->compute_ctx->hpriv->taskpid, -PIDTYPE_PID ); +mutex_lock (&hdev->fpriv_list_lock); + +/* This section must be protected because we are dereferencing +* pointers that are freed if the process exits +*/ +list_for_each_entry (hpriv, &hdev->fpriv_list, dev_node) { + task =get_pid_task (hpriv->taskpid,PIDTYPE_PID );if (task) { -dev_info (hdev->dev,"Killing user processes\n" ); +dev_info (hdev->dev,"Killing user process\n" );send_sig (SIGKILL , task,1 ); -msleep (100 ); +usleep_range (1000 ,10000 );put_task_struct (task); } } +mutex_unlock (&hdev->fpriv_list_lock); +/* We killed the open users, but because the driver cleans up after the * user contexts are closed (e.g. mmu mappings), we need to wait again * to make sure the cleaning phase is finished before continuing with
@@ -609,19 +590,18 @@ static void device_kill_open_processes(struct hl_device *hdev) pending_cnt = pending_total; -while ((atomic_read (&hdev->fd_open_cnt)) && (pending_cnt)) { +while ((!list_empty (&hdev->fpriv_list)) && (pending_cnt)) { +dev_info (hdev->dev, +"Waiting for all unmap operations to finish before hard reset\n" ); pending_cnt--;ssleep (1 ); } -if (atomic_read (&hdev->fd_open_cnt)) +if (!list_empty (&hdev->fpriv_list))dev_crit (hdev->dev,"Going to hard reset with open user contexts\n" ); - -mutex_unlock (&hdev->fd_open_cnt_lock); - }static void device_hard_reset_pending (struct work_struct *work)
@@ -677,13 +657,16 @@ int hl_device_reset(struct hl_device *hdev, bool hard_reset,/* This also blocks future CS/VM/JOB completion operations */ hdev->disabled = true; -/* -* Flush anyone that is inside the critical section of enqueue +/* Flush anyone that is inside the critical section of enqueue * jobs to the H/W */ hdev->asic_funcs->hw_queues_lock (hdev); hdev->asic_funcs->hw_queues_unlock (hdev); +/* Flush anyone that is inside device open */ +mutex_lock (&hdev->fpriv_list_lock); +mutex_unlock (&hdev->fpriv_list_lock); +dev_err (hdev->dev,"Going to RESET device!\n" ); }
@@ -759,12 +742,16 @@ int hl_device_reset(struct hl_device *hdev, bool hard_reset,for (i =0 ; i < hdev->asic_prop.completion_queues_count ; i++)hl_cq_reset (hdev, &hdev->completion_queue[i]); +mutex_lock (&hdev->fpriv_list_lock); +/* Make sure the context switch phase will run again */ if (hdev->compute_ctx) {atomic_set (&hdev->compute_ctx->thread_ctx_switch_token,1 ); hdev->compute_ctx->thread_ctx_switch_wait_token =0 ; } +mutex_unlock (&hdev->fpriv_list_lock); +/* Finished tear-down, starting to re-initialize */ if (hard_reset) {
@@ -1126,13 +1113,16 @@ void hl_device_fini(struct hl_device *hdev)/* Mark device as disabled */ hdev->disabled = true; -/* -* Flush anyone that is inside the critical section of enqueue +/* Flush anyone that is inside the critical section of enqueue * jobs to the H/W */ hdev->asic_funcs->hw_queues_lock (hdev); hdev->asic_funcs->hw_queues_unlock (hdev); +/* Flush anyone that is inside device open */ +mutex_lock (&hdev->fpriv_list_lock); +mutex_unlock (&hdev->fpriv_list_lock); + hdev->hard_reset_pending = true;hl_hwmon_fini (hdev);
drivers/misc/habanalabs/goya/goya_hwmgr.c +17 -8
@@ -254,11 +254,11 @@ static ssize_t pm_mng_profile_store(struct device *dev, goto out; } -mutex_lock (&hdev->fd_open_cnt_lock); +mutex_lock (&hdev->fpriv_list_lock); -if (atomic_read (&hdev->fd_open_cnt) >0 ) { +if (hdev->compute_ctx) {dev_err (hdev->dev, -"Can't change PM profile while user process is opened on the device\n" ); +"Can't change PM profile while compute context is opened on the device\n" ); count = -EPERM ; goto unlock_mutex; }
@@ -266,24 +266,33 @@ static ssize_t pm_mng_profile_store(struct device *dev,if (strncmp ("auto" , buf,strlen ("auto" )) ==0 ) {/* Make sure we are in LOW PLL when changing modes */ if (hdev->pm_mng_profile ==PM_MANUAL ) { -atomic_set (&hdev->curr_pll_profile,PLL_HIGH ); + hdev->curr_pll_profile =PLL_HIGH ;hl_device_set_frequency (hdev,PLL_LOW ); hdev->pm_mng_profile =PM_AUTO ; } }else if (strncmp ("manual" , buf,strlen ("manual" )) ==0 ) { -/* Make sure we are in LOW PLL when changing modes */ if (hdev->pm_mng_profile ==PM_AUTO ) { -flush_delayed_work (&hdev->work_freq); +/* Must release the lock because the work thread also +* takes this lock. But before we release it, set +* the mode to manual so nothing will change if a user +* suddenly opens the device +*/ hdev->pm_mng_profile =PM_MANUAL ; + +mutex_unlock (&hdev->fpriv_list_lock); + +/* Flush the current work so we can return to the user +* knowing that he is the only one changing frequencies +*/ +flush_delayed_work (&hdev->work_freq); } }else {dev_err (hdev->dev,"value should be auto or manual\n" ); count = -EINVAL ; - goto unlock_mutex; } unlock_mutex: -mutex_unlock (&hdev->fd_open_cnt_lock); +mutex_unlock (&hdev->fpriv_list_lock); out:return count; }
drivers/misc/habanalabs/habanalabs.h +11 -13
@@ -909,6 +909,7 @@ struct hl_debug_params {* @ctx_mgr: context manager to handle multiple context for this FD. * @cb_mgr: command buffer manager to handle multiple buffers for this FD. * @debugfs_list: list of relevant ASIC debugfs. +* @dev_node: node in the device list of file private data * @refcount: number of related contexts. * @restore_phase_mutex: lock for context switch and restore phase. */
@@ -920,6 +921,7 @@ struct hl_fpriv {struct hl_ctx_mgr ctx_mgr;struct hl_cb_mgr cb_mgr;struct list_head debugfs_list; +struct list_head dev_node;struct kref refcount;struct mutex restore_phase_mutex; };
@@ -1161,12 +1163,6 @@ struct hl_device_reset_work {* @cpu_accessible_dma_pool: KMD <-> ArmCP shared memory pool. * @asid_bitmap: holds used/available ASIDs. * @asid_mutex: protects asid_bitmap. -* @fd_open_cnt_lock: lock for updating fd_open_cnt in hl_device_open. Although -* fd_open_cnt is atomic, we need this lock to serialize -* the open function because the driver currently supports -* only a single process at a time. In addition, we need a -* lock here so we can flush user processes which are opening -* the device while we are trying to hard reset it * @send_cpu_message_lock: enforces only one message in KMD <-> ArmCP queue. * @debug_lock: protects critical section of setting debug mode for device * @asic_prop: ASIC specific immutable properties.
@@ -1182,6 +1178,9 @@ struct hl_device_reset_work {* @hl_debugfs: device's debugfs manager. * @cb_pool: list of preallocated CBs. * @cb_pool_lock: protects the CB pool. +* @fpriv_list: list of file private data structures. Each structure is created +* when a user opens the device +* @fpriv_list_lock: protects the fpriv_list * @compute_ctx: current compute context executing. * @dram_used_mem: current DRAM memory consumption. * @timeout_jiffies: device CS timeout value.
@@ -1189,10 +1188,9 @@ struct hl_device_reset_work {* value is saved so in case of hard-reset, KMD will restore this * value and update the F/W after the re-initialization * @in_reset: is device in reset flow. -* @curr_pll_profile: current PLL profile. -* @fd_open_cnt: number of open user processes. * @cs_active_cnt: number of active command submissions on this device (active * means already in H/W queues) +* @curr_pll_profile: current PLL profile. * @major: habanalabs KMD major. * @high_pll: high PLL profile frequency. * @soft_reset_cnt: number of soft reset since KMD loading.
@@ -1211,7 +1209,7 @@ struct hl_device_reset_work {* @mmu_enable: is MMU enabled. * @device_cpu_disabled: is the device CPU disabled (due to timeouts) * @dma_mask: the dma mask that was set for this device -* @in_debug: is device under debug. This, together with fd_open_cnt, enforces +* @in_debug: is device under debug. This, together with fpriv_list, enforces * that only a single user is configuring the debug infrastructure. */ struct hl_device {
@@ -1239,8 +1237,6 @@ struct hl_device {struct gen_pool *cpu_accessible_dma_pool;unsigned long *asid_bitmap;struct mutex asid_mutex; -/* TODO: remove fd_open_cnt_lock for multiple process support */ -struct mutex fd_open_cnt_lock;struct mutex send_cpu_message_lock;struct mutex debug_lock;struct asic_fixed_properties asic_prop;
@@ -1259,15 +1255,17 @@ struct hl_device {struct list_head cb_pool;spinlock_t cb_pool_lock; +struct list_head fpriv_list; +struct mutex fpriv_list_lock; +struct hl_ctx *compute_ctx;atomic64_t dram_used_mem;u64 timeout_jiffies;u64 max_power;atomic_t in_reset; -atomic_t curr_pll_profile; -atomic_t fd_open_cnt;atomic_t cs_active_cnt; +enum hl_pll_frequency curr_pll_profile;u32 major;u32 high_pll;u32 soft_reset_cnt;
drivers/misc/habanalabs/habanalabs_drv.c +38 -40
118 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v3 6/7 habanalabs: change device_setup_cdev() to be more generic
In reply to Oded GabbayThis patch re-factors the device_setup_cdev() function to make it more generic. It doesn't manipulate members of the driver's internal device structure but instead works only on the arguments that are sent to it.
This is in preparation for using this function to create an additional char device per ASIC.
--- drivers/misc/habanalabs/device.c | 47 +++++++++++++++++--------------- 1 file changed, 25 insertions(+), 22 deletions(-)
drivers/misc/habanalabs/device.c +25 -22
@@ -131,48 +131,43 @@ static const struct file_operations hl_ops = {* @hdev: pointer to habanalabs device structure * @hclass: pointer to the class object of the device * @minor: minor number of the specific device -* @fpos : file operations to install for this device +* @fpos: file operations to install for this device +* @name: name of the device as it will appear in the filesystem +* @cdev: pointer to the char device object that will be created +* @dev: pointer to the device object that will be created * * Create a cdev and a Linux device for habanalabs's device. Need to be * called at the end of the habanalabs device initialization process, * because this function exposes the device to the user */ static int device_setup_cdev (struct hl_device *hdev,struct class *hclass, -int minor,const struct file_operations *fops) +int minor,const struct file_operations *fops, +char *name,struct cdev *cdev, +struct device **dev) {int err, devno =MKDEV (hdev->major, minor); -struct cdev *hdev_cdev = &hdev->cdev; -char *name; - - name =kasprintf (GFP_KERNEL ,"hl%d" , hdev->id); -if (!name) -return -ENOMEM ; -cdev_init (hdev_cdev, fops); - hdev_cdev->owner =THIS_MODULE ; - err =cdev_add (hdev_cdev, devno,1 ); +cdev_init (cdev, fops); + cdev->owner =THIS_MODULE ; + err =cdev_add (cdev, devno,1 );if (err) {pr_err ("Failed to add char device %s\n" , name); - goto err_cdev_add; +return err; } - hdev->dev =device_create (hclass,NULL , devno,NULL ,"%s" , name); -if (IS_ERR (hdev->dev)) { + *dev =device_create (hclass,NULL , devno,NULL ,"%s" , name); +if (IS_ERR (*dev)) {pr_err ("Failed to create device %s\n" , name); - err =PTR_ERR (hdev->dev); + err =PTR_ERR (*dev); goto err_device_create; } -dev_set_drvdata (hdev->dev, hdev); - -kfree (name); +dev_set_drvdata (*dev, hdev);return 0 ; err_device_create: -cdev_del (hdev_cdev); -err_cdev_add: -kfree (name); +cdev_del (cdev);return err; }
@@ -875,9 +870,17 @@ int hl_device_reset(struct hl_device *hdev, bool hard_reset,int hl_device_init (struct hl_device *hdev,struct class *hclass) {int i, rc, cq_ready_cnt; +char *name; + + name =kasprintf (GFP_KERNEL ,"hl%d" , hdev->id); +if (!name) +return -ENOMEM ;/* Create device */ - rc =device_setup_cdev (hdev, hclass, hdev->id, &hl_ops); + rc =device_setup_cdev (hdev, hclass, hdev->id, &hl_ops, name, + &hdev->cdev, &hdev->dev); + +kfree (name);if (rc) goto out_disabled;
Signature · 1 line
2.17.1
PATCH v3 7/7 habanalabs: create two char devices per ASIC
In reply to Oded GabbayThis patch changes the driver to create two char devices for each ASIC it discovers. This is done to allow system/monitoring applications to query the device for stats, information, idle state and more, while also allowing the deep-learning application to send work to the ASIC.
One char device is the original device, hlX. IOCTL calls through this device file can perform any task on the device (compute, memory, queries). The open function for this device will fail if it was called before but the file-descriptor it created was not completely released yet (the release callback function is not called from the kernel until all instances of that FD are closed). The driver needs to keep this behavior to support backward compatibility with existing userspace, which count that the open will fail if the device is "occupied".
The second char device is called "hl_controlDx", where x is the same index of the main device with a minor number of the original char device + 1. Applications that open this device can only call the INFO IOCTL. There is no limitation on the number of applications opening this device.
---
Changes in v3:
- control device use minor of main + 1
- remove simulator code
drivers/misc/habanalabs/device.c | 53 +++++++++++- drivers/misc/habanalabs/habanalabs.h | 14 +++- drivers/misc/habanalabs/habanalabs_drv.c | 89 +++++++++++++++----- drivers/misc/habanalabs/habanalabs_ioctl.c | 95 +++++++++++++++------- 4 files changed, 199 insertions(+), 52 deletions(-)
drivers/misc/habanalabs/device.c +50 -3
@@ -95,6 +95,24 @@ static int hl_device_release(struct inode *inode, struct file *filp)return 0 ; } +static int hl_device_release_ctrl (struct inode *inode,struct file *filp) +{ +struct hl_fpriv *hpriv = filp->private_data; +struct hl_device *hdev; + + filp->private_data =NULL ; + + hdev = hpriv->hdev; + +mutex_lock (&hdev->fpriv_list_lock); +list_del (&hpriv->dev_node); +mutex_unlock (&hdev->fpriv_list_lock); + +kfree (hpriv); + +return 0 ; +} +/* * hl_mmap - mmap function for habanalabs device *
@@ -125,6 +143,14 @@ static const struct file_operations hl_ops = { .compat_ioctl = hl_ioctl }; +static const struct file_operations hl_ctrl_ops = { + .owner =THIS_MODULE , + .open = hl_device_open_ctrl, + .release = hl_device_release_ctrl, + .unlocked_ioctl = hl_ioctl_control, + .compat_ioctl = hl_ioctl_control +}; +/* * device_setup_cdev - setup cdev and device for habanalabs device *
@@ -567,7 +593,8 @@ static void device_kill_open_processes(struct hl_device *hdev)list_for_each_entry (hpriv, &hdev->fpriv_list, dev_node) { task =get_pid_task (hpriv->taskpid,PIDTYPE_PID );if (task) { -dev_info (hdev->dev,"Killing user process\n" ); +dev_info (hdev->dev,"Killing user process pid=%d\n" , +task_pid_nr (task));send_sig (SIGKILL , task,1 );usleep_range (1000 ,10000 );
@@ -872,7 +899,7 @@ int hl_device_init(struct hl_device *hdev, struct class *hclass)int i, rc, cq_ready_cnt;char *name; - name =kasprintf (GFP_KERNEL ,"hl%d" , hdev->id); + name =kasprintf (GFP_KERNEL ,"hl%d" , hdev->id /2 );if (!name)return -ENOMEM ;
@@ -885,10 +912,25 @@ int hl_device_init(struct hl_device *hdev, struct class *hclass)if (rc) goto out_disabled; + name =kasprintf (GFP_KERNEL ,"hl_controlD%d" , hdev->id /2 ); +if (!name) { + rc = -ENOMEM ; + goto release_device; + } + +/* Create control device */ + rc =device_setup_cdev (hdev, hclass, hdev->id_control, &hl_ctrl_ops, + name, &hdev->cdev_ctrl, &hdev->dev_ctrl); + +kfree (name); + +if (rc) + goto release_device; +/* Initialize ASIC function pointers and perform early init */ rc =device_early_init (hdev);if (rc) - goto release_device; + goto release_control_device;/* * Start calling ASIC initialization. First S/W then H/W and finally
@@ -1064,6 +1106,9 @@ int hl_device_init(struct hl_device *hdev, struct class *hclass) hdev->asic_funcs->sw_fini (hdev); early_fini:device_early_fini (hdev); +release_control_device: +device_destroy (hclass, hdev->dev_ctrl->devt); +cdev_del (&hdev->cdev_ctrl); release_device:device_destroy (hclass, hdev->dev->devt);cdev_del (&hdev->cdev);
@@ -1179,6 +1224,8 @@ void hl_device_fini(struct hl_device *hdev)device_early_fini (hdev);/* Hide device from user */ +device_destroy (hdev->dev_ctrl->class, hdev->dev_ctrl->devt); +cdev_del (&hdev->cdev_ctrl);device_destroy (hdev->dev->class, hdev->dev->devt);cdev_del (&hdev->cdev);
drivers/misc/habanalabs/habanalabs.h +12 -2
@@ -912,6 +912,7 @@ struct hl_debug_params {* @dev_node: node in the device list of file private data * @refcount: number of related contexts. * @restore_phase_mutex: lock for context switch and restore phase. +* @is_control: true for control device, false otherwise */ struct hl_fpriv {struct hl_device *hdev;
@@ -924,6 +925,7 @@ struct hl_fpriv {struct list_head dev_node;struct kref refcount;struct mutex restore_phase_mutex; +u8 is_control; };
@@ -1010,7 +1012,7 @@ struct hl_dbg_device_entry {*/ /* Theoretical limit only. A single host can only contain up to 4 or 8 PCIe -* x16 cards. In extereme cases, there are hosts that can accommodate 16 cards +* x16 cards. In extreme cases, there are hosts that can accommodate 16 cards. */ #define HL_MAX_MINORS 256
@@ -1143,7 +1145,9 @@ struct hl_device_reset_work {* @pcie_bar: array of available PCIe bars. * @rmmio: configuration area address on SRAM. * @cdev: related char device. -* @dev: realted kernel basic device structure. +* @cdev_ctrl: char device for control operations only (INFO IOCTL) +* @dev: related kernel basic device structure. +* @dev_ctrl: related kernel device structure for the control device * @work_freq: delayed work to lower device frequency if possible. * @work_heartbeat: delayed work for ArmCP is-alive check. * @asic_name: ASIC specific nmae.
@@ -1196,6 +1200,7 @@ struct hl_device_reset_work {* @soft_reset_cnt: number of soft reset since KMD loading. * @hard_reset_cnt: number of hard reset since KMD loading. * @id: device minor. +* @id_control: minor of the control device * @disabled: is device disabled. * @late_init_done: is late init stage was done during initialization. * @hwmon_initialized: is H/W monitor sensors was initialized.
@@ -1217,7 +1222,9 @@ struct hl_device {void __iomem *pcie_bar[6 ];void __iomem *rmmio;struct cdev cdev; +struct cdev cdev_ctrl;struct device *dev; +struct device *dev_ctrl;struct delayed_work work_freq;struct delayed_work work_heartbeat;char asic_name[16 ];
@@ -1271,6 +1278,7 @@ struct hl_device {u32 soft_reset_cnt;u32 hard_reset_cnt;u16 id; +u16 id_control;u8 disabled;u8 late_init_done;u8 hwmon_initialized;
@@ -1376,6 +1384,7 @@ static inline bool hl_mem_area_crosses_range(u64 address, u32 size, }int hl_device_open (struct inode *inode,struct file *filp); +int hl_device_open_ctrl (struct inode *inode,struct file *filp);bool hl_device_disabled_or_in_reset (struct hl_device *hdev);enum hl_device_status hl_device_status (struct hl_device *hdev);int hl_device_set_debug_mode (struct hl_device *hdev,bool enable);
@@ -1615,6 +1624,7 @@ static inline void hl_debugfs_remove_ctx_mem_hash(struct hl_device *hdev,/* IOCTLs */ long hl_ioctl (struct file *filep,unsigned int cmd,unsigned long arg); +long hl_ioctl_control (struct file *filep,unsigned int cmd,unsigned long arg);int hl_cb_ioctl (struct hl_fpriv *hpriv,void *data);int hl_cs_ioctl (struct hl_fpriv *hpriv,void *data);int hl_cs_wait_ioctl (struct hl_fpriv *hpriv,void *data);
drivers/misc/habanalabs/habanalabs_drv.c +70 -19
129 lines not on this page · raw message
drivers/misc/habanalabs/habanalabs_ioctl.c +67 -28
157 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH v3 7/7] habanalabs: create two char devices per ASIC
Quoted 14 lines · Oded Gabbay
This patch changes the driver to create two char devices for each ASIC it discovers. This is done to allow system/monitoring applications to query the device for stats, information, idle state and more, while also allowing the deep-learning application to send work to the ASIC.
One char device is the original device, hlX. IOCTL calls through this device file can perform any task on the device (compute, memory, queries). The open function for this device will fail if it was called before but the file-descriptor it created was not completely released yet (the release callback function is not called from the kernel until all instances of that FD are closed). The driver needs to keep this behavior to support backward compatibility with existing userspace, which count that the open will fail if the device is "occupied".
Quoted 6 lines
The second char device is called "
hl_controlDx", where x is the same index of the main device with a minor number of the original char device + 1. Applications that open this device can only call the INFO IOCTL. There is no limitation on the number of applications opening this device.Signed-off-by Oded Gabbay <[email protected]>
Looks good, thanks for changing the minor allocation:
Quoted 33 lines · Oded Gabbay
v3 introduces a few fixes and changes the control char device's minor number to be 1 above that of the main device.
Original text:
Today, the driver allows only a single user application (the deep-learning application) to perform queries of the device's stats, information, idle state and more.
This is a serious limitation in data centers, where there are multiple system/monitorining applications that want to continuously retrieve that information, while allowing the deep-learning application to perform work on the device.
This patch-set allows unlimited number of user applications to perform the above queries (by calling the INFO IOCTL), while the deep-learning application is running.
This is done by creating an additional char device per ASIC, that is dedicated to information retrieval only (allows only to call the INFO IOCTL). This method will maintain backward compatibility with existing userspace applications.
- Patches 1-4 makes small improvements to existing code.
- Patch 5 removes the accounting of the number of open file-descriptors and replace it with tracking of the driver's internal file private data strcuture.
- Patch 6 is a pre-requisite to creating the two char devices
- Patch 7 introduce the additional char deviceThanks,
Oded
Quoted 8 lines
Oded Gabbay (7):
habanalabs: add handle field to context structure
habanalabs: kill user process after CS rollback
habanalabs: show the process context dram usage
habanalabs: renameuser_ctxascompute_ctx
habanalabs: maintain a list of file private data objects
habanalabs: changedevice_setup_cdev()to be more generic
habanalabs: create two char devices per ASIC
Whole series is: