Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2131.outbound.protection.outlook.com [40.107.243.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 40F911E877 for ; Wed, 10 Apr 2024 21:51:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.243.131 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712785901; cv=fail; b=rdlD/Fb8bVk+IzAL+5yWt6it+qjpNuUWTyzxzDwXLPCqW1XXHjbaiJuzelUrpyYlj80QSfEMHR015+gFx1dUAQldLIHcFcboRUXrCqwYNOjCaL+7/nv9Iii3skDKVsVJzi1oThkzilUmSN9SmYO3a4FaA65BCD3J2xziY4GSgps= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712785901; c=relaxed/simple; bh=9bn5hby8aDVrZ3Wa1lVT/oWn87VtvMvg9BESD0UzwMQ=; h=Message-ID:Date:Subject:To:Cc:References:From:In-Reply-To: Content-Type:MIME-Version; b=UD+nri0ebNu2VfvmK/F6vMcxfRMZi11gl9BoDENS+I3cmO+oH2KzllLE3EymaZzFUb3/X/0PGOhGh0vlqs56uclfHNnqrbRsz8H9HAXSmpW/xvpqWG6HQNd+lP9ayv+yrFuUKo6hmNEmLtSdP3GKA+LyWQH+ICKFS9MYJstaizY= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=yZfqKt2W; arc=fail smtp.client-ip=40.107.243.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="yZfqKt2W" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=K4fvaSRUn8JJNj2uiEw7AD9tnkVeW/K4HtqQuiupFe7TS0PsRfFUbap5pEGvIT0x9BsMXzfL3oUONX9URldtyuWUzY82WL6Ix/8qMl0T3RbUFFH4GfJ6XoKAOttysiY2DgIvK+g0/6dNn0/SIC0uqwmuEMHqsDT4UvFzTypbRWIdvI3gZsSdTuumh1BT7//ElrIUouad8bGw7RCOCZQWU2CdOo2x7i1gGeTbxOW32M7J+5ClXxZc9Z+o7hXYnJBhPmVcmhfz/h/NXBTBfYj7V4k0Rd3ElmY+HH7yur10BhiLbzbbRawz/pEdYSfVa9aQS679EYMVxXSzMhsyL3A5mQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=67DE2y+v1SM7MC0EG0lGQ9UhkG4CUMMrHpLw2VLEGjs=; b=MqBlHuS96unLStDbEUqZ9if6a9fKFGnkjeDit21fLupp4JiMrctcTHhP0XSWD/Y0cEGYe5PYZApgMs9fhmXufGLo9NONQuu++BV2dOl7GfY42CegTHeLDN9SNMo2lw45bBKp5EVNvQ0OtT2hOp8a2cD7fdXcOqrgTM2CZ0fiybN8i1eNpjaIflB34NwyOeLcUDWfyBCiJce3a14iy0MM85Lvlp0JdweyvkOf1zTxj8MhUd0fLf1jL5dHH8h2ov5cFgmFkr2u6316e0ZttgM37n3kph7krQU9nX+lkiTTGRsDa/0pquFE1s5gaeOXz10K1k9E+T4CAzXDVtgXE3uymw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=67DE2y+v1SM7MC0EG0lGQ9UhkG4CUMMrHpLw2VLEGjs=; b=yZfqKt2WJX+Wx/GCDIjplFM2/bQ8zdYORGaSicwUyegKX6itKA7yCCJoXjXyKAgyxuQ1tKR6P7yGrFOR97Mi5+smxe90KI2hSH7OKPrM7uDJPbeHNP/OMEH7vP6m8NDu+agDGS5ULttLrV7OrsFTDCHWuukiyAfBhYtsuW5aAEw= Received: from DS0PR12MB6583.namprd12.prod.outlook.com (2603:10b6:8:d1::12) by SJ0PR12MB8089.namprd12.prod.outlook.com (2603:10b6:a03:4eb::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.46; Wed, 10 Apr 2024 21:51:35 +0000 Received: from DS0PR12MB6583.namprd12.prod.outlook.com ([fe80::11ac:76e:24ce:4cb1]) by DS0PR12MB6583.namprd12.prod.outlook.com ([fe80::11ac:76e:24ce:4cb1%4]) with mapi id 15.20.7409.042; Wed, 10 Apr 2024 21:51:35 +0000 Message-ID: Date: Wed, 10 Apr 2024 14:51:32 -0700 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v1 net 2/4] net: ena: Wrong missing IO completions check order Content-Language: en-US To: darinzon@amazon.com, David Miller , Jakub Kicinski , netdev@vger.kernel.org Cc: "Woodhouse, David" , "Machulsky, Zorik" , "Matushevsky, Alexander" , Saeed Bshara , "Wilson, Matt" , "Liguori, Anthony" , "Bshara, Nafea" , "Belgazal, Netanel" , "Saidi, Ali" , "Herrenschmidt, Benjamin" , "Kiyanovski, Arthur" , "Dagan, Noam" , "Agroskin, Shay" , "Itzko, Shahar" , "Abboud, Osama" , "Ostrovsky, Evgeny" , "Tabachnik, Ofir" , Netanel Belgazal , Sameeh Jubran , Amit Bernstein References: <20240410091358.16289-1-darinzon@amazon.com> <20240410091358.16289-3-darinzon@amazon.com> From: "Nelson, Shannon" In-Reply-To: <20240410091358.16289-3-darinzon@amazon.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: BYAPR11CA0072.namprd11.prod.outlook.com (2603:10b6:a03:80::49) To DS0PR12MB6583.namprd12.prod.outlook.com (2603:10b6:8:d1::12) Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR12MB6583:EE_|SJ0PR12MB8089:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: vNYD1neI9+8V2TWI/VsWQmkPoWDY+kJmAPdqJV5womxtILuxH/HavSToJgFOlmEuaPGbSoJeLxSNMbCCkS3ond7YdLspjspX+ruRDf0YICv+y8UtsjXciSQBGeCDZlHpuuQfvma8pYu6qdCqfR3SykODDrw0uSfTfhFyJzSzrqstqqnIK00nrZrGYSzu4pg1FbSRj4zi+ur1/nhj49xhOTmepeuiwOQXUl7Z5t6KvVXh7qeB/M+zqfXI7Bg/mm+GGh8PI9WOJMC1lWRbblCY9Iku702HqE1Tm63y7SbvxzSrtSqbZLcrvJ0ik3xR6qJaHrXB2y4HCywzAi3i5P1Voa35c/ayJ7hDu1MXq8rl5Kb1ITLgdFCUhH4Qg0+VZjqEbhFE8x0iRnHILKYo4/MbJyROFbrByGPJL+xA3iZpZzSH3PFyvmp249i+KYuv+84BZhfPwjA92qvm5FjnWr5nUuMfE0O5WMSFZDh/Pfpc1xT2Zfi0T7cfIAQlIa3r7YGKkDGGfU5kTqhcOOPNJBl03aSr139vfNXbWF+PpCdkpC5AvDCZYi5ZMOkUuKBgLoa26BwpxGmb8HN13hoEhbhXuICYXrBITYHE5I8YDx6Kur7i+rlgxOojz87rqaaC5SMvMY07otdAfZ6ebPUNLfyfqE5Pz35vXg75kSuANst1TUI= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS0PR12MB6583.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(7416005)(366007)(376005);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?aElCRktoWExyTWF5QVpYZVpua3B4ZHVYYUlKSVdqdEMvSFQrMHNQY1IwKyt6?= =?utf-8?B?a0NzZkhHWEoxcVpodWNXY1o3ZUhKK1RyUjQ3Z29RTWtEVmZEUEprdnY1Y3gw?= =?utf-8?B?WkNCeUdSWjdHOWN4TnhlSVQ3UG9KYlJOYmhkejZUOGZsaHNsb0JqeTlXOERa?= =?utf-8?B?dWxTeC9ZeFhLTFY1TE5vMGczcDViQWFkQlVrMDkyV0s0aTNhMFdaMGhUd3E5?= =?utf-8?B?Z1lrdG4rZndPeWJkdFJqYk8vWVprYjRmeTVKU1JId1hFeFlIOXg3SlpTc3Fq?= =?utf-8?B?RGNpRXJCR2psbDRGRitLSGJqVzh0OEJqTGE5YzJDVm1yT1l4eWtLWW5QMFZh?= =?utf-8?B?Q20rdmZETDR4bDJDcmNEWi9SV0t6cUVhZ0QvdFJ0V096b04wMENzQjNEQXo3?= =?utf-8?B?UkRWaVE5eUlvNU1BblhJR0ErblcrZDJJVndEdXBwNFNacmRUcW5IQVhwZEJz?= =?utf-8?B?dERzdDZ4L0NrWFp1TXBIM2hqNm1UOHhxYk1LVFl0NzRjWXN4eU96VXFEaXFZ?= =?utf-8?B?OFp3eThUbDUwQ242aUYrb1lpdElLTjJPREtYYkozOVlmUWE5clhsb3RaT0Yv?= =?utf-8?B?aHcyNCtoOGtENDhUSTloS3ppNlVOeUxYbDAwQWwySWJIdXVlRFE2L1NGaHFH?= =?utf-8?B?MHQybGc4MmpaSndoaG8wSWtnNEV6VU56bTJIUVcyU2xsbFcxRm8rMUc3cy9u?= =?utf-8?B?cmxEN3Z3ZDNCQktOQXRiUkNRSUdIOHlVOThTQ3ZTQlhaVHdhZ0dLdC9ZOTlJ?= =?utf-8?B?dFgwRDZrSzVGQ2VkcHpwUHlkczZ0Q1VYWkk5TkUvNjhNTlRKZ3JmcElZcENC?= =?utf-8?B?UlczbWRZb2JtV3ZYdHFvVHNhVVFEVE1HWTNYOWQxVmlHd2orZldSSjVUMjFC?= =?utf-8?B?QUhyU0p3Nm5lNjE4NGp5VFNydDhXRDR1dWV4S05TVVNKOWhyZ0Y5YmxndXF5?= =?utf-8?B?cWc0UU9KNkRyRmhQTnJtdnZRSXBQMDNuQmljdW1JbFc5OGtFK2hja3MxTVNq?= =?utf-8?B?Zk80RTFLS1VrMjBQTGVYbjBQallKRmJNaHp1anVBZ0xudVQwdnN4Nmo3WjB0?= =?utf-8?B?elFHNlFSYU5pYVNSd2ZaQkRiUkk0bjkxUFdIL0p2a05TQm8wcW1pRmpBenRr?= =?utf-8?B?aFZ2aE9qR3ZpWkhaNGNtZzVrOUFwK0x5dTdITUQ0S2w3VVg4dHhBWDF0MnlJ?= =?utf-8?B?Q1Q3d0Z4U3pIQlNiTHdCZk05UFdyRWlXWUFoK1FYNEZGNmxQWkJvcHRIZGdo?= =?utf-8?B?d0RwVlVIeTBaMEIyNmZERWw4cld2U08rdXNuUnByN2hmMUJiSVhhYnZIMU84?= =?utf-8?B?bndGV0tiRWZIWnEzM1VlSHdFM05VbkZ4Si82Z29IK1BLVDFMSnRUNXM4OUZI?= =?utf-8?B?NlMxckZwS003TXQxV1dJOTl5dFdvdVhHRFVIREpDMDQvdm9nY3BNcWZON1lv?= =?utf-8?B?dmxHK2REQWk2aHFUSHIxZjNKYjFQL1ZxM3psK3Y0b2VyQjZwcUY4V2VxRStx?= =?utf-8?B?cTJ0OGFYcjNlbTJMOUlQMG9ydHRiRUZUcGRleXQ5VlE5UUZXaTdlVjZNZ1VY?= =?utf-8?B?aGlsemY4d3dXMFFaVHQ2V0x1SGl3NHpIb2FBQTJFWWh6b0xpTXRwNWlidDgv?= =?utf-8?B?V2Q0endJc0pxUDZ1UHg3cXpJVEZlU0FsM2VaaUxia1lTbEhLWXFNQ29SZXFP?= =?utf-8?B?S2kyQXdmRkwxbzVqTVphT3dKYktzcjNCTG15SHdwcnVNQmJTbldOZk54Y1NX?= =?utf-8?B?czlLc014QVhhVWlXL3VCRys4dGdiSjFNR3V6ejBpMWo0dGpBWXcvQkdTejVC?= =?utf-8?B?M0RjeGZKQU5EL2ZROFhlQVQxV0ppUDZ1aWY5bjFWZHNyTmJoa2VaYlZHbzJu?= =?utf-8?B?YytUTUxGeWtVUWhKOTZQSmE1R0JZdUFWYmVETGJpSnZKV3lNK3R5TDYyUENV?= =?utf-8?B?TGFhL2RyMzNkc1lGRUxUYjd3V2NSSkJQZGZIc04zVXBBSHNUTDhqU1gyRXpn?= =?utf-8?B?SnBsYVhZRzlFbld3cEM3UUkrVFZLQk82VWhsMVl2NkpEUHhaMFJqTkdib3Jr?= =?utf-8?B?N2dXRkRleXBoU2c3N005UUxXL05RRlNWNDR0MWdYbFoyRmtQSEw2SWd3R2pB?= =?utf-8?Q?FCbFLJrH99TGoAM3EFuwCmcpK?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: f6c734a8-3758-44d2-3847-08dc59a86459 X-MS-Exchange-CrossTenant-AuthSource: DS0PR12MB6583.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Apr 2024 21:51:35.3936 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 2G7RS7OiKQeHuqnEVEaD3qHT3qWSnLPKggWLWlhL2RVwoFsbmNbFzTE1HYoRMuAaayuFz6qMj9sLA5p4VcyR7g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB8089 On 4/10/2024 2:13 AM, darinzon@amazon.com wrote: > From: David Arinzon > > Missing IO completions check is called every second (HZ jiffies). > This commit fixes several issues with this check: > > 1. Duplicate queues check: > Max of 4 queues are scanned on each check due to monitor budget. > Once reaching the budget, this check exits under the assumption that > the next check will continue to scan the remainder of the queues, > but in practice, next check will first scan the last already scanned > queue which is not necessary and may cause the full queue scan to > last a couple of seconds longer. > The fix is to start every check with the next queue to scan. > For example, on 8 IO queues: > Bug: [0,1,2,3], [3,4,5,6], [6,7] > Fix: [0,1,2,3], [4,5,6,7] > > 2. Unbalanced queues check: > In case the number of active IO queues is not a multiple of budget, > there will be checks which don't utilize the full budget > because the full scan exits when reaching the last queue id. > The fix is to run every TX completion check with exact queue budget > regardless of the queue id. > For example, on 7 IO queues: > Bug: [0,1,2,3], [4,5,6], [0,1,2,3] > Fix: [0,1,2,3], [4,5,6,0], [1,2,3,4] > The budget may be lowered in case the number of IO queues is less > than the budget (4) to make sure there are no duplicate queues on > the same check. > For example, on 3 IO queues: > Bug: [0,1,2,0], [1,2,0,1] > Fix: [0,1,2], [0,1,2] > > Fixes: 1738cd3ed342 ("net: ena: Add a driver for Amazon Elastic Network Adapters (ENA)") > Signed-off-by: Amit Bernstein > Signed-off-by: David Arinzon Reviewed-by: Shannon Nelson > --- > drivers/net/ethernet/amazon/ena/ena_netdev.c | 21 +++++++++++--------- > 1 file changed, 12 insertions(+), 9 deletions(-) > > diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c > index 09e7da1a..59befc0f 100644 > --- a/drivers/net/ethernet/amazon/ena/ena_netdev.c > +++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c > @@ -3481,10 +3481,11 @@ static void check_for_missing_completions(struct ena_adapter *adapter) > { > struct ena_ring *tx_ring; > struct ena_ring *rx_ring; > - int i, budget, rc; > + int qid, budget, rc; > int io_queue_count; > > io_queue_count = adapter->xdp_num_queues + adapter->num_io_queues; > + > /* Make sure the driver doesn't turn the device in other process */ > smp_rmb(); > > @@ -3497,27 +3498,29 @@ static void check_for_missing_completions(struct ena_adapter *adapter) > if (adapter->missing_tx_completion_to == ENA_HW_HINTS_NO_TIMEOUT) > return; > > - budget = ENA_MONITORED_TX_QUEUES; > + budget = min_t(u32, io_queue_count, ENA_MONITORED_TX_QUEUES); > > - for (i = adapter->last_monitored_tx_qid; i < io_queue_count; i++) { > - tx_ring = &adapter->tx_ring[i]; > - rx_ring = &adapter->rx_ring[i]; > + qid = adapter->last_monitored_tx_qid; > + > + while (budget) { > + qid = (qid + 1) % io_queue_count; > + > + tx_ring = &adapter->tx_ring[qid]; > + rx_ring = &adapter->rx_ring[qid]; > > rc = check_missing_comp_in_tx_queue(adapter, tx_ring); > if (unlikely(rc)) > return; > > - rc = !ENA_IS_XDP_INDEX(adapter, i) ? > + rc = !ENA_IS_XDP_INDEX(adapter, qid) ? > check_for_rx_interrupt_queue(adapter, rx_ring) : 0; > if (unlikely(rc)) > return; > > budget--; > - if (!budget) > - break; > } > > - adapter->last_monitored_tx_qid = i % io_queue_count; > + adapter->last_monitored_tx_qid = qid; > } > > /* trigger napi schedule after 2 consecutive detections */ > -- > 2.40.1 > >