Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B043C43381 for ; Sun, 3 Mar 2019 21:33:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 38DB720835 for ; Sun, 3 Mar 2019 21:33:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Eg0KXDMt" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726650AbfCCVdV (ORCPT ); Sun, 3 Mar 2019 16:33:21 -0500 Received: from mail-it1-f196.google.com ([209.85.166.196]:40541 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726621AbfCCVdU (ORCPT ); Sun, 3 Mar 2019 16:33:20 -0500 Received: by mail-it1-f196.google.com with SMTP id l139so4732841ita.5 for ; Sun, 03 Mar 2019 13:33:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=OYZkYw/s/m2QI3096HGkHec/5rkbxY26hpPCMiM+nuQ=; b=Eg0KXDMtiiOQ+wH8jpz1T62xFbbjAooFOl02/wKHvPRf3hLc5W29mkhHBQco4SCkZU 73ybPxJy5BviFMNPr0JRMNIZ5jvJFz0Ycnd68k1R9CfGkrjQZR6DDaA3+eoLl6O+fm4X 4iKgalt+N37MQ53IcEkhXzkLC1BC0Liapl0al001E/KgL/BHYVC+Hwt1+t8DYLePxmMA Tl1suuVetw/kZeRD3ofX7XqbQbHbLE0RnnHXq4X8cwie7f9EWL1QZnlEPwANFnU11gbl 0dZKkl6sGZ80FJpecs99EOMh+ta7R1bkDUa6zF7rDsFUBzahGBOl61Azy7fvd60yM3cG t1aA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=OYZkYw/s/m2QI3096HGkHec/5rkbxY26hpPCMiM+nuQ=; b=mC6hMPDaOhTordlSntnfUFE42szsGtqUmVN2DydvsSGcsPD7uwBzg10H2ppMyXIEJ6 YgcTzYqF9PnC4IByjYGWX2dE0DvSHuaaGe3qCrxVKuMvvJSjwH2CBxdJ127xj/bYJfFZ /BStY3Y3hdW576VozBNkc5Gj3d+0b1iXuc55Os3T4Jf2bpku5peqIbsb3k6Hs/G0/ZYP UHpvKOGHSjoObcc1ceTRs+8OQ9z7aAw+HM/O6N0r/vATsXwq2SkrwSLeGmfM+cHpd9PZ 8xEyzOJnPahSRABA7FNSW5JZpgriV6GSaXnwmOUxZO1iJ6bfWQkk5ns6hjQdTdLrAARy 93AA== X-Gm-Message-State: AHQUAuZLmrtPd8KLJlipeez7tbSXekxnqh4tnbiHy95Wgn0qb1Wi6LpF /38Zmc8ZMIQCdOrI8MXR5zc= X-Google-Smtp-Source: AHgI3IbFpkmZClB27N6BcSXH8TyhEkgiL8ehGTZqo4jyciZunlNH0A7pkunfO8dIw2lFxG8gqVRk2A== X-Received: by 2002:a24:568f:: with SMTP id o137mr9291114itb.128.1551648799308; Sun, 03 Mar 2019 13:33:19 -0800 (PST) Received: from [192.168.86.250] ([184.63.162.180]) by smtp.gmail.com with ESMTPSA id b36sm2063714itd.2.2019.03.03.13.33.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 03 Mar 2019 13:33:18 -0800 (PST) Subject: Re: [PATCH net] net: sched: put back q.qlen into a single location To: Eric Dumazet , "David S . Miller" Cc: netdev , Eric Dumazet , Jamal Hadi Salim , Cong Wang , Jiri Pirko References: <20190228205543.172713-1-edumazet@google.com> From: John Fastabend Message-ID: <6d8025d5-bb1e-8f97-c98f-39fff5d3677b@gmail.com> Date: Sun, 3 Mar 2019 13:32:50 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: <20190228205543.172713-1-edumazet@google.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On 2/28/19 12:55 PM, Eric Dumazet wrote: > In the series fc8b81a5981f ("Merge branch 'lockless-qdisc-series'") > John made the assumption that the data path had no need to read > the qdisc qlen (number of packets in the qdisc). > > It is true when pfifo_fast is used as the root qdisc, or as direct MQ/MQPRIO > children. > > But pfifo_fast can be used as leaf in class full qdiscs, and existing > logic needs to access the child qlen in an efficient way. > > HTB breaks badly, since it uses cl->leaf.q->q.qlen in : > htb_activate() -> WARN_ON() > htb_dequeue_tree() to decide if a class can be htb_deactivated > when it has no more packets. > > HFSC, DRR, CBQ, QFQ have similar issues, and some calls to > qdisc_tree_reduce_backlog() also read q.qlen directly. > > Using qdisc_qlen_sum() (which iterates over all possible cpus) > in the data path is a non starter. > Yep. > It seems we have to put back qlen in a central location, > at least for stable kernels. > I can't think of anything better either unfortunately. > For all qdisc but pfifo_fast, qlen is guarded by the qdisc lock, > so the existing q.qlen{++|--} are correct. > > For 'lockless' qdisc (pfifo_fast so far), we need to use atomic_{inc|dec}() > because the spinlock might be not held (for example from > pfifo_fast_enqueue() and pfifo_fast_dequeue()) > > This patch adds atomic_qlen (in the same location than qlen) > and renames the following helpers, since we want to express > they can be used without qdisc lock, and that qlen is no longer percpu. > > - qdisc_qstats_cpu_qlen_dec -> qdisc_qstats_atomic_qlen_dec() > - qdisc_qstats_cpu_qlen_inc -> qdisc_qstats_atomic_qlen_inc() > > Later (net-next) we might revert this patch by tracking all these > qlen uses and replace them by a more efficient method (not having > to access a precise qlen, but an empty/non_empty status that might > be less expensive to maintain/track). Right I think this is the next step. > > Another possibility is to have a legacy pfifo_fast version that would > be used when used a a child qdisc, since the parent qdisc needs > a spinlock anyway. But then, future lockless qdiscs would also > have the same problem. > I would prefer to optimize the qlen checks out for each qdisc but haven't checked its possible yet in all cases. > Fixes: 7e66016f2c65 ("net: sched: helpers to sum qlen and qlen for per cpu logic") > Signed-off-by: Eric Dumazet > Cc: John Fastabend > Cc: Jamal Hadi Salim > Cc: Cong Wang > Cc: Jiri Pirko > --- > include/net/sch_generic.h | 31 +++++++++++++------------------ > net/core/gen_stats.c | 2 -- > net/sched/sch_generic.c | 13 ++++++------- > 3 files changed, 19 insertions(+), 27 deletions(-) > Thanks! Acked-by: John Fastabend