Skip to contentSkip to conversation

lib/telemetry: add support to fetch global metrics

PATCH Superseded Reshma Pattan 5 participants 13 messages May 17, 2019 – Jun 25, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 lib/telemetry: add support to fetch global metrics

telemetry has support for fetching port based stats from metrics library.

Metrics library also has global stats which are
not fetched by telemetry, so extend telemetry to
fetch the global metrics.

---
 doc/guides/howto/telemetry.rst                |   9 +-
 doc/guides/rel_notes/release_19_08.rst        |   5 +
 lib/librte_telemetry/rte_telemetry.c          | 119 ++++++++++++----
 lib/librte_telemetry/rte_telemetry_internal.h |  31 ++++-
 lib/librte_telemetry/rte_telemetry_parser.c   | 130 +++++++++++++++---
 usertools/dpdk-telemetry-client.py            |  13 +-
 6 files changed, 253 insertions(+), 54 deletions(-)
doc/guides/howto/telemetry.rst +5 -4
index 00f8f7a85..cacc08216 100644
@@ -11,9 +11,10 @@ Introduction
 ------------
 
 The ``librte_telemetry`` provides the functionality so that users may query
-metrics from incoming port traffic. The application which initializes packet
-forwarding will act as the server, sending metrics to the requesting application
-which acts as the client.
+metrics from incoming port traffic and global stats(application stats).
+The application which initializes packet forwarding will act as the server,
+sending metrics to the requesting application which acts as the client.
+
 
 In DPDK, applications are used to initialize the ``telemetry``. To view incoming
 traffic on featured ports, the application should be run first (ie. after ports
@@ -79,7 +80,7 @@ any DPDK application is applicable.
    the menu.
 
 #. Send traffic to any or all available ports from a traffic generator.
-   Select a query option(recursive or singular polling).
+   Select a query option(recursive or singular polling or global stats).
    The metrics will then be displayed on the client terminal in JSON format.
 
 #. Once finished, unregister the client using the menu command.
doc/guides/rel_notes/release_19_08.rst +5 -0
index b9510f93a..a0bf9923a 100644
@@ -54,6 +54,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **Updated telemetry library for global metrics support.**
+
+  Updated ``librte_telemetry`` to fetch the global metrics from the
+  ``librte_metrics`` library.
+
 
 Removed Items
 -------------
lib/librte_telemetry/rte_telemetry.c +89 -30

176 lines not on this page · raw message

lib/librte_telemetry/rte_telemetry_internal.h +29 -2
index c298c3919..6a91d0c44 100644
@@ -24,6 +24,8 @@ extern int telemetry_log_level;
 #define TELEMETRY_LOG_INFO(fmt, args...) \
 	TELEMETRY_LOG(INFO, fmt, ## args)
 
+#define MAX_METRICS 256
+
 typedef struct telemetry_client {
 	char *file_path;
 	int fd;
@@ -48,6 +50,28 @@ enum rte_telemetry_parser_actions {
 	ACTION_DELETE = 2
 };
 
+enum rte_telemetry_stats_type {
+	PORT_STATS = 0,
+	GLOBAL_STATS = 1
+};
+
+/* @internal */
+struct telemetry_encode_param {
+	enum rte_telemetry_stats_type type;
+	union {
+		struct port_param {
+			uint32_t num_metric_ids;
+			uint32_t metric_ids[MAX_METRICS];
+			uint32_t num_port_ids;
+			uint32_t port_ids[RTE_MAX_ETHPORTS];
+		} pp;
+		struct global_param {
+			uint32_t num_metric_ids;
+			uint32_t metric_ids[MAX_METRICS];
+		} gp;
+	};
+};
+
 int32_t
 rte_telemetry_parse_client_message(struct telemetry_impl *telemetry, char *buf);
 
@@ -72,10 +96,13 @@ int32_t
 rte_telemetry_is_port_active(int port_id);
 
 int32_t
-rte_telemetry_send_ports_stats_values(uint32_t *metric_ids, int num_metric_ids,
-	uint32_t *port_ids, int num_port_ids, struct telemetry_impl *telemetry);
+rte_telemetry_send_ports_stats_values(struct telemetry_encode_param *ep,
+	struct telemetry_impl *telemetry);
 
 int32_t
 rte_telemetry_socket_messaging_testing(int index, int socket);
 
+int32_t
+rte_telemetry_send_global_stats_values(struct telemetry_encode_param *ep,
+	struct telemetry_impl *telemetry);
 #endif
lib/librte_telemetry/rte_telemetry_parser.c +114 -16

196 lines not on this page · raw message

usertools/dpdk-telemetry-client.py +11 -2
index ce0c7a905..572ff56bb 100755
@@ -12,6 +12,7 @@
 METRICS_REQ = "{\"action\":0,\"command\":\"ports_all_stat_values\",\"data\":null}"
 API_REG = "{\"action\":1,\"command\":\"clients\",\"data\":{\"client_path\":\""
 API_UNREG = "{\"action\":2,\"command\":\"clients\",\"data\":{\"client_path\":\""
+GLOBAL_METRICS_REQ = "{\"action\":0,\"command\":\"global_stat_values\",\"data\":null}"
 DEFAULT_FP = "/var/run/dpdk/default_client"
 
 class Socket:
@@ -79,12 +80,18 @@ def repeatedlyRequestMetrics(self, sleep_time): # Recursively requests metrics f
             self.requestMetrics()
             time.sleep(sleep_time)
 
+    def requestGlobalMetrics(self): #Requests global metrics for given client
+        self.socket.client_fd.send(GLOBAL_METRICS_REQ)
+        data = self.socket.client_fd.recv(BUFFER_SIZE)
+        print "\nResponse: \n", str(data)
+
     def interactiveMenu(self, sleep_time): # Creates Interactive menu within the script
-        while self.choice != 3:
+        while self.choice != 4:
             print("\nOptions Menu")
             print("[1] Send for Metrics for all ports")
             print("[2] Send for Metrics for all ports recursively")
-            print("[3] Unregister client")
+            print("[3] Send for global Metrics")
+            print("[4] Unregister client")
 
             try:
                 self.choice = int(input("\n:"))
@@ -95,6 +102,8 @@ def interactiveMenu(self, sleep_time): # Creates Interactive menu within the scr
                 elif self.choice == 2:
                     self.repeatedlyRequestMetrics(sleep_time)
                 elif self.choice == 3:
+                    self.requestGlobalMetrics()
+                elif self.choice == 4:
                     self.unregister()
                     self.unregistered = 1
                 else:
Signature · 1 line
2.21.0
Acked-by on v1 1/1
#1 lore raw
Quoted 8 lines · Reshma Pattan

telemetry has support for fetching port based stats from metrics library.

Metrics library also has global stats which are
not fetched by telemetry, so extend telemetry to
fetch the global metrics.

Quoted 8 lines
---
  doc/guides/howto/telemetry.rst                |   9 +-
  doc/guides/rel_notes/release_19_08.rst        |   5 +
  lib/librte_telemetry/rte_telemetry.c          | 119 ++++++++++++----
  lib/librte_telemetry/rte_telemetry_internal.h |  31 ++++-
  lib/librte_telemetry/rte_telemetry_parser.c   | 130 +++++++++++++++---
  usertools/dpdk-telemetry-client.py            |  13 +-
  6 files changed, 253 insertions(+), 54 deletions(-)

Looks good to me, thanks!

#2 lore raw
In reply to Reshma Pattan

17/05/2019 19:07, Reshma Pattan:

Quoted 9 lines

telemetry has support for fetching port based stats from metrics library.

Metrics library also has global stats which are
not fetched by telemetry, so extend telemetry to
fetch the global metrics.

---

I see some errors with GCC:

rte_telemetry_parser.c:362:27: error: unused variable ‘p’

rte_telemetry.c:550:11: error: unused variable ‘i’

rte_telemetry.c:613:16: error: comparison of integer expressions of different signedness: ‘int’ and ‘uint32_t’

#3 lore raw
Quoted 4 lines

-----Original Message-----
From: Thomas Monjalon [mailto:[email protected]]

<snip>

Quoted 9 lines

I see some errors with GCC:

rte_telemetry_parser.c:362:27: error: unused variable ‘p’

rte_telemetry.c:550:11: error: unused variable ‘i’

rte_telemetry.c:613:16: error: comparison of integer expressions of different
signedness: ‘int’ and ‘uint32_t’

Hmm, yes these have to be fixed, strange they were not caught on my board with gcc (GCC) 9.0.1 20190312 (Red Hat 9.0.1-0.10)

Will fix and send v2.

Thanks,
Reshma

#4 lore raw
In reply to Thomas Monjalon
Quoted 4 lines

-----Original Message-----
From: Thomas Monjalon [mailto:[email protected]]

<snip>

Quoted 2 lines

I see some errors with GCC:

On which GCC version the errors are shown?

Quoted 7 lines

rte_telemetry_parser.c:362:27: error: unused variable ‘p’

rte_telemetry.c:550:11: error: unused variable ‘i’

rte_telemetry.c:613:16: error: comparison of integer expressions of different
signedness: ‘int’ and ‘uint32_t’

#5 lore raw

18/06/2019 18:55, Pattan, Reshma:

Quoted 7 lines
Quoted 5 lines

-----Original Message-----
From: Thomas Monjalon [mailto:[email protected]]

I see some errors with GCC:

On which GCC version the errors are shown?

GCC 8.3

Given the type of the errors I think it should be visible with older versions.

Quoted 6 lines
Quoted 6 lines

rte_telemetry_parser.c:362:27: error: unused variable ‘p’

rte_telemetry.c:550:11: error: unused variable ‘i’

rte_telemetry.c:613:16: error: comparison of integer expressions of different
signedness: ‘int’ and ‘uint32_t’

#6 lore raw

PATCH v2 1/1 lib/telemetry: add support to fetch global metrics

In reply to Reshma Pattan

telemetry has support for fetching port based stats from metrics library.

Metrics library also has global stats which are not fetched by telemetry, so extend telemetry to fetch the global metrics.

---
v2: fix GCC compilation issues.
rebase with latest release notes.

---
 doc/guides/howto/telemetry.rst                |   9 +-
 doc/guides/rel_notes/release_19_08.rst        |   5 +
 lib/librte_telemetry/rte_telemetry.c          | 122 +++++++++++-----
 lib/librte_telemetry/rte_telemetry_internal.h |  31 ++++-
 lib/librte_telemetry/rte_telemetry_parser.c   | 130 +++++++++++++++---
 usertools/dpdk-telemetry-client.py            |  13 +-
 6 files changed, 255 insertions(+), 55 deletions(-)
doc/guides/howto/telemetry.rst +5 -4
index 00f8f7a85..cacc08216 100644
@@ -11,9 +11,10 @@ Introduction
 ------------
 
 The ``librte_telemetry`` provides the functionality so that users may query
-metrics from incoming port traffic. The application which initializes packet
-forwarding will act as the server, sending metrics to the requesting application
-which acts as the client.
+metrics from incoming port traffic and global stats(application stats).
+The application which initializes packet forwarding will act as the server,
+sending metrics to the requesting application which acts as the client.
+
 
 In DPDK, applications are used to initialize the ``telemetry``. To view incoming
 traffic on featured ports, the application should be run first (ie. after ports
@@ -79,7 +80,7 @@ any DPDK application is applicable.
    the menu.
 
 #. Send traffic to any or all available ports from a traffic generator.
-   Select a query option(recursive or singular polling).
+   Select a query option(recursive or singular polling or global stats).
    The metrics will then be displayed on the client terminal in JSON format.
 
 #. Once finished, unregister the client using the menu command.
doc/guides/rel_notes/release_19_08.rst +5 -0
index 8c3932d06..083a738e6 100644
@@ -88,6 +88,11 @@ New Features
   * Added multi-queue support to allow one af_xdp vdev with multiple netdev
     queues
 
+* **Updated telemetry library for global metrics support.**
+
+  Updated ``librte_telemetry`` to fetch the global metrics from the
+  ``librte_metrics`` library.
+
 
 Removed Items
 -------------
lib/librte_telemetry/rte_telemetry.c +91 -31

180 lines not on this page · raw message

lib/librte_telemetry/rte_telemetry_internal.h +29 -2
index c298c3919..6a91d0c44 100644
@@ -24,6 +24,8 @@ extern int telemetry_log_level;
 #define TELEMETRY_LOG_INFO(fmt, args...) \
 	TELEMETRY_LOG(INFO, fmt, ## args)
 
+#define MAX_METRICS 256
+
 typedef struct telemetry_client {
 	char *file_path;
 	int fd;
@@ -48,6 +50,28 @@ enum rte_telemetry_parser_actions {
 	ACTION_DELETE = 2
 };
 
+enum rte_telemetry_stats_type {
+	PORT_STATS = 0,
+	GLOBAL_STATS = 1
+};
+
+/* @internal */
+struct telemetry_encode_param {
+	enum rte_telemetry_stats_type type;
+	union {
+		struct port_param {
+			uint32_t num_metric_ids;
+			uint32_t metric_ids[MAX_METRICS];
+			uint32_t num_port_ids;
+			uint32_t port_ids[RTE_MAX_ETHPORTS];
+		} pp;
+		struct global_param {
+			uint32_t num_metric_ids;
+			uint32_t metric_ids[MAX_METRICS];
+		} gp;
+	};
+};
+
 int32_t
 rte_telemetry_parse_client_message(struct telemetry_impl *telemetry, char *buf);
 
@@ -72,10 +96,13 @@ int32_t
 rte_telemetry_is_port_active(int port_id);
 
 int32_t
-rte_telemetry_send_ports_stats_values(uint32_t *metric_ids, int num_metric_ids,
-	uint32_t *port_ids, int num_port_ids, struct telemetry_impl *telemetry);
+rte_telemetry_send_ports_stats_values(struct telemetry_encode_param *ep,
+	struct telemetry_impl *telemetry);
 
 int32_t
 rte_telemetry_socket_messaging_testing(int index, int socket);
 
+int32_t
+rte_telemetry_send_global_stats_values(struct telemetry_encode_param *ep,
+	struct telemetry_impl *telemetry);
 #endif
lib/librte_telemetry/rte_telemetry_parser.c +114 -16

196 lines not on this page · raw message

usertools/dpdk-telemetry-client.py +11 -2
index ce0c7a905..572ff56bb 100755
@@ -12,6 +12,7 @@
 METRICS_REQ = "{\"action\":0,\"command\":\"ports_all_stat_values\",\"data\":null}"
 API_REG = "{\"action\":1,\"command\":\"clients\",\"data\":{\"client_path\":\""
 API_UNREG = "{\"action\":2,\"command\":\"clients\",\"data\":{\"client_path\":\""
+GLOBAL_METRICS_REQ = "{\"action\":0,\"command\":\"global_stat_values\",\"data\":null}"
 DEFAULT_FP = "/var/run/dpdk/default_client"
 
 class Socket:
@@ -79,12 +80,18 @@ def repeatedlyRequestMetrics(self, sleep_time): # Recursively requests metrics f
             self.requestMetrics()
             time.sleep(sleep_time)
 
+    def requestGlobalMetrics(self): #Requests global metrics for given client
+        self.socket.client_fd.send(GLOBAL_METRICS_REQ)
+        data = self.socket.client_fd.recv(BUFFER_SIZE)
+        print "\nResponse: \n", str(data)
+
     def interactiveMenu(self, sleep_time): # Creates Interactive menu within the script
-        while self.choice != 3:
+        while self.choice != 4:
             print("\nOptions Menu")
             print("[1] Send for Metrics for all ports")
             print("[2] Send for Metrics for all ports recursively")
-            print("[3] Unregister client")
+            print("[3] Send for global Metrics")
+            print("[4] Unregister client")
 
             try:
                 self.choice = int(input("\n:"))
@@ -95,6 +102,8 @@ def interactiveMenu(self, sleep_time): # Creates Interactive menu within the scr
                 elif self.choice == 2:
                     self.repeatedlyRequestMetrics(sleep_time)
                 elif self.choice == 3:
+                    self.requestGlobalMetrics()
+                elif self.choice == 4:
                     self.unregister()
                     self.unregistered = 1
                 else:
Signature · 1 line
2.21.0
#7 lore raw
Quoted 9 lines · Reshma Pattan

telemetry has support for fetching port based stats from metrics library.

Metrics library also has global stats which are not fetched by telemetry, so extend telemetry to fetch the global metrics.

Hi Reshma, Kevin,

Not related to this patch, but when telemetry library enabled, it is causing some warnings [1] with gcc9

The [-Waddress-of-packed-member] warnings already disable for gcc, but you need following update to "lib/librte_telemetry/Makefile":
-CFLAGS += -I$(SRCDIR)
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR)

Can you please take care of this?

[1]
In file included from .../dpdk/x86_64-native-linuxapp-gcc/include/rte_ethdev.h:160,
from .../dpdk/lib/librte_telemetry/rte_telemetry_parser.c:13:
.../dpdk/x86_64-native-linuxapp-gcc/include/rte_ether.h: In function ‘rte_is_broadcast_ether_addr’:
.../dpdk/x86_64-native-linuxapp-gcc/include/rte_ether.h:152:2: warning:
converting a packed ‘const struct rte_ether_addr’ pointer (alignment 1) to a ‘unaligned_uint16_t’ {aka ‘const short unsigned int’} pointer (alignment 2) may result in an unaligned pointer value [-Waddress-of-packed-member]

  152 |  const unaligned_uint16_t *ea_words = (const unaligned_uint16_t *)ea;
      |  ^~~~~
#8 lore raw
Quoted 7 lines

Hi Reshma, Kevin,

Not related to this patch, but when telemetry library enabled, it is causing some warnings [1] with gcc9

Quoted 7 lines

The [-Waddress-of-packed-member] warnings already disable for gcc, but you need following update to "lib/librte_telemetry/Makefile":
-CFLAGS += -I$(SRCDIR)
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR)

Can you please take care of this?

Sure, I'll look into it. Thanks.

Quoted 15 lines

[1]
In file included from .../dpdk/x86_64-native-linuxapp- gcc/include/rte_ethdev.h:160,
from .../dpdk/lib/librte_telemetry/rte_telemetry_parser.c:13:
.../dpdk/x86_64-native-linuxapp-gcc/include/rte_ether.h: In function ‘rte_is_broadcast_ether_addr’:
.../dpdk/x86_64-native-linuxapp-gcc/include/rte_ether.h:152:2: warning:
converting a packed ‘const struct rte_ether_addr’ pointer (alignment 1) to a ‘unaligned_uint16_t’ {aka ‘const short unsigned int’} pointer (alignment 2) may result in an unaligned pointer value [-Waddress-of-packed-member]
152 | const unaligned_uint16_t *ea_words = (const unaligned_uint16_t

*)ea;
      |  ^~~~~
#9 lore raw
In reply to Reshma Pattan

18/06/2019 15:49, Reshma Pattan:

Quoted 7 lines

telemetry has support for fetching port based stats from metrics library.

Metrics library also has global stats which are not fetched by telemetry, so extend telemetry to fetch the global metrics.

Quoted 5 lines

---
v2: fix GCC compilation issues.
rebase with latest release notes.

Applied, thanks

#10 lore raw

Hi Reshma,
I am still seeing build issue when rebasing to this : ie. checking uint32_t is negative in rte_telemetry.c:558 Thanks
Nic

-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Thomas Monjalon
Sent: Monday, June 24, 2019 7:54 AM
To: Pattan, Reshma <[email protected]>
Cc: [email protected]; Laatz, Kevin <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v2] lib/telemetry: add support to fetch global metrics

18/06/2019 15:49, Reshma Pattan:

Quoted 11 lines

telemetry has support for fetching port based stats from metrics library.

Metrics library also has global stats which are not fetched by telemetry, so extend telemetry to fetch the global metrics.

---
v2: fix GCC compilation issues.
rebase with latest release notes.

Applied, thanks

#11 lore raw
Quoted 11 lines

-----Original Message-----
From: Chautru, Nicolas
Sent: Monday, June 24, 2019 6:37 PM
To: Thomas Monjalon <[email protected]>; Pattan, Reshma <[email protected]>
Cc: [email protected]; Laatz, Kevin <[email protected]>
Subject: RE: [dpdk-dev] [PATCH v2] lib/telemetry: add support to fetch global metrics

Quoted 4 lines

Hi Reshma,
I am still seeing build issue when rebasing to this : ie. checking uint32_t is negative in rte_telemetry.c:558 Thanks Nic

Which compiler and version , can you give the more details. <snip>

#12 lore raw
In reply to Chautru, Nicolas
Quoted 22 lines

-----Original Message-----
From: Pattan, Reshma
Sent: Tuesday, June 25, 2019 9:39 AM
To: Chautru, Nicolas <[email protected]>; Thomas Monjalon <[email protected]>
Cc: [email protected]; Laatz, Kevin <[email protected]>
Subject: RE: [dpdk-dev] [PATCH v2] lib/telemetry: add support to fetch global metrics

Quoted 9 lines

-----Original Message-----
From: Chautru, Nicolas
Sent: Monday, June 24, 2019 6:37 PM
To: Thomas Monjalon <[email protected]>; Pattan, Reshma <[email protected]>
Cc: [email protected]; Laatz, Kevin <[email protected]>
Subject: RE: [dpdk-dev] [PATCH v2] lib/telemetry: add support to fetch global metrics

Quoted 7 lines
Quoted 4 lines

Hi Reshma,
I am still seeing build issue when rebasing to this : ie. checking uint32_t is negative in rte_telemetry.c:558 Thanks Nic

Which compiler and version , can you give the more details. <snip>

Hi,

WERROR_FLAGS is missing in the make file so we missed to see these errors, me and Kevin are looking into this. Kevin will send the patch.

Thanks,
Reshma