examples/vm_power: 100% Busy Polling

PATCH Superseded David Hunt 4 participants 46 messages Jun 7, 2018 – Jul 20, 2018

Share & tools

David Hunt

author

2018-06-07 14:39 UTC #0 lore raw

cover examples/vm_power: 100% Busy Polling

This patch set adds the capability to do out-of-band power monitoring on a systemi by detecting when a core is doing 100% busy polling, but not handling any packets.

It uses a thread to monitor the branch counters in the targeted cores, and calculates the branch ratio if the running code.

If the branch ratop is low (0.01), then
the code is most likely running in a tight poll loop and doing nothing, i.e. receiving no packets. In this case we scale down the frequency of that core.

If the branch ratio is higher (>0.01), then it is likely that the code is receiving and processing packets. In this case, we scale up the frequency of that core.

The cpu counters are read via /dev/cpu/x/msr, so requires the msr kernel module to be loaded. Because this method is used, the patch set is implemented with one file for x86 systems, and another for non-x86 systems, with conditional compilation in the Makefile. The non-x86 functions are stubs, and do not currently implement any functionality.

The vm_power_manager app has been modified to take a new parameter
--core-list or -l
which takes a list of cores in a comma-separated list format, e.g. 1,3,5-7,9, which resolvest to a core list of 1,3,5,6,7,9 These cores will then be enabled for oob monitoring. When the OOB monitoring thread starts, it reads the branch hits/miss counters of each monitored core, and scales up/down accordingly.

[1/6] examples/vm_power: add check for port count [2/6] examples/vm_power: add core list parameter [3/6] examples/vm_power: add oob monitoring functions [4/6] examples/vm_power: allow greater than 64 cores [5/6] examples/vm_power: add thread for oob core monitor [6/6] examples/vm_power: add port-list to command line

David Hunt

author

2018-06-07 14:39 UTC #1 lore raw

PATCH 1/6 examples/vm_power: add check for port count

If we don't pass any ports to the app, we don't need to create any mempools, and we don't need to init any ports.

Signed-off-by David Hunt <[email protected]>

---
 examples/vm_power_manager/main.c | 81 +++++++++++++++++---------------
 1 file changed, 43 insertions(+), 38 deletions(-)

examples/vm_power_manager/main.c +43 ~~-38~~

index c9805a461..043b374bc 100644
@@ -280,51 +280,56 @@ main(int argc, char **argv)
 
 	nb_ports = rte_eth_dev_count_avail();
 
-	mbuf_pool = rte_pktmbuf_pool_create("MBUF_POOL", NUM_MBUFS * nb_ports,
-		MBUF_CACHE_SIZE, 0, RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id());
+	if (nb_ports > 0) {
+		mbuf_pool = rte_pktmbuf_pool_create("MBUF_POOL",
+				NUM_MBUFS * nb_ports, MBUF_CACHE_SIZE, 0,
+				RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id());
 
-	if (mbuf_pool == NULL)
-		rte_exit(EXIT_FAILURE, "Cannot create mbuf pool\n");
+		if (mbuf_pool == NULL)
+			rte_exit(EXIT_FAILURE, "Cannot create mbuf pool\n");
 
-	/* Initialize ports. */
-	RTE_ETH_FOREACH_DEV(portid) {
-		struct ether_addr eth;
-		int w, j;
-		int ret;
+		/* Initialize ports. */
+		RTE_ETH_FOREACH_DEV(portid) {
+			struct ether_addr eth;
+			int w, j;
+			int ret;
 
-		if ((enabled_port_mask & (1 << portid)) == 0)
-			continue;
+			if ((enabled_port_mask & (1 << portid)) == 0)
+				continue;
 
-		eth.addr_bytes[0] = 0xe0;
-		eth.addr_bytes[1] = 0xe0;
-		eth.addr_bytes[2] = 0xe0;
-		eth.addr_bytes[3] = 0xe0;
-		eth.addr_bytes[4] = portid + 0xf0;
+			eth.addr_bytes[0] = 0xe0;
+			eth.addr_bytes[1] = 0xe0;
+			eth.addr_bytes[2] = 0xe0;
+			eth.addr_bytes[3] = 0xe0;
+			eth.addr_bytes[4] = portid + 0xf0;
 
-		if (port_init(portid, mbuf_pool) != 0)
-			rte_exit(EXIT_FAILURE, "Cannot init port %"PRIu8 "\n",
+			if (port_init(portid, mbuf_pool) != 0)
+				rte_exit(EXIT_FAILURE,
+					"Cannot init port %"PRIu8 "\n",
 					portid);
 
-		for (w = 0; w < MAX_VFS; w++) {
-			eth.addr_bytes[5] = w + 0xf0;
-
-			ret = rte_pmd_ixgbe_set_vf_mac_addr(portid,
-						w, &eth);
-			if (ret == -ENOTSUP)
-				ret = rte_pmd_i40e_set_vf_mac_addr(portid,
-						w, &eth);
-			if (ret == -ENOTSUP)
-				ret = rte_pmd_bnxt_set_vf_mac_addr(portid,
-						w, &eth);
-
-			switch (ret) {
-			case 0:
-				printf("Port %d VF %d MAC: ",
-						portid, w);
-				for (j = 0; j < 6; j++) {
-					printf("%02x", eth.addr_bytes[j]);
-					if (j < 5)
-						printf(":");
+			for (w = 0; w < MAX_VFS; w++) {
+				eth.addr_bytes[5] = w + 0xf0;
+
+				ret = rte_pmd_ixgbe_set_vf_mac_addr(portid,
+							w, &eth);
+				if (ret == -ENOTSUP)
+					ret = rte_pmd_i40e_set_vf_mac_addr(
+							portid, w, &eth);
+				if (ret == -ENOTSUP)
+					ret = rte_pmd_bnxt_set_vf_mac_addr(
+							portid, w, &eth);
+
+				switch (ret) {
+				case 0:
+					printf("Port %d VF %d MAC: ",
+							portid, w);
+					for (j = 0; j < 5; j++) {
+						printf("%02x:",
+							eth.addr_bytes[j]);
+					}
+					printf("%02x\n", eth.addr_bytes[5]);
+					break;
 				}
 				printf("\n");
 				break;

Signature · 1 line

2.17.0

David Hunt

author

2018-06-21 13:25 UTC #2 lore raw

v2 cover examples/vm_power: 100% Busy Polling

This patch set adds the capability to do out-of-band power monitoring on a system. It uses a thread to monitor the branch counters in the targeted cores, and calculates the branch ratio if the running code.

If the branch ratop is low (0.01), then
the code is most likely running in a tight poll loop and doing nothing, i.e. receiving no packets. In this case we scale down the frequency of that core.

If the branch ratio is higher (>0.01), then it is likely that the code is receiving and processing packets. In this case, we scale up the frequency of that core.

The guest_cli app has also been modified to allow sending of a policy of type BRANCH_RATIO where all of the cores included in the policy will be monitored by the vm_power_manager oob thread.

v2 changes:
* Add the guest_cli patch into this patch set, including the ability to set the policy to BRANCH_RATIO.
http://patches.dpdk.org/patch/40742/
* When vm_power_manger receives a policy with type BRANCH_RATIO, add the relevant cores to the monitoring thread.

[1/8] examples/vm_power: add check for port count [2/8] examples/vm_power: add core list parameter [3/8] examples/vm_power: add oob monitoring functions [4/8] examples/vm_power: allow greater than 64 cores [5/8] examples/vm_power: add thread for oob core monitor [6/8] examples/vm_power: add port-list to command line [7/8] examples/vm_power: add branch ratio policy type [8/8] examples/vm_power: add cli args to guest app

David Hunt

author

2018-06-21 13:25 UTC #3 lore raw

PATCH v2 1/8 examples/vm_power: add check for port count

If we don't pass any ports to the app, we don't need to create any mempools, and we don't need to init any ports.

Signed-off-by David Hunt <[email protected]>

---
 examples/vm_power_manager/main.c | 81 +++++++++++++++++---------------
 1 file changed, 43 insertions(+), 38 deletions(-)

examples/vm_power_manager/main.c +43 ~~-38~~

index c9805a461..043b374bc 100644
@@ -280,51 +280,56 @@ main(int argc, char **argv)
 
 	nb_ports = rte_eth_dev_count_avail();
 
-	mbuf_pool = rte_pktmbuf_pool_create("MBUF_POOL", NUM_MBUFS * nb_ports,
-		MBUF_CACHE_SIZE, 0, RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id());
+	if (nb_ports > 0) {
+		mbuf_pool = rte_pktmbuf_pool_create("MBUF_POOL",
+				NUM_MBUFS * nb_ports, MBUF_CACHE_SIZE, 0,
+				RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id());
 
-	if (mbuf_pool == NULL)
-		rte_exit(EXIT_FAILURE, "Cannot create mbuf pool\n");
+		if (mbuf_pool == NULL)
+			rte_exit(EXIT_FAILURE, "Cannot create mbuf pool\n");
 
-	/* Initialize ports. */
-	RTE_ETH_FOREACH_DEV(portid) {
-		struct ether_addr eth;
-		int w, j;
-		int ret;
+		/* Initialize ports. */
+		RTE_ETH_FOREACH_DEV(portid) {
+			struct ether_addr eth;
+			int w, j;
+			int ret;
 
-		if ((enabled_port_mask & (1 << portid)) == 0)
-			continue;
+			if ((enabled_port_mask & (1 << portid)) == 0)
+				continue;
 
-		eth.addr_bytes[0] = 0xe0;
-		eth.addr_bytes[1] = 0xe0;
-		eth.addr_bytes[2] = 0xe0;
-		eth.addr_bytes[3] = 0xe0;
-		eth.addr_bytes[4] = portid + 0xf0;
+			eth.addr_bytes[0] = 0xe0;
+			eth.addr_bytes[1] = 0xe0;
+			eth.addr_bytes[2] = 0xe0;
+			eth.addr_bytes[3] = 0xe0;
+			eth.addr_bytes[4] = portid + 0xf0;
 
-		if (port_init(portid, mbuf_pool) != 0)
-			rte_exit(EXIT_FAILURE, "Cannot init port %"PRIu8 "\n",
+			if (port_init(portid, mbuf_pool) != 0)
+				rte_exit(EXIT_FAILURE,
+					"Cannot init port %"PRIu8 "\n",
 					portid);
 
-		for (w = 0; w < MAX_VFS; w++) {
-			eth.addr_bytes[5] = w + 0xf0;
-
-			ret = rte_pmd_ixgbe_set_vf_mac_addr(portid,
-						w, &eth);
-			if (ret == -ENOTSUP)
-				ret = rte_pmd_i40e_set_vf_mac_addr(portid,
-						w, &eth);
-			if (ret == -ENOTSUP)
-				ret = rte_pmd_bnxt_set_vf_mac_addr(portid,
-						w, &eth);
-
-			switch (ret) {
-			case 0:
-				printf("Port %d VF %d MAC: ",
-						portid, w);
-				for (j = 0; j < 6; j++) {
-					printf("%02x", eth.addr_bytes[j]);
-					if (j < 5)
-						printf(":");
+			for (w = 0; w < MAX_VFS; w++) {
+				eth.addr_bytes[5] = w + 0xf0;
+
+				ret = rte_pmd_ixgbe_set_vf_mac_addr(portid,
+							w, &eth);
+				if (ret == -ENOTSUP)
+					ret = rte_pmd_i40e_set_vf_mac_addr(
+							portid, w, &eth);
+				if (ret == -ENOTSUP)
+					ret = rte_pmd_bnxt_set_vf_mac_addr(
+							portid, w, &eth);
+
+				switch (ret) {
+				case 0:
+					printf("Port %d VF %d MAC: ",
+							portid, w);
+					for (j = 0; j < 5; j++) {
+						printf("%02x:",
+							eth.addr_bytes[j]);
+					}
+					printf("%02x\n", eth.addr_bytes[5]);
+					break;
 				}
 				printf("\n");
 				break;

Signature · 1 line

2.17.1

David Hunt

author

2018-06-26 09:23 UTC #4 lore raw

cover examples/vm_power: 100% Busy Polling

If the branch ratop is low (0.01), then
the code is most likely running in a tight poll loop and doing nothing, i.e. receiving no packets. In this case we scale down the frequency of that core.

If the branch ratio is higher (>0.01), then it is likely that the code is receiving and processing packets. In this case, we scale up the frequency of that core.

The guest_cli app has also been modified to allow sending of a policy of type BRANCH_RATIO where all of the cores included in the policy will be monitored by the vm_power_manager oob thread.

v3 changes:
* Added a command line parameter to alloe changing of the default branch ratio threshold. can now use -b 0.3 or --branch-ratio=0.3 to set the ratio for scaling up/down.

[1/9] examples/vm_power: add check for port count [2/9] examples/vm_power: add core list parameter [3/9] examples/vm_power: add oob monitoring functions [4/9] examples/vm_power: allow greater than 64 cores [5/9] examples/vm_power: add thread for oob core monitor [6/9] examples/vm_power: add port-list to command line [7/9] examples/vm_power: add branch ratio policy type [8/9] examples/vm_power: add cli args to guest app [9/9] examples/vm_power: make branch ratio configurable

David Hunt

author

2018-06-26 09:23 UTC #5 lore raw

PATCH v3 1/9 examples/vm_power: add check for port count

If we don't pass any ports to the app, we don't need to create any mempools, and we don't need to init any ports.

Signed-off-by David Hunt <[email protected]>

Acked-by Radu Nicolau <[email protected]>

---
 examples/vm_power_manager/main.c | 81 +++++++++++++++++---------------
 1 file changed, 43 insertions(+), 38 deletions(-)

examples/vm_power_manager/main.c +43 ~~-38~~

index c9805a461..043b374bc 100644
@@ -280,51 +280,56 @@ main(int argc, char **argv)
 
 	nb_ports = rte_eth_dev_count_avail();
 
-	mbuf_pool = rte_pktmbuf_pool_create("MBUF_POOL", NUM_MBUFS * nb_ports,
-		MBUF_CACHE_SIZE, 0, RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id());
+	if (nb_ports > 0) {
+		mbuf_pool = rte_pktmbuf_pool_create("MBUF_POOL",
+				NUM_MBUFS * nb_ports, MBUF_CACHE_SIZE, 0,
+				RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id());
 
-	if (mbuf_pool == NULL)
-		rte_exit(EXIT_FAILURE, "Cannot create mbuf pool\n");
+		if (mbuf_pool == NULL)
+			rte_exit(EXIT_FAILURE, "Cannot create mbuf pool\n");
 
-	/* Initialize ports. */
-	RTE_ETH_FOREACH_DEV(portid) {
-		struct ether_addr eth;
-		int w, j;
-		int ret;
+		/* Initialize ports. */
+		RTE_ETH_FOREACH_DEV(portid) {
+			struct ether_addr eth;
+			int w, j;
+			int ret;
 
-		if ((enabled_port_mask & (1 << portid)) == 0)
-			continue;
+			if ((enabled_port_mask & (1 << portid)) == 0)
+				continue;
 
-		eth.addr_bytes[0] = 0xe0;
-		eth.addr_bytes[1] = 0xe0;
-		eth.addr_bytes[2] = 0xe0;
-		eth.addr_bytes[3] = 0xe0;
-		eth.addr_bytes[4] = portid + 0xf0;
+			eth.addr_bytes[0] = 0xe0;
+			eth.addr_bytes[1] = 0xe0;
+			eth.addr_bytes[2] = 0xe0;
+			eth.addr_bytes[3] = 0xe0;
+			eth.addr_bytes[4] = portid + 0xf0;
 
-		if (port_init(portid, mbuf_pool) != 0)
-			rte_exit(EXIT_FAILURE, "Cannot init port %"PRIu8 "\n",
+			if (port_init(portid, mbuf_pool) != 0)
+				rte_exit(EXIT_FAILURE,
+					"Cannot init port %"PRIu8 "\n",
 					portid);
 
-		for (w = 0; w < MAX_VFS; w++) {
-			eth.addr_bytes[5] = w + 0xf0;
-
-			ret = rte_pmd_ixgbe_set_vf_mac_addr(portid,
-						w, &eth);
-			if (ret == -ENOTSUP)
-				ret = rte_pmd_i40e_set_vf_mac_addr(portid,
-						w, &eth);
-			if (ret == -ENOTSUP)
-				ret = rte_pmd_bnxt_set_vf_mac_addr(portid,
-						w, &eth);
-
-			switch (ret) {
-			case 0:
-				printf("Port %d VF %d MAC: ",
-						portid, w);
-				for (j = 0; j < 6; j++) {
-					printf("%02x", eth.addr_bytes[j]);
-					if (j < 5)
-						printf(":");
+			for (w = 0; w < MAX_VFS; w++) {
+				eth.addr_bytes[5] = w + 0xf0;
+
+				ret = rte_pmd_ixgbe_set_vf_mac_addr(portid,
+							w, &eth);
+				if (ret == -ENOTSUP)
+					ret = rte_pmd_i40e_set_vf_mac_addr(
+							portid, w, &eth);
+				if (ret == -ENOTSUP)
+					ret = rte_pmd_bnxt_set_vf_mac_addr(
+							portid, w, &eth);
+
+				switch (ret) {
+				case 0:
+					printf("Port %d VF %d MAC: ",
+							portid, w);
+					for (j = 0; j < 5; j++) {
+						printf("%02x:",
+							eth.addr_bytes[j]);
+					}
+					printf("%02x\n", eth.addr_bytes[5]);
+					break;
 				}
 				printf("\n");
 				break;

Signature · 1 line

2.17.1

David Hunt

author

2018-07-13 14:23 UTC #6 lore raw

v4 cover examples/vm_power: 100% Busy Polling

If the branch ratio is low (0.01), then
the code is most likely running in a tight poll loop and doing nothing, i.e. receiving no packets. In this case we scale down the frequency of that core.

If the branch ratio is higher (>0.01), then it is likely that the code is receiving and processing packets. In this case, we scale up the frequency of that core.

The guest_cli app has also been modified to allow sending of a policy of type BRANCH_RATIO where all of the cores included in the policy will be monitored by the vm_power_manager oob thread.

v3 changes:
* Added a command line parameter to allow changing of the default branch ratio threshold. can now use -b 0.3 or --branch-ratio=0.3 to set the ratio for scaling up/down.

v4 changes:
* Removed some un-needed header file includes

David Hunt

author

2018-07-13 14:23 UTC #7 lore raw

PATCH v4 1/9 examples/vm_power: add check for port count

If we don't pass any ports to the app, we don't need to create any mempools, and we don't need to init any ports.

Signed-off-by David Hunt <[email protected]>

Acked-by Radu Nicolau <[email protected]>

---
 examples/vm_power_manager/main.c | 81 +++++++++++++++++---------------
 1 file changed, 43 insertions(+), 38 deletions(-)

examples/vm_power_manager/main.c +43 ~~-38~~

index 8911f2659..0d3846971 100644
@@ -278,51 +278,56 @@ main(int argc, char **argv)
 
 	nb_ports = rte_eth_dev_count_avail();
 
-	mbuf_pool = rte_pktmbuf_pool_create("MBUF_POOL", NUM_MBUFS * nb_ports,
-		MBUF_CACHE_SIZE, 0, RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id());
+	if (nb_ports > 0) {
+		mbuf_pool = rte_pktmbuf_pool_create("MBUF_POOL",
+				NUM_MBUFS * nb_ports, MBUF_CACHE_SIZE, 0,
+				RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id());
 
-	if (mbuf_pool == NULL)
-		rte_exit(EXIT_FAILURE, "Cannot create mbuf pool\n");
+		if (mbuf_pool == NULL)
+			rte_exit(EXIT_FAILURE, "Cannot create mbuf pool\n");
 
-	/* Initialize ports. */
-	RTE_ETH_FOREACH_DEV(portid) {
-		struct ether_addr eth;
-		int w, j;
-		int ret;
+		/* Initialize ports. */
+		RTE_ETH_FOREACH_DEV(portid) {
+			struct ether_addr eth;
+			int w, j;
+			int ret;
 
-		if ((enabled_port_mask & (1 << portid)) == 0)
-			continue;
+			if ((enabled_port_mask & (1 << portid)) == 0)
+				continue;
 
-		eth.addr_bytes[0] = 0xe0;
-		eth.addr_bytes[1] = 0xe0;
-		eth.addr_bytes[2] = 0xe0;
-		eth.addr_bytes[3] = 0xe0;
-		eth.addr_bytes[4] = portid + 0xf0;
+			eth.addr_bytes[0] = 0xe0;
+			eth.addr_bytes[1] = 0xe0;
+			eth.addr_bytes[2] = 0xe0;
+			eth.addr_bytes[3] = 0xe0;
+			eth.addr_bytes[4] = portid + 0xf0;
 
-		if (port_init(portid, mbuf_pool) != 0)
-			rte_exit(EXIT_FAILURE, "Cannot init port %"PRIu8 "\n",
+			if (port_init(portid, mbuf_pool) != 0)
+				rte_exit(EXIT_FAILURE,
+					"Cannot init port %"PRIu8 "\n",
 					portid);
 
-		for (w = 0; w < MAX_VFS; w++) {
-			eth.addr_bytes[5] = w + 0xf0;
-
-			ret = rte_pmd_ixgbe_set_vf_mac_addr(portid,
-						w, &eth);
-			if (ret == -ENOTSUP)
-				ret = rte_pmd_i40e_set_vf_mac_addr(portid,
-						w, &eth);
-			if (ret == -ENOTSUP)
-				ret = rte_pmd_bnxt_set_vf_mac_addr(portid,
-						w, &eth);
-
-			switch (ret) {
-			case 0:
-				printf("Port %d VF %d MAC: ",
-						portid, w);
-				for (j = 0; j < 6; j++) {
-					printf("%02x", eth.addr_bytes[j]);
-					if (j < 5)
-						printf(":");
+			for (w = 0; w < MAX_VFS; w++) {
+				eth.addr_bytes[5] = w + 0xf0;
+
+				ret = rte_pmd_ixgbe_set_vf_mac_addr(portid,
+							w, &eth);
+				if (ret == -ENOTSUP)
+					ret = rte_pmd_i40e_set_vf_mac_addr(
+							portid, w, &eth);
+				if (ret == -ENOTSUP)
+					ret = rte_pmd_bnxt_set_vf_mac_addr(
+							portid, w, &eth);
+
+				switch (ret) {
+				case 0:
+					printf("Port %d VF %d MAC: ",
+							portid, w);
+					for (j = 0; j < 5; j++) {
+						printf("%02x:",
+							eth.addr_bytes[j]);
+					}
+					printf("%02x\n", eth.addr_bytes[5]);
+					break;
 				}
 				printf("\n");
 				break;

Signature · 1 line

2.17.1

David Hunt

author

2018-07-13 14:23 UTC #8 lore raw

PATCH v4 2/9 examples/vm_power: add core list parameter

In reply to David Hunt

Add in the '-l' command line parameter (also --core-list) So the user can now pass --corelist=4,6,8-10 and it will expand out to 4,6,8,9,10 using the parse function provided in parse.c (parse_set).

This list of cores is then used to enable out-of-band monitoring to scale up and down these cores based on the ratio of branch hits versus branch misses. The ratio will be low when a poll loop is spinning with no packets being received, so the frequency will be scaled down.

Also , as part of this change, we introduce a core_info struct which keeps information on each core in the system, and whether we're doing out of band monitoring on them.

Signed-off-by David Hunt <[email protected]>

Acked-by Radu Nicolau <[email protected]>

---
 examples/vm_power_manager/Makefile        |  2 +-
 examples/vm_power_manager/main.c          | 34 +++++++++-
 examples/vm_power_manager/parse.c         | 81 +++++++++++++++++++++++
 examples/vm_power_manager/parse.h         | 20 ++++++
 examples/vm_power_manager/power_manager.c | 31 +++++++++
 examples/vm_power_manager/power_manager.h | 20 ++++++
 6 files changed, 185 insertions(+), 3 deletions(-)
 create mode 100644 examples/vm_power_manager/parse.c
 create mode 100644 examples/vm_power_manager/parse.h

examples/vm_power_manager/Makefile +1 -1

index ef2a9f959..0c925967c 100644
@@ -19,7 +19,7 @@ APP = vm_power_mgr
 
 # all source are stored in SRCS-y
 SRCS-y := main.c vm_power_cli.c power_manager.c channel_manager.c
-SRCS-y += channel_monitor.c
+SRCS-y += channel_monitor.c parse.c
 
 CFLAGS += -O3 -I$(RTE_SDK)/lib/librte_power/
 CFLAGS += $(WERROR_FLAGS)

examples/vm_power_manager/main.c +32 -2

index 0d3846971..613a40af0 100644
@@ -29,6 +29,7 @@
 #include "channel_monitor.h"
 #include "power_manager.h"
 #include "vm_power_cli.h"
+#include "parse.h"
 #include <rte_pmd_ixgbe.h>
 #include <rte_pmd_i40e.h>
 #include <rte_pmd_bnxt.h>

@@ -133,18 +134,22 @@ parse_portmask(const char *portmask)
 static int
 parse_args(int argc, char **argv)
 {
-	int opt, ret;
+	int opt, ret, cnt, i;
 	char **argvopt;
+	uint16_t *oob_enable;
 	int option_index;
 	char *prgname = argv[0];
+	struct core_info *ci;
 	static struct option lgopts[] = {
 		{ "mac-updating", no_argument, 0, 1},
 		{ "no-mac-updating", no_argument, 0, 0},
+		{ "core-list", optional_argument, 0, 'l'},
 		{NULL, 0, 0, 0}
 	};
 	argvopt = argv;
+	ci = get_core_info();
 
-	while ((opt = getopt_long(argc, argvopt, "p:q:T:",
+	while ((opt = getopt_long(argc, argvopt, "l:p:q:T:",
 				  lgopts, &option_index)) != EOF) {
 
 		switch (opt) {

@@ -156,6 +161,27 @@ parse_args(int argc, char **argv)
 				return -1;
 			}
 			break;
+		case 'l':
+			oob_enable = malloc(ci->core_count * sizeof(uint16_t));
+			if (oob_enable == NULL) {
+				printf("Error - Unable to allocate memory\n");
+				return -1;
+			}
+			cnt = parse_set(optarg, oob_enable, ci->core_count);
+			if (cnt < 0) {
+				printf("Invalid core-list - [%s]\n",
+						optarg);
+				break;
+			}
+			for (i = 0; i < ci->core_count; i++) {
+				if (oob_enable[i]) {
+					printf("***Using core %d\n", i);
+					ci->cd[i].oob_enabled = 1;
+					ci->cd[i].global_enabled_cpus = 1;
+				}
+			}
+			free(oob_enable);
+			break;
 		/* long options */
 		case 0:
 			break;

@@ -261,6 +287,10 @@ main(int argc, char **argv)
 	uint16_t portid;
 
 
+	ret = core_info_init();
+	if (ret < 0)
+		rte_panic("Cannot allocate core info\n");
+
 	ret = rte_eal_init(argc, argv);
 	if (ret < 0)
 		rte_panic("Cannot init EAL\n");

examples/vm_power_manager/parse.c new file +81 -0

81 lines not on this page · raw message

examples/vm_power_manager/parse.h new file +20 -0

new file mode 100644
index 000000000..a5971e9a2
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef PARSE_H_
+#define PARSE_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+int
+parse_set(const char *, uint16_t [], unsigned int);
+
+#ifdef __cplusplus
+}
+#endif
+
+
+#endif /* PARSE_H_ */

examples/vm_power_manager/power_manager.c +31 -0

index 35db25591..a7849e48a 100644
@@ -12,6 +12,7 @@
 #include <dirent.h>
 #include <errno.h>
 
+#include <sys/sysinfo.h>
 #include <sys/types.h>
 
 #include <rte_log.h>

@@ -54,6 +55,7 @@ struct freq_info {
 
 static struct freq_info global_core_freq_info[POWER_MGR_MAX_CPUS];
 
+struct core_info ci;
 static uint64_t global_enabled_cpus;
 
 #define SYSFS_CPU_PATH "/sys/devices/system/cpu/cpu%u/topology/core_id"

@@ -76,6 +78,35 @@ set_host_cpus_mask(void)
 	return num_cpus;
 }
 
+struct core_info *
+get_core_info(void)
+{
+	return &ci;
+}
+
+int
+core_info_init(void)
+{
+	struct core_info *ci;
+	int i;
+
+	ci = get_core_info();
+
+	ci->core_count = get_nprocs_conf();
+	ci->cd = malloc(ci->core_count * sizeof(struct core_details));
+	if (!ci->cd) {
+		RTE_LOG(ERR, POWER_MANAGER, "Failed to allocate memory for core info.");
+		return -1;
+	}
+	for (i = 0; i < ci->core_count; i++) {
+		ci->cd[i].global_enabled_cpus = 1;
+		ci->cd[i].oob_enabled = 0;
+		ci->cd[i].msr_fd = 0;
+	}
+	printf("%d cores in system\n", ci->core_count);
+	return 0;
+}
+
 int
 power_manager_init(void)
 {

examples/vm_power_manager/power_manager.h +20 -0

index 8a8a84aa4..45385de37 100644
@@ -8,6 +8,26 @@
 #ifdef __cplusplus
 extern "C" {
 #endif
+struct core_details {
+	uint64_t last_branches;
+	uint64_t last_branch_misses;
+	uint16_t global_enabled_cpus;
+	uint16_t oob_enabled;
+	int msr_fd;
+};
+
+struct core_info {
+	uint16_t core_count;
+	struct core_details *cd;
+};
+
+struct core_info *
+get_core_info(void);
+
+int
+core_info_init(void);
+
+#define RTE_LOGTYPE_POWER_MANAGER RTE_LOGTYPE_USER1
 
 /* Maximum number of CPUS to manage */
 #define POWER_MGR_MAX_CPUS 64

Signature · 1 line

2.17.1

David Hunt

author

2018-07-13 14:23 UTC #9 lore raw

PATCH v4 3/9 examples/vm_power: add oob monitoring functions

In reply to David Hunt

This patch introduces the out-of-band (oob) core monitoring functions.

The functions are similar to the channel manager functions. There are function to add and remove cores from the list of cores being monitored. There is a function to initialise the monitor setup, run the monitor thread, and exit the monitor.

The monitor thread runs in it's own lcore, and is separate functionality to the channel monitor which is epoll based. THis thread is timer based. It loops through all monitored cores, calculates the branch ratio, scales up or down the core, then sleeps for an interval (~250 uS).

The method it uses to read the branch counters is a pread on the /dev/cpu/x/msr file, so the 'msr' kernel module needs to be loaded. Also, since the msr.h file has been made unavailable in recent kernels, we have #defines for the relevant MSRs included in the code.

The makefile has a switch for x86 and non-x86 platforms, and compiles stub function for non-x86 platforms.

Signed-off-by David Hunt <[email protected]>

Acked-by Radu Nicolau <[email protected]>

---
 examples/vm_power_manager/Makefile          |   5 +
 examples/vm_power_manager/oob_monitor.h     |  68 +++++
 examples/vm_power_manager/oob_monitor_nop.c |  38 +++
 examples/vm_power_manager/oob_monitor_x86.c | 259 ++++++++++++++++++++
 4 files changed, 370 insertions(+)
 create mode 100644 examples/vm_power_manager/oob_monitor.h
 create mode 100644 examples/vm_power_manager/oob_monitor_nop.c
 create mode 100644 examples/vm_power_manager/oob_monitor_x86.c

examples/vm_power_manager/Makefile +5 -0

index 0c925967c..13a5205ba 100644
@@ -20,6 +20,11 @@ APP = vm_power_mgr
 # all source are stored in SRCS-y
 SRCS-y := main.c vm_power_cli.c power_manager.c channel_manager.c
 SRCS-y += channel_monitor.c parse.c
+ifeq ($(CONFIG_RTE_ARCH_X86_64),y)
+SRCS-y += oob_monitor_x86.c
+else
+SRCS-y += oob_monitor_nop.c
+endif
 
 CFLAGS += -O3 -I$(RTE_SDK)/lib/librte_power/
 CFLAGS += $(WERROR_FLAGS)

examples/vm_power_manager/oob_monitor.h new file +68 -0

68 lines not on this page · raw message

examples/vm_power_manager/oob_monitor_nop.c new file +38 -0

new file mode 100644
index 000000000..7e7b8bc14
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2014 Intel Corporation
+ */
+
+#include "oob_monitor.h"
+
+void branch_monitor_exit(void)
+{
+}
+
+__attribute__((unused)) static float
+apply_policy(__attribute__((unused)) int core)
+{
+	return 0.0;
+}
+
+int
+add_core_to_monitor(__attribute__((unused)) int core)
+{
+	return 0;
+}
+
+int
+remove_core_from_monitor(__attribute__((unused)) int core)
+{
+	return 0;
+}
+
+int
+branch_monitor_init(void)
+{
+	return 0;
+}
+
+void
+run_branch_monitor(void)
+{
+}

examples/vm_power_manager/oob_monitor_x86.c new file +259 -0

259 lines not on this page · raw message

Signature · 1 line

2.17.1

David Hunt

author

2018-07-13 14:23 UTC #10 lore raw

PATCH v4 4/9 examples/vm_power: allow greater than 64 cores

In reply to David Hunt

To facilitate more info per core, change the global_cpu_mask from a uint64_t to an array. This also removes the limit on 64 cores, allocing the aray at run-time based on the number of cores found in the system.

Signed-off-by David Hunt <[email protected]>

Acked-by Radu Nicolau <[email protected]>

---
 examples/vm_power_manager/power_manager.c | 115 +++++++++++-----------
 1 file changed, 58 insertions(+), 57 deletions(-)

examples/vm_power_manager/power_manager.c +58 ~~-57~~

index a7849e48a..4bdde23da 100644
@@ -19,14 +19,14 @@
 #include <rte_power.h>
 #include <rte_spinlock.h>
 
+#include "channel_manager.h"
 #include "power_manager.h"
-
-#define RTE_LOGTYPE_POWER_MANAGER RTE_LOGTYPE_USER1
+#include "oob_monitor.h"
 
 #define POWER_SCALE_CORE(DIRECTION, core_num , ret) do { \
-	if (core_num >= POWER_MGR_MAX_CPUS) \
+	if (core_num >= ci.core_count) \
 		return -1; \
-	if (!(global_enabled_cpus & (1ULL << core_num))) \
+	if (!(ci.cd[core_num].global_enabled_cpus)) \
 		return -1; \
 	rte_spinlock_lock(&global_core_freq_info[core_num].power_sl); \
 	ret = rte_power_freq_##DIRECTION(core_num); \

@@ -37,7 +37,7 @@
 	int i; \
 	for (i = 0; core_mask; core_mask &= ~(1 << i++)) { \
 		if ((core_mask >> i) & 1) { \
-			if (!(global_enabled_cpus & (1ULL << i))) \
+			if (!(ci.cd[i].global_enabled_cpus)) \
 				continue; \
 			rte_spinlock_lock(&global_core_freq_info[i].power_sl); \
 			if (rte_power_freq_##DIRECTION(i) != 1) \

@@ -56,28 +56,9 @@ struct freq_info {
 static struct freq_info global_core_freq_info[POWER_MGR_MAX_CPUS];
 
 struct core_info ci;
-static uint64_t global_enabled_cpus;
 
 #define SYSFS_CPU_PATH "/sys/devices/system/cpu/cpu%u/topology/core_id"
 
-static unsigned
-set_host_cpus_mask(void)
-{
-	char path[PATH_MAX];
-	unsigned i;
-	unsigned num_cpus = 0;
-
-	for (i = 0; i < POWER_MGR_MAX_CPUS; i++) {
-		snprintf(path, sizeof(path), SYSFS_CPU_PATH, i);
-		if (access(path, F_OK) == 0) {
-			global_enabled_cpus |= 1ULL << i;
-			num_cpus++;
-		} else
-			return num_cpus;
-	}
-	return num_cpus;
-}
-
 struct core_info *
 get_core_info(void)
 {

@@ -110,38 +91,45 @@ core_info_init(void)
 int
 power_manager_init(void)
 {
-	unsigned int i, num_cpus, num_freqs;
-	uint64_t cpu_mask;
+	unsigned int i, num_cpus = 0, num_freqs = 0;
 	int ret = 0;
+	struct core_info *ci;
+
+	rte_power_set_env(PM_ENV_ACPI_CPUFREQ);
 
-	num_cpus = set_host_cpus_mask();
-	if (num_cpus == 0) {
-		RTE_LOG(ERR, POWER_MANAGER, "Unable to detected host CPUs, please "
-			"ensure that sufficient privileges exist to inspect sysfs\n");
+	ci = get_core_info();
+	if (!ci) {
+		RTE_LOG(ERR, POWER_MANAGER,
+				"Failed to get core info!\n");
 		return -1;
 	}
-	rte_power_set_env(PM_ENV_ACPI_CPUFREQ);
-	cpu_mask = global_enabled_cpus;
-	for (i = 0; cpu_mask; cpu_mask &= ~(1 << i++)) {
-		if (rte_power_init(i) < 0)
-			RTE_LOG(ERR, POWER_MANAGER,
-					"Unable to initialize power manager "
-					"for core %u\n", i);
-		num_freqs = rte_power_freqs(i, global_core_freq_info[i].freqs,
+
+	for (i = 0; i < ci->core_count; i++) {
+		if (ci->cd[i].global_enabled_cpus) {
+			if (rte_power_init(i) < 0)
+				RTE_LOG(ERR, POWER_MANAGER,
+						"Unable to initialize power manager "
+						"for core %u\n", i);
+			num_cpus++;
+			num_freqs = rte_power_freqs(i,
+					global_core_freq_info[i].freqs,
 					RTE_MAX_LCORE_FREQS);
-		if (num_freqs == 0) {
-			RTE_LOG(ERR, POWER_MANAGER,
-				"Unable to get frequency list for core %u\n",
-				i);
-			global_enabled_cpus &= ~(1 << i);
-			num_cpus--;
-			ret = -1;
+			if (num_freqs == 0) {
+				RTE_LOG(ERR, POWER_MANAGER,
+					"Unable to get frequency list for core %u\n",
+					i);
+				ci->cd[i].oob_enabled = 0;
+				ret = -1;
+			}
+			global_core_freq_info[i].num_freqs = num_freqs;
+
+			rte_spinlock_init(&global_core_freq_info[i].power_sl);
 		}
-		global_core_freq_info[i].num_freqs = num_freqs;
-		rte_spinlock_init(&global_core_freq_info[i].power_sl);
+		if (ci->cd[i].oob_enabled)
+			add_core_to_monitor(i);
 	}
-	RTE_LOG(INFO, POWER_MANAGER, "Detected %u host CPUs , enabled core mask:"
-					" 0x%"PRIx64"\n", num_cpus, global_enabled_cpus);
+	RTE_LOG(INFO, POWER_MANAGER, "Managing %u cores out of %u available host cores\n",
+			num_cpus, ci->core_count);
 	return ret;
 
 }

@@ -156,7 +144,7 @@ power_manager_get_current_frequency(unsigned core_num)
 				core_num, POWER_MGR_MAX_CPUS-1);
 		return -1;
 	}
-	if (!(global_enabled_cpus & (1ULL << core_num)))
+	if (!(ci.cd[core_num].global_enabled_cpus))
 		return 0;
 
 	rte_spinlock_lock(&global_core_freq_info[core_num].power_sl);

@@ -175,15 +163,26 @@ power_manager_exit(void)
 {
 	unsigned int i;
 	int ret = 0;
+	struct core_info *ci;
 
-	for (i = 0; global_enabled_cpus; global_enabled_cpus &= ~(1 << i++)) {
-		if (rte_power_exit(i) < 0) {
-			RTE_LOG(ERR, POWER_MANAGER, "Unable to shutdown power manager "
-					"for core %u\n", i);
-			ret = -1;
+	ci = get_core_info();
+	if (!ci) {
+		RTE_LOG(ERR, POWER_MANAGER,
+				"Failed to get core info!\n");
+		return -1;
+	}
+
+	for (i = 0; i < ci->core_count; i++) {
+		if (ci->cd[i].global_enabled_cpus) {
+			if (rte_power_exit(i) < 0) {
+				RTE_LOG(ERR, POWER_MANAGER, "Unable to shutdown power manager "
+						"for core %u\n", i);
+				ret = -1;
+			}
+			ci->cd[i].global_enabled_cpus = 0;
 		}
+		remove_core_from_monitor(i);
 	}
-	global_enabled_cpus = 0;
 	return ret;
 }

@@ -299,10 +298,12 @@ int
 power_manager_scale_core_med(unsigned int core_num)
 {
 	int ret = 0;
+	struct core_info *ci;
 
+	ci = get_core_info();
 	if (core_num >= POWER_MGR_MAX_CPUS)
 		return -1;
-	if (!(global_enabled_cpus & (1ULL << core_num)))
+	if (!(ci->cd[core_num].global_enabled_cpus))
 		return -1;
 	rte_spinlock_lock(&global_core_freq_info[core_num].power_sl);
 	ret = rte_power_set_freq(core_num,

Signature · 1 line

2.17.1

David Hunt

author

2018-07-13 14:23 UTC #11 lore raw

PATCH v4 5/9 examples/vm_power: add thread for oob core monitor

In reply to David Hunt

Change the app to now require three cores, as the third core will be used to run the oob montoring thread.

Signed-off-by David Hunt <[email protected]>

Acked-by Radu Nicolau <[email protected]>

---
 examples/vm_power_manager/main.c | 37 +++++++++++++++++++++++++++++---
 1 file changed, 34 insertions(+), 3 deletions(-)

examples/vm_power_manager/main.c +34 -3

index 613a40af0..aef97b9ae 100644
@@ -29,6 +29,7 @@
 #include "channel_monitor.h"
 #include "power_manager.h"
 #include "vm_power_cli.h"
+#include "oob_monitor.h"
 #include "parse.h"
 #include <rte_pmd_ixgbe.h>
 #include <rte_pmd_i40e.h>

@@ -267,6 +268,17 @@ run_monitor(__attribute__((unused)) void *arg)
 	return 0;
 }
 
+static int
+run_core_monitor(__attribute__((unused)) void *arg)
+{
+	if (branch_monitor_init() < 0) {
+		printf("Unable to initialize core monitor\n");
+		return -1;
+	}
+	run_branch_monitor();
+	return 0;
+}
+
 static void
 sig_handler(int signo)
 {

@@ -285,12 +297,15 @@ main(int argc, char **argv)
 	unsigned int nb_ports;
 	struct rte_mempool *mbuf_pool;
 	uint16_t portid;
+	struct core_info *ci;
 
 
 	ret = core_info_init();
 	if (ret < 0)
 		rte_panic("Cannot allocate core info\n");
 
+	ci = get_core_info();
+
 	ret = rte_eal_init(argc, argv);
 	if (ret < 0)
 		rte_panic("Cannot init EAL\n");

@@ -365,16 +380,23 @@ main(int argc, char **argv)
 		}
 	}
 
+	check_all_ports_link_status(enabled_port_mask);
+
 	lcore_id = rte_get_next_lcore(-1, 1, 0);
 	if (lcore_id == RTE_MAX_LCORE) {
-		RTE_LOG(ERR, EAL, "A minimum of two cores are required to run "
+		RTE_LOG(ERR, EAL, "A minimum of three cores are required to run "
 				"application\n");
 		return 0;
 	}
-
-	check_all_ports_link_status(enabled_port_mask);
+	printf("Running channel monitor on lcore id %d\n", lcore_id);
 	rte_eal_remote_launch(run_monitor, NULL, lcore_id);
 
+	lcore_id = rte_get_next_lcore(lcore_id, 1, 0);
+	if (lcore_id == RTE_MAX_LCORE) {
+		RTE_LOG(ERR, EAL, "A minimum of three cores are required to run "
+				"application\n");
+		return 0;
+	}
 	if (power_manager_init() < 0) {
 		printf("Unable to initialize power manager\n");
 		return -1;

@@ -383,8 +405,17 @@ main(int argc, char **argv)
 		printf("Unable to initialize channel manager\n");
 		return -1;
 	}
+
+	printf("Running core monitor on lcore id %d\n", lcore_id);
+	rte_eal_remote_launch(run_core_monitor, NULL, lcore_id);
+
 	run_cli(NULL);
 
+	branch_monitor_exit();
+
 	rte_eal_mp_wait_lcore();
+
+	free(ci->cd);
+
 	return 0;
 }

Signature · 1 line

2.17.1

David Hunt

author

2018-07-13 14:23 UTC #12 lore raw

PATCH v4 6/9 examples/vm_power: add port-list to command line

In reply to David Hunt

add in the long form of -p, which is --port-list

Signed-off-by David Hunt <[email protected]>

Acked-by Radu Nicolau <[email protected]>

---
 examples/vm_power_manager/main.c | 1 +
 1 file changed, 1 insertion(+)

examples/vm_power_manager/main.c +1 -0

index aef97b9ae..f9990f153 100644
@@ -145,6 +145,7 @@ parse_args(int argc, char **argv)
 		{ "mac-updating", no_argument, 0, 1},
 		{ "no-mac-updating", no_argument, 0, 0},
 		{ "core-list", optional_argument, 0, 'l'},
+		{ "port-list", optional_argument, 0, 'p'},
 		{NULL, 0, 0, 0}
 	};
 	argvopt = argv;

Signature · 1 line

2.17.1

David Hunt

author

2018-07-13 14:23 UTC #13 lore raw

PATCH v4 7/9 examples/vm_power: add branch ratio policy type

In reply to David Hunt

Add the capability for the vm_power_manager to receive a policy of type BRANCH_RATIO. This will add any vcpus in the policy to the oob monitoring thread.

Signed-off-by David Hunt <[email protected]>

Acked-by Radu Nicolau <[email protected]>

---
 examples/vm_power_manager/channel_monitor.c | 23 +++++++++++++++++++--
 lib/librte_power/channel_commands.h         |  3 ++-
 2 files changed, 23 insertions(+), 3 deletions(-)

examples/vm_power_manager/channel_monitor.c +21 -2

index 73bddd993..7fa47ba97 100644
@@ -27,6 +27,7 @@
 #include "channel_commands.h"
 #include "channel_manager.h"
 #include "power_manager.h"
+#include "oob_monitor.h"
 
 #define RTE_LOGTYPE_CHANNEL_MONITOR RTE_LOGTYPE_USER1

@@ -92,6 +93,10 @@ get_pcpu_to_control(struct policy *pol)
 	struct vm_info info;
 	int pcpu, count;
 	uint64_t mask_u64b;
+	struct core_info *ci;
+	int ret;
+
+	ci = get_core_info();
 
 	RTE_LOG(INFO, CHANNEL_MONITOR, "Looking for pcpu for %s\n",
 			pol->pkt.vm_name);

@@ -100,8 +105,22 @@ get_pcpu_to_control(struct policy *pol)
 	for (count = 0; count < pol->pkt.num_vcpu; count++) {
 		mask_u64b = info.pcpu_mask[pol->pkt.vcpu_to_control[count]];
 		for (pcpu = 0; mask_u64b; mask_u64b &= ~(1ULL << pcpu++)) {
-			if ((mask_u64b >> pcpu) & 1)
-				pol->core_share[count].pcpu = pcpu;
+			if ((mask_u64b >> pcpu) & 1) {
+				if (pol->pkt.policy_to_use == BRANCH_RATIO) {
+					ci->cd[pcpu].oob_enabled = 1;
+					ret = add_core_to_monitor(pcpu);
+					if (ret == 0)
+						printf("Monitoring pcpu %d via Branch Ratio\n",
+								pcpu);
+					else
+						printf("Failed to start OOB Monitoring pcpu %d\n",
+								pcpu);
+
+				} else {
+					pol->core_share[count].pcpu = pcpu;
+					printf("Monitoring pcpu %d\n", pcpu);
+				}
+			}
 		}
 	}
 }

lib/librte_power/channel_commands.h +2 -1

index 5e8b4ab5d..ee638eefa 100644
@@ -48,7 +48,8 @@ enum workload {HIGH, MEDIUM, LOW};
 enum policy_to_use {
 	TRAFFIC,
 	TIME,
-	WORKLOAD
+	WORKLOAD,
+	BRANCH_RATIO
 };
 
 struct traffic {

Signature · 1 line

2.17.1

David Hunt

author

2018-07-13 14:23 UTC #14 lore raw

PATCH v4 8/9 examples/vm_power: add cli args to guest app

In reply to David Hunt

Add new command line arguments to the guest app to make
testing and validation of the policy usage easier. These arguments are mainly around setting up the power management policy that is sent from the guest vm to to the vm_power_manager in the host

New command line parameters:
-n or --vm-name
sets the name of the vm to be used by the host OS.
-b or --busy-hours
sets the list of hours that are predicted to be busy
-q or --quiet-hours
sets the list of hours that are predicted to be quiet
-l or --vcpu-list
sets the list of vcpus to monitor
-p or --port-list
sets the list of posts to monitor when using a workload policy.
-o or --policy
sets the default policy type
TIME
WORKLOAD
TRAFFIC
BRANCH_RATIO

The format of the hours or list paramers is a comma-separated list of integers, which can take the form of
a. x e.g. --vcpu-list=1
b. x,y e.g. --quiet-hours=3,4
c. x-y e.g. --busy-hours=9-12
d. combination of above (e.g. --busy-hours=4,5-7,9)

Signed-off-by David Hunt <[email protected]>

Acked-by Radu Nicolau <[email protected]>

---
 examples/vm_power_manager/guest_cli/Makefile  |   2 +-
 examples/vm_power_manager/guest_cli/main.c    | 151 +++++++++++++++++-
 examples/vm_power_manager/guest_cli/parse.c   |  82 ++++++++++
 examples/vm_power_manager/guest_cli/parse.h   |  19 +++
 .../guest_cli/vm_power_cli_guest.c            | 113 +++++++------
 .../guest_cli/vm_power_cli_guest.h            |   6 +
 6 files changed, 319 insertions(+), 54 deletions(-)
 create mode 100644 examples/vm_power_manager/guest_cli/parse.c
 create mode 100644 examples/vm_power_manager/guest_cli/parse.h

examples/vm_power_manager/guest_cli/Makefile +1 -1

index d710e22d9..8b1db861e 100644
@@ -14,7 +14,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 APP = guest_vm_power_mgr
 
 # all source are stored in SRCS-y
-SRCS-y := main.c vm_power_cli_guest.c
+SRCS-y := main.c vm_power_cli_guest.c parse.c
 
 CFLAGS += -O3 -I$(RTE_SDK)/lib/librte_power/
 CFLAGS += $(WERROR_FLAGS)

examples/vm_power_manager/guest_cli/main.c +143 -8

178 lines not on this page · raw message

examples/vm_power_manager/guest_cli/parse.c new file +82 -0

82 lines not on this page · raw message

examples/vm_power_manager/guest_cli/parse.h new file +19 -0

new file mode 100644
index 000000000..c8aa0ea50
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef PARSE_H_
+#define PARSE_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+int
+parse_set(const char *, uint16_t [], unsigned int);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* PARSE_H_ */

examples/vm_power_manager/guest_cli/vm_power_cli_guest.c +68 ~~-45~~

135 lines not on this page · raw message

examples/vm_power_manager/guest_cli/vm_power_cli_guest.h +6 -0

index 75a262967..fd77f6a69 100644
@@ -11,6 +11,12 @@ extern "C" {
 
 #include "channel_commands.h"
 
+struct channel_packet *get_policy(void);
+
+int set_policy_mac(int port, int idx);
+
+void set_policy_defaults(struct channel_packet *pkt);
+
 void run_cli(__attribute__((unused)) void *arg);
 
 #ifdef __cplusplus

Signature · 1 line

2.17.1

David Hunt

author

2018-07-13 14:23 UTC #15 lore raw

PATCH v4 9/9 examples/vm_power: make branch ratio configurable

In reply to David Hunt

For different workloads and poll loops, the theshold may be different for when you want to scale up and down.

This patch allows changing of the default branch ratio by using the -b command line argument (or --branch-ratio=)

Signed-off-by David Hunt <[email protected]>

Acked-by Radu Nicolau <[email protected]>

---
 examples/vm_power_manager/main.c            | 16 +++++++++++++++-
 examples/vm_power_manager/oob_monitor_x86.c |  3 +--
 examples/vm_power_manager/power_manager.c   |  1 +
 examples/vm_power_manager/power_manager.h   |  3 +++
 4 files changed, 20 insertions(+), 3 deletions(-)

examples/vm_power_manager/main.c +15 -1

index f9990f153..58c5fa45c 100644
@@ -141,17 +141,19 @@ parse_args(int argc, char **argv)
 	int option_index;
 	char *prgname = argv[0];
 	struct core_info *ci;
+	float branch_ratio;
 	static struct option lgopts[] = {
 		{ "mac-updating", no_argument, 0, 1},
 		{ "no-mac-updating", no_argument, 0, 0},
 		{ "core-list", optional_argument, 0, 'l'},
 		{ "port-list", optional_argument, 0, 'p'},
+		{ "branch-ratio", optional_argument, 0, 'b'},
 		{NULL, 0, 0, 0}
 	};
 	argvopt = argv;
 	ci = get_core_info();
 
-	while ((opt = getopt_long(argc, argvopt, "l:p:q:T:",
+	while ((opt = getopt_long(argc, argvopt, "l:p:q:T:b:",
 				  lgopts, &option_index)) != EOF) {
 
 		switch (opt) {

@@ -184,6 +186,18 @@ parse_args(int argc, char **argv)
 			}
 			free(oob_enable);
 			break;
+		case 'b':
+			branch_ratio = 0.0;
+			if (strlen(optarg))
+				branch_ratio = atof(optarg);
+			if (branch_ratio <= 0.0) {
+				printf("invalid branch ratio specified\n");
+				return -1;
+			}
+			ci->branch_ratio_threshold = branch_ratio;
+			printf("***Setting branch ratio to %f\n",
+					branch_ratio);
+			break;
 		/* long options */
 		case 0:
 			break;

examples/vm_power_manager/oob_monitor_x86.c +1 -2

index 62d503ca5..589c604e5 100644
@@ -22,7 +22,6 @@ void branch_monitor_exit(void)
 /* Number of microseconds between each poll */
 #define INTERVAL 100
 #define PRINT_LOOP_COUNT (1000000/INTERVAL)
-#define RATIO_THRESHOLD 0.03
 #define IA32_PERFEVTSEL0 0x186
 #define IA32_PERFEVTSEL1 0x187
 #define IA32_PERFCTR0 0xc1

@@ -89,7 +88,7 @@ apply_policy(int core)
 
 	ratio = (float)miss_diff * (float)100 / (float)hits_diff;
 
-	if (ratio < RATIO_THRESHOLD)
+	if (ratio < ci->branch_ratio_threshold)
 		power_manager_scale_core_min(core);
 	else
 		power_manager_scale_core_max(core);

examples/vm_power_manager/power_manager.c +1 -0

index 4bdde23da..b7769c3c3 100644
@@ -74,6 +74,7 @@ core_info_init(void)
 	ci = get_core_info();
 
 	ci->core_count = get_nprocs_conf();
+	ci->branch_ratio_threshold = BRANCH_RATIO_THRESHOLD;
 	ci->cd = malloc(ci->core_count * sizeof(struct core_details));
 	if (!ci->cd) {
 		RTE_LOG(ERR, POWER_MANAGER, "Failed to allocate memory for core info.");

examples/vm_power_manager/power_manager.h +3 -0

index 45385de37..605b3c8f6 100644
@@ -19,8 +19,11 @@ struct core_details {
 struct core_info {
 	uint16_t core_count;
 	struct core_details *cd;
+	float branch_ratio_threshold;
 };
 
+#define BRANCH_RATIO_THRESHOLD 0.1
+
 struct core_info *
 get_core_info(void);

Signature · 1 line

2.17.1

Thomas Monjalon

Applied

2018-07-20 22:06 UTC #16 lore raw

In reply to David Hunt

13/07/2018 16:22, David Hunt:

Quoted 9 lines

[1/9] examples/vm_power: add check for port count [2/9] examples/vm_power: add core list parameter [3/9] examples/vm_power: add oob monitoring functions [4/9] examples/vm_power: allow greater than 64 cores [5/9] examples/vm_power: add thread for oob core monitor [6/9] examples/vm_power: add port-list to command line [7/9] examples/vm_power: add branch ratio policy type [8/9] examples/vm_power: add cli args to guest app [9/9] examples/vm_power: make branch ratio configurable

Applied, thanks

Some features could be hosted in a DPDK library. It has been agreed in the technical board to use examples as a staging area, but it must re-considered regularly whether the code must stay, move or be removed.

David Hunt

author

2018-06-26 09:23 UTC #17 lore raw

PATCH v3 2/9 examples/vm_power: add core list parameter

In reply to David Hunt

Also , as part of this change, we introduce a core_info struct which keeps information on each core in the system, and whether we're doing out of band monitoring on them.

Signed-off-by David Hunt <[email protected]>

Acked-by Radu Nicolau <[email protected]>

---
 examples/vm_power_manager/Makefile        |  2 +-
 examples/vm_power_manager/main.c          | 34 ++++++++-
 examples/vm_power_manager/parse.c         | 93 +++++++++++++++++++++++
 examples/vm_power_manager/parse.h         | 20 +++++
 examples/vm_power_manager/power_manager.c | 31 ++++++++
 examples/vm_power_manager/power_manager.h | 20 +++++
 6 files changed, 197 insertions(+), 3 deletions(-)
 create mode 100644 examples/vm_power_manager/parse.c
 create mode 100644 examples/vm_power_manager/parse.h

examples/vm_power_manager/Makefile +1 -1

index ef2a9f959..0c925967c 100644
@@ -19,7 +19,7 @@ APP = vm_power_mgr
 
 # all source are stored in SRCS-y
 SRCS-y := main.c vm_power_cli.c power_manager.c channel_manager.c
-SRCS-y += channel_monitor.c
+SRCS-y += channel_monitor.c parse.c
 
 CFLAGS += -O3 -I$(RTE_SDK)/lib/librte_power/
 CFLAGS += $(WERROR_FLAGS)

examples/vm_power_manager/main.c +32 -2

index 043b374bc..cc2a1289c 100644
@@ -29,6 +29,7 @@
 #include "channel_monitor.h"
 #include "power_manager.h"
 #include "vm_power_cli.h"
+#include "parse.h"
 #include <rte_pmd_ixgbe.h>
 #include <rte_pmd_i40e.h>
 #include <rte_pmd_bnxt.h>

@@ -135,18 +136,22 @@ parse_portmask(const char *portmask)
 static int
 parse_args(int argc, char **argv)
 {
-	int opt, ret;
+	int opt, ret, cnt, i;
 	char **argvopt;
+	uint16_t *oob_enable;
 	int option_index;
 	char *prgname = argv[0];
+	struct core_info *ci;
 	static struct option lgopts[] = {
 		{ "mac-updating", no_argument, 0, 1},
 		{ "no-mac-updating", no_argument, 0, 0},
+		{ "core-list", optional_argument, 0, 'l'},
 		{NULL, 0, 0, 0}
 	};
 	argvopt = argv;
+	ci = get_core_info();
 
-	while ((opt = getopt_long(argc, argvopt, "p:q:T:",
+	while ((opt = getopt_long(argc, argvopt, "l:p:q:T:",
 				  lgopts, &option_index)) != EOF) {
 
 		switch (opt) {

@@ -158,6 +163,27 @@ parse_args(int argc, char **argv)
 				return -1;
 			}
 			break;
+		case 'l':
+			oob_enable = malloc(ci->core_count * sizeof(uint16_t));
+			if (oob_enable == NULL) {
+				printf("Error - Unable to allocate memory\n");
+				return -1;
+			}
+			cnt = parse_set(optarg, oob_enable, ci->core_count);
+			if (cnt < 0) {
+				printf("Invalid core-list - [%s]\n",
+						optarg);
+				break;
+			}
+			for (i = 0; i < ci->core_count; i++) {
+				if (oob_enable[i]) {
+					printf("***Using core %d\n", i);
+					ci->cd[i].oob_enabled = 1;
+					ci->cd[i].global_enabled_cpus = 1;
+				}
+			}
+			free(oob_enable);
+			break;
 		/* long options */
 		case 0:
 			break;

@@ -263,6 +289,10 @@ main(int argc, char **argv)
 	uint16_t portid;
 
 
+	ret = core_info_init();
+	if (ret < 0)
+		rte_panic("Cannot allocate core info\n");
+
 	ret = rte_eal_init(argc, argv);
 	if (ret < 0)
 		rte_panic("Cannot init EAL\n");

examples/vm_power_manager/parse.c new file +93 -0

93 lines not on this page · raw message

examples/vm_power_manager/parse.h new file +20 -0

new file mode 100644
index 000000000..a5971e9a2
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef PARSE_H_
+#define PARSE_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+int
+parse_set(const char *, uint16_t [], unsigned int);
+
+#ifdef __cplusplus
+}
+#endif
+
+
+#endif /* PARSE_H_ */

examples/vm_power_manager/power_manager.c +31 -0

index 35db25591..a7849e48a 100644
@@ -12,6 +12,7 @@
 #include <dirent.h>
 #include <errno.h>
 
+#include <sys/sysinfo.h>
 #include <sys/types.h>
 
 #include <rte_log.h>

@@ -54,6 +55,7 @@ struct freq_info {
 
 static struct freq_info global_core_freq_info[POWER_MGR_MAX_CPUS];
 
+struct core_info ci;
 static uint64_t global_enabled_cpus;
 
 #define SYSFS_CPU_PATH "/sys/devices/system/cpu/cpu%u/topology/core_id"

@@ -76,6 +78,35 @@ set_host_cpus_mask(void)
 	return num_cpus;
 }
 
+struct core_info *
+get_core_info(void)
+{
+	return &ci;
+}
+
+int
+core_info_init(void)
+{
+	struct core_info *ci;
+	int i;
+
+	ci = get_core_info();
+
+	ci->core_count = get_nprocs_conf();
+	ci->cd = malloc(ci->core_count * sizeof(struct core_details));
+	if (!ci->cd) {
+		RTE_LOG(ERR, POWER_MANAGER, "Failed to allocate memory for core info.");
+		return -1;
+	}
+	for (i = 0; i < ci->core_count; i++) {
+		ci->cd[i].global_enabled_cpus = 1;
+		ci->cd[i].oob_enabled = 0;
+		ci->cd[i].msr_fd = 0;
+	}
+	printf("%d cores in system\n", ci->core_count);
+	return 0;
+}
+
 int
 power_manager_init(void)
 {

examples/vm_power_manager/power_manager.h +20 -0

index 8a8a84aa4..45385de37 100644
@@ -8,6 +8,26 @@
 #ifdef __cplusplus
 extern "C" {
 #endif
+struct core_details {
+	uint64_t last_branches;
+	uint64_t last_branch_misses;
+	uint16_t global_enabled_cpus;
+	uint16_t oob_enabled;
+	int msr_fd;
+};
+
+struct core_info {
+	uint16_t core_count;
+	struct core_details *cd;
+};
+
+struct core_info *
+get_core_info(void);
+
+int
+core_info_init(void);
+
+#define RTE_LOGTYPE_POWER_MANAGER RTE_LOGTYPE_USER1
 
 /* Maximum number of CPUS to manage */
 #define POWER_MGR_MAX_CPUS 64

Signature · 1 line

2.17.1

David Hunt

author

2018-06-26 09:23 UTC #18 lore raw

PATCH v3 3/9 examples/vm_power: add oob monitoring functions

In reply to David Hunt

This patch introduces the out-of-band (oob) core monitoring functions.

The makefile has a switch for x86 and non-x86 platforms, and compiles stub function for non-x86 platforms.

Signed-off-by David Hunt <[email protected]>

Acked-by Radu Nicolau <[email protected]>

---
 examples/vm_power_manager/Makefile          |   5 +
 examples/vm_power_manager/oob_monitor.h     |  68 +++++
 examples/vm_power_manager/oob_monitor_nop.c |  38 +++
 examples/vm_power_manager/oob_monitor_x86.c | 282 ++++++++++++++++++++
 4 files changed, 393 insertions(+)
 create mode 100644 examples/vm_power_manager/oob_monitor.h
 create mode 100644 examples/vm_power_manager/oob_monitor_nop.c
 create mode 100644 examples/vm_power_manager/oob_monitor_x86.c

examples/vm_power_manager/Makefile +5 -0

index 0c925967c..13a5205ba 100644
@@ -20,6 +20,11 @@ APP = vm_power_mgr
 # all source are stored in SRCS-y
 SRCS-y := main.c vm_power_cli.c power_manager.c channel_manager.c
 SRCS-y += channel_monitor.c parse.c
+ifeq ($(CONFIG_RTE_ARCH_X86_64),y)
+SRCS-y += oob_monitor_x86.c
+else
+SRCS-y += oob_monitor_nop.c
+endif
 
 CFLAGS += -O3 -I$(RTE_SDK)/lib/librte_power/
 CFLAGS += $(WERROR_FLAGS)

examples/vm_power_manager/oob_monitor.h new file +68 -0

68 lines not on this page · raw message

examples/vm_power_manager/oob_monitor_nop.c new file +38 -0

new file mode 100644
index 000000000..7e7b8bc14
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2014 Intel Corporation
+ */
+
+#include "oob_monitor.h"
+
+void branch_monitor_exit(void)
+{
+}
+
+__attribute__((unused)) static float
+apply_policy(__attribute__((unused)) int core)
+{
+	return 0.0;
+}
+
+int
+add_core_to_monitor(__attribute__((unused)) int core)
+{
+	return 0;
+}
+
+int
+remove_core_from_monitor(__attribute__((unused)) int core)
+{
+	return 0;
+}
+
+int
+branch_monitor_init(void)
+{
+	return 0;
+}
+
+void
+run_branch_monitor(void)
+{
+}

examples/vm_power_manager/oob_monitor_x86.c new file +282 -0

282 lines not on this page · raw message

Signature · 1 line

2.17.1

Thomas Monjalon

2018-07-12 19:13 UTC #19 lore raw

Re: [PATCH v3 3/9] examples/vm_power: add oob monitoring functions

26/06/2018 11:23, David Hunt:

Quoted 18 lines

examples/vm_power_manager/oob_monitor_x86.c new file +18 -0

+#include <unistd.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <signal.h>
+#include <errno.h>
+#include <string.h>
+#include <sys/types.h>
+#include <sys/epoll.h>
+#include <sys/queue.h>
+#include <sys/time.h>
+#include <fcntl.h>
+
+#include <rte_log.h>
+#include <rte_memory.h>
+#include <rte_malloc.h>
+#include <rte_atomic.h>
+#include <rte_cycles.h>

examples/vm_power_manager/oob_monitor_x86.c line 32

Quoted 10 lines

examples/vm_power_manager/oob_monitor_x86.c new file +10 -0

+#include <rte_ethdev.h>
+#include <rte_pmd_i40e.h>
+
+#include <libvirt/libvirt.h>
+#include "oob_monitor.h"
+#include "power_manager.h"
+#include "channel_manager.h"
+
+#include <rte_log.h>
+#define RTE_LOGTYPE_CHANNEL_MONITOR RTE_LOGTYPE_USER1

I'm sure you don't need all these headers.
rte_log.h is included twice.
rte_pmd_i40e is more than suspicious...

This is a hint that the whole file was probably written too fast :)

Stephen Hemminger

2018-07-12 22:18 UTC #20 lore raw

Re: [PATCH v3 3/9] examples/vm_power: add oob monitoring functions

Quoted 34 lines · On Thu, 12 Jul 2018 21:13:26 +0200 Thomas Monjalon <[email protected]> wrote:

26/06/2018 11:23, David Hunt:
Quoted 18 lines
examples/vm_power_manager/oob_monitor_x86.c new file +18 -0
+#include <unistd.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <signal.h>
+#include <errno.h>
+#include <string.h>
+#include <sys/types.h>
+#include <sys/epoll.h>
+#include <sys/queue.h>
+#include <sys/time.h>
+#include <fcntl.h>
+
+#include <rte_log.h>
+#include <rte_memory.h>
+#include <rte_malloc.h>
+#include <rte_atomic.h>
+#include <rte_cycles.h>
examples/vm_power_manager/oob_monitor_x86.c line 32
Quoted 10 lines
examples/vm_power_manager/oob_monitor_x86.c new file +10 -0
+#include <rte_ethdev.h>
+#include <rte_pmd_i40e.h>
+
+#include <libvirt/libvirt.h>
+#include "oob_monitor.h"
+#include "power_manager.h"
+#include "channel_manager.h"
+
+#include <rte_log.h>
+#define RTE_LOGTYPE_CHANNEL_MONITOR RTE_LOGTYPE_USER1  
I'm sure you don't need all these headers.
rte_log.h is included twice.
rte_pmd_i40e is more than suspicious...

Quoted 2 lines

This is a hint that the whole file was probably written too fast :)

This tool can help
https://include-what-you-use.org/

Hunt, David

author

2018-07-13 08:22 UTC #21 lore raw

Re: [PATCH v3 3/9] examples/vm_power: add oob monitoring functions

In reply to Thomas Monjalon

Hi Thomas,

Quoted 33 lines · Thomas Monjalon

26/06/2018 11:23, David Hunt:
Quoted 18 lines
examples/vm_power_manager/oob_monitor_x86.c new file +18 -0
+#include <unistd.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <signal.h>
+#include <errno.h>
+#include <string.h>
+#include <sys/types.h>
+#include <sys/epoll.h>
+#include <sys/queue.h>
+#include <sys/time.h>
+#include <fcntl.h>
+
+#include <rte_log.h>
+#include <rte_memory.h>
+#include <rte_malloc.h>
+#include <rte_atomic.h>
+#include <rte_cycles.h>
examples/vm_power_manager/oob_monitor_x86.c line 32
Quoted 10 lines
examples/vm_power_manager/oob_monitor_x86.c new file +10 -0
+#include <rte_ethdev.h>
+#include <rte_pmd_i40e.h>
+
+#include <libvirt/libvirt.h>
+#include "oob_monitor.h"
+#include "power_manager.h"
+#include "channel_manager.h"
+
+#include <rte_log.h>
+#define RTE_LOGTYPE_CHANNEL_MONITOR RTE_LOGTYPE_USER1
I'm sure you don't need all these headers.
rte_log.h is included twice.
rte_pmd_i40e is more than suspicious...

Quoted 1 line

This is a hint that the whole file was probably written too fast :)

Apologies, it was a cut-and-paste from another file in that same directory. I can clean it up and re-spin.

Regards,
Dave.

David Hunt

author

2018-06-26 09:23 UTC #22 lore raw

PATCH v3 4/9 examples/vm_power: allow greater than 64 cores

In reply to David Hunt

Signed-off-by David Hunt <[email protected]>

Acked-by Radu Nicolau <[email protected]>

---
 examples/vm_power_manager/power_manager.c | 115 +++++++++++-----------
 1 file changed, 58 insertions(+), 57 deletions(-)

examples/vm_power_manager/power_manager.c +58 ~~-57~~

index a7849e48a..4bdde23da 100644
@@ -19,14 +19,14 @@
 #include <rte_power.h>
 #include <rte_spinlock.h>
 
+#include "channel_manager.h"
 #include "power_manager.h"
-
-#define RTE_LOGTYPE_POWER_MANAGER RTE_LOGTYPE_USER1
+#include "oob_monitor.h"
 
 #define POWER_SCALE_CORE(DIRECTION, core_num , ret) do { \
-	if (core_num >= POWER_MGR_MAX_CPUS) \
+	if (core_num >= ci.core_count) \
 		return -1; \
-	if (!(global_enabled_cpus & (1ULL << core_num))) \
+	if (!(ci.cd[core_num].global_enabled_cpus)) \
 		return -1; \
 	rte_spinlock_lock(&global_core_freq_info[core_num].power_sl); \
 	ret = rte_power_freq_##DIRECTION(core_num); \

@@ -37,7 +37,7 @@
 	int i; \
 	for (i = 0; core_mask; core_mask &= ~(1 << i++)) { \
 		if ((core_mask >> i) & 1) { \
-			if (!(global_enabled_cpus & (1ULL << i))) \
+			if (!(ci.cd[i].global_enabled_cpus)) \
 				continue; \
 			rte_spinlock_lock(&global_core_freq_info[i].power_sl); \
 			if (rte_power_freq_##DIRECTION(i) != 1) \

@@ -56,28 +56,9 @@ struct freq_info {
 static struct freq_info global_core_freq_info[POWER_MGR_MAX_CPUS];
 
 struct core_info ci;
-static uint64_t global_enabled_cpus;
 
 #define SYSFS_CPU_PATH "/sys/devices/system/cpu/cpu%u/topology/core_id"
 
-static unsigned
-set_host_cpus_mask(void)
-{
-	char path[PATH_MAX];
-	unsigned i;
-	unsigned num_cpus = 0;
-
-	for (i = 0; i < POWER_MGR_MAX_CPUS; i++) {
-		snprintf(path, sizeof(path), SYSFS_CPU_PATH, i);
-		if (access(path, F_OK) == 0) {
-			global_enabled_cpus |= 1ULL << i;
-			num_cpus++;
-		} else
-			return num_cpus;
-	}
-	return num_cpus;
-}
-
 struct core_info *
 get_core_info(void)
 {

@@ -110,38 +91,45 @@ core_info_init(void)
 int
 power_manager_init(void)
 {
-	unsigned int i, num_cpus, num_freqs;
-	uint64_t cpu_mask;
+	unsigned int i, num_cpus = 0, num_freqs = 0;
 	int ret = 0;
+	struct core_info *ci;
+
+	rte_power_set_env(PM_ENV_ACPI_CPUFREQ);
 
-	num_cpus = set_host_cpus_mask();
-	if (num_cpus == 0) {
-		RTE_LOG(ERR, POWER_MANAGER, "Unable to detected host CPUs, please "
-			"ensure that sufficient privileges exist to inspect sysfs\n");
+	ci = get_core_info();
+	if (!ci) {
+		RTE_LOG(ERR, POWER_MANAGER,
+				"Failed to get core info!\n");
 		return -1;
 	}
-	rte_power_set_env(PM_ENV_ACPI_CPUFREQ);
-	cpu_mask = global_enabled_cpus;
-	for (i = 0; cpu_mask; cpu_mask &= ~(1 << i++)) {
-		if (rte_power_init(i) < 0)
-			RTE_LOG(ERR, POWER_MANAGER,
-					"Unable to initialize power manager "
-					"for core %u\n", i);
-		num_freqs = rte_power_freqs(i, global_core_freq_info[i].freqs,
+
+	for (i = 0; i < ci->core_count; i++) {
+		if (ci->cd[i].global_enabled_cpus) {
+			if (rte_power_init(i) < 0)
+				RTE_LOG(ERR, POWER_MANAGER,
+						"Unable to initialize power manager "
+						"for core %u\n", i);
+			num_cpus++;
+			num_freqs = rte_power_freqs(i,
+					global_core_freq_info[i].freqs,
 					RTE_MAX_LCORE_FREQS);
-		if (num_freqs == 0) {
-			RTE_LOG(ERR, POWER_MANAGER,
-				"Unable to get frequency list for core %u\n",
-				i);
-			global_enabled_cpus &= ~(1 << i);
-			num_cpus--;
-			ret = -1;
+			if (num_freqs == 0) {
+				RTE_LOG(ERR, POWER_MANAGER,
+					"Unable to get frequency list for core %u\n",
+					i);
+				ci->cd[i].oob_enabled = 0;
+				ret = -1;
+			}
+			global_core_freq_info[i].num_freqs = num_freqs;
+
+			rte_spinlock_init(&global_core_freq_info[i].power_sl);
 		}
-		global_core_freq_info[i].num_freqs = num_freqs;
-		rte_spinlock_init(&global_core_freq_info[i].power_sl);
+		if (ci->cd[i].oob_enabled)
+			add_core_to_monitor(i);
 	}
-	RTE_LOG(INFO, POWER_MANAGER, "Detected %u host CPUs , enabled core mask:"
-					" 0x%"PRIx64"\n", num_cpus, global_enabled_cpus);
+	RTE_LOG(INFO, POWER_MANAGER, "Managing %u cores out of %u available host cores\n",
+			num_cpus, ci->core_count);
 	return ret;
 
 }

@@ -156,7 +144,7 @@ power_manager_get_current_frequency(unsigned core_num)
 				core_num, POWER_MGR_MAX_CPUS-1);
 		return -1;
 	}
-	if (!(global_enabled_cpus & (1ULL << core_num)))
+	if (!(ci.cd[core_num].global_enabled_cpus))
 		return 0;
 
 	rte_spinlock_lock(&global_core_freq_info[core_num].power_sl);

@@ -175,15 +163,26 @@ power_manager_exit(void)
 {
 	unsigned int i;
 	int ret = 0;
+	struct core_info *ci;
 
-	for (i = 0; global_enabled_cpus; global_enabled_cpus &= ~(1 << i++)) {
-		if (rte_power_exit(i) < 0) {
-			RTE_LOG(ERR, POWER_MANAGER, "Unable to shutdown power manager "
-					"for core %u\n", i);
-			ret = -1;
+	ci = get_core_info();
+	if (!ci) {
+		RTE_LOG(ERR, POWER_MANAGER,
+				"Failed to get core info!\n");
+		return -1;
+	}
+
+	for (i = 0; i < ci->core_count; i++) {
+		if (ci->cd[i].global_enabled_cpus) {
+			if (rte_power_exit(i) < 0) {
+				RTE_LOG(ERR, POWER_MANAGER, "Unable to shutdown power manager "
+						"for core %u\n", i);
+				ret = -1;
+			}
+			ci->cd[i].global_enabled_cpus = 0;
 		}
+		remove_core_from_monitor(i);
 	}
-	global_enabled_cpus = 0;
 	return ret;
 }

@@ -299,10 +298,12 @@ int
 power_manager_scale_core_med(unsigned int core_num)
 {
 	int ret = 0;
+	struct core_info *ci;
 
+	ci = get_core_info();
 	if (core_num >= POWER_MGR_MAX_CPUS)
 		return -1;
-	if (!(global_enabled_cpus & (1ULL << core_num)))
+	if (!(ci->cd[core_num].global_enabled_cpus))
 		return -1;
 	rte_spinlock_lock(&global_core_freq_info[core_num].power_sl);
 	ret = rte_power_set_freq(core_num,

Signature · 1 line

2.17.1

David Hunt

author

2018-06-26 09:23 UTC #23 lore raw

PATCH v3 5/9 examples/vm_power: add thread for oob core monitor

In reply to David Hunt

Change the app to now require three cores, as the third core will be used to run the oob montoring thread.

Signed-off-by David Hunt <[email protected]>

Acked-by Radu Nicolau <[email protected]>

---
 examples/vm_power_manager/main.c | 37 +++++++++++++++++++++++++++++---
 1 file changed, 34 insertions(+), 3 deletions(-)

examples/vm_power_manager/main.c +34 -3

index cc2a1289c..4c6b5a990 100644
@@ -29,6 +29,7 @@
 #include "channel_monitor.h"
 #include "power_manager.h"
 #include "vm_power_cli.h"
+#include "oob_monitor.h"
 #include "parse.h"
 #include <rte_pmd_ixgbe.h>
 #include <rte_pmd_i40e.h>

@@ -269,6 +270,17 @@ run_monitor(__attribute__((unused)) void *arg)
 	return 0;
 }
 
+static int
+run_core_monitor(__attribute__((unused)) void *arg)
+{
+	if (branch_monitor_init() < 0) {
+		printf("Unable to initialize core monitor\n");
+		return -1;
+	}
+	run_branch_monitor();
+	return 0;
+}
+
 static void
 sig_handler(int signo)
 {

@@ -287,12 +299,15 @@ main(int argc, char **argv)
 	unsigned int nb_ports;
 	struct rte_mempool *mbuf_pool;
 	uint16_t portid;
+	struct core_info *ci;
 
 
 	ret = core_info_init();
 	if (ret < 0)
 		rte_panic("Cannot allocate core info\n");
 
+	ci = get_core_info();
+
 	ret = rte_eal_init(argc, argv);
 	if (ret < 0)
 		rte_panic("Cannot init EAL\n");

@@ -367,16 +382,23 @@ main(int argc, char **argv)
 		}
 	}
 
+	check_all_ports_link_status(enabled_port_mask);
+
 	lcore_id = rte_get_next_lcore(-1, 1, 0);
 	if (lcore_id == RTE_MAX_LCORE) {
-		RTE_LOG(ERR, EAL, "A minimum of two cores are required to run "
+		RTE_LOG(ERR, EAL, "A minimum of three cores are required to run "
 				"application\n");
 		return 0;
 	}
-
-	check_all_ports_link_status(enabled_port_mask);
+	printf("Running channel monitor on lcore id %d\n", lcore_id);
 	rte_eal_remote_launch(run_monitor, NULL, lcore_id);
 
+	lcore_id = rte_get_next_lcore(lcore_id, 1, 0);
+	if (lcore_id == RTE_MAX_LCORE) {
+		RTE_LOG(ERR, EAL, "A minimum of three cores are required to run "
+				"application\n");
+		return 0;
+	}
 	if (power_manager_init() < 0) {
 		printf("Unable to initialize power manager\n");
 		return -1;

@@ -385,8 +407,17 @@ main(int argc, char **argv)
 		printf("Unable to initialize channel manager\n");
 		return -1;
 	}
+
+	printf("Running core monitor on lcore id %d\n", lcore_id);
+	rte_eal_remote_launch(run_core_monitor, NULL, lcore_id);
+
 	run_cli(NULL);
 
+	branch_monitor_exit();
+
 	rte_eal_mp_wait_lcore();
+
+	free(ci->cd);
+
 	return 0;
 }

Signature · 1 line

2.17.1

David Hunt

author

2018-06-26 09:23 UTC #24 lore raw

PATCH v3 6/9 examples/vm_power: add port-list to command line

In reply to David Hunt

add in the long form of -p, which is --port-list

Signed-off-by David Hunt <[email protected]>

Acked-by Radu Nicolau <[email protected]>

---
 examples/vm_power_manager/main.c | 1 +
 1 file changed, 1 insertion(+)

examples/vm_power_manager/main.c +1 -0

index 4c6b5a990..4088861f1 100644
@@ -147,6 +147,7 @@ parse_args(int argc, char **argv)
 		{ "mac-updating", no_argument, 0, 1},
 		{ "no-mac-updating", no_argument, 0, 0},
 		{ "core-list", optional_argument, 0, 'l'},
+		{ "port-list", optional_argument, 0, 'p'},
 		{NULL, 0, 0, 0}
 	};
 	argvopt = argv;

Signature · 1 line

2.17.1

David Hunt

author

2018-06-26 09:23 UTC #25 lore raw

PATCH v3 7/9 examples/vm_power: add branch ratio policy type

In reply to David Hunt

Add the capability for the vm_power_manager to receive a policy of type BRANCH_RATIO. This will add any vcpus in the policy to the oob monitoring thread.

Signed-off-by David Hunt <[email protected]>

Acked-by Radu Nicolau <[email protected]>

---
 examples/vm_power_manager/channel_monitor.c | 23 +++++++++++++++++++--
 lib/librte_power/channel_commands.h         |  3 ++-
 2 files changed, 23 insertions(+), 3 deletions(-)

examples/vm_power_manager/channel_monitor.c +21 -2

index 73bddd993..7fa47ba97 100644
@@ -27,6 +27,7 @@
 #include "channel_commands.h"
 #include "channel_manager.h"
 #include "power_manager.h"
+#include "oob_monitor.h"
 
 #define RTE_LOGTYPE_CHANNEL_MONITOR RTE_LOGTYPE_USER1

@@ -92,6 +93,10 @@ get_pcpu_to_control(struct policy *pol)
 	struct vm_info info;
 	int pcpu, count;
 	uint64_t mask_u64b;
+	struct core_info *ci;
+	int ret;
+
+	ci = get_core_info();
 
 	RTE_LOG(INFO, CHANNEL_MONITOR, "Looking for pcpu for %s\n",
 			pol->pkt.vm_name);

@@ -100,8 +105,22 @@ get_pcpu_to_control(struct policy *pol)
 	for (count = 0; count < pol->pkt.num_vcpu; count++) {
 		mask_u64b = info.pcpu_mask[pol->pkt.vcpu_to_control[count]];
 		for (pcpu = 0; mask_u64b; mask_u64b &= ~(1ULL << pcpu++)) {
-			if ((mask_u64b >> pcpu) & 1)
-				pol->core_share[count].pcpu = pcpu;
+			if ((mask_u64b >> pcpu) & 1) {
+				if (pol->pkt.policy_to_use == BRANCH_RATIO) {
+					ci->cd[pcpu].oob_enabled = 1;
+					ret = add_core_to_monitor(pcpu);
+					if (ret == 0)
+						printf("Monitoring pcpu %d via Branch Ratio\n",
+								pcpu);
+					else
+						printf("Failed to start OOB Monitoring pcpu %d\n",
+								pcpu);
+
+				} else {
+					pol->core_share[count].pcpu = pcpu;
+					printf("Monitoring pcpu %d\n", pcpu);
+				}
+			}
 		}
 	}
 }

lib/librte_power/channel_commands.h +2 -1

index 5e8b4ab5d..ee638eefa 100644
@@ -48,7 +48,8 @@ enum workload {HIGH, MEDIUM, LOW};
 enum policy_to_use {
 	TRAFFIC,
 	TIME,
-	WORKLOAD
+	WORKLOAD,
+	BRANCH_RATIO
 };
 
 struct traffic {

Signature · 1 line

2.17.1

David Hunt

author

2018-06-26 09:23 UTC #26 lore raw

PATCH v3 8/9 examples/vm_power: add cli args to guest app

In reply to David Hunt

Signed-off-by David Hunt <[email protected]>

Acked-by Radu Nicolau <[email protected]>

---
 examples/vm_power_manager/guest_cli/Makefile  |   2 +-
 examples/vm_power_manager/guest_cli/main.c    | 151 +++++++++++++++++-
 examples/vm_power_manager/guest_cli/parse.c   |  93 +++++++++++
 examples/vm_power_manager/guest_cli/parse.h   |  19 +++
 .../guest_cli/vm_power_cli_guest.c            | 113 +++++++------
 .../guest_cli/vm_power_cli_guest.h            |   6 +
 6 files changed, 330 insertions(+), 54 deletions(-)
 create mode 100644 examples/vm_power_manager/guest_cli/parse.c
 create mode 100644 examples/vm_power_manager/guest_cli/parse.h

examples/vm_power_manager/guest_cli/Makefile +1 -1

index d710e22d9..8b1db861e 100644
@@ -14,7 +14,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 APP = guest_vm_power_mgr
 
 # all source are stored in SRCS-y
-SRCS-y := main.c vm_power_cli_guest.c
+SRCS-y := main.c vm_power_cli_guest.c parse.c
 
 CFLAGS += -O3 -I$(RTE_SDK)/lib/librte_power/
 CFLAGS += $(WERROR_FLAGS)

examples/vm_power_manager/guest_cli/main.c +143 -8

178 lines not on this page · raw message

examples/vm_power_manager/guest_cli/parse.c new file +93 -0

93 lines not on this page · raw message

examples/vm_power_manager/guest_cli/parse.h new file +19 -0

new file mode 100644
index 000000000..c8aa0ea50
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef PARSE_H_
+#define PARSE_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+int
+parse_set(const char *, uint16_t [], unsigned int);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* PARSE_H_ */

examples/vm_power_manager/guest_cli/vm_power_cli_guest.c +68 ~~-45~~

135 lines not on this page · raw message

examples/vm_power_manager/guest_cli/vm_power_cli_guest.h +6 -0

index 75a262967..fd77f6a69 100644
@@ -11,6 +11,12 @@ extern "C" {
 
 #include "channel_commands.h"
 
+struct channel_packet *get_policy(void);
+
+int set_policy_mac(int port, int idx);
+
+void set_policy_defaults(struct channel_packet *pkt);
+
 void run_cli(__attribute__((unused)) void *arg);
 
 #ifdef __cplusplus

Signature · 1 line

2.17.1

David Hunt

author

2018-06-26 09:23 UTC #27 lore raw

PATCH v3 9/9 examples/vm_power: make branch ratio configurable

In reply to David Hunt

For different workloads and poll loops, the theshold may be different for when you want to scale up and down.

This patch allows changing of the default branch ratio by using the -b command line argument (or --branch-ratio=)

Signed-off-by David Hunt <[email protected]>

Acked-by Radu Nicolau <[email protected]>

---
 examples/vm_power_manager/main.c            | 16 +++++++++++++++-
 examples/vm_power_manager/oob_monitor_x86.c |  3 +--
 examples/vm_power_manager/power_manager.c   |  1 +
 examples/vm_power_manager/power_manager.h   |  3 +++
 4 files changed, 20 insertions(+), 3 deletions(-)

examples/vm_power_manager/main.c +15 -1

index 4088861f1..784d928bd 100644
@@ -143,17 +143,19 @@ parse_args(int argc, char **argv)
 	int option_index;
 	char *prgname = argv[0];
 	struct core_info *ci;
+	float branch_ratio;
 	static struct option lgopts[] = {
 		{ "mac-updating", no_argument, 0, 1},
 		{ "no-mac-updating", no_argument, 0, 0},
 		{ "core-list", optional_argument, 0, 'l'},
 		{ "port-list", optional_argument, 0, 'p'},
+		{ "branch-ratio", optional_argument, 0, 'b'},
 		{NULL, 0, 0, 0}
 	};
 	argvopt = argv;
 	ci = get_core_info();
 
-	while ((opt = getopt_long(argc, argvopt, "l:p:q:T:",
+	while ((opt = getopt_long(argc, argvopt, "l:p:q:T:b:",
 				  lgopts, &option_index)) != EOF) {
 
 		switch (opt) {

@@ -186,6 +188,18 @@ parse_args(int argc, char **argv)
 			}
 			free(oob_enable);
 			break;
+		case 'b':
+			branch_ratio = 0.0;
+			if (strlen(optarg))
+				branch_ratio = atof(optarg);
+			if (branch_ratio <= 0.0) {
+				printf("invalid branch ratio specified\n");
+				return -1;
+			}
+			ci->branch_ratio_threshold = branch_ratio;
+			printf("***Setting branch ratio to %f\n",
+					branch_ratio);
+			break;
 		/* long options */
 		case 0:
 			break;

examples/vm_power_manager/oob_monitor_x86.c +1 -2

index 485ec5e3f..ea327b819 100644
@@ -45,7 +45,6 @@ void branch_monitor_exit(void)
 /* Number of microseconds between each poll */
 #define INTERVAL 100
 #define PRINT_LOOP_COUNT (1000000/INTERVAL)
-#define RATIO_THRESHOLD 0.03
 #define IA32_PERFEVTSEL0 0x186
 #define IA32_PERFEVTSEL1 0x187
 #define IA32_PERFCTR0 0xc1

@@ -112,7 +111,7 @@ apply_policy(int core)
 
 	ratio = (float)miss_diff * (float)100 / (float)hits_diff;
 
-	if (ratio < RATIO_THRESHOLD)
+	if (ratio < ci->branch_ratio_threshold)
 		power_manager_scale_core_min(core);
 	else
 		power_manager_scale_core_max(core);

examples/vm_power_manager/power_manager.c +1 -0

index 4bdde23da..b7769c3c3 100644
@@ -74,6 +74,7 @@ core_info_init(void)
 	ci = get_core_info();
 
 	ci->core_count = get_nprocs_conf();
+	ci->branch_ratio_threshold = BRANCH_RATIO_THRESHOLD;
 	ci->cd = malloc(ci->core_count * sizeof(struct core_details));
 	if (!ci->cd) {
 		RTE_LOG(ERR, POWER_MANAGER, "Failed to allocate memory for core info.");

examples/vm_power_manager/power_manager.h +3 -0

index 45385de37..605b3c8f6 100644
@@ -19,8 +19,11 @@ struct core_details {
 struct core_info {
 	uint16_t core_count;
 	struct core_details *cd;
+	float branch_ratio_threshold;
 };
 
+#define BRANCH_RATIO_THRESHOLD 0.1
+
 struct core_info *
 get_core_info(void);

Signature · 1 line

2.17.1

Thomas Monjalon

2018-07-12 19:09 UTC #28 lore raw

In reply to David Hunt

26/06/2018 11:23, David Hunt:

Quoted 21 lines

This patch set adds the capability to do out-of-band power monitoring on a system. It uses a thread to monitor the branch counters in the targeted cores, and calculates the branch ratio if the running code.
If the branch ratop is low (0.01), then
the code is most likely running in a tight poll loop and doing nothing, i.e. receiving no packets. In this case we scale down the frequency of that core.
If the branch ratio is higher (>0.01), then it is likely that the code is receiving and processing packets. In this case, we scale up the frequency of that core.
The cpu counters are read via /dev/cpu/x/msr, so requires the msr kernel module to be loaded. Because this method is used, the patch set is implemented with one file for x86 systems, and another for non-x86 systems, with conditional compilation in the Makefile. The non-x86 functions are stubs, and do not currently implement any functionality.

Quoted 7 lines

The vm_power_manager app has been modified to take a new parameter
--core-list or -l
which takes a list of cores in a comma-separated list format, e.g. 1,3,5-7,9, which resolvest to a core list of 1,3,5,6,7,9 These cores will then be enabled for oob monitoring. When the OOB monitoring thread starts, it reads the branch hits/miss counters of each monitored core, and scales up/down accordingly.

It looks to be a feature which could be integrated in DPDK libs. Why choosing to implement it fully in an example?

Hunt, David

author

2018-07-13 08:29 UTC #29 lore raw

Hi Thomas,

Quoted 29 lines · Thomas Monjalon

26/06/2018 11:23, David Hunt:
Quoted 21 lines
This patch set adds the capability to do out-of-band power monitoring on a system. It uses a thread to monitor the branch counters in the targeted cores, and calculates the branch ratio if the running code.
If the branch ratop is low (0.01), then
the code is most likely running in a tight poll loop and doing nothing, i.e. receiving no packets. In this case we scale down the frequency of that core.
If the branch ratio is higher (>0.01), then it is likely that the code is receiving and processing packets. In this case, we scale up the frequency of that core.
The cpu counters are read via /dev/cpu/x/msr, so requires the msr kernel module to be loaded. Because this method is used, the patch set is implemented with one file for x86 systems, and another for non-x86 systems, with conditional compilation in the Makefile. The non-x86 functions are stubs, and do not currently implement any functionality.
Quoted 7 lines
The vm_power_manager app has been modified to take a new parameter
--core-list or -l
which takes a list of cores in a comma-separated list format, e.g. 1,3,5-7,9, which resolvest to a core list of 1,3,5,6,7,9 These cores will then be enabled for oob monitoring. When the OOB monitoring thread starts, it reads the branch hits/miss counters of each monitored core, and scales up/down accordingly.

Quoted 2 lines

It looks to be a feature which could be integrated in DPDK libs. Why choosing to implement it fully in an example?

I needed to set up a thread that looped tightly (~100uS interval) and run it on it's own core. From what I have seen in other cases, it is usually the application that allocates cores and decides what to run on them. I did think about putting some of it in a library, but for this case I thought it made more sense to keep it purely as a sample app.

Regards,
Dave.

Thomas Monjalon

2018-07-13 08:33 UTC #30 lore raw

13/07/2018 10:31, Hunt, David:

Quoted 35 lines

Hi Thomas,
Quoted 29 lines · Thomas Monjalon
26/06/2018 11:23, David Hunt:
Quoted 21 lines
This patch set adds the capability to do out-of-band power monitoring on a system. It uses a thread to monitor the branch counters in the targeted cores, and calculates the branch ratio if the running code.
If the branch ratop is low (0.01), then
the code is most likely running in a tight poll loop and doing nothing, i.e. receiving no packets. In this case we scale down the frequency of that core.
If the branch ratio is higher (>0.01), then it is likely that the code is receiving and processing packets. In this case, we scale up the frequency of that core.
The cpu counters are read via /dev/cpu/x/msr, so requires the msr kernel module to be loaded. Because this method is used, the patch set is implemented with one file for x86 systems, and another for non-x86 systems, with conditional compilation in the Makefile. The non-x86 functions are stubs, and do not currently implement any functionality.
Quoted 7 lines
The vm_power_manager app has been modified to take a new parameter
--core-list or -l
which takes a list of cores in a comma-separated list format, e.g. 1,3,5-7,9, which resolvest to a core list of 1,3,5,6,7,9 These cores will then be enabled for oob monitoring. When the OOB monitoring thread starts, it reads the branch hits/miss counters of each monitored core, and scales up/down accordingly.
Quoted 2 lines
It looks to be a feature which could be integrated in DPDK libs. Why choosing to implement it fully in an example?

Quoted 8 lines

I needed to set up a thread that looped tightly (~100uS interval) and run it on it's
own core. From what I have seen in other cases, it is usually the application that
allocates cores and decides what to run on them. I did think about putting some of it in a library, but for this case I thought it made more sense to keep
it purely as a sample app.

I feel some code deserves to be in a library.
For instance, having different implementations per CPU is a good reason to make a library.

Hunt, David

author

2018-07-13 08:41 UTC #31 lore raw

Quoted 43 lines · Thomas Monjalon

13/07/2018 10:31, Hunt, David:
Quoted 34 lines
Hi Thomas,
Quoted 29 lines · Thomas Monjalon
26/06/2018 11:23, David Hunt:
Quoted 21 lines
This patch set adds the capability to do out-of-band power monitoring on a system. It uses a thread to monitor the branch counters in the targeted cores, and calculates the branch ratio if the running code.
If the branch ratop is low (0.01), then
the code is most likely running in a tight poll loop and doing nothing, i.e. receiving no packets. In this case we scale down the frequency of that core.
If the branch ratio is higher (>0.01), then it is likely that the code is receiving and processing packets. In this case, we scale up the frequency of that core.
The cpu counters are read via /dev/cpu/x/msr, so requires the msr kernel module to be loaded. Because this method is used, the patch set is implemented with one file for x86 systems, and another for non-x86 systems, with conditional compilation in the Makefile. The non-x86 functions are stubs, and do not currently implement any functionality.
Quoted 7 lines
The vm_power_manager app has been modified to take a new parameter
--core-list or -l
which takes a list of cores in a comma-separated list format, e.g. 1,3,5-7,9, which resolvest to a core list of 1,3,5,6,7,9 These cores will then be enabled for oob monitoring. When the OOB monitoring thread starts, it reads the branch hits/miss counters of each monitored core, and scales up/down accordingly.
Quoted 2 lines
It looks to be a feature which could be integrated in DPDK libs. Why choosing to implement it fully in an example?
Quoted 8 lines
I needed to set up a thread that looped tightly (~100uS interval) and run it on it's
own core. From what I have seen in other cases, it is usually the application that
allocates cores and decides what to run on them. I did think about putting some of it in a library, but for this case I thought it made more sense to keep
it purely as a sample app.

Quoted 4 lines

I feel some code deserves to be in a library.
For instance, having different implementations per CPU is a good reason to make a library.

Sure, I can look at moving some of the code into the library in a future release. However, I believe it's OK as it is for the current merge window.

Regards,
Dave.

Thomas Monjalon

2018-07-18 15:23 UTC #32 lore raw

13/07/2018 10:43, Hunt, David:

Quoted 45 lines · Thomas Monjalon

Quoted 43 lines
13/07/2018 10:31, Hunt, David:
Quoted 42 lines
Hi Thomas,
Quoted 29 lines · Thomas Monjalon
26/06/2018 11:23, David Hunt:
Quoted 21 lines
This patch set adds the capability to do out-of-band power monitoring on a system. It uses a thread to monitor the branch counters in the targeted cores, and calculates the branch ratio if the running code.
If the branch ratop is low (0.01), then
the code is most likely running in a tight poll loop and doing nothing, i.e. receiving no packets. In this case we scale down the frequency of that core.
If the branch ratio is higher (>0.01), then it is likely that the code is receiving and processing packets. In this case, we scale up the frequency of that core.
The cpu counters are read via /dev/cpu/x/msr, so requires the msr kernel module to be loaded. Because this method is used, the patch set is implemented with one file for x86 systems, and another for non-x86 systems, with conditional compilation in the Makefile. The non-x86 functions are stubs, and do not currently implement any functionality.
Quoted 7 lines
The vm_power_manager app has been modified to take a new parameter
--core-list or -l
which takes a list of cores in a comma-separated list format, e.g. 1,3,5-7,9, which resolvest to a core list of 1,3,5,6,7,9 These cores will then be enabled for oob monitoring. When the OOB monitoring thread starts, it reads the branch hits/miss counters of each monitored core, and scales up/down accordingly.
Quoted 2 lines
It looks to be a feature which could be integrated in DPDK libs. Why choosing to implement it fully in an example?
I needed to set up a thread that looped tightly (~100uS interval) and run it on it's
own core. From what I have seen in other cases, it is usually the application that
allocates cores and decides what to run on them. I did think about putting some of it in a library, but for this case I thought it made more sense to keep
it purely as a sample app.

Quoted 8 lines

Quoted 4 lines
I feel some code deserves to be in a library.
For instance, having different implementations per CPU is a good reason to make a library.
Sure, I can look at moving some of the code into the library in a future release. However, I
believe it's OK as it is for the current merge window.

I will to pull it in 18.08-rc2 if compilation is fine.

David Hunt

author

2018-06-21 13:25 UTC #33 lore raw

PATCH v2 2/8 examples/vm_power: add core list parameter

In reply to David Hunt

Also , as part of this change, we introduce a core_info struct which keeps information on each core in the system, and whether we're doing out of band monitoring on them.

Signed-off-by David Hunt <[email protected]>

---
 examples/vm_power_manager/Makefile        |  2 +-
 examples/vm_power_manager/main.c          | 34 ++++++++-
 examples/vm_power_manager/parse.c         | 93 +++++++++++++++++++++++
 examples/vm_power_manager/parse.h         | 20 +++++
 examples/vm_power_manager/power_manager.c | 31 ++++++++
 examples/vm_power_manager/power_manager.h | 20 +++++
 6 files changed, 197 insertions(+), 3 deletions(-)
 create mode 100644 examples/vm_power_manager/parse.c
 create mode 100644 examples/vm_power_manager/parse.h

examples/vm_power_manager/Makefile +1 -1

index ef2a9f959..0c925967c 100644
@@ -19,7 +19,7 @@ APP = vm_power_mgr
 
 # all source are stored in SRCS-y
 SRCS-y := main.c vm_power_cli.c power_manager.c channel_manager.c
-SRCS-y += channel_monitor.c
+SRCS-y += channel_monitor.c parse.c
 
 CFLAGS += -O3 -I$(RTE_SDK)/lib/librte_power/
 CFLAGS += $(WERROR_FLAGS)

examples/vm_power_manager/main.c +32 -2

index 043b374bc..cc2a1289c 100644
@@ -29,6 +29,7 @@
 #include "channel_monitor.h"
 #include "power_manager.h"
 #include "vm_power_cli.h"
+#include "parse.h"
 #include <rte_pmd_ixgbe.h>
 #include <rte_pmd_i40e.h>
 #include <rte_pmd_bnxt.h>

@@ -135,18 +136,22 @@ parse_portmask(const char *portmask)
 static int
 parse_args(int argc, char **argv)
 {
-	int opt, ret;
+	int opt, ret, cnt, i;
 	char **argvopt;
+	uint16_t *oob_enable;
 	int option_index;
 	char *prgname = argv[0];
+	struct core_info *ci;
 	static struct option lgopts[] = {
 		{ "mac-updating", no_argument, 0, 1},
 		{ "no-mac-updating", no_argument, 0, 0},
+		{ "core-list", optional_argument, 0, 'l'},
 		{NULL, 0, 0, 0}
 	};
 	argvopt = argv;
+	ci = get_core_info();
 
-	while ((opt = getopt_long(argc, argvopt, "p:q:T:",
+	while ((opt = getopt_long(argc, argvopt, "l:p:q:T:",
 				  lgopts, &option_index)) != EOF) {
 
 		switch (opt) {

@@ -158,6 +163,27 @@ parse_args(int argc, char **argv)
 				return -1;
 			}
 			break;
+		case 'l':
+			oob_enable = malloc(ci->core_count * sizeof(uint16_t));
+			if (oob_enable == NULL) {
+				printf("Error - Unable to allocate memory\n");
+				return -1;
+			}
+			cnt = parse_set(optarg, oob_enable, ci->core_count);
+			if (cnt < 0) {
+				printf("Invalid core-list - [%s]\n",
+						optarg);
+				break;
+			}
+			for (i = 0; i < ci->core_count; i++) {
+				if (oob_enable[i]) {
+					printf("***Using core %d\n", i);
+					ci->cd[i].oob_enabled = 1;
+					ci->cd[i].global_enabled_cpus = 1;
+				}
+			}
+			free(oob_enable);
+			break;
 		/* long options */
 		case 0:
 			break;

@@ -263,6 +289,10 @@ main(int argc, char **argv)
 	uint16_t portid;
 
 
+	ret = core_info_init();
+	if (ret < 0)
+		rte_panic("Cannot allocate core info\n");
+
 	ret = rte_eal_init(argc, argv);
 	if (ret < 0)
 		rte_panic("Cannot init EAL\n");

examples/vm_power_manager/parse.c new file +93 -0

93 lines not on this page · raw message

examples/vm_power_manager/parse.h new file +20 -0

new file mode 100644
index 000000000..a5971e9a2
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef PARSE_H_
+#define PARSE_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+int
+parse_set(const char *, uint16_t [], unsigned int);
+
+#ifdef __cplusplus
+}
+#endif
+
+
+#endif /* PARSE_H_ */

examples/vm_power_manager/power_manager.c +31 -0

index 35db25591..a7849e48a 100644
@@ -12,6 +12,7 @@
 #include <dirent.h>
 #include <errno.h>
 
+#include <sys/sysinfo.h>
 #include <sys/types.h>
 
 #include <rte_log.h>

@@ -54,6 +55,7 @@ struct freq_info {
 
 static struct freq_info global_core_freq_info[POWER_MGR_MAX_CPUS];
 
+struct core_info ci;
 static uint64_t global_enabled_cpus;
 
 #define SYSFS_CPU_PATH "/sys/devices/system/cpu/cpu%u/topology/core_id"

@@ -76,6 +78,35 @@ set_host_cpus_mask(void)
 	return num_cpus;
 }
 
+struct core_info *
+get_core_info(void)
+{
+	return &ci;
+}
+
+int
+core_info_init(void)
+{
+	struct core_info *ci;
+	int i;
+
+	ci = get_core_info();
+
+	ci->core_count = get_nprocs_conf();
+	ci->cd = malloc(ci->core_count * sizeof(struct core_details));
+	if (!ci->cd) {
+		RTE_LOG(ERR, POWER_MANAGER, "Failed to allocate memory for core info.");
+		return -1;
+	}
+	for (i = 0; i < ci->core_count; i++) {
+		ci->cd[i].global_enabled_cpus = 1;
+		ci->cd[i].oob_enabled = 0;
+		ci->cd[i].msr_fd = 0;
+	}
+	printf("%d cores in system\n", ci->core_count);
+	return 0;
+}
+
 int
 power_manager_init(void)
 {

examples/vm_power_manager/power_manager.h +20 -0

index 8a8a84aa4..45385de37 100644
@@ -8,6 +8,26 @@
 #ifdef __cplusplus
 extern "C" {
 #endif
+struct core_details {
+	uint64_t last_branches;
+	uint64_t last_branch_misses;
+	uint16_t global_enabled_cpus;
+	uint16_t oob_enabled;
+	int msr_fd;
+};
+
+struct core_info {
+	uint16_t core_count;
+	struct core_details *cd;
+};
+
+struct core_info *
+get_core_info(void);
+
+int
+core_info_init(void);
+
+#define RTE_LOGTYPE_POWER_MANAGER RTE_LOGTYPE_USER1
 
 /* Maximum number of CPUS to manage */
 #define POWER_MGR_MAX_CPUS 64

Signature · 1 line

2.17.1

David Hunt

author

2018-06-21 13:25 UTC #34 lore raw

PATCH v2 3/8 examples/vm_power: add oob monitoring functions

In reply to David Hunt

This patch introduces the out-of-band (oob) core monitoring functions.

The makefile has a switch for x86 and non-x86 platforms, and compiles stub function for non-x86 platforms.

Signed-off-by David Hunt <[email protected]>

---
 examples/vm_power_manager/Makefile          |   5 +
 examples/vm_power_manager/oob_monitor.h     |  68 +++++
 examples/vm_power_manager/oob_monitor_nop.c |  38 +++
 examples/vm_power_manager/oob_monitor_x86.c | 282 ++++++++++++++++++++
 4 files changed, 393 insertions(+)
 create mode 100644 examples/vm_power_manager/oob_monitor.h
 create mode 100644 examples/vm_power_manager/oob_monitor_nop.c
 create mode 100644 examples/vm_power_manager/oob_monitor_x86.c

examples/vm_power_manager/Makefile +5 -0

index 0c925967c..13a5205ba 100644
@@ -20,6 +20,11 @@ APP = vm_power_mgr
 # all source are stored in SRCS-y
 SRCS-y := main.c vm_power_cli.c power_manager.c channel_manager.c
 SRCS-y += channel_monitor.c parse.c
+ifeq ($(CONFIG_RTE_ARCH_X86_64),y)
+SRCS-y += oob_monitor_x86.c
+else
+SRCS-y += oob_monitor_nop.c
+endif
 
 CFLAGS += -O3 -I$(RTE_SDK)/lib/librte_power/
 CFLAGS += $(WERROR_FLAGS)

examples/vm_power_manager/oob_monitor.h new file +68 -0

68 lines not on this page · raw message

examples/vm_power_manager/oob_monitor_nop.c new file +38 -0

new file mode 100644
index 000000000..7e7b8bc14
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2014 Intel Corporation
+ */
+
+#include "oob_monitor.h"
+
+void branch_monitor_exit(void)
+{
+}
+
+__attribute__((unused)) static float
+apply_policy(__attribute__((unused)) int core)
+{
+	return 0.0;
+}
+
+int
+add_core_to_monitor(__attribute__((unused)) int core)
+{
+	return 0;
+}
+
+int
+remove_core_from_monitor(__attribute__((unused)) int core)
+{
+	return 0;
+}
+
+int
+branch_monitor_init(void)
+{
+	return 0;
+}
+
+void
+run_branch_monitor(void)
+{
+}

examples/vm_power_manager/oob_monitor_x86.c new file +282 -0

282 lines not on this page · raw message

Signature · 1 line

2.17.1

David Hunt

author

2018-06-21 13:25 UTC #35 lore raw

PATCH v2 4/8 examples/vm_power: allow greater than 64 cores

In reply to David Hunt

Signed-off-by David Hunt <[email protected]>

---
 examples/vm_power_manager/power_manager.c | 115 +++++++++++-----------
 1 file changed, 58 insertions(+), 57 deletions(-)

examples/vm_power_manager/power_manager.c +58 ~~-57~~

index a7849e48a..4bdde23da 100644
@@ -19,14 +19,14 @@
 #include <rte_power.h>
 #include <rte_spinlock.h>
 
+#include "channel_manager.h"
 #include "power_manager.h"
-
-#define RTE_LOGTYPE_POWER_MANAGER RTE_LOGTYPE_USER1
+#include "oob_monitor.h"
 
 #define POWER_SCALE_CORE(DIRECTION, core_num , ret) do { \
-	if (core_num >= POWER_MGR_MAX_CPUS) \
+	if (core_num >= ci.core_count) \
 		return -1; \
-	if (!(global_enabled_cpus & (1ULL << core_num))) \
+	if (!(ci.cd[core_num].global_enabled_cpus)) \
 		return -1; \
 	rte_spinlock_lock(&global_core_freq_info[core_num].power_sl); \
 	ret = rte_power_freq_##DIRECTION(core_num); \

@@ -37,7 +37,7 @@
 	int i; \
 	for (i = 0; core_mask; core_mask &= ~(1 << i++)) { \
 		if ((core_mask >> i) & 1) { \
-			if (!(global_enabled_cpus & (1ULL << i))) \
+			if (!(ci.cd[i].global_enabled_cpus)) \
 				continue; \
 			rte_spinlock_lock(&global_core_freq_info[i].power_sl); \
 			if (rte_power_freq_##DIRECTION(i) != 1) \

@@ -56,28 +56,9 @@ struct freq_info {
 static struct freq_info global_core_freq_info[POWER_MGR_MAX_CPUS];
 
 struct core_info ci;
-static uint64_t global_enabled_cpus;
 
 #define SYSFS_CPU_PATH "/sys/devices/system/cpu/cpu%u/topology/core_id"
 
-static unsigned
-set_host_cpus_mask(void)
-{
-	char path[PATH_MAX];
-	unsigned i;
-	unsigned num_cpus = 0;
-
-	for (i = 0; i < POWER_MGR_MAX_CPUS; i++) {
-		snprintf(path, sizeof(path), SYSFS_CPU_PATH, i);
-		if (access(path, F_OK) == 0) {
-			global_enabled_cpus |= 1ULL << i;
-			num_cpus++;
-		} else
-			return num_cpus;
-	}
-	return num_cpus;
-}
-
 struct core_info *
 get_core_info(void)
 {

@@ -110,38 +91,45 @@ core_info_init(void)
 int
 power_manager_init(void)
 {
-	unsigned int i, num_cpus, num_freqs;
-	uint64_t cpu_mask;
+	unsigned int i, num_cpus = 0, num_freqs = 0;
 	int ret = 0;
+	struct core_info *ci;
+
+	rte_power_set_env(PM_ENV_ACPI_CPUFREQ);
 
-	num_cpus = set_host_cpus_mask();
-	if (num_cpus == 0) {
-		RTE_LOG(ERR, POWER_MANAGER, "Unable to detected host CPUs, please "
-			"ensure that sufficient privileges exist to inspect sysfs\n");
+	ci = get_core_info();
+	if (!ci) {
+		RTE_LOG(ERR, POWER_MANAGER,
+				"Failed to get core info!\n");
 		return -1;
 	}
-	rte_power_set_env(PM_ENV_ACPI_CPUFREQ);
-	cpu_mask = global_enabled_cpus;
-	for (i = 0; cpu_mask; cpu_mask &= ~(1 << i++)) {
-		if (rte_power_init(i) < 0)
-			RTE_LOG(ERR, POWER_MANAGER,
-					"Unable to initialize power manager "
-					"for core %u\n", i);
-		num_freqs = rte_power_freqs(i, global_core_freq_info[i].freqs,
+
+	for (i = 0; i < ci->core_count; i++) {
+		if (ci->cd[i].global_enabled_cpus) {
+			if (rte_power_init(i) < 0)
+				RTE_LOG(ERR, POWER_MANAGER,
+						"Unable to initialize power manager "
+						"for core %u\n", i);
+			num_cpus++;
+			num_freqs = rte_power_freqs(i,
+					global_core_freq_info[i].freqs,
 					RTE_MAX_LCORE_FREQS);
-		if (num_freqs == 0) {
-			RTE_LOG(ERR, POWER_MANAGER,
-				"Unable to get frequency list for core %u\n",
-				i);
-			global_enabled_cpus &= ~(1 << i);
-			num_cpus--;
-			ret = -1;
+			if (num_freqs == 0) {
+				RTE_LOG(ERR, POWER_MANAGER,
+					"Unable to get frequency list for core %u\n",
+					i);
+				ci->cd[i].oob_enabled = 0;
+				ret = -1;
+			}
+			global_core_freq_info[i].num_freqs = num_freqs;
+
+			rte_spinlock_init(&global_core_freq_info[i].power_sl);
 		}
-		global_core_freq_info[i].num_freqs = num_freqs;
-		rte_spinlock_init(&global_core_freq_info[i].power_sl);
+		if (ci->cd[i].oob_enabled)
+			add_core_to_monitor(i);
 	}
-	RTE_LOG(INFO, POWER_MANAGER, "Detected %u host CPUs , enabled core mask:"
-					" 0x%"PRIx64"\n", num_cpus, global_enabled_cpus);
+	RTE_LOG(INFO, POWER_MANAGER, "Managing %u cores out of %u available host cores\n",
+			num_cpus, ci->core_count);
 	return ret;
 
 }

@@ -156,7 +144,7 @@ power_manager_get_current_frequency(unsigned core_num)
 				core_num, POWER_MGR_MAX_CPUS-1);
 		return -1;
 	}
-	if (!(global_enabled_cpus & (1ULL << core_num)))
+	if (!(ci.cd[core_num].global_enabled_cpus))
 		return 0;
 
 	rte_spinlock_lock(&global_core_freq_info[core_num].power_sl);

@@ -175,15 +163,26 @@ power_manager_exit(void)
 {
 	unsigned int i;
 	int ret = 0;
+	struct core_info *ci;
 
-	for (i = 0; global_enabled_cpus; global_enabled_cpus &= ~(1 << i++)) {
-		if (rte_power_exit(i) < 0) {
-			RTE_LOG(ERR, POWER_MANAGER, "Unable to shutdown power manager "
-					"for core %u\n", i);
-			ret = -1;
+	ci = get_core_info();
+	if (!ci) {
+		RTE_LOG(ERR, POWER_MANAGER,
+				"Failed to get core info!\n");
+		return -1;
+	}
+
+	for (i = 0; i < ci->core_count; i++) {
+		if (ci->cd[i].global_enabled_cpus) {
+			if (rte_power_exit(i) < 0) {
+				RTE_LOG(ERR, POWER_MANAGER, "Unable to shutdown power manager "
+						"for core %u\n", i);
+				ret = -1;
+			}
+			ci->cd[i].global_enabled_cpus = 0;
 		}
+		remove_core_from_monitor(i);
 	}
-	global_enabled_cpus = 0;
 	return ret;
 }

@@ -299,10 +298,12 @@ int
 power_manager_scale_core_med(unsigned int core_num)
 {
 	int ret = 0;
+	struct core_info *ci;
 
+	ci = get_core_info();
 	if (core_num >= POWER_MGR_MAX_CPUS)
 		return -1;
-	if (!(global_enabled_cpus & (1ULL << core_num)))
+	if (!(ci->cd[core_num].global_enabled_cpus))
 		return -1;
 	rte_spinlock_lock(&global_core_freq_info[core_num].power_sl);
 	ret = rte_power_set_freq(core_num,

Signature · 1 line

2.17.1

David Hunt

author

2018-06-21 13:25 UTC #36 lore raw

PATCH v2 5/8 examples/vm_power: add thread for oob core monitor

In reply to David Hunt

Change the app to now require three cores, as the third core will be used to run the oob montoring thread.

Signed-off-by David Hunt <[email protected]>

---
 examples/vm_power_manager/main.c | 37 +++++++++++++++++++++++++++++---
 1 file changed, 34 insertions(+), 3 deletions(-)

examples/vm_power_manager/main.c +34 -3

index cc2a1289c..4c6b5a990 100644
@@ -29,6 +29,7 @@
 #include "channel_monitor.h"
 #include "power_manager.h"
 #include "vm_power_cli.h"
+#include "oob_monitor.h"
 #include "parse.h"
 #include <rte_pmd_ixgbe.h>
 #include <rte_pmd_i40e.h>

@@ -269,6 +270,17 @@ run_monitor(__attribute__((unused)) void *arg)
 	return 0;
 }
 
+static int
+run_core_monitor(__attribute__((unused)) void *arg)
+{
+	if (branch_monitor_init() < 0) {
+		printf("Unable to initialize core monitor\n");
+		return -1;
+	}
+	run_branch_monitor();
+	return 0;
+}
+
 static void
 sig_handler(int signo)
 {

@@ -287,12 +299,15 @@ main(int argc, char **argv)
 	unsigned int nb_ports;
 	struct rte_mempool *mbuf_pool;
 	uint16_t portid;
+	struct core_info *ci;
 
 
 	ret = core_info_init();
 	if (ret < 0)
 		rte_panic("Cannot allocate core info\n");
 
+	ci = get_core_info();
+
 	ret = rte_eal_init(argc, argv);
 	if (ret < 0)
 		rte_panic("Cannot init EAL\n");

@@ -367,16 +382,23 @@ main(int argc, char **argv)
 		}
 	}
 
+	check_all_ports_link_status(enabled_port_mask);
+
 	lcore_id = rte_get_next_lcore(-1, 1, 0);
 	if (lcore_id == RTE_MAX_LCORE) {
-		RTE_LOG(ERR, EAL, "A minimum of two cores are required to run "
+		RTE_LOG(ERR, EAL, "A minimum of three cores are required to run "
 				"application\n");
 		return 0;
 	}
-
-	check_all_ports_link_status(enabled_port_mask);
+	printf("Running channel monitor on lcore id %d\n", lcore_id);
 	rte_eal_remote_launch(run_monitor, NULL, lcore_id);
 
+	lcore_id = rte_get_next_lcore(lcore_id, 1, 0);
+	if (lcore_id == RTE_MAX_LCORE) {
+		RTE_LOG(ERR, EAL, "A minimum of three cores are required to run "
+				"application\n");
+		return 0;
+	}
 	if (power_manager_init() < 0) {
 		printf("Unable to initialize power manager\n");
 		return -1;

@@ -385,8 +407,17 @@ main(int argc, char **argv)
 		printf("Unable to initialize channel manager\n");
 		return -1;
 	}
+
+	printf("Running core monitor on lcore id %d\n", lcore_id);
+	rte_eal_remote_launch(run_core_monitor, NULL, lcore_id);
+
 	run_cli(NULL);
 
+	branch_monitor_exit();
+
 	rte_eal_mp_wait_lcore();
+
+	free(ci->cd);
+
 	return 0;
 }

Signature · 1 line

2.17.1

David Hunt

author

2018-06-21 13:25 UTC #37 lore raw

PATCH v2 6/8 examples/vm_power: add port-list to command line

In reply to David Hunt

add in the long form of -p, which is --port-list

Signed-off-by David Hunt <[email protected]>

---
 examples/vm_power_manager/main.c | 1 +
 1 file changed, 1 insertion(+)

examples/vm_power_manager/main.c +1 -0

index 4c6b5a990..4088861f1 100644
@@ -147,6 +147,7 @@ parse_args(int argc, char **argv)
 		{ "mac-updating", no_argument, 0, 1},
 		{ "no-mac-updating", no_argument, 0, 0},
 		{ "core-list", optional_argument, 0, 'l'},
+		{ "port-list", optional_argument, 0, 'p'},
 		{NULL, 0, 0, 0}
 	};
 	argvopt = argv;

Signature · 1 line

2.17.1

David Hunt

author

2018-06-21 13:25 UTC #38 lore raw

PATCH v2 7/8 examples/vm_power: add branch ratio policy type

In reply to David Hunt

Add the capability for the vm_power_manager to receive a policy of type BRANCH_RATIO. This will add any vcpus in the policy to the oob monitoring thread.

Signed-off-by David Hunt <[email protected]>

---
 examples/vm_power_manager/channel_monitor.c | 23 +++++++++++++++++++--
 lib/librte_power/channel_commands.h         |  3 ++-
 2 files changed, 23 insertions(+), 3 deletions(-)

examples/vm_power_manager/channel_monitor.c +21 -2

index 73bddd993..7fa47ba97 100644
@@ -27,6 +27,7 @@
 #include "channel_commands.h"
 #include "channel_manager.h"
 #include "power_manager.h"
+#include "oob_monitor.h"
 
 #define RTE_LOGTYPE_CHANNEL_MONITOR RTE_LOGTYPE_USER1

@@ -92,6 +93,10 @@ get_pcpu_to_control(struct policy *pol)
 	struct vm_info info;
 	int pcpu, count;
 	uint64_t mask_u64b;
+	struct core_info *ci;
+	int ret;
+
+	ci = get_core_info();
 
 	RTE_LOG(INFO, CHANNEL_MONITOR, "Looking for pcpu for %s\n",
 			pol->pkt.vm_name);

@@ -100,8 +105,22 @@ get_pcpu_to_control(struct policy *pol)
 	for (count = 0; count < pol->pkt.num_vcpu; count++) {
 		mask_u64b = info.pcpu_mask[pol->pkt.vcpu_to_control[count]];
 		for (pcpu = 0; mask_u64b; mask_u64b &= ~(1ULL << pcpu++)) {
-			if ((mask_u64b >> pcpu) & 1)
-				pol->core_share[count].pcpu = pcpu;
+			if ((mask_u64b >> pcpu) & 1) {
+				if (pol->pkt.policy_to_use == BRANCH_RATIO) {
+					ci->cd[pcpu].oob_enabled = 1;
+					ret = add_core_to_monitor(pcpu);
+					if (ret == 0)
+						printf("Monitoring pcpu %d via Branch Ratio\n",
+								pcpu);
+					else
+						printf("Failed to start OOB Monitoring pcpu %d\n",
+								pcpu);
+
+				} else {
+					pol->core_share[count].pcpu = pcpu;
+					printf("Monitoring pcpu %d\n", pcpu);
+				}
+			}
 		}
 	}
 }

lib/librte_power/channel_commands.h +2 -1

index 5e8b4ab5d..ee638eefa 100644
@@ -48,7 +48,8 @@ enum workload {HIGH, MEDIUM, LOW};
 enum policy_to_use {
 	TRAFFIC,
 	TIME,
-	WORKLOAD
+	WORKLOAD,
+	BRANCH_RATIO
 };
 
 struct traffic {

Signature · 1 line

2.17.1

David Hunt

author

2018-06-21 13:25 UTC #39 lore raw

PATCH v2 8/8 examples/vm_power: add cli args to guest app

In reply to David Hunt

Signed-off-by David Hunt <[email protected]>

---
 examples/vm_power_manager/guest_cli/Makefile  |   2 +-
 examples/vm_power_manager/guest_cli/main.c    | 151 +++++++++++++++++-
 examples/vm_power_manager/guest_cli/parse.c   |  93 +++++++++++
 examples/vm_power_manager/guest_cli/parse.h   |  19 +++
 .../guest_cli/vm_power_cli_guest.c            | 113 +++++++------
 .../guest_cli/vm_power_cli_guest.h            |   6 +
 6 files changed, 330 insertions(+), 54 deletions(-)
 create mode 100644 examples/vm_power_manager/guest_cli/parse.c
 create mode 100644 examples/vm_power_manager/guest_cli/parse.h

examples/vm_power_manager/guest_cli/Makefile +1 -1

index d710e22d9..8b1db861e 100644
@@ -14,7 +14,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 APP = guest_vm_power_mgr
 
 # all source are stored in SRCS-y
-SRCS-y := main.c vm_power_cli_guest.c
+SRCS-y := main.c vm_power_cli_guest.c parse.c
 
 CFLAGS += -O3 -I$(RTE_SDK)/lib/librte_power/
 CFLAGS += $(WERROR_FLAGS)

examples/vm_power_manager/guest_cli/main.c +143 -8

178 lines not on this page · raw message

examples/vm_power_manager/guest_cli/parse.c new file +93 -0

93 lines not on this page · raw message

examples/vm_power_manager/guest_cli/parse.h new file +19 -0

new file mode 100644
index 000000000..c8aa0ea50
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef PARSE_H_
+#define PARSE_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+int
+parse_set(const char *, uint16_t [], unsigned int);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* PARSE_H_ */

examples/vm_power_manager/guest_cli/vm_power_cli_guest.c +68 ~~-45~~

135 lines not on this page · raw message

examples/vm_power_manager/guest_cli/vm_power_cli_guest.h +6 -0

index 75a262967..fd77f6a69 100644
@@ -11,6 +11,12 @@ extern "C" {
 
 #include "channel_commands.h"
 
+struct channel_packet *get_policy(void);
+
+int set_policy_mac(int port, int idx);
+
+void set_policy_defaults(struct channel_packet *pkt);
+
 void run_cli(__attribute__((unused)) void *arg);
 
 #ifdef __cplusplus

Signature · 1 line

2.17.1

Radu Nicolau

2018-06-21 14:29 UTC #40 lore raw

In reply to David Hunt

Quoted 40 lines · David Hunt

This patch set adds the capability to do out-of-band power monitoring on a system. It uses a thread to monitor the branch counters in the targeted cores, and calculates the branch ratio if the running code.
If the branch ratop is low (0.01), then
the code is most likely running in a tight poll loop and doing nothing, i.e. receiving no packets. In this case we scale down the frequency of that core.
If the branch ratio is higher (>0.01), then it is likely that the code is receiving and processing packets. In this case, we scale up the frequency of that core.
The cpu counters are read via /dev/cpu/x/msr, so requires the msr kernel module to be loaded. Because this method is used, the patch set is implemented with one file for x86 systems, and another for non-x86 systems, with conditional compilation in the Makefile. The non-x86 functions are stubs, and do not currently implement any functionality.
The vm_power_manager app has been modified to take a new parameter
--core-list or -l
which takes a list of cores in a comma-separated list format, e.g. 1,3,5-7,9, which resolvest to a core list of 1,3,5,6,7,9 These cores will then be enabled for oob monitoring. When the OOB monitoring thread starts, it reads the branch hits/miss counters of each monitored core, and scales up/down accordingly.
The guest_cli app has also been modified to allow sending of a policy of type BRANCH_RATIO where all of the cores included in the policy will be monitored by the vm_power_manager oob thread.
v2 changes:
* Add the guest_cli patch into this patch set, including the ability to set the policy to BRANCH_RATIO.
http://patches.dpdk.org/patch/40742/
* When vm_power_manger receives a policy with type BRANCH_RATIO, add the relevant cores to the monitoring thread.

Quoted 9 lines

[1/8] examples/vm_power: add check for port count [2/8] examples/vm_power: add core list parameter [3/8] examples/vm_power: add oob monitoring functions [4/8] examples/vm_power: allow greater than 64 cores [5/8] examples/vm_power: add thread for oob core monitor [6/8] examples/vm_power: add port-list to command line [7/8] examples/vm_power: add branch ratio policy type [8/8] examples/vm_power: add cli args to guest app

Series Acked-by: Radu Nicolau <[email protected]>

David Hunt

author

2018-06-07 14:39 UTC #41 lore raw

PATCH 2/6 examples/vm_power: add core list parameter

In reply to David Hunt

Also , as part of this change, we introduce a core_info struct which keeps information on each core in the system, and whether we're doing out of band monitoring on them.

Signed-off-by David Hunt <[email protected]>

---
 examples/vm_power_manager/Makefile        |  2 +-
 examples/vm_power_manager/main.c          | 34 ++++++++-
 examples/vm_power_manager/parse.c         | 93 +++++++++++++++++++++++
 examples/vm_power_manager/parse.h         | 20 +++++
 examples/vm_power_manager/power_manager.c | 31 ++++++++
 examples/vm_power_manager/power_manager.h | 20 +++++
 6 files changed, 197 insertions(+), 3 deletions(-)
 create mode 100644 examples/vm_power_manager/parse.c
 create mode 100644 examples/vm_power_manager/parse.h

examples/vm_power_manager/Makefile +1 -1

index ef2a9f959..0c925967c 100644
@@ -19,7 +19,7 @@ APP = vm_power_mgr
 
 # all source are stored in SRCS-y
 SRCS-y := main.c vm_power_cli.c power_manager.c channel_manager.c
-SRCS-y += channel_monitor.c
+SRCS-y += channel_monitor.c parse.c
 
 CFLAGS += -O3 -I$(RTE_SDK)/lib/librte_power/
 CFLAGS += $(WERROR_FLAGS)

examples/vm_power_manager/main.c +32 -2

index 043b374bc..cc2a1289c 100644
@@ -29,6 +29,7 @@
 #include "channel_monitor.h"
 #include "power_manager.h"
 #include "vm_power_cli.h"
+#include "parse.h"
 #include <rte_pmd_ixgbe.h>
 #include <rte_pmd_i40e.h>
 #include <rte_pmd_bnxt.h>

@@ -135,18 +136,22 @@ parse_portmask(const char *portmask)
 static int
 parse_args(int argc, char **argv)
 {
-	int opt, ret;
+	int opt, ret, cnt, i;
 	char **argvopt;
+	uint16_t *oob_enable;
 	int option_index;
 	char *prgname = argv[0];
+	struct core_info *ci;
 	static struct option lgopts[] = {
 		{ "mac-updating", no_argument, 0, 1},
 		{ "no-mac-updating", no_argument, 0, 0},
+		{ "core-list", optional_argument, 0, 'l'},
 		{NULL, 0, 0, 0}
 	};
 	argvopt = argv;
+	ci = get_core_info();
 
-	while ((opt = getopt_long(argc, argvopt, "p:q:T:",
+	while ((opt = getopt_long(argc, argvopt, "l:p:q:T:",
 				  lgopts, &option_index)) != EOF) {
 
 		switch (opt) {

@@ -158,6 +163,27 @@ parse_args(int argc, char **argv)
 				return -1;
 			}
 			break;
+		case 'l':
+			oob_enable = malloc(ci->core_count * sizeof(uint16_t));
+			if (oob_enable == NULL) {
+				printf("Error - Unable to allocate memory\n");
+				return -1;
+			}
+			cnt = parse_set(optarg, oob_enable, ci->core_count);
+			if (cnt < 0) {
+				printf("Invalid core-list - [%s]\n",
+						optarg);
+				break;
+			}
+			for (i = 0; i < ci->core_count; i++) {
+				if (oob_enable[i]) {
+					printf("***Using core %d\n", i);
+					ci->cd[i].oob_enabled = 1;
+					ci->cd[i].global_enabled_cpus = 1;
+				}
+			}
+			free(oob_enable);
+			break;
 		/* long options */
 		case 0:
 			break;

@@ -263,6 +289,10 @@ main(int argc, char **argv)
 	uint16_t portid;
 
 
+	ret = core_info_init();
+	if (ret < 0)
+		rte_panic("Cannot allocate core info\n");
+
 	ret = rte_eal_init(argc, argv);
 	if (ret < 0)
 		rte_panic("Cannot init EAL\n");

examples/vm_power_manager/parse.c new file +93 -0

93 lines not on this page · raw message

examples/vm_power_manager/parse.h new file +20 -0

new file mode 100644
index 000000000..a5971e9a2
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef PARSE_H_
+#define PARSE_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+int
+parse_set(const char *, uint16_t [], unsigned int);
+
+#ifdef __cplusplus
+}
+#endif
+
+
+#endif /* PARSE_H_ */

examples/vm_power_manager/power_manager.c +31 -0

index 35db25591..a7849e48a 100644
@@ -12,6 +12,7 @@
 #include <dirent.h>
 #include <errno.h>
 
+#include <sys/sysinfo.h>
 #include <sys/types.h>
 
 #include <rte_log.h>

@@ -54,6 +55,7 @@ struct freq_info {
 
 static struct freq_info global_core_freq_info[POWER_MGR_MAX_CPUS];
 
+struct core_info ci;
 static uint64_t global_enabled_cpus;
 
 #define SYSFS_CPU_PATH "/sys/devices/system/cpu/cpu%u/topology/core_id"

@@ -76,6 +78,35 @@ set_host_cpus_mask(void)
 	return num_cpus;
 }
 
+struct core_info *
+get_core_info(void)
+{
+	return &ci;
+}
+
+int
+core_info_init(void)
+{
+	struct core_info *ci;
+	int i;
+
+	ci = get_core_info();
+
+	ci->core_count = get_nprocs_conf();
+	ci->cd = malloc(ci->core_count * sizeof(struct core_details));
+	if (!ci->cd) {
+		RTE_LOG(ERR, POWER_MANAGER, "Failed to allocate memory for core info.");
+		return -1;
+	}
+	for (i = 0; i < ci->core_count; i++) {
+		ci->cd[i].global_enabled_cpus = 1;
+		ci->cd[i].oob_enabled = 0;
+		ci->cd[i].msr_fd = 0;
+	}
+	printf("%d cores in system\n", ci->core_count);
+	return 0;
+}
+
 int
 power_manager_init(void)
 {

examples/vm_power_manager/power_manager.h +20 -0

index 8a8a84aa4..45385de37 100644
@@ -8,6 +8,26 @@
 #ifdef __cplusplus
 extern "C" {
 #endif
+struct core_details {
+	uint64_t last_branches;
+	uint64_t last_branch_misses;
+	uint16_t global_enabled_cpus;
+	uint16_t oob_enabled;
+	int msr_fd;
+};
+
+struct core_info {
+	uint16_t core_count;
+	struct core_details *cd;
+};
+
+struct core_info *
+get_core_info(void);
+
+int
+core_info_init(void);
+
+#define RTE_LOGTYPE_POWER_MANAGER RTE_LOGTYPE_USER1
 
 /* Maximum number of CPUS to manage */
 #define POWER_MGR_MAX_CPUS 64

Signature · 1 line

2.17.0

David Hunt

author

2018-06-07 14:39 UTC #42 lore raw

PATCH 3/6 examples/vm_power: add oob monitoring functions

In reply to David Hunt

This patch introduces the out-of-band (oob) core monitoring functions.

The makefile has a switch for x86 and non-x86 platforms, and compiles stub function for non-x86 platforms.

Signed-off-by David Hunt <[email protected]>

---
 examples/vm_power_manager/Makefile          |   5 +
 examples/vm_power_manager/oob_monitor.h     |  68 +++++
 examples/vm_power_manager/oob_monitor_nop.c |  38 +++
 examples/vm_power_manager/oob_monitor_x86.c | 282 ++++++++++++++++++++
 4 files changed, 393 insertions(+)
 create mode 100644 examples/vm_power_manager/oob_monitor.h
 create mode 100644 examples/vm_power_manager/oob_monitor_nop.c
 create mode 100644 examples/vm_power_manager/oob_monitor_x86.c

examples/vm_power_manager/Makefile +5 -0

index 0c925967c..13a5205ba 100644
@@ -20,6 +20,11 @@ APP = vm_power_mgr
 # all source are stored in SRCS-y
 SRCS-y := main.c vm_power_cli.c power_manager.c channel_manager.c
 SRCS-y += channel_monitor.c parse.c
+ifeq ($(CONFIG_RTE_ARCH_X86_64),y)
+SRCS-y += oob_monitor_x86.c
+else
+SRCS-y += oob_monitor_nop.c
+endif
 
 CFLAGS += -O3 -I$(RTE_SDK)/lib/librte_power/
 CFLAGS += $(WERROR_FLAGS)

examples/vm_power_manager/oob_monitor.h new file +68 -0

68 lines not on this page · raw message

examples/vm_power_manager/oob_monitor_nop.c new file +38 -0

new file mode 100644
index 000000000..7e7b8bc14
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2014 Intel Corporation
+ */
+
+#include "oob_monitor.h"
+
+void branch_monitor_exit(void)
+{
+}
+
+__attribute__((unused)) static float
+apply_policy(__attribute__((unused)) int core)
+{
+	return 0.0;
+}
+
+int
+add_core_to_monitor(__attribute__((unused)) int core)
+{
+	return 0;
+}
+
+int
+remove_core_from_monitor(__attribute__((unused)) int core)
+{
+	return 0;
+}
+
+int
+branch_monitor_init(void)
+{
+	return 0;
+}
+
+void
+run_branch_monitor(void)
+{
+}

examples/vm_power_manager/oob_monitor_x86.c new file +282 -0

282 lines not on this page · raw message

Signature · 1 line

2.17.0

David Hunt

author

2018-06-07 14:39 UTC #43 lore raw

PATCH 4/6 examples/vm_power: allow greater than 64 cores

In reply to David Hunt

Signed-off-by David Hunt <[email protected]>

---
 examples/vm_power_manager/power_manager.c | 115 +++++++++++-----------
 1 file changed, 58 insertions(+), 57 deletions(-)

examples/vm_power_manager/power_manager.c +58 ~~-57~~

index a7849e48a..4bdde23da 100644
@@ -19,14 +19,14 @@
 #include <rte_power.h>
 #include <rte_spinlock.h>
 
+#include "channel_manager.h"
 #include "power_manager.h"
-
-#define RTE_LOGTYPE_POWER_MANAGER RTE_LOGTYPE_USER1
+#include "oob_monitor.h"
 
 #define POWER_SCALE_CORE(DIRECTION, core_num , ret) do { \
-	if (core_num >= POWER_MGR_MAX_CPUS) \
+	if (core_num >= ci.core_count) \
 		return -1; \
-	if (!(global_enabled_cpus & (1ULL << core_num))) \
+	if (!(ci.cd[core_num].global_enabled_cpus)) \
 		return -1; \
 	rte_spinlock_lock(&global_core_freq_info[core_num].power_sl); \
 	ret = rte_power_freq_##DIRECTION(core_num); \

@@ -37,7 +37,7 @@
 	int i; \
 	for (i = 0; core_mask; core_mask &= ~(1 << i++)) { \
 		if ((core_mask >> i) & 1) { \
-			if (!(global_enabled_cpus & (1ULL << i))) \
+			if (!(ci.cd[i].global_enabled_cpus)) \
 				continue; \
 			rte_spinlock_lock(&global_core_freq_info[i].power_sl); \
 			if (rte_power_freq_##DIRECTION(i) != 1) \

@@ -56,28 +56,9 @@ struct freq_info {
 static struct freq_info global_core_freq_info[POWER_MGR_MAX_CPUS];
 
 struct core_info ci;
-static uint64_t global_enabled_cpus;
 
 #define SYSFS_CPU_PATH "/sys/devices/system/cpu/cpu%u/topology/core_id"
 
-static unsigned
-set_host_cpus_mask(void)
-{
-	char path[PATH_MAX];
-	unsigned i;
-	unsigned num_cpus = 0;
-
-	for (i = 0; i < POWER_MGR_MAX_CPUS; i++) {
-		snprintf(path, sizeof(path), SYSFS_CPU_PATH, i);
-		if (access(path, F_OK) == 0) {
-			global_enabled_cpus |= 1ULL << i;
-			num_cpus++;
-		} else
-			return num_cpus;
-	}
-	return num_cpus;
-}
-
 struct core_info *
 get_core_info(void)
 {

@@ -110,38 +91,45 @@ core_info_init(void)
 int
 power_manager_init(void)
 {
-	unsigned int i, num_cpus, num_freqs;
-	uint64_t cpu_mask;
+	unsigned int i, num_cpus = 0, num_freqs = 0;
 	int ret = 0;
+	struct core_info *ci;
+
+	rte_power_set_env(PM_ENV_ACPI_CPUFREQ);
 
-	num_cpus = set_host_cpus_mask();
-	if (num_cpus == 0) {
-		RTE_LOG(ERR, POWER_MANAGER, "Unable to detected host CPUs, please "
-			"ensure that sufficient privileges exist to inspect sysfs\n");
+	ci = get_core_info();
+	if (!ci) {
+		RTE_LOG(ERR, POWER_MANAGER,
+				"Failed to get core info!\n");
 		return -1;
 	}
-	rte_power_set_env(PM_ENV_ACPI_CPUFREQ);
-	cpu_mask = global_enabled_cpus;
-	for (i = 0; cpu_mask; cpu_mask &= ~(1 << i++)) {
-		if (rte_power_init(i) < 0)
-			RTE_LOG(ERR, POWER_MANAGER,
-					"Unable to initialize power manager "
-					"for core %u\n", i);
-		num_freqs = rte_power_freqs(i, global_core_freq_info[i].freqs,
+
+	for (i = 0; i < ci->core_count; i++) {
+		if (ci->cd[i].global_enabled_cpus) {
+			if (rte_power_init(i) < 0)
+				RTE_LOG(ERR, POWER_MANAGER,
+						"Unable to initialize power manager "
+						"for core %u\n", i);
+			num_cpus++;
+			num_freqs = rte_power_freqs(i,
+					global_core_freq_info[i].freqs,
 					RTE_MAX_LCORE_FREQS);
-		if (num_freqs == 0) {
-			RTE_LOG(ERR, POWER_MANAGER,
-				"Unable to get frequency list for core %u\n",
-				i);
-			global_enabled_cpus &= ~(1 << i);
-			num_cpus--;
-			ret = -1;
+			if (num_freqs == 0) {
+				RTE_LOG(ERR, POWER_MANAGER,
+					"Unable to get frequency list for core %u\n",
+					i);
+				ci->cd[i].oob_enabled = 0;
+				ret = -1;
+			}
+			global_core_freq_info[i].num_freqs = num_freqs;
+
+			rte_spinlock_init(&global_core_freq_info[i].power_sl);
 		}
-		global_core_freq_info[i].num_freqs = num_freqs;
-		rte_spinlock_init(&global_core_freq_info[i].power_sl);
+		if (ci->cd[i].oob_enabled)
+			add_core_to_monitor(i);
 	}
-	RTE_LOG(INFO, POWER_MANAGER, "Detected %u host CPUs , enabled core mask:"
-					" 0x%"PRIx64"\n", num_cpus, global_enabled_cpus);
+	RTE_LOG(INFO, POWER_MANAGER, "Managing %u cores out of %u available host cores\n",
+			num_cpus, ci->core_count);
 	return ret;
 
 }

@@ -156,7 +144,7 @@ power_manager_get_current_frequency(unsigned core_num)
 				core_num, POWER_MGR_MAX_CPUS-1);
 		return -1;
 	}
-	if (!(global_enabled_cpus & (1ULL << core_num)))
+	if (!(ci.cd[core_num].global_enabled_cpus))
 		return 0;
 
 	rte_spinlock_lock(&global_core_freq_info[core_num].power_sl);

@@ -175,15 +163,26 @@ power_manager_exit(void)
 {
 	unsigned int i;
 	int ret = 0;
+	struct core_info *ci;
 
-	for (i = 0; global_enabled_cpus; global_enabled_cpus &= ~(1 << i++)) {
-		if (rte_power_exit(i) < 0) {
-			RTE_LOG(ERR, POWER_MANAGER, "Unable to shutdown power manager "
-					"for core %u\n", i);
-			ret = -1;
+	ci = get_core_info();
+	if (!ci) {
+		RTE_LOG(ERR, POWER_MANAGER,
+				"Failed to get core info!\n");
+		return -1;
+	}
+
+	for (i = 0; i < ci->core_count; i++) {
+		if (ci->cd[i].global_enabled_cpus) {
+			if (rte_power_exit(i) < 0) {
+				RTE_LOG(ERR, POWER_MANAGER, "Unable to shutdown power manager "
+						"for core %u\n", i);
+				ret = -1;
+			}
+			ci->cd[i].global_enabled_cpus = 0;
 		}
+		remove_core_from_monitor(i);
 	}
-	global_enabled_cpus = 0;
 	return ret;
 }

@@ -299,10 +298,12 @@ int
 power_manager_scale_core_med(unsigned int core_num)
 {
 	int ret = 0;
+	struct core_info *ci;
 
+	ci = get_core_info();
 	if (core_num >= POWER_MGR_MAX_CPUS)
 		return -1;
-	if (!(global_enabled_cpus & (1ULL << core_num)))
+	if (!(ci->cd[core_num].global_enabled_cpus))
 		return -1;
 	rte_spinlock_lock(&global_core_freq_info[core_num].power_sl);
 	ret = rte_power_set_freq(core_num,

Signature · 1 line

2.17.0

David Hunt

author

2018-06-07 14:39 UTC #44 lore raw

PATCH 5/6 examples/vm_power: add thread for oob core monitor

In reply to David Hunt

Change the app to now require three cores, as the third core will be used to run the oob montoring thread.

Signed-off-by David Hunt <[email protected]>

---
 examples/vm_power_manager/main.c | 37 +++++++++++++++++++++++++++++---
 1 file changed, 34 insertions(+), 3 deletions(-)

examples/vm_power_manager/main.c +34 -3

index cc2a1289c..4c6b5a990 100644
@@ -29,6 +29,7 @@
 #include "channel_monitor.h"
 #include "power_manager.h"
 #include "vm_power_cli.h"
+#include "oob_monitor.h"
 #include "parse.h"
 #include <rte_pmd_ixgbe.h>
 #include <rte_pmd_i40e.h>

@@ -269,6 +270,17 @@ run_monitor(__attribute__((unused)) void *arg)
 	return 0;
 }
 
+static int
+run_core_monitor(__attribute__((unused)) void *arg)
+{
+	if (branch_monitor_init() < 0) {
+		printf("Unable to initialize core monitor\n");
+		return -1;
+	}
+	run_branch_monitor();
+	return 0;
+}
+
 static void
 sig_handler(int signo)
 {

@@ -287,12 +299,15 @@ main(int argc, char **argv)
 	unsigned int nb_ports;
 	struct rte_mempool *mbuf_pool;
 	uint16_t portid;
+	struct core_info *ci;
 
 
 	ret = core_info_init();
 	if (ret < 0)
 		rte_panic("Cannot allocate core info\n");
 
+	ci = get_core_info();
+
 	ret = rte_eal_init(argc, argv);
 	if (ret < 0)
 		rte_panic("Cannot init EAL\n");

@@ -367,16 +382,23 @@ main(int argc, char **argv)
 		}
 	}
 
+	check_all_ports_link_status(enabled_port_mask);
+
 	lcore_id = rte_get_next_lcore(-1, 1, 0);
 	if (lcore_id == RTE_MAX_LCORE) {
-		RTE_LOG(ERR, EAL, "A minimum of two cores are required to run "
+		RTE_LOG(ERR, EAL, "A minimum of three cores are required to run "
 				"application\n");
 		return 0;
 	}
-
-	check_all_ports_link_status(enabled_port_mask);
+	printf("Running channel monitor on lcore id %d\n", lcore_id);
 	rte_eal_remote_launch(run_monitor, NULL, lcore_id);
 
+	lcore_id = rte_get_next_lcore(lcore_id, 1, 0);
+	if (lcore_id == RTE_MAX_LCORE) {
+		RTE_LOG(ERR, EAL, "A minimum of three cores are required to run "
+				"application\n");
+		return 0;
+	}
 	if (power_manager_init() < 0) {
 		printf("Unable to initialize power manager\n");
 		return -1;

@@ -385,8 +407,17 @@ main(int argc, char **argv)
 		printf("Unable to initialize channel manager\n");
 		return -1;
 	}
+
+	printf("Running core monitor on lcore id %d\n", lcore_id);
+	rte_eal_remote_launch(run_core_monitor, NULL, lcore_id);
+
 	run_cli(NULL);
 
+	branch_monitor_exit();
+
 	rte_eal_mp_wait_lcore();
+
+	free(ci->cd);
+
 	return 0;
 }

Signature · 1 line

2.17.0

David Hunt

author

2018-06-07 14:39 UTC #45 lore raw

PATCH 6/6 examples/vm_power: add port-list to command line

In reply to David Hunt

add in the long form of -p, which is --port-list

Signed-off-by David Hunt <[email protected]>

---
 examples/vm_power_manager/main.c | 1 +
 1 file changed, 1 insertion(+)

examples/vm_power_manager/main.c +1 -0

index 4c6b5a990..4088861f1 100644
@@ -147,6 +147,7 @@ parse_args(int argc, char **argv)
 		{ "mac-updating", no_argument, 0, 1},
 		{ "no-mac-updating", no_argument, 0, 0},
 		{ "core-list", optional_argument, 0, 'l'},
+		{ "port-list", optional_argument, 0, 'p'},
 		{NULL, 0, 0, 0}
 	};
 	argvopt = argv;

Signature · 1 line

2.17.0