Skip to contentSkip to conversation

COLO net and runstate bugfix/optimization

PATCH Superseded Zhang Chen 2 participants 13 messages Mar 9, 2022 – Apr 1, 2022 Networking

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Zhang Chen

author
#0 lore raw

cover COLO net and runstate bugfix/optimization

This series fix some COLO related issues in internal stress testing.

Zhang Chen (4):
softmmu/runstate.c: add RunStateTransition support form COLO to
PRELAUNCH
net/colo: Fix a "double free" crash to clear the conn_list net/colo.c: No need to track conn_list for filter-rewriter net/colo.c: fix segmentation fault when packet is not parsed correctly

 net/colo-compare.c    |  2 +-
 net/colo.c            | 11 +++++++++--
 net/filter-rewriter.c |  2 +-
 net/trace-events      |  1 +
 softmmu/runstate.c    |  1 +
 5 files changed, 13 insertions(+), 4 deletions(-)
Signature · 1 line
2.25.1
author
#1 lore raw

PATCH 1/4 softmmu/runstate.c: add RunStateTransition support form COLO to PRELAUNCH

If the checkpoint occurs when the guest finishes restarting but has not started running, the runstate_set() may reject the transition from COLO to PRELAUNCH with the crash log:

{"timestamp": {"seconds": 1593484591, "microseconds": 26605},\
"event": "RESET", "data": {"guest": true, "reason": "guest-reset"}}
qemu-system-x86_64: invalid runstate transition: 'colo' -> 'prelaunch'

Long-term testing says that it's pretty safe.

---
 softmmu/runstate.c | 1 +
 1 file changed, 1 insertion(+)
softmmu/runstate.c +1 -0
index e0d869b21a..c021c56338 100644
@@ -127,6 +127,7 @@ static const RunStateTransition runstate_transitions_def[] = {
     { RUN_STATE_RESTORE_VM, RUN_STATE_PRELAUNCH },
 
     { RUN_STATE_COLO, RUN_STATE_RUNNING },
+    { RUN_STATE_COLO, RUN_STATE_PRELAUNCH },
     { RUN_STATE_COLO, RUN_STATE_SHUTDOWN},
 
     { RUN_STATE_RUNNING, RUN_STATE_DEBUG },
Signature · 1 line
2.25.1
author
#2 lore raw

PATCH 2/4 net/colo: Fix a "double free" crash to clear the conn_list

In reply to Zhang Chen

We notice the QEMU may crash when the guest has too many incoming network connections with the following log:

[email protected]:colo_proxy_main : colo proxy connection hashtable full, clear it free(): invalid pointer
[1] 15195 abort (core dumped) qemu-system-x86_64 ....

This is because we create the s->connection_track_table with g_hash_table_new_full() which is defined as:

GHashTable * g_hash_table_new_full (GHashFunc hash_func,

                       GEqualFunc key_equal_func,
                       GDestroyNotify key_destroy_func,
                       GDestroyNotify value_destroy_func);

The fourth parameter connection_destroy() will be called to free the memory allocated for all 'Connection' values in the hashtable when we call g_hash_table_remove_all() in the connection_hashtable_reset().

It's unnecessary because we clear the conn_list explicitly later, and it's buggy when other agents try to call connection_get() with the same connection_track_table.

---
 net/colo-compare.c    | 2 +-
 net/filter-rewriter.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
net/colo-compare.c +1 -1
index 62554b5b3c..ab054cfd21 100644
@@ -1324,7 +1324,7 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp)
     s->connection_track_table = g_hash_table_new_full(connection_key_hash,
                                                       connection_key_equal,
                                                       g_free,
-                                                      connection_destroy);
+                                                      NULL);
 
     colo_compare_iothread(s);
 
net/filter-rewriter.c +1 -1
index bf05023dc3..c18c4c2019 100644
@@ -383,7 +383,7 @@ static void colo_rewriter_setup(NetFilterState *nf, Error **errp)
     s->connection_track_table = g_hash_table_new_full(connection_key_hash,
                                                       connection_key_equal,
                                                       g_free,
-                                                      connection_destroy);
+                                                      NULL);
     s->incoming_queue = qemu_new_net_queue(qemu_netfilter_pass_to_next, nf);
 }
 
Signature · 1 line
2.25.1
#3 lore raw

Re: [PATCH 2/4] net/colo: Fix a "double free" crash to clear the conn_list

Quoted 35 lines · Zhang Chen

We notice the QEMU may crash when the guest has too many incoming network connections with the following log:

[email protected]:colo_proxy_main : colo proxy connection hashtable full, clear it free(): invalid pointer
[1] 15195 abort (core dumped) qemu-system-x86_64 ....

This is because we create the s->connection_track_table with g_hash_table_new_full() which is defined as:

GHashTable * g_hash_table_new_full (GHashFunc hash_func,

                        GEqualFunc key_equal_func,
                        GDestroyNotify key_destroy_func,
                        GDestroyNotify value_destroy_func);

The fourth parameter connection_destroy() will be called to free the memory allocated for all 'Connection' values in the hashtable when we call g_hash_table_remove_all() in the connection_hashtable_reset().

It's unnecessary because we clear the conn_list explicitly later, and it's buggy when other agents try to call connection_get() with the same connection_track_table.

---
  net/colo-compare.c    | 2 +-
  net/filter-rewriter.c | 2 +-
  2 files changed, 2 insertions(+), 2 deletions(-)
net/colo-compare.c +0 -0
index 62554b5b3c..ab054cfd21 100644

net/colo-compare.c static void colo_compare_complete(UserCreatable *uc, Error **errp) · line 1327

Quoted 5 lines
net/colo-compare.c +1 -1
@@ -1324,7 +1324,7 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp)
      s->connection_track_table = g_hash_table_new_full(connection_key_hash,
                                                        connection_key_equal,
                                                        g_free,
-                                                      connection_destroy);
+                                                      NULL);

202 /* if not found, create a new connection and add to hash table */ 203 Connection *connection_get(GHashTable *connection_track_table, 204                            ConnectionKey *key,
205                            GQueue *conn_list)
206 {
207     Connection *conn = g_hash_table_lookup(connection_track_table, key); 208
209     if (conn == NULL) {
210         ConnectionKey *new_key = g_memdup(key, sizeof(*key)); 211
212         conn = connection_new(key);
213
214         if (g_hash_table_size(connection_track_table) > HASHTABLE_MAX_SIZE) { 215             trace_colo_proxy_main("colo proxy connection hashtable full," 216                                   " clear it");
217 connection_hashtable_reset(connection_track_table);

197 void connection_hashtable_reset(GHashTable *connection_track_table) 198 {
199 g_hash_table_remove_all(connection_track_table);
200 }

IIUC,  above subroutine will do some cleanup explicitly. And before your patch, connection_hashtable_reset() will release all keys and their values in this hashtable. But now, you remove all keys and just one value(conn_list) instead. Does it means other values will be leaked ?

218 /*
219              * clear the conn_list
220 */
221             while (!g_queue_is_empty(conn_list)) {
222 connection_destroy(g_queue_pop_head(conn_list));
223 }
224 }
225
226         g_hash_table_insert(connection_track_table, new_key, conn); 227 }
228
229     return conn;
230 }

Thanks
Zhijian

Quoted 16 lines
net/colo-compare.c +0 -0
      colo_compare_iothread(s);
 
net/filter-rewriter.c +1 -1
index bf05023dc3..c18c4c2019 100644
@@ -383,7 +383,7 @@ static void colo_rewriter_setup(NetFilterState *nf, Error **errp)
      s->connection_track_table = g_hash_table_new_full(connection_key_hash,
                                                        connection_key_equal,
                                                        g_free,
-                                                      connection_destroy);
+                                                      NULL);
      s->incoming_queue = qemu_new_net_queue(qemu_netfilter_pass_to_next, nf);
  }
 
#4 lore raw

RE: [PATCH 2/4] net/colo: Fix a "double free" crash to clear the conn_list

Quoted 83 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Monday, March 21, 2022 11:06 AM
To: Zhang, Chen <[email protected]>; Jason Wang <[email protected]>; [email protected]
Cc: qemu-dev <[email protected]>; Like Xu <[email protected]>
Subject: Re: [PATCH 2/4] net/colo: Fix a "double free" crash to clear the conn_list

net/colo-compare.c static void

Quoted 36 lines · Zhang Chen

We notice the QEMU may crash when the guest has too many incoming network connections with the following log:

[email protected]:colo_proxy_main : colo proxy connection hashtable full, clear it
free(): invalid pointer
[1] 15195 abort (core dumped) qemu-system-x86_64 ....

This is because we create the s->connection_track_table with g_hash_table_new_full() which is defined as:

GHashTable * g_hash_table_new_full (GHashFunc hash_func,

                        GEqualFunc key_equal_func,
                        GDestroyNotify key_destroy_func,
                        GDestroyNotify value_destroy_func);

The fourth parameter connection_destroy() will be called to free the memory allocated for all 'Connection' values in the hashtable when we call g_hash_table_remove_all() in the connection_hashtable_reset().

It's unnecessary because we clear the conn_list explicitly later, and it's buggy when other agents try to call connection_get() with the same connection_track_table.

---
  net/colo-compare.c    | 2 +-
  net/filter-rewriter.c | 2 +-
  2 files changed, 2 insertions(+), 2 deletions(-)
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

62554b5b3c..ab054cfd21 100644

net/colo-compare.c +0 -0
@@ -1324,7 +1324,7 @@ static void

colo_compare_complete(UserCreatable *uc, Error **errp)

net/colo-compare.c static void · line 1324

Quoted 1 line
net/colo-compare.c +0 -0
      s->connection_track_table =

g_hash_table_new_full(connection_key_hash,

net/colo-compare.c static void colo_compare_complete(UserCreatable *uc, Error **errp) · line 1327

Quoted 4 lines
net/colo-compare.c +1 -1
                                                        connection_key_equal,
                                                        g_free,
-                                                      connection_destroy);
+                                                      NULL);

202 /* if not found, create a new connection and add to hash table */ 203 Connection *connection_get(GHashTable *connection_track_table, 204                            ConnectionKey *key,
205                            GQueue *conn_list)
206 {
207     Connection *conn = g_hash_table_lookup(connection_track_table, key);
208
209     if (conn == NULL) {
210         ConnectionKey *new_key = g_memdup(key, sizeof(*key)); 211
212         conn = connection_new(key);
213
214         if (g_hash_table_size(connection_track_table) > HASHTABLE_MAX_SIZE) {
215             trace_colo_proxy_main("colo proxy connection hashtable full," 216                                   " clear it");
217 connection_hashtable_reset(connection_track_table);

197 void connection_hashtable_reset(GHashTable *connection_track_table) 198 {
199 g_hash_table_remove_all(connection_track_table);
200 }

Quoted 4 lines

IIUC,  above subroutine will do some cleanup explicitly. And before your patch, connection_hashtable_reset() will release all keys and their values in this hashtable. But now, you remove all keys and just one value(conn_list) instead. Does it means other values will be leaked ?

Thanks Zhijian. Re-think about it. Yes, I think you are right.
It looks need a lock to avoid into connection_get() when triggered the clear hashtable operation. What do you think?

Thanks
Chen

Quoted 39 lines

218 /*
219              * clear the conn_list
220 */
221             while (!g_queue_is_empty(conn_list)) {
222 connection_destroy(g_queue_pop_head(conn_list));
223 }
224 }
225
226         g_hash_table_insert(connection_track_table, new_key, conn); 227 }
228
229     return conn;
230 }

Thanks
Zhijian

net/filter-rewriter.c static void colo_rewriter_setup(NetFilterState *nf,

Quoted 8 lines
net/colo-compare.c +0 -0
      colo_compare_iothread(s);
 
net/filter-rewriter.c index renamed from net/filter-rewriter.c +0 -0

bf05023dc3..c18c4c2019 100644

net/filter-rewriter.c +0 -0
@@ -383,7 +383,7 @@ static void colo_rewriter_setup(NetFilterState *nf,

Error **errp)

net/filter-rewriter.c static void colo_rewriter_setup(NetFilterState *nf, · line 383

Quoted 1 line
net/filter-rewriter.c +0 -0
      s->connection_track_table =

g_hash_table_new_full(connection_key_hash,

net/filter-rewriter.c static void colo_rewriter_setup(NetFilterState *nf, · line 387

Quoted 5 lines
net/filter-rewriter.c +1 -1
                                                        connection_key_equal,
                                                        g_free,
-                                                      connection_destroy);
+                                                      NULL);
      s->incoming_queue =

qemu_new_net_queue(qemu_netfilter_pass_to_next, nf);

Quoted 2 lines
net/filter-rewriter.c +0 -0
  }
 
#5 lore raw

Re: [PATCH 2/4] net/colo: Fix a "double free" crash to clear the conn_list

connection_track_table
-----+----------
key1 | conn |-----------------------------------------------------------+

-----+----------                                                           |
key2 | conn    |----------------------------------+                        |
-----+----------                                  |                        |
key3 | conn    |---------+                        |                        |
-----+----------         |                        |                        |
                          |                        |                        |
                          |                        |                        |
     + CompareState ++    |                        |                        |
     |               |    V                        V                        V
     +---------------+   +---------------+         +---------------+
     |conn_list      +--->conn           +--------->conn           |     connx
     +---------------+   +---------------+         +---------------+
     |               |     |           |             |          |
     +---------------+ +---v----+  +---v----+    +---v----+ +---v----+
                       |primary |  |secondary    |primary | |secondary
                       |packet  |  |packet  +    |packet  | |packet  +
                       +--------+  +--------+    +--------+ +--------+
                           |           |             |          |
                       +---v----+  +---v----+    +---v----+ +---v----+
                       |primary |  |secondary    |primary | |secondary
                       |packet  |  |packet  +    |packet  | |packet  +
                       +--------+  +--------+    +--------+ +--------+
                           |           |             |          |
                       +---v----+  +---v----+    +---v----+ +---v----+
                       |primary |  |secondary    |primary | |secondary
                       |packet  |  |packet  +    |packet  | |packet  +
                       +--------+  +--------+    +--------+ +--------+

I recalled that we should above relationships between connection_track_table conn_list and conn. That means both connection_track_table and conn_list reference to the same conn instance.

So before this patch, connection_get() is possible to use-after-free/double free conn. where 1st was in connection_hashtable_reset() and 2nd was
221             while (!g_queue_is_empty(conn_list)) {
222 connection_destroy(g_queue_pop_head(conn_list));
223 }

I also doubt that your current abort was just due to above use-after-free/double free. If so, looks it's enough we just update to g_queue_clear(conn_list) in the 2nd place.

Thanks
Zhijian

Quoted 130 lines · Chen
Quoted 81 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Monday, March 21, 2022 11:06 AM
To: Zhang, Chen <[email protected]>; Jason Wang <[email protected]>; [email protected]
Cc: qemu-dev <[email protected]>; Like Xu <[email protected]>
Subject: Re: [PATCH 2/4] net/colo: Fix a "double free" crash to clear the conn_list

net/colo-compare.c static void

Quoted 36 lines · Zhang Chen

We notice the QEMU may crash when the guest has too many incoming network connections with the following log:

[email protected]:colo_proxy_main : colo proxy connection hashtable full, clear it
free(): invalid pointer
[1] 15195 abort (core dumped) qemu-system-x86_64 ....

This is because we create the s->connection_track_table with g_hash_table_new_full() which is defined as:

GHashTable * g_hash_table_new_full (GHashFunc hash_func,

                         GEqualFunc key_equal_func,
                         GDestroyNotify key_destroy_func,
                         GDestroyNotify value_destroy_func);

The fourth parameter connection_destroy() will be called to free the memory allocated for all 'Connection' values in the hashtable when we call g_hash_table_remove_all() in the connection_hashtable_reset().

It's unnecessary because we clear the conn_list explicitly later, and it's buggy when other agents try to call connection_get() with the same connection_track_table.

---
   net/colo-compare.c    | 2 +-
   net/filter-rewriter.c | 2 +-
   2 files changed, 2 insertions(+), 2 deletions(-)
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

62554b5b3c..ab054cfd21 100644

net/colo-compare.c +0 -0
@@ -1324,7 +1324,7 @@ static void

colo_compare_complete(UserCreatable *uc, Error **errp)

net/colo-compare.c static void · line 1324

Quoted 1 line
net/colo-compare.c +0 -0
       s->connection_track_table =

g_hash_table_new_full(connection_key_hash,

net/colo-compare.c static void colo_compare_complete(UserCreatable *uc, Error **errp) · line 1327

Quoted 4 lines
net/colo-compare.c +1 -1
                                                         connection_key_equal,
                                                         g_free,
-                                                      connection_destroy);
+                                                      NULL);

202 /* if not found, create a new connection and add to hash table */ 203 Connection *connection_get(GHashTable *connection_track_table, 204                            ConnectionKey *key,
205                            GQueue *conn_list)
206 {
207     Connection *conn = g_hash_table_lookup(connection_track_table, key);
208
209     if (conn == NULL) {
210         ConnectionKey *new_key = g_memdup(key, sizeof(*key)); 211
212         conn = connection_new(key);
213
214         if (g_hash_table_size(connection_track_table) > HASHTABLE_MAX_SIZE) {
215             trace_colo_proxy_main("colo proxy connection hashtable full," 216                                   " clear it");
217 connection_hashtable_reset(connection_track_table);

197 void connection_hashtable_reset(GHashTable *connection_track_table) 198 {
199 g_hash_table_remove_all(connection_track_table);
200 }

Quoted 4 lines

IIUC,  above subroutine will do some cleanup explicitly. And before your patch, connection_hashtable_reset() will release all keys and their values in this hashtable. But now, you remove all keys and just one value(conn_list) instead. Does it means other values will be leaked ?

Thanks Zhijian. Re-think about it. Yes, I think you are right.
It looks need a lock to avoid into connection_get() when triggered the clear hashtable operation. What do you think?

Thanks
Chen

Quoted 37 lines

218 /*
219              * clear the conn_list
220 */
221             while (!g_queue_is_empty(conn_list)) {
222 connection_destroy(g_queue_pop_head(conn_list));
223 }
224 }
225
226         g_hash_table_insert(connection_track_table, new_key, conn); 227 }
228
229     return conn;
230 }

Thanks
Zhijian

net/filter-rewriter.c static void colo_rewriter_setup(NetFilterState *nf,

Quoted 7 lines
net/colo-compare.c +0 -0
       colo_compare_iothread(s);
 
net/filter-rewriter.c index renamed from net/filter-rewriter.c +0 -0

bf05023dc3..c18c4c2019 100644

net/filter-rewriter.c +0 -0
@@ -383,7 +383,7 @@ static void colo_rewriter_setup(NetFilterState *nf,

Error **errp)

net/filter-rewriter.c static void colo_rewriter_setup(NetFilterState *nf, · line 383

Quoted 1 line
net/filter-rewriter.c +0 -0
       s->connection_track_table =

g_hash_table_new_full(connection_key_hash,

net/filter-rewriter.c static void colo_rewriter_setup(NetFilterState *nf, · line 387

Quoted 5 lines
net/filter-rewriter.c +1 -1
                                                         connection_key_equal,
                                                         g_free,
-                                                      connection_destroy);
+                                                      NULL);
       s->incoming_queue =

qemu_new_net_queue(qemu_netfilter_pass_to_next, nf);

Quoted 2 lines
net/filter-rewriter.c +0 -0
   }
 
#6 lore raw

RE: [PATCH 2/4] net/colo: Fix a "double free" crash to clear the conn_list

Quoted 55 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Thursday, March 31, 2022 9:15 AM
To: Zhang, Chen <[email protected]>; Jason Wang <[email protected]>
Cc: qemu-dev <[email protected]>; Like Xu <[email protected]>
Subject: Re: [PATCH 2/4] net/colo: Fix a "double free" crash to clear the conn_list

connection_track_table
-----+----------
key1 | conn |-----------------------------------------------------------+

-----+----------                                                           |
key2 | conn    |----------------------------------+                        |
-----+----------                                  |                        |
key3 | conn    |---------+                        |                        |
-----+----------         |                        |                        |
                          |                        |                        |
                          |                        |                        |
     + CompareState ++    |                        |                        |
     |               |    V                        V                        V
     +---------------+   +---------------+         +---------------+
     |conn_list      +--->conn           +--------->conn           |     connx
     +---------------+   +---------------+         +---------------+
     |               |     |           |             |          |
     +---------------+ +---v----+  +---v----+    +---v----+ +---v----+
                       |primary |  |secondary    |primary | |secondary
                       |packet  |  |packet  +    |packet  | |packet  +
                       +--------+  +--------+    +--------+ +--------+
                           |           |             |          |
                       +---v----+  +---v----+    +---v----+ +---v----+
                       |primary |  |secondary    |primary | |secondary
                       |packet  |  |packet  +    |packet  | |packet  +
                       +--------+  +--------+    +--------+ +--------+
                           |           |             |          |
                       +---v----+  +---v----+    +---v----+ +---v----+
                       |primary |  |secondary    |primary | |secondary
                       |packet  |  |packet  +    |packet  | |packet  +
                       +--------+  +--------+    +--------+ +--------+

I recalled that we should above relationships between connection_track_table conn_list and conn.
That means both connection_track_table and conn_list reference to the same conn instance.

So before this patch, connection_get() is possible to use-after-free/double free conn. where 1st was in
connection_hashtable_reset() and 2nd was
221             while (!g_queue_is_empty(conn_list)) {
222 connection_destroy(g_queue_pop_head(conn_list));
223 }

Quoted 4 lines

I also doubt that your current abort was just due to above use-after- free/double free.
If so, looks it's enough we just update to g_queue_clear(conn_list) in the 2nd place.

Make sense, but It also means the original patch works here, skip free conn in connection_hashtable_reset() and do it in:
221 while (!g_queue_is_empty(conn_list)) {
222 connection_destroy(g_queue_pop_head(conn_list));
223 }.
It also avoid use-after-free/double free conn.
Maybe we can keep the original version to fix it?

Thanks
Chen

Quoted 142 lines

Thanks
Zhijian

Quoted 84 lines · Chen
Quoted 84 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Monday, March 21, 2022 11:06 AM
To: Zhang, Chen <[email protected]>; Jason Wang <[email protected]>; [email protected]
Cc: qemu-dev <[email protected]>; Like Xu <[email protected]>
Subject: Re: [PATCH 2/4] net/colo: Fix a "double free" crash to clear the conn_list

net/colo-compare.c static void

Quoted 36 lines · Zhang Chen

We notice the QEMU may crash when the guest has too many incoming network connections with the following log:

[email protected]:colo_proxy_main : colo proxy connection hashtable full, clear it
free(): invalid pointer
[1] 15195 abort (core dumped) qemu-system-x86_64 ....

This is because we create the s->connection_track_table with g_hash_table_new_full() which is defined as:

GHashTable * g_hash_table_new_full (GHashFunc hash_func,

                         GEqualFunc key_equal_func,
                         GDestroyNotify key_destroy_func,
                         GDestroyNotify value_destroy_func);

The fourth parameter connection_destroy() will be called to free the memory allocated for all 'Connection' values in the hashtable when we call g_hash_table_remove_all() in the connection_hashtable_reset().

It's unnecessary because we clear the conn_list explicitly later, and it's buggy when other agents try to call connection_get() with the same connection_track_table.

---
   net/colo-compare.c    | 2 +-
   net/filter-rewriter.c | 2 +-
   2 files changed, 2 insertions(+), 2 deletions(-)
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

62554b5b3c..ab054cfd21 100644

net/colo-compare.c +0 -0
@@ -1324,7 +1324,7 @@ static void

colo_compare_complete(UserCreatable *uc, Error **errp)

net/colo-compare.c static void · line 1324

Quoted 1 line
net/colo-compare.c +0 -0
       s->connection_track_table =

g_hash_table_new_full(connection_key_hash,

net/colo-compare.c static void colo_compare_complete(UserCreatable *uc, Error **errp) · line 1327

Quoted 4 lines
net/colo-compare.c +1 -1
                                                         connection_key_equal,
                                                         g_free,
-                                                      connection_destroy);
+                                                      NULL);

202 /* if not found, create a new connection and add to hash table */ 203 Connection *connection_get(GHashTable *connection_track_table, 204                            ConnectionKey *key,
205                            GQueue *conn_list)
206 {
207     Connection *conn = g_hash_table_lookup(connection_track_table,
key);
208
209     if (conn == NULL) {
210         ConnectionKey *new_key = g_memdup(key, sizeof(*key)); 211
212         conn = connection_new(key);
213
214         if (g_hash_table_size(connection_track_table) > HASHTABLE_MAX_SIZE) {
215             trace_colo_proxy_main("colo proxy connection hashtable full," 216                                   " clear it");
217 connection_hashtable_reset(connection_track_table);

197 void connection_hashtable_reset(GHashTable *connection_track_table)
198 {
199 g_hash_table_remove_all(connection_track_table); 200 }

Quoted 6 lines
Quoted 4 lines

IIUC,  above subroutine will do some cleanup explicitly. And before your patch, connection_hashtable_reset() will release all keys and their values in this hashtable. But now, you remove all keys and just one value(conn_list) instead. Does it means other values will be leaked ?

Thanks Zhijian. Re-think about it. Yes, I think you are right. It looks need a lock to avoid into connection_get() when triggered the clear

hashtable operation.

Quoted 45 lines

What do you think?

Thanks
Chen

Quoted 39 lines

218 /*
219              * clear the conn_list
220 */
221             while (!g_queue_is_empty(conn_list)) { 222 connection_destroy(g_queue_pop_head(conn_list)); 223 }
224 }
225
226         g_hash_table_insert(connection_track_table, new_key, conn);
227 }
228
229     return conn;
230 }

Thanks
Zhijian

net/filter-rewriter.c static void colo_rewriter_setup(NetFilterState · line 383

Quoted 8 lines
net/colo-compare.c +0 -0
       colo_compare_iothread(s);
 
net/filter-rewriter.c index renamed from net/filter-rewriter.c +0 -0

bf05023dc3..c18c4c2019 100644

net/filter-rewriter.c +0 -0
@@ -383,7 +383,7 @@ static void colo_rewriter_setup(NetFilterState
 *nf,

Error **errp)

net/filter-rewriter.c static void colo_rewriter_setup(NetFilterState · line 384

Quoted 1 line
net/filter-rewriter.c +0 -0
       s->connection_track_table =

g_hash_table_new_full(connection_key_hash,

net/filter-rewriter.c static void colo_rewriter_setup(NetFilterState · line 388

Quoted 5 lines
net/filter-rewriter.c +1 -1
                                                         connection_key_equal,
                                                         g_free,
-                                                      connection_destroy);
+                                                      NULL);
       s->incoming_queue =

qemu_new_net_queue(qemu_netfilter_pass_to_next, nf);

Quoted 2 lines
net/filter-rewriter.c +0 -0
   }
#7 lore raw

Re: [PATCH 2/4] net/colo: Fix a "double free" crash to clear the conn_list

Quoted 58 lines · Chen
Quoted 54 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Thursday, March 31, 2022 9:15 AM
To: Zhang, Chen <[email protected]>; Jason Wang <[email protected]>
Cc: qemu-dev <[email protected]>; Like Xu <[email protected]>
Subject: Re: [PATCH 2/4] net/colo: Fix a "double free" crash to clear the conn_list

connection_track_table
-----+----------
key1 | conn |-----------------------------------------------------------+

-----+----------                                                           |
key2 | conn    |----------------------------------+                        |
-----+----------                                  |                        |
key3 | conn    |---------+                        |                        |
-----+----------         |                        |                        |
                           |                        |                        |
                           |                        |                        |
      + CompareState ++    |                        |                        |
      |               |    V                        V                        V
      +---------------+   +---------------+         +---------------+
      |conn_list      +--->conn           +--------->conn           |     connx
      +---------------+   +---------------+         +---------------+
      |               |     |           |             |          |
      +---------------+ +---v----+  +---v----+    +---v----+ +---v----+
                        |primary |  |secondary    |primary | |secondary
                        |packet  |  |packet  +    |packet  | |packet  +
                        +--------+  +--------+    +--------+ +--------+
                            |           |             |          |
                        +---v----+  +---v----+    +---v----+ +---v----+
                        |primary |  |secondary    |primary | |secondary
                        |packet  |  |packet  +    |packet  | |packet  +
                        +--------+  +--------+    +--------+ +--------+
                            |           |             |          |
                        +---v----+  +---v----+    +---v----+ +---v----+
                        |primary |  |secondary    |primary | |secondary
                        |packet  |  |packet  +    |packet  | |packet  +
                        +--------+  +--------+    +--------+ +--------+

I recalled that we should above relationships between connection_track_table conn_list and conn.
That means both connection_track_table and conn_list reference to the same conn instance.

So before this patch, connection_get() is possible to use-after-free/double free conn. where 1st was in
connection_hashtable_reset() and 2nd was
221             while (!g_queue_is_empty(conn_list)) {
222 connection_destroy(g_queue_pop_head(conn_list));
223 }

Quoted 4 lines

I also doubt that your current abort was just due to above use-after- free/double free.
If so, looks it's enough we just update to g_queue_clear(conn_list) in the 2nd place.

Quoted 5 lines

Make sense, but It also means the original patch works here, skip free conn in connection_hashtable_reset() and do it in:
221 while (!g_queue_is_empty(conn_list)) {
222 connection_destroy(g_queue_pop_head(conn_list));
223 }.
It also avoid use-after-free/double free conn.

Although you will not use-after-free here, you have to consider other situations carefully that g_hash_table_remove_all() g_hash_table_destroy() were called where the conn_list should also be freed with you approach.

Quoted 1 line

Maybe we can keep the original version to fix it?

And your commit log should be more clear.

Thanks
Zhijian

Quoted 142 lines

Thanks
Chen

Quoted 138 lines

Thanks
Zhijian

Quoted 82 lines · Chen
Quoted 82 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Monday, March 21, 2022 11:06 AM
To: Zhang, Chen <[email protected]>; Jason Wang <[email protected]>; [email protected]
Cc: qemu-dev <[email protected]>; Like Xu <[email protected]>
Subject: Re: [PATCH 2/4] net/colo: Fix a "double free" crash to clear the conn_list

net/colo-compare.c static void

Quoted 36 lines · Zhang Chen

We notice the QEMU may crash when the guest has too many incoming network connections with the following log:

[email protected]:colo_proxy_main : colo proxy connection hashtable full, clear it
free(): invalid pointer
[1] 15195 abort (core dumped) qemu-system-x86_64 ....

This is because we create the s->connection_track_table with g_hash_table_new_full() which is defined as:

GHashTable * g_hash_table_new_full (GHashFunc hash_func,

                          GEqualFunc key_equal_func,
                          GDestroyNotify key_destroy_func,
                          GDestroyNotify value_destroy_func);

The fourth parameter connection_destroy() will be called to free the memory allocated for all 'Connection' values in the hashtable when we call g_hash_table_remove_all() in the connection_hashtable_reset().

It's unnecessary because we clear the conn_list explicitly later, and it's buggy when other agents try to call connection_get() with the same connection_track_table.

---
    net/colo-compare.c    | 2 +-
    net/filter-rewriter.c | 2 +-
    2 files changed, 2 insertions(+), 2 deletions(-)
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

62554b5b3c..ab054cfd21 100644

net/colo-compare.c +0 -0
@@ -1324,7 +1324,7 @@ static void

colo_compare_complete(UserCreatable *uc, Error **errp)

net/colo-compare.c static void · line 1324

Quoted 1 line
net/colo-compare.c +0 -0
        s->connection_track_table =

g_hash_table_new_full(connection_key_hash,

net/colo-compare.c static void colo_compare_complete(UserCreatable *uc, Error **errp) · line 1327

Quoted 4 lines
net/colo-compare.c +1 -1
                                                          connection_key_equal,
                                                          g_free,
-                                                      connection_destroy);
+                                                      NULL);

202 /* if not found, create a new connection and add to hash table */ 203 Connection *connection_get(GHashTable *connection_track_table, 204                            ConnectionKey *key,
205                            GQueue *conn_list)
206 {
207     Connection *conn = g_hash_table_lookup(connection_track_table,
key);
208
209     if (conn == NULL) {
210         ConnectionKey *new_key = g_memdup(key, sizeof(*key)); 211
212         conn = connection_new(key);
213
214         if (g_hash_table_size(connection_track_table) > HASHTABLE_MAX_SIZE) {
215             trace_colo_proxy_main("colo proxy connection hashtable full," 216                                   " clear it");
217 connection_hashtable_reset(connection_track_table);

197 void connection_hashtable_reset(GHashTable *connection_track_table)
198 {
199 g_hash_table_remove_all(connection_track_table); 200 }

Quoted 6 lines
Quoted 4 lines

IIUC,  above subroutine will do some cleanup explicitly. And before your patch, connection_hashtable_reset() will release all keys and their values in this hashtable. But now, you remove all keys and just one value(conn_list) instead. Does it means other values will be leaked ?

Thanks Zhijian. Re-think about it. Yes, I think you are right. It looks need a lock to avoid into connection_get() when triggered the clear

hashtable operation.

Quoted 44 lines

What do you think?

Thanks
Chen

Quoted 38 lines

218 /*
219              * clear the conn_list
220 */
221             while (!g_queue_is_empty(conn_list)) { 222 connection_destroy(g_queue_pop_head(conn_list)); 223 }
224 }
225
226         g_hash_table_insert(connection_track_table, new_key, conn);
227 }
228
229     return conn;
230 }

Thanks
Zhijian

net/filter-rewriter.c static void colo_rewriter_setup(NetFilterState · line 383

Quoted 8 lines
net/colo-compare.c +0 -0
        colo_compare_iothread(s);
 
net/filter-rewriter.c index renamed from net/filter-rewriter.c +0 -0

bf05023dc3..c18c4c2019 100644

net/filter-rewriter.c +0 -0
@@ -383,7 +383,7 @@ static void colo_rewriter_setup(NetFilterState
 *nf,

Error **errp)

net/filter-rewriter.c static void colo_rewriter_setup(NetFilterState · line 384

Quoted 1 line
net/filter-rewriter.c +0 -0
        s->connection_track_table =

g_hash_table_new_full(connection_key_hash,

net/filter-rewriter.c static void colo_rewriter_setup(NetFilterState · line 388

Quoted 5 lines
net/filter-rewriter.c +1 -1
                                                          connection_key_equal,
                                                          g_free,
-                                                      connection_destroy);
+                                                      NULL);
        s->incoming_queue =

qemu_new_net_queue(qemu_netfilter_pass_to_next, nf);

Quoted 2 lines
net/filter-rewriter.c +0 -0
    }
#8 lore raw

RE: [PATCH 2/4] net/colo: Fix a "double free" crash to clear the conn_list

Quoted 75 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Friday, April 1, 2022 9:47 AM
To: Zhang, Chen <[email protected]>; Jason Wang <[email protected]>
Cc: qemu-dev <[email protected]>; Like Xu <[email protected]>
Subject: Re: [PATCH 2/4] net/colo: Fix a "double free" crash to clear the conn_list

Quoted 55 lines · Chen
Quoted 55 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Thursday, March 31, 2022 9:15 AM
To: Zhang, Chen <[email protected]>; Jason Wang <[email protected]>
Cc: qemu-dev <[email protected]>; Like Xu <[email protected]>
Subject: Re: [PATCH 2/4] net/colo: Fix a "double free" crash to clear the conn_list

connection_track_table
-----+----------
key1 | conn |-----------------------------------------------------------+

-----+----------                                                           |
key2 | conn    |----------------------------------+                        |
-----+----------                                  |                        |
key3 | conn    |---------+                        |                        |
-----+----------         |                        |                        |
                           |                        |                        |
                           |                        |                        |
      + CompareState ++    |                        |                        |
      |               |    V                        V                        V
      +---------------+   +---------------+         +---------------+
      |conn_list      +--->conn           +--------->conn           |     connx
      +---------------+   +---------------+         +---------------+
      |               |     |           |             |          |
      +---------------+ +---v----+  +---v----+    +---v----+ +---v----+
                        |primary |  |secondary    |primary | |secondary
                        |packet  |  |packet  +    |packet  | |packet  +
                        +--------+  +--------+    +--------+ +--------+
                            |           |             |          |
                        +---v----+  +---v----+    +---v----+ +---v----+
                        |primary |  |secondary    |primary | |secondary
                        |packet  |  |packet  +    |packet  | |packet  +
                        +--------+  +--------+    +--------+ +--------+
                            |           |             |          |
                        +---v----+  +---v----+    +---v----+ +---v----+
                        |primary |  |secondary    |primary | |secondary
                        |packet  |  |packet  +    |packet  | |packet  +
                        +--------+  +--------+    +--------+ +--------+

I recalled that we should above relationships between connection_track_table conn_list and conn.
That means both connection_track_table and conn_list reference to the same conn instance.

So before this patch, connection_get() is possible to use-after-free/double free conn. where 1st was in connection_hashtable_reset() and 2nd was
221             while (!g_queue_is_empty(conn_list)) {
222 connection_destroy(g_queue_pop_head(conn_list));
223 }

Quoted 5 lines
Quoted 4 lines

I also doubt that your current abort was just due to above use-after- free/double free.
If so, looks it's enough we just update to g_queue_clear(conn_list) in the 2nd place.

Make sense, but It also means the original patch works here, skip free conn

in connection_hashtable_reset() and do it in:

Quoted 10 lines
Quoted 4 lines

221 while (!g_queue_is_empty(conn_list)) {
222 connection_destroy(g_queue_pop_head(conn_list));
223 }.
It also avoid use-after-free/double free conn.

Although you will not use-after-free here, you have to consider other situations carefully that
g_hash_table_remove_all() g_hash_table_destroy() were called where the conn_list should also be freed with you approach.

I re-checked the code, it looks fine to me.

Quoted 4 lines
Quoted 3 lines

Maybe we can keep the original version to fix it?

And your commit log should be more clear.

OK, I will update V2 for commit log.

Thanks
Chen

Quoted 152 lines

Thanks
Zhijian

Quoted 23 lines

Thanks
Chen

Quoted 19 lines

Thanks
Zhijian

Quoted 14 lines · Chen
Quoted 14 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Monday, March 21, 2022 11:06 AM
To: Zhang, Chen <[email protected]>; Jason Wang <[email protected]>; [email protected]
Cc: qemu-dev <[email protected]>; Like Xu <[email protected]>
Subject: Re: [PATCH 2/4] net/colo: Fix a "double free" crash to clear the conn_list

Quoted 1 line · Zhang Chen

We notice the QEMU may crash when the guest has too many

incoming

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

network connections with the following log:

[email protected]:colo_proxy_main : colo proxy connection hashtable full, clear it
free(): invalid pointer
[1] 15195 abort (core dumped) qemu-system-x86_64 ....

This is because we create the s->connection_track_table with g_hash_table_new_full() which is defined as:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

GHashTable * g_hash_table_new_full (GHashFunc hash_func,

                          GEqualFunc key_equal_func,
                          GDestroyNotify key_destroy_func,
                          GDestroyNotify value_destroy_func);

The fourth parameter connection_destroy() will be called to free the memory allocated for all 'Connection' values in the hashtable when we call g_hash_table_remove_all() in the

connection_hashtable_reset().

Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines

net/colo-compare.c static void

Quoted 17 lines

It's unnecessary because we clear the conn_list explicitly later, and it's buggy when other agents try to call connection_get() with the same connection_track_table.

---
    net/colo-compare.c    | 2 +-
    net/filter-rewriter.c | 2 +-
    2 files changed, 2 insertions(+), 2 deletions(-)
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

62554b5b3c..ab054cfd21 100644

net/colo-compare.c +0 -0
@@ -1324,7 +1324,7 @@ static void

colo_compare_complete(UserCreatable *uc, Error **errp)

net/colo-compare.c static void · line 1324

Quoted 1 line
net/colo-compare.c +0 -0
        s->connection_track_table =

g_hash_table_new_full(connection_key_hash,

net/colo-compare.c static void colo_compare_complete(UserCreatable *uc, Error **errp) · line 1327

Quoted 4 lines
net/colo-compare.c +1 -1
                                                          connection_key_equal,
                                                          g_free,
-                                                      connection_destroy);
+                                                      NULL);

202 /* if not found, create a new connection and add to hash table */
203 Connection *connection_get(GHashTable *connection_track_table, 204                            ConnectionKey *key,
205                            GQueue *conn_list)
206 {
207     Connection *conn = g_hash_table_lookup(connection_track_table,
key);
208

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

209     if (conn == NULL) {
210         ConnectionKey *new_key = g_memdup(key, sizeof(*key)); 211
212         conn = connection_new(key);
213
214         if (g_hash_table_size(connection_track_table) > HASHTABLE_MAX_SIZE) {
215             trace_colo_proxy_main("colo proxy connection hashtable

full,"

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

216                                   " clear it"); 217 connection_hashtable_reset(connection_track_table);

197 void connection_hashtable_reset(GHashTable *connection_track_table)
198 {
199 g_hash_table_remove_all(connection_track_table); 200 }

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

IIUC,  above subroutine will do some cleanup explicitly. And before your patch, connection_hashtable_reset() will release all keys and their values in this hashtable. But now, you remove all keys and just one value(conn_list) instead. Does it means other values will be

leaked ?

Quoted 48 lines
Quoted 48 lines
Quoted 3 lines

Thanks Zhijian. Re-think about it. Yes, I think you are right. It looks need a lock to avoid into connection_get() when triggered the clear

hashtable operation.

Quoted 44 lines

What do you think?

Thanks
Chen

Quoted 38 lines

218 /*
219              * clear the conn_list
220 */
221             while (!g_queue_is_empty(conn_list)) { 222 connection_destroy(g_queue_pop_head(conn_list)); 223 }
224 }
225
226         g_hash_table_insert(connection_track_table, new_key, conn);
227 }
228
229     return conn;
230 }

Thanks
Zhijian

net/filter-rewriter.c static void colo_rewriter_setup(NetFilterState · line 383

Quoted 8 lines
net/colo-compare.c +0 -0
        colo_compare_iothread(s);
 
net/filter-rewriter.c index renamed from net/filter-rewriter.c +0 -0

bf05023dc3..c18c4c2019 100644

net/filter-rewriter.c +0 -0
@@ -383,7 +383,7 @@ static void colo_rewriter_setup(NetFilterState
 *nf,

Error **errp)

net/filter-rewriter.c static void colo_rewriter_setup(NetFilterState · line 384

Quoted 1 line
net/filter-rewriter.c +0 -0
        s->connection_track_table =

g_hash_table_new_full(connection_key_hash,

net/filter-rewriter.c static void colo_rewriter_setup(NetFilterState · line 388

Quoted 5 lines
net/filter-rewriter.c +1 -1
                                                          connection_key_equal,
                                                          g_free,
-                                                      connection_destroy);
+                                                      NULL);
        s->incoming_queue =

qemu_new_net_queue(qemu_netfilter_pass_to_next, nf);

Quoted 2 lines
net/filter-rewriter.c +0 -0
    }
author
#9 lore raw

PATCH 3/4 net/colo.c: No need to track conn_list for filter-rewriter

In reply to Zhang Chen

Filter-rewriter no need to track connection in conn_list. This patch fix the glib g_queue_is_empty assertion when COLO guest keep a lot of network connection.

---
 net/colo.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/colo.c +1 -1
index 1f8162f59f..694f3c93ef 100644
@@ -218,7 +218,7 @@ Connection *connection_get(GHashTable *connection_track_table,
             /*
              * clear the conn_list
              */
-            while (!g_queue_is_empty(conn_list)) {
+            while (conn_list && !g_queue_is_empty(conn_list)) {
                 connection_destroy(g_queue_pop_head(conn_list));
             }
         }
Signature · 1 line
2.25.1
Reviewed-by on v1 3/4
#10 lore raw

Re: [PATCH 3/4] net/colo.c: No need to track conn_list for filter-rewriter

Quoted 5 lines · Zhang Chen

Filter-rewriter no need to track connection in conn_list. This patch fix the glib g_queue_is_empty assertion when COLO guest keep a lot of network connection.

LGTM.

Quoted 17 lines
---
  net/colo.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
net/colo.c +1 -1
index 1f8162f59f..694f3c93ef 100644
@@ -218,7 +218,7 @@ Connection *connection_get(GHashTable *connection_track_table,
              /*
               * clear the conn_list
               */
-            while (!g_queue_is_empty(conn_list)) {
+            while (conn_list && !g_queue_is_empty(conn_list)) {
                  connection_destroy(g_queue_pop_head(conn_list));
              }
          }
author
#11 lore raw

PATCH 4/4 net/colo.c: fix segmentation fault when packet is not parsed correctly

In reply to Zhang Chen

When COLO use only one vnet_hdr_support parameter between filter-redirector and filter-mirror(or colo-compare), COLO will crash with segmentation fault. Back track as follow:

Thread 1 "qemu-system-x86" received signal SIGSEGV, Segmentation fault. 0x0000555555cb200b in eth_get_l2_hdr_length (p=0x0)
at /home/tao/project/COLO/colo-qemu/include/net/eth.h:296
296 uint16_t proto = be16_to_cpu(PKT_GET_ETH_HDR(p)->h_proto);
(gdb) bt
0 0x0000555555cb200b in eth_get_l2_hdr_length (p=0x0)
at /home/tao/project/COLO/colo-qemu/include/net/eth.h:296
1 0x0000555555cb22b4 in parse_packet_early (pkt=0x555556a44840) at net/colo.c:49
2 0x0000555555cb2b91 in is_tcp_packet (pkt=0x555556a44840) at net/filter-rewriter.c:63

So wrong vnet_hdr_len will cause pkt->data become NULL. Add check to raise error and add trace-events to track vnet_hdr_len.

---
 net/colo.c       | 9 ++++++++-
 net/trace-events | 1 +
 2 files changed, 9 insertions(+), 1 deletion(-)
net/colo.c +8 -1
index 694f3c93ef..6b0ff562ad 100644
@@ -46,7 +46,14 @@ int parse_packet_early(Packet *pkt)
     static const uint8_t vlan[] = {0x81, 0x00};
     uint8_t *data = pkt->data + pkt->vnet_hdr_len;
     uint16_t l3_proto;
-    ssize_t l2hdr_len = eth_get_l2_hdr_length(data);
+    ssize_t l2hdr_len;
+
+    if (data == NULL) {
+        trace_colo_proxy_main_vnet_info("This packet is not parsed correctly, "
+                                        "pkt->vnet_hdr_len", pkt->vnet_hdr_len);
+        return 1;
+    }
+    l2hdr_len = eth_get_l2_hdr_length(data);
 
     if (pkt->size < ETH_HLEN + pkt->vnet_hdr_len) {
         trace_colo_proxy_main("pkt->size < ETH_HLEN");
net/trace-events +1 -0
index d7a17256cc..6af927b4b9 100644
@@ -9,6 +9,7 @@ vhost_user_event(const char *chr, int event) "chr: %s got event: %d"
 
 # colo.c
 colo_proxy_main(const char *chr) ": %s"
+colo_proxy_main_vnet_info(const char *sta, int size) ": %s = %d"
 
 # colo-compare.c
 colo_compare_main(const char *chr) ": %s"
Signature · 1 line
2.25.1
Reviewed-by on v1 4/4
#12 lore raw

Re: [PATCH 4/4] net/colo.c: fix segmentation fault when packet is not parsed correctly

Quoted 16 lines · Zhang Chen

When COLO use only one vnet_hdr_support parameter between filter-redirector and filter-mirror(or colo-compare), COLO will crash with segmentation fault. Back track as follow:

Thread 1 "qemu-system-x86" received signal SIGSEGV, Segmentation fault. 0x0000555555cb200b in eth_get_l2_hdr_length (p=0x0)
at /home/tao/project/COLO/colo-qemu/include/net/eth.h:296
296 uint16_t proto = be16_to_cpu(PKT_GET_ETH_HDR(p)->h_proto);
(gdb) bt
0 0x0000555555cb200b in eth_get_l2_hdr_length (p=0x0)
at /home/tao/project/COLO/colo-qemu/include/net/eth.h:296
1 0x0000555555cb22b4 in parse_packet_early (pkt=0x555556a44840) at net/colo.c:49
2 0x0000555555cb2b91 in is_tcp_packet (pkt=0x555556a44840) at net/filter-rewriter.c:63

Quoted 1 line

So wrong vnet_hdr_len will cause pkt->data become NULL.

Not sure if we can check this earlier, well

Quoted 44 lines

Add check to
raise error and add trace-events to track vnet_hdr_len.

---
  net/colo.c       | 9 ++++++++-
  net/trace-events | 1 +
  2 files changed, 9 insertions(+), 1 deletion(-)
net/colo.c +8 -1
index 694f3c93ef..6b0ff562ad 100644
@@ -46,7 +46,14 @@ int parse_packet_early(Packet *pkt)
      static const uint8_t vlan[] = {0x81, 0x00};
      uint8_t *data = pkt->data + pkt->vnet_hdr_len;
      uint16_t l3_proto;
-    ssize_t l2hdr_len = eth_get_l2_hdr_length(data);
+    ssize_t l2hdr_len;
+
+    if (data == NULL) {
+        trace_colo_proxy_main_vnet_info("This packet is not parsed correctly, "
+                                        "pkt->vnet_hdr_len", pkt->vnet_hdr_len);
+        return 1;
+    }
+    l2hdr_len = eth_get_l2_hdr_length(data);
 
      if (pkt->size < ETH_HLEN + pkt->vnet_hdr_len) {
          trace_colo_proxy_main("pkt->size < ETH_HLEN");
net/trace-events +1 -0
index d7a17256cc..6af927b4b9 100644
@@ -9,6 +9,7 @@ vhost_user_event(const char *chr, int event) "chr: %s got event: %d"
 
  # colo.c
  colo_proxy_main(const char *chr) ": %s"
+colo_proxy_main_vnet_info(const char *sta, int size) ": %s = %d"
 
  # colo-compare.c
  colo_compare_main(const char *chr) ": %s"