Skip to contentSkip to conversation

huawei: modem channel gets closed after PPP disconnection

PATCH Superseded Christophe Ronco 2 participants 8 messages Feb 16, 2017 – Feb 21, 2017 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

Hi,

I use a huawei E3372 modem. It connects to GPRS using PPP. I am able to activate context and then deactivate it. After that, I am not able to reactivate the context.
During deactivation, the AT channel used by PPP gets closed and nobody reopens it. During reactivation, the first AT command (AT+CGDCONT) fails. Channel shutting down occurs just after NO CARRIER is received on modem channel (channel is back to AT state) along with a bunch of wrong data. Here is the log:
Modem: < \r\nNO CARRIER\r\n~\377}#\200!}%}"} }$\210*~ Shutting down is detected in function received_data (gatio.c). Flags G_IO_HUP and G_IO_ERR are received.

I have done a patch in huawei plugin that correct the problem@least with my modem. The idea is for plugin to get notified when modem AT channel gets closed and when this happens to remove gprs-context, reopen modem AT channel and restart gprs-context driver. This is possible because all other drivers (sim, gprs, ...) use PCUI AT channel and not modem AT channel.
But I am not sure this is:
- the right place to do it (in plugin or in gprs-context driver). I choose plugin because there are some AT commands to send just after channel opening and this is a plugin job.
- the right way to do it (removing the whole gprs-context driver might
be too much).
Please find attached my current patch. It might not apply exactly on HEAD of master. At this stage, this is more to have comment on my fix than to really send a patch proposal.

If you need more info on this problem, I should be able to send traces with AT debug. It does not help a lot and I had to add traces in gatchat module to investigate.

Best Regards,

Christophe

Quoted 1 line

From f840da4307634c98d6f1a100f0012d211521ed2a Mon Sep 17 00:00:00 2001

From: Christophe Ronco <[email protected]>
Date: Wed, 15 Feb 2017 12:11:13 +0100
Subject: [PATCH] huawei: reopen modem channel if it disconnects

On E3372 after a GPRS disconnection, modem AT channel (used by PPP) gets disconnected (G_IO_HUP and G_IO_ERR is detected in received_data in gatio.c). After that all connection attempts fail because we are not able to send any AT commands on modem channel (it is closed).

With this patch, when this behavior is detected, we close gprs context driver, reopen AT channel and gprs context driver.

---
 plugins/huawei.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 47 insertions(+), 7 deletions(-)
plugins/huawei.c +47 -7
index 5dc9bc3b..4f9b3956 100644
@@ -78,6 +78,8 @@ enum {
 struct huawei_data {
 	GAtChat *modem;
 	GAtChat *pcui;
+	struct ofono_gprs *gprs;
+	struct ofono_gprs_context *gc;
 	gboolean have_sim;
 	int sim_state;
 	guint sysinfo_poll_source;
@@ -584,6 +586,45 @@ static GAtChat *open_device(struct ofono_modem *modem,
 	return chat;
 }
 
+static void modem_disconnect(gpointer user_data)
+{
+	struct ofono_modem *modem = user_data;
+	struct huawei_data *data = ofono_modem_get_data(modem);
+
+	DBG("");
+
+	ofono_warn("modem channel disconnected");
+
+	/* clean and close modem device */
+	g_at_chat_cancel_all(data->modem);
+	g_at_chat_unregister_all(data->modem);
+	g_at_chat_unref(data->modem);
+	data->modem = NULL;
+
+	/* close gprs context driver */
+	ofono_gprs_context_remove(data->gc);
+
+	/* reopen modem channel */
+	data->modem = open_device(modem, "Modem", "Modem: ");
+
+	if (data->modem == NULL) {
+		DBG("Can't reopen device");
+		return;
+	}
+
+	/* configure modem channel */
+	g_at_chat_set_disconnect_function(data->modem, modem_disconnect, modem);
+	g_at_chat_set_slave(data->modem, data->pcui);
+	g_at_chat_send(data->modem, "ATE0 +CMEE=1", NULL, NULL, NULL, NULL);
+
+	/* reopen gprs context driver */
+	data->gc = ofono_gprs_context_create(modem, OFONO_VENDOR_HUAWEI,
+						"atmodem", data->modem);
+
+	if (data->gprs && data->gc)
+		ofono_gprs_add_context(data->gprs, data->gc);
+}
+
 static int huawei_enable(struct ofono_modem *modem)
 {
 	struct huawei_data *data = ofono_modem_get_data(modem);
@@ -594,6 +635,8 @@ static int huawei_enable(struct ofono_modem *modem)
 	if (data->modem == NULL)
 		return -EINVAL;
 
+	g_at_chat_set_disconnect_function(data->modem, modem_disconnect, modem);
+
 	data->pcui = open_device(modem, "Pcui", "PCUI: ");
 	if (data->pcui == NULL) {
 		g_at_chat_unref(data->modem);
@@ -820,9 +863,6 @@ static void huawei_post_sim(struct ofono_modem *modem)
 	}
 
 	if (data->have_gsm == TRUE) {
-		struct ofono_gprs *gprs;
-		struct ofono_gprs_context *gc;
-
 		ofono_phonebook_create(modem, 0, "atmodem", data->pcui);
 		ofono_radio_settings_create(modem, 0,
 						"huaweimodem", data->pcui);
@@ -830,13 +870,13 @@ static void huawei_post_sim(struct ofono_modem *modem)
 		ofono_sms_create(modem, OFONO_VENDOR_HUAWEI,
 						"atmodem", data->pcui);
 
-		gprs = ofono_gprs_create(modem, OFONO_VENDOR_HUAWEI,
+		data->gprs = ofono_gprs_create(modem, OFONO_VENDOR_HUAWEI,
 						"atmodem", data->pcui);
-		gc = ofono_gprs_context_create(modem, OFONO_VENDOR_HUAWEI,
+		data->gc = ofono_gprs_context_create(modem, OFONO_VENDOR_HUAWEI,
 						"atmodem", data->modem);
 
-		if (gprs && gc)
-			ofono_gprs_add_context(gprs, gc);
+		if (data->gprs && data->gc)
+			ofono_gprs_add_context(data->gprs, data->gc);
 	}
 }
 
Signature · 1 line
2.11.0
#2 lore raw

Hi Christophe,

Quoted 13 lines · Christophe Ronco

Hi,

I use a huawei E3372 modem. It connects to GPRS using PPP. I am able to activate context and then deactivate it. After that, I am not able to reactivate the context.
During deactivation, the AT channel used by PPP gets closed and nobody reopens it. During reactivation, the first AT command (AT+CGDCONT) fails. Channel shutting down occurs just after NO CARRIER is received on modem channel (channel is back to AT state) along with a bunch of wrong data. Here is the log:
Modem: < \r\nNO CARRIER\r\n~\377}#\200!}%}"} }$\210*~ Shutting down is detected in function received_data (gatio.c). Flags G_IO_HUP and G_IO_ERR are received.

This is a common problem with these devices, a hold-over from olden serial days. Basically the modem signals a hangup and in addition to sending a NO CARRIER, it also twiddles with some serial status bits. This causes the socket to receive a 'shutdown' event. I think that on some devices the serial port is also physically removed for a short time, so even opening the port again is likely to fail at least some of the time.

We use AT&C0 to turn off this behavior, but it sounds like your firmware is ignoring this.

Quoted 7 lines

I have done a patch in huawei plugin that correct the problem at least with my modem. The idea is for plugin to get notified when modem AT channel gets closed and when this happens to remove gprs-context, reopen modem AT channel and restart gprs-context driver. This is possible because all other drivers (sim, gprs, ...) use PCUI AT channel and not modem AT channel.

Doing it in the modem driver (e.g. plugins/huawei.c) is the right place.
The fix you have come up with looks okayish. However, you might want
to strace/ptrace ofonod to see what the kernel is actually sending you to trigger this. Also, a udev event monitor might help to tell if the tty is being physically removed / reinserted.

We have had a similar fix in the past and it simply didn't work due to the race conditions outlined.

Regards,
-Denis

#3 lore raw

Hello Denis,

Quoted 7 lines · Denis Kenzior

Doing it in the modem driver (e.g. plugins/huawei.c) is the right place. The fix you have come up with looks okayish. However, you might want to strace/ptrace ofonod to see what the kernel is actually sending you to trigger this. Also, a udev event monitor might help to tell if the tty is being physically removed / reinserted.

I've done the connect/disconnect operation with strace and a udev event monitor. The PPP modem channel is /dev/ttyUSB1. Here is the result of the udev event monitor:
root(a)klk-lpbs-040070:~ # udevadm monitor
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing KERNEL - the kernel uevent

KERNEL[15443.637223] add      /devices/virtual/net/ppp0 (net)
KERNEL[15443.637377] add      /devices/virtual/net/ppp0/queues/rx-0 (queues)
KERNEL[15443.637464] add      /devices/virtual/net/ppp0/queues/tx-0 (queues)
UDEV  [15443.735949] add      /devices/virtual/net/ppp0 (net)
UDEV  [15443.742095] add      /devices/virtual/net/ppp0/queues/rx-0 (queues)
UDEV  [15443.754166] add      /devices/virtual/net/ppp0/queues/tx-0 (queues)
KERNEL[15449.168724] remove   /devices/virtual/net/ppp0/queues/rx-0 (queues)
KERNEL[15449.172883] remove   /devices/virtual/net/ppp0/queues/tx-0 (queues)
KERNEL[15449.176000] remove   /devices/virtual/net/ppp0 (net)
UDEV  [15449.183343] remove   /devices/virtual/net/ppp0/queues/tx-0 (queues)
UDEV  [15449.184839] remove   /devices/virtual/net/ppp0/queues/rx-0 (queues)
UDEV  [15449.275684] remove   /devices/virtual/net/ppp0 (net)

So only network interface appears (connection) and vanishes (disconnection). The strace result is too big to be sent on the list, you can see it here:

https://framabin.org/?7e392864cc149771#8gyA/NLLDVVpnwxUV0FpR0wH5U6rwlogzy7S8yXmSMk=

What I learnt from it:

line 5839: open("/dev/ttyUSB1", O_RDWR|O_NOCTTY|O_NONBLOCK) = 10 => open line 9041: read(10, "\r\nNO CARRIER\r\n~\377}#\200!}%}\"} }$\210*~", 8192) = 31 => last data read
line 9049: poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}, {fd=6, events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLIN}, {fd=10, events=POLLIN}, {fd=11, events=POLLIN}], 8, -1) = 1 ([{fd=10, revents=POLLIN|POLLERR|POLLHUP}]) => poll error

So nothing more than a POLLERR|POLLHUP event on modem channel.

After that, /dev/ttyUSB1 is reopened (line 9068, before close, that's strange). This is because I 've done the trace with my patch. It does not give more information on why error happens, but we do know that /dev/ttyUSB1 does not disappear.

If you are OK, I'll send patch next week.

Best regards,

Christophe

#4 lore raw

Hi Christophe,

Quoted 3 lines

So only network interface appears (connection) and vanishes (disconnection).

Okay, good.

Quoted 2 lines

So nothing more than a POLLERR|POLLHUP event on modem channel.

okay, but the socket still has to be reopened I guess due to the POLLHUP.

Quoted 1 line

If you are OK, I'll send patch next week.

Sure.

Regards,
-Denis

#5 lore raw

PATCH 1/1 huawei: reopen modem channel if it disconnects

On E3372 after a GPRS disconnection, modem AT channel (used by PPP) gets disconnected (G_IO_HUP and G_IO_ERR are detected in received_data in gatio.c).
After that all connection attempts fail because we are not able to send any AT commands on modem channel (it is closed).

With this patch, when this behavior is detected, we close gprs context driver, reopen AT channel and gprs context driver.

---
 plugins/huawei.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 47 insertions(+), 7 deletions(-)
plugins/huawei.c +47 -7
index 59995b30..fcec7994 100644
@@ -78,6 +78,8 @@ enum {
 struct huawei_data {
 	GAtChat *modem;
 	GAtChat *pcui;
+	struct ofono_gprs *gprs;
+	struct ofono_gprs_context *gc;
 	gboolean have_sim;
 	int sim_state;
 	guint sysinfo_poll_source;
@@ -584,6 +586,45 @@ static GAtChat *open_device(struct ofono_modem *modem,
 	return chat;
 }
 
+static void modem_disconnect(gpointer user_data)
+{
+	struct ofono_modem *modem = user_data;
+	struct huawei_data *data = ofono_modem_get_data(modem);
+
+	DBG("");
+
+	ofono_warn("modem channel disconnected");
+
+	/* clean and close modem device */
+	g_at_chat_cancel_all(data->modem);
+	g_at_chat_unregister_all(data->modem);
+	g_at_chat_unref(data->modem);
+	data->modem = NULL;
+
+	/* close gprs context driver */
+	ofono_gprs_context_remove(data->gc);
+
+	/* reopen modem channel */
+	data->modem = open_device(modem, "Modem", "Modem: ");
+
+	if (data->modem == NULL) {
+		DBG("Can't reopen device");
+		return;
+	}
+
+	/* configure modem channel */
+	g_at_chat_set_disconnect_function(data->modem, modem_disconnect, modem);
+	g_at_chat_set_slave(data->modem, data->pcui);
+	g_at_chat_send(data->modem, "ATE0 +CMEE=1", NULL, NULL, NULL, NULL);
+
+	/* reopen gprs context driver */
+	data->gc = ofono_gprs_context_create(modem, OFONO_VENDOR_HUAWEI,
+						"atmodem", data->modem);
+
+	if (data->gprs && data->gc)
+		ofono_gprs_add_context(data->gprs, data->gc);
+}
+
 static int huawei_enable(struct ofono_modem *modem)
 {
 	struct huawei_data *data = ofono_modem_get_data(modem);
@@ -594,6 +635,8 @@ static int huawei_enable(struct ofono_modem *modem)
 	if (data->modem == NULL)
 		return -EINVAL;
 
+	g_at_chat_set_disconnect_function(data->modem, modem_disconnect, modem);
+
 	data->pcui = open_device(modem, "Pcui", "PCUI: ");
 	if (data->pcui == NULL) {
 		g_at_chat_unref(data->modem);
@@ -820,9 +863,6 @@ static void huawei_post_sim(struct ofono_modem *modem)
 	}
 
 	if (data->have_gsm == TRUE) {
-		struct ofono_gprs *gprs;
-		struct ofono_gprs_context *gc;
-
 		ofono_phonebook_create(modem, 0, "atmodem", data->pcui);
 		ofono_radio_settings_create(modem, 0,
 						"huaweimodem", data->pcui);
@@ -830,13 +870,13 @@ static void huawei_post_sim(struct ofono_modem *modem)
 		ofono_sms_create(modem, OFONO_VENDOR_HUAWEI,
 						"atmodem", data->pcui);
 
-		gprs = ofono_gprs_create(modem, OFONO_VENDOR_HUAWEI,
+		data->gprs = ofono_gprs_create(modem, OFONO_VENDOR_HUAWEI,
 						"atmodem", data->pcui);
-		gc = ofono_gprs_context_create(modem, 0,
+		data->gc = ofono_gprs_context_create(modem, 0,
 						"atmodem", data->modem);
 
-		if (gprs && gc)
-			ofono_gprs_add_context(gprs, gc);
+		if (data->gprs && data->gc)
+			ofono_gprs_add_context(data->gprs, data->gc);
 	}
 }
 
Signature · 1 line
2.11.0
#6 lore raw

v2 cover Fix possible seg fault in previous patch

A segmentation fault is possible with my previous patch.
It occurs if modem is removed. In this case, modem_disconnect is called after huawei_remove.
ofono_modem_get_data(modem) will return NULL in this case.

Christophe Ronco (1):
huawei: reopen modem channel if it disconnects

 plugins/huawei.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 50 insertions(+), 7 deletions(-)
Signature · 1 line
2.11.0
#7 lore raw

PATCH v2 1/1 huawei: reopen modem channel if it disconnects

On E3372 after a GPRS disconnection, modem AT channel (used by PPP) gets disconnected (G_IO_HUP and G_IO_ERR are detected in received_data in gatio.c).
After that all connection attempts fail because we are not able to send any AT commands on modem channel (it is closed).

With this patch, when this behavior is detected, we close gprs context driver, reopen AT channel and gprs context driver.

---
 plugins/huawei.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 50 insertions(+), 7 deletions(-)
plugins/huawei.c +50 -7
index 59995b30..f02315d5 100644
@@ -78,6 +78,8 @@ enum {
 struct huawei_data {
 	GAtChat *modem;
 	GAtChat *pcui;
+	struct ofono_gprs *gprs;
+	struct ofono_gprs_context *gc;
 	gboolean have_sim;
 	int sim_state;
 	guint sysinfo_poll_source;
@@ -584,6 +586,48 @@ static GAtChat *open_device(struct ofono_modem *modem,
 	return chat;
 }
 
+static void modem_disconnect(gpointer user_data)
+{
+	struct ofono_modem *modem = user_data;
+	struct huawei_data *data = ofono_modem_get_data(modem);
+
+	if (data == NULL) {
+		DBG("Modem has already been removed");
+		return;
+	}
+
+	ofono_warn("Modem channel disconnected");
+
+	/* clean and close modem device */
+	g_at_chat_cancel_all(data->modem);
+	g_at_chat_unregister_all(data->modem);
+	g_at_chat_unref(data->modem);
+	data->modem = NULL;
+
+	/* close gprs context driver */
+	ofono_gprs_context_remove(data->gc);
+
+	/* reopen modem channel */
+	data->modem = open_device(modem, "Modem", "Modem: ");
+
+	if (data->modem == NULL) {
+		DBG("Can't reopen device");
+		return;
+	}
+
+	/* configure modem channel */
+	g_at_chat_set_disconnect_function(data->modem, modem_disconnect, modem);
+	g_at_chat_set_slave(data->modem, data->pcui);
+	g_at_chat_send(data->modem, "ATE0 +CMEE=1", NULL, NULL, NULL, NULL);
+
+	/* reopen gprs context driver */
+	data->gc = ofono_gprs_context_create(modem, OFONO_VENDOR_HUAWEI,
+						"atmodem", data->modem);
+
+	if (data->gprs && data->gc)
+		ofono_gprs_add_context(data->gprs, data->gc);
+}
+
 static int huawei_enable(struct ofono_modem *modem)
 {
 	struct huawei_data *data = ofono_modem_get_data(modem);
@@ -594,6 +638,8 @@ static int huawei_enable(struct ofono_modem *modem)
 	if (data->modem == NULL)
 		return -EINVAL;
 
+	g_at_chat_set_disconnect_function(data->modem, modem_disconnect, modem);
+
 	data->pcui = open_device(modem, "Pcui", "PCUI: ");
 	if (data->pcui == NULL) {
 		g_at_chat_unref(data->modem);
@@ -820,9 +866,6 @@ static void huawei_post_sim(struct ofono_modem *modem)
 	}
 
 	if (data->have_gsm == TRUE) {
-		struct ofono_gprs *gprs;
-		struct ofono_gprs_context *gc;
-
 		ofono_phonebook_create(modem, 0, "atmodem", data->pcui);
 		ofono_radio_settings_create(modem, 0,
 						"huaweimodem", data->pcui);
@@ -830,13 +873,13 @@ static void huawei_post_sim(struct ofono_modem *modem)
 		ofono_sms_create(modem, OFONO_VENDOR_HUAWEI,
 						"atmodem", data->pcui);
 
-		gprs = ofono_gprs_create(modem, OFONO_VENDOR_HUAWEI,
+		data->gprs = ofono_gprs_create(modem, OFONO_VENDOR_HUAWEI,
 						"atmodem", data->pcui);
-		gc = ofono_gprs_context_create(modem, 0,
+		data->gc = ofono_gprs_context_create(modem, 0,
 						"atmodem", data->modem);
 
-		if (gprs && gc)
-			ofono_gprs_add_context(gprs, gc);
+		if (data->gprs && data->gc)
+			ofono_gprs_add_context(data->gprs, data->gc);
 	}
 }
 
Signature · 1 line
2.11.0
#8 lore raw

Re: [PATCH v2 1/1] huawei: reopen modem channel if it disconnects

Hi Christophe,

Quoted 6 lines · Christophe Ronco

On E3372 after a GPRS disconnection, modem AT channel (used by PPP) gets disconnected (G_IO_HUP and G_IO_ERR are detected in received_data in gatio.c).
After that all connection attempts fail because we are not able to send any AT commands on modem channel (it is closed).

Quoted 6 lines

With this patch, when this behavior is detected, we close gprs context driver, reopen AT channel and gprs context driver.

---
 plugins/huawei.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 50 insertions(+), 7 deletions(-)

Applied, thanks.

Regards,
-Denis