Skip to contentSkip to conversation

Remote management of embedded devices

DISCUSSION Alex J Lennon 7 participants 12 messages Mar 7, 2014 – Mar 10, 2014

lore
Share & tools
lore
#0 lore raw

Hi,

I'm looking into remote management solutions for an upcoming headless mesh edge router running Poky. I think, at least in the initial rollout we're going to need something more than, say, a cron-based package update facility.

I'm currently thinking of going down the route of a cloud based server providing SSH port forwarding to the embedded devices, and then perhaps putting some scripting together on top of that to enable monitoring, configuration, and control.

I was wondering if there are better solutions already supported by the Yocto project which people might be using to good effect in production systems ?

Thanks,

Alex

#1 lore raw

Alex,

Quoted 14 lines

-----Original Message-----
From: [email protected] [mailto:yocto- [email protected]] On Behalf Of Alex J Lennon
Sent: Friday, March 07, 2014 7:30 AM
To: [email protected]
Subject: [yocto] Remote management of embedded devices

Hi,

I'm looking into remote management solutions for an upcoming headless mesh edge router running Poky. I think, at least in the initial rollout we're going to need something more than, say, a cron-based package update facility.

Quoted 4 lines

I'm currently thinking of going down the route of a cloud based server providing SSH port forwarding to the embedded devices, and then perhaps putting some scripting together on top of that to enable monitoring, configuration, and control.

This is similar to what we have running. We're using opkg for the package management system and allow firmware upgrade through the USB stick or through remote access. We have an SSH server for remote access, and each device has its own private key for access into the server. If the device is in the middle of an SSH session with the remote access server, we can then SSH into the device from our server if we want to do some deeper diagnostics on an issue with a device. We have separate HTTP server which each device queries to see if it needs to check in. So instead of having to do an ssh login each time to check if there's a firmware upgrade available, it just needs to do a HTTP GET to see if there's firmware available or if we want to check on its status.

The biggest issues we've had have been due to our network setup and handling upgrade both through the network and the USB stick. We are using "opkg upgrade --download-only" as the first step of the upgrade process to make sure that we don't do a partial upgrade. opkg-0.1.8 doesn't do --download-only for file:// sources; why download a file that is already on the filesystem? So I had to add a patch so it would download files from the USB stick. We also had an issue with DNS names because the server has a different name when the device finds it locally on our intranet then when it connects remotely, so we had to setup separate mirrors. Other than that it's been working pretty well.

I'd also love to hear other people's solutions to see if they have done something similar or came up with a different solution.

Regards,
Bryan

Quoted 12 lines

I was wondering if there are better solutions already supported by the Yocto project which people might be using to good effect in production systems ?

Thanks,

Alex

_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto

#2 lore raw

We're experimenting with software called yalertunnel (yaler.net). It causes embedded devices to maintain a connection to a relay server through which they can be accessed. We are using a 3g broadband dongle with no static ip or port forwarding and results have been quite good. We have been using it only for accessing our web-interface but they also support ssh.

Regards,
Mark

Quoted 70 lines · Bryan Evenson

Alex,

Quoted 11 lines

-----Original Message-----
From: [email protected] [mailto:yocto- [email protected]] On Behalf Of Alex J Lennon
Sent: Friday, March 07, 2014 7:30 AM
To: [email protected]
Subject: [yocto] Remote management of embedded devices

Hi,

I'm looking into remote management solutions for an upcoming headless mesh edge router running Poky. I think, at least in the initial rollout

we're

Quoted 7 lines

going to need something more than, say, a cron-based package update facility.

I'm currently thinking of going down the route of a cloud based server providing SSH port forwarding to the embedded devices, and then perhaps putting some scripting together on top of that to enable monitoring, configuration, and control.

This is similar to what we have running. We're using opkg for the package management system and allow firmware upgrade through the USB stick or through remote access. We have an SSH server for remote access, and each device has its own private key for access into the server. If the device is in the middle of an SSH session with the remote access server, we can then SSH into the device from our server if we want to do some deeper diagnostics on an issue with a device. We have separate HTTP server which each device queries to see if it needs to check in. So instead of having to do an ssh login each time to check if there's a firmware upgrade available, it just needs to do a HTTP GET to see if there's firmware available or if we want to check on its status.

The biggest issues we've had have been due to our network setup and handling upgrade both through the network and the USB stick. We are using "opkg upgrade --download-only" as the first step of the upgrade process to make sure that we don't do a partial upgrade. opkg-0.1.8 doesn't do --download-only for file:// sources; why download a file that is already on the filesystem? So I had to add a patch so it would download files from the USB stick. We also had an issue with DNS names because the server has a different name when the device finds it locally on our intranet then when it connects remotely, so we had to setup separate mirrors. Other than that it's been working pretty well.

I'd also love to hear other people's solutions to see if they have done something similar or came up with a different solution.

Regards,
Bryan

Quoted 2 lines

I was wondering if there are better solutions already supported by the

Yocto

Quoted 1 line

project which people might be using to good effect in production systems

?

Quoted 9 lines

Thanks,

Alex

_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto

_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto

#3 lore raw

Slightly OT: Administrivia (add "-- " to footers)

If I could make a small suggestion for the Yocto list owners.

I also run MailMan and have set all the footers to be preceded by the "-- " (dash dash space).

For the mailers that understand it (like Thunderbird), everything past the "-- " is excluded in replies.

This would cut down a lot on needed trim of appended footers on the list.

Just an item for the suggestion box.

Cheers,

-Ben

#4 lore raw

Re: Slightly OT: Administrivia (add "-- " to footers)

Quoted 10 lines · Ben Kamen

If I could make a small suggestion for the Yocto list owners.

I also run MailMan and have set all the footers to be preceded by the "-- " (dash dash space).

For the mailers that understand it (like Thunderbird), everything past the "-- " is excluded in replies.

This would cut down a lot on needed trim of appended footers on the list.

I think its a very good suggestion. I am sure Michael H would be interested.

Quoted 10 lines

Just an item for the suggestion box.

Cheers,

-Ben

_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto

#5 lore raw

Re: Slightly OT: Administrivia (add "-- " to footers)

Quoted 6 lines · Khem Raj
Quoted 5 lines · Ben Kamen

If I could make a small suggestion for the Yocto list owners.

I also run MailMan and have set all the footers to be preceded by the "-- " (dash dash space).

Quoted 6 lines
Quoted 5 lines

For the mailers that understand it (like Thunderbird), everything past the "-- " is excluded in replies.

This would cut down a lot on needed trim of appended footers on the list.

I think its a very good suggestion. I am sure Michael H would be interested.

Good idea. I've prepended '-- \r\n' to the footers of all Yocto Project and OE lists.

Michael Halstead
Yocto Project / Sys Admin

Quoted 15 lines
Quoted 10 lines

Just an item for the suggestion box.

Cheers,

-Ben

_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto

_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto

#6 lore raw

Re: Slightly OT: Administrivia (add "-- " to footers)

Quoted 2 lines · Michael Halstead

Good idea. I've prepended '-- \r\n' to the footers of all Yocto Project and OE lists.

thanks Michael

#7 lore raw
In reply to Mark O'Donovan

Hi Mark,

Quoted 2 lines · Mark O'Donovan

We're experimenting with software called yalertunnel (yaler.net).

Is it open-source? Is there any cost to use it?

Regards,

--
*dS
Diego Sueiro

Administrador do Embarcados
www.embarcados.com.br

/*long live rock 'n roll*/

#8 lore raw

It is open-source.
I've just looked through the license file, the main point seems to be that is is free for non-commercial purposes.
It claims to be based on the sleepycat license.

See here for the source code and license file:
http://hg.yaler.org/yaler/src/

Regards,
Mark

Quoted 19 lines · Diego Sueiro

Hi Mark,

Quoted 2 lines · Mark O'Donovan

We're experimenting with software called yalertunnel (yaler.net).

Is it open-source? Is there any cost to use it?

Regards,

--
*dS
Diego Sueiro

Administrador do Embarcados
www.embarcados.com.br

/*long live rock 'n roll*/

#9 lore raw
In reply to Mark O'Donovan
Quoted 7 lines · Mark O'Donovan

We're experimenting with software called yalertunnel (yaler.net
<http://yaler.net>).
It causes embedded devices to maintain a connection to a relay server through which they can be accessed. We are using a 3g broadband dongle with no static ip or port forwarding and results have been quite good. We have been using it only for accessing our web-interface but they also support ssh.

Thanks for that Mark. I will take a look.

Best, Alex

#10 lore raw
In reply to Bryan Evenson

+

Quoted 16 lines · Bryan Evenson

Alex,

Quoted 14 lines

-----Original Message-----
From: [email protected] [mailto:yocto- [email protected]] On Behalf Of Alex J Lennon
Sent: Friday, March 07, 2014 7:30 AM
To: [email protected]
Subject: [yocto] Remote management of embedded devices

Hi,

I'm looking into remote management solutions for an upcoming headless mesh edge router running Poky. I think, at least in the initial rollout we're going to need something more than, say, a cron-based package update facility.

Quoted 7 lines
Quoted 4 lines

I'm currently thinking of going down the route of a cloud based server providing SSH port forwarding to the embedded devices, and then perhaps putting some scripting together on top of that to enable monitoring, configuration, and control.

This is similar to what we have running. We're using opkg for the package management system and allow firmware upgrade through the USB stick or through remote access. We have an SSH server for remote access, and each device has its own private key for access into the server. If the device is in the middle of an SSH session with the remote access server, we can then SSH into the device from our server if we want to do some deeper diagnostics on an issue with a device. We have separate HTTP server which each device queries to see if it needs to check in. So instead of having to do an ssh login each time to check if there's a firmware upgrade available, it just needs to do a HTTP GET to see if there's firmware available or if we want to check on its status.

The biggest issues we've had have been due to our network setup and handling upgrade both through the network and the USB stick. We are using "opkg upgrade --download-only" as the first step of the upgrade process to make sure that we don't do a partial upgrade. opkg-0.1.8 doesn't do --download-only for file:// sources; why download a file that is already on the filesystem? So I had to add a patch so it would download files from the USB stick. We also had an issue with DNS names because the server has a different name when the device finds it locally on our intranet then when it connects remotely, so we had to setup separate mirrors. Other than that it's been working pretty well.

Thanks Bryan, thanks really interesting. We do something similar with HTTP requests on various projects and that might well be something to apply here. I guess the limitation of this approach is that you are limited in when you can obtain your back channel to the device to the frequency with which they connect to the HTTP server. I do wonder just how much load there would be on a server handling a lot of SSH connections if almost all of those connections were just idling, sending keep alives now and again. I keep meaning to try to put together some metrics on that as, if I'm right and the load is low (as long as we ensure connections from devices are never synchronised, e.g. power failure) then I'd like to have an "always on" back-channel available

Interestesting about opkg too - thanks for that!

Quoted 1 line

I'd also love to hear other people's solutions to see if they have done something similar or came up with a different solution.

Me too

#11 lore raw

Alex,

Quoted 52 lines

-----Original Message-----
From: Alex J Lennon [mailto:[email protected]]
Sent: Saturday, March 08, 2014 4:18 AM
To: Bryan Evenson
Cc: [email protected]
Subject: Re: [yocto] Remote management of embedded devices

+

Quoted 16 lines · Bryan Evenson

Alex,

Quoted 14 lines

-----Original Message-----
From: [email protected] [mailto:yocto- [email protected]] On Behalf Of Alex J Lennon
Sent: Friday, March 07, 2014 7:30 AM
To: [email protected]
Subject: [yocto] Remote management of embedded devices

Hi,

I'm looking into remote management solutions for an upcoming headless mesh edge router running Poky. I think, at least in the initial rollout we're going to need something more than, say, a cron-based package update facility.

Quoted 5 lines
Quoted 4 lines

I'm currently thinking of going down the route of a cloud based server providing SSH port forwarding to the embedded devices, and then perhaps putting some scripting together on top of that to enable monitoring, configuration, and control.

This is similar to what we have running. We're using opkg for the package

management system and allow firmware upgrade through the USB stick or through remote access. We have an SSH server for remote access, and each device has its own private key for access into the server. If the device is in the middle of an SSH session with the remote access server, we can then SSH into the device from our server if we want to do some deeper diagnostics on an issue with a device. We have separate HTTP server which each device queries to see if it needs to check in. So instead of having to do an ssh login each time to check if there's a firmware upgrade available, it just needs to do a HTTP GET to see if there's firmware available or if we want to check on its status.

Quoted 2 lines

The biggest issues we've had have been due to our network setup and

handling upgrade both through the network and the USB stick. We are using "opkg upgrade --download-only" as the first step of the upgrade process to make sure that we don't do a partial upgrade. opkg-0.1.8 doesn't do -- download-only for file:// sources; why download a file that is already on the filesystem? So I had to add a patch so it would download files from the USB stick. We also had an issue with DNS names because the server has a different name when the device finds it locally on our intranet then when it connects remotely, so we had to setup separate mirrors. Other than that it's been working pretty well.

Quoted 11 lines

Thanks Bryan, thanks really interesting. We do something similar with HTTP requests on various projects and that might well be something to apply here. I guess the limitation of this approach is that you are limited in when you can obtain your back channel to the device to the frequency with which they connect to the HTTP server. I do wonder just how much load there would be on a server handling a lot of SSH connections if almost all of those connections were just idling, sending keep alives now and again. I keep meaning to try to put together some metrics on that as, if I'm right and the load is low (as long as we ensure connections from devices are never synchronised, e.g. power
failure) then I'd like to have an "always on" back-channel available

We considered the same thing. We weren't certain how scalable the idle SSH connection would be, as every device doesn't need to be connected at any given time. Depends on how many devices you plan on deploying, how many you think need to be connected at any given time, and what kind of response time you need. In our case, an HTTP request followed by the SSH connection if needed was a better solution.

Regards,
Bryan