Skip to contentSkip to conversation

Install Fuego Release Test dependencies

PATCH In review Guilherme Campos Camargo 2 participants 6 messages Mar 5, 2018 – Mar 8, 2018

lore
Share & tools
lore
#0 lore raw

cover Install Fuego Release Test dependencies

Hello, Tim.

I was talking to @barbieri and we were thinking that due to the issues that we're currently going through with a new repo that extends fuego-base (most probably due to docker version incompatibilities), it would be more productive if we kept working in two fronts from now on:

- Primay: One that's focused on the Fuego meta-test itself (using SeleniumHQ and Pexpect - for ftc commands)

- Secondary: Focused on the fuego-release-test - that extends fuego-base, install dependencies, and adds the tests through an overlay - and that might be used in the future for users to add their own tests/dependencies/toolchains to Fuego in what seems to be a more extendable/reusable way.

Under the scope of the Primary front:

- We wrote a new patch that only implements the minimal changes needed on Fuego for installing the test dependencies. You will see that the part that changes fuego is short, consisting of just a few additions.

- We also have a new patch that will actually add the new Functional.fuegotest to fuego-core, as usual, so that you can focus on reviewing/testing it - not being blocked by docker or environment issues. Currently there are just a few helper functions and tests implemented (employing Pexpect and SeleniumHq), but we plan to add many more during this week.

I'm sending those two patches to the list for review.

Thank you!

Guilherme Campos Camargo (1):
Install Fuego Release Test dependencies and add fuego-test board

Guilherme Iscaro (1):
Do not use httpredir.debian.org links.

 Dockerfile                                         | 50 +++++++++++++++++++++-
 fuego-host-scripts/docker-create-container.sh      |  1 +
 .../docker-create-usb-privileged-container.sh      |  1 +
 fuego-ro/boards/fuego-test.board                   | 13 ++++++
 setup/apt/sources/fuego-debian-jessie.list         |  4 +-
 5 files changed, 66 insertions(+), 3 deletions(-)
 create mode 100644 fuego-ro/boards/fuego-test.board
Signature · 1 line
2.16.2
#1 lore raw

PATCH 1/2 Do not use httpredir.debian.org links.

From: Guilherme Iscaro <[email protected]>

httpredir.debian.org is known to have some availability issues and out-of-date mirrors [1]. To avoid problems during the image instalation use the default debian PPA servers.

Note: We were actually having trouble with the previous mirrors when trying to build the image from scratch. Did you notice any problems with them as well?

[1]: https://lists.debian.org/debian-project/2016/04/msg00012.html

---
 Dockerfile                                 | 2 +-
 setup/apt/sources/fuego-debian-jessie.list | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)
Dockerfile +1 -1
index cc79351..aa52fd8 100644
@@ -13,7 +13,7 @@ MAINTAINER [email protected]
 ARG DEBIAN_FRONTEND=noninteractive
 
 COPY setup/apt/sources/fuego-debian-jessie.list \
-        /etc/apt/sources.list.d/fuego-debian-jessie.list
+        /etc/apt/sources.list
 RUN apt-get update && \
     apt-get -yV install \
         apt-utils \
setup/apt/sources/fuego-debian-jessie.list +2 -2
index d224f15..02f210c 100644
@@ -1,2 +1,2 @@
-deb http://httpredir.debian.org/debian jessie main non-free
-deb http://httpredir.debian.org/debian jessie-updates main non-free
+deb http://deb.debian.org/debian jessie main non-free
+deb http://deb.debian.org/debian jessie-updates main non-free
Signature · 1 line
2.16.2
#2 lore raw

Re: [Fuego] [PATCH 1/2] Do not use httpredir.debian.org links.

Quoted 23 lines

-----Original Message-----
From: Guilherme Campos Camargo
httpredir.debian.org is known to have some availability issues and out-of-date mirrors [1]. To avoid problems during the image instalation use the default debian PPA servers.

Note: We were actually having trouble with the previous mirrors when trying to build the image from scratch. Did you notice any problems with them as well?

[1]: https://lists.debian.org/debian-project/2016/04/msg00012.html

---
 Dockerfile                                 | 2 +-
 setup/apt/sources/fuego-debian-jessie.list | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)
Dockerfile +0 -0
index cc79351..aa52fd8 100644

Dockerfile MAINTAINER [email protected] · line 16

Quoted 5 lines
Dockerfile +1 -1
@@ -13,7 +13,7 @@ MAINTAINER [email protected]
 ARG DEBIAN_FRONTEND=noninteractive
 
 COPY setup/apt/sources/fuego-debian-jessie.list \
-        /etc/apt/sources.list.d/fuego-debian-jessie.list
+        /etc/apt/sources.list

You are overwriting the main sources.list, instead of just added additional sources (in the .d) directory. This ends up eliminating a line:
deb http://security.debian.org/ Jessie/updates main that is in the current sources.list file.

Is this what is intended??

setup/apt/sources/fuego-debian-jessie.list line 2

Quoted 13 lines
Dockerfile +0 -0
 RUN apt-get update && \
     apt-get -yV install \
         apt-utils \
setup/apt/sources/fuego-debian-jessie.list +2 -2
index d224f15..02f210c 100644
@@ -1,2 +1,2 @@
-deb http://httpredir.debian.org/debian jessie main non-free
-deb http://httpredir.debian.org/debian jessie-updates main non-free
+deb http://deb.debian.org/debian jessie main non-free
+deb http://deb.debian.org/debian jessie-updates main non-free

I'm OK with this change.

Quoted 2 lines

--
2.16.2

I'm applying, but may revert depending on answer to above.

This is in my 'next' branch.
-- Tim

#3 lore raw

Re: [Fuego] [PATCH 1/2] Do not use httpredir.debian.org links.

Hello, Tim.

Yes, that was intended. Sorry for not pointing that out in the commit message.

I just tested again, adding the deb http://security.debian.org/ jessie/updates main to /etc/apt/sources.list, and had the error below while Docker was building the image. See that apt-get fails to install some toolchains.

We couldn't find exactly why this was happening, but it makes us think that eventually the security repo and the tools repo are not sync'ed. By removing the security repo, this problem does not happen anymore.

```
Step 7/33 : RUN echo deb http://emdebian.org/tools/debian/ jessie main >
/etc/apt/sources.list.d/crosstools.list &&     curl
http://emdebian.org/tools/debian/emdebian-toolchain-archive.[0/1981]
t-key add - &&     dpkg --add-architecture armhf &&     apt-get update &&
   apt-get -yV install         binutils-arm-linux-gnueabihf
 cpp-arm-linux-gnueabihf         crossbuild-essenti
al-armhf         gcc-arm-linux-gnueabihf &&     rm -rf /var/lib/apt/lists/*
 ---> Running in bfddc5d95ba3
  % Total    % Received % Xferd  Average Speed   Time    Time     Time
Current
                                 Dload  Upload   Total   Spent    Left
Speed
100  2398  100  2398    0     0    780      0  0:00:03  0:00:03 --:--:--
 781
OK
Get:1 http://security.debian.org jessie/updates InRelease [63.1 kB]
Get:2 http://emdebian.org jessie InRelease [30.7 kB]
Get:3 http://security.debian.org jessie/updates/main amd64 Packages [632 kB]
Ign http://deb.debian.org jessie InRelease
Get:4 http://deb.debian.org jessie-updates InRelease [145 kB]
Get:5 http://deb.debian.org jessie Release.gpg [2434 B]
Get:6 http://security.debian.org jessie/updates/main armhf Packages [606 kB]
Get:7 http://deb.debian.org jessie Release [148 kB]
Get:8 http://emdebian.org jessie/main amd64 Packages [17.2 kB]
Get:9 http://emdebian.org jessie/main armhf Packages [8217 B]
Get:10 http://deb.debian.org jessie-updates/main amd64 Packages [23.1 kB]
Get:11 http://deb.debian.org jessie-updates/non-free amd64 Packages [449 B]
Get:12 http://deb.debian.org jessie-updates/main armhf Packages [23.1 kB]
Get:13 http://deb.debian.org jessie-updates/non-free armhf Packages [450 B]
Get:14 http://deb.debian.org jessie/main amd64 Packages [9064 kB]
Get:15 http://deb.debian.org jessie/non-free amd64 Packages [101 kB]
Get:16 http://deb.debian.org jessie/main armhf Packages [8864 kB]
Get:17 http://deb.debian.org jessie/non-free armhf Packages [74.9 kB]
Fetched 19.8 MB in 32s (612 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 crossbuild-essential-armhf : Depends: g++-arm-linux-gnueabihf (>= 4.9.1-1)
but it is not going to be installed
 gcc-arm-linux-gnueabihf : Depends: gcc-4.9-arm-linux-gnueabihf (>=
4.9.2-10) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
The command '/bin/sh -c echo deb http://emdebian.org/tools/debian/ jessie
main > /etc/apt/sources.list.d/crosstools.list &&     curl
http://emdebian.org/tools/debian/emdebian-toolchain-archive.
key | apt-key add - &&     dpkg --add-architecture armhf &&     apt-get
update &&     apt-get -yV install         binutils-arm-linux-gnueabihf
   cpp-arm-linux-gnueabihf         crossbuild
-essential-armhf         gcc-arm-linux-gnueabihf &&     rm -rf
/var/lib/apt/lists/*' returned a non-zero code: 100

```

I'll take the change to take another look into it. Will let you know If I discover something.

Quoted 60 lines · On Thu, Mar 8, 2018 at 5:56 PM, <[email protected]> wrote:
Quoted 23 lines

-----Original Message-----
From: Guilherme Campos Camargo
httpredir.debian.org is known to have some availability issues and out-of-date mirrors [1]. To avoid problems during the image instalation use the default debian PPA servers.

Note: We were actually having trouble with the previous mirrors when trying to build the image from scratch. Did you notice any problems with them as well?

[1]: https://lists.debian.org/debian-project/2016/04/msg00012.html

---
 Dockerfile                                 | 2 +-
 setup/apt/sources/fuego-debian-jessie.list | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)
Dockerfile +0 -0
index cc79351..aa52fd8 100644

Dockerfile MAINTAINER [email protected] · line 16

Quoted 5 lines
Dockerfile +1 -1
@@ -13,7 +13,7 @@ MAINTAINER [email protected]
 ARG DEBIAN_FRONTEND=noninteractive
 
 COPY setup/apt/sources/fuego-debian-jessie.list \
-        /etc/apt/sources.list.d/fuego-debian-jessie.list
+        /etc/apt/sources.list

You are overwriting the main sources.list, instead of just added additional sources (in the .d) directory. This ends up eliminating a line:
deb http://security.debian.org/ Jessie/updates main that is in the current sources.list file.

Is this what is intended??

setup/apt/sources/fuego-debian-jessie.list line 2

Quoted 13 lines
Dockerfile +0 -0
 RUN apt-get update && \
     apt-get -yV install \
         apt-utils \
setup/apt/sources/fuego-debian-jessie.list +2 -2
index d224f15..02f210c 100644
@@ -1,2 +1,2 @@
-deb http://httpredir.debian.org/debian jessie main non-free
-deb http://httpredir.debian.org/debian jessie-updates main non-free
+deb http://deb.debian.org/debian jessie main non-free
+deb http://deb.debian.org/debian jessie-updates main non-free

I'm OK with this change.

Quoted 2 lines

--
2.16.2

I'm applying, but may revert depending on answer to above.

This is in my 'next' branch.
-- Tim

#4 lore raw

PATCH 2/2 Install Fuego Release Test dependencies and add fuego-test board

In reply to Guilherme Campos Camargo

A new session has been added to the Dockerfile, mainly responsible for installing the Docker binary, pexpect and seleniumHQ (that are needed for Fuego to test a new Fuego release).

Also, a fuego-test board (a simple loopback that points to localhost) has been added.

The create scripts have also been modified to allow mapping the hosts' dockerd socket into the container. By doing this we allow Fuego and the host to share the same docker daemon.

---
 Dockerfile                                         | 48 ++++++++++++++++++++++
 fuego-host-scripts/docker-create-container.sh      |  1 +
 .../docker-create-usb-privileged-container.sh      |  1 +
 fuego-ro/boards/fuego-test.board                   | 13 ++++++
 4 files changed, 63 insertions(+)
 create mode 100644 fuego-ro/boards/fuego-test.board
Dockerfile +48 -0
index aa52fd8..42b2fc8 100644
@@ -106,6 +106,54 @@ RUN curl -L -O ${JENKINS_URL} && \
     dpkg -i jenkins_${JENKINS_VERSION}_all.deb && \
     rm jenkins_${JENKINS_VERSION}_all.deb
 
+
+# ==============================================================================
+# Install Fuego Release Test Dependencies
+# ==============================================================================
+
+# TODO: This session should be moved to a separate Dockerfile in the future,
+# that simply extends a fuego-base image and compiles a Fuego that's capable of
+# testing itself.
+
+# Install Dependencies
+RUN apt-get update && \
+    apt-get -yV install \
+        apt-transport-https \
+        ca-certificates \
+        chromium \
+        curl \
+        gnupg2 \
+        python3 \
+        python3-pip \
+        software-properties-common && \
+    pip3 install \
+        docker \
+        pexpect \
+        selenium
+
+# Install Docker
+RUN curl -fsSL https://download.docker.com/linux/$(source /etc/os-release; \
+        echo "$ID")/gpg | sudo apt-key add - && \
+    add-apt-repository \
+        "deb [arch=amd64] https://download.docker.com/linux/$(\
+            source /etc/os-release; echo "$ID") $(lsb_release -cs) stable" && \
+    apt-get update && \
+    apt-get -yV install \
+        docker-ce
+
+# Install Chrome Driver for SeleniumHQ
+RUN CHROME_DRIVER_VERSION=$(curl --silent --fail \
+        https://chromedriver.storage.googleapis.com/LATEST_RELEASE) && \
+    curl https://chromedriver.storage.googleapis.com/$(\
+        echo ${CHROME_DRIVER_VERSION})/chromedriver_linux64.zip \
+            -o chrome-driver.zip && \
+    unzip chrome-driver.zip -d /usr/local/bin && rm chrome-driver.zip && \
+    chmod +x /usr/local/bin/chromedriver
+
+# Setting jenkins as a sudoer. Needed for accessing the dockerd socket.
+RUN echo "jenkins ALL = (root) NOPASSWD:ALL" >> /etc/sudoers
+
+
 # ==============================================================================
 # Post installation
 # ==============================================================================
fuego-host-scripts/docker-create-container.sh +1 -0
index 3d133bd..e837614 100755
@@ -29,6 +29,7 @@ sed -i "s/docker_jenkins_uid=1000/docker_jenkins_uid=$uid/" $DIR/../fuego-ro/con
 sed -i "s/docker_jenkins_gid=500/docker_jenkins_gid=$gid/" $DIR/../fuego-ro/conf/fuego.conf
 
 sudo docker create -it --name ${DOCKERCONTAINER} \
+    -v /var/run/docker.sock:/var/run/docker.sock \
     -v /boot:/boot:ro \
     -v $DIR/../fuego-rw:/fuego-rw \
     -v $DIR/../fuego-ro:/fuego-ro:ro \
fuego-host-scripts/docker-create-usb-privileged-container.sh +1 -0
index b314160..2329cbf 100755
@@ -30,6 +30,7 @@ sed -i "s/docker_jenkins_gid=500/docker_jenkins_gid=$gid/" $DIR/../fuego-ro/conf
 
 sudo docker create -it --name ${DOCKERCONTAINER} \
     --privileged \
+    -v /var/run/docker.sock:/var/run/docker.sock \
     -v /boot:/boot:ro \
     -v /dev/bus/usb:/dev/bus/usb \
     -v /dev/ttyACM0:/dev/ttyACM0 \
fuego-ro/boards/fuego-test.board new file +13 -0
new file mode 100644
index 0000000..185e0b4
@@ -0,0 +1,13 @@
+inherit "base-board"
+include "base-params"
+
+IPADDR=127.0.0.1
+SRV_IP=127.0.0.1
+#LOGIN="root"
+#PASSWORD="root"
+#SSH_KEY="path/to/id_rsa"
+BOARD_TESTDIR="$FUEGO_RW/tests"
+PLATFORM="x86_64"
+TRANSPORT="local"
+ARCHITECTURE="x86_64"
+DISTRIB="nosyslogd.dist"
Signature · 1 line
2.16.2
#5 lore raw
In reply to Guilherme Campos Camargo
Quoted 28 lines

-----Original Message-----
From: Guilherme Campos Camargo
I was talking to @barbieri and we were thinking that due to the issues that we're currently going through with a new repo that extends fuego-base (most
probably due to docker version incompatibilities), it would be more productive
if we kept working in two fronts from now on:

- Primay: One that's focused on the Fuego meta-test itself (using SeleniumHQ
and Pexpect - for ftc commands)

- Secondary: Focused on the fuego-release-test - that extends fuego-base, install dependencies, and adds the tests through an overlay - and that might be used in the future for users to add their own tests/dependencies/toolchains to Fuego in what seems to be a more extendable/reusable way.

Under the scope of the Primary front:

- We wrote a new patch that only implements the minimal changes needed on Fuego
for installing the test dependencies. You will see that the part that changes fuego is short, consisting of just a few additions.

Quoted 8 lines

- We also have a new patch that will actually add the new Functional.fuegotest
to fuego-core, as usual, so that you can focus on reviewing/testing it - not being blocked by docker or environment issues. Currently there are just a few
helper functions and tests implemented (employing Pexpect and SeleniumHq),
but we plan to add many more during this week.

I think it is a very good ideas to separate these activities. Sorry I got behind a bit on my patch review. I'll look at these patches today.

Thanks,
-- Tim