Install Fuego Release Test dependencies
cover Install Fuego Release Test dependencies
Hello, Tim.
I was talking to @barbieri and we were thinking that due to the issues that we're currently going through with a new repo that extends fuego-base (most probably due to docker version incompatibilities), it would be more productive if we kept working in two fronts from now on:
- Primay: One that's focused on the Fuego meta-test itself (using SeleniumHQ and Pexpect - for ftc commands)
- Secondary: Focused on the fuego-release-test - that extends fuego-base, install dependencies, and adds the tests through an overlay - and that might be used in the future for users to add their own tests/dependencies/toolchains to Fuego in what seems to be a more extendable/reusable way.
Under the scope of the Primary front:
- We wrote a new patch that only implements the minimal changes needed on Fuego for installing the test dependencies. You will see that the part that changes fuego is short, consisting of just a few additions.
- We also have a new patch that will actually add the new Functional.fuegotest to fuego-core, as usual, so that you can focus on reviewing/testing it - not being blocked by docker or environment issues. Currently there are just a few helper functions and tests implemented (employing Pexpect and SeleniumHq), but we plan to add many more during this week.
I'm sending those two patches to the list for review.
Thank you!
Guilherme Campos Camargo (1):
Install Fuego Release Test dependencies and add fuego-test board
Guilherme Iscaro (1):
Do not use httpredir.debian.org links.
Dockerfile | 50 +++++++++++++++++++++- fuego-host-scripts/docker-create-container.sh | 1 + .../docker-create-usb-privileged-container.sh | 1 + fuego-ro/boards/fuego-test.board | 13 ++++++ setup/apt/sources/fuego-debian-jessie.list | 4 +- 5 files changed, 66 insertions(+), 3 deletions(-) create mode 100644 fuego-ro/boards/fuego-test.board
Signature · 1 line
2.16.2
PATCH 1/2 Do not use httpredir.debian.org links.
From: Guilherme Iscaro <[email protected]>
httpredir.debian.org is known to have some availability issues and out-of-date mirrors [1]. To avoid problems during the image instalation use the default debian PPA servers.
Note: We were actually having trouble with the previous mirrors when trying to build the image from scratch. Did you notice any problems with them as well?
[1]: https://lists.debian.org/debian-project/2016/04/msg00012.html
--- Dockerfile | 2 +- setup/apt/sources/fuego-debian-jessie.list | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)
Dockerfile +1 -1
@@ -13,7 +13,7 @@ MAINTAINER [email protected] ARG DEBIAN_FRONTEND=noninteractive COPY setup/apt/sources/fuego-debian-jessie.list \ - /etc/apt/sources.list.d/fuego-debian-jessie.list + /etc/apt/sources.list RUN apt-get update && \ apt-get -yV install \ apt-utils \
setup/apt/sources/fuego-debian-jessie.list +2 -2
@@ -1,2 +1,2 @@ -deb http://httpredir.debian.org/debian jessie main non-free -deb http://httpredir.debian.org/debian jessie-updates main non-free +deb http://deb.debian.org/debian jessie main non-free +deb http://deb.debian.org/debian jessie-updates main non-free
Signature · 1 line
2.16.2
Re: [Fuego] [PATCH 1/2] Do not use httpredir.debian.org links.
Quoted 23 lines
-----Original Message-----
From: Guilherme Campos Camargo
httpredir.debian.org is known to have some availability issues and out-of-date mirrors [1]. To avoid problems during the image instalation use the default debian PPA servers.Note: We were actually having trouble with the previous mirrors when trying to build the image from scratch. Did you notice any problems with them as well?
[1]: https://lists.debian.org/debian-project/2016/04/msg00012.html
--- Dockerfile | 2 +- setup/apt/sources/fuego-debian-jessie.list | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)Dockerfile +0
-0
Dockerfile MAINTAINER [email protected] · line 16
Quoted 5 lines
Dockerfile +1
-1@@ -13,7 +13,7 @@ MAINTAINER [email protected] ARG DEBIAN_FRONTEND=noninteractive COPY setup/apt/sources/fuego-debian-jessie.list \ - /etc/apt/sources.list.d/fuego-debian-jessie.list + /etc/apt/sources.list
You are overwriting the main sources.list, instead of just added additional sources (in the .d) directory. This ends up eliminating a line:
deb http://security.debian.org/ Jessie/updates main that is in the current sources.list file.
Is this what is intended??
setup/apt/sources/fuego-debian-jessie.list line 2
Quoted 13 lines
Dockerfile +0
-0RUN apt-get update && \ apt-get -yV install \ apt-utils \setup/apt/sources/fuego-debian-jessie.list +2
-2@@ -1,2 +1,2 @@ -deb http://httpredir.debian.org/debian jessie main non-free -deb http://httpredir.debian.org/debian jessie-updates main non-free +deb http://deb.debian.org/debian jessie main non-free +deb http://deb.debian.org/debian jessie-updates main non-free
I'm OK with this change.
Quoted 2 lines
--
2.16.2
I'm applying, but may revert depending on answer to above.
This is in my 'next' branch.
-- Tim
Re: [Fuego] [PATCH 1/2] Do not use httpredir.debian.org links.
Hello, Tim.
Yes, that was intended. Sorry for not pointing that out in the commit message.
I just tested again, adding the deb http://security.debian.org/ jessie/updates main to /etc/apt/sources.list, and had the error below while Docker was building the image. See that apt-get fails to install some toolchains.
We couldn't find exactly why this was happening, but it makes us think that eventually the security repo and the tools repo are not sync'ed. By removing the security repo, this problem does not happen anymore.
``` Step 7/33 : RUN echo deb http://emdebian.org/tools/debian/ jessie main > /etc/apt/sources.list.d/crosstools.list && curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.[0/1981] t-key add - && dpkg --add-architecture armhf && apt-get update && apt-get -yV install binutils-arm-linux-gnueabihf cpp-arm-linux-gnueabihf crossbuild-essenti al-armhf gcc-arm-linux-gnueabihf && rm -rf /var/lib/apt/lists/* ---> Running in bfddc5d95ba3 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2398 100 2398 0 0 780 0 0:00:03 0:00:03 --:--:-- 781 OK Get:1 http://security.debian.org jessie/updates InRelease [63.1 kB] Get:2 http://emdebian.org jessie InRelease [30.7 kB] Get:3 http://security.debian.org jessie/updates/main amd64 Packages [632 kB] Ign http://deb.debian.org jessie InRelease Get:4 http://deb.debian.org jessie-updates InRelease [145 kB] Get:5 http://deb.debian.org jessie Release.gpg [2434 B] Get:6 http://security.debian.org jessie/updates/main armhf Packages [606 kB] Get:7 http://deb.debian.org jessie Release [148 kB] Get:8 http://emdebian.org jessie/main amd64 Packages [17.2 kB] Get:9 http://emdebian.org jessie/main armhf Packages [8217 B] Get:10 http://deb.debian.org jessie-updates/main amd64 Packages [23.1 kB] Get:11 http://deb.debian.org jessie-updates/non-free amd64 Packages [449 B] Get:12 http://deb.debian.org jessie-updates/main armhf Packages [23.1 kB] Get:13 http://deb.debian.org jessie-updates/non-free armhf Packages [450 B] Get:14 http://deb.debian.org jessie/main amd64 Packages [9064 kB] Get:15 http://deb.debian.org jessie/non-free amd64 Packages [101 kB] Get:16 http://deb.debian.org jessie/main armhf Packages [8864 kB] Get:17 http://deb.debian.org jessie/non-free armhf Packages [74.9 kB] Fetched 19.8 MB in 32s (612 kB/s) Reading package lists... Reading package lists... Building dependency tree... Reading state information... Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: crossbuild-essential-armhf : Depends: g++-arm-linux-gnueabihf (>= 4.9.1-1) but it is not going to be installed gcc-arm-linux-gnueabihf : Depends: gcc-4.9-arm-linux-gnueabihf (>= 4.9.2-10) but it is not going to be installed E: Unable to correct problems, you have held broken packages. The command '/bin/sh -c echo deb http://emdebian.org/tools/debian/ jessie main > /etc/apt/sources.list.d/crosstools.list && curl http://emdebian.org/tools/debian/emdebian-toolchain-archive. key | apt-key add - && dpkg --add-architecture armhf && apt-get update && apt-get -yV install binutils-arm-linux-gnueabihf cpp-arm-linux-gnueabihf crossbuild -essential-armhf gcc-arm-linux-gnueabihf && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 100 ```
I'll take the change to take another look into it. Will let you know If I discover something.
Quoted 60 lines · On Thu, Mar 8, 2018 at 5:56 PM, <[email protected]> wrote:
Quoted 23 lines
-----Original Message-----
From: Guilherme Campos Camargo
httpredir.debian.org is known to have some availability issues and out-of-date mirrors [1]. To avoid problems during the image instalation use the default debian PPA servers.Note: We were actually having trouble with the previous mirrors when trying to build the image from scratch. Did you notice any problems with them as well?
[1]: https://lists.debian.org/debian-project/2016/04/msg00012.html
--- Dockerfile | 2 +- setup/apt/sources/fuego-debian-jessie.list | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)Dockerfile +0
-0Dockerfile MAINTAINER [email protected] · line 16
Quoted 5 lines
Dockerfile +1
-1@@ -13,7 +13,7 @@ MAINTAINER [email protected] ARG DEBIAN_FRONTEND=noninteractive COPY setup/apt/sources/fuego-debian-jessie.list \ - /etc/apt/sources.list.d/fuego-debian-jessie.list + /etc/apt/sources.listYou are overwriting the main sources.list, instead of just added additional sources (in the .d) directory. This ends up eliminating a line:
deb http://security.debian.org/ Jessie/updates main that is in the current sources.list file.Is this what is intended??
setup/apt/sources/fuego-debian-jessie.list line 2
Quoted 13 lines
Dockerfile +0
-0RUN apt-get update && \ apt-get -yV install \ apt-utils \setup/apt/sources/fuego-debian-jessie.list +2
-2@@ -1,2 +1,2 @@ -deb http://httpredir.debian.org/debian jessie main non-free -deb http://httpredir.debian.org/debian jessie-updates main non-free +deb http://deb.debian.org/debian jessie main non-free +deb http://deb.debian.org/debian jessie-updates main non-freeI'm OK with this change.
Quoted 2 lines
--
2.16.2I'm applying, but may revert depending on answer to above.
This is in my 'next' branch.
-- Tim
PATCH 2/2 Install Fuego Release Test dependencies and add fuego-test board
In reply to Guilherme Campos CamargoA new session has been added to the Dockerfile, mainly responsible for installing the Docker binary, pexpect and seleniumHQ (that are needed for Fuego to test a new Fuego release).
Also, a fuego-test board (a simple loopback that points to localhost) has been added.
The create scripts have also been modified to allow mapping the hosts' dockerd socket into the container. By doing this we allow Fuego and the host to share the same docker daemon.
--- Dockerfile | 48 ++++++++++++++++++++++ fuego-host-scripts/docker-create-container.sh | 1 + .../docker-create-usb-privileged-container.sh | 1 + fuego-ro/boards/fuego-test.board | 13 ++++++ 4 files changed, 63 insertions(+) create mode 100644 fuego-ro/boards/fuego-test.board
Dockerfile +48 -0
@@ -106,6 +106,54 @@ RUN curl -L -O ${JENKINS_URL} && \ dpkg -i jenkins_${JENKINS_VERSION}_all.deb && \ rm jenkins_${JENKINS_VERSION}_all.deb + +# ============================================================================== +# Install Fuego Release Test Dependencies +# ============================================================================== + +# TODO: This session should be moved to a separate Dockerfile in the future, +# that simply extends a fuego-base image and compiles a Fuego that's capable of +# testing itself. + +# Install Dependencies +RUN apt-get update && \ + apt-get -yV install \ + apt-transport-https \ + ca-certificates \ + chromium \ + curl \ + gnupg2 \ + python3 \ + python3-pip \ + software-properties-common && \ + pip3 install \ + docker \ + pexpect \ + selenium + +# Install Docker +RUN curl -fsSL https://download.docker.com/linux/$(source /etc/os-release; \ + echo "$ID")/gpg | sudo apt-key add - && \ + add-apt-repository \ + "deb [arch=amd64] https://download.docker.com/linux/$(\ + source /etc/os-release; echo "$ID") $(lsb_release -cs) stable" && \ + apt-get update && \ + apt-get -yV install \ + docker-ce + +# Install Chrome Driver for SeleniumHQ +RUN CHROME_DRIVER_VERSION=$(curl --silent --fail \ + https://chromedriver.storage.googleapis.com/LATEST_RELEASE) && \ + curl https://chromedriver.storage.googleapis.com/$(\ + echo ${CHROME_DRIVER_VERSION})/chromedriver_linux64.zip \ + -o chrome-driver.zip && \ + unzip chrome-driver.zip -d /usr/local/bin && rm chrome-driver.zip && \ + chmod +x /usr/local/bin/chromedriver + +# Setting jenkins as a sudoer. Needed for accessing the dockerd socket. +RUN echo "jenkins ALL = (root) NOPASSWD:ALL" >> /etc/sudoers + + # ============================================================================== # Post installation # ==============================================================================
fuego-host-scripts/docker-create-container.sh +1 -0
@@ -29,6 +29,7 @@ sed -i "s/docker_jenkins_uid=1000/docker_jenkins_uid=$uid/" $DIR/../fuego-ro/con sed -i "s/docker_jenkins_gid=500/docker_jenkins_gid=$gid/" $DIR/../fuego-ro/conf/fuego.conf sudo docker create -it --name ${DOCKERCONTAINER} \ + -v /var/run/docker.sock:/var/run/docker.sock \ -v /boot:/boot:ro \ -v $DIR/../fuego-rw:/fuego-rw \ -v $DIR/../fuego-ro:/fuego-ro:ro \
fuego-host-scripts/docker-create-usb-privileged-container.sh +1 -0
@@ -30,6 +30,7 @@ sed -i "s/docker_jenkins_gid=500/docker_jenkins_gid=$gid/" $DIR/../fuego-ro/conf sudo docker create -it --name ${DOCKERCONTAINER} \ --privileged \ + -v /var/run/docker.sock:/var/run/docker.sock \ -v /boot:/boot:ro \ -v /dev/bus/usb:/dev/bus/usb \ -v /dev/ttyACM0:/dev/ttyACM0 \
fuego-ro/boards/fuego-test.board new file +13 -0
@@ -0,0 +1,13 @@ +inherit "base-board" +include "base-params" + +IPADDR=127.0.0.1 +SRV_IP=127.0.0.1 +#LOGIN="root" +#PASSWORD="root" +#SSH_KEY="path/to/id_rsa" +BOARD_TESTDIR="$FUEGO_RW/tests" +PLATFORM="x86_64" +TRANSPORT="local" +ARCHITECTURE="x86_64" +DISTRIB="nosyslogd.dist"
Signature · 1 line
2.16.2
Quoted 28 lines
-----Original Message-----
From: Guilherme Campos Camargo
I was talking to @barbieri and we were thinking that due to the issues that we're currently going through with a new repo that extends fuego-base (most
probably due to docker version incompatibilities), it would be more productive
if we kept working in two fronts from now on:- Primay: One that's focused on the Fuego meta-test itself (using SeleniumHQ
and Pexpect - for ftc commands)- Secondary: Focused on the fuego-release-test - that extends fuego-base, install dependencies, and adds the tests through an overlay - and that might be used in the future for users to add their own tests/dependencies/toolchains to Fuego in what seems to be a more extendable/reusable way.
Under the scope of the Primary front:
- We wrote a new patch that only implements the minimal changes needed on Fuego
for installing the test dependencies. You will see that the part that changes fuego is short, consisting of just a few additions.
Quoted 8 lines
- We also have a new patch that will actually add the new Functional.fuegotest
to fuego-core, as usual, so that you can focus on reviewing/testing it - not being blocked by docker or environment issues. Currently there are just a few
helper functions and tests implemented (employing Pexpect and SeleniumHq),
but we plan to add many more during this week.
I think it is a very good ideas to separate these activities. Sorry I got behind a bit on my patch review. I'll look at these patches today.
Thanks,
-- Tim