Skip to contentSkip to conversation

RFC: layer index docker fixes

layerindex-web Paul Eggleton 2 participants 13 messages Jul 10, 2018 – Oct 2, 2018

lore
Share & tools
lore
#0 lore raw

Hi folks

I've been trying to get the OE layer index example docker setup in shape, based upon improvements from myself, Michael and Konrad. For the moment I've taken most of what Michael and I did (with some fixes and tweaks, and broken up into smaller patches) and a few pieces of the Dockerfile from Konrad's recent patch and put it here:

http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log/?h=paule/dockerfixes

Konrad, I know you were a little bit more ambitious with your changes but I wanted to get something closer to what Michael is using and then we can build upon it. In particular I haven't yet explored docker-compose as you have, although it looks like it would make deploying all of this a bit easier.

Let me know what you think.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#1 lore raw
Quoted 8 lines · Paul Eggleton

Hi folks

I've been trying to get the OE layer index example docker setup in shape, based upon improvements from myself, Michael and Konrad. For the moment I've taken most of what Michael and I did (with some fixes and tweaks, and broken up into smaller patches) and a few pieces of the Dockerfile from Konrad's recent patch and put it here:

Quoted 8 lines

http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log/?h=paule/dockerfixes

Konrad, I know you were a little bit more ambitious with your changes but I wanted to get something closer to what Michael is using and then we can build upon it. In particular I haven't yet explored docker-compose as you have, although it looks like it would make deploying all of this a bit easier.

Let me know what you think.

Hello Paul,

I have attached a patch to reduce the layerindex-web image size further. Docker image layers are like git objects and RUN commands can only hide files, not remove them once they have been committed. The pip upgrade caused the following 'pip install' to fail and was removed.

I recommend docker-compose because much of the information in docker/README can be captured in a compose yaml file. It handles the network setup and makes managing shared volumes much simpler.

Signature · 1 line
Konrad Scherer, MTS, Linux Products Group, Wind River

From 22274ea8298b4cb0c8ea3e545367a9c2c4c379c4 Mon Sep 17 00:00:00 2001
From: Konrad Scherer <[email protected]>
Date: Wed, 11 Jul 2018 12:19:15 -0400
Subject: [PATCH] Dockerfile: Reduce image size by merging RUN stages together

Docker commits changes after each RUN stage and commands that delete files must part of the sane RUN command to actually reduce image size.

Debian stretch contains pip 9.0.3 and the upgrade to 10 caused pip install to fail.

---
 Dockerfile | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)
Dockerfile +12 -12
index 523280d..0aff2ec 100644
@@ -12,8 +12,8 @@ ENV PYTHONUNBUFFERED=1 \
 #ENV https_proxy https://your.proxy.server:port
 
 COPY requirements.txt /
-RUN apt-get update
-RUN apt-get install -y --no-install-recommends \
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends \
 	autoconf \
 	g++ \
 	gcc \
@@ -31,20 +31,20 @@ RUN apt-get install -y --no-install-recommends \
 	locales \
 	netcat-openbsd \
 	curl \
-	vim
-RUN echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen \
+	vim \
+    && echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen \
 	&& locale-gen en_US.UTF-8 \
-	&& update-locale
-RUN pip install --upgrade pip
-RUN pip3 install gunicorn
-RUN pip install setuptools
-RUN pip3 install setuptools
-RUN pip install -r /requirements.txt
-RUN pip3 install -r /requirements.txt
-RUN apt-get purge -y autoconf g++ gcc make python3-dev libjpeg-dev libmariadbclient-dev \
+	&& update-locale \
+    && pip3 install gunicorn \
+    && pip install setuptools \
+    && pip3 install setuptools \
+    && pip install -r /requirements.txt \
+    && pip3 install -r /requirements.txt \
+    && apt-get purge -y autoconf g++ gcc make python-dev python3-dev libjpeg-dev libmariadbclient-dev \
 	&& apt-get autoremove -y \
 	&& rm -rf /var/lib/apt/lists/* \
 	&& apt-get clean
+
 COPY . /opt/layerindex
 COPY docker/settings.py /opt/layerindex/settings.py
 COPY docker/refreshlayers.sh /opt/refreshlayers.sh
Signature · 1 line
2.18.0
#2 lore raw

Hi Konrad

Quoted 4 lines · CEST Konrad Scherer

I have attached a patch to reduce the layerindex-web image size further. Docker image layers are like git objects and RUN commands can only hide files, not remove them once they have been committed. The pip upgrade caused the following 'pip install' to fail and was removed.

Right, overall you don't save any storage space with separate stages - but the downside of merging these is that you lose the ability to incrementally update the container within those stages; any time anything before changes the whole block will need to be re-run. Having said that we still have the separation between requirements.txt and everything else which is probably sufficient.

Oddly I did not see a failure from pip when I built this yesterday, but I guess it's probably not strictly necessary to upgrade it anyway.

I've pushed your patch to the paule/dockerfixes branch.

Quoted 3 lines

I recommend docker-compose because much of the information in docker/README can be captured in a compose yaml file. It handles the network setup and makes managing shared volumes much simpler.

Right, we'd just need to make sure everything we now have in the README is covered by the compose file (everything that's practical to cover, anyway).

Cheers,
Pau;

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#3 lore raw
Quoted 14 lines · Paul Eggleton

Hi Konrad

Quoted 4 lines · CEST Konrad Scherer

I have attached a patch to reduce the layerindex-web image size further. Docker image layers are like git objects and RUN commands can only hide files, not remove them once they have been committed. The pip upgrade caused the following 'pip install' to fail and was removed.

Right, overall you don't save any storage space with separate stages - but the downside of merging these is that you lose the ability to incrementally update the container within those stages; any time anything before changes the whole block will need to be re-run. Having said that we still have the separation between requirements.txt and everything else which is probably sufficient.

Quoted 4 lines

Oddly I did not see a failure from pip when I built this yesterday, but I guess it's probably not strictly necessary to upgrade it anyway.

I've pushed your patch to the paule/dockerfixes branch.

Thanks, I just noticed that the image doesn't contain git. Was this deliberate or an oversight?

Quoted 6 lines
Quoted 3 lines

I recommend docker-compose because much of the information in docker/README can be captured in a compose yaml file. It handles the network setup and makes managing shared volumes much simpler.

Right, we'd just need to make sure everything we now have in the README is covered by the compose file (everything that's practical to cover, anyway).

I am heading out on vacation soon. I will rework my docker-compose configuration with the new image and make some documentation when I get back.

Signature · 1 line
Konrad Scherer, MTS, Linux Products Group, Wind River
#4 lore raw
Quoted 2 lines · CEST you

I just noticed that the image doesn't contain git. Was this deliberate or an oversight?

Ah, no, it wasn't deliberate - perhaps it was there earlier with buildpack- deps? It doesn't appear that we ever explicitly had it in there. I've added it and pushed the branch again.

Quoted 11 lines
Quoted 7 lines
Quoted 3 lines

I recommend docker-compose because much of the information in docker/README can be captured in a compose yaml file. It handles the network setup and makes managing shared volumes much simpler.

Right, we'd just need to make sure everything we now have in the README is covered by the compose file (everything that's practical to cover , anyway).

I am heading out on vacation soon. I will rework my docker-compose configuration with the new image and make some documentation when I get back.

OK great, thanks.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#5 lore raw
Quoted 7 lines · Paul Eggleton
Quoted 2 lines · CEST you

I just noticed that the image doesn't contain git. Was this deliberate or an oversight?

Ah, no, it wasn't deliberate - perhaps it was there earlier with buildpack- deps? It doesn't appear that we ever explicitly had it in there. I've added it and pushed the branch again.

I recently noticed that if gcc is removed in the 'apt-get purge' step, the bitbake sanity checker will fail due to missing 'gcc' when doing a layer update.

Signature · 1 line
Konrad Scherer, MTS, Linux Products Group, Wind River
#6 lore raw
Quoted 5 lines · CEST Konrad Scherer
Quoted 4 lines · Paul Eggleton
Quoted 2 lines · CEST you

I just noticed that the image doesn't contain git. Was this deliberate or an oversight?

Quoted 7 lines
Quoted 3 lines

Ah, no, it wasn't deliberate - perhaps it was there earlier with buildpack-deps? It doesn't appear that we ever explicitly had it in there. I've added it and pushed the branch again.

I recently noticed that if gcc is removed in the 'apt-get purge' step, the bitbake sanity checker will fail due to missing 'gcc' when doing a layer update.

Hmm, well that's a different problem - those sanity checks shouldn't be running. We don't need those tools since we're not building anything. I will have a look into this.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#7 lore raw
Quoted 8 lines · CEST Paul Eggleton
Quoted 3 lines · CEST Konrad Scherer

I recently noticed that if gcc is removed in the 'apt-get purge' step, the bitbake sanity checker will fail due to missing 'gcc' when doing a layer update.

Hmm, well that's a different problem - those sanity checks shouldn't be running. We don't need those tools since we're not building anything. I will have a look into this.

I was wrong - I remembered that we explicitly set HOSTTOOLS = "gcc" in our own conf/local.conf because there are other places in the metadata where we check for gcc which we do not override, and thus it's expected to be there. I've fixed and pushed the branch again, thanks for pointing out the issue (this time I actually tested running an update ;).

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#8 lore raw
Quoted 10 lines · Paul Eggleton
Quoted 8 lines · CEST Paul Eggleton
Quoted 3 lines · CEST Konrad Scherer

I recently noticed that if gcc is removed in the 'apt-get purge' step, the bitbake sanity checker will fail due to missing 'gcc' when doing a layer update.

Hmm, well that's a different problem - those sanity checks shouldn't be running. We don't need those tools since we're not building anything. I will have a look into this.

Quoted 5 lines

I was wrong - I remembered that we explicitly set HOSTTOOLS = "gcc" in our own conf/local.conf because there are other places in the metadata where we check for gcc which we do not override, and thus it's expected to be there. I've fixed and pushed the branch again, thanks for pointing out the issue (this time I actually tested running an update ;).

I am also guilty of not running a full update before I sent the last patch. I leave on vacation tomorrow and will work on a docker-compose setup when I get back.

Have a great summer!

Signature · 1 line
Konrad Scherer, MTS, Linux Products Group, Wind River
#9 lore raw

Hi folks

I know Konrad promised to update the docker-compose setup to be closer to what we'd done recently with the docker setup, but I needed something urgently and wanted to learn how to use docker-compose anyway, so I've put this together:

http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log/?h=paule/docker-compose

Let me know what you think - it works well enough for me here, although the setup commands are now a bit ugly on their own.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#10 lore raw
Quoted 11 lines · NZST Paul Eggleton

Hi folks

I know Konrad promised to update the docker-compose setup to be closer to what we'd done recently with the docker setup, but I needed something urgently and wanted to learn how to use docker-compose anyway, so I've put this together:

http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log/?h=paule/docker-compose

Let me know what you think - it works well enough for me here, although the setup commands are now a bit ugly on their own.

I realised this morning that "docker-compose run" can simplify a lot of the docker run commands, so I've just converted most of the commands in docker/README and pushed the branch again.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#11 lore raw
Quoted 6 lines · NZST Paul Eggleton
Quoted 5 lines · NZST Paul Eggleton

I know Konrad promised to update the docker-compose setup to be closer to what we'd done recently with the docker setup, but I needed something urgently and wanted to learn how to use docker-compose anyway, so I've put this together:

Quoted 8 lines
Quoted 4 lines

http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log/?h=paule/docker-compose

Let me know what you think - it works well enough for me here, although the setup commands are now a bit ugly on their own.

I realised this morning that "docker-compose run" can simplify a lot of the docker run commands, so I've just converted most of the commands in docker/README and pushed the branch again.

So, any comments?

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#12 lore raw
Quoted 9 lines · Paul Eggleton
Quoted 8 lines · NZST Paul Eggleton
Quoted 7 lines · NZST Paul Eggleton

I know Konrad promised to update the docker-compose setup to be closer to what we'd done recently with the docker setup, but I needed something urgently and wanted to learn how to use docker-compose anyway, so I've put this together:

http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log/?h=paule/docker-compose

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Let me know what you think - it works well enough for me here, although the setup commands are now a bit ugly on their own.

I realised this morning that "docker-compose run" can simplify a lot of the docker run commands, so I've just converted most of the commands in docker/README and pushed the branch again.

So, any comments?

Sorry for the delay. I just rebased our internal layerindex to the latest and made some updates to the Dockerfiles and other files.

https://github.com/WindRiver-OpenSourceLabs/layerindex-web/tree/rebase-20181002

Dockerfile:
- added the wheel package to avoid errors in pip package installation
- Used ADD --chown to add all the files with the correct permissions to avoid a second chown RUN step
- Used an entrypoint.sh script instead of CMD to handle things like db initialization, creating a superuser, overriding default configuration with environment variables and collecting static files. The same entrypoint is used to start either the celery-worker or gunicorn.

https://github.com/WindRiver-OpenSourceLabs/layerindex-web/blob/rebase-20181002/docker/entrypoint.sh

docker/docker-compose.yaml:
- separating the gunicorn and the celery-worker is a good idea
- the blacklabelops/nginx image generates the nginx config from env variables which makes development easier. Changing an env var in docker-compose.yaml is easier than rebuilding the image or bind-mounting in a new version.
- One thing I didn't put in passing in the DB username and password because I am using this as a test instance, not a production one. Should be a simple change.

I had problems setting up a reverse proxy for the layerindex-web app when using a subpath (ie. http://<host>/li/). In urls.py, the leading slash on the last redirect works as a redirect to an absolute path and breaks the redirect when using a subpath . Fortunately the fix is simple and doesn't break usage without a subpath.

https://github.com/WindRiver-OpenSourceLabs/layerindex-web/commit/4586ff3f4fa477daa6e0bee6c6e9e0602105c8d2#diff-de6dd4b4c889fe0882cfd3f6df5aa451L29

Thanks!

Signature · 1 line
Konrad Scherer, MTS, Linux Products Group, Wind River