Skip to contentSkip to conversation

Docker related fixes

PATCH New Tim Orling 1 participant 7 messages Jan 15, 2024

lore
Share & tools
lore

Tim Orling

author
#0 lore raw

cover Docker related fixes

This series updates all the depedencies in requirements.txt, notably a fix for a GitPython vulnerability and fixes enabling latest mysqlclient (v2.2.1) wheel to be buildable.

Also included is a move to the mariadb:lts container in docker-compose, since in practice this is closer to what we are running in production and is the version we should be striving to be compatible with (unless upstream Django has breakage we cannot fix).

The following changes since commit b0af957ac734b76bc7cc2b45b0beb1cde74df0a6:

layerindex: improve updates for actual_branch (2024-01-03 13:33:37 -0800)

are available in the Git repository at:

https://github.com/moto-timo/layerindex-web docker-fixes
https://github.com/moto-timo/layerindex-web/tree/docker-fixes

Tim Orling (6):
dockersetup: use .zst not .zstd
Dockerfile: sort apt packages alphabetically
Dockerfile: fix mysqlclient==2.2.1 build
requirements.txt: bump all to latest
docker-compose.yml: mariadb:lts
docker-compose.yml: drop version '3'

 Dockerfile         | 31 ++++++++++++++++---------------
 docker-compose.yml |  3 +--
 dockersetup.py     |  4 ++--
 requirements.txt   | 14 +++++++-------
 4 files changed, 26 insertions(+), 26 deletions(-)
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH 1/6 dockersetup: use .zst not .zstd

The default file extension for zstd compression is .zst

---
 dockersetup.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
dockersetup.py +2 -2
index 56458d3..21be0da 100755
@@ -54,7 +54,7 @@ def get_args():
     parser.add_argument('-s', '--https-proxy', type=str, help='https proxy in the format http://<myproxy:port>', default=default_https_proxy, required=False)
     parser.add_argument('-S', '--socks-proxy', type=str, help='socks proxy in the format socks://myproxy:port>', default=default_socks_proxy, required=False)
     parser.add_argument('-N', '--no-proxy', type=str, help='Comma-separated list of hosts that should not be connected to via the proxy', default=default_no_proxy, required=False)
-    parser.add_argument('-d', '--databasefile', type=str, help='Location of your database file to import. Must be a .sql, .sql.gz or .sql.zstd file.', required=False)
+    parser.add_argument('-d', '--databasefile', type=str, help='Location of your database file to import. Must be a .sql, .sql.gz or .sql.zst file.', required=False)
     parser.add_argument('-e', '--email-host', type=str, help='Email host for sending messages (optionally with :port if not 25)', required=False)
     parser.add_argument('--email-user', type=str, help='User name to use when connecting to email host', required=False)
     parser.add_argument('--email-password', type=str, help='Password to use when connecting to email host', required=False)
@@ -808,7 +808,7 @@ if not args.update:
     # Import the user's supplied data
     if args.databasefile:
         filename, file_extension = os.path.splitext(args.databasefile)
-        if file_extension == ".zstd":
+        if file_extension == ".zst":
             return_code = subprocess.call("zstd -t %s > /dev/null 2>&1" % quote(args.databasefile), shell=True)
             if return_code == 0:
                 catcmd = 'zstdcat'
Signature · 1 line
2.34.1
author
#2 lore raw

PATCH 2/6 Dockerfile: sort apt packages alphabetically

In reply to Tim Orling

Make it is a bit easier to see what is installed by sorting the package list alphabetically.

---
 Dockerfile | 26 +++++++++++++-------------
 1 file changed, 13 insertions(+), 13 deletions(-)
Dockerfile +13 -13
index 02948b6..0f3afc4 100644
@@ -23,26 +23,26 @@ RUN DEBIAN_FRONTEND=noninteractive apt-get update \
         && update-locale \
     && apt-get install -y --no-install-recommends \
 	autoconf \
+	cpio \
+	curl \
 	g++ \
 	gcc \
-	make \
-	python2 \
-	python3-pip \
-	python3-mysqldb \
-	python3-dev \
-	python3-wheel \
-	zlib1g-dev \
+	git \
 	libfreetype6-dev \
 	libjpeg-dev \
 	libmariadb-dev-compat \
+	make \
 	netcat-openbsd \
-	curl \
-	wget \
-	git \
-	vim \
-	rpm2cpio \
+	python2 \
+	python3-dev \
+	python3-mysqldb \
+	python3-pip \
+	python3-wheel \
 	rpm \
-	cpio \
+	rpm2cpio \
+	vim \
+	wget \
+	zlib1g-dev \
     && echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen \
 	&& locale-gen en_US.UTF-8 \
 	&& update-locale \
Signature · 1 line
2.34.1
author
#3 lore raw

PATCH 3/6 Dockerfile: fix mysqlclient==2.2.1 build

In reply to Tim Orling

To build mysqlclient==2.2.1 we need to add default-libmysqlclient-dev and pkg-config

Drop conflicting libmariadb-dev-compat
Drop python3-mysqldb to allow requirements.txt to override version

Upgrade pip and setuptools before installing from requirements.txt as newer versions are required to properly build mysqlclient wheel.

---
 Dockerfile | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
Dockerfile +5 -4
index 0f3afc4..72f57d2 100644
@@ -30,12 +30,12 @@ RUN DEBIAN_FRONTEND=noninteractive apt-get update \
 	git \
 	libfreetype6-dev \
 	libjpeg-dev \
-	libmariadb-dev-compat \
+	default-libmysqlclient-dev \
 	make \
 	netcat-openbsd \
+	pkg-config \
 	python2 \
 	python3-dev \
-	python3-mysqldb \
 	python3-pip \
 	python3-wheel \
 	rpm \
@@ -46,10 +46,11 @@ RUN DEBIAN_FRONTEND=noninteractive apt-get update \
     && echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen \
 	&& locale-gen en_US.UTF-8 \
 	&& update-locale \
+    && pip3 install pip --upgrade \
+    && pip3 install setuptools --upgrade \
     && pip3 install gunicorn \
-    && pip3 install setuptools \
     && pip3 install -r /requirements.txt \
-    && apt-get purge -y autoconf g++ make python3-dev libjpeg-dev \
+    && apt-get purge -y autoconf g++ make python3-dev pkg-config libjpeg-dev \
 	&& apt-get autoremove -y \
 	&& rm -rf /var/lib/apt/lists/* \
 	&& apt-get clean
Signature · 1 line
2.34.1
author
#4 lore raw

PATCH 4/6 requirements.txt: bump all to latest

In reply to Tim Orling

Bump all the dependencies to latest, using
pur -r requirements.txt

We are now able to build the mysqlclient==2.2.1 package, so the only constraint is Django>=4.2,<4.3 to keep us pinned at the desired LTS version.

---
 requirements.txt | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
requirements.txt +7 -7
index 20bcd30..ac06cfb 100644
@@ -11,20 +11,20 @@ confusable-homoglyphs==3.2.0
 diff-match-patch==20230430
 Django>=4.2,<4.3
 django-appconf==1.0.6
-django-axes==6.2.0
+django-axes==6.3.0
 django-cors-headers==4.3.1
 django-ipware==6.0.3
 django-ranged-response==0.2.0
 django-registration==3.4
-django-reversion==5.0.9
+django-reversion==5.0.10
 django-reversion-compare==0.16.2
 django-simple-captcha==0.6.0
 djangorestframework==3.14.0
 gitdb==4.0.11
-GitPython==3.1.40
+GitPython==3.1.41
 kombu==5.3.4
-mysqlclient==2.1.1
-Pillow==10.1.0
+mysqlclient==2.2.1
+Pillow==10.2.0
 prompt-toolkit==3.0.43
 python-dateutil==2.8.2
 pytz==2023.3.post1
@@ -33,6 +33,6 @@ smmap==5.0.1
 soupsieve==2.5
 sqlparse==0.4.4
 typing_extensions==4.9.0
-tzdata==2023.3
+tzdata==2023.4
 vine==5.1.0
-wcwidth==0.2.12
+wcwidth==0.2.13
Signature · 1 line
2.34.1
author
#5 lore raw

PATCH 5/6 docker-compose.yml: mariadb:lts

In reply to Tim Orling

We should really be striving to be compatible with the latest mariadb LTS (currently 10.11).

---
 docker-compose.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
docker-compose.yml +1 -1
index e57f59d..8b43946 100644
@@ -1,7 +1,7 @@
 version: '3'
 services:
   layersdb:
-    image: mariadb:10.4
+    image: mariadb:lts
     command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci --wait_timeout=28800 --max_allowed_packet=128M
     environment:
      - "MYSQL_DATABASE=layersdb"
Signature · 1 line
2.34.1
author
#6 lore raw

PATCH 6/6 docker-compose.yml: drop version '3'

In reply to Tim Orling

Drop the obsolete version: '3' property, it is no longer needed since late 2020 (Docker Compose 1.27+) and is only informative:

https://nickjanetakis.com/blog/docker-tip-51-which-docker-compose-api-version-should-you-use
https://github.com/compose-spec/compose-spec/blob/master/04-version-and-name.md

---
 docker-compose.yml | 1 -
 1 file changed, 1 deletion(-)
docker-compose.yml +0 -1
index 8b43946..8704599 100644
@@ -1,4 +1,3 @@
-version: '3'
 services:
   layersdb:
     image: mariadb:lts
Signature · 1 line
2.34.1