This year we have been working on the long over due task of upgrading server operating systems:
It is currently in progress, but I’ve been asked to give a summary. The core team maintains 15 servers, which are on varying versions of Ubuntu. The table below lists each server, their current version and support level
A note on Ubuntu versions:
We choose the LTS (Long Term Support) versions of Ubuntu, which receive general Maintenance & Security support for 5 years from the release date. With a Ubuntu Pro subscription, Expanded Security Maintenance (ESM) is provided for another 5 years.
The current LTS version is 24 (Ubuntu versions are named for the year they are released). More details about Ubuntu support are summarised at Ubuntu | endoflife.date.
To consider
Ideally, we would upgrade all servers to the current version Ubuntu 24, but our server provisioning scripts (ofn-install) only support up to Ubuntu 20. To go further requires an upgrade to our scripts to support a later version of Ansible and Python. So we have considered multiple options:
Update the oldest servers (Ubuntu 16) to version 20 to bring them into general support in the short term. (in progress)
Upgrade ofn-install for the latest version of Ansible/Python
Re-write ofn-install using less Ansible and more shell scripts
Develop a Docker container for provisioning servers, as an alternative to Ansible.
These are not necessarily mutually exclusive, and each have advantages and disadvantages to be discussed…
We need to choose a path forward, and estimate it so that we can incorporate it into long term planning and budgeting.
Get Expanded Security Maintenance for older servers via Ubuntu Pro
From what I can see, Ubuntu Pro is $225-500 per server (depending on our needs, which need to be evaluated).
Or, you can get a free personal subscription:
Free, personal subscription for 5 machines for you or any business you own
It seems appropriate then that each instance manager could get their own personal subscription, thus delaying the urgency to upgrade.
For me, the beating heart of any Free/Libre software project is the self-hosting community. I wonder which of these options would best support people who would like to “tinker” with OFN to spin up an instance ?
I suspect the docker image, but I’m curious what others think ?
I would agree that we want it to be as easy and accessible as possible to manage a server (which obviously benefits ourselves as well!)
Our experience with using Docker in development is that it has brought extra layers of challenges, with an additional dependency. I think most developers give up on that approach actually. So it hasn’t really delivered on the promise of accessibility thus far.
I would like to think that we can move more of the tasks done by Ansible to shell scripts that can be executed directly on the server (perhaps they could live in the openfoodnetwork repo), which I think would make it more accessible.
We need to proceed to upgrade the following servers to Ubuntu 20:
uk
de
be
au
us
Estimate 3x5= 15hrs remaining
According to Clockify AU, 64.5hrs already spent (this includes meetings, liaising with instance managers etc). (Sorry I wish it was less)
Future direction
Our provisioning scripts are on a very old version of Ansible/Python. We think that moving to Docker is too unknown. Replacing parts of Ansible scripts with shell scripts seems like a good way to go, as it reduces the dependency on Ansible, and would perform faster. But we are cautious about spending lots of time on that.
So the most pragmatic approach seems to upgrade Ansible. Where we have problems to solve, we take the opportunity to replace with shell scripts (where relevant). This means we progressively work towards both the goal of upgrading ansible, and reducing reliance on it.
First, we will do a spike on upgrading ansible/python directly to latest version (1-2hrs). (Probably Maikel).
The result of the spike is that we will be able to decide on the exact upgrade path (eg upgrade minor increments or all in one go), and provide an estimate.
Our plan then would be to schedule in the update, so that we can be prepared to upgrade the next round of servers.
Metabase
This server is on Ubuntu 18 and needs upgrading too.
Gaetan will proceed with updating Metabase and Postgres on the current server for now, due to project requirements.
We will need to update the OS at some point in the future. It doesn’t depend on ansible scripts so we should be able to use latest version of Ubuntu. We would consider trying the pre-prepared Docker installation for Metabase at that time.
