There are two items in the current product development backlog around “Being a single coherent OFN unit” that relate to the standardisation of SysAdmin across our servers. We decided that this was important because at the moment it is almost impossible for anyone without an intimate knowledge of a particular server to do any useful sysadmin work on it without fear of breaking things.
The main issues which have been identified so far are the following:
- there is not really a master list anywhere that lists all of the servers that we consider to be a part of the global pool
- SSH access to servers is very restricted and inconsistent, it is unclear who has access to which servers, and therefore who should be asked for access
- there is minimal documentation about the configuration used on a given server (ie. hosting, S3, backups, ofn-install, SendGrid/Mandrill, Stripe)
- there is no standardised way of access the login credentials of services associated with a given server
If there are any other issues that I have missed, please post a comment below or edit this post yourself.
I’ll have a crack at listing all of the servers that I know about that should be considered a part of the global pool. Please add any that I have forgotten.
Global site (openfoodnetwork.org)
Australia CI (hosts our Buildkite agent)
To resolve the first issue (SSH), the proposal is that we add the keys of all core developers to all known servers. This should mean that there is an least one person available at all times to at least temporarily grant access to others if they are not in a position to do the work themselves. I am happy to do this for the servers that I have access to but that is pretty much just Australian servers and UK prod, so someone else will need to at least add me to France, UK staging, US, Canada, Scandinavia, etc.
In terms of server configurations, the most important thing is to do an audit of the current servers and document the configuration that they use, preferably in the ofn-install Wiki. For servers that were not provisioned using ofn-install, it is important that we also include information such as the user to use when logging in. I believe @maikel has made a start on this, but it is quite a large job. I’ve done the Australian production server, and I think if we could work together to answer all of the following points for each server listed above that would be a good start:
Operating System: Ubuntu 16.04.4 LTS
CPU: 4 x Intel® Xeon® CPU E5-2620 v3 @ 2.40GHz
Memory: 7.5GB + 1GB swap
Hosting: Rimu hosting VPS with SSD
Backups: S3 (every 4 hours)
Email: Gmail (email@example.com)
Once we have this information we can work out next steps for the standardisation process.
It would be good to have a central place to store access credentials for all services (hosting, AWS, stripe, mail servers etc, etc) for each instance. I am open to suggestions as to the best approach to this, but I thought that a good starting point would be the creation of an account on something like LastPass (or another password manager if other people have a strong preference). We could use it to document logins for services associated with any server in the global pool. I propose that we grant access to this account to a very restricted set of users: core devs + Kirsten and Myriam would be my preference. Then these people can opt to give others passwords for specific services as required. If anyone has an alternative suggestion let me know.