Configurations of OFN instances
To become a single coherent OFN unit, we need to find out what we do differently, where it makes sense and where we should agree on a unified process. This document discusses different configuration aspects of each OFN instance and if we need to improve the current situation. Some of this may be just my opinion. The summary at the bottom contains the key points to discuss for a more unified setup.
We all use the ofn-install project to provision our servers, but there are a lot of differences. Some differences come from the different times when the servers were installed. Other differences are configured within ofn-install. And last, some instances have been modified to compensate shortcomings of ofn-install or to customise certain aspects.
Operating system
As far as I know, everybody uses Ubuntu LTS for their server. Our install scripts are tailored to a recent LTS version at the time.
- Ubuntu 16.04: Aus prod, UK prod, Mex stage
- Ubuntu 14.04: Can prod, France prod, France stage, Norway prod
- Ubuntu 12.04: Aus stage, UK stage?
At least the Australian staging server should be updated. But the Canadian, French and Norwegian servers need updating within one year as well. From the Ubuntu wiki:
- Ubuntu 12.04 has been unsupported since April 28, 2017.
- Ubuntu 14.04 will be supported till April 2019.
- Ubuntu 16.04 will be supported till April 2021.
I don’t see any issue with using Ubuntu as default operating system. It’s well supported and the support period is quite good. Debian may be a bit more value aligned and supporting it would be nice as well. But there is no urgency for us.
Deploy methods
There are two ways to update the application on a server.
- Use the ofn-install deploy scripts. It creates a snapshot of the current application data to enable a rollback. It then builds the whole application from scratch and restarts the application server.
- Use the Git post-receive hook script. When you push updates via Git to the server, it triggers the post-receive hook. That script checks out the latest files in place and reloads the application.
Both methods keep the database, run migrations, compile the assets and restart DelayedJob. There is no agreement of which method is better. But developers expressed that both methods have there benefits and it would be good to create one method with all the benefits.
The benefits of using ofn-install are:
- provide a snapshot for rollbacks
- resetting the application configuration according to the inventory
- latest updates from ofn-install can improve the server configuration
The benefits of using the Git post-receive hook are:
- no Ansible needed for deployments
- fast in-place application update without downtime
- allowance of deploy customisations per server in the post-receive hook
- lower risk of bugs in ofn-install affecting the deploy process
Methods in use:
- ofn-install: UK, Spain, France (
ofn_deployment), ? - Git hook: Aus, Canada, Norway, ?
Email methods
Since the email method credentials are secret, they are not contained in ofn-install and I have no idea who uses what. The Australian team started with using a Gmail account to send emails from. The UK and Canada were set up with Gmail as well, but we had some complications with the Gmail configuration and password changes.
Gmail upsides:
- people are familiar with Gmail
- one account to manage all emails from
- sent emails are stored
- up to 60,000 emails per month (2000 per day) with G Suite
- up to 15,000 emails per month (500 per day) with personal Gmail account
Downsides:
- Gmail account needs configuration to allow SMTP
- If the Gmail account user changes the password, OFN mailing stops working
- OFN can only send with one identity
Mandrill used to be a free option to send emails, but has only paid plans now.
Free alternatives are SendGrid and mailgun.
SendGrid and mailgun upsides:
- Professional tool for email sending.
- Any identity of owned domain possible.
- Easy configuration (I guess, I haven’t done it)
Downsides:
- mailgun allows only 10,000 emails per month for free
- SendGrid allows only 3,000 emails per month for free
- You still need email hosting for your domain
Setting up a local mail server is possible as well, but it’s not included in the ofn-install scripts and needs some DNS configuration to ensure deliverability.
Own mail server upsides:
- No dependency to other services.
- No explicit limits to number of emails.
Downsides:
- Additional configuration and maintenance.
- Additional knowledge required to ensure deliverability of emails.
File storage
We have two options. The OFN default is to store uploaded images locally on the server. This is simple and easy, but images can take up a lot of disk space after a while.
The bigger OFN instances use Amazon S3 to store images. This adds the complexity of setting up the S3 bucket with the right permissions and API keys. We also rely on Amazon. If they are down, we don’t have images. Amazon also doesn’t seem to be aligned with our values. But the big upside is that Amazon is very reliable and very, very cheap (for file storage). It also takes load off our server.
Backup methods
Similar to images, we have the option to store backups with Amazon S3. This kind of off-site backup is crucial if a server is lost. The downside is that we give sensitive data to Amazon.
Backing up on other servers or using a redundant database setup is also possible, but I don’t know of any OFN instance that does it. It’s more work and more expensive.
TLS/SSL certificates for HTTPS
The ofn-install project uses certbot to automatically retrieve a certificate from Letsencrypt. The certbot software is still under rapid development and in many cases ofn-install failed to set it up correctly. While this should be solved in ofn-install, admins added some manual configuration to deal with it. We should check regularly if ofn-install is installing certbot correctly.
In Australia, we use Wormly (see Monitoring below) to check the validity of our certificates. It warns us 20 days before expiry in case the server is not configured correctly.
Monitoring
There are several optional tools we can use to monitor our applications:
- New Relic: performance monitoring on the application level
- Bugsnag: aggregating raised exceptions, can be used like an issue tracker
- Wormly: external monitoring tool, tests the availability of servers and can monitor resources
- Skylight: performance monitoring
Discourse integration
We can connect an OFN instance to a Discourse forum.
OFN customers can then log into Discourse with their OFN account.
Summary
- Operating system: At least Aus staging needs updating.
- Deployment: We haven’t found the perfect way yet and need to shape it.
- Email: Gmail has high rate limits, but is not ideal as email gateway. Mailgun seems to be the best alternative. Integrating the setup of a mailserver in ofn-install would not be friction free (DNS setup), but could be an option.
- File storage: Amazon S3 is recommended. A more value aligned alternative would be nice though.
- Backups: Offsite backups are highly recommended. Amazon S3 is the way to go at the moment, but there are privacy concerns (especially for Europeans).
- TLS/SSL configuration needs checking regularly.
- Monitoring: We have several good tools. Use whatever you like.
- Discourse: no recommendation
See also: Managing access to servers and associated services