Sharing `secrets.yml` for ofn-install


#1

We provision servers with ofn-install. And most people deploy with that as well. While the code includes some configuration files for each server, we also need a secrets.yml file for each server to run the scripts. There have been several practices so far:

  • Sharing the file in a private slack channel.
  • Encrypting the file with ansible-vault and share it through a private Github repository.
  • Copying the file onto the server, e.g. scp inventory/host_vars/openfoodnetwork.net/secrets.yml openfoodnetwork.net:secrets.yml

I found several downsides to these approaches:

  • Slack cannot be trusted. People may forget that secrets where shared months ago and invite someone into a channel who should not have the secrets. It’s also a bit cumbersome to copy and paste all the time.
  • Sharing via Github adds the overhead of permission management for the additional repository. Private repositories are not for free on Github. And we need to enter the vault password every time we want to run Ansible or edit the file.
  • Having it as simple file on the server does not provide any version control. If one person empties the file, other people easily overwrite their copies as well and the contents can become lost. It’s also a lengthy scp command and we have to remember to sync it.

New proposal

We create a Git repository on each server holding the secrets file. The initial setup would look like this:

ssh ofn-admin@openfoodnetwork.net 'git --bare init secrets.git'
cd inventory/host_vars/openfoodnetwork.net
git init
git remote add origin ofn-admin@openfoodnetwork.net:secrets.git
git add secrets.yml
git commit -m 'Initial secrets file'
git push --set-upstream origin master

The ofn-install repository is not affected by this. It ignores the additional .git folder. This gives everybody with admin access to the server access to the file needed for provisioning. No additional permission management. We also get version control and synchronising the secrets is as easy as git push and git pull. Okay, you man need to go to the directory first: (cd inventory/host_vars/openfoodnetwork.net && git pull)

We can document the setup in ofn-install and have that as the default location.

What do you think? @enricostn @sauloperez @luisramos0 @Matt-Yorkley @sigmundpetersen @paco

A note on security on dev machines: My disk is encrypted and that’s why I don’t worry about storing sensitive information on there. If this is a concern, we can still use ansible-vault for an additional layer of security. It doesn’t matter on the server, because the server has all the secrets in application.yml and the database anyway.


#2

I like this solution. I think keeping the secrets file in the home folder on the server is the best method we’ve had so far, and the addition of versioning to this method will be a definite improvement. :+1:


#3

We’ve been working on public repos, no need for private repos. Also I think this is the easiest/more practical way.

In any case it could be joined with your proposal. Even if I don’t understand the use case of having the secrets stored in your server, if you really want your secrets stored in your server you can clone the public repo there.

As per the permission management issue I would love to see ALL the secrets public repos under openfoodnetwork Github org and use the permission management we already use for all the other repos.

Let’s keep it simple, same rules for all the repos. But it’s just my 2 cents of course.


#4

I don’t think that public repositories protect the secrets well enough. Somebody can just download the vault file and start a brute-force attack on the password. I don’t think our current passwords are strong enough for that.

With a stronger password that may not be successful at the moment, but all encryptions have been cracked at some point. It’s a matter of time. While encryption methods are updated as well, the old vault with the old encryption has been made public and can still be in the hands of an attacker. That would force us to change all passwords regularly. And we would need to change the cookie encryption secret which is a pain.


#5

I thought the idea was to initialise a local git repo on the server to allow for tracking the file history, but not have a corresponding github repo?


#6

That’s what I did with the US instance. It works well so far.

On a tangent: I’m thinking of having a copy of the ofn-install repo in the server as well to facilitate deployments:

  • single point of truth of which code was used for the last deployment
  • performance improvement by executing Ansible locally on the server
  • easy integration with Semaphore or other tools to trigger one command on the server for deploys

I will share more info once I’ve actually done it and know if it’s worth it adapting.


#7

Ok, after procrastinating on this for a long time I finally consulted a sysadmin friend (@magec) as this is not my biggest area of expertise. My ideas don’t differ much about what has been said here but let me answer each of the topics.

We create a Git repository on each server holding the secrets file.

IMO github is better suited for this. They have a dedicated service for hosting git remotes, with all the guarantees that brings: backups, service availability, etc. and we won’t implement.

Also, I think this better aligns with our strategy of having a single team, a single repo for the app, a single repo for the deploy and provisioning scripts. It would feel natural to have also a single repo for all secrets, which I hope it’d make it easy for new core team members to understand. I’m afraid storing secrets in a repo in each server will be confusing for both, newcomers and current core team members. The simpler, the better.

On the other hand, I think having all secrets in a single repo rather than scattered in each server will make it easier to have them all in sync and ease its maintenance. Not to forget that we’re giving up on our code management practices such as PRs and code review, which to me is a very big downside. It’s been a big effort to adopt them in all our repos and it’d be a pity to have an exception now.

I don’t think that public repositories protect the secrets well enough.

As @magec told me, it’s probably better to have the secrets in a private Github repo, as this keeps malicious people unaware of them. That costs money, yes, but the time we’ve spent talking about this is much more than what a team plan costs ($9 user/month), so I think we can totally afford it.

Somebody can just download the vault file and start a brute-force attack on the password. I don’t think our current passwords are strong enough for that.

I disagree here. Having the repo private saves us a bit from anyone downloading it, although not completely, and a brute-force attack doesn’t seem feasible to me. According to ansible vault docs The default cipher is AES (which is shared-secret based), actually AES256, which is the one used in SSL/TLS. I understand then that the password is just the passphrase. In any case, ansible vault’s use case is exactly ours.

While talking to @magec I also realized that ansible vault allows encrypting only values, which would make it easier to track the changes in our files. It looks like the following:

some_foo: !vault |
      $ANSIBLE_VAULT;1.1;AES256
      62313365396662343061393464336163383764373764613633653634306231386433626436623361
      6134333665353966363534333632666535333761666131620a663537646436643839616531643561
      63396265333966386166373632626539326166353965363262633030333630313338646335303630
      3438626666666137650a353638643435666633633964366338633066623234616432373231333331
      6564

Finally

Sharing via Github adds the overhead of permission management for the additional repository.

I don’t think that is a problem. We managed to fix permissions once and for all and adding new repo it’s just giving access to the core devs group.

Sharing passwords

To keep things secure I suggest we start using a password manager to avoid having to share the vault’s password as well as any other. At Coopdevs are pretty happy with Bitwarden and I’ve also been using (LastPass)[https://lastpass.com] personally for a long time.

What do you think?


#8

I agree that Github provides a better service for Git hosting.

I’m glad that you agree on having a private repository for the secrets. I wouldn’t be surprised if some people scanned all public Github repositories for Ansible Vault files and stored them somewhere just in case the encryption is broken in a few years. Every encryption is broken one day.

The feature of encrypting only values is great. That kind of enables us to do “code reviews”. At least it will enable us to quickly assess the configs when config parameters change.

I have a few questions about a private Github repository though:

  • Who will have access to the repository? Our Administrators team has 9 members. That would be $81 a month or $972 a year. We can fund the hosting of two OFN instances for that.
  • What do people do when they are just starting out? They want to set up a new instance and are not part of the OFN core team yet. I guess they just have to find their own solution until they are trusted by the community to have access to all servers and all secrets. In the mean time, we may not have access to that secrets file, because they found their own solution. Otherwise they have to share it with us somehow.

#9

That is very valid point. I thought all core team devs would have access to secrets, but it turns out @luisramos0 doesn’t feel comfortable deploying so distinguish between devs and deployers. More about this in https://github.com/openfoodfoundation/ofn-install/issues/267.

So, if we have a subset of devs with deployment permissions we could lower that price down considerably, right? What do you think?

What do people do when they are just starting out? They want to set up a new instance and are not part of the OFN core team yet. I guess they just have to find their own solution until they are trusted by the community to have access to all servers and all secrets. In the meantime, we may not have access to that secrets file, because they found their own solution. Otherwise, they have to share it with us somehow.

I like that you raise this point. Here we’re getting to the actual topic which is the organizational change OFN is going through IMO. I believe this thread is more about this than anything else.

So, the way I see it is that new instances trust the OFN core team to deal with the instance set up. It is no longer a bunch of projects with their own tech team and infrastructure that simply share a code base but a single project with a single team and infrastructure, which happens to be distributed in regions throughout the world (but it could be different). We communalize not only the software but the servers as well and their management.

I see this shift as a big opportunity to make the network bigger. I think that for example the barrier of entry for Belgium it’s being much shorter than others before them, am I right @Matt-Yorkley?

What do others think? I believe this is a critical point on the transition OFN is undergoing this year.

To answer your question, I then think is more about what fits us best than them. I also think the ansible vault solution gives room for exceptions though because we simply specify the path to the specific secrets file, which could be anywhere.


#10

VERY relevant for this discussion https://github.com/nonprofit. I believe we meet all the requirements so then, as soon as we get the unlimited private repositories, we get rid of a downside.

It’s time to move to action. This is one of the major blockers for the sysadmin standarization that we started back in December 2017. To move to a conclusion, and hearing the opinions shared in this thread, here is what I propose

Final proposal

First iteration

  • Apply for https://github.com/nonprofit
  • Create per-instance private repos in Openfoodfoundation’s GH organization to hold their secrets.yml. See an example in https://github.com/coopdevs/katuma_secrets
  • Create a Core Team in Github with all its members. There are people in the current groups that are no longer around and to whom I wouldn’t give access.
  • Give this team permissions to edit all secrets repos
  • Create a Bitwarden account for OFN giving access to the core team
  • Store the encryption passphrase of each instance’s secrets.yml in Bitwarden

Second iteration

  • Switch to ansible’s value-only encryption in all secrets repos
  • Create a deployers team, subset of the core team with permissions to edit all secrets repos

This last step will answer @luisramos0’s concern, already tracked in https://github.com/openfoodfoundation/ofn-install/issues/267.

Resulting processes

As a result, this is how our processes would look like

First time setup

# Replace france with any other instance name
# Let's assume this repo lives next to openfoodnetwork's repo
$ git clone git@github.com:openfoodfoundation/france_secrets.git

Provisioning

Copy the french instance’s vault password from Bitwarden’s browser extension

$ ansible-playbook playbooks/provision.yml --limit=fr-staging -e "@../france_secrets/staging.yml" --ask-vault-pass
Vault password: # Paste the vault password you just copied

-e stands for –extra-vars, Ansible’s mechanism to pass variables on the command line. @ is the way to tell it to get the vars from a file.

Sit back and relax while the provisioning takes place :tropical_drink:

Deployment

Likewise, copy the French instance’s vault password from Bitwarden’s browser extension

$ ansible-playbook playbooks/deploy.yml --limit es-staging --ask-vault-pass -e "@../france_secrets/staging.yml"
Vault password: # Paste the vault password you just copied

Enjoy! :rocket:

So please, if you think there’s something relevant worth modifying from this proposal do so in the coming days. Do you see a little improvement on top of this which wouldn’t change the proposal much? let’s iterate and include them one by one but let’s get the ball rolling!.

Once we agree on the steps I’ll start creating the necessary issues in ofn-install.


#11

Sounds good. I haven’t worked with Bitwarden yet, but it sounds exciting. :slight_smile:


#12

lets do this!
I have used lastpass to manage passwords within large teams, it’s very good. I am happy to try a new one! Bitwarden ftw


#13

I just created the epic https://github.com/openfoodfoundation/ofn-install/issues/307 with all the tasks of the first iteration under it. I moved only the first one to Dev Ready as I’m sure this will take long.