Update OFN Physical Architecture - separate webapp server from DB server

Tech
1

In the context of very slow pages in the frontoffice we discussed increasing server capacity:

Increasing existing servers capacity is not the only solution, there are other options.
This simple article lists the options we have:

We are currently on “1. Everything On One Server” and it’s probably time (specially with the idea of moving to a single global instance) to move to “2. Separate Database Server”. This basically means that for each instance that wants to upgrade we would buy a new server and install the DB there. with this, because webapp and DB are separate, things go a lot faster. This is a relatively simple sysadmin tasks and maintenance efforts don’t change much. The only difference is really the cost of the server. So, it costs money but if well done it’s a lot less expensive for the project than the negative impact that very poor performance has on users and the community.

There is no silver bullet and app performance improvements will have to be done as well, but maintaining an appropriate physical architecture with appropriate size is also very important

Thoughts?

2

moving the conversation in the issue above to here:
@sauloperez wrote in the issue:

“Splitting our single servers into app server + DB server it’s something we’ll definitely do at some point but… Comparing implementing pagination vs. implementing a DB server, the former undoubtedly wins; it’s way quicker and straightforward. Besides, it has to happen sooner or later. Limits must be put on all endpoints.”

@luisramos0 wrote:
“what do you mean with “wins”?
in terms of estimation, implementing a DB server is a lot smaller, right?
pagination is something that will require UX, right?”

@sauloperez wrote:

" > in terms of estimation, implementing a DB server is a lot smaller, right?

there is where I disagree. To me, it is incomparably harder and longer (by few orders of magnitude :trollface: ) than implementing pagination, which requires moving some logic from the frontend to the backend, let’s not forget. And with pagination I mean http://ui-patterns.com/patterns/Pagination.

With my far-from-perfect sysadmin skills these are the steps I foresee this would have:

  • Buying a new DB server
    • we can do it as we do now, one for each instance
    • we can move towards the single-instance and buy a single one that hosts multiple DBs for now. This makes it even harder as nothing on ofn-install is thought for it and we’ll need a thorough investigation first.
  • Creating a bunch of new ansible roles to provision a DB server
  • Refactor ofn-install to treat app server and db server separately
  • Defining a strategy to migrate all current instances to the new DB + app server set up.
  • Refactor ofn-install and the database.yml to allow setting a different DB server
  • Provision all instances for the changes to take effect
  • Deploy all instances for the app to connect to a different DB

there might be more though.

Also, the sysadmin processes are one of our weakest points where we’re still taking our first steps. Just think of how long is taking us to set up Datadog or provision and deploy all instances, as an example.

I believe we’re not ready for such big change on the infrastructure. Many things need to happen at organizational/processes level for a change like this to not take several months and end successfully.

However, I do think having a separate DB server it’ll be the next infrastructure change on OFN, but just not now."

3

awesome @sauloperez thanks for detailing the steps to get that task done!!! it’s the first step!

there a few points I’d like to add on to:

  • ofn-install deploy will be done on the app server only, the db migrations can be done from there, so, ofn-install work will only be for provisioning the DB server, correct?
  • “Defining a strategy to migrate all current instances to the new DB” - 1. this will be a shutdown app, 2. export/import, 3. deploy new version pointing to new DB, correct?
  • I dont think we should mix this with moving towards a single instance, it’s related but different. So this task will only be about making each instance use a DB that is on a separate server.

so, I’d refactor your list of tasks into this:

  • buying a new DB server
  • update ofn-install
    ** new ansible roles to provision a DB server
    ** separate app server and db server provisioning
    ** refactor database.yml to allow a different DB server
  • define and execute DB migration strategy, something like
    ** shutdown + export/import and then:
    ** ofn-install.provision each instance
    ** ofn-install.deploy each instance
4
  • ofn-install deploy will be done on the app server only, the db migrations can be done from there, so, ofn-install work will only be for provisioning the DB server, correct?

not only that. Now the provisioning playbook does it all since there’s a single server. We’ll need to refactor that. One thing will be to provision an app server and another to provision a db server.

Also, we can’t forget about monitoring that new server. Particularly postgres stats.

  • “Defining a strategy to migrate all current instances to the new DB” - 1. this will be a shutdown app, 2. export/import, 3. deploy new version pointing to new DB, correct?

we’ll need to think about possible failure scenarios like having recent db backups, etc. it needs careful investigation but yes I guess it’s more or less what you said.

  • I dont think we should mix this with moving towards a single instance, it’s related but different. So this task will only be about making each instance use a DB that is on a separate server.

Sure. Unaffordable to do it all at once.

That list looks good. Thanks for taking the time.

1 Like