Hi everyone!
Time to share some updates and move this forward. Last week @matt and I had
a catch-up to get on the same page and see how to proceed now.
We touched on the architecture of the system so far and understood its challenges. I won’t go into details here but he explained it very well at https://github.com/openfoodfoundation/ofn-data/wiki.
These are the things that stood out:
- We need to add the security stuff @matt worked on before going to production
- There’s some concern about the resource impact this will have in all instances. We need to try and monitor.
- The current setup is ready for prime time. No big extra tasks need to be done. We might only need to add monitoring to this new server.
- We might need a separate purpose-specific server (extra RAM and better disk?) to host the DB replicas pretty soon.
The single aggregated database
We came to the agreement that besides the per-instance read-only replicas what we need is a single read-only database that replicates data from all instances together so that we can have the global perspective of OFN as a single entity and start gathering relevant metrics at many levels.
@Matt-Yorkley thinks Kafka connect (the underlying tech) could fill this need as well but this something wasn’t yet implemented and will need investigation.
Updated roadmap
What we did so far
We haven’t moved much the proposed pilot above. We did the following.
- Connect Metabase to Katuma’s production DB (exactly what we wanted to avoid with the DB replicas…) to start gatering billing information.
- Prepare some basic usage dashboards we used to share metrics with the Dimmons research group.
- Connect Metabase to UK staging with a somewhat old production DB dump to start playing with metrics.
What we propose from now on
The sudden crisis at Katuma due to the lack of instance manager (working on it) makes it a less good candidate to pilot.
Also, due to concerns about the extra load, all this might add to our servers we think is best to choose a standard 8Gb RAM instance server (Katuma is still to be upgraded).
So, to summarize, these are the next actions (in order):
- Update all dependencies before hitting production https://github.com/openfoodfoundation/ofn-data/issues/7
- Change Datadog’s in-depth performance monitoring (Called APM) from the UK to FR. We pay for a single server.
- Provision FR production with ofn-data.
- Keep a close look at the monitoring to spot performance implications.
- Onboard and help @Rachel get acquainted with Metabase and report insights back to the team.
- Check whether Kafka Connect can replicate from multiple instances to a single database.
Of course, we need @Rachel’s and @lin_d_hop 's agreement first, at least.
Anything else you might want to add @Matt-Yorkley ?
If there’s some consensus, I’ll prepare issues to add at least one to the next delivery train.