About after-deploys checks

paco · September 3, 2020, 6:42pm

We all have our routines after new deployments, but I am wondering if I am not missing something in my own ones, so I decided to start a post to share best practices regarding this topic.

Of course it depends on what you are deploying and how you are doing it, but I will choose the case of the Global Team deployments, as it is the one I know the best and the practices for other approaches would be very similar.

I will post my own routine, and then we can discuss about it and improve it. I will at the end synthetize the discussion into a specific wiki on ofn-install projects if it sounds good to you.

AFTER-DEPLOY CHECKS:

Check general process from #devops-notifications channel
- Look at Ansible execution info
- Check there is no new bug
Check bug trackers
- Be sure on Bugsnag that existing bugs are not increasing
Check performances
- Take a look at Datadog dashboards
- Unsual behaviors: https://app.datadoghq.com/watchdog
- Metrics: https://app.datadoghq.com/dash/integration/1/system-metrics?from_ts=1599106501689&live=true&to_ts=1599120901689

sauloperez · September 9, 2020, 7:36am

that’s an interesting very tech-specific approach @paco. I wonder what others do. I’m guessing other instance managers place an order in a demo shop. @Kirsten @Rachel @lin_d_hop @tschumilas?

The cool thing about your approach is that very easily automatable. We can get email notifications for most of those steps.

sigmundpetersen · September 9, 2020, 7:42am

We also have https://ofn-user-guide.gitbook.io/ofn-testing-handbook/testing-handbook-v2-from-google-drive/how-to-do-a-sanity-check-in-production-after-a-release

Ping @Rachel @filipefurtado

Rachel · September 9, 2020, 7:48am

Yup I do as the doc says

sigmundpetersen · September 9, 2020, 7:53am

We have also discussed automating what the doc says https://github.com/openfoodfoundation/ofn-install/issues/493

filipefurtado · September 9, 2020, 8:27am

Great that you’ve raised this issue @paco.

Sure it is tech oriented, but I feel these steps could be added to the our gitbook-document. Maybe as advanced “Advanced Checks” or so. It would perhaps benefit from some additional context but I would add this immediately and build as we go.

I’d love to have a chat on Datadog - I’ve never really used this platform before. The current log-in info in Bitwarden seems outdated. If anyone is available for this please do ping me on Slack.

Thanks as well for pinging here @sigmundpetersen - and especially for pointing out #493 in ofn-install!

tschumilas · September 9, 2020, 4:31pm

we also do what the doc says - and we also let some of the users we work most closely with know to ‘have an eye’ that morning as they do their usual stuff, for anything unusual. I totally agree with automating the approach - but honestly, we have good relationships with a group of diverse users who can also be our ‘checkers’ in ‘real life’.

paco · October 11, 2020, 2:26pm

Sorry for the late reply but I got good information there.

@sigmundpetersen Thanks for pointing me these 2 good links about tests, there are very useful for the functional tests.

As @sauloperez and @filipefurtado mentioned, I was more looking at technical checks, and I think both approach can be complimentary. The most important is for sure the functional approach (eg. the checkout is working fine), but the technical approach can unveil some invisible performance issue (eg. the checkout is taking 2 times more resources than before) or point new invisible issue (a new error on a non-essential XHR call).

I really agree that this could be specified inside the OFN Testing Handbook gitbook, maybe we could add a ‘Devops Testing Handbook’ or something like that. It could just be the same checks I listed or we could setup a more significant list.