I was wondering if we can make our PR testing good enough for releases. The process would be to make release testing more efficient and include more into PR testing until they are the same.
Very well. It sounds reasonable to me to test the checkout as the most business critical feature. For example, the production and staging environment is communicating with Stripe while the tests simulate the communication with Stripe. But we don’t need to test five different enterprise fees manually, because that should be tested automatically and the communication to Stripe is not affected by the type of enterprise fee.
I think it’s good to review the manual testing from time to time and identify parts that can be automated so that they don’t need to be tested manually. And I agree that there will always be some parts of integration testing that can’t be covered by automated tests.
As Rachel asked: What is the difference to a staging server? I think you are saying that our staging servers don’t have that complex data. You would like production data on a staging server. That is almost possible. The process of copying the data from production to staging involves some changes though. Secrets like Stripe keys need to be replaced or removed so that we are not transferring money into real accounts. The email setup needs to be changed so that we are not sending emails to real people. We need to put the application in a sandbox to test with production data without affecting the real world. That’s a medium sized project.
We would. With Continuous Deployment master is always the latest release. We can give a name or leave it.
Ofn-install and our custom deploy script are just two ways to update a server to a specific version of the code. Both work and are efficient enough for continuous deployment. With either of them you can deploy master or a release. We invested work in ofn-install so that it works for all instances, is more reliable and we can share the sysadmin work globally.
Talking to the other core developers, we agree that we would like to work towards continuous deployment. But that’s a slow process, because we need to clarify the impacts on the whole process and adapt our testing and deployment methods. We changed a lot within the last year and it still feels like we are experimenting with out processes. We need to run the current model for a few cycles to identify what works and doesn’t work and then iterate. We are trying to become more efficient and make more and smaller releases. And one day the smallest release contains only one pull request.