More than a year later, we made some progress on specifying test data, but we haven’t implemented anything yet.
So the idea is that we have a standardised set of test data that everybody can use. This data will be available on all staging servers and on our local development machines. Yay. But in discussion with @sstead, @oeoeaio and @Kirsten, we raised a few more issues that are not solved by this data set.
Changing the data set
Writing this data set requires a developer and is more laboursome than creating the data through the admin interface. While it would be great to have all test data defined in code, it also means that changes will go through the whole pull request, review, staging and testing process.
In some cases it would help testers a lot if they could save the current state of a staging server and reload it when testing again. In Australia, we have a process in place to always save the staging database when a pull request is accepted and merged. It allows us to persist data. The database will be reset after each failed pull request. This is not ideal either, because the tester has no control over saving and resetting data. But it worked quite well for us.
Enabling testers to change data and persist it would speed up testing.
Custom and private data
Some data cannot be included in a public data set. For example connecting an enterprise to Stripe involves some secrets that should not be shared. I’m also not sure if the same keys would work on several servers. If we don’t want to run through the set up every time we want to test Stripe payments, for example, then we need a way to persist custom private data on the staging server.
Again, custom data speeds up testing.
Default user accounts and credentials
I like Pau’s idea to use Mailinator to access email inboxes of test accounts. Does this have any security implications? Since everybody can sign up as enterprise user, we can also provide public access to enterprise users. That is no problem.
I do see an issue with a public admin user though. Admins can access some private API keys and change configurations. That opens possibilities for misuse of our staging server. I’m happy to have a standard admin account with a password we all share, but I don’t think it should be public.