Seed data [development] [provisioning] [deployment]

Software improvement 5-Done
#3

More than a year later, we made some progress on specifying test data, but we haven’t implemented anything yet.

So the idea is that we have a standardised set of test data that everybody can use. This data will be available on all staging servers and on our local development machines. Yay. But in discussion with @sstead, @oeoeaio and @Kirsten, we raised a few more issues that are not solved by this data set.

Changing the data set

Writing this data set requires a developer and is more laboursome than creating the data through the admin interface. While it would be great to have all test data defined in code, it also means that changes will go through the whole pull request, review, staging and testing process.

In some cases it would help testers a lot if they could save the current state of a staging server and reload it when testing again. In Australia, we have a process in place to always save the staging database when a pull request is accepted and merged. It allows us to persist data. The database will be reset after each failed pull request. This is not ideal either, because the tester has no control over saving and resetting data. But it worked quite well for us.

Enabling testers to change data and persist it would speed up testing.

Custom and private data

Some data cannot be included in a public data set. For example connecting an enterprise to Stripe involves some secrets that should not be shared. I’m also not sure if the same keys would work on several servers. If we don’t want to run through the set up every time we want to test Stripe payments, for example, then we need a way to persist custom private data on the staging server.

Again, custom data speeds up testing.

Default user accounts and credentials

I like Pau’s idea to use Mailinator to access email inboxes of test accounts. Does this have any security implications? Since everybody can sign up as enterprise user, we can also provide public access to enterprise users. That is no problem.

I do see an issue with a public admin user though. Admins can access some private API keys and change configurations. That opens possibilities for misuse of our staging server. I’m happy to have a standard admin account with a password we all share, but I don’t think it should be public.

#4

I like the idea of having easy and consistent test data available on every staging server so that people can hop onto any one to test the feature that is staged there. But weighed into this discussion with @maikel and @sstead as I think the lag created by needing a dev to update test data would be extraordinarily painful and bad. Is there a way we can get a hybrid of this? Like

  • have test data that is loaded by default
  • allow key / lead testers? to update that data when testing
  • have a process for them to nominate when their changed data should be loaded into the base? and have that happen easily . .

nothing is easy kirsten . . but something like that would be great

1 Like
#5

Hey @Kirsten, I think you want a acceptance test framework. I have used fitnesse in the past. It’s a wiki page testers can edit and change the test data (and the tests!). That is the future. OFN will require an acceptance testing framework on top of rspec, like cucumber or turnip or some other nice tool I dont know yet.

But before that we need to improve the basic data set. We could train the testers to change this data as they think appropriate with a simple git workflow:

  • checkout
  • edit db/seeds.rb or other
  • create PR

We need to do this very soon. IMO we should attach this task to the spree upgrade. Before we start the functional validation of ofn v2 (ofn with spree v2), we should create this data set in the v2 branch. I think that, at this stage, making this in v1 and then migrating to v2 would be a waste of time.

What I am proposing is that we make this https://github.com/openfoodfoundation/openfoodnetwork/issues/2072 a mandatory part of the spree upgrade.

OFN Australia adopting the common release cycle
Delivery Circle Meeting - 15th march
#6

ohhh @luisramos0 this sounds exciting - @sstead @MyriamBoure @Rachel what do you think?

#7

@luisramos0 I’ve used behat in the past, is fitnesse in the same spirit? But yes in a nutshell I fully agree with you and I’m free to go forward with this. As this is part of the task that was assign to me and @maikel at our last meeting, I would love to have his opinion as well :slight_smile:

#8

Whaooo ! That seems cool @luisramos0 ! Ok I never used those type of tools of course but if I understand well the idea is:

  • that as testers we will have some sample data set like the content of this file but that we will be able to update them as we go as we will be trained to ourselves commit PRs to change it. So we can iteratively enrich step by step this data set with what we need and that will be saved somehow for later testings. I like this idea :slight_smile:
  • then the second point is about writing acceptance tests with the framework tool you propose. Is it like “writing the test cases” we write today in the PR or stories for new features to check if we consider they pass or fail? or in the testing template when we write “test cases”? I’m not super clear about what it would be concretely… would this replace the first dot point about changing sample data through PRs in GH? I guess so. Anyway I feel like it is the good way to go, I guess it would “professionalize” a bit our testing processes :slight_smile:
    It would be good to “test” FitNesse to see what it looks like and get a more concrete sense of it.
#9

First we need to see if other devs know this space and have opinions.

I read just a little bit more, looks like that turnip is really the cucumber that plays with rspec (we use rspec already). In these tools the data tables have to be programmed, for example, devs would have to create a “product table” that could be used in tests (testers could create their specific product list), or a “enterprise table”.
Fitnesse is another thing because testers can create database data directly and edit it as a wiki. It’s awesome but a longer setup/learning curve.

Interesting article about this subject and about turnip with rspec in specific, I think the beginning is useful even if you are not a dev (we already use capybara as well): https://medium.com/@bobbytables/acceptance-testing-with-turnip-and-rails-dbd7d65398f2

#10

I agree with above :slight_smile:

#11

I have no particular experience with these tools. I know cucumber exists and how it looks like but I’ve never used it. If we want to give this a try I’d go with Cucumber as is the most known tool and has a big community (I guess) rather than Turnip. But before that I’d like some things to be considered.

But before that we need to improve the basic data set

We’ve worked with @MyriamBoure on that doc you linked but we never actually implemented it because there are always 100 more things with greater priority. If we think this is now affecting our velocity, let’s find the resources and give it the priority to implement it.

We could train the testers to change this data as they think appropriate with a simple git workflow

That could be a next step but do we have the resources? changing this data means being able to code in Rails. Are testers ready to do this or will it add extra overhead to our already small core dev team? How much time will we spend training them? how much maintenance overhead will this add in the long term even if they manage to code them themselves? I absolutely encourage everyone learning to code but it feels far from our capacity training them to me, at least now.

What I am proposing is that we make this #2072 a mandatory part of the spree upgrade.

I’m fine with that but not more than that. I’m afraid of making the 2-0-stable branch bigger and the upgrade itself even longer.

After that, we could consider Cucumber IMO, which to me is different from implementing new sample data.

#12

I feel like the discussion went a bit off-topic. Behat, Cucumber and Turnip are basically the same thing. But they are about writing automated tests. This topic was about test data. And I would like to throw in three options:

  1. We define test data and code it. This is a lengthy process and needs developers for implementing, but then it’s very easy to share between instances. It’s very good for basic data, but we can’t store configurations containing secrets, like a Stripe setup in there.
  2. We use fitnesse which allows more people (non-devs) to create the test data. I haven’t worked with it, but I guess it’s very easy to share, but we would have the same restriction of not coding secrets in a public database.
  3. We allow testers do take database snapshots on staging servers which are the base for new staging deployments. They can create the setup as they like and then save it. This wouldn’t be easy to replicate between staging servers automatically, but testers can do it themselves. They just add the data they are missing naturally while testing. It creates unique databases which can be a downside (not reproducible on other servers), but also be an upside (complex data reveals more bugs and resemble production data more closely). It also enables us to store unique secrets in the databases.

My favourite would be to combine all of these. We already have a very small set of programmed test data. Let’s extend it. Let the testers create and save more data on the staging servers while we work on adding more data to the programmed data pool. And let’s consider fitness as an alternative way to generate the programmed data to speed up the process.

All of these things don’t depend on each other and can happen in parallel. What do you think?

#13

Fine for me @maikel. We can plan the work on 1. while someone first spends time thinking about how to best use 3. and then we spend some time figuring out how 2. would fit for us.

#14

@MyriamBoure @Rachel I almost implemented all data defined in https://docs.google.com/document/d/1_va0Aw2yJ5BRe-HtuBT-OC8HNHIui0gvQ0uElWjDVs0. There is just one thing I would like to discuss.

The mailinator email addresses would leave all accounts open for abuse. I suggest to use invalid addresses like jane@example.org. We can still read all emails by using a dedicated email address that is configured in the mail method as “Send Copy of All Mails To”. It could be staging-emails@openfoodnetwork.org.

What do you think?

#15

@maikel this is awesome :slight_smile: Looking forward to see this going further. I like your suggestion, let’s go with that!

#16

Awesome @maikel!!!
is there an issue on GH for this? I’d love to have a look at a wip PR :slight_smile:

#17

@luisramos0 I opened my pull request with most of the seeding data. It would be good to get some feedback before doing the next iteration: https://github.com/openfoodfoundation/openfoodnetwork/pull/3209

1 Like
#18

@MyriamBoure @maikel @luisramos0 @sauloperez One small proposal for improvement after using this a bit:

would you guys mind if we put first names that would be radically different?

When I test mobile or when I use a dropdown for example, I only see the beginning of the enterprise name and it already happened that I changed something on backoffice for Mary and in frontoffice I checked Maryse’s shop on mobile trying to understand what was wrong :exploding_head:

So for dummies like me I suggest this:

  • Freddy’s Farm shop => George ?
  • Fredo’s Farm hub => Bob ?
  • Maryse’ Private Shop => Phoebe ?

Or we don’t put first names in the enterprise names.

#19

Good idea. I got confused with the names as well. I like your suggestions.

#20

yeah, confusing names.
what if we follow personas names?
Shannon and Jane?

#21

@luisramos0 Jane is our final buyer. But yes we can use Shannon, I don’t know why I thought she was already somewhere. So we just need to find names for Fredo and Freddy who are derived from Fred our farmer persona (who is used here for the producer profile).

1 Like
#22

Actually I understand about the confusing names… but I put those names to try to stick to the persona, yes, as @Rachel guessed, Fred and derived for farmers, Mary and derives for distributors… but if you want to change, no objection as long as we document carefully so any tester can use. Or we could just in the name put “George (prod)”, “Sarah (retail)”, “Sylvia (wholesale)” so it’s obvious what role they play.