Seed data [development] [provisioning] [deployment]

Software improvement 5-Done
1

What

This is one of the action items that resulted from the last developer mumble meeting.

Tha main goal of this task is to provide some meaningful seed data. From the official Rails guide:

To add initial data after a database is created, Rails has a built-in ‘seeds’ feature that makes the process quick and easy. This is especially useful when reloading the database frequently in development and test environments. It’s easy to get started with this feature: just fill up db/seeds.rb with some Ruby code, and run rails db:seed:

[…]

This is generally a much cleaner way to set up the database of a blank application.

Current status

In OFN repository we use seed data only to populate Spree::Country and Spree::State tables for Australia. Look at db/seed.rb for more details.

Furthermore we have a different behavior between running the app in development environment in a local machine and provisioning a production or staging environment with our Ansible playbooks. More info here.

In any case we just seed country and state data, nothing else.

The problem

The problem is that running a new instance of OFN (no matter if development or production) will lead you to an empty shell that cannot be used right away. For instance you cannot try the app since to see a valid shop you need to create an enterprise and at least one product. But when you’re going to create a product you discover that you must create some category (taxonomy) first… and so on.

This makes difficult and tedious to manually test basic behavior while developing and difficult to understand trying to run a production instance for the first time.

When I asked people in #dev channel how do they deal with this problem the response I got was that we move around DB dumps. Basically snapshots of the state of the DB at some (unknown) point. This is not ideal.

Proposal

What we agreed in the meeting is to start this discussion and try to detect the basic data needed to make a health check of the app. I’ll start proposing a basic setup so we can discuss:

Users

Right now we only have the super admin user that Spree seed data provides: spree@example.com. In addition to that would be nice to have:

  • a normal enterprise admin
  • a customer

Enterprises

Nice to have:

  • an enterprise (1 for each type if possible, we can iterate over it)

Taxonomy

Some category for the products. The problem here is that every OFN instance uses a different taxonomy. Maybe we can create a default one just to test the app and override in production or find a way to seed data based on some conditional.

For the spanish OFN instance we started working on a repo to use for production and staging. Maybe we can find a way to use it seeding the DB in development too, still not sure about it.

Products

Some product for each enterprise.

Disclaimer

Since I’m new to OFN please feel free to correct me if I’m wrong.

Discussion

I was thinking about working on a first PR to make some real proposal. What do you think? Am I missing something important? Thought? Rants? :joy_cat:

2

I think that the current minimal seeding is for production deploys. I don’t like the idea of populating a fresh production instance with example data. As you said, every instance has a different taxonomy. And unfortunately, you can’t even delete everything properly through the admin interface.

Did you see lib/tasks/dev.rake? It contains openfoodnetwork:dev:load_sample_data which does what you described. It is three years old though. I think, a good first step would be to rewrite that task to do what you want and to document it. Then you have one command to load example data.

If that is not enough, we could discuss integrating some of that in seeds.rb which should distinguish between environments for that.

What do you think?

3

More than a year later, we made some progress on specifying test data, but we haven’t implemented anything yet.

So the idea is that we have a standardised set of test data that everybody can use. This data will be available on all staging servers and on our local development machines. Yay. But in discussion with @sstead, @oeoeaio and @Kirsten, we raised a few more issues that are not solved by this data set.

Changing the data set

Writing this data set requires a developer and is more laboursome than creating the data through the admin interface. While it would be great to have all test data defined in code, it also means that changes will go through the whole pull request, review, staging and testing process.

In some cases it would help testers a lot if they could save the current state of a staging server and reload it when testing again. In Australia, we have a process in place to always save the staging database when a pull request is accepted and merged. It allows us to persist data. The database will be reset after each failed pull request. This is not ideal either, because the tester has no control over saving and resetting data. But it worked quite well for us.

Enabling testers to change data and persist it would speed up testing.

Custom and private data

Some data cannot be included in a public data set. For example connecting an enterprise to Stripe involves some secrets that should not be shared. I’m also not sure if the same keys would work on several servers. If we don’t want to run through the set up every time we want to test Stripe payments, for example, then we need a way to persist custom private data on the staging server.

Again, custom data speeds up testing.

Default user accounts and credentials

I like Pau’s idea to use Mailinator to access email inboxes of test accounts. Does this have any security implications? Since everybody can sign up as enterprise user, we can also provide public access to enterprise users. That is no problem.

I do see an issue with a public admin user though. Admins can access some private API keys and change configurations. That opens possibilities for misuse of our staging server. I’m happy to have a standard admin account with a password we all share, but I don’t think it should be public.

4

I like the idea of having easy and consistent test data available on every staging server so that people can hop onto any one to test the feature that is staged there. But weighed into this discussion with @maikel and @sstead as I think the lag created by needing a dev to update test data would be extraordinarily painful and bad. Is there a way we can get a hybrid of this? Like

  • have test data that is loaded by default
  • allow key / lead testers? to update that data when testing
  • have a process for them to nominate when their changed data should be loaded into the base? and have that happen easily . .

nothing is easy kirsten . . but something like that would be great

1 Like
5

Hey @Kirsten, I think you want a acceptance test framework. I have used fitnesse in the past. It’s a wiki page testers can edit and change the test data (and the tests!). That is the future. OFN will require an acceptance testing framework on top of rspec, like cucumber or turnip or some other nice tool I dont know yet.

But before that we need to improve the basic data set. We could train the testers to change this data as they think appropriate with a simple git workflow:

  • checkout
  • edit db/seeds.rb or other
  • create PR

We need to do this very soon. IMO we should attach this task to the spree upgrade. Before we start the functional validation of ofn v2 (ofn with spree v2), we should create this data set in the v2 branch. I think that, at this stage, making this in v1 and then migrating to v2 would be a waste of time.

What I am proposing is that we make this https://github.com/openfoodfoundation/openfoodnetwork/issues/2072 a mandatory part of the spree upgrade.

6

ohhh @luisramos0 this sounds exciting - @sstead @MyriamBoure @Rachel what do you think?

7

@luisramos0 I’ve used behat in the past, is fitnesse in the same spirit? But yes in a nutshell I fully agree with you and I’m free to go forward with this. As this is part of the task that was assign to me and @maikel at our last meeting, I would love to have his opinion as well :slight_smile:

8

Whaooo ! That seems cool @luisramos0 ! Ok I never used those type of tools of course but if I understand well the idea is:

  • that as testers we will have some sample data set like the content of this file but that we will be able to update them as we go as we will be trained to ourselves commit PRs to change it. So we can iteratively enrich step by step this data set with what we need and that will be saved somehow for later testings. I like this idea :slight_smile:
  • then the second point is about writing acceptance tests with the framework tool you propose. Is it like “writing the test cases” we write today in the PR or stories for new features to check if we consider they pass or fail? or in the testing template when we write “test cases”? I’m not super clear about what it would be concretely… would this replace the first dot point about changing sample data through PRs in GH? I guess so. Anyway I feel like it is the good way to go, I guess it would “professionalize” a bit our testing processes :slight_smile:
    It would be good to “test” FitNesse to see what it looks like and get a more concrete sense of it.
9

First we need to see if other devs know this space and have opinions.

I read just a little bit more, looks like that turnip is really the cucumber that plays with rspec (we use rspec already). In these tools the data tables have to be programmed, for example, devs would have to create a “product table” that could be used in tests (testers could create their specific product list), or a “enterprise table”.
Fitnesse is another thing because testers can create database data directly and edit it as a wiki. It’s awesome but a longer setup/learning curve.

Interesting article about this subject and about turnip with rspec in specific, I think the beginning is useful even if you are not a dev (we already use capybara as well): https://medium.com/@bobbytables/acceptance-testing-with-turnip-and-rails-dbd7d65398f2

10

I agree with above :slight_smile:

11

I have no particular experience with these tools. I know cucumber exists and how it looks like but I’ve never used it. If we want to give this a try I’d go with Cucumber as is the most known tool and has a big community (I guess) rather than Turnip. But before that I’d like some things to be considered.

But before that we need to improve the basic data set

We’ve worked with @MyriamBoure on that doc you linked but we never actually implemented it because there are always 100 more things with greater priority. If we think this is now affecting our velocity, let’s find the resources and give it the priority to implement it.

We could train the testers to change this data as they think appropriate with a simple git workflow

That could be a next step but do we have the resources? changing this data means being able to code in Rails. Are testers ready to do this or will it add extra overhead to our already small core dev team? How much time will we spend training them? how much maintenance overhead will this add in the long term even if they manage to code them themselves? I absolutely encourage everyone learning to code but it feels far from our capacity training them to me, at least now.

What I am proposing is that we make this #2072 a mandatory part of the spree upgrade.

I’m fine with that but not more than that. I’m afraid of making the 2-0-stable branch bigger and the upgrade itself even longer.

After that, we could consider Cucumber IMO, which to me is different from implementing new sample data.

12

I feel like the discussion went a bit off-topic. Behat, Cucumber and Turnip are basically the same thing. But they are about writing automated tests. This topic was about test data. And I would like to throw in three options:

  1. We define test data and code it. This is a lengthy process and needs developers for implementing, but then it’s very easy to share between instances. It’s very good for basic data, but we can’t store configurations containing secrets, like a Stripe setup in there.
  2. We use fitnesse which allows more people (non-devs) to create the test data. I haven’t worked with it, but I guess it’s very easy to share, but we would have the same restriction of not coding secrets in a public database.
  3. We allow testers do take database snapshots on staging servers which are the base for new staging deployments. They can create the setup as they like and then save it. This wouldn’t be easy to replicate between staging servers automatically, but testers can do it themselves. They just add the data they are missing naturally while testing. It creates unique databases which can be a downside (not reproducible on other servers), but also be an upside (complex data reveals more bugs and resemble production data more closely). It also enables us to store unique secrets in the databases.

My favourite would be to combine all of these. We already have a very small set of programmed test data. Let’s extend it. Let the testers create and save more data on the staging servers while we work on adding more data to the programmed data pool. And let’s consider fitness as an alternative way to generate the programmed data to speed up the process.

All of these things don’t depend on each other and can happen in parallel. What do you think?

13

Fine for me @maikel. We can plan the work on 1. while someone first spends time thinking about how to best use 3. and then we spend some time figuring out how 2. would fit for us.

14

@MyriamBoure @Rachel I almost implemented all data defined in https://docs.google.com/document/d/1_va0Aw2yJ5BRe-HtuBT-OC8HNHIui0gvQ0uElWjDVs0. There is just one thing I would like to discuss.

The mailinator email addresses would leave all accounts open for abuse. I suggest to use invalid addresses like jane@example.org. We can still read all emails by using a dedicated email address that is configured in the mail method as “Send Copy of All Mails To”. It could be staging-emails@openfoodnetwork.org.

What do you think?

15

@maikel this is awesome :slight_smile: Looking forward to see this going further. I like your suggestion, let’s go with that!

16

Awesome @maikel!!!
is there an issue on GH for this? I’d love to have a look at a wip PR :slight_smile:

17

@luisramos0 I opened my pull request with most of the seeding data. It would be good to get some feedback before doing the next iteration: https://github.com/openfoodfoundation/openfoodnetwork/pull/3209

1 Like
18

@MyriamBoure @maikel @luisramos0 @sauloperez One small proposal for improvement after using this a bit:

would you guys mind if we put first names that would be radically different?

When I test mobile or when I use a dropdown for example, I only see the beginning of the enterprise name and it already happened that I changed something on backoffice for Mary and in frontoffice I checked Maryse’s shop on mobile trying to understand what was wrong :exploding_head:

So for dummies like me I suggest this:

  • Freddy’s Farm shop => George ?
  • Fredo’s Farm hub => Bob ?
  • Maryse’ Private Shop => Phoebe ?

Or we don’t put first names in the enterprise names.

19

Good idea. I got confused with the names as well. I like your suggestions.

20

yeah, confusing names.
what if we follow personas names?
Shannon and Jane?