Testing OFN v2

Now that we are about to be ready to test OFN v2 in staging we need a strategy to do it.

First the scope, we separated the full scope in 2 and we have “spree upgrade phase 2”.

We will reach a green build soon and we already have OFN v2 staged.

So, there are 3 points in time to consider:
1 - now (almost green build)
2 - green build phase 1
3 - phase 2 dev ready

We could wait until the end to start testing, I dont think that is the best approach. Early feedback from testers will be very very valuable! So I recommend the following approach:

At point 1 in time, NOW, I think a couple of hours of testing time on v2 will be very very useful to get a general sense of how the system is behaving. There are know issues, the only significant one is that the orders admin page is not working yet (we could wait for this page to be working to start testing).

At point 2, phase 1 green build, we will need an extensive validation of the system by testers. This will allow us to fix any issues in the core part of the system without having to wait for phase 2.

At point 3, phase 2 green build, we should validate not only the non-core features now working but also the core of the system with the new changes (not many surprises are expected here but I think it needs to be done, it’s a major release afterall).

What do you guys think about this? @Rachel @MyriamBoure @sstead @sigmundpetersen

An integral part of the test strategy is the definition of what tests to execute @sstead has started a doc for this. Maybe you guys can use this thread as well to discuss the test documentation/procedures for v2. or maybe create a new thread…


@luisramos0 one clarification question: how to separate the “core” from the “non-core” ? Where is the line ? Did you make a list of the core features so we know what we can test in 1 and 2 ? Like private shop, is it core? Tagging ? etc.
I understand then in 3 we need to test everything as deeply as possible, but we need to draw the line to know what to test on the first round.

When we understand better what is core we can work with @Rachel and @sstead to specify more into details the scope of the stage 1 testing. We can run the test one time after order page is done, then we can run them a second time at step 2.

And then we can prepare for step 3 and make a more extensive list of all features to test. We can use maybe this list I had done 2 years ago (so needs update) https://drive.google.com/open?id=1kuOkkg-PIAcWGYL5Ru8COlKoSN8bwFPgOvcix0HQb_o but it was pretty detailed normally…

the issue I linked has the list of non-core features: https://github.com/openfoodfoundation/openfoodnetwork/issues/2953


  • reports
  • subscriptions
  • product import
  • account invoices
  • paypal integration

everything else is core.

Oups, sorry, I should have got that, thanks for clarifying :wink: @Rachel @sstead I greyed in the file that Sally shared https://docs.google.com/document/d/1KAUBztCvoJYTCcN9g1wai_XLExiFkBZQXw3W4XRwqkY/edit#, the things we shouldn’t test at that stage, I added the missing ones as greyed so we have them for next stage testing.

Let’s start testing parts untouched by “orders page” ! @Rachel given the time you have next week I might be able to give a hand. And hopefully @sstead can do a bit more next Friday !

1 Like

so, two months later and we are well into phase 1 testing (point 2 above).
we have this epic to track the bugs: https://github.com/openfoodfoundation/openfoodnetwork/issues/2743
Currently only one major bug: product cache not working #3391

I think that, between now and the final round of tests, we need to fetch data from one of the main live instances into a staging server so that we test v2 with real data. What live instance should we get the data from?

Just to keep track of what I’ve added on slack:

There might be more than just product cache, given for example that you cannot update stock info: https://github.com/openfoodfoundation/openfoodnetwork/issues/3573
Things that haven’t yet be tested:

  1. Enterprise fee (waiting for some test cases)
  2. Stripe (waiting on French staging to be cleaned up, but moving to AU staging will alos help out)

Tests cases are documented here (WIP): https://docs.google.com/document/d/15EdXKb4B3BS8gekD6_9PDdPTZsjUCAVS2011zRf55lM/edit# any feedback would be awesome as it would also improve testing in general :slight_smile:

1 Like