Reports Project

The Objective

To create a report project roadmap that systematically addresses all prioritised feedback and bugs identified in github, discourse, user and admin feedback to result in a delightful user experience.

For more background see;

The Need

User and admin insight indicates the current report system no longer meets requirements. High level feedback is to meet requirements it needs to;

  • be able to customise report creation and remove any manual process

  • consolidate reports to reduce complexity

  • ensure no data fields are missing

  • have consistent use of data fields and labelling

  • include an intuitive user experience including add default dates, add favourite function, and being able to reorder fields

  • export to an alternative format to CSV

  • fix existing bugs and performance issues

  • remove unused fields

  • reduce manual testing of reports and ensure test coverage includes all filter options in the UI

Complete;

  • Requirements were rated based on value to the user, value to the admins, priority level and technical feasibility
  • this was a first step to help us see where the cluster of priorities lie

Next steps

  • a workshop will be held to play back requirements analysis, discuss and agree technical strategy with all key players who wish to be involved, and create first draft of the roadmap
3 Likes

Awesome! I think we all know reports is probably the number one part of OFN that desperately needs a revamp. It’s critical for a lot of users, and is bad on many different levels (performance, UX, etc). I’ll give my initial thoughts here before I dive in to the details of the requirements/roadmap.

I see a really broad range of problems in reports, some are technical and probably quite difficult to resolve, some could be fixed with small tweaks here and there.

I also see that what we want to do here is actually an incredibly common task; generating reports from some datasets and displaying them or making them downloadable in spreadsheets (or other formats). As such, I think this particular set of technical problems will have already been solved before, many times over, by other people, and we can probably just utilise a “generating nice reports in Ruby on Rails” solution that already exists, instead of painfully reinventing the wheel.

I’d recommend a bit of a spike on existing reports solutions that we can use via gems. 60 seconds of browsing brings up several potential candidates with taglines such as “Easy SQL based report generation with the ability to accept request parameters and render multiple formats”:




It might be that we can solve almost all of these problems at once with a minimal amount of effort by just switching to a nice reports gem, so I think it’s worth investigating that option and seeing what comes up.

3 Likes

hey @Matt-Yorkley can you explain a bit the difference between those gems and metabase?

Hi Holly. I have added my scores to the doc for user and admin value judgements where I have an opinion.
Looking forward to see how it progresses. Thanks for starting the journey

Metabase is outside of the app, and does different things, and will just be used by instance managers for now. This would be in the app, and would handle the things like displaying the tables when a reports page loads, or exporting a selected report for a daterange to a CSV file (from the UI, as an enterprise user for example).

Metabase is outside of the app, and does different things, and will just be used by instance managers for now. This would be in the app, and would handle the things like displaying the tables when a reports page loads, or exporting a selected report for a daterange to a CSV file (from the UI, as an enterprise user for example).

hmmm but if the only difference is whether it is in or out the app I still think we should include it in the potential solution and compare the advantages and disadvantages (you can download CSV with metabase, even better you can download .xls). I know that in France users wouldn’t mind using an external app. They already do it because all of our scrips are on spreadsheet (online or local). Don’t know about other instances.

I think in terms of big push forward we can going to be comparing ease value of Metabase vs API vs Gems to see where come to. The perfect solution may well be all three… but our world is far from perfect :-/

I can sense a big pros and cons chat about our future reports visions on the horizon… We’ll want a lot of voices in that discussion.

@HarrietHolley I just need a bit more clarity on the categories

  • value to user is an enterprise user? i.e. farm or hub?
  • value to admin is super-admin i.e. instance manager?
  • do i have a total number that i as an individual can put in priority? just thinking to force consistency on how many ‘priority points’ we can allocate?

on closer investigation i am finding this a little overwhelming and wondering if it might worth attempting some grouping / summarising prior to voting? It’s pretty hard to see which are duplicates, versions of similar things etc from this. I’m wondering if it is best use of time to have everyone trying to apply constrained brain space to working that out separately, all interpreting things differently?

started to look at this, and I concur - we are likely to end up with 4s & 5s everywhere. So it will be a process in dropping off the list 3 and under - which will be maybe 6/7 things only. I think we need to agree on how we will take a more critical and hard perspective. Like - what are my top 5? That’s tough - but it might be more helpful.

Hi everyone, thanks for looking at it and for the feedback.

Am happy to change the approach if it’s easier and more time efficient.

@Kirsten for the groupings, does it help to filter by category?

To start maybe we could rate the category, rather than the individual requirements? Once we have the high priority categories we can have a workshop to get into the detail to avoid misinterpretation.

Does that sound like a reasonable approach?

@Kirsten to answer your question hopefully this helps :slight_smile: I referred to ‘admin’ as an internal user, one of us, and ‘user’ as a customer, and external user.

So to align with OFN terminology the ‘user’ would be a farmer, hub owner, end customer or anyone using the system that is not an admin or one of us.

I am still getting to understand all the user types and terminology but hopefully that helps?!

Great thank you Louise! How’s your availability for a workshop on the 13th May? Will be good to meet you!

Cool, a lot has been added to the report since I first looked at it. Seems like it’s still evolving with the incoming feedback. Nice work @HarrietHolley :+1:

I like the suggestion of shifting things towards the API here! The technical process for generating reports is basically three phases:

  1. Fetch a bunch of data from the database
  2. Do some data-wrangling to organise the fetched data in the way we need it
  3. Render the data; either on the screen as HTML, as a downloadable CSV / XLS / whatever file, or maybe as json from an API endpoint for external consumption

The current way we do all three is quite problematic and all mashed together. If the first two phases were done nicely on the API side, the rendering in different formats could be separated, which would be really good. I still think both the initial data fetching / wrangling and the subsequent rendering in different formats might be something that a reports gem could potentially do very well, but as I say it would need a spike…

As a rambling side-note:

I feel like this is one of those moments where we are faced with the choice of two approaches; ignore the structural problems, delicately plaster over all the cracks, and cross our fingers that it doesn’t continue crumbling, or: smash the crumbling parts down with a sledgehammer and build something in it’s place that will last.

When the maps page was taking 54 seconds to load in production, I approached it with the cautious mindset, tried tip-toeing around the existing code and looking for small ways to improve it without disrupting things and got absolutely nowhere. I switched to the sledgehammer mindset (ruthlessly ripping apart any existing code that got in the way) and within about 3 hours the maps page was loading in 0.7 seconds.

If you wanna make an omelette, you gotta break some eggs.

2 Likes

@HarrietHolley just to be sure how to fill the doc: if I had a 5 in the priority column does it mean that it is high priority for me or low priority?

EDIT: also I’m not sure how to fill the value for admin: currently if it has a high value for users, it means the admin will be involved to make it happened so it is taking time to the admin to do it. So… it ends up with a high value for the admin as well… should I lower down the value for the admin in that case?

Hi Rachel,

Thanks for looking into this :slight_smile:

So the context behind value to admin and user columns is, I am trying to make sure we keep the external customer/end user needs at the heart of what we create (farmer, shopper, hub owner etc).

I imagine some requirements are of value to both but this approach validates we don’t accidentally create a system which addresses all of our issues and miss some of the end user needs.

Does that help provide clarity?

And yes the higher the number the higher the priority. That way the total scores can be used during analysis.

Hope that helps!

and this is why you are tech lead on this project @Matt-Yorkley :wink:

Hi All,

Thanks all so much for your input so far. I have caught up with most of you individually and good progress has been made so thank you for you for taking the time.

I am hoping to look at analysis this week on Wednesday morning Bangkok time. Does that allow enough time for anyone still wishing to input?

To make it easier I am removing the ‘priority column’ as this will come from the overall score. Matts dealing with tech feasibility so it’s just the two columns (value to user and value to admin) we are looking for 0-5 ratings on.

If you have any questions or problems with the timeline please DM me and we can find a work around.

Many thanks!