[API] API V1 Discussions (from the Portugal delivery team mini-gathering)

This post is to summarise some of the discussions on high level API implementation. What follows is essentially a proposal for the wider delivery team, summarising a lot of content discussed in other places alongside some of the discussions we had in person.

1. Documentation

Documentation will be automatically generated by RSwag gem. You can find an example of what this will look like here. Documentation takes info in specs and turns it into human readable Swagger docs. This documentation will be automatically updated when the code is updated, which is obviously excellent. There was some discussion that we might want to add a little additional detail to aid useability, like more descriptive details on attributes. We might also want to add a wiki somewhere that outlines the different ways filtering can be applied.

2. Pagination Strategy

We propose that pagination is implemented on every v1 endpoint. We discussed a few different options and realised that perhaps we just want to start conservative and increase numbers when we have more data about API use. Our proposed starting points are:

  • Set the max number of rows returned per request at 200
  • Set the default number of rows returned (when no number of rows is specified in the request) to 50. We recognise is low but setting the default to a low number seems sensible.

3. Authentication Strategy

Essentially we’re all in agreement that the authentication strategy remains unchanged between v0 and v1. This means:

  • Basic auth via login when API is accessed via browser
  • API Key as param
  • API Key in header

4. Implementation

The technical framework has been decided through conversations with techie folks in Slack and Github. We propose that new API controllers should be implemented with full CRUD functions from the start. @Matt-Yorkley suggests that there is no point implementing only specific actions on API controllers, that we might as well define and scope all actions upon creation of each API controller.

5. Bug Severity

We discussed how we assign API endpoint severity, and essentially propose that the same fundamental severity ranking will apply as now in v1. If endpoints are suddenly down these will be s2 at least. For performance issues we agree performance is a feature, as always. In designing endpoints the proposal is that we keep calculated fields (eg customer balance) to an absolute minimum and never show field like this on high level index. Endpoints should be nested. Attributes that require heavy calculations might be hidden by default and might exist only on :show. Datadog already monitors endpoints currently, it gives detailed information so we should be able to stay ahead or at least have good visibility on performance. This is important as for the foreseeable future the API will be served over production servers so performance issues could impact all users.

6. V0 Maintenance

The proposal is that v0 endpoints are not maintained going forward. Obviously v0 endpoints that are used by the app will need to be maintained specifically for app use as the app will continue to use v0 endpoints until we have fully migrated to ReactiveRails. To be clear, We won’t transition the app to use v1 endpoints at all. V1 of the API is an entirely separate product to the app. That being said, v1 endpoints might be used in modularisation processes eg /shops page might be turned into an independent and modular app when we work on Discovery, and this app would use v1 (or v2 or vwhatever). The same would be true of modular Shopfronts implemented via the API.

We propose that officially v0 endpoints are now scheduled for deletion and deprecated. This means that by default v0 bugs are not fixed (unless directly needed by the app). Enhancements on v0 not prioritised and rejected by default. V0 documentation won’t be maintained. We understand that a number of instances are using v0 but we propose that if changes are needed any instance using the API should support the development of v1 endpoints if they require changes (via the funded features process). The idea is that once a v1 endpoint is created, we declare an End of Life date for the v0 version (for any endpoint not used by the app we declare an EOL date). In this way the intention is to encourage the community onto v1.

2 Likes

Great! I’m super happy with these outcomes.

Shall we spec the performance? Since we have pagination for every endpoint and a maximum page size, we could put a hard limit on that, for example 200 records have to be served within 4 seconds. That would mean that the default 50 records are usually served in less than a second.

The performance depends on the server specs and the production database and therefore it would be best to measure performance in production. We can do that either via Datadog or add a little bit of code to measure the execution time and report to Bugsnag if a request takes too long.

We could then say something like

  • 4 seconds is fine,
  • 8 seconds is an S3 level bug,
  • 16 seconds is an S2 level bug,
  • 30 seconds is an S1 because we would like to limit all requests to 30 seconds.

Since Datadog APM seems to be already plugged in, it should be great to measure performance? (rather than adding more code to instrument API performance?).

4 seconds seems very long to me for an API call but it depends on the applications using the API. Would you have some use cases that are going to be using the API?

Hi Adrian,
Loads of work has been done getting to this point on our API discussions over the past year. On this thread you will find a lot of our journey, including notes on the adventure from React to ReactiveRails (which was key to the API discussions). Toward the end you’ll find this post that summarises some explorations of use cases and some loose prioritisation.

On prioritised use cases:

  • We’ll implement much as funded features (more on ‘funded features’ here, here and here)
  • The DFC, and in particular the v1 products endpoint, is amongst the first funded API work as the funding for this work has been confirmed. More here on auth but the next stage is products interoperability.

Thanks Lynne lot’s of reading :).

Just to clarify my question: if the API is used only for backoffice tasks to sync data between platforms, or maybe as the background loading step of a frontend, 4s might be ok. But if it’s used in some interactive tasks (in new frontends), I fear the UX might be poor.

My vote is - pick any number and iterate later with more data.

Hi there,

Here are some thoughts :wink: :

  • Use bearer token authentication from the beginning (it makes it a bit clearer how you handle permissions). You could then put the apiUserId, tenantId (entrepriseId) and scopes inside the token. As scopes, I thought of product:read, product:write, catalog:read, catalog write, entreprise:read, entreprise:write, ordercycle:read, ordercycle:write (inspired by https://guide.openfoodnetwork.org/basic-features/enterprise-profile/enterprise-to-enterprise-permissions-e2es and github scopes)
  • Use open api 3 for defining the api schema and model the api (you could have the file on a specific github repo so contributors can make suggestions)
  • Put something like Redoc on top of this file to make it nice to read
  • Consider graphQL (every ecommerce is following this path), I think it’s easier to reason about a graphQL schema than an open api one. (one I did for local food system logistic modelling : https://raw.githubusercontent.com/qalincalabs/koala/main/model.graphql)
  • I wouldn’t use jsonapi : it’s not as “pure” as json (adds clutter to you json), you will be tempted to hook it to your database and your API will look like your database (changing from postgresql to mongodb shouldn’t affect your API). If you want the nice stuff that jsonapi brings, just go for graphQL …
  • json schema rocks
  • make the json as simple as possible. API responses are used directly in automation tools (Zapier, n8n), templates (liquid templates). You don’t really want to rebuild your objects …
  • think i18n from the beginning (I work for OFN Belgium and we need the possibility to add a product description in french, dutch and german).

I forgot some thoughts :slight_smile:

  • The notion of events !! The ability to subscribe to events such as OrderPlaced, … (through webhooks)
  • GraphQL is super good for reads but a bit more complex for writes. Maybe an hybrid API might do the trick …
  • Endpoints are a good start to know how to compartmentalize the API : what would OFN look like if it was build with microservices (doesn’t mean you need to go this way but it’s good to identify the different contexts).

I identified the following big contexts :

  • MarketPlace
  • Entreprise
  • Customer
  • Catalog
  • Offer
  • Order
  • Shipping
  • Billing/Payment
  • Stock management
  • Analytics

You could apply vertical design on this : like split database, separate controllers into folders, separate API endpoints (don’t forget to group them in a main API for better developer experience) , build independent microservices (or use some from outside OFN)

Heyall: OFN n00b here, in process of evaluating the system for potential adoption by our local food co-op (in Portugal, as it happens :slight_smile: ), with a little systems integration challenge i am trying to scope out.

Specifically: we’ve got a central warehouse using Moloni SaaS for Stock Management and Invoicing (latter function being integrated with PT’s tax authority, is why we’re kinda wedded to this), which- tho it has some Zapier integrations with other SaaS platforms- offers no integration with OFN software as it stands.

So i am wondering: Is there any API endpoint currently available for such integration? I’m looking at this “products_and_inventory” report (via CSV download), and thinking this could be all the data we need for OFN<->Moloni interop, but am not clear what methods/ endpoints (if any) are available for production use at this time. ?

hi @walt - all our current endpoints are considered ‘not released’ but we are aiming to deliver a support release of products / inventory endpoint in the next few months. In the meantime Build, Collaborate & Integrate APIs | SwaggerHub is one that you could use, but please be aware it is not supported and will change

1 Like

Thanks @Kirsten -good to know. I’ll be watching this forum for news of a stable API release, and meanwhile will use the prototype to explore possibilities.

One Q: in the readme, it says “Talk to us to get your credentials set up.” Who should i ask for such credentials grant (as a tester) and how -can you say?

Creds on Katuma? @walt
@filipefurtado can help with that.

2 Likes

Conversation continues at API v1 Discussions (Aus February 2023)