Toward a roadmap for our API

Product Development
1

What is the need / problem ?

OFN has an API. It is used by the app, by Zapier, and increasingly by people wanting to use OFN and solve problems we’ve not solved. Increasing we are seeing the need for a clearly specified, well defined API.

For a long time the API has not been treated as a first class citizen. Much of the API was originally part of Spree. Although the API was brought into the OFN code base about a year ago the json responses still are based on the Spree implementation. In many cases this results in ugly json that requires a lot of post-processing to create something meaningful. This is particularly true for anything that touches adjustments!

We all agree that the API should be a fully supported feature of OFN - a first class citizen. There is reasonable agreement that this should happen as fast as possible. This post aims to make some first proposals of a roadmap for what needs to happen to make the API a first class citizen.

What does it mean for the API to be a first class citizen?

For the API to be a fully supported feature I propose the following acceptance criteria:

  1. Official OFN API docs - the current docs are listed under a personal swagger account belonging to Luis and marked at v0.1. We need an official OFN account to own the documentation. This is apparently very easy to switch.

  2. Clarity over which endpoints we are officially supporting now. Once endpoints are officially supported we will have a responsibility to users upon changing them. We will have a responsibility to fix bugs. Although we don’t officially have SLAs and Ts&Cs (another story) if the API is to be an official feature we will prioritise rollouts and bug fixes as part of our processes.

  3. We will have clarity as a team over how the API fits into our processes. At the very least we will share an understanding of constitutes a bug of different severity. No doubt many of our processes will develop over the coming months, and the API will be included.

Some bigger picture questions

A bad API will likely create more work for us than no API. Certainly more than a clean, clear, well designed API. API design is a skill. Do we want to invest in API design or do we have the skills in the existing team?

Some endpoints are really messy. How much time are we putting into the API now? What is in scope and out of scope for v1.0?

How do we handle endpoints we need (and are used) but would benefit from refactor before being officially supported? For example do we want to release the /orders endpoint in v1.0 as is? Or will we exclude from v1.0, refactor it and release it later? Or will we work on it before v1.0?

What are the next steps?

  1. Discussions on which endpoints will be officially supported in OFN API v1.

    • Who will participate in these decisions?
    • What factors are important to consider? Eg usability, need, work required
    • When do we want to know this by?
    • How will we prioritise work that needs to be done?

  2. Checking and confirming the existing docs for clarity and accuracy

  3. Agree some processes around the API

    • What constitutes a bug of different severity within the API?
    • Do we want to agree a process for API releases? For rolling out API changes?
    • Do we want to consider acceptance criteria for future API updates?

  4. Publish the documentation in an official OFN account at v1.0.

[EDIT]

What is the need?

To be able to CRUD OFN data on OFN data objects (eg enterprises, order cycles, shoppers, orders, products, inventory).

Use Cases

This is the beginning of a list of things that either we are currently using either the API, the Zapier DB link or reports downloads to do, or that people have asked for:

  • to maintain orders within an accounting system
  • to plan distribution routes
  • to create shoppers
  • to place orders
  • to extract data for packing lists
  • to extract shopper data for CRM tools and newsletters
  • to update and edit orders
  • to find enterprise addresses
  • to find order cycle details
  • TBC

[/EDIT]

A Roadmap for the API

To come…

Ping @Rachel @Kirsten @luisramos0 @Matt-Yorkley @sauloperez @maikel @Jana @Erioldoesdesign @apb

2 Likes
2

Nice, thanks!

I think first we need to make an inventory of what’s the current state so that we have an idea of how much effort would be required: some endpoints will be ready to be supported, some will require a lot of effort to be ready to be supported.

Yes, we do have the skills. We need to prepare a proposal for API best practices, we need to finish what I started “how to build an API endpoint”: API Development · openfoodfoundation/openfoodnetwork Wiki · GitHub

Not an easy one. Who’s going to make this call?
I think the orders endpoint is pretty decent but I understand we want to rethink/simplify the structure of how we list adjustments for example.

4

I think we need to document use cases for API endpoints which can meet several needs:

  • State the need for a particular endpoint and specify what it needs to do.
  • Document examples of how the API can be used.
  • One use case can cover multiple endpoints and demonstrate how they play together.
5

What problems do they want to solve that we’re not? Where is the proof of that increasing need? What does the API unlock for OFN? Who agrees it should be a supported feature? When was this agreement made?

I don’t doubt there is a need strategically for this, but it’s not articulated in the detail above so I’m not sure beyond the technical benefits / clean up from Spree separation what exactly makes this anymore interesting to the average instance manager than the spree upgrade was…it sounds like just another thing getting in the way of having features built. But it’s not just that, so what is that detail, the thing that will make the decision a no-brainer to anyone who reads this?

3 Likes
6

What a wonderful question @danielle

Why is the API a fundamental mid/long term strategic pillar for OFN?

There are already several places in discourse that address this question.
The answer is in slide 9 here:
https://docs.google.com/presentation/d/1dWbxwekBLtUMzj-9XuCXDH0Yib10P-tL6G7VcsmZ2n8/edit?usp=sharing (this is my presentation about API on the global gathering 2019).
Copy pasting from there:

  • building external shopfronts (and other tools that support our users)
  • integration with other systems (POS, other websites, Odoo, etc)
  • implement de-centralization of OFN, for example, managing distributed product catalogs
  • DATA interoperability
  • technological diversity (for example, in terms of frontend dev)

In terms of architecture: slide 5 and 6 show how the API can be the center of the OFN solutions ecosystem. Basically, the API is the center stone of anything related to breaking away from the monolith.

This is not related to technical debt or Spree separation.

3 Likes
7

Great!

And so the next question then, is of those opportunities the API unlocks what has the most benefit for OFN in terms of achieving mission / raising funds / bringing on more enterprises / developing new products for the sector? And what data do we have to back this up? What size is the opportunity in terms of external shopfronts, what does that unlock in terms of new customers or the such for our instances? Does POS integration have a greater benefit than the external shopfront? Why? How do you know? What does technological diversity provide OFN that makes it potentially equally/more important than any of the instance-benefitting parts? What opportunity does DATA interoperability unlock, why do we need this, etc etc?

The API as a standalone thing in our roadmap doesn’t feel as important as the things that it unlocks. Well, I would say this to be true from an instance manager’s perspective. Therefore, what we should actually be talking about from a product perspective is the opportunity, not the API itself. Be it a little thing like Kirsten’s query that broke, or whatever that first prioritised thing is, it will lead to part one of the work being to set up and support the API, and then doing the endpoint work for that specific opportunity.

So to sum up (or repeat myself :woman_shrugging: ):

  • firstly, what opportunities does the API unlock for the product in terms of our strategic direction?

  • secondly, how do we know of all the opportunities which is the one we should put first in priority and get included on product roadmap? And second? etc?

  • thirdly, how does that fit into our other priorities around network and reports and metrics and dependency management etc etc?

  • and then finally, how do we set up the API to then extend it and unlock that opportunity? I think that last question was answered by the original post. It’s the rest that isn’t clear.

8

Caveat: the only reason I would say the API deserves to be a standalone thing on our roadmap is because we plan to productise and sell it. Then, it will be considered a product in its own right and have all the strategic thinking around it and the discovery work will be about understanding the market this product will serve and the value derived to then understand pricing and take a lean approach to validate that there is product-market fit. So still, requires product thinking but if that’s the path OFN wants to go down then I see yep, API on roadmap.

9

Final thought: the API feels like it should form part of the technology strategy that should underpin the coming product strategy that will be worked on by the product team (incorporating broader community input in whatever way that’s planned out).

If the strategic path points to us having capabilities that require the API, then it makes sense to then develop a technology strategy and roadmap that identifies the path that needs to be taken from a technical perspective to achieve each step in the product path.

It would include stuff like like dependency management, API, services (if that’s what is needed), etc etc. The reason any of this done is in service of the tool as a usable and useful thing for our instances and their customers, so perhaps the conversation is extended to become the strategic path for both the product and technological aspects of the platform, rather than just the product view?

10

ok, I understand your point.

Let me bring an experience from Farfetch where I worked. They are a platform with quite a few similarities with OFN (it’s a platform for small enterprises that cant afford a tech solution themselves), and also some major differences off course. I am mentioning Farfetch because API was not just the top product priority. API was the top business priority. For example, back in 2017, Farfetch’s business goal number 2 was expanding to China (this ended in FF receiving 400Musd investment from a JD) but guess what was the only thing above China in terms of overall business priorities: API. I am adding this context just to clarify I am not inventing any of this. I am just describing what I have seen at Farfetch (and this approach was not created at FF). If we want to be a platform (we are in the business of platform coops), our API needs to become not only a product in itself but a top priority in our vision.
From this perspective, API is not a technical decision, it’s not even a Product decision. It’s a a business decision.

Having said this, I understand where you are coming from and I understand if API is put in a second layer behind as a tech solution for other business needs :+1:

1 Like
11

I think I tried to hard to be impartial in my post above :slight_smile: I forgot to add my informed opinion :heart_eyes:

I am not only saying how Farfetch did things. I am convinced this is how OFN should go ahead as a business strategy. Long term we must break away from the monolith (from being a single app). We will be an ecosystem of interconnected apps that integrate with many more apps. There’s only one way to achieve this vision. (how can we do this without an API?)

In a way this is about changing the basic identity of the OFN solution: we are no longer an ecommerce app that enterprises can use to be more efficient. We are a platform which is the foundation of an diverse technological ecosystem that supports the regeneration and growth of a new food system.

image
image723×47 8.4 KB

The API is really the stepping stone of this vision.

I am sure I am not the only one really excited about this :slight_smile:

3 Likes
12

100% agree. It’s about being deliberate and articulating what we believe the world will look like for OFN in a few years, which the API will underpin. This is the work going on in the strategy development, and will no doubt include the API, the network stuff, at least. It’s about putting the API into context with that visioning and associated roadmap, including measures.

1 Like
13

I’m finding this thread very useful in helping me understand api (ping @dthomas @JessC. One question re: our open source licensing and API. If/when we have multiple stable API endpoints, and another tech company wants to use these to build something - lets say a front end app for farmers markets (because this came up here during covid) - does our open source license apply to what they use our API to build? So - would they have to to license this front end app they build as open source? Anyone know? @Kirsten maybe?

14

@tschumilas thanks for flagging this

15

Instance Ts&Cs will need to cover use of the API as well as use of the platform.
Use of the API will be covered by a license of use rather than the license on the code.

It would be unusual to dictate users of an API need to make their code open source and probably stifles use and innovation. Instead you should be able to capture and charge for API use through your pricing model.

16

@Mario had commented on slack somewhere about this question of ‘productising’ and charging for API use. I know nothing about this. It seems on one hand like we should and on the other hand like we shouldn’t (I feel like there is a ‘standard’ of not specifically charging for the API but making sure your usage fees are adequate - that people would baulk if we tried to charge them per api call or something).

Prompted by the DFC discussion, I am wondering how / when we move this forward. Do we need to discuss it in Product Curation? It isn’t currently on the list - should it be?

17

To me it is a topic that is too big for just a product circle decision. We need a community consent that everyone agrees working on this becomes a recurring thing. The first step that @lin_d_hop has described is the correct one. We should discuss during a meeting the answers to those questions:

1 Like
18

So we need an API specific meeting - but the question is whether product-curation gives the nod for resources being used on what is effectively a mega-process inception. Perhaps already has as @lin_d_hop is leading on it, in which case perhaps it should be on the product-curation list

19

hehe this is interesting process-wise :slight_smile:

I think that Lynne is working on it as part of the product-pipe. So to me it’s on the same level as doing a competitor analysis or working on vision. We are at a very high / governance level for now on it, not at the inception level. Product-curation, as we have defined it lately (if I’m not mistaken) is only about the inception-pipe.

But maybe I just don’t understand yet what other people see in product curation?

20

I am highly interested in this topic. I worked at my previous company at something very similar. And the earlier this is tackled the better for the long term strategy. This can achieve wider adoption, far more reach and an ecosystem of services that can be really breakthrough.
If possible, (I am just the new kid on the block), I would love to attend to this meetings. Even if it is as a listener/passive contributor.

1 Like
21

Following on from the conversations in “Reactive Rails” we have a big strategic decision to make regarding our direction for the OFN API.

Without further ado - an interlude from Hamlet.

To dogfood, or not to dogfood

That is the question. Whether 'tis nobler in the mind to suffer an API as a side project. The slings and arrows of outrageous fortune of having the DFC, Or to take arms against a sea of troubled BE developers. And by opposing end them with business decisions.

Enough of that.

What has emerged from the Reactive Rails discussions are two potential OFN strategic directions:

Strategic Direction the First
OFN continues to dogfood its API. We take a FE framework like React that enforces a rigourous FE and BE split. We use the API to connect the app together and in turn begin to move in a direction of multiple servers and an ecosystem of applications, both build by us and not, that create the OFN community. We polish the JSON API for public use. The DFC is a plugin that consumes the JSON API.

Strategic Direction the Second
OFN ceases to consume its API between FE and BE. We use coupling framework between FE and BE like Reactive Rails. The JSON API becomes a side project. It is pretty clear that having two API side projects (JSON and DFC) is a bad idea so we move away from the JSON API and have our DFC implementation as our sole API. API work (bug fixes, endpoints, documentation etc )is prioritised alongside all the things.

Strategic direction the second is bold, and a big bet on ourselves. Personally I want to make sure we are all on board with the implications of this decision before we make it because it impacts our business strategy, our funding strategy, our delivery strategy… it is one of the biggest decisions we’ve made for a while.

Questions to Consider

  • What time implications does this have for usable OFN API endpoints?
  • What scaling implications does this have? How hard is it to move away from the monolith?
  • What are the budget implications? Ballpark investment required.
  • What are the business implications?

We have big decisions to make. Seems like we need a meeting for this.

1 Like