Global Gathering 2020 - Day 10 Presenting the Data Food Consortium (DFC)

International Community
1

This post feels as long overdue and I apologize for that.

What is Data Food Consortium? It’s an initiative started and currently hosted by Open Food France to create and use an interoperability standard for short food chains tools / platforms.

What is interoperability? The ability for tools to “communicate” together.

You will find here the slides I’ve used during this gathering session:

And here the link to the recording:

https://drive.google.com/file/d/1w3gAmWWZ67m7A20Ty-eM6D9aNiHYbp4M/view?usp=sharing

It’s not a topic that is super easy to understand at first, and I don’t want to write a novel. So I really encourage people to try to follow at least the first 30min of the recording to understand what this is all about.

Here in discourse I would like to talk about the next steps for OFN.

Why is it interesting for OFN to follow a standard ?

As mentioned in the presentation they are many use cases, but we could argue that this would only be achieved once many other platforms and tools do accept the standard.
True. But I believe that the main thing that got me excited about it is the potential for current OFN instances to become interoperable.
That means plenty of stuff :

  • Network without borders. Producers would only need 1 account in OFN and sell through as many OFN instance as they wish. You may argue that this only works for Europe, but that also means we can forget about our rule that says 1 country = 1 instance. We could have many instances within a given geographical perimeter. This could enable local instances, but also instances with particular rules. For example in France the organic association contacted us to launch their own instance, but they want to accept only organic producers on their instance. In a world where all OFN instances can communicate together, this is not an issue: producers selling on this instance could also be selling on the “national” platform with the same profile. The “national” instance would not “loose” them.

  • Mapping! Once v1 of the DFC standard implemented, we could build an OFN Map on the global website with all shops all over the world :heart_eyes: Yes this means we will need to map our taxonomies against a standard taxonomy. I will publish a dedicated post on that as soon as I have worked out the details

  • Following a standard might require us to change our login system (in order for tools to communicate easily, they need to know that the user is correctly authenticated in all tools). Sounds boring but this actually opens up the opportunity to offer more easy solution to login to the platform. The discussion on this particular point is happening here: OpenID / OAuth2 authentication solution for DFC (and maybe more)

Challenges for DFC’s within OFN next steps

Interaction with global core dev team

During the 2019 gathering, it was decided that @paco should become the OFN dev lead within DFC. It means that François became in charge to build the first brick of the DFC standard within OFN. This was chosen because we didn’t want to impact the OFN pipe with DFC work.

The first idea of DFC was to implement full semantic API . So the original idea was to develop OFN’s API and use a tools called Semantic Bus to convert OFN’s API into a semantic API.
After some iteration, the DFC tech standard evolved to something more easy to handle: simple JSON-LD format.
As this was more easy, François developed the v1 of the DFC engine directly on the database.

From the discussion with @luisramos0 it appears it would have been better to develop the DFC API on top of the OFN API, to be able to develop the OFN API at the same time.

So having François working totally separated from the OFN core team of developers might not be a good solution. We need to find a better way of communicating (maybe discourse + a monthly call?)

ping @Matt-Yorkley @maikel @apb @sauloperez

Amount of work to deliver

DFC is currently funded. It means that the work making OFN “compliant” with DFC can be paid for. But this funding is limited in time. If OFN is not capable to show progress (finish step 1 by mid 2021), we might loose the funding opportunity to another platform.
I think it would be ok to leave DFC make its own journey without OFN for now because we have other priorities. But DFC will need to know soon if OFN does not want to be implement the prototype.

So I don’t see any other choice than speaking about it at a product curation meeting and make a plan (either we make room for review /strategic decision and sometimes developments within our pipe, or leave it aside for now) ping @Kirsten @sauloperez @Jana @lin_d_hop But happy to discuss to see if they are other options.

The case of the DFC engine

DFC compliance is currently developed in an engine separate from the OFN main app. I think we could start to think in the future about the way we want to maintain engines. We could say for example that some engines become “core” engines, so the core team maintains them. But we could also choose that the maintenance of the engine is externalize to another team.

Today the DFC engine is maintained by the OFN core team (yeah we never managed something that would never impact the pipe). Should we continue to do so?

2 Likes
2

Famous last words :smiley:

3 Likes
3

Thanks Rachel.

Re mapping: the global map is something it should be very easy to accomplish with the rest api already. I think the coordinates of the enterprises are already there available. The rest api also have a endpoint to create enterprises so it should be straight forward to read all enterprises from existing instances and load them into a single instance to see that map. A few tweaks to the code may be needed.

Re the engine: I think the core devs now agree the DFC engine should be removed from the main app and moved into a separate github repo and included in ofn as an eternal gem (this is something easy to do, mostly cut and paste the code).

re discussing DFC in product curation, in my opinion, it’s pretty obvious the global budget should be invested in getting our existing rest api into a good level that can be leveraged for so many use cases. We should discuss the work that needs to happen on the rest api in the product curation meeting.
imo having DFC api development in the global team is a diversion of attention we shouldn’t take. I think we can continue to support other devs doing it (like it happened so far).
I think OICD is an exception and can be handled in a different way. Because it’s part of the rest api roadmap, we could maybe get it into our pipe: OpenID / OAuth2 authentication solution for DFC (and maybe more)

1 Like
4

I feel my inadequate understanding alongside my strong desire to understand and move all this forward. I am concerned that a product curation meeting may not have adequate space but happy to try. It seems like we need to have a detailed discussion about the OFN API and what our strategy and priorities (and resourcing) might be there alongside / before we can commit anything re. DFC? I wouldn’t want to lose the opportunity for OFN to be implementing DFC.

Was there also an issue with @paco maybe not having enough time to continue and perhaps needing another dev to pick it up? Is that something that should be under consideration in dev hiring?

5

@luisramos0 I understand the fact that the DFC project is not amongst priorities of OFN. The existing code is almost autonomous and we could extract it inside a gem easily. About the OIDC integration, I was really busy lately on personal issues and couldn’t advance much on this topic, I will try to continue this week.

@Kirsten It is true I don’t have much time and more resources will help to move on with DFC integration. Internally, resources are limited and workflow quite intense, so it makes sense to look for external help, I would be glad to help and share about DFC project.

6

Thank you all for your quick feedbacks!

Yes of course, but using a standard for data can really make it interesting. Just mapping addresses is nice, but being able to search across common taxonomies for example is nicer :slight_smile:

Alright I wasn’t aware of this :+1: Where was this discussed?

The thing is, currently we have the opportunity to not use the global budget for this, but a dedicated budget.

If we agree to this:

imo having DFC api development in the global team is a diversion of attention we shouldn’t take. I think we can continue to support other devs doing it (like it happened so far).

Then it is obvious to me that we should stop OFN’s involvement in DFC and not pursue trying to implement the standard.

I think OICD is an exception

In terms of funding, we cannot have funding from DFC just for OICD. So if DFC is not a priority for OFN, OICD should go as a wishlist and be prioritized by the community when needed.

We should discuss the work that needs to happen on the rest api in the product curation meeting.

First we need to be sure the community agrees it is our next up priority :slight_smile: This is not clear yet.

7

re converting the engine into a gem.

I think this was on the dev catch up in the gathering, I briefly mentioned it and Maikel said: “yes, that makes sense”. It’s very easy to do and it doesn’t have implications. When the DFC work resumes we can discuss this with the dev doing it and evaluate this in detail, ok?

I think the question is who is “we”? There are multiple entities involved here. From an OFN perspective we can continue to encorage the DFC adoption and implementation but not as part of our core dev pipe (where the I think the rest api should be the priority). We can continue to do what “we” have done so far, to support other devs implementing it. So, as a joint project between DFC, OFF and OFN we can continue to support and drive the pilot in OFN. OFN core team can continue to support it as we can with tech guidance and code reviews. Which I think is a lot! From an organizational perspective we could say that OFN considers DFC a business priority by assigning it some of its scarce dev resources to support and review DFC pilot dev efforts done by other teams.

re rest api,

ok, fair enough. In terms of rest api I think it’s about the opportunity cost of every month we postpone getting our rest api in shape. It’s urgent.

8

You are right I should have explained that better :+1:. Here is how I’m seeing things:

  • OFF kicked-off the DFC idea and hosted the legal structure to start the project
  • Once it became a project that could impact the software, OFF turned to OFN global (last gathering) to understand how this could be done
  • After Mai 2019, @paco became OFN global’s tech representative within DFC
  • DFC is transitioning out of OFF’s legal structure and since September has a dedicated community and project manager. This person has among their task to create a separate legal entity with all participating platforms.

As for my personal status: I’ve joined the project in January 2019 as a contribution from OFF on product & UX skills. But of course I also have the double hat with OFN global as I’m helping François navigate into OFN’s product and processes and making sure OFN delivers on time.
It’s with this second hat that I’m writing this post: “we” is OFN global here.
To ease things, we can consider that OFF is not part of the equation anymore other than being the legal representative of OFN global in France?

Does it make sense? :see_no_evil:

Maybe I’m wrong but I don’t feel we can continue as is if we want this work to be funded. @paco has not enough time availability. Even if he does, the tech guidance of OICD and CRUD actions on product catalog are going to take OFN core devs some time to agree and decide on the best approach. I’m struggling to see how these discussions will take place outside of the pipe…

Same if we hire new devs who would only work on this? I will let @paco answer what he thinks about it, but it seems to me that doing something like DFC without knowing OFN is not working. And even if we would do it, aside of core devs, who can onboard a separate team :face_with_head_bandage: ?

My conclusion is that if we continue with the same time commitment, it’s highly possible that we don’t meet the DFC deadlines.

If this is a project we see as a volunteer project on the side it’s fine.
But if it’s supposed to be a work funded by DFC then I’m looking for help on how to make this work :slight_smile:

9

ok, thanks for clarifying.
It makes sense.

< my-opinions >
I dont think it’s a lot of work to make decisions and support some else to implement DFC phase 2, that’s why I think global team should support that.
I think it’s a lot of work getting DFC phase 2 done and that’s why I dont think global team should do it.
In my mind DFC is a research project, with the dev budget we have right now, I dont think we can take such a project, even if funded, there’s too much in our plate already.
From the rest api perspective, OICD is only a nice to have. It will come down on priorities if seen only as a rest api need.
If DFC needs core devs from the OFN global team to implement phase 2 as part of the global pipe I think we should drop the initiative in OFN. If there are funds available, let’s hire a dev to do this work outside the global pipe.
This DFC api road/project is not a short one if we ever want this to really work, it will require some significant investment (I mean after this non-trivial pilot phase). I find that cost outrageous mostly because I’m looking at the simplicity of just making the rest api work properly.
< /my-opinions >

1 Like
10

I’m looking for the ‘have cake and eat it too’ answer -

  • it seems that @luisramos0 opinions is that OFN Global is happy to continue to support someone else doing the work, whether that be @paco or other
  • and/or might need to be done in the way we are talking about contract stuff i.e. that there is an overhead paid to OFN to contribute to product, design, testing etc - on top of the payment to the dev doing the work
  • DFC has funds to pay for that. But there is a need / desire for that person to be having some immersion in OFN as well
  • it seems that we have more likely looking devs in the current recruitment round than we have jobs. However I am not clear how many of these people are front-end only and wouldn’t be able to take on DFC work - would we have been recruiting for someone else completely if actually looking for this? Or might we have someone in the wings who could do it?

If putting the funds directly into the global pipe enabled us to hire an additional new potential OFN person, who will work initially and primarily on DFC, would that not be pretty much the same outcome?

To me this is similar to when we pulled Weights & Measures into product curation because there were volunteers working on it, and therefore we made sure we were ready to support them.

I am not sure if the resistance here is to the DFC vision / implementation in general vs just going REST API <-> REST API in every case. Personally I am very compelled by the DFC vision, but I have no real understanding of the technical complexity it might add

11

Thanks again for all your inputs, it starts to take shape into my head but I still have a lot of questions :see_no_evil: .

hehe me too! I’m still unclear why this is turning as a discussion REST API vs DFC API. I wonder what’s blocking us to do both :thinking:

DFC work is purely backend / not visible for the end user. All current product / design / testing is on the DFC prototype, so completely out of OFN. This will stay this way unless OFN wants to start building an interface to use DFC API’s. But this is not the scope of the topic here.

@Kirsten you mean back-end only here right?

I start to feel it would be more effective. But maybe I think about it with the wrong hypothesis? I’m mainly thinking this because so far in what our competitors are doing, I don’t see a massive work being done on DFC. So I’m not seeing a scenario where a person would be “primarily” working on DFC. More like 1 day a week at max.
So maybe we need to dive a bit more into estimates? In any case that would be interesting feedback for DFC. The standard became already more easy as other platform in France started to work on it. Maybe there is a similar case here? OFN can shape the DFC standard to its own needs, it does not need to be something that is forced on OFN. That’s actually the whole point of the DFC project: get platforms to agree how they want to communicate data.

I have a question for the separate repo @luisramos0: when doing that, what does it imply for instances who wants on their server to have OFN code deploy + the DFC code? Would they need a separate and dedicated deployment that is out of the current sysadmin pool? I’m trying to assess costs in each scenario.

12

It’s totally different Kirsten. Weights & Measures was complete. This is a huge initiative that has just started.

About doing both. We have a very limited budget, we need to careful select the projects we work on.

Getting this to work in OFN will take not only time but it will significantly add to the complexity of the OFN solution that will have a continuous cost to maintain.

A dev outside the pipe and a dev inside in the pipe. I think it’s a huge difference in terms of cost: our attention will be diverted to DFC instead of where it should be: our core initiatives like network feature, etc. I am worried we are making this decision based on money.

I think that’s a big problem. Not many of us know what DFC is.
The main problem is the core team inheriting a second API. How can we afford maintaining 2 APIs? This is what I find a bit shocking. This is a big strategic product decision that will cost us a lot of time and money in the future.

I think we need to hear from more people.

13

Wow, first of all: I’m super excited by the prospect of an API to work with other instances and maybe with other software as well. I’m very grateful for the DFC work so far.

Now we are getting to crunch time on a decision here. The implications could be huge.

It looks like nobody wants two APIs within OFN. And maybe that’s where the friction with the DFC code so far came from. We all want OFN to have an API though and I don’t understand yet why it can’t be the same for OFN and DFC.

My vision would be that we focus on the OFN API as @luisramos0 suggested. It’s really important for our mission to provide commons to be interoperable and reduce complexity. My outdated knowledge from 2019 is though that we can make the OFN API compatible with DFC. And in cases where this ideal is not practical, we may be able to introduce another layer that converts OFN API into DFC API.

In order to make this work, the API (including DFC) needs to be a priority for OFN. If someone external is working on it, we still need to have good communication with the core team to make sure that the solution is good for everyone.

I think that we can do it but I’m not sure about the deadline mid 2021. I don’t know how big DFC step 1 is. We would need to map out the work and estimate it to make sure that we can delivery within six months.

If the global team sees this as priority, I’m happy to work on this. I’m a big fan of APIs. The timezone difference between Australia and France could be a hurdle. While timezones have been a factor in the delay of mobile, I do think that delays in responses for days or weeks were the biggest problem.

1 Like
14

Hi @maikel DFC is json-ld, REST API is not. Having a single API with a mix of DFC (json-ld) and vanilla REST (no json-ld) doesnt make sense imo. I think it would be really useful if you could have a look at the DFC data format: JSON-LD. I think mixing the two would make a mess. I think DFC API will have to be on the side or on top of the REST API. The OFN REST API should not use json-ld like DFC.

15

@luisramos0 Can you clarify the difference between “converts” (what Maikel’s said) and “on top”?

For a non technical person like me it’s the same :see_no_evil:

16

The engine and the gem solutions only differ in the location of the code. But now that you mention it I think along with the gem we can have an ofn-install config to activate or deactivate DFC API. It would work as a plugin: more code, more risk. So, while it’s not fully functional I’d recommend deactivating it in most instances.
But if we want we can keep the gem always there in all versions and deploys like any other gem: stripe, paypal, rails, etc.
The main advantage of having a gem is that it wont break the OFN build and the maintenance/development process can be a bit more decoupled.

1 Like
17

Due to my limited knowledge please forget this comment if I’m totally off topic :smiley: On the project for the CSA network we have used API platform: https://api-platform.com/ to create the API. What is interesting is that it makes the API compatible with several data format:

image

Also it is uses swagger for the doc.
Maybe this is a PHP specific framework but I thought I would mentioned it in case a Rails framework exist as well.

18

yes there is an option to have the same API produce both formats. I think that would be crazy in our context. Maybe a good option for smaller/simpler apps? Also, this would make the endpoints of both APIs be the same or similar at least, I dont think thast’s what we want because of the difference between the DFC ontology and the OFN data model.