Connecting food cooperatives

Exciting discussion here!

After all an open decentralized federated food network is all about interoperability and networking

A full data sync can be quite complicated and expensive. Apart from an initial sync I could imagine most of the time two systems connect, they would already be (almost) in sync. E.g. farming offline and getting connectivity again, syncing up the changes made in the meantime and downloading the latest order updates.

Afaics spree/ofn offers a REST-style API, i.e. request/response PULL-based data retrieval. In a dezentralized network of food participants, maybe some event-driven messaging could be “easier” to keep everything in sync? Kind of a PUSH-based publish/subscribe API, where updates get pushed proactively to connected subscribers?

E.g.: subscribe to change-feed since timestamp:

  • new producer…
  • new product…
  • updated price…
  • deleted product…
  • updated order status…

kind of a stream of changes to subscribe to
ordered by time and stays open for realtime updates

What do you think?

I see different requirement when we need to work with read&write vs. read only interop. Communecter most likely can start with read only API and doesn’t need to write. While we work on full read/write API we may already get started with read only by embedding structured data with https://schema.org and/or http://ogp.me

BTW for more broad approach to interoperability please check out https://valueflo.ws which I would like to make a core of https://github.com/ouisharelabs/food-dashboard experiment.

Ping @oceatoon, I think the discussion is for you :slight_smile:

Great to se some discussion on interoperability. I agree with @elf-pavlik that a read-only API may be sensible to start with, and a lot easier. It is good to know that Spree has an API supporting more advanced use-cases too. Still OFN has introduced quite some changes so I’m not sure how well this is usable (e.g. I wonder if access control for enterprises is present in the API).
OFN’s API documentation is probably (somewhat?) up-to-date as it was, as far as I know (also see the current routes file).

Interesting to mention RDFa or Microdata. It would certainly help search engines and spiders to access the data in a structured way (schema.org has a Product schema). I do wonder if developers of apps wanting to connect are familiar with this approach, where I feel REST is more prevalent.

e.g.
a mobile phone downloads (read-only) enterprises to the phone book
or two OFN instances share (read-only) enterprises in both directions

Which use cases do you have in mind?

how to detect deleted enterprises?
delete everything and re-download?
a REST query since=last_modified?

How “cheaply” detect that you are already in sync?

microformats, valueflows, ogp and schema.org products look exciting! A common shared vocabulary and compatible data types makes interoperability. Standards seem to emerge out of widely used best practices.

Beside data formats is the question of how to communicate updates (protocol).
Would we simply (restfully) http put/post/delete each other in real-time for everyone to stay up-to-date? What if someone goes offline for some time and then reconnects to catch up? Or would we constantly poll each other? How to handle participants joining and leaving an open food network?

Maybe we can (re)use some ideas from dat https://github.com/maxogden/dat Like git but for data. Similar to event sourcing or kappa architecture the main idea is to have an immutable append-only log of data changes that enables versioning and branch/merge synchronizing data (in a merkle DAG structure) like with git pull/push.

Rather than have to insert Microdata or RDFa into HTML, what about to generate JSON-LD and insert that into pages? Many of the OFN pages now pull JSON from the server anyway. This could make overall implementation easier?

:+1:

Ping @sylvain :slight_smile: You have some ideas about this tech part I guess?

Absolutely. I totally agree with @pmackay, as using json-ld would make the API understandable by other web applications. If LesPaniers bio du Val de Loire do the same, you’re compatible!

Yes, standard data formats would be nice to have. Until common standards emerge from widely used best practices we can simply translate between different data formats.

What’s not clear to me is how a replication protocol would work :confused:

Is it possible with the current OFN rest API to get the products that have been modified since some (last sync) timestamp? How to detect deletes? What is soft delete mentioned in the API docs?

Most of the time two systems connect they will already be almost in sync. It would be great to quickly cheaply detect that… Any ideas?

@wvengen afaics you do quite some sophisticated data synchronisation between the central product database and various foodsoft instances. Do you delete and reimport everything to catch deleted products? Are there also (two-way-sync) changes uploaded from the instance to the central database?

@orangeman I am a programmer / designer community activist / CMS person
interested in using OFN here in the states for local food emphasis in
Colorado and upstate NY initially.

I have been working on getting a local copy of the software running via
docker or vagrant…

Once I have a local copy to experiment with, I’d be happy to construct a
query for you against the database that could be used with the rest API for
our future specialized needs like you are requiring…

all best good vibes to your work!
Greg

1 Like

Thx @NorthernColorado Sounds great!

How would you go about this?
extending the rails app rest api?
or build some kind of api node
that uses the same postgres?

Are you using the ansible playbooks to build the docker images?
Currently I’m stuck at the last webserver ansible task: update unicorn
it errors: "Could not find the requested service "‘unicorn_openfoodnetwork’"
any idea? @pmackay

Edit: ansible debian topic Connecting food cooperatives

Hi Orangeman…

yes, my thoughts were to add a new db class that the api talks to, and make that result available throygh the rest api…

looking here for how ‘unicorn_openfoodnetwork’ could be named as a service.
what order did you get that error in? were there any other steps completed beforehand?

thanks
Greg

ref1: https://github.com/openfoodfoundation/ofn-install/tree/master/playbooks

ref2: http://www.mzoo.org/open-food-network-on-new-vps/

trying product post example from spree product api docs
http://guides.spreecommerce.org/api/products.html#create

curl -X POST -H “X-Spree-Token: abc…”
https://www.fairteil.de/api/products?product[name]=Headphones&product[price]=100

the api responds that supplier is missing but supplier is not part of spree products, right? At least the product json from the api does not contain a supplier. Apending product[supplier]=foo seems to be ignored by the api

Is anyone using the product rest api so far?
write a separate ofn product api?
or decorate the spree api?

it seems to work! with supplier_id instead of supplier

curl -X POST -H “X-Spree-Token: xyz” “http://localhost:3000/api/products?product[name]=Headphones&product[price]=100&product[supplier_id]=2&product[variant_unit]=items&product[variant_unit_name]=kiste&product[unit_value]=1&product[primary_taxon_id]=5

How to detect (and handle) the case when the same product changed in between a sync on both sides?

@NorthernColorado @orangeman hi guys, could I please suggest to create separate threads, or perhaps better to raise GH issues in ofn-install for problems with the install process? Its certainly not strongly related to this thread so would be better pulled out.

Hi orangeman

I was reading through these yesterday to help answer your question…

http://guides.spreecommerce.org/api/products.html

https://spreecommerce.com/blog/updated-rest-api

http://guides.spreecommerce.org/api/variants.html

it sounds like you may need to add a supplier valiue or tell the api to
stop asking for it…

We shall figure it out!
G

Here a product data sync thread
Product data sychronization

I’ve mentioned the Diaspora project here already:

It is a decentralized social network.
Perhaps this could be very interesting to look at in terms of decentralization and privacy for the OFN project as well.