From open source software to open source interoperable modules... co-sensing the future

Integrations
1

Continuing discussion from Inception session: product chain data model

I am involved in lots of commons “prospective” networks, with both “product” and “tech people” in France. I want to share with you some updates about what is coming, what is happening right now, and how it can impact and influence our strategic choices for OFN.
We all share that vision in this ecosystem that the web is evolving toward small “modules” that do little but very well. Instead of big apps trying to do everything but not managing to do anything well. APIsation of the world mean that it becomes much and much easier to build a nice front end, taking various blocks and combining them through APIs into a nice offer. Of course for that you need open and democratically run standards so that all those blocks can easily be plugged with one another, like Lego blocks.

A- An analogy

I wanted to share with you a quick analogy that I use all the time now to describe this “world”:

  • Software are roads. You have many roads to drive you to many places. Not one road that try to go everywhere.
  • Roads are connected to one another through “road connectors”, “highway interchange”, that enable easily to move from one road to the other, switch from one direction to another and make my own custom path to go where I want. These are the APIs.
  • Standards are the “highway code”, the “rules”, a form of “law”, that we decide collectively to use peacfuly our shared infrastructures.
  • Data are cars and trucks on the road.

I also sometimes say that French is my standard. If I speak French and you speak English, we need either a “pivot language” to understand each other, either a translator. A pivot language is a common, shared standard (here a “de facto standard”, not open and democratic. Esperanto would have been). If we hire a translator each time we talk to another individual, it becomes very costly. APIs are my voice and your ear, the “support” for the communication to technically happen. Data are the content of my speech.

B- From open source software to open source interoperable modules

Let’s take the example of the product catalog. An agent (producer or distributor) needs a nice tool to manage its catalog, stock, and pricing table for their different customer categories. The agent wants to be able to share this catalog potentially through many distribution platforms, just like an agent offering a carsharing road wants to push it through many carsharing apps to maximize the chance to find passengers.
Concretely ALL producers today work with more than one distribution channel, and the different distributors they sell through are unlikely to use the same management tool.

There are activists that I know in France who want to build a dedicated tool for catalog management, based on the semantic web standard, interoperable with any distribution platform (using the Data Food Consortium standard hopefully). This module should of course handle connections between catalogs through the “product chain model” we have worked on, and “recipe” that makes the connection. So we can, whatever the application, display the product chain of any product in the world (utopia is what drives evolution :-)) This catalog module should be agnostic of the nature of the product sold, it can be food but it can be other things, the logic remains the same on what is and should do a catalog.

C- What is OFN? Where is the line between what we do and where we integrate?

I am deeply asking myself that question: should we persist in “internalizing the product catalogs into OFN”? Or "shall we join forces with those type of actors to build together a very well thought open source module that we will “consume” through a nice semantic web API in our marketplace? I think we should stop talking about open source software. And start thinking open source interoperable modules.

Maybe you’ve seen the open source software for CSA OpenOlitor? It is a platform presented as many modules. One for instance is “membership management”. There are hundreds of apps proposing membership management. Let’s stop building again other membership management modules. Let’s take it out of all our OS software and build together a nice module as a base block we can all use to offer membership management to our users. Same for volunteer planning for packing. Same for accounting. Invoicing. Maybe Shops ?

We are now doing that in France with the network of commoners and coops (like energy coop, telecom coop, etc.), we are building on Odoo a shared module to handle the life cycle of “shares”.

I like the question we asked ourselves in our last gatherings: what is OFN ? And where do we draw the line of what we do and where we integrate with others? I think the thing we should really build ourselves might be thinner than what we do today. Product catalog for me should be out of OFN, but we could be cobuilder of that module. Invoicing, even the basic module, should be out of OFN.

We can do customer integrations for some users who have specific needs. BUT we need in our mainstream offer, in our public platform to offer a basic aggregation of blocks that work well together in a single user experience. Like the user will not see that we use some external invoicing block. Or some external product catalog. So she can create and manage her catalog within OFN, event if it uses that external block. She can invoice directly within OFN, even if it uses that block, etc. It will not really be “within OFN” but it will look like when she is in OFN. But if she wants to access her catalog in OpenOlitor platform she will be able to, without any friction. She will have a single source for her product catalog and can handle her stock, etc.

I know this is pretty aligned with the discussions we had about “breaking OFN into domains” and building our API and displaying our front end based on that API. But I think my vision goes further than that. I think we should get rid of some of those domains purely and start building some of those blocks with others.

Happy to discuss on some concrete cases and opportunities soon…

Ping @luisramos0 @Kirsten @Rachel @sauloperez @maikel @Matt-Yorkley and others

2 Likes
2

I have the same vision Myriam. When thinking about OFN working with other tools, the product catalogue always comes to my mind. But I never spent more than 20 minutes thinking about it and always ended up with: It’s not easy. It needs a proper inception. And it would be great if we had a good product catalogue tool to integrate with as an example.

Managing products can be as simple as a spread sheet for a producer. Product Import is the first interface to communicate with other tools and bring these products into OFN. There are more things the OFN does though:

  • Products are displayed in a shop. The OFN can’t ask another API every time it wants to display a shop front. In fact, we don’t even ask the OFN’s database, because it’s too slow. Whenever products change, the OFN needs to be notified and we can compile the shopfront data again. Stock levels always have to be synchronised. Is stock management included in the catalogue or should it be another small tool?
  • Products are organised in order cycles. As long as order cycle management is in OFN, we need to save some data about products that are included in order cycles. We can’t just have them stored somewhere else.
  • The OFN is a directory. Enterprises should be discoverable by product names. This could trigger a lot of API calls to the catalogues.

There is probably more. These things can all be designed in a way to work with a product catalogue API. But the result won’t be a clear separation where all the products are outside the OFN. The OFN needs to know about products. And stock management is actually more difficult as it has to be synchronised during checkouts. Travel agents can reserve a seat while you decide if you want to book the flight. The stock API needs something similar.

3

Very interesting post Myriam. You got to some core software engineering concepts. To be more precise I recommend you use more specific words like libraries, frameworks or applications/saas. For example, discussing frameworks (it’s a well studied field in academic software engineering) is different from discussing the re-use/integration of applications or even re-use of saas.

The most important thing I have to say is: “there is a reason why things are the way they are”. and I believe it’s just because software is a very complex.
Yesterday I activated database logs as I run a simple OFN test, one simple test and my screen looked like the Matrix with all the db queries going in different colours for about 10 seconds. Software is very complex. And that’s the reason why people choose to rebuild.

In the long term your vision is good, on the decade timescale, after a few more revolutions like Docker (I recommend you read about Docker, it plays really well on the roads/bridges analogy).

On the short term, for OFN, this means we need to think API, we need to break it into smaller parts, we need to re-use libraries and frameworks for sure, but we will always have custom. Look at Spree (it’s the common framewwork for ecommerce), and we want to build our own stuff now because Spree doesnt fix all problems we have, all our specific needs. This is a very good example in this context.

All I am saying is that we live in the age of custom software. You are calling for the age of standards that is in its early early stages.

More pragmatically, I think after we break OFN in domains, we can start to think about decoupling some of these domains from OFN (standalone checkout), having multiple installations of specific parts (multiple catalogs), trash a domain and re-use common component/application (maybe for entity management or maybe we plug a CMS on the Web domain), etc. The first step is to break it.

Thanks for sharing the vision and inspiration!

4

For the first time - after dozens of people have tried to explain it to me - I am beginning to understand API. Thanks @MyriamBoure for the roads analogy. I’m sure - like any analogy - its too simple - but that is what is needed in the OFN community so we can get tech and non-tech people on the page. For sure - the developers who will build this will have more precise and complex language @luisramos0 - but you will use that when you ‘talk among yourselves’. (In the same way I use precise food sovereignty and governance language when I am with my co-researchers, and more general terms when I post on the OFN community.) SO - finally I feel included in this important visioning discussion!!

Maybe - based on what @maikel describes (and that post is also VERY helpful to me - I"m starting to get it) products are not the first interoperable module to think about given the complexity. Maybe we think of some function/need that OFN does not currently do at all — like membership management & communications?

Finally - what I’m sure is a stupid question: How do we identify these open source apps that are ‘outside’ of OFN, that we might integrate with? How do you find them? It seems that this is an important early step - just to even check out what apps are possible. Is there a database of these somewhere (I’m afraid you are going to tell me we find them through google searches. Yuk. I wish I could design my own search engine!)

1 Like
5

Hi Theresa, good to have your feedback. I am sorry for saying things you don’t understand! I’ll try to keep it easy next time. Also, please let me know if I can clarify any concept or idea :heart:

6

Everything is in everything in the universe @tschumilas, there are common patterns, or “meta models to describe the world”, and that’s why I find analogies are so powerful :slight_smile: Thank you for your feedback that makes me really happy :rofl:
So yes, what we started to talk about @tschumilas was more about integrating curent OFN with external systems like POS, but can also be membership management and emailing campaigns tools. As you say there is a work to investigate the external tools we would like to integrate with. Like for POS we identified Odoo POS and Unicenta for instance. So in the “brainstorming solutions” stage, when you have a need, looking for potential external solutions we could integrate with is something important to do.

@maikel @luisramos0 I understand that it wouldn’t be easy and agree that it might not happen in one year, but vision starts now. I have a meeting on Friday with those activists working on a super semantic web friendly interoperable products catalog, so that’s why I opened that discussion on vision. @luisramos0 I would be happy to use technical terms but I don’t know them either, I’m not sure I understand the difference between libraries and frameworks. I think I was more talking about applications, like small applications like “modules” that you can plug and play. Each “module” can be built on “librairies”, likea road is made of technical construction blocks, isn’t it? Frameworks are more like standards no? But no all standards are frameworks? They are more like conventions on how to build a construction block for instance, or how to build a road, no? They are not the same type of standards than the highway code…? Frameworks are more technical specifications, while interoperability standards are more “functional specifications”, they are linked to the job done by the individuals on the road. Tell me if I get it :slight_smile:

7

Hey, I dont think my point about libraries/frameworks/applications is so important to this thread. Also because I realize you are talking about integration with other applications.

Open Source has grown a lot due to open source libraries and frameworks. Libraries like openssl (open source library for criptography) and frameworks like Rails (the web application framework we use to build OFN faster than just using Ruby).
Open source applications is a different game with Firefox or OpenOffice where full functional applications are offered. On top of this, some of these applications can be offered as a service, Matomo is the closest app we are using as a saas (which means that we use someone elses installation of it). WordPress is another good example where you can use the app (install it and run it as we do in OFN) or use wordpress.org as a sass (same software but someone else’s installation).

I think I mentioned this because you mentioned modules, a word used more to refer to libraries/frameworks… but actually, when you use a saas, for example matomo for analytics or lets say wordpress for CMS, you can think of these applications as modules of OFN. But they are separate applications really.

Anyway, I think there are two dimensions to this game of integrations, one is the standard stuff:

  • CMS, if you want to integrate with CMS choose one, wordpress, october, etc
  • CRM - same same, choose one, oddo?
  • POS - …
  • Accounting/Finance -
  • Analytics - Matomo
  • Mapping - OSM

But the adventure begins when we start talking about what you mentioned like Product Catalogs, or Order Management Systems. This is the traditional world of the ERPs like SAP.
afaik there arent many standards neither standard solutions for these problems.
It’s quite an interesting challenge to build a standardized open source product catalog/inventory solution…

8

Interestingly, in France some activists organization have taken the problem the other way around, they started from Odoo applications (modules) that they plugged with one another, so they have a product catalog, stock inventory, connect that with an online shop, accounting, invoicing, and CRM and mailing. GRAP is a cooperative in France who has customized Odoo to feed food enterprises needs and offer those modules as Saas to their members. They are missing a marketplace, a powerful online shopping tool, but they start from ERP. We have done the contrary. When we discussed we had a real challenge understanding who should be the master / slave in the integration process. I tend to think that an independent product and stock management module should be the master, and marketplace should “consume” that catalog. For orders, it would be the contrary. The fact that we have a product catalog within OFN make it complex for integrating as people can update their product catalog both in OFN and in a product catalog module for example in the case of a POS module. But anyway, whatever, modularizing and building API will be a first step before we can integrate anything and replace any module from inside…

9

nice, that is the SAP way of life :slight_smile:
I will have to check Odoo now :slight_smile:
I am pretty sure it would be a difficulty for any off-the-shelf solution to handle OFNs marketplace type of logic… Odoo Multi Vendor Marketplace | Odoo Apps Store

10

Hello, nice to meet you all here!

I try to get into this discussion that I think to be very important and very helpful. Thank you for allegories and all this!

Is there anybody drawing pictures of what you are talking about? It seems to me that we need grafical output to come to common understanding and to be able to make decisions. Am I right? So if anybody can make a scetch of theses different languages to link things together, please do it!

To explain my point of view. I am one of those that try to connect the different stakeholders on the german speaking countries and I want to try to organize people and support the work if it makes sense for OFN. Reading the different threads I feel a bit overwhelmed by the amount of possibilities. Is it in the end a lot of different tools that only need a server somewhere? Where lies the compulsory common in this?

Educated as an architect I am used to have space allocation plans and catalogues of requirements, these are light or heavy books - depend on the building :wink: - that people look in every now and then to remember the orginal ideas and have to’s. It seems to me that the way you are talking about the different languages have not found this common ground yet where we can look in… Am I right? Please be patient with me if I am not on the highway yet.

@MyriamBoure @luisramos0

11

Thanks for your feedback @Karin
I will make some drawings :slight_smile:

12

Ah ah ! My personal answer to that question would be that the only compulsory common in this is a shared standard, a common language, that all system can connect with through their own API. Databases can be hosted in many places, actually a producer could have their own server hosting their own data and give access to them to multiple applications like the OFN.

But you are right in a sense, that OFN wants to take some of those tools and deploy them all in a server as you say (somehow) and connect them through their respective APIs (so data can be shared by those tools) so that we can offer a comprehensive service to our users. So we need a standard to interoperate those tools (or if no standard, we need to build connectors), and we need APIs using that standard so that the whole mix of tools work well together.

As we said here we are not yet there, there are steps before we can reach those kind of architectures. 1- Modularize OFN and build our API and use it, 2- Connect missing bits that our users want to propose comperhensive offeres, 3- See if some internal modules with OFN are better done outside and it would make sense to replace by another module (like catalog management), etc.

Great if you have some drawing @luisramos0 :slight_smile:

13

Some very draft drawing about integrations here:

2 Likes