Bulk import of products

I have several 100th of product to upload into the Norwegian OFN instance so my question is: how can I bulk import a product list from an Excel or a CSV? Is there such a functionality available? If yes, how do I proceed and what is the accepted format for the file?

This has probably been raised before but could not find anything here on the community site.

Hi Olivier, this feature does not exist yet, but I’m pretty sure you wouldn’t be the only one to be happy if it did! I am going to move this post into the ‘feature - wishlist’ category so it can be easily seen alongside the other things for dev consideration

Thanks for the answer even if it is not the one I was hoping for :wink:

quick notes from some Aus discussions today

This would probably be done in a couple of phases depending on use case. Q. whether something like this could help with the UK data migration issues. Pending more info from @lin_d_hop on data migration

Phase 1: basic upload for new user

  • simple csv format that a new producer can use to import all products / variants to get started e.g. population most of the info they need
  • this would probably also do the job for data migration use case

Phase 2: select update / override

  • ability to update existing p/v info from csv e.g. price changes
  • ability to select from options around update e.g. yes / no

Looks great! I would also add:

Phase 2:

  • ability to select whether products not present in the csv file should be removed (useful if a supplier provides regularly a full list of available product)

I do think this would be a useful feature for a lot of people. In theory the implementation would not be too difficult. The challenge would more be in matching up the appropriate relationships fields to the necessary values. So to create an import spreadsheet in the appropriate format to not result in data issues after the import would take quite a lot of processing.

My thinking is that a requirement that the data comes in a specific format that matches our database would not really be that useful. I did a wee bit of searching and found this gem:

which might just be right up our alley. Takes xml, csv, word and ODF files then specify the structure of the new data using YAML. Effectively this would mean that from a dev perspective each time a new person wanted data imported someone would just have to express their file in YAML to match up to our models. For a dev it would not take long and I imagine it would become a revenue stream to offer this service.

Updates are not currently supported by this gem but it looks like it is a feature in the pipeline. This potentially means it doesn’t really help the current UK requirements. My hunch is to implement updates in a flexible way would be tricky; supporting the range of producer csvs while not dropping data accidentally. I’m sure it would be possible but might take a little thought. Might even be better off spending the time implementing updates in the gem itself.

Anyway, a little food for thought…

Maybe a first step would just be to propose a default structure to be used, csv based for instance. So anyone can do an import, even those who dont want to play around with XML but want to stay with a tool they are confident with like excel.

Yep for sure Oliver. That would be implicit. Just a lot of work in most
cases for the individual involved.

I would like to make sure that we make best use the code that Aidan has written already. He has been busy with other projects but in our Skype this morning he has said that he is happy to pick up where he let off with his code. Please can I check whether anyone has done any work on his original code and whether we are all happy for him to carry on with where he left off?

ping @RohanM @MikeiLL @oeoeaio please see nicks question above

@NickWeir, @lin_d_hop: Noone has done any work on Aidan’s code to my knowledge. I’m happy for him to carry on, but also curious about where the code is up to now (I no longer have access to the github repo) and what direction it would be taken in. I’m happy to help with planning & design for whatever the next steps are.

@RohanM thanks very much for offering to help with planning and design. I will ask @Aidan to give you an update and send you the latest version of the code. Thanks both for your work on this

We also need to plan for importing Tamar Valley data from the old Stroudco software into OFN - I will ask @Sara to give us an idea of how many products and users she has

A rough estimate 60-70 producers, must be 5-600 products, 350 customers,

@nick, @Aidan: Any update on this?

I’m on half term holiday at the moment. Will repost the repo soon as I’m back on the 2nd Nov. The repo was taken down because of customer data - will need to remove any customer details from the XML outputs.

Repo for data migration tool is now at https://github.com/aidandunsdon/stroudco-data-migration

I have just talked this through with Aidan. He is happy to help do a final migrate of data from DFFH and Stroudco into OFN using this tool (big thanks Aidan) but for the time being he does not have any time to do any more development work on this tool to get it ready to migrate data from any other legacy systems into OFN.

What would be the final migration involve? Wasnt the last migration going to cover the bulk of the product list and other stuff be done manually?

there are about 10 new producers who have joined Stroudco since Aidan did the migrate in July. I guess this is about 80-120 new products. It would be great if we didn’t have to enter them all (although we will still need to do the photos and unit/values manually)

We have decided to input all of our producers manually! Big job but it gives us a chance to have a tidy up. We’d love to import our customers though?..