Internationalisation project notes

International
1

Introduction

Here are some preparatory notes on the project to internationalise and localise the Open Food Network. For this project, I’m only considering the scope of the customer-facing part of the site. The admin interface can be tackled in a subsequent project.

There is a spree_multi_lingual project. From a quick glance it doesn’t appear production-ready (1.5 years since last commit, builds failing). However, its code could be a great reference for achieving some of the things below.

In this doc, I’ll split up some concerns into i18n (internationalisation) and l10n (localisation). i18n is the process of making the app flexible for translation, and l10n is the process of using that flexibility to localise it to a particular region/language.

Internationalisation

With i18n, we have a few different aspects to address. The first is to replace strings used in the app with calls to t(), and extract the strings into config/locales/en.yml. I haven’t been able to find a tool to automate this process, but although laborious, I don’t think the volume of work is too large; an estimated 162 strings at 1 string per minute comes in under three hours.

Secondly, there is data that users enter for which we might want to collect and provide versions in multiple languages. An example of this is enterprise descriptions. I think it’d be nice to provide an interface for enterprise users to provide this content in multiple languages. Luckily, the globalize gem addresses this. The SitePoint article provides a good run-down on how it works. The simplest UX is a bit clunky - the admin would need to enter the data, save, switch locales and then re-enter the data in that locale. I’m sure we could do better, but this is probably enough as a starting point. I also have some uncertainty about whether this will work out of the box for derived fields - ie. entering variant unit_value and unit_description fields. Maybe it’ll require no extra effort, maybe it’ll be quite difficult. I’ve budgeted some time in the estimate for this uncertainty.

Thirdly, there are a number of small details that might need addressing. The icon font contains some icons with English text (most notably the “Open” and “Closed” signs for shops), so we either need to find a way to convey the same graphical meaning without using text (much preferred), or make a number of translations of the font. The currency symbol may need customising in a few places, although I think this has already been accomplished for the UK. We probably won’t need to touch date formats in this round, but it’s worth keeping that concern in mind. And pluralisation may need work (the SitePoint article has details).

One thing that I’m uncertain about is internationalising strings that are within javascript/angular code. The strategy that first comes to mind is to have Rails inject localisations with erb. See app/assets/javascripts/darkswarm/services/geo.js.erb.coffee for an example of this technique.

Localisation

Once the site is internationalised, we can move on to localisation. Rails allows translations to be provided via yml files (ie. config/locales/fr.yml). However, there are a number of online tools that make it easy for translators to make their translations (in some cases inline within the website), and easily sync them with the project (in some cases in real time). Many of them are free to use for non-profit organisations, although it may be worth paying money for the best option if required.

I wasn’t able to find a clear winner between these in the time I had available. It might be worth asking some people who will be involved in translation to test out their UIs and solicit feedback about the tradeoffs between the different choices.

As a starting point for exploring the options, Discourse tackled this decision a few years ago. They settled on Transifex, which is an excellent contender. It offers a sophisticated UI and GitHub integration. LocaleApp offers tighter Rails integration and real-time display of translations for development and staging environments, although its UI appears less sophisticated. PhraseApp offers a in-page translation, and looks very schmick.

As part of this process, we’ll need to determine the workflow we use for accepting translations into the codebase. It seems that Transifex’s GitHub integration would cause a commit to be made to master every time a translation was changed. This could automate things significantly, but maybe it’d be really messy. Worth investigating. I believe the other options offer some similar kind of workflow. We’d need to consider how long it would take from a region completing some translations to them being able to see the results. Shortening this feedback loop would be a big plus, which might be an advocacy for Localeapp’s real-time synchronisation for dev and staging environments.

WBS

Here’s a breakdown and estimate of the work. All estimates are based on 6-hour days. I’ve included some space in the upper bound of the estimate for various things that I anticipate could be difficult. Often in my experience these things end up being easier than expected, but then an entirely unanticipated problem will take up a lot of time and even things out. Therefore, my best guess would be towards the upper range of this estimate.

Infrastructure - 2 - 3.5 days

  • Locale choice affordance - by IP and/or by dropdown
  • Admins can choose their locale so that they can enter details in multiple languages
  • Set up l10n library / app
  • Set up translation workflow

Concerns - 1 day

  • Content within angular - do we have erb?
  • Allow a day for working with this

i18n strings - 0.5 day

  • 162 total strings @ 1s/m = 2:42 h

i18n model fields - 1.5 - 4 days

  • ~6 models, ~11 fields
  • Some fiddliness - derived fields, ie. with variant names
  • Install globalize
  • Set up for each model
  • Get it to work with derived fields (ie. variant unit_description)

Icon font - designer + dev. half day each -> 1.5 - 3 days

  • Open/closed sign

Misc others - 0.5 day

  • Currency symbol (already done?)
  • Date format
  • Pluralisation

Total: 7 to 12.5 days

Scoping

As part of preparing the estimate above, I did a fairly quick review of the website for elements needing i18n. Here’s what I found. All string counts are rough estimates.

Header/footer - 45 strings

Home - 15 strings

Shops

  • 15 strings
  • Hub names
  • Taxons
  • Producer names and descriptions
  • Icon font - open/closed sign

Shopfront

  • 6 strings
  • Hub name, description
  • Producer names
  • Top info message
  • Product and variant names

Cart - 11 strings, currency display

Checkout - 29 strings, shipping and payment method names and details

Order confirmation - 10 strings

Map - 1 string

Producers - Producer details, 1 string

Groups - 1 string

Group - Group details

Producers signup - 9 strings

Hubs signup - 9 strings

Groups signup - 10 strings

Resources

  1. OFN in foreign languages
  2. https://github.com/jipiboily/spree_multi_lingual
  3. http://www.sitepoint.com/go-global-rails-i18n/
  4. http://guides.rubyonrails.org/i18n.html
  5. https://github.com/globalize/globalize
  6. https://meta.discourse.org/t/translation-tools-transifex-localeapp/7763/21
  7. https://www.transifex.com/
  8. https://www.localeapp.com/
  9. https://phraseapp.com/
2 Likes
2

@RohanM that looks awesome!!! Even if I don’t understand everything :wink: But thank you so much for dedicating time & energy to that reflexion, which is a priority for Norway and France! @sigmundpetersen @olemd @sylvain @marito59 @gnollet, any comment on that?

3

I have used transifex.com within mailchimp and other apps, easy and user friendly :smile:

5

Hello everyone,

I had a go on the internationalisation stuff. I started with the first
step: replacing all language strings used in the ruby code and move them
to the English language file. Unfortunately, there were a lot more
strings than expected. I found around 400. That took me 16 hours, so we
thought we’d check in and discuss a plan for getting to MVP within the
budgeted resources - which might mean a lighter touch on some other
areas.

So where are we now? Rohan listed the following steps of
internationalisation:

1. replace strings in ruby code

This is done. So the localisation can start. That means that someone can
take the new English language file with all the strings and translates
them into another language. I’m sure there will be difficulties. In some
cases the context of a string might not be clear which makes it hard to
find the right translation. In other cases it might be grammatically
difficult, because there are strings that represent only a part of a
sentence. For example “Filter by” or “products at” assume some data
afterwards. That might not work in all languages. We’ll find out. So
give plenty of feedback.

@RohanM mentioned tools like Transifex and LocaleApp that can help
translating. I think that we should get some experience in translating
first, see were the problems are and then try some of the tools to see
if they solve these problems. Just taking the language file and
translating the strings is the simplest way, so let’s get started with
that.

Currently, the English language file is here:
https://github.com/openfoodfoundation/openfoodnetwork/blob/i18n-replace-strings-in-views/config/locales/en.yml

It contains roughly 450 lines looking like this:
email_signup_greeting: Hello!

“Hello!” is the part to be translated. The labels on the left side
should not be touched. Some translations contain variables like:
email_signup_shop_html: “You can start shopping online now at %{link}.”

It’s important to keep the symbols as they are, but the text around them
can be changed. If the label on the left side ends with “_html” (like
the one above), then you can use HTML tags in your text, but then you
also have to encode HTML entities like & as &. That’s as complicated
as it gets.

So hop to it @lawrence @sylvain @MyriamBoure :smile:

2. use “globalize” to provide multi language versions of user data like enterprise descriptions

The second one is only needed if you would like to have one website
serving several languages. That would make sense in Switzerland for
example. But I think that we should plan that as a separate project,
because there has already been discussion that this is not required in
the first round.

3. change the icon font and replace the open/closed signs

The third one involves some graphics design. As soon as someone
contributes some new, cool and non-language symbols, we can change the
symbol font. We don’t actually have a designer on the team here at the
moment - does someone want to have a go at them? @CynthiaReynolds

###4. replace strings in Javascript code
The fourth one is a bit more tricky. Rohan mentioned already that he’s
not certain how that should be solved. I need some more time on
investigation here, and may post some questions if we need more input
before making a decision how to do it.

Maikel & Kirsten

6

A little update on the Javascript files. There are roughly 123 Javascript files in the front end. I went through all of them to identify the strings that have to be replaced. There are 17 files containing 37 strings of which five contain variables.

I guess that this is small enough to go ahead with the simplest idea. Send the Javascript code through ruby and let it replace strings as we do in ruby code. That would happen when compiling Javascript, at most once per deploy. The running system would not be affected and the language can’t be changed without a new deployment.
Estimated development time: 1d

For the long term, I would like to discuss with Rohan and Rob if we should use a more complex technique to deal with Javascript (link) or we move the language strings from Javascript to application view files to be consistent.

7

Another update with a question how to proceed. It’s about the internationalisation of the Javascript files. There are these 17 Javascript files with a few strings. But there are also another 42 HTML files used by Javascript that are treated the same way. To replace all strings in there is a bigger bunch of work, maybe 2-3 days. But there are two different ways doing that.

Option 1 - language set at compile time

We can replace the strings the same way as we did in the ruby code. But since Javascript is pre-compiled during deployment, we won’t be able to provide switch languages during runtime. That means that a user will never be able to choose a language. The admin chooses the language for the whole website.
Estimate: 2-3 days

If we later want that users can change the language, we have to do all that again.

Option 2 - flexible language file loading

We can replace the strings in a different way so that they are loaded at runtime. If we want that users can change the language, we don’t have to do extra work then - but we would need to do it now. Therefore, this solution will probably take 1-2 days longer. (total 3-5 days)

Comparison

Option 1: 2-3d plus 3-5d later
Option 2: 3-5d total

@MyriamBoure Does that make sense? Can you please let us know asap what you want to do so we can get on with it.

Thanks!

8

Hi Maikel, thanks for the detailed update.
Based on option 1, how far over are we based on our original estimate of total time being 7-12.5 days? adding 2-3 days for option 2.

  • Make built-in text translatable - 3.5 - 5 days
  • Make user-added text translatable - 1.5 - 4 days
  • Other details - 2 -3.5 days
    Total: 7 - 12.5 days

Regarding icons for the open/closed icons, I will give it a go and put something together.

9

@maikel that sounds like a great job!!!

@sigmundpetersen or @CynthiaReynolds could you have a try and set up the translation file for norwegian? So that we can also evaluate the pure translation work now?
@marito59 @gnollet @sylvain can you have a look at it for the French instance?

@maikel if we need to find “in context” what the strings refer to, how can we do? Is there an easy manipulation?

About

Great @CynthiaReynolds that you can work on that :slight_smile:

About the javascript files, I’m not sure I understand the technical part :-o But it seems to me of course much more cost and time efficient to go for option 2. However as @CynthiaReynolds mentionned it, we of course need to have a more precise idea about an updated budget for the whole operation, just to be sure we can finance it. But even for the scandinavian instance, we would like to be able to change language from norwegian to swedish or danish… we are not yet in Sweden and Denmark, but that could happen soon :smile:

@Kirsten I need asap a quotation for that whole translation framework development for a grant application which is to send before Monday (will send you a quick email about that)

Keep up the good job, and thanks a lot for putting your energy in it @maikel and @Kirsten :slight_smile:

10

I’ve spent 3 days in total. Most of them on the built-in text translatables. A little bit on admin and details.

  • Make built-in text translatable: That needs a few more days, option 1 or option 2.
  • Make user-added text translatable: This requires option 2. I would skip this if we are running out of budget.
  • Other details: I spent only a couple of hours on admin, research and so on. There will be a bit more and it’s a buffer for unknown bits.

So, it looks like we are doing well. We are not over yet. Option 2 brings the risk of running over, but we could just skip the user-added text in that case.

11

One way is to search in the project files for the label on the left. But it needs a technical person to understand the context. Or, you give us a list of unclear things and we can give you examples and document them in the language file.

12

Ok @maikel, we’ll see while doing. Maybe @sigmundpetersen will know how to do that :slight_smile: Anyway @sigmundpetersen and @CynthiaReynolds , we need to “tag” the unclear strings in the translation process. About the “risk of running over” with option 2, it’s ok if we go a bit over the 12,5 days, but would be great not to go over 15…

13

@maikel is it OK if we add our no.yml file directly into the branch in github in the locales folder? Or is there a “tidier” way of doing it?

14

@sigmundpetersen which branch are you talking about?
Putting them all in the locales folder is our way to go for now. We’ll see how well that works and how we’ll maintain all the files.

15

Here https://github.com/openfoodfoundation/openfoodnetwork/tree/i18n-replace-strings-in-views/config/locales. Not that experienced working with github so I was just asking so i don’t mess up anything.

16

I think Rohan would prefer if we choose the option “Create a new branch for this commit and start a pull request.” so that he can see what have been added and check if everything is ok. Do you confirm @maikel? I will do the same for the french file then.

17

I agree, opening a pull request is the best way for us.

18

Bonjour,

Comment souhaites tu traiter ce fichier ?
Je peux télécharger le fichier en.yml et le mettre de google drive en le
renommant fr.yml
Ensuite il faudra un peu de temps pour traduire chacune des phrases

Gilles

19

Hi @gnollet, the file is already on the French drive, I saw that, thanks :slight_smile: Useful info maybe, there is a Drive Notepad connected to google Drive, it is proposed when you open an yml file in the drive… can be useful for collab work on yml before uploading in Github and ask for merging :slight_smile: So @gnollet I saw that, it’s in my plan to work on it in the coming days, probably this week-end, but if someone else have time and can work on it please feel free :slight_smile:

20

Ok, for information, it took me 5 hours to do the translations in French, for indication for other languages. I’m not sure of course if all is ok, we have to see “in context” and see what works well and not. Just a practical question @maikel : each time we modify a translation in the local translation file, do we have to create a branch and ask a pull request? (for fr.yml I just created the branch for the moment but didn’t as th pull request, I wait for some feedback for the team before asking to merge)… I just feel it’ a bit “heavy” to ask a new pull request each time we change a word in that file… but anyway if you want us to do it, we can do it :slight_smile:

21

@maikel as I saw you created a branch already with i18n, I put directly the fr.yml file in it, in the “locale” folder. Let us know when all that has been merged and we can upgrade to see the result :slight_smile: Cheers!