Author Topic: Translation tools  (Read 16301 times)

Offline SupSuper

  • Lazy Developer
  • Administrator
  • Commander
  • *****
  • Posts: 2159
    • View Profile
Translation tools
« on: September 20, 2013, 04:45:02 am »
So I'm planning on doing some big changes to translations. And by big I mean "oh gawd I'm gonna have to update all these files and strings by hand", so first I thought about making a program to make this whole process easier, and while I'm at it it could also check and report outdated translations, and then I figured maybe there's things you guys find boring and repetitive that you could use a program to make translating easier too, since our current process is... uh... "help yourselves". :P

But I'm not gonna reinvent the wheel. I know there's already tons of translator tools out there that make stuff a lot easier like managing multiple translators and languages, checking untranslated strings, adding new strings, etc. They're also more attractive to new translators.
Currently GetLocalization is my top pick since it's free and integrates with GitHub and it's free and did I mention free?

However, there are some advantages to the current direct-text translations, such as being able to put them in your favorite text editor, doing mass find/replace, directly testing it ingame, etc.

So, what do you translators think? Are you happy with the current way of translating? Would you like switching to a big fancy webby thing? Are you gonna give me some more complicated options? :P

Offline winterheart

  • Colonel
  • ****
  • Posts: 180
  • Fellow Squaddie
    • View Profile
Re: Translation tools
« Reply #1 on: September 20, 2013, 08:58:44 am »
If new strings will appears automagically after adding in code and they will have original English context (like in Gettext or XLIFF) it would be OK for me.

Offline kkmic

  • Commander
  • *****
  • Posts: 582
  • Undefined
    • View Profile
Re: Translation tools
« Reply #2 on: September 20, 2013, 11:51:39 am »
How any strings are there in total? (I'm too lazy to check)

Offline SupSuper

  • Lazy Developer
  • Administrator
  • Commander
  • *****
  • Posts: 2159
    • View Profile
Re: Translation tools
« Reply #3 on: September 20, 2013, 04:14:15 pm »
1216 strings (8245 words) ATM.

Offline sjamaan

  • Sergeant
  • **
  • Posts: 10
    • View Profile
Re: Translation tools
« Reply #4 on: September 21, 2013, 01:57:31 pm »
If new strings will appears automagically after adding in code and they will have original English context (like in Gettext or XLIFF) it would be OK for me.

The main problem I've found in practice with (traditional) gettext is that it won't allow context-dependent translations.  So if the same English word or short string is used in an entirely different context, you cannot have two distinct translations. It appears that GNU's version of gettext has a pgettext() macro which allows the user to distinguish between contexts so maybe that's acceptable. It might cause portability problems, though, since it's not "standard" (for example, my NetBSD system does not have it (all BSD systems use Citrus gettext), nor would a Solaris system (it uses a proprietary Sun implementation), but it's probably possible to link OpenXCOM against GNU gettext on all these systems). I also don't know how many of the GUI frontends for gettext support contexts. Another problem with gettext is the way translations are done using C printf-format strings; it's sometimes difficult to see exactly which placeholder corresponds to what variable. This can be solved using position-swapping and xgettext comments, but it feels a bit kludgy and requires great care on the part of the programmer, to assist the translators as much as he can.

I'm not familiar with XLIFF at all, but from afar it sounds like a slightly better format (with support for translation status; translated, not translated, needs review), at the disadvantage of being a bit newer, so  the tool support may be somewhat less mature. OTOH, there's Pootle and Virtaal, both of which support multiple formats. And the GetLocalisation tool you mentioned supports both formats as well, and it sounds very good; integration with Github is very nice. If you use XLIFF, I guess you'll need to start looking for a library that can use the format, whereas gettext is a format and library rolled into one.

If you're going to overhaul the translation strings, it would be great if you could start using more placeholders/templates. For example, STR_WE_CAN_NOW_RESEARCH should probably use a placeholder for the thing to research, so it can be translated to Dutch as "Onderzoek naar %s is nu mogelijk". Note that this is not inherent to the technical translation system in use, but to the way the code constructs "sentences" out of translatable strings; it's always better to work with larger preconstructed "template" strings which can be translated as a whole rather than assembling them together in code. I think this has already improved, since I saw a few old assembled strings while I was translating to Dutch, which had been since replaced by better preconstructed template strings.

Finally, there are some problems with grammatical gender of words, which may cause trouble in other languages than Dutch and English, and of course, pluralisation can cause issues even in English if not handled carefully (gettext has some pluralisation facilities which may be of help here). These are fundamental problems with the current translation system, which any new system should address if it's worth overhauling the code for.

Offline SupSuper

  • Lazy Developer
  • Administrator
  • Commander
  • *****
  • Posts: 2159
    • View Profile
Re: Translation tools
« Reply #5 on: September 21, 2013, 06:01:23 pm »
The main problem I've found in practice with (traditional) gettext is that it won't allow context-dependent translations.  So if the same English word or short string is used in an entirely different context, you cannot have two distinct translations. It appears that GNU's version of gettext has a pgettext() macro which allows the user to distinguish between contexts so maybe that's acceptable. It might cause portability problems, though, since it's not "standard" (for example, my NetBSD system does not have it (all BSD systems use Citrus gettext), nor would a Solaris system (it uses a proprietary Sun implementation), but it's probably possible to link OpenXCOM against GNU gettext on all these systems). I also don't know how many of the GUI frontends for gettext support contexts. Another problem with gettext is the way translations are done using C printf-format strings; it's sometimes difficult to see exactly which placeholder corresponds to what variable. This can be solved using position-swapping and xgettext comments, but it feels a bit kludgy and requires great care on the part of the programmer, to assist the translators as much as he can.
Most translation formats are pretty similar, they all come down to matching IDs to strings (or stuff like gettext which matches strings to strings :P) sometimes with context comments.
The advantage of using software like this is the format doesn't matter. The software takes care of all the extra stuff like keeping track of "untranslated"/"needs updating", matching the master file with the translated files, keeping everything in sync and notified, etc.

If you're going to overhaul the translation strings, it would be great if you could start using more placeholders/templates. For example, STR_WE_CAN_NOW_RESEARCH should probably use a placeholder for the thing to research, so it can be translated to Dutch as "Onderzoek naar %s is nu mogelijk". Note that this is not inherent to the technical translation system in use, but to the way the code constructs "sentences" out of translatable strings; it's always better to work with larger preconstructed "template" strings which can be translated as a whole rather than assembling them together in code. I think this has already improved, since I saw a few old assembled strings while I was translating to Dutch, which had been since replaced by better preconstructed template strings.

Finally, there are some problems with grammatical gender of words, which may cause trouble in other languages than Dutch and English, and of course, pluralisation can cause issues even in English if not handled carefully (gettext has some pluralisation facilities which may be of help here). These are fundamental problems with the current translation system, which any new system should address if it's worth overhauling the code for.
That was the end goal of all this. ;) We've had support for templating and pluralization for a while, but going through the whole master file and correcting all sentence strings is quite the effort, and would pretty much break all current translations and make it a nightmare to fix them up again. Using software like this will at least get rid of the latter problem. I'm also working on getting rid of hardcoded language graphics and the like to make translation much simpler.

Anyways I'll try to get it set up this weekend so you can test it around and if it proves useful we'll switch to it.

Offline sjamaan

  • Sergeant
  • **
  • Posts: 10
    • View Profile
Re: Translation tools
« Reply #6 on: September 22, 2013, 12:11:54 am »
Most translation formats are pretty similar, they all come down to matching IDs to strings (or stuff like gettext which matches strings to strings :P) sometimes with context comments.

True that! :)

That was the end goal of all this. ;) We've had support for templating and pluralization for a while

I wasn't aware of that, thanks for the heads up (somehow I thought the templating was ad-hoc one-off stuff).


I'm also working on getting rid of hardcoded language graphics and the like to make translation much simpler.

Anyways I'll try to get it set up this weekend so you can test it around and if it proves useful we'll switch to it.

Awesome! Thanks a lot! Especially getting rid of the hardcoded graphics is a big improvement.

Offline hsbckb

  • Colonel
  • ****
  • Posts: 275
  • Gill Man
    • View Profile
Re: Translation tools
« Reply #7 on: September 22, 2013, 04:11:08 pm »
I'm also working on getting rid of hardcoded language graphics and the like to make translation much simpler.

Will this make Chinese or Japanese languages support possible?

Offline LCSand

  • Captain
  • ***
  • Posts: 73
    • View Profile
Re: Translation tools
« Reply #8 on: September 22, 2013, 04:57:27 pm »
Will this make Chinese or Japanese languages support possible?
I think the main problems with that are the fonts and lack of pixelspace.
I think unicode is already supported, I do not know about right to left and top to bottom text and such though.

Offline hsbckb

  • Colonel
  • ****
  • Posts: 275
  • Gill Man
    • View Profile
Re: Translation tools
« Reply #9 on: September 23, 2013, 05:40:54 am »
I think the main problems with that are the fonts and lack of pixelspace.
I think unicode is already supported, I do not know about right to left and top to bottom text and such though.
Left to right reading of the text is not uncommon for Chinese and Japanese language. I agree that the main problem is the fonts. So, I suspend the Chinese Translation of the New text strings. :'(

Offline SupSuper

  • Lazy Developer
  • Administrator
  • Commander
  • *****
  • Posts: 2159
    • View Profile
Re: Translation tools
« Reply #10 on: September 23, 2013, 06:20:26 pm »
Adding RTL is no problem, though vertical would require a UI redesign. And yes we have full Unicode support, though I dunno how good SDL's Unicode input is (supposedly it's better in 2.0).
For reference, these are the current fonts:



As long as you can make characters that fit, they can go in. You should be able to generate a CJK bitmap font for those sizes but I dunno about readability for the smallest one (though you'll see it resized x2 in-game). I wouldn't rule it out just yet though, as previously discussed.
As for this related discussion, I couldn't figure out the format of the PSX files to extract their font (they're in RAW/IDF files which are different from the PC formats), and figuring out the text relies on knowing the fonts.

On the bright side, CJK text uses much shorter strings so we wouldn't have any length problems that we often encounter with typical latin languages. :P
« Last Edit: September 23, 2013, 06:26:58 pm by SupSuper »

Offline djemon_lda

  • Captain
  • ***
  • Posts: 52
    • View Profile
Re: Translation tools
« Reply #11 on: September 25, 2013, 10:56:17 pm »
@SupSuper:
does this tool support handling debatable propositions in translations? I've just tried out the polish language version and some things are really off the original meaning by a lot. can I use the tool to propose a change, or the proposition should be done on the forums ?

Offline SupSuper

  • Lazy Developer
  • Administrator
  • Commander
  • *****
  • Posts: 2159
    • View Profile
Re: Translation tools
« Reply #12 on: September 25, 2013, 11:45:16 pm »
Yes it does support discussion and multiple proposed translations and such, so you can add them there if you want, but most translators don't hang around much, so you're better off just posting and being patient.

Offline cort

  • Sergeant
  • **
  • Posts: 43
    • View Profile
Re: Translation tools
« Reply #13 on: October 15, 2013, 07:17:50 pm »
So, here is the list of strings and the screens in which they are used.

Now a question for SupSuper:
What resolution would be good for the screenshots? Are there any restrictions on GetLocalization?
« Last Edit: October 15, 2013, 07:31:36 pm by cort »

Offline tyran_nick

  • Captain
  • ***
  • Posts: 59
    • View Profile
Re: Translation tools
« Reply #14 on: October 15, 2013, 08:09:41 pm »
This list is quite helpful!