Translation sources¶

The translations amaGama imports are listed in a sources file, amagama/updater/sources.toml unless UPDATER_SOURCES names another. Each [[block]] is a set of files fetched and imported together, such as one GNOME release:

[[block]]
name = "gnome-51"
title = "GNOME"
version = "51"
source = "damned-lies"
release = "gnome-51"
licence = "GPL-2.0-or-later"

A sources file can include others, named relative to it, whose blocks follow its own; block names must be unique across them all:

include = ["weblate-fedora.toml"]

The included Weblate files are written by weblate_sources; transifex_sources writes blocks the same way from repositories’ Transifex configurations, and discover_sources from repositories it finds by the localisation configuration they commit.

A [defaults] table gives settings for every block in its file (not in the files it includes); a block’s own settings take precedence. name and versions can’t be defaults:

[defaults]
source = "git"
min_age_days = 60

name

The block’s fixed reference: it names the block’s directory in the cache and its imports, so never rename or reuse one. Blocks are read in the order they appear in the file.

title

The name shown to people, such as GNOME or Legacy OSS. Titles are translated, so keep release numbers in version.

version

Optional release, such as "51"; shown after the title (GNOME 51).

versions

Instead of version, a list of releases: the block stands for one block per release, with {version} replaced in every setting, so adding a release is one more item. The name must contain {version}. An item can be a table giving values that differ by release:

[[block]]
name = "gnome-{version}"
title = "GNOME"
source = "damned-lies"
release = "gnome-{version}"
versions = ["50", "51"]

[[block]]
name = "libreoffice-{version}"
title = "LibreOffice"
source = "git"
url = "https://git.libreoffice.org/translations"
files = ["source/{lang}/**/*.po"]
versions = [
  { version = "26.2", ref = "libreoffice-26-2" },
  { version = "26.8", ref = "libreoffice-26-8" },
]

source

How to fetch the files; the other keys depend on it (below).

licence

Optional SPDX licence expression, checked as for build_tmdb.

files

Optional filemasks choosing which of the block’s files to import, relative to its directory. * matches within a directory, ** any number of directories, and {lang} the target language (a code such as af, pt_BR, sr@latin or b+sr+Latn), which then overrides the language the source gives:

files = ["addons/*/i18n/{lang}.po"]
files = ["translations/*_{lang}.ts"]
files = ["**/locale/{lang}/LC_MESSAGES/django.po"]

Without it every file is imported. A block whose masks match no file fails, rather than loading nothing.

exclude

Optional masks, without {lang}, of files to leave out, such as ["**/qqq.json"].

keep

Optional masks, without {lang}, of files the import needs beside the translations but that aren’t imported themselves, such as the .grdp parts a Chromium .grd includes.

languages

Optional table of a source’s language codes to amaGama’s, applied before the normalising below; map a code to "" to leave it out:

languages = { no = "nb", xxx = "" }

format

Optional reader for the block’s files, when their extension doesn’t say: po, mo, ts, tmx, tbx, xliff, csv or nextcloud-json (Nextcloud’s l10n/<lang>.json, keyed by the English text). Without it the extension picks the reader.

Monolingual formats hold one language per file, keyed by an id, and need a template: android (strings.xml), php, properties (Java’s ISO-8859-1; properties-utf8 and properties-utf16 for files in those encodings), joomla-ini, json (key to message), fluent and dtd; and xliff-monolingual, XLIFF whose <source> is the message’s id and whose template’s <target> is the English (Symfony’s).

For android, {lang} must be a locale qualifier alone (af, pt-rBR, b+sr+Latn): directories that add another qualifier, such as values-af-watch or values-fr-feminine, and the pseudo-locales en-rXA, ar-rXB and en-rXC are left out.

xtb is Chromium’s: each .xtb translation is keyed by a fingerprint of the English message, which is in a .grd (and the .grdp parts it includes, which keep must keep). Each .grd lists its .xtb files and their languages, so without a template that list pairs them:

format = "xtb"
files = ["**/*.xtb"]
keep = ["**/*.grdp"]

A template pairs by path instead, with files giving the language:

format = "xtb"
files = ["*/strings/translations/*_{lang}.xtb"]
template = "*/strings/*.grd"
keep = ["**/*.grdp"]

template

For a monolingual format, the English file each translation pairs with by id: a path with the same wildcards as files, each taking what it matched in the translation’s path:

files = ["res/values-{lang}/strings.xml"]
template = "res/values/strings.xml"

files = ["{lang}/*.php"]
template = "en/*.php"

The template is kept but not imported itself. Messages with no English, and plurals, are left out; a translation whose template is missing is listed as unreadable.

columns and header_row

For format = "csv": what each column holds, from "source", "target" and "location", with "" for a column to ignore; and whether the first row is a header to skip, or the header row’s first cell, so that the rows before it (a notice, say) are skipped too. Files are then read as plain comma-separated values:

format = "csv"
columns = ["", "source", "target"]
header_row = true

xliff_states

For XLIFF files: the target states that count as translated, from XLIFF 1.2’s new, needs-translation, needs-review-translation, translated, final, signed-off and the like, x- custom states, and "none" for a target without a state. A unit with approved="yes" always counts. Without it, translated, final and signed-off count:

xliff_states = ["none", "translated", "final", "signed-off"]

segment

Optional true to also import each multi-sentence message as sentence pairs, so a single sentence of a long message can match: worthwhile for documentation. A message is split only when its English and translation have the same number of sentences and no inline tag crosses a sentence (an element around the whole message, such as LibreOffice’s <ahelp>, is set aside first). The whole message is imported too.

weight

Optional number from 0 to 10, default 1: how suggestions from the block rank in lookups, by quality times weight, unless a client gives its own weight. A translation also found in a block of higher weight takes that weight. Set it in a file’s [defaults] to weight all of an included file’s blocks.

tags

Optional list grouping blocks apart from the file they’re in, so a server can build a set of them. A file’s [defaults] tags add to each block’s own. A block has at most one tier, how it was chosen:

• core: by hand (sources.toml)

• platform: from a translation platform’s listing (weblate_sources)

• discovered: by a search of repositories (discover_sources, transifex_sources)

and a content tag, ui, docs or terminology; without one it is docs if segment is set, else ui. trial marks blocks being tried out:

tags = ["terminology", "trial"]

disabled

Optional reason to leave the block out: it isn’t fetched, imported or listed, but keeps its name. The commands that write blocks keep it when they regenerate the file:

disabled = "duplicates Damned Lies module meld"

min_age_days and max_age_days

Both count days since the block was last fetched; min_age_days is a floor and max_age_days a ceiling.

min_age_days (default 0): until the block’s copy is this old, it isn’t checked at all, so its server gets no request; this keeps many small sources from being fetched more often than they’re worth. Blocks first fetched together would all come due on one day, so a block’s first refetch comes after a fraction of min_age_days taken from a hash of its name. 0 checks the block on every run.

max_age_days (default 30): a copy this old is fetched again, though nothing says it changed. For a source with one download per language (damned-lies, translationproject, ubuntu-langpacks, http, pontoon, wordpress, drupal, moodle, joomla, svn) it applies to each language, and the downloads are spread out: a run takes its share of the languages for the days since the last fetch (rounded up; a day’s share the first time), languages never fetched first, then those fetched longest ago. With 57 languages a daily run fetches 2, so the first fetch takes max_age_days too, and the block isn’t due again until max_age_days / 57 days later. Such a source is refreshed by age, not by its fingerprint, so its server never gets every language at once. 0 turns that off: the whole block is fetched whenever its fingerprint changes, or on every run without one.

An old release that hardly changes can have a longer max_age_days than the current ones, as an item of versions.

Within min_age_days a block isn’t touched, even past max_age_days.

exclude_ids

Optional patterns of message ids to leave out, such as ["@metadata*"] (MediaWiki), ["*.description"] (Signal) or a TMX tuid prefix (["foundation-website-content:*"]). JSON ids are matched without their leading ..

language_in_ids

Optional true for a block with a template whose message ids name their file’s language, such as CLDR’s main.de.localeDisplayNames.... or Rails’ de.errors....: the id segment where every one of the template’s ids has the same English code is left out of each id before a translation is matched with its English.

Templates (.pot) are never imported.

Every language code is turned into one form, language with optional _REGION and @variant, whichever way the source writes it: pt-BR, Android’s pt-rBR and pt_br become pt_BR; sr-Latn, b+sr+Latn and sr@Latn become sr@latin; zh-Hans and zh-Hant become zh_CN and zh_TW; ca-valencia becomes ca@valencia; replaced codes such as iw become he; a country’s code used for its language (cn, ua) becomes the language’s (zh_CN, uk); and a three-letter ISO 639 code with a two-letter one becomes it (arg and deu become an and de). Codes that aren’t languages are left out and listed by fetch_sources and update.

fetch_sources downloads the blocks into UPDATER_CACHE_DIR, and update rebuilds the database from them.

$ amagama-manage sources_pot