.. _poterminology_stopword_file:

Stopword file format
********************

.. versionadded:: 1.2

The default stopword file for :doc:`poterminology` describes the syntax of
these files and provides a good default for most applications using English
source text.  You can find the location of the default stopword file by looking
at the output of poterminology :opt:`--help`, or using the following command::

  poterminology --manpage | sed -n '/STOPFILE/s/.*(\(.*\)).*/\1/p'

.. _poterminology_stopword_file#overview:

Overview
========

The basic syntax of this file is line-oriented, with the first character of
each line determining its function.  The order of the lines is generally not
significant (with one exception noted below), and the selection of function
characters was made so that an ASCII sort of the file would leave it in a
generally logical order (except for comment lines).

Apart from comment lines (which begin with '#') and empty lines (which are also
ignored), there are three general types of lines, which may appear in any
order:

* case mapping specifiers
* stoplist regular expressions
* stoplist words

.. _poterminology_stopword_file#case_mapping_specifiers:

Case mapping specifiers
-----------------------

A line beginning with a '**!**' specifies upper-/lower-case mapping for words
or phrases before comparison with this stoplist (no mapping is applied to the
words or regular expressions in this file, only to the source messages).  The
second character on this line must be one of the following:

* **C** no uppercase / lowercase mapping is performed
* **F** 'Title Case" words / terms are folded to lower case (default)
* **I** all words are mapped to lowercase

These correspond to the equivalent :opt:`--preserve-case` /
:opt:`--fold-titlecase` / :opt:`--ignore-case` options to poterminology, but
are completely independent and only apply to stoplist matching.  You can run
poterminology with :opt:`-I` to map all terms to lowercase, and if the case
mapping specifier in the stopword file is '**!C**' a stoplist with "pootle" in
it will not prevent a term containing "Pootle" from passing the stoplist (and
then being mapped to "pootle").

There should only be one case mapping specifier in a stoplist file; if more
than one are present, the last one will take precedence over the others, and
its mapping will apply to all entries.  If multiple stoplist files are used,
the last case mapping specifier processed will apply to all entries **in all
files**.

.. _poterminology_stopword_file#stoplist_regular_expressions:

Stoplist regular expressions
----------------------------

Lines beginning with a '**/**' are regular expression patterns -- any word that
matches will be ignored by itself, and any phrase containing it will be
excluded as well.  The regular expression consists of all characters on the
line following the initial '/' -- these are extended regular expressions, so
grouping, alternation, and such are available.

Regular expression patterns are only checked if the word itself does not appear
in the stoplist file as a word entry.  The regular expression patterns are
always applied to individual words, not phrases, and must match the entire word
(i.e. they are anchored both at the start and end).

Use regular expressions sparingly, as evaluating them for every word in the
source files can be expensive.  In addition to stoplist regular expressions,
poterminology has precompiled patterns for C and Python format specifiers (e.g.
%d) and XML/HTML <elements> and &entities; -- these are removed before stoplist
processing and it is not possible to override this.

.. _poterminology_stopword_file#stoplist_words:

Stoplist words
--------------

All other lines should begin with one of the following characters, which
indicate whether the word should be **ignored** (as a word alone),
**disregarded** in a phrase (i.e. a phrase containing it is allowed, and the
word does not count against the :opt:`--term-words` length limit), or any
phrase containing it should be **excluded**.

* **+** allow word alone, allow phrases containing it
* **:** allow word alone, disregarded (for :opt:`--term-word-length`) inside
  phrase
* **<** allow word alone, but exclude any phrase containing it
* **=** ignore word alone, but allow phrases containing it
* **>** ignore word alone, disregarded (for :opt:`--term-word-length`) inside
  phrase
* **@** ignore word alone, and exclude any phrase containing it

Generally '+' is only needed for exceptions to regular expression patterns, but
it may also be used to override an entry in a previous stoplist if you are
using multiple stoplists.

Note that if a word appears multiple times in a stoplist file with different
function characters preceding it, the *last entry will take precedence* over
the others.  This is the only exception to the general rule that order is not
important in stopword files.

.. _poterminology_stopword_file#default_file_example:

Default file example
====================

::

  # apply title-case folding to words before comparing with this stoplist
  !F

The fold-titlecase setting is the default, even if it were not explicitly
specified.  This allows capitalized words at the start of a sentence (e.g.
"Who") to match a stopword "who" but allows acronyms like WHO (World Health
Organization) to be included in the terminology.  If you are using
poterminology with source files that contain large amounts of ALL UPPERCASE
TEXT you may find the ignore-case setting to be preferable.

::

  # override regex match below for phrases with 'no'
  +no

The regular expression /..? below would normally match the word 'no' and both
ignore it as a term and exclude any phrases containing it.  The above will
allow it to appear as a term and in phrases.

::

  # ignore all one or two-character words (unless =word appears below)
  /..?
  # ignore words with parenthesis, typically function() calls and the like
  /.*\(.*
  # ignore numbers, both cardinal (e.g. 1,234.0) and ordinal (e.g. 1st, 22nd)
  /[0-9,.]+(st|nd|rd|th)?

These regular expressions ignore a lot of uninteresting terms that are
typically code or other things that shouldn't be translated anyhow.  There are
many exceptions to the one or two-character word pattern in the default
stoplist file, not only with = like '=in' but also '+no' and ':on' and '<ok'
and '>of'.

::

  # allow these words by themselves and don't count against length for phrases
  :off
  :on

These prepositions are common as button text and thus useful to have as terms;
they also form an important part of phrases so are disregarded for term word
count to allow for slightly longer phrases including them.

::

  # allow these words by themselves, but ignore any phrases containing them
  <first
  <hello
  <last

These are words that are worth including in a terminology, as they are common
in applications, but which aren't generally part of idiomatic phrases.

::

  # ignore these words by themselves, but allow phrases containing them
  =able
  =about
  =actually
  =ad
  =as
  =at

This is the largest category of stoplist words, and these are all just rather
common words.  The purpose of a terminology list is to provide specific
translation suggestions for the harder words or phrases, not provide a general
dictionary, so these words are not of interest by themselves, but may well be
part of an interesting phrase.

::

  # ignore these words by themselves, but allow phrases containing them,   and
  # don't count against length for phrases
  #
  # (possible additions to this list for multi-lingual text: >di >el >le)
  #
  >a
  >an
  >and

These very common words aren't of interest by themselves, but often form an
important part of phrases so are disregarded for term word count to allow for
slightly longer phrases including them.

::

  # ignore these words and any phrases containing them
  @ain't
  @aint
  @al
  @are

These are "junk" words that are not only uninteresting by themselves, they
generally do not contribute anything to the phrases containing them.