The Okapi project’s main purpose is to architect a set of building blocks for the creation of larger open source localization and translation tools. But many Okapi components are generic enough to be of interest to the text mining, natural language processing, and text retrieval communities. Okapi’s many text filters (HTML, Properties, XML (ITS XPath-based rules), OpenXML, ODF, Regex etc.) provide a straightforward way to access the text of multiple document formats. Its document events and pipeline can be made to integrate with other frameworks such as UIMA, LingPipe, OpenPipeline, OpenNLP, GATE, and Lucene. The advantage of Okapi’s text filters is that not only is text extracted, but all non-textual formatting is preserved. It is possible to decompose a document into events, process them via the pipeline, and then rebuild the input document without loss. Structural information can be added to Okapi document events so that tables, lists, links, titles etc. are grouped together and treated as a unit. This is useful when context based on a “universal” document structure is needed. The Okapi event model supports user configurable annotations, similar to UIMA, but simpler and more restricted in scope. User can annotate spans of text or add new resources such as translation memory matches, terminology, token types, or part of speech information.
libuninum is a library for converting Unicode strings to integers and integers to Unicode strings. Internal computation is done using arbitrary precision arithmetic, so there is no limit on the size of the integer that can be converted. Values are passed and returned as ASCII decimal strings, GNU MP mpz_t objects, or unsigned long integers. Auto-detection of the number system is provided. Very many number systems are supported. Group delimitation for output strings is fully controllable. Command line and graphical interfaces are also provided.
The Gettext Commons project provides Java classes for internationalization through GNU gettext and Java ResourceBundles. This makes it possible to use the original text instead of custom property keys, which is less cumbersome and makes programs easier to read. It supports the easy extraction of user visible strings, marks strings as fuzzy when the original text changes to allow translators to check translations, powerful plural handling, and build process integration through Maven.
HEBCI is a technique that allows a Web form handler to transparently detect the character set with which its data was encoded. By using carefully-chosen character references, the browser's encoding can be inferred. Thus, it is possible to guarantee that data is in a standard encoding without relying on (often unreliable) Web server/browser encoding interactions.
PHP Net_IDNA is a class to convert between the Punycode and Unicode formats. Punycode is a standard described in RFC 3492 and part of IDNA (Internationalizing Domain Names in Applications [RFC3490]) . This class allows PHP scripts to convert these domain names without having one of the PHP extensions installed. It supports both IDNA 2003 and IDNA 2008.
Mozilla Localization with PO Files provides Mozilla DTD and properties files converted to the Gettext PO format to allow easier localization using one of the graphical PO editors such as KBabel, GTranslator, or poEdit. These files are created using the moz2po application from the Translate package.