summaryrefslogtreecommitdiff
path: root/src/locale
AgeCommit message (Collapse)AuthorFilesLines
2018-09-14drop lazy plural forms init in dcngettextRich Felker1-18/+17
there is no good reason to wait to find and process the plural rules for a translated message file until a gettext form requesting plural rule processing is used. it just imposes additional synchronization, here in the form of clunky use of atomics. it looks like there may also have been a race condition where nplurals could be seen without plural_rule being seen, possibly leading to null pointer dereference. if so, this commit fixes it.
2018-09-12split internal lock API out of libc.h, creating lock.hRich Felker3-2/+3
this further reduces the number of source files which need to include libc.h and thereby be potentially exposed to libc global state and internals. this will also facilitate further improvements like adding an inline fast-path, if we want to do so later.
2018-09-12reduce spurious inclusion of libc.hRich Felker10-11/+0
libc.h was intended to be a header for access to global libc state and related interfaces, but ended up included all over the place because it was the way to get the weak_alias macro. most of the inclusions removed here are places where weak_alias was needed. a few were recently introduced for hidden. some go all the way back to when libc.h defined CANCELPT_BEGIN and _END, and all (wrongly implemented) cancellation points had to include it. remaining spurious users are mostly callers of the LOCK/UNLOCK macros and files that use the LFS64 macro to define the awful *64 aliases. in a few places, new inclusion of libc.h is added because several internal headers no longer implicitly include libc.h. declarations for __lockfile and __unlockfile are moved from libc.h to stdio_impl.h so that the latter does not need libc.h. putting them in libc.h made no sense at all, since the macros in stdio_impl.h are needed to use them correctly anyway.
2018-09-12apply hidden visibility to various remaining internal interfacesRich Felker1-1/+3
2018-09-12overhaul internally-public declarations using wrapper headersRich Felker3-9/+2
commits leading up to this one have moved the vast majority of libc-internal interface declarations to appropriate internal headers, allowing them to be type-checked and setting the stage to limit their visibility. the ones that have not yet been moved are mostly namespace-protected aliases for standard/public interfaces, which exist to facilitate implementing plain C functions in terms of POSIX functionality, or C or POSIX functionality in terms of extensions that are not standardized. some don't quite fit this description, but are "internally public" interfacs between subsystems of libc. rather than create a number of newly-named headers to declare these functions, and having to add explicit include directives for them to every source file where they're needed, I have introduced a method of wrapping the corresponding public headers. parallel to the public headers in $(srcdir)/include, we now have wrappers in $(srcdir)/src/include that come earlier in the include path order. they include the public header they're wrapping, then add declarations for namespace-protected versions of the same interfaces and any "internally public" interfaces for the subsystem they correspond to. along these lines, the wrapper for features.h is now responsible for the definition of the hidden, weak, and weak_alias macros. this means source files will no longer need to include any special headers to access these features. over time, it is my expectation that the scope of what is "internally public" will expand, reducing the number of source files which need to include *_impl.h and related headers down to those which are actually implementing the corresponding subsystems, not just using them.
2018-09-12add internal header for declaring __pleval function (used by gettext)Rich Felker3-1/+8
locale_impl.h could have been used, but this function is completely independent of anything else, and preserving that property seems nice.
2018-09-12move __loc_is_allocated declaration to locale_impl.hRich Felker1-2/+0
2018-06-01fix output size handling for multi-unicode-char big5-hkscs charactersRich Felker1-5/+13
since this iconv implementation's output is stateless, it's necessary to know before writing anything to the output buffer whether the conversion of the current input character will fit. previously we used a hard-coded table of the output size needed for each supported output encoding, but failed to update the table when adding support for conversion to jis-based encodings and again when adding separate encoding identifiers for implicit-endianness utf-16/32 and ucs-2/4 variants, resulting in out-of-bound table reads and incorrect size checks. no buffer overflow was possible, but the affected characters could be converted incorrectly, and iconv could potentially produce an incorrect return value as a result. remove the hard-coded table, and instead perform the recursive iconv conversion to a temporary buffer, measuring the output size and transferring it to the actual output buffer only if the whole converted result fits.
2018-06-01fix iconv mapping of big5-hkscs characters that map to two unicode charsRich Felker1-1/+1
this case is handled with a recursive call to iconv using a specially-constructed conversion descriptor. the constant 0 was used as the offset for utf-8, since utf-8 appears first in the charmaps table, but the offset used needs to point into the charmap entry, past the name/aliases at the beginning, to the byte identifying the encoding. as a result of this error, junk was produced. instead, call find_charmap so we don't have to hard-code a nontrivial offset. with this change, the code has been tested and found to work in the case of converting the affected hkscs characters to utf-8.
2018-05-09fix iconv conversion to UTF-32 with implicit (big) endiannessWill Dietz1-0/+2
maintainer's notes: commit 95c6044e2ae85846330814c4ac5ebf4102dbe02c split UTF-32 and UTF-32BE but neglected to add a case for the former as a destination encoding, resulting in it wrongly being handled by the default case. the intent was that the value of the macro be chosen to encode "big endian" in the low bits, so that no code would be needed, but this was botched; instead, handle it the way UCS2 is handled.
2018-05-09fix iconv buffer overflow converting to legacy JIS-based encodingsWill Dietz1-0/+1
maintainer's notes: commit a223dbd27ae36fe53f9f67f86caf685b729593fc added the reverse conversions to JIS-based encodings, but omitted the check for remining buffer space in the case where the next character to be written was single-byte, allowing conversion to continue past the end of the destination buffer.
2018-03-07fix nl_langinfo_l(CODESET, loc) reporting wrong locale's valueRich Felker1-1/+1
use of MB_CUR_MAX encoded a hidden dependency on the currently active locale for the calling thread, whereas nl_langinfo_l is supposed to report for the locale passed as an argument.
2018-01-09revise the definition of multiple basic locks in the codeJens Gustedt3-3/+3
In all cases this is just a change from two volatile int to one.
2017-12-18fix iconv output of surrogate pairs in ucs2Rich Felker1-1/+1
in the unified code for handling utf-16 and ucs2 output, the check for ucs2 wrongly looked at the source charset rather than the destination charset.
2017-12-18add support for BOM-determined-endian UCS2, UTF-16, and UTF-32 to iconvRich Felker1-3/+40
previously, the charset names without endianness specified were always interpreted as big endian. unicode specifies that UTF-16 and UTF-32 have BOM-determined endianness if BOM is present, and are otherwise big endian. since commit 5b546faa67544af395d6407553762b37e9711157 added support for stateful encodings, it is now possible to implement BOM support via the conversion descriptor state. for conversions to these charsets, the output is always big endian and does not have a BOM.
2017-12-18add cp866 (dos cyrillic) to iconvRich Felker1-0/+12
2017-12-12add ibm1047 codepage (ebcdic representation of latin1) to iconvRich Felker1-0/+20
2017-11-14add reverse iconv mappings for JIS-based encodingsRich Felker2-1/+612
these encodings are still commonly used in messaging protocols and such. the reverse mapping is implemented as a binary search of a list of the jis 0208 characters in unicode order; the existing forward table is used to perform the comparison in the search.
2017-11-13generalize iconv framework for 8-bit codepagesRich Felker3-246/+273
previously, 8-bit codepages could only remap the high 128 bytes; the low range was assumed/forced to agree with ascii. interpretation of codepage table headers has been changed so that it's possible to represent mappings for up to 256 slots (fewer if the initial portion of the map is elided because it coincides with unicode codepoints). this requires consuming a bit more of the 10-bit space of characters that can be represented in 8-bit codepages, but there's still a plenty left. the size of the legacy_chars table is actually reduced now by eliding the first 256 entries and considering them to map implicitly via the identity map. before these changes, there seem to have been minor bugs/omissions in codepage table generation, so it's likely that some actual bug fixes are silently included in this commit. round-trip testing of a few codepages was performed on the new version of the code, but no differential testing against the old version was done.
2017-11-11reformat cjk iconv tables to be diff-friendly, match tool outputRich Felker3-2755/+2808
the new version of the code used to generate these tables forces a newline every 256 entries, whereas at the time these files were originally generated and committed, it only wrapped them at 80 columns. the new behavior ensures that localized changes to the tables, if they are ever needed, will produce localized diffs. other tables including hkscs were already committed in the new format. binary comparison of the generated object files was performed to confirm that no spurious changes slipped in.
2017-11-10add iso-2022-jp support (decoding only) to iconvRich Felker1-2/+45
this implementation aims to match the baseline defined by rfc1468 (the original mime charset definition) plus the halfwidth katakana extension included in the whatwg definition of the charset. rejection of si/so controls and newlines in doublebyte state are not currently enforced. the jis x 0201 mode is currently interpreted as having the yen sign and overline character in place of backslash and tilde; ascii mode has the standard ascii characters in those slots.
2017-11-10add iconv framework for decoding stateful encodingsRich Felker2-3/+24
assuming pointers obtained from malloc have some nonzero alignment, repurpose the low bit of iconv_t as an indicator that the descriptor is a stateless value representing the source and destination character encodings.
2017-11-10simplify/optimize iconv utf-8 caseRich Felker1-4/+3
the special case where mbrtowc returns 0 but consumed 1 byte of input does not need to be considered, because the short-circuit for low bytes already covered that case.
2017-11-10handle ascii range individually in each iconv caseRich Felker1-2/+10
short-circuiting low bytes before the switch precluded support for character encodings that don't coincide with ascii in this range. this limitation affected iso-2022 encodings, which use the esc byte to introduce a shift sequence, and things like ebcdic.
2017-11-10move iconv_close to its own translation unitRich Felker2-5/+6
this is in preparation to support stateful conversion descriptors, which are necessarily allocated and thus must be freed in iconv_close. putting it in a separate TU will avoid pulling in free if iconv_close is not referenced.
2017-11-10refactor iconv conversion descriptor encoding/decodingRich Felker1-6/+20
this change is made to avoid having assumptions about the encoding spread out across the file, and to facilitate future change to a form that can accommodate allocted, stateful descriptors when needed. this commit should not produce any functional changes; with the compiler tested the only change to code generation was minor reordering of local variables on stack.
2017-07-31add _NL_LOCALE_NAME extension to nl_langinfoRich Felker1-0/+4
since setlocale(cat, NULL) is required to return the setting for the global locale, there is no standard mechanism to obtain the name of the currently active thread-local locale set by uselocale. this makes it impossible for application/library software to load appropriate translations, etc. unless using the gettext implementation provided by libc, which has privileged access to libc internals. to fill this gap, glibc introduced the _NL_LOCALE_NAME macro which can be used with nl_langinfo to obtain the name. GNU gettext/gnulib code already use this functionality on glibc, and can easily be adapted to make use of it on non-glibc systems if it's available; for other systems they poke at locale implementation internals, which we want to avoid. this patch provides a compatible interface to the one glibc introduced.
2017-07-04fix missing volatile qualifier on lock in __get_localeJens Gustedt1-1/+1
2017-06-20fix iconv conversions for iso88592-iso885916Bartosz Brachaczek1-1/+1
commit 97bd6b09dbe7478d5a90a06ecd9e5b59389d8eb9 refactored the table lookup into a function and introduced an error in index computation. the error caused garbage to be read from the table if the given charmap had a non-zero number of elided entries.
2017-06-14catopen: set errno to EOPNOTSUPPA. Wilcox1-0/+2
Per 1003.1-2008 (2016 ed.), catopen must set errno on failure. We set errno to EOPNOTSUPP because musl does not currently support message catalogues.
2017-05-27fix iconv conversions to legacy 8bit encodingsRich Felker1-9/+12
there was missing reverse-conversion logic for the case, handled specially in the character set tables, where a byte represents a unicode codepoint with the same value. this patch adds code to handle the case, and refactors the two-level 10-bit table lookup for legacy character sets into a function to avoid repeating it yet another time as part of the fix.
2017-03-21search locale name variants for gettext translationsRich Felker1-32/+55
often translations will be named only by language, whereas locale names may also include a territory code, modifier, and codeset portion. previously, only translations exactly matching the locale name were loaded. this was a major usability issue, requiring workarounds like symlinks or tweaking of the locale name. with these changes, gettext now searches for translations by first removing the codeset portion of the locale name, then trying the remainder in full, with modifier (@mod) removed, with territory code (_XX) removed, and with both removed. part of the reason gettext lacked support for searching fallbacks before is that the candidate pathname for a translation file was constructed on each call and used as the key to lookup an already-mapped translation file. this was very costly/inefficient. we now use the tuple of textdomain binding pointer, locale map pointer, and integer category id as the key for looking up a translation file mapping. based on patch by He X.
2017-03-21make setlocale return a single name for LC_ALL if all categories matchRich Felker1-2/+5
when called for LC_ALL, setlocale has to return a string representing the state of all locale categories. the simplest way to do this was to always return a delimited list of values for each category, but that's not friendly in the fairly common case where all categories have the same setting. He X proposed a patch to check for this case and return a single name; this patch is a simplified approach to do the same.
2017-01-29avoid unbounded strlen in gettext functionsRich Felker1-3/+3
use the standard strnlen idiom for cases where lengths greater than an imposed limit are going to be rejected immediately anyway.
2017-01-29fix use of uninitialized pointer in gettext coreRich Felker1-2/+2
the plural_rule field of allocated msgcat structures was assumed to be initially-null but was never initialized. for future-proofing, the nplurals field which was left uninitialized should also be cleared. likewise, in the binding structure, the active field could be used uninitialized by a technicality: the a_store which stores the initial value of 0 may be implemented as a cas operation, which reads the old value. rather than fixing these issues individually, just use calloc for both allocations. this does result in wasteful clearing of name buffers (up to NAME_MAX+PATH_MAX) before filling them, but since the size if bounded and the time is dominated by filesystem operations, it really doesn't matter; simplicity and future-proofing have more value here. modified from patch submitted by He X.
2017-01-29fix bindtextdomain logic error deactivating other domainsRich Felker1-1/+1
this loop was only supposed to deactivate other bindings for the same text domain name, but due to copy-and-paste error, deactivated all other bindings. patch by He X.
2015-11-10fix return value of nl_langinfo for invalid item argumentsRich Felker1-5/+5
it was wrongly returning a null pointer instead of an empty string.
2015-10-01make nl_langinfo(CODESET) always return "ASCII" in byte-based C localeRich Felker1-1/+1
commit 844212d94f582c4e3c5055e0a1524931e89ebe76, which did not make it into any releases, changed nl_langinfo(CODESET) to always return "UTF-8", even in the byte-based C locale. this was problematic because application software was found to use the string match for "UTF-8" to activate its own UTF-8 processing. this both undermines the byte-based functionality of the C locale, and if mixed with with calls to the standard multibyte functions, which happened in practice, could result in severe mis-handling of input. the motive for the previous change was that, to avoid widespread compatibility problems, the string returned by nl_langinfo(CODESET) needs to be accepted by iconv and by third-party character conversion code. thus, the only remaining choice is "ASCII". this choice accurately represents the intent that high bytes do not have individual meaning in the C locale, but it does mean that iconv, when passed nl_langinfo(CODESET) in the C locale, will produce errors in cases where mbrtowc would have succeeded. for reference, glibc behaves similarly in this regard, so I don't think it will be a problem.
2015-09-24fix localeconv field value for unavailable valuesRich Felker1-14/+15
per ISO C, CHAR_MAX, not -1, is the value used to indicate that a char field in struct lconv is unavailable. patch by Julien Ramseier.
2015-09-09fix breakage in nl_langinfo from previous commitRich Felker1-1/+1
2015-09-09make nl_langinfo(CODESET) always return "UTF-8"Rich Felker1-2/+1
this restores the original behavior prior to the addition of the byte-based C locale and fixes what is effectively a regression in musl's property of always providing working UTF-8 support. commit 1507ebf837334e9e07cfab1ca1c2e88449069a80 introduced the codeset name "UTF-8-CODE-UNITS" for the byte-based C locale to represent that the semantic content is UTF-8 but that it is being processed as code units (bytes) rather than whole multibyte characters. however, many programs assume that the codeset name is usable with iconv and/or comes from a set of standard/widely-used names known to the application. such programs are likely to produce warnings or errors, run with reduced functionality, or mangle character data when run explicitly in the C locale. the standard places basically no requirements for the string returned by nl_langinfo(CODESET) and how it interacts with other interfaces, so returning "UTF-8" is permissible. moreover, it seems like the right thing to do, since the identity of the character encoding as "UTF-8" is independent of whether it is being processed as bytes of characters by the standard library functions.
2015-06-16byte-based C locale, phase 2: stdio and iconv (multibyte callers)Rich Felker1-0/+6
this patch adjusts libc components which use the multibyte functions internally, and which depend on them operating in a particular encoding, to make the appropriate locale changes before calling them and restore the calling thread's locale afterwards. activating the byte-based C locale without these changes would cause regressions in stdio and iconv. in the case of iconv, the current implementation was simply using the multibyte functions as UTF-8 conversions. setting a multibyte UTF-8 locale for the duration of the iconv operation allows the code to continue working. in the case of stdio, POSIX requires that FILE streams have an encoding rule bound at the time of setting wide orientation. as long as all locales, including the C locale, used the same encoding, treating high bytes as UTF-8, there was no need to store an encoding rule as part of the stream's state. a new locale field in the FILE structure points to the locale that should be made active during fgetwc/fputwc/ungetwc on the stream. it cannot point to the locale active at the time the stream becomes oriented, because this locale could be mutable (the global locale) or could be destroyed (locale_t objects produced by newlocale) before the stream is closed. instead, a pointer to the static C or C.UTF-8 locale object added in commit commit aeeac9ca5490d7d90fe061ab72da446c01ddf746 is used. this is valid since categories other than LC_CTYPE will not affect these functions.
2015-06-16byte-based C locale, phase 1: multibyte character handling functionsRich Felker1-1/+2
this patch makes the functions which work directly on multibyte characters treat the high bytes as individual abstract code units rather than as multibyte sequences when MB_CUR_MAX is 1. since MB_CUR_MAX is presently defined as a constant 4, all of the new code added is dead code, and optimizing compilers' code generation should not be affected at all. a future commit will activate the new code. as abstract code units, bytes 0x80 to 0xff are represented by wchar_t values 0xdf80 to 0xdfff, at the end of the surrogates range. this ensures that they will never be misinterpreted as Unicode characters, and that all wctype functions return false for these "characters" without needing locale-specific logic. a high range outside of Unicode such as 0x7fffff80 to 0x7fffffff was also considered, but since C11's char16_t also needs to be able to represent conversions of these bytes, the surrogate range was the natural choice.
2015-06-06make static C and C.UTF-8 locales available outside of newlocaleRich Felker3-21/+21
2015-06-05fix uselocale((locale_t)0) not to modify localeTimo Teräs1-3/+1
commit 68630b55c0c7219fe9df70dc28ffbf9efc8021d8 made the new locale to be assigned unconditonally resulting in crashes later on.
2015-05-27implement fail-safe static locales for newlocaleRich Felker3-13/+46
this frees applications which need to make temporary use of the C locale (via uselocale) from the possibility that newlocale might fail. the C.UTF-8 locale is also provided as a static locale. presently they behave the same, but this may change in the future.
2015-05-27rename internal locale file handling locale mapsRich Felker1-0/+0
since the __setlocalecat function was removed, the filename __setlocalecat.c no longer made sense.
2015-05-27overhaul locale internals to treat categories roughly uniformlyRich Felker6-129/+107
previously, LC_MESSAGES was treated specially as the only category which could be set to a locale name without a definition file, in order to facilitate gettext message translations when no libc locale was available. LC_NUMERIC was completely un-settable, and LC_CTYPE stored a flag intended to be used for a possible future byte-based C locale, instead of storing a __locale_map pointer like the other categories use. this patch changes all categories to be represented by pointers to __locale_map structures, and allows locale names without definition files to be treated as valid locales with trivial definition when used in any category. outwardly visible functional changes should be minor, limited mainly to the strings read back from setlocale and the way gettext handles translations in categories other than LC_MESSAGES. various internal refactoring has also been performed, and improvements in const correctness have been made.
2015-05-27replace atomics with locks in locale-setting codeRich Felker2-32/+51
this is part of a general program of removing direct use of atomics where they are not necessary to meet correctness or performance needs, but in this case it's also an optimization. only the global locale needs synchronization; allocated locales referenced with locale_t handles are immutable during their lifetimes, and using atomics to initialize them increases their cost of setup.
2015-05-21remove outdated and misleading comment in iconv.cRich Felker1-6/+0
the comment claimed that EUC/GBK/Big5 are not implemented, which has been incorrect since commit 19b4a0a20efc6b9df98b6a43536ecdd628ba4643.