summaryrefslogtreecommitdiff
path: root/src/ctype
AgeCommit message (Collapse)AuthorFilesLines
2021-12-27fix wcwidth of hangul combining (vowel/final) lettersRich Felker1-58/+60
these characters combine onto a base character (initial) and therefore need to have width 0. the original binary-search implementation of wcwidth handled them correctly, but a regression was introduced in commit 1b0ce9af6d2aa7b92edaf3e9c631cb635bae22bd by generating the new tables from unicode without noticing that the classification logic in use (unicode character category Mn/Me/Cf) was insufficient to catch these characters.
2020-01-01fix wcwidth wrongly returning 0 for most of planes 4 and upRich Felker1-1/+1
commit 1b0ce9af6d2aa7b92edaf3e9c631cb635bae22bd introduced this bug back in 2012 and it was never noticed, presumably since the affected planes are essentially unused in Unicode.
2019-10-25update case mappings to unicode 12.1.0Rich Felker1-85/+92
2019-10-25update ctype data to unicode 12.1.0u_quark4-201/+232
2019-10-25overhaul wide character case mapping implementationRich Felker2-290/+345
the existing implementation of case mappings was very small (typically around 1.5k), but unmaintainable, requiring manual addition of new case mappings with each new edition of Unicode. often, it turned out that newly-added case mappings were not easily representable in the existing tightly-constrained table structures, requiring new hacks to be invented and delaying support for new characters. the new implementation added here follows the pattern used for character class membership, with a two-level table allowing Unicode blocks for which no data is needed to be elided. however, rather than single-bit data, each character maps to a one of up to 6 case-mapping rules available to its block, where 6 is floor(cbrt(256)) and allow 3 characters to be represented per byte (vs 8 with bit tables). blocks that would need more than 6 rules designate one as an exception and let lookup pass into a binary search of exceptional cases for the block. the number 6 was chosen empirically; many blocks would be ok with 4 rules (uncased, lower, upper, possible exceptions), some even just with 2, but the latter are rare and fitting 4 characters per byte rather than 3 does not save significant space. moreover, somewhat surprisingly, there are sufficiently many blocks where even 4 rules don't suffice without a lot of exceptions (blocks where some case pairs are laced, others offset) that originally I was looking at supporting variable-width tables, with 1-, 2-, or 3-bit entries, thereby allowing blocks with 8 rules. as implemented in my experiments, that version was significantly larger and involved more memory accesses/cache lines. improvements in size at the expense of some performance might be possible by utilizing iswalpha data or merging the table of case mapping identity with alphabetic identity. these were explored somewhat when the code was first written, and might be worth revisiting in the future.
2019-10-25add missing case mapping between U+03F3 and U+037FRich Felker1-0/+1
somehow this seems to have been overlooked. add it now so that subsequent overhaul of case mapping implementation will not introduce a functional change at the same time.
2018-09-12reduce spurious inclusion of libc.hRich Felker29-29/+1
libc.h was intended to be a header for access to global libc state and related interfaces, but ended up included all over the place because it was the way to get the weak_alias macro. most of the inclusions removed here are places where weak_alias was needed. a few were recently introduced for hidden. some go all the way back to when libc.h defined CANCELPT_BEGIN and _END, and all (wrongly implemented) cancellation points had to include it. remaining spurious users are mostly callers of the LOCK/UNLOCK macros and files that use the LFS64 macro to define the awful *64 aliases. in a few places, new inclusion of libc.h is added because several internal headers no longer implicitly include libc.h. declarations for __lockfile and __unlockfile are moved from libc.h to stdio_impl.h so that the latter does not need libc.h. putting them in libc.h made no sense at all, since the macros in stdio_impl.h are needed to use them correctly anyway.
2017-12-18update case mappings to unicode 10.0Rich Felker1-2/+41
the mapping tables and code are not automatically generated; they were produced by comparing the output of towupper/towlower against the mappings in the UCD, ignoring characters that were previously excluded from case mappings or from alphabetic status (micro sign and circled letters), and adding table entries or code for everything else missing. based very loosely on a patch by Reini Urban.
2017-12-18update ctype tables to unicode 10.0Rich Felker4-220/+305
2017-12-18reformat ctype tables to be diff-friendly, match tool outputRich Felker4-263/+276
the new version of the code used to generate these tables forces a newline every 256 entries, whereas at the time these files were originally generated and committed, it only wrapped them at 80 columns. the new behavior ensures that localized changes to the tables, if they are ever needed, will produce localized diffs. commit d060edf6c569ba9df4b52d6bcd93edde812869c9 made the corresponding changes to the iconv tables.
2017-05-31towupper/towlower: fast path for ascii charsNatanael Copa1-3/+3
Make a fast path for ascii chars which is assumed to be the most common case. This has significant performance benefit on xml json and similar
2015-06-16byte-based C locale, phase 1: multibyte character handling functionsRich Felker1-2/+3
this patch makes the functions which work directly on multibyte characters treat the high bytes as individual abstract code units rather than as multibyte sequences when MB_CUR_MAX is 1. since MB_CUR_MAX is presently defined as a constant 4, all of the new code added is dead code, and optimizing compilers' code generation should not be affected at all. a future commit will activate the new code. as abstract code units, bytes 0x80 to 0xff are represented by wchar_t values 0xdf80 to 0xdfff, at the end of the surrogates range. this ensures that they will never be misinterpreted as Unicode characters, and that all wctype functions return false for these "characters" without needing locale-specific logic. a high range outside of Unicode such as 0x7fffff80 to 0x7fffffff was also considered, but since C11's char16_t also needs to be able to represent conversions of these bytes, the surrogate range was the natural choice.
2015-06-06add macro version of ctype.h isascii functionRich Felker1-0/+1
presumably internal code (ungetwc and fputwc) was written assuming a macro implementation existed; otherwise use of isascii is just a pessimization.
2014-09-05fix case mapping for U+00DF (ß)Rich Felker2-2/+1
U+00DF ('ß') has had an uppercase form (U+1E9E) available since Unicode 5.1, but Unicode lacks the case mappings for it due to stability policy. when I added support for the new character in commit 1a63a9fc30e7a1f1239e3cedcb5041e5ec1c5351, I omitted the mapping in the lowercase-to-uppercase direction. this choice was not based on any actual information, only assumptions. this commit adds bidirectional case mappings between U+00DF and U+1E9E, and removes the special-case hack that allowed U+00DF to be identified as lowecase despite lacking a mapping. aside from strong evidence that this is the "right" behavior for real-world usage of these characters, several factors informed this decision: - the other "potentially correct" mapping, to "SS", is not representable in the C case-mapping system anyway. - leaving one letter in lowercase form when transforming a string to uppercase is obviously wrong. - having a character which is nominally lowercase but which is fixed under case mapping violates reasonable invariants.
2014-08-13add inline isspace in ctype.h as an optimizationSzabolcs Nagy1-0/+1
isspace can be a bottleneck in a simple parser, inlining it gives slightly smaller and faster code src/locale/pleval.o already had this optimization, the size change for other libc functions for i386 is src/internal/intscan.o 2134 2118 -16 src/locale/dcngettext.o 1562 1552 -10 src/network/res_msend.o 1961 1940 -21 src/network/lookup_name.o 2627 2608 -19 src/network/getnameinfo.o 1814 1811 -3 src/network/lookup_serv.o 643 624 -19 src/stdio/vfscanf.o 2675 2663 -12 src/stdlib/atoll.o 117 107 -10 src/stdlib/atoi.o 95 91 -4 src/stdlib/atol.o 95 91 -4 src/time/strptime.o 1515 1503 -12 (TOTALS) 432451 432321 -130
2014-07-02consolidate *_l ctype/wctype functions into their non-_l source filesRich Felker29-9/+256
the main practical purposes of this commit are to remove a huge amount of clutter from the src/locale directory, to cut down on the length of the $(AR) and $(LD) command lines, and to reduce the amount of space wasted by object file headers in the static libc.a. build time may also be reduced, though this has not been measured. as an additional justification, if there ever were a need for the behavior of these functions to vary by locale, it would be necessary for the non-_l versions to call the _l versions, so that linking the former without the latter would not be possible anyway.
2013-12-12include cleanups: remove unused headers and add feature test macrosSzabolcs Nagy4-6/+3
2013-11-11iswspace: fix handling of 0rofl0r1-2/+1
2013-03-04fix types for wctype_t and wctrans_tRich Felker1-4/+4
wctype_t was incorrectly "int" rather than "long" on x86_64. not only is this an ABI incompatibility; it's also a major design flaw if we ever wanted wctype_t to be implemented as a pointer, which would be necessary if locales support custom character classes, since int is too small to store a converted pointer. this commit fixes wctype_t to be unsigned long on all archs, matching the LSB ABI; this change does not matter for C code, but for C++ it affects mangling. the same issue applied to wctrans_t. glibc/LSB defines this type as const __int32_t *, but since no such definition is visible, I've just expanded the definition, int, everywhere. it would be nice if these types (which don't vary by arch) could be in wctype.h, but the OB XSI requirement in POSIX that wchar.h expose some types and functions from wctype.h precludes doing so. glibc works around this with some hideous hacks, but trying to duplicate that would go against the intent of musl's headers.
2013-02-02make some arrays constrofl0r3-4/+4
this way they'll go into .rodata, decreasing memory pressure.
2012-08-02fix argument type error on wcwidth functionRich Felker1-2/+2
since the correct declaration was not visible, and since the representation of the types wchar_t and wint_t always match, a compiler would have to go out of its way to make this bug manifest, but better to fix it anyway.
2012-06-20fix broken wcwidth tablesRich Felker1-7/+8
unicode char data has both "W" and "F" wide types and the old table only included the "W" ones. this omitted U+3000 (ideographic space) and all the wide-ascii, etc.
2012-06-05fix ctype abi junk (pointer should point to 0 slot, not -128 slot)Rich Felker3-3/+3
2012-06-02add LSB abi junk for ctype functionsRich Felker3-0/+104
this should be the last major fix needed to support running glibc-linked conforming POSIX programs with musl in place of glibc, as long as musl provides the features they need and they don't use pthread cancellation (which is implemented as c++ exceptions in glibc, and fundamentally incompatible with musl).
2012-04-24new wcwidth implementation (fast table-based)Rich Felker3-179/+125
i tried to go with improving the old binary-search-based algorithm, but between growth in the number of ranges, bad performance, and lack of confidence in the binary search code's stability under changes in the table, i decided it was worth the extra 1.8k to have something clean and maintainable. also note that, like the alpha and punct tables, there's definitely room to optimize the nonspacing/wide tables by overlapping subtables. this is not a high priority, but i've begun looking into how to do it, and i suspect the table sizes can be roughly halved. if that turns out to be true, the new, fast, table-based implementation will be roughly the same size as if i had just extended the old binary search one.
2012-04-23sync case mappings with unicode 6.1Rich Felker2-8/+30
also special-case ß (U+00DF) as lowercase even though it does not have a mapping to uppercase. unicode added an uppercase version of this character but does not map it, presumably because the uppercase version is not actually used except for some obscure purpose...
2012-04-23optimize iswprintRich Felker1-3/+12
2012-04-23fix spurious punct class for some surrogate codepoints (invalid)Rich Felker1-59/+56
this happened due to their entries in UnicodeData.txt
2012-04-23destubify iswalpha and update iswpunct to unicode 6.1Rich Felker5-135/+252
alpha is defined as unicode property "Alphabetic" plus category Nd minus ASCII digits minus 2 special-cased Thai punctuation marks supposedly misclassified by Unicode as letters. punct is defined as all of unicode except control, alphanumeric, and space characters. the tables were generated by a simple tool based on the code posted previously to the mailing list. in the future, this and other code used for maintaining locale/iconv/i18n data will be published either in the main source repository or in a separate locale data generation repository.
2012-02-09document iswspace and remove wrongly-included zwsp characterRich Felker1-1/+5
2012-02-09fix typo in iswspace space list tableRich Felker1-1/+1
2011-02-14more header fixes, minor warning fixRich Felker1-0/+1
2011-02-12initial check-in, version 0.5.0v0.5.0Rich Felker34-0/+854