Windows: encode and decode text as UTF-8 instead of trusting the locale
setlocale(LC_ALL, "") maps to the ANSI code page on Windows -- 936 (GBK)
on a Chinese system -- so wcrtomb() produces GBK where the protocol wants
UTF-8, and mbrtowc() reads UTF-8 as GBK. Typed Chinese therefore reached
the model as mojibake, and once the buffer held correct UTF-8 the input
line rendered as mojibake instead. Windows' C runtime has no ".UTF-8"
locale to switch to, so the conversion has to be explicit.
os_compat.[ch]
* add os_utf8_encode() and os_utf8_decode(), locale-independent, with
range and overlong-form checks
src/tui.c
* input_put_wc(), the command-palette filter and the ask_user buffer use
os_utf8_encode() instead of wcrtomb()
src/textsel.c
* cst_utf8_decode() uses os_utf8_decode() instead of mbrtowc()
This commit is contained in:
@@ -111,6 +111,29 @@ void os_localtime(const time_t *t, struct tm *out);
|
||||
int os_getpid(void);
|
||||
int os_chmod_private(const char *path); /* best-effort 0600; no-op/ok on Win */
|
||||
|
||||
/* --- UTF-8 --------------------------------------------------------------
|
||||
* Encode one code point as UTF-8, independent of the C locale.
|
||||
*
|
||||
* wcrtomb() follows setlocale(). On Linux the environment normally selects a
|
||||
* UTF-8 locale, but on Windows the C runtime maps setlocale(LC_ALL, "") to the
|
||||
* ANSI code page -- 936 (GBK) on a Chinese system -- so wcrtomb() hands back
|
||||
* GBK bytes where the protocol requires UTF-8, and the model receives
|
||||
* mojibake. Windows' C runtime has no ".UTF-8" locale to switch to.
|
||||
*
|
||||
* Returns the number of bytes written (1..4), or 0 if the code point cannot be
|
||||
* encoded. Surrogate halves (as they arrive on 16-bit wchar_t platforms for
|
||||
* non-BMP code points) return 0 and are skipped. */
|
||||
int os_utf8_encode(unsigned long cp, char out[4]);
|
||||
|
||||
/* Decode one UTF-8 sequence from s (at most avail bytes).
|
||||
* Returns the number of bytes consumed (1..4) and stores the code point in
|
||||
* *cp, or 0 if the bytes are not a valid, complete, non-overlong sequence.
|
||||
*
|
||||
* The counterpart of the encoder above, and needed for the same reason:
|
||||
* mbrtowc() also follows setlocale(), so on Windows it would decode UTF-8
|
||||
* text as GBK and the screen would show mojibake. */
|
||||
int os_utf8_decode(const char *s, size_t avail, unsigned long *cp);
|
||||
|
||||
/* --- libc gaps on Windows ------------------------------------------------
|
||||
* MinGW-w64 provides neither strcasestr() (a GNU extension) nor
|
||||
* setenv()/unsetenv() (POSIX): they are simply absent from its headers.
|
||||
|
||||
Reference in New Issue
Block a user