Multibyte String

&reftitle.intro; While there are many languages in which every necessary character can be represented by a one-to-one mapping to a 8-bit value, there are also several languages which require so many characters for written communication that cannot be contained within the range a mere byte can code. Multibyte character encoding schemes were developed to express that many (more than 256) characters in the regular bytewise coding system. When you manipulate (trim, split, splice, etc.) strings encoded in a multibyte encoding, you need to use special functions since two or more consecutive bytes may represent a single character in such encoding schemes. Otherwise, if you apply a non-multibyte-aware string function to the string, it probably fails to detect the beginning or ending of the multibyte character and ends up with a corrupted garbage string that most likely loses its original meaning. mbstring provides these multibyte specific string functions that help you deal with multibyte encodings in PHP, which is basically supposed to be used with single byte encodings. In addition to that, mbstring handles character encoding conversion between the possible encoding pairs. mbstring is also designed to handle Unicode-based encodings such as UTF-8 and UCS-2 and many single-byte encodings for convenience (listed below), whereas mbstring was originally developed for use in Japanese web pages.

Encodings of the following types are safely used with PHP. A singlebyte encoding, which has ASCII-compatible (ISO646 compatible) mappings for the characters in range of 00h to 7fh. A multibyte encoding, which has ASCII-compatible mappings for the characters in range of 00h to 7fh. which don't use ISO2022 escape sequences. which don't use a value from 00h to 7fh in any of the compounded bytes that represents a single character. These are examples of character encodings that are unlikely to work with PHP. Although PHP scripts written in any of those encodings might not work, especially in the case where encoded strings appear as identifiers or literals in the script, you can almost avoid using these encodings by setting up the mbstring's transparent encoding filter function for incoming HTTP queries. It's highly discouraged to use SJIS, BIG5, CP936, CP949 and GB18030 for the internal encoding unless you are familiar with the parser, the scanner and the character encoding. If you have some database connected with PHP, it is recommended that you use the same character encoding for both database and the internal encoding for ease of use and better performance. If you are using PostgreSQL, the character encoding used in the database and the one used in the PHP may differ as it supports automatic character set conversion between the backend and the frontend.

&reference.mbstring.configure; &reference.mbstring.ini;

&reftitle.resources; &no.resource;

&reference.mbstring.constants;

HTTP input/output character encoding conversion may convert binary data also. Users are supposed to control character encoding conversion if binary data is used for HTTP input/output. In PHP 4.3.2 or earlier versions, there was a limitation in this functionality that mbstring does not perform character encoding conversion in POST data if the enctype attribute in the form element is set to multipart/form-data. So you have to convert the incoming data by yourself in this case if necessary. Beginning with PHP 4.3.3, if enctype for HTML form is set to multipart/form-data and mbstring.encoding_translation is set to On in &php.ini; the POST'ed variables and the names of uploaded files will be converted to the internal character encoding as well. However, the conversion isn't applied to the query keys. HTTP Input There is no way to control HTTP input character conversion from PHP script. To disable HTTP input character conversion, it has to be done in &php.ini;. When using PHP as an Apache module, it is possible to override those settings in each Virtual Host directive in &httpd.conf; or per directory with &htaccess;. Refer to the Configuration section and Apache Manual for details. HTTP Output There are several ways to enable output character encoding conversion. One is using &php.ini;, another is using ob_start with mb_output_handler as ob_start callback function. PHP3-i18n users should note that mbstring's output conversion differs from PHP3-i18n. Character encoding is converted using output buffer. ]]>

Currently the following character encodings are supported by the mbstring module. Any of those Character encodings can be specified in the encoding parameter of mbstring functions. The following character encoding is supported in this PHP extension: UCS-4 UCS-4BE UCS-4LE UCS-2 UCS-2BE UCS-2LE UTF-32 UTF-32BE UTF-32LE UTF-16 UTF-16BE UTF-16LE UTF-7 UTF7-IMAP UTF-8 ASCII EUC-JP SJIS eucJP-win SJIS-win ISO-2022-JP JIS ISO-8859-1 ISO-8859-2 ISO-8859-3 ISO-8859-4 ISO-8859-5 ISO-8859-6 ISO-8859-7 ISO-8859-8 ISO-8859-9 ISO-8859-10 ISO-8859-13 ISO-8859-14 ISO-8859-15 byte2be byte2le byte4be byte4le BASE64 HTML-ENTITIES 7bit 8bit EUC-CN CP936 HZ EUC-TW CP950 BIG-5 EUC-KR UHC (CP949) ISO-2022-KR Windows-1251 (CP1251) Windows-1252 (CP1252) CP866 (IBM866) KOI8-R &php.ini; entry, which accepts encoding name, accepts "auto" and "pass" also. mbstring functions, which accepts encoding name, and accepts "auto". If "pass" is set, no character encoding conversion is performed. If "auto" is set, it is expanded to the list of encodings defined per the NLS. For instance, if the NLS is set to Japanese, the value is assumed to be "ASCII,JIS,UTF-8,EUC-JP,SJIS". See also mb_detect_order

You might often find it difficult to get an existing PHP application work in a given multibyte environment. That's mostly because lots of PHP applications out there are written with the standard string functions such as substr, which are known to not properly handle multibyte-encoded strings. mbstring supports 'function overloading' feature which enables you to add multibyte awareness to such an application without code modification by overloading multibyte counterparts on the standard string functions. For example, mb_substr is called instead of substr if function overloading is enabled. This feature makes it easy to port applications that only support single-byte encodings to a multibyte environment in many cases. To use the function overloading, set mbstring.func_overload in &php.ini; to a positive value that represents a combination of bitmasks specifying the categories of functions to be overloaded. It should be set to 1 to overload the mail function. 2 for string functions, 4 for regular expression functions. For example, if is set for 7, mail, strings and regular expression functions should be overloaded. The list of overloaded functions are shown below. value of mbstring.func_overload original function overloaded function 1 mail mb_send_mail 2 strlen mb_strlen 2 strpos mb_strpos 2 strrpos mb_strrpos 2 substr mb_substr 2 strtolower mb_strtolower 2 strtoupper mb_strtoupper 2 substr_count mb_substr_count 4 ereg mb_ereg 4 eregi mb_eregi 4 ereg_replace mb_ereg_replace 4 eregi_replace mb_eregi_replace 4 split mb_split It is not recommended to use the function overloading option in the per-directory context, because it's not confirmed yet to be stable enough in a production environment and may lead to undefined behaviour.

It is often said quite hard to figure out how Japanese texts are handled in the computer. This is not only because Japanese characters can only be represented by multibyte encodings, but because different encoding standards are adopted for different purposes / platforms. Moreover, not a few character set standards are used there, which are slightly different from one another. Those facts have often led developers to inevitable mess-up. To create a working web application that would be put in the Japanese environment, it is important to use the proper character encoding and character set for the task in hand. Storage for a character can be up to six bytes Most of multibyte characters often appear twice as wide as a single-byte character on display. Those characters are called "zen-kaku" in Japanese which means "full width", and the other (narrower) characters are called "han-kaku" - means half width. However the graphical properties of the characters depend on the glyphs of the type faces used to display them or print them out. Some character encodings use shift(escape) sequences defined in ISO2022 to switch the code map of the specific code area (00h to 7fh). ISO-2022-JP should be used in SMTP/NNTP, and headers and entities should be reencoded as per RFC requirements. Although those are not requisites, it's still a good idea because several popular user agents cannot recognize any other encoding methods. Webpages created for mobile phone services such as i-mode, Vodafone live!, or EZweb are supposed to use Shift_JIS.

Multibyte character encoding schemes and the related issues are very complicated. There should be too few space to cover in sufficient details. Please refer to the following URLs and other resources for further readings. Unicode materials &url.unicode; Japanese/Korean/Chinese character information &url.oreilly.cjk-inf;

&reference.mbstring.encodings; &reference.mbstring.functions;