307 lines
12 KiB
HTML
307 lines
12 KiB
HTML
|
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
|
<HTML><HEAD><TITLE>Man page of PCREUNICODE</TITLE>
|
|
</HEAD><BODY>
|
|
<H1>PCREUNICODE</H1>
|
|
Section: C Library Functions (3)<BR>Updated: 27 February 2013<BR><A HREF="#index">Index</A>
|
|
<A HREF="/cgi-bin/man/man2html">Return to Main Contents</A><HR>
|
|
|
|
<A NAME="lbAB"> </A>
|
|
<H2>NAME</H2>
|
|
|
|
PCRE - Perl-compatible regular expressions
|
|
<A NAME="lbAC"> </A>
|
|
<H2>UTF-8, UTF-16, UTF-32, AND UNICODE PROPERTY SUPPORT</H2>
|
|
|
|
|
|
<P>
|
|
As well as UTF-8 support, PCRE also supports UTF-16 (from release 8.30) and
|
|
UTF-32 (from release 8.32), by means of two additional libraries. They can be
|
|
built as well as, or instead of, the 8-bit library.
|
|
<A NAME="lbAD"> </A>
|
|
<H2>UTF-8 SUPPORT</H2>
|
|
|
|
|
|
<P>
|
|
In order process UTF-8 strings, you must build PCRE's 8-bit library with UTF
|
|
support, and, in addition, you must call
|
|
|
|
<B>pcre_compile()</B>
|
|
|
|
with the PCRE_UTF8 option flag, or the pattern must start with the sequence
|
|
(*UTF8) or (*UTF). When either of these is the case, both the pattern and any
|
|
subject strings that are matched against it are treated as UTF-8 strings
|
|
instead of strings of individual 1-byte characters.
|
|
<A NAME="lbAE"> </A>
|
|
<H2>UTF-16 AND UTF-32 SUPPORT</H2>
|
|
|
|
|
|
<P>
|
|
In order process UTF-16 or UTF-32 strings, you must build PCRE's 16-bit or
|
|
32-bit library with UTF support, and, in addition, you must call
|
|
|
|
<B>pcre16_compile()</B>
|
|
|
|
or
|
|
|
|
<B>pcre32_compile()</B>
|
|
|
|
with the PCRE_UTF16 or PCRE_UTF32 option flag, as appropriate. Alternatively,
|
|
the pattern must start with the sequence (*UTF16), (*UTF32), as appropriate, or
|
|
(*UTF), which can be used with either library. When UTF mode is set, both the
|
|
pattern and any subject strings that are matched against it are treated as
|
|
UTF-16 or UTF-32 strings instead of strings of individual 16-bit or 32-bit
|
|
characters.
|
|
<A NAME="lbAF"> </A>
|
|
<H2>UTF SUPPORT OVERHEAD</H2>
|
|
|
|
|
|
<P>
|
|
If you compile PCRE with UTF support, but do not use it at run time, the
|
|
library will be a bit bigger, but the additional run time overhead is limited
|
|
to testing the PCRE_UTF[8|16|32] flag occasionally, so should not be very big.
|
|
<A NAME="lbAG"> </A>
|
|
<H2>UNICODE PROPERTY SUPPORT</H2>
|
|
|
|
|
|
<P>
|
|
If PCRE is built with Unicode character property support (which implies UTF
|
|
support), the escape sequences \p{..}, \P{..}, and \X can be used.
|
|
The available properties that can be tested are limited to the general
|
|
category properties such as Lu for an upper case letter or Nd for a decimal
|
|
number, the Unicode script names such as Arabic or Han, and the derived
|
|
properties Any and L&. Full lists is given in the
|
|
|
|
<B>pcrepattern</B>
|
|
|
|
and
|
|
|
|
<B>pcresyntax</B>
|
|
|
|
documentation. Only the short names for properties are supported. For example,
|
|
\p{L} matches a letter. Its Perl synonym, \p{Letter}, is not supported.
|
|
Furthermore, in Perl, many properties may optionally be prefixed by "Is", for
|
|
compatibility with Perl 5.6. PCRE does not support this.
|
|
|
|
<A NAME="lbAH"> </A>
|
|
<H3>Validity of UTF-8 strings</H3>
|
|
|
|
|
|
<P>
|
|
When you set the PCRE_UTF8 flag, the byte strings passed as patterns and
|
|
subjects are (by default) checked for validity on entry to the relevant
|
|
functions. The entire string is checked before any other processing takes
|
|
place. From release 7.3 of PCRE, the check is according the rules of RFC 3629,
|
|
which are themselves derived from the Unicode specification. Earlier releases
|
|
of PCRE followed the rules of RFC 2279, which allows the full range of 31-bit
|
|
values (0 to 0x7FFFFFFF). The current check allows only values in the range U+0
|
|
to U+10FFFF, excluding the surrogate area. (From release 8.33 the so-called
|
|
"non-character" code points are no longer excluded because Unicode corrigendum
|
|
#9 makes it clear that they should not be.)
|
|
<P>
|
|
|
|
Characters in the "Surrogate Area" of Unicode are reserved for use by UTF-16,
|
|
where they are used in pairs to encode codepoints with values greater than
|
|
0xFFFF. The code points that are encoded by UTF-16 pairs are available
|
|
independently in the UTF-8 and UTF-32 encodings. (In other words, the whole
|
|
surrogate thing is a fudge for UTF-16 which unfortunately messes up UTF-8 and
|
|
UTF-32.)
|
|
<P>
|
|
|
|
If an invalid UTF-8 string is passed to PCRE, an error return is given. At
|
|
compile time, the only additional information is the offset to the first byte
|
|
of the failing character. The run-time functions <B>pcre_exec()</B> and
|
|
<B>pcre_dfa_exec()</B> also pass back this information, as well as a more
|
|
detailed reason code if the caller has provided memory in which to do this.
|
|
<P>
|
|
|
|
In some situations, you may already know that your strings are valid, and
|
|
therefore want to skip these checks in order to improve performance, for
|
|
example in the case of a long subject string that is being scanned repeatedly.
|
|
If you set the PCRE_NO_UTF8_CHECK flag at compile time or at run time, PCRE
|
|
assumes that the pattern or subject it is given (respectively) contains only
|
|
valid UTF-8 codes. In this case, it does not diagnose an invalid UTF-8 string.
|
|
<P>
|
|
|
|
Note that passing PCRE_NO_UTF8_CHECK to <B>pcre_compile()</B> just disables the
|
|
check for the pattern; it does not also apply to subject strings. If you want
|
|
to disable the check for a subject string you must pass this option to
|
|
<B>pcre_exec()</B> or <B>pcre_dfa_exec()</B>.
|
|
<P>
|
|
|
|
If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set, the result
|
|
is undefined and your program may crash.
|
|
|
|
<A NAME="lbAI"> </A>
|
|
<H3>Validity of UTF-16 strings</H3>
|
|
|
|
|
|
<P>
|
|
When you set the PCRE_UTF16 flag, the strings of 16-bit data units that are
|
|
passed as patterns and subjects are (by default) checked for validity on entry
|
|
to the relevant functions. Values other than those in the surrogate range
|
|
U+D800 to U+DFFF are independent code points. Values in the surrogate range
|
|
must be used in pairs in the correct manner.
|
|
<P>
|
|
|
|
If an invalid UTF-16 string is passed to PCRE, an error return is given. At
|
|
compile time, the only additional information is the offset to the first data
|
|
unit of the failing character. The run-time functions <B>pcre16_exec()</B> and
|
|
<B>pcre16_dfa_exec()</B> also pass back this information, as well as a more
|
|
detailed reason code if the caller has provided memory in which to do this.
|
|
<P>
|
|
|
|
In some situations, you may already know that your strings are valid, and
|
|
therefore want to skip these checks in order to improve performance. If you set
|
|
the PCRE_NO_UTF16_CHECK flag at compile time or at run time, PCRE assumes that
|
|
the pattern or subject it is given (respectively) contains only valid UTF-16
|
|
sequences. In this case, it does not diagnose an invalid UTF-16 string.
|
|
However, if an invalid string is passed, the result is undefined.
|
|
|
|
<A NAME="lbAJ"> </A>
|
|
<H3>Validity of UTF-32 strings</H3>
|
|
|
|
|
|
<P>
|
|
When you set the PCRE_UTF32 flag, the strings of 32-bit data units that are
|
|
passed as patterns and subjects are (by default) checked for validity on entry
|
|
to the relevant functions. This check allows only values in the range U+0
|
|
to U+10FFFF, excluding the surrogate area U+D800 to U+DFFF.
|
|
<P>
|
|
|
|
If an invalid UTF-32 string is passed to PCRE, an error return is given. At
|
|
compile time, the only additional information is the offset to the first data
|
|
unit of the failing character. The run-time functions <B>pcre32_exec()</B> and
|
|
<B>pcre32_dfa_exec()</B> also pass back this information, as well as a more
|
|
detailed reason code if the caller has provided memory in which to do this.
|
|
<P>
|
|
|
|
In some situations, you may already know that your strings are valid, and
|
|
therefore want to skip these checks in order to improve performance. If you set
|
|
the PCRE_NO_UTF32_CHECK flag at compile time or at run time, PCRE assumes that
|
|
the pattern or subject it is given (respectively) contains only valid UTF-32
|
|
sequences. In this case, it does not diagnose an invalid UTF-32 string.
|
|
However, if an invalid string is passed, the result is undefined.
|
|
<A NAME="lbAK"> </A>
|
|
<H3>General comments about UTF modes</H3>
|
|
|
|
|
|
<P>
|
|
1. Codepoints less than 256 can be specified in patterns by either braced or
|
|
unbraced hexadecimal escape sequences (for example, \x{b3} or \xb3). Larger
|
|
values have to use braced sequences.
|
|
<P>
|
|
|
|
2. Octal numbers up to \777 are recognized, and in UTF-8 mode they match
|
|
two-byte characters for values greater than \177.
|
|
<P>
|
|
|
|
3. Repeat quantifiers apply to complete UTF characters, not to individual
|
|
data units, for example: \x{100}{3}.
|
|
<P>
|
|
|
|
4. The dot metacharacter matches one UTF character instead of a single data
|
|
unit.
|
|
<P>
|
|
|
|
5. The escape sequence \C can be used to match a single byte in UTF-8 mode, or
|
|
a single 16-bit data unit in UTF-16 mode, or a single 32-bit data unit in
|
|
UTF-32 mode, but its use can lead to some strange effects because it breaks up
|
|
multi-unit characters (see the description of \C in the
|
|
|
|
<B>pcrepattern</B>
|
|
|
|
documentation). The use of \C is not supported in the alternative matching
|
|
function <B>pcre[16|32]_dfa_exec()</B>, nor is it supported in UTF mode by the
|
|
JIT optimization of <B>pcre[16|32]_exec()</B>. If JIT optimization is requested
|
|
for a UTF pattern that contains \C, it will not succeed, and so the matching
|
|
will be carried out by the normal interpretive function.
|
|
<P>
|
|
|
|
6. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly
|
|
test characters of any code value, but, by default, the characters that PCRE
|
|
recognizes as digits, spaces, or word characters remain the same set as in
|
|
non-UTF mode, all with values less than 256. This remains true even when PCRE
|
|
is built to include Unicode property support, because to do otherwise would
|
|
slow down PCRE in many common cases. Note in particular that this applies to
|
|
\b and \B, because they are defined in terms of \w and \W. If you really
|
|
want to test for a wider sense of, say, "digit", you can use explicit Unicode
|
|
property tests such as \p{Nd}. Alternatively, if you set the PCRE_UCP option,
|
|
the way that the character escapes work is changed so that Unicode properties
|
|
are used to determine which characters match. There are more details in the
|
|
section on
|
|
|
|
|
|
generic character types
|
|
|
|
in the
|
|
|
|
<B>pcrepattern</B>
|
|
|
|
documentation.
|
|
<P>
|
|
|
|
7. Similarly, characters that match the POSIX named character classes are all
|
|
low-valued characters, unless the PCRE_UCP option is set.
|
|
<P>
|
|
|
|
8. However, the horizontal and vertical white space matching escapes (\h, \H,
|
|
\v, and \V) do match all the appropriate Unicode characters, whether or not
|
|
PCRE_UCP is set.
|
|
<P>
|
|
|
|
9. Case-insensitive matching applies only to characters whose values are less
|
|
than 128, unless PCRE is built with Unicode property support. A few Unicode
|
|
characters such as Greek sigma have more than two codepoints that are
|
|
case-equivalent. Up to and including PCRE release 8.31, only one-to-one case
|
|
mappings were supported, but later releases (with Unicode property support) do
|
|
treat as case-equivalent all versions of characters such as Greek sigma.
|
|
<A NAME="lbAL"> </A>
|
|
<H2>AUTHOR</H2>
|
|
|
|
|
|
<P>
|
|
<PRE>
|
|
Philip Hazel
|
|
University Computing Service
|
|
Cambridge CB2 3QH, England.
|
|
</PRE>
|
|
|
|
<A NAME="lbAM"> </A>
|
|
<H2>REVISION</H2>
|
|
|
|
|
|
<P>
|
|
<PRE>
|
|
Last updated: 27 February 2013
|
|
Copyright (c) 1997-2013 University of Cambridge.
|
|
</PRE>
|
|
|
|
<P>
|
|
|
|
<HR>
|
|
<A NAME="index"> </A><H2>Index</H2>
|
|
<DL>
|
|
<DT id="1"><A HREF="#lbAB">NAME</A><DD>
|
|
<DT id="2"><A HREF="#lbAC">UTF-8, UTF-16, UTF-32, AND UNICODE PROPERTY SUPPORT</A><DD>
|
|
<DT id="3"><A HREF="#lbAD">UTF-8 SUPPORT</A><DD>
|
|
<DT id="4"><A HREF="#lbAE">UTF-16 AND UTF-32 SUPPORT</A><DD>
|
|
<DT id="5"><A HREF="#lbAF">UTF SUPPORT OVERHEAD</A><DD>
|
|
<DT id="6"><A HREF="#lbAG">UNICODE PROPERTY SUPPORT</A><DD>
|
|
<DL>
|
|
<DT id="7"><A HREF="#lbAH">Validity of UTF-8 strings</A><DD>
|
|
<DT id="8"><A HREF="#lbAI">Validity of UTF-16 strings</A><DD>
|
|
<DT id="9"><A HREF="#lbAJ">Validity of UTF-32 strings</A><DD>
|
|
<DT id="10"><A HREF="#lbAK">General comments about UTF modes</A><DD>
|
|
</DL>
|
|
<DT id="11"><A HREF="#lbAL">AUTHOR</A><DD>
|
|
<DT id="12"><A HREF="#lbAM">REVISION</A><DD>
|
|
</DL>
|
|
<HR>
|
|
This document was created by
|
|
<A HREF="/cgi-bin/man/man2html">man2html</A>,
|
|
using the manual pages.<BR>
|
|
Time: 00:05:52 GMT, March 31, 2021
|
|
</BODY>
|
|
</HTML>
|