Draft:UCE-8
Where to get help
How to improve a draft
You can also browse Wikipedia:Featured articles and Wikipedia:Good articles to find examples of Wikipedia's best writing on topics similar to your proposed article. Improving your odds of a speedy review To improve your odds of a faster review, tag your draft with relevant WikiProject tags using the button below. This will let reviewers know a new draft has been submitted in their area of interest. For instance, if you wrote about a female astronomer, you would want to add the Biography, Astronomy, and Women scientists tags. Editor resources
|
| Alias(es) | Unicode Compact Encoding |
|---|---|
| Standard | None (experimental) |
| Current status | Experimental; not adopted by any standards body |
| Classification | Unicode Transformation Format-like, variable-width |
| Extends | ASCII |
| Transforms / Encodes | Unicode |
UCE-8 (Unicode Compact Encoding) is an experimental variable-width character encoding for Unicode that represents each code point in one, two or three bytes. It was designed to reduce the storage and transmission cost of scripts that UTF-8 encodes in three bytes, principally the writing systems of South Asia, Southeast Asia, East Asia and the Horn of Africa.
Unlike UTF-8, in which the leading byte identifies both the length and the general range of a character, UCE-8 places the identifying information in the final byte of a sequence. The high bit of each byte acts solely as a continuation flag, and the terminating byte doubles as an index into one of 68 character "pages".
Design
UCE-8 assigns byte lengths by script rather than by position in the Unicode code space. The difference from UTF-8 falls entirely on the third tier, which UCE-8 promotes to two bytes and UTF-8 does not:
| Category | UTF-8 | UCE-8 |
|---|---|---|
| ASCII text | 1 byte | 1 byte |
| European and Middle Eastern scripts | 2 bytes | 2 bytes |
| Asian and African scripts | 3 bytes | 2 bytes |
| Historic scripts and other supplementary-plane characters | 4 bytes | 3 bytes |
The saving applies only to the characters the two-byte tier actually holds; the scripts concerned, and how completely each is covered, are listed under Page map below.
Byte structure
The most significant bit of every byte indicates whether another byte follows. A byte with the high bit set ("lead byte") is followed by at least one more byte; a byte with the high bit clear ("trail byte") terminates the character. The encoding is therefore self-terminating and requires no length field.
| Tier | Byte pattern | Length | Characters reachable | Range covered |
|---|---|---|---|---|
| 1 | 0xxxxxxx |
1 byte | 128 | ASCII |
| 2 | 1xxxxxxx 0xxxxxxx |
2 bytes | 8,704 | 68 pages of 128 characters |
| 3 | 1xxxxxxx 1xxxxxxx 0xxxxxxx |
3 bytes | 1,114,112 | All remaining code points |
The capacity of the three-byte tier, 128 × 128 × 68, equals 1,114,112, which is also the total size of the Unicode code space (17 × 65,536).
Addressing
In a two-byte sequence the trail byte selects a page of 128 characters and the lead byte selects a position, or "slot", within it. This is the reverse of UTF-8, where the leading byte carries the identifying information.
For example, the Devanagari letter na (U+0928) is encoded as the byte pair A8 35:
A8=1 0101000— high bit set, slot 28 hex35=0 0110101— high bit clear, page index 35 hex
Page 35 hex corresponds to the Unicode block beginning at U+0900, so the sequence resolves to U+0900 + 28 hex = U+0928. Holding the lead byte constant and changing only the trail byte selects the same slot in a different script.
Restricted trail bytes
Of the 128 possible trail-byte values, 60 are excluded: the 33 control codes, the space character, and 26 punctuation marks that have structural meaning in common data formats such as JSON, CSV, URLs, file paths and shell syntax. Only six punctuation marks remain usable as trail bytes: !, (, ), ^, _ and ~.
This leaves 68 usable page indices, giving the two-byte tier a capacity of 128 × 68 = 8,704 characters.
Page map
The following table lists every trail-byte value. The row gives the high nibble and the column the low nibble; the small character in each cell is the ASCII character that byte represents, and the value beneath it is the page selected. WLD, CHN and KOR mark indexed pages, which hold a curated list rather than an aligned Unicode block.
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | A | B | C | D | E | F | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | The 32 control codes occupy rows 0 and 1 and can never be a page index. | |||||||||||||||
| 1 | ||||||||||||||||
| 2 | SP | ! 008 |
" | # | $ | % | & | ' | ( 010 |
) 038 |
* | + | , | - | . | / |
| 3 | 0 040 |
1 050 |
2 058 |
3 060 |
4 068 |
5 090 |
6 098 |
7 0A0 |
8 0A8 |
9 0B0 |
: | ; | < | = | > | ? |
| 4 | @ | A 0C0 |
B 0C8 |
C 0D0 |
D 0D8 |
E 0E0 |
F 0E8 |
G 0F0 |
H 100 |
I 120 |
J 128 |
K 130 |
L 178 |
M 1E8 |
N 300 |
O 308 |
| 5 | P WLD |
Q WLD |
R 4E0 |
S 4E8 |
T 540 |
U 5B8 |
V 628 |
W 7E8 |
X 8B8 |
Y 8D0 |
Z CHN |
[ | \ | ] | ^ CHN |
_ CHN |
| 6 | ` | a CHN |
b CHN |
c CHN |
d CHN |
e CHN |
f CHN |
g CHN |
h CHN |
i CHN |
j CHN |
k CHN |
l CHN |
m CHN |
n CHN |
o CHN |
| 7 | p CHN |
q CHN |
r CHN |
s CHN |
t CHN |
u CHN |
v KOR |
w KOR |
x KOR |
y KOR |
z KOR |
{ | | | } | ~ KOR |
DEL |
| Colour | Category | Pages |
|---|---|---|
| World scripts | 30 | |
| Chinese | 32 | |
| Korean | 6 | |
| Excluded | 60 |
Devanagari occupies row 3, column 5 — the ASCII digit 5 — which is why text in that script produces byte streams containing that digit at every second position.
The page values in the table above are Unicode block addresses with the final hexadecimal digit removed; page 090 therefore denotes the 128 code points beginning at U+0900. The scripts they correspond to are:
| Page | Script | Unicode block | Speakers (millions) | Coverage |
|---|---|---|---|---|
090 |
Devanagari | U+0900–U+097F | 600 | Complete block |
098 |
Bengali | U+0980–U+09FF | 284 | Complete block |
0A0 |
Gurmukhi | U+0A00–U+0A7F | 30 | Complete block |
0A8 |
Gujarati | U+0A80–U+0AFF | 57 | Complete block |
0B0 |
Odia | U+0B00–U+0B7F | 41 | Complete block |
0B8 |
Tamil | U+0B80–U+0BFF | 78 | Indexed: the 61 assigned characters, the rest of the block being reserved |
0C0 |
Telugu | U+0C00–U+0C7F | 83 | Complete block |
0C8 |
Kannada | U+0C80–U+0CFF | 43 | Complete block |
0D0 |
Malayalam | U+0D00–U+0D7F | 45 | Complete block |
0D8 |
Sinhala | U+0D80–U+0DFF | 17 | Complete block |
0E0 |
Thai | U+0E00–U+0E7F | 71 | Complete block |
0E8 |
Lao | U+0E80–U+0EFF | 7.7 | Complete block |
0F0 |
Tibetan | U+0F00–U+0FFF | 6 | First half of the block |
100 |
Myanmar | U+1000–U+109F | 33 | First 128 code points |
108 |
Georgian | U+10A0–U+10FF | 3.6 | Indexed, on a shared world page |
120 |
Ethiopic | U+1200–U+137F | 135 | Three pages |
178 |
Khmer | U+1780–U+17FF | 18 | Complete block |
180 |
Mongolian | U+1800–U+18AF | 10 | Indexed: 32 traditional-script characters. Mongolia writes in Cyrillic, which is unaffected |
1E8 |
Vietnamese | U+1E00–U+1EFF | 86 | Shared with Yoruba, Igbo and Welsh; most Vietnamese text is already ASCII |
300 |
Japanese | U+3040–U+30FF | 122 | Two pages: kana complete, kanji only where they coincide with the Chinese index |
4E0 |
Chinese | U+4E00–U+9FFF | 1,400 | 32 pages, 4,096 most frequent characters; simplified only |
AC0 |
Korean | U+AC00–U+D7A3 | 78 | Indexed: 768 most frequent syllables of 11,172 |
Population figures are approximate and are given for the writing system rather than for any one language. The scripts listed all fall in the range UTF-8 encodes in three bytes.
Page allocation
| Category | Pages | Slots | Contents |
|---|---|---|---|
| Alphabetic world scripts | 30 | 3,840 | Latin, Greek, Cyrillic, Armenian, Hebrew, Arabic, the Indic scripts, Thai, Lao, Tibetan, Myanmar, Georgian, Ethiopic, Khmer, Vietnamese, Kana, Mongolian, currency signs |
| Chinese | 32 | 4,096 | Most frequent Han characters |
| Korean | 6 | 768 | Most frequent Hangul syllables |
| Total | 68 | 8,704 |
Pages are of two kinds. Block pages map a 128-code-point-aligned slice of Unicode, so the slot is computed arithmetically. Indexed pages hold a curated list of 128 entries, so the slot requires a table lookup. Chinese and Korean use indexed pages because Unicode orders Han characters by radical and Hangul syllables by jamo composition, neither of which correlates with frequency of use.
Comparison with other encodings
UCE-8 is not the first encoding to represent East Asian scripts in two bytes. GB 18030, Shift JIS, EUC-KR and Big5 all predate it and encode their respective scripts in two bytes, but each covers a single language community and they are mutually incompatible. UTF-16 encodes the entire Basic Multilingual Plane in two bytes but requires two bytes for ASCII as well.
| Script | UTF-8 | UTF-16 | GB 18030 | UCE-8 |
|---|---|---|---|---|
| ASCII | 1 | 2 | 1 | 1 |
| Cyrillic, Greek | 2 | 2 | 2 | 2 |
| Hebrew, Arabic | 2 | 2 | 4 | 2 |
| Indic, Thai, Ethiopic | 3 | 2 | 4 | 2 |
| Chinese, Japanese kana | 3 | 2 | 2 | 2 |
| Korean | 3 | 2 | 4 | 2 |
| Supplementary planes | 4 | 4 | 4 | 3 |
Unicode also defines two compression schemes with related goals, the Standard Compression Scheme for Unicode (UTS #6) and BOCU-1 (UTS #12), neither of which is widely deployed.
Limitations
- Incompatibility. A UCE-8 byte stream is not valid UTF-8, and vice versa, except for pure ASCII, which is identical in both.
- Byte order does not follow code-point order. Because indexed pages are ordered by frequency rather than by code point, sorting encoded bytes does not sort characters, affecting binary search and range queries over raw encoded data.
- Context-dependent bytes. A trail byte cannot be distinguished from a standalone ASCII character by inspection alone; determining which requires examining the preceding byte. UTF-8, by contrast, marks continuation bytes with a reserved
10xxxxxxprefix. The designer describes this as a deliberate trade: reserving such a prefix would reduce the two-byte tier from 8,704 characters to 4,096. - Substring search. Because trail bytes are ordinary ASCII digits and letters, a naive byte-level substring search can report false positives. These can be eliminated by testing the byte preceding each candidate match, described under Character boundaries below. The test applies only to code that performs the search, so tools that are unaware of the encoding — grep, SQL
LIKE, or a general-purpose search index — remain affected. - Fixed repertoire. The curated character tables cannot be reordered without invalidating previously encoded data.
- Interaction with compression. Much of the size advantage over UTF-8 is reduced when general-purpose compression such as gzip or Brotli is applied, since UTF-8's repeated lead bytes are themselves highly compressible.
Implementation
A reference implementation in Python, together with the frozen character tables and a browser-based demonstration, is published by the designer under a public repository.[1]
Character tables
The 32 indexed pages are backed by three ordered strings, one per category. A character's slot is its position in the string, so the tables are the encoding: reordering them would change the meaning of every byte sequence already written. They are consequently frozen.
| Table | Pages | Characters | Contents |
|---|---|---|---|
CHINESE_CHARS |
24 | 3,072 | Han characters by frequency, excluding those the eight Chinese block pages already reach |
| 十厂卜八入儿匕几刁刀力又干工土士才大小口山巾千川夕勺凡及 … | |||
KOREAN_CHARS |
6 | 768 | Hangul syllables ordered by the frequency of their component jamo |
| 아이가으나기사하다오니그시히디안어우느자마스고흐드알인바 … | |||
WORLD_CHARS |
2 | 256 | Cyrillic extensions, Georgian, Mongolian, Tamil, currency signs and additional Latin letters |
| ӘәҒғҚқҢңӨөҰұҮүҺһҖҗҘҙҠҡҪҫҜҝҸҹ … | |||
| Total | 32 | 4,096 | |
Each table fills its pages exactly, 32 × 128 = 4,096 entries with none unused. The complete tables are too long to reproduce here; they appear in full in the reference implementation.[1]
Encoding
The encoder tries the block pages first, since their slot is arithmetic, then the three indexed tables, and finally falls back to the three-byte tier. LEAD_BASE is 0x80, the bit that marks a byte as non-terminating, and TAIL_COUNT is 68, the number of usable trail values.
def encode(code_point):
"""Encode a Unicode code point into 1, 2 or 3 bytes."""
# TIER 1 - plain ASCII passes straight through
if code_point < 0x80:
return bytes([code_point])
# TIER 2a - block pages: an aligned 128-code-point slice of Unicode
trail = PAGE_TO_TRAIL.get((code_point >> 7) << 3)
if trail is not None:
return bytes([LEAD_BASE | (code_point & 0x7F), trail])
# TIER 2b-d - indexed pages: the slot comes from a curated table
for table, trails in ((CHINESE_SLOT, CHINESE_TRAILS),
(KOREAN_SLOT, KOREAN_TRAILS),
(WORLD_SLOT, WORLD_TRAILS)):
slot = table.get(code_point)
if slot is not None:
return bytes([LEAD_BASE | (slot & 0x7F), trails[slot >> 7]])
# TIER 3 - everything else, spread across the 68 legal terminators
index = code_point - 0x80
rest, tail = divmod(index, TAIL_COUNT)
return bytes([
LEAD_BASE | (rest >> 7),
LEAD_BASE | (rest & 0x7F),
TAIL_BYTES[tail],
])
The third tier has no page table left to consult, so it distributes the remaining code points arithmetically: the remainder modulo 68 selects one of the legal terminating bytes and the quotient becomes the two lead bytes. This is why the tier's capacity is exactly 128 × 128 × 68, and why the encoder cannot emit an illegal terminator even in its fallback path.
Decoding
The decoder reads the high bit of the second byte to choose the tier, then uses the trail byte as the page index.
def decode(data):
"""Decode a 1, 2 or 3 byte sequence back into a Unicode code point."""
first = data[0]
if first < 0x80:
return first
second = data[1]
# TIER 2 - the second byte clears the high bit, so it terminates
if second < 0x80:
page = TRAIL_TO_PAGE[second]
if page is WORLD:
return WORLD_CODE_POINTS[(WORLD_PAGE[second] << 7) | (first & 0x7F)]
if page is CHINESE:
return CHINESE_CODE_POINTS[(CHINESE_PAGE[second] << 7) | (first & 0x7F)]
if page is KOREAN:
return KOREAN_CODE_POINTS[(KOREAN_PAGE[second] << 7) | (first & 0x7F)]
return ((page >> 3) << 7) | (first & 0x7F)
# TIER 3 - the second byte keeps the high bit, so a third follows
tail = TAIL_INDEX[data[2]]
rest = ((first & 0x7F) << 7) | (second & 0x7F)
return 0x80 + rest * TAIL_COUNT + tail
Character boundaries
Because a trail byte is indistinguishable from a standalone ASCII character in isolation, a byte-level substring search over a UCE-8 stream can match the interior of a multi-byte character. The effect is pronounced for scripts whose page index happens to be a common ASCII character: Devanagari's page index is 0x35, the digit 5, so Devanagari text contains that byte at every second position. Searching such a stream for the digit 5 returns one match per character in addition to any genuine digit.
The ambiguity is resolved by a single test on the preceding byte. A low byte always terminates a character, so a position begins a character precisely when the byte before it is also low, or when it is the start of the stream:
def is_char_start(data, i):
"""A low byte terminates a character, so position i begins one."""
return i == 0 or not (data[i - 1] & 0x80)
Only the first byte of a candidate match needs this test. If that byte begins a character and is itself low, it also terminates one, so the following byte begins a character in turn, and the property carries along the whole search term. The same test governs where a stream may be split: a cut is safe unless the byte before it is a lead byte.
See also
- Comparison of Unicode encodings
- Standard Compression Scheme for Unicode
- BOCU-1
- GB 18030
- Variable-width encoding
References
- ^ a b "UCE-8 reference implementation". GitLab. Retrieved 2026-08-11.
External links
Category:Unicode Transformation Formats Category:Character encoding Category:Computer-related introductions in 2026
Content Disclaimer
Informasi ini disarikan dari Wikipedia dan disajikan kembali untuk tujuan edukasi. Konten tersedia di bawah lisensi CC BY-SA 3.0. Kami tidak bertanggung jawab atas ketidakakuratan data yang bersumber dari kontribusi publik tersebut.
- The information displayed on this website is sourced in part or in whole from Wikipedia and has been adapted for the purpose of restating it. We strive to provide accurate and relevant information, however:
- There is no guarantee of absolute accuracy. Wikipedia is an open, collaborative project that can be edited by anyone, so information is subject to change.
- It is not intended to constitute professional advice. The content displayed is for informational and educational purposes only. For important decisions (e.g., medical, legal, or financial), please consult a professional.
- Content copyright. Wikipedia is licensed under the Creative Commons Attribution-ShareAlike License (CC BY-SA). This means that content may be reused with appropriate attribution and shared under a similar license.
- Responsible use. Any risk arising from the use of information from this website is entirely the responsibility of the user.

- provide significant coverage: discuss the subject in detail, not just brief mentions or routine announcements;
- are reliable: from reputable outlets with editorial oversight;
- are independent: not connected to the subject, such as interviews, press releases, the subject's own website, or sponsored content.
Please add references that meet all three of these criteria. If none exist, the subject is not yet suitable for Wikipedia.