Draft:UCE-8

UCE-8
Alias(es)Unicode Compact Encoding
StandardNone (experimental)
Current statusExperimental; not adopted by any standards body
ClassificationUnicode Transformation Format-like, variable-width
ExtendsASCII
Transforms / EncodesUnicode

UCE-8 (Unicode Compact Encoding) is an experimental variable-width character encoding for Unicode that represents each code point in one, two or three bytes. It was designed to reduce the storage and transmission cost of scripts that UTF-8 encodes in three bytes, principally the writing systems of South Asia, Southeast Asia, East Asia and the Horn of Africa.

Unlike UTF-8, in which the leading byte identifies both the length and the general range of a character, UCE-8 places the identifying information in the final byte of a sequence. The high bit of each byte acts solely as a continuation flag, and the terminating byte doubles as an index into one of 68 character "pages".

Design

UCE-8 assigns byte lengths by script rather than by position in the Unicode code space. The difference from UTF-8 falls entirely on the third tier, which UCE-8 promotes to two bytes and UTF-8 does not:

Bytes per character, by category of script
Category UTF-8 UCE-8
ASCII text 1 byte 1 byte
European and Middle Eastern scripts 2 bytes 2 bytes
Asian and African scripts 3 bytes 2 bytes
Historic scripts and other supplementary-plane characters 4 bytes 3 bytes

The saving applies only to the characters the two-byte tier actually holds; the scripts concerned, and how completely each is covered, are listed under Page map below.

Byte structure

The most significant bit of every byte indicates whether another byte follows. A byte with the high bit set ("lead byte") is followed by at least one more byte; a byte with the high bit clear ("trail byte") terminates the character. The encoding is therefore self-terminating and requires no length field.

Tier Byte pattern Length Characters reachable Range covered
1 0xxxxxxx 1 byte 128 ASCII
2 1xxxxxxx 0xxxxxxx 2 bytes 8,704 68 pages of 128 characters
3 1xxxxxxx 1xxxxxxx 0xxxxxxx 3 bytes 1,114,112 All remaining code points

The capacity of the three-byte tier, 128 × 128 × 68, equals 1,114,112, which is also the total size of the Unicode code space (17 × 65,536).

Addressing

In a two-byte sequence the trail byte selects a page of 128 characters and the lead byte selects a position, or "slot", within it. This is the reverse of UTF-8, where the leading byte carries the identifying information.

For example, the Devanagari letter na (U+0928) is encoded as the byte pair A8 35:

  • A8 = 1 0101000 — high bit set, slot 28 hex
  • 35 = 0 0110101 — high bit clear, page index 35 hex

Page 35 hex corresponds to the Unicode block beginning at U+0900, so the sequence resolves to U+0900 + 28 hex = U+0928. Holding the lead byte constant and changing only the trail byte selects the same slot in a different script.

Restricted trail bytes

Of the 128 possible trail-byte values, 60 are excluded: the 33 control codes, the space character, and 26 punctuation marks that have structural meaning in common data formats such as JSON, CSV, URLs, file paths and shell syntax. Only six punctuation marks remain usable as trail bytes: !, (, ), ^, _ and ~.

This leaves 68 usable page indices, giving the two-byte tier a capacity of 128 × 68 = 8,704 characters.

Page map

The following table lists every trail-byte value. The row gives the high nibble and the column the low nibble; the small character in each cell is the ASCII character that byte represents, and the value beneath it is the page selected. WLD, CHN and KOR mark indexed pages, which hold a curated list rather than an aligned Unicode block.

Trail-byte values and the pages they select
0 1 2 3 4 5 6 7 8 9 A B C D E F
0 The 32 control codes occupy rows 0 and 1 and can never be a page index.
1
2 SP !
008
" # $ % & ' (
010
)
038
* + , - . /
3 0
040
1
050
2
058
3
060
4
068
5
090
6
098
7
0A0
8
0A8
9
0B0
: ; < = > ?
4 @ A
0C0
B
0C8
C
0D0
D
0D8
E
0E0
F
0E8
G
0F0
H
100
I
120
J
128
K
130
L
178
M
1E8
N
300
O
308
5 P
WLD
Q
WLD
R
4E0
S
4E8
T
540
U
5B8
V
628
W
7E8
X
8B8
Y
8D0
Z
CHN
[ \ ] ^
CHN
_
CHN
6 ` a
CHN
b
CHN
c
CHN
d
CHN
e
CHN
f
CHN
g
CHN
h
CHN
i
CHN
j
CHN
k
CHN
l
CHN
m
CHN
n
CHN
o
CHN
7 p
CHN
q
CHN
r
CHN
s
CHN
t
CHN
u
CHN
v
KOR
w
KOR
x
KOR
y
KOR
z
KOR
{ | } ~
KOR
DEL
Legend
Colour Category Pages
  World scripts 30
  Chinese 32
  Korean 6
  Excluded 60

Devanagari occupies row 3, column 5 — the ASCII digit 5 — which is why text in that script produces byte streams containing that digit at every second position.

The page values in the table above are Unicode block addresses with the final hexadecimal digit removed; page 090 therefore denotes the 128 code points beginning at U+0900. The scripts they correspond to are:

Scripts reachable in two bytes, and the pages that hold them
Page Script Unicode block Speakers (millions) Coverage
090 Devanagari U+0900–U+097F 600 Complete block
098 Bengali U+0980–U+09FF 284 Complete block
0A0 Gurmukhi U+0A00–U+0A7F 30 Complete block
0A8 Gujarati U+0A80–U+0AFF 57 Complete block
0B0 Odia U+0B00–U+0B7F 41 Complete block
0B8 Tamil U+0B80–U+0BFF 78 Indexed: the 61 assigned characters, the rest of the block being reserved
0C0 Telugu U+0C00–U+0C7F 83 Complete block
0C8 Kannada U+0C80–U+0CFF 43 Complete block
0D0 Malayalam U+0D00–U+0D7F 45 Complete block
0D8 Sinhala U+0D80–U+0DFF 17 Complete block
0E0 Thai U+0E00–U+0E7F 71 Complete block
0E8 Lao U+0E80–U+0EFF 7.7 Complete block
0F0 Tibetan U+0F00–U+0FFF 6 First half of the block
100 Myanmar U+1000–U+109F 33 First 128 code points
108 Georgian U+10A0–U+10FF 3.6 Indexed, on a shared world page
120 Ethiopic U+1200–U+137F 135 Three pages
178 Khmer U+1780–U+17FF 18 Complete block
180 Mongolian U+1800–U+18AF 10 Indexed: 32 traditional-script characters. Mongolia writes in Cyrillic, which is unaffected
1E8 Vietnamese U+1E00–U+1EFF 86 Shared with Yoruba, Igbo and Welsh; most Vietnamese text is already ASCII
300 Japanese U+3040–U+30FF 122 Two pages: kana complete, kanji only where they coincide with the Chinese index
4E0 Chinese U+4E00–U+9FFF 1,400 32 pages, 4,096 most frequent characters; simplified only
AC0 Korean U+AC00–U+D7A3 78 Indexed: 768 most frequent syllables of 11,172

Population figures are approximate and are given for the writing system rather than for any one language. The scripts listed all fall in the range UTF-8 encodes in three bytes.

Page allocation

Category Pages Slots Contents
Alphabetic world scripts 30 3,840 Latin, Greek, Cyrillic, Armenian, Hebrew, Arabic, the Indic scripts, Thai, Lao, Tibetan, Myanmar, Georgian, Ethiopic, Khmer, Vietnamese, Kana, Mongolian, currency signs
Chinese 32 4,096 Most frequent Han characters
Korean 6 768 Most frequent Hangul syllables
Total 68 8,704

Pages are of two kinds. Block pages map a 128-code-point-aligned slice of Unicode, so the slot is computed arithmetically. Indexed pages hold a curated list of 128 entries, so the slot requires a table lookup. Chinese and Korean use indexed pages because Unicode orders Han characters by radical and Hangul syllables by jamo composition, neither of which correlates with frequency of use.

Comparison with other encodings

UCE-8 is not the first encoding to represent East Asian scripts in two bytes. GB 18030, Shift JIS, EUC-KR and Big5 all predate it and encode their respective scripts in two bytes, but each covers a single language community and they are mutually incompatible. UTF-16 encodes the entire Basic Multilingual Plane in two bytes but requires two bytes for ASCII as well.

Bytes per character
Script UTF-8 UTF-16 GB 18030 UCE-8
ASCII 1 2 1 1
Cyrillic, Greek 2 2 2 2
Hebrew, Arabic 2 2 4 2
Indic, Thai, Ethiopic 3 2 4 2
Chinese, Japanese kana 3 2 2 2
Korean 3 2 4 2
Supplementary planes 4 4 4 3

Unicode also defines two compression schemes with related goals, the Standard Compression Scheme for Unicode (UTS #6) and BOCU-1 (UTS #12), neither of which is widely deployed.

Limitations

  • Incompatibility. A UCE-8 byte stream is not valid UTF-8, and vice versa, except for pure ASCII, which is identical in both.
  • Byte order does not follow code-point order. Because indexed pages are ordered by frequency rather than by code point, sorting encoded bytes does not sort characters, affecting binary search and range queries over raw encoded data.
  • Context-dependent bytes. A trail byte cannot be distinguished from a standalone ASCII character by inspection alone; determining which requires examining the preceding byte. UTF-8, by contrast, marks continuation bytes with a reserved 10xxxxxx prefix. The designer describes this as a deliberate trade: reserving such a prefix would reduce the two-byte tier from 8,704 characters to 4,096.
  • Substring search. Because trail bytes are ordinary ASCII digits and letters, a naive byte-level substring search can report false positives. These can be eliminated by testing the byte preceding each candidate match, described under Character boundaries below. The test applies only to code that performs the search, so tools that are unaware of the encoding — grep, SQL LIKE, or a general-purpose search index — remain affected.
  • Fixed repertoire. The curated character tables cannot be reordered without invalidating previously encoded data.
  • Interaction with compression. Much of the size advantage over UTF-8 is reduced when general-purpose compression such as gzip or Brotli is applied, since UTF-8's repeated lead bytes are themselves highly compressible.

Implementation

A reference implementation in Python, together with the frozen character tables and a browser-based demonstration, is published by the designer under a public repository.[1]

Character tables

The 32 indexed pages are backed by three ordered strings, one per category. A character's slot is its position in the string, so the tables are the encoding: reordering them would change the meaning of every byte sequence already written. They are consequently frozen.

The frozen tables, with the opening characters of each
Table Pages Characters Contents
CHINESE_CHARS 24 3,072 Han characters by frequency, excluding those the eight Chinese block pages already reach
十厂卜八入儿匕几刁刀力又干工土士才大小口山巾千川夕勺凡及 …
KOREAN_CHARS 6 768 Hangul syllables ordered by the frequency of their component jamo
아이가으나기사하다오니그시히디안어우느자마스고흐드알인바 …
WORLD_CHARS 2 256 Cyrillic extensions, Georgian, Mongolian, Tamil, currency signs and additional Latin letters
ӘәҒғҚқҢңӨөҰұҮүҺһҖҗҘҙҠҡҪҫҜҝҸҹ …
Total 32 4,096

Each table fills its pages exactly, 32 × 128 = 4,096 entries with none unused. The complete tables are too long to reproduce here; they appear in full in the reference implementation.[1]

Encoding

The encoder tries the block pages first, since their slot is arithmetic, then the three indexed tables, and finally falls back to the three-byte tier. LEAD_BASE is 0x80, the bit that marks a byte as non-terminating, and TAIL_COUNT is 68, the number of usable trail values.

def encode(code_point):
    """Encode a Unicode code point into 1, 2 or 3 bytes."""

    # TIER 1 - plain ASCII passes straight through
    if code_point < 0x80:
        return bytes([code_point])

    # TIER 2a - block pages: an aligned 128-code-point slice of Unicode
    trail = PAGE_TO_TRAIL.get((code_point >> 7) << 3)
    if trail is not None:
        return bytes([LEAD_BASE | (code_point & 0x7F), trail])

    # TIER 2b-d - indexed pages: the slot comes from a curated table
    for table, trails in ((CHINESE_SLOT, CHINESE_TRAILS),
                          (KOREAN_SLOT, KOREAN_TRAILS),
                          (WORLD_SLOT, WORLD_TRAILS)):
        slot = table.get(code_point)
        if slot is not None:
            return bytes([LEAD_BASE | (slot & 0x7F), trails[slot >> 7]])

    # TIER 3 - everything else, spread across the 68 legal terminators
    index = code_point - 0x80
    rest, tail = divmod(index, TAIL_COUNT)

    return bytes([
        LEAD_BASE | (rest >> 7),
        LEAD_BASE | (rest & 0x7F),
        TAIL_BYTES[tail],
    ])

The third tier has no page table left to consult, so it distributes the remaining code points arithmetically: the remainder modulo 68 selects one of the legal terminating bytes and the quotient becomes the two lead bytes. This is why the tier's capacity is exactly 128 × 128 × 68, and why the encoder cannot emit an illegal terminator even in its fallback path.

Decoding

The decoder reads the high bit of the second byte to choose the tier, then uses the trail byte as the page index.

def decode(data):
    """Decode a 1, 2 or 3 byte sequence back into a Unicode code point."""

    first = data[0]
    if first < 0x80:
        return first

    second = data[1]

    # TIER 2 - the second byte clears the high bit, so it terminates
    if second < 0x80:
        page = TRAIL_TO_PAGE[second]

        if page is WORLD:
            return WORLD_CODE_POINTS[(WORLD_PAGE[second] << 7) | (first & 0x7F)]
        if page is CHINESE:
            return CHINESE_CODE_POINTS[(CHINESE_PAGE[second] << 7) | (first & 0x7F)]
        if page is KOREAN:
            return KOREAN_CODE_POINTS[(KOREAN_PAGE[second] << 7) | (first & 0x7F)]

        return ((page >> 3) << 7) | (first & 0x7F)

    # TIER 3 - the second byte keeps the high bit, so a third follows
    tail = TAIL_INDEX[data[2]]
    rest = ((first & 0x7F) << 7) | (second & 0x7F)

    return 0x80 + rest * TAIL_COUNT + tail

Character boundaries

Because a trail byte is indistinguishable from a standalone ASCII character in isolation, a byte-level substring search over a UCE-8 stream can match the interior of a multi-byte character. The effect is pronounced for scripts whose page index happens to be a common ASCII character: Devanagari's page index is 0x35, the digit 5, so Devanagari text contains that byte at every second position. Searching such a stream for the digit 5 returns one match per character in addition to any genuine digit.

The ambiguity is resolved by a single test on the preceding byte. A low byte always terminates a character, so a position begins a character precisely when the byte before it is also low, or when it is the start of the stream:

def is_char_start(data, i):
    """A low byte terminates a character, so position i begins one."""
    return i == 0 or not (data[i - 1] & 0x80)

Only the first byte of a candidate match needs this test. If that byte begins a character and is itself low, it also terminates one, so the following byte begins a character in turn, and the property carries along the whole search term. The same test governs where a stream may be split: a cut is safe unless the byte before it is a lead byte.

See also

References

  1. ^ a b "UCE-8 reference implementation". GitLab. Retrieved 2026-08-11.

Category:Unicode Transformation Formats Category:Character encoding Category:Computer-related introductions in 2026

Content Disclaimer

Informasi ini disarikan dari Wikipedia dan disajikan kembali untuk tujuan edukasi. Konten tersedia di bawah lisensi CC BY-SA 3.0. Kami tidak bertanggung jawab atas ketidakakuratan data yang bersumber dari kontribusi publik tersebut.

  1. The information displayed on this website is sourced in part or in whole from Wikipedia and has been adapted for the purpose of restating it. We strive to provide accurate and relevant information, however:
  2. There is no guarantee of absolute accuracy. Wikipedia is an open, collaborative project that can be edited by anyone, so information is subject to change.
  3. It is not intended to constitute professional advice. The content displayed is for informational and educational purposes only. For important decisions (e.g., medical, legal, or financial), please consult a professional.
  4. Content copyright. Wikipedia is licensed under the Creative Commons Attribution-ShareAlike License (CC BY-SA). This means that content may be reused with appropriate attribution and shared under a similar license.
  5. Responsible use. Any risk arising from the use of information from this website is entirely the responsibility of the user.