UTF-8, national data and names in Indian scripts

The name अनीता देशमुख is 12 characters and 34 bytes in UTF-8, so a PIC X(30) field that looks roomy cuts it, and a PIC X(10) field cuts the letter त in half. COBOL’s PIC X counts bytes, and names in Indian scripts take about three bytes a character.

This lesson measures names in six scripts, shows what MOVE, UPPER-CASE and REVERSE do to UTF-8 text in GnuCOBOL, and names what national data and IBM’s UTF-8 items count instead.

A PIC X field counts bytes, and UTF-8 needs up to four for a character

In UTF-8UTF-8: A Unicode encoding that stores each code point in one to four bytes: one for ASCII characters, three for most characters of the Indian scripts. A byte from X'80' to X'BF' only continues a character. a character takes one to four bytes; IBM’s Programming Guide says its encoding “is always between one and four bytes”. The Latin letters of ASCII take one, and a Devanagari letter takes three. A PIC X field, and every statement that works on it, counts bytes. names.cob puts the name in byte-based fields and calls three text functions on it, and valuecut.cob gives the name, as a VALUE, to a field of 30 bytes:

       IDENTIFICATION DIVISION.
       PROGRAM-ID. NAMES.
      * A name in Devanagari in byte-based PIC X fields.
       DATA DIVISION.
       WORKING-STORAGE SECTION.
       01  NAME-40                 PIC X(40).
       01  NAME-10                 PIC X(10).
       01  WS-BYTES                PIC 99.
       01  NAME-REV                PIC X(10).
       PROCEDURE DIVISION.
           MOVE "अनीता देशमुख" TO NAME-40
           MOVE FUNCTION LENGTH (FUNCTION TRIM (NAME-40)) TO WS-BYTES
           DISPLAY "bytes used:   " WS-BYTES
           DISPLAY "as written:   [" FUNCTION TRIM (NAME-40) "]"
           MOVE NAME-40 TO NAME-10
           DISPLAY "into X(10):   [" NAME-10 "]"
           DISPLAY "its bytes:    " FUNCTION HEX-OF (NAME-10)
           DISPLAY "UPPER-CASE:   ["
               FUNCTION UPPER-CASE (FUNCTION TRIM (NAME-40)) "]"
           MOVE FUNCTION REVERSE (NAME-10) TO NAME-REV
           DISPLAY "REVERSE:      " FUNCTION HEX-OF (NAME-REV)
           STOP RUN.

from session import files, run

# A name in Devanagari in byte-based PIC X fields, then as the VALUE of a field four bytes too short.
# -A passes an option to the C compiler: it quiets clang's warning about the UTF-8 bytes of the literals.
files("names.cob", "valuecut.cob")
run("""
cobc -x -A -Wno-invalid-source-encoding names.cob && ./names
cobc -x -A -Wno-invalid-source-encoding valuecut.cob && ./valuecut | xxd
""")

$ cobc -x -A -Wno-invalid-source-encoding names.cob && ./names
bytes used:   34
as written:   [अनीता देशमुख]
into X(10):   [अनी�]
its bytes:    E0A485E0A4A8E0A580E0
UPPER-CASE:   [अनीता देशमुख]
REVERSE:      E080A5E0A8A4E085A4E0
$ cobc -x -A -Wno-invalid-source-encoding valuecut.cob && ./valuecut | xxd
valuecut.cob:7: warning: value size exceeds data size [-Wothers]
    5 |        WORKING-STORAGE SECTION.
    6 |        01  NAME-30                 PIC X(30)
    7 >            VALUE "अनीता देशमुख".
    8 |        PROCEDURE DIVISION.
    9 |            DISPLAY NAME-30
valuecut.cob:7: note: value size is 34 [-Wothers]
00000000: e0a4 85e0 a4a8 e0a5 80e0 a4a4 e0a4 be20  ...............
00000010: e0a4 a6e0 a587 e0a4 b6e0 a4ae e0a5 0a    ...............

The name took 34 bytes. The MOVE into PIC X(10) stopped after byte 10, which is the first of the three bytes of त, so the field ends with a lone E0 that DISPLAY shows as a replacement mark. UPPER-CASE left the name unchanged, since Devanagari has no capitals, and REVERSE reversed bytes, not characters, so its result is not UTF-8 at all. The VALUE clause compiled with only a warning, “value size exceeds data size”, and a note that the value is 34 bytes, and its xxd listing ends with the bytes e0 a5 before DISPLAY’s newline: the vowel sign ु was cut in two at byte 30. Step through the MOVE:

Names in Indian scripts take three bytes a character, and national data counts characters

name_sizes.py measures seven invented names in six scripts, in code pointscode point: The number Unicode gives a character, such as U+0905 for the Devanagari letter अ. What a reader sees as one letter may be several code points, such as a consonant with a vowel sign., in UTF-8 bytes, which a PIC X field holds, and in UTF-16UTF-16: A Unicode encoding that stores most characters, those of the Indian scripts included, in two bytes. IBM's Enterprise COBOL keeps national data in UTF-16, CCSID 1200. bytes, which national data holds:

"""How big a name is in characters (Unicode code points), in UTF-8 bytes, which is what a PIC X field counts, and in
UTF-16 bytes, which is what national data holds. The names are invented; add your own to NAMES and run it again."""

NAMES = [
    "Anita Deshmukh",
    "José Fernandes",
    "अनीता देशमुख",
    "கவிதா ராமன்",
    "সুমিত ঘোষ",
    "ਹਰਪ੍ਰੀਤ ਕੌਰ",
    "లక్ష్మి రెడ్డి",
]

print(f"{'code points':>12} {'UTF-8 bytes':>12} {'UTF-16 bytes':>13}  name")
for name in NAMES:
    print(f"{len(name):12} {len(name.encode('utf-8')):12} {len(name.encode('utf-16-be')):13}  {name}")

widest = max(len(name.encode("utf-8")) for name in NAMES)
print(f"The longest name here needs PIC X({widest}) in UTF-8.")

 code points  UTF-8 bytes  UTF-16 bytes  name
          14           14            28  Anita Deshmukh
          14           15            28  José Fernandes
          12           34            24  अनीता देशमुख
          11           31            22  கவிதா ராமன்
           9           25            18  সুমিত ঘোষ
          11           31            22  ਹਰਪ੍ਰੀਤ ਕੌਰ
          14           40            28  లక్ష్మి రెడ్డి
The longest name here needs PIC X(40) in UTF-8.

Names in Indian scripts took about three UTF-8 bytes a code point: the Telugu name has 14 code points and needs 40 bytes, while the Latin name of 14 takes 14, and José takes 15 for its é. In UTF-16 every one of these names takes two bytes a code point, so names in Indian scripts are smaller in UTF-16 than in UTF-8, and Latin names the other way round. A vowel sign or a virama is a code point of its own, so a name of five visible letters may be more code points than that.

Add a name of your own to NAMES and run the program again to see how many bytes its field needs.

COBOL has a second way to hold text. National datanational data: COBOL data items with USAGE NATIONAL and PICTURE N, which Enterprise COBOL holds as UTF-16, so that a field counts characters rather than the bytes of UTF-8., USAGE NATIONAL with PICTURE N, is UTF-16 in Enterprise COBOL: “Define national data items with the USAGE NATIONAL clause to hold national (UTF-16) character strings.” Enterprise COBOL 6.5 also has UTF-8 data items, PICTURE U, which count code points rather than bytes. In GnuCOBOL 3.2, NEWS lists only “parsing support” for USAGE UTF-8, so a program there keeps UTF-8 text in PIC X and must do the counting itself.

Practise: bytes, characters and fields

These questions use the name, the runs and the table of sizes.

Sizing and validating UTF-8 fields: a safe cut, a validity check and a conjunct kept whole Pro: opens with a Pro, Premium or Ultimate pass

National and UTF-8 data in Enterprise COBOL 6.5 and GnuCOBOL 3.2, compared Pro: opens with a Pro, Premium or Ultimate pass

Interview questions

A warm-up with its answer, then three questions about names that did not fit:

Key takeaways

• PIC X fields and the statements on them count bytes, so a MOVE, a VALUE or REVERSE can cut or scramble UTF-8 text, and a Devanagari name of 12 characters needs 34 bytes.
• Names in Indian scripts take about three UTF-8 bytes a code point, and two in UTF-16, so national data holds them in fewer bytes than UTF-8, and Latin names the other way round.
• Enterprise COBOL 6.5 has national data in UTF-16 and UTF-8 items that count code points, while GnuCOBOL 3.2 only parses USAGE UTF-8, so portable programs keep UTF-8 in PIC X and do the counting themselves.