D for C, C++ and Python developers

In D, writing s[0] after s = arr[1 .. 4] changes arr[1] as well, as a pointer into the array would in C, and int.max + 1 is −2,147,483,648 without a word, while the same lines in Python copy the list and give 2,147,483,648. D reads like C and C++ with much of Python’s convenience, and its surprises sit where it behaves like one of them in one place and like another elsewhere.

This lesson runs one small program per idea on LDC 1.43.0: slices and associative arrays, UTF-8 strings, the integer and rounding rows of the concept mapconcept map: A table of the questions on which languages differ in meaning, such as how whole numbers divide or how a half rounds, answered for each new language with a run of your own. More in “A one-week plan for learning any language”, lazy range pipelines, structs against classes with scope guards and exceptions, and the @safe attribute, which decides whether a release build still checks array bounds.

A slice is a view, and .dup makes a copy

A D array variable holds two things, a pointer to the first element and a length, and arr[1 .. 4] makes another such pair over the same memory: a sliceslice (D): A view of part of an array: a pointer to its first element and a length, written arr[1 .. 4]. Nothing is copied, so writing through the slice changes the array; .dup makes a copy with memory of its own.. Nothing is copied, so writing through either name changes both, as in C with a pointer and a length, and unlike Python’s arr[1:4], which copies. Step through the program and watch which memory each name points to:

Here is the program, with a second one about associative arrays, and the session that ran both:

import std.stdio;

void main()
{
    int[] arr = [1, 2, 3, 4, 5];
    int[] s = arr[1 .. 4];       // a view of arr[1], arr[2] and arr[3]
    s[0] = 20;
    writeln("arr = ", arr, "  s = ", s);

    int[] t = s.dup;             // a copy with memory of its own
    t[0] = 99;
    writeln("arr = ", arr, "  t = ", t);

    writeln("s.capacity = ", s.capacity, ", arr.capacity = ", arr.capacity);
    s ~= 40;                     // s does not end where arr's data ends: the runtime copies first
    s[0] = 7;
    writeln("arr = ", arr, "  s = ", s);

    int[] small = [1, 2, 3];
    writeln(small[5]);
}

import std.stdio;

void main()
{
    int[string] stock = ["pens": 12, "pads": 3];
    stock["clips"] = 40;
    writeln(stock.length, " keys; ", "pens" in stock ? "pens are in stock" : "no pens");
    if (auto count = "pads" in stock)    // in gives a pointer to the value, or null
        *count += 1;
    writeln("pads: ", stock["pads"], ", ink: ", stock.get("ink", 0));
    writeln(stock["ink"]);
}

from session import run

# Slices of one array, a copy, an append that copies, and an index past the end; then an associative array, a lookup
# with in and with get, and a key that is not there.
run("""
cd slices
ldc2 slices.d
!./slices
ldc2 stock.d
!./stock
""")

$ cd slices
$ ldc2 slices.d
$ ./slices
arr = [1, 20, 3, 4, 5]  s = [20, 3, 4]
arr = [1, 20, 3, 4, 5]  t = [99, 3, 4]
s.capacity = 0, arr.capacity = 7
arr = [1, 20, 3, 4, 5]  s = [7, 3, 4, 40]
core.exception.ArrayIndexError@slices.d(20): index [5] is out of bounds for array of length 3
[exit status 1]
$ ldc2 stock.d
$ ./stock
3 keys; pens are in stock
pads: 4, ink: 0
core.exception.RangeError@stock.d(11): Range violation
[exit status 1]

.dup gave t memory of its own, so t[0] = 99 left arr alone. The append is the subtle line. s.capacity was 0, because s ends where arr still has an element, so s ~= 40 could not grow in place without writing over arr[4]: the runtime copied s first, and from then on the two were separate arrays. Whether names still share memory after an append depends on a capacity that the code does not show, so take a .dup whenever you mean a copy.

Indexing is checked: small[5] stopped the program with core.exception.ArrayIndexError, naming the index, 5, and the length, 3, where C would read whatever lies there. Associative arrays, D’s built-in hash tables, follow the same rule. in gives a pointer to the value, or null, get gives a default, and reading a missing key, stock["ink"], threw a RangeError: the counterpart of Python’s KeyError, but an Error, which the section on classes comes back to.

A string is UTF-8, and length counts bytes

string is a slice too: immutable(char)[], an array of UTF-8 code unitscode unit: The unit in which an encoding stores text: one byte in UTF-8, two in UTF-16. ï takes two UTF-8 code units and 😀 four, and a D string's length counts UTF-8 code units. that cannot be changed through it. So length counts bytes, not letters, and the four strings of this program show where the three ways of counting part:

import std.stdio;
import std.range : walkLength;
import std.uni : byGrapheme;

void main()
{
    string city = "Pune";
    string word = "naïve";
    string face = "😀";
    string flag = "🇮🇳";
    writeln("length:     ", city.length, " ", word.length, " ", face.length, " ", flag.length);
    writeln("walkLength: ", city.walkLength, " ", word.walkLength, " ", face.walkLength, " ", flag.walkLength);
    writeln("graphemes:  ", city.byGrapheme.walkLength, " ", word.byGrapheme.walkLength, " ", face.byGrapheme.walkLength, " ", flag.byGrapheme.walkLength);
    writeln(typeof(word).stringof, ", ", typeof(word[0]).stringof);
    foreach (dchar c; word)
        write(c, '|');
    writeln();
    foreach (char c; word)
        write(cast(int) c, ' ');
    writeln();
    writeln(word[0 .. 3], " ", word[0 .. 4]);
    writeln("abc" == "abc", " ", "abc" < "abd", " ", "Zebra" < "apple");
}

from session import run

run("""
cd strings
ldc2 -run strings.d
""")

$ cd strings
$ ldc2 -run strings.d
length:     4 6 4 8
walkLength: 4 5 1 2
graphemes:  4 5 1 1
string, immutable(char)
n|a|ï|v|e|
110 97 195 175 118 101
na� naï
true true true

walkLength walks the string as a range, and Phobos’s range functions decode a string into code points, dchar, as foreach (dchar c; word) did when it printed the five letters of naïve. The flag is two code points that a reader sees as one character, a grapheme, which only byGrapheme counts. The foreach (char c; …) line shows the bytes: ï is 195 175. Indexing and slicing work on code units, so word[0 .. 3] cut ï in half and the terminal showed the leftover byte as �, while word[0 .. 4] kept it whole. Comparison goes by code unit as well: "Zebra" < "apple" is true, because capital letters come before small ones in UTF-8, as in ASCII.

Integers divide like C and wrap without a word

Here are the division, rounding and overflow rows of the concept map, answered for D. The D specification says that for integers the quotient rounds towards zero and the remainder takes the sign of the dividend, as in C, and that addition wraps: int.max + 1 == int.min. The program checks both, and a few neighbours:

import std.stdio;
import std.math : lrint, round;
import core.checkedint : adds;

void main()
{
    writeln("-7 / 2 = ", -7 / 2, ", -7 % 2 = ", -7 % 2, ", 7 / 2 = ", 7 / 2, ", 7.0 / 2 = ", 7.0 / 2);

    int big = int.max;
    big += 1;
    writeln("int.max + 1 = ", big);
    uint none = 0;
    none -= 1;
    writeln("0u - 1 = ", none);
    bool overflow;
    immutable checked = adds(int.max, 1, overflow);
    writeln("adds(int.max, 1) = ", checked, ", overflow: ", overflow);

    writeln("round(2.5) = ", round(2.5), ", round(-2.5) = ", round(-2.5), ", lrint(2.5) = ", lrint(2.5));
    writeln("0.1 + 0.2 = ", 0.1 + 0.2, ", == 0.3: ", 0.1 + 0.2 == 0.3);
    writefln("0.1 + 0.2 = %.17g", 0.1 + 0.2);

    byte b = 100;
    auto sum = b + b;
    writeln("byte + byte is ", typeof(sum).stringof, ": ", sum);
}

from session import run

run("""
cd numbers
ldc2 -run numbers.d
!ldc2 narrow.d
""")

$ cd numbers
$ ldc2 -run numbers.d
-7 / 2 = -3, -7 % 2 = -1, 7 / 2 = 3, 7.0 / 2 = 3.5
int.max + 1 = -2147483648
0u - 1 = 4294967295
adds(int.max, 1) = -2147483648, overflow: true
round(2.5) = 3, round(-2.5) = -3, lrint(2.5) = 2
0.1 + 0.2 = 0.3, == 0.3: false
0.1 + 0.2 = 0.30000000000000004
byte + byte is int: 200
$ ldc2 narrow.d
narrow.d(4): Error: cannot implicitly convert expression `total` of type `int` to `byte`
    byte small = total;
                 ^
[exit status 1]

The wrap is defined behaviour in D, so every compiler gives the same −2,147,483,648, while in C a signed overflow is undefined behaviour. When you need to know about it, core.checkedint tells you: adds returned the same wrapped sum and set its overflow flag to true, and an unsigned 0 − 1 is 4,294,967,295. round takes a half away from zero, as C’s round does, while lrint rounds in the current mode, to the even neighbour by default, so lrint(2.5) is 2, what Python’s round gives.

The last two lines are traps for a Python programmer. writeln prints a double with six significant digits, so 0.1 + 0.2 shows as 0.3 while comparing it with 0.3 gives false; %.17g shows all of it, 0.30000000000000004. And byte + byte is an int, which does not narrow to a byte by itself: byte small = total; was refused at compile time. Write cast(byte) total when you know the value fits, or to!byte(total), which throws when it does not.

Ranges and UFCS build lazy pipelines

A Phobos algorithm takes a rangerange (D): An object that hands out values one at a time through empty, front and popFront. Phobos's algorithms take and return ranges, and most of them are lazy: nothing is computed until a value is taken., something that hands out values one at a time, and most of them return another range without computing anything yet. With UFCSUFCS: Uniform Function Call Syntax: in D, a call f(a, b) may be written a.f(b), so that calls chain from left to right, as in iota(1, 11).filter!(x => x % 2 == 0)., f(a, b) can be written a.f(b), so a pipeline of calls reads from left to right:

import std.stdio;
import std.algorithm : filter, map, sum;
import std.array : array;
import std.range : iota, take;

int square(int x)
{
    writeln("  square(", x, ")");
    return x * x;
}

void main()
{
    auto evens = iota(1, 11).filter!(x => x % 2 == 0).map!(x => x * x);
    writeln(evens);
    writeln(iota(1, 101).sum, " ", sum(iota(1, 101)));

    writeln("building a pipeline over a billion numbers");
    auto squares = iota(1, 1_000_000_000).map!square;
    writeln("taking two");
    writeln(squares.take(2).array);
}

iota(1, 11).filter!(x => x % 2 == 0).map!(x => x * x) is three nested calls, and the ! passes each lambda as a template argument, so the compiler builds a filter and a map for those lambdas. Step through how writeln pulls the values out of it:

from session import run

run("""
cd ranges
ldc2 -run pipeline.d
ldc2 -run topwords.d
!ldc2 -c noempty.d
""")

$ cd ranges
$ ldc2 -run pipeline.d
[4, 16, 36, 64, 100]
5050 5050
building a pipeline over a billion numbers
taking two
  square(1)
  square(2)
[1, 4]
$ ldc2 -run topwords.d
cat 3; the 3; hat 2;
ek 3; do 2; teen 1;
café 3; naïve 1;
$ ldc2 -c noempty.d
noempty.d(7): Error: no property `empty` for type `string`, perhaps `import std.range;` is needed?
    foreach (word; "a cat's hat".splitter!(c => !c.isAlpha).filter!(w => !w.empty))
                                                                           ^
…/bin/../import/std/algorithm/iteration.d(1506):        instantiated from here: `FilterResult!(__lambda_L7_C69, SplitterResult!(__lambda_L7_C44, string))`
        return FilterResult!(unaryFun!predicate, Range)(range);
               ^
noempty.d(7):        instantiated from here: `filter!(SplitterResult!(__lambda_L7_C44, string))`
    foreach (word; "a cat's hat".splitter!(c => !c.isAlpha).filter!(w => !w.empty))
                                                           ^
[exit status 1]

The second half of pipeline.d shows the laziness: a map over a billion numbers was built at once, and take(2) made square run exactly twice. The same rule of UFCS makes iota(1, 101).sum and sum(iota(1, 101)) one call, 5,050 both times. topwords.d puts the pieces together:

import std.stdio;
import std.algorithm : filter, multiSort, splitter;
import std.array : array;
import std.range : empty, take;
import std.uni : isAlpha, toLower;

/// The k most frequent words of text. A word is a run of letters, compared in lower case; more frequent words come
/// first, and words with the same count in alphabetical order.
auto topWords(string text, size_t k)
{
    int[string] counts;
    foreach (word; text.splitter!(c => !c.isAlpha).filter!(w => !w.empty))
        counts[word.toLower]++;
    auto pairs = counts.byKeyValue.array;
    pairs.multiSort!("a.value > b.value", "a.key < b.key");
    return pairs.take(k);
}

void main()
{
    foreach (text; ["The cat and the hat. The CAT sat; a cat's hat!", "Ek, do, teen: ek do. Ek!", "Café, café, CAFÉ; naïve"])
    {
        foreach (p; topWords(text, 3))
            write(p.key, " ", p.value, "; ");
        writeln();
    }
}

splitter!(c => !c.isAlpha) cuts the text at every character that is not a letter, so cat's gives cat and s, and filter drops the empty pieces between two separators. The associative array counts, byKeyValue.array turns its pairs into an array, and multiSort orders them by count and then by word, which is why cat came before the at 3 each. Note the line import std.range : empty, take;: .empty on a string is a function of Phobos, not of the language, and noempty.d, the same pipeline without that import, failed with “no property empty for type string”, the compiler suggesting import std.range; itself.

Structs are values, classes are references

D has two kinds of user type, and each behaves like one of its neighbours. A struct is a value, as a C++ object is: assigning or passing it copies its fields. A class object lives on the garbage-collected heap, as in Java, and a class variable is a reference to it:

import std.stdio;

struct Point { int x, y; }

class Box
{
    int size;
    this(int size) { this.size = size; }
}

void scaled(Point p) { p.x *= 10; }
void grown(Box b) { b.size *= 10; }

void main()
{
    Point a = Point(1, 2);
    Point b = a;            // a copy of the struct
    b.x = 9;
    scaled(a);              // scaled changes its own copy
    writeln("a = ", a, ", b = ", b);

    Box p = new Box(1);
    Box q = p;              // a second reference to the same object
    q.size = 9;
    grown(p);
    writeln("p.size = ", p.size, ", q.size = ", q.size, ", p is q: ", p is q);
    writeln("Point(1, 2) == Point(1, 2): ", Point(1, 2) == Point(1, 2), "; new Box(1) == new Box(1): ", new Box(1) == new Box(1));

    {
        scope (exit) writeln("3. scope(exit): leaving the block");
        scope (success) writeln("2. scope(success): no exception");
        writeln("1. in the block");
    }
    try
    {
        scope (failure) writeln("scope(failure): an exception is leaving");
        throw new Exception("bad mark");
    }
    catch (Exception e)
        writeln("caught: ", e.msg);
}

from session import run

run("""
cd values
ldc2 -run values.d
ldc2 errors.d
!./errors
ldc2 nobox.d
!./nobox
""")

$ cd values
$ ldc2 -run values.d
a = Point(1, 2), b = Point(9, 2)
p.size = 90, q.size = 90, p is q: true
Point(1, 2) == Point(1, 2): true; new Box(1) == new Box(1): false
1. in the block
2. scope(success): no exception
3. scope(exit): leaving the block
scope(failure): an exception is leaving
caught: bad mark
$ ldc2 errors.d
$ ./errors
ConvException: Unexpected 'a' when converting from type string to type int
core.exception.ArrayIndexError@errors.d(13): index [3] is out of bounds for array of length 3
[exit status 1]
$ ldc2 nobox.d
$ ./nobox
none is null: true
[exit status 139]

scaled(a) changed only its own copy, so a stayed Point(1, 2), while grown(p) changed the one Box that p and q both name. Two separately made boxes were not ==, because the default opEquals of a class is identity, the same test as is. A class variable that was never assigned is null, and using it ended nobox with exit status 139, a crash on signal 11, not an exception that a handler could catch.

scope(exit), scope(success) and scope(failure) are scope guardsscope guard: A statement scope(exit), scope(success) or scope(failure) that runs when the enclosing block ends: always, only without an exception, or only while one leaves. Several run in reverse order, the last written first.: each runs its statement when the block ends, always, only without an exception, or only while one leaves. Several run in reverse order, the last written first, as the lines numbered 2 and 3 show, which lets the clean-up sit next to what it cleans.

errors.d shows the line between the two kinds of throwable. to!int("12a") threw a ConvException, which the program caught. marks[3] threw an ArrayIndexError, which catch (Exception e) does not catch: it is an ErrorError (D): A throwable for a failure a program is not expected to recover from, such as ArrayIndexError, RangeError or AssertError. catch (Exception e) does not catch it, and the documentation calls it not safe to catch., which the documentation of object calls unrecoverable and not safe to catch, so the program stopped. Catch exceptions, and fix the code that raises errors.

@safe keeps bounds checks in a release build

@safe marks a function whose memory safety the compiler checks: among other rules, it may not do pointer arithmetic or call a function that is @system, the default. Here a struct has a toString, and main is @safe:

import std.stdio;
import std.format : format;

struct Mark
{
    string name;
    int score;

    string toString() const
    {
        return format("%s: %d", name, score);
    }
}

void main() @safe
{
    auto m = Mark("Asha", 91);
    writeln(m);
}

from session import run

# @safe and a struct's toString; what --release does to bounds checks, read from ldc2 --help and then tried on one
# program built twice; a unittest block that fails, and the fix. The lines of the long compiler error that show the
# chain of calls are picked out with grep.
run("""
cd safety
ldc2 marks.d 2>&1 | grep -e 'Error' -e 'which calls'
ldc2 -run marks_fixed.d
ldc2 --help | grep -e '--release '
ldc2 bounds.d -of=bounds_debug
!./bounds_debug
!./bounds_debug safe
ldc2 --release bounds.d -of=bounds_release
./bounds_release
!./bounds_release safe
ldc2 -unittest -main grades.d -of=grades_test
!./grades_test
ldc2 -unittest -main grades_fixed.d -of=grades_fixed
./grades_fixed
""")

$ cd safety
$ ldc2 marks.d 2>&1 | grep -e 'Error' -e 'which calls'
marks.d(18): Error: `@safe` function `D main` cannot call `@system` function `std.stdio.writeln!(Mark).writeln`
…/bin/../import/std/stdio.d(1663):        which calls `write`
…/bin/../import/std/format/write.d(527):        which calls `formattedWrite`
…/bin/../import/std/format/write.d(1255):        which calls `formatValue`
…/bin/../import/std/format/internal/write.d(2522):        which calls `formatValueImpl`
…/bin/../import/std/format/internal/write.d(2157):        which calls `formatObject`
marks.d(9):        which calls `toString`
$ ldc2 -run marks_fixed.d
Asha: 91
$ ldc2 --help | grep -e '--release '
  --release                                    - Compile release version, defaulting to disabled asserts/contracts/invariants, and bounds checks in @safe functions only
$ ldc2 bounds.d -of=bounds_debug
$ ./bounds_debug
core.exception.ArrayIndexError@bounds.d(3): index [3] is out of bounds for array of length 3
[exit status 1]
$ ./bounds_debug safe
core.exception.ArrayIndexError@bounds.d(4): index [3] is out of bounds for array of length 3
[exit status 1]
$ ldc2 --release bounds.d -of=bounds_release
$ ./bounds_release
@system read: 61
$ ./bounds_release safe
core.exception.ArrayIndexError@bounds.d(4): index [3] is out of bounds for array of length 3
[exit status 1]
$ ldc2 -unittest -main grades.d -of=grades_test
$ ./grades_test
grades.d(13): [unittest] 75 is a B
1/1 modules FAILED unittests
[exit status 1]
$ ldc2 -unittest -main grades_fixed.d -of=grades_fixed
$ ./grades_fixed
1 modules passed unittests

The first command fails, and the reason is attribute inferenceattribute inference: The compiler working out attributes such as @safe, pure, nothrow and @nogc from a function's body. D does it for templates, function literals, auto functions and nested functions, and not for other functions.. writeln is a template, so the compiler works out its attributes from its body, and they would be @safe; but it calls Mark’s toString, an ordinary member function. The specification infers attributes for function literals, auto functions, nested functions and templates only, so toString counts as @system, and the message follows the chain of calls from writeln down to it. Marking toString as @safe fixed the build, and marks_fixed.d printed Asha: 91.

Why bother with @safe? Because of the next lines. ldc2 --help says that --release switches off asserts, contracts and invariants, and keeps bounds checks in @safe functions only. bounds.d reads one element past the end of a three-element slice, once through a @system function and once through a @safe one:

The 61 is the fourth element of the array the slice was cut from, read without a word. dub’s release build type is release mode, so the same holds for dub build --build=release.

unittest blocks test the code beside it. -unittest compiles them in and -main adds an empty main, so ldc2 -unittest -main grades.d makes a program that only runs the tests. Its first run reported grades.d(13): [unittest] 75 is a B, with the message of the failing assert; with >= in place of >, the second reported 1 modules passed unittests.

Practise: port topWords to Python

A second implementation is a good test of a first: the exercise asks for topWords of topwords.d in Python, giving the same pairs as the D program. Its sample tests are the three texts of the session, with the lists that topwords.d printed, and a few cases of its rules.

def top_words(text, k):
    """The k most frequent words of text, as topwords.d finds them."""
    counts = {}
    for word in text.split():
        counts[word] = counts.get(word, 0) + 1
    pairs = sorted(counts.items(), key=lambda pair: pair[1], reverse=True)
    return pairs[:k]

from topwords import top_words


def test_the_first_text_of_the_session():
    """topwords.d printed cat 3; the 3; hat 2;"""
    assert top_words("The cat and the hat. The CAT sat; a cat's hat!", 3) == [("cat", 3), ("the", 3), ("hat", 2)]


def test_words_in_another_language():
    """topwords.d printed ek 3; do 2; teen 1;"""
    assert top_words("Ek, do, teen: ek do. Ek!", 3) == [("ek", 3), ("do", 2), ("teen", 1)]


def test_letters_beyond_ascii():
    """topwords.d printed café 3; naïve 1; with only two words to give"""
    assert top_words("Café, café, CAFÉ; naïve", 3) == [("café", 3), ("naïve", 1)]


def test_ties_are_alphabetical():
    """four words once each, in alphabetical order"""
    assert top_words("to be or not", 4) == [("be", 1), ("not", 1), ("or", 1), ("to", 1)]


def test_digits_and_apostrophes_end_words():
    """r2d2 gives r and d; it's gives it and s"""
    assert top_words("r2d2 it's it", 3) == [("it", 2), ("d", 1), ("r", 1)]


def test_nothing_to_count():
    """an empty text, a text without letters, and k = 0"""
    assert top_words("", 3) == []
    assert top_words("42, 7; 108!", 3) == []
    assert top_words("the cat", 0) == []

Then the self-check, on the six ideas of this lesson. The idea you have answered least well comes up first, and each answer is explained when you pick it:

Interview questions

Warm-up (fresher to mid-level): In D, what does arr[1 .. 4] give you, and when does it stop sharing memory with arr?

It gives a slice: a pointer into arr’s memory at element 1 and a length of 3, with nothing copied, so writing s[0] = 20 changes arr[1] too. The two share memory until one of them is given new memory. .dup makes a copy at once. An append can do it too: when the slice cannot grow in place, because its capacity is 0 or full, the runtime copies it before appending, and from then on writes through the slice no longer reach arr. In the lesson, s.capacity was 0, since arr still had an element after the slice, so s ~= 40 moved s to new memory and s[0] = 7 left arr as [1, 20, 3, 4, 5]. Because the capacity is not visible in the code, use .dup whenever you mean a copy.

Two longer questions, on an append that sometimes changes the caller’s data and on a release build that reads past an array:

Key takeaways

• A slice shares its array’s memory, so writes through it change the array; .dup copies, and an append copies only when the slice cannot grow in place, which the code does not show.
• A string is an array of UTF-8 code units: length counts bytes, walkLength code points and byGrapheme the characters a reader sees, and slicing can cut a character in half.
• Integer division truncates and overflow wraps, by the specification; round goes half away from zero, lrint to even, and writeln prints a double with six significant digits.
• Phobos pipelines are lazy ranges chained with UFCS: nothing runs until a value is taken, and take(2) did the work of two values.
• Structs are copied and classes are shared references; scope guards run in reverse order when a block ends, and an Error such as ArrayIndexError is not caught by catch (Exception e).
• A release build keeps bounds checks only in @safe functions, so mark indexing code @safe, and run unittest blocks with -unittest or dub test.