Crystal macros, fibers and C bindings

A macro of seven lines in units.cr writes three conversion methods while the program compiles, and crystal tool expand prints the code it wrote. Three fibers on one thread send three squares to the main fiber, and two of them never print their last line: concurrency in Crystal follows rules you can predict from the run.

This lesson covers the three things Crystal adds to a Ruby developer’s toolbox: macros that write code before the program runs, fibers and channels that share one thread, and lib blocks that call C functions directly, with the strings and the callbacks that C cannot take.

Macros write code while the program compiles

units.cr needs one conversion method per unit. Instead of typing three methods, it calls a macromacro (Crystal): Code that runs inside the compiler and writes Crystal code where it is called. Its arguments arrive as pieces of the program, {% %} runs loops and conditions, and {{ }} pastes values into the code it writes., define_units, which writes them:

# A macro that writes one conversion method per unit while the program compiles.
macro define_units(base, **factors)
  {% for unit, factor in factors %}
    def self.{{unit.id}}_to_{{base.id}}(value : Float64) : Float64
      value * {{factor}}
    end
  {% end %}
end

module Length
  define_units metre, inch: 0.0254, foot: 0.3048, yard: 0.9144
end

puts Length.inch_to_metre(12.0)
puts Length.foot_to_metre(10.0)
puts Length.yard_to_metre(100.0)
p {{ Length.class.methods.map(&.name.stringify).sort }}

A macro runs inside the compiler. Its arguments arrive as pieces of the program, not as values: base is the name metre, and factors holds three named number literals. Code between {% and %} runs while compiling, so the loop runs three times, and {{ }} pastes a value into the code the macro writes. Step through the call:

from session import run

# define_units writes three methods while units.cr compiles; tool expand shows the code it wrote; a name it did not
# write is a compile error.
run("""
cd macros
crystal run units.cr
crystal tool expand -c units.cr:11:3 units.cr
crystal run accessors.cr
!crystal run typo.cr
""")

$ cd macros
$ crystal run units.cr
0.30479999999999996
3.048
91.44
["foot_to_metre", "inch_to_metre", "yard_to_metre"]
$ crystal tool expand -c units.cr:11:3 units.cr
1 expansion found
expansion 1:
   define_units(metre, inch: 0.0254, foot: 0.3048, yard: 0.9144)

# expand macro 'define_units' (units.cr:2:1)
~> def self.inch_to_metre(value : Float64) : Float64
     value * 0.0254
   end

   def self.foot_to_metre(value : Float64) : Float64
     value * 0.3048
   end

   def self.yard_to_metre(value : Float64) : Float64
     value * 0.9144
   end
$ crystal run accessors.cr
C7: 3 guests
["guests", "guests=", "initialize", "seat"]
$ crystal run typo.cr
Showing last frame. Use --error-trace for full trace.

In typo.cr:14:13

 14 | puts Length.feet_to_metre(10.0)
                  ^------------
Error: undefined method 'feet_to_metre' for Length:Module

Did you mean 'foot_to_metre'?
[exit status 1]

crystal tool expand -c units.cr:11:3 names the line and column of the call and printed the three methods the macro wrote, exactly as if they had been typed. The last line of units.cr runs macro code too: {{ Length.class.methods.map(&.name.stringify).sort }} listed the class methods of Length while compiling, and the program printed that list. Ruby developers know the reflection the guide for Rubyists says Crystal leaves out, such as methods; this is where Crystal does it, at compile time.

You have used macros already. property and getter in accessors.cr are macros of the standard library: the compiler’s list of Booking’s methods shows guests and guests= for property guests, and seat alone for getter seat. And a name that no macro wrote does not exist: typo.cr calls feet_to_metre, the compiler stops with undefined method 'feet_to_metre', and it suggests the method the macro did write, foot_to_metre.

Fibers take turns on one thread, and channels hand values over

spawn starts a fiberfiber: A lightweight unit of execution that Crystal's scheduler runs, started with spawn. Fibers take turns on a thread: one runs until it waits on a channel, input, output or a sleep, or until it ends., a lightweight unit of execution that Crystal’s scheduler runs. The concurrency guide states the rule that explains every output in this section: a program runs one fiber at a time by default, and fibers are cooperative, so a fiber keeps the thread until it waits, on a channel, input, output or a sleep, or ends. round_trip.cr sends three squares through a channelchannel (Crystal): A typed queue that fibers send values through. Channel(Int32).new has no room, so send waits until a fiber receives; Channel(Int32).new(3) has room for three values, so send goes on until it is full. with no room in it, Channel(Int32).new:

# Three fibers each send one square through an unbuffered channel; the main fiber receives three times.
squares = Channel(Int32).new

3.times do |i|
  spawn do
    puts "fiber #{i} sends #{i * i}"
    squares.send(i * i)
    puts "fiber #{i} is done"
  end
end

puts "main waits"
got = Array.new(3) { squares.receive }
puts "main got #{got}"

from session import run

# Three fibers send one square each to the main fiber, through a channel without room and through one with room for
# three: the order of the lines shows when each fiber runs and when it waits.
run("""
cd fibers
crystal run round_trip.cr
crystal run round_trip_buffered.cr
""")

$ cd fibers
$ crystal run round_trip.cr
main waits
fiber 0 sends 0
fiber 0 is done
fiber 1 sends 1
fiber 2 sends 4
main got [0, 1, 4]
$ crystal run round_trip_buffered.cr
main waits
fiber 0 sends 0
fiber 0 is done
fiber 1 sends 1
fiber 1 is done
fiber 2 sends 4
fiber 2 is done
main got [0, 1, 4]

The first run printed fiber 0 is done and never the same line for fibers 1 and 2. Step through it in the order of its lines:

A send on a channel without room waits until a fiber receives. Fiber 0 found main already waiting and went on; fibers 1 and 2 found nobody and waited. By the time main had taken their values it was the last thing to run, and a Crystal program ends when its main fiber does, whatever other fibers still have to do.

The second run made one change, Channel(Int32).new(3), a channel with room for three values. Now no send had to wait, so each fiber printed both of its lines before the next one started, and main found the 1 and the 4 waiting in the channel. The values were the same, [0, 1, 4]; the order of everything else changed with the capacity.

A pool of fibers shares the work, and select takes the first answer

pool.cr checks the status of six hosts with three worker fibers. The statuses come from a table in the program, so the run needs no network; a real checker would make a request where the table is read, and each wait for an answer would let another fiber run:

# A status checker: three worker fibers take hosts from one channel and send results to another.
# The statuses come from a table in the program, so the run needs no network.
STATUS = {
  "api.local"     => 200,
  "billing.local" => 503,
  "cdn.local"     => 200,
  "docs.local"    => 404,
  "mail.local"    => 200,
}

record Result, index : Int32, host : String, status : Int32?, worker : Int32

hosts = ["api.local", "billing.local", "cdn.local", "docs.local", "mail.local", "wiki.local"]
jobs = Channel({Int32, String}).new
results = Channel(Result).new

3.times do |worker|
  spawn do
    while job = jobs.receive?
      index, host = job
      results.send(Result.new(index, host, STATUS[host]?, worker))
    end
  end
end

spawn do
  hosts.each_with_index { |host, index| jobs.send({index, host}) }
  jobs.close
end

arrived = Array.new(hosts.size) { results.receive }
arrived.each { |r| puts "worker #{r.worker}: #{r.host}" }
puts "-- in input order --"
arrived.sort_by(&.index).each do |r|
  puts "#{r.host}: #{r.status || "unknown host"}"
end

The sender fiber closes jobs after the last host. receive? gives nil once a channel is closed and empty, which ends each worker’s while loop, and the record Result carries the index of its host:

from session import run

# A pool of three worker fibers over six hosts whose statuses come from a table in the program, and select with a
# timeout.
run("""
cd fibers
crystal run pool.cr
crystal run select_timeout.cr
""")

$ cd fibers
$ crystal run pool.cr
worker 0: api.local
worker 0: docs.local
worker 1: billing.local
worker 2: cdn.local
worker 0: mail.local
worker 0: wiki.local
-- in input order --
api.local: 200
billing.local: 503
cdn.local: 200
docs.local: 404
mail.local: 200
wiki.local: unknown host
$ crystal run select_timeout.cr
cache answered
no answer within 200 ms

Worker 0 handled four of the six hosts. Each time it sent a result it went straight back to jobs, while workers 1 and 2 were still waiting to hand their results over: the order of arrival comes from the scheduler, not from the input, and with parallelism switched on it would change from run to run. That is why each result carries its index and main sorts by it: the report in input order is the same whoever did the work. wiki.local is not in the table, so STATUS[host]? gave nil and the report says unknown host.

select_timeout.cr waits on two channels and a timeout at once. In the first round the fiber that stands for the cache sent its answer while select waited, so select took it; in the second nothing came, and after 200 milliseconds the timeout branch ran. A checker uses the same pattern to give up on a host that does not answer.

lib and fun call C directly, with rules for strings and callbacks

A lib blocklib block: A Crystal declaration that names C functions with their C types, such as fun strlen(s : UInt8*) : LibC::SizeT inside lib LibText, so that the program can call them like class methods. declares C functions with their C types, and the program then calls them like class methods. The C library is linked into every program already, so these need no @[Link] annotation:

# C functions of the system library, declared in lib blocks and called from Crystal.
lib LibText
  fun strlen(s : UInt8*) : LibC::SizeT
  fun toupper(c : Int32) : Int32
end

lib LibMath
  fun cbrt(x : Float64) : Float64
end

puts LibText.strlen("chai")
puts LibText.toupper('q'.ord).chr
puts LibMath.cbrt(27.0)

hindi = "चाय"
puts "#{hindi}: size #{hindi.size}, bytesize #{hindi.bytesize}, strlen #{LibText.strlen(hindi)}"

order = "tea\0coffee"
puts "order: size #{order.size}, bytesize #{order.bytesize}, strlen #{LibText.strlen(order)}"
begin
  order.check_no_null_byte
rescue e : ArgumentError
  puts "check_no_null_byte: #{e.message}"
end

# qsort from the C library sorts a Crystal array in place, calling back into a Crystal proc.
lib LibSort
  fun qsort(base : Void*, count : LibC::SizeT, size : LibC::SizeT, compare : (Void*, Void*) -> Int32)
end

marks = [72, 18, 95, 43, 60]
LibSort.qsort(marks.to_unsafe.as(Void*), marks.size, sizeof(Int32), ->(a : Void*, b : Void*) {
  a.as(Int32*).value <=> b.as(Int32*).value
})
p marks

# A proc that uses a local variable is a closure, and C cannot hold one.
lib LibSort
  fun qsort(base : Void*, count : LibC::SizeT, size : LibC::SizeT, compare : (Void*, Void*) -> Int32)
end

marks = [72, 18, 95, 43, 60]
descending = true
LibSort.qsort(marks.to_unsafe.as(Void*), marks.size, sizeof(Int32), ->(a : Void*, b : Void*) {
  order = a.as(Int32*).value <=> b.as(Int32*).value
  descending ? -order : order
})
p marks

from session import run

# C functions called from Crystal: strlen, toupper and cbrt, qsort with a Crystal callback, and a callback C refuses.
run("""
cd cbind
crystal run libc.cr
crystal run qsort.cr
!crystal run closure.cr
""")

$ cd cbind
$ crystal run libc.cr
4
Q
3.0
चाय: size 3, bytesize 9, strlen 9
order: size 10, bytesize 10, strlen 3
check_no_null_byte: String contains null byte
$ crystal run qsort.cr
[18, 43, 60, 72, 95]
$ crystal run closure.cr
Showing last frame. Use --error-trace for full trace.

In closure.cr:8:69

 8 | LibSort.qsort(marks.to_unsafe.as(Void*), marks.size, sizeof(Int32), ->(a : Void*, b : Void*) {
                                                                         ^
Error: can't send closure to C function (closured vars: descending)
[exit status 1]

The types are C’s, written the Crystal way. C’s char * is UInt8*, a pointer to bytes, and a String passed for it goes as a pointer to its bytes, through its to_unsafe. C counts bytes, not characters: चाय, tea in Hindi, has 3 characters and 9 bytes in UTF-8, and strlen said 9. It also stops at the first zero byte, so for "tea\0coffee" it said 3 where bytesize said 10. A C function given that string sees only tea; check_no_null_byte raises an ArgumentError for such a string, so call it before you pass text from outside the program to C.

qsort.cr sorts a Crystal array with the C library’s qsort, which calls a Crystal proc back to compare two elements. Step through the call:

The proc in closure.cr reads descending, a local variable, so it is a closureclosure: A function, such as a Crystal proc, that uses variables of the code around it and carries them along. A C function pointer has no room for those variables, so Crystal refuses to pass a closure to C., and C has no room for the variable it carries: the compiler said can't send closure to C function (closured vars: descending). The callbacks reference shows the way out for C functions that take a user-data pointer: pass a proc without a closure and send the data through that pointer instead.

Practise: macro output and C strings, in Python

Both steps of the lab rebuild something this lesson ran. The first writes the methods that define_units wrote, as text, and its sample tests compare your text with what crystal tool expand printed; the second works out what a C function sees of a Crystal string. Each step has starter code, sample tests and a first hint:

def define_units(base, factors):
    """The Crystal methods that define_units writes for base and the (unit, factor) pairs."""
    code = ""
    for unit, factor in factors:
        code += f"def {unit}_to_{base}(value)\n  value * {factor}\nend\n"
    return code

from expand import define_units

LENGTH = """def self.inch_to_metre(value : Float64) : Float64
  value * 0.0254
end

def self.foot_to_metre(value : Float64) : Float64
  value * 0.3048
end

def self.yard_to_metre(value : Float64) : Float64
  value * 0.9144
end"""


def test_the_lessons_three_units():
    """what crystal tool expand printed for units.cr"""
    factors = [("inch", "0.0254"), ("foot", "0.3048"), ("yard", "0.9144")]
    assert define_units("metre", factors) == LENGTH


def test_one_unit():
    """a single method, with no empty line after it"""
    assert define_units("gram", [("ounce", "28.349523125")]) == (
        "def self.ounce_to_gram(value : Float64) : Float64\n  value * 28.349523125\nend"
    )


def test_order_follows_the_factors():
    """the methods come in the order of the pairs"""
    code = define_units("litre", [("pint", "0.568261"), ("cup", "0.25")])
    assert code.index("pint_to_litre") < code.index("cup_to_litre")
    assert code.count("\n\n") == 1


def test_no_units():
    """no pairs, no methods"""
    assert define_units("metre", []) == ""

def lengths(text):
    """(size, bytesize, strlen) of a Crystal string that holds text."""
    return (len(text), len(text), len(text))


def safe_for_c(text):
    """True when a C function would see the whole string."""
    return True

from lengths import lengths, safe_for_c


def test_the_lessons_strings():
    """chai, चाय and tea\\0coffee, as libc.cr printed them"""
    assert lengths("chai") == (4, 4, 4)
    assert lengths("चाय") == (3, 9, 9)
    assert lengths("tea\0coffee") == (10, 10, 3)


def test_more_characters():
    """a cup of tea in one character, three bytes; an empty string"""
    assert lengths("☕") == (1, 3, 3)
    assert lengths("") == (0, 0, 0)
    assert lengths("\0") == (1, 1, 0)


def test_zero_bytes_are_unsafe_for_c():
    """check_no_null_byte refuses a string with a zero byte"""
    assert safe_for_c("चाय") is True
    assert safe_for_c("tea\0coffee") is False
    assert safe_for_c("") is True

Then the self-check on the four ideas of the lesson. It asks about the idea your answers show least first, and every answer comes with the reason:

Interview questions

Warm-up (mid-level to senior): When do you write a Crystal macro instead of a method?

When the code to write depends on the program itself rather than on values: methods named after the units of a list, accessors for instance variables, or code that needs the program’s types. A macro runs inside the compiler, takes its arguments as syntax, and writes code that the compiler then checks like any other, so a mistake in what it writes shows up as a compile error with a name, as feet_to_metre did. A method runs while the program runs and takes values. Prefer a method whenever one can do the job: it is easier to read, to test and to debug. When you do write a macro, crystal tool expand shows exactly what it wrote, and the standard library’s property, getter and record are the models to follow.

Two longer questions, on a C call that stalls a server and on a pool whose report changes between machines:

Key takeaways

• A macro runs inside the compiler, takes its arguments as syntax, loops with {% %} and pastes with {{ }}; the methods it writes are checked like any code, and crystal tool expand shows them.
• property, getter and record are macros of the standard library, and macro code can read the program’s own methods while compiling.
• Fibers take turns on one thread: a fiber runs until it waits or ends, a send on a channel without room waits for a receiver, and the program ends with its main fiber.
• In a pool, results arrive in the scheduler’s order, so they carry an index; receive? ends the workers once the jobs channel is closed, and select with timeout gives up on an answer that does not come.
• A lib block declares C functions with C types: strings go as bytes and stop at the first zero byte, and a proc that is a closure cannot be passed to C.