Numeric types: double, single, integers, complex

Add 0.1 ten times in Octave and the total is 0.99999999999999989, so s == 1 is false; subtract 5 from the 8-bit unsigned integer 3 and you get 0. Neither is a bug: each numeric class stores numbers its own way, and its rules decide every result you compute with it.

Here you see how doubles store fractions and how far apart they are, compare results with a tolerance, predict what integer classes do when a result is a fraction or out of range, mix classes, use complex numbers and the two transposes, and convert between types without losing data by accident.

Numbers are doubles, and a double is a binary fraction

Every number you type is a double unless you ask for another class: 8 bytes in the 64-bit format of the IEEE 754 standard, which MATLAB and Octave both use. A double is a sum of powers of two, and no finite sum of powers of two equals one tenth, in the same way that no finite decimal equals one third. Octave therefore keeps the double closest to 0.1, about 5.6e-18 too large, as the first line of the output shows, and such tiny errors add up over several operations. This program shows it, then compares two ways of testing results:

% float_traps.m: equality tests that fail on doubles, and comparisons that work
printf('0.1 is stored as %.25f\n', 0.1);
printf('0.1 + 0.2 = %.17g, and 0.1 + 0.2 == 0.3 is %d\n', 0.1 + 0.2, 0.1 + 0.2 == 0.3);

s = 0;                          % add 0.1 ten times
for k = 1:10
  s = s + 0.1;
  printf('after %2d additions: %.17g\n', k, s);
end
printf('s == 1 is %d; abs(s - 1) < 1e-12 is %d\n', s == 1, abs(s - 1) < 1e-12);

% an absolute tolerance has no right size: the same slip is larger on larger numbers
a = 1e6 * (0.1 + 0.2);
b = 1e6 * 0.3;
printf('a - b = %.3g; abs(a - b) < 1e-12 is %d\n', a - b, abs(a - b) < 1e-12);

% a relative tolerance scales with the numbers; NaN is never close to anything
isclose = @(x, y, rtol, atol) abs(x - y) <= max(atol, rtol * max(abs(x), abs(y)));
printf('isclose(a, b, 1e-12, 0) is %d\n', isclose(a, b, 1e-12, 0));
printf('isclose(NaN, NaN, 1e-12, 0) is %d\n', isclose(NaN, NaN, 1e-12, 0));

from session import prompt, run, version

version()
run("octave-cli --no-init-all float_traps.m")
prompt("eps_demo.m")

# GNU Octave (aarch64-apple-darwin25.4.0) version 11.3.0
$ octave-cli --no-init-all float_traps.m
0.1 is stored as 0.1000000000000000055511151
0.1 + 0.2 = 0.30000000000000004, and 0.1 + 0.2 == 0.3 is 0
after  1 additions: 0.10000000000000001
after  2 additions: 0.20000000000000001
after  3 additions: 0.30000000000000004
after  4 additions: 0.40000000000000002
after  5 additions: 0.5
after  6 additions: 0.59999999999999998
after  7 additions: 0.69999999999999996
after  8 additions: 0.79999999999999993
after  9 additions: 0.89999999999999991
after 10 additions: 0.99999999999999989
s == 1 is 0; abs(s - 1) < 1e-12 is 1
a - b = 5.82e-11; abs(a - b) < 1e-12 is 0
isclose(a, b, 1e-12, 0) is 1
isclose(NaN, NaN, 1e-12, 0) is 0
$ octave-cli --quiet
octave:1> printf('%.17g\n', eps(1), eps(1000), eps(1e6))
2.2204460492503131e-16
1.1368683772161603e-13
1.1641532182693481e-10
octave:2> eps(2^53)
ans = 2
octave:3> flintmax
ans = 9.0072e+15
octave:4> flintmax + 1 == flintmax
ans = 1
octave:5> eps('single')
ans = 1.1921e-07
octave:6> flintmax('single')
ans = 1.6777e+07
octave:7> single(16777216) + 1 == single(16777216)
ans = 1

The ten additions drift: after 3 the sum is 0.30000000000000004, after 6 it is 0.59999999999999998, and after 10 it is 0.99999999999999989, so s == 1 is 0, false. Comparing with a tolerance, abs(s - 1) < 1e-12, accepts it.

A fixed tolerance has no right size, though. a = 1e6 * (0.1 + 0.2) and b = 1e6 * 0.3 differ by 5.82e-11, larger than 1e-12, because the same relative slip is a million times larger on numbers a million times larger. A relative tolerance, as in the program’s isclose, scales with the numbers, and it says 1. It also says 0 for two NaN, which is right: NaN is not close to anything, itself included.

eps is the gap to the next double, and it grows with the number

Doubles are spaced unevenly: there are as many between 1 and 2 as between 1,000 and 2,000. eps(x) gives the gap from x to the next larger double, which machine epsilonmachine epsilon: The gap between 1 and the next larger double, eps(1) = 2.2204e-16 in MATLAB and Octave. eps(x) gives the gap next to any x, which grows with the size of x., eps(1), sets for numbers near 1. The prompt part of the session above measured it:

The gap is between about 1.1e-16 and 2.2e-16 times the number, which is why a relative tolerance works and an absolute one does not. At 2^53 the gap reaches 2: flintmax, the largest whole number below which every whole number is exact, is 9,007,199,254,740,992, and flintmax + 1 == flintmax is 1, true, because the sum rounds back. Above it only every second whole number is a double, so counters and IDs there skip values or collide.

single is the 32-bit format, 4 bytes: about 7 significant digits, eps('single') 1.1921e-07, and exact whole numbers only up to flintmax('single'), 16,777,216, where adding 1 changed nothing.

Integer classes round and saturate instead of wrapping

The integer classes int8, uint8, int16, uint16, int32, uint32, int64 and uint64 hold whole numbers in 1, 2, 4 or 8 bytes. Every result is first worked out exactly, then rounded to a whole number, halves away from zero, then set to the nearest end of the class’s range when it falls outside, which is called saturationsaturation: Setting an integer result that falls outside its class's range to the nearest end of the range instead of wrapping round, so that int8(127) + 1 is 127 and uint8(3) - 5 is 0.. Step through five results from the sessions:

from session import prompt, version

version()
prompt("int_rules.m")

# GNU Octave (aarch64-apple-darwin25.4.0) version 11.3.0
$ octave-cli --quiet
octave:1> int8(127) + 1
ans = 127
octave:2> int8(-128) - 1
ans = -128
octave:3> uint8(3) - 5
ans = 0
octave:4> int8(2.5)
ans = 3
octave:5> int8(-2.5)
ans = -3
octave:6> int32(7) / int32(2)
ans = 4
octave:7> int32(-7) / int32(2)
ans = -4
octave:8> idivide(int32(7), int32(2))
ans = 3
octave:9> idivide(int16([-7 -4 7 11]), int16(10))
ans =

  -1  -1   0   1

octave:10> int16(fix(double(int16([-7 -4 7 11])) / 10))
ans =

  0  0  0  1

octave:11> [intmin('int8'), intmax('int8')]
ans =

  -128  127

int8(127) + 1 is 127 and int8(-128) - 1 is -128: the result stays at the edge of the range instead of wrapping round to the other end, as integers do in many other languages. uint8(3) - 5 is 0. int8(2.5) is 3 and int8(-2.5) is -3. Division rounds like everything else, so int32(7) / int32(2) is 4 and int32(-7) / int32(2) is -4, while idivide lets you choose the rounding and is documented to round towards zero by default: idivide(int32(7), int32(2)) is 3.

Mixed classes follow fixed rules, and integers win

When a calculation mixes classes, the result’s class follows rules that MATLAB and Octave share, and most of them surprise people at first:

% adc_counts.m: 12-bit ADC counts to volts around a mid-scale offset
counts = uint16([2048 3000 1000 4095 0]);   % raw readings, 0 to 4095
offset = 2048;                              % the count at 0 V
scale = 3.3 / 4096;                         % volts per count

wrong = (counts - offset) * scale;          % computed in uint16
printf('wrong (%s): %s\n', class(wrong), mat2str(wrong));

right = (double(counts) - offset) * scale;  % computed in double
printf('right (%s): %s\n', class(right), mat2str(right, 4));

from session import prompt, run, version

version()
prompt("mixed_rules.m", errors=1)
run("octave-cli --no-init-all adc_counts.m")

# GNU Octave (aarch64-apple-darwin25.4.0) version 11.3.0
$ octave-cli --quiet
octave:1> m = int8(1) + 2.7
m = 4
octave:2> class(m)
ans = int8
octave:3> int8([1 2]) + [1.5 2.5]
ans =

  3  5

octave:4> int8(1) + int16(1)
error: binary operator '+' not implemented for 'int8 scalar' by 'int16 scalar' operations
octave:5> class(single(1) + 1)
ans = single
octave:6> t = true + true
t = 2
octave:7> class(t)
ans = double
$ octave-cli --no-init-all adc_counts.m
wrong (uint16): [0 1 0 2 0]
right (double): [0 0.767 -0.8443 1.649 -1.65]

MATLAB adds one condition to the first row: a double that meets an integer array must be a scalar, as MathWorks’ page on floating-point numbers states. Octave 11.3 accepts a double row as well: int8([1 2]) + [1.5 2.5] gave the int8 values 3 and 5. Code for both programs therefore keeps such doubles scalar, or converts first.

The first row causes the classic bug, and adc_counts.m has it. A 12-bit converter reads 0 to 4095, stored as uint16, with 2048 meaning 0 V. (counts - offset) * scale is worked out in uint16: every reading below 2048 is clamped to 0, and the volts are rounded to whole numbers, [0 1 0 2 0]. Converting first, double(counts) - offset, gives the right volts, [0 0.767 -0.8443 1.649 -1.65].

Which class to store data in depends on what you will do with it. Answer three questions:

Complex numbers need no setup, but one transpose conjugates them

Complex numbers need no special setup: 3 + 4i is one, and functions that leave the real numbers, such as sqrt(-1), return one. The session also shows the two transposes on a row with a complex element:

from session import prompt, version

version()
prompt("complex_demo.m")

# GNU Octave (aarch64-apple-darwin25.4.0) version 11.3.0
$ octave-cli --quiet
octave:1> z = 3 + 4i;
octave:2> [abs(z), real(z), imag(z)]
ans =

   5   3   4

octave:3> w = [1+2i, 3];
octave:4> w'
ans =

   1 - 2i
   3 - 0i

octave:5> w.'
ans =

   1 + 2i
   3 + 0i

octave:6> sqrt(-1)
ans =  0 + 1i
octave:7> log(0)
ans = -Inf
octave:8> (-8)^(1/3)
ans =  1.0000 + 1.7321i
octave:9> nthroot(-8, 3)
ans = -2

abs(3 + 4i) is 5, the length, and real and imag take the parts. The important line is the transpose: for w = [1+2i, 3], w' turned 1 + 2i into 1 - 2i, because ' is the conjugate transposeconjugate transpose: The transpose that also changes the sign of every imaginary part, written ' in MATLAB and Octave. The plain transpose, written .', only swaps rows and columns; for real data they agree., while w.' only rearranged the elements. For real data the two give the same result, which is why the difference goes unnoticed until complex values arrive, for example from an fft.

Two more lines are worth a look. log(0) is -Inf, not an error. And (-8)^(1/3) is 1.0000 + 1.7321i, the principal complex cube root, while nthroot(-8, 3) is the real root, -2.

cast, typecast and literals convert on purpose

Converting a value to another class is a choice about what to keep. cast(x, 'uint8') and uint8(x) keep the number, rounded and saturated as above. typecast keeps the bits and reads them as another class, and integer literals choose a class as you type them:

from session import prompt, version

version()
prompt("convert_demo.m")

# GNU Octave (aarch64-apple-darwin25.4.0) version 11.3.0
$ octave-cli --quiet
octave:1> cast(300, 'uint8')
ans = 255
octave:2> double(int8(-5)) / 2
ans = -2.5000
octave:3> typecast(single(1), 'uint32')
ans = 1065353216
octave:4> x = 0x1F
x = 31
octave:5> class(x)
ans = uint8
octave:6> b = 0b1010
b = 10
octave:7> v = 0x1FFu16;
octave:8> class(v)
ans = uint16
octave:9> n = 0xF0s8
n = -16

• cast(300, 'uint8') is 255: the number is kept as well as the class allows. double(int8(-5)) / 2 is -2.5, because the division happens in double.
• typecast(single(1), 'uint32') is 1065353216: the same 32 bits read as an unsigned integer. It is the tool for reading raw bytes from a file or a device, and wrong for changing a number’s class.
• Hexadecimal and binary literals such as 0x1F and 0b1010 are uint8 in Octave 11.3, the smallest unsigned class that holds them, and a suffix picks another: 0x1FFu16 is a uint16 and 0xF0s8 is the int8 -16, read in two’s complement. MathWorks documents the same rule and suffixes for MATLAB, where these literals arrived in R2019b.

Practise: classes on your own data

Do these in Octave, and check each against what Octave prints:

1. Add 0.1 a hundred times and compare the total with 10, first with ==, then with a relative tolerance.
2. Store the counts uint16([100 2048 4000]) and convert them to volts around 2048 with the scale 3.3 / 4096, first wrongly in uint16, then correctly in double. The correct second value is 0.
3. Predict int8(100) * 2, uint8(250) + 10, int16(-7) / int16(2) and idivide(int16(-7), int16(2)), then check. Which of them would MATLAB’s documentation give differently?
4. Make a complex column from w = [1+2i, 3-1i] with w.', then with w', and say which line changes the signs.

The self-check below asks up to twelve questions on doubles, integers, complex numbers and conversions, with a reason for each answer:

Interview questions

Warm-up: why is 0.1 + 0.2 == 0.3 false in MATLAB and Octave, and how should you compare such values?

MATLAB and Octave store numbers as IEEE 754 doubles, sums of powers of two, and none of 0.1, 0.2 or 0.3 is such a sum, so each is held as the closest double instead. The rounding errors of 0.1 and 0.2 add up to 0.30000000000000004, which is one double away from the stored 0.3, so == says false. This is how the arithmetic is defined in every language that uses IEEE doubles, not a fault of MATLAB or Octave. Compare computed values with a tolerance instead: an absolute one, abs(x - y) < 1e-12, for numbers near 1, and a relative one, abs(x - y) <= 1e-12 * max(abs(x), abs(y)), for anything else, because the gap between doubles grows with their size, from 2.2e-16 at 1 to 1.2e-10 at a million.

Key takeaways

• Numbers are IEEE 754 doubles by default, binary fractions in which 0.1 is not exact, so sums drift and computed values are compared with a tolerance, not with ==.
• eps(x) is the gap to the next double and grows with x, so tolerances are relative; whole numbers are exact up to flintmax, 2^53, and single has about 7 digits and stops at 16,777,216.
• Integer classes round to the nearest whole number, halves away from zero, and saturate at intmin and intmax instead of wrapping; integer division rounds too.
• An integer mixed with a double gives the integer class, two integer classes cannot mix, and single wins over double, so convert counts with double before calculating.
• ' is the conjugate transpose and .' the plain one; complex results such as (-8)^(1/3) appear without a warning.
• cast keeps the number, typecast keeps the bits, and 0x and 0b literals pick the smallest unsigned class unless a suffix chooses another.