Map, reduce & aggregates

Package: MAPREDUCE_PACKAGE. Registered by createEngine(); for a slimmer engine, register it explicitly (see choosing packages).

Map and reduce are the two ways of working through a list with one expression. Map applies the expression to each element in turn and gives back the list of answers, one for each element. Reduce folds the list down to a single value, carrying a running result from one element to the next, the way a total or a largest-so-far is kept by hand.

Ranges

A range is start:end, the whole numbers from the start to the end, inclusive at both ends. A colon between two numbers means a range only where a list is expected: as the list map, reduce, sum (or total, its synonym) and prod work through. Anywhere else it is a clock time, because that reading is far more common in a document: a bare 0:3, (0:3), and max(9:30, 10:15), which compares two times of day.

map(10*x, 0:3) // [0, 10, 20, 30]

Only the list is read as a range. The expression before it is worked out once for each element, so it is never a range itself, and a colon there is a clock time: sum(9:30, 10:15) is two times of day, the same as total(9:30, 10:15), and both are refused by name, since a time of day is a moment rather than an amount to add up. A length of time adds up as a quantity.

sum(9:30, 10:15) // ERROR: A date or time cannot be added: only numbers and quantities can.
total(9:30, 10:15) // ERROR: A date or time cannot be added: only numbers and quantities can.
sum(2 hours, 3 hours) // 5 hours
sum(x, 1:3) // 6

The same holds for prod, map and reduce: their first argument is the expression, so prod(9:30, 10:15) multiplies a time of day and is refused by name. A list written in square brackets is not a range either. Its items are values, so a colon pair inside it is a clock time, and a list holds numbers, so a list of times is refused as one. Write the range without the brackets, sum(1:3), to add up whole numbers.

prod(9:30, 10:15) // ERROR: A date or time cannot be multiplied: it is a moment, not an amount. A length of time is written 1h30m, 90 minutes or 1:30:00.
sum(x, [9:30, 10:15]) // ERROR: A date or time cannot be a cell of a list: each cell holds one number.
sum(x, [1 hour, 30 min]) // 1.50 hours

A range is the list, so it is always the last argument, and a range followed by more values is not a form: sum(100:200, 50) does not add 50 to the whole numbers from 100 to 200. A pair that can only be a range (whole numbers counting up, with no clock’s shape of an hour and two-digit minutes) is refused by name there, and the refusal gives the call that answers, with the other values added outside it. It used to be refused as the time “100:200”, which named something the line never meant. A pair that is a real time, such as sum(10:12, 5), is still a time, and one in a clock’s shape, such as sum(24:30, 1), is still refused as a time.

sum(100:200, 50) // ERROR: In sum(...), 100:200 is a range, and a range is read only as the last argument, the list sum works through. To add other numbers in as well, write them outside the call: sum(100:200) + 50.
sum(100:200) + 50 // 15,200
map(100:200, 5) // ERROR: In map(...), 100:200 is a range, and a range is read only as the last argument, the list map works through. Write the expression first and the range last, as in map(x * 2, 100:200).

Slicing a matrix, m[0:1, 0:1], keeps its ranges: there the brackets follow a name, and pick a range of its rows and a range of its columns.

Large bounds

A large number is easier to read with its thousands grouped, 1,000, and a range’s bound can be written that way. Inside a call’s brackets a comma usually separates one argument from the next (so max(1,000, 2) is the largest of 1, 0 and 2, see amounts inside a call), but a comma between digits straight against a range’s colon is read as grouping, since nobody writes a range as starting at 000 and a colon after three digits is never a clock time. So sum(1,000:2,000) is the range from one thousand to two thousand, and sum(1,000:1), a range that counts down, is refused by name rather than read as 1 and the range 000:1:

sum(1,000:2,000) // 1,501,500
sum(1,000:1,005) // 6,015
map(x * 2, 1,000:1,002) // [2,000, 2,002, 2,004]
sum(1,000:1) // A range's min (1000) cannot be greater than its max (1). Did you mean "1:1000"?

The second bound is grouped when the first cannot be the hour of a clock time: three or more digits (100:1,000), a name or a bracket. After a short first bound, as in sum(1:2,000), the comma still separates, because 1:2 could be a time. The boundary is the reading the rule cannot tell apart: sum(1,100:200) is the range from 1,100 down to 200, and refused, not the element form adding 1 for each of 100 to 200. To mean two arguments, put a space after the comma:

sum(1, 100:200) // 101
max(1,000, 2) // 2

A grouping comma always has exactly three digits after it, so a comma straight against the colon with any other number of digits after it is neither a grouped bound nor plainly two arguments. sum(1,0000:1) could be the mistyped bound 10,000 or the two values 1 and 0000:1, and it used to be read as the second without a word and answered 2. It is refused by name instead, and the refusal says how to write two values: put a space after the comma.

sum(1,0000:1) // ERROR: "1,0000" is not a number: a grouping comma needs exactly three digits after it. To give two values, put a space after the comma: 1, 0000.
sum(12,3456:1) // ERROR: "12,3456" is not a number: a grouping comma needs exactly three digits after it. To give two values, put a space after the comma: 12, 3456.
sum(1, 0000:1) // 2

The refusal covers a run of two digits, or of four or more, against the colon. Two shapes keep the separator reading, because that is how two values are written: a single digit (sum(1,1:3) adds 1 for each of 1, 2 and 3) and two digits followed by two more after the colon, which may be a clock time (max(9:00,17:30)).

sum(1,1:3) // 3
sum(1,12:30) // 19

Map

map(expression, list) works the expression out once for each element, with x standing for the element, and returns the answers as a list of the same length. It is how a whole list is converted, scaled or squared in one line.

map(10*x, [1,2,3]) // [10, 20, 30]
map(x^2, [4, 5, 6]) // [16, 25, 36]

Reduce

reduce(expression, list) folds a list to one value. acc, the accumulator, holds the result so far and x the next element; the expression says how the two combine. acc starts as the first element, and the fold runs over the rest, so a list of one element reduces to that element.

reduce(acc+x, [1,2,3]) // 6
reduce(acc*x, [2,3,4]) // 24
reduce(acc - x, [10, 1, 2]) // 7
reduce(max(acc, x), [3, 9, 4]) // 9

The third line is 10 - 1 - 2: the 10 is where acc starts, not a value taken away. The last keeps the larger of the running result and each element, so it ends on the largest value in the list.

Sum and product

Shorthand for the two most common reductions. sum(expression, list) adds up the expression worked out for each element, and prod(expression, list) multiplies them.

sum(x, [10, 20, 30]) // 60
prod(x, [2,3,4]) // 24
sum(x, 0:4) // 10
sum(x^2, 1:3) // 14

Given the list alone, they add or multiply its elements as they are, which is the same as writing x for the expression:

sum(1:3) // 6
sum([10, 20, 30]) // 60
prod(1:5) // 120

total is another name for sum, the word a spreadsheet or a receipt uses, and with a single list it reads the same way: a range or a bracketed list inside total(...) is added up. With commas, total(1, 2, 3) adds the values one by one, as sum(1, 2, 3) does, and over lines total(line 1 : line 3) adds a span of the document (see line references). The element form, sum(x^2, 1:3), is sum’s alone.

total(1:3) // 6
total([10, 20, 30]) // 60
total(1, 2, 3) // 6

With two arguments, sum has two readings: the element form, an expression worked out for each item of a list, and two values added up. It is the element form when the second argument is written as a list or a range, or when the first uses x, the name that stands for each item. Otherwise the first argument would be the same for every item, so sum(a, b) of two names from the lines above, or of a name and a number, adds the two values, as total(a, b) does.

price = $5 // $5.00
fee = $7 // $7.00
sum(price, fee) // $12.00
sum(price, 2) // $7.00
xs = [1, 2, 3] // [1, 2, 3]
sum(x * 2, xs) // 12

The boundary: x always stands for the item, so sum(x, y) is the element form even where a line above defines x, and adding a value named x to another is written total(x, y) or x + y. A name that holds a list is not written as one, so sum(a, xs) is the two values added, and refused as total(a, xs) is, since a list is not one value; sum(x, xs) + a adds a value to the list’s sum.

A single argument has to be a list: a range, a bracketed list, or a name holding one. sum(5) is refused, because there is nothing to add it to, and the refusal says so in the words of the call typed; a run of plain values is written with commas, sum(1, 2, 3). prod, map and reduce refuse a single value the same way, each naming what it does with a list. A range counts up in whole numbers, so sum(3:1) and sum(1.5:3) are refused by name rather than read another way.

The refusal of a range that counts down names each bound as it was written, with the number it came to when that differs, since a bound can be worked out from a sum. In total(1 + 24:00) the colon is the range’s, so its bounds are 1 + 24 and 00, not the time 24:00, and the refusal says so in the reader’s own text rather than as two numbers nobody typed:

sum(5:1) // ERROR: A range's min (5) cannot be greater than its max (1). Did you mean "1:5"?
total(1 + 24:00) // ERROR: A range's min (1 + 24, which is 25) cannot be greater than its max (00, which is 0). Did you mean "0:25"?
total(2*3:1) // ERROR: A range's min (2*3, which is 6) cannot be greater than its max (1). Did you mean "1:6"?

sum(5) // sum adds up the items of a list or a range, such as [1, 2, 3] or 1:3, and this is a single number; to add values one by one, list them, as in sum(5, 6).
prod(5) // prod multiplies together the items of a list or a range, such as [1, 2, 3] or 1:3, and this is a single number.
map(x * 2, 5) // map works through the items of a list or a range, such as [1, 2, 3] or 1:3, and this is a single number.
sum(1, 2, 3) // 6

Using another line’s value

The expression of map, reduce, sum and prod is not worked out on the line where it is written: it is held and worked out once for each element, away from the document, so it has no lines above it to read. A reference to another line inside it, prev, line 1, total above, a tag or a table column, is refused by name, and the refusal says how to write it: give the line’s value a name on a line of its own, and use the name in the expression.

5 // 5
map(x + prev, 1:3) // ERROR: map's expression reads other lines of the document, and it is worked out away from the line, where there are no lines to read: give the line's value a name first, as in p = prev, and use p in the expression
p = 5 // 5
map(x + p, 1:3) // [6, 7, 8]
sum(x + p, 1:3) // 21

These used to be refused as though they reached live data, “no weather/stocks/currency calls”, which named something the line never did. A function body (f(x) = x + prev) is refused the same way, for the same reason (see variables), and so are a plot’s expression and the expression of der, solve and integral. The boundary: an expression that does reach live data, a weather or price lookup, keeps its own refusal, since it waits for the network rather than for the document.