jq: query and reshape JSON from the shell

One line of jq pulls three paid orders out of a four-order JSON blob and sums their amounts: jq '[.orders[] | select(.status == "paid") | .amount] | add' orders.json prints 1870. The same work in a hand-written script is a loop, a parser and an accumulator. jq ships that pipeline as its everyday idiom and runs it on a stream, one value at a time.

You will build filters out of five building blocks (., .key, .[], the pipe | and the comma ,); use select, map, group_by, sort_by and unique_by to filter, transform and summarise an orders file; pass shell values in with --arg and --argjson instead of pasting them into the filter; format output with @csv, @tsv, @sh and -r so a bash loop can read it; and read jq’s exit status in a script, including the special -e flag. The toolchain pins jq 1.8.2, with its filters and format strings documented in the manual sections cited below.

A filter is a pipeline of generators

A jq filterjq filter: A jq program; a chain of operators that reads JSON on standard input and writes JSON on standard output, one value at a time. is a short program that reads JSON on standard input and writes JSON on standard output. Five building blocks make almost every filter: the identity ., the object identifier-index .key or .["key"], the array index .[0] and iterator .[], the pipe | and the comma ,. The jq manual says each one on its own, but the lesson’s examples use them together, which is how a real filter looks.

"""A filter is a pipeline of generators: ., .key, .[], the pipe and the comma.

The session shows:
1. The identity filter . reads JSON on standard input and pretty-prints it on standard output.
2. .orders is the object identifier-index; .orders[0] indexes an array; .orders[].id iterates it.
3. The pipe | feeds the output of the filter on its left into the one on its right (jq manual, Pipe).
4. The comma , runs two filters on the same input and prints both streams in order.
5. -c (--compact-output) prints each result on one line; -r (--raw-output) drops the quotes on a string.
6. .foo? uses the optional operator so a type mismatch produces no output instead of an error.
"""
from session import run

run("""
# A tiny JSON file of three orders with id, city, items, amount and status
printf '%s\\n' '{"orders":[{"id":"o1","city":"Pune","items":2,"amount":450,"status":"paid"},{"id":"o2","city":"Chennai","items":1,"amount":220,"status":"pending"},{"id":"o3","city":"Pune","items":4,"amount":880,"status":"paid"}]}' > orders.json
jq --version

# . is the identity filter: it reads the input and pretty-prints it
jq . orders.json

# .orders is an object index; it picks one field of the object
jq '.orders' orders.json

# .[0] indexes an array; combine with .orders to pick the first order
jq '.orders[0]' orders.json

# .[] iterates the array and emits each element as its own value
jq '.orders[].id' orders.json

# The pipe | combines two filters: the output of the left feeds the right
jq '.orders[] | .amount' orders.json

# The comma , runs two filters on the same input and concatenates their outputs
jq '.orders[0] | .id, .amount' orders.json

# -c prints each result on one line (compact); good for a wc -l count
jq -c '.orders[]' orders.json

# -r drops the surrounding quotes on a string result; integers print as themselves
jq -r '.orders[].city' orders.json

# A mixed array holds numbers and objects; .[].city then raises an error on the number
printf '%s\\n' '[{"city":"Pune"},42,{"city":"Chennai"}]' > mixed.json
!jq '.[].city' mixed.json

# .foo? is the optional operator: a type error gives no output and the stream continues
jq '.[].city?' mixed.json
""")

# A tiny JSON file of three orders with id, city, items, amount and status
$ printf '%s\n' '{"orders":[{"id":"o1","city":"Pune","items":2,"amount":450,"status":"paid"},{"id":"o2","city":"Chennai","items":1,"amount":220,"status":"pending"},{"id":"o3","city":"Pune","items":4,"amount":880,"status":"paid"}]}' > orders.json
$ jq --version
jq-1.8.2
# . is the identity filter: it reads the input and pretty-prints it
$ jq . orders.json
{
  "orders": [
    {
      "id": "o1",
      "city": "Pune",
      "items": 2,
      "amount": 450,
      "status": "paid"
    },
    {
      "id": "o2",
      "city": "Chennai",
      "items": 1,
      "amount": 220,
      "status": "pending"
    },
    {
      "id": "o3",
      "city": "Pune",
      "items": 4,
      "amount": 880,
      "status": "paid"
    }
  ]
}
# .orders is an object index; it picks one field of the object
$ jq '.orders' orders.json
[
  {
    "id": "o1",
    "city": "Pune",
    "items": 2,
    "amount": 450,
    "status": "paid"
  },
  {
    "id": "o2",
    "city": "Chennai",
    "items": 1,
    "amount": 220,
    "status": "pending"
  },
  {
    "id": "o3",
    "city": "Pune",
    "items": 4,
    "amount": 880,
    "status": "paid"
  }
]
# .[0] indexes an array; combine with .orders to pick the first order
$ jq '.orders[0]' orders.json
{
  "id": "o1",
  "city": "Pune",
  "items": 2,
  "amount": 450,
  "status": "paid"
}
# .[] iterates the array and emits each element as its own value
$ jq '.orders[].id' orders.json
"o1"
"o2"
"o3"
# The pipe | combines two filters: the output of the left feeds the right
$ jq '.orders[] | .amount' orders.json
450
220
880
# The comma , runs two filters on the same input and concatenates their outputs
$ jq '.orders[0] | .id, .amount' orders.json
"o1"
450
# -c prints each result on one line (compact); good for a wc -l count
$ jq -c '.orders[]' orders.json
{"id":"o1","city":"Pune","items":2,"amount":450,"status":"paid"}
{"id":"o2","city":"Chennai","items":1,"amount":220,"status":"pending"}
{"id":"o3","city":"Pune","items":4,"amount":880,"status":"paid"}
# -r drops the surrounding quotes on a string result; integers print as themselves
$ jq -r '.orders[].city' orders.json
Pune
Chennai
Pune
# A mixed array holds numbers and objects; .[].city then raises an error on the number
$ printf '%s\n' '[{"city":"Pune"},42,{"city":"Chennai"}]' > mixed.json
$ jq '.[].city' mixed.json
jq: error (at mixed.json:1): Cannot index number with string ("city")
"Pune"
[exit status 5]
# .foo? is the optional operator: a type error gives no output and the stream continues
$ jq '.[].city?' mixed.json
"Pune"
"Chennai"

The session’s jq . orders.json pretty-printed the whole object; jq '.orders[0]' orders.json picked the first order out of the array; jq '.orders[].id' orders.json iterated the array and printed three ids, "o1", "o2" and "o3". The pipe | and the comma , are the two glue operators: .orders[] | .amount printed 450, 220 and 880 by feeding each order into .amount, while .orders[0] | .id, .amount ran the same input through both filters and printed "o1" and 450 in order. Each operator is a generator: .[] on an array of four values emits four values, and the next stage runs once per value.

Four command-line flags change what jq reads and writes. -n runs the program once with null as input, so the filter builds JSON from the command line without reading anything (jq -n '{a:1}' prints {"a":1}). -s (slurpslurp: jq's --slurp or -s flag, which reads every input JSON value into one array and runs the filter once on that array, instead of once per value.) reads every input value into one array first, so a filter sees it as a single value instead of a stream. -c prints each output on one line (good for wc -l); -r (raw outputraw output: jq's --raw-output or -r flag, which prints a string result without the enclosing double quotes, so the shell reads it as plain text.) drops the quotes on a string result, so jq -r '.orders[].city' printed Pune, Chennai, Pune (plain words, not JSON strings). The session’s run of jq --version confirmed jq 1.8.2 on the toolchain.

The optional operator .foo? catches a type error and emits no output instead. In basics_session.py the mixed array [{"city":"Pune"},42,{"city":"Chennai"}] failed with jq '.[].city' (Cannot index number with string ("city"), exit 5) because the middle element was a number. The ? form jq '.[].city?' printed "Pune" and "Chennai" and skipped the error. Use it when a stream can hold mixed shapes.

select, map and the toolkit of builtins

The filter pipelinefilter pipeline: A chain of jq filters joined by |, so each value flows one at a time from the stage on its left to the stage on its right. gets short when a few builtins do the heavy lifting. select(cond) keeps a value when cond is true and drops it otherwise; map(f) applies f to every element of an array and returns a new array. The combination is the “filter a list and reshape it” everyday case, written as .orders | map(select(.status == "paid")). The manual’s “Builtin operators and functions” section lists every form; the lesson’s session uses the ones a daily script reaches for.

"""Builtins: select, map, add, length, group_by, sort_by, unique_by, min_by, to_entries and //.

The session shows:
1. select(cond) keeps a value in the stream only if cond is true; map(f) applies f to every array element.
2. [filter] wraps a stream of outputs back into an array; add then sums or concatenates the array.
3. length counts the elements of an array, the characters of a string or the keys of an object (jq manual).
4. group_by(.city) returns groups sorted by the key (jq manual), so a totals report is deterministic.
5. sort_by(.amount), unique_by(.city) and min_by/max_by take any expression as the sorting key.
6. The alternative operator a // b takes a when it is non-null and non-false, else b: a default without an if.
7. Object construction {id: .id, city: .city, total: .amount * .items} builds a new JSON value from fields.
"""
from session import run

run("""
# The orders file from the basics session, with one more order to make groups non-trivial
printf '%s\\n' '{"orders":[{"id":"o1","city":"Pune","items":2,"amount":450,"status":"paid"},{"id":"o2","city":"Chennai","items":1,"amount":220,"status":"pending"},{"id":"o3","city":"Pune","items":4,"amount":880,"status":"paid"},{"id":"o4","city":"Kochi","items":3,"amount":540,"status":"paid"}]}' > orders.json

# select(cond) keeps a value only when cond is true; .[] | select(...) is the filtering idiom
jq -c '.orders[] | select(.status == "paid")' orders.json

# map(f) applies f to every array element, then returns a new array
jq '.orders | map(.amount)' orders.json

# length counts the elements; works on arrays, strings and objects
jq '.orders | length' orders.json
jq '.orders[0] | length' orders.json
jq '.orders[0].city | length' orders.json

# add sums or concatenates the elements of an array; [filter] wraps a stream back into an array
jq '[.orders[].amount] | add' orders.json

# sort_by(.amount) sorts the array of orders by amount, ascending
jq -c '.orders | sort_by(.amount) | .[].id' orders.json

# group_by(.city) returns groups sorted by the key; map builds the totals report
jq -c '.orders | group_by(.city) | map({city: .[0].city, total: (map(.amount) | add)})' orders.json

# unique_by(.city) keeps one order per city, in sorted key order
jq -c '.orders | unique_by(.city) | .[].city' orders.json

# min_by and max_by take any expression as the key: here the smallest and largest amount
jq -c '[(.orders | min_by(.amount) | .id), (.orders | max_by(.amount) | .id)]' orders.json

# to_entries turns an object into a list of {key, value}; good when the keys are not fixed
printf '%s\\n' '{"Pune":1330,"Chennai":220,"Kochi":540}' | jq -c 'to_entries | map("\\(.key)=\\(.value)")'

# a // b falls back to b when a is null or false: a default without an if
jq -r '.orders[] | .discount // 0, .status // "unknown"' orders.json

# Object construction builds a new JSON value from a few fields of the input
jq -c '.orders[] | {id, city, total: (.amount * .items)}' orders.json
""")

# The orders file from the basics session, with one more order to make groups non-trivial
$ printf '%s\n' '{"orders":[{"id":"o1","city":"Pune","items":2,"amount":450,"status":"paid"},{"id":"o2","city":"Chennai","items":1,"amount":220,"status":"pending"},{"id":"o3","city":"Pune","items":4,"amount":880,"status":"paid"},{"id":"o4","city":"Kochi","items":3,"amount":540,"status":"paid"}]}' > orders.json
# select(cond) keeps a value only when cond is true; .[] | select(...) is the filtering idiom
$ jq -c '.orders[] | select(.status == "paid")' orders.json
{"id":"o1","city":"Pune","items":2,"amount":450,"status":"paid"}
{"id":"o3","city":"Pune","items":4,"amount":880,"status":"paid"}
{"id":"o4","city":"Kochi","items":3,"amount":540,"status":"paid"}
# map(f) applies f to every array element, then returns a new array
$ jq '.orders | map(.amount)' orders.json
[
  450,
  220,
  880,
  540
]
# length counts the elements; works on arrays, strings and objects
$ jq '.orders | length' orders.json
4
$ jq '.orders[0] | length' orders.json
5
$ jq '.orders[0].city | length' orders.json
4
# add sums or concatenates the elements of an array; [filter] wraps a stream back into an array
$ jq '[.orders[].amount] | add' orders.json
2090
# sort_by(.amount) sorts the array of orders by amount, ascending
$ jq -c '.orders | sort_by(.amount) | .[].id' orders.json
"o2"
"o1"
"o4"
"o3"
# group_by(.city) returns groups sorted by the key; map builds the totals report
$ jq -c '.orders | group_by(.city) | map({city: .[0].city, total: (map(.amount) | add)})' orders.json
[{"city":"Chennai","total":220},{"city":"Kochi","total":540},{"city":"Pune","total":1330}]
# unique_by(.city) keeps one order per city, in sorted key order
$ jq -c '.orders | unique_by(.city) | .[].city' orders.json
"Chennai"
"Kochi"
"Pune"
# min_by and max_by take any expression as the key: here the smallest and largest amount
$ jq -c '[(.orders | min_by(.amount) | .id), (.orders | max_by(.amount) | .id)]' orders.json
["o2","o3"]
# to_entries turns an object into a list of {key, value}; good when the keys are not fixed
$ printf '%s\n' '{"Pune":1330,"Chennai":220,"Kochi":540}' | jq -c 'to_entries | map("\(.key)=\(.value)")'
["Pune=1330","Chennai=220","Kochi=540"]
# a // b falls back to b when a is null or false: a default without an if
$ jq -r '.orders[] | .discount // 0, .status // "unknown"' orders.json
0
paid
0
pending
0
paid
0
paid
# Object construction builds a new JSON value from a few fields of the input
$ jq -c '.orders[] | {id, city, total: (.amount * .items)}' orders.json
{"id":"o1","city":"Pune","total":900}
{"id":"o2","city":"Chennai","total":220}
{"id":"o3","city":"Pune","total":3520}
{"id":"o4","city":"Kochi","total":1620}

The orders file holds four orders in three cities, with totals 450, 220, 880 and 540. The session’s jq -c '.orders[] | select(.status == "paid")' orders.json kept the three paid orders in a stream; jq '[.orders[].amount] | add' orders.json wrapped the stream of amounts back into an array and summed it to 2090. length is overloaded by input type: .orders | length is 4 (an array), .orders[0] | length is 5 (the keys of an object) and .orders[0].city | length is 4 (the characters of "Pune"), as the manual’s length entry describes.

sort_by(.amount) sorts the array ascending, so the ids came back as "o2", "o1", "o4", "o3". group_by(.city) returns groups ordered by the key (jq manual, group_by), which is why the recorded totals-by-city report starts with Chennai 220, then Kochi 540, then Pune 1330, deterministic across runs. unique_by(.city) is the one-order-per-city form, also in sorted key order. min_by(.amount) and max_by(.amount) on the lesson’s four orders returned "o2" and "o3".

The alternative operator // takes the left-hand value when it is neither null nor false, otherwise the right. .discount // 0 printed 0 for every order because no order has a discount, and .status // "unknown" printed each real status. The jq manual explains that a // b works on a stream too: it emits the non-null, non-false values of a and falls back to b only when there are none. Pairing // with an optional operator is how a filter stays robust against missing fields.

Two short forms show up repeatedly in reports. {id, city} is shorthand for {id: .id, city: .city} and is a cheap way to pick a few fields of an object; "\(.id): \(.amount)" is string interpolationstring interpolation: jq syntax: inside a string, \(expr) is replaced by the expression's value. @format "..." applies the format to every interpolation only.. The session’s final command built per-order totals with {id, city, total: (.amount * .items)}, which printed {"id":"o1","city":"Pune","total":900} for the first order.

Pass shell values with the arg option, not into the filter text

Pasting a shell variable into the filter text looks fine until a value contains a jq operator. The result is a two-stage parse: the shell expands the variable, and jq then parses whatever the shell produced. The right form is --arg name value, which the manual’s “Invoking jq” section documents as “a value that will be treated as a string” and bound to $name in the filter.

"""Pass shell values into jq with --arg and --argjson, not with string concatenation.

The session shows:
1. Pasting a shell variable into the filter text is a two-stage parse: the shell expands it, then jq
   parses the result. A value with a jq operator ("Pune\\" or \\"Chennai") reshapes the filter and
   the output is silently wrong, with no error for a script to catch.
2. --arg name value binds a shell string to a jq variable; its value is read exactly, so any shape of
   text stays a string ($city, $query).
3. --argjson name JSON parses the value as JSON, so a number comes through as a number and jq compares
   it as a number, not a string.
4. jq -n with --arg builds a JSON object from the command line, with no input file at all.
"""
from session import run

run("""
printf '%s\\n' '{"orders":[{"id":"o1","city":"Pune","amount":450,"status":"paid"},{"id":"o2","city":"Chennai","amount":220,"status":"pending"},{"id":"o3","city":"Pune","amount":880,"status":"paid"},{"id":"o4","city":"Kochi","amount":540,"status":"paid"}]}' > orders.json
city='Pune'

# Pasting the shell value into the filter text looks fine for a plain word
jq ".orders[] | select(.city == \\"$city\\")" orders.json

# The safe form: --arg city $city binds the shell string to $city inside jq, exactly as written
jq --arg city "$city" '.orders[] | select(.city == $city)' orders.json

# A value that holds a jq operator reshapes the pasted filter and the output is silently wrong
awkward='Pune" or "Chennai'
jq ".orders[] | select(.city == \\"$awkward\\")" orders.json | jq -s length

# The --arg form is unchanged by the awkward value, because jq reads it as a string, not as code
jq --arg city "$awkward" '.orders[] | select(.city == $city)' orders.json | jq -s length

# --argjson parses the value as JSON, so a number stays a number and the comparison is numeric
min=500
jq --argjson min "$min" '.orders[] | select(.amount >= $min) | .id' orders.json

# --arg with a number binds it as a STRING, so the comparison then fails: no match
jq --arg min "$min" '.orders[] | select(.amount >= $min) | .id' orders.json

# -n (--null-input) runs the filter once with null as input: build JSON from the command line only
jq -n --arg city "$city" --argjson amount 1330 '{report: {city: $city, total: $amount}}'
""")

$ printf '%s\n' '{"orders":[{"id":"o1","city":"Pune","amount":450,"status":"paid"},{"id":"o2","city":"Chennai","amount":220,"status":"pending"},{"id":"o3","city":"Pune","amount":880,"status":"paid"},{"id":"o4","city":"Kochi","amount":540,"status":"paid"}]}' > orders.json
$ city='Pune'
# Pasting the shell value into the filter text looks fine for a plain word
$ jq ".orders[] | select(.city == \"$city\")" orders.json
{
  "id": "o1",
  "city": "Pune",
  "amount": 450,
  "status": "paid"
}
{
  "id": "o3",
  "city": "Pune",
  "amount": 880,
  "status": "paid"
}
# The safe form: --arg city $city binds the shell string to $city inside jq, exactly as written
$ jq --arg city "$city" '.orders[] | select(.city == $city)' orders.json
{
  "id": "o1",
  "city": "Pune",
  "amount": 450,
  "status": "paid"
}
{
  "id": "o3",
  "city": "Pune",
  "amount": 880,
  "status": "paid"
}
# A value that holds a jq operator reshapes the pasted filter and the output is silently wrong
$ awkward='Pune" or "Chennai'
$ jq ".orders[] | select(.city == \"$awkward\")" orders.json | jq -s length
4
# The --arg form is unchanged by the awkward value, because jq reads it as a string, not as code
$ jq --arg city "$awkward" '.orders[] | select(.city == $city)' orders.json | jq -s length
0
# --argjson parses the value as JSON, so a number stays a number and the comparison is numeric
$ min=500
$ jq --argjson min "$min" '.orders[] | select(.amount >= $min) | .id' orders.json
"o3"
"o4"
# --arg with a number binds it as a STRING, so the comparison then fails: no match
$ jq --arg min "$min" '.orders[] | select(.amount >= $min) | .id' orders.json
# -n (--null-input) runs the filter once with null as input: build JSON from the command line only
$ jq -n --arg city "$city" --argjson amount 1330 '{report: {city: $city, total: $amount}}'
{
  "report": {
    "city": "Pune",
    "total": 1330
  }
}

The session set city='Pune' and ran two filters that look alike: jq ".orders[] | select(.city == \"$city\")" orders.json and jq --arg city "$city" '.orders[] | select(.city == $city)' orders.json. Both returned the two Pune orders. Then it set awkward='Pune" or "Chennai' and ran the same two forms. The pasted form expanded to the filter .orders[] | select(.city == "Pune" or "Chennai"), which jq read as (.city == "Pune") or "Chennai"; because the string "Chennai" is neither false nor null, or is true for every row. jq -s length on the result printed 4: every order came back. The --arg form on the same awkward value printed 0: no city equals the awkward string, so the stream was empty. One form reshaped the program, the other kept it as written.

The two options differ in parsing. --arg min 500 binds $min to the string "500", so select(.amount >= $min) on the orders file matched nothing (jq’s sort order places numbers below strings, so 450 >= "500" is false; the session’s run printed nothing). --argjson min 500 binds $min to the number 500, so the same filter printed "o3" and "o4", the orders at or above 500. Use --arg for a string and --argjson for a value that must stay a number, a boolean or a parsed object. jq -n --arg city "$city" --argjson amount 1330 '{report: {city: $city, total: $amount}}' built the report {"report":{"city":"Pune","total":1330}} from the command line without reading any file.

The lesson of quoting-single-double-and-ansi-c covers the shell side of this: single-quote the jq program itself so the shell leaves $name, |, * and ( alone, and quote "$value" on --arg city "$value" so a space in the value does not split the word.

Format strings and raw output for the shell

A jq filter that ends with a string can go straight into a bash loop, as long as its output follows a format the loop expects. -r (raw output) is the first flag: it drops the enclosing quotes on a string result. The @format family then shapes a value for a target syntax: @csv, @tsv, @sh, @uri, @html, @base64. The manual’s “Format strings and escaping” section lists the full set; the lesson’s session uses the four that come up in shell integration.

"""Shell-friendly output: @csv, @tsv, @sh, @uri, raw strings and the while IFS read loop.

The session shows:
1. @csv takes an array and prints a CSV row that quotes strings and escapes a double quote by doubling it.
2. @tsv prints a tab-separated row; it escapes a tab, newline, carriage return and backslash with \\t \\n \\r \\\\.
3. @sh escapes a value for a POSIX shell: an apostrophe becomes '\\'' and the whole value is single-quoted.
4. String interpolation "\\(exp)" reads an expression inside a string; @format "..." applies the format to
   every interpolation only (jq manual, Format strings and escaping).
5. jq -r | while IFS=$'\\t' read -r is the standard way to iterate a jq result in a bash loop.
"""
from session import run

run("""
printf '%s\\n' '{"orders":[{"id":"o1","city":"Pune","amount":450,"status":"paid","note":"needs invoice"},{"id":"o2","city":"Chennai","amount":220,"status":"pending","note":"gift wrap"},{"id":"o3","city":"Pune","amount":880,"status":"paid","note":"split \\"pay\\""}]}' > orders.json

# @csv needs an array as input; strings get double quotes and an inner quote is doubled
jq -r '.orders[] | [.id, .city, .amount, .note] | @csv' orders.json

# @tsv is tab-separated; a tab, newline, CR or backslash in a value is written as a two-letter escape
jq -r '.orders[] | [.id, .city, .amount] | @tsv' orders.json

# @sh escapes each element for a POSIX shell: an apostrophe becomes '\\'' so the shell sees it as text
jq -r '[.orders[0].id, "O\\u0027Hara\\u0027s Ale"] | @sh' orders.json

# @uri "..." applies percent-encoding to every interpolation; the fixed text stays as written
jq -rn '@uri "https://example.test/search?q=\\("needs invoice")&city=\\("Pune")"'

# -r (raw-output) is a string without the outer quotes; the shell reads it with IFS set to a tab
jq -r '.orders[] | [.id, .city, .amount] | @tsv' orders.json | while IFS=$'\\t' read -r id city amount; do printf 'order=%s city=%s amount=%s\\n' "$id" "$city" "$amount"; done

# --raw-output0 (jq 1.7+; checked here on jq 1.8.2) prints a NUL after each value
jq --raw-output0 '.orders[].note' orders.json | tr '\\0' '|'
""")

$ printf '%s\n' '{"orders":[{"id":"o1","city":"Pune","amount":450,"status":"paid","note":"needs invoice"},{"id":"o2","city":"Chennai","amount":220,"status":"pending","note":"gift wrap"},{"id":"o3","city":"Pune","amount":880,"status":"paid","note":"split \"pay\""}]}' > orders.json
# @csv needs an array as input; strings get double quotes and an inner quote is doubled
$ jq -r '.orders[] | [.id, .city, .amount, .note] | @csv' orders.json
"o1","Pune",450,"needs invoice"
"o2","Chennai",220,"gift wrap"
"o3","Pune",880,"split ""pay"""
# @tsv is tab-separated; a tab, newline, CR or backslash in a value is written as a two-letter escape
$ jq -r '.orders[] | [.id, .city, .amount] | @tsv' orders.json
o1	Pune	450
o2	Chennai	220
o3	Pune	880
# @sh escapes each element for a POSIX shell: an apostrophe becomes '\'' so the shell sees it as text
$ jq -r '[.orders[0].id, "O\u0027Hara\u0027s Ale"] | @sh' orders.json
'o1' 'O'\''Hara'\''s Ale'
# @uri "..." applies percent-encoding to every interpolation; the fixed text stays as written
$ jq -rn '@uri "https://example.test/search?q=\("needs invoice")&city=\("Pune")"'
https://example.test/search?q=needs%20invoice&city=Pune
# -r (raw-output) is a string without the outer quotes; the shell reads it with IFS set to a tab
$ jq -r '.orders[] | [.id, .city, .amount] | @tsv' orders.json | while IFS=$'\t' read -r id city amount; do printf 'order=%s city=%s amount=%s\n' "$id" "$city" "$amount"; done
order=o1 city=Pune amount=450
order=o2 city=Chennai amount=220
order=o3 city=Pune amount=880
# --raw-output0 (jq 1.7+; checked here on jq 1.8.2) prints a NUL after each value
$ jq --raw-output0 '.orders[].note' orders.json | tr '\0' '|'
needs invoice|gift wrap|split "pay"|

The session’s jq -r '.orders[] | [.id, .city, .amount, .note] | @csv' orders.json wrote one CSV row per order; the note on o3 contained a literal "pay", which @csv quoted by doubling the inner double quote ("split ""pay"""), as the RFC 4180 convention and jq manual’s @csv entry require. @tsv produced the same three fields separated by tabs. @sh on ["o1", "O'Hara's Ale"] wrote 'o1' 'O'\''Hara'\''s Ale': every element surrounded by single quotes, with an inner apostrophe written as the five characters '\'', which is what a POSIX shell reads back as one word with an apostrophe inside.

The @format "..." template form applies the format to every \(expr) interpolation only: @uri "https://example.test/search?q=\("needs invoice")&city=\("Pune")" wrote https://example.test/search?q=needs%20invoice&city=Pune, so the fixed https:// and the & stayed as they were, while the two interpolated values were percent-encoded. For a CSV template the input to each \(…) must still be a value @csv can quote; the simple form [.a, .b] | @csv is the one that always works.

A bash loop reads jq’s output with IFS set to the field separator. jq -r '.orders[] | [.id, .city, .amount] | @tsv' orders.json | while IFS=$'\t' read -r id city amount; do printf 'order=%s city=%s amount=%s\n' "$id" "$city" "$amount"; done printed order=o1 city=Pune amount=450 and two more lines, one per order. The same pattern works for CSV when values do not contain commas, which is why serious CSV work goes through @tsv first and uses the tab. The reading-files-line-by-line lesson covers the loop in detail. jq --raw-output0 (jq 1.7 and later, checked here on 1.8.2) writes a NUL byte between values instead of a newline, so a value that contains a newline can still be read by while IFS= read -r -d '' value.

Exit status in scripts, and when to use the e flag

jq’s exit status is where a script decides whether the data was there. The manual’s --exit-status entry gives the four values a script watches for: 0 when the last output was neither false nor null; 1 when the last output was false or null; 4 when the filter produced no output at all; and the generic ones a shell knows, 2 for a usage or system error and 3 for a program (compile) error. Invalid JSON input returns 5 by default, which is halt_error’s default exit status.

"""jq's exit status: 0, 1, 4 and 5, and how a script reads them.

The session shows:
1. Without -e, jq exits 0 when the program compiles and runs, even if the last output is null or false.
2. With -e (--exit-status), jq exits 1 when the last output is false or null, and 4 when the filter
   produced no output. 0 means "the last value was neither false nor null" (jq manual, --exit-status).
3. Invalid JSON input raises a parse error; the default halt_error exit status is 5.
4. A missing path through .[] on a non-array raises a type error with the same exit 5.
5. jq -e inside an if is the idiom for "fail the script when the field is missing".
"""
from session import run

run("""
printf '%s\\n' '{"orders":[{"id":"o1","city":"Pune","amount":450,"status":"paid"},{"id":"o2","city":"Chennai","amount":220,"status":"pending"}]}' > orders.json
jq --version

# Without -e, a filter that produces null still exits 0: the run itself was fine
jq '.orders[2]' orders.json
echo "exit=$?"

# With -e, the same filter now exits 1 (last output was null)
!jq -e '.orders[2]' orders.json
echo "exit=$?"

# With -e, a filter that produces nothing at all exits 4 (no valid result was ever produced)
!jq -e '.orders[] | select(.status == "cancelled")' orders.json
echo "exit=$?"

# With -e, a filter whose last output is a non-null, non-false value exits 0
jq -e '.orders[0].id' orders.json
echo "exit=$?"

# Invalid JSON on standard input exits 5 (parse error)
!printf 'not json\\n' | jq .
echo "exit=$?"

# A compile error in the filter itself exits 3
!jq 'this is not jq' orders.json
echo "exit=$?"

# jq -e in an if is the usual "did the field show up?" test
if jq -e '.orders[0].city' orders.json >/dev/null; then echo "city is set"; else echo "city is missing"; fi
if jq -e '.orders[0].discount' orders.json >/dev/null; then echo "discount is set"; else echo "discount is missing"; fi
""")

$ printf '%s\n' '{"orders":[{"id":"o1","city":"Pune","amount":450,"status":"paid"},{"id":"o2","city":"Chennai","amount":220,"status":"pending"}]}' > orders.json
$ jq --version
jq-1.8.2
# Without -e, a filter that produces null still exits 0: the run itself was fine
$ jq '.orders[2]' orders.json
null
$ echo "exit=$?"
exit=0
# With -e, the same filter now exits 1 (last output was null)
$ jq -e '.orders[2]' orders.json
null
[exit status 1]
$ echo "exit=$?"
exit=1
# With -e, a filter that produces nothing at all exits 4 (no valid result was ever produced)
$ jq -e '.orders[] | select(.status == "cancelled")' orders.json
[exit status 4]
$ echo "exit=$?"
exit=4
# With -e, a filter whose last output is a non-null, non-false value exits 0
$ jq -e '.orders[0].id' orders.json
"o1"
$ echo "exit=$?"
exit=0
# Invalid JSON on standard input exits 5 (parse error)
$ printf 'not json\n' | jq .
jq: parse error: Invalid numeric literal at line 1, column 4
[exit status 5]
$ echo "exit=$?"
exit=5
# A compile error in the filter itself exits 3
$ jq 'this is not jq' orders.json
jq: error: syntax error, unexpected IDENT, expecting end of file at <top-level>, line 1, column 6:
    this is not jq
         ^^
jq: 1 compile error
[exit status 3]
$ echo "exit=$?"
exit=3
# jq -e in an if is the usual "did the field show up?" test
$ if jq -e '.orders[0].city' orders.json >/dev/null; then echo "city is set"; else echo "city is missing"; fi
city is set
$ if jq -e '.orders[0].discount' orders.json >/dev/null; then echo "discount is set"; else echo "discount is missing"; fi
discount is missing

The session’s four jq runs showed the whole picture. jq '.orders[2]' orders.json printed null and exited 0: a plain run always exits 0 when it ran, regardless of what came out. jq -e '.orders[2]' orders.json on the same filter printed null and exited 1. jq -e '.orders[] | select(.status == "cancelled")' orders.json printed nothing and exited 4 (no value was ever produced). jq -e '.orders[0].id' orders.json printed "o1" and exited 0 (a real value). jq 'this is not jq' exited 3 with a syntax error, and printf 'not json\n' | jq . exited 5 with a parse error. Each exit number names a different failure, so a script that catches one does not catch another.

The -e flag is tailor-made for an if test. if jq -e '.orders[0].city' orders.json >/dev/null; then echo "city is set"; else echo "city is missing"; fi printed city is set; the same block on .orders[0].discount printed city is missing. The pipe discards jq’s output and keeps only its status, so the if reads just the yes/no. For a pipeline that ends with jq -e, pipelines-pipestatus-and-pipefail is the companion lesson: set -o pipefail makes the whole pipeline return the first non-zero status, so curl ... | jq -e .data.id reports a curl failure as well as a jq one, and ${PIPESTATUS[@]} names each stage.

Practise: pick the right jq for each ask

Try these at your own terminal, using an orders file like the one above. Each one is one or two jq commands.

1. From orders.json, print the ids of orders whose amount is strictly less than 500. Hint: select with a numeric comparison; -r to drop the quotes.
2. Print a totals-by-city report sorted by total descending. Hint: group_by(.city) | map(...) for the per-city object, then sort_by(-.total) to reverse.
3. Build a CSV with a header line and three data lines, each with id, city and amount. Hint: ["id","city","amount"], (.orders[] | [.id, .city, .amount]) | @csv.
4. From a shell variable limit=500, print the paid orders whose amount is at or above it, as a safe filter. Hint: --argjson limit "$limit" and select(.status == "paid" and .amount >= $limit).
5. In a script, exit 4 when .orders holds no paid order; exit 0 otherwise. Hint: jq -e 'first(.orders[] | select(.status == "paid"))' orders.json >/dev/null.

The questions below use jq programs you have not seen. Every answer rests on a filter or flag from the sections above; settle on yours before you read the explanation:

Interview questions

Warm-up: what does .[] do versus ., and when do you reach for each (fresher to 2–5 years)?

. is the identity filter: it emits its input unchanged, so jq '.' orders.json pretty-prints the whole object and nothing else. .[] is the array or object iterator: it emits each element of an array (or each value of an object) as its own value, so the next stage of a filter sees one element at a time instead of the whole collection. Reach for . when you only want jq to validate and pretty-print, or at the start of a filter where the input flows in as is. Reach for .[] whenever the next stage is “do something per item”: .orders[] | select(...), .orders[] | .id, or [.orders[] | .amount] | add to sum. The two forms combine: .orders[] is read as .orders | .[], and in the lesson’s session that pattern iterated the four orders in order. The manual’s “Array/Object Value Iterator” entry gives both forms side by side.

Before opening the answers, write down your own first move for each of the three scenarios the questions describe:

Key takeaways

• A jq filter is a chain of generators built from ., .key, .[], the pipe | and the comma ,; the flags -r, -c, -s and -n change how input and output are shaped.
• select filters a stream, map transforms an array, and group_by with sort_by and unique_by returns results in a deterministic order sorted by the key.
• Shell values reach jq through --arg name value (a string) or --argjson name JSON (a parsed value); pasting a shell variable into the filter text is a parse-reshape trap that fails silently.
• @csv, @tsv, @sh, @uri and jq -r make jq’s output shell-friendly; a bash loop reads a @tsv stream with while IFS=$'\t' read -r.
• jq’s exit status is 0 for a run that finished (and with -e for a last value that was neither null nor false), 1 for a last null or false value, 4 for no output, 3 for a compile error and 5 for invalid JSON input.