How to Use the Command 'jq' (with Examples)

jq is a powerful command-line JSON processor that allows users to interactively explore, filter, and manipulate JSON data in a flexible way. With its domain-specific language, jq enables rich data querying and transformation processes that are simple to integrate into shell scripts. Its diverse set of functionalities makes it indispensable for developers and data analysts working intensively with JSON data formats.

Use case 1: Execute a Specific Expression Only Using the jq Binary

Code:

jq '.' /path/to/file.json

Motivation for using the example:

Sometimes, one may want to quickly view the contents of a JSON file in a readable, formatted manner. This task is particularly common when dealing with JSON data that isn’t indented or neatly organized, making it difficult to interpret. Using jq in this manner provides a quick, efficient way to enhance readability without modifying the original content of the file.

Explanation for every argument given in the command:

• jq: Invokes the jq command, initializing the JSON processor.
• '.': A jq expression that represents the whole JSON document, thus instructing jq to print it.
• /path/to/file.json: The path to the JSON file that is to be processed and displayed in a clean format.

Example output:

{
  "name": "John Doe",
  "age": 30,
  "city": "New York"
}

Use case 2: Execute a Specific Script

Code:

cat path/to/file.json | jq --from-file path/to/script.jq

Motivation for using the example:

There are scenarios where the operations to be performed on a JSON file are too complex or extensive to fit conveniently into a single jq command. This is where putting jq expressions in a separate file becomes beneficial. It simplifies maintenance and the sharing of jq programs between users or systems, while also improving readability and organization of complex JSON processing tasks.

Explanation for every argument given in the command:

• cat path/to/file.json: This command reads out the JSON file’s content, preparing it for piping.
• |: This pipe symbol directs the output of the cat command into jq for further processing.
• jq --from-file path/to/script.jq: The --from-file option allows jq to read and execute jq expressions stored in a separate file pointed to by path/to/script.jq.

Example output:

Depends on the content of the jq script file. For example, if the script extracts names from a list of users, the output may be:

[
  "Alice",
  "Bob",
  "Charlie"
]

Use case 3: Pass Specific Arguments

Code:

cat path/to/file.json | jq --arg "name1" "value1" --arg "name2" "value2" '. + $ARGS.named'

Motivation for using the example:

Adding specific data to a JSON document on-the-fly without modifying the source file is frequently necessary. This approach allows dynamic adjustment of JSON data, enabling the injection of external values into jq expressions. It is particularly useful during data transformations in batch processing or automated workflows, where JSON data must be altered based on user input or environment-specific parameters.

Explanation for every argument given in the command:

• cat path/to/file.json: Retrieves the content of the JSON file to be processed.
• |: Directs the produced output into the jq command.
• jq --arg "name1" "value1": Specifies an argument to be used in the jq expression, by the name name1 with the value value1.
• --arg "name2" "value2": Specifies an additional argument, name2 with its corresponding value.
• '. + $ARGS.named': A jq expression that adds the named arguments as fields to the JSON object.

Example output:

If the original data was:

{
  "greeting": "Hello"
}

After execution, it could look like this:

{
  "greeting": "Hello",
  "name1": "value1",
  "name2": "value2"
}

Use case 4: Create New JSON Object via Old JSON Objects from Multiple Files

Code:

cat path/to/multiple_json_file_*.json | jq '{newKey1: .key1, newKey2: .key2.nestedKey}'

Motivation for using the example:

In situations where data needs to be extracted from multiple JSON files and consolidated into a single database object or structured differently, this method is ideal. It is especially relevant in data migrations or integrations where individual JSON records must be combined or transformed to fit into new systems or databases through automation.

Explanation for every argument given in the command:

• cat path/to/multiple_json_file_*.json: Reads all matching JSON files, aggregating their content into one stream.
• |: Passes the combined JSON data to jq for processing.
• jq '{newKey1: .key1, newKey2: .key2.nestedKey}': Constructs a new JSON object from existing ones, mapping old keys key1 and key2’s nestedKey to new keys newKey1 and newKey2 within the output.

Example output:

Given initial contents from the files could be:

{
  "key1": "Value1",
  "key2": { "nestedKey": "Value2" }
}

The output would be:

{
  "newKey1": "Value1",
  "newKey2": "Value2"
}

Use case 5: Print Specific Array Items

Code:

cat path/to/file.json | jq '.[index1], .[index2]'

Motivation for using the example:

When dealing with large JSON arrays where only specific elements are of interest, it is efficient to directly extract and display those elements. This selective approach minimizes data processing overhead and enhances clarity, particularly when only a subset of data is needed for visual inspection or debugging.

Explanation for every argument given in the command:

• cat path/to/file.json: Reads the JSON file content, outputting it for jq to process.
• |: Facilitates the movement of data from the cat command to jq.
• jq '.[index1], .[index2]': Extracts the array elements at index positions index1 and index2, printing them out sequentially.

Example output:

For a sample JSON array:

[
  "apple",
  "banana",
  "cherry",
  "date"
]

The command might output:

"banana"
"cherry"

Use case 6: Print All Array/Object Values

Code:

cat path/to/file.json | jq '.[]'

Motivation for using the example:

In several instances, it can be immensely helpful to scan through every element of a JSON array or list the values of all keys in an object to evaluate them individually. This is especially useful for visualizing items within structures that are more complex and necessitate a straightforward method for accessing each element without manually iterating through them.

Explanation for every argument given in the command:

• cat path/to/file.json: Outputs the full JSON content from the specified file.
• |: Directs the textual output into jq’s processing pipeline.
• jq '.[]': Indicates all elements in the current data structure (array/object) shall be printed.

Example output:

For an array like:

["red", "green", "blue"]

The output would list each color:

"red"
"green"
"blue"

Use case 7: Print Objects with 2-condition Filter in Array

Code:

cat path/to/file.json | jq '.[] | select((.key1=="value1") and .key2=="value2")'

Motivation for using the example:

When parsing through JSON data arrays, it is often necessary to extract specific elements that satisfy multiple conditions or criteria. Such a filtered view is particularly valuable for preparing reports or summaries from log data or user profiles, where only specific items meeting precise conditions are of interest.

Explanation for every argument given in the command:

• cat path/to/file.json: Retrieves the JSON data for further manipulation.
• |: Streams the output from cat into jq’s processing capabilities.
• jq '.[] | select((.key1=="value1") and .key2=="value2")': Examines each object within the array, applying a filter via select() to only include those matching key1 as “value1” and key2 as “value2”.

Example output:

Given data such as:

[
  {"key1": "value1", "key2": "value2"},
  {"key1": "value3", "key2": "value4"}
]

It would result in:

{
  "key1": "value1",
  "key2": "value2"
}

Use case 8: Add/Remove Specific Keys

Code:

cat path/to/file.json | jq '. +|- {"key1": "value1", "key2": "value2"}'

Motivation for using the example:

There are many situations where data needs modification at the key level, specifically adding or removing certain keys and their associated data from JSON objects. Such cases are commonly observed when reconciling data entry forms with updates or deletions, or even adapting schemas between multiple APIs or databases.

Explanation for every argument given in the command:

• cat path/to/file.json: Outputs content from the given JSON file.
• |: Funnels data through to jq for operation.
• jq '. +|- {"key1": "value1", "key2": "value2"}': The + operator adds the specified keys and values to the JSON object, whereas - would remove keys designated in the following JSON object syntax.

Example output:

Starting with:

{
  "existingKey": "existingValue"
}

Adding could result in:

{
  "existingKey": "existingValue",
  "key1": "value1",
  "key2": "value2"
}

Alternatively, if removing:

{
  "existingKey": "existingValue"
}

Would remain unchanged, given no specified key is originally present in the object.

Conclusion:

jq is an incredibly versatile tool for anyone working with JSON data. Its commands, while initially appearing complex due to their DSL, provide powerful methods for parsing, querying, and manipulating JSON structures both interactively and programmatically. By mastering various jq examples, developers and analysts can significantly streamline their data processing workflows, allowing them to focus more effectively on decision-making and analysis.