How to use the command 'uniq' (with examples)

The ‘uniq’ command in Unix-like operating systems is used to display or filter out the unique lines from a given input or file. It compares adjacent lines and eliminates duplicates, but it does not work across non-adjacent lines. Therefore, it is often used in combination with the ‘sort’ command, as ‘sort’ arranges similar lines together.

Use case 1: Display each line once

Code:

sort path/to/file | uniq

Motivation: This use case is helpful when we want to remove duplicate lines from a file and only display each line once.

Explanation:

• ‘sort path/to/file’ sorts the lines in the file alphabetically.
• ‘|’ is a pipe operator used to redirect the output of the ‘sort’ command to the ‘uniq’ command.
• ‘uniq’ filters out the duplicate lines from the sorted output and displays each line only once.

Example output:

apple
banana
grape
orange

Use case 2: Display only unique lines

Code:

sort path/to/file | uniq -u

Motivation: This use case is useful when we want to obtain only the unique lines from a sorted input or file, discarding any duplicates.

Explanation:

• ‘sort path/to/file’ sorts the lines in the file alphabetically.
• ‘|’ is a pipe operator used to redirect the output of the ‘sort’ command to the ‘uniq’ command.
• ‘-u’ is an option passed to the ‘uniq’ command, which causes it to display only unique lines.

Example output:

grape
orange

Use case 3: Display only duplicate lines

Code:

sort path/to/file | uniq -d

Motivation: This use case is beneficial when we want to identify and display only the duplicate lines from the sorted input or file, excluding the unique lines.

Explanation:

• ‘sort path/to/file’ sorts the lines in the file alphabetically.
• ‘|’ is a pipe operator used to redirect the output of the ‘sort’ command to the ‘uniq’ command.
• ‘-d’ is an option passed to the ‘uniq’ command, which causes it to display only duplicate lines.

Example output:

apple
banana

Use case 4: Display number of occurrences of each line along with that line

Code:

sort path/to/file | uniq -c

Motivation: This use case is useful when we want to count the occurrences of each line in a sorted file and display them along with the corresponding line.

Explanation:

• ‘sort path/to/file’ sorts the lines in the file alphabetically.
• ‘|’ is a pipe operator used to redirect the output of the ‘sort’ command to the ‘uniq’ command.
• ‘-c’ is an option passed to the ‘uniq’ command, which causes it to prefix each line with the count of occurrences.

Example output:

      2 apple
      1 banana
      1 grape
      1 orange

Use case 5: Display number of occurrences of each line, sorted by the most frequent

Code:

sort path/to/file | uniq -c | sort -nr

Motivation: This use case is helpful when we want to count the occurrences of each line in a sorted file, display them along with the corresponding line, and sort the output by the most frequent lines.

Explanation:

• ‘sort path/to/file’ sorts the lines in the file alphabetically.
• ‘|’ is a pipe operator used to redirect the output of the ‘sort’ command to the ‘uniq’ command.
• ‘-c’ is an option passed to the ‘uniq’ command, which causes it to prefix each line with the count of occurrences.
• ‘|’ is another pipe operator used to redirect the output of the ‘uniq’ command to the ‘sort’ command.
• ‘-nr’ is an option passed to the ‘sort’ command, which sorts the input numerically and in reverse order.

Example output:

      2 apple
      1 grape
      1 orange
      1 banana

Conclusion:

The ‘uniq’ command in Unix-like systems is a versatile tool for filtering and displaying unique or duplicate lines from a sorted input or file. By combining it with the ‘sort’ command, various use cases can be achieved, including removing duplicates, counting occurrences, and sorting the output. Understanding the different options of ‘uniq’ allows for efficient manipulation and analysis of data.