pipe

How to Pipe and Redirect Output in Bash

In Bash, piping and redirecting are essential concepts that allow you to manipulate and control the flow of data between commands and files. These features provide powerful ways to handle command output and input, making your workflows more efficient and flexible.

Here’s a guide to using pipes and redirects in Bash.

---

1. Redirecting Output

Redirecting output means sending the output of a command to a file or another destination instead of displaying it on the terminal.

Redirect Standard Output (> and >>)

• > (Overwrite): This operator redirects the output of a command to a file, overwriting the file if it exists.

  echo "Hello, World!" > output.txt
  

  • This command writes "Hello, World!" to output.txt. If the file already exists, its contents will be replaced.

• >> (Append): This operator appends the output to the end of an existing file.

  echo "New line" >> output.txt
  

  • This command appends "New line" to the end of output.txt without overwriting the existing contents.

Redirecting Standard Error (2> and 2>>)

Sometimes, a command will produce errors. You can redirect standard error (stderr) to a file, separate from regular output.

• 2> (Overwrite): Redirects standard error to a file, overwriting the file if it exists.

  ls non_existent_directory 2> error.log
  

  • This command tries to list a non-existent directory, and any error is written to error.log.

• 2>> (Append): Appends standard error to a file.

  ls non_existent_directory 2>> error.log
  

  • This command appends errors to error.log instead of overwriting it.

Redirecting Both Standard Output and Standard Error

To redirect both stdout (standard output) and stderr (standard error) to the same file, use the following syntax:

• Redirect both stdout and stderr to the same file: bash command > output.log 2>&1
  • This command writes both the regular output and errors from the command to output.log.

---

2. Piping Output

Piping allows you to send the output of one command as the input to another command. This is useful for chaining commands together, creating powerful command-line workflows.

| (Pipe) Operator

• Pipe (|): Sends the output of one command to another command. bash ls | grep "Documents"
  • This command lists the files and directories (ls), and pipes the output to grep, which filters and shows only lines containing "Documents."

Combining Pipes

You can chain multiple commands together using pipes:

cat file.txt | grep "search_term" | wc -l

• cat file.txt: Outputs the contents of file.txt.
• grep "search_term": Filters lines containing the word "search_term."
• wc -l: Counts the number of lines returned by grep.

This will output the number of lines in file.txt that contain "search_term."

---

3. Redirecting Input

In addition to redirecting output, you can redirect input. This means providing a file as input to a command rather than typing it manually.

< (Input Redirect)

• <: Redirects input from a file to a command. bash sort < input.txt
  • This command reads the contents of input.txt and sorts it.

<< (Here Document)

A here document allows you to provide multi-line input directly within a script or command line.

• <<: Used to input multiple lines to a command. bash cat << EOF Line 1 Line 2 Line 3 EOF
  • The command prints the input lines until the delimiter (EOF) is reached.

---

4. Using tee Command

The tee command reads from standard input and writes to both standard output (the terminal) and one or more files.

tee (Write to File and Standard Output)

• tee: Redirects output to a file while also displaying it on the terminal.

  echo "Hello, World!" | tee output.txt
  

  • This writes "Hello, World!" to both the terminal and output.txt.

• tee -a: Appends the output to the file, instead of overwriting it.

  echo "New line" | tee -a output.txt
  

  • This command appends "New line" to output.txt and also displays it on the terminal.

---

5. Using File Descriptors

In Bash, file descriptors represent open files. Standard input (stdin), standard output (stdout), and standard error (stderr) are associated with file descriptors 0, 1, and 2, respectively.

Redirecting Output to a File Using File Descriptors

You can explicitly reference file descriptors when redirecting input and output.

• Redirect stdout (1>):

  command 1> output.txt
  

  • This is equivalent to command > output.txt since stdout is file descriptor 1 by default.

• Redirect stderr (2>):

  command 2> error.log
  

• Redirect both stdout and stderr:

  command > output.txt 2>&1
  

---

6. Common Use Cases for Pipe and Redirection

Here are a few practical examples of how piping and redirection can be used in real-world scenarios:

Example 1: Count the Number of Files in a Directory

ls -1 | wc -l

• ls -1: Lists files one per line.
• wc -l: Counts the number of lines, which equals the number of files in the directory.

Example 2: Find a Word in a File and Save the Results

grep "error" logfile.txt > results.txt

• grep "error": Searches for the word "error" in logfile.txt.
• > results.txt: Redirects the output to results.txt.

Example 3: Show Disk Usage of Directories and Sort by Size

du -sh * | sort -h

• du -sh *: Displays the disk usage of directories/files in a human-readable format.
• sort -h: Sorts the output by size, with the smallest at the top.

---

7. Summary of Key Redirection and Piping Operators

Operator	Action
>	Redirects standard output to a file (overwrite)
>>	Redirects standard output to a file (append)
2>	Redirects standard error to a file (overwrite)
2>>	Redirects standard error to a file (append)
|	Pipes the output of one command to another command
<	Redirects input from a file to a command
<<	Here document: Allows multiple lines of input to a command
tee	Writes output to both the terminal and a file
2>&1	Redirects stderr to stdout (useful for combining both error and output)
&>	Redirects both stdout and stderr to the same file (in some shells)

---

Conclusion

Piping and redirecting output are fundamental features of the Bash shell. They allow you to efficiently manage and manipulate data in the terminal. By understanding how to use pipes (|), redirections (>, >>, 2>, <), and tools like tee, you can streamline your workflows and perform complex tasks more easily. These techniques are powerful tools that every Linux user and Bash script writer should master.