Command Line Interface

How to Use the find Command to Search Files in Linux

The find command is one of the most powerful and versatile tools in Linux for searching files and directories. It allows you to locate files based on various criteria such as name, size, permissions, time of modification, and more. Here’s a guide on how to use the find command effectively.

---

1. Basic Syntax of the find Command

The basic syntax for the find command is:

find [path] [expression]

• path: The directory (or directories) where you want to search. You can specify one or more directories, or use . to search in the current directory.
• expression: The conditions or filters you want to apply (e.g., file name, size, type).

---

2. Searching by File Name

To search for a file by its name, use the -name option. The search is case-sensitive by default.

Case-Sensitive Search

find /path/to/search -name "filename.txt"

This command searches for filename.txt in the specified directory and its subdirectories.

Case-Insensitive Search

To make the search case-insensitive, use -iname:

find /path/to/search -iname "filename.txt"

This will match files like Filename.txt, FILENAME.TXT, etc.

Using Wildcards in Name Search

You can use wildcards (*, ?, etc.) to match patterns: - *: Matches any sequence of characters. - ?: Matches a single character.

For example, to search for all .txt files:

find /path/to/search -name "*.txt"

---

3. Searching by File Type

The find command allows you to filter files based on their type. The -type option can be used to specify the following types:

• f: Regular file
• d: Directory
• l: Symbolic link
• s: Socket
• p: Named pipe (FIFO)
• c: Character device
• b: Block device

Search for Regular Files

find /path/to/search -type f

This command finds all regular files in the specified directory and its subdirectories.

Search for Directories

find /path/to/search -type d

This command finds all directories.

---

4. Searching by File Size

You can search for files based on their size using the -size option. Sizes can be specified in various units:

• b: 512-byte blocks (default)
• c: Bytes
• k: Kilobytes
• M: Megabytes
• G: Gigabytes

Find Files of a Specific Size

• Exact size:

  find /path/to/search -size 100M
  

  This finds files that are exactly 100 MB in size.

• Greater than a size:

  find /path/to/search -size +100M
  

  This finds files greater than 100 MB.

• Less than a size:

  find /path/to/search -size -100M
  

  This finds files smaller than 100 MB.

---

5. Searching by Modification Time

The find command allows you to search for files based on when they were last modified. The -mtime option specifies the modification time in days:

• -mtime +n: Files modified more than n days ago.
• -mtime -n: Files modified less than n days ago.
• -mtime n: Files modified exactly n days ago.

Find Files Modified Within the Last 7 Days

find /path/to/search -mtime -7

Find Files Not Modified in the Last 30 Days

find /path/to/search -mtime +30

Find Files Modified Exactly 1 Day Ago

find /path/to/search -mtime 1

---

6. Searching by Permissions

You can search for files based on their permissions using the -perm option.

Find Files with Specific Permissions

For example, to find files with 777 permissions:

find /path/to/search -perm 0777

Find Files with at Least Specific Permissions

To find files that have at least rw-r--r-- permissions, use the - before the permission value:

find /path/to/search -perm -644

Find Files with Specific Permissions for User, Group, or Others

You can also use symbolic notation to search for files with specific permissions for the user (u), group (g), or others (o). For example:

find /path/to/search -perm /u+x

This finds files that have the executable permission for the user.

---

7. Searching by File Owner

The -user option allows you to find files owned by a specific user.

Find Files Owned by a Specific User

find /path/to/search -user username

Find Files Owned by a Specific Group

Similarly, use the -group option to search for files owned by a specific group:

find /path/to/search -group groupname

---

8. Executing Commands on Search Results

You can use the -exec option to perform actions on the files that match your search criteria. The {} placeholder represents the current file.

Example: Delete All .log Files

find /path/to/search -name "*.log" -exec rm -f {} \;

This command finds all .log files and deletes them.

Example: Display the File Details

find /path/to/search -name "*.txt" -exec ls -l {} \;

This command lists the details (using ls -l) of each .txt file found.

Note: The \; at the end is required to terminate the -exec action.

---

9. Using find with xargs for Efficiency

When executing commands on large numbers of files, xargs is often more efficient than -exec, because it minimizes the number of times the command is run.

Example: Delete Files Using xargs

find /path/to/search -name "*.log" | xargs rm -f

This command finds all .log files and passes the list of files to rm using xargs.

---

10. Combining Multiple Conditions

You can combine multiple search conditions using logical operators like -and, -or, and -not.

Example: Find Files Larger Than 10MB and Modified in the Last 7 Days

find /path/to/search -size +10M -and -mtime -7

Example: Find Files That Are Not Directories

find /path/to/search -not -type d

---

11. Limiting Search Depth with -maxdepth

The -maxdepth option restricts the depth of directories find will search into.

Example: Search Only in the Top-Level Directory

find /path/to/search -maxdepth 1 -name "*.txt"

This will find .txt files only in the top-level directory (/path/to/search), not in subdirectories.

---

12. Summary of Useful find Command Options

Option	Description
-name	Search by file name
-iname	Search by file name (case-insensitive)
-type	Search by file type (f = file, d = directory, l = symlink)
-size	Search by file size
-mtime	Search by modification time (in days)
-user	Search by file owner
-group	Search by file group
-perm	Search by file permissions
-exec	Execute a command on each found file
-not	Negate a condition
-and / -or	Combine multiple conditions (default is -and)
-maxdepth	Limit the depth of directory traversal
-mindepth	Limit the minimum depth of directory traversal

---

Conclusion

The find command is an indispensable tool for searching and managing files in Linux. With its wide range of options, you can tailor your search to meet almost any criteria. Whether you're looking for files by name, size, type, or modification time, find can help you locate exactly what you need, and its ability to execute commands on the results makes it incredibly powerful for automating tasks.

How to Pipe and Redirect Output in Bash

In Bash, piping and redirecting are essential concepts that allow you to manipulate and control the flow of data between commands and files. These features provide powerful ways to handle command output and input, making your workflows more efficient and flexible.

Here’s a guide to using pipes and redirects in Bash.

---

1. Redirecting Output

Redirecting output means sending the output of a command to a file or another destination instead of displaying it on the terminal.

Redirect Standard Output (> and >>)

• > (Overwrite): This operator redirects the output of a command to a file, overwriting the file if it exists.

  echo "Hello, World!" > output.txt
  

  • This command writes "Hello, World!" to output.txt. If the file already exists, its contents will be replaced.

• >> (Append): This operator appends the output to the end of an existing file.

  echo "New line" >> output.txt
  

  • This command appends "New line" to the end of output.txt without overwriting the existing contents.

Redirecting Standard Error (2> and 2>>)

Sometimes, a command will produce errors. You can redirect standard error (stderr) to a file, separate from regular output.

• 2> (Overwrite): Redirects standard error to a file, overwriting the file if it exists.

  ls non_existent_directory 2> error.log
  

  • This command tries to list a non-existent directory, and any error is written to error.log.

• 2>> (Append): Appends standard error to a file.

  ls non_existent_directory 2>> error.log
  

  • This command appends errors to error.log instead of overwriting it.

Redirecting Both Standard Output and Standard Error

To redirect both stdout (standard output) and stderr (standard error) to the same file, use the following syntax:

• Redirect both stdout and stderr to the same file: bash command > output.log 2>&1
  • This command writes both the regular output and errors from the command to output.log.

---

2. Piping Output

Piping allows you to send the output of one command as the input to another command. This is useful for chaining commands together, creating powerful command-line workflows.

| (Pipe) Operator

• Pipe (|): Sends the output of one command to another command. bash ls | grep "Documents"
  • This command lists the files and directories (ls), and pipes the output to grep, which filters and shows only lines containing "Documents."

Combining Pipes

You can chain multiple commands together using pipes:

cat file.txt | grep "search_term" | wc -l

• cat file.txt: Outputs the contents of file.txt.
• grep "search_term": Filters lines containing the word "search_term."
• wc -l: Counts the number of lines returned by grep.

This will output the number of lines in file.txt that contain "search_term."

---

3. Redirecting Input

In addition to redirecting output, you can redirect input. This means providing a file as input to a command rather than typing it manually.

< (Input Redirect)

• <: Redirects input from a file to a command. bash sort < input.txt
  • This command reads the contents of input.txt and sorts it.

<< (Here Document)

A here document allows you to provide multi-line input directly within a script or command line.

• <<: Used to input multiple lines to a command. bash cat << EOF Line 1 Line 2 Line 3 EOF
  • The command prints the input lines until the delimiter (EOF) is reached.

---

4. Using tee Command

The tee command reads from standard input and writes to both standard output (the terminal) and one or more files.

tee (Write to File and Standard Output)

• tee: Redirects output to a file while also displaying it on the terminal.

  echo "Hello, World!" | tee output.txt
  

  • This writes "Hello, World!" to both the terminal and output.txt.

• tee -a: Appends the output to the file, instead of overwriting it.

  echo "New line" | tee -a output.txt
  

  • This command appends "New line" to output.txt and also displays it on the terminal.

---

5. Using File Descriptors

In Bash, file descriptors represent open files. Standard input (stdin), standard output (stdout), and standard error (stderr) are associated with file descriptors 0, 1, and 2, respectively.

Redirecting Output to a File Using File Descriptors

You can explicitly reference file descriptors when redirecting input and output.

• Redirect stdout (1>):

  command 1> output.txt
  

  • This is equivalent to command > output.txt since stdout is file descriptor 1 by default.

• Redirect stderr (2>):

  command 2> error.log
  

• Redirect both stdout and stderr:

  command > output.txt 2>&1
  

---

6. Common Use Cases for Pipe and Redirection

Here are a few practical examples of how piping and redirection can be used in real-world scenarios:

Example 1: Count the Number of Files in a Directory

ls -1 | wc -l

• ls -1: Lists files one per line.
• wc -l: Counts the number of lines, which equals the number of files in the directory.

Example 2: Find a Word in a File and Save the Results

grep "error" logfile.txt > results.txt

• grep "error": Searches for the word "error" in logfile.txt.
• > results.txt: Redirects the output to results.txt.

Example 3: Show Disk Usage of Directories and Sort by Size

du -sh * | sort -h

• du -sh *: Displays the disk usage of directories/files in a human-readable format.
• sort -h: Sorts the output by size, with the smallest at the top.

---

7. Summary of Key Redirection and Piping Operators

Operator	Action
>	Redirects standard output to a file (overwrite)
>>	Redirects standard output to a file (append)
2>	Redirects standard error to a file (overwrite)
2>>	Redirects standard error to a file (append)
|	Pipes the output of one command to another command
<	Redirects input from a file to a command
<<	Here document: Allows multiple lines of input to a command
tee	Writes output to both the terminal and a file
2>&1	Redirects stderr to stdout (useful for combining both error and output)
&>	Redirects both stdout and stderr to the same file (in some shells)

---

Conclusion

Piping and redirecting output are fundamental features of the Bash shell. They allow you to efficiently manage and manipulate data in the terminal. By understanding how to use pipes (|), redirections (>, >>, 2>, <), and tools like tee, you can streamline your workflows and perform complex tasks more easily. These techniques are powerful tools that every Linux user and Bash script writer should master.

How to Use the find Command to Search Files in Linux

The find command is one of the most powerful and versatile tools in Linux for searching files and directories. It allows you to locate files based on various criteria such as name, size, permissions, time of modification, and more. Here’s a guide on how to use the find command effectively.

---

1. Basic Syntax of the find Command

The basic syntax for the find command is:

find [path] [expression]

• path: The directory (or directories) where you want to search. You can specify one or more directories, or use . to search in the current directory.
• expression: The conditions or filters you want to apply (e.g., file name, size, type).

---

2. Searching by File Name

To search for a file by its name, use the -name option. The search is case-sensitive by default.

Case-Sensitive Search

find /path/to/search -name "filename.txt"

This command searches for filename.txt in the specified directory and its subdirectories.

Case-Insensitive Search

To make the search case-insensitive, use -iname:

find /path/to/search -iname "filename.txt"

This will match files like Filename.txt, FILENAME.TXT, etc.

Using Wildcards in Name Search

You can use wildcards (*, ?, etc.) to match patterns: - *: Matches any sequence of characters. - ?: Matches a single character.

For example, to search for all .txt files:

find /path/to/search -name "*.txt"

---

3. Searching by File Type

The find command allows you to filter files based on their type. The -type option can be used to specify the following types:

• f: Regular file
• d: Directory
• l: Symbolic link
• s: Socket
• p: Named pipe (FIFO)
• c: Character device
• b: Block device

Search for Regular Files

find /path/to/search -type f

This command finds all regular files in the specified directory and its subdirectories.

Search for Directories

find /path/to/search -type d

This command finds all directories.

---

4. Searching by File Size

You can search for files based on their size using the -size option. Sizes can be specified in various units:

• b: 512-byte blocks (default)
• c: Bytes
• k: Kilobytes
• M: Megabytes
• G: Gigabytes

Find Files of a Specific Size

• Exact size:

  find /path/to/search -size 100M
  

  This finds files that are exactly 100 MB in size.

• Greater than a size:

  find /path/to/search -size +100M
  

  This finds files greater than 100 MB.

• Less than a size:

  find /path/to/search -size -100M
  

  This finds files smaller than 100 MB.

---

5. Searching by Modification Time

The find command allows you to search for files based on when they were last modified. The -mtime option specifies the modification time in days:

• -mtime +n: Files modified more than n days ago.
• -mtime -n: Files modified less than n days ago.
• -mtime n: Files modified exactly n days ago.

Find Files Modified Within the Last 7 Days

find /path/to/search -mtime -7

Find Files Not Modified in the Last 30 Days

find /path/to/search -mtime +30

Find Files Modified Exactly 1 Day Ago

find /path/to/search -mtime 1

---

6. Searching by Permissions

You can search for files based on their permissions using the -perm option.

Find Files with Specific Permissions

For example, to find files with 777 permissions:

find /path/to/search -perm 0777

Find Files with at Least Specific Permissions

To find files that have at least rw-r--r-- permissions, use the - before the permission value:

find /path/to/search -perm -644

Find Files with Specific Permissions for User, Group, or Others

You can also use symbolic notation to search for files with specific permissions for the user (u), group (g), or others (o). For example:

find /path/to/search -perm /u+x

This finds files that have the executable permission for the user.

---

7. Searching by File Owner

The -user option allows you to find files owned by a specific user.

Find Files Owned by a Specific User

find /path/to/search -user username

Find Files Owned by a Specific Group

Similarly, use the -group option to search for files owned by a specific group:

find /path/to/search -group groupname

---

8. Executing Commands on Search Results

You can use the -exec option to perform actions on the files that match your search criteria. The {} placeholder represents the current file.

Example: Delete All .log Files

find /path/to/search -name "*.log" -exec rm -f {} \;

This command finds all .log files and deletes them.

Example: Display the File Details

find /path/to/search -name "*.txt" -exec ls -l {} \;

This command lists the details (using ls -l) of each .txt file found.

Note: The \; at the end is required to terminate the -exec action.

---

9. Using find with xargs for Efficiency

When executing commands on large numbers of files, xargs is often more efficient than -exec, because it minimizes the number of times the command is run.

Example: Delete Files Using xargs

find /path/to/search -name "*.log" | xargs rm -f

This command finds all .log files and passes the list of files to rm using xargs.

---

10. Combining Multiple Conditions

You can combine multiple search conditions using logical operators like -and, -or, and -not.

Example: Find Files Larger Than 10MB and Modified in the Last 7 Days

find /path/to/search -size +10M -and -mtime -7

Example: Find Files That Are Not Directories

find /path/to/search -not -type d

---

11. Limiting Search Depth with -maxdepth

The -maxdepth option restricts the depth of directories find will search into.

Example: Search Only in the Top-Level Directory

find /path/to/search -maxdepth 1 -name "*.txt"

This will find .txt files only in the top-level directory (/path/to/search), not in subdirectories.

---

12. Summary of Useful find Command Options

Option	Description
-name	Search by file name
-iname	Search by file name (case-insensitive)
-type	Search by file type (f = file, d = directory, l = symlink)
-size	Search by file size
-mtime	Search by modification time (in days)
-user	Search by file owner
-group	Search by file group
-perm	Search by file permissions
-exec	Execute a command on each found file
-not	Negate a condition
-and / -or	Combine multiple conditions (default is -and)
-maxdepth	Limit the depth of directory traversal
-mindepth	Limit the minimum depth of directory traversal

---

Conclusion

The find command is an indispensable tool for searching and managing files in Linux. With its wide range of options, you can tailor your search to meet almost any criteria. Whether you're looking for files by name, size, type, or modification time, find can help you locate exactly what you need, and its ability to execute commands on the results makes it incredibly powerful for automating tasks.

Understanding the Bash Shell's History Feature

The Bash shell provides a history feature that records commands entered during previous sessions. This allows you to quickly recall, reuse, and manipulate commands from the past without having to type them again. The history feature is incredibly useful for streamlining your work in the terminal and for quickly repeating or modifying past commands.

---

1. What is Bash History?

The Bash history refers to the list of commands that have been executed in the terminal. These commands are stored in a history file, which by default is located in the user's home directory as .bash_history.

• Location of history file:
  • ~/.bash_history

This file stores the commands you enter, allowing you to recall or search them later. Bash automatically updates this history file as you exit a session or periodically during the session, depending on settings.

---

2. Basic History Commands

Here are some basic commands to interact with your Bash history:

2.1. Viewing Command History

To view the history of commands you've previously executed, use the history command:

history

This will display a list of previously executed commands with a number beside each command. You can also specify how many commands to display:

history 20  # Show the last 20 commands

2.2. Recall a Previous Command

To quickly execute a command from your history, you can use the history expansion feature. Here are a few ways to recall commands:

• Recall the last command:

  !!
  

  This will execute the last command you ran.

• Recall a specific command by number: Each command in the history list has a number. To run a command from history, use:

  !number
  

  For example, if history shows:

  1 ls -l
  2 cat file.txt
  

  You can run the cat file.txt command again by typing:

  !2
  

2.3. Search the History

You can search through your command history using reverse search:

• Press Ctrl + r and start typing part of the command you want to search for. Bash will automatically search for commands matching the text you're typing.
• Press Ctrl + r repeatedly to cycle through previous matches.

To cancel the search, press Ctrl + g.

2.4. Clear the History

To clear your Bash history for the current session:

history -c

This will clear the history stored in memory for the current session but will not delete the .bash_history file.

If you want to delete the history file entirely, use:

rm ~/.bash_history

However, the history will not be permanently cleared until the session ends and Bash writes the history to the .bash_history file.

---

3. Configuring Bash History

Bash provides several configuration options for how history is handled. These are set in the ~/.bashrc file or /etc/bash.bashrc for system-wide settings. Here are some useful environment variables for configuring history behavior:

3.1. Setting the History File

By default, Bash saves history in ~/.bash_history. You can change this by modifying the HISTFILE variable:

export HISTFILE=~/.my_custom_history

3.2. History Size

You can configure how many commands are stored in the history file and the in-memory history list.

• HISTSIZE: Specifies the number of commands to store in memory during the current session.
• HISTFILESIZE: Specifies the number of commands to store in the .bash_history file.

To set the history size to 1000 commands:

export HISTSIZE=1000
export HISTFILESIZE=1000

3.3. Ignoring Duplicate and Repeated Commands

You can prevent duplicate commands from being saved in the history file by setting HISTCONTROL. Here are some useful options: - ignoredups: Ignores commands that are identical to the previous one. - ignorespace: Ignores commands that start with a space.

Example:

export HISTCONTROL=ignoredups:ignorespace

This will prevent duplicate commands and commands starting with a space from being saved to history.

3.4. Appending to History

By default, Bash overwrites the history file when the session ends. To make sure new commands are appended to the history file (instead of overwriting it), set the shopt option:

shopt -s histappend

This ensures that when a new session starts, the history is appended to the existing file, rather than replacing it.

3.5. Timestamping History Commands

If you want to save the timestamp for each command, you can set the HISTTIMEFORMAT variable:

export HISTTIMEFORMAT="%F %T "

This will add the date and time before each command in the history file (e.g., 2024-12-20 14:15:34 ls -l).

---

4. Using the History with Scripting

You can also use the history feature within scripts. For instance, to extract a specific command from history in a script, you can use history with the grep command:

history | grep "some_command"

This is useful when you need to look back and automate the execution of previously used commands in scripts.

---

5. Advanced History Expansion

Bash offers some advanced features for working with history. One of the key features is history expansion, which allows you to reference and manipulate previous commands. Some common history expansions include:

• !!: Repeat the last command.
• !$: Use the last argument of the previous command.
• !n: Execute the command with history number n.
• !string: Run the most recent command that starts with string.

Example:

$ echo "hello"
hello
$ !echo  # Repeats the last echo command
echo hello

---

6. Best Practices for Using Bash History

• Security Considerations: Be mindful that sensitive information (like passwords or API keys) entered in the terminal can be saved in the history file. Avoid typing sensitive data directly, or configure the history to ignore such commands using HISTCONTROL and ignoredups.

• Efficiency: Use the history search (Ctrl + r) to quickly find previously executed commands. This can significantly speed up repetitive tasks.

• Customization: Tweak history behavior (such as setting HISTCONTROL to avoid saving certain commands) to improve your workflow and avoid unnecessary clutter in your history file.

---

Conclusion

The Bash history feature is a powerful tool that makes working with the terminal more efficient by allowing you to recall and reuse previously executed commands. By understanding the different history commands and configuring Bash to suit your workflow, you can significantly improve your productivity. Whether you’re repeating common tasks, searching for past commands, or scripting, mastering the history feature will make you more effective in using Bash.

Working with Environment Variables in Bash

Environment variables in Bash are variables that define the environment in which processes run. They store system-wide values like system paths, configuration settings, and user-specific data, and can be accessed or modified within a Bash session.

Environment variables are essential for: - Configuring system settings. - Customizing the behavior of scripts and programs. - Storing configuration values for users and applications.

Here’s an overview of how to work with environment variables in Bash.

---

1. Viewing Environment Variables

To see all the current environment variables, use the env or printenv command:

env

or

printenv

This will print a list of all environment variables and their values.

To view a specific variable:

echo $VARIABLE_NAME

Example:

echo $HOME

This will display the path to the home directory of the current user.

---

2. Setting Environment Variables

To set an environment variable, you can use the export command. This makes the variable available to any child processes or scripts launched from the current session.

Syntax:

export VARIABLE_NAME="value"

Example:

export MY_VAR="Hello, World!"

Now, the MY_VAR variable is available to the current session and any child processes.

To check its value:

echo $MY_VAR

---

3. Unsetting Environment Variables

If you no longer need an environment variable, you can remove it with the unset command.

Syntax:

unset VARIABLE_NAME

Example:

unset MY_VAR

After running this command, MY_VAR will no longer be available in the session.

---

4. Temporary vs. Permanent Environment Variables

Temporary Environment Variables:

Environment variables set using export are only valid for the duration of the current shell session. Once the session ends, the variable will be lost.

Permanent Environment Variables:

To set an environment variable permanently (so that it persists across sessions), you need to add it to one of the shell initialization files, such as: - ~/.bashrc for user-specific variables in interactive non-login shells. - ~/.bash_profile or ~/.profile for login shells.

Example: Setting a permanent variable

1. Open the file for editing (e.g., ~/.bashrc):

   nano ~/.bashrc
   

2. Add the export command at the end of the file:

   export MY_VAR="Permanent Value"
   

3. Save the file and reload it:

   source ~/.bashrc
   

Now, MY_VAR will be available every time a new shell is started.

---

5. Common Environment Variables

Here are some common environment variables you’ll encounter:

• $HOME: The home directory of the current user.
• $USER: The username of the current user.
• $PATH: A colon-separated list of directories where executable files are located.
• $PWD: The current working directory.
• $SHELL: The path to the current shell.
• $EDITOR: The default text editor (e.g., nano, vim).

Example:

echo $PATH

This will print the directories that are included in your executable search path.

---

6. Using Environment Variables in Scripts

Environment variables can be used within Bash scripts to customize behavior or store settings.

Example Script:

#!/bin/bash

# Use an environment variable in the script
echo "Hello, $USER! Your home directory is $HOME"

You can also pass variables into scripts when you run them:

MY_VAR="Some Value" ./myscript.sh

Inside myscript.sh, you can access $MY_VAR as if it were set in the environment.

---

7. Modifying $PATH

The $PATH variable is a crucial environment variable that defines the directories the shell searches for executable files. If you install new software or custom scripts, you may want to add their location to $PATH.

Example: Adding a directory to $PATH

export PATH=$PATH:/path/to/my/custom/bin

This command appends /path/to/my/custom/bin to the existing $PATH.

To make this change permanent, add the export command to your ~/.bashrc or ~/.bash_profile.

---

8. Environment Variables in Subshells

When you open a new subshell (e.g., by running a script or launching another terminal), the environment variables are inherited from the parent shell. However, changes to environment variables in a subshell will not affect the parent shell.

For example:

export MY_VAR="New Value"
bash  # Open a new subshell
echo $MY_VAR  # This will show "New Value" in the subshell
exit
echo $MY_VAR  # The parent shell's $MY_VAR is unaffected

---

9. Example of Using Multiple Environment Variables in a Script

#!/bin/bash

# Setting multiple environment variables
export DB_USER="admin"
export DB_PASSWORD="secret"

# Using the variables
echo "Connecting to the database with user $DB_USER..."
# Here, you would use these variables in a real script to connect to a database, for example.

---

Conclusion

Working with environment variables in Bash is a key part of managing system configuration and making your scripts flexible and portable. By using commands like export, echo, and unset, you can configure, view, and manage variables both temporarily and permanently. Mastering environment variables will help you manage your Linux environment more effectively, automate tasks, and write more dynamic Bash scripts.

Top 10 Bash Commands Every New Linux User Should Learn

If you're new to Linux and Bash, learning some essential commands is the best way to start. These commands will help you navigate the system, manage files, and perform basic tasks. Here’s a list of the top 10 commands every new Linux user should master:

---

1. ls – List Files and Directories

The ls command displays the contents of a directory.

• Basic usage: bash ls
• Common options:
  • ls -l: Long listing format (shows details like permissions and file sizes).
  • ls -a: Includes hidden files.
  • ls -lh: Displays file sizes in human-readable format.

---

2. cd – Change Directory

Navigate through the file system with the cd command.

• Basic usage: bash cd /path/to/directory
• Tips:
  • cd ..: Move up one directory.
  • cd ~: Go to your home directory.
  • cd -: Switch to the previous directory.

---

3. pwd – Print Working Directory

The pwd command shows the current directory you're working in.

• Usage: bash pwd

---

4. touch – Create a New File

The touch command creates empty files.

• Basic usage: bash touch filename.txt

---

5. cp – Copy Files and Directories

Use cp to copy files or directories.

• Basic usage: bash cp source_file destination_file
• Copy directories: bash cp -r source_directory destination_directory

---

6. mv – Move or Rename Files

The mv command moves or renames files and directories.

• Move a file: bash mv file.txt /new/location/
• Rename a file: bash mv old_name.txt new_name.txt

---

7. rm – Remove Files and Directories

The rm command deletes files and directories.

• Basic usage: bash rm file.txt
• Delete directories: bash rm -r directory_name
• Important: Be cautious with rm as it permanently deletes files.

---

8. mkdir – Create Directories

The mkdir command creates new directories.

• Basic usage: bash mkdir new_directory
• Create parent directories: bash mkdir -p parent/child/grandchild

---

9. cat – View File Content

The cat command displays the content of a file.

• Basic usage: bash cat file.txt
• Combine files: bash cat file1.txt file2.txt > combined.txt

---

10. man – View Command Documentation

The man command shows the manual page for a given command.

• Usage: bash man command_name
• Example: bash man ls

---

Bonus Commands

• echo: Prints text to the terminal or a file. bash echo "Hello, World!"
• grep: Searches for patterns in text files. bash grep "search_term" file.txt
• sudo: Runs commands with superuser privileges. bash sudo apt update

---

Tips for Learning Bash Commands

1. Practice regularly: The more you use these commands, the easier they will become.
2. Explore options: Many commands have useful flags; use man to learn about them.
3. Be cautious with destructive commands: Commands like rm and sudo can have significant consequences.

With these commands, you'll be well on your way to mastering Linux and Bash!

Creating and Using Bash Aliases for Faster Commands

A Bash alias is a shortcut for a longer command or a sequence of commands. Aliases help improve productivity by saving time and effort. Here's a guide to creating and using Bash aliases:

---

1. Temporary Aliases

Temporary aliases are created in the current shell session and last until the terminal is closed.

Syntax:

alias alias_name='command'

Examples:

• Create a short alias for listing files: bash alias ll='ls -al'
• Create an alias to navigate to a frequently used directory: bash alias docs='cd ~/Documents'
• Create an alias to remove files without confirmation: bash alias rmf='rm -rf'

Using Temporary Aliases:

Once created, type the alias name to execute the command:

ll    # Equivalent to 'ls -al'
docs  # Changes directory to ~/Documents

---

2. Permanent Aliases

To make aliases persist across sessions, add them to your shell's configuration file. The most common file is ~/.bashrc, but it could also be ~/.bash_profile or another file depending on your system setup.

Steps to Create Permanent Aliases:

1. Open your shell configuration file: bash vi ~/.bashrc
2. Add the alias definition at the end of the file: bash alias ll='ls -al' alias docs='cd ~/Documents' alias gs='git status'
3. Save and exit the file.
4. Reload the configuration file to apply changes: bash source ~/.bashrc

---

3. Viewing Existing Aliases

To see all active aliases in the current shell session, use:

alias

If you want to check the definition of a specific alias:

alias alias_name

Example:

alias ll
# Output: alias ll='ls -al'

---

4. Removing an Alias

To remove a temporary alias in the current session, use:

unalias alias_name

Example:

unalias ll

To remove a permanent alias, delete its definition from ~/.bashrc and reload the configuration:

vi ~/.bashrc
# Delete the alias definition, then:
source ~/.bashrc

---

5. Advanced Alias Tips

• Use Parameters with Functions:
  If you need an alias that accepts arguments, use a shell function instead:

  myfunction() {
  ls -l "$1"
  }
  alias ll='myfunction'
  

• Chaining Commands in Aliases:
  Combine multiple commands using && or ;:

  alias update='sudo apt update && sudo apt upgrade -y'
  

• Conditional Aliases:
  Add logic to aliases by wrapping them in functions:

  alias checkdisk='df -h && du -sh *'
  

---

6. Examples of Useful Aliases

• Simplify ls commands: bash alias l='ls -CF' alias la='ls -A' alias ll='ls -alF'
• Git shortcuts: bash alias gs='git status' alias ga='git add .' alias gc='git commit -m' alias gp='git push'
• Networking: bash alias myip='curl ifconfig.me' alias pingg='ping google.com'
• Custom cleanup command: bash alias clean='rm -rf ~/.cache/* && sudo apt autoremove -y'

---

Conclusion

Using aliases can greatly speed up your workflow by reducing repetitive typing. Start with simple aliases for your most-used commands and progressively add more as you identify opportunities to save time. With permanent aliases, you’ll have a customized environment that boosts efficiency every time you open the terminal.

How to Customise Your Bash Prompt for Better Productivity

The Bash prompt is the text that appears in your terminal before you type a command. By default, it displays minimal information, such as your username and current directory. Customizing your Bash prompt can enhance productivity by providing quick access to important information and making your terminal visually appealing.

---

What is the Bash Prompt?

The Bash prompt is controlled by the PS1 variable, which defines its appearance. For example:

PS1="\u@\h:\w\$ "

• \u: Username.
• \h: Hostname.
• \w: Current working directory.
• \$: Displays $ for normal users and # for the root user.

---

Why Customise Your Bash Prompt?

1. Enhanced Information: Display details like the current Git branch, exit status of the last command, or time.
2. Improved Visuals: Use colors and formatting to make the prompt easier to read.
3. Increased Productivity: Quickly identify useful information without typing additional commands.

---

Steps to Customise Your Bash Prompt

1. Temporary Customization

You can modify your prompt temporarily by setting the PS1 variable:

PS1="\[\e[32m\]\u@\h:\w\$\[\e[0m\] "

This changes the prompt to green text with the username, hostname, and working directory.

2. Permanent Customisation

To make changes permanent:

1. Edit the .bashrc file in your home directory: bash vi ~/.bashrc
2. Add your custom PS1 line at the end of the file.
3. Save and reload the file: bash source ~/.bashrc

---

Common Customizations

Add Colors

Use ANSI escape codes for colors:

• \[\e[31m\]: Red

• \[\e[32m\]: Green

• \[\e[34m\]: Blue

• \[\e[0m\]: Reset to default color.

Example:

PS1="\[\e[34m\]\u@\h:\[\e[32m\]\w\$\[\e[0m\] "

This makes the username and hostname blue and the working directory green.

Include Time

Display the current time:

PS1="\[\e[33m\]\t \[\e[34m\]\u@\h:\w\$ \[\e[0m\]"

• \t: Time in HH:MM:SS format.

• \@: Time in 12-hour AM/PM format.

Show Git Branch

Display the current Git branch when inside a repository:

PS1="\[\e[32m\]\u@\h:\[\e[34m\]\w\[\e[31m\]\$(__git_ps1)\[\e[0m\] \$ "

• Ensure you have Git installed and the git-prompt.sh script sourced in your .bashrc file.

Add Command Exit Status

Show the exit status of the last command:

PS1="\[\e[31m\]\$? \[\e[34m\]\u@\h:\w\$ \[\e[0m\]"

• \$?: Exit status of the last command.

---

Advanced Customisations with Tools

Starship

Starship is a modern, highly customizable prompt written in Rust. Install it and add this line to your .bashrc:

eval "$(starship init bash)"

---

Best Practices for Customizing Your Prompt

1. Keep it Simple: Avoid cluttering the prompt with too much information.
2. Use Colors Sparingly: Highlight only the most critical details.
3. Test Changes: Test new prompts before making them permanent.
4. Backup Your .bashrc: Keep a backup before making extensive changes.

---

Example Custom Prompt

Here’s a full-featured example:

PS1="\[\e[32m\]\u@\h \[\e[36m\]\w \[\e[33m\]\t \[\e[31m\]$(git_branch)\[\e[0m\]\$ "

• Green username and hostname.
• Cyan working directory.
• Yellow current time.
• Red Git branch (requires Git integration).

---

Customizing your Bash prompt is a great way to make your terminal more functional and visually appealing. Experiment with different configurations and find the one that works best for you!

Here are three common ways to determine which process is listening on a particular port in Linux:

---

1. Using lsof (List Open Files)

• Command: bash sudo lsof -i :<port_number>
• Example: bash sudo lsof -i :8080
• Output:
  • The command shows the process name, PID, and other details of the process using the specified port.

---

2. Using netstat (Network Statistics)

• Command: bash sudo netstat -tuln | grep :<port_number>
• Example: bash sudo netstat -tuln | grep :8080
• Output:
  • Displays the protocol (TCP/UDP), local address, foreign address, and the process (if run with -p option on supported versions).

Note: If netstat is not installed, you can install it via: bash sudo apt install net-tools

---

3. Using ss (Socket Statistics)

• Command: bash sudo ss -tuln | grep :<port_number>
• Example: bash sudo ss -tuln | grep :8080
• Output:
  • Displays similar information to netstat but is faster and more modern.

---

Bonus: Using /proc Filesystem

• Command: bash sudo grep <port_number> /proc/net/tcp
• Example: bash sudo grep :1F90 /proc/net/tcp > Replace :1F90 with the hexadecimal representation of the port (e.g., 8080 in hex is 1F90).
• This is a more manual approach and requires converting the port to hexadecimal.

In Bash, managing timers typically involves the use of two primary tools: Bash scripts with built-in timing features (like sleep or date) and Cron jobs (via crontab) for scheduled task execution. Both tools are useful depending on the level of complexity and frequency of the tasks you're managing.

1. Using Timers in Bash (CLI)

In Bash scripts, you can manage timers and delays by using the sleep command or date for time-based logic.

a. Using sleep

The sleep command pauses the execution of the script for a specified amount of time. It can be used for simple timing operations within scripts.

Example:

#!/bin/bash

# Wait for 5 seconds
echo "Starting..."
sleep 5
echo "Done after 5 seconds."

You can also specify time in minutes, hours, or days:

sleep 10m  # Sleep for 10 minutes
sleep 2h   # Sleep for 2 hours
sleep 1d   # Sleep for 1 day

b. Using date for Timing

You can also use date to track elapsed time or to schedule events based on current time.

Example (Calculating elapsed time):

#!/bin/bash

start_time=$(date +%s)  # Get the current timestamp
echo "Starting task..."
sleep 3  # Simulate a task
end_time=$(date +%s)  # Get the new timestamp

elapsed_time=$((end_time - start_time))  # Calculate elapsed time in seconds
echo "Elapsed time: $elapsed_time seconds."

2. Using crontab for Scheduling Tasks

cron is a Unix/Linux service used to schedule jobs to run at specific intervals. The crontab file contains a list of jobs that are executed at scheduled times.

a. Crontab Syntax

A crontab entry follows this format:

* * * * * /path/to/command
│ │ │ │ │
│ │ │ │ │
│ │ │ │ └── Day of the week (0-7) (Sunday=0 or 7)
│ │ │ └── Month (1-12)
│ │ └── Day of the month (1-31)
│ └── Hour (0-23)
└── Minute (0-59)

• * means "every," so a * in a field means that job should run every minute, hour, day, etc., depending on its position.
• You can use specific values or ranges for each field (e.g., 1-5 for Monday to Friday).

b. Setting Up Crontab

To view or edit the crontab, use the following command:

crontab -e

Example of crontab entries: - Run a script every 5 minutes: bash */5 * * * * /path/to/script.sh - Run a script at 2:30 AM every day: bash 30 2 * * * /path/to/script.sh - Run a script every Sunday at midnight: bash 0 0 * * 0 /path/to/script.sh

c. Managing Crontab Entries

• List current crontab jobs: bash crontab -l
• Remove crontab entries: bash crontab -r

d. Logging Cron Jobs

By default, cron jobs do not provide output unless you redirect it. To capture output or errors, you can redirect both stdout and stderr to a file:

* * * * * /path/to/script.sh >> /path/to/logfile.log 2>&1

This saves both standard output and error messages to logfile.log.

3. Combining Bash Timers and Cron Jobs

Sometimes you might need to use both cron and timing mechanisms within a Bash script. For example, if a task needs to be scheduled but also requires some dynamic timing based on elapsed time or conditions, you could use cron to trigger the script, and then use sleep or date inside the script to control the flow.

Example:

#!/bin/bash

# This script is triggered every day at midnight by cron

# Wait 10 minutes before executing the task
sleep 600  # 600 seconds = 10 minutes

# Execute the task after the delay
echo "Executing task..."
# Your task here

4. Advanced Scheduling with Cron

If you need more complex scheduling, you can take advantage of specific cron features: - Use ranges or lists in the time fields: bash 0 0,12 * * * /path/to/script.sh # Run at midnight and noon every day - Run a task every 5 minutes during certain hours: bash */5 9-17 * * * /path/to/script.sh # Every 5 minutes between 9 AM and 5 PM

5. Practical Examples

• Backup Every Night:

  0 2 * * * /home/user/backup.sh
  

  This runs the backup.sh script every day at 2 AM.

• Check Server Health Every Hour:

  0 * * * * /home/user/check_server_health.sh
  

  This runs a script to check the server's health every hour.

Conclusion

Managing timers in Bash using cron and sleep allows you to automate tasks, control timing, and create sophisticated scheduling systems. sleep is suitable for in-script delays, while cron is ideal for recurring scheduled jobs. Combining these tools lets you create flexible solutions for a wide range of automation tasks.

OS Package Managers: Keeping Your System Up-to-Date

Package managers are essential tools in modern operating systems (OS) that help automate the process of installing, updating, and removing software packages. These tools manage the software installed on a system, making it easier for users and administrators to keep their systems up-to-date with the latest versions of software. They provide a streamlined and efficient way to manage dependencies, handle software updates, and ensure system stability by preventing compatibility issues.

Importance of Package Managers

Package managers are crucial for maintaining system health and security, and they provide several benefits:

1. Automatic Updates: Package managers track software versions and allow you to update all installed software at once with a single command. This ensures that you always have the latest security patches, performance improvements, and new features without needing to manually search for and download updates.

2. Dependency Management: Many software packages depend on other libraries and programs to function. Package managers ensure that these dependencies are correctly installed and maintained, which reduces the likelihood of conflicts between different versions of libraries or missing dependencies.

3. Security: Security is a major reason to use a package manager. Package managers allow you to easily update software to close vulnerabilities that could be exploited by attackers. Often, package repositories are curated and include only trusted, verified packages.

4. Reproducibility: Package managers allow administrators to set up systems with the exact same configuration across multiple machines. This is especially important in server environments, where you want all systems to have the same set of software, libraries, and dependencies.

5. Software Removal: Package managers make it easy to remove unwanted software. This ensures that unnecessary files, dependencies, and configurations are cleaned up, saving disk space and reducing the attack surface.

6. Centralized Repository: Most package managers use centralized repositories where software is pre-compiled and tested, so users don’t need to manually compile code or find external download sources, minimizing risks from malicious software.

---

Types of Package Managers

There are different types of package managers depending on the operating system. Below, we will explore examples from different OS environments to see how package managers work.

1. Linux Package Managers

Linux distributions (distros) typically use package managers that vary based on the type of distribution. The most common Linux package managers are:

• APT (Advanced Package Tool): Used in Debian-based systems such as Ubuntu.
• YUM/DNF (Yellowdog Updater, Modified / Dandified YUM): Used in Red Hat-based systems such as CentOS, Fedora, and RHEL.
• Zypper: Used in openSUSE and SUSE Linux Enterprise Server.
• Pacman: Used in Arch Linux and Manjaro.

Examples of Commands to Install and Update Software on Linux:

1. APT (Ubuntu/Debian)

- Install a package:

sudo apt install <package-name>

Example:

sudo apt install vim

• Update the system:

sudo apt update
sudo apt upgrade

This updates the package list and upgrades all installed software to the latest available version in the repositories.

• Upgrade a specific package:

sudo apt install --only-upgrade <package-name>

Example:

sudo apt install --only-upgrade vim

• Remove a package:

sudo apt remove <package-name>

Example:

sudo apt remove vim

2. YUM/DNF (CentOS/Fedora/RHEL)

- Install a package:

sudo yum install <package-name># YUM for older versions
sudo dnf install <package-name># DNF for newer Fedora/CentOS/RHEL

Example:

sudo dnf install vim

• Update the system:

sudo dnf update

This command updates the entire system, installing the latest versions of all packages.

• Upgrade a specific package:

sudo dnf upgrade <package-name>

• Remove a package:

sudo dnf remove <package-name>

Example:

sudo dnf remove vim

3. Zypper (openSUSE)

- Install a package:

sudo zypper install <package-name>

Example:

sudo zypper install vim

• Update the system:

sudo zypper update

• Remove a package:

sudo zypper remove <package-name>

Example:

sudo zypper remove vim

4. Pacman (Arch Linux)

- Install a package:

sudo pacman -S <package-name>

Example:

sudo pacman -S vim

• Update the system:

sudo pacman -Syu

• Remove a package:

sudo pacman -R <package-name>

Example:

sudo pacman -R vim

---

2. macOS Package Manager

On macOS, Homebrew is the most popular package manager, although there are alternatives such as MacPorts.

• Homebrew:
  Homebrew allows macOS users to install software and libraries not included in the macOS App Store. It works by downloading and compiling the software from source or installing pre-built binaries.

Examples of Commands to Install and Update Software on macOS:

• Install a package:

brew install <package-name>

Example:

brew install vim

• Update the system:

brew update
brew upgrade

• Upgrade a specific package:

brew upgrade <package-name>

• Remove a package:

brew uninstall <package-name>

Example:

brew uninstall vim

---

3. Windows Package Managers

Windows traditionally didn't include package managers like Linux or macOS, but with the advent of Windows Package Manager (winget) and Chocolatey, this has changed.

• winget (Windows Package Manager):
  Windows 10 and newer include winget, a command-line package manager for installing software.

Examples of Commands to Install and Update Software on Windows:

• Install a package:

winget install <package-name>

Example:

winget install vim

• Update a package:

winget upgrade <package-name>

• Update all installed software:

winget upgrade --all

• Remove a package:

winget uninstall <package-name>

Example:

winget uninstall vim

• Chocolatey:
  Chocolatey is another popular package manager for Windows, with a large repository of software.

Install a package:

choco install <package-name>

Example:

choco install vim

Update a package:

choco upgrade <package-name>

Remove a package:

choco uninstall <package-name>

---

Conclusion

Package managers provide a streamlined, automated way to manage software installation, updates, and removal. Whether you're on a Linux, macOS, or Windows system, a package manager ensures that your software is up-to-date, secure, and properly configured. By using package managers, you can easily manage dependencies, get the latest versions of software with minimal effort, and maintain system stability.

Having the ability to run a single command to install or update software, like sudo apt update on Linux or brew upgrade on macOS, saves time and reduces the risks associated with manually downloading and managing software. Package managers have become a fundamental tool for system administrators, developers, and power users, making system maintenance and software management easier, faster, and more reliable.

Introduction

Signals are used to interact between processes and can occur at anytime, typically they are kill signals however processes can opt to handle them programatically unless they are SIGKILL or SIGSTOP signals.

List Of Available Signals

Table of signals

SIGNAL	VALUE	DEFAULT ACTION	POSIX?	MEANING
SIGHUP	1	Terminate	Yes	Hangup detected on controlling terminal or death of controlling process
SIGINT	2	Terminate	Yes	Interrupt from keyboard
SIGQUIT	3	Core dump	Yes	Quit from keyboard
SIGILL	4	Core dump	Yes	Illegal instruction
SIGTRAP	5	Core dump	No	Trace/breakpoint trap for debugging
SIGABTR SIGIOT	6	Core dump	Yes	Abnormal termination
SIGBUS	7	Core dump	Yes	Bus error
SIGFPE	8	Core dump	Yes	Floating point exception
SIGKILL	9	Terminate	Yes	Kill signal (cannot be caught or ignored)
SIGUSR1	10	Terminate	Yes	User-defined signal 1
SIGSEGV	11	Core dump	Yes	Invalid memory reference
SIGUSR2	12	Terminate	Yes	User-defined signal 2
SIGPIPE	13	Terminate	Yes	Broken pipe: write to pipe with no readers
SIGALRM	14	Terminate	Yes	Timer signal from alarm
SIGTERM	15	Terminate	Yes	Process termination
SIGSTKFLT	16	Terminate	No	Stack fault on math co-processor
SIGCHLD	17	Ignore	Yes	Child stopped or terminated
SIGCONT	18	Continue	Yes	Continue if stopped
SIGSTOP	19	Stop	Yes	Stop process (can not be caught or ignored)
SIGTSTP	20	Stop	Yes	Stop types at tty
SIGTTIN	21	Stop	Yes	Background process requires tty input
SIGTTOU	22	Stop	Yes	Background process requires tty output
SIGURG	23	Ignore	No	Urgent condition on socket (4.2 BSD)
SIGXCPU	24	Core dump	Yes	CPU time limit exceeded (4.2 BSD)
SIGXFSZ	25	Core dump	Yes	File size limit exceeded (4.2 BSD)
SIGVTALRM	26	Terminate	No	Virtual alarm clock (4.2 BSD)
SIGPROF	27	Terminate	No	Profile alarm clock (4.2 BSD)
SIGWINCH	28	Ignore	No	Window resize signal (4.3 BSD, Sun)
SIGIO SIGPOLL	29	Terminate	No	I/O now possible (4.2 BSD) (System V)
SIGPWR	30	Terminate	No	Power Failure (System V)
SIGSYS SIGUNUSED	31	Terminate	No	Bad System Called. Unused signal

Introduction

A computer doing more than one thing at a time is using processes, these require resources, CPU time, memory and access to other devices like CD/DVD/USB drives, etc. Each process is allocated an amount of system resources to perform its function which is controlled by the operating system whose job it is to facilitate these processes. Signals have an important part to play on the interaction of the processes, usually these send exit signals and other information to each other, or to itself.

Programs, Processes, and Threads

A program is a set of instructions to be carried out which may local data such as information for output to the terminal via read or external data which may come from a database. Common programs include ls, cat and rm which would reside outside of the user operating system thus they have their own operating program on disk.

A process is a program which is executing which has associated resources, furthermore. These might include environments, open files or signal handlers etc. At the same time, two or more tasks or processes may share system resources meaning they are multi-threaded processes.

In other operating systems there may be a distinction between heavy-weight and light-weight processes; furthermore, heavy-processes may contain a number of lightweight processes. However, in Linux, each process is considered heavy or light by the amount of shared resources they consume, for faster context switching. Again, unlike other operating systems, Linux is much faster at switching between processes, creating and destroying them too. This means the model for multi-threaded processes is similar to that of simultaneous processes as they are treated as one. All the same, Linux respects POSIX and other standards for multi-threaded processes, where each thread returns the same process ID plus returning a thread ID.

Processes

Processes are running programs in execution, either running or sleeping. Every process has a process ID (pid), parent process ID (ppid) and a parent group ID (ppgid). In addition every process has program code, data, variables, file descriptions and an environment.

In Linux, init is the first process that is run, thus becoming the ancestor to all other programs executed on the system; unless they are started directly from the Linux Kernel, for which they are grouped with brackets ([]) in the ps listing. Other commonalities are the processes pid numbering, if a parent process ends before the child, the process will become adopted by init, setting its process ID (pid) to 1. If the system is more recent, this will result in the pid being set to 2 inline with systemd’s kernel thread kthreadd. In the circumstance where a child dies before its parent it enters a zombie state, using no resources and retaining only the exit code. It is the job of init to allow these processes to die gracefully and is often referred to as being the zombie killer or child reaper. Finally, the processes ID can not go above 16bit definition hence 32768 is the largest pid you will find on a Linux system; to alter this value see /proc/sys/kernel/pid_max; once this limit is reached they are restarted at 300.

Process Attributes

• All processes have certain attributes, as shown below:
  • Program
  • Context
  • Permissions
  • Resources

The program is the process itself, maybe it is a set of commands in a script or a loop which never ends checking against external data and performing an action when necessary.

The context is the state which the program is in at any point in time, thus a snapshot of CPU registers, what is in RAM and other information. Furthermore, processes can be scheduled in and out when sharing CPU time or put to sleep when waiting for user input, etc. Being able to swap out and put back the process context is known as context switching.

Permissions are inherited from the user who executed the program however programs themselves can have an s property assigned to them In order to define their effective user ID, rather than their real ID and are referred to as setuid programs. These setuid programs run with the permissions of the creator of the program, not the user who ran it. A commonly found setuid program is passwd.

Process Resource Isolation

This is the practice of isolating the process from other processes upon execution, promoting security and stability. Furthermore, processes do not have direct access to hardware devices, etc. Hardware is managed by the kernel meaning system calls must be made in order to interact with said devices.

Controlling Processes (ulimit)

This program, umlimit, reports and resets a number of resource limits associated with processes running under its shell. See below for a typical output of ulimit -a.

Here you see a screenshot of ulimit -a which contains values important for a system administrator to be aware of because it identifies the allocation of resources. You may want to restrict or expand resource allocation depending on the requirements of the processes and / or file access limits.

Hard and soft resource allocation terms come into play here, with hard limits imposed by the administrator and soft limited by those restrictions but allowing for user-level limits to be imposed. Run ulimit -H -n to see hard limits and ulimit -S -n for soft limits. File descriptors are probably the most common thing that may need changing, to allow more resources, typically this is set to 1024 which may make execution of an application virtually impossible so you can change this with ulimit -n 1600 which would change it to 1600. In order to make more permanent changes you would need to edit /etc/security/limits.conf then perform a reboot.

Process States

Processes can take on many different states, which is managed by the scheduler. See below for the main process states: - Running - Sleeping (waiting) - Stopped - Zombie

A running process is either using the CPU or will be waiting for its next CPU time execution slice, usually cited in the run queue - resuming when the scheduler deems it satisfactory to re-enter CPU execution.

A sleeping process is one that is waiting on its request that can not be processed until completed. Upon completion, the kernel will wake the process and reposition it into the run queue.

A stopped process is one that has been suspended and is used normally by developers to take a look at resource usage at any given point. This can be activated with CTRL-Z or by using a debugger.

A zombie process is one that has entered terminated state and no other process has inquired about it, namely “reaped it” - this can often be called a “defunct process”. The process releases all of its resources except its exit state.

Introduction

After reading this document you should be able to identify why Linux defines its filesystem hierarchy in one big tree and explain the role of the filesystem hierarchy standard, explain what is available at boot in the root directory (/), explain each subdirectory purpose and typical contents. The aim here is to be able to create a working bash script which knows where to put its different data stores including lockfiles, database(s) or temporary files; including the script itself.

One Big Filesystem

As with all Linux installations there is a set protocol to follow which could be looked at as one big tree starting from its root, /. This often contains different access points not just typical file or folder components, but in fact mounted drives, USB or CD/DVD media volumes and so on. Even more adapt, these can span many partitions; as one filesystem. Regardless, the end result is one big filesystem, meaning applications generally do not care what volume or partition the data resides upon. The only drawback you may encounter is different naming conventions however there are now standards to follow in the Linux-ecosystem for cross-platform conformity.

Defining The Data

There has to be one method for defining all data in order to satisfy clear distinctions. Firstly, you may examine data and identify whether it is shareable or not. For instance, /home data may be shared across many hosts however .pid lock files will not. Another angle you may look at is are the files variable or static, meaning if no administrator input is given they will remain the same, ie. static - or else the data changes when the filesystem is in operation without human interaction hence it is called variable. With this in mind, you must identify which trees and sub-trees are accessible by your application or command prompt to identify if they can be manipulated at runtime and where they should reside if you are creating these filetypes.

To summarise: - Shared Data is Common To All Systems - Non-Shareable Data is Local To One System - Static Data Never Changes When Left Alone - Variable Data Will Change During Application Processing

The Filesystem Hierarchy Standard aims to help achieve unity across all platforms however different distributions can often invent new methodologies that will generally become standard over time. While the FHS (Filesystem Hierarchy Standard) publishes its standard new conventions are currently in play despite this document, see here: linuxfoundation.org...

DIRECTORY	FHS Approved	PURPOSE
/	Yes	Primary directory of the entire filesystem hierarchy.
/bin	Yes	Essential executable programs that must be available in single user mode.
/boot	Yes	Files needed to boot the system, such as the kernel, initrd or initramfs |images, and boot configuration files and bootloader programs.
/dev	Yes	Device Nodes, used to interact with hardware and software devices.
/etc	Yes	System-wide configuration files.
/home	Yes	User home directories, including personal settings, files, etc.
/lib	Yes	Libraries required by executable binaries in /bin and /sbin.
/lib64	No	64-bit libraries required by executable binaries in /bin and /sbin, for systems which can run both 32-bit and 64-bit programs.
/media	Yes	Mount points for removable media such as CDs, DVDs, USB sticks, etc.
/mnt	Yes	Temporarily mounted filesystems.
/opt	Yes	Optional application software packages.
/proc	Yes	Virtual pseudo-filesystem giving information about the system and processes running on it. Can be used to alter system parameters.
/sys	No	Virtual pseudo-filesystem giving information about the system and processes running on it. Can be used to alter system parameters. Similar to a device tree and is part of the Unified Device Model.
/root	Yes	Home directory for the root user.
/sbin	Yes	Essential system binaries.
/srv	Yes	Site-specific data served up by the system. Seldom used.
/tmp	Yes	Temporary files; on many distributions lost across a reboot and may be a ramdisk in memory.
/usr	Yes	Multi-user applications, utilities and data; theoretically read-only.
/var	Yes	Variable data that changes during system operation.

Run du --max-depth=1 -hx / to see the output of your root filesystem hierarchy.

The Root Directory (/)

Starting with our first directory, the root directory (/) this is often the access point mounted across multiple (or single) partitions with other locations such as /home, /var and /opt mounted later. This root partition must contain all root directories and files at boot in order to serve the system. Therefore it needs boot loader information and configuration files plus other essential startup data, which must be adequate to perform the following operations: - Boot the system - Restore the system on external devices such as USB, CD/DVD or NAS - Recover and/or repair the system (ie. in rescue mode)

The root directory / should never have folders created directly within it; period.

Binary Files (/bin)

• The /bin directory must be present for a system to function containing scripts which are used indirectly by other scripts. It's important because non-privileged users and system administrators all have access to this directory plus it contains scripts needed to be served before the filesystem is even mounted. It is common place to store non-essential scripts which do not merit going in /bin to be served from /usr/bin, however /bin is becoming more acceptable to be used in common-place operation, in fact in RHEL they are the same directory. Often symbolic links are used from/bin to other folder locations in order to preserve two-way folder listings.

They are as follows: cat, chgrp, chmod, chown, cp, date, dd, df, dmesg, echo, false, hostname, kill, ln, login, ls, mkdir, mknod, more, mount, mv, ps, pwd, rm, rmdir, sed, sh, stty, su, sync, true, umount and uname

Other binaries that may be present during boot up and in normal operation are: test, csh, ed, tar, cpio, gunzip, zcat, netstat, ping

The Boot Directory (/boot)

This folder contains the vmlinuz and intramfs (also known as initrd) files which are put there in order to serve the boot operation, the first is a compressed kernel and the second is the initial RAM filesystem. Other files include config and system.map.

Device Files (/dev)

Device files are often used to store device nodes; commonly serving various hardware references including nodes - network cards however are more likely to be named eth0 or wlan0 meaning they are referenced by name. The directory /dev/ will automatically create nodes using udev when system hardware is found. Quite aptly, ls /dev | grep std will show you some useful output references which can be used to process data either to the terminal or into the obis.

Configuration Files (/etc)

Used to contain config directives (or contained folders with config directives) for system-wide programs and more importantly system services.

Some examples are: csh.login, exports, fstab, ftpusers, gateways, gettydefs, group, host.conf, hosts.allow, hosts.deny, hosts,equiv, hosts.lpd, inetd.conf, inittab, issue, ld.so.conf, motd, mtab, mtools.conf, networks, passwd, printcap, profile, protocols, resolv.conf, rpc, securetty, services, shells, syslog.conf

More crucially, the following helps keep the system correctly configured: - /etc/skel This folder contains the skeleton of any new users /home directory - /etc/systemd - Points to or contains configuration scripts for system services, called by service - /etc/init.d - Contains startup and shutdown scripts used by System V initialisation

System Users (/home)

On Linux, users working directories are given in the /home/{username} format and are typically named in a naming convention such as /home/admin /home/projects/home/stagingor/home/production. Typically, this could be their name or nickname or purpose, eg./home/steve,/home/steve-work` and so on.

With Linux, this folder can be accessed via the ~ symbol which is given to system users in order to direct the user to the currently logged in users home directory, eg. ls ~/new-folder etc; which is also accessible by using $home. The only caveat to this is that the root user is placed in /root - all other users reside in /home, typically mirroring /etc/skel as previously outlined. (see “Configuration Files (/etc)”)

System Libraries (/lib and /lib64)

These folders are for libraries serving binary files found in /bin or other locations where scripts are found. These libraries are important because they maintain the upkeep of essential system programs (binaries) which help boot the system and then are used by the system once booted, fundamentally. Kernel modules (device or system drivers) are stored in /lib/modules and PAM (Pluggable Authentication Modules) are stored in /lib/security.

For systems running 32bit and 64bit the /lib64 is usually present. More common place is to use the one folder with symbolic links to the actual destination of the library, similar to how /bin has reformed back into one folder, providing the same structure with separation of differing program importance (using symbolic links) while maintaining the single source for all scripts of that type.

External Devices (/media)

The /media folder is often found to be a single source for all removable media. USB, CD, DVD even the ancient Floppy Disk Drive. Linux mounts these automatically using “udev” which in turn creates references inside /media making for simple access to external devices, autonomously. As the device is removed, the name of the file in this directory is removed also.

Temporary Mounts (/mnt)

This folder is used for mount points, usually temporary ones. During the development of the FHS this would typically contain removable devices however /media is favoured on modern systems.

Typical use scenarios are: - NFS - Samba - CIFS - AFS

Generically, this should not be used by applications, instead mounted disks should be located elsewhere on the system.

Software Packages (/opt)

This location is where you would put system-wide software packages with everything included in one place. This is for services that want to provide everything in one place, so you would have /opt/project/bin etc. all in this folder. The directories /opt/bin, /opt/doc, /opt/include, /opt/info, /opt/lib, and /opt/man are saved for administrator usage.

System Processes (/proc)

These are special files which are mounted like with /dev and are constantly changing. They only contain data at the point you make the request, so typically a file may be 0kb but if you look, may contain many lines of data; this is accessed only when you run the cat or vi operation, it does indeed remain empty. Important pseudo files are /proc/interrupts, /proc/meminfo, /proc/mounts, and /proc/partitions.

System Filesystems (/sys)

This directory is the mount point for the sysfs pseudo-filesystem where all information resides only in memory, not on disk. This is again very much like /dev/ and /proc in that it contains mounted volumes which are created on system boot. Containing information about devices and drivers, kernel modules and so on.

Root (/root)

This is generally called “slash-route” pronounced phonetically and is simply the primary system administrators home folder. Other user accounts are encouraged with specific access details for better security.

System Binaries (/sbin)

This is very similar to /bin and as mentioned may very well have symbolic link references inside /bin with the actual program residing here. This allows for a one-solution-fits-all because /bin will display all programs whereas /sbin would be designated to the programs listed there. Some of these programs include: fdisk, fsck, getty, halt, ifconfig, init, mkfs, mkswap, reboot, route, swapon, swapoff and update.

System Services (/srv)

Popular for some administrators this is designed to provide system service functionality. You can be fairly lax with naming conventions here you may want to group applications into folders such as ftp, rsync, www, and cvs etc. Popular by those that use it and may be overlooked by those what don’t.

Temporary Files (/tmp)

Used by programs that do not want to keep the data between system boots and it may be periodically refreshed by the system. Use at your own discretion. Be aware as this is truly temporary data any large files may cause issues as often the information is stored in memory.

System User (/usr)

This should be thought of as a second system hierarchy. Containing non-local data it is best practice to serve administrator applications here and is often used for files and packages or software that is not needed for booting.

DIRECTORY	PURPOSE
/usr/bin	Non-essential command binaries
/usr/etc	Non-essential configuration files (usually empty)
/usr/games	Game data
/usr/include	Header files used to compile applications
/usr/lib	Library files
/usr/lib64	Library files for 64-bit
/usr/local	Third-level hierarchy (for machine local files)
/usr/sbin	Non-essential system binaries
/usr/share	Read-only architecture-independent files
/usr/src	Source code and headers for the Linux kernel
/usr/tmp	Secondary temporary directory

Other common destinations are /usr/share/man and /usr/local, the former is for manual pages and the latter is for predominantly read-only binaries.

Variable Data (/var)

This directory is intended for variable (volatile) data and as such is often updated quite frequently. Contains log files, spool directories and files administrator files and transient files such as cache data.

SUBDIRECTORY	PURPOSE
/var/ftp	Used for ftp server base
/var/lib	Persistent data modified by programs as they run
/var/lock	Lock files used to control simultaneous access to resources
/var/log	Log files
/var/mail	User mailboxes
/var/run	Information about the running system since the last boot
/var/spool	Tasks spooled or waiting to be processed, such as print queues
/var/tmp	Temporary files to be preserved across system reboot. Sometimes linked to /tmp
/var/www	Root for website hierarchies

Transient Files (/run)

These are meant to be files that are updated quite regularly and are not often maintained during reboots, useful for containing temporary files and runtime information. The use of run is quite new and you may find /var/run and /var/lock symbolic link references. The use of this is more common place in modern systems.

When it happens that your VPS is eating data by the second and there is disk read/write issues one port of call you are bound to visit is searching and identifying large files on your system.

Now, you would have been forgiven for thinking this is a complicated procedure considering some Linux Bash solutions for fairly simple things, but no. Linux Bash wins again!

du -sh /path/to/folder/* | sort -rh

Here, du is getting the sizes and sort is organising them, -h is telling du to display human-readable format.

The output should be something like this:

2.3T    /path/to/directory
1.8T    /path/to/other

It does take a while to organise as it is being done recursively however given 3-5mins and most scenarios will be fine.

So yeah, getting used to Bash is about finding the right way to do things. However, learning one-liners and picking up information here and there is all very useful, finding something you don't need to Google in order to recall is extremely important.

Take the case of recursive find and replace. We've all been there, it needs to be done frequently but you're either a) scared or b) forgetful. Then you use the same snippet from a web resource again and again and eventually make a serious error and boom, simplicity is now lost on you!

So here it is, something you can remember so that you don't use different methods depending on what Google throws up today.

grep -Rl newertext . | xargs sed -i 's/newertext/newesttext/g'

Notice, we use grep to search -R recursively with results fed to xargs which does a simple sed replace with g global/infinite properties.

Say you want to keep it simple though and find and review the files before doing the "simple" replace. Well, do this.

grep -R -l "foo" ./*

The uppercase -R denotes to follow symlinks and the ./* indicates the current directory. The -l requests a list of filenames matched. Neat.