How to Use the Command `sysdig` (with examples)

Sysdig is a powerful, open-source, system monitoring tool that acts as a versatile Swiss Army knife for troubleshooting, analyzing, and exploring system behavior in real-time. By capturing system calls and other OS-level interactions, sysdig provides valuable insights into how applications and systems function, making it invaluable for developers, system administrators, and security professionals.

Sysdig shines in various scenarios, such as when detailed, real-time monitoring of system calls and network activity is required, or when a post-mortem analysis of a system is needed to diagnose what went wrong. It’s particularly adept at revealing issues that generally go unnoticed, providing a deep dive into the intricacies of process interactions and resource usage.

Use Case 1: Capture All Events from the Live System and Print Them to Screen

Code:

sysdig

Motivation:

Capturing all events from a live system and immediately printing them to the screen is the most straightforward way to start understanding the flow of system calls and their effects. This is particularly helpful when you want to monitor a live system and are unsure what kind of problem might currently exist.

Explanation:

• sysdig: This command, by itself, runs sysdig in its default mode, where it begins to capture all system calls and OS events and prints them in real time to the terminal screen. There are no additional arguments, which means that every call happening in the system will be shown as it occurs.

Example Output:

The output consists of a stream of system call events, each detailed with attributes such as timestamp, process ID, process name, user ID, system call type, and more:

11:57:06.258289231 12345 ls (1234) < open fd=3(<f>/etc/ld.so.cache) dirfd=-100(<f>/etc/ld.so.cache) name=/etc/ld.so.cache flags=4098(O_RDONLY|O_CLOEXEC) mode= (0)
11:57:06.258289470 12345 ls (1234) > open res=3

Use Case 2: Capture All Events from the Live System and Save Them to Disk

Code:

sysdig -w path/to/file.scap

Motivation:

Saving events to disk is particularly useful when you need a historical record to analyze later or wish to conduct a more in-depth analysis on another machine without being connected to the live system. This use case is great for collecting data over a period of time for later examination.

Explanation:

• sysdig: The base command that indicates we wish to capture system calls.
• -w path/to/file.scap: The -w flag tells sysdig to write the captured events to a file. The specified path, path/to/file.scap, determines where these events will be stored. It’s important to use the .scap extension as a convention for sysdig capture files.

Example Output:

This command does not produce output immediately on the screen but captures system activity into file.scap, which is stored on disk:

(no immediate screen output; data is saved to path/to/file.scap)

Use Case 3: Read Events from a File and Print Them to Screen

Code:

sysdig -r path/to/file.scap

Motivation:

Reading events from a file and printing them to a screen is primarily used for post-analysis of system state or behavior. This allows you to revisit previously captured events, which is useful for diagnosing past issues or understanding how a particular process interacted with the system.

Explanation:

• sysdig: Calls the sysdig utility.
• -r path/to/file.scap: The -r flag stands for “read,” and instructs sysdig to read events from a pre-existing capture file (path/to/file.scap) instead of live events, replaying them to the screen.

Example Output:

As this command replays the events stored in the path/to/file.scap, you see detailed system call and OS events as they occurred at capture time:

11:50:22.457823456 23456 ping (2345) < open fd=3(<f>/etc/resolv.conf) dirfd=-100(<f>/etc/resolv.conf) name=/etc/resolv.conf flags=4096(O_RDONLY) mode= (0)
11:50:22.457823670 23456 ping (2345) > open res=3

Use Case 4: Filter and Print All Open System Calls Invoked by Cat

Code:

sysdig proc.name=cat and evt.type=open

Motivation:

Filtering and printing only the open system calls invoked by cat helps you focus on specific interactions of interest, highlighting file access attempts made by a single command. This is essential when diagnosing file access issues or understanding which resources a particular process requires.

Explanation:

• sysdig: The primary command executes sysdig.
• proc.name=cat: This filter focuses sysdig on capturing events where the process name matches “cat”.
• and evt.type=open: This additional filter indicates that only open system calls should be displayed, narrowing the focus to file-opening operations performed by cat.

Example Output:

Output consists of only those open system calls initiated by the cat command:

12:00:05.789012345 12357 cat (4567) < open fd=3(<f>myfile.txt) dirfd=-100(<f>myfile.txt) name=myfile.txt flags=4098(O_RDONLY|O_CLOEXEC) mode= (0)
12:00:05.789012456 12357 cat (4567) > open res=3

Use Case 5: Register Any Found Plugin and Use Dummy as Input Source Passing to It Open Params

Code:

sysdig -I dummy:'parameter'

Motivation:

This use case is geared towards those who are experimenting with sysdig plugins. By using dummy as an input source, it processes whichever parameters are passed, allowing for flexible testing and development of plugins in the sysdig ecosystem.

Explanation:

• sysdig: Invokes the sysdig tool.
• -I: This tells sysdig to register any found plugins.
• dummy:'parameter': Specifies the input source as dummy and passes parameter to it. This provides a controlled testing environment via a mock input source.

Example Output:

The command doesn’t produce a regular system call output but would be used to initiate a plugin, often tailored for development contexts. Actual outputs depend on the specifics of the plugin and parameters used.

(Plugin output varies based on implementation)

Use Case 6: List the Available Chisels

Code:

sysdig -cl

Motivation:

Listing available chisels is fundamental for users who want to explore the additional built-in scripts that extend sysdig’s functionality. Chisels provide an array of analytical and monitoring tools, customizable to specific tasks.

Explanation:

• sysdig: Activates the sysdig program.
• -cl: This shorthand commands sysdig to present all available chisels, where ‘c’ denotes chisels and ’l’ signifies listing.

Example Output:

This outputs a comprehensive list of predefined chisels, each with an explanation of its purpose:

ID    NAME                   DESCRIPTION
1     topfiles_bytes        Top files by R+W bytes
2     topprocs_latency      Top processes by latency
3     spy_ip                Inspect the network traffic

Use Case 7: Use the Spy_ip Chisel to Look at Data Exchanged with an IP Address

Code:

sysdig -c spy_ip ip_address

Motivation:

Using the spy_ip chisel is incredibly useful for network security analysis. It allows a user to monitor and inspect all data exchanged with a specific IP address, providing insights into potential breaches, leaks or unauthorized access attempts.

Explanation:

• sysdig: Launches the sysdig application.
• -c spy_ip: Signals sysdig to run the spy_ip chisel, which is designed to track and report on IP-specific network traffic.
• ip_address: User-specified IP address whose traffic will be monitored.

Example Output:

This command results in a direct stream of network data associated with the provided IP address:

IP 10.0.0.1: Received 150 bytes: GET /index.html HTTP/1.1
IP 10.0.0.1: Sent 200 bytes: HTTP/1.1 200 OK

Conclusion

Sysdig is an invaluable resource for those interested in delving into the real-time and historical activities occurring within their systems. It provides indispensable assistance in capturing, analyzing, and filtering system calls and network activities, and extends its functionality through a versatile plugin and chisel ecosystem. This utility not only enhances system visibility and troubleshooting capabilities but also contributes to improving security posture by enabling detailed monitoring and control.