Tracing a process in Linux is essential for debugging, performance monitoring, or understanding the behavior of a program. Process investigation in Linux is crucial for several reasons:
Troubleshooting
When a system encounters issues, understanding which processes are running and how they interact can help identify the root cause of the problem. This is essential for effective troubleshooting and quick resolution.
Resource Management
By monitoring processes, administrators can identify resource-intensive tasks and optimize system performance. This ensures efficient utilization of CPU, memory, and other resources.
Security
Investigating processes helps in detecting unusual or unauthorized activities, which could indicate security breaches or malware. This is vital for maintaining the integrity and security of the system.
System Maintenance
Regular process investigation helps in maintaining system health by identifying and terminating unnecessary or rogue processes that may consume resources or cause instability.
Forensics
In the event of a security incident, process investigation is key to understanding what happened, how it happened, and potentially who was responsible. This is crucial for forensic analysis and legal proceedings.
Here are some common ways to trace a process.
strace (System Call Tracing)
Command: strace -p
Purpose: Tracks system calls and signals made by a process.
strace -p 1234 # Attach to a running process
[root@client2 ~]# strace -p 496
strace: Process 496 attached
restart_syscall(<… resuming interrupted read …>) = 1
epoll_wait(18, [], 6, 0) = 0
write(3, “\1\0\0\0\0\0\0\0”, 8) = 8
strace ps # Trace the `ps` command
[root@client2 ~]# strace ps
execve(“/usr/bin/ps”, [“ps”], 0x7ffe5539cf40 /* 28 vars */) = 0
brk(NULL) = 0x5592c9cc8000
arch_prctl(0x3001 /* ARCH_??? */, 0x7ffe9b050230) = -1 EINVAL (Invalid argument)
ltrace (Library Call Tracing)
Purpose: Tracks library calls made by a process (e.g., printf, malloc).
Command: ltrace -p <pid>
[root@client2 ~]# ltrace ps # To trace the library calls of ps command
__cxa_atexit(0x563e6802b9c0, 0, 0x563e68033f60, 0x563e68033f58) = 0
strrchr(“ps”, ‘/’)
= nil
gdb (GNU Debugger)
Command: gdb or gdb –pid=1234
Purpose: Debugs the process, allowing breakpoints, memory inspection, etc.
gdb –pid=1234 #To trace a process with id 1234
perf (Performance Analysis)
Purpose: Monitors performance metrics like CPU usage, cache misses.
perf record -p 33795 //Record stats of process 33795, Ctrl+C to quit
perf report //To list the report
perf record -u apache //To record stats of apache processes
top or htop (Real-time Process Monitoring)
Command: top -p <pid> or htop
Purpose: Provides a real-time view of resource usage (CPU, memory).
top -p 1234 #Display usage of process 1234
htop # Interactive, allows filtering by PID
pidstat (Per-Process Statistics)
Purpose: Reports CPU, memory, I/O usage per process.
Command: pidstat -p <pid> <interval>
pidstat -p 1234 1 # Report every second
[root@client2 ~]# pidstat -p 33795 //To display stats for process 33795
05:13:13 AM UID PID %usr %system %guest %wait %CPU CPU Command
05:13:13 AM 0 33795 0.00 0.00 0.00 0.00 0.01 0 httpd
dmesg (Kernel Messages)
Purpose: Tracks kernel messages related to the process.
Command: dmesg | grep <pid> or dmesg -w
dmesg -w # Continuous monitoring
ps (Process Status)
Purpose: Provides detailed process information.
Command: ps -p <pid> -o <fields>
[root@client2 ~]# ps -p 33795 -o pid,ppid,cmd,%cpu,%mem
PID PPID CMD %CPU %MEM
33795 1 /usr/sbin/httpd -DFOREGROUN 0.0 0.5
[root@client2 ~]#
bpftrace (eBPF Tracing)
Allows powerful custom tracing using eBPF.
Key Use Cases
Performance Profiling:
- Analyze CPU usage, memory allocation, and I/O performance.
- Identify performance bottlenecks in applications or the kernel.
System Debugging:
- Track system calls, function calls, and network packets.
- Debug kernel issues or application-specific problems without modifying source code.
Security Monitoring:
- Detect unusual activity like unauthorized system calls.
- Monitor network traffic or file access for anomalies.
Application Monitoring:
- Trace application-specific functions for performance and behavior analysis.
- Capture metrics like latency or error rates dynamically
How bpftrace Works
- Scripts: bpftrace scripts define probes (e.g., function entry, system calls) and attach actions to them.
- Probes: Common types include:
kprobe/kretprobe: Trace kernel functions.uprobe/uretprobe: Trace user-space functions.tracepoint: Attach to predefined kernel tracepoints.usdt: User-defined static tracepoints in applications.profile: Collect samples at regular intervals.
Use Cases in IT Infrastructure (like TechRise)
Monitoring Kubernetes workloads.
Debugging performance issues in containers or VMs.
Enhancing observability for distributed systems.
bpftrace -e ‘tracepoint:syscalls:sys_enter_* { printf(“%s\\n”, comm); }’
bpftrace -e ‘tracepoint:syscalls:sys_enter_open { printf(“%s\\n”, comm); }’
ptrace (Programmatic Tracing)
A system call in Linux that allows one process (the tracer) to observe and control the execution of another process (the tracee). It is commonly used for debugging, system call monitoring, and tamper detection.
Key Use Cases for ptrace
1 Debugging: Core mechanism used by debuggers like gdb.
2 System Call Tracing: Tools like strace use it to trace syscalls.
3 Fault Injection: Modify registers or memory to test software resilience.
4 Security Tools: Monitor or restrict certain actions of a process.
auditctl and ausearch (Audit Framework)
Purpose: Audits all system calls made by the process
auditctl -a always,exit -F pid=<pid> and ausearch -p <pid>
auditctl -a always,exit -F pid=1234
ausearch -p 1234
[root@client2 ~]# ausearch -p 33795 //Tracing an apache process
—-
time->Fri Nov 29 05:59:49 2024
type=PROCTITLE msg=audit(1732877989.366:3523432):
proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=SYSCALL msg=audit(1732877989.366:3523432): arch=c000003e syscall=61 success=yes exit=0 a0=ffffffff a1=7ffdc1809fa4 a2=3 a3=0 items=0 ppid=1 pid=33795 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=”httpd” exe=”/usr/sbin/httpd” key=(null)
coredump for Postmortem Analysis
Purpose: Analyzes the core dump using gdb or apport.
(ulimit -c unlimited) // to enable core dumps.
Start debugging:
gdb ./segfault core.12345
Backtrace (show the call stack):
(gdb) bt
Inspect registers:
(gdb) info registers
Inspect variables:
(gdb) print variable_name
These tools offer various levels of granularity for tracing system calls, library calls, performance, or debugging issues with a process.