The operating system (OS) determines the viable techniques employed in binary exploitation. Different OS architectures offer fun exploit opportunities due to their distinct handling of system-level mechanics like memory management, calling conventions, and security features.
This blog explores how foundational OS topics like System V, POSIX, UNIX, and BSD influence binary exploitation on macOS and Linux. I’ll also cover the System V ABI, inter-process communication (IPC), threading, signaling, and security features like ASLR and stack canaries.
I was inspired to write this post after hearing about some of the DEF CON CTF challenges, and learning that many universities are not covering this information in their OS courses. If you have some spare time, I recommend taking an OS course since that is how I was able to learn the information displayed in this post.
🌸👋🏻 Join 10,000+ followers! Let’s take this to your inbox. You’ll receive occasional emails about whatever’s on my mind—offensive security, open source, boats, reversing, software freedom, you get the idea.
System V (SysV)
SysV Application Binary Interface (ABI)
System V introduced the SysV ABI, which establishes the low-level binary interface for compiled programs. It outlines conventions for function calls, argument passing, and stack management.
Many UNIX-like systems, including Linux, use this ABI. The SysV ABI is a core element of binary exploitation, as it governs how I can manipulate the stack, registers, and control flow during an attack.
Specifically, knowledge of the ABI allows me to craft exploits that interact with the target system’s calling conventions, stack management, and register usage.
Calling convention
Calling conventions define how function arguments are passed (e.g., via registers or the stack) and how return values are handled.
In x86-64 SysV ABI (Linux and macOS), the first six integer or pointer arguments are in registers RDI, RSI, RDX, RCX, R8, and R9. The return value is in RAX.
Read more: MIPS Assembly: Data, Registers, and Mimicking Scope – Olivia A. Gallucci
Simple function call
- _start
- Entry point of the program.
- Function call
mov rdi, 5andmov rsi, 10load the first and second arguments (5 and 10) intoRDIandRSI, respectively.- The
call add_numbersinstruction transfers control to theadd_numbersfunction.
- Function logic
- Inside
add_numbers, the two arguments (RDIandRSI) are added together withadd rdi, rsi. - The result is stored back in
RDI, and then moved toRAXto be returned.
- Inside
- Return to caller
- The
retinstruction returns control to the caller, which is_startin this case.
- The
- Storing result
- The return value in
RAXis stored in the memory locationresult.
- The return value in
- Exit
- The program exits using the
exitsystem call (mov rax, 60,xor rdi, rdi,syscall).
- The program exits using the
Stack management
Within the SysV ABI, the stack management specifies how to use the stack for local variables, return addresses, and function arguments.
Remember, the stack stores local variables, return addresses, and sometimes function arguments. The stack pointer (RSP) is adjusted as data is pushed onto or popped from the stack.
🌸👋🏻 Join 10,000+ followers! Let’s take this to your inbox. You’ll receive occasional emails about whatever’s on my mind—offensive security, open source, boats, reversing, software freedom, you get the idea.
Stack usage for local variables
This program calls the use_stack function, which sets up a stack frame to allocate space for two local variables, stores the values 42 and 24 in these variables, and then cleans up the stack frame before returning to the caller. Finally, the program exits.
“Cleans up the stack frame before returning to the caller” means that the function restores the stack pointer (rsp) and base pointer (rbp) to their original values, ensuring the stack is in the same state as before the function was called, preventing any disruption or corruption in the rest of the program.
Register usage
In the SysV ABI, register usage specifies which registers are for which purposes, and which must preserve across function calls.
Certain registers are caller-saved (volatile), and others are callee-saved (non-volatile). For example, registers like RDI, RSI, RDX, etc., are caller-saved, while RBX, RBP, R12–R15 are callee-saved.
Preserving registers across function calls
This program calls a function modify_registers to temporarily change the values of registers RBX and R12, preserving their original values using the stack, and then exits the program.
System calls
System calls define how software interacts with the kernel, detailing how to pass arguments to system calls and handle return values.
In Linux, system calls are made by placing the syscall number in RAX, and the arguments in RDI, RSI, RDX, etc. The syscall instruction is then used to transition to kernel mode.
🌸👋🏻 Join 10,000+ followers! Let’s take this to your inbox. You’ll receive occasional emails about whatever’s on my mind—offensive security, open source, boats, reversing, software freedom, you get the idea.
System call for writing to standard output
This program prints “Hello, world!” to the terminal and exits.
Resources
- System V ABI osDev
- System V Application Binary Interface AMD64 Architecture Processor Supplement (With LP64 and ILP32 Programming Models) by H.J. Lu, Michael Matz, Milind Girkar, Jan Hubicka, Andreas Jaeger, Mark Mitchell
SysV Inter-Process Communication (IPC)
System V also introduced various IPC mechanisms, like message queues, semaphores, and shared memory. These mechanisms enable processes to communicate and synchronize with each other. Exploiting vulnerabilities in how a binary handles IPC can be a vector for attacks.
Message queues
Message queues are an IPC method for exchanging messages in a controlled and asynchronous manner. Each message queue is identified by a unique key, and processes can send and receive messages via these queues (source).
The messages are often stored in a queue and retrieved in a First-In-First-Out (FIFO) order, although priority-based retrieval is also possible. Message queues are useful for tasks where processes need to communicate discrete packets of data.
Example of how message queues work in a unix-like OS
Example
A server process sends task completion status updates to a client process via a message queue. If a process doesn’t properly validate messages, I could send malicious messages to manipulate the process’s behavior.
Semaphores
Another IPC mechanism is semaphores. Semaphores are synchronization tools that control multiple processes’ access to a shared resource to prevent race conditions. They can be thought of as counters that manage access to shared resources.
Two operations are associated with semaphores: wait (or P operation) and signal (or V operation). The wait operation decrements the semaphore value, and the process is blocked if the value becomes negative. The signal operation increments the semaphore value, potentially waking up a blocked process.
Example
Semaphores are used to ensure that only one process at a time can access a critical section of code, such as writing to a log file. If I can manipulate semaphore values, I could disrupt process synchronization, causing deadlocks or race conditions.
Shared memory
Shared memory is an IPC mechanism that allows multiple processes to access the same block of memory.
Unlike message queues, where data is copied between processes, shared memory allows processes to directly read and write to the same memory location, making it the fastest IPC method.
However, it requires careful synchronization (often using semaphores) to prevent data corruption caused by concurrent access.
🌸👋🏻 Join 10,000+ followers! Let’s take this to your inbox. You’ll receive occasional emails about whatever’s on my mind—offensive security, open source, boats, reversing, software freedom, you get the idea.
Example
Two processes can share a buffer in memory, where one process writes data to the buffer and the other reads from it. If access to shared memory is controlled improperly, I might try to read or modify sensitive data, leading to information leaks or corruption.
POSIX
Now, having established a solid understanding of the System V ABI and its role in binary exploitation, it’s essential to recognize the broader standards that have influenced UNIX-like OS-es. The Portable Operating System Interface (POSIX) is one of the most significant.
POSIX standardizes application interaction with UNIX-like OS-es, ensuring portability across platforms like Linux and macOS through consistent APIs, threading models, and signal handling.
These APIs are not the same as a web API! These APIs work with your computer’s OS rather than a service over the web like the GitHub API.
Standardization
The IEEE maintains the standards and specifies how applications should interact with the OS kernel and system libraries. This standardization means that binary exploitation techniques developed on one POSIX-compliant system (like Linux) are often transferable to another (like macOS).
For instance, the POSIX’s standardization implies that some buffer overflows, format string attacks, or return-oriented programming (ROP) can be developed and tested on one POSIX-compliant system and then applied to another with minimal adjustments.
However, while the underlying POSIX standards provide a common ground, there are differences in implementation, system architecture, and security mechanisms between different OS-es.
For example, macOS uses the Mach-O binary format, while Linux typically uses the ELF format. They also have different memory protections and security features. These differences mean that the specifics of an attack usually require adaptation to the target system’s particularities.
The Mach-O and ELF formats are binary file formats used by OS-es to execute programs and manage their executable code, shared libraries, and other binary data.
Let’s explore some subdomains of POSIX.
APIs for system calls
Example
open- System call opens or creates a file. It’s part of the POSIX standard. The first argument is the file name, the second specifies flags (e.g.,
O_CREATto create the file if it doesn’t exist), and the third is the file mode (e.g.,0644for read/write permissions).
- System call opens or creates a file. It’s part of the POSIX standard. The first argument is the file name, the second specifies flags (e.g.,
write- System call writes data to the file. The first argument is the file descriptor, the second is the data to write, and the third is the number of bytes to write.
read- System call reads data from the file. The first argument is the file descriptor, the second is the buffer where the data will be stored, and the third is the number of bytes to read.
close- System call closes the file descriptor.
Cross-platform compatibility
Because this code uses POSIX-compliant system calls (open, write, read, close), it will work on any POSIX-compliant OS, such as Linux or macOS. This portability means I can compile and run this code on both OS-es without modifications.
… or very little modification depending on the situation.
Threading and signals
Cool podcast under a similar name: Signals and Threads.
POSIX also defines how threading and signal handling should be implemented, which can also be avenues for exploitation.
For example, race conditions in multi-threaded applications or improper signal handling can lead to vulnerabilities.
In fact, race conditions and signal handling flaws are common avenues for exploitation in POSIX-compliant systems.
TLDR definitions:
- Threading is about managing multiple sequences of execution.
- Signaling is about communication through notifications between processes or threads.
- Multi-threading involves running multiple threads concurrently within a single process.
- Race conditions arise when the timing or order of thread execution leads to unpredictable or erroneous program behavior, often resulting in vulnerabilities.
Threading
Threading is the process of creating and managing multiple sequences of execution within a single program. A thread is the smallest unit of processing that an OS can schedule. In a threaded application, multiple threads can run concurrently, sharing the same memory space and resources.
Viewing threads
top or htop
top or htop can be used on Linux and macOS.
top output example 
htop output exampleps
To list all threads for a specific process, use:
ps -T -p <PID>This command shows all of the process’ threads with the specified PID.
Signaling
Signaling is how processes and threads can communicate with each other or with the OS to notify of events, trigger actions, or handle exceptional conditions.
Signals are asynchronous notifications sent to a process or thread to prompt the execution of a signal handler, a specific function designed to handle such signals.
🌸👋🏻 Join 10,000+ followers! Let’s take this to your inbox. You’ll receive occasional emails about whatever’s on my mind—offensive security, open source, boats, reversing, software freedom, you get the idea.
Viewing signals
strace (linux) or dtruss (macOS)
strace traces system calls and signals. To trace signals sent to a specific process:
# Linux
strace -e signal=<SIGNAL> -p <PID>
# macOS
sudo dtruss -p <PID>Here, I would replace <SIGNAL> with the signal name (e.g., SIGINT) and <PID> with the Process ID I want to monitor.
To trace all signals, use:
# Linux
strace -e trace=signal -p <PID>
# macOS
sudo dtrace -n 'proc:::signal-send /pid == <PID>/ { printf("%s -> %s: signal %d\n", execname, pid, arg1); }'proc:::signal-send- This probes the signal sending events.
/pid ==/ - This condition restricts the trace to the specified PID.
printf- This command prints out the process name, the PID, and the signal number.
kill
kill is used to send signals to processes. I view the signals I can send with:
kill -lThis command lists all available signals. Again, each signal has a number and a name (e.g., SIGTERM, SIGKILL).
POSIX threads (Pthreads)
POSIX specifies a threading model (Pthreads) that allows for concurrent execution within a process. However, using multiple threads can introduce race conditions—where the operation outcome depends on the threads’ unpredictable timing.
If a program does not correctly manage access to shared resources (e.g., using mutexes or other synchronization mechanisms), it could lead to inconsistent states that I might exploit to gain unauthorized access or cause unintended behavior.
Mutexes are synchronization primitives used in multi-threaded programming to ensure that only one thread can access a shared resource or critical section at a time, preventing race conditions.
Many real world examples are found of this: “Time-of-Check to Time-of-Use” (TOCTOU) vulnerabilities.
Read more: How to manipulate the execution flow of TOCTOU attacks – Olivia A. Gallucci
Signal handling
Lastly, POSIX defines how signals (asynchronous notifications sent to a process or thread) should be managed. Improper signal handling—such as not properly blocking signals during critical sections of code or incorrectly handling signals that interrupt system calls—can create vulnerabilities. For instance, I might exploit a race condition or signal mismanagement to crash a program or execute arbitrary code.
There are many ways of doing this:
- Exploiting signal handler reentrancy,
- Exploiting race conditions with signals,
- Exploiting signal handlers to bypass security checks, and
- Exploiting signals to interrupt system calls.
POSIX conclusion
The primary takeaway from this is that the POSIX standardization facilitates the transfer of exploitation techniques across systems, but the real challenge lies in the unique implementation details of each system.
With a solid understanding of POSIX and its role in standardizing interactions across UNIX-like systems, let’s delve into the UNIX OS itself. UNIX, the foundation upon which POSIX was built, has a profound impact on computing. By exploring UNIX, I feel I can better appreciate how these standards shaped the development of Linux and macOS, and how they influence binary exploitation.
🌸👋🏻 Join 10,000+ followers! Let’s take this to your inbox. You’ll receive occasional emails about whatever’s on my mind—offensive security, open source, boats, reversing, software freedom, you get the idea.
UNIX
Unix philosophy
The UNIX philosophy emphasizes small, modular programs that do one thing well, often composed to perform complex tasks. Understanding this philosophy helps me anticipate patterns in program design and locate potential vulnerabilities.
Example
Command chaining with pipes
cat /var/log/syslog | grep "error" | sort | uniqThis command is a classic example of the UNIX philosophy in action. Here’s what each component does:
cat /var/log/syslog- Read the contents of the syslog file.
grep "error"- Filters the log entries to show only those containing the word “error.”
sort- Sorts the filtered log entries.
uniq- Removes duplicate entries from the sorted list.
Each command performs a single, well-defined task, and they are chained together using pipes (|) to create a more complex operation. The modular nature of this setup allows flexibility and reusability, but it also introduces potential vulnerabilities if not carefully managed.
Attack: Command injection
Since I know this pattern, I might target the command chain for injection attacks. For instance, if the command were part of a script that takes user input, such as:
#!/bin/bash
cat $1 | grep "error" | sort | uniqHere, $1 is a user-supplied argument, potentially leading to command injection if not properly sanitized. I could supply a malicious input like:
; rm -rf / ; #When passed as an argument, this input could cause the script to execute the rm -rf / command, attempting to delete the root directory and cause catastrophic damage.
File permissions and system calls
UNIX systems rely heavily on file permissions and system calls, which are often targets for binary exploitation. For example, privilege escalation often involves exploiting binaries that mishandle file permissions or system calls.
🌸👋🏻 Join 10,000+ followers! Let’s take this to your inbox. You’ll receive occasional emails about whatever’s on my mind—offensive security, open source, boats, reversing, software freedom, you get the idea.
Example
Exploiting SUID binary to achieve privilege escalation
Imagine I have a Set User ID (SUID) binary on a UNIX system. SUID binaries run with the privileges of the file owner, not the user who runs them. If a binary is owned by root and has the SUID bit set, it will execute with root privileges, regardless of which user runs it.
Suppose this SUID binary has a vulnerability in how it handles file operations, such as improper use of the open() system call.
Vulnerability
Consider the binary allows a user to write to a file that should only be writable by the root. The binary doesn’t properly check the file path before performing the write operation, and the open() system call is used in a vulnerable manner.
Here is a simple example:
The program writes the string “Hello, world!\n” to a file.
Exploitation
1. Identify the SUID binary
The first step is to identify the SUID binary. I can do this with the following command:
find / -perm -4000 2>/dev/null2. Examine the binary
Let’s say I find a SUID binary /usr/local/bin/vulnprog that matches the above code. The binary is owned by root and has the SUID bit set:
-rwsr-xr-x 1 root root 12345 Aug 11 08:00 /usr/local/bin/vulnprog3. Exploit the vulnerability
The binary allows me to specify a filename as an argument. Because the open() system call does not check for symbolic links, I can create a symbolic link to a critical system file, like /etc/passwd, where user credentials are stored.
ln -s /etc/passwd /tmp/malicious_fileNow, execute the vulnerable SUID binary with the symbolic link as the argument:
/usr/local/bin/vulnprog /tmp/malicious_file4. Privilege escalation
The vulnerable binary writes to /tmp/malicious_file, which is a symbolic link to /etc/passwd. If I modify the content being written, I can inject a new user with root privileges or alter the root password.
For example, adding a new root user:
echo "attacker::0:0:root:/root:/bin/bash" >> /etc/passwdAfter this, I can switch to the new user:
su attackerAnd now I have root privileges.
Berkeley Software Distribution (BSD)
Derivatives and influence
BSD is a branch of UNIX, and many of its innovations have influenced other OS-es, including macOS.
MacOS is based on Darwin, a BSD variant.
How BSD-derived systems manage memory and processes informs how I craft exploits, mainly when targeting vulnerabilities like buffer overflows or race conditions.
🌸👋🏻 Join 10,000+ followers! Let’s take this to your inbox. You’ll receive occasional emails about whatever’s on my mind—offensive security, open source, boats, reversing, software freedom, you get the idea.
Memory management
Let’s compare memory management in SysV and BSD. We will start with SysV, since we covered that previously.
SysV
Memory management in SysV systems traditionally involves a more segmented memory model, and the use of shared memory, semaphores, and message queues (SysV IPC). These mechanisms have implications for how memory is allocated and managed, and thus, impact how certain classes of vulnerabilities might be exploited.
SysV systems often employ a simpler memory model, which can limit the types of memory-related vulnerabilities (such as certain heap exploits) compared to more modern systems.
In SysV systems, heap exploits such as heap overflows and use-after-free vulnerabilities are limited due to the simpler and more segmented memory model, reducing the likelihood of complex memory corruption scenarios.
BSD
BSD systems, including macOS, tend to use a more advanced virtual memory system. It introduced concepts like copy-on-write, and memory-mapped files (via mmap).
The VM system in BSD is often more sophisticated, which allows for complex memory exploit strategies but also involves different mitigations (like Address Space Layout Randomization (ASLR)) that can make exploitation challenging.
Calling convention and system-level mechanics
Now, let’s compare SysV’s and BSDs calling conventions and system-level mechanics.
The calling convention and system-level mechanics in macOS largely follow the System V ABI. However, it does exhibit some differences from the standard System V ABI found in Linux.
TLDR comparison:
- Calling convention
- Very similar, with a small but critical difference in stack alignment due to macOS’s handling of the return address.
- System calls
- The mechanics are similar, but macOS has a different numbering scheme and additional abstraction layers.
- Stack management
- Largely the same, with the noted alignment difference.
- Register usage
- Mostly the same across both systems.
- Exception handling
- More OS-specific differences, particularly when considering Objective-C.
Below is a more detailed comparison of macOS specifics to the System V ABI.
Calling convention
- The first six integer/pointer arguments are passed in
RDI,RSI,RDX,RCX,R8, andR9. - Functions return results in
RAX(for integers) orXMM0(for floating-point values). - Additional arguments beyond the sixth are passed on the stack.
- macOS: However, macOS has a quirk where stack alignment on entry to a function is typically 16 bytes minus 8 (i.e., 8-byte alignment). This is due to the way macOS handles the return address on the stack.
- After pushing the return address, the stack is 16-byte aligned.
System calls
- System calls in Linux are invoked using the
syscallinstruction. - The system call number is placed in
RAX, and arguments are passed inRDI, RSI, RDX, R10, R8,andR9. - The result of the system call is returned in
RAX, with negative values indicating errors. - macOS: System call numbers are different, often prefixed by
0x2000000. macOS also uses higher-level APIs, reducing directsyscallusage. Explanation:- Specific macOS system calls may have different numbers and slight behavioral differences due to the Mach kernel and BSD heritage.
- The numbering scheme for system calls in macOS differs significantly from Linux. Many macOS system calls are prefixed by
0x2000000. - macOS often uses special APIs or libraries (like Core Foundation, Grand Central Dispatch) that abstract some direct system call usage, which may lead to fewer direct invocations of raw system calls compared to Linux.
Stack management
- Stack frames are 16-byte aligned when a function is called.
- macOS: Stack alignment is 8-byte aligned upon function entry. This difference in alignment can cause subtle bugs if assembly code is ported from Linux to macOS without accounting for this difference.
Register usage – identical for Linux and macOS
- Caller-saved:
RAX, RCX, RDX, RSI, RDI, R8, R9, R10, R11. - Callee-saved:
RBX, RBP, R12, R13, R14, R15. - Instruction pointer:
RIP. - Condition flags:
RFLAGS.
Exception handling and stack unwinding
Stack unwinding is the process during exception handling where the program traverses the call stack, cleaning up resources and executing destructors for objects as it exits each function, until it reaches the appropriate exception handler. Here, stack unwinding affects binary exploitation by potentially altering the control flow during exception handling.
- Both systems follow a standardized approach with DWARF debug information for stack unwinding.
- DWARF is a standardized debugging file format.
- macOS: Differences exist in Objective-C runtime exceptions (differ from C++ exceptions). It uses different metadata and mechanisms for stack unwinding, particularly with the Mach-O binary format.
Security features
BSD systems often include security features like Write XOR Execute (W^X), Address Space Layout Randomization (ASLR), and stack canaries. Traditional buffer overflow exploits are often ineffective due to these defenses.
As a result, I might need to rely on techniques like ROP, which chains together small code snippets (gadgets) already present in the binary to achieve code execution.
🌸👋🏻 Join 10,000+ followers! Let’s take this to your inbox. You’ll receive occasional emails about whatever’s on my mind—offensive security, open source, boats, reversing, software freedom, you get the idea.
Write XOR execute (W^X)
This security policy ensures that memory pages cannot be both writable and executable simultaneously. This policy is a defense against exploits that attempt to inject and execute code on the stack or heap.
On BSD systems like OpenBSD, which pioneered W^X, this feature is strictly enforced, complicating traditional exploitation techniques (aka plain binary exploitation). In these situations, crafting an exploit in a W^X environment may involve finding writable and executable memory regions or leveraging ROP to execute code without violating the W^X policy.
What to look for
I usually follow these steps to check if the Write XOR Execute (W^X) policy is enabled on macOS or Linux.
Linux
1. Check kernel parameters
I often check if W^X is enforced by looking at certain kernel parameters.
cat /proc/sys/vm/mmap_min_addrThis command shows the minimum address that can be mapped, which can indicate if certain security features are enabled.
I also check the dmesg logs for any mentions of “NX” (No-eXecute), which indicates that memory pages cannot be both writable and executable.
dmesg | grep NXIf W^X is enforced, I usually see something like:
NX (Execute Disable) protection: active2. Using proc filesystem
I inspect /proc/self/maps to see the memory regions of a process. If W^X is enforced, I should see that writable segments are not executable.
cat /proc/self/mapsIn the output, I usually see regions marked with either r-xp (readable, executable, private) or rw-p (readable, writable, private), but not rwxp (readable, writable, executable, private).
macOS
1. Check system integrity protection (SIP)
macOS enforces a similar protection as W^X through System Integrity Protection (SIP). To check if SIP is enabled:
csrutil statusIf SIP is enabled, the output will be:
System Integrity Protection status: enabled.2. Use vmmap or procinfo
On macOS, I inspect the memory layout of a running process using vmmap. If W^X is enforced, I should not see any memory regions that are both writable and executable.
vmmap <pid>Address Space Layout Randomization (ASLR)
ASLR randomizes the memory addresses used by system and application processes, making it more difficult to predict the location of my payloads.
In BSD-based systems like macOS, ASLR is implemented to reduce the chances of successful memory corruption attacks. Thus, I need to devise methods to bypass ASLR, such as through information leaks or ROP.
🌸👋🏻 Join 10,000+ followers! Let’s take this to your inbox. You’ll receive occasional emails about whatever’s on my mind—offensive security, open source, boats, reversing, software freedom, you get the idea.
Stack canaries
Stack canaries are security mechanisms placed before the return pointer on the stack to detect buffer overflows. If a buffer overflow occurs and modifies the return address, the canary value will also be altered, triggering a security exception and preventing the exploit. BSD systems, including macOS, use stack canaries to protect against stack-based buffer overflows, requiring exploit developers to find ways to bypass or defeat this protection.
What to look for
This program shows how a stack canary can detect a buffer overflow that alters the return address. Note that modern systems implement stack canaries with additional complexity, using randomization and XOR operations to make them harder to guess or manipulate. This code does not reflect the full complexity of real-world implementations.
- Canary initialization
- The
canary_valueis initialized to0xDEADBEEF. This is a dummy value representing the canary in this example.
- The
- Copy of the canary
- When
vulnerable_functionis called, a copy of the canary is made locally.
- When
- Buffer overflow simulation
- The
strcpyfunction copies user input into a fixed-size buffer,buffer[64]. Sincestrcpydoes not perform bounds checking, the input can overflow the buffer and potentially overwrite the return address and the canary.
- The
- Canary check
- After the buffer operation, the canary is checked to see if it still matches the original value. If the canary value has been altered, it indicates a buffer overflow occurred, and the program prints an error message and exits.
- Output
- If the canary value remains unchanged, the function executes successfully without detecting a buffer overflow.
How to test
- Normal input
- Providing an input string shorter than 64 characters will result in no overflow, and the function will execute normally.
- Overflow attempt
- Providing an input string longer than 64 characters may alter the canary value, trigger the overflow detection, and terminate the program.
Summarized relevance to binary exploitation
Control flow hijacking
Exploits like buffer overflows rely on knowledge of the ABI, calling conventions, and memory layout (all influenced by System V and POSIX).
Exploiting system calls
Many binary exploits involve invoking system calls in unintended ways. POSIX and the underlying UNIX/BSD kernel define the specific system calls available and their behavior.
Privilege escalation
On UNIX and UNIX-like systems, gaining higher privileges (e.g., root) is often the goal of an exploit. Understanding the nuances of file permissions, user IDs, and process control, which are all part of the UNIX and POSIX standards, can help achieve that goal.
Conclusion
Binary exploitation on macOS and Linux require distinct methods as these OS-es manage system-level mechanics differently. System V, POSIX, UNIX, and BSD have shaped how binaries are structured and executed on these platforms, influencing their ABIs and their implementation of security mechanisms like ASLR, stack canaries, and W^X. Knowledge of how they operate enables me to predict, identify, and exploit vulnerabilities in binaries running on these platforms.
If you enjoyed this post on OS-es and binary exploitation, consider reading how to manipulate the execution flow of TOCTOU attacks.
You must be logged in to post a comment.