RLCTHD

/home

/posts

Home > Posts > Why Userspace ELF Loading isn't Stealthy

Why Userspace ELF Loading isn't Stealthy

Quickly On AI generated content #

When reading technical details about how Linux works (this probably applies to most content online nowadays and chats with LLMs), treat the following as red flags:

  • Invented function names that only sound plausible
  • Collapsing multiple call layers into fake high-level summaries
  • Mixing historical kernels and academic descriptions
  • Mentioning functions that cannot be found in the kernel via grep
  • Avoiding concrete call graphs or code references
  • Relying on phrases like “high level actions” instead of actually showing the control flow

The following is a very brief example of an incorrect description I encountered while assembling notes and doing my research on this topic. It might look okay at a glance, because there’s a lot of buzzwords in the article, lots of simplified bulletpoints, images related to the subject, etc.

Example:

“Inside kernel_init(), the kernel calls do_basic_setup() and then init_post() to launch userspace init.”

There is no init_post() function in modern Linux. init_post() was a specific function in older Linux kernel versions (you can find it in versions like v2.6 for instance) that handled the final transition from kernel space to user space. There is no justification for a recent 2026 article to describe current Linux behavior using obsolete code references.

Userspace init is launched by kernel_init() via run_init_process() after late initialization has completed.

What I’m trying to say is that any explanation that invents functions or obscures the call chain should be treated as untrusted. Read kernel code. Verify claims. And if I get something wrong in the articles that follow, correct me.

How This Started #

An academic assignment requiring a custom loader capable of handling statically linked PIE and non-PIE binaries is what pushed me to dig deeper into how this actually works. I ended up building a custom loader: https://github.com/realcathode/elf-loader (which is a much simplified version of the linux/fs/binfmt_elf.c loader at https://github.com/torvalds/linux)

Custom Loaders & Misconceptions #

A custom ELF loader reimplements the ELF loading semantics normally performed inside the kernel during execve(), while still relying on an already existing process. In practice, creating an ELF loader means parsing ELF headers, mapping loadable segments, constructing a valid memory layout, and transferring control to an entry point.

Unlike Windows, where custom loaders are often used to evade user-mode visibility, writing an ELF loader on Linux does not meaningfully reduce kernel-level observability. It only reimplements part of the execve() loading contract in userspace.

I initially assumed it works similarly on Linux, largely because largely because my background at the time was focused on offensive tooling and understanding how EDR and antivirus products detect malicious execution on Windows. That assumption turns out to be incorrect.

On Windows, custom loaders are a standard technique for more stealthy execution because the PE loader lives in user space and is heavily instrumented by defensive tooling. Replacing it can materially reduce user-mode visibility. This condition does not exist on Linux, where program loading is a kernel-enforced operation.

How Programs are Run #

Every process on Linux (except init) is born as a clone of its parent via fork(). But fork() only duplicates the existing code. To run new code, the process calls execve().

Historically, execve() was the only syscall used to execute a new program image. Since Linux 3.19 (early 2015), this role is shared with execveat(), which generalizes program execution to file descriptors and directory relative paths.

The first userspace process (PID 1) is not forked from another userspace process. It originates as a kernel thread, created during late kernel initialization. That kernel thread is then transformed into a userspace process via execve().

init is the root of the userspace process tree, and the only process whose creation does not involve fork().

When a new program is executed via execve() or execveat(), the kernel takes on the responsibility of loading the ELF binary and establishing the process’s initial execution context. It is Parsing the ELF headers to identify loadable segments (PT_LOAD), interpreters (PT_INTERP), and entry points. After that it maps virtual memory areas (VMAs) for code, data, and BSS segments, with correct protections. Then set up the initial user stack, including command-line arguments (argv), environment variables (envp), and the auxiliary vector (auxv) containing execution metadata provided by the kernel. Finally Transfer control to the binary’s entry point (usually _start), marking the transition from kernel space to user space.

Trust the Kernel #

The kernel is the trusted transition point, because it enforces memory isolation, ensures the ABI contract, and maintains security invariants. For example, EDRs and Linux Security Modules rely on this boundary to validate the legitimacy of executed programs, audit activity, and enforce access control.

Custom Loaders are Not Stealthy #

Normal ELF execution results in:

  • RX text segments
  • RW data segments
  • No RWX mappings
  • Predictable stack placement
  • VMAs consistent with ELF headers

Red flags are:

  • Anonymous RX mappings
  • Code executing from heap or stack
  • RX pages with no file backing
  • Frequent mprotect(PROT_EXEC) on anonymous memory

Suspicious behavior

  • Jumping into executable memory without a corresponding exec event
  • Sudden instruction pointer relocation without exec
  • Self‑modifying executable regions

EDRs and sandboxes validate:

  • Presence of AT_ENTRY, AT_PHDR, AT_RANDOM
  • Reasonable stack layout
  • A valid auxv vector

Everything described so far can be directly observed on a running system. This code is going to be used as an example:

#include <stdio.h>  
#include <stdlib.h>  
  
int main(int argc, char *argv[], char *envp[]) {  
   printf("[*] Hello payload!\n");  
   printf("[*] Received %d arguments.\n", argc);  
  
   for(int i = 0; i < argc; i++) {  
       printf("    argv[%d]: %s\n", i, argv[i]);  
   }  
  
   if (envp[0]) {  
       printf("[*] First environment variable: %s\n", envp[0]);  
   }  
  
   return 69;  
}

$ gcc -static-pie payload.c -o payload

$ file payload  
payload: ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), static-pie linked, BuildID[sha1]=198488d86a4eaeb31e5d445dad1e68725ebfe3fd, for GNU/Linux 3.2.0, not stripped

This is a statically linked PIE executable that requires relocation but no dynamic linker, making it ideal for experimentation with userspace ELF loading. After running it through the custom loader, everything works like expected:

$ ./elf-loader ../tests/payload  
[DEBUG] SP after setting AUXV: 0x706c3f1feec0 (0)  
[DEBUG] Final Entry Point: 0x706c3f1feb90 (0)  
[DEBUG] Transfer control  
  
[*] Hello payload!  
[*] Received 1 arguments.  
   argv[0]: ../tests/payload  
[*] First environment variable: SHELL=/bin/bash

$ echo $?
69

To learn more about what happens when this executable is run, let’s use strace:

strace is a useful diagnostic, instructional, and debugging tool. System administrators, diagnosticians, and troubleshooters will find it invaluable for solving problems with programs for which source code is not readily available, as recompilation is not required for tracing. Students, hackers, and the overly-curious will discover that a great deal can be learned about a system and its system calls by tracing even ordinary programs. Programmers will find that since system calls and signals occur at the user/kernel interface, a close examination of this boundary is very useful for bug isolation, sanity checking, and attempting to capture race conditions. - https://man7.org/linux/man-pages/man1/strace.1.html

$ strace ./elf-loader ../tests/payload  
execve("./elf-loader", ["./elf-loader", "../tests/payload"], 0x7ffc143e3638 /* 78 vars */) = 0  
brk(NULL)                               = 0x55556cc1a000  
brk(0x55556cc1ad00)                     = 0x55556cc1ad00  
arch_prctl(ARCH_SET_FS, 0x55556cc1a380) = 0  
set_tid_address(0x55556cc1a650)         = 14852  
set_robust_list(0x55556cc1a660, 24)     = 0  
rseq(0x55556cc1aca0, 0x20, 0, 0x53053053) = 0  
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0  
readlinkat(AT_FDCWD, "/proc/self/exe", "/home/cation/Documents/elf-loade"..., 4096) = 48  
getrandom("\xe4\xf7\x22\x04\xa8\xde\x23\xff", 8, GRND_NONBLOCK) = 8  
brk(NULL)                               = 0x55556cc1ad00  
brk(0x55556cc3bd00)                     = 0x55556cc3bd00  
brk(0x55556cc3c000)                     = 0x55556cc3c000  
mprotect(0x7beb008de000, 20480, PROT_READ) = 0  
openat(AT_FDCWD, "../tests/payload", O_RDONLY) = 3  
fstat(3, {st_mode=S_IFREG|0775, st_size=822656, ...}) = 0  
mmap(NULL, 822656, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7beb00756000  
close(3)                                = 0  
mmap(NULL, 765952, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7beb0069b000  
mmap(0x7beb0069b000, 28672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7beb0069b000  
mprotect(0x7beb0069b000, 28672, PROT_READ) = 0  
mmap(0x7beb006a2000, 520192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7beb006a2000  
mprotect(0x7beb006a2000, 520192, PROT_READ|PROT_EXEC) = 0  
mmap(0x7beb00721000, 163840, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7beb00721000  
mprotect(0x7beb00721000, 163840, PROT_READ) = 0  
mmap(0x7beb00749000, 53248, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7beb00749000  
mprotect(0x7beb00749000, 53248, PROT_READ|PROT_WRITE) = 0  
mmap(NULL, 8388608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7beaffe00000  
getrandom("\x5a\xf1\xb4\x0d\x5f\x96\xcd\xa7\xf2\x4a\x8c\x73\x88\xb7\x4a\x5d", 16, 0) = 16  
fstat(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0x1), ...}) = 0  
write(1, "[DEBUG] SP after setting AUXV: 0"..., 50[DEBUG] SP after setting AUXV: 0x7beb005feec0 (0)  
) = 50  
write(1, "[DEBUG] Final Entry Point: 0x7be"..., 46[DEBUG] Final Entry Point: 0x7beb005feb90 (0)  
) = 46  
write(1, "[DEBUG] Transfer control\n", 25[DEBUG] Transfer control  
) = 25  
write(1, "\n", 1  
)                       = 1  
brk(NULL)                               = 0x55556cc3c000  
brk(0x55556cc3cd00)                     = 0x55556cc3cd00  
arch_prctl(ARCH_SET_FS, 0x55556cc3c380) = 0  
set_tid_address(0x55556cc3c650)         = 14852  
set_robust_list(0x55556cc3c660, 24)     = 0  
rseq(0x55556cc3cca0, 0x20, 0, 0x53053053) = -1 EINVAL (Invalid argument)  
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0  
readlinkat(AT_FDCWD, "/proc/self/exe", "/home/cation/Documents/elf-loade"..., 4096) = 48  
getrandom("\x03\x83\x48\x72\x9c\x37\x31\x21", 8, GRND_NONBLOCK) = 8  
brk(NULL)                               = 0x55556cc3cd00  
brk(0x55556cc5dd00)                     = 0x55556cc5dd00  
brk(0x55556cc5e000)                     = 0x55556cc5e000  
mprotect(0x7beb00749000, 20480, PROT_READ) = 0  
fstat(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0x1), ...}) = 0  
write(1, "[*] Hello payload!\n", 19[*] Hello payload!  
)    = 19  
write(1, "[*] Received 1 arguments.\n", 26[*] Received 1 arguments.  
) = 26  
write(1, "    argv[0]: ../tests/payload\n", 30    argv[0]: ../tests/payload  
) = 30  
write(1, "[*] First environment variable: "..., 48[*] First environment variable: SHELL=/bin/bash  
) = 48  
exit_group(69)                          = ?  
+++ exited with 69 +++

First thing we notice, there is indeed only one execve call. Does it mean the ELF binary executed silently and nobody noticed?

From the kernel, LSM, and EDR perspective: elf-loader is the executed program, test1PIE is just data, not a program and there is no second exec event. Everything that follows happens inside the same process image.

Then there is this:

mmap(..., PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)  
mmap(... PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0)  
mprotect(..., PROT_READ) = 0

mmap(..., PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0)
mprotect(..., PROT_READ|PROT_EXEC)

Two of the red flags mentioned before: Anonymous RX mappings, RX pages with no file backing.

Every loadable segment follows a pattern of mapping readable/writable anonymous memory, and then doing mprotect(..., PROT_READ, PROT_EXEC). This is allowed, but abnormal and noisy. So we also have Frequent mprotect(PROT_EXEC) on anonymous memory from the list.

After this:

write(1, "[DEBUG] Transfer control\n", 25[DEBUG] Transfer control

There is no system call corresponding to execve or execveat, but from the kernel’s perspective the same PID/task is suddenly executing code that did not exist. This means Sudden instruction pointer relocation without exec.

By looking at the code, notice how the stack is created properly, it follows the ABI guidelines, however, from a sandbox perspective, the stack is not kernel-generated, its virtual memory area is anonymous. These are strong signals for: EDRs and sandboxes often validate reasonable stack layout.

One more huge tell is this sequence that appears twice:

arch_prctl(ARCH_SET_FS, ...)
set_tid_address(...)
set_robust_list(...)
rseq(...)
prlimit64(...)
getrandom(...)
brk(...)

Once for the elf-loader, and the second time for the payload. In a normal execution, libc init happens once, here it is initialized twice, witihout an exec. This alone is probably enough to trigger an alert. So Jumping into executable memory without a corresponding exec event

What Does Not Trigger #

The current loader avoids, RWX mappings, executable stack, broken auxv, and invalid ELF headers. This is why the program doesn’t crash.

Blend In #

The kernel decides when a program begins, what memory layout is valid, how the stack must look, and what metadata accompanies the transition into user space. Security tools rely that boundary: “Why is PID x executing code at 0x…?” There is no answer, because the kernel never approved that code and technically no ELF was executed.

Techniques that feel powerful on Windows fail conceptually on Linux. On Windows, replacing the user-mode loader can meaningfully reduce visibility. On Linux, the loader is the kernel. You cannot replace it, and trying to approximate it from user space produces stronger signals than simply executing a binary normally.

On Linux, the most reliable way to avoid detection is not to invent new execution paths, but to remain inside the ones the kernel already expects.

Get Involved #

I think knowledge should be shared and discussions encouraged. So, don’t hesitate to ask questions, or suggest topics you’d like me to cover in future posts.

Stay Connected #

You can contact me at ion.miron@tutanota.com

←
HackTheBox - Soulmate [Linux][Post Available Upon Challenge Retirement]
back to top