May 2, 2025
8 mins read
One of the most annoying things to debug when solving binary exploitation challenges is a solution script that works locally, but not remotely. This blogpost will be an explanation of every single thing I can think of that can cause this difference, sorted by likelihood.
Here are each of them as a quick list:
Your exploit requires having an exact copy of the remote server’s libc, and whatever libc you’re using is wrong. The two common subcases of this are A) you are using your own computer’s libc, which is probably not the remote’s B) you have grabbed the wrong copy.
You are usually either directly given a libc/ld binary to use or a Dockerfile that you can run and copy the necessary libraries out of. If you are given neither of these, the solution probably doesn’t require knowing the remote server’s libc.
Once you have a copy, you can automatically patch the program to use the given libraries with pwninit or manually with patchelf like so:
patchelf ./vuln --set-interpreter ./ld-local-copy.so --replace-needed libc.so.6 ./libc-local-copy.so.6
MAJOR FOOTGUN: If you need to grab the libraries from the Docker container, double check whether the Dockerfile uses redpwn’s jail. This jail is very commonly used for binary exploitation challenges, and if it is, the necessary libraries will be in /srv/lib instead of /lib. Again, do not copy the libraries inside /lib.
Crossing my fingers that these get indexed by search engines, here are the build IDs of the libc/ld inside of /lib/:
4286bd11475e673b194ee969f5f9e9759695e644
0401bd8da6edab3e45399d62571357ab12545133
If the libc/ld you’re using has this build ID, there is an extremely high chance you incorrectly copied the libc from /lib/libc.so.6 instead of /srv/lib/x86_64-linux-gnu/libc.so.6.
Your attack relies on ASLR being disabled. There is a very good chance this isn’t true on the remote server.
GDB disables ASLR by default, which may be the source of your confusion. You can reenable this in GDB with set disable-randomization off.
If your exploit has a ROP chain that works locally but not remotely, this is a pretty common reason why. In short: try adding a gadget that just runs ret to your chain.
There are a few x86 instructions that require the stack to be 16-byte aligned when executed, or in other words “$rsp must be a multiple of 16 when this instruction is ran”. If this isn’t true, the program will immediately crash. The most common instructions that cause this crash are movaps and movapd.
Due to Stack Inconsistency, your ROP chain might be 16-byte aligned locally but not remotely. To fix this, you can add a gadget to your ROP chain that runs ret and does nothing else. This will do nothing to your chain other than shift $rsp by 8 bytes, which should hopefully align your ROP chain on the remote server.
You can read more about this in the “Data Alignment” section of the AMD64 Manual.
Your exploit relies some data that exists on the stack (ex. a leftover stack pointer) that doesn’t exist on the remote system. This is a pretty common problem with format string challenges, since you commonly rely on leftover stack data to perform your exploit.
The two ways I know the stack can differ are:
I believe this happens due to a difference in CPU capabilities– GLIBC contains optimized versions of common functions like memcpy/strlen/strrchr that use whatever SIMD capabilities your CPU supports. If the remote server doesn’t have the same capabilities as your CPU does, it’ll use a different function that will end up doing different things to the stack.
A quick example of this I found is that disabling AVX support on my CPU (by starting the kernel with the argument clearcpuid=avx) would cause the stack to contain a pointer to dl_main that disappeared when I re-enabled AVX support.
A much simpler reason that this can happen is if you’re using the wrong libc/ld. If the stack of the remote server seems wildly different than your local machine, this is probably what’s happening.
Your exploit guesses the address of an mmap() call based on knowing what a different mmap() call returned– for example, guessing the base of ld given the base of libc. This guess can change on a remote system.
Where mmap() decides to allocate things is completely up to the kernel, and the algorithm for this frequently changes between kernel versions. This means that if you aren’t running the same kernel version as the remote, the ordering of all mmap() calls may be different than yours.
If you can’t determine this offset on the remote server, your best shot is to bruteforce it. Decrease/increase your guess in intervals of 0x1000 (since mmap() calls are page-aligned) and hopefully one of those guesses will be right.
You are exploiting an outdated or incorrect version of the vulnerable program. There are three ways this usually happens:
Your exploit is using things that the Docker container the program runs in doesn’t have.
A good example would be the BuckeyeCTF 2024 challenge no-handouts, where the vulnerable program was in an ultra-minimized Docker container that contained nothing but the program itself. While spawning a shell with system() would work locally, it wouldn’t remotely because there was no /bin/sh binary in the container.
Another good example is how redpwn’s jail works. The files necessary to run the challenge must be copied into the /srv directory of the container because the jail chroots /srv to run the challenge. This means that if the Dockerfile copies the flag to, say, /srv/app/flag.txt, that file will actually be located at /app/flag.txt.
patchelf is a well-known program to change how an ELF works. It’s commonly used in pwn challenges to force a program to use a specific libc/ld instead of your own system’s. This is also what pwninit uses under the hood.
This tool is very janky and it can make some pretty significant changes to the ELF. You may be accidentally using one of these changes in your exploit.
One problem patchelf can commonly introduce is changing a program’s base address– if the program is normally loaded at 0x400000, patchelf might decide to load it at 0x3f0000 instead. This can obviously break a remote exploit.
patchelf can also massively change the virtual memory layout of an ELF. Some things I noticed it does include:
.symtab section header into memoryRW LOAD program header that overlaps the data of all the following program headers, making multiple parts of the program that were previously read-only now writableSeriously, if you’re ever bored try to dissect what the hell patchelf does to the program. I still don’t even understand how the .symtab section gets loaded!
Anyways, your exploit may or may not be relying on this mangling patchelf is doing, and will fail on remote because the remote binary obviously hasn’t gone through this mangling.
To fix this, you can use unvariant’s version of patchelf, which makes much more minor changes to the ELF.
This is incredibly unlikely to be what’s going on, but it’s worth mentioning.
Your attack (likely unknowingly) relies on something about your CPU that isn’t true on the remote server’s. This can commonly happen during extremely tight race conditions or side channel attacks, but can even show up in ‘simple’ userland exploitation.
An interesting userland example is what happens if you supply a negative size to memcpy, aka memcpy(dst,src,-1). Rather than crashing the program like you might expect, this actually causes a backwards buffer overflow varying on what SIMD instructions your CPU supports!
This fact was first shown off in the PlaidCTF 2025 challenge Bounty Board, where the CPU on the remote server supported AVX512 instructions. Competitors having CPUs that didn’t support these instructions would have to slightly edit their solve scripts to work remotely.
Sharing is caring!