loading . . . Bálint Réczey: Think you can’t interpose static binaries with LD_PRELOAD? Think again! Well, you are right, you can’t. At least not directly. This is well documented in many projects relying on interposing binaries, like faketime.
But what if we could write something that would take a static binary, replace at least the direct syscalls with ones going through libc and load it with the dynamic linker? We are in luck, because the excellent QEMU project has a user space emulator! It can be compiled as a dynamically linked executable, honors LD_PRELOAD and uses the host libc’s syscall – well, at least sometimes. Sometimes syscalls just bypass libc.
The missing piece was a way to make QEMU always take the interposable path and call the host libc instead of using an arch-specifix assembly routine (`safe_syscall_base`) to construct the syscall and going directly to the kernel. Luckily, this turned out to be doable. A small patch later, QEMU gained a switch that forces all syscalls through libc. Suddenly, our static binaries started looking a lot more dynamic!
$ faketime '2008-12-24 08:15:42' qemu-x86_64 ./test_static_clock_gettime
2008-12-24 08:15:42.725404654
$ file test_static_clock_gettime
test_clock_gettime: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, ...
With this in place, Firebuild can finally wrap even those secretive statically linked tools. QEMU runs them, libc catches their syscalls, `LD_PRELOAD` injects `libfirebuild.so`, and from there the usual interposition magic happens. The result: previously uncachable build steps can now be traced, cached, and shortcut just like their dynamic friends.
There is one more problem though. Why would the static binaries deep in the build be run by QEMU? Firebuild also intercepts the `exec()` calls and now it rewrites them on the fly whenever the executed binary would be statically linked!
$ firebuild -d comm bash -c ./test_static
...
FIREBUILD: fd 9.1: ({ExecedProcess 161077.1, running, "bash -c ./test_static", fds=[0: {FileFD ofd={FileO
FD #0 type=FD_PIPE_IN r} cloexec=false}, 1: {FileFD ofd={FileOFD #3 type=FD_PIPE_OUT w} {Pipe #0} close_o
n_popen=false cloexec=false}, 2: {FileFD ofd={FileOFD #4 type=FD_PIPE_OUT w} {Pipe #1} close_on_popen=fal
se cloexec=false}, 3: {FileFD NULL} /* times 2 */]})
{
"[FBBCOMM_TAG]": "exec",
"file": "test_static",
"// fd": null,
"// dirfd": null,
"arg": [
"./test_static"
],
"env": [
"SHELL=/bin/bash",
...
"FB_SOCKET=/tmp/firebuild.cpMn75/socket",
"_=./test_static"
],
"with_p": false,
"// path": null,
"utime_u": 0,
"stime_u": 1017
}
FIREBUILD: -> proc_ic_msg() (message_processor.cc:782) proc={ExecedProcess 161077.1, running, "bash -c
./test_static", fds=[0: {FileFD ofd={FileOFD #0 type=FD_PIPE_IN r} cloexec=false}, 1: {FileFD ofd={FileOF
D #3 type=FD_PIPE_OUT w} {Pipe #0} close_on_popen=false cloexec=false}, 2: {FileFD ofd={FileOFD #4 type=F
D_PIPE_OUT w} {Pipe #1} close_on_popen=false cloexec=false}, 3: {FileFD NULL} /* times 2 */]}, fd_conn=9.
1, tag=exec, ack_num=0
FIREBUILD: -> send_fbb() (utils.cc:292) conn=9.1, ack_num=0 fd_count=0
Sending message with ancillary fds []:
{
"[FBBCOMM_TAG]": "rewritten_args",
"arg": [
"/usr/bin/qemu-user-interposable",
"-libc-syscalls",
"./test_static"
],
"path": "/usr/bin/qemu-user-interposable"
}
...
FIREBUILD: -> accept_ic_conn() (firebuild.cc:139) listener=6
...
FIREBUILD: fd 9.2: ({Process NULL})
{
"[FBBCOMM_TAG]": "scproc_query",
"pid": 161077,
"ppid": 161073,
"cwd": "/home/rbalint/projects/firebuild/test",
"arg": [
"/usr/bin/qemu-user-interposable",
"-libc-syscalls",
"./test_static"
],
"env_var": [
"CCACHE_DISABLE=1",
...
"SHELL=/bin/bash",
"SHLVL=0",
"_=./test_static"
],
"umask": "0002",
"jobserver_fds": [],
"// jobserver_fifo": null,
"executable": "/usr/bin/qemu-user-interposable",
"// executed_path": null,
"// original_executed_path": null,
"libs": [
"/lib/x86_64-linux-gnu/libatomic.so.1",
"/lib/x86_64-linux-gnu/libc.so.6",
"/lib/x86_64-linux-gnu/libglib-2.0.so.0",
"/lib/x86_64-linux-gnu/libm.so.6",
"/lib/x86_64-linux-gnu/libpcre2-8.so.0",
"/lib64/ld-linux-x86-64.so.2"
],
"version": "0.8.5.1"
}
The QEMU patch is forwarded to qemu-devel. If it lands, anyone using QEMU user-mode emulation could benefit — not just Firebuild.
For Firebuild users, though, the impact is immediate. Toolchains that mix dynamic and static helpers? Cross-builds that pull in odd little statically linked utilities? Previously “invisible” steps in your builds? All now fair game for caching.
Firebuild 0.8.5 ships this new capability out of the box. Just update, make sure you’re using a patched QEMU, and enjoy the feeling of watching even static binaries fall neatly into place in your cached build graph. Ubuntu users can get the prebuilt patched QEMU packages from the Firebuild PPA already.
Static binaries, welcome to the party! https://balintreczey.hu/blog/think-you-cant-interpose-static-binaries-with-ld_preload-think-again/
