Update on rlbox wasm sandbox crashes
Hi guys.
So those who are familiar with this will know I dug into what was
crashing Firefox on startup. Which was expat crashing inside a wasm
compiled sandbox. An issue known about for years and worked around by
disabling the sandboxing. An existing fix for s390x. Which fixed the
Firefox build on Debian ppc/64 so it worked again. (Thanks Adrian.)
I pretty much threw myself into the deep end as I wasn't familiar with
RLBox nor how it was integrated into Firefox but did know about wasm. So
I had to learn what was crashing in GDB and undo my confusion as to what
was causing it and why sources for symbols were missing. I've spent time
since investigating it and learning about all the processes involving
this sandboxing.
(TL;DR, or afraid to but still reading this line, it looks like wasm2c
did it.)
There's three main components involved:
* rbox - The core sandbox.
* rlbox_wasm2c_sandbox - Sand boxing using wasm as a middle man.
* rlbox-book - Example code and tutorials for porting and testing
sandboxing.
These come in different levels with their own sources. The first, rlbox
is small. This easily compiled on ppc64 and passed all tests. The logic
appears fine. Next, rlbox_wasm2c_sandbox, is more complicated. This is
designed to compile the whole toolchain before you can even use it to
compile and test any code for sandboxing. The last, rlbox-book, has code
for a tutorial and examples for basic "no-op" false sandbox source
testing and a real "wasm" sandbox source example. This relies on
rlbox_wasm2c_sandbox but, the sources are obsolete or mismatched, as the
example code expects sources generated from the build that don't exist.
So you cannot build it all without the examples still breaking. Somewhat
time wasting if you need to hunt down what it's looking for. Which I did.
RLBox and WASM have been available for some time now days so there
should be no need to build the entire toolchain just to do some simple
test compiles. And there are prebult packages. In particular there is
wabt which provides wasm2c, wasi-sdk which provides wasm build tools,
and clang actually can compile source to wasm code itself. Of these
Debian ppc64 doesn't have wasi-sdk. But does have wabt and a wasi-libc.
And the ppc64 clang build does support wasm. As expected there are no
ppc/64 releases of wasi-sdk. However, tools are available here in ppc64
for what building an rlbox wasm example needs, if you find where to
look. :-)
The whole RLBox and WASM integration is one big convoluted monster. It
will take a nice small source code and bloat it up into a big
complicated mess. All for the purpose of a software VM to isolate code
and protect the host. To simplify the process, a source needs to be
compiled into wasm, then wasm code reverse engineered (or back compiled)
into C, then finally wrapped in the rlbox API and compiled as C/C++ into
the end binary. Sounds simple! I wanted to compile this on ppc64 since
this process breaks binaries on ppc64 and I had a devil of a time doing
so. But, I also wanted to avoid building the whole toolchain, since the
tools do exist. First, I needed a generic clang to work for me, since
the wasm clang build is custom. A search revealed clang needed
--target=wasm32-wasi so was a simple fix. Next I needed wasm2c to work
normally. Unfortunately, the documentation is a bit sparse. For example,
wasm2c from wabt kept generating source that caused clang (and gcc for
that matter) to spew out errors everywhere. All the labels had been
surrounded by 0x24 and 0x2e (dollar and dot if you know your ASCII hex)
and it kept wrecking the whole build, which looked strange, as the
sources it output were isolated to one C and header file. Turns out,
with the wasm2c I was using, I needed to pass --no-debug-names. Looks
simple, but this took me ages to find out. It also didn't help that some
options give errors about being unimplemented as if was2c was still an
unfinished beta. I looked on the net for info on wasm2c putting 0x24 and
0x2e in labels and got no results at all! In the end I just added that
debug option as a guess since it looked related and that was it. Well
they could have mentioned it! How about the manual actually mentioning
that by default this wasm2c will put 0x24 and 0x2e in labels? Or wreck
it so it won't compile? :-facepalm.
After getting over all that I could finally do some real testing in gdb.
At this point, rlbox and an rlbox wasm noop-hello-example from
rlbox-book compiled and worked fine. Now I had the wasm-hello-example
from rlbox-book compiled which was the real challenge. I ran the test
binary and it crashed. Actually, unlike expat in Firefox, it didn't
actually crash but aborted itself. Either way this was good as I now
knew what code was broken and what built the code. I've now ran the code
through gdb and also compared with running on x64. A few things I have
ascertained:
* Code wasm2c generates is big endian aware. Despite lacking tier 1
support the host code generated checks for big endian and runs alternate
code to fix bytes. So regardless of host endian the output of wasm2c
code is designed to and should work on big endian/ppc native. This is
good as not only does the upstream code take big endian into account it
is designed to run correctly on a big endian host. I checked and both
x64 and ppc64 builds of wasm2c produced big endian aware sources.
* Related to my first point there is a quirky macro that fixes endian
when ran from big. It actually will slow down the code so although it's
designed to fix endian it should be fixed to be optimised. It's a load
data routine that normally just does a mem copy. On big endian it does a
mem copy and then, then reverses the code in place, almost neatly by
reading a byte at one end and swapping with a byte at the opposite end
with a half length array loop. Although it looks novel I wonder why they
just didn't stick a reverse mem copy in? But that's just how I would do
it. Any built in mem copy is rendered useless by the whole operation.
I have found some issues. The example code I compiled from the
wasm-hello-example breaks on big endian. I don't mean to repeat the
obvious, but relating to my first point, any code compiled on big endian
will not include the macro. It relies on a WABT_BIG_ENDIAN define being
set to 1 when the final wasm2c C code is compiled. On my system this was
not set although I expected the upper headers included would test and
set it. This could relate to my test build but is one to watch out for
upstream. It is tested with from a CMake list in rlbox_wasm2c_sandbox.
But the example just has a simple Makefile with no checks for configuring.
Next is loading data. The wasm2c code has data built in. There are both
data tables and some function tables all built from byte arrays. Plus
what looks like hex encoded ASCII string tables. Looking at the data it
appears there are scalars embedded in little endian order. This would be
consistent with wasm being little endian but will obviously cause
trouble in big endian. Although the code should be aware of any byte
positions. I suspect this could be what is crashing expat where just one
offset among a few parameters to a function was byte swapped. I don't
know how clean the code is, since it doesn't look clean from the onset,
but if data load routines are doing tricks, such as reading the LSB or
reading one byte from a bigger scalar it will badly break. There could
be big trouble in little endian here! :-D
The macro has more issues. It is designed for scalars but accepts any
value. I don't know if they intended it to be like that to me it looks
like a bug. The routine is designed to load data but is abused/confused
by a macro that reverses the loaded data. However, since it wasn't
enabled in my build, this wasn't what was aborting the code. For fun I
enabled the WABT_BIG_ENDIAN macro on x64. Which I'm also doing
comparative testing with. It broke it and caused the binary to abort.
Lol. But, on ppc64, enabling WABT_BIG_ENDIAN almost made it worse. The
code doesn't crash or abort but ends up in an endless loop! :-?
After examining more code I can see this getting more complex. Due to it
coding these tables in. This will result in mixed endian data throughout
the code. There are fields code assigns and reads back in native endian
from source. There's scalars read from data. But since they are both set
and read in the same data table it's hard to see what endian will be
what! Only wasm2c would know what the offset is. If it knows what an
offset does in the first place. Then the code itself appears to be
simulating a CPU. It assumes all these local variables it spends time
reading and writing to, doing conditionals on with gotos, passing values
around and doing what looks like the least optimised way of an
operations. Doing operations atomically. It actually looks like a C
version of RISC ASM. :-)
This is running deep. Of course, this is due to taking perfectly
portable code, compiling it into little endian machine code, then
reverse engineering that into an analogous C source. The result has
ended up with introducing a false dependence of little endian. Be good
if wasm2c could be told to generate big endian or portable code. But
since the world is now running on a little endian web, this problem
won't go away now.
After gathering the evidence so far I can only conclude the entire
process going on here has made an already convoluted process more
complicated. That becomes more worse on an endian it isn't designed for.
I've already dealt with code like this before that, based on
examination, was generated by reverse engineering x86 code into a non
portable convoluted mess of C. It wasn't pretty. Yet I manged to find
every byte out of place and fix to be portable. So now, I can see that
disabling the sandboxing is for now likely the best fix, because if it
worked the code would be even slower. But, the compromise is here is
more security, so getting it working down the track is a good idea
regardless and even for an "exotic endian" platform. In the mean time,
I'll continue digging, now it's becoming clearer where to point the
finger. And what I can use as an MVE.
Links:
https://github.com/PLSysSec/rlbox
https://github.com/PLSysSec/rlbox_wasm2c_sandbox
https://github.com/PLSysSec/rlbox-book
The wasm2c [excuse for a] man page example:
https://manpages.debian.org/bookworm/wabt/wasm2c.1.en.html
That quirky macro:
static inline void load_data(void* dest, const void* src, size_t n) {
if (!n) {
return;
}
size_t i = 0;
u8* dest_chars = dest;
wasm_rt_memcpy(dest, src, n);
for (i = 0; i < (n >> 1); i++) {
u8 cursor = dest_chars[i];
dest_chars[i] = dest_chars[n - i - 1];
dest_chars[n - i - 1] = cursor;
}
}
--
My regards,
Damien Stewart.
Reply to: