Re: segmentation fault with libcrypto.so (but not libcrypto.a)

[Goswin von Brederlow <goswin-v-b@web.de>]

N N <nothere44@gmail.com> writes:

> Apologies if this is the wrong list.  If so, please direct me to the
> appropriate one.
>
> Consider the following C code:
>
> include <openssl/hmac.h>
> #include <stdio.h>
>
> int main(int argc, char** argv) {
>   unsigned char foo[10] = "boo";
>   printf("%s\n", SHA1(foo, 3, 0));
> }
>
> in file test-hmac.c.
>
> gcc -static test-hmac.c -lcrypto; ./a.out
>
> This works correctly, spewing garbage to the terminal.

Since accidentally the SHA1 sum is followed by some allocated memory
containing a 0 byte at some point.

> gcc test-hmac.c -lcrypto; ./a.out
>
> This segmentation faults.

Here you don't have a 0 byte after the SHA1 before the allocated
memory runs out and you get a segfault.

> Why?  What is wrong here?  So far, my best guess is that it has to
> do with how SHA1 allocates the return value when passed the null
> pointer (the third argument, 0).  The SHA1 function creates a static
> pointer large enough to hold the result which it then returns.  Does
> the fact that this operation occurs in a shared library change the
> allocation to the static pointer so that accessing after it returns
> is outside the allowed memory for the calling program?  Any help is
> appreciated.

What is wrong is your code.

While SHA1 allocates the value it does not allocate a 0 terminated C
string but rather a fixed size array filled with binary data. You can
never print that with %s or even %32s. Apart from your segfault just
think what happens if the SHA1 starts with a zero byte or contains a
zero byte.

You can also not compare 2 SHA1 with strncmp. Don't even try.

> Thanks.
>
> P.S. libcrypto version 0.9.8k, Debian version squeeze/sid.

MfG
        Goswin