I have another such case where I would be upstream. I currently have a library
that I am the only copyright holder of.
If you're the sole copyright holder, you can do whatever you like with the code. The license is 100% irrelevant as concerns anything you are doing with the code.
Do I then get into trouble with the GPL, as the source tarball contains code
that links to non-GPL compatible code?
You can't get in trouble for anything you do with your own code to which you own the copyright. You only run into issues when you do something with code someone else holds copyright to. In this case, you're free to link OpenSSL against your own software, that belongs to you, to which you own the copyright.
This BTW affects the FreeBSD project too