Advice wanted: handling weird vendoring situation

[Julian Gilbey <julian@d-and-j.net>]

Hi,

I'm working towards packaging pydevd (which is a recursive dependency
of Spyder via IPython), and it's a somewhat challenging package!  I
have hit an issue and would appreciate some thoughts on how best to
handle it.

Background:

pydevd is a debugging package which can attach to running scripts.  It
is used by PyDev, PyCharm, VSCode and Spyder, and with Spyder, it is
imported through debugpy, which in turn is imported into IPython.

In order to ensure that the libraries it is using have not been
modified by gevent, it uses vendored copies of various standard
library modules rather than just saying "import inspect" etc.

I thought I could address this issue by replacing the vendored copies
of the library modules by symlinks to /usr/lib/python3.X/, but now
I've hit another snag: some of these modules import other modules.
For example:

pydev_imps/_pydev_SimpleXMLRPCServer.py
is a very old version of /usr/lib/python3.X/xmlrpc/server.py.  It
contains within it the following lines:

from _pydev_imps import _pydev_xmlrpclib as xmlrpclib
from _pydev_imps._pydev_xmlrpclib import Fault
from _pydev_imps import _pydev_SocketServer as SocketServer
from _pydev_imps import _pydev_BaseHTTPServer as BaseHTTPServer

These libraries are:
_pydev_imps._pydev_xmlrpclib -> xmlrpc.client
_pydev_imps._pydev_SocketServer -> socketserver
_pydev_imps._pydev_BaseHTTPServer -> http.server

So what should I do?

One solution is just to symlink from _pydev_SimpleXMLRPCServer.py to
/usr/lib/python3.X/xmlrpc/server.py and not worry about the other
modules.  But that might break things in non-obvious ways, so I don't
want to do that.

Another possible solution is to update all of the vendored copies in
_pydev_imps using the relevant /usr/lib/python3.X/* modules and making
the same modifications to the imports to load local copies.  But then
we will have duplicate copies of standard library modules, which I
also don't want to do.

Perhaps another possibility is to have symlinks in the _pydev_imps
directory to the standard library versions and then temporarily modify
sys.path to look in _pydev_imps before looking in standard locations.
I don't know whether this will work, though.

(There is another snag, too, which is that the path will depend on the
Python version.  So I will probably have
_pydev_imps/python_39/socketserver.py -> /usr/lib/python3.9/socketserver.py
and
_pydev_imps/python_310/socketserver.py -> /usr/lib/python3.10/socketserver.py
But that's a simpler issue.)

Any thoughts on these ideas would be much appreciated!

Best wishes,

   Julian