That's the problem: An internal SSD is connected relatively directly to the CPU. Almost all USB keys aren't optimised for fast data transfer other than, perhaps, reading/buffering large single files. Running a VM means constant read/write, constant update - it would be all the same as pushing a CPU to constant use of a swap file/swap partition. It's massively stressful on I/O - if you've a USB 2.x device, it transfers (very) slowly compared to USB 3.x.
[For an illustration: even identical USB sticks will also vary: writing 2.7GB DVD images to a set of USB 2.0 sticks a couple of weeks ago: most of them took six minutes, one took 35 - poor i/o and constantly swapping to check (I used dd with the oflag=sync option - essential for data integrity.) ]
The nearest comparison would be using a nice new NVME device. Inside a laptop, in an M.2 slot - as fast as you like. Put it into an external USB connected caddy - via USB 3.1 - for data transfer to another NVME inside the laptop and it will be slightly to significantly slower. Same chip, same laptop, moderated through a slower connection. That's the difference between your VM on internal disk and your VM on USB stick