You're using Xfce without a desktop? IOW you are using Thunar, even if you don't use it as a file manager.I don't use any file manager, I work only with urxvt terminals
I seldom use a file manager, I prefer to use CLI, but when Xfce is installed, the desktop does need Thunar as a dependency. The desktop, the thing with the wallpaper that can show icons, already is Thunar, even if you suppress to show icons on the desktop.
Thunar does use gvfs, but we are free to use Thunar without gvfs and mount everything using another way.