Synopsis
If you are interested in utilizing your Nvidia GPU/primary gpu while also passing it through occasionally, I suggest you look at the scripts present in this guide. They can be used to unbind and bind a nvidia gpu successfully. Note you should only run these when your x11 or wayland session is not running.
Libvirt Hooks
Effectively just follow this guide for setting up hooks and your problems should be resolved. Make sure your gpu is bound to vfio on boot though either via kernel parameters or some other means. bryansteiner-guide
Have applications use that specific gpu.
I used nvidia prime to do this operation (which works in sway additionally) but effectively install the nvidia-prime package. Then use the following syntax to run the application with a different gpu as a renderer
prime-run <application>