There’s one old game that works better by chosing “Scaled by the system”; however this make other x11 apps look slightly blurry. Is it possible to either choose “Apply scaling themselves” and exclude one app from it, or exclude every other app from “scaled by the system”, so it only affect this one game?
Yes.
This is most easily achieved by using gamescope as they suggest.
Right, but gamescope opens in a single new window. That’s different. I’m wondering if it’s possible to have the same compositor show windows from two Xwayland instances, because from what I’ve seen it has to be integrated into the compositor somehow, gamescope doesn’t.