Strange, my XFCE tends to have an uptime of weeks and months and it doesn't slowdown into a crawl. Are you sure you don't have a distro issue? (version issue, compilation, issue, etc?)
My server's running on a certain Amazon Web Services hardware that rate-limits the CPU, to save money. I can only use 10% of each of my two cores without paying extra. It's always the X server that's taking up the CPU cycles but this mainly happens when I start a desktop environment. I think the DE was loading a lot of things.
You shouldn't be doing this. Why bother running an UI in a remote server anyway?
But if you HAVE TO, there are ways to run Xorg apps in a server client fashion. As for xfce, it doesn't matter if you use it, obviously something like flux, i3 or whatever will be faster and leaner.
Still, opening an Xorg program in your computer from a remote server shouldn't take too much cpu server side (the graphic work is done by the client). Unless you are doing something silly like streaming the thing teamviewer/vnc style rather than using the proper Xorg server client mechanism. Or perhaps something easier like X2Go/Nomachine.
You have your own computer to do graphical stuff, use it.
