This is mostly true .. in newer versions of Xen there is options for PCI passthrough, where it maps the physical device to a specific VM. So it is possible, but not very easy, and I wouldn't count on it not taking the whole box down if something went wrong.
Last time I looked, PCI passthrough didn't work with GPUs.
It works fine, parallels even sells their Extreme Workstation product specifically for the task. You do however need to make sure your chipset supports VT-d (Intel) or IOMMU (AMD) for hardware level support of direct IO.
