First off, I'm just speaking for myself here.
This is a great question. Indeed, there is currently no way to verify that the Ubuntu binaries came from the source code.
Lets start with a description of the way that Debian/Ubuntu packaging works. With Debian, the package maintainer creates one of the packages (the one for the architecture that the maintainer is building on) and the Debian infrastructure builds the rest of the packages for the other architectures. I'm not as familiar with Ubuntu, but I think Ubuntu builds all the packages on their infrastructure (though I may just be thinking of the way PPAs work).
There is a reproducible build effort underway to make Debian as a whole reproducible. You can
check out the progress. The percentage of packages that are reproducible is a little above 80% as of the writing of this post. In the future, it looks like it will be a requirement that uploaded packages be reproducible, otherwise they won't be accepted into the Debian Archive. But right now there is no requirement that a package be reproducible.
How does Debian's effort affect Ubuntu? In 2013, Colin Watson
stated that Ubuntu will inherit the changes over the natural merging and syncing of packages from Debian. I don't know how many packages are reproducible in Ubuntu, because it doesn't seem like Ubuntu has the infrastructure setup to rebuild and compare packages for reproducibility like Debian has.
Now I probably should have put this earlier, because a lot of you are probably wondering what reproducible builds are and how do they affect the verification that binaries came from the source code. Reproducible builds involve multiple people building the binaries from a particular version of the source code. The hashes of the binaries should match those of all the people building the binaries. If there is a discrepancy in the hashes, then it means that there was some difference in the resulting binaries (whether that means someone's computer was compromised or there was something non-deterministic about the build process that introduced differences). The reproducible build process means that every single builder needs to be compromised to compromise the resulting binaries. If the current method that Debian uses of building packages is used instead, then there is a single target (either the maintainer of the package or the Debian build infrastructure) to compromise. There are some things that reproducible builds just don't protect against. If everyone is using the same OS to do the builds, then it is possible to just compromise the OS and influence the build that way. But you have to define the threat you are trying to protect against and focus on that. Even if you have a foolproof way of making sure your software stack is secure, there is still the hardware you are running. Do you know for sure that your hardware isn't compromised? Of course that worry can be mitigated by having the builders using a variety of hardware, so that it is less likely that all the hardware is compromised.
That explains the Debian/Ubuntu reproducible build effort. However, there is also Gitian, which was originally developed for Bitcoin Core, but is also used by Tor Browser Bundle. Armory is also using Gitian along with the Debian reproducible build toolchain to make Armory reproducible, but that isn't finished yet (I'm pleasantly surprised how many people have asked questions related to reproducible builds after the Armory effort was started). With Gitian, the build is performed in a VM and there is this assert file produced that contains the hashes of the resulting binaries (among other data). The assert file is signed by the builder. In Bitcoin Core's case the assert file and signature get added to a repo called Gitian.sigs on GitHub. Then users can download the signatures and verify them. You should verify the signatures of people you trust.
I think this is a pretty comprehensive answer, but let us know if there is something I should go into more detail on. Unfortunately the reproducible build efforts for OSes aren't complete yet, so you still have to trust in the build processes used and the people that do the builds.