Actually, a lot of people.
I don't know about Ubuntu specifically, but most Linux distributions are well-audited and every line is accounted for.
Debian, which is the distro Ubuntu is based off, prevented two backdoors from being placed in 2003 and 2006.
The one in 2003 was planted when a hacker broke into the source control server and removed a SINGLE character in one core developers repository in order to introduce a root privilege escalation exploit. The developer later merged his changes into the main repository, making it very hard for anybody to spot especially considering these changes (including the backdoor) were coming from a "trusted" developer who had no idea he had been hacked. Despite all of this it was still noticed and patched.
https://freedom-to-tinker.com/blog/felten/the-linux-backdoor-attempt-of-2003/
tl;dr; while it's possible, it's very difficult to insert a backdoor into an open source project.
On the topic, programmers nowadays have very powerful tools like IDE's, Diff/Meld, Version control software such as Git, which makes putting a backdoor into anything more difficult than most people would think.
Most changes are not so big and are incremental changes, it is easy to review them by an experienced programmer.