It's not that simple. For one thing, there are more people willing to buy or sell than the number of people who have positions on the orderbook. I may be willing to sell, but may not want to let the market know that I'm willing to sell by putting an ask/sell order on an orderbook. Instead, I might just execute a market order and sell directly into the buy orders. Similarly, buyers may wish to buy, but might not advertise that on the orderbook, and might, instead, buy some of the ask/sell orders sitting on the book.
Or: someone might actually be willing to buy, but will put a big sell order into the book to scare the market down.