That's exactly right. However, Congress is only 1/3 of the power in the Government. The Judicial branch has already declared on two occasions that cryptography is a protected form of free speech.
Congress can't just pass any law they feel like. In the event that they do pass an unconstitutional law, the Judicial branch has the power to nullify it.
Considering the fact that there's already a precedent regarding cryptography, Congress passing a law hindering cryptography is no different than passing a law regulating religion, written words in a book, speaking negatively about the President, etc... It's without the force of law no matter how much they want it.
The law is made in the Legislative branch, enforced by the Executive branch, and judged by the Judicial branch. The Judicial branch already made a ruling that crypto is protected by the Constitution; any future law passed will be judged in court as if it were no a law at all; they can't so easily go back on their decision.
"In cases addressing other encryption software, however, two federal appeals courts have established the rule that cryptographic software source code is speech protected by the First Amendment (the Ninth Circuit Court of Appeals in the Bernstein case and the Sixth Circuit Court of Appeals in the Junger case)."