Additionally, if you want to backup label of your address/TX (usually for accounting or privacy purpose), you should also backup the wallet file.
In electrum, you connect to a server and get only the transactions data of your own wallet, so that it can display your transactions history, the correct balance of your wallet and the list of your UTXOs.
If you have made/received many transactions before, it may take a few minutes. If you have never made any transaction, it would take a few seconds.
In addition, when your wallet has thousand address/non-empty address sync could take even longer or even unable to complete sync due to DDoS protection on Electrum server. There's also performance concern if you use really old or slow device. Althouugh most user don't need to worry about this problem.