You first have to work through at worst all 24! (equals 620,448,401,733,239,439,360,000) possibilities to arrange the 24 words to decide if your tried arrangements yield a valid checksum. If yes, you can go through the address derivation process to compare if your known address is among a certain derivation path or range of it.
Going through the whole process to derive addresses based on common derivation paths is computationally expensive because you have to go through a 2048 rounds of PBKDF2 to get to the master private key from which you go further on.
But to perform statistically at least half of 620,448,401,733,239,439,360,000 arrangements and computing SHA-256 alone to only validate a correct checksum doesn't look achievable within centuries or more (I'll leave it to your own calculation to estimate a needed timeframe, not to speak of needed amount of energy to perform such a brute-force attack). Doable for half the words, unfeasible for 24 words of unknown arrangement.