Your private key is (or better: should be) only stored in your wallet on your PC / mobile / whatever device you are using.
They are not stored on the blockchain at all.
Your private key is used to sign transactions. These signatures can be verified using the corresponding public key.
No one needs access to your private key to verify signatures.
So in case of etherdelta for example, it is an online wallet, do they have my private key? When you open an account there you are given a private key.
In the case of etherdelta, they don't have your private key. The private key is generated locally in your browser.
You are not given the key from their server.
I am just wondering if the blockchain has no memory how does it know whether the secret key i entered is correct or not.
You don't ever enter your private key anywhere.
You sign a transaction using your private key. And everyone can verify it by using the public key.
For visualization:
A transaction = "I want to transact X BTC from address 1 to address 2
Then you sign the transaction using the private key of address 1.
Everyone can then verify the signature by using the public key of address 1.
Please note that this visualization is not completely technically correct. But it gives you an idea how it works.