So the DLL/SO survelliances that the EXE requesting API access is correctly signed and not modified, and that not critical parts are modified, and the EXE checks that the DLL/SO is not modified.
Also both the EXE and the DLL/SO can check itself in a way too.
If its too tough to make secure, you could have 2 identical DLL/SO, that survelliance each other.
How does the dll keep me from modifying it while it isn't running?
No matter how many complicated layers you add, I am still running code on your system. I can do whatever I want to circumvent those security layers because I am responsible for enforcing them in the first place.