1. for "each" input
2. read related outpoint's scriptpub
2.1. figure out the type of it
2.2. prepare the message digest or digests based on the type
3. hash it/them and get a final digest
4. sign it.
the steps 2.1 and 2.2 can be entirely different depending on what step 2. returned. and it has to be a pre-defined type. for example if it returned P2PKH type then it empties all the other txin scriptsigs and replaces the corresponding input's script with that P2PKH script and then signs the whole modified transaction.
if it was P2WPKH then the entire hashing mechanism changes. you have to concatenate different parts together hash them then concatenate the hashes together and hash them again (BIP143).
if it were non-standard or not pre-defined...
there is no way any code could predict how to produce a valid signature for this.
I mean the case when the step 2 is returned one of the predefined standard scriptpub. I need to sign nonstandard redeem script only.
For example when I have an P2SH transaction hash (from my grandfather as an inheritance) and I known that redeem script should by like this
So I can create a raw transaction with this redeem script manually and only one thing that I can not to do - I can not to sign RAW transaction because redeem script is nonstandard. Of course, if I am a programmer, I just can write a program. But I think that this can be done without programming with openssl for example as a hard way. Or this feature can be placed in one of the next version of bitcoin core API
BTW may be I do not undestand the signing process and API "signrawtransactionwithkey" can sign my unstandard redeem script. Can you explain me?