I have created a hash value using createrawtransaction and signrawtransaction, and I guess I could work out the byte size of that hash value. What I find confusing is that I don't get the hash value until createrawtransaction has been executed, and I am supposed to type in the fee value into createrawtransaction.
Createrawtransaction doesn't return a hash, it returns raw bytes, which can be counted. Count the bytes, work out the required fee, then subtract it from your change output (altering the value of an output doesn't affect the size of the transaction).
You can also estimate the tx size so that you can put a desired tx fee in your first attempt. Then like foxpup said compute the tx size and if needed go back change the fee amount by making change smaller. Think of it as potentially being a two pass process. This assumes you are not trying to spend lots of dust inputs* and you should setup a tx such that you have a change output with sufficient value that it can be reduced to cover the potential (increased) fee. If the change output is less than the amount of the fee increase then it can get more complicated.
For "normal" (Pay2PubKeyHash) txs:
signed_tx_size (in bytes) ~= 10 + (numInputs * 148 ) + (numOutputs * 34)
That assumes all inputs are compressed PubKeys. Uncompressed PubKey are 32 bytes larger so the size of the input would be 180 bytes instead. Bitcoin uses varint for #Inputs, #Outputs, and script lengths so this should only be used as an estimate.
So for example if you were creating a tx which has 4 inputs (one of which is uncompressed), three outputs (including change output). If all inputs/outputs are Pay2PubKeyHash then the size would be roughly (10 + 148*3 +32 + 3*34) = 588 bytes. Rounding up nearest KB the tx size is 1KB. If you wanted a fee of 0.1 mBTC per KB it would be a total fee of 0.1 mBTC. So set the change output such that the inputs - outputs = 0.1 mBTC. Create and Sign. Now double check the size of the signed tx. Is it <1KB (in this case there is no chance of it going over 1000 bytes but for tx close to the limit it might)? If so you are done. If not (say tx size is 1038 bytes) then reduce the change by another 0.1 mBTC (resulting in a 0.2 mBTC fee on 2KB tx = same 0.1 mBTC per KB).
* A word about dust inputs. Things start to get tricky if you are trying to spend lots of tiny dust outputs. It can lead to a more complicated tx creation algorithm or a flawed one which results in insufficient fees to ensure timely inclusion in a block. If on your "2nd pass" you need to raise the fee = lower the value of the change output and the change output is "too small" to cover the increase then you will need to add another inputs. This could result in the tx being large enough that the fee needs to be increased as well (fee is per KB) and you could end up with insufficient available output requiring a second or third pass. Once all the available inputs are below the dust threshold you will find that it becomes impossible to the create the tx (i.e. adding the outputs increases the fee more than the value it adds and thus you will go through a number of passes until you exhaust all outputs and end up with a giant tx that still has insufficient fees). If you want to keep your algorithm simple I would recommend excluding dusty outputs from input selection.
