I can think of another option that might work: if I use the sorted list to get the new addresses, I can get those out of the daily update while keeping the chronological order. This way I only have to deal with two 20 MB files which is easy. After this, all I have to do is add them to the total file.
I found a bit of time to write this. Testing it now..
Just to check with you, I was sorta wrong here:
Given two sorted lists:
n = 1 5 10 11 12 13 14 15 16 19 20
k = 3 6 18
We can read n from disk line by line and compare it to the current position in k.
1 < 3, write 1 to new file.
5 > 3, write 3 to file.
5 < 6, write 5 to file.
10 > 6, write 6 to file.
10 < 18, write 10 to file.
11 < 18, write 11 to file.
....
16 < 18, write 16 to file.
19 > 18, write 18 to file.
19 & nothing left in k, write 19 to file.
20 & nothing left in k, write 20 to file.
That's n + k instead of n * k, right?
n = 1 5 10 11 12 13 14 15 16 19 20
k = 3 6 18
We can read n from disk line by line and compare it to the current position in k.
1 < 3, write 1 to new file.
5 > 3, write 3 to file.
5 < 6, write 5 to file.
10 > 6, write 6 to file.
10 < 18, write 10 to file.
11 < 18, write 11 to file.
....
16 < 18, write 16 to file.
19 > 18, write 18 to file.
19 & nothing left in k, write 19 to file.
20 & nothing left in k, write 20 to file.
That's n + k instead of n * k, right?
Since we're sorting as strings it would actually be:
n = 1 10 11 12 13 14 15 16 19 20 5
k = 18 3 6
The whole list would then become:
all = 1 10 11 12 13 14 15 16 18 19 20 3 5 6
Correct?
) is to take the transaction_hex field and pass it as the argument to the "decoderawtransaction" RPC call. It'll return JSON where the signature script is located at [N]["vin"]["scriptSig"]["hex"] for each input index N and then get the compressed public key in the last 33 bytes of the hex.
But even then, sorting data means reading and writing the same data several times.