Topic: Get private keys when each was used (Read 331 times)

You have all the operations in this thread.

I don't know why comm complained about your file not sorted, but it did.
When I look into files, there is difference about case sensitivity, so lower case letters are after upper case - that's the difference.

I don't have duplicates because I use "sort -u". The input data, which I use to create this file, contains every Bitcoin transaction every made, and many addresses have been reused.

With "-u" too

I tried to test the performance difference, but perl used up my 16 GB RAM:
So I'll stick to "uniq" for this.

I'm not sure which awk command you mean: in another thread, I've mentioned removing duplicates while keeping addresses in their original order, and that indeed requires a lot of memory. But that's more complicated than simply removing duplicates.

That's incorrect: the command sorts the data, then removes duplicates. The output is sorted.

-u, --unique  this mean you have more then same addresses at different lines, actually mean duplicate, btw you dont have duplicate as i see last year your files

without -u your sort will be right in order

for duplicate remove you can use perl command for big files, no memory issue/error

perl -ne'print unless $_{$_}++' big-file.txt >> dup-remove.txt

for duplicate awk command will give errors, even you have big ram, but perl work best

sort -u will no right in order results, better use simple sort

Why?
This has the same result as sort | uniq, and gets the job done.

Don't use -u switch

Can you share what you did exactly? I tested it, and comm didn't complain about the sorting order:
I create all_Bitcoin_addresses_ever_used_sorted.txt.gz by piping all data through sort -u -S30%, so no additional sort should be necessary.

Indeed I can see that the file is sorted, but different-case-sensitive than the expected by comm program.

Linux sort by default uses /tmp for sorting files that don't fit it's RAM. I assume the WSL-version does something similar.
But the file you've downloaded from me should have been sorted already.

That's a public addresses file.

I sorted it in 20 min with WSL sort -u.

16 GiB of RAM was enough, even pagefile didn't extended.

And it was the larger file, not the latest smaller.

The Private Key list file is gigantic, about a few gigabytes large, so expect the sorting with 'awk' take a lot of memory, just to warn you (and that is if it doesn't run out of RAM first)

Will try some other day.

first sort used big file of addresses , if you have already sorted, thats good
2nd your dumped addresses from your wallet file
3rd sort your dumped addresses

simple command will let you get result

join your-dump-sorted-file.txt your full-big-sorted-list.txt >> result.txt

extra note if you have latest addresses files from web, its look example
1111111111111111111114oLvT2     28314777925
111111111111111111112BEH2ro     10940
111111111111111111112czxoHN     282440
111111111111111111112xT3273     5945

then extract addresses command
awk '{print $1}' addresses-list-18-gb.txt >> result-addresses.txt

then sort

sort result-addresses.txt >> result-addresses-sorted.txt

then start from top

It is not that crucial. These are experiments.

Additionally if anyone stumble upon this thread:

If you want to get private/public keys/addresses from Bitcoin Core, make two things:

1. export transactions
2. dump wallet

And with the above work you are ready to get in effect used private keys.

Glad it worked I have only one thing to add: all_Bitcoin_addresses_ever_used_sorted.txt.gz was last updated 5 weeks ago (which reminds me to run an update now, but it takes a few hours). If that's okay for your data, you're good. If not, you may have to add a few recently used keys.

Ok, got it!

What have I done:

- downloaded Bitcoin_addresses_LATEST.txt.gz and unpacked it
- dumped wallet to a text file (I will need private addresses later), dumpwallet command
- dumped all transactions (this gave me my public addresses), export in Bitcoin Core
- removed unneeded data from files (headers, footers, blank lines)
- awk -F "\"*,\"*" '{print $5}' pub-used.csv | sort -u > pub-used-addr-only.csv
- sorted -u the addresses file (comm said it was not sorted)
- comm -12 Bitcoin_addresses_LATEST-sorted.txt pub-used-addr-only.csv > result.txt
- result.txt goes to other folder with name pub-my.txt; priv-pub.txt is dumped wallet without unneeded data

-
while IFS= read -r p; do
        if grep -q "$p" priv-pub.txt; then
                echo $p
        fi
done < "pub-my.txt"

As a result I should receive public addresses that I have private key for them. I received 0 (zero). So wrong was selection of LATEST file (1-1,5 GiB), I should use whole all_Bitcoin_addresses_ever_used_sorted.txt.gz (22 GiB).

EDIT:

So later I've done:

- put pub-user-addr-only.csv in other folder, also file with all addresses ever used - needed to sort it (also uniqed it)
- comm -12 all_Bitcoin_addresses_ever_used_sorted-sorted.txt pub-used-addr-only.csv > result.txt
- put files in even other folder: my wallet dump, pub-used.txt
-

while IFS= read -r p; do
        grep "$p" priv-pub.txt | cut -d " " -f1 >> result.txt
done < "pub-used.txt"

- in result.txt I've receive private keys for all used public addresses
- sort -u result.txt > result-sort-uniq.txt

And at last I receive all private keys (WIF) which are the only that had movement on the blockchain.

OP is looking for addresses that had movement (maybe to recover Forkcoins?), so he needs a much longer list: List of all Bitcoin addresses ever used.
But that's only useful if you have the addresses with the private keys. If you only have private keys, you'll need to convert them into addresses first (with most likely 3 different address types per private key).

If you have an address list it's quick:
Then all you need to do is match those addresses to their private keys again.

Hello
You can use python to process this in multithread.
For example read file split each then process with Key to address with fast python lib (https://ofek.dev/bit/).
Match with known funded address (both compressed & uncompressed) if match save his output WIF .
This won't need internet i am unsure about process speed i tested myself 1000 line take less then 1min.

I have wallet dump in text format and I need to take out from it all private keys that had movement (incoming/outgoing) as some of the keys are brain wallets and were not used.
Not an option is going through each private key and exporting it from Bitcoin Core console.
Simply there is too much keys (23643 lines=keys).

How can I simply check each private key for any movement and then save it in other file?

This is a task where I need to limit the wallet for keys that have been used and getting rid of unused.