Pages:
Author

Topic: VanitySearch (Yet another address prefix finder) - page 5. (Read 31225 times)

full member
Activity: 1050
Merit: 219
Shooters Shoot...
Can you enlighten me to what exactly the bug is/is doing? Is it only the -check unit test?

It's not just in the unit test, it's also in the main loop when it's generating Base58 addresses (the checksums made after an equality comparison or regex match, specifically).

In the above code piece for creating the checksum, it runs fine without any optimizations at all.

But, when -fstrict-aliasing is switched on as a result of using -O2, the compiler assumes that variables of different types being assigned to each other have the same size (as in sizeof()) and optimizes out some internal ASM that would've otherwise been generated.

The ASM being removed here is parts of the assignment statements in the snippet above (and in the non-SSE version). Those areas still have junk values from when those automatic variables were declared.

So the SHA256 checksum contains some garbage bytes. When this is encoded into Base58, it can only generate wrong addresses. And this is despite all private keys being 100% correct.

That's why it works in debug mode and not in production.
So is it just when the -check is used? I'm confused lol.

How does it impact searching for keys...
legendary
Activity: 1568
Merit: 6660
bitcoincleanup.com / bitmixlist.org
Can you enlighten me to what exactly the bug is/is doing? Is it only the -check unit test?

It's not just in the unit test, it's also in the main loop when it's generating Base58 addresses (the checksums made after an equality comparison or regex match, specifically).

In the above code piece for creating the checksum, it runs fine without any optimizations at all.

But, when -fstrict-aliasing is switched on as a result of using -O2, the compiler assumes that variables of different types being assigned to each other have the same size (as in sizeof()) and optimizes out some internal ASM that would've otherwise been generated.

The ASM being removed here is parts of the assignment statements in the snippet above (and in the non-SSE version). Those areas still have junk values from when those automatic variables were declared.

So the SHA256 checksum contains some garbage bytes. When this is encoded into Base58, it can only generate wrong addresses. And this is despite all private keys being 100% correct.

That's why it works in debug mode and not in production.
full member
Activity: 1050
Merit: 219
Shooters Shoot...
An update about the chronic address bug on VanitySearch:

I have found the source of the error. In case you didn't know, everything released after VanitySearch 1.13 fails to math any base58 address, on newer GCC (11+, and maybe 10 as well)

This has to do with the sha256.cpp and sha256_sse.cpp files inside the hash/ folder. Specifically, sha256_checksum and sha256sse_checksum functions.

These functions fail to make the correct checksum when you are running with optimization flags on, but I managed to get even more specific than that.

It appears that a single O2 optimization flag, -fstrict-aliasing is causing all the chaos. Taking the SSE version of this function for example:

Code:
void sha256sse_checksum(uint32_t *i0, uint32_t *i1, uint32_t *i2, uint32_t *i3,
  uint8_t *d0, uint8_t *d1, uint8_t *d2, uint8_t *d3) {

  __m128i s[8];

  _sha256sse::Initialize(s);
  _sha256sse::Transform2(s, i0, i1, i2, i3);

#ifndef WIN64
  uint32_t *s32 = (uint32_t *)(&s[0]);
  *((uint32_t *)d0) = __builtin_bswap32(s32[3]);
  *((uint32_t *)d1) = __builtin_bswap32(s32[2]);
  *((uint32_t *)d2) = __builtin_bswap32(s32[1]);
  *((uint32_t *)d3) = __builtin_bswap32(s32[0]);
#else
  *((uint32_t *)d0) = _byteswap_ulong(s[0].m128i_u32[3]);
  *((uint32_t *)d1) = _byteswap_ulong(s[0].m128i_u32[2]);
  *((uint32_t *)d2) = _byteswap_ulong(s[0].m128i_u32[1]);
  *((uint32_t *)d3) = _byteswap_ulong(s[0].m128i_u32[0]);
#endif

}

Aliasing is an optimization technique where the compiler assumes that you're never going to cast variables into crazy types of different sizes and makes some fast but unsafe memory writes as a result.

It's unsafe because if the type indeed is cast into a larger size type, then some garbage will be written into the higher parts of the variable. That's where the gibberish in the checksum characters comes from. The parameters d0-3 are expanded into uint32 and then assigned, and that's where strict aliasing messes things up.

Something similar happens with sha256_checksum but with memcpy instead.

These functions were introduced in the following commit: https://github.com/JeanLucPons/VanitySearch/commit/ea177b7b36c0db66f110d4358fd4fd4704a6603d right before v1.14 was released.

To fix this mess, all you have to do is pass -fno-strict-aliasing (please write this carefully) in the CXXFLAGS in the makefile.

I am working on a patched version of this where only the offending functions get this flag. So my codebase is spaghetti right now. Please stand by.
Can you enlighten me to what exactly the bug is/is doing? Is it only the -check unit test?
legendary
Activity: 1568
Merit: 6660
bitcoincleanup.com / bitmixlist.org
An update about the chronic address bug on VanitySearch:

I have found the source of the error. In case you didn't know, everything released after VanitySearch 1.13 fails to math any base58 address, on newer GCC (11+, and maybe 10 as well)

This has to do with the sha256.cpp and sha256_sse.cpp files inside the hash/ folder. Specifically, sha256_checksum and sha256sse_checksum functions.

These functions fail to make the correct checksum when you are running with optimization flags on, but I managed to get even more specific than that.

It appears that a single O2 optimization flag, -fstrict-aliasing is causing all the chaos. Taking the SSE version of this function for example:

Code:
void sha256sse_checksum(uint32_t *i0, uint32_t *i1, uint32_t *i2, uint32_t *i3,
  uint8_t *d0, uint8_t *d1, uint8_t *d2, uint8_t *d3) {

  __m128i s[8];

  _sha256sse::Initialize(s);
  _sha256sse::Transform2(s, i0, i1, i2, i3);

#ifndef WIN64
  uint32_t *s32 = (uint32_t *)(&s[0]);
  *((uint32_t *)d0) = __builtin_bswap32(s32[3]);
  *((uint32_t *)d1) = __builtin_bswap32(s32[2]);
  *((uint32_t *)d2) = __builtin_bswap32(s32[1]);
  *((uint32_t *)d3) = __builtin_bswap32(s32[0]);
#else
  *((uint32_t *)d0) = _byteswap_ulong(s[0].m128i_u32[3]);
  *((uint32_t *)d1) = _byteswap_ulong(s[0].m128i_u32[2]);
  *((uint32_t *)d2) = _byteswap_ulong(s[0].m128i_u32[1]);
  *((uint32_t *)d3) = _byteswap_ulong(s[0].m128i_u32[0]);
#endif

}

Aliasing is an optimization technique where the compiler assumes that you're never going to cast variables into crazy types of different sizes and makes some fast but unsafe memory writes as a result.

It's unsafe because if the type indeed is cast into a larger size type, then some garbage will be written into the higher parts of the variable. That's where the gibberish in the checksum characters comes from. The parameters d0-3 are expanded into uint32 and then assigned, and that's where strict aliasing messes things up.

Something similar happens with sha256_checksum but with memcpy instead.

These functions were introduced in the following commit: https://github.com/JeanLucPons/VanitySearch/commit/ea177b7b36c0db66f110d4358fd4fd4704a6603d right before v1.14 was released.

To fix this mess, all you have to do is pass -fno-strict-aliasing (please write this carefully) in the CXXFLAGS in the makefile.

I am working on a patched version of this where only the offending functions get this flag. So my codebase is spaghetti right now. Please stand by.
hero member
Activity: 630
Merit: 731
Bitcoin g33k
Anyone want to lend me their GPU for a few days so I can fix my broken VanitySearch build (and maybe Kangaroo as well as people are starting to complain about the hash table not working and stuff)?

I have $10 in Vast.ai from a while ago but the card used to find it has since expired and I have no way to fund it now.

@NotATether
I've prepared a machine for you. Check your mailbox (DM)
legendary
Activity: 1568
Merit: 6660
bitcoincleanup.com / bitmixlist.org
Anyone want to lend me their GPU for a few days so I can fix my broken VanitySearch build (and maybe Kangaroo as well as people are starting to complain about the hash table not working and stuff)?

I have $10 in Vast.ai from a while ago but the card used to find it has since expired and I have no way to fund it now.
You want me to ship you one? Smiley

Well that won't really make any sense to me, as I only got (quite old) laptops handy at the moment, and the power utility is a b*tch.

I think just put it in a DIY box and slap some temporary SSH access on it for me to get in and debug VanitySearch should do the trick.
full member
Activity: 1050
Merit: 219
Shooters Shoot...
Anyone want to lend me their GPU for a few days so I can fix my broken VanitySearch build (and maybe Kangaroo as well as people are starting to complain about the hash table not working and stuff)?

I have $10 in Vast.ai from a while ago but the card used to find it has since expired and I have no way to fund it now.
You want me to ship you one? Smiley
legendary
Activity: 1568
Merit: 6660
bitcoincleanup.com / bitmixlist.org
Anyone want to lend me their GPU for a few days so I can fix my broken VanitySearch build (and maybe Kangaroo as well as people are starting to complain about the hash table not working and stuff)?

I have $10 in Vast.ai from a while ago but the card used to find it has since expired and I have no way to fund it now.
hero member
Activity: 630
Merit: 731
Bitcoin g33k
I'll be surprised if this bug is in JLP's VanitySearch build as well.

original VanitySearch of JLP works fine, no issues. We certainly would heard about existing bugs
legendary
Activity: 1568
Merit: 6660
bitcoincleanup.com / bitmixlist.org
what's the output of
Code:
./VanitySearch -check

read HERE and HERE

try to compile with gcc 9 instead of the latest gcc 11

in Linux try to set optimization flag -O2 to -O1 in Makefile
In Windows / cygwin64 the same

try IntMod.cpp change
Code:
static Int _P; // Field characteristic in IntMod.cpp

to
Code:
static Int Field_P; 
and all other occurences in file

VanitySearch -check unit test is failing because of "Invalid private key checksum", and indeed, all parts of the address are equal except for the checksum part. So I will investigate this.

I'll be surprised if this bug is in JLP's VanitySearch build as well.
hero member
Activity: 630
Merit: 731
Bitcoin g33k
what's the output of
Code:
./VanitySearch -check

read HERE and HERE

try to compile with gcc 9 instead of the latest gcc 11

in Linux try to set optimization flag -O2 to -O1 in Makefile
In Windows / cygwin64 the same

try IntMod.cpp change
Code:
static Int _P; // Field characteristic in IntMod.cpp

to
Code:
static Int Field_P; 
and all other occurences in file
legendary
Activity: 1568
Merit: 6660
bitcoincleanup.com / bitmixlist.org
A strange error happens when I run my custom build of VanitySearch in non-debug mode - I get:

"Warning wrong private key generated!"

Keep in mind that I did not modify any key generation code, my mods were limited to adding regex support for the prefixes.

In the debug mode, it gets the address right every single time. So what gives?
hero member
Activity: 630
Merit: 731
Bitcoin g33k
how to ask vanitysearch to search for classic bitcoin address and segwit too?

Quote
(P2PKH, P2SH or BECH32 allowed at once)

you cannot mix them up. You have to run and search for a bc1q... address and later on you could run a search for a 1... address
legendary
Activity: 1568
Merit: 6660
bitcoincleanup.com / bitmixlist.org
PSA: If you're getting:

Code:
/usr/bin/ld: obj/main.o: in function `std::vector >::vector(std::vector > const&)':
main.cpp:(.text._ZNSt6vectorIiSaIiEEC2ERKS1_[_ZNSt6vectorIiSaIiEEC5ERKS1_]+0x96): undefined reference to `std::__throw_bad_array_new_length()'
collect2: error: ld returned 1 exit status

while compiling VanitySearch, you need to clear the object files and rebuild the project with:

Code:
make clean
make

It's a GCC bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71757 (despite being closed as WONTFIX, it still pops around apparently) so it should not affect Windows and Visual Studio doesn't need any fix.



Anyone knows how I can add a 3rd party lib to a Visual Studio project?



how to ask vanitysearch to search for classic bitcoin address and segwit too?

It will detect the "1" or "3" or "bc1" at the beginning and automatically choose the correct address type.

newbie
Activity: 6
Merit: 0
how to ask vanitysearch to search for classic bitcoin address and segwit too?


legendary
Activity: 2394
Merit: 5531
Self-proclaimed Genius
im getting memory error.
when I loaded 6219 prefixes (no error), but when i loaded 82 full address i get "illegal memory access error", when i reduce the full address to 15 addresses, it worked.
Now, this is an entirely different issue from your first post, I presume?
Because the command line doesn't show that it stopped after running for hours.

For that error, it's could be the issue when searching for multiple full addresses with Nvidia GPU, try to downgrade to v1.17.
Reference (issue #81): https://github.com/JeanLucPons/VanitySearch/issues/81#issuecomment-745085166
newbie
Activity: 6
Merit: 0
hi
im getting memory error.
when I loaded 6219 prefixes (no error), but when i loaded 82 full address i get "illegal memory access error", when i reduce the full address to 15 addresses, it worked.
I ran GPU memory test and it says no error.

any idea what is the problem?



[Building lookup32 100.0%]
Search: 6219 prefixes (Lookup size 2) [Compressed]
Start Tue Jan  3 21:29:45 2023
Base Key: BD6291C1DE2447068E663194E0DDFBE0C72CD78A8177B51CAE080716AD7496F3
Number of CPU thread: 0
GPU: GPU #0 NVIDIA GeForce RTX 2060 SUPER (34x64 cores) Grid(272x128)
[404.78 Mkey/s][GPU 404.78 Mkey/s][Total 2^32.43][Prob 31.4%][50% in 00:00:11][Found 33]  ^C
C:\Users\abc\Desktop>


C:\Users\abc\Desktop>VanitySearch.exe -stop -i addresses.txt -o found.txt -t 0 -gpu
VanitySearch v1.19
Search: 82 addresses (Lookup size 82,[1,1]) [Compressed]
Start Tue Jan  3 21:32:26 2023
Base Key: 5323DAB06CE0DAC8D3C8E5191F1D6DA70ABFC8F6D9188B89C186845E543C2674
Number of CPU thread: 0
GPU: GPU #0 NVIDIA GeForce RTX 2060 SUPER (34x64 cores) Grid(272x128)
GPUEngine: Launch: an illegal memory access was encountered


C:\Users\abc\Desktop>VanitySearch.exe -stop -i addresses.txt -o found.txt -t 0 -gpu
VanitySearch v1.19
Search: 15 addresses (Lookup size 15,[1,1]) [Compressed]
Start Tue Jan  3 21:32:34 2023
Base Key: A6D841113DFC9539F68F8FF5345D7AECC1A62A4A3651FD86ECF6D3696C3A9644
Number of CPU thread: 0
GPU: GPU #0 NVIDIA GeForce RTX 2060 SUPER (34x64 cores) Grid(272x128)
[1295.00 Mkey/s][GPU 1295.00 Mkey/s][Total 2^32.88][Prob 0.0%][50% in 2.48055e+31y][Found 0]

Again.
What version are of VS you running?
What version of cuda?
It's a probably a Cuda error combined with a poor VM setup.
We need more information.
And initial feedback to previous suggestions would help us troubleshoot. 



thx for the reply.
nvm i just return the card, could be faulty.
hero member
Activity: 1423
Merit: 504
hi
im getting memory error.
when I loaded 6219 prefixes (no error), but when i loaded 82 full address i get "illegal memory access error", when i reduce the full address to 15 addresses, it worked.
I ran GPU memory test and it says no error.

any idea what is the problem?



[Building lookup32 100.0%]
Search: 6219 prefixes (Lookup size 2) [Compressed]
Start Tue Jan  3 21:29:45 2023
Base Key: BD6291C1DE2447068E663194E0DDFBE0C72CD78A8177B51CAE080716AD7496F3
Number of CPU thread: 0
GPU: GPU #0 NVIDIA GeForce RTX 2060 SUPER (34x64 cores) Grid(272x128)
[404.78 Mkey/s][GPU 404.78 Mkey/s][Total 2^32.43][Prob 31.4%][50% in 00:00:11][Found 33]  ^C
C:\Users\abc\Desktop>


C:\Users\abc\Desktop>VanitySearch.exe -stop -i addresses.txt -o found.txt -t 0 -gpu
VanitySearch v1.19
Search: 82 addresses (Lookup size 82,[1,1]) [Compressed]
Start Tue Jan  3 21:32:26 2023
Base Key: 5323DAB06CE0DAC8D3C8E5191F1D6DA70ABFC8F6D9188B89C186845E543C2674
Number of CPU thread: 0
GPU: GPU #0 NVIDIA GeForce RTX 2060 SUPER (34x64 cores) Grid(272x128)
GPUEngine: Launch: an illegal memory access was encountered


C:\Users\abc\Desktop>VanitySearch.exe -stop -i addresses.txt -o found.txt -t 0 -gpu
VanitySearch v1.19
Search: 15 addresses (Lookup size 15,[1,1]) [Compressed]
Start Tue Jan  3 21:32:34 2023
Base Key: A6D841113DFC9539F68F8FF5345D7AECC1A62A4A3651FD86ECF6D3696C3A9644
Number of CPU thread: 0
GPU: GPU #0 NVIDIA GeForce RTX 2060 SUPER (34x64 cores) Grid(272x128)
[1295.00 Mkey/s][GPU 1295.00 Mkey/s][Total 2^32.88][Prob 0.0%][50% in 2.48055e+31y][Found 0]

Again.
What version are of VS you running?
What version of cuda?
It's a probably a Cuda error combined with a poor VM setup.
We need more information.
And initial feedback to previous suggestions would help us troubleshoot. 

newbie
Activity: 6
Merit: 0
hi
im getting memory error.
when I loaded 6219 prefixes (no error), but when i loaded 82 full address i get "illegal memory access error", when i reduce the full address to 15 addresses, it worked.
I ran GPU memory test and it says no error.

any idea what is the problem?



[Building lookup32 100.0%]
Search: 6219 prefixes (Lookup size 2) [Compressed]
Start Tue Jan  3 21:29:45 2023
Base Key: BD6291C1DE2447068E663194E0DDFBE0C72CD78A8177B51CAE080716AD7496F3
Number of CPU thread: 0
GPU: GPU #0 NVIDIA GeForce RTX 2060 SUPER (34x64 cores) Grid(272x128)
[404.78 Mkey/s][GPU 404.78 Mkey/s][Total 2^32.43][Prob 31.4%][50% in 00:00:11][Found 33]  ^C
C:\Users\abc\Desktop>


C:\Users\abc\Desktop>VanitySearch.exe -stop -i addresses.txt -o found.txt -t 0 -gpu
VanitySearch v1.19
Search: 82 addresses (Lookup size 82,[1,1]) [Compressed]
Start Tue Jan  3 21:32:26 2023
Base Key: 5323DAB06CE0DAC8D3C8E5191F1D6DA70ABFC8F6D9188B89C186845E543C2674
Number of CPU thread: 0
GPU: GPU #0 NVIDIA GeForce RTX 2060 SUPER (34x64 cores) Grid(272x128)
GPUEngine: Launch: an illegal memory access was encountered


C:\Users\abc\Desktop>VanitySearch.exe -stop -i addresses.txt -o found.txt -t 0 -gpu
VanitySearch v1.19
Search: 15 addresses (Lookup size 15,[1,1]) [Compressed]
Start Tue Jan  3 21:32:34 2023
Base Key: A6D841113DFC9539F68F8FF5345D7AECC1A62A4A3651FD86ECF6D3696C3A9644
Number of CPU thread: 0
GPU: GPU #0 NVIDIA GeForce RTX 2060 SUPER (34x64 cores) Grid(272x128)
[1295.00 Mkey/s][GPU 1295.00 Mkey/s][Total 2^32.88][Prob 0.0%][50% in 2.48055e+31y][Found 0]
newbie
Activity: 17
Merit: 7
bech32m support incoming? 🤔
Pages:
Jump to: