Need Help Understanding Base58Check Encoding...

achow101

staff

Activity: 3458

Merit: 6793

Just writing some code

Quote from: ExabyteMiner on April 13, 2016, 07:29:11 PM

Quote from: achow101 on April 13, 2016, 07:11:39 PM

Quote from: ExabyteMiner on April 13, 2016, 07:03:42 PM

Sorry, again, I am self-taught when it comes to coding, I do not fully understand the pseudocode... From what I understand I have to change "8044d00f6eb2e5491cd7ab7e7185d81b67a23c4980f62b2ed0914d32b7eb1c55815b0e9051" into a "big integer" and then perform some math on it. How do i convert a plain text string to a "big integer"?

It means treat that hex string as a really big integer encoded in hex (base16). The easiest thing is to go to a site like http://www.wolframalpha.com/ and type that string followed by " convert to decimal" and it will give you a really large decimal integer. Then you just use that number to do the conversions.

Also, when working with hex, you always need to think of hex not as a string but as bytes and integers.

When i go to that site and type:

Code:

8044d00f6eb2e5491cd7ab7e7185d81b67a23c4980f62b2ed0914d32b7eb1c55815b0e9051 convert to decimal

I get:

Code:

Wolfram|Alpha doesn't understand your query

Showing instead result for query: decimal

Oh, you need to prepend it with a 0x and use base10, not decimal, sorry. So 0x8044d.... convert to base10

ExabyteMiner

member

Activity: 82

Merit: 10

Quote from: achow101 on April 13, 2016, 07:11:39 PM

Quote from: ExabyteMiner on April 13, 2016, 07:03:42 PM

Sorry, again, I am self-taught when it comes to coding, I do not fully understand the pseudocode... From what I understand I have to change "8044d00f6eb2e5491cd7ab7e7185d81b67a23c4980f62b2ed0914d32b7eb1c55815b0e9051" into a "big integer" and then perform some math on it. How do i convert a plain text string to a "big integer"?

It means treat that hex string as a really big integer encoded in hex (base16). The easiest thing is to go to a site like http://www.wolframalpha.com/ and type that string followed by " convert to decimal" and it will give you a really large decimal integer. Then you just use that number to do the conversions.

Also, when working with hex, you always need to think of hex not as a string but as bytes and integers.

When i go to that site and type:

Code:

8044d00f6eb2e5491cd7ab7e7185d81b67a23c4980f62b2ed0914d32b7eb1c55815b0e9051 convert to decimal

I get:

Code:

Wolfram|Alpha doesn't understand your query

Showing instead result for query: decimal

achow101

staff

Activity: 3458

Merit: 6793

Just writing some code

Quote from: ExabyteMiner on April 13, 2016, 07:03:42 PM

Sorry, again, I am self-taught when it comes to coding, I do not fully understand the pseudocode... From what I understand I have to change "8044d00f6eb2e5491cd7ab7e7185d81b67a23c4980f62b2ed0914d32b7eb1c55815b0e9051" into a "big integer" and then perform some math on it. How do i convert a plain text string to a "big integer"?

It means treat that hex string as a really big integer encoded in hex (base16). The easiest thing is to go to a site like http://www.wolframalpha.com/ and type that string followed by " convert to decimal" and it will give you a really large decimal integer. Then you just use that number to do the conversions.

Also, when working with hex, you always need to think of hex not as a string but as bytes and integers.

ExabyteMiner

member

Activity: 82

Merit: 10

Quote from: achow101 on April 13, 2016, 06:51:26 PM

Quote from: ExabyteMiner on April 13, 2016, 06:39:46 PM

OK, I am very close, I am able to reliably produce correct results all the way up to the actual Base58 encoding now.

After finding and adding the checksum to the end of my string from step 3, I have:

Code:

8044d00f6eb2e5491cd7ab7e7185d81b67a23c4980f62b2ed0914d32b7eb1c55815b0e9051

Now, how do I do the final Base58 encoding to turn that result into:

Code:

5JLbJxi9koHHvyFEAERHLYwG7VxYATnf8YdA9fiC6kXMghkYXpk

The base58 conversion is specific to Bitcoin. There is pseudocode and a chart of characters here: https://en.bitcoin.it/wiki/Base58Check_encoding#Base58_symbol_chart

Sorry, again, I am self-taught when it comes to coding, I do not fully understand the pseudocode... From what I understand I have to change "8044d00f6eb2e5491cd7ab7e7185d81b67a23c4980f62b2ed0914d32b7eb1c55815b0e9051" into a "big integer" and then perform some math on it. How do i convert a plain text string to a "big integer"?

achow101

staff

Activity: 3458

Merit: 6793

Just writing some code

Quote from: ExabyteMiner on April 13, 2016, 06:39:46 PM

OK, I am very close, I am able to reliably produce correct results all the way up to the actual Base58 encoding now.

After finding and adding the checksum to the end of my string from step 3, I have:

Code:

8044d00f6eb2e5491cd7ab7e7185d81b67a23c4980f62b2ed0914d32b7eb1c55815b0e9051

Now, how do I do the final Base58 encoding to turn that result into:

Code:

5JLbJxi9koHHvyFEAERHLYwG7VxYATnf8YdA9fiC6kXMghkYXpk

The base58 conversion is specific to Bitcoin. There is pseudocode and a chart of characters here: https://en.bitcoin.it/wiki/Base58Check_encoding#Base58_symbol_chart

ExabyteMiner

member

Activity: 82

Merit: 10

OK, I am very close, I am able to reliably produce correct results all the way up to the actual Base58 encoding now.

After finding and adding the checksum to the end of my string from step 3, I have:

Code:

8044d00f6eb2e5491cd7ab7e7185d81b67a23c4980f62b2ed0914d32b7eb1c55815b0e9051

Now, how do I do the final Base58 encoding to turn that result into:

Code:

5JLbJxi9koHHvyFEAERHLYwG7VxYATnf8YdA9fiC6kXMghkYXpk

ExabyteMiner

member

Activity: 82

Merit: 10

Quote from: DannyHamilton on April 13, 2016, 09:36:07 AM

The problem isn't your understanding of the process (other than perhaps your understanding of the difference between a hex value and an ASCII string representation of that hex value).

The main problem is in your understanding of the tool you've chosen to use:

https://searchcode.com/codesearch/view/11088282/

Quote

cmdhashgen is a Command Line Utility that can be used to generate
various hashes for a given String or File

Since you don't want to hash the "string", that means that you'll need to create a file that contains ONLY the 32 byte value that you want to hash.

This can be done with xxd if you are using a system that has it...

First use a file editor to create a file that contains the representation that you would have gotten if you had used xxd to convert a file that originally had the value:

Code:

0000000: 8044 d00f 6eb2 e549 1cd7 ab7e 7185 d81b  ................
0000010: 67a2 3c49 80f6 2b2e d091 4d32 b7eb 1c55  ................
0000020: 81                                       .

You can see here that if you strip out the first column (which is the byte offset into the file where that line of data occurs) and the last column of dots (which is just a placeholder for the ASCII character that each byte represents) the remaining data is your 8044d00f6eb2e5491cd7ab7e7185d81b67a23c4980f62b2ed0914d32b7eb1c5581.

Next, use the "reverse" functionality of xxd to convert back to a binary representation:

Code:

xxd -r fileWeJustCreated > ourNewFileContainingTheDesiredValueToHash

We now have a file named "ourNewFileContainingTheDesiredValueToHash" that contains a 32 byte value.

Hash this file, and the result should be b54909e60352ef3ebdd448b65beda938e103624b207c91229bde6e9ba60351e1. I don't use cmdhashgen, but here's what I get when I use openssl:

Code:

openssl dgst -sha256 ourNewFileContainingTheDesiredValueToHash
SHA256(ourNewFileContainingTheDesiredValueToHash)= b54909e60352ef3ebdd448b65beda938e103624b207c91229bde6e9ba60351e1

Now if I simply captured this output, I'd have an ASCII string representation of "b54909e60352ef3ebdd448b65beda938e103624b207c91229bde6e9ba60351e1". Since we need to hash this value, we want the binary output from openssl. openssl has a "-binary" command line option that will output the actual 32 byte value instead of the 64 character ASCII string representation.

Code:

openssl dgst -sha256 -binary ourNewFileContainingTheDesiredValueToHash > newerFileContainingFinalValueToHash

We now have a file named "newerFileContainingFinalValueToHash" that contains a 32 byte value. If cmdhashgen does not have the ability to output the 32 byte value, you'll need to repeat the "reverse xxd" process that we did with the previous value to create your file.

I can now hash this file to get the final value that will be used for the checksum:

Code:

openssl dgst -sha256 newerFileContainingFinalValueToHash
SHA256(newerFileContainingFinalValueToHash)= 5b0e90510294762f4553a6f717a13d5dce4c4aa6137311b950f2ca90ffc9a035

Thank you for your reply!

OK, I have xxd and openssl both and I seem to be getting some strange results.

When i create NewFile.txt with the data:

Code:

8044d00f6eb2e5491cd7ab7e7185d81b67a23c4980f62b2ed0914d32b7eb1c5581

and then run:

Code:

xxd -r NewFile.txt > NewFile2.txt

NewFile2.txt comes back empty...

Also, if I run:

Code:

xxd NewFile.txt > NewFile2.txt

I get:

Code:

0000000: 3830 3434 6430 3066 3665 6232 6535 3439  8044d00f6eb2e549
0000010: 3163 6437 6162 3765 3731 3835 6438 3162  1cd7ab7e7185d81b
0000020: 3637 6132 3363 3439 3830 6636 3262 3265  67a23c4980f62b2e
0000030: 6430 3931 3464 3332 6237 6562 3163 3535  d0914d32b7eb1c55
0000040: 3831                                     81

Which looks a bit different than the result you got...

EDIT:
Eureka! I needed to add the -p flag! So:

Code:

xxd -r -p NewFile.txt > NewFile2.txt

and then

Code:

openssl dgst -sha256 NewFile2.txt > hashed.txt

Outputs the correct:

Code:

b54909e60352ef3ebdd448b65beda938e103624b207c91229bde6e9ba60351e1

You have definitely placed me on the correct track, thank you so much! Going to see if I can finish the process from here.

DannyHamilton

legendary

Activity: 3472

Merit: 4801

Quote from: ExabyteMiner on April 13, 2016, 07:59:49 AM

Bumping in hopes of an answer...

I appreciate the help given so far, I just need to understand this one process.

I understand, and even if you don't feel like explaining it yourself, could you please point me in the right direction by linking me to a resource or giving me proper google search terms that will explain how:

Code:

8044D00F6EB2E5491CD7AB7E7185D81B67A23C4980F62B2ED0914D32B7EB1C5581

becomes:

Code:

0xb54909e60352ef3ebdd448b65beda938e103624b207c91229bde6e9ba60351e1

Thank you in advance for any help finding the answer here...

The problem isn't your understanding of the process (other than perhaps your understanding of the difference between a hex value and an ASCII string representation of that hex value).

The main problem is in your understanding of the tool you've chosen to use:

https://searchcode.com/codesearch/view/11088282/

Quote

cmdhashgen is a Command Line Utility that can be used to generate
various hashes for a given String or File

Since you don't want to hash the "string", that means that you'll need to create a file that contains ONLY the 32 byte value that you want to hash.

This can be done with xxd if you are using a system that has it...

First use a file editor to create a file that contains the representation that you would have gotten if you had used xxd to convert a file that originally had the value:

Code:

0000000: 8044 d00f 6eb2 e549 1cd7 ab7e 7185 d81b  ................
0000010: 67a2 3c49 80f6 2b2e d091 4d32 b7eb 1c55  ................
0000020: 81                                       .

You can see here that if you strip out the first column (which is the byte offset into the file where that line of data occurs) and the last column of dots (which is just a placeholder for the ASCII character that each byte represents) the remaining data is your 8044d00f6eb2e5491cd7ab7e7185d81b67a23c4980f62b2ed0914d32b7eb1c5581.

Next, use the "reverse" functionality of xxd to convert back to a binary representation:

Code:

xxd -r fileWeJustCreated > ourNewFileContainingTheDesiredValueToHash

We now have a file named "ourNewFileContainingTheDesiredValueToHash" that contains a 32 byte value.

Hash this file, and the result should be b54909e60352ef3ebdd448b65beda938e103624b207c91229bde6e9ba60351e1. I don't use cmdhashgen, but here's what I get when I use openssl:

Code:

openssl dgst -sha256 ourNewFileContainingTheDesiredValueToHash
SHA256(ourNewFileContainingTheDesiredValueToHash)= b54909e60352ef3ebdd448b65beda938e103624b207c91229bde6e9ba60351e1

Now if I simply captured this output, I'd have an ASCII string representation of "b54909e60352ef3ebdd448b65beda938e103624b207c91229bde6e9ba60351e1". Since we need to hash this value, we want the binary output from openssl. openssl has a "-binary" command line option that will output the actual 32 byte value instead of the 64 character ASCII string representation.

Code:

openssl dgst -sha256 -binary ourNewFileContainingTheDesiredValueToHash > newerFileContainingFinalValueToHash

We now have a file named "newerFileContainingFinalValueToHash" that contains a 32 byte value. If cmdhashgen does not have the ability to output the 32 byte value, you'll need to repeat the "reverse xxd" process that we did with the previous value to create your file.

I can now hash this file to get the final value that will be used for the checksum:

Code:

openssl dgst -sha256 newerFileContainingFinalValueToHash
SHA256(newerFileContainingFinalValueToHash)= 5b0e90510294762f4553a6f717a13d5dce4c4aa6137311b950f2ca90ffc9a035

achow101

staff

Activity: 3458

Merit: 6793

Just writing some code

Code:

0xb54909e60352ef3ebdd448b65beda938e103624b207c91229bde6e9ba60351e1

Is the hash of

Code:

8044D00F6EB2E5491CD7AB7E7185D81B67A23C4980F62B2ED0914D32B7EB1C5581

when that is hashed as binary data and not as a string.

ExabyteMiner

member

Activity: 82

Merit: 10

Bumping in hopes of an answer...

I appreciate the help given so far, I just need to understand this one process.

I understand, and even if you don't feel like explaining it yourself, could you please point me in the right direction by linking me to a resource or giving me proper google search terms that will explain how:

Code:

8044D00F6EB2E5491CD7AB7E7185D81B67A23C4980F62B2ED0914D32B7EB1C5581

becomes:

Code:

0xb54909e60352ef3ebdd448b65beda938e103624b207c91229bde6e9ba60351e1

Thank you in advance for any help finding the answer here...

ExabyteMiner

member

Activity: 82

Merit: 10

Quote from: DannyHamilton on April 08, 2016, 02:58:28 PM

Because, when you used the site, you calculated the hash on a single 32 byte value represented in hexadecimal radix.

When you used cmdhashgen, you calculated the hash on a string of 64 alphanumeric characters represented in ASCII.

The site calculated the hash using a 32 byte input of:
0xb54909e60352ef3ebdd448b65beda938e103624b207c91229bde6e9ba60351e1

cmdhashgen calculated the hash of a 64 byte input of:
0x62353439303965363033353265663365626464343438623635626564613933386531303336323 462323037633931323239626465366539626136303335316531

I see... Very interesting, thank you for the clarification.

For future reference how does one convert a 64 character alphanumeric hexadecimal representation to the proper 32 byte hexadecimal radix...?

In other words, how do I turn:

Code:

8044D00F6EB2E5491CD7AB7E7185D81B67A23C4980F62B2ED0914D32B7EB1C5581

into the proper:

Code:

0xb54909e60352ef3ebdd448b65beda938e103624b207c91229bde6e9ba60351e1

so it can be passed to cmdhashgen for hashing...?

DannyHamilton

legendary

Activity: 3472

Merit: 4801

Quote from: ExabyteMiner on April 08, 2016, 09:25:35 AM

Why is that result different than what I got when I used cmdhashgen to hash the string in SHA256...? Because I took the same string, ran it through cmdhashgen's SHA256 encoder, took that result and repeated but I got:

Code:

2897b55427f86f73136d55c1729cb4ce56d72e40e33baf8ca10e9e02cb963434

Because, when you used the site, you calculated the hash on a single 32 byte value represented in hexadecimal radix.

When you used cmdhashgen, you calculated the hash on a string of 64 alphanumeric characters represented in ASCII.

The site calculated the hash using a 32 byte input of:
0xb54909e60352ef3ebdd448b65beda938e103624b207c91229bde6e9ba60351e1

cmdhashgen calculated the hash of a 64 byte input of:
0x62353439303965363033353265663365626464343438623635626564613933386531303336323 462323037633931323239626465366539626136303335316531

ExabyteMiner

member

Activity: 82

Merit: 10

Quote from: achow101 on April 08, 2016, 06:47:49 AM

You have to put the hex string in the text box on the field labeled "binary hash". Then scroll down and find the one labeled "sha256" and copy the hex string and put that in text box in "binary hash". Then scroll down again and the hash in" sha256" is the hash that you want for step 4

OK, I took the hex string from step 3 (8044D00F6EB2E5491CD7AB7E7185D81B67A23C4980F62B2ED0914D32B7EB1C5581) and ran it through the site which gave me:

Code:

b54909e60352ef3ebdd448b65beda938e103624b207c91229bde6e9ba60351e1

Then i took that result and ran it through the site again, I got:

Code:

5b0e90510294762f4553a6f717a13d5dce4c4aa6137311b950f2ca90ffc9a035

Why is that result different than what I got when I used cmdhashgen to hash the string in SHA256...? Because I took the same string, ran it through cmdhashgen's SHA256 encoder, took that result and repeated but I got:

Code:

2897b55427f86f73136d55c1729cb4ce56d72e40e33baf8ca10e9e02cb963434

achow101

staff

Activity: 3458

Merit: 6793

Just writing some code

Quote from: ExabyteMiner on April 08, 2016, 12:02:42 AM

Quote from: achow101 on April 07, 2016, 11:25:10 PM

No. You are converting it back to binary. What you just need to do is keep the hex string as it is but instead of hashing it as a string, hash it as bytes. Use the website I posted earlier.

I tried using that website, but it outputs 16 different fields only one of which contains the word bytes and that just shows the same information from step 5 except uppercase letters have been replaced with lower case letters... Is that what I am looking for...?

You have to put the hex string in the text box on the field labeled "binary hash". Then scroll down and find the one labeled "sha256" and copy the hex string and put that in text box in "binary hash". Then scroll down again and the hash in" sha256" is the hash that you want for step 4

ExabyteMiner

member

Activity: 82

Merit: 10

Quote from: achow101 on April 07, 2016, 11:25:10 PM

No. You are converting it back to binary. What you just need to do is keep the hex string as it is but instead of hashing it as a string, hash it as bytes. Use the website I posted earlier.

I think I understand what you mean now, I want to convert it to the hex form that looks like:

Code:

0x80 0x44 0xD0 0x0F 0x6E 0xB2 0xE5 0x49 0x1C 0xD7 0xAB 0x7E 0x71 0x85 0xD8 0x1B 0x67 0xA2 0x3C 0x49 0x80 0xF6 0x2B 0x2E 0xD0 0x91 0x4D 0x32 0xB7 0xEB 0x1C 0x55 0x81 0xb0 0x92 0x6e 0x9a

I'm sure that's not the correct string, but that's what it should basically look like?

ExabyteMiner

member

Activity: 82

Merit: 10

Quote from: DannyHamilton on April 07, 2016, 11:49:14 PM

Are you using a programming language for this process?

If so, which language, and which packages or libraries are you using?

I'm not sure if I should be presenting suggestions and examples in javascript, java, C, C++, perl, ruby, php, or something else entirely.

So far, I have been doing all of these steps using basic DOS/batch scripts that I am writing as I go aside from cmdhashgen which is being used for the SHA256 hashing.

I have some experience and understanding in VBS, C, C++, perl and php.

ExabyteMiner

member

Activity: 82

Merit: 10

Quote from: achow101 on April 07, 2016, 11:25:10 PM

No. You are converting it back to binary. What you just need to do is keep the hex string as it is but instead of hashing it as a string, hash it as bytes. Use the website I posted earlier.

I tried using that website, but it outputs 16 different fields only one of which contains the word bytes and that just shows the same information from step 5 except uppercase letters have been replaced with lower case letters... Is that what I am looking for...?

DannyHamilton

legendary

Activity: 3472

Merit: 4801

Are you using a programming language for this process?

If so, which language, and which packages or libraries are you using?

I'm not sure if I should be presenting suggestions and examples in javascript, java, C, C++, perl, ruby, php, or something else entirely.

achow101

staff

Activity: 3458

Merit: 6793

Just writing some code

Quote from: ExabyteMiner on April 07, 2016, 10:56:36 PM

Thank you so much for your help, I'm getting closer and closer...

OK, so i THINK I have that step figured out...

If I'm not mistaken, when I convert the hex string from step 5 back to hex bytes it becomes:

Code:

1000 0000 0100 0100 1101 0000 0000 1111 0110 1110 1011 0010 1110 0101 0100 1001
0001 1100 1101 0111 1010 1011 0111 1110 0111 0001 1000 0101 1101 1000 0001 1011
0110 0111 1010 0010 0011 1100 0100 1001 1000 0000 1111 0110 0010 1011 0010 1110
1101 0000 1001 0001 0100 1101 0011 0010 1011 0111 1110 1011 0001 1100 0101 0101

Not really sure what to do from here though...

No. You are converting it back to binary. What you just need to do is keep the hex string as it is but instead of hashing it as a string, hash it as bytes. Use the website I posted earlier.

ExabyteMiner

member

Activity: 82

Merit: 10

Quote from: achow101 on April 07, 2016, 09:01:48 PM

So what you have is a hex string which is different from hex bytes. Hex bytes are the human readable form of binary. If you were to convert a set of hex bytes to binary, you would get the 1's and 0's that a computer reads. On the other hand, if you have a string, you have something completely different. At the binary level, each character in the string is a set of bytes, and when you hash the string, you are hashing those bytes, and not the bytes that you actually want which you read in the string.

To hash the bytes, you can use this site: http://www.fileformat.info/tool/hash.htm. Enter the hex string into "Binary Hash" field and it will give you the hashes down below. When you hash again, make sure you do the same thing and hash the bytes.

Thank you so much for your help, I'm getting closer and closer...

OK, so i THINK I have that step figured out...

If I'm not mistaken, when I convert the hex string from step 5 back to hex bytes it becomes:

Code:

1000 0000 0100 0100 1101 0000 0000 1111 0110 1110 1011 0010 1110 0101 0100 1001
0001 1100 1101 0111 1010 1011 0111 1110 0111 0001 1000 0101 1101 1000 0001 1011
0110 0111 1010 0010 0011 1100 0100 1001 1000 0000 1111 0110 0010 1011 0010 1110
1101 0000 1001 0001 0100 1101 0011 0010 1011 0111 1110 1011 0001 1100 0101 0101
1000 0001 0000 1001 0010 0110 1001 1001 1001 1001

Not really sure what to do from here though...

EDIT:
I am definitely doing something wrong because when I put in the hex string from step 5 into the Hexadecimal field at http://convertxy.com/index.php/numberbases/ I get:

Code:

1000 0000 0100 0100 1101 0000 0000 1111 0110 1110 1011 0010 1110 0101 0100 1001 
0001 1100 1101 0111 1010 1011 0111 1110 0111 0001 1000 0101 1101 1000 0001 1011
0110 0111 1010 0010 0011 1100 0100 1001 1000 0000 1111 0110 0010 1011 0010 1110
1101 0000 1001 0001 0100 1101 0011 0010 1011 0111 1110 1011 0001 1100 0101 0101
1000 0001 1011 0000 1001 0010 0110 1110 1001 1010

Which is exactly the same all the way up until the last 8 sets of numbers...

Topic: Need Help Understanding Base58Check Encoding... (Read 1378 times)