Pages:
Author

Topic: Help a newbie; why is hashing not done once but twice during Bitcoin transaction - page 5. (Read 1615 times)

sr. member
Activity: 333
Merit: 506
Or just that length extension attacks and protection against them using double hashing was already a well know thing that nobody bothered asking about it! For example in chapter 6.3.1 of the book Practical Cryptography published in 2003 (Bitcoin was released in 2008) it explains that a countermeasure against such attacks is to compute SHA-X(SHA-X(m)) where X is 1, 256, 384 or 512.
HMACSHA functions also use the same principle to solve the length extension attacks which is a problem in MACs not so much in simple hashes.

Agreed.  

I think it's fair enough to question though.
With any hashing function, you lose information and risk an increased number of collisions. If you hash twice, you risk more collisions from what I understand. This is a risk - but we agree that this is an acceptable risk so that we can sign transactions, as long as we start off with enough random information.

The risk that Satoshi really played with here though is the long term building of a currency based off of this.
I mean, fair enough, if you want to use methods like this for your one off encrypted projects, your personal project, your messenger services, or even your large corporations. Most of these are personal and don't require long term security that a currency needs - which has to be resilient to both change and attacks. Most of these don't involve multi-billion dollar risks if it's broken, or have the ability to adapt over time to changing technology. Bitcoin needs to be set as a protocol and have long term resilience unless you want to force everyone to transact their entire life savings frequently (which you don't). I think it was a risk for Satoshi to have played with the double hash due to this need for longevity, although that risk was clearly much less by the larger risk of a single hash. (Risks everywhere? The benefit of hindsight..) That the double hash has worked for 20 years generally (edit: I guess longer, actually!) and 13 years in bitcoin is a credit to it, but, man, the difference between a few years isn't that much for something hoped to last decades or longer. The longevity of the double hash was still an unknown, even if it was better in theory. There are several other things that have been broken and fixed over time in bitcoin that I'm surprised as a novice at how resilient the double hash has been.

I was responding more to why it wasn't discussed in the early forum. You had varying degrees of understanding. The first few forum members still seemed to be wrapping their head about the basics without going into detail about much more complicated topics. The very few people who had been working with Satoshi pre-forum and later more serious developers like Andresen had probably already thought about it or looked things like that up quickly. So I agree with you, they didn't need to discuss it. I also think things may not always have been openly discussed when they present possible flaws, because you have groups that want to patch before exploitation or to use those exploits for their own greed - the double hash might have been caught up into a phenomenon like that even if it never had that weakness. So there are several reasons why it wasn't discussed.
legendary
Activity: 2268
Merit: 18587
i thought thats what the nonce was for.
The nonce is only 4 bytes, meaning a total of 4.3 billion possibilities. Miners can exhaust this search space in under a second. They need to be able to change other things in the block in addition to the nonce. The timestamp is one such option for them, alongside extraNonce, which transactions they include, and the order of those transactions.

I don't know. I just thought they were.
Blockchain.com is not a reputable company. I would not even use their products, let alone attach any degree of authority to them.

Without an accurate timestamp on your transaction how do you properly convince a tax authority exactly when a transaction occurred?
But this becomes trivial to fake, since a signed transaction can be not broadcasted for any length of time. I could easy sign and broadcast a transaction with a timestamp of 5 years ago to gain beneficial tax treatment. As far as most governments are concerned, the transaction occurs when it was confirmed. Usually the time between a transaction being broadcast and being confirmed is a few hours at most, so irrelevant from a tax point of view.
legendary
Activity: 3472
Merit: 10611
I'm going to guess that it wasn't discussed when Satoshi was around for two reasons: 1) It was implicit. Satoshi didn't explain a lot of what he did unless questioned on it. Length extension attack would have been an obvious vector if he had played with it. Besides, questioning all potential flaws publicly in a developing system might cause people to try to break it before you can secure it, when you prefer to keep testing its resilience privately. 2) Either the users were still developing and didn't have enough understanding of the code and mathematics to question it, or understood your reason as the reason. There was nothing to question for the last group.
Or just that length extension attacks and protection against them using double hashing was already a well know thing that nobody bothered asking about it! For example in chapter 6.3.1 of the book Practical Cryptography published in 2003 (Bitcoin was released in 2008) it explains that a countermeasure against such attacks is to compute SHA-X(SHA-X(m)) where X is 1, 256, 384 or 512.
HMACSHA functions also use the same principle to solve the length extension attacks which is a problem in MACs not so much in simple hashes.
sr. member
Activity: 333
Merit: 506
Secondly why is hashing not done once but twice in Bitcoin?
That is a good question. It turns out that it has to do with the so-called length extension attack. I have never seen this been discussed back in the early days, though, so there may be another reason.

Oooo.
You hit the nail on the head there.
I'm going to guess that it wasn't discussed when Satoshi was around for two reasons: 1) It was implicit. Satoshi didn't explain a lot of what he did unless questioned on it. Length extension attack would have been an obvious vector if he had played with it. Besides, questioning all potential flaws publicly in a developing system might cause people to try to break it before you can secure it, when you prefer to keep testing its resilience privately. 2) Either the users were still developing and didn't have enough understanding of the code and mathematics to question it, or understood your reason as the reason. There was nothing to question for the last group.

Thus, it never came up.
legendary
Activity: 4354
Merit: 3260
Quote
Adding timestamp to transaction require soft/hard fork, so you need to convince Bitcoin community instead. Although it's different case if it's opt-in where you simply utilize OP_RETURN.
But you can stuff anything you want to in OP_RETURN. Including fake timestamp.

Even if a transaction had a timestamp, you could fake that, too. There is nothing that could ensure that a block's and transaction's timestamps are accurate. Remember that it can take days for a transaction to be included in a block, it can take minutes for a block to propagate, and node times are not synchronized.
legendary
Activity: 2870
Merit: 7490
Crypto Swap Exchange
Quote
But most of those vendor are big IT company with big net worth. Can you give few example of vendor or application who cares about quality?
Someone that is writing a program for their internal use only.

I doubt it, especially if internal use refer to small-medium or non-profit company.

Someone that develops a smart contract that has to be misssion critical and contain no errors so it has to be done not with spaghetti code but in a structured well thought out manner. now time tells how good their contract is. and it might be better than bitcoin in that sense that it has no bugs.

It's valid example, although the scale between developing smart contract and complex application/OS is very different.

--snip--

Apparantly such transaction timestamps were not considered to be needed or beneficial. (It doesn't matter if YOU find it necessary or beneficial or plain nice-to-have. Of course, you're free to try to convince a majority of Bitcoin nodes... Grin)

Adding timestamp to transaction require soft/hard fork, so you need to convince Bitcoin community instead. Although it's different case if it's opt-in where you simply utilize OP_RETURN.
legendary
Activity: 1512
Merit: 7340
Farewell, Leo
well i see what you mean but it is still counterintuitive to see it happening. is that what the timestamp was really meant for is something that miners could manipulate for their own purposes?
It's meant to establish the difficulty. As said, there's a range they're allowed to manipulate, which doesn't affect the adjustment by a lot. You cannot avoid this limit; otherwise they'll perpetually sacrifice computational power for blocks that aren't complied with your rules.

i thought thats what the nonce was for.
The nonce is just a number used once by miners on their way to find a valid block hash. Check: Nonce.

Well i can think of alot of reason why a timestamp on a transaction is helpful. Every transaction in e-commerce uses timestamps.
Yes, but that has a point only in trust-based systems. In Bitcoin, that works asynchronously, you can't know when blockchain dot com's node received the transaction or when you did. When it gets confirmed, your running node will immediately receive it, but one who's syncing won't have it yet. If the signer can set the timestamp, which doesn't make sense, you can't know if it's faked.
sr. member
Activity: 1036
Merit: 350
that's what i thought but if that's the case then blocks can't be in the wrong order based on timestamp.
They can, as I explained above. Here is an example:

145045 - 16:05
145046 - 16:00
145047 - 15:53
145048 - 16:04
145049 - 16:08

well i see what you mean but it is still counterintuitive to see it happening. is that what the timestamp was really meant for is something that miners could manipulate for their own purposes? i thought thats what the nonce was for.


Quote

What makes you think what blockchain.com does is or should in any way be "authoritive" for the Bitcoin space?

I don't know. I just thought they were.

Quote
A Bitcoin transaction is "done" when the block has been mined which contains this transaction and this block remains in the longest consensus chain of blocks. For the Bitcoin protocol and consensus it doesn't matter at all when exactly this transaction has been made aware into a node's mempool and how long it takes to be mined from a miner's mempool into a block. Adding timestamps and the space it needs to save such details makes only sense if you need those details.
Well i can think of alot of reason why a timestamp on a transaction is helpful. Every transaction in e-commerce uses timestamps. Without an accurate timestamp on your transaction how do you properly convince a tax authority exactly when a transaction occurred? You can't. All you can do is point to the block timestamp but what if that's not the same as when you actually submitted the transaction and it affects you negatively? Did you ever buy anything from amazon and not see a timestamp on your transaction?

Quote
Apparantly such transaction timestamps were not considered to be needed or beneficial.
Well whoever that was didn't have e-commerce in mind.


Quote
Adding timestamp to transaction require soft/hard fork, so you need to convince Bitcoin community instead. Although it's different case if it's opt-in where you simply utilize OP_RETURN.
But you can stuff anything you want to in OP_RETURN. Including fake timestamp.



hero member
Activity: 714
Merit: 1010
Crypto Swap Exchange
Blockchain.com timestamps each transaction inside each block. Not every transaction has the same timestamp.
As above, this is when blockchain.com's node first learns of that transaction. This is not related to bitcoin at a protocol level.
well it's a nice feature to be able to see when a transaction was done. maybe they should add that to the protocol. everything should have a timestamp i would think. or some way of deciding the ordering of objects in time. even down to the transaction level. if blockchain.com can do it why can't bitcoin?
What makes you think what blockchain.com does is or should in any way be "authoritive" for the Bitcoin space?

A Bitcoin transaction is "done" when the block has been mined which contains this transaction and this block remains in the longest consensus chain of blocks. For the Bitcoin protocol and consensus it doesn't matter at all when exactly this transaction has been made aware into a node's mempool and how long it takes to be mined from a miner's mempool into a block. Adding timestamps and the space it needs to save such details makes only sense if you need those details. Apparantly such transaction timestamps were not considered to be needed or beneficial. (It doesn't matter if YOU find it necessary or beneficial or plain nice-to-have. Of course, you're free to try to convince a majority of Bitcoin nodes... Grin)
legendary
Activity: 2268
Merit: 18587
that's what i thought but if that's the case then blocks can't be in the wrong order based on timestamp.
They can, as I explained above. Here is an example:

145045 - 16:05
145046 - 16:00
145047 - 15:53
145048 - 16:04
145049 - 16:08

So a miner can just stamp a block with any old time they want? Like 50 years in the future?
The timestamp can range from the median time of the last 11 blocks (plus 1 second) to 2 hours in to the future (based on network adjusted time, not local time).

If that's the case then something isn't right. And I don't see how bitcoin could function like that. Seems like a bug.
Because the timestamp is used to determine the difficulty, not to determine the order of the blocks. It doesn't need to be accurate beyond a couple of hours.
legendary
Activity: 3472
Merit: 10611
that's what i thought but if that's the case then blocks can't be in the wrong order based on timestamp. blocks with earlier timestamps have to come before blocks with later timestamps. otherwise, there is no well defined notation of "time between 2 blocks".
No, the order is, as I said, achieved using cryptography. Meaning block C references block B by including its hash in the header. That way the order can not change.
The block time can actually be different meaning block B can have a time that is bigger than block C. That is why the adjustment is not done every block but postponed to 2016 blocks/2 weeks and is a median time. There is also limitations on how wrong the time can be (up to 2 hours IIRC).
sr. member
Activity: 1036
Merit: 350
Note that the timestamp of blocks is irrelevant to deciding the ordering of them. Blocks are ordered based on their height, not their timestamp.
So a miner can just stamp a block with any old time they want? Like 50 years in the future?

Quote
There have been plenty of occasions where blocks have had earlier timestamps than blocks which came before them.
If that's the case then something isn't right. And I don't see how bitcoin could function like that. Seems like a bug.
legendary
Activity: 3472
Merit: 10611
I don't know I thought that timestamps were supposed to be a way of putting things into order. If that's not the case then what's their purpose?
We put things in order by creating a "chain" using cryptography, where block C comes after block B and before block D.
The timestamp is only used in block headers to be used for difficulty adjustment so that we can come up with a target value that lets use find 2016 blocks every 2 weeks.
sr. member
Activity: 1036
Merit: 350

Blockchain can do it because they are a single node recording the time that their node first saw a transaction, and recording it on a centralized database they control. What if my node sees a transaction now, and yours sees it 5 minutes later? Which timestamp to we go with? What if my mempool is full, but your node has different parameters meaning it accepts transaction I am ignoring, and we end up seeing the transaction days or even weeks apart? What if I sign a transaction with a timestamp of today, but choose not to broadcast it for a day, a month, a year? And what do we have to gain from it? It will make every transaction larger and therefore more expensive, and I can't see any meaningful benefit.

I don't know I thought that timestamps were supposed to be a way of putting things into order. If that's not the case then what's their purpose? Like for example if you had a way to timestamp transactions then you could create blocks based on transactions timestamps.

Quote
Note that the timestamp of blocks is irrelevant to deciding the ordering of them. Blocks are ordered based on their height, not their timestamp. There have been plenty of occasions where blocks have had earlier timestamps than blocks which came before them.

Then I'm not even sure why blocks contain a timestamp. If the timestamp isn't used.

Quote
But most of those vendor are big IT company with big net worth. Can you give few example of vendor or application who cares about quality?
Someone that is writing a program for their internal use only. Someone that develops a smart contract that has to be misssion critical and contain no errors so it has to be done not with spaghetti code but in a structured well thought out manner. now time tells how good their contract is. and it might be better than bitcoin in that sense that it has no bugs.
legendary
Activity: 2268
Merit: 18587
ok well how am i supposed to know that?
Because as I said above, transactions do not have timestamps.

well it's a nice feature to be able to see when a transaction was done. maybe they should add that to the protocol. everything should have a timestamp i would think. or some way of deciding the ordering of objects in time. even down to the transaction level. if blockchain.com can do it why can't bitcoin?
Blockchain can do it because they are a single node recording the time that their node first saw a transaction, and recording it on a centralized database they control. What if my node sees a transaction now, and yours sees it 5 minutes later? Which timestamp to we go with? What if my mempool is full, but your node has different parameters meaning it accepts transaction I am ignoring, and we end up seeing the transaction days or even weeks apart? What if I sign a transaction with a timestamp of today, but choose not to broadcast it for a day, a month, a year? And what do we have to gain from it? It will make every transaction larger and therefore more expensive, and I can't see any meaningful benefit.

Note that the timestamp of blocks is irrelevant to deciding the ordering of them. Blocks are ordered based on their height, not their timestamp. There have been plenty of occasions where blocks have had earlier timestamps than blocks which came before them.
legendary
Activity: 2870
Merit: 7490
Crypto Swap Exchange
Blockchain.com timestamps each transaction inside each block. Not every transaction has the same timestamp.
As above, this is when blockchain.com's node first learns of that transaction. This is not related to bitcoin at a protocol level.
well it's a nice feature to be able to see when a transaction was done. maybe they should add that to the protocol. everything should have a timestamp i would think. or some way of deciding the ordering of objects in time. even down to the transaction level. if blockchain.com can do it why can't bitcoin?

It's possible, but
1. It introduce small bloat. Could be 4 or 8 bytes depending on whether you concerned about year 2038/2106 problem.
2. Since transaction could broadcaster far before it's included on block, it's hard to verify acuuracy of the timestamp on transaction.

Quote
You should spend few minutes on website which list CVE (Common Vulnerabilities and Exposures) such as https://www.cvedetails.com/cvss-score-distribution.php (https://www.cvedetails.com/cvss-score-distribution.php). If you browse The "Top 50", you'll find out popular product/vendor have hundred to thousand CVE.

that's because they don't care about quality, they are concerned with pumping crap out the door. getting it to market to make more bucks.

But most of those vendor are big IT company with big net worth. Can you give few example of vendor or application who cares about quality?
sr. member
Activity: 1036
Merit: 350
Yeah on that particular block explorer, but not on blockchain.com. If you look at blockchain.com, you can see that block 91880 has the same timestamp as block 91722 for the coinbase transaction.
That's a fault of blockchain.com and has nothing to do with the bitcoin protocol. Blockchain.com assigns a timestamp to each transaction based on when they first saw that transaction in their mempool, which is why the duplicate second transaction has the same timestamp as the first transaction. The internal workings of blockchain.com are irrelevant to how the network treats these transactions.

ok well how am i supposed to know that? it doesn't exactly say that on their website. so i figured its part of the blockchain the timestamp data.


Blockchain.com timestamps each transaction inside each block. Not every transaction has the same timestamp.
As above, this is when blockchain.com's node first learns of that transaction. This is not related to bitcoin at a protocol level.
well it's a nice feature to be able to see when a transaction was done. maybe they should add that to the protocol. everything should have a timestamp i would think. or some way of deciding the ordering of objects in time. even down to the transaction level. if blockchain.com can do it why can't bitcoin?

i would think by now with computer software and technology development being what it is, automated systems have taken over finding and fixing bugs.
You have some very unrealistic views on how software is developed.
well if you think bitcoin is the best software development model ever developed then i would have to disagree. i think there are better ones out there. but as long as bitcoin keeps working its fine but hopefully no big cve comes along and makes bitcoin go to 0. that could happen. it almost did once. luckily we had satoshi to save our ass that time.
 


Quote
You should spend few minutes on website which list CVE (Common Vulnerabilities and Exposures) such as https://www.cvedetails.com/cvss-score-distribution.php (https://www.cvedetails.com/cvss-score-distribution.php). If you browse The "Top 50", you'll find out popular product/vendor have hundred to thousand CVE.

that's because they don't care about quality, they are concerned with pumping crap out the door. getting it to market to make more bucks.

legendary
Activity: 2870
Merit: 7490
Crypto Swap Exchange
Quote
It is impossible to reproduce the same bug again, as this is prevented by both BIP30 and BIP34. Are there other bugs out there we are unaware of which could permanently remove some bitcoin from the supply? Most likely.
i would think by now with computer software and technology development being what it is, automated systems have taken over finding and fixing bugs. if they would use that on bitcoin, it might help. maybe it's more complicated than that but i think it's a reasonable goal.

You should spend few minutes on website which list CVE (Common Vulnerabilities and Exposures) such as https://www.cvedetails.com/cvss-score-distribution.php. If you browse The "Top 50", you'll find out popular product/vendor have hundred to thousand CVE.

what would be so hard about programming a system to find bugs like this duplicate transaction id?

First of all, thinking all possible bug or unwanted behavior.

Quote
Their bank never has bugs that affect their account balance or if they do they get fixed so that no one loses any money...
I've read countless stories about bugs in bank account balances. Say 10 billion dollar balance, or transactions that get processed twice. Usually the bank indeed fixes it, but it can still cause problems (such as spending money you don't have).

Even worse, your balance suddenly reduced. In such cases, it's unlikely the bank give any compensation (unless they have decent customer service or PR).
legendary
Activity: 1512
Merit: 7340
Farewell, Leo
what would be so hard about programming a system to find bugs like this duplicate transaction id?
That it may function falsely. Welcome to software engineering. Smiley

You can't know you have a bug until you do something you'd never done before which appeared an unexpected result. The only way to lower your chances is to do structured programming which makes your code easier to read and develop further by third parties. It also takes less time to build things that way.

From my experience, when I start over-complicating my code for some reason, it's usually when I know I'll have trouble.

Their bank never has bugs that affect their account balance or if they do they get fixed so that no one loses any money...
Start reading: https://www.ptsecurity.com/ww-en/analytics/vulnerabilities-rbo-2019/
legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
Yeah on that particular block explorer, but not on blockchain.com. If you look at blockchain.com, you can see that block 91880 has the same timestamp as block 91722 for the coinbase transaction.
That's a good reminder you shouldn't always trust block explorers (and especially not to trust blockchain.com, they have far too many bugs).

Quote
Their bank never has bugs that affect their account balance or if they do they get fixed so that no one loses any money...
I've read countless stories about bugs in bank account balances. Say 10 billion dollar balance, or transactions that get processed twice. Usually the bank indeed fixes it, but it can still cause problems (such as spending money you don't have).
Pages:
Jump to: