Topic: Segwit details? N + 2*numtxids + numvins > N, segwit uses more space than 2MB HF - page 10. (Read 21415 times)

Now that that's all settled: What's Stolfi on about here? The 75% discount?

As long as it fully validates all of the NEW blocks and transactions that it receives. HISTORICAL blocks and the transactions within them are not validated because they are HISTORICAL and are tens of thousands of blocks deep.

And how are you checking the txids if they are not provided? A tx message can be sent unsolicited with a new transaction and it does not contain the txid. In fact, there is no network message that I could find that sends a transaction with its txid. Of course, I think it is safe to assume that if a node requested a specific transaction that it would check the hash of the data it received so that it knows whether that data is correct. But for unsolicited transactions, then the only way to verify them is to check the signature.

No, when bootstrapping historical blocks the witness data is not required because it doesn't need to validate historical blocks. See above.

If you are going to run your node from now until the end of time continuously and save all of the data relevant to the blocks and transactions that it receives and call of that data "permanent blockchain data", then yes, I think it does require more storage than a simple 2 Mb fork.

Since when has anyone ever claimed that segwit is "a way to save permanent blockchain space"?

Since you keep saying stuff about sending transactions between nodes, I don't think you understand how Bitcoin transactions work. It isn't sending between things but creating outputs from inputs after proving that the transaction creator can spend from those inputs. The inputs of a transaction don't affect the outputs of a transaction except for the amounts.

A transaction that spends a segwit input can still create a p2pkh and p2pk output which current nodes and wallets understand. p2pkh and p2pk are two output types that wallets currently understand. Those p2pkh and p2pk outputs can be spent from just like every other p2pkh and p2pk output is now. That will not change. The inputs and the scriptsigs of spending from those outputs will be the exact same as they are today. Segwit doesn't change that.

Rather segwit spends to a special script called a witness program. This script becomes a p2sh address, another output type which current wallets know about and can spend to.

Segwit wallets would instead always create p2sh addresses because that is the only way that segwit can implement witness programs to be backwards compatible. Those p2sh addresses are distributed normally but can only be spent from with a witness program.

Then the attacker is just sending the owner of an address a bunch of Bitcoin. If it is a bunch of spam outputs, then it can be annoying, but that is something that people can already do today.

Well firstly, full nodes don't mine blocks.

The data that composes the block is the data that currently comprises of a block. The header is the same. The Coinbase transaction just has the OP_RETURN output to add the witness root to the blockchain. The transactions are the transactions with the current format. If a block is requested by another node that wants the witness data, then the block is sent with the transactions serialized in the witness serialization format.

And in the future, what is to say that simpletons will be able to understand segwit? In the future, someone would still be saying that segwit is too complicated and that we should not use it. In the future it will still be large changes and it will still be prone to misunderstandings. Nothing will change in the future except instead of increasing the block size limit from 1 Mb to 2 Mb, they will be clamoring to increase the block size limit from 2 Mb to 4 Mb. The situation would literally be the same.

---

Yeah, I asked on #bitcoin-core-dev as achow101 (I go by achow101 pretty much everywhere else except here, although I am also achow101 here). I logged off of IRC because I went to sleep, probably should have asked it earlier.

I will look at the BIP pulls and see if there is anything there.

---

It was originally proposed as a hard fork, but someone (luke-jr I think) pointed out that it could be done as a soft fork. Soft forks are preferred because they are backwards compatible. In this case, the backwards compatibility is that if you run non-upgraded software, you can continue as you were and have no ill effect. You just won't be able to take advantage of the new functionality provided by segwit.

Alternatively, if this were done as a hard fork, then everyone would be required to upgrade in order to deploy segwit and then that would essentially force everyone to use segwit.

Have you been in a cave for the last 6 months?  Did you miss https://bitcoincore.org/en/2016/01/26/segwit-benefits/ ?

Segwit has been explained in many ways, from technical BIPS to colorful info-graphics.

Most wallet, etc. providers had little to no trouble adding segwit, because they like the idea: https://bitcoincore.org/en/segwit_adoption/

jl777 is only whining about his difficulties because he doesn't like anything Core supports and because he's a terrible dev who never finishes a single project he starts (eg SuperNET).

I find the shadowy linkages between alt-coin scammer jl777 and Classic fascinating.  The bitco.in alliance between him, the DashHoles, and the Frap.doc crew make for an interesting demographic.

Since you feel I am deceiving people by using uppercase, I changed it to lower case. The first responses I got did not make it clear the overhead was 2 bytes per tx and 1 byte per vin.

I was under the misunderstanding that segwit saved blockchain space, you know cuz I am idiotic and believed stuff on the internet.

I am glad that now we all know that a 2MB hardfork would be more space efficient than segwit as far as permanent blockchain space.

What I still dont understand is how things will work when a segwit tx is sent to a non-segwit node and that is spent to another non-segwit node. How will the existing wallets deal with that? What happens if an attacker created segwit rawtransactions and sent them to non-segwit nodes? there are no attack vectors? what about in zeroconf environments? how does a full relaying node mine a block with segwit inputs? or do existing full nodes cease to be able to mine blocks after segwit softfork?

And even a simpleton like me can understand how to increase blocksizes with a hardfork, so why not do that before adding massive new changes like segwit? especially since it is more space efficient and not prone to misunderstandings

... and for a while we didn't see any realistic way to deploy it short of rebooting the whole blockchain in a great big flag day (which would inevitably end up unintentionally confiscating some peoples' coins)-- not just a hard fork but an effective _rewrite_.  The clever part of segwit was reorganizing things a bit-- the signature field is still part of the txid but we don't use it for signatures anymore, we use a separate set of fields stapled onto the end to achieve exactly the same effect; but without blowing everything up...

Yet.
Coming soon via core soft fork.

I was told by gmax himself that a node that doesnt validate all signatures should call itself a fully validating node.

Also, I am making an optimized bitcoin core and one of these optimizations is rejecting a tx whose contents doesnt match the txid. The thinking being that if the hashes dont match, there is no point in wasting time calculating the signature

not sure what libsecp256k1's speed has anything to do with the fact that it is still much slower to calculate than SHA256.

So my point again, is that all witness data needs to be stored permanently for a full node that RELAYS historical blocks to a bootstrapping node. If we are to lose this, then we might as well make bitcoin PoS as that is the one weakness for PoS vs PoW. So if you are saying that we need to view bitcoin as fully SPV all the time with PoS level security for bootstrapping nodes, ok, with those assumptions lots and lots of space is saved.

However, with such drastic assumptions I can (and have) already saved lots more space without adding a giant amount of new protocol and processing.

So this controversy has at least clarified that segwit INCREASES the size of the permanently needed data for fully validating and relaying node. Of course for SPV nodes things are much improved, but my discussion is not about SPV nodes.

So the powers that be can call me whatever names they want. I still claim that:

N + 2*numtx + numvins > N

And as such segwit as way to save permanent blockchain space is an invalid claim.Now the cost of 2*numtx+numvins is not that big, so maybe it is worth the cost for all the benefits we get.

However on the benefits claims, one of them is the utxo dataset is becoming a lot more manageable. this is irrelevant as that is a local inefficiency that can be optimized without any external effects. I have it down to 4 bytes of RAM per utxo, but I could make it smaller if needed

It just seems a lot of unsupported (or plain wrong) claims are made to justify the segwit softfork. And the most massive change by far is being slipped in as a minor softfork update?

So this is kind-of like a merge-mined witness chain?

I am pretty sure that the idea of segwit grew from this idea to ignore signatures in the txid calculation. This was proposed in BIP 140: https://github.com/bitcoin/bips/blob/master/bip-0140.mediawiki. It basically proposed a normalized txid which was the txid but ignoring the signatures. The other stuff in that BIP was because the author wanted it deployed as a soft fork rather than a hard fork. Otherwise a hard fork would be needed.

The segwit transactions are non-standard to old nodes. This means that old nodes/wallets ignore them until they are confirmed-- they don't show them in the wallet, they don't relay them, they don't mine them, so even confusion about unconfirmed transactions is avoided.

I'll happily say otherwise.  It's a change of somewhat more complexity than P2SH; certainly less than all combined. The implementation, however is smaller than the BIP101 implementation (comparing with tests removed). The Bitcoin community is getting better at documenting changes, so there is more documentation written about this than many prior ones.  Conceptually segwit's changes are very simple; based on signaling in the scriptPubkey, scriptsigs can be moved to the ends of transactions, where they are not included in the txid. An additional hashtree is added to the coinbase transaction to commit to the signatures. The new scriptsigs begin with a version byte that describes how the scripts are interperted, two kinds are defined now, the rest are treated as "return true".

This is what segwit effectively does, among other improvements. The first version of segwit that was created for elements alpha does _EXACTLY_ that, but there was no way to deploy that design in bitcoin because it would deeply break every piece of Bitcoin software ever written-- all nodes, all lite wallets, all thin clients, all hardware wallets, all web front ends, all block explorers, all pre-signed nlocked timed transactions, even many pieces of mining hardware; we learned about how impactful doing that was with elements alpha when it was very difficult getting existing software working with it... and for a while we didn't see any realistic way to deploy it short of rebooting the whole blockchain in a great big flag day (which would inevitably end up unintentionally confiscating some peoples' coins)-- not just a hard fork but an effective _rewrite_.  The clever part of segwit was reorganizing things a bit-- the signature field is still part of the txid but we don't use it for signatures anymore, we use a separate set of fields stapled onto the end to achieve exactly the same effect; but without blowing everything up.

That leads me to another question I've been having that hasn't been answered as far as I know. If segregating the signatures out of the tx leads to a stable txid (malleability fixed), then why can't we simply fix malleability independantly by simply ingoring the signatures when hashing the txid?

I agree with that, after all, there are 6 separate BIPs for it of which 4 (or 5?) are being implemented in one go.

Again, no. Full nodes don't have to validate the signatures of transactions in super old blocks. This is something that Bitcoin Core currently does and will continue to do. Since it doesn't check the signatures of transactions in historical blocks those witnesses don't need to be downloaded, although I suppose they could. There will of course be people who run nodes that store all of that data (I will probably be one of those people) in case someone wants it.

From the view point of a new node some years in the future, that data isn't needed and it most certainly won't be downloaded or needed. But for now, yes, you will be storing that data but it can be pruned away like the majority of the blockchain can be now. Keep in mind that a pruned node is still a full node. It still validates and relays every transaction and block it receives. Full Nodes do not have to store the entire blockchain, they just need to download it. That extra data is also not counted as what is officially defined as the blockchain.

Anymore? It was never done in the first place. Verifying the transaction has always been checking the signatures because the creating and verifying signatures involve the hash of the transaction.

It won't increase the CPU load because that is what is currently being done and has always been done. In fact, signature validation in Bitcoin Core 0.12+ is significantly faster than in previous versions due to the use of libsecp256k1 which dramatically increased the performance of the validation.

That is so wrong.

"If you're going to argue that you don't want a system where hashpower consensus allows new script rules for users to use to voluntarily contract with themselves, you should have left Bitcoin in 2010 or 2011"

I don't see jl777 sgbett arguing that. You want hashpower, consencus or not, to blind existing nodes. Introduce trust or obscurity. A hard fork in disguise.


"-- and I would call denying users the ability to choose whatever contract terms they want out of principle rather than considerations like resource usage both weird and harmful--"

What has jl777 sgbett done to harm Bitcoin?
Segwit however could destroy it. (unproven obviously. just opinion)

Something better will come in due course.
Segwit needs more thought.

Segwit needs to be hard forked.


edited - gmax does appear to be responding to sgbett. apologies for wrong accreditation. 

It saves space in that it does not require that the signatures be downloaded. Since clients that are syncing do not need verify the signatures of old transactions, the witness data is then not needed. This would mean that they do not need to download the witnesses and can thus save bandwidth and space. This is only for several years in the future after segwit has been used for a while.

Not every transaction needs that hash, just like how not every transaction now has its txid stored in the blockchain. Just like it is done today, the merkle root of the wtxids is stored, and that is it.

The witnesses are not permanent and can be deleted. After a few years, they won't even need to be downloaded.

Checking the transaction hash is not part of the validation of transactions. Rather the signature is validated with the witness data and checks that the signature is valid for that transaction. The block contains a merkle root. This merkle root is calculated by hashing all of the transaction hashes. The same is done for the witness root hash which is just the hash of all of the wtxids.

See above.

It is okay to ask questions, but when you spread around false information (like the title of this thread) then it is not okay. I admit, sometimes I do say false things, but I usually do say things with stuff like "I think" or "from what I understand" to indicate that what I say may not be the truth.

Kind of, I can't build one right now and I don't have a segnet node set up yet so I can't pull one from there. I will give you a similar example, pulled from the BIP. In this example there are two inputs, one from p2pk (currently used) and one from a p2wpkh (segwit but this format will probably not be used).

Take for example this transaction:

Here it is broken down:
An upgraded node would request and receive this transaction like this. It would contain all of the witness data.

But a non-upgraded node (and a node that doesn't want witness data) would only receive
which is considerably smaller than the witness serialization because it doesn't include the witness.

Those three bytes (and whatever other overhead because I am fairly sure there is other overhead in the witness data) are not part of the transaction that goes into the count of the size of a block.

Because Segwit adds more useful stuff and 2 Mb hard fork doesn't solve the O(n^2) hashing problem. The Classic 2 Mb hard fork had some weird hackish workaround for the O(n^2) hashing problem. In terms of lines of code, I believe that there was an analysis done on segwit and the classic hard fork that found that the amount of code lines added was roughly the same. This is because the hashing problem needed to be fixed with that hard fork.

I suppose you could also say that segwit is more "efficient". Segwit, with roughly the same amount of code as the classic hard fork, brings much more functionality to Bitcoin than a 2 Mb hard fork does.

I apologize for my technical ineptitudes.

I am trying to understand how segwit saves blockchain space as that is what it is being marketed as and with a softfork in the upcoming weeks.

So i look at the BIP, I find each segwit tx needs a commitment hash, plus more. this is 32 bytes per tx that to my ignorant simple minded thinking is in addition to what would be needed if it was a normal tx.

Now I am being told that we dont need to count the space used by the witness data. This confuses me. I like to count all permanently required data as the permanent space cost.

I apologize for asking questions like this, but I am just a simple C programmer trying to allocate space for the segwit and noticing it seems to take more space for every tx. I await to be properly learned about how the commitment hash is not actually needed to be stored anywhere and how that would still allow a full node to properly validate all the transactions.

Or did I misunderstand? Does segwit mean that it is no business of full nodes to verify all the transactions?

James

****
OK, it was made clear that the commitment hash is just for the wtxid calculation so it doesnt really exist anywhere. The cost is 2 bytes per tx and 1 byte per vin. Still it is increasing HDD space used permanently, which is what confused me as segwit was marketed as saving HDD space and helping with scaling

Oh, the amount of misinformation in this thread!!

jl777 you have misunderstood the segwit BIPs.

NO NO NO!!!!. AND PLEASE STOP SPREADING THIS!! IT IS WRONG.

The above with the OP_RETURN is how the witness root hash (the hash of the wtxids where the wtxids are leaves on a hash tree). This is added to the Coinbase transaction of the block. The point of doing this is to commit the witnesses to the blockchain without adding all of the witnesses to the size calculation of the blockchain except for these 38 bytes.

The wtxid (also called witness hash) is the hash of the transaction with all of the witness data and the transaction data. The transaction format if witness serialization is specified is 4 bytes for the transaction version (currently used), 1 byte for a marker (new, is always 0), 1 byte for a flag (new), the input count (currently used), the inputs (currently used), the output count (currently used), the outputs (currently used), the witnesses (new), and the locktime (currently used). All of this is what is hashed to get the wtxid. However, the regular txid is just the hash of everything that is currently used (so it contains none of the new stuff) so as to maintain backwards compatibility. If witness serialization is not specified then only the currently used stuff is sent in the tx message.

Likewise, with blocks, from what I understand, the block sent will contain the new transaction format if witness serialization is specified. Otherwise it will just include the transaction data that is currently used so that it maintains backwards compatibility. The only "bloat" caused by segwit is the 38 bytes in a second output of the Coinbase to commit the wtxids in a similar manner to the way that regular txids are committed and 2 extra bytes per transaction. Only upgraded nodes would get the witness data and those 2 extra bytes.

The addresses are different. An upgraded node will use p2sh addresses, which we currently use. Those p2sh addresses can be spent to using the current methods so non-upgraded users can still spend to those addresses. To spend from those addresses requires segwit, and the way that the scriptsig is set up, non-upgraded nodes will always validate those transactions as valid even if the witness (which those nodes cannot see) is invalid. Those segwit transactions still create outputs the regular way, so they can still send to p2pkh or p2pk outputs which non-upgraded users can still receive and spend from. Segwit transactions are considered by old nodes as transactions which spent an anyonecanspend output and thus are treated with a grain of salt. The best course of action is to of course wait for confirmations as we already should still be doing now.

Again, NO. See above.

---

---

Ah *sigh of relief*; here comes somebody who actually knows what they are talking about.

Could you also let me know if I presented any misinformation? I have been trying my best to not and to make jl777 understand why he is wrong but I may have accidentally (either due to misunderstanding the BIPs or just really bad typing) given him false information.

Since Pieter and no one on the IRC reponded either, I will ask this again. Will there be a full write up (preferably before segwit's release) of all of the changes that segwit entails so that wallet developers can get working on implementing segwit? AFAIK the segwit implementation contains omissions and changes from what was specified in the BIPs.

Wow. The deceptive misinformation in this thread is really astonishing.

Contrary to the claims here, segwit doesn't increase transaction sizes (as was noted, it adds a single coinbase commitment per block).


jl777, to be blunt, and offer some unsolicited advice: You have almost no chance of actually writing that bitcoin full node you say you want to be working on when you are so unwilling to spend more than a second reading or take any time at all to understand how existing Bitcoin software works.  Virtually every post of yours contains one or another fundamental misunderstanding of the existing system/software-- and your abrasive an accusatory approach leave other people disinterested in spending their time educating you. Even here, I am not responding to for your benefit-- as I would otherwise-- but because other people are repeating the misinformation you've unintentionally generated due to your ignorance. Please take a step back: Bitcoin is not "bitcoin dark", "nxt", or the other altcoins you've worked on in the past where an abusive/armwaving style that leans heavily on native intelligence while eschewing study will itself establish you as an expert. Bitcoin is full of really remarkably intelligent people, so simply being smarter than average doesn't make you a shining star as it may in some places.

The text you are quoting is instructions on computing a hash. None of the data involved in it is stored, not any more than the tens of times the transaction size of data used for the sighashes on a large transaction is stored.

If the carefully constructed, peer reviewed specifications are not to your liking; you could also spend some time studying the public segnet testnet. Given that there are both specifications and a running public network, the continued inquisitory "needs to be answered" conspiracy theory nonsense-- even after being given a _direct_ and specific answer ("segwit adds one coinbase output that contains the root hash of the Merkle tree that commits to the witness transaction ids. It uses 47 extra bytes per block")-- is highly inappropriate. Please do not subject other contributors to this forum to that kind of hostility.  

The content of your scriptpubkey, beyond the resource costs to the network, is a private contract between the sender of the funds and the receiver of the funds. It is only the business of these parties, no one else. Ideally, it would not be subject to "consensus", in any way/shape/form-- it is a _private matter_. It is not any of your business how I spend my Bitcoins; but unfortunately, script enhancing softforks do require consensus of at least the network hashpower.

Bitcoin Script was specifically designed because how the users contract with it isn't the network's business-- though it has limitations. And, fundamentally, even with those limitations it already, at least theoretically, impossible to prevent users from contracting however they want. For example, Bitcoin has no Sudoku implementation in Script, and yet I can pay someone conditionally on them solving one (or any other arbitrary program).

Bitcoin originally had an OP_VER to enable versioned script upgrades. Unfortunately, the design of this opcode was deeply flawed-- it allowed any user of the network, at their unannounced whim, to hardfork the network between different released versions of Bitcoin.  Bitcoin's creator, removed it and in its place put in facilities for softforks. Softforks have been used many times to compatibly extend the system-- first by Bitcoin's creator, and later by the community. The segwit script versioning brings back OP_VER but with a design that isn't broken---- it makes it faster and safer to design and deploy smart contracting/script improvements (for example, a recently proposed one will reduce transaction sizes by ~30% with effectively no costs once deployed); but doesn't change the level of network consensus required to deploy softforks; only perhaps the ease of achieving the required consensus because the resulting improvements are safer.

If you're going to argue that you don't want a system where hashpower consensus allows new script rules for users to use to voluntarily contract with themselves, you should have left Bitcoin in 2010 or 2011 (though it's unclear how any blockchain cryptocurrency could _prevent_ this from happening).  Your views, if not just based on simple misunderstandings, are totally disjoint with how Bitcoin works. I don't begrudge you the freedom to want weird or even harmful things-- and I would call denying users the ability to choose whatever contract terms they want out of principle rather than considerations like resource usage both weird and harmful--, but Bitcoin isn't the place for them, and the restrictions you're asking for appear to be deeply disjoint with Bitcoin's day-one and every-day-since design, which has a huge amount of complexity in the original design for user (not consensus) determined smart contracting and where softforks (hashpower consensus) have been frequently used to extend the system.

Great posts. I think your smart enough to understand segwit. If anyone can.

In "a few weeks" Bitcoin will be fundamentally changed by segwit. (soft forked by core, Bitcoin guardians)

I agree with your earlier comment. Segwit must be abandoned postponed.

2mb blocks first, soon, then reassess segwit. At least hard fork.
(core could do this, 2mb blocks are road mapped in core?)

Segwit is not my bitcoin. Not at this point in time at least.
"a much more complicated error prone system with potential attack vectors that is not peer reviewed"