Topic: The swarm client proposal - Reminder: 15 BTC pledged so far, now worth 3255$! - page 4. (Read 12127 times)

Keep in mind too that Bitcoin doesn't have to have micropayments on the P2P network for it to serve its fundamental purposes of being a currency.  The P2P network doesn't have to replace Visa to be a huge success, it merely need to replace the central bank as the central issuer of a currency.

What do today's banks do now?  They settle transactions quietly amongst themselves - and use the Federal Reserve as the backbone for wires and lump settlements between banks.  Replace the dollar with the Bitcoin, and the only thing that is different is the backing (no Federal Reserve) - banks still do what they do, the way they do it now.  If banks can switch from dealing in dollars to euros or drachmas, Bitcoins are just another unit of account they could accommodate if they felt inclined.  Grown big enough, and with enough controls to keep the banks from running with your money, space and bandwidth on the Bitcoin block chain may become scarce enough that it is prohibitive for all but the biggest transactions (i.e. settlements and wires), the same way Fedwire (bank wires) only makes sense for big payments like buying houses and cars, not for buying snacks and sodas, with the fees setting the bar for what kind of transaction makes sense to put through as a fed wire.  There is nothing wrong with this.

Being good enough only to be the backbone for a world currency - with no ability to settle micropayments or even everyday transactions - would still be a huge success story for Bitcoin.

A distinction ought be made between the average and marginal costs of a transaction. It certainly can make sense to have transactions worth less than the average cost to process if the marginal cost is low enough.

Not sure where you got that from.

Here's an example:

May 9, 2011 - Gavin Andresen:


 - http://gigaom.com/2011/05/09/can-flattr-plus-twitter-make-micropayments-a-reality/#comment-622793

How can I relax at a time like this? Horrifying realization and all. Horrifying.

This is very much an old SCCS vs. RCS argument.

SCCS is forward-delta storage (started in 1972).
RCS is reverse-delta storage (started in 1982).

Currently Bitcoin is forward-delta. I'm going to guess that the optimal storage for Bitcoin will be some combination of both approaches. But I'm positive that the reference implementation will have to be started afresh to achieve any meaningfull progress.

Edit: added a quote from a thread that is now locked, but shouldn't be.

Yes, yes panic mode set to off I get it.

... I will however save an "I told you so" for the eventuality of further scaling issues than a penny/tx.

1-2 years.

I could see natural growth like that happening and you won't get that much fancy future-ware in that time.

Relax dude I believe in BTC as well, but many still consider it an experiment.

Anyway my point was simply that arguing against new things because they are new is a null argument.

Well spotted/thought.

I see two solutions:
1. Clients looking at the same address origin to the invalid transaction would instantly be able to see the error. The rest of the network could choose to just trust a multitude of reports (no harm in throwing blocks away).

2. As you said someone could download the entire blockchain and do a check not based solely on "voting with error reports".

Likely what would happen would be 1., there are few risks to dismissing a widely error reported block, but lots of mining power wasted working on a bad block.

If rampant false reporting happened (say to win a "mining war") the originator of the false reports would eventually have his ips blocked.

There is less rush here as you can do that retroactively and then never worry about the blocked folks again. You could take your sweet time downloading everything and verifying everything before judging.

Thanks, kinda thought I would have to just leave this topic alone with no support.

I arrived at this idea after bouncing ideas off TangibleCryptography in the SD thread so he gets some credit for re-explaining block verification to me.

A x-year-ledger/snapshot solution would also make it easier to fully verify error reports as it eases downloading the entire chain.

How could they do that check? You would have to trace a transaction back far enough in the chain to be trustworthy. For a transaction chain that is used by a major service like MtGox, a single false report could force the entire network to download and backtrace hundreds of thousands of transactions.

Admittedly, this works great in the case where everyone is honest, and it still works better than what we currently have when most people are honest, so I'll give you that. Combined with a merge-mined ledger, Bitcoin would be able to scale up even further - possibly infinitely. Nice work.

I think partial verification makes a lot of sense, but you are being ridiculous and counterproductive.

Bitcoin is more tested than the vast vast majority of software in existance. Thousands of people trust it for thousands of dollars of value on a continual basis with nearly no problems except bad user error.

A super node requires nothing to implement, I'm already running one.

Those are some good numbers I'll give you that.

I still say we should save that 1m$/day.

I do not think cost of super nodes will be an issue at "VISA" transaction volumes.

4000 transactions per second at 0.0005BTC fee each = $1m per day in fees.

So a good few exchanges, merchants, mining pools will happily be paying $200k per year to run a super node or 10.

Yes debate is evil.

One of the selling points of BTC is micro transactions, but with penny fees per tx (and more in the future), that at least disappears.


Hugely complex
So is BTC itself; lets abandon it for simple fiat.
Super nodes are also hugely complex.

untested
Well yeah.. so is BTC mostly.

will require significant amount of resources to implement
Less than super nodes.

Swarm nodes would run just fine on regular users computers. A giant long term saving.

and solves a "problem" which doesn't exist.
I am a software engineer and I see a problem.

The official devs (Gavin at least) also seem quite open to some kind of update of this nature.

My solution would not fork the chain and would be fully operational with 2 or more clients.

1. Gavin sees the problem.
2. I am a software engineer already working on an open source project that should help the world a bunch.
3. If everyone has been too incompetent/lazy to fix this in a year or two I WILL do it myself.
4. I was however hoping others more trained would do this so I could focus on developing my idea for a BTC credit card or perhaps stock exchange.
5. I have pledged 15 btc, more than I usually give in gifts to my own family, not just "talk", not much, but I'm poor-ish.

So yeah, anything else?

Well if you feel paying a penny to send up to millions of dollars of value anonymously, securely, irreversibly and nearly instantly "too much" well there really is no point in further discussion.  Still the nice thing about Bitcoin is even if you are an illogical cheap ass you can be just don't expect first class service for free.


Hugely complex, untested, will require significant amount of resources to implement and solves a "problem" which doesn't exist.

Still Bitcoin is open source so implement it and fork the block chain and if enough people follow you then it will happen.   Still I get the impression it isn't your intention to actually code this but rather convince other people to code it for you when they neither see the value nor the problem it solves.  Right?

I don't really think there is a problem now, but rather there MAY be one some years from now.

The users already give a VAST subsidy to miners, donate a decent chunk of their computing power and tx volume is by all standards trivial - so why do they have to pay extra to compete for quick transaction already now?

Considering sending BTC is a few electrons in a wire 1 penny really is too much - especially when relatively simple solutions exist (not just my idea here).


What are the counter arguments to an update?

You're the man. Fixed it.

I doubt they are that sinister. This is just a somewhat daunting update.

Additionally it likely wouldn't work as someone at some point would just make an S-node client and it would slowly overtake the network.

Until then your BTC is as safe as your password as always.

1. No, but if the network is groaning now its not a good sign.
2. Once you go from over-counter standard hardware to even simple super nodes, you need a geek to put it together and run it. Geeks have a high pay check so that is either a lot of money/time you ask others to forego or fees you will pay.

(20$/hour 5 hours/day 5 days/week 4 weeks/month 12m/y = 24.000$ then add other costs - it will be pricey)

The technical complexity of putting such a node together is likely equal to making an update of SOME kind anyway.


Its a non-problem now, it won't be forever.

You may have discovered the "exit strategy" of the "core development team" and other current Bitcoin luminaries.

Edit: I assume "levee" was a misspell of "levy".

Read that like 3 times now.

It mentions pruning and that super nodes may scale - which I mentioned.

However I don't think that is enough.


Fitting you should mention those:
1. They use server farms to do that, huge ones.
2. They do load balancing, not everyone gets their youtube stream from one server.

The data of BTC is NOT the issue, the bad comm protocol IS.

To run a VISA network - according to the wiki - you would need a few racks of machines and an optic fiber internet connection to run a node.

The end points of said connection would also have to be reasonably fast.

This would all be very expensive, minimum 20.000 - 200.000$ to construct/maintain for just a year. (engineer time + pricy computers + beefed connection/subscription).

How many of such pricy nodes would we really end up with and what would the fee levels become?

Even a thousand not is not that believable and how vulnerable does that make us?

I sincerely doubt that Satoshi thought that:
1. O(n) was good design (you don't if you do software).
2. You should scale hardware rather than slightly fix issues.
3. That BTC should have ~1000 vulnerable/expensive nodes when ONE voluntary update could give us a million cheaper ones.


Even if the US does nothing about the "drug/terrorist/gun-money" that is bitcoin for their friends the banks (lol, they will) such super nodes would be in a position to levy very high fees.

We would be back to VISA level prices.

You again bring up an interesting point with torrents:
1. Does torrents have such things as super nodes? - NO, period.
2. Does any rich benevolent (magic) "hobbyists" successfully operate super nodes? - NO, period.

Traffic hiding is easy to add, TOR does it.

Our program has a huge flaw... OKAY: SCALE THE HARDWARE!

S-nodes would help adoption and give everyone lower fees. Complexity is low to medium.

Yes, it may take half a year for two programmers to do it, but it would pave the way for the future of bitcoin and allow 1000 fold increase in usage.

Just building ONE of your super nodes would like take more TIME and MONEY than doing this update.


Anyway unless someone can find a flaw in my proposal, my work is done. Use it when your mining pools and various clients crash over satoshidice 2.0 - you get to listen to all the media laughing at bitcoin in the meantime.

I think you should read this first, if you didn't already:

  https://en.bitcoin.it/wiki/Scalability

Many software projects that at first seem impossible to scale actually later turn out to be feasible. Examples: Facebook photo storage, YouTube, etc. Even at VISA levels of traffic it would be quite feasible for small companies or rich hobbyists to run full verifying nodes with todays technology, and tomorrows technology will be a lot better. There can easily be tens of thousands of such nodes and perhaps that's all you need, Satoshi certainly thought so before he left.

The idea that a supernode structure makes Bitcoin vulnerable to being "shut down by the US Government" is not realistic. It's not like BitTorrent has gone away, right? Bitcoin doesn't try and hide its traffic, so if a government wanted to make running nodes illegal they could easily do so regardless of how many nodes there are.

There is no need for complicated changes to the protocol. Just some boring and difficult work to scale the implementations.

[UPDATED 10-04-2013]


In response to this thread:
https://bitcointalksearch.org/topic/huge-increase-in-satoshidice-spam-over-the-past-day-87444 (satoshidice spam)

Problem:
The bitcoin network basically has what is called O(n) algorithmic performance.

In short this means that if the network grows 100 times, you need pretty much 100 times more powerful nodes and 100 times more bandwidth for EACH node.

While this is possible from a technical perspective it would quickly make the entire BTC network dependent on a few exposed super nodes.

Such nodes would very likely extort huge fees and/or be shut down by the US government.

Pruning or ledger blocks solves the HDD issues, but does nothing to spread out CPU, RAM or bandwidth loads.

Solution:
Future clients should be swarm clients that do the roles of full nodes, but only on parts of the chain.

The simplest explanation I have is this:
Imagine "cutting the blockchain in two halves". Now you run two clients on two computers that look at each their half of the chain or rather each their half of the addresses in the blockchain.
Now if one client sees a block spending an output already spent it reports it to the other client and they both reject the block. To prove such claims the reporter includes the TX and its minified merkle-branch up to the block header it occurred in - so the reporter cannot fake a report.
To the outsider such two clients would act and behave in every way as a full node - imagine chopping up the blockchain further and you have a system of swarm nodes!

I have already pledged 15 BTC to the implementer of this - mark me a scammer if I don't pay up please. Some escrow solution is also possible.
I will also accept less elegant solutions that split up the blockchain just 5 times or more.

Feel free to pledge along with me, it will be worth some of your BTC to secure the value of the rest I think

Nature of my solution
+ No fork.
+ No new blockchain standards.
- New communication protocol.
+ Bridge can be made between old and new clients.

Detailed solution:
Abbreviations:
F-node: Full node.
Could choose to mine independently of a pool, does verification on the whole blockchain and so on.

S-node: Swarm node.
In a swarm acts the same as a network of full nodes, but only does verification on parts of the chain.

L-node: Light node.
Is dependent on super nodes.

B-Node: Bridge node.
Bridges two ways of sharing blocks by "translating" the format of one to the other and broadcasting it.
Only one such node is theoretically needed to prevent forking.

Branches/trees:
See Merkle trees in wikipedia - the existing way Bitcoin stores txs.
The important thing here is that S-nodes send block fragments in the form of merkle trees tied to the block header whereas todays clients send whole blocks.

Subscriber:
A client which has a standing request for updates concerning specific addresses.

W-addrs: Watched addresses.
The Bitcoin addresses a swarm client is watching and storing the full history for. To keep itself updated it will subscribe to information about these addresses with other S-nodes.
S-nodes will relay information and TXs to each other even if

S-node group:
Say half the S-nodes watch addrs ending with an equal hex number and the other half unequal - each half would then be a "group". Members of the same group watch the same addresses, can then more easily share data and will have to connect to fewer relay nodes.

NEW IMPORTANT - Addresses:
The blockchain does in fact not store BTC in addresses, this is an abstraction, but rather allows movement of BTC if a script evaluates to true. With normal Bitcoin addresses this script looks something like this:
    1 2 3
However the script could also require multiple signatures or go by IP.
Hence S-nodes will not actually organize by addresses, but by the scripts - a redeem script will always look the same for the same address and/or the same conditions, hence it can be used as an identifier.
In most normal cases this will mean S-nodes basically use addresses as expected, but a multisig or IP script will just be handled as another address would: it has a balance or it doesn't.
If we imagine the hash or just first 50 bits of a script are between 0-1 then one S-node group could be 0-0.5 and another 0.5-1 and it would not matter at all what kinds of scripts were thought up or used.
would be all an S-node saw.

Miners forming a new block entirely with S-node system:
1. Miners in a pool will split themselves up into groups by the addresses their S-clients watch.
2. Each group will handle TXs that send funds from their W-addrs. (If there are multiple input addrs in the TX the first input will determine which S-node group puts the TX in the block, but each relevant group validates their W-addrs inputs.)
2A. Bad TXs inserted by dishonest miners will be rejected/reported by the other miners of the same S-node group.
3. From this they construct branches of the block and transmit these.
4. A pool master collects branch headers and makes a header hash which can be mined - no super node required to validate the whole thing.
5. Header hash is broadcasted and solved by the miner clients.
6. Bridge node gathers all the block strands into one traditional block.
7. Bridge node broadcasts the block to the older clients (or vice versa).

Clients with S-node system
1. Single small block branch is received from same S-node group.
2. Client checks all transactions it can and block header is valid.
3. The miners who solved the block send the sections to the various S-node groups on a on-demand basis branch by branch.

Security:
Bad TX:
If in step 2 a transaction is invalid due to either a hash error OR insufficient balance this is reported to the network.

1. Report is trustworthy because everyone can check that chain part themselves or that hash operation themselves.
2. The IP that sends the invalid branch/block is automatically added to a client IP-filter.
3. If an balance error report is received that turns out to be false, then that IP is blocked automatically.

If your connection is hijacked:
You don't need a web of trust or anything like that, just PKC; each client randomly selects a PK and publishes it.
Each client would then have its own list of Pub.Ks or "client addresses". For instance I might add MtGox, my brother and some others.
This doesn't mean I trust these guys it just means I find it near impossible that they would ALL know what the attacker wanted left out AND either have been hacked or colluding with him.
If I cannot connect to them because the attacker cannot fake them without their Pri.Ks I will know my connection has been hijacked.

Notes
All of this means that no S-client has to process, receive nor store the full blockchain. Or: O(log(n)) -> network grows 100 times you only need say 2 as powerful nodes.

Full trust-less security is maintained.

Even in the case of reports you can either trust the multitude of clients watching a certain address balance or reconstruct the entire chain from branches around the network.

Implementation:
It is my understanding that today the client has certain "get block" methods.

While the block standards would remain the same the S clients would basically ignore such calls and replace them with at least:
1. Reporting.
2. Get branch.
3. Subscribe to address/group updates.

By having S-node groups sharing information with themselves we can continue operating with just a couple of peers (many peers overload routers and stuff).

Using a bridge no one would even need know you had S-nodes behind it.

Am I missing anything?

EDIT: Changed title to be more informing.
EDIT, Update 10-04-13: Clarified some things and improved solution with my now clearer understanding of the Bitcoin protocol.