Pages:
Author

Topic: Scalability (Read 26227 times)

newbie
Activity: 1
Merit: 0
April 21, 2011, 08:21:02 PM
#51
This has probably been discussed allready but it seem to me that the scalability and performance problems may be relatively easy to solve.

Consider a peer to peer file sharing app. A file is split into a collection of blocks. Each peer downloads the blocks they need in order of priority. Even if none of the peers have all the blocks (original source has gone offline) it is still possible for the peers to piece together their collective copy. I suppose the point is that no one bitcoin node needs to have all the blocks and transactions at any one time for the collective to hold a secure and trused copy between them. So a bitcoin client could have a set limit on the amount of resources that it will use, based on the hardware availiable and the users preference. An indiviual user can hold the information of highest priority to themself and use the remaining resources for other randomly chosen blocks. I suppose this is where transaction fees come in, encouraging users to commit more resources by rewarding them financialy for processing certain transactions of high priority.

I hope that made some sense. I think that is how it is supposed to work, but that comes from reading beteen the lines, so I may be way off.
sr. member
Activity: 252
Merit: 250
September 17, 2010, 09:38:57 AM
#50
The design outlines a lightweight client that does not need the full block chain.  In the design PDF it's called Simplified Payment Verification.  The lightweight client can send and receive transactions, it just can't generate blocks.
Is it possible to generate blocks if a lightweight generator  ask many nodes for hash of previous blockchain and transactions in current block?
sr. member
Activity: 252
Merit: 250
September 17, 2010, 09:33:08 AM
#49
I agree, I've been looking into packaging "blocks" for download so that you can install the client, unzip a fresh stack of blocks and let the client finish off the rest to get people up and running faster. I'm experimenting with Windows and Linux clients (don't have a Mac that can run the Mac client, sorry folks)

Hopefully that will help with the issue.

Of course, the trust factor comes in (should I download these from this guy instead of letting the network do it for me)

 Wink
Well, I don't see problems with trust if properly implemented.

The whole idea that downloading zipped blocks is faster than downloading "from network" means performance problems in BitCoin client.

I know that long time ago bitcoin client utilized very little of available bandwidth because of cache flushing after each single block or something like that. But Satoshi fixed that. Did someone test how much bandwidth (of available bandwidth) is used and how much data is downloaded when a new chain is downloaded?

If bandwidth is underutilized - then networking code must be rewritten. If zipping helps to compress chain (I'm not sure because chain contains a lot of hashes and those are random and thus incompressible) - then compression must be built in.  If chain is downloaded many times - then trust scheme must be changed to Emule AICH or similar:
---
Newer versions of eMule support AICH - Advanced Intelligent Corruption Handling. It is meant to make eMule's corruption handling competitive with BitTorrent. SHA-1 hashes are computed for each 180kb sub-chunk and a whole SHA-1 hash tree is formed. AICH is processed purely with peer-to-peer source exchanges. eMule requires 10 agreeing peers regarding the SHA-1 hash, so rare files generally do not benefit from AICH.
---

So instead of asking each peer for blocks, Bitcoin client can ask for block hashes and even for hashes of block ranges (to save hash traffic) to download each block only once.

Also, transaction clearinghouse function can be easily be built in the current network. We can have a "trust network to validate transactions" setting.

Also, blockchain is not needed to generate blocks, so a purely generating node can ask other nodes for hashes needed to build its block instead of asking for chain. And people will be able to contribute either bandwidth or CPU or both. Now they are forced to contribute bandwidth.

I am for example in a remote village in Nigeria connecting over EDGE. 30 MB of chain costs me 9 NGN that is 0.06 USD. But with terabytes of traffic that will not be affordable. Even now 30 MB is not affordable because a lot of people earn less than $200 a month and 0.06 USD is the cheapest possible wholesale price because I pay $80 a month for a 3 GB cap at 120 kbps down / 80 up I measured.

Debit/credit cards industry is not developed here, but everyone has a cheap fake Chinese phone with J2ME and thus capable to run some client software.
hero member
Activity: 574
Merit: 513
September 11, 2010, 03:22:48 AM
#48
Has there been any progress or further discussion on establishing better scalability for future of Bitcoin?
newbie
Activity: 12
Merit: 0
August 15, 2010, 02:29:05 PM
#47
I think then now would be a good point to work out a separate "server" and "client" for the next release.

The design outlines a lightweight client that does not need the full block chain.  In the design PDF it's called Simplified Payment Verification.  The lightweight client can send and receive transactions, it just can't generate blocks.  It does not need to trust a node to verify payments, it can still verify them itself.

The lightweight client is not implemented yet, but the plan is to implement it when it's needed.  For now, everyone just runs a full network node.

When these "client" or "lightwight client" will be released? Or something like sample code?
It would be interesting to implement it on, say, java smartcard environment.

lfm
full member
Activity: 196
Merit: 104
August 15, 2010, 10:01:18 AM
#46
I guess it's not quite as bad as I initially thought, but it's still enough to make it impractical for many people to run their own bitcoin clients in a world where bitcoin is popular. The likely scenario (assuming no change to how Bitcoin works) is that most people use big payment providers instead and those providers use Bitcoin between each other.

I suppose this might be a good reason to make www.mybitcoin.com or an equivalent your primary bitcoin account/wallet. All you need for it is a browser.

member
Activity: 70
Merit: 11
July 24, 2010, 03:34:19 PM
#45
What's funny?

A server lying about whether or not your transactions are valid would be like your ISP lying about whether or not your HTTP requests are valid or not.

If they lie, you'll very quickly find another service provider (or download a Bitcoin iPhone app that doesn't suck and say that your transactions are invalid).

If you are a merchant and the server tells you that your transaction with a certain client is valid when in fact it is not, then the goods or services you provide, you provided for free. Now suppose this person buys a Ferrari... You're screwed. And that's, in fact, one of the well known types of attacks: an attack on trust. You will of course find another service provider, but the damage will already be done.

If you are a merchant, you will be your own server, no? Now if your own server gets hacked somehow with a dummy placed in between, this could be a problem...
newbie
Activity: 37
Merit: 0
July 24, 2010, 05:49:26 AM
#44
What's funny?

A server lying about whether or not your transactions are valid would be like your ISP lying about whether or not your HTTP requests are valid or not.

If they lie, you'll very quickly find another service provider (or download a Bitcoin iPhone app that doesn't suck and say that your transactions are invalid).

If you are a merchant and the server tells you that your transaction with a certain client is valid when in fact it is not, then the goods or services you provide, you provided for free. Now suppose this person buys a Ferrari... You're screwed. And that's, in fact, one of the well known types of attacks: an attack on trust. You will of course find another service provider, but the damage will already be done.
Red
full member
Activity: 210
Merit: 115
July 23, 2010, 07:31:37 PM
#43
What's funny?

I'm in agreement on your points. However you have to admit that what you wrote at the end sounded funny!

Gee, why would an anonymous server lie about my money?  :-)

In reality you are correct. There are very few things to lie about. I could think of a few things though.
legendary
Activity: 1652
Merit: 2311
Chief Scientist
July 23, 2010, 07:10:31 PM
#42
What's funny?

A server lying about whether or not your transactions are valid would be like your ISP lying about whether or not your HTTP requests are valid or not.

If they lie, you'll very quickly find another service provider (or download a Bitcoin iPhone app that doesn't suck and say that your transactions are invalid).
Red
full member
Activity: 210
Merit: 115
July 23, 2010, 06:33:38 PM
#41
Well, you do have to trust that the server doesn't lie about whether your transactions are valid or not, but why would the server lie about that?
lMAO!!!
full member
Activity: 158
Merit: 100
July 23, 2010, 10:57:45 AM
#40
I anticipate there will never be more than 100K nodes, probably less.  It will reach an equilibrium where it's not worth it for more nodes to join in.  The rest will be lightweight clients, which could be millions.

At equilibrium size, many nodes will be server farms with one or two network nodes that feed the rest of the farm over a LAN.

Could you, or anyone else, speculate about the numbers of transaction per second limit for the current system?

What could be the maximum transaction throughput (in the number of transactions, not in the volume)? What could it depend on?

What is the future limits on that number, in the case the system will grow?

Just consider Bitcoin as a system for micro- or even nano-payments?
Like paying 0.000001 (or less) for posting in a forum? Or paying even less for just reading the forum (with advertising stripped)?
That would generate far more than a million transactions per day.
That would create unprecedented transaction flow.
Will it hurt the other participants, which are not nano-payers?

We could test the "system" speed if we start sending small amounts in a circle between wallets and thus flooding the system.
Or we couldn't?
If we do that, what effect will it cause?

Would you mind against such an experiment? If you veto that, then how could you stop it?
full member
Activity: 210
Merit: 105
July 14, 2010, 10:47:24 PM
#39
It's good the spec already addresses this problem. I'd like to see some more in depth access to things like the coin list (so we can choose to spend coins based upon who sent them to us, etc) and the client list. For example, all of my computers at home -connect to my router, which runs (but does not generate) with port 8333 open.

I wish there were some way to prioritize network traffic to my own local nodes over remote nodes.
founder
Activity: 364
Merit: 7423
July 14, 2010, 09:10:52 PM
#38
The design outlines a lightweight client that does not need the full block chain.  In the design PDF it's called Simplified Payment Verification.  The lightweight client can send and receive transactions, it just can't generate blocks.  It does not need to trust a node to verify payments, it can still verify them itself.

The lightweight client is not implemented yet, but the plan is to implement it when it's needed.  For now, everyone just runs a full network node.

I anticipate there will never be more than 100K nodes, probably less.  It will reach an equilibrium where it's not worth it for more nodes to join in.  The rest will be lightweight clients, which could be millions.

At equilibrium size, many nodes will be server farms with one or two network nodes that feed the rest of the farm over a LAN.
member
Activity: 77
Merit: 10
July 14, 2010, 07:23:50 PM
#37
I second the DHT idea for maintaining a client list - we can't have millions of people relying upon an IRC channel, etc.  As far as the scaling issue goes, the issue is not at all HDD space, its network bandwidth.  Everyone is forgetting, its not bytes_per_transaction*transactions, which is the number everyone is using.  That number, as everyone has said, is fully manageable.  No, the number we're interested in is bytes_per_transaction * transactions * number_of_clients * total_hops_beyond_first_between_all_clients_combined

THIS is the amount of bandwidth which the protocol for BTC consumes as the network scales.  We're not just talking about sending one copy of each transaction to each client - we're talking about multiple clients broadcasting potentially redundant data to one another, and doing it across numerous hops, meaning numerous rebroadcasts.  Much larger number, much more difficult to handle.  However, it is manageable, just not in the current incarnation of network handling in the client.

Perhaps in the "popular" phase, BTC chains could be broken up by region, similar to the purviews of domain name authorities now - and there could be an alternative protocol for transactions across these regional boundaries?  This would help the raw numbers of the problem, and also cut down on latency and related issues.  Not that I think this is an excellent solution - but P2P flooding across all active clients is obviously out barring some massive breakthrough in quantum computing or whatnot.
hero member
Activity: 532
Merit: 505
July 14, 2010, 05:52:30 PM
#36
And I expect most of us will be running lightweight clients that just keep our wallets, sign transactions, and send and receive transactions to the ultra-fast nodes that ARE looking at every transaction.

Is this possible? What would this look like? From a technical perspective what does a "lightweight client" look like for you? My understanding is that the Bitcoin client needs the entire block chain in order to establish trust.
I'm imagining:
....

you don't even have to imagine , actually it's already possible to "remote-control" the node, so just create ur own little lightweight-client, that just sends and gets some info to/from your (highspeed-connected, hdd-packed) homeserver.

some kind of "managed server-client-version" already exists in MyBitcoin, you could run something like that on your own webhost and connect to it from wherever u are on whatever connection-speed.

the "lightweight client" doesnt have to be one, but connect to it and tell it what todo.
we can do that with JSON.
newbie
Activity: 8
Merit: 0
July 14, 2010, 05:44:12 PM
#35
Quote
spaceshaker: yes I agree there will have to be nodes that act as proxies for mobile devices.

Um. That's exactly what a supernode server would do.

Um. Sure. I think I've gone full circle. I think Gavin said it best:
A lightweight client would have a wallet with coins in it (public+private key pairs).

And a secure way of sending messages to, and getting messages from, any of the ultra-fast, always-connected heavyweight nodes.

The lightweight client sends money by:
  creating a transaction (signing coins with the private key)
  sending the signed transaction securely to the ultra-fast server, which puts it on the network.
  receiving confirmation that the transaction was valid and sent, and updating its wallet (marks coins as spent)
   (or getting a "you already spent those coins" error from the server)

The lightweight client receives money by:
  Either polling the server every once in a while, asking "Any payments to these BC addresses that I have in my wallet?"
   ... or asking the server to tell it whenever it sees a transaction to a list of BC addresses (or maybe when it sees
    a relevant transaction with N confirmations)
  When transactions occur, the lightweight client updates its wallet (adds the coins).

You don't have to trust the server; it never has your private keys.

Well, you do have to trust that the server doesn't lie about whether your transactions are valid or not, but why would the server lie about that?

In this scenario, the Bitcoin client could remain largely the same as it is today, although the focus would be that it is used on the "super-nodes" or "transaction servers" or "proxy servers" (these systems would probably serve all three roles) or by anyone wishing to play in that game. If the Bitcoin client was augmented to use DHT then that may be improvement but there is still a need for a "lightweight client" as Gavin described above. It seem's Gavin's "lightweight client" concept obviates my scalability concerns somewhat.
sr. member
Activity: 308
Merit: 258
July 14, 2010, 05:12:22 PM
#34
I've had luck with testing of packaging blocks into a zip file and dumping them on a new Bit Coin to speed up the block download process. I'll leave the link in my signature and if anyone finds it useful or time saving, feel free to donate.  Wink
member
Activity: 60
Merit: 10
July 14, 2010, 04:42:16 PM
#33
Quote
Why? E-Mail and Jabber work the same way. Everyone can run their own server, and many do, but most users prefer to use someone else's server(s).
Sure it can work that way but is that the ideal? Doesn't that make the network less robust and more vulnerable to attacks and manipulation?

Not really. If you're worried, you can always run your very own server.

Quote
What happens if some attackers start running a cluster of supernodes?

Nothing. Unless you happen to use one of those supernodes, which you don't have to, because you can run your own supernode. Or use a trusted one, much like people trust Google with their E-Mail.

Quote
The main point is why rely on this more vulnerable architecture when you don't have to? It isn't easier to implement.

It isn't easier to implement? I'd like to see some proof here.

Counterexample: The totally distributed E-Mail system developed at Rice University is a hell of a lot more complex than running a (network of) semi-centralized mail server(s).

Quote
spaceshaker: yes I agree there will have to be nodes that act as proxies for mobile devices.

Um. That's exactly what a supernode server would do.
sr. member
Activity: 308
Merit: 258
July 14, 2010, 04:34:44 PM
#32
Having to download the entire chain also causes issues with new user acceptance.  I installed bitcoin on an older box I have and it took about 4 hours to download the entire chain.  As time goes on this is only going to get worse.  While this is a minor annoyance for most people here with fast connections and servers, if bitcoin were to actually get wider acceptance outside the tech circle it would be a deal breaker.

Say your neighbor wanted to buy something with bitcoins.  You tell him to install the program, then go to the exchange and buy some coins.  Sounds great, right?  How do you think he will feel about the transaction when he has to wait 8 hours after installing the program to get his bitcoins because he has to download a chain with 200,000 blocks?  Do you think he would recommend it to his friends?  To gain more widespread acceptance, initial transactions would need to overcome this factor.
I agree, I've been looking into packaging "blocks" for download so that you can install the client, unzip a fresh stack of blocks and let the client finish off the rest to get people up and running faster. I'm experimenting with Windows and Linux clients (don't have a Mac that can run the Mac client, sorry folks)

Hopefully that will help with the issue.

Of course, the trust factor comes in (should I download these from this guy instead of letting the network do it for me)

 Wink
Pages:
Jump to: