The ping rate seems to be the indicator of a slow synchronisation. For a couple of days I've been disconnecting peers with over 1000ms pings, and the sync' seems to be flying through. A 30Kb/s connection over public WiFi seems to be reasonable for running a full node on either Windows or Ubuntu.
In fact, this morning I ran two synchronisations on HP notebooks. McDonalds has two customer channels, and I logged the machines into different channels, and both seemed to behave normally. I was watching political YouTube videos on the larger machine at the same time, and this didn't seem to slow anything down. I checked with the other customers here who were using the WiFi, and they didn't notice any degradation.
Topic: Processor speed and blockchain synchronisation (Read 2320 times)
Ok then, you have nothing to worry in terms of CPU though. Your issue is network transfer rate - it's too low. You need ~35 seconds to download a block and verification would only last half a second or less.
30kb/sec = the bottleneck is on the network (transfer rate, not pings). Your other components should be good for handling at least 1mb+ / sec. Try it with an ethernet cable if possible.
I can't use ethernet because it's part of my project to run a full node by using public WiFi.
30Kb is usually fine, and it often updates a couple of days worth fairly quickly. When I shut down core and restarted, it used a different set of peers, and thiose ran through fairly quickly. I was in the same restaurant, and used the same WiFi channel, I just restarted core within a few seconds of the shutdown.
30kb/sec = the bottleneck is on the network (transfer rate, not pings). Your other components should be good for handling at least 1mb+ / sec. Try it with an ethernet cable if possible.
OK - here is a bit of an update on this.
After a number of observations, I believe that the most significant factor for me is the ping time for my connected nodes.Because the public wifi services that I use the most limit me to 8 connections, and run fairly consistently at around 30 Kb/s, I don't think that storage or processor speed is a significant factor. I've been running a node for about 4 hours, and it started at 38 hours behind. After 4 hours it was 39 hours behind. I checked the nodes, and the best ping rate I had was 450 ms, and one was 870034ms. I closed core, and I re-opened it immediately. It has now cleared the backlog in fewer than 20 minutes. Half of the nodes in the new connections were around 30ms, and all were under 120ms.
It's looking as if I should have a preferred connection list. I'm mulling over the ramifications of this if I am operating from a variety of sites. The nodes are international, so I don't think location is significant. I wonder if network provider could be important though.
After a number of observations, I believe that the most significant factor for me is the ping time for my connected nodes.Because the public wifi services that I use the most limit me to 8 connections, and run fairly consistently at around 30 Kb/s, I don't think that storage or processor speed is a significant factor. I've been running a node for about 4 hours, and it started at 38 hours behind. After 4 hours it was 39 hours behind. I checked the nodes, and the best ping rate I had was 450 ms, and one was 870034ms. I closed core, and I re-opened it immediately. It has now cleared the backlog in fewer than 20 minutes. Half of the nodes in the new connections were around 30ms, and all were under 120ms.
It's looking as if I should have a preferred connection list. I'm mulling over the ramifications of this if I am operating from a variety of sites. The nodes are international, so I don't think location is significant. I wonder if network provider could be important though.
Well this morning I seem to be getting a lot of bad transmission messages for example
TCP previous segment not captured
duplicate ACK
TCP retransmission
I've found an interesting book on digital foresics, and it includes a chapter on the Bitcoin blockchain. I'll see if I can read and understand that.
TCP previous segment not captured
duplicate ACK
TCP retransmission
I've found an interesting book on digital foresics, and it includes a chapter on the Bitcoin blockchain. I'll see if I can read and understand that.
Here is a small extract from the packet stream.
I need to learn more about WiFi. I've got Acrylic installed, but I decided to install WireShark as well. It's about 47Mb in size, and I was trying for about 90 minutes to download it. I've switched channels, and it downloaded in under 10 minutes.
More info.
McDonalds uses a Cisco hub with a number of channels. Two of them are public - McDonalds and O2, they both have the same MAC address, and have a rated max speed of 54Mbps. I tend to use the McD channel. I shut down my PC and restarted to use the O2 channel - That was the one with the "quick" download. I'm starting to think that interference by other users may be a big part of the problem. afaik most users use McD.
I'll install Wireshark and see if that tells me anything interesting.
More info.
McDonalds uses a Cisco hub with a number of channels. Two of them are public - McDonalds and O2, they both have the same MAC address, and have a rated max speed of 54Mbps. I tend to use the McD channel. I shut down my PC and restarted to use the O2 channel - That was the one with the "quick" download. I'm starting to think that interference by other users may be a big part of the problem. afaik most users use McD.
I'll install Wireshark and see if that tells me anything interesting.
Another tip is perhaps disabling full journaling of the filesystem for initial sync as that does tend to slow it down, whether SSD or mechanic. Plus on SSD it would double all the writes (reducing the disk's lifetime).
That sounds like a good idea, but I can't find any info about it. By initial sync, do you mean the initial downloading of the blockchain? With this computer, I copied the blockchain directory from the i5 computer, and started the new node with the address of the copied file. That seems to have worked without any problems.
It's usually a mount option of the filesystem.
You can check with the mount command.
for example:
mount
/dev/sda5 on /home/alex/100GB type ext4 (rw,noatime,noacl,data=journal)
There are three modes:
https://www.kernel.org/doc/Documentation/filesystems/ext4.txt
data=journal All data are committed into the journal prior to being
written into the main file system. Enabling
this mode will disable delayed allocation and
O_DIRECT support.
note: this writes data twice, it's the slowest but is the safest option
for data corruption prevention.
data=ordered All data are forced directly out to the main file
system prior to its metadata being committed to the
journal.
data=writeback Data ordering is not preserved, data may be written
into the main file system after its metadata has been
committed to the journal.
note: this is the fastest but more unsafe if there is a sudden shutdown
They have both finished now. There seems to be a bit of a pattern here ( if 3 days can be considered a pattern). On startup core doesn't seem to be doing much, but there is lots of traffic for an hour or two. It even seems to go backwards slightly. It then seems to settle down, and it finishes the job at its old speed.
I'm planning to get a lenovo with Celeron N3060 1.6GHz upto 2.48GHz 2MB cache with 4GB RAM and GeForce 920M 1GB
I will install win 7 what manual configuration do I need to do for best blockchain sync?
I will install win 7 what manual configuration do I need to do for best blockchain sync?
Is it this one?
http://www.currys.co.uk/gbuk/computing/laptops/laptops/lenovo-ideapad-110-15-6-laptop-black-10146807-pdt.html
If it is, I'd try to get a copy of the blockchain directory on a dvd, or use the ethernet to connect to a machine that is already running a full node.
It takes forever to download the full blockchain.
I have internet speed up to 10MB/s
I wouldn't zip it. If you aren't sure that you can get an honest copy, then you may have to download it . I downloaded it over public wifi when it was about half the current size. I expect somebody could give you an estimate of the time it would taske to download on your machine.
The good news is that you don't have to do it in one hit, and it runs perfectly well in the background. Make sure you wait for core to close down completely if you do interrupt the process. You can do this as many times as you like, as long as it closes off completely before you turn off the machine.
I seem to be struggling with the sync on both machines today. I started a couple of hours ago, and I was 22 hours behind, I'm now 23 hours behind.
This is a picture of the network traffic on the i5
This is a picture of the network traffic on the i5
I'm planning to get a lenovo with Celeron N3060 1.6GHz upto 2.48GHz 2MB cache with 4GB RAM and GeForce 920M 1GB
I will install win 7 what manual configuration do I need to do for best blockchain sync?
I will install win 7 what manual configuration do I need to do for best blockchain sync?
Is it this one?
http://www.currys.co.uk/gbuk/computing/laptops/laptops/lenovo-ideapad-110-15-6-laptop-black-10146807-pdt.html
If it is, I'd try to get a copy of the blockchain directory on a dvd, or use the ethernet to connect to a machine that is already running a full node.
It takes forever to download the full blockchain.
I have internet speed up to 10MB/s
I'm planning to get a lenovo with Celeron N3060 1.6GHz upto 2.48GHz 2MB cache with 4GB RAM and GeForce 920M 1GB
I will install win 7 what manual configuration do I need to do for best blockchain sync?
I will install win 7 what manual configuration do I need to do for best blockchain sync?
Is it this one?
http://www.currys.co.uk/gbuk/computing/laptops/laptops/lenovo-ideapad-110-15-6-laptop-black-10146807-pdt.html
If it is, I'd try to get a copy of the blockchain directory on a dvd, or use the ethernet to connect to a machine that is already running a full node.
It takes forever to download the full blockchain.
Is there any serious profiling made with coin software to identity performance bottle neck ?
Yes. Many thousands of hours....
Bandwidth can be bottleneck in terms of block propagation speed.
Processing power is a bottleneck too, especially for initial chain syncing.
Is there any attempt in compressing block data sent over network ? Would that improve block propagation time ?
Is there a place I can find such kind of test and benchmarking ?
The transaction data in blocks is already highly optimized and compression would yield very minimal gains. However, there are numerous approaches to increasing block propagation... check out compact blocks and x-thin blocks.
I would believe there is still potentially repetitive data in blocks that could gain from compression, but not sure how much. But there are still adresses hashes in the script and certain thing that could gain a bit from compression, not sure the gain in transfert speed would over come the loss in compression/decompression time though.
Oki i'll check those
Found this :
"Datastream compression: Testing various compression libraries such as LZO-1x and Zlib have shown it is possible to further reduce block and transaction sizes by 20 to 30% without affecting response times which could also be applied to thinblocks"
Not so marginal
I'm planning to get a lenovo with Celeron N3060 1.6GHz upto 2.48GHz 2MB cache with 4GB RAM and GeForce 920M 1GB
I will install win 7 what manual configuration do I need to do for best blockchain sync?
I will install win 7 what manual configuration do I need to do for best blockchain sync?
Is there any serious profiling made with coin software to identity performance bottle neck ?
Yes. Many thousands of hours....
Bandwidth can be bottleneck in terms of block propagation speed.
Processing power is a bottleneck too, especially for initial chain syncing.
Is there any attempt in compressing block data sent over network ? Would that improve block propagation time ?
Is there a place I can find such kind of test and benchmarking ?
The transaction data in blocks is already highly optimized and compression would yield very minimal gains. However, there are numerous approaches to increasing block propagation... check out compact blocks and x-thin blocks.
Is there any serious profiling made with coin software to identity performance bottle neck ?
Yes. Many thousands of hours....
Bandwidth can be bottleneck in terms of block propagation speed.
Processing power is a bottleneck too, especially for initial chain syncing.
I can't do much about the processor or the ram unfortunately.
The wallet is on the internal SSD, and the blockchain is on the external SSD - it's on a USB3 port, and that is about the fastest connection I can get. Unfortunately this model doesn't have an RJ45 port ( is that the correct name these days? ).
If I get a chance, I'll do a bit of resource analysis to see if there is an apparent bottleneck.
I need to look at the WiFi, and make sure I've got the best drivers. I suspect the aerial on the i5 is twice the width of the Celery, and maybe this cuts down on resends. I'll see if I can find some software to check this.
Is there any serious profiling made with coin software to identity performance bottle neck ?
Yes. Many thousands of hours....
Bandwidth can be bottleneck in terms of block propagation speed.
Processing power is a bottleneck too, especially for initial chain syncing.
Is there any attempt in compressing block data sent over network ? Would that improve block propagation time ?
Is there a place I can find such kind of test and benchmarking ?
Is there any test using asynchronous io too to see if that can make a difference in synch speed or if it's mostly processing power on signature checking that take most of the time ?
Another tip is perhaps disabling full journaling of the filesystem for initial sync as that does tend to slow it down, whether SSD or mechanic. Plus on SSD it would double all the writes (reducing the disk's lifetime).
That sounds like a good idea, but I can't find any info about it. By initial sync, do you mean the initial downloading of the blockchain? With this computer, I copied the blockchain directory from the i5 computer, and started the new node with the address of the copied file. That seems to have worked without any problems.
Jump to: