how to maximize block download speed on a single local node

lontivero

full member

Activity: 164

Merit: 128

Amazing times are coming

Thanks for the explanation about ping/pong messages.

Yes, I was testing with bitcoind and NBitcoin in the same local machine. And yes, CPU and Disk IO are very low usage (CPU is almost idle).

Nicolas Dorier

hero member

Activity: 714

Merit: 662

lonteviro, the numbers are in bytes / sec.
I witnessed some peaks at 15 MB/sec like you, but this was more or less random and did not seem to depends on the version of the node.

Quote

Another wierd thing that I saw (i was not able to reproduce it) is that while downloading the blocks, a ping/pong exchange in the middle took 4 seconds.

This is normal, the messages sent from the node to the client are queued in bitcoind.

If client ask 10 000 blocks then does 1 ping, then the pong of the server will be queued at the end of the 10 000 blocks.
This is also why the pong timeout is 20min in bitcoind.

Did you tested on a local node ? your results are very coherent with mine.

I notice no bottleneck of disk IO or CPU. The socket buffer being never full means that NBitcoin is processing blocks faster than they are coming from bitcoind.

Did someone managed (on any library) to fetch block from a local node at a highest speed than 10MB/s ?

lontivero

full member

Activity: 164

Merit: 128

Amazing times are coming

Something is wrong somewhere because I've tested several times and I don't understand what is going on between NBitcoin and bitcoind. Here a share some of the results:

This is one of several run that I did with 0.10.1

And this is the test output: (those numbers are the kb/s)

And the following are the results for the same test against 0.11rc

(run 1)

As we can see, for a while it performed much better however, after a minute more or less it went down.

(run 2)

(run 3)

(run 4)

I tested this in exactly the same way. Bitcoind was connected only with NBitcoin.

Another wierd thing that I saw (i was not able to reproduce it) is that while downloading the blocks, a ping/pong exchange in the middle took 4 seconds.

Nicolas Dorier

hero member

Activity: 714

Merit: 662

I just tested the 0.11rc1.

The headers are downloaded 3 times faster. (Downloading 3MB/s instead of previous 1MB/s)
My block download speed have not changed very much.

I don't notice the 100ms wait time consistently anymore.

Nicolas Dorier

hero member

Activity: 714

Merit: 662

I am already fetching the blocks with getdata 10000 blocks.

The 100ms I have seen is not the same as "100ms per block".
I precisely mean that when the receive buffer of my socket is empty, then it takes consistently 100ms before new data comes inside.

I measure this just before reading the next message.

It makes no differences if those are 1MB blocks or 1kb blocks.

I feel the PR gmaxwell talks about fixed my problem. I'll let you know on 0.11.

TierNolan

legendary

Activity: 1232

Merit: 1094

Quote from: Nicolas Dorier on May 16, 2015, 07:48:44 AM

Cool I'll try that. If the problem really come from this sleep, it will be more than 30% improvement, this might be a 10x improvement for my case. (Since my code does not verify the blocks, only download them, most of the download time is wasted waiting 100ms)

I ran with the lastest on master on Linux, and it is 0-1ms per block for pure downloading. I made sure to download 1000 blocks with each getdata request.

On Linux, 0.10.1.rc2 is quick as well, but only if I request multiple blocks at once. Each getdata message costs 100ms, so sending one for every 1000 blocks is much faster.

Weirdly, when I run the java download code on windows with the latest download (0.10.1), it is 20-30ms per block.

"git checkout master"

getdata(1000 blocks): 0-1ms per block
getdata(1 block): 37-42ms per block

"git checkout v0.10.1rc2" [Linux]

getdata(1000 blocks): 1-3ms per block
getdata(1 block): 100-101ms per block

Nicolas Dorier

hero member

Activity: 714

Merit: 662

I tried to follow the build process.

Quote

./bin/gbuild --commit bitcoin=master ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml

Quote

Stopping target if it is up
Making a new image copy
Starting target
Checking if target is up
Preparing build environment
Updating apt-get repository (log in var/install.log)
Installing additional packages (log in var/install.log)
Upgrading system, may take a while
Creating package manifest
Creating build script (var/build-script)
Running build script (log in var/build.log)
Grabbing results
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
./bin/gbuild:21:in `system!': failed to run copy-from-target out build (RuntimeError)
from ./bin/gbuild:267:in `block (2 levels) in

'
from ./bin/gbuild:259:in `each'
from ./bin/gbuild:259:in `block in

'
from ./bin/gbuild:257:in `each'
from ./bin/gbuild:257:in `

'

install.log and build.log are empty.

build-script is

Quote

NICO@bitcoin-build:~/gitian-builder$ tail var/build-script
  find . -name "lib*.la" -delete
  find . -name "lib*.a" -delete
  rm -rf ${DISTNAME}/lib/pkgconfig
  find ${DISTNAME} | sort | tar --no-recursion --mode='u+rw,go+r-w,a+X' --owner=0 --group=0 -c -T - | gzip -9n > ${OUTDIR}/${DISTNAME}-${i}.tar.gz
  cd ../../
done
mkdir -p $OUTDIR/src
mv $SOURCEDIST $OUTDIR/src
mv ${OUTDIR}/${DISTNAME}-x86_64-*.tar.gz ${OUTDIR}/${DISTNAME}-linux64.tar.gz
mv ${OUTDIR}/${DISTNAME}-i686-*.tar.gz ${OUTDIR}/${DISTNAME}-linux32.tar.gz

Well, except I forgot something obvious, I guess I will just wait for 0.11 to be out, I guess none of you have time to help me to build successfully, and I don't have time to figure out linux voodoo by myself.
I'll update this post when 0.11 is out to publish the result that 0.11 have on my perfs problems.

Nicolas Dorier

hero member

Activity: 714

Merit: 662

Quote from: gmaxwell on May 16, 2015, 05:05:50 AM

There is an intentional sleep in message handling.

It was removed in 0.11 (git master right now) via Patrick Strateman PR 5971.

There are excruciatingly precise building instructions, https://github.com/bitcoin/bitcoin/blob/master/doc/gitian-building.md sufficient to get someone who isn't terribly technical producing bit identical binaries to the release.

(A simpler build process works too, of course, if you don't care about reproducing the builds-- but those instructions also have the benefit of being very precise.)

Cool I'll try that. If the problem really come from this sleep, it will be more than 30% improvement, this might be a 10x improvement for my case. (Since my code does not verify the blocks, only download them, most of the download time is wasted waiting 100ms)

gmaxwell

staff

Activity: 4284

Merit: 8808

There is an intentional sleep in message handling.

It was removed in 0.11 (git master right now) via Patrick Strateman PR 5971.

There are excruciatingly precise building instructions, https://github.com/bitcoin/bitcoin/blob/master/doc/gitian-building.md sufficient to get someone who isn't terribly technical producing bit identical binaries to the release.

(A simpler build process works too, of course, if you don't care about reproducing the builds-- but those instructions also have the benefit of being very precise.)

Nicolas Dorier

hero member

Activity: 714

Merit: 662

Quote from: TierNolan on May 15, 2015, 02:32:29 PM

Are you using your own software as receiver? I think CPU rather than network bandwidth is normally the issue.

Yes the client side is my own software. But my CPU is mostly idle, and the rcv buffer of my client does not have time to fill up, meaning that I am processing faster than the speed to which bitcoind is sending me blocks.
What bother me is the consistent delay for filling the rcv buffer once I emptied it by reading my socket... It takes 100ms. (+- 20ms)

TierNolan

legendary

Activity: 1232

Merit: 1094

Are you using your own software as receiver? I think CPU rather than network bandwidth is normally the issue.

Nicolas Dorier

hero member

Activity: 714

Merit: 662

thanks, I'll take a look. I'm not linux wizard but it should help

hhanh00

sr. member

Activity: 467

Merit: 267

Quote from: Nicolas Dorier on May 13, 2015, 10:55:09 AM

No, generally speaking, everytimes I tried compiling bitcoind, I gave up after several hours wasted and frustrated, so I just imagine this is caused by the sleep. The 100ms timing seems very consistent.

Did you try looking at the github autobuilder log? I think that if Travis can do it, it should be easy to do manually too.
https://travis-ci.org/bitcoin/bitcoin/jobs/62665884

Nicolas Dorier

hero member

Activity: 714

Merit: 662

Just tried to connect with a bitcoind on my dev machine. There was some dash to 1MB/s, then it settled around 30kb/s. (I stopped at block 50K)
I guess this is normal since bitcoind checks incoming blocks. Would have loved to try at higher blocks though. (but my dev machine SSD is full Sad

)

Nicolas Dorier

hero member

Activity: 714

Merit: 662

Just to be sure, I'll also try to sync a bitcoind from my dev machine with the one of my server to see if I get the same speed, just to rule out any possibility that something in my code is wrong.

Nicolas Dorier

hero member

Activity: 714

Merit: 662

Quote from: 2112 on May 13, 2015, 11:53:11 AM

Quote from: Nicolas Dorier on May 13, 2015, 10:55:09 AM

The 100ms timing seems very consistent.

I was wondering if I was missing anything obvious, any "magic parameter" somewhere.

TCP_NODELAY?

Interestingly John Nagle has an account on this forum:

https://bitcointalksearch.org/user/nagle-28488

Nice try, but did not changed anything.
I think the NoDelay is more for the sender. If I manage to compile bitcoind, I'll try to set the no delay on it, along with the sleep change.

2112

legendary

Activity: 2128

Merit: 1073

Quote from: Nicolas Dorier on May 13, 2015, 10:55:09 AM

The 100ms timing seems very consistent.

I was wondering if I was missing anything obvious, any "magic parameter" somewhere.

TCP_NODELAY?

Interestingly John Nagle has an account on this forum:

https://bitcointalksearch.org/user/nagle-28488

Edit: I'm actually posting this from Windows, not my normal machine. It took me a while to locate the place where the OS include files are located. In a search for the correct spelling I found a funny comment in one of the Windows' SDKs:

Code:

// turn off nagling

Obviously making a verb out of somebody's last name is a sign of respect. I'm just wondering whether starting it with lower case 'n' means more respect or less respect?

Nicolas Dorier

hero member

Activity: 714

Merit: 662

Quote

Is this with Bitcoin Core 0.10.1?

Yes

Quote

Are you only syncing the block chain over your local network to one other machine? If you can sync from multiple machines, that can speed things up.

Yes, but this is not what I want to do : I am currently benchmarking my code on NBitcoin, and noticed that the download was slow and that my receive buffer was never full, which indicate bitcoind is slow.

Quote

HDD or SSD on the machines?

4 HDD in RAID5, 10K RPM (the server has his own RAID controller)

Quote

CPU spec

I own a Dell Powerdge T620, the CPU is not capped, nor the Disk IO.

Quote

Have you tried dropping the 100ms sleep to 1ms?

No, generally speaking, everytimes I tried compiling bitcoind, I gave up after several hours wasted and frustrated, so I just imagine this is caused by the sleep. The 100ms timing seems very consistent.

Quote

There are a lot of conditions there. I'd also try to set for a start the sleep to a very low value and if there is a real improvement I'd debug a little.

Thanks, well, I guess I will have to try compiling bitcoin one more time on windows.

I was wondering if I was missing anything obvious, any "magic parameter" somewhere.

NeuroticFish

legendary

Activity: 3668

Merit: 6382

Looking for campaign manager? Contact icopress!

There are a lot of conditions there. I'd also try to set for a start the sleep to a very low value and if there is a real improvement I'd debug a little.
It's too easy to miss something there.

TierNolan

legendary

Activity: 1232

Merit: 1094

Quote from: Nicolas Dorier on May 13, 2015, 07:42:54 AM

However by reviewing the code, I have seen no mistake : if the send buffer of the bitcoin node is not full, I should not experience any 100ms delay.
Once again, trying to change the buffers size of bitcoind had no effect.

It looks like it will only skip the sleep if at least one node received data and it is unprocessed. I don't why that is connected to the sending buffer.

Have you tried dropping the 100ms sleep to 1ms?

Topic: how to maximize block download speed on a single local node (Read 2370 times)