Anyone running a small pool in US-West, make sure you are connected to at least one of the other relays ...
Edit: it was fixed - in case anyone was still wondering
Topic: Re: How (and why) to use the Relay Network (Read 730 times)
US-West relay been dead for 2.25 4 hours.
Anyone running a small pool in US-West, make sure you are connected to at least one of the other relays ...
Edit: it was fixed - in case anyone was still wondering
Anyone running a small pool in US-West, make sure you are connected to at least one of the other relays ...
Edit: it was fixed - in case anyone was still wondering
"Failed to parse -udpport option, not starting FIBRE"
...
udpport=8444,8555,8666
Not sure exactly what you're trying to do here...As o_solo_miner pointed out, to use the public relay network for receiving-blocks-quickly, you should simply run unmodified Bitcoin Core 14/15 and addnode to one of the FIBRE nodes (see http://bitcoinfibre.org/public-network.html)....
udpport=8444,8555,8666
If you wish to run your own FIBRE network to propagate blocks quickly between your own nodes, note that the udpport syntax (as pointed out in the --help output) is udpport=port,group,bw where port is a port number, group is a monotonically-increasing group id (ie it should be 0, the second port should use 1, etc) and bw is a bandwidth to use for sending to that group, in Mbps.
The Relaynode Network has been replaced by fibre and it is build in Core 14 / 15.01.
All you need is to register on the fibre Network: http://bitcoinfibre.org/public-network.html
Please read the first Page of this topic.
https://bitcointalksearch.org/topic/m.8635670
All you need is to register on the fibre Network: http://bitcoinfibre.org/public-network.html
Please read the first Page of this topic.
https://bitcointalksearch.org/topic/m.8635670
idk if this is still managed.,
but i tried building it from source and when i started it
i got
"Failed to parse -udpport option, not starting FIBRE"
even though i had
udpport=8444,8555,8666
in my config file.
i even checked logs for details and found above message.
but i tried building it from source and when i started it
i got
"Failed to parse -udpport option, not starting FIBRE"
even though i had
udpport=8444,8555,8666
in my config file.
i even checked logs for details and found above message.
I really appreciate your work, Matt. You have no idea how many problems your simple post has solved for me. Thanks a lot!
Seattle (us-west) fibre relay has been dead for about 45 minutes 2 hours
Fixed Thanks Matt
Back to your problem
I do not think that dropped connections have anything in common with bitcoin rpc max threads at all.
Relay client is not using RPC at all. It talks raw to btcd and relay servers
Other thing to check is your BTC max peer count. I had this problem when I reached max peer connections and I have restarted relay client it would never connect.
So one of the hacks to my btcd deamon is to use peer -1 and always to have a room for relay client Wink
I am running btcd 0.11 and relay with kano hack (because it never worked without it-thanks kano) and after block downloading again after mats incident it works just fine no issues
I do not think that dropped connections have anything in common with bitcoin rpc max threads at all.
Relay client is not using RPC at all. It talks raw to btcd and relay servers
Other thing to check is your BTC max peer count. I had this problem when I reached max peer connections and I have restarted relay client it would never connect.
So one of the hacks to my btcd deamon is to use peer -1 and always to have a room for relay client Wink
I am running btcd 0.11 and relay with kano hack (because it never worked without it-thanks kano) and after block downloading again after mats incident it works just fine no issues
The udpnode stuff is not exposed on the public FIBRE instances because I dont significantly trust that there isnt some bug in the implementation somewhere which could cause the nodes to crash with data sent corrupted in some way. The other option (the fibrenetworkclient thing from my RelayNode github) exists only to make sure connections stay up reliably. See two posts up (where we talked about https://github.com/bitcoin/bitcoin/pull/9319), which will make -addnode a much more reliable "keep this connection open" option in Bitcoin Core 0.14. fibrenetworkclient shouldn't make any performance difference at all.
Lets talk Metrics here.... I run a full public bitcoin node with p2pool attached.... my getblock latency mean averages around .25s ... but gets as low as .12s on occasion....
I do not yet have the Relay set up and running.
If my latency is .25s or less, do I NEED a relay?
This has nothing to do with getblock latency. This has to do with the time until you see a block. p2pool has no idea how long it took from when someone else found a block to when you received it, and this is the part that relay networks work to solve.I do not yet have the Relay set up and running.
If my latency is .25s or less, do I NEED a relay?
OK.. got it. Next Question:
I have whitelisted my IP at the US East coast relay and my regular v13 bitcoin node has it listed as a peer.
What performance improvements will result from using the specialized "bitcoinfibre" node and using the "addtrustedudpnode" (or "addudpnode") command(s)?
And how can I measure it or see the improvement locally?
Lets talk Metrics here.... I run a full public bitcoin node with p2pool attached.... my getblock latency mean averages around .25s ... but gets as low as .12s on occasion....
I do not yet have the Relay set up and running.
If my latency is .25s or less, do I NEED a relay?
This has nothing to do with getblock latency. This has to do with the time until you see a block. p2pool has no idea how long it took from when someone else found a block to when you received it, and this is the part that relay networks work to solve. I do not yet have the Relay set up and running.
If my latency is .25s or less, do I NEED a relay?
Lets talk Metrics here.... I run a full public bitcoin node with p2pool attached.... my getblock latency mean averages around .25s ... but gets as low as .12s on occasion....
I do not yet have the Relay set up and running.
If my latency is .25s or less, do I NEED a relay?
./fasbit
I do not yet have the Relay set up and running.
If my latency is .25s or less, do I NEED a relay?
./fasbit
Any work on-going on making this step easier? For example a "better" -addnode argument that maintains the connection to the added seednode?
See https://github.com/bitcoin/bitcoin/pull/9319.Thank you very much. Excellent stuff.
Any work on-going on making this step easier? For example a "better" -addnode argument that maintains the connection to the added seednode?
See https://github.com/bitcoin/bitcoin/pull/9319. "Note that, because bitcoind treats the -addnode argument as an extra seednode it may use, and not as a connection which it should maintain, it is recommended that you use an external daemon to keep the connection reliable.
Any work on-going on making this step easier? For example a "better" -addnode argument that maintains the connection to the added seednode?
So since it's now next to impossible to see what's going on in the fibre relay network, you wouldn't happen to know yourself what happened at UTC 2016-12-11 16:43pm for the blocks that came though during that minute?
I can see both sides of the network seeing the two different blocks, but no idea about the timing of the opposing blocks crossing the GFW.
It would seem that the fibre network wasn't getting into china very well at all for the orphan I got then ...
The two I see are mine at 16:43:57.141 UTC and one in china later at 16:43:58.910 UTC (that won the orphan race when antpoo confirmed it)
The old relay stopped 15 minutes before that.
Edit: you should see my block coming in to all non-chinese fibre nodes directly, during that, before the block was found inside china.
Though that depends on bitcoind (0.13.0) and when it got around to deciding to send it to your fibre relay, and when you accepted it.
I can see both sides of the network seeing the two different blocks, but no idea about the timing of the opposing blocks crossing the GFW.
It would seem that the fibre network wasn't getting into china very well at all for the orphan I got then ...
The two I see are mine at 16:43:57.141 UTC and one in china later at 16:43:58.910 UTC (that won the orphan race when antpoo confirmed it)
The old relay stopped 15 minutes before that.
Edit: you should see my block coming in to all non-chinese fibre nodes directly, during that, before the block was found inside china.
Though that depends on bitcoind (0.13.0) and when it got around to deciding to send it to your fibre relay, and when you accepted it.
For those using the new FIBRE-based Relay Network, I just added the following paragraph to the public-network page:
"Note that, because bitcoind treats the -addnode argument as an extra seednode it may use, and not as a connection which it should maintain, it is recommended that you use an external daemon to keep the connection reliable. There is a simple connect-two-bitcoinds-together client available in the Relay Network Client Source Tree (called fibrenetworkclient after you run make) which takes, as arguments, two bitcoind instances, and connects them to each other, reliably reconnecting if one goes down."
"Note that, because bitcoind treats the -addnode argument as an extra seednode it may use, and not as a connection which it should maintain, it is recommended that you use an external daemon to keep the connection reliable. There is a simple connect-two-bitcoinds-together client available in the Relay Network Client Source Tree (called fibrenetworkclient after you run make) which takes, as arguments, two bitcoind instances, and connects them to each other, reliably reconnecting if one goes down."
Do you have numbers to compare - coz that's really the point here.
I of course have the old compression numbers for every block since the beginning of time since I've used your relay
If the new one is better compression (= faster block transmission) then you can say that if you have numbers
First of all, you're making a common mistake here - less data != faster block transmission when you're talking about things in the 1-20k range. The extra cost of a packet isnt a ton (loss is not hugely likely if you're on a decent connection until you start talking about 100k+ data) and the extra transmission time of a packet is virtually nothing - so an additional 20ms in receive->ProcessNewBlock time is often worth it. With stuff like https://github.com/bitcoin/bitcoin/pull/9125 this is very realistic.I of course have the old compression numbers for every block since the beginning of time since I've used your relay
If the new one is better compression (= faster block transmission) then you can say that if you have numbers
Post a run of any 100 blocks and I'll get the same blocks and see (like my 50 block post above) - that's really what I want to see.
I dont have tons of numbers handy, but if you look at a bitcoind with -debug running that receives blocks not over the RN, then you can easily see compact block reconstruction info (eg blocktxn and cmpctblock message sizes). In my recent log for blocks received over compact blocks most blocks took between 12k and 17k for the blocktxn, and few needed a blocktxn roundtrip (printed in log as having received a 33-byte blocktxn, which isnt a real message on the wire).Edit2: there's also another question in the interface design - what's the comparison of the 2 for back and forward communication?
e.g. comparing the two (I don't know what that are)
As noted in the previous post, the old RN protocol is one-way as far as blocks are concerned...it makes strong assumptions about knowledge of the other side's mempool (requires you to store the last N txn sent to you by the server), and can thus just send you the compressed block and knows you can reconstruct it. The Compact Blocks protocol isnt so nice, and sometimes requires a round-trip to request missing transactions, though being able to send the cmpctblock message 5-10ms sooner and saving 10-20ms on the receiving side can almost make up for that if you're (reasonably) close to one of the RN servers (eg AWS us-east or us-west should meet that criteria easily). Hopefully further optimizations to the compact block implementation (that the protocol designed for) in 0.14 will will make the likelyhood of round-trips much lower.e.g. comparing the two (I don't know what that are)
My main concern is that bitcoind sux.
It's hopeless at keeping connections and has too much overhead.
The relay seemed to be able to keep the connection for a long time (I have counters that display in a different colour when it loses/regains the connections) so I've seen how reliable that can be.
(I don't "watch" the old relay log, I instead have a script run a simple analysis of the last 9999 lines in the log every 10 seconds)
I suspect that the relay was pretty minimal in back and forth communication and data size vs bitcoind.
Yea, this is something I thought about a lot when considering the protocol change, actually this more than the latency concerns. While I dont know why it would be bad at keeping connections, aside from not reconnecting agressively enough, the agressive reconnection stuff that I have in the old RN is super nice since I can pretty freely restart the servers, etc. As for protocol-chatty-ness, I think they're not that far off, bitcoind will send more transactions, but thats a good thing - more transactions to match the compact block against It's hopeless at keeping connections and has too much overhead.
The relay seemed to be able to keep the connection for a long time (I have counters that display in a different colour when it loses/regains the connections) so I've seen how reliable that can be.
(I don't "watch" the old relay log, I instead have a script run a simple analysis of the last 9999 lines in the log every 10 seconds)
I suspect that the relay was pretty minimal in back and forth communication and data size vs bitcoind.
. ...
...
Do you have numbers to compare - coz that's really the point here.With this new 'connect a bitcoind' directly to your network, all traffic from your network comes and goes directly to the main bitcoind.
How is the block compression compared to the current (old) relay?
Compact Blocks get much better compression than the old relay network in real-world testing. There are a few minor tweaks that need to happen that will take them from reasonably-often-missing-transactions to rarely-missing-transactions (including orphan pool, recent-replacements, etc).How is the block compression compared to the current (old) relay?
...
I of course have the old compression numbers for every block since the beginning of time since I've used your relay
If the new one is better compression (= faster block transmission) then you can say that if you have numbers
Post a run of any 100 blocks and I'll get the same blocks and see (like my 50 block post above) - that's really what I want to see.
Edit: well to make it simpler - all blocks 439000 to whenever you can find the stats (currently 439277)
Edit2: there's also another question in the interface design - what's the comparison of the 2 for back and forward communication?
e.g. comparing the two (I don't know what that are)
1) 'send block'-> (finished)
2) 'message'-> 'reply'<- 'block'-> (finished)
0r
3) something else added to handle the txns
Or
4) ...
etc.
My main concern is that bitcoind sux.
It's hopeless at keeping connections and has too much overhead.
The relay seemed to be able to keep the connection for a long time (I have counters that display in a different colour when it loses/regains the connections) so I've seen how reliable that can be.
(I don't "watch" the old relay log, I instead have a script run a simple analysis of the last 9999 lines in the log every 10 seconds)
I suspect that the relay was pretty minimal in back and forth communication and data size vs bitcoind.
OK, so correct me if I'm wrong, but why is a slower relay considered better?
If I understand the limitations you've added correctly, all you have done is speed up your internal UDP relay, but replaced the external connections to it with a slower relay and added excessive burden on the bitcoind.
Not sure what "excessive burden" you're referring to here - replacing one P2P connection with another shouldn't change the load on your bitcoind, even if one is remote...If you really want to split hairs you should see less load on your bitcoind because the message hashing for an entire block no longer happens.If I understand the limitations you've added correctly, all you have done is speed up your internal UDP relay, but replaced the external connections to it with a slower relay and added excessive burden on the bitcoind.
The current (old) relay means there's a program (normally) local to the bitcoind that's hit with transactions at whatever rate is happening with the relay, then when a block is found, the traffic from your service is minimal, usually just a small compressed block, and then the local relay pushes a whole block to the local bitcoind that happens to (normally) be 0ms away, so no network issue at all.
Not quite...the old relay network protocol is bandwidth-limited to only send out a certain number of transactions per second (I think its like 5Mbps, so it shouldnt ever really hit that, but it does means that if you get a burst it could get behind.With this new 'connect a bitcoind' directly to your network, all traffic from your network comes and goes directly to the main bitcoind.
How is the block compression compared to the current (old) relay?
Compact Blocks get much better compression than the old relay network in real-world testing. There are a few minor tweaks that need to happen that will take them from reasonably-often-missing-transactions to rarely-missing-transactions (including orphan pool, recent-replacements, etc).How is the block compression compared to the current (old) relay?
Do you send transactions from your new relay to the bitcoind? Coz that will also be an unwanted thing to happen.
This isnt a change from the old behavior - the old client used to send transactions it received to bitcoind locally ((re-)starting a relay network client remains one of the easiest ways to get your mempool filled after restart a node, at least until 0.14).Basically it sounds like the new relay is really a transaction improvement relay, not a block relay improvement, so an improvement for payment sources, but a slower block relay for pools.
Your internal network may be faster, but the external network seems slower by design.
Compact Blocks is a far, far superior protocol to use on the wire vs the old realy network protocol, and the fact that its in bitcoind directly instead of via an external daemon means the latency of it can be much lower (it can skip, for example, the hash-message-check-message-hash stuff that happens now). Indeed, there is a drawback of occasionally having an extra round-trip from clients to relay network servers, but current testing shows that a) the improvement in average latency between the servers far outweighs most RTTs to clients (eg within a continent you shouldn't see more than 50-100ms RTT), b) its relatively easy to shave another 10-30ms (yes, really, its that bad) off the total receive->ProcessNewBlock time, which should all happen in 0.14, in compact blocks, whereas for the RN code its really not so trivial, and c) we can hopefully make the RTTs much less common with some simple fixes which the compact blocks protocol already supports, just aren't yet implemented. Your internal network may be faster, but the external network seems slower by design.
OK, so correct me if I'm wrong, but why is a slower relay considered better?
If I understand the limitations you've added correctly, all you have done is speed up your internal UDP relay, but replaced the external connections to it with a slower relay and added excessive burden on the bitcoind.
The current (old) relay means there's a program (normally) local to the bitcoind that's hit with transactions at whatever rate is happening with the relay, then when a block is found, the traffic from your service is minimal, usually just a small compressed block, and then the local relay pushes a whole block to the local bitcoind that happens to (normally) be 0ms away, so no network issue at all.
With this new 'connect a bitcoind' directly to your network, all traffic from your network comes and goes directly to the main bitcoind.
How is the block compression compared to the current (old) relay?
Do you send transactions from your new relay to the bitcoind? Coz that will also be an unwanted thing to happen.
Basically it sounds like the new relay is really a transaction improvement relay, not a block relay improvement, so an improvement for payment sources, but a slower block relay for pools.
Your internal network may be faster, but the external network seems slower by design.
Edit: here's the last 50 old relay block sizes in transferred bytes (that last number). What's the new one for these?
If I understand the limitations you've added correctly, all you have done is speed up your internal UDP relay, but replaced the external connections to it with a slower relay and added excessive burden on the bitcoind.
The current (old) relay means there's a program (normally) local to the bitcoind that's hit with transactions at whatever rate is happening with the relay, then when a block is found, the traffic from your service is minimal, usually just a small compressed block, and then the local relay pushes a whole block to the local bitcoind that happens to (normally) be 0ms away, so no network issue at all.
With this new 'connect a bitcoind' directly to your network, all traffic from your network comes and goes directly to the main bitcoind.
How is the block compression compared to the current (old) relay?
Do you send transactions from your new relay to the bitcoind? Coz that will also be an unwanted thing to happen.
Basically it sounds like the new relay is really a transaction improvement relay, not a block relay improvement, so an improvement for payment sources, but a slower block relay for pools.
Your internal network may be faster, but the external network seems slower by design.
Edit: here's the last 50 old relay block sizes in transferred bytes (that last number). What's the new one for these?
Code:
[2016-11-16 02:48:11.106+00] ..00092682eadc186df.. recv'd, size 480071 with 7759 bytes
[2016-11-16 02:48:43.987+00] ..00090e83c28f2e244.. recv'd, size 40890 with 372 bytes
[2016-11-16 02:58:44.541+00] ..001df2eba7161c6f3.. recv'd, size 815374 with 6467 bytes
[2016-11-16 03:17:48.229+00] ..00194859b7edddd08.. recv'd, size 749215 with 4347 bytes
[2016-11-16 03:21:53.697+00] ..000b8a18e37427efc.. recv'd, size 995350 with 107909 bytes
[2016-11-16 03:24:56.709+00] ..001f852409054b6b9.. recv'd, size 379872 with 55638 bytes
[2016-11-16 03:45:50.157+00] ..00390e6e74393934d.. recv'd, size 998890 with 43410 bytes
[2016-11-16 03:54:36.501+00] ..000177238c9682e4f.. recv'd, size 998190 with 4671 bytes
[2016-11-16 04:12:17.972+00] ..0044f0b258077f23c.. recv'd, size 998142 with 5109 bytes
[2016-11-16 04:18:16.561+00] ..00048ef2fddab806c.. recv'd, size 775518 with 3192 bytes
[2016-11-16 04:19:00.934+00] ..0033c9e6d5f71d09b.. recv'd, size 102301 with 54094 bytes
[2016-11-16 04:19:12.686+00] ..002f51db6db619e07.. recv'd, size 42063 with 414 bytes
[2016-11-16 04:19:30.174+00] ..002a16788c9ad4aef.. recv'd, size 55625 with 382 bytes
[2016-11-16 04:19:39.133+00] ..000dc7a6eed875d98.. recv'd, size 8488 with 6485 bytes
[2016-11-16 04:42:00.685+00] ..00335deb00d796f75.. recv'd, size 999967 with 5814 bytes
[2016-11-16 04:48:45.110+00] ..0030cc64619151b69.. recv'd, size 859533 with 3710 bytes
[2016-11-16 04:52:07.216+00] ..0020d7cec45f78a15.. recv'd, size 998197 with 709974 bytes
[2016-11-16 04:58:05.087+00] ..001994c1f03adbbe9.. recv'd, size 362497 with 2132 bytes
[2016-11-16 05:11:25.948+00] ..0036ef48e33c117c7.. recv'd, size 999164 with 82598 bytes
[2016-11-16 05:13:17.240+00] ..0038a1e01b6de37cf.. recv'd, size 145171 with 693 bytes
[2016-11-16 05:27:12.666+00] ..000761cf6e37b57af.. recv'd, size 998038 with 4244 bytes
[2016-11-16 05:27:30.403+00] ..00398a5b97f368fec.. recv'd, size 97689 with 632 bytes
[2016-11-16 05:36:41.080+00] ..003c56ee2ea460182.. recv'd, size 755314 with 3028 bytes
[2016-11-16 05:43:48.049+00] ..003ddbca9351653f9.. recv'd, size 592994 with 34850 bytes
[2016-11-16 05:51:45.980+00] ..000ed0e3df1e5d3f9.. recv'd, size 993582 with 358432 bytes
[2016-11-16 06:03:21.346+00] ..00354d61136d8659a.. recv'd, size 879448 with 9050 bytes
[2016-11-16 06:06:48.633+00] ..003738fe282036c86.. recv'd, size 341092 with 86181 bytes
[2016-11-16 06:13:56.436+00] ..00335e2307e39a9eb.. recv'd, size 543532 with 2902 bytes
[2016-11-16 06:35:00.886+00] ..003659797406d34c9.. recv'd, size 999045 with 91518 bytes
[2016-11-16 06:37:33.387+00] ..002fe93e6a654620e.. recv'd, size 562524 with 2522 bytes
[2016-11-16 06:39:35.114+00] ..00048cf85b8a28f1c.. recv'd, size 111666 with 826 bytes
[2016-11-16 06:42:45.695+00] ..0019e8ee8a754b8c3.. recv'd, size 997973 with 722715 bytes
[2016-11-16 06:47:39.188+00] ..002a3bb0e0abf4450.. recv'd, size 275764 with 1653 bytes
[2016-11-16 06:48:22.713+00] ..0013298e19f4ab876.. recv'd, size 120658 with 469 bytes
[2016-11-16 06:49:07.700+00] ..003fe9c7f91392e0f.. recv'd, size 135217 with 78859 bytes
[2016-11-16 07:11:23.336+00] ..001ece483a24a3504.. recv'd, size 997997 with 6018 bytes
[2016-11-16 07:12:14.726+00] ..00415a03fd0ad8d16.. recv'd, size 749156 with 80903 bytes
[2016-11-16 07:23:08.089+00] ..000a731efbfec2080.. recv'd, size 997997 with 2541 bytes
[2016-11-16 07:25:22.546+00] ..0012d0155e0a287c1.. recv'd, size 683120 with 36350 bytes
[2016-11-16 07:36:08.687+00] ..002073bf93047c1bd.. recv'd, size 684824 with 3809 bytes
[2016-11-16 07:46:41.966+00] ..001feaae195e3d206.. recv'd, size 849255 with 4149 bytes
[2016-11-16 07:47:09.905+00] ..003d2504008e560cc.. recv'd, size 997374 with 918590 bytes
[2016-11-16 07:47:41.005+00] ..00252cb551eb2ba3e.. recv'd, size 49173 with 329 bytes
[2016-11-16 08:13:43.015+00] ..0029b70d5b108efbb.. recv'd, size 997890 with 5902 bytes
[2016-11-16 08:17:14.009+00] ..001b4ec9166d9a8b6.. recv'd, size 998211 with 4539 bytes
[2016-11-16 08:22:23.174+00] ..001c33f82658413f9.. recv'd, size 807397 with 3143 bytes
[2016-11-16 08:26:39.030+00] ..000001c04a845d6d1.. recv'd, size 307998 with 1756 bytes
[2016-11-16 08:28:48.756+00] ..001caa93923bb5cec.. recv'd, size 201068 with 1220 bytes
[2016-11-16 08:28:50.152+00] ..0006c8dbd2026ce8a.. recv'd, size 191 with 127 bytes
[2016-11-16 08:56:17.868+00] ..003c616e0ac5df293.. recv'd, size 998106 with 5367 bytes
[2016-11-16 02:48:43.987+00] ..00090e83c28f2e244.. recv'd, size 40890 with 372 bytes
[2016-11-16 02:58:44.541+00] ..001df2eba7161c6f3.. recv'd, size 815374 with 6467 bytes
[2016-11-16 03:17:48.229+00] ..00194859b7edddd08.. recv'd, size 749215 with 4347 bytes
[2016-11-16 03:21:53.697+00] ..000b8a18e37427efc.. recv'd, size 995350 with 107909 bytes
[2016-11-16 03:24:56.709+00] ..001f852409054b6b9.. recv'd, size 379872 with 55638 bytes
[2016-11-16 03:45:50.157+00] ..00390e6e74393934d.. recv'd, size 998890 with 43410 bytes
[2016-11-16 03:54:36.501+00] ..000177238c9682e4f.. recv'd, size 998190 with 4671 bytes
[2016-11-16 04:12:17.972+00] ..0044f0b258077f23c.. recv'd, size 998142 with 5109 bytes
[2016-11-16 04:18:16.561+00] ..00048ef2fddab806c.. recv'd, size 775518 with 3192 bytes
[2016-11-16 04:19:00.934+00] ..0033c9e6d5f71d09b.. recv'd, size 102301 with 54094 bytes
[2016-11-16 04:19:12.686+00] ..002f51db6db619e07.. recv'd, size 42063 with 414 bytes
[2016-11-16 04:19:30.174+00] ..002a16788c9ad4aef.. recv'd, size 55625 with 382 bytes
[2016-11-16 04:19:39.133+00] ..000dc7a6eed875d98.. recv'd, size 8488 with 6485 bytes
[2016-11-16 04:42:00.685+00] ..00335deb00d796f75.. recv'd, size 999967 with 5814 bytes
[2016-11-16 04:48:45.110+00] ..0030cc64619151b69.. recv'd, size 859533 with 3710 bytes
[2016-11-16 04:52:07.216+00] ..0020d7cec45f78a15.. recv'd, size 998197 with 709974 bytes
[2016-11-16 04:58:05.087+00] ..001994c1f03adbbe9.. recv'd, size 362497 with 2132 bytes
[2016-11-16 05:11:25.948+00] ..0036ef48e33c117c7.. recv'd, size 999164 with 82598 bytes
[2016-11-16 05:13:17.240+00] ..0038a1e01b6de37cf.. recv'd, size 145171 with 693 bytes
[2016-11-16 05:27:12.666+00] ..000761cf6e37b57af.. recv'd, size 998038 with 4244 bytes
[2016-11-16 05:27:30.403+00] ..00398a5b97f368fec.. recv'd, size 97689 with 632 bytes
[2016-11-16 05:36:41.080+00] ..003c56ee2ea460182.. recv'd, size 755314 with 3028 bytes
[2016-11-16 05:43:48.049+00] ..003ddbca9351653f9.. recv'd, size 592994 with 34850 bytes
[2016-11-16 05:51:45.980+00] ..000ed0e3df1e5d3f9.. recv'd, size 993582 with 358432 bytes
[2016-11-16 06:03:21.346+00] ..00354d61136d8659a.. recv'd, size 879448 with 9050 bytes
[2016-11-16 06:06:48.633+00] ..003738fe282036c86.. recv'd, size 341092 with 86181 bytes
[2016-11-16 06:13:56.436+00] ..00335e2307e39a9eb.. recv'd, size 543532 with 2902 bytes
[2016-11-16 06:35:00.886+00] ..003659797406d34c9.. recv'd, size 999045 with 91518 bytes
[2016-11-16 06:37:33.387+00] ..002fe93e6a654620e.. recv'd, size 562524 with 2522 bytes
[2016-11-16 06:39:35.114+00] ..00048cf85b8a28f1c.. recv'd, size 111666 with 826 bytes
[2016-11-16 06:42:45.695+00] ..0019e8ee8a754b8c3.. recv'd, size 997973 with 722715 bytes
[2016-11-16 06:47:39.188+00] ..002a3bb0e0abf4450.. recv'd, size 275764 with 1653 bytes
[2016-11-16 06:48:22.713+00] ..0013298e19f4ab876.. recv'd, size 120658 with 469 bytes
[2016-11-16 06:49:07.700+00] ..003fe9c7f91392e0f.. recv'd, size 135217 with 78859 bytes
[2016-11-16 07:11:23.336+00] ..001ece483a24a3504.. recv'd, size 997997 with 6018 bytes
[2016-11-16 07:12:14.726+00] ..00415a03fd0ad8d16.. recv'd, size 749156 with 80903 bytes
[2016-11-16 07:23:08.089+00] ..000a731efbfec2080.. recv'd, size 997997 with 2541 bytes
[2016-11-16 07:25:22.546+00] ..0012d0155e0a287c1.. recv'd, size 683120 with 36350 bytes
[2016-11-16 07:36:08.687+00] ..002073bf93047c1bd.. recv'd, size 684824 with 3809 bytes
[2016-11-16 07:46:41.966+00] ..001feaae195e3d206.. recv'd, size 849255 with 4149 bytes
[2016-11-16 07:47:09.905+00] ..003d2504008e560cc.. recv'd, size 997374 with 918590 bytes
[2016-11-16 07:47:41.005+00] ..00252cb551eb2ba3e.. recv'd, size 49173 with 329 bytes
[2016-11-16 08:13:43.015+00] ..0029b70d5b108efbb.. recv'd, size 997890 with 5902 bytes
[2016-11-16 08:17:14.009+00] ..001b4ec9166d9a8b6.. recv'd, size 998211 with 4539 bytes
[2016-11-16 08:22:23.174+00] ..001c33f82658413f9.. recv'd, size 807397 with 3143 bytes
[2016-11-16 08:26:39.030+00] ..000001c04a845d6d1.. recv'd, size 307998 with 1756 bytes
[2016-11-16 08:28:48.756+00] ..001caa93923bb5cec.. recv'd, size 201068 with 1220 bytes
[2016-11-16 08:28:50.152+00] ..0006c8dbd2026ce8a.. recv'd, size 191 with 127 bytes
[2016-11-16 08:56:17.868+00] ..003c616e0ac5df293.. recv'd, size 998106 with 5367 bytes
Jump to: