Topic: Re: How (and why) to use the Relay Network - page 2. (Read 675 times)

Really not sure what you're trying to say here, but - New network is UDP between its own servers (dedicated 1Gbps on every link except the Beijing one, but its fast enough, too), but then Bitcoin-P2P protocol between the servers and clients. Hopefully you're using Bitcoin 0.13+ so you'll be using compact blocks, which are generally more effecient than the old relay network protocol by a decent margin (plus I have some hacks to force compact block clients into low-latency mode). If you have a server which is particularly far from the nearest relay network server I'm happy to talk about setting up something that isnt Bitcoin-P2P, and am thinking about doing FIBRE-based peering (so UDP floods for block announce).

Nope, see https://github.com/bitcoinfibre/bitcoinfibre/commit/335f87424a60af1fc02bffb7c84e90b8e2eadeca which will fast-relay compact blocks out to peers within a few ms of receiving the block if either a) you're a segwit peer or b) segwit has not activated.

Ahh, I see the misunderstanding - nope, its only Bitcoin-P2P publicly for now.

So correct me if I'm wrong but ...

Old network
'Relay' to 'local' using TCP but only a small 'compressed' block and that passes it to the local bitcoind for full verification

New network
'Relay' to 'remote' using UDP, full size 'compressed' block?, full verification?, then pass to 'local' bitrcoind for full verification again

So ... that looks like it's slower than before ...

... and also I must run segwit ... which I don't want to do.

... and of course any pool that would open their main bitcoind to UDP traffic (to remove the extra step) clearly has never considered the risks of doing that or has no idea about how UDP works ... as I brought up before ...

The new peering instructions are up at http://bitcoinfibre.org/public-network.html.

@Matt I think it is good if u please update OP with FIBRE related information.

Yea, the HK one died after the host decided to wipe the drive on the server when the NIC failed....yes, you read that right. I didn't bother to re-install since a) I dont actually remember how to reinstall that stuff, and havent had time to look it up (FIBRE is so much easier...) and b) I want to turn the thing off for SegWit (though getting a public FIBRE network up has taken longer than expected :/.

Indeed, one of the things I'm planning on doing is listing those sponsoring on the site by putting their logo up. I assume if folks are paying 2-300$/month for something they will use it. Indeed, it was an uphill battle to get people to use it initially, but its much better these days. Bitfury still has some insanity occasionally, but I'm working with them to solve it.

Yea, they're all dedicated servers since the random lag introduced by VPSes was incredibly annoying. Sadly, for dedicated servers, unlike virtual ones, it can be hard to get them in good locations in some places (APAC especially), and its also very nice to have them all on one network, so they're all on Softlayer. They're a bit more expensive than neccessary, but they have really, really good latency between any two points on their network (with the exception of JPY<->EU as they dont have TEA bandwidth), as well as solid peering in Tokyo/HK with PCCW/Pacnet, which is important for China.

Well ... a few things.

I take it that the current relay network has been crappy for a while coz you decided you weren't getting enough sponsorship?
i.e. when the sponsorship isn't enough in your (unstated) opinion you just let it go to crap?
(No I've never sponsored it - but I do connect to every relay ... even the long dead hk one - our ckpool block relay is faster than using the relay to distribute to other relay nodes so I connect to all nodes to get our blocks out faster)

Unless you have the top pools connected (i.e. every pool over a few % of the network) it's value is marginal so I'd think you should actually list the pools that are sponsoring and are connected - and update it when that changes - since that's not a large list.
I'd also suggest you whitelist access to most of it. That way you know who is connected and know it is still useful.
e.g. f2pool wasn't on the relay until long after the relay existed, and I've no idea if all of the 3 bitmain pools use it. Bitfury's high orphan rate at times makes me wonder if they have always used it.

Well I'm certainly no where near 7% of the network but I'll put forward ~$100 in BTC a month towards it (as long as the biggest pools are on it)
N.B. none of my servers but the single main back end cost me even half of that So I guess this network is either using an expensive provider or high end servers on every node

Hi folks,

I'd like to turn off the relay network on December 1st. For that to happen, I'd like to first have a public relay network of nodes based on FIBRE available ASAP.

In order to make the network much more reliable/lower-latency than the original relay network, I'd like to use real servers with good,
dedicated, bandwidth. I speced out a network which would cost about $1550/month and would have public peering nodes in Hong Kong, Tokyo, Seattle, Washington DC, Beijing, and one of London/Amsterdam/Frankfurt. Additionally, there would be (at least) one node in Siberia to utilize the TEA Cable route.

A few folks have already agreed to chip in to make this a reality, but I'd love to see a few more so that we can make it incredibly cheap for
everyone involved. If you'd like to help, I'm looking for people willing to make commitments to cover a fair share of the above costs, and would list the sponsors on bitcoinrelaynetwork.org (which would be updated with new info).

This has been misbehaving for the last week or so. Any chance it could get a kick?

If anyone is looking to set up a FIBRE-based network, I'm happy to share the slow-as-fuck bash script which generates the slow-to-render pretty graphs on test network stats page.

Yep good solution.
One collection of 'satellites' I have are sub 1ms (around 1/3ms)

Well looks like I'll be behind the times for a while if it's been merged, coz yeah not really interested in adding code to centralise bitcoin and give away mining fees to those centralising it so they can take those fees.

If you're talking about running your own FIBRE-based network: No need to enable incoming UDP on pool servers to run FIBRE. TCP/Compact Block relay to servers only a few ms away is plenty fast, so you could just as easily put a relay network server on a separate server (and probably should, given that its still beta and based on Bitcoin Core master just before segwit was merged).

If you're talking about connecting to someone else's: I never really envisioned it as running over UDP between miners and the first-hop server. While this is definitely an option (and FIBRE has an "untrusted" mode for this), there is little advantage to spending 10ms calculating FEC data over just eating the 10ms RTT for a dropped packet or two if the servers are nearby. FIBRE is great for long-haul links, but Compact Blocks work just as well if you're talking about short-hops.

General note: FIBRE's UDP does have a shared-key HMAC in each packet, so that packets are easily and trivially authenticated.

I block all UDP and run no UDP services coz it's been the majority cause of DDoS that occurred since I started the pool ...
... since there is no 'network security' applied to UDP on the internet.

So I'm sure everyone has seen, by now, my blog post on the future of the relay network. I do have a test-network based on FIBRE which has shown incredibly good performance, but I'm not quite sure if I will allow public connections to it (it currently only connects to random nodes on the public P2P network and to the original relay network) or allow others to use FIBRE to take up the mantle of running high-speed relay networks. For now, the original relay network continues to run and the FIBRE network running side-by-side is only making it faster!
If you want any help running your own FIBRE-based network, I'm more than happy to help out!

For those paying attention...The relay network donation address was recently nearly entirely depleted to pay for a new set of servers...The network is going to switch entirely to a new infrastructure based on bitcoind with UDP-FEC (ie no more delays from TCP lag) which allows for cut-through routing by rumoring UDP-FEC packets around the entire network before receiving the entire block. You can find the source for this patch at https://github.com/TheBlueMatt/bitcoin/tree/udp-wip...The API to it is being refactored to enable the cut-through routing optimizations, and note that it includes some LGPL components in the binary, so its probably all secretly under LGPL, but IANAL, please dont sue me.

See https://github.com/TheBlueMatt/bips/blob/master/bip-TODO.mediawiki for something similar which is more targeted at the P2P nature of Bitcoin. See-also https://github.com/TheBlueMatt/bitcoin/tree/udp-wip for something which is intended to eventually replace the Relay Network - a whitelisted-peers-only secondary relay channel that is easy for miners to use between their own nodes (ie its just an additional flag) and will also be much faster than the relay network today in its worst-case (and probably average) relay times.

Yeah I asked that already ... the problem is that at least some of the bitcoin devs are morons.
https://github.com/bitcoin/bitcoin/issues/7049

... especially when you get the part where gmaxwell considers exceptionally useful code black magic

I didn't want to wade through 19 pages of text and I know this has been answered before, but why would it be a bad idea to integrate this into the P2P network on a bilateral basis? Wouldn't it still help a bit?

If worst comes to worst could you throw together a short docu on how the server components fit together? Trying to piece together how and with what each of them talk to by digging through the C code is less than optimal.

Servers go down sometimes .

Until other things that are much higher priority (ie the current drama) is dealt with, I will keep restarting servers when they go down but no other changes are gonna happen...I'll revisit things in a month or three.

Hey Matt. Any update on what the plans are for your invaluable network? Thanks.