Topic: Increasing outgoing connection limit (Read 393 times)

You make it complicated for no reason. This comment from the PR explains how accessible full node operators have priority in propagating transactions. If they get whitelisted by others then even better.
I think it's makes sense that node operators will get something in return, not just the miners. They provide valuable services the community too. Don't you think node operators should be rewarded?

Yes, which was released in 0.21 on January 14th, 2021.

https://bitcoincore.org/en/releases/0.21.0/

Search for #19988 in the list at the end.

Sometimes changes are written years before they make it into a release.


The "and" in which sentence?

There is no constant two second delay.  When a peer offers you a transaction if the peer was one you connected out to and you haven't already requested the transaction from someone else you will instantly request it.  If the peer is a peer connecting in to you, you will remember that and start a two second timer. During the timer you will immediately request the txn if a peers you are connecting out to offers it, and if other inbound peers offer it you will remember their offers.  After the two second timer expires, if any peer you are connecting out to has still not offered the the transaction you will randomly select one of the available offers and request the transaction, this fairly seldom happens. There are additional penalties for peers that you have many txn requests already outstanding.  The average effect is just that transactions get requested in a preferred direction across links (because every link has an inbound and an outbound side) without changing the delay much, but it substantially blocks tarpitting attackers since they depend on connecting many times to a target and triggering request timeouts.

It was introduced here, PR was merged on Oct 14, 2020, that would be 8 months ago.

https://github.com/bitcoin/bitcoin/commit/242d16477df1a024c7126bad23dde39cad217eca

The "and" in that sentence is actually "or". From reading the code it replaced the random delay with constant 2 seconds.

Preferred status is set *automatically* by the node. It can also be set as a side effect of the noban permission.  I was attempting to discover if your complaint is about the automatic behavior, which is an important protection against attack, or if your complaint is about the fact that it's also a side effect of noban.  I still don't know, because you seem to just be ignoring that it's also set automatically.

The latter isn't important, except that some tests use noban to eliminate delays that otherwise get in the way of performing some tests. It's not even a documented behavior and could probably just go away except for whatever tests are depending on it.

Anyone could turn off the behavior generally on their own node, but doing so will just reintroduce the tarpitting attack vulnerability (see section 5.3) and reduce their privacy.

It seems to me that ratholed yourself on some extremely minor, relatively newly introduced behavior, that has fairly little effect on transaction propagation overall (except against tarpitting attackers), particular since it's just a control for which peers out of multiple offers you'll request a transaction from. ... and you're continually ignoring that the your fundamental premise that users are somehow disadvantaged by small differences in transaction relay is just false as many people have pointed out.

It's a configuration flag, it can only be configured by operators of other nodes by adding --whitelist as i showed you.
There are many ways of testing software that don't require preferential treatment of other nodes.
I get your strategy, just circling the drain until I give up.

That it applies to peers automatically.

It's unclear to me what you're asking.

Is your complaint that the 'no ban' whitelist option bypasses some delays? You'd have to figure out how to fix the functional tests that need to bypass the delays for their testing purposes-- which is pretty much why it's there AFAIK.  But who cares if I'd accept it? I am already not whitelisting any peers so it would make no difference.  Outside of testing the whitelist stuff is hardly used, and doesn't generate reports even when it's been broken in various ways.

Or is your complaint just that the transaction download manager treats some peers differently, without any configuration?  That is an important protection against denial of service and for user privacy, and it shouldn't be removed-- and the only reason I could see someone wanting to remove it is if they were a malicious party.

Where is the rest of your response?  We're all still waiting to hear why you believe some disadvantage is created for some user somewhere and why you're obsessed about code that sometimes delays the speed at which you _request_ transactions (vs the speed people offer them) which was released just a couple months ago.

Here's how it is configured: https://github.com/bitcoin/bitcoin/blob/master/src/init.cpp#L471-L473
Just set the NoBan (VIP) for whoever you want.

https://github.com/bitcoin/bitcoin/blob/master/src/net_processing.cpp#L1054

https://github.com/bitcoin/bitcoin/blob/master/src/net_processing.cpp#L1062

What am i missing here?


Since it's so useless, would you accept a PR to remove the preferential propagation?
Anyone can add it to their own fork anyway

Because it wouldn't benefit them, it would just waste their bandwidth and cause other participants to banlist them due to wasting their bandwidth.

As I explained above, that has absolutely nothing to any of the whitelisting settings, preferred nodes are random peers selected by the software, and not something users can configure.  It doesn't apply to all requests but only some peers, it there because it protects nodes against a tarpitting dos attack where malicious peers offer it transactions but don't reply when they're requested.  It also helps protect nodes privacy by making it less clear to peers if a node already had a transaction which they offered to it.

That particular code wasn't even added until 0.21, which was only release four and a half months ago.

There isn't any whitelisting there the transaction download manager's preferences are its own, your fixation on conspiracy theories appears to be clouding your judgement.

As many people have pointed out here, there is generally no benefit to small differences in transaction relay speed for network participants.  Continually making arguments that only make sense if there were one doesn't cause there to be one, it just makes you look a bit unhinged.

Also you keep downplaying the transaction delay which is set to 2 seconds for non-preferred nodes
https://github.com/bitcoin/bitcoin/blob/master/src/net_processing.cpp#L1062
https://bitnodes.io/dashboard/#transactions-propagation
Maybe people sell that whitelisting so whales can have guaranteed service
Can you add a list of preferred wallets too? I think that would be even more popular than preferred addresses

You're missing the point with the extreme numbers, why wouldn't people configure their nodes to have 125 outgoing connections and 8 incoming instead of 8 and 125 respectively. I think it's in order to support the "network", providing services to all those nodes using NAT. It's just like upload/download using P2P file sharing software.
Also you can get a list of nodes from bitnodes.io api so you don't have to waste time crawling

You can't have it both ways, if using outgoing connections & NAT works as well as running a node as far as availability & performance, why would anyone spends money running a full node? Most of them are on hosting providers.
If full nodes have an advantage (maybe higher availability during big market moves), then the privileged access flag is damning.

It also means Bitcoin is very vulnerable to community backlash. If public will pressure AWS, Google Cloud and other hosting providers to kick bitcoin nodes off their services there will not be enough resources to support all those NAT nodes. Most public nodes i can reach are hosted on such services. Amazon is buying carbon credits and touts their ESG goals. Seems hypocritical to me...
5k * 125 incoming connections < 83 * 11 outgoing

Do you think they run their nodes with only 10 peer connections or do they increase the outgoing connection limit?

seems the topic creator wants 'fast' tx relaying so he can gain a vew millisecond advantage to see bitcoin deposits going into exchanges before they are even confirmed

what he doesnt realise is although a whale might be depositing funds into an exchange. it may not result in a on exchange market order that day or week
also the deposit does not reveal if that funds leads to a long or a short order. thus becomes redundant metric to use
lastly that whale tx being relayed may not actually be a deposit resulting in an on-exchange market order. but could also be just the exchange itself re-organising its cold store reserves. thus making analysing the tx relay and blockchain useless for predicting in-exchange activity

i found it myself. depositing funds into an exchange made me flag up as a whale and suddenly i had to fill in some KYC stuff and a 48 hour admin time.. to get level X utility within the exchange.
so the topic creator stressing over a few milliseconds of tx relay wont benefit him in predicting when trades are going to happen because a deposit does not = instant trade on exchange
and a deposit does not reveal which direction a trader wants to swing. (long or short)


the disconnect. search ip list, connect. handshake send tx disconnect cycle of operations wastes more seconds to 83k peers. rather than the time it takes to just pass data through the network the normal way peer-to-peer relay. to 83k

also the peers you wish to connect to must have available slots to even allow the connection to a new peer and also not already ban hammered you with spam connections from previous tries. so its not something that would last long if you were to try it.
you end up just wasting time and bandwidth and eventually treated like a bad node the network refuses to connect with

.. in short the very thought of a max of 6 hops and max of 2 second delay per hop in your scenario is 12 seconds to get from originator to all nodes.
batch connect send disconnect and repeat for all 83k nodes would end up taking over 12 seconds to attempt and still no guarantee you reached all 83k nodes in your attempts. thus redundant

sending a tx 83k time in atleast 12 seconds is more of a waste than peer-to-peer network relay of upto 12 seconds where your node only sends the tx 10 times

NAT.  For quite a few years Bitcoin hasn't had any automatic hole punching by default, after it got turned off due to repeated vulnerabilities in the standard UPNP library (and the upnp code was determined to be too dubious to keep risking it).

That will likely change somewhat in the next major release as it adds NAT-PMP support (nat-pmp doesn't require a @#$@ XML parser). PMP isn't as widely supported by routers as UPNP, however.

It's also the case that users get some security and privacy advantages in being not connectable, so even when nat isn't an issue some users choose to disable inbound connections, but it's mostly due to the lack of by-default portmapping.

Why do most nodes don't accept incoming connections?

There are a *lot* more than 10k nodes, a good estimate is about 64k nodes.  Most nodes don't accept inbound connections, so the only way you know they are there is when they connect to you.  That means 64k nodes connecting out to 9k or so listeners.  Connections aren't made uniformly either... if you are on a network block with few nodes you'll get more than your fair share (and fewer than if you are on a network with many nodes). There are also a few mass connectors, including "forked.net" which may be making 10-15 connections to you from a single /23, if you don't have them banned.  It's typical to lose 15-20 connections when you apply a banlist.

Last question for you.. Let's say there are 10,000 nodes in the network all playing by the rules and opening total of 110,000 outgoing connections. How come my node receives 125 connections? Extrapolating that would mean There are 1,250,000 outgoing connections in the network. We somehow have more than 1m unexpected connections. Is it possible others are using unrestricted forks too?
Would love to hear other explanations.

That is by far one of the dumbest things I have ever heard.
Almost no trading is done with live transactions.
Coinbase / Gemini / Binance / etc are where trading happens. You have to have your BTC on an exchange with several confirms before you can sell.
And you have to have your cash sitting there before you can buy.

--Dave

What the heck? In the 2+ years I've been involved in Bitcoin and a whole year of running a node, I have never heard of or even seen people running "privileged nodes".

There is no such thing as privileged nodes. All nodes are solely identifiable by their IP address (some have domain names but as most of those are arbitrary too, that's missing the point). So it would be a PITA for colluding "elite" users to compile and distribute a list of IP addresses and ports to each other, let alone feeding it to the command line or config file using --addnode flags. Hence why nobody does it.

Indeed. Obtaining 81920 IPs will not allow you to replace all the entries in the table, that itsn't how it works.

In what sense? Node availability changes very slowly.



This isn't meaningful. Bitcoin transactions just move bitcoin, they don't establish prices in any markets.  You're hand waving.



Your post was full of errors wrong, from top to bottom.  But, as I mentioned-- Bitcoin intentionally doesn't relay transactions as fast as possible, and for intentional and good reasons.

Here you're still mistaken.  Wallets originate transactions, they don't pay any attention to transactions other than ones paying them.  A user is not disadvantaged by learning that a payment was made a few seconds later, and if the user cares to know that information they can have the payer tell them directly, which is much faster.


Simply asserting it doesn't make it true.  You seem to think that Bitcoin transactions are spot market trades which offer or establish some price. They are not and do not.  Trades in the spot markets don't require any bitcoin transactions, because they use funds which are in confirmed deposits in exchanges.  Funds which just moved *cannot* be traded, typically not until after 3 or 6 blocks have passed.  Nothing in a bitcoin transaction establishes a price or even indicates an intent to trade.

Moreover, to whatever limited extent that a user might be disadvantaged or traded against based on their transaction -- protecting their privacy is important, which is the primary purpose of relay delays in the first place.

The 'preferred' term doesn't even have anything to do with the noban setting, and it cannot be configured to apply to particular peers.

The 'preferred' term you're discussing comes from heuristics used in selecting which of multiple offering peers to request a transaction from to avoid tar-pitting attacks. A peer could offer you a transaction and then fail to transmit it when you request it, leaving you to wait until the requests times out.  To avoid this, the node will prefer to fetch from a well behaved peer which is less likely to attack.  If the first peer to offer the transaction wasn't one of these preferred peers it will wait up to a small timeout for a preferred peer to offer the transaction, and if the timeout is reached it will pick a source at random from the offering parties.


You could start by reading the PR that introduced the whitelisting a number of years ago...
or simply look at the test suite in bitcoin/test/function where it's used by 18 different tests.


Many people run their own automation to detect abusive nodes (mass connecting spies, dos attackers, and other resource wasters) and banlist them, there is no coordination in that case.

Some people run that automation and publish lists on the internet which other people can adopt if they want.

Thanks for reviewing my content, i'll fix the mistakes.

The reference of 10 connections refers to someone launching official software on their computer in order to make a transaction. They can wait until other nodes will connect to them and then submit the transaction or send it to the limited pool of outgoing nodes, thanks for clarifying it's even more restricted than i thought.

81,920 is not a lot. You can just buy it on hosting providers for about 810$/hr. Feeler interval is also very restrictive.

Bitcoin transactions obviously determine prices in other markets and vice versa. So it's not a stock market on it's own yet it's futures are traded in the stock market.

That gives wallet software even lower priority, proving my pecking order right. Also makes my node with 1400 incoming connections receive transactions earlier than the average node.
I'm sure many trade Bitcoin futures using such privileged information or trade Bitcoin using high frequency data from futures market.

The elephant in the room is still with us, why some nodes are preferred and are not subjected to delay?
How do you know who uses that flag and for what?
How are they coordinating the various black lists you mentioned?

There are small differences, basically because you control who you outgoing connect to but an attacker controls that they connect in to you,  so outbound connections are somewhat safer against malicious behavior and are slightly preferred in some places.  Overall the protocol is highly symmetrical but where a peers misbehavior could cause delays and it could freely choose between outbound and inbound peers, it will prefer outbound peers.


In the current networking protocol the aggregate cost to the network for transaction relay is multiplied by the connection degree of nodes. Communication cost ~= constant_1 * nodes * degree * transactions + constant_2 * nodes * transactions.   There are proposals to change this, but that is how it is today (and in every other similar system).

Because of outgroup batching making more connections won't significantly speed up your reception, intentionally so, in order to act against transaction surveillance.  -- which is something your post completely fails to mention,  the processing batching isn't some flaw, error, or limitation--  it is an intentional designed functionality which is extremely important for participant privacy.

There are also potential delays in requesting novel transactions which are important for protecting nodes from denial of service attacks.

Your post takes for granted that there is some issue created by transaction propagation taking a couple seconds, but there isn't-- blocks are found with a 600 second expectation (meaning at every instance the mean time to the next block is about 600 seconds from now) and by comparison to this number a couple seconds are close to irrelevant.  For a user these tiny differences in transaction propagation serve them no disadvantage-- if your transaction is propagated now or a second from now it makes no difference. Transaction's aren't a race.  No protocol that depended on transactions being a race could be secure in any case: because an attacker could get extraordinary delay advantages (at considerable cost) through geographic distribution no matter what the node behavior was.

The only place where small transaction delays matter at all is in mining. The gain a miner has in learning of a new transaction is the marginal increase in fees that transaction offers over the worst transaction it was previously going to include and will now leave out.  Because fees are relatively isotropic this improvement is typically very small.  And if a miner does include a transaction which is poorly propagated they do so at great cost:  Block propagation is many times faster when all transactions in the block are known to the recipient, since in that case the propagation can happen without a round trip.

As things are, most mining infrastructure adds considerable delay post block template creation, the average appears to be greater than 30 seconds except when there is a new chain tip.  If miners were concerned about the lost marginal income of the newest transactions that would obviously be the first place to optimize.

Going into more detail-- your specific claims about the operation of the system are false.  If you'd like to suggest improvements it's important to understand what it's actually doing (and why) and not just substitute the actual operation with your assumptions.


Maybe someone dosed your punch with blockchain-braindamage-koolaid.  Bitcoin is not a stock exchange. Users are not vulnerable to front running. Even Franky1 pointed this out to you.  It's not clear to me if you've been personally buzzword addled or if you're just trying to dupe buzzword addled users into running some dubious non-peer-reviewed code.