...
Network routing is one of those arcane specialties filled with scary-smart people who know all the minutiae of complex systems which most developers are barely aware exist. (How does the Internet really work? How do all those little packets know where to go? Magic!) I am not in that specialty, and I’m not familiar with its research literature
I'm also not a specialist, but understand enough on the basics. So no, not magic, and I bet you know as well :-)
IP protocol is well defined, with routing information (or destination) in the header of each packet. In the very early days a router would ask his neighbor, do you have a route for network x? (neighbor discovery protocol). Someone to whom I was connected would answer accordingly (or not, then "PATH NOT FOUND"). And the question of routing optimization started quite early. There was this ATM network model at the same time (goal to unify networks with telephony and IP in 53 byte packets!)... Remember good old CISCO 27xx or 29xx models with configs for RIPv1 and v2, and then open shortest path first (OSPF)? These were the internal routing protocols. Then there was the bigger networks in the late 80s and they needed a professional base layer (border gateway protocol, BGP and IGRP/EGRP, or EGP). And of course a DNS. btw: why is DNSsec not used today?
... what first occured to me was that Spanning Tree Protocol might somehow be applicable. Of course, that’s not an Internet routing protocol; but it is the standard staple for organizing the network topology on LANs.
Spanning tree was the protection layer in bridges, to prevent loops in larger networks, with the "routing" of MAC addresses (bridge = layer 2 device). Yup, no routing at layer 4 here ...
The general question is: Given a global set of nodes which form and remove links between each other unpredictably, how does each node organize its own view of potential routes and choose optimal paths?
Lookup tables?
I took a look into the routing in Lightning - you come to deal with SPINX, HMACs, and of course onion routings. I think the difficulty layer of lightning routing stems from the fact, that the node only knows the predecessor and the successor of a route. Nothing else. Not the origin, not the final destination, not the amount (but then, how to know, that the channel supports the requested value transfer?), and for sure in a way bullet proof, that the node cannot benefit from forwarding the package to a different target. There I find this table in the data structure of 20 entries @65 bytes (hops_data), which makes me think on what it is used for? Is there a max of 20 hops?
Also the flare white paper (
http://bitfury.com/content/5-white-papers-research/whitepaper_flare_an_approach_to_routing_in_lightning_network_7_7_2016.pdf) says:
Hence, it is in the interest of the sender to optimize fees and make the final decision on which route to choose to the recipient (otherwise the sender gets potentially unpredictable expenses, as other nodes are not incentivized to optimize for the cheapest path).
and
Source routing leads to the requirement that the sender node should be able to collect information on fees and available channel capacity to pick the best route (as well as knowing which nodes are currently online). Thus, an overlay mechanism should exist to enable requesting information about a channel from any of its owners.
Interesting, that would mean no dynamic routing?
I don't understand (yet) how routing achieves a dynamic management, when it is pre-defined from the beginning. And this would mean, it does not require IP routing specialists? There's work to be done!