Topic: The Lightning Network node experience - page 8. (Read 3380 times)

I could have done the same, but I wanted to test out the backup plugin in case I ever need to use it due to drive failure. Hopefully, this never happens.

Interesting. When I backed my C-Lightning node up and restored it recently, I just had to back up hsm_secret and the database. But I just copied the files after shutting down the service. No tools necessary. Worked flawlessly Just copied them to the new installation after it was done, and before starting it up; that's it.

I decided to scrap my home server so I had to move my Bitcoin and Lightning nodes back to a Raspberry Pi. The migration process was painless. Here's what I did:

1) I made sure that my old VM won't boot anymore. You really don't want to accidentally run two instances at the same time or an outdated database.
2) I copied my hsm_secret to .lightning/bitcoin folder on my Raspberry Pi.
3) I ran ./backup-cli restore file:///mnt/external/location ~/.lightning/bitcoin/lightningd.sqlite3 to restore my channel backup.
4) I had to remove my backup file and initialize the backup plugin again: ./backup-cli init --lightning-dir ~/.lightning/bitcoin file:///mnt/external/location/file.bkp

My node is up and running with the same Tor address and channels. It is worth mentioning that LND docs explicitly state that migration between different operating systems and system architectures might lead to data corruption. I guess that's another legitimate reason to choose c-lightning over LND.

November was so far the best month. My node routed 59 transactions and collected 548.46 satoshi in fees. I earned the most from payments incoming from the channels which were opened this month.

I have been getting a lot of "Local_failed" routing attempts in RTL since the triangle swap. lightning-cli listforwards returns WIRE_TEMPORARY_CHANNEL_FAILURE for all of these recent transactions so I decided to investigate further. I turned on debug logging and here's what I found:


This way, you can learn which channel failed to route the payment and how large it was. RTL shows only the "IN" channel.


Thanks! I won't be able to come close to your node's capacity, but I will do whatever I can to help the network.

Glad it's working out! I forwarded 288 payments yesterday so sometimes I miss individual ones! Keep growing that node!

---

Let's say I have a channel to node A. I set that fee to say 100 sats per 1M sats. If that channel is empty the script is confident that if it was repleted it would make 100 sats per 1M again. So it looks for a rebalance path that would be less than 40 sats/1M (I playing around with the cutoffs, but thats the general approach).

It's worth noting that you can make others wait longer to claim the funds than yourself. Currently, my node makes other people wait 432 blocks (~3 days), but I still accept channels with up to 2016 blocks (~2 weeks) delay. I don't mind waiting that long. Most people don't change the default value of 144 blocks anyway.

If you want to reject channels with excessive timelocks change either bitcoin.maxlocaldelay (LND) or max-blocktime-watch (c-lightning). If you want to force the other party to wait longer if they broadcast their commitment transaction change bitcoin.defaultremotedelay or watchtime-blocks (c-lightning). Note that those parameters do not affect existing channels.

That's good to mention! I have had to force close channels before and indeed, sometimes it can be quite inconvenient and even a bit worrisome when you don't see the funds coming back for days the first time you do a force close. A buddy recently did his first force close and he was quite irritated as well. Definitely worth to consider, maybe even reducing lock time during channel creation if you anticipate the possibility of needing a quick force close in the future.

It can also be argued that it's a feature, not a bug, but yeah I see how it can be disturbing especially for large amounts of BTC.

For those who have not followed my other posts I have been running a few nodes in a box: https://bitcointalksearch.org/topic/nodes-in-a-box-5364113 today the fan in one of them failed: https://bitcointalksearch.org/topic/nodes-in-a-box-5364113.msg58485606#msg58485606

Since it's an RPi the fan is more or less not needed. BUT since it was making a really annoying noise I decided to close all the channels and shut it down. Well, since 2 of the other ends were offline I now waiting for them to close. Whatever, not a big deal. But, if I did have a lot of BTC locked up in a channel that I needed I could see that being a real annoyance. I know on an intellectual level that this could happen and it's just a wait it out. But it's still something people should think about. And since this is about the experience of running a LN node. Well, I guess dealing with things like this is worth a mention.

-Dave

Very interesting. How does it anticipate the expected future fees? (what assumptions does it make)

I suspect that when LN has an increased adoption rate, the most successful LN node operators will automate their channel balances such that they are balanced (no pun intended), and that there is a low risk channels will be closed due to an unbalanced channel.

Yesterday, I performed a triangle swap with two other bitcointalk members. Each of us opened a 0.05 BTC channel to the other person. This way, we gained additional 0.10 BTC capacity, which means that this swap has doubled my node's capacity. I paid 222 sat for the opening transaction. We dropped the fees to 0 and TheJuice's script took care of the rebalancing. On the same day, I routed 6 transactions from the channel I opened and I got paid 222.76 satoshi. The channel paid itself off in just a few hours.

Note that on some days (ex. today) I routed a few small payments (< 1 sat). It's impossible to tell it from the graph at this point.

I have 2 -- one dynamically adjust fees based on the proportion of the sats on each side (and some other parameters). The rebalance script looks to get atleast 2M sats outgoing on each channel -- however it will only rebalance if the expected fees from future transactions are higher (by 3x) than the fee to rebalance.

What does your rebalance script do? Does it adjust fees once one side of a channel reaches a certain point? Or does it send a invoice to yourself that is paid to a channel with a high balance to a channel with a low balance?

There are 4 major reason:

1) It's part of the game to get higher rankings. A lot of the rankings take into consideration average channel size. When I first started out I had the limit at 1M sats and have been slowly increasing. Higher rankings help with #4 below.
2) I've found that smaller channels almost never generate significant transactions -- I suspect this is both b/c smaller nodes open smaller channels and it's harder to keep sats on both sides of a small channel.
3) My rebalance script targets 2M sats on each side; just makes the scripting easy with no small channels.
4) Once you get a rather large node you start having "gravity", meaning people open channels to you randomly. This is great b/c your node grows without having to invest more capital. I found most times people open the smallest possible channel -- so by now using a 5M sat make people who want to connect open a larger channel than they might have otherwise. The other reason is closing channels can be expensive (esp force closed ones), so I don't want small nodes who may shut down in a month to open to me. I want long term node partners.

If BTC crosses 100k, I'll probably drop my min channel size to 4M sats.

I have considered if I should let people connect for less who want to use my node as a connection for their lightning node wallet (as opposed to another routing node) -- maybe one of these days i'll do that.

I am actually thinking about opening a channel to your node. Is there any particular reason why you reject channels smaller than 5 million satoshi? Do you want to avoid frequent rebalancing this way?

my node also runs for about 4 months and has 17 channels. it forwarded 616 payments which collected 978 sats. i've set the basefee to 1 sat and no percentage fee for all channels. i played around with the fees in the beginning (thats why i earned more sats than the number of forwards)

i only added the summary plugin to the node for now

some weeks ago i wanted to rebalance a channel with a trusted peer (all of the balance is on my side, pushing over 50% and the peer sending the amount onchain back to me), but the keysend to the lnd node didn't work. i've sent christian decker an email with the logs and he said he will look into it to debug it, but i never heard back... it still isn't working

apart from that i didn't have any bigger problems, but i am also not using my lightning node for a lot of my own economic activity at the moment (only some small donations)

I've been meaning to do a larger write up of my experience once I hit 1M sats in fees.

I run the node Prince, https://1ml.com/node/02644f80b5d32ed9a9888690571159692a17d7ead7db2df5124a8e2a72a8447d30

I am currently ranked 8th on the BOS rankings (https://fulmo.org/bos-score.html) and 12th on lightning terminal rankings. I have 8-9 BTC on my node with about 110 channels (I have more incoming that outcoming liquidity, so that 8BTC is not my investment).

Top line numbers: Over the past 5 months have routed over 6700 payments worth over 1.7 billion sats earning me about 900k sats in routing fees; I've spent about 100k sats in on-chain fees and rebalancing fees. For those doing the math, that comes out to about 0.5% APR on my locked-up capital. Although my volume is steadily increasing and I may be able to get closer to 1% in the next year.

I have everything automated including fees and re-balancing.
-- I use lnd-charge for fees: https://github.com/accumulator/charge-lnd
-- I use c-otto's script for automatic re-balancing: https://c-otto.de/

Happy to answer more questions -- oh, and feel free to open a channel to me. The best way to get started is to open liquidity triangles with people (ie A->B; B->C; C->A). Happy to be a part of a couple here if there is interest. I do require a minimum of 5M sats for any incoming channels.

Yes, I know about the listforwards command outputting the exact reasons, but not for all failed payments. I also know about the description of each error. However, at least me, cannot understand by only reading the descriptions in that link. So, it's almost useless to me to read them. :p

I'm not sure we can check if a payment was split or not. What it seems to me is that, at least for some payments, there are several attempts and as the values changes by small amounts, I assume it tried different routes, hence, different fees.

You need to use lightning-cli listforwards command to see the exact error. Here you can find a list of errors with explanations.


Maybe the payment was split into multiple chunks and some of them failed? It would make sense to route one part through you again if it worked in some previous attempt.

No problem @_Rath.

I still have tons of failed transactions with no apparent reason. I have 5 pages of failed transactions at November 9th. It's 48 failed attempts, of which only 6 were LOCAL_FAILED. All the others are simply FAILED.
I have enough liquidity in the outgoing channels of these attempts, so I can't understand why, even the LOCAL_FAILED ones, failed.



There is no outgoing channel for these LOCAL_FAILED. Have no idea why.

Then, the other attempts all failed further down in the route. What I question in my mind is if when a payment fails, for a given amount and route, the next attempts will try to use exactly the same route or will it try new routes? If it tries the same route, what could be the reasons for the failures? And if it tries a different route for each attempt, is it normal to fail like 10 or 12 times before it finally gives up on trying to route this payment?