Pages:
Author

Topic: The Lightning Network FAQ - page 74. (Read 33805 times)

member
Activity: 194
Merit: 74
November 06, 2019, 02:21:56 PM
Hmm that's `lightningd.sqlite3`.. I don't think there is much sense in having another file which is updated at the same time `lightningd.sqlite3` is updated and that contains the same informations.
In this case I can have one lightningd.sqlite3 on my node and another one in a cloud (I used NFS storage on a cloud vps)
lightningd.sqlite3 stores not only channel balances but lots of other stuff. lightningd.sqlite3 changes too often

Quote
For safeguard I use:
- ..................  Grin
Let's imagine my hard drive failed or there was a power outage or something else happened and I am not sure if the lightningd.sqlite3 is intact (or I lost it). I restored onchain data from hsm_secret. What should I do to get my node up and running again? What about my channels and funds in them?

Quote
I think the change has to come from the protocol
Exactly!
sr. member
Activity: 279
Merit: 435
November 06, 2019, 05:57:02 AM
You can backup your onchain funds just by backing up the (potentially encrypted) `hsm_secret`.
Ok, that's clear.

Quote
You could backup your `lightningd.sqlite3` but it changes often (very often) and you should be very careful with restoring from backups.
Is there anything like lnd backup? An independent backup file, which changes every time when channels' balance changes.
Hmm that's `lightningd.sqlite3`.. I don't think there is much sense in having another file which is updated at the same time `lightningd.sqlite3` is updated and that contains the same informations.

For what it worth, efforts are being made for real-time database replication through plugins (some have been proposed) but it's harder than it seems and we have not yet found a satisfiable enough solution.
Are these plugins available somewhere?
You have a lot of plugins here, the plugin I had specifically in mind is this one by SimonVrouwe which you should not use because the database structure changed since.

What do you use?
In terms of plugins or safeguards ?
For plugins I use:
- summary
- rebalance
- drain
- probe
- reckless
- and soon the JIT routing one

For safeguard I use:
- ..................  Grin

No more seriously I have some hacky and nasty scripts for my database but I think the change has to come from the protocol. For example we now have (and I'm working on making C-lightning default to this) `option_static_remotekey` which allows you to recover your funds after the remote side unilateral close just with you hsm_secret (related but I made a mistake I have a tool for fund recovery which I plan to clean up and re-PR when we default to this).
member
Activity: 194
Merit: 74
November 05, 2019, 06:48:08 PM
You can backup your onchain funds just by backing up the (potentially encrypted) `hsm_secret`.
Ok, that's clear.

Quote
You could backup your `lightningd.sqlite3` but it changes often (very often) and you should be very careful with restoring from backups.
Is there anything like lnd backup? An independent backup file, which changes every time when channels' balance changes.

Quote
For what it worth, efforts are being made for real-time database replication through plugins (some have been proposed) but it's harder than it seems and we have not yet found a satisfiable enough solution.
Are these plugins available somewhere? What do you use?
sr. member
Activity: 279
Merit: 435
November 05, 2019, 05:59:29 PM
Hello Samuel,

I'm glad that you are pleased with C-lightning  Smiley

As far as I got it, c-lightning creates a wallet inside lightningd.sqlite3 file and there is a file hsm_secret, which is some kind of key, right?

In the `hsm_secret` is stored the seed used to derive all keys (channels and onchain ones). You can consider it as the HD wallet seed.
In the `lightningd.sqlite3` (also available as a postgresql backend) is notably stored the channels state (aside a lot of other things !).

So, the first question is how to backup onchain and offchain data? Before I start using it, I need to setup backups. It was quite clear with c-lightning, it gives me mnemonic key and has a backup file with all channel information. What about c-lightning?
You can backup your onchain funds just by backing up the (potentially encrypted) `hsm_secret`.
You could backup your `lightningd.sqlite3` but it changes often (very often) and you should be very careful with restoring from backups.

For what it worth, efforts are being made for real-time database replication through plugins (some have been proposed) but it's harder than it seems and we have not yet found a satisfiable enough solution.
member
Activity: 194
Merit: 74
November 05, 2019, 04:37:57 PM
Hi everybody!
Finally, I installed c-lightning, it uses about 22Mb of RAM and does not load CPU at all, everything looks good.
As far as I got it, c-lightning creates a wallet inside lightningd.sqlite3 file and there is a file hsm_secret, which is some kind of key, right?

So, the first question is how to backup onchain and offchain data? Before I start using it, I need to setup backups. It was quite clear with c-lightning, it gives me mnemonic key and has a backup file with all channel information. What about c-lightning?
sr. member
Activity: 279
Merit: 435
November 05, 2019, 05:44:11 AM
An arguable downside to c-lightning IMO is the lack of available builds (I imagine the devs want to make them reproducible before doing that).
Being in C (a lot of functionalities used are low level and system dependant, e.g. daemons interop) , I think we'll ever only have builds for POSIX systems.

Other downsides might be features... lnd probably has more features than c-lightning, but I think they're pretty close as far as the actual spec of the Lightning protocol goes (it seems c-lightning & lnd teams are coordinating development pretty closely along with the ACINQ team doing Zap/Eclair, although I know even less about their clients).
Hmmm I'm not sure ? What feature would you like to see on C-lightning ?

But if you look at the work on efficient routing algorithms, it seems as though c-lightning might get ahead of lnd soon, according to what their development priorities seem to be recently. Also, c-lightning has the plugin system that Darosior mentions, I don't know enough to say whether there are not some available plugins that do things lnd can't.
Development priorities are actually on dual funding and v1.1 spec (new invoices, AMP, etc..).
An example of a plugin doing JIT routing (i.e. rebalancing channels if needed while forwarding an HTLC) : https://github.com/lightningd/plugins/pull/66.
About LND pathfinding algorithm, I don't know much. I know however that they implemented scoring...
member
Activity: 194
Merit: 74
November 04, 2019, 06:31:06 PM
Thanks guys! I will give c-lightning a shot.
legendary
Activity: 3430
Merit: 3083
November 04, 2019, 05:59:16 PM
An arguable downside to c-lightning IMO is the lack of available builds (I imagine the devs want to make them reproducible before doing that). That's what's strange about lnd; they're releasing a product that's more like an alpha release than a beta release, yet they're distributing binaries of that alpha-ish product. As I said, I think golang makes this a little too easy to do, and so the LL people are running before they can walk.

Other downsides might be features... lnd probably has more features than c-lightning, but I think they're pretty close as far as the actual spec of the Lightning protocol goes (it seems c-lightning & lnd teams are coordinating development pretty closely along with the ACINQ team doing Zap/Eclair, although I know even less about their clients). But if you look at the work on efficient routing algorithms, it seems as though c-lightning might get ahead of lnd soon, according to what their development priorities seem to be recently. Also, c-lightning has the plugin system that Darosior mentions, I don't know enough to say whether there are not some available plugins that do things lnd can't.

overall, I get the impression c-lightning is in a better state of production-readiness than lnd, despite any lack of extra features (and I might be getting that part wrong).
sr. member
Activity: 279
Merit: 435
November 04, 2019, 04:30:52 PM
if you're running multiple nodes I would guess resource usage per-daemon could be important to you.
You are damn right! 400Mb of RAM is a lot for a single lnd process with no channels at all.

Quote
FWIW, I'm running c-lightning on a Raspberry Pi 3B+ (1GB RAM) with a handful of channels, so no problems on low resource devices.
Tell me about c-lightning service, how reliable and stable it is? How much memory does it need?

PS I have spent so much time on lnd, automation, scripting and so on, and now I came across the bug that pushes me to use something else. Sad
*Shilling mode on*
How reliable ==> releases are quite stable, but it depends of how much you tweak it with plugins  Wink
How much memory ==> just checked, with around 10-15 channels and more than 300 connections (yeah I'm doing test atm..) it does not use more than 10% of the 2GB of my node. Idle mode, might of course be higher when computing routes (I think that's the more usage incentive thing).

Anyway C-lightning aims at being as minimal (all non-vital things as plugins) and economic as possible : for example Rusty did a lot of performance improvements for last releases (tests are done on a RPi to use the less powerful hardware as possible), and a lot of bandwidth improvements for this one (v0.7.3) (gossip).
*Shilling mode off*
member
Activity: 194
Merit: 74
November 04, 2019, 03:03:56 PM
if you're running multiple nodes I would guess resource usage per-daemon could be important to you.
You are damn right! 400Mb of RAM is a lot for a single lnd process with no channels at all.

Quote
FWIW, I'm running c-lightning on a Raspberry Pi 3B+ (1GB RAM) with a handful of channels, so no problems on low resource devices.
Tell me about c-lightning service, how reliable and stable it is? How much memory does it need?

PS I have spent so much time on lnd, automation, scripting and so on, and now I came across the bug that pushes me to use something else. Sad
legendary
Activity: 3430
Merit: 3083
November 04, 2019, 02:52:31 PM
After the graph is updated, an lnd service is supposed to go into idle mode with low cpu usage. In reality, this happens but not for a long time. At first, CPU usage stays low but only for a couple of hours, then all services one by one started loading CPU heavily with no reason. The lnd services will not stop doing that until I restart them, then everything happens all over again. One interesting thing is that the bitcoind service does not use almost any CPU resourses; it means that lnd services do not use bitcoind, when they load CPU. I could not find anything unusual in the lnd logs, everything seems normal.

as you've seen, this is a known bug in lnd, for which there is currently no fix. I don't know if this is affecting arm64 builds (RasPi nodes are arm64), but if so this is pretty bad news, as most people are opt for lnd because the daemon is provided as a pre-built package (LL have been good in that respect, there are alot of OS's and platforms supported with builds and prebuilt binaries, don't know if that's because golang makes that easy, I suspect so)

try using c-lightning as the fix Cheesy lnd has other resource usage issues right now (uses alot of RAM), if you're running multiple nodes I would guess resource usage per-daemon could be important to you. FWIW, I'm running c-lightning on a Raspberry Pi 3B+ (1GB RAM) with a handful of channels, so no problems on low resource devices. That's not to say that c-lightning is perfect (they've been wrangling bugs too), but resource usage has not been an issue. You'll need to sync the channel map as with lnd, but maybe you'll appreciate that you can use PostgreSql with the db seeing as you're doing something that sounds demanding Smiley
member
Activity: 194
Merit: 74
November 04, 2019, 02:21:22 PM
Hi, I have been working on a project where I need to run lots of lnd services on a single machine (physical or virtual, does not matter). In theory, it seems easy, I created a directory structure for each node with its own conf file, tcp ports and so on. It was not difficult and I started 16 nodes for a test on a single VM machine. (CentOS 7)
After starting them, each node begins to update network graph (file channel.db in /data/graph/mainnet); this process is quite long and CPU consuming. After the graph is updated, an lnd service is supposed to go into idle mode with low cpu usage. In reality, this happens but not for a long time. At first, CPU usage stays low but only for a couple of hours, then all services one by one started loading CPU heavily with no reason. The lnd services will not stop doing that until I restart them, then everything happens all over again. One interesting thing is that the bitcoind service does not use almost any CPU resourses; it means that lnd services do not use bitcoind, when they load CPU. I could not find anything unusual in the lnd logs, everything seems normal.
I don't understand what's going on and ho to fix it. My host's hardware should be enought for the purpose (Core i3-3.8Ghz/16Gb RAM/SSD) Maybe somenody has already come accross with this issue. I fould the similar issue on the lnd's github page but there is no solution there.
https://github.com/lightningnetwork/lnd/issues/3370
I am interested to listen to everyone who has any thoughts about it or solutions how to fix it.
legendary
Activity: 3430
Merit: 3083
October 24, 2019, 07:29:34 PM
As far as I understand the LN architecture, that "private network" would use the same channels (i.e. the same multisig channel opening transactions) than the "real LN", only that the "communication layer" would change, so no additional on-chain TX would be necessary which is the important feature we are seeking ... (correct me if I'm wrong)

yes, you commit the channel states onchain, but do not connect to nodes on the "main" Lightning network (and so you don't announce your channels on the main network either). Of course, the opening tx's are can only be created using the public internet, but then your alternative Lightning network can be set up on a completely private network (and that comes with all the liquidity and trust trade-offs as mentioned before)

Connecting to the main Lightning network today is like a trip back to the 1990's; you look up a list of LN node IP addresses on a listing site, connect to one, it tells you the current (entire) channel map, then you start making channels at your discretion using the current map. There's absolutely nothing stopping you getting together with a bunch of other people and starting an alternative LN network from scratch, you just need LN nodes and each others IP addresses.
legendary
Activity: 3430
Merit: 3083
October 24, 2019, 04:35:08 PM
How was it even possible to lose bitcoin using this network?

they didn't know the risks, and screwed up trying to close a channel using an outdated transaction (i.e. an old channel state)


This user reported they lost 4btc using lightning. https://bitcointalksearch.org/topic/m.52855065

yep, that was too risky considering they didn't close their channel competently. Every time you send a large amount of money anywhere, on any system, you should be triple-sure you know what you're doing.


Wasnt it created inorder to not being susceptible for such things to happen? Undecided

nope.

You're probably thinking of the eltoo form of Lightning, which avoids this problem. You cannot make that mistake using eltoo, but it's a long way off (new opcode required, and the details are still being debated, in an already long debate)
legendary
Activity: 3906
Merit: 6249
Decentralization Maximalist
October 24, 2019, 04:32:50 PM
set up that private Lightning network, and keep it very exclusive to people within a given area. The crucial part would be: don't use it, except in outage situations. Use uptime of the general internet to store BTC in the private network, but use the regular Lightning network for everyday payments while internet remains up.
Thanks for the suggestion. If a mesh network can be established in the region, then this can be a good solution, but it would also be more suitable for urban areas (in most places of Venezuela, however, it should work). As far as I understand the LN architecture, that "private network" would use the same channels (i.e. the same multisig channel opening transactions) than the "real LN", only that the "communication layer" would change, so no additional on-chain TX would be necessary which is the important feature we are seeking ... (correct me if I'm wrong)

@rdbase: Seems to be the same case like the one Khaos77 mentioned, the user simply broadcasted the wrong transaction to close the channel. I would strongly favour to not discuss this topic here as it's the user's fault (it's possible however that the LN software could have been designed to prevent this).
legendary
Activity: 2828
Merit: 1497
Join the world-leading crypto sportsbook NOW!
October 24, 2019, 03:58:22 PM
How was it even possible to lose bitcoin using this network?
This user reported they lost 4btc using lightning. https://bitcointalksearch.org/topic/m.52855065
Wasnt it created inorder to not being susceptible for such things to happen? Undecided
member
Activity: 194
Merit: 74
October 24, 2019, 03:39:33 PM
Actually, you don't have to run a full Bitcoin node if you are using LND! LND supports neutrino which lets you interact with the blockchain using third-party nodes. Keep in mind that it's still experimental but it's been vastly improved since it was released. You might give it a try. Thanks for clarifying your previous message.
I thought about it, I really did, but

lnd.conf

[neutrino]
; Connect only to the specified peers at startup. This creates a persistent
; connection to a target peer. This is recommended as there aren't many
; neutrino compliant full nodes
on the test network yet.
; neutrino.connect=


So, LND needs a full node and I can specify the only one of them. What happens to my LND node if that neutrino node stops working? It would be much more convenient if I did not need to specify a node. I wish it could find a compliant full node on the network itself and connect to it. Smiley Like torrent clients do.

One more question about tor nodes. I often see in logs that my node fails to connect to nodes with tor addresses (*.onion). Do I need to choose between Tor and the regular Internet or I can install tor and connect to both of them?
legendary
Activity: 1876
Merit: 3139
October 24, 2019, 12:33:24 PM
I'm sorry, what is DLP?

Data Loss Protection, it was mentioned in the LND v0.6 changelog which you quoted.

PS I would have gladly put everything on a cloud if I had not needed to store the entire blockchain there. Keeping 300Gb on a cloud server is quite expensive.

Actually, you don't have to run a full Bitcoin node if you are using LND! LND supports neutrino which lets you interact with the blockchain using third-party nodes. Keep in mind that it's still experimental but it's been vastly improved since it was released. You might give it a try. Thanks for clarifying your previous message.
member
Activity: 194
Merit: 74
October 24, 2019, 12:17:57 PM
Once the DLP is initiated, the remote node sends the latest commitment transaction and forcefully closes the channel.
I'm sorry, what is DLP?

Quote
The only problem with your proposal is that your NFS backup might become invalid anytime unless you stop participating in the payment routing. How often would you backup the file? Every time it was modified?
You did not get the idea; let me clarify.
I mounted a remote directory to my local directory /mnt/nfsstorage. So, everything I put in there ends up on my NFS server (in amazon cloud). Then I added the backup file path in the lnd.conf file (backupfilepath=/mnt/nfsstorage/lndbackup/channel.backup). The lnd service keeps channel.backup updated constantly. Thus, I do not need to copy anything; the backup file will be updated by lnd directly on the cloud server.

The only problem I might have is with connection to the cloud server. However, the NFS protocol allows to automatically reconnect to a server if the previous connection was interrupted (that's what manual says Smiley.
I installed the NFS server yesterday, It seems to be working fine.

PS I would have gladly put everything on a cloud if I had not needed to store the entire blockchain there. Keeping 300Gb on a cloud server is quite expensive.
legendary
Activity: 1876
Merit: 3139
October 24, 2019, 12:51:35 AM
-snip

I totally forgot about this method, sorry! Since SCBs make use of a feature built into the protocol, they should be reliable. Personally, I have never had a chance to restore my node from such a backup. Once the DLP is initiated, the remote node sends the latest commitment transaction and forcefully closes the channel. The only problem with your proposal is that your NFS backup might become invalid anytime unless you stop participating in the payment routing. How often would you backup the file? Every time it was modified?
Pages:
Jump to: