[Guide] C-Lightning backup guide | Bitcointalksearch.org

n0nce

hero member

Activity: 924

Merit: 5943

not your keys, not your coins!

Quote from: Jason Brendon on February 01, 2023, 01:30:31 AM

great. thank you two for the explanation.
One other question, considering that i don't have many channels open. Is there an intensive writing to the disk? the reason i am asking is i am considering buying a tf card or a usb. If it requires constant and high density writing, i need to buy a good tf card. If not, i will just buy a cheaper one..

The most R/W will probably be regular OS operations which you cannot really prevent. If your operating system runs on such a type of storage, I'd recommend making sure it is a good quality product.
The backup drive is probably not accessed too frequently in your case, so feel free to use what you have on hand. If you really want to make sure, you can check in RTL how many transactions you typically route per day / week / month, but I'd guess it's not a lot.

Quote from: bnbstorm on February 01, 2023, 07:40:46 AM

This seems interesting. I would try to use it for my next project. Is this guide still up to date?

It is! Feel free to try it on the latest release and post a reply if there are any issues.

bnbstorm

copper member

Activity: 126

Merit: 35

This seems interesting. I would try to use it for my next project. Is this guide still up to date?

Jason Brendon

member

Activity: 162

Merit: 65

Quote from: n0nce on January 31, 2023, 10:03:31 AM

Quote from: Jason Brendon on January 31, 2023, 02:35:25 AM

let's say I have just 5 channels open to the others. and let's say in the next 2 days, I am not receiving nor sending anything. So, the channel status stays the same, right?

Not entirely correct. Even if you are not sending / receiving, someone else may use your node to route their payment, thus changing your channel states.
The recommended backup methods all update the backup automatically whenever a channel state happens.

Your manual method would not work, if you only update backups whenever you make payments. You'd have to keep track of every payment going through your node, at which point you're simply coding the backup plugin all over again. Wink

great. thank you two for the explanation.
One other question, considering that i don't have many channels open. Is there an intensive writing to the disk? the reason i am asking is i am considering buying a tf card or a usb. If it requires constant and high density writing, i need to buy a good tf card. If not, i will just buy a cheaper one..

n0nce

hero member

Activity: 924

Merit: 5943

not your keys, not your coins!

Quote from: Jason Brendon on January 31, 2023, 02:35:25 AM

let's say I have just 5 channels open to the others. and let's say in the next 2 days, I am not receiving nor sending anything. So, the channel status stays the same, right?

Not entirely correct. Even if you are not sending / receiving, someone else may use your node to route their payment, thus changing your channel states.
The recommended backup methods all update the backup automatically whenever a channel state happens.

Your manual method would not work, if you only update backups whenever you make payments. You'd have to keep track of every payment going through your node, at which point you're simply coding the backup plugin all over again. Wink

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

Quote from: Jason Brendon on January 31, 2023, 02:35:25 AM

[...]

The lightningd.sqlite3 file is updated whenever there is a change in the state of the channel (such as if you send a payment, receive a payment, or route a payment). However, if you're trying to backup your wallet, this way is not recommended by the developers. It is nowhere stated that copying the database file elsewhere is safe. The backup page describes that if you want to backup your wallet, use either the --wallet parameter properly, or the backup plugin.

Jason Brendon

member

Activity: 162

Merit: 65

hi op,

one question that often stays on top of my mind is that:

is the sq file constantly writing like every minute or second?
let's say I have just 5 channels open to the others. and let's say in the next 2 days, I am not receiving nor sending anything. So, the channel status stays the same, right?

my question is that: if i take the old sq file (2 days ago) and th hsm secret, will i be able to restore the node and not be punished for being caught cheating? (the sq file is 2 days old.)

need people with a confirmed answer. Thanks a lot!

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

Quote from: Jason Brendon on September 19, 2022, 08:56:21 PM

Well, thanks, buddy. yes tBtTC, I see veterans do experiments with testnet all the time. Me haven't tried anything like that. because i am afraid of ending up messing things up....

No problem. Just ask some in that faucet-thread.

Jason Brendon

member

Activity: 162

Merit: 65

Quote from: BlackHatCoiner on September 11, 2022, 06:34:57 AM

From lightning.readthedocs.io:

Quote

Recovery of the hsm_secret is sufficient to recover any onchain funds. Recovery of the hsm_secret is necessary, but insufficient, to recover any in-channel funds. To recover in-channel funds, you need to use one or more of the other backup strategies below.

To backup both off-chain and on-chain you need both the hsm_secret and the lightningd.sqlite3.

I'd advice Jason Brendon to test recovering a Lightning node (on-chain & off-chain funds) with fake money first (tBTC). That way, you don't need to be afraid of losing anything. That's why testnet exists. If you don't have tBTC, acquire here: https://bitcointalksearch.org/topic/m.57912622

Well, thanks, buddy. yes tBtTC, I see veterans do experiments with testnet all the time. Me haven't tried anything like that. because i am afraid of ending up messing things up....

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

From lightning.readthedocs.io:

Quote

Recovery of the hsm_secret is sufficient to recover any onchain funds. Recovery of the hsm_secret is necessary, but insufficient, to recover any in-channel funds. To recover in-channel funds, you need to use one or more of the other backup strategies below.

To backup both off-chain and on-chain you need both the hsm_secret and the lightningd.sqlite3.

I'd advice Jason Brendon to test recovering a Lightning node (on-chain & off-chain funds) with fake money first (tBTC). That way, you don't need to be afraid of losing anything. That's why testnet exists. If you don't have tBTC, acquire here: https://bitcointalksearch.org/topic/m.57912622

n0nce

hero member

Activity: 924

Merit: 5943

not your keys, not your coins!

Quote from: Jason Brendon on September 09, 2022, 09:54:03 PM

Quote from: n0nce on February 02, 2022, 11:47:58 PM

[6] If you get no errors starting up the service or anything, you're good! For good measure, you could send yourself some sats from a mobile wallet, restart the service with sudo service lightningd restart and even restore the node from the backup, then check if everything's fine.

Thanks mate. I mean how can i test a restore? By what? by deleting the hsm_secret, will i then be able to restore the node? Thanks man.

Yes; first of all make sure you don't have a lot of funds on it.
Then, shut the node down and delete everything. You can even wipe the machine.

Or spin up a fresh Virtual Machine, or use a different device - you get the idea. The 'real' node must be offline for good, though.
Then restore your backup as explained in the guide and check that your channels are still there.

Now you know you have a good backup.
Again: it's not just about hsm_secret, if you don't have the channel state database, the channels are gone.

Jason Brendon

member

Activity: 162

Merit: 65

Quote from: n0nce on September 09, 2022, 05:55:19 PM

Quote from: Jason Brendon on September 08, 2022, 09:37:21 PM

Quote from: n0nce on February 02, 2022, 11:47:58 PM

[6] If you get no errors starting up the service or anything, you're good! For good measure, you could send yourself some sats from a mobile wallet, restart the service with sudo service lightningd restart and even restore the node from the backup, then check if everything's fine.

Thanks mate. I mean how can i test a restore? By what? by deleting the hsm_secret, will i then be able to restore the node? Thanks man.

n0nce

hero member

Activity: 924

Merit: 5943

not your keys, not your coins!

Quote from: Jason Brendon on September 08, 2022, 09:37:21 PM

Quote from: n0nce on February 02, 2022, 11:47:58 PM

[6] If you get no errors starting up the service or anything, you're good! For good measure, you could send yourself some sats from a mobile wallet, restart the service with sudo service lightningd restart and even restore the node from the backup, then check if everything's fine.

Question:

1. Is restoring the lightning node from the same device or even an entirely different device going to cause my existing channels to close forcibly?

No, that's the idea of the backups. While your old / broken node is offline, the channel will just be offline. The counterparty could force-close, if they don't believe you're going to come back online, though.
If they don't, all the channels should be back just as they were before.

Quote from: Jason Brendon on September 08, 2022, 09:37:21 PM

2. How to restore a c-lightning node? Do i simply remove the hsm_secret file and that is called a restore?
Thanks guys.

As the guide says:

Quote from: https://lightning.readthedocs.io/BACKUP.html#sqlite3-wallet-main-backup-and-remote-nfs-mount

To recover, simply get all the backup database files. Note that SQLITE3 will sometimes create a -journal or -wal file, which is necessary to ensure correct recovery of the backup; you need to copy those too, with corresponding renames if you use a different filename for the backup database, e.g. if you named the backup backup.sqlite3 and when you recover you find backup.sqlite3 and backup.sqlite3-journal files, you rename backup.sqlite3 to lightningd.sqlite3 and backup.sqlite3-journal to lightningd.sqlite3-journal. Note that the -journal or -wal file may or may not exist, but if they do, you must recover them as well (there can be an -shm file as well in WAL mode, but it is unnecessary; it is only used by SQLITE3 as a hack for portable shared memory, and contains no useful data; SQLITE3 will ignore its contents always). It is recommended that you use the same filename for both main and backup databases (just on different directories), and put the backup in its own directory, so that you can just recover all the files in that directory without worrying about missing any needed files or correctly renaming.

Jason Brendon

member

Activity: 162

Merit: 65

Quote from: n0nce on February 02, 2022, 11:47:58 PM

[6] If you get no errors starting up the service or anything, you're good! For good measure, you could send yourself some sats from a mobile wallet, restart the service with sudo service lightningd restart and even restore the node from the backup, then check if everything's fine.

Question:

1. Is restoring the lightning node from the same device or even an entirely different device going to cause my existing channels to close forcibly?
2. How to restore a c-lightning node? Do i simply remove the hsm_secret file and that is called a restore?

Thanks guys.

n0nce

hero member

Activity: 924

Merit: 5943

not your keys, not your coins!

Quote from: xyris on May 06, 2022, 09:31:17 AM

Indeed, there is no file "requirements.txt" anywhere, I checked in directories backup and plugins

What is going wrong here? Please help, thanks!

Quote from: BlackHatCoiner on August 11, 2022, 08:31:01 AM

This guide must be outdated. Once I run the backup-cli command, I get:

Code:

Traceback (most recent call last):
  File "backup/backup-cli", line 2, in 
    from backends import get_backend
  File "/home/bitcoin/lightning/plugins/backup/backends.py", line 5, in 
    from backend import Backend
  File "/home/bitcoin/lightning/plugins/backup/backend.py", line 7, in 
    from tqdm import tqdm
ModuleNotFoundError: No module named 'tqdm'

Quote from: ABCbits on August 13, 2022, 04:08:53 AM

Quote from: n0nce on August 12, 2022, 04:12:21 PM

But I remember that I did minimal modification to my original install / workflow which is described here, to get it working again. It could be as simple as: pip install tqdm after Step 2. @BlackHatCoiner, this is worth trying..

I'd advice to install exact version of tqdm (pip install tqdm==4.62.3) on poerty.lock file to avoid possible package conflict.

Hey everybody! Guide fixed. At last. Replaced the backup plugin entirely in favor of the new built-in backup feature; this way we skip all this poetry packet manager stuff and the whole setup process is much cleaner and simpler to do.
I did spin up a whole new VM in which I installed Core Lightning as per my own install guide as well as following the steps I described here for backing up, so it should work, but if anyone's got any issues don't hesitate to ask.

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Quote from: n0nce on August 12, 2022, 04:12:21 PM

Quote from: ?? on ??

--snip--

I just checked their repository and found out the installation guide is also outdated[1]. The guide mention you use pip to install requirements, but today they use Poetry[2] which is different library package manager.

[1] https://github.com/lightningd/plugins#additional-dependencies
[2] https://github.com/lightningd/plugins/blob/master/backup/poetry.lock

Actually, within the backup plugin's README file, it tells you to just use poetry install. And to then invoke ./backup-cli using poetry run ./backup-cli ....

Thanks for the correction, i didn't notice they put README.md file on their directory. Even so, i still find wording on main README.md file means you can install dependency for all plugin using pip

Quote from: n0nce on August 12, 2022, 04:12:21 PM

But I remember that I did minimal modification to my original install / workflow which is described here, to get it working again. It could be as simple as: pip install tqdm after Step 2. @BlackHatCoiner, this is worth trying..

I'd advice to install exact version of tqdm (pip install tqdm==4.62.3) on poerty.lock file to avoid possible package conflict.

n0nce

hero member

Activity: 924

Merit: 5943

not your keys, not your coins!

Quote from: ?? on ??

Quote from: n0nce on August 11, 2022, 05:31:04 PM

Quote from: BlackHatCoiner on August 11, 2022, 08:31:01 AM

This guide must be outdated. Once I run the backup-cli command, I get:

Code:

Traceback (most recent call last):
  File "backup/backup-cli", line 2, in 
    from backends import get_backend
  File "/home/bitcoin/lightning/plugins/backup/backends.py", line 5, in 
    from backend import Backend
  File "/home/bitcoin/lightning/plugins/backup/backend.py", line 7, in 
    from tqdm import tqdm
ModuleNotFoundError: No module named 'tqdm'

It is! I will add a red note to the original post, right now and update the guide as soon as possible. Sorry for the inconvenience!

I just checked their repository and found out the installation guide is also outdated[1]. The guide mention you use pip to install requirements, but today they use Poetry[2] which is different library package manager.

[1] https://github.com/lightningd/plugins#additional-dependencies
[2] https://github.com/lightningd/plugins/blob/master/backup/poetry.lock

Actually, within the backup plugin's README file, it tells you to just use poetry install. And to then invoke ./backup-cli using poetry run ./backup-cli ....

But I remember that I did minimal modification to my original install / workflow which is described here, to get it working again. It could be as simple as: pip install tqdm after Step 2. @BlackHatCoiner, this is worth trying..

I will check my history file once I find the time and have SSH access to my node (the two are not always synchronous.. Roll Eyes

).

n0nce

hero member

Activity: 924

Merit: 5943

not your keys, not your coins!

Quote from: BlackHatCoiner on August 11, 2022, 08:31:01 AM

This guide must be outdated. Once I run the backup-cli command, I get:

Code:

Traceback (most recent call last):
  File "backup/backup-cli", line 2, in 
    from backends import get_backend
  File "/home/bitcoin/lightning/plugins/backup/backends.py", line 5, in 
    from backend import Backend
  File "/home/bitcoin/lightning/plugins/backup/backend.py", line 7, in 
    from tqdm import tqdm
ModuleNotFoundError: No module named 'tqdm'

It is! I will add a red note to the original post, right now and update the guide as soon as possible. Sorry for the inconvenience!

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

This guide must be outdated. Once I run the backup-cli command, I get:

Code:

Traceback (most recent call last):
  File "backup/backup-cli", line 2, in 
    from backends import get_backend
  File "/home/bitcoin/lightning/plugins/backup/backends.py", line 5, in 
    from backend import Backend
  File "/home/bitcoin/lightning/plugins/backup/backend.py", line 7, in 
    from tqdm import tqdm
ModuleNotFoundError: No module named 'tqdm'

n0nce

hero member

Activity: 924

Merit: 5943

not your keys, not your coins!

Quote from: xyris on May 06, 2022, 09:31:17 AM

~

Oh, indeed, thanks for bringing this up! It seems the file has disappeared in the last two commits (after almost a year of silence): https://github.com/lightningd/plugins/commit/42f88ce24ab0469597bed3e90754a880ac9f956d

Now somehow, they switched to poetry instead of using Python's built-in pip. I generally prefer to stick with the popular, well-known and 'tried and tested' tools. You could certainly check-out the repo at an earlier point in time (before the two commits of last week) or try doing it with poetry.

I will update my guide as soon as possible, but I can't promise that it'll happen today.

xyris

newbie

Activity: 1

Merit: 2

Hi, thanks, this is really helpful. I'm a (non-technical) Raspiblitz user and I don't have enough know-how to do a c-lightning backup just with the available (too technical) documentation I could find online.

Unfortunately I stumbled over this command:

Code:

pip install -r backup/requirements.tx

where I got this error:

Code:

ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'

Indeed, there is no file "requirements.txt" anywhere, I checked in directories backup and plugins

What is going wrong here? Please help, thanks!

Topic: [Guide] C-Lightning backup guide (Read 613 times)