How usable is CLN currently (vs LND)?

Jodmapht

member

Activity: 74

Merit: 21

Quote from: DaveF on December 19, 2022, 01:19:57 PM

Quote from: Jodmapht on December 19, 2022, 08:18:36 AM

Quote from: DaveF on December 16, 2022, 06:58:41 AM

Quote from: ?? on ??

Ok I looked at the details using htop, and what I see is that cln's bcli is writing to the disk at 10 MB/s constantly...

To me that looks like it's still trying to do something with the initial sync, but it really should not be taking this kind of time. If it's still doing it now 15+ hours from your last post and a long time after your initial seeing of the issue. I would open an issue on github. It is obviously working since you can see the channel you created in 1ml so something else is going on.

Sorry I could not help more, but this looks more like it's not a configuration issue but something withing CLN / your setup.

-Dave

Thanks. It ended up settling down after another day. I wish I understood more what it was doing and why it was not performing these operations in memory...

Either some setting has to be tweaked to get it to use ram instead of the drive OR it was just doing some table manipulation in the database and its just reading & writing until it's done.
Glad it's working.
Make sure to do regular backups if you have any funds in there, RPis are good and all, but you still do not have RAID in case of a disk issue.
Been there, done that.

-Dave

Thanks,

Do you think using PostgreSQL instead of SQLite could have prevented this?

Yes I got some Lexar USB sticks with a metal enclosure I will use for backup...
Thanks!

DaveF

legendary

Activity: 3500

Merit: 6320

Crypto Swap Exchange

Quote from: Jodmapht on December 19, 2022, 08:18:36 AM

Quote from: DaveF on December 16, 2022, 06:58:41 AM

Quote from: ?? on ??

Ok I looked at the details using htop, and what I see is that cln's bcli is writing to the disk at 10 MB/s constantly...

To me that looks like it's still trying to do something with the initial sync, but it really should not be taking this kind of time. If it's still doing it now 15+ hours from your last post and a long time after your initial seeing of the issue. I would open an issue on github. It is obviously working since you can see the channel you created in 1ml so something else is going on.

Sorry I could not help more, but this looks more like it's not a configuration issue but something withing CLN / your setup.

-Dave

Thanks. It ended up settling down after another day. I wish I understood more what it was doing and why it was not performing these operations in memory...

Either some setting has to be tweaked to get it to use ram instead of the drive OR it was just doing some table manipulation in the database and its just reading & writing until it's done.
Glad it's working.
Make sure to do regular backups if you have any funds in there, RPis are good and all, but you still do not have RAID in case of a disk issue.
Been there, done that.

-Dave

Jodmapht

member

Activity: 74

Merit: 21

Quote from: DaveF on December 16, 2022, 06:58:41 AM

Quote from: ?? on ??

Ok I looked at the details using htop, and what I see is that cln's bcli is writing to the disk at 10 MB/s constantly...

To me that looks like it's still trying to do something with the initial sync, but it really should not be taking this kind of time. If it's still doing it now 15+ hours from your last post and a long time after your initial seeing of the issue. I would open an issue on github. It is obviously working since you can see the channel you created in 1ml so something else is going on.

Sorry I could not help more, but this looks more like it's not a configuration issue but something withing CLN / your setup.

-Dave

Thanks. It ended up settling down after another day. I wish I understood more what it was doing and why it was not performing these operations in memory...

runam0k

legendary

Activity: 1092

Merit: 1001

Touchdown

Quote from: HeRetiK on December 14, 2022, 09:57:21 AM

Actually I've been using the rebalance plugin, rebalancing individual channels manually:
https://github.com/lightningd/plugins/tree/master/rebalance

Running a larger node with good liquidity will definitely require some automation though and the LND ecosystem seems to have a lower learning curve in that regard.

CLBOSS automates things and generally works well, though it's purpose is more about managing fees than purely rebalancing https://github.com/ZmnSCPxj/clboss

DaveF

legendary

Activity: 3500

Merit: 6320

Crypto Swap Exchange

Quote from: ?? on ??

Ok I looked at the details using htop, and what I see is that cln's bcli is writing to the disk at 10 MB/s constantly...

To me that looks like it's still trying to do something with the initial sync, but it really should not be taking this kind of time. If it's still doing it now 15+ hours from your last post and a long time after your initial seeing of the issue. I would open an issue on github. It is obviously working since you can see the channel you created in 1ml so something else is going on.

Sorry I could not help more, but this looks more like it's not a configuration issue but something withing CLN / your setup.

-Dave

Jodmapht

member

Activity: 74

Merit: 21

Quote from: DaveF on December 15, 2022, 12:46:10 PM

Quote from: Jodmapht on December 15, 2022, 10:15:06 AM

These seem to all apply to LND and not CLN though, correct? I can open an issue on the CLN's repository.

Sorry yeah, I am remote into my desktop that is logged into the forum and the last part "There might be something similar in CLightning" did not make it.

Quote from: Jodmapht on December 15, 2022, 11:29:11 AM

Ok, so it looks like this could be all normal and due to the fact that my LN node is new. It is supposed to calm down...

At least for the CLN part, I'm not sure why core is using anything. Mine only has activity when blocks are found and even then not much.

How are you powering the RPi? I have seen some 5V adapters cause issues.
This is one of my favorites.... 5% over voltage with no load, add a load and I can get it to 5.5V / 10% over voltage, then bad things^tm happen.

-Dave

Do you recall having your service using a significant amount of CPU and IO for a few days initially? When comparing to downloading and verifying the blockchain, CLN does not seem to be downloading much data or using much disk space, but it saturates the SSD IO and puts significant load on the CPU. My whole bitcoin directory for CLN currently uses only 82M of disk space. I see about 20 KB/s or less of up or down traffic for the node. lightningd at 36% and bitcoind at 110% CPU core usage.

I have been using the official 15W power supply. I have not had any obvious issue with it. What do you have exactly? I have not done any overclocking for my node, but as I mentioned got UASP and trim working for the SSD, and I use it for the OS as well, I do not use any SD card, I activated noatime. I would expect IO performance to be quite optimized for my RPI.

Maybe I should bump up bitcoind's dbcache temporarily to help CLN syncing faster with less IO operations? I had dbcache configured at 7368 MB when I downloaded and verified the blockchain, but since I brought it down to 300 MB... Maybe it is bitcoind that is currently creating the IO strain and not lightningd...

I will try increasing dbcache to 5000

Ok I looked at the details using htop, and what I see is that cln's bcli is writing to the disk at 10 MB/s constantly...

[moderator's note: consecutive posts merged]

DaveF

legendary

Activity: 3500

Merit: 6320

Crypto Swap Exchange

Quote from: Jodmapht on December 15, 2022, 10:15:06 AM

These seem to all apply to LND and not CLN though, correct? I can open an issue on the CLN's repository.

Sorry yeah, I am remote into my desktop that is logged into the forum and the last part "There might be something similar in CLightning" did not make it.

Quote from: Jodmapht on December 15, 2022, 11:29:11 AM

Ok, so it looks like this could be all normal and due to the fact that my LN node is new. It is supposed to calm down...

At least for the CLN part, I'm not sure why core is using anything. Mine only has activity when blocks are found and even then not much.

How are you powering the RPi? I have seen some 5V adapters cause issues.
This is one of my favorites.... 5% over voltage with no load, add a load and I can get it to 5.5V / 10% over voltage, then bad things^tm happen.

-Dave

Jodmapht

member

Activity: 74

Merit: 21

Ok, so it looks like this could be all normal and due to the fact that my LN node is new. It is supposed to calm down...

Jodmapht

member

Activity: 74

Merit: 21

Quote from: DaveF on December 15, 2022, 10:05:14 AM

Found this for an older version but from reading the issue may still exist.

https://github.com/lightningnetwork/lnd/issues/6837

https://github.com/Bitcoinite/RaspiBolt/commit/cc5374ff073c499f912d0321673283c9ab15d81b

Although there are a lot of helpful people here, looking at github to see if others have a similar issue can save you some time.
I personally don't contribute a lot to things there, but I do read a lot to find answers and comment if it works.

Also, see what db.bolt.auto-compact= is set to.
IIRC you need to set it to 1 to compact / clean the db.

-Dave

These seem to all apply to LND and not CLN though, correct? I can open an issue on the CLN's repository.

DaveF

legendary

Activity: 3500

Merit: 6320

Crypto Swap Exchange

Found this for an older version but from reading the issue may still exist.

https://github.com/lightningnetwork/lnd/issues/6837

https://github.com/Bitcoinite/RaspiBolt/commit/cc5374ff073c499f912d0321673283c9ab15d81b

Although there are a lot of helpful people here, looking at github to see if others have a similar issue can save you some time.
I personally don't contribute a lot to things there, but I do read a lot to find answers and comment if it works.

Also, see what db.bolt.auto-compact= is set to.
IIRC you need to set it to 1 to compact / clean the db.

-Dave

Jodmapht

member

Activity: 74

Merit: 21

Quote from: Jodmapht on December 15, 2022, 12:37:20 AM

Quote from: DaveF on December 14, 2022, 08:57:04 PM

Something is definitely not right. You should not be using that much CPU for either of them.
Stupid as it may be, can you reboot and see if it clears up. Could be something else hung, but that is a guess.

If you look at 1ml.com do you see the channel open?

I'm exhausted, but I'm sure someone else can chime in with other ideas.

-Dave

Code:

top - 00:31:48 up 11 min, 6 users, load average: 1.04, 0.68, 0.48
Tasks: 175 total, 1 running, 174 sleeping, 0 stopped, 0 zombie
%Cpu(s): 29.0 us, 12.4 sy, 0.0 ni, 57.1 id, 0.9 wa, 0.0 hi, 0.7 si, 0.0 st
MiB Mem : 7862.9 total, 87.0 free, 1717.7 used, 6058.2 buff/cache
MiB Swap: 7872.0 total, 7872.0 free, 0.0 used. 6020.7 avail Mem

   PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
   1633 cln 20 0 71608 69236 5460 S 49.0 0.9 0:22.59 lightningd
   1671 cln 20 0 20196 18856 3232 S 34.4 0.2 0:13.67 lightning_gossi
   747 bitcoin 20 0 3342684 633720 230944 S 13.2 7.9 1:36.80 bitcoind
   712 debian-+ 20 0 53320 49208 18376 S 10.6 0.6 0:41.84 tor

Thanks. This is strange, CPU usage was normal for about 5 minutes after the reboot, then it started again. Initially lightning_gossipd uses some CPU as well, then it goes away and lightningd becomes the main consumer.
Yes I see my channel as active on 1ml.com

So when I look at the debugging information, I see thousands of lines like these

Code:

Updated pending announce with update 759871x572x1/0
Received channel_announcement for channel 759871x1337x0

and lots of lines like these as well

Code:

Bad gossip order: WIRE_CHANNEL_UPDATE before announcement 766980x708x1/1
Bad gossip order: WIRE_CHANNEL_UPDATE before announcement 765702x318x0/0

HeRetiK

legendary

Activity: 3108

Merit: 2177

Playgram - The Telegram Casino

Quote from: Jodmapht on December 14, 2022, 04:44:35 PM

Quote from: HeRetiK on December 14, 2022, 09:57:21 AM

Actually I've been using the rebalance plugin, rebalancing individual channels manually:
https://github.com/lightningd/plugins/tree/master/rebalance

Running a larger node with good liquidity will definitely require some automation though and the LND ecosystem seems to have a lower learning curve in that regard.

Thanks. My understanding though is that for what I describe, I need a swap and not a rebalancing between LN channels?

Right, the method I describe is only for balancing between LN channels. To balance channels by moving coins on- and off-chain, similar to Loop's service, you'll need a swap.

Jodmapht

member

Activity: 74

Merit: 21

Quote from: DaveF on December 14, 2022, 08:57:04 PM

Something is definitely not right. You should not be using that much CPU for either of them.
Stupid as it may be, can you reboot and see if it clears up. Could be something else hung, but that is a guess.

If you look at 1ml.com do you see the channel open?

I'm exhausted, but I'm sure someone else can chime in with other ideas.

-Dave

Code:

top - 00:31:48 up 11 min, 6 users, load average: 1.04, 0.68, 0.48
Tasks: 175 total, 1 running, 174 sleeping, 0 stopped, 0 zombie
%Cpu(s): 29.0 us, 12.4 sy, 0.0 ni, 57.1 id, 0.9 wa, 0.0 hi, 0.7 si, 0.0 st
MiB Mem : 7862.9 total, 87.0 free, 1717.7 used, 6058.2 buff/cache
MiB Swap: 7872.0 total, 7872.0 free, 0.0 used. 6020.7 avail Mem

   PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
   1633 cln 20 0 71608 69236 5460 S 49.0 0.9 0:22.59 lightningd
   1671 cln 20 0 20196 18856 3232 S 34.4 0.2 0:13.67 lightning_gossi
   747 bitcoin 20 0 3342684 633720 230944 S 13.2 7.9 1:36.80 bitcoind
   712 debian-+ 20 0 53320 49208 18376 S 10.6 0.6 0:41.84 tor

Thanks. This is strange, CPU usage was normal for about 5 minutes after the reboot, then it started again. Initially lightning_gossipd uses some CPU as well, then it goes away and lightningd becomes the main consumer.
Yes I see my channel as active on 1ml.com

DaveF

legendary

Activity: 3500

Merit: 6320

Crypto Swap Exchange

When possible when posting things like that use the

Code:

tags it makes it easier to read.

[code]
No throttling (RPI4 with 8 GB of RAM, using UASP+Trim for SSD):
> for src in arm core h264 isp v3d uart pwm emmc pixel vec hdmi dpi ; do \
> echo -e "$src:\t$(vcgencmd measure_clock $src)" ; \
> done
arm: frequency(48)=1500345728
core: frequency(1)=500000992
h264: frequency(28)=0
isp: frequency(45)=0
v3d: frequency(46)=499987808
uart: frequency(22)=48001464
pwm: frequency(25)=0
emmc: frequency(50)=250000496
pixel: frequency(29)=81000000
vec: frequency(10)=0
hdmi: frequency(0)=0
dpi: frequency(4)=0

> vcgencmd measure_temp
temp=40.4'C

> top
Tasks: 178 total, 1 running, 177 sleeping, 0 stopped, 0 zombie
%Cpu(s): 17.8 us, 10.4 sy, 0.0 ni, 69.6 id, [b]1.5 wa[/b], 0.0 hi, 0.8 si, 0.0 st
MiB Mem : 7862.9 total, 99.2 free, 2454.8 used, 5308.9 buff/cache
MiB Swap: 7872.0 total, 7872.0 free, 0.0 used. 5283.3 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
417629 cln 20 0 98872 94388 3332 S [b]38.3[/b] 1.2 105:12.34 [b]lightningd[/b]
415815 bitcoin 20 0 4350500 890128 113516 S [b]15.2[/b] 11.1 104:25.51 [b]bitcoind[/b]

> iostat 1 1000

avg-cpu: %user %nice %system %iowait %steal %idle
16.91 0.00 1.64 0.15 0.00 81.30

Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
sda 22.92 4863.75 633.34 604.88 3876127573 504738369 482052560

avg-cpu: %user %nice %system %iowait %steal %idle
16.24 0.00 15.74 1.52 0.00 66.50

Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
sda 172.00 13656.00 7860.00 17028.00 13656 7860 17028

Something is definitely not right. You should not be using that much CPU for either of them.
Stupid as it may be, can you reboot and see if it clears up. Could be something else hung, but that is a guess.

If you look at 1ml.com do you see the channel open?

I'm exhausted, but I'm sure someone else can chime in with other ideas.

-Dave[/code]

Jodmapht

member

Activity: 74

Merit: 21

No throttling (RPI4 with 8 GB of RAM, using UASP+Trim for SSD):

Code:

> for src in arm core h264 isp v3d uart pwm emmc pixel vec hdmi dpi ; do \
> echo -e "$src:\t$(vcgencmd measure_clock $src)" ; \
> done
arm: frequency(48)=1500345728
core: frequency(1)=500000992
h264: frequency(28)=0
isp: frequency(45)=0
v3d: frequency(46)=499987808
uart: frequency(22)=48001464
pwm: frequency(25)=0
emmc: frequency(50)=250000496
pixel: frequency(29)=81000000
vec: frequency(10)=0
hdmi: frequency(0)=0
dpi: frequency(4)=0

> vcgencmd measure_temp
temp=40.4'C

> top
Tasks: 178 total, 1 running, 177 sleeping, 0 stopped, 0 zombie
%Cpu(s): 17.8 us, 10.4 sy, 0.0 ni, 69.6 id, [b]1.5 wa[/b], 0.0 hi, 0.8 si, 0.0 st
MiB Mem : 7862.9 total, 99.2 free, 2454.8 used, 5308.9 buff/cache
MiB Swap: 7872.0 total, 7872.0 free, 0.0 used. 5283.3 avail Mem

   PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
417629 cln 20 0 98872 94388 3332 S [b]38.3[/b] 1.2 105:12.34 [b]lightningd[/b]
415815 bitcoin 20 0 4350500 890128 113516 S [b]15.2[/b] 11.1 104:25.51 [b]bitcoind[/b]

> iostat 1 1000

avg-cpu: %user %nice %system %iowait %steal %idle
   16.91 0.00 1.64 0.15 0.00 81.30

Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
sda 22.92 4863.75 633.34 604.88 3876127573 504738369 482052560

avg-cpu: %user %nice %system %iowait %steal %idle
   16.24 0.00 15.74 1.52 0.00 66.50

Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
sda 172.00 13656.00 7860.00 17028.00 13656 7860 17028

Jodmapht

member

Activity: 74

Merit: 21

Quote from: DaveF on December 14, 2022, 06:16:03 PM

Quote from: Jodmapht on December 14, 2022, 04:48:28 PM

Thanks! I opened a single (public) channel on CLN using RTL. It has been using 40%+ of the RPI's CPU and bugging down the SSD for over 1 hour. I am not sure what it could be doing given that there can't be anything going on in this channel since my node is not connected to any other channel and I am not doing any transaction? What am I missing?

If you run top what process is grabbing the CPU?

I see lightningd using between 40% and 100%, and bitcoind using about 15%.

Quote from: DaveF
Also, are you sure it's not thermal throttling? Can you run vcgencmd to check?
[url

https://elinux.org/RPI_vcgencmd_usage[/url]

It's easy to get 40% load when the processor is running at 25% speed.

Last year I built a bunch of nodes in a box: https://bitcointalksearch.org/topic/nodes-in-a-box-5364113 and over time have had a few fan issues. Only once did it cause it to thermal throttle but when it did it caused what looked like a CPU spike but it was just running a such a slow speed that you could do anything.

-Dave

It does not seem to be thermal throttling. It uses passive cooling through a case that is basically a big aluminum heat sink in thermal contact with all the heat-generating chips. I have never seen CPU temp above 45C I think. I looked many times today and the highest I saw was 42C. I can give a try with vcgencmd.

DaveF

legendary

Activity: 3500

Merit: 6320

Crypto Swap Exchange

Quote from: Jodmapht on December 14, 2022, 04:48:28 PM

Thanks! I opened a single (public) channel on CLN using RTL. It has been using 40%+ of the RPI's CPU and bugging down the SSD for over 1 hour. I am not sure what it could be doing given that there can't be anything going on in this channel since my node is not connected to any other channel and I am not doing any transaction? What am I missing?

If you run top what process is grabbing the CPU?

Also, are you sure it's not thermal throttling? Can you run vcgencmd to check?
https://elinux.org/RPI_vcgencmd_usage

It's easy to get 40% load when the processor is running at 25% speed.

Last year I built a bunch of nodes in a box: https://bitcointalksearch.org/topic/nodes-in-a-box-5364113 and over time have had a few fan issues. Only once did it cause it to thermal throttle but when it did it caused what looked like a CPU spike but it was just running a such a slow speed that you could do anything.

-Dave

Jodmapht

member

Activity: 74

Merit: 21

Quote from: BlackHatCoiner on December 14, 2022, 10:23:21 AM

CLN is like the lightweight, bug-safer LND. If you're running a node on a Pi, you'll notice difference in performance. While CLN does have plugins, they don't come pre-installed as in LND, because it focuses on efficiency. Stuff such as rebalance and backup-plugin must have a better recognition and development from the LND community though.

In terms of users, I think LND is more used.

Thanks! I opened a single (public) channel on CLN using RTL. It has been using 40%+ of the RPI's CPU and bugging down the SSD for over 1 hour. I am not sure what it could be doing given that there can't be anything going on in this channel since my node is not connected to any other channel and I am not doing any transaction? What am I missing?

Jodmapht

member

Activity: 74

Merit: 21

Quote from: HeRetiK on December 14, 2022, 09:57:21 AM

Quote from: Jodmapht on December 14, 2022, 08:07:05 AM

Quote from: HeRetiK on December 14, 2022, 06:51:10 AM

I've been running a minor CLN node for a while and it definitely feels like the grass is greener on the LND side of things. At least without digging really deep into the guts of it.

The rebalancing I've been doing via commandline, though there might be a simpler way by now. Haven't followed the progress that closely lately.

Thanks. When you say you have been rebalancing via commandline, how do you do that? For now I expect to mostly only receive LN payments and not have to send any myself, so I would be overall using it in a very unbalanced way...

Edit: Based on what I read from the links posted by DaveF, that would be PeerSwap, correct? Thanks!

Actually I've been using the rebalance plugin, rebalancing individual channels manually:
https://github.com/lightningd/plugins/tree/master/rebalance

Running a larger node with good liquidity will definitely require some automation though and the LND ecosystem seems to have a lower learning curve in that regard.

Thanks. My understanding though is that for what I describe, I need a swap and not a rebalancing between LN channels?

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

CLN is like the lightweight, bug-safer LND. If you're running a node on a Pi, you'll notice difference in performance. While CLN does have plugins, they don't come pre-installed as in LND, because it focuses on efficiency. Stuff such as rebalance and backup-plugin must have a better recognition and development from the LND community though.

In terms of users, I think LND is more used.

Topic: How usable is CLN currently (vs LND)? (Read 344 times)