Cannot connect to my electrs server (“shutting down connection”)

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

Quote from: vv181 on October 03, 2021, 10:50:42 AM

[...]

Okay, I figured it out with the logs. It turns out that it couldn't verify the certificate of 192.168.2.7. It must have had another one saved from one of my previous attempts and each time I was trying to connect, it was failing. I deleted the certificate, retried and it now works!

I now have both main net and test net server running! Thanks vv181!

vv181

legendary

Activity: 1932

Merit: 1273

Quote from: BlackHatCoiner on October 03, 2021, 08:02:33 AM

~

Your Electrs journal doesn't indicate that the Electrum is making a connection at all.

After you opened up Electrum, pull up the Electrum debug logging here. Also post the "netstat -an | grep 60002" and "sudo ufw status | grep 60002" output from your Pi here.

Quote from: BlackHatCoiner on October 03, 2021, 04:19:21 PM

I now have both main net and test net server running! Thanks vv181!

Grats!

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

Quote from: vv181 on October 02, 2021, 06:11:48 PM

You can try #242 or #256, see if that works.

Thanks, man! I set jsonrpc_import = true on my electrs_testnet.conf and it is now running properly.

The only thing left that I don't understand why it's not working is the LAN connection failure. As I've previously said, I can normally connect to my electrum server that is running on main net, but not for the one that runs on test net.

It should establish a connection on LAN:60002, but it doesn't.

Electrs on test net works fine:

BTW, that's my nginx.conf:

Code:

ssl_session_timeout 4h;
  ssl_protocols TLSv1.2 TLSv1.3;
  ssl_prefer_server_ciphers on;

  upstream electrs {
    server 127.0.0.1:50001;
  }

upstream electrs-testnet {
    server 127.0.0.1:60001;
  }

  server {
    listen 50002 ssl;
    proxy_pass electrs;
  }

server {
    listen 60002 ssl;
    proxy_pass electrs-testnet;
  }
}

vv181

legendary

Activity: 1932

Merit: 1273

Your pi might be run out of memory or maybe blk files corruption, no? Also, do you monitor your RAM usage when indexing it?

You can try #242 or #256, see if that works.

Either way, you might want to consider upgrading to 0.9.0, but it needs to reindex the whole thing since the DB schema is changed. Refer to https://stadicus.github.io/RaspiBolt/raspibolt_50_electrs.html#electrs-upgrade and https://github.com/romanz/electrs/blob/master/doc/usage.md#upgrading.

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

So, here's a problem. I've synced with the main net and my electrum server on main net works properly. I can connect in it, make transactions with it etc. It shows that it's online.

I've also synced with the test net, but my electrum server on test net doesn't work properly, because it never successfully indexes the chain.

Code:

...
Oct 02 08:28:13 raspibolt electrs[1965]: 2021-10-02T08:27:29.640+01:00 - TRACE - indexed "/mnt/ext/bitcoin/testnet3/blocks/blk00169.dat": 0 rows
Oct 02 08:28:13 raspibolt electrs[1965]: 2021-10-02T08:27:29.807+01:00 - TRACE - indexed "/mnt/ext/bitcoin/testnet3/blocks/blk00168.dat": 0 rows
Oct 02 08:28:13 raspibolt electrs[1965]: memory allocation of 134147382 bytes failed
Oct 02 08:28:13 raspibolt systemd[1]: electrs_testnet.service: Main process exited, code=killed, status=6/ABRT
Oct 02 08:28:13 raspibolt systemd[1]: electrs_testnet.service: Failed with result 'signal'.
Oct 02 08:28:43 raspibolt systemd[1]: electrs_testnet.service: Service RestartSec=30s expired, scheduling restart.
Oct 02 08:28:43 raspibolt systemd[1]: electrs_testnet.service: Scheduled restart job, restart counter is at 1.
Oct 02 08:28:43 raspibolt systemd[1]: Stopped Electrs daemon.
Oct 02 08:28:43 raspibolt systemd[1]: Started Electrs daemon.

As you can see it reaches a point and then it fails allocating memory which leads to restarting the Electrs daemon. Then, the whole thing repeats. What could be the fault?

vv181

legendary

Activity: 1932

Merit: 1273

Quote from: BlackHatCoiner on September 29, 2021, 03:23:02 PM

It doesn't work right now, though. I tried it with port 60002 too.

What does the Electrum debug logging show right now?

Make sure you pointed the correct Electrs RPC port to Nginx, and you opened the UFW firewall of that port.

Code:

$ ufw status
50002/tcp                  ALLOW       192.168.1.50               # allow electrs mainnet
60002/tcp                  ALLOW       192.168.1.50               # allow electrs testnet

Quote from: BlackHatCoiner on September 29, 2021, 03:23:02 PM

Both networks are now ran by my electrs.

Great!

I tried to run Electrum main net pointed to Electrs testnet, and I'm able to reproduce @OP error code, so it's likely the issue.

Code:

20210929T201340.355814Z |     INFO | interface.[192.168.1.5:60002] | disconnecting due to GracefulDisconnect('server chain conflicts with checkpoints')
20210929T201341.233874Z |     INFO | network | connecting to 192.168.1.5:60002:s as new interface
20210929T201341.290862Z |     INFO | interface.[192.168.1.5:60002] | connection established. version: ['electrs 0.8.10', '1.4']
20210929T201341.298321Z |     INFO | interface.[192.168.1.5:60002] | set blockchain with height 702750
20210929T201341.298770Z |     INFO | interface.[192.168.1.5:60002] | can't connect 2096866
20210929T201341.299126Z |     INFO | interface.[192.168.1.5:60002] | requesting block header 702751 in mode backward
20210929T201341.300134Z |     INFO | network | switching to 192.168.1.5:60002:s
20210929T201341.377811Z |     INFO | interface.[192.168.1.5:60002] | requesting block header 659231 in mode backward
20210929T201341.456084Z |     INFO | interface.[192.168.1.5:60002] | disconnecting due to GracefulDisconnect('server chain conflicts with checkpoints'

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

Thanks, I didn't know I could do this!

Alright, so I have few questions. First off, how do I connect to my test net Electrum server from the client? For example, I knew that I should run the electrum executable with the following parameters if I wanted to do the same in main net:

Code:

--oneserver --server 192.168.2.7:50002:s

I remember running this when I was having a test net electrum server:

Code:

--testnet --oneserver --server 192.168.2.7:50002:s

It doesn't work right now, though. I tried it with port 60002 too.

Thanks again! Both networks are now ran by my electrs.

vv181

legendary

Activity: 1932

Merit: 1273

Quote from: BlackHatCoiner on September 29, 2021, 01:21:09 PM

And how can I have separate systemd services? Obviously, I could just have an electrs_mainnet.conf and an electrs_testnet.conf, but what about the systemd services? (I don't know much from Linux)

You can simply make 2 copies of electrs.service, one for main net and one for test net, then fill it with its own configuration. I'll give you an example below.

Quote from: BlackHatCoiner on September 29, 2021, 01:21:09 PM

I should rephrase my words: Can you run both main net and test net at the same time from electrs? If yes, how?

Yes, what I mean is I'm able to run main net and test net with electrs at the same time.

Here is my configuration

Electrs configurations:

Code: (electrs.conf)

# Bitcoin Core settings
network = "bitcoin"
daemon_dir= "/home/bitcoin/.bitcoin"
daemon_rpc_addr = "127.0.0.1:8332"

# Electrs settings
electrum_rpc_addr = "127.0.0.1:50001"
db_dir = "/home/bitcoin/.electrs/db"

Code: (electrs-testnet.conf)

# Bitcoin Core settings
network = "testnet"
daemon_dir= "/home/bitcoin/.bitcoin"
daemon_rpc_addr = "127.0.0.1:18332"

# Electrs settings
electrum_rpc_addr = "127.0.0.1:60001"
db_dir = "/home/bitcoin/.electrs/db"

Here is what I mean by a separate systemd service. Electrs systemd units:

Code: (electrs.service)

[Unit]
Description=Electrs daemon
Wants=bitcoind.service
After=bitcoind.service

[Service]

# Service execution
ExecStart=/usr/local/bin/electrs --conf /home/bitcoin/.electrs/electrs.conf

Code: (electrs-testnet.service)

[Unit]
Description=Electrs testnet daemon
Wants=bitcoind-testnet.service
After=bitcoind-testnet.service

[Service]
ExecStart=/usr/local/bin/electrs --conf /home/bitcoin/.electrs/electrs-testnet.conf

Nginx Configurations:

Code: (nginx.conf)

  upstream electrs {
    server 127.0.0.1:50001;
  }
  upstream electrs-testnet {
    server 127.0.0.1:60001;
  }
  server {
    listen 50002 ssl;
    proxy_pass electrs;
  }
  server {
    listen 60002 ssl;
    proxy_pass electrs-testnet;
  }

I show the part where the lines are different(except the db_dir and daemon_dir), everything else is just the same as the Raspibolt tutorial. It all works fine whether it automatically started with systemd or with manually running the application (electrs --conf=).

Quote from: BlackHatCoiner on September 29, 2021, 01:21:09 PM

Once I run electrs --conf= while it already runs another network, it'll stop it to start with the other network.

It's hard to judge if I didn't know your configuration. If you are still interested to run both main net and test net, try to use my configurations, I'll try to help if something goes wrong.

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

Quote from: vv181 on September 29, 2021, 01:10:48 PM

Actually, it is okay to have the same folder, electrs will do the job as long as you have separate electrs.conf and systemd services.

And how can I have separate systemd services? Obviously, I could just have an electrs_mainnet.conf and an electrs_testnet.conf, but what about the systemd services? (I don't know much from Linux)

Quote from: vv181 on September 29, 2021, 01:10:48 PM

I tried to reproduce it, it works fine on my end.

I should rephrase my words: Can you run both main net and test net at the same time from electrs? If yes, how? Once I run electrs --conf= while it already runs another network, it'll stop it to start with the other network.

vv181

legendary

Activity: 1932

Merit: 1273

Quote from: BlackHatCoiner on September 25, 2021, 05:55:48 AM

the directory's name. I had chosen “electrs” for my test net server so I couldn't choose the same for main net. Thus, I named it “electrs_main”.

Actually, it is okay to have the same folder, electrs will do the job as long as you have separate electrs.conf and systemd services.

+1 for using the wrong network or an corruption.

Quote from: BlackHatCoiner on September 27, 2021, 07:35:05 AM

Question 2: Is it possible to run both main net and test net from the same Pi with electrs?

I tried to reproduce it, it works fine on my end.

Anyway, when you doing it, did you also modify the nginx.conf?

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

Testnet must have been the issue. I deleted the whole electrs directory, redid the installation, but this time for main net and it worked properly after re-indexing the chain.

Question 1: Every time that I start the electrs server with electrs --conf /mnt/ext/electrs/electrs.conf, is it required to re-index the block chain? I may want to turn it off or a power outage may occur. It shouldn't be practical for me to wait for the reindexing and it should also have them saved. What will it achieve by reindexing again and again?

Question 2: Is it possible to run both main net and test net from the same Pi with electrs?

HCP

legendary

Activity: 2086

Merit: 4314

Sounds like your electrs server has not indexed the blockchain correctly... or indexed the "wrong" blockchain as nc50lc has suggested.

I've not had this issue with electrs... but then I don't think I've tried using testnet and mainnet at the same time Huh

nc50lc

legendary

Activity: 2394

Merit: 5531

Self-proclaimed Genius

Quote from: BlackHatCoiner on September 26, 2021, 02:22:05 AM

I dug up the code a little bit, searching what's that GracefulDisconnect and it turns out that it can't search the block headers backwards. Although, I don't understand how it conflicts with checkpoints and which checkpoints are those.

Maybe it's pertaining to this 'checkpoints' file:
https://github.com/spesmilo/electrum/blob/master/electrum/checkpoints.json - contains block hash and target;
Which are from: https://github.com/spesmilo/electrum/blob/b431d8e9b8f75b99a43540ed6672772abfb68834/electrum/constants.py#L63-L64

I've got the same error and behavior by changing some "block hash" from the client's 'checkpoints.json' and connect to a known working server.
But I guess in your case, it's Electrs server's blockchain at fault; perhaps it's using testnet blockchain rather than mainnet or have some minor corruption?

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

Quote from: Abdussamad on September 25, 2021, 09:04:50 AM

enable debug logging in the client and see what it says:

I did and I guess that's the faulty part:

Code:

20210926T061438.669179Z |     INFO | interface.[192.168.2.7:50002] | connection established. version: ['electrs 0.8.11', '1.4']
20210926T061438.669179Z |     INFO | interface.[192.168.2.7:50002] | set blockchain with height 702243
20210926T061438.669179Z |     INFO | interface.[192.168.2.7:50002] | can't connect 2096356
20210926T061438.669179Z |     INFO | interface.[192.168.2.7:50002] | requesting block header 702244 in mode backward
20210926T061438.669179Z |     INFO | network | switching to 192.168.2.7:50002:s
20210926T061438.684803Z |     INFO | gui.qt.ElectrumGui | Qt GUI starting up... Qt=5.15.2, PyQt=5.15.4
20210926T061438.684803Z |     INFO | network | fee_estimates {25: 1000, 10: 1000, 5: 1000, 2: 1000}
20210926T061438.684803Z |     INFO | interface.[192.168.2.7:50002] | requesting block header 659231 in mode backward
20210926T061438.684803Z |     INFO | network | fee_histogram []
20210926T061438.684803Z |     INFO | interface.[192.168.2.7:50002] | disconnecting due to GracefulDisconnect('server chain conflicts with checkpoints')

I dug up the code a little bit, searching what's that GracefulDisconnect and it turns out that it can't search the block headers backwards. Although, I don't understand how it conflicts with checkpoints and which checkpoints are those.

But, that's probably what's wrong. It, then, retries connecting, but gets the same disconnection error.

Edit: No one has faced this before. Undecided

Abdussamad

legendary

Activity: 3584

Merit: 1560

enable debug logging in the client and see what it says:

https://electrum.readthedocs.io/en/latest/faq.html#how-to-enable-debug-logging

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

I've setup my electrs as it's written in here except from the directory's name. I had chosen “electrs” for my test net server so I couldn't choose the same for main net. Thus, I named it “electrs_main”.

Node works fine, server works fine and once I run:

Code:

electrum.exe --oneserver --server=192.168.2.7:50002:s

This is what it returns me in the Linux console (from my Pi):

Code:

Sep 25 10:46:15 raspibolt electrs[2722]: 2021-09-25T10:46:15.378+01:00 - TRACE - RPC Request("{\"jsonrpc\": \"2.0\", \"method\": \"blockchain.block.header\", \"id\": 22, \"params\": [659231]}\n")
Sep 25 10:46:15 raspibolt electrs[2722]: 2021-09-25T10:46:15.383+01:00 - TRACE - RPC Done
Sep 25 10:46:15 raspibolt electrs[2722]: 2021-09-25T10:46:15.383+01:00 - DEBUG - [127.0.0.1:48976] shutting down connection
Sep 25 10:46:15 raspibolt electrs[2722]: 2021-09-25T10:46:15.384+01:00 - INFO - [127.0.0.1:48976] disconnected peer

And then it repeats. It continuously tries to establish a connection and reads my balance, but fails and I remain disconnected.

(The test net server works fine)

Topic: Cannot connect to my electrs server (“shutting down connection”) (Read 278 times)