Author

Topic: An efficient re-implementation of Electrum Server in Rust (Read 630 times)

HCP
legendary
Activity: 2086
Merit: 4363
Are you running electrs as a background process or are you running it in a terminal window and can see all of it's debug output?

When you run Electrum, do you see any notifications in the electrs output that shows any connection attempts? You might also want to run manually running Electrum from a terminal with the 'debug' option (-v) to see why it isn't able to connection.

There is also an option in Electrum to log to disk: Tools menu > Preferences > General tab, and tick "Write logs to file"
newbie
Activity: 4
Merit: 1
Unable to get Electrum wallet to connect to electrs

Unsure if this is the right thread to ask this, however I couldn't find a thread that related to this issue so here goes -

Have electrs running (confirmed with: sudo systemctl status electrs). Have installed Electrum wallet, & edited the electrum.desktop file as follows to connect to localhost -

Code:
Exec=sh -c "PATH=\"\\$HOME/.local/bin:\\$PATH\"; electrum --oneserver --server localhost:50001:t %u"

Network Server tab in Electrum wallet shows "Server: localhost 50001" & all other servers greyed out, however within Network OverviewStatus: shows "Not connected" & Server: shows "None"

Perhaps there are config edits I need to make elsewhere for Electrum to connect to electrs?

Any suggestions greatly appreciated! Thank you in advance.
newbie
Activity: 4
Merit: 1

Quote
So, try this:
Code:
cargo run --release -- -vvv --timestamp --db-dir ./db --daemon-dir /mnt/HDD/ --electrum-rpc-addr="127.0.0.1:50001" --cookie-file /home/user/.bitcoin/.cookie

It worked! Electrs is up & running, financial self-sovereignty is within very close reach. Yes my apologies, I meant to refer to electrs not Electrum in my previous comment - rookie mistake.

As a non-techie noob I'm blown away by this world of open source development & the remarkable minds working together to make it all happen.

HCP Sir, a MASSIVE thank you for walking me through this. Means a lot. Hope to be able to contribute in some way myself in future.
HCP
legendary
Activity: 2086
Merit: 4363
Great, thank you for suggesting! Doesn't seem to do the trick unfortunately. I assume that   --db-dir ./db  within the argument refers to the Electrum database, so I've left this unchanged given I have Electrs located in the default /home/electrs/ location.
Did you mean Electrum database... or electrs database? Huh db-dir is the location where electrs stores it's database.



Quote
The .cookie file is not produced within the /mnt/HDD/ folder but instead within /Home/User/.bitcoin, together with the bitcoin.conf file, & I have the following argument within the bitcoin.conf file that sets /mnt/HDD/ as the data directory: "datadir=/mnt/HDD"

I have the following argument - "rpccookiefile=/home/user/.bitcoin/.cookie" - within the bitcoin.conf file to create the .cookie file there so that Bitcoind will run from the default /Home/User/.bitcoin directory but use the /mnt/HDD to store the blocks.
Ahhh ok, yeah... you've added an extra layer of complexity by moving some of the Bitcoin data to your /mnt/HDD location, but then leaving some of it in the default "/home/user/.bitcoin" location.

By default, electrs simply looks for the cookie in the daemon-dir... If you have the cookie in a custom location, you'd need to specify that. There is a cookie-file argument that you can use for this.

So, try this:
Code:
cargo run --release -- -vvv --timestamp --db-dir ./db --daemon-dir /mnt/HDD/ --electrum-rpc-addr="127.0.0.1:50001" --cookie-file /home/user/.bitcoin/.cookie
newbie
Activity: 4
Merit: 1
Ok... so I use a shell script to start it up... it's basically one line:

Code:
cargo run --release -- -vvv --timestamp --db-dir /mnt/e/electrs --daemon-dir /mnt/e/Bitcoin --electrum-rpc-addr="127.0.0.1:50001"

As you can see... I've got --daemon-dir added into the "cargo run" command... and specified the path to my Bitcoin blocks folder... "/mnt/e/Bitcoin" (I've also put the electrs db-dir in a custom location too)


Great, thank you for suggesting! Doesn't seem to do the trick unfortunately. I assume that   --db-dir ./db  within the argument refers to the Electrum database, so I've left this unchanged given I have Electrs located in the default /home/electrs/ location. Thus my code is as follows -

Code:
cargo run --release -- -vvv --timestamp --db-dir ./db --daemon-dir /mnt/HDD/ --electrum-rpc-addr="127.0.0.1:50001"

This returns the following recurring error:

"WARN - reconnecting to bitcoind: failed to read cookie from /mnt/HDD/.cookie"
.
The .cookie file is not produced within the /mnt/HDD/ folder but instead within /Home/User/.bitcoin, together with the bitcoin.conf file, & I have the following argument within the bitcoin.conf file that sets /mnt/HDD/ as the data directory: "datadir=/mnt/HDD"

I have the following argument - "rpccookiefile=/home/user/.bitcoin/.cookie" - within the bitcoin.conf file to create the .cookie file there so that Bitcoind will run from the default /Home/User/.bitcoin directory but use the /mnt/HDD to store the blocks.

I've tried setting /Home/User/.bitcoin as the --daemon-dir in your suggested argument however this returned the original error, where Electrs appears to look for blocks within /Home/User/.bitcoin instead of /mnt/HDD.

Hope this makes sense, very new to Ubuntu/Linux & setting up a node etc. How to proceed from here?

Thank you in advance!


HCP
legendary
Activity: 2086
Merit: 4363
Ok... so I use a shell script to start it up... it's basically one line:

Code:
cargo run --release -- -vvv --timestamp --db-dir /mnt/e/electrs --daemon-dir /mnt/e/Bitcoin --electrum-rpc-addr="127.0.0.1:50001"

As you can see... I've got --daemon-dir added into the "cargo run" command... and specified the path to my Bitcoin blocks folder... "/mnt/e/Bitcoin" (I've also put the electrs db-dir in a custom location too)
newbie
Activity: 4
Merit: 1
"Before anyone asks... you use --daemon-dir to specify your custom Bitcoin directory location Wink"

Where do you specify this exactly? I'm an Ubuntu / Linux noob trying to set up Electrum Rust using laptop + external HDD, & have run into a problem I can't seem to solve -

Node is synced to the tip & fully indexed, with blocks & data stored on external HDD, while bitcoin.conf & .cookie files appear in .bitcoin folder in Home directory. However in my first Electrum Rust "usage" ie. index sync ($ cargo run --release -- -vvv --timestamp --db-dir ./db --electrum-rpc-addr="127.0.0.1:50001"), as per the "Usage" section (https://github.com/romanz/electrs/blob/master/doc/usage.md) I get the following error -


"2020-03-02T16:09:42.188+10:30 - INFO - indexing 0 blk*.dat files
2020-03-02T16:09:42.189+10:30 - DEBUG - found 0 indexed blocks
2020-03-02T16:10:03.639+10:30 - DEBUG - applying 619796 new headers from height 0

thread 'bulk_writer' panicked at 'no indexed header found', src/libcore/option.rs:1188:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
thread 'main' panicked at 'writer panicked: Any', src/libcore/result.rs:1188:5"

My data directory is on my external HDD, however it appears Electrs is looking for blocks in the default directory?

Any help would be very much appreciated!





hero member
Activity: 692
Merit: 500
Thank you, the update went smoothly.  The Cargo build only took 45 seconds or so.
newbie
Activity: 8
Merit: 1
The underlying storage format didn't change, so the instructions above should work Smiley
hero member
Activity: 692
Merit: 500
What would you recommend are the steps to update to the newest release v0.7 (in my case I have v0.62 from early June) ?
I have not made any local changes to the code.

Code:
$ git clone https://github.com/romanz/electrs
$ cd electrs
$ git fetch origin
$ git status
$ git pull
$ cargo build --release
newbie
Activity: 8
Merit: 1
It should work with latest Bitcoin Core (0.16+).
Didn't test with Parity or with Bitcoin Cash nodes...
newbie
Activity: 12
Merit: 7
Which full node implementations of bitcoin and bitcoin* cash are compatible with this Electrum server? I tried Parity, so that I could run bitcoin and bitcoin cash while only having to upgrade one codebase.  However, ElectrumX will not speak to Parity. From my perspective, Parity and bitcoin core follow the same network protocol, but within my localhost they use a different database formats.  Hence why ElectrumX rejects Parity's RPC responses with:

Code:
 
Jan 30 05:55:12  electrumx_server[4937]: ERROR:root:task crashed: Jan 30 05:55:12  electrumx_server[4937]: Traceback (most recent call last):
Jan 30 05:55:12  electrumx_server[4937]:   File "/home/electrum/.local/lib/python3.7/site-packages/aiorpcx/util.py", line 118, in
.
.
Jan 30 05:55:12  electrumx_server[4937]: ERROR:Daemon:Supplied content type is not allowed. Content-Type: application/json is required

I didn't see any specs for the full node implementation compatibility on either https://github.com/romanz/electrs/blob/master/doc/usage.md or https://github.com/kyuupichan/electrumx/tree/master/docs . Also, is there a way to run one full node implementation Thanks for reading.

*I use lower case to denote that I am referring to the bitcoin/bitcoin cash networks and not the units of account.
newbie
Activity: 8
Merit: 1
Thanks!
This project was designed for personal use (e.g. optimized for HDD, no DoS prevention mechanisms).

Currently, it doesn't utilize SSD for better performance (so it would be hard to support thousands users), but there is a WIP branch for a new (more SSD-friendly) indexing schema here:
https://github.com/romanz/electrs/commits/new-index
sr. member
Activity: 441
Merit: 250
Interesting project! Electrum server could certainly be improved on.

Is this software for running a personal server or could it be used to run a public server?

How well would it handle the load of thousands simultaneous users, asking for many addresses each?
newbie
Activity: 8
Merit: 1
HCP
legendary
Activity: 2086
Merit: 4363
I guess the lack of SSL isn't a "big deal", since I'm connecting to localhost... but are you planning on implementing SSL in case I'd like to host it on a separate/remote box?
newbie
Activity: 8
Merit: 1
Cool, glad to hear that it works Smiley
HCP
legendary
Activity: 2086
Merit: 4363
Has been slow going... My router died the other day and after a day of being bounced around various departments at my ISP to get a replacement, a friend loaned me one and I managed to get back online...

Since then the various Ubuntu and Rust repositories have been spitting out random errors... So I think my connection still isn't quite right.

I think I finally managed to get it all synced up and set it compiling before heading to work... Will check it after my shift Wink



EDIT: ok, so I managed to get it compiled and running and it is picking up new blocks:
Code:
2018-07-20T19:36:14.665+12:00 - DEBUG - Server listening on 127.0.0.1:42024
2018-07-20T19:36:14.669+12:00 - DEBUG - Running accept thread
2018-07-20T19:36:14.682+12:00 - DEBUG - BlockchainInfo { chain: "main", blocks: 532757, headers: 532757, bestblockhash: "00000000000000000021489dbeba2486cb06231787a01248d5279b478d44d29d", size_on_disk: 199680846175, pruned: false }
2018-07-20T19:36:14.684+12:00 - DEBUG - NetworkInfo { version: 160100, subversion: "/Satoshi:0.16.1/" }
2018-07-20T19:36:14.684+12:00 - DEBUG - opening DB at "/mnt/s/db/mainnet"
2018-07-20T19:36:15.476+12:00 - INFO - listing block files at "/mnt/e/Bitcoin/blocks/blk*.dat"
2018-07-20T19:36:15.504+12:00 - INFO - indexing 1321 blk*.dat files
2018-07-20T19:37:08.956+12:00 - DEBUG - applying 532758 new headers from height 0
2018-07-20T20:07:53.198+12:00 - DEBUG - skipping block 000000000000000001b982e5989a72cd799130f71cf3acd6db6214de024a8ad2
2018-07-20T20:07:53.669+12:00 - DEBUG - skipping block 000000000000000000352b9ad3494acf88b0aa73a4c6f8987bab247e6c32d9ca
2018-07-20T20:08:13.935+12:00 - DEBUG - skipping block 0000000000000000009085be97279a7ab4d20e983be9cb20fe0c2e0d49313cdd
...
snip Bunch of "DEBUG - skipping block lines"
...
2018-07-20T20:28:43.430+12:00 - DEBUG - no more blocks to index
2018-07-20T20:28:44.543+12:00 - DEBUG - last indexed block: best=00000000000000000021489dbeba2486cb06231787a01248d5279b478d44d29d height=532757 @ 2018-07-20T07:32:29Z
2018-07-20T20:28:44.774+12:00 - INFO - starting full compaction
2018-07-20T21:38:04.565+12:00 - INFO - finished full compaction
2018-07-20T21:38:09.008+12:00 - DEBUG - applying 532758 new headers from height 0
2018-07-20T21:38:09.518+12:00 - DEBUG - downloading new block headers (532758 already indexed) from 0000000000000000000dc8a8b42d286c1796337a3cc44622143320e98a40415c
2018-07-20T21:38:09.539+12:00 - INFO - best=0000000000000000000dc8a8b42d286c1796337a3cc44622143320e98a40415c height=532779 @ 2018-07-20T09:26:49Z (22 left to index)
2018-07-20T21:38:13.016+12:00 - DEBUG - applying 22 new headers from height 532758
2018-07-20T21:38:13.822+12:00 - INFO - RPC server running on 127.0.0.1:50001
2018-07-20T21:55:05.336+12:00 - DEBUG - downloading new block headers (532780 already indexed) from 00000000000000000003972c3d95015978a4b7fa5662aee1417b677058178318
2018-07-20T21:55:05.338+12:00 - INFO - best=00000000000000000003972c3d95015978a4b7fa5662aee1417b677058178318 height=532780 @ 2018-07-20T09:54:40Z (1 left to index)
2018-07-20T21:55:06.067+12:00 - DEBUG - applying 1 new headers from height 532780

However, when I attempt to connect with Electrum (v3.2.2)... it starts generating "UTF8" errors
Code:
2018-07-20T23:40:28.775+12:00 - INFO - [127.0.0.1:61899] connected peer
2018-07-20T23:40:28.783+12:00 - INFO - [127.0.0.1:61899] shutting down connection
2018-07-20T23:40:28.783+12:00 - ERROR - [127.0.0.1:61899] receiver failed: invalid UTF8 "\u{16}\u{3}\u{1}\u{2}\u{0}\u{1}\u{0}[b][REDACTED][/b]\n": FromUtf8Error { bytes: [22, 3, 1, 2, 0, 1, 0, [b][REDACTED][/b]], error: Utf8Error { valid_up_to: 8, error_len: Some(1) } }
2018-07-20T23:40:28.786+12:00 - INFO - [127.0.0.1:61900] connected peer
2018-07-20T23:40:28.786+12:00 - INFO - [127.0.0.1:61899] disconnected peer
2018-07-20T23:40:28.792+12:00 - INFO - [127.0.0.1:61900] shutting down connection
2018-07-20T23:40:28.793+12:00 - ERROR - [127.0.0.1:61900] receiver failed: invalid UTF8 "\u{16}\u{3}\u{1}\u{2}\u{0}\u{1}\u{0}[b][REDACTED][/b]\n": FromUtf8Error { bytes: [22, 3, 1, 2, 0, 1, 0, [b][REDACTED][/b]], error: Utf8Error { valid_up_to: 8, error_len: Some(1) } }
2018-07-20T23:40:28.794+12:00 - INFO - [127.0.0.1:61900] disconnected peer

Any ideas? NOTE: also created an issue on GitHub



EDIT2: as per the note on Github... it is related to SSL connections... currently there is no way to disable SSL in the GUI... but starting Electrum and forcing to use TCP with the following command:
Code:
"C:\Program Files (x86)\Electrum\electrum-3.2.2.exe" --server=127.0.0.1:50001:t

And it all connects and syncs up... pretty happy to have my own Electrum server now Wink
newbie
Activity: 8
Merit: 1
Quote
I'll update how I get on.

Thanks a lot!
Please let me know if there are any issues/problems Smiley
HCP
legendary
Activity: 2086
Merit: 4363
Before anyone asks... you use --daemon-dir to specify your custom Bitcoin directory location Wink

All commandline args are listed here: https://github.com/romanz/electrs/blob/master/src/config.rs

I'm currently attempting to see if this will work on Ubuntu 16.04 running on the Windows Linux Subsystem... combined with Bitcoin Core running under Windows. I'll update how I get on. Wink
newbie
Activity: 8
Merit: 1
Thanks for the compliment Smiley

Quote
I will suggest though is that you list the limitations of your implementation so others, if not you, can improve on it as it grows.

Good point, will add a document for that!

N.B.
https://github.com/romanz/electrs/issues/8
sr. member
Activity: 1344
Merit: 307
Hello all,

I was working on this project for the last few months, so a user could run his own Electrum server, with required hardware resources not much beyond those of a full node (using ideas from ElectrumX [1], Electrum Personal Server [2] and bitcoincore-indexd [3]).

The code and usage instructions can be found here:
https://github.com/romanz/electrs

The server indexes the entire Bitcoin blockchain, and the resulting index [4] enables fast queries for any given user wallet, allowing the user to keep real-time track of his balances and his transaction history using the Electrum wallet [5].
Since it runs on the user's own machine, there is no need for the wallet to communicate with external Electrum servers, thus preserving the privacy of the user's addresses and balances.

Features:
 * Supports latest Electrum protocol [6].
 * Maintains an index of transaction inputs and outputs, allowing fast balance queries
 * Fast synchronization of the Bitcoin blockchain (~2.5 hours for ~185GB @ June 2018) on modest hardware [7]
 * Low CPU & memory usage (after initial indexing)
 * Low index storage overhead (~20%), relying on a local full node for transaction retrieval
 * Efficient mempool tracker allowing better fee estimation [8].
 * `-txindex` is not required for the Bitcoin node
 * Uses `rust-bitcoin` library [9] for efficient serialization/deserialization of Bitcoin transactions
 * Uses a single RocksDB [10] database, for better consistency and crash recovery

Hope you'll find it useful Smiley
Questions, suggestions and pull requests are welcome!

[1] https://github.com/kyuupichan/electrumx
[2] https://github.com/chris-belcher/electrum-personal-server
[3] https://github.com/jonasschnelli/bitcoincore-indexd
[4] https://github.com/romanz/electrs/blob/master/doc/schema.md
[5] https://electrum.org
[6] https://electrumx.readthedocs.io/en/latest/protocol.html
[7] https://gist.github.com/romanz/cd9324474de0c2f121198afe3d063548
[8] https://github.com/spesmilo/electrum/blob/59c1d03f018026ac301c4e74facfc64da8ae4708/RELEASE-NOTES#L34-L46)
[9] https://github.com/rust-bitcoin/rust-bitcoin
[10] https://github.com/spacejam/rust-rocksdb

About a year ago, I attempted to redo electrum in rust (mainly the server side of it) mainly to push away from using python on a server, and to create something that would have high performance while remaining scalable, secured and have a low footprint.  Decided to move to a different route than to re-implement it, but I will say you got WAY further than me. Even though you do have a TODO list, one thing I will suggest though is that you list the limitations of your implementation so others, if not you, can improve on it as it grows.
newbie
Activity: 8
Merit: 1
Hello all,

I was working on this project for the last few months, so a user could run his own Electrum server, with required hardware resources not much beyond those of a full node (using ideas from ElectrumX [1], Electrum Personal Server [2] and bitcoincore-indexd [3]).

The code and usage instructions can be found here:
https://github.com/romanz/electrs

The server indexes the entire Bitcoin blockchain, and the resulting index [4] enables fast queries for any given user wallet, allowing the user to keep real-time track of his balances and his transaction history using the Electrum wallet [5].
Since it runs on the user's own machine, there is no need for the wallet to communicate with external Electrum servers, thus preserving the privacy of the user's addresses and balances.

Features:
 * Supports latest Electrum protocol [6].
 * Maintains an index of transaction inputs and outputs, allowing fast balance queries
 * Fast synchronization of the Bitcoin blockchain (~2.5 hours for ~185GB @ June 2018) on modest hardware [7]
 * Low CPU & memory usage (after initial indexing)
 * Low index storage overhead (~20%), relying on a local full node for transaction retrieval
 * Efficient mempool tracker allowing better fee estimation [8].
 * `-txindex` is not required for the Bitcoin node
 * Uses `rust-bitcoin` library [9] for efficient serialization/deserialization of Bitcoin transactions
 * Uses a single RocksDB [10] database, for better consistency and crash recovery

Hope you'll find it useful Smiley
Questions, suggestions and pull requests are welcome!

[1] https://github.com/kyuupichan/electrumx
[2] https://github.com/chris-belcher/electrum-personal-server
[3] https://github.com/jonasschnelli/bitcoincore-indexd
[4] https://github.com/romanz/electrs/blob/master/doc/schema.md
[5] https://electrum.org
[6] https://electrumx.readthedocs.io/en/latest/protocol.html
[7] https://gist.github.com/romanz/cd9324474de0c2f121198afe3d063548
[8] https://github.com/spesmilo/electrum/blob/59c1d03f018026ac301c4e74facfc64da8ae4708/RELEASE-NOTES#L34-L46)
[9] https://github.com/rust-bitcoin/rust-bitcoin
[10] https://github.com/spacejam/rust-rocksdb
Jump to: