Electrum server discussion thread - page 7.

it-zone

newbie

Activity: 37

Merit: 0

Quote from: tl121 on July 08, 2015, 07:19:24 PM

Electrum-server used 5832 seconds to process one block! This is a bug. Not a slow processor.

Unfortunately I must agree, the architecture of electrum server is screwed. If we use power of the internet to make electrum possible, we should share resources between electrum servers.

But there is no clear solution. Which Key-Value database engine is able work on the WAN scale?

tl121

sr. member

Activity: 278

Merit: 254

Quote from: molecular on July 08, 2015, 04:00:14 PM

Quote from: tl121 on July 08, 2015, 02:23:58 PM

I believe it should be high priority for the Electrum system to show unconfirmed payments as rapidly as possible, so as to make for a Newbie friendly user experience.

I agree to an extent. However I think the problem is mitigated for "newbie users" because they will likely use "auto connect" and I'm assuming lagging servers will not be autoconnected to.

Of course the electrum server software could probably be made to perform better, but I'm guessing it'd be quite a bit of work.

Until then and although it's a bit sad and maybe even mildly concerning (centralization), maybe your little atom (not to discredit it, but I assume it's the bottleneck in your system when electrum is digesting transactions) will have to leave the work to faster machines regarding servicing the masses?

A couple of years ago, I played around with Electrum and rejected it because of the "server lagging" issues and the unreliability of connections to the servers. It seemed dodgy. These problems are not new and not the fault of my particular server hardware. At the time I was not running a server and was using the auto selection feature.

I chose the little Atom deliberately, so that it would be a canary in the coal mine. My little machine is not servicing the masses. It's just servicing me. If tracking the block chain can not be done efficiently, then the future of Electrum is at risk. I would say the future of bitcoin as well as a decentralized system, except that the little Atom manages to keep up with the present load using only a few percent of CPU time to run bitcoind.

Electrum-server used 5832 seconds to process one block! This is a bug. Not a slow processor.

molecular

donator

Activity: 2772

Merit: 1019

Quote from: tl121 on July 08, 2015, 02:23:58 PM

I believe it should be high priority for the Electrum system to show unconfirmed payments as rapidly as possible, so as to make for a Newbie friendly user experience.

I agree to an extent. However I think the problem is mitigated for "newbie users" because they will likely use "auto connect" and I'm assuming lagging servers will not be autoconnected to.

Of course the electrum server software could probably be made to perform better, but I'm guessing it'd be quite a bit of work.

Until then and although it's a bit sad and maybe even mildly concerning (centralization), maybe your little atom (not to discredit it, but I assume it's the bottleneck in your system when electrum is digesting transactions) will have to leave the work to faster machines regarding servicing the masses?

tl121

sr. member

Activity: 278

Merit: 254

Quote from: molecular on July 08, 2015, 01:47:52 PM

Quote from: tl121 on July 08, 2015, 12:24:19 PM

Worse, when I use an Electrum client to access the node I observe that the client reports an erroneous number of blocks lagging (more blocks than actual).

Does a client restart fix this particular issue?

If so, maybe you're looking at this issue instead.

Thanks for the suggestion.

Because I had previously experienced the problem that you linked, I had already tried a number of client restarts. They didn't help. At present my server is only 3 blocks behind (according to getinfo via SSH) and this is consistent with what my Electrum client reports. However, the unconfirmed transaction has been sitting for three hours (presumably due to the "test"). Blockchain.info has had the transaction for three hours, but my Electrum client still doesn't show it. (It will show it if I connect it to a different Electrum server, so I know that this is server related. After changing servers, I restarted the client and it reverted back to showing the transaction unconfirmed. I will have to wait until my server catches up and then I will see if it picks up the unconfirmed transaction.)

I believe it should be high priority for the Electrum system to show unconfirmed payments as rapidly as possible, so as to make for a Newbie friendly user experience.

molecular

donator

Activity: 2772

Merit: 1019

Quote from: tl121 on July 08, 2015, 12:24:19 PM

Worse, when I use an Electrum client to access the node I observe that the client reports an erroneous number of blocks lagging (more blocks than actual).

Does a client restart fix this particular issue?

If so, maybe you're looking at this issue instead.

tl121

sr. member

Activity: 278

Merit: 254

My private Electrum server is having trouble keeping up with the present network activity, being up to 11 blocks behind. The bitcoin node on the same machine has no trouble keeping up and the CPU time used by electrum-server is now 7 times greater than that used by bitcoind.

Worse, when I use an Electrum client to access the node I observe that the client reports an erroneous number of blocks lagging (more blocks than actual). In addition, it does not pull out a new unconfirmed received transactions that have been in the mempool for over an hour. (Switching the client to another server does show this unconfirmed transaction.) This is a private server and only one client is accessing it. Perhaps a better balance in thread priority can be taken between resources spent tracking the network and updating the database vs. providing timely interaction to clients.

Looking at the logs, it looks like one block gave the server serious cramps:

[08/07/2015-12:24:49] utxo not in database; postponing mempool update
[08/07/2015-12:24:49] blockchain: 364428 (5284.813s)
[08/07/2015-12:25:15] mempool_update 16.0112478733

The machine has a Haswell Atom processor, but it has 8 GB of RAM and an SSD. Running Ubuntu 14.04, Electrum Server updated from github yesterday.

BadAss.Sx

legendary

Activity: 1526

Merit: 1002

Bulletproof VPS/VPN/Email @ BadAss.Sx

Quote from: ?? on ??

There was an orphan block at height 363959. Your bitcoind should have rejected the orphan block, if you are using Bitcoin-core v0.9.5 and above - which version are you using?

I used 0.10.0, but have upgraded now to 0.10.2. Also redownloaded the db, because mine was corrupt and couldn't get past the orphaned block.

memvola

hero member

Activity: 938

Merit: 1002

Quote from: ThomasV on July 06, 2015, 07:00:18 AM

There is a robot, 'electra', that detects servers with a corrupted DB, and kicks them from the irc channel.
Unfortunately this robot is temporarily broken (I will try to fix it this week), and it seems that a few servers have been running with db corruption without noticing it.

Is there a way to check for corruption locally?

ThomasV

legendary

Activity: 1896

Merit: 1353

Quote from: memvola on July 06, 2015, 06:43:12 AM

Hi!

Unfortunately my server couldn't get past the reorg; seems like it can't revert. bitcoind running fine, updated to the latest version, regenerated/imported blockchain_headers to no avail. Any ideas?

...

This traceback means that your database was corrupted.
There is a robot, 'electra', that detects servers with a corrupted DB, and kicks them from the irc channel.
Unfortunately this robot is temporarily broken (I will try to fix it this week), and it seems that a few servers have been running with db corruption without noticing it.
This reorg made the job that electra usually does

JWU42

legendary

Activity: 1666

Merit: 1000

memvola

hero member

Activity: 938

Merit: 1002

Hi!

Unfortunately my server couldn't get past the reorg; seems like it can't revert. bitcoind running fine, updated to the latest version, regenerated/imported blockchain_headers to no avail. Any ideas?

Code:

electrum-server[2001]: INFO:electrum:blockchain reorg 363959 000000000000000010f14e0687867ae37dafce2a918cdf3f9eccdfa992445eb5 00000000000000001262efe584ec7ac039dba15b56d6ed239249c483636b15e
electrum-server[2001]: Exception in thread Thread-4:
electrum-server[2001]: Traceback (most recent call last):
electrum-server[2001]: File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
electrum-server[2001]: self.run()
electrum-server[2001]: File "/usr/lib/python2.7/threading.py", line 763, in run
electrum-server[2001]: self.__target(*self.__args, **self.__kwargs)
electrum-server[2001]: File "/usr/lib/python2.7/site-packages/electrumserver/blockchain_processor.py", line 76, in do_catch_up
electrum-server[2001]: self.catch_up(sync=False)
electrum-server[2001]: File "/usr/lib/python2.7/site-packages/electrumserver/blockchain_processor.py", line 658, in catch_up
electrum-server[2001]: self.import_block(block, self.storage.last_hash, self.storage.height, sync, revert=True)
electrum-server[2001]: File "/usr/lib/python2.7/site-packages/electrumserver/blockchain_processor.py", line 408, in import_block
electrum-server[2001]: self.storage.revert_transaction(txid, tx, block_height, touched_addr, undo)
electrum-server[2001]: File "/usr/lib/python2.7/site-packages/electrumserver/storage.py", line 602, in revert_transaction
electrum-server[2001]: self.revert_add_to_history(addr, txid, x.get('index'), x.get('value'), block_height)
electrum-server[2001]: File "/usr/lib/python2.7/site-packages/electrumserver/storage.py", line 511, in revert_add_to_history
electrum-server[2001]: self.delete_address(key + txo)
electrum-server[2001]: File "/usr/lib/python2.7/site-packages/electrumserver/storage.py", line 416, in delete_address
electrum-server[2001]: items.pop(letter)
electrum-server[2001]: KeyError: '7'

TheButterZone

legendary

Activity: 3038

Merit: 1032

RIP Mommy

Submitted
https://github.com/spesmilo/electrum-server/issues/98
for
https://bitcointalksearch.org/topic/electrum-servers-should-indicate-which-version-of-bitcoind-they-are-using-1109497

JWU42

legendary

Activity: 1666

Merit: 1000

Quote from: BadAss.Sx on July 05, 2015, 10:56:13 AM

Have a new problem:

Quote

[05/07/2015-17:43:36] Starting Electrum server on electrum.badass.sx
[05/07/2015-17:43:37] Database version 3
[05/07/2015-17:43:37] Blockchain height 363959
[05/07/2015-17:43:37] UTXO tree root hash: 83d729bc609764ee6bfd7f8e33ece94e33d68aada186554fe1c9579b622f2bc9
[05/07/2015-17:43:37] Coins in database: 1434845972693919
[05/07/2015-17:43:37] TCP server started on 94.102.50.70:50001
[05/07/2015-17:43:37] SSL server started on 94.102.50.70:50002
[05/07/2015-17:43:39] blockchain reorg 363959 000000000000000010f14e0687867ae37dafce2a918cdf3f9eccdfa992445eb5 00000000000000001262efe584ec7ac039dba15b56d6ed239249c483636b15ef

How can i pass this? Doesn't matter if i restart the server, the problem remains and electrum server isn't public.

Not a problem - it hit my log as well. Reorg's happen...

Code:

[05/07/2015-15:25:23] blockchain reorg 363959 000000000000000010f14e0687867ae37dafce2a918cdf3f9eccdfa992445eb5 00000000000000001262efe584ec7ac039dba15b56d6ed239249c483636b15ef

BadAss.Sx

legendary

Activity: 1526

Merit: 1002

Bulletproof VPS/VPN/Email @ BadAss.Sx

Have a new problem:

Quote

[05/07/2015-17:43:36] Starting Electrum server on electrum.badass.sx
[05/07/2015-17:43:37] Database version 3
[05/07/2015-17:43:37] Blockchain height 363959
[05/07/2015-17:43:37] UTXO tree root hash: 83d729bc609764ee6bfd7f8e33ece94e33d68aada186554fe1c9579b622f2bc9
[05/07/2015-17:43:37] Coins in database: 1434845972693919
[05/07/2015-17:43:37] TCP server started on 94.102.50.70:50001
[05/07/2015-17:43:37] SSL server started on 94.102.50.70:50002
[05/07/2015-17:43:39] blockchain reorg 363959 000000000000000010f14e0687867ae37dafce2a918cdf3f9eccdfa992445eb5 00000000000000001262efe584ec7ac039dba15b56d6ed239249c483636b15ef

How can i pass this? Doesn't matter if i restart the server, the problem remains and electrum server isn't public.

BadAss.Sx

legendary

Activity: 1526

Merit: 1002

Bulletproof VPS/VPN/Email @ BadAss.Sx

Thanks, it is working now....apparently is was the size limits issue, because server came online and i can connect now with my client

JWU42

legendary

Activity: 1666

Merit: 1000

Quote from: BadAss.Sx on July 04, 2015, 02:45:49 PM

Now it is syncing i guess, because cpu load it around 20% all the time, but no specific new lines in the log...so i wait until it is done with its job and see what happens then.

The log only outputs every 1000 blocks.

You can do...

Code:

electrum-server getinfo

To get an output of what block it is working on...

BadAss.Sx

legendary

Activity: 1526

Merit: 1002

Bulletproof VPS/VPN/Email @ BadAss.Sx

I did that and all went fine without errors, but now i read this in my log and can't connect to my server:

Quote

[04/07/2015-21:38:48] Signal handler called with signal 15
[04/07/2015-21:38:48] Stopping Stratum
[04/07/2015-21:38:49] Closing database...
[04/07/2015-21:38:49] Database is closed
[04/07/2015-21:38:49] Electrum Server stopped
[04/07/2015-21:40:10] Starting Electrum server on electrum.badass.sx
[04/07/2015-21:40:17] Database version 3
[04/07/2015-21:40:17] Blockchain height 363830
[04/07/2015-21:40:17] UTXO tree root hash: d229909f0021c67165fd317a9bb519125996b68335b2aff3ef4651339d5174d7
[04/07/2015-21:40:17] Coins in database: 1434314010655316
[04/07/2015-21:40:17] TCP server started on 94.102.50.70:50001
[04/07/2015-21:40:17] SSL server started on 94.102.50.70:50002
[04/07/2015-21:40:17] bitcoind still warming up...
[04/07/2015-21:40:27] bitcoind still warming up...
[04/07/2015-21:40:37] bitcoind still warming up...
[04/07/2015-21:40:47] bitcoind still warming up...
[04/07/2015-21:40:57] bitcoind still warming up...
[04/07/2015-21:41:07] bitcoind still warming up...
[04/07/2015-21:41:17] bitcoind still warming up...
[04/07/2015-21:41:27] bitcoind still warming up...
[04/07/2015-21:41:37] bitcoind still warming up...
[04/07/2015-21:41:47] bitcoind still warming up...

Any idea what is going on? In top i don't see anything strange.

When i do bitcoin-cli getblockchaininfo

Quote

{
   "chain" : "main",
   "blocks" : 363834,
   "headers" : 363834,
   "bestblockhash" : "0000000000000000011cef43d1c689b097bd15410151f6b2666a6f683a7ff659",
   "difficulty" : 49402014931.22746277,
   "verificationprogress" : 0.99999607,
   "chainwork" : "00000000000000000000000000000000000000000008295a3c16c8d1fb7190dc"
}

But when i read the electrum-server log

Quote

[04/07/2015-21:57:55] Starting Electrum server on electrum.badass.sx
[04/07/2015-21:57:56] Database version 3
[04/07/2015-21:57:56] Blockchain height 363830
[04/07/2015-21:57:56] UTXO tree root hash: d229909f0021c67165fd317a9bb519125996b68335b2aff3ef4651339d5174d7
[04/07/2015-21:57:56] Coins in database: 1434314010655316
[04/07/2015-21:57:56] TCP server started on 94.102.50.70:50001
[04/07/2015-21:57:56] SSL server started on 94.102.50.70:50002

It looks like the blockchain height isn't synchronising properly because it says 363830 instead of 363834.

EDIT: changed the file limits to 64k to see what that does, but i had it already static in limits.conf, but when i did ulimit -n it told me 1024. Because electrum-server needs to be run under root i have added the same lines for root also in the limits.conf. Now it is syncing i guess, because cpu load it around 20% all the time, but no specific new lines in the log...so i wait until it is done with its job and see what happens then.

JWU42

legendary

Activity: 1666

Merit: 1000

Assumes you are starting in your home directory and the server is in ~/electrum-server

Code:

cd electrum-server
git pull
sudo python setup.py install

BadAss.Sx

legendary

Activity: 1526

Merit: 1002

Bulletproof VPS/VPN/Email @ BadAss.Sx

Hi, how do i upgrade...is there a simple method?

Sorry for asking if it is too much asked

tl121

sr. member

Activity: 278

Merit: 254

Quote from: ThomasV on July 01, 2015, 03:06:26 AM

if you run an electrum server, it is time to upgrade!

Two fixes were added recently:
- support for arbitrary p2sh scripts (will be required by electrum 2.4 client)
- memory pool efficiency fix, added after the recent 'stress test' (servers without this patch are too slow)

note: 2.4 clients will only connect to servers that have upgraded

Thanks for the update. It was quick and easy to install.

The server seems to run smoother between blocks. However, when a new full block shows up the server still takes a long time to process the block, over two minutes for a full 1 MB block. In the past 12 hours electrum-server has consumed more than four times as much CPU time as bitcoin core. The server machine is completely CPU bound while processing a new block. The machine hardware is a Haswell Atom machine, running with 8 GB of RAM and SSD.

Topic: Electrum server discussion thread - page 7. (Read 61487 times)