Obelisk: An Electrum server using libbitcoin

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Quote from: Parazyd on April 21, 2021, 02:52:35 AM

If you have an unstable power source or something, you can enable the "flush_writes" option in the config file. This will slow down the sync a bit, but it should avoid corrupting the databases.

Thanks, i'll try it one more time. But how come it's not mentioned on configuration file, GitHub's wiki or GitHub's issue.

Edit: I don't do anything, but the file suddenly corrupted in middle of sync. However quick SMART test/info doesn't show any problem. Looks like this time i encountered bug.

Code:

FATAL [node] Failure organize block for slot (0), store is now corrupted: lock failure

Quote from: Parazyd on April 21, 2021, 02:52:35 AM

The comments in the config file tell you what needs to be changed to work on mainnet/testnet.

Yeah, i already do as what config file said.

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Quote from: Parazyd on April 16, 2021, 09:40:02 AM

Finished most of the code now, and almost all is covered by test units

Great. I'm still syncing libbitcoin-server testnet after modifying bs.cfg few times (had to change many checkpoint, BIP checkpoint and other variable specific for testnet).

Edit: i didn't know libbitcoin-server is fragile. Power outage and stop the application using SIGTERM (because it didn't respond to Ctrl-C and SIGINT) causes the database to corrupt and the only way i know to solve the problem is by sync from 0 again.

Edit 2: Since i use regular computer which isn't turned on 24/7, i decide to turn off the computer. However libbitcoin-server doesn't respond to Ctrl-C and SIGINT again. I waited about half an hour where there's no progress mentioned on console and log file. I'll just wait until libbitcoin v4 is stable.

Parazyd

hero member

Activity: 812

Merit: 587

Space Lord

Quote from: ABCbits on April 18, 2021, 07:52:24 AM

Quote from: Parazyd on April 16, 2021, 09:40:02 AM

Finished most of the code now, and almost all is covered by test units

Great. I'm still syncing libbitcoin-server testnet after modifying bs.cfg few times (had to change many checkpoint, BIP checkpoint and other variable specific for testnet).

Edit: i didn't know libbitcoin-server is fragile. Power outage and stop the application using SIGTERM (because it didn't respond to Ctrl-C and SIGINT) causes the database to corrupt and the only way i know to solve the problem is by sync from 0 again.

Edit 2: Since i use regular computer which isn't turned on 24/7, i decide to turn off the computer. However libbitcoin-server doesn't respond to Ctrl-C and SIGINT again. I waited about half an hour where there's no progress mentioned on console and log file. I'll just wait until libbitcoin v4 is stable.

If you have an unstable power source or something, you can enable the "flush_writes" option in the config file. This will slow down the sync a bit, but it should avoid corrupting the databases. By default libbitcoin decides to keep most of the stuff in memory and just do periodic writes to disk.

But yeah, v4 is still in development so there are bugs for sure. However, Eric tells me that he's back on working on libbitcoin full time again so I expect it to be more stable sooner than later.

On my local testnet libbitcoin-server, I modified the configuration to the testnet values and didn't change anything else except "use_libconsensus=true" from the defaults. The comments in the config file tell you what needs to be changed to work on mainnet/testnet.

Parazyd

hero member

Activity: 812

Merit: 587

Space Lord

Finished most of the code now, and almost all is covered by test units

libbitcoin-server v4 still needs some code written, regarding tx broadcasting, but hopefully this will be finished soon.

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Quote from: Parazyd on April 14, 2021, 04:52:44 AM

One thing I recommend is to first sync testnet, it's about 30 gigabytes only and play with that before going mainnet.

Good thing you mentioned it, i was thinking which file should i move to external drive. Coincidentally, i also run Bitcoin Core on testnet, so i can save some time and internet bandwidth.

Parazyd

hero member

Activity: 812

Merit: 587

Space Lord

Quote from: ABCbits on April 14, 2021, 04:43:40 AM

Quote from: Parazyd on April 13, 2021, 07:10:43 AM

Strange. if this prefix is correct, then the binary should be in $HOME/.local/bin/bs, and the config file should be in $HOME/.local/etc/libbitcoin/bs.cfg
You can run it with the full path, after you configure bs.cfg. If they're not there, then something did error (or you perhaps ran install.sh with another user's privileges so it ended up in a different $HOME, like /root/.local).

I'm sure i didn't use different user or sudo to run the script. However, i solved the problem simply by replacing $HOME/.local with absolute path (on new directory solely for libbitcoin). I can run ./bs -v without any problem.

Code:

Version Information:

libbitcoin-server: 4.0.0
libbitcoin-protocol: 4.0.0
libbitcoin-node: 4.0.0
libbitcoin-blockchain: 4.0.0
libbitcoin: 4.0.0

Now i just need to configure libbitcoin-server and perform initial sync. However, i just notice i'll have 2 copy of Bitcoin blockchain with different format (owned by Bitcoin Core and libbitcoin), so i'll free my HDD for libbitcoin's blockchain. There shouldn't be any more problem and i assume libbitcoin will take few days to sync since i use HDD before i can test Obelisk.

Quote from: Parazyd on April 13, 2021, 07:10:43 AM

For what it's worth, the autotools is generally the tool that is ran in a standalone context. The install.sh script also downloads all other necessary dependencies, like libbitcoin-system, and libbitcoin-node and does all that for you transparently.

Yeah, i noticed that.

Ok, glad you got it set up.
One thing I recommend is to first sync testnet, it's about 30 gigabytes only and play with that before going mainnet.

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Quote from: Parazyd on April 13, 2021, 07:10:43 AM

Strange. if this prefix is correct, then the binary should be in $HOME/.local/bin/bs, and the config file should be in $HOME/.local/etc/libbitcoin/bs.cfg
You can run it with the full path, after you configure bs.cfg. If they're not there, then something did error (or you perhaps ran install.sh with another user's privileges so it ended up in a different $HOME, like /root/.local).

I'm sure i didn't use different user or sudo to run the script. However, i solved the problem simply by replacing $HOME/.local with absolute path (on new directory solely for libbitcoin). I can run ./bs -v without any problem.

Code:

Version Information:

libbitcoin-server:     4.0.0
libbitcoin-protocol:   4.0.0
libbitcoin-node:       4.0.0
libbitcoin-blockchain: 4.0.0
libbitcoin:            4.0.0

Now i just need to configure libbitcoin-server and perform initial sync. However, i just notice i'll have 2 copy of Bitcoin blockchain with different format (owned by Bitcoin Core and libbitcoin), so i'll free my HDD for libbitcoin's blockchain. There shouldn't be any more problem and i assume libbitcoin will take few days to sync since i use HDD before i can test Obelisk.

Quote from: Parazyd on April 13, 2021, 07:10:43 AM

For what it's worth, the autotools is generally the tool that is ran in a standalone context. The install.sh script also downloads all other necessary dependencies, like libbitcoin-system, and libbitcoin-node and does all that for you transparently.

Yeah, i noticed that.

Parazyd

hero member

Activity: 812

Merit: 587

Space Lord

Quote from: ?? on ??

Quote from: Parazyd on April 13, 2021, 06:31:13 AM

Ah, for libbitcoin-server, it's best if you run the install.sh script.

Something like:

Code:

git clone https://github.com/libbitcoin/libbitcoin-server
cd libbitcoin-server
./install.sh --build-boost --build-zmq --disable-shared --prefix=$HOME/.local

These install scripts are made for easy deployment and a self-contained build process.

Actually i've used the install.sh and follow the guide step by step. I also tried Autotools, but there's error when i run ./configure which says libbitcoin-node is missing. But since you mention different parameter, i'll try it now.

Edit: Although there are many warning (such as "extra ;"), there's no single error. However, i can't find the executable bs even though i check $HOME/.local is on $PATH variable. Using find command, i also can't find the executable on $HOME/.local.

Strange. if this prefix is correct, then the binary should be in $HOME/.local/bin/bs, and the config file should be in $HOME/.local/etc/libbitcoin/bs.cfg
You can run it with the full path, after you configure bs.cfg. If they're not there, then something did error (or you perhaps ran install.sh with another user's privileges so it ended up in a different $HOME, like /root/.local).

If it's not a hassle for you, I can help you out if there's a way for you to share the install.sh build output somehow.

For what it's worth, the autotools is generally the tool that is ran in a standalone context. The install.sh script also downloads all other necessary dependencies, like libbitcoin-system, and libbitcoin-node and does all that for you transparently.

Parazyd

hero member

Activity: 812

Merit: 587

Space Lord

Ah, for libbitcoin-server, it's best if you run the install.sh script.

Something like:

Code:

git clone https://github.com/libbitcoin/libbitcoin-server
cd libbitcoin-server
./install.sh --build-boost --build-zmq --disable-shared --prefix=$HOME/.local

These install scripts are made for easy deployment and a self-contained build process.

By the way, if you're interested, also check out libbitcoin-explorer, which is complementary to libbitcoin-server and can be used for querying.

Parazyd

hero member

Activity: 812

Merit: 587

Space Lord

Quote from: ABCbits on April 13, 2021, 04:07:19 AM

Quote from: Parazyd on April 12, 2021, 12:54:42 PM

Keep in mind that libbitcoin v4 is still not fully finished, but Eric and the community are working on it.

Yeah, the README.md isn't complete and didn't mention that libzmq is required for compiling. libzmq repository shows there's .deb file, but there are dependency problem when i attempt to install it with apt and dpkg. I would rather not risking broke my system right now.

In apt-based distros there is python3-zmq which should do the trick.

EDIT: I added some more notes in the Readme. Thanks for the feedback.

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Quote from: Parazyd on April 12, 2021, 12:54:42 PM

Keep in mind that libbitcoin v4 is still not fully finished, but Eric and the community are working on it.

Yeah, the README.md isn't complete and didn't mention that libzmq is required for compiling. libzmq repository shows there's .deb file, but there are dependency problem when i attempt to install it with apt and dpkg. I would rather not risking broke my system right now.

Parazyd

hero member

Activity: 812

Merit: 587

Space Lord

Quote from: ABCbits on April 11, 2021, 04:08:33 AM

Quote from: Parazyd on April 11, 2021, 03:33:57 AM

Yeah, 1.4 up to 1.4.2., but still a few things need to be finished which I need help with, or just more spare time.

Basically Electrum 3.3.0 and newer are supported, at least according to these
https://github.com/spesmilo/electrum/blob/3.3.0/electrum/version.py#L4
https://github.com/spesmilo/electrum/blob/3.2.4/electrum/version.py#L4

Quote from: Parazyd on April 11, 2021, 03:33:57 AM

Will do @ setup.py
I've been planning to make such a thing so Obelisk can even be installed with pip.

That's great, i'll try it if i can setup libbitcoin-server without any major problem.

I'm still reading how to setup libbitcoin-server and found out how to make it connect only to my Bitcoin Core client with peer = localhost:8333 on bs.cfg. However, i can't find out how to configure RAM usage and whether HDD will cause bottleneck or not.

Thanks

Keep in mind that libbitcoin v4 is still not fully finished, but Eric and the community are working on it.
I'm not sure if RAM usage is configurable, but you shouldn't go OOM or something.

~~I'll try to get some setup.py ready soon.~~ Done.

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Quote from: Parazyd on April 11, 2021, 03:33:57 AM

Yeah, 1.4 up to 1.4.2., but still a few things need to be finished which I need help with, or just more spare time.

Basically Electrum 3.3.0 and newer are supported, at least according to these
https://github.com/spesmilo/electrum/blob/3.3.0/electrum/version.py#L4
https://github.com/spesmilo/electrum/blob/3.2.4/electrum/version.py#L4

Quote from: Parazyd on April 11, 2021, 03:33:57 AM

Will do @ setup.py
I've been planning to make such a thing so Obelisk can even be installed with pip.

That's great, i'll try it if i can setup libbitcoin-server without any major problem.

I'm still reading how to setup libbitcoin-server and found out how to make it connect only to my Bitcoin Core client with peer = localhost:8333 on bs.cfg. However, i can't find out how to configure RAM usage and whether HDD will cause bottleneck or not.

Parazyd

hero member

Activity: 812

Merit: 587

Space Lord

Yeah, 1.4 up to 1.4.2., but still a few things need to be finished which I need help with, or just more spare time.

Will do @ setup.py
I've been planning to make such a thing so Obelisk can even be installed with pip.

Parazyd

hero member

Activity: 812

Merit: 587

Space Lord

Quote from: NotATether on April 10, 2021, 02:03:23 AM

Quote from: Parazyd on April 09, 2021, 12:44:10 PM

The eventual next step would be to merge Obelisk directly into Electrum and avoid the server boilerplate.

Do you think it's doable? Sure it would be nice if we could just fire up an Electrum server with a --server switch, but the libbitcoin dependency could be an obstacle to merging. You have to go to the package manager to install deps like Qt5 and libsecp256k1 so the logical thing to do is make sure that it's in distribution's official repositories, and not some PPA or community maintained repo.

libbitcoin is not a direct dependency. Obelisk simply uses zeromq to talk to a libbitcoin server and queries for information.

It is very much doable to have it merged into Electrum, but it will mostly depend on the maintainers and the community to choose if they want to move forward this way or not.

NotATether

legendary

Activity: 1568

Merit: 6660

bitcoincleanup.com / bitmixlist.org

Quote from: Parazyd on April 09, 2021, 12:44:10 PM

The eventual next step would be to merge Obelisk directly into Electrum and avoid the server boilerplate.

Do you think it's doable? Sure it would be nice if we could just fire up an Electrum server with a --server switch, but the libbitcoin dependency could be an obstacle to merging. You have to go to the package manager to install deps like Qt5 and libsecp256k1 so the logical thing to do is make sure that it's in distribution's official repositories, and not some PPA or community maintained repo.

Parazyd

hero member

Activity: 812

Merit: 587

Space Lord

The eventual next step would be to merge Obelisk directly into Electrum and avoid the server boilerplate.

Parazyd

hero member

Activity: 812

Merit: 587

Space Lord

Quote from: Dabs on April 09, 2021, 12:34:03 PM

Quote from: Parazyd on April 09, 2021, 12:31:08 PM

That one requires Bitcoin Core. All servers like this that use Bitcoin core are inefficient in one way or another Wink

Oh, so how does your Electrum server work then? Where does it get it's blockchain data for the addresses and coins and all those other stuff?

*edit* oh, libbitcoin = is another full node, like Bitcoin Core but different?

Yes, libbitcoin-server is a full node and a query server.

The existing implementations do their separate indexing on top of Core which is extremely inefficient.
libbitcoin is the fastest implementation of Bitcoin and it already indexes everything that is necessary.

You can use your own libbitcoin server or public instances, it's up to the user to choose.

Dabs

legendary

Activity: 3416

Merit: 1912

The Concierge of Crypto

Quote from: Parazyd on April 09, 2021, 12:31:08 PM

That one requires Bitcoin Core. All servers like this that use Bitcoin core are inefficient in one way or another Wink

Oh, so how does your Electrum server work then? Where does it get it's blockchain data for the addresses and coins and all those other stuff?

*edit* oh, libbitcoin = is another full node, like Bitcoin Core but different?

Parazyd

hero member

Activity: 812

Merit: 587

Space Lord

That one requires Bitcoin Core. All servers like this that use Bitcoin core are inefficient in one way or another Wink

Dabs

legendary

Activity: 3416

Merit: 1912

The Concierge of Crypto

https://github.com/cculianu/Fulcrum/releases

This one is Fulcrum. Have you tried that? There was an update just today I think, version 1.5.2. How does your server compare with that one? It also runs on Windows.

Parazyd

hero member

Activity: 812

Merit: 587

Space Lord

Quote from: NotATether on April 09, 2021, 08:59:01 AM

Well it's still a work in progress since the methods for peers and drawing a histogram are stuns.

Histograms can be drawn with matplotlib but I don't think libbitcoin can help with making a list of peers or inserting yourself into other people's peer lists. I wonder how many more lines of Python will that electrum-specific functionality take plus the existing 1000.

The peer stuff is trivial, since it's just about telling other servers about itself, and other servers can be retrieved through DNS seeds.
I didn't implement this yet because I don't really need it, and because I want the code reviewed first.

The mempool fee histogram stuff is deprecated since protocol 1.4.2, so I'll just drop that. The 1.5 protocol is going to have some new additions which I will have to follow up after the spec is written.

NotATether

legendary

Activity: 1568

Merit: 6660

bitcoincleanup.com / bitmixlist.org

Well it's still a work in progress since the methods for peers and drawing a histogram are stuns.

Histograms can be drawn with matplotlib but I don't think libbitcoin can help with making a list of peers or inserting yourself into other people's peer lists. I wonder how many more lines of Python will that electrum-specific functionality take plus the existing 1000.

Parazyd

hero member

Activity: 812

Merit: 587

Space Lord

Quote from: ABCbits on April 09, 2021, 06:05:33 AM

Since i never use libbitcoin, i'll just give basic feedback,
1. Have you compare the performance with ElectrumX or other Electrum server implementation?
2. AFAIK Electrum have it's own protocol and updated few times, so which Electrum (client wallet) version supported by Obelisk?
3. Is it right to assume Obelisk only support Linux?

1.
I haven't yet. This is something I would appreciate help with, ideally from someone who is experienced in this kind of profiling.

2.
Electrum uses the ElectrumX server protocol as found in https://electrumx-spesmilo.readthedocs.io/en/latest/protocol-methods.html
I followed and implemented the spec.

3.
I develop and use it on Linux, but there should be no blockers to running it on any system that can use Python 3.

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Since i never use libbitcoin, i'll just give basic feedback,
1. Have you compare the performance with ElectrumX or other Electrum server implementation?
2. AFAIK Electrum have it's own protocol and updated few times, so which Electrum (client wallet) version supported by Obelisk?
3. Is it right to assume Obelisk only support Linux?

Parazyd

hero member

Activity: 812

Merit: 587

Space Lord

I've been working on an Electrum server implementation that uses ZeroMQ and libbitcoin as its backend so that I can use Electrum with my libbitcoin node/server.
It is written with using libbitcoin v4 in mind, so with public servers, only testnet is available because v4 is not yet finished, or if you have your own mainnet v4 server, then you can use mainnet.

The code can be found here: https://github.com/parazyd/obelisk

In total, it's about 1000 lines of Python code, which is relatively straightforward. This means it's 10 times smaller than ElectrumX for example.

If you're interested, I'd appreciate some feedback, help, and code review Cheesy

Happy hacking!

Topic: Obelisk: An Electrum server using libbitcoin (Read 280 times)