Topic: Electrum, high availability and horizontal scalability (Read 194 times)

The risk of SPV wallet in general, even though in reality it almost never happen (AFAIK it's only when Bitcoin Cash is born/forked from Bitcoin).


4. Use third-party hosting. But obviously it's not good option and BTCPay itself made an article about it at https://docs.btcpayserver.org/ThirdPartyHosting/

running a full node is not some complex endeavor. you just run bitcoin core software and it does the sync automatically. it just takes time to download the blockchain. you can enable pruning to save disk space.

Thanks for your answer, especially the points you mentioned about Electrum server-side limitations.

You all agree on the fact that Electrum is not the best solution in that case so I'm going to stop swimming against the tide from now )

I'm gonna give BTCPayserver a second chance and alongside with that, trying to run a full node!

 

Honestly, the best solution is a full node... attempting to "workaround" Electrum's limitations with regards to large volumes of transactions/addresses etc is only going to end in tears

You'll either create a solution that doesn't scale well or becomes a nightmare to manage.

Note that the limitations of "large volume" aren't necessarily a "client" issue, so much as a "server" bandwidth imposed limit. The most common server (ElectrumX) has some "default" limits that will be triggered by a client requesting large amounts of data (ie. when trying to sync a wallet with 1000s of addresses/transactions etc). When that happens, the client is throttled/timed out and it appears as though "syncing" is stuck or broken.

If you run your own Electrum server, these limits can easily be modified to workaround this limitation... However this leads to this:
You'll find that all Electrum servers require access to a Bitcoin Node anyway... so we're back to the solution of: "just run a Bitcoin Node"


So... in summary, I can see 3 options:

1. Find a friendly Electrum Server admin who would be willing to setup a server for you with modified limits
or
2. Setup your own Bitcoin Node + Electrum Server setup with modified limits
or
3. Setup your own Bitcoin Node + BTCPayServer

Personally, I'd probably go with #3... it's the least "hacky" solution using tools designed specifically for the job you appear to be trying to accomplish.

Like I said, currently I can't build a payment process on top of a full node (and going live with it)... i need more experience in Bitcoin/Blockchain architecture to achieve this efficiently.

Although Electrum seems to have issues with large wallets, its interface comply wth most of my actual business requirements.

Except large wallets, could you tell me what, in your opinion, what could possibly go wrong with Electrum ?


Yes BTCPayserver is obviously not centralized. I was basically mentioning that BTCPay is compatible with Bitpay API. Sorry for that confusion.

I tried to run a self-hosted BTCPay instance some weeks ago to test. I gave up because there was a lot of dependencies, options and a complexity which is not required in my case.

Thanks for your answer

no just run a full node


btcpayserver is not centralized. it needs bitcoin core running. with bitcoin core i.e. a full node you have the blockchain on disk so you can handle very large wallets.

In the other hand, if I run a new wallet instance every time "10" addresses have been assigned to a payment request (funded or not), it could become a substantial waste of resource on my side...not to mention managing hundreds or thousands of wallets is probably a funny nightmare.

Another solution would be to not reuse an address for a "very raisonnable" period of time in order to mitigate this risk.


I'm actually trying to estimate that design weakness and finding a creative way to bypass it... even though I don't really know where I'm going at right now but it's getting better)

As far as I know you can run your own Electrum server and make all your wallets connect to that particular server only. However, it could become a single point of failure. Of course you can also run your own Bitcoin node if this is a critical business requirement. But in the end (IMO) we always have to rely on a third-party provider at some point.


I guess this architecture could survive as long as it follows very simple rules and avoid complexity as much as possible (I mean, in terms of wallet management/swap policy)


Yes I already took a (short) look at Bitpay, as well as BTCPay server which is API compliant, as a possible alternative. However at this point I actually need to avoid centralized services as well as third-party payment providers. This is a technical/business requirement.

Thanks again for your answers and point of view!

That's an interesting point of view. I guess JSON is a core design choice made by the developer(s) to avoid dependencies and because they assume that most users won't have to deal with a large wallet. However a database backend would be a merchant requirement for sure. It should be studied more closely.
 

It's more of a hassle if the User A accidentally sends funds to the address after it's expiration but it has already been assigned to User B. Such a scenario can be tricky to handle.

That being said, I think you might be able to do with having addresses that doesn't have any transactions since those wouldn't incur any resource penalty when querying them with an Electrum server. I missed that.


I wouldn't say Electrum is the best candidate because it isn't designed for heavy workload in the first place and having to rely on someone else's Bitcoin node to tell you the correct information doesn't sit too well with me. You do have a point with the issue about xpub, since Bitcoin Core won't derive any xpub. IMO, it would be complicated to manage and swap wallets every now and then. If that's something that you're able to accommodate then I don't think there's too much of a problem. If you'd like, there are services like Bitpay which helps to streamline the process, at the expense of it being a centralised service.

Yes I was wrong, gab limit is not a limitation nor an issue (except when you have to restore a wallet using a seed if I understood well). I assumed gab limit could be used to delimit how many address should be used per wallet instance, but this is not relevant at all. I still need to do some homework)

In my situation I guess I can mitigate this by expiration times and limitating how many pending requests a user can generate at the same time.

On architecture side, one solution would be to look for expired requests and reuse addresses, assuming that reusing an address which has never been funded is an acceptable privacy risk. But again, I may be missing something.



I didn't know about solutions such as getrawmempool thanks!

I'm mainly using Electrum because I guess this wallet provides a "lightweight" additionnal layer as well as some valuable features (secret seed, client-side stored private keys, socks5 proxy support, etc.). At this point, I don't have enough knowledge nor experience to deal with a full node directly.

The problem comes from the fact that Electrum writes the wallet file in simple JSON format which it looks up later so a better approach would be to use everything in Electrum except parts of the wallet layer.
It would require more work but theoretically with some modification to the code you could replace the wallet file (the way it is written to disk and read) with a database instead so that looking up inside millions of inputs becomes super fast and use that with the rest of the Electrum code for signing, downloading txs and blocks from nodes, ...

I imagine that it'll be a hassle to keep track of the wallets. You'll probably have to change out for a new wallet everyday. Gap limit is not a limitation, it's just how far your wallet will read ahead so no issues there. You would have to account for those who would generate an address for payment but not pay in the end as well.
It's more of the few developers-centric RPCs (getrawmempool,etc) which wouldn't really affect normal users.

Hi @ranochigo,

Thanks for your answer.

Maybe I should have started by looking at Bitcoin Core to have a better overview. I will take a look at it asap.

To be more precise, the idea behind this would be to automatically create a new Electrum wallet and run it as a read-only wallet (in a new containerized instance) each time a limitation is reached (e.g gap limit), avoiding as much as possible the limitations you mentioned. This is theory.

Could you give me some details about what is missing (in your opinion) in Electrum RPC interface ?



  




  

The way I've seen and recommend people to implement if they're using Electrum is to only include the master public key on the production server and store the seeds and the main wallet on a separate server. The payment is usually done by checking against some block explorer APIs or through an Electrum running in watch-only mode on the server.

Okay that aside, the main problem doesn't arise until you need to reload the wallet and/or spend the huge number of UTXOs. Spending such transactions or loading them will cause Electrum to lag as it will involves large amount of data being transferred from the Electrum servers and IIRC, some has restrictions on it as well.

I don't recommend people using Electrum as a merchant because it has loads of limitations combined with the lackluster RPC interface (perhaps I'm asking for too much) that I've encountered. Bitcoin Core is a much better choice in comparison and you won't face much problems either.

Hey,

I've been reading some topics on the message board stating that Electrum wasn't design to handle wallets with a large addresses/transactions history, although there are no hard-coded limitations.

In a merchant situation where a lot of addresses and transactions occur on a daily basis (e.g naivly 100 addresses for 100 transactions a day), does it make sense to "load balance" payments using multiple wallets (e.g naivly 1 wallet for 10 transactions a day = 10 instances of wallets for 100 tansactions a day) ? Does anybody experienced such architecture ?

Thanks