Pruning Blockchain and creating multi wallets

ABCbits

legendary

Activity: 2856

Merit: 7410

Crypto Swap Exchange

Quote from: faak on May 24, 2022, 08:57:42 AM

1. The library is popular 200 stars on github
2. I can generate a private key and use it to made operations with the library. That's using https://libbitcoin.info/
3. As I understand, that is only cryptography operations, so that can handle thousand address.

AFAIK libbitcoin is fine library. But libbitcoin have 10+ library where each have different function, make sure you use one that meets your need.

Quote from: faak on May 24, 2022, 08:57:42 AM

But yes I will have to maintain a bitcoin core even if I use this option, so why not using directly my bitcoin core RPC API with one wallet and multiple address ?

Not if you use libbitcoin-server which is full node client with query function. But based on my experience, i would recommend Bitcoin Core over libbitcoin-server.

ranochigo

legendary

Activity: 2954

Merit: 4158

Quote from: faak on May 24, 2022, 08:57:42 AM

But yes I will have to maintain a bitcoin core even if I use this option, so why not using directly my bitcoin core RPC API with one wallet and multiple address ?

Exactly.

Quote from: faak on May 24, 2022, 08:57:42 AM

Someone told me that I can use this service from blockchain.com : https://github.com/blockchain/service-my-wallet-v3
I have to install a service on my server and I will be able to use an API.
Do you know more about that ?

Of course 3rd party API is the best way but the cost is incredible few thousands $ by month... (Bitgo for example)

They are not the best way. Most 3rd party APIs don't hold any responsibilities for any inaccuracies or any downtime in their APIs. You are much better off (and cheaper) to just setup your own Bitcoin Core and run it on your own server. Relying on someone else when you're hosting such an important service is not the way to go.

faak

newbie

Activity: 7

Merit: 5

Quote from: ?? on ??

Quote from: faak on May 24, 2022, 05:28:02 AM

@NotATether you are suggesting to generate a private key for every user, that something I can do. I found library to do that, I am able to generate private key of each user and get associated btc address without having bitcoin core. So I have to find the solution to create and sign a transaction from a private key.

1. How popular is the library?
2. Does the library have option to backup/restore private keys?
3. Have you checked whether it can handle thousand address?

Quote from: ranochigo on May 24, 2022, 05:39:57 AM

The problem is that you still need a source to get the transactions, as well as the data for your transactions. You cannot do this without running an instance of Bitcoin Core or another full node/client.

You could use 3rd party API, but it's definitely not ideal option when you handle money.

1. The library is popular 200 stars on github
2. I can generate a private key and use it to made operations with the library. That's using https://libbitcoin.info/
3. As I understand, that is only cryptography operations, so that can handle thousand address.

But yes I will have to maintain a bitcoin core even if I use this option, so why not using directly my bitcoin core RPC API with one wallet and multiple address ?

Someone told me that I can use this service from blockchain.com : https://github.com/blockchain/service-my-wallet-v3
I have to install a service on my server and I will be able to use an API.
Do you know more about that ?

Of course 3rd party API is the best way but the cost is incredible few thousands $ by month... (Bitgo for example)

ranochigo

legendary

Activity: 2954

Merit: 4158

Quote from: faak on May 24, 2022, 05:28:02 AM

Many of you are saying there is not the good option to use Bitcoin core for my use-case, thanks for the information.
@NotATether you are suggesting to generate a private key for every user, that something I can do. I found library to do that, I am able to generate private key of each user and get associated btc address without having bitcoin core. So I have to find the solution to create and sign a transaction from a private key.

So If I resume, the best solution to manage the creation and the managing of thousands of wallets, is not to use bitcoin core but to generate private key for each user and create raw bitcoin transaction and sign it with this private key.

Everyone agree with it ?

The problem is that you still need a source to get the transactions, as well as the data for your transactions. You cannot do this without running an instance of Bitcoin Core or another full node/client.

The solution that you've proposed can work, but you have to be careful to audit and understand the code when you're using it and not just ripping the code off somewhere else. Bitcoin Core is still a better solution, because you have the entire RPC API for you to utilize and all of the functions being built into it. The threshold for Bitcoin Core to slow down is quite high, and you shouldn't really face a problem with it unless you're having an extremely big operation, for which you should invest in a better server.

Again, you don't have to manage thousands of wallets. You can easily just create a single wallet with all of the addresses and tag each unique address to each user. Best practice is to generate a new address for the user after a single transaction.

faak

newbie

Activity: 7

Merit: 5

Thanks all for you reply, you are very helpful

I have the budget to store all the blockchain but if I can avoid it, I will.

Many of you are saying there is not the good option to use Bitcoin core for my use-case, thanks for the information.
@NotATether you are suggesting to generate a private key for every user, that something I can do. I found library to do that, I am able to generate private key of each user and get associated btc address without having bitcoin core. So I have to find the solution to create and sign a transaction from a private key.

So If I resume, the best solution to manage the creation and the managing of thousands of wallets, is not to use bitcoin core but to generate private key for each user and create raw bitcoin transaction and sign it with this private key.

Everyone agree with it ?

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: faak on May 23, 2022, 03:30:40 PM

So yes, anyway I will parse the transactions and store it in database, so I have the balance but if a security breach exist and someone are able to withdrawal more than he really have in his wallet, he will be able because the main wallet will have the money of everybody. It is very dangerous whereas if I create a wallet for each user, this case is not possible.
What do you think ? Maybe you know some security tips to avoid it ?

I think you shouldn't build an exchange from scratch. Ten years ago "homebrew" solutions were the reason so many exchanges lost a lot of money. Nowadays, I don't think you should do this if your budget doesn't even allow to store the full blockchain on a node.

NotATether

legendary

Activity: 1568

Merit: 6660

bitcoincleanup.com / bitmixlist.org

Quote from: faak on May 23, 2022, 03:30:40 PM

The only bad thing with this option is to retrieve a balance of specific address. I checked and the only possiblity to get a address balance is to parse all the transactions.
So yes, anyway I will parse the transactions and store it in database, so I have the balance but if a security breach exist and someone are able to withdrawal more than he really have in his wallet, he will be able because the main wallet will have the money of everybody. It is very dangerous whereas if I create a wallet for each user, this case is not possible.
What do you think ? Maybe you know some security tips to avoid it ?

Even creating a single wallet with many addresses inside, one for each user, will pose a scalability problem for you when you get thousands of users, as the load time for opening the wallet will increase severely.

You should look into creating the transactions from the private keys manually - this does mean that you'll need to generate a private key for each user (and store it inside a database), and there are some libraries that can help facilitate this (depending on the programming language you choose), but it's not a good idea to outsource this functionallity to Bitcoin Core. It wasn't designed to be an exchange backend.

ranochigo

legendary

Activity: 2954

Merit: 4158

Quote from: faak on May 23, 2022, 03:30:40 PM

The only bad thing with this option is to retrieve a balance of specific address. I checked and the only possiblity to get a address balance is to parse all the transactions.
So yes, anyway I will parse the transactions and store it in database, so I have the balance but if a security breach exist and someone are able to withdrawal more than he really have in his wallet, he will be able because the main wallet will have the money of everybody. It is very dangerous whereas if I create a wallet for each user, this case is not possible.
What do you think ? Maybe you know some security tips to avoid it ?

Using a wallet system for this is not a solution, because as your user base grows, you'll run out of resources to handle that many wallets in Bitcoin Core. The problem that you highlighted isn't really a concern if you know how to properly code and maintain your database. Using a wallet system to segregate your funds isn't a solution because you are operating an exchange, not a wallet. There are tons of transfers within the users themselves either through buying or selling other coins. If you were to initiate a transfer everytime the user makes a transaction, then things can get very messy quite quickly.

I would encourage you to explore the possibility of building and managing your own backend which would greatly simplify your workflow. Do hire an experienced developer if you want to properly manage your site, the community here can only guide you so much.

faak

newbie

Activity: 7

Merit: 5

Quote from: LoyceV on May 23, 2022, 09:21:02 AM

Quote from: faak on May 23, 2022, 08:54:33 AM

I already thought about the possibility of generating only address on the same wallets but the problem will be on the sending transaction. I need to send BTC from specific address but if I use the same wallet with many address, the sending address will always be the same.

If you enable Coin Control in Bitcoin Core, you can manually select each input.

Thanks for the tip with Coin Control, you are a BTC core genious !

The only bad thing with this option is to retrieve a balance of specific address. I checked and the only possiblity to get a address balance is to parse all the transactions.
So yes, anyway I will parse the transactions and store it in database, so I have the balance but if a security breach exist and someone are able to withdrawal more than he really have in his wallet, he will be able because the main wallet will have the money of everybody. It is very dangerous whereas if I create a wallet for each user, this case is not possible.
What do you think ? Maybe you know some security tips to avoid it ?

Quote from: LoyceV on May 23, 2022, 09:21:02 AM

Quote

I am working on a crypto exchange platform, so I will have to generate multiple wallets on differents blockchains for each of users.

For an exchange, there's no need to use the user's deposited inputs for their withdrawals. Usually people withdraw a different coin than they deposited.

Users can use their wallets as a simple wallet, so they can receive and send crypto on it, it will be not only used to exchanges cryptos.

ranochigo

legendary

Activity: 2954

Merit: 4158

Quote from: faak on May 23, 2022, 08:47:19 AM

I don't know if it can be use on my case because I have to generate new wallets from my server and not on the user device. Do you think it can be possible to set my server as a client to generate multiple wallets ?

SPV wallets would probably not be very good for any service in general, simply because they are very sensitive on both the security and the privacy aspect. I would probably recommend you to get a stronger server instead, because if you can't hold the entire blockchain with your storage, then your server has a good chance of not being strong enough.

Using multiple wallets won't be viable for use-case with loads of individual wallets because that would result in a huge delay when synchronizing and loading up the various wallets while offering little to no benefits in terms of segregation. Having a database with the different addresses tagged to each user would be a far better option both in terms of organization and efficiency.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: faak on May 23, 2022, 08:54:33 AM

I already thought about the possibility of generating only address on the same wallets but the problem will be on the sending transaction. I need to send BTC from specific address but if I use the same wallet with many address, the sending address will always be the same.

If you enable Coin Control in Bitcoin Core, you can manually select each input.

Quote

I am working on a crypto exchange platform, so I will have to generate multiple wallets on differents blockchains for each of users.

For an exchange, there's no need to use the user's deposited inputs for their withdrawals. Usually people withdraw a different coin than they deposited.

faak

newbie

Activity: 7

Merit: 5

Quote from: LoyceV on May 23, 2022, 06:17:13 AM

Quote from: faak on May 22, 2022, 03:54:09 PM

this node will allow me to create multiple wallets and send transactions from each of them. Is it possible to create a wallet now and have to send a transaction in 5 years.

Is there a reason to use multiple wallets, instead of many different addresses in one wallet? That might be easier.
Also, if you're keeping funds on an online wallet for 5 years, depending on the value you may want to look at offline cold storage instead.

Quote

But I have to regulary check if on each wallet a new transaction is coming on, thats why I need a long history of data.

Depending on what service you're offering (I'm thinking it's either an exchange, casino or webshop), you'll need to check each wallet very offen, so you won't need a long pruned history.

Quote

I would like to save data space because I will have to sync 10+ differents blockchains...

Can you share what you're planning?

Hi @LoyceV,

Thanks for reply.

I already thought about the possibility of generating only address on the same wallets but the problem will be on the sending transaction. I need to send BTC from specific address but if I use the same wallet with many address, the sending address will always be the same.

You have right on this point, I will regulary check and store transactions on my database for each wallets, so I don't really need to have the whole blockchain history. That's a good point

I am working on a crypto exchange platform, so I will have to generate multiple wallets on differents blockchains for each of users.
So we need to have something sure and secure on the long terms.

NeuroticFish

legendary

Activity: 3500

Merit: 6205

Looking for campaign manager? Contact icopress!

Quote from: faak on May 23, 2022, 08:47:19 AM

I don't know if it can be use on my case because I have to generate new wallets from my server and not on the user device. Do you think it can be possible to set my server as a client to generate multiple wallets ?

If you want to create something like e-commerce, creating different deposit address for each of your customers can be done in any wallet.
And if the number of customers (and transactions) is not too big, even a SPV wallet can handle. But if it's something more serious, the wallet software will have to be Bitcoin Core, not a SPV wallet.
But: for this you need only one wallet, you don't need to create different wallet per customer, it's enough to create new address per customer, all in the same wallet.

However, you better confirm your actual use case....

faak

newbie

Activity: 7

Merit: 5

Quote from: ranochigo on May 23, 2022, 04:03:28 AM

Quote from: faak on May 22, 2022, 03:54:09 PM

@DaveF : You have right, this node will allow me to create multiple wallets and send transactions from each of them. Is it possible to create a wallet now and have to send a transaction in 5 years. But I have to regulary check if on each wallet a new transaction is coming on, thats why I need a long history of data. But as you said, maybe I can use my node for wallet creation/send transaction and an external API to check if new transaction exist for wallets.
I would like to save data space because I will have to sync 10+ differents blockchains...

If storage space is really a concern for you, then wouldn't an SPV wallet be a more suitable alternative for you? Electrum would suffice, without having external API or stuff like that.

The wallet will be synchronized from the point which you've closed it. It shouldn't really be a problem because I think that all of the wallets are being initialized and synchronized at the same time regardless of if you're opening it or not. CMMIW on that, but it obviously wouldn't apply if you're importing a wallet in the future.

Hi @ranochigo,

Thanks for help.
I don't know if it can be use on my case because I have to generate new wallets from my server and not on the user device. Do you think it can be possible to set my server as a client to generate multiple wallets ?

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: faak on May 22, 2022, 03:54:09 PM

this node will allow me to create multiple wallets and send transactions from each of them. Is it possible to create a wallet now and have to send a transaction in 5 years.

Is there a reason to use multiple wallets, instead of many different addresses in one wallet? That might be easier.
Also, if you're keeping funds on an online wallet for 5 years, depending on the value you may want to look at offline cold storage instead.

Quote

But I have to regulary check if on each wallet a new transaction is coming on, thats why I need a long history of data.

Depending on what service you're offering (I'm thinking it's either an exchange, casino or webshop), you'll need to check each wallet very offen, so you won't need a long pruned history.

Quote

I would like to save data space because I will have to sync 10+ differents blockchains...

Can you share what you're planning?

ranochigo

legendary

Activity: 2954

Merit: 4158

Quote from: faak on May 22, 2022, 03:54:09 PM

@DaveF : You have right, this node will allow me to create multiple wallets and send transactions from each of them. Is it possible to create a wallet now and have to send a transaction in 5 years. But I have to regulary check if on each wallet a new transaction is coming on, thats why I need a long history of data. But as you said, maybe I can use my node for wallet creation/send transaction and an external API to check if new transaction exist for wallets.
I would like to save data space because I will have to sync 10+ differents blockchains...

If storage space is really a concern for you, then wouldn't an SPV wallet be a more suitable alternative for you? Electrum would suffice, without having external API or stuff like that.

The wallet will be synchronized from the point which you've closed it. It shouldn't really be a problem because I think that all of the wallets are being initialized and synchronized at the same time regardless of if you're opening it or not. CMMIW on that, but it obviously wouldn't apply if you're importing a wallet in the future.

faak

newbie

Activity: 7

Merit: 5

Thanks all for your relevant replies !

@DaveF : You have right, this node will allow me to create multiple wallets and send transactions from each of them. Is it possible to create a wallet now and have to send a transaction in 5 years. But I have to regulary check if on each wallet a new transaction is coming on, thats why I need a long history of data. But as you said, maybe I can use my node for wallet creation/send transaction and an external API to check if new transaction exist for wallets.
I would like to save data space because I will have to sync 10+ differents blockchains...

Cookdata

hero member

Activity: 910

Merit: 875

Not Your Keys, Not Your Bitcoin

Quote from: ranochigo on May 22, 2022, 11:35:32 AM

No good way to really do this because prune works by the size of the blockchain and not an absolute block height. It is not what it is really designed to do anyways.

I didn't put some things into consideration. At first, I thought that if the OP could view the transaction history of his addresses before pruning, he might be able to trace the first transaction of his address to the specific block where those files are saved on the local drive, and then calculate the total amount of space from that block to the current block and prune the rest of the file. For example:
If OP had its first transaction in block K and was able to determine the overall size of blocks from K to the present one, he may simply prune=N where N is the total amount he wishes to keep (MB) on his computer. I'm not sure the strategy would work.

The best option is to restart a full blockchain download, or never prune your node in the first place.

ranochigo

legendary

Activity: 2954

Merit: 4158

Quote from: Cookdata on May 22, 2022, 09:36:14 AM

Another suggestion is that, if you complete initial blockchain download and you have all your wallet history verify, you can prune the node to first day your wallets had it first transaction and delete the rest of the file.

No good way to really do this because prune works by the size of the blockchain and not an absolute block height. It is not what it is really designed to do anyways.

I'm currently running a pruned wallet on one of my PCs and I've never opened the other wallets for a few months and they seem to be synchronized just fine. I'm inclined to think that they are actually initialized at the start of the synchronize and they synchronize in tandem so it isn't really an issue if you don't open any of the wallets for a long time. I could be wrong, but I know that at least the wallets are checked at start up.

DaveF

legendary

Activity: 3458

Merit: 6231

Crypto Swap Exchange

If you are ever going to need any old transaction data from before the data that you have not pruned it's going to be an issue.

I have been getting grumpy about this but will say it again, if you want to run a node for business and cost the storage difference of a pruned node vs a full node makes a difference they you should not be doing it that way. There are many ways to get the data using API calls to services. That one time you need something that is not on your node for some reason the entire process comes crashing down. Use the right tools for the job.

-Dave

Topic: Pruning Blockchain and creating multi wallets (Read 273 times)