Download and Upload big file to blockchain program

kzv

legendary

Activity: 1722

Merit: 1285

OpenTrade - Open Source Cryptocurrency Exchange

Quote from: BrewMaster on October 30, 2019, 12:50:09 PM

since i think continuing to discuss an ancient subject is pointless i redirect you to here: https://github.com/bitcoin/bitcoin/pull/3737

https://github.com/bitcoin/bitcoin/pull/3737#issuecomment-36383977

Quote

gavinandresen commented on 1 Mar 2014

I agree with @gidgreen .

Words like "abuse" and "rape" aren't helpful. If you think the size of the blockchain is a big issue, then please help optimize it. Because even without OP_RETURN, the blockchain will get bigger. Forever.

I think this comment no nead any comment?

BTW arbitrary data may be stored (and long and successfully saved) not only with OP_RETURN. I hope you heard something about "cryptografity"?

Moreover I can say that OP_RETURN is very impractical way for storing data in blockchain.
Much more effective is the using P2SH (or P2WSH) scripts like this

Code:

OP_DROP
OP_DROP
OP_TRUE

This is a more complex but with this way we can store over 1000 bytes in each output of standart transaction

BrewMaster

legendary

Activity: 2128

Merit: 1293

There is trouble abrewing

Quote from: kzv on October 29, 2019, 11:26:19 AM

...

of course you can't prevent people from using a "feature" that bitcoin has. but that doesn't make uploading an arbitrary file to bitcoin blockchain less of an abuse. and this is not just my opinion! most others are sharing the same views.
https://en.bitcoin.it/wiki/OP_RETURN

Quote

Many members of the Bitcoin community believe that use of OP_RETURN is irresponsible in part because Bitcoin was intended to provide a record for financial transactions, not a record for arbitrary data.

since i think continuing to discuss an ancient subject is pointless i redirect you to here: https://github.com/bitcoin/bitcoin/pull/3737

nc50lc

legendary

Activity: 2534

Merit: 6080

Self-proclaimed Genius

Quote from: Maktubeer on October 29, 2019, 09:52:50 PM

Quote from: nc50lc on October 29, 2019, 02:28:29 AM

Don't you think POS (Proof of Stake) is a better solution than POW for this kind of projects?

I think about it but i afraid create super-centralized system because if will not competition miner set censure data... I think we can sacrifice the stability of the system due to greater decentralization... But you can create analog with POS if you want.. the more options we create, the sooner we will find the best.

If the problem is centralization; then whether it's POW or POS, in the end the number of full nodes will be limited to a number or users due to an inevitably huge blockchain size.

But for POS, the incentive to run a full node works "natively" (like I explained above) and I'm sure that you'll likely encounter problems with POW with a file-sharing blockchain.
It's just a suggestion, you can try it anytime later after some tests with your POW coin.

Maktubeer

newbie

Activity: 14

Merit: 2

Quote from: HeRetiK on October 28, 2019, 07:11:05 AM

Apart from that you should be able to find the default constants being set here (starting at line 54):
https://github.com/bitcoin/bitcoin/blob/b688b859dbb2b5af2e9d19cae9dce3e3e14bd2c1/src/validation.h

its work! i edit: DEFAULT_ANCESTOR_LIMIT DEFAULT_ANCESTOR_SIZE_LIMIT DEFAULT_DESCENDANT_LIMIT DEFAULT_DESCENDANT_SIZE_LIMIT and now i can make up loading speed... Thank!!!

Quote from: nc50lc on October 29, 2019, 02:28:29 AM

Don't you think POS (Proof of Stake) is a better solution than POW for this kind of projects?

I think about it but i afraid create super-centralized system because if will not competition miner set censure data... I think we can sacrifice the stability of the system due to greater decentralization... But you can create analog with POS if you want.. the more options we create, the sooner we will find the best.

Quote from: LoyceV on October 29, 2019, 03:14:31 AM

A blockchain is only as "forever" as the people who use it.

I understand it. if people want pay for "forever" options, this is what they need. if not they can use torrent or ipfs or youtube or ..... e.t.c

Quote from: kzv on October 29, 2019, 08:32:31 AM

Custom fork is excessive for this task IMHO

Yes. i just want find a optimal parameters allowing to store max number bit without big problem for miners...

Quote from: BrewMaster on October 29, 2019, 09:44:53 AM

the "bitcoin" blockchain is a place to only store [money] transactions not files. the fact that it offers a special feature with an OP code shouldn't lead you to attempt doing that.

but firs block in bitcoin have text message (array of bits). what does that mean in you opinion? may by we should use bitcoin like twitter? why not?

Quote from: ABCbits on October 29, 2019, 12:58:11 PM

That's because miner don't need to run full node to mine Bitcoin, only pool or miners who use P2Pool who need to run full nodes.

I took about solo miners... i think it not be great problem because i can buy for example this EC2 it's not very expensive especially if i will have explorer with advertising or something like this

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Quote from: Maktubeer on October 28, 2019, 11:40:34 PM

Quote from: joniboini on October 28, 2019, 10:33:12 PM

Not to mention storing huge data is not an easy task, and considering the block space, it might take weeks to save a single file.

but bitcoin blockchain size > 120 Gb and it not problem for miners. if blockchain size will be > 2 TB miners can use ZFS + ssd raid + Broadband network.
Users don't need have full node. they can use thin client for download/upload files.. or not?

That's because miner don't need to run full node to mine Bitcoin, only pool or miners who use P2Pool who need to run full nodes.

Besides, while most user won't run full nodes, most Bitcoiner agree that running full nodes should be cheap.

Quote from: BrewMaster on October 29, 2019, 09:44:53 AM

the "bitcoin" blockchain is a place to only store [money] transactions not files. the fact that it offers a special feature with an OP code shouldn't lead you to attempt doing that.

As long as the consensus allows it, people will try to do it. Even without OP_RETURN, there are many attempt to store files on Bitcoin's blockchain.
See http://www.righto.com/2014/02/ascii-bernanke-wikileaks-photographs.html

kzv

legendary

Activity: 1722

Merit: 1285

OpenTrade - Open Source Cryptocurrency Exchange

Quote from: Dabs on October 29, 2019, 09:54:59 AM

If you need a relatively small file (less than 1 GB would be small enough) to last longer than a year, you can pay for a few services that will keep your file backed up or available for download.

Not all people live in countries where there is freedom of speech.
Imagine that there are countries in which Internet access is strictly regulated.
A decentralized repository of information will be a real breath of fresh air for people from these countries.
Cloud storages of commercial companies are forced to cooperate with corrupt authoritarian leaders in order to be able to conduct business in their countries.

Quote from: BrewMaster on October 29, 2019, 09:44:53 AM

in my opinion you shouldn't think about the cost (fees) or size of blockchain and scalability, and things like that.

in my opinion no one has the right to tell others what and how they should think

Quote from: BrewMaster on October 29, 2019, 09:44:53 AM

the "bitcoin" blockchain is a place to only store [money] transactions not files.

No way!
This is your opinion, but not a mandatory rule.
Anyone can manage their own bitcoins whatever they want. So if someone want to store files in Bitcoin blockchain then he can pay fee and do that. No one should try to dictate terms here!

Dabs

legendary

Activity: 3416

Merit: 1912

The Concierge of Crypto

So, I go on the pirate bay, click on movies, and find one that was first uploaded in 2012, it still has 600 seeders today, 7 years later. They have no financial incentive to seed it, it's just a popular movie, ... about 1.5 gigs only.

So, what do you need stored on a blockchain that can be stored anywhere else anyway? At one time (don't know if true today) Blizzard let you download their latest games or updates using some sort of torrent or peer to peer tech, that included Diablo, Warcraft and Starcraft.

If you need a relatively small file (less than 1 GB would be small enough) to last longer than a year, you can pay for a few services that will keep your file backed up or available for download.

Otherwise, you can look up filecoin, storj as others have mentioned or a few of those other storage coins. As well as trying to save larger files on BSV as they increased their block sizes.

There are or were hybrid attempts where the files are stored on IPFS and a blockchain was used to index certain files, keeping only a hash, or the hash of a hash or something small. They were called video coins or something similar.

BrewMaster

legendary

Activity: 2128

Merit: 1293

There is trouble abrewing

in my opinion you shouldn't think about the cost (fees) or size of blockchain and scalability, and things like that. the only argument that matters is the purpose of the blockchain you are trying to insert the file into.

the "bitcoin" blockchain is a place to only store [money] transactions not files. the fact that it offers a special feature with an OP code shouldn't lead you to attempt doing that.

Quote from: Maktubeer on October 25, 2019, 06:54:42 PM

now i create my own cryptocurrency(file hosting)

the question you should be thinking about is, why do you need an immutable distributed ledger to store files? and what would be the advantage of that.

for example the current existing systems have the advantage of being able to remove a file from existence. something that is not possible in an immutable ledger. so what is your plan for when (note that it is not a matter of if only when) someone uploads a malware to this ledger? or worse, things that are illegal?

kzv

legendary

Activity: 1722

Merit: 1285

OpenTrade - Open Source Cryptocurrency Exchange

Quote from: LoyceV on October 29, 2019, 08:06:38 AM

On the other hand, there's a new blockchain without any proven record of being able to store even moderate amounts of data.
Needless to say, I pick the first one

Anyone can store data in Bitcoin blockchain. It will be expensive but as reliable as possible. This is for most critical data.
If data is not too critical, anyone can store data in Litecoin, Dogecoin or other compatible forks.

Custom fork is excessive for this task IMHO

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: kzv on October 29, 2019, 07:36:52 AM

Why such difficulties if there is a method of reliable storage in just one place?

I think we'll have to disagree on which method is more reliable:
On the one hand, there's mirrors that have stored huge amounts of data for decades.
On the other hand, there's a new blockchain without any proven record of being able to store even moderate amounts of data.
Needless to say, I pick the first one

kzv

legendary

Activity: 1722

Merit: 1285

OpenTrade - Open Source Cryptocurrency Exchange

Quote from: LoyceV on October 29, 2019, 03:14:31 AM

If you want to prove a certain file existed on a certain date, all you have to do is store a sha256 hash on the blockchain, and keep the file itself on several mirrors.

Hashing it that way is only may by a proof that file not changed. But saving data in blockchain is a guarantee that the data will never lost. These are slightly different things.
How many mirrors are needed to reliably protect data from loss?
Where and how should information about these mirrors be stored? May be on several other mirors?

Why such difficulties if there is a method of reliable storage in just one place?

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: Maktubeer on October 27, 2019, 09:01:45 AM

ipfs and torrent can't give guarantees saved data... if you load file in blockchain file can't be deleted and you can find it anytime and anyway.

This is true, unless:

Quote from: Maktubeer on October 25, 2019, 06:54:42 PM

now i create my own cryptocurrency(file hosting)

A blockchain is only as "forever" as the people who use it. Bitcoin's blockchain has been running stable for 10 years thanks to a financial reward for miners.

If you want to prove a certain file existed on a certain date, all you have to do is store a sha256 hash on the blockchain, and keep the file itself on several mirrors. Mirroring data has worked very well for decades, and is much easier to use than getting a file that's scattered in tiny pieces on a blockchain.

nc50lc

legendary

Activity: 2534

Merit: 6080

Self-proclaimed Genius

Don't you think POS (Proof of Stake) is a better solution than POW for this kind of projects?
Since your coin is all about storing files (big ones), then the "mining" should be done by those who's willing to store the whole blockchain.

Plus the coin will eventually become centralized to limited number of nodes in the future (for obvious reasons), so it's best to create it as a POS.
Unless there's a limit on how much a user can store (since you've mentioned that it can store unlimited data),
the blockchain will grow from 1GB to 10TB in just weeks/a month, and no one will be able to run a node unless he owns a data center.

NeuroticFish

legendary

Activity: 3668

Merit: 6382

Looking for campaign manager? Contact icopress!

Quote from: Maktubeer on October 25, 2019, 06:54:42 PM

now i create my own cryptocurrency(file hosting) and i need disable or reset too-long-mempool-chain protection in source and make it more faster... where i can find this parameters? and why this parameter set in bitcoin?

Keep in mind that you'll need to think on some good incentive for the full nodes, else nobody would store extremely big blockchain.
You may also think on some sort of sharding for your blockchain to keep the sizes at a reasonable level, just then you have to fine tune it to avoid data loss when nodes leave the network.

Maybe you discuss with the guys from DxBox project, they also store files in the blockchain https://bitcointalksearch.org/topic/dropbox-modern-cloud-storage-or-dxbox-blockchain-based-cloud-storage-solution-5189356

Maktubeer

newbie

Activity: 14

Merit: 2

Quote from: HeRetiK on October 28, 2019, 07:11:05 AM

I think this thread should answer you which settings to add to the config file:
https://bitcointalksearch.org/topic/64-too-long-mempool-chain-3791899

Apart from that you should be able to find the default constants being set here (starting at line 54):
https://github.com/bitcoin/bitcoin/blob/b688b859dbb2b5af2e9d19cae9dce3e3e14bd2c1/src/validation.h

thank you very much!

i try modified *.conf file with

limitancestorsize=99999999
limitdescendantsize=99999999

limitancestorcount=10000
imitdescedantcount=10000

maxmempool=9999999999

walletrejectlongchains=0

but this did not solve the problem. i try edit validation.h file and recompile... i write later

Quote from: joniboini on October 28, 2019, 10:33:12 PM

Won't this give another problem? Those 'low-cost coins' hash is not as high as Bitcoin, so if there's an attack, your file might be affected too.

Maybe. 51% attack might be easier in this case. but i think in bitcoin and litcoin difficulty is redundant and miner can transfer part of the cost from creating 00000000... in hash to file storage without risk.
Of course if set max_block_size very bigger we will have problem with low hash rate. Just need find a balance(compromise) between security and system bandwidth.

Quote from: joniboini on October 28, 2019, 10:33:12 PM

Not to mention storing huge data is not an easy task, and considering the block space, it might take weeks to save a single file.

but bitcoin blockchain size > 120 Gb and it not problem for miners. if blockchain size will be > 2 TB miners can use ZFS + ssd raid + Broadband network.
Users don't need have full node. they can use thin client for download/upload files.. or not?

joniboini

legendary

Activity: 2170

Merit: 1789

Quote from: Maktubeer on October 27, 2019, 09:01:45 AM

Yes, it true if we took about bitcoin... but if using existing coin with low fee, like dogecoin cost can be low..
it is more environmentally friendly because miner need buy ssd instead ASIC and it spends less electricity.

Won't this give another problem? Those 'low-cost coins' hash is not as high as Bitcoin, so if there's an attack, your file might be affected too. Not to mention storing huge data is not an easy task, and considering the block space, it might take weeks to save a single file.

Dabs

legendary

Activity: 3416

Merit: 1912

The Concierge of Crypto

Torrents will survive as long as there are seeders. Files stored on IPFS, as long as at least one node is keeping it. There are entire movies still available that were uploaded 3 to 5 years ago.

If you need files accessible for a shorter time period and smaller file sizes, there are plenty of file hosting sites that will keep your file for 30 days or 3 months or something like that.

I use a few of those when travelling internationally and don't want any data stored on the laptop or computer as devices can be searched at airports or worse copied. You could also just encrypt the files with a secure and long password.

If you really want to store large files on a blockchain, you should try BCH and BSV, they have really large blocks.

HeRetiK

legendary

Activity: 3122

Merit: 2178

Playgram - The Telegram Casino

Quote from: Maktubeer on October 27, 2019, 09:01:45 AM

Quote from: joniboini on October 26, 2019, 11:21:24 PM

Blockchain is not suitable to store a huge file. The cost outweighs the benefit.

Yes, it true if we took about bitcoin... but if using existing coin with low fee, like dogecoin cost can be low..
it is more environmentally friendly because miner need buy ssd instead ASIC and it spends less electricity.

Be aware that the moment you start storing huge files on low-fee cryptocurrencies you'll see either (a) the fee increasing significantly, due to lack of available block-space or (b) the blockchain facing serious scalability issues due to each node having to validate and propagate huge amounts of data. It sounds like a fun project nonetheless though.

Quote from: Maktubeer on October 25, 2019, 06:54:42 PM

now i create my own cryptocurrency(file hosting) and i need disable or reset too-long-mempool-chain protection in source and make it more faster... where i can find this parameters? and why this parameter set in bitcoin?

I think this thread should answer you which settings to add to the config file:
https://bitcointalksearch.org/topic/64-too-long-mempool-chain-3791899

Apart from that you should be able to find the default constants being set here (starting at line 54):
https://github.com/bitcoin/bitcoin/blob/b688b859dbb2b5af2e9d19cae9dce3e3e14bd2c1/src/validation.h

Maktubeer

newbie

Activity: 14

Merit: 2

Quote from: joniboini on October 26, 2019, 11:21:24 PM

Why not use IPFS or torrent?

ipfs and torrent can't give guarantees saved data... if you load file in blockchain file can't be deleted and you can find it anytime and anyway.

Quote from: joniboini on October 26, 2019, 11:21:24 PM

Blockchain is not suitable to store a huge file. The cost outweighs the benefit.

Yes, it true if we took about bitcoin... but if using existing coin with low fee, like dogecoin cost can be low..
it is more environmentally friendly because miner need buy ssd instead ASIC and it spends less electricity.

Quote from: joniboini on October 26, 2019, 11:21:24 PM

That being said, you might be interested in Filecoin.

Thank you. But i find it troubles because it seems like ipfs and data can be removed after time. or I'm wrong?
however i think cryptocurrency like dogecoin and cash can give cheap recording space... but need less time between blocks

joniboini

legendary

Activity: 2170

Merit: 1789

Why not use IPFS or torrent?

Blockchain is not suitable to store a huge file. The cost outweighs the benefit. That being said, you might be interested in Filecoin.

Topic: Download and Upload big file to blockchain program (Read 461 times)