Topic: Massive Download [ETH] smart contracts (Read 99 times)

@starmyc thanks for the feedback once again.

By the way, I'm still in the process of setting up a full node in order to store the data in a traditional database.

Do you have any idea how this can be achieved? I'm running geth, and it has been doing operations for about 2 hours already:


I'm not even sure what it is doing...

Assuming it's synchronising but I honestly have no idea what's going on. And if so, once the process is done... where is the data being stored?

Contract owners can upload their contract to etherscan.io, which compile it to check that the contract is the same in the blockchain, before showing it on their site. This is why you'll find contracts with source code and others without on it.

I never studied contract decompilation or disassembling though..

I'll proceed to modify the source code or make a small extension, and try with a small portion of the blockchain, if it works, then it's just about scaling. I'll report here in case it works.

Hi starmyc,

Thanks for helping to sort out this equation, which I find really interesting.

I have found that etherscan.io actually hold information about the solidity source code of each contract. However I don't really know if this information is the actual code before being compiled to bytecode, or if it's a simple reconstruction of the code using a decompiler, which isn't the same for my goal.

https://etherscan.io/address/0xa701122c1b67220a8b6883d03c8ad67896b12466#code

Seems like etherscan.io has an API as well, which is available here: https://etherscan.io/apis

However it's limited to 5 requests per second, and the options are quite limited, hence it will probably take an ethernity to clone all desired information.

I have found that it's possible to get closer to the goal, using geth, and then processing the data, moreover, I have been recommended to use QuickBlocks (https://quickblocks.io/) which is mostly based on C++ and seems to work quite fast.

Yet, the code is compiled and I'm not yet aware of any pre-coded function that allows to clone each smart contract into a db (even the compiled version).

I've found as well that it's possible to decompile the code, at least partially using comaeio/porosity (https://github.com/comaeio/porosity) which makes a quick replacement based on the detected expressions. As following:

I'm still wondering exactly which functions or which modifications might be required to make QuickBlocks store the desired information into BigQuery or MongoDB.

The idea seems possible, however not simple to achieve.

You won't be able to use the Ethereum blockchain to retrieve contracts because they are stored in their compiled form in it, and you would need to disassemble them unfortunately.

A lot of contract creators upload their contracts on ethercan.io though. But etherscan.io doesn't provide an API to download them neither.

I'm looking for the most valid path to download and keep downloading every single smart contract on the ethereum network.

As I've been programming for a few years, I've been thinking about using cURL to make this. But I'm aware this is probably very outdated as it should be possible to be done somehow using the blockchain...

I'm willing to store all the contracts in a traditional MySQL database, in order to run multiple security audits on them. Is there any other database engine compatible with PHP that you would recommend to store them?

How would you approach this?

Kind regards;
Chris