But probably right, because it is not a Bitcoin technical discussion. 
Topic: Ethereum scanner for USDT transfers (Read 83 times)
Yeah, I posted it first in "Development & Technical Discussion", but some admin moved it But probably right, because it is not a Bitcoin technical discussion.
But probably right, because it is not a Bitcoin technical discussion. 
Thanks, this would explain it. If it is traded within an exchange, then it doesn't leave a trace on the blockchain.
Yeap, and I think the term should be used to determine what you refer to as on-chain transfer volume. As I know, other on-chain analytic website does offer to show the transfer volume data, like Glassnode for example. I suggest you can compare between the number calculated by the tool with other trusted on-chain analytical sites. As an example, you can see a successful on-chain transfer volume of USDT on Glassnode, the number shows ~3.8B USD on 20th November. You can see it here. I think it might be helpful if you compare what those sites number tells and with what your tool show, it would help to test whether the number on the tools given is probable or not.
Just a side note, I think altcoin boards are having a lack of technical discussion or solutions, specifically about some of your issues, unfortunately, others may find it hard to help you, so do I since I didn't have experience playing around with web3 yet.
Thanks, this would explain it. If it is traded within an exchange, then it doesn't leave a trace on the blockchain.
But I guess there is still a remaining bug. If I sum all transaction from 20th November, the sum is only 2.5 billion USDT. But coinmarketcap says it was 72 billion USDT. So maybe I missed another function call?
If I'm not mistaken, your tools are only gathering data from the on-chain transactions. On other hand, there are some movements within the exchange that not present in the data, which it makes the number is vastly different. And CMC does calculate their volume based on the reported exchanges data.See reference below:
Volume (Cryptoasset)
The volume of any cryptoasset is the total spot trading volume reported by all exchanges over the last 24 hours for that cryptoasset. Some market pairs are excluded from the sum, denoted by two asterisks (**) on the markets tab, if the exchange does not enforce a trading fee or otherwise offers significant incentives to trade on the market pair. Market pairs with these characteristics are rather susceptible to wash trading, resulting in artificially inflated reported volumes. From our experience, we have found that it is better to exclude these markets to give a better representation of relative trading volumes for the crypto market.
The volume of any cryptoasset is the total spot trading volume reported by all exchanges over the last 24 hours for that cryptoasset. Some market pairs are excluded from the sum, denoted by two asterisks (**) on the markets tab, if the exchange does not enforce a trading fee or otherwise offers significant incentives to trade on the market pair. Market pairs with these characteristics are rather susceptible to wash trading, resulting in artificially inflated reported volumes. From our experience, we have found that it is better to exclude these markets to give a better representation of relative trading volumes for the crypto market.
I fixed the problem with the "Approve" function call. I created a hash from the signature "Transfer(address,address,uint256)", which is then used in the topics array as the first element. With this I could test it and don't get the "Approve" calls anymore.
But I guess there is still a remaining bug. If I sum all transaction from 20th November, the sum is only 2.5 billion USDT. But coinmarketcap says it was 72 billion USDT. So maybe I missed another function call?
BTW, pretty high volume. I did a sort for the highest transactions, and there were 2 with 111 million USDT, just yesterday:
https://etherscan.io/tx/0x95220146c7ec639d001171b4323bb0acafb59da223272b556c0fc8c940250252
https://etherscan.io/tx/0x71c663b408ec4e52fe1bb9899e1f0f4b02023c875f34222aed1d87aabefaf0d7
Some big whales.
But I guess there is still a remaining bug. If I sum all transaction from 20th November, the sum is only 2.5 billion USDT. But coinmarketcap says it was 72 billion USDT. So maybe I missed another function call?
BTW, pretty high volume. I did a sort for the highest transactions, and there were 2 with 111 million USDT, just yesterday:
https://etherscan.io/tx/0x95220146c7ec639d001171b4323bb0acafb59da223272b556c0fc8c940250252
https://etherscan.io/tx/0x71c663b408ec4e52fe1bb9899e1f0f4b02023c875f34222aed1d87aabefaf0d7
Some big whales.
I improved the code: Now it uses a binary search to find the first block to check. So if you want to list all transaction from one day of last year, the previous version needed hours, now only a few minutes. Probably could be improved further with web3 batch requests.
I wrote a scanner which prints all transfers of a specified time and amount range:
https://github.com/FrankBuss/ethscan
Example output:
See the TODO section for possible improvements. And as with my last project, I'm still relatively new to Rust and alls these Rust crypto crates, so my code might not use it very idiomaticly or could be written shorter. Feel free to critizise my code, or send PRs.
If other tokens should be integrated, the contract needs to be read to determine the number of decimal places, and maybe the ticker label etc. And I'm not sure if transaction_receipt is the right way to get the transfer amount (and from/to addresses, currently unused). Shouldn't all the required data be in the transaction?
https://github.com/FrankBuss/ethscan
Example output:
Quote
search filter:
date from: 2021-11-20 00:00:00 UTC
date to: 2021-11-21 00:00:00 UTC
amount from: 10000000 USDT
amount to: 1000000000000000000000000000000000000000000 USDT
time (UTC),amount (USDT),from address,to address,transaction
2021-11-20 21:50:23,50000000.000000,0xed344fa1075499dac4e7eb0b868a1874dcdd36cf,0xb8ac6dec186496ad25ef7fdc7f341d93db928d30,0x32cc61b3fc064fd7e7ceab562755316645b72af158a7f4925cdc4f86210d17f5
2021-11-20 16:02:08,10000000.000000,0x6262998ced04146fa42253a5c0af90ca02dfd2a3,0x46340b20830761efd32832a74d7169b29feb9758,0x1ceb25243b19c0ee13bacf476e29ec8a95a317c49ea8581c1984115bb6544e46
2021-11-20 13:03:42,111000000.000000,0x7119cb953db332a3b60730d5d39e7b1ba5120e86,0x0548f59fee79f8832c299e01dca5c76f034f558e,0x95220146c7ec639d001171b4323bb0acafb59da223272b556c0fc8c940250252
2021-11-20 10:01:23,18010797.000000,0x2dc754a674bb130fca3b0e28e5faa1fe719e6756,0x28c6c06298d514db089934071355e5743bf21d60,0xd18cc0c4fad232ad36799bd36f5df05c9d5b9b3233e3aa4ffda215a45f421486
2021-11-20 09:13:42,10000000.000000,0xad6eaa735d9df3d7696fd03984379dae02ed8862,0xf2103b01cd7957f3a9d9726bbb74c0ccd3f355d3,0xf8fb2666804b859d919a3f5888a1a478697334188e6ffe0f9a4af1d8bfd9ed9a
2021-11-20 07:52:42,15800000.000000,0xcf04a13e8b3cbc7770c2347ba8d6f1a1b6605edc,0xda816e2122a8a39b0926bfa84edd3d42477e9efd,0x95dadd60f1ed11d373795da8d8fafc45acc2abce543195601ac3df2f885a2b6b
2021-11-20 07:44:50,13011811.600000,0x4862733b5fddfd35f35ea8ccf08f5045e57388b3,0x83ca25fc236d7dd31e6e5536f94283e3005dc93f,0x87be78f1c81185dcb00d4db1b2deca75d7c304b361aa2c9fc4b9b997d02490d0
2021-11-20 01:56:32,111000000.000000,0x4862733b5fddfd35f35ea8ccf08f5045e57388b3,0x7119cb953db332a3b60730d5d39e7b1ba5120e86,0x71c663b408ec4e52fe1bb9899e1f0f4b02023c875f34222aed1d87aabefaf0d7
2021-11-20 01:30:33,20000000.000000,0x6262998ced04146fa42253a5c0af90ca02dfd2a3,0x46340b20830761efd32832a74d7169b29feb9758,0x7ded214dd8d12948dbcd95d1cfa2a671ce73cefe581dcbedb64bd80fcb2065c3
2021-11-20 01:09:23,49999976.000000,0x21a31ee1afc51d94c2efccaa2092ad1028285549,0x4862733b5fddfd35f35ea8ccf08f5045e57388b3,0xb0818a17731109a91b7b8bf02d926b5423d626c7ed8ad697308e49efabc984e7
2021-11-20 00:35:36,59999976.000000,0xdfd5293d8e347dfe59e90efd55b2956a1343963d,0x4862733b5fddfd35f35ea8ccf08f5045e57388b3,0x1c0fa7d9233130fd311b24e4aad142fed9002845e8a92b55501410afd24b9479
2021-11-20 00:31:52,29305092.000000,0xdfd5293d8e347dfe59e90efd55b2956a1343963d,0x4862733b5fddfd35f35ea8ccf08f5045e57388b3,0xc574481cf7eab8f51386c1cb746f1bf6a00f1383d35cb9afd898018c1dba53ce
2021-11-20 00:22:18,19000000.000000,0x28c6c06298d514db089934071355e5743bf21d60,0x46340b20830761efd32832a74d7169b29feb9758,0x13201a4b51c534e86ee425ea37985b579aaac1390dffc8d2d68c526c6850e1b0
number of transactions: 13, amount sum: 517127652.60 USDT
date from: 2021-11-20 00:00:00 UTC
date to: 2021-11-21 00:00:00 UTC
amount from: 10000000 USDT
amount to: 1000000000000000000000000000000000000000000 USDT
time (UTC),amount (USDT),from address,to address,transaction
2021-11-20 21:50:23,50000000.000000,0xed344fa1075499dac4e7eb0b868a1874dcdd36cf,0xb8ac6dec186496ad25ef7fdc7f341d93db928d30,0x32cc61b3fc064fd7e7ceab562755316645b72af158a7f4925cdc4f86210d17f5
2021-11-20 16:02:08,10000000.000000,0x6262998ced04146fa42253a5c0af90ca02dfd2a3,0x46340b20830761efd32832a74d7169b29feb9758,0x1ceb25243b19c0ee13bacf476e29ec8a95a317c49ea8581c1984115bb6544e46
2021-11-20 13:03:42,111000000.000000,0x7119cb953db332a3b60730d5d39e7b1ba5120e86,0x0548f59fee79f8832c299e01dca5c76f034f558e,0x95220146c7ec639d001171b4323bb0acafb59da223272b556c0fc8c940250252
2021-11-20 10:01:23,18010797.000000,0x2dc754a674bb130fca3b0e28e5faa1fe719e6756,0x28c6c06298d514db089934071355e5743bf21d60,0xd18cc0c4fad232ad36799bd36f5df05c9d5b9b3233e3aa4ffda215a45f421486
2021-11-20 09:13:42,10000000.000000,0xad6eaa735d9df3d7696fd03984379dae02ed8862,0xf2103b01cd7957f3a9d9726bbb74c0ccd3f355d3,0xf8fb2666804b859d919a3f5888a1a478697334188e6ffe0f9a4af1d8bfd9ed9a
2021-11-20 07:52:42,15800000.000000,0xcf04a13e8b3cbc7770c2347ba8d6f1a1b6605edc,0xda816e2122a8a39b0926bfa84edd3d42477e9efd,0x95dadd60f1ed11d373795da8d8fafc45acc2abce543195601ac3df2f885a2b6b
2021-11-20 07:44:50,13011811.600000,0x4862733b5fddfd35f35ea8ccf08f5045e57388b3,0x83ca25fc236d7dd31e6e5536f94283e3005dc93f,0x87be78f1c81185dcb00d4db1b2deca75d7c304b361aa2c9fc4b9b997d02490d0
2021-11-20 01:56:32,111000000.000000,0x4862733b5fddfd35f35ea8ccf08f5045e57388b3,0x7119cb953db332a3b60730d5d39e7b1ba5120e86,0x71c663b408ec4e52fe1bb9899e1f0f4b02023c875f34222aed1d87aabefaf0d7
2021-11-20 01:30:33,20000000.000000,0x6262998ced04146fa42253a5c0af90ca02dfd2a3,0x46340b20830761efd32832a74d7169b29feb9758,0x7ded214dd8d12948dbcd95d1cfa2a671ce73cefe581dcbedb64bd80fcb2065c3
2021-11-20 01:09:23,49999976.000000,0x21a31ee1afc51d94c2efccaa2092ad1028285549,0x4862733b5fddfd35f35ea8ccf08f5045e57388b3,0xb0818a17731109a91b7b8bf02d926b5423d626c7ed8ad697308e49efabc984e7
2021-11-20 00:35:36,59999976.000000,0xdfd5293d8e347dfe59e90efd55b2956a1343963d,0x4862733b5fddfd35f35ea8ccf08f5045e57388b3,0x1c0fa7d9233130fd311b24e4aad142fed9002845e8a92b55501410afd24b9479
2021-11-20 00:31:52,29305092.000000,0xdfd5293d8e347dfe59e90efd55b2956a1343963d,0x4862733b5fddfd35f35ea8ccf08f5045e57388b3,0xc574481cf7eab8f51386c1cb746f1bf6a00f1383d35cb9afd898018c1dba53ce
2021-11-20 00:22:18,19000000.000000,0x28c6c06298d514db089934071355e5743bf21d60,0x46340b20830761efd32832a74d7169b29feb9758,0x13201a4b51c534e86ee425ea37985b579aaac1390dffc8d2d68c526c6850e1b0
number of transactions: 13, amount sum: 517127652.60 USDT
See the TODO section for possible improvements. And as with my last project, I'm still relatively new to Rust and alls these Rust crypto crates, so my code might not use it very idiomaticly or could be written shorter. Feel free to critizise my code, or send PRs.
If other tokens should be integrated, the contract needs to be read to determine the number of decimal places, and maybe the ticker label etc. And I'm not sure if transaction_receipt is the right way to get the transfer amount (and from/to addresses, currently unused). Shouldn't all the required data be in the transaction?
Jump to: