Binance Raw trading data | Bitcointalksearch.org

tardis_thad

newbie

Activity: 4

Merit: 0

Can't speak for other solutions but in case of https://tardis.dev price is not only for trade data, but also for tick by tick order book data which given it's vast size (in case of Binance only it's ~150GB compressed data per month) you may find price not that unbelievable (add VM + storage costs and it quickly adds up).

K_I_C_K

newbie

Activity: 1

Merit: 0

I've looked for historical trade data as well .. and I can't believe the price that is being asked for this data. I designed a python program to grab all the trade data from the api and place it in a database server .. the only problem is that the api only allows you to grab a maximum of 1000 trades per request ... so in order to grab a years worth of data, it's going to take about 10-20 days. If any one knows of a better solution, could you let me know ? and if any one would like the python code to grab the data, let me know and i'll pass it on.

silviumatei

newbie

Activity: 1

Merit: 0

Quote from: glaz330 on March 28, 2019, 10:25:40 AM

Quote from: Tytanowy Janusz on March 28, 2019, 09:08:38 AM

This is incomplete in my opinion and might be confusing for AI bots without full order book all the time. Big (fake or real) walls in order book very often push price in various directions that won't be visible for AI bot with only trade history.

Fair enough....

So do you think, that raw data in sync with an order book of say 100 positions, in both directions, is a good sample?

Ty in advance

Hi, how r u doing?

Very interesting project, particularly for reinforcement learning.
Did you see this by any chance? Looks pretty much like what you are describing: https://tardis.dev/

Frank37

sr. member

Activity: 770

Merit: 250

CryptoTalk.Org - Get Paid for every Post!

Unfortunately, many users do not understand what you are trying to do here. Even experienced BTCT users overlook the fact that you want to collect tick by tick raw data.
All here mentioned free services and the binance API can deliver this:

https://api.binance.com/api/v1/klines?symbol=BTCUSDT&interval=1m

But the smallest time interval is 1 min.

It is not trivial to do this task tick by tick, because you have to read out the trade list cyclically this way:

https://api.binance.com/api/v1/trades?symbol=BTCUSDT&limit=50

and merge it at the right place together again.

Since sometimes 10 or 20 trades per second are executed, this is not easy, because your script have to act fast.
But if your script requests data to fast you can quickly get a HTTP 429 return code (breaking a request rate limit).
And this request is only for one pair - BTCUSDT. All BTC-pairs create much more traffic and timing problems.
So it would need several data collection clients which are working together.
Keep up your work.

glaz330

newbie

Activity: 21

Merit: 0

New dates added.

glaz330

newbie

Activity: 21

Merit: 0

Quote from: upsidedown75 on March 29, 2019, 02:31:14 PM

Quote from: glaz330 on March 28, 2019, 10:16:49 AM

Quote from: Mpamaegbu on March 28, 2019, 08:21:54 AM

OP, honestly if anyone were interested initially and then seeing that you're just a Newbie would make them lose interest in your project. Who even pays for a bot services that high in price. BTW, there's nothing special about what you intend to supply as it can be found online already.

How exactly did you conclude that I am a Newbie? (on the forum yes)

It can be really funny how people can just judge the expertise of a forum user by their forum tag, I do that a lot too and until one day, it dawn on me that I was already an expert in trading before getting to know about bitcointalk and opening an account with on it.

So for the fact that you are a newbie here doesn’t mean you are not an expert in your field but the service you intend rendering is quite on a high side for now and if you can proof to us that it will help the market out of the situation it is now, maybe I can try it out too and probably broadcast it to more crypto users too who might find it useful.

I have updated the post with a link...

I 'll really appreciate your feedback.

Thank you

glaz330

newbie

Activity: 21

Merit: 0

github link with first dataset

glaz330

newbie

Activity: 21

Merit: 0

Quote from: hyunee on March 31, 2019, 12:44:50 PM

Quote from: glaz330 on March 27, 2019, 02:11:18 AM

Quote from: BitMaxz on March 26, 2019, 06:46:52 PM

They don't need to pay for raw or historical data for trading because you can just get it free from well-known tools online.

Check this sample of historical data below.

- https://www.coingecko.com/en/coins/binance-coin/historical_data/usd
- https://coinmarketcap.com/currencies/binance-coin/historical-data/

So you don't need to offer a service like this because no one would be interested.

Thank you for your answer.

I am talking for tick by tick raw data - all trades.

Can you find this kind of info for free?

Ty in advance!

I'm really sorry but I don't know what you're talking about. Can you explain it in details why do we need tick raw data?

Paper trade (test) your strategy + train an AI bot + various analyses of the data

glaz330

newbie

Activity: 21

Merit: 0

Quote from: Sujoris on March 31, 2019, 01:56:27 PM

Quote from: glaz330 on March 27, 2019, 02:11:18 AM

Quote from: BitMaxz on March 26, 2019, 06:46:52 PM

They don't need to pay for raw or historical data for trading because you can just get it free from well-known tools online.

Check this sample of historical data below.

- https://www.coingecko.com/en/coins/binance-coin/historical_data/usd
- https://coinmarketcap.com/currencies/binance-coin/historical-data/

So you don't need to offer a service like this because no one would be interested.

Thank you for your answer.

I am talking for tick by tick raw data - all trades.

Can you find this kind of info for free?

Ty in advance!

Get an API access and pull all the data to your own server. So yes, it is available for free. You only need to be smart & code-literate enough to access that data.

You are making this look like the simplest thing on the planet.

In my research I couldn't find some smart & code-literate enough to access that data AND give it for free.

Sujoris

member

Activity: 80

Merit: 46

Quote from: glaz330 on March 27, 2019, 02:11:18 AM

Quote from: BitMaxz on March 26, 2019, 06:46:52 PM

They don't need to pay for raw or historical data for trading because you can just get it free from well-known tools online.

Check this sample of historical data below.

- https://www.coingecko.com/en/coins/binance-coin/historical_data/usd
- https://coinmarketcap.com/currencies/binance-coin/historical-data/

So you don't need to offer a service like this because no one would be interested.

Thank you for your answer.

I am talking for tick by tick raw data - all trades.

Can you find this kind of info for free?

Ty in advance!

Get an API access and pull all the data to your own server. So yes, it is available for free. You only need to be smart & code-literate enough to access that data.

hyunee

full member

Activity: 476

Merit: 100

Quote from: glaz330 on March 27, 2019, 02:11:18 AM

Quote from: BitMaxz on March 26, 2019, 06:46:52 PM

They don't need to pay for raw or historical data for trading because you can just get it free from well-known tools online.

Check this sample of historical data below.

- https://www.coingecko.com/en/coins/binance-coin/historical_data/usd
- https://coinmarketcap.com/currencies/binance-coin/historical-data/

So you don't need to offer a service like this because no one would be interested.

Thank you for your answer.

I am talking for tick by tick raw data - all trades.

Can you find this kind of info for free?

Ty in advance!

I'm really sorry but I don't know what you're talking about. Can you explain it in details why do we need tick raw data?

glaz330

newbie

Activity: 21

Merit: 0

Quote from: Theb on March 31, 2019, 12:27:02 PM

So the price range for Binance's historical data range from 350$ to 1500$ which I don't think normal traders would be interested to buy since most of the data can already be provided by other free services online. The only thing I can think off that will buy this data are either their competitors or some companies in the financial industry trying to create data solutions for others. So if you are thinking that you can piece out their data and sell it cheaper it may work but you will only have a limited market for it as I don't think people will be comfortable on shelling out hundreds of dollars for a data they don't even know how to take advantage of.

Could you please provide a link with free raw historical Binance data, like the one showing in my post above?

Thank you.

Theb

hero member

Activity: 1680

Merit: 655

So the price range for Binance's historical data range from 350$ to 1500$ which I don't think normal traders would be interested to buy since most of the data can already be provided by other free services online. The only thing I can think off that will buy this data are either their competitors or some companies in the financial industry trying to create data solutions for others. So if you are thinking that you can piece out their data and sell it cheaper it may work but you will only have a limited market for it as I don't think people will be comfortable on shelling out hundreds of dollars for a data they don't even know how to take advantage of.

glaz330

newbie

Activity: 21

Merit: 0

First 50 entries starting at 2019-03-30 00:00:00 UTC

to get a glimpse

Code:

id |event_type|event_time |symbol |aggregate_trade_id|price |quantity|first_trade_id|last_trade_id|trade_time |market_maker|ignore|
---|----------|-------------|---------|------------------|----------|--------|--------------|-------------|-------------|------------|------|
1|aggTrade |1553904000025|NULSBTC | 4213613| 0.000187| 108| 4834249| 4834249|1553904000023|false |true |
2|aggTrade |1553904000059|CNDBTC | 3980877|0.00000407| 9766| 5473717| 5473717|1553904000058|false |true |
3|aggTrade |1553904000059|CNDBTC | 3980878|0.00000408| 10000| 5473718| 5473718|1553904000058|false |true |
4|aggTrade |1553904000059|CNDBTC | 3980879|0.00000409| 18663| 5473719| 5473721|1553904000058|false |true |
5|aggTrade |1553904000062|ARNBTC | 10846535|0.00011232| 19| 12083863| 12083863|1553904000058|false |true |
6|aggTrade |1553904000062|ARNBTC | 10846536|0.00011233| 744| 12083864| 12083864|1553904000058|false |true |
7|aggTrade |1553904000062|ARNBTC | 10846537|0.00011248| 100| 12083865| 12083865|1553904000058|false |true |
8|aggTrade |1553904000062|ARNBTC | 10846538| 0.0001126| 1039| 12083866| 12083867|1553904000058|false |true |
9|aggTrade |1553904000073|ADABTC | 19667521|0.00001733| 231| 25492811| 25492811|1553904000070|false |true |
10|aggTrade |1553904000109|ADABTC | 19667522|0.00001733| 84| 25492812| 25492812|1553904000106|false |true |
11|aggTrade |1553904000131|BCHSVBTC | 5616643| 0.015673| 0| 6848467| 6848467|1553904000129|true |true |
12|aggTrade |1553904000138|BNBBTC | 35103603| 0.0040185| 10| 40594951| 40594951|1553904000136|false |true |
13|aggTrade |1553904000138|BNBBTC | 35103604| 0.0040198| 27| 40594952| 40594952|1553904000136|false |true |
14|aggTrade |1553904000144|ETHBTC | 104442122| 0.034891| 0| 114690882| 114690882|1553904000142|true |true |
15|aggTrade |1553904000153|ARNBTC | 10846539| 0.0001126| 22| 12083868| 12083868|1553904000152|false |true |
16|aggTrade |1553904000192|NULSBTC | 4213614| 0.000187| 391| 4834250| 4834250|1553904000191|false |true |
17|aggTrade |1553904000248|ADABTC | 19667523|0.00001733| 5164| 25492813| 25492813|1553904000246|false |true |
18|aggTrade |1553904000251|EVXBTC | 3361090|0.00028006| 8| 4288698| 4288699|1553904000250|true |true |
19|aggTrade |1553904000251|EVXBTC | 3361091|0.00028001| 3| 4288700| 4288700|1553904000250|true |true |
20|aggTrade |1553904000278|BNBBTC | 35103605| 0.00402| 23| 40594953| 40594953|1553904000275|true |true |
21|aggTrade |1553904000278|BNBBTC | 35103606| 0.0040185| 0| 40594954| 40594954|1553904000275|true |true |
22|aggTrade |1553904000304|PPTBTC | 3740263| 0.0003651| 40| 4315780| 4315780|1553904000303|false |true |
23|aggTrade |1553904000343|NULSBTC | 4213615| 0.000187| 352| 4834251| 4834251|1553904000341|false |true |
24|aggTrade |1553904000351|ETHBTC | 104442123| 0.034896| 3| 114690883| 114690883|1553904000349|false |true |
25|aggTrade |1553904000352|ETHBTC | 104442124| 0.034898| 3| 114690884| 114690884|1553904000352|false |true |
26|aggTrade |1553904000367|BATBTC | 7191680| 0.0000693| 1375| 8602061| 8602061|1553904000365|false |true |
27|aggTrade |1553904000369|ETHBTC | 104442125| 0.034898| 1| 114690885| 114690885|1553904000367|false |true |
28|aggTrade |1553904000372|ETCBTC | 11879170| 0.001187| 51| 14399174| 14399177|1553904000369|true |true |
29|aggTrade |1553904000443|ADABTC | 19667524|0.00001733| 8440| 25492814| 25492814|1553904000440|false |true |
30|aggTrade |1553904000477|NULSBTC | 4213616| 0.000187| 2631| 4834252| 4834252|1553904000476|false |true |
31|aggTrade |1553904000496|AMBBTC | 3279987|0.00001682| 400| 3942009| 3942009|1553904000493|true |true |
32|aggTrade |1553904000496|AMBBTC | 3279988| 0.0000168| 11234| 3942010| 3942012|1553904000493|true |true |
33|aggTrade |1553904000515|EVXBTC | 3361092|0.00028007| 107| 4288701| 4288701|1553904000514|false |true |
34|aggTrade |1553904000521|NULSBTC | 4213617| 0.000187| 248| 4834253| 4834253|1553904000520|false |true |
35|aggTrade |1553904000521|ADABTC | 19667525|0.00001733| 7197| 25492815| 25492815|1553904000520|false |true |
36|aggTrade |1553904000530|ADABTC | 19667526|0.00001733| 53442| 25492816| 25492818|1553904000527|false |true |
37|aggTrade |1553904000533|LTCBTC | 20357376| 0.014972| 0| 22637179| 22637179|1553904000532|false |true |
38|aggTrade |1553904000545|ADABTC | 19667527|0.00001733| 5146| 25492819| 25492819|1553904000543|false |true |
39|aggTrade |1553904000586|EVXBTC | 3361093|0.00028007| 85| 4288702| 4288702|1553904000585|false |true |
40|aggTrade |1553904000657|DOCKBTC | 1600866|0.00000329| 1367| 2530555| 2530556|1553904000655|true |true |
41|aggTrade |1553904000665|ADABTC | 19667528|0.00001733| 1809| 25492820| 25492820|1553904000663|false |true |
42|aggTrade |1553904000721|NULSBTC | 4213618| 0.000187| 443| 4834254| 4834254|1553904000721|false |true |
43|aggTrade |1553904000743|EVXBTC | 3361094|0.00028007| 84| 4288703| 4288703|1553904000741|false |true |
44|aggTrade |1553904000826|NULSBTC | 4213619| 0.000187| 211| 4834255| 4834255|1553904000825|false |true |
45|aggTrade |1553904000850|ADABTC | 19667529|0.00001732| 115| 25492821| 25492821|1553904000847|true |true |
46|aggTrade |1553904000858|FUELBTC | 2706758|0.00000339| 2950| 4152898| 4152898|1553904000856|false |true |
47|aggTrade |1553904000946|NULSBTC | 4213620| 0.000187| 144| 4834256| 4834256|1553904000943|false |true |
48|aggTrade |1553904001006|OAXBTC | 2748544|0.00006215| 359| 3605942| 3605942|1553904001004|false |true |
49|aggTrade |1553904001019|BNBBTC | 35103607| 0.0040184| 3| 40594955| 40594955|1553904001016|true |true |
50|aggTrade |1553904001060|NULSBTC | 4213621| 0.000187| 93| 4834257| 4834257|1553904001058|false |true |

Indamuck

hero member

Activity: 1120

Merit: 554

Data has immense value. This is why all these free platforms exist, they want to collect every bit of information they can about you.

As people have said before you need a lot of trading data if you are going to make a trading bot that has any chance of turning consistent profit.

glaz330

newbie

Activity: 21

Merit: 0

Quote from: GregH37 on March 30, 2019, 02:55:27 AM

Quote from: Tytanowy Janusz on March 29, 2019, 04:56:53 AM

That's a lot of data that can feed AI bot. You can test your AI bot on those data. 1 clever algorithm and you have AI bot that has multiple times better performance than traditional bots and is "learning" and improving with every new trade that came to database.
This is not data than average trader will look in and perform his trades better. Its huge database for smart programists and their algorithm

Keep pushing it and probably those who are interested will have a rethink of it and see the need for it. Is there no way you can provide a free test for this that any trader can try and see if it will work or not, because it is only your work that can speak for you because getting people to commit fund now will be quite hard as everyone is virtually a suspect now, kudos to those idiots called ICO scammers that is now making everyone to be scared and also see a good project as a bad one.

Anyway, I wish you luck but that $1500 is not feasible, you can make the first test very cheap, like $200, from the test, many of them will come to testify, then you can increase the value for your work.

Thank you.

Please see my answer above.

glaz330

newbie

Activity: 21

Merit: 0

Quote from: upsidedown75 on March 29, 2019, 02:31:14 PM

Quote from: glaz330 on March 28, 2019, 10:16:49 AM

Quote from: Mpamaegbu on March 28, 2019, 08:21:54 AM

OP, honestly if anyone were interested initially and then seeing that you're just a Newbie would make them lose interest in your project. Who even pays for a bot services that high in price. BTW, there's nothing special about what you intend to supply as it can be found online already.

How exactly did you conclude that I am a Newbie? (on the forum yes)

It can be really funny how people can just judge the expertise of a forum user by their forum tag, I do that a lot too and until one day, it dawn on me that I was already an expert in trading before getting to know about bitcointalk and opening an account with on it.

So for the fact that you are a newbie here doesn’t mean you are not an expert in your field but the service you intend rendering is quite on a high side for now and if you can proof to us that it will help the market out of the situation it is now, maybe I can try it out too and probably broadcast it to more crypto users too who might find it useful.

Thank you for your constructive criticism....

I am testing the backend right now at it is working as expected. I am capturing data as we speak.

My database is already one week old with all BTC pairs.

I 'll provide a link, when I finish the validation and write a README with the specs of the database.
Plus, I'll keep it free on github probably, for the time being.

GregH37

legendary

Activity: 1302

Merit: 1037

Quote from: Tytanowy Janusz on March 29, 2019, 04:56:53 AM

That's a lot of data that can feed AI bot. You can test your AI bot on those data. 1 clever algorithm and you have AI bot that has multiple times better performance than traditional bots and is "learning" and improving with every new trade that came to database.
This is not data than average trader will look in and perform his trades better. Its huge database for smart programists and their algorithm

Keep pushing it and probably those who are interested will have a rethink of it and see the need for it. Is there no way you can provide a free test for this that any trader can try and see if it will work or not, because it is only your work that can speak for you because getting people to commit fund now will be quite hard as everyone is virtually a suspect now, kudos to those idiots called ICO scammers that is now making everyone to be scared and also see a good project as a bad one.

Anyway, I wish you luck but that $1500 is not feasible, you can make the first test very cheap, like $200, from the test, many of them will come to testify, then you can increase the value for your work.

upsidedown75

legendary

Activity: 1288

Merit: 1036

Quote from: glaz330 on March 28, 2019, 10:16:49 AM

Quote from: Mpamaegbu on March 28, 2019, 08:21:54 AM

OP, honestly if anyone were interested initially and then seeing that you're just a Newbie would make them lose interest in your project. Who even pays for a bot services that high in price. BTW, there's nothing special about what you intend to supply as it can be found online already.

How exactly did you conclude that I am a Newbie? (on the forum yes)

It can be really funny how people can just judge the expertise of a forum user by their forum tag, I do that a lot too and until one day, it dawn on me that I was already an expert in trading before getting to know about bitcointalk and opening an account with on it.

So for the fact that you are a newbie here doesn’t mean you are not an expert in your field but the service you intend rendering is quite on a high side for now and if you can proof to us that it will help the market out of the situation it is now, maybe I can try it out too and probably broadcast it to more crypto users too who might find it useful.

Tytanowy Janusz

legendary

Activity: 2156

Merit: 1622

Quote from: StarofBTC on March 29, 2019, 03:33:45 AM

Raw data from Binance? is it not the same way you got it that others too can get it, I doubt if many people will make this as priority because I personally don’t see how their raw data can be of help to my trade, if you need to develop anything that will really be of help to many people, I will advise you develop what will help people make profit in their trading.

Moreover, the amount you are talking about is quite too expensive, do you know how much I will get if I just invest that in bitcoin and leave it for a very long term. $20 dollars would have been a fair price for many potential people that will patronize it.

You do not understand the value of this data. That's a lot of data that can feed AI bot. You can test your AI bot on those data. 1 clever algorithm and you have AI bot that has multiple times better performance than traditional bots and is "learning" and improving with every new trade that came to database.
This is not data than average trader will look in and perform his trades better. Its huge database for smart programists and their algorithm

Topic: Binance Raw trading data (Read 575 times)