Author

Topic: Pool API Standard (Read 4468 times)

legendary
Activity: 2576
Merit: 1186
June 18, 2012, 08:34:04 AM
#45
Elapsed should 'almost' never be used unless there is an actual specific reason for it
How long has a block been around is a good example of when it shouldn't be used - that should be a timestamp
It is a timestamp...

Also - I like float MH/s not H/s since that is what everyone thinks in nowadays and yeah in the future it will be even bigger
1 TH/s is 1000000000000 H/s - that number is just way too big already ...
1000000.000000 MH/s is 'nicer' IMO Smiley
1e12 is valid JSON.
legendary
Activity: 4592
Merit: 1851
Linux since 1997 RedHat 4
June 18, 2012, 07:56:54 AM
#44
A suggestion (or 2) ... welcome to be ignored Smiley


Elapsed should 'almost' never be used unless there is an actual specific reason for it
How long has a block been around is a good example of when it shouldn't be used - that should be a timestamp
(and saying it makes it easier for recipient XYZ simply means you got it wrong)
Timestamp things and return those integer sec timestamps or float timeval if higher accuracy is required
Also, as I did in the cgminer API, include a 'now' with any data set that has timestamps
(the cgminer API STATUS header always has 'When' for that reason ... once I realised it was missing)
This resolves any issue of ever creating correct elapsed values if anyone wants to display them


Also - I like float MH/s not H/s since that is what everyone thinks in nowadays and yeah in the future it will be even bigger
1 TH/s is 1000000000000 H/s - that number is just way too big already ...
1000000.000000 MH/s is 'nicer' IMO Smiley
legendary
Activity: 2576
Merit: 1186
June 06, 2012, 10:22:29 PM
#43
I (we!) missed a very important metric up there, invalid/stale shares. So I've tweaked it a bit.
Getting rather complicated now though! Didn't intend for it to go this way, but at least it's pretty complete!

https://bitcointalksearch.org/topic/m.921816
Did you update the wiki page with the specification?

I should probably do that...
Please try to edit the existing specification draft, rather than replace it entirely. I combined the two, but please double-check that I didn't miss anything. Note that JSON does not have Float or Time types, just Number.

I also made a quick pass over it to try to make it sensible for Bitcoin address mining.
legendary
Activity: 1795
Merit: 1208
This is not OK.
June 06, 2012, 09:44:32 PM
#42
Hoppable prop pools will not be wanting to give out that much information... of course that's their problem.
That's why everything is optional.

Word.
legendary
Activity: 2576
Merit: 1186
June 06, 2012, 09:37:57 PM
#41
Hoppable prop pools will not be wanting to give out that much information... of course that's their problem.
That's why everything is optional.
-ck
legendary
Activity: 4088
Merit: 1631
Ruu \o/
June 06, 2012, 09:18:12 PM
#40
Hoppable prop pools will not be wanting to give out that much information... of course that's their problem.
legendary
Activity: 1795
Merit: 1208
This is not OK.
June 06, 2012, 05:08:22 PM
#39
Reminds me of



Certainly does Cheesy

but no standard exists for this yet... all have been proprietary.
hero member
Activity: 686
Merit: 500
June 06, 2012, 04:35:59 PM
#38
Reminds me of

legendary
Activity: 1795
Merit: 1208
This is not OK.
June 06, 2012, 03:47:08 PM
#37
I (we!) missed a very important metric up there, invalid/stale shares. So I've tweaked it a bit.
Getting rather complicated now though! Didn't intend for it to go this way, but at least it's pretty complete!

https://bitcointalksearch.org/topic/m.921816
Did you update the wiki page with the specification?

I should probably do that...
legendary
Activity: 2576
Merit: 1186
June 06, 2012, 03:26:26 PM
#36
I (we!) missed a very important metric up there, invalid/stale shares. So I've tweaked it a bit.
Getting rather complicated now though! Didn't intend for it to go this way, but at least it's pretty complete!

https://bitcointalksearch.org/topic/m.921816
Did you update the wiki page with the specification?
legendary
Activity: 1795
Merit: 1208
This is not OK.
June 06, 2012, 03:22:25 PM
#35
I (we!) missed a very important metric up there, invalid/stale shares. So I've tweaked it a bit.
Getting rather complicated now though! Didn't intend for it to go this way, but at least it's pretty complete!

https://bitcointalksearch.org/topic/m.921816
full member
Activity: 133
Merit: 100
June 01, 2012, 05:26:05 PM
#34
block =
{
    "currency" =
    "id" = ,
    "duration" = ,
    "shares_total" = ,
    "shares_submitted" =
}

balance =
{
    "currency" = ,
    "confirmed" = =120 valid blocks) in Satoshis>,
    "unconfirmed" = ,
    "estimate" = ,
    "last_pay" = ,
    "last_pay_time" =

+1

I would love to see this implemented.
legendary
Activity: 1795
Merit: 1208
This is not OK.
May 25, 2012, 11:41:20 AM
#33
shares =
{
    "submitted" = ,
    "stale" = ,
    "invalid" =
}

block =
{
    "currency" =
    "id" = ,
    "duration" = ,
    "shares_round" = {<'shares' for entire pool for the round (as defined above)>},
    "shares_user" = {<'shares' for all workers of the user for the round (as defined above)>}
}

balance =
{
    "currency" = ,
    "confirmed" = =120 valid blocks) in Satoshis>,
    "unconfirmed" = ,
    "estimate" = ,
    "last_pay" = ,
    "last_pay_time" =
legendary
Activity: 1795
Merit: 1208
This is not OK.
May 24, 2012, 11:21:51 AM
#32
CSV for block data sounds reasonable to me.
donator
Activity: 2058
Merit: 1007
Poor impulse control.
May 24, 2012, 10:27:43 AM
#31
I would agree, historical data is better provided as CSV.  The API should be for relevant current information that changes frequently.

Json has overhead compared to csv.  However, csv is not extensible, and the information isn't that big, so the overhead doesn't matter that much.  I would prefer json over csv.  A simple (online) convertor can still be made for those that prefer to work with csv instead of json.  If it is a standard, such a convertor would be easy to write for all pools at once...

csv is only considered for historical data that would be easier to manipulate as a dataframe and needs updating only when a block is found.
sr. member
Activity: 263
Merit: 250
Pool operator of Triplemining.com
May 24, 2012, 08:58:08 AM
#30
I would agree, historical data is better provided as CSV.  The API should be for relevant current information that changes frequently.

Json has overhead compared to csv.  However, csv is not extensible, and the information isn't that big, so the overhead doesn't matter that much.  I would prefer json over csv.  A simple (online) convertor can still be made for those that prefer to work with csv instead of json.  If it is a standard, such a convertor would be easy to write for all pools at once...
legendary
Activity: 1260
Merit: 1000
May 24, 2012, 07:40:42 AM
#29
I would agree, historical data is better provided as CSV.  The API should be for relevant current information that changes frequently.
legendary
Activity: 1386
Merit: 1097
May 24, 2012, 01:13:43 AM
#28
As Inaba said the number and order of fields in JSON is immaterial

To be exact, this is true only for keys of an array. Lists (and history of blocks should be definitely stored in list) are ordered. But I see your point. Actually I'm for defining JSON API as an standard, because it's usually easier to handle in applications, and provide alternate CSV API with block history only (from pool side it's just an iteration over original JSON and columns with expected order).
donator
Activity: 2058
Merit: 1007
Poor impulse control.
May 23, 2012, 08:12:32 PM
#27
I think pool round history is probably better as a .csv than in JSON. It's much easier to use a dataframe when working with aggregates of data. Also much easier for the poor sods using excel to get a feel for their pool's recent history.

As Inaba said the number and order of fields in JSON is immaterial, but that's not the case in a dataframe, especially if you want to perform exactly the same operations on different dataframes from different pools.

sr. member
Activity: 263
Merit: 250
Pool operator of Triplemining.com
May 23, 2012, 05:18:16 PM
#26
I'd suggest providing 2 API's/url's

One API that requires an API key, showing all the information related to that user.

A second api/url containing the pool data, global data that is not related to a specific user, so it can be retrieved by any user without an API key.  Examples are global hashrate, found blocks history and such.

Seperating them would make it easy for people to get full data on any pool without an account...

Perhaps a third url can be made like /standard_api_locations.txt where the location of both those url's are placed, so the user doesn't have to enter anything except the main pool url.
legendary
Activity: 1795
Merit: 1208
This is not OK.
May 23, 2012, 12:43:10 PM
#25
Yes and I see it overcomplicated, especially that "collab" construction.
Yes it is rather. I didn't intend such complexity.

Better to have more APIs than one huge monster, isn't it? I even think that current API should be split into more calls like account state (balances, address, etc) and worker states.

That was my original suggestion Smiley

but having structure:
balance =
{
  type = BTC
  confirmed = xxx
  unconfimed = xxx
  ...etc...
}

As an array inside the main structure also works for me.

user =
{
    ...
    balances = {array of balance}
    ...
}
legendary
Activity: 1386
Merit: 1097
May 23, 2012, 12:10:57 PM
#24
Have you guys seen Luke's BIP?

Yes and I see it overcomplicated, especially that "collab" construction. Better to have more APIs than one huge monster, isn't it? I even think that current API should be split into more calls like account state (balances, address, etc) and worker states.
legendary
Activity: 1386
Merit: 1097
May 23, 2012, 12:08:13 PM
#23
I did that specifically so you could add nmc_balance, nmc_unconfirmed, and nmc_paid.  Although I suppose we could make them a sub array of BTC: or NMC: (or whatever other currency) and then just use balance, unconfirmed, paid.

Having currency name in variable doesn't look smart. Then I prefer - separate calls completely (and have an api for every currency) or have separate section for every currency, like ...'balance': {'BTC': {'unconfirmed': 0, 'confirmed': 0, 'paid': 0}, 'NMC': {...}} ...
legendary
Activity: 1795
Merit: 1208
This is not OK.
May 23, 2012, 11:20:56 AM
#22
Have you guys seen Luke's BIP?

https://en.bitcoin.it/wiki/BIP_PoolAPI

Some interesting ideas on displaying balances. Very generic, so can be used for any currency.

Things like the API call and call rate I should think can be totally up to the pool. I don't think there's any need to standardise these.
legendary
Activity: 1260
Merit: 1000
May 23, 2012, 07:50:56 AM
#21
Quote
The number and order of fields in each api should be standardised.

JSON makes both of these immaterial.  No need to have them ordered and it would impose an undue burden on the pool side.  The number of fields would also be immaterial, just discard the fields you don't want to use.  That way it allows a pool to provide additional information beyond the standard.

Quote
The api call should be standardised -
eg the following could call worker stats:

This is nice in theory, but I'm not sure it can work in practice, given all the different systems out there. In your example it would create a supremely difficult problem for me to provide since I'm not setup to handle calls in that fashion.  If we are going to standardize in this fashion, HTTP GET is probably the best solution, since it should be universally compatible.

Quote
What if user (or few of his workers) are PPS? null value?

Zero would be the logical amount, since it would be an accurate value.

Quote
UTC, not GMT! GMT has summer/winter time, which is always pain in applications.

Yes!  Sorry I meant UTC not GMT!

Quote
Let's rename it to balance, balance_unconfirmed, balance_paid. Otherwise we'll rename those fields for NMC?

I did that specifically so you could add nmc_balance, nmc_unconfirmed, and nmc_paid.  Although I suppose we could make them a sub array of BTC: or NMC: (or whatever other currency) and then just use balance, unconfirmed, paid.

Quote
Also, we should define some standard on ask rate from clients. I'm caching API calls for 30 seconds, but still I see some stupid users fetching JSON data every second. That's not only annoying, that also needs much more transfers than mining itself of those users (especially when they're CPU miners).

Yes, I absolutely agree.  I see the same things.

legendary
Activity: 1386
Merit: 1097
May 23, 2012, 04:54:21 AM
#20
I'm not against this initiative if it will need just to rename columns and URLs in my own API (which was - as Tycho mentioned - the first API, and there's no general issue with that). That means - I don't see any reason to change token management etc.

Also, we should define some standard on ask rate from clients. I'm caching API calls for 30 seconds, but still I see some stupid users fetching JSON data every second. That's not only annoying, that also needs much more transfers than mining itself of those users (especially when they're CPU miners).
legendary
Activity: 1386
Merit: 1097
May 23, 2012, 04:48:44 AM
#19
estimated_reward
Estimated reward the user will receive on next payout in BTC
What if user (or few of his workers) are PPS? null value?

Quote
last_activity
Last miner activity in GMT

UTC, not GMT! GMT has summer/winter time, which is always pain in applications.

Quote
btc_balanceCurrent available balance for the user in BTC
btc_unconfirmedUnconfirmed/unavailable balance for the user in BTC
btc_paidTotal amount paid to the user in BTC

Let's rename it to balance, balance_unconfirmed, balance_paid. Otherwise we'll rename those fields for NMC? :-)

donator
Activity: 2058
Merit: 1007
Poor impulse control.
May 23, 2012, 02:34:55 AM
#18
One more thing - a pool history request shouldn't require a key (Ahem maxbtc, I'm looking at you Smiley)
donator
Activity: 2058
Merit: 1007
Poor impulse control.
May 23, 2012, 01:14:07 AM
#17
I think UNIX timestamp is the way to go.  If a pool wants to add extra "human" time, then that's easily done.  I think the standard as UNIX time is the right choice.  Really, all time should be given in unix time stamps and extra fields added for human time if desired.

So my proposal for the standard:

All time displays presented as UNIX epoc.  This will also do away with the need for timezoning questions/issues.
All hash rates presented as hashes per second.
All BTC's presented as Satoshis


Good proposal Inaba, and thanks to Inaba and P_Shep for getting this going.

Can I add:
  • All fields to be integer (no char, no added commas etc, no double)
  • Apis should all be the same format (preferably JSON).
  • The number and order of fields in each api should be standardised.
  • The api call should be standardised -
    eg the following could call worker stats:
Code:
poolname.com/api/workername

  • Any api call that includes historical data should include an option for selection of amount of data. Eg if X is the last X rounds of historical data:[\li]
    Code:
    poolname.com/api/username/history?X

legendary
Activity: 1795
Merit: 1208
This is not OK.
May 22, 2012, 10:29:11 PM
#16
I think UNIX timestamp is the way to go.  If a pool wants to add extra "human" time, then that's easily done.  I think the standard as UNIX time is the right choice.  Really, all time should be given in unix time stamps and extra fields added for human time if desired.

So my proposal for the standard:

All time displays presented as UNIX epoc.  This will also do away with the need for timezoning questions/issues.
All hash rates presented as hashes per second.
All BTC's presented as Satoshis

Sounds good to me
legendary
Activity: 1260
Merit: 1000
May 22, 2012, 09:19:24 PM
#15
I think UNIX timestamp is the way to go.  If a pool wants to add extra "human" time, then that's easily done.  I think the standard as UNIX time is the right choice.  Really, all time should be given in unix time stamps and extra fields added for human time if desired.

So my proposal for the standard:

All time displays presented as UNIX epoc.  This will also do away with the need for timezoning questions/issues.
All hash rates presented as hashes per second.
All BTC's presented as Satoshis


donator
Activity: 2058
Merit: 1007
Poor impulse control.
May 22, 2012, 08:14:43 PM
#14
and the timestamp should be in the standard unix  %Y-%m-%d %H:%M:%S.
In my opinon formatting has no place in an API that is a presentation layer task.  Unix/Posix time is better since it is an integer and every language has tools to format it.

For example as of writing Unix/Posix time is 1337632227
I would definitely prefer it to be unixtime for my own purposes, although I get a seamless conversion when I need it. However I proposed the above format because I thought that the average miner might not be able to use unixtime so easily.
Quote
Still the larger point is that data format should be part of the spec.
Yes, that's the main thing. Anyone wanting to - for example - create a website to list miner api results or pool history will need all data standardised if she or he's going to list all pools equally easily. Btc-poolwatch.com was a constant job for koo (the owner/writer) since adding an api meant writing a new code to interpret the api, and any api changes broke the service.
donator
Activity: 1218
Merit: 1079
Gerald Davis
May 22, 2012, 07:58:06 PM
#13
Data should be standardised too.

Agree

Quote
'Round duration' should be in seconds

Agreed (as in integer unformated # of seconds)

Quote
and the timestamp should be in the standard unix  %Y-%m-%d %H:%M:%S.
In my opinon formatting has no place in an API that is a presentation layer task.  Unix/Posix time is better since it is an integer and every language has tools to format it.

For example as of writing Unix/Posix time is 1337632227

Still the larger point is that data format should be part of the spec.

Quote
Current difficulty is important to include.
Agreed.
vip
Activity: 980
Merit: 1001
May 22, 2012, 06:54:35 PM
#12
A standardised api for pool stat history would be nice.
standardisation is slowly creeping into bitcoin - which is good

Ozcoin shows per worker hashrate as well as total, this was on user request - I think it needs to be included for people that use API to monitor rigs up/down. e.g. our desktop widget will not work without separated worker hashrates.
At Ozcoin payout method is by user not worker (this might change in the future)
Currently we are doing some heavy development on the backend and site - whilst a great idea a standardised API is not high on our priorities.


donator
Activity: 2058
Merit: 1007
Poor impulse control.
May 22, 2012, 05:40:58 PM
#11
For a pool history API, for pools like deepbit where pool history will be thousands of blocks there might need to be a protocol limiting the number of blocks to 100 recent blocks, and then have a slightly different url for blocks further back.

I'd like to see a JSON or other API present a pool history table like this:

Code:
Block height/hash | Block timestamp | Round duration | Total submitted shares | Current difficulty | 
Data should be standardised too. 'Round duration' should be in seconds and the timestamp should be in the standard unix  %Y-%m-%d %H:%M:%S. Current difficulty is important to include. It's easily available elsewhere, but this being a block history API it's a pain for a miner to search the difficulty history for each and every block.

The field for can be used for any results specific to a payout method. I had SMPPS in mind (a field for buffer), but I'm sure other payout methods would find a use for it.
legendary
Activity: 1795
Merit: 1208
This is not OK.
May 22, 2012, 05:00:19 PM
#10

"rewardalgo" may have to go per worker. Some pools you can select what worker does what... EMC & Ozcoin allows this. Perhaps this field can be used to list available algorithms.
I quite liked my idea for a round/block structure. Allows for block history to use the same structure repeated. And everyone likes re-using code Smiley
donator
Activity: 2058
Merit: 1007
Poor impulse control.
May 22, 2012, 04:36:56 PM
#9
A standardised api for pool stat history would be nice.
legendary
Activity: 1795
Merit: 1208
This is not OK.
May 22, 2012, 04:31:57 PM
#8
p_shep approves this message Smiley
legendary
Activity: 2576
Merit: 1186
legendary
Activity: 1428
Merit: 1000
May 22, 2012, 03:32:50 PM
#6
I assume he mean reward method (PPS, SMPSS, DGM, etc)?

yes, thanks
would be easy to develop an app to monitor multiple pools then
donator
Activity: 1218
Merit: 1079
Gerald Davis
May 22, 2012, 03:31:17 PM
#5
I assume he mean reward method (PPS, SMPSS, DGM, etc)?

Including things like fee %, payout tx fees, merge mining would make it possible to keep something like this actually up to date. Smiley
https://en.bitcoin.it/wiki/Comparison_of_mining_pools
hero member
Activity: 504
Merit: 502
May 22, 2012, 03:31:03 PM
#4
I really like these suggestions, I will soon be adding proper json and may aswell make use of the suggested options once "finalised" in this thread I assume.
legendary
Activity: 1260
Merit: 1000
May 22, 2012, 03:26:41 PM
#3
Uris? 

What do you mean by payout method exactly?
legendary
Activity: 1428
Merit: 1000
May 22, 2012, 03:21:49 PM
#2
what do you think about adding a field about the payout-method used?
and i would love to see standardized uris too.
legendary
Activity: 1260
Merit: 1000
May 22, 2012, 03:10:41 PM
#1
SMF won't format tables properly (surprise!) ... I will reformat it if it's too confusing.

Here is a first draft proposal for a JSON API standards for Bitcoin Pools:

Generic keys:
JSON KeyDescription
versionVersion of the API in use

Pool keys:
JSON KeyDescription
round_durationDuration of the current round
round_sharesNumber of shares submitted in current round
hashrateTotal hashrate of the pool


User keys:
JSON KeyDescription
round_shares
Total shares submitted by user for current round
estimated_reward
Estimated reward the user will receive on next payout in BTC
hashrateTotal hashrate of the user
last_activity
Last miner activity in GMT
btc_balanceCurrent available balance for the user in BTC
btc_unconfirmedUnconfirmed/unavailable balance for the user in BTC
btc_paidTotal amount paid to the user in BTC
last_payout_amountLast amount paid to the user in BTC
last_payout_timeTime user was paid in GTM
Jump to: