[ANN] Stratum mining protocol - ASIC ready - page 13.

slush

legendary

Activity: 1386

Merit: 1097

Quote from: fireduck on February 14, 2013, 10:21:52 AM

That makes sense to me. I know the ship has long since sailed, but I kinda with these weren't json arrays but instead where key value pairs so that it would be a little more self describing and easier to add various extensions. Maybe any extensions should be done as pairs rather than tacking values on to keep things clear?

Then next logical step is to get rid of json entirely and propose protobuf messages. I picked json for good reasons and I still think it was smart choice, but for future development protobuf would be better. If anyone thinks about protocol change, then please take protobuf in account.

slush

legendary

Activity: 1386

Merit: 1097

Quote from: fireduck on February 14, 2013, 10:17:24 AM

So, I've just made SockThing use the same extranonce1 for all connections. I am also adding an extra random number to the coinbase and generate a new coinbase for each job.

Why is this better than generating custom extranonce1? Extranonce1 has been designed exactly for this. If you can achieve the same block template on all backends, then you can make session_id==extranonce1 and my proposal will work for you.

fireduck

sr. member

Activity: 392

Merit: 251

Quote from: slush on February 14, 2013, 09:03:12 AM

Quote from: -ck on February 14, 2013, 08:29:39 AM

If just one pool op decides to implement this, I will support it in cgminer.

No problem. I just hope that the first daredevil will implement it properly, so response of mining.subscribe will have one extra argument 'session-id':

Quote

{"id": 1, "result": [[], "08000002", 4, "some-session-id"], "error": null}\n

If this parameter is set and non-null, the miner will store sesssion id internally and then, after the crash of the connection, miner will call mining.resume('session-id') instead of mining.subscribe(...). Pool will respond with true/false. If true, miner can continue in the previous session (so re-upload pending shares, ...), if false, miner must call mining.subscribe and throw away pending shares.

Correct?

That makes sense to me. I know the ship has long since sailed, but I kinda with these weren't json arrays but instead where key value pairs so that it would be a little more self describing and easier to add various extensions. Maybe any extensions should be done as pairs rather than tacking values on to keep things clear?

fireduck

sr. member

Activity: 392

Merit: 251

Quote from: slush on February 14, 2013, 03:01:26 AM

Quote from: fireduck on February 13, 2013, 07:23:59 PM

1) Should a work unit (from a mining.notify message) be usable by a client if the TCP connection breaks and they reconnect? It makes sense to me that it should, but I am having trouble because the mining.subscribe seems to occur before the mining.authorize so I need to issue (or reissue) the same extranonce1 before I know which user it is.

Please propose some protocol extension before you implement anything in this area. I was thinking about some session identifiers and resuming the connection, but I didn't find any usable (simple and scalable) solution yet. From my perspective this is actually much smaller problem that it seems to be; properly implemented server and client and standard internet connection won't drop the connection for days. And one second of lost work per day is definitely NOT worth all the effort.

So, I've just made SockThing use the same extranonce1 for all connections. I am also adding an extra random number to the coinbase and generate a new coinbase for each job.

Now reconnects should work fine, except clients have no way of knowing that they will.

I'm thinking maybe adding a field to mining.subscribe that indicates this feature exists.

However, I don't imagine other pool operators are inclined to build a new coinbase for each user so I don't imagine my solution will be for everyone.

slush

legendary

Activity: 1386

Merit: 1097

Quote from: -ck on February 14, 2013, 08:29:39 AM

If just one pool op decides to implement this, I will support it in cgminer.

No problem. I just hope that the first daredevil will implement it properly, so response of mining.subscribe will have one extra argument 'session-id':

Quote

{"id": 1, "result": [[], "08000002", 4, "some-session-id"], "error": null}\n

If this parameter is set and non-null, the miner will store sesssion id internally and then, after the crash of the connection, miner will call mining.resume('session-id') instead of mining.subscribe(...). Pool will respond with true/false. If true, miner can continue in the previous session (so re-upload pending shares, ...), if false, miner must call mining.subscribe and throw away pending shares.

Correct?

-ck

legendary

Activity: 4088

Merit: 1631

Ruu \o/

Quote from: slush on February 14, 2013, 08:16:27 AM

If something, then go with mining.resume() and let the pool decide if he'll accept it or not. However I don't think anybody sane will want to implement full mechanism of session resuming.

You have certainly expressed your disinterest in implementing it, but that is not the case for all other pool ops. If just one pool op decides to implement this, I will support it in cgminer, and as per always, once one pool has set the standard, it would be crazy for others to not follow. So who's first?

slush

legendary

Activity: 1386

Merit: 1097

Quote from: Luke-Jr on February 14, 2013, 07:58:16 AM

I'd suggest just adding an optional field to the end of mining.submit for extranonce1 that the miner can use for old share submissions on reconnect.
Let the pool decide whether to honour them or not, and for how long.

But pool cannot validate the share unless the backend has the same block template (with the same job ID). Yes, we can start adding "optional fields", but that's not the direction of development which I want to go.

If something, then go with mining.resume() and let the pool decide if he'll accept it or not. However I don't think anybody sane will want to implement full mechanism of session resuming.

Luke-Jr

legendary

Activity: 2576

Merit: 1186

I'd suggest just adding an optional field to the end of mining.submit for extranonce1 that the miner can use for old share submissions on reconnect.
Let the pool decide whether to honour them or not, and for how long.

slush

legendary

Activity: 1386

Merit: 1097

Quote from: kano on February 14, 2013, 04:30:58 AM

Quote from: slush on February 14, 2013, 03:01:26 AM

properly implemented server and client and standard internet connection won't drop the connection for days.

Except ... for anyone with an unreliable internet connection (happens with wireless and 3g and such) they lose more shares the more often they lose the connection.

Important part highlighted.

Quote

Of course if you have a dynamic IP address, you also lose shares during that change

Why? Providers are dropping TCP connection because of moving the customer to new IP address?

Quote

... pity it's a new problem added by Stratum (that didn't already exist)

A lot of other and more important issues existed before, which are solved by Stratum.

Quote

Yes a proper implementation would have to handle a pool restart also - but that is the pools problem not the protocol's problem.

Ok, adding session identifier to the response of mining.subscribe() is trivial, this identifier can be used by mining.resume() on the beginning of another TCP connection. But you're literally opening Pandora box:

a) Backends must have some backbone channel for retrieving session data of given session identifier
or
b) Backends must use exactly the same template, so session identifier can be the same as extranonce1, but backends must have backbone for validating duplicated submissions.

Both solutions have drastical impact to scalability and both solutions are extremely hard to implement. It also open another doors for attacks (DoS, share double spends). That additional infrastructure is definitely not worth of <0.001% lost shares of users with non-standard mining setup.

I agree with you kano, that it's not perfect. But adding another magnitude of complexity won't help much.

kano

legendary

Activity: 4634

Merit: 1851

Linux since 1997 RedHat 4

Quote from: slush on February 14, 2013, 03:01:26 AM

Quote from: fireduck on February 13, 2013, 07:23:59 PM

1) Should a work unit (from a mining.notify message) be usable by a client if the TCP connection breaks and they reconnect? It makes sense to me that it should, but I am having trouble because the mining.subscribe seems to occur before the mining.authorize so I need to issue (or reissue) the same extranonce1 before I know which user it is.

Please propose some protocol extension before you implement anything in this area. I was thinking about some session identifiers and resuming the connection, but I didn't find any usable (simple and scalable) solution yet. From my perspective this is actually much smaller problem that it seems to be; properly implemented server and client and standard internet connection won't drop the connection for days. And one second of lost work per day is definitely NOT worth all the effort.

Except ... for anyone with an unreliable internet connection (happens with wireless and 3g and such) they lose more shares the more often they lose the connection.
Of course if you have a dynamic IP address, you also lose shares during that change, however often that happens - but obviously that's not often.
... pity it's a new problem added by Stratum (that didn't already exist)

Yes in general it is a small number of shares, but it also means that if the pool itself does a restart, everyone loses the work they are working on and can never submit it.
Yes a proper implementation would have to handle a pool restart also - but that is the pools problem not the protocol's problem.

However, the problem is simply that it is hard for the pools to implement, so the pools have shied away from doing it or are unable to do it due to their own design.

slush

legendary

Activity: 1386

Merit: 1097

Quote from: fireduck on February 13, 2013, 07:23:59 PM

1) Should a work unit (from a mining.notify message) be usable by a client if the TCP connection breaks and they reconnect? It makes sense to me that it should, but I am having trouble because the mining.subscribe seems to occur before the mining.authorize so I need to issue (or reissue) the same extranonce1 before I know which user it is.

Please propose some protocol extension before you implement anything in this area. I was thinking about some session identifiers and resuming the connection, but I didn't find any usable (simple and scalable) solution yet. From my perspective this is actually much smaller problem that it seems to be; properly implemented server and client and standard internet connection won't drop the connection for days. And one second of lost work per day is definitely NOT worth all the effort.

eleuthria

legendary

Activity: 1750

Merit: 1007

Quote from: fireduck on February 14, 2013, 01:18:15 AM

I've put my code on github in case anyone is interested:
https://github.com/fireduck64/SockThing

It seems to work but is certainly not done. I need to add metrics, probably some job management and clean up the multi-connection same user story a bit.

Ahh, the Java, it burns! Glad to see another fresh implementation of Stratum. I'll try to take a deeper look at it tomorrow, but so far I must say it's one of the cleaner java-based pool servers I've seen in quite some time.

fireduck

sr. member

Activity: 392

Merit: 251

Quote from: fireduck on February 13, 2013, 07:23:59 PM

I have made a java stratum pool server implementation. It seems to be working pretty well. Once I pull my passwords and settings out of the source into a config file I'll put it up on github.

Questions:

1) Should a work unit (from a mining.notify message) be usable by a client if the TCP connection breaks and they reconnect? It makes sense to me that it should, but I am having trouble because the mining.subscribe seems to occur before the mining.authorize so I need to issue (or reissue) the same extranonce1 before I know which user it is.

2) I should be able to support a bunch of authenticated workers on a single TCP connection, right? This means they are all going to work with the same mining.notify messages and with the same extranonce1 values?

If I do this, it makes me dedup code a little more complex. If I don't do it, it probably disables some interesting proxy use cases.

I could also see an issue with select-able difficulty pools (like mine) where a user creates a stratum connection with a bunch of workers at different difficulty levels and then when they find a share they submit it as whichever worker has the highest difficulty that it matches, thus greatly increasing their pay at my expense.

I've put my code on github in case anyone is interested:
https://github.com/fireduck64/SockThing

It seems to work but is certainly not done. I need to add metrics, probably some job management and clean up the multi-connection same user story a bit.

eleuthria

legendary

Activity: 1750

Merit: 1007

Quote from: fireduck on February 13, 2013, 07:23:59 PM

I have made a java stratum pool server implementation. It seems to be working pretty well. Once I pull my passwords and settings out of the source into a config file I'll put it up on github.

Questions:

1) Should a work unit (from a mining.notify message) be usable by a client if the TCP connection breaks and they reconnect? It makes sense to me that it should, but I am having trouble because the mining.subscribe seems to occur before the mining.authorize so I need to issue (or reissue) the same extranonce1 before I know which user it is.

2) I should be able to support a bunch of authenticated workers on a single TCP connection, right? This means they are all going to work with the same mining.notify messages and with the same extranonce1 values?

If I do this, it makes me dedup code a little more complex. If I don't do it, it probably disables some interesting proxy use cases.

I could also see an issue with select-able difficulty pools (like mine) where a user creates a stratum connection with a bunch of workers at different difficulty levels and then when they find a share they submit it as whichever worker has the highest difficulty that it matches, thus greatly increasing their pay at my expense.

For #2) Yes, work with Stratum is defined per connection, so is difficulty. If people pool multiple workers over one connection, they have to work at whatever the highest difficulty worker is set to.

kano

legendary

Activity: 4634

Merit: 1851

Linux since 1997 RedHat 4

Quote from: fireduck on February 13, 2013, 07:23:59 PM

...
Questions:

1) Should a work unit (from a mining.notify message) be usable by a client if the TCP connection breaks and they reconnect? It makes sense to me that it should, but I am having trouble because the mining.subscribe seems to occur before the mining.authorize so I need to issue (or reissue) the same extranonce1 before I know which user it is.
...

As far as I know, none of the pools support this yet Sad

cgminer currently assumes that a disconnect means throw away the work, since all pools do that.

Be the first! Make the others do it too!

fireduck

sr. member

Activity: 392

Merit: 251

I have made a java stratum pool server implementation. It seems to be working pretty well. Once I pull my passwords and settings out of the source into a config file I'll put it up on github.

Questions:

1) Should a work unit (from a mining.notify message) be usable by a client if the TCP connection breaks and they reconnect? It makes sense to me that it should, but I am having trouble because the mining.subscribe seems to occur before the mining.authorize so I need to issue (or reissue) the same extranonce1 before I know which user it is.

2) I should be able to support a bunch of authenticated workers on a single TCP connection, right? This means they are all going to work with the same mining.notify messages and with the same extranonce1 values?

If I do this, it makes me dedup code a little more complex. If I don't do it, it probably disables some interesting proxy use cases.

I could also see an issue with select-able difficulty pools (like mine) where a user creates a stratum connection with a bunch of workers at different difficulty levels and then when they find a share they submit it as whichever worker has the highest difficulty that it matches, thus greatly increasing their pay at my expense.

-ck

legendary

Activity: 4088

Merit: 1631

Ruu \o/

Quote from: DrHaribo on February 11, 2013, 04:55:14 PM

I believe I saw somewhere a way for a server to ask a client to identify itself (software name and version). Is this something the current crop of clients support?

client.get_version

Supported by cgminer.

DrHaribo

legendary

Activity: 2730

Merit: 1034

Needs more jiggawatts

I believe I saw somewhere a way for a server to ask a client to identify itself (software name and version). Is this something the current crop of clients support?

generalfault

newbie

Activity: 26

Merit: 0

Marty,

it looks like the getwork proxy cannot connect to the local stratum server. My guess is you have a firewall blocking the stratum port. If that is not the case, please post the top section (listening ports) of netstat -an

martychubbs

hero member

Activity: 742

Merit: 503

thanks, that fixed it.

BTW, I am using git://github.com/generalfault/stratum-mining-proxy.git

located ./stratum-mining/externals/stratum-mining-proxy

Code:

GW_ENABLE = True		# Enable the Proxy (If enabled you MUST run update_submodules)
GW_PORT = 8331			# Getwork Proxy Port
GW_DISABLE_MIDSTATE = False	# Disable midstate's (Faster but breaks some clients)
GW_SEND_REAL_TARGET = False	# Propigate >1 difficulty to Clients (breaks some clients)

Midstate is detected and I ran update_submodules.

Issue I am having is: getwork is not detected on port 8331, however, stratum is detected on port 3333. (so says latest version of GUIminer)

thoughts? BTW, great support Slush and General!

Update: http://pastebin.com/BwuBWVQP

Is the debug normal? There is a connection timeout..?

Code:

2013-02-07 21:42:07-0600 [-] Unhandled error in Deferred:
2013-02-07 21:42:07-0600 [-] Unhandled Error
        Traceback (most recent call last):
        Failure: stratum.custom_exceptions.TransportException: SocketTransportClientFactory connection timed out
 
2013-02-07 21:42:07,328 DEBUG socket_transport socket_transport.clientConnectionFailed # [Failure instance: Traceback (failure with no frames): : User timeout caused connection failure.
]

and

Code:

2013-02-07 21:51:37,329 ERROR proxy client_service.on_timeout # Connection to upstream pool timed out

Upstream pool is on localhost. That is the stratum mining server... ?

also

Webstats output on port 8889:

Code:

web.Server Traceback (most recent call last):
exceptions.TypeError: int() argument must be a string or a number, not 'NoneType'
/usr/local/lib/python2.7/dist-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/web/server.py:192 in process
191                    self._encoder = encoder
192            self.render(resrc)
193        except:
/usr/local/lib/python2.7/dist-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/web/server.py:241 in render
240        try:
241            body = resrc.render(self)
242        except UnsupportedMethod as e:
/usr/local/lib/python2.7/dist-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/web/resource.py:250 in render
249            raise UnsupportedMethod(allowedMethods)
250        return m(request)
251
/opt/stratum-mining/lib/basic_stats.py:99 in render_GET
98 r+="%s%s/%s%s/%s%s"%(
99 wc,wi,format(int(wd["speed"]),"n"),wd["difficulty"],format(int(wd["total_shares"]),"n"),
100 format(int(wd["total_rejects"]),"n"),format(int(wd["total_found"]),"n"))
exceptions.TypeError: int() argument must be a string or a number, not 'NoneType'

Thanks all!

Topic: [ANN] Stratum mining protocol - ASIC ready - page 13. (Read 146166 times)