Topic: Unified miners communication protocol - page 2. (Read 6255 times)
I would suggest that the best way to go about this would be to draft a document and let everyone pick it apart. Of course getting a concensus is almost impossible. When all is said and done, the important part would be getting the hardware people on board. Once they are in agreement, the software people may bitch, but they still need to support the hardware or someone else will.
As long as you're having the device request work, might as well have it specify how many it needs.
If NeedWork is implemented - yes, there is. If not implemented, but hashrate is - the predefined fixed value may be a good start and much more easier to implement as fixed number answer.If you notice that returned timestamps are increasing too fast, send work faster.
Example: You have 500 BF chips and you are given a single work (be it with MTime 300), but don't have nonce range division (HW limitation) and don't have NeedWork implemented. When you start you wait 1.2sec (with 200 chips idle), before sending nonces back. MS sees them coming fast and sends more, but still for 300 + a bit more until your full hashrate is reached. If GotHashrate is implemented and you return a fixed value 500x3GH - you will be given enough work from the first command.Having either NeedWork or GotHasrate implemented will speed-up the start, while GotHasrate is easier to implement and if fully implemented gives detailed information, which the MS may not need for it's normal operation, but can be provided to a customised (for the hardware) web interface for example.
How is the number of jobs it needs calculated then? I mean when we have just started and discovered X unknown devices from unknown manufacturer (otherwise we just won't ask)
If you notice that returned timestamps are increasing too fast, send work faster. 
IMO there is no real value for asking the device for its current hashrate.
How is the number of jobs it needs calculated then?I mean when we have just started and discovered X unknown devices from unknown manufacturer (otherwise we just won't ask)
Also if there are 500 chips and the hardware keeps an eye on them (to know when to pool them), so it has that detailed info - the MS may only keep the overall hashrate and ask.
It might be nice if the protocol were prepared to handle non-Bitcoin POW algorithms too, but this could probably be done as an extension.
What about ShaWork/ShaNonce in replacement for DoWork/GotNonce and then ScptWork/ScptNonce etc. as each may need it's own parameters?Of more important note, I think I forgot to mention: There should probably be a single command to cancel all pending jobs ASAP, for use on block changes.
IMO there is no real value for asking the device for its current hashrate.
How is the number of jobs it needs calculated then?I mean when we have just started and discovered X unknown devices from unknown manufacturer (otherwise we just won't ask)
Also if there are 500 chips and the hardware keeps an eye on them (to know when to pool them), so it has that detailed info - the MS may only keep the overall hashrate and ask.
It might be nice if the protocol were prepared to handle non-Bitcoin POW algorithms too, but this could probably be done as an extension.
What about ShaWork/ShaNonce in replacement for DoWork/GotNonce and then ScptWork/ScptNonce etc. as each may need it's own parameters?But you've already missed something here ... when defining a 'one ring to rule them all' protocol, one version should be able to support different configurations, so version is not complete.
Again e.g. BFL SC firmware can enable and disable different options, so not only version would be required, but also a configuration response (like the BFL SC GetInfo)
So ... I'd ask directly: what is the point of a defined interface?
Not missed, but will I succeed is another question ... Again e.g. BFL SC firmware can enable and disable different options, so not only version would be required, but also a configuration response (like the BFL SC GetInfo)
So ... I'd ask directly: what is the point of a defined interface?
"to support different configurations, so version is not complete" - trying to define it the way that even if "_list_of_supported_actions" are improperly returned there are means to correct the result based on actual responses and not expect any specific response.
"not only version would be required, but also a configuration response (like the BFL SC GetInfo)" - on my opinion all the info should be available as separate commands (and their answers), not as a bunch of text. This way, in case we change the frequency/voltage of chip X, but the driver has reset it back on it's own (as protection masure or other) - there is no need to get all info again, but ask for "GetFreq,EngineID" and get a single number back, without occupying the communication channel (and limited power MCU) for a long time.
"what is the point of a defined interface" - for an easy jump start by implementing the bare minimum and still allowing future extensions without reinventing the wheel
Please, stop both!
Lets not make this tread a flame war, but focus on the subject.
Lets not make this tread a flame war, but focus on the subject.
I've always worked directly with manufacturers, often writing the code myself, so this isn't a real problem for BFGMiner.
And here comes the second problem - someone should do the same for CGminer and then someone else for EasyMiner and then for BitMinter etc.Managing dozens of hardware and software versions is hard if not impossible and the end user (miner) will be left with no choice, but to use a specific miner and version, because of his hardware.
I hope you will agree that there is a need in such protocol at least because of this and i aim to design it as flexible as possible.
(EasyMiner is a BFGMiner GUI)
Unfortunately, cgminer wants to reinvent the drivers themselves for no gain. It's called not-invented-here syndrome.
There are drivers that MUST use libusb ... even in your crappy clone there are libusb drivers.
How on earth is that reinventing, when all the drivers in cgminer use the same libusb.
Get a life and learn how to program intelligently rather than flipping burgers at McDonalds.
I've always worked directly with manufacturers, often writing the code myself, so this isn't a real problem for BFGMiner.
And here comes the second problem - someone should do the same for CGminer and then someone else for EasyMiner and then for BitMinter etc.Managing dozens of hardware and software versions is hard if not impossible and the end user (miner) will be left with no choice, but to use a specific miner and version, because of his hardware.
I hope you will agree that there is a need in such protocol at least because of this and i aim to design it as flexible as possible.
(EasyMiner is a BFGMiner GUI)
Unfortunately, cgminer wants to reinvent the drivers themselves for no gain. It's called not-invented-here syndrome.
I've always worked directly with manufacturers, often writing the code myself, so this isn't a real problem for BFGMiner.
And here comes the second problem - someone should do the same for CGminer and then someone else for EasyMiner and then for BitMinter etc.Managing dozens of hardware and software versions is hard if not impossible and the end user (miner) will be left with no choice, but to use a specific miner and version, because of his hardware.
I hope you will agree that there is a need in such protocol at least because of this and i aim to design it as flexible as possible.
One of the problems i see is that for a manufacturer it will be difficult to follow (if at all) mining software development and to back it up with the words of ckolivas from https://bitcointalksearch.org/topic/m.3017560
"Oh and it's not "the current cgminer", it's the forked code based on an older version."
What about the miner in case a (security) bug is discovered in that older version several versions later, which is then fixed, but is missing the manufacturer's code and for him it will mean starting with a new fork?
By separating them, a fix or improvement in the mining software will not affect custom hardware module and the opposite an updated module will not require downloading and recompiling of new mining software.
I've always worked directly with manufacturers, often writing the code myself, so this isn't a real problem for BFGMiner."Oh and it's not "the current cgminer", it's the forked code based on an older version."
What about the miner in case a (security) bug is discovered in that older version several versions later, which is then fixed, but is missing the manufacturer's code and for him it will mean starting with a new fork?
By separating them, a fix or improvement in the mining software will not affect custom hardware module and the opposite an updated module will not require downloading and recompiling of new mining software.
Thus, protocol complexity gets pushed to the host driver, where it lives now.
Agree and this is the reason i am using the word 'driver'.The communication can be implemented in the microcontroller if it has enough power or on the host and only pass simple instructions to the hardware or by directly accessing internal registers.
When implemented as a kernel driver it will be able to better handle the communication to hardware when it is time sensitive and provide a socket in /var or /sys for the communication.
For Windows or when the microcontroller is powerfull enough a network socket can be used locally or remotely.
The protocol interface is the easiest part of most devices, handling the hardware requirements is where the trouble starts 
And it is exactly the opposite for the hardware people 
So why not to separate the two, but leave a wide bridge between?
One of the problems i see is that for a manufacturer it will be difficult to follow (if at all) mining software development and to back it up with the words of ckolivas from https://bitcointalksearch.org/topic/m.3017560
"Oh and it's not "the current cgminer", it's the forked code based on an older version."
What about the miner in case a (security) bug is discovered in that older version several versions later, which is then fixed, but is missing the manufacturer's code and for him it will mean starting with a new fork?
By separating them, a fix or improvement in the mining software will not affect custom hardware module and the opposite an updated module will not require downloading and recompiling of new mining software.
Protocol design affects speed, and because speed is the #1 optimization criteria, it is unlikely there will ever be a standard hardware protocol.
Each ASIC has unique performance characteristics that are in part determined by what is computed off-chip (queuing, pre-compute, scheduling).
Advancements that affect what is computed off-chip may be non-obvious or complex, to the point that implementing a standard interface in the micro-controller firmware would drive up costs (i.e. more costly micro-controllers or FPGAs).
Thus, protocol complexity gets pushed to the host driver, where it lives now.
Each ASIC has unique performance characteristics that are in part determined by what is computed off-chip (queuing, pre-compute, scheduling).
Advancements that affect what is computed off-chip may be non-obvious or complex, to the point that implementing a standard interface in the micro-controller firmware would drive up costs (i.e. more costly micro-controllers or FPGAs).
Thus, protocol complexity gets pushed to the host driver, where it lives now.
That's why VersionString was introduced as required field:
Manufacturers will implement only Work and gotNonce, because that's all they care about, but ...
Developers and users from the community will add more and more actions and logic behind them and if at at least some of the optional commands are available as direct translation to some registers based on EngineID translation - the software will be able to manage the hardware based on the VersionString, no matter how old is the driver implementation used from the user.
or at least that's the idea and end result i am aiming at
OK - old idea - even BFL have that in their current SC line ...Manufacturers will implement only Work and gotNonce, because that's all they care about, but ...
Developers and users from the community will add more and more actions and logic behind them and if at at least some of the optional commands are available as direct translation to some registers based on EngineID translation - the software will be able to manage the hardware based on the VersionString, no matter how old is the driver implementation used from the user.
or at least that's the idea and end result i am aiming at
I asked BitBurner to add that to theirs and they did.
But you've already missed something here ... when defining a 'one ring to rule them all' protocol, one version should be able to support different configurations, so version is not complete.
Again e.g. BFL SC firmware can enable and disable different options, so not only version would be required, but also a configuration response (like the BFL SC GetInfo)
So ... I'd ask directly: what is the point of a defined interface?
Best one I've seen so far is the Klondike - it doesn't handle everything needed for other boards, but for it's own purposes it's well designed.
The protocol interface is the easiest part of most devices, handling the hardware requirements is where the trouble starts
A different one each time is no issue at all IMO
Shoe horn? Lulz - got one example of that?
I mine 8 different devices on my cgminer instances ...
Icarus, AsicMinerUSB, BitBurner, Jalapeno, SC Single, BlackArrow Lancelot, BFL FPGA, ModMinerQuad
(I'm getting a Cairnsmore1 shortly also and probably a Klondike soon too)
Still can't see the shoe horn there anywhere.
I mine 8 different devices on my cgminer instances ...
Icarus, AsicMinerUSB, BitBurner, Jalapeno, SC Single, BlackArrow Lancelot, BFL FPGA, ModMinerQuad
(I'm getting a Cairnsmore1 shortly also and probably a Klondike soon too)
Still can't see the shoe horn there anywhere.
That's why VersionString was introduced as required field:
Manufacturers will implement only Work and gotNonce, because that's all they care about, but ...
Developers and users from the community will add more and more actions and logic behind them and if at at least some of the optional commands are available as direct translation to some registers based on EngineID translation - the software will be able to manage the hardware based on the VersionString, no matter how old is the driver implementation used from the user.
or at least that's the idea and end result i am aiming at
Manufacturers will implement only Work and gotNonce, because that's all they care about, but ...
Developers and users from the community will add more and more actions and logic behind them and if at at least some of the optional commands are available as direct translation to some registers based on EngineID translation - the software will be able to manage the hardware based on the VersionString, no matter how old is the driver implementation used from the user.
or at least that's the idea and end result i am aiming at
Users don't want different applications and interfaces for each of their devices.
That depends. If the two devices are completely different in their feature set, I'd rather have two applications open, than one common one that shoehorns everything into the same format that either throws away a lot of features, or requires an overly complex specification to capture all possibilities. Here we go....
The problem is that the information from the hardware is unstructured. One piece of hardware may have a board temperature sensor, another piece of hardware may have two. Or, maybe one for the board, and one for the power regulator, and maybe one integrated into each of the 512 chips. It's going to be impossible to find a common way to bind all these different kinds of properties, and find a nice way to format them all in a single line on a screen.
And the bizarre thing is that all the information about LEDs, clocks, voltages, error rates, and chip temperatures goes into the mining software, which doesn't really have a clue what to do with that. So, you need equally complicated config files to tell it that LED A needs to turn color B when temperature sensor C is >= D degrees.
Really, this information is something that only concerns the mining hardware and the user. There's no need to standardize it all, just for the benefit of the middle man who doesn't really care anyway. A better approach would be to create a mining library that's responsible for reliable network communication, and provides a standard get/put work interface. That way, the vendor can just write his own application, communicating with it's own hardware, and just call the mining library to get fresh work and submit the results.
Rather, the driver is the non-common component here. Users don't want different applications and interfaces for each of their devices.The problem is that the information from the hardware is unstructured. One piece of hardware may have a board temperature sensor, another piece of hardware may have two. Or, maybe one for the board, and one for the power regulator, and maybe one integrated into each of the 512 chips. It's going to be impossible to find a common way to bind all these different kinds of properties, and find a nice way to format them all in a single line on a screen.
And the bizarre thing is that all the information about LEDs, clocks, voltages, error rates, and chip temperatures goes into the mining software, which doesn't really have a clue what to do with that. So, you need equally complicated config files to tell it that LED A needs to turn color B when temperature sensor C is >= D degrees.
Really, this information is something that only concerns the mining hardware and the user. There's no need to standardize it all, just for the benefit of the middle man who doesn't really care anyway. A better approach would be to create a mining library that's responsible for reliable network communication, and provides a standard get/put work interface. That way, the vendor can just write his own application, communicating with it's own hardware, and just call the mining library to get fresh work and submit the results.
BFGMiner has this common driver abstraction already in place.
Here we go....
The problem is that the information from the hardware is unstructured. One piece of hardware may have a board temperature sensor, another piece of hardware may have two. Or, maybe one for the board, and one for the power regulator, and maybe one integrated into each of the 512 chips. It's going to be impossible to find a common way to bind all these different kinds of properties, and find a nice way to format them all in a single line on a screen.
And the bizarre thing is that all the information about LEDs, clocks, voltages, error rates, and chip temperatures goes into the mining software, which doesn't really have a clue what to do with that. So, you need equally complicated config files to tell it that LED A needs to turn color B when temperature sensor C is >= D degrees.
Really, this information is something that only concerns the mining hardware and the user. There's no need to standardize it all, just for the benefit of the middle man who doesn't really care anyway. A better approach would be to create a mining library that's responsible for reliable network communication, and provides a standard get/put work interface. That way, the vendor can just write his own application, communicating with it's own hardware, and just call the mining library to get fresh work and submit the results.
The problem is that the information from the hardware is unstructured. One piece of hardware may have a board temperature sensor, another piece of hardware may have two. Or, maybe one for the board, and one for the power regulator, and maybe one integrated into each of the 512 chips. It's going to be impossible to find a common way to bind all these different kinds of properties, and find a nice way to format them all in a single line on a screen.
And the bizarre thing is that all the information about LEDs, clocks, voltages, error rates, and chip temperatures goes into the mining software, which doesn't really have a clue what to do with that. So, you need equally complicated config files to tell it that LED A needs to turn color B when temperature sensor C is >= D degrees.
Really, this information is something that only concerns the mining hardware and the user. There's no need to standardize it all, just for the benefit of the middle man who doesn't really care anyway. A better approach would be to create a mining library that's responsible for reliable network communication, and provides a standard get/put work interface. That way, the vendor can just write his own application, communicating with it's own hardware, and just call the mining library to get fresh work and submit the results.
Thank You! This is exactly the type of feedback i was looking
The commands for get LEDs, temps and get/set clocks and voltages (listed in my second post here) will be added too, but i wanted to hear first if there is any preference for some of them to be optional or recommended for implementation.
Also i wanted to get some feedback about the EngineID idea as identification for each of them - not to replace the same text in each command later.
For the rest:
"when a job has been started ..." - most of the MH will probably not have them implemented, but i will add them as 'Debug commands' section ... probably separate command for each one with "DebugXXX,1" for On and "DebugXXX,0" for Off
"which specific slave board and chip found the nonce" - will add it as optional parameter for GotNonce (EngineID after timestamp)
About error checking - good catch 10x ... the idea was that MS will verify the nonce and have it's own information, but at least for BF chips there is a need for the driver to test several nonces, so it will be double work to do that again ... maybe 'TestOK'/'TestNA' as mandatory parameter with the returned nonce after JobID (before nonce) to indicate that the nonce was tested, so it can be hardcoded string
For other errors like communication problems and bad nonces - also debug on/off or recommended for implementation as GotError,EngineID,ErrCode[.ErrDescr] with a list of default codes ?
I think the later is better and maybe with status - triggered on/off or counter, but then it will be one way information only or may add GetErrors/ClearErrors ... will think a bit more on the subject and will see what i will came with
EDIT: instead of 'TestOK'/'TestNA', i think returning 0 or the share diff is a much better idea, as MS will still need it and will have to DblSha() anyway if ommited
The commands for get LEDs, temps and get/set clocks and voltages (listed in my second post here) will be added too, but i wanted to hear first if there is any preference for some of them to be optional or recommended for implementation.
Also i wanted to get some feedback about the EngineID idea as identification for each of them - not to replace the same text in each command later.
For the rest:
"when a job has been started ..." - most of the MH will probably not have them implemented, but i will add them as 'Debug commands' section ... probably separate command for each one with "DebugXXX,1" for On and "DebugXXX,0" for Off
"which specific slave board and chip found the nonce" - will add it as optional parameter for GotNonce (EngineID after timestamp)
About error checking - good catch 10x ... the idea was that MS will verify the nonce and have it's own information, but at least for BF chips there is a need for the driver to test several nonces, so it will be double work to do that again ... maybe 'TestOK'/'TestNA' as mandatory parameter with the returned nonce after JobID (before nonce) to indicate that the nonce was tested, so it can be hardcoded string
For other errors like communication problems and bad nonces - also debug on/off or recommended for implementation as GotError,EngineID,ErrCode[.ErrDescr] with a list of default codes ?
I think the later is better and maybe with status - triggered on/off or counter, but then it will be one way information only or may add GetErrors/ClearErrors ... will think a bit more on the subject and will see what i will came with
EDIT: instead of 'TestOK'/'TestNA', i think returning 0 or the share diff is a much better idea, as MS will still need it and will have to DblSha() anyway if ommited
Jump to: