Topic: Flexible mining proxy - page 13. (Read 88658 times)

Took me a while, but I got it set up! Thanks!

1 BTC heading your way

can you give me a name of web hoster where your proxy can work ?

All i test, had 'php_auth_user' disable ......

Sorry for my poor english

I just cranked up a new kind of sniffer, one that logs each unique tuple into a separate file.  What I found is that upon receiving my input text (fed to your proxy via telnet) as follows:


Your proxy actually connected to deepbit twice.  The first time was apparently to setup the actual long-polling session for my telnet to the proxy.  The second communication was completely blank - neither side said anything.  After a couple of minutes, my telnet session spat back the following:


This was not in response to anything being sent from deepbit to the proxy - deepbit never at any time uttered another word.  I ran the same testing iteration again and got a successful long poll when the pool jumped to the next block, but on the next iteration it was exactly the same behavior as above.  (note that all of this is still based on a 2-day old pull, I did not pull the code again per your mention above).

Does that help?

Hmm... I'm not going to discount this explanation, but I don't see any evidence that sending an HTTP 1.0 request is causing problems.  My workers are issuing long-polling requests to deepbit through the proxy and they appear to be working just fine.

I've just done a push.  I recommend that everyone update from master.

This update fixes some PHP warnings that would get triggered on a failover, effectively breaking failover entirely on configurations where PHP warnings are emitted.  The pool timeout has also been lowered from 5 seconds to 2 since the default configuration of some miners is to request work every 5 seconds.  These miners will break if the proxy takes too long to respond, terminating the active getwork request and issuing another.  (This timeout will be made configurable in a later push so that you can customize it to your configuration.)

An error_reporting() call has also been added to the config.inc.php.sample file.  It's strongly recommended that you copy this into your config.inc.php to mitigate any possible future issues caused by PHP warnings getting mixed into the JSON response for workers.

Ahhh I see the problem.  From your proxy to deepbit:

GET /listenChannel HTTP/1.0
Authorization: Basic youcanthavemypassword;]
Host: deepbit.net:8332

I watched it send this to deepbit once and it worked, but the next time it didn't.  The request was the same, but it appears to be failing because you're sending the request as HTTP 1.0.  Based on the reply from the pool, you can see that the pool wants to serve HTTP/1.1.  So my guess is we're hitting nginx, which is setup as a loadbalancer, and one of the backend machines is configured to work with HTTP 1.0 and the other is not.  You can also see from my previous post that the client normally asks in HTTP/1.1 (though in the long polling specification at deepbit's website, they don't mention that it needs HTTP/1.1).  If that all looks sensible, can you commit a fix to make the long polling requests in HTTP/1.1?

Thanks!

Hmm, I thought I did.  Maybe I just did that in my own php.ini or something.  Anyway, thanks for the comment.  I'll add this in a future push.

hey cdhowie,

great proxy, just checking it out.

Had problems with the php notices spamming the pages to the point where chrome refused to render them (they get inserted before DOCTYPE decl):

Example:

I simply put
in my config.inc.php, that fixed the issue nicely.

Maybe you want to set the error_reporting level somewhere centrally to avoid this problem happening to others and also to avoid having to explain all the notices.

Yup, that's why it's there.    Not all of the pages have proper JSON support implemented, simply because for most of them it wouldn't be terribly useful.  But at some point, the proxy should allow full manipulation of the database objects via JSON, allowing you to code an alternative interface (like a desktop app) against the JSON API.  That probably wouldn't be a good thing to do, but it would at least be possible.

Alright, I'll have a look at it at some point later.  Thanks for the patch!

Great, it works for me too, just had to change the URL in my scripts. I think using such an API to run cronjobs against is more useful than having it crammed into the web interface.

Agreed

I guess I can drop the stats branch then, since it already works out of the box from your dashboard. On the worker-hashing branch I made the average interval configurable because it strongly depends on the hashing speed (high hashing speeds can get a more accurate reading with shorter intervals, while slow miners will have the average jump wildly around for small values). I think it is ready to be pulled. Once again sorry for all the noise I created in here ^^

There is support in the MVC framework for JSON output.  The dashboard view page already implements IJsonView so try visiting /admin/?format=json.  This works for me.

You're right, I'm not sure why that's there.  That being said, PHP is a whiny little bitch and it likes to complain about things that aren't problems.  

I haven't looked at it in-depth yet.  Let me know when you are done with it and have tested it and I'll review the pull request.

I'm not sure I could include the IP since that's not logged anywhere.  But I could include the worker's database ID as well as the worker name.

To keep database schema changes to a minimum, I'm not sure if I would want to do this.  But if I pass the worker's ID to the script, you could have your script look at the ID and figure out what to do.

Hmm.  This capture is helpful, but without seeing the long-polling request and response that the proxy itself sends out, I can't really say why it's doing this.  It is possible that deepbit itself is returning an error from the long-polling request and I'm just forwarding it on to the worker.

Being a stats junky I'm working on a simple page that dumps the data in the dashboard to JSON so that I can then write a munin plugin to monitor the status of my miners, I guess we could use that one with a cronjob to kick off custom scripts. It would make it much easier and more flexible since the web server user is usually quite constrained in what he can do ^^

BTW: htdocs/index.php@71 is throwing a lot of errors since in getwork requests the lpurl parameter is not set, and since we already check 2 lines below and then set $lpurl again it is redundant and should be removed ^^

Edit: My first attempt on a JSON stats dump is here: https://github.com/cdecker/Bitcoin-mining-proxy/commit/82f8ad9e352352cdbe81d84a8939b817204cb59d

Edit2: hope you didn't yet look at the stats and the worker-hashing branch because I was missing two GROUP BYs in the shares subquery which resulted in the hashing rate of all miners to be reported on a single miner instead of distinguishing. I fixed it in the branches ^^

Correct.  One could specify a shell script in the database, and perhaps two other user-defined arguments which would be called like:

`$script $minerip $timedelta $userdefinedarg1 $userdefinedarg2`

And would hopefully cause the script to send me an email letting me know that miner 1.2.3.4 has been down for 300 seconds.

No problem!  I'm glad you find the software useful. 
[/quote]

Useful indeed, if only I could make it function as flawlessly in my environment as it does in yours.  So as I mentioned, Phoenix 1.2 prints periodic "disconnected" messages.  Phoenix 1.45 seems to sporadically connect to the long-polling URL.  Phoenix 1.2 will connect, but it disconnects after a period of time.  I think I see why.  I hit my local mining proxy and fetched the long polling URL.  The first time it worked, and actually sent me info on a new block.  But the next time I tried it, I encountered a problem that almost surely explains what I'm seeing in Phoenix:

GET /index.php?lpurl=http%3A%2F%2Fdeepbit.net%3A8332%2FlistenChannel&pool=1 HTTP/1.1
Connection: close
Host: bitproxy.xxxxx.com
Authorization: Basic xxx
User-Agent: phoenix/v1.2


(((2 minutes elapse, and then...)))


HTTP/1.1 200 OK
Date: Tue, 10 May 2011 06:00:12 GMT
Server: Apache/2.2.3 (CentOS)
X-Powered-By: PHP/5.1.6
X-Source-Code: https://github.com/cdhowie/Bitcoin-mining-proxy
Set-Cookie: PHPSESSID=4j2r53dsse5afoa5u9k3oratn0; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Content-Length: 78
Connection: close
Content-Type: application/json-rpc

{"error":"Invalid response from long-poll request.","result":null,"id":"json"}Connection closed by foreign host.


Based on the 3 or 4 times I've done this, it seems to happen when there is in fact a new block.  This should be easily duplicable for you.  If not, I can PM you the info on my proxy and let you see for yourself.  Hope that helps.

Cool, keep me posted on that.

Sure!  I planned to do this at one point but forgot about it later.  Thanks for reminding me.

Yeah, this is probably something that could be done.  The red display is pretty easy to do; the shell script will require DB schema changes and a bit more work, but will still be possible.  I assume this would be used to, for example, email you if one of the miners stops requesting work?

I could define three new configuration options: one for the amount of time a miner must have not requested work for it to display differently, one for the amount of time a miner must have not requested work for the script to be run, and one for the script to run.

No problem!  I'm glad you find the software useful. 

Just wanted to update this to let you know that Phoenix 1.2 (version I've run for the last couple weeks) seemed to exhibit the problem, while version 1.45 has not yet exhibited the problem during several hours of testing.  I'm going to let this thing run for several more hours, then switch over a few more miners to the proxy and see how it goes.

Two additional feature requests:

1) Can you add in code (maybe configured by config.inc.php) to adjust the timestamp according to a user-defined timezone?  I've become accustomed to looking at UTC and knowing the offset, but still would be a nice thing to have.

2) The purpose of my push for your software, and what makes it valuable to me, is being able to know if a miner stops pulling and submitting shares (in addition to all of the wonderful features you've coded in so far).  Is this a feature you plan to add in the future, i.e. with simply listing all of the miners in red/green status like deepbit does, and having the option to act on it somehow, like with a shell script?

Thanks again for all of your hard work!

If I'm reading this right, the miner is actually closing the long-polling connection of its own accord, mid-stream.  What miner are you using?  Perhaps I can reproduce the issue locally.

Ok further to my issue, I captured one of those sessions per below:


This is mid-stream, i.e. I let the miner run for maybe 2 minutes, then turned on the sniffer when I thought it would throw the error, so it sent in maybe 2 or 3 successful shares, then the miner threw a disconnection error per below:

[09/05/2011 13:43:22] Result: f780500d accepted
[09/05/2011 13:43:53] Result: 98a41184 accepted
[09/05/2011 13:44:06] Disconnected from server
[09/05/2011 13:44:07] Connected to server

The capture looks completely normal to me, based on my (very lay) knowledge of the protocol. 

No.  The first is a user in the MySQL database; all of the workers as well as the admin account will cause the proxy to authenticate to the database server with these credentials.  The admin user is a separate set of credentials that you use to log in to the proxy management console.  You can pick whatever you want for the admin credentials, but the MySQL credentials must match a MySQL user.

No.  When you use the "/admin" URI, it will ask for admin credentials.  When you use the "/" URI it will ask for worker credentials.

yes sorry....

In your Install file you say:
2.  Create a MySQL user for the proxy, giving it the privileges SELECT, INSERT, UPDATE, DELETE, and LOCK TABLES on the proxy's database only.  (Optional, but recommended.)

Is it the same user than:

3.  Set the admin user and password.  This will be used when logging in to the web management console.


and an other question is, when i connect on the "/" of my proxy, it ask me a login and password.
Is it the same than in the admin section ?

sorry for my bad english