Topic: Flexible mining proxy - page 12. (Read 88658 times)

Sweet.  I have what looks like a working implementation, minus the aforementioned difficulties with long-polling that me and several others have experienced.  But it wasn't in use today for that reason.  I'm hoping to find a simple http proxying script and just change the LP URL handed out by cdhowie's script, to make the miners hit a different script to handle long polling in a non-problematic way.  Or so I hope.

It's pretty transparently switching between pools. The miner never notices a thing.

For those of you affected by deepbit/slush outages today, how well did this code hold up for you?

Thanks for the proxy. Seems to work pretty well so far.

I did have to make one change to fix a problem with the proxy out of the box. Whenever I submitted a form, I'd be redirected to some other web site (localhost) which has no server running. It appears you're reading SERVER_NAME, which may or may not be correct. It's more reliable to read HTTP_HOST:

10h should be ok.  It is needed when a miner eventually submits a share.  The proxy uses the information in the work_data table to look up which pool the miner got that work from so that it gets submitted back to the right place.  So don't delete everything in the table, but it is not needed as soon as your miners have finished working on their current work task.

Is the stuff in work_data needed for something importent? I added to my runnning script a function which remove all entries older than 10h.

Yes, there is no standard.  But it only turned out to be a couple lines of code specific to each type of pool to fish the balance out of each pools JSON API.

It actually doesn't work that well and is the main thing I am working on right now before releasing the code in the next day or two.  Making all of the JSON API calls to retrieve current balances is making the dashboard load slowly, so I'm moving balance updating to a background thread.

Just seeing now that you have a payout monitor in there, that would be a cool feature, though it adds a lot of problems since some pools don't provide that info and even worse, they don't adhere to a consistent standard for publishing them.

I do, I definitely want to see the code.

Hey, cdhowie.  I'm still not clear on what was happening with the proxy, but I have a couple clues.

I did determine that the latest phoenix behavior is to re-request the long polling URL immediatlly if the LP does not return a valid getwork response.  So, I think the issue with phoenix hammering the web server was because the proxied long-poll address must have been immediately returning, which caused phoenix to request it again right away, which caused...  well, a very fast loop of LP requests to the proxy.

In the end, I only know enough about PHP to be dangerous, and less about how to run an apache web server.  In the end, I was so inspired by the idea of this proxy that I ended up recoding it in ASP.NET, not because it is better than ASP.NET but just because I have 10 years of experience with it and know how to debug.  For the actual proxying of the JSON-RPC requests, I followed your algorithm almost exactly, just in C#, ASP.NET MVC, IIS and SQL Server.  I now have it working and long polling is working as well although the whole design of long polling tie-ing up a web service thread for 10-20 minutes doesn't play all nicely with IIS, but it doesn't matter too much for my 4 measly miners.  A side bonus is that ASP.NET has a very capable charting component and so I was able to quickly whip up some graphs for the dashboard as well.

To be clear, I doubt anyone will care about my implementation.  An IIS web server and SQL Server are much harder to come by than a Linux, Apache, MySQL, and PHP server.  I just happen to have a suitable windows server in my house already, so it worked well for me.  That said, as soon as I polish it a little more, I will post the source code somewhere given that I did borrow the idea and some of the assets from your project.

Just pushed some updates.

• Applied patch from cdecker implementing hash speed estimation for workers (issue 5).
• Implemented timezone conversion (issue 7).

These changes require modifications to your config.inc.php.  If you pull without updating your config.inc.php, the software will break.

To address some posts:

Some miners will bail on a connection if it doesn't return work within 3-5 seconds.  This is not always enough time for the proxy to detect a fail condition, and so the miner gives up before the proxy has a chance to try another pool.

I have started seeing similar problems on my install.  This is being actively researched.

I have been seeing the same with poclbm lately.  I'm not sure what changed to make this happen, but I suspect it is related to the stale shares problem.

I haven't used that particular miner and so I'm not sure if the proxy supports it.  If the miner isn't doing anything too wacky it should work...  If I get some free time I might investigate this.

There are some issues with long polling that are not easily addressed by the current model.  Long polling should in theory work just fine if your preferred pool doesn't go down, but during a failover scenario it can get a little bit wonky.  Some miners behave differently with respect to how they handle changing long-polling URLs, and so this is something that will take a lot of engineering to fix.

In the meantime I may supply a config option to disable long polling entirely; broken long polling is worse than none at all.

Sounds like a bug in Phoenix.  Can you take some sniffs of this behavior with tcpdump -w bmp.pcap port 80 and email me the pcap file(s)?  This will help me figure out what's going on.  (Please compress them, and you can optionally encrypt them to my GPG key.  This should not expose your mining account passwords, only your local proxy worker and possibly admin passwords.)

As long as the computers can make outbound connections on whatever port you are running your web server on, yes.  Transparent (or non-transparent) HTTP proxies should be able to process these requests too, though they might strip long-polling info.

Watching (very nice project btw)

I'd like to set this up to manage a mining for computers I have in different locations, but I am hesitant because of reports by some users of stale shares.

Thank you so very much cdhowie for even releasing what looks like an amazing piece of software.


Btw if I have access to computers at school and they are behind a firewall and so cannot mine, can I use this software to get around that?

I really like the idea of this tool, but have failed to get it working well.

I use phoenix as my miner on ubuntu 11.04.  I have tried a couple different web hosts including my own ubuntu 11.04 box, and I have the following symptoms:

1. lots more stale shares than when connected directly.  long polling does not seem to work with phoenix through the proxy but works fine when connected directly.
2. phoenix miner *hammers* the web server with requests to index.php.  Something like 10-20 requests per second as seen in the apache access.log.  When connected directly and observed with tcpdump it behaves much more normally (1 request every 5 seconds or so).
3. lower than normal hash rate when using the proxy.  I suspect this may be because the miner is so busy doing #2 as fast as it can.

I also tried poclbm on my Mac and it had similar problems with stale shares.  I did see an occasional LP message, but not frequently enough as compared to blocks being found in the network.  I also regularly got errors about "NoneType" not being indexable.

From this thread, though, it sounds like some people are having great success with it, so I wish I knew what to do differently.

hi

there are problems with php-authentication, if php is running as fast-cgi, but after switching to mod_php, login to admin-panel is working.

i believe ufa cpu-miner is not supported at the moment? all my tests ran in timeouts

Actually this did not run as well as expected. I've got a very high rate of rejected (about 10%) and after running for two hours phoenix stalled again:

Traceback (most recent call last):
  File "phoenix.py", line 125, in
  File "twisted\internet\base.pyc", line 1162, in run
  File "twisted\internet\base.pyc", line 1171, in mainLoop
--- ---
  File "twisted\internet\base.pyc", line 793, in runUntilCurrent
  File "minerutil\RPCProtocol.pyc", line 127, in callback
  File "minerutil\RPCProtocol.pyc", line 319, in handleWork
exceptions.TypeError: argument of type 'NoneType' is not iterable
[23/05/2011 01:04:42] Result: 4b18edf7 rejected

My connection has some packet loss (~2-4%) but the latency in general is good (under 150ms). Could this be a problem?

Great peace of software. Managed to get it running in under 10 minutes.
I've always had some cases in the past where there were connection problems and the miner was in a stale state after connectivity has been restored. Looking forward to see if this is solved when using the proxy.

It also would be an interesting idea to take such a service and offer it for hosting, so that the average miner does not have to worry about pool failover any more as he can simply point to some public proxy which handles the requests.

Switches will never occur on work submission.  Submissions always route back to the pool that the work came from.

Sounds strange. The switch should occur either upon work request or work submission. The latter isn't very nice because a share might be lost. We are working on exponential backoff to temporarily blacklist flapping pools (intermittent connection issues). Should it occur again please provide a tcpdump (just let tcpdump run along capturing traffic from/to the pool proxy port 80, and use temporary passwords between pool proxy and the workers).

Actually this just didn't work very well for me   Set it up with BTC Guild as primary and Slush as secondary.  BTC Guild is having intermittent connection issues, so I wanted a backup.  However, overnight all of my GUIMiner clients got stuck with a "connection problem".  My one phoenix client (which is also on same machine as my proxy) continued working fine however.