Topic: M's Miner (Ant/SP) Monitor v5.2: alerts,auto/mass/scheduled reboot/mobile miner - page 31. (Read 209500 times)

Here's another beta.  This version fixes a few things, including re-enabling the web scraping code.  I don't think there's any gain to the web scrapers in this code, but it works again.

I also added a configurable "display refresh" delay the defaults to 1 second.  That's how long the display holds off refreshing from the time an Ant refresh is initiated.  Users with slow video or large numbers of Ants will benefit from making this value larger. 

For example, when set to '1', every Ant that refreshes in the first second will refresh on the grid at once.  Every Ant that hasn't yet responded, or hasn't yet been refreshed, will show up automatically as it's refreshed.  Using the test results posted earlier, where it took 5 seconds to refresh 50 Ants when the grid wasn't shown, if you set it to 5 seconds, most if not all should refresh after 5 seconds.

This is pretty close to the final build for this version number.

As always, feedback is welcome and appreciated.

Download link: MAntMonitor32b2.zip

M

That's an interesting test and results.  I am purposely rendering the grid after each one returns, as there could be situations where if I wait for them all to return, it'd never refresh.  (Say a few aren't responding, or the refresh rate is so low that it restarts before it's finished.)

That said, I don't have to *instantly* refresh the grid.  I'll make a minor tweak that will slow down the refreshing of the grid, but it'll still be fast enough to appear to be pretty quick.  In your situation it should speed things up.

Thanks!

M

Matt, I'm not sure the biggest performance issue here is the actual polling of the Ants ... I think it might be the drawing of the status table.

So I was testing out the new beta with the thread slider on a 2 vCPU Win7 VM against 54 Ants (mixed S1/S3).  Testing results while viewing the status window on "Ants" tab showed:

1 thread - ~35s
2 - 3 threads - ~20s
4+ threads - ~10s
From 4 to 64 threads there was no difference in performance.

While I was playing around with the different thread values, I initiated a manual refresh while still on the "Config" tab accidently, not switching back to the status grid before doing the refresh.  While on that tab:

1 - 3 threads - ~6s
4 - 8 threads - ~5s
12 threads - ~4s
32 threads - 64 threads - ~2s

It was amazing the performance difference when not on the "Ants" tab showing the status grid.

Not sure how your logic is done, but are you waiting for the return of the values for the Ant per thread and drawing each one at the end of the thread before moving to the next?  If so it might make sense to do the poll in the background, write the info to an output area, maybe a file, and then you can generate the status table from there.  Either way it may be limited by the graphics processing part of this.

Now, I'm on a VM, which has crappy video performance in general being that it's a VM and meant for server purposes, not high end resolutions and performance.  Maybe that has something to do with it but I don't have a regular machine with a good video card to test it that way yet.  Could do on Tuesday with my laptop.

Your change requires a bit of reworking my code.  I intend to address your issues when I redo the UI for config.

M

If you're using the latest beta, and the ant grid is the wrong size again, please redownload.  It changed sizes of it's own accord again.

M

Here's hoping the next release is geared towards me ;-p

Here's another beta of 3.2.

This version adds a slider bar on the config screen where you can configure how many threads you want to use (2 to 64).  It defaults to 5 (1 dispatch, 4 worker).  You can change it midstream, however it'll abort the existing threads regardless of what they're doing.  For best results you shouldn't do it while it's rescanning, but I think it'll continue to work either way.  Of course you can save your thread setting so next time it'll start out at your new value

It also tweaks a few more spots that could cause some data corruption because they weren't synchronized across threads.

Download link: MAntMonitor32b1.zip

You're going to have to experiment with whatever threadcount works best for your configuration.  Please keep me posted on your findings.

Thanks!

M

Thank you.  I think I've worked out all the multi-threaded issues in this version.  Now I need to clean up the code, get the web scraping working, make sure everything works,  and add the ability to control the number of worker threads, and this will become the next stable version.  I may include revising the scanning in this version too, depends on how much time I have.

After that I intend to completely revise the Ant configuration methodology.  That should address the rest of the outstanding issues and requests.

M

I'm not doing the sort, the control is.  It's not a numeric value, as it contains periods.  It's a string value.

If I was sorting it myself I wouldn't have a problem.  I may go there; for now I was posting a quick work around.

M

Just downloaded and tried it out.  I'm using the most simplest functions of just getting status so I'm not a good measure of whether any other functionality is broken, but I am using the API and not web scraping.  It definitely seems faster so far, and you can tell the refresh is working on multiples as they used to fill in one by one, now it's in groups of 4 or so just like you mentioned.

We've got 50+ Ants being monitored, and it takes about 10 seconds now to go through all of them, whereas it used to be more than double that, close to 30 seconds for a full refresh.  This is on a Win7 VM with 2 vCPU's.  I can bump the vCPU to 4 and see if it makes any difference as well.

try sorting the IPs as a version object using linq sort. Should work

I was thinking about this.  I think you can fix this by entering all the ips as 3 digit ips, so 192.168.1.2 would be 192.168.001.002.  Then I think it'll sort properly.

M

Here's a beta version of 3.2.  The huge change to this was it was restructured to do multi-threaded polling of your Ants via the API.

Things to note:

- This is a beta version.  Chances are good I broke something. 
- That said, you should be able to revert back to the prior version simply by running the older version.
- This is for API users only.  I'm pretty sure the web scraping piece is broken, simply because I haven't taken the time yet to redo it to work with the new code.
- It has 5 threads running behind the scenes: 1 dispatch thread, and 4 worker threads.  I intend to have the worker threadcount controllable in the final version.
- You'll see in the Log which worker thread is checking which Ant.
- My main PC is an i7.  That means have 4 CPU cores with hyperthreading, so the OS sees 8 cores available.  I don't know how this will respond on CPUs with less cores available. 
- I only have 3 Ants: 1 S1, 1 S2, and 1 S3.  I can't really test how this will work with more than 3 Ants, especially not 1000.

As always, feedback is most welcome.

Thanks!

Download link: MAntMonitor32b.zip

M

I believe you

Have a nice weekend!

I have a plan that I think will address your remaining issues and the issues with having 1000 of them.

M

It probably has to do with my unique setup to which I suspect is not the same as yours.

Thanks.  That's not what I get when I scrape or use the API.  I'll have to look at this closer.

M

I got it, tyvm!


I'm working on recoding the refresh algorithm to be multi-threaded.  That should speed it up quite a bit.  

I'll also work on revising the search routine to see if it can help you.


Removing the machines from the config doesn't currently remove them from the output grid.  I'll add that to the todo list.

Someone also suggested having a way of removing/disabling from the output grid.  I'll be adding that as well, it should make it easier to remove/disable Ants from the output.


I see what you mean about the sorting.  I'll have to think about that.  Right now it's a string value, and .6 comes before .61 in strings.

Thank you for your support, I'll be getting these changes out as soon as I can.  It may take a few days.

M

See what I'm talking about now ?

It shows the full URL stuff that's used when say visiting the miner in a browser.

So instead of S3:IP:PORT you get what you see in the pic.

That's what I get wen scraping, I assume it's different with the API which last you said you were working on for my unique situation ;-)