Author

Topic: [ANN][BURST] Burst | Efficient HDD Mining | New 1.2.3 Fork block 92000 - page 647. (Read 2171078 times)

sr. member
Activity: 416
Merit: 250
I offer 1k BURST to someone who can tell me how can I (pool)mine with 5.5 TB without running out of 8 GB memory in a few minutes. I used all kinds of stagger sizes from 1024 to 16k, plot sizes from 2 GB to 5.5 TB using bipben's gpu plotter and tried uray's burst-miner and blago & dcct's miner. Using win7 and tried disabling prefetch/superfetch and various windows services without success.

It's Win7 problem
https://bitcointalksearch.org/topic/m.9086411
sr. member
Activity: 397
Merit: 250
legendary
Activity: 2002
Merit: 1051
ICO? Not even once.
Mine when you're not busy on your comp,and then when youre busy stop Wink

My main rig is always busy and while it has more memory I can't have a loose app eating all the memory over time on it. The plots are on one of my gpu mining rigs. I stopped gpu mining while I was fooling around with plotting so that didn't interfere either. It's not even about profit anymore, just bugs the crap out of me that I can't figure it out Cheesy
hero member
Activity: 588
Merit: 500
Strange. It should read everything....what is your stagger atm?

1024

Use a vm...one running on a usb stick or something...

It won't make a difference because it seems it doesn't matter which application reads the plots, be it a miner, a file viewer or chkdsk, the result seem to be the same which is win7 choking on it's own file reading cache which I can't control.
Mine when you're not busy on your comp,and then when youre busy stop Wink
legendary
Activity: 2002
Merit: 1051
ICO? Not even once.
Strange. It should read everything....what is your stagger atm?

1024

Use a vm...one running on a usb stick or something...

It won't make a difference because it seems it doesn't matter which application reads the plots, be it a miner, a file viewer or chkdsk, the result seem to be the same which is win7 choking on it's own file reading cache which I can't control.
hero member
Activity: 588
Merit: 500
Strange. It should read everything....what is your stagger atm?

1024

Use a vm...one running on a usb stick or something...
legendary
Activity: 2002
Merit: 1051
ICO? Not even once.
Strange. It should read everything....what is your stagger atm?

1024
hero member
Activity: 588
Merit: 500
I offer 1k BURST to someone who can tell me how can I (pool)mine with 5.5 TB without running out of 8 GB memory in a few minutes. I used all kinds of stagger sizes from 1024 to 16k, plot sizes from 2 GB to 5.5 TB using bipben's gpu plotter and tried uray's burst-miner and blago & dcct's miner. Using win7 and tried disabling prefetch/superfetch and various windows services without success.
I'm surprised that is a problem with 8GB. I have a machine on Windows 7 with 4GB of ram running 2 instances of the java miner, one for each of two 2TB drives all plotted with stagger 8191 by another machine. Disabling prefetch was vital, and the system is unusably slow, but it's been mining stable for weeks.

Disabling prefetch/superfetch is the registry tweak.

Interesting. I disabled the pagefile completely and now when it climbs up to 100% memory usage it trashes itself back to ~13% at which point the plot reading is at ~58%. When that happes it stops completely along with disk reading until the next block. So with insufficient memory it doesn't seem to read everything. This is the case with both C miners and the Java miner crashes with error reading file.
Strange. It should read everything....what is your stagger atm?
legendary
Activity: 2002
Merit: 1051
ICO? Not even once.
I offer 1k BURST to someone who can tell me how can I (pool)mine with 5.5 TB without running out of 8 GB memory in a few minutes. I used all kinds of stagger sizes from 1024 to 16k, plot sizes from 2 GB to 5.5 TB using bipben's gpu plotter and tried uray's burst-miner and blago & dcct's miner. Using win7 and tried disabling prefetch/superfetch and various windows services without success.
I'm surprised that is a problem with 8GB. I have a machine on Windows 7 with 4GB of ram running 2 instances of the java miner, one for each of two 2TB drives all plotted with stagger 8191 by another machine. Disabling prefetch was vital, and the system is unusably slow, but it's been mining stable for weeks.

Disabling prefetch/superfetch is the registry tweak.

Interesting. I disabled the pagefile completely and now when it climbs up to 100% memory usage it trashes itself back to ~13% at which point the plot reading is at ~58%. When that happes it stops completely along with disk reading until the next block. So with insufficient memory it doesn't seem to read everything. This is the case with both C miners and the Java miner crashes with error reading file.
member
Activity: 111
Merit: 10
Just throwing this out here, but has anyone tried storage with de-duplication to get more plots on disk?  Like this free up to 1TB DXiV1000 virtual appliance VM - http://www.quantum.com/products/disk-basedbackup/dxiv1000/index.aspx .  I turned one on at work as a test for our backup system and it de-duped about 1TB of disk (12VMs) down to 50GB (40:1) compression.  

I am guessing that the way the plots are created this is not possible but I would like to toss this out there...
hero member
Activity: 588
Merit: 500
I offer 1k BURST to someone who can tell me how can I (pool)mine with 5.5 TB without running out of 8 GB memory in a few minutes. I used all kinds of stagger sizes from 1024 to 16k, plot sizes from 2 GB to 5.5 TB using bipben's gpu plotter and tried uray's burst-miner and blago & dcct's miner. Using win7 and tried disabling prefetch/superfetch and various windows services without success.
I'm surprised that is a problem with 8GB. I have a machine with 4GB of ram running 2 instances of the java miner, one for each of two 2TB drives all plotted with stagger 8191 by another machine. Disabling prefetch was vital, and the system is unusably slow, but it's been mining stable for weeks.

You may want to update the OP about the bmpool asset....
hero member
Activity: 588
Merit: 500
I offer 1k BURST to someone who can tell me how can I (pool)mine with 5.5 TB without running out of 8 GB memory in a few minutes. I used all kinds of stagger sizes from 1024 to 16k, plot sizes from 2 GB to 5.5 TB using bipben's gpu plotter and tried uray's burst-miner and blago & dcct's miner. Using win7 and tried disabling prefetch/superfetch and various windows services without success.

Use a vm with linux.

OR

Mine on dev pool with the java miner....doesn't use the ram up like that


Just tried pocminer_pool_v1 and memory usage went from 8% to 99 plus some pagefile in less than minute.
There has to be some memory management config to prevent this via the registry beucase not the actual miner is using that amount of memory but some kind of read caching which uses the memory 10-20 minutes after the miner has been closed and even when the plots are relocated.

I'd rather not use linux - for now.

Windows vm in windows? You can limit resources like that...
sr. member
Activity: 280
Merit: 250
I offer 1k BURST to someone who can tell me how can I (pool)mine with 5.5 TB without running out of 8 GB memory in a few minutes. I used all kinds of stagger sizes from 1024 to 16k, plot sizes from 2 GB to 5.5 TB using bipben's gpu plotter and tried uray's burst-miner and blago & dcct's miner. Using win7 and tried disabling prefetch/superfetch and various windows services without success.
I'm surprised that is a problem with 8GB. I have a machine on Windows 7 with 4GB of ram running 2 instances of the java miner, one for each of two 2TB drives all plotted with stagger 8191 by another machine. Disabling prefetch was vital, and the system is unusably slow, but it's been mining stable for weeks.

Disabling prefetch/superfetch is the registry tweak.
legendary
Activity: 2002
Merit: 1051
ICO? Not even once.
I offer 1k BURST to someone who can tell me how can I (pool)mine with 5.5 TB without running out of 8 GB memory in a few minutes. I used all kinds of stagger sizes from 1024 to 16k, plot sizes from 2 GB to 5.5 TB using bipben's gpu plotter and tried uray's burst-miner and blago & dcct's miner. Using win7 and tried disabling prefetch/superfetch and various windows services without success.

Use a vm with linux.

OR

Mine on dev pool with the java miner....doesn't use the ram up like that


Just tried pocminer_pool_v1 and memory usage went from 8% to 99% plus some pagefile in less than minute.
There has to be some memory management config to prevent this via the registry because  it's not the actual miner using that amount of memory but some kind of read caching which uses the memory 10-20 minutes after the miner has been closed and even when the plots are relocated.

I'd rather not use linux - for now.
hero member
Activity: 588
Merit: 500
I am wondering if mining on a 5400 rpm HDD has the same performance as a 7200 rpm hdd.If their capacity is the same.


You're wondering loudly. lol.

I honestly don't know since I haven't run anything but 7200 RPMs, but I don't believe there should be that much of a difference. Maybe someone with a little more knowledge on the underside of how burst actulally mines could tell you.

But I would say it's definitely fine to mine on them, and it won't be that much of a difference.

Both mine just fine...just the plots will be read a little faster on the 7200rpm drive. If you did the thing with splitting up scoops then there would be an advantage for faster drives, but as it is with only reading 1/4096 scoops per plot it doesn't matter. Even USB 2.0 drives work fine....at least pinballdude says it does...dunno if he's using like 4tb, cause that would cause problems i'm guessing.
hero member
Activity: 588
Merit: 500
I offer 1k BURST to someone who can tell me how can I (pool)mine with 5.5 TB without running out of 8 GB memory in a few minutes. I used all kinds of stagger sizes from 1024 to 16k, plot sizes from 2 GB to 5.5 TB using bipben's gpu plotter and tried uray's burst-miner and blago & dcct's miner. Using win7 and tried disabling prefetch/superfetch and various windows services without success.

Use a vm with linux.

OR

Mine on dev pool with the java miner....doesn't use the ram up like that
legendary
Activity: 2002
Merit: 1051
ICO? Not even once.
I offer 1k BURST to someone who can tell me how can I (pool)mine with 5.5 TB without running out of 8 GB memory in a few minutes. I used all kinds of stagger sizes from 1024 to 16k, plot sizes from 2 GB to 5.5 TB using bipben's gpu plotter and tried uray's burst-miner and blago & dcct's miner. Using win7 and tried disabling prefetch/superfetch and various windows services without success.
hero member
Activity: 588
Merit: 500
Since this has been around for a while, I think it's important to ask... Is there any wear and tear done by this kind of mining, any loss of performance? Specially on SSD's...

Make sure your HDDs aren't parking often, this will kill them fast. If they do, add something like this to 'crontab -e':

Code:
*/2 * * * * echo `/bin/date` > /Volumes/hdd1/keepalive.txt
*/2 * * * * echo `/bin/date` > /Volumes/hdd2/keepalive.txt

Also, ensure plot files aren't fragmented and large enough stagger is used so they're read sequentially. This way tear should be minimal.

For SSDs the above doesn't matter much, reading during mining shouldn't put much stress on them.

p.s. Regarding storj: most ppl I talked to don't like the idea of storing files on their hdds (even encrypted ones). They're afraid of illegal content potentially being stored. Burst doesn't store any real data and many view this as an advantage.


I agree about the storage part,  I don't think burst needs to worry about becoming decentralized storage, the issue you mentioned as well as bandwidth come into play, also, not sure how they would work it with so many people having different connections and speeds. but yea... BURST doesn't need to be storage, it's the mining that's the innovation here.

it COULD however, be a decentralized marketplace, with escrow. That could already happen, and would be a great plan. IMO

Hold on, could we not enable "forging" again...but with a twist. Here's the idea.

You can upload photos/videos for the marketplace
.whenever an account is "forging" it is downloading photos, not all of them, but a few of them at least. The fee involved with a photo/video is calculated based on its size, and therefore hosting the photo is profitable, because part of the fee goes to you, the host/"forger". This way it is optional, but useful. Also, the photo only needsnto be in a couple of places, as the owner of the photo is responsible for making sure its always safe on his wallet.
newbie
Activity: 44
Merit: 0
I am wondering if mining on a 5400 rpm HDD has the same performance as a 7200 rpm hdd.If their capacity is the same.


It depends on how long it takes to read a set of scoops.  If it takes more than 30s or so, you may sometimes miss a short deadline.


Sequential read speed matters the most. My new 5400 rpm drive with 170 mb/s outperforms older 7200 rpm WDBlack which reads 120 mb/s. But they finish reading plots simultaneously because the latter has lower capacity.

Some 7200 rpm drives also tend to vibrate more and require extra measures to prevent resonance effect, so I'm not sure if they're better if you're building a rack.

This mostly matters for pools where you accumulate results before sending them to the pool as you can miss a block if one of HDDs is really slow.
sr. member
Activity: 462
Merit: 250
I am wondering if mining on a 5400 rpm HDD has the same performance as a 7200 rpm hdd.If their capacity is the same.


It depends on how long it takes to read a set of scoops.  If it takes more than 30s or so, you may sometimes miss a short deadline.
Jump to: