[1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool - page 593.

spiccioli

legendary

Activity: 1379

Merit: 1003

nec sine labore

Quote from: forrestv on January 22, 2013, 12:36:17 AM

I'm investigating the memory leak; one person experiencing it gave me SSH access and I'm waiting for it to occur. If anyone else wants to lend SSH access, that would be helpful too.

forrestv,

this is my public p2pool entry point at p2pool.soon.it:9332 after 10 days

Code:

Date Memory Usage/(B)
Wed Jan 23 2013 07:00:00 GMT+0100 (ora solare Europa occidentale) 576M
Tue Jan 22 2013 14:12:00 GMT+0100 (ora solare Europa occidentale) 578M
Mon Jan 21 2013 21:24:00 GMT+0100 (ora solare Europa occidentale) 578M
Mon Jan 21 2013 04:36:00 GMT+0100 (ora solare Europa occidentale) 570M
Sun Jan 20 2013 11:48:00 GMT+0100 (ora solare Europa occidentale) 570M
Sat Jan 19 2013 19:00:00 GMT+0100 (ora solare Europa occidentale) 570M
Sat Jan 19 2013 02:12:00 GMT+0100 (ora solare Europa occidentale) 570M
Fri Jan 18 2013 09:24:00 GMT+0100 (ora solare Europa occidentale) 570M
Thu Jan 17 2013 16:36:00 GMT+0100 (ora solare Europa occidentale) 570M
Wed Jan 16 2013 23:48:00 GMT+0100 (ora solare Europa occidentale) 570M
Wed Jan 16 2013 07:00:00 GMT+0100 (ora solare Europa occidentale) 580M
Tue Jan 15 2013 14:12:00 GMT+0100 (ora solare Europa occidentale) 518M
Mon Jan 14 2013 21:24:00 GMT+0100 (ora solare Europa occidentale) 518M
Mon Jan 14 2013 04:36:00 GMT+0100 (ora solare Europa occidentale) 511M
Sun Jan 13 2013 11:48:00 GMT+0100 (ora solare Europa occidentale) 416M
Sat Jan 12 2013 19:00:00 GMT+0100 (ora solare Europa occidentale) 335M
Sat Jan 12 2013 02:12:00 GMT+0100 (ora solare Europa occidentale) 275M
Fri Jan 11 2013 09:24:00 GMT+0100 (ora solare Europa occidentale) 276M
Thu Jan 10 2013 16:36:00 GMT+0100 (ora solare Europa occidentale) 178M
Wed Jan 09 2013 23:48:00 GMT+0100 (ora solare Europa occidentale) 0.00
Wed Jan 09 2013 07:00:00 GMT+0100 (ora solare Europa occidentale) 0.00

as you can see memory seems to have reached a maximum around 580 Mb.

I'm using python here but I've also used pypy in the past when I was experiencing high memory usage.

I'm going to restart it with pypy just to see if it makes any difference at all.

This is a fedora 16, 32 bit system with PAE and 6 GB of ram.

Regards.

spiccioli.

ps. memory usage does not work on freebsd (I've got a system running because I really like the zfs filesystem

)

zvs

legendary

Activity: 1680

Merit: 1000

https://web.archive.org/web/*/nogleg.com

Quote from: forrestv on January 22, 2013, 12:36:17 AM

Quote from: Rogue Star on January 22, 2013, 12:29:56 AM

Quote from: K1773R on January 18, 2013, 10:28:34 PM

Quote from: Rogue Star on January 18, 2013, 10:21:35 PM

obviously, but what are the effects on p2pool as a whole? does it report orphaned shares correctly in this scenario, are there other effects?

% of orphan increases @ stats?! yes it does "report" them correctly. a side effect would be the wasted traffic, altough this isnt much.

well they certainly aren't reported on the local graphs, otherwise it would show a much higher waste rate in the mining graphs, so that's one of the reasons I ask. i suppose it makes sense since you can't fully verify a share until it's been included in the sharechain. the other reason I ask is the variance we see when there are problems with the nodes/sharechain seem to imply that the orphan/dead rate perhaps isn't reported correctly.

another question I had was if sharechain 11 is a hardfork, why does p2pool still connect to clients on sharechain 9 after the fork?

i've updated from Ubuntu Server x64 12.04 LTS to 12.10 and I'm still seeing the memory leak and increased orphaned/dead as i approach a day of node uptime. The leak appears to start after about 12 hour, although it's not as pronounced as on 12.04 LTS.

Versions 10 and 11 were not hardforks. They triggered the upgrade notifications, but there was no switchover to cut version 9's off.

I'm investigating the memory leak; one person experiencing it gave me SSH access and I'm waiting for it to occur. If anyone else wants to lend SSH access, that would be helpful too.

well, I know if you start p2pool with tons of old share files, it'll start you out at like 500MB+ memory usage. it'll say something like 35000/35000 shares verified and it'll gradually start removing the old share files, but your memory usage will still stay at 500MB

forrestv

hero member

Activity: 516

Merit: 643

Quote from: Rogue Star on January 22, 2013, 12:29:56 AM

Quote from: K1773R on January 18, 2013, 10:28:34 PM

Quote from: Rogue Star on January 18, 2013, 10:21:35 PM

obviously, but what are the effects on p2pool as a whole? does it report orphaned shares correctly in this scenario, are there other effects?

% of orphan increases @ stats?! yes it does "report" them correctly. a side effect would be the wasted traffic, altough this isnt much.

well they certainly aren't reported on the local graphs, otherwise it would show a much higher waste rate in the mining graphs, so that's one of the reasons I ask. i suppose it makes sense since you can't fully verify a share until it's been included in the sharechain. the other reason I ask is the variance we see when there are problems with the nodes/sharechain seem to imply that the orphan/dead rate perhaps isn't reported correctly.

another question I had was if sharechain 11 is a hardfork, why does p2pool still connect to clients on sharechain 9 after the fork?

i've updated from Ubuntu Server x64 12.04 LTS to 12.10 and I'm still seeing the memory leak and increased orphaned/dead as i approach a day of node uptime. The leak appears to start after about 12 hour, although it's not as pronounced as on 12.04 LTS.

Versions 10 and 11 were not hardforks. They triggered the upgrade notifications, but there was no switchover to cut version 9's off.

I'm investigating the memory leak; one person experiencing it gave me SSH access and I'm waiting for it to occur. If anyone else wants to lend SSH access, that would be helpful too.

Rogue Star

member

Activity: 89

Merit: 10

Quote from: K1773R on January 18, 2013, 10:28:34 PM

Quote from: Rogue Star on January 18, 2013, 10:21:35 PM

obviously, but what are the effects on p2pool as a whole? does it report orphaned shares correctly in this scenario, are there other effects?

% of orphan increases @ stats?! yes it does "report" them correctly. a side effect would be the wasted traffic, altough this isnt much.

well they certainly aren't reported on the local graphs, otherwise it would show a much higher waste rate in the mining graphs, so that's one of the reasons I ask. i suppose it makes sense since you can't fully verify a share until it's been included in the sharechain. the other reason I ask is the variance we see when there are problems with the nodes/sharechain seem to imply that the orphan/dead rate perhaps isn't reported correctly.

another question I had was if sharechain 11 is a hardfork, why does p2pool still connect to clients on sharechain 9 after the fork?

i've updated from Ubuntu Server x64 12.04 LTS to 12.10 and I'm still seeing the memory leak and increased orphaned/dead as i approach a day of node uptime. The leak appears to start after about 12 hour, although it's not as pronounced as on 12.04 LTS.

zvs

legendary

Activity: 1680

Merit: 1000

https://web.archive.org/web/*/nogleg.com

Quote from: K1773R on January 19, 2013, 09:36:01 PM

Quote from: zvs on January 19, 2013, 09:15:51 PM

Quote from: Red Emerald on January 19, 2013, 07:59:26 PM

Quote from: kano on January 19, 2013, 07:31:09 PM

Quote from: Prattler on January 19, 2013, 07:33:44 AM

Quote from: zvs on January 18, 2013, 08:57:22 PM

p2pool's main problem is being penalized for including transactions.

P2pool miners are protecting the network against >50% attacks. That is a HUGE benefit for the bitcoin network.

Although the concept sounds good, since p2pool is only around 300-400GH/s - it really isn't protecting anything

... and if p2pool miners do as you suggest below, the larger pools, that include more transactions, are indeed better for BTC ...

Quote

Most p2pool users don't have the powerful servers and gigabit internet connections, so they should not include every possible 0-fee or 0.0005 BTC/kB transaction. It's not a huge problem if we leave that to the centralized pools.

And I thought forrest made it so that the transactions are really quickly between peers anyways, so are miners really still being penalized?

i'd think that it would be moved in a way that wouldnt involve that much more data, but you can see a clear difference in orphans when you set your blocksize to 1000 vs 250000. though i'd go for something like 5000 so it can cherry pick the occasional huge fees.

anyway, re: memory leak, check this bad boy out:

http://5.9.157.150:9332/static/graphs.html?Month

maybe it has something to do with DOA shares?

is your node public? interested in stats.

i put it back up about 4-5 hours ago, not mining on it again yet, though.. just for a relay... it's @ http://nogleg.com:9332

K1773R

legendary

Activity: 1792

Merit: 1008

/dev/null

Quote from: zvs on January 19, 2013, 09:15:51 PM

Quote from: Red Emerald on January 19, 2013, 07:59:26 PM

Quote from: kano on January 19, 2013, 07:31:09 PM

Quote from: Prattler on January 19, 2013, 07:33:44 AM

Quote from: zvs on January 18, 2013, 08:57:22 PM

p2pool's main problem is being penalized for including transactions.

P2pool miners are protecting the network against >50% attacks. That is a HUGE benefit for the bitcoin network.

Although the concept sounds good, since p2pool is only around 300-400GH/s - it really isn't protecting anything

... and if p2pool miners do as you suggest below, the larger pools, that include more transactions, are indeed better for BTC ...

Quote

Most p2pool users don't have the powerful servers and gigabit internet connections, so they should not include every possible 0-fee or 0.0005 BTC/kB transaction. It's not a huge problem if we leave that to the centralized pools.

And I thought forrest made it so that the transactions are really quickly between peers anyways, so are miners really still being penalized?

i'd think that it would be moved in a way that wouldnt involve that much more data, but you can see a clear difference in orphans when you set your blocksize to 1000 vs 250000. though i'd go for something like 5000 so it can cherry pick the occasional huge fees.

anyway, re: memory leak, check this bad boy out:

http://5.9.157.150:9332/static/graphs.html?Month

maybe it has something to do with DOA shares?

is your node public? interested in stats.

zvs

legendary

Activity: 1680

Merit: 1000

https://web.archive.org/web/*/nogleg.com

Quote from: Red Emerald on January 19, 2013, 07:59:26 PM

Quote from: kano on January 19, 2013, 07:31:09 PM

Quote from: Prattler on January 19, 2013, 07:33:44 AM

Quote from: zvs on January 18, 2013, 08:57:22 PM

p2pool's main problem is being penalized for including transactions.

P2pool miners are protecting the network against >50% attacks. That is a HUGE benefit for the bitcoin network.

Although the concept sounds good, since p2pool is only around 300-400GH/s - it really isn't protecting anything

... and if p2pool miners do as you suggest below, the larger pools, that include more transactions, are indeed better for BTC ...

Quote

Most p2pool users don't have the powerful servers and gigabit internet connections, so they should not include every possible 0-fee or 0.0005 BTC/kB transaction. It's not a huge problem if we leave that to the centralized pools.

And I thought forrest made it so that the transactions are really quickly between peers anyways, so are miners really still being penalized?

i'd think that it would be moved in a way that wouldnt involve that much more data, but you can see a clear difference in orphans when you set your blocksize to 1000 vs 250000. though i'd go for something like 5000 so it can cherry pick the occasional huge fees.

anyway, re: memory leak, check this bad boy out:

http://5.9.157.150:9332/static/graphs.html?Month

maybe it has something to do with DOA shares?

Red Emerald

hero member

Activity: 742

Merit: 500

Quote from: kano on January 19, 2013, 07:31:09 PM

Quote from: Prattler on January 19, 2013, 07:33:44 AM

Quote from: zvs on January 18, 2013, 08:57:22 PM

p2pool's main problem is being penalized for including transactions.

P2pool miners are protecting the network against >50% attacks. That is a HUGE benefit for the bitcoin network.

Although the concept sounds good, since p2pool is only around 300-400GH/s - it really isn't protecting anything

... and if p2pool miners do as you suggest below, the larger pools, that include more transactions, are indeed better for BTC ...

Quote

Most p2pool users don't have the powerful servers and gigabit internet connections, so they should not include every possible 0-fee or 0.0005 BTC/kB transaction. It's not a huge problem if we leave that to the centralized pools.

And I thought forrest made it so that the transactions are really quickly between peers anyways, so are miners really still being penalized?

kano

legendary

Activity: 4592

Merit: 1851

Linux since 1997 RedHat 4

Quote from: Prattler on January 19, 2013, 07:33:44 AM

Quote from: zvs on January 18, 2013, 08:57:22 PM

p2pool's main problem is being penalized for including transactions.

P2pool miners are protecting the network against >50% attacks. That is a HUGE benefit for the bitcoin network.

Although the concept sounds good, since p2pool is only around 300-400GH/s - it really isn't protecting anything

... and if p2pool miners do as you suggest below, the larger pools, that include more transactions, are indeed better for BTC ...

Quote

Most p2pool users don't have the powerful servers and gigabit internet connections, so they should not include every possible 0-fee or 0.0005 BTC/kB transaction. It's not a huge problem if we leave that to the centralized pools.

Prattler

full member

Activity: 192

Merit: 100

Quote from: zvs on January 18, 2013, 08:57:22 PM

p2pool's main problem is being penalized for including transactions.

P2pool miners are protecting the network against >50% attacks. That is a HUGE benefit for the bitcoin network.

Most p2pool users don't have the powerful servers and gigabit internet connections, so they should not include every possible 0-fee or 0.0005 BTC/kB transaction. It's not a huge problem if we leave that to the centralized pools.

Quote from: zvs on January 18, 2013, 08:57:22 PM

the bonus for solving a block should be the entirety of the transaction fees instead of what it is now... simple fix

Very good idea!

mdude77

legendary

Activity: 1540

Merit: 1001

Quote from: organofcorti on January 18, 2013, 10:16:29 PM

Quote from: mdude77 on January 18, 2013, 10:07:32 PM

I am. It doesn't make sense to use p2pool w/o supporting the author. And the 0.5% "fee" on bitminter is so I have access to API stats. In my mind, bitminter comes out ahead.

M

You could have chosen a 0.5% donation to forrestv. The fee comparison is not valid.

and I could go zero on both.

in my mind, the minimum for both is 1% for p2pool, and 0.5% for bitminer.

M

K1773R

legendary

Activity: 1792

Merit: 1008

/dev/null

Quote from: Rogue Star on January 18, 2013, 10:21:35 PM

Quote from: K1773R on January 18, 2013, 09:51:55 PM

if ur bitcoind isnt synced u get 100% orphan Wink

obviously, but what are the effects on p2pool as a whole? does it report orphaned shares correctly in this scenario, are there other effects?

% of orphan increases @ stats?! yes it does "report" them correctly. a side effect would be the wasted traffic, altough this isnt much.

Rogue Star

member

Activity: 89

Merit: 10

Quote from: K1773R on January 18, 2013, 09:51:55 PM

if ur bitcoind isnt synced u get 100% orphan Wink

obviously, but what are the effects on p2pool as a whole? does it report orphaned shares correctly in this scenario, are there other effects?

organofcorti

donator

Activity: 2058

Merit: 1007

Poor impulse control.

Quote from: mdude77 on January 18, 2013, 10:07:32 PM

I am. It doesn't make sense to use p2pool w/o supporting the author. And the 0.5% "fee" on bitminter is so I have access to API stats. In my mind, bitminter comes out ahead.

M

You could have chosen a 0.5% donation to forrestv. The fee comparison is not valid.

mdude77

legendary

Activity: 1540

Merit: 1001

Quote from: organofcorti on January 18, 2013, 10:06:08 PM

Quote from: mdude77 on January 18, 2013, 09:54:04 PM

Quote from: organofcorti on January 18, 2013, 09:33:14 PM

Quote from: mdude77 on January 18, 2013, 09:30:37 PM

Quote from: organofcorti on January 18, 2013, 09:14:13 PM

Quote from: mdude77 on January 18, 2013, 09:05:41 PM

I'm been using p2pool for a while, and while there are bad streaks, there's never been one this bad.

Yes, there have:
http://organofcorti.blogspot.com.au/2012/06/51-p2pool-bad-luck-or-flawed.html

Quote from: mdude77 on January 18, 2013, 09:05:41 PM

I was paying 1% fee on p2pool....

No you weren't. p2Pool does not have a fee.

It's optional, and called "1% donation to author".

M

I apologise, I misunderstood. However, comparing an optional 1% donation to an optional 0.5% donation makes not sense. Can you not select 0.5% as the optional donation to the p2Pool?

Of course I can. But why?

M

You're comparing the fee you pay on Bitminter compared to what you paid on p2Pool:

Quote from: mdude77 on January 18, 2013, 09:05:41 PM

I was paying 1% fee on p2pool, now I'm paying 0.5% on bitminter, I get less variance, and I still get namecoins.

You might not have meant it that way, but it reads as if you're comparing the two fees.

I am. It doesn't make sense to use p2pool w/o supporting the author. And the 0.5% "fee" on bitminter is so I have access to API stats. In my mind, bitminter comes out ahead.

M

organofcorti

donator

Activity: 2058

Merit: 1007

Poor impulse control.

Quote from: mdude77 on January 18, 2013, 09:54:04 PM

Quote from: organofcorti on January 18, 2013, 09:33:14 PM

Quote from: mdude77 on January 18, 2013, 09:30:37 PM

Quote from: organofcorti on January 18, 2013, 09:14:13 PM

Quote from: mdude77 on January 18, 2013, 09:05:41 PM

I'm been using p2pool for a while, and while there are bad streaks, there's never been one this bad.

Yes, there have:
http://organofcorti.blogspot.com.au/2012/06/51-p2pool-bad-luck-or-flawed.html

Quote from: mdude77 on January 18, 2013, 09:05:41 PM

I was paying 1% fee on p2pool....

No you weren't. p2Pool does not have a fee.

It's optional, and called "1% donation to author".

M

I apologise, I misunderstood. However, comparing an optional 1% donation to an optional 0.5% donation makes not sense. Can you not select 0.5% as the optional donation to the p2Pool?

Of course I can. But why?

M

You're comparing the fee you pay on Bitminter compared to what you paid on p2Pool:

Quote from: mdude77 on January 18, 2013, 09:05:41 PM

I was paying 1% fee on p2pool, now I'm paying 0.5% on bitminter, I get less variance, and I still get namecoins.

You might not have meant it that way, but it reads as if you're comparing the two fees.

mdude77

legendary

Activity: 1540

Merit: 1001

Quote from: organofcorti on January 18, 2013, 09:33:14 PM

Quote from: mdude77 on January 18, 2013, 09:30:37 PM

Quote from: organofcorti on January 18, 2013, 09:14:13 PM

Quote from: mdude77 on January 18, 2013, 09:05:41 PM

I'm been using p2pool for a while, and while there are bad streaks, there's never been one this bad.

Yes, there have:
http://organofcorti.blogspot.com.au/2012/06/51-p2pool-bad-luck-or-flawed.html

Quote from: mdude77 on January 18, 2013, 09:05:41 PM

I was paying 1% fee on p2pool....

No you weren't. p2Pool does not have a fee.

It's optional, and called "1% donation to author".

M

I apologise, I misunderstood. However, comparing an optional 1% donation to an optional 0.5% donation makes not sense. Can you not select 0.5% as the optional donation to the p2Pool?

Of course I can. But why?

M

K1773R

legendary

Activity: 1792

Merit: 1008

/dev/null

Quote from: Rogue Star on January 18, 2013, 09:42:10 PM

While I appreciate the suggestions, I do not want to "degrade" the network by being "greedy". I prefer to promote network health while being passively greedy. You might disagree with saying it degrades the network, but if I were solo mining I'd be able to include all transactions and allow port forwarding without any meaningful cost. You can also say I should be greedy, but again if I were solo mining the cost of including transactions is negligible. Whatever the problem is, I'm pretty sure it's p2pool.

I've also noticed in the network graphs that there is no incoming data from peers in the places where the data is "stuck", so I assume that would explain the very high orphan/dead rates, but perhaps not the above average rates I'm seeing otherwise. I do see outgoing data at those times, which I would think would be somewhat harmful to the network is things are out of sync.

I'm not entirely convinced p2pool is having a bad streak, but I do think the p2pool network is not as healthy as it may appear. One thing I don't think I've seen a good explanation about is the effect of share and/or block chains not being entirely in sync on the network health. I'm sure most people do a good job keeping them in sync, but what of the minority that are not?

if ur bitcoind isnt synced u get 100% orphan Wink

Rogue Star

member

Activity: 89

Merit: 10

While I appreciate the suggestions, I do not want to "degrade" the network by being "greedy". I prefer to promote network health while being passively greedy. You might disagree with saying it degrades the network, but if I were solo mining I'd be able to include all transactions and allow port forwarding without any meaningful cost. You can also say I should be greedy, but again if I were solo mining the cost of including transactions is negligible. Whatever the problem is, I'm pretty sure it's p2pool.

I've also noticed in the network graphs that there is no incoming data from peers in the places where the data is "stuck", so I assume that would explain the very high orphan/dead rates, but perhaps not the above average rates I'm seeing otherwise. I do see outgoing data at those times, which I would think would be somewhat harmful to the network is things are out of sync.

I'm not entirely convinced p2pool is having a bad streak, but I do think the p2pool network is not as healthy as it may appear. One thing I don't think I've seen a good explanation about is the effect of share and/or block chains not being entirely in sync on the network health. I'm sure most people do a good job keeping them in sync, but what of the minority that are not?

organofcorti

donator

Activity: 2058

Merit: 1007

Poor impulse control.

Quote from: mdude77 on January 18, 2013, 09:30:37 PM

Quote from: organofcorti on January 18, 2013, 09:14:13 PM

Quote from: mdude77 on January 18, 2013, 09:05:41 PM

I'm been using p2pool for a while, and while there are bad streaks, there's never been one this bad.

Yes, there have:
http://organofcorti.blogspot.com.au/2012/06/51-p2pool-bad-luck-or-flawed.html

Quote from: mdude77 on January 18, 2013, 09:05:41 PM

I was paying 1% fee on p2pool....

No you weren't. p2Pool does not have a fee.

It's optional, and called "1% donation to author".

M

I apologise, I misunderstood. However, comparing an optional 1% donation to an optional 0.5% donation makes not sense. Can you not select 0.5% as the optional donation to the p2Pool?

Topic: [1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool - page 593. (Read 2591920 times)