Topic: [ANN]: cpuminer-opt v3.8.8.1, open source optimized multi-algo CPU miner - page 3. (Read 444096 times)

I think it's time to reflect on the cpuminer-opt project.

TLDR:

Not going to compete with already optimized new CPU algos like RandomX.
Next big phase is AVX512.
I know I'm flogging a dead horse.
Everything depends on my motivation.

The long version:

Mining is in the dumps in general, exchange rates way down. Competition from ASICs
& FPGAs is making it worse for GPUs and even worse for CPUs.

One exception is the CPU-friendly algos like yespower and randomx, and Ryzen CPUs.

I started this project in better times with the purpose of importing existing X11
functions that were optimized for AES & SSE2. That was mostly cut & paste.

Some of those functions were easilly promoted to AVX2 like cubehash and lyra2.

I then started to optimize other functions to hash in parallel similar to how GPUs hash.
SIMD has some limitations so not all functions can be parallelized efficiently.

I also attempted to optimize CPU algos in various ways with very limited success.
Many CPU algos can't be vectorized at all or can't be vectorized using SIMD.
Most ot of those than can have already been optimized.

So that brings things up to date. I don't anticipate there is much more performance I can
squeeze out of the code. cpuminer-opt 3.9.10 is probably as fast as it will get with AVX2.

The next big phase is AVX512. All of the parallel AVX2 can be promoted and some of the linear
AVX2 can also be promoted to AVX512. A rough estimate is a 1.5x improvement in algos
with a significant portion of AVX2.

There are a few factor that limit the gains: only the portions of algos that currently use AVX2 will
see improvement, memory usage is doubled possibly creating bottlenecks, AVX512 code runs
at a lower clock speed than AVX2, and AVX512 has extra overhead for some operations.

Starting serious AVX512 development requires a CPU with AVX512. With the release of Intel's
Cascade Lake X CPUs and the big price reductions, combined with the failure of my core2-quad
system, it may be time to buy a new Cascade Lake X.

The loss of my core2 means I don't have a CPU to test SSE2 natively. I don't expect that to cause
any problems. SSE2 code will be very stable going forward and I can still test the SSE2  build.

Why am I still doing this when these algos are even dead from a GPU perspective?

One reason is no one else is doing it. For many algos the existing CPU miners are not optimized and for
others CPU miners don't exist.

Another reason is for fun. I'm retired and this is a hobby. I have a lot of experience in development and
support but not on Intel CPUs and not using C and not using Linux. So this has been a big learning
experience. I've made a lot of mistakes, fixed those I've discovered but there are probably many more.

In addition to getting used to the environment I also learned about SIMD programming (linear and parallel)
using intrinsic functions and, more recently, using ASM.

The learning aspect is very important because it's exercise for my aging brain. Doing hex arithmetic
and permuting bytes in my head is quite a workout.

As usual I make no promisses. Development of AVX512 will depend on my level of motivation which
is influenced by many factors.

cpuminer-opt-3.9.9

https://github.com/JayDDee/cpuminer-opt/releases

Added power2b algo for MicroBitcoin.

Added generic yespower-b2b (yespower + blake2b) algo to be used with
the parameters introduced in v3.9.7 for yespower & yescrypt.

Display additional info when a share is rejected.

Some low level enhancements and minor tweaking of log output.

Additional notes:

yespower-b2b is yespower with sha256 replaced with blake2b. There is no default parameter
configuration when using yespower-b2b, the N and R parameters and the personalization key,
if used, must be specified. The parameters work in the same way as yespower (sha256) and
yescrypt.  power2b is just yespower-b2b with the parameters preset for power2b.

Before this becomes a FAQ...

Power2b algo now used by MicroBitcoin is incompatible with current yespower code
and can't be mined with cpuminer-opt at this time.

A new version is required. I'm working on it.

cpuminer-opt-3.9.8.1

Summary log report will be generated on stratum diff change or after 5 minutes,
whichever comes first, to prevent incorrect data in the report.

Removed phi2-lux alias introduced in v3.9.8 due to Luxcoin's planned fork
to a new algo. The new Luxcoin algo is not supported by cpuminer-opt.
Until the fork Luxcoin can be mined using phi2 algo.

--hide-diff option is deprecated and has no effect. It will be removed in a
future release.

Edit: TTF estimates and hash rates for algos like x16r that have variable hash rates are
innacurate. No solution is apparent.

All the VS files are garbage, the project needs to be rebuilt from scratch.
Even so the code probably wouldn't compile due to incompatibilities
introduced over time.

It would be a bigger task than trying to compile on Apple Mac which a few have tried and
given up.

The process of using a Linux VM to cross compile using mingw is a lot of work to setup but
once done it's trivial to compile new versions. It's essentially 3 commands now to build the
binary release package.

There is a Microsoft Visual Studio .sln file. Is it compiling with MVS?

Randomx is under heavy development. I'm waiting until the dust settles to look at it in any detail.

Do you support randomx or will you support it?

cpuminer-opt-3.9.8

https://github.com/JayDDee/cpuminer-opt/releases/tag/v3.9.8

Changes to log output to provide data more relevant to actual mining performance.
phi2 can now handle pools with a mix of coins that use and don't use roots.
phi2-lux added as an alias for phi2 as they are identical except for roots.
Add x16rv2 algo for Ravencoin fork.

Detailed notes...

Please read the folllowing before reporting log questions.

The main motivation for the log changes was to focus on data that the pool uses to calculate earnings instead
of the traditional hash rate displayed by most miners. That hash rate is calculated by the miner counting hash
iterations over time.

Pools use the rate of share submission over time and the stratum difficulty to determine a user's share of a block.

As a result more focus has been placed on these data with the addition of detailed timing data in the 5 minute
summary report.

The diff adjusted average share submission rate is converted to a share equivalent hash rate and displayed in the
summary report. The miner reference hash rate is also displayed there for comparison and monitoring of CPU performance.
The miner reference hash rate is also displayed in the TTF calculation of the block report as it is used to
calculate the TTF for a block and share. The share equivalent hash rate should better match the pool's reported hash rate.
Although most pools use a 5 minute window they are not synchronized with the miner and will sometimes differ.
Edit: The summary data will be inacurrate if the stratum difficulty changed during the period.

A share's difficulty is also not used by the pool, unless it solves a block, and is no longer displayed on the first line of the
share report. The display of share difficulty and share ratio is FYI only due to it's lack if significance and is suppressed
if --quiet option is used.

The first line of the share report focusses on time, displaying the time since the last share and the network latency,
the time between submitting the share and receiving the reply from the pool. Timing information is also prominent
in the summary report.

Some compromises were made to avoid adding too much overhead that could reduce mining performance. For example the 5
minute summary is not exactly 5 minutes. Also, at startup block reports will show some zero values until a baseline
reference hash rate has been calculated. This can take a few minutes.

Everything else should be intuitive or easilly deduced if you've mined before. If you still can't figure it out please ask.

Keep up the good work, looking forward to test the new version once it is ready!

I'm considering more changes to the share display to address 2 specific issues.

1. Share difficulty is not used by pools except when it solves a block. All valid shares are
otherwise considered equal regardless of difficulty. This makes the sharediff and share ratio
(block %) irrelevent. I am proposing eliminating the  reporting of share ratio and removing
the colour ranking from the share diff. The share diff will still be reported without any
highlighting for high diff shares.

This creates a bit of a dilema for calculating the effective share hash rate as it uses the
sharediff as well as time while the pool uses only time. To match the pool's calculations
I would need to know the expected share rate, which is not available to the miner AFAIK.
The existing calculation using share diff seems to track the pool's reported hashrate fairly
well in the summary report so until I find a better way I won't change it.

2. Job ID reporting was introduced due to a specific issue with persistent stale shares in
one pool affecting a couple of algos. I've debugged down to the stratum/curl level and
it appears the pool is sending stale jobs. There has been little interest from the pool admin
in pursuing this issue but all indications are it's a pool issue.

The only other benefit to job id reporting is to confirm expected stale rejects where it shows
a new job received between the submission of a share and it's acceptance by the pool.
I don't think this is enough to justify the added verbosity in the output. I am proposing to remove
all reporting of job ids.

I am aslo planning on removing the timestamp on all lines of a multiline report. Only the first
line will have a timestamp.

I may also make some changes to the share rate reporting to place more emphasis on this statistic
as it has a direct effect on mining performance.
Comments?

My take on RandomX, cryptonight, Monero, etc.

As previously reported I have stopped trying to support the high level of algo development
by Monero and the variants of cryptonight. There exist excellent open source miners already
for this family of algos that are being actively developped. They are very well written with
little to no potential for further optimization.

In other words I have nothing to contribute. Trying to keep up would be a lot of work for no
benefit beyond the convenience of the additional algos in a single miner. Many Monero
miners are dedicated and don't usually mine other algos so the extra 100 algos would be just
bloat to them.

When and if development stabilizes and the dust settles I may consider re-importing
Cryptonight and RandomX and resume support.

I make no promises and there is no timeline.

Hi Joblo ...

may be you interest to add CNX-v1 algo to CPUminer
https://bitcointalksearch.org/topic/ann-galaxia-project-cnx-v1-algorithm-5172572
 thanks

It's a coin issue, there's nothing for me to fix.

You need the correct parameter values, ask the coin's devs.

Edit: Actually the real problem is too funny. The litb miner screwed up the key length:


The string is actually 75 chars but the length is hardcoded at 73. This means the key is actually the first 73 chars of the string:

Copy & paste the entire key including the quotes.

cpuminer-opt-3.9.7 makes changes to command line options.

-R is no longer used as a shortcut for retry-pause, users must use the long option.

New options have been added to facilitate mining algos based on yescrypt or
yespower with different parameters:


These parameters can be used to mine existing algos or new algos.
Existing algos may still be mined using their original unique name without extra parameters.

Any new algos based on yescrypt or yespower which only differ in parameters can be mined immediately
without requiring a new release of cpuminer-opt by using the generic algo name and specifying the parameters.

For example the recent cpupower algo can be mined using the following parameters:


Edit: Be carefull with the key, it's very long in this case and contains spaces and special characters, the quotes are necessary.

Parameter values are determined by the developpers of the coin and/or algo and should be listed in the coin's specifications
or mining instructions. Cpuminer has no knowledge of the parameter values except for algos previously
defined with a customized name.

Only the N parameter is supported for scrypt, R is harcoded with 1.  -a scrypt -N n has the same effect as -a scrypt:n.

-R option is deprecated

Plans have accelerated. There will be no grace period which would delay new features.
In the next release -R will have a totally different meaning, there will be no warning message.

I Initially declined to include cpupower algo in cpuminer-opt due to a number of  concerns.

Algo bloat: the number of supported algos is now around 100, that's 100 different
options for the -a argument. Many are almost identical with only parameter differences.

Single coin algos: I want to avoid algos that are tied to one specific coin and unlikely to be
used by any other coin in the future.

Algo branding: I prefer generic algo names instead of names that are closely associated with
a single coin.

Planned obsolescence: Coins that have plans to change algos are more work to support.

Unfortunately cpuchain/cpupower hits on all those points. The devs have expressed a desire
to maintain a tight coupling between the coin and algo, objected to a generic and more
descriptive algo name, and have plans to change algos.

All these factors led me to conclude the algo did not have much of a future and would likely
become obsolete.

Given the level of control the devs wanted over the algo I suggested it would be better to have
a dedicated cpuchain miner where the devs would have total control and freedom over their
algo of choice.

The existing cpuchain miner is forked from cpuminer-opt and can easilly import any missing
optimizations.

I'm looking at ways to simplify support for multiple algo variants to reduce the bloat and provide
some forward compatibility with new variants that don't require code changes.

This is a heads up for a proposed user interface change to cpuminer-opt.

I am intending in the next release to remove -R as a shortcut for the --retry-pause option:


I have plans to use -R for something else, --retry-pause will still be avaiable.
Retry is a rarely used option as most users prefer to exit and failover to a backup
pool when a connection is lost.

Until -R is repurposed its use will produce a warning message.

https://github.com/JayDDee/cpuminer-opt/issues/206