Topic: New demonstration CPU miner available - page 18. (Read 386150 times)

The current solution in git is sufficient: We re-use the CURL object in each miner thread, which implies we fully cache DNS entries according to their cache timeouts.

Yes, however, libcurl has its internal DNS cache set to expire once every 60 seconds. So I'd be saving 1440 DNS requests per day and per thread even after the changes that jgarzik recently made to the networking code are released. This will be an increase in performance of about 1.6% over the current code in the git repository (assumes one second to do a DNS look-up, which is what I experience at my machine).

Also the current release of the program does one DNS request for every time it does work since it does not re-use curl objects. My local DNS server blows and takes almost a full second to do a look-up, so you can imagine that this gets very costly, since the network code is synchronous with the crunching code. I was spending 10-20% of my time doing DNS requests instead of crunching hashes before making changes to the code to prevent this. This is the reason that I started looking at the code: I wanted to know why my processors were sitting idle for so much time instead of making hashes.

EDIT:

I think I just changed some of the files in the git repository:
-DNS caching now lasts indefinitely, until a connection attempt fails, at which point the cache is cleared and the look-up happens again.
-All threads use a single shared CURL object which is initialized only once.
-Yay and Boo results are now timestamped.

I've been testing this in slush's pool for the last few minutes and it's working fine so far, it's found about 5 shares.

Yes, this is better. But, it is necessary? DNS typically expires once every 8 hours, so if your DNS cache works properly, you're saving rountrip of 3 requests daily. Worthless.

So instead, perhaps we make the DNS cache not expire, except in cases where the connection fails. When a failure occurs, we assume that the IP address of the server might have changed, so we will clear the cache and do the DNS look-up again. I'll look into how to do this with CURL.

No please! No for my pool! Hardcoded IP makes system migration almost impossible or with significant drop in hashrate until all of 400 workers is fixed. I already migrated server to stronger machine and I'll do it again soon.

OK, that sounds like a good plan. I'll see what I can do to get that working. Maybe I can get an average time for the mining thread to process one getwork(), and the average time for a getwork() to resolve. I'll look into how the threads might be scheduled more effectively.

Also, it may be a good idea to set CURLOPT_DNS_CACHE_TIMEOUT to -1 or suggest that people use the IP address of the server they wish to connect to instead of the hostname. I used both, and the performance of the networking code improved significantly in both cases. However, this might cause a problem if the IP address of the server changes while crunching is happening. Maybe causing the cache to reset whenever a connection fails would be a good idea. Again, I'll look into doing these things.

I'll throw my suggestion in too.. any chance the miner could optionally time-stamp the successful results?

Maybe something like: "PROOF OF WORK RESULT: true (yay!!!) 18:24:32 02.10.2011"

Someone implemented a work queue a long time ago.  The reason that changed was never merged into upstream:  it basically guaranteed that you are always working on "old work," lagging a second or three.  Ideally, a background thread polls for work, carefully observing timings sufficient that it issues a 'getwork' JSON-RPC call, downloads and parses the JSON response just in time for a thread to request new work from the queue.

Just committed two changes to git, based on suggestions here:

• Re-use CURL object.  This means persistent HTTP/1.1 connections, if your pool server supports it!
• Use bswap_32(), if compiler does not provide intrinsic.  Useful for older OS's.

Well, the queue length I was testing was three. My processor can get through a getwork() in about 5 seconds per core, so I guess I was falling outside of that range for your server. Well, I guess this can't be solved in the way I was thinking. Maybe there's some other way to do it, I'll have to think through how I could do it without a queue and without taking a long time. Maybe having the miner threads call to the main thread to get the next work unit ready just before they crunch an existing one would be a good idea. I'll look into doing that.

Also, it looks like an entire new curl object is created every time that you want to get more work. It might be a good idea to use the existing object over again, since I'm sure that wastes some resources and time. All requests for more work have the same form anyway. I'll see if I can get that working as well, and then I'll submit my changes here. I'm not so sure how to git, so if anyone could link me a good tutorial on how to use it to make those patch files that seem so popular, I would be grateful.

How old are these calls? Miners need to have jobs as fresh as possible. Crunching too old jobs leads to rejecting by pool server. Usualy the ask rate is 5 seconds, this leads only to decent overhead.

So, I was looking at the processor meter in my machine while the miner was running, and noticed something strange: Every time a work-unit is completed, that processor sits idle for a few seconds.

I looked through the code to see what was causing this, and I found that the I/O for the getwork() calls is done inside the mining threads while the main thread sits idle and waits for the miners to exit.

To fix this, I implemented a blocking bounded queue and had the main thread constantly do getwork() commands while the mining threads crunch the resulting work. This seems to have removed the potentially long wait for a response form the RPC server, especially when doing pooled mining from a remote server.

However, something strange is occurring. When I set the queue up such that each mining thread has a small work queue of length 1, there is no problem and the system works as intended. However, when the queue length goes above one for each queue, all the results get a response to the effect that they are invalid. I have verified that the correct data is getting into the work structures, so I'm asking in here because I don't know much about how this RPC system works.

Is there some reason that the same thread can't do this queueing? For instance, some kind of an ID for the block that gets overwritten if a new getwork() request is made? I'm doing this while mining in slush's pool, so is that causing a problem?

Hi,

I'm trying to run the miner on my Ubuntu 8.04 (Hardy) system, with an Intel Pentium Dual CPU E2140 .

The  CFLAGS="-O3 -Wall -msse2" ./configure command seems to work fine, but then after the "make" command I get the following:

make  all-recursive
make[1]: Entering directory `/home/roldansu/Desktop/cpuminer-0.6.1'
Making all in compat
make[2]: Entering directory `/home/roldansu/Desktop/cpuminer-0.6.1/compat'
Making all in jansson
make[3]: Entering directory `/home/roldansu/Desktop/cpuminer-0.6.1/compat/jansson'
make[3]: Nothing to be done for `all'.
make[3]: Leaving directory `/home/roldansu/Desktop/cpuminer-0.6.1/compat/jansson'
make[3]: Entering directory `/home/roldansu/Desktop/cpuminer-0.6.1/compat'
make[3]: Nothing to be done for `all-am'.
make[3]: Leaving directory `/home/roldansu/Desktop/cpuminer-0.6.1/compat'
make[2]: Leaving directory `/home/roldansu/Desktop/cpuminer-0.6.1/compat'
make[2]: Entering directory `/home/roldansu/Desktop/cpuminer-0.6.1'
gcc  -O3 -Wall -msse2 -pthread  -o minerd cpu-miner.o sha256_generic.o sha256_4way.o sha256_via.o sha256_cryptopp.o util.o -lcurl -Wl,-Bsymbolic-functions -lgssapi_krb5 compat/jansson/libjansson.a -lpthread
sha256_via.o: In function `scanhash_via':
sha256_via.c:(.text+0x4c): undefined reference to `__builtin_bswap32'
sha256_via.c:(.text+0x70): undefined reference to `__builtin_bswap32'
sha256_via.c:(.text+0x120): undefined reference to `__builtin_bswap32'
sha256_via.c:(.text+0x12e): undefined reference to `__builtin_bswap32'
sha256_via.c:(.text+0x13c): undefined reference to `__builtin_bswap32'
sha256_via.o:sha256_via.c:(.text+0x14a): more undefined references to `__builtin_bswap32' follow
collect2: ld returned 1 exit status
make[2]: *** [minerd] Error 1
make[2]: Leaving directory `/home/roldansu/Desktop/cpuminer-0.6.1'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home/roldansu/Desktop/cpuminer-0.6.1'
make: *** [all] Error 2

My programing knowledge is almost zero, so I'd appreciate some advice on this.

Thanks

This optimization (__builtin_bswap32) is only available on gcc 4.3 or later, and it's a bit silly because you can get this optimization from via bswap_32() (which is smart enough to use the bswap opcode) on gcc 2.0 or later. AFAIK, as long as you have optimizations turned on, the generated code is identical. I appreciate your work here, I am using your miner on my servers to mine with their idle cpu-time -- the power is already bought, after all. I've attached a patch if you don't want to trust bswap_32() to be equivalent:

Code:

diff --git a/miner.h b/miner.h
index 539b5d6..eef4ee0 100644
--- a/miner.h
+++ b/miner.h
@@ -1,6 +1,7 @@
 #ifndef __MINER_H__
 #define __MINER_H__
 
+#include
 #include
 #include
 #include
@@ -24,7 +25,11 @@
 
 static inline uint32_t swab32(uint32_t v)
 {
+#if (__GNUC__ >= 4 && __GNUC_PATCHLEVEL__ >= 3)
        return __builtin_bswap32(v);
+#else
+       return bswap_32(v);
+#endif
 }
 
 static inline void swap256(void *dest_p, const void *src_p)

cpuminer version 0.6.1 released (see top of thread for URLs).

Changes:
- Fully validate "hash < target", rather than simply stopping our scan
  if the high 32 bits are 00000000.
- Add --retry-pause, to set length of pause time between failure retries
- Display proof-of-work hash and target, if -D (debug mode) enabled
- Fix max-nonce auto-adjustment to actually work.  This means if your
  scan takes longer than 5 seconds (--scantime), the miner will slowly
  reduce the number of hashes you work on, before fetching a new work unit.

SHA1: dca174e0e433cda072a1acd7884dbdc15fa4a53c  cpuminer-installer-0.6.1.zip
MD5: 675242118b337c6d2bfc23a13eb2dd82  cpuminer-installer-0.6.1.zip

cpuminer version 0.6 released (see top of thread for URLs).

Changes:
- Fetch new work unit, if scanhash takes longer than 5 seconds (--scantime)
- BeeCee1's sha256 4way optimizations (faster!)
- lfm's byte swap optimization (faster! improves via, cryptopp)
- Fix non-working short options -q, -r

SHA1: d4c21a3fc1c3f433771101cf1e71186df3a94032  cpuminer-installer-0.6.zip
MD5: 1936f4f71574643825f8d8d31e456a3a  cpuminer-installer-0.6.zip

add " --quiet >>miner.log 2>&1 & " onto your start command and it will run in background just fine

Hi :

I'm running minerd in a Debian environment.
Is there any way to run it as a daemon, with no terminal needed ?

Tks
Gustavo

You're right.  since it compiled with 4way support on a different machine I thought the configure script was making the choice.  On further reflection it seems more likely it is because the other machine is running in 64bit mode (which always has sse2).  -march=native didn't give me better results, I don't thing GCC recognizes my chip so uses more generic instructions, I still had to give -march=core2 for best performance.

Yeah, the configure script is not intended to "recognize your cpu."  If you are doing your own builds, you are expected to be able to supply the best compiler flags for your CPU through the CFLAGS environment variable.  This is the standard, recognized method for passing compiler flags to 95% of all configure scripts:


-march=native tells recent gcc versions to build programs tuned specificaly for the build machine's CPU (and may not work on other CPUs).  By default, the compiler tries to generate code that is best suited to a wide range of CPUs.