Topic: CKPOOL - Open source pool/proxy/passthrough/redirector/library in c for Linux - page 20. (Read 124153 times)

What's your ifconfig look like? You need to bind it to whatever the interface is that looks out to the world (probably a 192.x address).

I run ckpool -A and if the serverurl (in ckpool.conf) is set to anything other than 127.0.0.1:3333 than I get this error: [2014-10-14 00:13:58] Failed to bind socket for ckpool.org:3333 <- or any domain name or IP address
[2014-10-14 00:13:58] Connector failed to bind to socket, retrying in 5s

If I use 127.0.0.1:3333 then ckpool runs but I cannot connect any miner to it.  My miner connects to solo.ckpool.org:3333 just fine.  My firewall is open for port 3333.

I also have ensured that I can write to my socket directory as a user (not root) in /tmp.  I have also tried -s /run to change the socket directory to no avail.

Any thoughts?  Thanks again.

To be clear, merge mining coins which were just invented to monetise yet another coin that's stuffed into the bitcoin blockchain is a complete waste in my opinion. Something that uses the bitcoin blockchain but doesn't directly add value to the bitcoin economy is just free-riding. Extensions to bitcoin itself that depend on side chains are a valid potential use though (in my opinion), but none have matured at this stage. Please don't try and convince me otherwise, the noise is best left to other threads where I can more meaningfully ignore it.

The author said this:


Personally, I had no trouble getting it to work.

Does this work for altcoins?

Thanks for the awesome software - I was able to compile and configure a private pool in about five minutes! 

Hi, read the FAQ linked in the top post. This is software, not a pool, so the payscheme is not fixed in the software design and entirely up to the pool. The ckdb code has a pplns calculation in it, but that doesn't mean the pool that adopts ckpool has to use pplns at all. Code to perform payments is not anywhere in the included code. Furthermore, if you are using ckproxy and not ckpool, the payment part is entirely dependent on your upstream pool and the proxy has zero effect on it. If you're asking specifically about the current pools that are running ckpool, hashmine.io and ckpool/kano.is are using pplns for payment, while solo.ckpool.org is using entire block reward payments.

Payout schemes are not discussed for ckpool.  Is it just solo or are others possible like PPLNS?  I have read everything i could find on ckpool and i do not belive this was ever explicitly stated.  Hopefully, this will be my last question for awhile.  Thanks!

Server URL is only when run in ckpool mode. The proxy entry is when you are running it in proxy mode. If you are connecting clients to the proxy, you can only have one upstream connection to the pool with one login only. If you allowed multiple logins it would end up just having multiple connections to the upstream pool so the proxy wouldn't actually be achieving anything. The -P option ONLY works when the upstream pool is a ckpool instance as the manual explicitly states and nothing else and does not decrease bandwidth, only the number of socket connections.

Simplest p2pool proxy entry would be just the proxy entry and nothing else. eg:


EDIT: I've been thinking of ways to allow multiple clients attach to the upstream pool with their username but it's a very invasive change to the code since it is designed around the concept of only one upstream server at any one time. Unless multiple clients are connected with the same username attach they're not going to benefit from the proxy being there, but this change will likely eventually make its way into the code, along with ways to open up extra connections once the proxy is "full" of clients.

First off, thanks for ckpool - it really sounds like a great product and I am trying to learn more.  I have a p2pool server and I am trying to get a ckpool proxy setup in front of it since it sounds like this will reduce load on my server (provided I have more users).  First, with ckpool -A -p running will all downstream users continue to maintain their individual presence on the p2pool (like it is described to work with -P)?  Within ckpool.conf there is a proxy url and a server url that need to be set.  Running this on my p2pool server the proxy url seems to want to be set to my p2pool server with port 9332.  That what is server URL?  The IP and port that outside clients would connect to, to use the proxy before hitting the p2pool server?  I have tried multiple configurations and cannot get it to work.  Perhaps, there is a sample .conf file out there already for p2pool?  Thanks for all of the help.

I've added logging and statistics internally to ckpool on a combined per-worker basis, storing stats for each worker in the logs/workers directory.

I've also added the ability to set a user chosen minimum worker difficulty, though this will require some ckdb and front-end changes to allow this to be set.

In addition, I've implemented support for the stratum suggest_difficulty extension and updated the docs to reflect this. I've modified the cgminer git master to support this new extension with the --suggest-diff command.

I've substantially updated the ckproxy code to work much better in concert with p2pool, and to increase the maximum speed of clients that are connected via the proxy when connected to a non-ckpool. On most pools, you will now be able to connect up to 256 clients to the proxy (down from what was 64k) but the maximum hashrate of each connected miner has been pushed up from 5TH to 1.2PH.

Added support for using ckproxy to p2pool and large coinbase pools. Be aware that most non-ckpool based pools (p2pool included) have work limitations per connection when proxied as discussed in the ckproxy post.

very interesting concept for multi pools to trust each other... this will change the pooled mining landscape. Bitcoin mining is on the change again, fascinating future ahead for the mining scene.

@conman & kanoi; Congrats on the release and thank you for sharing your code with the community.

Cheers,
QG

We are the owners of hashmine.io and commissioned the development of ckpool.

We've been working with Con & Kano on this for a few months now, and I must say it's a pleasure to see the masters at work. In a sense, the work was a natural progression of the solo-mining/gbt work on cgminer we commissioned earlier this year.

I started on basic visualization/GUI layer here: http://cpui.hashmine.io/ - changes to that sync with Kano's web implementation @ http://ice.kano.is/
Current focus is on browser compatibility, making it work and look serviceable on all devices. Priority is bugfixes and basic user management workflows, will look @ i18n and l10n after that.
This current UI is mainly a test for the REST API I'm building on Kano's ckdb work and testing the servicing of historical reporting. Will add more visualizations & reporting in the coming days.
Still very much in debug mode as daily changes are coming in, but will try to keep up with Kano & Con as time goes on.

hashmine.io takes a flat rate 1.5% fee, of which 0.5% goes directly to ckpool development (controlled by Con, and issued @ coinbase at time of blocksolve). Newly minted coins go directly into cold storage and I process them manually when the blocks mature. A payout history is available @ the above-mentioned UI. Will do this until I've worked through the security aspects of automation.

There's still much to do on this technical path (hardware, software, architecture, distribution, etc.) and this project will probably always be on-going. Once things calm down a bit we'll go into the unique-selling points of hashmine and the direction we'd like to take with it.

Will update this post with progress as I push changes.

Discussion/Future direction/Feature requests

At this time the development of the code pool code has been in an extensive testing, debugging and consolidation, having been completed months ago and has had a great deal of real world testing, minting many blocks along the way. The ckdb code is now at completion to the point where public pools can run the code, which is why we're finally able to announce this project. Future features are definitely planned once this consolidation phase is over, and bug reports/adoption/feature requests/code pushes are all to be expected.

The next major feature we wish to work on is to develop the node code to allow multiple trusted nodes in different locations share data, but this is primarily work to be done at the database level.

In the future I wish to develop the concept of sibling pools which will allow collaboration to minimise variance for miners while sharing rewards. The idea is to expand beyond the need for trusted nodes to be able to link up with other pools that can prove to each other the work they are performing towards a common reward. In essence it will involve proof-of-work much like miners do to the pool itself but with auditing of each other's work. The details of the protocol are not yet finalised and is still some way off.

FAQs

Q: I get permission errors on the sockets?
A: You cannot talk via unix sockets using sudo. Make sure your users have write permissions to the directory where your sockets go. In particular, apache/httpd cannot use /tmp so you will have to choose a different directory such as /opt and give write permission to the user/group that you run ckpool/ckdb as.

Q: System requirements?
A: This depends entirely on the mode of running ckpool and the number of clients you wish to connect to it.
In proxy passthrough mode even a single CPU 32MB ram VPS is enough and requires virtually no disk space.
Standalone proxy mode requires more ram and CPU.
Running a standalone pool without ckdb requires only a few hundred megabytes of ram and will run on about 256MB of ram and at least one virtual CPU if the bitcoind is run on a separate machine (not recommended but doable).
However bitcoind itself usually requires a minimum of 2GB of ram with the wallet disabled and almost 30GB of HD space.
So a recommended setup for a minimal pool with bitcoind would be a 4GB ram 4 virtual CPU without ckdb.
Ckdb however uses a lot of ram due to its nature of having all data memory resident for maximum speed so the ram requirements for a fully featured pool would be at least 16GB ram.

Q: Cross platform / other operating system support?
A: Linux is by far the most proven, tested, deployed server operating system of choice today, and by limiting ourselves to writing for it we are able to use many unique linux features and not have surprise differences in operating system behaviour. When working with low level socket code as ckpool does without a library, the actual operating system behaviour and APIs themselves matter, so it is not worth trying to port them across which is one of the major complaints about the protocol.

Q: What payscheme is it? Paying of orphans? Paying of transactions?
A: This is a choice at a pool operator level and has nothing to do with the core ckpool code. While intrinsically there is code for PPLNS in ckdb, the pay scheme and mechanism is up to the pool operator who sets up the pool with this code.

Q: Why not p2pool as a backend?
A: p2pool as a concept has failed as a scalable concept due to continually marginalising its smallest miners and increasing variance the more miners it has instead of decreasing variance the way a traditional pool does. I have not completely discounted this as a module for ckpool if we can come up with valid workarounds and we are always considering new options. We are instead proposing future development of sibling pools with ckpool to decrease variance by combining rewards from multiple pools into smaller pools. More on this to come.

Q: Other mining protocols?
A: While stratum may not be a perfect mining protocol, the adoption of it as the defacto standard for scalability and its robustness has been well proven. Instead of rewriting the protocol, we believe that a solid implementation is enough to offset any disadvantages to the protocol, and add extensions as the need arises. We support reporting of transaction hashes included in work templates as a security measure for validation by miners and auditors.

Q: GUI?
A: A GUI has no place in the core code of a server or database.

Q: Simple setup tutorial?
A: It is a cinch to set up a simple ckproxy or solo ckpool. If you need help setting up a fully featured pool then you shouldn't be doing it.

Q: Merged mining?
A: Any presumed profit increase from adding virtually worthless altcoins is not worth the disadvantage of introducing far more instability associated with running multiple lesser unstable coin daemons. The running overheads of ckpool should be so small as to allow low fees from pool operators and stability. Some purported free pools with multiple coin support have so much downtime that any extra profit from them is offset by their downtime and poor design.

Q: Altcoins?
A: The author has no interest in support for anything but bitcoin. You are free to modify the code yourself as you see fit, but no altcoin support will ever be merged into the master ckpool code.

Will be added to as the questions arise.

Stratum extensions:

In addition to support of all the common stratum features, ckpool supports extensions that are either not yet in use, along with its own interpretation of further stratum features as the need arises.

mining.passthrough

This is a unique stratum extension supported only by ckpools upstream and creates a stratum-through-stratum passthrough. It is not intended to be used by miners directly except through the use of the ckpool code itself in passthrough mode for connection combining.

mining.node

This is unique stratum extension supported only by ckpools upstream and creates a stratum-through-stratum passthrough and additionally receives workinfo information from the upstream pool to allow local block submission.

mining.get_transactions

This controversial bandwidth eating feature has been absorbed into ckpool but instead of it handing out the full transactions which are a complete waste of bandwidth, the number of transactions is returned. In addition to supporting the originally documented form with job_id embedded in the method, ckpool will accept the valid json job_id as a single array parameter eg: params: ["$jobid"] .The functionality of the original feature request has been supplanted by the following call instead:

mining.get_txnhashes

Instead of handing out the full transactions incorporated into a job, this version returns only the hashes of the transactions included in each job in a space separated string as the result. The returned hashes are less than 1/10th the bandwidth that returning the full transactions would have amounted to, but is ample information for auditing tools or external authorities to confirm what the running ckpool is working on. This feature is on and rate limited to return only one response per miner per job_id to avoid it becoming a source of DoS. eg:

mining.suggest_difficulty

Special handling is done for accepting a diff as requested by each mining connection. This is done separately from the per-worker connection so it allows overriding that. When a miner suggests a difficulty, ckpool will allow a slightly lower diff than it would have otherwise chosen, but still will not exceed 30 shares per minute to prevent miners from using this to flood the server with low diff shares.  In addition to supporting the originally documented form where the difficulty was embedded in the method name, ckpool will accept the valid json where the difficulty is a single arrray parameter in params. eg:

mining.ping

This is as it says a simple ping sent to or from the miner without parameters. Ideally this would be used to ensure a miner is still alive or to inform the miner that the pool is still alive and ideally they should respond with a result of "pong" (or vice versa). Since most mining software does not currently support this it is not expected to be used for confirmation, although sending data is a good way to ensure a raw socket is still open.

CKPOOL configuration

At least one bitcoind (aka btcd) is mandatory in ckpool mode with the minimum requirements of server, rpcuser and rpcpassword set. If no btcd is set up in the configuration, ckpool will look for a running bitcoind on the local machine on the default port 8332 and the username "user" and password "pass" set up.

At the very least you will need the following lines in your bitcoin.conf file:


It is also recommended you start bitcoind with the -blocknotify feature calling the ckpool notifier as described here: https://bitcointalksearch.org/topic/m.8907597


Ckpool takes a json encoded configuration file in ckpool.conf by default or ckproxy.conf in proxy or passthrough mode unless specified with -c. Sample configurations for ckpool and ckproxy are included with the source. Entries after the valid json are ignored and the space there can be used for comments.

Sample ckpool.conf file:

Sample ckproxy.conf file:
The options recognised are as follows:


"btcd" : This is an array of bitcoind(s) with the options url, auth  and pass
which match the configured bitcoind. This is mandatory in pool mode. The
optional boolean field notify tells ckpool this btcd is using the notifier
and does not need to be polled for block changes.

"proxy" : This is an array in the same format as btcd above but is used in
proxy and passthrough mode to set the upstream pool and is mandatory.

"btcaddress" : This is the bitcoin address to try to generate blocks to.

"btcsig" : This is an optional signature to put into the coinbase of mined
blocks.

"blockpoll" : This is the frequency in milliseconds for how often to check for
new network blocks and is 100 by default. It is intended to be a backup only
for when the notifier is not set up and only polls if the "notify" field is
not set on a btcd.

"update_interval" : This is the frequency that stratum updates are sent out to
miners and is set to 30 seconds by default to help perpetuate transactions for
the health of the bitcoin network.

"serverurl" : This is the IP to try to bind ckpool uniquely to, otherwise it
will attempt to bind to all interfaces in port 3333 by default in pool mode
and 3334 in proxy mode.

"mindiff" : Minimum diff that vardiff will allow miners to drop to. Default 1

"startdiff" : Starting diff that new clients are given. Default 42

"logdir" : Which directory to store pool and client logs. Default "logs"



It is anticipated that pool operators wishing to set up a full database based installation of ckpool+ckdb will be familiar with setting up postgresql and associated permissions to the directories where the various processes will communicate with each other and a web server so these will not be documented.

If you are not capable of setting up and running a database and/or webserver then you should be questioning whether you should be running a public pool or not.

Building ckpool

Apart from a basic development package installation, as ckpool is mostly hand written without using libraries, it has zero external dependencies. If ckpool is built with ckdb support, only the postgres development library needs to be installed to build it. Most proxy users and simple solo ckpool setups will not require ckdb.


(Note that this is only for a tarball release which does not yet exist):
sudo apt-get install build-essential
./configure --without-ckdb
make


Building with ckdb requires installation of the postgresql development library.

sudo apt-get install build-essential libpq-dev
./configure
make


Building from git also requires autoconf and automake

sudo apt-get install build-essential libpq-dev autoconf automake libtool
./autogen.sh
./configure
make


Binaries will be built in the src/ subdirectory.

Installation is NOT required and ckpool can be run directly from the directory
it's built in but it can be installed with:
sudo make install