Topic: CKPOOL - Open source pool/proxy/passthrough/redirector/library in c for Linux - page 21. (Read 124144 times)

About libckpool

While not strictly feature complete as a library that can be installed, dynamically linked etc. as per other Linux libraries (yet), libckpool is part of the ckpool suite that is designed to be easily included into other projects without requiring any of the other ckpool code. Any and all functions that can be put into the libckpool.c/h files without being dependent on ckpool are placed there for convenience for others to use in their project, thus benefiting from the many common functions a c/c++ application might need, along with many of the unique ck* functions created for the cgminer and ckpool projects that others might find useful.


About ckpmsg:

When ckpool is run, a series of unix sockets are created: listener, generator, stratifier, connector usually in /tmp/ckpool (though configurable) and ckdb creates its own listener socket in a ckdb directory.

Ckpmsg is a simple application designed to talk locally to the various ckpool/ckdb proceses using the message structure that ckpool expects, usually to the listener socket by default if called with no arguments, looking for ckpool by default but can be aimed at ckproxy (with -p), a named ckpool (with -n) or a user chosen socket directory (with -s). Currently ckpool only accepts a limited number of commands on its listener:

shutdown
restart
loglevel=$val (from 0-7)
ping

It looks for a response to any command sent, and it currently can't distinguish those that will and those that won't return a response so it will complain if none is returned.

Although the same ckpmsg can be used for any of the low level functions within any of the processes, it is expected that admins will mostly be using the simple functions. It can also be used for any number of ckdb queries.


About notifier:

This is an extremely simple binary that is designed to be run with the blocknotify feature of bitcoind. It simply tells ckpool to look for new block information and can be run with the -s option to tell it which socket directory to use to tell ckpool about block updates. Standard usage would be something like:

Create a notify script such as notify.sh:

Start bitcoind with blocknotify telling it to call that script:

Using a script as blocknotify allows you to change the script without having to restart bitcoind.

About ckproxy

Ckproxy uses the same core code as ckpool and acts as a proxy with a simple switch (-p). It acts as a stratum-to-stratum proxy, or stratum-through-stratum proxy in passthrough mode (see below).

It is most useful to regular miners who wish to consolidate their mining hardware into one connection for minimal upstream/downstream bandwidth in standalone mode (with the additional -A switch). Ckproxy uses multiple modes to maintain as many miners as possible communicating with the upstream pool. It works by splitting up nonce2 sizes if the upstream pool's nonce2 is large enough and then it recruits extra upstream connections as needed to allow the number of downstream miners to scale indefinitely.

Ckproxy can be started in "userproxy" mode which monitors login names of miners attaching that don't match the master proxy username and recruits new upstream connections with each unique username, proxying all workers of the same username to the unique upstream proxy connections.

It can be configured to run with a database as per an actual pool entirely, thus acting like a child pool for a parent pool elsewhere, or simply for miners who wish full logging of every detail of their mining operation.

It can also act as a unique stratum-through-stratum passthrough mode which absolutely requires a ckpool as the parent pool with the -P switch. In this mode it does not decrease the bandwidth talking to the upstream pool but minimises the number of open connections instead, but requires extremely low resources to run. This mode is most useful for pools wishing to isolate their main pool instance from the outside world and set up multiple VPSs as a kind of front end proxy/firewall to the outside world. This also removes any realistic limit on the number of open connections since each passthrough instance could easily handle 10k connections, consolidating each into just 1 connection to the upstream pool.

In addition to the passthrough mode, there is a more advanced "node" mode which connects as a passthrough, but needs a local bitcoind connection of its own. It monitors all traffic and shares between the pool and miners, being able to display local hashrates and will submit any blocks found to the local bitcoind in addition to sending the shares to the upstream pool, circumventing the delay of block submission that would otherwise happen with remote nodes.

Finally ckproxy can be started in "redirector" mode which acts as a regular passthrough, but monitors share responses from the upstream pool, and once a valid share is recognised from the upstream pool it will redirect miners that support redirection (all cgminer based clients do) to the URL of your choice. Should the miners not support redirection, such as rental services, ckredirector will continue to act as an ordinary passthrough.

About ckdb

Backend database for full user management of ckpool.

Basic concept is a memory resident database with PostgreSQL as the backend DB for permanent storage.
Performance is the priority of ckdb.

Running live and stable on my pool and the hashmine pool.

Still in development ... see the git source code

About ckpool

Ckpool is a project by the cgminer development team designed to complete the circle between miners and pools from cgminer to ckpool. The bulk of the code work for this was commissioned and paid for, however I provided an incentive to the group commissioning the code to open the software up to all by charging less for free software than for closed code. Ultimately this should benefit bitcoin far more than me simply working on code for another pool. I intentionally chose to write most of the code from the ground up and not use external libraries to have complete control over all the code given past experiences of difficulty to debug failures in library code that was outside our domain with cgminer. The separation of the pool code from the database code allows the pool code to be ultra-scalable and extremely low overhead by dissociating any concept of share counts, user accounts, authentication, front ends, payschemes etc. etc. These can be as vastly different from the envisioned ckdb+postgresql+php interface as the administrator desires without any change to the core pool functionality.


Rationale:

The aim of this project is to provide the most low overhead, massively scalable pool software available by creating the code from the ground up using the same programming techniques and tools as those used when the author codes for the linux kernel. It was decided early on that to speed development and maintain consistent behaviour that only Linux would be supported. Even with massive numbers of miners, the core ckpool code uses very little CPU and only 100s of megabytes of ram.

We aim to remove the dependency of pool owners on custom software and the requirement for massive hardware infrastructure to run a stable and performant mining pool, thus dramatically lowering the barrier for entry for more pools to be created. In the process, we also aim to be providing tools to allow large mining farms to mine for themselves instead of directing their power to a large pool which would otherwise make them even larger and risking the security of the bitcoin network. While it may seem counter to the small miner's benefit to make large miners capable of running their own pool, it is an acceptance of where the bulk of the mining power is coming from and not make that mining power a risk to bitcoin. By providing them with software to mine in a manner that isn't designed to just mine blocks quickly but does so in a way that can lay down all transactions currently in flight without detriment to their block generating capacity, this will benefit the bitcoin network far more than them adding their hashes to a large pool.

Despite the pool software being provided to large miners to run standalone, the bulk of the development effort was actually directed to providing a reliable consistent and performing pooled mining solution (in the traditional pooled mining sense) with a stratum mining module at the core of the miner's experience that is not hindered in its ability to serve any sized miners with expedience and to create a platform for expanding on the existing stratum design as the need arises. As the core is modular in design, there is scope for plugging in extra components in the future as desired. Also as the same code is used by the pool software, the optional proxy included, the associated library, and the developers maintain the code at both ends, there is assurance the two ends will talk optimally.


DESIGN:

Architecture:
- Low level hand coded architecture relying on minimal outside libraries beyond
basic glibc functions for maximum flexibility and minimal overhead that can be
built and deployed on any Linux installation.
- Multiprocess+multithreaded design to scale to massive deployments and
capitalise on modern multicore/multithread CPU designs.
- Minimal memory overhead.
- Utilises ultra reliable unix sockets for communication with dependent
processes.
- Modular code design to streamline further development.
- Standalone library code that can be utilised independently of ckpool.
- Same code can be deployed in many different modes designed to talk to each
other on the same machine, local lan or remote internet locations.


Modes of deployment:
- Comprehensive pooled mining solution with a postgresql database interface.
- Passthrough node(s) that combine connections to a single socket which can
be used to scale to millions of clients and allow the main pool to be isolated
from direct communication with clients.
- Proxy nodes with a database that act as a single client to the upstream pool
while storing full client data of their own.
- Simple proxy without the limitations of hashrate inherent in other proxy
solutions when talking to ckpool.
- Simple pool without a database.
- Library for use by other software.


Features:
- Bitcoind communication to unmodified bitcoind with multiple failover to local
or remote locations.
- Local pool instance worker limited only by operating system resources and
can be made virtually limitless through use of multiple downstream passthrough
nodes.
- Proxy and passthrough modes can set up multiple failover upstream pools.
- Optional share logging.
- Virtually seamless restarts for upgrades through socket handover from exiting
instances to new starting instance.
- Configurable custom coinbase signature.
- Configurable instant starting and minimum difficulty.
- Rapid vardiff adjustment with stable unlimited maximum difficulty handling.
- New work generation on block changes incorporate full bitcoind transaction
set without delay or requiring to send transactionless work to miners thereby
providing the best bitcoin network support and rewarding miners with the most
transaction fees.
- Event driven communication based on communication readiness preventing
slow communicating clients from delaying low latency ones.
- Stratum messaging system to running clients.
- Accurate pool and per client statistics.
- Multiple named instances can be run concurrently on the same machine.


Processes:
When ckpool is run, a number of separate processes are spawned in order to capitalise on multi-core CPUs by offloading dedicated tasks to one process at a time in order to not allow any one process to slow any other. They talk to each other using unix sockets which by only talking on the local system cannot be influenced by the outside world.

The processes generated are:

The main process, AKA listener - this is the core process that listens for incoming requests from an admin socket and watches the running of the child processes.

The generator - this process is designed to only talk to bitcoind to generate work templates, check the current block hashes, check bitcoin addresses, submit block solves, or in proxy mode talks to an upstream stratum server to emulate generation of work. This also runs a watchdog thread to check the running of the upstream bitcoind/or upstream stratum pool and can perform failover should the bitcoind fail for whatever reason.

The connector - this process accepts incoming connections from miners, handling requests for new worker connections, receiving and sending data to the miners. It does this in an event driven manner with multiple send/receive/listen threads, automatically prioritising sends to downstream miners according to their readiness to receive data, thus preventing any one worker affecting any others.

The stratifier - this is the core stratum code implementation that receives data from the connector and runs multiple threads to create work templates, process incoming data, check for valid work, queue outgoing data, authenticate new users, and optionally talks to the ckdb process to store all manner of data for running a full database based pool.

Each process runs additional threads to perform logging in a queued manner that does not hinder the underlying running code.


Advantages to miners:

A stratum implementation with extremely rapid and accurate adjustment of diff to an optimal value both up and down in diff per worker.
Once diff has stabilised, it is extremely rare for diff to be adjusted again unless you add or remove hashes.
Diff is actually properly associated with work templates which means you don't "leak" rejected low diff shares when the diff changes.
Stable diff levels and proper work association with diff leads to lower rates of rejects.
Very rapid block propagation support to all miners means all miners get notification of, and work on, the latest block work as quickly as possible to not waste work.
No extra work restart messages for the inefficient zero-transaction work that other pool software means much less lost work.
Hashrates should hold much more stable at higher levels as a result of all these benefits.
Reconnect support: If you have temporary glitches in your networking, ckpool supports resuming and accepting of valid cached shares.
Messaging: Miners watching their console will see stratum messages as sent to them by the pool (such as block solves).
Virtually no downtime for pool restart: Pool restarts only register as an interruption and then reconnect instead of failing over to backup pools due to the unique socket handover mechanism in use.
Rewards: Maximum rewards are guaranteed by incorporating transactions into every work template instead of mining transaction-less blocks.
Confidence: Improving the bitcoin network through seamless transaction processing increases the confidence in bitcoin in general allowing expansion of the bitcoin network in terms of numbers of transactions and block sizes.
Code auditing: With the ckpool code all freely available, miners are able to audit the code for themselves.
Transaction auditing: Ckpool automatically includes transaction hash reporting to enable auditing of mining work for miners/auditors that request it.

CKPOOL + libckpool by the cgminer development team.

Ultra low overhead massively scalable multi-process, multi-threaded modular
bitcoin mining pool, proxy, passthrough, library and database interface in c
for Linux.

ckpool code by ckolivas / -ck
ckdb code by kano

Note that as maintainer of ckpool am FOR segwit and AGAINST Emergent Consensus.

Source code downloads:

https://bitbucket.org/ckolivas/ckpool

Current release status: Beta v0.94, from git source only.

IRC channel: irc://irc.freenode.net/ckpool

LICENSE: GNU Public license V3. See included COPYING for details.

About ckpool: https://bitcointalksearch.org/topic/m.8907250

About ckdb: https://bitcointalksearch.org/topic/m.8907583

About ckproxy: https://bitcointalksearch.org/topic/m.8907589

About libckpool: https://bitcointalksearch.org/topic/m.8907597

About ckpmsg: https://bitcointalksearch.org/topic/m.8907597

About notifier: https://bitcointalksearch.org/topic/m.8907597

Building: https://bitcointalksearch.org/topic/m.8907600

Configuration: https://bitcointalksearch.org/topic/m.8907601

Stratum extensions: https://bitcointalksearch.org/topic/m.8907604

FAQs: https://bitcointalksearch.org/topic/m.8908863

Discussion/Future directions: https://bitcointalksearch.org/topic/m.8908866



List of current ckpools:

http://www.kano.is
http://hashmine.io (private for now)
http://solo.ckpool.org (anonymous solo mining)
http://testnet.ckpool.org (testnet only)