Topic: [ANNOUNCE] ecoinpool - A brand new pool mining software written in Erlang - page 7. (Read 28491 times)

Hmmmmm. I wonder how well this might be suited to creating p2pool "supernodes" or "minipools" that link in with p2pool, but provide normal (I.E., diff 1) work.

Hi eleuthria, hi rjk,

it's nice to have some discussion going on here. I've been thinking about this "federated" or "distributed" pools idea a lot, also got some input from various chats and other sources. Here are some problems and tentative solutions or thoughts that came up, feel free to add comments:

Workers

• Sharing workers between servers is no problem if you have a database which offers replication
• CouchDB is very good at this, syncing up changes in nearly realtime; after a server crashes and recovers, all changes which have been made are grabbed from peer servers; also it offers streaming changes to applications (like ecoinpool)
• MySQL also has replication features, though I've never tried that out; polling for changes from the application would be required though
• A custom database and replication engine for ecoinpool could also solve the task of worker sharing and would be even faster

Shares and Statistics

• Besides worker configuration, all other pool services revolve around shares
• Large pools have to handle shares in masses
• CouchDB can easily get to its limit there and I wouldn't recommend it for shares storage anymore
• Having a central database would produce a bottleneck or single point of failure; only a hidden dedicated server where only the shares database is running would be ok
• The other option is to store & process shares on each server separately; however, this requires a mechanism to sum up the number of shares from each server and present them correctly to the user
• Creating long-term statistics out of short-term shares is also not an easy task; yet this could again be achieved with replication-enabled databases like CouchDB as the amount of data is reduced greatly and record sets drop in comparatively slowly
• Live monitoring is something really cool, everyone who experienced that in ecoinpool's testing phases instantly fell in love with it and wanted to keep it - but it relied on CouchDB which isn't built for monster rigs...
• Work has started on an alternative way to provide live monitoring without CouchDB, but this gets complicated if multiple servers are involved
• The current idea is to have a server-to-server protocol for shares announcement; like this, shares could be pushed via HTTP longpolling from any server to the user's browser

Daemons and Block Solves

• Current pool website software (i.e. cron jobs) detect block solves by querying a coin daemon (bitcoind, namecoind, litecoind...)
• This requires either to connect to all servers in order or having a central instance of the coin daemon or setting all daemons to the same payout address of the central server (currently not supported for namecoin)
• ecoinpool, having a built-in block monitor, could do block solve detection by itself and publish the information somewhere (block hash, Tx ID, number of confirmations, etc.) leveraging the requirement of those cron jobs

Payout

• Payout usually involve processing shares
• If not done correctly, shares could be paid out more than once
• First solution is again having a central shares database and payout
• The alternative is to let each server manage their own presumably partial payout; this is a good decentralized solution but requires some server-to-server communication to make it convenient for the user
• Different payout mechanisms put different demands on the shares database; not all payout mechanisms can be used in a decentralized way
• Legacy systems rely on existing structures (e.g. cron jobs working on SQL databases)

Sadly that wouldn't stop it .  I tried that in the past.  The problem with it is a VPS is much more likely to get blacklisted and suspended if targetted by an attack.  So the result of the DDoS would be a suspension of your services very quickly.  They wouldn't have to hold all your servers down at once, they'd just have to hold them down long enough for the VPS provider to turn it off, then move to the next one.

Not to mention a gigabit VPS where you're allowed to use the whole gigabit without overage charges is not that cheap ;p

However, that isn't saying that VPSes are bad.  They're quite useful in quickly scaling up/down for legit users.

If it is truly that scalable, then you could have tiny VPSes all over the world and one big DB server somewhere - thus making DDOS just that little bit harder for the idiot troublemakers that try to pull them off.

20 VPSes each with a gigbit connection somewhere in the world, all on a custom DNS rotation. That sounds like it could be a little fun to manage

I'm going to keep an eye on this, maybe even offer some of BTC Guild's large-scale miners an option to help test it out.  PoolServerJ has worked great for me, but that memory footprint is huge.

Thank you very much. I really like the idea of bringing some life into the scene. It's the competition that takes us to our limits and improves the overall quality of software to the benefit of everyone. You always need a little challenge to advance in your life.

Now, the reason why it's so lean on resources is, well, because it's not Java. And the reason why it performs so nice is because of Erlang's way of handling concurrency. Erlang comes with its own process scheduler; an Erlang process can be seen as a thread but is actually more lightweight than a regular operating system thread. There can be tens of thousands of Erlang processes at the same time without any significant performance loss, whereas on Java, every Thread is a little pain and a few hundred Java Threads can bring a system down to its knees.

Taken all the features of Erlang, the functional programming paradigm is imho more than ideal for developing realtime server applications. Plus, implementing a parser with any functional language is a piece of cake thanks to pattern matching. Erlang finally extends pattern matching to binaries, making it easy to use for any kind of binary network protocol like Bitcoin's. And if you do it right, your code will stay short and crisp and thus easy to maintain. Functional languages can be very powerful and expressive. Every line of code can be worth ten lines in other programming languages. On the other hand, every line will take you ten times as much to think about (well, if you're a perfectionist like me).

It's also worth to say that obeying to some basic programming principles can significantly improve the quality of your software: DRY - Do not Repeat Yourself. Work with encapsulation. Readability counts. Explicit is better than implicit. Flat is better than nested. Modularize where possible. Try unit tests and source code analyzers.

Congrats on the release.  I've been out of action for a couple of months due to personal commitments and poolserverj support has suffered quite a bit as a result  and pushpool doesn't seem to be getting a lot of developer love lately either so I'm glad to see the pool engine scene get a bit of new life injected into it. 

Look forward to spending some time running up a test instance and checking it out.  I don't know the first thing about Erlang or CouchDb (actually the first time someone mentioned Erlang to me I thought they were yanking my chain).  Though I've been ass deep in Scala tutorials lately and had a vague idea to rewrite psj v0.5 in Scala to see how I could apply the functional paradigm.  Very interested to get to know the code a bit and investigate how it manages to be so lean on resources without sacrificing performance. 

About two months ago, I started writing my own pool mining software from scratch using Erlang and CouchDB. After hundreds of hours of development, it has finally reached beta-quality and is hereby released open source to the public. ecoinpool moves ahead in a new direction, so I'd like to call it the next generation pool mining software.

Feature headlines

• Configuration completely through the database
• No delays, no restarts, no polling, no SQL, no Java
• Host multiple chains at once (Bitcoin, Namecoin, Litecoin and Fairbrix)
• Merged mining supported for Bitcoin+Namecoin
• Fast internal work creation with getmemorypool
• Fault tolerant, self-restarting on crashes
• Live shares monitoring through the browser
• Prepares data for displaying statistics on the fly
• Integrated web frontend, yet usable in existing websites
• Built-in Block Monitor and Mini-Blockexplorer
• Scales across multiple servers
• Backwards compatibility layer to MySQL

Target audience

ecoinpool tries both to be easy enough for solo-mining setups and scalable and fast enough for large pools with multiple servers around the world.

Main links

• Project Homepage
• Project Wiki
• GitHub Repository

About this project and Erlang

I always wanted to implement a large server project with the right tools. Erlang/OTP, a functional programming language and platform developed by Ericsson (nowadays open-source and community-driven), is the ultimate tool for this task. Some things that other languages or platforms can achieve only with tons of libraries are integrated right into the core language. Concurrency and inter-process communication are the pillars of Erlang, combined with a powerful pattern matching implementation which is typical for functional languages. Erlang is dynamically typed and allows reloading modules while the application is still running, allowing rapid development while reducing downtime to zero (if done correctly). A sophisticated error handling mechanism concludes this little features list.

Erlang has two things in common with Java: It is compiled to byte code and runs on a virtual machine. At this point the similarities end. Erlang requires only a fraction of Java's memory and CPU requirements and can scale from a single core desktop up to large clusters of multi-core servers. It has literally been built for that very reason. So, another part why I started this project is that a Java solution already exists and set the standard somewhat high. So I was wondering if I could write a software that can surpass it by using all of Erlang's powers against Java's weaknesses.

Other links

• README
• Setup on Ubuntu Linux
• Test Server
• Why CouchDB?

Coming soon

Actually it is "sooner or later", because ecoinpool has already consumed lots of my free time and I got other things to do now.

• Remove requirement for CouchDB and integrate it's notification and replication features directly into ecoinpool
• Configurable shares logging, support for other data sources for configuration and workers
• More documentation
• Better web frontend for the Mini-Blockexplorer
• Include payout mechanisms in the software
• Support for an external Coinbaser
• Extend Block Monitor to a full Bitcoin client

Changelog

v0.3.10 - Jan 3, 2012

• Implemented the ebitcoin configuration frontend

v0.3.9 - Jan 1, 2012

• Added support for Fairbrix (getmemorypool patch required)
• More dynamic block monitor/explorer to add new chains faster

v0.3.8 - Dec 31, 2011

• Added support for Litecoin
• Slightly improved web interface

v0.3.7 - Dec 30, 2011

• First public release

Closing words

So far I have talked to wtfman/cuqa of btcserv.net (Germany) and Graet of ozco.in (Australia) and they said that they will give it a try within the next weeks. If you are a Bitcoin miner, better keep an eye on those pools.

ecoinpool is successfully running on Elitist Jerks (LiteCoin), set up in cooperation with WKNiGHT (USA).

I'm looking forward to your feedback. If you have questions about some aspects of ecoinpool or need help with installing and setting up, feel free to contact me in #ecoinpool on Freenode.