It was the Bitcointalk forum that inspired us to create Bitcointalksearch.org - Bitcointalk is an excellent site that should be the default page for anybody dealing in cryptocurrency, since it is a virtual gold-mine of data. However, our experience and user feedback led us create our site; Bitcointalk's search is slow, and difficult to get the results you need, because you need to log in first to find anything useful - furthermore, there are rate limiters for their search functionality.
The aim of our project is to create a faster website that yields more results and faster without having to create an account and eliminate the need to log in - your personal data, therefore, will never be in jeopardy since we are not asking for any of your data and you don't need to provide them to use our site with all of its capabilities.
We created this website with the sole purpose of users being able to search quickly and efficiently in the field of cryptocurrency so they will have access to the latest and most accurate information and thereby assisting the crypto-community at large.
I just had to figure out how to repackage them into bigger blocks. After a couple of false starts over a couple of frustrating days (wouldn't it be nice if our ideas always worked perfectly the first time?), I came up with the following scheme:
First, I hacked the reference implementation and changed some constants related to the block size (MAX_BLOCK_SIZE, DEFAULT_BLOCK_MAX_SIZE, MAX_BLOCKFILE_SIZE). That part is easy.
The tricky bit is how to take transactions from the 1-MB and repackage them in bigger blocks. For most transactions, it is trivial, because ordinary transactions don't care what block they are in.
There are two exceptions:
1. Transactions that use the 'locktime' feature so they are not valid until a particular block height or date. I dealt with these by simply treating all transactions as 'final'.
2. There are special rules for 'coinbase' transactions: there can be only one coinbase transaction per block, it must be the first transaction, etc. My first couple of false starts went down the path of relaxing those rules, but I gave up that approach because I found myself changing more and more code. I think if I was 22 years old I probably would have just forged ahead, unwilling to let go of a couple of days of programming effort because "just one more change and it will probably all start to work..."
I wonder if old programmers throw away more code than young programmers. I bet we do.
Anyway, after a rethink I came up with a much cleaner solution for handling coinbase transactions from the real blockchain: I write them to a 'coinbasetx.dat' file which the hacked bitcoind reads at startup and stores in memory (there are only 73 megabytes of them). As blocks are added to the big-block chain, those coinbases are added to the "unspent transaction output set" so they are available to be spent (and I set the COINBASE_MATURITY constant to zero instead of 100, so they can be spent right away, in the same big block).
I wrote a tool ("gen_megablocks") that creates the coinbasetx.dat file and blk*.dat files that are valid-but-oversized, -regtest-mode blocks.
static const unsigned int MAX_BLOCK_SIZE = 20000000;
static const unsigned int MAX_BLOCK_SIZE_GEN = MAX_BLOCK_SIZE/2;
static const unsigned int MAX_BLOCK_SIGOPS = MAX_BLOCK_SIZE/50;
static const unsigned int MAX_ORPHAN_TRANSACTIONS = MAX_BLOCK_SIZE/100;
static const unsigned int MAX_INV_SZ = 50000;
static const int64_t MIN_TX_FEE = 1000000;
.....