Topic: [ANN][COMB] Haircomb - Quantum proof, anonymity and more - page 4. (Read 6890 times)

Well that's useful.

I've done all the other stuff, but I don't see how you're getting BTC to run properly with a blank rpc user/pass. If the bitcoin.conf has "rpcuser=" and "rpcpassword=", it's smart enough to know nothing is there and instead spawn and use a cookie file for the relevant info.

Unless I'm missing something, if we want to allow the user to operate COMB without using a config, then we need to set default info (i.e. "user" and "pass") and tell them to insert those into the bitcoin.conf file. I've set it up and modded the error message, but let me know if I'm missing something.

I also just merged the "btc is behind" alert into the "wallet isn't synced" alert, let me know if you think it should be more specific or not.

last things

1. make log file optional and specify the log file name in config.txt, if none or by default don't log.
2. config file should also be optional, we will change the default port to 2121 again, users are used to it.
3. Default rpc password / username should be empty string. It can be used no problem with BTC if you set
that in BTC config (if someone is in rush and doesn't care about security/syncing they won't be making a config file).
4. up the version counter for beta in deployment.go (need to increase it every time when releasing something,
because otherwise there would be versions in the wild that we won't be able to tell what it is later, it's ok to
skip versions)

minor things

message about use 546 sats to claim, change to 330 sats, (it's cheaper).

the message on the main page "COMB height is greater than BTC" is by itself ok but should be reworded,
probably something like "COMB height is different than BTC".

check_btc_is_caught_up() had no boolean result, check it before release, go build should pass

remove fmt.Println(hash), fmt.Println(segments_stack) from stack.go, it just
spams every time I sweep coins. It's not needed.

in general, turn our debug statements that we added so far into log printing or something,

When releasing beta, the less printing on console the better. Except when the coin
crashes. When a new dev joins he/she can put their own debug prints and see them,
not search for their own prints in a pile of spam.

Testnet tools:

To decode bc1 address into witness program:

http://bitcoin.sipa.be/bech32/demo/demo.html

To encode witness program back, but into testnet tb1 address (can be used to claim tesntet comb).

https://bc-2.jp/tools/bech32demo/index.html

Bitcoin command for testnet (prune needs to be high, because with low prune the BTC
will escape from and Comb will get fall behind unable to sync the blocks - "wrong height" ):


Can generate brainwallet for testnet using:

/">http://127.0.0.1:2121/wallet/brain//

We've already synced with testnet. Let's hit faucet, get some testnet bitcoins, and claim some testnet comb.

Holy cow dude, talk about industrious rofl. A lot of this stuff is Golang that I don't know yet, so that's great for me, time to do some learning lol. I guess now I'll start expanding the fake_block code.

One thing to note, your code removed the duration from the WaitForBlock struct, I re-added it in when merging. Looking at the code more it seems like I could've also just removed the duration check from the parse_wait_for_block() function though, I guess it's not exactly used now.

One thing I noticed recently, that may be an issue, every now and then when I was testing stuff in regtest the Haircomb mining loop wouldn't trigger the BTC rpc. It started happening after implementing the new waitfornewblock call; I'm not sure if a problem that's unique to regtest or not, but it's something to look out for. The solution was just to restart the BTC, everything started working again no problem.

I've pushed the code to my github so if anybody else is reading and wants to test it, go right ahead. I've just commented out some of the incompatible commitfile.go and left the rest in there for possible future inclusion of that legacy commit file option you were talking about before.

new  parse_wait_for_block looks good!

I went ahead and implemented the proposed format into the leveldb as well as the use of leveldb.Batch. The leveldb options
are also in use namely to disable compression and enable Sync (this is needed for db to survive power loss - was not tested actually
doing the computer power loss, but should work).

commitfilelvldb.go

https://pastebin.com/raw/c1GDhpAy

mine.go

https://pastebin.com/raw/bM50zwAR

newminer.go

https://pastebin.com/raw/4GrvKpUq

utxotag.go

https://pastebin.com/raw/7rDLYNRY

Overall I've fixed a few bugs:

1. In miner(), when reorging with direction -1 the second topmost block must be compared with previous hash, not the topmost, and the termination condition should be one_or_no_blocks()
2. miner() again, it should Put the block only with direction +1
3. rare race condition in mine_blocks. A local copy of next_download variable is needed to terminate the for loop, because if we use the global one, the goroutines that we spawn may increase that same global next_download, which could then lead to two goroutines to download a certain height and mine the same block twice.


What remains is, use btc_is_caught_up_mutex on the main page, remove mining subrouter s3 from main.go, clear stale code from commitfile.go

You're almost certainly right about this, my lack of computing experience is what led me to assume something that wasn't byte related lol. It'd be idiotic to have a DB that took values in a byte format only to store them in some other ordering.


Oh cool, I didn't know that that was a call that existed. That makes everything much simpler lol.


Maybe the best way to handle this is to just let the user know with a warning, but don't reorg. Similar to how the user gets told that the comb chain is way behind, so balances may look wrong, we can just display a warning like "Commits DB is ahead of the BTC chain. If you are rebuilding the BTC chain please wait for a full resync before operating to make sure your commits are accurate." Or something like that.

EDIT: I slotted in that code, along with the other small fixes mentioned before. Over the next bit I'll do a bit of testing, then move on to refactoring the metadata storage format and processing.

EDIT2: Made a minor change to the code, new parse_wait_for_block() is here: https://pastebin.com/raw/psXFsZbU
If you launch Haircomb before launching the BTC, and then launch the BTC, Haircomb's first (few?) RPC results will be "null", but not be detected as an error. This changes that.

huh I think it just orders in raw bytes, it would be shame to have blocks wrong-ordered IF leveldb sorts by UTF-8.


Yeah those stalls in the GUI are really annoying. I think I fixed it somehow, you know that outer loop in new_miner_start (newminer.go)

This is a standalone test of BTC behavior I needed to check. It can be integrated to comb.

https://goplay.space/#E5EVTWCocOD

Basically I sleep 1/10seconds, then call waitfornewblock RPC. That RPC is almost exactly what we need:

1. it waits until a block added to chain with configurable timeout
2. it returns height+hash in one call! awesome.

One last annoyance is the db completely getting erased when BTC is syncing + at lower height + at the same
chain branch that comb is.

That can be fixed by not reorging in case 1 when btc_is_caught_up=false. Basically we would wait for BTC to reach it's headers height and THEN reorg if needed.

Tell me what you think that is, if there is a worry that the longest valid (heaviest) blockchain would be actually few block shorter than the one comb is on.

I would've thought that the keys would have been ordered in numerical order, not "alphabetical" (so to speak lol), but that makes sense.


The problem with this is that BTC likes to stall on RPC responses if it's busy validating blocks. If all we're doing is alerting the user to the BTC connection status, then I can just modify the current loop to only run if the timestamp has been longer than X seconds so that the other calls can also trigger it.


I have some time now so I'll bang out all of this stuff over the couple of days.


Lol as far as I'm concerned none of it is yet, but I guess for now I'll go with the mining one because that seems like it's more dynamically integrated and could have more shifting variables, so I'm more likely to mess something up that'll affect it and be aware that it's happened

1. metadata and the complexity on initial commits db load is caused by not using
the proper keys for information.

- Block key in database should be just height uint64 in bytes, not prefixed by 99999.
- Commitment key should be the 128bit utxo tag in bytes (it too starts with 64bit height).

Then you can load everything using one NewIterator loop over the whole db because
block and its commitments are sorted by height and are alternating. You don't need any maps/lists or lookups whatsoever.

So when you see a block (a 64bit key) you flush the previous block (if any) and open a new Sha256 writer for checksum.
When you see a commitment (a 128bit key) you keep mining it as well hashing it into checksum writer.
Now, when you see a 1+ higher block you verify the previous block's checksum matches what you squeeze out of Sha256 writer.
Eventually, the whole db will be loaded or you will experience some kind of an error.
Make sure to check the checksum and flush last block when complete database is loaded (if the db contained at least 1 block of course).

In case of an error:

- A new function needs to be written that will clear commit_cache, commit_tag_cache
while commit_cache_mutex is taken.

- delete all commits and the block at the errored height using a new iterator with just that height as prefix.

- Then you should flip to deletion mode, that is continue the original iterator loop,
just keep deleting everything having a 64bit/128bit key at higher than the errored height.

2. the whole ping_btc_loop+miner_command_channel+miner_output_channel thing should be deleted.
It just makes our normal goroutine stuck communicating a pointless thing inside set_connected(false) when I restart BTC.
btc_is_connected should be a mutexed timestamp integer and set_connected(true) should set it to current time each time BTC responds.
Then, if btc_is_connected is a recent timestamp (not older than 2 seconds) we are CONNECTED.

3. slow CommitLvlDbUnWriteVal is still in place

4. get_block_info_for_height was not fixed. It can't contain loop and must get hash from internal source when dir = -1 NOT from the BTC.

5. what about "case 1" (ourhash_at_btc_height == hash)? Use switch u_config.reorg_type to use handle_reorg_direct in that scenario too.

6. When parsing block, we need to copy PBH (string `json:"previousblockhash"`) to the parsed block.
The function miner (func miner) needs to be modified to return error. It will return error when
PBH is not equal to the topmost block hash in case we are mining with direction 1 and there is at least one (previous=topmost) block already.
The above mentioned previous block error needs to be handled in all three places, there are two places in downloader(): handle them using 
stop_run();return. The remaining place is in mine_blocks(), there, just break the loop in case of this error.

7. I don't know whether direct / miner reorg_type should be the default. That depends on you and your confidence about which one is more
production ready.

There was a bug in the metadata loading process; if there were no commits in a block, the block's metadata would be skipped during loading, and the next attempt at pushing a blocks metadata would push a higher block that was expected and cause a crash. I've updated the CommitLvlDbLoad() with a fix, code is here: https://pastebin.com/qt2EGK0y

In the future a full refactor would make sense once we start including headers; first we can load all the metadata into a temp map, checking as we go along that a) the metadata for a block exists, and b) the headers match up and connect properly for the whole chain. Any break in either of those triggers the orphaning to be set at that height. Then we can go back and load the commits, comparing against their appropriate temp-loaded block metadata; unmatching fingerprint means that orphan mark would be triggered. After we've loaded all the non-orphaned commits, and removed any orphans and their block's metadata, we can then load the remaining blocks into their final map.

Actually I guess we could just load the metadata straight into their map and then remove any orphaned ones, that makes more sense.

That's a good idea lol, I hadn't considered just running them both xD

I'm just going to use "reorg_type" for both the struct key and the config.txt entry. Ideally we set one as the default and have the second as a fallback, so that unless it was needed the user wouldn't have to care about the entry. Which do you think makes more sense to be the default? Miner or direct?


These both make sense. The way I had it operating before was that a failed BTC call would trigger the btc_manager to send a signal to the miner that would turn "run" off, and then each loop cycle the get_block_info_for_height() would check that run value, and if it was off then it would shut itself down. Looking back on it though it feels a bit overcomplicated.


Oof yea that's not great. In the future we can jump store the commits but this works for now.


Yea I added this func for handle_reorg for this exact reason, the code I used was




Especially considering you wanted to add like 14 digits to the height number, yea adding 50000000 might not be so effective xD

Although if we're getting rid of the +50000000 method then we should probably just straight up remove the utag_mining_sign(tag) portion of the code; it won't serve any function. We'll need to change the flush detection as well at that point.

With interest I looked at the code.

do we intend to support both bidirectional mining and the new and much more
performant handle_reorg function? I suppose the answer is yes, I mean both
codes do pretty much the same thing and it makes sense to let the user choose.

And, supporting both codes using a config option will mean in case we fuckup
we can just tell users to reconfigure their clients instead of upgrading.

Now here is an example of problem that I have in mind (when using the old bidirectional unmining):

1. get_block_info_for_height() is wrong. That code absolutely must request
the block that needs to be reorged by it's block height and hash read from the db/ram.
Because BTC node might switched to different (better) block chain branch.
At that point a call to getblockhash(500000) will return block 500000 from the better
chain that BTC is on, not from the worse chain that we are on (assuming the reorg is
deeper than 500000).

2. the infinite loop inside get_block_info_for_height() should be removed,
we should just return error on any failure, that will terminate the 6 goroutines and
the downloader will reconsider what to do next after 1 second in the main loop.

But being a new function, I notice the imperfection in handle_reorg(), namely
it corrupts the aes checksum, because it erases commitments in random order,
I think the simplest fix would be to sort each block's commitments by their
utxo tag in decreasing order. So, once temp_commits_map gets populated:


3. I also think I know why the bidirectional unmining is now slow because there is the
function CommitLvlDbUnWriteVal which loops over the entire db I think it can be
eliminated instead:



4. we need to set direction_mine_unmine to UNMINE when reorging (this was previously
done by adding 5000000 to height but thats ugly), in miner_mine_commit_internal:

5. At this point when you set your fake block simulator to quickly reorg blocks,
it will eventually return to the initial situation with checksum of zero

also the database at that point should be cleared. But I still see inside db
the blocks hashes starting at 9999. These need clearing too.

Holy crap, I was planning on making a JSON api where we could just type out fake block info in a text file, but that's way better. That's awesome.

I actually removed the multi-directional mining because I think it makes more sense to reorg based on stored values. In some unknown case where a commit is stored in a block that gets reorged, but the commit isn't real and doesn't exist on the block, the commit won't get unmined. I guess that maybe this won't matter in the end; storing block fingerprints means this should be detected on start up, but just iterating through the commitsdb and removing all commits that have a height above the reorg target seems like the most simple and effective way to go.

The reorg code I'm using atm is here, your instructions were helpful: https://pastebin.com/raw/HFERMEvy. I had to make some changes to the check/find so they're included, as well as the main() so the integration code is there too. The CommitsLvlDbUnWrite is just a delete func based on the given key.

Let me know if you think there's any advantage to having the multidirectional mine option.

EDIT: Implemented the miner, I'll run a speed test tonight after my BTC catches up. Currently using your reorg but like I said, let me know what you think about just pulling straight from the DB.

EDIT2: Modded the DB load to push the block hashes to the map: https://pastebin.com/raw/31ieUe73

EDIT3: Made some minor rough changes to the fake blocker to receive commands. Right now it's just changing the speed it runs at and pausing/starting it; https://pastebin.com/raw/eqZkJTRY

EDIT4: Ran a full build test; the miner took ~10 hours, so it's as fast the the previous versions, and you fixed whatever was wrong with my variation, so yay!

EDIT5: I got the first bit of the fingerprint/block metadata code working. Right now it doesn't check the fingerprint on load, but it does store the block metadata (fingerprint, hash, height) on mine and load it on load. Code's on my github, there were too many scattered changes to drop it in a pastebin. The miner.go have been removed, and the relevant bits merged into newminer.go. Let me know what you think about the straight-to-db reorg vs the multi-dir mine reorg.

EDIT6: Added fingerprint checking, code's on the github. There's a TON of datatype switching that can be optimized/removed later, but it works for now. I also modded the initial fingerprinting so that now the final fingerprint includes the hash of the block too.

okay if you can't batch unwrite, we will need to clean up at startup.

take a look at this newminer.go

https://pastebin.com/raw/2RCREVsy


and here is a test RPC server that serves test blocks, the initial height is 500000

https://goplay.space/#2TGEZUC9ezM

It doesn't look like you can batch unwrite with this leveldb, just batch write. Let me know if I'm missing something. It should be fine; if the first thing removed from the commitsdb is the associated headers, then even if the process crashes mid-reorg the on-start orphan clean-up should take care of the rest. I'll just pull the relevant commit keys when I'm iterating through the commits map, then remove them from the commitsdb afterwards one by one.

I'm also not sure what you mean by "2. make the  utxo tag correspond to the target height, set it's txnum=0, outnum=0, direction=false." What utxo tag? EDIT: Do you mean the commit_currently_loaded?

I'm assuming you mean the tx_leg code in the the commits_rollback section, not the commits_cache section, correct?

This won't unmine in order of commits within the same block, but that's obvious so I'm assuming its fine.

EDIT2: I got the reorg working, but I ran a full pull and something's wrong with the way I implemented the mining code it seems like, so I'll have to figure that out.

what needs to happen when reorg back to a specific height (target height):

1. lock the commit_cache_mutex and commits_mutex,
2. make the  utxo tag correspond to the target height, set it's txnum=0, outnum=0, direction=false.
3. Run a for loop from the max height down towards target height:
4. - loop over the commits map. For each commit (key) whose height is equal to the currently unrolled height:
5. - - delete it from the combbases map, if it was there also call segments_coinbase_unmine() at that commit and unrolled height.
6. - - delete it from the commits map.
7. - - if used keys feature is enabled call used_key_commit_reorg(key, currently_unrolled_height)
8. - - call merkle_unmine(key)
9. - - also call the block of code in mine.go starting with txleg_mutex.RLock() and ending with txleg_mutex.RUnlock(), probably refactored to a separate function.
10. - - set unwritten = true if unwritten at least 1 commit
11. - don't do the thing below for every commit (key) anymore, but just for every height:
12. - if unwritten && enable_used_key_feature {used_key_height_reorg(reorg_height);}
13. don't do the thing below for every height anymore, but just once for the entire long range reorg:
14. commit_rollback = nil // to be sure
15. commit_rollback_tags = nil // to be sure
16. lazyopen = false // to be sure
17. resetgraph() // to reflow the entire balance graph
18. Truncate from LEVELDB everything above target height using a batch.
19. adios (unlock the 2 mutexes from step 1)


the nice thing is that you will be able to reorg back to any height, calling this new function once, not just 1 block back.

Woah that use of the next_mine/next_download variables is cool. Yea I'll try to hook it up, it might take me a sec to fully understand but it looks good.



Can we hard-code the header for block 481822 into haircomb and just use that? It won't work for regtest though, so we can potentially code an option to pull from genesis for those blocks, but if we keep a hard coded pseudo genesis haircomb block then we can cut down of the data storage and the build time. The only risk is somebody 51%ing the entire BTC chain back to 481821, right?

I used 481824 in my code because that's the first block that a commit ever appears on, but Natasha's Modded BTC pulls from 481822 so it makes sense to start from there.



Makes sense, a million blocks will only add like 40 mb of memory. But key = hash and value = height, or the other way around? The current leveldb is key = height, value = hash.


This is way better than what I was thinking, I figured we'd have to go through and mod all the code to use a new UTXOtag lol.

I'll try to implement a rough version of this and your new pull code over the next little bit, but I want to confirm the new UTXO struct first. Previously you laid out the following:


Do we need to include the CommitPositionInBlock? I'm not sure what the advantage is to including this.

I'm also concerned about the removal of the fork, because I'm not sure why it was implemented as a fork rather than an entire change like you're proposing here, so the unknown is spooky. But we'll be able to test if this is a problem via fingerprinting, so it should be fine to experiment with.

I'm also going to include the block hash in the block commits fingerprint, unless you think there's a problem doing it. This way there's an explicit tying of the commits to the block header itself, it feels better for some reason.

EDIT: I modded your code to be able to operate from high to low for unmining: https://goplay.space/#2_WoPvMPlLQ. I'll finish merging it tomorrow hopefully.

EDIT2: Running a speed test on the merged code.

EDIT3: As it stands the current code is super slow. I think this is because the info pulling and info mining are tied together; if there are a bunch of blocks queued up to be mined, mining them locks all the other downloaders from depositing their payload and pinging the BTC node for more. In order for this to be fast, the BTC node needs to be under 100% load all the time, which it isn't right now.

EDIT4: I modded the code to have separate mining/pulling processes. It still seems slower for some reason, however my pc is acting up and the old version also seems somewhat slower so I can't really trust it, you might want to give it a shot and see how quick it is for you in comparison. Code's up on my github.

I have here a skeleton for the sleep-less channel-less downloading of blocks, could it be integrated? I don't think its a big change.

https://goplay.space/#Yl0k34__XjH

Your explanation about sequential reorg is fine then, let's keep linear.

To finalize the format I think we need to also pull block headers since genesis using getblockheader I think.

Thinking about it, perhaps to simplify things, it would be ok to have a mutex protected global in memory map keyed by blockhash containing the block height.

You know, what you previously did in the level db, but just in memory. In the level db, on disk, there would be the other direction (from height to full header).

Then on startup, the leveldb can be looped over and values cached to memory.

Advice about the migration to the new utxo tag:

1. put the on disk utxo tag struct to the code
2. write a function convert the on disk utxo tag to the old utxo tag (the height is just casted to uint32, based on height before/after fork fill the txnum and outnum or split the output order num uint32 into two uint16 dividing/remaindering by 10000 and return them in the old struct)
2. when downloaded the block, generate the on disk utxo tag and store it next to the commitment.
3. when going to mine the block, don't pass the utxo tag as string but simply as on disk utxo tag struct
4. the strictly_monotonic_vouts_bugfix_fork_height if inside miner_mine_commit_internal can then be removed.
5. all the remaining functions just look at height
6. finally put the old version of the utxo tag to the main map (commits map[[32]byte]utxotag) where the old code does
7. but serialize the new on-disk version of the utxo tag to level db.

There was a bug in the initial release. When I added in the mining redundancy prevention, I forgot to modify the start block to reflect that, so it misses mining the very first comb commit. This will NOT show up in the AES fingerprint, as it doesn't take into account the height of the commit, only the commits themselves and the order they're loaded, but it will give an inaccurate number for how many total comb have been spawned.

I'll have a newer version up in the next couple of days, I want to test out the struct-based reading first. The correction is to just change the number in the following code to 481823, inst4ead of 481824


That I see and I've fixed, but I was wondering about your change from a bool to an incrementing int.



I already swapped over to structs. It feels better using more static typing anyways, so I'll run a few commit-build benchmarks and make sure it doesn't slow anything down. If it does then we can switch back.



Gotcha, that makes sense.


My understanding was the the fork Natasha did fixes the 9999 max by combining the two into one number. If you look at the formatting of commits that have been processed after the fork, IIRC they're all merged to be stored as one large number. Natasha described it here: https://bitcointalk.org/index.php?topic=5195815.140

I may be misunderstanding something about the fix though.


Oh store them together, that makes way more sense lol.



The bisect option makes sense, however the nature of the BTC chain, at least right now, means it's INSANELY likely that a reorg will occur within the fist 2 blocks of the chain edge. I think it makes the most sense to just iterate down the chain, and if the reorg isn't found within the first X iterations then it switches to a bisect search.



This is essentially what the current mining system does. The blocks are requested in incremental order as the counter ticks up, but the order it receives and reads in is not fixed. Unfortunately the chokepoint is not how quickly we can read the blocks, but how quickly the BTC can spit them out at us. Although the funneling method you described seems way better than having a bunch of loops checking a timer over and over again, so if that was your intent then nvm that makes sense lol.

You have it written that the goroutine that makes the call is also the one that does the mining. Do you think that this is better than having one dedicated routine to query the map and mine the blocks, and just have the callers dump their blocks in the map? I guess that it makes sense that you're removing a loop that is essentially doing nothing most of the time, waiting for blocks to come in.

I think I get what you're trying to build here, like building a skeleton of the BTC chain and then filling in the blanks as it goes along. Like I said, unfortunately right now the biggest limiter from what I can tell is BTC's speed at supplying the information. But structurally it makes sense.

yes I am right about the run boolean, there was really a race condition bug, I found it using:

 go build -race

The reader of that boolean must take the mutex too, not just the writer.

Sorry I was wrong, you aren't actually looping over the map[string]interface{}, just lookup one value. Thus it's safe.

Yes we need block height ON DISK to be uint64. This is because there will be fast altcombs (with block time 1 second or faster). If we don't do this, we are just repeating the Nakamoto 32bit timestamp bug. (Bitcoin Timestamp Year 2106 Overflow)


Utxo tag IN MEMORY can stay in it's current format for now. ([2]uint32). Can be bumped to match on disk format later.

I also think that the LEVELDB integers (inside keys) should be actual binary (big endian=starting by zeroes from the left) not in Binary coded decimals.
This will ensure that the transaction counter and output counter (both uint16) will be capped at 65535 not 9999.


Inside leveldb blocks (uint64 keys), you can store both the new 32byte block checksum and block header (80byte) concatenated.

The new 32byte block checksum will be SHA256 of Key 1 CAT Commitment 1 CAT Key 2 Commitment 2 CAT etc (all the keys and previously unseen commitments inside the block)




He=BTC Full node

1. read his best hash. if same as our best hash, end, otherwise start loop:
2. read his height
(optional) if his height is 0, feed him all our block headers (by submitheader) starting from genesis+1 and goto 2 (continue loop).
3. read his best hash, if different than read previously goto 2 (continue loop). else goto 4 (break loop).
4. read our block header at his height+do the header's hash.
5. if our hash at his height == his best hash from steps 1 & 3, reorg to his height, end.
6. if his height was less or equal to ours, launch the seek reorg height-by-height backwards procedure, then reorg to it's result. then fast forward, end.
7. read his hash at our height, if equal to ours top hash, fast forward, end.
8. launch the seek reorg height-by-height backwards procedure, then reorg to it's result, then fast forward, end.




seek reorg height-by-height backwards:
1. keep checking his hash at decreasing height until it matches our hash at that height.
2. return the height of highest match.
note: can be done using bisection just keep trying the central height between "highest match"+1 and "lowest non-match"-1.
if the central height was a match, increase "highest match" integer. If the central height was not a match, decrease "lowest non-match" integer. in both cases set the integer to central height and pick a new central height. Terminate when "highest match"+1 == "lowest non-match". This search is efficient even without goroutines.



fast forward:
1. check his hash at our height +1, +2, +3, ....
2. download that block(s).
3. apply them in order from lowest.
4. if applying any block fails, terminate all goroutines and end.
note: can be done by multiple goroutines, if the goroutine's downloaded block is
at current height +1 she will try applying it. otherwise she puts it into a shared map keyed by height and tries
applying the current height+1 block from the map (if it's there).
once the goroutine successfully applies her lowest block, she inspects the map to see if there's another lowest block.
if yes she applies that block too, if no, she goes to check another hash and download another block.
note 2: no need to have unlimited goroutines, just have like 6 of them and wait for them to apply everything using a wait group.
note 3: works without the use of channels, just needs a mutex protected shared map and two shared integer counters (all protected by the same mutex).

Weird. I get why the fix works, but I think the problem source may be that I put the "finished := false" within the for loop by accident. The reason I delay the break is because if the config doesn't have an empty line at the end, it looks like it'd break before actually parsing the last line. If you move the "finished := false" to before the for loop and remove the break you inserted, does it still cause a problem on Linux? Am I mistaken about the early break not causing problems on config files with no empty final line?

EDIT: I made a mistake that was included in the first version I uploaded to github, I've since corrected it but if you downloaded a version before I did then this might be the issue.
https://github.com/nixed18/combfullui/commit/3074fbd709e9602eeaccd639dfcb63dcad7dee66

If you've still got that "continue" on line 70, that's what causing the permaloop.



My understanding was that, because its creating a new Index instance for each mine_start() to use as a reference, mine_start() instance 1 and mine_start() instance 2 (and all their respective sub-routines) would never be referencing the same run value. Was I incorrect on this?


Yea that'd be bad. I haven't had it happen yet, all my fingerprints (that I've tested at least) have matched up, but I didn't know that that was a possibility. More reading for me lol.

EDIT: It's been a while since I wrote the initial interp code, but from reading it again it doesn't look like it's pulling the relevant TX sections into map[string]interface{}, but that it's pulling them into []interface{}. This was done for both the initial array of TXes in the block, and the array of outputs in the TX (https://github.com/nixed18/combfullui/blob/master/miner.go, lines 420 and 427). Even if I'm right and it's stable as is, it's still probably worth moving over to a struct system; from what I've read it's faster than using map[string]interface{}.

EDIT2: Could I see your struct implementaion? I've been unable to reuse make_bitcoin_call() without having it output an interface/map[string]interface{} value, and at that point I can't convert back to a struct without marshalling and unmarshalling the JSON for a second time. That seems slow.

EDIT3: I was able to do it by unmarshalling the "result" json into another json.RawMessage, then unmarshalling again. No clue if it's faster but it works, and lets make_bitcoin_call() be used for multiple return values, so that's cool.


Gotcha, I didn't know you could do that, that's pretty sick.


This gets into next-step territory; modifying the actual miner_mine_commit_internal() process. Cool.

While building the API to get headers, another thing to consider adding is the option to pull commits in a json format. While I don't know about pulling ALL the commits in one go, considering there's like over 3.5 million of them rofl, pulling them in chunks/pages might be practical. It'd allow other programs (i.e. ClaimVision) to access the commits file locally for use, without having to use an html parser to deal with (127.0.0.1:3232/utxo/).

EDIT: I'm confused about your last bit;

Right now the block hash information is stored to check for reorgs as the value, not the key. Are you referring to value here? Just making sure, but you're suggesting that we toss block hashes and just use the headers to check for reorgs, right? Why store the block height as a 64-bit, wouldn't a 32-bit be fine?

The way I'm reading the commit proposal is as follows

On-Disk Commits

The On-Disk Hashes are currently just

Is the replacement format you're suggesting below?