Pages:
Author

Topic: [BOUNTY] fix mobile WLCj wallet for android - page 3. (Read 7396 times)

full member
Activity: 217
Merit: 100
CEO WINC e. V.
My phone made it past 10k, and is around 17k and syncing right now. I'll continue testing with adb.

Edit: I've fetched up to the second to last block. Was this version built before my last change?

Hmm this apk included the changes in Transaction.java in BlockMergedMining.java and in Block.java

EDIT: I put another .5 BTC and 400 WLC to the bounty
sr. member
Activity: 392
Merit: 268
Tips welcomed: 1CF4GhXX1RhCaGzWztgE1YZZUcSpoqTbsJ
My phone made it past 10k, and is around 17k and syncing right now. I'll continue testing with adb.

Edit: I've fetched up to the second to last block. Was this version built before my last change?
full member
Activity: 217
Merit: 100
CEO WINC e. V.
The error above is from intellij on my phone it is crashing at block 10k already.

Download is here: http://www.winc-ev.org/english/downloads-contact/
sr. member
Activity: 392
Merit: 268
Tips welcomed: 1CF4GhXX1RhCaGzWztgE1YZZUcSpoqTbsJ
That means it ran out of memory. What device are you running it on?

I have a fairly high-end Android I can test it on, to see if it's just a need for resources, or an actual bug. Can you send me an APK file?
full member
Activity: 217
Merit: 100
CEO WINC e. V.
Yes sure,
it is giving me a Java heap space error when running with that new code hm, did i set something wrong?

Code:
New block: 18539:2830f1d3fe790817ff4703373c2689eaa65470553ed254041e673904dfc1655d
04:14:34 13 AbstractBlockChain.add: 6 blocks per second
New block: 18540:c967137ed7a6affd2618b64ae23339e4b1281c5dbc82dac9bea00a3c0a972f86
Peer #1 disconnected: [127.0.0.1]:55889
04:14:36 13 PeerGroup.handlePeerDeath: [127.0.0.1]:55889: Peer died
04:14:36 11 PeerGroup.connectToAnyPeer: Waiting 1000 msec before next connect attempt
04:14:36 13 PeerGroup.handlePeerDeath: [127.0.0.1]:55889: Peer died
04:14:36 13 PeerGroup.handlePeerDeath: Download peer died. Picking a new one.
04:14:36 13 PeerGroup.setDownloadPeer: Unsetting download peer: [127.0.0.1]:55889
04:14:37 11 PeerGroup.connectToAnyPeer: Waiting 2250 msec before next connect attempt
Peer #0 disconnected: [127.0.0.1]:55889
Exception in thread "NioClientManager" java.lang.OutOfMemoryError: Java heap space
at org.bitcoinj.core.Utils.copyOf(Utils.java:457)
sr. member
Activity: 392
Merit: 268
Tips welcomed: 1CF4GhXX1RhCaGzWztgE1YZZUcSpoqTbsJ
Would you mind sharing the logcat and errors at the stage where it crashes? I'm going to try to get this to build and run on my phone soon.
full member
Activity: 217
Merit: 100
CEO WINC e. V.
Awesome! I wish i had a portion of that skill of yours. The last step will be to get the wallet onto the phone. It is stll crashing at 10k, hmm..
sr. member
Activity: 392
Merit: 268
Tips welcomed: 1CF4GhXX1RhCaGzWztgE1YZZUcSpoqTbsJ
Nope, my previous reply was bogus. However, I also got the chain to sync 100% without errors using another fix.

Code:
09:37:39 15 DownloadListener.progress: Chain download 100% done with 0 blocks to go, block date Aug 21, 2015 9:12:56 AM

The issue was in the constructor of AbstractBlockChain$OrphanBlock:

Code:
 if (!shouldVerifyTransactions())
                this.block = block.cloneAsHeader();
            else
                this.block = block;

This optimization is invalid for merged mining since it will wipe out ALL of the merged mining data. A naive fix is to simply change it to this.block = block in ALL cases, but this will keep a bunch of payload data nobody cares around in many cases. There's a better solution that I'm testing right now.

Edit: I've tested my solution and it seems to work properly. To integrate it:

1) Don't change OrphanBlock from the original. Keep that if-statement.

2) Add the following to BlockMergeMined:

Code:
    public BlockMergeMined cloneAsHeader(Block mainBlock){
        BlockMergeMined bmm = new BlockMergeMined(this.params, null, 0, mainBlock);
        bmm.payload = this.payload;
        return bmm;
    }

3) Replace cloneAsHeader in Block with:

Code:
    public Block cloneAsHeader() {
        maybeParseHeader();
        Block block = new Block(params);
        block.nonce = nonce;
        block.prevBlockHash = prevBlockHash.duplicate();
        block.merkleRoot = getMerkleRoot().duplicate();
        block.version = version;
        block.time = time;
        block.difficultyTarget = difficultyTarget;
        block.transactions = null;
        block.hash = getHash().duplicate();
        if(mmBlock!=null)
            block.mmBlock = mmBlock.cloneAsHeader(block);
        return block;
    }

I've tested this with a successful sync up to 22397 (the latest block at the time).

Edit: I've synced up and am able to process new blocks successfully as they come in.
sr. member
Activity: 392
Merit: 268
Tips welcomed: 1CF4GhXX1RhCaGzWztgE1YZZUcSpoqTbsJ
So it seems the merge mining information can not be found mmBlock == null returns true.

I dont know why but the c++ wallet somehow handles the newly generated blocks in another way than namecoin and devcoin.

Are you sure that the last block is specifically merge-mined? Having merge mined data isn't a requirement for a block to be valid.
full member
Activity: 217
Merit: 100
CEO WINC e. V.
So it seems the merge mining information can not be found mmBlock == null returns true.

I dont know why but the c++ wallet somehow handles the newly generated blocks in another way than namecoin and devcoin.
full member
Activity: 217
Merit: 100
CEO WINC e. V.
Cool, i would have thought that the transactionversion is not set before parsing, but as long as it works  Grin.

The difficulty fail is really unexpected. Why is it only the last block? Totally confusing. The check for the merged mining work is copy-paste from devcoin (which copy-pasted it from namecoin) and no rocket science. As you said it is taking the blockhash from the parent chain as POW. So i would exclude mistakes in the c++ wallet if there are no further hints.

So either the diffucultyfilter is wrong for all the blocks (or all merge miningblocks) but the last,
or the difficultyfilter is correct but fails to validate the most current block.

To me the last option is the most probalbe which throws up the question:
what is special about the current block that hinders the wallet to get the right hash?

-----------------------------------------------------------------
Ok, of course it is set before parsing, but i thought the "new Transaction" in BlockMergeMinedPayload is setting it to Version 2 and that is the problem
staff
Activity: 3458
Merit: 6793
Just writing some code
sr. member
Activity: 392
Merit: 268
Tips welcomed: 1CF4GhXX1RhCaGzWztgE1YZZUcSpoqTbsJ
I managed to sync successfully up to height 22103, by modifying org.bitcoinj.core.Transaction.

Edit: Now synced to 22194 before crashing. I think I've managed to sync almost all of the chain, however it's rejecting the last block with an invalid difficulty. The C++ client accepts it, and the two clients are in agreement about difficulty.

The parse() method in that class should now be:

Code:
 @Override
    void parse() throws ProtocolException {

        if (parsed)
            return;

        cursor = offset;

        version = readUint32();
        optimalEncodingMessageSize = 4;

        // First come the inputs.
        long numInputs = readVarInt();
        optimalEncodingMessageSize += VarInt.sizeOf(numInputs);
        inputs = new ArrayList((int) numInputs);
        for (long i = 0; i < numInputs; i++) {
                TransactionInput input = new TransactionInput(params, this, payload, cursor, parseLazy, parseRetain);
                inputs.add(input);
                long scriptLen = readVarInt(TransactionOutPoint.MESSAGE_LENGTH);
                optimalEncodingMessageSize += TransactionOutPoint.MESSAGE_LENGTH + VarInt.sizeOf(scriptLen) + scriptLen + 4;
                cursor += scriptLen + 4;
        }
        // Now the outputs
        long numOutputs = readVarInt();
        optimalEncodingMessageSize += VarInt.sizeOf(numOutputs);
        outputs = new ArrayList((int) numOutputs);
        for (long i = 0; i < numOutputs; i++) {
            TransactionOutput output = new TransactionOutput(params, this, payload, cursor, parseLazy, parseRetain);
            outputs.add(output);
            long scriptLen = readVarInt(8);
            optimalEncodingMessageSize += 8 + VarInt.sizeOf(scriptLen) + scriptLen;
            cursor += scriptLen;
        }
        lockTime = readUint32();
        optimalEncodingMessageSize += 4;
        if(this.version==2) {
            refHeight = readUint32();
            optimalEncodingMessageSize += 4 + 4;
        }
        length = cursor - offset;
    }

Here's the new error:

Code:
org.bitcoinj.core.VerificationException: Could not verify block f26842324aca22db7c16ddf087277ee99117d0fa29ca5d5f6eaaaf4d42d9d8f0
v1114369 block:
   previous block: c85bfc960881fa6cfa833feef10e2842fe7b56f56aa962fdd0e2300b59c4da3e
   merkle root: 505a544293220d73c2f1b1ca3285906828fc8abcc39aa41a3f4155fc347b3c3a
   time: [1440012311] Wed Aug 19 15:25:11 EDT 2015
   difficulty target (nBits): 436643261
   nonce: 0
   Merged-mining info:
      version: v1114369
      time: [Wed Aug 19 15:25:11 EDT 2015] Wed Aug 19 15:25:11 EDT 2015
      difficulty target (nBits): 436643261
      nonce: 0
      Network: org.worldleadcurrency.production
      parent block coin base transaction:
  679a73c3467721d6fc3ffdcf6865da6b0283d50664b1d2914d67b111bbe072fd: Unknown confidence level.
     == COINBASE TXN (scriptSig PUSHDATA(3)[a4a705] PUSHDATA(44)[fabe6d6d7f15c02dd05c3eacefde83dd74b0f9b5ed7155c9ec92ef280acdda2bacd1f5c32000000000000000])  (scriptPubKey DUP HASH160 PUSHDATA(20)[3d47986f727402feb3df5134dff583af4ace8074] EQUALVERIFY CHECKSIG)

      coinbase link:
          hash of parent block header: 000000000000023a97f467afb051ac1480e27bed9788e8fdb9ac2d892614f551
      parent block header:
v3 block:
   previous block: 00000000000000000759c455cd9b9402a461885fe76e0261092a5cd7a575038d
   merkle root: cac502a2d3a6865430f959b445ff563b4bcd49f60ab9ad8579d6ab54ff829108
   time: [1440012417] Wed Aug 19 15:26:57 EDT 2015
   difficulty target (nBits): 404020484
   nonce: 3237608803


   with 1 transaction(s):
  505a544293220d73c2f1b1ca3285906828fc8abcc39aa41a3f4155fc347b3c3a: Unknown confidence level.
     == COINBASE TXN (scriptSig PUSHDATA(4)[bda5061a] PUSHDATA(1)[01] 2)  (scriptPubKey PUSHDATA(33)[02bc41065c462d75844cf77624b712a29064cd21630e9b8c3c68990e4881a9b58e] CHECKSIG)

at org.bitcoinj.core.AbstractBlockChain.add(AbstractBlockChain.java:286)
at org.bitcoinj.core.Peer.processBlock(Peer.java:872)
at org.bitcoinj.core.Peer.processMessage(Peer.java:360)
at org.bitcoinj.core.PeerSocketHandler.receiveBytes(PeerSocketHandler.java:178)
at org.bitcoinj.net.ConnectionHandler.handleKey(ConnectionHandler.java:217)
at org.bitcoinj.net.NioClientManager.handleKey(NioClientManager.java:75)
at org.bitcoinj.net.NioClientManager.run(NioClientManager.java:111)
at com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:60)
at com.google.common.util.concurrent.Callables$3.run(Callables.java:93)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.bitcoinj.core.VerificationException: Hash is higher than target: 13e6b9f88b5a8bd9a36adb475588d6108d8594b8eb37c5122ae70d3c4beface7 vs 6a5bd0000000000000000000000000000000000000000000000
at org.bitcoinj.core.Block.checkProofOfWork(Block.java:766)
at org.bitcoinj.core.Block.verifyHeader(Block.java:886)
at org.bitcoinj.core.AbstractBlockChain.add(AbstractBlockChain.java:397)
at org.bitcoinj.core.AbstractBlockChain.tryConnectingOrphans(AbstractBlockChain.java:825)
at org.bitcoinj.core.AbstractBlockChain.add(AbstractBlockChain.java:424)
at org.bitcoinj.core.AbstractBlockChain.add(AbstractBlockChain.java:275)
... 9 more

sr. member
Activity: 392
Merit: 268
Tips welcomed: 1CF4GhXX1RhCaGzWztgE1YZZUcSpoqTbsJ
I might have an idea on how this could be done. I think there should be extra fields to differentiate the merged chain used, probably a version or something. I don't think we can just check an expected value in a neat way. I'm going to try it shortly, and see if my idea works in practice.
full member
Activity: 217
Merit: 100
CEO WINC e. V.
Why is reading varints for the last thousands of blocks OK? Is the format different for that specific piece of data? Or are the parser reads misordered, misaligned to the actual structure, or calling for a varint where the format isn't a varint?
The data for block 11418 is different because it is merge mined, which is probably causing the problem. Actually, it might not be VarInt but the offset passed into it. After a little more analysis, it looks like the offset is not set correctly, which is causing VarInt to return incorrect numbers. I could be wrong.

https://github.com/FreicoinAlliance/freicoinj/commit/8c27b562cd814f2a6a026e76f0e11bcc0960ab4d

As you can see in this hash the offset for transactions has to be changed because of the extra property "refHeight" of the transactions of the coin.

Merged mining with freicoin as parent chain is working ok (Blocks 1 - 1405).

When merge mining with bitcoin as parent starts, the offset is suddenly wrong,
so maybe the wallet is expecting also the parent chain to have refHeight in its transactions?

I would like to test it somehow but right now i have no clue how to implement a solution.

Maybe something like:

If (length !="expected value")
    length = "value without refHeight";   
staff
Activity: 3458
Merit: 6793
Just writing some code
I think I may have found where the problem is, but I will need to do some testing. This is just from looking at the code. I think it has to do with VarInt and the readVarInt() method in message.java. This method is called to get the length of the array and I think it isn't doing something correctly which causes the length to be too small. I need to check this out with debugging though.

Hexafraction, can you send me the main class that you are using?

readVarInt isn't the culprit, as far as I can tell. It's being called when the array pointer is already pointing in an invalid location. I'll pastebin the main class I'm using shortly.

Edit: http://paste.ubuntu.com/12112739/
It's nothing fancy, but it fetches blocks and gives me a debuggable program. Chain is static so the debugger can easily access it for expression evaluation and conditional breakpoints.
I think the error is definitely in VarInt or readVarInt because at line 259 of Block.java returns number of transactions to be 132 when there is only 1 and in line 590 of Transaction.java it also returns 132 from readVarInt(). The specific ArrayIndexOutOfBounds error is caused by the length being too long, which is also set by ReadVarInt(). I am going to take a look at VarInt to see whether I can find the problem.

Why is reading varints for the last thousands of blocks OK? Is the format different for that specific piece of data? Or are the parser reads misordered, misaligned to the actual structure, or calling for a varint where the format isn't a varint?
The data for block 11418 is different because it is merge mined, which is probably causing the problem. Actually, it might not be VarInt but the offset passed into it. After a little more analysis, it looks like the offset is not set correctly, which is causing VarInt to return incorrect numbers. I could be wrong.
sr. member
Activity: 392
Merit: 268
Tips welcomed: 1CF4GhXX1RhCaGzWztgE1YZZUcSpoqTbsJ
I think I may have found where the problem is, but I will need to do some testing. This is just from looking at the code. I think it has to do with VarInt and the readVarInt() method in message.java. This method is called to get the length of the array and I think it isn't doing something correctly which causes the length to be too small. I need to check this out with debugging though.

Hexafraction, can you send me the main class that you are using?

readVarInt isn't the culprit, as far as I can tell. It's being called when the array pointer is already pointing in an invalid location. I'll pastebin the main class I'm using shortly.

Edit: http://paste.ubuntu.com/12112739/
It's nothing fancy, but it fetches blocks and gives me a debuggable program. Chain is static so the debugger can easily access it for expression evaluation and conditional breakpoints.
I think the error is definitely in VarInt or readVarInt because at line 259 of Block.java returns number of transactions to be 132 when there is only 1 and in line 590 of Transaction.java it also returns 132 from readVarInt(). The specific ArrayIndexOutOfBounds error is caused by the length being too long, which is also set by ReadVarInt(). I am going to take a look at VarInt to see whether I can find the problem.

Why is reading varints for the last thousands of blocks OK? Is the format different for that specific piece of data? Or are the parser reads misordered, misaligned to the actual structure, or calling for a varint where the format isn't a varint?
staff
Activity: 3458
Merit: 6793
Just writing some code
I think I may have found where the problem is, but I will need to do some testing. This is just from looking at the code. I think it has to do with VarInt and the readVarInt() method in message.java. This method is called to get the length of the array and I think it isn't doing something correctly which causes the length to be too small. I need to check this out with debugging though.

Hexafraction, can you send me the main class that you are using?

readVarInt isn't the culprit, as far as I can tell. It's being called when the array pointer is already pointing in an invalid location. I'll pastebin the main class I'm using shortly.

Edit: http://paste.ubuntu.com/12112739/
It's nothing fancy, but it fetches blocks and gives me a debuggable program. Chain is static so the debugger can easily access it for expression evaluation and conditional breakpoints.
I think the error is definitely in VarInt or readVarInt because at line 259 of Block.java returns number of transactions to be 132 when there is only 1 and in line 590 of Transaction.java it also returns 132 from readVarInt(). The specific ArrayIndexOutOfBounds error is caused by the length being too long, which is also set by ReadVarInt(). I am going to take a look at VarInt to see whether I can find the problem.
staff
Activity: 3458
Merit: 6793
Just writing some code
How does the coin format the merge mining data in the block? Also, is there a block explorer I can use for this coin? That would make it much easier to figure this whole thing out.

The Blockexplorer can be found at: 109.73.173.119:81 (the balances are in satoshi not in wlc).

I try to find out more about the merge mining data in the block, can you specify a little bit what information you need?
Thanks. The blockexplorer helps. It gives me the data I need, which is the raw format of the blocks in a human-readable format.
sr. member
Activity: 392
Merit: 268
Tips welcomed: 1CF4GhXX1RhCaGzWztgE1YZZUcSpoqTbsJ
How does the coin format the merge mining data in the block? Also, is there a block explorer I can use for this coin? That would make it much easier to figure this whole thing out.

I'm not sure, but https://github.com/WorldLeadCurrency/WLC/blob/master/src/auxpow.cpp might help you. I can't read others' C++ very well, however.
Pages:
Jump to: