Qora | 100% POS | Assets | Names | Voting | Open Source - page 27.

calvinstm

full member

Activity: 229

Merit: 101

Quote from: Vrontis on February 23, 2015, 10:18:01 AM

I have also asked MapDB founder about that and he told me to check MapDB Code for thread interrupts.

Also a new version of MapDB has been released

Nice job! Thanks for doing this! Grin

Okay, I just wanna quickly summarize and tell you what we need to do here:

1. We need to come up with a new workaround that does NOT involve the use of Memory Mapped Files since Memory Mapped Files only works on 64-bit system. Using MMF is not an option since we need qora to work on all systems including 32-bit. We need to find out what's causing the thread to get interrupted before all file I/O operations are completed.

rlh's comment got me thinking, and I think that it's possible the client does not exit cleanly (FC was closed while the changes to database were not fully committed). Since I've only run Qora on 64-bit machines, it's difficult for me to replicate Poloniex's problems and do a debug. Someone definitely needs to follow up on this and identify which calls are causing the problems.

2. I hope we can incorporate the following changes before we roll out the next version of Qora wallet - 1) Skerberus' Installer, 2) MMF workaround (for 32-bit system' sake), 3) the new Wallet design Pondsea was working on, 4) encoding method as suggested by deMap.

Keep up the good work, guys! Grin

calvinstm

full member

Activity: 229

Merit: 101

Quote from: delulo on February 23, 2015, 04:21:59 PM

Seem like you got everything out you could

Thanks for the info!

Sorry, but currently there is no documentation for Qora's forging algorithm. However, like what Vrontis said, the best place to start is Qora's Block Generator (BlockGenerator.java). Half way through the code, you will see how the new block is generated.

I think someone should definitely write a paper on that! Grin

Vrontis

sr. member

Activity: 351

Merit: 250

Quote from: delulo on February 23, 2015, 04:21:59 PM

Seem like you got everything out you could

Thanks for the info!

I am not able to read and understand the POS from the source code.
If anyone is capable enough it would be great to have a review.

delulo

sr. member

Activity: 441

Merit: 250

Quote from: Vrontis on February 23, 2015, 03:53:34 PM

Quote from: delulo on February 23, 2015, 02:42:44 PM

Quote from: Vrontis on February 23, 2015, 09:14:13 AM

Quote from: delulo on February 23, 2015, 08:39:05 AM

I'd like to know about the mechanics / incentives structures of the Qora POS. Any documentation about it?

Quote from: qora

A Proof-of-Stake algorithm that is different from anything that is described here https://en.bitcoin.it/wiki/Proof_of_Stake.

The algorithm will allow accounts to start generating blocks after only 10 confirmations.
Each account will also know at exactly which point they will generate the next block.

Source

That is a little little

Is there not more information?

Below are quoted all that I have found from the dev about POS

Quote from: qora on March 19, 2014, 07:46:04 AM

A Proof-of-Stake algorithm that is different from anything that is described here https://en.bitcoin.it/wiki/Proof_of_Stake.

The algorithm will allow accounts to start generating blocks after only 10 confirmations.
Each account will also know at exactly which point they will generate the next block.

Quote from: qora on April 08, 2014, 12:15:13 PM

To generate blocks you will need to unlock your wallet. the POS relies on the fact that the client has access to a somewhat reliable time source. Basicly it uses time but does not have to rely on a centralized time server to work. Currently it still uses NTP but later on this will be changed so it works completely decentralized.

Also a short review from a core dev of NXT

Quote from: kushti on December 23, 2014, 09:21:07 AM

Spent a little bit of my time on Qora sources during last weekend and even made Scala version of QORA's BlockGenerator class: https://gist.github.com/kushti/605dfe4f17d0dd75cedf (220 lines of code instead of original 400), also submitted the bug on the way: https://github.com/Qora/Qora/issues/10 . Haven't investigated forging algo in details, but that's definitely not TF. Read this epic topic about TF

https://nxtforum.org/general/nxt-forging-algorithm-simulating-approach/

Qora is used on the Forging Simulation Tool created for the Consensus research.

Seem like you got everything out you could

Thanks for the info!

Vrontis

sr. member

Activity: 351

Merit: 250

Quote from: delulo on February 23, 2015, 02:42:44 PM

Quote from: Vrontis on February 23, 2015, 09:14:13 AM

Quote from: delulo on February 23, 2015, 08:39:05 AM

I'd like to know about the mechanics / incentives structures of the Qora POS. Any documentation about it?

Quote from: qora

A Proof-of-Stake algorithm that is different from anything that is described here https://en.bitcoin.it/wiki/Proof_of_Stake.

The algorithm will allow accounts to start generating blocks after only 10 confirmations.
Each account will also know at exactly which point they will generate the next block.

Source

That is a little little

Is there not more information?

Below are quoted all that I have found from the dev about POS

Quote from: qora on March 19, 2014, 07:46:04 AM

A Proof-of-Stake algorithm that is different from anything that is described here https://en.bitcoin.it/wiki/Proof_of_Stake.

The algorithm will allow accounts to start generating blocks after only 10 confirmations.
Each account will also know at exactly which point they will generate the next block.

Quote from: qora on April 08, 2014, 12:15:13 PM

To generate blocks you will need to unlock your wallet. the POS relies on the fact that the client has access to a somewhat reliable time source. Basicly it uses time but does not have to rely on a centralized time server to work. Currently it still uses NTP but later on this will be changed so it works completely decentralized.

Also a short review from a core dev of NXT

Quote from: kushti on December 23, 2014, 09:21:07 AM

Spent a little bit of my time on Qora sources during last weekend and even made Scala version of QORA's BlockGenerator class: https://gist.github.com/kushti/605dfe4f17d0dd75cedf (220 lines of code instead of original 400), also submitted the bug on the way: https://github.com/Qora/Qora/issues/10 . Haven't investigated forging algo in details, but that's definitely not TF. Read this epic topic about TF

https://nxtforum.org/general/nxt-forging-algorithm-simulating-approach/

Qora is used on the Forging Simulation Tool created for the Consensus research.

delulo

sr. member

Activity: 441

Merit: 250

Quote from: Vrontis on February 23, 2015, 09:14:13 AM

Quote from: delulo on February 23, 2015, 08:39:05 AM

I'd like to know about the mechanics / incentives structures of the Qora POS. Any documentation about it?

Quote from: qora

A Proof-of-Stake algorithm that is different from anything that is described here https://en.bitcoin.it/wiki/Proof_of_Stake.

The algorithm will allow accounts to start generating blocks after only 10 confirmations.
Each account will also know at exactly which point they will generate the next block.

Source

That is a little little

Is there not more information?

Vrontis

sr. member

Activity: 351

Merit: 250

Quote from: calvinstm on February 22, 2015, 11:01:23 PM

Quote from: DeMap on February 22, 2015, 11:37:36 AM

I am asking because i have googled both errors (ClosedByInterruptException, ClosedChannelException) together with the term MapDB and i have found several websites
e.g.:
https://groups.google.com/forum/#!topic/mapdb/9MBNRFbX3g4
https://github.com/jankotek/MapDB/issues/352

that claim memory mapped files (mmapFileEnable()) might solve both problems, but they only work on 64 bit VMs.
So i looked into the sourcecode relase and found:
.mmapFileEnableIfSupported()
in https://github.com/Qora/Qora/blob/master/Qora/src/database/DBSet.java

Then i looked at the decompiled v18 but didn't find it there.

mmapFileEnable() - Enables Memory Mapped Files, much faster storage option. However on 32bit JVM this mode could corrupt your DB thanks to 4GB memory addressing limit.

You genius! Grin

I think you actually solved the mystery!

Very good job, DeMap! Wink

PS: I've been running Qora on 64-bit windows (64-bit JVM), and I can attest that my Qora client never once encountered any problem (No exception thrown)... I'm pretty sure it's a JVM problem.

----------------

PS 2: If this is the case, what Poloniex needs to do is to compile a new Qora wallet with memory map file always enabled, and switch to 64-bit JVM. I think someone needs to contact Poloniex or Tristen and let them know about the solution.

Indeed, this seems to be a workaround that solves this problem

https://groups.google.com/d/msg/mapdb/9MBNRFbX3g4/otD5fYEQBM0J

Quote

ClosedChannelException happens if one of the threads is interrupted while doing IO. The FileChannel gets closed on interruption and all future IOs will throw CCE. There is workaround to use memory-mapped files (mmapFileEnable()), but they only work on 64bit systems.

So either enable mmap or investigate what is interrupting your threads.

https://github.com/jankotek/MapDB/issues/352#issuecomment-48128816

Quote

Some threads are interrupted while reading/writing from FileChannel. As result FC is closed and always throws ClosedByInterruptException. That is documented behavior of FileChannel, and there is not much I can do without sacrificing performance.

So my suggestion is to stop interrupting threads while they are reading/writing.

Other solution is to use memory mapped files (mmapFileEnable()). Perhaps they are not affected by this

https://github.com/jankotek/MapDB/issues/384#issuecomment-68485183

Quote

I would recommend to use mmap files as
alternative to prevent this.

Also make sure that no thread is being
interrupted while doing IO. That would close
FileChannel

mmapFileEnable

Quote

public DBMakerT mmapFileEnable()
Enables Memory Mapped Files, much faster storage option. However on 32bit JVM this mode could corrupt your DB thanks to 4GB memory addressing limit.

You may experience java.lang.OutOfMemoryError: Map failed exception on 32bit JVM, if you enable this mode.

mmapFileEnableIfSupported

Quote

public DBMakerT mmapFileEnableIfSupported()
Enable Memory Mapped Files only if current JVM supports it (is 64bit)

I have also asked MapDB founder about that and he told me to check MapDB Code for thread interrupts.

Also a new version of MapDB has been released

Vrontis

sr. member

Activity: 351

Merit: 250

Quote from: delulo on February 23, 2015, 08:39:05 AM

I'd like to know about the mechanics / incentives structures of the Qora POS. Any documentation about it?

Quote from: qora

A Proof-of-Stake algorithm that is different from anything that is described here https://en.bitcoin.it/wiki/Proof_of_Stake.

The algorithm will allow accounts to start generating blocks after only 10 confirmations.
Each account will also know at exactly which point they will generate the next block.

Source

delulo

sr. member

Activity: 441

Merit: 250

Quote from: calvinstm on February 23, 2015, 04:02:36 AM

Quote from: delulo on February 23, 2015, 03:59:26 AM

the qora wiki @ https://wiki.qora.org/ isn't working?

Qora had a forum and a wiki set up by the dev, but as far as I'm aware, they're both down at the moment..

However, most info in the Wiki can be found in the API guide.. If there is something you need to know, just post them here!

Another good source of info is Qora's source code at github.. The source is pretty organized.. Check it out if you haven't!

-----------------------------------

A little bit of intro to Qora's API --

There are two ways to access Qora's API --

1. Through the built-in debug console.
2. Through jetty's html (JSON) interface using your browser or make calls from inside your applications...

Grin

I'd like to know about the mechanics / incentives structures of the Qora POS. Any documentation about it?

rlh

hero member

Activity: 804

Merit: 1004

I think this is the bug. I vaguely recall Qora mentioning that the issue (or multiple issues) were related to MapDb, and that he needed to refactor some code so that hard-kills of the app wouldn't break the db.

wizzardTim

legendary

Activity: 1708

Merit: 1000

Reality is stranger than fiction

Quote from: calvinstm on February 22, 2015, 11:01:23 PM

Quote from: DeMap on February 22, 2015, 11:37:36 AM

I am asking because i have googled both errors (ClosedByInterruptException, ClosedChannelException) together with the term MapDB and i have found several websites
e.g.:
https://groups.google.com/forum/#!topic/mapdb/9MBNRFbX3g4
https://github.com/jankotek/MapDB/issues/352

that claim memory mapped files (mmapFileEnable()) might solve both problems, but they only work on 64 bit VMs.
So i looked into the sourcecode relase and found:
.mmapFileEnableIfSupported()
in https://github.com/Qora/Qora/blob/master/Qora/src/database/DBSet.java

Then i looked at the decompiled v18 but didn't find it there.

mmapFileEnable() - Enables Memory Mapped Files, much faster storage option. However on 32bit JVM this mode could corrupt your DB thanks to 4GB memory addressing limit.

You genius! Grin

I think you actually solved the mystery!

Very good job, DeMap! Wink

PS: I've been running Qora on 64-bit windows (64-bit JVM), and I can attest that my Qora client never once encountered any problem (No exception thrown)... I'm pretty sure it's a JVM problem.

----------------

PS 2: If this is the case, what Poloniex needs to do is to compile a new Qora wallet with memory map file always enabled, and switch to 64-bit JVM. I think someone needs to contact Poloniex or Tristen and let them know about the solution.

Thank you DeMap! It is very important for qora to have poloniex up and running.

I tried to give some ideas in the slackroom, regarding the ways to reproduce the scenario and help find the bug, but if this is the bug, you saved all a lot of time for the community.

Contrats!

calvinstm

full member

Activity: 229

Merit: 101

Quote from: delulo on February 23, 2015, 03:59:26 AM

the qora wiki @ https://wiki.qora.org/ isn't working?

Qora had a forum and a wiki set up by the dev, but as far as I'm aware, they're both down at the moment..

However, most info in the Wiki can be found in the API guide.. If there is something you need to know, just post them here!

Another good source of info is Qora's source code at github.. The source is pretty organized.. Check it out if you haven't!

-----------------------------------

A little bit of intro to Qora's API --

There are two ways to access Qora's API --

1. Through the built-in debug console.
2. Through jetty's html (JSON) interface using your browser or make calls from inside your applications...

Grin

calvinstm

full member

Activity: 229

Merit: 101

Quote from: HalFinneysBrain on February 23, 2015, 03:49:04 AM

Thanks!

Is the problem that it has connecting to peers (usually fails to connect) normal?

Nice! Glad to be of service

Yes, I had the same problem with peers too (it used to take about 5 to 10 secs before I got any connection)...

Here is how to fix it --

So the problem is that some of the nodes in the known peer list are dead. Open the file Settings.json in notepad and you will find a list of known peers. Your job is to remove the ones that are dead.

So go back to your client - go to File -> Debug -> Logger (last tab)... If you look at the log, it will show you which nodes have failed (take note of those)..

Now, go back to your Settings.json and remove the peers that are dead..

Now, someone might have a list of new peers but I don't know where to find it.. Maybe someone can supply the list to us??

Let me know if you have other questions! Grin

delulo

sr. member

Activity: 441

Merit: 250

the qora wiki @ https://wiki.qora.org/ isn't working?

HalFinneysBrain

full member

Activity: 201

Merit: 100

Quote from: calvinstm on February 22, 2015, 11:07:02 PM

Quote from: HalFinneysBrain on February 22, 2015, 09:23:16 PM

I also had a problem where I closed QORA and then the next time I opened it, it gave and error that QORA had not been closed correctly, and wouldnt open. I had to reinstall and regenerate my wallet from the key. Why did this occur?

Thanks!

Okay, what you need to do is to always close the Qora client first (File -> Quit) and not close the command prompt window.

Thanks!

Is the problem that it has connecting to peers (usually fails to connect) normal?

Skerberus

hero member

Activity: 913

Merit: 664

Quote from: HalFinneysBrain on February 22, 2015, 09:23:16 PM

I'm running the QORA wallet (v18).

A window appears which says "connecting to peer" and an IP address, and then it almost always says "fail to connect to "
Every once in a while it actually connects.

Is this normal or am I doing something wrong?

I also had a problem where I closed QORA and then the next time I opened it, it gave and error that QORA had not been closed correctly, and wouldnt open. I had to reinstall and regenerate my wallet from the key. Why did this occur?

Thanks!

If you get this error again, you just have to delete the data folder! I made a pull request in order to do this automatically, but by now it is not integrated into the qora code.

calvinstm

full member

Activity: 229

Merit: 101

Quote from: HalFinneysBrain on February 22, 2015, 09:23:16 PM

I also had a problem where I closed QORA and then the next time I opened it, it gave and error that QORA had not been closed correctly, and wouldnt open. I had to reinstall and regenerate my wallet from the key. Why did this occur?

Thanks!

Okay, what you need to do is to always close the Qora client first (File -> Quit) and not close the command prompt window.

calvinstm

full member

Activity: 229

Merit: 101

Quote from: DeMap on February 22, 2015, 11:37:36 AM

I am asking because i have googled both errors (ClosedByInterruptException, ClosedChannelException) together with the term MapDB and i have found several websites
e.g.:
https://groups.google.com/forum/#!topic/mapdb/9MBNRFbX3g4
https://github.com/jankotek/MapDB/issues/352

that claim memory mapped files (mmapFileEnable()) might solve both problems, but they only work on 64 bit VMs.
So i looked into the sourcecode relase and found:
.mmapFileEnableIfSupported()
in https://github.com/Qora/Qora/blob/master/Qora/src/database/DBSet.java

Then i looked at the decompiled v18 but didn't find it there.

mmapFileEnable() - Enables Memory Mapped Files, much faster storage option. However on 32bit JVM this mode could corrupt your DB thanks to 4GB memory addressing limit.

You genius! Grin

I think you actually solved the mystery!

Very good job, DeMap! Wink

PS: I've been running Qora on 64-bit windows (64-bit JVM), and I can attest that my Qora client never once encountered any problem (No exception thrown)... I'm pretty sure it's a JVM problem.

----------------

PS 2: If this is the case, what Poloniex needs to do is to compile a new Qora wallet with memory map file always enabled, and switch to 64-bit JVM. I think someone needs to contact Poloniex or Tristen and let them know about the solution.

HalFinneysBrain

full member

Activity: 201

Merit: 100

I'm running the QORA wallet (v18).

A window appears which says "connecting to peer" and an IP address, and then it almost always says "fail to connect to "
Every once in a while it actually connects.

Is this normal or am I doing something wrong?

I also had a problem where I closed QORA and then the next time I opened it, it gave and error that QORA had not been closed correctly, and wouldnt open. I had to reinstall and regenerate my wallet from the key. Why did this occur?

Thanks!

DeMap

newbie

Activity: 9

Merit: 0

Quote from: Vrontis on February 21, 2015, 08:05:12 PM

Quote from: DeMap on February 21, 2015, 07:19:41 PM

Quote from: Vrontis on February 21, 2015, 12:56:27 PM

Yes, poloniex had tested their wallet recently but still crashes.

Are they running the precompiled V18 version that Qora released or are they using a compiled version of the released source code?

Are they using a 64 bit Java-VM or a 32 bit Java-VM?

I have reached out poloniex to let them know that there will be a new wallet version and it might be possible to work also on their wallet issue, but I noted that there is not any guarantees that it will be fixed.Once the new wallet version is ready we will try to simulate the problem into a testnet to see if it can be re produced.Then we might work on it and try to solve it.Your questions (hopefully among others) will be passed to poloniex once we reach that level.
If you can contribute to this then please do it.We've got slack that hosting our discussions, we recently launched an IRC channel and there is a skype room as well between me, vbcs and skerberus.Also here as well is a place to host those questions.

I am asking because i have googled both errors (ClosedByInterruptException, ClosedChannelException) together with the term MapDB and i have found several websites
e.g.:
https://groups.google.com/forum/#!topic/mapdb/9MBNRFbX3g4
https://github.com/jankotek/MapDB/issues/352

that claim memory mapped files (mmapFileEnable()) might solve both problems, but they only work on 64 bit VMs.
So i looked into the sourcecode relase and found:
.mmapFileEnableIfSupported()
in https://github.com/Qora/Qora/blob/master/Qora/src/database/DBSet.java

Then i looked at the decompiled v18 but didn't find it there.

Topic: Qora | 100% POS | Assets | Names | Voting | Open Source - page 27. (Read 113804 times)