Topic: HEAT Discussion and Technical info - page 36. (Read 61463 times)

$64K question: Would I earn more HEAT leasing it to your mega-pool than running my own node?

I am only aware of the OrgiOrg pool, and the one I am offering. There are some accounts in heatwallet which contain the name of "pool" but none of them are actually containing multiple lessors.

Thus far heatpool.org is operating very stable, but using a manual calculation, awaiting the release of microservices to gradually move over to automation. You can at least keep an overview of the open balance of all lessors at the website of heatpool.org, and I have thus far processed all payments within a reasonable period of time, most often 24 hours. If a lease period has ended without extension I will also process the payment asap. Any open balance will also be included in your effective stake in the pool, so even with a later payment none of your rewards are "lost".

I am still surprised (and delighted of course!) to see heatpool as the #2 forger in the total list of forgers at heatnodes.org. The current effective balance is around 1.5M HEAT, with most of the times around 35 lessors.

it's my forging pool ... no lessors until now 

but the 0%-offer with automatically payout was extended till end of march 2018 (also look here: http://heatledger.net/index.php?topic=137.msg726#msg726)

Anyone have an automatic payout pool available or try this one?

We should contact coinmarketcap and let them know about supply changes and announcement page change.
https://bitcointalk.org/index.php?topic=199685.5020

This is starting to get very interesting. Thanks Dennis for your explanations and development updates.

I've heard HEAT is going to be fully open source at some point. Any estimates when source will be released?

Hi,

I'd like to address some questions about the specifics of the `2000 tps` test.

Lets start with the specs of the PC mentioned before, the one on which the test was performed;

This would be my daily work horse PC:

   OS: Ubuntu 16.x
   RAM: 16GB
   CPU: Intel Core i5
   HD: 500GB SSD

Now about the test setup.

The HEAT server tested actually runs in my Eclipse Java IDE, this as opposed to running running HEAT server from the command line (its own JVM).
Running in Eclipse IDE, and in our case in DEBUG mode is quite the limiting factor i've experienced.
Running HEAT from the command line does not have the burden of also having to run the full Eclipse Java IDE coding environment as well as who knows what Eclipse is doing with regards to breakpoints and all and being able to pauze execution and all.

We have not yet tested this while HEAT runs from the command line, I expect it to be quite faster in that case.

Now about the websocket client app.

With the help of our newly beloved AVRO binary encoding stack we have been able to generate, sign and store to a file our 500,000 transactions. This process takes a while, a few minutes at least. But I dont think this matters much since in a real-life situation with possibly 100,000s of users the cost of creating and signing the transactions is divided over all those users.

The client app was written in Java and opens a Websocket connection to the HEAT server, since both are running on the same machine we would use the localhost address.

Now you might say; "Hey! wait a minute. Isn't localhost really fast? Isn't that cheating?"

The short awnser.. "No! And you might fail to understand what's important here."

While its absolutely true that localhost is much faster than your average external network, what should be obvious here is what levels of bandwidth we are talking about. I suppose anyone reading this has downloaded a movie before, be that over torrent or maybe streamed from the network. Well there is your proof that testing this over localhost has zero meaning here.

Your PC while downloading a movie or the youtube server streaming to you and probably 1000s of others the most recent PewDiePie video will process MUCH MUCH more data than our little test here.

One transaction in binary form is about 120 bytes in size, times 2000 and you'll need a network that has to support 240KB data transferred a second. Not sure what internet connections are normal in your country, but it seems here in Holland we can get 400Mbit connections straight to our homes, talking about standard consumer speeds here (looked it up just now).

To put that in perspective, 240KB a second is about 1/200th the amount of bandwidth you get with your 400Mbit(50MB) connection. You should understand by now that the network is not the bottle neck, its the cryptocurrency server.

---

So whats so special about HEAT you might ask, why does HEAT what HEAT can do?

Well for that you'd have to dive into the source code of our competitors, be them Waves, Lisk, NEM, IOTA, NXT etc.. Just this afternoon I've taken a look at IOTA source code, which is something thats always interesting (did the same with the others mentioned).

But I can tell you right now that none of the other currencies (Waves, Lisk, NEM, IOTA, NXT etc) will be able to reach similar speeds as HEAT has now shown it can.

Why I can claim this is pretty simple.

Cryptocurrencies, all of them basically (blockchain or tangle makes no difference here), all follow a similar internal design. And all of them need to store their; balances, transactions, signatures, etc etc... and they all use different databases to do so.

Some like NXT use the most slowest of all solutions which is a full fledged SQL database, others have improved models optimized for higher speed in the form of key/value datastores. IOTA today i've learned uses rocksdb, Waves is on H2's key value store, Bitcoin is on leveldb etc.

Afaik HEAT is the only one that does not use a database. Instead we've modeled our data in such a way that it can be written to a Memory Mapped flat file, which is how we store blocks and transactions. Our balances are kept all in memory and to support on-disk persistence we use https://github.com/OpenHFT/Chronicle-Map as our memory/on-disk hybrid indexing solution.

If you'd look at ChronicleMap's website you'll see that they: "Chronicle Map provides in-memory access speeds, and supports ultra-low garbage collection. Chronicle Map can support the most demanding of applications." Oh and did I mention this grew from the needs of HFT trading systems be faster than anything else available?

---

Anyways..

The next test is gonna be even cooler. We'll be hosting the single HEAT instance which will be forging blocks on a nice and powerfull machine, much faster than my PC, probably something with 64GB RAM and perhaps some 32 or 64 cores.. My estimate is that we can push the TPS to much higher levels on such a server.

Right now we are adding binary AVRO encoding support to HEAT-SDK and after that we'll release one of those samples we used to do a month or so ago with which you can fire transactions from your browser to our test setup. I bet that'll be fun.

- Dennis

One can have a opinion I guess.

I think they shouldnt make it open source at all until everything is developed 100% in order to get a first mover advantage.

Why give this advantage away? One can only wonder.

Or you lease your HEAT to a forging pool ...

1. Log into your account in the HEAT wallet (web/desktop does not matter)
2. Use the top-right "hamburger" menu to select "lease balance"
3. As recipient use 1842674054425679153
4. Choose a period of blocks, with a minimum of 1440 and a maximum of 300000. This will define the leasing period, with 1440 being approx. half a day of leasing. Do note that a 1440 block period might almost go unnoticed, so would suggest to set it a bit higher
5. Click on "send" and you're all set!
6. You can repeat this action to define a follow-up leasing period, which means that automatically your new lease will start when the current one is over
7. For a withdrawal, send a message to the heat forging pool account in the HEAT wallet.

I fucking love heat!

Thanks for the update, Dennis. And congrats on such a huge technical milestone!

WE DID IT, 2000 transactions PER SECOND OVER THE NETWORK!!
_{proof will follow, test it for yourself from your browser}

Hi fellow HEATers, please allow me to share a short development status update.

Some interesting and major milestones have been achieved over the past week. To list a few:

Blockchain-Event-Cache

Big steps have been made towards finalizing the Blockchain-Event-Cache, the event cache gave us some real hard challenges to solve. An example being how to store to disk the tip or last N events of a never ending stream of events, without running out of disk space! And do it in such a way that no matter if the HEAT server crashes or shuts down, that at all times the persisted state of the blockchain matches exactly with all persisted events.

The significance of the event cache really can't be overstated. It's workings and design are unique and as far as I know exclusive to HEAT. The event cache should be considered as an ordered list of events where most events map to a whole or part of a blockchain transaction. The event cache has a cursor which points to the current latest event and this cursor can move up and down the list of events.

Observers of the events (microservices listening to transactions for instance) will be notified when an event first arrives. HEAT being a decentralized consensus mechanism must account for the inevitable slowness of reaching full network wide consensus simply because nodes need time to talk to each other, the network speed determines this. The event cache is your friend that watches blocks come and go, switches to better forks and reorganizations of transactions, all the while giving you a normalized view of the transaction stream as if you observed centralized service.

When building business applications on top of HEAT using the blockchain as a place to transport and store your data having this 'normal' view of the transaction stream is of vital importance for any kind of application you build. 

High-speed-RPC-channel

What was probably the coolest part of the week was to fire up the new high speed RPC channel we added to HEAT core. From unit tests we could already see that HEAT has been very fast when it comes to processing transactions, we've seen numbers of well over 2000 transactions a second when feeding transactions directly from memory into HEAT.

What hasn't been achieved yet was to process such high numbers over a network, until now that is!!. Two major obstacles of feeding lots of transactions to a HEAT server over the network are:

• Everything is encoded as JSON, parsing lots of JSON costs memory and downloading large blocks of JSON makes things slow
• Every transaction broadcasted did so over its own unique and new HTTP connection

The data encoding issue has been something we've looked at for some time now. What we wanted is basically write out raw binary data which is as dense and optimized as possible, yet still be able to handle this new format somewhat easily and preferably portable so it could be used for instance with the https://www.npmjs.com/package/heat-sdk. We would be using such for:

• Microservices user defined messaging protocols
• P2P message encoding
• To encode high speed RPC messages

We decided to go with https://avro.apache.org/ which is the data serialization stack from http://hadoop.apache.org/ [HADOOP=The Apache™ Hadoop® project develops open-source software for reliable, scalable, distributed computing.]. A major factor to choose AVRO is its support for JavaScript and the fact it supports versioning of schemas. When developing microservices those aspects are extremely important and allow service operators to easily declare their custom binary protocols which can be shared and used in a decentralized way where service users only ever need the public schema (hosted perhaps on the blockchain - since even the schemas can be efficiently encoded to binary data) in order to communicate with a microservice.
 
To solve the slowness of sending each transaction over a new connection we've added a new channel to the existing websocket handlers. You can now open a websocket connection and leave it open, not needing to connect again you can now send binary transactions over the already open connection. For now this channel supports only one operation, to broadcast a single transaction, but we expect this RPC mechanism to grow to support more methods in the future.

And the cool thing is you can use this super fast transport from everywhere where there is Websocket support, this includes everywhere the HEAT-SDK can be used (browser, mobile, nodeJS).

Playing around, we've already seen huge speeds. On my PC alone we could already reach over 2000 transactions a second for the duration of 500,000 transactions. This was done with 4 websocket connections and after pre-generating and signing the 500,000 transactions. That particular node was forging but with no connected peers, for that to be possible we need to update the P2P layer still.

Currently we are working on completing the RPC and AVRO binary data support to HEAT-SDK so that all this can be used simply by getting HEAT-SDK.
When that is ready we'll be hosting the custom HEAT server somewhere and we'll release some in-browser samples that people could use to fire transactions at the HEAT instance. Would be fun to see what numbers it can do, I know you guys like breaking things. 

Thanks for sticking around! And let's make HEAT great again.

You need to set up a forging node, hope this can help you...
https://heatnodes.org/heatnodes.pdf