Author

Topic: NXT :: descendant of Bitcoin - Updated Information - page 1004. (Read 2761642 times)

legendary
Activity: 1890
Merit: 1086
Ian Knowles - CIYAM Lead Developer
I know it is not perfect, that is why I am asking for feedback to make it right. I think it is at least closer and things like http://www.wired.com/wiredscience/2014/02/cryptography-breakthrough/ give me hope that a solution is possible.

Am pretty sure that the kind of complexity required for that kind of approach would make the whole VM thing far too expensive to consider running for probably many, many years (so "stop right there" unless you are planning years in advance now).

...then why cant all the peers conclude properly that the correct plugin was indeed run on the correct input data (which is the output of NXT VM script)?

What a peer would do is have to run the script itself (get used to the fact that the script will need to run on *every* peer except perhaps for some lightweight ones) and compare its result to that being suggested by another peer.

This is why the script's state must be "deterministic" if a peer tells you that the answer to running script A with state X results in state Y but you think it results in state Z then you ignore that block as being invalid and block the peer.
legendary
Activity: 1176
Merit: 1134
Understand that any "step in the script" to "verify a plugin" can *simply be ignored* by Evil Bob's script processor!

It is the NXT core that would be verifying the plugin. Not sure if that makes any difference. Evil Bob's script processor has to generate the same output or it will get ignored by peers since the AM data from the script is wrong if changed.
legendary
Activity: 1176
Merit: 1134
I imagine trusting the plugins because they will be tested and source reviewed and realtime checksummed against tampering and peers will be able to validate that it actually ran and for mission critical plugins, we probably need an external verification, eg. blockchain.info for bitcoind operations

This is just wrong - it's like your trusting that other peers are running the correct NRS - of course they may not be and you have no way to know. The only thing you can do is verify results and compare them to what you know (and blacklist peers that disagree with you).

So your verification idea here is "ass about" (it would only be useful for a server to use to check that they are running the correct plugin themselves - it doesn't help the "script" owner in any way at all).

I know it is not perfect, that is why I am asking for feedback to make it right. I think it is at least closer and things like http://www.wired.com/wiredscience/2014/02/cryptography-breakthrough/ give me hope that a solution is possible.

So you are saying that Evil Bob will spoof that he is running the correct everything. I think the referenced article says that it is possible to do what I want, even though it is supposed to be impossible, even for Evil Bob

Also, I must be too tired to understand why if the forging node can verify they are running the correct plugin and it puts the result of running the plugin into the forged block and the forged block is verified cryptographically by the peers, then why cant all the peers conclude properly that the correct plugin was indeed run on the correct input data (which is the output of NXT VM script)?

What am I missing? I am thinking that all script owners will be assured that the output of their script will be processed only by a validated NXTplugin.

James

Edit: ah, so we need to make sure that the forging node did the validation of the plugin. Couldnt we use zeroknowledge proof for that?
legendary
Activity: 1890
Merit: 1086
Ian Knowles - CIYAM Lead Developer
I imagine trusting the plugins because they will be tested and source reviewed and realtime checksummed against tampering and peers will be able to validate that it actually ran and for mission critical plugins, we probably need an external verification, eg. blockchain.info for bitcoind operations

This is just wrong - it's like your trusting that other peers are running the correct NRS - of course they may not be and you have no way to know. The only thing you can do is verify results and compare them to what you know (and blacklist peers that disagree with you).

So your verification idea here is "ass about" (it would only be useful for a server to use to check that they are running the correct plugin themselves which would really just be like a "version check" - it doesn't help the "script" itself in any way at all).

Understand that any "step in the script" to "verify a plugin" can *simply be ignored* by Evil Bob's script processor!
legendary
Activity: 1176
Merit: 1134
I think we can verify the checksum of the running code for plugin matches the source code. Similar to signing of .jar files

Can you describe with details few things:
  • what the plugins could do and why they should be used
  • HOW they should be made? (that is should they operate on API or they should be REAL plugins)
  • what about people who don't want plugins? how limiting it (lack of plugins) will be?
  • How do you actually imagine this smtp plugin would work like (I'd like really detailed description along with use case description https://en.wikipedia.org/wiki/Use_case)
  • How do you imagine TRUSTing the plugins?

To be clear, I'm asking those questions, as most likely I will criticize the idea, once you answer to those questions.
https://bitcointalksearch.org/topic/m.5007726 has the original post about NXTplugins. Probably lost on this thread.

I use the term plugin just as a placeholder, I am beginning to think it means different specific things to different people and is not the best term. When I say NXTplugin, I mean the code that is invoked when the forging node scans the AM data and finds that it is input meant for a specific plugin.

I have a bounty out on how to make plugins, it depends on what is needed to be able to do realtime checksum/hash of its code space in memory.

A node does not have to have any NXTplugins, however if a NXTplugin is not on the servers that are forging most of the time, it could be a long time before it will get called. TF can be used to find the node that will actually call the plugin and so all the other nodes will know when to look for the result and error code.

NXTsmtp is to flesh out the details of the NXTplugin architecture, eg. is it the right way to expand NXT VM power? How to implement it? What sorts of requirements will we need to constrain the actual production NXTplugins, etc.

I imagine trusting the plugins because they will be tested and source reviewed and realtime checksummed against tampering and peers will be able to validate that it actually ran and for mission critical plugins, we probably need an external verification, eg. blockchain.info for bitcoind operations

James
legendary
Activity: 866
Merit: 1002
I think we can verify the checksum of the running code for plugin matches the source code. Similar to signing of .jar files

Can you describe with details few things:
  • what the plugins could do and why they should be used
  • HOW they should be made? (that is should they operate on API or they should be REAL plugins)
  • what about people who don't want plugins? how limiting it (lack of plugins) will be?
  • How do you actually imagine this smtp plugin would work like (I'd like really detailed description along with use case description https://en.wikipedia.org/wiki/Use_case)
  • How do you imagine TRUSTing the plugins?

To be clear, I'm asking those questions, as most likely I will criticize the idea, once you answer to those questions.
legendary
Activity: 1176
Merit: 1134
I think we can verify the checksum of the running code for plugin matches the source code. Similar to signing of .jar files

I don't see how running the "right plugin" is going to help if the plugin deals with any 3rd party software or protocol at all as Evil Bob doesn't need to *change the plugin* he will make his changes to the 3rd party software or intercept and modify the protocol commands.

The problem of using blockchain.info is that it is a website and that opens it up to all the problems of websites being hacked. It seemed a lot more secure to be able to verify that bitcoind running matches the bitcoind source code.

It simply isn't relevant if you are going to have other servers "check the script execution" which you will *have* to do in order for it to be correctly verified (which is why sending an email would be silly).

I am not thinking that there are ANY third party softwares. The NXTplugin needs to incorporate some or all of the third party software into itself. Otherwise there is no way it can be trusted.

The peer servers will verify the output of the plugin. We rely on the source code to know what the plugin did. So all the peers can verify that Evil Bob made no changes to the plugin and the plugin ran and output the result. How can Evil Bob modify the plugin if NXT core is verifying checksum/hash of the executing code in memory? Any changes would change the checksum/hash of the in memory copy of the plugin.

If I am wrong in that, THAT is the feedback I am looking for. How can Evil Bob change the plugin without changing the checksum/hash of the in memory copy that is being called by NXTcore?

The peers dont have to send the email, they just need to verify that the checksum/hash + result/errorcode matches what would have been expected given the input data to the plugin, which is the output data from NXT VM script

James

Edit: Ideally we would have an externally verifiable action, eg. unmodified bitcoind issued command and it is verified on blockchain.info as an example
full member
Activity: 189
Merit: 100
I am a bit concerned that there has been very little feedback on my recent proposals, blockchain FIFO and NXT plugin architecture.

I've been reading your posts with interest and trying to digest them.  Much of it sounds good, but is mostly over my head so it's hard to give good feedback.  My biggest concern is security right now especially after the recent scare.  New features often bring new security holes, so I'd rather not be in too much of a rush to beat the competition for every little thing.  Nxt already has a strong niche (zero inflation, proof-of-stake) and just needs steady, but not rushed, development to bring in the new features which may or may not be embraced by the market.

Has Dr. Evil been hired to continue to looking for exploits and weaknesses and consult?  I saw a couple posts requesting this, but it should be a priority.  He's proven himself by brute forcing something like 3% of Nxt accounts (including genesis) and discovering an x-spend attack.  If we have community funds available then I think we should try to keep him on board as long as we can.

+1
edit: new features often bring security holes, but this brainstorming is just project ideas, doesn't mean all will be implemented, but it will be discussed and possibly will be considered.
legendary
Activity: 1890
Merit: 1086
Ian Knowles - CIYAM Lead Developer
I think we can verify the checksum of the running code for plugin matches the source code. Similar to signing of .jar files

I don't see how running the "right plugin" is going to help if the plugin deals with any 3rd party software or protocol at all as Evil Bob doesn't need to *change the plugin* he will make his changes to the 3rd party software or intercept and modify the protocol commands.

The problem of using blockchain.info is that it is a website and that opens it up to all the problems of websites being hacked. It seemed a lot more secure to be able to verify that bitcoind running matches the bitcoind source code.

Of course using "blockchain.info" would be a hack but it isn't that relevant if you are going to have other servers "check the script execution" which you will *have* to do in order for it to be correctly verified (which is why sending an email would be silly).
legendary
Activity: 1176
Merit: 1134
James,

First of all thank you for all your great ideas, but ...

My background is IT project manager and I am going crazy by you.

You throw 10 projects on the table but have not one worked out.

Please for starters pick on project, work it out from start to finish, than pick another.

As of now your way of working getting us nowhere.

You are ddossing us.


I'm no IT project manager but right now, brainstorming some ideas and projects could actually be quite healthy. The problem is maybe that notevery idea (and the interaction/interference) gets discusses here. We have no overview right now of projects, ideas, developments, developers right now.

I had thought there was a cry to make sure NXT handles 1000TPS, that we add new tech features, etc. After CfB's set of posts the other day, I have worked very hard to come up with a technical roadmap for after April. If

Did I miss your roadmap?
NXTcash
NXTlayers
NXTplugins
cross chain
automated DAC gateway
probably more, getting tired I think it is already tomorrow

James

Edit: Where else are dev ideas being actively discussed. Only very small activity on nxtcrypto.
legendary
Activity: 1176
Merit: 1134
I didnt think there was a chance to find bitcoind in Java form that jean-luc would consider adding to NXT core.

Why would you need/want that rather than just doing RPC commands to a "real bitcoind"?

It is MUCH simpler to solve the issues with a hardcoded NXTplugin since we dont have to deal with Evil Bob changing the executable. Not having to worry about Evil Bob seemed prudent for the first attempt at adding parsing of AM data to see what plugins to call, etc.

You are *always* going to have Evil Bob using the "wrong plugin" and *you will not be able to tell* especially if your plugin has no way to be verified (which was the point about an SMTP plugin).

If we cant solve the issue with simple hardcoded plugin, no chance for complex external plugin. That is why I chose email as the proof of concept.

Exactly my concern - a plugin that issues a "bitcoind" RPC command (hell - why not just use "blockchain.info" for that matter) is at least *verifiable* in that given x servers running the "supposedly same" plugin you would get the exact same result from all of them (if they are able to give a result at all that is).

If you want a "dead simple" plugin then how about one that just does this:

return "hello";

I think we can verify the checksum of the running code for plugin matches the source code. Similar to signing of .jar files

The problem of using blockchain.info is that it is a website and that opens it up to all the problems of websites being hacked. It seemed a lot more secure to be able to verify that bitcoind running matches the bitcoind source code.

I couldn't justify paying a bounty for a program that returned "hello"

We need more people familiar with the NXT core. I wanted simple enough project that had some utility that would get people to see how easy it is to add functionality to NXT
legendary
Activity: 1176
Merit: 1134

Here is my idea for avoiding blockchain bloat while doing 1000TPS. I am surprised nobody mentioned this, so it is either because it is a really bad idea for some reason, or too obvious nobody bothered??

From my understanding, it will be possible to have regular snapshots of the entire state of the NXT blockchain so that you wont need to parse all the blocks from genesis.

Secondly, very little of this "entire state of NXT" data has a life beyond 1440 blocks.

So, my proposal is to generate a daily snapshot, peer reviewed by nodes, checksummed, fingerprinted, signatured, whatever we need to make sure it is a valid unmolested snapshot. We actually dont even need these snapshots, but while we are doing this, might as well avoid having to download the entire blockchain. Bandwidth savings alone makes it worthwhile.

OK, so one way or another, let us assume the node is current. Now the problem is keeping up with 1000TPS (lets make the overall network adaptive so we can handle bursts of 1000TPS, sustained 250 TPS) and that requires bandwidth, though with binary data, 250 TPS should be around what 100TPS will be now. So 100kbps would be enough to handle bursts of 1000TPS and sustained 250TPS

But where does all that data go?

Actually, I say just throw it away! Why can't we use a FIFO that stores the most recent 1441 blocks for all the blockchains (when we go parallel). Since we synced the the full blockchain with the last days checkpoint, then as long as we are properly updating the "entire state of NXT network" at any given block, we are able to forge a valid block.

Unless I totally missed some reason why we need to locally store more than the last 1441 blocks, this should work. We can then specify that any NXT VM (Turing scripts) will need to be designed to use only data from prior 1440 blocks. This I do not see as a big limitation as if it is really important the client issuing script can just get data it needs and put it into AM. So, the NXTcore would need to implement the FIFO, preferably based on a web.xml parameter. That would allow people who run NXT VM generating clients to have access to exactly the window of time they need.

I hope this puts useful forging back on the table for raspis. 100kbps to fully support 1000TPS (peak output) NXT network

James

I think, snapshots were mentioned or spoken about a couple of times.
Yes, snapshots were, but not using a blockchain FIFO (other than xyzz's post that was ignored). That is what allows the raspi to keep up. Even without snapshots, a raspi would be able to keep up, as long as it ever could catch up. That is why I combined it with snapshots to make sure all raspis could catch up to the current block.

Using snapshots alone would require much more frequent snapshots to be made and created all sorts of issues with delays, etc. If the blockchain FIFO is implemented, we can get by with weekly snapshots. This is because the blockchain is not stored locally other than the most recent day.

Also, we dont need parallel blockchains because of HDD usage if we had blockchain FIFO. There could be other reasons like partitioning workload, etc., but it is a quick way to get to 1000TPS and beyond without any software magic.

I saw the post about this, but nobody else seemed to recognize the significance. I just felt that it solved the 1000TPS on raspis issue and so wanted feedback on it to make sure it would work. It just seems so simple an idea, I wonder why it wasnt already done

James
hero member
Activity: 910
Merit: 1000
James,

First of all thank you for all your great ideas, but ...

My background is IT project manager and I am going crazy by you.

You throw 10 projects on the table but have not one worked out.

Please for starters pick on project, work it out from start to finish, than pick another.

As of now your way of working getting us nowhere.

You are ddossing us.


I'm no IT project manager but right now, brainstorming some ideas and projects could actually be quite healthy. The problem is maybe that notevery idea (and the interaction/interference) gets discusses here. We have no overview right now of projects, ideas, developments, developers right now.

I had thought there was a cry to make sure NXT handles 1000TPS, that we add new tech features, etc. After CfB's set of posts the other day, I have worked very hard to come up with a technical roadmap for after April. If

Did I miss your roadmap?
legendary
Activity: 2142
Merit: 1010
Newbie
For the people with less knowledge: Clarify exactly with more than 3 words what's your definition of "service providers" is.

Off-chain service using Nxt to accept subscription payments.
hero member
Activity: 910
Merit: 1000

The problem is that I see all of the things I am posting about as connected. Like the elephant described by different people. All sounds very different, but it is all the same elephant. If I described the elephant in its entirety, it wouldnt fit in posts. I feel a great sense of urgency due to competitive pressures.

Could you say some words about the "same elephant" and give some context within this post please (especially target audience and feature list)?

https://bitcointalksearch.org/topic/m.4959522
sr. member
Activity: 644
Merit: 250
Updates - 07/02/2014
- Shows multiple balances.
- Shows node status (Green = ON, Blue = OFF)

Now effective balance 136'722.00 NXT!!!

NXTCoin first automated forging platform!
Website: http://www.nxtio.org/

Is that really what we want? Centralization?
Not think it is as big concentration. My goal is to create different accounts as you get to an acceptable amount.

I thank the pool operators for trying to provide a service for the community, but I think forging should remain individual.  Granted, Nxt is resistant to up to 90% concentration attacks, but I don't think this is a path we want to follow for multiple reasons.  Sooner or later, someone will open a pool and run off with everyone's Nxt.  I'm not saying the current ops will do this, but it's bound to happen.

You don't get any more using a pool, you just get more consistent payouts.
Those with very few nxt could forge something through my system. Also, consider that you can be forged without being individually 24hr online.

I think it has advantages and disadvantages.

Yet to be seen, if it is efficient. Is a long-term experiment.
And think that in less than a week running, seem going to to work very well.


If you do not use all your NXTcoin's and you never open your client, what purpose are your NXTcoin's? I think NXTio is the answer to this problem, so I decided to give a solution.

Sorry the bad English, I really speak Spanish.

You make some valid points.  People who can't run their client 24/7 benefit and the network benefits.  Nxt is certainly in a better position to deal with security concentration than it's PoW counterparts.

Thanks again for providing a service to the community.
legendary
Activity: 1890
Merit: 1086
Ian Knowles - CIYAM Lead Developer
I didnt think there was a chance to find bitcoind in Java form that jean-luc would consider adding to NXT core.

Why would you need/want that rather than just doing RPC commands to a "real bitcoind"?

It is MUCH simpler to solve the issues with a hardcoded NXTplugin since we dont have to deal with Evil Bob changing the executable. Not having to worry about Evil Bob seemed prudent for the first attempt at adding parsing of AM data to see what plugins to call, etc.

You are *always* going to have Evil Bob using the "wrong plugin" and *you will not be able to tell* especially if your plugin has no way to be verified (which was the point about an SMTP plugin).

If we cant solve the issue with simple hardcoded plugin, no chance for complex external plugin. That is why I chose email as the proof of concept.

Exactly my concern - a plugin that issues a "bitcoind" RPC command (hell - why not just use "blockchain.info" for that matter) is at least *verifiable* in that given x servers running the "supposedly same" plugin you would get the exact same result from all of them (if they are able to give a result at all that is).

If you want a "dead simple" plugin then how about one that just does this:

return "hello";
hero member
Activity: 910
Merit: 1000
What about securing the network by giving the nodes running with the full blockchain a higher forging chance motivating the owners to go on with the full blockchain?
Others, like the guys on a rasp or on a smartphone would use a "light" version of the blockchain (something like James proposed)

If we start pruning someone needs to keep a full copy I guess so some kind of motivation would be needed to maintain this as a service to the network with associated bandwidth etc.

Phone clients should not need a block chain at all (unless someone really wants to burn battery and data bundles forging) when we have proper clients and should just access a public node with a full copy for any block chain needs.

Lightweight phone clients will not forge but verify transactions itself and send it to nodes. I hope someone is developing this "client side verification process" right now.
hero member
Activity: 910
Merit: 1000
ind ourself that the POSSIBILITY of 1000TPS brings us to 1000TPS in a few years. If we now state "not possible", we lose one feature.

1000TPS will not even touch the sides in a few years, if NXT manages to get it right.

In the adoption of e-currency in emerging markets are light years ahead of western markets. I know its not crypto but the average citizen in the emerging market cares about the outcome not the technology and in this model crucially the bank or telco offering the service helps the person get the money back if it goes astray - when your income is $10 / day then you are very careful where you put your money.

There have until recently been just 2 african countries where this has taken off but from 2012 this phenomenon has been spreading to other countries where telco's or banks basically take away the FIAT and provide electronic tokens that can be used by dumb text phones, to move money electronically and instantly via and e-wallet and yes these phones will be around a long time. no PCs no smart phones - these countries are still toxic to these technologies.

The recent GSMA report I've seen shows at least 10 countries where adoption of this way of paying is following a hyperbolic or hockey stick trajectory. In one of these countries the transaction platform can perform around between 300-400tps and resides in Europe not Africa.

Over a day in one of the major African markets demand can go from 0tps to 500tps, this is with only around 15m active customers and the market is less than 30% penetrated in terms of retail transactions and this is only one country. Average transaction time is <10s with full confirmation (yes this includes shipping everything from Africa to Germany and back) and for small transactions the cost is ZERO.

This kind of stuff is now being trialled in India, and there the transaction volumes will be way higher than 1000tps if you break into the for retail space, but for retail unlike general P2P you need instant transactions - VISA fakes it (authorise and then clear later), the e-money platforms do it all realtime and currently the crypto currencies do not address the problem of the retailer. Although some posts from I think cfb about how to implement instant transactions is the first I've seen of a crypto solution to this problem and i would hang onto it even if its implemented later.

So I would plan for far higher than 1000tps, solve the accompanying block chain problem and make sure you can do instant transactions, unless you just want to nibble around the crumbs that VISA / Mastercard will drop - both of whom have bough e-money platform companies...

With parallel chains, we could tackle the >1000TPS and other things like touring stuff... who is in charge of parallel chains? BCNext? JL? Community?
legendary
Activity: 1176
Merit: 1134
I don't see why you would want a Nxt VM script to "output an email" (or do anything else outside of the blockchain for that matter) - you do understand that whether such email was actually even really sent simply *cannot be proven* (as you are dealing with SMTP rather than a blockchain)?

Also SMTP is going to require accounts that need to be signed into and you don't want to end up with people effectively running "relay servers" or they'll end up on email blacklists.

Wouldn't it make more sense for such things to be services instead?

About walking vs. running - you are getting far too excited jl777 - can you just take something to slow down to a pace that we can keep up with (by the time we've tried to discuss one of your ideas you typically have posted 3 others).

NXTsmtp is just for proof of concept that incorporates something everybody is familiar with. I want to verify the peer verifiability of hardcoded NXTplugin followed by external NXTplugin. I am not worried about NXTsmtp for anything other than proving that NXTplugins work and are peer verified. At first I couldnt understand how on earth a DAC could be implemented. When I started thinking about email plugin, it became not as hard.

If the source to SMTP server is reviewed that it does send the email (backed up with test results) and as part of the sending process it adds a hash value of email to the blockchain. I think that allows peer verification, please explain where I am wrong. I am certain I have made mistakes somewhere and I am still coming up to speed with this whole decentralized blockchain approach.

The problem is that I see all of the things I am posting about as connected. Like the elephant described by different people. All sounds very different, but it is all the same elephant. If I described the elephant in its entirety, it wouldnt fit in posts. I feel a great sense of urgency due to competitive pressures.

James

P.S. I usually dont post when I am sleeping or flying Smiley

I could think of useful ideas for onchain plugins. But offchain?

It's the same as with other offchain activities like crosschain exchange, fiat exchange. Nxt has no control of process outside of it.

Anyway, what I wanted to add is: we should reserve a branch for each plugin on the blockchain. That is nodes that do not want to contribute to that chain does not need to download the data.
What if NXT was extended to deal with additional features and then added that to the blockchain. If we extend the meaning of NXT blockchain, then what used to be offchain becomes onchain.

THAT is what I am trying to do. Open the possibility of making anything we can fit into an AM into something that we can see was called, onchain. Now the issue of is running the code enough is answered by as long as all the error cases are reported and handled properly, I think it very well could be.
Jump to: