Hi, say I want to run a data mining project in this elastic supercomputer.
This project would require a somewhat large data set to download to the system.
Then it would require some software to read the and conduct the analysis using the supercomputer.
Let say I could write a program to do so if needed (though could be complex).
How far away are we from elastic capability to allow something like this.
Or are we already there in principle?
In principle it can do what you stated; however, it would certainly depend on your definition of a large data set. Here are a couple of things to note:
1) We need EK to confirm that max storage size that the network can handle....right now the miner is coded to only allow 256MB memory use, but that is just a random number I picked...it will depend on what EK determines the network can handle as part of the message traffic.
2) Currently, the Elastic UI doesn't allow the job author to upload a starting dataset....the current miner / node design accommodate this, but the UI would need to be modified to provide a way to upload the data
3) In the existing design, for smaller datasets (i.e. < 100K or so) you'd just include the dataset in the ElasticPL code and use it to initialize the variables during the first round.
So again, it would depend on what you consider a large dataset...as we've pointed out numerous times, the current design of Elastic was not intended for data intensive tasks such as rendering, machine learning, etc.
Its very interesting point of view. 