strip the reference client to a border router

kjj

legendary

Activity: 1302

Merit: 1026

Quote from: jl2012 on October 28, 2013, 04:45:52 AM

Is it possible to build a "no dependency" client? Just integrate all (and only) necessary external codes, e.g. SSL, so people can build it without depending on any package.

Just FYI, I routinely make static-ish builds. My mining distro is based on Debian, but the bitcoind binary on it is build on an ancient slackware box with wildly different libraries. Fudging the makefile for a static build gives me a binary that will run just about anywhere.

On the original topic, rather than a fork, why not build options? This would reduce the maintenance delta to a hopefully manageable level. It is even something that could be done incrementally.

piotr_n

legendary

Activity: 2058

Merit: 1416

aka tonikt

Quote from: Pieter Wuille on October 28, 2013, 05:32:20 AM

Quote from: piotr_n on October 28, 2013, 05:29:16 AM

Quote from: jl2012 on October 28, 2013, 05:21:00 AM

Can we just embed the relevant part of OpenSSL to the code, so we don't need to worry about compatibility?

Personally, I would rather advise you to replace it with the code made by sipa, so you would not need to worry about NSA backdoors

In due time. At this point, it hasn't by far received enough auditing to become responsible for securing millions of dollars in transfers.

It's an admirable stand, but we are talking about wallet-less node here and something that only a small percentage of the net would use anyway.

I don't see a potential for a big damage - the worst thing that can happen is a chain fork, that would only speed up the process of maturing your fine lib.

If someone wants to secure receiving millions of dollars, I'd rather advise him to use more than one node while making sure that he received the money - a totally different ones, using not only different ECDSA libs, but basically different everything. And run them on different PCs..

Pieter Wuille

legendary

Activity: 1072

Merit: 1189

Quote from: piotr_n on October 28, 2013, 05:29:16 AM

Quote from: jl2012 on October 28, 2013, 05:21:00 AM

Can we just embed the relevant part of OpenSSL to the code, so we don't need to worry about compatibility?

Personally, I would rather advise you to replace it with the code made by sipa, so you would not need to worry about NSA backdoors

In due time. At this point, it hasn't by far received enough auditing to become responsible for securing millions of dollars in transfers.

piotr_n

legendary

Activity: 2058

Merit: 1416

aka tonikt

Quote from: jl2012 on October 28, 2013, 05:21:00 AM

Can we just embed the relevant part of OpenSSL to the code, so we don't need to worry about compatibility?

Personally, I would rather advise you to replace it with the code made by sipa, so you would not need to worry about NSA backdoors

jl2012

legendary

Activity: 1792

Merit: 1122

Quote from: wumpus on October 28, 2013, 04:58:30 AM

Quote from: jl2012 on October 28, 2013, 04:45:52 AM

Is it possible to build a "no dependency" client? Just integrate all (and only) necessary external codes, e.g. SSL, so people can build it without depending on any package.

You could use sipa's secp256 library instead of OpenSSL, that makes the blockchain handling code less dependent of third parties: https://github.com/sipa/secp256k1 (as a bonus, it's faster too!)
Note: this is at your own risk, the library is far from as well-tested as OpenSSL is. Handling cryptography is very fickle and a decision to switch the main client over is not taken lightly.

Quote

We agree that modularization is a goal, my question is if we should we tackle it with highest priority.

That's up to you! Everyone in the core team has their own priorities, and they may or may not overlap with yours (though we all agree that modularization is a good thing, but it's just not our pet project). You're welcome to join and patches are welcome!

Can we just embed the relevant part of OpenSSL to the code, so we don't need to worry about compatibility?

piotr_n

legendary

Activity: 2058

Merit: 1416

aka tonikt

Quote from: wumpus on October 28, 2013, 05:16:41 AM

Quote from: piotr_n on October 28, 2013, 05:10:37 AM

With this code, just getting rid of boost seems quite impossible.

Until the parts that we use of boost are part of the C++ library (which may happen in the future), getting rid of boost is not even a goal.

Yeah. Part of the C++ library, which I need like an hour to build.

Quote from: wumpus on October 28, 2013, 05:17:12 AM

Quote from: piotr_n on October 28, 2013, 05:10:37 AM

EDIT:
Oh, wait... WTF is protobuf?
You guys are not giving up adding more crap in, are you?

Protobuf is only used for the GUI (payment requests) and not for the core, so it is already optional.

I thought someone once said here that the payment protocol does not add any more external libs.
Well, I guess he was wrong again...

wumpus

hero member

Activity: 812

Merit: 1022

No Maps for These Territories

Quote from: piotr_n on October 28, 2013, 05:10:37 AM

EDIT:
Oh, wait... WTF is protobuf?
You guys are not giving up adding more crap in, are you?

Protobuf is only used for the GUI (payment requests) and not for the core, so it is already optional. The only required external dependencies for the core are:

- boost (extended C++ library)
- berkelydb (wallets, would be nice if this was optional when nowallet mode is used)

And one of
- OpenSSL
- secp256k1 library

Believe me, that's not a lot for an open source project.

wumpus

hero member

Activity: 812

Merit: 1022

No Maps for These Territories

Quote from: piotr_n on October 28, 2013, 05:10:37 AM

With this code, just getting rid of boost seems quite impossible.

Until the parts that we use of boost are part of the C++ library (which may happen in the future), getting rid of boost is not even a goal.
This means cross-platform thread handling, synchronization handling, asynchronous network handling, etc.
The only other option would be to embed cross platform compatibility layers in the code base itself, but that would increase the amount of code (NIH syndrome) to audit, test and maintain not decrease it.

C++ is just not a good language if you want self-contained code. There are no batteries included. This improved with C++11 though and we do plan on switching over to that and using C++11 functionality where appropriate instead of boost.

piotr_n

legendary

Activity: 2058

Merit: 1416

aka tonikt

Quote from: jl2012 on October 28, 2013, 04:45:52 AM

Is it possible to build a "no dependency" client? Just integrate all (and only) necessary external codes, e.g. SSL, so people can build it without depending on any package.

With this code, just getting rid of boost seems quite impossible.

LevelDB - one would need to create a new db engine, from scratch, wanting to get rid of this one.
But LevelDB is an internal code already, so I guess it could stay. It's a fairly simple one and easy to audit.

As for OpenSSL - I guess you could embed the hashing functions into the code and use the ecdsa functions made by sipa.

The rest (berkeley, upnp, zlip, png & qr) should be much easier to remove.

EDIT:
Oh, wait... WTF is protobuf?
You guys are not giving up adding more crap in, are you?

wumpus

hero member

Activity: 812

Merit: 1022

No Maps for These Territories

Quote from: jl2012 on October 28, 2013, 04:45:52 AM

Is it possible to build a "no dependency" client? Just integrate all (and only) necessary external codes, e.g. SSL, so people can build it without depending on any package.

You could use sipa's secp256 library instead of OpenSSL, that makes the blockchain handling code less dependent of third parties: https://github.com/sipa/secp256k1 (as a bonus, it's faster too!)
Note: this is at your own risk, the library is far from as well-tested as OpenSSL is. Handling cryptography is very fickle and a decision to switch the main client over is not taken lightly.

Quote

We agree that modularization is a goal, my question is if we should we tackle it with highest priority.

That's up to you! Everyone in the core team has their own priorities, and they may or may not overlap with yours (though we all agree that modularization is a good thing, but it's just not our pet project). You're welcome to join and patches are welcome!

jl2012

legendary

Activity: 1792

Merit: 1122

Is it possible to build a "no dependency" client? Just integrate all (and only) necessary external codes, e.g. SSL, so people can build it without depending on any package.

piotr_n

legendary

Activity: 2058

Merit: 1416

aka tonikt

Quote from: jgarzik on October 27, 2013, 02:56:52 PM

Quote from: piotr_n on October 27, 2013, 11:20:15 AM

I dare to state (and I am dying to be proven wrong) that making a wallet-less reference node, with minimal external dependencies will never happen - at least not made by this team.
It just isn't on their employers' agenda, despite of all the rumors and gossips that you might have heard.

no-wallet mode will be in version 0.9.

but from I understand, it will only be controlled by a command line switch.
all the crap, plus tones of a new crap, will be still linked with the node, won't it?

so that is definitely not what I meant by "wallet-less reference node, with minimal external dependencies".
command line switches disabling functionality do not make a code easier to audit, nor less open for a potential exploits.

and BTW man, thanks for being one of a few out there who at least try to do it the right way
https://github.com/bitcoin/bitcoin/pull/2901

jgarzik

legendary

Activity: 1596

Merit: 1100

Quote from: piotr_n on October 27, 2013, 11:20:15 AM

I dare to state (and I am dying to be proven wrong) that making a wallet-less reference node, with minimal external dependencies will never happen - at least not made by this team.
It just isn't on their employers' agenda, despite of all the rumors and gossips that you might have heard.

no-wallet mode will be in version 0.9.

piotr_n

legendary

Activity: 2058

Merit: 1416

aka tonikt

Quote from: justusranvier on October 27, 2013, 12:10:06 PM

Quote from: piotr_n on October 27, 2013, 11:20:15 AM

The only way to have such a piece of software, is by forking the existing bitcoind and stripping it from all the crap, without even looking at the corrupt bitcoin elite, nor letting them to waste your time. If you want to have such a piece of software, then trust me: sooner you will make it yourself. I'd help you, but I prefer to raise my own child instead

Conformal Systems, and bits of proof, and Amir Taaki have already done that.

Thanks for being so well informed, but I have also done this, so please mind not letting my fine project behind

It has everything a bitcoin node needs and performs just great; I can challenge any of the software you've listed.
And if anyone needs a mining API, just let me know - shouldn't be hard to add.

justusranvier

legendary

Activity: 1400

Merit: 1013

Quote from: piotr_n on October 27, 2013, 11:20:15 AM

The only way to have such a piece of software, is by forking the existing bitcoind and stripping it from all the crap, without even looking at the corrupt bitcoin elite, nor letting them to waste your time. If you want to have such a piece of software, then trust me: sooner you will make it yourself. I'd help you, but I prefer to raise my own child instead

Conformal Systems, and bits of proof, and Amir Taaki have already done that.

piotr_n

legendary

Activity: 2058

Merit: 1416

aka tonikt

Guys, you are talking to a wall here.

I dare to state (and I am dying to be proven wrong) that making a wallet-less reference node, with minimal external dependencies will never happen - at least not made by this team.
It just isn't on their employers' agenda, despite of all the rumors and gossips that you might have heard.

The only way to have such a piece of software, is by forking the existing bitcoind and stripping it from all the crap, without even looking at the corrupt bitcoin elite, nor letting them to waste your time. If you want to have such a piece of software, then trust me: sooner you will make it yourself. I'd help you, but I prefer to raise my own child instead

jl2012

legendary

Activity: 1792

Merit: 1122

I like this idea. I use Armory only. It is way more secure than the reference client. A fully modularized system is easier to audit, maintain, and expand.

justusranvier

legendary

Activity: 1400

Merit: 1013

It would be cool if the Satoshi client, btcd, and BoP could all use the same APIs to talk to the various components such that the components become interchangeable.

grau

hero member

Activity: 836

Merit: 1030

bits of proof

This sounds promising Jeff. One could think we face similar challenges Wink

A chance of such refactoring could make me picking up C++ again ... after 12 years of not missing it.

jgarzik

legendary

Activity: 1596

Merit: 1100

A branch would increase work required, rather than decrease.

Here is the current rough plan:

The first step is basically complete: a runtime no-wallet mode: https://github.com/bitcoin/bitcoin/pull/2901 Later on, this may be extended to #ifdef'ing out wallet code, once the "no wallet" runtime code has been proven to work.

Next, headers-first sync, which fixes many issues with the bitcoin block download process in general: https://github.com/sipa/bitcoin/tree/headersfirst

Once these features in place, it becomes feasible to have the wallet client operate in a separate process from the public blockchain engine.

At that point you have a stripped down "blockchain engine" (border router) that focuses on mesh networking and accurately maintaining the chain.

grau

hero member

Activity: 836

Merit: 1030

bits of proof

Quote from: Mike Hearn on October 07, 2013, 01:25:26 PM

The Satoshi client does have a kind of internal API, if you look at how the wallet and GUI interacts with the rest of the code then you can see it. It's not perfect but polishing it hardly seems the highest priority.

I think you're right that lots of people use their software connected to a trusted Satoshi node that they run themselves, but realistically the wallet and GUI aren't just going to be deleted from the core Bitcoin code - people use them!

I wish for an isolation much higher than a more or less defined set of function signatures and structures within the same code base.
Such isolation was not a goal for Satoshi and is apparently not a priority for the core team.

I suggested such a fork (of code) to reduce the pain of code reviews and increase the pace of innovation.
It would come at a cost. Do you think it is feasible to branch and regularly rebase such a code subset for a "border router" ?

Mike Hearn

legendary

Activity: 1526

Merit: 1136

The Satoshi client does have a kind of internal API, if you look at how the wallet and GUI interacts with the rest of the code then you can see it. It's not perfect but polishing it hardly seems the highest priority.

I think you're right that lots of people use their software connected to a trusted Satoshi node that they run themselves, but realistically the wallet and GUI aren't just going to be deleted from the core Bitcoin code - people use them!

grau

hero member

Activity: 836

Merit: 1030

bits of proof

Quote from: gweedo on October 07, 2013, 11:50:36 AM

https://github.com/jgarzik/pynode this is kinda what you are talking about, this would be better suited than stripping out things from the reference client, which would be the hacky way to do this.

Thinks I have that kinda stuff too, check my signature.

The question is if we want more that kind of stuff or rather make satoshi welcoming to extensions.

grau

hero member

Activity: 836

Merit: 1030

bits of proof

Quote from: gmaxwell on October 07, 2013, 11:46:02 AM

If Gavin was interested in (/thought his time was best spent on) those other things he would spent them on them. You can't dictate how Gavin spends his time by creating a fork.

I certainly can't or want to dictate what he does or doesn't. We could however lower the barrier for those who could do stuff that does not require in depth knowledge of the real core, so he gets more help and can consider if that is where he want to be (too).

Quote from: gmaxwell on October 07, 2013, 11:46:02 AM

As an aside, though the code ends up in the same binary, as I recall the payment protocol didn't touch consensus mechanism stuff, it was largely self contained and nowhere near the core stuff. Increased modularity is a long term goal to making review easier, though it's currently come a long way from the original code that freely intermixed the GUI wallet stuff with the consensus code.

Even if it is better separated than older cross-cutting concepts, would't be nice if it was built on top of an API isolating the consensus layer and be in a separate project?

We agree that modularization is a goal, my question is if we should we tackle it with highest priority.

gweedo

legendary

Activity: 1498

Merit: 1000

https://github.com/jgarzik/pynode this is kinda what you are talking about, this would be better suited than stripping out things from the reference client, which would be the hacky way to do this.

gmaxwell

staff

Activity: 4326

Merit: 8951

If Gavin was interested in (/thought his time was best spent on) those other things he would spent them on them. You can't dictate how Gavin spends his time by creating a fork.

As an aside, though the code ends up in the same binary, as I recall the payment protocol didn't touch consensus mechanism stuff, it was largely self contained and nowhere near the core stuff. Increased modularity is a long term goal to making review easier, though it's currently come a long way from the original code that freely intermixed the GUI wallet stuff with the consensus code.

grau

hero member

Activity: 836

Merit: 1030

bits of proof

The goal is to
1. better support for feature development without touching or re-implementing functions of the reference client.
2. reduce number of lines of code and complexity in reference client to increase stability and scaleability.

Although wallet or RPC is harmless at runtime if not used, I do believe that every line of extra code is harmful at development as it makes code review and refactoring more complex and risky.

To give a practical example:
I think payment protocol is much needed, but I wish it could be implemented without touching the code base running the consensus and it would be implemented by other than Gavin. His in depth knowledge of the Satoshi code would be better spent on DoS protection and scaleability issues. With a stripped down reference client and a better API that development could be much better outsourced to the community.

gmaxwell

staff

Activity: 4326

Merit: 8951

Wouldn't this increase the amount of software which must be reviewed, tested, patched, and otherwise maintained? Is this just for the sake of removing parts which are largely inactive and harmless when not used?

I often don't understand proposals stated in terms of mechanisms rather than goals. Perhaps the benefits are apparent to those who would want it, but for my sake what benefit(s) would this serve which would justify the costs?

There is already have code to turn off the wallet. If there is a real application for turning off other things I don't see a reason that a fork would be required.

grau

hero member

Activity: 836

Merit: 1030

bits of proof

Would you welcome a fork of satoshi aimed to strip it of all but the consensus algorithm, that is without wallet and the RPC replaced with a bus?

Motivation
I recommend using the 'reference' client as border router to the P2P network in production environments, then connect the BOP server inside your company to it, to get a convenient API to Bitcoin in form of a message bus that serves applications and client side wallets.

I guess that above suggestion is not unique to BOP but is how those operate who developed significant added value around satoshi client and wonder if this use case should be the priority for reference client development instead of further extending the feature set of that fragile and crucial code base.

Topic: strip the reference client to a border router (Read 1913 times)