Pages:
Author

Topic: [ANNOUNCE] picocoin and libccoin -- C-based bitcoin library and client - page 6. (Read 30592 times)

legendary
Activity: 1120
Merit: 1150
As I've noted: I've forgot that you are one of the very few Bitcoin developers with regular access to big-endian hardware. I'm thinking that you will continue to be the only one developer who can test picocoin on the big-endian architecture. The mojibake-endian problem will continue to badly affect the future contributors/users of your library if you aren't going to use your own medicine (that is "sparse" and "__le32")

FWIW while I've never tried doing it QEMU can apparently emulate the big-endian PowerPC architecture. Here is one guys notes about getting it working from two years ago, specifically to test endian issues.
legendary
Activity: 1596
Merit: 1091
Thanks to jrmithdobbs for helping in bringing up OpenBSD (and more FreeBSD help from denisx).

Here's the autogen.sh and configure magic he used for OpenBSD.  Some are specific to his system, but it is an illlustrative example:

Code:
AUTOMAKE_VERSION=1.11 AUTOCONF_VERSION=2.68 ./autogen.sh && \
CC=clang LD=llvm-ld CFLAGS='-O2 -Wall -g -I/usr/local/include/event2 \
-I/opt/OpenBSD/5.1/amd64/jansson-2.4/include -I/usr/local/include/event2' \
 LDFLAGS='-L/opt/OpenBSD/5.1/amd64/jansson-2.4/lib -L/usr/local/lib -L/usr/lib' \
      ./configure --prefix=/opt/OpenBSD/5.1/amd64/picocoin-0.0 && \
      make all check
hero member
Activity: 836
Merit: 1021
bits of proof
Great to see a well written alternative for the thin client!

Follows the philosophy of "do, not hype."  This library is already far more secure and capable than other libraries hyped as the "future of bitcoin" by their authors.
Add do not FUD.
legendary
Activity: 2128
Merit: 1068
Removing the GLib dependency is straightforward for any developer.  GLib is used for a few ADTs like variable length strings, hash tables and arrays.

Not sure you understand the problem space...  First, newlib is wholly different from GLib, and does not provide the ADTs that GLib provides.  Second, replacing GLib and OpenSSL would not change resource usage much at all.  ADTs, Big Integer support, SHA hashing and ECDSA are required regardless of lib chosen.  They would simply be replaced with...  code that did the same thing under name other than "OpenSSL" or "GLib."

If I don't understand the problem space then you don't understand the solution space. GLib's ADT heavily depend on glibc-style memory allocation and both depend on GCC extensions and cross-compiling with MinGW instead of Visual Studio or any other native Windows compiler.

Both GLib and OpenSSL are extremely resource intensive and the only way to get respectable performance out of them is by running on a host with paging MMU and TLB. This excludes majority of standalone / lightweight OSes.

The problem/solution space doesn't demand OpenSSL. It requires BigNums, ECDSA, SHA-256. Any multiuser OS is just a serious hindrance for an lightweight implementation on a lean hardware. All those tasks are best performed not on the main CPU but on the attached GPU or DSP array. When the GPU can be directly called without the overhead of going through OpenGL/OpenCL the whole new world of possibilities opens up. As far as I know currently only PowerVR GPUs have an open source toolchain. This means Texas Instruments OMAP chips. In a year or two probably someone will do the same for the Broadcom VideoCore chips in Raspberry Pi. The performance and power efficiency gained from MIMD implemenations will be immense.

It will soon be building on MacOSX and Windows.
The salient point for Windows is: is it going to be possible to build it using a native Windows compiler like Visual Studio, OpenWATCOM or Borland/Embarcadero? Because another MinGW-only deliverable will be impossible to integrate into the Windows ecosystem.
legendary
Activity: 1470
Merit: 1005
Bringing Legendary Har® to you since 1952
First thing, i like the idea very much.

One small suggestion though:
Do you think you could give a distinct (and cool) name for this library?  There are several now and it gets confusing.  For example:

libcoin - Gronager
libccoin - jgarzik
libbitcoin - genjix
cbitcoin - MathhewLM

How about GarzikBitcoinLib?

+1

Seriously jgarzik, these names are getting more and more confusing.

How about calling it glibbitcoin, jgbitcoin or libbitcoin-jg ?
legendary
Activity: 1596
Merit: 1091
Thanks to denisx on IRC, who helped get picocoin/libccoin going on FreeBSD and GLib < 2.30 as well.
legendary
Activity: 2128
Merit: 1068
It continues to be true for all known compilers I've seen or heard of, when compared with a simple, native machine integer as used by the current implementation.
OK, show of hands. You must have heard of the existience of Microsoft Visual Studio C++/C compiler. I wrote a short test program that tests strong-typed little-endian integer on the Intel architecture, which will be the most important case.
Code:
#include

union le_int {
unsigned char b[4];
int i;
void operator =(int a) { i = a; }
operator int() { return i; }
};

int lehmer1(int n)
{
int r = 1;
while (n--)
r = (16807 * r) % 2147483647;
return r;
}

int lehmer2(int n)
{
union le_int r;

r = 1;
while (n--)
r = (16807 * r) % 2147483647;
return r;
}

void main()
{
printf("%d\n",lehmer1(2112));
printf("%d\n",lehmer2(2112));
}

and the generated "Release Win32" code, with everything at default:

Code:
00311000  push        edi 
printf("%d\n",lehmer1(2112));
00311001  mov         ecx,840h 
00311006  mov         edx,1 
0031100B  mov         edi,7FFFFFFFh 
00311010  imul        edx,edx,41A7h 
00311016  mov         eax,edx 
00311018  cdq 
00311019  idiv        eax,edi 
0031101B  dec         ecx 
0031101C  jne         main+10h (0311010h) 
0031101E  push        edx 
0031101F  push        312100h 
00311024  call        dword ptr ds:[312090h] 
0031102A  add         esp,8 
printf("%d\n",lehmer2(2112));
0031102D  mov         ecx,840h 
00311032  mov         edx,1 
00311037  imul        edx,edx,41A7h 
0031103D  mov         eax,edx 
0031103F  cdq 
00311040  idiv        eax,edi 
00311042  dec         ecx 
00311043  jne         main+37h (0311037h) 
00311045  push        edx 
00311046  push        312100h 
0031104B  call        dword ptr ds:[312090h] 
00311051  add         esp,8

Could anyone here can give an example of a popular compiler that produces different code for those two little functions? Maybe the test program needs to be somehow changed to trip up the optimizer? I'm seriously asking. I have code like this (and pure C equivalent with #define macros instead of operators()) in wide deployment for many years now on a variety of platforms.

I really only have my Windows laptop with me. I would have to ask people to unpack and boot my usual development environment for remote access.

Meanwhile I'm going to see if I can trip-up the Visual Studio somehow by rewriting this code in various ways.
hero member
Activity: 868
Merit: 1007
Very nice!  looking forward to playing around with it.
legendary
Activity: 1596
Merit: 1091
Everything builds and tests pass on MacOS X 10.7.  Good job on this code, you've written the client Satoshi should have.

Appreciated.  Thanks in particular for testing OSX.  That was a big item on the checklist.

member
Activity: 118
Merit: 10
Everything builds and tests pass on MacOS X 10.7.  Good job on this code, you've written the client Satoshi should have.
legendary
Activity: 1596
Merit: 1091
For starters, you'll have heck of a time to remove your dependency on glib.h from GTK.

You mean "remove your dependency on glib.h from libccoin", I presume?  GTK has no relevance to this, besides being the administrative umbrella project for GLib.

Removing the GLib dependency is straightforward for any developer.  GLib is used for a few ADTs like variable length strings, hash tables and arrays.

Quote
Both GLib and OpenSSL are heavy duty harvester combines. Removing the dependencies on them will at least halve the resource usage.

For a RedHat-ter at least make the code work with newlib instead of glib. The required knowledge should be no more than the internal phone call away for you. This includes a whole sermon on why you don't want to build a dependency on them.

Not sure you understand the problem space...  First, newlib is wholly different from GLib, and does not provide the ADTs that GLib provides.  Second, replacing GLib and OpenSSL would not change resource usage much at all.  ADTs, Big Integer support, SHA hashing and ECDSA are required regardless of lib chosen.  They would simply be replaced with...  code that did the same thing under name other than "OpenSSL" or "GLib."

Quote
The performance loss for passing unions around may have been true for GCC 2.96. I'm not normally working with Linux kernel and/or GCC.

It continues to be true for all known compilers I've seen or heard of, when compared with a simple, native machine integer as used by the current implementation.

Quote
All in all, I apologise for jumping in into the battle between you and MatthewLM. Regretfully I can't easily build neither of yours code, because I'm almost exclusively working on the closed-source platforms like Windows, MacOS, etc. and standalone environments not relying on GCC/GLIB/autogen.

It will soon be building on MacOSX and Windows.  Will post when this happens (volunteers willing to post building outputs are helpful).  As stated, it is a developer-only release right now, so there are plenty of rough edges.

Quote
But it is a shame that another C implementation of Bitcoin is useless for the developers of Bitcoin hardware wallets.

I respectfully disagree with that assessment.  Anyone looking at the code may see precisely what dependencies need replacing, for an embedded environment.  If anyone actually developing hardware or embedded bitcoin solutions wishes to contact me, I would be happy to illustrate how it would work.

legendary
Activity: 1596
Merit: 1091
So am I understanding right? I could run picocoin and it would be lightweight like Electrum but also support the Bitcoin network by interacting on the network like a full satoshi client. I've been using Electrum mostly to get away from having to use around 5GB disk space and for quick install/startup but I'd really prefer to be supporting Bitcoin more by being another validating node in it's network.

picocoin is only a lightweight client.  It is not a "full node" and probably never will be.

The included library, libccoin, does provide all the tools and gadgetry needed to build your own "full node" bitcoin client that takes 5GB disk space, if you have the programming chops to do that.

legendary
Activity: 2128
Merit: 1068
This seems quite an exaggeration, with no supporting evidence of what needs a major rework.  I do embedded programming for the day job, so I don't see this.  The Linux kernel is as bare metal as it gets, and must work in all environments, all platforms.  Smiley
For starters, you'll have heck of a time to remove your dependency on glib.h from GTK. But there's more. You could actually try to compile your code from within the Linux kernel tree to see for yourself.

For a die-hard Linux developer like you I think the minimum definition of bare-metal would be uClinux. Full Linux doesn't count anymore for a true embedded development.

The main chore of embedded use is simply the dependencies, most notably OpenSSL, and to a lesser extent GLib.  Once you have that, or its replacement, embedded use is straightforward.
Both GLib and OpenSSL are heavy duty harvester combines. Removing the dependencies on them will at least halve the resource usage.

For a RedHat-ter at least make the code work with newlib instead of glib. The required knowledge should be no more than the internal phone call away for you. This includes a whole sermon on why you don't want to build a dependency on them.

Using "union" decreases the performance and quality of code output for zero gain.  It is trivial enough to use sparse with a type specifier to guarantee endian purity, like the Linux kernel does, if that is desired ("__le32" etc.)

The code has long since been verified to work on big endian.
As I've noted: I've forgot that you are one of the very few Bitcoin developers with regular access to big-endian hardware. I'm thinking that you will continue to be the only one developer who can test picocoin on the big-endian architecture. The mojibake-endian problem will continue to badly affect the future contributors/users of your library if you aren't going to use your own medicine (that is "sparse" and "__le32")

The performance loss for passing unions around may have been true for GCC 2.96. I'm not normally working with Linux kernel and/or GCC.

But I have a great, positive experience with code of the following style(C++ used for brevity, ... expands to long #ifdef trees using platform- and compiler-specific intrinsics)
Code:
union be_int {
  unsigned char b[4];
  int i;
  inline operator =(int a) { ... }
  inline operator int() { ... }
};
The explicit strong-typing of endianness blocks the vast majority of possible endian errors and allows to produce meaningfull, endian-bug-free contributions from people who are exclusively using little-endian hardware. In my (rather extensive) experience it actually increases the performance because programmers can visualize the data structures better and can spend more time designing the algorithms instead of debugging trivial omissions or transpositions.

All in all, I apologise for jumping in into the battle between you and MatthewLM. Regretfully I can't easily build neither of yours code, because I'm almost exclusively working on the closed-source platforms like Windows, MacOS, etc. and standalone environments not relying on GCC/GLIB/autogen. But it is a shame that another C implementation of Bitcoin is useless for the developers of Bitcoin hardware wallets.
hero member
Activity: 784
Merit: 1009
firstbits:1MinerQ
So am I understanding right? I could run picocoin and it would be lightweight like Electrum but also support the Bitcoin network by interacting on the network like a full satoshi client. I've been using Electrum mostly to get away from having to use around 5GB disk space and for quick install/startup but I'd really prefer to be supporting Bitcoin more by being another validating node in it's network.
hero member
Activity: 686
Merit: 500
Wat
Do you think you could give a distinct (and cool) name for this library?  There are several now and it gets confusing.  For example:

libcoin - Gronager
libccoin - jgarzik
libbitcoin - genjix
cbitcoin - MathhewLM

How about GarzikBitcoinLib?

Just call it gbitcoin  Smiley
legendary
Activity: 1596
Merit: 1091
But even the definitions in your "ccoin" include files are deeply incompatible with any standalone environment, even if it uses GCC toolchain. Major rework would be required even now, when there's just couple of hundred lines of code.

This seems quite an exaggeration, with no supporting evidence of what needs a major rework.  I do embedded programming for the day job, so I don't see this.  The Linux kernel is as bare metal as it gets, and must work in all environments, all platforms.  Smiley

The main chore of embedded use is simply the dependencies, most notably OpenSSL, and to a lesser extent GLib.  Once you have that, or its replacement, embedded use is straightforward.

Quote
The big-endian compatibility is already super-confusing. Istead of using uint32_t in bu256_t use an "union le_int32" to at least flag the mixing of little endian uint32_t with guint32_t.

Using "union" decreases the performance and quality of code output for zero gain.  It is trivial enough to use sparse with a type specifier to guarantee endian purity, like the Linux kernel does, if that is desired ("__le32" etc.)

Quote
1) from the start build also on Mac Leopard or Snow Leopard to verify the endian-neutrality with "-arch ppc" with Rosetta on Intel CPU. Edit: Never mind, I forgot that you have access to the Linux on POWER and z/Arch through your association with RedHat.

The code has long since been verified to work on big endian.

legendary
Activity: 2128
Merit: 1068
Yes, embedded usage is another target.

Each module in libccoin is carefully designed to minimize internal dependencies.  The core data structures, address generation, script execution and transaction validation are wholly independent of any filesystem or network design.

These modules may be used on a non-POSIX flash filesystem, with zero network support, today.

picocoin (the client) requires certain network, filesystem and process features, but libccoin (the library) does not.  This is by design.
But even the definitions in your "ccoin" include files are deeply incompatible with any standalone environment, even if it uses GCC toolchain. Major rework would be required even now, when there's just couple of hundred lines of code.

The big-endian compatibility is already super-confusing. Istead of using uint32_t in bu256_t use an "union le_int32" to at least flag the mixing of little endian uint32_t with guint32_t.

From my extensive commercial experience of porting from Linux/Posix/Intel-only code to anything standalone or lightweight OS will unfortunatly go through the "first deny then rewrite from almost scratch" stages.

If I may suggest two things:

1) from the start build also on Mac Leopard or Snow Leopard to verify the endian-neutrality with "-arch ppc" with Rosetta on Intel CPU. Edit: Never mind, I forgot that you have access to the Linux on POWER and z/Arch through your association with RedHat.
2) obtain any embedded development/simulation environment. If you have any Xilinx Spartan6 boards (with memory) laying around you could use them to test using MicroBlaze CPU with Xilinx EDK/SDK.

Thank you for yor time.
legendary
Activity: 1596
Merit: 1091
Quote
AES encryption is applied to the wallet.  Passphrase is specified via environment variable PICOCOIN_PASSPHRASE.

Is this secure? My understanding is that the environment isn't a safe place to store/transmit information.

That's temporary.  It will input via fd like GPG soon.

legendary
Activity: 1974
Merit: 1029
Quote
AES encryption is applied to the wallet.  Passphrase is specified via environment variable PICOCOIN_PASSPHRASE.

Is this secure? My understanding is that the environment isn't a safe place to store/transmit information.
legendary
Activity: 1304
Merit: 1014
Do you think you could give a distinct (and cool) name for this library?  There are several now and it gets confusing.  For example:

libcoin - Gronager
libccoin - jgarzik
libbitcoin - genjix
cbitcoin - MathhewLM

How about GarzikBitcoinLib?
Pages:
Jump to: