Topic: subvertx command line utilities (proof of concept using libbitcoin) (Read 17860 times)

sorry for the necro but are there any news about this tool?

Electrum now accepts plugins.
It would be awesome to write a txradar plugin, that connect to a server running txrad.
It would display the propagation rate in the client, in real time

bump!

I followed all the hints, and finally built libbitcoin library and installed it in ubuntu 12.04
But the last clone from subvertx git doesn't want to compile correctly. Is the latest version of subvertx compatible with latest version of libbitcoin?
The errors during compilation:

Gentlemen,

I think I've run into a serious bug in mktx (and let me apologize in advance if this is the wrong place to post - newbie warning!):

Simply put: Sometimes the coins are sent to the address 1111111111111111111114oLvT2 rather than to the specified receiving address. It happens only with certain addresses, my theory is that it happens when the second character in the address is greater than C - when the second character in the address is C or lower, it appears to work fine.

The example in the first post of this thread works fine. However, to recreate the problem try the same command, only change the second receving address, for example:

mktx create -p priv@c524c555aad1932c24c26ec20439a9caefc49f7c0df6d6bccced890ef980b45c:0 -k priv@keys/privkey -r 12oabCifvHuxzXtYVGhkxVfWZDvKcU743s:2000000 -r 1Def9ia5QENGf8CYVVnhRTNvpop67gs6Jp:58000000

Deserializing the output from that command you can see it has set the second output script to OP_DUP OP_HASH160 0000000000000000000000000000000000000000 OP_EQUALVERIFY OP_CHECKSIG.

BTW, I am using the subvertx ubuntu package on 12.04. I haven't set up my environment to be able to recompile the utilities and such, but I guess I'll have to dive into the source myself unless I hear back...

In either case, thanks for a great bunch of tools!

1) The postgresql_blockchain in libbitcoin (which subvertx uses) plugin's underlying library cppdb can use MySQL/postgres/sqlite/... MySQL doesn't exist simply because I wasn't using it. So it might be possible to remove the postgres only stuff to make it work with mysql but I haven't looked into it.

2) Nobody has compiled for windows yet, but there is no linux specific stuff (and there shouldn't be). The code is even made to work on big endian systems too. Getting a windows port would be good Feel free to hit me up on IRC.

Very interesting tools and something I am very interested to use with my own future open source project.

Just a couple of questions:

1) Any possibility of optionally using MySQL?

2) Will these tools work in Windows also?


Cheers,

Ian.

I recommend L-AGPL over LGPL.

It's basically the same as LGPL, except it applies even in cases where the software is used "as a service" (instead of as an installable program.)

See above for more info.

subvertx has a transaction radar tool like http://www.transactionradar.com

https://gitorious.org/libbitcoin/subvertx/blobs/master/src/txrad.cpp

it's a simple tool. just sits there connected to 100 nodes and monitors for tx inventories.

 I'll look into it

Ah, right. Thank you.

Genjix, are you not considering the Lesser/Library extension?

Thanks for the info. Yes, they provided me with the exemption. The bug I'm referring to isn't to do with OpenSSL though. It's that the AGPL requires not only that you give the sourcecode on demand, but that you're proactive in providing the sourcecode visibly. This is problematic because if you make a small modification while developing or building on a compilation based system (like gentoo) then the act of connecting to the p2p network means you have to provide source. There is no means to do this in the bitcoin sourcecode and the idea itself is problematic; every build must store the sourcecode in a bundle which is served using custom bitcoin protocol extensions.

From my understanding, modifying the AGPL for them is complicated and so far the last proposal was:


This means that you are able to provide the source code through some other means (such as sending it via email if requested) and it solves the issue.

------

In an ideal world, copyrights, patents, trademarks and censorship wouldn't exist. Proprietary software would not be able to support itself against rampant piracy. And I would put my sourcecode out in the open for anyone to use as they like.

However, we don't live in that ideal world. And I will use what little power I have in the form of the GPL to fight this asymmetric warfare. The balance is totally tilted away from our favour, so why wouldn't you use the only protection afforded to you as a developer.

Licenses like the GPL hack the law, twisting it to our will. It is the best example of subversion.

OpenSSL doesn't necessarily conflict with the GPL!  All you have to do is, provide a waiver with your license, allowing your users to link OpenSSL while using your library.

See the OT headers for examples of this, regarding OpenSSL as well as the Lucre library:  https://github.com/FellowTraveler/Open-Transactions/blob/master/LICENSE-AND-CREDITS.txt

Wget, for example, uses this "waiver" trick, and other software. Google for more info.
YOU can provide additional permissions on top of the AGPL (which is exactly how the LAGPL itself works.)
Since you can add permissions (NEVER restrictions) to any GPL license, then you may also be able to fix your AGPL "bug" using that technique.


GPL licenses make exemptions for other GPL licenses, for interoperability reasons. So LGPL is different than BSD in terms of this interoperability. (For example, the above-described waiver is not necessary with LGPL...)

FYI, here is my own "easy english" description of the L-AGPL's operation:

Whether you are distributing binaries that link to the OT-API, or whether you are using the OT-API in your website — so-called “software as a service” — either way, your own code may remain private, but any improvements to the OT library or API code itself must be made available open-source.

My understanding is that the BSD license does not do this -- IBM could take all the libbitcoin code tomorrow if it were under BSD license, and use it internally, without having to open any source code, and without having to deal with the developers of libbitcoin. They could then sell it as closed-source, at a profit, and cut out the original developers, and cut out the open source community. They would not have to make available any of their improvements to libbitcoin, but could keep these proprietary. IMO that is a big distinction between BSD and LGPL.

I'm not opposed to the BSD license, but the link I provided in the previous post demonstrates why the GPL license can be important when it comes to maintaining a thriving open-source community around a given piece of software. Especially convincing is the amount of money spent by (normally competing) corporations such as Microsoft, et al towards improving linux, where such entities would normally never invest funds to benefit their competitors, but instead would tend to choose a proprietary solution (or co-opt a piece of BSD-licensed code in order to create a proprietary solution, as described in that article.)

Thanks for the links, particularly Charles M. Hannum's comments on NetBSD licensing. It contrasts the opinions of O'Reily which have been quite influential over the decade. I had suggested a playful BSD-like license (Poetic License), but perhaps the Affero extensions and strong copyleft are necessary with respect to bitcoin, and the Lesser extension for libraries such as libbitcoin. Though isn't LGPL (in contrast to GPL) essentially BSD?

Thanks fellowtraveller Actually there are a bunch of licensing issues that have to be resolved. Me and the SFLC/FSF are trying to fix up the AGPL since there is a bug in it where bitcoin is concerned. Also OpenSSL conflicts with the GPL and LAGPL was discussed since they get a lot of requests for it.

But still got a lot of time since the project is far from maturity.

Congratulations on the excellent library work.

I also notice you have chosen the GNU Affero GPL for your license. (The 'AGPL'.)

I still use AGPL, but only for my server software. My OT library now uses the L-AGPL, which preserves the Affero SaaS protections, while allowing LGPL-like permissions in places where OT is used as a library. It's the best of both worlds!

You can read more about the LAGPL here: http://mo.morsi.org/blog/node/270

I think this is the best argument in favor of using the GPL for (certain) open-source projects: http://www.dwheeler.com/blog/2006/09/01/#gpl-bsd

Also, FYI I actually wrote to the FSF about this, and they directed me to the same LAGPL article.

The critical point, legally, is that the LAGPL is created by layering additional permissions on top of the AGPL.

(Versus adding restrictions to the LGPL, which would be removable downstream.)

In the case of the LAGPL, the additional permissions are also removable, but who is ever going to do THAT? Therefore, that is the proper structure for it.

Ahh I had the http and git address mixed up.  Silly of me.

Works fine now.  Thanks.

https://gitorious.org/libbitcoin/subvertx is the repo webpage, it contains all the info you need.

In short, here are the actual repo urls:

git:	git://gitorious.org/libbitcoin/subvertx.git
http:	https://git.gitorious.org/libbitcoin/subvertx.git
ssh:	[email protected]:libbitcoin/subvertx.git

Can you remind us the git repo please?  I tried https://gitorious.org/libbitcoin/subvertx with no success

Be advised, subvertx 0.1.0 is now tagged in the repository.

Gentoo ebuild for this version is available in the "bitcoin" overlay.
Binary packages are not available at this time and will possibly arrive with soon-to-be-tagged 0.1.1

This version explicitly requires libbitcoin 0.1.0 alpha 1.
See this announcement for more info.

amazing. just awesome.

This looks very useful.

Nicee!

You don't copy paste though. You write to a file then read it in.

i.e

mktx send < /tmp/serialised-tx

And that file can just as easily be a named pipe (which are faster and not really files).

Or you can run it through a base64 decoder:

echo ... | openssl base64 -d | mktx send

I don't need to derail your thread with my own development activities, so we could PM about this.  In a nutshell, I'm doing Tx construction on the command line, and I usually end up with potential Tx's I want to broadcast, depending whether my SelectCoins did what I wanted.  If so, then I copy the serialized-tx hex into the tool for broadcasting (right now, Nibor's webpage), otherwise I ignore it, tweak my code and try again.  Copying raw binary from ASCII displays always made me uncomfortable... but maybe it works...
(I see that you had direct output from another process in mind for piping into your mktx-send operations, but I didn't)

OK!

But why do you need hex? Just use "openssl base64 ..." to base64 decode your stream and pipe it on.

Also I don't think you can use echo ..., you'd need to use cat to read until EOF. You can create a named pipe then open the mktx send with that pipe in bash.

Good intro to named pipes: http://www.linuxjournal.com/article/2156

Thanks for thinking of me, genjix   How do I put the serialized transaction into the stdin?  raw binary characters?  Would it be:


It's a tad easier if I can pass around hex instead.  On the other hand, I will probably just pick apart your code, adapt to my client, and integrate it into my codebase   Until then, I'll probably use it for experimenting until I get to that point.  Looks really good, thanks for releasing this!

Made updates to subvertx (updated OP but posting below also). Packages have also been refreshed so you may wish to update.

I split the process between creating a transaction and actually broadcasting it into 2 parts rather one command (for etotheipi mainly ). This artifice allows us all the whizz-bang toys like offline transactions, piping a transaction to multiple sends, a flyweight client, piping transactions over network shares to a single bitcoind and whatever guile schemes you can imagine. In shell we trust.


There are 2 commands there. The 'create' actually constructs the transaction and then dumps its binary network format to STDOUT. 'send' reads from STDIN and sends it to the network. By default it sends to localhost:8333, but you can change it using the --host and --port options.

Creating a transaction

A transaction consists of inputs. You can use blockexplorer to look them up. Here are some examples.

1. We want to send from transaction c524c555aad1932c24c26ec20439a9caefc49f7c0df6d6bccced890ef980b45c's 0th output (which was sent to an address we own) to 2 addresses: 0.02 BTC to 12oabCifvHuxzXtYVGhkxVfWZDvKcU743s and 0.58 BTC to 14DDgj2r8WQEwfTDEjJFBn3wRnHmXzgB3z

There is 1 previous output:

00535291532821f2e4879cf670f61396be32b9579400ae1119497f36f268eb40:1

There are 2 recipients:

12oabCifvHuxzXtYVGhkxVfWZDvKcU743s:2000000
14DDgj2r8WQEwfTDEjJFBn3wRnHmXzgB3z:58000000

Note that we use internal BTC amounts (decimal value * 10^8 - see help text).

Now to spend that output, it has to have been sent to an address you own, hopefully generated using the priv tool earlier


If we wish to store that output we can use a redirection > to save it. Maybe store it on your USB and take it to another computer. Or we can pipe it straight to the send command.


The hash at the end is your transaction hash. If you do: tail -f .bitcoin/debug.log, or you look it up on bitcoincharts.com/bitcoin you will see that hash without the spaces (i.e CTRL-F for c53ea3 to see your transaction).

If you noticed, I loaded several private keys there using -k (or --keypair) KEYPAIR-NAME@FILENAME. You then reference which keypair belongs to which input this way.

2. Sending from three, different outputs to 14DDgj2r8WQEwfTDEjJFBn3wRnHmXzgB3z.

Previous outputs:

00535291532821f2e4879cf670f61396be32b9579400ae1119497f36f268eb40:1
637f001eb4cbe165946c02a56bcb73822610f5886516169f98da6252266b7d8a:1
85b423b9c8c5c5277575b87d94dbcd4f87c1be578756eff6a9fde8b7d55749fb:1

All the outputs (in this case) use a different private key: ./keys/foo, ./keys/bar and ./keys/abc.

We can load the keys and name them (to have a way to refer to them) using:

 -k foo@keys/foo
 -k bar@keys/bar
 -k abc@keys/abc

And then indicate to mktx which key belongs with which inputs from above:

foo@00535291532821f2e4879cf670f61396be32b9579400ae1119497f36f268eb40:1
bar@637f001eb4cbe165946c02a56bcb73822610f5886516169f98da6252266b7d8a:1
abc@85b423b9c8c5c5277575b87d94dbcd4f87c1be578756eff6a9fde8b7d55749fb:1

1 recipient:

14DDgj2r8WQEwfTDEjJFBn3wRnHmXzgB3z:60000000

Thanks, that looks like a really nice set of tools.

Updated the OP with the ebuild made by Kamil Domański.

No problem for me.  For now, I plan to have the satoshi client open anyway, so I can just connect to localhost and avoid peer discovery entirely.  In the future I might work on something more independent, but for now this is perfect!

I'm not too familiar with the various licenses and terminologies around them.  It looks like your code is also AGPL v3, so there shouldn't be any compatibility problems.  But I would like to do things right, and apply attribution regardless if I really need to -- I like to give credit where it's deserved.  It sounds like it won't be difficult (licensing-wise) to bring in some of your code.  I will battle the code details later.

@grondilu, I changed it according to your suggestions.


I think you want this:
https://gitorious.org/libbitcoin/subvertx/blobs/master/src/mktx.cpp

However it doesn't do node discovery yet, so you have to know who you're connecting to beforehand.

You will need to modify mktx to be a new program that reads in a byte stream for the raw transaction.

See this:

https://gitorious.org/libbitcoin/libbitcoin/blobs/master/tests/dialect-test.cpp

If there is a problem reading the stream then it will throw an exception called end_of_stream. That exception is thrown by the internal deserialising object inside that dialect object:

https://gitorious.org/libbitcoin/libbitcoin/blobs/master/include/bitcoin/util/serializer.hpp


Of course. It's a library. Don't worry about attribution.

Also, I changed the PPA from zgenjix -> genjix. Please update your sources.list and add the new keyserver
$ wget -q "http://keyserver.ubuntu.com:11371/pks/lookup?op=get&search=0x3D1978972EC26E7B" -O- | sudo apt-key add -

+1 To this.  I'm extremely interested in having a minimal networking interface just for getting txs not in the blockchain yet, and transmit new blocks that we just signed.  How much work would that be--it sounds like tests/net is like 90% of the way there.   If I'm working on an AGPL v3 project, will I be able to use your code (modified or not) in my project with proper attribution?

I've been dreading how I'm going to deal with networking in my client, but this sounds like it will be the perfect solution!

This looks awesome.  Is it possible to connect to another host, so that we can make a transaction without having bitcoin actually running nor installed on localhost?

This would be great, as it would be the cheapest (and possibly the safest) to use bitcoin.

I also think this should be related to the Bitcoin Off-The-Grid project.

You're a god damn rock star!

btw in libbitcoin there are a bunch of other tools but I didn't include them as they're more complicated to setup:

examples/poller
download the bitcoin blockchain into a postgresql database

examples/subvertx/balance
find the balance of a bitcoin address using the downloaded database after running the poller above

tests/net
connect, handshake (swap versions, wait for veracks) and then download the first 500 blocks.

These are a simple set of utilities for working with the bitcoin network/protocol. Together you can use them as a 'flyweight' client. I like small sets of inter-working programs because it is the UNIX philosophy about building small bricks which you can piece together to create a set of highly configurable inter-locking mechanism of programs and daemons.

This is a proof of concept built off of the work-in-progress libbitcoin.

Subvertx is a suite of 4 tools:
- poller, downloads the blockchain from a bitcoind into a PostgreSQL database.
- balance, queries the PostgreSQL database downloaded with poller to find the balance of a bitcoin address.
- priv, create a new private key, sign data, verify data or show the bitcoin address of that key.
- mktx, connects to a bitcoind and sends some money from an output to a new address. You can use this in conjunction with a site like blockexplorer.com as a tool to make transactions very simply. Or even make offline transactions or use it like a super lightweight client that can run on embedded devices.
- txrad, like the website http://www.transactionradar.com/. It connects to 100 random nodes and then monitors for transaction messages. Simply run txrad.

Show help:

Make a new private key, sign and verify a transaction hash:

Show your bitcoin address:

Send bitcoins (offline & flyweight transactions possible)

This is done in 2 steps: creating the transaction, then piping it to send the transaction somewhere. Pipes allow you to do really clever things like send to multiple nodes, send it over a network file share, buffer it to send later (like on USB stick with a different machine that doesn't have your private key).


There are 2 commands there. The 'create' actually constructs the transaction and then dumps its binary network format to STDOUT. 'send' reads from STDIN and sends it to the network. By default it sends to localhost:8333, but you can change it using the --host and --port options.

A transaction consists of inputs. You can use blockexplorer to look them up. Here are some examples.

1. We want to send from transaction c524c555aad1932c24c26ec20439a9caefc49f7c0df6d6bccced890ef980b45c's 0th output (which was sent to an address we own) to 2 addresses: 0.02 BTC to 12oabCifvHuxzXtYVGhkxVfWZDvKcU743s and 0.58 BTC to 14DDgj2r8WQEwfTDEjJFBn3wRnHmXzgB3z

There is 1 previous output:

00535291532821f2e4879cf670f61396be32b9579400ae1119497f36f268eb40:1

There are 2 recipients:

12oabCifvHuxzXtYVGhkxVfWZDvKcU743s:2000000
14DDgj2r8WQEwfTDEjJFBn3wRnHmXzgB3z:58000000

Note that we use internal BTC amounts (decimal value * 10^8 - see help text).

Now to spend that output, it has to have been sent to an address you own, hopefully generated using the priv tool earlier


If we wish to store that output we can use a redirection > to save it. Maybe store it on your USB and take it to another computer. Or we can pipe it straight to the send command.


The hash at the end is your transaction hash. If you do: tail -f .bitcoin/debug.log, or you look it up on bitcoincharts.com/bitcoin you will see that hash without the spaces (i.e CTRL-F for c53ea3 to see your transaction).

If you noticed, I loaded several private keys there using -k (or --keypair) KEYPAIR-NAME@FILENAME. You then reference which keypair belongs to which input this way.

2. Sending from three, different outputs to 14DDgj2r8WQEwfTDEjJFBn3wRnHmXzgB3z.

Previous outputs:

00535291532821f2e4879cf670f61396be32b9579400ae1119497f36f268eb40:1
637f001eb4cbe165946c02a56bcb73822610f5886516169f98da6252266b7d8a:1
85b423b9c8c5c5277575b87d94dbcd4f87c1be578756eff6a9fde8b7d55749fb:1

All the outputs (in this case) use a different private key: ./keys/foo, ./keys/bar and ./keys/abc.

We can load the keys and name them (to have a way to refer to them) using:

 -k foo@keys/foo
 -k bar@keys/bar
 -k abc@keys/abc

And then indicate to mktx which key belongs with which inputs from above:

foo@00535291532821f2e4879cf670f61396be32b9579400ae1119497f36f268eb40:1
bar@637f001eb4cbe165946c02a56bcb73822610f5886516169f98da6252266b7d8a:1
abc@85b423b9c8c5c5277575b87d94dbcd4f87c1be578756eff6a9fde8b7d55749fb:1

1 recipient:

14DDgj2r8WQEwfTDEjJFBn3wRnHmXzgB3z:60000000


poller

To use the poller you first need to import the schema which can be found in /usr/share/libbitcoin/bitcoin.sql (a pre-downloaded blockchain is here).


Poller is doing full blockchain validation. It will take a long time since it's an SQL database and it's validating all the blocks.

Balance needs the full blockchain otherwise you might not get an up-to-date balance reported back. It's fairly simple to use:

Ubuntu packages

The PPA can be viewed on launchpad.

Add these 2 lines to the end of your /etc/apt/sources.list


This PPA also has a package for libbitcoin-dev although the API is undocumented at the moment.

Gentoo ebuild

Kamil Domański has made a gentoo ebuild.


Source installation instructions

Source code for the non-Ubuntu paupers:
https://gitorious.org/libbitcoin/libbitcoin
https://gitorious.org/libbitcoin/subvertx

You will need:

  - g++ 4.6
  - cppdb
  - boost
  - libpq-dev
  - postgresql

Build cppdb using the instructions on their site.

  $ svn co http://cppcms.svn.sourceforge.net/svnroot/cppcms/cppdb/trunk cppdb-trunk
  $ cd cppdb-trunk
  $ mkdir build
  $ cd build

You can use -DCMAKE_INSTALL_PREFIX if you wish to install to a non-standard place such as a local directory.

  $ cmake -DCMAKE_INSTALL_PREFIX=/tmp/cppdb/ ..  -DPQ_BACKEND_INTERNAL=1
  $ make
  # make install

Clone repo and build.

  $ git clone git://gitorious.org/libbitcoin/libbitcoin.git
  $ cd libbitcoin
  $ autoreconf -i
  $ ./configure
  $ make
  # make install

Don't forget to initialise and create a database (see first post above for more details how).

  $ psql bitcoin < bitcoin.sql

Same for the subvertx suite

  $ git clone git://gitorious.org/libbitcoin/subvertx.git
  $ cd subvertx
  $ autoreconf -i
  $ ./configure
  $ make
  # make install

I can't stress enough for people to use this at their own peril as it's alpha software.

Thanks goes to,

Kamil Domański (kdomanski)
Jan Engelhardt (se7) <[email protected]>
Patrick Strateman (phantomcircuit)
Denis Roio (jaromil)
Amir Taaki (genjix) <[email protected]>