Topic: Bitcoind CentOS Clean Compile On AWS EC2 (Read 7402 times)

any updates on your progress on this sh script?

i had some problems saying the patch was already applied but i think this was because i ran the command twice (my bad)

i started again and i have successfully managed to build the rpm and also dogecoind 

Many many thanks for your uber-detailed instructions which are probably the only ones on the net, yet again. (been happening a lot lately). Kudos.

this has really helped but i'm getting an error running this last command with the second patch applied

cd ~ ; /usr/bin/mock --rebuild openssl-1.0.0-27.el6.EC.1.src.rpmcd ~ ; /usr/bin/mock --rebuild openssl-1.0.0-27.el6.EC.1.src.rpm

31337,
Thanks for showing the compile-from-source after force-installing the "improved" openssl library.

However, what we would like to do is to install the openssl library as well as boost, boost-devel, boost-thread, db4 and db4-devel in a "chroot" environment and then build the bitcoind in that environment.

That way, you don't overwrite your default openssl CentOS library (which is generally not recommended)

I think it will more or less require the following six steps:

1. Compile the ECDSA openssl library with mock as per previous instructions.  Installed in the build root,  i.e. DO NOT overwrite your normal openssl library.

2. Download bitcoind source and create a "spec" file for it so it can be converted into an rpm package.  

3. In the "spec" file, specify the following libraries as buildrequires: boost, boost-devel, boost-thread, db4, db4-devel and the ECDSA-capable openssl library. (http://www.rpm.org/max-rpm/ch-rpm-inside.html).  I also assume you have to note boost and db4 as library requires (if you link then dynamically).  Install the libraries (using mock?)

4. Compile the bitcoind source package in the same chroot environment - this ensures it can find the new openssl library. Mock seems to be the right tool for the job.

5. Ensure that "our" openssl library is STATICALLY linked - this prevents runtime link attempts to the system openssl library, which cannot do elliptic curves.

6. Install the compiled bitcoind binary rpm.

I'm not yet 100% sure how the script for the above 6 steps will look like, but it will be similar to yours, with the exception of using rpmbuild and/or mock in the make process.

If anyone who knows mock/rpmbuild/chroot want to attempt to script the above 6 steps, please jump right in!!!

and some commands to compile bitcoind

you forgot a few lines
there could be another arch instead of x86_64 but you've got the point

also there should be some commands like "userdel -rf abcd; rm -rf /var/lib/mock/epel-6-x86_64" but its up to you.

thanks for manual

Success! OpenSSL with EC Builds on CentOS 6.4

Finally, openssl with EC builds on Centos! This is basically due to the help from in #centos on freenode IRC.

The method was actually VERY close to Kano's instructions (see previous posts)

We start with the CentOS openssl source RPM, and "install" it (which means splitting it into Sources (consisting of patches and a .tar.bz2) and Specs (containing openssl.spec)

Then we download a different, but version-synchronized  .tar.gz version of the source from openssl.org. The new source will replace the existing .tar.bz2 file.

A patch file (to patch a test error) is also needed from openssl.org. The patch fixes this error http://openssl.6102.n7.nabble.com/OpenSSL-1-0-0b-testssl-fails-td11009.html

With the .tar.gz and the patch from openssl.org dowloaded, the openssl.spec file is edited to

1) Point to the newly downloaded .tar.gz as source
2) Change no-EC to enable-EC (enable ellptic curves)
3) Disable the "hobble" script (which erases elliptic curve source files)
4) Change the release number
5) Add a patch entry for the newly downloaded patch file

After that, we create a new source rpm using the modified spec file and replacement source (.tar.gz).

Finally, use mock, a Fedora tool used by Centos package builders, to build a new openssl package from the newly created source rpm.

Below are all the steps, condensed, that you need to take on a fresh CentOS 6.4 box, to compile openssl with enable-ec.

Code:

yum -y update  # Update all packages on new machine
yum -y groupinstall 'Development tools'
yum -y localinstall --nogpgcheck http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm # Install EPEL (EL6 extra packages) repository
yum -y install fedora-packager  # Install mock from EPEL repository
userdel -rf abcd ; useradd -G mock abcd ; su abcd
cd ~ ; curl -O http://vault.centos.org/6.4/os/Source/SPackages/openssl-1.0.0-27.el6.src.rpm
/usr/bin/mock ~/openssl-1.0.0-27.el6.src.rpm
rm -rf /home/abcd/build ; mv /var/lib/mock/epel-6-x86_64/root/builddir/build/ /home/abcd ; # Move to a safe place
cd /home/abcd/build/SOURCES
curl -O http://www.openssl.org/source/openssl-1.0.0.tar.gz # Download corresponding source tarball from openssl
curl -o patch300.patch http://cvs.openssl.org/patchset?cn=19998 # Download this patch to fix a test error
cd ../SPECS
sed -i -e "s/no-ec/enable-ec/; s/no-ecdh/enable-ecdh/; s/no-ecdsa/enable-ecdsa/" openssl.spec # Enable EC
sed -i -e "s/^Source1: hobble-openssl/#&/; s/^%.SOURCE1. /#&/" openssl.spec # Disable the "hobble" script
sed -i -e "s/^Release.*dist\}/&.EC.1/" openssl.spec # Also change release number by adding .EC.1
sed -i -e "s/-usa.tar.bz2/.tar.gz/" openssl.spec # Change the source tarball
sed -i -e "s/^Patch78.*/&\nPatch300: patch300.patch\n/" openssl.spec # Add the new patch
sed -i -e "s/^%patch78.*/&\n%patch300 -p1 \n/" openssl.spec # Add the new patch again
/usr/bin/mock --buildsrpm --spec  ~/build/SPECS/openssl.spec --sources  ~/build/SOURCES # Do a source rebuild
cp /var/lib/mock/epel-6-x86_64/root/builddir/build/SRPMS/openssl-1.0.0-27.el6.EC.1.src.rpm /home/abcd
cd ~ ; /usr/bin/mock --rebuild openssl-1.0.0-27.el6.EC.1.src.rpm

Tip Jar: 1KaJZTmvvk2CPYmRPWALU63o2AZkMEMvJk

A kind soul in #centos on freenode irc with the handle pj was very helpful.

First, he confirmed the build errors in p_lib.c.

Then he created a patch to fix that error, and immediately ran into another error in p_lib.c.

So, it seems as if it's not just a simple fix in openssl.spec.

Some patches will have to be created. Off to learning how to patch...

DAY 7

I've been asking for help in several places regarding compiling openssl with EC.

A frequently asked question was "Why are you not using mock to build the package?".

So, after some searching, mock is a package builder from Fedora. It is installed from the "EPEL" repository.

To make a long story short, when you build with mock, you get the same errors.

Below is a summary of the steps needed to try building openssl (with EC) using mock on a fresh CentOS 6.4 box.

Build instructions: http://pastebin.centos.org/3016/

DAY 6

Focusing on building openssl.

Instantiate a fresh CentOS 6.4 server and then...


We now have all the tools to rebuild openssl. We are going to build as user "abcd".


# The above rebuild of openssl-1.0.0j takes a long time - at least 20 minutes.
 
jobs  # Command to check if the compile is done
# [1]+  Running  rpmbuild -bb openssl.spec > e1 2> e2 &
 
tail -f e2 # or "tail -f e1" (ctl-c to exit)   command to check the build progress
 
# When the build finishes, the compiler output is in files e1 and e2 in ~/rpmbuild/SPECS
 
less -i e2  # Command to let you browse the build errors. Search for "error" with /error
 
# ====================================================================
# =========================== After the build ========================
# ====================================================================
 
Note: openssl-1.0.0-27.el6.src.rpm compiles 100% correct AS DOWNLOADED - evidence that the build environment is OK.
 
However, we make the following five changes (see the sed lines above) to enable elliptic curves:
 
s/no-ec/enable-ec/
s/no-ecdh/ /
s/no-ecdsa/ /
s/^Source1: hobble-openssl/#&/
s/^%.SOURCE1. /#&/
 
When the package is built with these five changes in, there are build errors.
 
THIS IS WHERE I NEED YOUR HELP.  
 
Build errors here, near the end of the file: http://pastebin.com/8aGxEd6n

I find them extremely hard to resolve due to my inexperience.

p_lib.c:318: error: expected declaration specifiers or '...' before 'EC_KEY'

Looking at the preprocessor output with gcc -E, I saw that EC_KEY is undefined.
 
I found this handy: http://fossies.org/dox/openssl-1.0.1e/index.html
 
EC_KEY is defined in /crypto/include/ec.h
 
If I manually edit p_lib.c to include ec.h, this error vanishes.

So, for some reason, ec.h is not included in p_lib.c. I wonder why and how to properly fix it.
 
Comments and suggestions welcome.

Here is the last bit of stderr at the end of the rpmbuild command. It doesn't look too bad. 

Any pointers and tips appreciated.

In the "Briefer Build Instructions" above the last line is


The "&" at the end of the command means that the compile process is sent to the background.

That means you get your prompt back while the compile is still running.

You can pull the job to the foreground using the fg command.

The stdout and stderr messages are sent to files e1 and e2 respectively.

You can watch the stdout and stderr output during the compile with the following:

tail -f e1

tail -f e2

At the end of the compile, all status and error messages are saved in files e1 and e2.

You can browse them with

less -i e1

less -i e2

While browsing with less, you can use search, for instance type /error to search for "error" text.

Briefer Build Instructions

Brief Build Instructions

Kano's Instructions Modified / Annotated for CentOS 6.4

Before Kano's stuff we do some preparation for a new user as well as prepare per http://wiki.centos.org/TipsAndTricks/YumAndRPM

We start as the root user, create a new user "abcd" and then continue work as this new user.  That way, everything is fresh.

[root] yum install rpm-build # Install the package builder
[root] useradd abcd # Add a new user named abcd
[root] su abcd # Become user abcd
[abcd] cd /home/abcd
[abcd] mkdir /home/abcd/rpmbuild/{BUILD,RPMS,SPECS,SOURCES,SRPMS} # Build the directory structure for rpmbuild
[abcd] # The next two lines create a special rpmbuild file that tells the packager where we want to build and who we are
[abcd] echo "%_topdir /home/abcd/rpmbuild" > /home/abcd/.rpmmacros
[abcd] echo "%packager Test User <[email protected]>" >> /home/abcd/.rpmmacros

1) Find the correct SRC RPM
Update fc16 to the latest everything and get  openssl-1.0.0j-1.fc16.src.rpm from any mirror

[abcd] cd /home/abcd ; curl -O http://vault.centos.org/6.4/os/Source/SPackages/openssl-1.0.0-27.el6.src.rpm

2) rpm -Uvh openssl-1.0.0j-1.fc16.src.rpm

[abcd] rpm --install openssl-1.0.0-27.el6.src.rpm

Note: I'm getting a warning which I'm going to ignore (for now) : group mockbuild does not exist - using root

3) cd ~/rpmbuild/SPECS

[abcd] cd ~/abcd/rpmbuild/SPECS

4) vim openssl.spec

look for "./Configure" at the start of a line
(in 1.0.0j-1.fc16 it is line 219)
about 3 lines down from that you will see:
 enable-cms enable-md2 no-idea no-mdc2 no-rc5 no-ec no-ecdh no-ecdsa \
change it to look like:
 enable-cms enable-md2 no-idea no-mdc2 no-rc5 enable-ec enable-ecdh enable-ecdsa \

I'm going to use the sed editor because you can find it on almost any linux box and you can do these changes as little one-line scripts.

[abcd] sed -i -e "s/no-ec/enable-ec/; s/no-ecdh/enable-ecdh/; s/no-ecdsa/enable-ecdsa/" /home/abcd/rpmbuild/SPECS/openssl.spec

search for
 Source1: hobble-openssl
(line 29) and change it (comment it out) to:
 #Source1: hobble-openssl

search for
 %{SOURCE1} > /dev/null
(line 133) and change it (comment it out) to:
 #%{SOURCE1} > /dev/null

[abcd] sed -i -e "s/^Source1: hobble-openssl/#&/; s/^%.SOURCE1. /#&/" ~/rpmbuild/SPECS/openssl.spec

go back to the top and increase "Release:"

line 24 increment the "Release:" number
e.g. change
 Release: 1%{?dist}
to
 Release: 2%{?dist}

For now, I'm going to keep the version unchanged unless it seems important to do so. (can always force the package update)

5) install rpm-build
yum install rpm-build

We completed this step right in the beginning when we were still the root user, so can ignore now

6) cd ~/rpmbuild/SOURCES/

in ~/rpmbuild/SOURCES/ there is a file called "openssl-1.0.0j-usa.tar.xz"

rename it to "openssl-1.0.0j-usa.tar.xz.orig" (or whatever else you like)

[abcd] cd ~/rpmbuild/SOURCES/ ; mv openssl-1.0.0-usa.tar.bz2 old-usa-bz2-file-not-needed

get a full replacement for the tar file, at http://www.openssl.org/source/

 http://www.openssl.org/source/openssl-1.0.0j.tar.gz

and put it in the directory: ~/rpmbuild/SOURCES/

then rename it to "openssl-1.0.0j-usa.tar.xz"

[abcd] cd ~/rpmbuild/SOURCES/ ; curl -O http://www.openssl.org/source/openssl-1.0.0j.tar.gz # Download sources from openssl.org
[abcd] gunzip -d openssl-1.0.0j.tar.gz ; bzip2 -z  openssl-1.0.0j.tar  # We have to convert to .bz2 because that is required in spec file openssl.spec
[abcd] mv openssl-1.0.0j.tar.bz2 openssl-1.0.0-usa.tar.bz2 # We have to rename it to this because that is what the spec file openssl.spec is looking for

7) cd ~/rpmbuild/SPECS

 rpmbuild -bb openssl.spec

I'm going with -ba instead of -bb because -ba seems to build everything. Hope that doesn't bite us later.

[abcd]  cd ~/rpmbuild/SPECS
[abcd] rpmbuild -ba openssl.spec

DOH - The instructions break here. The problem is as follows:

In the original openssl-1.0.0-usa.tar.bz2, all the source get extracted to rpmbuild/SOURCE/openssl-1.0.0

In the downloaded openssl-1.0.0j (which we rename), source go to rpmbuild/SOURCE/openssl-1.0.0j  <<< note the extra "j" at the end.

I somehow have to modify the "root" directory in the tar file to be openssl-1.0.0 instead of openssl-1.0.0j.

Here's the clue from the rpmbuild output. The last thing that works is the extraction of the archive.  
It then tries a cd openssl-1.0.0 which fails. However the directory rpmbuild/BUILD/openssl-1.0.0j (note the j at the end) exists and is populated with the source files.


Below is a fairly ugly hack as a workaround attempt.
Basically, I take the wrongly installed openssl-1.0.0j, rename it to1.0.0 and then reconvert to bz2.


With this hack, during rpmbuild, the archive extracts correctly to BUILD/openssl-1.0.0.

The build now fails because the patches in the SOURCES directory do not fully correlate to the source in he "fake" openssl-1.0.0-usa.bz2 we created.

This seems to be the flaw in kano's instructions, and we'll have to figure that out.

For those interested, here is some output from the build process so far http://pastebin.com/z5nx844J

You now have the RPM files you need in ~/rpmbuild/RPMS/

I wish...

[abcd] ls ~/rpmbuild/RPMS/
[abcd] echo woo hoo !!!

DAY 4...

I'm a little frustrated being so close AND so far away.

However, the Bitcoin Mayor of Las Vegas, Tuxavant, summed up my feelings nicely in this thread https://bitcointalksearch.org/topic/m.939240

I've already learned more about packages and source RPMs than what I thought I would have to

Amazingly, this is a classic problem according to Kano, who has been running Linux for 16 years.

Note to self and others: "that is easy to find on the net"  I'm trying to remember the search terms I originally typed in to find this information. I think it was some text from the build error during the compile.  

If you remember what you typed to get here, please add a comment with the text "My Search Terms" somewhere prominent.

Below is Kano's instructions repeated. It's for Fedora Core 16 though. I'm going to analyze and annotate them with what I've read so far.

I suspect one of the key reasons he was successful was because he did everything as root.  
The CentOS instructions explicitly say that's living dangerously.
From the "Yum and RPM Tricks" at http://wiki.centos.org/TipsAndTricks/YumAndRPM

With that warning in mind, Kano's Instructions from https://bitcointalksearch.org/topic/m.946289


I'll start annotating in the next post.

First Bitcoind Compile Attempt on CentOS 6.4

Woohoo! Today I found some time to prepare for an initial compile of bitcoind from https://github.com/bitcoin/bitcoin.github.com.

1. First I wiped the existing AWS EC2 machine, re-instantiating a fresh CentOS 6.4 machine on the Amazon Cloud and logging in via ssh/putty.


2. Install dependencies as far as I could figure them out


3. Add a new user abc and pull the bitcoin source code from github


4. Change the variable USE_UPNP in file makefile.unix so the library miniUPNP is not included.

I apologise for using the obscure sed editor here but this way anyone can accurately duplicate the result.


5. Finally, try a compile of bitcoind as per the instructions in the bitcoin/doc directory


6. The result? Well, a bunch of files compiled happily until this:


Doh. It seems as if a bunch of boost files are not found.

7. Not going to give up that easily. Go back to root, yum-install boost-development package, try again.


The above errors go away!


The new errors are kind-of expected, because I read somewhere that openssl gets compiled without EC (elliptic curves) due to some patent issue.  

8. New quest would be to find an openssl with EC turned on.

In https://bitcointalksearch.org/topic/m.991835 Jeff Garzik, bitcoin core dev team guru, says:

Not yet sure what all those steps would translate to on my AWS EC2 CentOS box, but will be finding out over the next few days.  


8. Ok, think I have Jeff's step 1 figured out - it's downloading a .src.rpm fetch from the CentOS mother code vault


Comments & tips welcome.

I also read this http://wiki.centos.org/TipsAndTricks/YumAndRPM - everything is not 100% clear yet but 14.1 seems important.

Material to read:

http://wiki.centos.org/HowTos/SetupRpmBuildEnvironment
http://wiki.centos.org/HowTos/RebuildSRPM

Thanks TrevorH1

9. So, below a first attempt at taking apart a Source RPM package...


Holy cow, that seems like a lot of work. Former east coast racing sailor, the guitar-playing Bradthemad says on his site

Ktks, got it Bradthemad. Time to suck it up and code on. And if you're still following this post you're as hardcore as yours truly to get bitcoind going on CentOS

10. Start fiddling with the source in openssl SRPM


11. Now have to figure out which config (or source) files to edit in order to re-enable the elliptic curve code in openssl on CentOS 6.4...


Hero Member kano on bitcointalk: https://bitcointalksearch.org/topic/bounty-openssl-with-ec-for-fedorarhcentos-36-btc-85228  says we have to change

enable-ec enable-ecdh enable-ecdsa  

Also, it seems that you have to replace the .bz2 file in the source with a .gz fresh file from www.openssl.org, and then update the openssl.spec file to reflect that.

Thank you kano!

12. A few more changes to the openssl.spec file and the we rebuild the package with rpmbuild.


The above rpmbuild of the openssl package results in an error


It's close, but not exactly there.  From the above, when  /usr/bin/gzip -dc /home/abc/rpmbuild/SOURCES/openssl-1.0.0j.tar.gz  runs, the archive gets extracted to openssl-1.0.0j instead of openssl-1.0.0.  Therefore, when the cd openssl-1.0.0 happens, a "no such directory" is thrown, because everything is in openssl-1.0.0j.

So, seems close, but not 100% there.  Comments welcomed.

DAY 3...

I found this link http://wiki.centos.org/HowTos/RebuildSRPM on the CentOS wiki.


I agree, so Google found me Tom van der Woerdt http://bitcoin.stackexchange.com/questions/10467/how-do-i-go-about-installing-a-bitcoin-daemon-in-centos-linux

He says

I think this is the way to go, though I would really like to build a non-qt version of bitcoind.

This might be a good start and a first attempt to know more about how an rpm package works.

Serp, if you have access to a bitcoind built on ubuntu, could you post the output of a


command?

In fact, any ldd output, especially on a 6.4 CentOS would be great.

A few things to watch if you are using the free tier is the number of I/O's you use on your EBS (you get 2 million/month free) and your bandwidth (you get 15 GB/month free).  I've not tried it but if you plan to run it continuously then you might want to keep an eye on these things.  I'd be interested in hearing how it goes.

Also, I understand if you want to stick with CentOS, but I wanted to mention it would probably be super easy if you wanted to just use one of their Ubuntu server images.  You could just install the O/S then type 'sudo apt-get install bitcoind' and probably be done.