Bitcoind CentOS Clean Compile On AWS EC2

anton000

newbie

Activity: 21

Merit: 0

Quote from: neonzeon on September 10, 2013, 03:02:29 AM

31337,
Thanks for showing the compile-from-source after force-installing the "improved" openssl library.

However, what we would like to do is to install the openssl library as well as boost, boost-devel, boost-thread, db4 and db4-devel in a "chroot" environment and then build the bitcoind in that environment.

That way, you don't overwrite your default openssl CentOS library (which is generally not recommended)

I think it will more or less require the following six steps:

1. Compile the ECDSA openssl library with mock as per previous instructions. Installed in the build root, i.e. DO NOT overwrite your normal openssl library.

2. Download bitcoind source and create a "spec" file for it so it can be converted into an rpm package.

3. In the "spec" file, specify the following libraries as buildrequires: boost, boost-devel, boost-thread, db4, db4-devel and the ECDSA-capable openssl library. (http://www.rpm.org/max-rpm/ch-rpm-inside.html). I also assume you have to note boost and db4 as library requires (if you link then dynamically). Install the libraries (using mock?)

4. Compile the bitcoind source package in the same chroot environment - this ensures it can find the new openssl library. Mock seems to be the right tool for the job.

5. Ensure that "our" openssl library is STATICALLY linked - this prevents runtime link attempts to the system openssl library, which cannot do elliptic curves.

6. Install the compiled bitcoind binary rpm.

I'm not yet 100% sure how the script for the above 6 steps will look like, but it will be similar to yours, with the exception of using rpmbuild and/or mock in the make process.

If anyone who knows mock/rpmbuild/chroot want to attempt to script the above 6 steps, please jump right in!!!

any updates on your progress on this sh script?

hypes

full member

Activity: 168

Merit: 100

i had some problems saying the patch was already applied but i think this was because i ran the command twice (my bad)

i started again and i have successfully managed to build the rpm and also dogecoind Grin

Many many thanks for your uber-detailed instructions which are probably the only ones on the net, yet again. (been happening a lot lately). Kudos.

hypes

full member

Activity: 168

Merit: 100

this has really helped but i'm getting an error running this last command with the second patch applied

cd ~ ; /usr/bin/mock --rebuild openssl-1.0.0-27.el6.EC.1.src.rpmcd ~ ; /usr/bin/mock --rebuild openssl-1.0.0-27.el6.EC.1.src.rpm

neonzeon

newbie

Activity: 23

Merit: 0

31337,
Thanks for showing the compile-from-source after force-installing the "improved" openssl library.

However, what we would like to do is to install the openssl library as well as boost, boost-devel, boost-thread, db4 and db4-devel in a "chroot" environment and then build the bitcoind in that environment.

That way, you don't overwrite your default openssl CentOS library (which is generally not recommended)

I think it will more or less require the following six steps:

1. Compile the ECDSA openssl library with mock as per previous instructions. Installed in the build root, i.e. DO NOT overwrite your normal openssl library.

2. Download bitcoind source and create a "spec" file for it so it can be converted into an rpm package.

3. In the "spec" file, specify the following libraries as buildrequires: boost, boost-devel, boost-thread, db4, db4-devel and the ECDSA-capable openssl library. (http://www.rpm.org/max-rpm/ch-rpm-inside.html). I also assume you have to note boost and db4 as library requires (if you link then dynamically). Install the libraries (using mock?)

4. Compile the bitcoind source package in the same chroot environment - this ensures it can find the new openssl library. Mock seems to be the right tool for the job.

5. Ensure that "our" openssl library is STATICALLY linked - this prevents runtime link attempts to the system openssl library, which cannot do elliptic curves.

6. Install the compiled bitcoind binary rpm.

I'm not yet 100% sure how the script for the above 6 steps will look like, but it will be similar to yours, with the exception of using rpmbuild and/or mock in the make process.

If anyone who knows mock/rpmbuild/chroot want to attempt to script the above 6 steps, please jump right in!!!

31337

newbie

Activity: 7

Merit: 0

and some commands to compile bitcoind

Code:

yum install boost boost-devel boost-thread db4 db4-devel
ln -s /usr/lib64/libboost_thread-mt.so /usr/lib64/libboost_thread.so
wget -qO- https://github.com/bitcoin/bitcoin/tarball/master --no-check-certificate | tar xzv --strip-components 1
mv src bitcoin
cd bitcoin
make -f makefile.unix bitcoind USE_UPNP=-

31337

newbie

Activity: 7

Merit: 0

you forgot a few lines

Code:

exit #to become root again
rpm --force -i /var/lib/mock/epel-6-x86_64/root/builddir/build/RPMS/openssl-1.0.0-27.el6.EC.1.x86_64.rpm #force reinstall of openssl
rpm --force -i /var/lib/mock/epel-6-x86_64/root/builddir/build/RPMS/openssl-devel-1.0.0-27.el6.EC.1.x86_64.rpm #force reinstall of openssl-devel

there could be another arch instead of x86_64 but you've got the point

also there should be some commands like "userdel -rf abcd; rm -rf /var/lib/mock/epel-6-x86_64" but its up to you.

thanks for manual

neonzeon

newbie

Activity: 23

Merit: 0

Success! OpenSSL with EC Builds on CentOS 6.4

Finally, openssl with EC builds on Centos! This is basically due to the help from in #centos on freenode IRC.

The method was actually VERY close to Kano's instructions (see previous posts)

We start with the CentOS openssl source RPM, and "install" it (which means splitting it into Sources (consisting of patches and a .tar.bz2) and Specs (containing openssl.spec)

Then we download a different, but version-synchronized .tar.gz version of the source from openssl.org. The new source will replace the existing .tar.bz2 file.

A patch file (to patch a test error) is also needed from openssl.org. The patch fixes this error http://openssl.6102.n7.nabble.com/OpenSSL-1-0-0b-testssl-fails-td11009.html

With the .tar.gz and the patch from openssl.org dowloaded, the openssl.spec file is edited to

1) Point to the newly downloaded .tar.gz as source
2) Change no-EC to enable-EC (enable ellptic curves)
3) Disable the "hobble" script (which erases elliptic curve source files)
4) Change the release number
5) Add a patch entry for the newly downloaded patch file

After that, we create a new source rpm using the modified spec file and replacement source (.tar.gz).

Finally, use mock, a Fedora tool used by Centos package builders, to build a new openssl package from the newly created source rpm.

Below are all the steps, condensed, that you need to take on a fresh CentOS 6.4 box, to compile openssl with enable-ec.

Code:

yum -y update  # Update all packages on new machine
yum -y groupinstall 'Development tools'
yum -y localinstall --nogpgcheck http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm # Install EPEL (EL6 extra packages) repository 
yum -y install fedora-packager  # Install mock from EPEL repository
userdel -rf abcd ; useradd -G mock abcd ; su abcd
cd ~ ; curl -O http://vault.centos.org/6.4/os/Source/SPackages/openssl-1.0.0-27.el6.src.rpm
/usr/bin/mock ~/openssl-1.0.0-27.el6.src.rpm
rm -rf /home/abcd/build ; mv /var/lib/mock/epel-6-x86_64/root/builddir/build/ /home/abcd ; # Move to a safe place
cd /home/abcd/build/SOURCES
curl -O http://www.openssl.org/source/openssl-1.0.0.tar.gz # Download corresponding source tarball from openssl
curl -o patch300.patch http://cvs.openssl.org/patchset?cn=19998 # Download this patch to fix a test error
cd ../SPECS
sed -i -e "s/no-ec/enable-ec/; s/no-ecdh/enable-ecdh/; s/no-ecdsa/enable-ecdsa/" openssl.spec # Enable EC
sed -i -e "s/^Source1: hobble-openssl/#&/; s/^%.SOURCE1. /#&/" openssl.spec # Disable the "hobble" script
sed -i -e "s/^Release.*dist\}/&.EC.1/" openssl.spec # Also change release number by adding .EC.1
sed -i -e "s/-usa.tar.bz2/.tar.gz/" openssl.spec # Change the source tarball
sed -i -e "s/^Patch78.*/&\nPatch300: patch300.patch\n/" openssl.spec # Add the new patch
sed -i -e "s/^%patch78.*/&\n%patch300 -p1 \n/" openssl.spec # Add the new patch again
/usr/bin/mock --buildsrpm --spec  ~/build/SPECS/openssl.spec --sources  ~/build/SOURCES # Do a source rebuild
cp /var/lib/mock/epel-6-x86_64/root/builddir/build/SRPMS/openssl-1.0.0-27.el6.EC.1.src.rpm /home/abcd
cd ~ ; /usr/bin/mock --rebuild openssl-1.0.0-27.el6.EC.1.src.rpm

Tip Jar: 1KaJZTmvvk2CPYmRPWALU63o2AZkMEMvJk

neonzeon

newbie

Activity: 23

Merit: 0

A kind soul in #centos on freenode irc with the handle pj was very helpful.

First, he confirmed the build errors in p_lib.c.

Then he created a patch to fix that error, and immediately ran into another error in p_lib.c.

So, it seems as if it's not just a simple fix in openssl.spec.

Some patches will have to be created. Off to learning how to patch...

neonzeon

newbie

Activity: 23

Merit: 0

DAY 7

I've been asking for help in several places regarding compiling openssl with EC.

A frequently asked question was "Why are you not using mock to build the package?".

So, after some searching, mock is a package builder from Fedora. It is installed from the "EPEL" repository.

To make a long story short, when you build with mock, you get the same errors.

Below is a summary of the steps needed to try building openssl (with EC) using mock on a fresh CentOS 6.4 box.

Build instructions: http://pastebin.centos.org/3016/

neonzeon

newbie

Activity: 23

Merit: 0

DAY 6

Focusing on building openssl.

Instantiate a fresh CentOS 6.4 server and then...

Code:

yum -y update  # Update everything
yum -y groupinstall 'Development tools'
yum -y install zlib-devel
yum -y install krb5-devel
 
cat /etc/centos-release
# CentOS release 6.4 (Final)
 
uname -rpo
# 2.6.32-279.el6.x86_64 x86_64 GNU/Linux
 
gcc --version
# gcc 4.4.7 20120313 (Red Hat 4.4.7-3)

We now have all the tools to rebuild openssl. We are going to build as user "abcd".

Code:

userdel -rf abcd ; useradd abcd ; su abcd # Wipe user abcd, create fresh user abcd, become user abcd
echo "%_topdir /home/$USER/rpmbuild" > ~/.rpmmacros ; # Tell rpm where to install files
echo "%packager Test User " >> ~/.rpmmacros  # Tell rpm a test user id
echo "%_query_all_fmt %%{name}-%%{version}-%%{release}.%%{arch}" >> ~/.rpmmacros # Tell rpm the architecture
cd ~ ; curl -O http://vault.centos.org/6.4/os/Source/SPackages/openssl-1.0.0-27.el6.src.rpm # Get openssl source
rpm --install openssl-1.0.0-27.el6.src.rpm 1>e1 2>e2 # Extract the source package to /rpmbuild/SOURCES
cd ~/rpmbuild/SPECS # This is where the package .spec file is extracted to
sed -i -e "s/no-ec/enable-ec/; s/no-ecdh/ /; s/no-ecdsa/ /" ~/rpmbuild/SPECS/openssl.spec
sed -i -e "s/^Source1: hobble-openssl/#&/; s/^%.SOURCE1. /#&/" ~/rpmbuild/SPECS/openssl.spec # Edit the .spec file
rpmbuild -bb openssl.spec 1>e1 2>e2 &    # Rebuild the package using the edited .spec file

# The above rebuild of openssl-1.0.0j takes a long time - at least 20 minutes.

jobs # Command to check if the compile is done
# [1]+ Running rpmbuild -bb openssl.spec > e1 2> e2 &

tail -f e2 # or "tail -f e1" (ctl-c to exit) command to check the build progress

# When the build finishes, the compiler output is in files e1 and e2 in ~/rpmbuild/SPECS

less -i e2 # Command to let you browse the build errors. Search for "error" with /error

# ====================================================================
# =========================== After the build ========================
# ====================================================================

Note: openssl-1.0.0-27.el6.src.rpm compiles 100% correct AS DOWNLOADED - evidence that the build environment is OK.

However, we make the following five changes (see the sed lines above) to enable elliptic curves:

s/no-ec/enable-ec/
s/no-ecdh/ /
s/no-ecdsa/ /
s/^Source1: hobble-openssl/#&/
s/^%.SOURCE1. /#&/

When the package is built with these five changes in, there are build errors.

THIS IS WHERE I NEED YOUR HELP.

Build errors here, near the end of the file: http://pastebin.com/8aGxEd6n

I find them extremely hard to resolve due to my inexperience.

p_lib.c:318: error: expected declaration specifiers or '...' before 'EC_KEY'

Looking at the preprocessor output with gcc -E, I saw that EC_KEY is undefined.

I found this handy: http://fossies.org/dox/openssl-1.0.1e/index.html

EC_KEY is defined in /crypto/include/ec.h

If I manually edit p_lib.c to include ec.h, this error vanishes.

So, for some reason, ec.h is not included in p_lib.c. I wonder why and how to properly fix it.

Comments and suggestions welcome.

neonzeon

newbie

Activity: 23

Merit: 0

Here is the last bit of stderr at the end of the rpmbuild command. It doesn't look too bad.

Any pointers and tips appreciated.

Code:


dsa_gen.c: In function 'FIPS_dsa_generate_pq':
dsa_gen.c:218: warning: unused variable 'mont'
dsa_gen.c:217: warning: unused variable 'g'

eng_openssl.c: In function 'bind_helper':
eng_openssl.c:144: warning: implicit declaration of function 'ECDH_OpenSSL'
eng_openssl.c:144: warning: passing argument 2 of 'ENGINE_set_ECDH' makes pointer from integer without a cast ../../include/openssl/engine.h:465: note: expected 'const struct CDH_METHOD *' but argument is of type 'int'
eng_openssl.c:147: warning: implicit declaration of function 'ECDSA_OpenSSL'
eng_openssl.c:147: warning: passing argument 2 of 'ENGINE_set_ECDSA' makes pointer from integer without a cast ../../include/openssl/engine.h:466: note: expected 'const struct CDSA_METHOD *' but argument is of type 'int'

evp_enc.c:102: warning: initialization from incompatible pointer type

p_lib.c:318: error: expected declaration specifiers or '...' before 'EC_KEY'
p_lib.c: In function 'EVP_PKEY_set1_EC_KEY':
p_lib.c:320: warning: implicit declaration of function 'EVP_PKEY_assign_EC_KEY'
p_lib.c:320: error: 'key' undeclared (first use in this function)
p_lib.c:320: error: (Each undeclared identifier is reported only once
p_lib.c:320: error: for each function it appears in.)
p_lib.c:322: warning: implicit declaration of function 'EC_KEY_up_ref'
p_lib.c: At top level: 
p_lib.c:326: error: expected '=', ',', ';', 'asm' or '__attribute__' before '*' token

make[2]: *** [p_lib.o] Error 1
make[1]: *** [subdirs] Error 1
make: *** [build_crypto] Error 1

error: Bad exit status from /var/tmp/rpm-tmp.aIHqo8 (%build)

neonzeon

newbie

Activity: 23

Merit: 0

In the "Briefer Build Instructions" above the last line is

Code:

rpmbuild -bb openssl.spec 1>e1 2>e2 &

The "&" at the end of the command means that the compile process is sent to the background.

That means you get your prompt back while the compile is still running.

You can pull the job to the foreground using the fg command.

The stdout and stderr messages are sent to files e1 and e2 respectively.

You can watch the stdout and stderr output during the compile with the following:

tail -f e1

tail -f e2

At the end of the compile, all status and error messages are saved in files e1 and e2.

You can browse them with

less -i e1

less -i e2

While browsing with less, you can use search, for instance type /error to search for "error" text.

neonzeon

newbie

Activity: 23

Merit: 0

Briefer Build Instructions

Code:

userdel -rf abcd ; useradd abcd ; su abcd # Wipe user abcd, create fresh user abcd, become user abcd
echo "%_topdir /home/$USER/rpmbuild" > ~/.rpmmacros ; echo "%packager Test User " >> ~/.rpmmacros  # Tell rpm where to install files and fake id
cd ~ ; curl -O http://vault.centos.org/6.4/os/Source/SPackages/openssl-1.0.0-27.el6.src.rpm # Download the source package from CentOS
rpm --install openssl-1.0.0-27.el6.src.rpm 1>e1 2>e2 # Extract the source package to /rpmbuild/SOURCES
cd ~/rpmbuild/SPECS # This is where the package .spec file is extracted to
sed -i -e "s/no-ec/enable-ec/; s/no-ecdh/enable-ecdh/; s/no-ecdsa/enable-ecdsa/" ~/rpmbuild/SPECS/openssl.spec # Edit the .spec file
sed -i -e "s/^Source1: hobble-openssl/#&/; s/^%.SOURCE1. /#&/" ~/rpmbuild/SPECS/openssl.spec # Edit the .spec file
rpmbuild -bb openssl.spec 1>e1 2>e2 & # Rebuild the package using the edited .spec file

neonzeon

newbie

Activity: 23

Merit: 0

Brief Build Instructions

Code:

userdel -rf abcd # Wipe user abcd
useradd abcd # Add a new user named abcd
su abcd # Become user abcd
mkdir ~/rpmbuild ; mkdir ~/rpmbuild/{BUILD,RPMS,SPECS,SOURCES,SRPMS} # Build the directory structure for rpmbuild
echo "%_topdir /home/abcd/rpmbuild" > ~/.rpmmacros
echo "%packager Test User " >> ~/.rpmmacros
cd ~ ; curl -O http://vault.centos.org/6.4/os/Source/SPackages/openssl-1.0.0-27.el6.src.rpm
rpm --install openssl-1.0.0-27.el6.src.rpm
cd ~/rpmbuild/SPECS
sed -i -e "s/no-ec/enable-ec/; s/no-ecdh/enable-ecdh/; s/no-ecdsa/enable-ecdsa/" ~/rpmbuild/SPECS/openssl.spec
sed -i -e "s/^Source1: hobble-openssl/#&/; s/^%.SOURCE1. /#&/" ~/rpmbuild/SPECS/openssl.spec
rpmbuild -ba openssl.spec 1>e1 2>e2 &

neonzeon

newbie

Activity: 23

Merit: 0

Kano's Instructions Modified / Annotated for CentOS 6.4

Before Kano's stuff we do some preparation for a new user as well as prepare per http://wiki.centos.org/TipsAndTricks/YumAndRPM

We start as the root user, create a new user "abcd" and then continue work as this new user. That way, everything is fresh.

[root] yum install rpm-build # Install the package builder
[root] useradd abcd # Add a new user named abcd
[root] su abcd # Become user abcd
[abcd] cd /home/abcd
[abcd] mkdir /home/abcd/rpmbuild/{BUILD,RPMS,SPECS,SOURCES,SRPMS} # Build the directory structure for rpmbuild
[abcd] # The next two lines create a special rpmbuild file that tells the packager where we want to build and who we are
[abcd] echo "%_topdir /home/abcd/rpmbuild" > /home/abcd/.rpmmacros
[abcd] echo "%packager Test User <[email protected]>" >> /home/abcd/.rpmmacros

1) Find the correct SRC RPM
Update fc16 to the latest everything and get openssl-1.0.0j-1.fc16.src.rpm from any mirror

[abcd] cd /home/abcd ; curl -O http://vault.centos.org/6.4/os/Source/SPackages/openssl-1.0.0-27.el6.src.rpm

2) rpm -Uvh openssl-1.0.0j-1.fc16.src.rpm

[abcd] rpm --install openssl-1.0.0-27.el6.src.rpm

Note: I'm getting a warning which I'm going to ignore (for now) : group mockbuild does not exist - using root

3) cd ~/rpmbuild/SPECS

[abcd] cd ~/abcd/rpmbuild/SPECS

4) vim openssl.spec

look for "./Configure" at the start of a line
(in 1.0.0j-1.fc16 it is line 219)
about 3 lines down from that you will see:
enable-cms enable-md2 no-idea no-mdc2 no-rc5 no-ec no-ecdh no-ecdsa \
change it to look like:
enable-cms enable-md2 no-idea no-mdc2 no-rc5 enable-ec enable-ecdh enable-ecdsa \

I'm going to use the sed editor because you can find it on almost any linux box and you can do these changes as little one-line scripts.

[abcd] sed -i -e "s/no-ec/enable-ec/; s/no-ecdh/enable-ecdh/; s/no-ecdsa/enable-ecdsa/" /home/abcd/rpmbuild/SPECS/openssl.spec

search for
Source1: hobble-openssl
(line 29) and change it (comment it out) to:
#Source1: hobble-openssl

search for
%{SOURCE1} > /dev/null
(line 133) and change it (comment it out) to:
#%{SOURCE1} > /dev/null

[abcd] sed -i -e "s/^Source1: hobble-openssl/#&/; s/^%.SOURCE1. /#&/" ~/rpmbuild/SPECS/openssl.spec

go back to the top and increase "Release:"

line 24 increment the "Release:" number
e.g. change
Release: 1%{?dist}
to
Release: 2%{?dist}

For now, I'm going to keep the version unchanged unless it seems important to do so. (can always force the package update)

5) install rpm-build
yum install rpm-build

We completed this step right in the beginning when we were still the root user, so can ignore now

6) cd ~/rpmbuild/SOURCES/

in ~/rpmbuild/SOURCES/ there is a file called "openssl-1.0.0j-usa.tar.xz"

rename it to "openssl-1.0.0j-usa.tar.xz.orig" (or whatever else you like)

[abcd] cd ~/rpmbuild/SOURCES/ ; mv openssl-1.0.0-usa.tar.bz2 old-usa-bz2-file-not-needed

get a full replacement for the tar file, at http://www.openssl.org/source/

http://www.openssl.org/source/openssl-1.0.0j.tar.gz

and put it in the directory: ~/rpmbuild/SOURCES/

then rename it to "openssl-1.0.0j-usa.tar.xz"

[abcd] cd ~/rpmbuild/SOURCES/ ; curl -O http://www.openssl.org/source/openssl-1.0.0j.tar.gz # Download sources from openssl.org
[abcd] gunzip -d openssl-1.0.0j.tar.gz ; bzip2 -z openssl-1.0.0j.tar # We have to convert to .bz2 because that is required in spec file openssl.spec
[abcd] mv openssl-1.0.0j.tar.bz2 openssl-1.0.0-usa.tar.bz2 # We have to rename it to this because that is what the spec file openssl.spec is looking for

7) cd ~/rpmbuild/SPECS

rpmbuild -bb openssl.spec

I'm going with -ba instead of -bb because -ba seems to build everything. Hope that doesn't bite us later.

[abcd] cd ~/rpmbuild/SPECS
[abcd] rpmbuild -ba openssl.spec

DOH - The instructions break here. The problem is as follows:

In the original openssl-1.0.0-usa.tar.bz2, all the source get extracted to rpmbuild/SOURCE/openssl-1.0.0

In the downloaded openssl-1.0.0j (which we rename), source go to rpmbuild/SOURCE/openssl-1.0.0j <<< note the extra "j" at the end.

I somehow have to modify the "root" directory in the tar file to be openssl-1.0.0 instead of openssl-1.0.0j.

Here's the clue from the rpmbuild output. The last thing that works is the extraction of the archive.
It then tries a cd openssl-1.0.0 which fails. However the directory rpmbuild/BUILD/openssl-1.0.0j (note the j at the end) exists and is populated with the source files.

Code:

+ /usr/bin/bzip2 -dc /home/abcd/rpmbuild/SOURCES/openssl-1.0.0-usa.tar.bz2 | /bin/tar -xf -
+ cd openssl-1.0.0
error:  line 38:  cd: openssl-1.0.0: No such file or directory

Below is a fairly ugly hack as a workaround attempt.
Basically, I take the wrongly installed openssl-1.0.0j, rename it to1.0.0 and then reconvert to bz2.

Code:

 cd /home/abcd/rpmbuild/BUILD
 mv openssl-1.0.0j openssl-1.0.0 # Get rid of the "j" at the end
 tar -cvf openssl-1.0.0j.tar openssl-1.0.0 # Archive to tar
 bzip2 -z openssl-1.0.0j.tar # Convert to bz2
 mv openssl-1.0.0j.tar.bz2 ~/rpmbild/SOURCES/ openssl-1.0.0-usa.tar.bz2 # Rename and move to SOURCES

With this hack, during rpmbuild, the archive extracts correctly to BUILD/openssl-1.0.0.

The build now fails because the patches in the SOURCES directory do not fully correlate to the source in he "fake" openssl-1.0.0-usa.bz2 we created.

This seems to be the flaw in kano's instructions, and we'll have to figure that out.

For those interested, here is some output from the build process so far http://pastebin.com/z5nx844J

Cool

You now have the RPM files you need in ~/rpmbuild/RPMS/

I wish...

[abcd] ls ~/rpmbuild/RPMS/
[abcd] echo woo hoo !!!

neonzeon

newbie

Activity: 23

Merit: 0

DAY 4...

I'm a little frustrated being so close AND so far away.

However, the Bitcoin Mayor of Las Vegas, Tuxavant, summed up my feelings nicely in this thread https://bitcointalksearch.org/topic/m.939240

Quote

Even so, this will prove to be a very enlightening and educational experience for me as I've wanted to help maintain other packages too. This ECDSA thing has motivate me to finally learn how this stuff works.

I've already learned more about packages and source RPMs than what I thought I would have to

Amazingly, this is a classic problem according to Kano, who has been running Linux for 16 years.

Quote

Since there really is no one who has written this, that is easy to find on the net, for the last 7+ years this issue has existed ...

Note to self and others: "that is easy to find on the net" I'm trying to remember the search terms I originally typed in to find this information. I think it was some text from the build error during the compile.

If you remember what you typed to get here, please add a comment with the text "My Search Terms" somewhere prominent.

Below is Kano's instructions repeated. It's for Fedora Core 16 though. I'm going to analyze and annotate them with what I've read so far.

I suspect one of the key reasons he was successful was because he did everything as root.
The CentOS instructions explicitly say that's living dangerously.
From the "Yum and RPM Tricks" at http://wiki.centos.org/TipsAndTricks/YumAndRPM

Quote

14. Get set up for rebuilding packages as yourself, not root

Sometimes you just have to rebuild that package - maybe only to use some configuration option which just isn't there in the official package. Or because you have found some great package which you really cannot find in the repositories, but the site only gives you RPMs for another distribution. So you have to grab the src.rpm and rebuild it for yourself. But you really do not want to do it as root. So here's how to rebuild your packages in your home directory - with your own user account.

With that warning in mind, Kano's Instructions from https://bitcointalksearch.org/topic/m.946289

Code:

Build openssl-1.0.0j with EC on fc16
------------------------------------

All this is run from root.

1) Find the correct SRC RPM

Update fc16 to the latest everything and get
 openssl-1.0.0j-1.fc16.src.rpm
from any mirror

or later if there is a later version
(this was written 7-Jun-2012)

N.B. these details are specific for openssl-1.0.0j
but most developers should be able to adapt this to later versions

2) rpm -Uvh openssl-1.0.0j-1.fc16.src.rpm

3) cd ~/rpmbuild/SPECS

4) vim openssl.spec

look for "./Configure" at the start of a line
(in 1.0.0j-1.fc16 it is line 219)
about 3 lines down from that you will see:
 enable-cms enable-md2 no-idea no-mdc2 no-rc5 no-ec no-ecdh no-ecdsa \
change it to look like:
 enable-cms enable-md2 no-idea no-mdc2 no-rc5 enable-ec enable-ecdh enable-ecdsa \

search for
 Source1: hobble-openssl
(line 29) and change it (comment it out) to:
 #Source1: hobble-openssl

search for
 %{SOURCE1} > /dev/null
(line 133) and change it (comment it out) to:
 #%{SOURCE1} > /dev/null

go back to the top and increase "Release:"

line 24 increment the "Release:" number
e.g. change
 Release: 1%{?dist}
to
 Release: 2%{?dist}

5) install rpm-build
yum install rpm-build

6) cd ~/rpmbuild/SOURCES/

in ~/rpmbuild/SOURCES/ there is a file called "openssl-1.0.0j-usa.tar.xz"

rename it to "openssl-1.0.0j-usa.tar.xz.orig" (or whatever else you like)

get a full replacement for the tar file, at http://www.openssl.org/source/

 http://www.openssl.org/source/openssl-1.0.0j.tar.gz

and put it in the directory: ~/rpmbuild/SOURCES/

then rename it to "openssl-1.0.0j-usa.tar.xz"

7) cd ~/rpmbuild/SPECS

 rpmbuild -bb openssl.spec

8) You now have the RPM files you need in ~/rpmbuild/RPMS/*/

I'll start annotating in the next post.

neonzeon

newbie

Activity: 23

Merit: 0

First Bitcoind Compile Attempt on CentOS 6.4

Woohoo! Today I found some time to prepare for an initial compile of bitcoind from https://github.com/bitcoin/bitcoin.github.com.

1. First I wiped the existing AWS EC2 machine, re-instantiating a fresh CentOS 6.4 machine on the Amazon Cloud and logging in via ssh/putty.

Code:

[root]$ echo "Your fresh AWS EC2 machine is $(cat /proc/version*)"
Your fresh AWS EC2 machine is Linux version 2.6.32-279.el6.x86_64 ([email protected])

2. Install dependencies as far as I could figure them out

Code:

yum update
yum install 'Development tools'
yum install openssl-devel
yum install boost
yum install db4

3. Add a new user abc and pull the bitcoin source code from github

Code:

adduser abc; su abc; mkdir src; cd src
git clone git://github.com/bitcoin/bitcoin.git
cd /home/abc/src/bitcoin/src

4. Change the variable USE_UPNP in file makefile.unix so the library miniUPNP is not included.

I apologise for using the obscure sed editor here but this way anyone can accurately duplicate the result.

Code:

 sed -i -e "s/^USE_UPNP.*/USE_UPNP:=-/" makefile.unix

5. Finally, try a compile of bitcoind as per the instructions in the bitcoin/doc directory

Code:

[abc]$ cd /home/abc/src/bitcoin/src
[abc]$ make -f makefile.unix

6. The result? Well, a bunch of files compiled happily until this:

Code:

g++ -c -O2 -pthread -Wall -Wextra -Wformat -Wformat-security -Wno-unused-parameter -g -DBOOST_SPIRIT_THREADSAFE -D_FILE_OFFSET_BITS=64 -I/home/abc/src/bitcoin/src -I/home/abc/src/bitcoin/src/obj -DUSE_IPV6=1 -I/home/abc/src/bitcoin/src/leveldb/include -I/home/abc/src/bitcoin/src/leveldb/helpers -DHAVE_BUILD_INFO -fno-stack-protector -fstack-protector-all -Wstack-protector -D_FORTIFY_SOURCE=2  -MMD -MF obj/alert.d -o obj/alert.o alert.cpp

alert.cpp:6:53: warning: boost/algorithm/string/classification.hpp: No such file or directory
alert.cpp:7:46: warning: boost/algorithm/string/replace.hpp: No such file or directory
alert.cpp:8:29: warning: boost/foreach.hpp: No such file or directory

In file included from alert.h:13, from alert.cpp:11:

util.h:23:29: warning: boost/version.hpp: No such file or directory
util.h:24:28: warning: boost/thread.hpp: No such file or directory
util.h:25:32: warning: boost/filesystem.hpp: No such file or directory

Doh. It seems as if a bunch of boost files are not found.

7. Not going to give up that easily. Go back to root, yum-install boost-development package, try again.

Code:

[abc]$ exit # To root user
[root]# yum install boost-devel
Package boost-devel-1.41.0-17.el6_4.x86_64 installed
[root]# su abc # Become user abc again
[abc]$ cd /home/abc/src/bitcoin/src # where the makefile is
[abc]$ make -f makefile.unix

The above errors go away!

Code:

/bin/sh ../share/genbuild.sh obj/build.h
g++ -c -O2 -pthread -Wall -Wextra -Wformat -Wformat-security -Wno-unused-parameter -g -DBOOST_SPIRIT_THREADSAFE -D_FILE_OFFSET_BITS=64 -I/home/abc/src/bitcoin/src -I/home/abc/src/bitcoin/src/obj -DUSE_IPV6=1 -I/home/abc/src/bitcoin/src/leveldb/include -I/home/abc/src/bitcoin/src/leveldb/helpers -DHAVE_BUILD_INFO -fno-stack-protector -fstack-protector-all -Wstack-protector -D_FORTIFY_SOURCE=2  -MMD -MF obj/key.d -o obj/key.o key.cpp
key.cpp:5:27: warning: openssl/ecdsa.h: No such file or directory
key.cpp:16: error: ‘EC_KEY’ was not declared in this scope
key.cpp:55: error: ‘ECDSA_SIG’ was not declared in this scope

The new errors are kind-of expected, because I read somewhere that openssl gets compiled without EC (elliptic curves) due to some patent issue.

8. New quest would be to find an openssl with EC turned on.

In https://bitcointalksearch.org/topic/m.991835 Jeff Garzik, bitcoin core dev team guru, says:

Quote

From: Jeff Garzik, bitcoin core dev team
Re: ECDSA dropped out of openssl 1.0.0b June 26, 2012, 05:39:39 PM

The preferred "fixes" for Red Hat, CentOS, Fedora systems are, if you want to do it yourself,

1) Download SRPM
2) Download associated source code from openssl.org
3) Edit SPECS/openssl.spec,
   a) replacing source tarball filename with the downloaded one
   b) comment out all references to source1
   c) remove the "no-ec" stuff from the configure line
4) rebuild with "rpmbuild -ba SPECS/openssl.spec" or similar
5) install build rpms found in RPMS/

Not yet sure what all those steps would translate to on my AWS EC2 CentOS box, but will be finding out over the next few days.

8. Ok, think I have Jeff's step 1 figured out - it's downloading a .src.rpm fetch from the CentOS mother code vault

Code:

[root]# su abc # Currently root, become user abc
[abc]$ cd /home/abc # Go to abc's home directory
[abcl]$ # Use curl to download the openssl SRPM (Source RPM) package
[abc]$  curl -O http://vault.centos.org/6.4/os/Source/SPackages/openssl-1.0.0-27.el6.src.rpm
(4.11 MB/s) - “openssl-1.0.0-27.el6.src.rpm” saved [3419292/3419292]

Comments & tips welcome.

I also read this http://wiki.centos.org/TipsAndTricks/YumAndRPM - everything is not 100% clear yet but 14.1 seems important.

Quote

14. Get set up for rebuilding packages as yourself, not root
Sometimes you just have to rebuild that package - maybe only to use some configuration option which just isn't there in the official package.
So here's how to rebuild your packages in your home directory - with your own user account.

Material to read:

http://wiki.centos.org/HowTos/SetupRpmBuildEnvironment
http://wiki.centos.org/HowTos/RebuildSRPM

Thanks TrevorH1

9. So, below a first attempt at taking apart a Source RPM package...

Code:

[root]# yum install rpm-build                                                   # from the instructions at http://wiki.centos.org/HowTos/SetupRpmBuildEnvironment
[root]# yum install redhat-rpm-config                                           # from the instructions at http://wiki.centos.org/HowTos/SetupRpmBuildEnvironment
[root]# su abc # become the non-root user
[abc]$ rpm -ivh /home/abc/openssl-1.0.0-27.el6.src.rpm                  # from the instructions at http://bradthemad.org/tech/notes/patching_rpms.php

Holy cow, that seems like a lot of work. Former east coast racing sailor, the guitar-playing Bradthemad says on his site

Quote

Part of the philosophy behind RPM is that the build process should be repeatable, as explained here. It may be small consolation when you just want make a simple change on your own system, where building once is good enough, but that's the way it is.

Ktks, got it Bradthemad. Time to suck it up and code on. And if you're still following this post you're as hardcore as yours truly to get bitcoind going on CentOS

10. Start fiddling with the source in openssl SRPM

Code:

These steps have been deleted

11. Now have to figure out which config (or source) files to edit in order to re-enable the elliptic curve code in openssl on CentOS 6.4...

Hero Member kano on bitcointalk: https://bitcointalksearch.org/topic/bounty-openssl-with-ec-for-fedorarhcentos-36-btc-85228 says we have to change

enable-ec enable-ecdh enable-ecdsa

Also, it seems that you have to replace the .bz2 file in the source with a .gz fresh file from www.openssl.org, and then update the openssl.spec file to reflect that.

Code:

[abc]$ cd /home/abc/rpmbuild/SOURCES
[abc]$ mv  openssl-1.0.0-usa.tar.bz2  openssl-1.0.0-usa.tar.bz2-original
[abc]$ curl -O http://www.openssl.org/source/openssl-1.0.0j.tar.gz
[abc]$ mv  openssl-1.0.0j.tar.gz  openssl-1.0.0-usa.tar.gz 

In SOURCES/openssl.spec:

search for
 Source1: hobble-openssl
(line 29) and change it (comment it out) to:
 #Source1: hobble-openssl

search for
 %{SOURCE1} > /dev/null
(line 133) and change it (comment it out) to:
 #%{SOURCE1} > /dev/null

search fpr 
  Source: openssl-%{version}-usa.tar.bz2
(line nnn) and change it to
Source: openssl-%{version}j.tar.gz

Thank you kano!

12. A few more changes to the openssl.spec file and the we rebuild the package with rpmbuild.

Code:

[abc]$ sed -i -e "s/no-ec/enable_ec/; s/no-ecdh/enable-ecdh/; s/no-ecdsa/enable-ecdsa/" /home/abc/rpmbuild/SPECS/openssl.spec  # Used sed editor to delete the no-ecdsa and no-ec directives from the specfile
[abc]$ cd /home/abc/rpmbuild
[abc]$ rpmbuild -ba SPECS/openssl.spec  # From instructions at http://wiki.centos.org/HowTos/RebuildSRPM

The above rpmbuild of the openssl package results in an error

Code:

+ umask 022
+ cd /home/abc/rpmbuild/BUILD
+ LANG=C
+ export LANG
+ unset DISPLAY
+ cd /home/abc/rpmbuild/BUILD
+ rm -rf openssl-1.0.0
+ /bin/tar -xf -
+ /usr/bin/gzip -dc /home/abc/rpmbuild/SOURCES/openssl-1.0.0j.tar.gz
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ cd openssl-1.0.0
/var/tmp/rpm-tmp.IOaGqm: line 38: cd: openssl-1.0.0: No such file or directory
error: Bad exit status from /var/tmp/rpm-tmp.IOaGqm (%prep)

It's close, but not exactly there. From the above, when /usr/bin/gzip -dc /home/abc/rpmbuild/SOURCES/openssl-1.0.0j.tar.gz runs, the archive gets extracted to openssl-1.0.0j instead of openssl-1.0.0. Therefore, when the cd openssl-1.0.0 happens, a "no such directory" is thrown, because everything is in openssl-1.0.0j.

So, seems close, but not 100% there. Comments welcomed.

neonzeon

newbie

Activity: 23

Merit: 0

DAY 3...

I found this link http://wiki.centos.org/HowTos/RebuildSRPM on the CentOS wiki.

Quote

Even if the software is not available in CentOS, you should always try to get or build a RPM for the software, since the advantages of using a package management system will compensate for the work you will have building the package.

I agree, so Google found me Tom van der Woerdt http://bitcoin.stackexchange.com/questions/10467/how-do-i-go-about-installing-a-bitcoin-daemon-in-centos-linux

He says

Quote

This is the bitcoin.spec file I use to build these packages

Code:

Name:           bitcoin
Version:        0.8.1
Release:        1%{?dist}
Summary:        bitcoin
License:        GPL
URL:            http://bitcoin.org
Source0:        http://sourceforge.net/projects/bitcoin/files/Bitcoin/bitcoin-%{version}/bitcoin-%{version}-linux.tar.gz
%description
bitcoin
%package qt
Summary:        bitcoin-qt
%description qt
bitcoin-qt
%prep
%setup -q -n bitcoin-%{version}-linux
%build
%install
mkdir -p %{buildroot}%{_bindir}
%ifarch x86_64
cp -R bin/64/* %{buildroot}%{_bindir}
%endif
%ifarch i686
cp -R bin/32/* %{buildroot}%{_bindir}
%endif
%clean
rm -rf %{buildroot}
%files
%doc README COPYING
/usr/bin/bitcoind
%files qt
/usr/bin/bitcoin-qt
%changelog

I think this is the way to go, though I would really like to build a non-qt version of bitcoind.

This might be a good start and a first attempt to know more about how an rpm package works.

neonzeon

newbie

Activity: 23

Merit: 0

Quote from: serp on June 21, 2013, 10:42:12 AM

I wanted to mention it would probably be super easy if you wanted to just use one of their Ubuntu server images.
You could just install the O/S then type 'sudo apt-get install bitcoind' and probably be done.

Serp, if you have access to a bitcoind built on ubuntu, could you post the output of a

Code:

ldd bitcoind

command?

In fact, any ldd output, especially on a 6.4 CentOS would be great.

serp

sr. member

Activity: 245

Merit: 250

@serp

A few things to watch if you are using the free tier is the number of I/O's you use on your EBS (you get 2 million/month free) and your bandwidth (you get 15 GB/month free). I've not tried it but if you plan to run it continuously then you might want to keep an eye on these things. I'd be interested in hearing how it goes.

Also, I understand if you want to stick with CentOS, but I wanted to mention it would probably be super easy if you wanted to just use one of their Ubuntu server images. You could just install the O/S then type 'sudo apt-get install bitcoind' and probably be done.

Topic: Bitcoind CentOS Clean Compile On AWS EC2 (Read 7401 times)