Author

Topic: Bitcoin Full Node Security Practices? (DevOps) (Read 424 times)

legendary
Activity: 2268
Merit: 1092
January 17, 2020, 08:27:41 PM
#12
Each coin runs on its own server with its own security measures (ports, user, rpc info, allowedips, etc).

The security measures you've described are implemented at the application level, which means you're still trusting that each coin client will do the right thing wrt security. (Whether it be a deliberate backdoor, or an exploit). You should still look into a more restrictive container or chroot based solution to run your client, to minimise the effects of a compromised client. Perhaps also a separate hardware firewall connecting your different servers, so a partial or full compromise on one cannot be used as a launching pad to access another server on the local network.

I would also considering disallowing direct RPC access, and perform all actions via an intermediate API, which only permits abstract actions such as confirm deposit to X, or withdraw to X, possibly communicating with another backend server (eg "does account X have at least X funds available") to audit the API request.
newbie
Activity: 17
Merit: 3
bitcoind: Runs under its own user

This makes it sound like you'll be running multiple daemons on the one server, relying only on the file permissions to deny or grant access? You really should be completely isolating each separate coin, so that a compromise of one wallet won't affect the others on the server.

I don't know about Linux containerization, but under FreeBSD you can use 'jail' to run an extreme chroot: only the coin client and a couple of required libraries (as well as the application data) are visible to the process. This means that a compromised coin client can't open a shell (no /bin/sh), can't recompile anything (no cc), and can't start any kind of remote access daemon such as ftpd or nc. Lock each one up tight.

Each coin runs on its own server with its own security measures (ports, user, rpc info, allowedips, etc). The idea is if in the worst case scenario someone is able to get into one of the servers they won't be able to do anything with the others. We're using VPN/tunneling to our servers as they are under a private subnet so you must have explicit access within our office to access the servers.

We will be having a hot wallet (which would be the server) for smaller amounts as well as cold wallet (offline storage) for larger amounts.

The non-stop sequence of micro-architectural side channels on intel-- plus a moderate number on other vendors makes me pretty skeptical of having private keys on anything that isn't physically isolated.

Private keys will be stored in Amazon KMS with CloudHSM to encrypt/decrypt, a little expensive but well worth the security.
legendary
Activity: 4004
Merit: 1250
Owner at AltQuick.com
You really should be completely isolating each separate coin, so that a compromise of one wallet won't affect the others on the server.

This is what pwned Crypsty.
staff
Activity: 4284
Merit: 8808
The non-stop sequence of micro-architectural side channels on intel-- plus a moderate number on other vendors makes me pretty skeptical of having private keys on anything that isn't physically isolated.
legendary
Activity: 2268
Merit: 1092
bitcoind: Runs under its own user

This makes it sound like you'll be running multiple daemons on the one server, relying only on the file permissions to deny or grant access? You really should be completely isolating each separate coin, so that a compromise of one wallet won't affect the others on the server.

I don't know about Linux containerization, but under FreeBSD you can use 'jail' to run an extreme chroot: only the coin client and a couple of required libraries (as well as the application data) are visible to the process. This means that a compromised coin client can't open a shell (no /bin/sh), can't recompile anything (no cc), and can't start any kind of remote access daemon such as ftpd or nc. Lock each one up tight.
newbie
Activity: 17
Merit: 3
gmaxwell thank you very much for your input! You have very good points that I will be looking into more detail. One thing we have been looking at is Bitbox security architecture, they did a public presentation were they exposed that one of the best security ideas is to consider every form of attack as if the attacker is already in the system, as you put it in one of your examples. As for the advice forum advice, we are definitely having a big security company go through our software and do some security tests as well as provide advice. I do find it helpful to get advice from people who are directly involved in blockchain and coin development projects through this forum.
staff
Activity: 4284
Merit: 8808
You should bind rpc address to localhost.
That's been the default for a long time.

Quote
What else would you take a look at to make sure the node is as secure as possible?

I strongly recommend using a firewall node:   Basically you run two bitcoin nodes instead of one.

An exterior one which talks to the outside world (probably no inbound connections, only only with a mixture of random connections and some addnodes),  and an inside node which your service runs against. The inside node connects only to the firewall node.

This protects your sensitive internal systems from some possible vulnerabilities in the node software. It also increases your flexibility in dealing with upgrades. E.g. if there is a softfork in bitcoin you can upgrade just the firewall node ... and reduce the risk of disrupting your operations by upgrading the inside node on a relaxed time frame.

If you're accepting low confirm count transactions from potentially untrustworthy users you will need to invest in network stability monitoring. You should know automatically if there are chain forks. You should also have redundant and diverse internet connectivity to make your bitcoin nodes hard to partition.

Seriously consider using encryption so that the system can't come up in a vulnerable state if power-cycled without intervention from your staff.  This might result in downtime if there is an unplanned reboot, but that's better than getting compromised.  Many methods of compromise result in a reboot-- e.g. social engineering data center remote hands into changing the root password on a host.  Several big users have been saved by having an encrypted wallet without the decryption key stored on the host (so they had to log in to unlock the wallet).

Datacenter facilities are often unprepared to handle the security requirement of cryptocurrency are are often shockingly easy to social engineer into tampering with hosts.  I know many parties that keep all their interactions with hosts under alternative names so that an attacker impersonating you doesn't know what names to use.

Obviously make sure you have good online AND offline backups and that your backups don't compromise your security.

Since you mention altcoins you should be aware that in the past some altcoins have been outright trojans and many are not maintained anywhere near as well as Bitcoin and have had serious vulnerabilities.  You should consider the possibility that someone hacks into one of your nodes and take measures to limit their ability to escalate their access to other parts of your system.   If FooCoinD ships a backdoor and everyone's FooCoins are stolen everywhere people might not hold that too harshly against you.  If the Bitcoin's you're holding due are also lost due to that attack it may look a lot less good for you.

"RPC allow from: Specific IP"  -- if you're running RPC to separate hosts you should setup an encrypted tunnel, unless the connection is otherwise via a physically isolated and secure network.

As far as other security stuff goes... standard host security applies though most advice isn't really made for targets as vulnerable as cryptocurrency users. I'm a fan of remote syslogging onto a system with no remote access (or extremely limited remote access).  Disable password authentication, ideally use ssh keys + U2F.

For bitcoincore.org we don't just use sshkey+u2f but we require two different U2Fs so that anyone that logs in has to get another user to help them log in.

Automate everything you can to minimize the number of people and frequency of manual interventions on your highly secure hosts. Don't give staff access that you don't need them to have just because you like them or they're important to the company.

Assume your staff's hosts will get compromised (this is something U2F helps a lot with).  It's worth time to setup tripwires-- logging that gets triggered (and sent out as alerts) when various commands get run that legit users would virtually never run but hackers might, selinux is very good at doing this. Make sure you don't flood yourself with false positives for any alarms you put in, however.

Consider pre-authoring sweep transactions that much each of your hot-wallet outputs over some nominal value to some secure cold wallet address(es), then don't broadcast these transactions but store them someplace you can easily get to in the event of a compromise. I've seen some hacked services struggle with tension between locking the hackers out of their systems and keeping their own access in order to move funds somewhere safer. It would be a lot better if you could just have a trip wire cause a fast shutdown of your wallet hosts, while separate systems could start sweeping funds out of harms way.

Consider having a canary wallet: e.g. a wallet.dat you put in a very visible location on every one of your supposedly secure systems, which contains enough funds to be worth stealing... which you monitor for theft. Hopefully a hacker that makes it onto your systems will find that first, rather than the really valuable stuff and can't resist stealing it and alerting you to their presence.

Keeping hackers out is obviously the highest goal but detecting, understanding, and mitigating compromise fast should be a close second.

And finally: forum comments aren't a replacement for competent professional security advice.

legendary
Activity: 1456
Merit: 1010
Ad maiora!
You should bind rpc address to localhost.

rpcbind=127.0.0.1 (or whatever localhost address)
rpcallowip=restric connection to rpc server from specific IP address. You can add more than one rpcallowip fields in bitcoin's config file.

-rpcbind=[:port]

Bind to given address to listen for JSON-RPC connections. Do not expose the RPC server to untrusted networks such as the public internet!

This option is ignored unless -rpcallowip is also passed. Port is optional and overrides -rpcport. Use [host]:port notation for IPv6. This option can be specified multiple times (default: 127.0.0.1 and ::1 i.e., localhost)
newbie
Activity: 17
Merit: 3
Debian is slower to release updates as they make sure they are using binaries which are very well tested and therefore avoid as much as possible security issues, etc.
hero member
Activity: 2086
Merit: 761
To boldly go where no rabbit has gone before...
IMHO,

i would have a frontend server with an API, and backend server with a wallet.

Mandatory firewall in between that would only let API calls through, and ofc the wallet network ports, needed for blockchain updates and such
legendary
Activity: 1612
Merit: 1608
精神分析的爸
Restrict SSH strictly to the static IPs that need SSH access via hosts.allow / hosts.deny .

Also make sure to restrict RPC access properly and secure/encrypt the communication between bitcoind and your rpc endpoint by proper means like an IPSec tunnel, SSH tunnel etc.
newbie
Activity: 17
Merit: 3
I need to run several coin nodes (Bitcoin, Bitcoin Cash, Ethereum, Dash, Tether, etc.) on the cloud. These nodes will all have full transaction indexes in order to view transactions and create/verify our own. I understand each node stores its wallet if you want it to, in this case we will, as we are creating an exchange. I wanted to know from others experience, what are the best security practices to avoid transfer of funds, etc.

Currently I create the server, per default everything is blocked in our firewall so I open port 22 (with specific IP access only) for SSH and 8333 for other peers to connect. Some people suggest using Bastion Hosts and others don't (see: https://medium.com/@henriksylvesterpedersen/you-dont-need-that-bastion-host-cd1b1717a9e7) in order to make it harder to access the server. I do make sure our config file uses hashed password with provided rpcauth.py in sources. What else would you take a look at to make sure the node is as secure as possible?

OS: Debian 9 Stretch
Ports: 22 (IP-bound), 8333 (all)
bitcoind: Runs under its own user
RPC pass: Hashed
RPC allow from: Specific IP
Jump to: