Five Technical Takeaways From The Bitcoin Bug

It’s been more than a week since the initial disclosure of CVE-2018-17144. In that time, more details have been filled in, including the fact that the bug could not only crash clients, it could also create inflation— and that the bug was discovered and responsibly reported by a developer working on the “Unlimited” Bitcoin Cash client.

There has also been much discussion in the crypto world about the nature of this bug and what lessons we should takeaway from it. This article will detail my biggest takeaways, looking at it primarily from a technical perspective.

#1 The risk of a critical software flaw disrupting Bitcoin, or another major crypto network, is still under-appreciated.

At the beginning of August, it was revealed that Bitcoin Core developer Corey Fields had discovered and disclosed a chain split but in Bitcoin Cash. In his write up detailing that incident, Corey had this to say:

Working through this bug, which certainly had the potential for catastrophe, has reaffirmed my belief that the threat of software bugs is severely underestimated in the cryptocurrency world. I’m presenting a detailed report of this incident… as a wake-up call to companies who have not adequately prepared for this type of scenario.

This seems especially prescient in light of this latest vulnerability, and I fear many people are still failing to account for the risk. I’ve seen many folks on Twitter say things like “bugs can always be fixed” or “whats the big deal— it wasn’t even exploited.” We should make no mistake: Bitcoin dodged a bullet.

#2 Bitcoin development is under-provisioned given the enormity of what is now at stake.

A couple of months ago, Naval Ravikant set off an uproar amongst Bitcoiners for saying that incentives for Bitcoin developers are out of whack. Rather than discussing a very good point, most seized on the use of the economic term “free-rider” to imply, incorrectly, that Naval was criticizing all Bitcoin “hodlers”.

When some in the community did get around to addressing Naval’s actual point, most insisted that Bitcoin did not have a developer problem, that the contributions of passionate experts are sufficient, and that no conversation around how to fund additional eyes on the code was needed. I tweeted about this earlier in the week.

 
 

The commit that introduced this vulnerability was made by a single developer and merged after a cursory review by only one other. This, despite the fact the change was to a critical section of the code dealing with transaction validation. It subsequently remained in the code for a eighteen months before being discovered by someone working for a different project.

For a project with a $100 Billion market cap, which aims to make itself a store of value for people’s wealth, and on which many of us are resting our hopes for a more decentralized monetary future, this is simply unacceptable. I’m not saying I have the answers for how to solve this problem, but we damn well better acknowledge there is a problem, here!

#3 It’s totally unclear whether having only one widely used implementation of Bitcoin is a good idea.

Bitcoin Core is far and away the most widely used client software on the Bitcoin network, with over 95% of all nodes running it. Other networks have much more diverse topologies, such as Ethereum, which has multiple widely used clients built in different languages and by different teams.

There is a downside to heterogeneous networks, most critically that a consensus difference between two or more clients could lead to a chain split. This vulnerability lends credence to those who argue the benefits to multiple clients is worth this risk.

 
 

Had the bug been exploited, virtually all clients on the Bitcoin network would have been affected. Only those still running very outdated versions of the software would have been safe. A heterogeneous network would be more robust to these sorts of vulnerabilities, and arguably might result in their discovery sooner as well.

#4 Mining is (still) centralized, and mining pool operators have an outsized impact on consensus.

There’s no real new information here, just a dramatic demonstration of the reality. When this bug was discovered, the Bitcoin Core developers reached out to the operators of a handful of mining pools. Once some were patched, it was “safe” to reveal the full nature of the bug, despite the fact that 85% full nodes (like my own!) had not yet been patched.

 

Bitcoin hashrate distribution via blockahin.com

 

Running a full node is still a good idea for one’s own security— you don’t have to trust anyone to carry out transactions. And it does impute some measure of greater decentralization to have more full nodes in operation. At the end of the day, though, those in control of the hash rate have an outsized control on consensus, and at the moment, that means just a handful of pool operators.

Something that should create some cognitive dissonance for those of us who care about decentralization: the fact the hash rate is relatively centralized actually made patching this issue easier.

#5 If/when a sophisticated nationstate decides to attack cryptocurrencies, things will get ugly, fast.

What are the odds that someone inside a government agency, be it the NSA, or an equivalent agency in China or Russia, knew about this vulnerability before it was discovered and patched? I’d say they’re relatively high. If I headed up one of those agencies, I would certainly have a team reviewing the code of Bitcoin and other major cryptocurrencies, looking for exactly these types of weaknesses.

Why? Because that would be my job! Bitcoin, and other networks, have become too important on the international stage not to warrant attention from such agencies. And make no mistake, while these agencies are bureaucratic, they still employ and provide resources to brilliant people who, working in small teams, can accomplish incredible feats.

If, or more likely, when a major nationstate decides to become hostile to crypto, things could get…messy

Talking Smart Contracts, Software Development, And The Future Of Crypto

This past summer, I was a guest on the QuantLayer podcast talking about software development and the broader crypto ecosystem. I really enjoyed the discussion and hope you will too!

 

Something I noticed when I went back and reviewed the transcript: Two of the things we opined about at the time have been partially addressed in only 2 months.

  1. The lack of recurring subscriptions for decentralized services is now being developed under EIP-1337.

  2. I complained about MetaMask’s UX, which has since gotten a major overall that makes it much better (though still far from ready for mainstream adoption).

Amazing to see how fast things move. Where will be in two years, let alone two more months? Buckle up!

A Developer's Guide To Running A Bitcoin Full Node And Local Block Explorer

The goal of this guide is set up a Bitcoin Node that is useful for tinkering with as a developer. If you complete this guide successfully, you'll have a node that is in sync with the network, contains the full history of the Bitcoin blockchain, and is fully validating all transactions.

 
Obligatory Image Evoking the idea of networked nodes

Obligatory Image Evoking the idea of networked nodes

 

You'll also be able to communicate with your node via simple JSON-RPC calls. This means you can interact with the Bitcoin Network at large without relying on any trusted third party. As a simple example of what can be done with this, we'll run an open source block-explorer, providing you with a trustless interface to view activity on the network.

Prerequisites

Because this guide is for developers, it assumes you're comfortable doing developery things, such as running Linux, using bash, and executing Python scripts. We will not utilize the Bitcoin QT GUI to set up our wallet-- in fact none of the steps in this guide will assume a GUI at all.

Where more in-depth usage of some other tech is an out of scope pre-requisite, I'll provide links to useful guides as appropriate for those portions. If you're looking to run a full node as a standard user, check out the guide on bitcoin.org.

There are a few hardware and software requirements, as enumerated below:

  • A modestly spec'd computer with at least 2 GB of RAM (4+ suggested)
  • At least 150 GB of free disk space
  • A reasonably fast network connection without stringent bandwidth requirements
  • An installation of Ubuntu 18.04
    • If you're running this on your dev machine that already has macOS or Windows installed, you can do so in VM
    • Older versions of Ubuntu probably work fine as well, but weren't tested
    • Other variants of Linux should also work fine, but you'll have to install bitcoind and handle dependencies as appropriate for your distro (more below)

I run my node with fast residential broadband on a 6 year old desktop PC. It has an Athlon FX processor, 4 GB of RAM, and slow 250 GB spinning disk. It runs like champ! The Bitcoin Core team's focus on making it possible for anyone to run a node truly has paid off.

Installing bitcoind

The Bitcoin network follows a protocol for communication and consensus. While several clients for this protocol exist, the "canonical" and most widely used version is referred to as Bitcoin Core.

To install Bitcoin Core on Ubuntu, you have a few choices:

  1. Clone the repository from GitHub and build from source
  2. Download and install official pre-compiled binaries
  3. Installing the binaries via apt from the Bitcoin Personal Package Archive (PPA)

Each method has its tradeoffs. In this guide, we'll opt for using apt and the PPA, because it's the simplest and easiest way, and makes upgrading to new versions of the software easy as well. One downside is it requires trusting the software that comes from the PPA which could, for example, be compromised by an attacker. Please evaluate these tradeoffs yourself and choose the method thats best for you.

If you're just using your node for development and tinkering and want to proceed with option #3, simply run the following commands and follow the associated prompts:

sudo apt-add-repository ppa:bitcoin/bitcoin
sudo apt-get update
sudo apt-get install bitcoind

When these commands are complete successfully, run bitcoind -version. You should see some output showing the version of the bitcoind software you're using.

Initial Block Download

You've now got the latest software needed to run a node, but you're missing 10+ years of the network's history. Remember, this is a full node! That means it stores and validates the full history of the blockchain, starting from the very first block mined by Satoshi himself. That history will need to be downloaded.

Unfortunately, this process takes a while and requires about 150 GB of disk space as of today. On my reasonably fast residential internet, I was able to download the full blockchain in about 4 days. Your mileage may vary based on your connection and geography.

Run the following command:

bitcoind -daemon -txindex=1

This runs the bitcoind process in the background with transaction indexing on. You may also want to start this process automatically on every reboot as well. You can do this by running: crontab -e and adding the following line:

@reboot bitcoind -daemon

Note that we didn't add the txindex option to the crontab. Later in this guide we'll put this option in a config file instead, which is a better way to set options for your node.

When the node software starts for the first time, it creates ~/.bitcoin/. Let's navigate there now and poke around:

cd ~/.bitcoin/
ls

You'll notice a number of files and directories have been created. We'll examine just a few of these now. First, run tail -f debug.log. You'll notice a series of log messages that look something like this:

2018-08-10 17:22:15 UpdateTip: new best=00000000000000000011b735b8fde14db39704c446dd2b1ae4b8db198c255e0d
height=536128 version=0x20000000 log2_work=89.424049 tx=333938835
date='2018-08-10 17:22:26' progress=1.000000 cache=71.6MiB(529075txo)

(When the node first begins syncing, these may be flying by pretty fast. Hit Ctrl-C to quit tailing the file and examine the messages).

These messages represent blocks being downloaded from the network. The one shown here represents the latest block mined at the time I'm writing this tutorial, but you'll be well behind this. You'll note each message shows a number of parameters, including the hash of the block (best=X), the block height (height=X), and a timestamp of when this block was mined (date=X).

To monitor the progress of the initial download, you can tail this file and look at the historical date you've caught up to. Note that, since most blocks early in the network's history were mostly empty, early blocks will download fast. As you get further along, download times will increase. My node spent a majority of the time downloading blocks from 2017 onwards.

You can also interact with the blockchain using the bitcoin-cli command line interface. For example, you can run bitcoin-cli stop to safely shutdown your node, or run bitcoin-cli -getinfo to get a summary of your node's status:

{
  "version": 160000,
  "protocolversion": 70015,
  "walletversion": 159900,
  "balance": 0.00000000,
  "blocks": 536128,
  "timeoffset": 0,
  "connections": 33,
  "proxy": "",
  "difficulty": 5949437371609.53,
  "testnet": false,
  "keypoololdest": 1531965982,
  "keypoolsize": 1000,
  "paytxfee": 0.00000000,
  "relayfee": 0.00001000,
  "warnings": ""
}

One more small piece of fun. The block data that your node is downloading is stored in binary files in the ~/.bitcoin/blocks/ directory. You won't normally interact with this data directly, but try running this quick command:

head -c 210 ~/.bitcoin/blocks/blk00000.dat && echo '\n'`.

You'll see something like this:

��;���z{�z�,>gv�a���Q2:���K^J)�_I���+|����M��E
The Times 03/Jan/2009 Chancellor on brink of second bailout for banks��

Do you recognize that sentence? If you do, it might send chills down your spine to see it on your machine! If you don't, go Google it and learn why it should.

As mentioned, initial block download is going to take at least a few days. During this phase, the daemon may use a lot of resources. Mine pegged my processor at a consistent 20-30%, utilized over 2 gigs of RAM, and generally made my admittedly old machine slow. Once you're caught up, running the node software has a negligible effect on performance with even a modest machine.

If you've made it this far, bookmark the page and come back in a few days when your node is in sync!

Server Configuration

Welcome back! It's time to configure your node to act as a JSON-RPC server, so we can run and write code in our favorite language to interact with it.

Navigate back to the bitcoin directory and create a flat-text file called bitcoin.conf.

cd ~/.bitcoin/
touch bitcoin.conf

Henceforth, this file will be read when bitcoind starts and will be used to configure options for your node. There are a number of options you can set, and each of these can also be set at command line as well. To get a sense of the options available to you, run:

bitcoind --help

Open up the bitcoin.conf file in your favorite text editor and add the following lines:

## Configuration for the Bitcoin Core Daemon

# Accept command line and JSON-RPC commands
server=1

This is a good start, but not sufficient to enable RPC interaction with our node. To do so, we also have to create authentication credentials for the user who will log in.

It used to possible to specify a username and password in flat text in the config file, but this method is clearly insecure and has been deprecated. Instead, we have to provide the username along with the hash value of the password-- in conjunction with a salt used during hashing.

To generate this, use the following commands to download the canonical Python script provided by the Bitcoin Core team and run it with your own username and password.

wget https://raw.githubusercontent.com/bitcoin/bitcoin/master/share/rpcauth/rpcauth.py
python3 rpcauth.py myusername mysecretpassword

Be sure to replace the username and password values with something sane! The script should produce a result like this:

String to be appended to bitcoin.conf:
rpcauth=myusername:d2a45f8bd56a2014461e2f70f6783b56$636211cdbf3f943a887df72d9944be13ae61c03ef43c6d06dd5044debd4fcbd7
Your password:
mysecretpassword

As instructed, add this line to your bitcoin.conf file, which should now look something like this:

## Configuration for the Bitcoin Core Daemon

# Accept command line and JSON-RPC commands
server=1

# Index all transactions
txindex=1

# Auth Credentials For JSON-RPC server
rpcauth=myusername:d2a45f8bd56a2014461e2f70f6783b56$636211cdbf3f943a887df72d9944be13ae61c03ef43c6d06dd5044debd4fcbd7

Next, restart your node to make sure the configuration changes take effect:

bitcoin-cli stop
bitcoind -daemon

At this point, you're ready to make JSON-RPC requests to your node, as long as they're coming from the node itself. To test this out, try running the following command (note: you may have to install curl first via apt).

curl --user myusername --data-binary '{"jsonrpc": "1.0", "id":"curltest", "method": "getinfo", "params": [] }' -H 'content-type: text/plain;' http://127.0.0.1:8332

Be sure to replace myusername with username you actually created. You'll be prompted for the associated password, and if all worked properly, you should see something like the following:

{
    "result": {
        "chain": "main",
        "blocks": 536148,
        "headers": 536148,
        "bestblockhash": "00000000000000000028fab7a6b5035e2745fe06ef58d817bb32f35924198f97",
        "difficulty": 5949437371609.53,
        "mediantime": 1533926692,
        "verificationprogress": 0.9999864596463776,
        "initialblockdownload": false,
        "chainwork": "000000000000000000000000000000000000000002af5dee686cd058c3ec064d",
        "size_on_disk": 203955272825,
        "pruned": false,
        "softforks": ["..."],
        "bip9_softforks": { "..." : "" },
        "warnings": ""
    },
    "error": null,
    "id": "curltest"
}

If this works, you're ready to start using your full node for building/running an app that interacts with it. To learn more about the full set of API calls available, and various libraries that exist that wrap the API, checkout this reference on bitcoin.org.

IP Whitelisting

By default, your node is configured to only accept JSON-RPC requests from localhost, i.e. the machine where it runs. This is fine if you'll be developing on the same box the node is on, and you can skip this section.

If you're developing from a different machine, like a Windows or macOS computer, you'll have to use the rpcallowip option to whitelist IP addresses which are allowed to make calls, and also ensure port 8332 is reachable. (Note that a VM, though running on the same hardware, does in fact have a unique IP).

My local development machine runs macOS, so I can use the ifconfig to determine my local ip, then whitelist it by adding something like rpcallowip=192.168.0.101 to my config file. Note that you can include this option more than once to whitelist multiple IP addresses.

The specifics of finding your development machine's IP and making port 8332 available on your node are dependent on your platform and network. These are left as an exercise to the reader. Here's an example of what your final bitcoin.conf might look like.

#### Configuration for the Bitcoin Core Daemon

# Keep my daily upload bandwith around 1 GB

maxuploadtarget=1024  # Note that if your node isn't exposed to the internet on
                      # Port 8333, you're not feeding data to other nodes and this
                      # setting won't matter.
                      # While not required, if you'd like to support the network
                      # by servering data, make sure your router is configured to
                      # forward port 8333 to the machine with your.

# Accept command line and JSON-RPC commands
server=1

# Index all transactions
txindex=1

# Auth Credentials For JSON-RPC
rpcauth=myusername:d2a45f8bd56a2014461e2f70f6783b56$636211cdbf3f943a887df72d9944be13ae61c03ef43c6d06dd5044debd4fcbd7

# Whitelist only my Mac's local IP for JSON-RPC calls
rpcallowip=192.168.0.101

Running A Block Explorer

With your full node up and running, synced to the network, and configured for use as JSON-RPC server, it's time to run your own local blockchain explorer! We'll use the open source btc-rpc-explorer, created by Dan Janosik.

This explorer is great because it doesn't use a database-- it just needs access to your node, which it interacts with via the RPC interface we just configured.

btc-rpc-explorer is written in JavaScript using Node.js, so you'll need to have the latest stable version of node and npm installed. If you don't already have these, checkout the instructions for your development platform of choice.

Once Node and npm are ready to go, clone the repo from GitHub and build it with the following commands (which assume a Unix dev environment):

git clone https://github.com/janoside/btc-rpc-explorer.git
cd btc-rpc-explorer
npm install
npm run

Next, open the app/credentials.js file in your text editor of choice and update the username and password to the values you used during the RPC configuration.

If you're dev box is not the same as your node, you'll also have to update the host parameter to the IP address of your node. This will likely be a local area network IP, unless your node is hosted outside your LAN. (Note also that your router may assign either you development box or your node a new IP in the future. Look into configuring static local IPs based on MAC addresses in your router's config panel to solve this issue).

With the credentials updated, you're ready to run the explorer. Execute the following command, then visit http://127.0.0.1:3002/ in your browser of choice.

npm start

If all has gone well, you should see something like this:

locally sourced, organic block explorers

Congratulations, you now have your very own Bitcoin full node, and a locally sourced, organic block explorer to go with it 🎉. Happy hacking!