From 9972f5a560928b816bad78bb19f8312e7787fa8f Mon Sep 17 00:00:00 2001 From: Martin Boehm Date: Wed, 28 Feb 2018 12:54:09 +0100 Subject: [PATCH] Update README --- README.md | 102 ++++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 87 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index df95f840..4e4e897f 100644 --- a/README.md +++ b/README.md @@ -13,11 +13,11 @@ cd blockbook/docker Setup go environment (Debian 9): ``` -sudo apt-get install git -apt-get install -y pkg-config lxc-dev -wget https://storage.googleapis.com/golang/go1.9.2.linux-amd64.tar.gz -sudo mv go /usr/local -sudo ln -s /usr/local/go/bin/go /usr/bin/go +sudo apt-get update && apt-get install -y \ + build-essential git wget pkg-config lxc-dev libzmq3-dev libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev liblz4-dev +cd /opt +wget https://storage.googleapis.com/golang/go1.9.2.linux-amd64.tar.gz && tar xf go1.9.2.linux-amd64.tar.gz +sudo ln -s /opt/go/bin/go /usr/bin/go go help gopath ``` @@ -25,7 +25,6 @@ Install RocksDB: https://github.com/facebook/rocksdb/blob/master/INSTALL.md and compile the static_lib and tools ``` -sudo apt-get install libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev liblz4-dev git clone https://github.com/facebook/rocksdb.git cd rocksdb make release @@ -81,6 +80,79 @@ To run blockbook with fast synchronization, connection to ZeroMQ and providing h ``` Blockbook logs only to stderr, logging to files is disabled. Verbosity of logs can be tuned by command line parameters *-v* and *-vmodule*, details at https://godoc.org/github.com/golang/glog +## Setup on the blockbook-dev server (including Bitcoin Core) +Get Bitcoin Core +``` +wget https://bitcoin.org/bin/bitcoin-core-0.15.1/bitcoin-0.15.1-x86_64-linux-gnu.tar.gz +tar -xf bitcoin-0.15.1-x86_64-linux-gnu.tar.gz +``` + +### TESTNET ### +Data are to be stored in */data/testnet*, in folders */data/testnet/bitcoin* for Bitcoin Core data, */data/testnet/blockbook* for Blockbook data. + +Create configuration file */data/testnet/bitcoin/bitcoin.conf* with content +``` +testnet=1 +daemon=1 +server=1 +rpcuser=rpc +rpcpassword=rpc +rpcport=18332 +txindex=1 +``` +Create script that starts the bitcoind daemon *run-testnet-bitcoind.sh* +``` +#!/bin/bash + +bitcoin-0.15.1/bin/bitcoind -datadir=/data/testnet/bitcoin -zmqpubhashtx=tcp://127.0.0.1:18334 -zmqpubhashblock=tcp://127.0.0.1:18334 -zmqpubrawblock=tcp://127.0.0.1:18334 -zmqpubrawtx=tcp://127.0.0.1:18334 +``` +Run the *run-testnet-bitcoind.sh* to get initial import of data. + +Create script that runs blockbook *run-testnet-blockbook.sh* +``` +#!/bin/bash + +cd go/src/blockbook +./blockbook -path=/data/testnet/blockbook/db -sync -parse -rpcurl=http://127.0.0.1:18332 -httpserver=:18335 -socketio=:18336 -certfile=server/testcert -zeromq=tcp://127.0.0.1:18334 -explorer=https://testnet-bitcore1.trezor.io $1 +``` +To run blockbook with logging to file (run with nohup or daemonize or using screen) +``` +./run-testnet-blockbook.sh 2>/data/testnet/blockbook/blockbook.log +``` + +### BTC ### +Data are to be stored in */data/btc*, in folders */data/btc/bitcoin* for Bitcoin Core data, */data/btc/blockbook* for Blockbook data. + +Create configuration file */data/btc/bitcoin/bitcoin.conf* with content +``` +daemon=1 +server=1 +rpcuser=rpc +rpcpassword=rpc +rpcport=8332 +txindex=1 +``` +Create script that starts the bitcoind daemon *run-btc-bitcoind.sh* +``` +#!/bin/bash + +bitcoin-0.15.1/bin/bitcoind -datadir=/data/btc/bitcoin -zmqpubhashtx=tcp://127.0.0.1:8334 -zmqpubhashblock=tcp://127.0.0.1:8334 -zmqpubrawblock=tcp://127.0.0.1:8334 -zmqpubrawtx=tcp://127.0.0.1:8334 +``` +Run the *run-btc-bitcoind.sh* to get initial import of data. + +Create script that runs blockbook *run-btc-blockbook.sh* +``` +#!/bin/bash + +cd go/src/blockbook +./blockbook -path=/data/btc/blockbook/db -sync -parse -rpcurl=http://127.0.0.1:8332 -httpserver=:8335 -socketio=:8336 -certfile=server/testcert -zeromq=tcp://127.0.0.1:8334 -explorer=https://bitcore1.trezor.io/ $1 +``` +To run blockbook with logging to file (run with nohup or daemonize or using screen) +``` +./run-btc-blockbook.sh 2>/data/btc/blockbook/blockbook.log +``` + + # Data storage in RocksDB Blockbook stores data the key-value store RocksDB. Data are stored in binary form to save space. @@ -127,17 +199,17 @@ The data are separated to different column families: ## Todo +- cleanup of the socket.io - do not send unnecessary data +- protobuf websocket interface +- parallel sync - rewrite - it is not possible to gracefully stop it now, can leave holes in the block +- disconnect blocks - keep map of transactions in the last 100 blocks +- handle different versions of Bitcoin Core +- limit number of transactions returned by rocksdb.GetTransactions +- xpub index +- tests - ~~mempool - return also input transactions~~ - ~~blockchain - return inputs from mempool~~ - ~~do not return duplicate txids~~ -- limit number of transactions returned by rocksdb.GetTransactions - probably by return value from callback function - ~~legacy socket.io JSON interface~~ -- cleanup of the socket.io - do not send unnecessary data -- protobuf websocket interface -- stream results to REST and websocket interfaces -- parallel sync - rewrite - it is not possible to gracefully stop it now, can leave holes in the block -- ~~parallel sync - let rocksdb to compact itself from time to time, otherwise it consumes too much disk space~~ - ~~disconnect blocks - optimize - full range scan is too slow and takes too much disk space (creates snapshot of the whole outputs), split to multiple iterators~~ -- disconnect blocks - keep map of transactions in the last 100 blocks -- xpub index -- tests +- ~~parallel sync - let rocksdb to compact itself from time to time, otherwise it consumes too much disk space~~