RanchiMall/blockbook
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'master' into v0.0.7

Browse Source
This commit is contained in:
Martin Boehm 2018-09-03 08:33:20 +02:00
commit d50198cbff
18 changed files with 1066 additions and 347 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

175
CONTRIBUTING.md Normal file
View File

@ -0,0 +1,175 @@
# Blockbook Contributor Guide
Blockbook is back-end service for Trezor wallet. Although it is open source, design and development of core packages
is done by Trezor developers in order to keep Blockbook compatible with Trezor.
Bug fixes and support for new coins are welcome. Please take note that non-fixing pull requests that change base
packages or another coin code will not be accepted. If you will have to change some of existing code, please file
an issue and discuss your request with Blockbook maintainers.
## Development environment
Instructions to set up your development environment and build Blockbook are described in separated
[document](/docs/build.md).
## How can I contribute?
### Reporting bugs
A great way to contribute to the project is to send a detailed report when you encounter an issue. We always appreciate
a well-written, thorough bug report, and will thank you for it!
Check that [our issue database](https://github.com/trezor/blockbook/issues) doesn't already include that problem or
suggestion before submitting an issue. If you find a match, you can use the "subscribe" button to get notified on
updates. Do not leave random "+1" or "I have this too" comments, as they only clutter the discussion, and don't help
resolving it. However, if you have ways to reproduce the issue or have additional information that may help resolving
the issue, please leave a comment.
Include information about Blockbook instance which is exposed at internal HTTP port. Ports are listed in
[port registry](/docs/ports.md). For example execute `curl -k https://localhost:9030` for Bitcoin.
Also include the steps required to reproduce the problem if possible and applicable. This information will help us
review and fix your issue faster. When sending lengthy log-files, consider posting them as a gist
(https://gist.github.com).
### Adding coin support
Trezor harware wallet supports over 500 coins, see https://trezor.io/coins/. You are free to add support for any of
them to Blockbook. Actually implemented coins are listed [here](/docs/ports.md).
You should follow few steps bellow to get smooth merge of your PR.
> Although we are happy for support of new coins we have not enough capacity to run them all on our infrastructure.
> Actually we can run Blockbook instances only for coins supported by Trezor wallet. If you want to have Blockbook
> instance for your coin, you will have to deploy your own server.
#### Add coin definition
Coin definitions are stored in JSON files in *configs/coins* directory. They are single source of Blockbook
configuration, Blockbook and back-end package definition and build metadata. Since Blockbook supports only single
coin index per running instance, every coin (including testnet) must have single definition file.
All options of coin definition are described in [config.md](/docs/config.md).
Because most of coins are fork of Bitcoin and they have similar way to install and configure their daemon, we use
templates to generate package definition and configuration files during build process. It is similar to build Blockbook
package too. Templates are filled with data from coin definition. Although build process generate packages
automatically, there is sometimes necessary see intermediate step. You can generate all files by calling
`go run build/templates/generate.go coin` where *coin* is name of definition file without .json extension. Files are
generated to *build/pkg-defs* directory.
Good examples of coin configuration are
[*configs/coins/bitcoin.json*](configs/coins/bitcoin.json) and
[*configs/coins/ethereum.json*](configs/coins/ethereum.json) for Bitcoin-like coins and different coins, respectively.
Usually you have to update only few options that differ from Bitcoin definition. At first there are base information
about coin in section *coin* – name, alias etc. Then update port information in *port* section. We keep port series as
listed in [port registry](/docs/ports.md). Select next port numbers in series. Port numbers must be unique across all
port definitions.
In section *backend* update information how to build and configure back-end service. When back-end package is built,
build process downloads installation archive, verify and extract it. How it is done is described in
[build guide](/docs/build.md#on-back-end-building). Naming conventions and versioning are described
also in [build guide](/docs/build.md#on-naming-conventions-and-versioning). You have to update *package_name*,
*package_revision*, *system_user*, *version*, *binary_url*, *verification_type*, *verification_source*,
*extract_command* and *exclude_files*. Also update information whether service runs mainnet or testnet network in
*mainnet* option.
In section *blockbook* update information how to build and configure Blockbook service. Usually they are only
*package_name*, *system_user* and *explorer_url* options. Naming conventions are are described
[here](/docs/build.md#on-naming-conventions-and-versioning).
Update *package_maintainer* and *package_maintainer_email* options in section *meta*.
Execute script *contrib/scripts/check-ports.go* that will check mandatory ports and uniquity of registered ports.
Execute script *contrib/scripts/generate-port-registry.go* that will update *docs/ports.md*.
Now you can try generate package definitions as described above in order to check outputs.
#### Add coin implementation
Coin implementation is stored in *bchain/coins* directory. Each coin must implement interfaces *BlockChain* and
*BlockChainParser* (both defined in [bchain/types.go][/bchain/types.go]) and has registered factory function by
*init()* function of package *blockbook/bchain/coins* ([bchain/coins/blockchain.go](/bchain/coins/blockchain.go)).
There are several approaches how to implement coin support in Blockbook, please see examples below.
Bitcoin package *blockbook/bchain/coins/btc* is reference implementation for Bitcoin-like coins. Most of functionality
is same so particular coin should embed it and override just different parts.
Bitcoin uses binary WIRE protocol thus decoding is very fast but require complex parser. Parser translate whole
pubkey-script to database ID and therefore it is usually possible store transactions without change.
ZCash package *blockbook/bchain/coins/zec* on the other side uses JSON version of RPCs therefore it doesn't require
specialized parser. Only responsibility that parser has is to translate address to database ID and vice versa.
Ethereum package *blockbook/bchain/coins/eth* must have stand alone implementation because Ethereum uses totally
different concept than Bitcoin.
##### BlockChain interface
Type that implements *bchain.BlockChain* interface ensures communication with block chain network. Because
it calls node RPCs it usually has suffix RPC.
Initialization of object is separated into two stages. At first there is called factory method (details described
in next section) and then *bchain.BlockChain.Initialize()* method. Separated initialization method allows you call
inherited methods during initialization. However it is common practice override fields of embedded structure in factory
method.
During initialization, there is usually loaded chain information, registered message queue callback and created mempool
and parser objects.
BitcoinRPC uses *btc.RPCMarshaller* ([btc/codec.go](/bchain/coins/btc/codec.go)) in order to distinguish API version of
Bitcoin RPC. Current API (*btc.JSONMarshalerV2*) uses JSON object with method arguments. Older API (*btc.JSONMarshalerV1*)
uses JSON array with method arguments and some arguments are defined differently (e.g. bool vs int).
For example see [zec/zcashrpc.go](/bchain/coins/zec/zcashrpc.go).
##### BlockChain factory function
Factory function must be *coins.blockChainFactory* type ([coins/blockchain.go](/bchain/coins/blockchain.go)). It gets
configuration as JSON object and handler function for PUSH notifications. All factory functions have registered by
*init()* function of package *blockbook/bchain/coins* ([coins/blockchain.go](/bchain/coins/blockchain.go)). Coin name
must correspond to *coin.name* in coin definition file (see above).
Configuration passed to factory method is coin specific. For types that embed *btc.BitcoinRPC,* configuration must
contain at least fields referred in *btc.Configuration* ([btc/bitcoinrpc.go](/bchain/coins/btc/bitcoinrpc.go)).
For types that embed base struct it is common practise call factory method of embedded type in order to
create & initialize it. It is much more robust than simple struct composition.
For example see [zec/zcashrpc.go](/bchain/coins/zec/zcashrpc.go).
##### BlockChainParser interface
Type that implements *bchain.BlockChainParser* interface ensures parsing and conversions of block chain data. It is
initialized by *bchain.BlockChain* during initialization.
There are several groups of methods defined in *bchian.BlockChainParser*:
* *GetAddrIDFromVout* and *GetAddrIDFromAddress* – Convert transaction addresses to *[]byte* ID that is used as database ID.
Most of coins use pubkey-script as ID.
* *AddressToOutputScript* and *OutputScriptToAddresses* – Convert address to output script and vice versa. Note that
*btc.BitcoinParser* uses pointer to function *OutputScriptToAddressesFunc* that is called from *OutputScriptToAddress*
method in order to rewrite implementation by types embedding it.
* *PackTxid* and *UnpackTxid* – Packs txid to store in database and vice versa.
* *ParseTx* and *ParseTxFromJson* – Parse transaction from binary data or JSON and return *bchain.Tx*.
* *PackTx* and *UnpackTx* – Pack transaction to binary data to store in database and vice versa.
* *ParseBlock* – Parse block from binary data and return *bchain.Block*.
Base type of parsers is *bchain.BaseParser*. It implements method *ParseTxFromJson* that would be same for all
Bitcoin-like coins. Also implements *PackTx* and *UnpackTx* that pack and unpack transactions using protobuf. Note
that Bitcoin store transactions in hex format.
*bchain.BaseParser* stores pointer to function *bchain.AddressFactoryFunc* that is responsible for making human readable
address representation. See [*bch.BCashParser*](/bchain/coins/bch/bcashparser.go) for example of implementation that uses
different approach for address representation than Bitcoin.
#### Add tests
Add unit tests and integration tests. Tests are described [here](/docs/testing.md).
#### Deploy public server
Deploy Blockbook server on public IP address. Blockbook maintainers will check implementation before merging.

160
README.md
View File

@ -1,169 +1,50 @@
# blockbook # Blockbook
## **blockbook is currently in the state of heavy development, do not expect this documentation to be up to date** > **Blockbook is currently in the state of heavy development, do not expect this documentation to be up to date**
## Build and install using docker ## Build and installation instructions
Run in the project root Officially supported platform is **Debian Linux** and **AMD64** architecture. Memory requirements for initial synchronization
of **Bitcoin mainnet** are around 32 GB RAM and 100 GB of disk size. Fully synchronized instance takes around 10 GB RAM.
Other coins should have lower requirements depending on size of their block chain. Note that fast SSD disks are highly
recommended.
``` User installation guide is [here](https://wiki.trezor.io/User_manual:Running_a_local_instance_of_Trezor_Wallet_backend_(Blockbook)).
make all
```
to create blockbook debian packages.
## Install manually
Setup go environment (Debian 9):
```
sudo apt-get update && apt-get install -y \
build-essential git wget pkg-config lxc-dev libzmq3-dev libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev liblz4-dev
cd /opt
wget https://storage.googleapis.com/golang/go1.9.2.linux-amd64.tar.gz && tar xf go1.9.2.linux-amd64.tar.gz
sudo ln -s /opt/go/bin/go /usr/bin/go
go help gopath
```
Install RocksDB: https://github.com/facebook/rocksdb/blob/master/INSTALL.md
and compile the static_lib and tools
```
git clone https://github.com/facebook/rocksdb.git
cd rocksdb
make release
```
Setup variables for gorocksdb: https://github.com/tecbot/gorocksdb
```
export CGO_CFLAGS="-I/path/to/rocksdb/include"
export CGO_LDFLAGS="-L/path/to/rocksdb -lrocksdb -lstdc++ -lm -lz -lbz2 -lsnappy -llz4"
```
Install ZeroMQ: https://github.com/zeromq/libzmq
Install go-dep tool:
```
RUN go get github.com/golang/dep/cmd/dep
```
Get blockbook sources, install dependencies, build:
```
cd $GOPATH/src
git clone https://github.com/trezor/blockbook.git
cd blockbook
dep ensure
go build
```
## Usage
```
./blockbook --help
```
## Example command
To run blockbook with fast synchronization, connection to ZeroMQ and providing https and socket.io interface, with database in local directory *data* and connected to local bitcoind with configuration specified by parameter *-blockchaincfg*:
```
./blockbook -sync -blockchaincfg=configs/bitcoin_testnet.json -internal=127.0.0.1:8333 -public=127.0.0.1:8334 -certfile=server/testcert -logtostderr
```
Blockbook logs to stderr *-logtostderr* or to directory specified by parameter *-log_dir* . Verbosity of logs can be tuned by command line parameters *-v* and *-vmodule*, details at https://godoc.org/github.com/golang/glog
Developer build guide is [here](/docs/build.md).
# Implemented coins # Implemented coins
- [Bitcoin](bchain/coins/btc/btc.md) The most significant coins implemented by Blockbook are:
- [Bitcoin Testnet](bchain/coins/btc/btctestnet.md)
- Bitcoin
- Bitcoin Testnet
- Bcash - Bcash
- Bcash Testnet - Bcash Testnet
- Bgold - Bgold
- [ZCash](bchain/coins/zec/zec.md) - ZCash
- ZCash Testnet - ZCash Testnet
- Dash - Dash
- Dash Testnet - Dash Testnet
- Litecoin - Litecoin
- Litecoin Testnet - Litecoin Testnet
- [Ethereum](bchain/coins/eth/eth.md) - Ethereum
- [Ethereum Testnet Ropsten](bchain/coins/eth/ethropsten.md) - Ethereum Testnet Ropsten
They are also supported by Trezor wallet. List of all coins is [here](/docs/ports.md).
# Data storage in RocksDB # Data storage in RocksDB
Blockbook stores data the key-value store RocksDB. Data are stored in binary form to save space. Blockbook stores data the key-value store RocksDB. Database format is described [here](/docs/rocksdb.md).
The data are separated to different column families:
- **default**
at the moment not used, will store statistical data etc.
- **height** - maps *block height* to *block hash*
*Block heigh* stored as array of 4 bytes (big endian uint32)
*Block hash* stored as array of 32 bytes
Example - the first four blocks (all data hex encoded)
```
0x00000000 : 0x000000000933ea01ad0ee984209779baaec3ced90fa3f408719526f8d77f4943
0x00000001 : 0x00000000b873e79784647a6c82962c70d228557d24a747ea4d1b8bbe878e1206
0x00000002 : 0x000000006c02c8ea6e4ff69651f7fcde348fb9d557a06e6957b65552002a7820
0x00000003 : 0x000000008b896e272758da5297bcd98fdc6d97c9b765ecec401e286dc1fdbe10
```
- **outputs** - maps *output script+block height* to *array of outpoints*
*Output script (ScriptPubKey)+block height* stored as variable length array of bytes for output script + 4 bytes (big endian uint32) block height
*array of outpoints* stored as array of 32 bytes for transaction id + variable length outpoint index for each outpoint
Example - (all data hex encoded)
```
0x001400efeb484a24a1c1240eafacef8566e734da429c000e2df6 : 0x1697966cbd76c75eb9fc736dfa3ba0bc045999bab1e8b10082bc0ba546b0178302
0xa9143e3d6abe282d92a28cb791697ba001d733cefdc7870012c4b1 : 0x7246e79f97b5f82e7f51e291d533964028ec90be0634af8a8ef7d5a903c7f6d301
```
- **inputs** - maps *transaction outpoint* to *input transaction* that spends it
*Transaction outpoint* stored as array of 32 bytes for transaction id + variable length outpoint index
*Input transaction* stored as array of 32 bytes for transaction id + variable length input index
Example - (all data hex encoded)
```
0x7246e79f97b5f82e7f51e291d533964028ec90be0634af8a8ef7d5a903c7f6d300 : 0x0a7aa90ea0269c79f844c516805e4cac594adb8830e56fca894b66aab19136a428
0x7246e79f97b5f82e7f51e291d533964028ec90be0634af8a8ef7d5a903c7f6d301 : 0x4303a9fcfe6026b4d33ba488df6443c9a99bca7b7fcb7c6f6cd65cea24a749b700
```
## Registry of ports ## Registry of ports
| coin | blockbook internal port | blockbook public port | backend rpc port | zmq port | Reserved ports are described [here](/docs/ports.md)
|--------------------------|-------------------------|-----------------------|--------------------|----------|
| Bitcoin | 9030 | 9130 | 8030 | 38330 |
| Bcash | 9031 | 9131 | 8031 | 38331 |
| Zcash | 9032 | 9132 | 8032 | 38332 |
| Dash | 9033 | 9133 | 8033 | 38333 |
| Litecoin | 9034 | 9134 | 8034 | 38334 |
| Bgold | 9035 | 9135 | 8035 | 38335 |
| Ethereum | 9036 | 9136 | 8036 ws, 8136 http | 38336* |
| Ethereum Classic | 9037 | 9137 | 8037 | 38337* |
| Dogecoin | 9038 | 9138 | 8038 | 38338 |
| Namecoin | 9039 | 9139 | 8039 | 38339 |
| Vertcoin | 9040 | 9140 | 8040 | 38340 |
| Monacoin | 9041 | 9141 | 8041 | 38341 |
| Bitcoin Testnet | 19030 | 1913 | 18030 | 48330 |
| Bcash Testnet | 19031 | 1913 | 18031 | 48331 |
| Zcash Testnet | 19032 | 1913 | 18032 | 48332 |
| Dash Testnet | 19033 | 1913 | 18033 | 48333 |
| Litecoin Testnet | 19034 | 1913 | 18034 | 48334 |
| Ethereum Testnet Ropsten | 19036 | 19136 | 18036 | 48336* |
| Vertcoin Testnet | 19040 | 19140 | 18040 | 48340 |
| Monacoin Testnet | 19041 | 19141 | 18041 | 48341 |
\* geth listens on this port, however not as zmq service
## Todo ## Todo
- add db data version (column data version) checking to db to avoid data corruption - add db data version (column data version) checking to db to avoid data corruption
- improve txcache (time of storage, number/size of cached txs, purge cache) - improve txcache (time of storage, number/size of cached txs, purge cache)
- update documentation
- create/integrate blockchain explorer - create/integrate blockchain explorer
- support all coins from https://github.com/trezor/trezor-common/tree/master/defs/coins - support all coins from https://github.com/trezor/trezor-common/tree/master/defs/coins
- full ethereum support (tokens, balance) - full ethereum support (tokens, balance)
@ -172,6 +53,7 @@ The data are separated to different column families:
- tests - tests
- fix program dependencies to concrete versions - fix program dependencies to concrete versions
- protect socket.io interface against illicit usage - protect socket.io interface against illicit usage
- ~~update documentation~~
- ~~collect blockbook db stats (number of items in indexes, etc)~~ - ~~collect blockbook db stats (number of items in indexes, etc)~~
- ~~optimize mempool (use non verbose get transaction, possibly parallelize)~~ - ~~optimize mempool (use non verbose get transaction, possibly parallelize)~~
- ~~update used paths and users according to specification by system admin~~ - ~~update used paths and users according to specification by system admin~~

50
bchain/coins/btc/btc.md
View File

@ -1,50 +0,0 @@
## BTC Setup
Get Bitcoin Core
```
wget https://bitcoin.org/bin/bitcoin-core-0.16.0/bitcoin-0.15.1-x86_64-linux-gnu.tar.gz
tar -xf bitcoin-0.16.0-x86_64-linux-gnu.tar.gz
```
Data are stored in */data/btc*, in folders */data/btc/bitcoin* for Bitcoin Core data, */data/btc/blockbook* for Blockbook data.
Create configuration file */data/btc/bitcoin/bitcoin.conf* with content
```
daemon=1
server=1
rpcuser=rpc
rpcpassword=rpc
rpcport=8030
txindex=1
```
Create script that starts the bitcoind daemon *run-btc-bitcoind.sh* with increased rpcworkqueue and configured zeromq
```
#!/bin/bash
bitcoin-0.15.1/bin/bitcoind -datadir=/data/btc/bitcoin -rpcworkqueue=32 -zmqpubhashtx=tcp://127.0.0.1:38330 -zmqpubhashblock=tcp://127.0.0.1:38330 -zmqpubrawblock=tcp://127.0.0.1:38330 -zmqpubrawtx=tcp://127.0.0.1:38330
```
Run the *run-btc-bitcoind.sh* to get initial import of data.
Create blockchain configuration file */data/testnet/blockbook/btc.json*
```
{
"rpcURL": "http://127.0.0.1:8030",
"rpcUser": "rpc",
"rpcPass": "rpc",
"rpcTimeout": 25,
"parse": true,
"zeroMQBinding": "tcp://127.0.0.1:38330"
}
```
Create script that runs blockbook *run-btc-blockbook.sh*
```
#!/bin/bash
cd go/src/blockbook
./blockbook -coin=btc -blockchaincfg=/data/btc/blockbook/btc.json -datadir=/data/btc/blockbook/db -sync -internal=:9030 -public=:9130 -certfile=server/testcert -explorer=https://bitcore1.trezor.io/ $1
```
To run blockbook with logging to file (run with nohup or daemonize or using screen)
```
./run-btc-blockbook.sh 2>/data/btc/blockbook/blockbook.log
```

49
bchain/coins/btc/btctestnet.md
View File

@ -1,49 +0,0 @@
## BTC Testnet Setup
Get Bitcoin Core
```
wget https://bitcoin.org/bin/bitcoin-core-0.16.0/bitcoin-0.15.1-x86_64-linux-gnu.tar.gz
tar -xf bitcoin-0.16.0-x86_64-linux-gnu.tar.gz
```
Data are stored in */data/testnet*, in folders */data/testnet/bitcoin* for Bitcoin Core data, */data/testnet/blockbook* for Blockbook data.
Create configuration file */data/testnet/bitcoin/bitcoin.conf* with content
```
testnet=1
daemon=1
server=1
rpcuser=rpc
rpcpassword=rpc
rpcport=18030
txindex=1
```
Create script that starts the bitcoind daemon *run-testnet-bitcoind.sh* with increased rpcworkqueue and configured zeromq
```
#!/bin/bash
bitcoin-0.15.1/bin/bitcoind -datadir=/data/testnet/bitcoin -rpcworkqueue=32 -zmqpubhashtx=tcp://127.0.0.1:48330 -zmqpubhashblock=tcp://127.0.0.1:48330 -zmqpubrawblock=tcp://127.0.0.1:48330 -zmqpubrawtx=tcp://127.0.0.1:48330
```
Run the *run-testnet-bitcoind.sh* to get initial import of data.
Create blockchain configuration file */data/testnet/blockbook/btc-testnet.json*
```
{
"rpcURL": "http://127.0.0.1:18030",
"rpcUser": "rpc",
"rpcPass": "rpc",
"rpcTimeout": 25,
"parse": true,
"zeroMQBinding": "tcp://127.0.0.1:48330"
}
```
Create script that runs blockbook *run-testnet-blockbook.sh*
```
#!/bin/bash
cd go/src/blockbook
./blockbook -coin=btc-testnet -blockchaincfg=/data/testnet/blockbook/btc-testnet.json -datadir=/data/testnet/blockbook/db -sync -internal=:19030 -public=:19130 -certfile=server/testcert -explorer=https://testnet-bitcore1.trezor.io $1
```
To run blockbook with logging to file (run with nohup or daemonize or using screen)
```
./run-testnet-blockbook.sh 2>/data/testnet/blockbook/blockbook.log
```

25
bchain/coins/eth/eth.md
View File

@ -1,25 +0,0 @@
## Ethereum Testnet Setup
Get Ethereum
```
git clone https://github.com/ethereum/go-ethereum
cd go-ethereum/
make geth
```
Data are stored in */data/eth*, in folders */data/eth/eth* for Ethereum data, */data/eth/blockbook* for Blockbook data.
Run geth with rpc and websocket interfaces, bound to all ip addresses - insecure! (run with nohup or daemonize or using screen)
```
go-ethereum/build/bin/geth --syncmode "full" --cache 1024 --datadir /data/eth/eth --port "35555" --rpc --rpcport 8545 -rpcaddr 0.0.0.0 --rpccorsdomain "*" --ws --wsaddr 0.0.0.0 --wsport 8546 --wsorigins "*" 2>/data/eth/eth/eth.log
```
Create script that runs blockbook *run-eth-blockbook.sh*
```
#!/bin/bash
cd go/src/blockbook
./blockbook -coin=eth -blockchaincfg=/data/eth/blockbook/eth.json -datadir=/data/eth/blockbook/db -sync -internal=:8555 -public=:8556 -certfile=server/testcert $1
```
To run blockbook with logging to file (run with nohup or daemonize or using screen)
```
./run-eth-blockbook.sh 2>/data/eth/blockbook/blockbook.log
```

25
bchain/coins/eth/ethropsten.md
View File

@ -1,25 +0,0 @@
## Ethereum Testnet Setup
Get Ethereum
```
git clone https://github.com/ethereum/go-ethereum
cd go-ethereum/
make geth
```
Data are stored in */data/eth-testnet*, in folders */data/eth-testnet/eth* for Ethereum data, */data/eth-testnet/eth/blockbook* for Blockbook data.
Run geth with rpc and websocket interfaces, bound to all ip addresses - insecure! (run with nohup or daemonize or using screen)
```
go-ethereum/build/bin/geth --testnet --datadir /data/eth-testnet/eth --rpc --rpcport 18545 -rpcaddr 0.0.0.0 --rpccorsdomain "*" --ws --wsaddr 0.0.0.0 --wsport 18546 --wsorigins "*" 2>/data/eth-testnet/eth/eth.log
```
Create script that runs blockbook *run-eth-testnet-blockbook.sh*
```
#!/bin/bash
cd go/src/blockbook
./blockbook -coin=eth-testnet -blockchaincfg=/data/eth-testnet/blockbook/eth-testnet.json -datadir=/data/eth-testnet/blockbook/db -sync -internal=:18555 -public=:18556 -certfile=server/testcert $1
```
To run blockbook with logging to file (run with nohup or daemonize or using screen)
```
./run-eth-testnet-blockbook.sh 2>/data/eth-testnet/blockbook/blockbook.log
```

57
bchain/coins/zec/zec.md
View File

@ -1,57 +0,0 @@
## Zcash Setup
Get Zcash client
```
wget https://z.cash/downloads/zcash-1.0.15-linux64.tar.gz
tar xzf zcash-1.0.15-linux64.tar.gz
```
Run command to download the parameters used to create and verify shielded transactions:
```
zcash-1.0.15/bin/zcash-fetch-params
```
Data are stored in */data/zec* , in folders */data/zec/zcash* for Zcash client data, */data/zec/blockbook* for Blockbook data.
Create configuration file */data/zec/zcash/zcash.conf* with content
```
daemon=1
server=1
rpcuser=rpc
rpcpassword=rpc
rpcport=8032
txindex=1
mainnet=1
addnode=mainnet.z.cash
```
Create script *run-zec-zcashd.sh* that starts the zcashd daemon with increased rpcworkqueue and configured zeromq
```
#!/bin/bash
zcash-1.0.15/bin/zcashd -datadir=/data/zec/zcash -rpcworkqueue=32 -zmqpubhashblock=tcp://127.0.0.1:38332 -zmqpubrawblock=tcp://127.0.0.1:38332 -zmqpubhashtx=tcp://127.0.0.1:38332 -zmqpubrawtx=tcp://127.0.0.1:38332
```
Run the *run-zec-zcashd.sh* to get initial import of data.
Create blockchain configuration file */data/zec/blockbook/zec.json*
```
{
"rpcURL": "http://127.0.0.1:8032",
"rpcUser": "rpc",
"rpcPass": "rpc",
"rpcTimeout": 25,
"parse": true,
"zeroMQBinding": "tcp://127.0.0.1:38332"
}
```
Create *run-zec-blockbook.sh* script that starts blockbook
```
#!/bin/bash
./blockbook -coin=zec -blockchaincfg=/data/zec/blockbook/zec.json -datadir=/data/zec/blockbook/db -sync -internal=:9032 -public=:9132 -certfile=server/testcert -explorer=https://zec-bitcore1.trezor.io $1
```
To run blockbook with logging to file (run with nohup or daemonize using screen)
```
./run-zec-blockbook.sh 2> /data/zec/blockbook/blockbook.log
```

4
build/docker/bin/Makefile
View File

@ -23,10 +23,10 @@ tools:
chown $(PACKAGER) /out/{ldb,sst_dump} chown $(PACKAGER) /out/{ldb,sst_dump}
test: prepare-sources test: prepare-sources
cd $(GOPATH)/src/blockbook && go test -tags unittest ./... $(ARGS) cd $(GOPATH)/src/blockbook && go test -tags unittest `go list ./... | grep -v '^blockbook/contrib'` $(ARGS)
test-all: prepare-sources test-all: prepare-sources
cd $(GOPATH)/src/blockbook && go test -tags 'unittest integration' ./... $(ARGS) cd $(GOPATH)/src/blockbook && go test -tags 'unittest integration' `go list ./... | grep -v '^blockbook/contrib'` $(ARGS)
prepare-sources: prepare-sources:
@ [ -n "`ls /src 2> /dev/null`" ] || (echo "/src doesn't exist or is empty" 1>&2 && exit 1) @ [ -n "`ls /src 2> /dev/null`" ] || (echo "/src doesn't exist or is empty" 1>&2 && exit 1)

2
configs/coins/ethereum.json
View File

@ -8,6 +8,8 @@
"ports": { "ports": {
"backend_rpc": 8036, "backend_rpc": 8036,
"backend_message_queue": 0, "backend_message_queue": 0,
"backend_p2p": 38336,
"backend_http": 8136,
"blockbook_internal": 9036, "blockbook_internal": 9036,
"blockbook_public": 9136 "blockbook_public": 9136
}, },

1
configs/coins/ethereum_testnet_ropsten.json
View File

@ -8,6 +8,7 @@
"ports": { "ports": {
"backend_rpc": 18036, "backend_rpc": 18036,
"backend_message_queue": 0, "backend_message_queue": 0,
"backend_p2p": 48336,
"blockbook_internal": 19036, "blockbook_internal": 19036,
"blockbook_public": 19136 "blockbook_public": 19136
}, },

83
contrib/scripts/check-ports.go Executable file
View File

@ -0,0 +1,83 @@
//usr/bin/go run $0 $@ ; exit
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"os"
"path/filepath"
)
const configDir = "configs/coins"
type Config struct {
Coin struct {
Name string `json:"name"`
}
Ports map[string]uint16 `json:"ports"`
}
func main() {
ports := make(map[uint16][]string)
status := 0
files, err := ioutil.ReadDir(configDir)
if err != nil {
panic(err)
}
for _, fi := range files {
if fi.IsDir() || fi.Name()[0] == '.' {
continue
}
path := filepath.Join(configDir, fi.Name())
f, err := os.Open(path)
if err != nil {
panic(fmt.Errorf("%s: %s", path, err))
}
defer f.Close()
v := Config{}
d := json.NewDecoder(f)
err = d.Decode(&v)
if err != nil {
panic(fmt.Errorf("%s: json: %s", path, err))
}
if _, ok := v.Ports["blockbook_internal"]; !ok {
fmt.Printf("%s: missing blockbook_internal port\n", v.Coin.Name)
status = 1
}
if _, ok := v.Ports["blockbook_public"]; !ok {
fmt.Printf("%s: missing blockbook_public port\n", v.Coin.Name)
status = 1
}
if _, ok := v.Ports["backend_rpc"]; !ok {
fmt.Printf("%s: missing backend_rpc port\n", v.Coin.Name)
status = 1
}
for _, port := range v.Ports {
if port > 0 {
ports[port] = append(ports[port], v.Coin.Name)
}
}
}
for port, coins := range ports {
if len(coins) > 1 {
fmt.Printf("port %d: registered by %q\n", port, coins)
status = 1
}
}
if status == 0 {
fmt.Println("OK")
} else {
fmt.Println("Got some errors")
}
os.Exit(status)
}

263
contrib/scripts/generate-port-registry.go Executable file
View File

@ -0,0 +1,263 @@
//usr/bin/go run $0 $@ ; exit
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"io/ioutil"
"math"
"os"
"path/filepath"
"sort"
"strings"
)
const (
inputDir = "configs/coins"
outputFile = "docs/ports.md"
)
type PortInfo struct {
CoinName string
BlockbookInternalPort uint16
BlockbookPublicPort uint16
BackendRPCPort uint16
BackendServicePorts map[string]uint16
}
type PortInfoSlice []*PortInfo
type Config struct {
Coin struct {
Name string `json:"name"`
}
Ports map[string]uint16 `json:"ports"`
}
func main() {
output := "stdout"
if len(os.Args) > 1 {
if len(os.Args) == 2 && os.Args[1] == "-w" {
output = outputFile
} else {
fmt.Fprintf(os.Stderr, "Usage: %s [-w]\n", filepath.Base(os.Args[0]))
fmt.Fprintf(os.Stderr, " -w write output to %s instead of stdout\n", outputFile)
os.Exit(1)
}
}
slice, err := loadPortInfo(inputDir)
if err != nil {
panic(err)
}
sortPortInfo(slice)
err = writeMarkdown(output, slice)
if err != nil {
panic(err)
}
}
func loadPortInfo(dir string) (PortInfoSlice, error) {
files, err := ioutil.ReadDir(dir)
if err != nil {
return nil, err
}
items := make(PortInfoSlice, 0, len(files))
for _, fi := range files {
if fi.IsDir() || fi.Name()[0] == '.' {
continue
}
path := filepath.Join(dir, fi.Name())
f, err := os.Open(path)
if err != nil {
return nil, fmt.Errorf("%s: %s", path, err)
}
defer f.Close()
v := Config{}
d := json.NewDecoder(f)
err = d.Decode(&v)
if err != nil {
return nil, fmt.Errorf("%s: json: %s", path, err)
}
item := &PortInfo{CoinName: v.Coin.Name, BackendServicePorts: map[string]uint16{}}
for k, v := range v.Ports {
if v == 0 {
continue
}
switch k {
case "blockbook_internal":
item.BlockbookInternalPort = v
case "blockbook_public":
item.BlockbookPublicPort = v
case "backend_rpc":
item.BackendRPCPort = v
default:
if len(k) > 8 && k[:8] == "backend_" {
item.BackendServicePorts[k[8:]] = v
}
}
}
items = append(items, item)
}
return items, nil
}
func sortPortInfo(slice PortInfoSlice) {
// normalizes values in order to sort zero values at the bottom of the slice
normalize := func(a, b uint16) (uint16, uint16) {
if a == 0 {
a = math.MaxUint16
}
if b == 0 {
b = math.MaxUint16
}
return a, b
}
// sort values by BlockbookPublicPort, then by BackendRPCPort and finally by
// CoinName; zero values are sorted at the bottom of the slice
sort.Slice(slice, func(i, j int) bool {
a, b := normalize(slice[i].BlockbookPublicPort, slice[j].BlockbookPublicPort)
if a < b {
return true
}
if a > b {
return false
}
a, b = normalize(slice[i].BackendRPCPort, slice[j].BackendRPCPort)
if a < b {
return true
}
if a > b {
return false
}
return strings.Compare(slice[i].CoinName, slice[j].CoinName) == -1
})
}
func writeMarkdown(output string, slice PortInfoSlice) error {
var (
buf bytes.Buffer
err error
)
fmt.Fprintf(&buf, "# Registry of ports\n\n")
header := []string{"coin", "blockbook internal port", "blockbook public port", "backend rpc port", "backend service ports (zmq)"}
writeTable(&buf, header, slice)
fmt.Fprintf(&buf, "\n> NOTE: This document is generated from coin definitions in `configs/coins`.\n")
out := os.Stdout
if output != "stdout" {
out, err = os.OpenFile(output, os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
return err
}
defer out.Close()
}
n, err := out.Write(buf.Bytes())
if err != nil {
return err
}
if n < len(buf.Bytes()) {
return io.ErrShortWrite
}
return nil
}
func writeTable(w io.Writer, header []string, slice PortInfoSlice) {
rows := make([][]string, len(slice))
for i, item := range slice {
row := make([]string, len(header))
row[0] = item.CoinName
if item.BlockbookInternalPort > 0 {
row[1] = fmt.Sprintf("%d", item.BlockbookInternalPort)
}
if item.BlockbookPublicPort > 0 {
row[2] = fmt.Sprintf("%d", item.BlockbookPublicPort)
}
if item.BackendRPCPort > 0 {
row[3] = fmt.Sprintf("%d", item.BackendRPCPort)
}
svcPorts := make([]string, 0, len(item.BackendServicePorts))
for k, v := range item.BackendServicePorts {
var s string
if k == "message_queue" {
s = fmt.Sprintf("%d", v)
} else {
s = fmt.Sprintf("%d %s", v, k)
}
svcPorts = append(svcPorts, s)
}
row[4] = strings.Join(svcPorts, ", ")
rows[i] = row
}
padding := make([]int, len(header))
for column := range header {
padding[column] = len(header[column])
for _, row := range rows {
padding[column] = maxInt(padding[column], len(row[column]))
}
}
content := make([][]string, 0, len(rows)+2)
content = append(content, paddedRow(header, padding))
content = append(content, delim("-", padding))
for _, row := range rows {
content = append(content, paddedRow(row, padding))
}
for _, row := range content {
fmt.Fprintf(w, "|%s|\n", strings.Join(row, "|"))
}
}
func maxInt(a, b int) int {
if a > b {
return a
}
return b
}
func paddedRow(row []string, padding []int) []string {
out := make([]string, len(row))
for i := 0; i < len(row); i++ {
format := fmt.Sprintf(" %%-%ds ", padding[i])
out[i] = fmt.Sprintf(format, row[i])
}
return out
}
func delim(str string, padding []int) []string {
out := make([]string, len(padding))
for i := 0; i < len(padding); i++ {
out[i] = strings.Repeat(str, padding[i]+2)
}
return out
}

8
docs/README.md Normal file
View File

@ -0,0 +1,8 @@
# Documentation index
* [Contributing](/CONTRIBUTING.md) – Blockbook contributor guide
* [Build](/docs/build.md) – Blockbook build guide
* [Config](/docs/config.md) – Description of Blockbook and back-end configuration and package definitions
* [Ports](/docs/ports.md) – Automatically generated registry of ports
* [RocksDB](/docs/rocksdb.md) – Description of RocksDB structures used by Blockbook
* [Testing](/docs/testing.md) – Description of tests used during Blockbook development

265
docs/build.md Normal file
View File

@ -0,0 +1,265 @@
# Blockbook Build Guide
## Setting up your development environment
Supported environment to develop Blockbook is Linux. Although it is possible build and run Blockbook on macOS
or Windows our build process is not prepared for it. But you can still build Blockbook [manually](#manual-build).
The only dependency required to build Blockbook is Docker. You can see how to install Docker [here](https://docs.docker.com/install/linux/docker-ce/debian/).
Manual build require additional dependencies that are described in appropriate section.
## Build in Docker environment
All build operations run in Docker container in order to keep build environment isolated. Makefile in root of repository
define few targets used for building, testing and packaging of Blockbook. With Docker image definitions and Debian
package templates in *build/docker* and *build/templates* respectively, they are only inputs that make build process.
Docker build images are created at first execution of Makefile and that information is persisted. (Actually there are
created two files in repository – .bin-image and .deb-image – that are used as tags.) Sometimes it is necessary to
rebuild Docker images, it is possible by executing `make build-images`.
### Building binary
Just run `make` and that is it. Output binary is stored in *build* directory. Note that although Blockbook is Go application
it is dynamically linked with RocksDB dependencies and ZeroMQ. Therefore operating system where Blockbook will be
executed still need that dependencies installed. See [Manual build](#manual-build) instructions below or install
Blockbook via Debian packages.
### Building debug binary
Standard binary contains no debug symbols. Execute `make build-debug` to get binary for debugging.
### Testing
How to execute tests is described in separate document [here](/docs/testing.md).
### Building Debian packages
Blockbook and particular coin back-end are usually deployed together. They are defined in same place as well.
So typical way to build Debian packages is build Blockbook and back-end deb packages by single command. But it is not
mandatory, of course.
> Early releases of Blockbook weren't so friendly for extending. One had to define back-end package, Blockbook package,
> back-end configuration and Blockbook configuration as well. There were many options that were duplicated across
> configuration files and therefore error prone.
>
> Actually all configuration options and also build options for both Blockbook and back-end are defined in single JSON
> file and all stuff required during build is generated dynamically.
Makefile targets follow simple pattern, there are few prefixes that define what to build.
* *deb-blockbook-&lt;coin&gt;* – Build Blockbook package for given coin.
* *deb-backend-&lt;coin&gt;* – Build back-end package for given coin.
* *deb-&lt;coin&gt;* – Build both Blockbook and back-end packages for given coin.
* *all-&lt;coin&gt;* – Similar to deb-&lt;coin&gt; but clean repository and rebuild Docker image before package build. It is useful
for production deployment.
* *all* – Build both Blockbook and back-end packages for all coins.
Which coins are possible to build is defined in *configs/coins*. Particular coin has to have JSON config file there.
For example we want to build some packages for Bitcoin and Bitcoin Testnet.
```bash
# make all-bitcoin deb-backend-bitcoin_testnet
...
# ls build/*.deb
build/backend-bitcoin_0.16.1-satoshilabs-1_amd64.deb build/backend-bitcoin-testnet_0.16.1-satoshilabs-1_amd64.deb build/blockbook-bitcoin_0.0.6_amd64.deb
```
We have built two back-end packages – for Bitcoin and Testnet – and Blockbook package for Bitcoin. Before build have
been performed there was cleaned build directory and rebuilt Docker image.
### Extra variables
There are few variables that can be passed to make in order to modify build process.
In general, build of Blockbook binary require some dependencies. They are downloaded automatically during build process
but if you need to build binary repeatedly it consumes a lot of time. Here comes variable *UPDATE_VENDOR* that if is
unset says that build process uses *vendor* (i.e. dependencies) from your local repository. For example:
`make deb-bitcoin UPDATE_VENDOR=0`. But before the command is executed there must be *vendor* directory populated,
you can do it by calling `dep ensure --vendor-only`. See [Manual build](#manual-build) instructions below.
All build targets allow pass additional parameters to underlying command inside container. It is possible via ARGS
variable. For example if you want run only subset of unit-tests, you will perform it by calling:
`make test ARGS='-run TestBitcoinRPC' UPDATE_VENDOR=0`
Common behaviour of Docker image build is that build steps are cached and next time they are executed much faster.
Although this is a good idea, when something went wrong you will need to override this behaviour somehow. Execute this
command: `make build-images NO_CACHE=true`.
### On naming conventions and versioning
All configuration keys described below are in coin definition file in *configs/coins*.
**install and data directories**
Both Blockbook and back-end have separated install and data directories. They use common preffix and are defined in
*configs/environ.json* and all templates use them.
* back-end install directory is */opt/coins/nodes/&lt;coin&gt;*.
* back-end data directory is */opt/coins/data/&lt;coin&gt;/backend*.
* Blockbook install directory is */opt/coins/blockbook/&lt;coin&gt;*.
* Blockbook data directory is */opt/coins/data/&lt;coin&gt;/blockbook*.
*coin* used above is defined in *coin.alias* in coin definition file.
**package names**
Package names are defined in *backend.package_name* and *blockbook.package_name* in coin definition file. We use
simple pattern *&lt;prefix&gt;-&lt;coin&gt;* to name packages where *prefix* is either *blockbook* or *backend* and
*coin* is made similarly to *coin.alias*. We use convention that coin name uses lowercase characters and dash '-' as
a word delimiter. Testnet versions of coins must have *-testnet* suffix. That differs from *coin.alias* because
underscore has a special meaning in Debian packaging. For example there are packages *backend-bitcoin* and
*blockbook-bitcoin-testnet*.
**user names**
User names are defined in *backend.system_user* and *blockbook.system_user* in coin definition file. We follow common
Linux conventions, user names use lowercase characters and dash '-' as a word delimiter.
Back-end user name use coin name only, including testnet services. For example there is *bitcoin* user for both
*backend-bitcoin* and *backend-bitcoin-testnet* packages.
Blockbook user name has *blockbook-* prefix and coin name (made same as back-end version). For example there is
*blockbook-bitcoin* user for both *blockbook-bitcoin* and *blockbook-bitcoin-testnet* packages.
**back-end versioning**
Since we have to distinguish version of coin distribution and version of our configuration we follow standard Debian
package versioning rules (for details see
[Debian policy](https://www.debian.org/doc/debian-policy/ch-controlfields.html#version)). There is upstream version
and revision both defined in coin definition file in *backend.version* and *backend.package_revision*, respectively.
**blockbook versioning**
Blockbook versioning is much simpler. There is only one version defined in *configs/environ.json*.
### On back-end building
Because we don't keep back-end archives inside out repository we download them during build process. Build steps
are these: download, verify and extract archive, prepare distribution and make package.
All configuration keys described below are in coin definition file in *configs/coins*.
**download archive**
URL from where is archive downloaded is defined in *backend.binary_url*.
**verify archive**
There are three different approaches how is archive verification done. Some projects use PGP sign of archive, some
have signed sha256 sums and some don't care about verification at all. So there is option *backend.verification_type* that
could be *gpg*, *gpg-sha256* or *sha256* and chooses particular method.
*gpg* type require file with digital sign and maintainer's public key imported in Docker build image (see below). Sign
file is downloaded from URL defined in *backend.verification_source*. Than is passed to gpg in order to verify archvie.
*gpg-sha256* type require signed checksum file and maintainer's public key imported in Docker build image (see below).
Checksum file is downloaded from URL defined in *backend.verification_source*. Then is verified by gpg and passed to
sha256sum in order to verify archive.
*sha256* type is used for coins that don't support verification at all. In *backend.verification_source* is defined
hexadecimal string that is compared with output of sha256sum. Although this solution is not secure, it avoid download
errors and other surprises at least.
*gpg* and *gpg-sha256* types require maintainer's public key imported in Docker build image. It is not expected that
maintainer's key will change requently while sing or checksum files are changed every release, so it is ideal to
store maintainer's key within image definition. Public keys are stored in *build/docker/deb/gpg-keys* directory. Docker
image must be rebuilt by calling `make build-images`.
**extract archive**
Extraction command is defined in *backend.extract_command*. Content of archive must be extracted to `./backend` directory.
See bitcoin.json and vertcoin.json for different approaches.
**prepare distribution**
There are two steps in this stage – exclude unnecessary files and generate configuration.
Some files are not required for server deployment, some binaries have unnecessary dependencies, so it is good idea to
extract these files from output package. Files to extract are listed in *backend.exclude_files*. Note that paths are
relative to *backend* directory where archive is extracted.
Configuration is described in [config.md](/docs/config.md).
## Manual build
Instructions below are focused on Debian 9 (Stretch). If you want to use another Linux distribution or operating system
like macOS or Windows, please read instructions specific for each project.
Setup go environment:
```
wget https://dl.google.com/go/go1.10.3.linux-amd64.tar.gz && tar xf go1.10.3.linux-amd64.tar.gz
sudo mv go /opt/go
sudo ln -s /opt/go/bin/go /usr/bin/go
# see `go help gopath` for details
mkdir $HOME/go
export GOPATH=$HOME/go
export PATH=$PATH:$GOPATH/bin
```
Install RocksDB: https://github.com/facebook/rocksdb/blob/master/INSTALL.md
and compile the static_lib and tools
```
sudo apt-get update && sudo apt-get install -y \
build-essential git wget pkg-config libzmq3-dev libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev liblz4-dev
git clone https://github.com/facebook/rocksdb.git
cd rocksdb
CFLAGS=-fPIC CXXFLAGS=-fPIC make release
```
Setup variables for gorocksdb: https://github.com/tecbot/gorocksdb
```
export CGO_CFLAGS="-I/path/to/rocksdb/include"
export CGO_LDFLAGS="-L/path/to/rocksdb -lrocksdb -lstdc++ -lm -lz -lbz2 -lsnappy -llz4"
```
Install ZeroMQ: https://github.com/zeromq/libzmq
Install go-dep tool:
```
go get github.com/golang/dep/cmd/dep
```
Get blockbook sources, install dependencies, build:
```
cd $GOPATH/src
git clone https://github.com/trezor/blockbook.git
cd blockbook
dep ensure -vendor-only
go build
```
### Example command
Blockbook require full node daemon as its back-end. You are responsible for proper installation. Port numbers and
daemon configuration are defined in *configs/coins* and *build/templates/backend/config* directories. You should use
specific installation process for particular coin you want run (e.g. https://bitcoin.org/en/full-node#other-linux-distributions for Bitcoin).
When you have running back-end daemon you can start Blockbook. It is highly recommended use ports described in [ports.md](/docs/ports.md)
for both Blockbook and back-end daemon. You can use *contrib/scripts/build-blockchaincfg.sh* that will generate
Blockbook's blockchain configuration from our coin definition files.
Example for Bitcoin:
```
contrib/scripts/build-blockchaincfg.sh
./blockbook -sync -blockchaincfg=build/blockchaincfg.json -internal=:9030 -public=:9130 -certfile=server/testcert -logtostderr
```
This command starts Blockbook with parallel synchronization and providing HTTP and Socket.IO interface, with database
in local directory *data* and established ZeroMQ and RPC connections to back-end daemon specified in configuration
file passed to *-blockchaincfg* option.
Blockbook logs to stderr (option *-logtostderr*) or to directory specified by parameter *-log_dir* . Verbosity of logs can be tuned
by command line parameters *-v* and *-vmodule*, for details see https://godoc.org/github.com/golang/glog.
You can check that Blockbook is running by simple HTTP request: `curl https://localhost:9130`. Returned data is JSON with some
run-time information. If port is closed, Blockbook is syncing data.

105
docs/config.md Normal file
View File

@ -0,0 +1,105 @@
# Configuration
Coin definitions are stored in JSON files in *configs/coins* directory. They are single source of Blockbook
configuration, Blockbook and back-end package definition and build metadata. Since Blockbook supports only single
coin index per running instance, every coin (including testnet) must have single definition file.
Because most of coins are fork of Bitcoin and they have similar way to install and configure their daemon, we use
templates to generate package definition and configuration files during build process. It is similar to build Blockbook
package too. Templates are filled with data from coin definition. Although build process generate packages
automatically, there is sometimes necessary see intermediate step. You can generate all files by calling
`go run build/templates/generate.go coin` where *coin* is name of definition file without .json extension. Files are
generated to *build/pkg-defs* directory.
Good examples of coin configuration are
[*configs/coins/bitcoin.json*](configs/coins/bitcoin.json) and
[*configs/coins/ethereum.json*](configs/coins/ethereum.json) for Bitcoin-like coins and different coins, respectively.
## Description of coin definition
* `coin` – Base information about coin.
* `name` – Name of coin used internally (e.g. "Bcash Testnet").
* `shortcut` – Ticker symbol (code) of coin (e.g. "TBCH").
* `label` – Name of coin used publicly (e.g. "Bitcoin Cash Testnet").
* `alias` – Name of coin used in file system paths and config files. We use convention that name uses lowercase
characters and underscore '_' as a word delimiter. Testnet versions of coins must have *_testnet*
suffix. For example "bcash_testnet".
* `ports` – List of ports used by both back-end and Blockbook. Ports defined here are used in configuration templates
and also as source for generated documentation.
* `backend_rpc` – Port of back-end RPC that is connected by Blockbook service.
* `backend_message_queue` – Port of back-end MQ (if used) that is connected by Blockbook service.
* `backend_*` – Additional back-end ports can be documented here. Actually the only purpose is to get them to
port table (prefix is removed and rest of string is used as note).
* `blockbook_internal` – Blockbook's internal port that is used for metric collecting, debugging etc.
* `blockbook_public` – Blockbook's public port that is used to comunicate with Trezor wallet (via Socket.IO).
* `ipc` – Defines how Blockbook connects its back-end service.
* `rpc_url_template` – Template that defines URL of back-end RPC service. See note on templates below.
* `rpc_user` – User name of back-end RPC service, used by both Blockbook and back-end configuration templates.
* `rpc_pass` – Password of back-end RPC service, used by both Blockbook and back-end configuration templates.
* `rpc_timeout` – RPC timeout used by Blockbook.
* `message_queue_binding_template` – Template that defines URL of back-end's message queue (ZMQ), used by both
Blockbook and back-end configuration template. See note on templates below.
* `backend` – Definition of back-end package, configuration and service.
* `package_name` – Name of package. See convention note in [build guide](/docs/build.md#on-naming-conventions-and-versioning).
* `package_revision` – Revision of package. See convention note in [build guide](/docs/build.md#on-naming-conventions-and-versioning).
* `system_user` – User used to run back-end service. See convention note in [build guide](/docs/build.md#on-naming-conventions-and-versioning).
* `version` – Upstream version. See convention note in [build guide](/docs/build.md#on-naming-conventions-and-versioning).
* `binary_url` – URL of back-end archive.
* `verification_type` – Type of back-end archive verification. Possible values are *gpg*, *gpg-sha256*, *sha256*.
* `verification_source` – Source of sign/checksum of back-end archive.
* `extract_command` – Command to extract back-end archive. It is required to extract content of archive to
*backend* directory.
* `exclude_files` – List of files from back-end archive to exclude. Some files are not required for server
deployment, some binaries have unnecessary dependencies, so it is good idea to extract these files from output
package. Note that paths are relative to *backend* directory where archive is extracted.
* `exec_command_template` – Template of command to execute back-end node daemon. Every back-end node daemon has its
service that is managed by systemd. Template is evaluated to *ExecStart* option in *Service* section of
service unit. See note on templates below.
* `logrotate_files_template` – Template that define log files rotated by logrotate daemon. See note on templates
below.
* `postinst_script_template` – Additional steps in postinst script. See [ZCash definition](configs/coins/zcash.json)
for more information.
* `service_type` – Type of service. Services that daemonize must have *forking* type and write their PID to
*PIDFile*. Services that don't support daemonization must have *simple* type. See examples above.
* `service_additional_params_template` – Additional parameters in service unit. See
[ZCash definition](configs/coins/zcash.json) for more information.
* `protect_memory` – Enables *MemoryDenyWriteExecute* option in service unit if *true*.
* `mainnet` – Set *false* for testnet back-end.
* `config_file` – Name of template of back-end configuration file. Templates are defined in *build/backend/config*.
For Bitcoin-like coins it is not necessary to add extra template, most options can be added via
*additional_params*. For coins that don't require configuration option should be empty (e.g. Ethereum).
* `additional_params` – Object of extra parameters that are added to back-end configuration file as key=value pairs.
Exception is *addnode* key that contains list of nodes that is expanded as addnode=item lines.
* `blockbook` – Definition of Blockbook package, configuration and service.
* `package_name` – Name of package. See convention note in [build guide](/docs/build.md#on-naming-conventions-and-versioning).
* `system_user` – User used to run Blockbook service. See convention note in [build guide](/docs/build.md#on-naming-conventions-and-versioning).
* `internal_binding_template` – Template for *-internal* parameter. See note on templates below.
* `public_binding_template` – Template for *-public* parameter. See note on templates below.
* `explorer_url` – URL of blockchain explorer.
* `additional_params` – Additional params of exec command (see [Dogecoin definition](configs/coins/dogecoin.json)).
* `block_chain` – Configuration of BlockChain type that ensures communication with back-end service. All options
must be tweaked for each individual coin separely.
* `parse` – Use binary parser for block decoding if *true* else call verbose back-end RPC method that returns
JSON. Note that verbose method is slow and not every coin support it. However there are coin implementations
that don't support binary parsing (e.g. ZCash).
* `mempool_workers` – Number of workers for UTXO mempool.
* `mempool_sub_workers` – Number of subworkers for UTXO mempool.
* `block_addresses_to_keep` – Number of blocks that are to be kept in blockaddresses column.
* `additional_params` – Object of coin-specific params.
* `meta` – Common package metadata.
* `package_maintainer` – Full name of package maintainer.
* `package_maintainer_email` – E-mail of package maintainer.
### Go template evaluation note
We use *text/template* package to generate package definitions and configuration files. Some options in coin definition
are also templates and are executed inside base template. Use `{{.path}}` syntax to refer values in coin definition,
where *.path* can be for example *.Blockbook.BlockChain.Parse*. Go uses CamelCase notation so references inside templates
as well. Note that dot at the beginning is mandatory. Go template syntax is fully documented
[here](https://godoc.org/text/template).

26
docs/ports.md Normal file
View File

@ -0,0 +1,26 @@
# Registry of ports
| coin | blockbook internal port | blockbook public port | backend rpc port | backend service ports (zmq) |
|--------------------------|-------------------------|-----------------------|------------------|-----------------------------|
| Bitcoin | 9030 | 9130 | 8030 | 38330 |
| Bcash | 9031 | 9131 | 8031 | 38331 |
| Zcash | 9032 | 9132 | 8032 | 38332 |
| Dash | 9033 | 9133 | 8033 | 38333 |
| Litecoin | 9034 | 9134 | 8034 | 38334 |
| Bgold | 9035 | 9135 | 8035 | 38335 |
| Ethereum | 9036 | 9136 | 8036 | 38336 p2p, 8136 http |
| Ethereum Classic | 9037 | 9137 | 8037 | |
| Dogecoin | 9038 | 9138 | 8038 | 38338 |
| Namecoin | 9039 | 9139 | 8039 | 38339 |
| Vertcoin | 9040 | 9140 | 8040 | 38340 |
| Monacoin | 9041 | 9141 | 8041 | 38341 |
| Testnet | 19030 | 19130 | 18030 | 48330 |
| Bcash Testnet | 19031 | 19131 | 18031 | 48331 |
| Zcash Testnet | 19032 | 19132 | 18032 | 48332 |
| Dash Testnet | 19033 | 19133 | 18033 | 48333 |
| Litecoin Testnet | 19034 | 19134 | 18034 | 48334 |
| Ethereum Testnet Ropsten | 19036 | 19136 | 18036 | 48336 p2p |
| Vertcoin Testnet | 19040 | 19140 | 18040 | 48340 |
| Monacoin Testnet | 19041 | 19141 | 18041 | 48341 |
> NOTE: This document is generated from coin definitions in `configs/coins`.

44
docs/rocksdb.md Normal file
View File

@ -0,0 +1,44 @@
# Data storage in RocksDB
Blockbook stores data the key-value store RocksDB. Data are stored in binary form to save space.
The data are separated to different column families:
- **default**
at the moment not used, will store statistical data etc.
- **height** - maps *block height* to *block hash*
*Block heigh* stored as array of 4 bytes (big endian uint32)
*Block hash* stored as array of 32 bytes
Example - the first four blocks (all data hex encoded)
```
0x00000000 : 0x000000000933ea01ad0ee984209779baaec3ced90fa3f408719526f8d77f4943
0x00000001 : 0x00000000b873e79784647a6c82962c70d228557d24a747ea4d1b8bbe878e1206
0x00000002 : 0x000000006c02c8ea6e4ff69651f7fcde348fb9d557a06e6957b65552002a7820
0x00000003 : 0x000000008b896e272758da5297bcd98fdc6d97c9b765ecec401e286dc1fdbe10
```
- **outputs** - maps *output script+block height* to *array of outpoints*
*Output script (ScriptPubKey)+block height* stored as variable length array of bytes for output script + 4 bytes (big endian uint32) block height
*array of outpoints* stored as array of 32 bytes for transaction id + variable length outpoint index for each outpoint
Example - (all data hex encoded)
```
0x001400efeb484a24a1c1240eafacef8566e734da429c000e2df6 : 0x1697966cbd76c75eb9fc736dfa3ba0bc045999bab1e8b10082bc0ba546b0178302
0xa9143e3d6abe282d92a28cb791697ba001d733cefdc7870012c4b1 : 0x7246e79f97b5f82e7f51e291d533964028ec90be0634af8a8ef7d5a903c7f6d301
```
- **inputs** - maps *transaction outpoint* to *input transaction* that spends it
*Transaction outpoint* stored as array of 32 bytes for transaction id + variable length outpoint index
*Input transaction* stored as array of 32 bytes for transaction id + variable length input index
Example - (all data hex encoded)
```
0x7246e79f97b5f82e7f51e291d533964028ec90be0634af8a8ef7d5a903c7f6d300 : 0x0a7aa90ea0269c79f844c516805e4cac594adb8830e56fca894b66aab19136a428
0x7246e79f97b5f82e7f51e291d533964028ec90be0634af8a8ef7d5a903c7f6d301 : 0x4303a9fcfe6026b4d33ba488df6443c9a99bca7b7fcb7c6f6cd65cea24a749b700
```

71
docs/testing.md Normal file
View File

@ -0,0 +1,71 @@
# Testing
There are two kinds of tests in Blockbook – unit tests and integration tests. Because integration tests require running
back-end daemon so they can't be executed at every build. Blockbook's build system uses conditional compilation to
distinguish which tests should be executed.
To execute unit tests run `make test`. To execute unit tests and integration tests run `make test-all`. You can use
Go's flag *-run* to filter which tests should be executed. Use *ARGS* variable, e.g.
`make test-all ARGS="-run TestBitcoin"`.
## Unit tests
Unit test file must start with constraint `// +build unittest` followed by blank line (constraints are described
[here](https://golang.org/pkg/go/build/#hdr-Build_Constraints)).
Every coin implementation should have unit tests. At least for parser. Usual test suite define real transaction data
and try pack and unpack them. Specialities of particular coin are tested too. See examples in
[bcashparser_test.go](/bchain/coins/bch/bcashparser_test.go),
[bitcoinparser_test.go](/bchain/coins/btc/bitcoinparser_test.go) and
[ethparser_test.go](/bchain/coins/eth/ethparser_test.go).
## Integration tests
Integration test file must start with constraint `// +build integration` followed by blank line (constraints are
described [here](https://golang.org/pkg/go/build/#hdr-Build_Constraints)).
> Tests that cannot connect back-end service are skipped. `go test` doesn't show any information about skipped test,
> so you must run it in verbose mode with flag *-v*.
### Blockbook integration tests
TODO
### Back-end RPC integration tests
This kind of tests test *bchain.BlockChain* implementation and its capability to communicate with back-end RPC. Tests
of most of coins are similar so there is single generalized implementation in package *blockbook/bchain/tests/rpc*. Test
functions of particular coin implementation can just initialize test object and call its methods. Configuration of tests
is stored in *blockbook/bchain/tests/rpc/config.json* and consists of back-end URL and credentials. Every test suite also
has fixtures stored in *blockbook/bchain/tests/rpc/testdata*. Content is obvious from existing files.
Tests listed below just call back-end RPC methods with parameters from fixture file and check results against same
fixture file. So data in fixture file must be related together.
* TestGetBlockHash – Calls *BlockChain.GetBlockHash* with height and checks returned hash.
* TestGetBlockHeader – Calls *BlockChain.GetBlockHeader* with hash and check returned header. Note that only fields
that are significant are *Hash* and *Height*, they are checked against fixtures.
* TestGetBlock – Calls *BlockChain.GetBlock* with hash and checks returned block (actually number of transactions and
their txids).
* TestGetTransaction – Calls *BlockChain.GetTransaction* with txid and checks result against transaction object, where
*txid* is key and* **transaction object* is value of *txDetails* object in fixture file.
* TestGetTransactionForMempool – Calls *BlockChain.GetTransactionForMempool* that should be version of
*BlockChain.GetTransaction* optimized for mempool. Implementation of test is similar.
* TestGetMempoolEntry – Calls *BlockChain.GetMempoolEntry* and checks result. Because mempool is living structure it
tries to load entry for random transaction in mempool repeatedly.
* TestEstimateSmartFee – Calls *BlockChain.EstimateSmartFee* for few numbers of blocks and checks if returned fee is
non-negative.
* TestEstimateFee – Calls *BlockChain.EstimateFee*; implementation is same as *TestEstimateSmartFee*.
* TestGetBestBlockHash – Calls *BlockChain.GetBestBlockHash* and verifies that returned hash matches the really last
block.
* TestGetBestBlockHeight – Calls *BlockChain.GetBestBlockHeight* and verifies that returned height matches the really
last block.
TODO: TestMempoolSync should be "Blockbook integration test"
* TestMempoolSync – Synchronize *BlockChain*'s mempool and verify if sync was successful.
For example see [bitcoinrpc_test.go](/bchain/coins/btc/bitcoinrpc_test.go) and
[implementation](/bchain/tests/rpc/rpc.go) of test suite.