From 814b8d6663753167f55bc5bcca8be52e9ecec23c Mon Sep 17 00:00:00 2001 From: Manuel Araoz Date: Thu, 21 Aug 2014 10:17:12 -0400 Subject: [PATCH 01/16] update readme 1 --- README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index abf3c08..203740a 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,11 @@ # *insight API* -*insight API* is an open-source bitcoin blockchain REST +*insight API* is an open-source bitcoin blockchain REST and websocket API. Insight API runs in NodeJS and use LevelDB for storage. +This is a backend-only service. If you're looking for the web frontend application, +take a look at https://github.com/bitpay/insight. + *Insight API* allows to develop bitcoin related applications such as wallets that require certain information from the blockchain that bitcoind does not provide. From d84ef323d04fe1058b50941b91de0dd79bc6d79e Mon Sep 17 00:00:00 2001 From: Manuel Araoz Date: Thu, 21 Aug 2014 10:17:19 -0400 Subject: [PATCH 02/16] update readme 2 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 203740a..6315654 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # *insight API* *insight API* is an open-source bitcoin blockchain REST -and websocket API. Insight API runs in NodeJS and use LevelDB for storage. +and websocket API. Insight API runs in NodeJS and uses LevelDB for storage. This is a backend-only service. If you're looking for the web frontend application, take a look at https://github.com/bitpay/insight. From aa488b1cbfbb9cdd361ace838182a563a46fb132 Mon Sep 17 00:00:00 2001 From: Manuel Araoz Date: Thu, 21 Aug 2014 10:17:51 -0400 Subject: [PATCH 03/16] update readme 3 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 6315654..a1539ce 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ and websocket API. Insight API runs in NodeJS and uses LevelDB for storage. This is a backend-only service. If you're looking for the web frontend application, take a look at https://github.com/bitpay/insight. -*Insight API* allows to develop bitcoin related applications such as wallets that +*Insight API* allows to develop bitcoin-related applications (such as wallets) that require certain information from the blockchain that bitcoind does not provide. A blockchain explorer front-end have been developed to top of *Insight API*, it can From 2e50e135a3fbcc4174f3f50f2e09581b14259ef8 Mon Sep 17 00:00:00 2001 From: Manuel Araoz Date: Thu, 21 Aug 2014 10:18:14 -0400 Subject: [PATCH 04/16] update readme 4 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index a1539ce..5ba2287 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ take a look at https://github.com/bitpay/insight. *Insight API* allows to develop bitcoin-related applications (such as wallets) that require certain information from the blockchain that bitcoind does not provide. -A blockchain explorer front-end have been developed to top of *Insight API*, it can +A blockchain explorer front-end has been developed on top of *Insight API*. It can be downloaded at [Github Insight Repository](https://github.com/bitpay/insight). ## IMPORTANT: Upgrading from v0.1 to v0.2 From 67050ff3d1eb9e9692e690723193057989ba1d67 Mon Sep 17 00:00:00 2001 From: Manuel Araoz Date: Thu, 21 Aug 2014 10:21:00 -0400 Subject: [PATCH 05/16] update readme 6 --- README.md | 32 +++++++++++++------------------- 1 file changed, 13 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index 5ba2287..9a449c8 100644 --- a/README.md +++ b/README.md @@ -12,25 +12,6 @@ require certain information from the blockchain that bitcoind does not provide. A blockchain explorer front-end has been developed on top of *Insight API*. It can be downloaded at [Github Insight Repository](https://github.com/bitpay/insight). -## IMPORTANT: Upgrading from v0.1 to v0.2 -In order to optimize some queries, the key-value layout in Level DB has been changed. -If you are running v0.1.x, a total resync needs to be done, running `$ util/sync.js -D`. This -needs to run while *insight-api* is shut off. See details at: **Synchronization**. - - -## IMPORTANT: v0.2 Caching schema - -In v0.2 a new cache schema has been introduced. Only information from transactions with -INSIGHT_SAFE_CONFIRMATIONS+ settings will be cached (by default SAFE_CONFIRMATIONS=6). There -are 3 different caches: - * nr. of confirmations - * transaction spent information - * scriptPubKey for unspent transactions - -Cache data is only populated on request, i.e., only after accessing the required data for -the first time, the information is cached, there is not pre-caching procedure. To ignore -cache by default, use INSIGHT_IGNORE_CACHE. Also, address related calls support `?noCache=1` -to ignore the cache in a particular API request. ## Prerequisites @@ -146,6 +127,19 @@ To run the tests Contributions and suggestions are welcome at [insight-api github repository](https://github.com/bitpay/insight-api). +### Caching schema + +Since v0.2 a new cache schema has been introduced. Only information from transactions with +INSIGHT_SAFE_CONFIRMATIONS settings will be cached (by default SAFE_CONFIRMATIONS=6). There +are 3 different caches: + * nr. of confirmations + * transaction spent information + * scriptPubKey for unspent transactions + +Cache data is only populated on request, i.e., only after accessing the required data for +the first time, the information is cached, there is not pre-caching procedure. To ignore +cache by default, use INSIGHT_IGNORE_CACHE. Also, address related calls support `?noCache=1` +to ignore the cache in a particular API request. ## API From 4017a3a6c02cdca056a134ec5b7d1f6747b8a4e0 Mon Sep 17 00:00:00 2001 From: Manuel Araoz Date: Thu, 21 Aug 2014 10:21:23 -0400 Subject: [PATCH 06/16] update readme 7 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 9a449c8..86fe0e5 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ be downloaded at [Github Insight Repository](https://github.com/bitpay/insight). * **bitcoind** - Download and Install [Bitcoin](http://bitcoin.org/en/download) *insight API* needs a *trusted* bitcoind node to run. *insight API* will connect to the node -thru the RPC API, Peer-to-peer protocol and will even read its raw .dat files for syncing. +through the RPC API, Peer-to-peer protocol and will even read its raw .dat files for syncing. Configure bitcoind to listen to RPC calls and set `txindex` to true. The easiest way to do this is by copying `./etc/bitcoind/bitcoin.conf` to your From abda6212570a3ec3eae7e333f0eb87c29eef618f Mon Sep 17 00:00:00 2001 From: Manuel Araoz Date: Thu, 21 Aug 2014 10:22:00 -0400 Subject: [PATCH 07/16] update readme 8 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 86fe0e5..5c5a9d0 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ be downloaded at [Github Insight Repository](https://github.com/bitpay/insight). * **bitcoind** - Download and Install [Bitcoin](http://bitcoin.org/en/download) *insight API* needs a *trusted* bitcoind node to run. *insight API* will connect to the node -through the RPC API, Peer-to-peer protocol and will even read its raw .dat files for syncing. +through the RPC API, bitcoin peer-to-peer protocol, and will even read its raw block .dat files for syncing. Configure bitcoind to listen to RPC calls and set `txindex` to true. The easiest way to do this is by copying `./etc/bitcoind/bitcoin.conf` to your From 04fcdc5e6c570880fce8ae3e6cb2370c48b2a94a Mon Sep 17 00:00:00 2001 From: Manuel Araoz Date: Thu, 21 Aug 2014 10:23:17 -0400 Subject: [PATCH 08/16] update readme 9 --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 5c5a9d0..0f9dea3 100644 --- a/README.md +++ b/README.md @@ -22,8 +22,8 @@ through the RPC API, bitcoin peer-to-peer protocol, and will even read its raw b Configure bitcoind to listen to RPC calls and set `txindex` to true. The easiest way to do this is by copying `./etc/bitcoind/bitcoin.conf` to your -bitcoin data directory (usually `"~/.bitcoin"` on Linux, `"%appdata%\Bitcoin\"` on Windows, -or `"~/Library/Application Support/Bitcoin"` on Mac OS X). +bitcoin data directory (usually `~/.bitcoin` on Linux, `%appdata%\Bitcoin\` on Windows, +or `~/Library/Application Support/Bitcoin` on Mac OS X). bitcoind must be running and must have finished downloading the blockchain **before** running *insight API*. From ca45c3110c3f06ff5e16bbe4ecbdd899493950da Mon Sep 17 00:00:00 2001 From: Manuel Araoz Date: Thu, 21 Aug 2014 10:24:20 -0400 Subject: [PATCH 09/16] update readme 10 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 0f9dea3..26812ab 100644 --- a/README.md +++ b/README.md @@ -53,7 +53,7 @@ bitcoind must be running and must have finished downloading the blockchain **bef Please note that the app will need to sync its internal database with the blockchain state, which may take some time. You can check - sync progress from within the web interface. + sync progress at http://localhost:3001/api/sync. ## Configuration From 8806abb5a6a3144d1d7d3cb99cf65f92e2caf1c9 Mon Sep 17 00:00:00 2001 From: Manuel Araoz Date: Thu, 21 Aug 2014 10:28:41 -0400 Subject: [PATCH 10/16] update readme 11 --- README.md | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 26812ab..4ed4fe0 100644 --- a/README.md +++ b/README.md @@ -83,15 +83,26 @@ In case the network is changed (testnet to livenet or vice versa) levelDB databa ## Synchronization -The initial synchronization process scans the blockchain from the paired bitcoind server to update addresses and balances. *insight* needs one (and only one) trusted bitcoind node to run. This node must have finished downloading the blockchain before running *insight*. +The initial synchronization process scans the blockchain from the paired bitcoind server to update addresses and balances. *insight-api* needs exactly one trusted bitcoind node to run. This node must have finished downloading the blockchain before running *insight-api*. While *insight* is synchronizing the website can be accessed (the sync process is embedded in the webserver), but there may be missing data or incorrect balances for addresses. The 'sync' status is shown at the `/api/sync` endpoint. -The blockchain can be read from bitcoind's raw `.dat` files or RPC interface. Reading the information from the `.dat` files is much faster so it's the recommended (and default) alternative. `.dat` files are scanned in the default location for each platform (for example, `~/.bitcoin` on Linux). In case a non-standard location is used, it needs to be defined (see the Configuration section). As of June 2014, using `.dat` files the sync process takes 9 hrs. for livenet and 30 mins. for testnet. +The blockchain can be read from bitcoind's raw `.dat` files or RPC interface. +Reading the information from the `.dat` files is much faster so it's the +recommended (and default) alternative. `.dat` files are scanned in the default +location for each platform (for example, `~/.bitcoin` on Linux). In case a +non-standard location is used, it needs to be defined (see the Configuration section). +As of June 2014, using `.dat` files the sync process takes 9 hrs. +for livenet and 30 mins. for testnet. -While synchronizing the blockchain, *insight* listens for new blocks and transactions relayed by the bitcoind node. Those are also stored on *insight*'s database. In case *insight* is shutdown for a period of time, restarting it will trigger a partial (historic) synchronization of the blockchain. Depending on the size of that synchronization task, a reverse RPC or forward `.dat` syncing strategy will be used. +While synchronizing the blockchain, *insight-api* listens for new blocks and +transactions relayed by the bitcoind node. Those are also stored on *insight-api*'s database. +In case *insight-api* is shutdown for a period of time, restarting it will trigger +a partial (historic) synchronization of the blockchain. Depending on the size of +that synchronization task, a reverse RPC or forward `.dat` syncing strategy will be used. -If bitcoind is shutdown, *insight* needs to be stopped and restarted once bitcoind is restarted. +If bitcoind is shutdown, *insight-api* needs to be stopped and restarted +once bitcoind is restarted. ### Syncing old blockchain data manually @@ -101,7 +112,7 @@ If bitcoind is shutdown, *insight* needs to be stopped and restarted once bitcoi Check util/sync.js --help for options, particulary -D to erase the current DB. - *NOTE* that there is no need to run this manually since the historic synchronization is embedded on the web application, so by running you will trigger the historic sync automatically. + *NOTE*: there is no need to run this manually since the historic synchronization is embedded on the web application, so by running you will trigger the historic sync automatically. ### DB storage requirement From d1324d35807f0a8853314ea93cfe60ef338387c7 Mon Sep 17 00:00:00 2001 From: Manuel Araoz Date: Thu, 21 Aug 2014 10:29:44 -0400 Subject: [PATCH 11/16] update readme 12 --- README.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 4ed4fe0..701ae3c 100644 --- a/README.md +++ b/README.md @@ -112,12 +112,15 @@ once bitcoind is restarted. Check util/sync.js --help for options, particulary -D to erase the current DB. - *NOTE*: there is no need to run this manually since the historic synchronization is embedded on the web application, so by running you will trigger the historic sync automatically. + *NOTE*: there is no need to run this manually since the historic synchronization + is built in into the web application. Running *insight-api* normally will trigger + the historic sync automatically. ### DB storage requirement -To store the blockchain and address related information, *insight* uses LevelDB. Two DBs are created: txs and blocks. By default these are stored on +To store the blockchain and address related information, *insight-api* uses LevelDB. +Two DBs are created: txs and blocks. By default these are stored on ``~/.insight/`` From f5299b787245ff76522872450ba158efa03accc1 Mon Sep 17 00:00:00 2001 From: Manuel Araoz Date: Thu, 21 Aug 2014 10:30:01 -0400 Subject: [PATCH 12/16] update readme 13 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 701ae3c..de0ad28 100644 --- a/README.md +++ b/README.md @@ -124,7 +124,7 @@ Two DBs are created: txs and blocks. By default these are stored on ``~/.insight/`` -Please note that previous version's of Insight-API store that on `/db` +Please note that some older versions of Insight-API store that on `/db` this can be changed on config/config.js. As of June 2014, storing the livenet blockchain takes ~35GB of disk space (2GB for the testnet). From 1df11d95e270683666f58fe5435b8255e02e878d Mon Sep 17 00:00:00 2001 From: Manuel Araoz Date: Thu, 21 Aug 2014 10:30:58 -0400 Subject: [PATCH 13/16] update readme 14 --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index de0ad28..32f3182 100644 --- a/README.md +++ b/README.md @@ -124,9 +124,9 @@ Two DBs are created: txs and blocks. By default these are stored on ``~/.insight/`` -Please note that some older versions of Insight-API store that on `/db` +Please note that some older versions of Insight-API store that on `/db`. -this can be changed on config/config.js. As of June 2014, storing the livenet blockchain takes ~35GB of disk space (2GB for the testnet). +This can be changed at config/config.js. As of June 2014, storing the livenet blockchain takes ~35GB of disk space (2GB for the testnet). ## Development @@ -141,7 +141,7 @@ To run the tests Contributions and suggestions are welcome at [insight-api github repository](https://github.com/bitpay/insight-api). -### Caching schema +## Caching schema Since v0.2 a new cache schema has been introduced. Only information from transactions with INSIGHT_SAFE_CONFIRMATIONS settings will be cached (by default SAFE_CONFIRMATIONS=6). There From 2397958b8bdb0b7433413909c1e9d08f1814caf8 Mon Sep 17 00:00:00 2001 From: Manuel Araoz Date: Thu, 21 Aug 2014 10:31:34 -0400 Subject: [PATCH 14/16] update readme 15 --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 32f3182..9c68f9c 100644 --- a/README.md +++ b/README.md @@ -146,8 +146,8 @@ Contributions and suggestions are welcome at [insight-api github repository](htt Since v0.2 a new cache schema has been introduced. Only information from transactions with INSIGHT_SAFE_CONFIRMATIONS settings will be cached (by default SAFE_CONFIRMATIONS=6). There are 3 different caches: - * nr. of confirmations - * transaction spent information + * Number of confirmations + * Transaction output spent/unspent status * scriptPubKey for unspent transactions Cache data is only populated on request, i.e., only after accessing the required data for From e2b737acba2fdc15ab58e0e6901f1581774081d7 Mon Sep 17 00:00:00 2001 From: Manuel Araoz Date: Thu, 21 Aug 2014 10:32:46 -0400 Subject: [PATCH 15/16] update readme 16 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 9c68f9c..34e8f4b 100644 --- a/README.md +++ b/README.md @@ -213,7 +213,7 @@ Sample return: } ] ``` -Please note that in case confirmations are cached (because the number of confirmations if bigger that INSIGHT_SAFE_CONFIRMATIONS) the return will include the pair confirmationsFromCache:true, and confirmations will equal INSIGHT_SAFE_CONFIRMATIONS. See noCache and INSIGHT_IGNORE_CACHE options for details. +Please note that in case confirmations are cached (which happens by default when the number of confirmations is bigger that INSIGHT_SAFE_CONFIRMATIONS) the response will include the pair confirmationsFromCache:true, and confirmations will equal INSIGHT_SAFE_CONFIRMATIONS. See noCache and INSIGHT_IGNORE_CACHE options for details. From 9367d9528b9e54e4ba7c1e347bff00b98d8a854c Mon Sep 17 00:00:00 2001 From: Manuel Araoz Date: Thu, 21 Aug 2014 10:33:27 -0400 Subject: [PATCH 16/16] update readme 17 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 34e8f4b..015021a 100644 --- a/README.md +++ b/README.md @@ -281,7 +281,7 @@ POST response: /api/peer ``` -### Status network +### Status of the bitcoin network ``` /api/status?q=xxx ```