Update documentation for protocol 1.4

docs/protocol-basics.rst

@@ -194,3 +194,43 @@ inputs are confirmed.
 4. The :dfn:`status` of the script hash is the :func:`sha256` hash of the
 full string expressed as a hexadecimal string, or :const:`null` if the
 string is empty because there are no transactions.
+
+
+Block Headers
+-------------
+
+Originally Electrum clients would download all block headers and
+verify the chain of hashes and header difficulty in order to confirm
+the merkle roots with which to check transaction inclusion.
+
+With the BTC and BCH chains now past height 500,000, the headers form
+over 40MB of raw data which becomes 80MB if downloaded as text from
+Electrum servers.  The situation is worse for testnet and coins with
+more frequent blocks.  Downloading and verifying all this data on
+initial use would take several minutes, during which Electrum was
+non-responsive.
+
+To facilitate a better experience for SPV clients, particularly on
+mobile, protocol :ref:`version 1.4 <version 1.4>` introduces an
+optional *cp_height* argument to the :func:`blockchain.block.header`
+and :func:`blockchain.block.headers` RPC calls.
+
+This requests the server provide a merkle proof, to a single 32-byte
+checkpoint hard-coded in the client, that the header(s) provided are
+valid in the same way the server proves a transaction is included in a
+block.  If several consecutive headers are requested, the proof is
+provided for the final header - the *prev_hash* links in the headers
+are sufficient to prove the others valid.
+
+Using this feature client software only needs to download the headers
+it is interested in up to the checkpoint.  HHeaders after the
+checkpoint must all be downloaded and validated as before.  The RPC
+calls return the merkle root, so to embed a checkpoint in a client
+simply make an RPC request to a couple of trusted servers for the
+greatest height to which a reorganisation of the chain is infeasible,
+and confirm the returned roots match.
+
+.. note:: with 500,000 headers of 80 bytes each, a naïve server
+  implementation would require hashing approximately 88MB of data to
+  provide a single merkle proof.  ElectrumX implements an optimization
+  such that it hashes only approximately 180KB of data per proof.

docs/protocol-changes.rst

@@ -121,15 +121,16 @@ Deprecated methods
 Version 1.4
 ===========
 
-This documents the current intent for protocol version 1.4 which is
+This version removes all support for :ref:`deserialized headers
-not yet implemented.  It removes all support for :ref:`deserialized
+<deserialized header>`.
-headers <deserialized header>`.
 
 Changes
 -------
 
   * The only valid value for :func:`blockchain.headers.subscribe`
     argument *raw* is :const:`True`
+  * Optional *cp_height* argument added to
+    :func:`blockchain.block.header` and :func:`blockchain.block.headers`
 
 Removed methods
 ---------------

docs/protocol-methods.rst

@@ -2,69 +2,6 @@
  Protocol Methods
 ==================
 
-blockchain.block.get_header
-===========================
-
-Return the :ref:`deserialized header <deserialized header>` of the
-block at the given height.
-
-**Signature**
-
-  .. function:: blockchain.block.get_header(height)
-  .. deprecated:: 1.3
-
-  *height*
-
-    The height of the block, an integer.
-
-**Result**
-
-  The coin-specific :ref:`deserialized header <deserialized header>`.
-
-**Example Result**
-
-::
-
-  {
-    "bits": 392292856,
-    "block_height": 510000,
-    "merkle_root": "297cfcc6a66e063692b20650d21cc0ac7a2a80f7277ebd7c5d6c7010a070d25c",
-    "nonce": 3347656422,
-    "prev_block_hash": "0000000000000000002292de0d9f03dfa15a04dbf09102d5d4552117b717fa86",
-    "timestamp": 1519083654,
-    "version": 536870912
-  }
-
-blockchain.block.get_chunk
-==========================
-
-Return a concatenated chunk of block headers from the main chain.
-Typically, a chunk consists of a fixed number of block headers over
-which difficulty is constant, and at the end of which difficulty is
-retargeted.
-
-In the case of Bitcoin a chunk is 2,016 headers, each of 80 bytes, so
-chunk 5 consists of the block headers from height 10,080 to 12,095
-inclusive.  When encoded as hexadecimal, the result string is twice as
-long, so for Bitcoin it takes 322,560 bytes, making this a
-bandwidth-intensive request.
-
-**Signature**
-
-  .. function:: blockchain.block.get_chunk(index)
-  .. deprecated:: 1.2
-
-  *index*
-
-    The zero-based index of the chunk, an integer.
-
-**Result**
-
-    The binary block headers as hexadecimal strings, in-order and
-    concatenated together.  As many as headers as are available at the
-    implied starting height will be returned; this may range from zero
-    to the coin-specific chunk size.
-
 blockchain.block.header
 =======================
 
@@ -72,16 +9,44 @@ Return the block header at the given height.
 
 **Signature**
 
-  .. function:: blockchain.block.header(height)
+  .. function:: blockchain.block.header(height, cp_height=0)
   .. versionadded:: 1.3
+  .. versionchanged:: 1.4
+     *cp_height* parameter added
 
   *height*
 
-    The height of the block, an integer.
+    The height of the block, a non-negative integer.
+
+  *cp_height*
+
+    Checkpoint height, a non-negative integer.  Ignored if zero,
+    otherwise the following must hold:
+
+      *height* <= *cp_height*
 
 **Result**
 
-  The raw block header as a hexadecimal string.
+  If *cp_height* is zero, the raw block header as a hexadecimal
+  string.
+
+  Otherwise a dictionary with the following keys.  This provides a
+  proof that the given header is present in the blockchain; presumably
+  the client has the merkle root hard-coded as a checkpoint.
+
+  * *header*
+
+    The raw block header as a hexadecimal string.
+
+  * *root*
+
+    The merkle root of all blockchain headers up to and including
+    *cp_height*.
+
+  * *branch*
+
+    The merkle branch of *header* up to *root*, deepest pairing first.
+
 
 **Example Result**
 
@@ -89,6 +54,7 @@ Return the block header at the given height.
 
    "0100000085144a84488ea88d221c8bd6c059da090e88f8a2c99690ee55dbba4e00000000e11c48fecdd9e72510ca84f023370c9a38bf91ac5cae88019bee94d24528526344c36649ffff001d1d03e477"
 
+
 blockchain.block.headers
 ========================
 
@@ -96,8 +62,10 @@ Return a concatenated chunk of block headers from the main chain.
 
 **Signature**
 
-  .. function:: blockchain.block.headers(start_height, count)
+  .. function:: blockchain.block.headers(start_height, count, cp_height=0)
   .. versionadded:: 1.2
+  .. versionchanged:: 1.4
+     *cp_height* parameter added
 
   *start_height*
 
@@ -107,6 +75,13 @@ Return a concatenated chunk of block headers from the main chain.
 
     The number of headers requested, a non-negative integer.
 
+  *cp_height*
+
+    Checkpoint height, a non-negative integer.  Ignored if zero,
+    otherwise the following must hold:
+
+      *start_height* + (*count* - 1) <= *cp_height*
+
 **Result**
 
   A dictionary with the following members:
@@ -128,6 +103,22 @@ Return a concatenated chunk of block headers from the main chain.
     The maximum number of headers the server will return in a single
     request.
 
+  The dictionary additionally has the following keys if *count* and
+  *cp_height* are not zero.  This provides a proof that all the given
+  headers are present in the blockchain; presumably the client has the
+  merkle root hard-coded as a checkpoint.
+
+  * *root*
+
+    The merkle root of all blockchain headers up to and including
+    *cp_height*.
+
+  * *branch*
+
+    The merkle branch of the last returned header up to *root*,
+    deepest pairing first.
+
+
 **Example Response**
 
 ::
@@ -135,7 +126,7 @@ Return a concatenated chunk of block headers from the main chain.
   {
     "count": 2,
     "hex": "0100000000000000000000000000000000000000000000000000000000000000000000003ba3edfd7a7b12b27ac72c3e67768f617fc81bc3888a51323a9fb8aa4b1e5e4a29ab5f49ffff001d1dac2b7c010000006fe28c0ab6f1b372c1a6a246ae63f74f931e8365e15a089c68d6190000000000982051fd1e4ba744bbbe680e1fee14677ba1a3c3540bf7b1cdb606e857233e0e61bc6649ffff001d01e36299"
-     "max": 2016
+    "max": 2016
   }
 
 blockchain.estimatefee
@@ -600,7 +591,7 @@ When *verbose* is :const:`True`::
 blockchain.transaction.get_merkle
 =================================
 
-Return the markle branch to a confirmed transaction given its hash
+Return the merkle branch to a confirmed transaction given its hash
 and height.
 
 **Signature**

docs/protocol-removed.rst

@@ -13,8 +13,7 @@ Return the confirmed and unconfirmed balances of a bitcoin address.
 **Signature**
 
   .. function:: blockchain.address.get_balance(address)
-  .. deprecated:: 1.2
+  .. deprecated:: 1.2 removed in version 1.3
-     *Removed in version 1.3.*
 
   * *address*
 
@@ -32,8 +31,7 @@ Return the confirmed and unconfirmed history of a bitcoin address.
 **Signature**
 
   .. function:: blockchain.address.get_history(address)
-  .. deprecated:: 1.2
+  .. deprecated:: 1.2 removed in version 1.3
-     *Removed in version 1.3.*
 
   * *address*
 
@@ -51,8 +49,7 @@ Return the unconfirmed transactions of a bitcoin address.
 **Signature**
 
   .. function:: blockchain.address.get_mempool(address)
-  .. deprecated:: 1.2
+  .. deprecated:: 1.2 removed in version 1.3
-     *Removed in version 1.3.*
 
   * *address*
 
@@ -70,8 +67,7 @@ Return an ordered list of UTXOs sent to a bitcoin address.
 **Signature**
 
   .. function:: blockchain.address.listunspent(address)
-  .. deprecated:: 1.2
+  .. deprecated:: 1.2 removed in version 1.3
-     *Removed in version 1.3.*
 
   * *address*
 
@@ -89,8 +85,7 @@ Subscribe to a bitcoin address.
 **Signature**
 
   .. function:: blockchain.address.subscribe(address)
-  .. deprecated:: 1.2
+  .. deprecated:: 1.2 removed in version 1.3
-     *Removed in version 1.3.*
 
   *address*
 
@@ -116,8 +111,7 @@ Subscribe to receive the block height when a new block is found.
 **Signature**
 
   .. function:: blockchain.numblocks.subscribe()
-  .. deprecated:: 1.0
+  .. deprecated:: 1.0 removed in version 1.1
-     *Removed in version 1.1.*
 
 **Result**
 
@@ -139,7 +133,7 @@ Return the address paid to by a UTXO.
 
   .. function:: blockchain.utxo.get_address(tx_hash, index)
 
-  *Optional in version 1.0.  Removed in version 1.1.*
+    *Optional in version 1.0, removed in version 1.1*
 
   *tx_hash*
 
@@ -155,3 +149,66 @@ Return the address paid to by a UTXO.
   doesn't exist, the index is out of range, or the output is not paid
   to an address, :const:`null` must be returned.  If the output is
   spent :const:`null` *may* be returned.
+
+blockchain.block.get_header
+===========================
+
+Return the :ref:`deserialized header <deserialized header>` of the
+block at the given height.
+
+**Signature**
+
+  .. function:: blockchain.block.get_header(height)
+  .. deprecated:: 1.3 removed in version 1.4
+
+  *height*
+
+    The height of the block, an integer.
+
+**Result**
+
+  The coin-specific :ref:`deserialized header <deserialized header>`.
+
+**Example Result**
+
+::
+
+  {
+    "bits": 392292856,
+    "block_height": 510000,
+    "merkle_root": "297cfcc6a66e063692b20650d21cc0ac7a2a80f7277ebd7c5d6c7010a070d25c",
+    "nonce": 3347656422,
+    "prev_block_hash": "0000000000000000002292de0d9f03dfa15a04dbf09102d5d4552117b717fa86",
+    "timestamp": 1519083654,
+    "version": 536870912
+  }
+
+blockchain.block.get_chunk
+==========================
+
+Return a concatenated chunk of block headers from the main chain.
+Typically, a chunk consists of a fixed number of block headers over
+which difficulty is constant, and at the end of which difficulty is
+retargeted.
+
+In the case of Bitcoin a chunk is 2,016 headers, each of 80 bytes, so
+chunk 5 consists of the block headers from height 10,080 to 12,095
+inclusive.  When encoded as hexadecimal, the result string is twice as
+long, so for Bitcoin it takes 322,560 bytes, making this a
+bandwidth-intensive request.
+
+**Signature**
+
+  .. function:: blockchain.block.get_chunk(index)
+  .. deprecated:: 1.2 removed in version 1.4
+
+  *index*
+
+    The zero-based index of the chunk, an integer.
+
+**Result**
+
+    The binary block headers as hexadecimal strings, in-order and
+    concatenated together.  As many as headers as are available at the
+    implied starting height will be returned; this may range from zero
+    to the coin-specific chunk size.