diff --git a/README.rst b/README.rst index 09643c4..688ca5c 100644 --- a/README.rst +++ b/README.rst @@ -3,84 +3,86 @@ .. image:: https://coveralls.io/repos/github/kyuupichan/electrumx/badge.svg :target: https://coveralls.io/github/kyuupichan/electrumx - -ElectrumX - Reimplementation of Electrum-server +=============================================== +ElectrumX - Reimplementation of electrum-server =============================================== :: - Licence: MIT Licence + Licence: MIT Author: Neil Booth Language: Python (>=3.5) - Getting Started =============== -See :code:`docs/HOWTO.rst`. +See `docs/HOWTO.rst`_. Motivation ========== -For privacy and other reasons, I have long wanted to run my own -Electrum server, but for reasons I cannot remember I struggled to set -it up or get it to work on my DragonFlyBSD system, and I lost interest -for over a year. +Mainly for privacy reasons, I have long wanted to run my own Electrum +server, but I struggled to set it up or get it to work on my +DragonFlyBSD system and lost interest for over a year. -More recently I heard that Electrum server databases were around 35GB -in size when gzipped, and had sync times from Genesis of over a week -(and sufficiently painful that no one seems to have done one for a -long time) and got curious about improvements. After taking a look at -the existing server code I decided to try a different approach. +In September 2015 I heard that electrum-server databases were getting +large (35-45GB when gzipped), and it would take several weeks to sync +from Genesis (and was sufficiently painful that no one seems to have +done it for about a year). This made me curious about improvements +and after taking a look at the code I decided to try a different +approach. I prefer Python3 over Python2, and the fact that Electrum is stuck on Python2 has been frustrating for a while. It's easier to change the -server to Python3 than the client. +server to Python3 than the client, so I decided to write my effort in +Python3. -It also seemed like a good way to learn about asyncio, which is a -wonderful and powerful feature of Python from 3.4 onwards. -Incidentally asyncio would also make a much better way to implement +It also seemed like a good opportunity to learn about asyncio, a +wonderful and powerful feature introduced in Python 3.4. +Incidentally, asyncio would also make a much better way to implement the Electrum client. Finally though no fan of most altcoins I wanted to write a codebase that could easily be reused for those alts that are reasonably compatible with Bitcoin. Such an abstraction is also useful for -testnets, of course. +testnets. Features ======== -- The full Electrum protocol is implemented with the exception of the - blockchain.address.get_proof RPC call, which is not used in normal - sessions and only sent from the Electrum command line. +- The full Electrum protocol is implemented. The only exception is + the blockchain.address.get_proof RPC call, which is not used by + Electrum GUI clients, and can only be invoked from the command line. - Efficient synchronization from Genesis. Recent hardware should synchronize in well under 24 hours, possibly much faster for recent CPUs or if you have an SSD. The fastest time to height 439k (mid - November 2016) reported is under 5 hours. Electrum-server would - probably take around 1 month. -- Subscription limiting both per-connection and across all connections. + November 2016) reported is under 5 hours. For comparison, JElectrum + would take around 4 days, and electrum-server probably around 1 + month, on the same hardware. +- Various configurable means of controlling resource consumption and + handling denial of service attacks. These include maximum + connection counts, subscription limits per-connection and across all + connections, maximum response size, per-session bandwidth limits, + and session timeouts. - Minimal resource usage once caught up and serving clients; tracking the - transaction mempool seems to take the most memory. -- Each client is served asynchronously to all other clients and tasks, - so busy clients do not reduce responsiveness of other clients' - requests and notifications, or the processing of incoming blocks. -- Daemon failover. More than one daemon can be specified; ElectrumX - will failover round-robin style if the current one fails for any - reason. -- Coin abstraction makes compatible altcoin support easy. - + transaction mempool appears to be the most expensive part. +- Fully asynchronous processing of new blocks, mempool updates, and + client requests. Busy clients should not noticeably impede other + clients' requests and notifications, nor the processing of incoming + blocks and mempool updates. +- Daemon failover. More than one daemon can be specified, and + ElectrumX will failover round-robin style if the current one fails + for any reason. +- Coin abstraction makes compatible altcoin and testnet support easy. Implementation ============== -ElectrumX does not do any pruning or throwing away of history. It -will retain this property for as long as feasible, and I believe it is +ElectrumX does not do any pruning or throwing away of history. I want +to retain this property for as long as it is feasible, and it appears efficiently achievable for the forseeable future with plain Python. -So how does it achieve a much more compact database than Electrum -server, which is forced to prune hisory for busy addresses, and yet -sync roughly 2 orders of magnitude faster? - -I believe all of the following play a part:: +The following all play a part in making ElectrumX very efficient as a +Python blockchain indexer: - aggressive caching and batching of DB writes - more compact and efficient representation of UTXOs, address index, @@ -105,17 +107,17 @@ I believe all of the following play a part:: eliminate CPU idling. As a Python program ElectrumX is unavoidably single-threaded in its essence; we must keep that CPU core busy. -Python's asyncio means ElectrumX has no (direct) use for threads and -associated complications. I cannot foresee any case where they might -be necessary. +Python's ``asyncio`` means ElectrumX has no (direct) use for threads +and associated complications. Roadmap Pre-1.0 =============== - minor code cleanups. -- there may be a DB format change to index the DB in a way purely +- there will be a DB format change to index the DB in a way purely dependent on the script and not on address prefix +- support bitcoin testnet with Satoshi bitcoind 0.13.1 - implement simple protocol to discover peers without resorting to IRC. This may slip to post 1.0 @@ -135,29 +137,179 @@ Roadmap Post-1.0 Database Format =============== -The database and metadata formats of ElectrumX are likely to change. -Such changes will render old DBs unusable. For now I do not intend to -provide converters as the time taken from genesis to synchronize to a -pristine database is quite tolerable. +The database and metadata format of ElectrumX is very likely to change +prior to 1.0 release. Existing DBs will not be unusable and you will +need to resync from Genesis, which is quite tolerable. -Miscellany -========== +ChangeLog +========= -As I've been researching where the time is going during block chain -indexing and how various cache sizes and hardware choices affect it, -I'd appreciate it if anyone trying to synchronize could tell me:: +Version 0.9.22 +-------------- - - the version of ElectrumX - - their O/S and filesystem - - their hardware (CPU name and speed, RAM, and disk kind) - - whether their daemon was on the same host or not - - whatever stats about sync height vs time they can provide (the - logs give it all in wall time) - - the network (e.g. bitcoin mainnet) they synced +* documentation updates (ARCHITECTURE.rst, ENVIRONMENT.rst) only. + +Version 0.9.21 +-------------- + +* moved RELEASE-NOTES into this README +* document the RPC interface in docs/RPC-INTERFACE.rst +* clean up open DB handling, issue `#89`_ + +Version 0.9.20 +-------------- + +* fix for IRC flood issue `#93`_ + +Version 0.9.19 +-------------- + +* move sleep outside semaphore (issue `#88`_) + +Version 0.9.18 +-------------- + +* last release of 2016. Just a couple of minor tweaks to logging. + +Version 0.9.17 +-------------- + +* have all the DBs use fsync on write; hopefully means DB won't corrupt in + case of a kernel panic (issue `#75`_) +* replace $DONATION_ADDRESS in banner file + +Version 0.9.16 +-------------- + +* logging improvements, including throttling of abusive logs +* permit large RPC requests (issue 85) + +Version 0.9.15 +-------------- + +* fix crash on reorg, issue #84 + +Version 0.9.14 +-------------- + +* don't start processing mempool until block processor has caught up. + Print server settings when servers start, not at startup. + +Version 0.9.13 +-------------- + +* fix to reduce verbosity of logging of deprioritised sessions. Sessions + are deprioritised if they are using high bandwidth, or if they are part + of a group using high bandwidth. Previously each delayed request scheduling + would be logged, now only changes in the delay (up or down) are logged. + +Version 0.9.12 +-------------- + +* enchancements to RPC and logging. getinfo output has changed, a couple + of fields renamed. + issue 77: add PID to getinfo + issue 78: start RPC immediately, don't wait for catch-up + issue 79: show IPv6 address-port combinations properly in [] + issue 80: show DB and daemon heights in getinfo + +Version 0.9.11 +-------------- + +* rework the fetch-and-process blocks loop. This regains some of the + sync efficiency we lost during 0.8.x and that was poorly hacked + around earlier in 0.9.x. Continuing to investigate where the rest + went. +* logging of block processing times fixes #58 +* moved the peer column to the end of the sessions RPC so that IPv6 addrs + don't mess up the formatting + +Version 0.9.10 +-------------- + +* logging improvements +* fixed issue #76 (RPCError namespace) + +Version 0.9.9 +------------- + +* prioritize mempool processing of sent txs. Closes issue 73. +* mempool tx processing needs to handle DBError exceptions. Fixes issue 74. + +Version 0.9.8 +------------- + +* cleanup up mempool handling, notify of addresses only once when a new block + comes in. Fixes issue 70. + +Version 0.9.7 +------------- + +* history and UTXO requests are now processed by the executor, i.e., + properly asynchronously. This was the last of the potential latency + bottlenecks. + +Version 0.9.6 +------------- + +* fix it properly this time + +Version 0.9.5 +------------- + +* fix issue introduced in 0.9.4 with paused connections + +Version 0.9.4 +------------- + +* new env var MAX_SESSIONS, see docs/ENV-NOTES. The default limit is + 1,000 sessions so raise this if you want to be able to take more. +* a couple of minor bug fixes relating to paused connections +* removed RPC calls numsessions and numpeers. They're not very interesting + and all that and more is in getinfo. + +Version 0.9.3 +------------- + +* unconfirmed flag indicating whether mempool txs have unconfirmed inputs + was inverted + +Version 0.9.2 +------------- + +* fix mempool busy waiting + +Version 0.9.1 +------------- + +* fix another couple of issues introduced in 0.9.0 + +Version 0.9.0a +-------------- + +* fix typo in 0.9.0 + +Version 0.9.0 +------------- + +* complete rewrite of mempool code to have minimal latency and fix a + couple of minor bugs. When a new block is found, ideally this + should be communicated to clients who addresses are affected with a + single notification. Previously this would happen with two + notifications: one because the TX got in the block, and one because + that TX was no longer in the mempool. Fundamentally this a race + condition that cannot be eliminated but its occurrence should be + minimized. -Neil Booth -kyuupichan@gmail.com -https://github.com/kyuupichan +**Neil Booth** kyuupichan@gmail.com https://github.com/kyuupichan + 1BWwXJH3q6PRsizBkSGm2Uw4Sz1urZ5sCj + + +.. _#75: https://github.com/kyuupichan/electrumx/issues/75 +.. _#88: https://github.com/kyuupichan/electrumx/issues/88 +.. _#89: https://github.com/kyuupichan/electrumx/issues/89 +.. _#93: https://github.com/kyuupichan/electrumx/issues/93 +.. _docs/HOWTO.rst: https://github.com/kyuupichan/electrumx/blob/master/docs/HOWTO.rst diff --git a/RELEASE-NOTES b/RELEASE-NOTES deleted file mode 100644 index c15e328..0000000 --- a/RELEASE-NOTES +++ /dev/null @@ -1,691 +0,0 @@ -version 0.9.20 --------------- - -- fix for IRC flood issue 93 - -NOTE: I will soon move the RELEASE-NOTES into the README - -version 0.9.19 --------------- - -- move sleep outside semaphore (issue 88) - -version 0.9.18 --------------- - -- last release of 2016. Just a couple of minor tweaks to logging. - -version 0.9.17 --------------- - -- have all the DBs use fsync on write; hopefully means DB won't corrupt in - case of a kernel panic (issue 75) -- replace $DONATION_ADDRESS in banner file - -version 0.9.16 --------------- - -- logging improvements, including throttling of abusive logs -- permit large RPC requests (issue 85) - -version 0.9.15 --------------- - -- fix crash on reorg, issue #84 - -version 0.9.14 --------------- - -- don't start processing mempool until block processor has caught up. - Print server settings when servers start, not at startup. - -version 0.9.13 --------------- - -- fix to reduce verbosity of logging of deprioritised sessions. Sessions - are deprioritised if they are using high bandwidth, or if they are part - of a group using high bandwidth. Previously each delayed request scheduling - would be logged, now only changes in the delay (up or down) are logged. - -version 0.9.12 --------------- - -- enchancements to RPC and logging. getinfo output has changed, a couple - of fields renamed. - issue 77: add PID to getinfo - issue 78: start RPC immediately, don't wait for catch-up - issue 79: show IPv6 address-port combinations properly in [] - issue 80: show DB and daemon heights in getinfo - -version 0.9.11 --------------- - -- rework the fetch-and-process blocks loop. This regains some of the - sync efficiency we lost during 0.8.x and that was poorly hacked - around earlier in 0.9.x. Continuing to investigate where the rest - went. -- logging of block processing times fixes #58 -- moved the peer column to the end of the sessions RPC so that IPv6 addrs - don't mess up the formatting - -** Please don't run version 0.10.0, it will corrupt your DB. - -version 0.9.10 --------------- - -- logging improvements -- fixed issue #76 (RPCError namespace) - -version 0.9.9 -------------- - -- prioritize mempool processing of sent txs. Closes issue 73. -- mempool tx processing needs to handle DBError exceptions. Fixes issue 74. - -version 0.9.8 -------------- - -- cleanup up mempool handling, notify of addresses only once when a new block - comes in. Fixes issue 70. - -version 0.9.7 -------------- - -- history and UTXO requests are now processed by the executor, i.e., - properly asynchronously. This was the last of the potential latency - bottlenecks. - -version 0.9.6 -------------- - -- fix it properly this time - -version 0.9.5 -------------- - -- fix issue introduced in 0.9.4 with paused connections - -version 0.9.4 -------------- - -- new env var MAX_SESSIONS, see docs/ENV-NOTES. The default limit is - 1,000 sessions so raise this if you want to be able to take more. -- a couple of minor bug fixes relating to paused connections -- removed RPC calls numsessions and numpeers. They're not very interesting - and all that and more is in getinfo. - -version 0.9.3 -------------- - -- unconfirmed flag indicating whether mempool txs have unconfirmed inputs - was inverted - -version 0.9.2 -------------- - -- fix mempool busy waiting - -version 0.9.1 -------------- - -- fix another couple of issues introduced in 0.9.0 - -version 0.9.0a --------------- - -- fix typo in 0.9.0 - -version 0.9.0 -------------- - -- complete rewrite of mempool code to have minimal latency and fix a - couple of minor bugs. When a new block is found, ideally this - should be communicated to clients who addresses are affected with a - single notification. Previously this would happen with two - notifications: one because the TX got in the block, and one because - that TX was no longer in the mempool. Fundamentally this a race - condition that cannot be eliminated but its occurrence should be - minimized. - -version 0.8.12 --------------- - -- pause serving sessions whose send buffers are full (anti-DoS). This - is currently logged; let me know if it's too verbose -- various tweaks to request handling - -version 0.8.11 --------------- - -- catch harmless socket exception -- show session count in groups RPC call - -version 0.8.10 --------------- - -- fix socket bug in 0.8.9 - -version 0.8.9 -------------- - -- RPC groups and sessions calls improved -- issues fixed: #62, #68 (slow socket closing, IRC) - -version 0.8.8 -------------- - -- put sessions in a priority queue to better prioritise serving. Low-bandwidth - sessions get served first -- new RPC command "groups" - shows information about session groups -- sessions output: session priority shown under Flags column; the lower the - number the higher the priority. txs column moved, new column reqs showns - the number of outstanding requests for that connection (includes subrequests - of batches) -- issued fixed: #67 - -version 0.8.7 -------------- - -- update systemd config (bauerj) -- temporary fix for initial sync times -- continued JSON code refactoring - -version 0.8.6 -------------- - -- fix JSON bugs from 0.8.5 -- fix issue #61 (valesi) - -version 0.8.5 -------------- - -- rework of JSON layer to better handle batch requests. This is - preparatory work for improved DoS resistance. - -I'm aware recent versions don't sync efficiently; please use 0.8.0 to sync -until I find time to fix it. - -version 0.8.4 -------------- - -- remove invalidated histories from cache on new block - -version 0.8.3 -------------- - -Minor tweaks to session logs: - -- sessions output now shows flags. All sessions are listed. The - session type column is gone, instead the first letter of RPC, SSL or - TCP is the first flag letter. A 'C' flag indicates the session is closing. - An 'L' that it's being logged. -- don't attempt to forcefully stale sockets; they remain in C state until - Python closes them (which can be a long time for some SSL sockets) -- don't consider all seeing eye connections as stale - -version 0.8.2 -------------- - -- process new blocks in the asyncio executor; essentially a python thread. - This should eliminate latency during block processing that caused sessions - to be dropped. -- bandwith limit is restored incrementally to a session over the hour - rather than in a lump when one hour has passed. Also, only the - limit is refunded each hour; previously the entire usage would be - refunded. So if the session uses 30MB bandwidth and your limit is - 10MB, it will take 3 hrs before the session is considered to have - used none of its allotted bandwidth; previously it would happen after 1 - hour. - -version 0.8.1 -------------- - -** NOTE: this version has a new Python package dependency: pylru - -- fix for IRC encoding killing IRC connection -- add lru cache for history - -version 0.8.0 ------------- - -- stale connections are periodically closed. See docs/ENV-NOTES for - SESSION_TIMEOUT, default is 10 minutes. Issue #56. -- each session gets its own ID which is used in the logs instead of its - network address; the network address is still shown on initial connection. - Issue #55. -- the session ID is also shown in the sessions list. You can use this ID - with the following new RPC commands which take a list of session ids: - - electrumx_rpc.py log - electrumx_rpc.py disconnect - - The first will toggle logging of the sessions. A logged sesssion - prints every incoming request to the logs. - The second will disconnect the sessions. - Example: "electrumx_rpc.py log 15 369" - -version 0.7.20 --------------- - -- fix for errors during batch requests (issue #54) -- don't log errors on shutdown after giving sockets time to close - -version 0.7.19 --------------- - -- revert mempool change of 0.7.18 - -version 0.7.18 --------------- - -- better IRC support for tor (valesi) -- issues: suppressed some uninteresting socket logging to fix #52 -- mempool: fixed small memory leak - -version 0.7.17 --------------- - -- upped read buffer limit to 1,000,000 bytes. - -version 0.7.16 --------------- - -- fix bug introduced in 0.7.12 that hit during reorgs - -version 0.7.15 --------------- - -The following meta variables in your banner file are now replaced in -addition to $VERSION described in the notes to 0.7.11. If you type -getnetworkinfo in your daemon's debug console you will see what they -are based on: - -- $DAEMON_VERSION is replaced with the daemon's version as a dot-separated - string. For example "0.12.1". -- $DAEMON_SUBVERSION is replaced with the daemon's user agent string. - For example, "/BitcoinUnlimited:0.12.1(EB16; AD4)/". - -version 0.7.14 --------------- - -Improved DoS protection: - -- incoming network request buffers - which hold incomplete requests - are limited to 150,000 bytes, which I believe is large for genuine - clients. I don't foresee a need to change this so it is hard-coded. - If an incoming request (for example, text without a newline) exceeds - this limit the connection is dropped and the event logged. -- RPC connections have high MAX_SEND and incoming buffer limits as these - connections are assumed to be trusted. -- new environment variable BANDWIDTH_LIMIT. See docs/ENV-NOTES. -- fixes: LOG_SESSIONS of 0.7.13 wasn't being properly interpreted. - Tweak to rocksdb close() that should permit db reopening to work. - -version 0.7.13 --------------- - -- the output of the RPC sessions and getinfo calls are now written to logs - periodically by default. See LOG_SESSIONS in docs/ENV-NOTES -- Litecoin update (santzi) - -version 0.7.12 --------------- - -- minor bug fixes: 2 in JSON RPC, 1 in get_utxos (affected addresslistunspent) -- leveldb / rocksdb are opened with a different maximum open files limit, - depending on whether the chain has been fully synced or not. If synced - you want the files for network sockets, if not synced for the DB engines. - Once synced the DB will be reopened with the lower limit to free up the - files for serving network connections -- various refactoring preparing for possible asynchronous block processing - -version 0.7.11 --------------- - -- increased MAX_SEND default value to 1 million bytes so as to be able - to serve large historical transactions of up to ~500K in size. The - MAX_SEND floor remains at 350,000 bytes so you can reduce it if you - wish. To serve any historical transaction for bitcoin youd should - set this to around 2,000,100 bytes (one byte becomes 2 ASCII hex chars) -- issue #46: fix reorgs for coinbase-only blocks. We would not distinguish - undo information being empty from it not existing -- issue #47: IRC reconnects. I don't get this issue so cannot be certain - it is resolved -- $VERSION in your banner file is now replaced with your ElectrumX version -- more work to isolate the DB from the block processor, working towards the - goal of asynchronous block updates - -version 0.7.10 --------------- - -- replaced MAX_HIST environment variable with MAX_SEND, see docs/ENV-NOTES. - Large requests are blocked and logged. The logs should help you determine - if the requests are genuine (perhaps requiring a higher MAX_SEND) or abuse. - -version 0.7.9 -------------- - -- rewrite jsonrpc.py to also operate as a client. Use this class - for a robust electrumx_rpc.py. Fixes issue #43 - -version 0.7.8 -------------- - -- hopefully fix failed assertion on reorgs, issue #44 - -version 0.7.7 -------------- - -- add MAX_HIST to throttle history requests; see docs/ENV-NOTES. One - provider of ElectrumX services was attacked by a loser requesting - long histories; this environment variable allows you to limit what - you attempt to serve. - -version 0.7.6 -------------- - -- Fix IRC regression of 0.7.5 - would always connect to IRC by default - -version 0.7.5 -------------- - -- refactoring of server manager and event handling. One side effect - is to fix a bug in 0.7.4 where after a reorg ElectrumX might create - a second mempool and/or kick off more servers. Your testing would - be appreciated. This is part of the refactoring necessary to - process incoming blocks asynchronously so client connections are not - left waiting for several seconds -- close connections on first bad JSON encoding. Do not process buffered - requests of a closing connection -- make IRC params a function of the coin (TheLazier) and supply them for - Dash - -version 0.7.4 -------------- - -- really fix reorgs, they still triggered an assertion. If you hit a reorg - I believe your DB is fine and all you need to do is restart with updated - software -- introduced a new debug env var FORCE_REORG which I used to simulate a - reorg and confirm they should work - -version 0.7.3 -------------- - -- fix reorgs - broken since 0.6 I think - -version 0.7.2 -------------- - -- don't log message decoding errors. Cut off a connection after it has sent - 10 ill-formed requests. -- doc improvements (cluelessperson) -- RPC ports for Dash (TheLazier) - -version 0.7.1 -------------- - -- fixes an unqualified use of RPCError - -version 0.7 ------------ - -- daemon failover is now supported; see docs/ENV-NOTES. As a result, - DAEMON_URL must now be supplied and DAEMON_USERNAME, DAEMON_PASSWORD, - DAEMON_HOST and DAEMON_PORT are no longer used. -- fixed a bug introduced in 0.6 series where some client header requests - would fail -- fully asynchronous mempool handling; blocks can be processed and clients - notified whilst the mempool is still being processed - -version 0.6.3 -------------- - -- new environment variables MAX_SUBS and MAX_SESSION_SUBS. Please read - docs/ENV-NOTES - I encourage you to raise the default values. -- fixed import bug in 0.6.2 that prevented initial sync -- issues closed: #30. Logs should be clean on shutdown now. - -version 0.6.2 -------------- - -- handle daemon errors properly that result from client requests; pass the - error onto the client -- start serving immediatley on catchup; don't wait for the mempool -- logging improvements, in particular logging software and DB versions -- issues closed: #29, #31, #32 - -version 0.6.1 -------------- - -- main focus was better logging - more concise and informative, particularly - when caught up -- issues closed: #26, #27 -- default reorg limit is now taken from the coin, with a high default for - bitcoin testnet - -version 0.6.0 -------------- - -- DB format has changed again. This doesn't give a performance gain - or reduction that I could measure, but is cleaner in that each table - entry is now a singleton and not an array, which I much prefer as a - cleaner solution. It may enable other goodness in the future. -- Logging is much less noisy when serving clients. In fact anything - in your logs that isn't just status updates probably is a bug that I - would like to know about. Unfortunately clean shutdown whilst - serving clients leads to massive log spew. This is harmless and I - believe because of my noob status with asyncio. I intend to fix - this in a nearby release. -- expensive client requests are intended to yield to other requests - sufficiently frequently that there should be no noticeable delays or - pauses under normal load from hog clients. -- Notifications to hog clients are now queued in sequence with their - request responses. They used to be sent immediately regardless of - pending requests which seems less than ideal. -- some trivial improvements and fixes to local RPC query output - -version 0.5.1 -------------- - -- 0.5 changed some cache defaults, only partially intentionally. For - some users, including me, the result was a regression (a 15hr HDD - sync became a 20hr sync). Another user reported their fastest sync - yet (sub 10hr SSD sync). What changed was memory accounting - all - releases until 0.5 were not properly accounting for memory usage of - unflushed transaction hashes. In 0.5 they were accounted for in the - UTXO cache, which resulted in much earlier flushes. 0.5.1 flushes - the hashes at the same time as history so I now account for it - towards the history cache limit. To get a reasonable comparison - with prior releases your HIST_MB environment variable should be - bumped by about 15% from 0.4 and earlier values. This will not - result in greater memory consumption - the additional memory - consumption was being ignored before but is now being included. -- 0.5.1 is the first release where Electrum client requests are queued - on a per-session basis. Previously they were in a global queue. - This is the beginning of ensuring that expensive / DOS requests - mostly affect that user's session and not those of other users. The - goal is that each session's requests run asynchronously parallel to - every other sessions's requests. The missing part of the puzzle is - that Python's asyncio is co-operative, however at the moment - ElectrumX does not yield during expensive requests. I intend that a - near upcoming release will ensure expensive requests yield the CPU - at regular fine-grained intervals. The intended result is that, to - some extent, expensive requests mainly delay that and later requests - from the same session, and have minimal impact on the legitimate - requests of other sessions. The extent to which this goal is - achieved will only be verifiable in practice. -- more robust tracking and handling of asynchronous tasks. I hope - this will reduce asyncio's logging messages, some of which I'm - becoming increasingly convinced I have no control over. In - particular I learned earlier releases were unintentionally limiting - the universe of acceptable SSL protocols, and so I made them the - default that had been intended. -- I added logging of expensive tasks, though I don't expect much real - information from this -- various RPC improvements - -version 0.5 ------------ - -- DB change: all UTXOs, including those that are not canonically paying to - an address, are stored in the DB. So an attempt to spend a UTXO not in - the DB means corruption. DB version bumped to 2; older versions will not - work -- fixed issue #17: the genesis coinbase is not in the UTXO set - -version 0.4.3 -------------- - -- fix exception introduced in 0.4.2 - -version 0.4.2 -------------- - -- split out JSON RPC protcol handling. Now more robust and we should - fully support JSON RPC 2.0 clients, including batch requests - (Electrum client does not yet support these) -- refactored and cleaned up server handling -- improved DASH support (thelazier) - -version 0.4.1 -------------- - -- tweak IRC version reporting so we appear in the Electrum client's - network dialog box - -version 0.4 ------------ - -- IRC connectivity. See the notes for environment variables, etc. -- logging improvements - -Version 0.3.2, 0.3.3 --------------------- - -- fixed silly bugs - -Version 0.3.1 -------------- - -- fixes issue #9 -- save DB version in DB; warn on DB open if incompatible format - -Version 0.3 ------------ - -- Database format has changed; old DBs are incompatible. They will - not work and will probably die miserably as I'm not yet versioning - them for helpful warnings (coming soon). -- The change in on-disk format makes UTXO flushes noticeably more - efficient. My gut feeling is it probably benefits HDDs more than - SSDs, but I have no numbers to back that up other than that my HDD - synced about 90 minutes (10%) faster. Until the treacle hits at - blocks 300k+ there will probably be little noticeable difference in - sync time. - -Version 0.2.3 -------------- - -- fixes issues #6, #11, #15 -- the UTXO cache is now merged with BlockProcessor, where it properly belongs. - cache.py no longer exists - -Version 0.2.2.1 ---------------- - -- fixes issues #12, #13 -- only attempt to flush on asyncio.CancelledError to avoid spurious - secondary errors - -Version 0.2.2 -------------- - -- mostly refactoring: controller.py is gone; cache.py is half-gone. - Split BlockProcessor into 3: DB, BlockProcessor and BlockServer. DB - handles stored DB and FS state; BlockProcessor handles pushing the - chain forward and caching of updates, and BlockServer will - additionally serve clients on catchup. More to come. -- mempool: better logging; also yields during initial seeding -- issues fixed: #10 - -Version 0.2.1 -------------- - -- fix rocksdb and lmdb abstractions (bauerj) -- limit concurrent daemon requests -- improve script + coin abstractions -- faster tx and script parsing -- minor bug fixes - -Version 0.2 ------------ - -- update sample run script, remove empty addresses from mempool - -Version 0.1 ------------- - -- added setup.py, experimental. Because of this server_main.py renamed - electrumx_server.py, and SERVER_MAIN environment variable was renamed - to ELECTRUMX. The sample run script was updated to match. -- improvements to logging of daemon connection issues -- removal of old reorg test code -- hopefully more accurate sync ETA - -Version 0.07 ------------- - -- fixed a bug introduced in 0.06 at the last minute - -Version 0.06 ------------- - -- mempool support. ElectrumX maintains a representation of the daemon's - mempool and serves unconfirmed transactions and balances to clients. - -Version 0.05 ------------- - -- fixed a bug in 0.04 that stopped ElectrumX serving once synced - -Version 0.04 ------------- - -- made the DB interface a little faster for LevelDB and RocksDB; this was - a small regression in 0.03 -- fixed a bug that prevented block reorgs from working -- implement and enable client connectivity. This is not yet ready for - public use for several reasons. Local RPC, and remote TCP and SSL - connections are all supported in the same way as Electrum-server. - ElectrumX does not begin listening for incoming connections until it - has caught up with the daemon's height. Which ports it is listening - on will appear in the logs when it starts listening. The complete - Electrum wire protocol is implemented, so it is possible to now use - as a server for your own Electrum client. Note that mempools are - not yet handled so unconfirmed transactions will not be notified or - appear; they will appear once they get in a block. Also no - responses are cached, so performance would likely degrade if used by - many clients. I welcome feedback on your experience using this. - - -Version 0.03 ------------- - -- merged bauerj's abstracted DB engine contribution to make it easy to - play with different backends. In addition to LevelDB this adds - support for RocksDB and LMDB. We're interested in your comparitive - performance experiences. - - -Version 0.02 ------------- - -- fix bug where tx counts were incorrectly saved -- large clean-up and refactoring of code, breakout into new files -- several efficiency improvements -- initial implementation of chain reorg handling -- work on RPC and TCP server functionality. Code committed but not - functional, so currently disabled -- note that some of the enivronment variables have been renamed, - see samples/scripts/NOTES for the list \ No newline at end of file diff --git a/docs/ARCHITECTURE.rst b/docs/ARCHITECTURE.rst index 81d9800..7c968fa 100644 --- a/docs/ARCHITECTURE.rst +++ b/docs/ARCHITECTURE.rst @@ -1,85 +1,85 @@ Components ========== -The components of the server are roughly like this:: - - ------- - - Env - - ------- - - ------- - - IRC - - ------- - < - ------------- ------------ - - ElectrumX -<<<<<- LocalRPC - - ------------- ------------ - < > - ---------- ------------------- - - Daemon -<<<<<<<<- Block processor - - ---------- ------------------- - < < > - -------------- ----------- - - Prefetcher - - FS + DB - - -------------- ----------- - +.. image:: https://docs.google.com/drawings/d/1Su_DR2c8__-4phm12hAzV65fL2tNm_1IhKr4XivkW6Q/pub?w=720&h=540 + :target: https://docs.google.com/drawings/d/1Su_DR2c8__-4phm12hAzV65fL2tNm_1IhKr4XivkW6Q/pub?w=960&h=720 Env --- -Holds configuration taken from the environment. Handles defaults -appropriately. Generally passed to the constructor of other -components which take their settings from it. +Holds configuration taken from the environment, with apprioriate +defaulting appropriately. Generally passed to the constructor of +other components which take their settings from it. + +Controller +---------- + +The central part of the server process initialising and coordinating +all the others. Manages resource usage. LocalRPC -------- Handles local JSON RPC connections querying ElectrumX server state. -Not started until the block processor has caught up with the daemon. +Started when the ElectrumX process starts. ElectrumX --------- Handles JSON Electrum client connections over TCP or SSL. One instance per client session. Should be the only component concerned -with the details of the Electrum wire protocol. Responsible for -caching of client responses. Not started until the block processor -has caught up with the daemon. Logically, if not yet in practice, a -coin-specific class. +with the details of the Electrum wire protocol. + +Not started until the block processor has caught up with the daemon. Daemon ------ -Used by the block processor, ElectrumX servers and prefetcher. -Encapsulates daemon RPC wire protcol. Logically, if not yet in -practice, a coin-specific class. +Encapsulates the RPC wire protcol with bitcoind for the whole server. +Transparently handles temporary daemon errors, and fails over if +necessary. + Block Processor --------------- Responsible for managing block chain state (UTXO set, history, -transaction and undo information) and processing towards the chain -tip. Uses the caches for in-memory state updates since the last -flush. Flushes state to the storage layer. Reponsible for handling -block chain reorganisations. Once caught up maintains a -representation of daemon mempool state. +transaction and undo information) and for handling block chain +reorganisations. -Database --------- +When caught up, processes new blocks as they are found, and flushes +the updates to the database immediately. -The database. Along with the host filesystem stores flushed chain state. +When syncing uses caches for in-memory state updates since the prior +flush. Occasionally flushes state to the storage layer when caches +get large. Prefetcher ---------- -Used by the block processor to asynchronously prefetch blocks from the -daemon. Holds fetched block height. Once it has caught up -additionally obtains daemon mempool tx hashes. Serves blocks and -mempool hashes to the block processor via a queue. +Cooperates with the block processor to asynchronously prefetch blocks +from bitcoind. Once it has caught up it additionally requests mempool +transaction hashes from bitcoind. Serves blocks to the block +processor via a queue, and the mempool hashes to the Mempool object. + +Mempool +------- + +Handles all the details of maintaining a representation of bitcoind's +mempool state. Obtains the list of current mempool transaction hashes +from the Daemon when notified by the Prefetcher. + +Notifies the controller that addresses have been touched when the +mempool refreshes (or implicitly when a new block is found). + +Database +-------- + +The database. Flushed chain state is stored in the DB backend, such +as leveldb, along with metadata on the host filesystem. IRC --- -Not currently imlpemented; will handle IRC communication for the -ElectrumX servers. +Handles advertising of ElectrumX services via IRC. diff --git a/docs/ENV-NOTES b/docs/ENV-NOTES deleted file mode 100644 index 840397f..0000000 --- a/docs/ENV-NOTES +++ /dev/null @@ -1,143 +0,0 @@ -The following environment variables are required: - -DB_DIRECTORY - path to the database directory (if relative, to `run` script) -USERNAME - the username the server will run as, if using `run` script -ELECTRUMX - path to the electrumx_server.py script (if relative, - to `run` script) -DAEMON_URL - A comma-separated list of daemon URLS. If more than one is - provided ElectrumX will failover to the next when one stops - working. The generic form is: - http://username:password@hostname:port/ - The leading 'http://' is optional, as is the trailing - slash. The ':port' part is also optional and will default - to the standard RPC port for COIN if omitted. - -The other environment variables are all optional and will adopt -sensible defaults if not specified. - -COIN - see lib/coins.py, must be a coin NAME. Defaults to Bitcoin. -NETWORK - see lib/coins.py, must be a coin NET. Defaults to mainnet. -DB_ENGINE - database engine for the transaction database. Default is - leveldb. Supported alternatives are rocksdb and lmdb. - You will need to install the appropriate python packages. - Not case sensitive. -REORG_LIMIT - maximum number of blocks to be able to handle in a chain - reorganisation. ElectrumX retains some fairly compact - undo information for this many blocks in levelDB. - Default is 200. -HOST - the host that the TCP and SSL servers will use. Defaults to - localhost. Set to blank to listen on all addresses (IPv4 - and IPv6). -TCP_PORT - if set will serve Electrum TCP clients on that HOST:TCP_PORT -SSL_PORT - if set will serve Electrum SSL clients on that HOST:SSL_PORT - If set, SSL_CERTFILE and SSL_KEYFILE must be filesystem paths. -RPC_PORT - Listen on this port for local RPC connections, defaults to - 8000. -BANNER_FILE - a path to a banner file to serve to clients. The banner file - is re-read for each new client. The string $VERSION in your - banner file will be replaced with the ElectrumX version you - are runnning, such as 'ElectrumX 0.7.11'. -LOG_SESSIONS - the number of seconds between printing session statistics to - the log. Defaults to 3600. Set to zero to suppress this - logging. -ANON_LOGS - set to anything non-empty to remove IP addresses from - logs. By default IP addresses will be logged. -DONATION_ADDRESS - server donation address. Defaults to none. - -These following environment variables are to help limit server -resource consumption and to prevent simple DoS. Address subscriptions -in ElectrumX are very cheap - they consume about 100 bytes of memory -each and are processed efficiently. I feel the defaults are low and -encourage you to raise them. - -MAX_SESSIONS - maximum number of sessions. Once reached, TCP and SSL - listening sockets are closed until the session count drops - naturally to 95% of the limit. Defaults to 1,000. -MAX_SEND - maximum size of a response message to send over the wire, - in bytes. Defaults to 1,000,000 and will treat values - smaller than 350,000 as 350,000 because standard Electrum - protocol header chunk requests are almost that large. - The Electrum protocol has a flaw in that address - histories must be served all at once or not at all, - an obvious avenue for abuse. MAX_SEND is a - stop-gap until the protocol is improved to admit - incremental history requests. Each history entry - is appoximately 100 bytes so the default is - equivalent to a history limit of around 10,000 - entries, which should be ample for most legitimate - users. If you increase this bear in mind one client - can request history for multiple addresses. Also, - the largest raw transaction you will be able to serve - to a client is just under half of MAX_SEND, as each raw - byte becomes 2 hexadecimal ASCII characters on the wire. -MAX_SUBS - maximum number of address subscriptions across all - sessions. Defaults to 250,000. -MAX_SESSION_SUBS - maximum number of address subscriptions permitted to a - single session. Defaults to 50,000. -BANDWIDTH_LIMIT - per-session periodic bandwith usage limit in bytes. - Bandwidth usage over each period is totalled, and - when this limit is exceeded each subsequent request - is stalled by sleeping before handling it, - effectively yielding processing resources to other - sessions. Each time this happens the event is - logged. The more bandwidth usage exceeds the limit - the longer the next request will sleep. Each sleep - is a round number of seconds with a minimum of one. - The bandwith usage counter is reset to zero at the - end of each period. Currently the period is - hard-coded to be one hour. The default limit value - is 2 million bytes. -SESSION_TIMEOUT - an integer number of seconds defaulting to 600. - Sessions with no activity for longer than this are - disconnected. - -If you want IRC connectivity to advertise your node: - -IRC - set to anything non-empty -IRC_NICK - the nick to use when connecting to IRC. The default is a - hash of REPORT_HOST. Either way 'E_' will be prepended. -REPORT_HOST - the host to advertise. Defaults to HOST. -REPORT_TCP_PORT - the TCP port to advertise. Defaults to TCP_PORT. - '0' disables publishing the port. -REPORT_SSL_PORT - the SSL port to advertise. Defaults to SSL_PORT. - '0' disables publishing the port. -REPORT_HOST_TOR - Tor .onion address to advertise. Appends '_tor" to nick. -REPORT_TCP_PORT_TOR - the TCP port to advertise for Tor. Defaults to - REPORT_TCP_PORT, unless it is '0', then use TCP_PORT. - '0' disables publishing the port. -REPORT_SSL_PORT_TOR - the SSL port to advertise for Tor. Defaults to - REPORT_SSL_PORT, unless it is '0', then use SSL_PORT. - '0' disables publishing the port. - -If synchronizing from the Genesis block your performance might change -by tweaking the following cache variables. Cache size is only checked -roughly every minute, so the caches can grow beyond the specified -size. Also the Python process is often quite a bit fatter than the -combined cache size, because of Python overhead and also because -leveldb consumes a lot of memory during UTXO flushing. So I recommend -you set the sum of these to nothing over half your available physical -RAM: - -HIST_MB - amount of history cache, in MB, to retain before flushing to - disk. Default is 300; probably no benefit being much larger - as history is append-only and not searched. - -UTXO_MB - amount of UTXO and history cache, in MB, to retain before - flushing to disk. Default is 1000. This may be too large - for small boxes or too small for machines with lots of RAM. - Larger caches generally perform better as there is - significant searching of the UTXO cache during indexing. - However, I don't see much benefit in my tests pushing this - too high, and in fact performance begins to fall. My - machine has 24GB RAM; the slow down is probably because of - leveldb caching and Python GC effects. However this may be - very dependent on hardware and you may have different - results. - -The following are for debugging purposes. - -FORCE_REORG - if set to a positive integer, it will simulate a reorg - of the blockchain for that number of blocks on startup. - Although it should fail gracefully if set to a value - greater than REORG_LIMIT, I do not recommend it as I have - not tried it and there is a chance your DB might corrupt. diff --git a/docs/ENVIRONMENT.rst b/docs/ENVIRONMENT.rst new file mode 100644 index 0000000..b5b73b3 --- /dev/null +++ b/docs/ENVIRONMENT.rst @@ -0,0 +1,304 @@ +===================== +Environment Variables +===================== + +ElectrumX takes no command line arguments, instead its behaviour is +controlled by environment variables. Only a few are required to be +given, the rest will have sensible defaults if not specified. Many of +the defaults around resource usage are conservative; I encourage you +to review them. + +Required +-------- + +These environment variables are always required: + +* **DB_DIRECTORY** + + The path to the database directory. Relative paths should be + relative to the parent process working directory. This is the + directory of the `run` script if you use it. + +* **DAEMON_URL** + + A comma-separated list of daemon URLs. If more than one is provided + ElectrumX will initially connect to the first, and failover to + subsequent ones round-robin style if one stops working. + + The generic form of a daemon URL is: + + `http://username:password@hostname:port/` + + The leading `http://` is optional, as is the trailing slash. The + `:port` part is also optional and will default to the standard RPC + port for **COIN** and **NETWORK** if omitted. + + +For the `run` script +-------------------- + +The following are required if you use the `run` script: + +* **ELECTRUMX** + + The path to the electrumx_server.py script. Relative paths should + be relative to the directory of the `run` script. + +* **USERNAME** + + The username the server will run as. + +Miscellaneous +------------- + +These environment variables are optional: + +* **COIN** + + Must be a *NAME* from one of the **Coin** classes in + `lib/coins.py`_. Defaults to `Bitcoin`. + +* **NETWORK** + + Must be a *NET* from one of the **Coin** classes in `lib/coins.py`_. + Defaults to `mainnet`. + +* **DB_ENGINE** + + Database engine for the transaction database. The default is + `leveldb`. Supported alternatives are `rocksdb` and `lmdb`. You + will need to install the appropriate python package for your engine. + The value is not case sensitive. Note that the current way + ElectrumX uses LMDB gives poor performance . + +* **REORG_LIMIT** + + The maximum number of blocks to be able to handle in a chain + reorganisation. ElectrumX retains some fairly compact undo + information for this many blocks in levelDB. The default is a + function of **COIN** and **NETWORK**; for Bitcoin mainnet it is 200. + +* **HOST** + + The host that the TCP and SSL servers will use. Defaults to + `localhost`. Set to blank to listen on all addresses (IPv4 and IPv6). + +* **TCP_PORT** + + If set ElectrumX will serve TCP clients on **HOST**:**TCP_PORT**. + +* **SSL_PORT** + + If set ElectrumX will serve SSL clients on **HOST**:**SSL_PORT**. + If set SSL_CERTFILE and SSL_KEYFILE must be defined and be + filesystem paths to those SSL files. + +* **RPC_PORT** + + ElectrumX will listen on this port for local RPC connections. + ElectrumX listens for RPC connections unless this is explicity set + to blank. It defaults appropriately for **COIN** and **NETWORK** + (e.g., 8000 for Bitcoin mainnet) if not set. + +* **DONATION_ADDRESS** + + The server donation address reported to Electrum clients. Defaults + to empty, which Electrum interprets as meaning there is none. + +* **BANNER_FILE** + + The path to a banner file to serve to clients. Relative file paths + must be relative to **DB_DIRECTORY**. The banner file is re-read + for each new client. + + You can place several meta-variables in your banner file, which will be + replaced before serving to a client. + + + **$VERSION** is replaced with the ElectrumX version you are + runnning, such as *ElectrumX 0.9.22*. + + **$DAEMON_VERSION** is replaced with the daemon's version as a + dot-separated string. For example *0.12.1*. + + **$DAEMON_SUBVERSION** is replaced with the daemon's user agent + string. For example, `/BitcoinUnlimited:0.12.1(EB16; AD4)/`. + + **$DONATION_ADDRESS** is replaced with the address from the + **DONATION_ADDRESS** environment variable. + +* **ANON_LOGS** + + Set to anything non-empty to replace IP addresses in logs with + redacted text like 'xx.xx.xx.xx:xxx'. By default IP addresses will + be written to logs. + +* **LOG_SESSIONS** + + The number of seconds between printing session statistics to the + log. The output is identical to the **sessions** RPC command except + that **ANON_LOGS** is honoured. Defaults to 3600. Set to zero to + suppress this logging. + +Resource Usage Limits +--------------------- + +The following environment variables are all optional and help to limit +server resource consumption and prevent simple DoS. + +Address subscriptions in ElectrumX are very cheap - they consume about +100 bytes of memory each (160 bytes from version 0.10.0) and are +processed efficiently. I feel the two subscription-related defaults +below are low and encourage you to raise them. + +* **MAX_SESSIONS** + + The maximum number of incoming connections. Once reached, TCP and + SSL listening sockets are closed until the session count drops + naturally to 95% of the limit. Defaults to 1,000. + +* **MAX_SEND** + + The maximum size of a response message to send over the wire, in + bytes. Defaults to 1,000,000. Values smaller than 350,000 are + taken as 350,000 because standard Electrum protocol header "chunk" + requests are almost that large. + + The Electrum protocol has a flaw in that address histories must be + served all at once or not at all, an obvious avenue for abuse. + **MAX_SEND** is a stop-gap until the protocol is improved to admit + incremental history requests. Each history entry is appoximately + 100 bytes so the default is equivalent to a history limit of around + 10,000 entries, which should be ample for most legitimate users. If + you use a higher default bear in mind one client can request history + for multiple addresses. Also note that the largest raw transaction + you will be able to serve to a client is just under half of + MAX_SEND, as each raw byte becomes 2 hexadecimal ASCII characters on + the wire. Very few transactions on Bitcoin mainnet are over 500KB + in size. + +* **MAX_SUBS** + + The maximum number of address subscriptions across all sessions. + Defaults to 250,000. + +* **MAX_SESSION_SUBS** + + The maximum number of address subscriptions permitted to a single + session. Defaults to 50,000. + +* **BANDWIDTH_LIMIT** + + Per-session periodic bandwith usage limit in bytes. This is a soft, + not hard, limit. Currently the period is hard-coded to be one hour. + The default limit value is 2 million bytes. + + Bandwidth usage over each period is totalled, and when this limit is + exceeded each subsequent request is stalled by sleeping before + handling it, effectively giving higher processing priority to other + sessions. Each time this happens the event is logged. + + The more bandwidth usage exceeds this soft limit the longer the next + request will sleep. Sleep times are a round number of seconds with + a minimum of 1. + + Bandwidth usage is gradually reduced over time by "refunding" a + proportional part of the limit every now and then. + +* **SESSION_TIMEOUT** + + An integer number of seconds defaulting to 600. Sessions with no + activity for longer than this are disconnected. Properly + functioning Electrum clients by default will send pings roughly + every 60 seconds. + +IRC +--- + +Use the following environment variables if you want to advertise +connectivity on IRC: + +* **IRC** + + Set to anything non-empty to advertise on IRC + +* **IRC_NICK** + + The nick to use when connecting to IRC. The default is a hash of + **REPORT_HOST**. Either way a prefix will be prepended depending on + **COIN** and **NETWORK**. + +* **REPORT_HOST** + + The host to advertise. Defaults to **HOST**. + +* **REPORT_TCP_PORT** + + The TCP port to advertise. Defaults to **TCP_PORT**. '0' disables + publishing the port. + +* **REPORT_SSL_PORT** + + The SSL port to advertise. Defaults to **SSL_PORT**. '0' disables + publishing the port. + +* **REPORT_HOST_TOR** + + The tor .onion address to advertise. If set, an additional + connection to IRC happens with '_tor" appended to **IRC_NICK**. + +* **REPORT_TCP_PORT_TOR** + + The TCP port to advertise for Tor. Defaults to **REPORT_TCP_PORT**, + unless it is '0', otherwise **TCP_PORT**. '0' disables publishing + the port. + +* **REPORT_SSL_PORT_TOR** + + The SSL port to advertise for Tor. Defaults to **REPORT_SSL_PORT**, + unless it is '0', otherwise **SSL_PORT**. '0' disables publishing + the port. + +Cache +----- + +If synchronizing from the Genesis block your performance might change +by tweaking the following cache variables. Cache size is only checked +roughly every minute, so the caches can grow beyond the specified +size. Also the Python process is often quite a bit fatter than the +combined cache size, because of Python overhead and also because +leveldb consumes a lot of memory during UTXO flushing. So I recommend +you set the sum of these to nothing over half your available physical +RAM: + +* **HIST_MB** + + The amount of history cache, in MB, to retain before flushing to + disk. Default is 300; probably no benefit being much larger as + history is append-only and not searched. + + I do not recommend setting this above 500. + +* **UTXO_MB** + + The amount of UTXO and history cache, in MB, to retain before + flushing to disk. Default is 1000. This may be too large for small + boxes or too small for machines with lots of RAM. Larger caches + generally perform better as there is significant searching of the + UTXO cache during indexing. However, I don't see much benefit in my + tests pushing this too high, and in fact performance begins to fall. + My machine has 24GB RAM; the slow down is probably because of + leveldb caching and Python GC effects. + + I do not recommend setting this above 2000. + +Debugging +--------- + +The following are for debugging purposes: + +* **FORCE_REORG** + + If set to a positive integer, it will simulate a reorg of the + blockchain for that number of blocks on startup. Although it should + fail gracefully if set to a value greater than **REORG_LIMIT**, I do + not recommend it as I have not tried it and there is a chance your + DB might corrupt. + +.. _lib/coins.py: https://github.com/kyuupichan/electrumx/blob/master/lib/coins.py diff --git a/docs/RPC-INTERFACE.rst b/docs/RPC-INTERFACE.rst new file mode 100644 index 0000000..195fbf0 --- /dev/null +++ b/docs/RPC-INTERFACE.rst @@ -0,0 +1,144 @@ +The ElectrumX RPC Interface +=========================== + +You can query the status of a running server, and affect its behaviour +using the RPC interface. + +The general form of invocation is: + + ``electrumx_rpc.py [arg1 [arg2...]`` + +The following commands are available: + +* **getinfo** + + Returns a summary of server state. This command takes no arguments. + A typical result is as follows (with annotated comments): + + .. code:: + + $ electrumx_rpc.py getinfo + { + "closing": 1, # The number of sessions being closed down + "daemon_height": 446231, # The daemon's height when last queried + "db_height": 446231, # The height to which the DB is processed + "errors": 1, # Errors across current sessions + "groups": 2, # The number of session groups + "logged": 0, # The number of sessions being logged + "paused": 0, # The number of paused sessions. + "peers": 62, # Number of peer servers (from IRC) + "pid": 126275, # The server's process ID + "requests": 0, # Number of unprocessed requests + "sessions": 85, # Number of current sessions (connections) + "subs": 6235, # Number of current address subscriptions + "txs_sent": 2 # Total transactions sent by ElectrumX + } + + Clients that are slow to consume data sent to them are *paused* + until their socket buffer drains sufficiently, at which point + processing of requests resumes. + + Each ill-formed request, or one that does not follow the Electrum + protocol, increments the error count of the session that sent it. + If the error count reaches a certain level (currently 10) that + client is disconnected. + + Apart from very short intervals, typically after a new block or when + a client has just connected, the number of unprocessed requests + should normally be zero. + + Sessions are put into groups, primarily as an anti-DoS measure. + Initially all connections made within a period of time are put in + the same group. High bandwidth usage by a member of a group + deprioritizes itself, and all members of its group to a lesser + extent. Low-priority sessions have their requests served after + higher priority sessions. ElectrumX will start delaying responses + to a sessions if it becomes sufficiently deprioritized. + +* **sessions** + + Returns a list of all current sessions. Takes no arguments. + + .. code:: + + $ electrumx_rpc.py sessions + + ID Flags Client Reqs Txs Subs Recv Recv KB Sent Sent KB Time Peer + 0 S1 2.7.12 0 0 293 352 34 355 35 0:49:27 192.168.0.1:4093 + 1 T1 2.5.2 0 0 87 141 12 144 13 0:49:25 xxx.xx.xx.x:39272 + 2 T1 all_seeing_eye 0 0 0 10 0 13 2 0:49:25 xxx.xx.xxx.xx:57862 + 3 S1 all_seeing_eye 0 0 0 10 0 13 2 0:49:25 xxx.xx.xxx.xx:41315 + 4 S1 2.6.4 0 0 2,048 2,104 215 2,108 122 0:49:25 xxx.xx.xxx.xx:35287 + ... + 435 R0 RPC 0 0 0 1 0 0 0 0:00:00 [::1]:1484 + + + The columns show the session ID, flags (see below), how the client + identifies itself - typically the Electrum client version, the + number of unprocessed requests, the number of transactions sent, the + number of address subscriptions, the number of requests received and + the bandwidth used, the number of messages sent and the bandwidth + used, how long the client has been connected, and the client's IP + address. + + The flags are: + + * S an SSL connection + * T a TCP connection + * R a local RPC connection + * L a logged session + * C a connection that is being closed + * the non-negative number is the connection priority, with lower + numbers having higher priority. RPC connections have priority + 0, normal connections have priority at least 1. + +* **groups** + + Returns a list of all current groups. Takes no arguments. + + The output is quite similar to the **sessions** command. + +* **disconnect** + + Disconnects the given session IDs. Session IDs can be seen in the + logs or with the **sessions** RPC command. + + .. code:: + + $ ./electrumx_rpc.py disconnect 2 3 + [ + "disconnected 2", + "disconnected 3" + ] + + ElectrumX initiates the socket close process for the passed + sessions. Whilst most connections close quickly, it can take + several minutes for Python to close some SSL connections down. + +* **log** + + Toggles logging of the given session IDs. All incoming requests for + a logged session are written to the server log. Session IDs can be + seen in the logs or with the **sessions** RPC command. + + .. code:: + + $ electrumx_rpc.py log 0 1 2 3 4 5 + [ + "log 0: False", + "log 1: False", + "log 2: False", + "log 3: True", + "log 4: True", + "unknown session: 5" + ] + + The return value shows this command turned off logging for sesssions + 0, 1 and 2. It was turned on for sessions 3 and 4, and there was no + session 5. + +* **peers** + + Returns a list of peer electrum servers. This command takes no arguments. + + Currently this is data gleaned from an IRC session. diff --git a/server/version.py b/server/version.py index 46ec404..47bceef 100644 --- a/server/version.py +++ b/server/version.py @@ -1 +1 @@ -VERSION = "ElectrumX 0.9.20" +VERSION = "ElectrumX 0.9.22"