RanchiMall/electrumx
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Prepare 0.9.21

Browse Source
This commit is contained in:
Neil Booth 2017-01-02 19:58:27 +09:00
parent b29ce0604d
commit d951898b01
2 changed files with 211 additions and 755 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

275
README.rst
View File

@ -3,84 +3,86 @@
.. image:: https://coveralls.io/repos/github/kyuupichan/electrumx/badge.svg
:target: https://coveralls.io/github/kyuupichan/electrumx
ElectrumX - Reimplementation of Electrum-server
===============================================
ElectrumX - Reimplementation of electrum-server
===============================================
::
Licence: MIT Licence
Licence: MIT
Author: Neil Booth
Language: Python (>=3.5)
Getting Started
===============
See :code:`docs/HOWTO.rst`.
See `docs/HOWTO.rst`_.
Motivation
==========
For privacy and other reasons, I have long wanted to run my own
Electrum server, but for reasons I cannot remember I struggled to set
it up or get it to work on my DragonFlyBSD system, and I lost interest
for over a year.
Mainly for privacy reasons, I have long wanted to run my own Electrum
server, but I struggled to set it up or get it to work on my
DragonFlyBSD system and lost interest for over a year.
More recently I heard that Electrum server databases were around 35GB
in size when gzipped, and had sync times from Genesis of over a week
(and sufficiently painful that no one seems to have done one for a
long time) and got curious about improvements. After taking a look at
the existing server code I decided to try a different approach.
In September 2015 I heard that electrum-server databases were getting
large (35-45GB when gzipped), and it would take several weeks to sync
from Genesis (and was sufficiently painful that no one seems to have
done it for about a year). This made me curious about improvements
and after taking a look at the code I decided to try a different
approach.
I prefer Python3 over Python2, and the fact that Electrum is stuck on
Python2 has been frustrating for a while. It's easier to change the
server to Python3 than the client.
server to Python3 than the client, so I decided to write my effort in
Python3.
It also seemed like a good way to learn about asyncio, which is a
wonderful and powerful feature of Python from 3.4 onwards.
Incidentally asyncio would also make a much better way to implement
It also seemed like a good opportunity to learn about asyncio, a
wonderful and powerful feature introduced in Python 3.4.
Incidentally, asyncio would also make a much better way to implement
the Electrum client.
Finally though no fan of most altcoins I wanted to write a codebase
that could easily be reused for those alts that are reasonably
compatible with Bitcoin. Such an abstraction is also useful for
testnets, of course.
testnets.
Features
========
- The full Electrum protocol is implemented with the exception of the
blockchain.address.get_proof RPC call, which is not used in normal
sessions and only sent from the Electrum command line.
- The full Electrum protocol is implemented. The only exception is
the blockchain.address.get_proof RPC call, which is not used by
Electrum GUI clients, and can only be invoked from the command line.
- Efficient synchronization from Genesis. Recent hardware should
synchronize in well under 24 hours, possibly much faster for recent
CPUs or if you have an SSD. The fastest time to height 439k (mid
November 2016) reported is under 5 hours. Electrum-server would
probably take around 1 month.
- Subscription limiting both per-connection and across all connections.
November 2016) reported is under 5 hours. For comparison, JElectrum
would take around 4 days, and electrum-server probably around 1
month, on the same hardware.
- Various configurable means of controlling resource consumption and
handling denial of service attacks. These include maximum
connection counts, subscription limits per-connection and across all
connections, maximum response size, per-session bandwidth limits,
and session timeouts.
- Minimal resource usage once caught up and serving clients; tracking the
transaction mempool seems to take the most memory.
- Each client is served asynchronously to all other clients and tasks,
so busy clients do not reduce responsiveness of other clients'
requests and notifications, or the processing of incoming blocks.
- Daemon failover. More than one daemon can be specified; ElectrumX
will failover round-robin style if the current one fails for any
reason.
- Coin abstraction makes compatible altcoin support easy.
transaction mempool appears to be the most expensive part.
- Fully asynchronous processing of new blocks, mempool updates, and
client requests. Busy clients should not noticeably impede other
clients' requests and notifications, nor the processing of incoming
blocks and mempool updates.
- Daemon failover. More than one daemon can be specified, and
ElectrumX will failover round-robin style if the current one fails
for any reason.
- Coin abstraction makes compatible altcoin and testnet support easy.
Implementation
==============
ElectrumX does not do any pruning or throwing away of history. It
will retain this property for as long as feasible, and I believe it is
ElectrumX does not do any pruning or throwing away of history. I want
to retain this property for as long as it is feasible, and it appears
efficiently achievable for the forseeable future with plain Python.
So how does it achieve a much more compact database than Electrum
server, which is forced to prune hisory for busy addresses, and yet
sync roughly 2 orders of magnitude faster?
I believe all of the following play a part::
The following all play a part in making ElectrumX very efficient as a
Python blockchain indexer:
- aggressive caching and batching of DB writes
- more compact and efficient representation of UTXOs, address index,
@ -105,17 +107,17 @@ I believe all of the following play a part::
eliminate CPU idling. As a Python program ElectrumX is unavoidably
single-threaded in its essence; we must keep that CPU core busy.
Python's asyncio means ElectrumX has no (direct) use for threads and
associated complications. I cannot foresee any case where they might
be necessary.
Python's ``asyncio`` means ElectrumX has no (direct) use for threads
and associated complications.
Roadmap Pre-1.0
===============
- minor code cleanups.
- there may be a DB format change to index the DB in a way purely
- there will be a DB format change to index the DB in a way purely
dependent on the script and not on address prefix
- support bitcoin testnet with Satoshi bitcoind 0.13.1
- implement simple protocol to discover peers without resorting to IRC.
This may slip to post 1.0
@ -135,29 +137,174 @@ Roadmap Post-1.0
Database Format
===============
The database and metadata formats of ElectrumX are likely to change.
Such changes will render old DBs unusable. For now I do not intend to
provide converters as the time taken from genesis to synchronize to a
pristine database is quite tolerable.
The database and metadata format of ElectrumX is very likely to change
prior to 1.0 release. Existing DBs will not be unusable and you will
need to resync from Genesis, which is quite tolerable.
Miscellany
==========
ChangeLog
=========
As I've been researching where the time is going during block chain
indexing and how various cache sizes and hardware choices affect it,
I'd appreciate it if anyone trying to synchronize could tell me::
Version 0.9.21
--------------
- the version of ElectrumX
- their O/S and filesystem
- their hardware (CPU name and speed, RAM, and disk kind)
- whether their daemon was on the same host or not
- whatever stats about sync height vs time they can provide (the
logs give it all in wall time)
- the network (e.g. bitcoin mainnet) they synced
* moved RELEASE-NOTES into this README
* document the RPC interface in docs/RPC-INTERFACE.rst
* clean up open DB handling, issue `#89`_
Version 0.9.20
--------------
* fix for IRC flood issue `#93`_
Version 0.9.19
--------------
* move sleep outside semaphore (issue `#88`_)
Version 0.9.18
--------------
* last release of 2016. Just a couple of minor tweaks to logging.
Version 0.9.17
--------------
* have all the DBs use fsync on write; hopefully means DB won't corrupt in
case of a kernel panic (issue `#75`_)
* replace $DONATION_ADDRESS in banner file
Version 0.9.16
--------------
* logging improvements, including throttling of abusive logs
* permit large RPC requests (issue 85)
Version 0.9.15
--------------
* fix crash on reorg, issue #84
Version 0.9.14
--------------
* don't start processing mempool until block processor has caught up.
Print server settings when servers start, not at startup.
Version 0.9.13
--------------
* fix to reduce verbosity of logging of deprioritised sessions. Sessions
are deprioritised if they are using high bandwidth, or if they are part
of a group using high bandwidth. Previously each delayed request scheduling
would be logged, now only changes in the delay (up or down) are logged.
Version 0.9.12
--------------
* enchancements to RPC and logging. getinfo output has changed, a couple
of fields renamed.
issue 77: add PID to getinfo
issue 78: start RPC immediately, don't wait for catch-up
issue 79: show IPv6 address-port combinations properly in []
issue 80: show DB and daemon heights in getinfo
Version 0.9.11
--------------
* rework the fetch-and-process blocks loop. This regains some of the
sync efficiency we lost during 0.8.x and that was poorly hacked
around earlier in 0.9.x. Continuing to investigate where the rest
went.
* logging of block processing times fixes #58
* moved the peer column to the end of the sessions RPC so that IPv6 addrs
don't mess up the formatting
Version 0.9.10
--------------
* logging improvements
* fixed issue #76 (RPCError namespace)
Version 0.9.9
-------------
* prioritize mempool processing of sent txs. Closes issue 73.
* mempool tx processing needs to handle DBError exceptions. Fixes issue 74.
Version 0.9.8
-------------
* cleanup up mempool handling, notify of addresses only once when a new block
comes in. Fixes issue 70.
Version 0.9.7
-------------
* history and UTXO requests are now processed by the executor, i.e.,
properly asynchronously. This was the last of the potential latency
bottlenecks.
Version 0.9.6
-------------
* fix it properly this time
Version 0.9.5
-------------
* fix issue introduced in 0.9.4 with paused connections
Version 0.9.4
-------------
* new env var MAX_SESSIONS, see docs/ENV-NOTES. The default limit is
1,000 sessions so raise this if you want to be able to take more.
* a couple of minor bug fixes relating to paused connections
* removed RPC calls numsessions and numpeers. They're not very interesting
and all that and more is in getinfo.
Version 0.9.3
-------------
* unconfirmed flag indicating whether mempool txs have unconfirmed inputs
was inverted
Version 0.9.2
-------------
* fix mempool busy waiting
Version 0.9.1
-------------
* fix another couple of issues introduced in 0.9.0
Version 0.9.0a
--------------
* fix typo in 0.9.0
Version 0.9.0
-------------
* complete rewrite of mempool code to have minimal latency and fix a
couple of minor bugs. When a new block is found, ideally this
should be communicated to clients who addresses are affected with a
single notification. Previously this would happen with two
notifications: one because the TX got in the block, and one because
that TX was no longer in the mempool. Fundamentally this a race
condition that cannot be eliminated but its occurrence should be
minimized.
Neil Booth
kyuupichan@gmail.com
https://github.com/kyuupichan
**Neil Booth** kyuupichan@gmail.com https://github.com/kyuupichan
1BWwXJH3q6PRsizBkSGm2Uw4Sz1urZ5sCj
.. _#75: https://github.com/kyuupichan/electrumx/issues/75
.. _#88: https://github.com/kyuupichan/electrumx/issues/88
.. _#89: https://github.com/kyuupichan/electrumx/issues/89
.. _#93: https://github.com/kyuupichan/electrumx/issues/93
.. _docs/HOWTO.rst: https://github.com/kyuupichan/electrumx/blob/master/docs/HOWTO.rst

691
RELEASE-NOTES
View File

@ -1,691 +0,0 @@
version 0.9.20
--------------
- fix for IRC flood issue 93
NOTE: I will soon move the RELEASE-NOTES into the README
version 0.9.19
--------------
- move sleep outside semaphore (issue 88)
version 0.9.18
--------------
- last release of 2016. Just a couple of minor tweaks to logging.
version 0.9.17
--------------
- have all the DBs use fsync on write; hopefully means DB won't corrupt in
case of a kernel panic (issue 75)
- replace $DONATION_ADDRESS in banner file
version 0.9.16
--------------
- logging improvements, including throttling of abusive logs
- permit large RPC requests (issue 85)
version 0.9.15
--------------
- fix crash on reorg, issue #84
version 0.9.14
--------------
- don't start processing mempool until block processor has caught up.
Print server settings when servers start, not at startup.
version 0.9.13
--------------
- fix to reduce verbosity of logging of deprioritised sessions. Sessions
are deprioritised if they are using high bandwidth, or if they are part
of a group using high bandwidth. Previously each delayed request scheduling
would be logged, now only changes in the delay (up or down) are logged.
version 0.9.12
--------------
- enchancements to RPC and logging. getinfo output has changed, a couple
of fields renamed.
issue 77: add PID to getinfo
issue 78: start RPC immediately, don't wait for catch-up
issue 79: show IPv6 address-port combinations properly in []
issue 80: show DB and daemon heights in getinfo
version 0.9.11
--------------
- rework the fetch-and-process blocks loop. This regains some of the
sync efficiency we lost during 0.8.x and that was poorly hacked
around earlier in 0.9.x. Continuing to investigate where the rest
went.
- logging of block processing times fixes #58
- moved the peer column to the end of the sessions RPC so that IPv6 addrs
don't mess up the formatting
** Please don't run version 0.10.0, it will corrupt your DB.
version 0.9.10
--------------
- logging improvements
- fixed issue #76 (RPCError namespace)
version 0.9.9
-------------
- prioritize mempool processing of sent txs. Closes issue 73.
- mempool tx processing needs to handle DBError exceptions. Fixes issue 74.
version 0.9.8
-------------
- cleanup up mempool handling, notify of addresses only once when a new block
comes in. Fixes issue 70.
version 0.9.7
-------------
- history and UTXO requests are now processed by the executor, i.e.,
properly asynchronously. This was the last of the potential latency
bottlenecks.
version 0.9.6
-------------
- fix it properly this time
version 0.9.5
-------------
- fix issue introduced in 0.9.4 with paused connections
version 0.9.4
-------------
- new env var MAX_SESSIONS, see docs/ENV-NOTES. The default limit is
1,000 sessions so raise this if you want to be able to take more.
- a couple of minor bug fixes relating to paused connections
- removed RPC calls numsessions and numpeers. They're not very interesting
and all that and more is in getinfo.
version 0.9.3
-------------
- unconfirmed flag indicating whether mempool txs have unconfirmed inputs
was inverted
version 0.9.2
-------------
- fix mempool busy waiting
version 0.9.1
-------------
- fix another couple of issues introduced in 0.9.0
version 0.9.0a
--------------
- fix typo in 0.9.0
version 0.9.0
-------------
- complete rewrite of mempool code to have minimal latency and fix a
couple of minor bugs. When a new block is found, ideally this
should be communicated to clients who addresses are affected with a
single notification. Previously this would happen with two
notifications: one because the TX got in the block, and one because
that TX was no longer in the mempool. Fundamentally this a race
condition that cannot be eliminated but its occurrence should be
minimized.
version 0.8.12
--------------
- pause serving sessions whose send buffers are full (anti-DoS). This
is currently logged; let me know if it's too verbose
- various tweaks to request handling
version 0.8.11
--------------
- catch harmless socket exception
- show session count in groups RPC call
version 0.8.10
--------------
- fix socket bug in 0.8.9
version 0.8.9
-------------
- RPC groups and sessions calls improved
- issues fixed: #62, #68 (slow socket closing, IRC)
version 0.8.8
-------------
- put sessions in a priority queue to better prioritise serving. Low-bandwidth
sessions get served first
- new RPC command "groups" - shows information about session groups
- sessions output: session priority shown under Flags column; the lower the
number the higher the priority. txs column moved, new column reqs showns
the number of outstanding requests for that connection (includes subrequests
of batches)
- issued fixed: #67
version 0.8.7
-------------
- update systemd config (bauerj)
- temporary fix for initial sync times
- continued JSON code refactoring
version 0.8.6
-------------
- fix JSON bugs from 0.8.5
- fix issue #61 (valesi)
version 0.8.5
-------------
- rework of JSON layer to better handle batch requests. This is
preparatory work for improved DoS resistance.
I'm aware recent versions don't sync efficiently; please use 0.8.0 to sync
until I find time to fix it.
version 0.8.4
-------------
- remove invalidated histories from cache on new block
version 0.8.3
-------------
Minor tweaks to session logs:
- sessions output now shows flags. All sessions are listed. The
session type column is gone, instead the first letter of RPC, SSL or
TCP is the first flag letter. A 'C' flag indicates the session is closing.
An 'L' that it's being logged.
- don't attempt to forcefully stale sockets; they remain in C state until
Python closes them (which can be a long time for some SSL sockets)
- don't consider all seeing eye connections as stale
version 0.8.2
-------------
- process new blocks in the asyncio executor; essentially a python thread.
This should eliminate latency during block processing that caused sessions
to be dropped.
- bandwith limit is restored incrementally to a session over the hour
rather than in a lump when one hour has passed. Also, only the
limit is refunded each hour; previously the entire usage would be
refunded. So if the session uses 30MB bandwidth and your limit is
10MB, it will take 3 hrs before the session is considered to have
used none of its allotted bandwidth; previously it would happen after 1
hour.
version 0.8.1
-------------
** NOTE: this version has a new Python package dependency: pylru
- fix for IRC encoding killing IRC connection
- add lru cache for history
version 0.8.0
------------
- stale connections are periodically closed. See docs/ENV-NOTES for
SESSION_TIMEOUT, default is 10 minutes. Issue #56.
- each session gets its own ID which is used in the logs instead of its
network address; the network address is still shown on initial connection.
Issue #55.
- the session ID is also shown in the sessions list. You can use this ID
with the following new RPC commands which take a list of session ids:
electrumx_rpc.py log
electrumx_rpc.py disconnect
The first will toggle logging of the sessions. A logged sesssion
prints every incoming request to the logs.
The second will disconnect the sessions.
Example: "electrumx_rpc.py log 15 369"
version 0.7.20
--------------
- fix for errors during batch requests (issue #54)
- don't log errors on shutdown after giving sockets time to close
version 0.7.19
--------------
- revert mempool change of 0.7.18
version 0.7.18
--------------
- better IRC support for tor (valesi)
- issues: suppressed some uninteresting socket logging to fix #52
- mempool: fixed small memory leak
version 0.7.17
--------------
- upped read buffer limit to 1,000,000 bytes.
version 0.7.16
--------------
- fix bug introduced in 0.7.12 that hit during reorgs
version 0.7.15
--------------
The following meta variables in your banner file are now replaced in
addition to $VERSION described in the notes to 0.7.11. If you type
getnetworkinfo in your daemon's debug console you will see what they
are based on:
- $DAEMON_VERSION is replaced with the daemon's version as a dot-separated
string. For example "0.12.1".
- $DAEMON_SUBVERSION is replaced with the daemon's user agent string.
For example, "/BitcoinUnlimited:0.12.1(EB16; AD4)/".
version 0.7.14
--------------
Improved DoS protection:
- incoming network request buffers - which hold incomplete requests
are limited to 150,000 bytes, which I believe is large for genuine
clients. I don't foresee a need to change this so it is hard-coded.
If an incoming request (for example, text without a newline) exceeds
this limit the connection is dropped and the event logged.
- RPC connections have high MAX_SEND and incoming buffer limits as these
connections are assumed to be trusted.
- new environment variable BANDWIDTH_LIMIT. See docs/ENV-NOTES.
- fixes: LOG_SESSIONS of 0.7.13 wasn't being properly interpreted.
Tweak to rocksdb close() that should permit db reopening to work.
version 0.7.13
--------------
- the output of the RPC sessions and getinfo calls are now written to logs
periodically by default. See LOG_SESSIONS in docs/ENV-NOTES
- Litecoin update (santzi)
version 0.7.12
--------------
- minor bug fixes: 2 in JSON RPC, 1 in get_utxos (affected addresslistunspent)
- leveldb / rocksdb are opened with a different maximum open files limit,
depending on whether the chain has been fully synced or not. If synced
you want the files for network sockets, if not synced for the DB engines.
Once synced the DB will be reopened with the lower limit to free up the
files for serving network connections
- various refactoring preparing for possible asynchronous block processing
version 0.7.11
--------------
- increased MAX_SEND default value to 1 million bytes so as to be able
to serve large historical transactions of up to ~500K in size. The
MAX_SEND floor remains at 350,000 bytes so you can reduce it if you
wish. To serve any historical transaction for bitcoin youd should
set this to around 2,000,100 bytes (one byte becomes 2 ASCII hex chars)
- issue #46: fix reorgs for coinbase-only blocks. We would not distinguish
undo information being empty from it not existing
- issue #47: IRC reconnects. I don't get this issue so cannot be certain
it is resolved
- $VERSION in your banner file is now replaced with your ElectrumX version
- more work to isolate the DB from the block processor, working towards the
goal of asynchronous block updates
version 0.7.10
--------------
- replaced MAX_HIST environment variable with MAX_SEND, see docs/ENV-NOTES.
Large requests are blocked and logged. The logs should help you determine
if the requests are genuine (perhaps requiring a higher MAX_SEND) or abuse.
version 0.7.9
-------------
- rewrite jsonrpc.py to also operate as a client. Use this class
for a robust electrumx_rpc.py. Fixes issue #43
version 0.7.8
-------------
- hopefully fix failed assertion on reorgs, issue #44
version 0.7.7
-------------
- add MAX_HIST to throttle history requests; see docs/ENV-NOTES. One
provider of ElectrumX services was attacked by a loser requesting
long histories; this environment variable allows you to limit what
you attempt to serve.
version 0.7.6
-------------
- Fix IRC regression of 0.7.5 - would always connect to IRC by default
version 0.7.5
-------------
- refactoring of server manager and event handling. One side effect
is to fix a bug in 0.7.4 where after a reorg ElectrumX might create
a second mempool and/or kick off more servers. Your testing would
be appreciated. This is part of the refactoring necessary to
process incoming blocks asynchronously so client connections are not
left waiting for several seconds
- close connections on first bad JSON encoding. Do not process buffered
requests of a closing connection
- make IRC params a function of the coin (TheLazier) and supply them for
Dash
version 0.7.4
-------------
- really fix reorgs, they still triggered an assertion. If you hit a reorg
I believe your DB is fine and all you need to do is restart with updated
software
- introduced a new debug env var FORCE_REORG which I used to simulate a
reorg and confirm they should work
version 0.7.3
-------------
- fix reorgs - broken since 0.6 I think
version 0.7.2
-------------
- don't log message decoding errors. Cut off a connection after it has sent
10 ill-formed requests.
- doc improvements (cluelessperson)
- RPC ports for Dash (TheLazier)
version 0.7.1
-------------
- fixes an unqualified use of RPCError
version 0.7
-----------
- daemon failover is now supported; see docs/ENV-NOTES. As a result,
DAEMON_URL must now be supplied and DAEMON_USERNAME, DAEMON_PASSWORD,
DAEMON_HOST and DAEMON_PORT are no longer used.
- fixed a bug introduced in 0.6 series where some client header requests
would fail
- fully asynchronous mempool handling; blocks can be processed and clients
notified whilst the mempool is still being processed
version 0.6.3
-------------
- new environment variables MAX_SUBS and MAX_SESSION_SUBS. Please read
docs/ENV-NOTES - I encourage you to raise the default values.
- fixed import bug in 0.6.2 that prevented initial sync
- issues closed: #30. Logs should be clean on shutdown now.
version 0.6.2
-------------
- handle daemon errors properly that result from client requests; pass the
error onto the client
- start serving immediatley on catchup; don't wait for the mempool
- logging improvements, in particular logging software and DB versions
- issues closed: #29, #31, #32
version 0.6.1
-------------
- main focus was better logging - more concise and informative, particularly
when caught up
- issues closed: #26, #27
- default reorg limit is now taken from the coin, with a high default for
bitcoin testnet
version 0.6.0
-------------
- DB format has changed again. This doesn't give a performance gain
or reduction that I could measure, but is cleaner in that each table
entry is now a singleton and not an array, which I much prefer as a
cleaner solution. It may enable other goodness in the future.
- Logging is much less noisy when serving clients. In fact anything
in your logs that isn't just status updates probably is a bug that I
would like to know about. Unfortunately clean shutdown whilst
serving clients leads to massive log spew. This is harmless and I
believe because of my noob status with asyncio. I intend to fix
this in a nearby release.
- expensive client requests are intended to yield to other requests
sufficiently frequently that there should be no noticeable delays or
pauses under normal load from hog clients.
- Notifications to hog clients are now queued in sequence with their
request responses. They used to be sent immediately regardless of
pending requests which seems less than ideal.
- some trivial improvements and fixes to local RPC query output
version 0.5.1
-------------
- 0.5 changed some cache defaults, only partially intentionally. For
some users, including me, the result was a regression (a 15hr HDD
sync became a 20hr sync). Another user reported their fastest sync
yet (sub 10hr SSD sync). What changed was memory accounting - all
releases until 0.5 were not properly accounting for memory usage of
unflushed transaction hashes. In 0.5 they were accounted for in the
UTXO cache, which resulted in much earlier flushes. 0.5.1 flushes
the hashes at the same time as history so I now account for it
towards the history cache limit. To get a reasonable comparison
with prior releases your HIST_MB environment variable should be
bumped by about 15% from 0.4 and earlier values. This will not
result in greater memory consumption - the additional memory
consumption was being ignored before but is now being included.
- 0.5.1 is the first release where Electrum client requests are queued
on a per-session basis. Previously they were in a global queue.
This is the beginning of ensuring that expensive / DOS requests
mostly affect that user's session and not those of other users. The
goal is that each session's requests run asynchronously parallel to
every other sessions's requests. The missing part of the puzzle is
that Python's asyncio is co-operative, however at the moment
ElectrumX does not yield during expensive requests. I intend that a
near upcoming release will ensure expensive requests yield the CPU
at regular fine-grained intervals. The intended result is that, to
some extent, expensive requests mainly delay that and later requests
from the same session, and have minimal impact on the legitimate
requests of other sessions. The extent to which this goal is
achieved will only be verifiable in practice.
- more robust tracking and handling of asynchronous tasks. I hope
this will reduce asyncio's logging messages, some of which I'm
becoming increasingly convinced I have no control over. In
particular I learned earlier releases were unintentionally limiting
the universe of acceptable SSL protocols, and so I made them the
default that had been intended.
- I added logging of expensive tasks, though I don't expect much real
information from this
- various RPC improvements
version 0.5
-----------
- DB change: all UTXOs, including those that are not canonically paying to
an address, are stored in the DB. So an attempt to spend a UTXO not in
the DB means corruption. DB version bumped to 2; older versions will not
work
- fixed issue #17: the genesis coinbase is not in the UTXO set
version 0.4.3
-------------
- fix exception introduced in 0.4.2
version 0.4.2
-------------
- split out JSON RPC protcol handling. Now more robust and we should
fully support JSON RPC 2.0 clients, including batch requests
(Electrum client does not yet support these)
- refactored and cleaned up server handling
- improved DASH support (thelazier)
version 0.4.1
-------------
- tweak IRC version reporting so we appear in the Electrum client's
network dialog box
version 0.4
-----------
- IRC connectivity. See the notes for environment variables, etc.
- logging improvements
Version 0.3.2, 0.3.3
--------------------
- fixed silly bugs
Version 0.3.1
-------------
- fixes issue #9
- save DB version in DB; warn on DB open if incompatible format
Version 0.3
-----------
- Database format has changed; old DBs are incompatible. They will
not work and will probably die miserably as I'm not yet versioning
them for helpful warnings (coming soon).
- The change in on-disk format makes UTXO flushes noticeably more
efficient. My gut feeling is it probably benefits HDDs more than
SSDs, but I have no numbers to back that up other than that my HDD
synced about 90 minutes (10%) faster. Until the treacle hits at
blocks 300k+ there will probably be little noticeable difference in
sync time.
Version 0.2.3
-------------
- fixes issues #6, #11, #15
- the UTXO cache is now merged with BlockProcessor, where it properly belongs.
cache.py no longer exists
Version 0.2.2.1
---------------
- fixes issues #12, #13
- only attempt to flush on asyncio.CancelledError to avoid spurious
secondary errors
Version 0.2.2
-------------
- mostly refactoring: controller.py is gone; cache.py is half-gone.
Split BlockProcessor into 3: DB, BlockProcessor and BlockServer. DB
handles stored DB and FS state; BlockProcessor handles pushing the
chain forward and caching of updates, and BlockServer will
additionally serve clients on catchup. More to come.
- mempool: better logging; also yields during initial seeding
- issues fixed: #10
Version 0.2.1
-------------
- fix rocksdb and lmdb abstractions (bauerj)
- limit concurrent daemon requests
- improve script + coin abstractions
- faster tx and script parsing
- minor bug fixes
Version 0.2
-----------
- update sample run script, remove empty addresses from mempool
Version 0.1
------------
- added setup.py, experimental. Because of this server_main.py renamed
electrumx_server.py, and SERVER_MAIN environment variable was renamed
to ELECTRUMX. The sample run script was updated to match.
- improvements to logging of daemon connection issues
- removal of old reorg test code
- hopefully more accurate sync ETA
Version 0.07
------------
- fixed a bug introduced in 0.06 at the last minute
Version 0.06
------------
- mempool support. ElectrumX maintains a representation of the daemon's
mempool and serves unconfirmed transactions and balances to clients.
Version 0.05
------------
- fixed a bug in 0.04 that stopped ElectrumX serving once synced
Version 0.04
------------
- made the DB interface a little faster for LevelDB and RocksDB; this was
a small regression in 0.03
- fixed a bug that prevented block reorgs from working
- implement and enable client connectivity. This is not yet ready for
public use for several reasons. Local RPC, and remote TCP and SSL
connections are all supported in the same way as Electrum-server.
ElectrumX does not begin listening for incoming connections until it
has caught up with the daemon's height. Which ports it is listening
on will appear in the logs when it starts listening. The complete
Electrum wire protocol is implemented, so it is possible to now use
as a server for your own Electrum client. Note that mempools are
not yet handled so unconfirmed transactions will not be notified or
appear; they will appear once they get in a block. Also no
responses are cached, so performance would likely degrade if used by
many clients. I welcome feedback on your experience using this.
Version 0.03
------------
- merged bauerj's abstracted DB engine contribution to make it easy to
play with different backends. In addition to LevelDB this adds
support for RocksDB and LMDB. We're interested in your comparitive
performance experiences.
Version 0.02
------------
- fix bug where tx counts were incorrectly saved
- large clean-up and refactoring of code, breakout into new files
- several efficiency improvements
- initial implementation of chain reorg handling
- work on RPC and TCP server functionality. Code committed but not
functional, so currently disabled
- note that some of the enivronment variables have been renamed,
see samples/scripts/NOTES for the list