--- /dev/null
+9th July 2020
+-------------
+
+IF YOU ARE ON THE MASTER BRANCH, STOP (I REPEAT) STOP READING THIS
+DOCUMENT AND READ UPGRADE.mojo FIRST.
+
+
+If you are not on an existing 'mojo' branch, or you have a build less
+than 260, please stop reading as well and read UPGRADE.mojo (in the
+section about installing packages), as you need to install some extra
+packages before restarting the new version of the code. If in doubt
+try installing the packages again as this will either confirm that all
+is already done or will install what's missing.
+
+Assuming you have done the above:
+
+The latest release of the Mojo branch of DXSpider contains a client
+for the Reverse Beacon Network (RBN). This is not a simple client, it
+attempts to make some sense of the 10s of 1000s of "spots" that the
+RBN can send PER HOUR. At busy times, actually nearly all the time, the
+spots from the RBN come in too quickly for anybody to get anything more
+than a fleeting impression of what's coming in.
+
+Something has to try to make this manageable - which is what I have
+tried to do with DXSpider's RBN client.
+
+The RBN has a number of problems (apart from the overwhelming quantity
+of data that it sends):
+
+* Spotted callsigns, especially on CW, are not reliably
+ decoded. Estimates vary as to how bad it is but, as far as I can
+ tell, even these estimates are unreliable!
+
+* The frequency given is unreliable. I have seen differences as great
+ as 600hz on CW spots.
+
+* There is far too much (in my view) useless information in each spot
+ - even if one had time to read, decode and understand it before the
+ spot has scrolled off the top of the screen.
+
+* The format of the comment is not regular. If one has both FTx and
+ "all the other" spots (CW, PSK et al) enabled at the same time,
+ one's eye is constantly having to re-adjust. Again, very difficult
+ to deal with on contest days. Especially if it mixed in with
+ "normal" spots.
+
+So what have I done about this? Look at the sample of input traffic
+below:
+
+05Jul2020@22:59:31 (chan) <- I SK0MMR DX de KM3T-2-#: 14100.0 CS3B CW 24 dB 22 WPM NCDXF B 2259Z
+05Jul2020@22:59:31 (chan) <- I SK0MMR DX de KM3T-2-#: 28263.9 AB8Z/B CW 15 dB 18 WPM BEACON 2259Z
+05Jul2020@22:59:31 (chan) <- I SK0MMR DX de LZ3CB-#: 7018.20 RW1M CW 10 dB 18 WPM CQ 2259Z
+05Jul2020@22:59:31 (chan) <- I SK0MMR DX de W9XG-#: 14057.6 K7GT CW 7 dB 21 WPM CQ 2259Z
+05Jul2020@22:59:31 (chan) <- I SK0MMR DX de G0LUJ-#: 14100.1 CS3B CW 18 dB 20 WPM NCDXF B 2259Z
+05Jul2020@22:59:32 (chan) <- I SK0MMR DX de LZ4UX-#: 7018.3 RW1M CW 13 dB 18 WPM CQ 2259Z
+05Jul2020@22:59:32 (chan) <- I SK0MMR DX de LZ4AE-#: 7018.3 RW1M CW 28 dB 18 WPM CQ 2259Z
+05Jul2020@22:59:32 (chan) <- I SK0MMR DX de W1NT-6-#: 28222.9 N1NSP/B CW 5 dB 15 WPM BEACON 2259Z
+05Jul2020@22:59:32 (chan) <- I SK0MMR DX de W1NT-6-#: 28297.0 NS9RC CW 4 dB 13 WPM BEACON 2259Z
+05Jul2020@22:59:32 (chan) <- I SK0MMR DX de F8DGY-#: 7018.2 RW1M CW 23 dB 18 WPM CQ 2259Z
+05Jul2020@22:59:33 (chan) <- I SK0MMR DX de 9A1CIG-#: 7018.30 RW1M CW 20 dB 18 WPM CQ 2259Z
+05Jul2020@22:59:33 (chan) <- I SK0MMR DX de LZ7AA-#: 7018.3 RW1M CW 16 dB 18 WPM CQ 2259Z
+05Jul2020@22:59:33 (chan) <- I SK0MMR DX de DK9IP-#: 7018.2 RW1M CW 21 dB 18 WPM CQ 2259Z
+05Jul2020@22:59:33 (chan) <- I SK0MMR DX de WE9V-#: 10118.0 N5JCB CW 15 dB 10 WPM CQ 2259Z
+05Jul2020@22:59:34 (chan) <- I SK0MMR DX de DJ9IE-#: 7028.0 PT7KM CW 15 dB 10 WPM CQ 2259Z
+05Jul2020@22:59:34 (chan) <- I SK0MMR DX de DJ9IE-#: 7018.3 RW1M CW 31 dB 18 WPM CQ 2259Z
+05Jul2020@22:59:34 (chan) <- I SK0MMR DX de DD5XX-#: 7018.3 RW1M CW 21 dB 18 WPM CQ 2259Z
+05Jul2020@22:59:34 (chan) <- I SK0MMR DX de DE1LON-#: 14025.5 EI5JF CW 13 dB 19 WPM CQ 2259Z
+05Jul2020@22:59:34 (chan) <- I SK0MMR DX de DE1LON-#: 7018.3 RW1M CW 24 dB 18 WPM CQ 2259Z
+05Jul2020@22:59:34 (chan) <- I SK0MMR DX de ON6ZQ-#: 7018.3 RW1M CW 22 dB 18 WPM CQ 2259Z
+05Jul2020@22:59:34 (chan) <- I SK0MMR DX de OH6BG-#: 3516.9 RA1AFT CW 15 dB 25 WPM CQ 2259Z
+05Jul2020@22:59:35 (chan) <- I SK0MMR DX de HA1VHF-#: 7018.3 RW1M CW 30 dB 18 WPM CQ 2259Z
+05Jul2020@22:59:35 (chan) <- I SK0MMR DX de F6IIT-#: 7018.4 RW1M CW 32 dB 18 WPM CQ 2259Z
+05Jul2020@22:59:36 (chan) <- I SK0MMR DX de HB9BXE-#: 7018.3 RW1M CW 23 dB 18 WPM CQ 2259Z
+05Jul2020@22:59:37 (chan) <- I SK0MMR DX de SM0IHR-#: 7018.3 RW1M CW 21 dB 18 WPM CQ 2259Z
+05Jul2020@22:59:37 (chan) <- I SK0MMR DX de DK0TE-#: 7018.3 RW1M CW 26 dB 18 WPM CQ 2259Z
+05Jul2020@22:59:37 (chan) <- I SK0MMR DX de OE9GHV-#: 7018.3 RW1M CW 40 dB 19 WPM CQ 2259Z
+05Jul2020@22:59:37 (chan) <- I SK0MMR DX de CX6VM-#: 10118.0 N5JCB CW 20 dB 10 WPM CQ 2259Z
+05Jul2020@22:59:37 (chan) -> D G1TST DX de F8DGY-#: 7018.3 RW1M CW 23dB Q:9* Z:20 16 2259Z 14
+05Jul2020@22:59:38 (chan) <- I SK0MMR DX de HB9JCB-#: 7018.3 RW1M CW 16 dB 18 WPM CQ 2259Z
+05Jul2020@22:59:39 (chan) <- I SK0MMR DX de HB9JCB-#: 3516.9 RA1AFT CW 9 dB 26 WPM CQ 2259Z
+05Jul2020@22:59:39 (chan) <- I SK0MMR DX de KO7SS-7-#: 14057.6 K7GT CW 6 dB 21 WPM CQ 2259Z
+05Jul2020@22:59:39 (chan) <- I SK0MMR DX de K9LC-#: 28169.9 VA3XCD/B CW 9 dB 10 WPM BEACON 2259Z
+05Jul2020@22:59:40 (chan) <- I SK0MMR DX de HB9DCO-#: 7018.2 RW1M CW 25 dB 18 WPM CQ 2259Z
+05Jul2020@22:59:40 (chan) <- I SK0MMR DX de EA5WU-#: 7018.3 RW1M CW 19 dB 18 WPM CQ 2259Z
+
+* As you can see, there are frequently more than one spotter for a
+ callsign:
+
+* I normalise the frequency and cache up to 9 copies from different
+ spots. In order to do this I have to wait a few (configurable) seconds
+ for the client to collect a reasonable number of copies. More copies
+ may come in after 9 copies have been received. Once I have enough
+ copies to be sure that the callsign is at least agreeed upon by more
+ than one skimmer, or the wait timer goes off, I emit a spot. By this
+ means I can reduce the number of spots sent to a node user by up to a
+ factor of around 10 for CW etc spots and about 8 for FTx spots.
+
+ For example, from the trace above, all the RW1M RBN spots become just
+ one line:
+
+DX de F8DGY-#: 7018.3 RW1M CW 23dB Q:9* Z:20 16 2259Z 14
+
+* No RBN spots can leak out of the node to the general cluster. Each
+ node that wants to use the RBN *must* establish their own
+ connections to the RBN.
+
+* Currently no RBN spots are stored. This may well change but how and
+ where these spots are stored is not yet decided. Only "DXSpider
+ curated" spots (like the example above) will be stored (if/when they
+ are). Sh/dx will be suitably modified if storage happens.
+
+* There are some things that need to be explained:
+
+a) The input format from the RBN is not the same as format emitted by
+the cluster node. This is part of the unhelpfulness to mixing a raw
+RBN feed with normal spots.
+
+b) Each spot sent out to a node user has a "Qwalitee" marker, In this
+case Q:9*. The '9' means that I have received 9 copies of this spot
+from different skimmers and, in this case, they did not agree on the
+frequency (7018.2 - 7018.4) which is indicated by a '*'. The frequency
+shown is the majority decision. If this station has been active for
+for a while and he is still calling CQ after some time (configurable,
+but currently 60 minutes), ignoring gaps for QSOs or tea breaks, then
+a '+' character will be added.
+
+If the "Qualitee" Q:1 is seen on a CW spot, then only one skimmer has
+seen that spot and the callsign *could* be wrong, but frequently, if
+it is wrong, it is more obvious than the example below. But if Q is
+Q:2 and above, then the callsign is much more likely to be correct.
+
+DX de DJ9IE-#: 14034.9 UN7BBD CW 4dB Q:5*+ 17 1444Z 14
+DX de OL7M-#: 14037.9 UA6LQ CW 13dB Q:7 16 1448Z 15
+DX de LZ3CB-#: 28050.2 DL4HRM CW 7dB Q:1 14 1448Z 20
+
+Having said that, I check every spots call with the standard DXSpider
+is_callsign routine, to check that the callign is a valid format. Then
+through the Prefix::search routine, to check that the prefix actually
+exists. If either of these checks fails then the spot is ignored.
+
+c) I ditch the WPM and the 'CQ' as not being hugely relevant.
+
+d) If there is a Z:nn[,mm...], then this spot was also heard by
+skimmers in other zones. In this example, it means that this call was
+also heard in CQ Zone 20. This list does NOT include the cq zone of
+the skimmer nor the spot. If you would like to see these then do
+'set/dxcq'. This setting is active for all the examples in this
+document. This is completely optional.
+
+There can be a ',' separated list of as many zones where this spot was
+also heard by another skimmers, up to the space available in the
+comment area.
+
+DX de LZ4UX-#: 14015.5 ON7TQ CW 6dB Q:9 Z:5,14,15,40 14 0646Z 20
+DX de VE7CC-#: 3573.0 N8ADO FT8 -14dB Q:4 Z:4,5 4 0647Z 3
+DX de DM7EE-#: 14027.5 R1AC CW 9dB Q:9* Z:5,15,17,20 16 0643Z 14
+DX de WE9V-#: 7074.0 EA7ALL FT8 -9dB Q:2+ Z:5 14 0641Z 4
+
+e) I shorten the skimmer callsign to 6 characters - having first
+chopped off any SSIDs, spurious /xxx strings from the end, leaving
+just the base callsign, before (re-)adding '-#' on the end. This is
+done to minimise the misalignment of the spot rightwards, as in the
+incoming skimmer spot from KO7SS-7-# below. There are some very
+strange skimmer callsigns with all sorts of spurious endings, all of
+which I attempt to reduce to the base callsign. Some skimmer base
+callsigns still might be shortened for display purposes. Things like
+'3V/K5WEM' won't fit in six characters but the whole base callsign is
+used for zone info internally, but only the first 6 characters are
+displayed in any spot.
+
+05Jul2020@22:59:39 (chan) <- I SK0MMR DX de HB9JCB-#: 3516.9 RA1AFT CW 9 dB 26 WPM CQ 2259Z
+05Jul2020@22:59:39 (chan) <- I SK0MMR DX de KO7SS-7-#: 14057.6 K7GT CW 6 dB 21 WPM CQ 2259Z
+05Jul2020@22:59:39 (chan) <- I SK0MMR DX de K9LC-#: 28169.9 VA3XCD/B CW 9 dB 10 WPM BEACON 2259Z
+
+f) I happen to have a filter set (accept/spot by_zone 14 and not zone
+14 or zone 14 and not by_zone 14) which will give me the first spot
+that either spot or skimmer is in zone 14 but its companion isn't. For
+those of us that are bad at zones (like me) sh/dxcq is your
+friend. You can have separate filters just for RBN spots if you want
+something different to your spot filters. Use acc/rbn or rej/rbn. NB:
+these will completely override your spot filters for RBN
+spots. Obviously "real" spots will will continue to use the spot
+filter(s). If you use set/dxcq, then unset/dxitu and unset/dxgrid
+first. You only need to this once.
+
+g) If there is NO filter in operation, then the skimmer spot with the
+LOWEST signal strength will be shown. This implies that if any extra
+zones are shown, then the signal will be higher in those zones.
+
+h) A filter can further drastically reduce the output sent to the
+user. As this STATS line shows:
+
+23:22:45 (*) RBN:STATS hour SK0MMR raw: 5826 sent: 555 delivered: 70 users: 1
+
+For this hour, I received 5826 raw spots from the CW etc RBN, which
+produced 555 possible spots, which my filter reduced to 70 that were
+actually delivered to G1TST. For the FTx RBN, I don't have a filter
+active and so I got all the possibles:
+
+23:22:45 (*) RBN:STATS hour SK1MMR raw: 13354 sent: 1745 delivered: 1745 users: 1
+
+---------------------------------------------------------------------
+
+So how do you go about using this:
+
+First you need to create one or two RBN user(s). Now you can use any call you
+like and it won't be visible outside of the node. I call mine SK0MMR
+and SK1MMR. One of these connects to the "standard" RBN port that
+outputs CW, BEACON, DXF, PSK and RTTY spots, and the other connects to
+the RBN port that just outputs FT4 and FT8 spots.
+
+set/rbn sk0mmr sk1mmr
+
+Now create connect scripts in /spider/connect/sk0mmr (and similarly
+sk1mmr). They look like this:
+
+/spider/connect/sk0mmr:
+
+connect telnet telnet.reversebeacon.net 7000
+'call:' '<node callsign here'
+
+/spider/connect/sk1mmr:
+
+connect telnet telnet.reversebeacon.net 7001
+'call:' '<node callsign here'
+
+Now put them in your local crontab in /spider/local_cmd/crontab:
+
+* * * * * start_connect('sk0mmr') unless connected('sk0mmr')
+* * * * * start_connect('sk1mmr') unless connected('sk1mmr')
+
+This will check once every minute to see if each RBN connection is
+active, you can check what is connected with the 'links' command:
+
+ Ave Obs Ping Next Filters
+ Callsign Type Started Uptime RTT Count Int. Ping Iso? In Out PC92? Address
+ GB7DJK DXSP 5-Jul-2020 1722Z 7h 6m 8s 0.02 2 300 89 Y 163.172.11.79
+ SK0MMR RBN 5-Jul-2020 1722Z 7h 6m 8s 0 0 198.137.202.75
+ SK1MMR RBN 5-Jul-2020 1722Z 7h 6m 8s 0 0 198.137.202.75
+
+The RBN connections are sometimes dropped or become stuck. I have a
+mechanism to detect this and it will disconnect that RBN connection
+and, by this means,it will be automatically reconnected by the
+crontab, just like any other (normal) node.
+
+I use the crontab, rather than restarting immediately after
+disconnection, to prevent race conditions (or just slow them down to
+one disconnection a minute).
+
+The first time a connection is made, after node startup, there is a 5 (configurable)
+minute pause before RBN spots come out for users. This is done to fill
+up (or "train") the cache. Otherwise the users will be overwhelmed by
+spots - it slows down reasonably quickly - but experiment shows that 5
+minutes is a reasonable compromise. The delay is configurable,
+globally, for all RBN connections, but in future is likely to be
+configurable per connection. Basically, because the FTx RBN data is
+much more bursty and there is more of it (except on CW contests), it
+could do with a somewhat longer training period than the CW etc RBN
+connection.
+
+If a connection drops and reconnects. There is no delay or extra
+training time. If the node is stopped and restarted within a few
+minutes, then no training time will occur.
+
+For users. At the moment. There is a single command that sets or
+unsets ALL RBN spot sorts:
+
+Users can select which type(s) of RBN/Skimmer spot they require using
+the 'set/skimmer' command. Users can enter the 'help set/skimmer'
+command for more information on this. They can also enter then 'help
+rbn' command for more information about how to interpret what DXSpider
+produces and some other useful background information.
+
+This still very much "work in progress" and will be subject to
+change. But I am grateful to the feedback I have received, so far,
+from:
+
+Kin EA3CV
+Andy G4PIQ
+Mike G8TIC
+Lee VE7CC
+
+But if you have comments, suggestions or even brickbats, please email
+me or the support list.
+
+Dirk G1TLH
+
projects / spider.git / blobdiff