6th July 2020

The latest release of the Mojo branch of DXSpider contains a client
for the Reverse Beacon Network (RBN). This is not a simple client, it
attempts to make some sense of the 10s of 1000s of "spots" that the
RBN can send PER HOUR. At busy times, actually nearly all the time, the
spots from the RBN come in too fast for anybody to get anything more
than a fleeting impression of what's coming in.

Something has to try to make this manageable - which is what I have
tried to do with DXSpider's RBN client.

The RBN has a number of problems (apart from the overwhelming quantity
of data that it sends):

* Spotted callsigns, especially on CW, and not reliably
  decoded. Estimates vary as to how bad it is but, as far as I can
  tell, even these estimates are unreliable!

* The frequency given is unreliable. I have seen differences as great
  as 600hz on CW spots.

* There is far too much (in my view) useless information in each spot
  - even if one had time to read, decode and understand it before the
  spot has scrolled off the top of the screen.

* The format of the comment is not regular. If one has both FTx and
  "all the other" spots (CW, PSK et al) enabled at the same time,
  one's eye is constantly having to re-adjust. Again, very difficult
  to deal with on contest days.


So what have I done about this:

* As you can see, there are frequently more than one spotter for a
callsign:

05Jul2020@22:59:31 (chan) <- I SK0MMR DX de KM3T-2-#:  14100.0  CS3B           CW    24 dB  22 WPM  NCDXF B 2259Z
05Jul2020@22:59:31 (chan) <- I SK0MMR DX de KM3T-2-#:  28263.9  AB8Z/B         CW    15 dB  18 WPM  BEACON  2259Z
05Jul2020@22:59:31 (chan) <- I SK0MMR DX de LZ3CB-#:   7018.20  RW1M           CW    10 dB  18 WPM  CQ      2259Z
05Jul2020@22:59:31 (chan) <- I SK0MMR DX de W9XG-#:    14057.6  K7GT           CW     7 dB  21 WPM  CQ      2259Z
05Jul2020@22:59:31 (chan) <- I SK0MMR DX de G0LUJ-#:   14100.1  CS3B           CW    18 dB  20 WPM  NCDXF B 2259Z
05Jul2020@22:59:32 (chan) <- I SK0MMR DX de LZ4UX-#:    7018.3  RW1M           CW    13 dB  18 WPM  CQ      2259Z
05Jul2020@22:59:32 (chan) <- I SK0MMR DX de LZ4AE-#:    7018.3  RW1M           CW    28 dB  18 WPM  CQ      2259Z
05Jul2020@22:59:32 (chan) <- I SK0MMR DX de W1NT-6-#:  28222.9  N1NSP/B        CW     5 dB  15 WPM  BEACON  2259Z
05Jul2020@22:59:32 (chan) <- I SK0MMR DX de W1NT-6-#:  28297.0  NS9RC          CW     4 dB  13 WPM  BEACON  2259Z
05Jul2020@22:59:32 (chan) <- I SK0MMR DX de F8DGY-#:    7018.2  RW1M           CW    23 dB  18 WPM  CQ      2259Z
05Jul2020@22:59:33 (chan) <- I SK0MMR DX de 9A1CIG-#:  7018.30  RW1M           CW    20 dB  18 WPM  CQ      2259Z
05Jul2020@22:59:33 (chan) <- I SK0MMR DX de LZ7AA-#:    7018.3  RW1M           CW    16 dB  18 WPM  CQ      2259Z
05Jul2020@22:59:33 (chan) <- I SK0MMR DX de DK9IP-#:    7018.2  RW1M           CW    21 dB  18 WPM  CQ      2259Z
05Jul2020@22:59:33 (chan) <- I SK0MMR DX de WE9V-#:    10118.0  N5JCB          CW    15 dB  10 WPM  CQ      2259Z
05Jul2020@22:59:34 (chan) <- I SK0MMR DX de DJ9IE-#:    7028.0  PT7KM          CW    15 dB  10 WPM  CQ      2259Z
05Jul2020@22:59:34 (chan) <- I SK0MMR DX de DJ9IE-#:    7018.3  RW1M           CW    31 dB  18 WPM  CQ      2259Z
05Jul2020@22:59:34 (chan) <- I SK0MMR DX de DD5XX-#:    7018.3  RW1M           CW    21 dB  18 WPM  CQ      2259Z
05Jul2020@22:59:34 (chan) <- I SK0MMR DX de DE1LON-#:  14025.5  EI5JF          CW    13 dB  19 WPM  CQ      2259Z
05Jul2020@22:59:34 (chan) <- I SK0MMR DX de DE1LON-#:   7018.3  RW1M           CW    24 dB  18 WPM  CQ      2259Z
05Jul2020@22:59:34 (chan) <- I SK0MMR DX de ON6ZQ-#:    7018.3  RW1M           CW    22 dB  18 WPM  CQ      2259Z
05Jul2020@22:59:34 (chan) <- I SK0MMR DX de OH6BG-#:    3516.9  RA1AFT         CW    15 dB  25 WPM  CQ      2259Z
05Jul2020@22:59:35 (chan) <- I SK0MMR DX de HA1VHF-#:   7018.3  RW1M           CW    30 dB  18 WPM  CQ      2259Z
05Jul2020@22:59:35 (chan) <- I SK0MMR DX de F6IIT-#:    7018.4  RW1M           CW    32 dB  18 WPM  CQ      2259Z
05Jul2020@22:59:36 (chan) <- I SK0MMR DX de HB9BXE-#:   7018.3  RW1M           CW    23 dB  18 WPM  CQ      2259Z
05Jul2020@22:59:37 (chan) <- I SK0MMR DX de SM0IHR-#:   7018.3  RW1M           CW    21 dB  18 WPM  CQ      2259Z
05Jul2020@22:59:37 (chan) <- I SK0MMR DX de DK0TE-#:    7018.3  RW1M           CW    26 dB  18 WPM  CQ      2259Z
05Jul2020@22:59:37 (chan) <- I SK0MMR DX de OE9GHV-#:   7018.3  RW1M           CW    40 dB  19 WPM  CQ      2259Z
05Jul2020@22:59:37 (chan) <- I SK0MMR DX de CX6VM-#:   10118.0  N5JCB          CW    20 dB  10 WPM  CQ      2259Z
05Jul2020@22:59:37 (chan) -> D G1TST DX de F8DGY-#:     7018.3 RW1M         CW  23dB Q:9* Z:20           16 2259Z 14
05Jul2020@22:59:38 (chan) <- I SK0MMR DX de HB9JCB-#:   7018.3  RW1M           CW    16 dB  18 WPM  CQ      2259Z
05Jul2020@22:59:39 (chan) <- I SK0MMR DX de HB9JCB-#:   3516.9  RA1AFT         CW     9 dB  26 WPM  CQ      2259Z
05Jul2020@22:59:39 (chan) <- I SK0MMR DX de KO7SS-7-#:  14057.6  K7GT           CW     6 dB  21 WPM  CQ      2259Z
05Jul2020@22:59:39 (chan) <- I SK0MMR DX de K9LC-#:    28169.9  VA3XCD/B       CW     9 dB  10 WPM  BEACON  2259Z
05Jul2020@22:59:40 (chan) <- I SK0MMR DX de HB9DCO-#:   7018.2  RW1M           CW    25 dB  18 WPM  CQ      2259Z
05Jul2020@22:59:40 (chan) <- I SK0MMR DX de EA5WU-#:    7018.3  RW1M           CW    19 dB  18 WPM  CQ      2259Z

* I normalise the frequency and cache up to 9 copies from different
spots. In order to do this I have to wait a few (comfigurable) seconds
for the client to collect a reasonable number of copies. More copies
may come in after 9 copies have been. Once I have enough copies to be
sure that the callsign is at least agreeed upon by more than one
skimmer, or the wait timer goes off, I emit a spot. An example of
which is shown above in the spot sent to G1TST. By this means I can
reduce the number of spots sent to a node user by up to a factor of 10
for CW etc spots and about 8 for FTx spots.

* No RBN spots can leak out of the node to the general cluster. Each
  node that wants to use the RBN *must* establish their own
  connections to the RBN.

* Currently no RBN spots are stored. This may well change but how and
  where these spots are stored is not yet decided. Only "new" spots
  will be stored (if they are).

* There are some things that need to be said:

a) The input format from the RBN is not the same as format emitted by
the cluster node. This is part of the unhelpfulness to mixing a raw
RBN feed with normal spots.

b) Each spot sent out to a node user has a "Qwalitee" marker, In this
case Q:9*. The '9' means that I have received 9 copies of this spot
from different skimmers and, in this case, they did not agree on the
frequency (7018.2 - 7018.4) which is indicated by a '*'. The frequency
shown is the majority decision. If this station has been active for
some time and he is still calling CQ after some time (configurable,
but currently 60 minutes) and gaps for QSOs or tea breaks are ignored,
then a '+' character will be added.

c) I ditch the WPM and the 'CQ' as not being hugely relevant. 

d) If there is a Z:nn[,mm...] is there it means that this call was also heard
in CQ Zone 20. There can a ',' separated list of as many zones as
there the space available (and this spot call was heard by :-). You
will notice the spot zone and skimmer call zone around the time. This
can be activated with a 'set/dxcq' command. This is completely
optional.

e) I shorten the skimmer callsign to 6 characters, before (re-)adding
'-#' on the end to minimise the movement rightwards as in the incoming
spot from KO7SS-7-# just two lines below G1TST. There are some very
strange skimmer callsigns.

f) I have a filter set (accept/spot by_zone 14 and not zone 14 or zone
14 and not by_zone 14) which will give me the first spot that either
spot or skimmer is in zone 14 but the other isn't. For those of us
that are bad at zones (like me) sh/dxcq is your friend. You can have
separate filters just for RBN spots if you want something different to
your spot filters. Use acc/rbn or rej/rbn. NB: these will completely
override your spot filters for RBN spots. Obviously "real" spots will
will continue to use the spot filter(s).

g) If there is NO filter in operation, then the skimmer spot with the
LOWEST signal strength will be shown. This implies that if any extra
zone are shown then the signal will be higher.

h) A filter can further drastically reduce the output sent to the
user. As this STATS line shows:

23:22:45 (*) RBN:STATS hour SK0MMR raw: 5826 sent: 555 delivered: 70 users: 1

For this hour, I received 5826 raw spots from the CW etc RBN, which
produced 555 possible spots, which my filter reduced to 70 that were
actually delivered to G1TST. For the FTx RBN, I don't have a filter
active and so I got all the possibles:

23:22:45 (*) RBN:STATS hour SK1MMR raw: 13354 sent: 1745 delivered: 1745 users: 1

---------------------------------------------------------------------

So how do you go about using this:

First you need to create an RBN user. Now you can use any call you
like and it won't be visible outside of the node. I call mine SK0MMR
and SK1MMR.

set/rbn sk0mmr sk1mmr

Now create connect scripts in /spider/connect/sk0mmr (and similarly
sk1mmr). They look like this:

/spider/connect/sk0mmr:

connect telnet telnet.reversebeacon.net 7000
'call:' '<node callsign here'

/spider/connect/sk1mmr:

connect telnet telnet.reversebeacon.net 7001
'call:' '<node callsign here'

RBN port 7000 is the "traditional" port for anything except FT4 or FT8
spots. They come from RBN port 7001.

Now put them in your local crontab in /spider/local_cmd/crontab:

* * * * * start_connect('sk0mmr') unless connected('sk0mmr')
* * * * * start_connect('sk1mmr') unless connected('sk1mmr')

This will check once every minute to see if each RBN connection is
active, you can check with the 'links' command:

                                                 Ave  Obs  Ping  Next      Filters
  Callsign Type Started                 Uptime    RTT Count Int.  Ping Iso? In  Out PC92? Address
    GB7DJK DXSP  5-Jul-2020 1722Z     7h 6m 8s   0.02   2    300    89               Y    163.172.11.79
    SK0MMR RBN   5-Jul-2020 1722Z     7h 6m 8s                 0     0                    198.137.202.75
    SK1MMR RBN   5-Jul-2020 1722Z     7h 6m 8s                 0     0                    198.137.202.75

The connections are sometimes dropped or become stuck, I have a
mechanism to detect this and it will disconnect that connection and
the normal reconnection will happen just as any other (normal) node.

It is put in the crontab, rather than started immediately, to prevent
race conditions (or just slow them down to one disconnection a
minute).

The first time a connection is made, after node startup, there is a 5
minute pause before RBN spots come out for users. This is done to fill
up (or "train") the cache. Otherwise the users will be overwhelmed by
spots - it slows down reasonably quickly - but experiment shows that 5
minutes is a reasonable compromise. The delay is configurable,
globally, for all RBN connections, but in future is likely to be
configurable per connection. Basically, because the FTx RBN data is
much more bursty and there is more of it (except on CW contests), it
could do with a somewhat longer training period.

If a connection drops and reconnects. There is no delay or extra
training time.

For users. At the moment. There is a single command that sets or
unsets ALL RBN spot sorts:

set/wantrbn
unset/wantrbn

Very soon this will be replaced with a '(un)set/skimmer' command that
allow the user to choose which categories they want. Filtering can be
used in conjunction with this proposed command to further refine
output.

This still very much "work in progress" and will be subject to
change. But I am grateful to the feedback I have received, so far,
from:

Kin EA3CV
Andy G4PIQ
Mike G8TIC
Lee VE7CC

But if you have comments, suggestions and brickbats please email me or
the support list.

Dirk G1TLH