+20Jan23=======================================================================
+1. Add the variable @main::localhost_names to allow other IP addresses to
+ be treated in the same way as localhost in item 1 on 19Jan23 below. NOTE
+ you must include ALL the normal localhost names + any other interface
+ names that you might want to include:
+
+ set/var @main::localhost_names qw(127.0.0.1 ::1 192.168.1.30)
+
+ using the qw() construction is easier than:
+
+ set/var @main::localhost_names ('127.0.0.1', '::1', '192.168.1.30')
+
+ but either will work. You can define as many IP addresses as you like and
+ they can be IPV4 or 6.
+
+ You do NOT need to fiddle with this unless you specifically have more
+ than just the normal definitions of localhost. So for 'normal' nodes with
+ one external interface, you DO NOT NEED TO DO ANY OF THIS.
+19Jan23=======================================================================
+1. Introduce aliasing for localhost in DX Spots and outgoing PC92 A records
+ on login. There are two variables which can be set with the alias to use:
+ $main::localhost_alias_ipv4
+ $main::localhost_alias_ipv6
+ These can be set in the /spider/scripts/startup, but this is only
+ necessary if the node has more than one interface, or virtual hosts. If
+ there is ONLY ONE ipv4 and/or ipv6 IP address on the node machine then
+ these variables will be automatically populated on first use. But the SAFE
+ thing to do is to set them in the startup file.
+
+ THIS FEATURE IS EXPERIMENTAL...
+18Jan23=======================================================================
+1. Make sure than *every* channel has an IP address. Thank you (I think) Kin
+ for pointing out that PC92 A records were not going out with IP addresses.
+ I'm guessing that other things (like spots) had a similar problem.
+15Jan23=======================================================================
+1. Fix strange errors for carp on missing route_*_cache files on startup.
+14Jan23=======================================================================
+1. Fixed route PC11 promotions so that a new PC61 is actually generated and
+ also sent instead of the original PC11 (to PC61 capable nodes).
+13Jan23=======================================================================
+1. Periodically store Routing tables and, if they are young enough (def: 3hrs)
+ autotically restore them on restart of the node. This will short circuit
+ the need to rebuild the routing tables from scratch on every restart -
+ which is normally for something like software update.
+2. Fix pc11 debugging stats with the correct figures. Sigh... Also move some
+ of the totals to a different place.
+3. Add show/spotstats command which gives the current spot statistics shown
+ during pc11 debugging (which means you don't need to set/deb pc11 unless
+ you really want that extra noise).
+12Jan23=======================================================================
+1. Regularise 'set/debug pc11' output to track all the routes through PC11 and
+ PC61 processing and statistics.
+11Jan23=======================================================================
+1. Improve (?) the PC11 -> PC61 upgrading process that delays incoming PC11s
+ for a very short time in the hope that a PC61 will come in to be used
+ instead. It will also upgrade a PC11 if we have an uptodate IP address
+ that has come in from the routing system PC92s. do a 'set/debug pc11' to
+ see it in action.
+2. I have chosen a definitive list of TOR exits and relays which can be
+ downloaded from http://www.dxspider.net/download/badip.torexit,
+ http://www.dxspider.net/download/badip.torrelay and finally, for those IP
+ addresses that are deemed to be 'bad':
+ http://www.dxspider.net/download/badip.global. I have added the following
+ lines to my /spider/local_cmd/crontab:
+
+24 * * * * spawn('cd /spider/local_data; wget -qN http://www.dxspider.net/download/badip.torexit')
+24 * * * * spawn('cd /spider/local_data; wget -qN http://www.dxspider.net/download/badip.torrelay')
+24 * * * * spawn('cd /spider/local_data; wget -qN http://www.dxspider.net/download/badip.global')
+25 * * * * run_cmd('load/badip')
+
+ The tor files are downloaded from https://lists.fissionrelays.net/tor/ at
+ 15 minutes past every hour, please would you use some other minute than
+ 23 or 24 to get your own local copies.
+
+ A 'set/debug badip' will show you what is being blocked.
+3. Fix set/badip so that it appends new IP addresses correctly.
+10Jan23=======================================================================
+1. Add baddx on incoming callsign in RBN.
+2. Search for all /spider/local_data/badip.* files to allow more control on
+ which IP addresses are detected. e.g. badip.torexit, badip.torrelay as well
+ as baddx.local. The suffixes, apart from .local (created by set/badip) are
+ completely arbitrary. You can use whichever suffix name you like. This is
+ a more useful arrangement for the ever increasing sources of "bad ip
+ addresses" that we need to deter.
+
+ NOTE: all badip.<suffix> are read only EXCEPT badip.local (which can be
+ altered in real time by the sysop using set/badip <ip address> ...).
+ If one uses periodic crontab jobs to update any other badip.<suffix>
+ files from web resources then don't forget to 'load/badip' afterwards.
+3. Add a /spider/data/baddx.issue file which can be copied to (or used as a
+ basis to create) /spider/local_data/baddx
+09Jan23=======================================================================
+1. Finish implemention of DXCIDR ip address filtering. This works on both
+ logins (treated the same as locked out - i.e. just disconnected) and also
+ with PC61s where these sentences are just dropped. Also attempt to prevent
+ any *following* PC11s with the same data getting through.
+
+ YOU WILL NEED either 'cpanm Net::CIDR::Lite' or debian/ubuntu based distros
+ 'apt install libnet-cidr-lite-perl'. RedHat based systems will have similar
+ packages available.
+
+2. Recognise PC18s coming from CC Clusters more nicely.
+04Jan23=======================================================================
+1. Fillout DXCIDR, attach checks in PC61 and logins. Login that fail will
+ simply disconnect, like locked out callsigns
+2. Fix qrz.com URL in stock Internet.pm.
+3. Fix DXHash issues (baddx, badnode, badspotter etc)
+30Dec22=======================================================================
+1. Add more BadWords (regex) checks.
+01Dec22=======================================================================
+1. Re-add some debugging to see which incoming PC protcol sentences are
+ being dumped because of any bad content (words or calls) if debugging
+ option 'nologchan' is set.
+2. All debugging for badwords has the word 'badword' in it. So that one can
+ do a 'grepdbg -1 badword' and one should see who said it and all of what
+ they said. Watchdbg will also work with this.
+3. My latest recommended additions to the badword list includes:
+ RU STOP SPOT DISGRACE CRM CRIM CRUD SUBHUMAN FASCIST DEGENERATE CATTLE CNT
+23Nov22=======================================================================
+1. The BadWord system has been rewritten. This change is pretty radical and
+ needs to be used with care as a word that is entered will be reduced to the
+ minimum sized string needed to match that word.
+
+ This is effective a sysop command changeable version of the file badw_regex
+ but in a much more sysop friendly form. The (un)set/badword <word>...
+ commands now update the /spider/local_data/badword.new file in real time
+ without having to mess about with editing files and running load/badword.
+ load/badword still works, but you should now never need it.
+19Nov22=======================================================================
+1. "Fix" Badword detection in spots and announces etc.
+ NOTE: setting $DXCommandmode::maxbadcount to 0 (default 3) will disable
+ autologout on swearing. But any (detected) swearing will be logged.
+ User generated swearing will not be propagated, not will incoming
+ protocol.
+2. Fix is_ipaddr so that it accepts more (most?) valid IPV6 addresses that
+ actual real IP addresses that might appear as endpoints.
+3. Add CTY-3237 prefixes.
+06Jun22=======================================================================
+1. Fix show/wcy so that it works even if no wcy broadcasts have been received
+ this month.
+01Jun22=======================================================================
+1. Fix show/(v)hfstats commands.
+2. Add CTY-3221 Prefixes.
+07Feb22=======================================================================
+1. (re)add set/seeme and show/seeme commands which don't appear to have
+ "taken".
+03Feb22=======================================================================
+1. Improve/add the help text for grepdbg and watchdbg.
+31Jan22=======================================================================
+1. Set $RBN::respottime to 3 minutes. NOTE change of name from
+ $RBN::minspottime.
+2. Allow callsigns connected to a node with an SSID to get SEEME RBN spots
+ for their base callsigns. So G1TLH-2, G1TLH-7 will get **SEEME** RBN spots
+ if G1TLH is spotted on the RBN system - even is are both connected to the
+ node and G1TLH is not. Each call is unique and each callsign (variation)
+ must have done a set/seeme + set/skimmer. .
+29Jan22=======================================================================
+1. Implement RBN set/seeme which displays any passing RBN spots for your
+ callsign in "raw" format.
+28Jan22=======================================================================
+1. Add Capabilities Line to logged in users.
+2. Make absolutely sure that all DB_Files are closed correctly.
+3. Introduce (un)set/debug rbnchan to control the visualisation of raw RBN
+ input lines.
+25Jan22=======================================================================
+1. Fixed grepdbg so that it does what -help says it does.
+24Jan22=======================================================================
+1. Change Local::pcprot() calling conventions to be the same as the rest of
+ software (see any 'sub handle_nn' function in DXProtHandle.pm.
+2. Corrected some interpolations in Messages.
+3. Tried to make users file deletes and updates more stable.
+22Jan22=======================================================================
+1. Refine spot display to retain the ':' and also reduce the likelyhood of
+ of misalignment by using the available space in the 39 characters more
+ intelligently. This is WIP.
+21Jan22=======================================================================
+1. Fix version tracking in PC92
+2. Bring the major parts of master and mojo together to make maintenance a
+ bit less challenging. This does add anything extra to mojo, but it's still
+ quite a bit of work.
+18Jan22=======================================================================
+1. fix (un)set/registered.
+09Jan22=======================================================================
+1. Add the New Year CTY-3201 prefix data.
+08Jan22=======================================================================
+1. I don't think there are any more warnings...
+06Jan22=======================================================================
+1. Fix the last of any frequent warnings caused by yesterday's modifications.
+05Jan22=======================================================================
+1. Mark nodes that send PC92 K records as spider. These will include VE7CC
+ nodes. NOTE: there appear to be user records marked as user or other sorts
+ of node, which (now) are actually spider (compatible) nodes and will be
+ marked accordingly.
+2. Adjust nodes currently marked as spider nodes, but are sending versions
+ not in the spider range of versions on PC92 A records as AK1A.
+3. Try to undo some damage where users have been autocreated with similar
+ attributes as nodes (locked out with privilege set to 1). This will
+ slowly fix this problem over time, but see item 4 for a 'big bang'
+ approach.
+4. It has come to my attention that there are a large number of users (of
+ all sorts) that have incompatible SSIDs. See 03Jan22/4 for details.
+
+ These are now being scrubbed out of the users file and also will present
+ as their normalised selves. If a -0* SSID is encountered then, if the
+ normalised version of that call is not present, it will be renamed to
+ that normalised call. If the normalised version of that user record is
+ already present, the un-normalised user record (-0*) will be removed.
+5. Make export_users do a batch clean (as in 3. above) and also get rid of
+ (default) 12+ year old unaccessed user records and (default) 2+ year old
+ "empty" records (with no qra/latlog/qth or handle).
+
+ NOTE: if you do an manual export_users (as opposed to the automatic one
+ done once a week), do not be alarmed by the number of old (i.e. more than
+ 12 years old) callsigns that it will get rid of. In my case it was about
+ ~2/5th of the users file. Still left me with over 100,000 "active" users.
+
+ In you are a bit twitchy about this, the code will copy the current
+ user_json and user_json.ooooo to user_json.keep and user_json.backstop
+ respectively. These files will never be overwritten unless you remove one
+ or both, when they will be regenerated on the next export_user.
+04Jan22=======================================================================
+1. Fix issue in the RBN (and probably other places) with callsigns that
+ contain trailing / in callsigns like: OH0K/6, K2PO/7 etc.
+2. Regard strange callsigns like DR4W-HB (seen in skimmer spots) as invalid.
+ This *should be* something like HB9/DR4W or (spit) DR4W/HB9.
+3. Fix the (probably) spurious locking out of users that are unknown to this
+ node, that come in from other nodes. These create new user records which
+ where then automatically locked.
+03Jan22=======================================================================
+1. Allow overrides (on modern versions of perl) with things in DXVars.pm, such
+ $clusterport. This is really only of use for people trying to run more than
+ one instance of DXSpider on the same machine.
+2. Fix who command to make RBN connections as RBN and not USER.
+3. Prevent other nodes claiming that $myalias or $mycall is a different type
+ (user or node) from changing our route table and thence the user type.
+4. Normalise callsigns of incoming connections to G1TST if G1TST-0 or G1TST-00
+ amd G1TST-2 if G1TST-02. There are 800+ instances of callsigns with extra
+ 0 characters in the SSID in my users file. Allow SSIDs up to 99.
+02Jan22=======================================================================
+1. Fix dx bug introduced to handle dx by ip <ipaddr> for webclusters.
+2. Remove _add_thingy dbg message from general view.
+3. Increase sh/ann cache to 130 to cope with some client programs' profligate
+ need to fill their users' screens with needless erm... data.
+01Jan22=======================================================================
+1. Fixed missing ';' in spoof
+30Dec21=======================================================================
+1. Try to fix (nuke) tabs in comment field.
+28Dec21=======================================================================
+1. Fix issues with wwv & wcy updates into the cache.
+2. Add a simple sh/announce cache to stop spawning when used (especially) in
+ a script.
+27Dec21=======================================================================
+1. Fix "kwalitee control" issue in /spider/cmd/dx.pl.
+2. Make sh/wcy and sh/wwv cacheble for simple queries.
+3. Fix set/nodxxxxxx etc.
+22Dec21=======================================================================
+1. Add the possibility to do "sender verify" that spots (and later on, other
+ things like announce etc) are coming from spotters that exist on the node
+ that the PC11/61 says it does. At the moment, this is advisory as many
+ users seem to be on web clusters and many of them do not send PC61 and also
+ don't update the node's routing tables that normal node usage would do.
+2. Prioritise PC61 spots over PC11 spots wherever possible. Also promote PC11
+ spots to PC61 where the spotter's IP is known.
+3. Add a shell script "perlbrew-dxspider" that can be used for starting the
+ node when using perlbrew (i.e. more modern perls) on older distros such
+ as Centos 7/8.
+4. Allow to "dx by <spotter> ip <ip address> <qrg> <spot call> <comment>.
+ This is added as a convenience measure for webcluster authors while waiting
+ for the internal websocket interface (or because they cannot use that).
+ This can only be used by suitably privileged users.
+ This likely to be a temporary measure and will be immediately withdrawn if
+ there is any sign of abuse.
+11Dec21=======================================================================
+1. Removed dependency on perl Git package.
+2. Force code to change directory to $DXSPIDER_ROOT || '/spider'. Also force
+ git -C $root describe.
+3. Change git commands AGAIN to work on old perls (which mojo won't work on,
+ but it makes my life easier).
+4. Change version strings to include the perl version in use.
+10Dec21=======================================================================
+1. Fix the output of set/dxgrid, set/usstate and set/dxcq or set/dxitu to how
+ it was always supposed to be since 2003.
+09Dec21=======================================================================
+1. Moved isregistered to DXChannel for safety...
+08Dec21=======================================================================
+1. Get to the bottom of why ssid'd calls were using the base call's
+ credentials. This should now work as expected.
+06Dec21=======================================================================
+1. Improve console.pl scrolling. Split long lines (eg on announcements.
+04Dec21=======================================================================
+1. Fix illogicalities in USDB creations and make sure that O_CREAT on tie does
+ NOT encounter an existing file to barf about. Even though it shouldn't.
+ Thanks Howard WB3FFV.
+2. Fixed a typo in show/registered that prevents a list of callsigns being
+ searched for. Got rid of some over complex code. Thanks Fabrizio iZ0UIN.
+3. Fix long line wrapping in console.pl
+03Dec21=======================================================================
+1. Move motd and issue files to local_data if not already there.
+30Nov21=======================================================================
+1. Fix sh/dx with callsigns that have /p or VE/G1TLH in them.
+2. Add unset/ak1a, unset/arcluster aliases and some minimal help for UNSET/
+ SPIDER, NODE, ARCLUSTER, AKIA and also SET/USER.
+26Nov21=======================================================================
+1. *Really* change spot display format and sh/dx format "back the way they
+ were. But They won't stay that way for long!!!! There are four (yes, count
+ them) completely unused spaces at the end of the line!!!!
+25Nov21=======================================================================
+1. Restored spot format back to the way it was.
+2. Added CTY-3127 prefixes
+24Nov21=======================================================================
+1. Make /spider/perl/create_sysop.pl work on a base git cloned tree.
+2. Add /spider/local /spider/local_cmd /spider/local_data to git.
+3. Make user supplied console width (set/width) sticky.
+19Nov21=======================================================================
+1. Add the ability of console.pl to resize and have dx spots also resize so
+ that the comment field's size will increase accordingly. Sh/dx amd sh/mydx
+ also work similarly. If you make the console width smaller or wider then
+ spots already on the screen will not resize.
+2. For client programs that parse the output (why? use set/ve7cc to get more
+ easily machine parsed output), a new command "set/width <width>" is
+ available.
+18Nov21=======================================================================
+1. Add a dx spot cache which will store 2 days worth (configurable) of spots.
+ This is then used by the vast majority of simple sh/dx queries without
+ using a sub process. A simple sh/dx is any sh/dx that does not have an
+ "expression" such as "sh/dx g1tlh". So "sh/dx", "sh/dx 200", "sh/mydx" will
+ always run in the main program via the cache. Certain client programs (no
+ names, no pack drill) will notice the difference immediately. It's also a
+ bit snappier.
+2. set/badspotter or set/badnode will work for RBN input.
+3. RBN input with invalid QRGs will be dropped with a nice friendly message
+ rather than a scary exception message about multiplication thrown by one
+ of the libraries that I use.
+4. Added CTY-3126 prefixes.
+5. Silently ignore self spotting below 1240000Khz (change
+ $Spot::minselfspot to 0 to disable or some other freq in Khz). NOTE this
+ will appear as a spot to the spotter's feed, but will not be passed on.
+15Jul21=======================================================================
+1. Fix long standing bug in crontabs that commands with " rather than ' around
+ strings could cause those commands not to work.
+2. Add CTY 3117 prefixes.
+15Mar21=======================================================================
+1. Fix DXUtil::localdata so that it does actually prefer the latest version
+ of a file regardless of whether it is in /spider/data or /spider/localdata.
+2. Add CTY-3105 prefixes.
+05Mar21=======================================================================
+1. Fix DXCron::spawn_cmd so that more than one spawn_cmd can be active at a
+ time. This affects many nodes where lots of regular spawn_cmds happen at
+ a time and this prevented crucial things like 'start_connect' from working!
+2. Sort user lockout issues to cope with all likely scenarios - including
+ "phantom" SSID working.
+3. Fix "close_gracefully" error messages which are useless, but benign. These
+ happen (mainly) when outgoing connects timeout.
+24Feb21=======================================================================
+1. modify RBN timing arrangements
+
+ There are two new twiddle pots:
+
+ $limbotime (default 5*60 secs) which allows potential spots to
+ hang around longer to allow $minqual *different* skimmers to spot
+ them. At which point they are emitted. This is to catch the "slow
+ burning" spots where a call is spotted by different skimmers but
+ separated in time by more than $dwelltime - in fact maybe by
+ minutes.
+
+ $maxqual (default 9 skimmers) which short circuits the normal
+ hard maximum $quality (9 skimmer spots) and $dwelltime (10secs)
+ to allow a spot that has $maxqual *different* skimmers (usually
+ as the result of a burst from the RBN) to defeat $dwelltime and
+ be emitted the moment that condition is satisfied. There maybe
+ a better name for this.
+21Feb21=======================================================================
+1. add data section on 160m in bands.pl.
+07Sep20=======================================================================
+1, Change interface to watchdbg & grepdbg slightly so that multiple search
+ regexes are ANDed rather than ORed together. ORing is easily achieved
+ already by the usual regex pattern 'PATT..|PATT..|..' whereas ANDing could
+ not be done as easily without resorting lots of 'PATT.*PATH' things which
+ would not necessarily get what was wanted.
+2. Make sure that the pc92 C record only contains nodes and users and not
+ other extranoeus things like skimmers...
+15Aug20=======================================================================
+1. Simplify the skimmer scoring mechanism.
+13Aug20=======================================================================
+1. Improve the (displayed) RBN frequency weighting the skimmers' frequencies
+ w.r.t majority view on each spot. Any skimmer that disagrees with a
+ gets docked a 'point' from their 'good' score (if any) and have a 'point'
+ added to their 'bad' score. This score is then taked into account during
+ the 'choosing the real frequency' process. If a skimmer agrees with the
+ majority the process is reversed. To see this in action: set/deb rbnskim
+06Aug20=======================================================================
+1, Add CTY-3013 Prefixes
+2. Make RBN more efficient. Start the process of skimmer node performance
+ caching. Add minimum quality allowed (at 2), which will be overrideable.
+ The format of the rbn_cache has changed and so a full restart will occur.
+3. Collect channel input/output stats. New command: show/data_stats to show
+ them.
+4. Add local::lib qw{/spider/perl5lib} to store cpanm loaded modules just for
+ DXSpider. This is done so that updates needed by future changes can be done
+ as the sysop user and doesn't have to be done as root. This paves the way
+ for UPDATE.pl which will pull down new modules that it needs automatically.
+ When it's written, which will be soon.
+29Jul20=======================================================================
+1. Add show/rbn command that allows one to see who is online and configured
+ for RBN. See help sh/rbn for details.
+2. Fixed issue with set/ve7cc mode adding unwanted newlines to output.
+3. Attempted to make the QRG normalisation more "intelligent". This would be
+ sooooooooo much easier if skimmers were calibrated more accurately.
+4. Added CTY-3012 prefixes.
+23Jul20=======================================================================
+1. fix Filtering so that it does less of what it was doing badly, thus
+ the perl interpreter to do more of the work. Which means you *should* be
+ able to define more or less any allowable operation for that type of
+ filter. Which is a complicated way of saying: you can now use brackets.
+2. Regex expressions now appear as readable text in sh/filter.
+3. Make the RBN status file JSON, rather than DD format. In future versions
+ it will also be VERSIONed to enable more features to be added
+ transparently.
+4. Add show/rbn command to allow sysops to see who is using rbn.
+11Jul20=======================================================================
+1. Fix (embarrassing) gratuitous '#'s in bands.pl
+2. IMPORTANT: certain PC92 strings can crash DXSpider if the nologchan debug
+ category is set (which reduces debug files to sizes that non-developers can
+ reasonably store (especially during big contests)),
+10Jul20=======================================================================
+1. Fix console.pl permissions problem when running as another user not in
+ debug mode. Console.pl WILL NOT WORK in debug mode unless it is running
+ as the sysop user.
+08Jul20=======================================================================
+1. "Finish" the RBN system :-)
+2. This includes enabling the coarse selection of spot modes using set/wantrbn
+ with arguments like 'set/wantrbn cw beacon'. This limits your output to
+ just CW, BCN and DXF modes.
+3. The RBN spot is now cached. With a following wind, this means that even a
+ node restart, done in a timely fashion (within a few minutes) will not
+ cause a "cache warmup" delay for users on a restart.
+4. Added the "full fat" set/wantrbn command and aliased it to 'set/skimmer'.
+ I use both terms (whenever I remembered) in the help text.
+5. Help text has been written.
+6. The UPGRADE.mojo file has been tweeked to point out the users file format
+ change.
+7. Merge in users.v3j to the mojo branch.
+8. Add CTY-3011 prefixes.
07Jul20=======================================================================
1. Fix show/node command.
2. Fix show/cluster command to take into account the presence of skimmer nodes
projects / spider.git / blobdiff