X-Git-Url: http://www.dxcluster.org/gitweb/gitweb.cgi?a=blobdiff_plain;f=Changes;h=ce05214ad2eea99e5c9140180ad30bd2281a7055;hb=refs%2Fheads%2Fstaging;hp=64efdc0007213128ec3b37d784b06a977a171804;hpb=cb319a377cdf61f610b57bb1336eba1b5873db2b;p=spider.git diff --git a/Changes b/Changes index 64efdc00..ce05214a 100644 --- a/Changes +++ b/Changes @@ -1,3 +1,386 @@ +20Jan23======================================================================= +1. Add the variable @main::localhost_names to allow other IP addresses to + be treated in the same way as localhost in item 1 on 19Jan23 below. NOTE + you must include ALL the normal localhost names + any other interface + names that you might want to include: + + set/var @main::localhost_names qw(127.0.0.1 ::1 192.168.1.30) + + using the qw() construction is easier than: + + set/var @main::localhost_names ('127.0.0.1', '::1', '192.168.1.30') + + but either will work. You can define as many IP addresses as you like and + they can be IPV4 or 6. + + You do NOT need to fiddle with this unless you specifically have more + than just the normal definitions of localhost. So for 'normal' nodes with + one external interface, you DO NOT NEED TO DO ANY OF THIS. +19Jan23======================================================================= +1. Introduce aliasing for localhost in DX Spots and outgoing PC92 A records + on login. There are two variables which can be set with the alias to use: + $main::localhost_alias_ipv4 + $main::localhost_alias_ipv6 + These can be set in the /spider/scripts/startup, but this is only + necessary if the node has more than one interface, or virtual hosts. If + there is ONLY ONE ipv4 and/or ipv6 IP address on the node machine then + these variables will be automatically populated on first use. But the SAFE + thing to do is to set them in the startup file. + + THIS FEATURE IS EXPERIMENTAL... +18Jan23======================================================================= +1. Make sure than *every* channel has an IP address. Thank you (I think) Kin + for pointing out that PC92 A records were not going out with IP addresses. + I'm guessing that other things (like spots) had a similar problem. +15Jan23======================================================================= +1. Fix strange errors for carp on missing route_*_cache files on startup. +14Jan23======================================================================= +1. Fixed route PC11 promotions so that a new PC61 is actually generated and + also sent instead of the original PC11 (to PC61 capable nodes). +13Jan23======================================================================= +1. Periodically store Routing tables and, if they are young enough (def: 3hrs) + autotically restore them on restart of the node. This will short circuit + the need to rebuild the routing tables from scratch on every restart - + which is normally for something like software update. +2. Fix pc11 debugging stats with the correct figures. Sigh... Also move some + of the totals to a different place. +3. Add show/spotstats command which gives the current spot statistics shown + during pc11 debugging (which means you don't need to set/deb pc11 unless + you really want that extra noise). +12Jan23======================================================================= +1. Regularise 'set/debug pc11' output to track all the routes through PC11 and + PC61 processing and statistics. +11Jan23======================================================================= +1. Improve (?) the PC11 -> PC61 upgrading process that delays incoming PC11s + for a very short time in the hope that a PC61 will come in to be used + instead. It will also upgrade a PC11 if we have an uptodate IP address + that has come in from the routing system PC92s. do a 'set/debug pc11' to + see it in action. +2. I have chosen a definitive list of TOR exits and relays which can be + downloaded from http://www.dxspider.net/download/badip.torexit, + http://www.dxspider.net/download/badip.torrelay and finally, for those IP + addresses that are deemed to be 'bad': + http://www.dxspider.net/download/badip.global. I have added the following + lines to my /spider/local_cmd/crontab: + +24 * * * * spawn('cd /spider/local_data; wget -qN http://www.dxspider.net/download/badip.torexit') +24 * * * * spawn('cd /spider/local_data; wget -qN http://www.dxspider.net/download/badip.torrelay') +24 * * * * spawn('cd /spider/local_data; wget -qN http://www.dxspider.net/download/badip.global') +25 * * * * run_cmd('load/badip') + + The tor files are downloaded from https://lists.fissionrelays.net/tor/ at + 15 minutes past every hour, please would you use some other minute than + 23 or 24 to get your own local copies. + + A 'set/debug badip' will show you what is being blocked. +3. Fix set/badip so that it appends new IP addresses correctly. +10Jan23======================================================================= +1. Add baddx on incoming callsign in RBN. +2. Search for all /spider/local_data/badip.* files to allow more control on + which IP addresses are detected. e.g. badip.torexit, badip.torrelay as well + as baddx.local. The suffixes, apart from .local (created by set/badip) are + completely arbitrary. You can use whichever suffix name you like. This is + a more useful arrangement for the ever increasing sources of "bad ip + addresses" that we need to deter. + + NOTE: all badip. are read only EXCEPT badip.local (which can be + altered in real time by the sysop using set/badip ...). + If one uses periodic crontab jobs to update any other badip. + files from web resources then don't forget to 'load/badip' afterwards. +3. Add a /spider/data/baddx.issue file which can be copied to (or used as a + basis to create) /spider/local_data/baddx +09Jan23======================================================================= +1. Finish implemention of DXCIDR ip address filtering. This works on both + logins (treated the same as locked out - i.e. just disconnected) and also + with PC61s where these sentences are just dropped. Also attempt to prevent + any *following* PC11s with the same data getting through. + + YOU WILL NEED either 'cpanm Net::CIDR::Lite' or debian/ubuntu based distros + 'apt install libnet-cidr-lite-perl'. RedHat based systems will have similar + packages available. + +2. Recognise PC18s coming from CC Clusters more nicely. +04Jan23======================================================================= +1. Fillout DXCIDR, attach checks in PC61 and logins. Login that fail will + simply disconnect, like locked out callsigns +2. Fix qrz.com URL in stock Internet.pm. +3. Fix DXHash issues (baddx, badnode, badspotter etc) +30Dec22======================================================================= +1. Add more BadWords (regex) checks. +01Dec22======================================================================= +1. Re-add some debugging to see which incoming PC protcol sentences are + being dumped because of any bad content (words or calls) if debugging + option 'nologchan' is set. +2. All debugging for badwords has the word 'badword' in it. So that one can + do a 'grepdbg -1 badword' and one should see who said it and all of what + they said. Watchdbg will also work with this. +3. My latest recommended additions to the badword list includes: + RU STOP SPOT DISGRACE CRM CRIM CRUD SUBHUMAN FASCIST DEGENERATE CATTLE CNT +23Nov22======================================================================= +1. The BadWord system has been rewritten. This change is pretty radical and + needs to be used with care as a word that is entered will be reduced to the + minimum sized string needed to match that word. + + This is effective a sysop command changeable version of the file badw_regex + but in a much more sysop friendly form. The (un)set/badword ... + commands now update the /spider/local_data/badword.new file in real time + without having to mess about with editing files and running load/badword. + load/badword still works, but you should now never need it. +19Nov22======================================================================= +1. "Fix" Badword detection in spots and announces etc. + NOTE: setting $DXCommandmode::maxbadcount to 0 (default 3) will disable + autologout on swearing. But any (detected) swearing will be logged. + User generated swearing will not be propagated, not will incoming + protocol. +2. Fix is_ipaddr so that it accepts more (most?) valid IPV6 addresses that + actual real IP addresses that might appear as endpoints. +3. Add CTY-3237 prefixes. +06Jun22======================================================================= +1. Fix show/wcy so that it works even if no wcy broadcasts have been received + this month. +01Jun22======================================================================= +1. Fix show/(v)hfstats commands. +2. Add CTY-3221 Prefixes. +07Feb22======================================================================= +1. (re)add set/seeme and show/seeme commands which don't appear to have + "taken". +03Feb22======================================================================= +1. Improve/add the help text for grepdbg and watchdbg. +31Jan22======================================================================= +1. Set $RBN::respottime to 3 minutes. NOTE change of name from + $RBN::minspottime. +2. Allow callsigns connected to a node with an SSID to get SEEME RBN spots + for their base callsigns. So G1TLH-2, G1TLH-7 will get **SEEME** RBN spots + if G1TLH is spotted on the RBN system - even is are both connected to the + node and G1TLH is not. Each call is unique and each callsign (variation) + must have done a set/seeme + set/skimmer. . +29Jan22======================================================================= +1. Implement RBN set/seeme which displays any passing RBN spots for your + callsign in "raw" format. +28Jan22======================================================================= +1. Add Capabilities Line to logged in users. +2. Make absolutely sure that all DB_Files are closed correctly. +3. Introduce (un)set/debug rbnchan to control the visualisation of raw RBN + input lines. +25Jan22======================================================================= +1. Fixed grepdbg so that it does what -help says it does. +24Jan22======================================================================= +1. Change Local::pcprot() calling conventions to be the same as the rest of + software (see any 'sub handle_nn' function in DXProtHandle.pm. +2. Corrected some interpolations in Messages. +3. Tried to make users file deletes and updates more stable. +22Jan22======================================================================= +1. Refine spot display to retain the ':' and also reduce the likelyhood of + of misalignment by using the available space in the 39 characters more + intelligently. This is WIP. +21Jan22======================================================================= +1. Fix version tracking in PC92 +2. Bring the major parts of master and mojo together to make maintenance a + bit less challenging. This does add anything extra to mojo, but it's still + quite a bit of work. +18Jan22======================================================================= +1. fix (un)set/registered. +09Jan22======================================================================= +1. Add the New Year CTY-3201 prefix data. +08Jan22======================================================================= +1. I don't think there are any more warnings... +06Jan22======================================================================= +1. Fix the last of any frequent warnings caused by yesterday's modifications. +05Jan22======================================================================= +1. Mark nodes that send PC92 K records as spider. These will include VE7CC + nodes. NOTE: there appear to be user records marked as user or other sorts + of node, which (now) are actually spider (compatible) nodes and will be + marked accordingly. +2. Adjust nodes currently marked as spider nodes, but are sending versions + not in the spider range of versions on PC92 A records as AK1A. +3. Try to undo some damage where users have been autocreated with similar + attributes as nodes (locked out with privilege set to 1). This will + slowly fix this problem over time, but see item 4 for a 'big bang' + approach. +4. It has come to my attention that there are a large number of users (of + all sorts) that have incompatible SSIDs. See 03Jan22/4 for details. + + These are now being scrubbed out of the users file and also will present + as their normalised selves. If a -0* SSID is encountered then, if the + normalised version of that call is not present, it will be renamed to + that normalised call. If the normalised version of that user record is + already present, the un-normalised user record (-0*) will be removed. +5. Make export_users do a batch clean (as in 3. above) and also get rid of + (default) 12+ year old unaccessed user records and (default) 2+ year old + "empty" records (with no qra/latlog/qth or handle). + + NOTE: if you do an manual export_users (as opposed to the automatic one + done once a week), do not be alarmed by the number of old (i.e. more than + 12 years old) callsigns that it will get rid of. In my case it was about + ~2/5th of the users file. Still left me with over 100,000 "active" users. + + In you are a bit twitchy about this, the code will copy the current + user_json and user_json.ooooo to user_json.keep and user_json.backstop + respectively. These files will never be overwritten unless you remove one + or both, when they will be regenerated on the next export_user. +04Jan22======================================================================= +1. Fix issue in the RBN (and probably other places) with callsigns that + contain trailing / in callsigns like: OH0K/6, K2PO/7 etc. +2. Regard strange callsigns like DR4W-HB (seen in skimmer spots) as invalid. + This *should be* something like HB9/DR4W or (spit) DR4W/HB9. +3. Fix the (probably) spurious locking out of users that are unknown to this + node, that come in from other nodes. These create new user records which + where then automatically locked. +03Jan22======================================================================= +1. Allow overrides (on modern versions of perl) with things in DXVars.pm, such + $clusterport. This is really only of use for people trying to run more than + one instance of DXSpider on the same machine. +2. Fix who command to make RBN connections as RBN and not USER. +3. Prevent other nodes claiming that $myalias or $mycall is a different type + (user or node) from changing our route table and thence the user type. +4. Normalise callsigns of incoming connections to G1TST if G1TST-0 or G1TST-00 + amd G1TST-2 if G1TST-02. There are 800+ instances of callsigns with extra + 0 characters in the SSID in my users file. Allow SSIDs up to 99. +02Jan22======================================================================= +1. Fix dx bug introduced to handle dx by ip for webclusters. +2. Remove _add_thingy dbg message from general view. +3. Increase sh/ann cache to 130 to cope with some client programs' profligate + need to fill their users' screens with needless erm... data. +01Jan22======================================================================= +1. Fixed missing ';' in spoof +30Dec21======================================================================= +1. Try to fix (nuke) tabs in comment field. +28Dec21======================================================================= +1. Fix issues with wwv & wcy updates into the cache. +2. Add a simple sh/announce cache to stop spawning when used (especially) in + a script. +27Dec21======================================================================= +1. Fix "kwalitee control" issue in /spider/cmd/dx.pl. +2. Make sh/wcy and sh/wwv cacheble for simple queries. +3. Fix set/nodxxxxxx etc. +22Dec21======================================================================= +1. Add the possibility to do "sender verify" that spots (and later on, other + things like announce etc) are coming from spotters that exist on the node + that the PC11/61 says it does. At the moment, this is advisory as many + users seem to be on web clusters and many of them do not send PC61 and also + don't update the node's routing tables that normal node usage would do. +2. Prioritise PC61 spots over PC11 spots wherever possible. Also promote PC11 + spots to PC61 where the spotter's IP is known. +3. Add a shell script "perlbrew-dxspider" that can be used for starting the + node when using perlbrew (i.e. more modern perls) on older distros such + as Centos 7/8. +4. Allow to "dx by ip . + This is added as a convenience measure for webcluster authors while waiting + for the internal websocket interface (or because they cannot use that). + This can only be used by suitably privileged users. + This likely to be a temporary measure and will be immediately withdrawn if + there is any sign of abuse. +11Dec21======================================================================= +1. Removed dependency on perl Git package. +2. Force code to change directory to $DXSPIDER_ROOT || '/spider'. Also force + git -C $root describe. +3. Change git commands AGAIN to work on old perls (which mojo won't work on, + but it makes my life easier). +4. Change version strings to include the perl version in use. +10Dec21======================================================================= +1. Fix the output of set/dxgrid, set/usstate and set/dxcq or set/dxitu to how + it was always supposed to be since 2003. +09Dec21======================================================================= +1. Moved isregistered to DXChannel for safety... +08Dec21======================================================================= +1. Get to the bottom of why ssid'd calls were using the base call's + credentials. This should now work as expected. +06Dec21======================================================================= +1. Improve console.pl scrolling. Split long lines (eg on announcements. +04Dec21======================================================================= +1. Fix illogicalities in USDB creations and make sure that O_CREAT on tie does + NOT encounter an existing file to barf about. Even though it shouldn't. + Thanks Howard WB3FFV. +2. Fixed a typo in show/registered that prevents a list of callsigns being + searched for. Got rid of some over complex code. Thanks Fabrizio iZ0UIN. +3. Fix long line wrapping in console.pl +03Dec21======================================================================= +1. Move motd and issue files to local_data if not already there. +30Nov21======================================================================= +1. Fix sh/dx with callsigns that have /p or VE/G1TLH in them. +2. Add unset/ak1a, unset/arcluster aliases and some minimal help for UNSET/ + SPIDER, NODE, ARCLUSTER, AKIA and also SET/USER. +26Nov21======================================================================= +1. *Really* change spot display format and sh/dx format "back the way they + were. But They won't stay that way for long!!!! There are four (yes, count + them) completely unused spaces at the end of the line!!!! +25Nov21======================================================================= +1. Restored spot format back to the way it was. +2. Added CTY-3127 prefixes +24Nov21======================================================================= +1. Make /spider/perl/create_sysop.pl work on a base git cloned tree. +2. Add /spider/local /spider/local_cmd /spider/local_data to git. +3. Make user supplied console width (set/width) sticky. +19Nov21======================================================================= +1. Add the ability of console.pl to resize and have dx spots also resize so + that the comment field's size will increase accordingly. Sh/dx amd sh/mydx + also work similarly. If you make the console width smaller or wider then + spots already on the screen will not resize. +2. For client programs that parse the output (why? use set/ve7cc to get more + easily machine parsed output), a new command "set/width " is + available. +18Nov21======================================================================= +1. Add a dx spot cache which will store 2 days worth (configurable) of spots. + This is then used by the vast majority of simple sh/dx queries without + using a sub process. A simple sh/dx is any sh/dx that does not have an + "expression" such as "sh/dx g1tlh". So "sh/dx", "sh/dx 200", "sh/mydx" will + always run in the main program via the cache. Certain client programs (no + names, no pack drill) will notice the difference immediately. It's also a + bit snappier. +2. set/badspotter or set/badnode will work for RBN input. +3. RBN input with invalid QRGs will be dropped with a nice friendly message + rather than a scary exception message about multiplication thrown by one + of the libraries that I use. +4. Added CTY-3126 prefixes. +5. Silently ignore self spotting below 1240000Khz (change + $Spot::minselfspot to 0 to disable or some other freq in Khz). NOTE this + will appear as a spot to the spotter's feed, but will not be passed on. +15Jul21======================================================================= +1. Fix long standing bug in crontabs that commands with " rather than ' around + strings could cause those commands not to work. +2. Add CTY 3117 prefixes. +15Mar21======================================================================= +1. Fix DXUtil::localdata so that it does actually prefer the latest version + of a file regardless of whether it is in /spider/data or /spider/localdata. +2. Add CTY-3105 prefixes. +05Mar21======================================================================= +1. Fix DXCron::spawn_cmd so that more than one spawn_cmd can be active at a + time. This affects many nodes where lots of regular spawn_cmds happen at + a time and this prevented crucial things like 'start_connect' from working! +2. Sort user lockout issues to cope with all likely scenarios - including + "phantom" SSID working. +3. Fix "close_gracefully" error messages which are useless, but benign. These + happen (mainly) when outgoing connects timeout. +24Feb21======================================================================= +1. modify RBN timing arrangements + + There are two new twiddle pots: + + $limbotime (default 5*60 secs) which allows potential spots to + hang around longer to allow $minqual *different* skimmers to spot + them. At which point they are emitted. This is to catch the "slow + burning" spots where a call is spotted by different skimmers but + separated in time by more than $dwelltime - in fact maybe by + minutes. + + $maxqual (default 9 skimmers) which short circuits the normal + hard maximum $quality (9 skimmer spots) and $dwelltime (10secs) + to allow a spot that has $maxqual *different* skimmers (usually + as the result of a burst from the RBN) to defeat $dwelltime and + be emitted the moment that condition is satisfied. There maybe + a better name for this. +21Feb21======================================================================= +1. add data section on 160m in bands.pl. +07Sep20======================================================================= +1, Change interface to watchdbg & grepdbg slightly so that multiple search + regexes are ANDed rather than ORed together. ORing is easily achieved + already by the usual regex pattern 'PATT..|PATT..|..' whereas ANDing could + not be done as easily without resorting lots of 'PATT.*PATH' things which + would not necessarily get what was wanted. +2. Make sure that the pc92 C record only contains nodes and users and not + other extranoeus things like skimmers... +15Aug20======================================================================= +1. Simplify the skimmer scoring mechanism. 13Aug20======================================================================= 1. Improve the (displayed) RBN frequency weighting the skimmers' frequencies w.r.t majority view on each spot. Any skimmer that disagrees with a