4 Aranea Orthogonal Communications Protocol
10 <Origin>,<Group>,<TimeSeq>,<Hop>|<Tag>,<Data>...
14 For many years DX Clusters have used a protocol which was designed
15 for a non-looped tree of nodes. This environment has probably never, reliably,
16 been achieved in practice; certainly not recently.
18 There have always been loops, sometimes bringing the network to its
19 knees. In modern usage, both in order to get some resilience and also
20 to expedite information flow, we use internet based, deliberately
21 looped networks with filtering. Whilst this works, after a fashion, there
22 are all sorts of problems that the current PC protocol can never
26 describes a complete replacement for the PC protocol. It allows a
27 fully looped network, is inherently extensible and should be simple
28 to implement (especially in perl).
30 All implementations of this protocol shall B<only> use this protocol
31 for inter-node communications.
36 designed to be an extensible basis for any type of one too many
37 "instant" line-based communications tasks.
39 This protocol is designed to be flood routed in a meshed network in
40 as efficient a manner as possible. The reason we have chosen this
41 mechanism is that most L</Messages> need to be broadcast to all nodes.
43 Experience has shown that nodes will appear and (more infrequently)
44 disappear without much (or any) notice.
45 Therefore, the constantly changing and uncoordinated
46 nature of the network doesn't lend itself to fixed routing policies.
48 Having said that: directed routing is available where routes have
49 been learned through past traffic.
50 Those L</Messages> that could be routed (mainly single line one to
51 one "talk" L</Messages>)
52 happen sufficiently infrequently that, should they need to be flood routed
53 (because no route has been learned yet) it is a small cost overall.
57 A message is a single line of UTF8 encoded and HTTP escaped text
58 terminated in the standard internet manner with a <CR><LF>.
60 Each message consists of a L</Routing Section> and a L</Command Section>.
61 The two sections are separated with the '|' character.
63 characters (as well as non-printable characters, <CR>, <LF> and
64 a small number of other reserved characters)
65 can only be sent escaped. This is described further in the
66 L</Command Section> and L</Fields>.
68 Most of this document is concerned with the L</Routing Section>, however
69 some L</Standard Commands> which all implementations should issue and
70 must accept are described.
74 In the past messaging applications such as DX Cluster software have maintained
75 a fairly strict division between L</Node>s and L</User>s". This protocol attempts
76 to get away from that distinction by allowing any entity to connect to any
79 Applications that use this protocol are essentially all peers and therefore
80 nodes the only real difference between L</Node>s and L</User>s is that a "node" has one or more
81 listeners running that will,
82 potentially, allow incoming connections, from other L</Node>s, L</Endpoint>s or L</User>s
84 Any application that is a sink and/or source of data for L</Group>s, is capable of obeying
85 the protocol message construction rules and understands how to deduplicate incoming messages
86 correctly can operate as a routeable entity in this protocol. It is called an L</Endpoint>.
88 An L</Endpoint> is called a L</Node> if it accepts connections from L</Endpoint>s and is
89 prepared to route messages on their behalf to other L</Node>s or L</Endpoint>. In addition it
90 may provide some other, usually simpler, interface (eg simple telnet access) for direct user access.
92 The concept of an L</Endpoint> has been invented because modern clients are
93 capable of being intelligent than simple
94 character based connections such as telnet or ax25. They wish to be able to
95 distinguish between the various classes of message, such as: DX spots,
96 announces, talk, logging info etc. It is a pain to have to do it, as now,
97 by trying to make sense of the (slightly different for each piece of node
98 software) human readable "user" version of the output. Far better to pass on
99 regular, specified, easily computer decodable versions of the message,
100 i.e. in this protocol, and leave
101 the human presentation to the application implementing the L</Endpoint>.
103 It also helps to modularise the various interfaces that may be implemented such
104 as the legacy, character based connections of existing PC protocol based nodes.
105 They should be treated
106 as local clients, in fact as L</User>s, B<not> as peers in this protocol. It is likely that, in order
107 to do this, some extra L</Tag>s will need to be defined at application level.
109 =head1 Connection Types
113 A L</User> is a connection to a L</Node> (that allows such connections)
114 that does not occur in protocol. All L</User>s shall be identified with a name
115 of up to 12 characters in the set [-0-9A-Z_]. All messages have to be routed via the
116 L</Node> to which this L</User> is connected.
120 An L</Endpoint> is a connection to a L<Node> that uses the protocol. From a routing point of
121 view, it is indistiguishable from a L</Node>. The L</Endpoint> is responsible for creating and decoding
122 well formed protocol messages. An L</Endpoint> does not route beyond the immediate L</Node>(s) to
123 which it is connected. It may also be a L</Service> connected to a L</Node> which provides some
124 addressable service that can be queried.
128 A L</Node> is connected to other L</Node>s. It is responsible for routing messages in protocol
129 from other L</Node>s or L</Endpoint>s, whether directly connected or not. Optionally, a L</Node>
130 may provide other interfaces, such as direct L</User> connections or legacy PC protocol speaking
133 =head1 Routing Section
135 The application that implements this protocol is essentially a line
136 oriented message router. One line equals one message. Each line is
137 effectively a datagram.
139 It is assumed that nodes are connected to
140 each other using a "reliable" streaming protocol such as TCP/IP or
141 AX25. Having said that: in context, L</Messages> in this protocol could be
142 multi/broadcast, either "as is" or wrapped in some other framing
145 Although the physical transport between L</Node>s is reliable, the actual message
146 is unreliable, because this is an unreliable, best effort, "please route my packets
147 through your node" protocol. There is no guarantee that a message
148 will get to the other side of a mesh of nodes. There may be a
149 discontinuity either caused by outage or deliberate filtering.
151 However, as it is envisaged that most L</Messages> will be flood routed or,
152 in the case of directed L</Messages> (those that have L</Group> and/or
153 L</ToUser> fields) down some/most/all interfaces showing a route for that
154 direction, it is unlikely that L</Messages> will be lost in practice.
156 Assuming that there is a path between all the L</Node>s in a network, then it is guaranteed
157 that a message will be delivered everywhere, eventually. It is possible (indeed likely) that
159 will arrive at L</Node>s more than once. L</Node>s are responsible for deduplicating those messages
160 using the information in the L</Routing Section>.
162 =head2 Field Description
164 The first four fields in the L</Routing Section> are compulsory. However,
165 a client connection can
167 Adding a L</Group> and/or L</ToUser> field will restrict the destinations
168 or recipients that receive this message.
170 The L</Hop> field is incremented on receipt of a message on a node.
172 Fields are separated by the comma ',' character with the last field
173 required followed by the vertical bar '|' character.
175 If trailing fields are missed out then superfluous commas can also
176 be left out. If intervening fields are missing then no space needs
177 to be left for the separating comma.
179 The characters allowed in the routing section are restricted. Any
180 invalid characters in any field will cause the whole message to be
183 More detailed descriptions of the fields follow:
189 This is a compulsory field. It is the name of the originating node.
190 The field can contain up to 12 characters in the set [-A-Z0-9_/] in
191 any order. Higher layers may restrict this further.
193 The field must not be changed by any other node.
197 This is the Group (or Channel) to be used for this data. It is compulsory. There
198 is always a L</Group>
200 It is a string of up to 12 characters
201 in the set [-A-Z0-9_] in any order. Optionally, for extra routing to
202 a specific end point (node or user), it may have another 12 character
203 field in the same set, concatenated with the string, separated by a ':'
206 This field is used either to indicate particular node destination
207 or to differentiate this broadcast in some way by making this
208 message as a member of a L</Group>. Any message can be sent
209 down any L</Group>. The names of L</Group>s and their usage
210 is entirely up to the implementor.
212 It is assumed that node names can be differentiated from user
213 names and L</Group> names.
215 If the field is set to a particular node destination, it will
216 be routed (rather than broadcast) to that node. However, any
217 intervening nodes are free to duplicate the message and send
218 it down more than one, likely looking, interface - depending on any
219 network policies that may pertain.
223 This is a compulsory field. It is a 10 hexadecimal digit string which
224 consists of a day no (1-31),
225 a flag to indicate NTP syncronisation in use,
226 seconds within that day (0-86399) [total of 6 hex digits]
227 that are concatenated with a sequence number (0-65535)
228 [4 hex digits] making the total of 10 hexadecimal digits.
230 The date portion is constructed as:
232 my $date = ((((gmtime)[3] < 1) | $ntpflag) < 18) | (time % 86400);
234 The sequence number is simply an unsigned short (or 16 bit) number
237 Each message originated at this node will increment the sequence
242 This is a compulsory field. It is the number of hops from the
243 originating node. It is incremented immediately on receipt and
244 before determining its value.
246 So the originating node sends a message with a L</Hop> of 0, the
247 neighbouring nodes must increment this field before passing
248 it on to higher layers for onward processing.
250 Implementations may have an upper limit to this field and may
251 silently drop incoming L</Messages> with a L</Hop> count greater than the
260 It is assumed that nodes will be connected in a looped network with
261 more than one route available (in many cases) to another node.
263 In anycase, most traffic is not directed, but broadcast to all users
266 Each message is uniquely identified by the (L</Origin>,L</TimeSeq>)
267 tuple. The basic system will learn which interfaces can see what nodes
268 by looking at the tuple and merging that with the L</Hop> count.
269 Each interface remembers the latest L</TimeSeq> with the lowest L</Hop>
270 for each L</Origin> that arrives on that interface. It also remembers
271 the number of L</Messages> for that L</Origin> that has been received on
274 Any message for onward broadcast is duplicated and sent out on all
275 interfaces that it did not come in on.
277 Any message that is directed to a particular node will be sent out on
278 the "best" interface based on routing information gathered so far. If there
279 is more than one possible route then, depending on network or local
280 policy, the message may be duplicated and sent on other interfaces
285 On receipt of a message, its unique tuple (L</Origin>,L</TimeSeq>) is
286 checked against a hash table. If it exists: the message is silently
287 dropped. If it does not exist in the hash table then the tuple is
290 The hash table is periodically cleaned, removing tuples that
291 have expired. The length of time a tuple remains in the hash table
292 is implementation dependant but could easily be several days, if
295 This mechanism only ensures that a message broadcast around the network
296 travels the least distance and through the fewest nodes possible. It
297 is up to higher layers to make sure that data carried is not, itself,
302 # on link startup from GB7BAA (both sides hello)
303 GB7TLH,ROUTE,3D02350001,0|HELLO,Aranea,1.2,24.123
304 GB7BAA,ROUTE,3D02355421,1|HELLO,Aranea,1.1,23.245
306 # on user startup to GB7TLH
307 GB7TLH,ROUTE,3D042506F2,0,G1TLH|HELLO,PClient,1.3
309 # on user disconnection
310 GB7TLH,ROUTE,3D9534F32D,0,G1TLH|BYE
312 # a talk (actually 'text') message to a user (some distance away
313 # from the origin node)
314 GB7TLH,G8TIC,3D03450019,3|T,G1TLH,Hiya Mike what's happening?
316 # a talk/chat/text message to a Group
317 GB7TLH,VHF,0413525F23,2|T,G1TLH,2m is opening on MS
319 # a ping to find the whereabouts and distance of a user from a node
320 # the hex number on the end is the ping ID
321 GB7TLH,G7BRN,1512346543,0|PING,G1TLH,9F4D
323 # this effectively asks whether the user is on-line on a particular node
324 GB7TLH,GB7BAA:G7BRN,1512346543,0|PING,G1TLH,35DE
326 # A possible reply, same ID as ping followed by the no of hops on the
327 # ping that was received thus telling you how far away it is.
328 GB7BAA,G1TLH,1512450534,3|PONG,G7BRN,35DE,3
331 =head1 Command Section
333 The L</Command Section> of the message contains the actual data being
334 passed. It is called the Command Section because all commands
335 are identified with a L</Tag> each of which is implemented by
336 the software using this protocol. Each </Tag> (usually) is followed by one
341 The L</Tag> consists of string of uppercase letters and digits, starting
342 with a leading, uppercase, letter. Tags should be as short as is meaningful.
350 Invalid tags include:
356 The L</Tag> is separated from its data L</Fields> by a comma ','.
361 in any subsequent data shall be separated by a comma ','.
363 be HTTP encoded such that reserved characters (comma ',',
367 and non printable characters less than 127 (or %7F in hex)
368 [including newline and carraige return] are translated to
369 their two hex digit equivalent preceeded by the percent '%' character.
373 "%0D%0A" is "<carriage return><linefeed>".
374 "hello%2C there" is "hello, there"
376 This is not standard CSV, fields are not quoted (delimited with either
379 All national characters above 127 are UTF8 encoded in the
380 standard perl 5.8.x way. It follows that all (perl) programs that
381 are written according to this specification must say:
385 A message (or line) is terminated with <carriage return><linefeed>
386 0x0d 0x0a. Incoming L</Messages> must be accepted even when terminated
387 with just <linefeed>.
389 Care must be taken to make sure that fields have any reserved characters
390 encoded. In particular: it is perfectly permissible to have <linefeed>
391 characters in a field - so long as they are escaped.
393 Fields come in two styles: either simple fields (just containing
394 data) or B<key>=B<value> pairs. Each pair must be separated from
395 the next by a comma ','. The B<key> must consist of the set of
396 characters [a-z0-9_] (ie lowercase letters, digits and underscore),
397 with a leading letter. The B<value> must be HTTP encoded as
398 specified above and can otherwise contain any character.
400 There is no maximum size specified for a message. It is up to each
401 implimentation to enforce one (if only for their own protection).
403 =head2 Standard Commands
405 There are a number of L</Standard Commands> which must be accepted by
412 HELLO,<software name>,<version>,<build>,<comments>
414 Command sent on connection to another node. Both sides send their information
415 to the other. All the possible arguments are optional, although some of the
416 arguments should be sent in order to help diagnose problems. This command is
423 Command sent to all connections when the software is shutting down. This is sent
424 by the node just before shutdown occurs. This is really only used to help the
425 network prune its routing tables. It isn't a requirement. The <comment> field
430 DISC,<node name>,<comments>
432 Command sent when a node has disconnected from this node. This message is sent when
433 an interface shuts down. It need not be sent if a L<BYE> from an interface for
434 that node has just been received. This command should be broadcast.
436 The <node name> is mandatory and is the name of the interface that has just
441 PING,<user>,<ping id>
443 Command to send a ping to a node or user. This command is used both by the software
444 and users to determine a) whether a node or user exists and b) how good the path is
447 The <ping id> is a unique string which is usually the hexadecimal equivalent of an
448 integer that is incremented every time it is used. But it can be anything that
449 will identify this ping using the tuple (L<Origin>,<ping id>) as unique.
453 PONG,<ping id>,<user>,<no of hops on ping>
455 Command to reply to a ping. This is sent as a reply to an incoming ping command.
456 The <ping id> is the one supplied and the <no of hops on ping> is the number of
457 hops it took for the ping to arrive.
463 All implementations must be able to send "text" (encoded as specified in
464 L</Fields>). There would be little point in doing all this otherwise!
470 Dirk Koopman, G1TLH, E<lt>djk@tobit.co.ukE<gt>
472 =head1 COPYRIGHT AND LICENSE
474 Copyright 2004-2005 by Dirk Koopman, G1TLH
476 This library is free software; you can redistribute it and/or modify
477 it under the same terms as Perl itself.