Fix and document mlat_config, add universal ULTRAFEEDER_CONFIG parameter and support (#8)

.env

@@ -7,5 +7,6 @@ FEEDER_ALT_M=xx
 FEEDER_ALT_FT=xxx
 FEEDER_NAME=my_feeder_name
 MULTIFEEDER_UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
+ADSBX_UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
 FEEDER_HEYWHATSTHAT_ID=xxxxxxxxxxxx
 FEEDER_HEYWHATSTHAT_ALTS=3000,10000,30000,40000

.hadolint.yaml

@@ -1,5 +1,6 @@
 ignored: 
   - DL3003
+  - DL3007
   - DL3008
   - SC3054
   - SC3044

.markdownlint.json

@@ -0,0 +1,6 @@
+{
+    "MD024": false,
+    "MD013": false,
+    "MD033": false,
+    "MD029": false
+}

README-grafana.md

@@ -129,8 +129,8 @@ docker compose up -d
 
 You should be able to point your web browser at:
 
-* `http://docker.host.ip.addr:9090/` to access the `prometheus` console.
+- `http://docker.host.ip.addr:9090/` to access the `prometheus` console.
-* `http://docker.host.ip.addr:3000/` to access the `grafana` console, use admin/admin as initial credentials, you should be prompted to change the password on first login.
+- `http://docker.host.ip.addr:3000/` to access the `grafana` console, use admin/admin as initial credentials, you should be prompted to change the password on first login.
 
 Remember to change `docker.host.ip.addr` to the IP address of your docker host.
 
@@ -145,7 +145,7 @@ After you have logged into the `grafana` console the following manual steps are
 Option | Input
 ------------- | -------------
 Name | readsb
-URL | http://prometheus:9090/
+URL | `http://prometheus:9090/`
 
 Clicking `Save & Test` should return a green message indicating success. The dashboard can now be imported with the following steps
 

README.md

@@ -103,8 +103,8 @@ The sections below describe how to configure each part of the container function
 
 Note:
 
-- to enable a parameter, you can set it to any of `1`, `true`, `on`, `enabled`, `enable`, `yes`, or `y`. In the table below, we'll simply use `true` for convenience.
+* to enable a parameter, you can set it to any of `1`, `true`, `on`, `enabled`, `enable`, `yes`, or `y`. In the table below, we'll simply use `true` for convenience.
-- to disable a parameter, you can set it to anything else or simply leave it undefined.
+* to disable a parameter, you can set it to anything else or simply leave it undefined.
 
 ### General Configuration
 
@@ -153,7 +153,7 @@ The following parameters must be set (mandatory) for the container to function:
 | `S6_SERVICES_GRACETIME` | Optional, set to 30000 when saving traces / globe_history | | `3000` |
 | `LOGLEVEL` | `verbose` (all messages), `error` (errors only), `none` (minimal) | | `verbose` |
 
-`READSB_EXTRA_ARGS` just passes arguments to the commandline, you can check this file for more options for wiedehofp's readsb fork: <https://github.com/wiedehopf/readsb/blob/dev/help.h>
+`READSB_EXTRA_ARGS` just passes arguments to the commandline, you can check this file for more options for wiedehopf's readsb fork: <https://github.com/wiedehopf/readsb/blob/dev/help.h>
 
 ### Getting ADSB data to the Ultrafeeder
 
@@ -218,7 +218,41 @@ docker exec -it ultrafeeder /usr/local/bin/autogain1090 reset
 
 In addition to (or instead of) connecting to a SDR or hardware device to get ADSB data, the container also supports ingesting data from a TCP port. Here are some parameters that you need to configure if you want to make this happen:
 
-##### Mandatory parameters
+##### All-in-One Configuration using `ULTRAFEEDER_CONFIG`
+
+`ULTRAFEEDER_CONFIG` is a new parameter that can be used instead of separately defining `READSB_NET_CONNECTOR`, `MLAT_NET_CONNECTOR`, `MLATHUB_NET_CONNECTOR`/`MLATHUB_CONFIG`. These legacy parameters will still work; however, we wanted to provide a single parameter that enables configuration of incoming and outgoing ADSB data, MLAT-client data, and MLATHUB data.
+
+Note that `ULTRAFEEDER_CONFIG` and `ULTRAFEEDER_NET_CONNECTOR` can be used interchangeably; in this documentation, we'll use `ULTRAFEEDER_CONFIG` as an example.
+
+`ULTRAFEEDER_CONFIG` syntax:
+
+The ULTRAFEEDER_CONFIG parameter can have multiple config strings, separated by a `;`. Please note that the config strings cannot contain `;` or `,` -- undefined things may happen if these characters are present.
+
+```yaml
+- ULTRAFEEDER_CONFIG=adsb,host,port,protocol[,uuid=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX][,extra-arguments]
+...or...
+- ULTRAFEEDER_CONFIG=mlat,host,port[,return_port][,uuid=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX][,extra-arguments]
+...or...
+- ULTRAFEEDER_CONFIG=mlathub,host,port,protocol[,uuid=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX][,extra-arguments]
+```
+
+In the above configuration strings:
+
+* `host` is an IP address. Specify an IP/hostname/containername for incoming or outgoing connections.
+* `port` and `return_port` are TCP port numbers
+* `protocol` can be one of the following:
+  * `beast_reduce_out`: Beast-format output with lower data throughput (saves bandwidth and CPU)
+  * `beast_reduce_plus_out`: Beast-format output with extra data (UUID). This is the preferred format when feeding the "new" aggregator services
+  * `beast_out`: Beast-format output
+  * `beast_in`: Beast-format input
+  * `raw_out`: Raw output
+  * `raw_in`: Raw input
+  * `sbs_out`: SBS-format output
+  * `vrs_out`: SBS-format output
+* `uuid=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX` is an optional parameter that sets the UUID for this specific instance. It will override the global `UUID` parameter. This is only needed when you want to send different UUIDs to different aggregators.
+* `extra-arguments` can be any additional command line argument you want to pass to readsb, mlathub, or mlat-client. Example: `--net-only`. Please make sure to only once pass in an extra argument for each of the adsb|mlat|mlathub service.
+
+##### Networking parameters
 
 | Environment Variable | Purpose | Default |
 |----------------------|---------|---------|
@@ -227,7 +261,7 @@ In addition to (or instead of) connecting to a SDR or hardware device to get ADS
 | `MLATHOST` | Legacy parameter. IP/Hostname of an MLAT provider (`mlat-client`). Note - using this parameter will not make the MLAT data part of the consolidated mlathub. The preferred way of ingesting MLAT results is using the `mlathub` functionality of the container, see below for details | |
 | `MLATPORT` | Legacy parameter used with `MLATHOST`. TCP port number of an MLAT provider (`mlat-client`) | 30105 |
 
-###### Alternate Configuration Method with `READSB_NET_CONNECTOR`
+##### Alternate Configuration Method with `READSB_NET_CONNECTOR`
 
 Instead of (or in addition to) using `BEASTHOST`, you can also define ADSB data ingests using the `READSB_NET_CONNECTOR` parameter. This is the preferred way if you have multiple sources or destinations for your ADSB data. This variable allows you to configure incoming and outgoing connections. The variable takes a semicolon (`;`) separated list of `host,port,protocol[,uuid=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX]`, where:
 
@@ -270,7 +304,7 @@ There are many optional parameters relating to the ingestion of data and the gen
 | `READSB_NET_RAW_OUTPUT_INTERVAL` | TCP output flush interval in seconds (maximum interval between two network writes of accumulated data). | `--net-ro-interval=<rate>` | `0.05` |
 | `READSB_NET_RAW_OUTPUT_SIZE` | TCP output flush size (maximum amount of internally buffered data before writing to network). | `--net-ro-size=<size>` | `1200` |
 | `READSB_NET_CONNECTOR_DELAY` | Outbound re-connection delay. | `--net-connector-delay=<seconds>` | `30` |
-| `READSB_NET_HEARTBEAT` | TCP heartbeat rate in seconds (0 to disable). | `--net-heartbeat=<rate>` | `60` |
+| `READSB_NET_HEARTBEAT` | TCP heartbeat rate in seconds (0 to disable). | `--net-heartbeat=<rate>` | `35` |
 | `READSB_NET_RAW_INPUT_PORT` | TCP raw input listen ports. | `--net-ri-port=<ports>` | `30001` |
 | `READSB_NET_RAW_OUTPUT_PORT` | TCP raw output listen ports. | `--net-ro-port=<ports>` | `30002` |
 | `READSB_NET_SBS_INPUT_PORT` | TCP BaseStation input listen ports. | `--net-sbs-in-port=<ports>` | Unset |
@@ -282,6 +316,33 @@ There are many optional parameters relating to the ingestion of data and the gen
 | `READSB_JSON_INTERVAL` | Update interval for the webinterface in seconds / interval between aircraft.json writes | `--write-json-every=<sec>` | `1.0` |
 | `READSB_JSON_TRACE_INTERVAL` | Per plane interval for json position output and trace interval for globe history | `--json-trace-interval=<sec>` | `15` |
 | `READSB_FORWARD_MLAT_SBS` | If set to anthing, it will include MLAT results in the SBS/BaseStation output. This may be desirable if you feed SBS data to applications like [VRS](https://github.com/sdr-enthusiasts/docker-virtualradarserver) or [PlaneFence](https://github.com/kx1t/docker-planefence) | Unset |
+| `UUID` | Sets the UUID that is sent on the `beast_reduce_plus` port if no individual UUIDs have been defined with the `READSB_NET_CONNECTOR` parameter. Similarly, it's also used with `mlat-client` (see below) if no individual UUIDs have been set with the `MLAT_CONFIG` parameter. | | unset |
+
+##### MLAT configuration
+
+The Ultrafeeder contains a capability to send MLAT data to MLAT servers to be processed, and to receive the MLAT results and integrate those with an MLAT Hub and the `tar1090` map.
+It will create a separate instance of `mlat-client` for each defined MLAT server. The parameters for these `mlat-client` instances is as follows:
+
+```yaml
+    environment:
+    ...
+      - MLAT_NET_CONNECTOR=
+           mlat-server1.com,port1,return_port1,uuid=1234-5678-90123,--extra-argument hello;
+           mlat-server2.com,port2,return_port2,uuid=5678-9012-34567,--extra-argument hello
+    ...
+```
+
+where:
+
+* `mlat-server.com` is the domain name or ip address of the target MLAT server (mandatory)
+* `port` is the port (TCP or UDP) of the target MLAT server (mandatory)
+* `return_port` is the port at which the MLAT results are made available in BEAST format. We recommend to sequentially number them starting at 39000 (optional)
+* `uuid=xxxx` defines a unique user ID for this MLAT server instance. If included, the string must start with `uuid=` (optional)
+* `extra-arguments` are any additional command line arguments that you would like to use for this MLAT Client instance (optional)
+
+Note - the three optional parameters (`return_port`, `uuid=`, and `extra-arguments`) can be given in any order.
+
+If no UUID is specified with the `MLAT_CONFIG` parameter, it will use the value of the `UUID` parameter if it exists. If that fails, no UUID parameter is included with `mlat-client`.
 
 ### Web Gui (`tar1090`) Configuration
 
@@ -291,7 +352,7 @@ The web interface is rendered to port `80` in the container. This can me mapped
 
 All of the variables below are optional.
 
-Note - due to design limitations of `readsb`, the `tar1090` graphical interface will by default ONLY show MLAT results from the aggregators/MLAT sources that were defined with the `MLAT_NET_CONNECTOR` parameter. If you want to show any additional MLAT results (for example, those from `piaware`), you should add a separate `READSB_NET_CONNECTOR` for them.
+Note - due to design limitations of `readsb`, the `tar1090` graphical interface will by default ONLY show MLAT results from the aggregators/MLAT sources that were defined with the `MLAT_NET_CONNECTOR` parameter. If you want to show any additional MLAT results (for example, those from `piaware`), you should add a separate `READSB_NET_CONNECTOR` entry for them.
 
 #### `tar1090` Core Configuration
 

docker-compose.yml

@@ -34,32 +34,27 @@ services:
       # --------------------------------------------------
       # Sources and Aggregator connections:
       # (Note - remove the ones you are not using / feeding)
-      - READSB_NET_CONNECTOR=
+      - ULTRAFEEDER_CONFIG=
-          feed.adsb.fi,30004,beast_reduce_plus_out;
+          adsb,feed.adsb.fi,30004,beast_reduce_plus_out;
-          in.adsb.lol,30004,beast_reduce_plus_out;
+          adsb,in.adsb.lol,30004,beast_reduce_plus_out;
-          feed.adsb.one,64004,beast_reduce_plus_out;
+          adsb,feed.adsb.one,64004,beast_reduce_plus_out;
-          feed.planespotters.net,30004,beast_reduce_plus_out;
+          adsb,feed.planespotters.net,30004,beast_reduce_plus_out;
-          feed.theairtraffic.com,30004,beast_reduce_plus_out
+          adsb,feed.theairtraffic.com,30004,beast_reduce_plus_out;
-      # If you really want to feed ADSBExchange, you can do so by adding this above: feed1.adsbexchange.com,30004,beast_reduce_plus_out
+          mlat,feed.adsb.fi,31090,39000;
-      - UUID=${MULTIFEEDER_UUID}
+          mlat,in.adsb.lol,31090,39001;
+          mlat,feed.adsb.one,64006,39002;
+          mlat,mlat.planespotters.net,31090,39003;
+          mlat,feed.theairtraffic.com,31090,39004;
+          mlathub,piaware,30105,beast_in;
+          mlathub,rbfeeder,30105,beast_in;
+          mlathub,radarvirtuel,30105,beast_in
+      # If you really want to feed ADSBExchange, you can do so by adding this above: 
+      #        adsb,feed1.adsbexchange.com,30004,beast_reduce_plus_out,uuid=${ADSBX_UUID};
+      #        mlat,feed.adsbexchange.com,31090,39005,uuid=${ADSBX_UUID}
       #
       # --------------------------------------------------
-      # MLAT and MLATHUB related parameters:
+      - UUID=${MULTIFEEDER_UUID}
-      # MLAT_CONFIG defines who you are feeding MLAT data to, and on what port(s) the results will become available
-      - MLAT_CONFIG=
-          feed.adsb.fi,31090,39000;
-          in.adsb.lol,31090,39001;
-          feed.adsb.one,64006,39002;
-          mlat.planespotters.net,31090,39003;
-          feed.theairtraffic.com,31090,39004
-      # If you really want to feed ADSBExchange, you can do so by adding this above: feed.adsbexchange.com,31090,39005
       - MLAT_USER=${FEEDER_NAME}
-      # MLATHUB automatically gets MLAT results data from the feeders defined in MLAT_CONFIG,
-      # and additionally from the Net Connectors defined with MLATHUB_NET_CONNECTOR
-      - MLATHUB_NET_CONNECTOR=
-          piaware,30105,beast_in;
-          rbfeeder,30105,beast_in;
-          radarvirtuel,30105,beast_in
       #
       # --------------------------------------------------
       # TAR1090 (Map Web Page) parameters:
@@ -100,4 +95,4 @@ services:
     tmpfs:
       - /run:exec,size=256M
       - /tmp:size=128M
-      - /var/log:size=32M
+      - /var/log:size=32M

markdownlint.config

@@ -1,3 +0,0 @@
-{
-    "MD024": false
-}

rootfs/etc/services.d/mlat-client/run

@@ -1,8 +1,6 @@
 #!/usr/bin/with-contenv bash
 # shellcheck shell=bash disable=SC1091,SC2015,SC2016
 
-source /scripts/common
-
 #---------------------------------------------------------------------------------------------
 # Copyright (C) 2023, Ramon F. Kolb (kx1t)
 #
@@ -20,7 +18,10 @@ source /scripts/common
 #---------------------------------------------------------------------------------------------
 
 APPNAME="mlat-client"
-echo "[$(date)][${APPNAME}] Started as an s6 service"
+echo "[$(date +"%Y-%m-%d %H:%M:%S")][${APPNAME}] Started as an s6 service"
+
+source /scripts/common
+source /scripts/interpret_ultrafeeder_config
 
 MLAT_CMD="/usr/bin/mlat-client"
 
@@ -29,35 +30,42 @@ declare -A pid_array
 
 if [[ -z "${MLAT_CONFIG}" ]]
 then
-    echo "[$(date)][${APPNAME}] Warning: MLAT_CONFIG not defined - MLAT will be disabled."
+    echo "[$(date +"%Y-%m-%d %H:%M:%S")][${APPNAME}] Warning: MLAT_CONFIG not defined - MLAT will be disabled."
     sleep infinity
 fi
 
 if [[ -z "${MLAT_USER}" ]] && [[ -z "${UUID}" ]]
 then
-    echo "[$(date)][${APPNAME}] ERROR: either UUID or MLAT_USER must be defined - MLAT will be disabled."
+    echo "[$(date +"%Y-%m-%d %H:%M:%S")][${APPNAME}] ERROR: either UUID or MLAT_USER must be defined - MLAT will be disabled."
     sleep infinity
 fi
 
 if [[ -z "$LAT$READSB_LAT" ]]; then
-    echo "[$(date)][${APPNAME}] ERROR: READSB_LAT or LAT must be defined - MLAT will be disabled."
+    echo "[$(date +"%Y-%m-%d %H:%M:%S")][${APPNAME}] ERROR: READSB_LAT or LAT must be defined - MLAT will be disabled."
     sleep infinity
 fi
 
 if [[ -z "$LONG$READSB_LON" ]]; then
-    echo "[$(date)][${APPNAME}] ERROR: READSB_LON or LONG must be defined - MLAT will be disabled."
+    echo "[$(date +"%Y-%m-%d %H:%M:%S")][${APPNAME}] ERROR: READSB_LON or LONG must be defined - MLAT will be disabled."
     sleep infinity
 fi
 if [[ -z "$ALT$READSB_ALT" ]]; then
-    echo "[$(date)][${APPNAME}] ERROR: READSB_ALT or ALT must be defined - MLAT will be disabled."
+    echo "[$(date +"%Y-%m-%d %H:%M:%S")][${APPNAME}] ERROR: READSB_ALT or ALT must be defined - MLAT will be disabled."
     sleep infinity
 fi
 
 # MLAT_CONFIG has the following format:
-# MLAT_CONFIG=mlatserver_1,mlatserver_port_1[,results_port_1[,extra_params_1]];mlatserver_2,mlatserver_port_2[,results_port_2[,extra_params_2]] etc
+# MLAT_CONFIG=mlatserver_1,mlatserver_port_1,x1,y1,z1;mlatserver_2,mlatserver_port_2,x2,y2,z2 etc
+# where x1,y1,z1; x22,y2,z2; etc are optional and are interpreted as follows:
+# if it's a number, we'll assume it's a port number for return messages
+# if it starts with 'uuid=', it's a UUID number for that instance
+# anything else will be appended as extra parameter(s) to the mlat-client command line
 
 # parse MLAT_CONFIG string into mlat_configs array
-readarray -td ";" mlat_configs < <(printf '%s' "${MLAT_CONFIG// /}")
+# Strip any extraneous spaces:
+MLAT_CONFIG="${MLAT_CONFIG#"${MLAT_CONFIG%%[![:space:]]*}"}"   # strip leading space
+MLAT_CONFIG="${MLAT_CONFIG//; /;}"
+readarray -td ";" mlat_configs < <(printf '%s' "${MLAT_CONFIG}")
 
 # Now loop through the MLAT_CONFIG items and start up an Mlat_client for each of them:
 for instance in "${mlat_configs[@]}"
@@ -66,15 +74,19 @@ do
     # put individual params into the $params array:
     readarray -td "," params < <(printf '%s' "${instance}")
 
+    uuid_arg=false
+    extra_args=""
+
     # Check if the params array has values for the mandatory elements:
     if [[ -z "${params[0]}" ]] || [[ -z "${params[1]}" ]]
     then
-        echo "[$(date)][${APPNAME}] ERROR -- MLAT_CONFIG is malformed: \"${instance}\". Stopping MLAT execution."
+        echo "[$(date +"%Y-%m-%d %H:%M:%S")][${APPNAME}] ERROR -- MLAT_CONFIG is malformed: \"${instance}\". Stopping MLAT execution."
          # shellcheck disable=SC2046
         kill $(ps -s $$ -o pid=)   
         sleep infinity
     fi
 
+    # ------------------------------------------------
     # Build the MLAT parameter string:
     MLAT_PARAM=(--input-type "${MLAT_INPUT_TYPE}")
     MLAT_PARAM+=(--input-connect localhost:30005)
@@ -98,40 +110,79 @@ do
         MLAT_PARAM+=(--alt "${READSB_ALT}")
     fi
 
-    [[ -n "${MLAT_USER}" ]] && MLAT_PARAM+=(--user "${MLAT_USER}") || MLAT_PARAM+=(--user "${UUID}")
+    if [[ -n "${MLAT_USER}" ]]; then 
-    [[ -n "${params[2]}" ]] && MLAT_PARAM+=("--results beast,listen,${params[2]}") || true
+        MLAT_PARAM+=(--user "${MLAT_USER}")
-    [[ -n "${params[3]}" ]] && MLAT_PARAM+=("${params[3]}") || true
+    else 
+        rnd="${RANDOM}"
+        echo "[$(date +"%Y-%m-%d %H:%M:%S")][${APPNAME}] WARNING: MLAT_USER is not set - using random number \"${rnd}\" as MLAT_USER"
+        MLAT_PARAM+=(--user "${rnd}")
+    fi
 
+    # Now process the rest of the arguments
+    # The order doesn't matter, we'll do pattern matching:
+    # If the argument is a number, then it must be the beast_results port
+    # If the argument starts with "uuid=", then it must be a UUID
+    # If the argument isn't any of the above, then it's an "extra argument"
+
+    for ((i=2; i<${#params[*]}; i++))
+    do
+        if [[ -n "${params[i]}" ]] && [[ "${params[i]}" =~ ^[0-9]+$ ]]; then
+            # It's a number so it must be the return port
+            MLAT_PARAM+=("--results beast,listen,${params[i]}")
+
+        elif header="${params[i]:0:5}" && [[ "${header,,}" == "uuid=" ]]; then
+            # It's a UUID
+            MLAT_PARAM+=("--uuid ${params[i]:5}")
+            uuid_arg=true
+
+        else
+            # It's an Extra Args string
+            extra_args="$extra_args ${params[i]}"
+
+        fi
+    done
+    # If no UUID was set in the argument strings, check if we can use the generic UUID parameter
+    if [[ $uuid_arg == false ]] && [[ -n "${UUID}" ]]; then
+        MLAT_PARAM+=("--uuid ${UUID}")
+    fi
+    # Now add the extra_args, if any:
+    [[ -n "${extra_args}" ]] && MLAT_PARAM+=("${extra_args}") || true
+
+    # ------------------------------------------------
+    # Create the command exec string:
     # shellcheck disable=SC2048,SC2086
     execstring="$(echo ${MLAT_CMD} ${MLAT_PARAM[*]} | xargs)"
 
-    # Wait a bit until readsb is established...
+    # ------------------------------------------------
-    if ! pgrep readsb >/dev/null 
+    # Ready to launch, but wait until readsb is established...
-then
+    if ! pgrep readsb >/dev/null; then
-    echo "[$(date)][mlathub] Delaying start of MLAT client(s) until container is established..."
+        echo "[$(date +"%Y-%m-%d %H:%M:%S")][$APPNAME] Delaying start of MLAT client(s) until container is established..."
-    while ! pgrep readsb >/dev/null
+        while ! pgrep readsb >/dev/null
-    do
+        do
-        sleep 2
+            sleep 2
-    done
+        done
-fi
+    fi
-sleep 10     # sleep a bit so everything is well established - starting readsb may take a bit
+    sleep 10     # sleep a bit so everything is well established - starting readsb may take a bit
 
     # run this Mlat_client instance in the background:
-    echo  "[$(date)][${APPNAME}] starting: ${MLAT_CMD} ${MLAT_PARAM[*]}"
+    echo  "[$(date +"%Y-%m-%d %H:%M:%S")][${APPNAME}] starting: ${MLAT_CMD} ${MLAT_PARAM[*]}"
 
     if [[ -z "${LOGLEVEL}" ]] || [[ "${LOGLEVEL,,}" == "verbose" ]]; then
-        { exec ${execstring} 2>&1 | stdbuf -o0 awk -v app="${APPNAME}" -v inst="${params[0]}" '{print "[" strftime("%Y/%m/%d %H:%M:%S", systime()) "][" app "][" inst "] " $0}'; } &
+        { exec ${execstring} 2>&1 | stdbuf -o0 awk -v app="${APPNAME}" -v inst="${params[0]}" '{print "[" strftime("%Y-%m-%d %H:%M:%S", systime()) "][" app "][" inst "] " $0}'; } &
     elif [[ "${LOGLEVEL,,}" == "error" ]]; then
-        { exec ${execstring} 2>&1 >/dev/null | stdbuf -o0 awk -v app="${APPNAME}" -v inst="${params[0]}" '{print "[" strftime("%Y/%m/%d %H:%M:%S", systime()) "][" app "][" inst "] " $0}'; } &
+        { exec ${execstring} 2>&1 >/dev/null | stdbuf -o0 awk -v app="${APPNAME}" -v inst="${params[0]}" '{print "[" strftime("%Y-%m-%d %H:%M:%S", systime()) "][" app "][" inst "] " $0}'; } &
     elif [[ "${LOGLEVEL,,}" == "none" ]]; then
        { exec ${execstring} >/dev/null 2>/dev/null; } &
     fi
 
+    # pid_array is indexed by the PID of each mlat_client and contains the MLAT_PARAMs for that instance
+    # This is done so we can monitor them and restart them if needed
     pid_array[$!]="${MLAT_PARAM[*]}"
     sleep 5
 done
 
-# All MLAT Clients are up, so set the flag:
+# All MLAT Clients are up, so set the flag. This flag is read by MLATHUB as a sign that it's
+# safe to start up
 touch /run/mlathub_up
 
 # Now iterate over all MLAT-client instances and check if they are still running:
@@ -143,14 +194,14 @@ do
         then
             # it exited - let's restart:
             sleep "${RESTARTTIMER}"
-            [[ "${LOGLEVEL,,}" != "none" ]] && echo  "[$(date)][${APPNAME}] MLAT_Client $(awk '{print $6}' <<< "${pid_array[$mlat_pid]}") exited. Attempting to restart." || true
+            [[ "${LOGLEVEL,,}" != "none" ]] && echo "[$(date +"%Y-%m-%d %H:%M:%S")][${APPNAME}] MLAT_Client $(awk '{print $6}' <<< "${pid_array[$mlat_pid]}") exited. Attempting to restart." || true
             # shellcheck disable=SC2086
             execstring="$(echo ${MLAT_CMD} ${pid_array[$mlat_pid]} | xargs)"
 
             if [[ -z "${LOGLEVEL}" ]] || [[ "${LOGLEVEL,,}" == "verbose" ]]; then
-                { exec ${execstring} 2>&1 | stdbuf -o0 awk -v app="${APPNAME}" -v inst="${params[0]}" '{print "[" strftime("%Y/%m/%d %H:%M:%S", systime()) "][" app "][" inst "] " $0}'; } &
+                { exec ${execstring} 2>&1 | stdbuf -o0 awk -v app="${APPNAME}" -v inst="${params[0]}" '{print "[" strftime("%Y-%m-%d %H:%M:%S", systime()) "][" app "][" inst "] " $0}'; } &
             elif [[ "${LOGLEVEL,,}" == "error" ]]; then
-                { exec ${execstring} 2>&1 >/dev/null | stdbuf -o0 awk -v app="${APPNAME}" -v inst="${params[0]}" '{print "[" strftime("%Y/%m/%d %H:%M:%S", systime()) "][" app "][" inst "] " $0}'; } &
+                { exec ${execstring} 2>&1 >/dev/null | stdbuf -o0 awk -v app="${APPNAME}" -v inst="${params[0]}" '{print "[" strftime("%Y-%m-%d %H:%M:%S", systime()) "][" app "][" inst "] " $0}'; } &
             elif [[ "${LOGLEVEL,,}" == "none" ]]; then
                 { exec ${execstring} >/dev/null 2>/dev/null; } &
             fi

rootfs/etc/services.d/mlathub/run

@@ -1,15 +1,17 @@
 #!/usr/bin/with-contenv bash
-# shellcheck shell=bash disable=SC1091,SC2016
+# shellcheck shell=bash disable=SC1091,SC2015,SC2016
 
+APPNAME="mlathub"
 source /scripts/common
+source /scripts/interpret_ultrafeeder_config
 
 if [[ -z "${MLAT_CONFIG}" ]] && [[ -z "$MLATHUB_NET_CONNECTOR" ]]; then
-    echo "[$(date)][mlathub] No MLAT servers have been defined in MLAT_CONFIG and no external sources have been defined in MLATHUB_NET_CONNECTOR - no need to start MLATHUB"
+    echo "[$(date "+%Y-%m-%d %H:%M:%S")][${APPNAME}] No MLAT servers have been defined in MLAT_CONFIG and no external sources have been defined in MLATHUB_NET_CONNECTOR - no need to start MLATHUB"
     sleep infinity
 fi
 
 if chk_enabled "${MLATHUB_DISABLED}"; then
-    echo "[$(date)][mlathub] MLATHUB is disabled."
+    echo "[$(date "+%Y-%m-%d %H:%M:%S")][${APPNAME}] MLATHUB is disabled."
     sleep infinity
 fi
 
@@ -43,33 +45,38 @@ do
     # put individual params into the $params array:
     readarray -td "," params < <(printf '%s' "${instance}")
 
-    if [[ -n "${params[2]}" ]]
+    for ((i=2; i<${#params[*]}; i++))
-    then
+    do
-        MLATHUB_CMD+=("--net-connector=localhost,${params[2]},beast_in")
+        if [[ -n "${params[i]}" ]] && [[ "${params[i]}" =~ ^[0-9]+$ ]]; then
-        (( mlat_result_sources++ )) || true
+            # the parameter is a number and it is assumed that this is the beast_results port
-    fi
+            MLATHUB_CMD+=("--net-connector=localhost,${params[i]},beast_in")
+            (( mlat_result_sources++ )) || true
+            break
+        fi
+    done
 done
+
 if (( mlat_result_sources == 0 ))
 then
-    echo "[$(date)][mlathub] No MLAT servers have been defined in MLAT_CONFIG - no need to start MLATHUB"
+    echo "[$(date "+%Y-%m-%d %H:%M:%S")][${APPNAME}] No MLAT servers have been defined in MLAT_CONFIG - no need to start MLATHUB"
     sleep infinity
 fi
 
-# removed as it will cause looping:
-# handle MLATHUB_ENABLE_ADSB_INGEST (if set, connect to readsb as input)
-# if [[ -n "$MLATHUB_ENABLE_ADSB_INGEST" ]]; then
-#    MLATHUB_CMD+=("--net-connector=localhost,30006,beast_in")
-# fi
-
 # Add any additional net_connectors:
 if [[ -n "$MLATHUB_NET_CONNECTOR" ]]; then
+        # Strip any extraneous spaces:
+    MLATHUB_NET_CONNECTOR="${MLATHUB_NET_CONNECTOR#"${MLATHUB_NET_CONNECTOR%%[![:space:]]*}"}"   # strip leading spaces
+    MLATHUB_NET_CONNECTOR="${MLATHUB_NET_CONNECTOR//; /;}"      # strip space after semicolon
     IFS=';' read -r -a MLATHUB_NET_CONNECTOR_ARRAY <<< "$MLATHUB_NET_CONNECTOR"
     for NET_CONNECTOR_ELEMENT in "${MLATHUB_NET_CONNECTOR_ARRAY[@]}"
     do
-        MLATHUB_CMD+=("--net-connector=${NET_CONNECTOR_ELEMENT// /}")
+        MLATHUB_CMD+=("--net-connector=${NET_CONNECTOR_ELEMENT}")
     done
 fi
 
+# Handle ULTRAFEEDER_CONFIG / ULTRAFEEDER_NET_CONNECTOR elements relevant to readsb:
+[[ "${#MLATHUB_CONF_ARR[@]}" -gt 0 ]] && MLATHUB_CMD+=("${MLATHUB_CONF_ARR[@]}") || true
+
 if chk_enabled "${READSB_DEBUG}"; then
     MLATHUB_CMD+=("--debug=$READSB_DEBUG")
 fi
@@ -101,7 +108,7 @@ MLATHUB_CMD+=("--net-heartbeat=${READSB_NET_HEARTBEAT:-35}")
 
 if [[ ! -f /run/mlathub_up ]]
 then
-    echo "[$(date)][mlathub] Delaying MLAT hub start until container is established..."
+    echo "[$(date "+%Y-%m-%d %H:%M:%S")][${APPNAME}] Delaying MLAT hub start until container is established..."
     while [[ ! -f /run/mlathub_up ]]
     do
         sleep 2
@@ -109,18 +116,18 @@ then
 fi
 sleep 5     # sleep a bit so everything is well established
 
-echo "[$(date)][mlathub] Starting MLATHUB..."
+echo "[$(date "+%Y-%m-%d %H:%M:%S")][${APPNAME}] Starting MLATHUB..."
 
 # shellcheck disable=SC2086
 if [[ -z "${LOGLEVEL}" ]] || [[ "${LOGLEVEL,,}" == "verbose" ]]; then
     "${MLATHUB_BIN}" "${MLATHUB_CMD[@]}" $MLATHUB_EXTRA_ARGS 2>&1 | \
-	mawk -W Interactive '{print "[" strftime("%Y/%m/%d %H:%M:%S", systime()) "][mlathub] " $0}'
+	mawk -W Interactive '{print "[" strftime("%Y-%m-%d %H:%M:%S", systime()) "][mlathub] " $0}'
 elif [[ "${LOGLEVEL,,}" == "error" ]]; then
     "${MLATHUB_BIN}" "${MLATHUB_CMD[@]}" $MLATHUB_EXTRA_ARGS 2>&1 >/dev/null | \
-	mawk -W Interactive '{print "[" strftime("%Y/%m/%d %H:%M:%S", systime()) "][mlathub] " $0}'
+	mawk -W Interactive '{print "[" strftime("%Y-%m-%d %H:%M:%S", systime()) "][mlathub] " $0}'
 elif [[ "${LOGLEVEL,,}" == "none" ]]; then
     "${MLATHUB_BIN}" "${MLATHUB_CMD[@]}" $MLATHUB_EXTRA_ARGS >/dev/null 2>/dev/null 
 fi
 
-echo "[$(date "+%Y-%m-%d %H:%M:%S")][mlathub] The mlathub instance has exited. Restarting in 10 seconds."
+echo "[$(date "+%Y-%m-%d %H:%M:%S")][${APPNAME}] The mlathub instance has exited. Restarting in 10 seconds."
 sleep 10

rootfs/etc/services.d/readsb/run

@@ -1,7 +1,10 @@
 #!/usr/bin/with-contenv bash
-# shellcheck shell=bash disable=SC2016,SC1091
+# shellcheck shell=bash disable=SC2015,SC2016,SC1091
+
+APPNAME="readsb"
 
 source /scripts/common
+source /scripts/interpret_ultrafeeder_config
 
 mkdir -p /run/readsb
 
@@ -61,12 +64,12 @@ if ! chk_enabled "$READSB_NET_SBS_DISABLE_REDUCE"; then
 fi
 
 if chk_enabled "${READSB_FORWARD_MLAT}"; then
-    echo "[$(date +"%Y-%m-%d %H:%M:%S")][readsb] WARNING -- READSB_FORWARD_MLAT has been set! Do not feed the output of this container to any aggregators!"
+    echo "[$(date +"%Y-%m-%d %H:%M:%S")][${APPNAME}] WARNING -- READSB_FORWARD_MLAT has been set! Do not feed the output of this container to any aggregators!"
     READSB_CMD+=("--forward-mlat")
 fi
 
 if chk_enabled "${READSB_FORWARD_MLAT_SBS}"; then
-    echo "[$(date +"%Y-%m-%d %H:%M:%S")][readsb] WARNING -- READSB_FORWARD_MLAT_SBS has been set! Do not feed the SBS (BaseStation) output of this container to any aggregators!"
+    echo "[$(date +"%Y-%m-%d %H:%M:%S")][${APPNAME}] WARNING -- READSB_FORWARD_MLAT_SBS has been set! Do not feed the SBS (BaseStation) output of this container to any aggregators!"
     READSB_CMD+=("--forward-mlat-sbs")
 fi
 
@@ -93,6 +96,7 @@ if [[ -n "$READSB_MAX_RANGE" ]]; then
 fi
 
 # Handle "--mlat"
+# shellcheck disable=SC2153
 if chk_enabled "$READSB_MLAT"; then
     READSB_CMD+=("--mlat")
 fi
@@ -168,6 +172,9 @@ fi
 
 # Handle "--net-connector=<ip,port,protocol>"
 if [[ -n "$READSB_NET_CONNECTOR" ]]; then
+    # Strip any extraneous spaces:
+    READSB_NET_CONNECTOR="${READSB_NET_CONNECTOR#"${READSB_NET_CONNECTOR%%[![:space:]]*}"}"   # strip leading space
+    READSB_NET_CONNECTOR="${READSB_NET_CONNECTOR//; /;}"
     IFS=';' read -r -a READSB_NET_CONNECTOR_ARRAY <<< "$READSB_NET_CONNECTOR"
     for NET_CONNECTOR_ELEMENT in "${READSB_NET_CONNECTOR_ARRAY[@]}"
     do
@@ -175,13 +182,16 @@ if [[ -n "$READSB_NET_CONNECTOR" ]]; then
     done
 fi
 
+# Handle ULTRAFEEDER_CONFIG / ULTRAFEEDER_NET_CONNECTOR elements relevant to readsb:
+[[ "${#READSB_CONF_ARR[@]}" -gt 0 ]] && READSB_CMD+=("${READSB_CONF_ARR[@]}") || true
+
 # Handle "--net-connector-delay=<seconds>"
 if [[ -n "$READSB_NET_CONNECTOR_DELAY" ]]; then
     READSB_CMD+=("--net-connector-delay=$READSB_NET_CONNECTOR_DELAY")
 fi
 
 # Handle "--net-heartbeat=<rate>"
-READSB_CMD+=("--net-heartbeat=${READSB_NET_HEARTBEAT:-30}")
+READSB_CMD+=("--net-heartbeat=${READSB_NET_HEARTBEAT:-35}")
 
 # Handle "--net-ri-port=<ports>"
 if [[ -n "$READSB_NET_RAW_INPUT_PORT" ]]; then
@@ -320,7 +330,7 @@ elif [[ "${LOGLEVEL,,}" == "none" ]]; then
     "${READSB_BIN}" "${READSB_CMD[@]}" $READSB_EXTRA_ARGS >/dev/null 2>/dev/null
 fi
 
-echo "[$(date "+%Y-%m-%d %H:%M:%S")][readsb] The main readsb instance has exited. Restarting in 10 seconds."
+echo "[$(date "+%Y-%m-%d %H:%M:%S")][${APPNAME}] The main readsb instance has exited. Restarting in 10 seconds."
 sleep 10
 
 

rootfs/scripts/interpret_ultrafeeder_config

@@ -0,0 +1,123 @@
+# shellcheck shell=bash disable=SC2015
+#
+# This scripts should be sourced by the /etc/services.d/xxx/run modules for
+# readsb, mlat-client, and mlathub. It interprets the ULTRAFEEDER_CONFIG / ULTRAFEEDER_NET_CONNECTOR parameter
+# and makes individual net-connector strings available for these three modules.
+#
+#---------------------------------------------------------------------------------------------
+# Copyright (C) 2023, Ramon F. Kolb (kx1t)
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation, either version 3 of the License, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+# or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along with this program.
+# If not, see <https://www.gnu.org/licenses/>.
+#---------------------------------------------------------------------------------------------
+#
+# Note:
+# ULTRAFEEDER_CONFIG and ULTRAFEEDER_NET_CONNECTOR can be used interchangeably
+# If both are defined, they will be considered jointly.
+#
+# Format:
+# ULTRAFEEDER_CONFIG=adsb,host,port,protocol[,uuid=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX][,extra-arguments]
+# ULTRAFEEDER_CONFIG=mlat,host,port[,return_port][,uuid=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX][,extra-arguments]
+# ULTRAFEEDER_CONFIG=mlathub,host,port,protocol[,uuid=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX][,extra-arguments]
+#
+# The ULTRAFEEDER_CONFIG parameter can have multiple config strings, separated by a `;`
+# Please note that the config strings cannot containe `;` or `,` -- undefined things may happen if these characters are present.
+#
+# In the above configuration strings:
+# `host` is an IP address. Specify an IP/hostname/containername for incoming or outgoing connections.
+# `port` is a TCP port number
+# `protocol` can be one of the following:
+#    `beast_reduce_out`: Beast-format output with lower data throughput (saves bandwidth and CPU)
+#    `beast_reduce_plus_out`: Beast-format output with extra data (UUID). This is the preferred format when feeding the "new" aggregator services
+#    `beast_out`: Beast-format output
+#    `beast_in`: Beast-format input
+#    `raw_out`: Raw output
+#    `raw_in`: Raw input
+#    `sbs_out`: SBS-format output
+#    `vrs_out`: SBS-format output
+# `uuid=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX` is an optional parameter that sets the UUID for this specific instance.
+#     It will override the global `UUID` parameter. This is only needed when you want to send different UUIDs to different aggregators.
+# `extra-arguments` can be any additional command line argument you want to pass to readsb, mlathub, or mlat-client
+#     Example: `--net-only`. Please make sure to only once pass in an extra argument for each of the adsb|mlat|mlathub service.
+
+# Do some prep work:
+# combine ULTRAFEEDER_CONFIG and ULTRAFEEDER_NET_CONNECTOR
+ULTRAFEEDER_CONFIG="${ULTRAFEEDER_CONFIG}${ULTRAFEEDER_CONFIG:+;}${ULTRAFEEDER_NET_CONNECTOR}"
+# Strip any extraneous spaces:
+ULTRAFEEDER_CONFIG="${ULTRAFEEDER_CONFIG#"${ULTRAFEEDER_CONFIG%%[![:space:]]*}"}"   # strip leading space
+ULTRAFEEDER_CONFIG="${ULTRAFEEDER_CONFIG//; /;}"
+
+# Now read each entry and parse it:
+
+READSB_CONF_ARR=()
+MLATHUB_CONF_ARR=()
+
+readarray -td ";" configs < <(printf '%s' "${ULTRAFEEDER_CONFIG}")
+
+# Now loop through the MLAT_CONFIG items and start up an Mlat_client for each of them:
+for instance in "${configs[@]}"
+do
+    [[ -z "${instance}" ]] && continue || true
+    # put individual params into the $params array:
+    readarray -td "," param < <(printf '%s' "${instance}")
+
+    # Process based on parameter string type in param[0]:
+    param[0]="${param[0],,}"
+    case "${param[0]}" in
+
+        adsb)
+            # parse arguments for main readsb
+            # build net-connector string into $readsb_str
+            # and add any extra arguments in as individual entries into READSB_CONF_ARR()
+            readsb_str="${param[1]},${param[2]},${param[3]}"
+            for ((i=4; i<${#param[*]}; i++))
+            do
+                if [[ "${param[i]:0:5}" == "uuid=" ]]; then
+                    readsb_str="${readsb_str},${param[i]}"
+                elif [[ -n "${param[i]}" ]]; then
+                    READSB_CONF_ARR+=("${param[i]}")
+                fi
+            done
+            # Now add $readsb_str to the READSB_CONF_ARR() as --net-connector argument:
+            READSB_CONF_ARR+=("--net-connector=${readsb_str}")
+        ;;
+
+        mlat)
+            # parse arguments for the individual MLAT clients.
+            # In this case, the mlat-client run module will interpret everything
+            # as long as it is in the MLAT_CONFIG parameter. So we'll just add that here.
+            MLAT_CONFIG="${MLAT_CONFIG}${MLAT_CONFIG:+;}${instance#*,}"
+        ;;
+
+        mlathub)
+            # parse arguments for mlat_hub
+            # build net-connector string into $readsb_str
+            # and add any extra arguments in as individual entries into MLATHUB_CONF_STR()
+            mlathub_str="${param[1]},${param[2]},${param[3]}"
+            for ((i=4; i<${#param[*]}; i++))
+            do
+                if [[ "${param[i]:0:5}" == "uuid=" ]]; then
+                    mlathub_str="${readsb_str},${param[i]}"
+                elif [[ -n "${param[i]}" ]]; then
+                    MLATHUB_CONF_ARR+=("${param[i]}")
+                fi
+            done
+            # Now add $readsb_str to the READSB_CONF_STR as --net-connector argument:
+            MLATHUB_CONF_ARR+=("--net-connector=${mlathub_str}")
+        ;;
+
+        *)
+            # unknown string; warn and skip
+            echo "[$(date)][${APPNAME}] Warning: skipping unknown ULTRAFEEDER_CONFIG config element \"${instance}\""
+    esac
+
+done