RadioReference on Facebook   RadioReference on Twitter   RadioReference Blog
 
Wiki Home
Page
View source
History


Personal Tools

Search the Wiki





 

Streaming with Trunk Recorder and Liquidsoap


Contents

Introduction

Trunk Recorder is a free Linux based application written by Luke Berndt that can decode the following:

  • Trunked P25 & SmartNet Systems
  • Conventional P25 & analog systems, where each group has a dedicated RF channel
  • P25 Phase 1, P25 Phase 2 & Analog voice channels

Using a number of supported SDR hardware dongles Trunk Recorder can take entire P25 systems and simultaneously record every single transmission occurring at any moment in time. This is accomplished by using enough SDR dongles to cover the entire spectrum of voice and control channels in a system.

From a stream provider perspective this gives us the ability to put all transmissions into a stream without losing those that occurred while a physical scanner would be "stuck" on another channel. The cost is a slight delay caused by queuing the transmissions, but the severity of that queue depth is dependent upon how many talk-groups you stream and how chatty they are.

Now that Trunk Recorder supports analog its possible to combine analog and digital transmissions into the stream.

The "streamthis" python script below was written to bridge the gap between the .wav files Trunk Recorder creates, and an idle liquidsoap client already connected to Broadcastify. It made it possible to build the Twinsburg, Hudson and Reminderville Police, Fire / EMS stream and has worked quite well. That particular stream is sourced from two separate computers, one of which is dedicated to conventional analog receive. As those files are generated they are copied to the primary computer via ssh, and the streamthis script is remotely executed to insert the newly delivered analog .mp3 file to the stream queue.

How Trunk Recorder Works

Under the hood Trunk Recorder relies upon GNURadio and OP25. When you install Trunk Recorder you tell it where your system's control channels are, and you define the "center frequency" for your SDR dongles to cover the entire range of a given system. For example, using the Summit County Ohio Regional 800 System, with 20 some voice channels and four possible control channels the configuration looks something like this using four $30 SDR dongles to cover the entire RF range:

Each dongle has a "center" frequency and can listen to approximately 1mhz above and below that center. For brevity, one dongle looks like the section below. This system supports both digital and analog trunked talk-groups, hence the two types of "recorders" which are processes dedicated to listening to a voice channel, decoding the digital data into analog if necessary, and outputting it into a .wav file:

       {
        "center": 851418750.0,
        "rate": 2148000.0,
        "squelch": -60,
        "error": 0,
        "ppm": 0,
        "gain": 350,
        "antenna": "RX",
	"modulation": "fsk4",
        "digitalRecorders": 5,
        "analogRecorders": 2,
        "driver": "osmosdr",
        "device": "rtl=1"
    },

This block defines the possible control channels, defines it as a SmartNet system, instructs Trunk Recorder that the system is rebanded, and defines a script to run after each transmission is recorded.

    "systems": [
	{
        "control_channels": [858712500, 855262500, 857962500, 858512500],
        "type": "smartnet",
	"bandplan": "800_reband",
	"uploadScript": "encode-upload.sh",
        "talkgroupsFile": "ChannelList.csv"
	}]

Just like a a typical consumer grade scanner Trunk Recorder listens to the control channel, but since it has a receiver dedicated to that frequency it is always listening and never stops listening. This keeps it aware of every possible transmission on any of the voice channels, where as a normal scanner would leave the control channel to decode a transmission and potentially miss other simultaneous traffic.

Watching it's output we can see it start a recorder when TG 2384 and TG 33712 started talking:

(info)   [sys_1]   TG: 33712       Freq: 8.51562e+08     Call not found for Update Message, Starting one...
(info)   [sys_1]   TG: 33712       Freq: 8.51562e+08     Starting Recorder on Src: rtl=1
(info)     - Starting P25 Recorder Num [0] TG: 33712     Freq: 8.51562e+08       TDMA: false     Slot: 0
(info)   [sys_1]   TG: 2384        Freq: 8.55088e+08     Starting Recorder on Src: rtl=0
(info)     - Starting P25 Recorder Num [9] TG: 2384      Freq: 8.55088e+08       TDMA: false     Slot: 0
(info)   [sys_1]   TG: 33712       Freq: 8.51562e+08     Assign Retuning - New Freq: 8.5125e+08  Elapsed: 6s     Since update: 2s
(info)   [sys_1]   TG: 2352        Freq: 8.55512e+08     Ending Recorded Call - Last Update: 4s  Call Elapsed: 25
(info)     - Stopping P25 Recorder Num [10]        TG: 2352        Freq: 8.55512e+08       TDMA: false     Slot: 0
(info)   Running upload script: 
./encode-upload.sh /home/scanner/jradio/trunk-recorder/sys_1/2017/7/9/2352-1499630871_8.55512e+08.wav

As you can see from the text above the output is a wav file that may include a single transmission or, should there be very quick back-to-back transmissions that occur on the same voice channel, a string of them in a single .wav file. Each transmission also has an identically named .json file with metadata including the radio IDs and duration of transmission in the file:

{
"freq": 8.55512e+08,
"start_time": 1499630871,
"stop_time": 1499630896,
"emergency": 0,
"talkgroup": 2352,
"srcList": [ {"src": 7609, "time": 1499630871, "pos": 0.000000}, {"src": 24884, "time": 1499630873, "pos": 1.476000}, 
{"src": 4616, "time": 1499630879, "pos"
: 6.300000}, {"src": 7609, "time": 1499630888, "pos": 13.140000}, {"src": 2360, "time": 1499630889, "pos": 13.140000}, 
{"src": 7609, "time": 1499630889, "pos
": 13.140000}, {"src": 2360, "time": 1499630890, "pos": 14.400000} ],
"play_length": 15.300000,
"system": 2,
"freqList": [ { "freq": 855512500.000000, "time": 1499630871, "pos": 0.000000, "len": 123664.000000, 
"error_count": 677.000000, "spike_count": 0.000000} ]
}

Trunk Recorder is similar to using DSD+ on Windows, but it is superior in some respects because it is a single application. There is no need to run multiple instances of the application per VCO or "recorder". There is no need for virtual audio cables. Everything Trunk Recorder does is self-contained within a single application that produces .wav files and then initiates an exteral script so you can do something automatically with those files.

Installing Trunk Recorder

It should be stated up front that what Trunk Recorder is doing is very CPU intensive. You're not going to have success running 3-4 dongles with 3-5 digital recorders on each dongle trying to decode a number of simultaneous convorsations without a reasonable amount of CPU. You'll know your issue is CPU if after running the recorder process you see a bunch of 0's or O's showing.

Installing Trunk Recorder is relatively simple even for a novice Linux user. The official directions can be found on the project's GitHub page, but here is a the quick approach in Ubuntu:

In Ubuntu you would start by installing some pre-requisites:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install gnuradio-dev gr-osmosdr libhackrf-dev libuhd-dev
sudo apt-get install git cmake build-essential libboost-all-dev libusb-1.0-0.dev libssl-dev

Then, make a directory to install Trunk Recorder into, download it with git and "compile" the application:

mkdir trunk-build 
git clone https://github.com/robotastic/trunk-recorder.git 
cd trunk-build
cmake ../trunk-recorder
make

If this works you'll see:

[100%] Built target recorder

Configuring Trunk Recorder

Again, the instructions on how to configure Trunk Recorder are found here on the project's GitHub page. You should especially read the lower portion of that page which explains all of the fields of the config.json file, the format of the ChannelsList.csv file.

The more challenging part of building your config.json is determining how to layout each of your SDR dongles by picking a center frequency that allows you to use as few dongles as possible.

The Trunk Player Web GUI

There are two separate web front ends designed to take the output of Trunk Player and present it in a visually pleasing interface. The author of Trunk Recorder has his own OpenMHZ hosting platform that supports .m4a audio files. More info on that can be found here: https://github.com/robotastic/trunk-recorder/wiki/Uploading-to-OpenMHz

The alternative is Trunk Player, written by Dylan Reinhold, which you can host on your own either entirely internal to your home network, or by utilizing Amazon AWS hosting to push the audio files to the internet (so you can play them from any internet connected device). The front end looks like this:

TrunkPlayer.png

The great thing about Trunk Player is it's scanlist concept. If you have 2-3 systems built either on your own or by working with friends, all pushing audio files from different digital systems, you can group talk groups into logical scanlists instead of system based scanlists.

For example, in the Northeast Ohio's Cuyahoga County there are two digital systems: The State of Ohio MARCS P25 and Cleveland, Ohio's regional "county wide" P25 system it lets suburbs use. The numerous communities that make up Cuyahoga County have either gone with MARCS or Cleveland's system. Everything in Summit County (to the south) is slowly moving from a SmartNet system to a regional P25 of it's own. If you just like to hear about any fire scene regardless of what system it came from then in Trunk Player you just define a "Scan List" with all of the Fire/EMS talk-groups you care about. Then you hear every single transmission (even if some of them occurred at the same time) regardless of which digital system they were derived from. You might hear something 30 or so seconds after it occurred, but you hear the entire transmission as opposed to missing things.

Note: Trunk Player is not necessarily the easiest thing to install, and it will require you to understand how to manipulate the encode_upload.sh scripts that come with Trunk Player to move your .wav files somewhere that they can be served up. Either by utilizing the Amazon AWS S3 concept, or by building your own in-house web server and making those .wav (or converted to .mp3) files available. The front end of Trunk Player simply presents a link to the audio file and the device visiting that webpage retrieves it and plays it.

How do we stream it?

Trunk Recorder will basically generate a folder full of .wav files. We need an application that will connect to Broadcastify and keep the stream online even when nothing is occuring. liquidsoap is the client used to create a stream connection to the broadcastify server's mount point. When there is nothing in it's queue liquidsoap will send silence towards the stream based on the script below.

Every time Trunk Recorder writes a .wav file it will run a script called encode-upload.sh. (The script is for pushing the files to an external location for the web gui above, but you can add commands to it)

For the purposes of streaming we add a call to the encode-upload.sh script to run a separate script called "streamthis" which is provided below.

Streamthis is a python script that compares the talk-group of a wav file (by looking at the prefix of the file name) to a list of defined interesting talk-groups that make up your Broadcastify stream content. If the file just created by Trunk Recorder is one we want to inject into our stream the python script will connect to an already running instance of liquidsoap and add the new .wav file to the queue. If something is in the queue everything will be played in a FIFO fashion.

If the stream is currently idle, the .wav file is immediately played into the stream. However, since a transmission may be as long as a few seconds to upwards of 90 seconds before it is even written into a .wav file and made available to us a slight delay is introduced. It is imperative that the quantity of talk-groups added to a particular stream not be excessive. Due to the way this solution will queue up transmissions and play every single one of them into the stream a very busy talk group may need to be the only one streamed as the potential to end up 5-10 minutes behind could happen. This is a design obligation on the stream owner to avoid over building.

When the queue is empty, the stream sends silence towards Broadcastify.

Basic LiquidSoap Configuration

This liquidsoap script presumes you've already installed liquidsoap on the system that you're going to use to run Trunk Recorder. On Ubuntu you can install liquidsoap easily:

apt-get install liquidsoap

Once installed, this script can be made executable and run on Linux to initiate your stream connection to Broadcastify. NOTE: This stream will connect and appear "Online" indefinately with zero audio until you build the logic to queue .wav files. The configuration below tells liquidsoap to listen on a socket for connections from the streamthis script:

#!/usr/bin/liquidsoap 
#                                  ^-- use -v for verbose otherwise tail the log
set("tag.encodings",["UTF-8","ISO-8859-1"])

# Log dir
set("log.file.path","./basic-radio.log")

# create a socket to send commands to this instance of liquidsoap
set("server.socket",true)
set("server.socket.path","./socket")

# This creates a 1 second silence period generated programmatically (no disk reads) 
silence = blank(duration=1.)

# This pulls the alpha tag out of the wav file 
def append_title(m) =
  # Grab the current title
	#title = '$(if $(title),"$(title)","..> Scanning <..")'
  #     # Return a new title metadata
	[("title",">> Scanning <<")]
end

silence = map_metadata(append_title, silence)

#
myqueue = request.queue()
myqueue = server.insert_metadata(id="S4",myqueue)

# If there is anything in the queue, play it.  If not, play the silence defined above repeatedly:
stream = fallback(track_sensitive=false, [myqueue, silence])

title = '$(if $(title),"$(title)","...Scanning...")'
stream = rewrite_metadata([("title", title)], stream)

output.icecast(%mp3(stereo=false, bitrate=16, samplerate=22050), host="audio1.broadcastify.com", 
 port = 80, password =  "YOURPASSWORD", genre="Scanner", description="Scanner audio", mount="/YOURMOUNTPOINT", 
 name="YOUR STREAM NAME HERE", stream) 

Streamthis Script

This is the streamthis python script that decides if a given .wav file just generated by Trunk Recorder should be injected into the liquidsoap queue. Think of it as the glue between Trunk Recorder's output and liquidsoap's queue mechanism.

The "touch" commands in this script are used by a watchdog process to see if the stream has died and are optional. These were built because my analog system would occasionally die, and sometimes liquidsoap just goes to lunch. The watchdog script pays attention to how often "things" are happening, and takes automatic action to restart processes if it thinks the stream has gone to lunch. Once I clean that logic up I'll upload the watchdog script as well.

#!/usr/bin/python
import socket
import sys
import os

filepath = sys.argv[1]
filename = os.path.basename(filepath)
a = filename.split('-')
tg,junk = a[0],a[1]

def touch(fname, times=None):
    with open(fname, 'a'):
        os.utime(fname, times)

# for debug purposes
#print "tg is %s" % tg

# This array defines all of the decimal trunk-groups you want to inject into the stream queue:
streamthese = ["33328", "41168", "41200", "42576","42608","33936","41264","33904","33872","6","7","8"]

if tg in streamthese:
    print ">>>>> STREAMTHIS:   tg is in list"
    match = 1
    touch('/tmp/streamthis-lastrun')
else:
    print ">>>>> STREAMTHIS:   tg is not in the list"
    match = 0
    # silently exit
    exit()

# Create a UDS socket
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)

# Connect the socket to the port where the server is listening
server_address = '/PATH/TO/WHERE/LIQUIDSOAP/socket'
#print >>sys.stderr, 'connecting to %s' % server_address
try:
        sock.connect(server_address)
except socket.error, msg:
        print >>sys.stderr, msg
        sys.exit(1)

try:
    # Send data
    message = 'queue.push {0}\n\r'.format(filepath)
    print "message: {0}".format(message)
    #print >>sys.stderr, 'sending "%s' % message
    sock.sendall(message)

    amount_received = 0
    amount_expected = len(message)
    
    data = sock.recv(16)
    amount_received += len(data)
    #print >>sys.stderr, 'received "%s' % data

finally:
    #print >>sys.stderr, 'closing socket'
    match = 0
    sock.close()

Modifying encode-upload.sh

As I wrote this wiki page I realized the current version of encode_upload.sh is dramatically different than what I've been using. Therefor, I'll show you the modifications I make that you can add to whatever the current version of encode_upload.sh happens to become:

The encode_upload.sh script is passed a path to the .wav file by Trunk Recorder. That value is stored in the variable $1 and used by the shell script. We need to tear that path down and define some variables, so at the top of your encode_upload.sh script add the following if parts of this do not exist already.

CSV_FILE="/home/scanner/radio/dev/trunk-recorder/ChannelList.csv"   # Path to YOUR Trunk Recorder CSV file
filename="$1"                                              # This should already exist
basename="${filename%.*}"                      # This should already exist
filename_only=$(basename $basename)
dir="$(dirname $filename)" 
mp3=${dir}/${filename_only}.mp3

I've started converting my .wav file to mp3 prior to sending it to liquidsoap. I believe this takes the burden of transcoding the audio away from liquidsoap, while also giving me an opportunity to inject the Alpha Tag into the .mp3 file. The Alpha tag is derived from the CSV file defined above, which is required to start Trunk Recorder in the first place. Unfortunately its not currently available in the .json.

I place this into the encode_upload.sh file somewhere near the bottom (after it has done any file transfer/uploads):

# this is for streamthis - lets convert to mp3 here, then delete the .wav on exit
# Start by finding the alpha tag for the talk-group:
tg="$( cut -d '-' -f 1 <<< "$filename_only" )"
ALPHA=`grep "^${tg}," ${CSV_FILE} | cut -d , -f 4`

if [ -z "$ALPHA" ]
then
	ALPHA="TG: ${tg}"
fi

The following command uses lame to both add the alpha tag and run a highpass filter. This removes any noise such as PL hum if you're injecting any analog audio:

# Insert the alpha tag into a mp3 with highpass:
lame --preset voice --highpass 150 --tt "${ALPHA}" $filename $mp3

Finally, we tell the streamthis script the fully qualified path to the MP3 file we just added an alpha tag to:

/home/scanner/liquid/streamthis $mp3

...if the talk-group in that MP3 matches those you defined above in the "streamthese" array, then the streamthis script will inject the path to the MP3 into the queue and the transmission will get played into your stream

From the logfile of liquidsoap we can see a new transmission arriving in the queue and the transition from silence, to queue, and back to silence. The liquidsoap script provided above will extract the Alpha Tag of a given transmissions out of the .mp3 file as it is added to the queue. When the queue is not being played the while() loop changes the ALpha Tag to ">>> Scanning <<<" or whatever text you choose to use. (Alpha tag insertion takes place back in encode_upload.sh)

2017/07/10 01:24:27 [map_metadata_4977:3] Inserting missing metadata.
2017/07/10 01:24:28 [map_metadata_4977:3] Inserting missing metadata.
2017/07/10 01:24:28 [server:3] New client unix socket "\232\177\143\127".
2017/07/10 01:24:28 [decoder:3] Method "MAD" accepted "/PATH/TO/sys_1/2017/7/10/42576-1499664243_8.51562e+08.mp3".
2017/07/10 01:24:28 [server:3] Client unix socket "\232\177\143\127" disconnected without saying goodbye..!
2017/07/10 01:24:28 [queue:3] Prepared "/PATH/TO/sys_1/2017/7/10/42576-1499664243_8.51562e+08.mp3" (RID 5).
2017/07/10 01:24:28 [fallback_4983:3] Switch to S4 with transition.
2017/07/10 01:24:38 [queue:3] Finished with "/PATH/TO/sys_1/2017/7/10/42576-1499664243_8.51562e+08.mp3".
2017/07/10 01:24:38 [fallback_4983:3] Switch to map_metadata_4977 with forgetful transition.
2017/07/10 01:24:38 [map_metadata_4977:3] Inserting missing metadata.
2017/07/10 01:24:39 [map_metadata_4977:3] Inserting missing metadata.

Note: If anyone familiar with liquidsoap can find a way to only insert the the "Scanning" metadata once instead of every single second while still being able to immediately interrupt with a queue MP3 I am open to suggestions. This constant inserting seems inefficient and unnecessary.

Multiple Streams from one System

Using the original contents of this article, I have modified it a little to fit my needs. I broadcast about 8 streams from the Starcom system in St. Clair County, IL.

Notes:

I placed the recordings in /tmp/capture because the files will be cleaned up daily by systemd. This prevents them from running the disk space out.

The system whee this is running is an Intel(R) Pentium(R) CPU G3258 @ 3.20GHz, with 8GB of ram, and a 60GB disk.

With this configuration, it can do about 8 streams and about 8 digital recorders which is enough for the system in this county.

top - 16:22:17 up 3 days,  1:37,  1 user,  load average: 3.27, 3.13, 3.15
Tasks: 154 total,   1 running, 153 sleeping,   0 stopped,   0 zombie
%Cpu(s): 63.5 us,  6.8 sy,  0.0 ni, 29.4 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem :  8067164 total,   125436 free,   915916 used,  7025812 buff/cache
KiB Swap:  2097148 total,  2097148 free,        0 used.  6716772 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 1166 medrc     20   0 2643756 150776  60644 S 137.3  1.9   5354:27 recorder
 1127 liquids+  20   0  624700  73120  12056 S   1.0  0.9  38:44.14 liquidsoap
 1038 liquids+  20   0  631464  22072   7372 S   0.7  0.3  23:06.58 liquidsoap
 1095 liquids+  20   0  624700  52236  12044 S   0.7  0.6  23:35.93 liquidsoap
 1108 liquids+  20   0  624700  72972  12024 S   0.7  0.9  18:06.16 liquidsoap
 1117 liquids+  20   0  624700  62960  12196 S   0.7  0.8  27:22.28 liquidsoap
  251 root      20   0  441356 258004 257172 S   0.3  3.2   2:28.86 systemd-journal
  799 systemd+  20   0   66288   6728   5384 S   0.3  0.1  10:54.13 systemd-resolve
 1026 liquids+  20   0  631464  22244   7228 S   0.3  0.3  22:52.92 liquidsoap
 1078 liquids+  20   0  624700  61992  12188 S   0.3  0.8  24:02.62 liquidsoap
 1138 liquids+  20   0  624700  60952  12296 S   0.3  0.8  23:42.38 liquidsoap
 1149 liquids+  20   0  624700  57928  11896 S   0.3  0.7  21:48.18 liquidsoap
 1159 liquids+  20   0  631464  22244   7592 S   0.3  0.3  22:36.31 liquidsoap

To make it all start on reboot:

sudo systemctl daemon-reload
sudo systemctl enable recorder
sudo systemctl enable liquidsoap

/opt/recorder/config.json

{
    "sources": [{
        "center": 854406250.0,
        "rate": 10000000,
        "error": 0,
        "squelch": -90,
        "gain": 54.0,
        "antenna": "RX2",
        "digitalLevels": 3,
        "digitalRecorders": 8,
        "analogRecorders": 2,
        "driver": "osmosdr",
        "device": "airspy",
        "modulation": "QPSK"
    }],
    "systems": [{
        "control_channels": [851225000,851562500,852162500],
        "type": "p25",
        "recordUnknown": true,
        "shortName": "stclairco",
        "apiKey": "",
	"uploadServer": "https://api.openmhz.com",
        "talkgroupsFile": "/opt/recorder/ChannelList.csv",
        "uploadScript": "encode_upload.sh"
     }],
    "defaultMode": "analog",
    "callTimeout": 5,
    "logFile": false,
    "uploadServer": "https://api.openmhz.com",
    "captureDir": "/tmp/capture"
}

/etc/systemd/system/recorder.service

[Unit]
Description = Starting Trunk Recorder Service
After = network.target liquidsoap.service

[Service]
Type=simple
User=medrc
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=recorder
WorkingDirectory=/opt/recorder
ExecStart=/opt/recorder/recorder --config=/opt/recorder/config.json
Restart=on-abort

[Install]
WantedBy = multi-user.target

/opt/recorder/stream.liq

This is the base stream file used by each stream specific file.

set("tag.encodings",["UTF-8","ISO-8859-1"])

# Configure Logging
set("log.file",false)
set("log.level",1)
set("log.stdout",false)
set("log.syslog",true)
set("log.syslog.facility","DAEMON")
set("log.syslog.program","liquidsoap-#{STREAMID}")

# create a socket to send commands to this instance of liquidsoap
set("server.socket",true)
set("server.socket.path","<sysrundir>/#{STREAMID}.sock")
set("server.socket.permissions",511)
# This creates a 1 second silence period generated programmatically (no disk reads)
silence = blank(duration=1.)

# This pulls the alpha tag out of the wav file
def append_title(m) =
	[("title",">> Scanning <<")]
end

silence = map_metadata(append_title, silence)

recorder_queue = request.queue()
recorder_queue = server.insert_metadata(id="S4",recorder_queue)

# If there is anything in the queue, play it.  If not, play the silence defined above repeatedly:
stream = fallback(track_sensitive=false, [recorder_queue, silence])

title = '$(if $(title),"$(title)","...Scanning...")'
stream = rewrite_metadata([("title", title)], stream)

output.icecast( %mp3(stereo=false, bitrate=16, samplerate=22050),
  host=HOST, port=80, password=PASSWORD, genre="Scanner",
  description="Scanner audio", mount=MOUNT,  name=NAME, user="source", stream)

/opt/recorder/liquidsoap/<streamid>.liq

Create a symlink to this file at /etc/liquidsoap/<streamid>.liq

sudo ln -s /opt/recorder/liquidsoap/<streamid>.liq /etc/liquidsoap/<streamid>.liq

This will cause liquidsoap to start each of these streams on startup.

#!/usr/bin/liquidsoap
HOST="audioX.broadcastify.com"
MOUNT="/mount"
PASSWORD="password"
NAME="name of the stream"
STREAMID="<streamid>"
%include "/opt/recorder/stream.liq"

/opt/recorder/encode_upload.sh

#! /bin/bash
CSV_FILE="/opt/recorder/ChannelList.csv"   # Path to YOUR Trunk Recorder CSV file
#echo "Encoding: $1"
filename="$1"
basename="${filename%.*}"
filename_only=$(basename $basename)
dir="$(dirname $filename)"
mp3=${dir}/${filename_only}.mp3
#mp3encoded="$basename.mp3"
#mp4encoded="$basename.m4a"
json="$basename.json"

# this is for streamthis - lets convert to mp3 here, then delete the .wav on exit
# Start by finding the alpha tag for the talk-group:
tg="$( cut -d '-' -f 1 <<< "$filename_only" )"
ALPHA=`grep "^${tg}," ${CSV_FILE} | cut -d , -f 4`
GROUP=`grep "^${tg}," ${CSV_FILE} | cut -d , -f 5`

if [ -z "$ALPHA" ]
then
  ALPHA="TG: ${tg}"
fi

# Insert the alpha tag into a mp3 with highpass:
#lame --quiet --preset voice --highpass 150 --tt "${ALPHA}" $filename $mp3
lame --quiet --preset voice --tt "${ALPHA}" $filename $mp3
#echo "Resulting Alpha is ${ALPHA}"
#echo "Resulting Group is ${GROUP}"
echo "Submitting [${tg}] ${ALPHA}" | logger -p info -t encode_upload
/opt/recorder/streamthis $mp3

/opt/recorder/streamthis

#!/usr/bin/python
import socket
import sys
import os
import logging
import logging.handlers


logger = logging.getLogger('streamthis')
logger.setLevel(logging.INFO)
handler = logging.handlers.SysLogHandler(address = '/dev/log')
formatter = logging.Formatter('streamthis: %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)


filepath = sys.argv[1]
filename = os.path.basename(filepath)
a = filename.split('-')
tg,junk = a[0],a[1]

def touch(fname, times=None):
    with open(fname, 'a'):
        os.utime(fname, times)

# This array defines all of the decimal trunk-groups you want to inject into the stream queue:
ofallon = ["7199","7200","7201","7202","7203","7205","7206","7207","7208","7209","7210","7211","7212","7214"]
fairview = ["7219","7220","7221","7222","7223","7224","7225","7226","7227","7228","7229","7230","7231"]
#ofallon = ["7199","7200","7201","7202","7203","7205","7206","7207","7208","7209","7210","7211","7212","7214", "7215", "7216", "7204", "7213", "7217", "7218", "7219","7220","7221","7222","7223","7224","7225","7226","7227","7228","7229","7230","7231"]
ares = ["1", "2"]

# idot
#ares = ["7776", "7777", "7778", "7781", "7782", "7788", "7789", "8612", "8613", "8614", "8615", "8616", "8617", "8618", "8619", "8620", "8621", "8622"]

#isp district 11 small
#ares = ["17000", "17001", "17003", "17017", "17018", "17031", "17037", "17043", "17049", "17055", "17078"]

#isp district 11 large
isp = ["17000", "17001", "17003", "17017", "17018", "17031", "17037", "17043", "17049", "17055", "17078", "17065", "17066", "17067", "17068", "17069", "17070", "17071", "17075", "17084", "17085", "17086", "17096", "17097", "17098", "17102", "17103"]
troy = ["7961", "8201"]
fddisp = ["7019", "7023", "7028", "7189", "7079", "7271", "7290", "7221", "7201", "7097"]
belleville = ["7236", "7237", "7252"]
swansea = ["7245", "7243", "7252"]
estl = ["7162", "7188", "7187", "7190", "7252"]
#ofallon = ["7199", "7200", "7201", "7202", "7209", "7211"]
#fairview = ["7219", "7229", "7220", "7221"]
scclaw = ["7017", "7056", "7021", "7018", "7027", "7022", "7188", "7250", "7024", "7236", "7220", "7200", "7211"]
match = 0

servers = []

if tg in ofallon:
    logger.info(">>>>> OFALLON STREAM: tg is in list")
    match = 1
    touch('/tmp/streamthis-ofallon-lastrun')
    # Connect the socket to the port where the server is listening
    server_address = '/var/run/liquidsoap/ofallon.sock'
    servers.append(server_address)

if tg in fairview:
    logger.info(">>>>> FAIRVIEW STREAM: tg is in list")
    match = 1
    touch('/tmp/streamthis-fairview-lastrun')
    # Connect the socket to the port where the server is listening
    server_address = '/var/run/liquidsoap/fairview.sock'
    servers.append(server_address)

if tg in isp:
    logger.info(">>>>> ISP STREAM: tg is in list")
    match = 1
    touch('/tmp/streamthis-isp-lastrun')
    # Connect the socket to the port where the server is listening
    server_address = '/var/run/liquidsoap/isp.sock'
    servers.append(server_address)

if tg in troy:
    logger.info(">>>>> TROY STREAM: tg is in list")
    match = 1
    touch('/tmp/streamthis-troy-lastrun')
    # Connect the socket to the port where the server is listening
    server_address = '/var/run/liquidsoap/troy.sock'
    servers.append(server_address)

if tg in fddisp:
    logger.info(">>>>> FIRE STREAM: tg is in list")
    match = 1
    touch('/tmp/streamthis-fddisp-lastrun')
    # Connect the socket to the port where the server is listening
    server_address = '/var/run/liquidsoap/fddisp.sock'
    servers.append(server_address)

if tg in belleville:
    logger.info(">>>>> BELLEVILLE STREAM: tg is in list")
    match = 1
    touch('/tmp/streamthis-belleville-lastrun')
    # Connect the socket to the port where the server is listening
    server_address = '/var/run/liquidsoap/bellevile.sock'
    servers.append(server_address)

if tg in swansea:
    logger.info(">>>>> SWANSEA STREAM: tg is in list")
    match = 1
    touch('/tmp/streamthis-swansea-lastrun')
    # Connect the socket to the port where the server is listening
    server_address = '/var/run/liquidsoap/swansea.sock'
    servers.append(server_address)

if tg in estl:
    logger.info(">>>>> ESTL STREAM: tg is in list")
    match = 1
    touch('/tmp/streamthis-estl-lastrun')
    # Connect the socket to the port where the server is listening
    server_address = '/var/run/liquidsoap/estl.sock'
    servers.append(server_address)

if tg in scclaw:
    logger.info(">>>>> SCCLAW STREAM: tg is in list")
    match = 1
    touch('/tmp/streamthis-scclaw-lastrun')
    # Connect the socket to the port where the server is listening
    server_address = '/var/run/liquidsoap/scclaw.sock'
    servers.append(server_address)

if match == 0:
    logger.debug(">>>>> STREAMTHIS:   tg is not in the list")
    exit()

for server_address in servers:

  logger.info('%s %s', "sending to: ", server_address)

  # Create a UDS socket
  sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)

  try:
    sock.connect(server_address)
  except socket.error, msg:
      print >>sys.stderr, msg
      sys.exit(1)

  try:
    # Send data
    message = 'queue.push {0}\n\r'.format(filepath)
    sock.sendall(message)

    amount_received = 0
    amount_expected = len(message)

    data = sock.recv(16)
    amount_received += len(data)
    #print >>sys.stderr, 'received "%s' % data

  finally:
    #print >>sys.stderr, 'closing socket'
    match = 0
    sock.close()

Example Log Output

As you can see, some talkgroups are in more than one stream. An example would be East St. Louis Police. You can see below they are in the "East St. Louis Fire, Police and EMS" stream as well as the "St. Clair County Law Enforcement" channel.

Mar 31 15:54:28 medrc-recorder-1 recorder[1166]: [2018-03-31 15:54:28.007373] (info)   Running upload script: ./encode_upload.sh /tmp/capture/stclairco/2018/3/31/8619-1522529624_8.53138e+08.wav &
Mar 31 15:54:28 medrc-recorder-1 recorder[1166]: [2018-03-31 15:54:28.044887] (info)   Currently Active Calls: 3
Mar 31 15:54:28 medrc-recorder-1 recorder[1166]: [2018-03-31 15:54:28.045001] (info)   #011[ 2 ] State: 3
Mar 31 15:54:28 medrc-recorder-1 recorder[1166]: [2018-03-31 15:54:28.045039] (info)   #011[ 3 ] State: 3
Mar 31 15:54:28 medrc-recorder-1 recorder[1166]: [2018-03-31 15:54:28.045073] (info)   #011[ 4 ] State: 3
Mar 31 15:54:28 medrc-recorder-1 recorder[1166]: [2018-03-31 15:54:28.045086] (info)   Recorders:
Mar 31 15:54:28 medrc-recorder-1 recorder[1166]: [2018-03-31 15:54:28.045096] (info)   [ airspy ]
Mar 31 15:54:28 medrc-recorder-1 recorder[1166]: [2018-03-31 15:54:28.045107] (info)   [ 0 ] State: 2
Mar 31 15:54:28 medrc-recorder-1 recorder[1166]: [2018-03-31 15:54:28.045120] (info)   [ 1 ] State: 2
Mar 31 15:54:28 medrc-recorder-1 recorder[1166]: [2018-03-31 15:54:28.045133] (info)   [ 2 ] State: 3
Mar 31 15:54:28 medrc-recorder-1 recorder[1166]: [2018-03-31 15:54:28.045145] (info)   [ 3 ] State: 3
Mar 31 15:54:28 medrc-recorder-1 recorder[1166]: [2018-03-31 15:54:28.045156] (info)   [ 4 ] State: 3
Mar 31 15:54:28 medrc-recorder-1 recorder[1166]: [2018-03-31 15:54:28.045167] (info)   [ 5 ] State: 2
Mar 31 15:54:28 medrc-recorder-1 recorder[1166]: [2018-03-31 15:54:28.045179] (info)   [ 6 ] State: 2
Mar 31 15:54:28 medrc-recorder-1 recorder[1166]: [2018-03-31 15:54:28.045191] (info)   [ 7 ] State: 2
Mar 31 15:54:28 medrc-recorder-1 encode_upload: Submitting [8619] T8 Bowman
Mar 31 15:54:34 medrc-recorder-1 recorder[1166]: [2018-03-31 15:54:34.011654] (info)   Running upload script: ./encode_upload.sh /tmp/capture/stclairco/2018/3/31/7951-1522529659_8.51175e+08.wav &
Mar 31 15:54:34 medrc-recorder-1 encode_upload: Submitting [7951] SIUE Police
Mar 31 15:54:39 medrc-recorder-1 recorder[1166]: [2018-03-31 15:54:39.012561] (info)   Running upload script: ./encode_upload.sh /tmp/capture/stclairco/2018/3/31/7993-1522529663_8.5375e+08.wav &
Mar 31 15:54:39 medrc-recorder-1 encode_upload: Submitting [7993] Monr SheriffLaw
Mar 31 15:54:48 medrc-recorder-1 recorder[1166]: [2018-03-31 15:54:48.009675] (info)   Running upload script: ./encode_upload.sh /tmp/capture/stclairco/2018/3/31/7027-1522529628_8.5385e+08.wav &
Mar 31 15:54:48 medrc-recorder-1 encode_upload: Submitting [7027] SCC Law Disp 4
Mar 31 15:54:48 medrc-recorder-1 streamthis: >>>>> SCCLAW STREAM: tg is in list
Mar 31 15:54:48 medrc-recorder-1 streamthis: sending to:  /var/run/liquidsoap/scclaw.sock
Mar 31 15:55:01 medrc-recorder-1 CRON[10784]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Mar 31 15:55:11 medrc-recorder-1 recorder[1166]: [2018-03-31 15:55:11.012547] (info)   Running upload script: ./encode_upload.sh /tmp/capture/stclairco/2018/3/31/7027-1522529690_8.527e+08.wav &
Mar 31 15:55:11 medrc-recorder-1 encode_upload: Submitting [7027] SCC Law Disp 4
Mar 31 15:55:11 medrc-recorder-1 streamthis: >>>>> SCCLAW STREAM: tg is in list
Mar 31 15:55:11 medrc-recorder-1 streamthis: sending to:  /var/run/liquidsoap/scclaw.sock
Mar 31 15:55:14 medrc-recorder-1 recorder[1166]: [2018-03-31 15:55:14.012353] (info)   Running upload script: ./encode_upload.sh /tmp/capture/stclairco/2018/3/31/7315-1522529684_8.53138e+08.wav &
Mar 31 15:55:14 medrc-recorder-1 encode_upload: Submitting [7315] MadisonSheriff 1
Mar 31 15:55:40 medrc-recorder-1 recorder[1166]: [2018-03-31 15:55:40.011816] (info)   Running upload script: ./encode_upload.sh /tmp/capture/stclairco/2018/3/31/7188-1522529722_8.5385e+08.wav &
Mar 31 15:55:40 medrc-recorder-1 encode_upload: Submitting [7188] EStL Police 1
Mar 31 15:55:40 medrc-recorder-1 streamthis: >>>>> ESTL STREAM: tg is in list
Mar 31 15:55:40 medrc-recorder-1 streamthis: >>>>> SCCLAW STREAM: tg is in list
Mar 31 15:55:40 medrc-recorder-1 streamthis: sending to:  /var/run/liquidsoap/estl.sock
Mar 31 15:55:40 medrc-recorder-1 recorder[1166]: [Errno 111] Connection refused
Mar 31 15:55:44 medrc-recorder-1 recorder[1166]: [2018-03-31 15:55:44.011828] (info)   Running upload script: ./encode_upload.sh /tmp/capture/stclairco/2018/3/31/7027-1522529730_8.517e+08.wav &
Mar 31 15:55:44 medrc-recorder-1 encode_upload: Submitting [7027] SCC Law Disp 4
Mar 31 15:55:44 medrc-recorder-1 streamthis: >>>>> SCCLAW STREAM: tg is in list
Mar 31 15:55:44 medrc-recorder-1 streamthis: sending to:  /var/run/liquidsoap/scclaw.sock
Mar 31 15:55:50 medrc-recorder-1 recorder[1166]: [2018-03-31 15:55:50.007195] (info)   Running upload script: ./encode_upload.sh /tmp/capture/stclairco/2018/3/31/7315-1522529731_8.53138e+08.wav &
Mar 31 15:55:50 medrc-recorder-1 encode_upload: Submitting [7315] MadisonSheriff 1
Mar 31 15:55:51 medrc-recorder-1 recorder[1166]: [2018-03-31 15:55:51.003113] (info)   Running upload script: ./encode_upload.sh /tmp/capture/stclairco/2018/3/31/7028-1522529744_8.5375e+08.wav &
Mar 31 15:55:51 medrc-recorder-1 encode_upload: Submitting [7028] Caseyville Fire
Mar 31 15:55:51 medrc-recorder-1 streamthis: >>>>> FIRE STREAM: tg is in list
Mar 31 15:55:51 medrc-recorder-1 streamthis: sending to:  /var/run/liquidsoap/fddisp.sock
Mar 31 15:55:53 medrc-recorder-1 recorder[1166]: [2018-03-31 15:55:53.013964] (info)   Running upload script: ./encode_upload.sh /tmp/capture/stclairco/2018/3/31/7096-1522529737_8.5385e+08.wav &
Mar 31 15:55:53 medrc-recorder-1 encode_upload: Submitting [7096] SAFB RampNet

Improved Multiple Stream Support

Combined the encode_upload.sh and the streamthis.py scripts into a single script. Added an additional column to the ChannelList.csv file with a pipe delimited list of streams for each talkgroup.

/opt/recorder/ChannelList.csv

Decimal,Hex,Mode,Alpha Tag,Description,Tag,Tag,Priority,Stream List 7199,1C1F,D,O Fallon Emergency Button,O Fallon Emergency Button,Law Dispatch,Law Dispatch,0,ofallon|scclaw 7219,1C33,D,Fairview Heights Emergency Button,Fairview Heights Emergency Button,Emergency Ops,Emergency Ops,15,fairview|scclaw 7235,1C43,De,Belleville Emergency Button,Belleville Emergency Button,Law Tac,Law Tac,30,belleville|scclaw

/opt/recorder/encode_upload.py

#!/usr/bin/env python3
import socket
import sys
import os
from subprocess import call
from subprocess import PIPE, run

import logging
from logging.config import fileConfig

fileConfig('logging_config.ini')
logger = logging.getLogger('streamthis')

CSV_FILE="/opt/recorder/ChannelList.csv"
FILE_TO_ENCODE=sys.argv[1]
logger.debug("file to encode: %s", FILE_TO_ENCODE)

def touch(fname, times=None):
    with open(fname, 'a'):
        os.utime(fname, times)

WAV_FILE=os.path.basename(FILE_TO_ENCODE)
logger.debug("WAV_FILE: %s", WAV_FILE)

DIRECTORY=os.path.dirname(FILE_TO_ENCODE)
logger.debug("DIRECTORY: %s", DIRECTORY)

a = WAV_FILE.split('.')
FILENAME = a[0]
logger.debug("FILENAME: %s", FILENAME)

MP3_FILE="{0}/{1}.mp3".format(DIRECTORY, FILENAME)
logger.debug("MP3_FILE: %s", MP3_FILE)

a = FILENAME.split('-')
TALKGROUP = a[0]
logger.debug("TALKGROUP: %s", TALKGROUP)

command = ['/bin/grep', TALKGROUP, CSV_FILE]
result = run(command, stdout=PIPE, stderr=PIPE, universal_newlines=True)
#print(result.returncode, result.stdout, result.stderr)
csvline = result.stdout
logger.debug("csvline: %s", csvline)
a = csvline.split(',')
ALPHA, STREAM_LIST = a[3], a[8]
logger.debug("ALPHA: %s", ALPHA)
logger.debug("STREAM_LIST: %s", STREAM_LIST)

if (STREAM_LIST.strip() == ''):
    logger.info("No Stream for [%s] %s", TALKGROUP, ALPHA)
    quit(0)

logger.debug("Calling: %s %s %s %s %s %s %s", "/usr/bin/lame", "--quiet", "--preset", "voice", "--tt", "\"{0}\"".format(ALPHA), FILE_TO_ENCODE)
call(["/usr/bin/lame", "--quiet", "--preset", "voice", "--tt", "\"{0}\"".format(ALPHA), FILE_TO_ENCODE])

if os.path.exists(FILE_TO_ENCODE):
  os.remove(FILE_TO_ENCODE)

streams = STREAM_LIST.split('|')
servers = []

for stream in streams:
    logger.debug(">>>>> %s, %s is in the list", stream, TALKGROUP)
    touch("/tmp/streamthis-{0}-lastrun".format(stream))
    stream_address = "/var/run/liquidsoap/{0}.sock".format(stream)
    logger.debug("[%s] matched for %s, sending to: %s", TALKGROUP, stream, stream_address)
    servers.append(stream_address)

if len(servers) < 1:
    logger.error("Talkgroup [%s] did not have any streams in the list: %s", TALKGROUP, STREAM_LIST)
    exit(1)

for server_address in servers:
    logger.debug("address: %s", server_address)
    # Create a UDS socket
    sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)

    try:
        sock.connect(server_address)
    except socket.error as msg:
        sys.exit(1)

    try:
        # Send data
        logger.info('%s %s %s', TALKGROUP, "sending to: ", server_address)
        logger.debug('queue.push {0}\n\r'.format(MP3_FILE))
        message = 'queue.push {0}\n\r'.format(MP3_FILE)
        sock.sendall(message)

        amount_received = 0
        amount_expected = len(message)

        data = sock.recv(16)
        amount_received += len(data)

    finally:
        match = 0
        sock.close()

Example Log Output

Dec 31 23:51:48 medrc-recorder-1 recorder[1433]: [2018-12-31 23:51:48.007674] (info)   Running upload script: ./encode_upload.sh /tmp/capture/stclairco/2018/12/31/7057-1546321892_8.5375e+08.wav &
Dec 31 23:51:48 medrc-recorder-1 encode_upload: lame --quiet --preset voice --tt SCC IREACH Patch /tmp/capture/stclairco/2018/12/31/7057-1546321892_8.5375e+08.wav
Dec 31 23:51:48 medrc-recorder-1 encode_upload: Submitting [7057] SCC IREACH Patch to scclaw

Code to Drop if Queue is Full

Big thanks to the above contributors. You examples are great. However my system is a very large metro area and without logic to limit the LiquidSoap queue, entries get stale very quickly. The following is crude code I wrote to enable a queue awareness to prevent adding more entries if queue is above a coded value. Once upon a time I wrote crude Perl scripts to do automation and this annoyance allowed me to learn a bit of python. My apologies to those whom know better.

It's a small thing but maybe it will help someone down the road if faced with a similar issue. The liquidsoap documentation was a struggle.

The logic is simple, but a quick rundown might help. First file is to give us an easy way to see when a file was recorded when it plays. Second file shows modifications to enable the telnet server on Liquidsoap and how I substituted shoutcast for icecast. Third file is where the work is done. Define a new array of tg that will always be added to the queue. Add a telnet job to gather the current secondary queue depth. Compare that value with an arbitrary number and drop if greater than.

========================================
~/trunk-build/encode-local-sys-0.sh
========================================
#(Optional:) Tack a timestamp on to the end of the ALPHA title for troubleshooting queue delay
ALPHA=`grep "^${tg}," ${CSV_FILE} | cut -d , -f 4` #as in original
CTIME=`date +%R`
FULLTAG="$ALPHA , Recorded at:$CTIME"
lame --nohist --gain 8 --preset voice --tt "${FULLTAG}" $filename $mp3encoded


========================================
~/liquid/startliq.sh
========================================
#Add a telnet interface to allow queue querries
set("server.telnet",true)
set("server.telnet.bind_addr","192.168.1.x")

#Modified for Shoutcast
output.shoutcast(%mp3(stereo=false, bitrate=16, samplerate=22050), host="localhost",
 port = 8000, password =  "YOURPASS", name="STREAMNAME", stream)


========================================
~/liquid/streamthis
========================================
import telnetlib     #add telnet module

# This array defines all of the decimal trunk-groups you want to inject into the stream queue:
streamthese = ["9101", "9102", "9103", "9104","9105","9106","9107","9108","9109","9112","9145"] #previously existed
queuepri = ["9112", "9145"]   #these are new to override queue dump further on


#below lines exist already, for landmarks
if tg in streamthese:
   <clip>
else:
   <clip>
#end existing lines

#Next part goes below the above reference if/else, so only runs if get through that.
#New code to telnet to LiquidSoap and see how deep secondary queue is and dump/exit if queue too big, unless given a pass
HOST = "192.168.1.x"
tn = telnetlib.Telnet(HOST, 1234)
tn.write("\nlist\n")
tn.read_until("END")
tn.write("queue.secondary_queue\n")
secq = tn.read_until("END")
## When there is no secondary queue will return 1 for the END. n-1 will be actual queue size
seclist = secq.split()
print ">>>>> STREAMTHIStel:   Sec len is ", len(seclist)

if len(seclist) > 4 and tg not in queuepri:
        print ">>>>> STREAMTHIStel:  Queue bigger than 3 and tg not priority"
        exit()



Copyright 2019 by RadioReference.com LLC Privacy Policy  |  Terms and Conditions