Actions

Streaming with Trunk Recorder and Liquidsoap

From The RadioReference Wiki

Introduction

Trunk Recorder is a free Linux based application written by Luke Berndt that can decode the following:

  • Trunked P25 & SmartNet Systems
  • Conventional P25 & analog systems, where each group has a dedicated RF channel
  • P25 Phase 1, P25 Phase 2 & Analog voice channels

Using a number of supported SDR hardware dongles Trunk Recorder can take entire P25 systems and simultaneously record every single transmission occurring at any moment in time. This is accomplished by using enough SDR dongles to cover the entire spectrum of voice and control channels in a system.

From a stream provider perspective this gives us the ability to put all transmissions into a stream without losing those that occurred while a physical scanner would be "stuck" on another channel. The cost is a slight delay caused by queuing the transmissions, but the severity of that queue depth is dependent upon how many talk-groups you stream and how chatty they are.

Now that Trunk Recorder supports analog its possible to combine analog and digital transmissions into the stream.

The "streamthis" python script below was written to bridge the gap between the .wav files Trunk Recorder creates, and an idle liquidsoap client already connected to Broadcastify. It made it possible to build the Twinsburg, Hudson and Reminderville Police, Fire / EMS stream and has worked quite well. That particular stream is sourced from two separate computers, one of which is dedicated to conventional analog receive. As those files are generated they are copied to the primary computer via ssh, and the streamthis script is remotely executed to insert the newly delivered analog .mp3 file to the stream queue.

How Trunk Recorder Works

Under the hood Trunk Recorder relies upon GNURadio and OP25. When you install Trunk Recorder you tell it where your system's control channels are, and you define the "center frequency" for your SDR dongles to cover the entire range of a given system. For example, using the Summit County Ohio Regional 800 System, with 20 some voice channels and four possible control channels the configuration looks something like this using four $30 SDR dongles to cover the entire RF range:

Each dongle has a "center" frequency and can listen to approximately 1mhz above and below that center. For brevity, one dongle looks like the section below. This system supports both digital and analog trunked talk-groups, hence the two types of "recorders" which are processes dedicated to listening to a voice channel, decoding the digital data into analog if necessary, and outputting it into a .wav file:

       {
        "center": 851418750.0,
        "rate": 2148000.0,
        "squelch": -60,
        "error": 0,
        "ppm": 0,
        "gain": 350,
        "antenna": "RX",
	"modulation": "fsk4",
        "digitalRecorders": 5,
        "analogRecorders": 2,
        "driver": "osmosdr",
        "device": "rtl=1"
    },

This block defines the possible control channels, defines it as a SmartNet system, instructs Trunk Recorder that the system is rebanded, and defines a script to run after each transmission is recorded.

    "systems": [
	{
        "control_channels": [858712500, 855262500, 857962500, 858512500],
        "type": "smartnet",
	"bandplan": "800_reband",
	"uploadScript": "encode-upload.sh",
        "talkgroupsFile": "ChannelList.csv"
	}]

Just like a a typical consumer grade scanner Trunk Recorder listens to the control channel, but since it has a receiver dedicated to that frequency it is always listening and never stops listening. This keeps it aware of every possible transmission on any of the voice channels, where as a normal scanner would leave the control channel to decode a transmission and potentially miss other simultaneous traffic.

Watching it's output we can see it start a recorder when TG 2384 and TG 33712 started talking:

(info)   [sys_1]   TG: 33712       Freq: 8.51562e+08     Call not found for Update Message, Starting one...
(info)   [sys_1]   TG: 33712       Freq: 8.51562e+08     Starting Recorder on Src: rtl=1
(info)     - Starting P25 Recorder Num [0] TG: 33712     Freq: 8.51562e+08       TDMA: false     Slot: 0
(info)   [sys_1]   TG: 2384        Freq: 8.55088e+08     Starting Recorder on Src: rtl=0
(info)     - Starting P25 Recorder Num [9] TG: 2384      Freq: 8.55088e+08       TDMA: false     Slot: 0
(info)   [sys_1]   TG: 33712       Freq: 8.51562e+08     Assign Retuning - New Freq: 8.5125e+08  Elapsed: 6s     Since update: 2s
(info)   [sys_1]   TG: 2352        Freq: 8.55512e+08     Ending Recorded Call - Last Update: 4s  Call Elapsed: 25
(info)     - Stopping P25 Recorder Num [10]        TG: 2352        Freq: 8.55512e+08       TDMA: false     Slot: 0
(info)   Running upload script: 
./encode-upload.sh /home/scanner/jradio/trunk-recorder/sys_1/2017/7/9/2352-1499630871_8.55512e+08.wav

As you can see from the text above the output is a wav file that may include a single transmission or, should there be very quick back-to-back transmissions that occur on the same voice channel, a string of them in a single .wav file. Each transmission also has an identically named .json file with metadata including the radio IDs and duration of transmission in the file:

{
"freq": 8.55512e+08,
"start_time": 1499630871,
"stop_time": 1499630896,
"emergency": 0,
"talkgroup": 2352,
"srcList": [ {"src": 7609, "time": 1499630871, "pos": 0.000000}, {"src": 24884, "time": 1499630873, "pos": 1.476000}, 
{"src": 4616, "time": 1499630879, "pos"
: 6.300000}, {"src": 7609, "time": 1499630888, "pos": 13.140000}, {"src": 2360, "time": 1499630889, "pos": 13.140000}, 
{"src": 7609, "time": 1499630889, "pos
": 13.140000}, {"src": 2360, "time": 1499630890, "pos": 14.400000} ],
"play_length": 15.300000,
"system": 2,
"freqList": [ { "freq": 855512500.000000, "time": 1499630871, "pos": 0.000000, "len": 123664.000000, 
"error_count": 677.000000, "spike_count": 0.000000} ]
}

Trunk Recorder is similar to using DSD+ on Windows, but it is superior in some respects because it is a single application. There is no need to run multiple instances of the application per VCO or "recorder". There is no need for virtual audio cables. Everything Trunk Recorder does is self-contained within a single application that produces .wav files and then initiates an exteral script so you can do something automatically with those files.

Installing Trunk Recorder

It should be stated up front that what Trunk Recorder is doing is very CPU intensive. You're not going to have success running 3-4 dongles with 3-5 digital recorders on each dongle trying to decode a number of simultaneous convorsations without a reasonable amount of CPU. You'll know your issue is CPU if after running the recorder process you see a bunch of 0's or O's showing.

Installing Trunk Recorder is relatively simple even for a novice Linux user. The official directions can be found on the project's GitHub page, but here is a the quick approach in Ubuntu:

In Ubuntu you would start by installing some pre-requisites:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install gnuradio-dev gr-osmosdr libhackrf-dev libuhd-dev
sudo apt-get install git cmake build-essential libboost-all-dev libusb-1.0-0.dev libssl-dev

Then, make a directory to install Trunk Recorder into, download it with git and "compile" the application:

mkdir trunk-build 
git clone https://github.com/robotastic/trunk-recorder.git 
cd trunk-build
cmake ../trunk-recorder
make

If this works you'll see:

[100%] Built target recorder

Configuring Trunk Recorder

Again, the instructions on how to configure Trunk Recorder are found here on the project's GitHub page. You should especially read the lower portion of that page which explains all of the fields of the config.json file, the format of the ChannelsList.csv file.

The more challenging part of building your config.json is determining how to layout each of your SDR dongles by picking a center frequency that allows you to use as few dongles as possible.

The Trunk Player Web GUI

There are two separate web front ends designed to take the output of Trunk Player and present it in a visually pleasing interface. The author of Trunk Recorder has his own OpenMHZ hosting platform that supports .m4a audio files. More info on that can be found here: https://github.com/robotastic/trunk-recorder/wiki/Uploading-to-OpenMHz

The alternative is Trunk Player, written by Dylan Reinhold, which you can host on your own either entirely internal to your home network, or by utilizing Amazon AWS hosting to push the audio files to the internet (so you can play them from any internet connected device). The front end looks like this:

TrunkPlayer.png

The great thing about Trunk Player is it's scanlist concept. If you have 2-3 systems built either on your own or by working with friends, all pushing audio files from different digital systems, you can group talk groups into logical scanlists instead of system based scanlists.

For example, in the Northeast Ohio area there are two digital systems: The State of Ohio MARCS P25 and Cleveland, Ohio's regional "county wide" system. The numerous communities that make up Cuyahoga County have either gone with MARCS or Cleveland's system -- but you might be interested in any FIre/EMS traffic in the county. In Trunk Player you just define the talk-groups you care about and you hear every single transmission (even if some of them occured at the same time).

Note: Trunk Player is not necessarily the easiest thing to install, and it will require you to understand how to manipulate the encode_upload.sh scripts that come with Trunk Player to move your .wav files somewhere that they can be served up. Either by utilizing the Amazon AWS S3 concept, or by building your own in-house web server and making those .wav (or converted to .mp3) files available. The front end of Trunk Player simply presents a link to the audio file and the device visiting that webpage retrieves it and plays it.

How do we stream it?

Trunk Recorder will basically generate a folder full of .wav files. We need an application that will connect to Broadcastify and keep the stream online even when nothing is occuring. liquidsoap is the client used to create a stream connection to the broadcastify server's mount point. When there is nothing in it's queue liquidsoap will send silence towards the stream based on the script below.

Every time Trunk Recorder writes a .wav file it will run a script called encode-upload.sh. See below for what the script normally is meant to do - but for the purposes of streaming we add a call to the encode-upload.sh script to run a separate script called streamthis.

Streamthis is a python script that compares the talk-group of a wav file (by looking at the prefix of the file name) to a list of defined interesting talk-groups. If the file just created is one we want to inject into our stream the python script will connect to an already running instance of liquidsoap and add the new .wav file to the queue.

If the stream is currently idle, the .wav file is immediately played into the stream. When the queue is empty, the stream sends silence towards Broadcastify.

Basic LiquidSoap Configuration

This liquidsoap script presumes you've already installed liquidsoap on the system that you're going to use to run Trunk Recorder. On Ubuntu you can install liquidsoap easily:

apt-get install liquidsoap

Once installed, this script can be made executable and run on Linux to initiate your stream connection to Broadcastify. NOTE: This stream will connect and appear "Online" indefinately with zero audio until you build the logic to queue .wav files. The configuration below tells liquidsoap to listen on a socket for connections from the streamthis script:

#!/usr/bin/liquidsoap 
#                                  ^-- use -v for verbose otherwise tail the log
set("tag.encodings",["UTF-8","ISO-8859-1"])

# Log dir
set("log.file.path","./basic-radio.log")

# create a socket to send commands to this instance of liquidsoap
set("server.socket",true)
set("server.socket.path","./socket")

# This creates a 1 second silence period generated programmatically (no disk reads) 
silence = blank(duration=1.)

# This pulls the alpha tag out of the wav file 
def append_title(m) =
  # Grab the current title
	#title = '$(if $(title),"$(title)","..> Scanning <..")'
  #     # Return a new title metadata
	[("title",">> Scanning <<")]
end

silence = map_metadata(append_title, silence)

#
myqueue = request.queue()
myqueue = server.insert_metadata(id="S4",myqueue)

# If there is anything in the queue, play it.  If not, play the silence defined above repeatedly:
stream = fallback(track_sensitive=false, [myqueue, silence])

title = '$(if $(title),"$(title)","...Scanning...")'
stream = rewrite_metadata([("title", title)], stream)

output.icecast(%mp3(stereo=false, bitrate=16, samplerate=22050), host="audio1.broadcastify.com", 
 port = 80, password =  "YOURPASSWORD", genre="Scanner", description="Scanner audio", mount="/YOURMOUNTPOINT", 
 name="YOUR STREAM NAME HERE", stream) 

Streamthis Script

This is the streamthis python script that decides if a given .wav file just generated by Trunk Recorder should be injected into the liquidsoap queue. Think of it as the glue between Trunk Recorder's output and liquidsoap's queue mechanism.

The "touch" commands in this script are used by a watchdog process to see if the stream has died and are optional. These were built because my analog system would occasionally die, and sometimes liquidsoap just goes to lunch. The watchdog script pays attention to how often "things" are happening, and takes automatic action to restart processes if it thinks the stream has gone to lunch. Once I clean that logic up I'll upload the watchdog script as well.

#!/usr/bin/python
import socket
import sys
import os

filepath = sys.argv[1]
filename = os.path.basename(filepath)
a = filename.split('-')
tg,junk = a[0],a[1]

def touch(fname, times=None):
    with open(fname, 'a'):
        os.utime(fname, times)

# for debug purposes
#print "tg is %s" % tg

# This array defines all of the decimal trunk-groups you want to inject into the stream queue:
streamthese = ["33328", "41168", "41200", "42576","42608","33936","41264","33904","33872","6","7","8"]

if tg in streamthese:
    print ">>>>> STREAMTHIS:   tg is in list"
    match = 1
    touch('/tmp/streamthis-lastrun')
else:
    print ">>>>> STREAMTHIS:   tg is not in the list"
    match = 0
    # silently exit
    exit()

tag=tg

# Create a UDS socket
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)

# Connect the socket to the port where the server is listening
server_address = '/PATH/TO/WHERE/LIQUIDSOAP/socket'
#print >>sys.stderr, 'connecting to %s' % server_address
try:
        sock.connect(server_address)
except socket.error, msg:
        print >>sys.stderr, msg
        sys.exit(1)

try:
    # Send data
    message = 'queue.push {0}\n\r'.format(filepath)
    print "message: {0}".format(message)
    #print >>sys.stderr, 'sending "%s' % message
    sock.sendall(message)

    amount_received = 0
    amount_expected = len(message)
    
    data = sock.recv(16)
    amount_received += len(data)
    #print >>sys.stderr, 'received "%s' % data

finally:
    #print >>sys.stderr, 'closing socket'
    match = 0
    sock.close()

Modifying encode-upload.sh

As I wrote this wiki page I realized the current version of encode_upload.sh is dramatically different than what I've been using. Therefor, I'll show you the modifications I make that you can add to whatever the current version of encode_upload.sh happens to become:

The encode_upload.sh script is passed a path to the .wav file by Trunk Recorder. That value is stored in the variable $1 and used by the shell script. We need to tear that path down and define some variables, so at the top of your encode_upload.sh script add the following if parts of this do not exist already.

CSV_FILE="/home/scanner/radio/dev/trunk-recorder/ChannelList.csv"   # Path to YOUR Trunk Recorder CSV file
filename="$1"                                              # This should already exist
basename="${filename%.*}"                      # This should already exist
filename_only=$(basename $basename)
dir="$(dirname $filename)" 
mp3=${dir}/${filename_only}.mp3

I've started converting my .wav file to mp3 prior to sending it to liquidsoap. I believe this takes the burden of transcoding the audio away from liquidsoap, while also giving me an opportunity to inject the Alpha Tag into the .mp3 file. The Alpha tag is derived from the CSV file defined above, which is required to start Trunk Recorder in the first place. Unfortunately its not currently available in the .json.

I place this into the encode_upload.sh file somewhere near the bottom (after it has done any file transfer/uploads):

# this is for streamthis - lets convert to mp3 here, then delete the .wav on exit
# Start by finding the alpha tag for the talk-group:
tg="$( cut -d '-' -f 1 <<< "$filename_only" )"
ALPHA=`grep "^${tg}," ${CSV_FILE} | cut -d , -f 4`

if [ -z "$ALPHA" ]
then
	ALPHA="TG: ${tg}"
fi

The following command uses lame to both add the alpha tag and run a highpass filter. This removes any noise such as PL hum if you're injecting any analog audio:

# Insert the alpha tag into a mp3 with highpass:
lame --preset voice --highpass 150 --tt "${ALPHA}" $filename $mp3

Finally, we tell the streamthis script the fully qualified path to the MP3 file we just added an alpha tag to:

/home/scanner/liquid/streamthis $mp3

...if the talk-group in that MP3 matches those you defined above in the "streamthese" array, then the streamthis script will inject the path to the MP3 into the queue and the transmission will get played into your stream