After installing, configuring and training SOPARE, you want naturally do something with the recognized results. I run some installations where SOPARE turns lights on and off, controls a magic mirror and a robotic arm. With my voice. On a Raspberry Pi. Offline and in real time. How does that work? Glad you are asking. This post should give you an overview about SOPARE in terms of the architecture and provide some insights how to write your own custom plugins for further processing.
SOPARE is written in Python and runs very well with Python 2.7 and was tested successfully on several Raspberry Pis running all kind of operating systems. In fact, SOPARE should run on all kind of *UNIX systems if the underlying hardware comes with a multi core CPU. Here you find more information about how to get started. Now let’s make a small walk through. When you start SOPARE with the command
we can look at this simplified list what’s happening:
- Check for arguments from the command line and init parameters
- Load the config file
- Evaluate the init parameters
- Initialize the audio
- Read input from microphone
- Check if the sound level is above the THRESHOLD
- Check for matching patterns
- Call plugin(s)
- Stop everything after the configured timeout is reached
Below is an architecture overview:
Now we can dig a bit deeper and do a more detailed view. The first thread (thread 1) is listening the whole time and records chunks of data. This small chunks of data are compared in terms of sound volume. Whenever one CHUNK volume is above the THRESHOLD the chunks are transformed, filtered and a characteristic is created. This happens in „thread 2“. The chain of characteristics are compared against trained results and the plugins are called in „thread 3“. Each thread runs on a different CPU core. This can be observed when SOPARE is running by starting the command
top -d 1
and then pressing „1“ to show all CPUs:
Press „q“ to exit the top output.
As we see in the above picture the SOPARE python processes with the corresponding PID 656, 655 and 649 I want to show two ways to kill SOPARE processes:
First via the „kill“ command and a list of all PIDs:
kill -3 649 655 656
Second via „pkill“ where the processes are killed based on a name:
pkill -3 -f ./sopare.py
I usually send a SIGQUIT as I get the dump to see what the program is doing but you can send any appropriate signal.
Ok. Now that we have touched the architecture and processes, let’s talk about plugins. Plugins are stored in the
directory. Below you see the plugin structure from my magic mirror pi:
To create a plugin I recommend to copy the print plugin and create a „custom_plugin_directory“ where you replace the name „custom_plugin_directory“ with any meaningful name for your plugin:
cd plugins cp -r print/ custom_plugin_directory
You now have a directory with all necessary files for your custom plugin in the directory „custom_plugin_directory“ or whatever name you have chosen. Let’s modify the plugin and see what’s inside:
The file „__init__.py“ is the one which is called from SOPARE when a sound is regognized. Open it:
You see some lines of code and the interesting part is this one:
def run(readable_results, data, rawbuf): print readable_results
The function „run“ is called and the value „readable_results“ is handed over as an array. This means that each recognized sound is inside this array with the corresponding ID. Let’s assume that you trained a sound with the ID „test“. When the same sound is recognized, then the ID „test“ shows up like this:
If more words are recognized, you get all of them in the array in the same order as they were recognized:
[u'test', u'another_id', u'whatever']
You can now work with the results and write your own conditions. For example, here is the code that just checks for two words:
def run(readable_results, data, rawbuf): if ('some_word' in readable_results and 'another_word' in readable_results): print ('Tschakka! Got my two words...now do some awesome stuff')
Here is an example where the results must be in a specific order:
def run(readable_results, data, rawbuf): if (len(readable_results) == 2 and 'word_1' in readable_results and 'word_2' in readable_results): print ('Tschakka! Got my two words in the right order...now do some awesome stuff')
With this knowledge you are now hopefully able to read and understand the robotic arm example and write your own plugins. There is one thing I want to mention. All plugins are called sequential. This means you should not execute any complex or long running code without threading!
Ok, I stop for now. I’ll make a video tutorial whenever I have the time. In the meantime don’t hold back and ask questions, make suggestions and give feedback.
Happy coding 🙂