History

« Previous - Version 3/31 (diff) - Next » - Current version
Martin Braun, 07/10/2012 02:04 pm
Added syntax highlighting

Out-of-tree modules

Extending GNU Radio with own functionality and blocks

This article borrows heavily from the original (but very outdated) How to write a block? written by Eric Blossom.

What is an out-of-tree module?

An out-of-tree module is a GNU Radio component that does not live within the GNU Radio source tree. Typically, if you want to extend GNU Radio with your own functions and blocks, such a module is what you create (i.e. you wouldn't usually add stuff to the actual GNU Radio source tree unless you're planning to submit it to the devs for upstream integration). This allows you to maintain the code yourself and have additional functionality alongside the main code.

A lot of such modules are hosted at CGRAN, which is a repository for GNU Radio-related projects. If you've developed some nice stuff yourself, please submit it to CGRAN!

One example of such a module is the spectral estimation toolbox, which extends GNU Radio with spectral estimation features. When installed, you have more blocks available (e.g. in the GNU Radio companion which behave just like the rest of GNU Radio; however, the developers and maintainers are different people.

Tools and resources at my disposal

There are a couple of tools, scripts and documents that are available as 3rd-party programs or as part of GNU Radio.

gr-howto-write-a-block

This is an example of an out-of-tree module which is delivered as part of the GNU Radio source tree. If you follow the tutorials later on in this document, you will end up with a module that looks a lot like gr-howto-write-a-block. Basically, this a good reference for you as a developer to see if you're module is looking as it should.

Because it is part of the GNU Radio tree, it is tested and maintained. A current version of GNU Radio will thus always come with an up-to-date module structure.

gr-modtool

When developing a module, there's a lot of boring, monotonous work involved: boilerplate code, makefile editing etc. gr-modtool is a script which aims to help with all these things by automatically editing makefiles, using templates and doing as much work as possible for the developer, such that you can jump straight into the DSP coding.

Note that gr-modtool makes a lot of assumptions on what the code looks like. The more your module is custom and has specific changes, the less useful gr-modtool becomes.

It is hosted on CGRAN and github.

create-out-of-tree-module

This is a script that comes with GNU Radio. Note that it's functionalities are a subset of gr-modtool's, so you don't really need it. However, if you don't like gr-modtool and want to do all the makefile editing etc. by hand anyway, you can use this script to create the initial directory.

Developer resources on the wiki

Most important is definitely the block coding guide. While this is written for the GNU Radio main tree, this should also be applied to all modules. Specifically, have a look at the naming conventions!

CMake, make, etc.

GNU Radio uses CMake as a build system. Building a module therefore requires you to have cmake installed, and whatever build manager you prefer (most often this is 'make', but you could also be using Eclipse or MS Visual Studio).

Structure of a module

Let's jump straight into the gr-howto-write-a-block module and see what it's made up of:

gnuradio/gr-howto-write-a-block [master] % ls
apps  cmake  CMakeLists.txt  docs  grc  include  lib  python  swig

It consists of several subdirectories. Anything that will be written in C++ (or C, or any language that is not Python) is put into lib/. For C++ files, we usually have headers which are put into include/ (if they are to be exported) or also in lib/ (if they're only relevant during compile time, but are not installed later).

Of course, Python stuff comes into python/. This includes unit tests (which are not installed) and parts of the Python module which are installed.

You probably know already that GNU Radio blocks are available in Python even if they were written in C++. This is done by the help of SWIG, the simplified wrapper and interface generator, which automatically creates glue code to make this possible. SWIG needs some instructions on how to do this, which are put into the swig/ subdirectory.

If you want your blocks to be available in the GNU Radio companion, the graphical UI for GNU Radio, you need to add XML descriptions of the blocks and put them into grc/.

For documentation, docs/ contains some instructions on how to extract documentation from the C++ files and Python files (we use Doxygen and Sphinx for this) and also make sure they're available as docstrings in Python. Of course, you can add custom documentation here as well.

Finally, the apps/ subdir contains any complete applications (both for GRC and standalone executables) which are installed to the system alongside with the blocks.

Some modules contain another directory, examples/, which can be used to save (guess what) examples, which are a great addendum to documentation, because other developers can simply look straight at the code to see how your blocks are used.

The build system brings some baggage along, as well: the CMakeLists.txt file (one of which is present in every subdirectory) and the cmake/ folder. You can ignore the latter for now, as it brings along mainly instructions for CMake on how to find GNU Radio libraries etc. The CMakeLists.txt files need to be edited a lot in order to make sure your module builds correctly.

But one step at a time! Now, let's start with our first tutorial.

Tutorial 1: Creating an out-of-tree module

The easy way: gr-modtool

If you have gr-modtool installed, just use that. It will create a new directory and all. If you're planning to use gr-modtool in the upcoming steps, it is highly recommended that you create the module this way.

Here's how it works:

1 ~/tmp % gr_modtool.py create howto
2 Module directory is "./gr-howto".
3 Creating directory...
4 Copying howto example...
5 Unpacking...
6 Replacing occurences of 'howto' to 'howto'...
7 Done.
8 Use 'gr_modtool add' to add a new block to this currently empty module.

Note that gr-modtool actually uses the gr-howto-write-a-block directory as a template. After installing, it renames anything related to 'howto' into whatever you called your block (being uncreative, we chose 'howto' here as well).

The hard way: by hand

If you want to understand all the inner workings of a module right now, and hate to install gr-modtool, do the following:

  1. Copy gr-howto-write-a-block to a new location (e.g. ~/src)
  2. Rename the directory (e.g. gr-howto)
  3. Rename the project name in the top-level CMakeLists.txt file
  4. Remove the *.cc, *.h, *.xml and *.grc files
  5. Remove all references to these files from all the CMakeLists.txt files

Make sure to not miss anything in the CMakeLists.txt. Best to open them all!

Tutorial 2: Writing a block (howto_square_ff) in C++

For our first example we'll create a block that computes the square of its single float input. This block will accept a single float input stream and produce a single float output stream, i.e. for every incoming float item, we output one float item which is the square of that input item.

Following the naming conventions, we'll use howto as our package prefix, and the block will be called howto_square_ff because it has float inputs, float outputs.

We are going to arrange that this block, as well as the others that we write in this article, end up in the howto Python module. This will allow us to access it from Python like this:

1 import howto
2 sqr = howto.square_ff()

Test Driven Programming

We could just start banging out the C++ code, but being highly evolved modern programmers, we're going to write the test code first. After all, we do have a good spec for the behavior: take a single stream of floats as the input and produce a single stream of floats as the output. The output should be the square of the input.

How hard could this be? Turns out that this is easy! Check out this code:

 1 from gnuradio import gr, gr_unittest
 2 import howto_swig # Can't import howto because that module does not yet exist
 3 
 4 class qa_howto (gr_unittest.TestCase):
 5 
 6     def setUp (self):
 7         self.tb = gr.top_block ()
 8 
 9     def tearDown (self):
10         self.tb = None
11 
12     def test_001_square_ff (self):
13         src_data = (-3, 4, -5.5, 2, 3)
14         expected_result = (9, 16, 30.25, 4, 9)
15         src = gr.vector_source_f (src_data)
16         sqr = howto_swig.square_ff ()
17         dst = gr.vector_sink_f ()
18         self.tb.connect (src, sqr)
19         self.tb.connect (sqr, dst)
20         self.tb.run ()
21         result_data = dst.data ()
22         self.assertFloatTuplesAlmostEqual (expected_result, result_data, 6)
23 
24 if __name__ == '__main__':
25     gr_unittest.main ()

gr_unittest is an extension to the standard Python module unittest. gr_unittest adds support for checking approximate equality of tuples of float and complex numbers. Unittest uses Python's reflection mechanism to find all methods that start with test_ and runs them. Unittest wraps each call to test_* with matching calls to setUp and tearDown. See the Python unittest documentation for details.

When we run the test, gr_unittest.main is going to invoke setUp, test_001_square_ff, and tearDown.

test_001_square_ff builds a small graph that contains three nodes. gr.vector_source_f(src_data) will source the elements of src_data and then say that it's finished. howto.square_ff is the block we're testing. gr.vector_sink_f gathers the output of howto.square_ff.

The run() method runs the graph until all the blocks indicate they are finished. Finally, we check that the result of executing square_ff on src_data matches what we expect.

Note that such a test is usually called before installing the module. This means that we need some trickery to be able to load the blocks when testing. CMake takes care of most things by changing PYTHONPATH appropriately. Also, we import howto_swig instead of howto in this file.

Build Tree vs. Install Tree

When you run cmake, you usually run it in a separate directory (e.g. build/). This is the build tree. The path to the install tree is $prefix/lib/pythonversion/site-packages, where $prefix is whatever you specified to CMake during configuration (usually /usr/local/) with the -DCMAKE_INSTALL_PREFIX switch.

During compilation, the libraries are copied into the build tree. Only during installation, files are installed to the install tree, thus making our blocks available to GNU Radio apps.

We write our applications such that they access the code and libraries in the install tree. On the other hand, we want our test code to run on the build tree, where we can detect problems before installation.

make test

We use make test to run our tests. This invokes a shell script which sets up the PYTHONPATH environment variable so that our tests use the build tree versions of our code and libraries. It then runs all files which have names of the form qa_*.py and reports the overall success or failure.

There is quite a bit of behind-the-scenes action required to use the non-installed versions of our code (look at the cmake/ directory for a cheap thrill.)

Of course, we can't call it right now--there's nothing to test on.

The C++ code (part 1)

Now that we've got a test case, let's write the C++ code. All signal processing blocks are derived from gr_block or one of its subclasses. Go check out the block documentation on the Doxygen-generated manual now!

A quick scan of the docs reveals that since general_work() is pure virtual, we definitely need to override that. general_work() is the method that does the actual signal processing. For our squaring example we'll need to override this and provide a constructor and destructor and a bit of stuff to take advantage of the boost shared_ptrs.

We now need to write a header file, a .cc file and then edit the include/CMakeLists.txt and lib/CMakeLists.txt. As before, we can use gr-modtool to do all of that:

tmp/gr-howto % gr_modtool.py add -t general square_ff
Operating in directory .
GNU Radio module name identified: howto
Code is of type: general
Block/code identifier: square_ff
Full block/code identifier is: howto_square_ff
Enter valid argument list, including default arguments: 
Add Python QA code? [Y/n] n
Add C++ QA code? [Y/n] n
Traversing lib...
Adding file 'howto_square_ff.h'...
Adding file 'howto_square_ff.cc'...
Traversing swig...
Editing swig/howto_swig.i...
Traversing python...
Editing python/CMakeLists.txt...
Traversing grc...
Adding file 'howto_square_ff.xml'...
Editing grc/CMakeLists.txt...

As you can see, gr-modtool does even more: it creates GRC bindings (the XML is not valid, though, but more on that later) and does something to the SWIG definitions. But most importantly, it creates the header- and cc-file we wanted. We can now edit them with our favourite editor. (Also note we skipped the automatic generation of a qa*.py-file because we already placed it in python/ in the previous section.)

Of course, we can do that by hand, too: copy a .h- and .cc-file from another projects, use search/replace to rename everything accordingly and edit the CMakeLists.txt.

Finally, we actually edit the code to do the squaring. Let's jump straight into the results. First, we start with the header file (howto_square_ff.h) which gets put into include/. Open it now:

source:gr-howto-write-a-block/include/howto_square_ff.h

The C++ file is called howto_square_ff.cc and resides in lib/. Open that, too:

source:gr-howto-write-a-block/lib/howto_square_ff.cc

Here's some things to pay attention to:
  • The header file contains pretty much only standard definitions etc. and looks very similar for all kinds of blocks. In fact, for this simple example, the header does not need any manual editing at all if you use gr-modtool (with the exception of documentation)
  • The io signature (in the constructor definition in the .cc-file) specifies that we have one input port, which accepts items of type 'float'. The output is the same.
  • All the work is done in the general_work() function. There is one pointer to the input- and output buffer, respectively, and a for-loop which copies the square of the input buffer to the output buffer.
  • The final two statements in the general_work() method tell GNU Radio how many items were read from the input buffer (i.e. consumed) and how many items were written to the output buffer (the return statement).
  • If you used gr-modtool, references to the howto_square_ff.h were added to include/CMakeLists.txt and howto_square_ff.cc was added to lib/CMakeLists.txt. Have a look at these files, too, to understand how the build system works.
  • Also, a reference to howto_square_ff.h was added to the file swig/howto_swig.i. Because this block is so simple, it is sufficient to point SWIG to the header file and tell it to "create a Python object that looks like this C++ class". Because GNU Radio comes with some of it's own SWIG magic, this works fine in most cases.

Simple, isn't it?

More C++ code (but better) - Subclasses for common patterns

gr_block allows tremendous flexibility with regard to the consumption of input streams and the production of output streams. Adroit use of forecast() and consume() (see below) allows variable rate blocks to be built. It is possible to construct blocks that consume data at different rates on each input, and produce output at a rate that is a function of the contents of the input data.

On the other hand, it is very common for signal processing blocks to have a fixed relationship between the input rate and the output rate. Many are 1:1, while others have 1:N or N:1 relationships. You must have thought the same thing in the general_work() function of the previous block: if the number of items consumed is identical the number of items produced, why do I have to tell GNU Radio the exact same number twice?

Another common requirement is the need to examine more than one input sample to produce a single output sample. This is orthogonal to the relationship between input and output rate. For example, a non-decimating, non-interpolating FIR filter needs to examine N input samples for each output sample it produces, where N is the number of taps in the filter. However, it only consumes a single input sample to produce a single output. We call this concept "history", but you could also think of it as "look-ahead".

  • gr_sync_block

gr_sync_block is derived from gr_block and implements a 1:1 block with optional history. Given that we know the input to output rate, certain simplifications are possible. From the implementor's point-of-view, the primary change is that we define a work method instead of general_work(). work() has a slightly different calling sequence; it omits the unnecessary ninput_items parameter, and arranges for consume_each() to be called on our behalf.

 1   /*!
 2    * \brief Just like gr_block::general_work, only this arranges to
 3    *  call consume_each for you.
 4    *
 5    * The user must override work to define the signal processing code
 6    */
 7   virtual int work (int noutput_items,
 8                     gr_vector_const_void_star &input_items,
 9                     gr_vector_void_star &output_items) = 0;

This gives us fewer things to worry about, and less code to write. If the block requires history greater than 1, call set_history() in the constructor, or any time the requirement changes.

gr_sync_block provides a version of forecast that handles the history requirement.

  • gr_sync_decimator

gr_sync_decimator is derived from gr_sync_block and implements a N:1 block with optional history.

  • gr_sync_interpolator

gr_sync_interpolator is derived from gr_sync_block and implements a 1:N block with optional history.

With this knowledge it should be clear that howto_square_ff should be a gr_sync_block with no history.

So let's write another block, which does the same as before, but is a sync block. Another invocation of gr-modtool is our friend:

tmp/gr-howto % gr_modtool.py add -t sync square2_ff
Operating in directory .
GNU Radio module name identified: howto
Code is of type: sync
Block/code identifier: square2_ff
Full block/code identifier is: howto_square2_ff
Enter valid argument list, including default arguments:
Add Python QA code? [Y/n] n
Add C++ QA code? [Y/n] n
Traversing lib...
Adding file 'howto_square2_ff.h'...
Adding file 'howto_square2_ff.cc'...
Traversing swig...
Editing swig/howto_swig.i...
Traversing grc...
Adding file 'howto_square2_ff.xml'...
Editing grc/CMakeLists.txt...

Again, we skip the QA file generation because we'll just use the other one.
In fact, the test is exactly the same. Here's a qa_howto.py file for both blocks:

source:gr-howto-write-a-block/python/qa_howto.py

Running make test now will spawn a test run with of qa_howto.py which should not fail.

Making your blocks available in GRC

You can now install your module, but it will not be available in GRC. That's because gr-modtool can't create valid XML files before you've even written a block. So, you will have to edit the grc/*.xml files by hand. The GRC wiki site has a description available.

For the blocks written in tutorial 2, the valid XML files look like this:

source:gr-howto-write-a-block/grc/howto_square_ff.xml
source:gr-howto-write-a-block/grc/howto_square2_ff.xml

There's more: additional gr_block-methods

If you've read the gr_block documentation (which you should have), you'll have noticed there are a great number of methods available to configure your block.

Here's some of the more important ones:

set_history()

If you're block needs a history (e.g. something like an FIR filter), call this in the constructor. GNU Radio then makes sure you have the given number of 'old' items available.

forecast()

Looking at general_work() you may have wondered how the system knows how much data it needs to ensure is valid in each of the input arrays. The forecast() method provides this information.

The default implementation of forecast() says there is a 1:1 relationship between noutput_items and the requirements for each input stream. The size of the items is defined by gr_io_signatures in the constructor of gr_block. The sizes of the input and output items can of course differ; this still qualifies as a 1:1 relationship.

 1 
 2   // default implementation:  1:1
 3   void
 4   gr_block::forecast (int noutput_items,
 5                       gr_vector_int &ninput_items_required)
 6   {
 7     unsigned ninputs = ninput_items_required.size ();
 8     for (unsigned i = 0; i < ninputs; i++)
 9       ninput_items_required[i] = noutput_items;
10   }

Although the 1:1 implementation worked for howto_square_ff, it wouldn't be appropriate for interpolators, decimators, or blocks with a more complicated relationship between noutput_items and the input requirements. That said, by deriving your classes from gr_sync_block, gr_sync_interpolator or gr_sync_decimator instead of gr_block, you can often avoid implementing forecast.

Note that if you've already got a history set, you usually don't need to set this.

set_output_multiple()

When implementing your general_work() routine, it's occasionally convenient to have the run time system ensure that you are only asked to produce a number of output items that is a multiple of some particular value. This might occur if your algorithm naturally applies to a fixed sized block of data. Call set_output_multiple in your constructor to specify this requirement. The default output multiple is 1.

Other types of blocks

Sources and sinks

Sources and sinks are derived from gr_sync_block. The only thing different about them is that sources have no inputs and sinks have no outputs. This is reflected in the gr_io_signatures that are passed to the gr_sync_block constructor. Take a look at "gr_file_source.{h,cc}":source:gnuradio-core/src/lib/io/gr_file_source.cc and gr_file_sink.{h,cc} for some very straight-forward examples. See also the tutorial on writing Python applications.

Hierarchical blocks

For the concept of hierarchical blocks, see this. Of course, they can also be written in C++. gr-modtool supports skeleton code for hierarchical blocks both in Python and C++.

tmp/gr-howto % gr_modtool.py add -t hiercpp hierblockcpp_ff
Operating in directory .
GNU Radio module name identified: howto
Code is of type: hiercpp
Block/code identifier: hierblockcpp_ff
Full block/code identifier is: howto_hierblockcpp_ff
Enter valid argument list, including default arguments: 
Add Python QA code? [Y/n] n
Add C++ QA code? [Y/n] n
Traversing lib...
Adding file 'howto_hierblockcpp_ff.h'...
Adding file 'howto_hierblockcpp_ff.cc'...
Traversing swig...
Editing swig/howto_swig.i...
Traversing grc...
Adding file 'howto_hierblockcpp_ff.xml'...
Editing grc/CMakeLists.txt...
tmp/gr-howto % gr_modtool.py add -t hierpython hierblockpy_ff
Operating in directory .
GNU Radio module name identified: howto
Code is of type: hierpython
Block/code identifier: hierblockpy_ff
Full block/code identifier is: howto_hierblockpy_ff
Enter valid argument list, including default arguments: 
Add Python QA code? [Y/n] n
Traversing python...
Adding file 'hierblockpy_ff.py'...
Traversing grc...
Adding file 'howto_hierblockpy_ff.xml'...
Editing grc/CMakeLists.txt...

Tutorial 3: Writing a signal processing block in Python

tbw

Debugging blocks

Debugging GNU Radio is available as a separate tutorial.