« Previous - Version 16/38 (diff) - Next » - Current version
Martin Braun, 11/02/2012 03:01 pm
gr_modtool makexml


Out-of-tree modules

Extending GNU Radio with own functionality and blocks

This article borrows heavily from the original (but very outdated) "How to write a block?" written by Eric Blossom.

What is an out-of-tree module?

An out-of-tree module is a GNU Radio component that does not live within the GNU Radio source tree. Typically, if you want to extend GNU Radio with your own functions and blocks, such a module is what you create (i.e. you wouldn't usually add stuff to the actual GNU Radio source tree unless you're planning to submit it to the devs for upstream integration). This allows you to maintain the code yourself and have additional functionality alongside the main code.

A lot of such modules are hosted at CGRAN, which is a repository for GNU Radio-related projects. If you've developed some nice stuff yourself, please submit it to CGRAN!

One example of such a module is the spectral estimation toolbox, which extends GNU Radio with spectral estimation features. When installed, you have more blocks available (e.g. in the GNU Radio companion) which behave just like the rest of GNU Radio; however, the developers are different people.

Tools and resources at my disposal

There are a couple of tools, scripts and documents that are available as 3rd-party programs or as part of GNU Radio.

gr-howto-write-a-block

This is an example of an out-of-tree module which is delivered as part of the GNU Radio source tree. If you follow the tutorials later on in this document, you will end up with a module that looks a lot like gr-howto-write-a-block. Basically, this a good reference for you as a developer to see if you're module is looking as it should.

Because it is part of the GNU Radio tree, it is tested and maintained. A current version of GNU Radio will thus always come with an up-to-date module structure.

gr-modtool

When developing a module, there's a lot of boring, monotonous work involved: boilerplate code, makefile editing etc. gr-modtool is a script which aims to help with all these things by automatically editing makefiles, using templates and doing as much work as possible for the developer, such that you can jump straight into the DSP coding.

Note that gr-modtool makes a lot of assumptions on what the code looks like. The more your module is custom and has specific changes, the less useful gr-modtool becomes.

It is hosted on CGRAN and github.

create-out-of-tree-module

This is a script that comes with GNU Radio. Note that it's functionalities are a subset of gr-modtool's, so you don't really need it. However, if you don't like gr-modtool and want to do all the makefile editing etc. by hand anyway, you can use this script to create the initial directory.

Developer resources on the wiki

Most important is definitely the block coding guide. While this is written for the GNU Radio main tree, this should also be applied to all modules. Specifically, have a look at the naming conventions!

If you're reading this, you're most likely familiar with all the GNU Radio jargon, but just in case you're not, have a peak at the core concepts tutorial. This contains definitive must-knows. Also, the tutorial on writing Python applications explains a lot of the key features.

CMake, make, etc.

GNU Radio uses CMake as a build system. Building a module therefore requires you to have cmake installed, and whatever build manager you prefer (most often this is 'make', but you could also be using Eclipse or MS Visual Studio).

Structure of a module

Let's jump straight into the gr-howto-write-a-block module and see what it's made up of:

gnuradio/gr-howto-write-a-block [master] % ls
apps  cmake  CMakeLists.txt  docs  grc  include  lib  python  swig

It consists of several subdirectories. Anything that will be written in C++ (or C, or any language that is not Python) is put into lib/. For C++ files, we usually have headers which are put into include/ (if they are to be exported) or also in lib/ (if they're only relevant during compile time, but are not installed later).

Of course, Python stuff comes into python/. This includes unit tests (which are not installed) and parts of the Python module which are installed.

You probably know already that GNU Radio blocks are available in Python even if they were written in C++. This is done by the help of SWIG, the simplified wrapper and interface generator, which automatically creates glue code to make this possible. SWIG needs some instructions on how to do this, which are put into the swig/ subdirectory.

If you want your blocks to be available in the GNU Radio companion, the graphical UI for GNU Radio, you need to add XML descriptions of the blocks and put them into grc/.

For documentation, docs/ contains some instructions on how to extract documentation from the C++ files and Python files (we use Doxygen and Sphinx for this) and also make sure they're available as docstrings in Python. Of course, you can add custom documentation here as well.

Finally, the apps/ subdir contains any complete applications (both for GRC and standalone executables) which are installed to the system alongside with the blocks.

Some modules contain another directory, examples/, which can be used to save (guess what) examples, which are a great addendum to documentation, because other developers can simply look straight at the code to see how your blocks are used.

The build system brings some baggage along, as well: the CMakeLists.txt file (one of which is present in every subdirectory) and the cmake/ folder. You can ignore the latter for now, as it brings along mainly instructions for CMake on how to find GNU Radio libraries etc. The CMakeLists.txt files need to be edited a lot in order to make sure your module builds correctly.

But one step at a time! Now, let's start with our first tutorial.

Tutorial 1: Creating an out-of-tree module

The easy way: gr-modtool

If you have gr-modtool installed, just use that. It will create a new directory and all. If you're planning to use gr-modtool in the upcoming steps, it is highly recommended that you create the module this way.

Here's how it works:

~/tmp % gr_modtool.py create howto
Module directory is "./gr-howto".
Creating directory...
Copying howto example...
Unpacking...
Replacing occurences of 'howto' to 'howto'...
Done.
Use 'gr_modtool add' to add a new block to this currently empty module.

Note that gr-modtool actually uses the gr-howto-write-a-block directory as a template. After installing, it renames anything related to 'howto' into whatever you called your block (being uncreative, we chose 'howto' here as well).

The hard way: by hand

If you want to understand all the inner workings of a module right now, and hate to install gr-modtool, do the following:

  1. Copy gr-howto-write-a-block to a new location (e.g. ~/src)
  2. Rename the directory (e.g. gr-howto)
  3. Rename the project name in the top-level CMakeLists.txt file
  4. Remove the *.cc, *.h, *.xml and *.grc files
  5. Remove all references to these files from all the CMakeLists.txt files

Make sure to not miss anything in the CMakeLists.txt. Best to open them all!

Using CMake

If you've never used CMake before, this is good time to give it a try. The typical workflow of a CMake-based project as seen from the command line is this:

$ mkdir build    # We're currently in the module's top directory
$ cd build/
$ cmake ../      # Tell CMake that all its config files are one dir up
$ make           # And start building

Now we have a new directory build/ in our module's directory. All the compiling etc. is done in here, so the actual source tree is not littered with temporary files. If we change any CMakeLists.txt file significantly, we should re-run cmake ../.

Tutorial 2: Writing a block (howto_square_ff) in C++

For our first example we'll create a block that computes the square of its single float input. This block will accept a single float input stream and produce a single float output stream, i.e. for every incoming float item, we output one float item which is the square of that input item.

Following the naming conventions, we'll use howto as our package prefix, and the block will be called howto_square_ff because it has float inputs, float outputs.

We are going to arrange that this block, as well as the others that we write in this article, end up in the howto Python module. This will allow us to access it from Python like this:

import howto
sqr = howto.square_ff()

Test Driven Programming

We could just start banging out the C++ code, but being highly evolved modern programmers, we're going to write the test code first. After all, we do have a good spec for the behavior: take a single stream of floats as the input and produce a single stream of floats as the output. The output should be the square of the input.

How hard could this be? Turns out that this is easy! Check out this code, which we save as python/qa_howto.py:

from gnuradio import gr, gr_unittest
import howto_swig # Can't import howto because that module does not yet exist

class qa_howto (gr_unittest.TestCase):

    def setUp (self):
        self.tb = gr.top_block ()

    def tearDown (self):
        self.tb = None

    def test_001_square_ff (self):
        src_data = (-3, 4, -5.5, 2, 3)
        expected_result = (9, 16, 30.25, 4, 9)
        src = gr.vector_source_f (src_data)
        sqr = howto_swig.square_ff ()
        dst = gr.vector_sink_f ()
        self.tb.connect (src, sqr)
        self.tb.connect (sqr, dst)
        self.tb.run ()
        result_data = dst.data ()
        self.assertFloatTuplesAlmostEqual (expected_result, result_data, 6)

if __name__ == '__main__':
    gr_unittest.main ()

gr_unittest is an extension to the standard Python module unittest. gr_unittest adds support for checking approximate equality of tuples of float and complex numbers. Unittest uses Python's reflection mechanism to find all methods that start with test_ and runs them. Unittest wraps each call to test_* with matching calls to setUp and tearDown. See the Python unittest documentation for details.

When we run the test, gr_unittest.main is going to invoke setUp, test_001_square_ff, and tearDown.

test_001_square_ff builds a small graph that contains three nodes. gr.vector_source_f(src_data) will source the elements of src_data and then say that it's finished. howto.square_ff is the block we're testing. gr.vector_sink_f gathers the output of howto.square_ff.

The run() method runs the graph until all the blocks indicate they are finished. Finally, we check that the result of executing square_ff on src_data matches what we expect.

Note that such a test is usually called before installing the module. This means that we need some trickery to be able to load the blocks when testing. CMake takes care of most things by changing PYTHONPATH appropriately. Also, we import howto_swig instead of howto in this file.

In order for CMake to actually know this test exists, we append a line to python/CMakeLists.txt, so it looks like this:

########################################################################
# Handle the unit tests
########################################################################
include(GrTest)

set(GR_TEST_TARGET_DEPS gnuradio-howto)
set(GR_TEST_PYTHON_DIRS ${CMAKE_BINARY_DIR}/swig)
GR_ADD_TEST(qa_howto ${PYTHON_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/qa_howto.py)

Build Tree vs. Install Tree

When you run cmake, you usually run it in a separate directory (e.g. build/). This is the build tree. The path to the install tree is $prefix/lib/pythonversion/site-packages, where $prefix is whatever you specified to CMake during configuration (usually /usr/local/) with the -DCMAKE_INSTALL_PREFIX switch.

During compilation, the libraries are copied into the build tree. Only during installation, files are installed to the install tree, thus making our blocks available to GNU Radio apps.

We write our applications such that they access the code and libraries in the install tree. On the other hand, we want our test code to run on the build tree, where we can detect problems before installation.

make test

We use make test to run our tests. This invokes a shell script which sets up the PYTHONPATH environment variable so that our tests use the build tree versions of our code and libraries. It then runs all files which have names of the form qa_*.py and reports the overall success or failure.

There is quite a bit of behind-the-scenes action required to use the non-installed versions of our code (look at the cmake/ directory for a cheap thrill.)

Of course, we can't call it right now--there's nothing to test on.

The C++ code (part 1)

Now that we've got a test case, let's write the C++ code. All signal processing blocks are derived from gr_block or one of its subclasses. Go check out the block documentation on the Doxygen-generated manual now!

A quick scan of the docs reveals that since general_work() is pure virtual, we definitely need to override that. general_work() is the method that does the actual signal processing. For our squaring example we'll need to override this and provide a constructor and destructor and a bit of stuff to take advantage of the boost shared_ptrs.

We now need to write a header file, a .cc file and then edit the include/CMakeLists.txt and lib/CMakeLists.txt. As before, we can use gr-modtool to do all of that:

tmp/gr-howto % gr_modtool.py add -t general square_ff
Operating in directory .
GNU Radio module name identified: howto
Code is of type: general
Block/code identifier: square_ff
Full block/code identifier is: howto_square_ff
Enter valid argument list, including default arguments: 
Add Python QA code? [Y/n] n
Add C++ QA code? [Y/n] n
Traversing lib...
Adding file 'howto_square_ff.h'...
Adding file 'howto_square_ff.cc'...
Traversing swig...
Editing swig/howto_swig.i...
Traversing python...
Editing python/CMakeLists.txt...
Traversing grc...
Adding file 'howto_square_ff.xml'...
Editing grc/CMakeLists.txt...

As you can see, gr-modtool does even more: it creates GRC bindings (the XML is not valid, though, but more on that later) and does something to the SWIG definitions. But most importantly, it creates the header- and cc-file we wanted. We can now edit them with our favourite editor. (Also note we skipped the automatic generation of a qa*.py-file because we already placed it in python/ in the previous section.)

Of course, we can do that by hand, too: copy a .h- and .cc-file from another projects, use search/replace to rename everything accordingly and edit the CMakeLists.txt.

Finally, we actually edit the code to do the squaring. Let's jump straight into the results. First, we start with the header file (howto_square_ff.h) which gets put into include/. Open it now:

source:gr-howto-write-a-block/include/howto_square_ff.h

The C++ file is called howto_square_ff.cc and resides in lib/. Open that, too:

source:gr-howto-write-a-block/lib/howto_square_ff.cc

Here's some things to pay attention to:
  • The header file contains pretty much only standard definitions etc. and looks very similar for all kinds of blocks. In fact, for this simple example, the header does not need any manual editing at all if you use gr-modtool (with the exception of documentation)
  • The io signature (in the constructor definition in the .cc-file) specifies that we have one input port, which accepts items of type 'float'. The output is the same.
  • All the work is done in the general_work() function. There is one pointer to the input- and output buffer, respectively, and a for-loop which copies the square of the input buffer to the output buffer.
  • The final two statements in the general_work() method tell GNU Radio how many items were read from the input buffer (i.e. consumed) and how many items were written to the output buffer (the return statement).
  • If you used gr-modtool, references to the howto_square_ff.h were added to include/CMakeLists.txt and howto_square_ff.cc was added to lib/CMakeLists.txt. Have a look at these files, too, to understand how the build system works.
  • Also, a reference to howto_square_ff.h was added to the file swig/howto_swig.i. Because this block is so simple, it is sufficient to point SWIG to the header file and tell it to "create a Python object that looks like this C++ class". Because GNU Radio comes with some of it's own SWIG magic, this works fine in most cases.

Simple, isn't it?

More C++ code (but better) - Subclasses for common patterns

gr_block allows tremendous flexibility with regard to the consumption of input streams and the production of output streams. Adroit use of forecast() and consume() (see below) allows variable rate blocks to be built. It is possible to construct blocks that consume data at different rates on each input, and produce output at a rate that is a function of the contents of the input data.

On the other hand, it is very common for signal processing blocks to have a fixed relationship between the input rate and the output rate. Many are 1:1, while others have 1:N or N:1 relationships. You must have thought the same thing in the general_work() function of the previous block: if the number of items consumed is identical the number of items produced, why do I have to tell GNU Radio the exact same number twice?

Another common requirement is the need to examine more than one input sample to produce a single output sample. This is orthogonal to the relationship between input and output rate. For example, a non-decimating, non-interpolating FIR filter needs to examine N input samples for each output sample it produces, where N is the number of taps in the filter. However, it only consumes a single input sample to produce a single output. We call this concept "history", but you could also think of it as "look-ahead".

  • gr_sync_block

gr_sync_block is derived from gr_block and implements a 1:1 block with optional history. Given that we know the input to output rate, certain simplifications are possible. From the implementor's point-of-view, the primary change is that we define a work method instead of general_work(). work() has a slightly different calling sequence; it omits the unnecessary ninput_items parameter, and arranges for consume_each() to be called on our behalf.

  /*!
   * \brief Just like gr_block::general_work, only this arranges to
   *  call consume_each for you.
   *
   * The user must override work to define the signal processing code
   */
  virtual int work (int noutput_items,
                    gr_vector_const_void_star &input_items,
                    gr_vector_void_star &output_items) = 0;

This gives us fewer things to worry about, and less code to write. If the block requires history greater than 1, call set_history() in the constructor, or any time the requirement changes.

gr_sync_block provides a version of forecast that handles the history requirement.

  • gr_sync_decimator

gr_sync_decimator is derived from gr_sync_block and implements a N:1 block with optional history.

  • gr_sync_interpolator

gr_sync_interpolator is derived from gr_sync_block and implements a 1:N block with optional history.

With this knowledge it should be clear that howto_square_ff should be a gr_sync_block with no history.

So let's write another block, which does the same as before, but is a sync block. Another invocation of gr-modtool is our friend:

tmp/gr-howto % gr_modtool.py add -t sync square2_ff
Operating in directory .
GNU Radio module name identified: howto
Code is of type: sync
Block/code identifier: square2_ff
Full block/code identifier is: howto_square2_ff
Enter valid argument list, including default arguments:
Add Python QA code? [Y/n] n
Add C++ QA code? [Y/n] n
Traversing lib...
Adding file 'howto_square2_ff.h'...
Adding file 'howto_square2_ff.cc'...
Traversing swig...
Editing swig/howto_swig.i...
Traversing grc...
Adding file 'howto_square2_ff.xml'...
Editing grc/CMakeLists.txt...

Again, we skip the QA file generation because we'll just use the other one.
In fact, the test is exactly the same. Here's a qa_howto.py file for both blocks:

source:gr-howto-write-a-block/python/qa_howto.py

Running make test now will spawn a test run with of qa_howto.py which should not fail.

Inside the work() function

If you're using a sync block (including decimator and interpolator), this is how the skeleton code looks like produced by gr_modtool:

int
my_block_name::work (int noutput_items,
      gr_vector_const_void_star &input_items,
      gr_vector_void_star &output_items)
{
    const float *in = (const float *) input_items[0];
    float *out = (float *) output_items[0];

    // Do <+signal processing+>

    // Tell runtime system how many output items we produced.
    return noutput_items;
}

So, given history, vectors, multiple input ports etc., is this really all you need? Yes it is! Because sync blocks have a fixed output to input rate, all you need to know is the number of output items, and you can calculate how many input items are available.

Example - the adder block: source:gnuradio-core/src/lib/gengen/gr_add_XX.cc.t

This block has an unknown number of inputs and variable vector lengths. The number of connected ports can be checked by input_items.size() and output_items.size(). The outer for loop, which goes over all the available items, goes up to noutput_items*d_vlen. The number of output items is identical to the number of input items because it is a sync block, and you can trust GNU Radio to have this number of items available. In this case, one item is a vector of samples, but we want to add the individual samples, so the for loop considers that.

Example - gr_unpack_k_bits_bb: source:gnuradio-core/src/lib/general/gr_unpack_k_bits_bb.cc

This is a block which picks apart bytes and produces the individual bits. Again, it is unknown at compile time how many bits per byte there are. However, there's a fixed number of output items per input item, so we can simply divide noutput_items/d_k to get the correct number of input items. It will always be correct, because GNU Radio knows the input to output ratio and will make sure that noutput_items is always an multiple of this integer ratio.

Example - history in source:gr-digital/lib/digital_diff_phasor_cc.cc

If you use history of length k, GNU Radio will keep k-1 entries of the input buffer instead of discarding them. This means that if GNU Radio tells you the input buffer has N items, it actually has N+k-1 items you may use.

Consider the example above. We need one previous item, so history is set to k=2. If you inspect the for loop closely, you'll find that out of noutput_items items, noutput_items+1 items are actually read. This is possible because there is an extra item in the input buffer from the history.

After consuming noutput_items items, the last entry is not discarded and will be available for the next call of work().

Help! My test fails!

Congratulations! If your test fails, your QA code has already paid for itself. Obviously, you want to fix everything before you continue.

You can use the command ctest -V (instead of make test) to get all the output from the tests. You can also use ctest -V -R REGEX to only run tests that match REGEX, if you have many tests and want to narrow it down. If you can't figure out the problem from the output of your QA code, put in print statements and show intermediary results. If you need more info on debugging blocks, check out the debugging tutorial.

Making your blocks available in GRC

You can now install your module, but it will not be available in GRC. That's because gr-modtool can't create valid XML files before you've even written a block. The XML code generated when you call gr_modtool add is just some skeleton code.

Once you've finished writing the block, gr_modtool has a function to help you create the XML code for you. For the howto example, you can invoke it on the square2_ff block by calling

$ gr_modtool.py makexml square2_ff

In most cases, gr_modtool can't figure out all the parameters by itself and you will have to edit the appropriate XML file by hand. The GRC wiki site has a description available.

For the blocks written in tutorial 2, the valid XML files look like this:

source:gr-howto-write-a-block/grc/howto_square_ff.xml
source:gr-howto-write-a-block/grc/howto_square2_ff.xml

There's more: additional gr_block-methods

If you've read the gr_block documentation (which you should have), you'll have noticed there are a great number of methods available to configure your block.

Here's some of the more important ones:

set_history()

If you're block needs a history (e.g. something like an FIR filter), call this in the constructor. GNU Radio then makes sure you have the given number of 'old' items available.

forecast()

Looking at general_work() you may have wondered how the system knows how much data it needs to ensure is valid in each of the input arrays. The forecast() method provides this information.

The default implementation of forecast() says there is a 1:1 relationship between noutput_items and the requirements for each input stream. The size of the items is defined by gr_io_signatures in the constructor of gr_block. The sizes of the input and output items can of course differ; this still qualifies as a 1:1 relationship.


  // default implementation:  1:1
  void
  gr_block::forecast (int noutput_items,
                      gr_vector_int &ninput_items_required)
  {
    unsigned ninputs = ninput_items_required.size ();
    for (unsigned i = 0; i < ninputs; i++)
      ninput_items_required[i] = noutput_items;
  }

Although the 1:1 implementation worked for howto_square_ff, it wouldn't be appropriate for interpolators, decimators, or blocks with a more complicated relationship between noutput_items and the input requirements. That said, by deriving your classes from gr_sync_block, gr_sync_interpolator or gr_sync_decimator instead of gr_block, you can often avoid implementing forecast.

Note that if you've already got a history set, you usually don't need to set this.

set_output_multiple()

When implementing your general_work() routine, it's occasionally convenient to have the run time system ensure that you are only asked to produce a number of output items that is a multiple of some particular value. This might occur if your algorithm naturally applies to a fixed sized block of data. Call set_output_multiple in your constructor to specify this requirement. The default output multiple is 1.

Finalizing your work and installing

First, go through this checklist:
  • Have you written one or more blocks, including QA codes?
  • Does make test pass?
  • Are there GRC bindings available (if that's what you want)?

In that case, you can go ahead and install your module. On a Linux machine, this would mean going back to the build directory and calling make install:

$ cd build/
$ sudo make install

With Ubuntu, you usually have to call ldconfig as well:

$ sudo ldconfig

Otherwise, you'll get an error message that the library you just installed cannot be found.

Other types of blocks

Sources and sinks

Sources and sinks are derived from gr_sync_block. The only thing different about them is that sources have no inputs and sinks have no outputs. This is reflected in the gr_io_signatures that are passed to the gr_sync_block constructor. Take a look at "gr_file_source.{h,cc}":source:gnuradio-core/src/lib/io/gr_file_source.cc and gr_file_sink.{h,cc} for some very straight-forward examples. See also the tutorial on writing Python applications.

Hierarchical blocks

For the concept of hierarchical blocks, see this. Of course, they can also be written in C++. gr-modtool supports skeleton code for hierarchical blocks both in Python and C++.

tmp/gr-howto % gr_modtool.py add -t hiercpp hierblockcpp_ff
Operating in directory .
GNU Radio module name identified: howto
Code is of type: hiercpp
Block/code identifier: hierblockcpp_ff
Full block/code identifier is: howto_hierblockcpp_ff
Enter valid argument list, including default arguments: 
Add Python QA code? [Y/n] n
Add C++ QA code? [Y/n] n
Traversing lib...
Adding file 'howto_hierblockcpp_ff.h'...
Adding file 'howto_hierblockcpp_ff.cc'...
Traversing swig...
Editing swig/howto_swig.i...
Traversing grc...
Adding file 'howto_hierblockcpp_ff.xml'...
Editing grc/CMakeLists.txt...
tmp/gr-howto % gr_modtool.py add -t hierpython hierblockpy_ff
Operating in directory .
GNU Radio module name identified: howto
Code is of type: hierpython
Block/code identifier: hierblockpy_ff
Full block/code identifier is: howto_hierblockpy_ff
Enter valid argument list, including default arguments: 
Add Python QA code? [Y/n] n
Traversing python...
Adding file 'hierblockpy_ff.py'...
Traversing grc...
Adding file 'howto_hierblockpy_ff.xml'...
Editing grc/CMakeLists.txt...

Everything at one glance: Cheat sheet for editing modules/components:

Here's a quick list for all the steps necessary to build blocks and out-of-tree modules:

  1. Create (do this once per module): gr_modtool create MODULENAME
  2. Add a block to the module: gr_modtool add BLOCKNAME
  3. Create a build directory: mkdir build/
  4. Invoke the make process: cd build && cmake .. && make (Note that you only have to call cmake if you've changed the CMake files)
  5. Invoke the testing: make test or ctest or ctest -V for more verbosity
  6. Install (only when everthing works and no tests fail): sudo make install
  7. Ubuntu users: reload the libs: sudo ldconfig
  8. Delete blocks from the source tree: gr_modtool rm REGEX
  9. Disable blocks by removing them from the CMake files: gr_modtool disable REGEX

Writing a signal processing block in Python

This lives on a separate page.

Debugging blocks

Debugging GNU Radio is available as a separate tutorial.