Cross Compiling GNU Radio for the Cell Processor

This page explains how to compile GNU Radio on an x86 or x86_64 platform, and generate code for the Cell processor. This is most useful when developing for the Sony PS3 which, because of it's small main memory size (256MB), is a poor platform for software development.

Overview

There are several ways one can go about cross compiling anything. There are always at least two machines involved: the build machine where the compilation takes place, and the host machine where the resulting software runs. Oftentimes these machines have differing architectures. In our case, the build machine is an x86 or x86-64 workstation running GNU/Linux, while the host machine is a Cell based machine also running GNU/Linux. (These instructions have been tested using a PS3, but are expected to work on a Cell bladeserver such as the IBM QS21 too.)

The strategy described here requires that the build machine have read-only access via NFS to the root file system of the Cell host, and that both machines have read/write access to a shared NFS file system.

Prerequisites

In order to be concrete, we call the build machine build and the host machine ps3.

  • Both build and ps3 shall have the free part of the IBM SDK 3.0 installed. It contains the cross compilers, linkers, etc. (the cross toolchain.)

At present the only RPM's available are for x86, x86_64 or Cell/BE machines running Fedora 7. They also work fine on Fedora 8. It should be possible to build the cross toolchain from source on virtually any machine, but that is beyond the scope of this page.

  • Make sure that both build and ps3 have the ppu32-gcc compiler in their PATHs, otherwise:

export PATH=/opt/cell/toolchain/bin:$PATH

  • Both build and ps3 shall function as both NSF servers and NFS clients.

Some PS3 kernels don't come with the NFS server module built by default. In those cases you'll have to configure, build and install a kernel with the appropriate features enabled. This kernel .config file is known to work with the linux-2.6.23-20070817 kernel.

  • Your numeric uid and gid shall be the same on build and ps3. E.g., 
    build$ id
uid=500(eb) gid=500(eb) groups=3(sys),10(wheel),500(eb),1000(usrp)

ps3$ id
uid=500(eb) gid=500(eb) groups=4(adm),10(wheel),500(eb),1000(usrp)
  • Both build and ps3 shall have ntp installed and operating and their clocks shall be synchronized. 
    $ sudo yum install ntp
$ sudo chkconfig --level 35 ntpd on
$ sudo service ntpd start

Details

Congratulations on making it this far! You have handled all the prerequisites described above, right? If not, please take care of them now. Google or your local sys admin can provide assistance.

To cross compile there is some one-time stuff that needs to be done to set up everything between build and ps3, and then there's the actual steps for cross compiling GNU Radio.

Setting up build and ps3 so that they can talk

We need to arrange for build to be able to see the installed packages, includes and libraries on ps3 so that we can compile and link against them. In addition, so that we can compile the software on build but run the QA code (make check) on ps3, we must arrange that they see a common file system, with identical path names to the shared file system from the root of each system. (configure hard-codes absolute file names into the generated Makefile's).

Export the ps3 root filesystem read-only

You'll need to edit or create /etc/exports on ps3 so that it contains something like this: 

/       192.168.64.0/24(ro,all_squash)

Use man exports if you've got questions. Then run exportfs to have it take effect. 

ps3$ sudo exportfs -a

Mount the ps3 root filesystem on build

On build create a mount point named /mnt/cell-root and mount the ps3 root there. 

build$ sudo mkdir /mnt/cell-root
build$ sudo mount -o ro ps3:/ /mnt/cell-root

At this point you should be able to see the ps3 root filesystem from build. (The warnings and question marks are normal. Special files aren't accessible across NFS.) 

build$ ls -l /mnt/cell-root
ls: cannot access /mnt/cell-root/net: No such file or directory
ls: cannot access /mnt/cell-root/sys: No such file or directory
ls: cannot access /mnt/cell-root/dev: No such file or directory
ls: cannot access /mnt/cell-root/spu: No such file or directory
ls: cannot access /mnt/cell-root/home: No such file or directory
ls: cannot access /mnt/cell-root/proc: No such file or directory
ls: cannot access /mnt/cell-root/misc: No such file or directory
total 156
drwxr-xr-x  2 root root  4096 2007-11-08 04:56 bin
drwxr-xr-x  4 root root  4096 2007-10-25 17:34 boot
??????????  ? ?    ?        ?                ? dev
drwxr-xr-x 99 root root 12288 2007-11-08 15:18 etc
??????????  ? ?    ?        ?                ? home
drwxr-xr-x 16 root root  4096 2007-11-08 04:55 lib
drwxr-xr-x  7 root root  4096 2007-11-08 04:55 lib64
drwx------  2 root root 16384 2007-10-04 13:22 lost+found
drwxr-xr-x  2 root root  4096 2007-11-07 15:07 media
??????????  ? ?    ?        ?                ? misc
drwxr-xr-x  5 root root  4096 2007-11-08 13:12 mnt
??????????  ? ?    ?        ?                ? net
drwxr-xr-x  3 root root  4096 2007-10-24 13:41 opt
??????????  ? ?    ?        ?                ? proc
drwxr-xr-x  2 root root  4096 2007-11-08 19:50 proj
drwxr-x---  6 root root  4096 2007-11-07 13:55 root
drwxr-xr-x  2 root root 12288 2007-11-08 04:56 sbin
drwxr-xr-x  2 root root  4096 2007-10-04 13:26 selinux
??????????  ? ?    ?        ?                ? spu
drwxr-xr-x  3 root root  4096 2007-10-04 14:14 srv
??????????  ? ?    ?        ?                ? sys
drwxrwxrwt 14 root root  4096 2007-11-10 00:16 tmp
drwxr-xr-x 16 root root  4096 2007-09-19 08:18 usr
drwxr-xr-x 21 root root  4096 2007-10-04 14:43 var

If this doesn't work, check that NFS is running on ps3 and build and that there aren't any firewall rules that might be keeping you from connecting. Look at /var/log/messages on both machines for clues.

Once this is working, add a line to /etc/fstab on build so that the filesystem is mounted automatically on boot. 

ps3:/                   /mnt/cell-root          nfs     ro,intr,bg      0 0

Export the shared filesystem and mount it on both systems

Now we need to create the shared directory on build, export it, and then mount it on both systems.

Find an existing filesystem on build that has at least a few gigs free, and create a directory called share there. It should be readable and writable by your regular login. The path isn't critical, just remember what you used. E.g., 

build$ mkdir /mnt/misc/share

Set up this directory for export by adding a line similar to this to /etc/exports on build: 

/mnt/misc/share         192.168.64.0/24(rw)

Then export it: 

build$ sudo exportfs -a

Create a mount point called /mnt/share on both machines: 

build$ sudo mkdir /mnt/share
ps3$ sudo mkdir /mnt/share

and mount it on both machines: 

build$ sudo mount build:/mnt/misc/share /mnt/share
ps3$ sudo mount build:/mnt/misc/share /mnt/share

You should be able to create a file from either machine and see it on the other. E.g., 

build$ date > /mnt/share/foo

ps3$ cat /mnt/share/foo
Sat Nov 10 02:29:44 PST 2007

If this isn't working, take a look at /var/log/messages on both machines.

Once this is working, add a line to /etc/fstab on both machines so that the mount will happen automatically at boot time: 

build:/mnt/misc/share /mnt/share                  nfs     rw,intr,bg      0 0

Tweak filesystem on PS3 for cross compilation

A few symlinks need to be created on ps3 and some linker scripts need to be edited. There's a script that will do this for us. The script is contained in the gnuradio trunk. As part of getting ready to do the cross compilation, check out the gnuradio trunk into a directory on /mnt/share. E.g., 

build$ mkdir /mnt/share/gr
build$ cd /mnt/share/gr
build$ svn co http://gnuradio.org/svn/gnuradio/trunk

Now, on ps3 do this: 

ps3$ sudo /mnt/share/gr/trunk/dtools/bin/tweak-cell-for-cross-compiling

Excellent! All the one-time set up is complete!

Do the Cross Compilation (Finally!)

Now on to the familar part...

To keep life simple we'll specify a --prefix that's on /mnt/share. (This isn't required, but start this way. Otherwise, depending on which machine you're on when you type "make install" you'll hose yourself with P(.5) by installing binaries for the wrong architecture into /usr/local). The key difference from native compilation is using the ./configure-cell-cross script instead of ./configure directly. It accepts all the normal configure options.

On build: 

build$ cd /mnt/share/gr/trunk
build$ ./bootstrap
build$ ./configure-cell-cross --prefix=/mnt/share/cell-install
build$ make  2>&1 | tee make.log

Assuming that configure and make completed successfully, we can switch to ps3 and run make check 

ps3$ cd /mnt/share/gr/trunk
ps3$ make check

This should also work fine. When you're happy: 

ps3$ make install

At this point the installed code should be ready to run. Be sure to set your PYTHONPATH and PATH to point into your prefix: 

ps3$ export PYTHONPATH=/mnt/share/cell-install/lib/python2.5/site-packages
ps3$ export PATH=$PATH:/mnt/share/cell-install/bin

Then try to run something: 

ps3$ usrp_fft.py ...

Trouble Shooting

What could possibly go wrong???