[ Timing Latency | ^ Top ^ ]

USB Controller FX2 Questions

• Am I correct in saying that the usrp code treats the device as a stream of data entering the system via the USB connection?

It's a little more complicated than that, but that's the basic idea. USB supports three kinds of transfers across the USB: command, bulk read/write and isochronous. We use command packets to configure the USRP, load firmware, the FPGA bitstream, etc. We use endpoint 0 for that interface. In addition, we use two other endpoints, one for streaming input data and the other for streaming output data. These endpoints are used with bulk transfers, since they are the USB transfers that support the highest throughput. I suggest reading chapter 9, "USB Device Framework" of the USB 2.0 specification. It covers most of what you need to know to build a USB peripheral.

• How many endpoints we can support with FX2?

The FX2 is limited in the number of endpoints that it supports. See page 15 of the CY7C68013 datasheet. Right now we are running with endpoints 2 and 6 "quad buffered", 4 512 byte buffers each. One is used for Tx data, the other for Rx data. It would be possible with a bit of hacking to run endpoints 2, 4, 6 and 8 "double buffered", 2 512 byte buffers each. That would give us 4 bulk endpoints to work with (and would also lower the worst case latency).

• I am looking at the tx_buffer module. From my understanding, that module does the interlacing of the data to be transmitted (2 I channels and 2 Q channels). Is that correct?

It implements the transmit direction part of the FX2/FPGA GPIF interface. It also demuxes the data to be transmitted and sends it to the appropriate DACs.

• Does FX2 reformat incoming/outgoing FPGA data?

No. The formatting of the data is all handled in the FPGA. The FX2 firmware doesn't even see the bulk data going in and out. The data is transferred by DMA to/from the GPIF from/to the FPGA USB buffers.

• Also, next to the bus_reset input declaration there is a comment saying “Used here for the 257 hack to fix the FX2 bug". Which bug is it talking about?

On the FX2, the WR pulse is asserted one clock longer than it should be.

• I think that along the way I misunderstood the purpose of the write_count register. How does it actually work? WR triggers every time a 16 bit packet is ready from the FX2 doesn't it?

write_count counts from 0 to 256, then back to 0. It's at 256 when WR is still asserted but there's really no data to receive. This works around some strange behavior in the FX2 GPIF interface and/or programming.

• The wreq trigger of the FIFO is triggered by (WR & ~write_count[8]). Does this mean that only 256 16 bit samples enter the FIFO before the WR is removed? Why is this? How could I determine exactly when there is I or Q sample that must be written into the FIFO?

wrreq tells the FIFO when data should be written to the FIFO. So, we write when (WR & ~write_count[8]). That is, when WR is asserted, but the count does not have 0x100 bit set. As I recall, WR is asserted an extra cycle, and the counter trick works around this.

The block called tx_fifo is one of their standard blocks. It's the dual clock version of the fifo, and is used (amongst other things), to bridge between the USB clock domain (.wrclk(usbclk)) and the signal processing clock (.rdclk(txclk)).

• When the FX2 detects the have_space pin on the FPGA, does it transfer 1 entire buffered USB packet to the FPGA, then re-check the have_space pin, right?

Yes. Have_space is used by the feeding FX2 to know if the FPGA can handle one more packet. The have_space pin is used by the FX2 to know if the FPGA FIFO can store AT LEAST one more packet.

• Would it be reasonable to assume a 1 clock delay between the last byte of one 512-byte packet being written to the FPGA and the first byte of a second 512-byte packet being written to the FPGA?

Yes. That shouldn't be a problem. There's software inside the FX2 that polls the pin. You've got at least 100 ns between packets, probably more.

• I couldn't find any documentation on what is stored in the USRP motherboard EEPROM?

On the motherboard, we're currently using a "C2" formatted EEPROM so that we can run code at power-up time to get the 9862's into a reasonably low-power state. The details of the C0 vs C2 FX2 boot can be found in the FX2 Technical Reference Manual. The EEPROM on the motherboard stores some USB info, and some very simple code to put the board in a low power state when it powers up. It also blinks one LED quickly. Once you start running anything on the USRP, a more complete firmware (which is much bigger than the EEPROM would hold) is sent over the USB bus.

• What about the USRP operation of Cypress EZ-USB FX2 USB Microcontroller?

Many points:

1) Official documentation

2) The main controlling program is usrp_main.c (compiled firmware for the 8051 in the FX2 USB chip by SDCC) found in usrp/firmware/src/usrp2.

3) The following loops were examined from the usrp_main.c

 main( ) 
 {
 Initialize USRP 
 Initialize GPIF 
 Patch USB Descriptors (read from EEPROM, set HW device ID) 
 Setup Autovectors 
 Install USB Handlers 
 Re-enumerate 
 Run main_loop( ) 
 }
 
 main_loop( )
 { 
 Check for USB setup packets 
 Check and log the RX Overruns and the TX Underruns 
 Check for packets to send back to host 
 Check for packets to send to FPGA 
 }

4) USB Endpoints:

Different USB endpoints (EP) are used to logically separate different operations occurring on the bus into separate flows. There are currently 3 USB endpoints checked/used within the main_loop() as described earlier.

Endpoint	Description
0	Control/status
2	Host -> FPGA
6	FPGA -> Host

5) USB Transfers are always 512-byte bulk transfers. All control information is written using endpoint 0 and the vendor commands. These commands are separated into two different categories: VRT_VENDOR_IN and VRT_VENDOR_OUT. These are processed in the app_vendor_cmd( ) function seen in usrp_main.c.

All control communication between the FPGA and FX2 microcontroller is done over the SPI. Daughterboards all seem to be controlled by the generic VRQ_I2C_* and VRQ_SPI_* commands.

6) VRT_VENDOR_IN Commands:

VRQ_GET_STATUS
GS_TX_UNDERRUN
GS_RX_OVERRUN
VRQ_I2C_READ
VRQ_SPI_READ
7) VRT_VENDOR_OUT Commands:

VRQ_SET_LED
VRQ_FPGA_LOAD
FL_BEGIN
FL_XFER
FL_END
VRQ_FPGA_SET_RESET
VRQ_FPGA_SET_TX_ENABLE
VRQ_FPGA_SET_RX_ENABLE
VRQ_FPGA_SET_TX_RESET
VRQ_FPGA_SET_RX_RESET
VRQ_I2C_WRITE
VRQ_SPI_WRITE

• How the USRP FX2 works?

The FX2 microcontroller contains an embedded USB 2.0 transceiver and handles all USB transfers with the upstream USB host. It presents a data bus to the outside world (in this case the FPG , with generic control signals which can be programmed to behave in a custom manner. This interface is called the GPIF (General Purpose Interface),

The FX2 also handles all USB control requests (via endpoint 0), which all USB-enabled devices must support to fully comply with the USB standard. These include responses to device capability interrogations and standard setup requests.

FX2 Internal Block Diagram (From Kalen Thesis)

A simplified view of the FX2 is shown in Figure above. It houses an industry-standard 8051 microcontroller core (with several extensions and enhancements), which handles all internal control. The 8051 initializes the transceiver, which handles the actual USB transactions. It is also responsible for configuring the FX2's general-purpose I/O ports (not shown) and the GPIF state machine. Data is transferred in a USB system via endpoints, which are similar to Ethernet network socket. Each endpoint must have a specified data-flow direction, either IN or OUT (USB-centric), except the control endpoint 0 which is bidirectional. The endpoints, must also have defined data transaction types which indicate their bandwidth requirements.

Data entering or leaving the FX2 on the USB host side is stored in the endpoint FIFO’s, which can be configured to have various sizes and levels of buffering. The GPIF has direct access to these FIFO’s, allowing for seamless data transfer between an external device and the USB host through a series of buffered FIFO’s.

• What about FX2 firmware?

As well as general house-keeping and configuration, the FX2 firmware is responsible for the following areas:

- GPIF initialization

Rather than attempt to handle data transfers directly at USB 2.0 high-speed (480MBits/s), data transfers are carried out by the GPIF. The FX2's 8051 core initializes this interface.

- USB control request handling

The 8051 core responds to control requests sent by the USB host over endpoint 0. All USB compliant devices must return responses in a pre-determined format to standard requests about the device (e.g. GetDeviceName which returns the device's description).

- USB transfer requests

These are implemented as a polling loop that loads GPIF setup registers with the transfer parameters as and when requests are received from the host. The FX2 firmware is written in C, and compiled with the open-source compiler, SDCC (Small Device Cross Compiler). The embedded functions are invoked using the usrp_basic and usrp_prims libraries.

• What is the FX2 GPIF?

The GPIF (General Purpose Interface) is a mechanism implemented by the FX2 to allow for simple interfacing with many different types of devices. Essentially it is a bi-directional data bus with a set of generic control signals which are governed by a configurable state machine. The GPIF can be configured to be the interface bus master or can be driven as a slave by the device itself. The GPIF is the bus master. Six states can be defined for a particular waveform, with decision points that dictate the state transitions depending on the values of the generic control lines.

There are four bus cycle waveforms available which can be configured. These are:

- Single Write

This bus cycle writes a single data word from the USB to the device (all transactions are defined to be USB-centric)

- Single Read

This cycle reads a single word from the device.

- FIFO Write

This cycle writes a block of data based on some previously setup parameters. During the transfer state, data flow can be throttled by the receiving device.

- FIFO Read

This cycle reads a block of data from the device, and is used to stream ADC data up to the host. An example of a simple GPIF FIFO read waveform is shown in figure below.

GPIF waveform Example (From Kalen Thesis)

The GPIF has six control inputs RDY5:0, and six control outputs CTL5:0. These can be used to implement a great variety of standard and proprietary bus control cycles. In the figure above, the decision point in state 1 will hold the cycle in a stalled state until RDY0 (in this case representing FIFO Empty Flag) is de-asserted when the GPIF state machine will advance to state 2 and latch in the data presented on the DATA bus. The FX2 implements the USB endpoints as internal endpoints EP0, EP1, EP2, EP4, EP6 and EP8, which can be configured in various ways to suit the application’s buffering requirements. EP0 is always a 64 byte CONTROL endpoint and EP1 may only be of the BULK or INTERRUPT types (also 64 bytes). The remaining endpoint FIFO’s can be configured to be double-, triple- or quad-buffered, with a maximum total buffer size of 4K. In this manner, data can be continually streamed to or from the host, provided adequate flow control is in place.

Cypress Semiconductor has developed GPIF Designer, a freely available application which can be used to design GPIF state machines with a graphical interface. The application allows a user to customize the state machines, after which the appropriate configuration and initialization source code is generated which can be included with the main FX2 firmware. This simplifies the process of configuring the GPIF waveform registers which would otherwise have to be coded manually. In the USRP's case, the developers chose to parse the GPIF Designer's output code, and to insert their own initialization routines. This was done in part because the output code generated by GPIFDesigner is somewhat ambiguous (bugs have been encountered by the GnuRadio team).

Internally in USRP, the GPIF acts autonomously from the actual FX2 8051 core and provides data to and from the internal endpoint FIFO’s for streaming up to and down from the USB host. The USRP FX2 interface at the application level is defined and implemented in the usrp_prims and usrp_basic source and header files.

The Usrp_prims defines a set of functions implemented on the FX2 in response to control requests via USB endpoint 0 that extend the generic USB function set. These function primitives typically write to or read from various FX2 registers, or initiate data transfers via the GPIF.

The Usrp_basic provides a simple API to the usrp_prims functions in the form of an object-oriented structure. The interface to the system provides higher-level functionality, hiding most of the underlying interface mechanisms.

• What is the usrper?

It is a data streaming test application. It can load FPGA configuration data (fed serially into the configuration lines of the FPGA bitwise), assert debug USRP LEDs, read and write data to the USRP.

On start up, usrper first attempts to create an interface object that can communicate with a valid USRP device. This creates a handle to a generic USB device, and then traverses all the devices found attached to the USB, searching for the FX2's vendor and product ID numbers (0x04B4 and 0x8613 respectively). On successfully locating a powered, un-configured FX2 device, usrper then downloads the GnuRadio firmware to the FX2, which sets up the system. After system setup, the firmware loads custom values into the Vendor and Product ID fields and initiates a software reset, causing the FX2 device to disconnect and then reconnect to the USB bus.

This process is called re-enumeration, and the Linux kernel now sees the USRP board as a USB device with the Product ID and Vendor ID 0xFFFE and 0x0002, which correspond to a configured USRP device developed by the Free Software Foundation.

Once this has been achieved, calls are made to USRP-specific methods which can be reads, writes or control requests.

- Control requests:

These enable or configure various aspects of the USRP system, and are made to USB endpoint 0. The firmware intercepts these calls (interrupts are generated on all USB control requests that are received from the host) and services them. They form vendor-extension functions, and carry out various data transfers to and from internal registers based on the host's control request parameters. For example to download the FPGA’s firmware, the configuration bit stream data is sent via endpoint 0 with the appropriate control parameters indicating a request to configure the FPGA. The FX2 intercepts each byte in this stream, and sends it bitwise over the FPGA’s configuration lines (the Altera EP1C12 FPGA supports serial configuration modes). While servicing the control request, the FX2 firmware sends back the appropriate handshaking signals to the host to indicate that the control request is being handled correctly.

- Read requests:
The USB requests data from the USRP by making read requests over endpoint 2 (set up for 512 byte bulk IN transfers). The FX2 intercepts these requests and primes the GPIF transfer state machine with the transfer parameters before relinquishing control to the GPIF.

The GPIF enters the FIFO read state machine and remains in this configuration until the transfer is complete, or an error has occurred. The FX2 handles all acknowledge and handshaking signals with the USB host transparently from the firmware.

- Write requests:

As with read requests, the host PC makes generic USB write requests, this time to endpoint 6 (configured for 512 byte bulk OUT transfers). The GPIF in this time set up for the FIFO write state machine, and transfers data until the transaction is complete, or an error is encountered.

• What does the FX2 low-level interface library (USRP_PRIMS.CC) do?

This library code forms the direct interface with the FX2 as a set of low-level functions. Data transfers and various control requests are passed to the FX2 via USB endpoint 0, while the data received is sent back via endpoint 2.

• What does the FX2 application-level interface library (USRP_BASIC.CC) do?

This library presents a higher-level interface to the FX2, in the form of object-oriented methods which call the functions in usrp_prims.cc.

• Why we use GPIF mode?

We use the FX2 in GPIF mode because it allows us to burst data across the GPIF interface at 96 MB/sec.

• USRP-FX2 interface - As I understand it, all control calls to endpoint 0 are handled within the FX2 and do not require the involvement of the FPGA, right?

Yes

• GPIF / FPGA firmware - How many GPIF waveform configurations do you implement?

Two: FIFORd and FIFOWr

• Do you handle FPGA configuration explicitly using the FX 8051 core?

We bit bang the FPGA configuration from the 8051.

• Please could you point me to the correct FX2 source to determine your GPIF CTLx/RDYx lines and waveform descriptions,?

We used the Cypress GPIF designer. See gpif.gpf.

• How we can modify the USRP which uses USB2.0 (480Mbit/sec) high speed interfaces to use USB 1.1 (12Mbyte/sec) full speed controller. The USB 1.1 sends 64 byte packets, while USB2.0 uses 512 byte packets?

In tx_buffer.v and look at line 94, you will see the test for "end of packet". write_count[8] goes high when we have put 256 elements (512 bytes) into the FIFO. You would need to modify this to write_count[5] which will go high when 32 elements (64 bytes) have been put into the FIFO.

You probably also need to modify some firmware. Also you have to decimate the source down to a very low data rate, so it won't overflow.

Note: There is a suggested patch to make USRP able to use USB 1.1.

• Doing suggested above modifications, it looks like data is coming out, but it looks like I get 64 bytes out, then there is a "hiccup" about 5 microseconds long. I am getting suspicious the PC doesn't get the data on the USB bus fast enough. I am wondering if there is a way to let the buffers in the FX2 chip fill up more before the FPGA starts pulling data from the FX2?

Unless your FX2 code sets up the transfer size to 64, then this is probably what is happening. That is, the host is sending 64 bytes, but the FX2 is ignoring the real length, and is assuming that it's 512 bytes. Remember that the GPIF is currently set up to DMA 256 16-bit values. Perhaps that part needs changing. The magic value is probably buried in the WaveData? table in usrp_gpif.c.

Another thing you could try is to set up the GPIF in a non-flowstate mode. You'd need to use the Cypress tool to do this (or Larry Doolittle's perl (?) code. Running in full speed, you don't need to be able to burst data at 96MB/sec between the FX2 and FPGA.

• I looked over the GPIF stuff with the Cypress tool I do not see any reference to transfer size in there. I am thinking the transfer size is set in the FPGA?

Nope. The GPIF is in charge of the transfer. I'm assuming that what you are seeing is that the FX2 is bursting a 256 word transfer, given how you have programmed everything. BTW, I'm not kidding about this being nearly impossible without a logic analyzer. What's currently really happening on the GPIF bus? (Bring out the relevant GPIF pins to the daughterboard debug headers.) Looking at page 10-16 (10.3.2.2.2 Decision Point States) of the FX2 Technical Reference Manual (and dusting off my memory), the waveform decision point is controlled by the "Transaction Count Expired" signal. Page 10-24, "LOGIC FUNCTION Register", TERMA and/or TERMB will be coded as RDY5 (or Transaction-Count Expiration, if GPIFREADYCFG.5 = 1 (which it is)). See page 10-41, 10.4.3.1 "Transaction Counter".

To use the Transaction Counter for FIFO "x" load GPIFTCB3:0 with the desired number of transactions. When a FIFO-READ or -WRITE waveform is triggered on that FIFO, the GPIF will transfer the specified number of bytes (or words, if WORDWIDE=1).

See ### HERE ### below (from usrp/firmware/src/usrp2/usrp_main.c):

Are you setting these to 32 instead of 256? If not, that's probably the root of the problem.

       // Next see if there are any "OUT" packets waiting for our attention, 
       // and if so, if there's room in the FPGA's FIFO for them. 
 
       if (g_tx_enable && !(EP24FIFOFLGS & 0x02)){  // USB end point fifo is not empty... 
 
         if (fpga_has_room_for_packet ()){   // ... and FPGA has room for packet 
 
           GPIFTCB1 = 0x01; SYNCDELAY;        #######  HERE ####### 
           GPIFTCB0 = 0x00; SYNCDELAY;        #######  HERE ####### 
 
           setup_flowstate_write (); 
 
           SYNCDELAY; 
           GPIFTRIG = bmGPIF_EP2_START | bmGPIF_WRITE; // start the xfer 
           SYNCDELAY; 
 
           while (!(GPIFTRIG & bmGPIF_IDLE)){ 
             // wait for the transaction to complete 
           } 
         } 
       } 
 
       // See if there are any requests for "IN" packets, and if so 
       // whether the FPGA's got any packets for us. 
 
       if (g_rx_enable && !(EP6CS & bmEPFULL)){ // USB end point FIFO is not full... 
 
         if (fpga_has_packet_avail ()){ // ... and FPGA has packet available 
 
           GPIFTCB1 = 0x01; SYNCDELAY;       ####### HERE ####### 
           GPIFTCB0 = 0x00; SYNCDELAY;       ####### HERE ####### 
 
           setup_flowstate_read (); 
 
           SYNCDELAY; 
           GPIFTRIG = bmGPIF_EP6_START | bmGPIF_READ; // start the xfer 
           SYNCDELAY; 
 
           while (!(GPIFTRIG & bmGPIF_IDLE)){ 
             // wait for the transaction to complete 
           } 
 
           SYNCDELAY; 
           INPKTEND = 6; // tell USB we filled buffer (6 is our endpoint num) 
         } 
       }

Unless you get lucky and the modification above labeled "HERE" works, you're _really_ going to want access to a logic analyzer.

• I'm looking into writing a native USB driver for Windows and the USRP; I’m trying to find out some information on the USB interface of the USRP.

See:
usrp/firmware/include/usrp_interfaces.h
usrp/firmware/include/usrp_commands.h

• Does anyone know where the best place is to find documentation (if any) of the USB data structures that are sent on interface 0, 1 & 2 (ep0, ep2 and ep6) so that I can achieve this. I assume the ep0 data structures are that of the USB Device Request structure on the default control pipe, but I also assume there are more control requests than the standard USB ones.
  
  Another thing I was wondering, since the vendor id is a standard free for all type, how do you actually determine if it is indeed a USRP that you are connected to, and on top of that if you are connected to an FX2 chip which is not the USRP and has not been initialized, can you determine that it is an FX2 that in on a USRP or not?

All the shipped USRPs have a non-Cypress USB VID/PID burned into them. There is no confusion between a USRP and an un-programmed FX2. We use: VID = 0xfffe, PID = 0x0002. We distinguish board revs and whether or not we've loaded our firmware via the DID.

See usrp/firmware/include/usrp_ids.h.

Pretty much every question about how to control the USRP is answered in one of the .h files in usrp/firmware/include. In particular; see fpga_regs_common.h and fpga_regs_standard.h for the definitions of the configuration registers in the FPGA.

• My problem is that I completely miss the low level USB programming skills and I hoped to find some sort of driver for the Cypress chip in Linux.

No need for any driver programming. Just use libusb.

• How do I ensure that the appropriate usb driver(s) are associated with my USB device when I connect to the system?

Here is the outline of a few steps:
1. Assign yourself a USB product ID in the space defined by: firmware/include/usrp_ids.h and get the USRP developers to acknowledge it.
2. Hack host/lib/usrp_prims.cc to accept your PID in addition to the normal USRP ones.
3. Program the FX2 EEPROM to your product ID using usrper i2c_write.
4. Add a usermap file and initialization script to /etc/hotplug/usb/

• Is it feasible to modify existing FX2 firmware to my own needs for testing purposes?

Absolutely. You can use a slightly modified usrper and test_usrp_standard_rx to get yourself going until you build your own customized executable from the libraries.

• We created a custom board based on the USRP. I'd like to avoid recompiling the FX (8051) source code if possible. Is there a function that will allow me to write directly to one of the FX registers? I want to change the frequency of the CLK output from the default of 12 MHz to 24 MHz.

No command. In USRP we run the FX2 at 48 MHz after our firmware is loaded.

• It was said that USRP FPGA FIFO Buffers is 2K lines. Does it mean 2048 bits?

No, it means 2K lines, each line is 32 bits. The FX2 also implements quad buffering in both TX and RX directions, each buffer is 512 bytes.

• What do the parameters fusb_nblock and fusb_block_size exactly do in how data is transmitted over the USB to the USRP?

The fusb_block_size is the size in bytes of the maximum transfer that we will ask the kernel to make to/from user-space. The fusb_nblock is the maximum number of transfers (of maximum size fusb_block_size) that we can have in flight at any given time. Take a look at fusb_linux.{h,cc} for the details.

• If there is less data available on the USRP (i.e. not fusb_nblock*fusb_block_size), does it still get sent over the USRP to the computer?

Yes. The USRP packages data into 512 byte USB packets and sends them as soon as it can. That's 128 complex samples (16-bit I & Q).

• How is the smallest amount of data necessary in the USRP defined such that a packet is sent over the USB to the computer? Is it fusb_block_size?

Its 512 bytes. It's set in the FX2 firmware.

• Looking through the FX2 data sheet, the GPIF designer app's generated code and your edit_gpif script output, I (hope) I have an understanding of the actual data flow bus cycle (I am only focusing on FIFORd): You use flow states. There is a flowstate in state S1, which constantly asserts REN and OE (and BOGUS) while the transaction has not yet expired. It de-asserts all CTLx signals when the transfer is complete. USRP do data transfers on both edges of IFCLK. The FLOWSTB register indicates that CTL4 is your master strobe. I thought CTL4 was a reset signal (CLRST). To summarize: OE and REN are active-high, and data is clocked out on both edges of IFCLK. Is this correct?

Yes, modulo problem with signals being asserted one cycle too long on WR.

• What part do the RDYx pins play in the flowstate?

None.

• Do they only affect transitions between S1, S2..S6?

Nope, not that either.

[ Timing Latency | ^ Top ^ ]