Difference between revisions of "TiLDA Debugging using JTAG"

Latest revision as of 14:35, 2 October 2014

The TiLDA badge has a JTAG port which can be used to perform remote firmware debugging. With the right hardware and software you can use this to run a GDB session on your development machine to insert breakpoints, inspect memory and poke around with the firmware running on the badge just like it was a local process running on your machine.

Prerequisites

I started the list below with the parts that worked for me. Feel free to add alternatives if you know they work for you.

X86-64 Linux machine for building the firmware and running GDB
TiLDA badge
- Plus micro USB cable for connecting TiLDA to development machine
JTAG header, 10 way (2 x 5 pins) 0.05" (or 1.27mm) through hole
- e.g. Samtec FTS-105-01-L-D
- or Harwin M50-3500542 (part number from the TiLDA schematic)
Fine tipped soldering iron
Olimex ARM-USB-TINY-H USB JTAG Interface
- Plus USB type A to B cable (the original & large square plug)
Olimex ARM-JTAG-20-10 20pin to 10pin adapter
OpenOCD software.
- This can control the Olimex JTAG interface and implements the GDB remote server protocol to talk with the debugger
GDB
- I used gdb-7.7.1-18.fc20.x86_64.
- An version of GDB for arm (arm-none-eabi-gdb) is bundled with the Arduino IDE and can be found in the Arduino/hardware/tools/gcc-arm-none-eabi-4.8.3-2014q1/bin folder of your arduino install (Note path may change slightly in later arduino build as newer version of gcc are included)
Arduino 1.5.7 tools
- For compiling & uploading the firmware image

Attach JTAG Header to badge

The header must be attached on the correct side of the board otherwise it won't work. The pins on the two sides of the header may be slightly different, one side is tin coated and appears silver/grey. These are the ones you should be soldering. The gold plated side should to be used by the connector.

JTAG header before being soldered on to the board

TiLDA with new JTAG header highlighted

Flip the board upside down and solder pins from the LCD side. The pitch is only 1.27mm (0.05") which is half the size of most normal connectors so you need a steady hand. After you have soldered the pins it is a good idea to test for any short circuits before you power it on.

Header soldered on LCD side of the board

Olimex adapter and USB-TINY-H

The cable from the Olimex adaper should be connected to the header so that the cable runs away from the board. Once again you may wish to check the connections appear to be the right way around, e.g. pin 1 on the adapter board connects to the 3v3 test point on the TiLDA. Connect the 20 pin side of the adapter into the main USB-TINY, this is keyed so it only fits one way around.

TiLDA connected to Olimex TINY-Y

Plug the USB-TINY into a USB port on your machine. The 'dmesg' output should report something like:

usb 2-2: new high-speed USB device number 111 using ehci-pci
usb 2-2: New USB device found, idVendor=15ba, idProduct=002a
usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 2-2: Product: Olimex OpenOCD JTAG ARM-USB-TINY-H
usb 2-2: SerialNumber: OLXxxxxx

Atmel ICE Basic

If you use an Atmel ICE then the cable needs to be connected so that it runs over the top of the board. The red stripe indicates pin 1.

TiLDA connected to Atmel ICE Basic

 
[jburgess@shark openocd]$ sudo openocd -f interface/cmsis-dap.cfg -f target/at91sam3XXX.cfg 
Open On-Chip Debugger 0.9.0-dev-00161-g9c47dc9-dirty (2014-09-30-23:42)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.sourceforge.net/doc/doxygen/bugs.html
Info : only one transport option; autoselect 'cmsis-dap'
adapter speed: 500 kHz
adapter_nsrst_delay: 100
cortex_m reset_config sysresetreq
Info : CMSIS-DAP: SWD  Supported
Info : CMSIS-DAP: JTAG Supported
Info : CMSIS-DAP: Interface Initialised (SWD)
Info : CMSIS-DAP: FW Version = 01.00.0021
Info : SWCLK/TCK = 1 SWDIO/TMS = 1 TDI = 1 TDO = 0 nTRST = 0 nRESET = 1
Info : DAP_SWJ Sequence (reset: 50+ '1' followed by 0)
Info : CMSIS-DAP: Interface ready
Info : clock speed 500 kHz
Info : IDCODE 0x2ba01477
Info : sam3.cpu: hardware has 6 breakpoints, 4 watchpoints

Install OpenOCD

I started with the current version available in the Fedora package repository (openocd-0.7.0-3.fc20.x86_64). Other Linux distros are likely to have packaged too.

OpenOCD stack pointer bug

After I got everything running I noticed that the GDB backtraces would show the current function (PC) and the caller (LR) but all further stack entries were missing. After some debugging I found that OpenOCD was incorrectly calculating the stack pointer when it tried to align it. I applied a quick fix to disable the broken alignment code:

--- openocd-0.7.0/src/rtos/rtos.c.bak   2014-09-14 17:28:10.000000000 +0100
+++ openocd-0.7.0/src/rtos/rtos.c       2014-09-14 17:29:39.000000000 +0100
@@ -454,12 +454,16 @@
        tmp_str_ptr = *hex_reg_list;
        new_stack_ptr = stack_ptr - stacking->stack_growth_direction *
                stacking->stack_registers_size;
+#if 0
+       // This code gives bad results for already aligned pointers
+       // and negative stack growth
        if (stacking->stack_alignment != 0) {
                /* Align new stack pointer to x byte boundary */
                new_stack_ptr =
                        (new_stack_ptr & (~((int64_t) stacking->stack_alignment - 1))) +
                        ((stacking->stack_growth_direction == -1) ? stacking->stack_alignment : 0);
        }
+#endif
        for (i = 0; i < stacking->num_output_registers; i++) {
                int j;
                for (j = 0; j < stacking->register_offsets[i].width_bits/8; j++) {

I reported the bug to the OpenOCD developers and proper fix will hopefully arrive in the trunk code soon: http://openocd.zylin.com/#/c/2301/ (an initial fix was committed but has since been reverted because it was broken for a different case).

Alternative (Debian/Ubuntu)

To do this you can download the code from http://sourceforge.net/projects/openocd/files/openocd/0.8.0/ and apply this patch http://openocd.zylin.com/#/c/2301/3/src/rtos/rtos.c,cm .

sudo apt-get install libusb-dev libusb-1.0-0-dev
./configure --enable-armjtagew
make 
sudo make install
./configure

OpenOCD configuration file

OpenOCD requires a set of configuration commands describing both the JTAG adapter and the target CPU/board. Copy this into a file called tilda.cfg:

# Recommended for OpenOCD-0.7
# script interface/olimex-arm-usb-tiny-h.cfg

# Recommended for OpenOCD-0.8
script interface/ftdi/olimex-arm-usb-tiny-h.cfg

# script for ATMEL sam3, a CORTEX-M3 chip
script target/at91sam3XXX.cfg

$_TARGETNAME configure -rtos FreeRTOS

Run OpenOCD

OpenOCD uses libusb to access the device and may need to run with sudo for it to work. If everything is connected together and the badge is powered on then you should see it detecting the CPU core as shown below:

$ sudo openocd -f tilda.cfg
Open On-Chip Debugger 0.8.0 (2014-09-17-20:52)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.sourceforge.net/doc/doxygen/bugs.html
Info : only one transport option; autoselect 'jtag'
adapter speed: 500 kHz
adapter_nsrst_delay: 100
jtag_ntrst_delay: 100
cortex_m reset_config sysresetreq
Info : clock speed 500 kHz
Info : JTAG tap: sam3.cpu tap/device found: 0x4ba00477 (mfg: 0x23b, part: 0xba00, ver: 0x4)
Info : sam3.cpu: hardware has 6 breakpoints, 4 watchpoints

CMSIS DAP

If you us an an CMSIS DAP compliant adapter such as the [Atmel ICE] then OpenOCD can be run with interface/cmsis-dap.cfg e.g.

$ sudo openocd -f interface/cmsis-dap.cfg -f target/at91sam3XXX.cfg
Open On-Chip Debugger 0.9.0-dev-00161-g9c47dc9-dirty (2014-09-30-23:42)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.sourceforge.net/doc/doxygen/bugs.html
Info : only one transport option; autoselect 'cmsis-dap'
adapter speed: 500 kHz
adapter_nsrst_delay: 100
cortex_m reset_config sysresetreq
Info : CMSIS-DAP: SWD  Supported
Info : CMSIS-DAP: JTAG Supported
Info : CMSIS-DAP: Interface Initialised (SWD)
Info : CMSIS-DAP: FW Version = 01.00.0021
Info : SWCLK/TCK = 1 SWDIO/TMS = 1 TDI = 1 TDO = 0 nTRST = 0 nRESET = 1
Info : DAP_SWJ Sequence (reset: 50+ '1' followed by 0)
Info : CMSIS-DAP: Interface ready
Info : clock speed 500 kHz
Info : IDCODE 0x2ba01477
Info : sam3.cpu: hardware has 6 breakpoints, 4 watchpoints

This seems to be a little slower at single-stepping the code than the Olimex adapter but is a little cheaper and in theory provides a more flexible debug interface (SWD).

Patch TiLDA firmware for OpenOCD FreeRTOS detection

The OpenOCD code knows how to find the FreeRTOS tasks and can use this to make each task running on the badge as a different thread in GDB. To make this work you need to apply a small patch to the TiLDA code.

diff --git a/EMF2014/TiLDATask.cpp b/EMF2014/TiLDATask.cpp
index 2cdc08d..701f940 100644
--- a/EMF2014/TiLDATask.cpp
+++ b/EMF2014/TiLDATask.cpp
@@ -58,6 +58,8 @@
 #include "logo.h"
 #include "TiLDA_64x128.h"
 
+// Hack to make FreeRTOS support work in OpenOCD
+unsigned portBASE_TYPE uxTopUsedPriority;
 
 TiLDATask::TiLDATask() {
 
@@ -68,6 +70,10 @@ String TiLDATask::getName() const {
 }
 
 void TiLDATask::task() {
+
+    // Hack to make FreeRTOS support work in OpenOCD
+    uxTopUsedPriority = configMAX_PRIORITIES - 1;
+
     Tilda::_realTimeClock = new RTC_clock(RC);
     Tilda::_appManager = new AppManager;

This adds a variable that the OpenOCD expects to be able to read from the target to determine how many queues are being used by the FreeRTOS scheduler. This was present in older FreeRTOS release but has since been removed.

Build and flash the badge firmware

To make the debugger work you must be sure that the firmware running on the badge exactly matches the source code and binary objects that you have locally on your development machine. I recommend you apply the patch from the previous section, build the TiLDA firmware and upload it to your badge before you go any further.

In the Arduino IDE you should build firmware (EMF2014 sketch) and upload it via the USB port on the TiLDA badge. (I have not attempted to upload firmware via the JTAG interface but that should be possible as well). The IDE writes a copy of the object files and the final executable to a directory in /tmp whenever you build the code. The firmware image file is EMF2014.cpp.elf and you want to find the most recently built one, e.g.

[jburgess@shark]$ ls -lt /tmp/build*/EMF2014.cpp.elf | head -n 1
-rwxrwxr-x. 1 jburgess jburgess 1475463 Sep 14 22:53 /tmp/build9091009552223609902.tmp/EMF2014.cpp.elf

Running GDB

First locate the EMF2014.cpp.elf file mentioned in the previous section. Run gdb with this file and with any luck it should be able to load it:

[jburgess@shark tmp]$ gdb /tmp/build9091009552223609902.tmp/EMF2014.cpp.elf
GNU gdb (GDB) Fedora 7.7.1-18.fc20
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /tmp/build9091009552223609902.tmp/EMF2014.cpp.elf...done.
(gdb)

Check debug symbols

Next try some simple commands to see whether GDB knows where symbols are located in the binary and source:

(gdb) info line main
Line 43 of "/home/jburgess/Documents/emf/Mk2-Firmware/hardware/emfcamp/sam/cores/rtos/main.cpp" starts at address 0x8fe78 <main()> and ends at 0x8fe7a <main()+2>.
(gdb) list main
38
39      /*
40       * \brief Main entry point of Arduino application
41       */
42      int main( void )
43      {
44              init();
45
46              initVariant();
47

If something goes wrong at this point you could try using the copy of GDB provided with the arduino tools instead, e.g. arduino-1.5.7/hardware/tools/gcc-arm-none-eabi-4.8.3-2014q1/bin/arm-none-eabi-gdb

Attach GDB to OpenOCD

The OpenOCD supports a couple of different interfaces. Before we try GDB we should first stop the CPU using the basic control interface and tell it to halt the CPU:

$ telnet localhost 4444
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Open On-Chip Debugger
> halt
target state: halted
target halted due to debug-request, current mode: Thread 
xPSR: 0x61000000 pc: 0x000870ba psp: 0x20074a30
>

To continue execution you can run "resume". Make sure you run "halt" to leave the target in the stopped before attaching GDB. Normally when GDB attaches it should stop the target automatically but this isn't working at the moment. If the target is running then you may when GDB attaches then you may see some odd behaviour.

The OpenOCD software runs a target server on localhost:3333 which we can ask GDB to attach to:

(gdb) target extended-remote localhost:3333
Remote debugging using localhost:3333
0x000870ba in prvCheckTasksWaitingTermination () at /home/jburgess/Documents/emf/Mk2-Firmware/hardware/emfcamp/sam/libraries/FreeRTOS_ARM/utility/tasks.c:2839
2839                    while( uxTasksDeleted > ( UBaseType_t ) 0U )
(gdb)

If you get "warning: Architecture rejected target-supplied description" try using "hardware/tools/gcc-arm-none-eabi-4.8.3-2014q1/bin/arm-none-eabi-gdb" from Arduino's folder instead of whatever gdb you're distro has installed.

List FreeRTOS tasks

To list the running tasks you can use "info threads".

(gdb) info threads
[New Thread 537348080]
[New Thread 537396632]
[New Thread 537402416]
[New Thread 537394216]
[New Thread 537405232]
[New Thread 537393096]
[New Thread 537348848]
[New Thread 537398824]
[New Thread 537400224]
[New Thread 537346984]
[New Thread 537403512]
[New Thread 537397728]
[New Thread 537395312]
  Id   Target Id         Frame 
  14   Thread 537395312 (MessageCh) 0x00086750 in vPortYield () at emfcamp/sam/libraries/FreeRTOS_ARM/utility/port.c:385
  13   Thread 537397728 (RadioTran) 0x00086750 in vPortYield () at emfcamp/sam/libraries/FreeRTOS_ARM/utility/port.c:385
  12   Thread 537403512 (IMUTask) 0x00086750 in vPortYield () at emfcamp/sam/libraries/FreeRTOS_ARM/utility/port.c:385
  11   Thread 537346984 (TiLDATask) 0x00086750 in vPortYield () at emfcamp/sam/libraries/FreeRTOS_ARM/utility/port.c:385
  10   Thread 537400224 (GUI) 0x00086750 in vPortYield () at emfcamp/sam/libraries/FreeRTOS_ARM/utility/port.c:385
  9    Thread 537398824 (LCD) 0x00086750 in vPortYield () at emfcamp/sam/libraries/FreeRTOS_ARM/utility/port.c:385
  8    Thread 537348848 (Tmr Svc) 0x00086750 in vPortYield () at emfcamp/sam/libraries/FreeRTOS_ARM/utility/port.c:385
  7    Thread 537393096 (RGBTask) 0x00086750 in vPortYield () at emfcamp/sam/libraries/FreeRTOS_ARM/utility/port.c:385
  6    Thread 537405232 (HomeScree) 0x00086750 in vPortYield () at emfcamp/sam/libraries/FreeRTOS_ARM/utility/port.c:385
  5    Thread 537394216 (BatterySa) 0x00086750 in vPortYield () at emfcamp/sam/libraries/FreeRTOS_ARM/utility/port.c:385
  4    Thread 537402416 (PMICTask) 0x00086750 in vPortYield () at emfcamp/sam/libraries/FreeRTOS_ARM/utility/port.c:385
  3    Thread 537396632 (RadioRece) 0x00086750 in vPortYield () at emfcamp/sam/libraries/FreeRTOS_ARM/utility/port.c:385
  2    Thread 537348080 (IDLE :  : Running) 0x000870ba in prvCheckTasksWaitingTermination ()
    at emfcamp/sam/libraries/FreeRTOS_ARM/utility/tasks.c:2839
* 1    Thread 537401320 (AppOpener) 0x000870ba in prvCheckTasksWaitingTermination ()
    at emfcamp/sam/libraries/FreeRTOS_ARM/utility/tasks.c:2839

Most threads should claim to be in "vPortYield()" which is one of the OS routines that a task will call when it wants to sleep for a while.Select a task and you can obtain its backtrace:

(gdb) thread 6
[Switching to thread 6 (Thread 537405232)]
#0  0x00086750 in vPortYield () at emfcamp/sam/libraries/FreeRTOS_ARM/utility/port.c:385
385             __asm volatile( "isb" );
(gdb) bt
#0  0x00086750 in vPortYield () at emfcamp/sam/libraries/FreeRTOS_ARM/utility/port.c:385
#1  0x00085c60 in xEventGroupWaitBits (xEventGroup=0x20082b78, uxBitsToWaitFor=1, xClearOnExit=0, xWaitForAllBits=0, xTicksToWait=<optimized out>)
    at emfcamp/sam/libraries/FreeRTOS_ARM/utility/event_groups.c:357
#2  0x0008453e in HomeScreenApp::task (this=0x20082710) at /tmp/build9091009552223609902.tmp/HomeScreenApp.cpp:157
#3  0x000818b4 in Task::taskCaller (this=0x20082710) at /tmp/build9091009552223609902.tmp/Task.cpp:78
#4  0x00081906 in Task::_task (self=<optimized out>) at /tmp/build9091009552223609902.tmp/Task.cpp:87
#5  0x00086768 in ulPortSetInterruptMask () at emfcamp/sam/libraries/FreeRTOS_ARM/utility/port.c:423
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Known issues

Normally when you hit a breakpoint GDB will automatically switch to the thread that hit it. This is not happening at the moment. If you interrupt the code or hit a breakpoint you may have to manually switch threads. The 'info threads' puts "Running" next to task being executed and this is probably the one you need to switch to.
- Fixed by applying this patch to OpenOCD http://openocd.zylin.com/#/c/2303/
When you attach and detach GDB doesn't automatically halt/resume the target. You can do this manually using the interface on port 3333 or just tell GDB to continue, then interrupt to stop it.
GDB hits an assert if you detach the target and asks whether you want a core dump.
- This appears to be: https://sourceware.org/bugzilla/show_bug.cgi?id=12228
I had a problem where the badge kept stopping at a breakpoint that I set in a previous GDB session and I was unable to fix it without power cycling the badge and restarting GDB & OpenOCD.