Debugging the ZombieVerter WebInterface

This post is going to be:

1. Super Nerdy
2. Only for people who REALLY like my deep dives
3. And even then only if you need to debug the particular Verhicle Control Unit I’m using in my Project sEVen conversion (ZombieVerter VCU)

I’ll probably do a video about all of this soon, because there’s way more to see than I can realistically add as images to this post. The link to the video will appear here if I get around to it.

But if you’re in the market for this kind of info then you’ll also be looking for scripts and config files that are less easy to use from a video, so I’ll include those below too.

And the story goes like this:

What Started All of This

So I decided my Project sEVen EV conversion needed a Vehicle Control Unit (VCU). I’d originally thought I’d be able to just use an AIM Technologies PDM32 to control the car. They’re pretty programmable and have screens, IMU, soft-fused power switching, etc etc. Then I realised I was going to need much more in the way of control than the simple graphical logic builder could deliver in the PDM32. There are people who will tell you differently, but I suspect they’re not software people!

I toyed around for a while thinking I might do a VCU with a Pi or an Arduino but they’re not really automotive grade and there’d be a lot of code needed.

The ZombieVerter had come across my radar before, but at the time (3 years ago) it had promise but I couldn’t see how I could use what they’d got at the time (though that’s almost certainly my lack of vision that made me decide that).

But I now needed a VCU and the ZombieVerter is based on an automotive grade STM32 processor and with a software framework that I could adapt to what I needed.

So I bought a ZombieVerter with all the cases and enclosures and put it aside while I concentrated on gearboxes.

But over the last month or so (Jan 2026) the gearbox had sort of stalled at the production stage and I could start looking at the electronics of the car.

After connecting up the VCU, various CANBuses, contactors and the AIM PDM it was mostly working. I could get all of the above to mostly talk to each other and to engage the contactors.

But… the ZombieVerter VCU comes with (at least I bought the option with one) a Wemos D1 Mini ESP8266 web interface board. The ESP8266 shows a web interface to control the VCU and talks to the VCU’s STM32 over a serial port. All well and good so far.

However, mine was proving to be unreliable. The ESP8266 sends commands to, and gets responses from, the STM32. And one of the main command/response activities is a request and response for a JSON object from the STM. The JSON contains all the parameters that you can set on the VCU along with all the spot values that the VCU is working with – voltages, currents, vehicle state etc etc.

But for some reason my setup was showing communications errors when trying to do this JSON grab. And this would manifest as a red banner at the top of the web interface an nothing showing for params or spot values.

I could work around this issue while I was bringing things up, but it got more and more annoying as I was trying to rely on the web interface to tell me what was going on.

And then it got too much and I decided to try and fix it.

I suspected a few things that might be going on. We’ll cover them in more detail later, or at least once I’ve spent a few more hours doing some further testing. The theories about what might be going on are:

1. Signal integrity on the STM32 to ESP8266 UART connection
2. Hardware FIFO overflow on the ESP8266 Rx
3. Software FIFO overflow on the ESP8266
4. Heap fragmentation on the ESP8266
5. Wifi drop-outs and retransmissions (this could probably be the cause of points 2, 3 and 4 above)

So…. after a few days detour, I’ve now got a debug environment set up for the STM32 and ESP8266. And I’m getting closer to a conclusion about what’s going on. There’ll be a section at the bottom of this post that will hopefully have that conclusion when I finally come to one.

But I thought a post here would be useful to show what I’ve done to get a debug environment up and running and so I can come back to what I’ve done here in the future, or else I’ll forget.

It must be said though… this process of debugging two different processor families talking to each other, is not a trivial project. I’ve been using Visual Studio Code for this sort of stuff for 7 or 8 years, debugging all sorts of projects, including remote real-time applications. And it still took me many hours to get all this hooked up. But it can be done, and here’s how I did it…

The Setup

 For the moment, I’m just trying to figure out the Red Banner problem. This setup may (and probably will) get used for all sorts of other future bits of the project but for the moment it’s just the Red Banner.

So lets go through each of the components, how they’re set up and how I use them.

ZombieVerter Connections

Here’s now I’ve got the ZombieVerter hooked up in my test rig at the moment…

And here’s the signal connections to the ST-LINK programmer/debugger.

You can see in these two pictures that I’ve cobbled together a perspex top for the enclosure… to stop me dropping nuts, bolts, screwdrivers and spanner onto the running board! 🙂

The ZombieVerter wiki entry talks about a red wire but for these cheap ST-LINK clone adapters there’s no need. If you’re using an official ST ST-LINK v2 (white puck) then you’ll need to also connect its VTref to a 3v3 somewhere on the ZombieVerter board. This ST adapter needs the VTref so it can decide what voltage levels to set its outputs to (5V, 3v3 etc).

 A bit of a side note: I spent an hour or two trying to get the NRST signal working from the genuine ST ST-LINK but connecting to the chip side of the Zombie’s reset capacitor. In theory it should work but no matter what incantation of launch.json I tried I couldn’t get the hard reset to work. I was hoping that would be a better way of entering and releasing debug mode. But… I couldn’t get it to work. And with all the connecting and disconnecting the ST-LINK, I think I blew it up. At least it doesn’t work anymore. The LED comes on but it doesn’t blink to say it’s connecting to the PC. I think I may have got the clock and data lines reversed, not sure if that’s what did it, but more care needed in the future.

The Logic Analyser is simply connected to the Rx, Tx and GND.

All of the Analyser, ST-LINK and ESP8266 Wemos boards are connected to my Mac over a USB cable (I use a 4-port USB hub near the test rig and run a 3m cable across my office to the Mac).

Visual Studio Code setup for ZombieVerter STM32f107 Debugging and Programming

 First of all, it’s, I think, obvious, but I’m going to say it anyway. Debugging the STM32 on a ZombieVerter should be considered VERY VERY carefully. You should absolutely not be suspending the STM32 execution in any OpMode other than “off”… unless you know what you’re doing! At best you’re going to be simulating what your system does when the STM32 crashes, at worst you’re going to do a lot of very expensive damage to things like inverters. 

With that out of the way, this is what you’ll need to get Visual Studio Code debugging and programming a ZombieVerter:

•  Visual Studio Code – duh
• OpenOCD
• Cortex-Debug Plugin
• Arm Toolchain
• Launch.json
• Tasks.json
• esptool.py
• mkspiffs
• arduino-cli
• The ZombieVerter code

This isn’t a full run through of what you’ll need to do, but if you’re into writing code for the ZombieVerter then you should know the steps to fill in here:

Visual Studio Code

There are loads of videos about how to get going on VSC, so I’m not going to cover the general setup here. But I will cover the STM32 and ESP8266 specific stuff later. There’s also a lot of code formatting and viewer plugins you’ll probably want, but VSC is pretty good at suggesting these as you open files up as and when you do so.

OpenOCD

OpenOCD is the Open On Chip Debugger. It includes the GDB install and allows you to use the GDB backend on the STM32 processor to pause, breakpoint and inspect your paused code.

brew install open-ocd

Note how this is open-ocd, not openocd… the cask name seems to have changed.

Cortex-Debug Plugin

This plugin connects the OpenOCD command line tools to the Visual Studio Code debugger environment.

Install the cortex-debug plugin from Marus25.

Arm Toolchain

The ARM toolchain is what allows you to compile ARM64 code.

In theory you should be able to use the arm toolchain from either Homebrew or the default ARM toolchain. In the end there were some bits that seemed to be missing from the Homebrew install so I just went with the ARM default toolchain that installs into /Applications.

So I installed the MacOS (Apple Silicon) 15.2rel1 toolchain from:

https://developer.arm.com/downloads/-/arm-gnu-toolchain-downloads

 ZombieVerter Code

Well then, if we’re going to be debugging code we better have it installed. The code comes in two chunks, the STM32 code and the ESP8266 code. Do this next bit wherever you want to editing you code…

They can be installed with:

git clone --recursive https://github.com/damienmaguire/Stm32-vcu.git

And the webserver code you’ll need for the ESP8266 is: 

git clone https://github.com/jsphuebner/esp8266-web-interface.git

That will create two folders containing the two code sets.

What I then do is open Visual Studio code and open each folder, one in each of the two VSS windows (i.e. open VSS then do File -> New Window and in each window open one of the two folders from the git clone operations). That way you get two launch.json and task.json setups (see below) and that makes things easier to remember what’s what, IMHO.

tasks.json

Now you’ve got some of the basic code installed you’ll need the magic to bring it all together. The first of these is a tasks.json that you use to build the ZombieVerter Stm32-vcu code. Once you’ve got this installed you don’t actually need to do a Shift-Cmd-P -> Task Run. You can do all of that automagically from the launch.json.

This is the tasks.json needed for the STM32 code window. You can either create this manually or get VSS to create it, the important thing is that this file (and launch.json) live in a sub-folder called .vscode.

{
    "version": "2.0.0",
    "tasks": [
        {
            "label": "Build stm32 VCU",
            "type": "shell",
            "command": "make",
            "options": {
                "cwd": "${workspaceRoot}",
                "env": {
                    "PATH": "/Applications/ArmGNUToolchain/15.2.rel1/arm-none-eabi/bin:$PATH"
                }
            },
            "group": "build",
            "presentation": {
                "echo": true,
                "reveal": "always",
                "focus": false,
                "panel": "shared"
            },
            "args": [
                "all"
            ],
            // Use the standard less compilation problem matcher.
            "problemMatcher": "$gcc",
        },
        {
            "label": "Rebuild stm32 VCU",
            "type": "shell",
            "command": "make",
            "options": {
                "cwd": "${workspaceRoot}",
                "env": {
                    "PATH": "/Applications/ArmGNUToolchain/15.2.rel1/arm-none-eabi/bin:$PATH"
                }
            },
            "group": {
                "kind": "build",
                "isDefault": true
            },
            "presentation": {
                "echo": true,
                "reveal": "always",
                "focus": false,
                "panel": "shared"
            },
            "args": [
                "clean",
                "all"
            ],
            // Use the standard less compilation problem matcher.
            "problemMatcher": "$gcc",
        },
        {
            "label": "Clean stm32 VCU",
            "type": "shell",
            "command": "make",
            "options": {
                "cwd": "${workspaceRoot}",
                "env": {
                    "PATH": "/Applications/ArmGNUToolchain/15.2.rel1/arm-none-eabi/bin:$PATH"
                }
            },
            "group": "build",
            "presentation": {
                "echo": true,
                "reveal": "always",
                "focus": false,
                "panel": "shared"
            },
            "args": [
                "clean"
            ],
            // Use the standard less compilation problem matcher.
            "problemMatcher": "$gcc",
        }
    ]
}

 And here’s the launch.json that starts/stops a debugging session.

Note: I also choose to build the code every time… just to make sure the symbols are up to date…. I hate chasing my tail with stale symbols!@

Launch.json

{
	"version": "0.2.0",
	"configurations": [
		{
			"name": "Launch (Build, Flash and Attach)",
			"type": "cortex-debug",
			"request": "launch",
			"servertype": "openocd",
			"cwd": "${workspaceFolder}",
			"executable": "${workspaceFolder}/stm32_vcu",
			"device": "STM32F107VC",
			"configFiles": [
				"interface/stlink.cfg",
				"target/stm32f1x.cfg"
			],
			"svdFile": "${workspaceFolder}/STM32F107xx.svd",
			"preLaunchTask": "Build stm32 VCU",
			"openOCDLaunchCommands": [
				"adapter speed 1000"
			]
		},
		{
			"name": "Attach - make sure elf matches target",
			"type": "cortex-debug",
			"request": "attach",
			"servertype": "openocd",
			"cwd": "${workspaceFolder}",
			"executable": "${workspaceFolder}/stm32_vcu",
			"device": "STM32F107VC",
			"configFiles": [
				"interface/stlink.cfg",
				"target/stm32f1x.cfg"
			],
			"svdFile": "${workspaceFolder}/STM32F107xx.svd",
			// "preLaunchTask": "Build stm32 VCU",
			"postStartSessionCommands": [
 				 "continue"
			],
			"openOCDLaunchCommands": [
				"adapter speed 1000"
			]
		}
	]
}

A couple of points here. I probably need to spend more time on this, but the Attach only options were struggling without doing a preLaunch build. I need to come back to that.

You’ll also need another file here, the STM32F107xx.svd. This is downloaded from a generic repository and once that’s unzipped I copied the .svd file into the workspace folder.

You can also play with the ST-LINK adapter speed settings. I’ve had good success at up to 4000 but a more conservative 1000 seems to do everything I need.

The two config files come from the openocd install and are relative to that install location, so as long as you’ve got openocd installed then the files should be found.

Ok, so now I’ve got a setup that can build, run and debug the STM32 code. Breakpoints and code inspection once paused worked well from inside VSC.

What about the ESP8266 code?

ESP8266 Code Setup

 While I could build and debug the STM32 code from within VSS, I haven’t got quite so far with the ESP8266. So at the moment I just edit the code in VSC and do the building and reflashing from the command line.

This could all be done from the Arduino IDE, but I prefer VSC AND… I prefer VSC 🙂

First though we need the Arduino command line tools and some way to flash the device:

brew install arduino-tools

and also 

brew install esptool

My Arduino sketch build, doing the ESP8266 code is:

arduino-cli compile -v --fqbn esp8266:esp8266:d1_mini esp8266-web-interface.ino --build-path ./build

• Pause the STM32 from within the STM32 debugger (obviously having run the code to start with)
• From the VSC debug console:
  • source tools/float_uart3.gdb
  • flash the ESP8266 board
  • source tools/restore_uart3.gdb
• Restart the STM32

To flash the ESP8266 from the command line I use:

curl -F 'data=@/data/ui.js' http://10.0.1.20/edit

git clone --recursive https://github.com/igrr/mkspiffs.git

make

and then we can build the filesystem

path-to-mkspiffs-build-dir/mkspiffs -c data -b 4096 -p 256 -s 0x100000 spiffs.bin

That gets us a filesystem in a .bin, now we can load it into the right place on the device:

ESP8266 Debugging

Now we can change code on the ESP8266, next up was to add some logging to the ESP8266.

Because the ESP8266 Wemos D1 Mini board share their USB and breakout pin serial ports, there’s no way of getting any Serial printf debugging out of the board while the ESP8266 is talking to the STM32. My solution was to add some simple UDP logging into the ESP8266 code…

#include <WiFiUdp.h>

WiFiUDP logUdp;
IPAddress logIP(10,0,1,6);   // <-- your Mac IP
const uint16_t logPort = 5514;

void logf(const char* fmt, ...) {
  static char buf[256];
  va_list ap;
  va_start(ap, fmt);
  int n = vsnprintf(buf, sizeof(buf), fmt, ap);
  va_end(ap);
  if (n <= 0) return;

  logUdp.beginPacket(logIP, logPort);
  logUdp.write((const uint8_t*)buf, (size_t)min(n, (int)sizeof(buf)-1));
  logUdp.endPacket();
}

Testing Signals on the Wire

Ok, now I’ve got a framework for debugging, I wanted to check that the JSON coming into the ESP8266 was valid.

The JSON code on the ESP8266 is structured to use a char* buffer to accept packets of up to 256 bytes in from the UART. It then appends them to a C++ String object.

The problem with that on the ESP8266 is that this sort of object is a bit of a heap monster. The JSON objects getting thrown around here are about 30kB. And the ESP8266 has only a small amount of RAM for stack and heap usage. While the C++ String control block is created on the Stack, behind the scenes the ESP8266 Arduino runtime uses malloc and free to implement the String storage. And so when every 256 bytes come into the sketch code it needs to malloc a new chunk of contiguous memory and then free the old one. But with all the other stuff going on on the chip (Wifi and Web Server) the heap is going to be pretty scrappy and may not have space for a 30kB chunk. Heap Fragmentation.

So I did some code to check if the String object did or did not have valid JSON in it (it was possible that the corruption was happening further on in the transmission over the Wifi, so I wanted to see if the JSON coming into the Arduino sketch was ok). It wasn’t.

The code below does a simple brace matching algorithm and I could output whether each JSON message was valid into my UDP logger. Whenever I saw the Web Browder fail to load JSON (Red Banner) I also saw my brace matcher say there was corrupted JSON.

So, my problem is getting JSON into the Arduino sketch.

bool looksLikeValidJsonStructure(const String& s) {
  int depth = 0;
  bool inString = false;
  bool escape = false;
  bool started = false;

  for (size_t i = 0; i < s.length(); i++) {
    char c = s[i];

    if (!started) {
      if (c == '{' || c == '[') {
        started = true;
        depth = 1;
      } else if (c > ' ') {
        return false; // non-whitespace before start
      }
      continue;
    }

    if (escape) { escape = false; continue; }

    if (inString) {
      if (c == '\\') { escape = true; continue; }
      if (c == '"') inString = false;
      continue;
    } else {
      if (c == '"') { inString = true; continue; }
      if (c == '{' || c == '[') depth++;
      if (c == '}' || c == ']') {
        depth--;
        if (depth == 0) {
          // after closing root, only whitespace allowed
          for (size_t j = i + 1; j < s.length(); j++) {
            if (s[j] > ' ') return false;
          }
          return true;
        }
        if (depth < 0) return false;
      }
    }
  }

  return false; // never closed root
}

And here’s some sample logging, isJson=0 showing the brace matching failing.

On the Wire Testing

Right, so the JSON coming into the Arduino sketch is corrupted. Note that I’m not saying it’s corrupted coming into the ESP8266 itself, we’ll get to that in a bit.

So next up I wanted to check if the STM32 was sending valid JSON messages… and that meant breaking out the logic analyser.

With my Salaea hookup up to the UART Rx and Tx I could easily grab the 500kbps traffic on the wire. The Salaea Logic 2 software has protocol disectors for a bunch of protocols (and of course serial is about as simple as they get) so it was easy to decode the serial 1’s and 0’s into hex and ascii. I could then export the hex to a .csv file and some simple Python to turn that back into a text file containing all the JSON messages.

Eagle eyed amongst you will see that in the image above, the STM32 echo of the json command from the ESP8266 is missing the first character – it should say “json” but just says “son”. So I’ll need to go and check why that echo is munching the first echoed character. Problem for future John!

Here’s the simple Python to take the Salaea CSV file and convert it to a flat text file.

#!/usr/bin/env python3

import sys
import csv
from pathlib import Path

def main():
    if len(sys.argv) != 2:
        print(f"Usage: {sys.argv[0]} <input.csv>")
        sys.exit(1)

    csv_path = Path(sys.argv[1])
    out_path = csv_path.with_suffix("").with_name(csv_path.stem + "_columnB.txt")

    chars = []

    with csv_path.open(newline='', encoding='utf-8') as f:
        reader = csv.reader(f)
        next(reader, None)  # skip header row

        for row_num, row in enumerate(reader, start=2):
            if len(row) < 2:
                continue

            value = row[1].strip()

            if value.startswith("0x"):
                try:
                    byte = int(value, 16)

                    if byte == 0x0D:
                        chars.append("\r")
                    elif byte == 0x0A:
                        chars.append("\n")
                    elif 0x20 <= byte <= 0x7E:
                        chars.append(chr(byte))
                    else:
                        chars.append("?")

                except ValueError:
                    print(f"Warning: invalid hex value '{value}' on row {row_num}")
                    chars.append("?")

    with out_path.open("w", encoding="utf-8", newline="") as f:
        f.write("".join(chars))

    print(f"Wrote {len(chars)} characters to {out_path}")

if __name__ == "__main__":
    main()

All of which showed no corruption of the JSON on the wire. All the JSON messages from the logic analyser captured UART traffic were whole and valid. No corruption there.

So at the same time that I could see the UDP logging was indicating corruption there, the data on the wire was ok – I tested a few hundred JSON messages and all were good. So at least that rules out any issues on the PCB or indeed with the transmission out of the STM32.

Transmission Speeds 

One other side show was to see if the speed of the UART comms was causing the problem – I suspect it is, but which bit of the chain of events does that relate to. Just lowering the data rate might solve the problem, and it sort of does, but why… and how can I get the transmission back up to where it should be if I don’t get to the root of the problem.

So I changed the ESP8266 and STM32 code to allow me to send an altered “fastuart” command (as well as the JSON messages the protocol between the browser client, ESP8266 and STM32 allows for other data and commands).

The new fastuart command allows me to send a specific baudrate to both the ESP8266 and the STM32, synching their uart speeds and allowing me to test different rates… the original code set allowed a couple of speeds to be selected, but I wanted more granularity.

And slowing things down did work. To a point. There were still problems even at slower rates. And with those slow rates it took ages for the web interface to update. Transferring 30kB at 115200 takes over 2 seconds, so my interface isn’t showing VCU mode changes or voltages/current very quickly after they change… and in fact at those rates I miss a lot of what’s going on with the VCU.

Where does that leave me?

Well, I suspect the problem relates to the ESP8266 hardware fifos. The chip is a pretty low powered device to be running a webserver and wifi stack. There are alls sorts of tasks that the web server and wifi could be doing that would hold off the rest of the processor from doing other things.

And the ESP8266 input fifo’s are only 128 bytes. Which at 500,000bps is around 2.5ms before the 128 bytes fill up. So if the web server or wifi hold off other tasks for longer than that then data will be lost on the UART RX. 

This could have been mitigated by hardware flow control (RTS/CTS) but the STM32 on the ZombieVerter uses those hardware pins for other things, and its just not going to get fixed or even jury rigged to test out, any time soon.

Other options are to try and port the webserver Arduino code to something with a bit more grunt. Say an ESP32-C or S. My money is on doing it with an S. Which has dual cores and the standard web server implementation works on the second core, leaving the first core to worry about servicing the UART. But that’s a lot of work that I wasn’t planning to do.

Why’s this affecting me? I suspect my Wifi is rather clogged and either the ESP8266 is not picking a good channel to work on (I tried this all out with both the inbuilt access point and connecting to my office network SSID) or retransmissions on the wifi network are causing processor starvation for UART servicing. I certainly see retransmission requests with Wireshark when I look at the traffic coming into the client browser. Maybe a different Wifi stack (ESP32-S) might be better at handling that sort of problem.

And that’s all for this ramble. Hopefully I’ll figure out what’s going on and include it here, or at least put a link to a new post here instead.

J