Fixing Faults in EV Charger Firmware

Why firmware faults are hard to pin down

A charger sits between four things that all behave slightly differently from their specifications: the vehicle, the supply, the backend and the human using it. A fault that appears in firmware is often the firmware behaving exactly as written while one of those four does something the author did not anticipate.

• The vehicle negotiates in a way the control pilot state machine did not expect, or takes longer than the timeout allows
• The supply sags, flickers or loses a phase in a pattern the protection logic reads as a different fault
• The backend sends a message sequence that is legal but unusual, or stops responding mid-transaction
• The user unplugs, replugs, badges twice or walks away at a moment nothing was written to handle

This is why the first question is never what the code says. It is what actually happened, in what order, and whether it can be made to happen again.

What we need before anything can be diagnosed

A fault report of the form the charger stopped working is not actionable. These are what turn a report into a diagnosis, roughly in order of how much they help.

1. 1

   The OCPP trace

   The message exchange between charger and backend around the failure, with timestamps. This alone resolves a large share of reported faults, because it shows whether the charger did what it was told and whether it was told anything sensible.

2. 2

   Device logs

   Whatever the unit recorded locally. Firmware that logs nothing is firmware that cannot be debugged remotely, and that is itself a finding.

3. 3

   The conditions

   Vehicle make and model, supply arrangement, whether the site was busy, whether it had rained. Faults that only appear on one vehicle or one phase configuration are common and the pattern is the diagnosis.

4. 4

   Reproduction steps

   If it can be made to happen on demand, the work is a day. If it happens once a fortnight on one site, it is a monitoring exercise first.

5. 5

   Firmware version and history

   Which build, and what changed in the build before it. A fault that started at a known release has a much smaller search space.

How the work runs

1. 1

   Reproduce

   On a bench with a charger simulator where possible, so the fault can be triggered repeatedly without a vehicle and without a site visit. Faults that cannot be reproduced are attacked by instrumenting the field units instead.

2. 2

   Isolate

   Establish which layer actually owns the behaviour: application logic, protocol stack, hardware abstraction, or something outside the charger entirely.

3. 3

   Fix, or advise against fixing

   Sometimes the correct answer is that firmware is the wrong place. A supply problem patched in firmware becomes a second fault later, in a form that is never connected to the first.

4. 4

   Regression test

   Against the message set and the vehicle behaviours that already work, because the risk in a fleet-wide firmware release is never the fault you fixed.

5. 5

   Stage the release

   A pilot group before the fleet, with a defined rollback. Anything else is testing in production on a customer's revenue.

Working on firmware we did not write

A large share of this work is on codebases inherited from a supplier who has moved on, or from a team that no longer exists. That is a normal situation rather than an awkward one, and it changes the approach rather than preventing it.

• With full source, the work proceeds normally, with an orientation period to map the architecture before anything is changed
• With source but no build environment, reconstructing a reproducible build is the first deliverable and is worth having regardless of the fault
• With binaries only, behaviour can still be characterised from the outside, and the honest recommendation is often to replace rather than patch

We will tell you which of those you are in early, because the third one is a different conversation about cost and it should not arrive as a surprise three weeks in.

What cannot be fixed in firmware

This is the part buyers least want to hear and the part that saves the most money, so it is worth being direct.

• Insufficient supply capacity. Firmware can manage it gracefully; it cannot create headroom
• Bad earthing or termination. A protective function correctly detecting a real installation fault is not a firmware bug
• Component selection that was wrong for the duty. Firmware can derate around it, which shortens the product rather than saving it
• Internal cable routing and quality, which produce failures months after the unit passed every test
• A vehicle behaving out of specification. Workarounds are possible and each one is a permanent liability in the codebase

Each of these can be worked around in software. Every workaround is code that stays forever, that the next engineer will not understand, and that will interact with something in three years. We will say when we think that trade is a bad one.

Releasing a fix to a fleet

Diagnosing a fault is usually the smaller half. Getting the fix onto units that are already in car parks, basements and forecourts is where programmes come unstuck.

If firmware cannot be updated over the air, reliably, with a rollback, then the fault is not really fixed. It is fixed on units built after today. Everything already installed keeps the fault until somebody drives to it. Where that is the situation, restoring a working update path is the first piece of work and it is worth more than the fix that prompted it.