Our subscriber came online without SmartOLT.
At 19:20 on a Tuesday in Witeithie, Kiambu, a PPPoE session authenticated for an ONU provisioned entirely by software the ISP owns: a Huawei MA5683T driven over SSH by FyberPay's own driver, an HG8546M driven over TR-069 by our own GenieACS. No tunnel to somebody else's cloud. No SmartOLT login anywhere in the chain.
Getting there took apart three layers of assumption: an address that looked like a typo and wasn't, a simulator we had shipped 143 green specs against that turned out to be fiction, and the ONU itself, which refused to be configured the way the standard says it can.
I have spent six months as lead engineer on FyberPay, a multi-tenant billing and network platform for ISPs. Our native OLT integration was built, deployed, covered by 143 passing specs, and had never spoken to a real OLT. The brief: point it at a live MA5683T, provision a real subscriber, delete SmartOLT. This is every hoop between "credentials from the networking guru" and "PPPoE session up".
The dependency worth deleting
236 ONUs, two OLTs, and every one of them managed through a subscription: the ISP's router dials an OpenVPN tunnel out to SmartOLT's cloud, and SmartOLT speaks to the OLT from there.
Read that from the subscriber's side: a paying customer's internet transits somebody else's SLA. Tunnel drops, the ISP cannot manage its own network. Subscription lapses, 236 households are one expiry away from unmanageable.
The address that looked like a typo
The credentials arrived: OLT at 172.168.x.x, management VLAN 1000, SSH user smartolt; an "eth port" at 10.x.x.x with admin credentials.
One problem: 172.168.x.x is not private space. RFC1918 stops at 172.31.255.255, and whois puts the block with RIPE NCC in the Netherlands. From our VPS it answered ping at 110ms, ports closed: a live stranger on the public internet.
The theory wrote itself: a typo for 172.16.x.x. What we did with the theory was the safest thing we did all week: stopped at a TCP connect, sent no credentials. Anyone who had run ssh smartolt@172.168.x.x from that server would have handed the OLT's password to RIPE space.
The router settled it. A read-only look at the ARP table of Mikrotik-witeithie, the WireGuard peer:
address: 172.168.x.x mac: 04:F9:38:xx:xx:xx interface: vlan1000 complete: true
▲
04:F9:38 is a Huawei OUI. The OLT really lives here.Duara simply uses public-looking space internally. The "eth port" was the dead one: incomplete ARP, 100% loss from the router itself.
Two rules left that afternoon and became permanent: trust the device's tables over any addressing theory, and never send credentials anywhere until you know what owns the socket.
The router nobody may touch by hand
The OLT sits behind a FyberPay-managed MikroTik whose config is rewritten by routes-reconciler.sh on a 5-second loop. Add the OLT to a peer's allowed-ips by hand and it is silently reverted before you finish typing the next command. That is not a bug; it is the reason nobody's improvisation survives to production.
The tempting shortcut was worse. Every NasService carries VLAN, gateway and DHCP pool, and the reconciler applies them. Creating one for the OLT's subnet would have built a bridge and a DHCP pool on Duara's live management network, as a side effect of a form submission.
So the platform learned a new concept: a management route, a CIDR into a device's tunnel that provisions nothing on the device. Access follows a discipline: route one /32, re-arm it every second, tear it down on exit, verify allowed-ips is back to its original three entries. A killed script self-heals in 5 seconds, because the reconciler is the safety net.
The near-miss that wrote a rule: the GenieACS entrypoint does not parse the routes payload as JSON. It regex-scrapes every CIDR-shaped substring and runs ip route replace on each. Name a route "OLT mgmt (was 192.168.88.0/24)" and you have routed MikroTik's factory LAN inside the ACS container, silently unreachable'd every CPE on that range, and logged success.
Three bugs the simulator hid
The first live session connected and read nothing. Zero ONUs, zero PON ports, zero boards.
All three causes shared one root: the bundled simulator emitted a format no Huawei OLT produces, and the code matched the fiction. 143 green specs proved only that the code matched the fiction.
The transport outran the device. The driver wrote the next command the instant a prompt appeared. A real MA5683T drops characters under that:
sent: display ont info summary 0/0
echo: display ontinfosummary0/0
% Unknown command, the error locates at '^'Same command, only the pause varying: 30ms and 150ms mangled it, 600ms and 2000ms arrived intact. Reads now complete only when the prompt is visible and the stream has been quiet 500ms.
The hardcoded slot was empty. The service assumed slotIds = [2]. The chassis: boards in slots 0 and 1, and display board 0/2 answered Failure: Board does not exist. The deployed sync would have run clean forever and persisted zero ONUs. Slot discovery now comes from the chassis.
Every parser returned zero rows. listPonPorts expected 0/1/0 GPON; the device prints 0 GPON. The summary parser expected one combined row; the device prints two tables keyed by ONT ID. The fix was not patched parsers. It was captured device output, committed as fixtures, and a simulator rebuilt to match the hardware, so divergence is loud instead of silent.
The error shapes rode along. Huawei rejects with Failure: ..., not %: a rejected deactivate once returned applied: true. And Failure: The service virtual port already exists is the device agreeing. Rejection and agreement now have different names in the code.
The same capture pass killed a guess that mattered: the ONU runs line profile 8 and service profile 2, not the 10/10 a dry-run had rendered. Profile IDs are dropdowns fed from a captured catalog now, not free-typed numbers.
OMCI says no, TR-069 says yes
Provisioning the subscriber's HG8546M came down to one question asked twice: which layers accept writes?
The OLT path owns one write, verified on this hardware: the per-SSID OMCI VLAN binding, inside interface gpon mode:
ont port native-vlan <pon> <ont> wifi <1-4> vlan <vid> priority <0-7>That joins the radios to their service VLANs: management WAN on 1001, PPPoE WAN on 1002. Everything else is off-limits by policy; guessing grammar is what produced the parser bugs above.
The ONU refused the rest. The PPPoE username and password could not be written from the OLT side; the ONU blocked those OMCI writes. We did not fight it. No retry storms against a device that has said no.
The turn: the ONU had registered with GenieACS over TR-069, and the standard parameter tree accepted setParameterValues where OMCI had refused. We set the PPPoE credentials and the SSID details through the ACS, waited out the device's apply, and the session came up.
The refusal was the architecture announcing itself. Each plane owns what it can actually write, and the feature encodes it: SSID-to-VLAN joins belong to the OLT, WAN and Wi-Fi credentials belong to TR-069, everything else in OLT grammar belongs to nobody. Where verification is impossible (no OLT read-back for Wi-Fi VLAN bindings exists), the run records unavailable instead of claiming success.
Feature, not fluke
The first provisioning worked because an operator held the wheel. The feature removes the operator.
A run is durable state with an explicit phase machine, and a partial unique index admits exactly one active run per CPE, so two operators clicking configure cannot interleave writes to one radio. The device's cooperation is bought, not assumed: the TR-069 inform is shortened to 60 seconds during the run and restored to the 300 seconds we observed before touching it. A callback timeout earns exactly one reset, then a stop: a second reset against a half-provisioned ONU is how one support call becomes a truck roll. An ambiguous OLT write classifies as ambiguous, not failed, and ends as manual_action_required: layering ACS writes over an unknown state is the one thing we never do.
What it cost, honestly
Trusting the router's tables over the credential sheet. Cost: a read-only probe and a paused hand. Bought: no credentials sent to RIPE space.
A management route concept. Cost: a new route type the reconciler must ignore. Bought: reachability nobody can accidentally break.
Capture-verbatim fixtures. Cost: rewriting the simulator against real output. Bought: 143 specs that mean something.
A 500ms settle gate. Cost: half a second per read. Bought: commands arrive intact on real hardware.
Two control planes instead of one. Cost: an architecture to reason about. Bought: every write goes to a plane that accepts it.
The pattern behind all five: when a device refuses a write, believe it, and route around the refusal, not through it.
Our subscriber never saw any of this. The fibre went live, the session authenticated, the Wi-Fi came up with the credentials the portal showed. That is the point.
Where to go from here
Pointing homegrown tooling at a real OLT? Capture the device's output verbatim before you write a single parser, and feed the captures to the simulator. One afternoon of work; the difference between specs that mean something and specs that mean nothing.
The full FyberPay case study is at kiragu.alkenacode.dev.
I take on fixed-scope ISP platform builds, OLT and TR-069 integrations, and payment engine work. Fastest reply is WhatsApp: +254 714 313 598. Email kiragu@alkenacode.dev, or pick a fixed-price engagement on Contra.