About the Role
We're looking for a senior engineer to own the quality, evaluation, and reliability of a production voice AI system used by real patients. This role is about measuring whether our voice assistant works well enough to deploy safely — and having the authority to say no when the evidence doesn't support a release.
You won't be building a speech stack from scratch. We use a third-party voice platform for telephony, speech recognition, synthesis, endpointing, and interruption handling. Your job is to build the evaluation and reliability layer around it.
What you'll own
You will own the complete evaluation and reliability function for our production voice assistant. This includes:
Measuring end-to-end conversational latency across all components, from the caller through backend round trips and back.
Evaluating endpointing, silence handling, and turn-taking behavior in production conditions.
Testing barge-in, interruptions, caller corrections, hesitation responses, and refusal handling.
Measuring speech-recognition quality (WER/CER and business-critical entity recognition) under real PSTN and VoIP conditions.
Building and maintaining automated conversation regression tests that catch regressions before release.
Creating realistic conversation scenario sets based on production calls and known failure modes.
Defining release thresholds and go/no-go criteria for each deployment.
Investigating failures across the voice platform, backend, prompts, and integrations.
Turning production incidents into permanent, reproducible regression tests.
The core question you will answer is: Did this change make the voice experience better, worse, or unsafe to release?
What we're looking for
Production voice experience. You have operated a phone or voice assistant used by real users. You can explain what its end-to-end latency was, how you measured it, which percentiles mattered, and where latency accumulated across the system.
Voice tuning in production. You have tuned endpointing, silence thresholds, barge-in, interruption handling, long pauses, false interruptions, and caller self-corrections. Experience with a closed or vendor-controlled voice platform is especially valuable.
Telephone ASR expertise. You understand how speech recognition degrades through PSTN or VoIP compared to high-quality microphones, and you have measured it. You're comfortable evaluating both WER/CER and business-critical recognition errors (names, dates, numbers, entities).
Evaluation systems. This is the centre of the role. You have built or owned a systematic way to test conversational systems: conversation replay, scenario-based test suites, simulation, golden conversation sets, automated regression testing, or human review combined with automated scoring. We care less about the framework and more about whether it reliably compared one release against another.
Strong Python engineering. You can build evaluation harnesses, API integrations, data pipelines, and automated tests. You're comfortable with the tools you'll likely encounter: pytest, Pydantic, Pandas/Polars, NumPy, jiwer, FFmpeg/SoX/librosa, PostgreSQL, S3-compatible storage, OpenTelemetry, Grafana/Metabase, GitHub Actions/GitLab CI, Twilio, and HTTP APIs. You don't need experience with every tool — we care about whether you understand the underlying problems and know how to build reliable evidence.
Nice to have: SIP, WebRTC, Asterisk, or Twilio experience; LLM or conversational-agent evaluation; experience with sensitive or regulated data; healthcare or safety-sensitive domains; German-language voice systems.
You do not need to speak German, build streaming audio infrastructure, build STT/TTS systems, build a turn-taking engine, train models from scratch, or know PHP or our existing backend. Another team owns the backend endpoints; your focus is the quality of the complete conversation.
What success looks like
Every important conversation change is tested before release. Production failures become reproducible regression tests. Voice latency is measured objectively. ASR quality is tested under realistic phone conditions. Interruption and endpointing changes are compared quantitatively. Every release has explicit quality thresholds. Most importantly, the company has someone who can independently answer: "Is this version good enough to put in front of patients?"