About the Role
Senior / Staff level
We’re building patient-facing AI across multiple channels: voice first, then web chat and whatever comes next.
The goal is not to build a separate assistant for every channel. It is to build one internal AI platform and API that every patient-facing experience can call, so the same rules, tools, models, safety boundaries, and business logic apply everywhere.
Alongside that, we need to decide how much of our AI infrastructure we should operate ourselves.
Today, speech recognition and language models are largely vendor-hosted. Over time, we may move parts of that stack to open-weight models running on infrastructure we control. That could give us better control over patient data, model versions, cost, and reliability.
But we have not decided that self-hosting is automatically the right answer.
Your first responsibility will be to produce the evidence: benchmark open models on our actual workloads, understand their accuracy and latency, calculate the real infrastructure cost, and tell us whether moving away from vendors makes sense.
If the numbers say we should keep paying a vendor for another year, we expect you to say so. We are not looking for someone who quietly builds whatever architecture was suggested in the interview. We want someone senior enough to challenge it.
What you’ll own
You’ll design and operate the AI platform behind our patient-facing products.
That includes:
Building the internal API used by voice, web chat, and future AI interfaces.
Serving and operating open-weight language and speech models in production.
Benchmarking vendor models against self-hosted alternatives.
Measuring latency, throughput, concurrency, GPU utilization, accuracy, and cost.
Building streaming inference paths with proper timeouts, retries, backpressure, and useful failure handling. Designing the boundary between the model and backend actions such as appointment booking.
Building safe tool-calling workflows where models can propose actions but application code decides whether those actions are allowed.
Grounding models on our own content and measuring where retrieval still fails.
Evaluating whether fine-tuning, LoRA, quantization, or other model adaptations are actually worth using.
Building regression and evaluation suites so model or infrastructure changes cannot quietly make the system worse.
Designing infrastructure that lets us switch between vendor-hosted and self-hosted models without rewriting every product.
Owning observability around inference, model behavior, GPU performance, and production failures.
You should have
Built a production API around an LLM or similar model that was consumed by more than one product or service. Operated a language model, speech-recognition model, or comparable ML workload on infrastructure you were responsible for.
Strong Python and Linux experience.
Production experience with containers and GPU inference.
Experience with a serving stack such as vLLM, SGLang, or an equivalent production inference system.
Enough understanding of NVIDIA GPUs, CUDA, memory utilization, batching, and concurrency to size and cost model serving.
Experience with streaming responses, asynchronous workloads, timeouts, retries, rate limits, and graceful failure handling.
Production experience with tool/function calling.
A strong opinion about why an LLM should not be treated as an authorization system.
Experience handling untrusted input reaching systems capable of triggering actions.
Experience with RAG or another approach to grounding models on private or domain-specific information. Experience evaluating model quality rather than relying only on public benchmarks.
Experience adapting open-weight models with LoRA, PEFT, QLoRA, or equivalent techniques.
The ability to estimate things such as: concurrent sessions per GPU, cost per million tokens, cost per hour of audio, latency at different loads, and where the system starts to degrade.
Enough independence to work without a large ML team around you.
The confidence to tell us when our assumptions are wrong.
You should also understand that using patient conversations to generate an answer and using those conversations to train or improve a model are different data-processing decisions. We expect that distinction to matter in the architecture.
Our likely stack
We care more about your ability to evaluate and operate these systems than whether you have used every exact tool below.
Our current direction is:
Application & APIs
Python
FastAPI
Pydantic
asyncio
SSE / WebSockets
Model serving
vLLM
SGLang
Hugging Face Transformers
Speech recognition
Whisper-family models / faster-whisper as an initial baseline
CTranslate2
FFmpeg
audio processing and resampling tools
Model adaptation
PyTorch
PEFT
LoRA / QLoRA
Retrieval
PostgreSQL
pgvector
Hugging Face / sentence-transformer embedding models
Infrastructure
Linux
Docker
NVIDIA CUDA
Terraform
Kubernetes and Helm where the operational complexity is justified
NVIDIA GPU Operator / DCGM at larger scale
Observability
OpenTelemetry
Prometheus
Grafana
structured logging
Evaluation
Python-based evaluation tooling
pytest
jiwer or equivalent ASR metrics
golden datasets
load and GPU benchmarking
automated regression gates
We do not expect you to blindly adopt this stack. Part of the role is deciding where it is appropriate and where it is not.
Nice to have
Quantization experience and a practical understanding of its quality/performance trade-offs.
Experience benchmarking different inference engines or GPU configurations.
Speech/audio knowledge: codecs, sample rates, VAD, resampling, telephone-quality audio, noise, and streaming. GPU scheduling and Kubernetes.
PostgreSQL/pgvector, Qdrant, OpenSearch, or similar retrieval infrastructure.
OpenTelemetry, Prometheus, Grafana, or similar observability tooling.
Shadow deployments, canaries, model routing, or A/B evaluation.
Experience running the same assistant across voice and text channels.
Experience in healthcare or another environment where privacy and auditability materially affected system design.
Working somewhere GDPR was treated as an engineering constraint rather than a line in a policy document.
You do not need
To have built a telephony or voice-assistant stack. Another role owns the voice layer.
To speak German.
To know PHP.
To have trained a foundation model from scratch.
To have used every library in our current stack.
We care much more about whether you can look at a proposed architecture, benchmark it properly, understand where it fails, and make a defensible technical decision.
This role is for someone who enjoys owning both the AI platform and the evidence that proves it is good enough to run in production.
Apply to marko@apexteamhiring.com