Prototype of a Voice AI Operator for Medical Clinics in Europe

Challenge

Our client, a digital solutions startup for medical institutions, wanted to test whether it was possible to build a proprietary voice assistant system that could replace a live operator while remaining fully GDPR-compliant. According to GDPR, personal data must be processed lawfully, fairly, and transparently for the data subject, with clear and unambiguous consent, and for a specific purpose.

The main challenge was that existing market solutions, while offering the required functionality (speech recognition, synthesis, dialogue management), typically operate in the cloud, meaning patient data passes through external services and can be exposed to leakage risks.

It was critical for the client to have a system that works fully on-premises, within their own infrastructure, while maintaining the same quality as cloud-based AI services.

Beyond security, it was necessary to evaluate how well modern open-source technologies perform in real calls: Can the assistant correctly recognize speech, schedule an appointment, provide reference information, and hold a conversation without noticeable delays or interruptions?

The project’s main goal was to create a working prototype to validate whether the chosen technology stack could support a full-fledged MVP and further development.

Solution

We implemented a voice assistant prototype by deploying all core components on-premises within the client’s infrastructure. The architecture included speech recognition (Whisper/Vosk), a dialogue engine based on Rasa, and speech synthesis via Coqui TTS.

The assistant accepted calls through a PBX, recognized patient requests, identified intent, executed actions via APIs (e.g., booking an appointment or providing information), and delivered responses in a natural voice.

Integration with corporate systems

As part of the prototype, we integrated with Bitrix24 CRM and Google Calendar, allowing the AI operator to schedule appointments or reschedule them directly.

Rasa Research

The main task of this PoC was to explore the Rasa platform.

Rasa is an open-source framework for building conversational systems and chatbots, based on machine learning and natural language understanding (NLU).

With NLU, Rasa detects intents and extracts entities from user messages. This formed the core idea of the project — recognizing client intent, extracting relevant entities, and acting on them.

Rasa is trained on example dialogues, learning to predict the bot’s responses or actions in different scenarios without hardcoding conditional rules. Bot responses can be linked to text outputs or Python scripts for more complex behavior.

The platform supports integration with external services and messengers, enabling seamless embedding of virtual assistants into business processes and communication channels.

During the project, we explored Rasa’s strengths and limitations:

• Pros: open source, flexible scenario configuration, local deployment support, modular architecture.
• Cons: high data quality requirements for training, manual intent configuration, fewer built-in monitoring tools compared to commercial solutions.

Multilingual support

Since the client plans to serve European medical clinics, they required multilingual functionality. Our prototype successfully handled two languages — English and Russian.

Results

As part of the prototype, we validated key hypotheses:

• How accurately the system recognizes and interprets speech,
• Whether the assistant can hold coherent conversations without noticeable delays,
• How a fully local setup performs without the cloud,
• How easily scenarios can be extended and CRM integration achieved.

The PoC confirmed the concept’s viability. The system successfully handled calls, recognized requests, and executed core scenarios: appointment booking, providing reference information, clarifying visit dates. Response times and speech naturalness were sufficient for real-world use. Local deployment proved that it is possible to avoid cloud services while preserving data privacy.

For the client, this confirmed that the chosen architecture can serve as the foundation for a full MVP.

Project outlook

Although the project was developed for medical clinics, the solution can be applied in many industries where voice communication automation and strict data confidentiality are critical: banking, insurance, logistics, and government services.

The key differentiator from most existing systems is full on-premises deployment, ensuring compliance with personal data confidentiality requirements and protecting clients’ data from leaks. Unlike cloud services, where data inevitably passes through third parties, our architecture remains entirely within the client’s perimeter.