Comma Soft’s open source-based GenAI technology is liberating European enterprises from LLM-as-a-service providers

According to the industry association Bitkom, one in six companies has plans to use generative AI. A study carried out by researchers at Stanford and Berkeley universities recently revealed that companies cannot currently rely on the quality of tools like ChatGPT. “In recent months, we have observed an increasing demand among our customers accompanied by a simultaneous critical attitude towards the LLM-as-a-service solutions currently available on the market,” Benjamin Schulte, COO of Comma Soft AG, confirms, adding: “For companies operating in Europe, tools from US vendors often raise concerns about whether or not regulatory requirements are being adequately met.” These regulations include, for example, the EU Data Protection Regulation (GDPR), the German Trade Secrets Protection Act (Geschäftsgeheimnisgesetz) and industry-specific requirements. Moreover, within the business context of a specific company, training data for GenAI solutions is often required that is not included in the existing tools, e.g., confidential data from the company’s own information systems. “Companies are in search of answers on how to deal with these limitations and are increasingly approaching us for advice. With more than a decade of experience in AI research, consulting and solution development, we are often their very first port of call. Through our self-developed LLM, we can now offer our customers the technological basis for secure and worry-free GenAI use and support them throughout the entire process, starting with evaluation and implementation right through to its productive use,” says Benjamin Schulte.

Safe GenAI deployment thanks to open source

Comma Soft’s LLM is based on an open-source foundation model, currently Llama 2 from Meta. It can be hosted both as on-premises software and in any cloud environment, in Germany for example. “Companies can use it to create an enterprise GenAI solution that gives them full control over their data. This is because the data is not shared externally and is not automatically used for model training,” explains Dr. Andrej Fischer, executive manager and one of the GenAI experts at Comma Soft AG. In this way, companies also have full control when it comes to retraining with the company’s own data. This makes it possible to retrain the LLM for industry-specific and company-specific use cases as well as to integrate existing specialized applications and databases via plug-in.

With that technological foundation in place, companies can:

• Implement a GenAI solution tailored to their requirements
• Verifiably meet regulatory & compliance requirements, e.g., the EU-DSGVO
• Use high-quality GenAI in the German language
• Support employees in their work processes in a goal-oriented and targeted manner
• Decide for themselves how their GenAI solution is hosted, e.g., as on-premises software or in a German cloud environment
• Map additional use cases flexibly & cost-efficiently now and in the future

“Companies simply can no longer hesitate when it comes to generative AI, as more and more people are using such solutions privately and value the quick results it delivers. Employees also want this kind of digital assistant in their day-to-day work – and may already be using it without the company’s knowledge. This is why we have been using our own ‘CommaGPT’ – a further development of ChatGPT-4 based on the Azure API – on a daily basis for all our employees and selected customers since May. But because not only the legal parameters but also quality, relevance and consistency in the corporate context are enormously important, the LLM-as-a-service solutions that are commonly available on the market will usually require a retrofit. This is associated with a considerable amount of time and effort as well as additional costs, the latter increasing further in the future due to pay-per-token models, especially for the productive use in volume processes,” says Dr. Andrej Fischer and adds: “This is where companies can support their employees by providing their own GenAI solutions that not only meet their business needs, but can also be more helpful than the models currently available on the market thanks to targeted further development and customization.”

Scalability and performance without any cost traps

The introduction of an in-house GenAI solution represents a major challenge in terms of IT resources, especially for medium-sized companies. But even large corporations are required to maintain a balance between performance and costs due to their high user numbers. Dr. Lars Flöer, Lead Consultant Artificial Intelligence at Comma Soft AG, and his team are developing the LLM to be suitably scalable for this purpose: “Unlike “as a service” solutions, companies need to keep in mind that with a self-powered LLM, in addition to the model and training data, they also need an appropriate IT infrastructure, and in particular, capacity for fine tuning and hosting. For the large mid-sized companies and even for corporations, this is often associated with high investments and implementation risks. Therefore, it was important for us to create a technological basis in our LLM that would solve this problem.”

To ensure both cost-efficient and high-performance operationalization, Comma Soft AG relies on the following building blocks, among others, for its LLM technology:

• Optimized training methods in which only the relevant model parts are (re)trained rather than the entire model, thereby requiring less training data and hardware (LoRa & QLoRA).
• More efficient hosting, where the ready-trained LLM is run in a hardware-saving way with few of the expensive and sometimes hard-to-get graphics cards (inference quantization).
• Maximum performance and scalability, where even thousands of users can access the LLM simultaneously without delays thanks to dynamic load distribution (continuous batching, autoscaling).

Development partnerships as a driver for Germany as a business location

With many companies currently facing the challenge of how to operate GenAI themselves cost-effectively and securely, Comma Soft is committed to development partnerships, as Benjamin Schulte points out: “Instead of working alone on one solution at a time, we can use our ‘swarm knowledge’ to make a much bigger difference and achieve far greater benefits. We would therefore like to invite all interested companies to join us in exchanging experiences and together make GenAI ready for the European market.”

If you would like to discuss this topic or be a partner in the development of Comma Soft’s LLM technology, please contact Comma Soft AG’s LLM Task Force Leader Dr. Valentin Kemper: You can contact us here.

You can also find further information here:

• GenAI solutions for businesses
• ChatGPT & Co under the microscope: What is generative AI capable of?