Introducing AI in a Mid-Sized Company: What the Managing Director Must Decide, Check and Answer For

Artificial intelligence has arrived in most companies - but the profit from it has not. 88 percent of the organizations surveyed by McKinsey use AI in at least one business function, yet only around 6 percent are among the leaders attributing at least 5 percent of their earnings (EBIT) to AI (McKinsey, State of AI 2025). This gap between usage and value creation is the real leadership task for managing directors and CEOs in 2026.

For decision-makers in small and medium-sized enterprises (SMEs), the question shifts fundamentally: no longer "should we use AI?", but "why do only a few manage to create real value - and how do we become one of them?". This guide answers exactly that. It shows where AI really pays off in SMEs, what obligations the EU AI Act 2026 brings, why so many AI projects fail - and provides a concrete 90-day roadmap for adoption.

Key Takeaways

• The value gap is the real problem: 88 percent use AI, but only around 6 percent attribute at least 5 percent of their earnings to AI (McKinsey 2025).
• Usage is growing fast: 54.4 percent of companies in Germany used AI in May 2026, up from 40.9 percent a year earlier (ifo 2026).
• Failure is common: Around 95 percent of the GenAI pilots studied deliver no measurable return, mostly because they are not embedded in workflows rather than because of the technology (MIT 2025).
• AI is a boardroom matter with liability: Since February 2025 the AI Act requires AI competence in the company; anyone who lets adoption run without the care § 43 GmbHG demands risks personal liability.
• Success is 70 percent a question of people and processes, not technology (the Boston Consulting Group's 10-20-70 principle).
• Structure beats chance: A clear 90-day roadmap delivers results faster than aimless experimentation.

AI in SMEs 2026 - where do we really stand?

The use of AI is no longer a niche topic in the German Mittelstand, but it is far from universal. According to a Bitkom survey of 604 companies with 20 or more employees from March 2026, 41 percent use AI, up from 17 percent the year before. At the same time, KfW Research shows in February 2026 that around 20 percent of mid-sized companies used AI in the period from 2022 to 2024, just under 780,000 businesses.

These figures only seem to contradict each other: they are based on different populations and periods. KfW looks at the entire Mittelstand in the years 2022 to 2024, Bitkom at companies with 20 or more employees in early 2026. The ifo Institute reports 54.4 percent of the companies surveyed for May 2026, with large enterprises at 67.2 percent, clearly ahead of small and medium-sized businesses (ifo Institute, June 2026).

One development is decisive: AI has become a boardroom matter. 72 percent of the CEOs surveyed by the Boston Consulting Group now see themselves as the main decision maker on AI, twice as many as the year before (BCG AI Radar 2026). That is logical, because AI is not a pure IT question but a strategic one.

Why 95 percent of AI pilots deliver no ROI

The sobering truth behind the AI boom: most projects do not pay off. A widely noted study by the MIT project NANDA concluded in August 2025 that 95 percent of enterprise-wide GenAI pilot projects have no measurable effect on the profit and loss statement.

The right interpretation matters: what is measured is the missing business value, not a technical failure. The technology works - but it is too rarely used in a way that ends up putting more money in the bank. This is precisely where the management's task lies.

What "AI as a boardroom matter" concretely means for managing directors

AI as a boardroom matter means that management sets direction, resources and framework - and does not delegate what is strategic. In concrete terms: prioritize use cases, release budget, take responsibility for governance and bring the workforce along.

The figures show why this counts: according to McKinsey, CEO oversight of AI governance is among the factors most closely linked to a measurable earnings contribution from AI. Yet only 28 percent of respondents said their CEO is responsible for AI governance (McKinsey, State of AI, March 2025). Anyone who leaves AI to chance or to the IT department alone gives away the very lever that decides between success and failure.

For managing directors in SMEs, this is good news: you do not have to become a data scientist. You have to ask the right questions, set clear priorities and organize responsibility.

Where AI really pays off in SMEs - three use cases with ROI

Not every AI deployment pays off equally well. McKinsey estimates that around 75 percent of the value potential of generative AI falls into four areas: customer service, marketing and sales, software development, and research and development (McKinsey, 2023). For SMEs this means: the best entry point is where processes are recurring, data-intensive and time-consuming.

The following three scenarios are illustrative examples of typical SME situations. The figures given come from the linked studies and should be understood as orientation, not as guaranteed results.

Scenario 1: Back office - the incoming invoice as a quick win

A typical entry scenario in mechanical engineering: a business with around 180 employees processes hundreds of incoming invoices manually every day. With AI-supported document recognition, invoice data can be automatically extracted, checked and prepared for posting; approval stays with a person. An additional driver is e-invoicing: since January 2025, companies in Germany must be able to receive e-invoices, and the obligation to issue them follows from 2027 and 2028. Because processing time per document is easy to measure before and after, the back office is well suited as a first use case. The duties that follow from GoBD and VAT law are covered in the article Automating Back-Office Processes with AI.

Scenario 2: Sales - faster to the right quote

A technical wholesaler with around 100 employees loses time and orders because quote generation is manual and slow. AI can prepare quotes from product data and customer history and prioritize incoming inquiries by closing probability. McKinsey reports that companies investing in AI for marketing and sales see 3 to 15 percent more revenue and a 10 to 20 percent higher sales ROI (McKinsey, 2023). The lever is not in replacing sales, but in accelerating it.

Scenario 3: Customer service - with a reality check

A service provider with around 60 employees wants to answer recurring customer inquiries automatically. An AI assistant can solve standard cases around the clock; McKinsey estimates the productivity potential of generative AI in customer service at 30 to 45 percent of the function's current costs. An NBER study from 2023 measured 14 percent more issues resolved per hour for service agents with AI support, and 34 percent for new and less experienced agents (NBER Working Paper 31161).

But honesty belongs here: the Klarna example shows the limits. According to the company, its AI assistant handled two thirds of service chats in its very first month (Klarna, February 2024). In 2025, however, Klarna announced it would hire people for customer service again: quality had suffered, and customers must always be able to talk to a person (Fortune, May 2025). The lesson for SMEs is: AI plus human, not AI instead of human.

Assessment: The first use case should not be the most spectacular one, but the most quickly measurable. A well-chosen first success finances and legitimizes the next steps. On the way there, the MIT study shows a clear pattern: solutions purchased from specialized vendors succeeded in around 67 percent of cases, in-house developments only one third as often.

Agentic AI - the top trend of 2026 with a reality check

AI agents are the dominant topic of 2026 - and at the same time the one with the greatest hype risk. An AI agent is a system that does not just respond, but independently plans and executes multi-step tasks: for example checking an order, creating it in the system and notifying the customer. The article Which AI agent types exist? shows which kinds of AI agents there are.

The market potential is enormous. Gartner forecasts that by the end of 2026 around 40 percent of enterprise applications will contain task-specific AI agents - compared to under 5 percent in 2025.

But the reality check belongs here too: Gartner also expects that over 40 percent of agentic AI projects will be discontinued by the end of 2027 - due to excessive costs, unclear benefits or insufficient risk control. Added to this is the phenomenon of "agent washing": of the thousands of providers advertising agents, Gartner estimates only around 130 offer genuine agentic capabilities.

For CEOs this means: take the trend seriously, but do not follow the hype. An AI agent only pays off along a clearly defined business problem - not because it happens to be in fashion.

Why AI projects fail - and how you avoid it

Many AI projects fail. RAND Corporation cites estimates putting the share at more than 80 percent, twice the rate of IT projects without AI (RAND, 2024). The reasons are rarely technical. They repeat themselves:

1. Missing data foundation: The data is incomplete, scattered or of poor quality.
2. Unclear use case: The project starts without a defined business problem and without a measurable goal.
3. Missing integration: The pilot remains an island and is never embedded into everyday work.
4. Neglected change management: The workforce is not brought along, acceptance is lacking.
5. Competence gap: The knowledge to use and steer AI sensibly is missing.
6. Wrong expectations: Impatience leads to premature cancellation.

Behind this is a pattern that the Boston Consulting Group summarizes in the 10-20-70 principle: success with AI consists of 10 percent algorithms, 20 percent technology and data - and 70 percent people and processes. The biggest lever is therefore not in the model, but in the organization.

McKinsey confirms this: redesigning workflows has the greatest influence on whether a company achieves an earnings contribution from generative AI. Yet only 21 percent of respondents said they had fundamentally redesigned at least some workflows (McKinsey, State of AI, March 2025). Anyone who simply layers AI over existing processes rarely reaps more than an expensive gadget.

EU AI Act 2026 - what managing directors need to know now

The EU AI Act is not an abstract Brussels topic, but concerns the obligations - and the liability - of every management. The legal situation moved once more in 2026: with the Digital Omnibus (Regulation (EU) 2026/1744, in force since 27 July 2026) the EU postponed the obligations for stand-alone high-risk systems under Annex III from 2 August 2026 to 2 December 2027, and those for high-risk systems embedded in products under Annex I to 2 August 2028. A detailed assessment for SMEs is provided in our article on the EU AI Act 2026.

What is decisive, however, is what has not been postponed and already applies:

Two points should be taken particularly seriously by managing directors. First, the AI competence obligation under Article 4: since February 2025, companies must take measures for the AI literacy of their employees. Since the Digital Omnibus it is enough to support its development; nobody has to guarantee a specific level. Certification is not required, and the regulation sets no separate fine for breaches of Article 4. Training should still be documented, because it shows that management fulfils its organisational duty. Second, the fines under Article 99: anyone using prohibited practices risks up to 35 million euros or 7 percent of global annual revenue; for breaches of most other obligations, such as the transparency obligations, it is up to 15 million euros or 3 percent. For small and medium-sized enterprises the lower of the two amounts applies in each case.

Personal liability is especially important. When it applies and which provisions carry it is set out in the next section.

When is the managing director personally liable for AI mistakes?

When adoption runs without the care that § 43 paragraph 1 GmbHG demands of a prudent businessperson: without documented selection, without rules, without training, without supervision. The machine's error is rarely the ground for liability. The absence of an organisation behind it is.

Four sets of provisions carry this statement, and three of them are older than any AI debate:

On top comes the company's external liability for what the AI tells customers. In Moffatt v. Air Canada the Civil Resolution Tribunal of British Columbia ruled in February 2024 that the airline was bound by false information given by its chatbot. The tribunal rejected the argument that the chatbot was a separate entity. The case was decided in Canada, but the lesson carries over: whatever a customer-service agent promises, the company should be able to honour.

What management should therefore keep on file fits on one page:

• The decision basis: which providers were assessed, why the choice fell as it did, which risk class the system has under the AI Act
• The AI policy: which tools are approved, which data may go in, who decides exceptions
• The training record: date, content, participants, repetition
• The contracts: data processing agreement with every AI provider, data residency, exclusion of training use
• The escalation: which cases the system hands to a human, who monitors the answers, how errors are reported

This is no substitute for legal advice, and assessing a specific damage case belongs to a lawyer. But the documentation a lawyer will later want to see is created in the first 90 days or not at all. That is exactly what the roadmap in the next section is built for.

The 90-day roadmap for AI in SMEs

The most common mistake is aimless experimentation. A structured entry over 90 days delivers results faster than years of trial and error. The following roadmap is divided into three phases.

Day 1-30: Stocktaking and quick win

1. AI inventory: where is AI already being used in the company - even unofficially?
2. Collect use cases: where do processes cost the most time and money?
3. Prioritize with the impact-effort matrix: high benefit, low effort first.
4. Select a quick win and define a clear metric by which success can be measured.

Day 31-60: Pilot and governance

1. Implement the quick win as a pilot - preferably with a proven purchased solution rather than in-house development.
2. Set up a lean AI policy: what is allowed, which data may be used?
3. Start AI competence training - this also fulfills Article 4 of the EU AI Act.
4. Clarify data protection: order processing, data residency and documented approvals.

Day 61-90: Scaling and anchoring

1. Measure the ROI of the pilot and assess it honestly.
2. Redesign the workflow - do not just set up the tool, but adapt the process.
3. Define responsibilities: who maintains, monitors and improves the solution?
4. Continue the roadmap and tackle the next use case.

After 90 days, the finished AI transformation is not in place - but a proven success, a functioning governance and an organization that has learned how AI works for it.

AI governance and risks - the CEO's duty

Governance is not a bureaucratic accessory, but the prerequisite for ensuring that AI does not become a risk. The reality is sobering: in IBM's Cost of a Data Breach Report 2025, 63 percent of the breached organizations studied had no fully developed AI governance policy (IBM, July 2025).

Three risks should be kept particularly in view by managing directors:

• Shadow AI: Employees use AI tools on their own initiative and share sensitive data in the process. Data breaches involving shadow AI cost an average of 4.63 million US dollars (IBM 2025).
• False information and liability: In the Moffatt v. Air Canada case, the company was held liable in 2024 for false information from its chatbot - the argument that the chatbot was independently responsible was rejected.
• Manipulation and fraud: At the engineering group Arup, an employee in Hong Kong transferred around 25 million US dollars to fraudsters in 2024 after a video conference with colleagues imitated by deepfake (CNN, May 2024).

The NIST AI Risk Management Framework and the ISO/IEC 42001 standard have established themselves as a framework for serious AI governance. They help to cleanly regulate responsibilities, risks and controls.

For SMEs there is also the question of which data may go into which tool. That requires data processing agreements, a legal basis for data transfers with providers outside the EU and a policy that defines data classes. What such a policy can look like is described in the article ChatGPT in the Company.

Conclusion: AI is a boardroom matter - but not a self-runner

The gap between usage and value creation is the central challenge of 2026: almost everyone uses AI, but only a few make money with it. The difference lies not in the technology, but in leadership - in clearly chosen use cases, a structured approach, lived governance and the willingness to truly rethink processes.

For managing directors and CEOs in SMEs, this is an opportunity: anyone who starts in a structured way now gains a lead that aimless competitors will not catch up on quickly. The 90-day roadmap is the pragmatic entry point.

Frequently Asked Questions (FAQ)

What does AI concretely bring to my small or medium-sized company? AI mainly saves time in recurring workflows: in the back office, in sales and in customer service. Typical entry points are processing incoming invoices or preparing quotes. The decisive thing is to start with a use case whose benefit can be measured against a metric.

Where should I start with AI as a managing director? With the use case that promises high benefit at low effort. Collect time-consuming processes, assess them in an impact-effort matrix and choose a quick win with a clear metric. The 90-day roadmap in this article provides the structure for it.

What does introducing AI in the company cost? That depends on the use case. For getting started: according to MIT's Project NANDA, solutions purchased from specialized vendors succeeded far more often than in-house developments. It is important to budget for the hidden costs: under the Boston Consulting Group's 10-20-70 principle, 70 percent of the effort belongs to people and processes, not the technology.

How do I calculate the ROI of an AI project? Define a metric in advance and measure before and after: time saved times personnel costs, a higher closing rate or shorter lead times. Be honest about the time horizon: in a Deloitte survey of 1,854 executives in Europe and the Middle East, most achieved a satisfactory ROI on a typical AI use case only after two to four years.

What do I need to consider as a managing director with the EU AI Act 2026? Three things already apply: the ban on certain AI practices, the AI competence obligation under Article 4 (train employees and document it) and, since August 2026, the transparency obligations. Obligations for high-risk AI under Annex III apply, following the Digital Omnibus, only from 2 December 2027. The highest fines of up to 35 million euros or 7 percent of global revenue apply to prohibited practices; the regulation sets no separate fine for the competence obligation.

Am I personally liable if the AI makes a mistake? Yes, if adoption runs without the care § 43 GmbHG demands: no documented selection, no AI policy, no training, no supervision. The company can then seek recourse, and § 130 OWiG threatens a fine for breach of the duty of supervision. The Canadian case Moffatt v. Air Canada also shows that a company can be bound by false information from its chatbot. Clear governance is therefore mandatory.

Can I use ChatGPT and other AI in a GDPR-compliant way? Yes, with the right framework: data processing agreements, clear rules on data use, contractually excluded training use and avoiding shadow AI. For sensitive data, services with processing in the EU and contractually assured confidentiality are the safer choice.

Why do so many AI projects fail - and how do I avoid it? RAND Corporation cites estimates that more than 80 percent of AI projects fail, mostly for organizational reasons: unclear goals, poor data, missing integration and a workforce that is not brought along. The most important lever is to redesign workflows instead of just layering AI over old processes.

---

Sources

• The state of AI in 2025: Agents, innovation, and transformation - McKinsey & Company
• The state of AI: How organizations are rewiring to capture value (March 2025) - McKinsey & Company
• AI-powered marketing and sales reach new heights with generative AI (2023) - McKinsey & Company
• The economic potential of generative AI (2023) - McKinsey & Company
• Digitalisation of the economy: almost every company is engaging with AI (press release of 11 March 2026) - Bitkom e.V.
• AI in the Mittelstand - Focus on Economics No. 533 - KfW Research
• More than half of companies in Germany use artificial intelligence (5 June 2026) - ifo Institute
• BCG AI Radar 2026: As AI Investments Surge, CEOs Take the Lead - Boston Consulting Group
• MIT study: 95 percent of GenAI pilots without ROI - Fortune / MIT Project NANDA
• Gartner: 40 percent of enterprise apps with AI agents by 2026 - Gartner
• Gartner: Over 40 percent of agentic AI projects canceled by 2027 - Gartner
• The Leader's Guide to Transforming with AI (10-20-70) - Boston Consulting Group
• Generative AI at Work - NBER
• The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed (2024) - RAND Corporation
• AI ROI: The paradox of rising investment and elusive returns - Deloitte
• Klarna AI assistant handles two-thirds of customer service chats in its first month (February 2024) - Klarna
• Klarna plans to hire humans again (May 2025) - Fortune
• Arup revealed as victim of $25 million deepfake scam (May 2024) - CNN
• EU AI Act - Article 4 (AI literacy) - EU AI Act
• EU AI Act - Article 26 (Obligations of deployers) - EU AI Act
• EU AI Act - Article 99 (Penalties) - EU AI Act
• § 43 GmbHG - Liability of managing directors - Gesetze im Internet
• § 130 OWiG - Breach of the duty of supervision - Gesetze im Internet
• IBM Report: 13% of organizations reported breaches of AI models or applications (July 2025) - IBM
• AI Risk Management Framework - NIST
• ISO/IEC 42001 - AI management systems - ISO