Part 7 of the SAP Business AI Series | Back to Series Hub

Pre-delivered AI handles common scenarios well. But every enterprise has unique processes, proprietary data assets, and competitive differentiation that standard configurations cannot capture. SAP AI Foundation on BTP is the infrastructure layer that enables organisations to build beyond the out-of-the-box — developing, deploying, and governing custom AI solutions within SAP’s trusted platform.

This post covers the full technical stack: the AI Foundation architecture, the Generative AI Hub, Joule Studio, the Orchestration Service, Document AI, the Knowledge Graph, and the HANA Vector Engine — everything a developer or technical architect needs to understand to build enterprise-grade AI on SAP BTP.

The AI Foundation: Architecture Overview

Think of the AI Foundation as the operating system for SAP Business AI. It provides a unified, scalable platform to build, orchestrate, and run custom AI solutions and agents — with governance, security, and resource management built in from the ground up.

The architecture is structured into four layers:

1. OS Interfaces Layer — the developer-facing entry point. Includes AI Playground, Joule Studio, and orchestration tools for building, testing, and managing AI capabilities
2. AI Kernel Layer — the runtime management layer. Handles agents, workloads, model lifecycle, and ensures secure, compliant, and efficient AI operations at scale
3. AI Integration Layer — the connective tissue. Manages data integration, model orchestration, and workload routing between services
4. Peripheral and Data Layer — the foundation. Integrates SAP and non-SAP data sources, underlying infrastructure, and partner or third-party AI models

What makes this architecture valuable is not any individual component — it is the fact that governance, versioning, security, and compliance are infrastructure concerns, not application concerns. Developers building on the AI Foundation inherit these capabilities automatically, rather than having to build them from scratch for each use case.

SAP Generative AI Hub: Enterprise Access to Frontier Models

The Generative AI Hub is the central component of AI Foundation for LLM-based development. It provides secure, managed access to multiple frontier AI models — both SAP-hosted foundation models and third-party models including GPT, Claude, Gemini, Mistral, and others — through a single, governed interface.

What the Generative AI Hub Provides

• Model Flexibility — access to multiple LLMs from a single platform, with the ability to switch models or run comparisons without changing application code
• Model Library — a searchable catalog of available models with performance benchmarks, cost indicators, and domain specialisation details. Chart mode enables visual comparison between models on any two parameters; Leaderboard mode shows ranked scores across benchmarks
• Chat Interface — a conversational environment for interactive testing and rapid prototyping, with configurable model parameters and session context management
• Prompt Editor and Prompt Management — a complete environment for designing, versioning, and governing prompts, with support for system/user/assistant roles, variables, and metadata tagging
• Governance and Compliance — privacy safeguards, legal vetting, content filtering, and commercial frameworks built in to support responsible AI adoption
• Bring Your Own Model (BYOM) — organisations can plug in fine-tuned models trained on proprietary data, while still benefiting from the Hub’s governance infrastructure

The Orchestration Service: Multi-Step AI Workflows Without Complexity

Most enterprise AI use cases are not a single LLM call. They involve retrieving data, filtering sensitive content, applying safety policies, translating between languages, calling the model, filtering the output, and delivering a compliant response — all coordinated in a specific sequence. The SAP Orchestration Service manages this as a single, managed component.

The execution order of orchestration modules is fixed to ensure security and compliance, but each module is individually configurable. The full pipeline looks like this:

1. User Input — the request originates from an end user or upstream business application
2. Grounding — relevant, factual information is retrieved from trusted enterprise sources to anchor the response in business reality
3. Templating — user input, grounded data, and system instructions are combined into a structured prompt using configurable templates
4. Input Masking — sensitive information (PII: names, emails, phone numbers, addresses) is detected and pseudonymised before reaching the LLM
5. Input Filtering — the masked prompt is scanned for harmful, toxic, or inappropriate content
6. Input Translation — if required, the prompt is translated into the LLM’s operating language
7. LLM Processing — the secured, filtered, grounded prompt is processed by the selected model
8. Output Filtering — the response is scanned to ensure it meets content safety and compliance standards
9. Output Translation — the response is translated back to the user’s original language if needed

From a developer perspective, the entire pipeline executes with a single API call. Once deployed, the workflow runs consistently across environments without requiring application code changes when models or policies evolve.

SAP Document AI: Automating Document Intelligence

SAP Document AI automates the processing of business documents regardless of format — structured tables, semi-structured forms, and unstructured PDFs or scanned images. Using a combination of OCR, transformer models, and LLMs, it extracts, classifies, and routes document content directly into SAP application workflows.

Pre-built templates cover the most common document types — invoices, purchase orders, delivery notes, quality certificates — enabling faster deployment without starting from scratch. Deep integration with S/4HANA, SAP Ariba, SAP Concur, and SAP SuccessFactors means extracted data flows directly into the relevant business process.

Measured outcomes include certificate processing time cut from approximately 10 minutes to 3 minutes, significant reduction in manual sales order entry errors, and automated invoice validation integrated into the SAP Business Network.

SAP Knowledge Graph: Semantic Context for AI

The SAP Knowledge Graph is the semantic layer that gives AI systems genuine business context — not just data access, but understanding of how business entities relate to each other. It connects information across finance, supply chain, HR, procurement, and customer data in a way that preserves the relationships between entities.

Key capabilities include natural language querying (users ask business questions in plain language and get meaningful answers), cross-domain intelligence (linking data across business functions), and a developer API that exposes metadata and relationship data for building analytics and AI applications.

In practice, the Knowledge Graph is what enables Joule to answer “Do I have any purchase orders at risk of delivery delay?” — because the agent can navigate from PO to supplier to delivery schedule to performance history, rather than relying on flat data queries that miss the connections between entities.

SAP HANA Vector Engine: The Retrieval Foundation

For RAG to work in an enterprise context, the retrieval step needs to be fast, accurate, and semantically aware. The SAP HANA Vector Engine provides this by enabling storage, processing, and similarity search of vector embeddings directly inside SAP HANA — without requiring a separate vector database.

Two core functions power semantic search:

• COSINE_SIMILARITY() — measures the directional similarity between two vectors. Widely used in text and embedding analysis to assess how semantically similar two documents or queries are, regardless of the exact words used
• L2DISTANCE() — calculates Euclidean distance between vectors in high-dimensional space. Used for clustering and nearest-neighbour searches

Developers query vectors using standard SQL — no specialist database skills required — and in-memory processing ensures performance at enterprise scale. The Vector Engine is the retrieval infrastructure that powers Document Grounding in SAP’s AI stack, enabling grounded LLM responses based on enterprise content stored in HANA.

Document Grounding: Connecting Enterprise Knowledge to LLMs

Document Grounding is the capability that connects enterprise content — internal documentation, policy files, process guides, SharePoint libraries, cloud storage — to the LLMs in SAP’s Generative AI Hub, ensuring that AI responses are based on organisational knowledge rather than generic model training.

The Grounding Management interface in SAP AI Launchpad manages the lifecycle of data pipelines that bring external content into the grounding infrastructure. These pipelines connect sources like SharePoint or Amazon S3, convert documents into vector embeddings, store them in the HANA Vector Engine, and make them available to the Orchestration Service for retrieval during inference.

There are two APIs for managing this process:

• Vector API — for developer-controlled ingestion where you handle chunking and embedding manually. Best for custom knowledge bases requiring fine-grained control.
• Pipeline API — a higher-level managed pipeline that fetches documents from configured sources, chunks them, generates embeddings, and stores them automatically. Best for large, unstructured document repositories.

Key Takeaways

• The AI Foundation is the operating infrastructure for custom AI on SAP BTP — governance, security, and compliance are built in, not bolted on
• The Generative AI Hub provides governed access to multiple frontier LLMs through a single interface, including BYOM support
• The Orchestration Service manages complex, multi-step AI workflows as a single API call, with configurable modules for grounding, masking, filtering, and translation
• Document AI automates document intake and extraction, deeply integrated with SAP application workflows
• The HANA Vector Engine provides enterprise-grade semantic search without a separate vector database
• Document Grounding connects enterprise content to LLMs, ensuring responses are based on organisational knowledge

Next in the series: Post 8 — Responsible AI: SAP’s Approach to Ethics & Governance →