Part 9 of the SAP Business AI Series | Back to Series Hub

Understanding SAP’s AI architecture and strategy is one thing. Actually building with it is another. This post is your practical starting point — covering the setup path for SAP Generative AI Hub, the prompt engineering techniques that matter most in enterprise contexts, the Prompt Registry for governing prompts at scale, and how the SAP Cloud SDK for AI brings it all together in code.

Whether you are experimenting for the first time or building a production AI workflow, this post gives you a clear sequence to follow.

Setting Up SAP Generative AI Hub: The Step-by-Step Path

Access to SAP’s Generative AI Hub requires a provisioned SAP AI Core instance in Extended edition and a SAP AI Launchpad instance in Standard edition. Here is the full setup sequence:

1. Set up your BTP Global Account — ensure you have a BTP global account with the necessary entitlements for SAP AI Core and SAP AI Launchpad
2. Provision SAP AI Core (Extended edition) — the Extended edition is required for Generative AI Hub access. The Standard edition of AI Core does not include Gen AI Hub capabilities
3. Provision SAP AI Launchpad (Standard edition) — the Standard edition of AI Launchpad is required for model visibility in the Gen AI Hub. Connect AI Launchpad to your AI Core instance to create a working connection
4. Assign roles — the following roles control Gen AI Hub access:
   • genai_manager — full management access
   • prompt_manager — manage and edit prompts
   • genai_experimenter — explore and test models
   • prompt_experimenter — create and test prompts
5. Deploy an Orchestration Service instance — this is the execution runtime for all Gen AI Hub operations. Without a running orchestration deployment, no LLM calls can be made. Think of it as starting the engine — until it is running, no prompts execute, no RAG pipelines activate, and no policy enforcement happens. It runs as a Kubernetes workload on SAP AI Core and is typically available in your default resource group

Once these steps are complete, you can access the Gen AI Hub through SAP AI Launchpad, through API clients like Postman, or programmatically via the SAP Cloud SDK for AI in Python or JavaScript.

Prompt Engineering: The Skill That Multiplies AI Value

Prompt engineering is the practice of designing and refining the inputs you give an LLM to consistently produce accurate, relevant, and useful outputs. Think of it as programming in natural language — carefully constructed instructions replace code to guide model behaviour.

LLMs are highly capable but inherently general-purpose. Without clear guidance, they produce responses that are broad, vague, or miss the specific business context you need. Prompt engineering provides the structure that unlocks the model’s full potential for a given use case.

The Three Elements of an Effective Enterprise Prompt

1. Define the task clearly — state exactly what the model should do: summarise, classify, extract, translate, generate code, answer a question. Vague tasks produce vague results
2. Add context — provide the background the model needs to give a relevant response. This includes business data, prior conversation, role context (“You are an SAP finance assistant”), and any constraints the model should know about
3. Set constraints on the output — specify format (JSON, bullet list, table), length (one paragraph, maximum 200 words), tone (formal, conversational), and any content restrictions. Output constraints make responses immediately usable in downstream systems

The Three Message Roles: System, User, and Assistant

Modern LLM APIs, including SAP’s Generative AI Hub, organise prompts using three distinct roles. Understanding these roles is foundational to building multi-turn AI applications:

• System role — sets the identity, behaviour, and boundaries of the AI. It answers: who are you, how should you behave, what rules must you follow? The system prompt carries the highest authority in the conversation. Example: “You are an SAP procurement assistant. Provide concise answers using SAP procurement terminology. Never speculate about data you cannot verify.”
• User role — contains the actual request from the user. Dynamic content — variable inputs, live data, user-provided context — goes here. It answers: what does the user want?
• Assistant role — represents prior AI responses in a multi-turn conversation. Including previous assistant turns helps the model maintain context across a longer interaction — important for conversational applications where the user asks follow-up questions

Advanced Prompt Techniques

Few-Shot Prompting

Few-shot prompting provides the model with a small number of examples showing the desired input-output pattern before asking it to complete a new, similar task. Instead of describing what you want in abstract terms, you show it. This technique is particularly effective for classification tasks, structured data extraction, and outputs that need to follow a specific format.

Basic structure of a few-shot prompt:

• Instruction: “Classify the following expense categories…”
• Example 1: Input → Output
• Example 2: Input → Output
• New input: [model completes]

Meta Prompting

Meta prompting is an advanced technique where the prompt defines not just what the model should do, but how it should think, reason, and validate its outputs. Rather than asking for an answer, you provide a reasoning framework the model must follow.

What makes a prompt “meta” is that you specify: how to analyse the task, how to structure reasoning, how to validate the output, and how to self-check results before responding. This produces more consistent, higher-quality responses for complex tasks like compliance checking, multi-criteria evaluation, or structured report generation.

Multi-Modal Prompting

Multi-modal prompting combines text with images or other data types to improve AI accuracy for problems with visual context. Many real-world business problems come with visual elements — a damaged component, a scanned form, a chart from a presentation — that text alone cannot adequately describe. Including an image alongside a text prompt gives the model visual context that improves accuracy and relevance, particularly for classification, diagnosis, and document processing use cases.

SAP Prompt Optimizer: Automating Model Migration

Switching AI models is expensive. Prompts optimised for one LLM often perform poorly on a different model — requiring days of manual rework for each use case when a better model becomes available. SAP’s Prompt Optimizer, developed in collaboration with Not Diamond, automates this process entirely.

When an organisation selects a new LLM in SAP’s Generative AI Hub, the Prompt Optimizer automatically converts all existing prompts into optimised versions for the new model using public performance benchmarks. The result: organisations can adopt the best available model for their use case without the manual migration overhead that previously made model updates a project in themselves.

Prompt Registry: Governing Prompts Like Code

In enterprise AI applications, prompts are not just instructions typed into a chat box. They are the configuration logic that drives business outputs — classifying supplier risk, generating contract language, summarising financial reports. They need to be versioned, governed, and managed with the same rigour as application code.

SAP’s Prompt Registry is an enterprise feature in AI Launchpad that provides centralised storage, version control, governance policies, and CI/CD integration for prompts. Think of it as Git plus a policy layer for your prompt library.

Two Interfaces for Different Stages

• Imperative API — used during development for active prompt engineering. Provides full CRUD capabilities: create, read, update, and delete prompt templates. Tracks all changes and retains version history for audit and debugging. Use this when you are still iterating on prompt structure and behaviour.
• Declarative API — used for production deployment via CI/CD pipelines. Prompts are defined as code (YAML files) stored in a Git repository. SAP AI Core watches the repo and synchronises changes into the Prompt Registry automatically. This makes prompt management identical to application code deployment — with pull requests, code reviews, and automated deployment.

Prompt Templates with Dynamic Placeholders

Prompt templates use placeholders (written as {{?placeholderName}}) to create reusable prompt structures that accept dynamic data at runtime. Instead of hardcoding specific values into every prompt, you define the template once and inject the relevant data — supplier name, invoice amount, contract date — when the prompt is called. This enables consistent prompt logic across thousands of runtime invocations with variable inputs.

SAP Cloud SDK for AI: Programmatic Integration

The SAP Cloud SDK for AI provides the programmatic interface for building AI-powered applications on SAP BTP. It wraps the AI Core and Orchestration Service APIs in language-specific functions for Python and JavaScript, enabling developers to focus on business logic rather than API authentication and request formatting.

Key capabilities accessible via the SDK:

• Prompt Registry access — create, retrieve, and use prompt templates programmatically
• Grounding and content filtering — configure RAG pipelines and safety filters in code
• Structured JSON output — define expected output schemas and validate model responses
• Orchestration workflow execution — trigger complete multi-step pipelines with a single function call
• Prompt evaluation — run automated tests across prompt variants with objective quality metrics (relevance, coherence, fluency) to identify the best-performing version before production deployment

Model Selection: Choosing the Right LLM for Your Use Case

The Generative AI Hub provides access to multiple LLMs — and selecting the right one for a specific use case has meaningful implications for quality, cost, and performance. Key considerations:

• Task suitability — different models excel at different tasks. One model may outperform others on summarisation; another on code generation; another on multilingual tasks. Test on your specific use case, not on general benchmarks alone
• Performance vs. cost — a cheaper model that achieves 95% of the quality of a premium model on your specific task is usually the right choice for production. The Model Library’s cost indicators help make this comparison transparent
• Scalability and redundancy — production deployments should consider having alternative models available. If a primary model experiences latency issues or a change in behaviour following an update, a fallback option reduces operational risk

The practical process: use the Model Library to explore available models and their characteristics, run the same representative prompt against multiple candidates, compare outputs and cost, then select based on actual performance on your specific use case — not reputation or marketing.

Key Takeaways

• Gen AI Hub requires SAP AI Core (Extended) and SAP AI Launchpad (Standard) — and an Orchestration Service deployment before any LLM calls can execute
• Effective prompts define the task, add context, and set output constraints — vague prompts produce vague results
• System/User/Assistant roles structure multi-turn conversations and give the model the context it needs to respond consistently
• Few-shot prompting, meta prompting, and multi-modal prompting are the three techniques that deliver the most value in enterprise contexts
• The Prompt Registry treats prompts like code — versioned, governed, and deployed via CI/CD pipelines
• Model selection should be based on actual performance on your use case, not general benchmarks — use the Model Library to compare

That completes the SAP Business AI series. Return to the Series Hub →

If you found this series useful, the next step is hands-on experimentation. Set up your Gen AI Hub access, run your first prompt through the Chat Interface, and explore the Model Library to compare how different LLMs handle your specific business questions. The best way to build intuition for enterprise AI is to use it — and SAP’s tooling makes that easier than ever.