• Agentic AI
• AI-native SDLC
• Human-in-the-Loop
• Least Privilege
• Model Context Protocol (MCP)
• Knowledge Management

Challenge: AI coding agents deliver code faster than teams can review it the traditional way. It only scales with clear roles, automated checks and a project memory that does not get lost in chat histories.

• Roles (Human-in-the-Loop).

  • The product owner formulates requirements and findings.
  • AI coding agents deliver, in the development environment and as cloud agents that create pull requests on their own.
  • More than 150 pull requests have been merged, almost all of them created by cloud agents.

• Automated checks instead of trust. Every pull request automatically goes through:

  • formatting, type checking and around 2,300 unit tests (approx. 2 minutes),
  • the logged-in end-to-end suite against a disposable database and a deterministic test model (approx. 12 minutes),
  • a preview build.

• Project memory as code (Knowledge Management for agents).

  • Conventions, measurement results and root causes live in the repository as a versioned knowledge base: 38 files, around 29,000 lines.
  • Every agent starts with the same knowledge; every change to it is a reviewable diff.
  • Knowledge only enters the repository through reviewed commits.

• Minimal permissions (Least Privilege).

  • Agents have no write access to production data and only the credentials their task requires.
  • Content from tickets, pull requests and error messages is treated as untrusted (prompt injection).

• Test Automation
• Shift-Left Testing
• Compliance as Code
• Mock-LLM
• Regression Testing
• CI/CD

Challenge: Classic tests expect exact outputs; a language model answers the same question slightly differently every time. Quality therefore comes from several levels: deterministic where possible, statistical where necessary.

• Policy as tests (Compliance as Code, Shift-Left). Tests check the entire codebase against fixed rules – violations surface before the merge, not in operation:

  • Every AI call is pinned to the EU region, zero data retention and exclusion from model training; the only disclosed exception is web research, which can be switched off.
  • Every scheduled job is monitored.
  • Both language versions are complete.
  • No tool description contradicts a defined process chain.

• Unit tests. Around 2,300 test cases in 218 test files, mainly for the calculating tools.

• End-to-end in the browser. 21 Playwright test files. The logged-in suite runs in CI against a disposable database, Redis and a deterministic mock model: reproducible and without model costs.

• Visual checks. Automated screenshots of all screens in both languages, light and dark.

• LLM-as-a-Judge
• Golden Set
• Grounding Checks
• Sycophancy Tests
• Judge Calibration
• Rubric-based Scoring

Challenge: No single assert decides whether an AI answer is good. Without reliable grading, every prompt, tool or model change remains a matter of opinion.

• Scenario catalogue (Golden Set).

  • Around 120 scenarios, as single questions and multi-turn dialogues, each with profile, portfolio and frozen market data.
  • Each scenario carries up to 26 checkable expectations. Each one is graded individually: met, partially met or missed.

• LLM-as-a-judge with a fixed rubric.

  • Six dimensions: profile fit, factual accuracy, consistency, process clarity, risk transparency, advisory experience.
  • 16 predefined finding categories in four groups: verifiability, content, process, form.
  • The categories are enforced by schema so that findings remain comparable across all runs.

• Deterministic checks alongside the judge (Grounding Checks).

  • 34 finding categories, including: invented links, internal identifiers in the answer, a call to trade, candidates silently passed over.
  • Figures are reconciled with the tool results.
  • Every answer is checked against the intended tool chain: missing step, wrong order.
  • Before going live, new rules are measured for false alarms against the stock of stored answers.

• Pushback tests (sycophancy). The simulated user contradicts or pushes. The test checks whether the system sticks to the facts instead of obligingly giving in.

• Tool Calling
• Context Engineering
• Guardrails
• Human-in-the-Loop
• Prompt Caching
• Single Source of Truth

Challenge: Language models phrase things convincingly but calculate unreliably and know no current data. In an enterprise, figures must be right, processes must follow an order and write access must be controlled.

• Division of labour as an architectural principle (Tool Calling).

  • The model understands the question, chooses tools and explains the results.
  • Calculating, checking, filtering and booking are done by 55 tools. Every figure in an answer comes from a tool call.

• Tool descriptions are interface contracts (Single Source of Truth).

  • When in doubt, the model follows the tool's description, not the system prompt.
  • That is why the process chains are stored once, centrally, as data – and a test prevents any tool description from deviating from them.

• Receipts instead of hope. Tools report back what was checked, what was skipped and which step is still open. This deterministically steers the model back into the process.

• Self-correction in three stages.

  1. Invalid inputs are rejected.
  2. Deviations come back as a warning.
  3. Where an objective target-versus-actual check is possible, a silent trial run calculates before execution.

• Grounding
• Structured Outputs
• Agentic Retrieval
• Data Provenance
• Hallucination Rate
• Prompt Injection Defence

Challenge: Enterprise knowledge sits in tables and key figures as much as in reports, news and free text. A language model is meant to use both – without inventing values and without treating third-party text as instructions.

• Grounding via tools (Agentic Retrieval). The model fetches data via tools when needed instead of answering from its training knowledge; every figure can be traced back to a tool call.

• Structured data.

  • An adapter layer normalises market data, such as currency units, and makes the provider replaceable by configuration (less vendor lock-in).
  • A nightly scoring run over a universe of around 2,000 securities archives key figures point in time.
  • Key figures are converted to a common currency basis before comparison.
  • Positions that cannot be scored are disclosed instead of being silently calculated with assumptions.

• Unstructured becomes structured (Structured Outputs).

  • Language models read news articles, annual reports and web sources and write into fixed schemas.
  • Examples: the theme assignment of every security with a conviction level, market catalysts with confidence and source (Data Provenance).

• Closed vocabulary, measured hallucination.

  • The model chooses from predefined categories. If it names one that does not exist, it is discarded and counted.
  • The reason: a silently rising hallucination rate is the error that looks healthiest from the outside.

• Privacy by Design
• Policy as Code
• Data Residency
• Zero Data Retention
• AI Security
• Supply Chain Security

Challenge: Policies that exist only in the prompt or in the manual are intentions. They become reliable when code enforces them and tests verify compliance with them.

• Data residency as a test.

  • Every AI call is pinned in code to the EU region, zero data retention and exclusion from model training. A test fails if a new call bypasses this.
  • The only disclosed exception is web research, which can be switched off. It transmits only a sanitised market question without personal or portfolio data.
  • This request goes directly to the model provider in the USA, likewise without retention.

• Data protection in the data model.

  • Account, card and tax numbers are masked before they are stored.
  • Uploaded images are removed after analysis.
  • Particularly sensitive information is never taken into memory.
  • Users can choose deletion periods for their chats.

• Protection against prompt injection.

  • Via the chat: approvals, permissions and output filters are enforced in code – no prompt can override them.
  • Via tools: external content such as news articles or web sources is data and never counts as an instruction.

• Permissions checked server-side. Identifiers supplied by the model are checked against the owner. Even read paths that are not billed are rate-limited.

• Model Benchmarking
• A/B Testing
• Unit Economics
• LLM-FinOps
• Observability

Challenge: With model changes, new prompts and new features, the loudest argument wins if nothing is measured. And without cost transparency, an AI system scales in the wrong direction.

• Model choice by experiment (Model Benchmarking).

  • A larger language model competed against the one in use.
  • Set-up: 38 scenarios with 81 dialogue turns, two runs per variant, two different judge models.
  • Quality result: in three of four comparisons, the difference lay within the variance of two identical runs, in one just above it, and it was not significant anywhere.
  • Cost and time result: around 2.2 times the cost per dialogue turn, median response time around 30 seconds longer.
  • Decision: no switch.

• Statistics instead of gut feeling. Paired comparisons with exact p-values, correction for multiple testing (Benjamini-Hochberg), pre-registered comparison pairs.

• Research before features.

  • More than 200 evaluation and research scripts, including 23 studies on the predictive power of individual signals.
  • Hypotheses that do not hold up do not enter the product.

• Costs under control (Unit Economics).

  • Usage is metered and capped per user and month.
  • Background work such as memory and research is reported in separate budgets.
  • Prompt caching and reading from the database instead of live calls reduce the cost per dialogue turn.