Testing business AI with synthetic data: acceptance tests for RAG and Odoo
Before entrusting a process to AI, teams need to reproduce failures without exposing company data. A Microsoft Research publication provides a starting point; here is our proposed acceptance-testing approach for RAG and business agents.
1. A published method, not an assumed product offering
On October 30, Microsoft Research described synthetic enterprise environments for evaluating Copilot without customer data. The method combines simulated organisations, query generation and quality checks. The publication describes an internal system; it does not establish availability for your organisation.
2. What changes for Belgian and French organisations
Our recommendation is to move part of acceptance testing into a reproducible business sandbox before opening production data. An SME can start with fictional customers, orders and documents. Mid-market companies, large enterprises and public bodies should add multiple departments, roles and decision histories. Business owners define the expected answers; IT retains configurations and results. This is a testing approach proposed by Underside, with no quantified savings promise.
3. Underside analysis: test the system’s boundaries
For RAG, build a fictional corpus with two conflicting versions of a procedure and an effective date. Ask for the applicable rule and check citations, dates and the ability to flag ambiguity. For an agent connected to Odoo Enterprise, prepare a separate test database: a fictional quotation, customers with similar names, an unauthorised discount and an already approved invoice. Check that the agent proposes an action without writing when its role or human approval is missing. These are our architecture proposals, not integrations announced by Microsoft.
4. Measurable, repeatable acceptance testing
Assign each scenario an expected outcome, an owner and a blocking criterion: cross-role disclosure, forbidden writes or missing sources. Include cases where the correct response is refusal or clarification. Retain model versions, prompts, corpora, connectors, permissions and logs. Rerun the same suite after each change; review differences before promoting a new configuration. Have business users review a sample so that automated generators and evaluators do not share unchecked blind spots.
5. Privacy and sovereignty: retain a second control
Fictional datasets should stay fictional: avoid copying real tickets, names or contracts into generation instructions. Document sources and inspect outputs before sharing. For local, cloud or hybrid deployment, also test network destinations, log retention and administrator access. Success on synthetic data proves neither regulatory compliance nor behaviour on all real data. Follow with a limited, authorised pilot, human oversight and a rollback procedure.
Acceptance-testing priority: select one workflow, create fictional data and block production release if permission or approval tests fail.
Plan AI acceptance testing