Salesforce Decode
Salesforcedecode
Back to questions
AgentforceAdvancedtestingregressionqasynthetic-data

Design synthetic conversation testing for agent regression suites

Real World Scenario

QA cannot manually test 200 topics before each release. Production incidents trace to utterance gaps and action failures that sandbox spot-checks missed.

Expected Answer

• Build synthetic conversation datasets covering happy path, edge cases, adversarial, and multilingual utterances • Automate eval runs in CI comparing expected topic, actions invoked, and response constraints • Score grounding fidelity—did the agent cite correct knowledge IDs for policy questions? • Include red-team prompts for injection, jailbreak, and over-permissioned action attempts • Track eval pass rate trends across releases; block deploy on regression below threshold • Seed production-like data in partial sandbox copies for action integration tests • Maintain human review sample of failed eval cases for taxonomy and prompt updates

Follow-Up Questions & Answers

Click to expand — each follow-up includes a direct, interview-ready answer

Test at three levels: unit (isolated logic), integration (end-to-end flow), and UAT (real user scenarios). Build synthetic conversation datasets covering happy path, edge cases, adversarial, and multilingual utterances. Use sandbox data that mirrors production volume. Manual chat-with-the-agent testing does not scale. Optimize for scale and operational observability.

Architect Perspective

Manual chat-with-the-agent testing does not scale. Invest in eval infrastructure comparable to unit tests for Apex—deterministic assertions on topic, tools, and guardrails.