Validating Conversational Analytics, Query by Query
How 1,140+ manual test cases across three database engines, AI-generated SQL, and enterprise-grade security gave a natural-language analytics platform the QA rigour mission-critical decisions demand.
Quality assurance for a natural-language enterprise analytics platform.
The client's platform is an AI-powered enterprise analytics tool that lets business users interact with structured data using natural language — eliminating the need for SQL knowledge or dependency on analytics teams to answer everyday business questions. The platform allows users to connect multiple types of data sources and simply ask questions in plain English. The AI automatically interprets intent, generates optimised SQL, executes it across the connected database, produces business-friendly summaries, recommends the best visualisation, builds interactive dashboards, and enforces enterprise-grade security and role-based access — all within seconds. This lets both technical and non-technical users explore data independently, replacing scheduled reports and dashboard-building bottlenecks with real-time, conversational access to enterprise data.
Proving that conversational analytics can be trusted with mission-critical business decisions.
FUNCTIONAL TEST CASES EXECUTED
ENTERPRISE DATABASES VALIDATED
CROSS-DATABASE TEST CASES
BILLING & SUBSCRIPTION CASES
Proving an AI can be trusted with the query, every time.
Organisations struggle to get fast, reliable insights from structured data spread across multiple systems — dependent on analysts, SQL expertise, and slow, inconsistent reporting. The platform's promise was to remove all of that friction; the QA challenge was proving it could do so safely, accurately, and securely across every supported database.
Multi-Database Consistency
The platform had to generate accurate SQL and return correct results regardless of whether it was connected to PostgreSQL, Microsoft SQL Server, or ClickHouse — each with its own SQL dialect and behaviour.
AI Accuracy at Scale
Natural language prompts spanning direct queries, aggregations, multi-table joins, filters, and time-based and multi-level aggregations all had to produce correct SQL and correct business insights, not just plausible-looking answers.
Enterprise Security & Access Control
Organisation isolation, role-based access control, and table- and column-level permissions all had to hold up under testing to ensure users could only ever reach authorised data.
Dashboard & Collaboration Complexity
Dashboard and widget creation, sharing, permissions, and access management all needed validation to ensure collaboration features didn't create unintended data exposure.
Usage-Based Billing Accuracy
A credit-based AI usage model — seat-based credits, workspace top-up pools, organisation usage analytics, and billing workflows — required careful validation to ensure customers were billed accurately for what they used.
Testing the AI, the data layer, and the access model — together.
A comprehensive manual testing strategy was executed across every supported database platform, AI workflow, dashboard capability, and security feature — resulting in 1,140+ functional test cases across all major product areas.
Multi-Database Validation
Validated database onboarding and connectivity, schema discovery, metadata generation, table and column validation, SQL generation accuracy across dialects, data retrieval consistency, and error handling across PostgreSQL, SQL Server, and ClickHouse.
AI Query Validation
Executed a large set of natural language prompts covering direct and aggregate queries, single- and multi-table joins, search, sorting, filtering, and time-based and multi-level aggregations — validating AI-generated SQL and business insights across tables and columns.
AI Response Validation
Validated every generated response for SQL correctness, result accuracy, business summaries, visualisation recommendations, chart rendering, regeneration behaviour, query explanation, clarification handling, and error messaging.
Dashboard & Collaboration Testing
Validated dashboard and widget creation, deletion, copying, and moving, dashboard sharing and permissions, access management, and role-based access to individual widgets.
Security & RBAC Testing
Validated organisation isolation, role-based access control, table- and column-level permissions, shared dashboard permissions, and user invitation and access flows.
Billing & Usage Validation
Validated the AI credit consumption model — seat-based credits, workspace top-up credits, organisation usage, member consumption, billing workflows, subscription management, and invoice generation.
Comprehensive UI & Functional Testing
Covered AI chat, dashboards, widgets, dashboard sharing, billing and subscription, team management, roles and permissions, audit logs, system prompts, feedback, the embeddable chatbot, and responsive UI validation.
functional test cases executed across all major product areas
from AI chat and dashboards to billing, security, and multi-database connectivity.
cross-database test cases validated consistent AI and query behaviour across ClickHouse, PostgreSQL, and Microsoft SQL Server simultaneously.
billing and subscription test cases confirmed accurate credit consumption, workspace top-ups, usage analytics, and invoice generation.
user invitation test cases and 10 RBAC-specific test cases confirmed organisation isolation and role-based access held up as intended.
test cases validated domain-specific reporting scenarios, confirming the platform's natural-language querying held up on real-world, sector-specific data.
test cases validated the embeddable chatbot experience, and 21 validated AI-generated starter questions — ensuring first-touch AI interactions were accurate and reliable.
Faster decision-making, reduced dependency on technical teams, improved data accessibility, and increased adoption of self-service analytics — with QA validating cross-database compatibility, AI response accuracy, dashboard collaboration, and enterprise security ahead of production readiness.
The stack.
Beyond validating what the platform does, the QA process proved the rigour behind it — cross-database compatibility, AI response accuracy, dashboard collaboration, and enterprise security all held up under 1,140+ test cases. That combination of breadth and depth significantly reduced deployment risk and gave the organisation confidence to adopt conversational analytics for decisions that matter.