Salesforce Advances Enterprise AI with Consistency Focus and New Benchmarks

Salesforce is addressing a critical challenge in enterprise artificial intelligence: the gap between an AI system's raw intelligence and its ability to perform consistently in unpredictable business environments. This issue, termed "jagged intelligence" by Salesforce, highlights how AI systems often excel in controlled tests but struggle with reliable task execution in complex, real-world settings.

To tackle this, Salesforce AI Research introduced several new benchmarks, models, and frameworks designed to enhance both the capabilities and consistency of AI agents tailored for enterprise use. Central to this effort is the concept of Enterprise General Intelligence (EGI), which focuses on AI optimized for business complexity and consistent performance rather than the broader, more theoretical pursuit of Artificial General Intelligence (AGI).

Measuring and Improving AI Consistency with SIMPLE Benchmark

Salesforce developed the SIMPLE dataset, a public benchmark consisting of 225 straightforward reasoning questions. This dataset quantifies the "jaggedness" or inconsistency in AI system capabilities, providing a measurable foundation to improve AI reliability. For enterprises, this consistency is mission-critical, as AI missteps can disrupt operations, damage customer trust, and cause financial losses.

CRMArena: Simulating Realistic Enterprise AI Scenarios

The CRMArena framework is a novel benchmarking environment that simulates realistic customer relationship management scenarios. It tests AI agents across roles such as service agents, analysts, and managers, revealing current performance gaps—agents succeed less than 65% of the time in key tasks. This tool helps Salesforce identify failure points and iteratively improve AI agents for enterprise readiness.

Advanced Embedding Models for Deeper Enterprise Context

Salesforce introduced SFR-Embedding, a model leading the Massive Text Embedding Benchmark across 56 datasets, designed for superior contextual understanding in enterprise data. A specialized variant, SFR-Embedding-Code, enhances code search capabilities for developers. These models will soon be integrated into Salesforce’s Data Cloud, enabling more precise and efficient AI-driven insights.

xLAM V2: Action-Focused AI Models for Enterprise Tasks

The xLAM V2 family of models emphasizes predicting and executing next actions rather than just generating text. These models range from 1 billion to 70 billion parameters, with smaller models delivering powerful, cost-effective performance. By fine-tuning large language models on action trajectories, Salesforce equips AI agents to interact autonomously and reliably with enterprise systems.

Ensuring Enterprise AI Safety with Trust Layers and Guardrails

To address safety and reliability, Salesforce developed SFR-Guard models trained on public and CRM-specific data, forming a Trust Layer that enforces guardrails on AI agent behavior. This ensures agents operate within business policies and standards, reducing risks. Additionally, the ContextualJudgeBench evaluates AI responses for accuracy, conciseness, and appropriateness, further enhancing trustworthiness.

Multimodal AI with TACO for Complex Problem Solving

Salesforce also introduced TACO, a multimodal action model family that addresses complex, multi-step problems by integrating chains of thought and action. This approach improves AI’s ability to interpret and respond to queries involving multiple media types, achieving up to 20% better performance on challenging benchmarks like MMVet.

Customer Co-Innovation Driving Enterprise AI Progress

Salesforce emphasizes collaboration with customers to refine AI solutions that meet stringent enterprise accuracy and relevance demands. Innovations like the Atlas reasoning engine and retrieval augmented generation techniques have doubled accuracy in customer deployments compared to competitors, demonstrating the impact of co-innovation on advancing enterprise AI capabilities.

The Future of Enterprise AI with Salesforce

Salesforce’s research and innovations come at a pivotal time as enterprises demand AI systems that balance advanced capabilities with dependable, consistent performance. By focusing on closing the consistency gap and prioritizing real-world business needs, Salesforce is shaping the future of enterprise AI. Upcoming rollouts will integrate these technologies into products like Data Cloud and Agentforce, empowering businesses to harness AI with confidence and control.