Independent benchmark data published Thursday by Artificial Analysis gives Meta’s Muse Spark 1.1 its first systematic third-party scorecard since the model’s July 9 API launch — and the results confirm a genuine improvement over April’s original, while also clarifying where a 9-point gap still separates Meta’s model from the agentic coding leaders developers pay the most to replace.
The key number for cost-conscious developer teams: Artificial Analysis estimates Muse Spark 1.1’s cost at approximately $0.26 per task on its Intelligence Index — well below GLM-5.2 at $0.37 and GPT-5.4 at $0.89, despite all three models landing at the same Intelligence Index score of 51. That gap is not a rounding artifact. It is driven by Muse Spark 1.1 generating 94 million output tokens to complete the index’s evaluation suite, compared with 109 million for GPT-5.4, 125 million for GPT-5.6 Luna, and 141 million for GLM-5.2 — a 33 percent reduction against the next most efficient peer at a lower per-token price.
Artificial Analysis, which runs benchmark evaluations for AI labs including Meta, disclosed that it supported Meta with pre-release evaluation of Muse Spark 1.1. The firm uses the same fixed methodology across all models and publishes a confidence interval of less than ±1 percent on the Intelligence Index, making it a more controlled comparison than vendor-reported scores — though the pre-release relationship is worth noting when interpreting the findings.
Eight Points in Three Months, With Gains Concentrated Where They Were Needed
The 8-point gain that moved Muse Spark 1.1 from 43 to 51 on the Intelligence Index was not evenly distributed. According to Artificial Analysis, it concentrated in scientific reasoning, coding, and knowledge — precisely the domains where the April model was openly regarded as a weak spot by developers who had tested it against contemporaries. The coding sub-score within the Intelligence Index rose 12 points (from 59 to 71). Performance on the SciCode scientific reasoning benchmark reached 58 percent, ranking third across all models Artificial Analysis has evaluated, trailing only Claude Fable 5 at 60 percent and Gemini 3.1 Pro Preview at 59 percent. On Humanity’s Last Exam, Muse Spark 1.1 reached 45 percent — within a point of Claude Opus 4.8 (46%) and ahead of GPT-5.5 (44%) and Grok 4.5 (40%).
Agentic knowledge work, measured by GDPval-AA v2, improved substantially — gaining 232 Elo points, from 1,144 to 1,376 — but continues to lag the frontier on that benchmark.
The context window grew from roughly 262,000 tokens on the original Muse Spark to 1 million on the 1.1 release, a meaningful change for repository-level coding tasks. Output speed on Meta’s first-party API runs at approximately 114 tokens per second median, with a time to first answer token of around 21 seconds — faster than the median among comparable-tier reasoning models.
Two Coding Scores, Two Different Workloads
A distinction the benchmark data makes clear — and that matters for any team considering Muse Spark 1.1 for a coding agent — is that Artificial Analysis runs two separate evaluations for code-related tasks, and they measure fundamentally different things.
The first is the Coding sub-score within the Intelligence Index. It is based on LiveCodeBench, a continuously updated benchmark that harvests fresh competitive programming problems from LeetCode, AtCoder, and CodeForces after their contest dates, so no problem in the test set could have appeared in a model’s training data at training time. This contamination-free design addresses one of the central reliability problems in AI evaluation: models trained on internet data are likely to have seen static benchmark questions before taking the test. Muse Spark 1.1 scored 71 on this evaluation.
The second is the Coding Agent Index — a separate evaluation that pairs models with full software engineering harnesses (Codex CLI for OpenAI models, Claude Code for Anthropic models, and Opencode for others) and measures multi-step autonomous software engineering performance across three evaluations: DeepSWE, Terminal-Bench v2, and SWE-Atlas-QnA. This is the benchmark that reflects what a coding agent actually does in production. GPT-5.6 Sol in Codex leads at 80 on the Coding Agent Index, followed by Claude Fable 5 in Claude Code and GPT-5.6 Terra, both at 77. For developers building agents that autonomously diagnose bugs, implement features in large codebases, or execute multi-step development workflows, the Coding Agent Index is the more relevant number.
A competitive programming score of 71 and a multi-step software engineering score of 69 are related but not equivalent. Teams evaluating Muse Spark 1.1 for agent workloads should test against both — and should note that the 9-point gap between Muse Spark 1.1 and the Coding Agent Index leader is real and measurable on that separate evaluation.
Hallucination Rate Fell, but Through Abstention, Not Accuracy
The mechanism behind Muse Spark 1.1’s improvement on AA-Omniscience — Artificial Analysis’s benchmark tracking both hallucination and factual accuracy — is a behavioral shift as much as a capability one, and developers should understand the tradeoff before deploying the model.
The AA-Omniscience score rose from 4 to 18, more than quadrupling. But the improvement came from the model declining to answer more frequently, not from becoming more accurate. The attempt rate dropped from 95 percent to 82 percent, the hallucination rate fell 35 points to 38 percent, and accuracy held roughly flat (from 45 percent to 41 percent). The model now answers fewer questions than its predecessor and gets a higher share of those answers right — primarily because it is no longer attempting questions where it would have previously guessed with false confidence.
For developers building coding agents that operate with limited human oversight, this is a meaningful improvement. A model that confidently generates wrong code is typically worse than one that flags uncertainty and asks for clarification. For general-purpose conversational use cases where users expect an answer to every query, a model that declines 18 percent of attempts is a different product than its predecessor. The right evaluation of this tradeoff depends on the deployment context.
Why Meta Can Price Below Cost
Meta’s API rate card — $1.25 per million input tokens and $4.25 per million output tokens — sits well below what Anthropic and OpenAI charge for models with comparable Intelligence Index scores. The structural reason is not a model architecture secret: it is a business model difference.
Anthropic and OpenAI depend on API margins to fund model development and company operations. Their output token pricing has to cover that overhead from API revenue alone. Meta generates more than $60 billion annually in advertising profit and can run the Meta Model API as a strategic acquisition channel for its developer ecosystem — one that spans WhatsApp, Instagram, Facebook, and Ray-Ban smart glasses — without requiring the API itself to be profitable. The model API is simultaneously a product and a distribution play for Meta’s consumer AI ecosystem.
That structural cost advantage is sustainable for Meta indefinitely. It is not replicable by a pure-play AI lab, which means the cost gap between Muse Spark 1.1 and comparable-tier models from Anthropic or OpenAI reflects a structural business condition, not a temporary promotional pricing decision. Developers budgeting for volume API workloads should treat Meta’s pricing as durable rather than introductory.
New Meta Model API accounts receive $20 in free credits. The public preview remains US-only and waitlisted. The API is designed to be compatible with OpenAI-style client libraries, which lowers the friction for teams integrated with GPT endpoints who want to run a cost comparison without rebuilding their client stack.
Independent Numbers vs. Self-Reported Claims
As with any frontier model launch, the spread between self-reported and independently measured scores warrants careful reading. Thursday’s Artificial Analysis data provides a more controlled comparison than vendor-reported scores — but it is not the only independent reading available, and independent evaluators have found different numbers.
On Terminal-Bench 2.1, Meta’s own evaluation report cited a score of 80.0. Vals AI, running the same benchmark on an independent harness, measured 69.29 — a gap of more than 10 points. That discrepancy surfaced within hours of the July 9 launch and has not been publicly resolved. The gap is not unique to Muse Spark 1.1: industry analysis consistently shows meaningful differences between lab-controlled evaluation conditions and independently administered runs of the same benchmarks.
Meta’s previous model family provides a relevant precedent. The Llama 4 family’s benchmark results came under scrutiny after the April 2026 launch, with Yann LeCun — who left Meta as chief scientist in November 2025 — telling the Financial Times that certain Llama 4 scores had used specialized model versions for specific evaluations while shipping a weaker general-purpose version. Meta’s evaluation report for Muse Spark 1.1 also discloses that the company’s own safety team could not rule out the model meeting “high risk” capability thresholds in Chemical & Biological and Cybersecurity domains without mitigations applied.
The practical reading of Thursday’s Artificial Analysis data: the Intelligence Index score of 51, the coding sub-score of 71, and the cost-per-task figure of $0.26 are from a single independent evaluation with a fixed methodology. They represent a genuine and useful signal. They are not a proxy for every real-world workload, and teams running coding agents at production scale should run their own evaluation on a representative task sample before routing significant traffic.
What Comes Next for Meta’s AI Strategy
Muse Spark 1.1 is the second model from Meta Superintelligence Labs (MSL), the unit assembled under Chief AI Officer Alexandr Wang following Meta’s pivot away from its open-weight Llama strategy. The model is positioned as a closed, hosted, metered product — a departure from the Llama family, which was distributed as open weights and reached more than 1.2 billion downloads. With Muse Spark 1.1, Meta is in direct monetization competition with Anthropic and OpenAI for the first time.
The next-generation model internally code-named Watermelon is reportedly in training. In a July 2 internal town hall, Wang described Watermelon as training on roughly 10 times the compute of Muse Spark and claimed it had reached GPT-5.5 parity in internal evaluations — claims that have not been independently verified, and for which no named benchmark or release date has been provided.
For developers evaluating today’s options, the message from Thursday’s data is specific: Muse Spark 1.1 is competitive at its Intelligence Index tier, its 8-point improvement over April’s original is independently confirmed, and its cost-per-task advantage over GPT-5.4 is real and structural. The 9-point gap on the Coding Agent Index between Muse Spark 1.1 and GPT-5.6 Sol is equally real. Whether that gap matters depends on whether the specific workload requires single-step coding problem solving — where the cost-efficiency case is strong — or multi-step autonomous software engineering, where the frontier models still lead.
Frequently Asked Questions
How does Muse Spark 1.1 compare to GPT-5.4 on independent benchmarks?
Both models score 51 on the Artificial Analysis Intelligence Index, making them effectively tied on that composite measure. The meaningful difference is cost: Artificial Analysis estimates Muse Spark 1.1 at approximately $0.26 per task versus $0.89 for GPT-5.4 — a more than three-to-one gap — because Muse Spark 1.1 generates fewer output tokens to complete the same evaluations and carries a lower per-token price. On the separate Coding Agent Index, which measures multi-step software engineering agent performance, GPT-5.6 Sol currently leads at 80, a benchmark that GPT-5.4 also lags.
What is the Artificial Analysis Intelligence Index, and is it truly independent?
The Artificial Analysis Intelligence Index is a composite evaluation suite that runs the same set of benchmarks across models using a fixed methodology, with a published confidence interval of less than ±1 percent. It is independent in the sense that Artificial Analysis applies the same evaluation conditions to all models rather than accepting vendor-reported numbers. However, Artificial Analysis disclosed that it supported Meta with pre-release evaluation of Muse Spark 1.1 before the model launched publicly — a commercial relationship worth noting when interpreting the scores. The firm also runs its own contamination-free coding evaluation that continuously harvests new programming problems from competitive platforms rather than using a static problem set, which addresses one of the main reliability problems in AI benchmark design.
Why does Muse Spark 1.1 score differently on coding depending on which benchmark you consult?
Because the different benchmarks measure different things. The Coding sub-score within the Intelligence Index (where Muse Spark 1.1 scored 71) uses LiveCodeBench, a competitive programming evaluation focused on single-turn problem solving. The Coding Agent Index (where GPT-5.6 Sol leads at 80 and Muse Spark 1.1 scored 69 in one independent harness) measures multi-step autonomous software engineering using full agentic scaffolds — writing code, running it, reading errors, and iterating. A developer building an agent that files bug fixes in production codebases should consult the Coding Agent Index; a developer evaluating code-generation quality in isolation should consult the single-turn coding benchmark. The two scores address different use cases and should not be compared directly.
How much does the Meta Model API cost, and who currently has access?
Meta charges $1.25 per million input tokens and $4.25 per million output tokens, with a cache-hit rate of $0.15 per million. New accounts receive $20 in free credits. The public preview is currently US-only and requires joining a waitlist. The API is built to be compatible with OpenAI-style client libraries, so teams using GPT endpoints can test Muse Spark 1.1 without rebuilding their integration layer.
Enjoyed this article? Sign up for our newsletter to receive regular insights and stay connected.