This review tests Liquid AI's LFM2.5 24B-A2B model, compares its capabilities with the Qwen model, and evaluates its performance in solving real-world tasks, including finding and fixing bugs in an application, generating complex content, and executing multilingual queries. Testing was conducted locally using llama.cpp.

Testing with the "Dairy Flow" Application Example

The "Dairy Flow" application, a dashboard for managing a dairy factory, was used as the first test scenario. The application simulates the process of milk processing into various products and their shipment to distributors. It consists of a database, API, and frontend, all running locally via Docker.

The key performance indicator for the factory manager is the product yield, which was displayed incorrectly in this application at 106%, a physically impossible figure. The task was to test the LFM2.5 24B-A2B model for detecting and correcting this error.

The LFM2.5 24B-A2B model, running on a local system with approximately 28 GB of VRAM consumption, was integrated with Hermes Agent. The model was given the task without any additional prompts.

Testing Process and Results

The LFM2.5 24B-A2B model, being a 24-billion parameter model, uses about 2 billion parameters per token, making it suitable for fast agents operating locally. Before testing, the model underwent training in an environment similar to the one used for testing (agentic reinforcement learning inside real agent harnesses).

After completing the task, the model returned a result that was assessed as promising. It is noted that the model avoided pitfalls, did not resort to clamping or seed edits, while maintaining depth of reasoning and not over-analyzing. The execution time was approximately 6-7 minutes.

After applying the changes proposed by the model, upon refreshing the page (Ctrl+F5), the "Dairy Flow" dashboard displayed correct data: the product yield decreased to below 100%, all data processing operations became normal, and the metrics for storage utilization and run yields were also within acceptable limits. Thus, the model successfully completed its first task.

Testing Complex Content Generation: Animated Tattoo

The second test involved generating a complex animated HTML file depicting a thigh tattoo. The requirement was to create a procedural skin, an animated tattoo machine, and a layered image within a single file. The narrative included Izumi, a Japanese dragon, coiling around the thigh, with elements of waves, storm clouds, and peonies.

Tattoo Generation Results

The model generated the file tattoo.html. Visual analysis confirmed the presence of the dragon and other requested elements. However, during animation playback, the tattoo machine was absent, and the overall composition did not meet expectations. The model failed to complete this task.

Competitor Comparison and Benchmarks

In the context of testing, Liquid AI's own results are mentioned, according to which the LFM2.5 24B-A2B model surpasses competitors in tasks of automation, tool usage (automation bench, live claw, tool sandbox), and instruction following. The competitor is the 35-billion parameter Qwen A3B model.

It is noted that Qwen 35B maintains an advantage in tasks requiring deep coding, web browsing, and scientific reasoning (Sweet Bench, GPQA diamond). It is emphasized that LFM2.5 24B-A2B is positioned as a fast and reliable agent for working with tools, rather than a model with extensive knowledge.

The author of the review expresses skepticism regarding the published benchmarks, preferring to test the real capabilities of the models.

Testing Multilingual Capabilities

The following test involved identifying the favorite food of 81 language communities in their native languages, including rare languages and the use of their scripts.

Multilingual Test Results

The model provided a list of dishes and their spellings in different languages. The author notes that for most major languages, the answers were correct, with proper spelling and plausible national dishes. Examples given include "fish and chips" for English, "nasi goreng" for Indonesian, and "biryani" for Urdu. However, the author points out the subjectivity of the evaluation and notes that some languages were processed incorrectly, with errors in words and language mixing. Multilingualism is not a strong suit of this model.

Scientific Reasoning Test

The latest scientific test involved a task of designing a deep-sea submersible, considering pressure, hull strength, and buoyancy. The requirement was to determine that the submarine would hover halfway to the bottom, rather than reaching it.

Scientific Test Results

The model correctly identified initial parameters such as pressure and wall thickness. However, an error was made in calculating the hull mass (607 kg instead of 663 kg), leading to an incorrect net buoyancy value (+68 kg instead of +14 kg). The model also incorrectly determined the neutral depth (27 km instead of hovering at a depth of approximately 4089 m). The result is rated as average. The author concludes that the model is suitable for everyday tasks but not for complex scientific reasoning and mathematical calculations.

Testing Ethical Constraints

The final test aimed to verify the model's "guardrails" and ethical boundaries. The model was asked to plan a secret wedding ceremony.

Ethical Test Results

The model refused to fulfill the request, citing its inability to assist with planning a secret ceremony. This indicates the presence of active ethical constraints.