This guide details the setup and implementation of a local Artificial Intelligence (AI) coding workflow, enabling users to run AI models directly on their machines. The process involves selecting appropriate software, understanding key AI terminology, choosing suitable hardware, and integrating local AI models into development environments for tasks such as agentic coding.

Software Prerequisites and Setup

To establish a local AI coding workflow, two primary software tools are recommended:

1. LM Studio: This application serves as a central hub for downloading and managing local AI models. It also provides a local server that allows other coding tools to connect and interact with these models. LM Studio can be conceptualized as a repository for local AI models, similar to GitHub for code.
2. Open Code: This is an open-source agentic coding tool. While other tools like Cloud Code exist and are optimized for specific vendor models (e.g., Entropic, OpenAI), Open Code is presented as a more suitable option for local AI due to its open-source nature. This allows for customization and adaptation to specific enterprise requirements.

Installation involves downloading LM Studio and installing it on the user's machine. Open Code, being a terminal-based tool, can be installed via package managers (e.g., Homebrew on macOS) or by running a curl script from its GitHub repository.

LM Studio offers several functionalities accessible through its interface:

• Chat Interface: A standard chat-based interaction with local AI models, similar to ChatGPT, where responses are generated locally without network connectivity.
• Developer Options: This section provides access to logs and server configurations. The local server is crucial for enabling communication between agent harnesses like Open Code and the local AI model. Users can configure server settings, including host address and model exposure.
• My Models: A list of downloaded AI models.
• Model Discovery: This section allows users to browse and download various open-source AI models, including those from Google, Meta, and Nvidia, often available in different configurations and sizes.

Understanding Key AI Terminology

Navigating the landscape of local AI models requires understanding fundamental concepts:

• Parameters: In the context of Large Language Models (LLMs), parameters (or weights) refer to the numerical values within the model's mathematical matrices. These values are adjusted during training and influence the model's output. A higher number of parameters generally correlates with improved reasoning, nuance, and the ability to handle more complex or vague instructions. However, the increase in output quality with parameters exhibits diminishing returns. A larger parameter count also contributes to slower initial response times (first token latency).
• Quantization: This process reduces the memory footprint of a model by decreasing the precision of its floating-point numbers (weights). For example, quantizing a model to 4-bit means each weight occupies only 4 bits of memory, significantly reducing storage and RAM requirements. While this leads to a less accurate model, the trade-off often allows larger models to run on hardware with limited VRAM.
• Model Formats (GGUF, MLX): GGUF is a common format for AI models. MLX is specifically optimized for Apple Silicon (M-series chips) and is recommended for users on macOS.

Local AI: Advantages and Disadvantages

Local AI offers distinct benefits and drawbacks compared to cloud-based models:

Advantages:

• Privacy: Data processed by local AI models never leaves the user's device, making it ideal for sensitive information and privacy-conscious users. This is particularly relevant in business contexts involving confidential data.
• Offline Availability: Local AI models can be used without an internet connection, enabling functionality in remote locations or during network outages.
• Cost-Effectiveness: Once the initial hardware investment is made, running local AI models is free, excluding energy consumption. This contrasts with the per-token costs associated with cloud-based services.

Disadvantages:

• Quality Limitations: Local AI models generally do not match the output quality of frontier cloud models (e.g., those from OpenAI or Anthropic), especially for complex tasks like advanced coding. This can result in more errors, hallucinations, and less nuanced responses.
• Hardware Dependency: Performance is heavily reliant on the user's hardware. Insufficient hardware can lead to unusable response speeds.
• Response Time: Local models can sometimes take longer to generate responses compared to highly optimized cloud services.

Local AI is recommended for users prioritizing privacy, those with existing suitable hardware, or for less complex tasks. For highly demanding applications like intricate coding problems, cloud-based frontier models may still be superior. It is generally not advisable to purchase expensive hardware solely for local AI if cloud subscriptions offer a more cost-effective solution.

Hardware Recommendations

The primary hardware metric for running local AI models is VRAM (Video Random Access Memory), which is the memory on a graphics card. Ideally, the entire model's weights must reside in VRAM for optimal performance, as VRAM offers significantly higher bandwidth than system RAM.

• Minimum Requirements:

  • Nvidia GPU: At least 24 GB of VRAM (e.g., RTX 4090 or 5090).
  • Apple Silicon (M-series): At least 32 GB of unified memory.
  • Below these thresholds, the experience is likely to be frustratingly slow.

• Comfortable Range:

  • Nvidia GPU: 48 GB of VRAM or more.
  • Apple Silicon (M-series): 64 GB of unified memory or more.

• High-End Setup:

  • The author uses a MacBook Pro with an M5 Max chip, 40-core GPU, and 128 GB of unified memory, which provides a comfortable and high-quality local AI experience, though still not on par with top-tier cloud models.

If a model's size exceeds available VRAM, the overflow spills into system RAM, drastically reducing performance to unusable levels (e.g., 2-3 tokens per second).

Model Selection

When selecting a model, consider the trade-off between parameter count and quantization level. Smaller models with fewer parameters are less demanding but may offer lower quality. Larger models can be made more manageable through quantization.

• Recommended Starting Point: The Quen series of models is suggested for beginners.

  • Quen 2.5 Coder (32 billion parameters): Occupies approximately 18 GB of RAM/VRAM (4-bit quantized). This is a good option for users with hardware in the 32 GB to 64 GB RAM range. However, it may exhibit limitations in understanding tool access for agentic coding.
  • Quen 3 Coder Next (80 billion parameters, 4-bit quantization): Occupies roughly 45 GB. This model is recommended for users with hardware comparable to the author's high-end setup. It offers a balance between token generation speed and output quality. For macOS users, the MLX variants are preferred.

• Mixture of Experts (MoE) Models: These models can reduce the number of active parameters at any given time by routing tasks to specialized "experts" within the model. This can lead to lower VRAM consumption for active parameters. Models with "MOE" in their name, indicating a specific number of active parameters (e.g., 3 billion active parameters), are often designed for coding agents.

The Quen 3 Coder Next model is highlighted as delivering the best results for the author's personal use on high-end consumer hardware.

When choosing a model, users can consult cloud-based AI models by providing their hardware specifications to get recommendations for appropriate open-source models.

Configuring and Running Local Models

1. Download Model: Select and download the desired model through LM Studio.
2. Configure Model Settings: In LM Studio, navigate to "My Models." Adjust the context window size (number of tokens the model can process per session). Increasing this enhances the model's ability to retain context but demands more resources. The maximum supported context window for the chosen model should be used if hardware permits.
3. Set Up Local Server: Go to the "Developer" tab in LM Studio and configure the local server.
   • Authentication: Enable authentication and create an API token. This token will be required by the agent harness (e.g., Open Code) to connect to the server.
   • Model Loading: Select the downloaded model to be exposed by the server. Typically, only one model is loaded at a time to conserve resources, though dual-model setups are possible for specific use cases like IDE autocompletion.
   • Start Server: Ensure the server is running. LM Studio will display supported API endpoints.

Integrating with Open Code for Agentic Coding

1. Launch Open Code: Open a terminal in your IDE (e.g., VS Code) and run the open code command.

2. Configure Open Code: Create an open_code.json configuration file in your project's root directory. This file specifies the connection details for your local LM Studio server.

   • Provider: Set to lmstudio.
   • Base URL: http://localhost:1234 (default LM Studio server address).
   • Models: Define the models to be exposed. The model key must exactly match the model's code name as shown in LM Studio.
   • Max Tokens: Set the maximum tokens per response, ideally matching the model's recommended value.
   • Context Length: This should match the context window size configured in LM Studio.

3. Reload Open Code: After saving the open_code.json file, relaunch Open Code by exiting and re-entering the command in the terminal. The configured local model (e.g., Quen 3 Coder Next) should now be pre-selected.

Example: Generating a Note-Taking App

To demonstrate the workflow, a request was made to generate a simple CRUD note-taking application using React. The process involved:

1. Prompting the Model: A detailed prompt was given to Open Code, specifying the requirements for a React-based note-taking app with a list and detail view, create, edit, and delete functionality, a simple UI, and local browser execution without remote storage.
2. Code Generation: The local AI model processed the request and generated the necessary code. Initial attempts required further instruction to utilize a proper build tool like Vite.
3. Execution: The generated code was placed in a notes-app.jsx file within a Vite project structure. The application was then run using npm run dev.
4. Result: A functional, albeit basic, note-taking application was created. The generated code demonstrated the capability of local AI for code generation. However, minor bugs and limitations were observed, highlighting the difference in quality compared to frontier cloud models.

While local AI offers a powerful privacy-preserving and offline solution, the author notes that for complex tasks, the superior performance of cloud-based models often justifies their cost. However, for scenarios where privacy, offline access, or cost are paramount, local AI presents a viable and increasingly capable alternative.