How to run a Qwen-Coder-based code completion tool locally on a Mac

Introduction

Leveraging Ollama and Continue, this guide elucidates the process of deploying the Qwen-2.5-Coder model on a MacBook for enhanced code completion, offering a compelling alternative to GitHub Copilot.

As developers increasingly prioritize privacy and localized tools, many seek local alternatives to cloud-based solutions like GitHub Copilot. This document outlines the configuration and execution of the Qwen-2.5-Coder model using Ollama and Continue on a MacBook M3 Max (36GB RAM), providing a performant and privacy-conscious code completion experience.

Why Qwen-2.5-Coder?

Developed by Alibaba DAMO Academy, Qwen-2.5-Coder is a highly efficient code generation model supporting numerous programming languages and excelling in code completion and refactoring suggestions. Local deployment offers several advantages:

• Preservation of Code Confidentiality: Eliminates the need for code uploads to the cloud, ensuring data security.
• Enhanced Development Velocity: Facilitates local inference, obviating network dependency.
• Optimal Hardware Utilization: Capitalizes on the potent computational capabilities of the MacBook M3 Max.

Tool Overview

Ollama

Ollama is a local AI model management tool facilitating the download and execution of various AI models. It provides an API for seamless integration with other applications.

Continue

Continue is a VSCode extension that integrates flawlessly with local or remote AI models, offering intelligent code completion and refactoring recommendations.

Prerequisites

Prior to commencement, ensure the following prerequisites are met:

• Operating System: macOS Sonoma or later.
• Hardware Requirements: MacBook M3 Max (36GB RAM, recommended for large-scale AI models).
• Required Tools:
  • Docker (for Ollama model execution).
  • Homebrew (for Ollama installation).
  • VSCode and the Continue extension (installable from the marketplace).

Ollama Installation

1. Installation package method, you can skip using the command line.

If you are not familiar with the command line, you can directly download the installation package. After installation, click the program to start it.

Download link

2. Using the command line method.

bash
brew install ollama

Verify successful installation:

bash
ollama --version

Initiate Ollama:

bash
/opt/homebrew/opt/ollama/bin/ollama serve

Model Selection and Memory Requirements

Considering the parameters and memory demands of the Qwen-2.5-Coder model, the following configurations are recommended for the MacBook M3 Max:

Memory Estimation Methodology

• Total Parameters × Floating-Point Precision Coefficient:
  • FP32 (32-bit precision): ~4 bytes/parameter.
  • FP16 (16-bit precision): ~2 bytes/parameter.
  • INT8 (8-bit precision): ~1 byte/parameter (requires model quantization support).
• Inference Memory ≈ Model Weight Memory × 2 (includes activation values and optimizer state).

Model Memory Requirements Estimation

Hardware Compatibility Recommendations

For the MacBook M3 Max (36GB RAM), the following models are recommended:

Recommended Model

• 7B Model: Offers optimal performance and quality balance on the M3 Max.
• 3B Model: A suitable alternative if memory is a constraint.

Downloading and Executing the Qwen-2.5-Coder Model

Download the chosen model using Ollama:

bash
ollama pull qwen2.5-coder:7b

After download, initiate the model service:

bash
ollama run qwen2.5-coder:7b

By default, Ollama initiates a local HTTP API service, listening on port 11434.

Configuring the Continue Extension

1. Launch VSCode and install the Continue extension.
2. Navigate to the extension settings, locate the model configuration options, and set the model address to http://localhost:11434.
3. Designate the model name as qwen2.5-coder:7b.

Configuration Example:

json
{
  "models": [
    {
      "title": "Ollama",
      "provider": "ollama",
      "model": "AUTODETECT"
    }
  ],
  "tabAutocompleteModel": {
    "title": "qwen2.5-coder:7b",
    "provider": "ollama",
    "model": "qwen2.5-coder:7b"
  },
  "tabAutocompleteOptions": {
    "debounceDelay": 1000,
    "maxPromptTokens": 1500
    // "disableInFiles": ["*.md"]
  }
}

User Experience

Upon completion of the configuration, utilize the Qwen-2.5-Coder model within VSCode for enhanced code completion and optimization suggestions:

• Intelligent Code Completion: Automated completion of code input.
• Refactoring Recommendations: Proactive suggestions for optimizing existing code.
• Error Diagnostics: Assistance in analyzing and rectifying erroneous code.

Performance Optimization Recommendations

1. Adjust VSCode Extension Settings: Refine the completion trigger frequency and context length to balance performance and completion quality.

By following these configuration steps, leverage the power of Ollama and Continue to locally deploy the Qwen-2.5-Coder model on your MacBook M3 Max, enjoying efficient code completion with enhanced privacy. This localized solution caters to developers prioritizing privacy and performance over cloud-based alternatives like GitHub Copilot.

Your feedback and inquiries are welcome in the comments section.

Reference Links

• Ollama
• Qwen2.5-Coder
• Continue