Going Fully Local: How to Set Up Ollama and Open-Source LLMs in Your IDE

The landscape of AI coding assistants has been dominated by cloud-based services like GitHub Copilot, OpenAI, and Claude. While these tools are incredibly capable, they come with three distinct drawbacks: recurring subscription costs, dependency on an active internet connection, and the risk of sending proprietary code to external servers.

For developers working on sensitive enterprise codebases, building personal projects offline, or seeking full control over their workflows, the solution is clear: Local AI Coding Assistants.

Thanks to Ollama and the open-source community, you can now run state-of-the-art Large Language Models (LLMs) like Llama 3, DeepSeek Coder, and Qwen Coder directly on your local machine.

Here is a step-by-step technical guide to setting up a fully local, offline AI assistant in Visual Studio Code.

---

Why Go Local? The Benefits of Local LLMs

Running models locally on your CPU or GPU is no longer a slow, experimental chore. With modern hardware (like Apple Silicon M-series chips or Nvidia RTX graphics cards), local inference is near-instantaneous. Going local offers:

• Total Code Privacy: Your code never leaves your local hardware. There is zero data leakage or telemetry sent to cloud providers.
• Zero Latency & Cost: No subscriptions, usage caps, or API tier charges. You run models completely for free, offline.
• Customizable Engines: You decide exactly which models to use—whether a small 1.5-billion parameter model for quick autocomplete or a large 14-billion model for architectural brainstorming.

---

Step 1: Install Ollama on Your System

Ollama is a lightweight, open-source framework that packages model weights, configurations, and a GPU-accelerated runner into a single desktop application.

1. Head to ollama.com and download the application matching your OS (macOS, Linux, or Windows).
2. Install and launch the application. You will see a small console icon in your menu bar indicating that Ollama is running its server in the background (typically serving at http://localhost:11434).

---

Step 2: Download Open-Source Coding Models

Now, pull the specialized coding models from the Ollama library. Open your terminal or command prompt and run the following commands:

# 1. Pull the chat model (used for conversation, refactoring, and code explanation)
ollama pull deepseek-coder:6.7b

# 2. Pull the autocomplete model (a lightweight model optimized for low-latency line completions)
ollama pull qwen2.5-coder:1.5b

# 3. Pull a general-purpose model (optional, great for general knowledge and markdown docs)
ollama pull llama3.1

Why two different models? Tab-autocomplete requires sub-second response times. Using a massive model for autocomplete will cause lag in your editor. We use a tiny 1.5B model like Qwen 2.5 Coder for inline completions, and reserve a larger, smarter model like DeepSeek Coder 6.7B for side-panel chat and refactoring.

You can verify your pulled models by running:

ollama list

---

Step 3: Install and Configure Continue.dev in VS Code

To connect VS Code to your local Ollama server, we will use Continue, a premium, open-source AI extension that acts as a custom client interface in your IDE.

1. Open VS Code and open the Extensions panel (Ctrl+Shift+X or Cmd+Shift+X).
2. Search for Continue (by Continue Dev, Inc.) and click Install.
3. Once installed, click the new Continue icon in your VS Code sidebar.
4. Click the gear icon at the bottom right of the Continue pane. This will open the Continue configuration file, config.json (or config.yaml depending on the version).

Replace the contents of the config file with the following setup to link it to your local Ollama server:

{
  "models": [
    {
      "title": "DeepSeek Coder 6.7B",
      "provider": "ollama",
      "model": "deepseek-coder:6.7b"
    },
    {
      "title": "Llama 3.1",
      "provider": "ollama",
      "model": "llama3.1"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Qwen Coder 1.5B",
    "provider": "ollama",
    "model": "qwen2.5-coder:1.5b"
  },
  "customCommands": [
    {
      "name": "test",
      "prompt": "Write a complete set of unit tests for this code using standard testing frameworks.",
      "description": "Generate unit tests"
    }
  ],
  "contextProviders": [
    {
      "name": "code",
      "params": {}
    },
    {
      "name": "docs",
      "params": {}
    }
  ]
}

Save the file. Continue will automatically detect the settings and connect to your local Ollama instance.

---

Step 4: How to Leverage Your Local AI Assistant

Now that your setup is complete, you can leverage it across three primary developer workflows:

1. In-Line Tab Autocomplete

As you type in a file, the lightweight qwen2.5-coder:1.5b model will suggest completions. Press Tab to accept suggestions, or keep typing to ignore them. Because the model is running on your local machine, the latency is extremely low.

2. Side-Panel Chat and Code Explanations

Highlight any block of code in your editor, press Ctrl+L (or Cmd+L), and Continue will copy the code block into the side chat panel. You can ask:

• "Explain how this recursive function works."
• "Refactor this block to use async/await."
• "Are there any memory leaks or security flaws in this code?"

The side chat utilizes the robust deepseek-coder:6.7b model to deliver deep, contextual code logic.

3. Inline Code Edits

Select a block of code and press Ctrl+I (or Cmd+I) to open the inline edit prompt. Type a request (e.g., "Convert this REST endpoint to handle pagination"), and the model will modify the code directly inside your active file, showing a git-style diff comparison so you can approve or reject the changes.

---

Optimizing for Performance: RAM & Hardware Guidelines

If you encounter sluggish response times, check your hardware allocation:

• 8GB RAM / Basic Laptops: Stick to 1.5B to 3B models. Use qwen2.5-coder:1.5b for both chat and autocomplete to avoid swapping memory.
• 16GB RAM / M-series Mac or Mid-range PC: Easily run 7B models (like deepseek-coder:6.7b or llama3) for chat, and 1.5B for autocomplete.
• 32GB+ RAM / High-end GPUs: You can run larger 14B to 32B models (like qwen2.5-coder:14b or codellama:34b) natively with exceptional reasoning and speed.

---

Final Thoughts: Complete Autonomy Over Your Workflow

Setting up a local LLM environment isn't just about saving money; it is about establishing complete autonomy over your development environment. You no longer have to worry about internet outages, API rate limits, or corporate compliance audits flagging external code transmission.

By combining Ollama's local engine with VS Code and Continue.dev, you gain a private, secure, and customizable partner that runs natively right on your machine, helping you build faster, safer, and entirely on your own terms.