Building a Private Intelligence Agency, Locally

Written By Kory Hayward

My "Second Brain" had a serious dependency problem.

For years, I’ve refined my Obsidian vault to be more than just a storage locker for links and a heap of one-off notes. I wanted a true graph: a system where ideas, individuals, and organizations link together to reveal patterns I might otherwise miss. To automate this, I built a set of Python tools, among them a News Agent that fetches updates, tags notes, and synthesizes weekly reviews.

Initially, I built this on top of OpenAI’s cloud API. It worked, but it felt fragile. I was outsourcing the core logic of my personal knowledge base to a server I didn't own.

So this week, I fired the cloud.

I migrated the entire system to run locally using new open-source models. It wasn't just a technical experiment; it was a move toward data sovereignty. Here is why I made the switch, and how I took a local system mobile.

The Goal: Data Engineering, Not Chatbots

My News Agent isn't a conversational chatbot; it is a data engineering pipeline disguised as a script. I didn't need a personality; I needed a robust system that could:

  • Ingest diverse formats: Handle HTML, PDFs, and messy URLs without complaining.

  • Standardize output: Convert noise into clean Markdown that fits my existing vault schema.

  • Synthesize connections: Link a policy paper read today with a meeting note from three months ago.

The value isn't in the AI writing the text; it's in the AI fostering connections.

Cloud vs. Local: The Strategic Trade-offs

Moving from a cloud provider to a local setup (using LM Studio and gpt-oss-20b) was a strategic calculation.

The Case for the Cloud

  • Raw Intelligence: Let's be honest: GPT-4o is still smarter than my MacBook. It handles nuance and satire better than a quantized local model.

  • Convenience: Zero server management, zero heat throttling.

The Case for Local (The Winner)

  • Cost: Cloud APIs are cheap until you process high-volume feeds. Local inference is free, once you’ve paid the "Apple Tax" for the RAM.

  • Privacy & Security: This is my personal vault. It contains private thoughts and drafts. If I wouldn't paste proprietary company data into a public chatbot, why should I treat my personal intellectual property any differently? Keeping that data on silicon I own is a massive security upgrade.

  • Control: This was the deciding factor. APIs change. Companies pivot. By running a local model, I own the infrastructure. My prompt engineering doesn't break just because a product manager in San Francisco decided to "align" the model differently this week.

Local Power, Anywhere Access

The most common criticism of local AI is the tether. “Great, you have a powerful customized agent, but it's stuck on your desk.”

To solve this, I used Tailscale.

By creating a secure mesh network between my devices, I can SSH into my MacBook Pro directly from my phone. This creates a hybrid workflow:

  1. Mobile Trigger: I capture a URL or have a thought on my phone while out for coffee.

  2. Heavy Lifting: I SSH in and trigger the news_agent.py script. My MacBook spins up, loads the 20B model, and crunches the data remotely.

  3. Unified Sync: The script writes the finished markdown files to my vault, which instantly syncs back to my phone via Obsidian Sync.

I’m not sacrificing mobility for privacy. I’m just moving the compute to where it belongs, on the hardware I own, while keeping the interface wherever I happen to be.

How It Works Under the Hood

The migration was surprisingly straightforward. I updated my Python scripts to point to a local server (hosted by LM Studio) rather than OpenAI's endpoint. The system runs in three distinct loops:

  1. Fetch: Scrapes my reading list, using the local LLM to extract key entities and strategic implications.

  2. Digest: Aggregates daily items into a thematic briefing, filtering signal from noise.

  3. Review: At the end of the week, it looks for macro-trends across the entire week's data.

The result is a system that feels robust rather than magic. It’s a tool I can trust because I understand and control every part of the stack.

You can find the code for this project in my Obsidian Vault Utilities repository on GitHub.

Kory Hayward https://koryhayward.com