I installed Khoj on Linux to build a personal memory system

I wanted to install Khoj for a simple reason.

I wanted to build a private second brain.

It was more than a place to store notes. I wanted a system that could hold my journals, life events, emotional patterns, memories, and personal reflections. I wanted to ask questions about my own life and get useful answers from my own writing.

That was the goal.

The process was not simple.

Why I Wanted Khoj

I have been thinking about a personal memory system for a while.

I want a place where I can write about what happened in my life. I want to track how I felt. I want to keep notes about important people, patterns, trauma, goals, and lessons.

Then I want an AI system to help me search through it.

Not in a vague way. I want to ask things like this.

“What patterns keep showing up in my journals?”

“What did I write about this person?”

“What life events shaped how I think now?”

“What did I believe back then that I no longer believe today?”

That is why Khoj interested me.

It is made to work like a second brain. It can search your own files and answer questions from them. That made it a good system to test.

The Install Was Harder Than Expected

I installed Khoj on my M1 MacBook running Fedora Asahi Linux.

That alone made things harder.

Most guides assume a normal Mac setup, a regular Linux computer, or Docker. My setup was different. I was using Apple Silicon, but through Linux. That caused problems.

The first problem was Python packages.

Some packages did not install cleanly. One package needed system tools before it could build. Another package did not support my setup the way I expected.

Then the database became the next problem.

Khoj needed PostgreSQL. I tried to use the embedded database option first. That did not work on my machine because one of the required pieces was not available for Linux on Apple Silicon.

So I had to install and set up PostgreSQL myself.

That led to another issue. PostgreSQL was running, but it would not let Khoj log in. The error was caused by a login rule called ident. In plain English, PostgreSQL was ignoring the password and checking the system user instead.

I had to change the PostgreSQL config so password login would work.

Then Khoj needed the pgvector extension. That was another thing I had to install before the database could work.

Each step fixed one problem and revealed the next one.

The Mistakes I Made

I made a few mistakes during the process.

The first mistake was assuming Khoj would automatically read anything inside my folder.

It does not work that way.

Khoj does not automatically scan every folder on your computer because it is installed. You have to upload files, configure a source, or create a sync process.

The second mistake was putting the Khoj app folder inside my PersonalMemory folder.

That caused a big problem.

When I made a sync script, it scanned the whole PersonalMemory folder. Since Khoj was installed inside that folder, the script uploaded hundreds of random software files. It uploaded things like license files, README files, package files, and other junk.

That polluted the memory system.

I had to clean the database and fix the script so it ignored the Khoj app folder.

The third mistake was editing a Python file too quickly.

At one point, I broke the indentation in a Khoj file. Python is strict about spacing, so one wrong indent caused the whole app to fail.

That was frustrating, but it taught me something important.

When fixing local software, small edits matter. One missing parenthesis or one bad indent can break everything.

What I Learned

I learned a lot from this install.

I learned that “local AI” is not always simple. Even when an app works, the setup can depend on your computer, your operating system, your Python version, your database, and your hardware.

I learned more about PostgreSQL.

I had to deal with users, passwords, login rules, extensions, migrations, and database cleanup. I did not plan to learn all of that. The install forced me to.

I learned that a second brain is about more than the AI tool.

The folder structure matters.

The file format matters.

The sync process matters.

The way I write my notes matters.

If the notes are messy, the AI will have less to work with. If the folders are clear, the AI has a better chance of giving useful answers.

I also learned that testing with one small file first is the right move.

Before uploading personal trauma or private journals, I tested Khoj with a simple phrase. The phrase was “silver bridge.” When Khoj could answer from that test journal, I knew the system was working.

That small test saved me from making bigger mistakes with sensitive files.

The System Works Now

After all the errors, Khoj finally worked.

I created a small personal memory folder with files for these categories.

• Journals
• Life events
• Emotional patterns
• Important people
• Therapy reflections
• Core themes

Then I synced those files into Khoj.

I asked Khoj what files were in my personal memory system, and it answered correctly.

That was the first real win.

It meant the setup was finally working.

Why This Was Worth It

The process was long and annoying.

But it was worth doing.

I learned more by fighting through the install than I would have learned by only watching a tutorial.

I now understand the pieces better.

I know Khoj needs a database. I know how files get indexed. I know the difference between installing an app and building a real workflow around it.

That matters because using these tools is not the point. Understanding them is.

I want hands-on experience with AI systems, second brains, local tools, agents, and personal knowledge workflows.

This was a messy install, but it gave me real experience.

What I Will Try Next

Next, I want to give Odysseus a try.

Khoj worked, but I still want to compare other systems.

I want to see which one is better for my personal memory project. I want to know which one is easier to use, which one feels more natural, and which one gives better answers from my own life notes.

I do not see this as wasted time.

Even if I do not stay with Khoj forever, I learned a lot from setting it up.

I plan to keep testing systems like Khoj and Odysseus for the experience of doing it. Each one teaches me something. Each one helps me understand what is possible with local AI and personal knowledge systems.

The goal is about more than finding the perfect tool.

The goal is to learn by building.