​

OpenClaw Long-Term Memory System — Full Research

Research date: 2026-03-25 Purpose: Understand openclaw memory architecture for porting to nanobot

​

Table of Contents

1. Architecture Overview
2. Two Plugin Systems
3. Core Memory Module (src/memory/)
4. Memory Tools Exposed to Agent
5. System Prompt Integration
6. Indexing & Sync Pipeline
7. Search Flow (Hybrid BM25 + Vector)
8. Conversation History Persistence
9. Session Memory Hook (Dated Files)
10. Pre-Compaction Memory Flush
11. LanceDB Plugin (Independent Long-Term Memory)
12. Configuration Reference
13. Data Flow Summary
14. Key Files Map
15. Porting Considerations for Nanobot

​

1. Architecture Overview

┌──────────────────────────────────────────────────────────────┐
│                     Agent Loop                                │
│                                                               │
│  ┌─────────────┐   ┌──────────────┐   ┌───────────────────┐ │
│  │ System Prompt│   │ memory_search│   │  memory_get       │ │
│  │ "## Memory  │   │ (tool call)  │   │  (tool call)      │ │
│  │  Recall"    │   └──────┬───────┘   └────────┬──────────┘ │
│  └──────┬──────┘          │                     │            │
│         │                 ▼                     ▼            │
│         │     ┌───────────────────────────────────┐         │
│         │     │     MemoryIndexManager (SQLite)    │         │
│         │     │  ┌─────────┐  ┌────────────────┐  │         │
│         │     │  │ FTS5    │  │ sqlite-vec     │  │         │
│         │     │  │ (BM25)  │  │ (cosine dist)  │  │         │
│         │     │  └─────────┘  └────────────────┘  │         │
│         │     │  ┌─────────────────────────────┐  │         │
│         │     │  │ Embedding Provider (auto)   │  │         │
│         │     │  │ openai/gemini/voyage/local  │  │         │
│         │     │  └─────────────────────────────┘  │         │
│         │     └───────────────────────────────────┘         │
│         │                      ▲                             │
│         │                      │ indexes                     │
│         │     ┌────────────────┴──────────────────┐         │
│         │     │         Source Files               │         │
│         │     │  MEMORY.md                         │         │
│         │     │  memory/*.md (dated files)         │         │
│         │     │  sessions/*.jsonl (experimental)   │         │
│         │     └───────────────────────────────────┘         │
│         │                      ▲                             │
│         │                      │ writes                      │
│         │     ┌────────────────┴──────────────────┐         │
│         │     │      Memory Writers                │         │
│         │     │  • Agent (direct file writes)      │         │
│         │     │  • Session-memory hook (/new)       │         │
│         │     │  • Pre-compaction flush             │         │
│         │     └───────────────────────────────────┘         │
└──────────────────────────────────────────────────────────────┘

​

2. Two Plugin Systems

​

2a. memory-core (default, built-in)

• Provides memory_search and memory_get tools
• Uses SQLite + FTS5 + sqlite-vec for indexing and search
• Indexes MEMORY.md + memory/*.md files
• Hybrid BM25 + vector search
• Multiple embedding providers with auto-selection
• Entry point: extensions/memory-core/index.ts

​

2b. memory-lancedb (optional, install-on-demand)

• Completely independent vector DB system using LanceDB
• Provides memory_recall, memory_store, memory_forget tools
• Auto-recall: injects relevant memories before agent starts
• Auto-capture: analyzes user messages after agent ends, stores important facts
• Uses OpenAI embeddings (text-embedding-3-small)
• DB stored at ~/.openclaw/memory/lancedb
• Entry point: extensions/memory-lancedb/index.ts

​

Plugin selection

plugins:
  slots:
    memory: "memory-core"     # default
    # memory: "memory-lancedb"  # alternative
    # memory: "none"            # disable

​

3. Core Memory Module

​

Class hierarchy

MemoryManagerSyncOps (abstract)
  → MemoryManagerEmbeddingOps (abstract)
    → MemoryIndexManager (concrete, implements MemorySearchManager)

​

Storage: SQLite

​

Memory file discovery

1. MEMORY.md or memory.md in workspace root
2. All *.md files recursively under memory/ directory
3. Additional paths from memorySearch.extraPaths config
4. Session JSONL files (if experimental.sessionMemory: true)

​

Document chunking

• Splits by lines, groups into ~400-token chunks with 80-token overlap
• Each chunk: startLine, endLine, text, hash (SHA-256)
• Token estimate: characters / 4

​

Embedding providers

1. Try local model if modelPath configured
2. Try remote in order: openai → gemini → voyage → mistral
3. If all fail (missing keys) → FTS-only mode (keyword search, no vectors)

​

4. Memory Tools Exposed to Agent

​

memory_search (from memory-core)

• Agent calls it with a text query
• Returns matching chunks with file path, line numbers, score, text
• Searches MEMORY.md, memory/*.md, optionally session transcripts

​

memory_get (from memory-core)

• Parameters: file path, from line, lines count
• Returns the requested lines from a memory file
• Used for targeted follow-up reads after search

​

memory_recall / memory_store / memory_forget (from memory-lancedb)

​

5. System Prompt Integration

• Tells the agent about memory_search and memory_get tools
• Instructs the agent to search memory before answering recall-dependent questions
• Controls citation mode (on/off/auto) for file paths and line numbers in replies
• Only appears if the tools are in availableTools

​

6. Indexing & Sync Pipeline

​

runSync() method (in manager-sync-ops.ts)

1. Memory files sync: List all MEMORY.md + memory/*.md, compare hashes against DB, re-index changed files
2. Session files sync: List session JSONL files, extract user/assistant messages, build text entries, index them
3. Atomic reindex: If embedding model/provider changes, wipe all chunks and re-index everything
4. Embedding batching: Chunks grouped by byte size (max 8000 tokens/batch), sent to provider in parallel (concurrency: 4)
5. Embedding cache: SHA-256 content hash as cache key — unchanged content is never re-embedded

​

Sync triggers

​

7. Search Flow

​

FTS-only mode (no embedding provider)

1. Extract keywords from query
2. Run BM25 full-text search via FTS5 MATCH
3. Merge and deduplicate

​

Hybrid mode (default)

1. Keyword search: FTS5 BM25 on chunks_fts table
2. Vector search: Embed query → vec_distance_cosine() on chunks_vec (or in-memory cosine fallback)
3. Merge: score = 0.7 * vectorScore + 0.3 * textScore
4. Optional temporal decay: Exponential, configurable half-life (default 30 days, disabled by default)
5. Optional MMR re-ranking: Jaccard similarity diversity, lambda 0.7 (disabled by default)
6. Filter: minScore ≥ 0.35
7. Limit: maxResults = 6

​

8. Conversation History Persistence

​

Session transcripts (JSONL)

• Location: ~/.openclaw/agents/<agentId>/sessions/<SessionId>.jsonl
• Each line: JSON record with type: "message", full message object (role, content, usage)
• Session metadata: ~/.openclaw/agents/<agentId>/sessions/sessions.json

​

Session scoping

• agent:<agentId>:<mainKey> — DM sessions
• agent:<agentId>:<channel>:group:<id> — group sessions
• DM scope modes: main, per-peer, per-channel-peer, per-account-channel-peer

​

Session lifecycle

• Daily reset: at 4 AM local time
• Idle reset: after configurable idle timeout
• Manual reset: /new or /reset commands
• Rotation: when transcript exceeds rotateBytes
• Pruning: session.maintenance.pruneAfter for old sessions

​

Session transcripts as searchable memory (experimental)

• Session JSONL files are parsed, user/assistant messages extracted
• Content is chunked, embedded, and indexed alongside memory files
• Becomes searchable via memory_search

​

9. Session Memory Hook (Dated Files)

​

Files

• src/hooks/bundled/session-memory/handler.ts
• src/hooks/bundled/session-memory/transcript.ts

​

Trigger

​

Process

1. Find the previous session’s transcript JSONL file
2. Read the last N messages (default: 15, configurable)
3. Send messages to LLM to generate a descriptive filename slug (e.g., “api-design”, “vendor-pitch”)
4. Write memory/YYYY-MM-DD-slug.md with:

# Session: 2026-01-16 14:30:00 UTC

- **Session Key**: agent:main:main
- **Session ID**: abc123def456
- **Source**: telegram

## Conversation Summary

user: What about the API design?
assistant: I suggest we use REST with...

​

Transcript reading (transcript.ts)

• Reads JSONL files, parses type: "message" entries
• Extracts user and assistant messages (skips / commands)
• Supports fallback to .reset. rotated transcript files
• Finds previous session files by session ID, topic variants, or most recent

​

10. Pre-Compaction Memory Flush

​

Files

• src/auto-reply/reply/memory-flush.ts
• src/auto-reply/reply/agent-runner-memory.ts

​

Trigger conditions (shouldRunMemoryFlush)

• Session tokens exceed contextWindow - reserveTokensFloor - softThresholdTokens (default soft threshold: 4000 tokens)
• OR transcript file size exceeds forceFlushTranscriptBytes (default: 2MB)
• Has not already flushed for the current compaction cycle

​

Execution

1. Read token usage from session transcript
2. Project next input size
3. Run a dedicated LLM agent turn with system prompt instructing it to write durable memories
4. Target file: memory/YYYY-MM-DD.md (uses user’s timezone)
5. Instructions to LLM: append only, never overwrite, never edit root memory files

​

Default prompt

“Pre-compaction memory flush. Store durable memories only in memory/YYYY-MM-DD.md. If nothing to store, reply with [silent].”

​

Configuration

agents:
  defaults:
    compaction:
      memoryFlush:
        enabled: true
        softThresholdTokens: 4000
        systemPrompt: "..."
        prompt: "..."

​

11. LanceDB Plugin

​

Auto-recall (before_agent_start hook)

1. Embed the incoming user message
2. Search top 3 memories (minScore: 0.3) from LanceDB
3. Inject as <relevant-memories> XML block into context

​

Auto-capture (agent_end hook)

1. Scan user messages for capturable content
2. Rule-based trigger detection:
   • Preference patterns (“I prefer…”, “I like…”)
   • Contact info
   • Decision language
   • Explicit “remember” instructions
3. Category detection: preference, fact, decision, entity, other
4. Duplicate check: skip if >0.95 cosine similarity with existing memory
5. Store up to 3 memories per conversation

​

Safety

• shouldCapture() filters: length 10-500 chars, skip injected memory context, skip system content, skip markdown-heavy output, skip emoji-heavy content
• looksLikePromptInjection() protection
• HTML escaping in memory injection

​

Storage

• LanceDB table "memories" with fields: id, text, vector, importance, category, createdAt
• L2 distance converted to similarity: 1 / (1 + distance)
• DB path: ~/.openclaw/memory/lancedb

​

12. Configuration Reference

​

Core memory search config (agents.defaults.memorySearch)

​

Memory backend config (memory)

memory:
  backend: "builtin"  # or "qmd"
  citations: "auto"   # "on" | "off" | "auto"

​

Plugin config

plugins:
  slots:
    memory: "memory-core"  # or "memory-lancedb" or "none"
  entries:
    memory-lancedb:
      config:
        embedding:
          provider: "openai"
          model: "text-embedding-3-small"
          apiKey: "..."
        autoCapture: true
        autoRecall: true

​

13. Data Flow Summary

​

Write paths (how memories get created)

1. Agent directly edits MEMORY.md or memory/*.md
   └── File watcher detects change → re-index

2. Session-memory hook (on /new or /reset)
   └── Reads last N messages from JSONL transcript
   └── LLM generates slug
   └── Writes memory/YYYY-MM-DD-slug.md → re-index

3. Pre-compaction flush (before context compaction)
   └── Dedicated LLM turn
   └── Appends to memory/YYYY-MM-DD.md → re-index

4. LanceDB auto-capture (if lancedb plugin active)
   └── Rule-based trigger on user messages
   └── Embeds + stores in LanceDB (separate from file-based memory)

​

Read paths (how memories get recalled)

1. Agent calls memory_search tool
   └── MemoryIndexManager.search()
   └── Hybrid BM25 + vector search on SQLite
   └── Returns ranked chunks with file/line references

2. Agent calls memory_get tool
   └── Direct file read with line range

3. LanceDB auto-recall (if lancedb plugin active)
   └── before_agent_start hook
   └── Embed user message → search LanceDB
   └── Inject <relevant-memories> into context

4. System prompt loads MEMORY.md content directly
   └── Always available, no search needed
   └── Only in main private session

​

Persistence across sessions

Session N ends (/new or /reset)
  │
  ├── JSONL transcript already on disk
  │     ~/.openclaw/agents/<id>/sessions/<sessionId>.jsonl
  │
  ├── Session-memory hook fires
  │     → memory/2026-03-25-api-design.md created
  │
  └── Next session starts
        │
        ├── MEMORY.md loaded into system prompt
        ├── memory/*.md files indexed (including new dated file)
        ├── Agent can memory_search for past conversations
        └── (optional) session transcripts indexed if experimental flag on

​

14. Key Files Map

​

Plugin entry points

• extensions/memory-core/index.ts — default memory plugin
• extensions/memory-lancedb/index.ts — LanceDB alternative

​

Core memory engine

• src/memory/manager.ts — MemoryIndexManager (877 lines, main class)
• src/memory/manager-sync-ops.ts — file/session sync operations
• src/memory/manager-embedding-ops.ts — embedding batch operations
• src/memory/manager-search.ts — vector + keyword search implementation
• src/memory/internal.ts — file listing, chunking, cosine similarity
• src/memory/hybrid.ts — hybrid search merge logic
• src/memory/mmr.ts — Maximal Marginal Relevance re-ranking
• src/memory/temporal-decay.ts — recency-aware scoring
• src/memory/memory-schema.ts — SQLite schema definitions
• src/memory/session-files.ts — session transcript indexing
• src/memory/search-manager.ts — factory with QMD fallback
• src/memory/prompt-section.ts — system prompt injection

​

Embedding providers

• src/memory/embeddings.ts — factory + local provider
• src/memory/embeddings-openai.ts
• src/memory/embeddings-gemini.ts
• src/memory/embeddings-voyage.ts
• src/memory/embeddings-mistral.ts
• src/memory/embeddings-ollama.ts

​

Session memory hook

• src/hooks/bundled/session-memory/handler.ts — creates dated memory files
• src/hooks/bundled/session-memory/transcript.ts — reads JSONL transcripts

​

Pre-compaction flush

• src/auto-reply/reply/memory-flush.ts — flush trigger logic
• src/auto-reply/reply/agent-runner-memory.ts — flush execution

​

Configuration

• src/config/zod-schema.agent-runtime.ts — MemorySearchSchema (lines 587-734)
• src/config/types.tools.ts — MemorySearchConfig type
• src/config/types.memory.ts — MemoryConfig, QMD types

​

Documentation

• docs/concepts/memory.md — conceptual overview
• docs/reference/memory-config.md — full config reference (712 lines)
• docs/cli/memory.md — CLI reference

​

15. Porting Considerations for Nanobot

​

What’s worth porting (ranked by impact vs effort)

1. Dated memory files (memory/YYYY-MM-DD-slug.md) — session end hook that summarizes conversation into a dated file. Needs: session end event, LLM call for slug generation, file write.
2. Pre-compaction memory flush — before context compaction, run a dedicated LLM turn to extract durable memories. Needs: token counting, compaction event hook, LLM call.
3. memory_search tool — let the agent search its own memory files. Needs: FTS at minimum, vector search is a bonus.

​

Minimal integration points needed

• Session end event → write dated memory file
• Pre-compaction event → run memory flush LLM turn
• Tool registration → add memory_search / memory_get to agent
• System prompt builder → add memory recall instructions
• File watcher (optional) → re-index on memory file changes

​

What to skip

• QMD backend (experimental, complex)
• Multimodal memory (niche)
• sqlite-vec (FTS-only is a good starting point)
• LanceDB plugin (separate concern, memory-core is sufficient)