Memory System

Persistent storage and retrieval of conversation history and user data using IndexedDB and unstorage

Overview

Airi uses a multi-tier storage system for persisting conversation history, user preferences, and application data. The system leverages browser storage APIs including IndexedDB for structured data and localStorage for simple key-value pairs.

Storage Architecture

Airi’s storage system is built on unstorage, a universal storage layer that supports multiple drivers:

• Memory: Fast in-memory cache (default)
• IndexedDB: Browser-based structured storage for conversations and data
• LocalStorage: Key-value pairs for settings and preferences

import { storage } from '@proj-airi/stage-ui/database/storage'

// Storage structure:
// - memory: In-memory cache (base driver)
// - local: IndexedDB for general data
// - outbox: IndexedDB for sync queue

Storage Configuration

The storage system is pre-configured with multiple mount points:

import { createStorage } from 'unstorage'
import indexedDbDriver from 'unstorage/drivers/indexedb'
import memoryDriver from 'unstorage/drivers/memory'

export const storage = createStorage({
  driver: memoryDriver()  // Base driver for fast access
})

// Mount IndexedDB for persistent storage
storage.mount('local', indexedDbDriver({ base: 'airi-local' }))

// Mount sync queue for background operations
storage.mount('outbox', indexedDbDriver({ base: 'airi-sync-queue' }))

Using Storage

Basic Operations

import { storage } from '@proj-airi/stage-ui/database/storage'

// Store data
await storage.setItem('local:user-preference', {
  theme: 'dark',
  language: 'en'
})

// Retrieve data
const preference = await storage.getItem('local:user-preference')
console.log(preference)
// { theme: 'dark', language: 'en' }

// Check if key exists
const exists = await storage.hasItem('local:user-preference')

// Remove data
await storage.removeItem('local:user-preference')

// Clear all data in a namespace
await storage.clear('local')

List Keys

// List all keys in a mount point
const keys = await storage.getKeys('local')
console.log(keys)
// ['user-preference', 'conversation-history', ...]

// List keys with prefix
const conversationKeys = await storage.getKeys('local:conversation')

Batch Operations

// Get multiple items
const items = await Promise.all([
  storage.getItem('local:key1'),
  storage.getItem('local:key2'),
  storage.getItem('local:key3')
])

// Set multiple items
await Promise.all([
  storage.setItem('local:key1', value1),
  storage.setItem('local:key2', value2),
  storage.setItem('local:key3', value3)
])

LocalStorage Management

For reactive localStorage with automatic serialization:

import { useLocalStorageManualReset } from '@proj-airi/stage-shared/composables'

// Create reactive localStorage ref
const settings = useLocalStorageManualReset('app-settings', {
  theme: 'dark',
  fontSize: 14,
  notifications: true
})

// Access value
console.log(settings.value.theme)

// Update value (automatically persisted)
settings.value.theme = 'light'

// Reset to default
settings.reset()

Versioned Storage

import { useVersionedLocalStorage } from '@proj-airi/stage-ui/composables/use-versioned-local-storage'

// Store data with version tracking
const data = useVersionedLocalStorage('my-data', {
  version: 2,
  data: {
    name: 'Example',
    count: 0
  }
}, {
  // Migration function for version upgrades
  migrate: (oldData, oldVersion) => {
    if (oldVersion === 1) {
      // Migrate from version 1 to 2
      return {
        version: 2,
        data: {
          ...oldData,
          count: oldData.count || 0
        }
      }
    }
    return oldData
  }
})

// Access versioned data
console.log(data.value.data.name)

Conversation Storage

While Airi doesn’t use DuckDB WASM for conversation storage (that dependency is listed but not actively used), conversations are stored in IndexedDB through the storage layer.

Storing Conversations

import { storage } from '@proj-airi/stage-ui/database/storage'
import { nanoid } from 'nanoid'

interface Message {
  id: string
  role: 'user' | 'assistant' | 'system'
  content: string
  timestamp: number
}

interface Conversation {
  id: string
  title: string
  messages: Message[]
  createdAt: number
  updatedAt: number
}

// Create conversation
const conversation: Conversation = {
  id: nanoid(),
  title: 'New Chat',
  messages: [],
  createdAt: Date.now(),
  updatedAt: Date.now()
}

await storage.setItem(`local:conversation:${conversation.id}`, conversation)

// Add message to conversation
conversation.messages.push({
  id: nanoid(),
  role: 'user',
  content: 'Hello!',
  timestamp: Date.now()
})

conversation.updatedAt = Date.now()
await storage.setItem(`local:conversation:${conversation.id}`, conversation)

Retrieving Conversations

// Get single conversation
const conversationId = 'abc123'
const conversation = await storage.getItem<Conversation>(
  `local:conversation:${conversationId}`
)

// List all conversations
const keys = await storage.getKeys('local:conversation')
const conversations = await Promise.all(
  keys.map(key => storage.getItem<Conversation>(`local:${key}`))
)

// Sort by most recent
conversations.sort((a, b) => b.updatedAt - a.updatedAt)

Searching Conversations

// Search by text content
async function searchConversations(query: string): Promise<Conversation[]> {
  const keys = await storage.getKeys('local:conversation')
  const conversations = await Promise.all(
    keys.map(key => storage.getItem<Conversation>(`local:${key}`))
  )
  
  return conversations.filter(conv => 
    conv.title.toLowerCase().includes(query.toLowerCase()) ||
    conv.messages.some(msg => 
      msg.content.toLowerCase().includes(query.toLowerCase())
    )
  )
}

const results = await searchConversations('quantum computing')

Deleting Conversations

// Delete single conversation
await storage.removeItem(`local:conversation:${conversationId}`)

// Delete all conversations
const keys = await storage.getKeys('local:conversation')
await Promise.all(
  keys.map(key => storage.removeItem(`local:${key}`))
)

Sync Queue (Outbox Pattern)

The outbox mount point is used for queuing operations that need to sync:

import { storage } from '@proj-airi/stage-ui/database/storage'
import { nanoid } from 'nanoid'

interface SyncOperation {
  id: string
  type: 'create' | 'update' | 'delete'
  collection: string
  data: any
  createdAt: number
  retries: number
}

// Queue an operation
const operation: SyncOperation = {
  id: nanoid(),
  type: 'create',
  collection: 'conversations',
  data: conversation,
  createdAt: Date.now(),
  retries: 0
}

await storage.setItem(`outbox:${operation.id}`, operation)

// Process outbox
async function processOutbox() {
  const keys = await storage.getKeys('outbox')
  
  for (const key of keys) {
    const operation = await storage.getItem<SyncOperation>(`outbox:${key}`)
    
    try {
      // Sync to server
      await syncToServer(operation)
      
      // Remove from outbox on success
      await storage.removeItem(`outbox:${key}`)
    } catch (error) {
      // Increment retry count
      operation.retries++
      await storage.setItem(`outbox:${key}`, operation)
    }
  }
}

Data Export/Import

Export All Data

async function exportAllData() {
  const keys = await storage.getKeys('local')
  const data: Record<string, any> = {}
  
  for (const key of keys) {
    data[key] = await storage.getItem(`local:${key}`)
  }
  
  // Create download
  const blob = new Blob([JSON.stringify(data, null, 2)], {
    type: 'application/json'
  })
  
  const url = URL.createObjectURL(blob)
  const a = document.createElement('a')
  a.href = url
  a.download = `airi-backup-${Date.now()}.json`
  a.click()
  URL.revokeObjectURL(url)
}

Import Data

async function importData(file: File) {
  const text = await file.text()
  const data = JSON.parse(text)
  
  // Restore all data
  for (const [key, value] of Object.entries(data)) {
    await storage.setItem(`local:${key}`, value)
  }
}

Storage Monitoring

Check Storage Usage

if ('storage' in navigator && 'estimate' in navigator.storage) {
  const estimate = await navigator.storage.estimate()
  const usageInMB = (estimate.usage || 0) / (1024 * 1024)
  const quotaInMB = (estimate.quota || 0) / (1024 * 1024)
  
  console.log(`Storage used: ${usageInMB.toFixed(2)} MB`)
  console.log(`Storage quota: ${quotaInMB.toFixed(2)} MB`)
  console.log(`Percentage used: ${((estimate.usage || 0) / (estimate.quota || 1) * 100).toFixed(1)}%`)
}

Storage Events

// Listen for storage quota warnings
if ('storage' in navigator && 'persist' in navigator.storage) {
  const isPersisted = await navigator.storage.persist()
  console.log(`Storage persisted: ${isPersisted}`)
}

// Request persistent storage
await navigator.storage.persist()

Data Maintenance

Cleanup Old Data

import { useDataMaintenance } from '@proj-airi/stage-ui/composables/use-data-maintenance'

const maintenance = useDataMaintenance()

// Remove conversations older than 90 days
async function cleanupOldConversations(maxAgeDays: number = 90) {
  const keys = await storage.getKeys('local:conversation')
  const cutoff = Date.now() - (maxAgeDays * 24 * 60 * 60 * 1000)
  
  for (const key of keys) {
    const conversation = await storage.getItem<Conversation>(`local:${key}`)
    
    if (conversation && conversation.updatedAt < cutoff) {
      await storage.removeItem(`local:${key}`)
      console.log(`Deleted old conversation: ${conversation.id}`)
    }
  }
}

await cleanupOldConversations()

Vacuum/Optimize

// Clear memory cache
await storage.clear()  // Only clears memory driver, mounts are preserved

// Reload critical data
const criticalKeys = ['user-preference', 'active-conversation']
for (const key of criticalKeys) {
  const value = await storage.getItem(`local:${key}`)
  await storage.setItem(key, value)  // Reload into memory cache
}

Best Practices

1. Use Appropriate Storage: Use IndexedDB for large/structured data, localStorage for simple settings
2. Namespace Keys: Always prefix keys with a namespace (e.g., ‘conversation:’, ‘settings:’)
3. Handle Errors: Storage operations can fail (quota exceeded, permissions, etc.)
4. Implement Versioning: Use versioned storage for data that may need migration
5. Regular Cleanup: Implement periodic cleanup of old/unused data
6. Batch Operations: Group multiple operations to reduce IndexedDB transactions
7. Cache Strategically: Use memory cache for frequently-accessed data
8. Monitor Quota: Check storage usage and warn users when approaching limits

Troubleshooting

Storage Quota Exceeded

try {
  await storage.setItem('local:data', largeData)
} catch (error) {
  if (error.name === 'QuotaExceededError') {
    console.error('Storage quota exceeded')
    // Cleanup old data
    await cleanupOldConversations(30)  // More aggressive cleanup
  }
}

IndexedDB Blocked

// Some browsers block IndexedDB in private/incognito mode
try {
  const testKey = 'test-key'
  await storage.setItem(`local:${testKey}`, 'test')
  await storage.removeItem(`local:${testKey}`)
} catch (error) {
  console.error('IndexedDB unavailable, falling back to memory storage')
  // Use memory-only storage
}

Data Corruption

// Validate data on read
async function getConversationSafe(id: string): Promise<Conversation | null> {
  try {
    const conversation = await storage.getItem<Conversation>(
      `local:conversation:${id}`
    )
    
    // Validate structure
    if (!conversation || !conversation.id || !Array.isArray(conversation.messages)) {
      console.error('Invalid conversation data')
      return null
    }
    
    return conversation
  } catch (error) {
    console.error('Failed to read conversation:', error)
    return null
  }
}