> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/moeru-ai/airi/llms.txt
> Use this file to discover all available pages before exploring further.

# Audio Package

> Audio processing and encoding utilities for AIRI

## Overview

The `@proj-airi/audio` package provides audio processing utilities including AudioContext management, audio encoding/decoding, and worklet-based audio processing.

## Installation

```bash theme={null}
npm install @proj-airi/audio
```

## Package Structure

The package provides three main export paths:

```typescript theme={null}
import * as audio from '@proj-airi/audio'
import * as audioContext from '@proj-airi/audio/audio-context'
import * as encoding from '@proj-airi/audio/encoding'
```

## Audio Context Management

Manage Web Audio API context with automatic initialization and cleanup.

### initializeAudioContext()

Initialize or get the shared AudioContext.

```typescript theme={null}
import { initializeAudioContext } from '@proj-airi/audio/audio-context'

const context = await initializeAudioContext(48000)
console.log('Sample rate:', context.sampleRate)
```

<ParamField path="requestedSampleRate" type="number" optional default="48000">
  Desired sample rate in Hz. The actual sample rate will be at least 48000Hz for high quality.
</ParamField>

<ResponseField name="context" type="AudioContext">
  The initialized AudioContext instance
</ResponseField>

### Audio Context State

Get current audio context state:

```typescript theme={null}
import {
  getAudioContextState,
  isAudioContextReady,
  getAudioContext,
  getCurrentTime
} from '@proj-airi/audio/audio-context'

const state = getAudioContextState()
// {
//   isReady: boolean
//   sampleRate: number
//   error: string
//   isInitializing: boolean
//   workletLoaded: boolean
//   currentTime: number
//   state: AudioContextState
// }

if (isAudioContextReady()) {
  const context = getAudioContext()
  const time = getCurrentTime()
}
```

### State Interface

```typescript theme={null}
interface State {
  isReady: boolean                    // Context is initialized and ready
  sampleRate: number                  // Actual sample rate
  error: string                       // Error message if initialization failed
  isInitializing: boolean             // Currently initializing
  workletLoaded: boolean              // Audio worklets are loaded
  currentTime: number                 // Current audio time
  state: AudioContextState            // Native context state
}
```

### Subscribe to State Changes

```typescript theme={null}
import { subscribeToAudioContext } from '@proj-airi/audio/audio-context'

const unsubscribe = subscribeToAudioContext((state) => {
  console.log('Audio context state:', state)
})

// Later: clean up subscription
unsubscribe()
```

## Audio Node Creation

### createAudioSource()

Create an audio source from a MediaStream.

```typescript theme={null}
import { createAudioSource } from '@proj-airi/audio/audio-context'

const stream = await navigator.mediaDevices.getUserMedia({ audio: true })
const source = createAudioSource(stream)
```

<ParamField path="mediaStream" type="MediaStream" required>
  MediaStream to create source from
</ParamField>

<ResponseField name="source" type="MediaStreamAudioSourceNode">
  Created audio source node
</ResponseField>

### createAudioAnalyser()

Create an analyser node for audio visualization.

```typescript theme={null}
import { createAudioAnalyser } from '@proj-airi/audio/audio-context'

const analyser = createAudioAnalyser({
  fftSize: 2048,
  smoothingTimeConstant: 0.8,
  minDecibels: -90,
  maxDecibels: -10
})

// Connect to source
source.connect(analyser)

// Get frequency data
const dataArray = new Uint8Array(analyser.frequencyBinCount)
analyser.getByteFrequencyData(dataArray)
```

<ParamField path="options.fftSize" type="number" optional>
  FFT size for frequency analysis (power of 2)
</ParamField>

<ParamField path="options.smoothingTimeConstant" type="number" optional>
  Smoothing between frames (0-1)
</ParamField>

<ParamField path="options.minDecibels" type="number" optional>
  Minimum decibel value for FFT scaling
</ParamField>

<ParamField path="options.maxDecibels" type="number" optional>
  Maximum decibel value for FFT scaling
</ParamField>

### createAudioGainNode()

Create a gain node for volume control.

```typescript theme={null}
import { createAudioGainNode } from '@proj-airi/audio/audio-context'

const gainNode = createAudioGainNode(0.5)  // 50% volume

// Connect source -> gain -> destination
source.connect(gainNode)
gainNode.connect(context.destination)

// Change volume
gainNode.gain.value = 0.8
```

<ParamField path="initialGain" type="number" optional default="1">
  Initial gain value (0-1)
</ParamField>

## Audio Resampling

### createResamplingWorkletNode()

Create a worklet node for real-time audio resampling.

```typescript theme={null}
import { createResamplingWorkletNode } from '@proj-airi/audio/audio-context'

const workletNode = createResamplingWorkletNode(source, {
  inputSampleRate: 48000,
  outputSampleRate: 16000,
  channels: 1,
  converterType: 2,  // SRC_SINC_MEDIUM_QUALITY
  bufferSize: 4096
})

workletNode.port.onmessage = (event) => {
  const { audio } = event.data
  // Process resampled audio
}
```

<ParamField path="inputNode" type="AudioNode" required>
  Source audio node to resample
</ParamField>

<ParamField path="options.inputSampleRate" type="number" optional default="48000">
  Input sample rate in Hz
</ParamField>

<ParamField path="options.outputSampleRate" type="number" optional default="16000">
  Output sample rate in Hz
</ParamField>

<ParamField path="options.channels" type="number" optional default="1">
  Number of audio channels
</ParamField>

<ParamField path="options.converterType" type="number" optional default="2">
  Resampling quality (0-4, higher is better)
</ParamField>

<ParamField path="options.bufferSize" type="number" optional default="4096">
  Processing buffer size
</ParamField>

## Audio Encoding

### WAV Encoding

Encode audio data to WAV format.

```typescript theme={null}
import { toWav, toWAVBase64 } from '@proj-airi/audio/encoding'

// Encode to WAV ArrayBuffer
const wavBuffer = toWav(audioData, 16000, 1)

// Encode to base64 WAV
const base64Wav = toWAVBase64(audioData, 16000)

// Use as data URL
const audio = new Audio(`data:audio/wav;base64,${base64Wav}`)
audio.play()
```

<ParamField path="buffer" type="ArrayBufferLike" required>
  Float32 audio samples
</ParamField>

<ParamField path="sampleRate" type="number" required>
  Sample rate in Hz
</ParamField>

<ParamField path="channels" type="number" optional default="1">
  Number of audio channels
</ParamField>

<ResponseField name="wavBuffer" type="ArrayBuffer">
  Complete WAV file as ArrayBuffer
</ResponseField>

<ResponseField name="base64Wav" type="string">
  Base64-encoded WAV file
</ResponseField>

## Resource Management

### Node Cleanup

Remove nodes when done:

```typescript theme={null}
import {
  removeAudioSource,
  removeAudioGainNode,
  removeAudioAnalyser,
  removeWorkletNode
} from '@proj-airi/audio/audio-context'

// Clean up nodes
removeAudioSource(source)
removeAudioGainNode(gainNode)
removeAudioAnalyser(analyser)
removeWorkletNode(workletNode)
```

### Context Suspension

Suspend and resume audio context:

```typescript theme={null}
import {
  suspendAudioContext,
  resumeAudioContext
} from '@proj-airi/audio/audio-context'

// Suspend to save CPU
await suspendAudioContext()

// Resume when needed
await resumeAudioContext()
```

### Complete Cleanup

Clean up all resources:

```typescript theme={null}
import { cleanupAudioContext } from '@proj-airi/audio/audio-context'

// Clean up all nodes and close context
await cleanupAudioContext()
```

## Complete Example

Microphone recording with resampling:

```typescript theme={null}
import {
  initializeAudioContext,
  createAudioSource,
  createResamplingWorkletNode,
  removeAudioSource,
  removeWorkletNode,
  cleanupAudioContext
} from '@proj-airi/audio/audio-context'
import { toWAVBase64 } from '@proj-airi/audio/encoding'

// Initialize
const context = await initializeAudioContext(48000)

// Get microphone access
const stream = await navigator.mediaDevices.getUserMedia({
  audio: {
    sampleRate: 48000,
    channelCount: 1,
    echoCancellation: true,
    noiseSuppression: true
  }
})

// Create audio pipeline
const source = createAudioSource(stream)
const worklet = createResamplingWorkletNode(source, {
  inputSampleRate: 48000,
  outputSampleRate: 16000,
  channels: 1
})

// Collect audio chunks
const chunks: Float32Array[] = []

worklet.port.onmessage = (event) => {
  if (event.data.audio) {
    chunks.push(new Float32Array(event.data.audio))
  }
}

// Start recording
console.log('Recording...')

// Stop after 5 seconds
setTimeout(async () => {
  // Stop recording
  stream.getTracks().forEach(track => track.stop())
  
  // Combine chunks
  const totalLength = chunks.reduce((sum, chunk) => sum + chunk.length, 0)
  const combined = new Float32Array(totalLength)
  let offset = 0
  for (const chunk of chunks) {
    combined.set(chunk, offset)
    offset += chunk.length
  }
  
  // Encode to WAV
  const wav = toWAVBase64(combined.buffer, 16000)
  console.log('Recording complete:', wav.length, 'bytes')
  
  // Clean up
  removeAudioSource(source)
  removeWorkletNode(worklet)
  await cleanupAudioContext()
}, 5000)
```

## Best Practices

<AccordionGroup>
  <Accordion title="Initialize early">
    Initialize AudioContext early in your app, preferably on user interaction due to browser autoplay policies.
  </Accordion>

  <Accordion title="Use high sample rates">
    Use 48kHz or higher for input to ensure quality. Resample down as needed for processing.
  </Accordion>

  <Accordion title="Clean up resources">
    Always remove nodes and clean up the context when done to prevent memory leaks.
  </Accordion>

  <Accordion title="Handle errors">
    Wrap audio operations in try-catch blocks and check the error state.
  </Accordion>

  <Accordion title="Suspend when inactive">
    Suspend the audio context when not actively processing to save CPU.
  </Accordion>
</AccordionGroup>

## Browser Compatibility

The package uses Web Audio API features:

* AudioContext
* AudioWorklet
* MediaStream

Supported in all modern browsers:

* Chrome/Edge 66+
* Firefox 76+
* Safari 14.1+

## Next Steps

<CardGroup cols={2}>
  <Card title="Stage UI Package" icon="window" href="/api/packages/stage-ui">
    Integrate audio with UI components
  </Card>

  <Card title="Server SDK" icon="server" href="/api/packages/server-sdk">
    Stream audio to server runtime
  </Card>
</CardGroup>