# Iterators

Read large Milvus result sets in batches with the Node.js query and search iterators, and close iterators when finished.

SDK: `@zilliz/milvus2-sdk-node@3.0.10` · Tested with Milvus 3.0.6

Canonical documentation: https://milvus-io.github.io/milvus-sdk-node/operations/iterators/

## Section links

- [Search Iterator](https://milvus-io.github.io/milvus-sdk-node/operations/iterators/#search-iterator)
- [Basic Usage](https://milvus-io.github.io/milvus-sdk-node/operations/iterators/#basic-usage)
- [Parameters](https://milvus-io.github.io/milvus-sdk-node/operations/iterators/#parameters)
- [Client-Side Filtering](https://milvus-io.github.io/milvus-sdk-node/operations/iterators/#client-side-filtering)
- [Query Iterator](https://milvus-io.github.io/milvus-sdk-node/operations/iterators/#query-iterator)
- [Basic Usage](https://milvus-io.github.io/milvus-sdk-node/operations/iterators/#basic-usage-1)
- [Parameters](https://milvus-io.github.io/milvus-sdk-node/operations/iterators/#parameters-1)
- [End-to-End Example: Full Collection Scan](https://milvus-io.github.io/milvus-sdk-node/operations/iterators/#end-to-end-example-full-collection-scan)
- [Best Practices](https://milvus-io.github.io/milvus-sdk-node/operations/iterators/#best-practices)
- [Next Steps](https://milvus-io.github.io/milvus-sdk-node/operations/iterators/#next-steps)

Iterators allow you to paginate through large result sets that exceed the `topk` limit (16,384). Instead of retrieving all results at once, iterators fetch data in configurable batches.

## Search Iterator

`searchIterator()` performs vector similarity search and returns results in batches.

### Basic Usage

```javascript
const iterator = await client.searchIterator({
  collection_name: 'my_collection',
  data: [0.1, 0.2, 0.3, 0.4],
  batchSize: 100,
  limit: 1000, // total results to return (-1 or omit for no limit)
  output_fields: ['id', 'text'],
  expr: 'age > 25',
});

for await (const batch of iterator) {
  console.log('Batch size:', batch.length);
  // Process each batch
}
```

### Parameters

| Parameter | Type | Required | Description |
| --- | --- | --- | --- |
| `collection_name` | `string` | Yes | Collection to search |
| `data` | `number[]` or `number[][]` | Yes | Search vector(s) |
| `batchSize` | `number` | Yes | Items per batch (max 16384) |
| `limit` | `number` | No | Total results limit (-1 for unlimited) |
| `expr` | `string` | No | Filter expression |
| `output_fields` | `string[]` | No | Fields to return |
| `anns_field` | `string` | No | Vector field name (auto-detected if only one) |
| `params` | `object` | No | Search parameters (e.g., `{ nprobe: 10 }`) |
| `external_filter_fn` | `function` | No | Client-side filter function (see below) |

### Client-Side Filtering

Use `external_filter_fn` to apply additional filtering on the client side after results are returned from the server:

```javascript
const iterator = await client.searchIterator({
  collection_name: 'my_collection',
  data: [0.1, 0.2, 0.3, 0.4],
  batchSize: 100,
  output_fields: ['text'],
  external_filter_fn: (row) => {
    // Only keep results where the text length > 50
    return row.text && row.text.length > 50;
  },
});

for await (const batch of iterator) {
  // All items in batch satisfy the external filter
  console.log(batch);
}
```

## Query Iterator

`queryIterator()` retrieves entities matching a filter expression in batches.

### Basic Usage

```javascript
const iterator = await client.queryIterator({
  collection_name: 'my_collection',
  expr: 'age > 30',
  output_fields: ['id', 'text', 'age'],
  batchSize: 100,
  limit: 5000,
});

for await (const batch of iterator) {
  console.log('Batch:', batch.length);
  // Process each batch of query results
}
```

### Parameters

| Parameter | Type | Required | Description |
| --- | --- | --- | --- |
| `collection_name` | `string` | Yes | Collection to query |
| `expr` | `string` | Yes | Filter expression |
| `batchSize` | `number` | Yes | Items per batch |
| `limit` | `number` | No | Total results limit |
| `output_fields` | `string[]` | No | Fields to return |
| `partition_names` | `string[]` | No | Partitions to query |

## End-to-End Example: Full Collection Scan

```javascript
import { MilvusClient, DataType } from '@zilliz/milvus2-sdk-node';

const client = new MilvusClient({ address: 'localhost:19530' });

// Create and populate a collection
await client.createCollection({
  collection_name: 'iterator_demo',
  consistency_level: 'Strong',
  fields: [
    {
      name: 'id',
      data_type: DataType.Int64,
      is_primary_key: true,
      autoID: true,
    },
    { name: 'category', data_type: DataType.VarChar, max_length: 64 },
    { name: 'vector', data_type: DataType.FloatVector, dim: 4 },
  ],
});

await client.createIndex({
  collection_name: 'iterator_demo',
  field_name: 'vector',
  index_type: 'AUTOINDEX',
  metric_type: 'COSINE',
});

await client.loadCollectionSync({ collection_name: 'iterator_demo' });

// Insert sample data
const data = Array.from({ length: 1000 }, (_, i) => ({
  category: `cat_${i % 10}`,
  vector: Array.from({ length: 4 }, () => Math.random()),
}));
await client.insert({ collection_name: 'iterator_demo', data });

// Scan all entities in category 'cat_5'
let totalCount = 0;
const iterator = await client.queryIterator({
  collection_name: 'iterator_demo',
  expr: 'category == "cat_5"',
  output_fields: ['id', 'category'],
  batchSize: 50,
});

for await (const batch of iterator) {
  totalCount += batch.length;
  console.log(`Fetched ${batch.length} items, total: ${totalCount}`);
}

console.log(`Total matching entities: ${totalCount}`);

// Cleanup
await client.dropCollection({ collection_name: 'iterator_demo' });
await client.closeConnection();
```

## Best Practices

1. **Batch size tuning** — Start with 100-500. Larger batches reduce round trips but increase memory usage per batch. Maximum is 16,384.
2. **Use filters** — Apply server-side filters (`expr`) to reduce data transfer. Use `external_filter_fn` only for logic that can’t be expressed as a Milvus filter.
3. **Output fields** — Only request fields you need to minimize data transfer.
4. **Memory** — For very large scans, process and discard each batch promptly rather than accumulating all results in memory.

## Next Steps

- Learn about [Hybrid Search](https://milvus-io.github.io/milvus-sdk-node/operations/hybrid-search/) for multi-vector search
- Explore [Query & Search](https://milvus-io.github.io/milvus-sdk-node/operations/data-operations-query/) for standard operations
