# Full-Text Search

Build BM25 full-text search in Milvus by configuring an analyzer and a function that turns text into searchable sparse vectors.

SDK: `@zilliz/milvus2-sdk-node@3.0.10` · Tested with Milvus 3.0.6

Canonical documentation: https://milvus-io.github.io/milvus-sdk-node/advanced/full-text-search/

## Section links

- [How It Works](https://milvus-io.github.io/milvus-sdk-node/advanced/full-text-search/#how-it-works)
- [Testing Analyzers](https://milvus-io.github.io/milvus-sdk-node/advanced/full-text-search/#testing-analyzers)
- [Analyzer Types](https://milvus-io.github.io/milvus-sdk-node/advanced/full-text-search/#analyzer-types)
- [Custom Dictionary Files](https://milvus-io.github.io/milvus-sdk-node/advanced/full-text-search/#custom-dictionary-files)
- [Creating a Full-Text Search Collection](https://milvus-io.github.io/milvus-sdk-node/advanced/full-text-search/#creating-a-full-text-search-collection)
- [Adding Functions to Existing Collections](https://milvus-io.github.io/milvus-sdk-node/advanced/full-text-search/#adding-functions-to-existing-collections)
- [Searching with Full-Text](https://milvus-io.github.io/milvus-sdk-node/advanced/full-text-search/#searching-with-full-text)
- [Highlighting Search Results](https://milvus-io.github.io/milvus-sdk-node/advanced/full-text-search/#highlighting-search-results)
- [Lexical Highlighting](https://milvus-io.github.io/milvus-sdk-node/advanced/full-text-search/#lexical-highlighting)
- [Semantic Highlighting](https://milvus-io.github.io/milvus-sdk-node/advanced/full-text-search/#semantic-highlighting)
- [Highlight Results](https://milvus-io.github.io/milvus-sdk-node/advanced/full-text-search/#highlight-results)
- [Managing Collection Functions](https://milvus-io.github.io/milvus-sdk-node/advanced/full-text-search/#managing-collection-functions)
- [Next Steps](https://milvus-io.github.io/milvus-sdk-node/advanced/full-text-search/#next-steps)

Full-text search enables keyword-based text retrieval using BM25 scoring. Milvus supports this through **collection functions** that automatically tokenize text and convert it to sparse vectors at insert and query time.

## How It Works

1. You define a VarChar field for text and a SparseFloatVector field for the generated vectors
2. You add a BM25 function to the collection that maps the text field to the sparse vector field
3. When you insert text, Milvus automatically tokenizes it and generates sparse vectors
4. When you search, you pass text queries and Milvus automatically converts them to sparse vectors for matching

## Testing Analyzers

Before creating a collection, you can test how an analyzer tokenizes text using `runAnalyzer()`:

```javascript
const result = await client.runAnalyzer({
  text: ['machine learning is great', 'deep learning fundamentals'],
  analyzer_params: {
    type: 'standard',
  },
});

console.log('Results:', result.results);
// Each text input produces an AnalyzerResult with tokens
```

You can also test analyzers that are configured on a collection field, request detailed token data, and include token hashes:

```javascript
const result = await client.runAnalyzer({
  db_name: 'default',
  collection_name: 'articles',
  field_name: 'body',
  analyzer_names: ['english_analyzer'],
  text: ['Running analyzers with offsets and hashes'],
  with_detail: true,
  with_hash: true,
});

result.results[0].tokens.forEach((token) => {
  console.log(token.token, token.start_offset, token.end_offset, token.hash);
});
```

`runAnalyzer()` accepts either `analyzer_params` for an ad-hoc analyzer configuration or collection context fields (`db_name`, `collection_name`, `field_name`, `analyzer_names`) to use analyzers defined in a schema.

### Analyzer Types

| Type | Description |
| --- | --- |
| `standard` | Standard tokenizer with lowercase filter |
| `english` | English-specific tokenizer with stemming |
| `chinese` | Chinese text tokenizer |

### Custom Dictionary Files

 Milvus 3.0 

For analyzers that load a custom dictionary or synonym file, register the server-visible object before referencing it from analyzer parameters:

```javascript
await client.addFileResource({
  name: 'product_dictionary',
  path: 'analyzer-resources/product-dictionary.txt',
});

const resources = await client.listFileResources();
console.log(resources.resources);
```

The file must already be available through the storage configured for Milvus. `addFileResource()` registers metadata; it does not upload a local Node.js file. Remove the registration when it is no longer used:

```javascript
await client.removeFileResource({ name: 'product_dictionary' });
```

## Creating a Full-Text Search Collection

```javascript
import {
  MilvusClient,
  DataType,
  FunctionType,
  IndexType,
  MetricType,
  ConsistencyLevelEnum,
} from '@zilliz/milvus2-sdk-node';

const client = new MilvusClient({ address: 'localhost:19530' });

// 1. Create collection with text and sparse vector fields
await client.createCollection({
  collection_name: 'articles',
  fields: [
    {
      name: 'id',
      data_type: DataType.Int64,
      is_primary_key: true,
      autoID: true,
    },
    {
      name: 'title',
      data_type: DataType.VarChar,
      max_length: 256,
      enable_analyzer: true,
      analyzer_params: { tokenizer: 'standard', filter: ['lowercase'] },
    },
    {
      name: 'body',
      data_type: DataType.VarChar,
      max_length: 10000,
      enable_analyzer: true,
      analyzer_params: { tokenizer: 'standard', filter: ['lowercase'] },
    },
    { name: 'sparse_vector', data_type: DataType.SparseFloatVector },
  ],
  functions: [
    {
      name: 'bm25',
      type: FunctionType.BM25,
      input_field_names: ['body'],
      output_field_names: ['sparse_vector'],
      params: {},
    },
  ],
});
```

## Adding Functions to Existing Collections

You can add the BM25 output field, function, and bound index to an existing collection in one schema alteration:

```javascript
await client.addFunctionField({
  collection_name: 'articles',
  field: {
    name: 'title_sparse',
    data_type: DataType.SparseFloatVector,
    is_function_output: true,
  },
  function: {
    name: 'title_bm25',
    type: FunctionType.BM25,
    input_field_names: ['title'],
    output_field_names: ['title_sparse'],
    params: {},
  },
  index_name: 'title_bm25_index',
  extra_params: {
    index_type: IndexType.SPARSE_INVERTED_INDEX,
    metric_type: MetricType.BM25,
  },
});
```

## Searching with Full-Text

Pass a text string as the search data. Milvus uses the BM25 function to convert it to a sparse vector automatically. The example uses Strong consistency and flushes the inserted documents before searching so that the small test dataset is ready for BM25 retrieval. Avoid flushing after every insert in a production ingestion pipeline.

```javascript
// Insert documents
await client.insert({
  collection_name: 'articles',
  data: [
    {
      title: 'Introduction to Machine Learning',
      body: 'Machine learning is a branch of AI...',
    },
    {
      title: 'Deep Learning Basics',
      body: 'Deep learning uses neural networks...',
    },
  ],
});

// Wait for the example documents to be flushed before searching
await client.flushSync({ collection_names: ['articles'] });

await client.createIndex({
  collection_name: 'articles',
  field_name: 'sparse_vector',
  index_type: 'SPARSE_INVERTED_INDEX',
  metric_type: 'BM25',
});
await client.loadCollectionSync({ collection_name: 'articles' });

// Search by text
const results = await client.search({
  collection_name: 'articles',
  data: ['machine learning algorithms'],
  anns_field: 'sparse_vector',
  metric_type: 'BM25',
  consistency_level: ConsistencyLevelEnum.Strong,
  limit: 10,
  output_fields: ['title', 'body'],
});

console.log('Results:', results.results);
```

## Highlighting Search Results

Use the `highlighter` parameter to highlight matched text fragments in search results.

### Lexical Highlighting

Highlights keyword matches using the text field’s analyzer. Set `highlight_search_text: true` to highlight the BM25 search terms, and explicitly set `metric_type: 'BM25'` on the search request. Creating a BM25 index alone does not supply this request parameter to the highlighter.

In this collection, the BM25 function takes `body` as its input, so search-term highlighting applies to `body`. Including `title` in `output_fields` returns its source text but does not automatically highlight it.

```javascript
const results = await client.search({
  collection_name: 'articles',
  data: ['machine learning'],
  anns_field: 'sparse_vector',
  metric_type: 'BM25',
  consistency_level: ConsistencyLevelEnum.Strong,
  limit: 10,
  output_fields: ['title', 'body'],
  highlighter: {
    type: 0, // HighlightType.Lexical
    highlight_search_text: true,
    pre_tags: ['<em>'],
    post_tags: ['</em>'],
    fragment_size: 100,
    num_of_fragments: 3,
  },
});
```

For the first example document, `results.results[0].highlight.body` contains:

```json
{
  "fragments": ["<em>Machine</em> <em>learning</em> is a branch of AI..."],
  "scores": []
}
```

### Semantic Highlighting

Highlights semantically similar text through a model service. Milvus must have the Zilliz model-service provider endpoint configured on the server, with access to an appropriate highlighting model deployment. The `highlighter` parameter does not configure or start that service.

A standalone Milvus instance without this provider configuration returns an error such as `Create SemanticHighlight failed: zilliz client config error, lost endpoint config`. Use the following example only after configuring the service, and replace `YOUR_HIGHLIGHT_MODEL_DEPLOYMENT_ID` with your deployment ID:

```javascript
const results = await client.search({
  collection_name: 'articles',
  data: ['machine learning'],
  anns_field: 'sparse_vector',
  metric_type: 'BM25',
  consistency_level: ConsistencyLevelEnum.Strong,
  limit: 10,
  output_fields: ['title', 'body'],
  highlighter: {
    type: 1, // HighlightType.Semantic
    model_deployment_id: 'YOUR_HIGHLIGHT_MODEL_DEPLOYMENT_ID',
    queries: ['machine learning'],
    input_fields: ['body'],
    pre_tags: ['<mark>'],
    post_tags: ['</mark>'],
    threshold: 0.5,
  },
});
```

`queries` must contain one string for each search query. Set `highlight_only: true` to request only highlighted fragments from the model service. The source fields requested through `output_fields` are still returned in the search hits.

### Highlight Results

Search hits include highlight data for the fields processed by the highlighter. Each highlighted field contains `fragments` and `scores` arrays. Lexical highlighting returns an empty `scores` array; semantic highlighting can return scores from the model service.

```javascript
results.results.forEach((hit) => {
  console.log('Title:', hit.title);
  console.log('Highlight:', hit.highlight);

  const bodyHighlight = hit.highlight?.body;
  bodyHighlight?.fragments.forEach((fragment, index) => {
    console.log('Fragment:', fragment);
    const score = bodyHighlight.scores[index];
    if (score !== undefined) {
      console.log('Score:', score);
    }
  });
});
```

## Managing Collection Functions

```javascript
// List functions on a collection
const desc = await client.describeCollection({ collection_name: 'articles' });
console.log('Functions:', desc.schema.functions);

// Drop the function, output field, and bound index
await client.dropFunctionField({
  collection_name: 'articles',
  function_name: 'bm25',
});
```

To change a BM25 function on Milvus 3.0.6, drop it and add the replacement with `addFunctionField()`.

## Next Steps

- Learn about [Hybrid Search](https://milvus-io.github.io/milvus-sdk-node/operations/hybrid-search/) for combining full-text with vector search
- Explore [Data Types & Schemas](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/) for SparseFloatVector details
