# Data Types and Collection Schemas

Design Milvus collection schemas in Node.js using scalar and vector data types, a primary key, and optional dynamic fields.

SDK: `@zilliz/milvus2-sdk-node@3.0.10` · Tested with Milvus 3.0.6

Canonical documentation: https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/

## Section links

- [Supported Data Types](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#supported-data-types)
- [Scalar Types](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#scalar-types)
- [Integer Types](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#integer-types)
- [Floating Point Types](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#floating-point-types)
- [Boolean Type](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#boolean-type)
- [String Types](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#string-types)
- [JSON Type](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#json-type)
- [Geometry](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#geometry)
- [Timestamptz](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#timestamptz)
- [Vector Types](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#vector-types)
- [FloatVector](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#floatvector)
- [BinaryVector](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#binaryvector)
- [SparseFloatVector](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#sparsefloatvector)
- [Float16Vector](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#float16vector)
- [BFloat16Vector](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#bfloat16vector)
- [Int8Vector](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#int8vector)
- [Nullable Vector Fields](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#nullable-vector-fields)
- [Complex Types](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#complex-types)
- [Array](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#array)
- [ArrayOfVector](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#arrayofvector)
- [StructArray](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#structarray)
- [Field Schema Definition](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#field-schema-definition)
- [Primary Key Fields](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#primary-key-fields)
- [Collection Schema Creation](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#collection-schema-creation)
- [Basic Schema](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#basic-schema)
- [Schema with Multiple Vector Fields](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#schema-with-multiple-vector-fields)
- [Schema with Consistency Level](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#schema-with-consistency-level)
- [Dynamic Schema](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#dynamic-schema)
- [Schema Validation](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#schema-validation)
- [Schema Best Practices](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#schema-best-practices)
- [Next Steps](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#next-steps)
- [Text](https://milvus-io.github.io/milvus-sdk-node/core-concepts/data-types-schemas/#text)

Milvus supports various data types for storing different kinds of data. Understanding these types is essential for designing effective schemas.

## Supported Data Types

### Scalar Types

#### Integer Types

- **Int8**: 8-bit signed integer (-128 to 127)
- **Int16**: 16-bit signed integer (-32,768 to 32,767)
- **Int32**: 32-bit signed integer
- **Int64**: 64-bit signed integer

```javascript
import { DataType } from '@zilliz/milvus2-sdk-node';

const ageField = {
  name: 'age',
  data_type: DataType.Int64,
};
```

#### Floating Point Types

- **Float**: 32-bit floating point number
- **Double**: 64-bit floating point number

```javascript
{
  name: 'score',
  data_type: DataType.Float,
}
```

#### Boolean Type

- **Bool**: Boolean value (true/false)

```javascript
{
  name: 'is_active',
  data_type: DataType.Bool,
}
```

#### String Types

- **VarChar**: Variable-length string (requires `max_length`)

```javascript
{
  name: 'title',
  data_type: DataType.VarChar,
  max_length: 256,
}
```

#### JSON Type

- **JSON**: JSON object

```javascript
{
  name: 'metadata',
  data_type: DataType.JSON,
}
```

#### Geometry

Geospatial data stored in WKT (Well-Known Text) format.

```javascript
const locationField = {
  name: 'location',
  data_type: DataType.Geometry,
};

// Insert as WKT string
const row = { location: 'POINT(121.47 31.23)' };
```

#### Timestamptz

Timestamp with timezone information.

```javascript
{
  name: 'created_at',
  data_type: DataType.Timestamptz,
}
```

### Vector Types

#### FloatVector

Dense float vector (most common):

```javascript
{
  name: 'embedding',
  data_type: DataType.FloatVector,
  dim: 128, // Required: dimension of the vector
}
```

#### BinaryVector

Binary vector (for binary embeddings):

```javascript
{
  name: 'binary_embedding',
  data_type: DataType.BinaryVector,
  dim: 128, // Must be multiple of 8
}
```

#### SparseFloatVector

Sparse vectors store only non-zero values with their indices. Commonly used for BM25 full-text search and keyword-based retrieval.

```javascript
{
  name: 'sparse_vector',
  data_type: DataType.SparseFloatVector,
}
```

The SDK accepts sparse vectors as dictionaries, dense arrays with missing positions, CSR objects, or COO arrays of objects. For example:

**Dictionary format** (recommended):

```javascript
// Keys are indices, values are float weights
const sparseVector = { 10: 0.5, 100: 0.3, 500: 0.8, 1200: 0.1 };
```

**COO format:**

```javascript
const sparseVector = [
  { index: 10, value: 0.5 },
  { index: 100, value: 0.3 },
  { index: 500, value: 0.8 },
  { index: 1200, value: 0.1 },
];
```

Note: Dimension is determined automatically from the maximum index across all vectors.

#### Float16Vector

16-bit float vector:

```javascript
{
  name: 'f16_embedding',
  data_type: DataType.Float16Vector,
  dim: 128,
}
```

#### BFloat16Vector

BFloat16 vector:

```javascript
{
  name: 'bf16_embedding',
  data_type: DataType.BFloat16Vector,
  dim: 128,
}
```

#### Int8Vector

8-bit integer vector:

```javascript
{
  name: 'int8_embedding',
  data_type: DataType.Int8Vector,
  dim: 128,
}
```

#### Nullable Vector Fields

Vector fields can be marked as nullable. Insert or upsert `null` for rows that do not have an embedding yet; query and search results return `null` for those fields.

```javascript
const optionalEmbedding = {
  name: 'optional_embedding',
  data_type: DataType.FloatVector,
  dim: 128,
  nullable: true,
};

await client.insert({
  collection_name: 'my_collection',
  data: [
    { id: 1, optional_embedding: Array(128).fill(0.1) },
    { id: 2, optional_embedding: null },
  ],
});
```

Nullable vector payloads are supported for `FloatVector`, `BinaryVector`, `Float16Vector`, `BFloat16Vector`, `SparseFloatVector`, and `Int8Vector`.

### Complex Types

#### Array

Array of scalar values:

```javascript
{
  name: 'tags',
  data_type: DataType.Array,
  element_type: DataType.VarChar,
  max_capacity: 100,
  max_length: 128,
}
```

#### ArrayOfVector

Use a vector child field in a StructArray to store multiple vectors per entity. The SDK encodes that child as `ArrayOfVector` when sending the schema to Milvus. Declare each child with its vector type and dimension, as shown below.

#### StructArray

 Milvus 3.0 

A StructArray is an `Array` whose elements share a nested struct schema. Define the child fields with `fields` and set `element_type` to `DataType.Struct`.

```javascript
{
  name: 'features',
  data_type: DataType.Array,
  element_type: DataType.Struct,
  max_capacity: 16,
  nullable: true,
  fields: [
    { name: 'label', data_type: DataType.VarChar, max_length: 64 },
    { name: 'score', data_type: DataType.Int32 },
    { name: 'active', data_type: DataType.Bool },
    { name: 'embedding', data_type: DataType.FloatVector, dim: 128 },
  ],
}
```

Insert arrays of objects that match the child schema:

```javascript
await client.insert({
  collection_name: 'documents',
  data: [
    {
      id: 1,
      features: [
        {
          label: 'title',
          score: 95,
          active: true,
          embedding: Array(128).fill(0.1),
        },
      ],
    },
  ],
});
```

Use bracketed paths for child fields and `element_filter` when predicates must match values from the same array element:

```javascript
const rows = await client.query({
  collection_name: 'documents',
  filter: 'element_filter(features, $[score] >= 90 AND $[active] == true)',
  output_fields: ['id', 'features[label]', 'features[score]'],
});
```

Vector and scalar child fields can be indexed with paths such as `features[embedding]`, `features[score]`, or `features[label]`. Element-level query and search results include an `offset` for the matching StructArray element. Separate StructArray fields may reuse the same child names.

To add a StructArray to an existing collection with `addCollectionField()`, set `nullable: true` so existing rows remain valid. Nested `Array` children inside a StructArray are not currently accepted by the Milvus 3.0 server.

## Field Schema Definition

Each field in a collection schema must include:

```javascript
{
  name: 'field_name',           // Required: field name
  data_type: DataType.Int64,    // Required: data type
  description: 'Field description', // Optional
  is_primary_key: false,        // Optional: primary key flag
  autoID: false,                // Optional: auto-generate IDs
  max_length: 256,              // Required for VarChar
  dim: 128,                     // Required for vector types
  nullable: false,              // Optional: allow null values
  default_value: undefined,     // Optional scalar default value
  external_field: undefined,    // Optional: source field name for external collections
}
```

### Primary Key Fields

Every collection must have exactly one primary key field:

```javascript
{
  name: 'id',
  data_type: DataType.Int64,
  is_primary_key: true,
  autoID: true, // Let Milvus generate IDs automatically
}
```

Or with manual IDs:

```javascript
{
  name: 'user_id',
  data_type: DataType.Int64,
  is_primary_key: true,
  autoID: false, // You provide IDs
}
```

## Collection Schema Creation

### Basic Schema

```javascript
import { MilvusClient, DataType } from '@zilliz/milvus2-sdk-node';

const schema = [
  {
    name: 'id',
    data_type: DataType.Int64,
    is_primary_key: true,
    autoID: true,
  },
  {
    name: 'vector',
    data_type: DataType.FloatVector,
    dim: 128,
  },
  {
    name: 'text',
    data_type: DataType.VarChar,
    max_length: 256,
  },
];

await client.createCollection({
  collection_name: 'my_collection',
  fields: schema,
});
```

### Schema with Multiple Vector Fields

```javascript
const schema = [
  {
    name: 'id',
    data_type: DataType.Int64,
    is_primary_key: true,
    autoID: true,
  },
  {
    name: 'text_vector',
    data_type: DataType.FloatVector,
    dim: 768,
  },
  {
    name: 'image_vector',
    data_type: DataType.FloatVector,
    dim: 512,
  },
  {
    name: 'metadata',
    data_type: DataType.JSON,
  },
];
```

### Schema with Consistency Level

```javascript
await client.createCollection({
  collection_name: 'my_collection',
  fields: schema,
  consistency_level: 'Bounded', // 'Strong', 'Session', 'Bounded', 'Eventually'
});
```

## Dynamic Schema

Enable dynamic schema to add fields without redefining the schema:

```javascript
await client.createCollection({
  collection_name: 'my_collection',
  fields: [
    {
      name: 'id',
      data_type: DataType.Int64,
      is_primary_key: true,
      autoID: true,
    },
    {
      name: 'vector',
      data_type: DataType.FloatVector,
      dim: 128,
    },
  ],
  enable_dynamic_field: true, // Enable dynamic fields
});
```

With dynamic schema enabled, you can insert data with additional fields:

```javascript
await client.insert({
  collection_name: 'my_collection',
  data: [
    {
      vector: [
        /* ... */
      ],
      dynamic_field_1: 'value1', // Automatically added
      dynamic_field_2: 123, // Automatically added
    },
  ],
});
```

## Schema Validation

The SDK validates schemas before creating collections. Common validation errors:

- Missing primary key field
- Multiple primary key fields
- Missing dimension for vector fields
- Missing max_length for VarChar fields
- Invalid dimension for BinaryVector (must be multiple of 8)

## Schema Best Practices

1. **Choose appropriate data types**: Use Int64 for IDs, FloatVector for embeddings
2. **Set reasonable max_length**: For VarChar fields, set max_length based on expected content
3. **Use autoID**: Let Milvus generate IDs unless you have specific requirements
4. **Enable dynamic schema**: For flexible schemas that may change over time
5. **Document fields**: Use descriptions to document field purposes

## Next Steps

- Learn about [Collection Management](https://milvus-io.github.io/milvus-sdk-node/management/collection-management/)
- Explore [Data Operations](https://milvus-io.github.io/milvus-sdk-node/operations/data-operations-insert/)
- Check out [Best Practices](https://milvus-io.github.io/milvus-sdk-node/advanced/best-practices/)

## Text

SDK 3.0.4 adds `DataType.Text` (25) for variable-length text without a `max_length` setting. It requires a Milvus deployment using **StorageV3**. Use VarChar for primary/partition keys and array string elements; Text cannot be used for these roles. Do not set `max_length` on Text fields.

```typescript
const contentField = {
  name: 'content',
  data_type: DataType.Text,
  nullable: true,
  enable_analyzer: true,
  enable_match: true,
  analyzer_params: { tokenizer: 'standard' },
};
```

Add this field alongside a primary key and vector field in `createCollection({ fields })`. Insert and upsert ordinary JavaScript strings (or `null` for nullable fields), and include `content` in `output_fields` to retrieve it. Text supports analyzer-based matching and BM25 input. Request and server resource limits still apply even though the schema has no VarChar length limit.
