Elasticsearch patterns -- client setup, index management, search DSL, aggregations, vector search, bulk operations, deep pagination
Quick Guide: Use
@elastic/elasticsearch(v8.x/v9.x) as the TypeScript client. Elasticsearch is near real-time -- documents are NOT searchable immediately after indexing; they become visible after a refresh (default: every 1 second on active indices). You MUST define explicit mappings before indexing -- dynamic mapping infers types from the first document, and mismatched types in later documents cause hard failures you cannot fix without reindexing. Usesearch_after+ Point in Time (PIT) for deep pagination -- NOTfrom/sizebeyond 10,000 hits and NOT the scroll API (deprecated for search). Use thebulkAPI orclient.helpers.bulk()for any batch operation -- never loop individual index calls.
<critical_requirements>
<philosophy> </philosophy> <patterns> </patterns>
All code must follow project conventions in CLAUDE.md (kebab-case, named exports, import ordering, import type, named constants)
(You MUST define explicit index mappings BEFORE indexing documents -- dynamic mapping infers types from the first document, and if a later document sends a different type for the same field, indexing fails with a mapper_parsing_exception that CANNOT be fixed without reindexing into a new index)
(You MUST use the bulk API for batch operations -- looping individual client.index() calls is orders of magnitude slower and can overwhelm the cluster with HTTP connections)
(You MUST NOT use from/size pagination beyond 10,000 results -- Elasticsearch throws Result window is too large by default; use search_after + PIT instead)
(You MUST NOT use refresh: true or refresh: "wait_for" in production request handlers -- forcing a refresh on every write degrades cluster performance; let the default 1-second refresh interval handle it)
</critical_requirements>
Additional resources:
Auto-detection: Elasticsearch, elasticsearch, @elastic/elasticsearch, client.search, client.index, client.bulk, client.indices.create, client.indices.putMapping, dense_vector, knn, search_after, point in time, openPIT, aggregations, aggs, bool query, match query, term query, multi_match, nested query, range query, client.helpers.bulk, client.helpers.scrollSearch, BulkResponse, SearchResponse, MappingProperty
When to use:
Key patterns covered:
When NOT to use:
Elasticsearch is a distributed search and analytics engine built on Apache Lucene. It excels at full-text search, structured queries, aggregations, and vector search at scale. Core principles:
from/size is capped at 10,000 hits by default (index.max_result_window). Deep pagination requires search_after + Point in Time (PIT). The scroll API is deprecated for search use cases.text fields are analyzed (tokenized, lowercased) for full-text search. keyword fields are exact-match only. Getting this wrong means either broken search or broken aggregations/filters.Initialize the client with node URL and authentication. The client supports Elastic Cloud, API keys, basic auth, and bearer tokens.
// Good Example -- Typed client setup with environment validation
import { Client } from "@elastic/elasticsearch";
function createElasticsearchClient(): Client {
const node = process.env.ELASTICSEARCH_URL;
if (!node) {
throw new Error("ELASTICSEARCH_URL environment variable is required");
}
return new Client({
node,
auth: {
apiKey: process.env.ELASTICSEARCH_API_KEY ?? "",
},
});
}
export { createElasticsearchClient };
Why good: Environment variable validation, named export, API key auth (preferred over basic auth in production)
// Bad Example -- Hardcoded credentials
import { Client } from "@elastic/elasticsearch";
const client = new Client({
node: "http://localhost:9200",
auth: { username: "elastic", password: "changeme" },
});
Why bad: Hardcoded node URL and credentials leak in version control, basic auth with default password
See examples/core.md for Elastic Cloud setup, health checks, and child clients.
Always define mappings before indexing. Dynamic mapping infers types from the first document -- if wrong, you must reindex.
// Good Example -- Explicit mappings with text + keyword multi-field
const INDEX_NAME = "products";
await client.indices.create({
index: INDEX_NAME,
settings: {
number_of_replicas: 1,
refresh_interval: "1s",
},
mappings: {
properties: {
name: {
type: "text",
fields: { keyword: { type: "keyword" } },
},
description: { type: "text", analyzer: "standard" },
price: { type: "float" },
categories: { type: "keyword" },
inStock: { type: "boolean" },
createdAt: { type: "date" },
},
},
});
Why good: Explicit types prevent mapping conflicts, text + keyword multi-field allows both full-text search and exact-match filtering/aggregation on name
// Bad Example -- No mappings, relying on dynamic mapping
await client.indices.create({ index: "products" });
await client.index({
index: "products",
document: { price: "29.99" }, // Oops -- "29.99" is a string, mapped as text
});
// All future numeric price documents will fail with mapper_parsing_exception
Why bad: Dynamic mapping infers price as text from the string "29.99", and this mapping is immutable -- all future documents with numeric price will fail
See examples/core.md for analysis settings, custom analyzers, and mapping migration.
The bool query is the workhorse of Elasticsearch. It combines must, should, must_not, and filter clauses.
// Good Example -- Bool query with filter context for exact matches
const MIN_PRICE = 10;
const MAX_PRICE = 100;
const result = await client.search<Product>({
index: INDEX_NAME,
query: {
bool: {
must: [{ match: { description: "wireless headphones" } }],
filter: [
{ range: { price: { gte: MIN_PRICE, lte: MAX_PRICE } } },
{ term: { inStock: true } },
],
},
},
size: 20,
});
// result.hits.hits[0]._source is typed as Product | undefined
Why good: filter context for exact matches (no scoring overhead, cacheable), must for full-text relevance scoring, named constants for range values, typed search with generic
// Bad Example -- Everything in must (no filter context)
const result = await client.search({
index: "products",
query: {
bool: {
must: [
{ match: { description: "wireless headphones" } },
{ range: { price: { gte: 10, lte: 100 } } }, // Wasteful scoring
{ term: { inStock: true } }, // Wasteful scoring
],
},
},
});
Why bad: Range and term queries in must waste CPU on relevance scoring for yes/no conditions; filter context skips scoring and enables Elasticsearch's filter cache
See examples/core.md for multi_match, nested queries, and function_score.
Aggregations compute analytics over search results. terms for category counts, range for bucketing, date_histogram for time series.
// Good Example -- Terms aggregation with sub-aggregation
const AGGREGATION_SIZE = 50;
const result = await client.search({
index: INDEX_NAME,
size: 0, // No hits needed, only aggregations
aggs: {
categories: {
terms: { field: "categories", size: AGGREGATION_SIZE },
aggs: {
avgPrice: { avg: { field: "price" } },
},
},
},
});
// result.aggregations?.categories.buckets -> [{ key: "electronics", doc_count: 42, avgPrice: { value: 89.5 } }]
Why good: size: 0 skips hits when only aggregations are needed (faster), nested sub-aggregation for metrics per bucket, named constant for aggregation size
See examples/aggregations.md for date_histogram, range, nested, and pipeline aggregations.
The bulk API batches multiple index/update/delete operations in a single request. Use client.helpers.bulk() for the best developer experience.
// Good Example -- Bulk helper with async generator
const result = await client.helpers.bulk<Product>({
datasource: products,
onDocument(doc) {
return { index: { _index: INDEX_NAME, _id: doc.productId } };
},
refreshOnCompletion: INDEX_NAME,
});
// result.total, result.successful, result.failed
Why good: Bulk helper handles batching, concurrency, retries, and back-pressure automatically; refreshOnCompletion triggers one refresh at the end instead of per-document
// Bad Example -- Looping individual index calls
for (const product of products) {
await client.index({
index: "products",
document: product,
refresh: true, // Refresh after EVERY document!
});
}
Why bad: N individual HTTP requests instead of 1 bulk request, refresh: true on every document causes N segment refreshes (devastating to cluster performance)
See examples/bulk-operations.md for error handling, update operations, and reindexing.
from/size is limited to 10,000 hits. For deep pagination, use search_after with a Point in Time (PIT) for consistent results.
// Good Example -- search_after with PIT
const PIT_KEEP_ALIVE = "1m";
const pit = await client.openPointInTime({
index: INDEX_NAME,
keep_alive: PIT_KEEP_ALIVE,
});
let searchAfter: Array<string | number> | undefined;
let allHits: Product[] = [];
while (true) {
const result = await client.search<Product>({
pit: { id: pit.id, keep_alive: PIT_KEEP_ALIVE },
sort: [{ createdAt: "desc" }, { _id: "asc" }], // Tiebreaker!
size: 100,
...(searchAfter ? { search_after: searchAfter } : {}),
});
const hits = result.hits.hits;
if (hits.length === 0) break;
allHits = allHits.concat(
hits.filter((h) => h._source !== undefined).map((h) => h._source!),
);
searchAfter = hits[hits.length - 1].sort as Array<string | number>;
}
await client.closePointInTime({ id: pit.id });
Why good: PIT ensures consistent snapshot across pages, tiebreaker _id prevents missing/duplicate documents, keep_alive refreshed on each request
See examples/pagination.md for from/size limits, scroll helpers, and pagination decision framework.
<decision_framework>
What kind of search do I need?
-- Full-text relevance search? -> match / multi_match in must
-- Exact value filtering? -> term / terms / range in filter
-- Combining text + filters? -> bool query (must for text, filter for exact)
-- Fuzzy matching? -> match with fuzziness: "AUTO"
-- Phrase matching? -> match_phrase
-- Complex nested objects? -> nested query with path
-- Vector similarity? -> knn with dense_vector field
-- Text + vector hybrid? -> query + knn in same request
How deep do results go?
-- Under 10,000 total? -> from/size (simplest)
-- Over 10,000 hits? -> search_after + PIT (recommended)
-- Bulk data export? -> client.helpers.scrollSearch() or scrollDocuments()
-- Real-time infinite scroll? -> search_after (no PIT needed for forward-only)
What will I do with this field?
-- Full-text search (tokenized, relevance)? -> text
-- Exact match, filtering, aggregations, sorting? -> keyword
-- Both? -> Multi-field: { type: "text", fields: { keyword: { type: "keyword" } } }
-- Neither (just stored, never queried)? -> { type: "keyword", index: false }
Does relevance scoring matter for this clause?
-- YES (affects result order) -> must
-- NO (binary yes/no filter) -> filter (cached, no scoring overhead)
-- Exclude documents -> must_not (in filter context)
-- Boost if present (optional) -> should with minimum_should_match: 0
</decision_framework>
<red_flags>
High Priority Issues:
mapper_parsing_exception that requires reindexing to fixfrom/size beyond 10,000 results -- Elasticsearch throws Result window is too large; use search_after + PITclient.index() calls instead of client.bulk() or client.helpers.bulk() -- orders of magnitude slower, can overwhelm the clusterrefresh: true or refresh: "wait_for" in production request handlers -- forces a segment refresh on every write, degrades cluster performance under loadMedium Priority Issues:
must instead of filter -- wastes CPU on scoring, misses filter cachetext type for fields that need exact matching or aggregation -- text fields are analyzed (tokenized), making aggregations return individual tokens instead of full valuessort when using search_after -- documents with identical sort values may be skipped or duplicated across pages_source check -- hit._source can be undefined if _source is disabled or fields are excluded; always handle thisGotchas & Edge Cases:
text, you cannot change it to keyword. The only fix is to create a new index with correct mappings and reindex all documentstext vs keyword confusion -- text fields are tokenized ("New York" becomes ["new", "york"]). Aggregating on a text field gives you individual tokens, not full values. Use keyword or a .keyword sub-field for aggregationsmatch vs term on text fields -- term on a text field often returns no results because term does NOT analyze the query but the field value IS analyzed (e.g., term "New York" won't match the analyzed tokens "new" and "york")refresh: "wait_for" or explicit client.indices.refresh()index.max_result_window is 10,000 -- increasing this is possible but NOT recommended; deep pagination with from/size holds all skipped results in memorynested mapping type -- arrays of objects are flattened by default, losing the association between fields within each object. If you need to query "color: red AND size: large" on the same object in an array, use nested_id is disabled by default (8.x+) -- use a separate id field if you need to aggregate by document ID_score is null in filter context -- clauses in filter do not contribute to scoring; if you need scoring, use mustresult.errors and iterate result.items to find failed operationssearch_after + PIT for deep pagination. Scroll is still valid for one-time data export but consumes cluster resources (open search contexts)</red_flags>
<critical_reminders>
All code must follow project conventions in CLAUDE.md (kebab-case, named exports, import ordering,
import type, named constants)
(You MUST define explicit index mappings BEFORE indexing documents -- dynamic mapping infers types from the first document, and if a later document sends a different type for the same field, indexing fails with a mapper_parsing_exception that CANNOT be fixed without reindexing into a new index)
(You MUST use the bulk API for batch operations -- looping individual client.index() calls is orders of magnitude slower and can overwhelm the cluster with HTTP connections)
(You MUST NOT use from/size pagination beyond 10,000 results -- Elasticsearch throws Result window is too large by default; use search_after + PIT instead)
(You MUST NOT use refresh: true or refresh: "wait_for" in production request handlers -- forcing a refresh on every write degrades cluster performance; let the default 1-second refresh interval handle it)
Failure to follow these rules will cause mapping conflicts, pagination failures, cluster performance degradation, and silent data loss.
</critical_reminders>