1. What is Elasticsearch?

Elasticsearch is a distributed, open-source, RESTful search and analytics engine built on top of Apache Lucene. It stores data as JSON documents and provides near real-time search — meaning data you index becomes searchable within ~1 second.

2. Why Use Elasticsearch?

Traditional databases (MySQL, PostgreSQL) use B-tree indexes which are great for exact lookups and range queries, but terrible for full-text search. Searching "best noise cancelling headphones" across millions of product descriptions would require a full table scan or a slow LIKE query.

Elasticsearch solves this with an inverted index — the same data structure used by Google, Wikipedia search, and every search engine.

3. Inverted Index — Deep Dive

This is the single most important concept in Elasticsearch. Every feature — search speed, relevance scoring, fuzzy matching — exists because of the inverted index.

What is a Forward Index?

A traditional database stores data like this (forward index):

// Forward Index — how a normal DB stores data
// To find "fast", you must scan EVERY row

Doc1 → "Elasticsearch is fast and scalable"
Doc2 → "Redis is fast for caching"
Doc3 → "MongoDB is a NoSQL database"

To find all documents containing "fast", the DB must scan every single row — this is O(n) and gets slower as data grows.

What is an Inverted Index?

Elasticsearch flips this around. At index time, it breaks every text into words (tokens) and builds a map from each word to its documents:

// Inverted Index — how Elasticsearch stores data
// To find "fast", just look up the word → instant O(1)

Term              → Documents (Postings List)
─────────────────────────────────────────────
"elasticsearch"   → [Doc1]
"is"              → [Doc1, Doc2, Doc3]
"fast"            → [Doc1, Doc2]          ← instant lookup!
"and"             → [Doc1]
"scalable"        → [Doc1]
"redis"           → [Doc2]
"caching"         → [Doc2]
"mongodb"         → [Doc3]
"nosql"           → [Doc3]
"database"        → [Doc3]

How Multi-Word Search Works Behind the Scenes

When you search for "fast scalable":

4. Core Concepts

5. Elasticsearch vs Traditional Database

6. How a Write (Index) Works — Behind the Scenes

When you send a document to Elasticsearch, here's every single step that happens internally:

7. How a Search Works — Behind the Scenes

Search is a two-phase process: Query phase and Fetch phase.

8. Analyzers & Tokenizers

An analyzer is the text processing pipeline that runs BOTH at index time (when you store data) and at search time (when you query). It determines how text is broken into searchable terms.

The 3-Stage Pipeline

Built-in Analyzers

Edge N-Gram — For Autocomplete

Breaks a word into progressive prefixes:

// edge_ngram with min_gram:2, max_gram:5
// Input: "iPhone"
// Tokens: ["iP", "iPh", "iPho", "iPhon"]
// (after lowercase): ["ip", "iph", "ipho", "iphon"]
// Now searching "iph" matches because "iph" is a stored token!

9. Mappings (Schema Definition)

Mapping defines how each field is stored and indexed. It's like a database schema, but more powerful because the same field can be indexed multiple ways simultaneously.

Field Types

Dynamic vs Explicit Mapping

10. Query DSL — Complete Guide

Query DSL (Domain Specific Language) is ES's JSON-based query language. Two categories: Leaf queries (search a single field) and Compound queries (combine multiple queries).

Query Context vs Filter Context

10.1 match — Full-Text Search

// match: Analyzes the query, then searches the inverted index
// "titanium chip" → ["titanium", "chip"] → OR search by default
{
  "query": {
    "match": {
      "description": {
        "query": "titanium chip",
        "operator": "and",          // "or" (default) = any term, "and" = all terms must match
        "fuzziness": "AUTO",         // Typo tolerance: AUTO = 0 edits for 1-2 chars, 1 for 3-5, 2 for 6+
        "minimum_should_match": "75%" // At least 75% of terms must match
      }
    }
  }
}

10.2 match_phrase — Exact Phrase (Order Matters)

// All terms must appear in EXACT order
{
  "query": {
    "match_phrase": {
      "description": {
        "query": "A17 Pro chip",
        "slop": 1    // Allow 1 word between terms ("A17 powerful Pro chip" = still matches)
      }
    }
  }
}

10.3 multi_match — Search Multiple Fields

{
  "query": {
    "multi_match": {
      "query": "Apple premium",
      "fields": ["name^3", "brand^2", "description"],  // ^N = boost that field's score by N
      "type": "best_fields",  // Score from best matching field (default)
      "fuzziness": "AUTO"     // "most_fields" = sum of all fields, "cross_fields" = treat as one field
    }
  }
}

10.4 term — Exact Value (No Analysis)

// term: Does NOT analyze the query — searches for exact token
// Use for: keywords, IDs, booleans, enums, numbers
// NEVER use term on text fields!
{ "query": { "term": { "brand.keyword": "Apple" } } }

// terms: Match any value (like SQL IN)
{ "query": { "terms": { "category.keyword": ["smartphones", "laptops", "tablets"] } } }

10.5 range — Number/Date Ranges

{ "query": { "range": { "price": { "gte": 500, "lte": 1500 } } } }

// Date range with relative math
{ "query": { "range": { "createdAt": {
  "gte": "now-30d/d",   // 30 days ago, rounded to start of day
  "lte": "now/d",        // today, rounded to end of day
  "time_zone": "+05:30"  // Adjust for timezone before comparison
} } } }

10.6 bool — Combine Queries (MOST IMPORTANT)

{
  "query": {
    "bool": {
      "must": [                // AND + scored — all must match, affects ranking
        { "match": { "description": "premium quality" } }
      ],
      "should": [              // OR + scored — boosts score if matched
        { "match": { "tags": "5g" } },
        { "match": { "tags": "camera" } }
      ],
      "minimum_should_match": 1, // At least 1 should clause must match
      "filter": [              // AND + NOT scored + CACHED — fastest for exact conditions
        { "term": { "brand.keyword": "Apple" } },
        { "range": { "price": { "gte": 500, "lte": 2000 } } },
        { "term": { "inStock": true } }
      ],
      "must_not": [            // NOT + NOT scored + CACHED — exclude results
        { "term": { "category.keyword": "accessories" } }
      ]
    }
  }
}

10.7 Other Useful Queries

// wildcard: Pattern matching (* = any, ? = single char)
{ "query": { "wildcard": { "name.keyword": "*Phone*" } } }

// prefix: Starts with
{ "query": { "prefix": { "name.keyword": "Mac" } } }

// exists: Field exists and is not null
{ "query": { "exists": { "field": "ratings" } } }

// ids: Match specific document IDs
{ "query": { "ids": { "values": ["1", "2", "3"] } } }

11. Aggregations (Analytics Engine)

Aggregations are ES's analytics engine — like SQL GROUP BY, COUNT, AVG, SUM on steroids. You can nest them infinitely and combine with any query.

Three Types

12. Pagination Strategies

13. Cluster Architecture

Node Roles

Cluster Health

14. Sharding

15. Replication

Every primary shard can have 0+ replica shards — exact copies on different nodes.

16. Performance Tuning

• Use filter over query for non-scoring conditions — filters are cached in bitsets
• Reduce _source — only fetch fields you need
• Bulk API for writes — 500-5000 docs per batch instead of one-by-one
• Refresh interval — set to 30s or -1 during bulk indexing, back to 1s after
• Shard sizing — 10-50GB per shard, not too many, not too few
• Index aliases — for zero-downtime reindexing
• Disable replicas during bulk load — set to 0, then back to 1 after
• Force merge read-only indices — merge to 1 segment for max read speed
• Use doc_values: false on fields you never sort/aggregate — saves disk
• Avoid wildcard queries with leading wildcards — *phone requires full scan

17. Security

ES security has 5 layers:

1. Authentication — Who are you? (API keys, SAML, LDAP, native users)
2. Authorization — What can you do? (RBAC — role-based access control)
3. Encryption — TLS for transport (node-to-node) and HTTP (client-to-node)
4. Audit logging — Who did what and when?
5. Field/document-level security — Restrict WHICH data specific roles can see

18. ELK Stack (Elastic Stack)

19. Scaling & Index Lifecycle Management

Hot-Warm-Cold Architecture

20. Cost Optimization

21. Setup & Connection

// Install: npm install @elastic/elasticsearch

const { Client } = require('@elastic/elasticsearch');

const client = new Client({
    node: 'http://localhost:9200',   // ES REST endpoint (port 9200)
    maxRetries: 5,                   // Retry failed requests 5 times
    requestTimeout: 60000,           // 60s timeout per request
    sniffOnStart: false,              // true = discover all nodes on connect
    // For Elastic Cloud:
    // cloud: { id: 'deployment:base64...' },
    // auth: { apiKey: 'your-key' }
});

// Test connection
async function ping() {
    const info = await client.info();
    console.log('Connected:', info.version.number);
}
ping();

22. Create Index with Mapping

async function createProductIndex() {
    await client.indices.create({
        index: 'products',
        settings: {
            number_of_shards: 1,        // 1 shard (small dataset). CANNOT change later!
            number_of_replicas: 1,      // 1 replica (2 total copies). CAN change later.
            refresh_interval: '1s',     // New data becomes searchable every 1 second
            analysis: {
                analyzer: {
                    autocomplete_analyzer: {
                        type: 'custom',
                        tokenizer: 'autocomplete_tokenizer',
                        filter: ['lowercase']
                    }
                },
                tokenizer: {
                    autocomplete_tokenizer: {
                        type: 'edge_ngram',
                        min_gram: 2,   // Minimum 2-char prefix: "iP"
                        max_gram: 15,  // Maximum 15-char prefix
                        token_chars: ['letter', 'digit']
                    }
                }
            }
        },
        mappings: {
            dynamic: 'strict',  // Reject docs with unmapped fields
            properties: {
                name: {
                    type: 'text',
                    fields: {
                        keyword: { type: 'keyword', ignore_above: 256 },
                        autocomplete: { type: 'text', analyzer: 'autocomplete_analyzer', search_analyzer: 'standard' }
                    }
                },
                brand:       { type: 'keyword' },
                category:    { type: 'keyword' },
                price:       { type: 'float' },
                description: { type: 'text' },
                inStock:     { type: 'boolean' },
                ratings:     { type: 'float' },
                tags:        { type: 'keyword' },
                createdAt:   { type: 'date' }
            }
        }
    });
    console.log('Index created');
}

23. CRUD Operations

CREATE — Index a Document

async function createDoc() {
    const result = await client.index({
        index: 'products',       // Target index
        id: '1',                  // Doc ID. Omit = ES auto-generates. Exists = REPLACES entire doc.
        document: {               // The JSON body to store
            name: 'iPhone 15 Pro',
            brand: 'Apple',
            price: 999,
            category: 'smartphones',
            description: 'Latest iPhone with A17 Pro chip and titanium design',
            inStock: true,
            ratings: 4.8,
            tags: ['premium', '5g', 'camera'],
            createdAt: new Date().toISOString()
        },
        refresh: 'wait_for',     // Wait until next refresh to make it searchable
    });
    console.log(result.result); // 'created' or 'updated'
}

READ — Get Document

const doc = await client.get({
    index: 'products',
    id: '1',
    _source_includes: ['name', 'price'], // Only return these fields (reduces payload)
});
console.log(doc._source);    // { name: 'iPhone 15 Pro', price: 999 }
console.log(doc._version);   // Version number (increments on every update)

// Get multiple docs at once
const multi = await client.mget({ index: 'products', ids: ['1', '2', '3'] });

UPDATE — Partial Update

await client.update({
    index: 'products',
    id: '1',
    doc: {                      // Only these fields change; rest stays untouched
        price: 899,
        inStock: false,
    },
    retry_on_conflict: 3,     // Retry 3 times if version conflict
});

// Update by query — update all docs matching a condition
await client.updateByQuery({
    index: 'products',
    query: { match: { brand: 'Apple' } },
    script: {
        source: "ctx._source.price = ctx._source.price * 0.9", // 10% discount
        lang: 'painless'
    },
    conflicts: 'proceed',    // Skip conflicts instead of aborting
});

DELETE

await client.delete({ index: 'products', id: '1' });

// Delete by query
await client.deleteByQuery({
    index: 'products',
    query: { range: { price: { lt: 100 } } }
});

// Delete entire index
await client.indices.delete({ index: 'products' });

24. Search & Filters

async function searchProducts({ query, brand, category, minPrice, maxPrice, inStock, page = 1, pageSize = 10 }) {
    const must = [];
    const filter = [];

    // Full-text search (scored)
    if (query) {
        must.push({
            multi_match: {
                query,
                fields: ['name^3', 'name.autocomplete^2', 'description', 'brand^2'],
                type: 'best_fields',
                fuzziness: 'AUTO',
            }
        });
    }

    // Exact filters (not scored, cached)
    if (brand) filter.push({ term: { brand } });
    if (category) filter.push({ term: { category } });
    if (minPrice || maxPrice) {
        const range = {};
        if (minPrice) range.gte = minPrice;
        if (maxPrice) range.lte = maxPrice;
        filter.push({ range: { price: range } });
    }
    if (inStock !== undefined) filter.push({ term: { inStock } });

    const result = await client.search({
        index: 'products',
        from: (page - 1) * pageSize,  // Offset (skip first N results)
        size: pageSize,               // Limit (return N results)
        query: {
            bool: {
                must: must.length ? must : [{ match_all: {} }],
                filter,
            }
        },
        highlight: {                  // Wrap matched words in tags
            fields: { description: {}, name: {} },
            pre_tags: ['<mark>'],
            post_tags: ['</mark>'],
        },
        _source: ['name', 'brand', 'price', 'category', 'ratings', 'inStock'],
        sort: ['_score', { price: 'asc' }],  // Primary: relevance, secondary: price
    });

    return {
        total: result.hits.total.value,
        results: result.hits.hits.map(h => ({
            id: h._id, score: h._score, ...h._source, highlight: h.highlight
        })),
    };
}

25. Aggregations

async function getDashboard() {
    const result = await client.search({
        index: 'products',
        size: 0,         // size:0 = only return agg results, no documents
        aggs: {
            // BUCKET: Group by brand
            brands: {
                terms: { field: 'brand', size: 20 },
                aggs: {  // Nested: avg price per brand
                    avg_price: { avg: { field: 'price' } }
                }
            },
            // BUCKET: Custom price ranges
            price_ranges: {
                range: { field: 'price', ranges: [
                    { key: 'Budget', to: 300 },
                    { key: 'Mid', from: 300, to: 1000 },
                    { key: 'Premium', from: 1000 },
                ]}
            },
            // METRIC: Overall stats
            price_stats: { stats: { field: 'price' } },
            // METRIC: Count unique brands
            unique_brands: { cardinality: { field: 'brand' } },
        }
    });
    console.log(result.aggregations.brands.buckets);
    // [{ key:'Apple', doc_count:4, avg_price:{value:1061} }, ...]
}

26. Pagination (search_after + PIT)

// Cursor-based deep pagination (unlimited, efficient)
async function cursorPaginate(lastSort = null) {
    const body = {
        index: 'products',
        size: 20,
        query: { match_all: {} },
        sort: [{ createdAt: 'desc' }, { _id: 'asc' }], // Tiebreaker required!
    };
    if (lastSort) body.search_after = lastSort; // Sort values of last doc from previous page

    const result = await client.search(body);
    const hits = result.hits.hits;
    return { hits, nextCursor: hits.length ? hits[hits.length-1].sort : null };
}

// PIT + search_after for consistent data export
async function exportAll() {
    const pit = await client.openPointInTime({ index: 'products', keep_alive: '5m' });
    let all = [], searchAfter;

    while (true) {
        const r = await client.search({
            size: 1000,
            pit: { id: pit.id, keep_alive: '5m' },
            sort: [{ _id: 'asc' }],
            ...(searchAfter && { search_after: searchAfter }),
        });
        if (!r.hits.hits.length) break;
        all.push(...r.hits.hits.map(h => h._source));
        searchAfter = r.hits.hits[r.hits.hits.length-1].sort;
    }
    await client.closePointInTime({ id: pit.id });
    return all;
}

27. Autocomplete

async function autocomplete(query) {
    const result = await client.search({
        index: 'products',
        size: 5,
        query: { match: { 'name.autocomplete': query } },
        _source: ['name', 'brand'],
    });
    return result.hits.hits.map(h => h._source.name);
}
// autocomplete('mac') → ['MacBook Pro M3 Max']

28. Bulk Operations

async function bulkIndex(products) {
    const operations = products.flatMap((doc, i) => [
        { index: { _index: 'products', _id: String(i + 1) } },  // Action line
        doc                                                         // Document body
    ]);

    const result = await client.bulk({
        operations,
        refresh: true,     // Make all docs searchable after bulk completes
    });

    if (result.errors) {
        const failed = result.items.filter(i => i.index?.error);
        console.error('Failed items:', failed);
    }
    console.log(`Indexed ${result.items.length} docs`);
}

29. Aliases & Zero-Downtime Reindex

// Step 1: Your app always uses the alias "products" (not the real index name)
await client.indices.putAlias({ index: 'products_v1', name: 'products' });

// Step 2: Need to change mapping? Create new index
await client.indices.create({ index: 'products_v2', /* new mapping */ });

// Step 3: Reindex data from v1 to v2
await client.reindex({
    source: { index: 'products_v1' },
    dest: { index: 'products_v2' },
});

// Step 4: Atomic alias swap — zero downtime!
await client.indices.updateAliases({
    actions: [
        { remove: { index: 'products_v1', alias: 'products' } },
        { add:    { index: 'products_v2', alias: 'products' } },
    ]
});
// Your app never noticed the switch!

30. Cluster Monitoring

// Cluster health
const health = await client.cluster.health({});
console.log(health.status);              // 'green' | 'yellow' | 'red'
console.log(health.unassigned_shards);   // > 0 means trouble

// Index stats
const stats = await client.indices.stats({
    index: 'products',
    metric: ['docs', 'store', 'search', 'indexing']
});
console.log(stats._all.primaries.docs.count);     // Total documents
console.log(stats._all.primaries.store.size);      // Disk usage

31. Complete Runnable Project

// ===== elasticsearch-demo.js =====
// Run: docker run -d -p 9200:9200 -e "discovery.type=single-node" -e "xpack.security.enabled=false" docker.elastic.co/elasticsearch/elasticsearch:8.12.0
// Then: npm install @elastic/elasticsearch && node elasticsearch-demo.js

const { Client } = require('@elastic/elasticsearch');
const client = new Client({ node: 'http://localhost:9200' });
const INDEX = 'products';

async function run() {
    // 1. Delete old index if exists
    try { await client.indices.delete({ index: INDEX }); } catch(e) {}

    // 2. Create index with mapping
    await client.indices.create({
        index: INDEX,
        settings: { number_of_shards: 1, number_of_replicas: 0,
            analysis: { analyzer: { ac: { type:'custom', tokenizer:'ac_tok', filter:['lowercase'] }},
                tokenizer: { ac_tok: { type:'edge_ngram', min_gram:2, max_gram:15, token_chars:['letter','digit'] }}}},
        mappings: { properties: {
            name: { type:'text', fields:{ keyword:{type:'keyword'}, ac:{type:'text',analyzer:'ac',search_analyzer:'standard'}}},
            brand:{type:'keyword'}, category:{type:'keyword'}, price:{type:'float'},
            description:{type:'text'}, inStock:{type:'boolean'}, ratings:{type:'float'},
            tags:{type:'keyword'}, createdAt:{type:'date'}
        }}
    });

    // 3. Bulk index sample data
    const data = [
        { name:'iPhone 15 Pro', brand:'Apple', price:999, category:'smartphones', description:'A17 Pro chip titanium design', inStock:true, ratings:4.8, tags:['premium','5g'], createdAt:'2024-09-22' },
        { name:'Samsung Galaxy S24', brand:'Samsung', price:799, category:'smartphones', description:'Galaxy AI powered with S Pen', inStock:true, ratings:4.7, tags:['ai'], createdAt:'2024-01-17' },
        { name:'MacBook Pro M3', brand:'Apple', price:2499, category:'laptops', description:'Professional laptop M3 Max chip', inStock:true, ratings:4.9, tags:['pro'], createdAt:'2023-11-07' },
        { name:'Sony WH-1000XM5', brand:'Sony', price:349, category:'headphones', description:'Industry leading noise cancelling', inStock:true, ratings:4.6, tags:['wireless','anc'], createdAt:'2023-05-18' },
        { name:'iPad Air M2', brand:'Apple', price:599, category:'tablets', description:'Lightweight tablet M2 creativity', inStock:false, ratings:4.5, tags:['portable'], createdAt:'2024-03-08' },
        { name:'Dell XPS 15', brand:'Dell', price:1499, category:'laptops', description:'Premium Windows OLED display', inStock:true, ratings:4.3, tags:['oled'], createdAt:'2024-02-20' },
        { name:'AirPods Pro 2', brand:'Apple', price:249, category:'headphones', description:'Active noise cancellation adaptive audio', inStock:true, ratings:4.7, tags:['wireless'], createdAt:'2023-09-12' },
        { name:'Pixel 8 Pro', brand:'Google', price:899, category:'smartphones', description:'AI-first best camera system', inStock:true, ratings:4.5, tags:['ai','camera'], createdAt:'2023-10-12' },
    ];
    await client.bulk({ operations: data.flatMap((d,i) => [{ index:{_index:INDEX,_id:String(i+1)}}, d]), refresh:true });
    console.log('Seeded', data.length, 'docs');

    // 4. Full-text search
    console.log('\n--- Search: "apple premium" ---');
    const s1 = await client.search({ index:INDEX, query:{ multi_match:{ query:'apple premium', fields:['name^3','brand^2','description'], fuzziness:'AUTO' }}, _source:['name','price'] });
    s1.hits.hits.forEach(h => console.log(`  ${h._source.name} — $${h._source.price} (score: ${h._score})`));

    // 5. Filtered search
    console.log('\n--- Filter: Laptops under $2000 ---');
    const s2 = await client.search({ index:INDEX, query:{ bool:{ filter:[{ term:{category:'laptops'}}, { range:{price:{lte:2000}}}]}}, _source:['name','price'] });
    s2.hits.hits.forEach(h => console.log(`  ${h._source.name} — $${h._source.price}`));

    // 6. Autocomplete
    console.log('\n--- Autocomplete: "mac" ---');
    const ac = await client.search({ index:INDEX, size:3, query:{ match:{ 'name.ac':'mac' }}, _source:['name'] });
    ac.hits.hits.forEach(h => console.log(`  ${h._source.name}`));

    // 7. Aggregations
    console.log('\n--- Aggregations ---');
    const agg = await client.search({ index:INDEX, size:0, aggs:{
        brands: { terms:{field:'brand'}, aggs:{ avg_price:{avg:{field:'price'}}} },
        price_stats: { stats:{field:'price'} },
    }});
    agg.aggregations.brands.buckets.forEach(b => console.log(`  ${b.key}: ${b.doc_count} products, avg $${b.avg_price.value.toFixed(0)}`));
    console.log('  Price stats:', agg.aggregations.price_stats);

    // 8. Cluster health
    console.log('\n--- Cluster Health ---');
    const h = await client.cluster.health({});
    console.log(`  Status: ${h.status}, Nodes: ${h.number_of_nodes}, Shards: ${h.active_shards}`);
}
run().catch(e => console.error(e.meta?.body?.error || e.message));

Interview Quick-Reference Cheatsheet