Magento Indexing Caching

Core Instructions

1. Understand Magento's indexers and their modes

   # List all indexers and their status
   bin/magento indexer:info
   bin/magento indexer:status
   
   # Example output:
   # catalog_category_product    Category Products    valid    Update on Save
   # catalog_product_category    Product Categories   valid    Update on Save
   # catalog_product_price       Product Price        invalid  Update by Schedule
   # catalogsearch_fulltext      Catalog Search       valid    Update by Schedule
   # catalogrule_product         Catalog Rule Product invalid  Update on Save
   

   Set all production indexers to Update by Schedule mode to prevent checkout blocking:

   bin/magento indexer:set-mode schedule catalog_category_product
   bin/magento indexer:set-mode schedule catalog_product_price
   bin/magento indexer:set-mode schedule catalogsearch_fulltext
   bin/magento indexer:set-mode schedule catalog_product_flat
   bin/magento indexer:set-mode schedule catalog_category_flat
   
   # Verify modes
   bin/magento indexer:show-mode
   

2. Manage indexers and partial reindexing

   # Reindex a single indexer (less disruptive than full reindex)
   bin/magento indexer:reindex catalog_product_price
   
   # Reindex all (avoid on production during business hours)
   bin/magento indexer:reindex
   
   # Reset indexer to "invalid" to force next scheduled run
   bin/magento indexer:reset catalog_product_price
   
   # Check mview (materialized view) changelog tables — lists pending rows to process
   bin/magento indexer:show-changelog
   

   For very large catalogs, use parallel indexing:

   # app/etc/env.php — enable parallel processing
   # In Admin → System → Index Management → Indexers → Configure
   # Or via config:
   bin/magento config:set dev/grid/async_indexing 1
   

   <?php
   // app/etc/env.php
   return [
       // ...
       'indexer' => [
           'batch_size' => [
               'catalog_product_price' => ['simple' => 200, 'configurable' => 50],
               'catalogsearch_fulltext' => ['simple' => 500],
           ],
       ],
   ];
   

3. Configure Varnish as Full-Page Cache

   # In Admin → System → Configuration → Advanced → System → Full Page Cache
   # Set Caching Application: Varnish Cache
   # Export Varnish configuration:
   bin/magento varnish:vcl:generate --export-version=6 --output-file=/etc/varnish/magento.vcl
   

   Key sections of the generated VCL to understand:

   # /etc/varnish/magento.vcl (excerpts from generated config)
   
   sub vcl_recv {
       # Do not cache if private cookie is set (logged-in customer)
       if (req.http.cookie ~ "X-Magento-Vary=") {
           # Magento sets this cookie to vary cache by context (customer group, currency)
           # The cookie value changes only when the context changes — not per session
       }
   
       # Strip _ga, fbclid, utm_* params from cache key to prevent cache fragmentation
       set req.url = regsuball(req.url, "(\?|&)(utm_[^&]+|fbclid|gclid|mc_[^&]+)(&|$)", "\1");
   }
   
   sub vcl_hash {
       # Include Magento's context cookie in the cache hash
       if (req.http.cookie ~ "X-Magento-Vary=") {
           hash_data(regsub(req.http.cookie, ".*X-Magento-Vary=([^;]+).*", "\1"));
       }
   }
   
   sub vcl_backend_response {
       # Respect Magento's Cache-Control headers
       if (beresp.http.Cache-Control ~ "no-store") {
           set beresp.uncacheable = true;
           set beresp.ttl = 120s;
       }
   
       # Strip Set-Cookie on cacheable responses
       if (beresp.ttl > 0s) {
           unset beresp.http.set-cookie;
       }
   }
   

4. Configure Redis for application cache and sessions

   <?php
   // app/etc/env.php — Redis cache and session configuration
   
   return [
       'cache' => [
           'frontend' => [
               'default' => [
                   'id_prefix' => 'b0c_',
                   'backend' => 'Magento\\Framework\\Cache\\Backend\\Redis',
                   'backend_options' => [
                       'server'            => '127.0.0.1',
                       'database'          => '0',
                       'port'              => '6379',
                       'password'          => '',
                       'compress_data'     => '1',
                       'compression_lib'   => 'gzip',
                       'persistent'        => 'mgto_cache',
                   ],
               ],
               'page_cache' => [
                   'id_prefix' => 'b0c_',
                   'backend' => 'Magento\\Framework\\Cache\\Backend\\Redis',
                   'backend_options' => [
                       'server'        => '127.0.0.1',
                       'database'      => '1', // Separate DB for FPC
                       'port'          => '6379',
                       'compress_data' => '0', // FPC data is already compressed HTML
                   ],
               ],
           ],
       ],
       'session' => [
           'save'                  => 'redis',
           'redis' => [
               'host'              => '127.0.0.1',
               'port'              => '6379',
               'password'          => '',
               'timeout'           => '2.5',
               'persistent_identifier' => 'mgto_sessions',
               'database'          => '2', // Separate DB for sessions
               'compression_threshold' => '2048',
               'compression_library'   => 'gzip',
               'log_level'             => '1',
               'max_concurrency'       => '6',
               'break_after_frontend'  => '5',
               'max_lifetime'          => '7200',
               'disable_locking'       => '1', // Improves performance but reduces session safety
           ],
       ],
   ];
   

5. Implement cache tag invalidation for custom modules

   Magento uses cache tags to selectively invalidate cached pages when data changes. Custom modules should tag and clean caches properly:

   <?php
   // Declare cache tags in your block/model
   namespace MyVendor\CustomModule\Block;
   
   use Magento\Framework\View\Element\Template;
   use Magento\Framework\DataObject\IdentityInterface;
   
   class CustomProductWidget extends Template implements IdentityInterface
   {
       private array $productIds = [];
   
       public function getIdentities(): array
       {
           // Return cache tags so Magento invalidates this block when any of these products change
           $tags = [\Magento\Catalog\Model\Product::CACHE_TAG];
           foreach ($this->productIds as $id) {
               $tags[] = \Magento\Catalog\Model\Product::CACHE_TAG . '_' . $id;
           }
           return $tags;
       }
   }
   

   <?php
   // In your observer or command — flush only relevant pages after product update
   namespace MyVendor\CustomModule\Observer;
   
   use Magento\Framework\Event\ObserverInterface;
   use Magento\Framework\App\Cache\TypeListInterface;
   use Magento\PageCache\Model\Cache\Type as PageCacheType;
   
   class ProductSaveObserver implements ObserverInterface
   {
       public function __construct(
           private readonly TypeListInterface $cacheTypeList,
           private readonly \Magento\Framework\App\Cache\Tag\Resolver $tagResolver
       ) {}
   
       public function execute(\Magento\Framework\Event\Observer $observer): void
       {
           $product = $observer->getEvent()->getProduct();
   
           // Invalidate only pages tagged with this product's cache tag
           // Varnish will purge via PURGE request using X-Magento-Tags-Pattern header
           $this->cacheTypeList->invalidate(PageCacheType::TYPE_IDENTIFIER);
       }
   }
   

Examples

Diagnose index lag causing wrong prices

# Check if mview changelog has pending rows
bin/magento indexer:show-changelog

# If catalog_product_price has > 0 pending rows, prices are stale
# Run the indexer to catch up
bin/magento indexer:reindex catalog_product_price

# Check cron_schedule table for missed cron jobs (should run indexers every minute)
bin/magento cron:run --group index &
# Or manually via:
mysql -u root -e "SELECT job_code, status, finished_at FROM magento.cron_schedule WHERE job_code LIKE '%index%' ORDER BY finished_at DESC LIMIT 20;"

Warm FPC after deployment

#!/bin/bash
# scripts/warm-cache.sh — warm the FPC after deploy
set -e

MAGENTO_BASE_URL="https://www.mystore.com"
SITEMAP_URL="$MAGENTO_BASE_URL/sitemap.xml"

echo "Flushing Magento caches..."
bin/magento cache:flush

echo "Warming FPC via sitemap..."
# Download sitemap and curl each URL with a warm-cache header
curl -s "$SITEMAP_URL" | \
  grep -oP '<loc>\K[^<]+' | \
  head -500 | \
  xargs -P 10 -I {} curl -s -o /dev/null -w "%{http_code} {}\n" \
    -H "X-Warm-Cache: 1" {}

echo "Cache warming complete."

Best Practices

• Always use Update by Schedule mode in production — Update on Save reindexes synchronously during admin saves, causing timeouts on large catalogs
• Separate Redis databases for cache, FPC, and sessions — using database 0 for everything causes key conflicts and makes it impossible to flush only one type
• Tag custom blocks with IdentityInterface — this ensures Varnish and Magento's FPC invalidate the right pages when your data changes
• Monitor indexer changelog table sizes — a catalog_product_price_cl table with millions of rows indicates cron is not running; fix cron before the table size causes reindex timeouts
• Use bin/magento cache:clean vs cache:flush — clean removes only invalid cache entries (safe); flush nukes everything including other framework caches sharing the same Redis
• Set distinct id_prefix per environment — staging and production can share a Redis server if each has a unique prefix, preventing cache poisoning
• Test Varnish with curl -I and X-Cache header — a HIT response confirms Varnish is serving from cache; MISS means it's hitting Magento for every request

Common Pitfalls

Related Skills

• @magento-module-development
• @magento-graphql
• @magento-multi-store
• @caching-strategies
• @infrastructure-performance