File Search

Elasticsearch: http://192.168.1.122:9200 Total Storage Indexed: ~18.5TB Total Documents: 3.4M+ files Active Crawlers: 9

Search API Examples

Basic File Search

curl -s "http://192.168.1.122:9200/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {"match": {"file.filename": "document.pdf"}},
  "size": 20
}'

Search by Path

curl -s "http://192.168.1.122:9200/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {"wildcard": {"path.real": "*Legal*"}}
}'

Search by Extension

curl -s "http://192.168.1.122:9200/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {"wildcard": {"file.filename": "*.pdf"}}
}'

Find Large Files (>1GB)

curl -s "http://192.168.1.122:9200/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {"range": {"file.filesize": {"gte": 1073741824}}},
  "sort": [{"file.filesize": {"order": "desc"}}]
}'

Recently Modified (Last 7 Days)

curl -s "http://192.168.1.122:9200/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {"range": {"file.last_modified": {"gte": "now-7d"}}},
  "sort": [{"file.last_modified": {"order": "desc"}}]
}'

Find Duplicate Files by Size

curl -s "http://192.168.1.122:9200/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "size": 0,
  "aggs": {
    "duplicate_sizes": {
      "terms": {"field": "file.filesize", "min_doc_count": 2, "size": 100}
    }
  }
}'

Search Across All Nodes

curl -s "http://192.168.1.122:9200/*-storage/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {"match": {"file.filename": "your-search-term"}}
}'

Crawler Inventory

Giratina (192.168.1.100) - Central Node

Talon (192.168.1.7) - Multi-Drive

Victini (192.168.1.115) - Large Storage

Hoopa (192.168.1.79)

Silvally (192.168.1.52)

Index Management

List All Indices

curl "http://192.168.1.122:9200/_cat/indices?v"

Get Storage Indices

curl "http://192.168.1.122:9200/_cat/indices/*-storage*?v&h=index,docs.count,store.size&s=index"

Check Cluster Health

curl "http://192.168.1.122:9200/_cluster/health?pretty"

Document Count for Index

curl "http://192.168.1.122:9200/talon-t9/_count?pretty"

Service Management

Check Crawler Status

ssh [email protected] "systemctl status fscrawler*"

Restart a Crawler

ssh [email protected] "systemctl restart fscrawler-NAME"

View Logs

ssh [email protected] "journalctl -u fscrawler-NAME -f"

Document Schema

Each indexed file has this metadata:

{
  "file": {
    "filename": "example.pdf",
    "extension": "pdf",
    "filesize": 1048576,
    "indexing_date": "2025-12-05T08:00:00.000Z",
    "last_modified": "2025-12-01T10:30:00.000Z"
  },
  "path": {
    "real": "/mnt/storage/expansion/Legal/example.pdf",
    "root": "/mnt/storage",
    "virtual": "/expansion/Legal/example.pdf"
  },
  "meta": {
    "title": "Example Document",
    "author": "John Doe"
  }
}

Adding a New Crawler

1. Create Configuration

mkdir -p /root/.fscrawler/new-crawler-name
cat > /root/.fscrawler/new-crawler-name/_settings.yaml << 'EOF'
---