Netryx Street Level Geolocation

GPU support auto-detected:

• NVIDIA GPU → CUDA (uses ALIKED, 1024 keypoints)
• Apple Silicon → MPS (uses DISK, 768 keypoints)
• No GPU → CPU (slower)

Optional Gemini API key for AI Coarse geolocation mode:

export GEMINI_API_KEY="your_key_here"

Launch GUI

python test_super.py

macOS blank GUI fix: brew install [email protected]

Core Workflow

1. Create an Index (Required First Step)

The index stores 512-dim CosPlace fingerprints for every crawled panorama in a geographic area.

Via GUI:

1. Select Create mode
2. Enter center lat/lon, radius (km), grid resolution (default: 300)
3. Click Create Index — saves incrementally to cosplace_parts/

Index size reference:

Interrupted builds resume automatically on next run.

Auto-build compiled index (runs after Create or manually):

cosplace_parts/*.npz  →  index/cosplace_descriptors.npy
                      →  index/metadata.npz

2. Search for a Location

Via GUI:

1. Select Search mode
2. Upload street-level photo
3. Choose method:
   • Manual: Provide center lat/lon + radius if you know the approximate region
   • AI Coarse: Gemini analyzes signs/architecture to estimate region (requires GEMINI_API_KEY)
4. Click Run Search → Start Full Search
5. Real-time candidate visualization → GPS result + confidence score on map

Pipeline Architecture

Query Image
    │
    ├─ CosPlace 512-dim descriptor
    ├─ Flipped descriptor (handles mirrored perspectives)
    │
    ▼
Cosine similarity search → radius filter (haversine) → Top 500–1000 candidates
    │                                                    (<1 second)
    ▼
Download panoramas → Rectilinear crops at 3 FOVs (70°, 90°, 110°)
    │
    ├─ ALIKED (CUDA) or DISK (MPS/CPU) keypoint extraction
    ├─ LightGlue deep feature matching
    ├─ RANSAC geometric verification
    │                                                    (2–5 min)
    ▼
Heading refinement: ±45° at 15° steps, top 15 candidates
    │
    ├─ Spatial consensus clustering (50m cells)
    ├─ Confidence scoring (uniqueness ratio)
    │
    ▼
📍 GPS Coordinates + Confidence Score

Ultra Mode (Difficult Images)

Enable Ultra Mode checkbox for night shots, motion blur, low-texture scenes.

Adds three extra passes:

1. LoFTR — detector-free dense matching (no keypoints needed, handles blur)
2. Descriptor hopping — if best match has <50 inliers, extract CosPlace from the matched clean panorama and re-search index
3. Neighborhood expansion — searches all panoramas within 100m of best match

Significantly slower; use when standard pipeline returns low confidence.

Project Structure

netryx/
├── test_super.py          # Main app: GUI + indexing + search pipeline
├── cosplace_utils.py      # CosPlace model loading + descriptor extraction
├── build_index.py         # Standalone high-performance index builder
├── requirements.txt
├── cosplace_parts/        # Raw .npz embedding chunks (created during indexing)
└── index/
    ├── cosplace_descriptors.npy   # All 512-dim descriptors (stacked)
    └── metadata.npz               # lat, lon, heading, panoid per descriptor

Code Examples

Extract a CosPlace Descriptor Programmatically

from cosplace_utils import load_cosplace_model, extract_descriptor
from PIL import Image
import torch

device = torch.device("cuda" if torch.cuda.is_available() else
                      "mps" if torch.backends.mps.is_available() else "cpu")

model = load_cosplace_model(device=device)

img = Image.open("query_photo.jpg").convert("RGB")
descriptor = extract_descriptor(model, img, device=device)
# descriptor.shape → (512,)

Load the Index and Run a Radius-Filtered Cosine Search

import numpy as np

# Load compiled index
descriptors = np.load("index/cosplace_descriptors.npy")   # (N, 512)
meta        = np.load("index/metadata.npz")
lats        = meta["lats"]      # (N,)
lons        = meta["lons"]      # (N,)
headings    = meta["headings"]  # (N,)
panoids     = meta["panoids"]   # (N,)

# Query descriptor (from extract_descriptor above)
query_vec = descriptor / np.linalg.norm(descriptor)

# Cosine similarity — single matrix multiply
norms     = np.linalg.norm(descriptors, axis=1, keepdims=True)
normed    = descriptors / (norms + 1e-8)
scores    = normed @ query_vec                              # (N,)

# Haversine radius filter
def haversine_km(lat1, lon1, lat2, lon2):
    R = 6371.0
    dlat = np.radians(lat2 - lat1)
    dlon = np.radians(lon2 - lon1)
    a = np.sin(dlat/2)**2 + np.cos(np.radians(lat1)) * np.cos(np.radians(lat2)) * np.sin(dlon/2)**2
    return R * 2 * np.arcsin(np.sqrt(a))

center_lat, center_lon = 48.8566, 2.3522   # Paris example
radius_km = 2.0

distances = haversine_km(lats, center_lon, center_lat, center_lon)
# Vectorised version:
distances = haversine_km(lats, lons, center_lat, center_lon)
mask      = distances <= radius_km

filtered_scores  = scores.copy()
filtered_scores[~mask] = -1.0

top_k = 500
top_idx = np.argsort(filtered_scores)[::-1][:top_k]

print("Top match:")
print(f"  lat={lats[top_idx[0]]:.6f}, lon={lons[top_idx[0]]:.6f}")
print(f"  heading={headings[top_idx[0]]}, panoid={panoids[top_idx[0]]}")
print(f"  score={filtered_scores[top_idx[0]]:.4f}")

Run LightGlue Matching Between Query and Candidate Crop

import torch
from PIL import Image
import numpy as np
from lightglue import LightGlue, ALIKED, DISK
from lightglue.utils import load_image, rbd

device = torch.device("cuda" if torch.cuda.is_available() else
                      "mps" if torch.backends.mps.is_available() else "cpu")

# Select extractor based on device
if device.type == "cuda":
    extractor = ALIKED(max_num_keypoints=1024).eval().to(device)