Simon Knight

Data Analytics & Visualization Ecosystem

High-performance tools for processing massive datasets, discovering patterns in time series, and creating information-dense visualizations that transform raw data into actionable insights.

Contents

The Vision

Modern data analysis fragments across disconnected tools: extract data with one tool, analyze with another, visualize with a third. This ecosystem provides an integrated workflow built on high-performance foundations (Rust + Polars) with a focus on three key problems:

  1. Scale: Process millions of rows interactively, not in overnight batch jobs
  2. Signal: Find patterns and anomalies automatically, not through manual exploration
  3. Clarity: Generate visualizations that reveal structure, not just plot points

Core Philosophy:

How They Work Together

┌────────────────────────────────────────────────────────────┐
│              Raw Data Sources                              │
│   (Geospatial · Time Series · Large CSVs · Streams)       │
└────────────┬───────────────────────────┬───────────────────┘
             │                           │
    ┌────────▼─────────┐        ┌────────▼─────────┐
    │ Tileserver Polars│        │ matrix-profile-rs│
    │  (Geospatial)    │        │  (Time Series)   │
    └────────┬─────────┘        └────────┬─────────┘
             │                           │
             └──────────┬────────────────┘
                        │
          ┌─────────────▼──────────────┐
          │   Interactive Frontends    │
          │  (Kepler.gl · Dashboards)  │
          └────────────────────────────┘

Typical Workflow:

  1. Ingest: Load massive datasets (geospatial points, time series) into Polars DataFrames
  2. Analyze: Use matrix-profile-rs for pattern discovery or Tileserver for spatial queries
  3. Visualize: Render interactive visualizations with sub-second query latency
  4. Iterate: Refine analysis based on visual feedback without waiting for batch jobs

The Tools

Tileserver Polars — Geospatial Analytics at Scale

Active Development · Full Details →

What It Is: Tile server that renders vector tiles (MVT) from Polars DataFrames for interactive geospatial visualization.

Key Features:

Example Workflow:

Load data:

import polars as pl
from tileserver_polars import TileServer

# Load massive point dataset
df = pl.read_csv("earthquakes_10M.csv")

# Start tile server
server = TileServer(df, lon_col="longitude", lat_col="latitude")
server.start(port=8080)

Configure Kepler.gl:

// Add custom tile layer
{
  type: "mvt",
  url: "http://localhost:8080/tiles/{z}/{x}/{y}.mvt",
  renderSubLayers: true
}

Query dynamically:

# Filter by magnitude on the fly
server.set_filter(pl.col("magnitude") > 5.0)
# Tiles regenerate automatically with filtered data

Performance:

Use Cases:

Current Status: Production-ready for point geometries, adding polygon/line support.

Tech Stack: Rust, Polars, protobuf for MVT encoding, Actix-web for HTTP

matrix-profile-rs — Time Series Pattern Discovery

Phase 2/5 (16%) · Full Details →

What It Is: A high-performance Rust implementation of Matrix Profile algorithms for time series analysis. Automatically discovers repeating patterns (motifs) and anomalies (discords) in univariate time series without domain knowledge or parameter tuning.

The Problem: Time series analysis traditionally requires:

The Solution: Matrix Profiles provide a universal representation:

Key Features:

Example: Predictive Maintenance

Input time series (vibration sensor):

use matrix_profile::{MatrixProfile, stomp};

// Load sensor data
let vibration_data: Vec<f64> = load_sensor_readings();

// Compute matrix profile (window size: 100 samples)
let mp = stomp(&vibration_data, 100)?;

// Find top 3 repeating patterns
let motifs = mp.motifs(3)?;
for (rank, motif) in motifs.iter().enumerate() {
    println!("Motif {}: occurs at indices {:?}",
             rank + 1, motif.occurrences);
    println!("  Distance: {:.4}", motif.distance);
}

// Find top 3 anomalies
let discords = mp.discords(3)?;
for (rank, discord) in discords.iter().enumerate() {
    println!("Anomaly {}: at index {}",
             rank + 1, discord.index);
    println!("  Severity: {:.4}", discord.distance);
}

Output:

Motif 1: occurs at indices [1234, 2456, 3678, 4890, ...]
  Distance: 0.0234
Motif 2: occurs at indices [890, 1890, 2890]
  Distance: 0.0456

Anomaly 1: at index 5432
  Severity: 12.3456
Anomaly 2: at index 7890
  Severity: 11.2345

Interpretation:

Polars Integration (Planned):

import polars as pl

df = pl.read_csv("sensor_data.csv")

# Compute matrix profile as DataFrame operation
result = df.with_columns([
    pl.col("vibration").mp.stomp(window=100).alias("mp_distance"),
    pl.col("vibration").mp.motifs(k=3).alias("top_motifs"),
    pl.col("vibration").mp.discords(k=3).alias("anomalies")
])

Performance Targets:

Use Cases:

Current Status: Phase 2 (Discovery Ergonomics) - building high-level APIs for motif/discord extraction.

Tech Stack: Rust, ndarray, rayon for parallelization, PyO3 for Python bindings

Philosophy: Why This Approach?

Performance Enables Interactivity

Sub-second query latency transforms the analysis workflow. Instead of “run batch job, wait, inspect results, adjust, repeat,” you get “adjust filter, see results immediately.” This tight feedback loop enables exploratory analysis that’s impossible with slow tools.

Rust + Polars for the Data Layer

Polars provides:

Algorithms, Not Heuristics

Matrix Profiles are mathematically sound—they guarantee finding the true nearest neighbor for every subsequence. This eliminates “tune epsilon until it looks right” parameter hell common in clustering/anomaly detection.

Composable Tools

Each tool solves one problem well:

Use the full stack or just the pieces you need. All built on Arrow for interoperability.

Open Source & Contributions

Active development, contributions welcome:

← Back to Projects View CV Network Automation Signal Processing Agentic Systems