Analytics basics: layers, parquet, similarity, clustering

The factory

Analytics turns raw events — a click, an add-to-cart, a purchase, an approval — into trustworthy, queryable data:

Raw logs → Cleaned data → Business metrics → Dashboards

Databricks layer names, same progression:

CSV is the starting interchange format; Parquet is where it settles — columnar, compressed, typed, readable by DuckDB, Spark and Trino. A pipeline’s job is movement between layers; the tool is whichever engine does that without becoming maintenance.

Grouping text without a model

Finding related chunks needs no LLM, only a similarity function over vectors:

cosine similarity = (A · B) / (‖A‖ ‖B‖)

Embedding each chunk with sentence-transformers and comparing vectors is fast, local, no API calls, good enough semantically; all-MiniLM-L6-v2 is the fast option, all-mpnet-base-v2 the better, slower one.

1
2

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

An LLM is optional, for writing the result up only.

HDBSCAN

Hierarchical Density-Based Spatial Clustering of Applications with Noise — DBSCAN improved: varying densities, a cluster hierarchy rather than one flat partition, the best level chosen automatically, outliers marked as noise rather than forced into a cluster.

Core distance — distance to the k-th nearest neighbour, capturing local density: small in dense regions, large in sparse ones.

Mutual reachability distance — not raw distance between two points:

max(core_dist(a), core_dist(b), distance(a, b))

The maximum stops two points in sparse regions being called close merely for proximity; both must sit in regions dense by their own standards, so loose points cannot form a cluster that looks real.

For text: cluster embedded chunks, let HDBSCAN label the rest as noise rather than inventing topics, then ask a model to name the clusters.