Cookbook / Data and storage
Full-text and vector search
Search text with PostgreSQL full-text, find similar items with pgvector, and combine both into one ranked list.
PostgreSQL can already do good search, so there is no need for a separate search server for many apps. axeon-orm wraps full-text search and the pgvector extension in typed queries, and lets you blend text rank and vector similarity into one score for hybrid search.
cargo add axeon-orm --features pgvector
# PostgreSQL with the extension, e.g.:
docker run --rm -p 5432:5432 -e POSTGRES_PASSWORD=postgres pgvector/pgvector:pg17
use axeon::{Json, Server, ServerError};
use axeon_orm::prelude::*;
use axeon_orm::search::{self, FullText};
use axeon_orm::vector::{self, Vector, VectorIndex};
#[derive(Model, serde::Serialize)]
struct Product {
id: i64,
name: String,
description: String,
#[serde(skip)]
#[model(dimensions = 3)] // use your embedding model's size, e.g. 1024
embedding: Vector,
}
fn fts() -> FullText<Product> {
FullText::columns([Product::NAME, Product::DESCRIPTION]).config("english")
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let db = Db::connect(&std::env::var("DATABASE_URL")?).await?;
db.execute("CREATE EXTENSION IF NOT EXISTS vector").await?;
db.create_table::<Product>().await?; // use migrations in real apps
// A GIN index matching the queries, and an HNSW index for nearest-neighbour.
db.execute(&search::create_index(&fts(), "products_fts")).await?;
db.execute(&vector::create_index::<Product>("embedding", VectorIndex::hnsw_cosine())).await?;
let mut app = Server::new();
app.state(db);
// GET /search?q=led lamp -solar (words, "phrases", OR and -excluded)
app.get("/search", |req| async move {
let q = req.query_param("q").unwrap_or_default().to_string();
let fts = fts();
let found = Product::query()
.filter(fts.matches(&q))
.order_by(fts.rank(&q).desc())
.limit(20)
.all(req.db()?)
.await?;
Ok::<_, ServerError>(Json(found))
});
// Hybrid: text rank and vector similarity in one score, best first.
app.get("/hybrid", |req| async move {
let q = req.query_param("q").unwrap_or_default().to_string();
let query_vector = Vector::from(vec![0.1, 0.9, 0.2]); // embed `q` with your model
let fts = fts();
let score = fts.rank(&q) * 0.6 + Product::EMBEDDING.cosine_distance(&query_vector).similarity() * 0.4;
let top: Vec<(Product, f64)> = Product::query()
.filter(fts.matches(&q) | Product::EMBEDDING.cosine_distance(&query_vector).lt(0.5))
.ranked(req.db()?, score, 20)
.await?;
let rows: Vec<_> = top.into_iter().map(|(p, s)| serde_json::json!({ "product": p, "score": s })).collect();
Ok::<_, ServerError>(Json(rows))
});
let listener = tokio::net::TcpListener::bind("127.0.0.1:3000").await?;
app.serve_with_shutdown(listener, axeon::shutdown_signal()).await?;
Ok(())
}
Notes#
- Ranks are normalised to 0..1, so they mix cleanly with vector similarities. Choose the weights by looking at your own results.
- Search a stored, generated
tsvectorcolumn instead of computing it per query withFullText::stored("search")andsearch::generated_column(..). That is faster on large tables. cosine_distance(<=>),l2_distance(<->) andinner_product(<#>) all sort and filter ordinary queries.Query::nearest(&db, distance, k)returns thekclosest rows with their distances.- HNSW and IVFFlat indexes make nearest-neighbour search approximate and fast. Filters on other columns apply to the index's candidates, so with very selective filters, raise
hnsw.ef_search. Query::to_sqlshows the SQL, which you canEXPLAIN.- PostgreSQL's full-text search has no word splitting for Chinese; install zhparser or pg_jieba and pass its configuration to
.config(..).