# Vantage Framework > Vantage is an open-source Rust framework for business data: typed entities and tables over SQL (PostgreSQL, MySQL, SQLite), SurrealDB, MongoDB, REST/GraphQL APIs and more, with relations, expressions, Rhai scripting and live (push or poll) data. It is the engine under Vantage UI (https://vantage-ui.com/). This book teaches it bottom-up, from a first query to live APIs and adding a new persistence. --- Source: https://vantage-ui.com/book/introduction.html # Vantage Framework Vantage is a data entity persistence and abstraction framework for Rust. Vantage changes the way you think about your data. Instead of writing queries and loading rows, you work with **sets** of records — "unpaid invoices older than 30 days", "customers who ordered today" — narrowing them, combining them, traversing from one to another, and acting on them where they live. Vantage offers two ways to work with your data. In **transactional** mode there is no state to manage: load a set, act on it, done — each operation goes straight to the backend. This is the natural shape of a REST endpoint: handle the request, read or update the data, respond. (The name describes that request/response shape, not database transactions — BEGIN/COMMIT is the backend's business, and Vantage doesn't wrap your operations in one unless you ask; see [Transactions](./transactions.md).) In **live** mode Vantage maintains a local representation of the data segment you're working with: you operate on your local copy, and it reconciles with the source over time. That's how a user interface behaves — any kind. Live mode is built on top of transactional mode, so everything you learn about the first carries into the second. This documentation tracks the **2.0** line of `vantage-table` and `vantage-vista`, and the drivers and `vantage-diorama` releases built on it. ## Ethos A few principles run through every layer of the framework: - **Let the backend do the work.** If a database or API can filter, join, aggregate, or paginate, Vantage pushes the work there — data is narrowed at the source, not in your process. - **Fill the gaps client-side, honestly.** Where a backend can't (a CSV file can't sort; two databases can't join), the layer above provides it — and every handle advertises exactly what it supports, so a missing capability is an explicit error, never a silent guess. - **Respect native types.** Each backend keeps its own type system end to end — decimals stay precise, dates stay dates, nothing is funnelled through JSON. - **Business logic lives on the entity.** Validation, audit, soft-delete, and domain methods attach to your model once and apply everywhere the entity is used. - **Fail loudly, retry safely.** No panics, no silent zeros or match-alls. Every write is retry-safe and stays inside the set it goes through ([Safe writes](./record-lifecycle.md#safe-writes)). - **Be aware of observers.** Data knows who is watching: changes stream to subscribers, and edits reconcile instead of clobbering. (This is live mode — the second half of the guide builds it up.) ## Three ways to work with data Vantage lets you choose how strongly typed your data access is — per component, not per project: 1. **Entity mode** — records are plain Rust structs (`Table`). Columns and conditions are compiler-checked, and business logic attaches to the entity: full type safety, the natural choice for hand-written Rust code. 2. **Record mode** — each record is an arbitrary structure of named values (`Record`), with schema introspectable at runtime. No entity struct required — built for generic code: data grids, admin tools, import/export. 3. **Rhai scripts** — more generic still: schema declared in YAML, custom expressions and logic in [Rhai](https://rhai.rs). Defined, loaded, and sealed at runtime with no recompiling — made for configuration-driven and agent-driven tooling. The three interoperate: a Rhai-declared table serves records, records deserialize into entities, and all of it sits behind the same capability-checked handles. ## What makes Vantage different Vantage is a **framework, not a library**. A library solves one concern and hands the rest back to you; Vantage takes over the data layer entirely — modelling, querying, types, caching, reactivity — and gives you well-defined places to plug in what's yours. The implementation spans 10+ crates (query builders, backend drivers, the entity layer, the reactive stack), all built on the same cohesive, extensible principles: what you learn in one crate applies in the next, and every extension point looks the same wherever you meet it. Vantage also doesn't mimic frameworks from other languages. No reflection, no runtime magic, no inheritance hierarchies. It leans on what Rust is uniquely good at — traits for composition, ownership for safe sharing, generics that compile away, async throughout — so the framework feels native rather than translated. ## The four layers Everything above maps onto four layers. Each builds on the one below, and you climb only as far as your application needs: ```text Table entity mode — typed, compiler-checked, transactional Vista record mode — schema known at runtime, capabilities honest Dio live mode begins — local cache, write queue, change events Scenery reactive views over a Dio — what a UI binds to and watches ``` A `Table` is where your model and business logic live. Wrap it into a `Vista` when generic code needs to consume it ([record mode](#three-ways-to-work-with-data), above). Bind the Vista to a `Dio` when you want a live local representation — caching, write routing, reconciliation. Open a `Scenery` over the Dio for an ordered table, a single record, or an aggregate that updates as the data changes. Layers never leak upward: a `Table` doesn't know it's being cached, and a `Scenery` consumer can't tell which backend is underneath. ## Vantage and Vantage UI [Vantage UI](https://vantage-ui.com) is a native admin console built directly on these crates — point it at your databases, APIs, and tools, and an AI agent configures tables, forms, and dashboards over them. It is a closed-source product (free download), and this framework is its open foundation: Vantage is open-sourced so that you can build your own services, CLIs, and UIs on the same data layer — and extend it, adding persistences and capabilities that custom builds can carry further than the stock app does. If you want the finished tool, start with Vantage UI. If you want the foundation, read on. ## Getting Started Vantage covers a lot of ground — multiple databases, type systems, entity frameworks, UI adapters — but none of that matters until you've seen it do something useful. This guide introduces Vantage concepts one at a time, each building on the last. We start with something you already know — SQL — and build a shared **foundation**: a small product catalog that begins as a CLI, becomes an HTTP API, and switches databases without touching its handlers, then erases into a runtime-generic data handle. From there the guide **forks by the shape of your problem**. The reactive stack that sits on top — caching, change events, watchable views — is the same whichever backend is underneath, so you follow it down whichever path is yours: - **A facade over an API you don't control** — a slow, read-only cloud API that can't sort, search, or paginate. Diorama caches it and fills the gaps. - **A live view in front of your own database** — a relational database you own and write to. Diorama gives it a cached, watchable facade, and the path moves the same app from SQLite to PostgreSQL with a single switch, then on to SurrealDB. Same abstractions, opposite backends — that contrast is the point. ### The foundation 1. **[SQLite and the Query Builder](./intro/step1-first-query.md)** — connect to a database, build and execute typed queries, map rows to structs. 2. **[Tables and Typed Data Access](./intro/step2-tables.md)** — define entities and tables, narrow sets with conditions, traverse relationships, add computed fields; CRUD becomes one-liners. 3. **[A Standalone Axum Server](./intro/step3-axum-server.md)** — put the model behind HTTP with one generic CRUD handler for every entity, then migrate the whole server from SQLite to MongoDB by editing only the model. 4. **[Vista — the Universal Data Handle](./intro/step4-vista.md)** — erase the entity and backend into a schema-bearing runtime handle, with explicit capability contracts. ### Choose your path With a Vista in hand, **[pick the path that matches your data](./intro/fork.md)** — a facade over an API you don't control, or a live view over your own relational database. Both build the same caching, reactive, watch-streaming stack; they differ only in the backend beneath it. You'll need basic Rust experience (structs, traits, async/await, cargo). No prior Vantage knowledge required. **Start here:** [SQLite and the Query Builder](./intro/step1-first-query.md) ## Beyond the guide The rest of the book is reference material — read it when the guide points at it, or jump straight to what you need: - [Expressions & Queries](./expressions.md) and the per-backend chapters ([SQL](./sql.md), [SurrealDB](./surrealdb.md)) — the query-building layer in depth. - [Records: Traversal, Invariants & Hooks](./record-lifecycle.md) — the write pipeline: audit stamps, validation, soft-delete. - [Model-Driven Architecture](./mda.md) — how Vantage expects you to structure business software. - [Config-Driven Vistas: YAML & Rhai](./config-driven-vistas.md) — define and reshape data handles from configuration, sealed at runtime without recompiling. - [Adding a New Persistence](./new-persistence.md) — nine incremental steps to connect Vantage to a backend it doesn't know yet; the main path for extending the framework. - [Augmentation](./augmentation.md) and the [Type System](./type-system.md) — enriching rows from a second source, and teaching backends your own types. - [Historical Timeline](./history.md) — how the framework got here, release by release. --- Source: https://vantage-ui.com/book/intro/step1-first-query.html # SQLite and the Query Builder Vantage is a big framework. It covers SQL databases, SurrealDB, MongoDB, CSV files, REST APIs — and ties them all together with a shared type system, expression engine, and data abstraction layer. We'll get to all of that. But right now, let's start with something familiar: **building SQL queries**. ```admonish info title="What is a Query Builder?" A query builder is a tool that assembles SQL from composable parts instead of string concatenation. You've probably seen this pattern before: - **Knex.js** (JavaScript) — `knex('users').where('age', '>', 18).select('name')` - **SQLAlchemy Core** (Python) — `select(users.c.name).where(users.c.age > 18)` - **JOOQ** (Java) — `dsl.select(USERS.NAME).from(USERS).where(USERS.AGE.gt(18))` - **Diesel** (Rust) — `users.filter(age.gt(18)).select(name)` Vantage has its own query builder too. Each supported database gets a dedicated builder — `SqliteSelect`, `PostgresSelect`, `SurrealSelect` — so you get the right quoting, parameter binding, and dialect features for your target. ``` For this chapter we'll use SQLite. It's lightweight, needs no server, and works with a plain file on disk. ```admonish example title="Goals for this chapter" By the end of this page you'll be able to: 1. Connect to an SQLite database from Rust 2. Build SELECT queries with fields and conditions 3. Execute queries and read results 4. Convert results into `Vec` with typed field access 5. Understand how Vantage keeps parameters separate from SQL (no injection risk) ``` --- ## Set up Create a new project: ```sh cargo init learn-1 && cd learn-1 cargo add vantage-sql --features sqlite cargo add vantage-expressions cargo add tokio --features full ``` Three dependencies — `vantage-sql` gives us the SQLite query builder and connection pool, `vantage-expressions` is needed by the `sqlite_expr!` macro, and `tokio` provides the async runtime because all database operations are async. ## Create and populate a database We'll make a small product catalog from scratch. Create `seed.sql` in your project root: ```sql CREATE TABLE product ( id INTEGER PRIMARY KEY, name TEXT NOT NULL, price INTEGER NOT NULL, category_id INTEGER, is_deleted BOOLEAN NOT NULL DEFAULT 0 ); INSERT INTO product VALUES (1, 'Cupcake', 120, 1, 0); INSERT INTO product VALUES (2, 'Doughnut', 135, 1, 0); INSERT INTO product VALUES (3, 'Tart', 220, 2, 0); INSERT INTO product VALUES (4, 'Pie', 299, 2, 0); INSERT INTO product VALUES (5, 'Cookies', 199, 1, 0); INSERT INTO product VALUES (6, 'A Stale Cake', 80, 1, 1); INSERT INTO product VALUES (7, 'Sourdough Loaf', 350, 3, 0); CREATE TABLE category ( id INTEGER PRIMARY KEY, name TEXT NOT NULL ); INSERT INTO category VALUES (1, 'Sweet Treats'); INSERT INTO category VALUES (2, 'Pastries'); INSERT INTO category VALUES (3, 'Breads'); ``` Run it: ```sh sqlite3 products.db < seed.sql ``` You now have `products.db` — 7 products (6 active, 1 deleted) across 3 categories. Quick check: ```sh sqlite3 products.db "SELECT name, price FROM product WHERE is_deleted = 0" ``` --- ## Start with an async main All database operations in Vantage are async, so we need a Tokio runtime. Replace `src/main.rs` with: ```rust use vantage_sql::prelude::*; #[tokio::main] async fn main() { if let Err(e) = run().await { e.report(); } } async fn run() -> VantageResult<()> { println!("Ready!"); Ok(()) } ``` A few things going on here: - **`use vantage_sql::prelude::*`** brings in everything we need for this chapter — [`SqliteDB`](vantage_sql::sqlite::SqliteDB), [`SqliteSelect`](vantage_sql::sqlite::statements::SqliteSelect), the `sqlite_expr!` macro, error types, and the traits that make builder and execution methods work. - **`VantageResult<()>`** is Vantage's own Result type. It uses `VantageError`, which tracks context and error chains for readable diagnostics. - **`e.report()`** prints the error in a structured format. We call it from `main()` because Rust's default `Result`-returning main uses `Debug` formatting, which is ugly. This pattern gives us clean error output instead. Run `cargo run` to make sure it compiles. --- ## Connect to SQLite Add this inside `run()`: ```rust,no_run # use vantage_sql::prelude::*; # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let db = SqliteDB::connect("sqlite:products.db?mode=ro") .await .context("Failed to connect to products.db")?; # Ok(()) # } ``` `SqliteDB` wraps an sqlx connection pool. The connection string is an [sqlx URL](https://docs.rs/sqlx/latest/sqlx/sqlite/struct.SqliteConnectOptions.html) — `?mode=ro` opens read-only, which is all we need for now. ```admonish tip title="Already have an sqlx pool?" If you're adding Vantage to an existing project that already has a `SqlitePool`, wrap it directly: ~~~rust let db = SqliteDB::new(existing_pool); ~~~ The reverse works too — `db.pool()` gives you the underlying `SqlitePool`, although Vantage expressions will eliminate any need to execute queries directly. ``` ```admonish info title=".context() — readable errors" `.context()` wraps any error with a human-readable message. If the database file doesn't exist, instead of a raw sqlx error you get: ~~~text Error: Failed to connect to products.db │ ╰─▶ error returned from database: (code: 14) unable to open database file ~~~ You'll see `.context()` used throughout Vantage code. It comes from `VantageError` and works on any `Result` with a standard error type. ``` --- ## Build a SELECT [`SqliteSelect`](vantage_sql::sqlite::statements::SqliteSelect) is the query builder for SQLite. Other persistences have their own — [`PostgresSelect`](vantage_sql::postgres::statements::PostgresSelect), [`MongoSelect`](vantage_mongodb::select::MongoSelect) — and they all implement the [`Selectable`](vantage_expressions::Selectable) trait, so the interface is identical apart from vendor-specific extensions. None of them need a database connection — they're just structs that accumulate query parts. You build them with a chain of `.with_*()` calls: ```rust # use vantage_sql::prelude::*; let select = SqliteSelect::new() .with_source("product") .with_field("name") .with_field("price"); println!("{}", select.preview()); // SELECT "name", "price" FROM "product" # assert_eq!(select.preview(), r#"SELECT "name", "price" FROM "product""#); ``` `.preview()` renders the final SQL as a string — handy for debugging, but never used for execution. ```admonish info title="Builder pattern" `.with_*()` consumes the builder and returns a new one. Call `.with_field()` as many times as you need; skip it entirely for `SELECT *`. Every `.with_*()` method has a corresponding `.add_*()` that mutates in place instead of consuming. Use whichever fits your code: ~~~rust // Builder style let select = SqliteSelect::new() .with_source("product") .with_field("name"); // Mutable style let mut select = SqliteSelect::new(); select.add_source("product", None); select.add_field("name"); ~~~ Same result. The `.with_*()` style is nicer for one-shot construction, `.add_*()` is useful when you're building a query conditionally in a loop. ``` --- ## Execute it ```rust,no_run # use vantage_sql::prelude::*; # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:products.db?mode=ro") # .await # .context("Failed to connect to products.db")?; # let select = SqliteSelect::new() # .with_source("product") # .with_field("name") # .with_field("price"); let result = db.execute(&select.expr()).await?; println!("{:?}", result); # Ok(()) # } ``` Two steps here: `.expr()` turns the builder into an [`Expression`](vantage_expressions::Expression) — Vantage's internal representation that keeps parameters separate from the SQL template. Then [`db.execute()`](vantage_expressions::ExprDataSource::execute) sends it to the database. The result is [`AnySqliteType`](vantage_sql::sqlite::AnySqliteType) — a type-tagged wrapper around whatever came back. The `Debug` output isn't pretty, but you should see all 6 product rows in there. ```admonish tip title="When does Vantage hit the database?" Only on `.await`. Everything before that — `with_source`, `with_field`, `with_condition` — is synchronous struct manipulation. You always know when a database call happens because you typed `.await`. ``` --- ## Adding conditions Our database has a soft-delete flag — "A Stale Cake" has `is_deleted = 1`. Let's filter it out: ```rust # use vantage_sql::prelude::*; let condition = sqlite_expr!("\"is_deleted\" = {}", false); let select = SqliteSelect::new() .with_source("product") .with_field("name") .with_field("price") .with_condition(condition); println!("{}", select.preview()); // SELECT "name", "price" FROM "product" WHERE "is_deleted" = 0 # assert_eq!( # select.preview(), # r#"SELECT "name", "price" FROM "product" WHERE "is_deleted" = 0"# # ); ``` `sqlite_expr!` creates an [`Expression`](vantage_expressions::Expression) — a SQL template with typed, bound parameters. That `{}` is **not** Rust's `format!`: the value `false` is stored separately and bound through sqlx's parameterized query interface at execution time — no injection risk, ever. The preview shows it inline for readability. In SQL persistences, a condition is just an expression. When you pass it to `.with_condition()`, it's nested inside the select's own expression tree. At execution time the whole tree is [flattened](vantage_expressions::ExpressionFlattener) into a single template + parameter list that the database driver can bind safely. Run it — you should see 5 rows, with "A Stale Cake" filtered out. ```admonish info title="Types and persistence rendering" Notice that you passed `false` but the preview shows `0`. SQLite has no native boolean — Vantage's `SqliteType` implementation for `bool` converts it to an integer automatically. PostgreSQL would render `FALSE` instead. Each persistence maps Rust types to the [correct native representation](../sql/type-conversions.md). The `{}` parameter accepts any type that implements `SqliteType`: `bool`, `i64`, `f64`, `String`, `chrono::NaiveDate`, `Option`, and more. You can implement `SqliteType` for your own types too. See [Persistence-aligned Type System](../type-system.md) for details. ``` ## Typed columns and operators Writing `\"is_deleted\"` in a raw expression works, but there's a cleaner way. [`Column`](vantage_table::column::core::Column) creates a typed column reference, then chain an [`SqliteOperation`](vantage_sql::sqlite::operation::SqliteOperation) like `.eq()` to build the condition. (`Column` lives in the `vantage-table` crate, but the `vantage_sql` prelude re-exports it — no new dependency needed.) ```rust # use vantage_sql::prelude::*; let is_deleted = Column::::new("is_deleted"); let condition = is_deleted.eq(false); ``` The same condition, rendered as `is_deleted = 0` (a typed column isn't quoted), but the type parameter `` ensures you can only compare against matching types. Try `is_deleted.eq(42)` — it won't compile. Other operators — `.gt()`, `.lt()`, `.ne()`, `.in_()` — enforce the same type safety. The result is a [`SqliteCondition`](vantage_sql::condition::SqliteCondition) — the backend's native condition type — ready to be passed directly to `.with_condition()`. ```admonish info title="Type safety and backend-specific operations" Each SQL backend has its own operation trait — `SqliteOperation`, `PostgresOperation`, `MysqlOperation` — imported automatically via the prelude. Type safety is enforced at the column level; conditions themselves are expressions, so operations chain further (`price.gt(10).eq(false)` reads "price > 10 is false"). Operations take ownership of their arguments — `.clone()` a column you want to reuse. The [Expressive trait](../expressions.md#expressive-trait) reference covers the chaining mechanics. ``` Multiple conditions combine with AND: ```rust # use vantage_sql::prelude::*; let is_deleted = Column::::new("is_deleted"); let price = Column::::new("price"); let select = SqliteSelect::new() .with_source("product") .with_field("name") .with_condition(is_deleted.eq(false)) .with_condition(price.gt(150)); // ... WHERE is_deleted = 0 AND price > 150 # assert_eq!( # select.preview(), # r#"SELECT "name" FROM "product" WHERE is_deleted = 0 AND price > 150"# # ); ``` ```admonish info title="Primitives for untyped access" Typed columns aren't the only way to reference a column. **Primitives** are reusable building blocks for SQL expressions — they handle quoting, escaping, and vendor-specific syntax. The identifier primitive builds the same condition without declaring a type: ~~~rust use vantage_sql::sqlite::sqlite_ident as ident; let condition = ident("is_deleted").eq(false); ~~~ Primitives are not part of the prelude — import them when needed. Besides identifiers, you get `Fx` (function calls), `Case`, `Concat`, `Interval`, and more. The [Primitives reference](../sql/primitives.md) has the full list, including the typed (`sqlite_ident()`, `pg_ident()`) versus generic (`ident()`) identifier variants. ``` Primitives, query builders, `Column`, conditions — even native values like `i64` and `bool` — all implement `Expressive`, where `T` is your database's any-type (`AnySqliteType` for SQLite, `AnyMongoType` for MongoDB). That shared trait is what makes any of them usable as a parameter inside any expression. --- ## Working with Any-types [`Record`](vantage_types::Record) is an ordered map (`IndexMap`) — one record per row, with column names as keys. It lives in the `vantage-types` crate, so add that dependency and import its prelude: ```sh cargo add vantage-types --features serde ``` ```rust use vantage_types::prelude::*; # fn main() {} ``` So far we've been printing raw `AnySqliteType` with `Debug`. That works for verifying queries, but it's useless for real work. When `db.execute()` returns a multi-row result, the [`AnySqliteType`](vantage_sql::sqlite::AnySqliteType) holds an array of maps internally. Convert it to `Vec>` to work with individual rows: ```rust,no_run # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:products.db?mode=ro") # .await # .context("Failed to connect to products.db")?; # let select = SqliteSelect::new() # .with_source("product") # .with_field("name") # .with_field("price"); let raw = db.execute(&select.expr()).await?; let records = Vec::>::try_from(raw) .context("Failed to convert to records")?; for rec in &records { let name: String = rec["name"].try_get::().unwrap(); let price: i64 = rec["price"].try_get::().unwrap(); println!("{} — {} cents", name, price); } // Cupcake — 120 cents // Doughnut — 135 cents // ... # Ok(()) # } ``` Access fields by column name with `rec["name"]`. Each value is still an `AnySqliteType`, so you call `.try_get::()` to extract a typed Rust value. If the type doesn't match (say you call `.try_get::()` on a text column), you get `None` — no panics, no garbage. ```admonish warning title="serde_json::Value conversion" If you need JSON-friendly records, call `.into_record()` on each `Record`: ~~~rust let json_rec: Record = rec.into_record(); ~~~ This is convenient for serialization, but `serde_json::Value` supports a narrower set of types — you'll lose precision on `Decimal` and `chrono` types (dates become strings, decimals become floats). Stick with `Record` when you need full type fidelity. ``` Under the hood, each persistence has its own type system for storing values. SQLite uses [CBOR](https://cbor.io/) — a compact binary format that preserves types like `Decimal`, `NaiveDate`, and `NaiveDateTime` through tagged values. MongoDB uses [BSON](https://bsonspec.org/) natively. The `AnySqliteType` / `AnyMongoType` wrappers hide these details — you interact with `.try_get::()` regardless of which persistence you're using. If you ever need to inspect the raw representation, `.value()` gives you the underlying CBOR value: ```rust,no_run # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:products.db?mode=ro") # .await # .context("Failed to connect to products.db")?; # let select = SqliteSelect::new() # .with_source("product") # .with_field("name") # .with_field("price"); # let records = Vec::>::try_from(db.execute(&select.expr()).await?)?; # let rec = &records[0]; let price_cbor = rec["price"].value(); println!("{:?}", price_cbor); // Integer(Integer(120)) # Ok(()) # } ``` See [Persistence-aligned Type System](../type-system.md) for the full picture. --- ## Mapping rows to structs Calling `.try_get::()` on every field gets tedious. The `#[entity]` macro generates [`TryFromRecord`](vantage_types::TryFromRecord) for your struct, so conversion happens in one call with no type information lost: ```rust,no_run # use vantage_sql::prelude::*; # use vantage_types::prelude::*; #[entity(SqliteType)] struct Product { name: String, price: i64, } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:products.db?mode=ro") # .await # .context("Failed to connect to products.db")?; # let select = SqliteSelect::new() # .with_source("product") # .with_field("name") # .with_field("price"); let raw = db.execute(&select.expr()).await?; let records = Vec::>::try_from(raw)?; for rec in records { let product = Product::from_record(rec)?; println!("{} — {} cents", product.name, product.price); } # Ok(()) # } ``` The macro needs `vantage-core` as a direct dependency (it's already a transitive dep through `vantage-sql`, but the generated code references it in your crate): ```sh cargo add vantage-core ``` The macro also supports multiple type systems in one attribute — `#[entity(SqliteType, PostgresType, MongoType)]` generates a separate `TryFromRecord` impl for each persistence. One struct, all backends. ```admonish info title="Serde alternative" `#[entity]` converts each field directly through the persistence's type system — your struct fields just need to implement `SqliteType`. You could instead convert to `Record` (as above) and derive `serde::Deserialize` — but that funnels values through JSON, with the same precision loss. Prefer `#[entity]` when your schema includes decimals or dates. ``` --- ## Putting it together Here's the complete `src/main.rs` — connect, query, convert to entities, print: ```rust,no_run use vantage_sql::prelude::*; use vantage_types::prelude::*; #[entity(SqliteType)] struct Product { name: String, price: i64, } #[tokio::main] async fn main() { if let Err(e) = run().await { e.report(); } } async fn run() -> VantageResult<()> { let db = SqliteDB::connect("sqlite:products.db?mode=ro") .await .context("Failed to connect to products.db")?; let select = SqliteSelect::new() .with_source("product") .with_field("name") .with_field("price") .with_condition(Column::::new("is_deleted").eq(false)); let raw = db.execute(&select.expr()).await?; let records = Vec::>::try_from(raw)?; for rec in records { let p = Product::from_record(rec)?; println!("{:<12} {:>3} cents", p.name, p.price); } Ok(()) } ``` ```sh cargo run # Cupcake 120 cents # Doughnut 135 cents # Tart 220 cents # Pie 299 cents # Cookies 199 cents ``` --- ## What we covered | Concept | What it does | More info | | -------------------------------------------------------------------- | --------------------------------------------------------------------------------- | ------------------------------------------------------- | | [`SqliteSelect`](vantage_sql::sqlite::statements::SqliteSelect) | Builds SELECT queries via builder pattern | [`Selectable`](vantage_expressions::Selectable) | | [`Column`](vantage_table::column::core::Column) | Typed column reference — enforces matching operand types | | | [`SqliteOperation`](vantage_sql::sqlite::operation::SqliteOperation) | Ext trait giving `.eq()`, `.gt()`, etc. → `SqliteCondition` | | | `sqlite_expr!` | Creates expressions with typed, bound parameters | [`Expression`](vantage_expressions::Expression) | | `db.execute()` | Runs an expression, returns [`AnySqliteType`](vantage_sql::sqlite::AnySqliteType) | [`ExprDataSource`](vantage_expressions::ExprDataSource) | | [`Record`](vantage_types::Record) | Ordered map of column names to values — row-level access | `.try_get::()` | | `#[entity(SqliteType)]` | Generates lossless record-to-struct conversion | [`TryFromRecord`](vantage_types::TryFromRecord) | ```admonish tip title="Going deeper" This chapter used the smallest useful slice of the query layer. When you need more, the reference half of the book has it: - **[Expressions & Queries](../expressions.md)** — how expressions nest and flatten, building lists with `from_vec` (multi-row INSERTs), and `defer()` — a parameter whose value is fetched from a *different* database at execution time. - **[SQL Primitives](../sql/primitives.md)** — the building blocks we skipped: `or_()` / `and_()` grouping, `fx!` function calls, `Case`, `ternary()`, `concat_!`, `Interval`, and portable `date_format()`. - **[SQL: PostgreSQL, MySQL & SQLite](../sql.md)** — the other two SQL backends (same interface, different dialects) and the [type-conversion tables](../sql/type-conversions.md) showing exactly how chrono and decimal types round-trip per column type. - **[Three Paths — Query Building](../three-paths.md)** — worked examples of the heavy artillery: JOINs, CTEs, window functions, `DISTINCT ON`. None of it is required for the next chapter — come back when a query calls for it. ``` --- Source: https://vantage-ui.com/book/intro/step2-tables.html # Tables and Typed Data Access A table is a structure representing a collection of records in a database: ```rust,no_run # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # #[entity(SqliteType)] # #[derive(Clone, Default)] # struct Product { # name: String, # price: i64, # } # impl Product { # fn table(db: SqliteDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:products.db") # .await # .context("Failed to connect to products.db")?; let products = Product::table(db); for (id, product) in products.list().await? { println!("{id}: {} — {} cents", product.name, product.price); } # Ok(()) # } ``` Think of `Table` as a `Vec` — except the records aren't in memory. They live somewhere else (a database, a file, an API), and you don't know how many there are or what they contain until you list them. If you've used an ORM before, you might expect Vantage to work with individual records — load one, change it, save it back. Vantage works differently: you always operate on a **set** of records. A set might contain one record, no records, or millions — you don't pull them into memory to find out. Operations like counting, filtering, and updating happen on the database side — Vantage builds the right query and sends it over. Even traversal operations and sub-queries are done by building queries and executing remotely — if the database supports it, of course. When you `.clone()` a table, you don't clone the data — you clone the _definition_. From there you can narrow it down by adding conditions, turning it into a subset. Some examples of sets you might work with: - all user records except those with a soft-delete flag - orders placed today - orders of paid customers - customers who have an unpaid invoice older than 30 days - products that sold at least 5 items today Each of these is a set — defined by conditions, not a list of IDs — and one set can serve as the condition of another ("orders *of paid customers*"). Acting on a set is a single call: notify those customers, archive those orders, apply that discount. Put together, even complex operations are easy to express in code: ```rust,no_run # use vantage_sql::prelude::*; # use vantage_sql::primitives; # use vantage_types::prelude::*; # #[entity(SqliteType)] # #[derive(Clone, Default)] # struct Customer { # name: String, # } # #[entity(SqliteType)] # #[derive(Clone, Default)] # struct Invoice { # status: String, # } # #[entity(SqliteType)] # #[derive(Clone, Default)] # struct Order { # total: i64, # } # impl Customer { # fn table(db: SqliteDB) -> Table { # Table::new("customer", db) # .with_id_column("id") # .with_column_of::("name") # .with_many("orders", "customer_id", Order::table) # } # } # impl Invoice { # fn table(db: SqliteDB) -> Table { # Table::new("invoice", db) # .with_id_column("id") # .with_column_of::("status") # .with_column_of::("issued_at") # .with_one("customer", "customer_id", Customer::table) # } # } # impl Order { # fn table(db: SqliteDB) -> Table { # Table::new("order", db) # .with_id_column("id") # .with_column_of::("total") # .with_column_of::("placed_at") # } # } # fn days_ago(days: i64) -> Expression { # let offset = format!("-{days} days"); # sqlite_expr!("date('now', {})", offset) # } # trait InvoiceTable { # fn ref_customer(&self) -> Table; # } # impl InvoiceTable for Table { # fn ref_customer(&self) -> Table { # self.get_ref_as("customer").unwrap() # } # } # trait CustomerTable { # fn subquery_orders(&self) -> Table; # async fn send_reminder(&self) -> VantageResult<()>; # } # impl CustomerTable for Table { # fn subquery_orders(&self) -> Table { # self.get_subquery_as("orders").unwrap() # } # async fn send_reminder(&self) -> VantageResult<()> { # for (_, customer) in self.list().await? { # println!("reminder sent to {}", customer.name); # } # Ok(()) # } # } # trait OrderTable { # fn only_this_month(self) -> Self; # fn field_sum(&self, column: &str) -> Expression; # } # impl OrderTable for Table { # fn only_this_month(self) -> Self { # let placed_at = self["placed_at"].clone(); # self.with_condition(placed_at.gt(sqlite_expr!("date('now', 'start of month')"))) # } # fn field_sum(&self, column: &str) -> Expression { # self.get_sum_query(&self[column]) # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:shop.db").await.context("Failed to connect")?; // "notify customers who have an unpaid invoice older than 30 days" let mut overdue = Invoice::table(db.clone()); overdue.add_condition(overdue["status"].eq("unpaid")); overdue.add_condition(overdue["issued_at"].lt(days_ago(30))); overdue.ref_customer().send_reminder().await?; // "apply a discount to customers who spent more than $500 this month" let customers = Customer::table(db) .with_expression("spent_this_month", |c| { c.subquery_orders().only_this_month().field_sum("total") }) .with_expression("discount", |c| { let spent = c.get_column_expr("spent_this_month").unwrap(); primitives::ternary(spent.gt(500), 10, 0).expr() }); # Ok(()) # } ``` Don't worry about the exact syntax — we'll get to all of it. `ref_customer`, `send_reminder`, `subquery_orders`, `only_this_month` and `field_sum` aren't framework calls: they're methods the model defines on its own tables, and [Extension traits](#extension-traits) below shows how. The point is that sets compose naturally: define one, use it to narrow another, then act on the result. ## Defining a Table The syntax for defining a table looks similar to query building from chapter 1, but there are important differences: | | Query | Table | | ---------------- | --------------------------------------- | ------------------------------------------------ | | **Lifetime** | Built, executed, dropped | Sticks around, spawns many queries | | **Operations** | One SQL statement (SELECT, INSERT, ...) | Higher-level CRUD: list, get, add, patch, delete | | **Columns** | String field names via `.with_field()` | Typed `Column` definitions | | **Database** | Not bound — just a struct | Holds a database reference (`Arc`) | | **Idempotency** | INSERT fails on duplicate key | every write is retry-safe ([Safe writes](../record-lifecycle.md#safe-writes)) | | **Data sources** | Only databases with a query language | Any data source: SQL, CSV, APIs, queues, etc. | A [`Table`](vantage_table::table::Table) is typically defined in its own file alongside the entity. Create `src/product.rs`: ```rust // src/product.rs use vantage_sql::prelude::*; use vantage_types::prelude::*; #[entity(SqliteType)] #[derive(Clone, Default)] pub struct Product { pub name: String, pub price: i64, } impl Product { pub fn table(db: SqliteDB) -> Table { let is_deleted = Column::::new("is_deleted"); Table::new("product", db) .with_id_column("id") .with_column_of::("name") .with_column_of::("price") .with_column_of::("is_deleted") .with_condition(is_deleted.eq(false)) } } # fn main() {} ``` ```admonish tip title="Hiding the db argument" In most cases you already know which database a table lives in — passing `db` every time is noise. A common pattern is to wrap the connection in a global accessor: ~~~rust impl Product { fn table() -> Table { Table::new("product", get_sqlite_db()) // ...columns... } } ~~~ We'll keep passing `db` explicitly in this tutorial for clarity, but real applications typically use this pattern. ``` This defines the `product` table once. You can now generate queries from it: ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # #[entity(SqliteType)] # #[derive(Clone, Default)] # struct Product { # name: String, # price: i64, # } # impl Product { # fn table(db: SqliteDB) -> Table { # let is_deleted = Column::::new("is_deleted"); # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("is_deleted") # .with_condition(is_deleted.eq(false)) # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite::memory:").await.context("Failed to connect")?; let table = Product::table(db); // Full SELECT with all columns and conditions applied let select = table.select(); println!("{}", select.preview()); // SELECT "id", "name", "price", "is_deleted" FROM "product" WHERE is_deleted = 0 // Count query (does not execute — just builds the expression) let count_query = table.get_count_query(); // (SELECT COUNT(*) FROM "product" WHERE is_deleted = 0) // Sum query for a specific column let sum_query = table.get_sum_query(&table["price"]); // (SELECT SUM(price) FROM "product" WHERE is_deleted = 0) # assert_eq!( # select.preview(), # r#"SELECT "id", "name", "price", "is_deleted" FROM "product" WHERE is_deleted = 0"# # ); # assert_eq!( # count_query.preview(), # r#"(SELECT COUNT(*) FROM "product" WHERE is_deleted = 0)"# # ); # assert_eq!( # sum_query.preview(), # r#"(SELECT SUM(price) FROM "product" WHERE is_deleted = 0)"# # ); # Ok(()) # } ``` These return the same `SqliteSelect` and `Expression` types from chapter 1 — the table just assembles them for you, with its conditions already applied. The count and sum queries come wrapped in parentheses so they can nest inside another query as a subquery. But most of the time you won't need the raw query at all. --- ## CRUD operations With the table defined, all four operations — create, read, update, delete — are one-liners. Each comes from a trait in the prelude: ```rust,no_run # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # #[entity(SqliteType)] # #[derive(Clone, Default)] # struct Product { # name: String, # price: i64, # } # impl Product { # fn table(db: SqliteDB) -> Table { # let is_deleted = Column::::new("is_deleted"); # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("is_deleted") # .with_condition(is_deleted.eq(false)) # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:products.db") # .await # .context("Failed to connect to products.db")?; let table = Product::table(db); // Read — list all, get one by ID let all = table.list().await?; // IndexMap let pie = table.get("4").await?; // Option — id 4 is Pie // Create — insert with a known ID let muffin = Product { name: "Muffin".into(), price: 175 }; table.insert("8", &muffin).await?; // Update — replace the entire record let updated = Product { name: "Blueberry Muffin".into(), price: 195 }; table.replace("8", &updated).await?; // Delete table.delete("8").await?; # Ok(()) # } ``` That's it. `list()` returns an `IndexMap` — ordered and keyed by ID. `get()` returns `Option` — `None` when the ID doesn't exist, so you can pattern-match or `.ok_or(...)` into your own not-found error. There's also `get_some()` which returns an `(Id, Product)` pair (still `Option`-wrapped) for sampling, and `insert_return_id()` for when you want the database to generate the ID; see [Safe writes](../record-lifecycle.md#safe-writes) for why a client-made id is the retry-safe choice. ```admonish info title="Ids are strings" Our `id` column is `INTEGER PRIMARY KEY`, yet every id above is a string. That's deliberate: at the Vantage API level ids are always strings — the one type that can carry any backend's key (SQLite rowids, Mongo ObjectIds, S3 object keys). The persistence binds the string into the id-column comparison when it builds the query, so `get("4")` matches the integer `4`. You'll meet the same convention wherever ids travel through the framework — including the JSON output in the next chapter. ``` ```admonish tip title="Idempotent operations" Try repeating the `insert`, `replace` and `delete` calls — the result is the same: an insert of an existing id returns the stored row, a replace with the same values changes nothing, and deleting a row that's gone succeeds. See [Safe writes](../record-lifecycle.md#safe-writes). ``` --- ## Listing any table `Table` is great when you know the types at compile time. Sometimes you want a function that lists _any_ table — products, orders, customers — without naming the entity type. With **generics** you keep full type safety, limited to Rust callers: ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; async fn list_table(table: &D) -> VantageResult<()> where D: ReadableDataSet, D::Id: std::fmt::Display, E: Entity + std::fmt::Debug, { for (id, entity) in table.list().await? { println!(" {}: {:?}", id, entity); } Ok(()) } # fn main() {} ``` The entity bound names the dataset's own value type (`Entity`): an entity is only an entity *of* a particular type system, and that's what lets `list()` decode rows losslessly. To go fully **type-erased** — iterate columns by name, get runtime values, and expose tables outside Rust (a web admin UI, FFI, scripting) — wrap the table in a **Vista**, the universal schema-bearing data handle. That's the subject of [its own chapter](./step4-vista.md); `Vista` replaced the older `AnyTable` carrier in 0.5. --- ## Relationships Our database has a `category` table that we haven't used yet. Let's define it and connect it to products. Create `src/category.rs`: ```rust // src/category.rs use vantage_sql::prelude::*; use vantage_types::prelude::*; use crate::product::Product; #[entity(SqliteType)] #[derive(Debug, Clone, Default)] pub struct Category { pub name: String, } impl Category { pub fn table(db: SqliteDB) -> Table { Table::new("category", db) .with_id_column("id") .with_column_of::("name") .with_many("products", "category_id", Product::table) } } # mod product { # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # #[entity(SqliteType)] # #[derive(Clone, Default)] # pub struct Product { # pub name: String, # pub price: i64, # } # impl Product { # pub fn table(db: SqliteDB) -> Table { # let is_deleted = Column::::new("is_deleted"); # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("is_deleted") # .with_condition(is_deleted.eq(false)) # } # } # } # fn main() {} ``` The important line is `.with_many("products", "category_id", Product::table)`. This declares: - **"products"** — the name of the relationship (you'll use this to traverse it) - **"category_id"** — the foreign key column on the `product` table - **`Product::table`** — a function that builds the target table (same one we defined earlier) No changes to `Product` are needed — the relationship is declared on the side that "owns" it. Category has many products; the foreign key lives on product. Given a set of categories, `get_ref_as` traverses the relationship and returns matching products: ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # #[entity(SqliteType)] # #[derive(Clone, Default)] # struct Product { # name: String, # price: i64, # } # impl Product { # fn table(db: SqliteDB) -> Table { # let is_deleted = Column::::new("is_deleted"); # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("is_deleted") # .with_condition(is_deleted.eq(false)) # } # } # #[entity(SqliteType)] # #[derive(Clone, Default)] # struct Category { # name: String, # } # impl Category { # fn table(db: SqliteDB) -> Table { # Table::new("category", db) # .with_id_column("id") # .with_column_of::("name") # .with_many("products", "category_id", Product::table) # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite::memory:").await.context("Failed to connect")?; let categories = Category::table(db.clone()); let products = categories.get_ref_as::("products")?; # Ok(()) # } ``` The result is a `Table` with an extra condition — only products whose `category_id` matches one of the category IDs. Narrow the categories first and the subquery narrows too. Let's use this in a CLI that accepts an optional search filter. The `.with_search()` method adds a LIKE condition across all columns of the table — handy for quick filtering. Update `src/main.rs`. Declare the two modules at the top with `mod category;` and `mod product;`, then: ```rust,no_run # mod product { # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # #[entity(SqliteType)] # #[derive(Clone, Default)] # pub struct Product { # pub name: String, # pub price: i64, # } # impl Product { # pub fn table(db: SqliteDB) -> Table { # let is_deleted = Column::::new("is_deleted"); # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("is_deleted") # .with_condition(is_deleted.eq(false)) # } # } # } # mod category { # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # use crate::product::Product; # #[entity(SqliteType)] # #[derive(Clone, Default)] # pub struct Category { # pub name: String, # } # impl Category { # pub fn table(db: SqliteDB) -> Table { # Table::new("category", db) # .with_id_column("id") # .with_column_of::("name") # .with_many("products", "category_id", Product::table) # } # } # } use category::Category; use product::Product; use vantage_sql::prelude::*; async fn list_products(table: &Table) -> VantageResult<()> { for (id, p) in table.list().await? { println!(" {:<4} {:<20} {:>3} cents", id, p.name, p.price); } Ok(()) } #[tokio::main] async fn main() { if let Err(e) = run().await { e.report(); } } async fn run() -> VantageResult<()> { let db = SqliteDB::connect("sqlite:products.db") .await .context("Failed to connect to products.db")?; let filter = std::env::args().nth(1); let products = match &filter { Some(search) => Category::table(db.clone()) .with_search(search) .get_ref_as::("products")?, None => Product::table(db), }; list_products(&products).await?; Ok(()) } ``` Try it: ```sh cargo run # all active products cargo run "Sweet" # Cupcake, Doughnut, Cookies cargo run "Pastries" # Tart, Pie cargo run "t" # matches "Sweet Treats" AND "Pastries" — 5 products ``` The search `"t"` matches two categories ("Sweet Treats" and "Pastries"), so products from both are returned. ```admonish info title="Sets, not joins" Notice what happened with `cargo run "t"`: the search matched **two** categories, and we got products from **both** — without writing a JOIN or collecting IDs manually. This is because Vantage relationships work through **sets**. `with_search("t")` narrowed the category table to two rows. `get_ref_as` then generated a subquery: ~~~sql SELECT ... FROM "product" WHERE "category_id" IN (SELECT "id" FROM "category" WHERE "name" LIKE '%t%') AND "is_deleted" = 0 ~~~ The subquery comes from whatever conditions are on the source table. Add more conditions and the subquery narrows further. The relationship definition stays the same — you never need to rewrite the traversal logic. ``` --- ## Computed fields with expressions So far, `list_products` only shows name and price. It would be nice to show the category name too — but `Product` doesn't have a `category` field, and the category name lives in a different table. Vantage solves this with **expressions** — computed fields that are evaluated as part of the SELECT query. You define them on the table, and they appear alongside regular columns. First, add a `with_one` relationship on Product (the reverse of `with_many` on Category): ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # #[entity(SqliteType)] # #[derive(Clone, Default)] # struct Product { # name: String, # price: i64, # } # #[entity(SqliteType)] # #[derive(Clone, Default)] # struct Category { # name: String, # } # impl Category { # fn table(db: SqliteDB) -> Table { # Table::new("category", db) # .with_id_column("id") # .with_column_of::("name") # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite::memory:").await.context("Failed to connect")?; # let products: Table = Table::new("product", db) # .with_id_column("id") # .with_column_of::("category_id") .with_one("category", "category_id", Category::table) # ; # Ok(()) # } ``` This says: each product has one category, linked through `category_id`. Now you can use `get_subquery_as` inside `with_expression` to build a correlated subquery: ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # #[entity(SqliteType)] # #[derive(Clone, Default)] # struct Product { # name: String, # price: i64, # } # #[entity(SqliteType)] # #[derive(Clone, Default)] # struct Category { # name: String, # } # impl Category { # fn table(db: SqliteDB) -> Table { # Table::new("category", db) # .with_id_column("id") # .with_column_of::("name") # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite::memory:").await.context("Failed to connect")?; # let products: Table = Table::new("product", db) # .with_id_column("id") # .with_column_of::("category_id") # .with_one("category", "category_id", Category::table) .with_expression("category", |t| { t.get_subquery_as::("category") .unwrap() .select_column("name") .expect("Category has a 'name' column") }) # ; # Ok(()) # } ``` `select_column("name")` builds a subquery like `SELECT "name" FROM "category" WHERE "id" = "product"."category_id"` — a correlated subquery that fetches the category name for each product row. It returns an `Option`, `None` when the target table has no such column; the name here is hard-coded and known to exist, so `.expect()` is the honest unwrap. Here's the updated `src/product.rs`: ```rust use vantage_sql::prelude::*; use vantage_types::prelude::*; use crate::category::Category; #[entity(SqliteType)] #[derive(Debug, Clone, Default)] pub struct Product { pub name: String, pub price: i64, pub category: Option, } impl Product { pub fn table(db: SqliteDB) -> Table { let is_deleted = Column::::new("is_deleted"); Table::new("product", db) .with_id_column("id") .with_column_of::("name") .with_column_of::("price") .with_column_of::("is_deleted") .with_condition(is_deleted.eq(false)) .with_one("category", "category_id", Category::table) .with_expression("category", |t| { t.get_subquery_as::("category") .unwrap() .select_column("name") .expect("Category has a 'name' column") }) } } # mod category { # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # use crate::Product; # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # pub struct Category { # pub name: String, # } # impl Category { # pub fn table(db: SqliteDB) -> Table { # Table::new("category", db) # .with_id_column("id") # .with_column_of::("name") # .with_many("products", "category_id", Product::table) # } # } # } # fn main() {} ``` A few things to note: - **`category: Option`** — the field is `Option` because a product might not have a category (NULL in the database). The expression result is deserialized into this field automatically. - **No `with_column_of` for "category"** — the expression replaces what would normally be a column. Vantage adds it to the SELECT as a computed field. - **`with_one` + `with_expression`** — the relationship defines how to traverse; the expression defines what to fetch. They work together but serve different purposes. Update `list_products` to show the category: ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # struct Product { # name: String, # price: i64, # category: Option, # } async fn list_products(table: &Table) -> VantageResult<()> { for (id, p) in table.list().await? { let cat = p.category.as_deref().unwrap_or("-"); println!(" {:<4} {:<20} {:>3} cents [{}]", id, p.name, p.price, cat); } Ok(()) } # fn main() {} ``` Run it: ```sh cargo run ``` ```text 1 Cupcake 120 cents [Sweet Treats] 2 Doughnut 135 cents [Sweet Treats] 3 Tart 220 cents [Pastries] 4 Pie 299 cents [Pastries] 5 Cookies 199 cents [Sweet Treats] 7 Sourdough Loaf 350 cents [Breads] ``` The category name comes from a subquery — no JOIN, no extra round trip. And the category filter still works: ```sh cargo run "t" ``` ```text 1 Cupcake 120 cents [Sweet Treats] 2 Doughnut 135 cents [Sweet Treats] 3 Tart 220 cents [Pastries] 4 Pie 299 cents [Pastries] 5 Cookies 199 cents [Sweet Treats] ``` ```admonish info title="Expressions compose" Expressions can reference other expressions. For example, we could add a `product_count` expression to Category that counts products via a subquery, then build a `title` expression that combines the name and count: ~~~rust // In Category::table() .with_expression("product_count", |t| { t.get_subquery_as::("products") .unwrap() .get_count_query() }) .with_expression("title", |t| { let name = t.get_column_expr("name").unwrap(); let count = t.get_column_expr("product_count").unwrap(); concat_!(name, " (", count, ")").expr() }) ~~~ `get_column_expr` returns the expression for either a real column or a computed expression by name. Here it resolves `"product_count"` to the count subquery, so `title` becomes something like `name || ' (' || (SELECT COUNT(*) FROM product WHERE ...) || ')'`. Now change Product's `category` expression to fetch `title` instead of `name`: ~~~rust .with_expression("category", |t| { t.get_subquery_as::("category") .unwrap() .select_column("title") .expect("Category has a 'title' column") }) ~~~ `select_column("title")` resolves the `title` expression on Category and builds it into the subquery. The result: ~~~text 1 Cupcake 120 cents [Sweet Treats (3)] 2 Doughnut 135 cents [Sweet Treats (3)] 3 Tart 220 cents [Pastries (2)] 4 Pie 299 cents [Pastries (2)] 5 Cookies 199 cents [Sweet Treats (3)] 7 Sourdough Loaf 350 cents [Breads (1)] ~~~ Vantage renders everything into a single SQL query. The nested subqueries might look redundant, but modern SQL databases optimise and execute them efficiently — there's no extra round trip to the database. ``` --- ## Extension traits Throughout this chapter you've seen calls like `categories.get_ref_as::("products")`. The turbofish `::` and the string `"products"` are repetitive and error-prone — get the string wrong and you get a runtime error. Rust's extension traits solve this. Define a trait on `Table` that wraps the traversal: ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # #[entity(SqliteType)] # #[derive(Clone, Default)] # pub struct Product { # name: String, # price: i64, # } # #[entity(SqliteType)] # #[derive(Clone, Default)] # pub struct Category { # name: String, # } pub trait CategoryTable { fn ref_products(&self) -> Table; } impl CategoryTable for Table { fn ref_products(&self) -> Table { self.get_ref_as("products").unwrap() } } # fn main() {} ``` The trait declares the vocabulary; the `impl` block supplies the bodies. Nothing is auto-implemented — `ref_products()` appears on `Table` the moment you write this impl, and on nothing else. The bodies must live impl-side, too: methods like `get_ref_as` are inherent to `Table`, so they're only reachable where `self` is concretely that table — inside the impl, not inside a trait default. The `unwrap()` is safe because we know "products" is defined on every `CategoryTable`. If the string were wrong, it would panic immediately during development, not silently fail at runtime. Now callers write `categories.ref_products()` — no turbofish, no string, no `?`, and the compiler catches typos. The same pattern works for typed column access: ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # #[entity(SqliteType)] # #[derive(Clone, Default)] # pub struct Product { # name: String, # price: i64, # } pub trait ProductTable { fn price(&self) -> Column; } impl ProductTable for Table { fn price(&self) -> Column { self.get_column("price").unwrap() } } # fn main() {} ``` With both traits in place, code reads naturally: ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # #[entity(SqliteType)] # #[derive(Clone, Default)] # pub struct Product { # name: String, # price: i64, # } # impl Product { # pub fn table(db: SqliteDB) -> Table { # let is_deleted = Column::::new("is_deleted"); # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("is_deleted") # .with_condition(is_deleted.eq(false)) # } # } # #[entity(SqliteType)] # #[derive(Clone, Default)] # pub struct Category { # name: String, # } # impl Category { # pub fn table(db: SqliteDB) -> Table { # Table::new("category", db) # .with_id_column("id") # .with_column_of::("name") # .with_many("products", "category_id", Product::table) # } # } # pub trait CategoryTable { # fn ref_products(&self) -> Table; # } # impl CategoryTable for Table { # fn ref_products(&self) -> Table { # self.get_ref_as("products").unwrap() # } # } # pub trait ProductTable { # fn price(&self) -> Column; # } # impl ProductTable for Table { # fn price(&self) -> Column { # self.get_column("price").unwrap() # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite::memory:").await.context("Failed to connect")?; # let categories = Category::table(db); let products = categories.ref_products(); let price = products.price(); let expensive = products.with_condition(price.gt(200)); # assert!(expensive.select().preview().contains("price > 200")); # Ok(()) # } ``` `price()` is read before `with_condition` because `with_condition` consumes the table it's called on; taking the column first keeps the borrow and the move apart. ```admonish tip title="Where to put extension traits" Define extension traits in the same file as the entity — `product.rs` gets `ProductTable`, `category.rs` gets `CategoryTable`. They're part of your model's public API: anyone who imports the entity gets the typed accessors for free. This is how Vantage scales to large codebases. The table definition (columns, conditions, relationships, expressions) is declared once. Extension traits give it a clean, typed API. Business logic code never sees raw strings or turbofish — just method calls. ``` ```admonish info title="Custom methods on extension traits" Extension traits aren't limited to column and relationship accessors — you can add any business logic. For example, the `vantage-cli-util` crate provides `print_table()`, which accepts any `&Table` and renders it for the terminal. Declare the method on your trait and implement it in the same impl block as the accessors: ~~~rust pub trait ProductTable { fn price(&self) -> Column; async fn print(&self) -> VantageResult<()>; } impl ProductTable for Table { fn price(&self) -> Column { self.get_column("price").unwrap() } async fn print(&self) -> VantageResult<()> { vantage_cli_util::print_table(self).await } } ~~~ Now `products.print().await?` gives you a formatted table in the terminal: ~~~text +----+----------------+-------+-------------------+ | id | name | price | category | +----+----------------+-------+-------------------+ | 1 | Cupcake | 120 | Sweet Treats (3) | | 2 | Doughnut | 135 | Sweet Treats (3) | | 3 | Tart | 220 | Pastries (2) | | ... | +----+----------------+-------+-------------------+ ~~~ Other methods you might add: `only_expensive()` that returns a filtered clone, `total_revenue()` that computes a sum, or `export_csv()` that writes to a file. The table is yours to extend. ``` --- ## Persistence Abstraction The files you have written so far — `Cargo.toml`, `product.rs`, `category.rs`, `main.rs` — are the makings of an extensible business software architecture: scalable, maintainable, testable, and portable across databases without rewriting a single line of business logic. What you have is four distinct components, each with a clear job: ```mermaid flowchart TD A["Business code
main.rs
minimal, domain-focused"] B["Model definitions
product.rs, category.rs
entities, tables, relationships, domain methods"] C["Persistence crate
vantage-sql (or surrealdb, mongodb, csv…)
query builder, CRUD, type system"] D["Vantage framework
vantage-table, vantage-expressions, vantage-types
the extensible mechanism that ties it together"] A --> B B --> C C --> D ``` The majority of your business code can work with data without any knowledge of where it's stored or how. It operates on typed entities, relationships, and domain methods — nothing else. The model definitions focus on describing _where_ and _how_ data is stored, but they don't micro-manage the process. If storage requirements change — a new backend, a schema migration, a cached layer in front — the model is where you make the tweaks. Business code remains untouched. This separation gives you the ergonomics scripting-language ORMs are known for — concise, domain-focused code — with Rust's type safety and performance underneath: the abstraction compiles down to direct calls. When storage requirements shift, the change stays contained in the model layer instead of rippling through business code, and the whole architecture travels as a single static binary — server, desktop, or embedded. ```admonish tip title="Going deeper" `Table` has considerably more surface than one chapter can show. The reference pages pick up where this one stops: - **[Records: Traversal, Invariants & Hooks](../record-lifecycle.md)** — lifecycle hooks (audit stamps, validation, soft-delete that intercepts a real delete), **set invariants** that fill and enforce foreign keys on every write through a relation, and traversing relations from a single loaded record. - **[Expressions & Queries](../expressions.md)** — the machinery underneath `with_expression` and every condition you wrote in this chapter. - **[Model-Driven Architecture](../mda.md)** — how entities, table constructors, and connection management are organised in a real model crate, including one entity with constructors for several backends. - **[Three Paths for Developers](../three-paths.md)** — how this chapter's entity framework relates to raw query building and custom persistences, and when to reach for each. ``` ```admonish tip title="A sneak peek at what's next" The pattern you've just built scales well beyond a product catalog: - **Client-side caching and reactive data** — the second half of this guide puts a model behind a persistent cache with live updates streaming to a UI, over a slow external API or your own relational database — whichever path matches your problem. - **Cross-persistence traversal** — follow a relationship from a Postgres table into a MongoDB collection, expressed in Rust, executed efficiently on each side. - **UI framework integration** — the same model drives desktop, terminal, and web UIs through `vantage-ui-adapters`: define once, render anywhere. ``` --- Source: https://vantage-ui.com/book/intro/step3-axum-server.html # A Standalone Axum Server Chapter 2 ended with a CLI that prints products to stdout. In this chapter we put the same model behind an HTTP API: ```sh curl "http://localhost:3001/categories" ``` ```json [{ "name": "Sweet Treats" }, { "name": "Pastries" }, { "name": "Breads" }] ``` ```sh # Products in a specific category curl "http://localhost:3001/categories/1/products" # Writes curl -X POST "http://localhost:3001/categories" \ -H 'content-type: application/json' -d '{"name":"Gluten-Free"}' ``` A few things about the shape of this API: - `/categories/{id}/products` is a **nested** route — products narrowed by the relationship we defined in chapter 2. The handler for it is the same generic `list` as `/categories`, just applied to a scoped table. - `/products` and `/categories` start out on SQLite, same as chapter 2. Toward the end of the chapter we **migrate** them to MongoDB. That migration is a change to the model file; the handlers and routes don't know the difference. The handler functions are each written **once** and mounted against any [`Table`](vantage_table::table::Table) the router has on hand — one `list`, one `get`, one `post`, one `patch`, one `delete`, parameterised over the backend and entity types. Adding another entity is a route registration, not a new handler. --- ## The minimum Axum skeleton Two pieces: a small adaptation to chapter 2's entities so they round-trip through HTTP, then the server itself. One new dependency: ```sh cargo add axum ``` Tokio and serde are already pulled in from earlier chapters. `axum` is the HTTP framework; it uses the existing `serde` to encode response bodies as JSON. **Entities.** Chapter 2's `Product` carried a computed `category` field, and the *Expressions compose* callout showed `Category` gaining a computed `title`. Both were assembled from subqueries — great for display, but they don't round-trip cleanly through a `POST` body because the JSON would try to write columns that don't exist. For a writable API we replace the computed fields with the plain FK column that sits under them. (See [`with_expression`](vantage_table::table::Table::with_expression) for the chapter-2 computed-field setup we're walking away from here.) `src/category.rs`: ```rust use serde::{Deserialize, Serialize}; use vantage_sql::prelude::*; use vantage_types::prelude::*; use crate::product::Product; #[entity(SqliteType)] #[derive(Debug, Clone, Default, Serialize, Deserialize)] pub struct Category { pub name: String, } impl Category { pub fn table(db: SqliteDB) -> Table { Table::new("category", db) .with_id_column("id") .with_column_of::("name") .with_many("products", "category_id", Product::table) } } # mod product { # use serde::{Deserialize, Serialize}; # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # use crate::Category; # #[entity(SqliteType)] # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # pub struct Product { # pub name: String, # pub price: i64, # #[serde(skip_serializing_if = "Option::is_none", default)] # pub category_id: Option, # #[serde(default)] # pub is_deleted: bool, # } # impl Product { # pub fn table(db: SqliteDB) -> Table { # let is_deleted = Column::::new("is_deleted"); # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("category_id") # .with_column_of::("is_deleted") # .with_condition(is_deleted.eq(false)) # .with_one("category", "category_id", Category::table) # } # } # } # fn main() {} ``` `src/product.rs`: ```rust use serde::{Deserialize, Serialize}; use vantage_sql::prelude::*; use vantage_types::prelude::*; use crate::category::Category; #[entity(SqliteType)] #[derive(Debug, Clone, Default, Serialize, Deserialize)] pub struct Product { pub name: String, pub price: i64, #[serde(skip_serializing_if = "Option::is_none", default)] pub category_id: Option, #[serde(default)] pub is_deleted: bool, } impl Product { pub fn table(db: SqliteDB) -> Table { let is_deleted = Column::::new("is_deleted"); Table::new("product", db) .with_id_column("id") .with_column_of::("name") .with_column_of::("price") .with_column_of::("category_id") .with_column_of::("is_deleted") .with_condition(is_deleted.eq(false)) .with_one("category", "category_id", Category::table) } } # mod category { # use serde::{Deserialize, Serialize}; # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # use crate::Product; # #[entity(SqliteType)] # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # pub struct Category { # pub name: String, # } # impl Category { # pub fn table(db: SqliteDB) -> Table { # Table::new("category", db) # .with_id_column("id") # .with_column_of::("name") # .with_many("products", "category_id", Product::table) # } # } # } # fn main() {} ``` Two things to notice: - `category_id` is now a real column in both the struct and the table, not a computed expression. Same information as before, but clients can read it from `GET` and supply it on `POST`. - `is_deleted` picks up `#[serde(default)]` so `POST /products` bodies don't have to carry it; new products default to live. The soft-delete condition from chapter 2 stays — `GET /products` will still hide rows with `is_deleted = true`. Now the server. Replace `src/main.rs` with the code below, keeping chapter 2's `mod category;` and `mod product;` declarations at the top: ```rust,no_run # mod product { # use serde::{Deserialize, Serialize}; # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # use crate::category::Category; # #[entity(SqliteType)] # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # pub struct Product { # pub name: String, # pub price: i64, # #[serde(skip_serializing_if = "Option::is_none", default)] # pub category_id: Option, # #[serde(default)] # pub is_deleted: bool, # } # impl Product { # pub fn table(db: SqliteDB) -> Table { # let is_deleted = Column::::new("is_deleted"); # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("category_id") # .with_column_of::("is_deleted") # .with_condition(is_deleted.eq(false)) # .with_one("category", "category_id", Category::table) # } # } # } # mod category { # use serde::{Deserialize, Serialize}; # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # use crate::product::Product; # #[entity(SqliteType)] # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # pub struct Category { # pub name: String, # } # impl Category { # pub fn table(db: SqliteDB) -> Table { # Table::new("category", db) # .with_id_column("id") # .with_column_of::("name") # .with_many("products", "category_id", Product::table) # } # } # } use std::sync::OnceLock; use axum::{routing::get, Json, Router}; use category::Category; use vantage_sql::prelude::*; static DB: OnceLock = OnceLock::new(); fn db() -> SqliteDB { DB.get().expect("database not initialised").clone() } async fn list_categories() -> Json> { let rows = Category::table(db()).list().await.unwrap(); Json(rows.into_values().collect()) } #[tokio::main] async fn main() -> VantageResult<()> { let conn = SqliteDB::connect("sqlite:products.db") .await .context("Failed to connect to products.db")?; DB.set(conn).ok(); let app = Router::new().route("/categories", get(list_categories)); let listener = tokio::net::TcpListener::bind("0.0.0.0:3001").await.unwrap(); axum::serve(listener, app).await.unwrap(); Ok(()) } ``` A few notes on what this is doing: - **`static DB: OnceLock<`[`SqliteDB`](vantage_sql::sqlite::SqliteDB)`>`.** The handler needs a database handle but we don't want to open a new connection per request. A program-wide `OnceLock` is the simplest possible holder — set once in `main`, read from anywhere. - **Each request calls `Category::table(db())`** — building the table (columns, relationships) and then lists it. - **The handler returns `Json>` directly.** No DTO layer — the entity's fields become the JSON fields. Run it: ```sh cargo run ``` In another terminal: ```sh curl http://localhost:3001/categories ``` ```json [ { "name": "Sweet Treats" }, { "name": "Pastries" }, { "name": "Breads" } ] ``` --- ## Caching table definitions We can cache each table definition so it's built once and handed out by reference on every subsequent call. In `src/category.rs`, add `use std::sync::OnceLock;` at the top and wrap the body of `Category::table`: ```rust # use serde::{Deserialize, Serialize}; # use vantage_sql::prelude::*; # use vantage_types::prelude::*; use std::sync::OnceLock; # #[entity(SqliteType)] # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # pub struct Category { # pub name: String, # } impl Category { pub fn table(db: SqliteDB) -> &'static Table { static CACHE: OnceLock> = OnceLock::new(); CACHE.get_or_init(|| { Table::new("category", db) .with_id_column("id") .with_column_of::("name") .with_many("products", "category_id", |db| Product::table(db).clone()) }) } } # #[entity(SqliteType)] # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # pub struct Product { # pub name: String, # pub price: i64, # #[serde(skip_serializing_if = "Option::is_none", default)] # pub category_id: Option, # #[serde(default)] # pub is_deleted: bool, # } # impl Product { # pub fn table(db: SqliteDB) -> &'static Table { # static CACHE: OnceLock> = OnceLock::new(); # CACHE.get_or_init(|| { # let is_deleted = Column::::new("is_deleted"); # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("category_id") # .with_column_of::("is_deleted") # .with_condition(is_deleted.eq(false)) # .with_one("category", "category_id", |db| Category::table(db).clone()) # }) # } # } # fn main() {} ``` Three changes from chapter 2's version: - **Return type is `&'static Table<...>` instead of `Table<...>`.** The cache owns the definition; callers get a shared reference. Most table operations ([`list`](vantage_dataset::traits::ReadableDataSet::list), [`get`](vantage_dataset::traits::ReadableDataSet::get), [`insert`](vantage_dataset::traits::WritableDataSet::insert), [`replace`](vantage_dataset::traits::WritableDataSet::replace), [`delete`](vantage_dataset::traits::WritableValueSet::delete), [`get_ref_as`](vantage_table::table::Table::get_ref_as)) take `&self`, so they work on the reference without any clone. - **`static CACHE: OnceLock<...>` lives inside the function.** A function-local static is still a program-wide single instance — Rust allows this and it keeps the cache private to the table that owns it. - **The [`with_many`](vantage_table::table::Table::with_many) callback is a closure: `|db| Product::table(db).clone()`.** The framework still hands us a db and expects an owned `Table` back, so we call the cached accessor and clone the reference. On the first call this triggers Product's cache to build; every subsequent call just clones a pre-built definition. `src/product.rs` follows the same pattern — wrap the body in `OnceLock::get_or_init`, change the return type to `&'static Table`, and rewrite `with_one` as `|db| Category::table(db).clone()`. The handler in `main.rs` doesn't change at all — `Category::table(db())` used to return an owned `Table`, and now it returns a `&'static Table` that auto-derefs through the same `.list()` call. Restart the server and hit `/categories` — the response is identical, but now the table definitions are built on the first request and reused forever after. ```admonish info title="When the cached table needs narrowing" Most operations work on `&Table<...>`, but the builder methods that narrow it — like `with_condition`, `with_search`, and `with_order` — consume `self` and return a new `Table`, and the setters (`add_condition`, `set_pagination`) need `&mut`. If a handler needs to narrow the cached definition, it clones first: ~~~rust let narrowed = Category::table(db()) .clone() // owned copy of the cached definition .with_condition(...) // now we can narrow it ~~~ That's chapter 2's rule in action — cloning a table copies the definition, and it's cheap. ``` --- ## Products of a category The [`with_many`](vantage_table::table::Table::with_many) relationship we kept from chapter 2 lets us serve a nested route — `/categories/{id}/products`. Before writing the handler, give the relationship a proper name with an extension trait on `Table`. Add this to `src/category.rs`: ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # pub struct Category { # pub name: String, # } # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # pub struct Product { # pub name: String, # } pub trait CategoryTable { fn ref_products(&self) -> Table; } impl CategoryTable for Table { fn ref_products(&self) -> Table { self.get_ref_as("products").unwrap() } } # fn main() {} ``` Chapter 2 introduced this pattern — a typed, discoverable name instead of the string-and-turbofish call. The `.unwrap()` is safe here because we're the ones who registered `"products"`; a typo surfaces immediately at startup. Now the handler. Take the cached category table, narrow it to a single id, and traverse the relationship: ```rust # mod product { # use std::sync::OnceLock; # use serde::{Deserialize, Serialize}; # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # use crate::category::Category; # #[entity(SqliteType)] # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # pub struct Product { # pub name: String, # pub price: i64, # #[serde(skip_serializing_if = "Option::is_none", default)] # pub category_id: Option, # #[serde(default)] # pub is_deleted: bool, # } # impl Product { # pub fn table(db: SqliteDB) -> &'static Table { # static CACHE: OnceLock> = OnceLock::new(); # CACHE.get_or_init(|| { # let is_deleted = Column::::new("is_deleted"); # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("category_id") # .with_column_of::("is_deleted") # .with_condition(is_deleted.eq(false)) # .with_one("category", "category_id", |db| Category::table(db).clone()) # }) # } # } # } # mod category { # use std::sync::OnceLock; # use serde::{Deserialize, Serialize}; # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # use crate::product::Product; # #[entity(SqliteType)] # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # pub struct Category { # pub name: String, # } # impl Category { # pub fn table(db: SqliteDB) -> &'static Table { # static CACHE: OnceLock> = OnceLock::new(); # CACHE.get_or_init(|| { # Table::new("category", db) # .with_id_column("id") # .with_column_of::("name") # .with_many("products", "category_id", |db| Product::table(db).clone()) # }) # } # } # pub trait CategoryTable { # fn ref_products(&self) -> Table; # } # impl CategoryTable for Table { # fn ref_products(&self) -> Table { # self.get_ref_as("products").unwrap() # } # } # } # use std::sync::OnceLock; # use axum::Json; # use category::Category; # use vantage_sql::prelude::*; # static DB: OnceLock = OnceLock::new(); # fn db() -> SqliteDB { # DB.get().expect("database not initialised").clone() # } use axum::extract::Path; use category::CategoryTable; use product::Product; async fn list_category_products(Path(id): Path) -> Json> { let id_col = Column::::new("id"); let products = Category::table(db()) .clone() .with_condition(id_col.eq(id)) .ref_products(); let rows = products.list().await.unwrap(); Json(rows.into_values().collect()) } # fn main() {} ``` Three things going on: - **`Category::table(db()).clone()`** — we need an owned [`Table`](vantage_table::table::Table) to chain [`with_condition`](vantage_table::table::Table::with_condition) onto, so we clone the cached definition — the shape, not any rows. - **`with_condition(id_col.eq(id))`** — narrows the category table to one row: the one we're asking for. Nothing hits the database yet. - **`ref_products()`** — traverses the `with_many` relationship we registered on Category in chapter 2, via [`get_ref_as`](vantage_table::table::Table::get_ref_as). The returned table is `Table`, already scoped to products whose `category_id` matches the narrowed category set. Chapter 2 walked through what the emitted SQL looks like; it's the same here. Register the route in `main.rs`: ```rust # mod product { # use std::sync::OnceLock; # use serde::{Deserialize, Serialize}; # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # use crate::category::Category; # #[entity(SqliteType)] # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # pub struct Product { # pub name: String, # pub price: i64, # #[serde(skip_serializing_if = "Option::is_none", default)] # pub category_id: Option, # #[serde(default)] # pub is_deleted: bool, # } # impl Product { # pub fn table(db: SqliteDB) -> &'static Table { # static CACHE: OnceLock> = OnceLock::new(); # CACHE.get_or_init(|| { # let is_deleted = Column::::new("is_deleted"); # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("category_id") # .with_column_of::("is_deleted") # .with_condition(is_deleted.eq(false)) # .with_one("category", "category_id", |db| Category::table(db).clone()) # }) # } # } # } # mod category { # use std::sync::OnceLock; # use serde::{Deserialize, Serialize}; # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # use crate::product::Product; # #[entity(SqliteType)] # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # pub struct Category { # pub name: String, # } # impl Category { # pub fn table(db: SqliteDB) -> &'static Table { # static CACHE: OnceLock> = OnceLock::new(); # CACHE.get_or_init(|| { # Table::new("category", db) # .with_id_column("id") # .with_column_of::("name") # .with_many("products", "category_id", |db| Product::table(db).clone()) # }) # } # } # pub trait CategoryTable { # fn ref_products(&self) -> Table; # } # impl CategoryTable for Table { # fn ref_products(&self) -> Table { # self.get_ref_as("products").unwrap() # } # } # } # use std::sync::OnceLock; # use axum::extract::Path; # use axum::{Json, Router, routing::get}; # use category::{Category, CategoryTable}; # use product::Product; # use vantage_sql::prelude::*; # static DB: OnceLock = OnceLock::new(); # fn db() -> SqliteDB { # DB.get().expect("database not initialised").clone() # } # async fn list_categories() -> Json> { # let rows = Category::table(db()).list().await.unwrap(); # Json(rows.into_values().collect()) # } # async fn list_category_products(Path(id): Path) -> Json> { # let id_col = Column::::new("id"); # let products = Category::table(db()) # .clone() # .with_condition(id_col.eq(id)) # .ref_products(); # let rows = products.list().await.unwrap(); # Json(rows.into_values().collect()) # } # fn main() { let app = Router::new() .route("/categories", get(list_categories)) .route("/categories/{id}/products", get(list_category_products)); # let _: Router = app; # } ``` Hit it: ```sh curl http://localhost:3001/categories/1/products ``` ```json [ { "name": "Cupcake", "price": 120, "category_id": "1", "is_deleted": false }, { "name": "Doughnut", "price": 135, "category_id": "1", "is_deleted": false }, { "name": "Cookies", "price": 199, "category_id": "1", "is_deleted": false } ] ``` ```sh curl http://localhost:3001/categories/2/products ``` ```json [ { "name": "Tart", "price": 220, "category_id": "2", "is_deleted": false }, { "name": "Pie", "price": 299, "category_id": "2", "is_deleted": false } ] ``` Swap the id in the URL and the products narrow accordingly. The relationship was declared once, back in chapter 2, and we haven't touched it since — every new nested route reuses the same declaration. Notice `"category_id": "1"` in the responses — a string, even though the SQLite column is INTEGER. This is chapter 2's id convention at work: ids travel as strings through Vantage, so the entity declares `category_id: Option` and the value converts on its way out of the database. --- ## A generic `crud` helper Two handlers so far — `list_categories` and `list_category_products` — and they're already doing the same thing: take a narrowed table, call `.list()`, return JSON. Adding `POST /categories`, `PATCH /categories/{id}`, and `DELETE /categories/{id}` would mean four more near-identical handlers per entity. That's not how you scale a codebase. The five HTTP methods of CRUD all map onto one of two operation shapes: - **List-level** at `/collection`: `GET` (list all) and `POST` (create one). - **Item-level** at `/collection/{id}`: `GET` (read one), `PATCH` (update), `DELETE` (remove). If we can describe the _set of rows this endpoint operates on_ with one closure, the same handler bodies serve every entity. That closure is `Fn(SqliteDB, &Params) -> Table` — given the database and whatever path params axum extracted, return the `Table` to act on. Put it in `main.rs`: ```rust # use std::sync::OnceLock; # use axum::extract::Path; # use axum::http::StatusCode; # use axum::{Json, Router, routing::get}; # use serde::{Serialize, de::DeserializeOwned}; # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # static DB: OnceLock = OnceLock::new(); # fn db() -> SqliteDB { # DB.get().expect("database not initialised").clone() # } use std::collections::HashMap; use std::sync::Arc; type Params = HashMap; fn crud(make_table: F) -> Router where F: Fn(SqliteDB, &Params) -> Table + Send + Sync + 'static, E: Entity + Serialize + DeserializeOwned + Send + Sync + 'static, >::Error: std::fmt::Debug, { let f = Arc::new(make_table); Router::new() .route( "/", get({ let f = f.clone(); move |p: Option>| async move { let params = p.map(|Path(p)| p).unwrap_or_default(); let rows = f(db(), ¶ms).list().await.unwrap(); Json::>(rows.into_values().collect()) } }) .post({ let f = f.clone(); move |p: Option>, Json(entity): Json| async move { let params = p.map(|Path(p)| p).unwrap_or_default(); let id = f(db(), ¶ms).insert_return_id(&entity).await.unwrap(); Json(serde_json::json!({ "id": id })) } }), ) .route( "/{id}", get({ let f = f.clone(); move |Path(params): Path| async move { let id = params["id"].clone(); let entity = f(db(), ¶ms).get(&id).await.unwrap().unwrap(); Json(entity) } }) .patch({ let f = f.clone(); move |Path(params): Path, Json(partial): Json| async move { let id = params["id"].clone(); let updated = f(db(), ¶ms).patch(&id, &partial).await.unwrap(); Json(updated) } }) .delete({ let f = f; move |Path(params): Path| async move { let id = params["id"].clone(); f(db(), ¶ms).delete(&id).await.unwrap(); StatusCode::NO_CONTENT } }), ) } # fn main() {} ``` Five handler bodies, each tiny, all generic over the entity type. The last bound, `TryIntoRecord::Error: Debug`, is what the table's write path asks of an entity: a record that fails to serialize has to be reportable. Mount `crud(...)` under a prefix with `.nest(...)` and every route below it gets the full CRUD verb set for free. `/categories` becomes a one-liner: ```rust # use std::collections::HashMap; # use std::sync::OnceLock; # use axum::Router; # use serde::{Deserialize, Serialize, de::DeserializeOwned}; # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # type Params = HashMap; # fn crud(make_table: F) -> Router # where # F: Fn(SqliteDB, &Params) -> Table + Send + Sync + 'static, # E: Entity + Serialize + DeserializeOwned + Send + Sync + 'static, # >::Error: std::fmt::Debug, # { # let _ = make_table; # Router::new() # } # #[entity(SqliteType)] # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # pub struct Category { # pub name: String, # } # impl Category { # pub fn table(db: SqliteDB) -> &'static Table { # static CACHE: OnceLock> = OnceLock::new(); # CACHE.get_or_init(|| { # Table::new("category", db) # .with_id_column("id") # .with_column_of::("name") # }) # } # } # fn main() { # let _app: Router = Router::new() .nest("/categories", crud(|db, _| Category::table(db).clone())) # ; # } ``` That gives us: | Method | Path | Does | | ------ | ---------------- | --------------------- | | GET | /categories | list all | | POST | /categories | insert, return new id | | GET | /categories/{id} | fetch one | | PATCH | /categories/{id} | partial update | | DELETE | /categories/{id} | remove | The nested `/categories/{cat_id}/products` route uses the same helper. The closure reads `cat_id` out of the params map to narrow the set: ```rust # use std::collections::HashMap; # use std::sync::OnceLock; # use axum::Router; # use serde::{Deserialize, Serialize, de::DeserializeOwned}; # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # type Params = HashMap; # fn crud(make_table: F) -> Router # where # F: Fn(SqliteDB, &Params) -> Table + Send + Sync + 'static, # E: Entity + Serialize + DeserializeOwned + Send + Sync + 'static, # >::Error: std::fmt::Debug, # { # let _ = make_table; # Router::new() # } # #[entity(SqliteType)] # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # pub struct Product { # pub name: String, # } # #[entity(SqliteType)] # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # pub struct Category { # pub name: String, # } # impl Category { # pub fn table(db: SqliteDB) -> &'static Table { # static CACHE: OnceLock> = OnceLock::new(); # CACHE.get_or_init(|| { # Table::new("category", db) # .with_id_column("id") # .with_column_of::("name") # }) # } # } # pub trait CategoryTable { # fn id(&self) -> Column; # fn ref_products(&self) -> Table; # } # impl CategoryTable for Table { # fn id(&self) -> Column { # self.get_column("id").unwrap() # } # fn ref_products(&self) -> Table { # self.get_ref_as("products").unwrap() # } # } # fn main() { # let _app: Router = Router::new() .nest( "/categories/{cat_id}/products", crud(|db, p| { let cat_id: i64 = p["cat_id"].parse().unwrap(); let mut c = Category::table(db).clone(); c.add_condition(c.id().eq(cat_id)); c.ref_products() }), ) # ; # } ``` `c.id()` comes from extending the `CategoryTable` trait we already set up — add it next to `ref_products`: ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # pub struct Category { # pub name: String, # } # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # pub struct Product { # pub name: String, # } pub trait CategoryTable { fn id(&self) -> Column; fn ref_products(&self) -> Table; } impl CategoryTable for Table { fn id(&self) -> Column { self.get_column("id").unwrap() } fn ref_products(&self) -> Table { self.get_ref_as("products").unwrap() } } # fn main() {} ``` Nesting `crud` under `/categories/{cat_id}/products` gives a full CRUD surface for _products belonging to that category_: | Method | Path | Does | | ------ | ---------------------------------- | -------------- | | GET | /categories/{cat_id}/products | list | | POST | /categories/{cat_id}/products | create | | GET | /categories/{cat_id}/products/{id} | fetch one | | PATCH | /categories/{cat_id}/products/{id} | partial update | | DELETE | /categories/{cat_id}/products/{id} | remove | Try it: ```sh curl -X POST http://localhost:3001/categories \ -H 'content-type: application/json' -d '{"name":"Gluten-Free"}' # {"id":"4"} curl http://localhost:3001/categories/1/products/1 # {"name":"Cupcake","price":120,"category_id":"1","is_deleted":false} curl -X PATCH http://localhost:3001/categories/1 \ -H 'content-type: application/json' -d '{"name":"Sweet Things"}' # {"name":"Sweet Things"} curl -X DELETE http://localhost:3001/categories/4 # HTTP/1.1 204 No Content ``` The inline `list_categories` and `list_category_products` functions can be deleted — `crud` covers them both. ```admonish info title="Why a HashMap for path params?" Axum only lets a handler run the `Path` extractor **once** per request — after that, the URL params are considered consumed. We need the outer `{cat_id}` inside the closure *and* the inner `{id}` to identify which record to fetch, which rules out calling `Path` plus a second `Path`. Grabbing all params in one shot as `HashMap` sidesteps the limit — the closure picks out what it needs by name, and the item-level handlers pull `id` from the same map. The cost is a small amount of stringly-typed parsing (`p["cat_id"].parse::()`), which is a fair trade for letting one `crud` function cover every route shape in the server. ``` ```admonish info title="What's inside crud(), briefly" Each HTTP method gets its own closure in the `Router`. They all need to share the same `make_table` function, but Rust closures that capture by move can't be cloned by default, and each axum handler is an independent `Fn` — so we wrap `make_table` in an `Arc` once and let each handler clone the `Arc` (cheap — just a refcount bump). Inside the async block, the closure can then invoke `f(db(), ¶ms)` to build the narrowed table for that request. ``` --- ## Error handling Every handler in `crud` still ends in `.unwrap()`. The happy path has been fine to demo, but an API that panics on the slightest database hiccup isn't usable. The worst part isn't the 500 — it's what happens on the wire when axum's request task panics: the connection is dropped and the client sees an empty reply, with no status code and no body to explain. A proper REST API needs three things: - Missing resources return **404**, not 500. - Bad JSON bodies return **400** with a useful message. - Everything else returns **500** but with a structured JSON body, not silence. Axum already gives us the middle one for free — it rejects malformed `Json` bodies with 400. The other two come down to converting [`VantageError`](vantage_core::VantageError) into an HTTP response. Add an `ApiError` type to `main.rs`: ```rust # use axum::Json; # use axum::http::StatusCode; # use vantage_sql::prelude::*; use axum::response::{IntoResponse, Response}; struct ApiError { status: StatusCode, message: String, } impl IntoResponse for ApiError { fn into_response(self) -> Response { ( self.status, Json(serde_json::json!({ "error": self.message })), ) .into_response() } } impl From for ApiError { fn from(e: VantageError) -> Self { eprintln!("API error: {:?}", e); Self { status: StatusCode::INTERNAL_SERVER_ERROR, message: e.to_string(), } } } fn not_found(id: &str) -> ApiError { ApiError { status: StatusCode::NOT_FOUND, message: format!("not found: {}", id), } } type ApiResult = Result; # fn main() {} ``` Three pieces matter here: - **`IntoResponse`** makes `ApiError` returnable from a handler; axum calls `into_response()` to assemble status, headers, and body. - **`From`** lets us use `?` inside a handler — every `.await?` short-circuits to an `ApiError` mapped to a 500, which axum will render for us. - **`not_found(id)`** is how we build a 404 explicitly. Vantage's `get` returns `Option`, so we `.ok_or_else(|| not_found(&id))?` the missing case straight into a 404 — no error-message string-matching, no brittleness. With that in place, the `unwrap`s inside `crud` turn into `?`, and each handler returns `ApiResult`: ```rust # use std::collections::HashMap; # use std::sync::{Arc, OnceLock}; # use axum::extract::Path; # use axum::http::StatusCode; # use axum::response::{IntoResponse, Response}; # use axum::{Json, Router, routing::get}; # use serde::{Serialize, de::DeserializeOwned}; # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # static DB: OnceLock = OnceLock::new(); # fn db() -> SqliteDB { # DB.get().expect("database not initialised").clone() # } # type Params = HashMap; # struct ApiError { # status: StatusCode, # message: String, # } # impl IntoResponse for ApiError { # fn into_response(self) -> Response { # (self.status, Json(serde_json::json!({ "error": self.message }))).into_response() # } # } # impl From for ApiError { # fn from(e: VantageError) -> Self { # Self { status: StatusCode::INTERNAL_SERVER_ERROR, message: e.to_string() } # } # } # type ApiResult = Result; # fn crud(make_table: F) -> Router # where # F: Fn(SqliteDB, &Params) -> Table + Send + Sync + 'static, # E: Entity + Serialize + DeserializeOwned + Send + Sync + 'static, # >::Error: std::fmt::Debug, # { # let f = Arc::new(make_table); # Router::new().route( # "/", get({ let f = f.clone(); move |p: Option>| async move { let params = p.map(|Path(p)| p).unwrap_or_default(); let rows = f(db(), ¶ms).list().await?; ApiResult::Ok(Json::>(rows.into_values().collect())) } }) # ) # } # fn main() {} ``` For the `GET /{id}` handler, convert the `Option` into a 404 at the handler: ```rust # use std::collections::HashMap; # use std::sync::{Arc, OnceLock}; # use axum::extract::Path; # use axum::http::StatusCode; # use axum::response::{IntoResponse, Response}; # use axum::{Json, Router, routing::get}; # use serde::{Serialize, de::DeserializeOwned}; # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # static DB: OnceLock = OnceLock::new(); # fn db() -> SqliteDB { # DB.get().expect("database not initialised").clone() # } # type Params = HashMap; # struct ApiError { # status: StatusCode, # message: String, # } # impl IntoResponse for ApiError { # fn into_response(self) -> Response { # (self.status, Json(serde_json::json!({ "error": self.message }))).into_response() # } # } # impl From for ApiError { # fn from(e: VantageError) -> Self { # Self { status: StatusCode::INTERNAL_SERVER_ERROR, message: e.to_string() } # } # } # fn not_found(id: &str) -> ApiError { # ApiError { status: StatusCode::NOT_FOUND, message: format!("not found: {}", id) } # } # type ApiResult = Result; # fn crud(make_table: F) -> Router # where # F: Fn(SqliteDB, &Params) -> Table + Send + Sync + 'static, # E: Entity + Serialize + DeserializeOwned + Send + Sync + 'static, # >::Error: std::fmt::Debug, # { # let f = Arc::new(make_table); # Router::new().route( # "/{id}", get({ let f = f.clone(); move |Path(params): Path| async move { let id = params["id"].clone(); let entity = f(db(), ¶ms) .get(id.clone()) .await? .ok_or_else(|| not_found(&id))?; ApiResult::Ok(Json(entity)) } }) # ) # } # fn main() {} ``` Do the same shape for `post`, `patch`, `delete` — drop the `.unwrap()`, add `?`, wrap the happy path in `ApiResult::Ok(...)`. Handlers that don't read by id just need the `?`. Try a few error cases: ```sh curl -w "\nstatus=%{http_code}\n" http://localhost:3001/categories/999 # {"error":"not found: 999"} # status=404 curl -w "\nstatus=%{http_code}\n" -X PATCH http://localhost:3001/categories/999 \ -H 'content-type: application/json' -d '{"name":"Ghost"}' # {"error":"Row not found (table: \"category\", id: \"999\")"} # status=500 curl -w "\nstatus=%{http_code}\n" -X POST http://localhost:3001/categories \ -H 'content-type: application/json' -d '{not-json}' # Failed to parse the request body as JSON: ... # status=400 curl -w "\nstatus=%{http_code}\n" -X DELETE http://localhost:3001/categories/999 # status=204 ``` Missing ids produce 404s with a clean JSON body. The malformed body gets axum's built-in 400 for free. `DELETE` on a missing id still returns 204 — vantage's `delete` is idempotent, and "the resource is gone" is true whether or not it was ever there. `PATCH` on a missing id fails with an error whose `is_not_found()` is true; this handler maps every non-`get` error to 500, so add `Err(e) if e.is_not_found() => not_found(&id)` to answer 404 there too. ```admonish info title="Why not match on the error message?" An earlier draft of this tutorial matched `e.to_string().contains("no row found")` to decide between 404 and 500 — brittle, because it hard-codes a vantage-internal error string. Once `ReadableDataSet::get` switched to `Result>`, the handler can map missing rows explicitly with `.ok_or_else(|| not_found(&id))?`. Errors are errors, misses are `None` — no string-matching required. ``` ```admonish info title="Logging with {:?} on the server side" `eprintln!("API error: {:?}", e);` prints the full error structure — location, context, nested sources — which is exactly what a human debugging a 500 needs. The client meanwhile only sees `e.to_string()`, the short one-liner message. That's the whole point of `.context("…")` from chapter 1: context accumulates on the server, a single sentence reaches the client. Swap `eprintln!` for `tracing::error!` if you're wired up for structured logging. The mechanics are identical. ``` --- ## Pagination and search Three categories is fine for dev; three thousand would crush any client that naively calls `GET /categories` and tries to render everything. Real APIs page through long lists and let callers filter by a search term — two query-string features that belong inside `crud` so every entity gets them. Axum parses query strings for us via the `Query` extractor. Add a small struct for the parameters, and a second extractor on the list handler: ```rust # use serde::Deserialize; use axum::extract::Query; use vantage_table::pagination::Pagination; #[derive(Deserialize, Default)] struct ListQuery { page: Option, per_page: Option, q: Option, } # fn main() {} ``` Inside `crud`, only the `GET /` handler changes. It takes both extractors, mutates the narrowed table with whatever the caller asked for, and lists: ```rust # use std::collections::HashMap; # use std::sync::{Arc, OnceLock}; # use axum::extract::{Path, Query}; # use axum::http::StatusCode; # use axum::response::{IntoResponse, Response}; # use axum::{Json, Router, routing::get}; # use serde::{Deserialize, Serialize, de::DeserializeOwned}; # use vantage_sql::prelude::*; # use vantage_table::pagination::Pagination; # use vantage_types::prelude::*; # static DB: OnceLock = OnceLock::new(); # fn db() -> SqliteDB { # DB.get().expect("database not initialised").clone() # } # type Params = HashMap; # struct ApiError { # status: StatusCode, # message: String, # } # impl IntoResponse for ApiError { # fn into_response(self) -> Response { # (self.status, Json(serde_json::json!({ "error": self.message }))).into_response() # } # } # impl From for ApiError { # fn from(e: VantageError) -> Self { # Self { status: StatusCode::INTERNAL_SERVER_ERROR, message: e.to_string() } # } # } # type ApiResult = Result; # #[derive(Deserialize, Default)] # struct ListQuery { # page: Option, # per_page: Option, # q: Option, # } # fn crud(make_table: F) -> Router # where # F: Fn(SqliteDB, &Params) -> Table + Send + Sync + 'static, # E: Entity + Serialize + DeserializeOwned + Send + Sync + 'static, # >::Error: std::fmt::Debug, # { # let f = Arc::new(make_table); # Router::new().route( # "/", get({ let f = f.clone(); move |p: Option>, Query(q): Query| async move { let params = p.map(|Path(p)| p).unwrap_or_default(); let mut t = f(db(), ¶ms); if q.page.is_some() || q.per_page.is_some() { t.set_pagination(Some(Pagination::new( q.page.unwrap_or(1), q.per_page.unwrap_or(50), ))); } if let Some(term) = q.q.as_deref() { t = t.with_search(term); } let rows = t.list().await?; ApiResult::Ok(Json::>(rows.into_values().collect())) } }) # ) # } # fn main() {} ``` Two things happen: - **[`set_pagination`](vantage_table::table::Table::set_pagination)`(Some(`[`Pagination::new`](vantage_table::pagination::Pagination::new)`(page, per_page)))`** takes a page number and a page size. Vantage applies these as `LIMIT … OFFSET …` on the SELECT. Missing params fall back to the defaults (page 1, 50 per page) — and if neither is supplied we don't touch pagination at all, so the unfiltered list still hits the whole set. - **[`with_search`](vantage_table::table::Table::with_search)`(term)`** is the same call we used in chapter 2 to add a LIKE filter across the table's search columns (every column, unless some are flagged searchable). It consumes the table and hands back the narrowed one, hence `t = t.with_search(term)`. Both end up as extra clauses on the query that vantage already compiles for us. Try it: ```sh curl "http://localhost:3001/categories?page=1&per_page=2" # [{"name":"Sweet Treats"},{"name":"Pastries"}] curl "http://localhost:3001/categories?page=2&per_page=2" # [{"name":"Breads"}] curl "http://localhost:3001/categories?q=Pastries" # [{"name":"Pastries"}] curl "http://localhost:3001/categories?q=e" # [{"name":"Sweet Treats"},{"name":"Pastries"},{"name":"Breads"}] ``` The nested route gets these for free — `crud` is the same function. `per_page` works on `/categories/{cat_id}/products` out of the box: ```sh curl "http://localhost:3001/categories/1/products?per_page=2" # [ # {"name":"Cupcake","price":120,"category_id":"1","is_deleted":false}, # {"name":"Doughnut","price":135,"category_id":"1","is_deleted":false} # ] ``` Because the closure for the nested mount narrows the table with `with_condition` *before* `crud` applies its own pagination/search, the filters compose cleanly: the category scope stays in effect, and pagination just counts rows within it. ```admonish info title="What about ordering?" A full `?order_by=price&dir=desc` pairing is the obvious next thing — and the Table API supports it via `with_order(OrderBy::ascending(…))` — but the `OrderBy` type is generic over the backend's condition type (`T::Condition`), not the easier-to-hand-you `Expression` that `get_column_expr` returns. Wiring it up at the generic `crud` level takes a small extra layer of `From` conversions that would balloon this section. For a single-entity handler, order the cloned definition directly: `Category::table(db()).clone().with_order(OrderBy::ascending(ident("name").into()))`, with `OrderBy` from `vantage_table::sorting`. Adding ordering to `crud` is a fine exercise once the rest of the server is in place — and a natural thing to push back into the framework so every backend picks it up uniformly. ``` ```admonish info title="Validating pagination params" Our `ListQuery` silently accepts `page=0`, `per_page=-5`, or `per_page=1000000`. `Pagination::new` clamps page and items-per-page to at least 1, so the first two can't crash us — but an API that hands out 1M rows because someone asked for it is a DoS target. For production, extend `ListQuery` with a `fn validate(&self) -> Result<(), ApiError>` that caps `per_page` at something like 200 and returns 400 otherwise. The plumbing is already there — `ApiError` already knows how to render a 400. ``` --- ## Migrating to MongoDB Chapter 2 closed on a claim: the model layer isolates business code from storage, so swapping databases is a change to the model, not to the routes, handlers, or business logic. Now we cash the check. `/categories`, `/categories/{id}`, and `/categories/{cat_id}/products` keep their exact URL shape, response bodies, and behaviour — but the data lives in [`MongoDB`](vantage_mongodb::MongoDB) instead of [`SqliteDB`](vantage_sql::sqlite::SqliteDB). Start Mongo — Docker is the easiest way: ```sh docker run -d --name mongo-learn -p 27017:27017 mongo:7 ``` ### Cargo.toml Drop `vantage-sql`, add `vantage-mongodb`: ```toml [dependencies] axum = "0.8.9" serde = { version = "1", features = ["derive"] } serde_json = "1" tokio = { version = "1", features = ["full"] } vantage-core = "1.1" vantage-dataset = "1.1" vantage-expressions = "1.2" vantage-mongodb = "2.0" vantage-table = "2.0" vantage-types = { version = "1.0", features = ["serde"] } ``` ### Entities Swap the `#[entity]` type tag from `SqliteType` to `MongoType` and the id column name from `id` to the MongoDB-idiomatic `_id`. Imports collapse to a single prelude use-line. `src/category.rs`: ```rust use std::sync::OnceLock; use vantage_mongodb::prelude::*; use crate::product::Product; #[entity(MongoType)] #[derive(Debug, Clone, Default, Serialize, Deserialize)] pub struct Category { pub name: String, } impl Category { pub fn table(db: MongoDB) -> &'static Table { static CACHE: OnceLock> = OnceLock::new(); CACHE.get_or_init(|| { Table::new("category", db) .with_id_column("_id") .with_column_of::("name") .with_many("products", "category_id", |db| Product::table(db).clone()) }) } } # mod product { # use std::sync::OnceLock; # use vantage_mongodb::prelude::*; # use crate::Category; # #[entity(MongoType)] # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # pub struct Product { # pub name: String, # pub price: i64, # #[serde(skip_serializing_if = "Option::is_none", default)] # pub category_id: Option, # #[serde(default)] # pub is_deleted: bool, # } # impl Product { # pub fn table(db: MongoDB) -> &'static Table { # static CACHE: OnceLock> = OnceLock::new(); # CACHE.get_or_init(|| { # let is_deleted = Column::::new("is_deleted"); # Table::new("product", db) # .with_id_column("_id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("category_id") # .with_column_of::("is_deleted") # .with_condition(is_deleted.eq(false)) # .with_one("category", "category_id", |db| Category::table(db).clone()) # }) # } # } # } # fn main() {} ``` `src/product.rs` gets the symmetric changes: ```rust use std::sync::OnceLock; use vantage_mongodb::prelude::*; use crate::category::Category; #[entity(MongoType)] #[derive(Debug, Clone, Default, Serialize, Deserialize)] pub struct Product { pub name: String, pub price: i64, #[serde(skip_serializing_if = "Option::is_none", default)] pub category_id: Option, #[serde(default)] pub is_deleted: bool, } impl Product { pub fn table(db: MongoDB) -> &'static Table { static CACHE: OnceLock> = OnceLock::new(); CACHE.get_or_init(|| { let is_deleted = Column::::new("is_deleted"); Table::new("product", db) .with_id_column("_id") .with_column_of::("name") .with_column_of::("price") .with_column_of::("category_id") .with_column_of::("is_deleted") .with_condition(is_deleted.eq(false)) .with_one("category", "category_id", |db| Category::table(db).clone()) }) } } # mod category { # use std::sync::OnceLock; # use vantage_mongodb::prelude::*; # use crate::Product; # #[entity(MongoType)] # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # pub struct Category { # pub name: String, # } # impl Category { # pub fn table(db: MongoDB) -> &'static Table { # static CACHE: OnceLock> = OnceLock::new(); # CACHE.get_or_init(|| { # Table::new("category", db) # .with_id_column("_id") # .with_column_of::("name") # .with_many("products", "category_id", |db| Product::table(db).clone()) # }) # } # } # } # fn main() {} ``` Nothing about the struct shape changed — `category_id` and `is_deleted` have been real columns since § 2. The `with_*` calls line up one-for-one with the SQLite version; only the id column name and the entity's type tag shift. The `CategoryTable` trait from § 3 also updates — MongoDB's `_id` is a string, not an integer: ```rust # use vantage_mongodb::prelude::*; # #[entity(MongoType)] # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # pub struct Category { # pub name: String, # } # #[entity(MongoType)] # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # pub struct Product { # pub name: String, # } pub trait CategoryTable { fn id(&self) -> Column; fn ref_products(&self) -> Table; } impl CategoryTable for Table { fn id(&self) -> Column { self.get_column("_id").unwrap() } fn ref_products(&self) -> Table { self.get_ref_as("products").unwrap() } } # fn main() {} ``` The nested route's closure gets a little simpler as a result — no parse step, just narrow by the URL's `cat_id` string directly: ```rust # use std::collections::HashMap; # use std::sync::OnceLock; # use axum::Router; # use vantage_mongodb::prelude::*; # type Params = HashMap; # fn crud(make_table: F) -> Router # where # F: Fn(MongoDB, &Params) -> Table + Send + Sync + 'static, # E: Entity + Serialize + DeserializeOwned + Send + Sync + 'static, # >::Error: std::fmt::Debug, # { # let _ = make_table; # Router::new() # } # #[entity(MongoType)] # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # pub struct Product { # pub name: String, # } # #[entity(MongoType)] # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # pub struct Category { # pub name: String, # } # impl Category { # pub fn table(db: MongoDB) -> &'static Table { # static CACHE: OnceLock> = OnceLock::new(); # CACHE.get_or_init(|| { # Table::new("category", db) # .with_id_column("_id") # .with_column_of::("name") # }) # } # } # pub trait CategoryTable { # fn id(&self) -> Column; # fn ref_products(&self) -> Table; # } # impl CategoryTable for Table { # fn id(&self) -> Column { # self.get_column("_id").unwrap() # } # fn ref_products(&self) -> Table { # self.get_ref_as("products").unwrap() # } # } # fn main() { # let _router: Router = crud(|db, p| { let mut c = Category::table(db).clone(); c.add_condition(c.id().eq(p["cat_id"].as_str())); c.ref_products() }) # ; # } ``` `.eq(&str)` works here because `vantage-mongodb`'s `From<&str>` for [`AnyMongoType`](vantage_mongodb::AnyMongoType) auto-promotes 24-character hex strings to `ObjectId` and leaves everything else as `String`. The comparison fires with the right BSON type whichever `_id` convention the collection uses. ### main.rs The `crud` function's generic bounds shift from `Entity` to `Entity` and its `Fn(SqliteDB, ...)` becomes `Fn(MongoDB, ...)`. Body unchanged. ```rust # use std::collections::HashMap; # use axum::Router; # use vantage_mongodb::prelude::*; # type Params = HashMap; fn crud(make_table: F) -> Router where F: Fn(MongoDB, &Params) -> Table + Send + Sync + 'static, E: Entity + Serialize + DeserializeOwned + Send + Sync + 'static, >::Error: std::fmt::Debug, # { # let _ = make_table; # Router::new() # } # fn main() {} ``` Connection and id handling are the only handler-level changes: ```rust,no_run # use std::collections::HashMap; # use vantage_mongodb::prelude::*; # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let params = HashMap::from([("id".to_string(), "69e2b9101a552c206f5f8468".to_string())]); // db() returns MongoDB; connect with URL + database name. let conn = MongoDB::connect("mongodb://localhost:27017", "learn3") .await .context("Failed to connect to MongoDB")?; // item-level ids are MongoId. String → MongoId dispatches to ObjectId when the // string is a 24-char hex, otherwise stays a plain String, via a `From` // smart-parse in vantage-mongodb. let id: MongoId = params["id"].clone().into(); # Ok(()) # } ``` The 404 path needs no change at all. Mongo's `get` returns `Result>` just like SQLite's, so the same `.ok_or_else(|| not_found(&id))?` maps a missing document to 404 — the promise of the *Why not match on the error message?* box holds across backends. ### Running it ```sh cargo run ``` The collection is empty, so the first request returns an empty array: ```sh curl http://localhost:3001/categories # [] ``` POST a few categories — responses carry the auto-generated MongoDB ObjectId as the new id: ```sh SWEETS=$(curl -s -X POST http://localhost:3001/categories \ -H 'content-type: application/json' -d '{"name":"Sweet Treats"}' \ | jq -r .id) echo "$SWEETS" # 69e2b9101a552c206f5f8468 ``` Create a product in that category by POSTing to the nested route, including the parent id as `category_id` in the body: ```sh curl -X POST "http://localhost:3001/categories/$SWEETS/products" \ -H 'content-type: application/json' \ -d "{\"name\":\"Cupcake\",\"price\":120,\"category_id\":\"$SWEETS\"}" # {"id":"69e2b9101a552c206f5f846a"} curl "http://localhost:3001/categories/$SWEETS/products" # [{"name":"Cupcake","price":120,"category_id":"69e2b9101a552c206f5f8468","is_deleted":false}] ``` Same URL shape as the SQLite version. Same JSON. Same error codes. Handlers, routing, pagination, filtering — nothing in the request path learned that storage moved from a local file to a document database. ```admonish info title="Cross-type $in in relationship traversal" `CategoryTable::ref_products()` still works. MongoDB's `_id` defaults to `ObjectId`, but application fields like `product.category_id` arrive as plain JSON strings and get stored as BSON `String`. A naive `$in: [ObjectId(...)]` wouldn't match. `vantage-mongodb` sidesteps that by pushing both representations into the `$in` inside `related_in_condition` — an `ObjectId` value also emits its 24-char hex string, and a hex-shaped `String` value also emits the parsed `ObjectId`. Traversal works regardless of which form the target stores. ``` ```admonish info title="String _ids are also an option" Nothing in MongoDB requires `_id` to be an `ObjectId` — it can be any BSON value, including a plain string. If the app supplies `_id` explicitly on insert (or the framework generates a UUID and writes it into `_id`), both `category._id` and `product.category_id` are strings and everything lines up without the $in dual-push. Useful when you want stable, human-legible ids or ids that came from an upstream system. ``` --- ## Scaling up: CRUD as a one-liner The `crud` function, `ApiError`, `Params`, and `ListQuery` aren't really tied to this app — they're generic over any `Table`. Move them into their own module and `main.rs` collapses to exactly what it's about: connecting the database and registering routes. Three files do the work: **`src/db.rs`** (17 lines) — the `static DB: OnceLock`, an `init(url, db)` helper that connects and stores the handle, and a `pub fn db() -> MongoDB` accessor that hands out cheap `MongoDB` clones: ```rust # mod db { use std::sync::OnceLock; use vantage_mongodb::prelude::*; static DB: OnceLock = OnceLock::new(); pub async fn init(url: &str, database: &str) -> VantageResult<()> { let conn = MongoDB::connect(url, database) .await .context("Failed to connect to MongoDB")?; DB.set(conn).ok(); Ok(()) } pub fn db() -> MongoDB { DB.get().expect("database not initialised").clone() } # } # fn main() {} ``` **`src/vantage_axum.rs`** (one file, ~130 lines) — everything HTTP: `ApiError`, `ListQuery`, and the `crud` helper. Accepts any entity `E` that implements `Entity` plus the usual serde bounds. It's the pieces from the sections above, collected: ```rust # mod db { # use std::sync::OnceLock; # use vantage_mongodb::prelude::*; # static DB: OnceLock = OnceLock::new(); # pub async fn init(url: &str, database: &str) -> VantageResult<()> { # let conn = MongoDB::connect(url, database) # .await # .context("Failed to connect to MongoDB")?; # DB.set(conn).ok(); # Ok(()) # } # pub fn db() -> MongoDB { # DB.get().expect("database not initialised").clone() # } # } # mod vantage_axum { use std::collections::HashMap; use std::sync::Arc; use axum::{ Json, Router, extract::{Path, Query}, http::StatusCode, response::{IntoResponse, Response}, routing::get, }; use vantage_mongodb::prelude::*; use crate::db::db; pub struct ApiError { pub status: StatusCode, pub message: String, } impl IntoResponse for ApiError { fn into_response(self) -> Response { ( self.status, Json(serde_json::json!({ "error": self.message })), ) .into_response() } } impl From for ApiError { fn from(e: VantageError) -> Self { eprintln!("API error: {:?}", e); Self { status: StatusCode::INTERNAL_SERVER_ERROR, message: e.to_string(), } } } fn not_found(id: &str) -> ApiError { ApiError { status: StatusCode::NOT_FOUND, message: format!("not found: {}", id), } } pub type Params = HashMap; pub type ApiResult = Result; #[derive(Deserialize, Default)] pub struct ListQuery { pub page: Option, pub per_page: Option, pub q: Option, } pub fn crud(make_table: F) -> Router where F: Fn(MongoDB, &Params) -> Table + Send + Sync + 'static, E: Entity + Serialize + DeserializeOwned + Send + Sync + 'static, >::Error: std::fmt::Debug, { let f = Arc::new(make_table); Router::new() .route( "/", get({ let f = f.clone(); move |p: Option>, Query(q): Query| async move { let params = p.map(|Path(p)| p).unwrap_or_default(); let mut t = f(db(), ¶ms); if q.page.is_some() || q.per_page.is_some() { t.set_pagination(Some(Pagination::new( q.page.unwrap_or(1), q.per_page.unwrap_or(50), ))); } if let Some(term) = q.q.as_deref() { t = t.with_search(term); } let rows = t.list().await?; ApiResult::Ok(Json::>(rows.into_values().collect())) } }) .post({ let f = f.clone(); move |p: Option>, Json(entity): Json| async move { let params = p.map(|Path(p)| p).unwrap_or_default(); let id = f(db(), ¶ms).insert_return_id(&entity).await?; ApiResult::Ok(Json(serde_json::json!({ "id": id }))) } }), ) .route( "/{id}", get({ let f = f.clone(); move |Path(params): Path| async move { let id = params["id"].clone(); let entity = f(db(), ¶ms) .get(id.clone()) .await? .ok_or_else(|| not_found(&id))?; ApiResult::Ok(Json(entity)) } }) .patch({ let f = f.clone(); move |Path(params): Path, Json(partial): Json| async move { let id: MongoId = params["id"].clone().into(); let updated = f(db(), ¶ms).patch(id.clone(), &partial).await?; ApiResult::Ok(Json(updated)) } }) .delete({ let f = f; move |Path(params): Path| async move { let id: MongoId = params["id"].clone().into(); f(db(), ¶ms).delete(id.clone()).await?; ApiResult::Ok(StatusCode::NO_CONTENT) } }), ) } # } # fn main() {} ``` **`src/main.rs`** — now down to ~30 lines. It declares the four modules (`mod category;`, `mod db;`, `mod product;`, `mod vantage_axum;`), then: ```rust,no_run # mod db { # use std::sync::OnceLock; # use vantage_mongodb::prelude::*; # static DB: OnceLock = OnceLock::new(); # pub async fn init(url: &str, database: &str) -> VantageResult<()> { # let conn = MongoDB::connect(url, database) # .await # .context("Failed to connect to MongoDB")?; # DB.set(conn).ok(); # Ok(()) # } # pub fn db() -> MongoDB { # DB.get().expect("database not initialised").clone() # } # } # mod product { # use std::sync::OnceLock; # use vantage_mongodb::prelude::*; # use crate::category::Category; # #[entity(MongoType)] # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # pub struct Product { # pub name: String, # pub price: i64, # #[serde(skip_serializing_if = "Option::is_none", default)] # pub category_id: Option, # #[serde(default)] # pub is_deleted: bool, # } # impl Product { # pub fn table(db: MongoDB) -> &'static Table { # static CACHE: OnceLock> = OnceLock::new(); # CACHE.get_or_init(|| { # let is_deleted = Column::::new("is_deleted"); # Table::new("product", db) # .with_id_column("_id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("category_id") # .with_column_of::("is_deleted") # .with_condition(is_deleted.eq(false)) # .with_one("category", "category_id", |db| Category::table(db).clone()) # }) # } # } # } # mod category { # use std::sync::OnceLock; # use vantage_mongodb::prelude::*; # use crate::product::Product; # #[entity(MongoType)] # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # pub struct Category { # pub name: String, # } # impl Category { # pub fn table(db: MongoDB) -> &'static Table { # static CACHE: OnceLock> = OnceLock::new(); # CACHE.get_or_init(|| { # Table::new("category", db) # .with_id_column("_id") # .with_column_of::("name") # .with_many("products", "category_id", |db| Product::table(db).clone()) # }) # } # } # pub trait CategoryTable { # fn id(&self) -> Column; # fn ref_products(&self) -> Table; # } # impl CategoryTable for Table { # fn id(&self) -> Column { # self.get_column("_id").unwrap() # } # fn ref_products(&self) -> Table { # self.get_ref_as("products").unwrap() # } # } # } # mod vantage_axum { # use std::collections::HashMap; # use std::sync::Arc; # use axum::{ # Json, Router, # extract::{Path, Query}, # http::StatusCode, # response::{IntoResponse, Response}, # routing::get, # }; # use vantage_mongodb::prelude::*; # use crate::db::db; # pub struct ApiError { # pub status: StatusCode, # pub message: String, # } # impl IntoResponse for ApiError { # fn into_response(self) -> Response { # (self.status, Json(serde_json::json!({ "error": self.message }))).into_response() # } # } # impl From for ApiError { # fn from(e: VantageError) -> Self { # eprintln!("API error: {:?}", e); # Self { status: StatusCode::INTERNAL_SERVER_ERROR, message: e.to_string() } # } # } # fn not_found(id: &str) -> ApiError { # ApiError { status: StatusCode::NOT_FOUND, message: format!("not found: {}", id) } # } # pub type Params = HashMap; # pub type ApiResult = Result; # #[derive(Deserialize, Default)] # pub struct ListQuery { # pub page: Option, # pub per_page: Option, # pub q: Option, # } # pub fn crud(make_table: F) -> Router # where # F: Fn(MongoDB, &Params) -> Table + Send + Sync + 'static, # E: Entity + Serialize + DeserializeOwned + Send + Sync + 'static, # >::Error: std::fmt::Debug, # { # let f = Arc::new(make_table); # Router::new() # .route( # "/", # get({ # let f = f.clone(); # move |p: Option>, Query(q): Query| async move { # let params = p.map(|Path(p)| p).unwrap_or_default(); # let mut t = f(db(), ¶ms); # if q.page.is_some() || q.per_page.is_some() { # t.set_pagination(Some(Pagination::new( # q.page.unwrap_or(1), # q.per_page.unwrap_or(50), # ))); # } # if let Some(term) = q.q.as_deref() { # t = t.with_search(term); # } # let rows = t.list().await?; # ApiResult::Ok(Json::>(rows.into_values().collect())) # } # }) # .post({ # let f = f.clone(); # move |p: Option>, Json(entity): Json| async move { # let params = p.map(|Path(p)| p).unwrap_or_default(); # let id = f(db(), ¶ms).insert_return_id(&entity).await?; # ApiResult::Ok(Json(serde_json::json!({ "id": id }))) # } # }), # ) # .route( # "/{id}", # get({ # let f = f.clone(); # move |Path(params): Path| async move { # let id = params["id"].clone(); # let entity = f(db(), ¶ms) # .get(id.clone()) # .await? # .ok_or_else(|| not_found(&id))?; # ApiResult::Ok(Json(entity)) # } # }) # .patch({ # let f = f.clone(); # move |Path(params): Path, Json(partial): Json| async move { # let id: MongoId = params["id"].clone().into(); # let updated = f(db(), ¶ms).patch(id.clone(), &partial).await?; # ApiResult::Ok(Json(updated)) # } # }) # .delete({ # let f = f; # move |Path(params): Path| async move { # let id: MongoId = params["id"].clone().into(); # f(db(), ¶ms).delete(id.clone()).await?; # ApiResult::Ok(StatusCode::NO_CONTENT) # } # }), # ) # } # } use axum::Router; use category::{Category, CategoryTable}; use vantage_axum::crud; use vantage_mongodb::prelude::*; #[tokio::main] async fn main() -> VantageResult<()> { db::init("mongodb://localhost:27017", "learn3").await?; let app = Router::new() .nest("/categories", crud(|db, _| Category::table(db).clone())) .nest( "/categories/{cat_id}/products", crud(|db, p| { let mut c = Category::table(db).clone(); c.add_condition(c.id().eq(p["cat_id"].as_str())); c.ref_products() }), ); let listener = tokio::net::TcpListener::bind("0.0.0.0:3001").await.unwrap(); axum::serve(listener, app).await.unwrap(); Ok(()) } ``` Adding a new entity to this server is now *two* things: 1. Write a `Table::new(...)` constructor for it — declarative, one function, same shape we learned in chapter 2. 2. Mount it. ```rust # use std::collections::HashMap; # use std::sync::OnceLock; # use axum::Router; # use vantage_mongodb::prelude::*; # type Params = HashMap; # fn crud(make_table: F) -> Router # where # F: Fn(MongoDB, &Params) -> Table + Send + Sync + 'static, # E: Entity + Serialize + DeserializeOwned + Send + Sync + 'static, # >::Error: std::fmt::Debug, # { # let _ = make_table; # Router::new() # } # #[entity(MongoType)] # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # pub struct Widget { # pub name: String, # } # impl Widget { # pub fn table(db: MongoDB) -> &'static Table { # static CACHE: OnceLock> = OnceLock::new(); # CACHE.get_or_init(|| { # Table::new("widget", db) # .with_id_column("_id") # .with_column_of::("name") # }) # } # } # fn main() { # let _app: Router = Router::new() .nest("/widgets", crud(|db, _| Widget::table(db).clone())) # ; # } ``` The full surface: `GET` list, `POST` create, `GET /{id}`, `PATCH /{id}`, `DELETE /{id}`, pagination via `?page=&per_page=`, full-text search via `?q=`, 404s for missing ids, 400s for malformed bodies, structured JSON errors. No new handler code. No per-entity error mapping. No per-entity query-param struct. One line per entity. Every route plays by the same rules because every route is served by the same `crud` — the single description of "what this route does" lives in the entity's `Table` definition, and the HTTP boundary just routes to it. That is the "one description, many operations" principle from chapter 2's `Table` carried all the way to the wire, unbroken. The `vantage_axum` module is generic enough to lift directly into a larger codebase — it has no knowledge of `Category`, `Product`, or your particular routes. Drop it into your own binary, give it a `db()` accessor, write entity files, mount routes. One thing it deliberately doesn't do is authentication — every endpoint here is anonymous. Auth belongs to the HTTP layer: wrap the `Router` in your usual tower middleware; Vantage stays out of it. --- Source: https://vantage-ui.com/book/intro/step4-vista.html # Vista — the Universal Data Handle Chapters 1–3 built a typed data layer: `Table`, conditions, relationships, CRUD. That layer is great when you know the entity at compile time. But step 3's generic `crud()` helper already showed the limitation — it could only exist because we erased the entity type with `Serialize + DeserializeOwned` and went through JSON. A CLI that lists "any table from any backend", a web admin that draws forms from a YAML schema, a UI data grid that works with whatever you point it at — none of these know your `Product` struct. They need a handle that carries its own schema, speaks a single value type, and works regardless of which database sits underneath. That handle is [`Vista`](vantage_vista::Vista). ```admonish question title="Why 'Vista'?" Vantage is named for the view. Stand at a vantage point — the peak above your infrastructure — and the landscape arranges itself below: every database, API, and file your organisation keeps, observable from one place. The framework's names stay true to that scene, borrowing from landscape and photography rather than reusing overloaded database words. A **table** isn't one of ours — it's what you already have. Vantage describes your tables; it doesn't reinvent them. A **vista** is the first thing the framework adds: in the landscape sense, a wide view opening up from where you stand. And that is precisely what a `Vista` is here — one data source seen in full, its schema, capabilities, and records composed into a single view, no matter what produces it. Further along the trail wait a **Lens**, a **Dio**rama, and **Scenery** — each another way of looking, never another thing to store. ``` ```admonish example title="Goals for this chapter" By the end of this page you'll be able to: 1. Wrap a typed `Table` into a Vista 2. Read schema metadata (columns, references, id column) from a Vista 3. Narrow a Vista with conditions, search, and ordering 4. Read a window of rows with `fetch_window`, or walk a cursor with `fetch_cursor` 5. Traverse relationships and cross-backend references 6. Understand capabilities — the explicit contract between Vista and its driver ``` --- ## What Vista actually is A `Vista` wraps a typed `Table` and erases both the backend and the entity. All data flows through `Record` — an ordered map of string keys to CBOR values. All schema lives on the Vista itself: columns (with types and flags), references, id column, and a set of capabilities. Think of the progression: ```text Table — typed entity, typed backend, compile-time safe Vista — fully erased: schema-bearing, CborValue, no generics ``` Vista trades away compile-time knowledge for runtime flexibility: everything is a string key and a CBOR value, and the schema travels with the handle rather than with your types. Wrapping a typed `Table` — this chapter's path — is also not the only way to get one: every driver ships a `VistaFactory` that can materialize a Vista from a declarative YAML spec, with Rhai scripts for the expressions YAML can't state. That path has its own guide — [Config-Driven Vistas](../config-driven-vistas.md) — and this chapter stays on the typed one. Scripts read and write Vistas through their own short vocabulary (`table("product").where(…)`), taught in [Scripting with Rhai](../rhai.md); the methods below are the Rust side of the same operations. ```admonish info title="CBOR, not JSON" Vista uses [`ciborium::Value`](https://docs.rs/ciborium) as its carrier type — a CBOR value. CBOR preserves type fidelity that JSON loses (integer vs float, binary blobs, precise decimals). You'll see `CborValue` in every Vista method signature. If you need JSON (for an HTTP response, for example), convert at the boundary — `Record` → `Record` is a one-liner. But inside the Vista layer, CBOR is the standard. ``` --- ## Wrapping a typed Table Each backend ships a factory that turns a typed table into a Vista. For SQLite, the database hands you one: ```rust,no_run # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # use vantage_vista::prelude::*; # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # struct Product { name: String, price: i64 } # impl Product { # fn table(db: SqliteDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("category_id") # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:products.db").await.context("connect")?; let table = Product::table(db.clone()); let vista: Vista = db.vista_factory().from_table(table)?; # Ok(()) # } ``` That's it. The factory harvests columns, id field, title fields, and references from the typed table definition you already built in chapter 2. No extra mapping code. For MongoDB the factory is a `MongoVistaFactory`, for AWS an `AwsVistaFactory` — same shape, different import. The resulting `Vista` is the same type regardless of which factory produced it. This chapter's snippets read `products.db`, the file chapters 1–3 built, so they compile but are not run by the book's tests. --- ## Reading schema Once you have a Vista, everything is runtime introspection: ```rust,no_run # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # use vantage_vista::prelude::*; # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # struct Product { name: String, price: i64 } # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # struct Category { name: String } # impl Product { # fn table(db: SqliteDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("category_id") # .with_one("category", "category_id", Category::table) # } # } # impl Category { # fn table(db: SqliteDB) -> Table { # Table::new("category", db) # .with_id_column("id") # .with_column_of::("name") # .with_many("products", "category_id", Product::table) # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:products.db").await.context("connect")?; # let vista = db.vista_factory().from_table(Product::table(db.clone()))?; // Columns — name, original type, flags for (name, column) in vista.columns() { let ty = column.original_type.as_deref().unwrap_or("?"); println!("{name}: {ty}"); } // price: i64 // … // ID column if let Some(id_column) = vista.id_column() { println!("id column: {id_column}"); } // Title columns — the human-readable ones for title in vista.title_columns() { println!("title: {title}"); } // References — relationships declared on the typed table for (name, reference) in vista.references() { println!("ref: {name} ({})", reference.kind.as_str()); } // ref: category (has_one) # Ok(()) # } ``` [`Column`](vantage_vista::Column) carries an `original_type` (the Rust type name the typed column was declared with, or `None` when nothing declares one) and a set of [`ColumnFlag`](vantage_vista::ColumnFlag)s. The flags the data layer acts on: | Flag | Meaning | | -------------- | -------------------------------------------------------- | | `Id` | This column is the primary key | | `Title` | Human-readable label column | | `Searchable` | Included in quicksearch | | `Orderable` | Can be sorted server-side | | `Hidden` | Don't show in default views | | `Mandatory` | Required on insert | | `RequestParam` | Sent to the backend as a request parameter (a URL slot) | | `Calculated` | Computed, not stored; read-only, and writes drop it | | `Auto` | The backend makes the value (auto-increment, record ids) | Any other name is kept as `ColumnFlag::Other` for the layer that reads it — a UI's presentation flag, for instance. Vista itself never interprets those. --- ## Narrowing Vista doesn't carry its own condition type. Instead, it delegates to a [`TableShell`](vantage_vista::TableShell) — the per-driver executor every Vista wraps, and the piece each factory actually builds — which translates the value into whatever the backend speaks: ```rust,no_run # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # use vantage_vista::prelude::*; # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # struct Product { name: String, price: i64 } # impl Product { # fn table(db: SqliteDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("category_id") # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:products.db").await.context("connect")?; # let vista = db.vista_factory().from_table(Product::table(db.clone()))?; let cakes = vista.clone().with_condition("category_id", FilterOp::Eq, 1)?; let rows = cakes.list_values().await?; // Only products with category_id == 1; `vista` still holds every product # Ok(()) # } ``` The value converts into a `CborValue`. The driver's shell translates the condition into a native one — an `Expression` for SQL, a `bson::Document` for MongoDB, an `AwsCondition` for AWS — and pushes it onto the wrapped table. [`FilterOp`](vantage_vista::FilterOp) has the other comparisons too (`Gt`, `In`, …); every shell takes `Eq`, and the rest need the `FilterOps` capability. ### Why `with_*` consumes A `Vista` is a value, the same as the `Table` it wraps. It owns its query state, so `vista.clone().with_condition(..)` hands back a new, smaller set and leaves `vista` exactly as it was. That's the property generic code leans on: a function that receives a Vista can narrow it for its own purposes without changing what the caller sees, and a cached "all products" Vista can be the starting point for every request that needs a subset. The `with_*` methods take `self` and return `Result`, so they chain: ```rust,no_run # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # use vantage_vista::prelude::*; # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # struct Product { name: String, price: i64 } # impl Product { # fn table(db: SqliteDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("category_id") # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:products.db").await.context("connect")?; # let vista = db.vista_factory().from_table(Product::table(db.clone()))?; let v = vista .clone() .with_condition("category_id", FilterOp::Eq, 1)? .with_order("price", SortDirection::Descending)?; # Ok(()) # } ``` When you hold a Vista of your own and want to change it in place — a UI handle whose search box the user keeps editing — the `&mut self` methods do the same work: `add_condition`, `set_search`, `set_order`. They exist for the owner of the value, not for narrowing someone else's. | Operation | Semantics | | ------------------------------------------------ | ------------------------------------------------------------ | | `with_condition`, `with_id` / `add_condition` | **Accrete** — each call narrows further; none can be removed | | `with_search`, `with_order` / `set_search`, `set_order` | **Replace** — a later call swaps the earlier one out; `None` clears it | | Widening | **Clone first** — narrow the clone, keep the original | A narrowed Vista is a *different, smaller set* — not a view you toggle filters on. Each clause the shell accepted is recorded on [`query()`](vantage_vista::Vista::query), so the layers above (the cache in the next chapter, for one) can tell two narrowed Vistas apart. ### Narrowing by id A common pattern — "I have an id and want the row": ```rust,no_run # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # use vantage_vista::prelude::*; # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # struct Product { name: String, price: i64 } # impl Product { # fn table(db: SqliteDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("category_id") # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:products.db").await.context("connect")?; # let vista = db.vista_factory().from_table(Product::table(db.clone()))?; let row = vista.clone().with_id(7)?.get_some_value().await?; // Some(("7", record)) when the row exists # Ok(()) # } ``` [`with_id`](vantage_vista::Vista::with_id) reads the id column name from the schema and adds the equality condition for you. --- ## Search and ordering ```rust,no_run # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # use vantage_vista::prelude::*; # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # struct Product { name: String, price: i64 } # impl Product { # fn table(db: SqliteDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("category_id") # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:products.db").await.context("connect")?; # let vista = db.vista_factory().from_table(Product::table(db.clone()))?; // Quicksearch — fans across columns flagged Searchable — then sort by column let v = vista .clone() .with_search("tart")? .with_order("price", SortDirection::Descending)?; let rows = v.list_values().await?; # Ok(()) # } ``` Both have **replace semantics** — a second search drops the first. Both return an error if the driver doesn't support them, and ordering also refuses a column that isn't flagged `Orderable`. Check capabilities first, or try and handle the error. ```admonish warning title="Not every driver supports these" A CSV file can't sort or search server-side — it loads everything into memory. DynamoDB can only order by its declared sort key. Each driver advertises its capabilities accordingly. Calling a method the driver doesn't advertise returns an `Unsupported` error naming the capability. This is by design — it's better to fail clearly than to silently return unfiltered results. ``` Generic code branches on the capabilities instead, here on a Vista it owns: ```rust,no_run # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # use vantage_vista::prelude::*; # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # struct Product { name: String, price: i64 } # impl Product { # fn table(db: SqliteDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:products.db").await.context("connect")?; # let vista = db.vista_factory().from_table(Product::table(db.clone()))?; let mut v = vista.clone(); if v.capabilities().has(Capability::Search) { v.set_search(Some("query"))?; } if v.capabilities().has(Capability::Order) { v.set_order(Some(("name", SortDirection::Ascending)))?; } # Ok(()) # } ``` --- ## Reading windows Vista has two ways to read part of a set. Not every driver supports both — capabilities tell you which to use. ### A window by offset: `fetch_window` Random access — any `[offset, offset + limit)` slice of the set, in its order: ```rust,no_run # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # use vantage_vista::prelude::*; # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # struct Product { name: String, price: i64 } # impl Product { # fn table(db: SqliteDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:products.db").await.context("connect")?; # let vista = db.vista_factory().from_table(Product::table(db.clone()))?; let first = vista.fetch_window(0, 25).await?; // rows 0..25 let second = vista.fetch_window(25, 25).await?; // rows 25..50 println!("{} rows, total {:?}", first.rows.len(), first.total); # Ok(()) # } ``` A [`Page`](vantage_vista::Page) holds the rows and, when the fetch learned it, the set's total — `None` means "didn't say", never zero. The leading window (`offset == 0`) works on every driver that lists; any other offset needs the `Window` capability. SQL databases and MongoDB have it, anything with `LIMIT … OFFSET`. ### A cursor: `fetch_cursor` Forward only — each call returns up to `limit` rows and a token for the next call: ```rust,no_run # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # use vantage_vista::prelude::*; # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # struct Product { name: String, price: i64 } # impl Product { # fn table(db: SqliteDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:products.db").await.context("connect")?; # let vista = db.vista_factory().from_table(Product::table(db.clone()))?; if vista.capabilities().has(Capability::Cursor) { let (page, token) = vista.fetch_cursor(None, 100).await?; // first page let (page, token) = vista.fetch_cursor(token, 100).await?; // second page // token is None when the set is exhausted } # Ok(()) # } ``` The token is **opaque** — its shape is driver-private (a DynamoDB `LastEvaluatedKey`, a REST `nextToken`, the last key S3 returned). Just round-trip it. Requires the `Cursor` capability, which SQLite doesn't advertise and the S3 listing in the next chapter does. ```admonish info title="When neither is available" Without `Window` and `Cursor`, the leading window and `list_values()` are what the driver offers. That's the CSV case — fine for small datasets. ``` --- ## Traversing references Same-persistence references (the `with_many` / `with_one` from chapter 2) come through automatically. Given a parent row, `get_ref` resolves the relationship: ```rust,no_run # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # use vantage_vista::prelude::*; # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # struct Product { name: String, price: i64 } # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # struct Category { name: String } # impl Product { # fn table(db: SqliteDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("category_id") # .with_one("category", "category_id", Category::table) # } # } # impl Category { # fn table(db: SqliteDB) -> Table { # Table::new("category", db) # .with_id_column("id") # .with_column_of::("name") # .with_many("products", "category_id", Product::table) # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:products.db").await.context("connect")?; let categories = db.vista_factory().from_table(Category::table(db.clone()))?; // Given a category row, traverse to its products if let Some((_, category)) = categories.get_some_value().await? { let products = categories.get_ref("products", Some(&category))?; for (id, row) in products.list_values().await? { println!(" {id} — {:?}", row.get("name")); } } # Ok(()) # } ``` `get_ref` takes the parent row as a `Record`. It reads the join field (e.g. `category_id`) out of that row to build the condition on the target, so a traversal starts from a row: fetch the parent with `get_some_value()` or `list_values()`, then traverse from each row. With `None` in place of the row, you get the bare target — the set a new related row goes into. `get_ref` resolves two kinds of relation: 1. **Contained relations** — records embedded in a column of the parent row (an order's `lines` array) come back as an editable sub-Vista. 2. **Foreign-key references** — the shell forwards to the `with_one` / `with_many` relationships declared on the typed table, and re-wraps the narrowed target as a fresh Vista. Either way the result is another `Vista`, so traversal chains without ever leaving the universal surface. Traversal needs the `Traverse` capability. ### Cross-backend references What if your categories live in PostgreSQL and your products in MongoDB? That is deliberately *not* a Vista's job — a Vista honestly describes one backend, and there is no engine that could execute a join across two. Cross-persistence traversal lives one layer up, in a [`Catalog`](vantage_vista::Catalog): register a loader per model name, and `reference.narrow(catalog.vista(&reference.target)?, &row)` builds the target in *its* backend and narrows it by values read from the parent row. The consumer still just receives a Vista — see [Config-Driven Vistas](../config-driven-vistas.md) for the catalog in action. --- ## Capabilities — the explicit contract [`VistaCapabilities`](vantage_vista::VistaCapabilities) is a set of [`Capability`](vantage_vista::Capability) values. The driver puts in exactly what it implements: ```rust,no_run # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # use vantage_vista::prelude::*; # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # struct Product { name: String, price: i64 } # impl Product { # fn table(db: SqliteDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:products.db").await.context("connect")?; # let vista = db.vista_factory().from_table(Product::table(db.clone()))?; let caps = vista.capabilities(); for capability in caps.iter() { println!("{capability}"); } // order, search, filter_ops, window, traverse, insert, update, delete, … if caps.has(Capability::Insert) { println!("writable"); } # Ok(()) # } ``` Each capability names one method family: `Insert`, `Update`, `Delete`, `Import`, `Subscribe`, `Order`, `Search`, `FilterOps`, `NativeConditions`, `Window`, `Cursor`, `Traverse` and `ConfineWrites` (writes through a narrowed Vista stay inside its set). Listing, getting a row, counting and equality conditions have no capability: every driver answers them. A CSV file advertises `Window` and `Traverse` — it's read-only. SQLite advertises reads, writes, `Window` and `Traverse`, but not `Cursor`. An S3 listing advertises `Cursor` and `Traverse`: no random access, no search, no writes. ```admonish warning title="Calling unsupported methods is an error" The capabilities aren't suggestions — they're a contract. If `Search` isn't in the set, `set_search()` returns an `Unsupported` error, which `VantageError::is_unsupported()` recognises. A contract test calls every advertised capability on every shell in the workspace, so a driver can't advertise a method it doesn't answer. UI adapters branch on these to decide which controls to show. A data grid checks `Window` to decide between a scrollbar (random access) and a "load more" button (`Cursor`). ``` --- ## Putting it together Here's a small function that takes any Vista and prints a summary — works with any backend: ```rust # use vantage_core::Result as VantageResult; # use vantage_vista::prelude::*; async fn print_vista(vista: &Vista) -> VantageResult<()> { let columns: Vec<&String> = vista.columns().keys().collect(); // Header print!(" {:>12} ", vista.id_column().unwrap_or("id")); for col in &columns { print!("{col:>16} "); } println!(); // Count — every driver answers it println!(" ({} rows)", vista.count().await?); // Rows let rows = vista.list_values().await?; for (id, record) in &rows { print!(" {id:>12} "); for col in &columns { let val = record .get(col.as_str()) .map(|v| format!("{v:?}")) .unwrap_or_default(); print!("{val:>16} "); } println!(); } Ok(()) } ``` No generics. No entity type. No backend knowledge. `print_vista` works with a SQLite Vista, a MongoDB Vista, an AWS Vista — anything the framework can produce. --- ## What we covered | Concept | What it does | | ------------------------------------------------------- | ------------------------------------------------------- | | [`Vista`](vantage_vista::Vista) | Universal schema-bearing data handle, wraps any `Table` | | [`TableShell`](vantage_vista::TableShell) | Per-driver executor that Vista delegates to | | [`VistaCapabilities`](vantage_vista::VistaCapabilities) | Explicit contract of what the driver supports | | [`Column`](vantage_vista::Column) | Column metadata with name, type, and flags | | `columns()` / `id_column()` / `references()` | Read the schema at runtime | | `clone().with_condition(..)` | Narrow a copy; the original is untouched | | `with_id` | Convenience: narrow by primary key | | `with_search` / `with_order` | Quicksearch and sorting (replace semantics) | | `add_condition` / `set_search` / `set_order` | The same, in place, on a Vista you own | | `fetch_window` / `fetch_cursor` | Offset windows and cursor pages | | `get_ref` | Traverse a relationship from a parent row | | [`Catalog`](vantage_vista::Catalog) | Cross-persistence traversal, one layer above Vista | ```admonish tip title="What's next" Vista gives you a universal read/write handle. But every call still hits the database — there's no caching, no reactivity, no way to push live updates to a UI. The next chapter introduces **Dio** and **Lens** — the caching and event layer that sits between a Vista (your master data) and a Scenery (the reactive view a UI consumes). ``` --- Source: https://vantage-ui.com/book/intro/fork.html # Choose Your Path You now have a **Vista** — a runtime handle over your data that works the same no matter what's behind it. Everything up to here was the foundation. Everything past here is the **reactive stack**: a local cache, an event bus, and watchable views that update as the data changes. That stack is backend-agnostic, and the rest of the guide proves it by forking here into two paths. Both build the *same* layers — Dio, Lens, Augmentation, Scenery, and a watch-streaming HTTP server — and differ only in the backend underneath. Pick the one shaped like your problem; the code you write is nearly identical either way. ## Which one is yours? ```admonish example title="Path A · A facade over an API you don't control" React / TUI ratatui, web Cache Dio · events Vista AWS S3 AWS API slow, read-only Your data lives behind something slow you cannot change — a cloud API, a legacy service, a third-party endpoint. It is read-only, hundreds of milliseconds away, and cannot sort, search, or paginate. You want a fast, queryable local view of it. The example builds an inventory of a public **S3** bucket (NOAA's climate archive) — thousands of files served seamlessly and responsively. Diorama caches the listing, **injects the capabilities S3 lacks**, and enriches each row from its contents; the path ends with a terminal UI (ratatui) and an Axum API feeding a React frontend. **[Take Path A →](./step5-dio-lens.md)** — steps 5–8. ``` ```admonish tip title="Path B · A live view in front of your own database" React / TUI ratatui, web Cache Dio · events Vista Postgres PostgreSQL you control it Your data lives in a **relational database you own** — you read it, write it, and it already sorts, searches, and joins. What you want is a *live, cached, watchable facade* in front of it: instant reads, changes streaming to every viewer, writes routed on your terms. The example builds a bar's product inventory whose stock ticks down in real time as items sell. The same caching, reactive, watch-streaming stack — and the path moves the app from **SQLite to PostgreSQL** with a single switch, then on to SurrealDB, proving the backend was never load-bearing. **[Take Path B →](./step5-sql-dio.md)** — steps 5–9. ``` ## Why the guide leads with S3 If you own a SQL database, an S3 bucket may look like a strange place to *start* teaching caching and reactivity. That is deliberate. A slow, read-only, capability-poor backend makes the value of the layer above **visible**: every millisecond the cache saves, and every sort the backend cannot do but Diorama can, is obvious. A capable SQL database would hide the very problem the reactive stack solves — so Path A teaches it where it shows, and Path B proves it was never about S3. | | Path A — Custom API (S3) | Path B — Relational DB (SQL) | | ------------------ | ----------------------------------------- | --------------------------------- | | Backend | slow, remote, read-only | fast, local, writable | | Sort / search | no — Diorama injects it | yes — natively | | Writes | routed elsewhere (the master can't take them) | routed to the database | | Reactivity source | periodic reconcile (S3 can't push) | your own writes + reconcile | | The payoff | a rich handle over a poor backend | a live facade, then a backend swap | Both paths converge on the same idea: **once a Dio wraps a Vista, nothing above it knows or cares what the backend is.** Read whichever path is yours — or both, and watch the code stay the same while the backend changes underneath it. --- Source: https://vantage-ui.com/book/intro/step5-dio-lens.html # Dio & Lens — Caching and Events Chapter 4 gave you Vista — a universal handle that works with any backend. But every call still hits the backend. That was easy to forgive with a local SQLite file; this chapter picks a data source where it hurts: a real cloud API, hundreds of milliseconds away, read-only, and unable to sort or search. We'll build a small CLI around it — an inventory of a public S3 bucket, with a cache. **Diorama** (`vantage-diorama`) is the layer that makes this pleasant. It sits between your Vista and whatever consumes it, and does three things: 1. **Transparent caching.** Keep a local copy of your data. Reads come from cache, not from the master — a listing that costs seconds over the network costs microseconds after the first fetch. 2. **Capability injection.** A Vista backed by S3 can't sort, search, or read a window at an offset. Diorama caches the dataset locally and answers those queries from cache — the consumer sees a richer handle than the backend actually offers. 3. **Custom write routing.** Writes don't have to go to the master — useful when the master can't take them at all. You decide what a write means: a queue, a different store, an API call. ```admonish question title="Why 'Diorama'?" The vista from the peak is magnificent — and far away. Sooner or later you want a piece of it close at hand: on your desk, under glass, alive. A **diorama** is exactly that — a crafted miniature of a real scene, small enough to hold, faithful enough to study. That's the layer this chapter builds. A **Lens** is the optics you capture the scene through — ground once, reused for every capture, deciding what is kept and how it refreshes. A **Dio** is one captured segment: your local, living copy of the data, reconciling with the world it depicts. And once the miniature is lit and running, **Scenery** (chapter 7) is what an audience actually watches. ``` --- ## Table, Vista, Dio This is the third handle to the same records — and like the query-vs-table comparison back in chapter 2, each one trades something away for something new: | | `Table` | `Vista` | `Dio` | | ---------------- | ------------------------------- | -------------------------------- | ------------------------------------------ | | **Purpose** | Model your data; business logic | Let generic code consume any table | Keep a live local copy of a data segment | | **Typing** | Compile-time entity & backend | Schema carried at runtime | Same records as Vista | | **Reads** | Query the backend | Query the backend | Served from a local cache | | **Writes** | Applied immediately | Applied immediately | Queued — through the master or a Lens writer | | **Changes** | You re-query | You re-query | Announced on an event bus | | **Capabilities** | Whatever the backend supports | Honestly advertised | Extended — the cache fills the gaps | | **Mode** | Transactional | Transactional | Live | | **Lifecycle** | A value — cheap to clone, narrowed per use | A value — cloned, narrowed, dropped | A shared handle — owns its cache, queue, and background tasks | ### Caching Diorama caches at two levels. **Page segments** hold windows of an ordered query result — they're what lets a viewport scroll a huge listing without re-asking the master, and they power the loading strategies you'll meet in chapter 7. Beneath them sits the **key/value record store** — one entry per record id, plain and dumb. We keep it simple for now: everything in this chapter runs on the record store alone. Dio master Vista — S3 listing ~200 ms per page cache redb — id → record µs away the Lens pumps master → cache facade Vista dio.cached() Sceneries chapter 7 reads — µs A **Dio** owns exactly the two blocks above: a *master* Vista — the source of truth, however far away — and a *cache*, the local copy. What it deliberately does **not** decide is policy: when the cache fills, when it goes stale, what a write means. There are too many valid answers for one default — seed once and keep forever, re-fetch on a timer, reconcile from a push stream, write through to the master, write somewhere else entirely. That's what the **Lens** is for: it describes how the cache is used. The simplest useful Lens pumps the master into the cache once, when the Dio is built: ```rust,no_run # use serde::{Deserialize, Serialize}; # use vantage_aws::prelude::*; # use vantage_diorama::prelude::*; # use vantage_diorama::{Cache, Loader, LoaderCx}; # use vantage_vista::prelude::*; # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # struct File { # #[serde(rename = "Key")] # filename: String, # #[serde(rename = "Size")] # size: String, # } # impl File { # fn table(aws: AwsAccount, bucket: &str, prefix: &str) -> Table { # Table::new( # "restxml/Contents@continuation-token=NextContinuationToken:s3/GET /{Bucket}?list-type=2", # aws, # ) # .with_id_column("Key") # .with_column_of::("Size") # .with_condition(eq("Bucket", bucket)) # .with_condition(eq("prefix", prefix)) # .with_condition(eq("max-keys", 100)) # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let aws = AwsAccount::public("us-east-1"); # let master = aws # .vista_factory() # .from_table(File::table(aws, "noaa-ghcn-pds", "csv/by_station/GM"))?; let lens = Lens::builder() .cache(Cache::redb("cache.redb")) .loader(Loader::eager(|cx: LoaderCx| async move { // Pump: read everything from the master, make the cache hold exactly that. let rows = cx.master().list_values().await?; cx.reconcile(rows).await })) .build()?; let dio = lens.dio(master).build().await?; // runs the eager start, returns a warm Dio # Ok(()) # } ``` With the Dio in place, there are two ways to consume it. The first is the **facade Vista** — `dio.cached()` — for *proactive* querying: you ask, it answers, exactly like the backend's own Vista from chapter 4. Same interface, same records — except the answers come from the cache, and the capability set can be *wider* than the master's: the facade searches, orders and reads windows whatever the master can do, and a read-only master gains writes when the Lens routes them somewhere. The second way in is the **Scenery** — a standing, *reactive* view that keeps itself current and tells you when it changed. UIs bind to sceneries; chapter 7 lives there. Dio master + cache facade Vista dio.cached() proactive: you ask — list, get, count… …it answers from the cache Scenery a standing view reactive: changes stream in — the generation counter announces each one ### The facade Vista Hand `dio.cached()` to chapter 4's `print_vista` and it just works — it never learns a cache is underneath. Each call returns a fresh, independent Vista, so you narrow it with `with_*` like any other and no other consumer sees the change. What makes the facade interesting is its capability set. Chapter 4's contract was honest but rigid: whatever the backend can't do, your application can't have. The facade advertises what the *pipeline* can do: `Search`, `Order`, `FilterOps` and `Window` always (a clause the master takes is pushed down, and the rest is evaluated over the cached rows), `Subscribe` because the event bus backs it, and the writes the master can take — or all of them once a Lens `writer` gives writes somewhere to go. The honesty contract still holds; it just describes a different engine. | | facade Vista | Scenery | | -------------------- | ------------------------------------------- | ------------------------------------------------ | | **Access style** | Proactive — you ask, it answers | Reactive — a standing view that stays current | | **Interface** | Chapter 4's Vista API, unchanged | Purpose-built: rows by index, a record, a scalar | | **Freshness** | As fresh as the cache when you ask | Recomputes on every change | | **Change awareness** | None — ask again | A generation counter that bumps on each change | | **Capabilities** | Search, order, windows, plus the writes the pipeline routes | Ordering, search, viewport — regardless of backend | | **Typical consumer** | Handlers, scripts, CLI commands | UIs, dashboards, live views | --- ## The project: a weather-station inventory NOAA publishes its daily climate archive — GHCN, one CSV per weather station — as a public S3 bucket in the [AWS Open Data registry](https://registry.opendata.aws/noaa-ghcn/). It's a perfect Diorama subject: listing it is slow, every listing request is paid again on every run, and the API can't sort or search. We'll grow one small tool across three chapters: - **This chapter** — a CLI that lists the station files from a persistent local cache. - **Chapter 6** — *augmentation*: each file gains columns computed from its contents (how many readings, how recent). - **Chapter 7** — a live terminal UI that scrolls the whole archive and fetches per-file data for exactly the rows on screen. ```admonish info title="No AWS account required" The bucket allows anonymous access, the way `aws s3 ls --no-sign-request` reads it. `AwsAccount::public(region)` is the vantage-aws equivalent: requests go out unsigned, so there are no credentials to configure — nothing to install, nothing to pay. ``` Set up the project: ```sh cargo new learn-4 && cd learn-4 cargo add serde --features derive cargo add tokio --features full cargo add vantage-aws vantage-diorama vantage-vista ``` `vantage-aws` is the S3/DynamoDB/IAM driver — it signs (or deliberately doesn't sign) requests itself, so there's no AWS SDK in the tree. `vantage-diorama` is what this chapter is about. The code splits into two files: `files.rs` holds the table definition, `main.rs` uses it. Every snippet that talks to S3 or opens `cache.redb` compiles in the book's tests but doesn't run there; the definitions without either do run. --- ## `files.rs` — the listing as a table One row per file in the bucket. This is chapter 2's pattern — an entity, and a `table()` constructor that describes the source — pointed at a cloud API instead of a database: ```rust use serde::{Deserialize, Serialize}; use vantage_aws::prelude::*; /// One file in the bucket. Field names match S3's wire XML /// (``). #[derive(Debug, Clone, Default, Serialize, Deserialize)] pub struct File { #[serde(rename = "Key")] pub filename: String, #[serde(rename = "Size")] pub size: String, } ``` The serde renames map S3's wire names onto the fields we actually want to write in Rust. `Size` is a `String` because that's what the XML carries — no silent coercion. ```rust # use serde::{Deserialize, Serialize}; # use vantage_aws::prelude::*; # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # pub struct File { # #[serde(rename = "Key")] # pub filename: String, # #[serde(rename = "Size")] # pub size: String, # } impl File { /// `ListObjectsV2` narrowed to one bucket and prefix. S3 sends at /// most `max-keys` keys per response; the `@continuation-token` /// cursor tells the framework to keep requesting pages until the /// listing is complete. pub fn table(aws: AwsAccount, bucket: &str, prefix: &str) -> Table { Table::new( "restxml/Contents@continuation-token=NextContinuationToken:s3/GET /{Bucket}?list-type=2", aws, ) .with_id_column("Key") .with_column_of::("Size") .with_condition(eq("Bucket", bucket)) .with_condition(eq("prefix", prefix)) .with_condition(eq("max-keys", 100)) } } ``` The table *name* is doing a lot of work here — for `vantage-aws`, it's the wire protocol spelled out: `restxml` is the protocol S3 speaks; `Contents` is the response element holding the rows; `s3/GET /{Bucket}?list-type=2` is the service and request. Conditions complete the request the same way chapter 2's conditions completed a WHERE clause: `Bucket` fills the path placeholder, and anything else (`prefix`, `max-keys`) becomes a query parameter. The `@continuation-token=NextContinuationToken` part is **auto-pagination**. S3 answers at most `max-keys` files per response, plus a continuation token when more exist; the cursor declaration tells the driver to keep re-issuing the request — token folded back in — until the listing is complete. One `list()` call, as many HTTP requests as it takes. ```admonish info title="The table-name grammar" The full shape vantage-aws parses is: ~~~text {protocol}/{array_key}[@cursor|@request=response]:{service}/{METHOD} {path}?{query} ~~~ `protocol` is one of `json1`, `json10`, `query`, `restxml`, `restjson`; `array_key` names the response element holding the rows; the optional `@` suffix declares the pagination cursor (one name if request and response fields match, `request=response` when they differ, as here). Get it wrong and the first query returns a `VantageError` quoting this grammar — an unknown protocol lists the valid ones. Nothing is guessed and nothing fails silently. ``` ## A first listing `main.rs`, shortest possible version. It starts with `mod files;`, then builds the table, lists it and prints: ```rust,no_run # mod files { # use serde::{Deserialize, Serialize}; # use vantage_aws::prelude::*; # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # pub struct File { # #[serde(rename = "Key")] # pub filename: String, # #[serde(rename = "Size")] # pub size: String, # } # impl File { # pub fn table(aws: AwsAccount, bucket: &str, prefix: &str) -> Table { # Table::new( # "restxml/Contents@continuation-token=NextContinuationToken:s3/GET /{Bucket}?list-type=2", # aws, # ) # .with_id_column("Key") # .with_column_of::("Size") # .with_condition(eq("Bucket", bucket)) # .with_condition(eq("prefix", prefix)) # .with_condition(eq("max-keys", 100)) # } # } # } use files::File; use vantage_aws::prelude::*; const BUCKET: &str = "noaa-ghcn-pds"; const PREFIX: &str = "csv/by_station/GM"; #[tokio::main] async fn main() { if let Err(e) = run().await { e.report(); } } async fn run() -> VantageResult<()> { let aws = AwsAccount::public("us-east-1"); let files = File::table(aws, BUCKET, PREFIX); for (filename, file) in files.list().await? { println!("{:>10} {filename}", file.size); } Ok(()) } ``` The `GM` prefix narrows the archive to Germany's stations — 1122 files, small enough to look at, big enough to feel: ```text 4796352 csv/by_station/GM000001153.csv 7057976 csv/by_station/GM000001474.csv 2978238 csv/by_station/GM000002277.csv ... 1122 files in 2.3s ``` 2.3 seconds: twelve HTTPS round-trips (1122 files at 100 per page), paid **on every run**, forever, because nothing remembers the answer. This is the itch the rest of the chapter scratches. --- ## The master Vista, and what it can't do The Dio wants a Vista for its master, so we erase the typed table the same way chapter 4 did: ```rust,no_run # use serde::{Deserialize, Serialize}; # use vantage_aws::prelude::*; # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # struct File { # #[serde(rename = "Key")] # filename: String, # #[serde(rename = "Size")] # size: String, # } # impl File { # fn table(aws: AwsAccount, bucket: &str, prefix: &str) -> Table { # Table::new( # "restxml/Contents@continuation-token=NextContinuationToken:s3/GET /{Bucket}?list-type=2", # aws, # ) # .with_id_column("Key") # .with_column_of::("Size") # .with_condition(eq("Bucket", bucket)) # .with_condition(eq("prefix", prefix)) # .with_condition(eq("max-keys", 100)) # } # } # const BUCKET: &str = "noaa-ghcn-pds"; # const PREFIX: &str = "csv/by_station/GM"; # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let aws = AwsAccount::public("us-east-1"); let master = aws .vista_factory() .from_table(File::table(aws, BUCKET, PREFIX))?; # Ok(()) # } ``` Ask this Vista what it can do, and you'll see why this backend needs a Diorama: ```rust,no_run # use serde::{Deserialize, Serialize}; # use vantage_aws::prelude::*; # use vantage_vista::prelude::*; # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # struct File { # #[serde(rename = "Key")] # filename: String, # #[serde(rename = "Size")] # size: String, # } # impl File { # fn table(aws: AwsAccount, bucket: &str, prefix: &str) -> Table { # Table::new( # "restxml/Contents@continuation-token=NextContinuationToken:s3/GET /{Bucket}?list-type=2", # aws, # ) # .with_id_column("Key") # .with_column_of::("Size") # .with_condition(eq("Bucket", bucket)) # .with_condition(eq("prefix", prefix)) # .with_condition(eq("max-keys", 100)) # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let aws = AwsAccount::public("us-east-1"); # let master = aws # .vista_factory() # .from_table(File::table(aws, "noaa-ghcn-pds", "csv/by_station/GM"))?; let caps = master.capabilities(); // Cursor and Traverse — and that's it. assert!(caps.has(Capability::Cursor)); assert!(!caps.has(Capability::Search) && !caps.has(Capability::Order)); assert!(!caps.has(Capability::Window) && !caps.has(Capability::Insert)); # Ok(()) # } ``` `Cursor` is worth pausing on — it's the one capability this chapter leans on. `fetch_cursor(token, limit)` is Vista's cursor pagination from chapter 4: pass `None` for the first page, pass back the returned token for the next one. For S3 the driver defines the token as **the last key of the previous page**, because S3 lists keys in order and accepts any key as a starting point (`start-after`). That has a consequence the generic contract doesn't promise: this cursor survives process restarts. Any key you already hold — say, the last key in a cache — resumes the listing right after it. ## The Lens: pump pages, resume where you left off Now the Lens. One setting does the work — an eager loader that, when the Dio is built, synchronizes the cache with the bucket: ```rust,no_run # use std::time::Instant; # use serde::{Deserialize, Serialize}; # use vantage_aws::prelude::*; # use vantage_diorama::prelude::*; # use vantage_diorama::{Cache, Loader, LoaderCx}; # use vantage_vista::prelude::*; # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # struct File { # #[serde(rename = "Key")] # filename: String, # #[serde(rename = "Size")] # size: String, # } # impl File { # fn table(aws: AwsAccount, bucket: &str, prefix: &str) -> Table { # Table::new( # "restxml/Contents@continuation-token=NextContinuationToken:s3/GET /{Bucket}?list-type=2", # aws, # ) # .with_id_column("Key") # .with_column_of::("Size") # .with_condition(eq("Bucket", bucket)) # .with_condition(eq("prefix", prefix)) # .with_condition(eq("max-keys", 100)) # } # } # const CACHE_FILE: &str = "cache.redb"; # const PAGE_SIZE: usize = 1000; # async fn sync(cx: &LoaderCx) -> VantageResult<()> { # let mut token: Option = # cx.cache().list_values().await?.keys().last().cloned().map(Into::into); # loop { # let start = Instant::now(); # let (page, next) = cx.master().fetch_cursor(token, PAGE_SIZE).await?; # let count = page.rows.len(); # cx.cache().insert_values(page.rows.into_iter().collect()).await?; # println!("fetched {count} files in {:?}", start.elapsed()); # if next.is_none() { # return Ok(()); # } # token = next; # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let aws = AwsAccount::public("us-east-1"); # let master = aws # .vista_factory() # .from_table(File::table(aws, "noaa-ghcn-pds", "csv/by_station/GM"))?; let lens = Lens::builder() .cache(Cache::redb(CACHE_FILE)) .loader(Loader::eager(|cx: LoaderCx| async move { sync(&cx).await })) .build() .context("Failed to build lens")?; let dio = lens.dio(master).build().await?; # Ok(()) # } ``` - **`Lens::builder()`** collects the settings; **`.build()`** checks them and returns the Lens, already in an `Arc` because every Dio it builds shares it. - **`.cache(Cache::redb(CACHE_FILE))`** opens a [redb](https://docs.rs/redb) file on disk. This is why the cache survives process restarts. `Cache::memory()` is the alternative for when persistence isn't wanted. - **`Loader::eager(...)`** runs once per Dio, inside `build()`, which waits for it. The callback gets a [`LoaderCx`](vantage_diorama::LoaderCx): an owned, cheap-to-clone context with the Dio's `master()` and `cache()`, so the future can hold it across `.await` without borrowing anything. - **`lens.dio(master).build().await`** does the rest in one call: opens a cache table named after the master, spawns the write worker, runs the eager start, and hands back the [`Dio`](vantage_diorama::Dio). `sync` is where the cursor pays off. Instead of one big `list_values()`, it pulls the listing **one page at a time** and seeds the cursor from the cache: ```rust # use std::time::Instant; # use vantage_aws::prelude::*; # use vantage_diorama::LoaderCx; # use vantage_vista::prelude::*; # const PAGE_SIZE: usize = 1000; /// Pump the master listing into the cache, one page per request. S3's /// paging cursor is simply "the last key seen" — so the last key already /// in the cache resumes the listing, and pages loaded by an earlier run /// (even one that was interrupted) are never fetched again. async fn sync(cx: &LoaderCx) -> VantageResult<()> { let mut token: Option = cx .cache() .list_values() .await? .keys() .last() .cloned() .map(Into::into); loop { let start = Instant::now(); let (page, next) = cx.master().fetch_cursor(token, PAGE_SIZE).await?; let count = page.rows.len(); cx.cache() .insert_values(page.rows.into_iter().collect()) .await?; println!("fetched {count} files in {:?}", start.elapsed()); if next.is_none() { return Ok(()); } token = next; } } ``` Read the first statement again: the initial token is *the last filename already cached*. On a cold cache that's `None` — start from the top. On a warm cache it's the end of what we have — S3 continues from there. Kill the process halfway through a sync and run it again: the pages you already paid for are never fetched twice. The cache isn't just an answer store; it's the resume point. `PAGE_SIZE` is the most keys one request asks for; S3 returns at most 1000. ## Reads come from the cache The listing loop stops asking S3 and starts asking the Dio — through the **facade Vista** from the first half of the chapter. Notice who talks to what: `sync`, being Lens plumbing, addresses the Dio's two sides through its `LoaderCx` (`cx.master()` to fetch, `cx.cache()` to store and resume); a *consumer* asks `dio.cached()` and never learns what's underneath. The records come back as `Record` — the erased form from chapter 4 — so field access goes through the `CborValueExt` helpers the prelude brings in: ```rust,no_run # use std::time::Instant; # use serde::{Deserialize, Serialize}; # use vantage_aws::prelude::*; # use vantage_diorama::prelude::*; # use vantage_diorama::Cache; # use vantage_vista::prelude::*; # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # struct File { # #[serde(rename = "Key")] # filename: String, # #[serde(rename = "Size")] # size: String, # } # impl File { # fn table(aws: AwsAccount, bucket: &str, prefix: &str) -> Table { # Table::new( # "restxml/Contents@continuation-token=NextContinuationToken:s3/GET /{Bucket}?list-type=2", # aws, # ) # .with_id_column("Key") # .with_column_of::("Size") # .with_condition(eq("Bucket", bucket)) # .with_condition(eq("prefix", prefix)) # .with_condition(eq("max-keys", 100)) # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let aws = AwsAccount::public("us-east-1"); # let master = aws # .vista_factory() # .from_table(File::table(aws, "noaa-ghcn-pds", "csv/by_station/GM"))?; # let dio = Lens::builder().cache(Cache::redb("cache.redb")).build()?.dio(master).build().await?; let start = Instant::now(); let listing = dio.cached().list_values().await?; for (filename, file) in &listing { let size = file.get("Size").and_then(|v| v.as_str()).unwrap_or(""); println!("{size:>10} {filename}"); } println!("{} files from cache in {:?}", listing.len(), start.elapsed()); # Ok(()) # } ``` First run — cold cache, the sync pump narrating each page: ```text fetched 100 files in 215ms fetched 100 files in 217ms ... fetched 22 files in 114ms 4796352 csv/by_station/GM000001153.csv 7057976 csv/by_station/GM000001474.csv ... 1122 files from cache in 18ms ``` Second run — same command, new process: ```text fetched 0 files in 370ms 1122 files from cache in 17ms ``` One round-trip — the resume request from the last cached key, confirming nothing new exists — and then the full listing in 17 milliseconds. New files that *do* appear under the prefix arrive on exactly that request, without re-fetching the thousand we already hold. ## Invalidating A resuming cache has one blind spot: it only ever looks *past* what it holds, so files deleted from the bucket linger locally. The honest fix is to start over. The Lens opens `cache.redb` when it is built, so the place to start over is before that: delete the file, and the eager start finds an empty cache and lists the bucket from the top. ```rust,no_run # use vantage_aws::prelude::*; # const CACHE_FILE: &str = "cache.redb"; # fn main() -> VantageResult<()> { // In `run`, before the Lens is built: if std::env::args().any(|a| a == "--invalidate") { remove_cache()?; } # Ok(()) # } fn remove_cache() -> VantageResult<()> { match std::fs::remove_file(CACHE_FILE) { Err(e) if e.kind() != std::io::ErrorKind::NotFound => { Err(e).context("Failed to remove the cache") } _ => Ok(()), } } ``` ```text $ cargo run -- --invalidate fetched 100 files in 215ms fetched 100 files in 217ms ... 1122 files from cache in 18ms ``` The CLI is complete: two seconds once, milliseconds forever after, resumable mid-sync, and an escape hatch back to the truth. --- ## The event bus One more thing `build()` set up, invisibly: every Dio carries a **broadcast event bus**, which `dio.subscribe_events()` listens to. Anything that changes data announces it there as a [`DioEvent`](vantage_diorama::DioEvent): `Seeded` when an eager start has put the whole set in the cache, `DatasetChanged` when the set was rewritten wholesale, `RecordChanged` / `RecordInserted` / `RecordRemoved` for row-level changes, `Refreshing` and `RefreshFailed` around a re-read, and `WritePending` / `WriteReverted` / `WriteFailed` as writes move through the queue. Nothing in this chapter listens yet; chapter 7's sceneries subscribe to this bus to know when to recompute, and chapter 6's hydration uses it to report progress. One rule is worth forming as a habit now: **direct cache writes are silent**. Our `sync` calls `cx.cache().insert_values(...)`, which stores rows and announces nothing. Inside an eager start that's fine — the Dio publishes `Seeded` once the start returns, and every scenery reloads from the cache. Anywhere else, use the calls that write *and* announce in one motion: `cx.reconcile(rows)` makes the cache hold exactly the set you hand it (writing what differs, deleting what's gone) and publishes one `DatasetChanged`; `dio.apply_change(...)` takes one upstream [`ChangeEvent`](vantage_diorama::ChangeEvent) — `Inserted`, `Updated`, `Deleted` or `Invalidated` — and publishes the matching row event. ```admonish info title="Poll or push — the Lens doesn't care" Our S3 listing has no way to notify us, so freshness is *pulled*: the eager start on every run, `--invalidate` by hand, or a `Refresher` on a timer (chapter 7's live view uses one). Backends that can push take the other route: `lens.dio(master).live()` follows a master that advertises `Subscribe` (a SurrealDB live query, Postgres `LISTEN/NOTIFY`), and anything else — a Kafka topic, a webhook — calls `dio.apply_change(...)` itself. Same cache, same events — only the trigger differs. ``` ## Writes, on a read-only master S3's listing Vista doesn't advertise `Insert`, and the facade won't pretend otherwise — by default. A write enters a Dio as a [`ChangeFlash`](vantage_diorama::ChangeFlash) handed to `dio.flash(...)` (the facade's own `insert_value` and friends become flashes too). The flash is staged in the cache at once, joins the Dio's write queue, and the queue writes it through the master. A master that can't take it fails the write: the staged row is rolled back, `WriteReverted` goes out on the bus, and `flash` returns the error — never a silent drop. ```rust,no_run # use serde::{Deserialize, Serialize}; # use vantage_aws::prelude::*; # use vantage_diorama::prelude::*; # use vantage_diorama::Cache; # use vantage_types::Record; # use vantage_vista::prelude::*; # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # struct File { # #[serde(rename = "Key")] # filename: String, # #[serde(rename = "Size")] # size: String, # } # impl File { # fn table(aws: AwsAccount, bucket: &str, prefix: &str) -> Table { # Table::new( # "restxml/Contents@continuation-token=NextContinuationToken:s3/GET /{Bucket}?list-type=2", # aws, # ) # .with_id_column("Key") # .with_column_of::("Size") # .with_condition(eq("Bucket", bucket)) # .with_condition(eq("prefix", prefix)) # .with_condition(eq("max-keys", 100)) # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let aws = AwsAccount::public("us-east-1"); # let master = aws # .vista_factory() # .from_table(File::table(aws, "noaa-ghcn-pds", "csv/by_station/GM"))?; # let dio = Lens::builder().cache(Cache::redb("cache.redb")).build()?.dio(master).build().await?; let mut row = Record::new(); row.insert("Size".to_string(), CborValue::from("0")); let written = dio.flash(ChangeFlash::insert("csv/by_station/NOTE.csv", row)).await; // Err: the S3 listing takes no inserts, and the staged row is rolled back # Ok(()) # } ``` To make the write mean something else, give the Lens a **`writer`**. Every flash the queue runs is then handed to your closure instead of the master — append to a journal, call a different API, publish to a queue. This is exactly the introduction's "a read-only CSV file accepts writes by routing them into a queue". `Ok` confirms the write and keeps the staged row; `Err` rolls it back. A writer also makes every Dio of the Lens advertise the writes on its facade: ```rust,no_run # use vantage_aws::prelude::*; # use vantage_diorama::prelude::*; # use vantage_diorama::{Cache, FlashOutcome, LoaderCx}; # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let lens = Lens::builder() .cache(Cache::redb("cache.redb")) .writer(|_cx: LoaderCx, flash: ChangeFlash| async move { println!("journal: {:?} {:?}", flash.kind(), flash.id()); Ok(FlashOutcome::applied()) }) .build()?; # Ok(()) # } ``` ## The four settings A Lens has four settings, each with its own moment: | Setting | When it runs | In this chapter | | ------- | ------------ | --------------- | | [`Loader`](vantage_diorama::Loader) | `eager`: once per Dio, inside `build()` (or in the background with `.blocking(false)`). `paged`: each window a table scenery asks for. `paged(..).or_eager(..)`: pages a master with `Window`, copies one without | `Loader::eager` — `sync`, resuming from the cache | | [`Refresher`](vantage_diorama::Refresher) | On `dio.refresh()`; every interval with `.every(..)`; at build and scenery open when the cache is older than `.max_age(..)` | Not used — chapter 7 refreshes on a timer | | `writer` | Every write the Dio's queue runs, in order, one at a time | Not used — the S3 listing is read-only | | [`Tuning`](vantage_diorama::Tuning) | Never runs; read by every Dio: write-queue size, viewport debounce, augmentation workers, refresh on open, aggregate debounce | Defaults | Everything a Dio adds beyond these — augmentation in chapter 6, a live follow with `.live()` — is chosen on `lens.dio(master)` before `build()`, because it belongs to that one Dio, not to every Dio the Lens builds. ### Why a builder that can refuse? `Lens::builder().build()` returns a `Result`, and it does refuse things. Some settings only make sense together, and each wrong combination would otherwise fail much later, in a way that's hard to trace back: `Loader::blocking` on a loader with no eager start (there's nothing to block on), `Refresher::every(Duration::ZERO)` (a timer with no period), a write queue of size zero (a queue that can never take a write), a Lens built outside a tokio runtime (nowhere to spawn its tasks). `build()` checks the combination once and names the setting at fault, so a misconfiguration is an error at startup, not a stall at the first write: ```rust # use std::time::Duration; # use vantage_diorama::prelude::*; # use vantage_diorama::{Cache, LoaderCx, Refresher}; # use vantage_vista::prelude::*; # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let refused = Lens::builder() .cache(Cache::memory()) .refresher( Refresher::new(|cx: LoaderCx| async move { let rows = cx.master().list_values().await?; cx.reconcile(rows).await }) .every(Duration::ZERO), ) .build(); assert!(refused.is_err()); // "Refresher::every needs a non-zero interval" # Ok(()) # } ``` The cost is one `?` at the call site. A builder that took anything and guessed would save that, and turn each of these mistakes into a hang. --- ## What we covered | Concept | What it does | | --------------------------------------------------- | ---------------------------------------------------------------- | | `AwsAccount::public(region)` | Unsigned requests — public buckets, no credentials | | Cursor in the table name (`@req=resp`) | Auto-pagination: one `list()`, as many requests as needed | | [`Lens`](vantage_diorama::Lens) | Shared policy: cache, loader, refresher, writer, tuning | | [`Cache`](vantage_diorama::Cache) | `redb(path)` on disk, `memory()` in process | | [`Dio`](vantage_diorama::Dio) | Master Vista + cache under a Lens; owns the queue and event bus | | `fetch_cursor(token, limit)` | One page per call; S3's token is the last key — durable | | [`LoaderCx`](vantage_diorama::LoaderCx) | The Dio's `master()` and `cache()`, handed to every callback | | `dio.cached()` | Facade Vista: reads from cache, writes through the queue | | `dio.flash(ChangeFlash::…)` | A write: staged, queued, written through, confirmed or reverted | | [`DioEvent`](vantage_diorama::DioEvent) | Bus event: seeded, dataset and row changes, write progress | ```admonish tip title="What's next" The inventory knows every file's name and size — and nothing about what's inside. The next chapter teaches the Dio to *augment* its rows: each station file gains a reading count and a latest-reading date, computed from the file itself, fetched once, cached forever. ``` --- Source: https://vantage-ui.com/book/intro/step6-augmentation.html # Augmentation — Enriching Rows Chapter 5's inventory knows every station file's name and size — the two things an S3 listing gives away for free. Everything interesting is *inside* the files: how many readings a station has recorded, and how recently. Getting that costs one download per file, which is exactly the kind of expense you want to pay once and remember. **Augmentation** is Diorama's answer: enrich the master's rows, one row at a time, from a *detail* source — and let the Dio's cache hold the result. The listing stays the cheap, fast spine it was in chapter 5; each row *gains columns* it never had, hydrated on demand and persisted alongside the row. This chapter adds two such columns to the inventory: ```text SIZE ROWS LATEST FILENAME 4796352 140629 19911231 csv/by_station/GM000001153.csv ``` `ROWS` — how many readings the file holds; `LATEST` — the date of the most recent one. Neither exists anywhere in S3's listing; both are computed from the file's contents. We build on chapter 5's crate unchanged — `learn-5` starts as a copy of `learn-4` (`files.rs` and `main.rs` exactly as they were) and this chapter only adds. One new file, `readings.rs`, describes the detail side; a handful of lines in `main.rs` wire it up. --- ## The detail side: `readings.rs` Every station file is a CSV of date-ordered readings: ```text ID,DATE,ELEMENT,DATA_VALUE,M_FLAG,Q_FLAG,S_FLAG,OBS_TIME GM000001153,18910101,TMAX,-20,,,I, GM000001153,18910102,TMAX,-40,,,I, ... ``` So the two columns we're after are cheap *derivations*: `rows` is the line count minus the header, `latest` is the `DATE` field of the last line. The expensive part is getting the contents at all. The detail source is a table — the same `ListObjectsV2` listing as the master, in fact, with two differences: it declares **no prefix** (the augmentation will narrow it to a single file per fetch), and it carries the derived columns. Start `readings.rs` with its entity: ```rust use serde::{Deserialize, Serialize}; use vantage_aws::prelude::*; /// One file, seen through the augmenter: the listing row plus columns /// computed from the file's contents. #[derive(Debug, Clone, Default, Serialize, Deserialize)] pub struct Readings { #[serde(rename = "Key")] pub filename: String, pub rows: i64, pub latest: String, } ``` ### Lazy expressions Where do `rows` and `latest` come from? Chapter 2's `with_expression` won't do it — those expressions lower *into the backend's query* (a SQL subselect), and S3 can't compute anything server-side. What we need runs on our side, **after** the row comes back. That's a **lazy expression**: a column computed in Rust, on the returned record. Lazy expressions apply in declaration order, and each callback *borrows the record as built so far* — including the columns earlier lazy expressions added. That ordering rule is the whole trick: ```rust # use serde::{Deserialize, Serialize}; # use vantage_aws::prelude::*; # #[derive(Debug, Clone, Default, Serialize, Deserialize)] # pub struct Readings { # #[serde(rename = "Key")] # pub filename: String, # pub rows: i64, # pub latest: String, # } impl Readings { pub fn table(aws: AwsAccount, bucket: &str) -> Table { let bucket = bucket.to_string(); Table::new("restxml/Contents:s3/GET /{Bucket}?list-type=2", aws.clone()) .with_id_column("Key") .with_condition(eq("Bucket", bucket.clone())) .with_lazy_expression("contents", move |row| { let aws = aws.clone(); let bucket = bucket.clone(); let key = row.get("Key").and_then(|v| v.as_str()).unwrap_or_default().to_string(); async move { Ok(s3::get_object(&aws, &bucket, &key).await?.into()) } }) .with_lazy_expression("rows", |row| { // Every line after the CSV header is one reading. let contents = row.get("contents").and_then(|v| v.as_str()).unwrap_or_default(); let rows = contents.lines().count().saturating_sub(1) as i64; async move { Ok(rows.into()) } }) .with_lazy_expression("latest", |row| { // Readings are date-ordered; take the last line's DATE column. let contents = row.get("contents").and_then(|v| v.as_str()).unwrap_or_default(); let latest = contents .lines() .last() .and_then(|line| line.split(',').nth(1)) .unwrap_or_default() .to_string(); async move { Ok(latest.into()) } }) } } ``` Read it as a pipeline over one record. The listing returns `{Key, Size, …}`; then: 1. **`contents`** reads `Key` off the record and downloads the file — `s3::get_object` is the driver's raw fetch, unsigned here just like the listing. Its return value is inserted into the record under `contents`. This is the expensive step, and it happens **once**. 2. **`rows`** never touches the network: it reads `row.get("contents")` — the column the previous expression just added — and counts lines. 3. **`latest`** reads the same `contents` and takes the last line's date. One download feeds every derived column declared after it. Each callback clones what it needs out of the borrowed record before going async, and each expression's name is also registered as a column on the table, so the derived fields are part of its schema like any other. ```admonish info title="Lazy expressions from YAML" Like most things declared on a table, lazy columns have a config-driven form: a column spec may carry `lazy: `, where the script sees the record built so far as `row` and its final expression becomes the value — `row.contents.split("\n").len() - 1`. Declaration order chains the same way. A `lazy:` script can't do I/O, so it suits the `rows` and `latest` steps but not the download. See [Computed Columns](../rhai/computed.md). ``` ## Why not just list this table? It's tempting to stop here — the augmenter table already produces every column we want, so why not make *it* the Dio's master? Because lazy expressions run **when data is fetched**. A `list()` on this table triggers the whole pipeline for every row it returns: a thousand files means a thousand downloads inside one blocking call, and nothing comes back until the last one lands. That's not a listing — it's a batch job wearing a listing's interface. What we're after is different. The list of files should reach the user **instantly** — it's chapter 5's cached listing, it costs milliseconds. The expensive columns should then be fetched **one row at a time**, each result written into the Dio and announced on the event bus, so anything watching updates progressively. Loading details for every station takes a long time no matter what; the win is in never blocking the listing on it, and in choosing the *order* — the next chapter puts a viewport in charge, so the rows the user is currently looking at are hydrated first, and rows nobody scrolls to are never fetched at all. That split — a cheap master everyone lists, an expensive detail source consulted per row — is what **augmentation** declares. ## Wiring the augmentation Back in `main.rs`. The augmenter becomes a Vista like any table, and the Dio is told how to use it. Chapter 5 built its Dio with `lens.dio(master).build()`; the augmentation slots in between the two calls, and that's the only structural change to `run()`: ```rust,no_run # use std::sync::Arc; # use vantage_aws::prelude::*; # use vantage_diorama::{Augmentation, Detail, Fetch, Lens, MergeRule, Source}; # use vantage_types::EmptyEntity; # use vantage_vista::Vista; # const BUCKET: &str = "noaa-ghcn-pds"; # struct Readings; # impl Readings { # fn table(aws: AwsAccount, bucket: &str) -> Table { # Table::new("restxml/Contents:s3/GET /{Bucket}?list-type=2", aws) # .with_id_column("Key") # .with_condition(eq("Bucket", bucket)) # } # } # async fn run(aws: AwsAccount, lens: Arc, master: Vista) -> vantage_core::Result<()> { let augmenter = aws .vista_factory() .from_table(Readings::table(aws.clone(), BUCKET))?; let dio = lens .dio(master) .augment(vec![Augmentation::new( Detail::Fixed(Arc::new(augmenter)), Source::Column { from: "Key".into(), to: Some("prefix".into()), }, Fetch::PerRow, MergeRule::lift(["rows", "latest"]), )]) .build() .await?; # let _ = dio; # Ok(()) # } ``` An [`Augmentation`](vantage_diorama::Augmentation) answers four questions: - **`detail`**: *where do detail records come from?* `Detail::Fixed` holds our augmenter Vista directly. The config-driven form is `Detail::Resolved { resolver, name }`: a config carries only a name, so the detail Vista is looked up by that name on each fetch, through a [`VistaResolver`](vantage_vista::VistaResolver) such as a [`Catalog`](vantage_vista::Catalog). The resolver travels with the detail, so the Dio needs no catalog of its own and `augment` takes nothing but the list. - **`source`** — *how does a master row select its detail record?* `Source::Column` maps the master's `Key` onto the detail table's `prefix` condition. A full filename used as an S3 prefix matches exactly one object — so each fetch narrows the augmenter to a one-row listing, and the lazy expressions run for precisely that file. - **`fetch`** — `Fetch::PerRow`: one detail fetch per master row. - **`merge`** — *which detail columns land on the master row?* Just `rows` and `latest`. `contents` is deliberately absent: it exists only inside the detail fetch, feeds the derived columns, and is never cached. The megabytes stay out of the Dio; the two numbers stay in. ## Reads hydrate Nothing has fetched anything yet — declaring an augmentation is free. The work happens on *read*: rows a facade read returns come back hydrated — any of them still missing its augment columns runs the detail fetch first, and the result is written back to the cache as complete. Dio master S3 listing cache redb facade Vista dio.cached() get / window — you ask hydrated rows — gaps filled first per row with a gap: Key → prefix {rows, latest} merged into the cache, RecordChanged on the bus augmenter Vista — Readings lazy: contents → rows → latest — one download Not every read, though. The listing must stay what chapter 5 made it — instant — so `list_values` through the facade returns the cheap rows untouched. Hydration belongs to **bounded reads**: `get_value` for one record, `fetch_window` for a range. The rows you ask for are the rows that pay: ```rust,no_run # use std::time::Instant; # use vantage_aws::prelude::*; # use vantage_diorama::Dio; # async fn run(dio: Dio) -> vantage_core::Result<()> { // The listing stays instant — cheap rows, no downloads. let start = Instant::now(); let listing = dio.cached().list_values().await?; println!("{} files (listed in {:?})", listing.len(), start.elapsed()); // Details are paid for by the rows you ask for: a window of ten. let start = Instant::now(); let window = dio.cached().fetch_window(0, 10).await?; for (filename, file) in &window.rows { let size = file.get("Size").and_then(|v| v.as_str()).unwrap_or(""); let rows = file.get("rows").and_then(|v| v.as_i64()).unwrap_or(0); let latest = file.get("latest").and_then(|v| v.as_str()).unwrap_or(""); println!("{size:>10} {rows:>8} {latest:>10} {filename}"); } println!("{} files detailed in {:?}", window.rows.len(), start.elapsed()); # Ok(()) # } ``` `dio.cached()` is the Dio's facade: a Vista with the master's schema that reads through the cache. `fetch_window` returns a page, whose `rows` keep the window's order. Ten downloads is still a wait worth narrating, and the Dio announces it: one `DioEvent::Hydrating` with the pending count before the first fetch, then a `RecordChanged` per row as each lands. A dozen lines of event plumbing become the progress display: ```rust,no_run # use vantage_diorama::{Dio, DioEvent}; # fn run(dio: Dio) { let mut events = dio.subscribe_events(); tokio::spawn(async move { while let Ok(event) = events.recv().await { match event { DioEvent::Hydrating { pending } => println!("hydrating {pending} files…"), DioEvent::RecordChanged { id } => println!(" {id}"), _ => {} } } }); # } ``` First run — chapter 5's sync, the instant listing, then the window doing its downloads: ```text fetched 100 files in 215ms ... 1122 files (listed in 16ms) hydrating 10 files… csv/by_station/GM000001153.csv csv/by_station/GM000001474.csv ... 4796352 140629 19911231 csv/by_station/GM000001153.csv 7057976 206874 20260531 csv/by_station/GM000001474.csv 2978238 87207 19990131 csv/by_station/GM000002277.csv ... 10 files detailed in 20.2s ``` Real data, and readable at a glance: station `GM000001474` has 206,874 readings and is still reporting (May 2026); its neighbour `GM000001153` went silent at the end of 1991. Second run — new process, warm cache: ```text fetched 0 files in 340ms 1122 files (listed in 16ms) 4796352 140629 19911231 csv/by_station/GM000001153.csv ... 10 files detailed in 16ms ``` No `hydrating` line at all: those ten rows already carry their augment columns, so the window finds no gaps and never touches the network. Twenty seconds became sixteen milliseconds; the derived numbers live in `cache.redb` with the rest of the row. Ask for a different window — `fetch_window(500, 10)` — and only *its* gaps download. (`--invalidate` still clears everything, derived columns included — derived data is data.) ```admonish question title="What if a download fails?" Nothing is swallowed. Each failing row broadcasts `DioEvent::RecordLoadFailed { id, error }` on the bus — the same listener printing our progress can print failures beside it. The bounded read that requested the row then returns an error ("augment hydration failed") rather than handing you a window that quietly misses data. Retry by asking again: rows that *did* land are already cached and are never re-downloaded. Chapter 7's reactive views are more forgiving — a failed row is marked `RowStatus::LoadFailed` while its cheap listing columns stay visible, and the fetch is retried the next time the viewport reaches for it. ``` --- ## What we covered | Concept | What it does | | --------------------------------------------------- | ----------------------------------------------------------------- | | `with_lazy_expression(name, callback)` | A column computed in Rust on the returned record; chains in order | | `s3::get_object` | Raw object fetch through the same (unsigned) driver | | [`Augmentation`](vantage_diorama::Augmentation) | Declares detail source, row→detail mapping, fetch style, merge | | `Detail::Fixed` / `Detail::Resolved` | Detail Vista held directly, or looked up by name through a resolver | | `lens.dio(master).augment(..).build()` | Declares the augmentations when the Dio is built | | `dio.cached()` | The Dio's facade Vista, read through the cache | | `Source::Column { from, to }` | Master column → detail condition (our `Key` → `prefix`) | | `MergeRule` | Which detail columns land on the master row — and which don't | | Bounded facade reads hydrate | `get_value` / `fetch_window` fill gaps for the rows they return; `list_values` stays cheap | | `DioEvent::Hydrating` | Fired before a hydration sweep — the cue for progress UI | ```admonish tip title="What's next" Our CLI asks for one fixed window and exits. A real interface is a *standing* view: the window should follow the user's cursor, rows should repaint as details land, and the whole thing should stay current as the bucket changes. The next chapter opens **Sceneries** over this same Dio — a live terminal UI scrolling the full 122,000-station archive, hydrating the rows the user is looking at. ``` ```admonish info title="Going deeper" Augmentation has a reference chapter of its own — batched fetches, the gap rule, demand gating, and the YAML form — at [Augmentation](../augmentation.md). ``` --- Source: https://vantage-ui.com/book/intro/step7-scenery.html # Scenery — Reactive Views Chapter 6's CLI asks for one window of details and exits — the *asking* is still on the programmer. A real interface is a standing view: the user scrolls, and the window should follow; details land one by one, and the rows should repaint; the bucket changes, and the listing should notice. And it has to hold up at scale — the full GHCN archive is about **122,000** station files, of which a screen shows forty. What a UI actually needs is an ordered row set it can read *by index*, hydration that follows the user's attention, and a signal whenever anything on screen changes. **Scenery** is that layer. Each Scenery is a reactive view onto a Dio, exposing one access pattern: - [`TableScenery`](vantage_diorama::TableScenery) — ordered rows by index, with a viewport - [`RecordScenery`](vantage_diorama::RecordScenery) — a single record by id - [`ValueScenery`](vantage_diorama::ValueScenery) — a computed scalar (count, sum, custom) All three share one reactivity mechanism: a [`Generation`](vantage_diorama::Generation) counter that bumps whenever the view's state changes. Consumers `subscribe()` to a `watch` channel and redraw on each bump — the channel only ever holds the *latest* generation, so a burst of changes costs one repaint, not one per change. ```admonish info title="Why 'Scenery'?" Point a camera at a vista and you never capture the whole of it. The lens frames a scene: a limited cut, chosen by where you aim. But what it frames is *alive* — pan, and the scene follows; wait, and the light changes in front of you. A Scenery makes the same trade. It will never hold the entire Vista — one window of rows, one record, one number — yet within that frame everything flows: rows repaint as details land, the count ticks as data arrives, the window follows the user's attention. Limited, but dynamic — that is the whole design. ``` A Scenery is not another handle to your data — it hands you no records to keep. It maintains one particular *view* of the Dio's copy, precomputed and ready for a widget to read: | | `Dio` | `Scenery` | | ------------- | ------------------------------------------------ | ------------------------------------------------------ | | **Purpose** | Keep a live local copy of a data segment | Present that copy to a consumer, one access pattern at a time | | **Shape** | Master + cache + write queue + event bus | Ordered rows / one record / one scalar | | **Reads** | You ask — the facade answers from the cache | Precomputed — `row(idx)` / `value()` return instantly | | **Changes** | Announces them on the event bus | Reacts to the bus, recomputes, bumps its generation | | **Queries** | The master's, carried by the facade | Its own: sort, search, filter, viewport — served locally | | **How many** | One per data segment | Many per Dio — one per open, rows shared per query | | **Lifecycle** | Long-lived; owns cache and background tasks | Opened by a view, dropped with it | | **Consumer** | Loaders, handlers, scripts | UI widgets, bound through an *adapter* | That last cell is where this chapter ends up. The consumer — a terminal table here, a desktop grid or a remote client such as a React page elsewhere — drives its scenery in one repeating loop: 1. The consumer declares what it shows: `set_viewport(40..80, 40..80)` — nothing more than "these rows are on screen." 2. The scenery turns that into work for the Dio: list pages for the spine, detail fetches for the rows in view. 3. Results land in the Dio's cache, and the event bus announces each one — `RecordChanged`, row by row. 4. The scenery reacts to the bus: it updates the affected slot and bumps its `Generation`. 5. The consumer's watch channel ticks; it re-reads `row(idx)` for what's visible and repaints. The code that runs this loop for one particular consumer is the **adapter** — the same handful of lines whether it paints a terminal, a desktop toolkit, or a wire to the browser. Dio cache event bus fetches run through the Lens Scenery rows by index Generation Consumer adapter UI widget or remote client ① set_viewport ② hydrate rows in view ③ RecordChanged ④ Generation tick ⑤ row(idx) — repaint ## Back to the inventory Chapter 6 left the CLI listing one prefix and detailing ten fixed rows. What we actually want from the inventory is the tool you'd reach for daily — and that sets the requirements: - the **whole archive**, not a prefix — all 122,000 station files, scrollable end to end; - **instant open**, even on the first run — an empty table that fills as the listing arrives, never a frozen prompt; - **details where the user is looking** — the rows on screen sprout `ROWS` and `LATEST` as their files are read, and the cursor drags that attention with it; - **standing freshness** — the bucket changes, the table follows, plus a grand total of readings across the archive. Measure chapter 6's ending against that. It read from the Dio with `dio.cached().fetch_window(0, 10)` — and it works fine, for a CLI: the read is bounded to ten records, and it resolves once all ten have their details. Nothing blocks — it's an async wait, the runtime stays free — but the *user* waits all the same: twenty seconds of silence on a cold cache, and every widening of the window widens the wait. For a better experience we need a more responsive UI: 1. display results as soon as we have them — don't hold rows hostage to their details; 2. show data immediately and fix it reactively as better values arrive; 3. register what the user is currently looking at, reducing what is fetched and transmitted. The Scenery implements exactly this: - rows are served as soon as the list pass has them, with `…` standing in for details still pending; - every landed detail updates its own row and bumps the `Generation` — one tick, one repaint; - `set_viewport(loaded, shown)` registers the user's attention, and only those rows hydrate. For the consumer, this chapter reaches for **ratatui** — a terminal-UI library — which Vantage updates in real time through a ready-made adapter. The `dataset-ui-adapters` crate ships such adapters for several UI frameworks (egui, Slint, GPUI, Cursive, Tauri among them), and they all speak the same scenery loop. Here is ours a few seconds into a cold start: the first two visible rows already hydrated, and the total still waiting for the rest of the archive: ```text FILENAME SIZE ROWS LATEST csv/by_station/ACW00011604.csv 41673 1231 19490814 csv/by_station/ACW00011647.csv 481021 14355 20260710 csv/by_station/AE000041196.csv 1998959 … … … 8000 rows · 2 augmented · total rows 0 · ↑/↓ PgUp/PgDn scroll · r refresh · q quit ``` `learn-6` starts as a copy of `learn-5`'s data side — `files.rs`, `readings.rs`, and the whole Dio setup carry over. Three things change: the prefix widens to the entire archive, the Lens learns to serve a UI, and the printing loop is replaced by a scenery-bound terminal table. ```sh cargo add dataset-ui-adapters --features ratatui ``` Widening the scope is two touches. The prefix loses its country filter: ```rust const PREFIX: &str = "csv/by_station/"; ``` And the listing pages grow. Every page is one HTTP round-trip: chapter 5 fetched 100 keys at a time, a comfortable twelve requests for its 1,122 files — but at 122,000 files the same setting means 1,220 round-trips of mostly latency. S3 caps a single response at 1,000 keys, so `files.rs` asks for the cap and the whole archive syncs in about 122 requests: ```rust # use vantage_aws::prelude::*; # use vantage_types::EmptyEntity; # fn table(aws: AwsAccount) -> Table { # Table::new("restxml/Contents:s3/GET /{Bucket}?list-type=2", aws) # .with_id_column("Key") // S3's per-response maximum — we're listing the whole station set. .with_condition(eq("max-keys", 1000)) # } ``` --- ## A Lens that serves a UI Chapter 5's Lens had one job: warm the cache before anyone reads. A UI inverts the priorities — the screen must appear *immediately*, on whatever is known so far, and data streams in behind it. Two changes to the builder: ```rust,no_run # use std::time::Duration; # use vantage_aws::prelude::*; # use vantage_diorama::{Cache, Lens, Loader, LoaderCx, Refresher}; # async fn sync(_cx: &LoaderCx) -> vantage_core::Result<()> { # Ok(()) # } # fn run() -> vantage_core::Result<()> { let lens = Lens::builder() .cache(Cache::redb("cache.redb")) // Don't wait for the sync: it warms the cache in the background, // while the view lists the bucket itself. .loader(Loader::eager(|cx: LoaderCx| async move { sync(&cx).await }).blocking(false)) // Reconcile against the bucket once a minute: new files appear // as un-hydrated rows, vanished ones drop out, changed ones are // demoted for re-hydration. The augmentation's refresh pass does // the work; the refresher only sets the interval. .refresher(Refresher::new(|_cx: LoaderCx| async { Ok(()) }).every(Duration::from_secs(60))) .build() .context("Failed to build lens")?; # let _ = lens; # Ok(()) # } ``` **`.blocking(false)`** detaches the sync. An eager start blocks the Dio's `build()` by default, which is what chapter 5 wanted: nobody read before the cache was warm. Here `build()` returns at once and the sync runs as a background task, so the UI's first frame never waits on 122 requests. The sync itself is chapter 5's, minus its `println!` narration (a terminal UI owns the screen now). It writes the cache directly, and writes to the raw cache raise no events, so the Dio does the announcing: when the detached start finishes it sends `DioEvent::Seeded`, and every open view re-reads. That covers the cache. What a `TableScenery` does with it needs a little more context. An augmented Dio drives **two-pass loading**: a *list pass* fetches cheap rows and establishes the view's order (its spine), and a *detail pass* hydrates individual rows' augment columns as the viewport reaches them. The detail pass is chapter 6's augmentation. The list pass picks its source when a view first lists: - while the cache isn't known to hold the whole set (a first run, or a sync still running), it asks the master, page by page; - once the Dio holds a **whole copy**, which the eager start records when it finishes, it reads the cache instead: one local read, zero network. So on a cold start the table lists the bucket itself while the sync runs, and on `Seeded` it re-lists from the warm cache. Nothing in the application chooses between the two; the Dio knows which answer it can trust. **The refresher** completes the freshness story. Once a minute the Dio reconciles its cache against the master (this *does* walk the listing: 122 requests, in the background). New files appear as un-hydrated rows, deleted ones drop out, and a file whose listing entry changed keeps showing its old numbers but is marked stale; the next time it's on screen, the detail pass re-fetches it. Chapter 5 needed `--invalidate` to notice deletions; a standing view just notices. The refresher's own callback does nothing here: on an augmented Dio the reconcile is the augmentation's refresh pass, and `Refresher::new(..).every(..)` only sets how often it runs. ### Why not always list from the cache? Because an empty cache and an empty bucket look the same. A view that read the cache during a first run would show zero files and call the listing complete, then jump to 122,000 rows minutes later. A half-written cache is worse: it reads as a complete listing of whichever files the sync reached first. The whole-copy record is how the Dio tells "the cache is the listing" from "the cache is what we have so far", and until it can say the first, the master answers. The cost is the master walk on a cold start, and that walk is the same one the sync is making anyway. ## Creating the reactive UI Our UI needs two things from the Dio: the rows, and a grand total. Each one is a scenery. Let's open them one at a time. ### The table ```rust,no_run # use vantage_diorama::Dio; # async fn run(dio: Dio) -> vantage_core::Result<()> { // One list page covers the whole cached listing (~122k station files); // the viewport's detail pass hydrates whatever is on screen first. let scenery = dio.table_scenery().page_size(200_000).open().await?; # let _ = scenery; # Ok(()) # } ``` `.open()` builds the scenery's rows (or joins the rows another open of the same query already built), spawns its reactor (the task that watches the Dio's event bus), and hands back `Arc`. What you get is deliberately small and synchronous: - `row_count()` and `row(idx)`, an [`EnrichedRecord`](vantage_diorama::EnrichedRecord): the record plus a per-row status; - `set_viewport(loaded, shown)`, more on this in a moment; - `subscribe()`, the generation channel from the start of the chapter; - `refresh_source()`, which asks the Dio to re-read its master; the view reloads as the fresh rows land. The builder can also chain `.order(column, SortDirection::Ascending)`, `.search(text)` and `.filter(condition)`, every one of them served locally on a backend that can do none of it. We don't use them here; the table shows the archive as-is. ```rust,no_run # use vantage_diorama::Dio; # use vantage_vista::SortDirection; # async fn run(dio: Dio) -> vantage_core::Result<()> { // The same table, from the end of the alphabet. let reversed = dio .table_scenery() .order("Key", SortDirection::Descending) .open() .await?; # let _ = reversed; # Ok(()) # } ``` A UI rarely commits to one order at open time, though, and it doesn't have to reopen. The handle mutates in place: `set_order(Some((column, direction)))` and `set_search(Some(text))`, bound to a key or a header click, re-point the view at the ordered index for the new query, swap the visible rows in one atomic step (the grid never blanks mid-reorder), and restart hydration for whatever is on screen. Sorting back reuses the already-built index: zero list calls. Runtime filters change the same way, through `set_filters(conditions)`, which takes a `Vec` and replaces the previous set (an empty vec clears it). One thing probably caught your eye: `page_size(200_000)`?! That's the *list pass*: how many spine rows one list page asks for. It's a separate axis from the viewport, which drives only the *detail pass* over rows the spine already holds. Left at the default, the spine would stop at the first hundred files and nothing in this UI asks for page two. Set past the archive size, one list page builds the entire spine. Once the Dio holds its whole copy, that page is a single read of the local cache. ### The viewport The **viewport** is the scenery's answer to "which rows is the user actually looking at?" — a plain range of row indexes, like `40..80`. It's declared through `set_viewport(loaded, shown)`, the method on the handle we just opened. `loaded` is the range to load and hydrate; `shown` is the range actually on screen, which feeds the view's "showing a–b" statistics and nothing else. A grid usually passes the same range twice. You rarely call it yourself: whoever renders the table calls it as the user scrolls. Later in this chapter, that's the ratatui adapter's job. It is the load-bearing call, because **the viewport drives hydration**. The rows a consumer declares as its viewport are the rows the detail pass works on. Framework code calls this the **demand gate**: no row is detailed unless some live view demands it. Everything else stays a cheap list row — and rows never observed never cost a download. Recognize it? This is chapter 6's bounded read with the asking automated: there the programmer chose the window; here the viewport is the window, and it moves with the user. ### The grand total ```rust,no_run # use vantage_diorama::Dio; # use vantage_diorama::aggregate::Sum; # async fn run(dio: Dio) -> vantage_core::Result<()> { // Grand total of the ROWS column. `rows` is augmented, so each // computation first hydrates every cached file: the total appears once // they have all landed, then follows the bucket's changes. let totals = dio.value_scenery(Sum::of("rows")).open().await?; # let _ = totals; # Ok(()) # } ``` A value scenery runs one aggregation (here [`Sum`](vantage_diorama::aggregate::Sum)) over the rows the Dio admits from its cache. Nothing is asked of the master. It recomputes when the Dio announces a change, and bumps its generation only when the value actually moved. `open()` computes the first value before it returns. Hold on: how can it sum `rows` when most files aren't augmented yet? It doesn't sum the gaps. The aggregation declares which columns it reads, and when one of them is an augmented column, the Dio hydrates the rows missing it before the aggregation sees them. A sum that skipped un-hydrated rows would show a number that looks like a total and isn't one. So the value is always a total over every cached file, and on a cold cache that means it waits for every file to download. That's an expensive number on 122,000 files, and it's meant to be: a total of readings genuinely needs every file read once. The cost is paid once, because the hydrated columns stay in `cache.redb`; a warm restart totals from the cache. When you want a cheap number instead, aggregate something the listing already has. [`Count::rows()`](vantage_diorama::aggregate::Count) reads no column at all, so it counts the cached rows without hydrating anything. (There is a third kind, **RecordScenery**: `dio.record_scenery(id).open()`, one record under the same contract: read it, subscribe, redraw on bump. A detail pane beside the table would use one; our table doesn't need it.) ### Open freely, drop when done Sceneries are inexpensive. A page opens as many as it needs (a grid, a total, a detail pane), and every `open()` returns a view of its own: its own viewport, its own order, search and filters. Views of the same query still read one set of rows underneath, so two grids over one query make one request and hydrate each file once. The other half of that contract: release them when the page closes. Dropping the handle stops the view's tasks and withdraws its demand, so a closed page stops pulling data. ### Why does each open get its own view? Because sharing the view meant sharing its setters. When identical opens returned one shared instance, a quick-search typed into one grid reshaped every other grid of the same query: `set_search` on one widget filtered its neighbour's rows too, and two remote clients asking the same question couldn't scroll to different pages. Sharing was right for the rows and wrong for everything the user controls. So the split runs along that line. The rows (the per-query index, the cached records, the hydration already paid for) are shared between every view of one query. The view (viewport, shown range, order, search, filters) belongs to whoever opened it. A setter moves only its own view to the rows of its new query, joining another view's rows when one already answers it. Chapter 8 leans on this: every HTTP connection opens its own view, and the downloads are still paid once. ## Binding to a terminal What remains is rendering — and none of it is specific to this app. Scrolling a virtualized table, forwarding the visible range as the viewport, redrawing on generation bumps: that's a reusable binding, and `dataset-ui-adapters` ships it for [ratatui](https://ratatui.rs) in its `ratatui_dio` module. The entire UI: ```rust,no_run # use std::sync::Arc; # use dataset_ui_adapters::ratatui_dio; # use vantage_aws::prelude::*; # use vantage_diorama::{TableScenery, ValueScenery}; # async fn ui( # scenery: Arc, # totals: Arc, # ) -> vantage_core::Result<()> { ratatui_dio::SceneryTable::new(scenery) .with_column("FILENAME", "Key", 0) .with_column("SIZE", "Size", 10) .with_column("ROWS", "rows", 8) .with_column("LATEST", "latest", 10) .with_status_value("total rows", totals) .run() .await .context("terminal UI failed") # } ``` Line by line: - **`ratatui_dio::SceneryTable::new(scenery)`** — hands the scenery to the adapter. The full module path marks the boundary: everything before this line was framework, everything after is the ratatui binding. - **`.with_column(header, field, width)`** — one table column: the header text, the record field it reads, and a width in characters (`0` = flexible fill). A field the row doesn't have yet renders as `…` — which is exactly how un-hydrated rows look. - **`.with_status_value(label, scenery)`** — pins a `ValueScenery` into the status bar; it repaints whenever that scenery's generation bumps, so `total rows` follows the Dio. - **`.run()`** — takes over the terminal until `q`. It draws only the visible rows (at 122,000 you don't build widgets for the rest), keeps the scenery's viewport on a **ten-row band around the cursor** — details load for the record the user is on and its neighbours, not the whole screen. Band-not-screen is the *adapter's* policy, not the scenery's: each detail fetch here is a multi-second download, and hydrating all forty visible rows would waste most of that work every time the user scrolls on. It listens to every subscribed generation for repaints, and renders the status bar: row count, an **augmented** counter (how many rows on hand are fully hydrated), your pinned values, and the key legend. `r` calls the scenery's `refresh_source()`: the Dio's reconcile on demand, ahead of the timer. ## Running it `cargo run` on a cold cache is the whole system visible at once. The table appears instantly, empty, while the view lists the bucket and the background sync copies it; when the sync lands, the view re-lists from the cache and the row counter passes 122,000. Meanwhile the rows around the cursor sprout numbers, one file at a time, as the detail pass works through the band (`…` becoming `14355 20260710`), and the `augmented` counter ticks upward with each one. Move the cursor, and the band follows; jump to `End`, and the last stations of the alphabet get their turn. `total rows` stays at its first value, zero over an empty cache, until the total has every file it needs. Quit, run again: everything already observed is back instantly, warm from `cache.redb`, and hydration resumes wherever you look next. Notice what the application never wrote: a render loop, a fetch, an event match. It reads `scenery.row(idx)` and `totals.value()` through a binding that repaints when a generation channel says so. Nothing more passes between data and display — and it's precisely how a real UI binds to Vantage. --- ## What we covered | Concept | What it does | | ------------------------------------------------- | --------------------------------------------------------------------- | | [`TableScenery`](vantage_diorama::TableScenery) | Ordered rows by index; sort/search/filter served locally | | [`ValueScenery`](vantage_diorama::ValueScenery) | One aggregation over the Dio's cached rows; hydrates augmented columns it reads | | [`RecordScenery`](vantage_diorama::RecordScenery) | One record by id, same subscribe/redraw contract | | [`Generation`](vantage_diorama::Generation) / `subscribe()` | Latest-value watch channel — one bump, one repaint | | One view per open | Own viewport, order, search and filters; rows shared per query | | Two-pass loading | List pass builds the spine; detail pass hydrates rows in view | | Whole copy | Once the cache holds the whole set, the list pass reads it, not the master | | `set_viewport(loaded, shown)` | Declares what loads, and thereby what hydrates; `shown` is what's on screen | | `set_order` / `set_search` / `set_filters` | Reshape one view in place, without reopening | | `Loader::eager(..).blocking(false)` | UI first: `build()` returns immediately, the sync runs behind it | | `Refresher::new(..).every(..)` | Standing freshness: reconcile on a timer; `refresh_source()` on demand | | `ratatui_dio::SceneryTable` | The ratatui binding: virtualized rows, viewport, status bar, keys | ```admonish tip title="What's next" One terminal, one viewport. A web server is the same picture multiplied: every connected browser is its own standing view, each on a different page, all expecting details to stream in — and none of them should ever download a file another view already paid for. The final chapter puts this Dio behind chapter 3's Axum server: kubernetes-style GET + watch endpoints, a React frontend, and a scheduler that serves every concurrent viewer fairly. ``` --- Source: https://vantage-ui.com/book/intro/step8-axum-dio.html # Serving Scenery — Axum & Watch Streams Chapter 7's client of the Dio was a terminal driving two sceneries — one showing the rows, the other aggregating a running sum. Sceneries scale well beyond that: the framework can drive many active viewers at once, and nothing requires them to live in the same process. This chapter sends them across the network — every connected browser tab becomes its own viewer, looking at its own page of the archive, with details streaming in as they land. The traditional plumbing for live updates over HTTP is a [WebSocket](https://en.wikipedia.org/wiki/WebSocket), [server-sent events](https://en.wikipedia.org/wiki/Server-sent_events), or the style the Kubernetes API made standard — the **watch**: a plain HTTP response that never ends, delivering one JSON line per change over [chunked transfer encoding](https://en.wikipedia.org/wiki/Chunked_transfer_encoding). We take the Kubernetes shape: every endpoint answers a plain `GET` with a snapshot, and the same URL with `?watch=true` keeps the connection open and streams changes for as long as the client stays. A watch is not a polling loop — it is a Scenery on the far end of an HTTP connection. This chapter re-uses chapter 3's Axum server, wired to the Dio through a new adapter crate — and finishes with a small React app browsing the bucket the way the AWS S3 Explorer does. Worth pausing on what the frontend gets for free: the page renders instantly from the cache, cells fill themselves in as augmentation lands, and every open tab stays current — the responsiveness chapter 7 built for the terminal, now delivered over a wire. ## One flight per row Before any of that can be safe, the framework needs an answer to a question chapter 7 never had to ask: what happens when **several** views drive augmentation at once? With one terminal and one viewport, it never mattered who ran the detail fetches. If each view ran its own, inline, for its own viewport, two browser tabs watching *overlapping* pages would each download the same CSVs; two tabs on *disjoint* pages would race each other with no ordering at all; and a tab opened mid-download would have no way to say "me next". So the fetches are the Dio's job, not the view's. Every consumer that wants rows hydrated — a scenery's viewport, a facade read blocking on its window — registers a queue with the Dio's **augment scheduler** and enqueues row ids into it. A worker pool drains the queues: - **Round-robin across consumers.** The worker takes one id from each queue in turn, so two views with disjoint pages interleave — neither starves behind the other's backlog. - **One flight per row.** An id already being fetched is never fetched again; every queue waiting on it is notified by the same completion. Overlapping views cost one download, not one per view. - **Closing a view withdraws its work.** The queue registration is owned by the view; when its handle drops, its queued ids vanish. A fetch already in the air completes and lands in the cache — paid-for work is kept. - **Workers are configurable.** The default single worker keeps fetch order deterministic; `Lens::builder().tuning(Tuning::default().with_augment_workers(4))` hydrates four rows at a time when the detail source can take it. Nothing in the example code asks for this: the scheduler sits under the same `set_viewport` and `fetch_window` calls the previous chapters used. What it buys is what you can safely do here: open as many sceneries as you have HTTP connections. Every open is its own view (chapter 7, "Why does each open get its own view?"). Two identical opens share one set of rows, which is right for two widgets showing the same grid, and each still keeps its own viewport, which is right for two *clients* watching different pages of the same query. Every view counts in the **demand union** — the Dio hydrates the union of the columns every open scenery demands — and is released on drop. ## The adapter: `DioRouter` Everything HTTP lives in a new adapter crate, `vantage-api-adapters` — the server-side sibling of the `dataset-ui-adapters` crate that provided chapter 7's ratatui binding. Where a UI adapter binds a scenery to a widget, an API adapter binds it to a route: ```rust,no_run use vantage_api_adapters::axum_dio::DioRouter; # use vantage_diorama::Dio; # fn api(dio: Dio) -> axum::Router { let api = DioRouter::new(dio.clone()) .with_column("filename", "Key") .with_column("size", "Size") .with_column("rows", "rows") .with_column("latest", "latest") .with_page_size(50) .into_router(); # api # } ``` Each `.with_column(name, field)` maps a record field to a JSON key — and doubles as the watch sceneries' *demand*: naming `rows` and `latest` here is exactly what makes a watch connection drive their hydration (chapter 7's demand gate, now per connection). `into_router()` yields a plain `axum::Router` with two routes, each in two modes: | Request | Answered by | Cost | | ------------------------------ | ----------------------------------------- | ----------------------------- | | `GET /?offset=&limit=` | A window over the cache | Instant; never fetches | | `GET /?watch=true&…` | A `TableScenery` view of its own | Streams while connected | | `GET /{id}` | A bounded facade read (`get_value`) | Hydrates that one row; cached | | `GET /{id}?watch=true` | A `RecordScenery` | Streams that record's changes | The split embodies the demand philosophy: a plain `GET` is not a standing view, so it serves the Dio's *current knowledge* instantly — augmented columns appear once some view has paid for them. A **watch is the standing view**: it declares its page as the viewport, hydration follows, and every change streams back as a Kubernetes-style NDJSON line: ```text {"type":"ADDED","object":{"index":3,"filename":"…","size":"4880965","rows":null,"latest":null}} {"type":"MODIFIED","object":{"index":3,"filename":"…","size":"4880965","rows":143676,"latest":"20260531"}} ``` The stream diffs against what it already sent — a generation bump that changed nothing on this page costs nothing on the wire. And the scenery is *owned by the response stream*: when the client disconnects, the stream drops, the scenery's guard aborts its tasks, its queued fetches are withdrawn, and its demand drains. A closed tab stops pulling — the same lifecycle rule as chapter 7's closing grid, now enforced by HTTP. One honesty note against the Kubernetes original: there is no resume token (no `resourceVersion`). A client that reconnects gets a fresh snapshot of `ADDED` lines and a new watch — not a replay of what it missed while away. ## The server `learn-7` is learn-6 with the terminal swapped for the router — and the prefix narrowed back to chapter 5's `GM` (1,122 files): a server should boot in seconds, and everything below works the same way on the full archive. The data plumbing is otherwise identical — `files.rs`, the augmenter, the Lens — with two differences. First, the cache is opened by hand, because we want a second table in the same redb file: ```rust,no_run # use std::sync::Arc; # use vantage_aws::prelude::*; # use vantage_diorama::{CacheBackend, CacheTable, RedbCache}; # struct ContentsCache; # impl ContentsCache { # fn new(_table: Arc) -> Arc { # Arc::new(Self) # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let cache = Arc::new(RedbCache::open("cache.redb").context("Failed to open cache")?); let contents = ContentsCache::new(cache.open_table("contents").await?); # let _ = contents; # Ok(()) # } ``` A Lens normally opens `cache.redb` itself (`Cache::redb(path)`); `Cache::backend(cache)` hands it ours instead. The Dio claims one named table for the listing, and `open_table("contents")` claims another for the cache we're about to meet. Second, the eager start stays **blocking**. That's `Loader::eager`'s default; learn-6 added `.blocking(false)` to open its UI on an empty table. A server should answer its first request from a warm cache, so the Dio's `build()` runs the chapter-5 sync to completion before `axum::serve` ever binds the port: ```rust,no_run # use std::sync::Arc; # use std::time::Duration; # use vantage_aws::prelude::*; # use vantage_diorama::{Cache, Lens, Loader, LoaderCx, RedbCache, Refresher}; # async fn sync(_cx: &LoaderCx) -> vantage_core::Result<()> { # Ok(()) # } # fn run(cache: Arc) -> vantage_core::Result<()> { let lens = Lens::builder() .cache(Cache::backend(cache)) // Pre-fetch: the eager start blocks the Dio's build (the default), // so the server starts answering only once the listing is cached. .loader(Loader::eager(|cx: LoaderCx| async move { sync(&cx).await })) .refresher(Refresher::new(|_cx: LoaderCx| async { Ok(()) }).every(Duration::from_secs(60))) .build() .context("Failed to build lens")?; # let _ = lens; # Ok(()) # } ``` A cold start lists the 1,122 files in two requests before the port opens; a restart resumes from redb and confirms with a single request: ```text fetched 1000 files in 831.454208ms fetched 122 files in 194.010875ms serving on http://localhost:3007 ``` The rest of `main.rs` is mounting: ```rust,no_run # use tower_http::services::ServeDir; # use vantage_aws::prelude::*; # async fn serve(api: axum::Router) -> vantage_core::Result<()> { let app = axum::Router::new() .nest("/api/files", api) .fallback_service(ServeDir::new("frontend/dist")); let listener = tokio::net::TcpListener::bind("0.0.0.0:3007") .await .context("Failed to bind :3007")?; println!("serving on http://localhost:3007"); axum::serve(listener, app).await.context("server failed") # } ``` Concurrency needs no further code. `Dio` is a cheap clone over shared state, redb reads run concurrently, and no lock is held across a network await anywhere in the read path — every simultaneous request proceeds on its own. Neither does authentication — because there is none. The endpoints are anonymous, as in chapter 3: wrap the router in your tower auth middleware before exposing it; Vantage deliberately stays out of authn. ## A cache that earns its keep The detail endpoint returns everything about one file, augment columns included — which means the first `GET /{id}` downloads the CSV. Chapter 6 deliberately kept `contents` out of the cache: at 122,000 stations, storing every casually-viewed file would grow `cache.redb` by gigabytes. But a server sees *repeat* traffic — the same popular stations, again and again — and re-downloading those is just as wasteful. `ContentsCache` splits the difference with **lazy admission**: a file must be requested twice before its contents earn a slot. It is application code, not framework — the framework contributes `open_table` (a named key-value table in the Dio's own redb file); the admission policy is entirely ours to write. ```rust # use std::collections::HashSet; # use std::sync::{Arc, Mutex}; # use vantage_diorama::CacheTable; pub struct ContentsCache { table: Arc, /// Keys downloaded at least once — the admission ledger. seen: Mutex>, } ``` Its one operation is cache-first with the admission rule on the miss path: ```rust # use std::collections::HashSet; # use std::future::Future; # use std::sync::{Arc, Mutex}; # use vantage_aws::prelude::{CborValueExt, VantageResult}; # use vantage_diorama::CacheTable; # use vantage_vista::prelude::*; # pub struct ContentsCache { # table: Arc, # seen: Mutex>, # } # impl ContentsCache { pub async fn get_or_fetch(&self, key: &str, fetch: F) -> VantageResult where F: FnOnce() -> Fut, Fut: Future>, { if let Some(row) = self.table.get_value(key).await? && let Some(body) = row.get("contents").and_then(|v| v.as_str()) { return Ok(body.to_string()); } let body = fetch().await?; let repeat = !self.seen.lock().unwrap().insert(key.to_string()); if repeat { let record = [("contents".to_string(), CborValue::from(body.clone()))] .into_iter() .collect(); self.table.insert_value(key, &record).await?; } Ok(body) } # } ``` A hit is served from redb. A miss runs `fetch` either way, but only a key already in `seen` (a repeat request) gets its body written to the `contents` table. One-off requests leave nothing behind but their key. (The ledger itself is never trimmed — bare keys, fine at 122,000 files; a bigger keyspace would want an eviction rule, and the policy being application code makes that your call.) The augmenter's `contents` lazy expression routes its download through it, and everything downstream — `rows`, `latest`, the detail endpoint — is unchanged: ```rust # use std::future::Future; # use std::sync::Arc; # use vantage_aws::prelude::*; # use vantage_types::EmptyEntity; # pub struct ContentsCache; # impl ContentsCache { # pub async fn get_or_fetch(&self, _key: &str, fetch: F) -> VantageResult # where # F: FnOnce() -> Fut, # Fut: Future>, # { # fetch().await # } # } # fn table(aws: AwsAccount, bucket: String, contents: Arc) -> Table { # Table::new("restxml/Contents:s3/GET /{Bucket}?list-type=2", aws.clone()) # .with_id_column("Key") .with_lazy_expression("contents", move |row| { let aws = aws.clone(); let bucket = bucket.clone(); let contents = contents.clone(); let key = row .get("Key") .and_then(|v| v.as_str()) .unwrap_or_default() .to_string(); async move { let fetch_key = key.clone(); let body = contents .get_or_fetch(&key, || async move { s3::get_object(&aws, &bucket, &fetch_key).await }) .await?; Ok(body.into()) } }) # } ``` ## Watching it work A plain `GET` is the cache, instantly — `rows` and `latest` are `null` because nothing has demanded them yet: ```text $ curl 'localhost:3007/api/files?offset=0&limit=3' {"total":1122,"offset":0,"limit":3,"items":[ {"index":0,"filename":"csv/by_station/GM000001153.csv","size":"4796352","rows":null,"latest":null}, … ``` A detail `GET` blocks on the bounded facade read — 1.8 s to download and digest 4.8 MB — and the same request again is a 21 ms cache hit: ```text $ curl 'localhost:3007/api/files/csv%2Fby_station%2FGM000001153.csv' {"Key":"csv/by_station/GM000001153.csv", …"Size":"4796352", …"rows":140629,"latest":"19911231"} ``` Now the standing view. `curl -N` holds the connection; the page arrives as `ADDED` rows, then each CSV lands as a `MODIFIED` line the moment its download completes: ```text $ curl -N 'localhost:3007/api/files?offset=3&limit=3&watch=true' {"type":"ADDED","object":{"index":3,"filename":"csv/by_station/GM000002288.csv","size":"4880965","rows":null,"latest":null}} {"type":"ADDED","object":{"index":4,"filename":"csv/by_station/GM000002698.csv","size":"6351302","rows":null,"latest":null}} {"type":"ADDED","object":{"index":5,"filename":"csv/by_station/GM000002716.csv","size":"5042384","rows":null,"latest":null}} {"type":"MODIFIED","object":{"index":3,…"rows":143676,"latest":"20260531"}} {"type":"MODIFIED","object":{"index":4,…"rows":186088,"latest":"20081031"}} {"type":"MODIFIED","object":{"index":5,…"rows":147011,"latest":"20250824"}} ``` And the scheduler is visible from outside. Two watches on **disjoint** pages, side by side: their `MODIFIED` lines alternate — B, A, B, A — the single worker taking one row from each view's queue in turn: ```text 19:35:35 [B] {"type":"MODIFIED","object":{"index":42,…"rows":201382,…}} 19:35:36 [A] {"type":"MODIFIED","object":{"index":40,…"rows":77595,…}} 19:35:38 [B] {"type":"MODIFIED","object":{"index":43,…"rows":132579,…}} 19:35:40 [A] {"type":"MODIFIED","object":{"index":41,…"rows":128335,…}} ``` Two watches on the **same** page: each row's `MODIFIED` reaches both connections in the same second — one download, fanned out by the event bus to every scenery holding the row: ```text 19:36:18 [B] {"type":"MODIFIED","object":{"index":50,…"rows":135802,…}} 19:36:18 [A] {"type":"MODIFIED","object":{"index":50,…"rows":135802,…}} ``` Run the first `GET` again afterwards: the page the watches paid for now answers with its numbers filled in. Current knowledge grew. ## A tiny React client The frontend is one component. It fetches its page with `watch=true` and reads the NDJSON stream directly off `fetch` — no client library, twenty lines: ```jsx const res = await fetch(`/api/files?offset=${offset}&limit=${LIMIT}&watch=true`, { signal: ctl.signal, }) const reader = res.body.getReader() // …accumulate chunks, split on '\n'… const event = JSON.parse(line) setRows(rs => ({ ...rs, [event.object.index]: event.object })) ``` `ADDED` and `MODIFIED` merge the same way — the object lands at its index — so the table renders the listing immediately and the `Rows` / `Latest` cells flip from `…` to numbers as the watch delivers them, exactly like chapter 7's terminal cells did. Paging aborts the fetch via the `AbortController`, which closes the connection, which drops the scenery server-side. Clicking a row hits the detail endpoint into a side panel. `npm run build` in `learn-7/frontend` emits `dist/`, the `ServeDir` fallback serves it, and `http://localhost:3007` is a bucket explorer: title, file count, Prev/Next, and a table filling itself in. --- ## What we covered | Concept | What it does | | ---------------------------------------- | ----------------------------------------------------------------------- | | Augment scheduler | Dio-owned detail fetches: round-robin across views, one flight per row | | `Tuning::with_augment_workers(n)` | Worker pool size — 1 (deterministic order) by default | | One view per open | Shared rows, own viewport — one standing view per HTTP connection | | `DioRouter` (vantage-api-adapters) | `.with_column()` + `.with_page_size()` → an axum router; columns double as demand | | `GET` vs `?watch=true` | Snapshot of current knowledge vs a Scenery streaming NDJSON events | | `ADDED` / `MODIFIED` lines | Kubernetes-style watch events, diffed per row per connection | | Connection drop = scenery drop | Queued fetches withdrawn, demand drained — a closed tab stops pulling | | Blocking `Loader::eager` | Pre-fetch: the port opens only once the listing is cached | | `Cache::backend` + `open_table` | One redb file, many named tables — the Dio's and our own | | `ContentsCache` | Lazy admission: downloaded once — remembered; twice — cached | ```admonish success title="The whole climb" Eight chapters ago this book started with one SQL query; it ends with a fair-scheduled, cache-backed watch API feeding a React frontend — and every layer still speaks through the one below it. From here, the reference half of the book takes over — [Augmentation](../augmentation.md) for batched fetches and demand gating, [Config-Driven Vistas](../config-driven-vistas.md) for defining all of this from YAML, [Model-Driven Architecture](../mda.md) for structuring a real application, and [Adding a New Persistence](../new-persistence.md) when you're ready to extend the framework itself. ``` --- Source: https://vantage-ui.com/book/intro/step5-sql-dio.html # Dio & Lens over SQL This path builds a live, cached, watchable view in front of a SQL database you own. We start from a Vista over a SQLite `product` table: a runtime handle that can already sort, search, paginate and write. That is a capable backend, which raises a fair question. If the database already does all that, what is a caching-and-events layer for? For the two things a capable backend doesn't hand you on its own. A [Dio](vantage_diorama::Dio) keeps a **live local copy** of a data segment, so reads are answered from memory instead of round-trips, and it **announces every change** on an event bus, so a view can watch the data instead of re-polling it. A slow, read-only API needs a third thing from Diorama: the *injection* of capabilities the backend lacks, like sorting a listing it cannot sort. A SQL database needs none of that. Here the live cache and the change stream are the whole story. ```admonish info title="This path assumes the Introduction" We build on two ideas from the [Introduction](./introduction.md): a [**Table**](./step2-tables.md), your typed model, and a [**Vista**](./step4-vista.md), the runtime handle a datasource's factory wraps that model in. If either is new, read the Introduction first; this path picks up where it ends. ``` We'll build a bar's shelf: a handful of products, each with a `stock` count, behind a cached, watchable API. A background "till" sells items so the data moves. This chapter builds the cache; the next serves it as a live watch; the one after moves the whole thing to PostgreSQL without touching the model. The full crate is [`learn-8`](https://github.com/romaninsh/vantage/tree/main/learn-8). ## The model The Introduction's `Product`, with one column added: `stock`, the units on the shelf. ```rust // src/product.rs use vantage_sql::prelude::*; use vantage_types::prelude::*; #[entity(SqliteType)] #[derive(Debug, Clone, Default)] pub struct Product { pub name: String, pub price: i64, pub stock: i64, } impl Product { pub fn table(db: SqliteDB) -> Table { Table::new("product", db) .with_id_column("id") .with_column_of::("name") .with_column_of::("price") .with_column_of::("stock") } } ``` Nothing here knows it is about to be cached, watched, or served from Postgres. The entity is written once, and the layers above it decide what to do with it. ## The master The master is the `product` table as a Vista, built the way the [Introduction](./step4-vista.md) built one: hand the table to its datasource's Vista factory. ```rust,no_run # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # pub struct Product { # pub name: String, # pub price: i64, # pub stock: i64, # } # # impl Product { # pub fn table(db: SqliteDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("stock") # } # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { const DB_PATH: &str = concat!(env!("CARGO_MANIFEST_DIR"), "/products.db"); let url = format!("sqlite:{DB_PATH}?mode=rwc"); let db = SqliteDB::connect(&url).await.context("connect sqlite")?; let master = db.vista_factory().from_table(Product::table(db.clone()))?; # Ok(()) # } ``` The database is a file beside the crate, and `mode=rwc` creates it on first run. A file rather than `sqlite::memory:` is deliberate: the next chapter edits it from another terminal with the `sqlite3` CLI and watches the change arrive. Ask this Vista what it can do and its [capabilities](vantage_vista::Capability) cover everything a listing needs: `Order`, `Search`, `FilterOps` and `Window`, plus `Insert`, `Update` and `Delete`. Contrast that with the S3 path, where the Vista admits it cannot sort or search. There is no gap to fill here, so this path injects no capabilities. What we still want is the cache. ## The lens: eager and reactive The S3 path's loader walks the bucket a page at a time, resuming from a cursor, and writes each page into a redb file. A SQL table needs none of that: it hands back the whole set in one call. So this path uses an eager loader that lists the table once, and a refresher that lists it again on a timer. ```rust,no_run # use std::time::Duration; # # use vantage_diorama::prelude::*; # use vantage_diorama::{Cache, Loader, LoaderCx, Refresher}; # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # pub struct Product { # pub name: String, # pub price: i64, # pub stock: i64, # } # # impl Product { # pub fn table(db: SqliteDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("stock") # } # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:products.db?mode=rwc").await.context("connect sqlite")?; # let master = db.vista_factory().from_table(Product::table(db.clone()))?; let lens = Lens::builder() .cache(Cache::memory()) .loader(Loader::eager(reload)) .refresher(Refresher::new(reload).every(Duration::from_secs(1))) .build() .context("build lens")?; let dio = lens.dio(master).build().await?; # Ok(()) # } /// Make the cache hold exactly the master's rows. async fn reload(cx: LoaderCx) -> VantageResult<()> { let rows = cx.master().list_values().await?; cx.reconcile(rows).await } ``` Three settings carry the behaviour: - **[`Cache::memory()`](vantage_diorama::Cache::memory)**: the cache is ephemeral, and a restart re-reads the table. The S3 path persists its cache with `Cache::redb("cache.redb")` because re-listing a remote bucket is expensive. Re-reading a local table is not. - **[`Loader::eager(reload)`](vantage_diorama::Loader::eager)**: before the server answers a request, `list_values()` pulls the whole table into the cache. `lens.dio(master).build()` waits for it by default, so the first read is already warm. - **[`Refresher::new(reload).every(..)`](vantage_diorama::Refresher)**: once a second, `reload` runs again. This is what makes the copy *live*: a sale the till commits, or an edit typed into `sqlite3`, lands on the next tick. The loader and the refresher share one function because both answer the same question: what should the cache hold? [`LoaderCx::reconcile`](vantage_diorama::LoaderCx::reconcile) takes the master's answer and makes the cache hold exactly that set. It writes the rows that differ, deletes the cached rows the master no longer returns (a product that sold out), leaves unchanged rows alone, and publishes one `DatasetChanged` for watchers. ### Why reconcile instead of clearing and reloading? The obvious refresh clears the cache and inserts the fresh rows. It has two problems. The first is the gap. Between the clear and the insert the cache is empty, and a reader that lands in that gap sees an empty shelf. Reconcile never empties the cache: each row is kept, rewritten, or removed. The second is writes in flight. A Dio takes writes optimistically: the cache shows the new value at once, and the master catches up. A refresh that listed the master just before that write landed would put the old value back. Reconcile skips any row with a write in flight, and any row whose write settled after this refresh began listing, so a refresh can't undo a write it raced. The cost is a comparison per row on every tick. For a shelf of products that is noise. For a table of a million rows, a one-second full reconcile is the wrong tool, and the PostgreSQL chapters replace the timer with push. ## Reads come from the cache From here on, reads don't touch SQLite. Everything a consumer asks of the Dio, a listing, a page or a single record, is answered from the in-memory cache the eager loader warmed. The database is consulted by the once-a-second reconcile and by nothing else. The next chapter puts a `DioRouter` over this Dio, and a plain `GET` returns the shelf without a query reaching the table. The backend stays the source of truth, but the handle you *read* is a local, live mirror of it. ```admonish note title="What a capable backend changes, and what it doesn't" The lens changed: a one-call list instead of a cursor walk, memory instead of redb, a refresher where S3 had none, and no augmentation because a SQL row already carries its columns. What *didn't* change is everything above `lens.dio(master).build()`: the `Dio`, its event bus, and the sceneries and watch server the next chapter opens over it are the same code the S3 path uses. The reactive stack doesn't know which backend it sits on. ``` ## Checklist - The model is a plain `Table` with an `#[entity]` struct; nothing in it mentions caching. - The master is `db.vista_factory().from_table(..)`, and a SQL Vista needs no capability injected. - One `reload` function lists the master and hands the rows to `cx.reconcile`; the loader and the refresher both call it. - `Cache::memory()` because re-reading a local table is cheap; `Cache::redb(..)` when it isn't. - `Refresher::new(..).every(..)` is what brings outside writes in, until the PostgreSQL chapters switch to push. Next: serve this Dio as a Kubernetes-style watch, and turn on the till so the shelf moves while you look at it. --- Source: https://vantage-ui.com/book/intro/step6-sql-watch.html # Serving & Watching Live Stock The last chapter left a `Dio` holding a live copy of the `product` table. Now we put it behind HTTP, using the **same adapter the S3 path uses**. `DioRouter` doesn't know or care that its Dio is backed by SQL instead of S3: it binds a Dio to a pair of routes, GET and watch, and that is all it needs. ```rust,no_run use vantage_api_adapters::axum_dio::DioRouter; # use std::time::Duration; # # use vantage_diorama::prelude::*; # use vantage_diorama::{Cache, Loader, LoaderCx, Refresher}; # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # pub struct Product { # pub name: String, # pub price: i64, # pub stock: i64, # } # # impl Product { # pub fn table(db: SqliteDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("stock") # } # } # # async fn reload(cx: LoaderCx) -> VantageResult<()> { # let rows = cx.master().list_values().await?; # cx.reconcile(rows).await # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:products.db?mode=rwc").await.context("connect sqlite")?; # let master = db.vista_factory().from_table(Product::table(db.clone()))?; # let lens = Lens::builder() # .cache(Cache::memory()) # .loader(Loader::eager(reload)) # .refresher(Refresher::new(reload).every(Duration::from_secs(1))) # .build() # .context("build lens")?; # let dio = lens.dio(master).build().await?; # let listener = tokio::net::TcpListener::bind("0.0.0.0:3008").await.context("bind :3008")?; let api = DioRouter::new(dio.clone()) .with_column("name", "name") .with_column("price", "price") .with_column("stock", "stock") .with_page_size(50) .into_router(); let app = axum::Router::new().nest("/api/products", api); axum::serve(listener, app).await.context("server failed")?; # Ok(()) # } ``` Each `with_column` maps a JSON key to a record field. The columns double as the demand of every watch the router opens, which matters on an augmented Dio and is harmless here. A plain `GET` is the cache, answered at once, never a query against SQLite: ```console $ curl 'localhost:3008/api/products?offset=0&limit=10' {"total":5,"offset":0,"limit":10,"items":[ {"index":0,"name":"Espresso","price":280,"stock":12}, {"index":1,"name":"Cappuccino","price":340,"stock":8}, {"index":2,"name":"Cold Brew","price":420,"stock":5}, {"index":3,"name":"Croissant","price":260,"stock":6}, {"index":4,"name":"Cheesecake","price":520,"stock":2}]} ``` ## A till to make it move A cache over a table nobody writes is a slow constant. To see the reactive stack work, the data has to change, so the app runs a **till**: a background task that sells one unit of a random product every 800ms, removes anything that sells out, and every tenth tick takes a delivery. These are ordinary SQL writes, sent with `sqlx::query(..).execute(db.pool())` on the same pool the master reads from (`src/sim.rs`): ```sql -- sell one unit of a random product that still has stock UPDATE product SET stock = stock - 1 WHERE id = (SELECT id FROM product WHERE stock > 0 ORDER BY RANDOM() LIMIT 1); -- anything that hit zero leaves the shelf DELETE FROM product WHERE stock <= 0; ``` The server never calls the till and the till never calls the server. They meet only through the database and the once-a-second reconcile, so **any** writer, in any process, moves the shelf. The `sqlite3` CLI will do as well, as we'll see. ### Why not have the till write through the Dio? A Dio is writable, and a write through it shows up in every watch at once instead of on the next tick. For an app that owns all its writes, that is the better path. This app doesn't own all its writes. A delivery script, a back-office tool, or a person with `sqlite3` changes the table behind the server's back, and a Dio that only learned of its own writes would drift from the database. The till writes straight to SQL on purpose: it stands in for every writer the server can't see. The price is latency, up to one refresh interval, and the PostgreSQL chapters remove it. ## The watch `curl -N` holds the connection open. The page arrives as `ADDED` lines, the current shelf, and then every sale streams back as a `MODIFIED` line once the reconcile picks it up: ```console $ curl -N 'localhost:3008/api/products?watch=true' {"type":"ADDED","object":{"index":0,"name":"Espresso","price":280,"stock":12}} {"type":"ADDED","object":{"index":1,"name":"Cappuccino","price":340,"stock":8}} {"type":"ADDED","object":{"index":2,"name":"Cold Brew","price":420,"stock":5}} {"type":"ADDED","object":{"index":3,"name":"Croissant","price":260,"stock":6}} {"type":"ADDED","object":{"index":4,"name":"Cheesecake","price":520,"stock":2}} {"type":"MODIFIED","object":{"index":4,"name":"Cheesecake","price":520,"stock":1}} {"type":"MODIFIED","object":{"index":2,"name":"Cold Brew","price":420,"stock":4}} {"type":"MODIFIED","object":{"index":0,"name":"Espresso","price":280,"stock":9}} ``` That is the reactive stack running over SQL, end to end: a SQL write, reconciled into the cache, one `DatasetChanged` event, a scenery re-derive, a diff against what this connection last saw, one NDJSON line on the wire. Nothing in that chain is specific to SQL. It is the same machinery the S3 path serves the NOAA bucket with. ```admonish note title="Two events, not three" `DioRouter` emits only `ADDED` and `MODIFIED` here, never `DELETED`, because it diffs each page **by position**. A delivery that grows the shelf appears as an `ADDED` at a new index, but a product that *sells out* shows up as the list shrinking and the rows below it shifting up (each shifted index reported as `MODIFIED`), not as an explicit removal. A watch client reconstructs "gone" from the shrinking `total`. A real `DELETED` line is a one-call opt-in: key the watch by identity with `.key_by("id")`, which the [live-push chapter](./step8-sql-notify.md) turns on. ``` ## It really is the database The strongest proof that this isn't an in-process illusion: change the table from **outside** the app. With a watch open, in another terminal: ```console $ sqlite3 learn-8/products.db "UPDATE product SET price=999 WHERE id='p1'" ``` and the open watch stream reflects it within the second: ```text {"type":"MODIFIED","object":{"index":0,"name":"Espresso","price":999,"stock":11}} ``` No API call and no shared memory: a separate process wrote a row, the reconcile read it, and the change fanned out to every watcher. The Dio is a mirror of the database, not a copy that happens to agree with it. ## Checklist - `DioRouter::new(dio).with_column(..).into_router()` serves GET and watch over any Dio; nest it under the resource's path. - A plain `GET` reads the cache; a `?watch=true` opens a scenery and streams NDJSON. - Writers outside the server reach watchers through the refresher's reconcile, within one interval. - Without `.key_by(..)`, a watch diffs by position and never sends `DELETED`. Next: the same app, the same shelf, the same watch, served from PostgreSQL instead of a file and changed by flipping one compile flag. --- Source: https://vantage-ui.com/book/intro/step7-sql-postgres.html # Moving to PostgreSQL A file on disk is a fine place for a bar to start. When it outgrows one, with concurrent writers, a second instance or a backup story, you reach for a database server. In most stacks that is a migration project. Here it is a refactor that adds one file and changes two attributes, and then a compile flag decides which backend the binary speaks. The finished crate is [`learn-9`](https://github.com/romaninsh/vantage/tree/main/learn-9): the same app as `learn-8`, made backend-parametric. ## Name the backend once `learn-8` wrote `SqliteDB` in a handful of places. The refactor pulls every one of them into a single alias, so the concrete backend is named in one file: ```rust,no_run // src/db.rs: the only file that names a database use vantage_sql::prelude::*; #[cfg(not(feature = "pg"))] pub type Db = vantage_sql::sqlite::SqliteDB; #[cfg(feature = "pg")] pub type Db = vantage_sql::postgres::PostgresDB; pub async fn connect() -> VantageResult { #[cfg(not(feature = "pg"))] let url = format!("sqlite:{}?mode=rwc", concat!(env!("CARGO_MANIFEST_DIR"), "/products.db")); #[cfg(feature = "pg")] let url = std::env::var("DATABASE_URL") .map_err(|_| vantage_core::error!("DATABASE_URL must be set for the `pg` build"))?; Db::connect(&url).await.context("connect db") } ``` `PostgresDB::connect`, `db.pool()` and `db.vista_factory().from_table(..)` are the same calls as their SQLite twins, so `connect()` is the whole of the connection difference: a file path on one side, a `DATABASE_URL` on the other. `db.rs` also holds the `CREATE TABLE`; it is plain SQL both backends accept. ## Two attributes on the model The entity marker names whichever backend is compiled in, and the table builder takes the `Db` alias instead of `SqliteDB`. That is the entire change to the model: ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # #[cfg(feature = "pg")] # use vantage_sql::postgres::AnyPostgresType; # #[cfg(not(feature = "pg"))] # type Db = vantage_sql::sqlite::SqliteDB; # #[cfg(feature = "pg")] # type Db = vantage_sql::postgres::PostgresDB; // src/product.rs #[cfg_attr(not(feature = "pg"), entity(SqliteType))] #[cfg_attr(feature = "pg", entity(PostgresType))] #[derive(Debug, Clone, Default)] pub struct Product { pub name: String, pub price: i64, pub stock: i64, } impl Product { pub fn table(db: Db) -> Table { Table::new("product", db) .with_id_column("id") .with_column_of::("name") .with_column_of::("price") .with_column_of::("stock") } } ``` The builder *body*, the columns and the id, doesn't move. `#[entity(..)]` takes a list of the backends a struct serves, not a choice of one: `#[entity(SqliteType, PostgresType)]` generates both sets of conversions. `learn-9` picks one with `cfg_attr` only so that the build it isn't running doesn't need the other driver compiled in. **Everything else is unchanged: `main.rs`, `sim.rs`, the lens, the Dio and the `DioRouter`.** They were written against `Db` and the portable data-set methods, so they never named a backend. The reconcile still calls `cx.master().list_values()`; the till still runs the same `UPDATE … ORDER BY RANDOM()`; the watch still streams `ADDED`/`MODIFIED`. ```admonish tip title="Why the SQL didn't have to change either" The till uses `$1` placeholders, `ON CONFLICT (id) DO NOTHING` and `RANDOM()`, which SQLite and PostgreSQL both accept, and the schema uses `BIGINT`, which is SQLite's INTEGER affinity and Postgres's 64-bit integer. Where the dialects differ (SQLite's `INTEGER` for booleans against Postgres's `BOOLEAN`, say), the difference lives in the schema strings in `db.rs`, never in the model or the app. ``` ### Why a compile flag and not a runtime switch? You could switch at runtime. Everything from the master up is a type-erased `Vista`, so a `match` on the URL scheme that builds the master from either factory would hand the same Dio code either backend. The parts below the master are typed by backend, though. `Table` and the till's `db.pool()` are a SQLite type or a Postgres type, so a runtime switch would compile both drivers and write those parts twice, once per arm. `learn-9` serves one backend per deployment, so it takes the flag: one alias, one driver in the binary. An app that must speak both at once keeps the `match`, and pays for both drivers. ## Run it on Postgres Start a local Postgres (the repo ships the script), point `DATABASE_URL` at it, and build with the flag: ```console $ vantage-sql/scripts/postgres/start.sh # docker: postgres:17-alpine on :5433 $ export DATABASE_URL='postgres://vantage:vantage@localhost:5433/vantage' $ cargo run -p learn-9 --no-default-features --features pg serving on http://localhost:3009 — try: curl -N 'localhost:3009/api/products?watch=true' ``` The shelf serves and streams as it did on SQLite, with the same routes, the same NDJSON and the same till: ```console $ curl -N 'localhost:3009/api/products?watch=true' {"type":"ADDED","object":{"index":0,"name":"Espresso","price":280,"stock":12}} {"type":"ADDED","object":{"index":1,"name":"Cappuccino","price":340,"stock":8}} ... {"type":"MODIFIED","object":{"index":4,"name":"Cheesecake","price":520,"stock":1}} ``` The it-really-is-the-database proof works over Postgres too. Edit a row with `psql` and watch it land: ```console $ psql "$DATABASE_URL" -c "UPDATE product SET price=777 WHERE id='p2'" {"type":"MODIFIED","object":{"index":1,"name":"Cappuccino","price":777,"stock":6}} ``` ## What this proves The reactive stack, cache, events, sceneries and watch server, ran unchanged over a file-backed SQLite database and a networked PostgreSQL server, and the code that moved between them was one alias and two attributes. The repo's `vantage-sql/tests/postgres/6_vista.rs` exercises the same Vista over a real Postgres. ```admonish success title="The fork converges" The guide's other path wraps a slow, read-only S3 bucket; this one wraps a fast, writable SQL table that then became a Postgres server. They need different lenses: a cursor walk into redb against a one-call list into memory, capability injection against none. Above `lens.dio(master).build()` they run the *same* Dio, the *same* sceneries and the *same* watch adapter. **The reactive layer does not depend on the backend.** Choose the backend your problem has; the stack above it is the one you just learned. ``` ## Checklist - One `Db` alias in `db.rs` names the backend; nothing else does. - `#[cfg_attr(.., entity(..))]` picks the entity marker for the compiled backend. - `main.rs`, the lens, the refresher and the router don't change. - The till's statements and the schema stay within the SQL both dialects accept. - `--no-default-features --features pg` and a `DATABASE_URL` run the same app on Postgres. There is one thing left on the table. The app reconciles on a one-second timer: it re-reads the table whether or not anything changed, and it is always up to a beat behind. A file on disk can't do better; a *server* can. Next, we commit to Postgres and trade that poll for push: the database tells us the moment a row changes. --- Source: https://vantage-ui.com/book/intro/step8-sql-notify.html # Real-Time Push with LISTEN/NOTIFY The app so far reconciles on a **timer**: `Refresher::new(reload).every(Duration::from_secs(1))`. That is polling. Once a second the Dio re-reads the master and reconciles. It works, but it is a beat behind every change, and it re-queries even when nothing moved. A file-backed SQLite database has no better option, because it can't call you back. A PostgreSQL *server* can: it can tell you the instant a row changes, and then the poll goes away. Having shown the app is portable, this chapter **commits to Postgres**: it drops the feature flag, deletes `sim.rs`, moves the writer into its own process, and replaces the refresher with one call on the Dio builder, `.live()`. The finished crate is [`learn-10`](https://github.com/romaninsh/vantage/tree/main/learn-10). ## Teach the database to announce changes Postgres `LISTEN/NOTIFY` is a publish/subscribe channel built into the server. A trigger calls `pg_notify` on every write, and anyone `LISTEN`ing on that channel wakes up. So the only schema added beyond the table is a trigger. `db::setup` sends these statements, each through `sqlx::query(..).execute(db.pool())`, after `CREATE TABLE product (..)`: ```sql CREATE OR REPLACE FUNCTION product_notify() RETURNS trigger AS $$ BEGIN PERFORM pg_notify('product_changed', ''); RETURN NULL; END; $$ LANGUAGE plpgsql; DROP TRIGGER IF EXISTS product_notify_trg ON product; CREATE TRIGGER product_notify_trg AFTER INSERT OR UPDATE OR DELETE ON product FOR EACH STATEMENT EXECUTE FUNCTION product_notify(); ``` `FOR EACH STATEMENT` fires the notify once per statement, not per row: a single "something changed" ping is all the Dio needs to reconcile. The payload is empty on purpose (more on that below). The channel is `product_changed` because the Postgres Vista listens on `{table}_changed`. ## Tell the Vista the trigger exists Postgres offers no way to ask "is anything going to notify me on this channel?". A `LISTEN` on a channel nobody feeds succeeds and then waits forever. A Vista that assumed it could push because the table is writable would advertise a feed that may never arrive, and no consumer could tell that apart from a quiet table. So the application, the only party that knows, says so: ```rust,no_run # use vantage_sql::postgres::{AnyPostgresType, PostgresDB}; # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # use vantage_vista::SortDirection; # # #[entity(PostgresType)] # #[derive(Debug, Clone, Default)] # pub struct Product { # pub name: String, # pub price: i64, # pub stock: i64, # pub created: i64, # } # # impl Product { # pub fn table(db: PostgresDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("stock") # .with_column_of::("created") # } # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let url = std::env::var("DATABASE_URL").context("DATABASE_URL must be set")?; # let db = PostgresDB::connect(&url).await.context("connect postgres")?; let mut master = db .vista_factory() .with_notify(true) // db::setup installed product_notify_trg .from_table(Product::table(db.clone()))?; master.set_order(Some(("created", SortDirection::Ascending)))?; # Ok(()) # } ``` `with_notify(true)` is what lets the Vista advertise [`Subscribe`](vantage_vista::Capability::Subscribe). Leave it off and the live follow below does nothing. That matters here, because this chapter deletes the refresher: push and a refresher's timer are the two ways the cache learns about a write from outside, and an app that declines both never updates. The model gained a `created` column, an epoch-millis stamp set on delivery. The shelf is ordered by it, so a drink keeps its place as it sells and new deliveries append at the end. ## One call replaces the poll The server builds the Lens as before, with an eager in-memory cache and a loader that reconciles, but with **no** refresher. The Dio builder gets `.live()` instead: ```rust,no_run # use vantage_diorama::prelude::*; # use vantage_diorama::{Cache, Loader, LoaderCx}; # use vantage_sql::postgres::{AnyPostgresType, PostgresDB}; # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(PostgresType)] # #[derive(Debug, Clone, Default)] # pub struct Product { # pub name: String, # pub price: i64, # pub stock: i64, # pub created: i64, # } # # impl Product { # pub fn table(db: PostgresDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("stock") # .with_column_of::("created") # } # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let url = std::env::var("DATABASE_URL").context("DATABASE_URL must be set")?; # let db = PostgresDB::connect(&url).await.context("connect postgres")?; # let master = db # .vista_factory() # .with_notify(true) # .from_table(Product::table(db.clone()))?; let lens = Lens::builder() .cache(Cache::memory()) .loader(Loader::eager(|cx: LoaderCx| async move { let rows = cx.master().list_values().await?; cx.reconcile(rows).await })) .build() .context("build lens")?; let dio = lens.dio(master).live().build().await?; # Ok(()) # } ``` That is the whole change. [`live()`](vantage_diorama::DioBuilder::live) checks whether the master advertises `Subscribe`. Because we opted in above, the Postgres Vista does, and the Dio opens a `LISTEN` on `product_changed`. Each notification arrives as a coarse change, "the set changed", and the Dio answers it with a [refresh](vantage_diorama::Dio::refresh). With no refresher on the Lens, a refresh re-runs the eager loader, so the same `cx.reconcile(rows)` that warmed the cache brings it up to date. Over a master that can't push, `live()` does nothing and a refresher carries the load instead. ```admonish note title="The server still never writes" The live follow only *reads and reconciles*. Nothing in the server process writes to `product`. That separation keeps the demo honest: whatever you see on screen arrived over the database. ``` ### Why not listen for NOTIFY yourself? You can. A `PgListener` on `product_changed` and a loop is a dozen lines, and an event source the Vista doesn't model (a webhook, a Kafka topic, a SaaS callback) needs exactly that. The loop hands each event to the Dio with [`apply_change`](vantage_diorama::Dio::apply_change): ```rust,no_run # use vantage_diorama::prelude::*; # use vantage_diorama::{Cache, Loader, LoaderCx}; # use vantage_sql::postgres::{AnyPostgresType, PostgresDB}; # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(PostgresType)] # #[derive(Debug, Clone, Default)] # pub struct Product { # pub name: String, # pub price: i64, # pub stock: i64, # pub created: i64, # } # # impl Product { # pub fn table(db: PostgresDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("stock") # .with_column_of::("created") # } # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let url = std::env::var("DATABASE_URL").context("DATABASE_URL must be set")?; # let db = PostgresDB::connect(&url).await.context("connect postgres")?; # let master = db.vista_factory().from_table(Product::table(db.clone()))?; # let lens = Lens::builder() # .cache(Cache::memory()) # .loader(Loader::eager(|cx: LoaderCx| async move { # let rows = cx.master().list_values().await?; # cx.reconcile(rows).await # })) # .build() # .context("build lens")?; # let dio = lens.dio(master).build().await?; // A payload-less ping: re-read the set. dio.apply_change(ChangeEvent::Invalidated).await?; // A source that names the row: drop it from the cache, no re-read. dio.apply_change(ChangeEvent::Deleted { id: "d42".into() }).await?; # Ok(()) # } ``` `live()` applies changes the same way, so for a Vista that advertises `Subscribe` the hand-written loop buys nothing and costs two things `live()` already gets right. It subscribes *before* the eager loader copies the set and applies what arrived once the copy lands, so a write during the copy is neither lost nor overwritten by it. And a follow that falls behind the stream re-reads the set instead of dropping changes. Write the loop for sources the Vista can't reach; use `live()` for the ones it can. ## The till becomes its own process To drive it, `learn-10` ships a second binary, the **mutator**, run alongside the server: ```console $ cargo run -p learn-10 # the reactive server, port 3010 $ cargo run -p learn-10 --bin mutator # a separate writer process ``` The mutator writes through Vantage's **active-entity** API, with no hand-written SQL. `list_entities()` hands back drinks that each carry their own id and datasource, so selling one is: ```rust,no_run # use rand::seq::SliceRandom; # use vantage_dataset::prelude::ActiveEntitySet; # use vantage_sql::postgres::{AnyPostgresType, PostgresDB}; # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(PostgresType)] # #[derive(Debug, Clone, Default)] # pub struct Product { # pub name: String, # pub price: i64, # pub stock: i64, # pub created: i64, # } # # impl Product { # pub fn table(db: PostgresDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("stock") # .with_column_of::("created") # } # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let url = std::env::var("DATABASE_URL").context("DATABASE_URL must be set")?; # let db = PostgresDB::connect(&url).await.context("connect postgres")?; # let table = Product::table(db.clone()); let mut shelf = table.list_entities().await?; let drink = shelf.choose_mut(&mut rand::thread_rng()).unwrap(); if drink.stock <= 1 { drink.delete().await?; // last unit: off the shelf } else { drink.stock -= 1; drink.save().await?; // one sale } # Ok(()) # } ``` Two independent processes that share nothing but the database and the notify channel: the strongest proof that the reactive stack tracks the *data*, not an in-process shortcut. ## Removals, and a UI to see them This app also keys its watch by identity, so a sold-out drink is reported as a removal rather than a list that shrinks: ```rust,no_run # use vantage_api_adapters::axum_dio::DioRouter; # use vantage_diorama::prelude::*; # use vantage_diorama::{Cache, Loader, LoaderCx}; # use vantage_sql::postgres::{AnyPostgresType, PostgresDB}; # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(PostgresType)] # #[derive(Debug, Clone, Default)] # pub struct Product { # pub name: String, # pub price: i64, # pub stock: i64, # pub created: i64, # } # # impl Product { # pub fn table(db: PostgresDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("stock") # .with_column_of::("created") # } # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let url = std::env::var("DATABASE_URL").context("DATABASE_URL must be set")?; # let db = PostgresDB::connect(&url).await.context("connect postgres")?; # let master = db # .vista_factory() # .with_notify(true) # .from_table(Product::table(db.clone()))?; # let lens = Lens::builder() # .cache(Cache::memory()) # .loader(Loader::eager(|cx: LoaderCx| async move { # let rows = cx.master().list_values().await?; # cx.reconcile(rows).await # })) # .build() # .context("build lens")?; # let dio = lens.dio(master).live().build().await?; let api = DioRouter::new(dio.clone()) .with_column("id", "id") .with_column("name", "name") .with_column("price", "price") .with_column("stock", "stock") .key_by("id") // identity-keyed: emits DELETED when a row leaves .with_page_size(50) .into_router(); # Ok(()) # } ``` With `key_by("id")`, the watch stream gains a third event, `DELETED`, alongside `ADDED` and `MODIFIED`. The key field has to be one of the `with_column` fields, which is why `id` is exposed now. A small React page (`frontend/index.html`, served by the same axum app through `ServeDir`) consumes the stream and animates each drink: arriving with a flash, ticking down as it sells, and leaving with a "sold out" stamp when a `DELETED` lands. It is plain React over the watch endpoint, not part of Vantage, and the next chapter reuses the same file unchanged. Watch it on the wire: ```console $ curl -N 'localhost:3010/api/products?watch=true' {"type":"ADDED","object":{"index":0,"id":"d51...","name":"Mojito","price":1000,"stock":11}} {"type":"ADDED","object":{"index":1,"id":"d59...","name":"Boulevardier","price":1400,"stock":12}} {"type":"MODIFIED","object":{"index":0,"id":"d51...","name":"Mojito","price":1000,"stock":10}} {"type":"DELETED","object":{"index":0,"id":"d59...","name":"Boulevardier","price":1400,"stock":1}} ``` Every line is a database write, announced by Postgres, reconciled into the cache, diffed by the scenery and fanned out, with no timer anywhere in the loop. ```admonish tip title="Coarse push: a ping, then a re-read" `NOTIFY` carries no row data; the payload is empty. So the Dio learns only *that* something changed, and responds by re-reading the whole set to reconcile. That is correct and, for a shelf of a few dozen drinks, instant. But it is a **coarse** signal: one write means one full reconcile. The next chapter switches to a database whose push carries the changed row itself, so no re-read is needed. ``` ## Checklist - A `FOR EACH STATEMENT` trigger calls `pg_notify('{table}_changed', '')` on every write. - `vista_factory().with_notify(true)` declares the trigger, so the Vista advertises `Subscribe`. - `lens.dio(master).live().build()` follows the channel; each ping refreshes the Dio, and with no refresher that re-runs the eager loader's reconcile. - Without `with_notify(true)` and without a refresher, the cache never sees an outside write. - `dio.apply_change(ChangeEvent::..)` is the entry for sources the Vista can't subscribe to. - `.key_by("id")` on the router turns sell-outs into `DELETED` events. Next: the same app, the same frontend, a different database. SurrealDB's live queries push the change itself rather than a ping. --- Source: https://vantage-ui.com/book/intro/step9-surreal.html # Switching to SurrealDB Postgres was a migration *within* SQL: a different server, the same dialect family. SurrealDB is a different database: multi-model, schemaless by default, and reached over a WebSocket rather than a pooled SQL connection. Adopting it tests the claim this path has been making. The goal is to change **as little as possible**, and then to cash in the one thing SurrealDB does better than a SQL server: it pushes the *changed row*, not a ping. The finished crate is [`learn-11`](https://github.com/romaninsh/vantage/tree/main/learn-11). Set it next to `learn-10` and the diff is small, which is the result we want. ## What changes Three places name the backend, and nothing else does. **The entity marker**: swap the backend in the attribute. ```rust # use vantage_surrealdb::types::AnySurrealType; # use vantage_types::entity; #[entity(SurrealType)] // was: #[entity(PostgresType)] #[derive(Debug, Clone, Default)] pub struct Product { pub name: String, pub price: i64, pub stock: i64, pub created: i64, } ``` **The table builder**: the same columns, built against `SurrealDB`. SurrealDB's native id is a `Thing` (`product:⟨key⟩`). ```rust # use vantage_surrealdb::surrealdb::SurrealDB; # use vantage_surrealdb::types::AnySurrealType; # use vantage_table::table::Table; # use vantage_types::entity; # # #[entity(SurrealType)] # #[derive(Debug, Clone, Default)] # pub struct Product { # pub name: String, # pub price: i64, # pub stock: i64, # pub created: i64, # } # # impl Product { pub fn surreal_table(db: SurrealDB) -> Table { Table::new("product", db) .with_id_column("id") .with_column_of::("name") .with_column_of::("price") .with_column_of::("stock") .with_column_of::("created") } # } ``` **Connect and setup**: a DSN instead of a `DATABASE_URL`, and **no trigger**. Where Postgres needed a `pg_notify` function and an `AFTER … FOR EACH STATEMENT` trigger, SurrealDB emits change frames on its own. Setup makes sure the table exists: ```rust,no_run # use surreal_client::SurrealConnection; # use vantage_core::{Result, error}; # use vantage_surrealdb::surrealdb::SurrealDB; # # #[tokio::main(flavor = "current_thread")] # async fn main() -> Result<()> { let dsn = std::env::var("SURREALDB_URL") .unwrap_or_else(|_| "cbor://root:root@localhost:8000/bar/v1".to_string()); let client = SurrealConnection::dsn(&dsn) .map_err(|e| error!("invalid SURREALDB_URL DSN", details = e.to_string()))? .connect() .await .map_err(|e| error!("connect surrealdb", dsn = &dsn, details = e.to_string()))?; // The one bit of schema: a table for LIVE SELECT to attach to. client .query("DEFINE TABLE IF NOT EXISTS product SCHEMALESS", None) .await .map_err(|e| error!("define product table", details = e.to_string()))?; let db = SurrealDB::new(client); # Ok(()) # } ``` The till (the mutator) is the same active-entity code as before. Only the id it hands `new_entity` is a `Thing`: ```rust,no_run # use vantage_core::{Context, Result}; # use vantage_dataset::prelude::ActiveEntitySet; # use vantage_surrealdb::surrealdb::SurrealDB; # use vantage_surrealdb::thing::Thing; # use vantage_surrealdb::types::AnySurrealType; # use vantage_table::table::Table; # use vantage_types::entity; # # #[entity(SurrealType)] # #[derive(Debug, Clone, Default)] # pub struct Product { # pub name: String, # pub price: i64, # pub stock: i64, # pub created: i64, # } # # impl Product { # pub fn surreal_table(db: SurrealDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("stock") # .with_column_of::("created") # } # } # # async fn deliver( # table: &Table, # key: String, # name: String, # price: i64, # stock: i64, # created: i64, # ) -> Result<()> { table .new_entity( Thing::new("product", key), // was: a String id Product { name, price, stock, created }, ) .save() .await?; # Ok(()) # } # # fn main() {} ``` ## What doesn't change The server's reactive core is untouched. The Vista comes from `SurrealVistaFactory::new(db.clone())` (the factory `db.vista_factory()` returns on a `SurrealDB`) instead of the Postgres factory, and the Lens and the Dio builder are line for line the Postgres chapter's, including the call that matters: ```rust,no_run # use surreal_client::SurrealConnection; # use vantage_core::{Context, Result, error}; # use vantage_dataset::prelude::*; # use vantage_diorama::prelude::*; # use vantage_diorama::{Cache, Loader, LoaderCx}; # use vantage_surrealdb::surrealdb::SurrealDB; # use vantage_surrealdb::types::AnySurrealType; # use vantage_surrealdb::vista::SurrealVistaFactory; # use vantage_table::table::Table; # use vantage_types::entity; # use vantage_vista::SortDirection; # # #[entity(SurrealType)] # #[derive(Debug, Clone, Default)] # pub struct Product { # pub name: String, # pub price: i64, # pub stock: i64, # pub created: i64, # } # # impl Product { # pub fn surreal_table(db: SurrealDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("stock") # .with_column_of::("created") # } # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> Result<()> { # let dsn = std::env::var("SURREALDB_URL") # .unwrap_or_else(|_| "cbor://root:root@localhost:8000/bar/v1".to_string()); # let client = SurrealConnection::dsn(&dsn) # .map_err(|e| error!("invalid SURREALDB_URL DSN", details = e.to_string()))? # .connect() # .await # .map_err(|e| error!("connect surrealdb", dsn = &dsn, details = e.to_string()))?; # let db = SurrealDB::new(client); let mut master = SurrealVistaFactory::new(db.clone()).from_table(Product::surreal_table(db.clone()))?; master.set_order(Some(("created", SortDirection::Ascending)))?; let lens = Lens::builder() .cache(Cache::memory()) .loader(Loader::eager(|cx: LoaderCx| async move { let rows = cx.master().list_values().await?; cx.reconcile(rows).await })) .build() .context("build lens")?; let dio = lens.dio(master).live().build().await?; // the same call, now backed by LIVE SELECT # Ok(()) # } ``` And the **frontend is the same file** as `learn-10`. The React page consumes `?watch=true` and animates the shelf; it never learns which database is behind the stream. Diff the two `frontend/index.html` files and you get nothing: the reactive layer, up to and including the UI, does not depend on the backend. ## The payoff: precise push, no re-read `live()` looks the same, but underneath it does something better than it could on Postgres. SurrealDB's `LIVE SELECT` delivers a **typed, per-row** notification for every change: the action (`CREATE`, `UPDATE`, `DELETE`) *and* the affected record. So the Dio applies each change directly to its cache, as [`ChangeEvent::Inserted`](vantage_diorama::ChangeEvent::Inserted), `Updated` or `Deleted` would through `dio.apply_change`. A create inserts one row, an update repaints one row, a delete removes one row. There is no "something changed, re-read everything" step, because the change arrived in full. ```console $ curl -N 'localhost:3011/api/products?watch=true' {"type":"ADDED","object":{"index":0,"id":{"@@TAGGED@@":[8,["product","d51..."]]},"name":"Mojito","price":1000,"stock":11}} {"type":"MODIFIED","object":{"index":0,"id":{"@@TAGGED@@":[8,["product","d51..."]]},"name":"Mojito","price":1000,"stock":10}} {"type":"DELETED","object":{"index":0,"id":{"@@TAGGED@@":[8,["product","d59..."]]},"name":"Boulevardier","price":1400,"stock":1}} ``` The id arrives as a SurrealDB `Thing` rather than a bare string. The frontend's `idOf` helper normalises either shape to a stable key, which is why the same UI serves both backends. ### Why does SurrealDB need no `with_notify(true)`? Because there is nothing to forget. Postgres pushes only through a trigger the application installs, and the driver can't see whether it exists, so the application has to declare it. SurrealDB's `LIVE SELECT` is part of the server: any table can be followed, with no schema of yours involved. The Surreal Vista advertises `Subscribe` from the moment you build it, and `live()` uses it. The asymmetry is the backend's, not Vantage's. A capability the driver can verify is advertised by default; one that depends on application setup is opt-in, so a Vista never promises a feed that may be silent. ```admonish success title="One capability, three freshness models" The `product` shelf has now run behind three backends. Above the master, what changed was the Lens losing its refresher and the Dio builder gaining `.live()`: | Backend | How the Dio learns of a change | Granularity | |---|---|---| | SQLite | a `Refresher::new(..).every(..)` timer (it can't push) | poll, whole set | | PostgreSQL | `LISTEN/NOTIFY`: a payload-less ping, then a re-read | **coarse** push, re-read on each write | | SurrealDB | `LIVE SELECT`: the action and the row | **fine-grained** push, the row applied directly | Poll, coarse push, fine push. `live()` picks from what the Vista advertises, and falls back to the refresher when it advertises nothing. You pick the database your problem has; the freshness you get is the best that database can offer, and the code that consumes it is the code you already wrote. Note what each one costs you: SurrealDB is watchable the moment you build the Vista, while the PostgreSQL row assumes the trigger from the previous chapter *and* the `with_notify(true)` that declares it. Every other backend Vantage speaks polls today; see [Live Data](../live-data.md) for the full picture. ``` ## Checklist - The entity marker becomes `#[entity(SurrealType)]`; the columns don't move. - `SurrealConnection::dsn(..).connect()` replaces `DATABASE_URL`, and `DEFINE TABLE` is the only schema. - New records take a `Thing` id: `Thing::new("product", key)`. - `SurrealVistaFactory::new(db).from_table(..)` builds the master; the Lens and `.live()` are unchanged. - Each `LIVE SELECT` frame is applied as one row change; nothing is re-read. From here the reference half of the book takes over: [SurrealDB](../surrealdb.md) for the driver details, [SQL](../sql.md) for the dialects, [Config-Driven Vistas](../config-driven-vistas.md) for declaring all of this from YAML, [Live Data](../live-data.md) for what every backend does about freshness, and [Adding a New Persistence](../new-persistence.md) when your backend isn't one Vantage already speaks. --- Source: https://vantage-ui.com/book/history.html # The Vantage Journey ```admonish note title="Historical snippets" Each snippet shows the API of the release its section describes. They aren't compiled against the current crates. ``` Vantage didn't start as a multi-backend entity framework. It started as a weekend experiment to see if Rust could build SQL queries without feeling like Rust. This page walks you through each release — what changed, why it mattered, and how the API evolved from raw Postgres queries to a universal persistence layer. ```mermaid graph LR D[0.0 DORM] -->|rename| V1[0.1 Vantage] V1 -->|entity framework| V2[0.2 MDA] V2 -->|crate split| V3[0.3 UI Adapters] V3 -->|type rewrite| V4[0.4 Type System] V4 -->|universal handle| V5[0.5 Vista] V5 -->|hardening| V6[0.6 Robustness] style D fill:#555,color:#fff style V1 fill:#4a7c59,color:#fff style V2 fill:#2d6a8f,color:#fff style V3 fill:#8f5a2d,color:#fff style V4 fill:#7c2d8f,color:#fff style V5 fill:#2d8f6a,color:#fff style V6 fill:#8f2d4a,color:#fff ``` --- ## 0.0 — "DORM" (April–November 2024) The project was originally called **DORM** — the Dry ORM. It was Postgres-only, monolithic, and proudly opinionated. Everything lived in one crate. The core idea was already there: **Data Sets**. Instead of loading records eagerly, you describe _what_ you want and let the framework figure out the query. ```rust,ignore let clients = Client::table(); // Table let paying = clients.with_condition( clients.is_paying_client().eq(&true) ); let orders = paying.ref_orders(); // Table for order in orders.get().await? { println!("#{} total: ${:.2}", order.id, order.total as f64 / 100.0); } ``` Behind this innocent-looking code, DORM generated a single SQL query with subqueries, joins, and soft-delete filters — all derived from model definitions: ```sql SELECT id, (SELECT name FROM client WHERE client.id = ord.client_id) AS client_name, (SELECT SUM((SELECT price FROM product WHERE id = product_id) * quantity) FROM order_line WHERE order_line.order_id = ord.id) AS total FROM ord WHERE client_id IN (SELECT id FROM client WHERE is_paying_client = true) AND is_deleted = false ``` No hand-written SQL. No query strings. The framework composed everything from relationship definitions and table extensions like `SoftDelete`. ```admonish tip title="The Arc breakthrough" Early commits show a battle with Rust's borrow checker — the infamous "lifetime hell." The solution came on April 28: switching to `Arc` for shared ownership. This unlocked clonable data sources and composable table references that define the framework to this day. ``` ### Milestones | Date | What happened | | ------ | --------------------------------------------------- | | Apr 11 | First commit — queries, expressions, SQLite binding | | Apr 18 | Insert/delete support, first Postgres tests | | Apr 28 | `Arc` adoption — escaped lifetime hell | | May 14 | `Query::Join` implemented | | May 25 | `has_one`, `has_many`, relationship traversal | | May 26 | Bakery model example — all entities defined | --- ## 0.1 — Vantage is born (December 2024) On December 12, the framework was renamed from DORM to **Vantage** and published to crates.io for the first time. The API stayed the same — this was a branding milestone, not an architectural one. ```admonish note title="Why 'Vantage'?" A vantage point gives you a clear view of the landscape below. The framework gives you a clear view of your data — no matter where it lives or how complex the relationships are. ``` The same month brought the first Axum integration (`bakery_api`), proving that data sets could drive REST endpoints naturally: ```rust,ignore async fn list_orders( client: axum::extract::Query, pager: axum::extract::Query, ) -> impl IntoResponse { let orders = Client::table() .with_id(client.client_id.into()) .ref_orders(); let mut query = orders.query(); query.add_limit(Some(pager.per_page)); Json(query.get().await.unwrap()) } ``` Tags `v0.1.0` and `v0.1.1` were published the same day. SQLx data source landed 11 days later. --- ## 0.2 — Entity Framework & MDA (February 2025) Version 0.2 repositioned Vantage as a full **Entity Framework** with Model-Driven Architecture. The README doubled in size. The vision expanded from "clever query builder" to "how enterprises should structure business logic." The key insight: entities aren't just database rows. They're **business objects** that might live in SQL, NoSQL, a REST API, or a message queue — and your code shouldn't care which. ```rust,ignore impl Client { fn table() -> Table { /* ... */ } fn registration_queue() -> impl Insertable { /* Kafka */ } fn admin_api() -> impl DataSet { /* REST */ } fn read_csv(file: String) -> impl ReadableDataSet { /* CSV */ } } ``` ```admonish example title="Same interface, any backend" A developer calling `Client::registration_queue().insert(id, client).await` doesn't need to know it's Kafka underneath. The SDK hides the transport — only the entity contract matters. ``` This release also introduced the idea of **struct projection** — using different Rust types against the same data set to control which fields get queried: ```rust,ignore struct MiniClient { name: String } struct FullClient { name: String, email: String, balance: Decimal } // Only fetches `name` from the database let name = clients.get_id_as::(42).await?.name; ``` The monolith was getting heavy, though. Everything still lived in one crate, and adding a new database meant touching core code. --- ## 0.3 — The Great Separation (July–October 2025) Version 0.3 broke the monolith into **dedicated crates** and bet heavily on SurrealDB as the primary backend. The trait-based architecture that defines Vantage today was born here. ```admonish info title="Crate explosion" One crate became many: `vantage-expressions`, `vantage-table`, `vantage-dataset`, `vantage-surrealdb`, `surreal-client`, `vantage-config`, `vantage-ui-adapters` — each with a focused responsibility. ``` Table definitions moved from static initialization to a **builder pattern**: ```rust,ignore // 0.2 — static, Postgres-only Table::new_with_entity("bakery", postgres()) .with_id_column("id") .with_column("name") .with_many("clients", "bakery_id", || Box::new(Client::table())) // 0.3 — builder, any datasource Table::::new("client", ds.clone()) .with_id_column("id") .with_column("name") .with_column("email") .with_many("orders", "client_id", || Client::order_table()) ``` Field accessors now return `Expression` instead of column objects — making them composable across query builders: ```rust,ignore // Build conditions from expressions let active = clients.is_paying_client().eq(true); let big_spenders = clients.balance().gt(1000); let query = clients.with_condition(active).with_condition(big_spenders); ``` The **`AnyTable`** type-erasure system arrived, enabling generic code that works with any datasource: ```rust,ignore let tables: Vec = vec![ AnyTable::new(Client::table()), // SurrealDB AnyTable::new(Product::table()), // SQLite ]; for table in &tables { println!("{}: {} records", table.name(), table.count().await?); } ``` This release culminated with **UI adapters for six frameworks** — egui, GPUI, Slint, Tauri, Ratatui, and Cursive — all driven by the same `AnyTable` interface. ```admonish success title="One data layer, six UIs" The same bakery model powered a native desktop app (GPUI), a web app (Tauri), a terminal dashboard (Ratatui), and three more — without changing a single line of business logic. ``` ![egui, GPUI, Slint, Tauri, Cursive, Ratatui — same data, six frameworks](images/ui-slideshow.gif) --- ## 0.4 — The Type System Rewrite (November 2025–April 2026) ```admonish info title="Vantage 0.4 — the type system" Version 0.4 rewrites the type system from the ground up. Custom types per datasource, CBOR protocol, 7 persistence backends (SurrealDB, Postgres, MySQL, SQLite, MongoDB, CSV, REST API), `ActiveEntity` / `ActiveRecord` patterns, typed columns, unified error handling, and a progressive trait model where each persistence only implements what its engine supports. **[Read the full 0.4 feature guide →](./whats-new-04.md)** ``` --- ## 0.5 — The Vista Era (May–June 2026) Version 0.4 made the type system precise. Version 0.5 asks the next question: how does code that _doesn't know your entity type_ talk to your data? A CLI that lists any table, a web admin that draws forms from a YAML schema, a UI data grid pointed at whatever you give it — none of them know your `Product` struct. 0.4's answer was `AnyTable`: type-erasure that funnelled everything through JSON. 0.5 replaces it with **`Vista`** — a universal, schema-bearing data handle. A Vista wraps any typed `Table`, erases both backend and entity, and carries its own schema (columns, references, capabilities) while delegating execution to a per-driver `TableShell`. ```rust,ignore // Wrap any typed table — SQLite, MongoDB, AWS, all the same shape let vista = SqliteVistaFactory::new(db).from_table(Product::table(db.clone()))?; // Everything is now runtime introspection — no generics, no entity type for name in vista.get_column_names() { let col = vista.get_column(name).unwrap(); println!("{}: {}", col.name, col.original_type); } let mut v = vista.clone(); v.add_condition_eq("category_id", 1.into())?; // driver translates to native condition v.add_search("tart")?; // fans across SEARCHABLE columns v.add_order("price", SortDirection::Descending)?; let rows = v.fetch_page(1).await?; ``` ```admonish info title="CBOR, not JSON" Where `AnyTable` narrowed every value to `serde_json::Value`, Vista carries [`ciborium::Value`](https://docs.rs/ciborium) end to end — preserving integer-vs-float, binary blobs, and precise decimals. JSON conversion happens only at the boundary, when you actually need it (an HTTP response, say). ``` The `AnyTable` carrier was deleted outright in `vantage-table 0.5.2` — type erasure now lives one layer up, in `Vista`. That decommission is what the 0.5 version bump marks. ### Capabilities — the explicit contract Not every backend can do everything. A CSV file can't sort server-side; DynamoDB can only order by its sort key; a REST API may paginate by cursor but not by page number. Vista makes this explicit with a struct of capability flags, and each driver declares exactly what it supports: ```rust,ignore let caps = vista.capabilities(); if caps.can_search { v.add_search("query")?; } if caps.can_fetch_page { /* random-access pager */ } else { /* load-more button */ } ``` Calling a method the driver doesn't advertise returns an `Unsupported` error — never a silent match-all, never a panic. UI adapters branch on these flags to decide which controls to render. ### Config-driven: YAML specs and Rhai scripting A Vista no longer needs a hand-written Rust definition. Tables, columns, and relations can be declared in a **YAML spec** (`VistaSpec`) and loaded by the driver's factory. For anything YAML can't express — vendor-specific expressions, derived sources, scripted reference traversal — there's an optional **Rhai** DSL that compiles to native queries: ```rust,ignore let users = table("users").alias("u"); select() .from(users) .expression(users["name"]) .where(users["age"] >= 18) .order_by(users["name"], "asc") ``` The same script renders dialect-correct SQL for SQLite, Postgres, and MySQL — automatic identifier quoting, `date_format()` mapping to `strftime`/`TO_CHAR`/`DATE_FORMAT`, `group_concat()` becoming `GROUP_CONCAT`/`STRING_AGG`. SurrealDB gets its own vocabulary (graph traversal, `RELATE`, record ids). YAML stays the primary format; Rhai is the targeted, serializable escape hatch. ### Contained relations and nested writes Embedded objects and arrays — an order's `lines`, a row's JSON column — now surface as a fully **editable sub-Vista**. SurrealDB backs them with native nested objects; SQL backends store them as JSON columns and patch the host column on writeback. And `insert_value` learned to walk relations: hand it a record whose keys name a relation and it sequences the writes, stamping foreign keys automatically. ### Diorama — caching, events, and reactive views The biggest new subsystem is **`vantage-diorama`** — a layer that sits between a Vista and whatever consumes it. Three concepts: - **Lens** — long-lived shared infrastructure: a cache backend, lifecycle callbacks, refresh policy. Built once per application. - **Dio** — a Vista bound to a Lens. Owns the cache table, a write queue, an event bus, and a refresh task. - **Scenery** — a reactive view onto a Dio (ordered tables, individual records, aggregates) that a UI binds to. ```rust,ignore let lens = Arc::new( Lens::new() .cache_at("./cache.redb") .on_start(|dio| { let dio = dio.clone(); async move { let rows = dio.master().list_values().await?; dio.cache().insert_values(rows).await?; Ok(()) }}) .refresh_every(Duration::from_secs(300)) .build()?, ); let products = lens.make_dio(products_vista).await?; let mut v = products.vista(); // facade: reads from cache, writes through the queue ``` ```admonish success title="Capability injection" Diorama caches the full dataset locally, so a read-only CSV Vista that _can't_ paginate, sort, or search server-side becomes one that can — the consumer sees a richer Vista than the backend actually supports. Two-pass progressive loading renders a cheap list immediately and hydrates expensive per-row detail lazily as rows scroll into view. ``` ### More backends, more reach 0.5 also widened the backend roster around the Vista abstraction: **redb** (embedded key-value store), **AWS** (DynamoDB and friends, all returning Vistas directly), a **Cmd** source that turns a shell script into a queryable table, and an **API pool**. Cross-persistence reference traversal — categories in Postgres, products in MongoDB — moved out of `Vista` and into **`vantage-vista-factory`**'s `VistaCatalog`, keeping each `Vista` strictly single-backend. ```admonish note title="A new introduction" The getting-started guide was rewritten around these concepts — chapters on [Vista](./intro/step4-vista.md), [Dio & Lens](./intro/step5-dio-lens.md), and [Scenery](./intro/step7-scenery.md) build a CLI up from a single SQLite query to a reactive, cached, multi-backend handle. **[Read the full 0.5 feature guide →](./whats-new-in-05.md)** ``` --- ## 0.6 — Hardening (June 2026) If 0.5 was about reach, 0.6 is about trust. The release is a coordinated, workspace-wide sweep with a single theme: **no panics, no silent failures.** Every path that could abort the process deep inside a write, or quietly return a wrong answer, was made fallible and explicit. - **Serialization is fallible.** A new `TryIntoRecord` trait replaces the infallible blanket: a failing `Serialize` (non-string map keys, an out-of-range number) now becomes a recoverable error instead of a process abort mid-write. `Entity` write paths (`insert` / `replace` / `patch`) propagate it. - **No silent zeros.** `get_count` unwraps the common `[{"count": N}]` shape and surfaces unrecognized result shapes as errors instead of returning `0`. - **No silent match-all.** Search on a backend that can't search server-side (CSV, redb) returns an `Unsupported` error — never a quietly unfiltered result set. - **Expression safety.** `f64::IntoValue` degrades NaN/Infinity to `Null` instead of panicking; `Expression::preview()` uses single-pass interleaving so a `{}` inside a rendered value can't corrupt the output. `select_column` became fallible. - **Clearer error classification.** `vantage-core`'s error-kind annotators were renamed to `mark_unsupported` / `mark_unimplemented` (the `is_*` names are now real predicates), and trace emission was decoupled — `mark_*` no longer logs implicitly; chain `.traced()` when you want the `tracing::error!` event. - **Security: SurrealDB identifier injection.** Identifier escaping was unified into `surreal-client`'s single `escape_identifier` authority and fixed for SurrealDB 3.x — a crafted `⟩` inside a column name could previously break out of `⟨…⟩` quoting and inject arbitrary SurrealQL. `similarity()` and `time_group()` literals are now bound as parameters rather than interpolated, closing the same hole on the Rhai-exposed search path. Alongside the hardening, 0.6 folded the standalone `vantage-data-script` crate into `vantage-vista`'s `rhai` feature, added random-access windowing (`fetch_window`) with REST lazy-loading, and introduced `ColumnFlag::Label` for status-tag display hints. --- ## The bigger picture Looking at the trajectory: | Version | Core idea | Backends | Crates | | ------- | ----------------------------- | ---------------------------------------------------------------- | ------ | | 0.0 | Can Rust build SQL smartly? | Postgres | 1 | | 0.1 | Let's publish this | Postgres | 1 | | 0.2 | Entity Framework for Rust | Postgres | 1 | | 0.3 | Traits, not inheritance | SurrealDB, SQLite | 10+ | | 0.4 | Strict types, any persistence | SurrealDB, SQLite, Postgres, MySQL, MongoDB, CSV, REST API | 20+ | | 0.5 | A universal data handle | + redb, AWS (DynamoDB), Cmd (shell scripts) | 25+ | | 0.6 | No panics, no silent failures | (same — hardening release) | 25+ | What started as 16 commits in April 2024 is now **800+ commits**, **300+ pull requests**, and a framework that can drive the same business logic across a dozen backends and six UI frameworks — typed when you know the entity, universal (via `Vista`) when you don't. The destination hasn't changed since day one: **describe your data once, use it everywhere.** Each version just made "everywhere" a little bigger. --- Source: https://vantage-ui.com/book/whats-new-04.html # What's New in Vantage 0.4 ```admonish note title="Historical snippets" The snippets show the 0.4 API. They aren't compiled against the current crates. ``` Version 0.4 is a **ground-up rewrite of the type system** and a massive expansion of persistence support. If 0.3 proved the architecture, 0.4 makes it production-ready. --- ## Progressive model — implement only what you need In earlier versions, adding a new backend meant implementing _everything_. Version 0.4 flips this: capabilities unlock **progressively** based on which traits your persistence (backend) implements. There are two parallel trait hierarchies — one for typed entities, one for schema-less records. **DataSet** works with your Rust structs: ```rust,ignore // A CSV file is read-only — and that's fine impl ReadableDataSet for CsvFile { /* ... */ } // A message queue can only append impl InsertableDataSet for KafkaTopic { /* ... */ } // A full database gets the complete toolkit impl ReadableDataSet for SurrealDB { /* ... */ } impl WritableDataSet for SurrealDB { /* ... */ } // ReadableDataSet + WritableDataSet automatically unlocks ActiveEntitySet! ``` **ValueSet** is the schema-less counterpart — same hierarchy, but operates on `Record` instead of entities. Useful for dynamic tables, config-driven schemas, or when you don't have (or want) a struct: ```rust,ignore // Read raw records — no struct needed impl ReadableValueSet for CsvFile { /* returns Record */ } // Full CRUD on raw values impl WritableValueSet for SurrealDB { /* accepts Record */ } // ReadableValueSet + WritableValueSet automatically unlocks ActiveRecordSet! ``` Each layer is opt-in. A CSV persistence never pretends it can build select queries. The **DataSource** layer is progressive too. These traits aren't a strict hierarchy — they combine independently, and each persistence implements only what its engine supports: ```text DataSource ← marker, all backends ├── TableSource ← CRUD, columns, conditions ├── ExprDataSource ← execute expressions, defer() └── SelectableDataSource ← query builder (Selectable) TableExprSource ← count/sum/max/min as composable expressions requires: TableSource + ExprDataSource TableQuerySource ← table definition → full query requires: TableSource + SelectableDataSource ``` Here's what each persistence implements: ```text CSV └── TableSource (read-only, in-memory conditions) REST API (vantage-api-client) ├── TableSource (read-only, HTTP GET) └── ExprDataSource (basic resolve) MongoDB ├── TableSource (full CRUD, native bson::Document conditions) └── SelectableDataSource (field projection, $match pipeline) └── aggregates via native $sum/$max/$min pipeline Postgres / MySQL / SQLite (via sqlx) ├── TableSource (full CRUD) ├── ExprDataSource (parametric SQL execution) ├── SelectableDataSource (JOINs, CTEs, window functions) └── TableQuerySource SurrealDB ├── TableSource (full CRUD) ├── ExprDataSource (CBOR execution) ├── SelectableDataSource (SurrealQL builder) ├── TableExprSource (composable aggregates) ├── TableQuerySource └── vendor extensions (graph queries, RELATE, live queries) ``` There's also **MockTableSource** in `vantage-table` and **ImDataSource** in `vantage-dataset` — a fully in-memory persistence backed by `HashMap>>`. Great for tests and prototyping without any database. ```admonish tip title="Build your own persistence" Want to add your own? The [Adding a New Persistence](./new-persistence.md) guide walks through each trait step by step — from type system to full CRUD to multi-backend CLI. ``` --- ## Persistence-specific Type Systems The `vantage-types` crate is the foundation of 0.4. Each persistence defines its own type universe via the `vantage_type_system!` macro — no more funnelling everything through JSON: ```rust,ignore vantage_type_system!(AnySurrealType, SurrealDB); // CBOR-native types vantage_type_system!(AnyMongoType, MongoDB); // BSON-native types vantage_type_system!(AnySqliteType, SqliteDB); // CBOR with integer/real/text markers vantage_type_system!(AnyPostgresType, PostgresDB); // CBOR with full PG type coverage vantage_type_system!(AnyMysqlType, MysqlDB); // CBOR with MySQL type coverage ``` All SQL backends (SQLite, Postgres, MySQL) now use **CBOR** as their internal value representation — not JSON. This preserves type fidelity that JSON loses (integer vs float, binary blobs, precise decimals) while keeping the same `vantage_type_system!` ergonomics. ```admonish note title="SQL type conversions" Each SQL backend has detailed type conversion tables covering chrono types (DATE, TIME, TIMESTAMP), numeric types (DECIMAL, BIGINT, FLOAT), and cross-type coercion rules. See the [Type Conversions](./sql/type-conversions.md) reference for exact round-trip behaviour per column type and Rust type. ``` Strict conversions replace silent casting — `try_into()` is explicit and fallible: ```rust,ignore let val: i64 = surreal_value.try_into()?; // fails if it's actually a string let val: String = mongo_value.try_into()?; // fails if it's actually a number ``` These types flow through the entire framework, not just storage. **Columns** carry their type as a generic parameter (`Column`, `Column`) — surviving type-erasure via `original_type` so UI adapters can inspect what a column actually holds. **`AssociatedExpression<'a, DS, T, R>`** is parameterised by the value type `T`, so a count query on SurrealDB returns `AnySurrealType` while the same query on Postgres returns `AnyPostgresType`. **`AssociatedQueryable`** then converts that into your expected Rust type — all type-checked at compile time. ```rust,ignore // Column preserves type through erasure let price = Column::::new("price").with_flag(ColumnFlag::Indexed); let erased = Column::::from_column(price); assert_eq!(erased.get_type(), "i64"); // original type survives // AssociatedExpression carries the persistence's value type let count: AssociatedExpression<'_, SurrealDB, AnySurrealType, usize> = table.get_expr_count(); let n: usize = count.get().await?; // executes, converts AnySurrealType → usize ``` This means the full range of your persistence's native types is preserved end-to-end — from column definition through query building to result extraction — without narrowing down to JSON variants. --- ## Table — the interface you get for free Once a persistence implements `TableSource`, the framework hands you `Table` — a fully-featured abstraction over your data with columns, conditions, ordering, pagination, relationships, and aggregates. You don't build any of this yourself; it comes from `vantage-table`. `Table` auto-implements a wide range of traits from the DataSet and ValueSet hierarchies: `ReadableDataSet`, `WritableDataSet`, `InsertableDataSet`, `ActiveEntitySet`, `ReadableValueSet`, `WritableValueSet`, `InsertableValueSet`, and `ActiveRecordSet`. All of these come for free once your persistence implements `TableSource`. ```rust,ignore let products = Product::table(db) .with_condition(products["is_deleted"].eq(false)) .with_order(products["price"].desc()) .with_pagination(Pagination::ipp(25)); // ReadableDataSet, WritableDataSet, ActiveEntitySet — all auto-implemented for (id, product) in products.list().await? { /* ... */ } let count = products.get_count().await?; ``` Additionally, `table.select()` yields a **vendor-specific query builder** — `SurrealSelect`, `SqliteSelect`, `PostgresSelect` — giving you full access to the persistence's native query capabilities when you need to go beyond what the generic `Table` API offers. This dramatically simplifies implementing new persistences. You implement `TableSource` methods (read, write, aggregate) and get the entire `Table` API for free — conditions compose automatically, columns carry types, references traverse between tables, and `AnyTable` type-erasure just works. ```mermaid graph LR P[Select Persistence] M[Create Model Layer] B[Build Business Logic] E[Export via FFI or API] P ==> M ==> B ==> E style P fill:#4a7c59,color:#fff style M fill:#2d6a8f,color:#fff style B fill:#8f5a2d,color:#fff style E fill:#7c2d8f,color:#fff ``` Pick your persistence (SurrealDB, Postgres, CSV — or several). Define entities and tables in a shared model crate. Build business logic against `Table` and `DataSet` traits — persistence-agnostic and testable with mocks. Finally, expose your model to other languages via FFI (C ABI, PyO3, UniFFI, WASM) or through API layers (Axum, gRPC). The model stays in Rust; consumers don't need to know. Read more in [Model-Driven Architecture](./mda.md) and [Three Paths for Developers](./three-paths.md). --- ## ActiveEntity and ActiveRecord The `vantage-dataset` crate introduces two flavours of the active record pattern: **`ActiveEntity`** — wraps a typed entity. Derefs straight to your struct, tracks the ID, and saves back to any `WritableDataSet`: ```rust,ignore let mut user = users.get_entity(&id).await?.unwrap(); user.email = "new@example.com".to_string(); // modify via DerefMut user.save().await?; // persists the change ``` **`ActiveRecord`** — same idea, but schema-less. Works with `Record` instead of a concrete struct, perfect for dynamic tables or config-driven entities: ```rust,ignore let mut rec = table.get_value_record(&id).await?; rec["status"] = json!("active"); rec.save().await?; ``` Both auto-unlock via blanket impls — if your datasource implements `ReadableDataSet` + `WritableDataSet`, you get `ActiveEntitySet` for free. No extra code. ```admonish example title="Get-or-create pattern" The API is designed for real-world patterns like get-or-create: ~~~rust,ignore let mut user = users.get_entity(&id).await? .unwrap_or_else(|| users.new_entity(id, User::default())); user.name = "Alice".into(); user.save().await?; ~~~ ``` --- ## Record\ — persistence-native value bags `Record` is an `IndexMap` wrapper — but `V` is **not** `serde_json::Value`. It's your persistence's own type. A SurrealDB table returns `Record`, Postgres returns `Record`, MongoDB returns `Record`. This ensures the full range of your persistence's native types is preserved — not narrowed down to what JSON can represent. ```rust,ignore // Each persistence speaks its native types let record: Record = surreal_table.get_value(&id).await?; let record: Record = pg_table.get_value(&id).await?; let record: Record = mongo_table.get_value(&id).await?; // Type-safe extraction — respects the persistence's type boundaries let name: String = record["name"].try_get::().unwrap(); ``` `Record` is the common currency across the framework — `ReadableValueSet`, `WritableValueSet`, `ActiveRecord`, and entity conversion all work through it. JSON conversion only happens at the boundary when you need `AnyTable` type-erasure (via `from_table()`). --- ## CBOR everywhere The SurrealDB client switched from JSON to **binary CBOR**, improving serialization performance and type fidelity. All backends followed — Postgres, MySQL, SQLite. --- ## New persistences MongoDB, CSV, and REST API persistences joined SurrealDB and SQLite. CSV evaluates conditions in-memory. MongoDB uses native BSON filters — no expression translation needed: ```rust,ignore // Same handle_commands function — different persistences handle_commands(SurrealDB::table("product")).await?; handle_commands(CsvFile::table("products.csv")).await?; handle_commands(MongoDB::table("product")).await?; ``` --- ## Unified error handling `vantage-core` introduced `VantageError` with structured context — replacing the patchwork of `Box` and `.unwrap()` calls: ```rust,ignore use vantage_core::{error, util::error::Context, Result}; connection.connect() .with_context(|| error!("Failed to connect", dsn = &dsn, timeout = 30))?; ``` --- ## AnyTable goes universal `AnyTable::from_table()` now wraps **any** datasource whose types convert to/from JSON, using an internal `JsonAdapter` for on-the-fly conversion. This means your generic code works with persistences that don't even share a value type: ```rust,ignore // Wrap a MongoDB table for use in generic code let any: AnyTable = AnyTable::from_table(mongo_products); let any: AnyTable = AnyTable::from_table(csv_products); // Both work through the same interface ``` --- ## What's still coming ```admonish warning title="Work in progress" **Trait boundary refinements** — Aggregates (`get_count`, `get_sum`) currently require `SelectableDataSource` but should only need `TableSource`, so non-query backends like MongoDB can use them directly. `column_table_values_expr` forces an `ExprDataSource` dependency that document-oriented backends don't need. **SurrealDB reference traversal** — `IN` subqueries return record objects instead of scalar values. Needs `SELECT VALUE id` — a SurrealDB-specific construct not yet in the generic `Selectable` trait. **Type system gaps** — `Vec` (binary data) and `Uuid` need type trait implementations across backends. Bind/read paths exist — just missing the `impl XxxType` wiring. **Query builder** — `sql_fx!()` macro for mixed-type function calls, `Expression::empty()` sweep, PostgreSQL ingress scripts. **Architecture** — Transaction support, Table JOIN preserving conditions and resolving alias clashes, `Condition::or()` beyond two arguments, expression refactor (split Owned/Lazy). **Someday** — Table aggregations (GROUP BY), disjoint subtypes pattern, replayable idempotent operations, "Realworld" example application. ``` ```admonish success title="The 0.4 philosophy" Don't force every persistence into the same mould. Let each one implement what it can, carry its own types, and unlock API surface progressively. The framework adapts to the datasource — not the other way around. ``` --- Source: https://vantage-ui.com/book/whats-new-in-05.html # What's New in Vantage 0.5 ```admonish note title="Historical snippets" The snippets show the 0.5 API. They aren't compiled against the current crates. ``` Version 0.4 made the type system precise — every backend carries its own types, and a persistence implements only what its engine supports. Version 0.5 answers the question that follows: **how does code that doesn't know your entity type talk to your data?** A CLI that lists any table, a web admin that draws forms from a YAML schema, a UI data grid pointed at whatever you hand it — none of them know your `Product` struct. 0.4's answer was `AnyTable`: type-erasure funnelled through JSON. 0.5 replaces it with **`Vista`**, a universal, schema-bearing data handle, and builds an entire reactive stack on top of it. --- ## Vista — the universal data handle A [`Vista`](vantage_vista::Vista) wraps any typed `Table` and erases **both** the backend and the entity. All data flows through `Record` — an ordered map of string keys to CBOR values. All schema lives on the Vista itself: columns (with types and flags), references, the id column, and a set of capability flags. Execution is delegated to a per-driver [`TableShell`](vantage_vista::TableShell). ```text Table — typed entity, typed backend, compile-time safe Vista — fully erased: schema-bearing, CborValue, no generics ``` Each backend ships a factory that turns a typed table into a Vista. The factory harvests columns, id field, title fields, and references from the table definition you already built — no extra mapping code: ```rust,ignore use vantage_sql::prelude::*; use vantage_vista::Vista; let table = Product::table(db.clone()); let vista = SqliteVistaFactory::new(db).from_table(table)?; ``` For MongoDB it's `MongoVistaFactory`, for AWS `AwsVistaFactory` — same shape, different import. The resulting `Vista` is identical regardless of which factory produced it. From there, everything is runtime introspection: ```rust,ignore // Schema — no entity type required for name in vista.get_column_names() { let col = vista.get_column(name).unwrap(); println!("{}: {}", col.name, col.original_type); // name: String, price: i64, ... } for (name, kind) in vista.list_references() { println!("ref: {} ({:?})", name, kind); // ref: products (HasMany) } // Query — Vista delegates the value to the driver's native condition type let mut v = vista.clone(); v.add_condition_eq("category_id", 1.into())?; v.add_search("tart")?; // fans across SEARCHABLE columns v.add_order("price", SortDirection::Descending)?; let rows = v.fetch_page(1).await?; ``` ```admonish info title="Conditions mutate; Table builds" Unlike `Table`'s `.with_condition()` (consume-and-return), Vista's `add_condition_eq` / `add_search` / `add_order` mutate in place — it's a runtime handle, not a builder. Search and order are **replace semantics** (calling again drops the previous one). Clone before narrowing if you need the unfiltered handle later. ``` --- ## CBOR, not JSON Where `AnyTable` narrowed every value to `serde_json::Value`, Vista carries [`ciborium::Value`](https://docs.rs/ciborium) end to end. CBOR preserves the type fidelity JSON loses — integer vs float, binary blobs, precise decimals — so a `Record` round-trips through the Vista layer without lossy hops. JSON conversion happens only at the boundary, when you actually need it (an HTTP response, for example), and it's a one-liner. This is the same carrier the backends already use internally (Surreal and SQL store CBOR; Mongo stores BSON), so wrapping a typed table into a Vista no longer crosses a JSON funnel. --- ## Capabilities — the explicit contract Not every backend can do everything. A CSV file can't sort or search server-side; DynamoDB orders only by its declared sort key; a token-paginated REST API offers a forward cursor but no page numbers. Vista makes this explicit with [`VistaCapabilities`](vantage_vista::VistaCapabilities) — a struct of booleans where each driver declares exactly what it supports: ```rust,ignore let caps = vista.capabilities(); if caps.can_search { v.add_search("query")?; } if caps.can_fetch_page { let page = v.fetch_page(2).await?; // random access } else if caps.can_fetch_next { let (rows, token) = v.fetch_next(None).await?; // forward cursor } ``` ```admonish warning title="Unsupported is an error, not a no-op" The flags aren't suggestions — they're a contract. Calling `add_search()` when `can_search` is `false` returns an `Unsupported` error. If a flag is `true` but the driver forgot to implement the method, you get `Unimplemented` instead. Both are `VantageError` variants you can match on. It's better to fail clearly than to silently return an unfiltered result set — a principle 0.6 then drove through every remaining path. ``` UI adapters branch on these flags directly: a data grid checks `can_fetch_page` to decide between a scrollbar (random access) and a "load more" button (cursor-based). --- ## Config-driven: YAML specs and Rhai scripting A Vista no longer needs a hand-written Rust definition. Tables, columns, and relations can be declared in a **YAML spec** ([`VistaSpec`](vantage_vista::VistaSpec)) and materialized by the driver's factory: ```yaml table: product columns: - { name: name, flags: [title, searchable] } - { name: price } references: - { name: orders, kind: has_many, foreign_key: product_id } ``` For everything YAML can't express — vendor-specific expressions, derived sources, scripted reference traversal — there's an optional **Rhai** DSL that compiles to native queries. The same script renders dialect-correct SQL across all three SQL backends: ```rust,ignore let users = table("users").alias("u"); select() .from(users) .expression(users["name"]) .where(users["age"] >= 18) .order_by(users["name"], "asc") ``` - Automatic identifier quoting (backticks for MySQL, double quotes for Postgres/SQLite) - Dialect-aware primitives: `date_format()` → `strftime()` / `TO_CHAR()` / `DATE_FORMAT()` - `group_concat()` → `GROUP_CONCAT` (SQLite/MySQL) / `STRING_AGG` (Postgres) SurrealDB gets its own Rhai vocabulary — graph traversal (`graph()`/`recurse()`), record ids (`thing`), `$parent` references, and SurrealDB-namespaced aggregates. A per-reference Rhai script can even override the default foreign-key traversal, evaluated lazily with the parent `row` in scope. ```admonish note title="YAML primary, Rhai targeted" YAML stays the canonical, declarative table format. Rhai is a serializable escape hatch you reach for only when a relationship or source needs vendor expressions a YAML key can't represent. Engine-less backends still understand a *conventional* uniform vocabulary (`table`, `with_id`, `add_condition_eq`, `add_order`) and only lose vendor-specific expression syntax — graceful degradation, not a hard requirement. ``` --- ## Contained relations and nested writes Embedded objects and arrays — an order's `lines` array, a row's JSON column — now surface as a fully **editable sub-Vista**: ```rust,ignore let order = Order::table(db) .with_contained_many("lines", |line| { line.with_column("product_id").with_column("quantity") }); ``` SurrealDB backs contained relations with native nested objects and arrays; the SQL backends store them as JSON columns and patch the host column on writeback. Reads project the column into records; writes re-serialize the whole collection and patch the parent row. Contained records can even traverse out to real tables (`line.product`). Insert learned to walk relations, too. Hand `insert_value` a record whose keys name a **relation** instead of a column, and Vista sequences the writes so foreign keys populate automatically: a has-one child is inserted first and its id stamped into the parent's FK; has-many children are inserted after the parent with the parent's id stamped into each. Arbitrary depth, same-persistence relations only. ```admonish warning title="Best-effort, non-atomic" Nested insert is best-effort and non-atomic — a mid-sequence failure leaves earlier writes committed. Transaction support is still on the roadmap. ``` --- ## Diorama — caching, events, and reactive views The largest new subsystem is **`vantage-diorama`**, a layer that sits between a Vista and whatever consumes it. It does three things: caches transparently, injects capabilities the backend lacks, and routes writes wherever you want. ```mermaid graph LR M[Master Vista] --> D[Dio] L[Lens
cache + callbacks] --> D D --> F[Facade Vista] D --> S[Scenery] D -. cache + events .-> D style M fill:#4a7c59,color:#fff style L fill:#2d6a8f,color:#fff style D fill:#8f5a2d,color:#fff style S fill:#7c2d8f,color:#fff style F fill:#7c2d8f,color:#fff ``` Four words you'll see throughout: - **Vista** — a single-backend data source (the master). - **Lens** — long-lived shared infrastructure: cache backend, lifecycle callbacks, refresh policy. Built once per application. - **Dio** — a Vista bound to a Lens. Owns the cache table, a write queue, an event bus, and a refresh task. Produced by `lens.make_dio(vista)`. - **Scenery** — a reactive view onto a Dio (ordered tables, individual records, aggregates). The UI binds here. ```rust,ignore let lens = Arc::new( Lens::new() .cache_at("./cache.redb") .on_start(|dio| { let dio = dio.clone(); async move { let rows = dio.master().list_values().await?; // seed cache from master dio.cache().insert_values(rows).await?; Ok(()) }}) .refresh_every(Duration::from_secs(300)) .build()?, ); let products = lens.make_dio(products_vista).await?; let mut v = products.vista(); // facade: reads from cache, writes through the queue let rows = v.list_values().await?; // cache hit — never touches the master ``` ```admonish success title="Capability injection" Diorama caches the full dataset locally and answers queries from it. So a read-only CSV Vista that _can't_ paginate, sort, or search server-side becomes one that can — the consumer sees a richer Vista than the backend actually supports. Register an `on_write` callback and the facade gains `can_insert` too, even though the master is read-only: writes land in the queue and you route them wherever you like (a Kafka topic, a different database). ``` Writes go through a queue as `WriteOp`s; the cache updates immediately and persistence happens asynchronously. A `broadcast` event bus publishes `DioEvent`s (record changed, inserted, removed, invalidated) that Sceneries subscribe to. Upstream changes — another user's edit, a database trigger, a webhook — feed in as `ChangeEvent`s through an `on_event` callback that reconciles them into the cache. **Two-pass progressive loading** handles slow sources: a cheap *list pass* renders immediately, and expensive per-row *detail* hydrates lazily as rows scroll into view, keyed per query so filter/sort variants share the detail store without blocking the UI. --- ## Cross-persistence traversal: VistaCatalog A reference can cross a backend boundary — categories in Postgres, products in MongoDB. In 0.5 that's no longer a `Vista` concern (a `Vista` is strictly single-backend). It moved up into **`vantage-vista-factory`**'s `VistaCatalog`: register a model loader per table name, then `build_vista(name)` materializes a Vista and `traverse(relation, parent_row)` resolves and narrows the related Vista regardless of which persistence backs it. ```rust,ignore let catalog = VistaCatalog::new(); catalog.register("category", |name| postgres_factory.build(name)); catalog.register("product", |name| mongo_factory.build(name)); let category = catalog.build_vista("category").await?; let products = catalog.traverse("products", &category_row).await?; // Postgres → MongoDB ``` --- ## More backends 0.5 widened the roster around the Vista abstraction: | Crate | Backend | | -------------------- | --------------------------------------------------------------- | | `vantage-redb` | Embedded key-value store (uses `ColumnFlag::Indexed` for real indexes) | | `vantage-aws` | DynamoDB and friends — `Factory::for_name` / `from_arn` return Vistas directly | | `vantage-cmd` | A shell script becomes a queryable table (separate `list` / `detail` scripts) | | `vantage-api-pool` | Pooled REST API access | Each implements `TableShell` and slots into the same Vista surface, the same YAML/Rhai config, and the same Diorama caching — nothing downstream changes. --- ## Migrating off AnyTable `AnyTable` is gone. The carrier, `AnyRecord`, the `CborAdapter`, `Table::get_ref` (the `AnyTable`-returning one), `Reference::resolve_as_any`, and the `model_cli` runner were all deleted across the 0.5.x line. ```admonish tip title="The replacement, in one line" Anywhere you wrote `AnyTable::from_table(table)`, write `T::vista_factory().from_table(table)?` to get a `Vista`. The typed `Table::get_ref_as` and `Table::get_subquery_as` survive untouched — they never went through `AnyTable`. For the row-driven case, prefer `Table::get_ref_from_row`. ``` The CLI runner story collapsed to one path: `vista_cli` (in `vantage-cli-util`), which has carried the full token grammar — operators, selectors, search, aggregates — since 0.4.5 and is what every in-tree consumer already uses. --- ## What's still coming ```admonish warning title="Work in progress" **Transactions** — nested insert and multi-step writes are best-effort and non-atomic today. **Live queries** — `can_subscribe` is wired through the capability struct but SurrealDB live-query push is still a later pass; Diorama's event bus is the interim path. **Multi-column ordering** — `add_order` is single-column for now; the signature already accommodates the multi-column future. **Type system gaps** — `Vec` (binary) and `Uuid` still need trait wiring across every backend. ``` ```admonish success title="The 0.5 philosophy" Keep the typed layer where you know your entity, and rise to a single universal handle — `Vista` — where you don't. Let each backend advertise exactly what it supports, and let a caching layer fill the gaps. The framework adapts to the datasource; the consumer writes against one shape. ``` --- Source: https://vantage-ui.com/book/migration-2.0.html # Migrating to 2.0 2.0 rebuilds the interface of every layer above `Table`: `vantage-vista`, `vantage-diorama` and every driver's `TableShell`. Two crates are gone, about twenty shell methods are gone, the Lens has four settings instead of eight hooks, and a Dio no longer changes after it is built. This guide maps each removed or renamed item to its replacement, page by page. A typical port is mostly mechanical: fix the imports, rename the calls the compiler points at, then read the behaviour notes for the areas your code leans on. The behaviour notes matter more than the renames. A rename fails to compile; a changed default compiles and then does something else. ## What changed and why 1.x had grown several ways to do most things, and a few ways to do them silently wrong: a Dio narrowed in place changed what every other holder saw, a paged view dropped the conditions its loader didn't understand, a capability flag could claim a method the shell didn't implement. 2.0 sorts every public type into one of three kinds and gives each kind one way to change: | Kind | Types | Narrowed by | Changed by | |---|---|---|---| | Value | `Table`, `Vista`, `VistaQuery`, `Reference`, `ChangeFlash` | `clone().with_*()`, a new value | `add_*` / `set_*` on a value you own | | Shared handle | `Lens`, `Dio` | `with_*` or `derive()`, a new handle | never after its builder | | View state | `TableScenery`, `RecordScenery`, `ValueScenery`, `Servo` | a builder opens a new one | `set_*`, by its one owner | Eleven rules hold that shape in place. The ones a migration runs into most: - **One way to do each thing.** No `_counted`, `_ordered`, `_with` or `_sugared` variants; a variant is a builder option or a parameter. - **No silent defaults.** A trait method either does the work or returns an `Unsupported` error. - **Capabilities map one-to-one to methods**, and a contract test calls each advertised one. - **Narrowing is never in place on shared objects.** `&self` on a Dio never changes what another holder sees. - **`#[non_exhaustive]` on every public enum and data struct that may grow.** Matches gain a `_ =>` arm; structs are built through constructors. - **One path per item.** A type is reachable from its module or the crate root, never both, and a crate re-exports nothing from another crate outside its prelude. The full list is in the spec (`agents/specs/2026-10-11-vista-2.0/00-overview.md`). Each page below opens with the rule or decision that drove its changes. ## Crates and features | 1.x | 2.0 | |---|---| | `vantage-vista-factory` | deleted; `vantage_vista::Catalog` ([references, catalog and rhai](./migration-2.0/references-catalog-rhai.md)) | | `vantage-diorama-aggregate` | deleted; `vantage_diorama::aggregate` ([aggregates](./migration-2.0/aggregates.md)) | | `vantage_vista::mocks` always compiled | the `mocks` feature | | `vantage_table::mocks`, `MockTableSource` always compiled | the `mocks` feature | | `DebugTap`, `stats`, `Dio::diagnostics` always compiled | vantage-diorama's `diagnostics` feature | | `Dio::take_write_worker_handle` (`#[doc(hidden)]`) | vantage-diorama's `test-util` feature | Neither deleted crate has a shim. Drop both from `Cargo.toml`, and turn on the features the code reads: ```toml [dependencies.vantage-diorama] features = ["diagnostics"] # the debug stream, stats, Dio::diagnostics [dev-dependencies.vantage-vista] features = ["mocks"] # MockShell ``` ## How to read this guide | Page | Covers | |---|---| | [Vista](./migration-2.0/vista.md) | reads, conditions, order and search, capabilities, metadata, the `TableShell` trait | | [References, catalog and rhai](./migration-2.0/references-catalog-rhai.md) | `Reference` constructors, `Catalog`, resolvers, the rhai host API and script verbs | | [Dio and Lens](./migration-2.0/dio-lens.md) | the Lens builder, loaders and refreshers, building and deriving a Dio, cache identity | | [Writes and ingress](./migration-2.0/writes-ingress.md) | `Dio::flash`, the writer, awaited facade writes, confinement, `apply_change`, `reconcile` | | [Sceneries](./migration-2.0/sceneries.md) | table, record and value sceneries: builders, setters, owned views, statuses | | [Aggregates](./migration-2.0/aggregates.md) | `vantage_diorama::aggregate`, value sceneries, derived Dios, `Where` matching | | [Surface](./migration-2.0/surface.md) | preludes, private modules, `#[non_exhaustive]`, features | | [Drivers](./migration-2.0/drivers.md) | spec resolvers, `keys:` and column `references:`, quicksearch, per-driver notes | | [Downstream](./migration-2.0/downstream.md) | what the vantage-ui and vantage-actions port has to change | Each change shows a "before" and an "after". Every "before" block is compiled by the book's tests and must fail with the error code on its fence, so a "before" that started compiling again would fail the build. A rename with no behaviour behind it goes in a table instead. If you are porting an application, start with [Surface](./migration-2.0/surface.md): most first-round compile errors are import paths, and the rest of the guide reads more easily once they are gone. ### Why not keep the old name as an alias? Because an alias is a second way to do the thing, forever. Rule 1 exists because 1.x had collected exactly those: `fetch_page` beside `fetch_window`, `flash_patch` beside `flash`, `make_dio` beside `make_dio_as`. Each pair looked harmless on its own, and together they meant two code paths to keep correct, two spellings in every example, and a reader who couldn't tell which was preferred. The usual answer is a deprecation cycle: keep the old name for a release, warn, then remove it. That is the right tool inside a major version, and a deprecated `clone_shell` would have been the right call for a 1.x release. 2.0 is a breaking release on purpose, and its one chance to remove the second ways without another major version. The compiler error is the migration checklist: each removed name fails at a line, and this guide names the replacement. A shim would turn that error into a warning that gets ignored, and leave the old path in the API for as long as anyone still calls it. ## Checklist - [ ] `vantage-vista-factory` and `vantage-diorama-aggregate` are out of every `Cargo.toml` - [ ] the crates that use `MockShell`, `DebugTap`, stats or `Dio::diagnostics` enable `mocks` / `diagnostics` - [ ] the code compiles against the [Surface](./migration-2.0/surface.md) rules: no private module paths, no prelude imports of items that left it - [ ] each page's checklist is done for the areas the code uses - [ ] the behaviour notes on [Writes and ingress](./migration-2.0/writes-ingress.md) and [Sceneries](./migration-2.0/sceneries.md) are checked against the code that waits for a write or reads a view's state --- Source: https://vantage-ui.com/book/migration-2.0/vista.html # Vista `Vista` is the type-erased `Table`, and 2.0 makes it behave like one: it is `Clone`, owns its query, narrows with consuming `with_*` and changes in place only through `add_*` / `set_*` on a value you own. The `TableShell` behind it lost about twenty of its 44 methods. The ones that went were variants (`fetch_page` beside `fetch_window`), silent defaults (a `can_*` flag the shell never checked), or reads of the shell's own state that `metadata()` now answers. Rule 3 drove the capability change: each `Capability` names one method family, and a contract test calls every one a shell advertises. The chapter to read alongside this page is [Vista: the Universal Data Handle](../intro/step4-vista.md). The snippets below run against an in-memory `MockShell` named `teams`, with rows `a` (red) and `b` (blue). ## One read primitive `fetch_page`, `fetch_next`, `set_page_size` and `fetch_window_counted` fold into `fetch_window(offset, limit)`, which returns a `Page` carrying the rows and, when the shell knows it, the total. Cursor paging is `fetch_cursor(token, limit)`. ```rust,compile_fail,E0599 # use vantage_types::Record; # use vantage_vista::mocks::MockShell; # use vantage_vista::prelude::*; # use vantage_vista::{Column, VistaMetadata}; # fn teams() -> Vista { # let row = |n: &str| -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() }; # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let vista = teams(); let rows = vista.fetch_page(0, 10).await?; # Ok(()) # } ``` ```rust # use vantage_types::Record; # use vantage_vista::mocks::MockShell; # use vantage_vista::prelude::*; # use vantage_vista::{Column, VistaMetadata}; # fn teams() -> Vista { # let row = |n: &str| -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() }; # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let vista = teams(); let page = vista.fetch_window(0, 10).await?; let (rows, total) = (page.rows, page.total); # assert_eq!(rows.len(), 2); # let _ = total; # Ok(()) # } ``` Every shell whose `list` works must answer `fetch_window(0, n)`; `Window` now means "any offset", and `Vista` refuses a later offset without it before the shell is called. ## One condition method `add_eq_condition`, `add_op_condition` and `add_condition_eq` are one `add_condition(column, op, value)` on an owned Vista, or `with_condition` to narrow a copy. ```rust,compile_fail,E0599 # use vantage_types::Record; # use vantage_vista::mocks::MockShell; # use vantage_vista::prelude::*; # use vantage_vista::{Column, VistaMetadata}; # fn teams() -> Vista { # let row = |n: &str| -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() }; # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let mut vista = teams(); vista.add_eq_condition("name", "red")?; # Ok(()) # } ``` ```rust # use vantage_types::Record; # use vantage_vista::mocks::MockShell; # use vantage_vista::prelude::*; # use vantage_vista::{Column, VistaMetadata}; # fn teams() -> Vista { # let row = |n: &str| -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() }; # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let red = teams().with_condition("name", FilterOp::Eq, "red")?; # assert_eq!(red.list_values().await?.len(), 1); # Ok(()) # } ``` ## Search and order are set, not added `add_search` / `clear_search` and `add_order` / `clear_order` become `set_search(Option)` and `set_order(Option)`: `None` clears. A Vista holds one search and one order, so "add" described something the shells never did. ```rust,compile_fail,E0599 # use vantage_types::Record; # use vantage_vista::mocks::MockShell; # use vantage_vista::prelude::*; # use vantage_vista::{Column, VistaMetadata}; # fn teams() -> Vista { # let row = |n: &str| -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() }; # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let mut vista = teams(); vista.add_order("name", SortDirection::Ascending)?; # Ok(()) # } ``` ```rust # use vantage_types::Record; # use vantage_vista::mocks::MockShell; # use vantage_vista::prelude::*; # use vantage_vista::{Column, VistaMetadata}; # fn teams() -> Vista { # let row = |n: &str| -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() }; # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String").with_flag("orderable")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let mut vista = teams(); vista.set_order(Some(("name", SortDirection::Ascending)))?; vista.set_search(None)?; # assert!(vista.query().order.is_some()); # Ok(()) # } ``` ## A query in, a residual out `with_query` (strict) and `with_query_partial` are one `with_query(&query)`, which returns the narrowed Vista and the residual: the clauses the shell didn't take. Nothing is dropped silently; the caller evaluates the residual, or refuses it. ```rust,compile_fail,E0599 # use vantage_types::Record; # use vantage_vista::mocks::MockShell; # use vantage_vista::prelude::*; # use vantage_vista::{Column, VistaMetadata}; # fn teams() -> Vista { # let row = |n: &str| -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() }; # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let query = VistaQuery::new().with_condition(Condition::new("name", FilterOp::Eq, "red")); let (narrowed, residual) = teams().with_query_partial(&query)?; # Ok(()) # } ``` ```rust # use vantage_types::Record; # use vantage_vista::mocks::MockShell; # use vantage_vista::prelude::*; # use vantage_vista::{Column, VistaMetadata}; # fn teams() -> Vista { # let row = |n: &str| -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() }; # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let query = VistaQuery::new().with_condition(Condition::new("name", FilterOp::Eq, "red")); let (narrowed, residual) = teams().with_query(&query)?; # assert!(residual.is_empty()); # let _ = narrowed; # Ok(()) # } ``` ## Capabilities are a typed set The `can_*` flags become `vista.capabilities().has(Capability::…)`. `can_fetch_page`, `can_fetch_next` and `can_set_page_size` map to `Window` and `Cursor`; the traversal flags map to `Traverse`; `can_count` and `can_invalidate` have no replacement, because `count` always works and nothing invalidates a Vista. ```rust,compile_fail,E0599 # use vantage_types::Record; # use vantage_vista::mocks::MockShell; # use vantage_vista::prelude::*; # use vantage_vista::{Column, VistaMetadata}; # fn teams() -> Vista { # let row = |n: &str| -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() }; # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let paged = teams().can_fetch_page(); # Ok(()) # } ``` ```rust # use vantage_types::Record; # use vantage_vista::mocks::MockShell; # use vantage_vista::prelude::*; # use vantage_vista::{Column, VistaMetadata}; # fn teams() -> Vista { # let row = |n: &str| -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() }; # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let paged = teams().capabilities().has(Capability::Window); # assert!(paged); # Ok(()) # } ``` ## Renames | 1.x | 2.0 | |---|---| | `get_column_names()`, `get_column(n)`, `get_id_column()`, `get_title_columns()` | `columns().keys()`, `column(n)`, `id_column()`, `title_columns()` | | `get_references()`, `list_references()`, `get_reference(n)` | `references()`, `reference(n)` | | `driver()`, `has_auto_id()`, `can_watch()` | `driver_name()`, the id column's `ColumnFlag::Auto`, `capabilities().has(Capability::Subscribe)` | | `preview_query()` (JSON with `driver`) | `preview()` → `Preview { query, note }`; the driver is `driver_name()` | | `stream_vista_values`, `Vista::stream_values` | `list_values()` | | `get_vista_value_with_row`, `Vista::get_value_with_row` | `get_value(id)`; a row hint goes to an augmentation's `Fetch::Custom` | | shell `get_vista_some_value` | `fetch_window(0, 1)`; `Vista::get_some_value` reads exactly that | | shell `insert_vista_return_id_value` | `TableShell::insert(None, record)` | | `get_ref_target`, `get_contained_ref`, `contained()`, `get_ref_kinds`, `list_contained` | `get_ref(relation, Option<&row>)`; the kind is `reference(n)?.kind` | | `aggregate_vista`, `Vista::aggregate`, `AggregateSpec` | removed; see [Aggregates](./aggregates.md) | | `with_id(&mut self)`, `set_name` | `with_id(self) -> Result`, `with_name(self)` | | `index_key`, the `narrowed` flag, `pub source` | `query().key()`, `is_narrowed()`, `map_shell` | | `EmptyShell`, `Vista::empty` | `vantage_vista::mocks::MockShell` (the `mocks` feature) | | `Column.original_type: String` | `Option`; `Column::untyped(name)` has no type | | `AnyExpression`, `ExpressionLike` | `vantage-expressions` | ## For shell authors | 1.x `TableShell` | 2.0 | |---|---| | `clone_shell` | the `Clone` supertrait (`CloneTableShell`) | | `columns()`, `references()`, `id_column()` | one `metadata() -> &VistaMetadata` | | a `&Vista` parameter on shell methods | gone | | `add_raw_condition` | `add_native_condition(NativeCondition)` | | `add_search` + `clear_search`, `add_order` + `clear_orders` | `set_search(Option<&str>)`, `set_order(Option<(&str, SortDirection)>)` | | `capability_flag`, `default_error` | `capabilities()`, `unsupported(Capability::…)` | | `fetch_page`, `fetch_next`, slicing a listing | `fetch_window -> Page`, `fetch_cursor`, `Page::from_listing` ([Drivers](./drivers.md)) | | `Preview::new(driver)` | `Preview::new()` | | `get_ref` required | defaults to `unsupported(Capability::Traverse)` | | a refusal for a dotted column | `condition_column_error(&metadata, column)` | | struct literals of `VistaSpec`, `ColumnSpec`, `ReferenceSpec`, `ContainedYaml`, `NoExtras` | `VistaSpec::new(name)`, `ColumnSpec::new()`, `NoExtras::default()` | | `resolve_id_column()`, `resolve_id_column_or(d)`, `foreign_key_or(r)` | `resolve_id_column(default)`, `foreign_key(relation)` | | `VistaMetadata::with_column_at`, `ContainedShell`, `refuse_path` | `with_columns_at`; crate-private; `condition_column_error` | | `eval::path_get(row, "a.b")` | `eval::cell_get(row, "a.b", &metadata)` | | `contract::check_leading_window`, `Preview::default()` | run by `contract::check_advertised`; `Preview::new()` | ## Behaviour notes - **One in-memory evaluator.** `vantage_vista::eval` (`Condition::matches`, `VistaQuery::matches`, `VistaQuery::sort_rows`, `cmp_rows`, `cell_get`, `cbor_cmp`) is what `MockShell`, the facade and every local refinement use. Search is a case-insensitive substring of any cell's text, numbers, bools, tagged values and nested cells included; a blank needle matches every row. Null and absent cells sort last in both directions, ties break on id, and `3 == 3.0`. - **Dotted paths.** A condition on `root.leaf` that a shell can't evaluate is refused as `Unimplemented` ("this shell doesn't evaluate a dotted path"); `with_query` hands it back in the residual. Test `is_unimplemented()` rather than the message. An unknown column reads "Unknown column for condition" with a `field` key on every shell, and the contract's dotted-path probe fails a shell that answers anything but a narrowing or that refusal. - **`MockShell`** gained `calls()` (`MockCalls`, shared by clones), `get_only()`, `gated(..)`, `get_log()` and `max_concurrent_gets()`; replace hand-rolled counting and get-only test shells. - **`get_some_value` reads `fetch_window(0, 1)` only**, with no fallback to `list()`. - **A contained relation without a row**: `get_ref(rel, None)` returns the bare target, with no records and no write capabilities. - **Untyped columns.** A YAML column without `type:` is `Column::untyped`; rhai `columns()` reports its type as `()`, and the rhai `new_id` gives it a UUIDv7. A numeric id column the rhai insert can't fill (`u128`, `usize`, `Option`) now errors unless it is flagged `auto` or an id is passed. - **Quicksearch reads `Searchable` columns**, or every column when none is flagged. See [Drivers](./drivers.md#quicksearch-reads-the-searchable-columns). ## Checklist - [ ] no `fetch_page`, `fetch_next` or `set_page_size`; windows are `fetch_window` → `Page` - [ ] every condition is `add_condition` / `with_condition` with a `FilterOp` - [ ] `set_search` / `set_order` take an `Option`; `None` clears - [ ] `with_query` callers handle the residual it returns - [ ] no `can_*` flags; `capabilities().has(Capability::…)` - [ ] metadata reads use the short names in the table above - [ ] a custom `TableShell` implements `metadata()`, derives `Clone`, and refuses with `unsupported(..)` / `condition_column_error(..)` --- Source: https://vantage-ui.com/book/migration-2.0/references-catalog-rhai.html # References, catalog and rhai 1.x had five ways to name "something that gives me a Vista for a name": `TargetResolver`, `ModelResolver`, `ModelLoader`, `ContainedRefResolver` and each driver's `*SpecResolver` alias. It had two relation types, vantage-vista's `Reference` and vantage-vista-factory's `Relation`, and a crate whose only job was a registry of loaders. 2.0 keeps one of each, following rule 1: one `VistaResolver` trait, one `Reference`, and a `Catalog` inside vantage-vista. The factory crate is deleted with no shim (decision 6), so each import fails and points here. The rhai layer follows rule 11: its items live under `vantage_vista::rhai` only, and its script verbs match the Rust ones. Read [Relations: Vista, YAML and Rhai](../relations/vistas.md) and [Config-Driven Vistas](../config-driven-vistas.md) for the current shape. ## Reference constructors `Reference::new(name, target, kind, keys)` and `Reference::contained(name, kind, host)` become one constructor per kind. A join can't be built without a key, and a contained kind can't carry one. ```rust,compile_fail,E0599 # use vantage_vista::{JoinKey, Reference, ReferenceKind}; let players = Reference::new("players", "players", ReferenceKind::HasMany, vec![JoinKey::new("id", "team_id")]); ``` ```rust # use vantage_vista::{JoinKey, Reference, ReferenceKind}; let players = Reference::has_many("players", "players", JoinKey::new("id", "team_id")); let captain = Reference::has_one("captain", "players", JoinKey::to_target_id("captain_id")); let kit = Reference::contains_many("kit", "kit_items"); # assert!(players.kind == ReferenceKind::HasMany && captain.kind == ReferenceKind::HasOne); # assert!(kit.kind.is_contained()); ``` A multi-key join comes from YAML `keys:` through `ReferenceSpec::to_reference(relation, source_id)`; `with_key` is crate-private. ## The catalog folds into vantage-vista `VistaCatalog` becomes `vantage_vista::Catalog`. A loader receives the name it was asked for, and the catalog is a `VistaResolver`, so it goes anywhere a resolver is held. A name it doesn't hold is an `is_not_found()` error. ```rust,compile_fail,E0432 # use std::sync::Arc; # use vantage_types::Record; # use vantage_vista::mocks::MockShell; # use vantage_vista::prelude::*; # use vantage_vista::{Catalog, Column, VistaMetadata}; # fn teams() -> Vista { # let row = |n: &str| -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() }; # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # fn main() -> vantage_core::Result<()> { use vantage_vista_factory::VistaCatalog; let mut catalog = VistaCatalog::new(); catalog.register("teams", Arc::new(|| Ok(teams()))); let vista = catalog.build_vista("teams")?; # Ok(()) # } ``` ```rust # use std::sync::Arc; # use vantage_types::Record; # use vantage_vista::mocks::MockShell; # use vantage_vista::prelude::*; # use vantage_vista::{Catalog, Column, VistaMetadata}; # fn teams() -> Vista { # let row = |n: &str| -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() }; # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # fn main() -> vantage_core::Result<()> { let catalog = Catalog::new().with("teams", |_: &str| Ok(teams())); let vista = catalog.vista("teams")?; # assert!(catalog.vista("players").is_err()); # let _ = vista; # Ok(()) # } ``` A cross-model relation is a `Reference`, traversed as `reference.narrow(catalog.vista(&reference.target)?, &row)?`. [Config-Driven Vistas](../config-driven-vistas.md#cross-persistence-the-catalog) has the full example. ## One resolver trait `TargetResolver` (`Arc Result>`) and api-client's `ModelResolver` become `Arc`. A closure still implements the trait when its parameter is typed `&str`; a call `resolver(name)` becomes `resolver.vista(name)`. ```rust,compile_fail,E0432 # use std::sync::Arc; # use vantage_types::Record; # use vantage_vista::mocks::MockShell; # use vantage_vista::prelude::*; # use vantage_vista::{Catalog, Column, VistaMetadata}; # fn teams() -> Vista { # let row = |n: &str| -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() }; # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # fn main() -> vantage_core::Result<()> { use vantage_vista::rhai::TargetResolver; let resolver: TargetResolver = Arc::new(|_: &str| Ok(teams())); let vista = resolver("teams")?; # Ok(()) # } ``` ```rust # use std::sync::Arc; # use vantage_types::Record; # use vantage_vista::mocks::MockShell; # use vantage_vista::prelude::*; # use vantage_vista::{Catalog, Column, VistaMetadata}; # fn teams() -> Vista { # let row = |n: &str| -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() }; # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # fn main() -> vantage_core::Result<()> { let resolver: Arc = Arc::new(Catalog::new().with("teams", |_: &str| Ok(teams()))); let vista = resolver.vista("teams")?; # let _ = vista; # Ok(()) # } ``` ## The rhai host API The rhai items leave the crate root for `vantage_vista::rhai`, and `DataVocab`'s four constructors become `DataVocab::new(terminals).with_resolver(resolver)`. `Terminals` is `Describe` or `ReadWrite`; a read-only host is `ReadWrite` with `Writes::Denied(message)`, so every write throws that message. ```rust,compile_fail,E0432 # use std::sync::Arc; # use vantage_vista::mocks::MockShell; # use vantage_vista::prelude::*; # use vantage_vista::Catalog; # fn main() { # let resolver: Arc = Arc::new(Catalog::new().with("teams", |_: &str| Ok(Vista::new("teams", MockShell::new())))); use vantage_vista::{DataVocab, Terminals, Writes}; let vocab = DataVocab::read_write(Some(resolver)); # } ``` ```rust # use std::sync::Arc; # use vantage_vista::mocks::MockShell; # use vantage_vista::prelude::*; # use vantage_vista::Catalog; # fn main() { # let resolver: Arc = Arc::new(Catalog::new().with("teams", |_: &str| Ok(Vista::new("teams", MockShell::new())))); use vantage_vista::rhai::{DataVocab, Terminals, Writes}; let vocab = DataVocab::new(Terminals::ReadWrite { limit: None, writes: Writes::Allowed }) .with_resolver(resolver.clone()); let read_only = DataVocab::new(Terminals::ReadWrite { limit: Some(100), writes: Writes::Denied("this host is read-only".into()), }) .with_resolver(resolver); # let _ = (vocab, read_only); # } ``` | 1.x | 2.0 | |---|---| | `vantage_vista::{DataVocab, Handle, RecordDraft, run_script, …}` | `vantage_vista::rhai::{…}` | | `vantage_diorama::rhai::{cbor_to_dynamic, dynamic_to_cbor, record_to_map}` | `vantage_vista::rhai::{…}` | | `DataVocab::describe(r)`, `read(r, l)`, `read_write(r)`, `read_write_with(r, l, w)` | `DataVocab::new(Terminals::Describe \| ReadWrite { limit, writes }).with_resolver(r)` | | `Terminals::Read { limit }` | `Terminals::ReadWrite { limit, writes: Writes::Denied(msg) }` | | `Handle::over`, `steps`, `Step` | crate-private; `Handle::query() -> Result` | | `Handle::with_base_vista(verb, FnOnce(&mut Vista))` | `Handle::with_native_condition(verb, NativeCondition)` | | `Handle::resolve(Option<&TargetResolver>)`, `run_script(s, TargetResolver, ..)`, `preview_script` | `resolve(Option<&dyn VistaResolver>)`; the runners take `Arc` | | `RecordDraft::{new_row, load, from_row}(handle, resolver, ..)` | the resolver rides on the handle: [Drivers](./drivers.md#the-resolver-rides-on-the-handle) | | `eval_augment_source`, `AugmentSourceFn`, `augment_source_closure`, `eval_lazy_expression`, `lazy_value_closure`, `LazyValueFn` | removed; `Column::with_expression`, or your own `Source::Build` closure | | `vantage_rhai::register_constant` | `vantage_rhai::Namespace` (`ns.var(..)`, read as `ns::name`) | | `reject_dropped_narrowing(ast, what)` | `reject_dropped_narrowing(ast, what, has_value)` | ## Script changes Scripts change in four places: the record verb, statement chaining, capability keys, and the backend query builders, which moved into namespaces so they can't shadow `table(name)`. ```rhai let o = table("order").record("o2"); o.status = "paid"; o.save(); if table("order").capabilities().can_insert { /* … */ } self.where("status", "open"); self.sort("total", "desc"); select().from("product").field(ident("name")) ``` ```rhai let o = table("order").record("o2"); o.status = "paid"; o.flash(); if table("order").capabilities().insert { /* … */ } self.where("status", "open").sort("total", "desc") sql::select().from("product").field(sql::ident("name")) ``` - **`save()` is `flash()`** on rhai records and on the servo (decision 5): it sends an insert, or a patch of the changed fields only. `save` is "Function not found". - **A handle script's value is its last statement.** In `modify:`, augment and reference scripts, a statement that narrows a handle and drops it is an error when the script compiles; chain the calls or assign them. Watch `if c { self.where(..) }` with no `else`: end it with `else { self }`. - **`capabilities()` keys come from `Capability::iter()`** (`insert`, `order`, `window`, `filter_ops`, …). The dropped `can_count`, `can_fetch_page`, `can_fetch_next`, `can_set_page_size`, `can_invalidate`, `can_traverse_*` and `can_build_ref_via_script` read as `()`, so an old script doesn't error; it reads unit. `count()` is always available. - **Namespaced builders.** SQL functions are `sql::` for SQLite, PostgreSQL and MySQL; SurrealDB's are `surreal::`, and its `me` anchor is `surreal::me`; MongoDB's `oid`, `date`, `doc` are `mongo::`. The `table("x")` alias for `ident` is gone: write `sql::ident("x")`. Methods, indexers, operators and `with_condition` are unchanged. - **`references()` reports the YAML kind names** (`has_one`, `has_many`, `contains_one`, `contains_many`), and `rejection()` is `#{ message }`. - **Writes are gated by the Vista.** A denied or unsupported write throws (`delete` used to return `false`); `upsert` needs `Update` only; `import_from` needs `Import`; `limit(n)` reads `fetch_window(0, n)` with no fallback to a truncated `list()`. ## Renames | 1.x | 2.0 | |---|---| | `Reference::contained(name, ReferenceKind::ContainsOne, host)` | `Reference::contains_one(name, host)` (`contains_many` likewise) | | `reference.narrow(target, row, coerce)` | `reference.narrow(target, row)`; a driver re-tags ids in its own `add_condition` | | `Reference::foreign_key()`, `ReferenceKind::default()` | read `reference.keys`; name the kind | | `with_build_script(s) -> Self` | `-> Result`; refused on a contained kind | | `ContainedSpec`, `ContainedKind`, the `ContainedYaml` lowering | `Reference` with a `Contains*` kind | | `ContainedRefResolver` (keyed by relation name) | `build_contained_vista(.., Option>)`, called with `reference.target` | | `ModelLoader`, `VistaCatalog::register` | a `Catalog::with(name, loader)` loader | | `VistaCatalog::{build_vista, has_model, model_names}` | `catalog.vista(name)` (`is_not_found()` for a missing name) | | `Relation`, `Relation::{single_key, multi_key, narrow}`, `traverse`, `traverse_from` | `Reference`, `reference.narrow(catalog.vista(&reference.target)?, &row)`, or `parent.get_ref(relation, Some(&row))` on one backend | | `vantage_diorama::prelude::VistaCatalog` | `vantage_vista::Catalog` | | each shell's "reference not found" error | `vantage_vista::unknown_relation(relation)`, `is_not_found()` | ## Behaviour notes - **A serialised `Reference` is validated**: a contained kind needs `host_column` and no `keys` or build script; a join needs at least one key and no `host_column`. - **`Reference.target` is a name a catalog holds**: the spec name for YAML references, the built table's name for typed ones (it used to be a Rust type name), so `catalog.vista(&r.target)` works on every driver. - **A contained record's resolver is called with `reference.target`** (`products`), not the relation name (`product`), so a `Catalog` can be passed as it is. - **Strict joins.** Traversal narrows the target by the join's `to` column, so the target must declare it. See [Drivers](./drivers.md#declare-the-foreign-key-on-the-target). ## Checklist - [ ] no `vantage_vista_factory` import; `Catalog::new().with(..)` and `catalog.vista(name)` - [ ] every `Reference::new` / `Reference::contained` is a per-kind constructor - [ ] resolvers are `Arc`; closures type their parameter `|name: &str|` - [ ] rhai items are imported from `vantage_vista::rhai`; `DataVocab::new(..).with_resolver(..)` - [ ] scripts use `flash()`, chain their handle calls, read `capabilities()` by capability name, and call `sql::`, `surreal::` and `mongo::` builders --- Source: https://vantage-ui.com/book/migration-2.0/dio-lens.html # Dio and Lens In 1.x a Lens had eight hooks, each handed a `&Dio`, and a Dio could be narrowed, re-pointed and augmented after it was built, by anyone holding it. Rule 5 ended that: a Dio is a shared handle, so `&self` never changes what another holder sees. Everything a Dio is (its master, its augmentations, its live follow) is chosen on a builder before `build()`, and narrowing derives a new Dio with its own cache identity. The Lens went the same way: four typed settings (`Loader`, `Refresher`, `writer`, `Tuning`), and a `build()` that checks the combination and returns `Result>`. One query type, `VistaQuery`, flows through every load path, which is what fixed paged loads that ignored conditions. The chapter to read alongside is [A Facade over a Custom API](../intro/step5-dio-lens.md). The Lens snippets use `copy`, an `async fn copy(cx: LoaderCx) -> Result<()>` that lists `cx.master()` and hands the rows to `cx.reconcile`. ## The Lens builder `Lens::new()` and `LensBuilder::new()` become `Lens::builder()`, and `build()` returns the Lens already in an `Arc`. The hooks become typed settings: `on_start` is `Loader::eager`, `on_load_chunk` is `Loader::paged`, `on_refresh` with `refresh_every` is `Refresher::new(..).every(..)`. Every callback takes an owned `LoaderCx` instead of `&Dio`, so the `let dio = dio.clone()` dance goes. ```rust,compile_fail,E0599 # use std::{sync::Arc, time::Duration}; # use vantage_diorama::{prelude::*, Cache, Loader, LoaderCx, Refresher}; # use vantage_vista::prelude::*; # async fn copy(cx: LoaderCx) -> vantage_core::Result<()> { let rows = cx.master().list_values().await?; cx.reconcile(rows).await } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let lens = Arc::new( Lens::new() .cache_in_memory() .on_start(|dio: &Dio| { let dio = dio.clone(); async move { dio.reconcile(dio.master().list_values().await?).await } }) .refresh_every(Duration::from_secs(60)) .build()?, ); # Ok(()) # } ``` ```rust # use std::{sync::Arc, time::Duration}; # use vantage_diorama::{prelude::*, Cache, Loader, LoaderCx, Refresher}; # use vantage_vista::prelude::*; # async fn copy(cx: LoaderCx) -> vantage_core::Result<()> { let rows = cx.master().list_values().await?; cx.reconcile(rows).await } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let lens: Arc = Lens::builder() .cache(Cache::memory()) .loader(Loader::eager(copy)) .refresher(Refresher::new(copy).every(Duration::from_secs(60))) .build()?; # Ok(()) # } ``` `build()` refuses a combination that would fail later, and names the setting in the error's `setting` context: no cache, no tokio runtime, `Loader::blocking` without an eager start, `or_eager` on a non-paged loader, `every(Duration::ZERO)`, a write queue or worker count of zero, a redb file that won't open. `LensBuildError` is gone; match the context instead. ## Building a Dio `make_dio(vista)` and `make_dio_as(vista, name)` become `lens.dio(vista)`, a builder, then `.build().await?`. There is no name argument: the cache identity comes from the vista (below). `build()` fails on a vista without an id column, naming the table, where 1.x fell back to `"id"`. ```rust,compile_fail,E0599 # use vantage_diorama::{prelude::*, Cache, Loader, LoaderCx}; # use vantage_types::Record; # use vantage_vista::{mocks::MockShell, prelude::*, Column, VistaMetadata}; # fn teams() -> Vista { # let row = |n: &str| -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() }; # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # async fn copy(cx: LoaderCx) -> vantage_core::Result<()> { let rows = cx.master().list_values().await?; cx.reconcile(rows).await } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let lens = Lens::builder().cache(Cache::memory()).loader(Loader::eager(copy)).build()?; let dio = lens.make_dio(teams()).await?; # Ok(()) # } ``` ```rust # use vantage_diorama::{prelude::*, Cache, Loader, LoaderCx}; # use vantage_types::Record; # use vantage_vista::{mocks::MockShell, prelude::*, Column, VistaMetadata}; # fn teams() -> Vista { # let row = |n: &str| -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() }; # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # async fn copy(cx: LoaderCx) -> vantage_core::Result<()> { let rows = cx.master().list_values().await?; cx.reconcile(rows).await } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let lens = Lens::builder().cache(Cache::memory()).loader(Loader::eager(copy)).build()?; let dio = lens.dio(teams()).live().build().await?; # assert_eq!(dio.cached().list_values().await?.len(), 2); # Ok(()) # } ``` `.live()` replaces `dio.watch()`: the follow starts at build, a subscribe failure fails the build, and over a master without `Subscribe` it does nothing. While a Dio with the same cache identity, augmentations (the same `Arc`) and `live` setting is held, `build()` returns that Dio. ## Augmentation goes on the builder `Dio::augment(resolver, augmentations)` after construction becomes `DioBuilder::augment(..)`, and the resolver moves into each detail: `Detail::Catalog(name)` is `Detail::Resolved { resolver, name }`. A `Fetch::Custom` closure now receives `(Vista, Record)`: the narrowed detail and a copy of the listed row. ```rust,compile_fail,E0599 # use std::sync::Arc; # use vantage_diorama::{prelude::*, Augmentation, Cache, Detail, Fetch, MergeRule, Source}; # use vantage_types::Record; # use vantage_vista::{mocks::MockShell, prelude::*, Catalog, Column, VistaMetadata}; # fn teams() -> Vista { # let row = |n: &str| -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() }; # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let lens = Lens::builder().cache(Cache::memory()).build()?; let labels: Arc = Arc::new(Catalog::new().with("labels", |_: &str| Ok(teams()))); let label = Augmentation::new(Detail::Catalog("labels".to_string()), Source::Id, Fetch::PerRow, MergeRule::lift(["name"])); let dio = lens.dio(teams()).build().await?; dio.augment(labels, vec![label]).await?; # Ok(()) # } ``` ```rust # use std::sync::Arc; # use vantage_diorama::{prelude::*, Augmentation, Cache, Detail, Fetch, MergeRule, Source}; # use vantage_types::Record; # use vantage_vista::{mocks::MockShell, prelude::*, Catalog, Column, VistaMetadata}; # fn teams() -> Vista { # let row = |n: &str| -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() }; # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let lens = Lens::builder().cache(Cache::memory()).build()?; let labels: Arc = Arc::new(Catalog::new().with("labels", |_: &str| Ok(teams()))); let label = Augmentation::new( Detail::Resolved { resolver: labels, name: "labels".into() }, Source::Id, Fetch::PerRow, MergeRule::lift(["name"]), ); let dio = lens.dio(teams()).augment(vec![label]).build().await?; # let _ = dio; # Ok(()) # } ``` ## Narrowing derives a new Dio `with_condition_eq` and the mutating `with_order` changed every holder's Dio. `dio.derive()` builds a new one and leaves the original alone; clauses the master refuses, and clauses on augmented columns, become the derived Dio's `residual()`. The phase-3 `Dio::with_query(&q)` is `dio.derive().with_query(&q)`. ```rust,compile_fail,E0599 # use vantage_diorama::{prelude::*, Cache, Loader, LoaderCx}; # use vantage_types::Record; # use vantage_vista::{mocks::MockShell, prelude::*, Column, VistaMetadata}; # fn teams() -> Vista { # let row = |n: &str| -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() }; # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # async fn copy(cx: LoaderCx) -> vantage_core::Result<()> { let rows = cx.master().list_values().await?; cx.reconcile(rows).await } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let dio = Lens::builder().cache(Cache::memory()).loader(Loader::eager(copy)).build()?.dio(teams()).build().await?; dio.with_condition_eq("name", "red").await?; # Ok(()) # } ``` ```rust # use vantage_diorama::{prelude::*, Cache, Loader, LoaderCx}; # use vantage_types::Record; # use vantage_vista::{mocks::MockShell, prelude::*, Column, VistaMetadata}; # fn teams() -> Vista { # let row = |n: &str| -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() }; # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # async fn copy(cx: LoaderCx) -> vantage_core::Result<()> { let rows = cx.master().list_values().await?; cx.reconcile(rows).await } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let dio = Lens::builder().cache(Cache::memory()).loader(Loader::eager(copy)).build()?.dio(teams()).build().await?; let red = dio .derive() .with_condition(Condition::new("name", FilterOp::Eq, "red")) .build() .await?; assert_eq!(dio.diagnostics().await?.cache_table, "teams"); assert!(red.diagnostics().await?.cache_table.starts_with("teams?")); # Ok(()) # } ``` The derived Dio's cache table is derived too: `name` when nothing narrows the vista, else `name?`, with `%`, `?` and control characters in the name percent-encoded. That replaces `make_dio_as` names, `get_ref`'s `{target}-via-{relation}-{id}` tables and hand-built `-where-` keys. A vista narrowed before `lens.dio(..)` gets its own table as well. Old cache files keep their old tables, which nothing opens again; sweep them with `CacheBackend::list_tables` and `drop_table` if the space matters. ## Reading through the Dio `fetch_window_ordered(offset, limit, query)` and its `_counted` twin become `dio.fetch(window, &query)`, returning `Fetched { page, refused }`. A clause the master refuses comes back in `refused` instead of a `tracing::warn!` and a silent drop. ```rust,compile_fail,E0599 # use vantage_diorama::{prelude::*, Cache, Loader, LoaderCx}; # use vantage_types::Record; # use vantage_vista::{mocks::MockShell, prelude::*, Column, VistaMetadata}; # fn teams() -> Vista { # let row = |n: &str| -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() }; # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # async fn copy(cx: LoaderCx) -> vantage_core::Result<()> { let rows = cx.master().list_values().await?; cx.reconcile(rows).await } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let dio = Lens::builder().cache(Cache::memory()).loader(Loader::eager(copy)).build()?.dio(teams()).build().await?; let (rows, total) = dio.fetch_window_ordered_counted(0, 10, &VistaQuery::new()).await?; # Ok(()) # } ``` ```rust # use vantage_diorama::{prelude::*, Cache, Loader, LoaderCx}; # use vantage_types::Record; # use vantage_vista::{mocks::MockShell, prelude::*, Column, VistaMetadata}; # fn teams() -> Vista { # let row = |n: &str| -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() }; # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # async fn copy(cx: LoaderCx) -> vantage_core::Result<()> { let rows = cx.master().list_values().await?; cx.reconcile(rows).await } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let dio = Lens::builder().cache(Cache::memory()).loader(Loader::eager(copy)).build()?.dio(teams()).build().await?; let fetched = dio.fetch(0..10, &VistaQuery::new()).await?; let (rows, total) = (fetched.page.rows, fetched.page.total); if !fetched.refused.is_empty() { // evaluate the refused clauses over a complete set, or refuse the read } # assert_eq!(rows.len(), 2); # let _ = total; # Ok(()) # } ``` ## Renames | 1.x | 2.0 | |---|---| | `cache_at(path)`, `cache_in_memory()`, `cache_source(b)` | `.cache(Cache::redb(path))`, `.cache(Cache::memory())`, `.cache(Cache::backend(b))` | | `on_start(f)` + `on_start_blocking(b)`, `on_load_chunk(f)`, both | `Loader::eager(f).blocking(b)`, `Loader::paged(f)`, `Loader::paged(chunk).or_eager(start)` | | `on_refresh(f)` + `refresh_every(d)`, `cache_ttl` | `Refresher::new(f).every(d)`, `Refresher::max_age(ttl)` | | `on_flash(f)` | `.writer(f)`: [Writes and ingress](./writes-ingress.md) | | `write_queue_capacity`, `viewport_debounce`, `augment_workers`, `refresh_on_open`, `LensDefaults` | `.tuning(Tuning::default().with_write_queue(n).with_viewport_debounce(d)…)` | | `on_event`, `Dio::handle_event` | `Dio::apply_change` | | `total_provider` | the paged loader calls `ChunkSink::set_total` | | `on_list_page`, `on_load_detail`, `QueryDescriptor` | an `Augmentation` on `lens.dio(v).augment(..)` | | `activity_signal`, `Activity`, `ActivitySignal`, `standby_refresh_every` | removed (decision 3) | | `ChunkQuery` handed to a chunk loader | the view's `VistaQuery`, by value | | `Dio::vista()` | `Dio::cached()`, the facade Vista | | `Dio::cache()`, `Dio::cache_table_name()` | `LoaderCx::cache()` / `reconcile()` in callbacks; `dio.diagnostics().await?.cache_table` | | `Dio::master() -> Arc` | `-> &Vista`; also `residual()` and `id_column()` | | `Dio::reload(vista)` | build a new Dio over the new vista | | `LoaderCx::id_column()` | `cx.master().id_column()`, which `build()` guarantees | | `Augmentation::augment_row(.., &VistaCatalog)` | `augment_row(dio_name, id_column, row)`, no resolver | | `Diorama`, `composition`, `master_aggregate`, `HotTier`, `dio::refresh`, `DioError` | removed; none had a working use | | `LensBuilder: Default` | `Lens::builder()` is the one entry point | | `LensCallbacks`, `SceneryChunkTarget`, `ChunkWrite`, `ChunkRow`, `FlushReport`, the `Dio*Callback` aliases | crate-private | ## Behaviour notes - **A paged loader is never called with a query it can't answer.** A paged view whose query leaves a clause the master doesn't take (a refused condition, search or order, one on an augmented column, the Dio's residual) is served from one whole copy made through the loader's `or_eager` start; without one the view fails. A grid over a REST source without `Order` or `Search` needs `Loader::paged(..).or_eager(..)`, and a `Refresher` to keep that copy fresh. The `or_eager` start runs only for a master without `Window` (it used to run for every Dio). - **`refresh()` re-reads the way the set was loaded**: a paged Dio announces `DatasetChanged`, an eager one re-runs the start, any other reconciles with `master.list()`. An augmented Dio runs its augmentation pass and then the refresher. There is no timer without `Refresher::every`, and `max_age` without `every` checks staleness at build and at open. - **A live follow holds each pushed row against the Dio's query**: a row that left the set applies as a delete, and over native conditions every change re-reads the set. A follow that lags or loses its subscription re-reads, retrying with backoff; a failed re-read is `DioEvent::LoadFailed { range: 0..0, .. }`, and a failed refresh is `DioEvent::RefreshFailed`. - **`get_ref` on a row without its join column is an error**; a `get_ref` target inherits `live` but not augmentation. - **A clause on a dotted path under an augmented column** (`detail.size`) stays in the residual, where 1.x offered it to the master. A Dio derived with an order its master refused keeps it as the residual's order: `fetch` names it in `refused`, and facade reads sort by it. - **The facade advertises `Search` and `FilterOps`** and evaluates what the master doesn't take. Every facade read keeps only rows the residual admits; an order on its own reads the cache and sorts it, and only a condition or search the master takes sends the read to the master. `preview().note` names what is applied over the cache. ## Checklist - [ ] every Lens is `Lens::builder()….build()?`, with no `Arc::new` around it - [ ] hooks are `Loader`, `Refresher`, `writer` and `Tuning`, each callback taking `LoaderCx` - [ ] every `make_dio` call is `lens.dio(v).build().await?`; `.augment(..)` and `.live()` are on the builder - [ ] no Dio is narrowed in place: `dio.derive()…build().await?` - [ ] `dio.vista()` is `dio.cached()`; window reads are `dio.fetch(range, &query)` and check `refused` - [ ] paged lenses over masters that refuse sort or search have an `or_eager` start --- Source: https://vantage-ui.com/book/migration-2.0/writes-ingress.html # Writes and ingress 1.x wrote through a family of `flash_*` methods, a facade that returned before the write landed (reporting a failure only as an event), and an id-less insert that skipped the Lens writer. Changes from outside came in through `patched`, `notify_record_changed`, `reconcile_values` and `handle_event`. Rule 1 cut both sides to one call each. Outbound, every write is a `ChangeFlash` handed to `Dio::flash`, which joins the Dio's write queue and returns once the write is confirmed or rolled back (decision 5: the outbound half of the photography lexicon). Inbound, one change is `apply_change(ChangeEvent)` and a full set is `reconcile(snapshot)`. Read [A Facade over a Custom API](../intro/step5-dio-lens.md#writes-on-a-read-only-master) and [Live Data](../live-data.md#changes-you-feed-in-yourself) for the current flow. The snippets run over a Dio of `teams` (rows `a` red, `b` blue) whose eager loader copies the master; `row(name)` builds a one-column record. ## One write call `flash_insert`, `flash_patch`, `flash_replace`, `flash_delete` and `flash_returning_id` become `dio.flash(ChangeFlash::…)`, which returns a `FlashOutcome`. `ChangeFlash::new` is private; each kind has its constructor (`insert`, `insert_new`, `partial`, `replace`, `upsert`, `delete`, `clear`), so a kind always comes with an id the queue can act on. ```rust,compile_fail,E0599 # use vantage_diorama::{prelude::*, Cache, DioEvent, FlashOutcome, Loader, LoaderCx}; # use vantage_types::Record; # use vantage_vista::{mocks::MockShell, prelude::*, Column, VistaMetadata}; # fn row(n: &str) -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() } # fn teams() -> Vista { # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # async fn copy(cx: LoaderCx) -> vantage_core::Result<()> { let rows = cx.master().list_values().await?; cx.reconcile(rows).await } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let dio = Lens::builder().cache(Cache::memory()).loader(Loader::eager(copy)).build()?.dio(teams()).build().await?; dio.flash_patch("a", row("green")).await?; let id = dio.insert_returning_id(row("white")).await?; # Ok(()) # } ``` ```rust # use vantage_diorama::{prelude::*, Cache, DioEvent, FlashOutcome, Loader, LoaderCx}; # use vantage_types::Record; # use vantage_vista::{mocks::MockShell, prelude::*, Column, VistaMetadata}; # fn row(n: &str) -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() } # fn teams() -> Vista { # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # async fn copy(cx: LoaderCx) -> vantage_core::Result<()> { let rows = cx.master().list_values().await?; cx.reconcile(rows).await } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let dio = Lens::builder().cache(Cache::memory()).loader(Loader::eager(copy)).build()?.dio(teams()).build().await?; dio.flash(ChangeFlash::partial("a", row("green"))).await?; let id = dio.flash(ChangeFlash::insert_new(row("white"))).await?.id; # let a = dio.cached().get_value("a").await?.and_then(|r| r.get("name").cloned()); # assert_eq!(a, Some(CborValue::Text("green".into()))); # assert!(id.is_some()); # Ok(()) # } ``` `insert_new` replaces the crate-private `insert_returning_id`: the id-less insert now goes through the queue and through the Lens writer when there is one, and `FlashOutcome::id` is the id the row settled under. ## The writer returns an outcome `on_flash` becomes `.writer(f)`. The callback takes `(LoaderCx, ChangeFlash)` and returns `Result`: `FlashOutcome::applied()` confirms the write, `.with_id(id)` reports the id an id-less insert got (and re-keys the staged row onto it), `.with_record(r)` hands back the stored record. ```rust,compile_fail,E0599 # use vantage_diorama::{prelude::*, Cache, DioEvent, FlashOutcome, Loader, LoaderCx}; # use vantage_types::Record; # use vantage_vista::{mocks::MockShell, prelude::*, Column, VistaMetadata}; # fn row(n: &str) -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() } # fn teams() -> Vista { # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let lens = Lens::builder() .cache(Cache::memory()) .on_flash(|flash: ChangeFlash| async move { println!("journal: {:?}", flash.kind()); Ok(()) }) .build()?; # Ok(()) # } ``` ```rust # use vantage_diorama::{prelude::*, Cache, DioEvent, FlashOutcome, Loader, LoaderCx}; # use vantage_types::Record; # use vantage_vista::{mocks::MockShell, prelude::*, Column, VistaMetadata}; # fn row(n: &str) -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() } # fn teams() -> Vista { # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let lens = Lens::builder() .cache(Cache::memory()) .writer(|_cx: LoaderCx, flash: ChangeFlash| async move { println!("journal: {:?}", flash.kind()); let outcome = FlashOutcome::applied(); Ok(match flash.id() { Some(_) => outcome, None => outcome.with_id("j-1"), // an id-less insert: report the id it got }) }) .build()?; let dio = lens.dio(teams()).build().await?; let outcome = dio.flash(ChangeFlash::insert_new(row("green"))).await?; # let _ = outcome; # Ok(()) # } ``` Writes on one Dio run one at a time, in order, and a writer must not flash the Dio it serves: that returns an error naming the Dio, where 1.x ran it inline. A writer that relied on concurrent calls now sees them one at a time per Dio. A writer that panics fails its flash, which is rolled back, and the queue carries on. A caller that stops waiting no longer cancels its write, and a rollback of stacked flashes restores the last confirmed row. ## Changes from outside `patched`, `removed` and `notify_record_changed` become `apply_change(ChangeEvent)`; `reconcile_value`, `reconcile_values` and `notify_dataset_changed` become `reconcile(snapshot)`, which makes the cache hold exactly that set; `handle_event` (which only dispatched to `on_event`) is gone. ```rust,compile_fail,E0599 # use vantage_diorama::{prelude::*, Cache, DioEvent, FlashOutcome, Loader, LoaderCx}; # use vantage_types::Record; # use vantage_vista::{mocks::MockShell, prelude::*, Column, VistaMetadata}; # fn row(n: &str) -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() } # fn teams() -> Vista { # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # async fn copy(cx: LoaderCx) -> vantage_core::Result<()> { let rows = cx.master().list_values().await?; cx.reconcile(rows).await } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let dio = Lens::builder().cache(Cache::memory()).loader(Loader::eager(copy)).build()?.dio(teams()).build().await?; dio.patched("a", row("green")).await?; dio.reconcile_values(dio.master().list_values().await?).await?; # Ok(()) # } ``` ```rust # use vantage_diorama::{prelude::*, Cache, DioEvent, FlashOutcome, Loader, LoaderCx}; # use vantage_types::Record; # use vantage_vista::{mocks::MockShell, prelude::*, Column, VistaMetadata}; # fn row(n: &str) -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() } # fn teams() -> Vista { # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # async fn copy(cx: LoaderCx) -> vantage_core::Result<()> { let rows = cx.master().list_values().await?; cx.reconcile(rows).await } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let dio = Lens::builder().cache(Cache::memory()).loader(Loader::eager(copy)).build()?.dio(teams()).build().await?; dio.apply_change(ChangeEvent::Updated { id: "a".into(), new: Some(row("green")) }).await?; dio.reconcile(dio.master().list_values().await?).await?; # Ok(()) # } ``` `Updated { id, new: None }` is the notify-only shape (`notify_record_changed`); `Deleted { id }` is `removed(id)`; `Invalidated` re-reads the set. `reconcile` skips rows with a write in flight, deletes cached rows absent from the snapshot, stamps the refresh time and publishes one `DatasetChanged`. Inside a loader or refresher, `cx.reconcile(rows)` is the same call. ## Facade writes are awaited `dio.cached()` inserts, replaces, upserts, patches and deletes no longer return before the write lands. Each one awaits `Dio::flash` and returns once the write is confirmed, or with its error once the staged row is rolled back. Polling loops that waited for a facade write to show up can go, and the returned error has to be handled where it used to be ignored. The record a facade write hands back carries the id under the Dio's id column, not a literal `"id"`. ```rust # use vantage_diorama::{prelude::*, Cache, DioEvent, FlashOutcome, Loader, LoaderCx}; # use vantage_types::Record; # use vantage_vista::{mocks::MockShell, prelude::*, Column, VistaMetadata}; # fn row(n: &str) -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() } # fn teams() -> Vista { # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # async fn copy(cx: LoaderCx) -> vantage_core::Result<()> { let rows = cx.master().list_values().await?; cx.reconcile(rows).await } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let dio = Lens::builder().cache(Cache::memory()).loader(Loader::eager(copy)).build()?.dio(teams()).build().await?; let mut events = dio.subscribe_events(); dio.cached().insert_value("c", &row("green")).await?; // confirmed when this returns dio.flash(ChangeFlash::delete("b")).await?; while let Ok(event) = events.try_recv() { match event { DioEvent::RecordInserted { id, .. } => println!("+ {id}"), DioEvent::RecordRemoved { id, .. } => println!("- {id}"), DioEvent::RecordChanged { id, .. } => println!("~ {id}"), _ => {} } } # Ok(()) # } ``` A confirmed write publishes its settle event by membership: a delete is `RecordRemoved`, a write of a row the cache didn't hold is `RecordInserted`, any other write `RecordChanged`. 1.x published `RecordChanged` for all three, so a subscriber that reacted to it after an insert or a delete handles the other two as well. A re-keyed row publishes `RecordRemoved` for the staged id and `RecordInserted` for the settled one. `DioEvent::WriteFailed` now announces only a failed write that staged no row (an id-less insert, `Clear`). ## Writes are confined to the Dio's set A derived Dio's writes are held to its full query. Diorama checks the residual itself: a record gets its `Eq` values filled in, and a record or patch result outside the set, or a write to an id held outside it, is `Conflict`. A patch of a row the set doesn't hold is `NotFound`, and a delete of one writes nothing. Under a Lens writer the whole query is checked, and a `Clear` (facade `delete_all`) becomes one `Delete` per row in the set. A native `import_values` is refused as `Conflict` when one record falls outside the residual. A native condition can't be judged in memory, so under a writer a Dio narrowed by one refuses every write, naming the clause. A write that relied on an `Eq` on an augmented column being filled into the record supplies the column itself. A set narrowed by a dotted path (`address.city = x`) records no invariant: a full-record write must hold `x` at `address.city` itself, and a patch must leave the row in the set, or it is `Conflict`. vantage-memory follows the same rule. ## Renames | 1.x | 2.0 | |---|---| | `flash_insert(id, r)`, `flash_patch(id, f)`, `flash_replace`, `flash_delete(id)` | `flash(ChangeFlash::insert(id, r))`, `::partial(id, f)`, `::replace`, `::delete(id)` | | `flash_returning_id`, `insert_returning_id` | `flash(ChangeFlash::insert_new(r)).await?.id` | | `ChangeFlash::new(FlashKind::Patch, Some(id), f)` | `ChangeFlash::partial(id, f)`, one constructor per kind | | writer returning `Result<()>` | `Result`: `FlashOutcome::applied()` | | `patched(id, r)`, `removed(id)`, `notify_record_changed(id)` | `apply_change(ChangeEvent::Updated { id, new: Some(r) })`, `Deleted { id }`, `Updated { id, new: None }` | | `reconcile_value(s)`, `notify_dataset_changed`, `notify_seeded` | `reconcile(snapshot)` or `refresh()`; the Dio announces a detached eager start itself | | rhai `record.save()`, `servo.save()` | `flash()` | ## Checklist - [ ] every write is `dio.flash(ChangeFlash::…)`; no `flash_*` method or `ChangeFlash::new` - [ ] the Lens writer returns `FlashOutcome`, with `.with_id(..)` for id-less inserts - [ ] facade write callers handle the returned error and drop their polling - [ ] `subscribe_events` consumers handle `RecordInserted` and `RecordRemoved`, and end their match with `_ =>` - [ ] external changes go through `apply_change` or `reconcile` - [ ] writers over narrowed Dios expect one `Delete` per row in place of `Clear` --- Source: https://vantage-ui.com/book/migration-2.0/sceneries.html # Sceneries In 1.x two identical opens returned the same `Arc`, so one grid's `set_search` changed what another grid showed, and an `.exclusive()` flag opted out. A scenery had two filter setters, a separate shown range, a cap wrapper and its own sort enum. 2.0 gives each scenery one owner: every open is its own view, and sharing moved underneath, to the fetched rows, so two views of one query still make one request (decision 7: only identical queries share). The filter vocabulary is vista's (`Condition`, `SortDirection`), so a grid's filter and a Dio's narrowing are the same type. Limit, tail and search became builder options. Read [Scenery: Reactive Views](../intro/step7-scenery.md) for the current shape. The snippets open views on a Dio of `teams` (rows `a` red, `b` blue). ## One sort direction, one condition type Diorama's `SortDir` and `OpCondition` are gone: use `vantage_vista::SortDirection` and `vantage_vista::Condition`, both in vista's prelude. ```rust,compile_fail,E0432 use vantage_diorama::{OpCondition, SortDir}; let direction = SortDir::Asc; ``` ```rust use vantage_vista::prelude::*; let direction = SortDirection::Ascending; let not_blue = Condition::new("name", FilterOp::Ne, "blue"); # let _ = (direction, not_blue); ``` ## The builder `where_eq` and `where_op` are one `filter(Condition)`; `sort` is `order`; `initial_range` is gone (the first viewport is `0..page_size`); `exclusive()` is gone because every view is exclusive. ```rust,compile_fail,E0599 # use vantage_diorama::{prelude::*, Cache, LoadState, Loader, LoaderCx, RecordStatus}; # use vantage_types::Record; # use vantage_vista::{mocks::MockShell, prelude::*, Column, VistaMetadata}; # fn row(n: &str) -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() } # fn teams() -> Vista { # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # async fn copy(cx: LoaderCx) -> vantage_core::Result<()> { let rows = cx.master().list_values().await?; cx.reconcile(rows).await } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let dio = Lens::builder().cache(Cache::memory()).loader(Loader::eager(copy)).build()?.dio(teams()).build().await?; let view = dio .table_scenery() .where_eq("name", "red") .sort("name", SortDirection::Ascending) .exclusive() .open() .await?; # Ok(()) # } ``` ```rust # use vantage_diorama::{prelude::*, Cache, LoadState, Loader, LoaderCx, RecordStatus}; # use vantage_types::Record; # use vantage_vista::{mocks::MockShell, prelude::*, Column, VistaMetadata}; # fn row(n: &str) -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() } # fn teams() -> Vista { # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # async fn copy(cx: LoaderCx) -> vantage_core::Result<()> { let rows = cx.master().list_values().await?; cx.reconcile(rows).await } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let dio = Lens::builder().cache(Cache::memory()).loader(Loader::eager(copy)).build()?.dio(teams()).build().await?; let view = dio .table_scenery() .filter(Condition::new("name", FilterOp::Eq, "red")) .order("name", SortDirection::Ascending) .page_size(50) .open() .await?; let latest = dio.table_scenery().order("name", SortDirection::Ascending).tail(10).open().await?; # let _ = (view, latest); # Ok(()) # } ``` `CappedScenery::wrap(s, n)` and ui-diorama's `open_tail(n)` are `.limit(n)` and `.tail(n)` on the builder. `tail` needs an order, and the two together are refused. ## The setters `set_sort(col, dir)` is `set_order(Option<(col, dir)>)`, with `None` clearing it. `set_filters` (eq pairs) and `set_filter_terms` (operator terms) are one `set_filters(Vec)`; an order-insensitively equal set is a no-op. `set_shown_range` merges into the viewport: `set_viewport(loaded, shown)`, where an empty `shown` means nothing is on screen. ```rust,compile_fail,E0599 # use vantage_diorama::{prelude::*, Cache, LoadState, Loader, LoaderCx, RecordStatus}; # use vantage_types::Record; # use vantage_vista::{mocks::MockShell, prelude::*, Column, VistaMetadata}; # fn row(n: &str) -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() } # fn teams() -> Vista { # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # async fn copy(cx: LoaderCx) -> vantage_core::Result<()> { let rows = cx.master().list_values().await?; cx.reconcile(rows).await } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let dio = Lens::builder().cache(Cache::memory()).loader(Loader::eager(copy)).build()?.dio(teams()).build().await?; let view = dio.table_scenery().open().await?; view.set_sort(Some("name".into()), SortDirection::Descending); view.set_filter_terms(vec![Condition::new("name", FilterOp::Ne, "blue")]); view.set_shown_range(Some(0..20)); if view.is_loading() { /* spinner */ } # Ok(()) # } ``` ```rust # use vantage_diorama::{prelude::*, Cache, LoadState, Loader, LoaderCx, RecordStatus}; # use vantage_types::Record; # use vantage_vista::{mocks::MockShell, prelude::*, Column, VistaMetadata}; # fn row(n: &str) -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() } # fn teams() -> Vista { # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # async fn copy(cx: LoaderCx) -> vantage_core::Result<()> { let rows = cx.master().list_values().await?; cx.reconcile(rows).await } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let dio = Lens::builder().cache(Cache::memory()).loader(Loader::eager(copy)).build()?.dio(teams()).build().await?; let view = dio.table_scenery().open().await?; view.set_order(Some(("name".into(), SortDirection::Descending))); view.set_filters(vec![Condition::new("name", FilterOp::Ne, "blue")]); view.set_viewport(0..50, 0..20); match view.load_state() { LoadState::Loading => { /* spinner */ } LoadState::Failed(failure) => eprintln!("{failure}"), // shown where the grid shows "empty" _ => {} } # Ok(()) # } ``` `LoadState::Failed(LoadFailure)` is new, and `LoadState` is no longer `Copy`. A view fails, shows no rows and emits `DioEvent::LoadFailed` when the master rejects its query, or when it is a paged view whose query the master can't answer and the loader has no `or_eager` start ([Dio and Lens](./dio-lens.md#behaviour-notes)). `LoadFailure.clause` names the `Clause`: `Filter(column)`, `Order(column)` or `Search`. ## Record sceneries open through a builder `record_scenery(id).await` becomes `record_scenery(id).open().await`, and the `_sugared` and `_with` variants become builder options. `load_on_miss()` is new: a cache miss reads the master, and caches the row only when the Dio admits it. ```rust,compile_fail,E0599 # use vantage_diorama::{prelude::*, Cache, LoadState, Loader, LoaderCx, RecordStatus}; # use vantage_types::Record; # use vantage_vista::{mocks::MockShell, prelude::*, Column, VistaMetadata}; # fn row(n: &str) -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() } # fn teams() -> Vista { # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # async fn copy(cx: LoaderCx) -> vantage_core::Result<()> { let rows = cx.master().list_values().await?; cx.reconcile(rows).await } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let dio = Lens::builder().cache(Cache::memory()).loader(Loader::eager(copy)).build()?.dio(teams()).build().await?; let record = dio.record_scenery_with("a", row("red")).await?; let ready = matches!(record.status(), RecordStatus::Fresh | RecordStatus::Stale); # Ok(()) # } ``` ```rust # use vantage_diorama::{prelude::*, Cache, LoadState, Loader, LoaderCx, RecordStatus}; # use vantage_types::Record; # use vantage_vista::{mocks::MockShell, prelude::*, Column, VistaMetadata}; # fn row(n: &str) -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() } # fn teams() -> Vista { # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # async fn copy(cx: LoaderCx) -> vantage_core::Result<()> { let rows = cx.master().list_values().await?; cx.reconcile(rows).await } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let dio = Lens::builder().cache(Cache::memory()).loader(Loader::eager(copy)).build()?.dio(teams()).build().await?; let record = dio.record_scenery("a").seed(row("red")).open().await?; let ready = match record.status() { RecordStatus::Fresh => true, RecordStatus::Loading | RecordStatus::Incomplete => false, _ => false, }; # let _ = ready; # Ok(()) # } ``` `RecordStatus::Stale`, `RowStatus::{Stale, Loading, NotFound}`, `ValueStatus::Stale` and `EnrichedRecord::dirty_fields` are removed; `RecordStatus::Loading` and `Incomplete` are now produced. The status enums are `#[non_exhaustive]`, so a match ends in `_ =>`. ## Renames | 1.x | 2.0 | |---|---| | `request_refresh()` on table, record and value sceneries | `refresh_source()` | | `filter_terms()` | `view_stats().filters` (the Dio's residual first, then the view's) | | `terms_supported(&[OpCondition])`, `count_where(Vec)` | `filters_supported(&[Condition])`, `count_where(Vec)` | | `record_scenery_sugared(id, s)` | `record_scenery(id).sugar(s).open()` | | `SceneryDiagnostic::refcount` | `shared_by`, one per open view | | `ViewStats.order: Option<(String, SortDir)>` | `Option<(String, SortDirection)>`, the view's declared order | | `TableScenery::demanded_columns` | removed from the trait | | default bodies on `TableScenery` methods | every method required; `row_by_id(id)` is new | ## Behaviour notes - **Owned views.** Two opens are two handles; `Arc::ptr_eq` on them is false. With `diagnostics`, `dio.diagnostics().await?.sceneries[i].shared_by` counts the views sharing one set of rows. - **A scenery on a derived Dio** layers the Dio's residual under its own clauses and, with no sort of its own, keeps the Dio's order. - **Local refinement uses vista's evaluator**: cells are read with `cell_get`, `3 == 3.0`, null and absent cells sort last, ties break on id. Two-pass views and titles-only pickers now apply their conditions, search and the Dio's residual over the listed index; a picker whose query names an augmented column fails, since it never hydrates. - **Augmented clauses hydrate first.** A refined two-pass view over a cold cache runs the detail pass for every listed row before it settles, and reads `Partial` while a row's detail fails. - **A paged view's loader receives every clause** (filters, search, order) in its `VistaQuery`; `set_filters` on a paged view refetches the viewport. A total stated under a condition is not remembered as the table's total. - **`search_supported()` is true for a paged view with an eager fallback**, so a UI shows its search box there. - **Record, servo and value reads see only rows the Dio's residual admits**: a row outside a derived Dio's set is `NotFound` or absent. ## Checklist - [ ] no `SortDir` or `OpCondition`; `SortDirection` and `Condition` from vantage-vista - [ ] builders use `filter`, `order`, `limit`, `tail`, `search`; no `exclusive()` or `initial_range` - [ ] `set_order(Option<..>)`, `set_filters(Vec)`, `set_viewport(loaded, shown)` - [ ] `is_loading()` is `load_state() == LoadState::Loading`, and `LoadState::Failed` is shown - [ ] `record_scenery(id)…open().await?`; status matches end in `_ =>` - [ ] `request_refresh` is `refresh_source` --- Source: https://vantage-ui.com/book/migration-2.0/aggregates.html # Aggregates 1.x had two aggregate vocabularies: diorama's own (`value_scenery().count()`, `.sum(c)`, `CustomAggregate`) and the vantage-diorama-aggregate crate's (`Count`, `Sum::new`, `CountWhere`, `AggregateLens`, `DerivedDio`), with a second Lens that persisted every value in its own cache table. 2.0 keeps one vocabulary, folded into `vantage_diorama::aggregate` (decision 6, no shim). A value scenery takes an `Aggregation`; a derived table comes from `dio.aggregate(..)` and is a plain `Dio`. Nothing is persisted: the source's cache is the warm start, and a reopen recomputes. Read [Scenery: Reactive Views](../intro/step7-scenery.md) for value sceneries in use. The snippets run over a Dio of `teams` (rows `a` red, `b` blue). ## The crate is a module Drop the `vantage-diorama-aggregate` dependency and import from `vantage_diorama::aggregate`. `value_scenery` takes the aggregation, and the column aggregates are built with `of`. ```rust,compile_fail,E0432 # use vantage_diorama::{prelude::*, Cache, Loader, LoaderCx}; # use vantage_types::Record; # use vantage_vista::{mocks::MockShell, prelude::*, Column, VistaMetadata}; # fn row(n: &str) -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() } # fn teams() -> Vista { # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # async fn copy(cx: LoaderCx) -> vantage_core::Result<()> { let rows = cx.master().list_values().await?; cx.reconcile(rows).await } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let dio = Lens::builder().cache(Cache::memory()).loader(Loader::eager(copy)).build()?.dio(teams()).build().await?; use vantage_diorama_aggregate::{Conditions, Count, Max, Where}; let last = dio.value_scenery(Max::new("name")).open().await?; let red = dio .value_scenery(Where::new(Conditions::new().eq("name", "red"), Count::rows())) .open() .await?; # Ok(()) # } ``` ```rust # use vantage_diorama::{prelude::*, Cache, Loader, LoaderCx}; # use vantage_types::Record; # use vantage_vista::{mocks::MockShell, prelude::*, Column, VistaMetadata}; # fn row(n: &str) -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() } # fn teams() -> Vista { # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # async fn copy(cx: LoaderCx) -> vantage_core::Result<()> { let rows = cx.master().list_values().await?; cx.reconcile(rows).await } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let dio = Lens::builder().cache(Cache::memory()).loader(Loader::eager(copy)).build()?.dio(teams()).build().await?; use vantage_diorama::aggregate::{Conditions, Count, Max, Where}; let last = dio.value_scenery(Max::of("name")).open().await?; let red = dio .value_scenery(Where::new(Conditions::new().eq("name", "red"), Count::rows())) .open() .await?; # let _ = (last.value(), red.value()); # Ok(()) # } ``` A log filter on the old crate's target moves too: `RUST_LOG=vantage_diorama_aggregate=debug` is `RUST_LOG=vantage_diorama::aggregate=debug`, and `vantage_diorama=warn` now covers aggregate warnings. ## A derived table is a Dio `AggregateLens::in_memory().build()?.derive(&src, name, agg)` becomes `src.aggregate(agg).await?`, and `derive_value(&src, name, alias, agg)` becomes `src.aggregate(AsRows::new(alias, agg))`. The returned `Dio` is the handle; `DerivedDio`, its `dio()` and its `Deref` are gone. Dropping every clone stops the recomputation. ```rust,compile_fail,E0433 # use vantage_diorama::aggregate::{AsRows, Count}; # use vantage_diorama::{prelude::*, Cache, Loader, LoaderCx}; # use vantage_types::Record; # use vantage_vista::{mocks::MockShell, prelude::*, Column, VistaMetadata}; # fn row(n: &str) -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() } # fn teams() -> Vista { # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # async fn copy(cx: LoaderCx) -> vantage_core::Result<()> { let rows = cx.master().list_values().await?; cx.reconcile(rows).await } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let dio = Lens::builder().cache(Cache::memory()).loader(Loader::eager(copy)).build()?.dio(teams()).build().await?; let aggregates = AggregateLens::in_memory().build()?; let counted = aggregates.derive(&dio, "teams.count", AsRows::new("n", Count::rows())).await?; # Ok(()) # } ``` ```rust # use vantage_diorama::aggregate::{AsRows, Count}; # use vantage_diorama::{prelude::*, Cache, Loader, LoaderCx}; # use vantage_types::Record; # use vantage_vista::{mocks::MockShell, prelude::*, Column, VistaMetadata}; # fn row(n: &str) -> Record { [("name".to_string(), CborValue::Text(n.into()))].into_iter().collect() } # fn teams() -> Vista { # let meta = VistaMetadata::new().with_column(Column::new("id", "String").with_flag("id")).with_column(Column::new("name", "String")).with_id_column("id"); # Vista::new("teams", MockShell::new().with_metadata(meta).with_record("a", row("red")).with_record("b", row("blue"))) # } # async fn copy(cx: LoaderCx) -> vantage_core::Result<()> { let rows = cx.master().list_values().await?; cx.reconcile(rows).await } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let dio = Lens::builder().cache(Cache::memory()).loader(Loader::eager(copy)).build()?.dio(teams()).build().await?; let counted = dio.aggregate(AsRows::new("n", Count::rows())).await?; let table = counted.cached(); // a Vista like any other # let _ = table; # Ok(()) # } ``` Every aggregate of one source is named `#aggregate` in the debug tap and diagnostics, whatever the aggregation, so two aggregates of one source can't be told apart by name. They are still two Dios on two private Lenses and never share a cache. ## A custom aggregation `value_scenery().custom(f)`, `CustomAggregate` and `boxed_custom_aggregate` become an `impl Aggregation`. The trait gained the schema on `compute` and a required `reads()`, which tells the Dio what the computation reads: `Reads::Count` (the row count only), `Reads::Columns(..)` or `Reads::Rows`. `GroupReducer::reduce` and the `Reduce` closure gain the schema the same way, and `GroupReducer` gains `reads()`. ```rust use vantage_diorama::aggregate::{Aggregation, Reads, Rows}; use vantage_vista::{CborValue, VistaMetadata}; /// The length of the longest `name`. struct Longest; impl Aggregation for Longest { type Output = CborValue; fn compute(&self, rows: &Rows, _schema: &VistaMetadata) -> CborValue { let longest = rows.values().filter_map(|r| r.get("name")?.as_text().map(str::len)).max(); longest.map_or(CborValue::Null, |n| CborValue::from(n as i64)) } fn reads(&self) -> Reads { Reads::Columns(vec!["name".to_string()]) } } # let _ = Longest; ``` `AggregateOutput` is sealed: an aggregation returns `CborValue` (a scalar, for value sceneries) or `DerivedRows` (a table, for `dio.aggregate`). ## `Where` compares by the column's type `Where` and the column reductions convert an operand by the column's declared type, and compare an untyped column as given. 1.x fell back to comparing rendered text, so a schemaless source loses that fallback (VAN-384). | Condition | Row | 1.x | 2.0 | |---|---|---|---| | `.eq("code", "200")` | `{code: 200}`, column untyped | matched | no match: declare `type: i64`, or write `.eq("code", 200)` | | `.eq("code", "200")` | `{code: 200}`, column `i64` | matched | matches (typed columns coerce both ways, bool included) | | `.eq("price", "9.99")` | a tag-10 decimal cell | matched | no match: write the operand in the cell's type | | `.eq("id", "user:1")` | a record-id cell | matched | no match | | `.eq("a.b", x)` | `{a: {b: x}}` | matched the path | matches the path; a literal `"a.b"` key only when the schema declares `a.b` | A derived `GroupBy` key column takes the source key's type (it was always `"String"`), and is untyped when the source key has none. ## Renames | 1.x | 2.0 | |---|---| | `value_scenery().count()`, `.sum(c)`, `.max(c)`, `.min(c)` | `value_scenery(Count::rows())`, `Sum::of(c)`, `Max::of(c)`, `Min::of(c)` | | `value_scenery().count_where(pairs)`, `CountWhere::new(conds)` | `value_scenery(Where::new(Conditions::new().eq(..), Count::rows()))` | | `Sum::new`, `Avg::new`, `Min::new`, `Max::new`, `Distinct::new` | `Sum::of` and so on | | `AggregateLens::value(&src, name, agg)` | `src.value_scenery(agg).open().await?` | | `AggregateLens::{at, in_memory, with_backend, retain, close}`, `AggregateLensBuilder` | removed: no persistence | | `AggregateLensBuilder::debounce(d)` | `Tuning::default().with_aggregate_debounce(d)` on the source's Lens | | `AggregateValue`, `seed_from_cache`, the per-aggregate cache table | none; a reopen recomputes from the source's cache | | `Aggregation::compute(rows)` | `compute(rows, schema)` + `reads()` | | `GroupReducer::reduce(key, rows)`, `Reduce::new(\|k, rows\| ..)` | `reduce(key, rows, schema)` + `reads()`, `Reduce::new(columns, \|k, rows, schema\| ..)` | | `AggregationCatalog::group_by` → `GroupBy>` | `GroupBy` | | `Conditions::matches(&row)`, `narrow(rows)` | `matches(&row, &schema)`, `narrow(rows, &schema)` | | `AggregationSpec { column, conditions }` | `AggregationSpec::new(column, conditions)` | | core `Aggregate`, `CustomAggregateFuture` | an `impl Aggregation` | ## Behaviour notes - **An aggregate over a derived Dio counts only the rows its narrowing admits**: it reads the source through `cached()`, residual included. - **A recomputation writes only the rows whose value changed** (through `Dio::reconcile`), and still publishes one `DatasetChanged`. - **`Reads::Rows` on an augmented Dio hydrates every cached row** on each recompute; prefer `Reads::Columns` when the computation names its columns. ## Checklist - [ ] no `vantage_diorama_aggregate` import or dependency - [ ] every `value_scenery()` call takes an aggregation; column aggregates use `of` - [ ] no `AggregateLens`, `DerivedDio` or `derive_value`: `src.aggregate(..)` returns a `Dio` - [ ] custom aggregations take the schema and implement `reads()` - [ ] `Where` conditions over untyped columns either declare the column type or write the operand in the cell's own type --- Source: https://vantage-ui.com/book/migration-2.0/surface.html # Surface Most first-round compile errors in a port are paths. Rule 11 gives every public item one path: its module or the crate root, never both, and no crate re-exports another crate's items outside its prelude. Rule 8 made the internal modules private. The preludes shrank to what nearly every user writes, each item under its home crate. Rule 7 put `#[non_exhaustive]` on the enums and data structs that may grow, so matches gain a `_ =>` arm and structs are built through constructors. Rule 9 moved test and debug helpers behind features. None of this changes behaviour; it changes where things are found. ## The preludes | Crate | Prelude | |---|---| | vantage-vista | `Vista`, `VistaQuery`, `Condition`, `FilterOp`, `SortDirection`, `Capability`, `VistaResolver`, `CborValue`, and the vantage-dataset value-set traits | | vantage-diorama | `Lens`, `Dio`, `ChangeFlash`, `ChangeEvent`, `TableScenery`, `RecordScenery`, `ValueScenery`, `Servo`, `Aggregation` | Everything else is imported from the crate root. Diorama's prelude no longer reaches `Cache`, `Loader`, `LoaderCx`, `Refresher`, `Tuning`, `Augmentation`, `Detail`, `Fetch`, `MergeRule`, `Source`, `DioEvent`, `Generation`, `WriteCapabilities`, `CacheBackend`, `CacheStatus`, `CacheTable`, `RedbCache`, `FlashKind`, `FlashOutcome`, `EnrichedRecord`, `RecordStatus`, `RowStatus`, `IdStrategy`, `ServoStatus` or vista's `VistaCatalog`. Vista's no longer reaches `VistaFactory`, `TableShell` or `VistaCapabilities`. ```rust,compile_fail,E0433 use vantage_diorama::prelude::*; # use vantage_vista::prelude::*; # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let lens = Lens::builder() .cache(Cache::memory()) .loader(Loader::eager(|cx| async move { let rows = cx.master().list_values().await?; cx.reconcile(rows).await })) .build()?; # Ok(()) # } ``` ```rust use vantage_diorama::{prelude::*, Cache, Loader}; # use vantage_vista::prelude::*; # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let lens = Lens::builder() .cache(Cache::memory()) .loader(Loader::eager(|cx| async move { let rows = cx.master().list_values().await?; cx.reconcile(rows).await })) .build()?; # let _ = lens; # Ok(()) # } ``` A factory or shell author imports the traits from the root: ```rust,compile_fail,E0405 use vantage_vista::prelude::*; fn register(_factory: F) {} ``` ```rust use vantage_vista::{prelude::*, VistaFactory}; fn register(_factory: F) {} ``` ## Private modules Diorama's `augment`, `dio`, `lens`, `ops`, `scenery` and `servo` modules are private; every public item they held is at the crate root. ```rust,compile_fail,E0603 use vantage_diorama::scenery::{TableScenery, RecordStatus}; # use vantage_diorama::LoadState; fn ready(view: &dyn TableScenery) -> bool { view.load_state() == LoadState::Complete } # fn main() { let _ = RecordStatus::Fresh; } ``` ```rust use vantage_diorama::{TableScenery, RecordStatus}; # use vantage_diorama::LoadState; fn ready(view: &dyn TableScenery) -> bool { view.load_state() == LoadState::Complete } # fn main() { let _ = RecordStatus::Fresh; } ``` | 1.x | 2.0 | |---|---| | `vantage_diorama::scenery::{table, record, value, sugar, enriched_record}::X` | `vantage_diorama::X` | | `vantage_diorama::lens::CacheTable`, `lens::Cache`, … | `vantage_diorama::CacheTable`, … | | `vantage_diorama::dio::event_bus::DioEvent`, `ops::change_flash::ChangeFlash` | `vantage_diorama::DioEvent`, `vantage_diorama::ChangeFlash` | | `vantage_diorama::servo::Servo`, `augment::Augmentation` | `vantage_diorama::Servo`, `vantage_diorama::Augmentation` | | `vantage_diorama::{DebugTap, ProcessStats, process_stats}` | `vantage_diorama::debug::{…}` (the `diagnostics` feature) | | `vantage_diorama::VistaCapabilities` | `vantage_vista::VistaCapabilities` | | `vantage_diorama::debug::{dur, bytes, pct}` | crate-private; `debug::num` stays public | | `DioShell`, `dio::shell` | crate-private; use the Vista from `dio.cached()` | | `vantage_vista::{DataVocab, Handle, RecordDraft, run_script, …}` | `vantage_vista::rhai::…` | | `vantage_memory::vista::{Catalog, MemoryTableShell, factory::…}` | `vantage_memory::{MemoryCatalog, MemoryTableShell, MemoryVistaFactory, …}` | | `pub mod dio::{worker, event_bus, impls, shell, hot_tier, refresh}` | crate-private | ## Matches gain a wildcard arm An exhaustive match on a diorama enum stops compiling, because a new variant is no longer a breaking change. The enums this hits most: `FlashKind`, `ChangeEvent`, `DioEvent`, `LoadState`, `Clause`, `RecordStatus`, `RowStatus`, `ValueStatus`, `ServoStatus`, `IdStrategy`, `CacheStatus`, `Detail`, `Fetch`, `Source`, and vista's `Capability`, `FilterOp`, `ColumnFlag`, `ReferenceKind`, `ReferenceSugar`, `Terminals` and `Writes`. `==` and `matches!` are unaffected. ```rust,compile_fail,E0004 # use vantage_diorama::{prelude::*, FlashKind}; fn label(flash: &ChangeFlash) -> &'static str { match flash.kind() { FlashKind::Insert => "insert", FlashKind::Patch => "patch", FlashKind::Replace => "replace", FlashKind::Upsert => "upsert", FlashKind::Delete => "delete", FlashKind::Clear => "clear", } } # fn main() { let _ = label(&ChangeFlash::clear()); } ``` ```rust # use vantage_diorama::{prelude::*, FlashKind}; fn label(flash: &ChangeFlash) -> &'static str { match flash.kind() { FlashKind::Insert => "insert", FlashKind::Patch => "patch", FlashKind::Replace => "replace", FlashKind::Upsert => "upsert", FlashKind::Delete => "delete", FlashKind::Clear => "clear", _ => "write", } } # fn main() { assert_eq!(label(&ChangeFlash::clear()), "clear"); } ``` ## Structs are built through constructors A struct literal of a `#[non_exhaustive]` struct from another crate doesn't compile. Reading its public fields still works. ```rust,compile_fail,E0639 # use std::sync::Arc; # use vantage_diorama::{Augmentation, Detail, Fetch, MergeRule, Source}; # use vantage_vista::{mocks::MockShell, Vista}; # fn main() { # let sizes = Arc::new(Vista::new("sizes", MockShell::new())); let size = Augmentation { detail: Detail::Fixed(sizes), source: Source::Id, fetch: Fetch::PerRow, merge: MergeRule { columns: vec!["size".to_string()] }, }; # } ``` ```rust # use std::sync::Arc; # use vantage_diorama::{Augmentation, Detail, Fetch, MergeRule, Source}; # use vantage_vista::{mocks::MockShell, Vista}; # fn main() { # let sizes = Arc::new(Vista::new("sizes", MockShell::new())); let size = Augmentation::new(Detail::Fixed(sizes), Source::Id, Fetch::PerRow, MergeRule::lift(["size"])); # assert_eq!(size.merge.columns, ["size"]); # } ``` | Struct | Build it with | |---|---| | `Augmentation`, `MergeRule`, `AggregationSpec` | `Augmentation::new(..)`, `MergeRule::lift(cols)` / `lift_all()`, `AggregationSpec::new(..)` | | `EnrichedRecord` | `EnrichedRecord::fresh(r)`, `incomplete`, `pending_write`, `write_failed`, `detail_failed` | | `Tuning` | `Tuning::default().with_*(..)` | | `VistaSpec`, `ColumnSpec`, `ReferenceSpec`, `ContainedYaml`, `NoExtras` | `VistaSpec::new(name)`, `ColumnSpec::new()`, deserialise, `NoExtras::default()` | | `ImportOutcome`, `DioDiagnostics`, `SceneryDiagnostic`, `Fetched`, `FlashOutcome`, `ViewStats` | read their fields; `FlashOutcome::applied()` | ## Traits you implement The open extension traits lost their default bodies (rule 2), so an implementation states every method: - **`TableShell`**: `metadata()`, `Clone`, and `unsupported(..)` for what it doesn't do; see [Vista](./vista.md#for-shell-authors). - **`CacheBackend` and `CacheTable`**: `close`, `list_tables`, `drop_table` and `name` on the backend; `meta_total`, `meta_refreshed_at` and `meta_copied_at` with their setters, and the `*_with_status` reads and writes, on the table. A wrapper forwards each one, status included, and `clear` forgets the refresh and copy stamps. A backend that can't enumerate or drop tables returns an error. - **`TableScenery`**: every method, including `row_by_id`; `demanded_columns` left the trait. - **`Aggregation` and `GroupReducer`**: `reads()`, and the schema parameter ([Aggregates](./aggregates.md#a-custom-aggregation)). ## Features | Feature | Gates | |---|---| | vantage-diorama `diagnostics` | `debug` (`DebugTap`, `num`, `ProcessStats`, `process_stats`), `stats`, `Dio::debug_tap`, `LensBuilder::debug_datasource`, `Dio::diagnostics`, `DioDiagnostics`, `SceneryDiagnostic`; a default build records no fetch ledger | | vantage-diorama `test-util` | `Dio::take_write_worker_handle` | | vantage-vista `mocks` | `vantage_vista::mocks` (`MockShell`, `MockCalls`); it enables tokio's `sync` | | vantage-table `mocks` | `vantage_table::mocks`, `prelude::MockTableSource` | `Dio::live_table_scenery_count()` is `dio.diagnostics().await?.sceneries.len()`, and `diagnostics()` returns a `Result`: a cache that can't count its rows is an error naming the table, where it read as `cache_rows: 0`. ## Checklist - [ ] imports outside the preludes come from the crate root - [ ] no `vantage_diorama::{scenery, lens, dio, ops, servo, augment}::` paths - [ ] every match on a vista or diorama enum ends in `_ =>` - [ ] no struct literal of a `#[non_exhaustive]` struct - [ ] `diagnostics`, `test-util` and `mocks` are enabled where their items are used --- Source: https://vantage-ui.com/book/migration-2.0/drivers.html # Drivers The drivers' own factory APIs (`from_table`, `from_yaml`, `with_resolver`) keep their shape; what changed is underneath and around them. Rule 1 removed the per-driver resolver aliases in favour of one `SpecResolver` trait, and rule 2 removed the places where a driver quietly ignored part of a spec: `keys:` lists, column-level `references:`, and the `searchable` flag. Each now means the same thing on every driver. The changes from the driver cleanup (PR #437) are on this page too, since they land in the same release. Read [Config-Driven Vistas](../config-driven-vistas.md) and [Adding a New Persistence](../new-persistence.md) for the current driver surface. ## The spec resolver is a trait `SqliteSpecResolver`, `PostgresSpecResolver`, `MysqlSpecResolver`, `SurrealSpecResolver`, `MongoSpecResolver` and `GraphqlSpecResolver` are gone. A factory holds `Arc>`, and any `Fn(&str) -> Option` closure implements the trait. ```rust,compile_fail,E0432 # use std::collections::HashMap; # use std::sync::Arc; # use vantage_sql::prelude::*; # use vantage_sql::sqlite::vista::SqliteVistaSpec; use vantage_sql::sqlite::vista::SqliteSpecResolver; # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite::memory:").await.context("connect")?; # let specs: HashMap = HashMap::from([("team".to_string(), SqliteVistaSpec::new("team"))]); let resolver: SqliteSpecResolver = Arc::new(move |name| specs.get(name).cloned()); let factory = db.vista_factory().with_resolver(resolver); # Ok(()) # } ``` ```rust # use std::collections::HashMap; # use std::sync::Arc; # use vantage_sql::prelude::*; # use vantage_sql::sqlite::vista::SqliteVistaSpec; use vantage_vista::SpecResolver; # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite::memory:").await.context("connect")?; # let specs: HashMap = HashMap::from([("team".to_string(), SqliteVistaSpec::new("team"))]); let resolver: Arc> = Arc::new(move |name: &str| specs.get(name).cloned()); let factory = db.vista_factory().with_resolver(resolver); # let _ = factory; # Ok(()) # } ``` Type the closure's parameter. `Arc::new(move |name| ..)` doesn't infer a parameter that takes every lifetime, and fails to coerce with "implementation of `Fn` is not general enough". The same holds for api-client's `ModelResolver`, now `with_model_resolver(Arc)`, and for any `|name: &str|` closure held as a `VistaResolver`. ### Why a trait and not the alias? An alias for `Arc Option>` only admits closures. A trait admits a type of your own, backed by a folder of YAML files or a hot-reloading inventory, with a `resolve` method you can test. The cost is the parameter annotation above, which the alias inferred for you. ## The resolver rides on the handle `RecordDraft::{new_row, load, from_row}` drop their resolver argument. `Handle::with_resolver` is public, and a named handle resolves through the resolver it carries. ```rust,compile_fail,E0061 # use std::sync::Arc; # use indexmap::IndexMap; # use vantage_vista::rhai::{Handle, RecordDraft, Writes}; # use vantage_vista::{mocks::MockShell, prelude::*, Catalog}; # fn main() { # let resolver: Arc = Arc::new(Catalog::new().with("row", |_: &str| Ok(Vista::new("row", MockShell::new())))); # let values: IndexMap = IndexMap::new(); let draft = RecordDraft::from_row(Handle::named("row"), Some(resolver), Writes::Allowed, "r1".to_string(), values); # } ``` ```rust # use std::sync::Arc; # use indexmap::IndexMap; # use vantage_vista::rhai::{Handle, RecordDraft, Writes}; # use vantage_vista::{mocks::MockShell, prelude::*, Catalog}; # fn main() { # let resolver: Arc = Arc::new(Catalog::new().with("row", |_: &str| Ok(Vista::new("row", MockShell::new())))); # let values: IndexMap = IndexMap::new(); let handle = Handle::named("row").with_resolver(resolver); let draft = RecordDraft::from_row(handle, Writes::Allowed, "r1".to_string(), values); # let _ = draft; # } ``` A caller holding an `Option>` attaches it only when present: `match resolver { Some(r) => handle.with_resolver(r), None => handle }`. ## Specs mean the same on every driver - **`keys:` is honoured.** A reference with a multi-column `keys:` list narrows on every pair. SQL, MongoDB and SurrealDB used to narrow on `foreign_key` alone and silently ignore the rest; vantage-memory used to refuse `keys:`. A spec without `keys:` lowers exactly as before. - **Column-level `references:` reach every driver's metadata.** `client_id: { references: client }` is a has-one named after the column, with the column as its foreign key; the full form `{ table, kind, foreign_key }` keeps its own kind. A `references:` entry of the same name wins. Before, only the table registered it, so it was missing from the Vista's `references()` on MongoDB and named by the built table on SurrealDB. MongoDB now applies its "a key is stored under its own field name" check to these too. - **`Reference.target` is the spec name** (`table:`), even when the spec reads another physical table, and a typed relation's target is its built table's name, so `catalog.vista(&r.target)` works everywhere. ### Declare the foreign key on the target Traversal narrows the target by the join's `to` column, so the target has to declare it. A has-many target that leaves out its foreign key column fails the traversal with an error naming the relation and the column, where 1.x injected or ignored it: ```yaml name: product columns: id: { type: string, flags: [id] } name: { type: string } bakery_id: { type: string } # the has-many `bakery.products` narrows by this ``` ## Quicksearch reads the searchable columns Search reads the columns flagged `Searchable`, or every column when none is. The evaluator, `MockShell` and every driver follow the rule through `VistaMetadata::search_columns` (vista) and `Table::search_columns` (vantage-table). A SQL table with only `name` flagged no longer searches every column; a MongoDB spec with no flag now searches every column, where it matched nothing. ```rust use vantage_vista::{Column, ColumnFlag, VistaMetadata}; let metadata = VistaMetadata::new() .with_column(Column::new("id", "String").with_flag(ColumnFlag::Id)) .with_column(Column::new("name", "String").with_flag(ColumnFlag::Searchable)) .with_column(Column::new("notes", "String")) .with_id_column("id"); assert_eq!(metadata.search_columns(), ["name"]); ``` GraphQL searches the flagged columns, or every text column but the id when none is flagged. Kubernetes drops `searchable_titles` and searches every declared column. `MemoryCondition::Search(t)` is `MemoryCondition::search(t)`, building a `Search { text, columns }` that reads every cell. ## Any window from a listing A shell that can only list builds its window with `Page::from_listing(rows, offset, limit)`: any offset, the listing's length as the total. It replaces csv's, cmd's and kubernetes' own slicing and the interim `Page::leading_window`. `Vista` still refuses an offset above zero without `Window` before the shell is called. ```rust,compile_fail,E0599 # use vantage_types::Record; # use vantage_vista::{CborValue, Page}; # fn main() -> vantage_core::Result<()> { # let listing: Vec<(String, Record)> = ["a", "b", "c"].iter().map(|id| (id.to_string(), Record::new())).collect(); let page = Page::leading_window(listing, 0, 2)?; # Ok(()) # } ``` ```rust # use vantage_types::Record; # use vantage_vista::{CborValue, Page}; # fn main() -> vantage_core::Result<()> { # let listing: Vec<(String, Record)> = ["a", "b", "c"].iter().map(|id| (id.to_string(), Record::new())).collect(); let page = Page::from_listing(listing, 2, 2); # assert_eq!((page.rows.len(), page.total), (1, Some(3))); # Ok(()) # } ``` ## Renames | 1.x | 2.0 | |---|---| | `vantage_memory::vista::Catalog`, `Catalog::shell(n)`, `Catalog::get(n)` | `MemoryCatalog`, a `VistaResolver`: `catalog.vista(n)` (`is_not_found()` for an unknown name) | | `MemoryTableShell::with_untyped_columns(cols)`, `capabilities_for_memory()` | `Column::untyped(name)`; `vista.capabilities()` | | vantage-table `Reference::cardinality() -> Cardinality`, `foreign_key()` | `Reference::vista_reference(name, target, source_id) -> vantage_vista::Reference` | | `Table::with_spec_references(&spec.references, ..)` | `with_spec_references(&spec, ..)`; `table::spec::spec_references(spec, source_id)` for metadata | | `Table::name_join_targets`, `name_spec_targets`, `Reference::target_type_name()` | removed; `target_table_name(data_source)` | | `ContainedRelation::new(name, host, kind, id_column, build)` | `ContainedRelation::one(name, host, build)`, `many(name, host, id_column, build)` | | `get_contained_ref(rel, row, parent_id, ..)` | `get_contained_ref(rel, Option<(&row, id)>, ..)`; `None` is the bare target | | `MockReference::new(cardinality, fk)` (public) | test-only | | serialised `Preview` `{driver, query, note}` | `{query, note}`; add `driver` from `vista.driver_name()` yourself | | faker shape YAML capabilities `fetch_next`, `fetch_window`, `count`, `fetch_page` | `cursor`, `window`; `count` and `fetch_page` removed | | request-parameter columns flagged `hidden` | `flags: [hidden, request_param]` (`ColumnFlag::RequestParam`) | | `temp_remove_condition(h)` then `temp_add_condition(c)` | `replace_temp_condition(old, new)`; removing a stale handle is `is_not_found()` | | `build_contained_vista(.., writeback, ..)` | `writeback: Option`; `None` builds the bare target | | `register_constructors!(engine, value: V)` and the other vantage-sql macros | `register_constructors!(engine, ns, value: V)`; `register_engine!` is unchanged | ## Behaviour notes - **Dotted conditions.** SQL refuses a dotted or imported column ("can't be filtered in SQL"); MongoDB and SurrealDB evaluate `root.leaf` natively when `root` is declared; GraphQL with push-down refuses an undeclared dotted path in `add_condition`. - **SurrealDB** re-tags a record id held as text on the id column and on record-link columns (typed `thing`/`record`, or read by one of the table's own has-one references) for every caller; any other column compares text as text. - **GraphQL** types each text operand by its column, so an integer column takes `_eq: 5`, and a non-integer text on one is an error before any request. - **REST, GraphQL and AWS** answer `fetch_window(0, n)` from one unpaged listing. - **AWS and cmd** apply request-parameter prefix filters (`PathPrefix`, S3 `prefix`, `logGroupNamePrefix`) to typed tables only; a YAML vista over them doesn't. - **REST, AWS and cmd client-side filters** pass a row lacking the field only for a `request_param` column, a `{placeholder}` or an undeclared field; a `hidden` column is a row field. - **cmd** answers `get(id)` only with the row keyed by `id`, or a single id-less row; on a narrowed set, `get` returns nothing for ids outside it. - **vantage-memory and faker** refuse a condition on a column the vista doesn't declare. Memory conforms seeded cells to their declared type and creates no table for an unknown traversal target. Faker counts a write only when it changed a row, and sims chain their handle calls. - **spacetimedb** refuses `IN`, `NOT IN` and `LIKE` (evaluated locally), advertises no writes on a keyless table, refuses writes through a narrowed handle and serves only the first window. - **REST** conditions accept declared columns and endpoint placeholders only; a non-scalar `Eq` value is refused, so it is evaluated locally. - **YAML flags reach the typed column.** `searchable`, `mandatory`, `label`, `indexed`, `hidden` and `request_param` all carry through `Column::from_spec`, and drivers with column flags (MongoDB, SurrealDB, GraphQL, SQL) show them on the Vista column. - **Unknown relations** are `vantage_vista::unknown_relation` on every driver: "unknown relation", `is_not_found()`. ## Checklist - [ ] no `*SpecResolver` alias; `Arc>` with `|name: &str|` closures - [ ] `RecordDraft` calls carry the resolver on the handle: `Handle::named(..).with_resolver(r)` - [ ] specs with `keys:` or column `references:` are checked against the traversals they now drive - [ ] has-many targets declare their foreign key column - [ ] the columns a search should read are flagged `searchable` - [ ] custom shells window a listing with `Page::from_listing` --- Source: https://vantage-ui.com/book/migration-2.0/downstream.html # Downstream: vantage-ui and vantage-actions This page is the work list for the port that follows 2.0: vantage-ui, its vantage-actions dependency (through vantage-lib), the skills bundle and the example apps. It is grouped by area and names files where the earlier phases found them. Nothing here is compiled against vantage-ui; the other pages hold the snippets. The file paths are relative to each repository and were found by read-only greps, so treat them as a starting point, not a complete list. ## vantage-actions - `src/row.rs`, `read_only_row`: the resolver moves onto the handle. `RecordDraft::from_row(Handle::named("row"), Some(no_table), Writes::Denied(denial.into()), id, values)` becomes `RecordDraft::from_row(Handle::named("row").with_resolver(no_table), Writes::Denied(denial.into()), id, values)`. - Any resolver closure it builds types its parameter: `Arc::new(|name: &str| ..)`. An untyped one fails with "implementation of `Fn` is not general enough". ## vantage-ui: resolvers and rhai - `crates/app/vantage-ui/src/infra/actions/run/body.rs`: `BodyRow::Writable` builds `Handle::named(table)`, attaches the `Option>` with `match resolver { Some(r) => handle.with_resolver(r), None => handle }`, and calls `RecordDraft::from_row(handle, Writes::Allowed, id, fields)`. - Every `SpecResolver` in type position is `Arc>`, and every closure behind one is `|name: &str|`. vantage-lib's callers need the same. - Quick filters (`quick-filters/script.rs`): `Handle::steps()` and `Step` are crate-private. Replace the steps loop with `let q = handle.query()?;` and derive the Dio with `dio.derive().with_query(&q).build().await?`. Quick filters refused `sort()` before; to keep that, reject `q.order.is_some()`. `ref()` and `limit()` errors now come from `query()`. - Form submit and the wizard proxy: `vantage_diorama::rhai::{cbor_to_dynamic, dynamic_to_cbor, record_to_map}` are `vantage_vista::rhai::…`; `DataVocab::read(..)` and friends are `DataVocab::new(Terminals::ReadWrite { limit, writes }).with_resolver(r)`. - Rhai `capabilities()` keys come from `Capability::iter()` (`insert`, `order`, `window`, …); the old `can_*` keys read as `()`. ## vantage-ui: Lens and Dio - The lens module: `Lens::new()` / `LensBuilder::new()` → `Lens::builder()`; `cache_at`, `cache_in_memory`, `cache_source` → `.cache(Cache::…)`; the hooks → `Loader`, `Refresher`, `writer`; `LensBuildError` → the error's `setting` context; `make_dio` / `make_dio_as` → `lens.dio(vista).build().await?`. - Its own cache TTL map → `Refresher::max_age(ttl)`. - Hand-built `-where-` cache keys go: a derived Dio's cache table is `name?`. Old redb tables are orphaned; sweep them with `CacheBackend::list_tables` / `drop_table` if needed. - `dio.vista()` → `dio.cached()` everywhere. - Every place that narrowed a Dio (`with_condition_eq`, the mutating `with_order`, quick filters) becomes one `dio.derive()…build().await?`. - `Dio::augment(resolver, ..)` → `lens.dio(v).augment(..)`, with each `Detail::Catalog(name)` as `Detail::Resolved { resolver, name }`; custom fetch closures take the listed row. - Entity: `dio.watch()` → `.live()` on the builder. - Inspection: `Dio::cache_table_name()` → `dio.diagnostics().await?.cache_table`. - Grids over REST sources without `Order` or `Search` fail on sort or search until their lens gives the paged loader an `or_eager` start and a `Refresher`; otherwise show the failure. - `!vista.can_fetch_page()`, used as "API-backed", is `!vista.capabilities().has(Capability::Window)`. ## vantage-ui: writes and events - The crud widget and adapters: facade writes return once confirmed or rolled back; drop the polling loops and handle the returned error. - `flash_insert` / `flash_patch` / `flash_delete` → `dio.flash(ChangeFlash::…)`; `handle_event`, `patched`, `notify_*`, `reconcile_values` → `apply_change` / `reconcile`. - `subscribe_events` consumers handle `RecordInserted` and `RecordRemoved`, not only `RecordChanged`, and may see `LoadFailed` with an empty range from a live follow. - Matches on `FlashKind`, `ChangeEvent` and `ServoStatus` (and every other diorama enum) need a `_ =>` arm. ## vantage-ui: sceneries and widgets - `SortDir` → `SortDirection`; `OpCondition` → `Condition`; builder `where_eq` / `where_op` → `filter`, `sort` → `order`; `set_sort` → `set_order(Option<..>)`; `set_filter_terms` → `set_filters(Vec)`; `set_shown_range` → the second range of `set_viewport`; `request_refresh` → `refresh_source`; `is_loading()` → `load_state() == LoadState::Loading`. - App routing: `CappedScenery` / `open_tail` → builder `limit` / `tail`. - App and form: `record_scenery(id).await`, `_sugared`, `_with` → `record_scenery(id)…open()`. - The inspector: `SceneryDiagnostic::refcount` → `shared_by`. - Show `LoadState::Failed(failure)` where a grid shows its empty state. - Value sceneries and dashboards: `value_scenery().count()` and the `AggregateLens` paths → `value_scenery(Count::rows())`, `dio.aggregate(..)`, imported from `vantage_diorama::aggregate`. ## vantage-ui: paths, features and diagnostics - `vantage_diorama::scenery::…` paths → the crate root. - `shell/view_summary.rs` reads `debug::num`, the one formatter that stays public; `dur`, `bytes` and `pct` are crate-private. - Enable `vantage-diorama/diagnostics` where the app opens the debug stream (`debug_datasource`), logs the session summary or reads `diagnostics()`. - Rust `ShapedShell` and `map_shell` call sites (form-tester, select-tester, the grid's `dio.rs`, select-vendored): `pub source` is gone; wrap with `map_shell`. - vantage-lib `slot.rs` and the test hosts: `vantage_memory::vista::Catalog` → `MemoryCatalog`. - The csv-dio and sqlite-dio loaders: struct literals of `VistaSpec`, `ColumnSpec`, `ReferenceSpec` → `VistaSpec::new(name)`, `ColumnSpec::new()`. ## vantage-ui: YAML, scripts and the skills bundle - Shape YAML capabilities: `fetch_next` → `cursor`, `fetch_window` → `window`; `count` and `fetch_page` are removed. - The `Preview.driver` key is gone from serialised previews; the MCP preview adds `driver` from `vista.driver_name()` itself. The skills that describe preview output change with it. - Record and servo `save()` → `flash()` in scripts and skills. - SQL scripts (`rhai:` sources, `base:` transforms, `mysql: { modify }`, reference `rhai:`, select `query:` options) prefix each builder function with `sql::`; SurrealDB scripts with `surreal::`, `me` → `surreal::me`; MongoDB's `oid`, `date`, `doc` with `mongo::`. `table("x")` as an identifier is `sql::ident("x")`. A missed call fails at build with "Function not found". - The skills bundle (`crates/domain/skills/resources/`, bump `metadata.version`): vantage-rhai, vantage-queries, vantage-dashboards, vantage-persistence-sqlite, -postgres, -mysql, -surrealdb, -rest, vantage-debugging, vantage-components and vantage-quality reference the old builder names, `can_*` capability keys or preview keys. - vantage-ui-examples: `apps/breg-bakery/table/bakery-surreal/*.yaml`, `apps/cashgpt/table/*.yaml` and `apps/cashgpt/page/*.yaml`, `apps/real-world/table/*.yaml` use the global builder names. - Has-many targets declare their foreign key column; a target that leaves it out fails traversal with an error naming the relation and the column. - Flag the columns a search should read `searchable`. - Untyped JSON columns under a `Where` aggregate need a declared type, or operands written in the cell's own type. - Generated schema descriptions (`page-schema-1.json`, `table-schema-1.json`) come from doc comments; regenerate them after the port. ## vantage-lib - Its `vista_loader/references.rs` lowers column `references:` sugar itself, naming the relation after the target (`author_id: { references: author }` gives `author`), where vantage-table and vantage-memory name it after the column (`author_id`). Pick one name before vantage-lib drops its copy for `vantage_table::table::spec::spec_references`. ## Checklist - [ ] vantage-actions `read_only_row` carries the resolver on the handle - [ ] vantage-ui compiles with no `vista()`, `make_dio`, `steps()` or `SortDir` - [ ] every diorama enum match in vantage-ui has a `_ =>` arm - [ ] the skills bundle and the example YAML use `flash()`, namespaced builders and the new capability keys, and `metadata.version` is bumped - [ ] the cache TTL map and `-where-` keys are deleted --- Source: https://vantage-ui.com/book/expressions.html # Expressions & Queries Vantage builds queries without string concatenation. The `vantage-expressions` crate provides a type-safe, composable expression system that works across all persistence backends — SQL, SurrealDB, MongoDB, CSV, and anything you add yourself. --- ## The core idea An `Expression` is a template string with typed parameters: ```rust # use vantage_sql::primitives::identifier::ident; # use vantage_sql::sqlite_expr; let expr = sqlite_expr!("SELECT {} FROM {} WHERE {} > {}", (ident("name")), (ident("product")), (ident("price")), 100i64); // → SELECT "name" FROM "product" WHERE "price" > ?1 (with 100 bound as i64) # assert_eq!(expr.preview(), r#"SELECT "name" FROM "product" WHERE "price" > 100"#); ``` Parameters are never interpolated into the string. They're carried separately, each tagged with a type marker from your persistence's type system. The bind layer uses these markers to call the right driver method — `bind_i64`, `bind_str`, `bind_bool` — no guessing, no silent coercion. (`preview()` renders the parameters inline for reading; it is not what is sent.) Three kinds of parameters: - **Scalar** — a typed value: `42i64`, `"hello"`, `true` - **Nested** — another expression, composed into the template - **Deferred** — a closure that executes later (cross-database resolution) --- ## Vendor macros Each persistence provides a convenience macro that produces `Expression` with the correct type wrapping: ```rust # use vantage_sql::{mysql_expr, postgres_expr, sqlite_expr}; # use vantage_surrealdb::surreal_expr; let e = sqlite_expr!("SELECT * FROM product WHERE price > {}", 100i64); let e = surreal_expr!("SELECT * FROM product WHERE price > {}", 100i64); let e = postgres_expr!("SELECT * FROM product WHERE price > {}", 100i64); let e = mysql_expr!("SELECT * FROM product WHERE price > {}", 100i64); # let _ = e; ``` Same syntax, different type universes. The compiler ensures you can't accidentally mix a `Expression` into a SurrealDB query. --- ## Composing expressions Expressions nest naturally. Parenthesised arguments call `.expr()` automatically: ```rust # use vantage_sql::primitives::identifier::ident; # use vantage_sql::sqlite_expr; let condition = sqlite_expr!("{} > {}", (ident("price")), 100i64); let query = sqlite_expr!("SELECT {} FROM {} WHERE {}", (ident("name")), (ident("product")), (condition)); # assert_eq!(query.preview(), r#"SELECT "name" FROM "product" WHERE "price" > 100"#); ``` The `ExpressionFlattener` collapses all nesting into a single flat template with positional parameters — each one still carrying its type marker. For building lists (e.g. multi-row INSERT), use `Expression::from_vec`: ```rust # use vantage_expressions::Expression; # use vantage_sql::sqlite_expr; let row1 = sqlite_expr!("({}, {})", "tart", 220i64); let row2 = sqlite_expr!("({}, {})", "pie", 299i64); let rows = Expression::from_vec(vec![row1, row2], ", "); # let _ = rows; ``` --- ## Identifier quoting SQL identifiers need quoting — and each database uses different quote characters. The `Identifier` struct handles this by implementing `Expressive` for each backend type: ```rust # use vantage_sql::primitives::identifier::ident; # use vantage_sql::{mysql_expr, sqlite_expr}; // Quoting adapts to the expression's type context sqlite_expr!("SELECT {}", (ident("name"))); // → SELECT "name" mysql_expr!("SELECT {}", (ident("name"))); // → SELECT `name` // Qualified identifiers sqlite_expr!("SELECT {}", (ident("name").dot_of("u"))); // → SELECT "u"."name" // Aliases mysql_expr!("SELECT {}", (ident("name").with_alias("n"))); // → SELECT `name` AS `n` ``` --- ## ExprDataSource — executing expressions The `ExprDataSource` trait connects expressions to a live database. Both calls below read a `product` table, so the block is compiled but not run: ```rust,no_run # use vantage_expressions::{AssociatedQueryable, ExprDataSource}; # use vantage_sql::sqlite::{AnySqliteType, SqliteDB}; # use vantage_sql::sqlite_expr; # use vantage_core::Context; # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:products.db").await.context("connect")?; # let expr = sqlite_expr!("SELECT name FROM product"); // Execute directly let result: AnySqliteType = db.execute(&expr).await?; // Associate with an expected return type let count: i64 = db.associate::(sqlite_expr!("SELECT COUNT(*) FROM product")) .get().await?; # let _ = (result, count); # Ok(()) # } ``` `AssociatedExpression<'a, DS, T, R>` carries both the expression and a reference to the datasource. Call `.get()` (from the `AssociatedQueryable` trait) to execute and convert in one step. The return type `R` is checked at compile time. --- ## Deferred expressions — cross-database values Sometimes a query on one database needs a value from another. `defer()` wraps a query as a closure that resolves at execution time. The block runs both queries against tables it doesn't create, so it is compiled but not run: ```rust,no_run # use vantage_expressions::{ExprDataSource, Expression, ExpressiveEnum}; # use vantage_sql::sqlite::{AnySqliteType, SqliteDB}; # use vantage_sql::sqlite_expr; # use vantage_core::Context; # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let config_db = SqliteDB::connect("sqlite:config.db").await.context("connect")?; # let shop_db = SqliteDB::connect("sqlite:shop.db").await.context("connect")?; // Query config_db for a threshold — but don't execute yet let threshold = config_db.defer( sqlite_expr!("SELECT value FROM config WHERE key = {}", "min_price") ); // Use the deferred value in a query against shop_db let expensive = Expression::::new( "SELECT name FROM product WHERE price >= {}", vec![ExpressiveEnum::Deferred(threshold)], ); let result = shop_db.execute(&expensive).await?; // 1. Resolves deferred → calls config_db, gets 150 // 2. Binds 150 as a scalar parameter // 3. Executes against shop_db # let _ = result; # Ok(()) # } ``` This is not a subquery — the deferred query runs first, produces a concrete value, and that value gets bound as a regular parameter. --- ## Selectable — the query builder interface The `Selectable` trait is the standard interface for building SELECT queries. Each persistence provides its own SELECT struct (`SqliteSelect`, `SurrealSelect`, `PostgresSelect`) implementing this trait: ```rust # use vantage_expressions::{Order, Selectable}; # use vantage_sql::primitives::identifier::ident; # use vantage_sql::sqlite::statements::SqliteSelect; # use vantage_sql::sqlite_expr; let select = SqliteSelect::new() .with_source("product") .with_field("name") .with_field("price") .with_condition(sqlite_expr!("{} = {}", (ident("is_deleted")), false)) .with_order(sqlite_expr!("{}", (ident("price"))), Order::Asc) .with_limit(Some(10), None); # let _ = select; ``` **Builder methods** come free from the trait — `with_source`, `with_field`, `with_condition`, `with_order` (an expression and an `Order`: `Order::Asc`, `Order::Desc`, optionally `.nulls_last()`), `with_limit`. You only implement the mutating methods (`add_field`, `add_where_condition`, etc.). Aggregate shortcuts clone the query and replace fields: ```rust # use vantage_expressions::Selectable; # use vantage_sql::primitives::identifier::ident; # use vantage_sql::sqlite::statements::SqliteSelect; # use vantage_sql::sqlite_expr; # let select = SqliteSelect::new().with_source("product").with_field("name"); let count = select.as_count(); // SELECT COUNT(*) FROM ... let total = select.as_sum(sqlite_expr!("{}", (ident("price")))); // SELECT SUM("price") FROM ... # let _ = (count, total); ``` --- ## SelectableDataSource — wiring it up `SelectableDataSource` connects the query builder to execution: it names the backend's SELECT type, hands out an empty one, and runs a finished one. Here it is for `ShopDB`, a wrapper around a SQLite connection, mirroring the impl `vantage-sql` ships for `SqliteDB`: ```rust # use vantage_core::Result; # use vantage_expressions::traits::datasource::DataSource; # use vantage_expressions::{ExprDataSource, Expressive, SelectableDataSource}; # use vantage_sql::prelude::SqliteCondition; # use vantage_sql::sqlite::statements::SqliteSelect; # use vantage_sql::sqlite::{AnySqliteType, SqliteDB}; # # pub struct ShopDB(SqliteDB); # # impl DataSource for ShopDB {} # impl SelectableDataSource for ShopDB { type Select = SqliteSelect; fn select(&self) -> Self::Select { SqliteSelect::new() } async fn execute_select(&self, select: &Self::Select) -> Result> { let result = self.0.execute(&select.expr()).await?; // The result is an array of row objects: one value per row. match result.value() { ciborium::Value::Array(rows) => { Ok(rows.iter().map(|row| AnySqliteType::untyped(row.clone())).collect()) } _ => Ok(vec![result]), } } } ``` Once implemented, `table.select()` returns your vendor-specific builder pre-populated with the table's columns, conditions, and ordering — ready for execution or further customisation. The full walkthrough for a new backend is [Step 4: Query Builder](./new-persistence/step4-query-builder.md). --- ## Expressive trait Anything that implements `Expressive` can be used inside an expression. This includes: - **Columns** — `table["price"]` - **Operations** — `table["price"].gt(100)` - **Identifiers** — `ident("name")` - **Query builders** — `select.expr()` - **Sort orders** — `table["name"].desc()` - **Scalar values** — `42i64`, `"hello"`, `true` - **Closures** — that's what `defer()` returns You can implement `Expressive` for your own types to make them composable into the expression system. Because conditions are themselves `Expressive`, operations **chain across type boundaries**: `price.gt(10)` returns a `SqliteCondition`, and `.eq(false)` works on it — reading "(price > 10) = 0". Type safety is enforced on the *first* operation, at the column level; once you hold a condition, any value compatible with the backend's any-type is accepted (`price.gt(10).eq("foobar")` compiles by design). Columns of the same type compare directly (`price.eq(price.clone())`), and operations take ownership of their arguments — clone a column you plan to reuse across conditions. ```rust # use vantage_sql::prelude::*; let price = Column::::new("price"); let cond = price.gt(10i64).eq(false); assert_eq!(cond.into_expr().preview(), "price > 10 = 0"); ``` ## Checklist To build and run a query with expressions: 1. **Pick the vendor macro** for your backend — `sqlite_expr!`, `postgres_expr!`, `mysql_expr!`, `surreal_expr!` — so every parameter carries that backend's type marker. 2. **Compose** with parenthesised arguments, `ident(..)` for names, and `Expression::from_vec` for lists. 3. **Build SELECTs** through `Selectable`: `with_source`, `with_field`, `with_condition`, `with_order(expr, Order::Asc)`, `with_limit`, and the `as_count`/`as_sum` shortcuts. 4. **Execute** through `ExprDataSource`: `execute` for a raw result, `associate::(..).get()` for a typed one, `defer` for a value one database computes for another. --- Source: https://vantage-ui.com/book/record-lifecycle.html # Records: Traversal, Invariants & Hooks Once a backend implements `TableSource` (see [Adding a New Persistence](./new-persistence.md)), `vantage-table` gives every `Table` a uniform write pipeline. On each write the record flows through three stages before it reaches the datasource: ```text record → lifecycle hooks (before) → set invariants → datasource write → hooks (after) ``` This page covers the consumer-facing features built on that pipeline: traversing a relation from a loaded record, the foreign-key **invariants** that traversal sets up, and the **lifecycle hooks** you can attach for audit, validation, soft-delete, and after-effects. They are backend-agnostic — everything here works the same on SQLite, Postgres, MySQL, Mongo, etc. ## Traversing from a loaded record: `get_ref` `Table::get_ref_as` / `get_ref_from_row` traverse a relation from a *table* (both covered in [Relations & Traversal](relations/traversal.md)). The `GetRefExt` trait adds the record-level equivalent: traverse straight from a loaded `ActiveEntity` (typed) or `ActiveRecord` (untyped). The block reads and writes a launch database, so it is compiled but not run: ```rust,no_run # use vantage_dataset::prelude::ActiveEntitySet; # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # pub struct Launch { # pub name: String, # } # # impl Launch { # pub fn table(db: SqliteDB) -> Table { # Table::new("launch", db) # .with_id_column("id") # .with_column_of::("name") # .with_many("launch_crew", "launch_id", LaunchCrew::table) # } # } # # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # pub struct LaunchCrew { # pub launch_id: Option, # pub astronaut_id: Option, # pub role: Option, # } # # impl LaunchCrew { # pub fn table(db: SqliteDB) -> Table { # Table::new("launch_crew", db) # .with_id_column("id") # .with_column_of::("launch_id") # .with_column_of::("astronaut_id") # .with_column_of::("role") # } # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:launches.db").await.context("connect")?; # let launches = Launch::table(db); # let (id, a) = ("apollo-11", "armstrong".to_string()); use vantage_table::prelude::GetRefExt; let launch = launches.get_entity(id).await?.expect("launch"); // A child set scoped to this launch — and carrying its foreign key (see invariants below). let crew = launch.get_ref::("launch_crew")?; crew.insert_return_id(&LaunchCrew { astronaut_id: Some(a), role: Some("Pilot".into()), ..Default::default() }).await?; # Ok(()) # } ``` `get_ref::(relation)` returns a `Table` scoped to the parent. For a typed `ActiveEntity` the entity's id is injected into the row before traversal (so has-many relations resolve); an untyped `ActiveRecord` already holds the raw row and forwards directly. The same method exists on both handles — `GetRefExt` carries blanket implementations for each, generic over any `TableSource` (the handles themselves live in `vantage-dataset` and know nothing about `Table`, which is why this arrives as an extension trait). ## Safe writes Every write is retry-safe, and a write through a narrowed table only touches rows in it. An unnarrowed table is the only way to write anywhere. The set is the table's conditions, equality and operator alike. For a Vista or a Rhai handle it is every narrowing step that decides membership. | operation | row in the set | row outside the set | no such row | | --- | --- | --- | --- | | `delete(id)` | deleted; `Ok(())` | left alone; `Ok(())` | `Ok(())` | | `insert(id, rec)` | nothing written; returns the stored row | `Conflict`, where the backend can tell | inserted; the record must fit the set | | `patch(id, partial)` | applied; `Conflict` if the row would leave the set | `NotFound` | `NotFound` | | `replace(id, rec)` | applied; `Conflict` if the row would leave the set | `Conflict`, where the backend can tell | created; the record must fit the set | | import | `skipped` | `rejected` | inserted; `rejected` if it doesn't fit the set | | `delete_all()` | deletes exactly the set | — | — | A write that isn't allowed fails with `ErrorKind::Unsupported`: writes are denied, the capability is off, or the backend can't apply one of the set's conditions. In Rhai, `delete` returns `false` instead and the other verbs throw. The data source enforces the set: SQL adds the conditions to the statement and checks a written row with a one-row probe; SurrealDB and MongoDB do the same in their own query language; memory, redb, DynamoDB and Dio evaluate the conditions on the row. A source that can't apply or evaluate a condition refuses the write. Ids: a record without an id gets a client-made UUIDv7 before the first attempt (Rhai `insert`, a new record, Servo `IdStrategy::Uuid`, [`IdGenerator::UuidV7`](vantage_table::table::IdGenerator)), so a retry lands on the idempotent insert. A table whose ids the server makes declares it ([`Table::with_auto_id`](vantage_table::table::Table::with_auto_id), YAML `flags: [id, auto]`) and inserts through `insert_return_id`, which is not retry-safe. ```admonish warning title="Eventually-consistent backends" A backend that can't see a row before it writes can't tell an insert of an id held outside the set from a new row. Such a backend documents what it does instead; vantage never overwrites silently. DynamoDB inserts with a conditional put, so it does detect the clash. ``` Each write lands on its own unless it runs inside a [transaction](./transactions.md). `replace` and `patch` check the row and then write it, so a concurrent writer can change the row between the two. On Postgres and MySQL, a locked read inside a transaction ([`Vista::lock`](vantage_vista::Vista::lock)) holds the row from your read to the commit. MySQL can't run an `UPDATE` or `DELETE` whose condition subqueries the table being written (error 1093), so a set defined that way can't be written through on MySQL. ## Set invariants A table narrowed by a literal `column = value` is a **set**, and every row written into it must *conform* to that definition. Vantage records the pair as an **invariant** and enforces it on write. The most common source is relationship traversal: `launch.get_ref::("launch_crew")` narrows the child set by `launch_id = `, so an inserted crew row's `launch_id` is filled automatically. Invariants are registered automatically wherever the scope is a plain `column = value`: - `Table::with_id(id)` — narrows by the id column. - has-many / has-one traversal (`Reference::resolve_from_row`) — narrows by the foreign key. - **every literal `column = value` passed to `add_condition` / `with_condition`**. - a Vista's `add_condition` with `FilterOp::Eq`, which `Vista::get_ref` uses for each join key. - Rhai `where(col, value)`. Expression scopes never register an invariant. You can also set one explicitly with `Table::with_invariant(column, value)` / `add_invariant`. On insert and replace, each invariant column is resolved by a four-way rule: | record's value for the column | result | | --- | --- | | absent | set to the invariant value | | present but null | set to the invariant value | | present and equal | kept | | present and **conflicting** | the write is rejected with an error | So a child row inserted through a relation needs no foreign key (it's filled), may state the matching one (kept), but cannot smuggle in a *different* one (error — it doesn't belong to this set). A patch never fills an invariant column. A present value must match it; a different or null value is rejected (`Conflict`), because it would move the row out of the set. Moving a row between sets goes through a handle whose set holds both. ```rust,no_run # use vantage_dataset::prelude::ActiveEntitySet; # use vantage_sql::prelude::*; # use vantage_table::prelude::GetRefExt; # use vantage_types::prelude::*; # # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # pub struct Launch { # pub name: String, # } # # impl Launch { # pub fn table(db: SqliteDB) -> Table { # Table::new("launch", db) # .with_id_column("id") # .with_column_of::("name") # .with_many("launch_crew", "launch_id", LaunchCrew::table) # } # } # # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # pub struct LaunchCrew { # pub launch_id: Option, # pub astronaut_id: Option, # pub role: Option, # } # # impl LaunchCrew { # pub fn table(db: SqliteDB) -> Table { # Table::new("launch_crew", db) # .with_id_column("id") # .with_column_of::("launch_id") # .with_column_of::("astronaut_id") # .with_column_of::("role") # } # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:launches.db").await.context("connect")?; # let launches = Launch::table(db); # let launch = launches.get_entity("apollo-11").await?.expect("launch"); # let a = "armstrong".to_string(); let crew = launch.get_ref::("launch_crew")?; // launch_id absent → filled from the set: crew.insert_return_id(&LaunchCrew { astronaut_id: Some(a), role: Some("Pilot".into()), ..Default::default() }).await?; // launch_id set to a *different* launch → Err: assert!(crew.insert_return_id(&LaunchCrew { launch_id: Some("other".into()), ..Default::default() }).await.is_err()); # Ok(()) # } ``` ```admonish info title="Backends and InvariantValue" Enforcement needs two operations on a backend's value type — a null check and an equality check — provided by the `InvariantValue` trait (`vantage-types`). The `vantage_type_system!` macro emits it for every generated `Any*Type`; pass a `null_when:` pattern (e.g. `null_when: ciborium::Value::Null` for the SQL backends) so genuine nulls are recognised. Non-nullable value types (e.g. CSV's `String`) simply never match the null branch. ``` ## Lifecycle hooks Attach async callbacks around writes with `Table::with_hook(Hook::…)`. Hooks are how audit stamps, normalization, validation, soft-delete, and after-effects are expressed — generically, on any backend. ```rust # use vantage_sql::prelude::*; # use vantage_table::table::{BeforeCx, BoxFuture}; # use vantage_types::prelude::*; # # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # pub struct Launch { # pub name: String, # } # # impl Launch { # pub fn table(db: SqliteDB) -> Table { # Table::new("launch", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("created") # } # } # # fn stamp_created(cx: BeforeCx<'_, SqliteDB>) -> BoxFuture<'_, VantageResult<()>> { # Box::pin(async move { # cx.record.insert("created".into(), AnySqliteType::new("2026-10-14".to_string())); # Ok(()) # }) # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite::memory:").await.context("connect")?; use vantage_table::prelude::{Hook, Phase}; use vantage_table::table::before_hook; let launches = Launch::table(db).with_hook(Hook::BeforeInsert(Phase::Populate, before_hook(stamp_created))); # let _ = launches; # Ok(()) # } ``` ### The `Hook` variants One enum carries a placement-specific closure. Each hook receives one context struct holding what's available at its stage, so new fields can be added without breaking existing hooks: | variant | when | receives | may | | --- | --- | --- | --- | | `BeforeInsert(Phase, _)` | before an insert | `BeforeCx { record: &mut Record, id: None, stored: None, context, table }` | mutate / `Err` to cancel | | `BeforeUpdate(Phase, _)` | before replace/patch | `BeforeCx`, with `id`, the `stored` row and `patch` (a patch's `record` holds only the patched fields; a replace's is the whole new row) | mutate / `Err` to cancel | | `BeforeSave(Phase, _)` | before insert **and** update | `BeforeCx` | mutate / `Err` to cancel | | `BeforeDelete(_)` | before a delete | `DeleteCx { id, record (former), context, table }` | `Err` to veto, or `HookReturn::Handled` to take over | | `AfterInsert` / `AfterUpdate` / `AfterSave` | after the write commits | `AfterCx { id, record, before, context, table }` | side-effects | | `AfterDelete(_)` | after a delete | `AfterCx` (`record` is the former row) | side-effects | | `BeforeDeleteAll(_)` / `AfterDeleteAll(_)` | around `delete_all` | `DeleteAllCx { context, table }` | `Err` to cancel (before) / side-effects | | `BeforeSelect(_)` | before a read: list, get, stream, count, sum/max/min | `SelectCx { context, table }` | `Err` to refuse the read | The `table` handed to a hook is entity-erased, so it can traverse relations (`get_ref`) and reach the datasource — enough for cross-row validation and after-effects. `context` is the [`WriteContext`](vantage_table::write_context::WriteContext) in scope for the write, if any. ### Ordering and control flow - **Before-write hooks run ahead of invariant enforcement**, ordered by `Phase`: `Normalize` → `Populate` (the default) → `Validate`, then registration order within a phase. So normalize inputs, then derive/stamp fields, then validate the final record. - **Returning `Err` from any before-hook cancels the operation** before anything is written. - **`BeforeDelete` returning `HookReturn::Handled`** skips the real delete and reports success — this is how soft-delete works (patch a `deleted` marker, return `Handled`). A delete with hooks loads the row once so before/after hooks see its contents. A delete with hooks on a row outside the set runs no hooks and deletes nothing. - **An update with after-update hooks reads the row first**, through the set, so `cx.before` holds the row as it was; without such hooks there is no extra read. - **`delete_all` runs `BeforeDeleteAll` / `AfterDeleteAll` once for the whole set**, never the per-row delete hooks. The hook gets the table, whose `conditions()` describe the set; an error from the before hook cancels it. A soft-delete `BeforeDelete` doesn't cover it: `delete_all` still removes the rows, so a soft-deleting table needs a `BeforeDeleteAll` that refuses it. - After-hooks run for side-effects only. Outside a transaction they run right after the write, and a failing after-hook returns its error without undoing the write. Inside a [transaction](./transactions.md#after-hooks-wait-for-the-commit) they wait for the commit: they run once it succeeds, under the writer's `WriteContext`, a rollback drops them, and a failure is logged rather than returned, since the rows are already committed. - Design after-effects to be safe to run again anyway. A transaction doesn't cover writes to a datasource without transactions (SpacetimeDB, the log writer, MongoDB standalone, SurrealDB before 3 write directly), and a retry after an after-hook failure, or after a commit that failed part-way across databases, runs them a second time. ### Read hooks `BeforeSelect` is the one hook on the read side. It runs before every read that returns rows or facts about rows, so a rule like "only signed-in callers read invoices" lives on the table rather than in every handler that lists it. A stream yields the hook's error as its first item. The hook reads `cx.context` (the same [`WriteContext`](vantage_table::write_context::WriteContext) writes see, so one scope covers a request's reads and writes) and `cx.table.conditions()`, which describe the set about to be read: ```rust use vantage_sql::sqlite::SqliteDB; use vantage_table::table::{SelectCx, select_hook}; let signed_in_only = select_hook(|cx: SelectCx<'_, SqliteDB>| { Box::pin(async move { match cx.context.and_then(|c| c.principal.as_ref()) { Some(_) => Ok(()), None => Err(vantage_core::error!("sign in to read this table")), } }) }); # let _ = signed_in_only; ``` It does **not** run when the table is only embedded in another query: `related_in_condition`, `column_table_values_expr`, `get_*_query` and `get_expr_*` build expressions, not reads of this table. A rule that must hold inside a subquery belongs in the table's conditions, added when the table is built (see the build injector in [Config-Driven Vistas](./config-driven-vistas.md#the-build-injector)). A vista shell that reads the data source directly, without going through `Table` (a SQL count, a REST window, a live watch), calls [`Table::check_read`](vantage_table::table::Table::check_read) first, so the hook guards those reads too. Reads a write makes run the hook as well, so a refusing `BeforeSelect` also refuses: - an update once the table has an after-update hook, since the update reads `cx.before` first; - a delete once the table has delete hooks, since the delete reads the row for them; - a write whose backend reads the row back after it (an insert, replace or patch that returns the stored row). So registering an audit hook can change whether a caller allowed to write but not to read can update. That is deliberate: a refused read must not leak the former row into a hook. Reads served by a Dio's cache are the exception; see the Dio note below. ### Writing a hook Hooks are boxed async closures over their context struct. `before_hook`, `before_delete_hook` and `after_hook` wrap a closure or a free `fn` into the hook type; their bounds guide closure inference: ```rust use vantage_types::Record; use vantage_sql::sqlite::{AnySqliteType, SqliteDB}; use vantage_table::prelude::HookReturn; use vantage_table::table::{BeforeCx, DeleteCx, before_delete_hook, before_hook}; # use vantage_dataset::WritableValueSet; # # fn now_iso8601() -> String { # "2026-10-14T09:00:00Z".to_string() # } // before-insert: stamp an audit field let stamp_created = before_hook(|cx: BeforeCx<'_, SqliteDB>| { Box::pin(async move { cx.record.insert("created".into(), AnySqliteType::new(now_iso8601())); Ok(()) }) }); // before-delete: soft-delete instead of removing the row let soft_delete = before_delete_hook(|cx: DeleteCx<'_, SqliteDB>| { Box::pin(async move { let mut patch = Record::new(); patch.insert("deleted".into(), AnySqliteType::new(now_iso8601())); cx.table.patch_value(cx.id.clone(), &patch).await?; Ok(HookReturn::Handled) }) }); # let _ = (stamp_created, soft_delete); ``` Who made a write comes from the [`WriteContext`](vantage_table::write_context::WriteContext): a host wraps its work in `WriteContext::new().with_principal(p).with_group(g).scope(fut)`, and every write awaited inside it hands the context to its hooks as `cx.context`. Outside a scope, `cx.context` is `None`. The context and the hook context structs are `#[non_exhaustive]`: build a `WriteContext` with `new()` and the `with_*` builders, and read a `cx` by field or destructure it with `..`, so a field added later breaks neither. Writes through a Dio need one more step, because a Dio runs them on its own worker task where the caller's scope doesn't reach. [`Dio::flash`](vantage_diorama::Dio::flash) records the context in scope where it is called on the [`ChangeFlash`](vantage_diorama::ChangeFlash), and the worker runs the write inside it, so the master table's hooks (and a Lens writer) see the same context. `ChangeFlash::with_context` sets one explicitly. A `clear` the Dio runs as one delete per row passes its context to each of those deletes. Reads are another matter. A Dio's `list`, `count` and window serve its cache, which the loader fills from its own task with no `WriteContext` in scope, so reads through a Dio don't run the master table's `BeforeSelect` per caller. A per-caller read rule belongs in the conditions of the table the Dio is built over, or in a Dio per principal. --- Source: https://vantage-ui.com/book/transactions.html # Transactions Moving 30 from Alice's account to Bob's takes two patches. Without a transaction, if the second one fails, Bob has the money and Alice still has it too. A [`Transaction`](vantage_vista::Transaction) makes the two patches land together or not at all, on every backend that has transactions, through the same `Vista` API you already write with. This chapter is the Rust side: the handle, what each backend does with it, what happens across several databases, and what a driver implements to take part. Rhai's `atomic { }` block is built on the same handle and gets one short section near the end. --- ## A transfer Wrap the Vistas you write through, write, commit. The example runs on the in-memory store, where `account` is narrowed to `balance >= 0`, so a patch that would overdraw fails: ```rust # use ciborium::Value as CborValue; # use vantage_core::{Result, error}; # use vantage_memory::{MemoryStore, MemoryVistaFactory}; # use vantage_types::Record; # use vantage_vista::prelude::*; # use vantage_vista::VistaFactory; # const ACCOUNT: &str = r#" # name: account # id_column: id # columns: # id: { type: string, flags: [id] } # balance: { type: integer } # "#; # fn balance(n: i64) -> Record { # [("balance".to_string(), CborValue::Integer(n.into()))].into_iter().collect() # } # async fn balance_of(accounts: &Vista, id: &str) -> Result { # let row = accounts.get_value(id).await?.ok_or_else(|| error!("no such account"))?; # match row.get("balance") { # Some(CborValue::Integer(n)) => Ok(i64::try_from(*n).unwrap_or_default()), # _ => Err(error!("no balance")), # } # } use vantage_vista::{Committed, Transaction}; async fn transfer(accounts: &Vista, from: &str, to: &str, amount: i64) -> Result { let tx = Transaction::new(); let accounts = tx.wrap(accounts); let (a, b) = (balance_of(&accounts, from).await?, balance_of(&accounts, to).await?); accounts.patch_value(to, &balance(b + amount)).await?; accounts.patch_value(from, &balance(a - amount)).await?; // overdrawn: Err, and `?` drops `tx` tx.commit().await } # #[tokio::main(flavor = "current_thread")] # async fn main() -> Result<()> { # let factory = MemoryVistaFactory::new(MemoryStore::new()); # let accounts = factory # .from_yaml(ACCOUNT)? # .with_condition("balance", FilterOp::Gte, CborValue::Integer(0.into()))?; # accounts.insert_value("alice", &balance(100)).await?; # accounts.insert_value("bob", &balance(100)).await?; let committed = transfer(&accounts, "alice", "bob", 30).await?; assert_eq!(committed.written, ["account"]); assert!(transfer(&accounts, "alice", "bob", 500).await.is_err()); assert_eq!(balance_of(&accounts, "bob").await?, 130); // the +500 never landed # assert_eq!(balance_of(&accounts, "alice").await?, 70); # Ok(()) # } ``` The second transfer patches Bob first, then fails on Alice. The early return drops the transaction and the wrapped Vista, and dropping them rolls back, so Bob's patch is gone too. ## The handle [`Transaction::new`](vantage_vista::Transaction::new) begins nothing. A transaction is a shared handle (cheap to clone) that collects the backends its Vistas touch. - [`tx.wrap(&vista)`](vantage_vista::Transaction::wrap) returns the Vista reading and writing through `tx`. Conditions, sort, hooks and references carry over. Every Vista it hands back from `get_ref` is wrapped too, so a child row inserted through a relation is in the same transaction. Live streams (`Subscribe`, `Cursor`) are switched off on the wrapped Vista: they would outlive the transaction. - [`tx.resolver(base)`](vantage_vista::Transaction::resolver) wraps a whole [`VistaResolver`](vantage_vista::VistaResolver): every Vista it resolves or adopts comes back wrapped. A host hands it to code that looks tables up by name. - [`tx.commit()`](vantage_vista::Transaction::commit) commits every backend that joined and returns [`Committed`](vantage_vista::Committed): `written` lists the tables written through the transaction, in first-written order, and `unprotected` the tables written directly by datasources without transactions (see [below](#datasources-without-transactions)). - [`tx.rollback()`](vantage_vista::Transaction::rollback) rolls them back. It is always `Ok`, and calling it on a finished transaction does nothing, so cleanup paths can call it freely. - Dropping the transaction and every Vista wrapped in it rolls back each backend that joined. Each one's open transaction rolls back when its handle drops (sqlx queues a `ROLLBACK`, MongoDB aborts the session, SurrealDB closes the connection). Writes by datasources without transactions have already landed and stay. A backend's own transaction starts at the first statement, not at `wrap`: wrapping a Vista you end up not using costs nothing. After commit or rollback, a wrapped Vista answers every call with `transaction finished`. ```rust # use std::sync::Arc; # use vantage_vista::mocks::MockShell; # use vantage_vista::{Transaction, Vista, VistaResolver}; # fn main() -> vantage_core::Result<()> { let base: Arc = Arc::new(|name: &str| Ok(Vista::new(name, MockShell::new()))); let tx = Transaction::new(); let tables = tx.resolver(base); let orders = tables.vista("order")?; // wrapped in `tx` # let _ = orders; # Ok(()) # } ``` A Vista whose shell can't take part still wraps without an error. The refusal comes at its first use: `"" can't take part in a transaction`. `wrap` doesn't return a `Result` because a host wraps whole resolvers up front, and most of the tables it wraps are never touched. ### A failed statement fails the commit When any statement inside the transaction fails, `commit` rolls back and returns `a statement failed inside the transaction; rolled back`. That includes an error you caught and moved past, and a statement dropped mid-flight by a timeout or a `select!`, which the server may still have run. #### Why not commit what did succeed? Postgres turns a `COMMIT` after an error into a rollback. MySQL, SQLite and SurrealDB don't: the failed statement is undone on its own, and the rest commit. Code that caught the error might then commit half of what it meant to write, and the result would depend on the database. Vantage keeps one rule for all of them. If you want to try a write that may fail and carry on regardless, do it outside the transaction. The in-memory store goes further: any failed operation fails its commit, including a refusal from the Vista layer (a write outside the set, say) that never reached the store. ## After-hooks wait for the commit A table's after-hooks (`AfterInsert`, `AfterUpdate`, `AfterSave`, `AfterDelete`, `AfterDeleteAll`; see [Records](./record-lifecycle.md#lifecycle-hooks)) don't run at the write inside a transaction. They run once the commit succeeds, in order, outside the transaction, under the [`WriteContext`](vantage_table::write_context::WriteContext) the write had. A rollback drops them. An after-hook that fails at that point is logged with `tracing::warn!`: the rows are already committed, so `commit` returns `Ok`. Before-hooks run at the write, inside the transaction, as they always do. A before-hook that returns `Err` fails the write. If you let the error leave the transaction, it rolls back. If you catch it and commit, the commit goes ahead on SQL, MongoDB and SurrealDB, because only a failed database statement fails the commit there. On memory, any failed operation on the store fails it, a refusal the caller caught included; a before-hook error stops the write before it reaches the store, so a caught one doesn't. ## Locking reads At Postgres's default isolation (read committed), a read-then-write inside a transaction can still race another writer: both read 10, both write 9. [`Vista::lock`](vantage_vista::Vista::lock) makes the wrapped Vista's reads lock the rows they return until the transaction ends: ```rust,no_run # use vantage_vista::prelude::*; # use vantage_vista::Transaction; # async fn reserve(stock: &Vista) -> vantage_core::Result<()> { let tx = Transaction::new(); let mut stock = tx.wrap(stock); stock.lock()?; let row = stock.get_value("sku-1").await?; // SELECT … FOR UPDATE on Postgres and MySQL // … patch the row from what was read # let _ = row; tx.commit().await?; # Ok(()) # } ``` - Postgres and MySQL read `FOR UPDATE`. A second locked reader of the row waits until the first transaction ends, then sees its write. - SQLite does nothing more: its transactions begin with `BEGIN IMMEDIATE`, which already holds the database's write lock. - A SQL Vista that isn't wrapped refuses with `lock() needs a transaction`. - Other backends refuse with `this shell can't lock rows`. There is no isolation setting. Each database runs at its own default: read committed on Postgres, repeatable read on MySQL, MongoDB's default read concern, serializable on SQLite. `lock()` is the tool for the read-then-write case. ## What each backend does | backend | inside a transaction | notes | | --- | --- | --- | | Postgres, MySQL | one connection from the pool, `BEGIN` | deadlocks are reported by the server and fail the write | | SQLite | one connection, `BEGIN IMMEDIATE` | writers queue on the file lock instead of deadlocking | | MongoDB | a session's transaction | replica sets and sharded clusters only | | SurrealDB | an interactive transaction on its own connection | SurrealDB 3 and later | | memory | a claim on the store and copies of the touched tables | see below | | SpacetimeDB, log-writer | write directly | listed in `unprotected` | | CSV, REST, GraphQL, Kubernetes, AWS, cmd | direct, read-only | a transaction only reads them | **MongoDB** needs a replica set (a single-node one is fine) for multi-document transactions. [`MongoDB::connect`](vantage_mongodb::MongoDB::connect) asks the server with `hello`; a standalone server then writes directly, as a datasource without transactions. A handle built with `MongoDB::new` doesn't ask, so its Vistas join, and on a standalone server the first operation in the transaction fails. A write to a document another transaction holds fails at once with a `WriteConflict`, which surfaces as an error and rolls the transaction back. Nothing retries it. **SurrealDB** runs the transaction with the `begin`, `commit` and `cancel` RPCs on a connection of its own: every clone of a `SurrealDB` shares one socket, and a transaction kept there would hold it for the whole block. That needs a client made by `SurrealConnection::connect`, which keeps the connection settings to open a second socket, and a SurrealDB 3+ server. Any other client writes directly. SurrealDB is optimistic: two transactions writing the same record both go ahead, and the second to commit fails with a conflict. **Memory** has no transactions of its own, so the driver builds one per store. On first use, read or write, the transaction claims the store. Another transaction on the same store waits at its own first use, for up to 2 seconds (`MemoryStore::with_transaction_wait`), then fails with `the memory store is busy with another transaction`. Each touched table is copied, and the transaction reads and writes the copy. Commit replays the transaction's row changes onto the store's tables as they are by then, so a write made outside meanwhile stays. An insert whose id was taken outside meanwhile fails the commit, with nothing applied. Callers outside a transaction never wait on the claim. Two transactions on one store run from the same task do: the second waits for a claim the first holds until it finishes, and gets the busy error. **DynamoDB and redb** aren't in the table. Neither has a Vista shell (DynamoDB has a `TableSource`, redb is a Dio cache), so no Vista over them can be wrapped. ## Several databases One transaction can span several datasources: wrap Vistas from each. Each datasource joins when its first Vista is wrapped, and at commit they commit one after another in that order. - If one fails before any has committed, all of them roll back. - If one fails after others committed, the rest roll back and the earlier ones stay committed. The error names their tables, and [`Transaction::committed_before_failure`](vantage_vista::Transaction::committed_before_failure) reads them back out: ```rust,no_run # use vantage_types::Record; # use vantage_vista::prelude::*; # use vantage_vista::Transaction; # async fn ship( # orders: &Vista, # stock: &Vista, # order: &Record, # left: &Record, # ) -> vantage_core::Result<()> { let tx = Transaction::new(); let (orders, stock) = (tx.wrap(orders), tx.wrap(stock)); // e.g. Postgres, then MongoDB orders.insert_value("o-1", order).await?; stock.patch_value("sku-1", left).await?; if let Err(e) = tx.commit().await { let landed = Transaction::committed_before_failure(&e); // ["order"] if only Postgres committed // undo or report what landed # let _ = landed; return Err(e); } # Ok(()) # } ``` Since order matters, wrap the datasource whose commit is likelier to fail first. If it fails, nothing has landed yet. ### Why not two-phase commit? Two-phase commit closes that window, but it needs a decision log: a durable record of "prepared, about to commit" that a recovering process reads to finish or undo the second phase. That log has to live in a database, and Vantage creates no tables in a client's databases. So a cross-database commit is atomic per database, and the failure window is reported instead of hidden. ## Datasources without transactions A shell over a datasource without transactions returns [`direct`](vantage_vista::direct) from its `rebind`. Its wrapped Vista writes as it always does: each write lands when made, and no rollback undoes it. The transaction lists those tables in `Committed::unprotected`, and [`tx.unprotected()`](vantage_vista::Transaction::unprotected) has them after a rollback or a failed commit too. Rhai hosts log each one as `atomic: "
" has no transactions; its writes landed directly`. That is why idempotence still matters. A direct write that landed before a rollback is still there on the retry, so the retry has to be safe to run again: an insert with a client-made id lands on the same row, a patch to a fixed value is the same patch twice. The same goes for a retry after a commit that failed part-way across databases. ## What doesn't join - **Dio.** A Dio's writes run on its worker task and go through its cache. A Dio shell wraps, and is refused at its first use (`"
" can't take part in a transaction`). Wrap the master Vista the Dio is built over, and refresh the Dio after the commit from `Committed::written` and `unprotected`. The Rhai hosts in vantage-server and vantage-ui do that for you, also for the tables that landed before a part-way failure. - **Contained relations.** A Vista over records embedded in a parent row refuses (its shell has no `rebind`). Write the parent row through a wrapped Vista instead. - **A forwarding shell that doesn't override `rebind`.** Refused, because only the wrapper knows how to rebuild itself over a rebound inner shell (see [below](#rebind)). - **DynamoDB and redb**, as above. ## In Rhai: `atomic { }` Scripts get the same handle as a block: ```rhai atomic { let accounts = table("account"); let alice = accounts.get("alice"); accounts.patch("bob", #{ balance: accounts.get("bob").balance + 30 }); if alice.balance < 30 { throw "not enough money"; } accounts.patch("alice", #{ balance: alice.balance - 30 }); } rollback (e, committed) { log(`transfer failed: ${e}; already committed: ${committed}`); } ``` Every table handle inside the block goes through one transaction per database, including handles made before the block. The block commits when it ends, or on `return` or `exit`. A `throw`, a `break` or `continue` out of it, a cancel, or its 30-second limit rolls it back. The optional `rollback (e)` or `rollback (e, committed)` arm then runs with a 10-second budget, and a cancel waits until it ends. `e` is what was thrown, or the error's text. `committed` is the array of tables that landed before a later database failed, `[]` otherwise. The arm may call actions; if it throws, the block's original error is the one re-thrown. An action marked `atomic: true` (`kind: script` only) runs its whole body as one block. Inside a block, `actions.x()`, a nested `atomic`, contained relations and Dio-only tables are refused. SQL lock waits and `secrets.get()` consent prompts aren't cut off by the 30-second limit, so keep blocks short. Nothing retries a failed block. When to use a block is a rule for script authors, in the **vantage-rhai** skill; the runtime doesn't check it. A Rust host wires the block with [`ResolverCell`](vantage_vista::rhai::ResolverCell), [`AtomicScope`](vantage_vista::rhai::AtomicScope) and [`AtomicVocab`](vantage_vista::rhai::AtomicVocab), and ends a block past its limit from `on_progress` through [`AtomicClock::should_stop`](vantage_vista::rhai::AtomicClock::should_stop). ## For driver authors ### `rebind` [`TableShell::rebind`](vantage_vista::TableShell::rebind) is the whole hook. `wrap` calls it, and the shell returns itself rebuilt to run on the transaction. There are three shapes. A datasource with transactions joins its participant there, keyed by a token every clone of the datasource shares, and returns a shell over the joined datasource. SQL's is two lines, the table keeping its hooks, conditions and references: ```rust,ignore fn rebind(&self, tx: &Transaction) -> Result> { let data_source = self.table.data_source().in_transaction(tx)?; // tx.participant(key, …) Ok(Box::new(Self { table: self.table.with_data_source(data_source), ..self.clone() })) } ``` Join in `rebind`, not at the first statement. The transaction notes which participants each rebind joined, and that is how it knows which tables landed when a later database fails. A participant joined later never shows up in `committed`. A datasource without transactions returns `Ok(direct(Box::new(self.clone())))`. A driver whose datasource has transactions only sometimes (MongoDB standalone, an old SurrealDB) decides per call. A forwarding shell rebinds its inner shell and wraps the result in a copy of itself. The default `rebind` refuses, and `forward_table_shell!` doesn't forward it, so a wrapper has to write it: ```rust use vantage_core::Result; use vantage_vista::{TableShell, Transaction}; #[derive(Clone)] struct Audited { inner: Box, } vantage_vista::forward_table_shell!(Audited, inner, { fn rebind(&self, tx: &Transaction) -> Result> { Ok(Box::new(Self { inner: self.inner.rebind(tx)? })) } }); ``` A shell that can lock rows also overrides `set_lock` (the default refuses). A [`TableSource`](vantage_table::traits::table_source::TableSource) in transaction mode answers `transaction()` with its transaction and `without_transaction()` with itself back on its pool: that is what holds after-hooks until the commit. ### `TxSlot` and `TxBackend` Most of a participant is the same whatever the backend: begin on first use, run one operation at a time, remember a failed one, refuse use after commit, roll back instead of committing after a failure. [`TxSlot`](vantage_vista::TxSlot) does all of that. A driver implements [`TxBackend`](vantage_vista::TxBackend): how to begin, commit and roll back. SQL, MongoDB and SurrealDB are each one `TxBackend` of about 40 lines. ```rust # use std::sync::{Arc, Mutex}; # use async_trait::async_trait; # use vantage_core::Result; use vantage_vista::{Transaction, TxBackend, TxKey, TxSlot}; /// Stands in for the driver's connection; it records what it ran. struct Conn(Arc>>); impl Conn { async fn run(&mut self, sql: &str) -> Result<()> { self.0.lock().unwrap().push(sql.to_string()); Ok(()) } } struct Begin(Arc>>); #[async_trait] impl TxBackend for Begin { type Open = Conn; async fn begin(&self) -> Result { let mut conn = Conn(self.0.clone()); conn.run("BEGIN").await?; Ok(conn) } async fn commit(&self, mut conn: Conn) -> Result<()> { conn.run("COMMIT").await } async fn rollback(&self, mut conn: Conn) -> Result<()> { conn.run("ROLLBACK").await } } #[derive(Clone)] struct Db { log: Arc>>, key: Arc<()>, tx: Option>>, } impl Db { /// What the shell's `rebind` calls. fn in_transaction(&self, tx: &Transaction) -> Result { let slot = tx.participant(TxKey::of(&self.key), || TxSlot::new(Begin(self.log.clone()), tx))?; Ok(Self { tx: Some(slot), ..self.clone() }) } async fn execute(&self, sql: &str) -> Result<()> { match &self.tx { None => Conn(self.log.clone()).run(sql).await, Some(slot) => { let mut open = slot.open().await?; // begins on first use slot.watch(open.run(sql)).await // an Err, or a drop mid-flight, fails the commit } } } } # #[tokio::main(flavor = "current_thread")] # async fn main() -> Result<()> { # let db = Db { log: Arc::default(), key: Arc::new(()), tx: None }; let tx = Transaction::new(); let inside = db.in_transaction(&tx)?; inside.execute("INSERT …").await?; tx.commit().await?; assert_eq!(*db.log.lock().unwrap(), ["BEGIN", "INSERT …", "COMMIT"]); # Ok(()) # } ``` The slot holds a weak handle to its transaction, so dropping the transaction drops the backend's open handle too. A backend with no transactions to begin, like memory, implements [`TxParticipant`](vantage_vista::TxParticipant) itself instead. ### The contract [`contract::check_transactions`](vantage_vista::contract::check_transactions) runs the transaction probes against a [`TxFixture`](vantage_vista::contract::TxFixture): commit, rollback, drop, isolation from outside readers, lazy begin, `get_ref` targets, concurrent calls, a failed statement, use after finish, two crossed writers, `lock()`, and two datasources at once. [`contract::check_atomic_rhai`](vantage_vista::contract::check_atomic_rhai) runs the same fixture through Rhai `atomic` blocks. Each check panics once, listing every probe that broke, and finishes inside 4.5 seconds. The fixture hands out two empty tables, `a(id, n)` and `b(id, a_id, n)`, with `n >= 0` enforced and a has-many relation `b` from `a`. The memory driver's, on a store per call: ```rust,no_run # use async_trait::async_trait; # use vantage_memory::{MemoryStore, MemoryVistaFactory}; # use vantage_vista::{FilterOp, VistaFactory}; use vantage_vista::contract::{self, TxFixture, TxTables}; # const TX_A: &str = "name: a\nid_column: id\ncolumns:\n id: { type: string, flags: [id] }\n n: { type: integer }\nreferences:\n b: { table: b, kind: has_many, foreign_key: a_id }\n"; # const TX_B: &str = "name: b\nid_column: id\ncolumns:\n id: { type: string, flags: [id] }\n a_id: { type: string }\n n: { type: integer }\n"; struct StoreTx; #[async_trait] impl TxFixture for StoreTx { async fn fresh(&self) -> TxTables { let f = MemoryVistaFactory::new(MemoryStore::new()); let a = f.from_yaml(TX_A).unwrap(); // The store checks no constraints; the set does it instead. let a = a.with_condition("n", FilterOp::Gte, ciborium::Value::Integer(0.into())).unwrap(); TxTables::new(a, f.from_yaml(TX_B).unwrap()) } async fn fresh_pool_of_one(&self) -> TxTables { self.fresh().await } fn detects_deadlocks(&self) -> bool { false } async fn fresh_other(&self) -> Option { Some(self.fresh().await) // another store: a second participant } } #[tokio::test] async fn memory_transactions_hold_the_contract() { contract::check_transactions(&StoreTx).await } #[tokio::test] async fn memory_atomic_holds_the_contract() { contract::check_atomic_rhai(&StoreTx).await } # fn main() {} ``` The other switches: `supports_lock()` for a backend whose `lock()` locks rows, `conflicts_fail_fast()` for one whose crossed writers both fail on a conflict instead of one waiting (MongoDB), and `supports_transactions() == false` for a direct datasource, which runs only the direct-write probe (or the direct-read one when its tables take no inserts). ## Checklist - Wrap every Vista you write through with `tx.wrap`, or the resolver with `tx.resolver`; an unwrapped Vista writes outside the transaction. - Commit explicitly. Every other way out (an error, `?`, a panic, a drop) rolls back the backends that joined; direct writes stay. - A caught database error still fails the commit (on memory, any caught error from the store); a caught before-hook error doesn't. - After-hooks run after the commit; their failures are logged, not returned. - Use `lock()` for read-then-write on Postgres and MySQL. - Across databases, the commit is per database: read `committed_before_failure` and wrap the likelier-to-fail database first. - Check `Committed::unprotected`: those writes landed directly, so they have to be safe to retry. - Dio, contained relations, DynamoDB and redb don't take part. - A driver: join in `rebind`, build on `TxSlot`, return `direct` when there are no transactions, and pass `check_transactions` and `check_atomic_rhai`. --- Source: https://vantage-ui.com/book/relations.html # Relations & Traversal Vantage treats a relation as something you *traverse*, not something you *load*. Declared once on the model, a relation gives you: 1. **Set-to-set traversal** — narrow "clients" down to the paying ones, traverse, and you hold "orders of paying clients" as a new set. One query when it executes; no loop over IDs. 2. **Related values inside queries** — a client row carrying its `order_count`, an order carrying its `client.name` — computed by the database through correlated subqueries or native link paths. 3. **The same vocabulary on every backend** — relations are declared on the model, not read from foreign-key constraints, so CSV files, REST APIs, and document stores get the same treatment as SQL. 4. **Relations that survive the stack** — the declaration travels from the typed `Table` through the type-erased `Vista` into the cached `Dio`; where two datasources meet, the Dio stitches the join client-side. If you come from an ORM, the shape is different enough to spell out: | | typical ORM | Vantage | | ---------------------- | ------------------------------------ | ------------------------------------------------ | | Traversal returns | loaded objects (lazy or eager) | a new set — a query definition, no data loaded | | N+1 queries | managed with eager-loading hints | don't arise: traversal is one subquery | | Declared | derived from schema / migrations | on the model; works without FK constraints | | Backends | the SQL database | SQL, SurrealDB, MongoDB, CSV, REST, DynamoDB, … | | Related fields | JOIN and map, or embedded objects | subquery expressions, implicit references | | Across two databases | out of scope | catalog traversal; augmentation joins per row in the cache | Our running example is `bakery_model3` — an example model crate that ships in the Vantage repository. It defines one set of entities (bakeries, clients, orders, products) with table constructors for several datasources — SQLite, PostgreSQL, SurrealDB, MongoDB, CSV, DynamoDB — and doubles as a test fixture for the framework itself. [Model-Driven Architecture](mda.md) walks through its anatomy; here we just borrow it as the example. Three of its tables belong together: a `Bakery` has many `Clients`, and each `Client` has many `Orders`. In SQL terms, `client.bakery_id` points at a bakery and `client_order.client_id` points at a client: ```mermaid erDiagram bakery ||--o{ client : "client.bakery_id" client ||--o{ client_order : "client_order.client_id" ``` Almost every real question you'd ask of this data crosses one of those links — "orders of paying clients", "which bakery does this order belong to" — so this guide is about how Vantage models those links and how you cross them. You met relations briefly in the [Introduction](intro/step2-tables.md), where a product catalog picked up `with_many` and `get_ref_as`; this guide covers the same machinery in depth, with a fuller model. We work SQL-first here (SQLite), with SurrealDB and other backends in notes along the way. The chapters after this one dig into traversal forms, subquery expressions, implicit references, and how relations survive into the type-erased `Vista` and cached `Dio` layers — this page sets up the vocabulary they all share. ## One relation, four layers An ORM or a data mapper typically has exactly one place where references live: on the mapped object. `client.orders()` loads objects, and that is the whole story. In Vantage the *same declared relation* is traversable at (approximately) four layers. Each layer traverses with what it has in hand and gives you back a handle native to that layer — you never drop down a level to cross a link: The traversal method is called `get_ref` at every layer — what changes is what you hold when you call it, and when you'd want to: | level | when you reach for it | | --- | --- | | Record → Table | You hold an `ActiveRecord` or `ActiveEntity` — physical row data, loaded. `client_record.get_ref("orders")?` hands back a table narrowed to that client. Models typically wrap this in typed methods, so application code calls `client_record.ref_orders()` and gets a `Table`. | | Table → Table | True set traversal — no data loaded on either side. Start with the table of VIP clients, traverse to their orders, aggregate: one query, and you hold the sum of all VIP clients' orders. | | Vista → Vista | Type-erased, for generic consumers. Record and table traversal stay inside one datasource; a `Vista` respects the relationships declared on the wrapped table, and a `Catalog` builds models by name, so a `Reference` can narrow a target that lives in a *different* datasource. | | Dio | Not a new traversal — the caching layer over the ones above. A traversed detail set gets its own live cache, and *augmentation* merges a related source's columns into the master's cached rows, one visible row at a time. | Why is there a Dio layer at all, if the catalog already crosses datasources? Because catalog traversal *navigates* — it hands you the related set, and every read of it is a live round-trip. The Dio owns what navigation lacks: a cache, a viewport, and change events. It caches each traversed detail set, and it hosts augmentation — not navigation *to* related rows but a per-row join *into* the master's rows, which needs exactly that cache and viewport. [Relations and Dio](relations/dio.md) works through both. The rest of this page covers the first vocabulary — declaring the relation. The child chapters then walk the layers: [tables](relations/traversal.md), [vistas](relations/vistas.md), and [dio](relations/dio.md). ## What a relation is A relation in Vantage is a declared, named link between two table definitions. It lives on the **table definition** — the model — not on the entity struct, and not in the database. Vantage never introspects foreign-key constraints; whether your SQLite schema actually declares `FOREIGN KEY (client_id) REFERENCES client(id)` is invisible to it. A relation is exactly three things: 1. **A name** — the string you traverse by, like `"orders"`. 2. **A foreign-key field** — the column that carries the link. 3. **A target-table constructor** — a function that builds the table on the other end. Two properties follow from this shape. **References are one-sided**: a declaration gives *this* table a way to reach the target — nothing is declared on the target, and no back-reference is required. If you want to traverse the other direction, declare that separately (the bakery model does: `orders` on `Client`, `client` on `Order`). And because traversal produces *sets*, **the same pair of tables can be linked by any number of references** — same target, same foreign key, different conditions. The constructor argument is what makes this natural: pass a closure that narrows the target, and the relation names a *meaningful subset*, not just a foreign key: ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # pub struct Invoice { # pub client_id: Option, # pub status: String, # } # # impl Invoice { # pub fn table(db: SqliteDB) -> Table { # Table::new("invoice", db) # .with_id_column("id") # .with_column_of::("client_id") # .with_column_of::("status") # } # } # # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # pub struct Client { # pub name: String, # } # # pub fn client_table(db: SqliteDB) -> Table { # Table::new("client", db) # .with_id_column("id") # .with_column_of::("name") .with_many("paid_invoices", "client_id", |db| { let mut t = Invoice::table(db); t.add_condition(t["status"].eq("paid")); t }) .with_many("due_invoices", "client_id", |db| { let mut t = Invoice::table(db); t.add_condition(t["status"].eq("due")); t }) # } ``` With the extension-trait convention (below), application code reads `client.ref_paid_invoices()` / `client.ref_due_invoices()` — one invoice table underneath, two named ways to reach it. ### Why not read the foreign keys from the database? Because most of the backends Vantage talks to don't have any. CSV files, REST APIs, and MongoDB have no foreign-key concept to introspect, yet their data is just as related. Declaring the relation on the model means the same declaration works everywhere — the link between clients and orders exists whether the rows live in Postgres or in a spreadsheet export. ## Two cardinalities There are two declaration methods, and the difference between them comes down to one question: which table holds the foreign key? - **`with_one("name", "fk_field", constructor)`** — the FK lives on **this** table. Traversal yields at most one row: many-to-one. An order has one client. - **`with_many("name", "fk_field", constructor)`** — the FK lives on the **target** table. Traversal yields a set: one-to-many. A client has many orders. Both declarations describe the same underlying link — one foreign-key column, viewed from either end: ```mermaid graph LR order["client_order
(carries client_id)"] client["client
(has id)"] order -- "with_one('client')" --> client client -- "with_many('orders')" --> order ``` The two are more similar than the names suggest. Both take the same closure and use it the same way: build the target table, then pin it with a single eq-condition. The only difference is the condition's orientation — `with_one` reads the foreign key *out of the source row* and pins the target's id; `with_many` reads the source row's *id* and pins the target's foreign-key column. Same closure, same eq-condition, read from opposite ends of the link. Internally these become `HasOne` and `HasMany` implementations of the `Reference` trait (`vantage-table/src/references/`), and the cardinality is surfaced as `ReferenceKind::HasOne` / `ReferenceKind::HasMany` for anything that needs to reason about relations generically — the vista layer will, later. ### You could write it yourself `with_one`, `with_many`, and `get_ref` are convenience. A traversal is just "build a table, add a condition from what I'm holding" — and nothing stops you from writing exactly that as an extension method on a loaded record, without registering any reference at all. It doesn't even have to stay in the same datasource. Here the client lives in SQLite and the mail log is a CSV export: ```rust # use vantage_csv::operation::CsvOperation; # use vantage_csv::{AnyCsvType, Csv}; # use vantage_dataset::ActiveEntity; # use vantage_sql::sqlite::{AnySqliteType, SqliteDB}; # use vantage_table::table::Table; # use vantage_types::entity; # # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # pub struct Client { # pub name: String, # pub email: String, # } # # #[entity(CsvType)] # #[derive(Debug, Clone, Default)] # pub struct MailLogEntry { # pub email: String, # pub subject: String, # } # # impl MailLogEntry { # pub fn table(csv: Csv) -> Table { # Table::new("mail_log", csv) # .with_column_of::("email") # .with_column_of::("subject") # } # } # pub trait ClientMailLog { fn ref_mail_log(&self) -> Table; } impl<'a> ClientMailLog for ActiveEntity<'a, Table, Client> { fn ref_mail_log(&self) -> Table { let mut log = MailLogEntry::table(Csv::new("exports")); log.add_condition(log["email"].eq(AnyCsvType::new(self.email.clone()))); log } } ``` (`self.email` works because `ActiveEntity` derefs to the entity struct — the loaded `Client`'s fields are right there. An untyped `ActiveRecord` derefs to the raw `Record` instead, where the same value is `self.get("email")`.) From the caller's side `client.ref_mail_log()` is indistinguishable from a declared relation — but there is no foreign key here, no registry entry, just a CSV-backed table conditioned by the client's email. What the declared forms add over hand-rolling is everything that needs the *registry*: string-addressable traversal for the generic layers (vistas, implicit references), and the foreign-key invariant on inserts. Reach for a hand-rolled method when the link doesn't fit an FK eq-match; declare a relation when you want the machinery. ### Coercion — pinning `get_ref`'s type A relation is addressed by string, and a string carries no type. `get_ref` therefore cannot *infer* what entity comes back — it is generic, and the caller chooses: `get_ref::("orders")`. The turbofish is an assertion, not a check: the registry knows which table the relation builds, but the entity type you name is simply coerced onto the result. Name the wrong one and you find out at runtime, when rows fail to deserialize — not from the compiler. This uncertainty is exactly why the model wraps every relation once, in an extension trait next to the table definition: ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Client { # pub name: String, # } # # impl Client { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client", db) # .with_id_column("id") # .with_column_of::("name") # .with_many("orders", "client_id", Order::sqlite_table) # } # } # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Order { # pub client_id: Option, # } # # impl Order { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client_order", db) # .with_id_column("id") # .with_column_of::("client_id") # .with_one("client", "client_id", Client::sqlite_table) # } # } # pub trait ClientTable { fn ref_orders(&self) -> Table; } impl ClientTable for Table { fn ref_orders(&self) -> Table { self.get_ref_as("orders").unwrap() } } ``` And the same on the other side — `Order` gets `ref_client()`: ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Client { # pub name: String, # } # # impl Client { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client", db) # .with_id_column("id") # .with_column_of::("name") # .with_many("orders", "client_id", Order::sqlite_table) # } # } # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Order { # pub client_id: Option, # } # # impl Order { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client_order", db) # .with_id_column("id") # .with_column_of::("client_id") # .with_one("client", "client_id", Client::sqlite_table) # } # } # pub trait OrderTable { fn ref_client(&self) -> Table; } impl OrderTable for Table { fn ref_client(&self) -> Table { self.get_ref_as("client").unwrap() } } ``` The coercion happens in one place — where the right answer is obvious — and the `unwrap()` is safe there too: `"orders"` is declared on every `Client` table the model produces, so a typo panics in development, not silently at runtime. This is the convention to follow throughout a model crate: every declared relation surfaces as a typed `ref_()` method, callers write `clients.ref_orders()` / `orders.ref_client()`, and the strings and turbofish exist in exactly one place with no opportunity to name the wrong entity. The [Introduction](intro/step2-tables.md) used the same pattern, and [Model-Driven Architecture](mda.md) covers where these traits live in a model crate. ## Declaring relations on the model Here is `Client` from `bakery_model3/src/client.rs` — the entity struct and its SQLite table constructor. The real file lists six type systems in `#[entity(..)]`, one per backend it targets; this excerpt keeps SQLite's: ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Bakery { # pub name: String, # } # # impl Bakery { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("bakery", db) # .with_id_column("id") # .with_column_of::("name") # } # } # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Order { # pub client_id: Option, # pub is_deleted: bool, # } # # impl Order { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client_order", db) # .with_id_column("id") # .with_column_of::("client_id") # .with_column_of::("is_deleted") # .with_one("client", "client_id", Client::sqlite_table) # } # } # #[entity(SqliteType)] #[derive(Debug, Clone, PartialEq, Default)] pub struct Client { pub name: String, pub email: String, pub contact_details: String, pub is_paying_client: bool, pub bakery_id: Option, } impl Client { pub fn sqlite_table(db: SqliteDB) -> Table { Table::new("client", db) .with_id_column("id") .with_column_of::("name") .with_column_of::("email") .with_column_of::("contact_details") .with_column_of::("is_paying_client") .with_column_of::("bakery_id") .with_one("bakery", "bakery_id", Bakery::sqlite_table) .with_many("orders", "client_id", Order::sqlite_table) } } ``` Notice that the struct knows nothing about relations. `bakery_id` is just an `Option` field like any other; the links are declared on the table definition, where the columns and conditions already live. And here is the other end, from `order.rs`: ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Client { # pub name: String, # } # # impl Client { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client", db) # .with_id_column("id") # .with_column_of::("name") # .with_many("orders", "client_id", Order::sqlite_table) # } # } # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Order { # pub client_id: Option, # pub is_deleted: bool, # } # impl Order { pub fn sqlite_table(db: SqliteDB) -> Table { Table::new("client_order", db) .with_id_column("id") .with_column_of::("client_id") .with_column_of::("is_deleted") .with_one("client", "client_id", Client::sqlite_table) } } ``` The same underlying relation appears from both ends: `Client` declares `with_many("orders", "client_id", …)` and `Order` declares `with_one("client", "client_id", …)`. Each side is declared independently and each is usable independently — you could declare only one of them if you never traverse the other direction. And both name the same column: `"client_id"` on the `client_order` table. For `with_many` that's the FK on the *target* table; for `with_one` it's the FK on the *source* — here `Order` happens to be the FK-carrying side both times. ### The constructor argument The third argument — `Order::sqlite_table` — is not a table, it's a function. It's the very same constructor the model already exposes; you're just passing it by name instead of calling it. That buys two things. First, declaring a relation costs nothing up front: the target table is only constructed when the relation is actually traversed. Second, and more importantly, when the target *is* constructed, it's the real model definition — its own columns, its own conditions, its own relations. If `Order::sqlite_table` filtered out soft-deleted rows, every traversal that reaches orders through a relation would inherit that filter. There is no second, weaker definition of "orders" that relations quietly use. ## A first traversal Traversal is set-to-set: narrow the source set, traverse, and the target set arrives already narrowed by the relationship. Here's "orders of paying clients": ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Client { # pub name: String, # pub is_paying_client: bool, # } # # impl Client { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("is_paying_client") # .with_many("orders", "client_id", Order::sqlite_table) # } # } # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Order { # pub client_id: Option, # } # # impl Order { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client_order", db) # .with_id_column("id") # .with_column_of::("client_id") # .with_one("client", "client_id", Client::sqlite_table) # } # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite::memory:").await.context("connect")?; let mut paying = Client::sqlite_table(db.clone()); paying.add_condition(paying["is_paying_client"].eq(true)); // Table, narrowed to orders of paying clients let orders = paying.get_ref_as::("orders")?; # let _ = orders; # Ok(()) # } ``` No query has run yet — `orders` is a `Table` like any other, and you can add conditions, select columns, or traverse further. When it does execute, the narrowing happens via a subquery — not a JOIN, and not a round-trip to fetch client IDs first. The shape (illustrative): ```sql SELECT ... FROM "client_order" WHERE "client_id" IN (SELECT "id" FROM "client" WHERE "is_paying_client" = 1) ``` This is the payoff of set-to-set thinking: "orders of paying clients" is one narrowing plus one traversal, executed as a single query — never a loop over IDs. In application code, the string and the turbofish disappear behind the wrapper from the [coercion section](#coercion--pinning-get_refs-type) above: ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Client { # pub name: String, # } # # impl Client { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client", db) # .with_id_column("id") # .with_column_of::("name") # .with_many("orders", "client_id", Order::sqlite_table) # } # } # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Order { # pub client_id: Option, # } # # impl Order { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client_order", db) # .with_id_column("id") # .with_column_of::("client_id") # } # } # # pub trait ClientTable { # fn ref_orders(&self) -> Table; # } # # impl ClientTable for Table { # fn ref_orders(&self) -> Table { # self.get_ref_as("orders").unwrap() # } # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite::memory:").await.context("connect")?; # let paying = Client::sqlite_table(db); let orders = paying.ref_orders(); // Table — type inferred, no string # let _ = orders; # Ok(()) # } ``` The next chapter covers the full range of traversal forms; this is just the first taste. ## SurrealDB: record links instead of foreign keys SurrealDB stores relations as **record links** — the FK column holds a typed record id (a `Thing`, rendered `table:key`) rather than a scalar. The declaration shape is identical; only the field differs. From the same `bakery_model3/src/client.rs`: ```rust # use vantage_surrealdb::surrealdb::SurrealDB; # use vantage_surrealdb::thing::Thing; # use vantage_surrealdb::types::AnySurrealType; # use vantage_table::table::Table; # use vantage_types::entity; # # #[entity(SurrealType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Bakery { # pub name: String, # } # # impl Bakery { # pub fn surreal_table(db: SurrealDB) -> Table { # Table::new("bakery", db) # .with_id_column("id") # .with_column_of::("name") # } # } # # #[entity(SurrealType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Order { # pub is_deleted: bool, # } # # impl Order { # pub fn surreal_table(db: SurrealDB) -> Table { # Table::new("order", db) # .with_id_column("id") # .with_column_of::("client") # .with_column_of::("is_deleted") # } # } # # #[entity(SurrealType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Client { # pub name: String, # pub email: String, # pub contact_details: String, # pub is_paying_client: bool, # } # # impl Client { pub fn surreal_table(db: SurrealDB) -> Table { Table::new("client", db) .with_id_column("id") .with_column_of::("name") .with_column_of::("email") .with_column_of::("contact_details") .with_column_of::("is_paying_client") .with_one("bakery", "bakery", Bakery::surreal_table) .with_many("orders", "client", Order::surreal_table) } # } ``` What changes is the column's *content*, not its name — that is a database-design choice. Any column can hold a record link: you could equally keep calling it `bakery_id` and store a `Thing` in it, and the declaration would name that column instead. `bakery_model3` happens to follow the common SurrealDB convention of naming the link column after the target (`"bakery"`), which is why its declarations differ from the SQLite ones only in that argument. Either way, the relation *name* is free to differ from the link field; nothing couples them. ```admonish info title="Other backends" MongoDB, CSV, and DynamoDB declare relations exactly the same way, with scalar foreign keys — `bakery_model3` has `mongo_table`, `csv_table`, and `dynamo_table` constructors of the same shape. Traversal itself is easy and every table supports it: it only *attaches a condition* to the target. The heavy lifting is honouring that condition when the related data is actually fetched — SQL pushes it down as a `WHERE`, while CSV filters in memory to fulfil the same `Table` contract. The next chapter covers what each traversal form generates per backend. ``` ## Writing into a traversed set A traversed set is not read-only. Traverse from a loaded client, insert into the result, and the new record *belongs* to the set you traversed to: ```rust,no_run # use vantage_dataset::prelude::ActiveEntitySet; # use vantage_sql::prelude::*; # use vantage_table::prelude::GetRefExt; # use vantage_types::prelude::*; # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Client { # pub name: String, # } # # impl Client { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client", db) # .with_id_column("id") # .with_column_of::("name") # .with_many("orders", "client_id", Order::sqlite_table) # } # } # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Order { # pub client_id: Option, # pub is_deleted: bool, # } # # impl Order { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client_order", db) # .with_id_column("id") # .with_column_of::("client_id") # .with_column_of::("is_deleted") # } # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:bakery.sqlite").await.context("connect")?; let clients = Client::sqlite_table(db); let client = clients.get_entity("marty").await?.expect("client exists"); let orders = client.get_ref::("orders")?; orders.insert_return_id(&Order::default()).await?; # Ok(()) # } ``` The order's `client_id` is filled in from the client — you don't set it, and if you set a *conflicting* one the write is rejected. The block reads a populated bakery database, so it is compiled but not run here. How that enforcement works (set invariants, lifecycle hooks) is covered in [Records: Traversal, Invariants & Hooks](record-lifecycle.md). ## Where this guide goes You now have the vocabulary: a relation is a named `(name, fk_field, constructor)` triple on the table definition, in one of two cardinalities, traversed set-to-set. The child chapters each take one thread further: - **[Traversing Sets and Records](relations/traversal.md)** — the full range of traversal forms: set-to-set (fetching and embedding), row-in-hand, record-level, and contained. - **[Expressions & Subqueries](relations/expressions.md)** — correlated lookups, counts, and how relation-derived expressions compose. - **[Implicit References](relations/implicit-references.md)** — dotted column names, the declarative way to pull a field across a relation. - **[Vista, YAML and Rhai](relations/vistas.md)** — vista factories, traversal after type erasure, capability checks, and declaring relations in YAML and Rhai. Scripts traverse with `ref(relation)`, covered in [Scripting with Rhai](rhai/tables.md#following-relations). - **[Relations and Dio](relations/dio.md)** — where cross-datasource enrichment lives. --- Source: https://vantage-ui.com/book/relations/traversal.html # Traversing Sets and Records The [previous chapter](../relations.md) declared the bakery model: Bakery → Clients (`client.bakery_id`) → Orders (`client_order.client_id`), with the relations `"orders"`, `"client"`, and `"bakery"` registered via `with_one` / `with_many`. Declaring a relation stores the join recipe. This chapter is about using it: turning "the orders of these clients" into an actual query. There are four traversal forms. They differ in *what you have in hand when you traverse* — a set (conditions, no data), a loaded row, or a loaded record object — and, between the two set forms, in *how the condition is applied* to the target. Each form exists because each starting point lets the backend do something different. ## Form 1 — set-to-set: `get_ref_as` You have a *set* — a table narrowed by conditions, possibly matching many rows, none of them loaded. `Table::get_ref_as::(relation)` traverses from the whole set at once: ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Client { # pub name: String, # pub is_paying_client: bool, # } # # impl Client { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("is_paying_client") # .with_many("orders", "client_id", Order::sqlite_table) # } # } # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Order { # pub client_id: Option, # } # # impl Order { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client_order", db) # .with_id_column("id") # .with_column_of::("client_id") # } # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite::memory:").await.context("connect")?; let mut paying = Client::sqlite_table(db.clone()); paying.add_condition(paying["is_paying_client"].eq(true)); let orders = paying.get_ref_as::("orders")?; // SELECT ... FROM "client_order" // WHERE "client_id" IN (SELECT "id" FROM "client" WHERE "is_paying_client" = 1) # let _ = orders; # Ok(()) # } ``` No client rows are fetched. The traversal generates an `IN (subquery)` condition on the target, and the subquery is whatever conditions the source set carries. Narrow `paying` further and the subquery narrows with it — the traversal call never changes. That composability is the point: you build the set that describes *which* parents you mean, and the child set inherits that description. In application code these calls hide behind the model's extension traits — `paying.ref_orders()`, `orders.ref_client()` — so the string and the turbofish live once, next to the table definition ([Coercion — pinning `get_ref`'s type](../relations.md#coercion--pinning-get_refs-type)). This chapter spells out the underlying calls so you can see what each form generates. How the `IN` condition is built is per-backend — each backend implements `TableSource::related_in_condition`: - **SQL** builds the subquery you see above: `"client_id" IN (SELECT "id" FROM "client" WHERE …)`. - **MongoDB** has no subqueries, so it builds a deferred `{ field: { "$in": [...] } }` document — the parent ids are resolved when the child query runs. - **CSV** fetches the join values in memory and builds an IN condition from them. ```admonish info title="MongoDB: the deferred condition" MongoDB's `related_in_condition` returns a **deferred** condition — a closure, not a document. Nothing runs when you traverse; the closure runs when the *child* query executes. At that point it issues a projected `find` on the source collection — only the join column, with the source set's own filter applied — collects the values, and emits `{ field: { "$in": [...] } }` for the child query. It also absorbs a representation mismatch: ids may be stored as `ObjectId` on one side and as the hex string on the other, so every collected value is pushed in *both* forms and `$in` matches either. The set-to-set contract survives — you still hand the target one condition — it just costs two round-trips at fetch time instead of one nested query. ``` ```admonish info title="CSV: an in-memory IN list" CSV has no query engine at all — its condition type is an ordinary Vantage expression that the driver *evaluates per row in Rust* while reading the file. `related_in_condition` builds `target_column.in_(values)`, where `values` is itself deferred: at fetch time it lists the source table (with its conditions applied the same in-memory way), pulls the join column out of each row, and yields the list as a single value. Filtering the target is then a per-row membership check. Same declared relation, same traversal call — no database anywhere. ``` ### Building and reading conditions by column `related_in_condition` has siblings on `TableSource` for code that builds or inspects a narrowing without a relation, such as an access rule or an audit log line: - `eq_value_condition(field, value)` and `in_values_condition(field, values)` build `field = value` and `field IN (values)` from native values. An empty list matches no row. REST accepts a single value only and refuses longer lists: a URL filter carries one value, and silently dropping the condition would widen the set. - `condition_column(&condition)` names the column a condition narrows to values: equality, `IN` (including the related and deferred forms above), comparisons, `LIKE`, each with a value (or, for `IN`, a value list or subquery) on the right. It returns `None` when it can't tell, as for a raw expression or an `OR`, and for anything that doesn't pin the column to values: a negation (`!=`, `NOT IN`, `NOT LIKE`, a null check), or a column or expression on the right (`tenant_id = tenant_id`, `tenant_id = 7 OR 1 = 1`). So a check that demands a condition on, say, `tenant_id` fails safe. - `describe_condition(&condition)` renders a condition for people (`client_id = 7`, `invoice_id IN `), for log entries. Don't parse it; use `condition_column` in code. Use this form when the parent is a set — filtered, unfiltered, one row or a thousand. ## Form 2 — set-to-set, for embedding: `get_subquery_as` Technically set-to-set as well — same starting point, same declared relation — but the condition is applied differently. Where `get_ref_as` narrows the target with `IN (subquery)` so the related rows can be *fetched* on their own, `Table::get_subquery_as::(relation)` attaches a **correlated** condition — target column against source column, row by row: ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Client { # pub name: String, # } # # impl Client { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client", db) # .with_id_column("id") # .with_column_of::("name") # .with_many("orders", "client_id", Order::sqlite_table) # } # } # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Order { # pub client_id: Option, # } # # impl Order { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client_order", db) # .with_id_column("id") # .with_column_of::("client_id") # } # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite::memory:").await.context("connect")?; # let clients = Client::sqlite_table(db); let orders = clients.get_ref_as::("orders")?; // fetch: WHERE client_id IN (SELECT id FROM client …) let orders = clients.get_subquery_as::("orders")?; // embed: WHERE client_id = client.id # let _ = orders; # Ok(()) # } ``` A correlated table is useless to fetch standalone — its condition references the source's rows — but it is exactly right to *embed* as a scalar subquery inside the source's own SELECT: a client's `order_count`, an order's `client.name`. The embedding recipe (`select_column`, aggregates, composition) is the subject of the [next chapter](./expressions.md). ```admonish warning title="SQL and SurrealDB only" This form needs the backend to express a correlated condition (`TableSource::related_correlated_condition`), and only SQL backends and SurrealDB can. MongoDB, CSV, REST, and CMD have no correlated-subquery expressions to lower to — the default implementation panics rather than degrading silently. The fetching forms (1, 3, 4) remain available everywhere. ``` ## Form 3 — row-in-hand: `get_ref_from_row` You already hold a loaded row — a `Record` that came back from the database. There is nothing to compute: the join value is sitting in the row. `Table::get_ref_from_row::(relation, &row)` reads it out and applies it as a single eq-condition on the target: ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Client { # pub name: String, # } # # impl Client { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client", db) # .with_id_column("id") # .with_column_of::("name") # .with_many("orders", "client_id", Order::sqlite_table) # } # } # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Order { # pub client_id: Option, # } # # impl Order { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client_order", db) # .with_id_column("id") # .with_column_of::("client_id") # } # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite::memory:").await.context("connect")?; # let clients = Client::sqlite_table(db); # let mut row: Record = Record::new(); # row.insert("id".to_string(), AnySqliteType::new("marty".to_string())); let orders = clients.get_ref_from_row::("orders", &row)?; // WHERE "client_id" = — one eq-condition # let _ = orders; # Ok(()) # } ``` Which field gets read depends on the relation's direction: a `HasOne` relation reads its stored foreign-key column from the row; a `HasMany` relation reads the source's id field. No subquery, no deferred fetch — `row` already carries the value. ```admonish info title="Plumbing, not user-facing API" You will rarely call this directly: it exists as the primitive beneath the blanket `get_ref` implementations for `ActiveRecord` and `ActiveEntity` (form 4), and the erased `Vista::get_ref` forwards here too. When a UI shows a clicked row's children, this is the form doing the work — one equality filter — but the caller holds a record handle or a Vista, not a raw row. ``` ## Form 4 — record-level: `GetRefExt::get_ref` The same traversal as form 3, called from a loaded handle instead of a raw row: `launch.get_ref::("launch_crew")` works on an `ActiveEntity` (typed — its id is injected into the row first, so has-many relations resolve) and on an `ActiveRecord` (untyped — the raw row forwards directly). The method comes from the `GetRefExt` extension trait in `vantage-table`, with blanket implementations for both handles over any `TableSource`. The worked example (loading a record, traversing, inserting a child) lives in [Records: Traversal, Invariants & Hooks](../record-lifecycle.md#traversing-from-a-loaded-record-get_ref). ## The bare target: `get_ref_target_erased` Sometimes you want the relation's target table with *no* condition at all — `Table::get_ref_target_erased(relation)` builds exactly that. The target's entity type is erased: a relation is built by a caller-supplied constructor, so the traversal site cannot name it. Where the traversal forms narrow the target to related rows, `get_ref_target_erased` hands you the table you'd insert a new related row into before any join value exists (this is what Vista's nested insert uses). ## When a target can't be built `with_one` and `with_many` take a constructor that always returns a table. Their siblings `try_with_one` and `try_with_many` take one that returns `Result`, plus the target's name for metadata. When the constructor fails, every traversal form returns that error. It never falls back to a table without the target's own conditions: if the target carried an access rule, a fallback would read rows the rule hides. Spec references (from YAML) are built this way. A target the resolver doesn't know, or one whose build fails (a build injector refusing it, say), makes the traversal an error. The relation still appears in metadata under its declared target name. ## Contained relations Some backends embed related data *inside* the row instead of linking to another table. SurrealDB's `Order` carries an embedded `array` of order lines — declared as a column (so it is selected) *and* as a contains-many relation whose record schema is built by a closure. From `bakery_model3/src/order.rs`: ```rust # use vantage_surrealdb::surrealdb::SurrealDB; # use vantage_surrealdb::thing::Thing; # use vantage_surrealdb::types::AnySurrealType; # use vantage_table::table::Table; # use vantage_types::entity; # # #[entity(SurrealType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Client { # pub name: String, # } # # impl Client { # pub fn surreal_table(db: SurrealDB) -> Table { # Table::new("client", db) # .with_id_column("id") # .with_column_of::("name") # } # } # # #[entity(SurrealType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Product { # pub name: String, # } # # impl Product { # pub fn surreal_table(db: SurrealDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # } # } # # #[entity(SurrealType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Order { # pub is_deleted: bool, # } # # impl Order { pub fn surreal_table(db: SurrealDB) -> Table { Table::new("order", db) .with_id_column("id") .with_column_of::("client") .with_column_of::("is_deleted") // `lines` is an embedded `array` of `{ product, quantity, // price }` — declared as a column so it's selected, and as a // contains-many relation whose record schema is built by the // closure (like `with_many`). .with_column_of::("lines") .with_one("client", "client", Client::surreal_table) .with_contained_many( "lines", "lines", |db| { Table::new("lines", db) .with_column_of::("product") .with_column_of::("quantity") .with_column_of::("price") // a line traverses out to the real product table .with_one("product", "product", Product::surreal_table) }, None, ) } # } ``` Contained relations are a K/V-and-document-store shape: the value already holds the related data, so "traversal" is descent into the row, not a query. Note that a contained record can still hold references back out — a line's `product` field is a `with_one` to the real product table, so from an embedded line you traverse out with the same forms as anywhere else. ```admonish note title="SurrealDB record ids in narrowings" Join values in SurrealDB are record ids (`Thing`), not scalar foreign keys. A string of the shape `"table:key"` handed into a narrowing — from a script surface or JSON, say — is coerced back into a record id by the backend (`coerce_reference_value` on `TableSource`; the identity function on scalar-FK backends). String ids narrow correctly; they don't silently match nothing. ``` ## Conclusion You can now: 1. **Traverse a filtered set** with `get_ref_as` — conditions compose into an `IN (subquery)` on the target. 2. **Correlate instead of fetch** with `get_subquery_as` — same relation, a per-row condition, built for embedding in the source's SELECT. 3. **Traverse from a loaded row or record** with `get_ref_from_row` / `get_ref` — one eq-condition, supported by every backend. 4. **Obtain a bare insert target** with `get_ref_target_erased` — the relation's target with no condition. 5. **Declare embedded/contained relations** on document stores with `with_contained_many`, where traversal is descent rather than a query. --- Source: https://vantage-ui.com/book/relations/expressions.html # Expressions & Subqueries The previous chapter used relations to *fetch* related rows — traverse from a client to their orders, get the orders back as a set. This chapter uses relations *inside* a query: a client row that carries its own order count, computed by the database, delivered as just another column. The recipe has three parts: 1. **`Table::with_expression(name, |t| …)`** — adds a computed field to the table. It's evaluated as part of the SELECT, alongside the physical columns. 2. **`Table::get_subquery_as::(relation)`** — builds the relation's target table with a **correlated condition**. For the clients → orders relation that condition is `client_order.client_id = client.id` — the direction is always `target. = source.`, supplied by the backend through `TableSource::related_correlated_condition`. This table is explicitly designed for embedding as a scalar subquery. 3. **`select_column("field")`** — projects a single column from that subquery, so it can sit where a scalar value is expected. Here's the shape, from the `select_column` rustdoc, on a product table that declares a `category` relation: ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # pub struct Category { # pub name: String, # } # # impl Category { # pub fn table(db: SqliteDB) -> Table { # Table::new("category", db) # .with_id_column("id") # .with_column_of::("name") # } # } # # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # pub struct Product { # pub name: String, # } # # pub fn product_table(db: SqliteDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("category_id") # .with_one("category", "category_id", Category::table) .with_expression("category", |t| { t.get_subquery_as::("category").unwrap() .select_column("name") .expect("Category has a 'name' column") }) # } ``` The key difference from the previous chapter: `get_ref_as` narrows with `IN (subquery)` to *fetch* related rows; `get_subquery_as` correlates to *embed* a related value per row. Same relation declaration, two different query shapes. `select_column` returns `Option>` — `None` when the field isn't a column on the target table. When the column is hardcoded and known to exist, `.expect(...)` is the honest choice: it panics in development, where you can fix the typo, rather than silently at runtime. ### Why not a JOIN? Vantage composes *sets*. A correlated scalar subquery keeps the row shape of the source table — one row per client, one extra column — with no join-key bookkeeping and no row multiplication when a client has ten orders. It also nests arbitrarily, which joins don't do gracefully. And you're not paying for the nesting: the database's optimizer flattens a correlated subquery where a join would be equivalent. ```admonish note title="A driver may still join" The correlated shape is the contract, not a mandate about execution. Lowering is a per-datasource concern, and a driver implementation is free to use a JOIN — or anything else — where that is the reliable choice for its engine, as long as the result keeps the source's row shape: one row per source record, the related value as a column. ``` ### Aggregates over a relation `get_subquery_as` returns a full `Table`, so the aggregate query-builders work on it: `get_count_query()` and `get_sum_query(&col)`. Both wrap their output in parentheses, so they nest safely inside the outer SELECT. Here's the real thing — verbatim from `bakery_model3/src/client.rs`, a client table whose `order_count` is computed by the database: ```rust # use vantage_surrealdb::surrealdb::SurrealDB; # use vantage_surrealdb::thing::Thing; # use vantage_surrealdb::types::AnySurrealType; # use vantage_table::table::Table; # use vantage_types::entity; # # #[entity(SurrealType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Bakery { # pub name: String, # } # # impl Bakery { # pub fn surreal_table(db: SurrealDB) -> Table { # Table::new("bakery", db) # .with_id_column("id") # .with_column_of::("name") # } # } # # #[entity(SurrealType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Order { # pub is_deleted: bool, # } # # impl Order { # pub fn surreal_table(db: SurrealDB) -> Table { # Table::new("order", db) # .with_id_column("id") # .with_column_of::("client") # .with_column_of::("is_deleted") # } # } # # #[entity(SurrealType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Client { # pub name: String, # } # # impl Client { pub fn surreal_table(db: SurrealDB) -> Table { Table::new("client", db) .with_id_column("id") .with_column_of::("name") .with_column_of::("email") .with_column_of::("contact_details") .with_column_of::("is_paying_client") .with_one("bakery", "bakery", Bakery::surreal_table) .with_many("orders", "client", Order::surreal_table) .with_expression("order_count", |t| { let orders = t.get_subquery_as::("orders").unwrap(); orders.get_count_query() }) } # } ``` The generated query has this shape (illustrative): ```sql SELECT *, (SELECT COUNT(*) FROM order WHERE order.client = client.id) AS order_count FROM client ``` The expression field shows up in `ReadableValueSet` results alongside the physical columns. To get it onto your entity struct, add a matching `Option` field — it deserializes like any other column. ### Composing expressions Expressions can reference other expressions. `get_column_expr(name)` resolves a name to either a real column or a computed expression, so one expression can build on another — a `title` composed of `name` and a count subquery via `concat_!`, for instance. All of it renders into a single query. The nesting looks redundant on paper, but SQL engines optimize it away; there is no extra round trip. The Table chapter walks through a concrete title-composition example step by step — see [Working with Tables](../intro/step2-tables.md) rather than repeating it here. ### Nesting arbitrary expressions — select_expression `select_column` is sugar over a more general form. `Table::select_expression(expr)` wraps *any* expression as a single-column subquery over the table's source and conditions — `SELECT FROM table WHERE …` — clearing fields and ordering, since a scalar subquery has exactly one output and no meaningful sort. This is what lets one correlated subquery nest inside another: a two-hop lookup is `outer.select_expression( (inner_subquery) )`. It is also the machinery under the next chapter's implicit references, which automate exactly this recipe. ```admonish info title="Cross-backend enrichment lives at the Dio layer" This whole chapter is a SQL/SurrealDB technique — the backend-support contract is on [Form 2 of the previous chapter](./traversal.md). If you need per-row related values across backends that can't correlate, there's a different tool for that: augmentation, covered in [Relations and Dio](./dio.md). ``` ### SurrealDB notes SurrealDB's record links give it a second way to express a related field: a native idiom path. `client.name` on an order traverses the link in place — no subquery at all, and often cheaper. The [implicit references](./implicit-references.md) chapter uses exactly that lowering automatically. Correlated subqueries still work too. The `order_count` example above runs on SurrealDB as written — `order.client = client.id` correlates through the record link. ### Conclusion At this point you should be able to: 1. **Embed a related field** as a computed column — `with_expression` + `get_subquery_as::(relation)` + `select_column("field")`. 2. **Add relation aggregates** — `get_count_query()` and `get_sum_query(&col)` on the subquery table. 3. **Compose expressions** — `get_column_expr(name)` resolves columns and expressions alike, so expressions can build on each other inside one query. 4. **Nest subqueries** — `select_expression(expr)` wraps any expression as a scalar subquery, one inside another. Next: [Implicit References](./implicit-references.md) — the declarative form of everything in this chapter's first half. --- Source: https://vantage-ui.com/book/relations/implicit-references.html # Implicit References The previous chapter ([Expressions & Subqueries](./expressions.md)) closed with a recipe: take a relation, call `get_subquery_as` to build a correlated subquery for one field, and register it with `with_expression` so it projects as a column. That recipe works, but surfacing a related row's field is common enough to deserve a declarative form. That form is a **dotted name** in `Table::with_active_columns`. Write `"client.name"` and the table traverses the declared `has_one` relation `"client"`, imports the target's `name` field, and projects it as a read-only column aliased under the literal dotted name — no hand-written expression. This feature is also called **implicit references**. The first segment addresses the relation by its **registry name** — whatever you named it in `with_one` — not by the foreign-key column. `bakery_model3` declares `with_one("client", "client_id", …)`, so the path is `client.name` on SQL and SurrealDB alike; a model that names its relations after the FK columns would write `client_id.name` instead. Either convention works — the segment just has to match the declared name. `with_active_columns` does two jobs at once: 1. **Plain names restrict projection.** Only the listed columns are selected — useful on its own for narrowing wide tables. The id column is always projected regardless. 2. **Dotted names traverse relations.** Each dot is one `has_one` hop from the current table to its target. ### One hop and two hops Using the bakery model from this guide — Orders relate to Client via `client_order.client_id`, and Client relates to Bakery via `client.bakery_id`, declared as relations `"client"` and `"bakery"` with `with_one`: ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Bakery { # pub name: String, # } # # impl Bakery { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("bakery", db) # .with_id_column("id") # .with_column_of::("name") # } # } # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Client { # pub name: String, # pub bakery_id: Option, # } # # impl Client { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("bakery_id") # .with_one("bakery", "bakery_id", Bakery::sqlite_table) # } # } # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Order { # pub client_id: Option, # } # # impl Order { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client_order", db) # .with_id_column("id") # .with_column_of::("client_id") # .with_one("client", "client_id", Client::sqlite_table) # } # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite::memory:").await.context("connect")?; // Plain names restrict projection; dotted names traverse has_one relations. let orders = Order::sqlite_table(db).with_active_columns(&[ "id", "client_id", "client.name", // one hop: client_order -> client "client.bakery.name", // two hops: client_order -> client -> bakery ])?; # let _ = orders; # Ok(()) # } ``` `client.name` is one hop. `client.bakery.name` recurses through two — traversal supports an arbitrary depth of `has_one` hops out of the box. On a SQL backend this lowers to nested correlated scalar subqueries: ```sql SELECT id, (SELECT name FROM client WHERE client.id = client_order.client_id) AS "client.name", (SELECT (SELECT name FROM bakery WHERE bakery.id = client.bakery_id) FROM client WHERE client.id = client_order.client_id) AS "client.bakery.name" FROM client_order ``` The alias is the dotted name itself, so rows come back with a **flat key equal to the dotted name**. Reading them needs a populated database, so this block is compiled but not run: ```rust,no_run # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Bakery { # pub name: String, # } # # impl Bakery { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("bakery", db) # .with_id_column("id") # .with_column_of::("name") # } # } # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Client { # pub name: String, # pub bakery_id: Option, # } # # impl Client { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("bakery_id") # .with_one("bakery", "bakery_id", Bakery::sqlite_table) # } # } # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Order { # pub client_id: Option, # } # # impl Order { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client_order", db) # .with_id_column("id") # .with_column_of::("client_id") # .with_one("client", "client_id", Client::sqlite_table) # } # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:bakery.sqlite").await.context("connect")?; # let orders = Order::sqlite_table(db) # .with_active_columns(&["id", "client.name", "client.bakery.name"])?; for row in orders.list_values().await?.values() { let get = |k: &str| row.get(k).map(|v| format!("{v}")).unwrap_or_default(); println!( " order {}: client.name={}, client.bakery.name={}", get("id"), get("client.name"), get("client.bakery.name"), ); } # Ok(()) # } ``` There is no nested object to unwrap — `row.get("client.bakery.name")` is a plain lookup on a flat record. The full example lives in `bakery_model3/examples/implicit-references.rs`; run it with `cargo run -p bakery_model3 --example implicit-references`. ### Validation at table construction Dotted names are ordinary strings, so the compiler cannot vet them. Instead, `with_active_columns` returns `Result` and validates everything as it runs — when the table definition is constructed, typically in model or startup code, before any query is made. A mistake surfaces there, never as a silently empty column at fetch time: - an unknown column or unknown relation in a dotted name; - a `has_many` hop — traversal is `has_one`-only, because a to-many field is a set, not a value (for an aggregate over the set, a count or a sum, use the previous chapter's `get_count_query`); - a backend that can lower neither a correlated subquery nor a native path nor a join (CSV, REST) — its `TableSource::supports_traversal` is `false`, and dotted names are refused up front. MongoDB lowers dotted names into `$lookup` stages, except over a `pipeline:` view. Traversal is also same-datasource only; enriching rows with fields from a *different* datasource belongs to Dio augmentation ([Dio](./dio.md)). ### Why not let a bad dotted name fail at fetch time? Because a fetch-time failure in a projection is invisible. The query still runs, the column comes back empty, and nothing tells you whether the relation was misspelled or the data is genuinely absent. Constructing the table is the moment you have full knowledge — the relation registry, the target's columns, the backend's capabilities — so that's where every check runs. ### Read-only semantics An imported column is a computed projection. No backend can honestly store `client.name` on the `client_order` table — the value lives in a different row of a different table. So imported columns carry write-path enforcement: - They are flagged `calculated` in vista metadata, so downstream consumers know the column is derived (see [Vistas](./vistas.md)). - They are **stripped from full-record write payloads** — insert, replace, generated-id insert. A read-modify-save round-trip never persists `client.name` as a real field. - They are **rejected outright in a `patch`**. A partial payload naming a read-only column is explicit intent, and silently dropping it would turn the patch into a successful no-op. The key difference: strip vs reject. A full-record payload naturally contains everything you read — including imported columns — so stripping them is the correct interpretation. A patch payload contains only what you chose to send, so a read-only column there is a mistake worth an error. Imported columns are also excluded from quicksearch and (on SurrealDB) not orderable — the dotted name is a projection alias, not a physical field. ### Per-backend lowering SQL backends lower dotted names through the generic chain: nested correlated scalar subqueries built on the same `get_subquery_as`/`select_expression` machinery from the previous chapter. A backend that already supports that recipe gets implicit references with nothing backend-specific to implement. SurrealDB overrides `TableSource::traversal_path_expr` and emits a native **idiom path** instead — `client.name` — with each segment escaped separately. Escaping the joined path as one identifier would produce a literal ⟨client.name⟩ field lookup: a dead column that matches nothing. The idiom descends each hop's **link field**, not the relation's registry name — a relation declared `with_one("owner", "client", …)` still lowers to `client.name`. Multi-hop comes for free in the idiom. ### Why not just use with_expression? You still can, and sometimes you should. The key difference: reach for `with_expression` + `get_subquery_as` when you need an *arbitrary expression* — arithmetic, aggregates, raw escapes. Reach for a dotted column when you just want a related field. The dotted form adds what the manual recipe can't: - **Proper errors** — the manual recipe's checks are `Option`/`unwrap` panics (`select_column` returns `None` for a misspelled column); the dotted form returns `Result`, with an error naming the bad relation, column, cardinality, or backend. - **A typed imported column** — the definition is cloned from the target's column, so type metadata travels with it. - **Enforced read-only write semantics** — a `with_expression` column has no write-path story at all. ### Gotchas - `Table::derive_from` does not inherit implicit references. A derived table starts with no active set, and listing an imported dotted column in its column inheritance copies only a bare definition. Re-declare the dotted names on the derived table. - Columns added after `with_active_columns` are not in the active set and won't project — declare the active set last. - Expression-only columns (registered via `with_expression` with no column definition) may be named in the active set; they stay projected. - Same-datasource only, by design — see [Dio](./dio.md) for the cross-datasource alternative. ### Conclusion At this point you should be able to: 1. **Restrict projection** with plain names in `with_active_columns` (the id column always projects). 2. **Import related fields** with dotted names — one hop or many, over declared `has_one` relations. 3. **Read them back** via flat keys equal to the dotted name. 4. **Predict every construction-time error** — unknown column, unknown relation, `has_many` hop, backend without traversal support. 5. **Explain the write-path semantics** — stripped from full-record writes, rejected in a patch. 6. **Choose the right tool** — dotted column for a related field, `with_expression` for an arbitrary expression. The next page, [Vista, YAML and Rhai](./vistas.md), covers how the same declarations work from YAML specs, and how the `calculated` flag surfaces at the erased layer. --- Source: https://vantage-ui.com/book/relations/vistas.html # Vista, YAML and Rhai Everything so far assumed you could write `Table` in your source code. Generic consumers can't. A UI grid, an admin panel, a scripting surface, a config-driven tool — none of them know your model types at compile time. What they hold is a `Vista`: the schema-bearing runtime handle introduced in [the Vista chapter](../intro/step4-vista.md). The question this page answers: what happens to relations when the type is erased? The answer is that they survive in three forms: 1. **Metadata** — the relation names, targets, kinds and join keys are introspectable, so a consumer can discover them at runtime. 2. **Traversal** — `get_ref` still works. Hand it a loaded row, get back a new `Vista` narrowed to the related rows. 3. **Capabilities** — an honest contract about whether this backend serves traversal at all, so a consumer knows what to offer before trying. ### The factory A `Vista` is built by a **vista factory**: each driver's factory implements the [`VistaFactory`](vantage_vista::VistaFactory) trait, covered in [Vista integration](../new-persistence/step8-vista-integration.md). The factory's defining ability is working **by name**: where typed code passes `User::table(db)`, a generic consumer asks for `"users"` and gets a Vista back. How the name resolves is the factory's business — typically it loads table specs from files — and that indirection is what lets config-driven tools, scripts, and by-name traversal address models they have no Rust types for. Behind a name, a vista is built one of two ways: 1. **From a typed table** — `from_table` wraps the table your model already builds. One handle type comes out, whichever backend went in: ```rust # use vantage_csv::{AnyCsvType, Csv}; # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # pub struct Product { # pub name: String, # } # # impl Product { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # } # } # # #[entity(CsvType)] # #[derive(Debug, Clone, Default)] # pub struct Order { # pub client_id: String, # } # # impl Order { # pub fn csv_table(csv: Csv) -> Table { # Table::new("order", csv).with_column_of::("client_id") # } # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let sqlite = SqliteDB::connect("sqlite::memory:").await.context("connect")?; # let csv = Csv::new("data"); let vistas = vec![ sqlite.vista_factory().from_table(Product::sqlite_table(sqlite.clone()))?, csv.vista_factory().from_table(Order::csv_table(csv.clone()))?, ]; # assert_eq!(vistas[0].name(), "product"); # Ok(()) # } ``` 2. **From a YAML spec** — no Rust model at all: columns, relations, and computed fields declared as data ([Config-Driven Vistas](../config-driven-vistas.md)). This path is how the declarations in the last section of this page become live, traversable relations. A spec's *source* doesn't have to be a physical table, either — a `rhai:` block builds it from a query (a query-sourced vista), and a `base:` block derives it from another spec. Whichever route constructs it, the factory folds the table's relations into `VistaMetadata` as [`Reference`](vantage_vista::Reference) entries — name, kind, target, join keys. The erased handle carries enough to introspect the relations and to traverse them. Nothing about the relation is lost in the erasure; only the compile-time names are. ### Metadata: introspecting relations A `Reference` is one model for every relation shape. It is built with the constructor of its kind, so the kind and its fields always agree: - `Reference::has_one(name, target, key)` — one target row, found by a key on the parent row. - `Reference::has_many(name, target, key)` — the target rows that carry the parent's key. - `Reference::contains_one(name, host_column)` — one record embedded as an object in a column. - `Reference::contains_many(name, host_column)` — records embedded as an array in a column. A join's key is a [`JoinKey`](vantage_vista::JoinKey): read the parent row's `from` column and constrain the target's `to` column to that value (`JoinKey::to_target_id(from)` constrains the target's id column). A join may carry several keys, and every one of them must match. `Vista` exposes the entries directly: `references()` is the ordered map of name to `Reference`, and `reference(name)` looks one up. ```rust # use vantage_vista::mocks::MockShell; # use vantage_vista::{Column, JoinKey, Reference, ReferenceKind, Vista, VistaMetadata}; # let metadata = VistaMetadata::new() .with_column(Column::new("id", "String").with_flag("id")) .with_column(Column::new("country_id", "String")) .with_id_column("id") .with_reference(Reference::has_many("orders", "order", JoinKey::new("id", "client_id"))) .with_reference(Reference::has_one("country", "country", JoinKey::to_target_id("country_id"))); let clients = Vista::new("client", MockShell::new().with_metadata(metadata)); for (name, reference) in clients.references() { println!("{name}: {} -> {}", reference.kind.as_str(), reference.target); } assert_eq!(clients.reference("orders").unwrap().kind, ReferenceKind::HasMany); ``` This is what lets a UI render relation tabs or context-menu entries without knowing the model. The loop prints "orders: has_many -> order" and "country: has_one -> country" — enough to label a tab and decide whether the target is a single row or a set — and it learned that from the handle, not from your source code. `ReferenceKind::is_contained` separates the embedded kinds from the joins. ### Traversal: get_ref on a loaded row The erased twin of the typed `get_ref_from_row` is [`Vista::get_ref`](vantage_vista::Vista::get_ref), which takes a relation name and an `Option<&Record>`. Hand it a loaded row — `Record` is the erased record type — and get back a new `Vista` narrowed to the related rows. The relation is named by string because that's all a generic consumer has; the reference's keys supply the columns. Each key becomes one equality condition on the target, applied through `Vista::add_condition`, so the target's `query()` records the join: ```rust # use ciborium::Value as CborValue; # use vantage_dataset::ReadableValueSet; # use vantage_types::Record; # use vantage_vista::mocks::MockShell; # use vantage_vista::{Column, JoinKey, Reference, Vista, VistaMetadata}; # # fn row(pairs: &[(&str, &str)]) -> Record { # pairs.iter().map(|(k, v)| (k.to_string(), CborValue::Text(v.to_string()))).collect() # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let orders = MockShell::new() # .with_metadata( # VistaMetadata::new() # .with_column(Column::new("id", "String").with_flag("id")) # .with_column(Column::new("client_id", "String")) # .with_id_column("id"), # ) # .with_record("o1", row(&[("id", "o1"), ("client_id", "marty")])) # .with_record("o2", row(&[("id", "o2"), ("client_id", "marty")])) # .with_record("o3", row(&[("id", "o3"), ("client_id", "doc")])); # let shell = MockShell::new() # .with_metadata( # VistaMetadata::new() # .with_column(Column::new("id", "String").with_flag("id")) # .with_id_column("id") # .with_reference(Reference::has_many("orders", "order", JoinKey::new("id", "client_id"))), # ) # .with_record("marty", row(&[("id", "marty")])) # .with_ref_target("orders", orders); # let clients = Vista::new("client", shell); let marty = clients.get_value("marty").await?.expect("client exists"); let orders = clients.get_ref("orders", Some(&marty))?; assert_eq!(orders.list_values().await?.len(), 2); let all_orders = clients.get_ref("orders", None)?; assert_eq!(all_orders.list_values().await?.len(), 3); # Ok(()) # } ``` Passing `None` gives the bare, unconditioned target. This is the insert destination for nested creates: when a UI wants to add a related row, it needs the target table without any narrowing applied. A contained relation works the same way: from a row, `get_ref` returns a Vista over the records embedded in the row's host column; with `None`, the empty contained schema. Both forms return `Vista`, so traversal chains and the consumer never leaves the erased world. A grid that drills from clients into orders into line items is calling `get_ref` three times, each time holding nothing more specific than a `Vista` and a row. ```admonish info title="Record ids that round-tripped through JSON" At the erased layer, SurrealDB record ids travel as strings (`"table:key"`). The backend coerces a string-shaped record id back into a typed record id when narrowing, so a row that round-tripped through JSON or a script still traverses correctly. ``` ### The scripting surface Rhai data scripts (the `vantage-vista` rhai feature) traverse with `ref(relation)` on a table handle, which follows the relation from every row of a set. A one-row set goes through `get_ref`, so it follows the same metadata and backend traversal described here. Several rows become an `in` condition on the bare target's join column (which needs a relation joined on one key), a `limit(n)` or `offset(n)` before the `ref` picks the rows it follows, and an empty set gives an empty target without a query. See [The Table Handle](../rhai/tables.md#following-relations). A relation's foreign key may be a [computed column](../rhai/computed.md) (a `lazy:` column). A has-one through one works, because the parent row carries the computed value. A has-many whose key is a computed column of the child is refused by `get_ref` with an error naming the relation and column, since the backend can't filter the child by a value it doesn't store. ### Capabilities: the honest contract Not every backend serves every traversal form — the [traversal chapter](./traversal.md) already showed that set-level traversal needs subqueries. At the typed layer, you knew that when you wrote the code. At the erased layer, the consumer has to ask. That's what [`VistaCapabilities`](vantage_vista::VistaCapabilities) is for: a set of [`Capability`](vantage_vista::Capability) values, each naming one method family the shell answers without refusing. The ones that matter for relations: - `Capability::Traverse` — the shell answers `get_ref`. A shell without it refuses every traversal with an `Unsupported` error naming the capability. - `Capability::FilterOps` — the shell takes operator conditions, `in` among them. A script's multi-row `ref` narrows the target with an `in` condition, so it needs this on the target. - `Capability::ConfineWrites` — writes through a narrowed Vista stay inside the narrowed set. An insert into the result of `get_ref` is such a write. Consumers branch on these before offering the corresponding affordance: a grid checks `vista.capabilities().has(Capability::Traverse)` before showing relation tabs, and a nested-create form checks the target's `Insert` and `ConfineWrites`. The same set appears in the Rhai `capabilities()` map under each capability's snake_case name (`traverse`, `filter_ops`), so scripts and config-driven tools can branch the same way. Two relation features need no flag of their own. A reference with a Rhai build script carries it in `Reference::build_script`, and the shell runs it in place of the key match (SurrealDB, MongoDB and MySQL accept one). A dotted column either lowers when the vista is built or fails the build, so a vista that exists already serves every dotted column it declares. ### Across datasources: the Catalog Everything above stays inside one datasource — a `Vista`'s references forward to the wrapped table, and a single Vista deliberately knows nothing about *other* datasources. Traversal across datasources lives one layer up, in a [`Catalog`](vantage_vista::Catalog): a map from model name to a loader that builds a fresh, unconditioned Vista, whichever backend backs it. You register models by name with `Catalog::with`, then narrow a target the catalog builds with `Reference::narrow`. The reference is the same `Reference` the metadata holds; here it names a model in another backend: ```rust # use ciborium::Value as CborValue; # use vantage_dataset::ReadableValueSet; # use vantage_types::Record; # use vantage_vista::mocks::MockShell; # use vantage_vista::{Catalog, Column, JoinKey, Reference, Vista, VistaMetadata, VistaResolver}; # # fn row(pairs: &[(&str, &str)]) -> Record { # pairs.iter().map(|(k, v)| (k.to_string(), CborValue::Text(v.to_string()))).collect() # } # # fn meta(columns: &[&str]) -> VistaMetadata { # columns # .iter() # .fold(VistaMetadata::new(), |m, c| m.with_column(Column::new(*c, "String"))) # .with_id_column("id") # } # # // Stands in for a client table in one backend. # fn client_vista() -> Vista { # let shell = MockShell::new() # .with_metadata(meta(&["id", "bakery_id"])) # .with_record("marty", row(&[("id", "marty"), ("bakery_id", "hill-valley")])); # Vista::new("client", shell) # } # # // Stands in for a bakery API in another. # fn bakery_vista() -> Vista { # let shell = MockShell::new() # .with_metadata(meta(&["id", "name"])) # .with_record("hill-valley", row(&[("id", "hill-valley"), ("name", "Hill Valley")])) # .with_record("twin-pines", row(&[("id", "twin-pines"), ("name", "Twin Pines")])); # Vista::new("bakery", shell) # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let catalog = Catalog::new() .with("client", |_: &str| Ok(client_vista())) .with("bakery", |_: &str| Ok(bakery_vista())); let bakery = Reference::has_one("bakery", "bakery", JoinKey::new("bakery_id", "id")); let marty = catalog.vista("client")?.get_value("marty").await?.expect("client exists"); let hers = bakery.narrow(catalog.vista(&bakery.target)?, &marty)?; assert_eq!(hers.list_values().await?.len(), 1); # Ok(()) # } ``` `narrow` applies one equality condition per key, through `Vista::add_condition`. How each condition is honoured — SQL `WHERE`, in-memory filter, REST path or query parameter — is the target driver's concern at fetch time. A name the catalog doesn't hold is a not-found error, and a loader that fails keeps its own error. Because loaders build a fresh Vista on every call, the catalog composes with hot-reloading inventories. The catalog is a [`VistaResolver`](vantage_vista::VistaResolver), so it goes wherever a resolver is expected — the Dio layer's augmentation resolves its detail sources through one, which the [next page](./dio.md) picks up. #### Why not register cross-datasource relations on the catalog? Because the relation already has a home: the `Reference`. A second relation type on the catalog would be one more spelling of "read this column, constrain that one", with its own constructor and its own traversal call. Keeping one model means a reference read out of a vista's metadata and a reference written by hand for a cross-backend join narrow the same way. The catalog stays what it is good at: building a model by name. ### Declaring relations in YAML and Rhai Config-driven vistas (full chapter: [Config-driven vistas](../config-driven-vistas.md)) declare relations in the table spec — either as column-level sugar or a top-level map. The factory reads the spec, registers the relations, and the resulting `Vista` traverses them exactly like one built `from_table`: ```yaml name: product columns: id: type: string flags: [id] batch: type: string references: batch # sugar: has_one named `batch`, FK = this column batch.name: # implicit reference through that relation type: string references: tags: table: tag kind: has_many foreign_key: product_id ``` The column-level `references: batch` sugar names the relation after the *column* and uses the column as the foreign key, matched to the target's id. That is the common case: a column named after the table it points at. A `references:` entry with the same name takes precedence over the sugar. The full form (`table`, `kind`, `foreign_key`) names the relation independently of any column — you need that when two relations point at the same table. A join that needs more than one column pair lists them under `keys:`. Each pair reads the parent row's `from` column and constrains the target's `to` column, and the child must match every pair: ```yaml references: deployments: table: deployment kind: has_many keys: - { from: product_id, to: product } - { from: version_id, to: version } ``` When `keys:` is given it describes the whole join; `foreign_key:` applies only when it is absent. When a relation needs more than key matches — extra conditions, ordering, a search — give the reference a `rhai:` build script in its driver block (SurrealDB, MongoDB and MySQL accept one): ```yaml references: due_orders: table: order kind: has_many foreign_key: client surreal: rhai: | table("order").where("client", row.id).where("status", "due") ``` The script runs lazily when the relation is traversed, with the parent record in scope as `row`, and must end on a table handle: start it with `table("")` and chain the [narrowing verbs](../rhai/tables.md#narrowing) (see [Reference build scripts](../rhai/surfaces.md#reference-build-scripts)). Without `rhai:`, the relation falls back to the plain key match. The script lands in the reference's `build_script`, which the shell evaluates instead of the keys. Imported dotted columns — like `batch.name` above, the YAML form of [implicit references](./implicit-references.md) — go through the same traversal import as the Rust API, with the same construction-time validation: a bad dotted column fails the spec load, not the first fetch. They arrive in metadata flagged `calculated`. That flag means read-only for consumers: the value comes from a traversal, not from a column you can write. This is how a UI knows not to offer editing on them. [Computed columns](../rhai/computed.md) (`lazy:`) carry the same flag for the same reason. The key difference: at the typed layer *you* know which traversal forms are safe — you wrote the code against a backend you chose. At the erased layer, the *capabilities* say so. Same relations, discovered instead of known. ### Conclusion At this point you should be able to: 1. **Enumerate a Vista's relations** — `references()` for every `Reference`, `reference(name)` for one, its `kind` for the cardinality and `keys` for the join. 2. **Build a reference by hand** — `Reference::has_one`, `has_many`, `contains_one`, `contains_many`, with `JoinKey::new` or `JoinKey::to_target_id`. 3. **Traverse from a loaded erased row** — `get_ref(relation, Some(&row))` returns a narrowed `Vista`; `get_ref(relation, None)` returns the bare target for nested creates. 4. **Check the capability first** — `Capability::Traverse` before offering traversal, `FilterOps` and `ConfineWrites` on the target for multi-row script traversal and nested inserts. 5. **Traverse across datasources** — register models with `Catalog::with`, then `reference.narrow(catalog.vista(&reference.target)?, &row)`. 6. **Declare relations in YAML and Rhai** — column-level `references:` sugar, the full top-level form, `keys:` for multi-column joins, implicit dotted columns through a relation, and `rhai:` build scripts for traversals that need more than a key match — all consumed by the same vista factory that wraps typed tables. 7. **Explain why `calculated` columns are read-only** in generic UIs — their values come from traversal, not from a writable column. Next: [what happens above the Vista](./dio.md) — combining erased handles across backends. --- Source: https://vantage-ui.com/book/relations/dio.html # Relations and Dio Everything in this guide so far has executed at or below the Vista: typed relations on the Table, traversal narrowing, subquery expressions, implicit references. This final chapter climbs one layer higher. A `Dio` keeps a live local copy of a data segment (a cache), fills it through a `Lens`, announces changes on an event bus, and serves reactive views (`Scenery`). You built one in the intro paths ([Dio over SQL](../intro/step5-sql-dio.md), [Scenery](../intro/step7-scenery.md)). Two questions matter here: what happens to the relation machinery you already know once a Dio sits on top — and what relation shape exists *only* at the Dio layer, because nothing below it can express one. ### Same-persistence relations resolve beneath the Dio Traversal, subquery expressions, and implicit references all happen in the Table or Vista *under* the Dio. The Dio caches the resulting rows like any others. If your table declares `with_active_columns(&["id", "client.name"])`, the `client.name` column is already part of each row when it enters the cache — nothing at the Dio layer knows or cares that it was traversed. From the cache's point of view, a traversed column and a plain column are indistinguishable. The practical consequence is the master/detail pattern in a UI: **traverse at the Vista, cache at the Dio**. Take a row from the cached master and call `Dio::get_ref(relation, &row)`. It runs the erased `Vista::get_ref` from the [Vistas chapter](./vistas.md) on the Dio's master and builds a new Dio over the narrowed target, on the same Lens: ```rust # use ciborium::Value as CborValue; # use vantage_dataset::ReadableValueSet; # use vantage_diorama::{Cache, Lens, Loader}; # use vantage_types::Record; # use vantage_vista::mocks::MockShell; # use vantage_vista::{Column, JoinKey, Reference, Vista, VistaMetadata}; # # fn row(pairs: &[(&str, &str)]) -> Record { # pairs.iter().map(|(k, v)| (k.to_string(), CborValue::Text(v.to_string()))).collect() # } # # fn clients() -> Vista { # let orders = MockShell::new() # .with_metadata( # VistaMetadata::new() # .with_column(Column::new("id", "String").with_flag("id")) # .with_column(Column::new("client_id", "String")) # .with_id_column("id"), # ) # .with_record("o1", row(&[("id", "o1"), ("client_id", "marty")])) # .with_record("o2", row(&[("id", "o2"), ("client_id", "marty")])) # .with_record("o3", row(&[("id", "o3"), ("client_id", "doc")])); # let shell = MockShell::new() # .with_metadata( # VistaMetadata::new() # .with_column(Column::new("id", "String").with_flag("id")) # .with_id_column("id") # .with_reference(Reference::has_many("orders", "order", JoinKey::new("id", "client_id"))), # ) # .with_record("marty", row(&[("id", "marty")])) # .with_record("doc", row(&[("id", "doc")])) # .with_ref_target("orders", orders); # Vista::new("client", shell) # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let lens = Lens::builder() .cache(Cache::memory()) .loader(Loader::eager(|cx| async move { let rows = cx.master().list_values().await?; cx.reconcile(rows).await })) .build()?; let clients = lens.dio(clients()).build().await?; let marty = clients.cached().get_value("marty").await?.expect("cached"); // A Dio of its own over marty's orders, with its own cache table. let orders = clients.get_ref("orders", &marty).await?; assert_eq!(orders.cached().list_values().await?.len(), 2); # Ok(()) # } ``` Each narrowed detail set gets its own cache table, keyed by the narrowed target's query, so switching between master rows switches between already-cached detail sets rather than re-narrowing one shared cache. The detail Dio inherits the master's live setting, but not its augmentations, which describe the master's table. `Dio::get_ref_target(relation)` is the same call without a row: a Dio over every eligible target row, which is what a reference picker in an edit form lists. When the detail lives in another datasource, build its Vista through a [`Catalog`](vantage_vista::Catalog) and narrow it with `Reference::narrow`, as the [Vistas chapter](./vistas.md#across-datasources-the-catalog) shows, then put a Dio in front of it with `lens.dio(narrowed).build()`. A narrower view of the *same* rows is a different operation: no relation, just more conditions. `dio.derive()` collects them and builds a new Dio over the narrowed set: ```rust # use ciborium::Value as CborValue; # use vantage_dataset::ReadableValueSet; # use vantage_diorama::{Cache, Lens, Loader}; # use vantage_types::Record; # use vantage_vista::mocks::MockShell; # use vantage_vista::{Column, Condition, FilterOp, Vista, VistaMetadata}; # # fn row(pairs: &[(&str, &str)]) -> Record { # pairs.iter().map(|(k, v)| (k.to_string(), CborValue::Text(v.to_string()))).collect() # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let orders = Vista::new( # "order", # MockShell::new() # .with_metadata( # VistaMetadata::new() # .with_column(Column::new("id", "String").with_flag("id")) # .with_column(Column::new("client_id", "String")) # .with_id_column("id"), # ) # .with_record("o1", row(&[("id", "o1"), ("client_id", "marty")])) # .with_record("o2", row(&[("id", "o2"), ("client_id", "doc")])), # ); # let lens = Lens::builder() # .cache(Cache::memory()) # .loader(Loader::eager(|cx| async move { # let rows = cx.master().list_values().await?; # cx.reconcile(rows).await # })) # .build()?; # let dio = lens.dio(orders).build().await?; let martys = dio .derive() .with_condition(Condition::new("client_id", FilterOp::Eq, "marty")) .build() .await?; assert_eq!(martys.cached().list_values().await?.len(), 1); # Ok(()) # } ``` `martys` keeps a cache table and an event bus of its own, but it isn't cut off from `dio`. Every Dio over the `order` table on this Lens joins the table's hub. What one of them settles (a confirmed flash, an `apply_change`, what a reconcile changed, a change its live follow received) reaches the others, and each judges it against its own conditions. Move `o1` to doc through `dio` and it leaves `martys`. Insert an order for marty and it shows up in `martys`. A row that only left a narrowed Dio's set is never taken as a delete: the others re-read it through their own masters. The hub is keyed by the master's driver and name. A layer that renames vistas, one name per session, passes the backend table to `DioBuilder::hub(key)` so its Dios still hear each other. An explicit key also names the cache table, so Dios keyed apart never share cached rows. A master narrowed by a rule its `query()` doesn't record (a table hook's) is built with `DioBuilder::recheck_facts()`, which re-reads each reported row through that master rather than judging it in memory. `recheck_facts` keeps nothing apart. Dios are told apart by vista name, recorded query and hub key only, so a hidden per-session rule needs its own vista name, a vista condition, or its own hub key. Without one, two sessions share a cache table and each can list the other's rows. ### The join the layers below cannot express A [`Catalog`](./vistas.md#across-datasources-the-catalog) can already *navigate* across datasources — hand a `Reference` a parent row and the catalog's target, and it returns the related set from another backend. What nothing below the Dio can do is *merge*: show master rows whose columns come from two sources at once. That is a join, and joins are governed by the capability contract — a SQL Vista can join its own tables, a REST Vista cannot, and two different backends can never push a join down to either engine. A Vista also has nowhere to put stitched rows: no cache, no viewport, no notion of "only visible rows pay". Where the backend can't, the layer above fills the gap — that is the whole job of the Dio. **Augmentation wires two Vistas into one `Dio`**: a *master* that is listed, and a *detail* source loaded one row at a time and merged on top. The detail is resolved by name through a [`VistaResolver`](vantage_vista::VistaResolver) such as a `Catalog`, so it is persistence-agnostic — a REST master enriched by a `cmd` detail, or a SQL master enriched from another database. It runs in two passes: 1. **List pass** — the master is listed cheaply; each row enters the cache marked `Incomplete`. 2. **Detail pass** — viewport-driven: for each *visible* incomplete row, the augmentation resolves its detail Vista, fetches the matching record, merges the chosen columns, and the row flips to `Fresh`. Rows off-screen are never fetched; hydrated rows are never re-fetched. A failed detail fetch marks only that row. ### Declaring an augmentation Augmentations are a property of the Dio, fixed when it is built: `DioBuilder::augment` takes the list, and supplying at least one is what engages the two-pass behaviour. Each augmentation answers four questions — which detail Vista, how a master row selects its detail, how it is fetched, and which columns to lift: ```rust # use std::sync::Arc; # # use vantage_diorama::{Augmentation, Cache, Detail, Fetch, Lens, MergeRule, Source}; # use vantage_vista::mocks::MockShell; # use vantage_vista::{Catalog, Column, Vista, VistaMetadata}; # # fn meta(columns: &[&str]) -> VistaMetadata { # columns # .iter() # .fold(VistaMetadata::new(), |m, c| m.with_column(Column::new(*c, "String"))) # .with_id_column("id") # } # # fn tfstate_detail() -> Vista { # Vista::new("tfstate_detail", MockShell::new().with_metadata(meta(&["id", "resources", "serial"]))) # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let buckets = Vista::new("buckets", MockShell::new().with_metadata(meta(&["id", "key", "size"]))); # let lens = Lens::builder().cache(Cache::memory()).build()?; let catalog = Catalog::new().with("tfstate_detail", |_: &str| Ok(tfstate_detail())); let dio = lens .dio(buckets) .augment(vec![Augmentation::new( Detail::Resolved { resolver: Arc::new(catalog), // resolves the detail by name name: "tfstate_detail".into(), }, Source::Column { from: "key".into(), to: None }, Fetch::PerRow, MergeRule::lift(["resources", "serial"]), )]) .build() .await?; # let _ = dio; # Ok(()) # } ``` - `detail` names the detail Vista: `Detail::Resolved` looks it up by name through a resolver; `Detail::Fixed(Arc)` takes one directly, for a side table no resolver knows. - `source` picks how a master row selects its detail: `Source::Id` matches master.id → detail.id; `Source::Column { from, to }` matches `master[from]` → `detail[to]`, or detail.id when `to` is `None`; `Source::Build(closure)` performs arbitrary narrowing from the whole row, and is per-row only. - `fetch` picks how the detail is read: `Fetch::PerRow` reads one record per master row; `Fetch::Custom(closure)` is a caller-supplied async fetch. - `merge` lists the detail columns to lift. `MergeRule::lift_all()` lifts all of them; on a name clash the detail wins. Diorama has no YAML form of these types. A consumer maps its own configuration onto them: vantage-ui reads an `augment:` block from a table's configuration and builds the same `Augmentation`: ```yaml augment: - table: tfstate_detail source: { kind: column, from: key } fetch: { kind: per_row } merge: [resources, serial, outputs] ``` ```admonish info title="Script sources" A scripted source is a `Build` closure the consumer writes; diorama never sees a script string. See [Augmentation](../augmentation.md#rhai-as-a-closure-factory). ``` To see the two passes run, try the example — two in-memory Vistas, list pass then detail pass: ```console cargo run -p vantage-diorama --example augmentation ``` The full treatment of augmentation — merge rules, capability reasoning, the planned batched fetch and keyed caching — is in the [Augmentation chapter](../augmentation.md); the tutorial form is [intro step 6](../intro/step6-augmentation.md). ### Why not model the detail as a foreign-key relation? A relation resolves *within* one persistence and can be pushed down as a join. The detail here may be a different backend entirely; there is nothing to join on and no engine that could honour it. Augmentation is the cross-Vista form, stitched by the Dio. ### Choosing between implicit references and augmentation Both produce the same user-visible result: a row enriched with related fields. Which one to use depends on where the data lives and who should pay for the fetch. #### Path A: implicit references (same datasource, in-query) One query; the backend does the work. The value is present on every row, including off-screen ones, it is validated when the table is constructed, and it carries read-only column semantics. Choose this when both tables live in one datasource that supports traversal (SQL, SurrealDB, MongoDB) — see [Implicit References](./implicit-references.md). #### Path B: augmentation (any two datasources, per-row client-side join) Two passes; expensive work follows the user's attention — only viewport rows pay, and only once. It works across arbitrary backends, and failed details degrade per-row, not per-page. Choose this when the sources differ, or when the detail is expensive and you want viewport-driven hydration even within one backend. The key difference: implicit references push the join *down* into the engine; augmentation lifts it *up* into the cache. Same user-visible result — a row enriched with related fields — chosen by where the data lives and who should pay for the fetch. ### Guide conclusion This closes the Relations & Traversal guide. At this point you should be able to: 1. **Say where each relation form executes** — traversal and implicit references in the engine, erased traversal through the Vista handle, augmentation in the Dio's cache. 2. **Build master/detail** — `dio.get_ref(relation, &row)` for a Dio over the related rows, each detail set in its own cache table; `dio.get_ref_target(relation)` for every eligible row. 3. **Narrow a Dio** — `dio.derive().with_condition(..).build().await?` for a narrower view of the same rows. 4. **Declare an augmentation** — `Detail::Resolved` or `Fixed`, a `Source`, a `Fetch` and a `MergeRule`, handed to `lens.dio(master).augment(..)` before `build()`. 5. **Choose deliberately** between implicit references and augmentation based on where the data lives and who should pay for the fetch. Relations in Vantage are declared once on the model and then travel — into queries, through erasure, and up into the cache — each layer serving the form the one below can't. --- Source: https://vantage-ui.com/book/mda.html # Model-Driven Architecture Vantage is opinionated about how you structure business software. The framework prescribes a **Model-Driven Architecture** (MDA) where entities, relationships, and business rules live in a shared model crate — decoupled from any specific persistence, UI, or API layer. --- ## The idea Most Rust projects scatter database queries across handlers, services, and utilities. When the schema changes or a new backend is needed, you're hunting through dozens of files. Vantage inverts this. You define your entities **once** in a model crate, then every consumer — REST API, CLI, desktop UI, background worker — uses the same `Table` and `DataSet` interfaces. The model is the source of truth. ```mermaid graph TD M[Model Crate] --> API[Axum API] M --> CLI[CLI Tool] M --> UI[Desktop UI] M --> Worker[Background Worker] M --> Test[Mock Tests] style M fill:#4a7c59,color:#fff ``` --- ## Anatomy of a model crate The `bakery_model3` crate in the Vantage repo demonstrates the pattern. Here's how it's structured: ```text bakery_model3/ ├── src/ │ ├── lib.rs ← re-exports, DB connection helpers │ ├── animal.rs ← a custom column type │ ├── bakery.rs ← Bakery entity + table constructors │ ├── client.rs ← Client entity + table constructors │ ├── order.rs ← Order entity + table constructors │ └── product.rs ← Product entity + table constructors └── examples/ ├── 0-intro.rs ← conditions and traversal over CSV ├── cli-vista.rs ← multi-source CLI using Vista ├── contained-surreal.rs ← embedded order lines on SurrealDB ├── dynamo-smoke.rs ← the same model on DynamoDB └── implicit-references.rs ← dotted columns ``` Each entity file follows the same pattern: **struct → trait impls → table constructors**. --- ## Defining entities An entity is a plain Rust struct. The `#[entity(...)]` macro generates `Record` conversions for each persistence's type system: ```rust # use vantage_csv::AnyCsvType; # use vantage_mongodb::AnyMongoType; # use vantage_sql::postgres::AnyPostgresType; # use vantage_sql::sqlite::AnySqliteType; # use vantage_surrealdb::types::AnySurrealType; # use vantage_types::entity; # #[entity(CsvType, SurrealType, SqliteType, PostgresType, MongoType)] #[derive(Debug, Clone, PartialEq, Default)] pub struct Client { pub name: String, pub email: String, pub contact_details: String, pub is_paying_client: bool, pub bakery_id: Option, } ``` One struct, five type systems. The same `Client` works with CSV files, SurrealDB, SQLite, Postgres, and MongoDB — each using its own native `Record` representation. The macro needs each `Any*Type` in scope, which is why a model file imports them from the driver crates. ```admonish tip title="No id field in the struct" The entity struct does **not** include an `id` field. IDs are managed by the table via `with_id_column()` — keeping the entity focused on business data. ``` --- ## Table constructors Each entity provides table constructors per persistence. These are plain functions that return `Table` with columns, relationships, and computed fields pre-configured: ```rust # use vantage_csv::{AnyCsvType, Csv}; # use vantage_sql::sqlite::{AnySqliteType, SqliteDB}; # use vantage_surrealdb::surrealdb::SurrealDB; # use vantage_surrealdb::thing::Thing; # use vantage_surrealdb::types::AnySurrealType; # use vantage_table::table::Table; # use vantage_types::entity; # # #[entity(CsvType, SurrealType, SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Bakery { # pub name: String, # } # # impl Bakery { # pub fn csv_table(csv: Csv) -> Table { # Table::new("bakery", csv).with_column_of::("name") # } # pub fn surreal_table(db: SurrealDB) -> Table { # Table::new("bakery", db).with_id_column("id").with_column_of::("name") # } # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("bakery", db).with_id_column("id").with_column_of::("name") # } # } # # #[entity(CsvType, SurrealType, SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Order { # pub is_deleted: bool, # } # # impl Order { # pub fn csv_table(csv: Csv) -> Table { # Table::new("order", csv) # .with_column_of::("client_id") # .with_column_of::("is_deleted") # } # pub fn surreal_table(db: SurrealDB) -> Table { # Table::new("order", db) # .with_id_column("id") # .with_column_of::("client") # .with_column_of::("is_deleted") # } # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client_order", db) # .with_id_column("id") # .with_column_of::("client_id") # .with_column_of::("is_deleted") # } # } # # #[entity(CsvType, SurrealType, SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Client { # pub name: String, # pub email: String, # pub is_paying_client: bool, # } # impl Client { pub fn csv_table(csv: Csv) -> Table { Table::new("client", csv) .with_column_of::("name") .with_column_of::("email") .with_column_of::("is_paying_client") .with_column_of::("bakery_id") .with_one("bakery", "bakery_id", Bakery::csv_table) .with_many("orders", "client_id", Order::csv_table) } pub fn surreal_table(db: SurrealDB) -> Table { Table::new("client", db) .with_id_column("id") .with_column_of::("name") .with_column_of::("email") .with_column_of::("is_paying_client") .with_one("bakery", "bakery", Bakery::surreal_table) .with_many("orders", "client", Order::surreal_table) .with_expression("order_count", |t| { let orders = t.get_subquery_as::("orders").unwrap(); orders.get_count_query() }) } pub fn sqlite_table(db: SqliteDB) -> Table { Table::new("client", db) .with_id_column("id") .with_column_of::("name") .with_column_of::("email") .with_column_of::("is_paying_client") .with_column_of::("bakery_id") .with_one("bakery", "bakery_id", Bakery::sqlite_table) .with_many("orders", "client_id", Order::sqlite_table) } } ``` Notice how the shape is consistent but details differ — SurrealDB uses embedded document references (`"bakery"` not `"bakery_id"`), and in this excerpt only SurrealDB gets the computed `order_count` expression field. The SQL backends support it too (the real `client.rs` declares it on SQLite and Postgres); CSV can't, because it has no correlated subqueries to lower it to. --- ## Relationships Relationships are declared on the table, not the entity. Two methods: - **`with_one("name", "fk_field", constructor)`** — foreign key to parent (many-to-one) - **`with_many("name", "fk_field", constructor)`** — parent to children (one-to-many) Traversal is synchronous and returns a new `Table` with conditions applied. Reading the result needs a populated database, so this block is compiled but not run: ```rust,no_run # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Client { # pub name: String, # pub is_paying_client: bool, # } # # impl Client { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("is_paying_client") # .with_many("orders", "client_id", Order::sqlite_table) # } # } # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Order { # pub client_id: Option, # pub is_deleted: bool, # } # # impl Order { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client_order", db) # .with_id_column("id") # .with_column_of::("client_id") # .with_column_of::("is_deleted") # } # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:bakery.sqlite").await.context("connect")?; let mut paying = Client::sqlite_table(db); paying.add_condition(paying["is_paying_client"].eq(true)); // Traverse — returns Table with a subquery condition let orders = paying.get_ref_as::("orders")?; // The generated query filters orders by paying clients automatically for (id, order) in orders.list().await? { println!("{id}: client {:?}", order.client_id); } # Ok(()) # } ``` --- ## Computed fields `with_expression` adds fields that don't exist in the database — they're computed via correlated subqueries: ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Order { # pub client_id: Option, # } # # impl Order { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client_order", db) # .with_id_column("id") # .with_column_of::("client_id") # } # } # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Client { # pub name: String, # } # # pub fn client_table(db: SqliteDB) -> Table { # Table::new("client", db) # .with_id_column("id") # .with_column_of::("name") # .with_many("orders", "client_id", Order::sqlite_table) .with_expression("order_count", |t| { let orders = t.get_subquery_as::("orders").unwrap(); orders.get_count_query() }) // SELECT *, (SELECT COUNT(*) FROM client_order WHERE client_order.client_id = client.id) // AS order_count FROM client # } ``` These fields appear in `ReadableValueSet` results alongside physical columns. --- ## Implicit references — dotted columns Surfacing a related row's field is common enough that it has a declarative form. A **dotted name** in `with_active_columns` traverses declared `has_one` relations and imports the target's field as a read-only column, aliased under the literal dotted name — no hand-written expression: ```rust # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Bakery { # pub name: String, # } # # impl Bakery { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("bakery", db) # .with_id_column("id") # .with_column_of::("name") # } # } # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Client { # pub name: String, # } # # impl Client { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("bakery_id") # .with_one("bakery", "bakery_id", Bakery::sqlite_table) # } # } # # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # pub struct Order { # pub client_id: Option, # } # # impl Order { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client_order", db) # .with_id_column("id") # .with_column_of::("client_id") # .with_one("client", "client_id", Client::sqlite_table) # } # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite::memory:").await.context("connect")?; let orders = Order::sqlite_table(db) .with_active_columns(&["id", "client.name", "client.bakery.name"])?; // SELECT id, // (SELECT name FROM client WHERE client.id = client_order.client_id) AS "client.name", // (SELECT (SELECT name FROM bakery WHERE bakery.id = client.bakery_id) // FROM client WHERE client.id = client_order.client_id) AS "client.bakery.name" // FROM client_order # let _ = orders; # Ok(()) # } ``` `client.name` is one hop; `client.bakery.name` recurses through two. Each backend lowers the traversal into its own query — SQL nests correlated scalar subqueries, SurrealDB emits a native idiom path (`client.name`, each segment escaped separately), and MongoDB adds `$lookup` stages. The row comes back with a **flat key** equal to the dotted name (`row.get("client.bakery.name")`). A plain (non-dotted) entry simply restricts projection to that column — `with_active_columns` is also how you narrow a wide table to the columns you actually read. Everything is validated when the table is built, so mistakes surface immediately rather than at fetch time: - an unknown column or relation is a **build-time error**; - a `has_many` hop is rejected (traversal is `has_one`-only — a to-many field is a set, not a value); - a backend that can lower neither a subquery, an idiom path nor a join (e.g. CSV, REST) refuses dotted names up front; same-datasource only. Imported columns are read-only: they are flagged `calculated` for consumers, stripped from full-record write payloads (insert, replace, generated-id insert) so a read-modify-save round-trip never tries to persist `client.name` as a real field, and rejected outright in a `patch` — a partial payload naming a read-only column is explicit intent, and silently dropping it would turn the patch into a successful no-op. This is the declarative counterpart to the manual `with_expression` + `get_subquery_as` recipe above — reach for `with_expression` when you need an arbitrary expression, and a dotted column when you just want a related field. The full treatment — traversal forms, validation, write semantics, per-backend lowering, and how relations surface on Vistas and Dio — is the [Relations & Traversal](relations.md) guide. --- ## Connection management The model crate owns database connections. `bakery_model3` keeps its SurrealDB handle in a `OnceLock`, set once by a DSN-based connection function. The same pattern for SQLite, which needs a database file, so the block is compiled but not run: ```rust,no_run # use std::sync::OnceLock; # # use vantage_core::{Context, Result, error}; # use vantage_sql::sqlite::SqliteDB; # static SQLITE: OnceLock = OnceLock::new(); pub async fn connect_sqlite() -> Result<()> { let dsn = std::env::var("SQLITE_URL").unwrap_or_else(|_| "sqlite:bakery.sqlite".into()); let db = SqliteDB::connect(&dsn).await.context("Failed to connect to SQLite")?; SQLITE.set(db).map_err(|_| error!("SQLite connection already set")) } pub fn sqlite() -> SqliteDB { SQLITE.get().expect("call connect_sqlite() first").clone() } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> Result<()> { # connect_sqlite().await?; # let _ = sqlite(); # Ok(()) # } ``` Consumers call `connect_sqlite()` once at startup, then use `Client::sqlite_table(sqlite())` anywhere. --- ## Using the model Once the model crate exists, consumers are simple. They don't know or care about SQL, SurrealQL, or BSON — they work with entities and tables. The handler and the CLI read a database, so the block is compiled but not run: ```rust,no_run # use std::sync::OnceLock; # # use axum::Json; # use serde::Serialize; # use vantage_memory::{AnyMemoryType, MemoryDB}; # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(SqliteType, MemoryType)] # #[derive(Debug, Clone, PartialEq, Default, Serialize)] # pub struct Client { # pub name: String, # pub is_paying_client: bool, # } # # impl Client { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("client", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("is_paying_client") # } # # pub fn memory_table(db: MemoryDB) -> Table { # Table::new("client", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("is_paying_client") # } # } # # static SQLITE: OnceLock = OnceLock::new(); # # fn sqlite() -> SqliteDB { # SQLITE.get().expect("connected").clone() # } # // In an Axum handler async fn list_clients() -> Json> { let mut clients = Client::sqlite_table(sqlite()); clients.add_condition(clients["is_paying_client"].eq(true)); Json(clients.list().await.unwrap().into_values().collect()) } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let _ = list_clients; // In a CLI tool let table = Client::sqlite_table(sqlite()); println!("{} clients", table.get_count().await?); // In a test — no database needed let table = Client::memory_table(MemoryDB::new()); table.insert_return_id(&Client { name: "Marty".into(), is_paying_client: true }).await?; assert_eq!(table.list().await?.len(), 1); # Ok(()) # } ``` The test reaches for `vantage-memory`, an in-process store with a typed `TableSource`. The model gives it one more constructor and one more type system in `#[entity(..)]`; nothing else about `Client` changes. ```admonish info title="Multi-persistence models" The same model crate can expose table constructors for multiple persistences. Your production code uses Postgres, your CLI reads CSV exports, your tests use mocks — all sharing the same entity definitions and business rules. ``` --- ## Type-erased access For truly generic code (UI grids, admin panels, config-driven tools), wrap tables in a [`Vista`](./new-persistence/step8-vista-integration.md) — the schema-bearing handle that replaced the removed `AnyTable` in 0.5. Each driver's factory wraps its own tables, and every result is the same type. Listing reads real data, so the block is compiled but not run: ```rust,no_run # use vantage_csv::{AnyCsvType, Csv}; # use vantage_dataset::ReadableValueSet; # use vantage_sql::sqlite::{AnySqliteType, SqliteDB}; # use vantage_table::table::Table; # use vantage_types::entity; # use vantage_core::Context; # # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # pub struct Product { # pub name: String, # } # # impl Product { # pub fn sqlite_table(db: SqliteDB) -> Table { # Table::new("product", db).with_id_column("id").with_column_of::("name") # } # } # # #[entity(CsvType)] # #[derive(Debug, Clone, Default)] # pub struct Order { # pub client_id: String, # } # # impl Order { # pub fn csv_table(csv: Csv) -> Table { # Table::new("order", csv).with_column_of::("client_id") # } # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let sqlite = SqliteDB::connect("sqlite:bakery.sqlite").await.context("connect")?; # let csv = Csv::new("bakery_model3/data"); let vistas = vec![ sqlite.vista_factory().from_table(Product::sqlite_table(sqlite.clone()))?, csv.vista_factory().from_table(Order::csv_table(csv.clone()))?, ]; // Same code handles both — different backends, same interface for v in &vistas { println!("{}: {} records", v.name(), v.list_values().await?.len()); } # Ok(()) # } ``` See also: `bakery_model3/examples/cli-vista.rs` for a complete multi-source CLI built on this pattern. --- ## The layered architecture Putting it all together, Vantage prescribes four layers: ```text ┌─────────────────────────────────────────────────┐ │ 4. Consumers │ │ Axum API · CLI · egui · Tauri · gRPC │ ├─────────────────────────────────────────────────┤ │ 3. Business Logic │ │ Traits on Table · custom methods · rules │ ├─────────────────────────────────────────────────┤ │ 2. Model Crate │ │ Entities · Table constructors · Relations │ ├─────────────────────────────────────────────────┤ │ 1. Persistence │ │ SurrealDB · Postgres · SQLite · CSV · API │ └─────────────────────────────────────────────────┘ ``` Layer 1 is implemented once per database (or use an existing Vantage crate). Layer 2 is your model crate — entity definitions, relationships, computed fields. Layer 3 adds business-specific traits and methods on top of `Table`. Layer 4 is any number of consumers that import the model and don't think about persistence. ```admonish success title="Why this works" The model is the only place that knows about database structure. Business logic works against abstract `Table` and `DataSet` interfaces. Consumers work against business logic traits. Change the database — only layer 1 and 2 change. Add a new UI — only layer 4 changes. The architecture scales to hundreds of entities and dozens of developers without coupling. ``` --- Source: https://vantage-ui.com/book/three-paths.html # Three Paths for Developers Vantage gives you three ways to work with data. Each path builds on the previous one, and all three converge into a unified data abstraction layer. --- ```mermaid graph TD EF["① Entity Framework
CRUD over business objects"] QB["② Query Building
Vendor-specific SQL power"] CP["③ Custom Persistences
APIs, caches, reactive data"] EF --> DAL["Universal Data Abstraction Layer"] QB --> DAL CP --> DAL DAL --> E["Export: FFI · API · UI · SDK"] style EF fill:#4a7c59,color:#fff style QB fill:#2d6a8f,color:#fff style CP fill:#8f5a2d,color:#fff style DAL fill:#7c2d8f,color:#fff style E fill:#555,color:#fff ``` --- ## ① Entity Framework The most common path. Define entities, build tables, use `DataSet` and `ActiveEntity` for CRUD — your code never touches SQL or any query language. The block reads and writes a populated database, so it is compiled but not run: ```rust,no_run # use vantage_dataset::prelude::ActiveEntitySet; # use vantage_sql::postgres::AnyPostgresType; # use vantage_sql::prelude::*; # use vantage_surrealdb::types::AnySurrealType; # use vantage_types::prelude::*; # // Define once #[entity(SurrealType, SqliteType, PostgresType)] #[derive(Debug, Clone, Default)] struct Product { name: String, price: i64, is_deleted: bool, } # # impl Product { # fn sqlite_table(db: SqliteDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # .with_column_of::("is_deleted") # } # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:bakery.sqlite").await.context("connect")?; # let id = "flux_cupcake"; // Use everywhere let table = Product::sqlite_table(db); let expensive = table.clone().with_condition(table["price"].gt(200i64)); let count = expensive.get_count().await?; let mut item = table.get_entity(id).await?.unwrap(); item.price = 350; item.save().await?; # let _ = count; # Ok(()) # } ``` This is the bread and butter of enterprise development — hundreds of entities, each with relationships, computed fields, and business rules, all persistence-agnostic and testable with mocks. ```admonish tip title="When to use this path" You're building business software with well-defined entities (Client, Order, Invoice). You want clean separation between persistence and logic. You don't need vendor-specific SQL features. ``` --- ## ② Query Building When you need the full power of your database — JOINs, CTEs, window functions, JSONB operators, array aggregation, recursive queries — drop into the vendor-specific query builder. ### SQLite — CASE, UNION, window functions ```rust # use vantage_expressions::{Expressive, Order, Selectable}; # use vantage_sql::primitives::alias::AliasExt; # use vantage_sql::sqlite::sqlite_ident as ident; # use vantage_sql::sqlite::statements::SqliteSelect; # use vantage_sql::sqlite_expr; use vantage_sql::primitives::case::Case; let select = SqliteSelect::new() .with_source("users") .with_field("name") .with_field("salary") .with_expression( Case::new() .when(sqlite_expr!("{} >= {}", (ident("salary")), 100000.0f64), sqlite_expr!("{}", "senior")) .when(sqlite_expr!("{} >= {}", (ident("salary")), 60000.0f64), sqlite_expr!("{}", "mid")) .else_(sqlite_expr!("{}", "junior")) .as_alias("band"), ) .with_order(ident("salary"), Order::Desc); # assert!(select.preview().starts_with(r#"SELECT "name", "salary", CASE WHEN"#)); ``` ### PostgreSQL — DISTINCT ON, LATERAL JOIN, array ops ```rust # use vantage_expressions::{Expressive, Order, Selectable}; # use vantage_sql::postgres::operation::PostgresOperation; # use vantage_sql::postgres::pg_ident as ident; # use vantage_sql::postgres::statements::PostgresSelect; # use vantage_sql::postgres::statements::select::join::PostgresSelectJoin; # use vantage_sql::primitives::alias::AliasExt; let select = PostgresSelect::new() .with_distinct_on(ident("user_id").dot_of("o")) .with_source_as("orders", "o") .with_expression(ident("name").dot_of("u")) .with_expression(ident("id").dot_of("o").with_alias("order_id")) .with_expression(ident("total").dot_of("o")) .with_join(PostgresSelectJoin::inner( "users", "u", ident("id").dot_of("u").eq(ident("user_id").dot_of("o")), )) .with_order(ident("user_id").dot_of("o"), Order::Asc) .with_order(ident("created_at").dot_of("o"), Order::Desc); ``` Both examples are type-safe — parameters are bound with the correct CBOR type markers, identifiers are quoted for the target dialect, and results deserialize into typed structs. See the full test suites: [SQLite complex queries](https://github.com/romaninsh/vantage/blob/main/vantage-sql/tests/sqlite/3_complex_queries.rs), [PostgreSQL complex queries](https://github.com/romaninsh/vantage/blob/main/vantage-sql/tests/postgres/3_complex_queries_pg.rs). ```admonish info title="Bridge back into the entity framework" Complex queries don't have to live in isolation. Wrap them as expression fields (`with_expression`) or as custom methods on your `Table` — they become part of the entity framework, available to every consumer of your model crate. ``` ```admonish tip title="When to use this path" You need vendor-specific features — recursive CTEs, window functions, `DISTINCT ON`, JSONB operators, `generate_series`, array aggregation. You want full control over the query while keeping type safety and parameterised binding. ``` --- ## ③ Custom Persistences Vantage isn't limited to databases. Implement the persistence traits for anything that stores or produces data: - **REST APIs** — `vantage-api-client` wraps any paginated JSON API as a read-only `TableSource` - **Resilient transport** — `vantage-api-pool` adds retry, a circuit breaker, rate limiting and auth refresh to API traffic - **CSV files** — `vantage-csv` reads structured files with in-memory conditions - **Local caches** — `vantage-diorama` keeps a local copy (in memory or in a redb file) of any slow backend (Postgres, SurrealDB, a REST API) for responsive UIs - **Message queues** — implement `InsertableDataSet` for append-only sources like Kafka topics - **Mixed sources** — read from SQL, write through a queue, cache in memory — same `Table` interface A cache sits above the backend rather than replacing it. Wrap the permanent table as a `Vista`, put a Dio in front of it, and the Dio hands back the cached copy as another `Vista`. The cache is filled from a database, so the block is compiled but not run: ```rust,no_run # use vantage_dataset::ReadableValueSet; # use vantage_diorama::{Cache, Lens, Loader}; # use vantage_sql::prelude::*; # use vantage_types::prelude::*; # # #[entity(SqliteType)] # #[derive(Debug, Clone, Default)] # struct Product { # name: String, # } # # impl Product { # fn sqlite_table(db: SqliteDB) -> Table { # Table::new("product", db).with_id_column("id").with_column_of::("name") # } # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:bakery.sqlite").await.context("connect")?; // The permanent backend, type-erased let products = db.vista_factory().from_table(Product::sqlite_table(db.clone()))?; // A local cache in front of it let lens = Lens::builder() .cache(Cache::redb("cache.redb")) .loader(Loader::eager(|cx| async move { let rows = cx.master().list_values().await?; cx.reconcile(rows).await })) .build()?; let cached = lens.dio(products).build().await?; // Reads come from the cache; your UI code doesn't care which it holds let rows = cached.cached().list_values().await?; # let _ = rows; # Ok(()) # } ``` ```admonish tip title="When to use this path" You're integrating with external APIs, building offline-first applications, implementing reactive data layers, or combining multiple data sources behind a single interface. Your custom persistence becomes a first-class citizen in the entity framework. ``` --- ## Convergence All three paths converge into the same **data abstraction layer**. Entity framework tables, raw query results, and custom persistence data all flow through `Table`, `DataSet`, and `Vista` — giving you one interface for the entire organisation's data. ```text ┌──────────────────────────────────────────────────────────────┐ │ Data Abstraction Layer │ │ │ │ Table ← entity framework │ │ Table ← complex queries │ │ Table ← custom persistence │ │ Dio over any Vista ← cached + reactive │ │ │ │ Tables implement: DataSet · ValueSet · ActiveEntitySet │ │ All wrap into: Vista (type-erased, generic code) │ └──────────────────────────────────────────────────────────────┘ │ ┌───────────────┼───────────────┐ ▼ ▼ ▼ Axum API egui/Tauri FFI / SDK ``` This is where Rust's type system pays off at scale. Every `Table` is checked at compile time. Every persistence has strict type boundaries. Every consumer works against abstract traits. Add a new database, API, or cache — nothing else changes. Add a new UI framework or language binding — the model stays the same. ```admonish success title="The architecture that scales" Vantage turns your data layer into a universal abstraction. Start with one database and a handful of entities. Over time, add more persistences, more entities, more consumers. The architecture doesn't bend — Rust's ownership model and trait system ensure each layer stays clean, each boundary stays enforced, and each team works independently. ``` --- Source: https://vantage-ui.com/book/config-driven-vistas.html # Config-Driven Vistas: YAML & Rhai The [Vista chapter](./intro/step4-vista.md) built Vistas by wrapping typed tables — Rust code, compiled in. This page covers the other construction path: **declare the Vista in YAML, load it at runtime, and seal it behind the same honest handle**. Change the YAML, rebuild the Vista, and the consumer sees the new shape — no recompiling. This is the path configuration-driven tooling takes: admin panels reading a folder of model files, AI agents writing schema on a user's behalf, and any application (Vantage UI among them) whose data layer is user-editable data rather than user-compiled code. For everything YAML can't express — vendor expressions, derived queries, scripted traversal — there's an optional **Rhai** layer that compiles to native queries. --- ## One page of YAML, one working Vista Every driver's factory implements [`VistaFactory`](vantage_vista::VistaFactory), whose `from_yaml` parses a spec and lowers it. `from_yaml` is a trait method, so the trait has to be in scope; it is imported by name from `vantage_vista`, not through a prelude. The block reads a populated `product` table, so it is compiled but not run: ```rust,no_run # use vantage_dataset::ReadableValueSet; # use vantage_sql::prelude::*; use vantage_vista::VistaFactory; # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite:products.db").await.context("connect")?; let yaml = r#" name: product_view columns: id: type: string flags: [id] name: type: string flags: [title, searchable] price: type: int sqlite: table: product "#; let vista = db.vista_factory().from_yaml(yaml)?; assert_eq!(vista.name(), "product_view"); assert_eq!(vista.id_column(), Some("id")); let rows = vista.list_values().await?; // reads the `product` table # let _ = rows; # Ok(()) # } ``` The factory builds a typed `Table` under the hood — each `type:` becomes a real `Column` — and wraps it exactly as `from_table` would. From here on, nothing downstream can tell how the Vista was made: same schema introspection, same conditions, same capability contract. Two implicit rules worth knowing: - **The Vista's name is the spec's `name`**, not the storage name. `product_view` is what a UI tab or a catalog key sees; `sqlite: { table: product }` is where the rows live. Omit the block and the spec name doubles as the table name. - **The id column resolves in a fixed order**: explicit `id_column:` at the top level wins, then the first column flagged `id`, then a driver default (`"id"` for SQL, `"_id"` for MongoDB). The explicit key exists so a YAML author can correct a wrong flag without touching anything else. --- ## Anatomy of a spec A [`VistaSpec`](vantage_vista::VistaSpec) has a uniform core that every driver understands, plus one driver-named block at each level: | Key | What it is | | ------------ | ------------------------------------------------------------------- | | `name` | The vista's public name (catalog key, UI label) | | `datasource` | Optional datasource key, for inventories that manage several | | `id_column` | Explicit id override (see resolution order above) | | `columns` | Ordered map of column name → `{ type, flags, references, lazy, }`| | `references` | Named relations to other vistas (see below) | | `contained` | Embedded-in-row relations (see below) | | `` | The driver's table-level block: `sqlite:`, `surreal:`, `mongo:`, `csv:` | ### Column types `type:` names lower to real typed columns; unknown names are a **parse error**, not a fallback: | YAML `type` | Rust column | | -------------------------------- | --------------------- | | `int`, `integer`, `i64`, `i32` | `Column` | | `float`, `double`, `f64`, `f32` | `Column` | | `bool`, `boolean` | `Column` | | `string`, `text`, `str` (default)| `Column` | | `decimal`, `numeric` | `Column` | | `date` / `time` / `datetime` | chrono naive types | | `timestamp` | `DateTime` | `flags:` is the same open vocabulary the Vista chapter introduced — `id`, `title`, `searchable`, `orderable`, `hidden`, `mandatory` — and drives the same consumer behaviour (title columns label rows, `searchable` feeds quicksearch, `hidden` drops out of default views). ### Typos fail at parse time Every driver block sets `deny_unknown_fields`, so `sqlite: { tabel: product }` is rejected when the YAML is parsed — with the offending key in the message — instead of silently falling back to a default and failing later at query time. Driver-block validation follows the same rule: an empty `nested_path`, a `a..b` path, mutually-exclusive options — all surface as parse errors naming the column. ### Column mapping per driver When the spec column name differs from the storage name, the column's driver block maps it: | Driver | Column block | Meaning | | ------- | -------------------------------- | -------------------------------------------------------- | | SQLite | `sqlite: { column: unit_price }` | SELECTed under the SQL name, aliased back to the spec name | | Surreal | `surreal: { field: unitPrice }` | Read/written under the Surreal field name | | MongoDB | `mongo: { field: unitPrice }` | Single-level BSON rename | | MongoDB | `mongo: { nested_path: price.amount }` | Dotted path into a nested document — projected on read, sub-documents rebuilt on write, dotted form pushed down on filter | | CSV | `csv: { source: "Unit Price" }` | CSV header to read from | | GraphQL | `graphql: { field: unit_price }` | Selected as a GraphQL alias, filtered and sorted under the server name ([GraphQL blocks](./http-apis/graphql-yaml.md)) | MongoDB adds the column types `object_id`, `datetime` (aliases `date`, `timestamp`; a BSON DateTime) and `decimal` (Decimal128). Writes keep these types, and text that doesn't parse as the column's type is rejected. A column named with a dotted path through a reference (`client.name`) joins via `$lookup`, which compares types strictly: a string foreign key does not join an `ObjectId` `_id`. A MongoDB table block can replace the collection read with an aggregation pipeline, or script it: | Table block | Meaning | | -------------------- | -------------------------------------------------------------------- | | `mongo: { pipeline: [...] }` | Read-only view over an aggregation pipeline (`MongoView`) | | `mongo: { rhai: "..." }` | Builds the pipeline in Rhai from `args` (`rhai` feature) | | `mongo: { modify: "..." }` | Rhai run last against the built vista as `self` | | `mongo: { view: true }` | The collection is a server-side view: read-only, not watched, not imported | Change streams (`Capability::Subscribe`) and atomic imports (`Capability::Import`) need a replica set; see [Live data](./live-data.md). Streams run on a dedicated client sized by `MongoDB::with_stream_pool_size` (default 100). CSV is also the one driver whose *table* block is mandatory — `csv: { path: data/products.csv }` — because without a path there is no file. --- ## References A relation to another vista, in the same uniform vocabulary the typed `with_one`/`with_many` uses: ```yaml name: category columns: id: { type: string, flags: [id] } name: { type: string, flags: [title] } references: products: table: product kind: has_many foreign_key: category_id ``` Each entry lowers to one [`Reference`](vantage_vista::Reference) in the vista's metadata. Three shapes are available: - **Full form** (above): `table`, `kind` (`has_one` / `has_many`), `foreign_key`. A `has_one` matches its `foreign_key` on this row to the target's id; a `has_many` matches this row's id to the target's `foreign_key`. Without `foreign_key:`, the entry's name is the key. - **Column sugar** — when the foreign key *is* the column, declare it inline. The relation takes the column's name, and a `references:` entry of the same name takes precedence: ```yaml columns: category_id: type: string references: category # has_one named `category_id`, foreign_key = the column itself ``` - **Multi-key joins** — `keys:` lists `{ from, to }` pairs when the child is narrowed by more than one parent field (a deployment matched on both `product_id` and `version_id`). Each pair reads the parent row's `from` column and constrains the target's `to` column, and traversal applies every pair. When `keys:` is present it is the whole join, and `foreign_key:` is not read: ```yaml references: deployments: table: deployment kind: has_many keys: - { from: product_id, to: product } - { from: version_id, to: version } ``` On MongoDB a reference key must be stored under its own field name. The vista fails to build when a reference's foreign key, or the id column it matches on the other side, is renamed with `field:` or moved with `nested_path:`. A reference with a `mongo: { rhai: }` script (below) is exempt, as its traversal runs the script. Joined columns (`client.city`) can still read nested fields of the target. ### The spec resolver A reference names its target *by spec name* — so the factory needs a way to find the target's current spec at traversal time. That's the **spec resolver**, attached once: ```rust # use std::collections::HashMap; # use std::sync::Arc; # # use ciborium::Value as CborValue; # use vantage_sql::prelude::*; # use vantage_sql::sqlite::vista::SqliteVistaSpec; # use vantage_types::Record; # use vantage_vista::{ColumnSpec, SpecResolver, VistaFactory}; # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite::memory:").await.context("connect")?; # let category_yaml = r#" # name: category # columns: # id: { type: string, flags: [id] } # name: { type: string, flags: [title] } # references: # products: # table: product # kind: has_many # foreign_key: category_id # "#; # let mut product = SqliteVistaSpec::new("product"); # product.columns.insert("id".to_string(), ColumnSpec::new().with_type("string").with_flag("id")); # product.columns.insert("category_id".to_string(), ColumnSpec::new().with_type("string")); # let specs: HashMap = HashMap::from([("product".to_string(), product)]); let resolver: Arc> = Arc::new(move |name: &str| specs.get(name).cloned()); let factory = db.vista_factory().with_resolver(resolver); let category = factory.from_yaml(category_yaml)?; // Traversal rebuilds `product` from its live spec: let row: Record = [("id".to_string(), CborValue::Text("pastry".into()))].into_iter().collect(); let products = category.get_ref("products", Some(&row))?; # assert!(products.is_narrowed()); # Ok(()) # } ``` [`SpecResolver`](vantage_vista::SpecResolver) is the same trait for every driver, with the driver's spec type as `S`, and a factory holds it as `Arc>`. Any `Fn(&str) -> Option` closure implements it (annotate its argument as `&str`, as above, or the closure won't take every lifetime), and a type of your own can implement `resolve` directly — back it with an in-memory map, a folder of `.yaml` files, or a hot-reloading inventory. Because targets are re-resolved on every traversal, editing a spec changes what the *next* traversal builds. A name the resolver doesn't know, or a target that fails to build, is a traversal error, never a wider table. Without a resolver at all, a traversal falls back to a column-less target and the next query fails loudly. ### The build injector Some rules belong on every table a factory builds, not on one spec: a tenant filter, an access-control condition, hooks that stamp who wrote a row. Copying them into each spec is how one gets forgotten. A **build injector** runs on every table the factory builds from a spec: the root, each reference target when it is traversed, each `base:` table under a derived vista, and tables built inside scripts. It runs after the table is built and before it becomes a vista, and it can add conditions, register hooks, or fail the build: ```rust # use std::sync::Arc; # # use vantage_sql::prelude::*; # use vantage_table::table::{BuildKind, InjectCx, TableInjector}; # use vantage_table::traits::table_source::TableSource; # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite::memory:").await.context("connect")?; let tenant_only: TableInjector = Arc::new(|cx: &InjectCx<'_, SqliteDB>, table| { if cx.kind == BuildKind::Root && cx.key == "audit_log" { return Err(vantage_core::error!("audit_log is not exposed")); } let tenant = AnySqliteType::new("acme".to_string()); let condition = table.data_source().eq_value_condition("tenant_id", tenant)?; table.add_condition(condition); Ok(()) }); let factory = db.vista_factory().with_injector(tenant_only); # let _ = factory; # Ok(()) # } ``` [`InjectCx`](vantage_table::table::InjectCx) carries the spec key, the [`BuildKind`](vantage_table::table::BuildKind) (`Root`, `Reference { from, relation }` or `Base { derived }`), and `build`, which builds another table by key through the same factory and injector. That is how an injector expresses "orders the caller may see are those of clients the caller may see": it builds `client` and narrows by its ids. A build that would re-enter a key already being built is an error naming the cycle (`a -> b -> a`), not a stack overflow. On a `Base` build only the conditions take effect. The derived vista is a fresh table over the base's `select()`, so the base's conditions carry into it but hooks an injector registers on the base never fire. The injector runs on the derived build too (as `Root` or `Reference`); register hooks there. `BuildKind` and `InjectCx` are `#[non_exhaustive]`, so match `BuildKind` with a wildcard arm. The SQLite, Postgres, MySQL, SurrealDB, MongoDB and CSV factories take `with_injector`. A reference target that fails injection fails its traversal, as above. A vista whose writes go through `Table`, so that the injector's hooks fire on them, advertises `Capability::WritesReachHooks` (`writes_reach_hooks`). --- ## The Rhai layer YAML declares structure. For anything with an expression in it, specs escalate to **Rhai** — a small, embeddable scripting language whose Vantage vocabulary compiles to native queries. The expression primitives are shared across backends where the concept overlaps (`count`, `avg`, `coalesce`, `case_when`, `date_format` …) — see [SQL Primitives](./sql/primitives.md) and [SurrealDB Primitives](./surrealdb/primitives.md) for the full vocabularies. A script calls them through its backend's namespace (`sql::count(..)`, `surreal::count(..)`, `mongo::oid(..)`), so they never shadow the data vocabulary's `table(name)`. Rhai appears in a spec in five places, each with a distinct job. The first two build native queries; the next two use the data vocabulary taught in [Scripting with Rhai](./rhai.md); the last computes a column from one row. ### 1. Query-sourced vistas — `rhai:` Replace the physical table with a script-built SELECT: ```yaml name: expensive_products columns: id: { type: string, flags: [id] } name: { type: string } sqlite: rhai: | sql::select().from("product").field("id").field("name").where(sql::expr("price > 150")) ``` The script runs once at build time and its SELECT becomes the vista's source. A query-sourced vista is **read-only** — the factory leaves `Insert`, `Update`, `Delete` and `ConfineWrites` out of its capabilities, because there's no single table a write could honestly land in. Capability honesty is preserved automatically; consumers find out by checking, same as ever. ### 2. Derived vistas — `base:` + `inherit` + transform Derive one vista from another, inheriting schema and transforming the query. The base's `select()` is seeded into the script's scope as `base`: ```yaml name: category_totals id_column: category_id columns: total_price: { type: int } sqlite: base: product inherit: columns: [category_id] rhai: | base.clear_fields().field("category_id") .expression(sql::expr("SUM(price) AS total_price")) .group_by(sql::expr("category_id")) ``` This is the aggregate pattern: group the base by a key, declare the aggregate outputs as the derived vista's own `columns:`, and re-key with `id_column`. `base:` resolves eagerly through the same spec resolver as references. Derived vistas are query-sourced, hence read-only. ### 3. Post-build tweaks — `modify:` (SurrealDB, MongoDB, MySQL) A script applied to the *finished* vista, exposed as `self`, using the [table handle's verbs](./rhai/tables.md) plus vendor expressions YAML keys can't state (see [`modify:` scripts](./rhai/surfaces.md#modify-scripts)): ```yaml name: active_products columns: id: { type: thing, flags: [id] } name: { type: string, flags: [title] } surreal: table: product modify: | self.with_condition(surreal::ident("is_deleted") == false) .sort("name", "asc") ``` Unlike a query source, `modify:` narrows a real table — so the vista **stays writable**. It runs last, composing with `table:`, `rhai:`, or `base:`. Writability follows the underlying source: a `modify:` on a `rhai:` or `base:` vista stays read-only, while on a plain table it stays writable. MySQL takes the same script in a `mysql:` block (`mysql: { modify: "..." }`), with the `with_condition(expr)` verb, which hands its expression to the shell as a native condition (from Rust: `Vista::add_native_condition`). ### 4. Scripted reference traversal — reference-level `rhai:` (SurrealDB, MongoDB, MySQL) When a relation can't be expressed as a foreign-key equality — a graph edge, a computed join — the reference carries its own build script, evaluated lazily at traversal time with the parent `row` in scope: ```yaml references: products: table: product kind: has_many foreign_key: category surreal: rhai: | table("product").where("category", row.id) ``` The script ends on a table handle, which becomes the narrowed target vista; `foreign_key` remains as metadata for consumers that introspect the relation. See [Reference build scripts](./rhai/surfaces.md#reference-build-scripts). MongoDB takes the same script in the reference's `mongo:` block (`rhai` feature): ```yaml references: orders: table: order kind: has_many foreign_key: client_id mongo: rhai: | table("order").where("client_id", row._id) ``` MySQL takes it in the reference's `mysql:` block (`mysql: { rhai: "..." }`, `rhai` feature). ### 5. Computed columns — column-level `lazy:` (every driver) A column whose value no backend stores, computed by the Vista from the row it read: ```yaml columns: net: { type: int } vat: { type: int, lazy: "row.net / 5" } ``` The script sees the row as `row`, and its last expression is the value. Columns compute in declaration order, writes drop them, and filtering, ordering or aggregating on one is an error naming the column. A has-one reference may use one as its foreign key; a has-many may not. Full details in [Computed Columns](./rhai/computed.md). ```admonish note title="YAML primary, Rhai targeted" The division of labour is deliberate: YAML stays the canonical, declarative format that every backend understands; Rhai is the serializable escape hatch you reach for only when a source, transform, or traversal needs an expression. A spec with no Rhai in it works on a build without the `rhai` feature; one that uses it fails loudly there instead of degrading. ``` --- ## Contained relations Embedded objects and arrays — an order's `lines`, a JSON column — declare as a `contained:` section and surface as editable sub-vistas: ```yaml name: order columns: id: { type: string, flags: [id] } lines: { type: string } # the host column — declare it so it's selected contained: lines: host_column: lines kind: contains_many id_column: line_id # omit for positional ids columns: product: { type: string } quantity: { type: int } ``` Reads project the embedded collection into records; writes patch it back into the host column. The mechanics — and the sharp edges — are covered in [Contained Relations](./new-persistence/step9-contained-relations.md). --- ## Cross-persistence: the Catalog A single Vista is strictly single-backend. When a system spans several — categories in Postgres, products behind a REST API — a [`Catalog`](vantage_vista::Catalog) sits one layer up: it holds a loader per model name, and a relation whose target lives in a *different* persistence is a [`Reference`](vantage_vista::Reference) narrowed over the target the catalog builds. Here two in-memory SQLite databases stand in for the two backends: ```rust # use ciborium::Value as CborValue; # use vantage_sql::prelude::*; # use vantage_types::Record; # use vantage_vista::{Catalog, JoinKey, Reference, VistaFactory, VistaResolver}; # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let pg = SqliteDB::connect("sqlite::memory:").await.context("connect")?; # let api = SqliteDB::connect("sqlite::memory:").await.context("connect")?; # let category_yaml = "name: category\ncolumns:\n id: { type: string, flags: [id] }\n".to_string(); # let product_yaml = # "name: product\ncolumns:\n id: { type: string, flags: [id] }\n category_id: { type: string }\n" # .to_string(); let (pg_factory, api_factory) = (pg.vista_factory(), api.vista_factory()); let catalog = Catalog::new() .with("category", move |_: &str| pg_factory.from_yaml(&category_yaml)) .with("product", move |_: &str| api_factory.from_yaml(&product_yaml)); let relation = Reference::has_many( "products", "product", JoinKey::new("id", "category_id"), // parent `id` constrains the target's `category_id` ); let row: Record = [("id".to_string(), CborValue::Text("pastry".into()))].into_iter().collect(); let products = relation.narrow(catalog.vista(&relation.target)?, &row)?; // one backend → the other # assert!(products.is_narrowed()); # Ok(()) # } ``` Loaders return a *fresh, unconditioned* Vista on every call, so the catalog composes with hot-reloading inventories the same way spec resolvers do. A name the catalog doesn't hold is a not-found error; a loader that fails keeps its own error. The catalog is a [`VistaResolver`](vantage_vista::VistaResolver), so it goes wherever a resolver is expected: it is also what [Augmentation](./augmentation.md) uses to resolve its detail sources — the `augment:` block in a table's configuration is this same machinery pointed at row enrichment. --- ## Sealed at runtime: data scripts Once vistas are config-defined, the last step of the story is *consuming* them from config too. `vantage-vista`'s `rhai` feature ships `run_script`, a sandboxed evaluator where `table(name)` resolves through a catalog-style resolver. It's the surface an AI agent or an MCP tool drives: the schema it sees, the capabilities it must respect, and the rows it reads all come from the same sealed handles this page built. The vocabulary, its limits and every other place it runs are covered in [Scripting with Rhai](./rhai.md). --- ## Choosing a path Both construction paths produce the same `Vista` — the choice is about who edits the definition: - **Typed `from_table`** when the model is code: business logic, compile-time safety, entity structs, `with_expression` closures. This is the path the [introduction guide](./intro/step4-vista.md) walks. - **YAML (+ Rhai)** when the model is data: inventories on disk, user- or agent-edited schema, hot-reload, no recompiling. Structure in YAML; expressions in Rhai; capabilities sealed either way. They mix freely — a catalog can hold typed-table loaders next to YAML loaders, and a YAML spec's reference can resolve to either. If you're implementing this machinery for a new backend, the driver-side walkthrough is [Adding a New Persistence, Step 8](./new-persistence/step8-vista-integration.md). --- Source: https://vantage-ui.com/book/rhai.html # Scripting with Rhai A Vantage application can keep much of its behaviour in configuration: YAML table specs, page definitions, action bodies, faker sims. Some of that configuration needs logic: a column derived from other columns, a filter that depends on the selected row, a write after a form is submitted, a simulated order that ships after twenty minutes. That logic is written in [Rhai](https://rhai.rs), a small scripting language embedded in Rust. Rhai is part of vantage: any application that holds Vistas can run these scripts. Vantage UI is one such application and is used as an example where it helps. This part of the book is the one place that teaches scripting across the framework. Other chapters link here instead of repeating the verbs. --- ## Why scripts Configuration is data. A user, an agent or a hot-reloading inventory can change it while the application runs, and nothing gets recompiled. Rust code can't follow it there, so the logic that travels with the configuration has to be data too: text that the application compiles and runs when it loads the YAML. Rhai fits that job: - **It runs in the process.** There is no interpreter to install and no subprocess. A script calls straight into the same Vistas the Rust code uses. - **It can only do what the host allows.** A Rhai script has no file system, network or process access of its own. It sees the functions the host registered and nothing else. - **It can't hang the caller.** Every host caps the number of operations a script may run. A script that loops forever fails with an error. - **It is backend-neutral.** Data scripts act on [Vistas](./intro/step4-vista.md), so the same script runs over SQLite, SurrealDB, a REST API or an in-memory store. ## What a script can do | Task | Example | Chapter | |---|---|---| | Read a set of rows, count it, follow a relation | `table("client").where("vip", true).ref("orders").count()` | [The table handle](./rhai/tables.md) | | Insert, patch, delete, copy rows between tables | `table("order").patch(id, #{ status: "paid" })` | [Writes](./rhai/writes.md) | | Compute a column from the other columns of a row | `lazy: "row.net + row.vat"` | [Computed columns](./rhai/computed.md) | | Edit one row field by field and write what changed | `row.status = "paid"; row.flash();` | [Records](./rhai/records.md) | | Narrow a table in YAML, or build a relation's target | `self.where("vip", true)` | [Surfaces](./rhai/surfaces.md) | | Build a backend's native query | `sql::select().from("product").field("id")` | [Expression dialects](./rhai/dialects.md) | ## Where scripts run A script always runs inside a **host**: a Rhai engine with fixed operation limits, a compile cache, and the vocabularies the host chose to register. The host decides what `table(name)` resolves to, whether writes are allowed, and how many rows a `list()` may return. The same script text can be legal in one place and refused in another. | Surface | Example | What the script can do | |---|---|---| | Agent data scripts (`run_script`) | an MCP tool reading or fixing data | read, and write if the host allows it | | Query preview (`preview_script`) | an MCP tool showing the query a script would run | describe a set, never read it | | YAML `modify:`, reference build scripts | `self.where("vip", true)` | describe a set | | YAML `lazy:` columns | `row.net / 5` | compute one value from one row | | An application's form `options:` (e.g. Vantage UI) | a dropdown filled from another table | read | | An application's action bodies, form `on_submit`, wizard workers (e.g. Vantage UI) | `row.status = "paid"; row.flash();` | read and write | | Faker sims | an order that ships, then disappears | read and write a memory store, plus time and random verbs | [Hosts](./rhai/hosts.md) explains how a host is put together, and [Surfaces](./rhai/surfaces.md) walks through each row of this table with a working example. ## Rhai in two minutes Rhai reads like a mix of Rust and JavaScript. The parts these chapters use: | Rhai | Meaning | |---|---| | `let x = 5;` | a variable; no type annotations | | `#{ name: "Ada", vip: true }` | an object map; read with `m.name` or `m["name"]` | | `[1, 2, 3]` | an array | | `\|c\| c.name` | a closure, as in `rows.map(\|c\| c.name)` and `rows.filter(\|c\| c.vip)` | | `()` | "nothing": a missing field, a row that wasn't found | | `a ?? b` | `a`, or `b` when `a` is `()` | | `` `total: ${t}` `` | string interpolation | | `if c { a } else { b }` | an expression; it has a value | | `try { … } catch (err) { … }`, `throw "message"` | catching and raising errors | A script's value is its last expression, with no `return` and no trailing `;`. Methods chain, so `table("order").where("status", "due").count()` is three calls on the result of the one before. The [Rhai book](https://rhai.rs/book/) covers the full language. ## A first script Every data script uses the same handful of words. `table(name)` names a table, a chain of narrowing verbs describes a set of its rows, and a terminal verb reads or writes: ```rhai let vips = table("client").where("vip", true); let due = vips.ref("orders").where("status", "due"); #{ vips: vips.count(), due: due.ids(), } ``` `vips` and `due` are **handles**: descriptions of a set. Nothing is read until `count()` or `ids()` runs, and the handle is then resolved into a [`Vista`](vantage_vista::Vista) and read through it. `ref("orders")` follows a relation from every row of the set, so `due` is the due orders of every VIP client. The script's last expression is its result; here a map, which a host can turn into JSON. The script never mentions SQL, SurrealDB or HTTP. The words act on Vistas, and every backend can be wrapped as a Vista. ## The layers Vista is the universal data handle: any backend, one type-erased interface. The data vocabulary lives in `vantage-vista` (behind its `rhai` feature) and knows nothing except Vistas. Three things sit above Vista, and each has its own scripting words: ```text ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ │ Servo │ │ Scenery │ │ faker sims │ │ form drafts │ │ view shapes │ │ time, random, │ │ (diorama) │ │ (vantage-ui) │ │ fake_row (faker) │ └──────┬───────┘ └──────┬───────┘ └────────┬─────────┘ │ │ │ ┌──────┴─────────────────┴───────────────────┴─────────┐ │ Vista data vocabulary: table, where, sort, ref, │ │ list, get, insert, patch, record, ... │ └──────────────────────────────────────────────────────┘ ``` The lower layer never refers to the upper ones. A Dio can hand a script a Vista whose writes go through its write queue, and the script can't tell the difference. Servo and Scenery keep their own words on purpose, because they describe different things: a form draft bound to one record, and the shape of a live view. [Layers](./rhai/layers.md) covers how they relate and which words they share. There is also a second kind of script: **expressions** that build a backend's native query, such as a SQL `select()` or a SurrealDB condition. They aren't data scripts and have their own chapters. [Expression dialects](./rhai/dialects.md) points to them. ## What this part covers - [Hosts](./rhai/hosts.md): `Host`, `Vocab`, `Limits`, the `DataVocab` configuration, the resolver behind `table(name)`, and how async reads run from synchronous scripts. - [The table handle](./rhai/tables.md): narrowing verbs, reads, relations, introspection, and errors. - [Writes](./rhai/writes.md): `insert`, `upsert`, `patch`, `delete` and `import_from`, and what each returns when a row is missing or outside the set. - [Computed columns](./rhai/computed.md): `lazy:` columns that the Vista fills on every read, and what they refuse. - [Records](./rhai/records.md): drafts of one row that stage edits and flash only what changed. - [Surfaces](./rhai/surfaces.md): every place the vocabulary runs, with a tested example for each, plus backend extensions such as SurrealDB's `with_condition`. - [Layers](./rhai/layers.md): Servo, Scenery and faker sims next to the data vocabulary. - [Faker sims](./rhai/faker.md): scripts that keep a memory store changing over sim time. - [Expression dialects](./rhai/dialects.md): query builders, templates and command scripts. - [Tables defined by a query](./rhai/query-tables.md): a table whose source is a script-built `SELECT`, and why that is the only place a script builds a query. - [Query builder functions](./rhai/query-builders.md): every function the SQL and SurrealDB query builders register. ```admonish note title="Tested examples" Every script in this part is copied from a test. The comment above a snippet names the test, for example `rhai_guide::tables::first_script` in `vantage-vista/tests/rhai_guide/`. Run them with `cargo nextest run -p vantage-vista --all-features -E 'binary(rhai_guide)'`. ``` ## Words at a glance | Group | Words | |---|---| | Start | `table(name)`; `self` in slot scripts; `table()` in faker sims | | Narrow | `where(col, value)`, `where(col, op, value)`, `sort(col)`, `sort(col, dir)`, `search(text)`, `limit(n)`, `offset(n)`, `ref(relation)` | | Read | `list()`, `get(id)`, `first()`, `count()`, `ids()`, `columns()`, `references()`, `capabilities()` (keyed by every `Capability`) | | Write | `insert(map)`, `upsert(id, map)`, `patch(id, map)`, `delete(id)`, `import_from(source)`, `import_from(source, mapping)` | | Record | `record()`, `record(id)`; on a record: `r.col`, `r["col"]`, `set(map)`, `id`, `is_dirty()`, `dirty(col)`, `baseline()`, `revert()`, `revert(col)`, `flash()`, `delete()`, `status()`, `rejection()` | | Computed column | `row` (the record as read); the last expression is the value | `where` takes the operators `=` / `==` / `eq`, `!=` / `ne`, `>` / `gt`, `>=` / `gte`, `<` / `lt`, `<=` / `lte`, `in`, `not_in` and `like`, plus a few aliases (`<>`, `nin`, `contains`). `sort` takes `"asc"` or `"desc"` (or `"ascending"` / `"descending"`). --- Source: https://vantage-ui.com/book/rhai/hosts.html # Hosts A script never runs on a bare Rhai engine. It runs in a [`Host`](vantage_rhai::Host) from `vantage-rhai`: one engine with fixed limits, a bounded cache of compiled scripts, and the vocabularies registered on it. This chapter is for the Rust side: how an application builds a host for the data vocabulary and what each setting decides. --- ## Host, Vocab, Env A host is built once and reused. Compiling through it is cached, so evaluating the same script on every row costs one parse: ```rust # use std::sync::Arc; # use ciborium::Value as CborValue; # use vantage_memory::{MemoryCatalog, MemoryStore}; # use vantage_rhai::{Block, Env, Host, Limits}; # use vantage_vista::rhai::{DataVocab, Terminals, Writes}; # use vantage_vista::{Column, ColumnFlag, VistaMetadata, VistaResolver}; # fn shop() -> MemoryStore { # let store = MemoryStore::new(); # for (id, status) in [("o1", "paid"), ("o2", "due"), ("o3", "paid"), ("o4", "due")] { # let row = [("status".to_string(), CborValue::Text(status.into()))] # .into_iter() # .collect(); # store.table("order").insert_as(id, row).unwrap(); # } # store # } # fn resolver(store: &MemoryStore) -> Arc { # let catalog = MemoryCatalog::new(store.clone()); # catalog.register( # "order", # VistaMetadata::new() # .with_column(Column::new("id", "String").with_flag(ColumnFlag::Id)) # .with_column(Column::new("status", "String")) # .with_id_column("id"), # ); # Arc::new(catalog) # } # let store = shop(); let terminals = Terminals::ReadWrite { limit: Some(50), writes: Writes::Allowed, }; let host = Host::builder(Limits::background()) .vocab(DataVocab::new(terminals).with_resolver(resolver(&store))) .build(); let script = host .compile(&Block::from( r#"table("order").where("status", "due").count()"#, )) .unwrap(); let due = script.eval(&Env::new()).unwrap(); # assert_eq!(due.as_int().unwrap(), 2); ``` The three pieces: - **[`Vocab`](vantage_rhai::Vocab)**: anything that registers functions and types on an engine. `DataVocab` is the data vocabulary. `ServoVocab` (diorama) and `SceneryVocab` (vantage-ui) are others. A host can carry several. When two register the same function under the same name and argument types, the one registered last wins. The backends keep their free functions in their own namespaces (`sql::`, `surreal::`, `mongo::`), so none of them can replace `table(name)` or another data verb, whatever the order. - **[`Limits`](vantage_rhai::Limits)**: `Limits::Ui` caps a script at 500,000 operations, for anything the UI thread waits on. `Limits::background()` allows 50 million, for scripts on worker threads: agent tools, imports, faker sims. A script that runs past its limit fails instead of hanging the caller. - **[`Env`](vantage_rhai::Env)**: the variables one evaluation sees. Slot evaluators push `row` and `self` this way. A host builds its `Env` per call, so the compiled script is shared and the variables are not. ## `DataVocab` ```rust # use std::sync::Arc; # use vantage_memory::{MemoryCatalog, MemoryStore}; # use vantage_vista::VistaResolver; # use vantage_vista::rhai::{DataVocab, Terminals}; # let terminals = Terminals::Describe; # let resolver: Arc = Arc::new(MemoryCatalog::new(MemoryStore::new())); let vocab = DataVocab::new(terminals).with_resolver(resolver); ``` [`DataVocab::new(terminals)`](vantage_vista::rhai::DataVocab::new) picks which verbs exist on top of the narrowing verbs. `with_resolver` backs `table(name)`. Without a resolver, `table` isn't registered at all, and scripts can only work with handles the host gives them (`self`, `row`). | `Terminals` | Verbs | Used by | |---|---|---| | `Describe` | none: the script returns a handle and the host reads its description | `modify:`, reference build scripts, query preview | | `ReadWrite { limit, writes }` | reads, writes, and records; under `Writes::Denied` every write throws | agent scripts, action bodies, form and wizard scripts, faker sims; with writes denied, form `options:` and action predicates | `limit` caps every `list()` and `ids()`, counting after any `offset(n)`. A script's own `limit(n)` still applies below it. `None` means no cap. `count()` isn't capped by the host's limit: it counts the whole set, or the rows the script's `limit(n)` and `offset(n)` leave. `Terminals::ReadWrite { limit: None, writes: Writes::Allowed }` is uncapped with writes allowed. `writes` is `Writes::Allowed` or `Writes::Denied(message)`. A denied host still registers every write verb, but each one throws `message`. That keeps the error readable: a script that calls `delete` on a host with writes turned off gets the host's explanation ("writes are off for agents") instead of Rhai's "function not found". A `Describe` host, by contrast, has no `count()` or `list()`, and calling one is a "function not found" error. ## The resolver The resolver is an `Arc`: anything with `fn vista(&self, name) -> Result`, which includes a closure `Fn(&str) -> Result` and a [`Catalog`](vantage_vista::Catalog). It is called each time a handle resolves, and must return a **fresh, unconditioned** Vista: the handle applies its own steps on top. A name it doesn't know is a not-found error. This is the resolver the guide's tests use, over an in-memory store: ```rust # use std::sync::Arc; # use vantage_memory::{MemoryCatalog, MemoryStore}; # use vantage_vista::VistaResolver; # fn catalog(store: &MemoryStore) -> MemoryCatalog { # MemoryCatalog::new(store.clone()) # } fn resolver(store: &MemoryStore) -> Arc { let catalog = catalog(store); Arc::new(move |name: &str| { catalog.store().table(name); catalog.vista(name) }) } # let store = MemoryStore::new(); # assert_eq!(resolver(&store).vista("order").unwrap().name(), "order"); ``` `catalog` is a `MemoryCatalog` holding each table's metadata. Creating the store table first means a name the shop doesn't hold gets a new, empty table instead of a not-found error. The resolver is where an application puts its policy, because `vantage-vista` only calls it. Some examples from the codebase: - A YAML catalog resolves through its spec resolver, so `table("order")` in a reference build script is the same Vista the spec declares. - Faker sims resolve over the sim's memory store; the engine's stats count the store's writes. - Vantage UI's page resolver accepts a full catalog key or its leaf name, reads from the data source rather than a cache, and wraps every Vista in a write guard. The guard refuses writes on read-only tables, coerces reference columns, gives an id-less insert a generated id, sends writes through the table's Dio (so they reach open pages), and wraps every `ref(...)` target in the target table's own guard. None of that is visible to the script. It calls `insert`, and the Vista it resolved decides what an insert means. ## Async reads from a synchronous script Rhai is synchronous. Vista reads and writes are `async`. Every terminal verb runs its future through one function, vista's [`block_on`](vantage_vista::rhai::block_on): - With a tokio runtime available (`Handle::try_current()` succeeds), it blocks on the runtime. - Without one, it polls the future once on the current thread. Memory Vistas finish on the first poll, so faker sims and tests run with no runtime at all. - A future that is still pending with no runtime (any network backend) is an error, "this data source needs a tokio runtime; run the script under spawn_blocking". It never hangs. Blocking on a runtime from inside an async task panics in tokio, so a host that has a runtime evaluates scripts on a blocking thread: ```rust # use std::sync::Arc; # use ciborium::Value as CborValue; # use vantage_memory::{MemoryCatalog, MemoryStore}; # use vantage_rhai::{Block, Env, Host, Limits}; # use vantage_vista::VistaResolver; # use vantage_vista::rhai::{DataVocab, Terminals, Writes}; # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let store = MemoryStore::new(); # for id in ["o1", "o2", "o3"] { # let row = [("status".to_string(), CborValue::Text("due".into()))] # .into_iter() # .collect(); # store.table("order").insert_as(id, row)?; # } # let resolver: Arc = Arc::new(MemoryCatalog::new(store)); let rows = tokio::task::spawn_blocking(move || { let terminals = Terminals::ReadWrite { limit: Some(2), writes: Writes::Denied("this host only reads".into()), }; let host = Host::builder(Limits::background()) .vocab(DataVocab::new(terminals).with_resolver(resolver)) .build(); host.compile(&Block::from(r#"table("order").list()"#)) .and_then(|s| s.eval(&Env::new())) .map(|v| vantage_rhai::to_json(&v)) }) .await .unwrap() .unwrap(); # assert_eq!(rows.as_array().unwrap().len(), 2); # Ok(()) # } ``` [`run_script`](vantage_vista::rhai::run_script) does exactly this for you. Action bodies in Vantage UI run under `Limits::Ui` on a background executor thread that has a runtime handle, which the bridge also accepts. ## Errors Script errors come back as Rhai errors whose text names what failed: - a narrowing step the backend can't apply (an unknown relation, a column that isn't orderable, a computed column) fails when the handle resolves, at the terminal verb, not where the step was written. The error names the step: `Step can't be applied (step: Ref("invoices")): …`; - a write the Vista's capabilities don't cover is an `unsupported` error naming the capability: "capability is not supported; method refused (capability: delete)"; - backend errors keep their message. Hosts that return JSON, like `run_script`, flatten all of these into the error string. --- Source: https://vantage-ui.com/book/rhai/tables.html # The Table Handle `table(name)` returns a handle: a table plus an ordered list of narrowing steps. Narrowing verbs add steps. Terminal verbs resolve the handle into a Vista, apply the steps, and read or write. --- ## Narrowing ```rhai let paid = table("order").where("status", "paid"); let big = paid.where("total", ">", 100); let top = paid.sort("total", "desc").first(); #{ paid: paid.count(), big: big.ids(), top: top.id, } ``` Each narrowing verb returns a **new** handle and leaves the one it was called on alone. `big` and the sorted handle both start from `paid`, and `paid.count()` still counts every paid order. You can keep a base handle in a variable and branch from it as often as you like. | Verb | Meaning | |---|---| | `where(col, value)` | rows whose `col` equals `value` | | `where(col, op, value)` | `op` is `=`/`==`/`eq`, `!=`/`ne`, `>`/`gt`, `>=`/`gte`, `<`/`lt`, `<=`/`lte`, `in`, `not_in` or `like`; `in` and `not_in` take an array | | `sort(col)`, `sort(col, "asc" \| "desc")` | order by `col`; the column must be flagged orderable | | `search(text)` | the backend's quicksearch | | `limit(n)` | at most `n` rows, the first `n` in sort order; `n` must be above 0 | | `offset(n)` | the rows after the first `n` in sort order, as SQL's `OFFSET`; `n` can't be negative | | `ref(relation)` | the related rows of every row in the set | Conditions add up: every `where` narrows further. Sorting keeps one column. A later `sort` replaces an earlier one, as `Vista::set_order` does. Several `limit` steps keep the smallest, and several `offset` steps add up, so a later step never widens the set. `offset` and `limit` combine as in SQL, whatever order you call them in: `offset(20).limit(10)` and `limit(10).offset(20)` are both rows 20 to 29. The offset is skipped first and the limit counts from there. `limit(n)` and `offset(n)` pick from the set itself, not just one read: `list()`, `ids()` and `count()` all see the same rows, a `ref` after them follows only those rows, and `import_from` copies only those (see [Following relations](#following-relations)). They don't carry over to the target of a `ref`; limit or offset the target again if you need to. A handle with `limit` or `offset` reads; it can't be written through ([Writes](./writes.md#writes-stay-in-the-set)). The column in `where` and `sort` must be one the backend stores. A [computed column](./computed.md) is refused with an error naming it. Nothing is checked while you narrow. A step the resolved Vista can't apply (an unknown relation, a column that isn't orderable, `search` on a backend without search) fails at the terminal verb: ```rhai let invoices = table("client").ref("invoices"); // nothing resolves yet invoices.count() // throws here ``` The error names the step, `Step can't be applied (step: Ref("invoices"))`, and its cause, an `unknown relation` naming `invoices`. An unknown relation is a not-found error. ## Reads ```rhai let clients = table("client"); let ada = clients.get("c1"); let ghost = clients.get("c9"); #{ name: ada.name, missing: ghost == (), first_two: clients.sort("name").limit(2).list().map(|c| c.name), } ``` | Verb | Returns | |---|---| | `list()` | an array of row maps, capped by the host's limit and any `limit(n)`, after any `offset(n)` | | `get(id)` | the row map, or `()` when no row has that id | | `first()` | the first row map in sort order (after `offset(n)`, the row at position `n`), or `()` for an empty set | | `count()` | the number of rows `list()` would return without the host's limit, as an integer: at most `n` after `limit(n)`, `n` fewer (never below 0) after `offset(n)` | | `ids()` | an array of id strings, in sort order (insertion order when unsorted) | Row maps always include the id column, filled from the row's key when the backend doesn't return it as a field. Ids are strings. `get`, and every verb that takes an id, also accepts an integer and uses its decimal form. A missing row is not an error for reads: `get` returns `()`. Compare with `()` as above, or use Rhai's `??`. A capped read (the host's limit or a `limit(n)`) reads the set's leading window, `fetch_window(0, n)`, which every backend that lists answers, so it fetches only the rows it returns. A backend that refuses the leading window makes the read throw; it never falls back to listing the whole set. With an `offset(n)` too, a backend advertising `window` reads `fetch_window(n, limit)`, so the skipped rows never leave it. Any other backend reads the leading window `fetch_window(0, n + limit)` and drops the first `n` rows. An offset with no cap at all (no `limit` and no host limit) lists the whole set and drops the first `n` rows, since there is no window to ask for. An offset past the end reads nothing: `list()` is empty, `first()` is `()` and `count()` is 0. ## Following relations `ref(relation)` works on sets, not single rows. The result is every row related to any row of the current set: ```rhai // The orders of one client... let ada_orders = table("client").where("id", "c1").ref("orders"); // ...and the clients behind every due order. let owing = table("order").where("status", "due").ref("client"); #{ ada: ada_orders.ids(), owing: owing.sort("name").list().map(|c| c.name), } ``` Relations come from the Vista's metadata: the YAML `references:` block, or the typed table's `with_one` / `with_many`. A `ref` step reads the rows it starts from when the handle resolves: - When the set has exactly one row, the step calls `Vista::get_ref(relation, row)`, so a backend's own traversal applies (SurrealDB record links, scripted references, cross-datasource references). - When it has several, the step takes the bare target and adds an `in` condition: on the foreign key for has-many (the target rows point at ours), on the target's id for has-one (our rows point at the target). - When it has none, or none of the rows has a has-one key, the target is an empty set. Nothing is queried: reads return nothing, `count()` is 0, and a preview reports `"query": null`. The target keeps its columns, and an insert into it throws: no row can belong to an empty set. - A `limit(n)` before the `ref` caps the rows it follows, and an `offset(n)` skips the first `n` of them. Without a limit, a `ref` over more than 1,000 rows (after the offset) is an error. Narrow first. ```rhai let biggest = table("order").sort("total", "desc").limit(2); let nobody = table("client").where("name", "Nobody"); #{ count: biggest.count(), clients: biggest.ref("client").sort("name").list().map(|c| c.name), none: nobody.ref("orders").count(), } ``` The two biggest orders are `o1` (Ada) and `o3` (Ben), so this returns `count: 2`, `clients: ["Ada", "Ben"]` and `none: 0`. Steps after `ref` narrow the target, so `ref("orders").where("status", "due")` is the due orders of the set. A relation keyed on a [computed column](./computed.md#relations-through-a-computed-column) works for has-one and is refused for has-many. ## Introspection Three reads describe the table instead of its rows: ```rhai let orders = table("order"); #{ insert: orders.capabilities().insert, columns: orders.columns().map(|c| c.name), references: orders.references().map(|r| r.name + " " + r.kind), } ``` This returns `insert: true`, the columns `id`, `client`, `total`, `status`, and the references `["client has_one"]`. - `capabilities()` is a map with one boolean per capability, keyed by its name. The keys come from [`Capability::iter()`](vantage_vista::Capability::iter), so every capability is listed: `insert`, `update`, `delete`, `import`, `subscribe`, `order`, `search`, `filter_ops`, `native_conditions`, `window`, `cursor`, `traverse` and `confine_writes`. There is no `count` key: every Vista can count. A script that may run against different backends can check before it writes. - `columns()` is an array of `#{ name, type, flags }`. `type` is `()` for a column declared without a type. - `references()` is an array of `#{ name, kind, contained }`. `kind` is the YAML name: `"has_one"`, `"has_many"`, `"contains_one"` or `"contains_many"`; `contained` marks relations stored inside the row (see [Contained Relations](../new-persistence/step9-contained-relations.md)). These resolve the handle, so they describe the table as narrowed. Capabilities rarely change with narrowing, but a query-sourced table reports no write capabilities at all. ## Handles in Rust A host that gets a handle back from a script reads it through [`Handle`](vantage_vista::rhai::Handle): `table_name()` is the starting table (`None` for a handle over a Vista), and `resolve(resolver)` builds the Vista with every step applied. Query preview works this way: the script returns a handle, and the host renders the resolved Vista's query. A handle can also start from a Vista instead of a name, which is how `self` reaches `modify:` scripts. The handle holds that Vista as its base and clones it on each resolve, so the same handle resolves as often as you like. Verbs on `self` return new handles, like any other narrowing. --- Source: https://vantage-ui.com/book/rhai/writes.html # Writes On a host with `Terminals::ReadWrite`, a handle also writes. Every write goes through the resolved Vista, so it follows the backend's rules and whatever policy the host's resolver wrapped around it. --- ## The write verbs ```rhai let orders = table("order"); let id = orders.insert(#{ client: "c2", total: 30, status: "due" }); orders.patch(id, #{ status: "paid" }); let deleted = orders.delete("o4"); // true: the row was there let again = orders.delete("o4"); // true: the row is gone either way let patched = orders.patch("o99", #{ status: "paid" }); // false #{ status: orders.get(id).status, deleted: deleted, again: again, patched: patched, } ``` | Verb | Returns | When the row is missing | |---|---|---| | `insert(map)` | the id; an id already in the set returns it | n/a | | `upsert(id, map)` | `id` | inserts it | | `patch(id, map)` | `true` | returns `false` | | `delete(id)` | `true` | returns `true` | | `import_from(source)`, `import_from(source, mapping)` | `#{ inserted, skipped, rejected, cancelled }` | n/a | `patch` returns `false` when the row is missing or outside the handle's set (the backend reported `NotFound`). `delete` returns `true` whenever the row is gone afterwards, including when it was never there or is outside the set. A write that isn't allowed (`Writes::Denied`, a capability the backend lacks) throws, `delete` included, and so does any other failure. Why: [Safe writes](../record-lifecycle.md#safe-writes). `patch` changes only the fields in the map. `upsert` replaces the whole row, so fields left out of the map are gone afterwards. [Computed columns](./computed.md) in the map are dropped before the write reaches the backend. ## Ids `insert` looks at the map's id column. With a value there, that id is used as given. Without one, `insert` mints a UUIDv7 before the first attempt, so a retry reuses it, and returns it. When the id column is flagged `auto`, the backend makes the id instead (`insert_return_id`, not retry-safe; see [Safe writes](../record-lifecycle.md#safe-writes)). A numeric id column without `auto` is an error: pass an id or flag the column. ```rhai let orders = table("order"); orders.insert(#{ id: "o10", client: "c3", total: 55, status: "due" }); orders.upsert("o10", #{ client: "c3", total: 60, status: "due" }); // replaces orders.upsert("o11", #{ client: "c3", total: 5, status: "due" }); // inserts table("client").where("id", "c3").ref("orders").ids() ``` Inserting an id that already exists in the set returns it; an id held by a row outside the set throws. This insert returns `"o1"` and changes nothing: ```rhai table("order").insert(#{ id: "o1", client: "c2", total: 1, status: "due" }) ``` Use `upsert` when a script owns stable ids and may run more than once. ## Writes stay in the set `where`, `search` and `ref` define the set; a write only touches rows in it, an insert fills the set's equality conditions (including a `ref`'s foreign key), and a write that would leave the set throws. `sort` doesn't change membership. A handle with `limit(n)` or `offset(n)` picks rows by position, not by condition, so it can't be written through: drop the limit or offset. ```rhai let paid = table("order").where("status", "paid"); let o2 = paid.delete("o2"); // o2 is due: outside the set let o9 = paid.patch("o2", #{ total: 1 }); // false: not in the set let ada = table("client").where("id", "c1"); ada.ref("orders").insert(#{ id: "o20", total: 9, status: "due" }); // client filled #{ orders: table("order").ids(), o20: table("order").get("o20").client, o2: o2, o9: o9, } ``` returns `#{ orders: ["o1", "o2", "o3", "o4", "o20"], o20: "c1", o2: true, o9: false }`: `o2` is still there. ## Capabilities Each write goes through the Vista method of the same name, and the Vista checks the one capability that names it before the backend sees the call. A missing capability throws an `unsupported` error naming it, such as `capability is not supported; method refused (capability: delete)`: | Verb | Needs | |---|---| | `insert` | `insert` | | `upsert` | `update` | | `patch` | `update` | | `delete` | `delete` | | `import_from` | `import`, checked before the source is read | A narrowed target imports row by row through inserts, so it also needs `insert` and a backend that keeps writes inside the set (`confine_writes`). A host can also turn writes off for every table with `Writes::Denied(message)`. The verbs are still there, and each throws `message`, `delete` included. Vantage UI does this for MCP agents unless the "Allow MCP agents to write data" setting is on. ## Importing `import_from(source)` copies every row of another handle into this table. `source` is read through its narrowing, so it can be any set: a filtered table, the target of a `ref`, a table on another datasource. A `limit(n)` on the source copies only its first `n` rows, and an `offset(n)` skips the first `n`: ```rhai let biggest = table("order").sort("total", "desc").limit(2); let report = table("archive").import_from(biggest); [report.inserted, table("archive").ids()] ``` returns `[2, ["o1", "o3"]]`. Without a mapping, each row keeps its id. With a mapping, each imported row is built from a source row. String values in the mapping may reference source columns as `${row.}`: ```rhai let report = table("archive").import_from( table("order").where("status", "paid"), #{ id: "a-${row.id}", amount: "${row.total}", note: "paid by ${row.client}" } ); #{ report: report, rows: table("archive").list() } ``` - A value that is exactly one reference (`"${row.total}"`) copies the column's value with its type: `amount` stays an integer. - Any other string interpolates into text (`"paid by ${row.client}"`). - Non-string values are literals. - The mapping must set the target's id column. Two source rows mapping to the same id are an error, and so is an id whose `table:` prefix names a different table. An id the target already holds is never overwritten; it counts as `skipped`. Rows that conflict with a narrowed target count as `rejected`. Into a whole table, the rows go through `import_values` in one call and the backend decides which ids it already holds. Into a narrowed target, each row is looked up with `get` first and inserted on its own if it is missing; a row outside the set is `rejected` and the import carries on; any other insert failure stops the import and throws. Importing the same rows twice into a memory table: ```rhai let first = table("backup").import_from(table("client")); let second = table("backup").import_from(table("client")); [first.inserted, second.inserted, second.skipped] ``` returns `[3, 0, 3]`. `cancelled` is `true` when the backend stopped part-way: a host's import can be cancellable (Vantage UI shows a progress dialog with a cancel button). `inserted` then counts the rows written before the stop. A backend signals this by returning an error carrying the [`IMPORT_CANCELLED`](vantage_vista::rhai::IMPORT_CANCELLED) context key (the rows inserted), and optionally [`IMPORT_CANCELLED_SKIPPED`](vantage_vista::rhai::IMPORT_CANCELLED_SKIPPED) (the rows skipped). ## Writes through a Dio A host may hand scripts Vistas whose writes go through a Dio: [`Dio::cached()`](vantage_diorama::Dio::cached) is a Vista over the Dio's cache whose inserts, patches and deletes are queued as flashes and written through to the master. To the script it is just a Vista. It is how action bodies in Vantage UI write: the page sees the change at once, and the master write follows. An `insert` without an id is queued like any other, with its minted UUIDv7. When the id column is flagged `auto`, it goes straight to the master instead (there is no id to stage until the master assigns one) and then seeds the cache, so an immediate `get(id)` finds the row. --- Source: https://vantage-ui.com/book/rhai/computed.html # Computed Columns A computed column is a column whose value no backend stores. It is a short Rhai expression over `row`, the record as the backend returned it, and its last expression is the column's value. The expression lives in the Vista's metadata, and the Vista fills the column on every read, so it works the same over SQLite, SurrealDB, a REST API, a CSV file or a memory store. In YAML it is the `lazy:` key on a column. This chapter covers what a computed column does, how to declare one, what it refuses, and how relations treat it. --- ## An example The examples run over two memory tables built from YAML. An order stores an invoice `code` and a `net` amount. Its client key is buried in the code, and its tax and gross amounts are derived: ```yaml name: order columns: id: { type: string, flags: [id] } code: { type: string } net: { type: int } client: type: string references: client lazy: | let parts = row.code.split("-"); // "INV-c1-0001" parts[1] vat: { type: int, lazy: "row.net / 5" } gross: { type: int, lazy: "row.net + row.vat" } ``` ```yaml name: client columns: id: { type: string, flags: [id] } name: { type: string } vip: { type: bool } badge: type: string lazy: | if row.vip { "★ " + row.name } else { row.name } references: orders: { table: order, kind: has_many, foreign_key: client } ``` A script reads computed columns like any other: ```rhai let o1 = table("order").get("o1"); #{ o1: [o1.client, o1.vat, o1.gross], badges: table("client").list().map(|c| c.badge), computed: table("order").columns() .filter(|c| c.flags.contains("calculated")) .map(|c| c.name), } ``` For order `INV-c1-0001` with `net: 100` this returns `o1: ["c1", 20, 120]`, the badges `["★ Ada", "Ben"]`, and `computed: ["client", "vat", "gross"]`. ## How the value is computed - Columns compute in declaration order, and each sees the values of those declared before it. `gross` reads `row.vat`, which `vat` has just added. - `row` is the record as the backend returned it. Nested records keep their shape (`row.Message.Headers.Subject`), and a missing field is `()`. The id column may not be in the record: memory and several other backends keep the id as the row's key, not as a field. - The script is a block: `let` statements and helper expressions are fine, and the last expression is the value. - It compiles once, when the column is declared. A script that doesn't parse fails the table build, not the first read. A script that throws on a row fails that read with "Computed column failed", naming the column. - It runs under the background operation limit (50 million), so a runaway loop errors instead of hanging the read. - There is no `table(name)` in scope. A computed column sees one row. To bring in data from another table, use [augmentation](../augmentation.md). Every read path fills computed columns: `list_values`, `get_value`, `fetch_window`, `fetch_cursor`, and watched changes. Anything built on a Vista sees them without knowing they exist: data scripts, Dio caches, grids. ## Declaring one In YAML, `lazy:` goes on the column, next to `type:` and `flags:`. Every driver's factory lowers it through [`ColumnSpec::lazy_column`](vantage_vista::ColumnSpec::lazy_column). The script can be inline, as above, or in a file. A loader that supports `!include`, as Vantage UI's does, replaces `lazy: !include ../script/client_key.rhai` with the file's text before the spec is parsed, so `vantage-vista` only ever sees the script. In Rust, [`Column::with_expression`](vantage_vista::Column::with_expression) turns a column into a computed one: ```rust # use vantage_vista::{Column, ColumnFlag}; let gross = Column::new("gross", "int") .with_flag(ColumnFlag::Orderable) .with_expression("row.net + row.vat") .unwrap(); assert!(gross.is_computed()); assert_eq!(gross.expression(), Some("row.net + row.vat")); assert!(gross.has_flag(ColumnFlag::Calculated)); assert!(!gross.has_flag(ColumnFlag::Orderable)); ``` `with_expression` needs the `rhai` feature and returns an error, naming the column, when the script doesn't compile. It flags the column `calculated` and removes `orderable` and `searchable`. A shell then reports the column in its `VistaMetadata` like any other. Typed tables take spec columns through `Table::add_spec_column`, which registers a `lazy:` column as computed. That is how the SQL, SurrealDB, MongoDB, CSV, REST and GraphQL factories fold `lazy:` into a Vista. The expression isn't serialized. A `Column` that goes through serde comes back as a stored column that keeps only its `calculated` flag. A Vista you build by hand takes a computed column the same way: put the [`Column::with_expression`](vantage_vista::Column::with_expression) column in the `VistaMetadata` the shell reports, and every read fills it. A `lines` column over a note's `contents`: ```rhai row.contents.split("\n").len() - 1 ``` ## What they refuse The backend holds no values for a computed column, so anything that would ask the backend about it is refused. **Writes drop them.** Inserts, patches, upserts and imports remove computed columns from the record before it reaches the backend: ```rhai let orders = table("order"); orders.insert(#{ id: "o4", code: "INV-c2-0004", net: 50, gross: 1 }); orders.patch("o1", #{ net: 200, vat: 0 }); [orders.get("o4").gross, orders.get("o1").vat] ``` This returns `[60, 40]`: the `gross: 1` and `vat: 0` were dropped, and the read computed both from the new `net`. Vantage UI keeps computed columns out of its forms for the same reason. **Filters, orders and aggregates error**, naming the column: ```rhai table("order").where("gross", ">", 100).count() // throws ``` ```rhai table("order").sort("gross", "desc").list() // throws ``` The first fails with `Step can't be applied (step: Where { col: "gross", … }): Computed column can't be used to filter, order or aggregate; the backend doesn't hold its values (column: "gross", operation: "condition")`. The error is marked unsupported, like any other narrowing a backend can't do. In Rust, `Vista::add_condition` and `set_order` refuse the same way. To filter on a derived value, filter on the stored columns it comes from (`net` rather than `gross`), or compute it in the backend's query: a [query-sourced table](./query-tables.md) or SurrealDB's `expr:` column (see [Other ways to compute a column](#other-ways-to-compute-a-column)). ## Relations through a computed column A has-one relation whose foreign key is a computed column of the parent works. The parent row already carries the computed value, and the target is narrowed by its own id, which it stores. `order.client` above is such a key. A has-many relation whose foreign key is a computed column of the child can't work: the child table would have to be filtered by a value it doesn't store. ```rhai let big = table("order").where("net", ">", 50); #{ one: table("order").where("id", "o3").ref("client").first().name, many: big.ref("client").sort("name").list().map(|c| c.name), } ``` Both forms of `ref` work for has-one: from one row (`"Ben"`), and from several (`["Ada", "Ben"]`), where the parents' computed keys become an `in` condition on the client id. The reverse direction throws: ```rhai table("client").where("id", "c1").ref("orders").ids() // throws ``` with `can't narrow the relation's target to the parent row`, naming the relation `orders` and the column `client`, caused by the computed-column refusal above. `Vista::get_ref` narrows through the same [`Reference::narrow`](vantage_vista::Reference::narrow), so it refuses the same way. A nested insert that would have to write a computed foreign key (into the parent for has-one, into each child for has-many) is refused before anything is written. ## Other ways to compute a column `lazy:` is one of several ways to derive a column in vantage. They differ in where the value is made, and that decides what the backend can do with it: | Where | How | Runs | Can filter/sort | Link | |---|---|---|---|---| | SQL / SurrealDB expression | `Table::with_expression(name, …)`, SurrealDB YAML `expr:` | in the query | yes | [`Table::with_expression`](vantage_table::table::Table::with_expression) | | Rust after the read | `Table::with_lazy_expression(name, …)` | after each read, in Rust | no | [`Table::with_lazy_expression`](vantage_table::table::Table::with_lazy_expression) | | Rust on write | a `BeforeSave` hook that stores the value | on insert and update | yes (it's stored) | [`Hook`](vantage_table::table::Hook) | | Rhai `lazy:` | this chapter | after each read, in the Vista | no | — | | Dio `sugar:` | [`Sugar`](vantage_diorama::Sugar) | on the rows one Scenery reads | no | — | - **`with_expression`** takes a closure that returns an expression in the backend's own language, rendered into the `SELECT` (the intro's [computed fields](../intro/step2-tables.md#computed-fields-with-expressions) use a correlated subquery). In Rust, `get_column_expr(name)` returns that expression, so a condition or an order can use it. SurrealDB's `expr:` column is the YAML form (see [Expression dialects](./dialects.md#surrealdb-expressions)). - **`with_lazy_expression`** runs an async Rust closure on each record a `list` or `get` returns. It can do I/O, which a `lazy:` script can't: the intro's [Augmentation step](../intro/step6-augmentation.md) uses it to download a file and derive columns from it. It never takes part in a query. The Vista reports its column as `calculated`. - **A `BeforeSave` hook** in the `Populate` phase sets a field before every insert and update (see [Lifecycle hooks](../record-lifecycle.md#lifecycle-hooks)). The value is then stored, so the backend can filter, sort and aggregate it, but it changes only when the row is written through that table. Pick the backend when the column must filter or sort; Rust when the logic needs the type system; `lazy:` when it must travel with YAML. ## `lazy:` and `sugar:` An application can also add derived columns per view with diorama's `Sugar`. Vantage UI, for example, exposes it as `sugar:` on a grid, binder or other component. The two look similar and are used for different things. | | `lazy:` (computed column) | `sugar:` (Vantage UI) | |---|---|---| | Declared on | the table | one component on one page | | Seen by | every reader of the Vista: pages, relations, data scripts, Dio caches | that component only | | Runs | inside the Vista, on each read | on its own thread, over the rows the component reads | | Script shape | one expression over `row` | a loop over `next_row()` that calls `emit(#{ … })`, with helper functions and setup before the first row | | Relations | has-one can follow it | none | Use `lazy:` for a value that is part of the data, such as a key, an amount or a status that relations and scripts need. Use `sugar:` for presentation on one page, or when the script needs setup (a lookup map, helpers) before the first row. Sugar is built on diorama's [`Sugar`](vantage_diorama::Sugar), which merges the outputs into the rows one Scenery reads and strips them from anything written back. Its YAML is documented in Vantage UI's skills. --- Source: https://vantage-ui.com/book/rhai/records.html # Records A record is a draft of one row. It reads like a map, stages edits, and flashes only what changed. Use it when a script edits a row field by field, or when a host hands a script "the current row" (Vantage UI's action bodies receive `row` as a record). --- ## Editing an existing row `record(id)` loads the row through the handle and throws if no row has that id: ```rhai let o = table("order").record("o2"); o.status = "paid"; o["total"] = 45; let was = o.baseline().status; let staged = [o.dirty("status"), o.dirty("client")]; o.flash(); // patches `status` and `total`, nothing else #{ was: was, staged: staged, dirty: o.is_dirty(), stored: table("order").get("o2"), } ``` A record keeps two layers: - the **baseline**: the row as last loaded or flashed (`baseline()` returns it as a map); - the **changes**: fields set since then. Reading a field (`o.status`, `o["status"]`) returns the change if there is one, else the baseline value, else `()`. Setting a field stages a change. `flash()` sends a `patch` with the changed fields only, then folds them into the baseline. A record with no changes writes nothing and returns its id. ## A new row `record()` with no id starts a draft of a row that doesn't exist yet: ```rhai let n = table("client").record(); n.set(#{ name: "Dee", vip: false }); let id = n.flash(); // an insert under a new UUIDv7 id #{ same_id: n.id == id, status: n.status(), stored: table("client").get(id).name, } ``` The first `flash()` of a new record is an `insert`. After it, the record has an id and later flashes are patches. `set(map)` stages several fields at once and returns the record, so `table("t").record().set(#{ … }).flash()` works in one line. A key with dots (`"inventory.stock"`) writes into a nested map, which is how a form field bound to an embedded object reaches the right place. The id column can't be set on a record. `n.id` is read-only, and setting the id column through `set` or an index throws: ```rhai let n = table("client").record(); n.set(#{ id: "c9", name: "Eve" }) // throws: `id` is the id column ``` To insert under an id you choose, use `insert(#{ id: …, … })` or `upsert(id, map)` on the handle. The record finds out which column is the id column on the first field set, by resolving its table, and keeps the answer, so later sets don't resolve the table again. A table with no id column falls back to `id`. ## Dirty state and revert ```rhai let c = table("client").record("c2"); c.name = "Benjamin"; c.vip = true; c.revert("vip"); let still_dirty = c.is_dirty(); // `name` is still staged c.revert(); [still_dirty, c.is_dirty(), c.name] ``` | Verb | Meaning | |---|---| | `is_dirty()` | any field staged | | `dirty(col)` | `col` staged | | `revert(col)` | drop the staged change to `col` | | `revert()` | drop every staged change | | `baseline()` | the last flashed row, as a map (empty for a new record) | ## Status and failures `status()` is `"tracking"` normally, `"pending"` while `flash()` or `delete()` runs, and `"failed"` when the last one failed. A failed flash throws, and the record keeps its staged changes, so the script can fix them and try again. `rejection()` then returns `#{ message }`, and returns `()` while the record isn't failed. ```rhai let o = table("order").record("o3"); table("order").delete("o3"); // the row goes away underneath the draft o.status = "refunded"; let threw = false; try { o.flash(); } catch (err) { threw = true; } #{ threw: threw, status: o.status(), message: o.rejection().message, staged: o.dirty("status"), } ``` Flashing a record whose row has gone fails with "record `o3` no longer exists". Servo drafts, which carry per-field validation errors from a Dio's flash route, also report a `fields` map in their rejection (see [Layers](./layers.md)). A new record mints its UUIDv7 on the first `flash()` and keeps it, so flashing again after a failure can't create a second row (see [Safe writes](../record-lifecycle.md#safe-writes)). A table whose id column is flagged `auto` leaves the id to the backend instead. ## Deleting `delete()` deletes the record's row and returns `true`, also when the row was already gone. A delete that isn't allowed throws, and the record is marked `failed` with the error as its rejection. A new record that was never flashed has nothing to delete and throws "record has no id; it was never flashed". ## Permissions Records write through the same checks as the write verbs: the target's capabilities (`insert` for a new record, `update` for an existing one, `delete` for `delete()`), and the host's `Writes` setting. Under `Writes::Denied(message)`, `record(id)` still loads and reads, and `flash()` or `delete()` throws `message`. Vantage UI uses this for action predicates (`when:`), which see `row` but must not change it. ## Records in Rust A host that already holds a row can give it to a script without a second fetch: [`RecordDraft::from_row(handle, writes, id, row)`](vantage_vista::rhai::RecordDraft::from_row). A named handle resolves through the resolver it carries, set with [`Handle::with_resolver`](vantage_vista::rhai::Handle::with_resolver). After the script runs, `changes()` returns what it staged and `values()` the row as the script last saw it. Vantage UI builds the action body's `row` this way, over the Vista that [`Dio::cached()`](vantage_diorama::Dio::cached) returns for the page's table, so the flash goes through the Dio's write queue. A row that belongs to no table, such as the `row` an action predicate reads, still needs a handle with a resolver: the first field set asks it for the id column. A Vista over an empty [`MockShell`](vantage_vista::mocks::MockShell) (feature `mocks`) serves: ```rust # use std::sync::Arc; # use ciborium::Value as CborValue; # use vantage_vista::mocks::MockShell; # use vantage_vista::rhai::{Handle, RecordDraft, Writes}; # use vantage_vista::{Vista, VistaResolver}; # let values = [ # ("status".to_string(), CborValue::Text("paid".into())), # ("total".to_string(), CborValue::Integer(120.into())), # ] # .into_iter() # .collect(); let no_table: Arc = Arc::new(|_: &str| Ok(Vista::new("row", MockShell::new()))); let row = RecordDraft::from_row( Handle::named("row").with_resolver(no_table), Writes::Denied("this row is read-only".into()), "o1".into(), values, ); # assert!(row.changes().is_empty()); ``` A script given this `row` can read it and stage fields, and every flash throws the denial: ```rhai row.status = "void"; // staged on the draft let flashed = true; try { row.flash(); } catch (err) { flashed = false; } #{ status: row.status, was: row.baseline().status, flashed: flashed } ``` returns `status: "void"`, `was: "paid"` and `flashed: false`. Setting `row["id"]` still throws, because the empty table has no id column and the draft falls back to `id`. --- Source: https://vantage-ui.com/book/rhai/surfaces.html # Surfaces The data vocabulary is the same everywhere. What changes from one place to the next is the host: which tables `table(name)` reaches, which terminal verbs exist, and what the script is given in scope. This chapter goes through each surface with a tested example. --- ## Overview Rows marked "(app)" are hosts an application builds for itself; the values shown are Vantage UI's. The rest come with vantage. | Surface | `table(name)` resolves | Terminals | Limits | In scope | |---|---|---|---|---| | `run_script` (agent tools) | the caller's resolver | ReadWrite, `list()` capped at 1 to 50 rows | Background | | | `preview_script` | the caller's resolver | Describe | Background | | | YAML `modify:` | the backend's spec resolver | Describe | Background | `self` | | reference build script | the backend's spec resolver | Describe | Background | `row` | | `lazy:` computed column | none | none: no data vocabulary | Background | `row` | | form `options:` (app) | the app's tables | ReadWrite, writes denied, capped | | form values | | action predicates, `when:` (app) | none | ReadWrite, writes denied | Ui | `row` | | action bodies (app) | the app's tables, writes through Dio | ReadWrite | Ui | `row`, `actions` | | form `on_submit`, wizard forms (app) | the app's tables, writes through Dio | ReadWrite | Background | `form` (a Servo), `wizard` | | wizard `worker:` (app) | the app's tables | ReadWrite, 50 rows | Background | `wizard`, `state` | | faker sims | the sim's memory store | ReadWrite | the sim's own budget | `args` | ## Agent scripts: `run_script` [`run_script`](vantage_vista::rhai::run_script) evaluates a script with every read and write verb and returns its last value as JSON. It runs under `spawn_blocking` for you: ```rhai let due = table("order").where("status", "due"); #{ count: due.count(), rows: due.sort("total", "desc").list() } ``` ```rust # use std::sync::Arc; # use ciborium::Value as CborValue; # use vantage_memory::{MemoryCatalog, MemoryStore}; # use vantage_vista::rhai::{Writes, run_script}; # use vantage_vista::{Column, ColumnFlag, VistaMetadata, VistaResolver}; # const AGENT_SCRIPT: &str = r#" # let due = table("order").where("status", "due"); # #{ count: due.count(), rows: due.sort("total", "desc").list() } # "#; # fn resolver(store: &MemoryStore) -> Arc { # let catalog = MemoryCatalog::new(store.clone()); # catalog.register( # "order", # VistaMetadata::new() # .with_column(Column::new("id", "String").with_flag(ColumnFlag::Id)) # .with_column(Column::new("total", "int").with_flag(ColumnFlag::Orderable)) # .with_column(Column::new("status", "String")) # .with_id_column("id"), # ); # Arc::new(catalog) # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let store = MemoryStore::new(); # for (id, total, status) in [("o1", 120, "paid"), ("o2", 40, "due"), ("o4", 15, "due")] { # let row = [ # ("total".to_string(), CborValue::Integer(total.into())), # ("status".to_string(), CborValue::Text(status.into())), # ] # .into_iter() # .collect(); # store.table("order").insert_as(id, row)?; # } let out = run_script( AGENT_SCRIPT.to_string(), resolver(&store), 1, Writes::Denied("writing data is turned off".into()), ) .await .unwrap(); # assert_eq!(out["count"], 2); # assert_eq!(out["rows"].as_array().unwrap().len(), 1); # Ok(()) # } ``` The limit is clamped to `MIN_LIMIT..=MAX_LIMIT` (1 to 50; `DEFAULT_LIMIT` is 5) and caps every `list()`, so `rows` above holds one row while `count` is 2. This is an inspection surface, not a bulk reader. Compile, runtime and backend errors all come back as the error string. Vantage UI's MCP `run_data_script` tool is this function. Its writes stay `Denied` unless the user turns on "Allow MCP agents to write data", a setting separate from the one that lets agents read. ## Query preview: `preview_script` [`preview_script`](vantage_vista::rhai::preview_script) runs a script on a `Describe` host, so it can build a handle but never read rows. The script must end on the handle. The resolved Vista renders its query through [`Vista::preview`](vantage_vista::Vista::preview): ```rhai table("order").where("status", "paid").sort("total", "desc") ``` The result is JSON with the Vista's `driver` and the `query` it would run, plus a `note` naming anything the query leaves out. Here the memory store reports `driver: "memory"` and `query: "scan order WHERE status = \"paid\" ORDER BY total DESC"`. One exception to "never reads": a `ref` step still reads the rows it starts from, because the target's condition depends on them. When it finds none, the preview reports `"query": null` with a note, because nothing would be queried. ## `modify:` scripts A YAML spec's `modify:` script runs on the finished Vista, exposed as `self`. A `Describe` host evaluates it with [`eval_modify_script`](vantage_vista::rhai::eval_modify_script): ```rhai self.where("vip", true).sort("name", "desc") ``` The script's value is its last statement, and it must be a table handle: that handle is resolved and becomes the table. A script that ends on anything else is an error, "The script must end on a table handle". `modify:` narrows a real table, so the Vista stays writable. Reference build scripts finish the same way. ### Backend extensions A backend can add its own verbs to handles through [`TableShell::register_rhai_extensions`](vantage_vista::TableShell::register_rhai_extensions), which registers its functions and constants in one call. SurrealDB adds its whole expression vocabulary, plus `with_condition(expr)`, which narrows by a native SurrealQL expression that `where` can't spell: ```rhai self.with_condition(surreal::ident("is_paying_client") == true) .sort("name", "asc") ``` and `surreal::me`, the current-record anchor for graph paths. MySQL adds the SQL expression vocabulary and the same `with_condition(expr)`; MongoDB's `with_condition` takes a filter written as a Rhai map, `self.with_condition(#{price: #{"$gt": 100}})`, with `mongo::oid`, `mongo::date` and `mongo::doc` for the BSON values a map can't spell. A backend's free functions and constants live in its own namespace: `surreal::` for SurrealDB, `sql::` for every SQL database, `mongo::` for MongoDB. So `table(name)` is always the data verb, and a backend function never shadows one. The verbs on handles, `with_condition` included, and the methods and operators on a backend's own types (`.from`, `.alias`, `["col"]`, `==`) take no prefix. `with_condition` narrows a Vista in hand, not a list of steps, so it works on `self` only. On a handle from `table(name)`, or after a `ref` step, it throws an error naming the verb. A backend implements it with [`Handle::with_native_condition`](vantage_vista::rhai::Handle::with_native_condition), which hands a [`NativeCondition`](vantage_vista::NativeCondition) to the shell's `add_native_condition`. This stand-in passes an equality as the native condition: ```rust # use vantage_rhai::rhai::{Dynamic, EvalAltResult}; # use vantage_vista::rhai::{Handle, dynamic_to_cbor}; # use vantage_vista::{Condition, FilterOp, NativeCondition}; fn with_condition(h: &mut Handle, col: &str, value: Dynamic) -> Result> { let key = format!("{col} = {value}"); let condition = Condition::new(col, FilterOp::Eq, dynamic_to_cbor(value)?); h.with_native_condition("with_condition", NativeCondition::new(condition, key)) .map_err(|e| e.to_string().into()) } ``` `with_native_condition` narrows a copy of the Vista and returns a new handle over it, so like every other verb it leaves the handle it was called on alone: ```rhai let all = self; let vip = all.with_condition("vip", true); all ``` still ends on every client. Verbs on `self` return new handles, so narrowing is a chain, whichever verbs it uses: ```rhai self.with_condition("vip", true).with_condition("name", "Cy") ``` leaves only Cy, and ```rhai self.with_condition("vip", true).sort("name", "desc") ``` leaves Cy and Ada, in that order. Chain the calls, or assign a step with `let` and continue from the variable. A statement that ends on a narrowing verb (`where`, `sort`, `search`, `limit`, `ref`, `with_condition`) outside the script's last statement drops its handle, whatever the chain starts from, and is refused when the script compiles, with "this statement's table handle is discarded; chain the calls or assign it": ```rhai self.with_condition("vip", true); // throws: this handle would be dropped self.sort("name", "desc"); ``` ```rhai let all = self; all.with_condition("vip", true); // throws: this handle would be dropped all ``` A last statement that isn't a handle is refused too. Watch an `if` with no `else`: when its condition is false its value is `()`, so end it with `else { self }`: ```rhai let vip_only = false; if vip_only { self.with_condition("vip", true) } // throws: a false `if` with no `else { self }` is () ``` Backends without a scripting vocabulary (CSV, REST, SQLite and PostgreSQL Vistas) keep the default no-op and get the data vocabulary alone. ## Reference build scripts A reference whose join isn't a plain foreign key can carry a `rhai:` build script. It runs at traversal time with the parent row as `row`, and must end on a handle: ```rhai table("order").where("client", row.id).where("status", "due") ``` In YAML: ```yaml references: due_orders: table: order kind: has_many foreign_key: client surreal: rhai: | table("order").where("client", row.id).where("status", "due") ``` [`eval_ref_script`](vantage_vista::rhai::eval_ref_script) evaluates it. ## Computed columns A `lazy:` column script runs on the shared background host, with no data vocabulary at all: no `table(name)`, no handles, only `row`. It computes one value from one row, every time the Vista reads that row. [Computed columns](./computed.md) covers it. ## Read-only hosts A read-only host is a `Terminals::ReadWrite` host with `writes: Writes::Denied(message)`. It reads freely, and every write throws the message. Records still load, so a script can read `row.status`, but flashing throws: ```rhai let o = table("order").record("o1"); o.status = "void"; o.flash() // throws: this host only reads ``` Table write verbs such as `delete` throw the same message. Vantage UI uses read-only hosts for form `options:` scripts, and for action predicates, which get `row` but no `table(name)`. ## Read-write hosts in an application An application decides what its action scripts see; Vantage UI, for example, gives action bodies and form `on_submit` scripts `table(name)` over the app's tables and, for row actions, `row` as a [record](./records.md): ```rhai let r = actions.cancel_order(row); row.status = "cancelled"; row.cancellation_reason = r.reason; row.flash(); ``` The resolver behind `table(name)` sends writes through each table's Dio, so open pages see them at once, and applies the app's write rules (read-only tables, reference coercion, generated ids). Form and wizard scripts also see Servo drafts, which have their own vocabulary ([Layers](./layers.md)). The action and form vocabularies themselves (`actions.*`, `notify`, `form`) are documented in Vantage UI's skills. ## Faker sims Sims are covered in [Faker Sims](./faker.md): they use this vocabulary over a memory store, with `table()` naming the sim's own table, plus time and random verbs. --- Source: https://vantage-ui.com/book/rhai/layers.html # Layers The data vocabulary acts on Vistas and nothing else. Three parts of the framework sit above Vista and give scripts words of their own: Servo drafts in `vantage-diorama`, Scenery shapes in Vantage UI, and faker sims. This chapter shows how each one relates to the data vocabulary, and why they stay separate. --- ## The rule `vantage-vista` never refers to Dio, Servo or Scenery. Each upper layer either hands scripts a plain Vista or registers its own vocabulary next to `DataVocab` on the same host. A host for a form's `on_submit` script, for example, registers both `ServoVocab` and `DataVocab`, so the script has `form` (a Servo) and `table(name)` (Vistas) side by side. Where an upper layer means the same thing as a data verb, it uses the same word: `where`, `sort`, `limit`, `ref`, `flash`, `is_dirty`. Where it means something different, it uses its own word, even if a data verb looks close. ## Dio: no vocabulary of its own A Dio doesn't add script words. [`Dio::cached()`](vantage_diorama::Dio::cached) returns a Vista whose reads come from the Dio's cache and whose writes are queued as flashes and written through to the master. A host that resolves `table(name)` to Dio Vistas gives scripts cached reads and optimistic writes, and the scripts are the same ones that would run against the bare backend. Vantage UI's action bodies work this way. ## Servo: a form draft A [Servo](vantage_diorama::Servo) is a draft of one record bound to a Dio. It tracks a baseline, staged edits, a status and per-field rejections from the Dio's flash route. Forms and wizards in Vantage UI hand one to their scripts as `form`. `ServoVocab` (`vantage_diorama::rhai`) registers its words: ```rhai servo.set("id", "tag:AB12-CD34"); servo.set("status", "unregistered"); if !servo.is_dirty() { throw "draft should be dirty"; } if !servo.dirty("status") { throw "field should be dirty"; } servo.flash() ``` A Servo and a data-vocabulary [record](./records.md) look alike, and some words match: | | record (`table(t).record(id)`) | Servo | |---|---|---| | read a field | `r.col`, `r["col"]` | `servo.get("col")` | | stage a field | `r.col = v`, `r.set(map)` | `servo.set("col", v)` | | id | `r.id` | `servo.id()` | | id of a new row | a UUIDv7 minted on the first flash | [`IdStrategy`](vantage_diorama::IdStrategy): `Uuid` (the default) mints one when the Servo opens | | the whole draft | (field reads) | `servo.record()` | | baseline | `r.baseline()` | `servo.baseline()` | | dirty | `r.is_dirty()`, `r.dirty(col)` | `servo.is_dirty()`, `servo.dirty(col)` | | revert | `r.revert(col)`, `r.revert()` | `servo.revert(col)`, `servo.revert_all()` | | fields that differ from the baseline | | `servo.error()` | | write | `r.flash()`: a patch or insert, now | `servo.flash()`: a flash through the Dio, waits for it to settle | | status | `r.status()`, `r.rejection()` (`#{ message }`) | `servo.status()`, `servo.rejection()` (`#{ message, fields }`, per-field errors) | They stay separate because they are different objects. A record is a script's own short-lived draft over any Vista, and is gone when the script ends. A Servo lives in the Dio and outlasts the script. The form's widgets are bound to it, its flash goes through the Dio's write queue and route, and a route can reject single fields. Folding the two into one type would make the record depend on the Dio, or hide the Servo's lifecycle behind data-vocabulary words. `servo.flash()` needs a tokio runtime context and blocks on the flash, so Servo scripts run under `spawn_blocking`, like any data script with a network backend. ### Which one to use Use a record for a script's own one-off edit over any Vista; it writes when the script says so and is gone after. Use a [Servo](vantage_diorama::Servo) when a person edits: it lives in the Dio, outlasts the script, takes per-field rejections, and its flash goes through the flash route. Unless the id is left to the backend (an `auto` id column, `IdStrategy::Auto`), both reuse their id when a flash is retried, so neither can create a row twice (see [Safe writes](../record-lifecycle.md#safe-writes)). ## Scenery: the shape of a view A Scenery is a live, reactive view over a Dio: a [`TableScenery`](vantage_diorama::TableScenery) keeps an ordered window of rows current as the Dio's cache changes, a [`ValueScenery`](vantage_diorama::ValueScenery) keeps one aggregate current, and a [`RecordScenery`](vantage_diorama::RecordScenery) one row (see [Scenery — Reactive Views](../intro/step7-scenery.md)). An application binds a widget to one. `vantage-diorama` has no Scenery script words. Vantage UI adds some: a component's `table:` binding can be a plain table key or a short script, `scenery(name)` followed by narrowing words, that describes which Scenery to open. The script builds a description, and the page opens the Scenery when it mounts: ```rhai scenery("top").sort("visitors", "desc").limit(10) ``` A child view through a relation, for the row selected in a grid: ```rhai scenery("launches").where("id", "42").ref("payloads") ``` In a page this reads `launches_grid.selected_id` instead of `"42"`, and the page remounts the binding when the selection changes. Shared words, with the same meaning as on a table handle: - `where(col, value)`: equality only. Values carry as text, so a page number needs no `to_string()`. - `sort(col, dir)`: the direction is required, `"asc"` or `"desc"`. - `limit(n)`: at most `n` rows. - `ref(relation)`: the related rows. On a Scenery it must follow exactly one `where` on the parent's id and nothing else, because the binding follows one parent row. Scenery-only words: - `tail(n)`: follow the last `n` rows as they arrive, for append-only feeds. - `arg(name, value)` and `args(#{ … })`: parameters passed to the table's own `rhai:` query script as its `args` map, which reshape the query itself rather than filtering its result (see [Arguments](./query-tables.md#arguments)). Scenery has no reads or writes: it describes a view, and the page's Dio opens it. The vocabulary lives in Vantage UI (`SceneryVocab` in `crates/app/src/infra/scenery_script.rs`), runs under `Limits::Ui`, and records every page value the script reads as a dependency of the binding. ## Faker sims Sims use the data vocabulary over a memory store plus time and random words. See [Faker Sims](./faker.md). --- Source: https://vantage-ui.com/book/rhai/faker.html # Faker Sims `vantage-faker` fills memory tables with generated rows. With its `sim` feature it also runs Rhai scripts as simulations over the same memory store: each sim is a script with its own clock, and a `SimDef` spawns copies of it (a burst at start, then a rate per sim minute, up to a maximum alive). Sims write through the data vocabulary from this part of the book, so a Dio, a grid or an API watching the store sees the rows arrive, change and leave. --- ## A sim script A faker sim is a Rhai script that runs from top to bottom on its own thread, with its own sim clock. Its local variables are its state, and `sleep` pauses it. The data vocabulary is there in full, over the engine's memory store, and `table()` with no name is the sim's own table: ```rhai if table().where("status", "Placed").count() >= 40 { done(); } let id = table().insert(#{ customer: fake("name"), total: rand_float(5.0, 80.0), status: "Placed" }); sleep(minutes(rand_int(5, 20))); table().patch(id, #{ status: "Shipped" }); table("order_event").insert(#{ order: id, note: "shipped" }); sleep(hours(2)); table().delete(id); ``` ## Sim words On top of the data vocabulary, sims get: - **data**: `fake_row()` on a handle (`table().fake_row()`, `table("t").fake_row()`) returns a map with a generated value for each column the table declares through `SimEngineBuilder::columns`, without the id column. Sequential generators (`walk`, evenly spread `date`) advance one step per call, shared by every sim of the def. - **time**: `seconds(n)`, `minutes(n)`, `hours(n)`, `days(n)`, `sleep(d)`, `wait_until(t)`, `now()`, `now_secs()`, `wall_now()`, `wall_in(d)`, `elapsed()`, `clock()`, `done()`. - **random**: `pick`, `pick_weighted`, `rand_int`, `rand_float`, `chance(p)`, `pattern("BA####")`, `sentence(min, max)`, `fake(kind)`, `date_between(from, to)`. - **spawn**: `spawn_sim(name, args?)`, `sim_id()`, `sim_name()`, and the spawner's `args`. - **geo**: `great_circle`, `bearing`, `interpolate`. ## The store underneath A sim's tables are memory Vistas, so reads and writes finish on the first poll and sims run without tokio (see [the bridge](./hosts.md#async-reads-from-a-synchronous-script)). Writes broadcast the store's change events to any watching Dio. During a warm start, each written table stays quiet and sends one `Reset` at the end. A missing table is a script error: the store's tables are created by the dataset, not by the sim. The `vantage-faker` README covers defs, spawners, clocks, the operations budget and the built-in sims. --- Source: https://vantage-ui.com/book/rhai/dialects.html # Expression Dialects Not every script in a Vantage configuration is a data script. Some build a backend's native query, some compute one value from one row, and some are text with a few holes in it. They run on the same `vantage-rhai` hosts with the same limits, but their vocabularies are separate from the data vocabulary, and from each other. This chapter tells them apart and points to where each is documented. --- ## Data scripts and expression scripts A data script works on sets of rows through Vistas: `table`, `where`, `list`, `insert`. It doesn't know which backend it talks to. An expression script builds a query in one backend's own terms: a SQL `SELECT`, a SurrealQL condition, a graph path. It runs once, when a Vista is built, and its result becomes part of that Vista's query. It reads no rows itself. The two meet in one place: a backend's extension verbs on a data handle, such as SurrealDB's `self.with_condition(expr)`, take an expression built with the backend's vocabulary (see [Backend extensions](./surfaces.md#backend-extensions)). ## SQL query builders A YAML spec for a SQL table can replace its source with a script-built `SELECT`. The script uses `vantage-sql`'s builder (`select`, `from`, `field`, `where`, `group_by`, `expr`) and the shared primitives (`count`, `avg`, `coalesce`, `case_when`, `date_format`, …). The free functions among them live in the `sql::` namespace, one for every SQL database; the methods don't take it: ```yaml name: expensive_products columns: id: type: string flags: [id] name: type: string sqlite: rhai: | sql::select().from("product").field("id").field("name").where(sql::expr("price > 15")) ``` A query-sourced Vista is read-only. A derived Vista (`base:` plus `rhai:`) gets the base's `select()` as `base` and transforms it. [Tables Defined by a Query](./query-tables.md) covers both, and [Query Builder Functions](./query-builders.md) lists every function the builders register. The Rust primitives are in [SQL Primitives](../sql/primitives.md). `where` here is the SQL builder's method and takes an expression. On a data handle, `where` takes a column and a value. The name is the same because both narrow. A `modify:` script has both vocabularies on one host, and each `where` dispatches on its receiver: a select or a table handle. The builder's free functions are namespaced so that none of them can shadow a data verb such as `table(name)`. ## SurrealDB expressions `vantage-surrealdb` registers the same builder shape for SurrealQL, the same primitive names where the concept exists in both, and SurrealDB's own terms where it doesn't (graph paths, `SPLIT`, `me`), in the `surreal::` namespace: `surreal::graph(surreal::me, "placed", "order")`. It backs `rhai:` sources, `base:` transforms, and the expressions handed to `with_condition`. See [SurrealDB Primitives](../surrealdb/primitives.md). The SQL and SurrealDB builders are two copies of a similar grammar. Merging them is a known follow-up and not part of the data vocabulary. A SurrealDB column spec may also carry `expr:`, a server-side computed column: the script is evaluated once with this vocabulary, and the backend projects the result as `() AS ` on every read. Unlike a `lazy:` column it is part of the query, so it can follow record links and call SurrealDB functions. Other backends reject or ignore it. ## `lazy:` columns A column's `lazy:` script is a third kind: it computes one value from one row, in the Vista, after the backend has returned the row. It uses neither the data vocabulary nor a query builder, only `row`. [Computed columns](./computed.md) covers it. ## Templates Many YAML strings are templates: text with `${ … }` holes. `vantage-rhai`'s [`Template`](vantage_rhai::Template) slot evaluates each hole as a Rhai expression against the host's scope and joins the results. A template that is exactly one hole (surrounded by whitespace at most) keeps the hole's value with its type, so `"${ row.total }"` stays a number and a map stays a map. Two look-alikes are not Rhai: - `import_from`'s mapping (see [Writes](./writes.md#importing)) only understands `${row.}` column references. Other holes stay in the text as written. - Vantage UI's action texts (`title:`, `description:`, `terminal.args`) substitute `${row.x}` and `${form.x}` paths, with no expressions. ## Command scripts `vantage-cmd` tables read by running a Rhai script that calls `run(...)` for a subprocess and `parse_json` / `parse_jsonl` on its output, then returns an array of row maps. The script sees the read's conditions, columns, limit and offset as variables. It is a third dialect, for shaping command output into rows, and is documented in the `vantage-cmd` crate. --- Source: https://vantage-ui.com/book/rhai/query-tables.html # Tables Defined by a Query A table usually reads one physical table or collection. A YAML spec for SQLite, PostgreSQL, MySQL or SurrealDB can instead take its rows from a `SELECT` that a Rhai script builds. The script uses the backend's query builder ([Query Builder Functions](./query-builders.md)), not the data vocabulary, and it runs when the table is built, never when a data script runs. --- ## A query as the source `rhai:` in the driver block (`sqlite:`, `postgres:`, `mysql:` or `surreal:`) replaces `table:`. The script's last expression must be a `select()`, built with the backend's namespaced functions (`sql::` for the SQL databases, `surreal::` for SurrealDB): ```yaml name: expensive_products columns: id: type: string flags: [id] name: type: string sqlite: rhai: | sql::select().from("product").field("id").field("name").where(sql::expr("price > 15")) ``` - The script runs once, when the Vista is built. A script that fails or doesn't end on a `select()` fails the build. - The `SELECT` becomes a subquery in the table's `FROM`, aliased under the spec's name: `SELECT "id", "name" FROM (SELECT … FROM "product" WHERE price > 15) AS "expensive_products"`. - `columns:` declares which of the query's output columns the table has, as for any table. - Reads, conditions, sort, search and paging apply on top, as on any table. A data script can narrow `table("expensive_products")` with `where` and `sort` like any other table. - Scalars the script passes to the builder (`sql::expr("price > {}", [15])`, a comparison with a number) are bound as parameters. ## Deriving from another table `base:` names another spec. It is resolved when the table is built, its `select()` is put in the script's scope as `base`, and `inherit:` copies the columns and relations to keep. The script transforms the base query instead of building one: ```yaml name: client_totals id_column: client_id columns: total_due: { type: int } sqlite: base: orders inherit: columns: [client_id] rhai: | base.clear_fields().field("client_id").expression(sql::expr("SUM(total) AS total_due")).group_by(sql::expr("client_id")) ``` This is the aggregate pattern: drop the base's fields, group by a key, declare the aggregate as the table's own column, and re-key the table with `id_column`. `clear_fields()` comes first because PostgreSQL and MySQL reject non-grouped columns that SQLite tolerates. Without `rhai:`, `base:` uses the base query as it is. An inherited relation still works: following it from the derived table narrows the target by the derived query, as in `client_id IN (SELECT "id" FROM (…) AS "debtors")`. ## Read-only A table with `rhai:` or `base:` in its driver block is read-only: the factory turns off the `insert`, `update` and `delete` capabilities, because there is no single table a write could land in. Callers find out the usual way, by checking capabilities. In Rhai, `insert`, `upsert`, `patch`, `delete` and a record's `flash()` throw (see [Capabilities](./writes.md#capabilities)). To keep a table writable while narrowing it with a native expression, use SurrealDB's `modify:` instead, which changes a real table (see [`modify:` scripts](./surfaces.md#modify-scripts)). ## Arguments A query-sourced table can take arguments. The host sets them on the driver block when it opens the table; YAML can't set them. The script sees them as `args`, a map of strings, where `""` means "not set". They reach `rhai:` sources and `base:` transforms, not `modify:` scripts. ```rhai let q = surreal::select().from("order"); if "bakery" in args && args.bakery != "" { q = q.where(surreal::expr("bakery.name = {}", [args.bakery])); } q ``` With `bakery` set to `"Breg"` the query gains the condition, with the name bound as a parameter. Without it the script skips the condition instead of failing. Arguments change the query itself, so they can filter rows before a `GROUP BY`, which a condition on the finished table can't. Vantage UI, for example, sets them from a Scenery binding's `arg(name, value)` (see [Layers](./layers.md#scenery-the-shape-of-a-view)). ## Why scripts can't run their own queries Query building is allowed in one place: a table's source, written by whoever writes the YAML. (SurrealDB's `modify:` scripts, `expr:` columns and reference scripts are configuration too, and use the same builder.) A data script, such as an action body or an agent script, can only narrow tables with `where`, `search` and `ref`. It can't send SQL, can't template a query, and actions can't run custom queries. That keeps every read inside the tables an application declared, every write inside the set rules of [Safe writes](../record-lifecycle.md#safe-writes), and every value a data script passes a parameter, never text spliced into a query. Allowing actions to run queries may come later, behind its own permission. [Config-Driven Vistas](../config-driven-vistas.md#the-rhai-layer) shows where these scripts sit among the other YAML keys. --- Source: https://vantage-ui.com/book/rhai/query-builders.html # Query Builder Functions This page lists every function the SQL and SurrealDB query builders register for Rhai. They are available where a script builds a native query: a [query-sourced table](./query-tables.md)'s `rhai:` and `base:` scripts, and on SurrealDB and MySQL also `modify:` scripts and reference build scripts (SurrealDB `expr:` columns too). Data scripts don't get them (see [Why scripts can't run their own queries](./query-tables.md#why-scripts-cant-run-their-own-queries)). The data vocabulary is listed in [Words at a glance](../rhai.md#words-at-a-glance). SQLite, PostgreSQL and MySQL share one registration (`vantage_sql::register_engine!`), so the "SQL" column holds for all three; where the output differs per database, the row says so. SurrealDB registers its own copy (`vantage_surrealdb::rhai_engine::register_surreal_onto`) with the same names where the concept exists in both. The Rust side of each primitive is in [SQL Primitives](../sql/primitives.md) and [SurrealDB Primitives](../surrealdb/primitives.md). **Free functions are namespaced.** Every function in the tables below that isn't called on a receiver lives in its backend's namespace: `sql::` for all three SQL databases, `surreal::` for SurrealDB. A script writes `sql::select().from(sql::ident("users"))` or `surreal::count(surreal::graph(surreal::me, "placed", "order"))`; the tables leave the prefix out. Methods (`.from`, `.where`, `.alias`, `.when`, `.map`, …), indexers and operators are called on their receiver as written. The namespace keeps the builders from shadowing the data vocabulary's `table`, `where` and `first` on a host that has both. In the signatures, `x` is an expression or an identifier, `sel` a `select()`, and `"…"` a string. Where a function takes a plain value as well, it says so. --- ## Values in a query Two rules hold in both builders: - A number, bool or string passed as an argument (to `expr`'s array, to a comparison, to `coalesce` and `thing` on SurrealDB) becomes a bound parameter, never query text. - A string passed as a name (`ident`, `from`, `field`, `alias`) or as a template (`expr`, `cast`'s type) is written into the query. These come from the configuration's author, not from data. ## Constructors | Function | SQL | SurrealDB | Builds | |---|---|---|---| | `select()` | yes | yes | an empty `SELECT` | | `ident("name")` | yes | yes | a quoted identifier | | `expr("text")` | yes | yes | raw query text | | `expr("a = {}", [args])` | yes | yes | text with `{}` holes; each argument is an expression, an identifier, or a value bound as a parameter | | `fx("name", [x, …])` | yes | yes | a function call `name(x, …)`; arguments must be expressions or identifiers | | `case_when()` | yes | yes | an empty conditional (see [Conditionals](#conditionals)) | | `window()` | yes | — | an empty window spec (see [Window functions](#window-functions)) | | `thing("table", "id")` | — | yes | a record id, `type::record(…)` with both parts bound as parameters | | `param("name")` | — | yes | a parameter, `$name` | | `parent()`, `parent("field")` | — | yes | `$parent`, `$parent.field` | | `time_now()` | — | yes | `time::now()` | | `me` (a variable: `surreal::me`) | — | yes | the current record, the anchor of a graph path | ## Identifiers and expressions | Function | SQL | SurrealDB | Builds | |---|---|---|---| | `id.dot_of("s")` | yes | yes | **differs**: SQL puts `s` in front (`sql::ident("name").dot_of("u")` is `"u"."name"`); SurrealDB puts it after (`surreal::ident("t").dot_of("f")` is `t.f`) | | `id["col"]` | yes | yes | SQL: the column `col` qualified by the identifier's alias or name; SurrealDB: the path `id.col` | | `x["field"]`, `x[n]` | — | yes | a field of an expression, or element `n` of an array | | `id.alias("a")` | yes | yes | SQL: the identifier with an alias, rendered `"users" AS "a"`; SurrealDB: the expression `id AS a` | | `x.alias("a")` | yes | yes | `x AS a`, with `a` quoted as an identifier | | `x.clone()` | yes | yes | a copy | ## Select methods Each method returns the select, so they chain. | Method | SQL | SurrealDB | Adds | |---|---|---|---| | `from("table")`, `from(id)` | yes | yes | a source | | `from(x)` | — | yes | an expression as the source, such as a graph path or a record id | | `from_as("table", "alias")` | yes | — | an aliased source | | `from_as(sel, "alias")` | yes | — | a derived table, `FROM (SELECT …) AS "alias"` | | `field("name")` | yes | yes | a column | | `clear_fields()` | yes | yes | drops every field so far; call it before grouping a `base` | | `expression(x)` | yes | yes | an expression as a field; name it with `.alias("a")` | | `where(x)` | yes | yes | a condition; several are joined with `AND` | | `group_by(x)` | yes | yes | a `GROUP BY` term | | `order_by(x, "asc"\|"desc")` | yes | yes | an `ORDER BY` term; any other direction is an error | | `having(x)` | yes | — | a `HAVING` condition | | `distinct()` | yes | yes | `DISTINCT` | | `limit(n, skip)` | yes | yes | `LIMIT` and offset; SurrealDB writes `LIMIT n START skip` and leaves out `START 0` | | `inner_join("table", "alias", on)` | yes | — | `INNER JOIN` | | `left_join("table", "alias", on)` | yes | — | `LEFT JOIN` | | `group_all()` | — | yes | `GROUP ALL`: the whole result as one aggregate row | | `split("field")`, `split(id)` | — | yes | `SPLIT`: one row per element of an array field | | `only()` | — | yes | `SELECT ONLY`: a single record | | `value()` | — | yes | `SELECT VALUE`: plain values instead of objects | | `sel.subquery()` | — | yes | the select as an expression, `(SELECT …)` | ## Aggregates | Function | SQL | SurrealDB | |---|---|---| | `count(x)` | `COUNT(x)` | `count(x)` | | `count()` | — | `count()` | | `count_distinct(x)` | `COUNT(DISTINCT x)` | `count(array::distinct(x))` | | `sum(x)` | `SUM(x)` | `math::sum(x)` | | `avg(x)` | `AVG(x)` | `math::mean(x)` | | `min(x)`, `max(x)` | `MIN(x)`, `MAX(x)` | `math::min(x)`, `math::max(x)` | | `group_concat(x, distinct)` | SQLite `GROUP_CONCAT(x, ',')`, PostgreSQL `STRING_AGG(x, ',')`, MySQL `GROUP_CONCAT(x SEPARATOR ',')`; `distinct` is a bool | — | | `median(x)`, `stddev(x)` | — | `math::median(x)`, `math::stddev(x)` | | `array_group(x)` | — | `array::group(x)` | ## Scalar functions | Function | SQL | SurrealDB | |---|---|---| | `coalesce(a, b)` | `COALESCE(a, b)`; `a` and `b` must be expressions or identifiers | `a ?? b`; values allowed | | `nullif(a, b)` | `NULLIF(a, b)`; expressions or identifiers | `IF a = b THEN NONE ELSE a END`; values allowed | | `cast(x, "type")` | `CAST(x AS type)` | `type::(x)` | | `round(x, n)` | `ROUND(x, n)` | `math::fixed(x, n)` | | `round(x)` | — | `math::round(x)` | | `date_format(x, "%Y-%m")` | SQLite `STRFTIME`, MySQL `DATE_FORMAT`, PostgreSQL `TO_CHAR` with the tokens translated (`%Y %m %d %H %M %S`) | `time::format(x, "fmt")` | | `time_group(x, "unit")` | — | `time::group(x, unit)`; the unit is bound | | `lower(x)` | — | `string::lowercase(x)` | | `words(x)` | — | `string::words(x)` | | `string_len(x)` | — | `string::len(x)` | | `similarity(x, "term")` | — | `string::similarity::jaro_winkler(x, term)`; the term is bound | | `first(x)` | — | `array::first(x)` | | `len(x)` | — | `array::len(x)` | | `object_entries(x)`, `object_values(x)` | — | `object::entries(x)`, `object::values(x)` | | `type_float(x)`, `type_int(x)` | — | `type::float(x)`, `type::int(x)` | ## Arithmetic and comparison | Function or operator | SQL | SurrealDB | Builds | |---|---|---|---| | `add(a, b)`, `sub(a, b)`, `mul(a, b)`, `div(a, b)` | yes | yes | `(a + b)` and so on; expressions or identifiers only | | `a + b`, `a - b`, `a * b`, `a / b` | — | yes | the same, where one side is an expression and the other an expression or a number | | `a == b`, `a != b`, `a < b`, `a > b`, `a <= b`, `a >= b` | yes | yes | a condition; `==` is written `=`. Either side may be an expression, an identifier or a value; SurrealDB also takes an array `[…]` or a map `#{…}` literal | ## Conditionals `case_when()` starts a conditional, `.when(cond, then)` adds a branch, `.else_(value)` sets the fallback, and `.expr()` turns it into an expression: | Method | SQL | SurrealDB | |---|---|---| | `.when(cond, then)` | `cond` an expression or identifier, `then` an expression | either may be a value | | `.else_(value)` | an expression or identifier | any value | | `.expr()` | `CASE WHEN … THEN … ELSE … END` | `IF … THEN … ELSE … END` | ## Window functions SQL only. `window()` starts a window spec; each method returns it. | Method | Adds | |---|---| | `.partition_by(x)` | a `PARTITION BY` term | | `.order_by(x, "asc"\|"desc")` | an `ORDER BY` term | | `.rows("from", "to")` | `ROWS BETWEEN from AND to`, such as `"UNBOUNDED PRECEDING"`, `"CURRENT ROW"` | | `.range("from", "to")` | `RANGE BETWEEN from AND to` | | `.apply(x)` | the expression `x OVER (…)`, such as `sql::window().partition_by(…).apply(sql::sum(…))` | ## Graph paths SurrealDB only. | Function | Builds | |---|---| | `graph(me, "edge", "table", …)` | an outward path `->edge->table…` from the anchor; 2 to 7 arguments | | `graph("table", "edge", …, me)` | an inward path `table<-edge<-…`; the anchor's position sets the direction | | `graph(…, graph(…), …)` | a nested path; the anchor may be a sub-path, which changes direction per hop | | `recurse(path, min, max)` | `@.{min..max}(path)` | | `id.arrow("edge")` | `id->edge` | | `id.back("edge")` | `id<-edge` | | `id.arrow_field("edge", "field")` | `id->edge.field` | Exactly one argument of `graph` is the anchor (`surreal::me` or a sub-path); it must be first or last. ## Array closures SurrealDB only. These take native Rhai closures and build SurrealQL from them; the closure's parameters become engine-named placeholders (`$value`, `$acc`). | Method | Builds | |---|---| | `x.map(\|v\| body)` | `x.map(\|$value\| body)` | | `x.filter(\|v\| body)` | `x.filter(\|$value\| body)` | | `x.fold(init, \|acc, v\| body)` | `x.fold(init, \|$acc, $value\| body)` | ## On a table handle SurrealDB and MySQL add one verb to the data vocabulary's handles: `self.with_condition(x)` narrows the table by a native expression, such as `self.with_condition(surreal::ident("is_paying_client") == true)`. It is a method, so it takes no namespace, and it works on `self` only (see [Backend extensions](./surfaces.md#backend-extensions)). MongoDB has the same verb, but it takes a filter written as a Rhai map rather than a query-builder expression. --- Source: https://vantage-ui.com/book/live-data.html # Live Data: Push, Poll & Freshness A cache only stays correct if it finds out when the underlying data changes. There are two ways it can: it asks (**poll**), or the database tells it (**push**). A Dio asks for push with one call on its builder, `DioBuilder::live()`: the Dio checks whether the master Vista advertises [`Capability::Subscribe`](vantage_vista::Capability), follows its change stream if so, and quietly does nothing if not. That last part is the bit worth knowing before you pick a backend. A `live()` Dio that builds without error does not mean you have a live feed; it means nothing went wrong. This page says exactly what each shipped driver delivers today, what — if anything — you have to do to get it, and how the Lens keeps a cache fresh when push isn't there. --- ## Three freshness models Every backend answers two questions, and the pair decides what you get: 1. **Can it tell you something changed at all?** A file on disk cannot. A database server can. 2. **When it does, does it hand you the row, or only a signal?** | Model | What arrives | What the Dio does | | -------------- | ------------------------------------------- | ------------------------------------- | | **poll** | nothing — you re-read on a timer | re-lists the whole set | | **coarse push** | "something changed", no payload | re-reads the set to find out what | | **fine push** | the action *and* the affected record | applies that one row to the cache | Fine push is the only model where a change costs you no query. Coarse push still beats polling: you re-read *because something happened*, not on the off-chance. There is a fourth position, currently held by one backend. Fine push still leaves a question open — *does this row belong in my filtered set?* — because a driver that watches the whole table will hand you rows your conditions exclude. **Filtered fine push** closes it: the conditions go into the subscription itself and the database maintains the result incrementally, so a row that changes *out* of the set arrives as a delete. Membership stops being something anyone has to reconcile. ## What each backend delivers today | Backend | Mechanism | Model | Setup required | | -------------- | -------------------------------------- | ----------- | --------------------------------------- | | SpacetimeDB | subscription over WebSocket (BSATN) | filtered fine push | none | | SurrealDB | `LIVE SELECT` over WebSocket | fine push | none | | PostgreSQL | `LISTEN/NOTIFY` on `{table}_changed` | coarse push | a trigger **and** an explicit opt-in | | SQLite | — | poll | — | | MySQL | — | poll | — | | MongoDB | change streams | fine push | a replica set or sharded cluster | | CSV files | — | poll | — | | REST / GraphQL | — | poll | — | | AWS | — | poll | — | | Kubernetes | — | poll | — | | CLI tools | — | poll | — | | Append logs | — (write-only sink) | n/a | — | MySQL is poll-only: it has no `LISTEN/NOTIFY` equivalent, and pushing changes would need a binlog replication client. Only four drivers implement `TableShell::watch` at all. A standalone MongoDB server has no change streams and polls. `MongoVistaFactory::with_change_streams(true)` is for overriding a failed (`Unknown`) topology probe, say behind a proxy that refuses `hello`; it cannot give a standalone server change streams, and forcing it on there makes the watch fail on the server. Everything in the poll rows inherits the trait default, which refuses `watch` and leaves `Capability::Subscribe` out of the shell's capabilities — so `live()` on those does nothing and the Lens's refresher carries the load. ```admonish note title="Poll is a supported answer, not a failure" A polled backend is not broken, and you do not need different application code for it. The Lens keeps its refresher, the scenery is content-aware, and an unchanged re-read bumps no generation and repaints nothing. The cost is latency and a query you didn't strictly need — not correctness. ``` ## Polling: the Lens's refresher A Lens re-reads its Dios through a [`Refresher`](vantage_diorama::Refresher). Its function gets a `LoaderCx` and does the re-read; `LoaderCx::reconcile` makes the cache hold exactly the snapshot it is given. Two settings decide when it runs: - `Refresher::every(interval)` runs it on a timer, `interval` after each Dio is built and then every `interval`. - `Refresher::max_age(ttl)` bounds how stale a cache may be when it is opened. A Dio whose last successful refresh is older than `ttl`, or unrecorded, gets a background refresh when it is built and whenever a table or record scenery opens on it. The refresh time is stored in the cache, so it holds across restarts. They combine freely. `max_age` alone runs no timer: a cache that was refreshed five minutes ago and opened again is left alone until it is older than its bound. `every` alone keeps refreshing whether anyone opens the cache or not. Both together is the usual setting for a polled backend behind a persistent cache: a refresh on open after a long absence, then a steady timer while the application runs. ```rust # use std::time::Duration; # # use ciborium::Value as CborValue; # use vantage_dataset::ReadableValueSet; # use vantage_diorama::{Cache, DioEvent, Lens, Loader, Refresher}; # use vantage_types::Record; # use vantage_vista::mocks::MockShell; # use vantage_vista::{Column, Vista, VistaMetadata}; # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let mut lobby = Record::new(); # lobby.insert("name".to_string(), CborValue::Text("Lobby".into())); # let rooms = Vista::new( # "rooms", # MockShell::new() # .with_metadata( # VistaMetadata::new() # .with_column(Column::new("id", "String").with_flag("id")) # .with_column(Column::new("name", "String")) # .with_id_column("id"), # ) # .with_record("r1", lobby), # ); let lens = Lens::builder() .cache(Cache::memory()) .loader(Loader::eager(|cx| async move { let rows = cx.master().list_values().await?; cx.reconcile(rows).await })) .refresher( Refresher::new(|cx| async move { let rows = cx.master().list_values().await?; cx.reconcile(rows).await }) .max_age(Duration::from_secs(15 * 60)) .every(Duration::from_secs(60)), ) .build()?; // Follows the master's change stream when it advertises Subscribe; // over a polled master the refresher above keeps the cache fresh instead. let dio = lens.dio(rooms).live().build().await?; let mut events = dio.subscribe_events(); tokio::spawn(async move { while let Ok(event) = events.recv().await { if let DioEvent::RefreshFailed { error, .. } = event { eprintln!("rooms may be stale: {error}"); } } }); # assert_eq!(dio.cached().list_values().await?.len(), 1); # Ok(()) # } ``` Without a refresher a Lens has no timer and no staleness check; the cache holds what the loader and the writes put there. ### Why not refresh on every open? Because for a persistent cache, "on every open" is "on every screen change". A grid that the user leaves and comes back to would re-list its whole set each time, which on a slow REST source is the latency the cache was there to hide. `max_age` keeps the refresh where it pays: after an absence long enough that the data has plausibly moved. ## Changes you feed in yourself A source with its own change feed that no driver wraps — a webhook, a message queue, a file watcher — feeds the Dio directly. Two calls cover it: - `Dio::apply_change(ChangeEvent)` applies one change: `Inserted` and `Updated` upsert the row, `Deleted` removes it, and `Invalidated` re-reads the whole set. Each publishes the matching `DioEvent`, so open sceneries repaint. - `Dio::reconcile(snapshot)` makes the cache hold exactly a full snapshot: rows absent from it are deleted, except one with a write in flight, and one `DatasetChanged` is published. ```rust # use ciborium::Value as CborValue; # use vantage_dataset::ReadableValueSet; # use vantage_diorama::{Cache, ChangeEvent, Lens}; # use vantage_types::Record; # use vantage_vista::mocks::MockShell; # use vantage_vista::{Column, Vista, VistaMetadata}; # # fn room(name: &str) -> Record { # let mut r = Record::new(); # r.insert("name".to_string(), CborValue::Text(name.to_string())); # r # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let rooms = Vista::new( # "rooms", # MockShell::new().with_metadata( # VistaMetadata::new() # .with_column(Column::new("id", "String").with_flag("id")) # .with_column(Column::new("name", "String")) # .with_id_column("id"), # ), # ); # let lens = Lens::builder().cache(Cache::memory()).build()?; let dio = lens.dio(rooms).build().await?; dio.apply_change(ChangeEvent::Inserted { id: "r1".into(), new: Some(room("Lobby")) }).await?; dio.apply_change(ChangeEvent::Updated { id: "r1".into(), new: Some(room("Atrium")) }).await?; let snapshot = [("r2".to_string(), room("Kitchen"))].into_iter().collect(); dio.reconcile(snapshot).await?; let cached = dio.cached().list_values().await?; assert!(cached.len() == 1 && cached.contains_key("r2")); # Ok(()) # } ``` A `None` row on `Inserted` or `Updated` publishes the event without touching the cache: the notify-only shape a coarse feed produces. A refresher's or loader's own re-read goes through `LoaderCx::reconcile` instead, which also leaves alone any row whose write settled while the read was in flight. ## SpacetimeDB — filtered fine push, no setup SpacetimeDB is a database that is also the application server: a WASM module holding the schema and the reducers runs inside it, and the client's normal way to read anything is to subscribe. Push is not a feature bolted on beside queries; it is the product. The driver sends the Vista's own conditions as the subscription query, so the server evaluates them and only matching rows are ever transmitted. Because it maintains that result incrementally, a row that changes so it *no longer* matches arrives as a delete — you are told the row left, not merely that something happened. That is why this is the only row in the table where set membership needs no reconciliation from anyone: not a re-read per notification as SurrealDB does, and not a whole-set reload as Postgres does. Nothing needs installing or enabling. There is no trigger, no extension, no opt-in flag — which is why its shells simply advertise `Subscribe` rather than something you have to declare. ```admonish note title="This backend gives you less everywhere else" The trade is real and worth stating. SpacetimeDB's SQL has no `ORDER BY`, `OFFSET`, `GROUP BY`, `IN` or `LIKE`, and the driver does not emulate them — a grid over a SpacetimeDB table shows rows in the server's natural order and its sort headers are inert. Aggregates for a chart have to be materialised by the module into a rollup table or computed in a view. You are choosing a backend that pushes beautifully and queries narrowly. ``` ## SurrealDB — fine push, no setup `LIVE SELECT` delivers a typed per-row notification: the action (`CREATE` / `UPDATE` / `DELETE`) and the affected record. The driver re-reads each notified id *through the vista's own conditions*, so a row that no longer belongs to a filtered set arrives as a delete rather than silently lingering. A real table is watchable the moment you build it; only query-sourced vistas (`rhai:` / `base:`) are not. ```admonish note title="Live queries ride the WebSocket connection" `LIVE SELECT` needs a transport the server can push frames down, which means WebSocket. That is not a trap you can fall into: `SurrealConnection::connect` accepts only `ws://`, `wss://` and `cbor://` and rejects anything else outright, so every SurrealDB connection you can actually open is one that can carry live queries. ``` ## PostgreSQL — coarse push, and why you must ask for it Postgres has no built-in change feed. It has `LISTEN/NOTIFY`, a general-purpose pub/sub channel, and a trigger you install yourself decides what gets announced. The driver listens on `{table}_changed` and yields a payload-less invalidation for every notification. Because you install that trigger, only you know whether it exists — and `LISTEN` on a channel nobody ever feeds *succeeds*, then blocks forever. A Vista that assumed it could push merely because the table is writable would advertise a feed that may never arrive, which a consumer cannot tell apart from a table where nothing happens. So it is opt-in. The block needs a running Postgres, so it is compiled but not run: ```rust,no_run # use vantage_sql::postgres::{AnyPostgresType, PostgresDB}; # use vantage_table::table::Table; # use vantage_types::entity; # use vantage_core::Context; # # #[entity(PostgresType)] # #[derive(Debug, Clone, Default)] # pub struct Product { # pub name: String, # } # # impl Product { # pub fn table(db: PostgresDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # } # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = PostgresDB::connect("postgres://localhost/bakery").await.context("connect")?; let master = db .vista_factory() .with_notify(true) // yes, I installed the trigger .from_table(Product::table(db.clone()))?; # let _ = master; # Ok(()) # } ``` The trigger itself, and the full walkthrough, are in [Real-Time Push with LISTEN/NOTIFY](./intro/step8-sql-notify.md). The declaration applies to every table a factory builds, so if only some of your tables carry triggers, use separate factories. It also carries across reference traversal — a relation target inherits the parent's opt-in rather than silently reverting to unwatchable. ## What a consumer must assume Push is best-effort, and the contract is deliberately loose so that a driver can tighten its scope later without breaking anyone. Callers always hand over the full Vista; the driver delivers what it can. Four things hold everywhere: - **The stream may be coarser than your Vista.** Watching the whole table and letting the consumer discard the rest is a legitimate implementation. - **A pushed row may fall outside your conditions**, precisely because of the above. Either the driver reconciles or you must. A live Dio holds each change against its own query, and applies a row that falls outside it as a delete. - **An invalidation means "re-read everything."** It carries no id and implies nothing about how much changed. - **The stream ending is normal.** Connections drop; consumers resubscribe with backoff and reconcile the gap. A `live()` Dio already does this: it re-subscribes, re-reads the set, and publishes `DioEvent::RefreshFailed` for each re-read that fails until one succeeds. The sceneries keep the rows they show meanwhile. The consequence worth internalising: `Capability::Subscribe` advertises an *attempt* to push, not a guarantee of delivery. No shipped backend is lossless, so anything that must not miss a change keeps a reconcile path — a refresher with `every`, a refresh on reconnect — rather than trusting push alone. ## Adding push to a driver If you are implementing a backend, the driver-side contract — what you must promise when you implement `watch`, and when you may honestly advertise `Subscribe` — is step 8 of [Adding a New Persistence](./new-persistence/step8-vista-integration.md). Start without the capability; turning it on the day the feed works is a healthier state than shipping a flag that lies. ## Checklist To keep a Dio fresh: 1. **Know your backend's model** — the table above; push needs `Capability::Subscribe` on the master. 2. **Ask for push** — `lens.dio(master).live().build()`. Over a polled master it does nothing. 3. **Give the Lens a refresher** — `Refresher::new(..)` with `max_age` for staleness on open, `every` for a timer, or both. 4. **Feed changes no driver wraps** — `dio.apply_change(..)` per change, `dio.reconcile(..)` for a full snapshot. 5. **Watch for `DioEvent::RefreshFailed`** — the cache may be out of step until a retry succeeds. --- Source: https://vantage-ui.com/book/optimising-diorama.html # Optimising Live Diorama Data An application reports its faults. It does not report its inefficiencies. A failed request raises an error, but a duplicate fetch stays unnoticed. Typical problems are: - refetch a static page in its entirety every 5 seconds - fetch all columns of a related table, just to fill a select field in a form - sort 120,000 rows in the client, and ignore the `sort` capability of the source Inefficiencies show as extra latency, high CPU usage, or network traffic. This chapter gives you the mechanisms to detect them and to avoid them. ```admonish info title="Why live data needs a cache" A user interface repaints many times each second. It cannot wait for the network on each frame, so the rows must already be in memory. When the user scrolls, the framework fetches rows ahead of the viewport to keep the movement smooth. Diorama keeps those rows in a [redb](https://www.redb.org/) file on disk, one file for each datasource. The rows therefore survive a move to another page, and they survive a restart. A user interface is not the only consumer of a cache. Vantage is also designed for facade APIs and for live caches at the edge. The goal is the same in each case: answer most requests without a request to the master source, and keep the local copy in agreement with that source. A correct cache answers immediately, and then pushes each change to the viewport when it arrives. UI viewport Facade API Edge cache Diorama cache redb, on disk Master source slow · remote declare viewport relevant live updates few fetches, batched whole-table changes ``` ## What slows the app down? | Inefficiency | What happens | The damage | | --- | --- | --- | | **Repeated fetch** | The application asks for a window that it already asked for. | One request for each timer tick or repaint | | **Redundant rows** | Rows arrive that the cache already holds. | Bandwidth, with no change on screen | | **Off-screen updates** | The source announces a change to a row that the user cannot see, and the view repaints. | CPU usage for data that nobody looks at | | **Undeclared columns** | Each row carries fields that no table lists. | Bandwidth. A grid of five columns can cost megabytes for each screen. | | **Work in the client** | The client sorts, searches, or counts locally. | CPU usage, and an answer that covers the loaded rows only | | **A full copy of a large table** | The source cannot serve windows, so the client reads all the rows at open. | Memory, and a wait before the first screen | | **A total that no fetch can fill** | The source states more rows than it serves. | Requests for the missing rows, for as long as the page stays open | There is no best caching strategy. Each use case needs a different approach. Vantage lets you design and implement your own strategy, but you must first understand how the cache behaves and how to monitor it. Vantage UI works well with its default behaviour. This chapter starts from that default, and then adjusts it. --- ## Overview | Section | What you do | What it answers | | --- | --- | --- | | [1. The setup](./optimising-diorama/stack.md) | Build the `optimising` project and open it in Vantage UI | How do I put a source that I control under a real client? | | [2. Reading the Debug Stream](./optimising-diorama/debug-stream.md) | Make the source slow, empty the cache, and switch on debug for one datasource | What does my application ask for? | | [3. Where the Work Happens](./optimising-diorama/capabilities.md) | Remove capabilities and find where the work moves | Which work moved into my process, and what did it cost? | | [4. Fetching What You Drop](./optimising-diorama/waste.md) | Find repeated fetches, redundant rows, and columns that no view shows | How many of these bytes did anybody need? | | [5. Faults: Errors, Outages & Lies](./optimising-diorama/faults.md) | Inject failures, outages, and dishonest totals | Does the application continue, or does it stop? | | [6. Locking It In](./optimising-diorama/regression.md) | Make assertions from your measurements, over a run with a fixed seed | Will I know when this becomes worse? | **Start here:** [The setup](./optimising-diorama/stack.md) --- Source: https://vantage-ui.com/book/optimising-diorama/stack.html # The setup An enterprise source holds millions of records, and there can be hundreds of changes each second. The change channels are coarse: one channel for each table. But no user observes millions of records. A user looks at 20 or 30 rows, and possibly at some aggregated values. A Vantage client makes the distinction. Vantage UI connects to the datasource and then narrows the data to a **viewport**, which is the set of rows on screen. A live cache at the edge follows the same principle when it drives a frontend application: it holds a fragment of the table, and it forwards the changes that touch that fragment. The cache is the same cache in both cases, so what you learn here against Vantage UI transfers to the other components. To measure a cache strategy you need three things: - a source whose behaviour you control: fast, slow, unreliable, or fast-changing - a consumer that you can navigate by hand and check by eye - full transparency into the cache mechanics, through a real-time log This chapter uses a **Vantage Faker** datasource, Vantage UI, and the full cache log. ## Create a project Vantage UI is a free download from [vantage-ui.com](https://vantage-ui.com). It installs as a normal macOS application. Start it normally, create a new project, and choose **Fake Data** in the wizard. Then find the project folder — this chapter calls it `optimising` — and examine the structure inside: ```text optimising/ ├── datasource/people.yaml where the rows come from ├── table/people.yaml the columns, and which datasource supplies them ├── page/people.yaml what the screen shows └── menu/left.yaml how to open the page ``` A project is a folder of YAML, and each folder holds one kind of declaration. Change the four files to match the listings below. The datasource is a **faker** source. It generates rows instead of fetching them, and you declare how it behaves. This one holds 200,000 personal records: names, surnames, emails, and some additional data. ```yaml # datasource/people.yaml type: faker effect: static count: 200000 seed: 103 shape: capabilities: [count, fetch_window, order, search] latency: window: 10ms..40ms get: 5ms..20ms ``` ```admonish note title="The capabilities list" `capabilities` is the contract of the source. A real driver declares what it can do, and faker declares what you tell it to. The list above mimics a SQL source: - **`count`** — the source reports how many rows the table holds. The grid then knows the size of the scrollbar before it has any row. - **`fetch_window`** — the source serves a range, such as rows 500 to 600. Without it, the only operation is "read everything". - **`order`** — the source sorts. A sort then applies to all the rows, and not to the loaded rows. - **`search`** — the source filters, with the same condition. The source refuses each operation that is not in the list, and the work then moves to the client. Section 3 removes them one at a time. [Vista — the Universal Data Handle](../intro/step4-vista.md) describes the full set of capability flags and how a driver declares them. The names here are Vantage UI's. In Rust, `fetch_window` is `Capability::Window`, and `count` has no capability at all, because every Vista can count. ``` Once the datasource exists, declare the tables on it. Each table names its columns and the datasource that supplies them: ```yaml # table/people.yaml datasource: people title: People columns: - { name: id, type: string, flags: [id] } - { name: name, type: string, flags: [title, searchable] } - { name: surname, type: string, flags: [title, searchable] } - { name: email, type: string } - { name: city, type: string } - { name: balance, type: decimal } ``` Each column name is significant. `name` generates human names, `surname` generates surnames, and `email` generates email addresses. Faker uses the column name first and the declared type second, so a name that faker does not recognise gets a value from its type. Every value is fake, and faker generates all of them at start-up from the `seed`. Use realistic names. Much of this chapter measures size, and a table of `row_1` and `row_2` values makes each of those measurements wrong. ```admonish note title="Use one faker datasource for each table" A table file follows the Vantage YAML specification, which is the same for every datasource type. It declares columns, types, and flags, and nothing more. A datasource file has no such restriction, so each faker knob is there: `count`, `seed`, `effect`, and the full `shape:` block. One faker datasource therefore describes one set of rows. Give each table its own datasource. This chapter builds four: `people`, `stubborn`, `flaky`, and `wide`, one for each behaviour that it measures. ``` The page shows a **binder**. A binder appears as a full CRUD screen: it lists the rows, it edits them when the source supports editing, it has a quick-search box and column ordering, and a click on a row opens a summary of that row. Behind the screen it keeps its viewport equal to the rows that you see. Vantage UI also has a `kind: grid` component. A grid has no search box, so this chapter uses a binder. ```yaml # page/people.yaml title: People body: - kind: binder name: people observe: people params: columns: name: { width: 160 } surname: { width: 160 } email: { width: 220 } city: { width: 180 } balance: { width: 120 } ``` The menu makes the page available: ```yaml # menu/left.yaml title: Optimising items: - section: Optimising children: - page: people label: People icon: Users ``` ## Run it Each later section starts Vantage UI again from the command line, and not from the Finder. The binary in the bundle accepts arguments, and the debug output goes to stdout. Make an alias: ```sh alias vantage-ui='/Applications/Vantage.app/Contents/MacOS/vantage-ui' ``` ```sh vantage-ui ./optimising --page=people ``` ![The binder over 200,000 generated people: the row list, the quick-search box, the row count, and the details panel for the selected row](../images/optimising-people-grid.png) The binder lists the rows, and a click on a row fills the details panel below it. The row count says 200000, because the source counted the set. Scrolling is smooth, and no row stays empty for long enough to see. The first argument is the project folder. `--page=` opens that page instead of the first entry in the menu. This matters: a page that you did not intend to measure also opens a Dio, and its requests go into the same statistics. ```admonish note title="Keep the project simple" Vantage UI builds one Dio for each observed table, when a page that observes it opens. The other YAML files in the folder cost nothing until you open them. Two conditions make a measurement unclear: 1. One page with components that observe different tables. Each component has its own Dio and sends its own requests. 2. Tabs that stay open behind the page that you measure. Each open tab keeps its Dios active. Use one table, one binder, and one variable. Close the pages that you do not measure. ``` ````admonish info title="Using faker in Rust" This chapter uses Vantage UI, because Vantage UI supplies the viewport. The faker source does not need Vantage UI. `vantage-faker` is a normal crate: a `DatasetGen` fills a vantage-memory store, and the `shape:` block above is a [`BackendShape`](https://docs.rs/vantage-faker/latest/vantage_faker/shape/struct.BackendShape.html) that a `ShapedShell` enforces over the memory table. This code builds the same datasource: ```rust,no_run # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { use std::time::Duration; use vantage_dataset::prelude::ReadableValueSet; use vantage_diorama::{Cache, Lens, Loader}; use vantage_faker::{BackendShape, DatasetGen, FakerColumn, Latency, LatencyModel, ShapedShell, TableGen}; use vantage_memory::{MemoryCatalog, MemoryStore, MemoryTableShell}; use vantage_vista::{Capability, Column, ColumnFlag, Vista, VistaMetadata}; // The rows: 200,000 people from seed 103. Faker fills the `id` column itself. let store = MemoryStore::new(); let people = TableGen::new("people") .column(FakerColumn::new("name", "string")) .column(FakerColumn::new("surname", "string")) .column(FakerColumn::new("email", "string")) .column(FakerColumn::new("city", "string")) .column(FakerColumn::new("balance", "decimal")) .count(200_000); DatasetGen::new(Some(103)).table(people).generate(&store)?; // The table file: columns and flags. let column = |name: &str, ty: &str| Column::new(name, ty).with_flag(ColumnFlag::Orderable); let metadata = VistaMetadata::new() .with_column(Column::new("id", "string").with_flag(ColumnFlag::Id)) .with_column(column("name", "string").with_flag(ColumnFlag::Title).with_flag(ColumnFlag::Searchable)) .with_column(column("surname", "string").with_flag(ColumnFlag::Title).with_flag(ColumnFlag::Searchable)) .with_column(column("email", "string")) .with_column(column("city", "string")) .with_column(column("balance", "decimal")) .with_id_column("id"); // The `shape:` block. `count` has no capability in Rust: every Vista counts. let shape = BackendShape { capabilities: [Capability::Window, Capability::Order, Capability::Search] .into_iter() .collect(), latency: LatencyModel { window: Some(Latency::between(Duration::from_millis(10), Duration::from_millis(40))), get: Some(Latency::between(Duration::from_millis(5), Duration::from_millis(20))), ..LatencyModel::default() }, seed: Some(103), ..BackendShape::default() }; let shell = MemoryTableShell::new(store.table("people"), metadata, MemoryCatalog::new(store.clone())); let vista = Vista::new("people", shell).map_shell(|inner| ShapedShell::new(inner, shape)); // From here, this is the normal reactive stack. Nothing is faker-specific. // Page a master that serves windows; copy one that doesn't. let loader = Loader::paged(|cx, range, query, sink| async move { let fetched = cx.fetch(range.clone(), &query).await?; if !fetched.refused.is_empty() { // A window can't be sorted or searched after the fact: fail the load. return Err(vantage_core::error!("the source refused part of the view's query")); } let page = fetched.page; if let Some(total) = page.total { sink.set_total(total.max(0) as usize); } for (i, (id, row)) in page.rows.into_iter().enumerate() { sink.push(range.start + i, id, row).await?; } Ok(()) }) .or_eager(|cx| async move { let rows = cx.master().list_values().await?; cx.reconcile(rows).await }); let lens = Lens::builder().cache(Cache::memory()).loader(loader).build()?; let dio = lens.dio(vista).build().await?; # let _ = dio; # Ok(()) # } ``` Use this code when your own code is the consumer: an integration test that calls `set_viewport`, a CLI, or a scenery in a terminal. `ShapedShell` only takes capabilities away: it advertises the ones the shape lists and the memory table has, and refuses the rest. ```` ## Switch on the debug stream Delete the cache and start Vantage UI again. Each measurement in this chapter starts from an empty cache, because a full cache serves the first screen and asks the source for nothing. ```sh rm -rf ~/Library/Caches/Vantage/optimising-* vantage-ui ./optimising --page=people ``` Without the debug flag, the output describes the start-up and nothing else: ```text INFO vantage_ui: starting root="…/optimising" has_project=true INFO catalog: loaded datasource key="people" INFO catalog: loaded table key="people" INFO catalog: loaded framework page key="people" INFO vantage_ui: Dio cache dir path="~/Library/Caches/Vantage/optimising-c79233058eb9f98f" INFO vantage_ui: configured faker datasource datasource=people effect=Some("static") INFO vantage_ui: entity open, by phase key=people/people ms=2891 breakdown=vista=2798ms cache=82ms dio=2ms scenery=7ms ``` This tells you what loaded. The last line is the only measurement: 2.8 seconds inside the Vista, which is faker generating 200,000 rows at start-up. Nothing here reports a request, a cache hit, or a row. Now add one line to the datasource: ```yaml # datasource/people.yaml type: faker effect: static debug: true # ← add this count: 200000 ``` Delete the cache and start it again: ```sh rm -rf ~/Library/Caches/Vantage/optimising-* vantage-ui ./optimising --page=people ``` A second stream now appears beside the log: ```text 11:53:45.591 people dio created over "people" — cache table "people_people", nothing held yet 11:53:45.592 people source can count, window, order, search · pages lazily, 100 rows at a time 11:53:45.592 people scenery opened cold — 0 rows seeded from cache 11:53:45.593 people census +1 table scenery — now 1 table, 0 record, 0 servo · rss 310MB 11:53:45.645 people viewport rows 0..200 — 2 scroll events coalesced into one 11:53:45.645 people dio fetch #1 asks for rows 0..200 — 200 missing, 0 already held 11:53:45.715 people cache +200 new, 0 updated — now holding 200 of 200,000 (0.1%) 11:53:45.715 people scenery opening → partial (chunk load succeeded) 11:53:45.716 people dio fetch #1 got 200 rows in 70ms 11:53:45.716 people payload 6 columns received · no view declared what it needs, so all were fetched · 23KB · id,name,surname,email,city,balance 11:53:45.717 people total 200,000 rows — stated by the source in the same response 11:53:45.821 people cache all 22 rows of 0..22 served locally — no fetch ``` That is one page opening, from an empty cache. The binder asked for 200 rows, one request supplied them, and the last line is the repaint that followed: 22 rows on screen, no request. The flag applies to one datasource. It is not a global setting. Your other datasources stay silent, so the question is always "what does **this** table do?". In Rust the same switch is on the Lens: [`LensBuilder::debug_datasource(name)`](vantage_diorama::LensBuilder::debug_datasource), behind vantage-diorama's `diagnostics` feature. Every Dio that Lens builds writes the stream to the `vantage_diorama::debug` tracing target, stamped with `name`. ```admonish tip title="What is next" The next section makes the source slow, so that each of these lines separates in time. It then reads the stream line by line. ``` --- Source: https://vantage-ui.com/book/optimising-diorama/debug-stream.html # Reading the Debug Stream The first thing that you learn from the stream is which line agrees with which moment on screen. Keep the window and the terminal beside each other. Sort a column, type in the quick-search box, and scroll, and watch which lines each action produces. That is much easier when each request needs three seconds than when it needs 40 milliseconds. ## Make the source slow A source that answers in 40ms hides its behaviour. The rows are simply present, and a load is one flicker between two frames. Change one value in the datasource. Each fetch then becomes an event with a start and an end. ```yaml # datasource/people.yaml shape: capabilities: [count, fetch_window, order, search] latency: window: 3s # each batch of rows costs three seconds get: 500ms..1s # each single record costs half a second or more ``` The two classes pay for different requests: | Latency class | The request that pays it | What causes it on screen | | --- | --- | --- | | `window` | A range of rows, such as rows 500 to 600 | The grid scrolls into rows that the cache does not hold | | `get` | One record, by its id | A click on a row, which opens the details panel | `window` is the class that this chapter measures. A grid asks for ranges, and each unnecessary range is a fetch that you can count. Give `get` a value as well, so that a click on a row is also visible in the stream. Nothing else changes: the same table, the same page, and the same 200,000 rows. But each decision of the framework now takes clock time that you can see. ![The grid during a load: the total is known, so the rows exist and the scrollbar has the correct size. The contents have not arrived.](../images/optimising-people-loading.png) Open the page and watch it load: 1. The window opens immediately, and the page then needs a few seconds to mount. That delay is the fixture, not the framework: `count: 200000` makes faker generate 200,000 rows at each start. Decrease `count` if you do not want it. 2. **Skeleton rows** appear. A skeleton row has a position, but no contents. The grid gave a position to each of the 200,000 rows, because the source declared `count`, so the scrollbar has the correct size before any row arrives. 3. The first batch arrives three seconds later, and those rows paint together. 4. Scroll past them. The same sequence occurs again. Now scroll back to the top. **Nothing happens.** There is no delay and there are no skeleton rows. That is the cache, and it is your first measurement: to return to a position that you visited costs nothing. ## Cold runs and warm runs There is one cache file for each datasource, under `~/Library/Caches/Vantage/-/`. Delete the folder for a cold run: ```sh rm -rf ~/Library/Caches/Vantage/optimising-* ``` A cold run and a warm run are two different experiments. A cold run answers "what does the first open cost?". A warm run starts from a full cache and answers "what does the application do with rows that it already holds?". Too many refreshes appear in the warm run. The two sets of numbers are not comparable. Record which run you did, each time. ## Understanding the log The debug stream shares the terminal with the ordinary log of the application: ```text 12:23:58.222 people dio created over "people" — cache table "people_people", 422 rows already held from a previous session 2026-07-31T11:23:58.242796Z INFO build_page_for_key{page_key="people" …}:build_entity{key="people" datasource=Some("people")}: vantage_ui: entity open, by phase key=people/people ms=2791 breakdown=vista=2722ms cache=49ms dio=0ms scenery=18ms 12:23:58.341 people dio fetch #1 asks for rows 0..22 — 0 missing, 22 already held 12:23:58.420 people dio fetch #1 got 22 rows in 79ms ``` The short lines are the debug stream. Each one has four parts, always in this sequence: | Part | Example | What it is | | --- | --- | --- | | time | `12:23:58.341` | Local time, to the millisecond | | table | `people` | The datasource with debug switched on. With one datasource for each table, this is also the table. | | origin | `dio` | The part of the framework that wrote the line | | message | `fetch #1 asks for rows 0..22 …` | One sentence, in units that a person reads: `3.0s`, `24KB`, `200,000`, `0.1%` | Each request has an identifier, such as `fetch #1`. The identifier connects a request to its result. There is one sequence for each Dio, and it starts at 1. ```admonish note title="The sequence of lines in one load is not a contract" The lines from one load do not always appear in the same sequence. The framework writes the cache line and the state line *inside* the operation that the result line completes. Both can therefore come before the result line. Use `fetch #N` to connect the lines. Do not use their positions. ``` The origin is what you search for. There are fourteen origins: | Origin | What the line reports | | --- | --- | | `dio` | The framework created a Dio. Or a fetch **asks for** a range, **got** rows, or **failed**. | | `source` | What the source can do and cannot do, and which load method follows. One line, at open. | | `scenery` | A view opened with a full or an empty cache. Or the load state changed. | | `census` | A consumer attached or detached. Consumers are table sceneries, record sceneries, and servos, which watch one value. The line gives the current counts and the memory. | | `viewport` | The range that a consumer declared. **Coalesced** means that several scroll events became one range. | | `cache` | The framework wrote rows. Or it **served** a viewport **locally**, with no fetch. | | `payload` | The columns that arrived, the columns that a view wants, and the size in bytes. | | `total` | The total count changed, and what supplied the new value. | | `sort` | A sort changed. Says whether the source does the work. | | `search` | A search changed. Says whether the source does the work. | | `filter` | The runtime filters changed. Says whether the source does the work. | | `derive` | An aggregate recomputed. The first line of a mount says **initial**: the computation that makes the schema of the derived table. After it, the aggregate waits for the source to change. | | `hydrate` | A second pass fetched the remaining columns of rows that are on screen. Section 4 explains the two passes. | | `summary` | The session summary, printed when Vantage UI quits. | ## Read one origin at a time Write the session to a file, then read one origin: ```sh vantage-ui ./optimising --page=people 2>&1 | tee /tmp/optimising.log grep ' dio ' /tmp/optimising.log ``` ```text 12:23:58.222 people dio created over "people" — cache table "people_people", 422 rows already held from a previous session 12:23:58.341 people dio fetch #1 asks for rows 0..22 — 0 missing, 22 already held 12:23:58.420 people dio fetch #1 got 22 rows in 79ms 12:24:11.295 people dio fetch #2 asks for rows 316..339 — 23 missing, 0 already held · sorted by city Desc 12:24:16.842 people dio fetch #2 got 23 rows in 5.5s ⚠ slow ``` Five lines report a whole session: - The cache was **warm**: 422 rows from an earlier run. A cold start says *nothing held yet* here. - `fetch #1` asked for 22 rows that the cache already held. A warm start still asks the source for the window on screen, to check that the held rows are current. - Between `12:23:58.420` and `12:24:11.295` there is no line. Thirteen seconds of use cost no request. - `fetch #2` carries `sorted by city Desc`. The sort went to the source, and the source applied it to all 200,000 rows, and not to the rows that the cache held. - That sorted request cost 5.5s. **⚠ slow** marks each request of one second or more. Read the other origins in the same way: ```sh grep ' cache ' /tmp/optimising.log # what the cache wrote, and what it served grep 'served locally' /tmp/optimising.log # the viewports that cost nothing grep -E 'asks for|got ' /tmp/optimising.log # the network, and nothing else ``` Scroll while you watch `tail -f` on the file. Each three-second delay must have one `asks for` line and one `got` line. Each immediate repaint must have a `served locally` line, or no line. **A delay with no request is a fault in the UI. A request with no delay is a request that you did not need.** ## A full cold open This is the output of the `optimising` project, with high latency and an empty cache, when the People page opens. It is a capture, not an example. ```text 21:20:15.397 people dio created over "people" — cache table "people_people", nothing held yet 21:20:15.399 people source can count, window, order, search · pages lazily, 100 rows at a time 21:20:15.401 people scenery opened cold — 0 rows seeded from cache 21:20:15.401 people census +1 table scenery — now 1 table, 0 record, 0 servo · rss 313MB 21:20:15.455 people viewport rows 0..200 — 2 scroll events coalesced into one 21:20:15.455 people dio fetch #1 asks for rows 0..200 — 200 missing, 0 already held 21:20:18.493 people cache +200 new, 0 updated — now holding 200 of 200,000 (0.1%) 21:20:18.494 people scenery opening → partial (chunk load succeeded) 21:20:18.494 people dio fetch #1 got 200 rows in 3.0s ⚠ slow 21:20:18.494 people payload 6 columns received · no view declared what it needs, so all were fetched · 23KB · id,name,surname,email,city,balance 21:20:18.494 people total 200,000 rows — stated by the source in the same response 21:20:18.551 people cache all 13 rows of 0..13 served locally — no fetch ``` The framework creates the Dio before any view can use it, and its cache table is empty. That is the cold start. The `source` line gives the contract that each following line obeys. The binder declares a viewport of 200 rows, and not the 13 rows that you can see. The loader reads ahead of the screen, so that the next scroll costs nothing. Section 4 measures this. The `source` line says 100 rows for each page, and the first fetch asks for 200. The first fetch covers the opening viewport, which the binder sets to 200 rows. Later fetches use the page size of 100. Then the delay: from `.455` to `18.493`, three seconds. Four facts then arrive together: - the cache holds the rows — 200 of 200,000, which is 0.1% - the view is no longer in the loading state - the fetch reports its cost - the total is known, because the source supplied it in the same response The last line is the result: the grid painted 13 rows and made no fetch. Now scroll. Most of the output is that same line: ```text 21:20:20.401 people viewport rows 29..43 — 5 scroll events coalesced into one 21:20:20.401 people cache all 14 rows of 29..43 served locally — no fetch ``` ## The summary at the end When you quit, the framework prints the summary of the session: ```text 21:20:33.374 — diorama session summary — 21:20:33.374 people summary 3 fetches, 0 repeats · 400 rows, 0 redundant · 9.1s waiting, 3.0s slowest 21:20:33.374 summary 1 dio, 1 table scenery, 0 record, 0 servo still alive 21:20:33.374 summary session 18.0s · cpu 15.0s · peak rss 352MB ``` Two numbers in the first line are important. The other sections keep both at zero. - **`repeats`** is the count of ranges that the application fetched more than one time. - **`redundant`** is the count of rows that arrived when the cache already held them. A warm revalidation gives a small number. A number equal to the count of rows that arrived means that the application fetched a full page for nothing. The second line finds leaks. Close a page, and the counts must decrease. If the count of sceneries increases while you move between pages, something keeps views alive after their page closed. --- ## What we covered | Item | What it does | | --- | --- | | `latency.window: 3s` | Separates the events, so that you can attribute each line to a moment | | `~/Library/Caches/Vantage/-/` | The location of the cache. Delete it for a cold run. | | `debug: true` | Switches the stream on for one datasource. The others stay silent. In Rust: `LensBuilder::debug_datasource(name)` | | `vantage_diorama::debug` | The one target, at `info`. It appears without `RUST_LOG`. | | time · datasource · tag · clause | The four parts of each line. `fetch #N` connects a request to its result. | | `served locally — no fetch` | The line that reports a viewport that cost nothing | | `repeats` / `redundant` | The two waste counters in the session summary | ```admonish tip title="What is next" The next section removes the capabilities of the source, one at a time, and finds where the work goes. A sort that the source cannot do does not stop. It moves to a more expensive place, and it uses only the rows that the application loaded. ``` --- Source: https://vantage-ui.com/book/optimising-diorama/capabilities.html # Where the Work Happens: Capabilities & Latency Until now the source was capable. When the source can do everything, the layers above it do nothing that you can measure. This section removes the capabilities. **Work does not disappear when a source cannot do it. The work moves.** ## One line controls everything Each session starts with this line: ```text people source can count, window, order, search · pages lazily, 100 rows at a time ``` That line is the full contract. The `capabilities:` list in the datasource produces it, and each later decision follows from it. Change that line, and you change the behaviour of the application. You change no other file. ## A source that can do nothing Remove all capabilities. One operation remains, because a source always supports `list`. ```yaml # datasource/stubborn.yaml type: faker effect: static debug: true count: 3000 seed: 109 shape: capabilities: [] # no count, no order, no search, no windows latency: list: 2500ms..3500ms # and the one operation that it has is slow ``` Open the page. The stream is very short: ```text 22:02:51.204 stubborn dio created over "stubborn" — cache table "stubborn_stubborn", nothing held yet 22:02:51.205 stubborn source can do nothing but list · cannot count, order, search · copied whole into the cache 22:02:51.205 stubborn scenery opened cold — 0 rows seeded from cache 22:02:51.206 stubborn census +1 table scenery — now 1 table, 0 record, 0 servo · rss 129MB 22:02:54.265 stubborn scenery opening → complete (reactor reseed) 22:02:54.337 stubborn cache all 22 rows of 0..22 served locally — no fetch ``` **There are no fetch lines.** The `people` source sends one request for each screen of rows. This source receives one read only — the three seconds between the census line and the `opening → complete` line. Each later scroll comes from the cache, because the cache holds all the rows. Diorama has two load methods, and the `source` line names the one in use. | Method | Log string | When the framework uses it | | --- | --- | --- | | **paged** | `pages lazily` | The source can serve `[offset, limit)` windows | | **eager** | `copied whole into the cache` | The source cannot serve windows | In Vantage UI you do not select the method. It follows from what the source can do. In Rust the choice is the Lens's loader: [`Loader::paged`](vantage_diorama::Loader::paged) fetches windows, [`Loader::eager`](vantage_diorama::Loader::eager) copies the set, and `Loader::paged(..).or_eager(..)` decides per Dio, paging a master that advertises `Window` and copying one that doesn't. That is the rule this section measures. | | **paged** | **eager** | | --- | --- | --- | | Reads | one window for each screen, on demand | one full read at open | | Memory | proportional to the rows that you looked at | proportional to the whole table | | Scrolling | can cost a request | costs nothing | | A large table | stays cheap | is a large read | Note the line `opening → complete`, and not `→ partial`. A paged view stays *partial* for most of its life: it holds 200 of 200,000 rows, and it knows this. An eager view goes directly to complete, because after the copy there is nothing more to get. ```admonish note title="Eager is correct for a small table, not for a large one" This source holds 3,000 rows. The same shape with 200,000 rows would read all of them at open, and the user would wait. If your source cannot serve windows and your table is large, do not look for a Diorama setting. Add windows to the source, or observe fewer rows. ``` ## The cost of a sort, and its location Sort the windowed table. The stream tells you where the work goes: ```text people sort by name descending — pushed to the source ``` The next fetch then includes the sort: ```text people dio fetch #3 asks for rows 521..621 (viewport 521..534) — 100 missing, 0 already held · sorted by name Desc people dio fetch #3 got 100 rows in 7.3s ⚠ slow ``` The framework did not sort the rows that it holds. It sent the sort to the source, because the source can order. When you hold 300 rows of 200,000, only the answer of the source is correct. The request also became more expensive: 7.3s for 100 rows, against 3.0s for 200 rows. The source must now order 200,000 rows for each window. The same action against the source that can do nothing gives a different line: ```text stubborn sort by name descending — the source cannot order; sorting the 3,000 rows held locally ``` No fetch follows. The cache holds all the rows, so the sort is a local operation. It is immediate, and it is correct because the cache holds the complete set. **The dangerous combination is a source that cannot order and a table that is too large to copy.** The framework would then have to sort the rows that it loaded, and "sorted" would mean "the first page of a sample". Diorama does not do this. A paged view whose source orders the rows discards its cached row positions instead of moving them, and it fetches the visible window again: ```text people scenery row positions dropped — the source re-orders, so cached rows must be re-placed ``` The cost is one more fetch. The result is correct. ## Search, and the control that the UI hides Search obeys the same rule: ```text people search "smith" — pushed to the source ``` A source that cannot search gives the opposite line, and the framework filters the rows that it holds. You see that line from the `stubborn` source, which holds all its rows. The UI adds one condition. **A grid over a source that cannot search shows no search box.** The UI hides the control instead of showing a control with a smaller scope than the user expects. ```admonish tip title="A grid has no search control" A `grid` component shows no search box, and this does not depend on the source. Use `kind: binder` for the full table controls. A grid over a source that can search therefore offers less than the source can do. ``` ## Latency multiplies the cost Capabilities control *where* the work happens. Latency controls *what the work costs*. | | fast source | slow source | | --- | --- | --- | | **can serve windows** | scrolling is almost free; waste is invisible | each unnecessary window is a visible delay | | **cannot serve windows** | one large read, and it completes quickly | one large read, and the user waits for it | The *count* of requests is the same in both conditions. **Measure the count, not the feeling.** --- ## What we covered | Item | What it does | | --- | --- | | `capabilities: []` | Removes all operations except `list`. This gives the eager method. | | the `source` line | Names what the source can do, and which load method follows | | **paged** and **eager** | One window for each screen, against one read at open | | the `sort` line | Says whether the source sorts, or the framework sorts the rows that it holds | | the `search` line | The same, for filters. `kind: grid` hides the control. | | `row positions dropped` | A paged view that refetches instead of showing a sample as the set | ```admonish tip title="What is next" Capabilities explain the requests that the application had to send. The next section finds the requests that it did not have to send, and adds a source that sends 19MB to fill a grid of five columns. ``` --- Source: https://vantage-ui.com/book/optimising-diorama/waste.html # Fetching What You Drop This section examines the requests that the application did not have to send. These are more difficult to find. A wasteful application and an efficient application look the same on screen. There are three types of waste, from the easiest to find to the most difficult: 1. **Repeats.** The application fetched the same window more than one time. 2. **Redundant rows.** Rows arrived when the cache already held them. 3. **Columns that no view shows.** Data crossed the network and then went in the bin. The first two have counters. The third has one line. ## The two counters that must be zero Quit Vantage UI. The session summary prints: ```text people summary 3 fetches, 0 repeats · 400 rows, 0 redundant · 9.1s waiting, 3.0s slowest summary 1 dio, 1 table scenery, 0 record, 0 servo still alive summary session 18.0s · cpu 15.0s · peak rss 352MB ``` **`repeats`** counts the windows that the application requested more than one time. The loader is built to avoid repeats, so a number above zero is a defect. There are three causes: - the cache is not operating - a total declares rows that the source will not serve - a viewport moves between two positions **`redundant`** counts the rows that arrived when the cache already held them. A cold run must give zero. A warm run gives a small number, and that is the cost of revalidation. If the redundant count equals the received count, the application fetched a full page for nothing. ```admonish note title="Read the counters with the session, not alone" `3 fetches` has no meaning until you know that you scrolled through 400 rows of a table of 200,000. Use the ratios: fetches for each screen of rows, and redundant rows for each received row. Record what you did together with what it cost. ``` ## Watch the cache do its work ```text 22:42:19.028 people viewport rows 77..90 — 6 scroll events coalesced into one 22:42:19.029 people cache all 13 rows of 77..90 served locally — no fetch 22:42:19.516 people viewport rows 125..138 — 3 scroll events coalesced into one 22:42:19.516 people cache all 13 rows of 125..138 served locally — no fetch 22:42:19.858 people viewport rows 257..270 — 11 scroll events coalesced into one 22:42:19.859 people dio fetch #2 asks for rows 257..357 (viewport 257..270) — 100 missing ``` Two efficiencies are visible here. **The framework combines scroll events.** Twenty scroll events produced three viewports. The loader waits for the movement to stop, and then declares one viewport for the final position. The intermediate positions never cause a fetch. **The loader reads ahead of the screen.** The viewport of `fetch #2` was `257..270`, which is 14 rows. The request was for `257..357`, which is 100 rows. The loader fetches extra rows in the direction of movement, so that the next screen costs nothing. This is why the two ranges differ. Read-ahead also explains the fetch count in the summary above: 3 fetches covered 400 rows, which is more screens than three. ## The waste with no counter Now add a source with large rows. The grid shows five short columns. Each row carries 200 more fields that nobody declared. ```yaml # datasource/wide.yaml type: faker effect: static debug: true count: 500 seed: 111 shape: capabilities: [count, fetch_window, order] latency: window: 50ms..150ms extra_fields: count: 200 # 200 extra fields in each row size: 500 # each extra field holds 500 bytes ``` Open it and read one line: ```text 22:06:34.148 wide dio fetch #1 got 200 rows in 2.3s ⚠ slow 22:06:34.148 wide payload 206 columns received · no view declared what it needs, so all were fetched · 19.6MB · id,name,surname,email,city,balance,extra_0001, extra_0002 +198 more ``` **The application moved 19.6MB to fill a grid of five columns.** The five columns are about 19KB of that total. Nothing failed. No counter changed. `repeats` is zero, and `redundant` is zero. Each measurement in the session summary says that the application is correct. This fault hides better than the others, because each other signal says that the application is correct. Only the `payload` line finds it. ## "No view declared what it needs" Diorama has a mechanism for this. A scenery can declare its **demand**, which is the list of columns that it shows. The Dio then combines the demands of the open views. When a view declares its demand, the line reports against that demand: ```text wide payload 5 of 206 columns wanted · 201 fetched and dropped · 19.6MB ``` This grid declares nothing. The combined demand is therefore "all columns". In Rust a view declares its demand when it opens, with [`TableSceneryBuilder::columns`](vantage_diorama::TableSceneryBuilder::columns). The framework cannot tell the difference between a view that wants all 206 columns and a view that never declared its columns. It assumes the first, and it fetches all the columns. ```admonish note title="Demand controls the second pass, not the width of the first" Diorama can load a row in two passes. The first pass fetches the window. The second pass — the **hydrate** pass — fetches the augmented columns of the rows on screen. Demand controls the second pass. A view that does not show an augmented column does not pay to hydrate it. Demand does not yet decrease the columns in one window fetch. The 19.6MB above is therefore not a configuration error that you can correct in YAML. Use the `payload` line as a measurement, and use the size of the number as the reason to make the source return fewer columns. ``` ## What to do about a large payload These actions are listed from the most effective to the least effective. - **Decrease the columns at the source.** Use a REST endpoint with a `fields=` parameter, a SQL view that selects the columns that you show, or a GraphQL query that names its columns. The cheapest byte is the byte that nobody sends. - **Move the large columns into an augment.** An `augment:` block on the table makes the large fields the second pass, which demand controls. See [Augmentation](../augmentation.md) for the block and its options. - **Observe fewer tables.** A page that observes two tables opens two Dios and pays for both. ## A checklist for one page Run the page cold, then warm. Read four numbers. | Number | Where | Correct value | | --- | --- | --- | | fetches | `summary` | fewer than one for each screen, because the loader reads ahead | | repeats | `summary` | **zero**, always | | redundant | `summary` | zero when cold; small when warm | | payload for each fetch | `payload` | proportional to the columns that you show | If the first three numbers are correct and the fourth is not, you have the invisible fault. --- ## What we covered | Item | What it does | | --- | --- | | `repeats` | Windows that the application fetched more than one time. Must be zero. | | `redundant` | Rows that arrived when the cache held them. Zero when cold, small when warm. | | coalescing | Scroll events that became one viewport, because the loader waits for the movement to stop | | read-ahead | The extra rows that the loader fetches in the direction of movement | | `payload` | The columns that arrived, the columns that a view wants, and the cost in bytes | | `extra_fields` | The faker setting that adds fields to each row | | demand · hydrate | The columns that a view declares, and the second pass that they control | ```admonish tip title="What is next" Each source until now answered its requests. The next section uses a source that fails one request in five, stops for ten seconds each minute, and reports a row count that is wrong. ``` --- Source: https://vantage-ui.com/book/optimising-diorama/faults.html # Faults: Errors, Outages & Lies Each source until now answered, and its answers were true. Production sources are not always honest. A source can fail one request while the user scrolls. It can stop for ten seconds. It can also report a row count that it will not serve. Here the cache stops being an optimisation. It becomes the component that keeps your application usable. ## A source with faults ```yaml # datasource/flaky.yaml type: faker effect: static debug: true count: 1000 seed: 107 shape: capabilities: [count, fetch_window, order] latency: window: 100ms..300ms faults: error_rate: 0.2 # one request in five fails offline: 10s/60s # /: down for 10s in each 60s total_lie: -13 # reports 13 fewer rows than it will serve ``` There are three independent faults, and each one repeats because the shape has a seed. ## What a failure looks like ```text 22:08:05.813 flaky dio fetch #1 asks for rows 0..200 — 200 missing, 0 already held 22:08:05.978 flaky dio fetch #1 failed after 164ms — shaped source request failed (injected fault) ``` The request identifier connects the two lines. The duration tells you how long you waited for the failure. The message is the error from the source. What is **absent** is more important. There is no state line, no cache line, and no total line. **A failed fetch changes nothing.** The rows that the view already had are still present, still in their correct positions, and still on screen. A failed refresh costs freshness, not data. ## The viewport causes the retry Nothing retries on a timer, and this is deliberate. The rows are still absent. The next time that a view declares a viewport that includes those rows, the loader asks again. A scroll, a repaint, or a window that gets focus each cause this. ```text flaky dio fetch #2 asks for rows 0..200 — 200 missing, 0 already held flaky dio fetch #2 got 200 rows in 210ms flaky cache +200 new, 0 updated — now holding 200 of 987 (20.3%) flaky total 987 rows — stated by the source in the same response ``` The retry is a result of demand. It is not a policy. It therefore needs no backoff configuration, and it cannot flood the source: if no view looks at a range, the loader sends no request for it. ```admonish note title="The one failure that does flood the source" There is an exception, and the `optimising` project produced it. A fetch can fail *immediately* and always — a rejected query, and not an unreliable connection. Each viewport movement then sends a request, each request fails immediately, and the application uses all available CPU. During the writing of this guide, a faker datasource advertised `order` while the source refused every ordered query. Vantage UI appeared to stop responding. The lesson is not about retry policy. **A capability that a source advertises but cannot do is worse than a capability that it never claims**, because nothing limits the rate of the failure. ``` ## The dishonest total Read the two last lines above again. The table has **1,000** rows. The source says 987. The grid uses the total from the source, because the source is the authority on how many rows it has. One condition overrides this rule, and the next paragraph gives it. A total that is too small is the safe direction. The grid makes its scrollbar 13 rows short, and the user cannot reach the last rows. A total that is too large is the dangerous direction. If a fetch asks for a range with missing rows, and it fills **none** of them, then the set ends at the first missing row, whatever the total says. The framework caps the total and reports the cap with the message *"capped: the source promises more than it serves"*. Without this rule, the grid shows rows that no fetch can fill. A missing row causes a fetch. Therefore the application asks the source again, one request each few seconds, for as long as the page stays open. ## Outages `offline: 10s/60s` stops the source on a schedule. In the stream, an outage looks like a sequence of failures. There is no separate offline state to handle, no reconnect procedure, and no separate path in your code. - Cached rows continue on screen, and they do not change. - Ranges that the application has not fetched stay empty. The loader asks for them again when a view looks at them. - When the source returns, the next viewport succeeds and fills the empty rows. The difficult condition is the first open with an empty cache during an outage. There is nothing to show and nothing to fetch. This is one more reason to measure the cold run and the warm run separately. ## What to check under faults | Question | Where to look | | --- | --- | | Did a failure delete data? | No `cache` line after the `failed` line, and rows still on screen | | Did the retry behave correctly? | One `asks for` line for each viewport movement, and not a sequence of them | | Is the total correct? | The `total` line, and whether the framework ever caps it | | Did recovery need a restart? | A `got` line after the failures, with no restart | An application that handles faults well produces a stream that looks normal: some failures, some retries when a view looks at the rows, and rows that never disappear. --- ## What we covered | Item | What it does | | --- | --- | | `faults.error_rate` | Makes a fraction of the requests fail | | `faults.offline` | Stops the source on a schedule, in the form `/` | | `faults.total_lie` | Changes the number that the grid uses to set its size | | `failed after …` | The request that did not complete, with its cost and the error from the source | | *capped: promises more than it serves* | The defence against a total that no fetch can fill | ```admonish tip title="What is next" The last section makes assertions from what you read. A number that becomes worse then makes a test fail, instead of nobody noticing. ``` --- Source: https://vantage-ui.com/book/optimising-diorama/regression.html # Locking It In: Cache Regression Tests The numbers that you measured are true of the code as it is today. This section answers a different question: what happens in six months, when somebody adds a refresh timer or makes a query wider, and nobody repeats the experiment? Three measurements automate. The rest is judgement. | Property | Assertion | Why it becomes worse without a signal | | --- | --- | --- | | **Requests for each open** | exactly *N* fetches for a known scroll | a second component that observes the table, a refresh at open, or a second Dio | | **No repeats** | `repeats == 0` | a total that is too large, or a viewport that moves between two positions | | **No redundant rows** | `redundant == 0` on a cold run | a refresh that makes too much data invalid | Put everything else in a report that you read. Do not put it in a test. Latency, memory, and payload size change for correct reasons, such as a larger fixture or a slower machine. A test that fails on those numbers gets disabled in one month. ```admonish note title="Assert on counts. Never assert on times." `3 fetches` is a property of your code. `9.1 seconds` is a property of the machine that ran the test. Put the first in CI. Put the second in a log that you read when something feels wrong. ``` ## What a test needs from the source **Assertions on exact counts need a source that you control.** A real source gives different numbers on each run, so you cannot assert on them. You can still point the `optimising` project at a real source and compare the two reports by hand, as section 1 describes. ```yaml type: faker effect: static debug: true count: 1000 # a small set, so the fetch count is small seed: 103 # the same rows, in the same sequence, on each run shape: capabilities: [count, fetch_window, order] latency: window: 20ms # one value, not a range — no variation in CI ``` There are three changes from the interactive project. - **`count`** decreases from 200,000 to 1,000, so that a short scroll covers a known part of the set. - **`seed`** was always present, and now it is necessary. A test that asserts on row 0 needs row 0 to be the same row on each machine. A seed fixes the sequence of generated values, so the same seed with the same `count` gives the same rows. A different `count` gives a different set. - **`latency`** becomes one value instead of a range. Variable latency is realistic in a measurement session. In a test it causes intermittent failures and gives no benefit. ## The structure of a test 1. **Start from a known cache state.** Delete the cache file of the datasource for a cold run. For a warm run, run Vantage UI one time to fill the cache, then measure the second start. 2. **Do a known interaction.** Open one page, declare one viewport, or scroll a fixed number of rows. Use `--page=`, so that no other page adds requests to your count. 3. **Wait for the requests to stop.** Do not wait for a fixed time. Poll the fetch count until it does not change for 500ms. A fixed `sleep` is slower than necessary, and it also fails on a busy CI machine. 4. **Assert on the session summary.** Read the counters, compare them, and fail with a clear message. ## How to read the numbers ```sh # each fetch of the session grep 'asks for' session.log | wc -l # the summary, printed when Vantage UI quits grep -A3 'session summary' session.log ``` In a Rust harness, you read the same numbers directly. The counters live in [`stats`](vantage_diorama::stats), behind vantage-diorama's `diagnostics` feature: ```rust,no_run use vantage_diorama::stats; // Limit the measurement to this interaction, not to the whole process. stats::reset_fetch_stats(); // ... drive the page ... let stats = stats::fetch_stats(); // `table` is the name of the Dio, which is the master table of the datasource. let people = stats.iter().find(|s| s.table == "people").expect("the table was read"); assert_eq!(people.repeats, 0, "the application fetched a window more than one time"); assert_eq!(people.rows_redundant, 0, "rows arrived that the cache already held"); ``` The block is compiled but not run: `// ... drive the page ...` stands for your own interaction, and without one there is no `people` entry to find. [`reset_fetch_stats`](vantage_diorama::stats::reset_fetch_stats) makes the measurement local to one test. The counters are global to the process and they accumulate. A test that does not reset them measures each test before it. To assert on the count of fetches, first measure the count for your interaction, then write that number into the test. The count depends on the read-ahead that section 4 describes, so you cannot calculate it from the number of screens. ## The assertion that finds faults If you write one assertion only, write this one: ```rust,no_run # use vantage_diorama::stats; # let stats = stats::fetch_stats(); # let people = stats.iter().find(|s| s.table == "people").expect("the table was read"); assert_eq!(people.repeats, 0); ``` It is cheap, it does not fail intermittently, and it finds every repeated fetch. It does not find a large payload. Section 4 shows a fault with 19.6MB of traffic, zero repeats, and zero redundant rows. Assert on the payload separately, or read the `payload` line. ## Views that stay open The second line of the session summary counts the views that are still alive. In Rust, ask the Dio itself. [`Dio::diagnostics`](vantage_diorama::Dio::diagnostics) is a snapshot of one Dio: its cache table, the rows it holds, and one entry for each open table view, with `shared_by` saying how many views read the same rows: ```rust # use vantage_diorama::{Cache, Lens}; # use vantage_vista::mocks::MockShell; # use vantage_vista::{Column, ColumnFlag, Vista, VistaMetadata}; # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let metadata = VistaMetadata::new() # .with_column(Column::new("id", "string").with_flag(ColumnFlag::Id)) # .with_id_column("id"); # let people = Vista::new("people", MockShell::new().with_metadata(metadata)); let lens = Lens::builder().cache(Cache::memory()).build()?; let dio = lens.dio(people).build().await?; let grid = dio.table_scenery().open().await?; let diagnostics = dio.diagnostics().await?; assert_eq!(diagnostics.sceneries.len(), 1, "one grid, one open view"); # drop(grid); # Ok(()) # } ``` Take the snapshot after the page closes, too. A view that outlives its page is the leak that the summary line reports. Reading the snapshot changes nothing, so a test can take as many as it needs. ## The result The result is not a faster application. The framework always did this work. The result is an application that you can **measure**. --- ## What we covered | Item | What it does | | --- | --- | | `seed:` with one latency value | Makes a run repeatable, so that you can assert on it | | `--page=` | Opens one page, so you count the requests of that page only | | `stats::reset_fetch_stats()` | Limits the counters to one interaction | | `stats::fetch_stats()` | Gives the counters in Rust, with no log to parse | | `dio.diagnostics().await?` | Lists the open views of one Dio, so a test can find a view that leaked | | `repeats == 0` | The one assertion to write, if you write one | ```admonish tip title="Where to go next" If you want to measure your own source, point the `optimising` project at it. The method works with any datasource, and the faker runs stay useful as the control. If you write a driver, [Adding a New Persistence](../new-persistence.md) is where you select the capabilities to advertise. This chapter is where you find out whether your driver does what it advertised. ``` --- Source: https://vantage-ui.com/book/new-persistence.html # Adding a New Persistence So you want to connect Vantage to a new database? This guide walks through the process in nine incremental steps — each one unlocks more framework features. You don't have to implement all nine; stop whenever your persistence has enough capability for your use case. --- ## Overview | Step | What you build | What it unlocks | Can skip? | | ------------------------------------------------------------- | ------------------------------------------------------------- | -------------------------------------------------------------------- | ---------------------------------------------- | | [1. Type System](./new-persistence/step1-types.md) | `vantage_type_system!` macro, `AnyType`, `Record` conversions | Type-safe values, struct ↔ record mapping | **Required** | | [2. Expressions](./new-persistence/step2-expressions.md) | Vendor macro, `ExprDataSource` | Execute raw queries, cross-database `defer()` | Skip for simple read-only sources (CSV) | | [3. Operators](./new-persistence/step3-operators.md) | Vendor operation trait, typed `.eq()`/`.gt()`/`.in_()` | Ergonomic typed conditions instead of raw expression macros | Skip if you don't expose typed columns | | [4. Query Builder](./new-persistence/step4-query-builder.md) | `Selectable`, `SelectableDataSource` | Composable SELECT with conditions, ordering, limits | Skip if your persistence has no query language | | [5. Table & CRUD](./new-persistence/step5-table-crud.md) | `TableSource`, entity tables, aggregates, writes | `Table`, full CRUD, `ReadableDataSet`, `WritableDataSet` | **Required** for table support | | [6. Relationships](./new-persistence/step6-relationships.md) | `with_one`, `with_many`, correlated subqueries | Reference traversal, expression fields | Skip if you don't need cross-table queries | | [7. Multi-Backend](./new-persistence/step7-multi-backend.md) | model crate, `vista_factory().from_table()`, CLI example | Type-erased tables, generic UI/API code | Skip if you only use one persistence | | [8. Vista](./new-persistence/step8-vista-integration.md) | `VistaFactory`, `TableShell`, YAML extras | YAML-defined data handles consumed by UI / scripting / agents | Skip if you don't need Vista support | | [9. Contained Relations](./new-persistence/step9-contained-relations.md) | embedded object/array columns as sub-tables | JSON/document sub-records surfaced as editable child sets | Skip if you have no embedded collections | --- ## Step 1: Type System Every database has its own idea of what types exist. The `vantage_type_system!` macro generates a type trait, variant enum, and type-erased `AnyType` wrapper that prevents silent casting between incompatible types. You'll implement the type trait for each Rust type your database supports, set up `Record` conversions (free via serde for JSON-based backends, or via `#[entity]` macro for custom value types), and add `TryFrom` for scalar extraction. **[Read Step 1 →](./new-persistence/step1-types.md)** --- ## Step 2: Expressions With types in place, build a vendor macro (`sqlite_expr!`, `surreal_expr!`) that produces `Expression` with typed parameters. Implement `ExprDataSource` to execute expressions against your database — handling parameter binding, deferred cross-database resolution, and result parsing. Skip this step if your persistence evaluates conditions in-memory (like CSV) — you can implement `TableSource` directly without an expression engine. **[Read Step 2 →](./new-persistence/step2-expressions.md)** --- ## Step 3: Operators Build vendor-specific operation traits so typed columns get ergonomic `.eq()`/`.gt()`/`.in_()` methods that produce your backend's native condition type. Blanket-implemented over all `Expressive` so users don't have to write raw expression macros for every condition. Skip this step if you don't expose typed columns to end users. **[Read Step 3 →](./new-persistence/step3-operators.md)** --- ## Step 4: Query Builder Build a SELECT struct implementing the `Selectable` trait — fields, conditions, ordering, limits, aggregates. Wire it up through `SelectableDataSource` so the rest of Vantage can create and execute queries through a standard interface. Skip this step if your persistence doesn't have a query language. MongoDB, for instance, skips `Selectable` and uses native BSON pipelines instead. **[Read Step 4 →](./new-persistence/step4-query-builder.md)** --- ## Step 5: Table & CRUD Implement `TableSource` to give Vantage full table abstraction — columns, conditions, ordering, pagination, entity CRUD, and aggregates. This is where `Table` comes alive and auto-implements `ReadableDataSet`, `WritableDataSet`, and `ActiveEntitySet`. Start with `todo!()` for every method and implement them incrementally, driven by tests. **[Read Step 5 →](./new-persistence/step5-table-crud.md)** --- ## Step 6: Relationships Declare `with_one` and `with_many` relationships on tables and traverse them with `get_ref_as`. Implement `column_table_values_expr` for subquery-based traversal and optionally `related_correlated_condition` for correlated subqueries (expression fields like computed counts). Skip this step if your persistence is flat (no foreign keys or cross-collection references). **[Read Step 6 →](./new-persistence/step6-relationships.md)** --- ## Step 7: Multi-Backend Applications Define your entities and table constructors once in a model crate, then erase the backend type with `db.vista_factory().from_table(table)` so generic UI, CLI, and API code works identically across SurrealDB, SQLite, CSV, MongoDB, or your new persistence. (Type erasure used to live in the now-removed `AnyTable`; it moved up to [`Vista`](./new-persistence/step8-vista-integration.md) in 0.5.) **[Read Step 7 →](./new-persistence/step7-multi-backend.md)** --- ## Step 8: Vista Wrap your typed `Table` as a `Vista` — the universal, schema-bearing handle that UI, scripting, and agents consume through a CBOR / `String` boundary. You write a `TableShell` that executes reads, narrowing and writes, and a factory that builds it from a typed table or a YAML spec. The shell advertises only the capabilities it honours, and the vista contract test checks each one. Skip this step if your driver is only consumed from typed Rust code. **[Read Step 8 →](./new-persistence/step8-vista-integration.md)** --- ## Step 9: Contained Relations Surface an embedded object or array column (a JSON document, a Mongo sub-document) as an editable child sub-table. Declare it with `with_contained_one` / `with_contained_many`; reads project the embedded collection as records and writes patch it back into the host column. Your shell's `get_ref` routes these relations to the shared `Table::get_contained_ref`, so native document backends and JSON-blob columns share the same path. Skip this step if your persistence has no embedded collections. **[Read Step 9 →](./new-persistence/step9-contained-relations.md)** --- Source: https://vantage-ui.com/book/new-persistence/step1-types.html # Step 1: Define Your Type System Every database has its own idea of what types exist. SQLite has 5 storage classes (NULL, INTEGER, REAL, TEXT, BLOB). Postgres has dozens. SurrealDB has its own set with Things and Geometry types. The vantage type system gives you two things: 1. **Type markers** — so you can tell the difference between "this is an integer" and "this is text" even when both are stored as `serde_json::Value` under the hood. 2. **Safe extraction** — `try_get::()` on a text value returns `None` instead of silently coercing. This prevents the kind of bugs where a string "42" gets treated as a number somewhere downstream. ### Why not just use serde_json::Value directly? You can! And for simple cases it works fine. The problem shows up when values move between contexts. A JSON number `42` could be an integer, a float, or even a boolean (SQLite stores bools as 0/1). Without type markers, you lose that distinction and get silent data corruption. ### Setting it up Use the `vantage_type_system!` macro. It generates a trait, an enum of variants, and a type-erased `AnyType` wrapper: ```rust # use vantage_types::vantage_type_system; vantage_type_system! { type_trait: SqliteType, method_name: json, value_type: serde_json::Value, type_variants: [Null, Integer, Text, Real, Numeric, Blob] } # impl SqliteTypeVariants { # fn from_json(value: &serde_json::Value) -> Option { # use serde_json::Value; # match value { # Value::Null => Some(Self::Null), # Value::Bool(_) => Some(Self::Integer), # Value::Number(n) if n.is_f64() => Some(Self::Real), # Value::Number(_) => Some(Self::Integer), # Value::String(_) => Some(Self::Text), # Value::Array(_) | Value::Object(_) => None, # } # } # } ``` This gives you `SqliteType` (trait), `SqliteTypeVariants` (enum), and `AnySqliteType` (the type-erased wrapper that remembers which variant a value belongs to). The `value_type` is whatever your database driver naturally speaks. For SQL databases that's usually `serde_json::Value`. SurrealDB uses `ciborium::Value` (CBOR). You could use any type — even `String` if your storage is that simple. Then implement the trait for each Rust type. Here's bool — SQLite stores it as 0/1: ```rust # use serde_json::Value; # use vantage_types::vantage_type_system; # vantage_type_system! { # type_trait: SqliteType, # method_name: json, # value_type: serde_json::Value, # type_variants: [Null, Integer, Text, Real, Numeric, Blob] # } # impl SqliteTypeVariants { # fn from_json(value: &Value) -> Option { # match value { # Value::Null => Some(Self::Null), # Value::Bool(_) => Some(Self::Integer), # Value::Number(n) if n.is_f64() => Some(Self::Real), # Value::Number(_) => Some(Self::Integer), # Value::String(_) => Some(Self::Text), # Value::Array(_) | Value::Object(_) => None, # } # } # } impl SqliteType for bool { type Target = SqliteTypeIntegerMarker; // bool lives in the Integer family fn to_json(&self) -> serde_json::Value { Value::Number(if *self { 1.into() } else { 0.into() }) } fn from_json(value: serde_json::Value) -> Option { match value { Value::Number(n) => n.as_i64().map(|i| i != 0), Value::Bool(b) => Some(b), // accept native bools too _ => None, } } } # assert_eq!(AnySqliteType::new(true).try_get::(), Some(true)); ``` And here's how the type safety works in practice: ```rust # use serde_json::Value; # use vantage_types::vantage_type_system; # vantage_type_system! { # type_trait: SqliteType, # method_name: json, # value_type: serde_json::Value, # type_variants: [Null, Integer, Text, Real, Numeric, Blob] # } # impl SqliteTypeVariants { # fn from_json(value: &Value) -> Option { # match value { # Value::Null => Some(Self::Null), # Value::Bool(_) => Some(Self::Integer), # Value::Number(n) if n.is_f64() => Some(Self::Real), # Value::Number(_) => Some(Self::Integer), # Value::String(_) => Some(Self::Text), # Value::Array(_) | Value::Object(_) => None, # } # } # } # impl SqliteType for i64 { # type Target = SqliteTypeIntegerMarker; # fn to_json(&self) -> Value { Value::from(*self) } # fn from_json(value: Value) -> Option { value.as_i64() } # } # impl SqliteType for String { # type Target = SqliteTypeTextMarker; # fn to_json(&self) -> Value { Value::String(self.clone()) } # fn from_json(value: Value) -> Option { value.as_str().map(str::to_string) } # } let val = AnySqliteType::new(42i64); assert_eq!(val.try_get::(), Some(42)); // same variant → works assert_eq!(val.try_get::(), None); // Integer ≠ Text → rejected let val = AnySqliteType::new("hello".to_string()); assert_eq!(val.try_get::(), Some("hello".to_string())); assert_eq!(val.try_get::(), None); // Text ≠ Integer → rejected ``` You also need `From` conversions so values can be created conveniently: ```rust # use serde_json::Value; # use vantage_types::vantage_type_system; # vantage_type_system! { # type_trait: SqliteType, # method_name: json, # value_type: serde_json::Value, # type_variants: [Null, Integer, Text, Real, Numeric, Blob] # } # impl SqliteTypeVariants { # fn from_json(value: &Value) -> Option { # match value { # Value::Null => Some(Self::Null), # Value::Bool(_) => Some(Self::Integer), # Value::Number(n) if n.is_f64() => Some(Self::Real), # Value::Number(_) => Some(Self::Integer), # Value::String(_) => Some(Self::Text), # Value::Array(_) | Value::Object(_) => None, # } # } # } # impl SqliteType for i64 { # type Target = SqliteTypeIntegerMarker; # fn to_json(&self) -> Value { Value::from(*self) } # fn from_json(value: Value) -> Option { value.as_i64() } # } # impl SqliteType for String { # type Target = SqliteTypeTextMarker; # fn to_json(&self) -> Value { Value::String(self.clone()) } # fn from_json(value: Value) -> Option { value.as_str().map(str::to_string) } # } # impl SqliteType for bool { # type Target = SqliteTypeIntegerMarker; # fn to_json(&self) -> Value { Value::from(i64::from(*self)) } # fn from_json(value: Value) -> Option { value.as_i64().map(|i| i != 0) } # } # impl From for AnySqliteType { # fn from(value: i64) -> Self { AnySqliteType::new(value) } # } # impl From<&str> for AnySqliteType { # fn from(value: &str) -> Self { AnySqliteType::new(value.to_string()) } # } # impl From for AnySqliteType { # fn from(value: bool) -> Self { AnySqliteType::new(value) } # } let val: AnySqliteType = 42i64.into(); let val: AnySqliteType = "hello".into(); let val: AnySqliteType = true.into(); # assert_eq!(val.try_get::(), Some(true)); ``` ### Records and struct conversion A `Record` is an ordered key-value map (field name → value). It's how vantage represents a single row of data regardless of the backend. The question is: how do your structs become Records and vice versa? There are two paths depending on your `value_type`. #### Path A: serde_json::Value (the easy path) If your type system uses `serde_json::Value` as the value type (like SQLite does), you get struct conversion for free. Vantage has blanket implementations of `IntoRecord` and `TryFromRecord` for any type that implements serde's `Serialize`/`Deserialize`: ```rust # use serde::{Deserialize, Serialize}; # use vantage_types::{IntoRecord, Record, TryFromRecord, TryIntoRecord}; #[derive(Serialize, Deserialize)] struct Product { name: String, price: i64, is_deleted: bool, } let product = Product { name: "Cupcake".into(), price: 120, is_deleted: false }; // Struct → Record — automatic via serde let record: Record = product.try_into_record().unwrap(); // Record → Struct — automatic via serde let restored: Product = Product::from_record(record).unwrap(); ``` This works because serde already knows how to turn structs into JSON objects and back. No extra code needed on your part. #### Path B: Custom value type (the `#[entity]` path) If your type system uses something other than `serde_json::Value` — like SurrealDB's `ciborium::Value` — then serde's blanket impls don't apply. You need the `#[entity]` proc macro to generate the conversion code: ```rust # use vantage_surrealdb::AnySurrealType; # use vantage_types::entity; #[entity(SurrealType)] #[derive(Debug, Clone)] struct Product { name: String, price: i64, is_deleted: bool, } ``` The `#[entity(SurrealType)]` macro looks at each field, and generates: - `IntoRecord` — calls `AnySurrealType::new(self.name)` for each field - `TryFromRecord` — calls `record["name"].try_get::()` for each field This is where the type markers from Step 1 become critical. When you read a record back from the database, each value is an `AnySurrealType` with a variant tag. The `try_get::()` call checks that the variant is `Text` before extracting. If someone stored an integer in a field that should be a string, you get an error instead of garbage. You can even target multiple type systems at once: ```rust # use vantage_csv::AnyCsvType; # use vantage_surrealdb::AnySurrealType; # use vantage_types::entity; #[entity(SurrealType, CsvType)] struct Product { name: String, price: i64, } ``` This generates conversions for both `Record` and `Record`, so the same struct works across different backends. #### Testing Record conversions Test `Record` in both modes — typed (write path) and untyped (read path). **Typed records** simulate what you'd build when inserting data. Values have variant tags, and `try_get` enforces them: ```rust # use serde_json::Value; # use vantage_types::{Record, vantage_type_system}; # vantage_type_system! { # type_trait: SqliteType, # method_name: json, # value_type: serde_json::Value, # type_variants: [Null, Integer, Text, Real, Numeric, Blob] # } # impl SqliteTypeVariants { # fn from_json(value: &Value) -> Option { # match value { # Value::Null => Some(Self::Null), # Value::Bool(_) => Some(Self::Integer), # Value::Number(n) if n.is_f64() => Some(Self::Real), # Value::Number(_) => Some(Self::Integer), # Value::String(_) => Some(Self::Text), # Value::Array(_) | Value::Object(_) => None, # } # } # } # impl SqliteType for i64 { # type Target = SqliteTypeIntegerMarker; # fn to_json(&self) -> Value { Value::from(*self) } # fn from_json(value: Value) -> Option { value.as_i64() } # } # impl SqliteType for String { # type Target = SqliteTypeTextMarker; # fn to_json(&self) -> Value { Value::String(self.clone()) } # fn from_json(value: Value) -> Option { value.as_str().map(str::to_string) } # } #[test] fn test_typed_record() { let mut record: Record = Record::new(); record.insert("name".into(), AnySqliteType::new("Cupcake".to_string())); record.insert("price".into(), AnySqliteType::new(120i64)); assert_eq!(record["name"].try_get::(), Some("Cupcake".to_string())); assert_eq!(record["name"].try_get::(), None); // Text ≠ Integer → blocked assert_eq!(record["price"].try_get::(), Some(120)); assert_eq!(record["price"].try_get::(), None); // Integer ≠ Text → blocked } ``` **Untyped records** simulate what comes back from the database. Values have `type_variant: None`, so `try_get` is permissive — it just attempts the conversion: ```rust # use serde_json::{Value, json}; # use vantage_types::{Record, vantage_type_system}; # vantage_type_system! { # type_trait: SqliteType, # method_name: json, # value_type: serde_json::Value, # type_variants: [Null, Integer, Text, Real, Numeric, Blob] # } # impl SqliteTypeVariants { # fn from_json(value: &Value) -> Option { # match value { # Value::Null => Some(Self::Null), # Value::Bool(_) => Some(Self::Integer), # Value::Number(n) if n.is_f64() => Some(Self::Real), # Value::Number(_) => Some(Self::Integer), # Value::String(_) => Some(Self::Text), # Value::Array(_) | Value::Object(_) => None, # } # } # } # impl SqliteType for i64 { # type Target = SqliteTypeIntegerMarker; # fn to_json(&self) -> Value { Value::from(*self) } # fn from_json(value: Value) -> Option { value.as_i64() } # } # impl SqliteType for f64 { # type Target = SqliteTypeRealMarker; # fn to_json(&self) -> Value { Value::from(*self) } # fn from_json(value: Value) -> Option { value.as_f64() } # } # impl SqliteType for String { # type Target = SqliteTypeTextMarker; # fn to_json(&self) -> Value { Value::String(self.clone()) } # fn from_json(value: Value) -> Option { value.as_str().map(str::to_string) } # } # impl AnySqliteType { # fn untyped(value: Value) -> Self { # Self { value, type_variant: None } # } # } #[test] fn test_untyped_record() { let mut record: Record = Record::new(); record.insert("name".into(), AnySqliteType::untyped(json!("Cupcake"))); record.insert("price".into(), AnySqliteType::untyped(json!(120))); assert_eq!(record["name"].try_get::(), Some("Cupcake".to_string())); assert_eq!(record["name"].try_get::(), None); // fails because "Cupcake" isn't a number assert_eq!(record["price"].try_get::(), Some(120)); assert_eq!(record["price"].try_get::(), Some(120.0)); // permissive — json 120 can be f64 } ``` The key difference: a typed `AnySqliteType::new(42i64)` blocks `try_get::()` because Integer ≠ Real. An untyped `AnySqliteType::untyped(json!(42))` allows it because there's no variant to check — it just asks "can JSON number 42 be read as f64?" Also test `Option` fields, null handling, and missing fields. ### TryFrom<AnyType> for common types You also need `TryFrom<AnyType>` implementations for scalar types and Records. These are used later by `AssociatedExpression::get()` in Step 2, but they belong here because they're part of the type system: ```text // Scalars — extract single value from single-row results impl TryFrom for i64 { ... } impl TryFrom for String { ... } // etc. // Records — extract first row from result array impl TryFrom for Record { ... } impl TryFrom for Record { ... } ``` For scalars, if the result is a single-row, single-column array like `[{"COUNT(*)": 3}]`, extract the value automatically. For Records, extract the first row and wrap each field as an untyped `AnyType`. ### Step 1 conclusion At this point you should have: 1. **Type impls** in `src//types/` — the `vantage_type_system!` macro call, trait implementations for each Rust type, `From` conversions on `AnyType`, variant detection in `TypeVariants::from_*()`, and `TryFrom<AnyType>` for scalars and Records. 2. **Tests** in `tests//1_types_round_trip.rs` covering: - In-memory `AnyType` round-trips for each supported type - Type mismatch rejections (wrong variant → `None`) - Struct ↔ Record conversions (including Option fields and error cases) - Values read from the actual database converting correctly ### How type markers flow through the system The `AnyType` wrapper has an `Option` field — `Some(Integer)` means "I know this is an integer", `None` means "I don't know, just try whatever conversion you need." **Writing (you → database):** Values created with `AnySqliteType::new(42i64)` get `type_variant: Some(Integer)`. The bind layer uses the variant to pick the right sqlx bind call — Integer binds as `i64`, Text as `&str`, Real as `f64`. No guessing. **Reading (database → you):** Values coming back from the database are created with `AnySqliteType::untyped(json_value)` which sets `type_variant: None`. This means `try_get::()` won't be blocked by a variant mismatch — it just attempts the conversion. The type checking happens later when you deserialize into a struct. ```rust # use serde_json::{Value, json}; # use vantage_types::vantage_type_system; # vantage_type_system! { # type_trait: SqliteType, # method_name: json, # value_type: serde_json::Value, # type_variants: [Null, Integer, Text, Real, Numeric, Blob] # } # impl SqliteTypeVariants { # fn from_json(value: &Value) -> Option { # match value { # Value::Null => Some(Self::Null), # Value::Bool(_) => Some(Self::Integer), # Value::Number(n) if n.is_f64() => Some(Self::Real), # Value::Number(_) => Some(Self::Integer), # Value::String(_) => Some(Self::Text), # Value::Array(_) | Value::Object(_) => None, # } # } # } # impl SqliteType for i64 { # type Target = SqliteTypeIntegerMarker; # fn to_json(&self) -> Value { Value::from(*self) } # fn from_json(value: Value) -> Option { value.as_i64() } # } # impl SqliteType for f64 { # type Target = SqliteTypeRealMarker; # fn to_json(&self) -> Value { Value::from(*self) } # fn from_json(value: Value) -> Option { value.as_f64() } # } # impl SqliteType for String { # type Target = SqliteTypeTextMarker; # fn to_json(&self) -> Value { Value::String(self.clone()) } # fn from_json(value: Value) -> Option { value.as_str().map(str::to_string) } # } # impl SqliteType for bool { # type Target = SqliteTypeIntegerMarker; # fn to_json(&self) -> Value { Value::from(i64::from(*self)) } # fn from_json(value: Value) -> Option { # match value { # Value::Number(n) => n.as_i64().map(|i| i != 0), # Value::Bool(b) => Some(b), # _ => None, # } # } # } # impl AnySqliteType { # fn untyped(value: Value) -> Self { # Self { value, type_variant: None } # } # } // Writing — typed, variant enforced let val = AnySqliteType::new(true); // type_variant: Some(Integer) val.try_get::(); // Some(true) — variant matches val.try_get::(); // Some(1) — bool lives in the Integer family val.try_get::(); // None — Integer ≠ Text, blocked // Reading — untyped, permissive let val = AnySqliteType::untyped(json!(1)); // type_variant: None val.try_get::(); // Some(1) — no variant check, json 1 parses as i64 val.try_get::(); // Some(true) — no variant check, json 1 parses as bool (≠0) val.try_get::(); // Some(1.0) — no variant check, json 1 parses as f64 val.try_get::(); // None — json 1 can't parse as String ``` Both directions use `Record`, but the values behave differently: ```text Writing: Struct → Record (typed) → bind_sqlite_value() → sqlx Reading: sqlx → Record (untyped) → try_get / serde → Struct ``` --- Source: https://vantage-ui.com/book/new-persistence/step2-expressions.html # Step 2: Make Expressions Work With the type system in place, you can now use `Expression` to build and execute queries. This step has two deliverables: a convenience macro and the `ExprDataSource` trait implementation. ### The vendor macro Define a macro that produces `Expression`. SurrealDB has `surreal_expr!`, we create `sqlite_expr!`: ```rust # use vantage_sql::sqlite_expr; let expr = sqlite_expr!("SELECT * FROM product WHERE price > {}", 100i64); ``` Under the hood, `100i64` gets wrapped as `AnySqliteType::new(100i64)` with variant `Integer`. When this expression hits the database, the bind layer knows to call `query.bind(100i64)` — not `query.bind("100")` or `query.bind(100.0)`. The macro handles three kinds of parameters: - `42i64` → scalar with type marker - `(sub_expr)` → nested expression (composed into the template) - `{deferred}` → lazy evaluation (resolved at execution time) ### Identifier quoting SQL identifiers (table names, column names) need quoting to handle reserved words, spaces, and special characters. Different databases use different quote styles — PostgreSQL and SQLite use double quotes (`"name"`), MySQL uses backticks (`` `name` ``), SurrealDB uses something else entirely. Vantage centralises this in the `Identifier` struct (`vantage-sql/src/primitives/identifier.rs`). `Identifier` is quote-agnostic — it stores the name parts and optional alias, but the actual quoting happens in the `Expressive` implementation for each backend type: ```text impl Expressive for Identifier { fn expr(&self) -> Expression { Expression::new(self.render_with('`'), vec![]) // `name` } } ``` When you add a new SQL backend, add an `Expressive` impl with your quote character. The compiler picks the right impl based on the expression type context. In practice you use the `ident()` shorthand and pass it into the vendor macro with parentheses — the `(...)` syntax calls `.expr()` automatically, so quoting is handled by the type context: ```rust # use vantage_sql::{mysql_expr, postgres_expr}; use vantage_sql::primitives::identifier::{Identifier, ident}; // The (ident(...)) syntax invokes Expressive — quoting is automatic let expr = mysql_expr!("SELECT {} FROM {} WHERE {} = {}", (ident("name")), (ident("product")), (ident("price")), 100i64); // → SELECT `name` FROM `product` WHERE `price` = 100 let expr = postgres_expr!("SELECT {} FROM {} WHERE {} = {}", (ident("name")), (ident("product")), (ident("price")), 100i64); // → SELECT "name" FROM "product" WHERE "price" = 100 ``` For qualified identifiers (table.column) and aliases: ```rust # use vantage_sql::primitives::identifier::{Identifier, ident}; # use vantage_sql::{mysql_expr, sqlite_expr}; let expr = sqlite_expr!("SELECT {}", (ident("name").dot_of("u"))); // → SELECT "u"."name" let expr = mysql_expr!("SELECT {}", (ident("name").with_alias("n"))); // → SELECT `name` AS `n` ``` Test identifier quoting in `tests//2_identifier.rs` — cover basic names, reserved words, spaces, hyphens, unicode, and names that start with numbers. These are all legal inside quoted identifiers in both PostgreSQL and MySQL. ### ExprDataSource Implement `DataSource` (marker) and `ExprDataSource` on your DB struct. The `execute` method takes an expression, flattens nested sub-expressions, converts `{}` placeholders to your driver's syntax (`?N` for SQLite, `$N` for Postgres), binds parameters using type markers, and returns results. Results come back as `AnySqliteType` with `type_variant: None` — the database doesn't preserve our markers, so results are permissive (see Step 1). For SQLite that's especially natural since it doesn't distinguish boolean from integer on the wire. If the persistence layer you're implementing _does_ preserve type information in responses (like SurrealDB with CBOR tags), set the correct `type_variant` when constructing result values in your `execute()` implementation. That way `try_get` enforces type boundaries on both sides of the round-trip. ### Validating with INSERT expressions The best way to test this is INSERT + SELECT round-trips. A single insert exercises all the pieces — macro, parameter binding, type markers, and result parsing: ```rust # use vantage_expressions::ExprDataSource; # use vantage_sql::sqlite::SqliteDB; # use vantage_sql::sqlite_expr; # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite::memory:").await.unwrap(); # db.execute(&sqlite_expr!( # "CREATE TABLE product (id TEXT PRIMARY KEY, name TEXT, price INTEGER, is_deleted BOOLEAN)" # )) # .await?; let insert = sqlite_expr!( "INSERT INTO product (id, name, price, is_deleted) VALUES ({}, {}, {}, {})", "cupcake", "Flux Cupcake", 120i64, false ); db.execute(&insert).await?; let select = sqlite_expr!("SELECT * FROM product WHERE id = {}", "cupcake"); let result = db.execute(&select).await?; # Ok(()) # } ``` Nested expressions let you build multi-row inserts from composable parts: ```rust # use vantage_expressions::{ExprDataSource, Expression, ExpressiveEnum}; # use vantage_sql::sqlite::{AnySqliteType, SqliteDB}; # use vantage_sql::sqlite_expr; # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite::memory:").await.unwrap(); # db.execute(&sqlite_expr!( # "CREATE TABLE product (id TEXT PRIMARY KEY, name TEXT, price INTEGER, is_deleted BOOLEAN)" # )) # .await?; let row1 = sqlite_expr!("({}, {}, {}, {})", "tart", "Time Tart", 220i64, false); let row2 = sqlite_expr!("({}, {}, {}, {})", "pie", "Sea Pie", 299i64, true); // Expression::from_vec joins sub-expressions with a delimiter let rows = Expression::from_vec(vec![row1, row2], ", "); // Nest into the INSERT — flattener resolves everything into a single query let insert = Expression::::new( "INSERT INTO product (id, name, price, is_deleted) VALUES {}", vec![ExpressiveEnum::Nested(rows)], ); db.execute(&insert).await?; # let count = db.execute(&sqlite_expr!("SELECT COUNT(*) FROM product")).await?; # assert_eq!(count.unwrap_scalar().try_get::(), Some(2)); # Ok(()) # } ``` The `ExpressionFlattener` collapses all nesting into one flat template with positional parameters — each one still carrying its type marker for correct binding. ### Deferring: cross-database value resolution Sometimes a query on one database needs a value from another database. That's what `defer()` is for — it wraps a query as a closure that executes later, when the outer query runs. This is not a subquery. The deferred query runs first, produces a concrete value, and that value gets bound as a regular parameter in the outer query. ```rust # use vantage_expressions::{ExprDataSource, Expression, ExpressiveEnum}; # use vantage_sql::sqlite::{AnySqliteType, SqliteDB}; # use vantage_sql::sqlite_expr; # async fn setup() -> (SqliteDB, SqliteDB) { # let config_db = SqliteDB::connect("sqlite::memory:").await.unwrap(); # for sql in [ # "CREATE TABLE config (key TEXT PRIMARY KEY, value INTEGER)", # "INSERT INTO config VALUES ('min_price', 150)", # ] { # config_db.execute(&sqlite_expr!(sql)).await.unwrap(); # } # let shop_db = SqliteDB::connect("sqlite::memory:").await.unwrap(); # for sql in [ # "CREATE TABLE product (name TEXT, price INTEGER)", # "INSERT INTO product VALUES ('Cheap', 50), ('Tart', 220), ('Pie', 299)", # ] { # shop_db.execute(&sqlite_expr!(sql)).await.unwrap(); # } # (config_db, shop_db) # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let (config_db, shop_db) = setup().await; // This doesn't execute yet — it's a closure let threshold_query = sqlite_expr!("SELECT value FROM config WHERE key = {}", "min_price"); let deferred_threshold = config_db.defer(threshold_query); // Use the deferred value as a parameter in a different database let shop_query = Expression::::new( "SELECT name FROM product WHERE price >= {} ORDER BY price", vec![ExpressiveEnum::Deferred(deferred_threshold)], ); // When shop_db.execute() runs: // 1. Resolves the deferred → calls config_db, gets 150 // 2. Replaces the Deferred param with Scalar(150) // 3. Flattens and binds: SELECT name FROM product WHERE price >= ?1 let result = shop_db.execute(&shop_query).await?; # Ok(()) # } ``` Your `execute()` implementation needs to resolve deferred parameters before flattening. Walk the parameter list, call `.call().await` on any `Deferred`, and leave `Scalar` and `Nested` untouched. The resolved value comes back as an untyped `AnySqliteType` (no variant marker), so it gets bound via JSON-inference. For SQLite this is fine — the loose type system handles it. For stricter databases, you may want `defer()` to preserve type information from the source query's result. ### Reading query results So far we've been calling `db.execute(&expr).await` which returns `AnySqliteType`. For a SELECT query, that value wraps a JSON array of row objects. To work with individual rows, you convert into Records: ```rust # use serde_json::Value as JsonValue; # use vantage_expressions::ExprDataSource; # use vantage_sql::sqlite::SqliteDB; # use vantage_sql::sqlite_expr; # use vantage_types::Record; # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite::memory:").await.unwrap(); # for sql in [ # "CREATE TABLE product (id TEXT PRIMARY KEY, name TEXT, price INTEGER)", # "INSERT INTO product VALUES ('a', 'Cheap', 50), ('b', 'Tart', 220), ('c', 'Pie', 299)", # ] { # db.execute(&sqlite_expr!(sql)).await?; # } let result = db.execute(&sqlite_expr!("SELECT * FROM product")).await?; // Result is AnySqliteType wrapping [{"id":"a","name":"Cheap","price":50}, ...] // Convert to records manually: let rows: Vec = match result.into_value() { ciborium::Value::Array(arr) => arr, _ => panic!("expected rows"), }; let record: Record = rows[0].clone().into(); # Ok(()) # } ``` That works but it's verbose. The `TryFrom` impls from Step 1 make this cleaner through `AssociatedExpression`. When you call `db.associate::(expr)`, you get an expression that knows its return type — `.get()` executes and converts in one step: ```rust # use serde_json::Value as JsonValue; # use vantage_expressions::ExprDataSource; # use vantage_sql::sqlite::SqliteDB; # use vantage_sql::sqlite_expr; # use vantage_types::Record; # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite::memory:").await.unwrap(); # for sql in [ # "CREATE TABLE product (id TEXT PRIMARY KEY, name TEXT, price INTEGER)", # "INSERT INTO product VALUES ('a', 'Cheap', 50), ('b', 'Tart', 220), ('c', 'Pie', 299)", # ] { # db.execute(&sqlite_expr!(sql)).await?; # } // Scalar — extracts single value from single-row result let count = db.associate::(sqlite_expr!("SELECT COUNT(*) FROM product")); assert_eq!(count.get().await?, 3); // Record — extracts first row let record: Record = db .associate(sqlite_expr!("SELECT * FROM product WHERE id = {}", "c")) .get().await?; # assert_eq!(record["name"], "Pie"); # Ok(()) # } ``` From a Record, you can deserialize into a struct. For the `#[entity]` path: ```rust # use vantage_expressions::ExprDataSource; # use vantage_sql::sqlite::{AnySqliteType, SqliteDB}; # use vantage_sql::sqlite_expr; # use vantage_types::{Record, TryFromRecord, entity}; # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let db = SqliteDB::connect("sqlite::memory:").await.unwrap(); # for sql in [ # "CREATE TABLE product (id TEXT PRIMARY KEY, name TEXT, price INTEGER)", # "INSERT INTO product VALUES ('a', 'Cheap', 50), ('b', 'Tart', 220), ('c', 'Pie', 299)", # ] { # db.execute(&sqlite_expr!(sql)).await?; # } #[entity(SqliteType)] struct Product { id: String, name: String, price: i64 } let record: Record = db .associate(sqlite_expr!("SELECT * FROM product WHERE id = {}", "c")) .get().await?; let product = Product::from_record(record)?; # assert_eq!(product.price, 299); # Ok(()) # } ``` Or for the serde path with `Record`: ```rust # use serde::Deserialize; # use serde_json::Value as JsonValue; # use vantage_expressions::ExprDataSource; # use vantage_sql::sqlite::SqliteDB; # use vantage_sql::sqlite_expr; # use vantage_types::{Record, TryFromRecord}; # #[tokio::main(flavor = "current_thread")] # async fn main() -> Result<(), Box> { # let db = SqliteDB::connect("sqlite::memory:").await?; # for sql in [ # "CREATE TABLE product (id TEXT PRIMARY KEY, name TEXT, price INTEGER)", # "INSERT INTO product VALUES ('a', 'Cheap', 50), ('b', 'Tart', 220), ('c', 'Pie', 299)", # ] { # db.execute(&sqlite_expr!(sql)).await?; # } #[derive(Deserialize)] struct Product { id: String, name: String, price: i64 } let record: Record = db .associate(sqlite_expr!("SELECT * FROM product WHERE id = {}", "c")) .get().await?; let product: Product = Product::from_record(record)?; # assert_eq!(product.name, "Pie"); # Ok(()) # } ``` Testing the failure modes (missing fields, NULL into required field, wrong types) can help spot issues in your implementation. ### Step 2 conclusion At this point you should have: 1. **A vendor macro** (`sqlite_expr!`, `surreal_expr!`, etc.) that produces `Expression` with typed parameters. 2. **Trait impls** in `src//impls/` — `DataSource` (marker) and `ExprDataSource` with `execute()` and `defer()`. 3. **Tests** in `tests//2_*.rs` covering: - INSERT with typed parameters, read back and verify - Multi-row INSERT using nested expressions and `from_vec` - Type marker verification (bool binds as bool, not as string "true") - Cross-database deferred value resolution - AssociatedExpression with scalar, Record, and entity results - Identifier quoting: basic names, reserved words, spaces, hyphens, unicode --- Source: https://vantage-ui.com/book/new-persistence/step3-operators.html # Step 3: Implement Operators Expressions let you build raw queries, but users shouldn't have to write `sqlite_expr!("{} > {}", (ident("price")), 100i64)` every time they want a condition. **Operators** give typed columns ergonomic methods like `.eq()`, `.gt()`, `.in_()` that produce your backend's native condition type. This step covers how to implement a vendor-specific operation trait for your persistence. ### Vendor-specific operation traits Each persistence defines its own operation trait that returns the backend's condition type directly. The trait is blanket-implemented for all `Expressive` where `T: Into`, so typed columns get the methods for free. For SQL backends, a macro generates the trait: ```text // In vantage-sql/src/sqlite/operation.rs define_sql_operation!( SqliteOperation, SqliteCondition, crate::sqlite::types::AnySqliteType ); ``` This produces: - A trait `SqliteOperation` with `.eq()`, `.gt()`, `.lt()`, `.ne()`, `.gte()`, `.lte()`, `.in_()`, `.in_list()`, `.cast()` — all returning `SqliteCondition` - A blanket impl for all `Expressive` where `T: Into` - An `Expressive` impl for `SqliteCondition`, enabling chaining ### How it works internally Each method builds an `Expression` from the two operands, then converts it to the backend's condition type via `From>`: ```text fn gt(&self, value: impl Expressive) -> SqliteCondition { let expr: Expression = Expression::new("{} > {}", vec![ ExpressiveEnum::Nested(self.expr()), ExpressiveEnum::Nested(value.expr()), ]); SqliteCondition::from(expr) // maps T → AnySqliteType via Into } ``` The `From> for SqliteCondition` impl (from Step 1's `define_sql_condition!` macro) handles the type mapping — it calls `ExpressionMap::map()` to convert all `F` scalars into `AnySqliteType`. ### Chaining across type boundaries Because `SqliteCondition` implements `Expressive`, the blanket gives it `SqliteOperation`. This enables: ```rust # use vantage_sql::sqlite::operation::SqliteOperation; # use vantage_table::column::core::Column; let price = Column::::new("price"); price.gt(10).eq(false) // => SqliteCondition wrapping: (price > 10) = 0 # ; ``` The first operation (`.gt(10)`) enforces type safety — `10` must be `Expressive`. The second operation (`.eq(false)`) operates on `SqliteCondition` where `bool: Expressive`, so any backend-compatible type is accepted. ### Implementing for a non-SQL backend For backends that don't use expression trees for conditions (like MongoDB), you implement the operation trait manually instead of using the macro. MongoDB produces BSON documents: ```text pub trait MongoOperation: Expressive { fn eq(&self, value: impl Into) -> MongoCondition { let field = self.expr().template.clone(); let bson_val = AnyMongoType::from(value).to_bson(); MongoCondition::Doc(doc! { field: { "$eq": bson_val } }) } fn gt(&self, value: impl Into) -> MongoCondition { let field = self.expr().template.clone(); let bson_val = AnyMongoType::from(value).to_bson(); MongoCondition::Doc(doc! { field: { "$gt": bson_val } }) } // ... } impl> MongoOperation for S {} ``` Key differences from SQL: - **Values use `Into`** not `Expressive` — MongoDB doesn't compose expression trees, it builds BSON documents from scalar values. - **Field name extraction** — `self.expr().template` gives the column name for simple columns. Complex expressions produce the template string as the field path. - **Chaining** — `MongoCondition` implements `Expressive` for the blanket, but boolean chaining (`.eq(false)` = negate) is handled via dedicated methods like `.eq_bool(false)` since MongoDB negation uses `$not` wrappers. ### Avoiding method name conflicts When multiple backend features are enabled, types like `Identifier` and `&str` implement `Expressive` for multiple backends. This causes ambiguity if the operation trait is generic. Each backend's operation trait lives in its own module (e.g. `sqlite::operation::SqliteOperation`). Users import only the trait they need: ```rust # mod sqlite { # pub mod operation { # pub use vantage_sql::sqlite::operation::SqliteOperation; # } # } // In your prelude: pub use crate::sqlite::operation::SqliteOperation; # fn main() {} ``` ### Condition type requirements Your condition type must satisfy `TableSource::Condition` bounds — `Clone + Send + Sync + 'static`. It also needs: - `From>` for any `F: Into` — so typed column operations convert cleanly - `From` — so `ident("field")` works with `with_condition()` - `Expressive` — so the condition can be chained with further operations - Any backend-specific conversions (e.g. `From` for MongoDB) For SQL backends, the `define_sql_condition!` macro generates all of these. ### Step 2b checklist 1. **Define your operation trait** — either via `define_sql_operation!` (SQL) or manually (document-oriented backends). 2. **Blanket-implement it** for all `Expressive` where `T` converts into your `AnyType`. 3. **Implement `Expressive` for your condition type** — enables chaining. 4. **Export from your prelude** — so users get the operation trait automatically. 5. **Tests** covering: - Typed column operations: `Column::::new("price").gt(150)` → condition - Boolean column: `Column::::new("active").eq(false)` → condition - Chaining: `price.gt(10).eq(false)` compiles and produces correct output - Cross-type rejection: `price.gt(false)` does NOT compile - Same-type column comparison: `price.eq(price.clone())` works - Condition usable with `table.add_condition()` and `select.with_condition()` --- Source: https://vantage-ui.com/book/new-persistence/step4-query-builder.html # Step 4: Statement Builders and SelectableDataSource In practice, nobody writes raw expressions for every query. This step adds the `Selectable` trait implementation for your SELECT builder and wires it up through `SelectableDataSource` so the rest of vantage can create and execute queries through a standard interface. ### Implement Selectable for your SELECT builder The `Selectable` trait is the standard interface for building SELECT queries across all vantage backends. Your SELECT struct needs to implement it. The trait has two kinds of methods: **Mutating methods** you must implement — `set_source`, `add_field`, `add_where_condition`, `add_order_by`, `add_group_by`, `set_limit`, `set_distinct`, the `clear_*` methods, the `has_*` methods, `as_count`, and `as_sum`. **Builder methods** you get for free — `with_source`, `with_field`, `with_condition`, `with_order`, `with_expression`, `with_limit`. These are default implementations that call the mutating methods and return `self`. This means your builder code is just the struct definition and `new()`: ```text pub struct SqliteSelect { pub fields: Vec, pub from: Vec, pub where_conditions: Vec, pub order_by: Vec<(Expr, bool)>, pub group_by: Vec, pub distinct: bool, pub limit: Option, pub skip: Option, } impl SqliteSelect { pub fn new() -> Self { /* initialize empty */ } } ``` The `Selectable` impl goes in its own file (e.g., `statements/select/impls/selectable.rs`) and the builder methods come from the trait: ```rust # use vantage_expressions::{Order, Selectable}; # use vantage_sql::sqlite::statements::SqliteSelect; # use vantage_sql::sqlite_expr; let select = SqliteSelect::new() .with_source("product") .with_field("name") .with_field("price") .with_condition(sqlite_expr!("\"is_deleted\" = {}", false)) .with_order(sqlite_expr!("\"price\""), Order::Desc) .with_limit(Some(10), None); ``` The `as_count()` and `as_sum()` methods should clone the current query, replace the fields with `COUNT(*)` or `SUM(column)`, drop the ORDER BY (unnecessary for aggregates), and render: ```rust # use vantage_expressions::{Order, Selectable}; # use vantage_sql::sqlite::statements::SqliteSelect; # use vantage_sql::sqlite_expr; # let select = SqliteSelect::new() # .with_source("product") # .with_condition(sqlite_expr!("\"is_deleted\" = {}", false)) # .with_order(sqlite_expr!("\"price\""), Order::Desc); let count_expr = select.as_count(); // SELECT COUNT(*) FROM product WHERE ... let sum_expr = select.as_sum(sqlite_expr!("\"price\"")); // SELECT SUM("price") FROM ... ``` ### Implement SelectableDataSource This trait connects the SELECT builder to execution. It tells vantage "this database can create SELECT queries and run them": ```text impl SelectableDataSource for SqliteDB { type Select = SqliteSelect; fn select(&self) -> Self::Select { SqliteSelect::new() } async fn execute_select(&self, select: &Self::Select) -> Result> { // delegate to ExprDataSource::execute() } } ``` ### Live tests Up to now, most tests used in-memory databases created in `setup()`. For Step 3, start running tests against a real pre-populated database. This catches issues that in-memory tests miss — schema mismatches, type affinity surprises, data edge cases. Set up a test database with known data (we use a bakery schema translated from SurrealDB's test fixture), and write tests that query it through the `Selectable` interface: ```rust,no_run # use vantage_expressions::{ExprDataSource, Expressive, Order, Selectable}; # use vantage_sql::sqlite::statements::SqliteSelect; # use vantage_sql::sqlite::{AnySqliteType, SqliteDB}; # use vantage_sql::sqlite_expr; # use vantage_types::{Record, TryFromRecord, entity}; # #[entity(SqliteType)] # struct Product { name: String, price: i64 } # #[tokio::main(flavor = "current_thread")] # async fn main() -> Result<(), Box> { let db = SqliteDB::connect("sqlite:../target/bakery.sqlite?mode=ro").await?; let select = SqliteSelect::new() .with_source("product") .with_condition(sqlite_expr!("\"price\" > {}", 200i64)) .with_order(sqlite_expr!("\"price\""), Order::Desc); let record: Record = db.associate(select.expr()).get().await?; let product = Product::from_record(record)?; assert_eq!(product.name, "Enchantment Under the Sea Pie"); # Ok(()) # } ``` ### Implementing complex queries The `Selectable` interface gives you the bare bones — fields, conditions, ordering, limits, aggregates. Your database can do much more: JOINs, subqueries, CTEs, window functions, HAVING, UNION, JSON operators, CASE expressions. The best way to build this out is incrementally, driven by real queries: 1. **Create a test database scaffold** with enough tables and data to exercise complex features. Foreign keys, self-referential hierarchies, many-to-many junctions, JSON columns, generated columns — the more variety, the better. (See `scripts/sqlite/db/v3.sql` for an example.) 2. **Write the queries first** in raw SQL, as comments in your test file. Start with queries you know work against your scaffold. Each query should target specific features (JOINs, GROUP BY + HAVING, window functions, etc.). 3. **Implement one query at a time.** For each query: - Read the SQL and identify which builder methods are missing - Add the methods to your SELECT struct (e.g., `add_join`, `add_having`, `with_cte`) - Write a render test that checks the generated SQL matches - Write a live test that executes against the scaffold and verifies results This approach has two advantages. First, you don't over-design — you only add features you actually need. Second, every new feature ships with a test that proves it works against a real database, not just string comparison. When your queries outgrow what the select struct offers directly, extract **primitives** and **nested structs** rather than bloating the builder. For example, `Identifier` (in `vantage-sql/src/primitives/`) handles qualified column names (`"u"."name"`) and aliases — it implements `Expressive` so it plugs straight into expressions. Similarly, `SqliteSelectJoin` lives inside the select module and renders its own `INNER JOIN ... ON ...` clause. The select struct just holds a `Vec` and calls `render()` on each one. This pattern — small struct with `Expressive` impl, composed into the builder — scales to CASE, CTE, window specs, and anything else without the select struct growing unbounded. ### Other statements `Selectable` only covers SELECT. INSERT, UPDATE, and DELETE don't need a trait at this stage — they just need to implement `Expressive` so they can be passed to `ExprDataSource::execute()`. The statement builders from earlier steps still work, they just need their expression type migrated from `JsonValue` to `AnySqliteType` to flow directly into `execute()`. ### Step 3 conclusion At this point you should have: 1. **`Selectable` impl** for your SELECT builder — all standard methods implemented, builder pattern provided by trait defaults. 2. **`SelectableDataSource` impl** for your DB struct — `select()` and `execute_select()`. 3. **Tests** in `tests//3_*.rs` covering: - SQL rendering via `preview()` — fields, conditions, ordering, limits, distinct, group by - `as_count()` and `as_sum()` render correctly - Live execution against a test database — SELECT, COUNT, SUM, ORDER+LIMIT - Entity deserialization from live query results --- Source: https://vantage-ui.com/book/new-persistence/step5-table-crud.html # Step 5: Table Abstraction and Entity CRUD The same entities get used hundreds of times across a codebase — constructing a query from scratch every single time is tedious and error-prone. Vantage offers `Table<>` as an abstraction over your entity definitions: it knows the table name, the columns, their types, and the ID field, so it can build queries for you. To use your persistence backend as a table source, you need to implement the `TableSource` trait. Most of the heavy-lifting is done by the `vantage-table` crate — your job is to implement `TableSource` trait methods. ### Implement TableSource with placeholder methods Start by adding the required dependencies: ```toml # in your backend's Cargo.toml vantage-table = { path = "../vantage-table" } async-trait = "0.1" ``` Create a new test file (e.g. `tests//4_table_def.rs`) that defines a table and populates its columns. The columns rely on the type system you built in Step 1. The `TableSource` implementation also declares several associated types: - **`Column`** — the `Column` type supplied by `vantage-table` is good enough for most backends. - **`AnyType`** and **`Value`** — your type-erased value type from Step 1 (e.g. `AnySqliteType`). - **`Id`** — use `String` for SQL databases, or a custom type if your IDs have special structure (e.g. SurrealDB's `Thing` which encodes `table:id`). Whatever you pick must be covered by your type system. ```text use async_trait::async_trait; use vantage_table::column::core::{Column, ColumnType}; use vantage_table::traits::table_source::TableSource; #[async_trait] impl TableSource for SqliteDB { type Column = Column where Type: ColumnType; type AnyType = AnySqliteType; type Value = AnySqliteType; type Id = String; // ... } ``` Implement the following methods first — they're all straightforward delegations: - **Column management** — `create_column`, `to_any_column`, `convert_any_column`: ```text fn create_column(&self, name: &str) -> Self::Column { Column::new(name) } fn to_any_column( &self, column: Self::Column, ) -> Self::Column { Column::from_column(column) } fn convert_any_column( &self, any_column: Self::Column, ) -> Option> { Some(Column::from_column(any_column)) } ``` - **Expression factory** — `expr()`: ```text fn expr( &self, template: impl Into, parameters: Vec>, ) -> Expression { Expression::new(template, parameters) } ``` Every other method — should start as `todo!()`. You'll implement them incrementally in the following sections, driven by tests. ### Define entity tables With `TableSource` in place, define your entity structs and table constructors. The pattern is the same across all backends — `#[entity(YourType)]` for the struct, plus a builder method that returns `Table`: ```rust use vantage_sql::sqlite::{SqliteType, SqliteDB, AnySqliteType}; use vantage_table::table::Table; use vantage_types::entity; #[entity(SqliteType)] #[derive(Debug, Clone, PartialEq, Default)] struct Product { name: String, calories: i64, price: i64, bakery_id: String, is_deleted: bool, inventory_stock: i64, } impl Product { fn sqlite_table(db: SqliteDB) -> Table { Table::new("product", db) .with_id_column("id") .with_column_of::("name") .with_column_of::("calories") .with_column_of::("price") .with_column_of::("bakery_id") .with_column_of::("is_deleted") .with_column_of::("inventory_stock") } } # #[tokio::main(flavor = "current_thread")] # async fn main() -> Result<(), Box> { # let table = Product::sqlite_table(SqliteDB::connect("sqlite::memory:").await?); # assert_eq!(table.columns().len(), 7); # Ok(()) # } ``` Note that the entity struct does **not** include the `id` field — that's handled separately by `with_id_column()`, which registers the column and sets the table's ID field. The remaining columns are added with `with_column_of::()`, which creates typed columns via your `TableSource::create_column` implementation. ### Verify with a query generation test Your first test should build a table, then call `table.select()`. Just like the Step 3 tests, you can use `preview()` to check the rendered SQL, and later execute it against a real database: ```rust # use vantage_sql::sqlite::{AnySqliteType, SqliteDB}; # use vantage_table::table::Table; # use vantage_types::entity; # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # struct Product { # name: String, # calories: i64, # price: i64, # bakery_id: String, # is_deleted: bool, # inventory_stock: i64, # } # impl Product { # fn sqlite_table(db: SqliteDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("calories") # .with_column_of::("price") # .with_column_of::("bakery_id") # .with_column_of::("is_deleted") # .with_column_of::("inventory_stock") # } # } #[tokio::test] async fn test_product_select() { let db = SqliteDB::connect("sqlite::memory:").await.unwrap(); let table = Product::sqlite_table(db); let select = table.select(); assert_eq!( select.preview(), "SELECT \"id\", \"name\", \"calories\", \"price\", \ \"bakery_id\", \"is_deleted\", \"inventory_stock\" FROM \"product\"" ); } ``` This works because `table.select()` (provided by `vantage-table`) calls your `SelectableDataSource::select()` to get a fresh SELECT builder, then applies the table name via `set_source()` and adds each registered column via `add_field()`. None of the `todo!()` methods are hit — only the column and expression infrastructure you already implemented. ### Implement the read methods `Table` implements two traits from `vantage-dataset` that provide read access: - **`ReadableValueSet`** — returns raw `Record` (untyped storage values): - `list_values()` → all records as `IndexMap>` - `get_value(id)` → `Option>` — `None` if no record matches the id - `get_some_value()` → one arbitrary record (or `None` if empty) - **`ReadableDataSet`** — returns deserialized entities (calls `E::try_from_record()` for you): - `list()` → all entities as `IndexMap` - `get(id)` → `Option` — `None` if no entity matches the id - `get_some()` → one arbitrary entity Both traits delegate to three `TableSource` methods: `list_table_values`, `get_table_value`, and `get_table_some_value`. The pattern is the same for all three: 1. Get the id field name from `table.id_field()` (falls back to `"id"`) 2. Build a SELECT using `table.select()` (which already applies columns, conditions, ordering) 3. Execute via `self.execute(&select.expr())` 4. Parse the result — split each row into an ID and a `Record` For `get_table_value`, add a WHERE condition on the id field and return `Ok(None)` when the lookup misses — errors are reserved for actual connection or parse failures. For `get_table_some_value`, set `LIMIT 1` and return the first row (or `None` if empty). Write tests for both `ReadableValueSet` and `ReadableDataSet` in separate files — import the traits from `vantage_dataset` and call `list_values()`, `get_value()`, `get_some_value()`, `list()`, `get()`, `get_some()` against your pre-populated test database. Keep these tests condition-free — conditions get their own test file next. ### Error handling All `TableSource` methods return `vantage_core::Result` (an alias for `Result`). Use the `error!` macro from `vantage_core` to create errors with structured context: ```text use vantage_core::error; // Simple error message return Err(error!("expected array result")); // With key = value context (NOT format args — the macro uses a different syntax) return Err(error!("row missing id field", field = id_field_name)); // For database-specific errors, convert them with map_err let rows = query.fetch_all(self.pool()).await .map_err(|e| error!("SQLite query failed", details = e.to_string()))?; ``` The macro automatically captures file, line, and column. The `key = value` pairs are stored as structured context, not interpolated into the message string. To wrap external errors with additional context, use the `Context` trait: ```rust,no_run # use vantage_core::error; # fn main() -> vantage_core::Result<()> { use vantage_core::Context; // Wraps the original error as the "source" of a new VantageError let data = std::fs::read("config.json") .context(error!("failed to load config"))?; # Ok(()) # } ``` This chains errors — the original `io::Error` is preserved as the source, so `Display` renders both messages and the source chain is available via `std::error::Error::source()`. ### Operation trait — condition building Each backend provides an operation trait (e.g. `SqliteOperation`) with `.eq()`, `.ne()`, `.gt()`, `.gte()`, `.lt()`, `.lte()`, and `.in_()` methods for building conditions. It has a **blanket implementation** for all `Expressive` types, so your columns get these methods automatically — no explicit impl needed. All methods accept `impl Expressive`, so you can pass native Rust values (`false`, `42`, `"hello"`), other columns (`table["other_field"]`), or full expressions. This requires your scalar types to implement `Expressive` — the same impls you added in Step 1 for the vendor macro. ### Testing conditions `Table` carries conditions set via `add_condition()`, and `table.select()` applies them automatically as WHERE clauses. Test a few patterns: - **Custom expression** — pass columns as expression arguments via `table["field"]`: ```rust # use vantage_sql::sqlite::{AnySqliteType, SqliteDB}; # use vantage_sql::sqlite_expr; # use vantage_table::table::Table; # use vantage_types::entity; # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # struct Product { # name: String, # calories: i64, # price: i64, # bakery_id: String, # is_deleted: bool, # inventory_stock: i64, # } # impl Product { # fn sqlite_table(db: SqliteDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("calories") # .with_column_of::("price") # .with_column_of::("bakery_id") # .with_column_of::("is_deleted") # .with_column_of::("inventory_stock") # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> Result<(), Box> { # let db = SqliteDB::connect("sqlite::memory:").await?; let mut table = Product::sqlite_table(db); table.add_condition(sqlite_expr!("{} > {}", (table["price"]), 130)); # Ok(()) # } ``` - **Multiple conditions** — combined with AND, including field-to-field comparison: ```rust # use vantage_sql::sqlite::{AnySqliteType, SqliteDB}; # use vantage_sql::sqlite_expr; # use vantage_table::table::Table; # use vantage_types::entity; # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # struct Product { # name: String, # calories: i64, # price: i64, # bakery_id: String, # is_deleted: bool, # inventory_stock: i64, # } # impl Product { # fn sqlite_table(db: SqliteDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("calories") # .with_column_of::("price") # .with_column_of::("bakery_id") # .with_column_of::("is_deleted") # .with_column_of::("inventory_stock") # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> Result<(), Box> { # let db = SqliteDB::connect("sqlite::memory:").await?; let mut table = Product::sqlite_table(db); table.add_condition(sqlite_expr!("{} > {}", (table["price"]), 130)); table.add_condition(sqlite_expr!("{} > {}", (table["price"]), (table["calories"]))); # Ok(()) # } ``` - **SqliteOperation::eq()** — the idiomatic way: ```rust # use vantage_sql::sqlite::{AnySqliteType, SqliteDB}; # use vantage_table::table::Table; # use vantage_types::entity; # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # struct Product { # name: String, # calories: i64, # price: i64, # bakery_id: String, # is_deleted: bool, # inventory_stock: i64, # } # impl Product { # fn sqlite_table(db: SqliteDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("calories") # .with_column_of::("price") # .with_column_of::("bakery_id") # .with_column_of::("is_deleted") # .with_column_of::("inventory_stock") # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> Result<(), Box> { # let db = SqliteDB::connect("sqlite::memory:").await?; use vantage_sql::sqlite::operation::SqliteOperation; let mut table = Product::sqlite_table(db); table.add_condition(table["is_deleted"].eq(false)); # Ok(()) # } ``` ### Implement aggregates Implement `get_table_count`, `get_table_sum`, `get_table_max`, and `get_table_min` in your `TableSource`. These build aggregate queries from `table.select()` and extract the scalar result. Once implemented, `Table` exposes shorter `get_count`, `get_sum`, `get_max`, `get_min` methods directly: ```rust,no_run # use vantage_sql::sqlite::{AnySqliteType, SqliteDB}; # use vantage_table::table::Table; # use vantage_types::entity; # #[entity(SqliteType)] # #[derive(Debug, Clone, PartialEq, Default)] # struct Product { # name: String, # calories: i64, # price: i64, # bakery_id: String, # is_deleted: bool, # inventory_stock: i64, # } # impl Product { # fn sqlite_table(db: SqliteDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("calories") # .with_column_of::("price") # .with_column_of::("bakery_id") # .with_column_of::("is_deleted") # .with_column_of::("inventory_stock") # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> Result<(), Box> { # let db = SqliteDB::connect("sqlite:../target/bakery.sqlite?mode=ro").await?; let table = Product::sqlite_table(db); assert_eq!(table.get_count().await.unwrap(), 5); assert_eq!(table.get_max(&table["price"]).await.unwrap().try_get::().unwrap(), 299); # Ok(()) # } ``` ### Implement write operations `Table` also implements `WritableDataSet` (insert, replace, patch, delete) and `InsertableDataSet` (insert with auto-generated ID). These delegate to six `TableSource` methods: - **`insert_table_value`** — INSERT with a known ID. Build an `SqliteInsert` with the id field and record fields, execute, then read back via `get_table_value`. - **`replace_table_value`** — full replacement. For SQLite, use `INSERT OR REPLACE INTO`. - **`patch_table_value`** — partial update. Build an `SqliteUpdate` with only the provided fields and a WHERE condition on the id field. - **`delete_table_value`** — DELETE with a WHERE condition on the id field. - **`delete_table_all_values`** — DELETE without conditions. - **`insert_table_return_id_value`** — INSERT without a known ID (auto-increment). Use `RETURNING "id"` to get the generated ID back from the database. Test both `WritableValueSet` (raw records, no entity) and `WritableDataSet` (typed entities) using in-memory SQLite: ```rust,no_run # use vantage_dataset::{InsertableDataSet, ReadableDataSet, WritableDataSet, WritableValueSet}; # use vantage_sql::sqlite::{AnySqliteType, SqliteDB}; # use vantage_table::table::Table; # use vantage_types::{Record, entity}; # #[entity(SqliteType)] # #[derive(Debug, Clone)] # struct Item { # name: String, # price: i64, # } # fn record(pairs: &[(&str, AnySqliteType)]) -> Record { # pairs.iter().map(|(k, v)| (k.to_string(), v.clone())).collect() # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> Result<(), Box> { # let db = SqliteDB::connect("sqlite::memory:").await?; # let table = Table::::new("item", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price"); // WritableValueSet — no entity needed let rec = record(&[("name", "Gamma".into()), ("price", 30i64.into())]); table.insert_value(&"c".to_string(), &rec).await.unwrap(); // WritableDataSet — typed entities let item = Item { name: "Gamma".into(), price: 30 }; table.insert(&"c".to_string(), &item).await.unwrap(); // InsertableDataSet — auto-generated ID let id = table.insert_return_id(&item).await.unwrap(); let fetched = table.get(id).await.unwrap(); # Ok(()) # } ``` --- Source: https://vantage-ui.com/book/new-persistence/step6-relationships.html # Step 6: Relationships Tables can declare relationships using `with_one` and `with_many`, then traverse them at runtime with `get_ref_as`. The relationship system is provided by `vantage-table` — your backend just needs `column_table_values_expr` implemented to make it work. ```admonish tip This step covers what a *driver* must implement for relationships. For the *consumer* side — traversing from a loaded record with `get_ref`, and the foreign-key invariants traversal sets up — see [Records: Traversal, Invariants & Hooks](../record-lifecycle.md). ``` Implement `column_table_values_expr` — it builds a subquery for a single column respecting current conditions. For SQL backends this is a `SELECT "col" FROM "table" WHERE ...` expression. Define relationships when constructing tables — `with_one` for foreign-key-to-parent, `with_many` for parent-to-children. Then traverse: ```rust,no_run # use vantage_dataset::ReadableDataSet; # use vantage_sql::sqlite::{AnySqliteType, SqliteDB}; # use vantage_sql::sqlite_expr; # use vantage_table::table::Table; # use vantage_types::{EmptyEntity, entity}; # #[entity(SqliteType)] # #[derive(Debug, Clone)] # struct ClientOrder { # client_id: String, # } # fn client_order_table(db: SqliteDB) -> Table { # Table::new("client_order", db) # .with_id_column("id") # .with_column_of::("client_id") # } # fn client_table(db: SqliteDB) -> Table { # Table::new("client", db) # .with_id_column("id") # .with_column_of::("is_paying_client") # .with_many("orders", "client_id", client_order_table) # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> Result<(), Box> { # let db = SqliteDB::connect("sqlite:../target/bakery.sqlite?mode=ro").await?; let mut clients = client_table(db); clients.add_condition(sqlite_expr!("{} = {}", (clients["is_paying_client"]), true)); let orders = clients.get_ref_as::("orders").unwrap(); // The generated query includes the subquery: // SELECT ... FROM "client_order" // WHERE client_id IN (SELECT "id" FROM "client" WHERE is_paying_client = 1) assert_eq!(orders.list().await.unwrap().len(), 3); # Ok(()) # } ``` `with_expression` adds computed fields to a table using correlated subqueries. It pairs with `get_subquery_as` which produces `target.fk = source.id` conditions (vs `get_ref_as` which uses `IN (subquery)`). ```text .with_many("orders", "client_id", Order::sqlite_table) .with_expression("order_count", |t| { t.get_subquery_as::("orders").unwrap().get_count_query() }) // Generates: (SELECT COUNT(*) FROM "client_order" // WHERE "client_order"."client_id" = "client"."id") AS "order_count" ``` **What to implement:** override `related_correlated_condition` in your `TableSource` to produce table-qualified equality. Default panics — backends without correlated subquery support (CSV) simply can't use this feature. ```text fn related_correlated_condition(&self, target_table: &str, target_field: &str, source_table: &str, source_column: &str) -> Self::Condition { sqlite_expr!("{} = {}", (ident(target_field).dot_of(target_table)), (ident(source_column).dot_of(source_table))) } ``` Requires `SelectableDataSource` (Step 3) since aggregate query builders use `table.select()`. --- Source: https://vantage-ui.com/book/new-persistence/step7-multi-backend.html # Step 7: Multi-Backend Applications At this point your backend works — you can define tables, query data, and traverse relationships. But a real application typically has a _model crate_ that defines entities once and offers table constructors for each backend. That's `bakery_model3` in the Vantage repo. The final piece is type erasure, which lets you treat tables from different backends uniformly. ```admonish info title="AnyTable is gone — use Vista" Earlier versions erased the backend with `AnyTable`. It was removed in 0.5.2; type erasure now lives one layer up in [`Vista`](./step8-vista-integration.md), reached through the driver's vista factory. The mechanism below is the supported replacement. ``` ### Vista: the type-erased handle `db.vista_factory().from_table(table)` erases the backend and entity types behind a uniform, schema-bearing [`Vista`](./step8-vista-integration.md) carrying `Record`. This is what makes it possible to write generic UI, CLI, or API code that doesn't care which database is behind it: ```rust # use vantage_sql::sqlite::{AnySqliteType, SqliteDB}; # use vantage_table::table::Table; # use vantage_types::entity; # #[entity(SqliteType)] # #[derive(Debug, Clone)] # struct Product { # name: String, # price: i64, # } # impl Product { # fn sqlite_table(db: SqliteDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # .with_column_of::("price") # } # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> Result<(), Box> { # let db = SqliteDB::connect("sqlite::memory:").await?; // Wrap any typed Table as a backend-agnostic Vista: let products = db.vista_factory().from_table(Product::sqlite_table(db.clone()))?; // (or surreal/csv/mongo — the call site is identical) # Ok(()) # } ``` Erasure works because each driver's vista factory bridges its native `AnyType` to the CBOR value the Vista exposes — the `vantage_type_system!` macro and your Step 1 conversions already provide both directions. See [Step 8](./step8-vista-integration.md) for the factory and `TableShell` you implement to enable this. ### Building a multi-source CLI The CLI example in `bakery_model3/examples/cli.rs` shows the pattern. A `build_table` function matches on the user's chosen source, calls the right entity constructor, and wraps it through the vista factory. Once you have a `Vista`, all commands are backend-agnostic — listing, counting, reading, inserting, and deleting all work identically regardless of which database is behind it. Because the values flow through as a typed CBOR record, the CLI renderer can inspect types at runtime — booleans like `is_deleted` display as `true`/`false` with color highlighting, numbers stay numeric, and nulls render cleanly. Your type system work in Step 1 ensures these values arrive with the right type rather than everything being a string. Try it out: ```bash # List products from CSV cargo run --example cli -- csv product list # Same thing from SQLite cargo run --example cli -- sqlite product list # Count bakeries in SurrealDB cargo run --example cli -- surreal bakery count # Get a single product record cargo run --example cli -- sqlite product get # Insert a new record cargo run --example cli -- surreal bakery add myid '{"name":"Test","profit_margin":10}' # Delete a record cargo run --example cli -- surreal bakery delete myid ``` That's the payoff of implementing a proper type system and `TableSource` — one line of `vista_factory().from_table()` bridges the gap between your backend's native types and a uniform record-based interface. --- Source: https://vantage-ui.com/book/new-persistence/step8-vista-integration.html # Step 8: Vista Integration By the end of Step 6 your backend is a typed persistence: `Table`, conditions, relationships. That's what business logic wants, and it's exactly what generic code can't use. A CLI that lists "any table from any backend" can't carry an entity generic. A web admin that draws forms from a YAML schema doesn't know your `Product` struct exists. A Rhai script that filters rows by a column name shouldn't compile against your backend's condition type. **Vista** is the bridge: a schema-bearing handle over one table of one backend, with CBOR values and string ids at its surface. This step adds two pieces to your backend: a **shell** that executes reads, narrowing and writes for a Vista, and a **factory** that builds a Vista from a typed `Table` or a YAML spec. ### What Vista is [`Vista`](vantage_vista::Vista) is a concrete struct, not a trait. It holds a name, the clauses applied to it (a [`VistaQuery`](vantage_vista::VistaQuery)) and a boxed [`TableShell`](vantage_vista::TableShell): your driver's executor. The schema and the capabilities live on the shell. `Vista` reads them from there and checks every call against them before the shell sees it. A `Vista` is a value. `vista.clone().with_condition(..)` gives a narrowed set and leaves `vista` alone, and it does that by cloning the shell and narrowing the clone. That's why `TableShell` has `Clone` as a supertrait: a shell's clone must carry its current narrowing and own it, while sharing the connection or the store. ### Why not hand around `Table`? You can, and wherever the entity is known at compile time you should: it's typed, it composes, and it's what the previous steps built. Vista is for the cases where the entity isn't known at compile time, or the backend is chosen at runtime: - a CLI driven by `--source surreal --table client list`, with no `Client` struct in scope; - an admin tool that reads its schema from YAML; - a Rhai callback that filters rows by a column name. The price of erasure is that values become `CborValue` and ids become `String`. Generic code renders values; it doesn't deserialize them into structs, so it doesn't miss the types. ### Setting it up The bridge is opt-in behind a `vista` feature, so a crate that only uses `Table` doesn't pull in `vantage-vista`. The contract checks used in the tests come from vantage-vista's `contract` feature, enabled for tests only: ```toml [features] default = [] vista = ["dep:vantage-vista"] [dependencies] vantage-vista = { path = "../vantage-vista", optional = true } [dev-dependencies] vantage-vista = { path = "../vantage-vista", features = ["contract"] } ``` Everything Vista-related sits under `#[cfg(feature = "vista")]`. CSV's layout is the one to copy: ```text /src/vista.rs re-exports, and ::vista_factory() /src/vista/spec.rs TableExtras, ColumnExtras, VistaSpec /src/vista/factory.rs VistaFactory and its impl VistaFactory /src/vista/source.rs TableShell and its impl TableShell ``` A backend whose native values aren't CBOR-shaped also needs a bridge between the two. MongoDB keeps its BSON ↔ CBOR conversion in `types/cbor.rs`. #### The shell Here is a complete shell over rows held in memory. It's the smallest backend there is, so everything in it is the trait's shape rather than a driver's detail. It evaluates every clause with the same evaluator a cache uses ([`VistaQuery::matches`](vantage_vista::VistaQuery::matches) and [`VistaQuery::sort_rows`](vantage_vista::VistaQuery::sort_rows)), so it can advertise operator filters, search and order: ```rust use std::sync::Arc; use async_trait::async_trait; use indexmap::IndexMap; use vantage_core::Result; use vantage_types::Record; use vantage_vista::{ Capability, CborValue, Column, ColumnFlag, Condition, FilterOp, Page, Preview, SortDirection, TableShell, Vista, VistaCapabilities, VistaMetadata, VistaQuery, condition_column_error, }; type Rec = Record; /// Rows held in memory. A clone shares the rows and owns its narrowing. #[derive(Clone)] struct VecShell { rows: Arc>, query: VistaQuery, metadata: VistaMetadata, capabilities: VistaCapabilities, } impl VecShell { fn new(rows: Arc>, metadata: VistaMetadata) -> Self { let capabilities = [ Capability::FilterOps, Capability::Search, Capability::Order, Capability::Window, ] .into_iter() .collect(); Self { rows, query: VistaQuery::new(), metadata, capabilities } } /// The rows the narrowing keeps, in its order. fn set(&self) -> Result> { let mut rows = Vec::new(); for (id, row) in self.rows.iter() { if self.query.matches(row, &self.metadata)? { rows.push((id.clone(), row.clone())); } } self.query.sort_rows(&mut rows, &self.metadata); Ok(rows) } /// A declared column, or a dotted path into one. fn check_column(&self, column: &str) -> Result<()> { let root = column.split('.').next().unwrap_or(column); let columns = &self.metadata.columns; if columns.contains_key(column) || columns.contains_key(root) { Ok(()) } else { Err(condition_column_error(&self.metadata, column)) } } } #[async_trait] impl TableShell for VecShell { fn metadata(&self) -> &VistaMetadata { &self.metadata } fn capabilities(&self) -> &VistaCapabilities { &self.capabilities } fn driver_name(&self) -> &'static str { "vec" } fn preview(&self) -> Preview { Preview::new().with_note("every clause is evaluated in memory") } async fn list(&self) -> Result> { Ok(self.set()?.into_iter().collect()) } async fn get(&self, id: &str) -> Result> { Ok(self.set()?.into_iter().find(|(key, _)| key == id).map(|(_, row)| row)) } async fn count(&self) -> Result { Ok(self.set()?.len() as i64) } async fn fetch_window(&self, offset: usize, limit: usize) -> Result { Ok(Page::from_listing(self.set()?, offset, limit)) } fn add_condition(&mut self, condition: &Condition) -> Result<()> { self.check_column(&condition.column)?; self.query.conditions.push(condition.clone()); Ok(()) } fn set_search(&mut self, text: Option<&str>) -> Result<()> { self.query.search = text.map(str::to_string); Ok(()) } fn set_order(&mut self, order: Option<(&str, SortDirection)>) -> Result<()> { if let Some((column, _)) = order { self.check_column(column)?; } self.query.order = order.map(|(column, dir)| (column.to_string(), dir)); Ok(()) } } fn row(id: &str, name: &str, category: &str) -> (String, Rec) { let cells = [("id", id), ("name", name), ("category", category)]; let record = cells.into_iter().map(|(k, v)| (k.to_string(), CborValue::from(v))).collect(); (id.to_string(), record) } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let metadata = VistaMetadata::new() .with_column(Column::new("id", "string").with_flag(ColumnFlag::Orderable)) .with_column(Column::new("name", "string").with_flag(ColumnFlag::Searchable)) .with_column(Column::new("category", "string")) .with_id_column("id"); let rows = Arc::new(vec![ row("p1", "Tea", "drink"), row("p2", "Cake", "food"), row("p3", "Coffee", "drink"), ]); let products = Vista::new("product", VecShell::new(rows, metadata)); let drinks = products.clone().with_condition("category", FilterOp::Eq, "drink")?; assert_eq!(drinks.count().await?, 2); assert_eq!(products.count().await?, 3); # Ok(()) # } ``` The methods split into what every shell answers and what it opts into. **Every shell** answers `metadata`, `capabilities`, `driver_name`, `preview`, `list`, `get`, `count`, `fetch_window` and `add_condition` with equality. - `metadata` is the schema. A shell over a typed table doesn't build it by hand: CSV's factory asks the table for it with `table.vista_metadata(VistaMetadataOptions { references: true, .. })`, which carries the column types, the id and title columns and the relations. - `fetch_window` with `offset == 0` is how a capped read and `get_some_value` read, so every shell that lists must answer it. A backend that can only list cuts the window from its listing with [`Page::from_listing`](vantage_vista::Page::from_listing), which also fills the page's total. A backend that pages natively pushes `LIMIT`/`OFFSET` and sets the total only when the fetch learned it (`Page::new(rows).with_total(n)`). - `add_condition` refuses a column the shell doesn't take with [`condition_column_error`](vantage_vista::condition_column_error). That gives a plain unknown-column error for a column the schema doesn't declare, and an `Unimplemented` refusal for a dotted path the shell can't evaluate. The kinds matter: `Vista::with_query` hands a refused clause back to the caller to evaluate locally, and passes a plain error on. A shell that errors must leave its narrowing unchanged, which is why `VecShell` checks before it pushes. - `preview` renders the query the next read would send, without sending it. Lossy is fine, wrong is not: it must not show a clause the shell wouldn't send. **A shell opts into** the rest by implementing the method and advertising its [`Capability`](vantage_vista::Capability). Each default answers [`unsupported`](vantage_vista::unsupported) with the capability that names it: | Capability | Methods | | ------------------ | -------------------------------------------------- | | `Insert` | `insert` | | `Update` | `replace`, `upsert`, `patch` | | `Delete` | `delete`, `delete_all` | | `Import` | `import`, one atomic batch that counts new rows | | `Subscribe` | `watch` | | `Order` | `set_order` | | `Search` | `set_search` | | `FilterOps` | `add_condition` with any operator other than `Eq` | | `NativeConditions` | `add_native_condition` | | `Window` | `fetch_window` at any offset | | `Cursor` | `fetch_cursor` | | `Traverse` | `get_ref` | | `ConfineWrites` | every write keeps to the narrowed set | `Vista` refuses most of these before the shell is called: writes, search, order, operator filters, windows past the first, cursors and subscriptions. `get_ref` and `add_native_condition` reach the shell, and the default refuses them there. CSV pushes only equality, so it doesn't advertise `FilterOps`, and a `Gt` never reaches it. Its `add_condition` still answers `unsupported(Capability::FilterOps)` for any other operator, so the shell is correct on its own. Two of the opt-in methods need more than the table above says: - `get_ref(relation, row)` is the one traversal method. With a parent row it returns the related rows; with `None` it returns the bare target, which is what an edit form lists to pick a related record from. A shell over a typed table forwards both to [`Table::traverse_erased`](vantage_table::table::Table::traverse_erased), passing its own factory to wrap the target, and answers an undeclared relation with [`unknown_relation`](vantage_vista::unknown_relation). CSV's `get_ref` looks the reference up in its metadata and makes that one call. Contained relations route through the same method (Step 9). - `add_native_condition` takes a [`NativeCondition`](vantage_vista::NativeCondition): a condition in your driver's own type, passed through Vista unread. Downcast it to the type you take. Another type is the caller's mistake, so it's an `IncorrectUsage` error, not `Unsupported`. #### The factory The factory is where both construction paths meet. Expose it as an inherent `vista_factory()` on the data source, so callers don't name an extra type. It carries two entry points: - `from_table(table: Table) -> Result`, inherent, because putting it on the trait would make vantage-vista depend on vantage-table; - `impl VistaFactory`, whose `build_from_spec` lowers a parsed YAML spec. The default `from_yaml` parses the YAML and calls it. Keep them on one path. CSV's `build_from_spec` builds a `Table` from the spec, hands it to `from_table`, and renames the result with `with_name(spec.name)`, so a spec called `client` over a file `clients.csv` shows as `client`. A capability added later is then a one-line edit in one place, and the two paths can't drift apart. For the memory shell above there is no typed table, so the factory lowers the spec straight into metadata: ```rust # use std::sync::Arc; # use async_trait::async_trait; # use indexmap::IndexMap; # use vantage_core::Result; # use vantage_types::Record; # use vantage_vista::{ # Capability, CborValue, Column, Condition, Page, Preview, SortDirection, TableShell, Vista, # VistaCapabilities, VistaMetadata, VistaQuery, condition_column_error, # }; # type Rec = Record; # #[derive(Clone)] # struct VecShell { # rows: Arc>, # query: VistaQuery, # metadata: VistaMetadata, # capabilities: VistaCapabilities, # } # impl VecShell { # fn new(rows: Arc>, metadata: VistaMetadata) -> Self { # let capabilities = [ # Capability::FilterOps, # Capability::Search, # Capability::Order, # Capability::Window, # ] # .into_iter() # .collect(); # Self { rows, query: VistaQuery::new(), metadata, capabilities } # } # fn set(&self) -> Result> { # let mut rows = Vec::new(); # for (id, row) in self.rows.iter() { # if self.query.matches(row, &self.metadata)? { # rows.push((id.clone(), row.clone())); # } # } # self.query.sort_rows(&mut rows, &self.metadata); # Ok(rows) # } # fn check_column(&self, column: &str) -> Result<()> { # let root = column.split('.').next().unwrap_or(column); # let columns = &self.metadata.columns; # if columns.contains_key(column) || columns.contains_key(root) { # Ok(()) # } else { # Err(condition_column_error(&self.metadata, column)) # } # } # } # #[async_trait] # impl TableShell for VecShell { # fn metadata(&self) -> &VistaMetadata { # &self.metadata # } # fn capabilities(&self) -> &VistaCapabilities { # &self.capabilities # } # fn driver_name(&self) -> &'static str { # "vec" # } # fn preview(&self) -> Preview { # Preview::new().with_note("every clause is evaluated in memory") # } # async fn list(&self) -> Result> { # Ok(self.set()?.into_iter().collect()) # } # async fn get(&self, id: &str) -> Result> { # Ok(self.set()?.into_iter().find(|(key, _)| key == id).map(|(_, row)| row)) # } # async fn count(&self) -> Result { # Ok(self.set()?.len() as i64) # } # async fn fetch_window(&self, offset: usize, limit: usize) -> Result { # Ok(Page::from_listing(self.set()?, offset, limit)) # } # fn add_condition(&mut self, condition: &Condition) -> Result<()> { # self.check_column(&condition.column)?; # self.query.conditions.push(condition.clone()); # Ok(()) # } # fn set_search(&mut self, text: Option<&str>) -> Result<()> { # self.query.search = text.map(str::to_string); # Ok(()) # } # fn set_order(&mut self, order: Option<(&str, SortDirection)>) -> Result<()> { # if let Some((column, _)) = order { # self.check_column(column)?; # } # self.query.order = order.map(|(column, dir)| (column.to_string(), dir)); # Ok(()) # } # } # fn row(id: &str, name: &str, category: &str) -> (String, Rec) { # let cells = [("id", id), ("name", name), ("category", category)]; # let record = cells.into_iter().map(|(k, v)| (k.to_string(), CborValue::from(v))).collect(); # (id.to_string(), record) # } use vantage_vista::{NoExtras, VistaFactory, VistaSpec}; struct VecVistaFactory { rows: Arc>, } impl VistaFactory for VecVistaFactory { type TableExtras = NoExtras; type ColumnExtras = NoExtras; type ReferenceExtras = NoExtras; fn build_from_spec(&self, spec: VistaSpec) -> Result { let mut metadata = VistaMetadata::new().with_id_column(spec.resolve_id_column("id")); for (name, column) in &spec.columns { let declared = match &column.col_type { Some(ty) => Column::new(name, ty), None => Column::untyped(name), }; let declared = column.flags.iter().cloned().fold(declared, Column::with_flag); metadata = metadata.with_column(declared); } Ok(Vista::new(spec.name, VecShell::new(self.rows.clone(), metadata))) } } const PRODUCT: &str = r#" name: product columns: id: { type: string, flags: [id, orderable] } name: { type: string, flags: [title, searchable] } category: { type: string } "#; # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let rows = vec![row("p1", "Tea", "drink"), row("p2", "Cake", "food"), row("p3", "Coffee", "drink")]; let factory = VecVistaFactory { rows: Arc::new(rows) }; let products = factory.from_yaml(PRODUCT)?; assert_eq!(products.id_column(), Some("id")); assert_eq!(products.title_columns(), ["name"]); # Ok(()) # } ``` Three spec rules hold for every driver: - **Resolve the id column in a fixed order**: `id_column:` first, then the first column flagged `id`, then the backend's own default (`"id"` for SQL, `"_id"` for MongoDB). `VistaSpec::resolve_id_column` does exactly that. Don't reverse it: an explicit `id_column:` is how a YAML author corrects a bad flag without editing the schema source. - **Driver blocks reject typos.** Driver-specific YAML lives under the three associated types, conventionally under a key named after the driver (`csv: { path }`, `mongo: { collection }`). Put `#[serde(deny_unknown_fields)]` on every extras struct, or `mongo: { collctiom: clients }` falls back to defaults and you debug a missing collection at runtime instead of at parse time. Make the fields `Option` with `#[serde(default)]` where the spec name is a sensible fallback. - **Validate up front.** Reject bad paths and conflicting options in `build_from_spec`, with the column name in the error, so the YAML author can find the entry. #### The boundary - **Ids stringify.** MongoDB's `ObjectId` becomes its hex form, SurrealDB's `Thing` its `table:id` form. Inside the shell you parse the string back. Round-trip it in a test: `get` must find a row by the very string `list` returned for it. - **Values become CBOR.** On the way in, unwrap `CborValue::Tag(_, inner)` and store the inner value. Stringify integers wider than your native type rather than truncating them. Keep the bridge module-private: a public BSON ↔ CBOR conversion gets called from places that should go through a Vista instead. ### Why must every advertised capability work? Because a capability is the only thing a generic caller checks. A UI draws an "add" button when the shell advertises `Insert`, a grid offers sorting when it advertises `Order`, and a cache pushes a filter down instead of evaluating it itself when it advertises `FilterOps`. Nothing else warns the caller, so a flag the shell can't honour turns a clean "not supported" into a failure in front of a user. `vantage_vista::contract` makes that a test failure instead. `check_advertised` calls the method behind each advertised capability on your fixture, and fails when one refuses, errors on a valid fixture, or does nothing: a search that counts as many rows as before, an ascending and a descending order that list the same row first, a `Gt` that keeps its own pivot row. It also holds every shell to the leading window and to the dotted-path rule above. `check_clone` checks that a narrowed clone leaves the original alone. Each driver runs both against its own fixture: ```rust # use std::sync::Arc; # use async_trait::async_trait; # use indexmap::IndexMap; # use vantage_core::Result; # use vantage_types::Record; # use vantage_vista::{ # Capability, CborValue, Column, ColumnFlag, Condition, Page, Preview, SortDirection, # TableShell, Vista, VistaCapabilities, VistaMetadata, VistaQuery, condition_column_error, # }; # type Rec = Record; # #[derive(Clone)] # struct VecShell { # rows: Arc>, # query: VistaQuery, # metadata: VistaMetadata, # capabilities: VistaCapabilities, # } # impl VecShell { # fn new(rows: Arc>, metadata: VistaMetadata) -> Self { # let capabilities = [ # Capability::FilterOps, # Capability::Search, # Capability::Order, # Capability::Window, # ] # .into_iter() # .collect(); # Self { rows, query: VistaQuery::new(), metadata, capabilities } # } # fn set(&self) -> Result> { # let mut rows = Vec::new(); # for (id, row) in self.rows.iter() { # if self.query.matches(row, &self.metadata)? { # rows.push((id.clone(), row.clone())); # } # } # self.query.sort_rows(&mut rows, &self.metadata); # Ok(rows) # } # fn check_column(&self, column: &str) -> Result<()> { # let root = column.split('.').next().unwrap_or(column); # let columns = &self.metadata.columns; # if columns.contains_key(column) || columns.contains_key(root) { # Ok(()) # } else { # Err(condition_column_error(&self.metadata, column)) # } # } # } # #[async_trait] # impl TableShell for VecShell { # fn metadata(&self) -> &VistaMetadata { # &self.metadata # } # fn capabilities(&self) -> &VistaCapabilities { # &self.capabilities # } # fn driver_name(&self) -> &'static str { # "vec" # } # fn preview(&self) -> Preview { # Preview::new().with_note("every clause is evaluated in memory") # } # async fn list(&self) -> Result> { # Ok(self.set()?.into_iter().collect()) # } # async fn get(&self, id: &str) -> Result> { # Ok(self.set()?.into_iter().find(|(key, _)| key == id).map(|(_, row)| row)) # } # async fn count(&self) -> Result { # Ok(self.set()?.len() as i64) # } # async fn fetch_window(&self, offset: usize, limit: usize) -> Result { # Ok(Page::from_listing(self.set()?, offset, limit)) # } # fn add_condition(&mut self, condition: &Condition) -> Result<()> { # self.check_column(&condition.column)?; # self.query.conditions.push(condition.clone()); # Ok(()) # } # fn set_search(&mut self, text: Option<&str>) -> Result<()> { # self.query.search = text.map(str::to_string); # Ok(()) # } # fn set_order(&mut self, order: Option<(&str, SortDirection)>) -> Result<()> { # if let Some((column, _)) = order { # self.check_column(column)?; # } # self.query.order = order.map(|(column, dir)| (column.to_string(), dir)); # Ok(()) # } # } # fn row(id: &str, name: &str, category: &str) -> (String, Rec) { # let cells = [("id", id), ("name", name), ("category", category)]; # let record = cells.into_iter().map(|(k, v)| (k.to_string(), CborValue::from(v))).collect(); # (id.to_string(), record) # } # fn products() -> Vista { # let metadata = VistaMetadata::new() # .with_column(Column::new("id", "string").with_flag(ColumnFlag::Orderable)) # .with_column(Column::new("name", "string").with_flag(ColumnFlag::Searchable)) # .with_column(Column::new("category", "string")) # .with_id_column("id"); # let rows = vec![row("p1", "Tea", "drink"), row("p2", "Cake", "food"), row("p3", "Coffee", "drink")]; # Vista::new("product", VecShell::new(Arc::new(rows), metadata)) # } use vantage_vista::contract; # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { contract::check_advertised(&products()).await; contract::check_clone(&products(), "category", CborValue::from("drink")).await; # Ok(()) # } ``` A third check, `check_advertised_writes`, probes the write capabilities. Run it only on a disposable fixture, since a successful probe stores a row. Leaving a flag off costs nothing: `Vista` refuses the call, and the caller evaluates the clause itself or hides the control. So start narrow and add a flag the day its method works. A shell that ships `Subscribe` the week LIVE queries land is healthier than one that advertised it on day one. Postgres is the cautionary case for `Subscribe`: `LISTEN` succeeds on a channel no trigger feeds, so its capability is opt-in (`PostgresVistaFactory::with_notify`), not inferred from the table being writable. A silent stream can't be told apart from an idle table. ### Tests Gate the Vista tests on `feature = "vista"`. The contract runs against the real backend, as the `TableSource` tests in earlier steps do. Around it: - **Unit tests** that need no backend: the CBOR bridge round-trips every native variant (lossy ones assert their documented form), and the YAML spec parses, with an unknown field in the driver block failing. - **Typed and YAML construction**: `from_table` and `from_yaml` list, get and count the seeded rows, and the spec name overrides the table name. - **Push-down**: a condition narrows both `count` and `list_values`, and a second one stacks. - **Traversal**: `get_ref` from a row and with `None`, for every relation. A test file that never traverses won't notice a shell that forgot to forward it. - **Ids**: `get_value` finds each row by the id `list_values` returned for it. ### Step 8 checklist At this point your backend has: 1. **A `vista` feature** gating the bridge, and `vantage-vista`'s `contract` feature in `[dev-dependencies]`. 2. **`::vista_factory()`**, returning a `VistaFactory` with an inherent `from_table` and an `impl VistaFactory` whose `build_from_spec` routes through the same path. 3. **`TableShell`** implementing `TableShell`: - `Clone` carries the narrowing and shares the connection; - `metadata` from `table.vista_metadata(..)`; - `list`, `get`, `count`, the leading window, and `add_condition` with equality, refusing columns through `condition_column_error`; - `get_ref` through `Table::traverse_erased`, from a row and with `None`; - `preview` and `driver_name`; - the opt-in methods the backend can honour, and `capabilities()` advertising exactly those. 4. **YAML extras** in `spec.rs`, each with `deny_unknown_fields`, and the id column resolved in the fixed order. 5. **Tests**: `check_advertised` and `check_clone` against a real fixture, `check_advertised_writes` where the fixture is disposable, and the construction, push-down, traversal and id tests. The same generic CLI, admin UI or script that drives CSV and MongoDB now drives your backend, with no entity import and no backend-specific line on the consumer side. --- Source: https://vantage-ui.com/book/new-persistence/step9-contained-relations.html # Step 9: Contained Relations Some data doesn't live in its own table. A product carries an `inventory` object; an order carries a `lines` array. The records are real: they have fields, and you want to list them, add to them and edit one. But they're embedded in a column of the parent row, not stored in a table of their own. `with_one` / `with_many` (Step 6) can't model this: they resolve to *another* table through a foreign key. A contained relation resolves to a **sub-Vista backed by one column of the same row**. Reads project that column into records; writes patch the column back in place. To the consumer it looks like any other relation: `get_ref("lines", Some(&row))`, then `list_values` / `insert_value` / `patch_value` / `delete`. This step is optional. Skip it unless your backend stores embedded objects or arrays that users should edit as records. ### What the framework gives you Almost all of it is backend-agnostic and already done. **Declaration** is on the typed `Table`, next to `with_one`. `with_contained_one` and `with_contained_many` each register a [`ContainedRelation`](vantage_table::references::ContainedRelation) (built by `ContainedRelation::one` or `ContainedRelation::many`), and `vista_contained()` describes them to Vista as references of a contained kind, [`Reference::contains_one`](vantage_vista::Reference::contains_one) or [`Reference::contains_many`](vantage_vista::Reference::contains_many): ```rust # use vantage_sql::sqlite::SqliteDB; # use vantage_table::table::Table; # use vantage_types::EmptyEntity; # use vantage_vista::Reference; # fn product_table(db: SqliteDB) -> Table { # Table::new("product", db) # .with_id_column("id") # .with_column_of::("name") # } # #[tokio::main(flavor = "current_thread")] # async fn main() -> Result<(), Box> { # let db = SqliteDB::connect("sqlite::memory:").await?; let orders = Table::::new("order", db) .with_id_column("id") .with_column_of::("lines") // the host column: declare it (see Sharp edges) .with_contained_many("lines", "lines", |db| { Table::new("lines", db) .with_column_of::("line_id") .with_column_of::("quantity") .with_column_of::("product") .with_one("product", "product", product_table) // a line can traverse out }, Some("line_id")); assert_eq!(orders.vista_contained(), [Reference::contains_many("lines", "lines")]); # Ok(()) # } ``` The closure builds the contained record's schema. It has the same shape as `with_one`'s target builder, uses the same type system, and runs lazily, at traversal. **The sub-Vista** is [`build_contained_vista`](vantage_vista::build_contained_vista). It holds the column's records in memory, serves reads from them, and after every write re-serializes the whole collection and hands it to a [`ContainedWriteback`](vantage_vista::ContainedWriteback). The writeback (patch the parent row's host column) is the only persistence-specific part: ```rust use std::sync::{Arc, Mutex}; use vantage_vista::prelude::*; use vantage_vista::{Column, ContainedWriteback, Reference, VistaMetadata, build_contained_vista}; fn line(id: &str, quantity: i64) -> CborValue { CborValue::Map(vec![ (CborValue::from("line_id"), CborValue::from(id)), (CborValue::from("quantity"), CborValue::from(quantity)), ]) } # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { let schema = VistaMetadata::new() .with_column(Column::new("line_id", "string")) .with_column(Column::new("quantity", "int")) .with_id_column("line_id"); let host = CborValue::Array(vec![line("l1", 2), line("l2", 1)]); // Stands in for "patch the parent row's host column". let written = Arc::new(Mutex::new(None)); let sink = written.clone(); let writeback: ContainedWriteback = Arc::new(move |collection: CborValue| { *sink.lock().unwrap() = Some(collection); Box::pin(async { Ok(()) }) }); let lines = Reference::contains_many("lines", "lines"); let records = build_contained_vista(&lines, schema, Some(&host), Some(writeback), None)?; assert_eq!(records.list_values().await?.len(), 2); records.delete("l1").await?; let collection = written.lock().unwrap().clone(); assert_eq!(collection, Some(CborValue::Array(vec![line("l2", 1)]))); # Ok(()) # } ``` **The skeleton** every driver shares is [`Table::get_contained_ref`](vantage_table::table::Table::get_contained_ref). It builds the contained schema from the closure, seeds the records from the parent row, wires the writeback to `patch_value` on the parent table, and resolves traverse-out (`line → product`) through the contained table's own relations. You pass it only what the driver knows: the parent row with its id in your native id type, a `wrap` closure that turns a target `Table` into a `Vista` through your factory, and the host column's codec. ### Wiring it into your shell Two changes, both small: - **Metadata.** Ask the table for its contained relations as well: `table.vista_metadata(VistaMetadataOptions { contained: true, .. })`. They join the joined relations in `metadata().references`, so consumers see one list. - **`get_ref`.** A contained relation routes through the same method as a join. When the reference's `kind.is_contained()`, call `get_contained_ref` instead of `traverse_erased`. With a row, pass the row and its id; with `None`, pass `None`, and you get the bare target: the contained schema with no records and no write capabilities, since there's no parent row to write into. The SQL drivers' shell is the worked source (`vantage-sql/src/sql_vista/refs.rs`). Their host column is `TEXT` holding JSON, so the codec parses on the way in and serializes on the way out: ```rust # use vantage_sql::sqlite::SqliteDB; # use vantage_table::table::Table; # use vantage_types::{EmptyEntity, PlainDialect, Record, cbor_to_json, json_to_cbor}; # use vantage_vista::prelude::*; # #[tokio::main(flavor = "current_thread")] # async fn main() -> Result<(), Box> { # let db = SqliteDB::connect("sqlite::memory:").await?; # let orders = Table::::new("order", db.clone()) # .with_id_column("id") # .with_column_of::("lines") # .with_contained_many("lines", "lines", |db| { # Table::new("lines", db) # .with_column_of::("line_id") # .with_column_of::("quantity") # }, Some("line_id")); // The parent row as the shell read it: `lines` is TEXT holding JSON. let order: Record = [ ("id".to_string(), CborValue::from("o1")), ("lines".to_string(), CborValue::from(r#"[{"line_id": "l1", "quantity": 2}]"#)), ] .into_iter() .collect(); let factory_db = db.clone(); let lines = orders.get_contained_ref( "lines", Some((&order, "o1".to_string())), move |target| factory_db.vista_factory().from_table(target), |host| match host { CborValue::Text(json) => serde_json::from_str(json).ok().map(json_to_cbor), _ => None, }, |collection| CborValue::Text(cbor_to_json(&PlainDialect, collection).to_string()), )?; assert_eq!(lines.list_values().await?.len(), 1); # Ok(()) # } ``` Nothing is written here: the records come from the row, and only a write through `lines` would patch `order.lines` on the database. The contract test from Step 8 covers the wiring. `check_advertised` traverses every reference, contained ones included, from a row and with `None`, so a shell that advertises `Traverse` and forgets the contained branch fails it. ### Native vs JSON-blob The codec is the one place native and JSON-blob backends differ. **Native backends** (SurrealDB, MongoDB) store the host column as a real nested object or array. The value arrives as `CborValue::Map` / `Array` already, and the writeback patches it back as it is (SurrealDB `UPDATE … MERGE`, MongoDB `$set`). Decode is a clone; encode is the identity. **JSON-blob backends** (SQL with no native nesting, the SQLite path above) store the collection as a JSON string in a `TEXT` column: parse on read, serialize on write. The SQL drivers' decoder also passes a `Map` / `Array` straight through, so the same code copes with a backend that parses JSON columns natively (Postgres `jsonb`, MySQL `json`). There, declaring the host column as `TEXT` keeps the round-trip a plain string and avoids binding a nested value on write, which is why Postgres and MySQL share the SQLite implementation. ### Why not just flatten the keys? You can, for *reading* a fixed shape: `inventory.stock` as a scalar column is fine when there's one known field. It falls apart the moment the embedded data is a *collection* (an order has N lines, not a fixed set), or the user needs to add and remove elements. A contained relation gives you a record set with ids, not a bag of dotted columns. ### Why not model it as a foreign-key relation? Because there's no other table to point at, and no foreign key to join on. The data is in the row. Forcing it into `with_many` would mean inventing a synthetic table and a join the storage engine can't honour. Traversal here is a column projection, not a query. ### Eager writeback Every mutation on the sub-Vista patches the parent row immediately; there's no `flush`. The sub-Vista and the parent row stay coherent at all times, and that's deliberate: batching belongs to a higher layer (a UI's write queue), not the storage boundary. The cost is one parent patch per edit, which is trivial for the small collections this targets. The flip side: the whole collection is re-serialized and written each time. A contained relation is for line items and embedded objects, not for a thousand-element array mutated in a tight loop. ### Sharp edges **Declare the host column.** This is the bug every backend hits. If `lines` isn't a declared column, your read path won't select or project it (SQL builds its `SELECT` from declared columns; MongoDB projects from its column paths), so the parent row arrives without it and traversal sees an empty collection. Declare it alongside the `with_contained_*` call. **Positional ids shift.** With no id column, contained-many records are keyed by position (`"0"`, `"1"`, …), and deleting element 0 renumbers the rest. Give the relation an id column (`with_contained_many(…, Some("line_id"))`) when callers hold on to ids across mutations. **Contained-one uses a fixed id.** A single embedded object is addressed as `"0"`. **The writeback is not atomic with the contained record's own children.** A contained record can traverse out (`line → product`), but each backend write is its own statement. That's the same best-effort contract as nested insert: fine for these shapes, not a transaction. ### From YAML Contained relations are declarable in a YAML spec too, in a `contained:` section next to `columns:` and `references:`: ```yaml name: order columns: id: { type: string, flags: [id] } lines: { type: string } # the host column: declare it so it's selected sqlite: table: order contained: lines: host_column: lines kind: contains_many # or contains_one, the default id_column: line_id # optional; omit for positional ids columns: line_id: { type: string } quantity: { type: int } ``` The factory lowers it through one generic helper, [`Table::with_contained_specs`](vantage_table::table::Table::with_contained_specs), which calls your driver's existing column builder on each contained column. Wiring it is one line in the factory's spec-to-table code: `table = table.with_contained_specs(&spec.contained, build_column)?;`. The YAML and code-first paths then register the same `ContainedRelation`. **Limitation:** YAML-declared contained records carry *columns only*, so traverse-out (`line.product`) isn't expressible from YAML yet; the code-first closure supports it, since it can add `with_one` to the contained table. Lifting this means letting contained columns carry `references:` sugar plus a resolver, the same machinery YAML foreign-key references need. ### Step 9 checklist A backend supports contained relations once it has: 1. **Metadata** built with `VistaMetadataOptions { contained: true, .. }`, so the relations appear in `metadata().references`. 2. **`get_ref`** routing a contained reference to `Table::get_contained_ref`, passing the parent row and its native id (or `None` for the bare target), the factory's `wrap`, and the host codec: passthrough for native nesting, JSON parse and serialize for a `TEXT` column. 3. **YAML**: one `with_contained_specs` line in the factory's spec-to-table code. 4. **Tests** against a real backend, gated on `feature = "vista"`: declare a host column holding a collection (code-first *and* through `from_yaml`), traverse it, insert and patch through the sub-Vista, and re-read the parent row to prove the writeback landed. The Step 8 contract test covers the traversal with and without a row. Native and JSON-blob backends differ only in two closures: how the host value comes in and how the writeback goes out. Everything between is shared. --- Source: https://vantage-ui.com/book/augmentation.html # Augmentation A `Vista` reads one table from one backend. But a row is often only half the story: an S3 bucket listing tells you a file's key and size, not how many resources the terraform state inside it declares. A REST list of repositories doesn't carry the CI status that lives behind a shell command. The data you want to show lives in a *second* source, keyed off the first. **Augmentation wires two Vistas into one `Dio`**: a *master* that is listed, and a *detail* source that is loaded one row at a time and merged on top. The two can be the same Vista (a backend whose list and detail are separate operations) or entirely different backends — a REST master enriched by a `cmd` detail, or the reverse. The detail source is resolved by name through a [`VistaResolver`](vantage_vista::VistaResolver), usually a [`Catalog`](vantage_vista::Catalog), so it is persistence-agnostic. ## Why this lives in the Dio, not the Vista This is a join, and joins are governed by the capability contract. A SQL Vista can join its own tables; a REST Vista cannot; two *different* backends can never push a join down to either engine. Where the backend can't, the layer above fills the gap — that is the whole job of the Dio. So augmentation is deliberately a `Dio`-layer feature. A `Vista` honestly describes one backend and has no cache, no viewport, and no catalog; declaring "enrich from another Vista, loaded lazily and cached" on it would be a promise it can't keep. The `Dio` has all three, so it owns the augmentation: it lists the master cheaply, and for each row the user actually scrolls to, it fetches the detail and stitches the columns together. It is a per-row join, executed client-side, exactly where a push-down is impossible. ```admonish note title="It reduces to the old two-pass" The previous progressive-loading path — a `cmd` table with a `detail` script — is just the special case where the detail source *is* the master and the key is the id. Expressed as augmentation it is `table: , source: id, fetch: per_row`. Nothing about that behaviour changed; it stopped being a backend quirk. ``` ## How it runs Augmentation engages the two-pass machinery: 1. **List pass** — the master is listed cheaply (a page at a time). Each row is written to the cache as `Incomplete`, carrying only the columns the list returns. When the master can serve windows (`Capability::Window`) the page is pushed down; otherwise the set is listed and windowed locally. 2. **Detail pass** — viewport-driven. For each visible `Incomplete` row, every augmentation resolves its detail Vista, fetches the matching record, and merges the chosen columns onto the cached row, which flips to `Fresh`. Rows off-screen are never fetched; rows already hydrated are never re-fetched. A failed detail fetch marks only that row, leaving its cheap columns intact — the rest of the page hydrates normally. ## Declaring it Augmentation is a property of the `Dio`, not the `Lens`: it is fixed when the Dio is built. Hand the [`DioBuilder`](vantage_diorama::DioBuilder) one or more augmentations between `lens.dio(master)` and `build()`: ```rust # use std::sync::Arc; # # use vantage_diorama::{Augmentation, Cache, Detail, Fetch, Lens, MergeRule, Source}; # use vantage_vista::mocks::MockShell; # use vantage_vista::{Column, Vista, VistaMetadata}; # # fn meta(columns: &[&str]) -> VistaMetadata { # columns # .iter() # .fold(VistaMetadata::new(), |m, c| m.with_column(Column::new(*c, "String"))) # .with_id_column("id") # } # # fn detail() -> Vista { # Vista::new("tfstate_detail", MockShell::new().with_metadata(meta(&["id", "resources", "serial"]))) # } # # #[tokio::main(flavor = "current_thread")] # async fn main() -> vantage_core::Result<()> { # let master = Vista::new("buckets", MockShell::new().with_metadata(meta(&["id", "key", "size"]))); # let lens = Lens::builder().cache(Cache::memory()).build()?; # let augmentation = Augmentation::new( # Detail::Fixed(Arc::new(detail())), # Source::Column { from: "key".into(), to: None }, # Fetch::PerRow, # MergeRule::lift(["resources", "serial"]), # ); let dio = lens .dio(master) .augment(vec![augmentation]) // ≥1 engages two-pass .build() .await?; # let _ = dio; # Ok(()) # } ``` Each `Augmentation` has four parts. Here the detail is resolved by name through a [`Catalog`](vantage_vista::Catalog): ```rust # use std::sync::Arc; # # use vantage_diorama::{Augmentation, Detail, Fetch, MergeRule, Source}; # use vantage_vista::mocks::MockShell; # use vantage_vista::{Catalog, Vista, VistaResolver}; # # fn detail() -> Vista { # Vista::new("tfstate_detail", MockShell::new()) # } # let resolver: Arc = Arc::new(Catalog::new().with("tfstate_detail", |_: &str| Ok(detail()))); let augmentation = Augmentation::new( Detail::Resolved { resolver, name: "tfstate_detail".into() }, // resolved by name Source::Column { from: "key".into(), to: None }, Fetch::PerRow, MergeRule::lift(["resources", "serial"]), ); # let _ = augmentation; ``` `Detail::Fixed(Arc)` takes a detail Vista directly instead, for a side table no resolver knows. `source` and `fetch` are two orthogonal axes: | `Source` | how a master row selects its detail | |---|---| | `Id` | `master.id → detail.id` | | `Column { from, to }` | `master[from] → detail[to` *or* `detail.id]` | | `Build(closure)` | arbitrary narrowing from the whole row — per-row only | | `Fetch` | how the detail is read | |---|---| | `PerRow` | one record per master row (`get_value`, or narrow-and-take-first) | | `Custom(closure)` | a caller-supplied async fetch | `Id` and id-keyed `Column` sources read by key through `get_value` — the uniform "one record by key" primitive (a `cmd` detail script, a SQL `WHERE id =`, a REST `GET /{id}`). `Build` returns an arbitrary narrowed Vista, so it is per-row only. ### Declaring one An `Augmentation` is built in Rust. There is no serde mirror of these types: a consumer already has its own config vocabulary — vantage-ui reads an `augment:` block naming a source table, a key and a column list — and maps that onto `Augmentation` directly. A second, parallel YAML shape in diorama would be one more spelling of the same idea for each consumer to ignore. ### Rhai as a closure factory The runtime types carry **closures**, not script strings — `Source::Build` is a `Fn(&row, base) -> Vista`. Rhai is one factory that produces such a closure; hand-written Rust is another. A consumer that wants a scripted narrowing builds the closure itself and hands it to `Source::Build`. The diorama core never sees a script string. The vocabulary such a script would use is the table handle's, covered in [Scripting with Rhai](./rhai/surfaces.md). ## Merging `merge.columns` lists the detail columns to lift; an empty list lifts all of them. The detail record is the authoritative hydration of the row, so on a name clash its value wins — it overwrites the cheap list-pass value and adds its new columns. (An empty list plus overwrite is exactly the old cmd two-pass: the detail script returns the full record, which replaces the stub.) The augmented `Dio` therefore advertises the union of the master's columns and the lifted detail columns — the "Dio advertises a superset" principle: below it, sources are partial; above it, the view is whole. ## Anticipated objections ### Why not model the detail as a foreign-key relation? A relation resolves *within* one persistence and can be pushed down as a join. The detail here may be a different backend entirely; there is nothing to join on and no engine that could honour it. Augmentation is the cross-Vista form, stitched by the Dio. ### Why one row at a time? Because the expensive work should follow the user's attention. The list pass is cheap and immediate; only rows that reach the viewport pay for their detail, and they pay once. ## What's next Three things are planned but not yet implemented; each fails honestly today rather than degrading silently: - **`Batched` fetch** — collect the window's distinct keys into one set query and scatter the results back, for `Id`/`Column` sources. - **Detail-key cache** — dedupe fetches across master rows that share a key (non-unique `Column` sources). - **Scripted fetch** — a script choosing how to pull. ## Checklist To augment a master with a second source: 1. **Register the detail model** in a `Catalog` by name (`Catalog::new().with(..)`), or hold it directly for `Detail::Fixed`. 2. **Build the `Augmentation`** — `Detail::Resolved { resolver, name }`, a `Source` (id, column, or a closure), a `Fetch` (`PerRow`, or `Custom`), and the `MergeRule` columns. 3. **Wire the Dio** — `lens.dio(master).augment(vec![..]).build().await?`; the two-pass machinery engages for every table scenery opened on it. 4. **Show the columns** — the lifted detail columns appear on hydrated rows alongside the master's. A runnable end-to-end example (two in-memory Vistas, list pass then detail pass) lives in `vantage-diorama/examples/augmentation.rs`: ```console cargo run -p vantage-diorama --example augmentation ``` --- Source: https://vantage-ui.com/book/type-system.html # Persistence-aligned Type System Every Vantage persistence defines its own type trait — `SqliteType`, `PostgresType`, `SurrealType`, `MongoType`, `CsvType` — that maps Rust values to and from the storage format. This is how expressions like `sqlite_expr!("price > {}", 150i64)` know how to bind `150i64` as a parameter. Built-in implementations cover the common ground: `bool`, `i64`, `f64`, `String`, `Option`, `chrono` date/time types, and more. Each persistence supports exactly the types its backend can handle natively. Because the trait is open, you can implement it for your own types — enums, newtypes, domain objects — and they'll work everywhere expressions are used: conditions, inserts, updates. See [Adding Custom Types](./type-system/custom-types.md) for a walkthrough. --- Source: https://vantage-ui.com/book/type-system/custom-types.html # Adding Custom Types You can teach Vantage to store any Rust type by implementing the persistence type trait. Here's the pattern, using an `Animal` enum that maps to a text column across every backend: 1. **Define your type** — a plain Rust enum (or struct). 2. **Implement the persistence trait** — `SqliteType`, `PostgresType`, etc. Each impl says how to convert to/from the storage format (CBOR for SQL backends, BSON for MongoDB). 3. **Use it in expressions** — once the trait is implemented, `sqlite_expr!("species = {}", animal)` just works. A single type can implement traits for multiple backends, so the same `Animal` enum works with SQLite, Postgres, SurrealDB, MongoDB, and CSV — each with its own serialization logic. --- Source: https://vantage-ui.com/book/sql.html # SQL: PostgreSQL, MySQL & SQLite The `vantage-sql` crate provides persistence implementations for three SQL databases via [sqlx](https://github.com/launchbadge/sqlx). All three share the same architecture — CBOR-based type systems, expression engine, query builder, and full `TableSource` implementation — but each has vendor-specific behaviour around type affinity, quoting, and parameter binding. ## Backends | Backend | Struct | Type System | Param style | ID quoting | | ---------- | ------------ | ----------------- | ----------- | ---------------- | | PostgreSQL | `PostgresDB` | `AnyPostgresType` | `$1, $2` | `"double_quote"` | | MySQL | `MysqlDB` | `AnyMysqlType` | `?, ?` | `` `backtick` `` | | SQLite | `SqliteDB` | `AnySqliteType` | `?1, ?2` | `"double_quote"` | `MysqlDB` supports MySQL 8 and MariaDB 11; both run the same test suite in CI. MariaDB differences: `JSON_EXTRACT` / `.as_json()` returns quoted JSON text; there is no `GROUPING()`, no `ORDER BY` with `WITH ROLLUP`, and no `REGEXP_LIKE` (use `REGEXP`); `DateTime` binds are rejected. ## What they implement All three implement the full trait stack: - `DataSource` — marker - `ExprDataSource` — parametric SQL execution with CBOR values - `SelectableDataSource` — query builder with JOINs, CTEs, window functions - `TableSource` — full CRUD, columns, conditions, aggregates - `TableQuerySource` — table definition → full query ## The Vista With the `vista` feature, each backend's factory builds a [Vista](./intro/step4-vista.md) from a YAML spec or a typed table. All three advertise the same reads, because SQL can do each of them in the query: `Order`, `Search`, `FilterOps`, `Window` (a `LIMIT`/`OFFSET` window, served by `fetch_window`) and `Traverse`. A table-sourced Vista adds the writes, `Insert`, `Update`, `Delete` and `ConfineWrites`. A query-sourced one (`rhai:` or `base:`) takes no writes, because no single table could receive them. Per backend: - **PostgreSQL** adds `Subscribe` when the factory is built `with_notify(true)`. Postgres can't tell whether a `{table}_changed` trigger exists, so the application has to say so. - **MySQL** adds `NativeConditions` (the `with_condition` verb in `modify:` scripts) and, on a table-sourced Vista, `Import`. A quicksearch is a `LIKE` on each searchable column, joined with `OR`. The searchable columns are the ones flagged `searchable`, or every column when none is flagged. Every driver reads that rule from `Table::search_columns` (`VistaMetadata::search_columns` states it over a Vista's columns), so no backend picks its own. Imported reference columns are left out, because they exist only as projection aliases, and a table with nothing to search matches no rows. PostgreSQL uses `ILIKE` on the column cast to text. SQLite's `LIKE` already ignores case for ASCII text, and MySQL's follows the column's collation. ## CBOR value representation All SQL backends use **CBOR** (`ciborium::Value`) as their internal value type — not JSON. This preserves type fidelity that JSON loses: - **Integer vs Float** — JSON's `42` is ambiguous; CBOR distinguishes `Integer(42)` from `Float(42.0)` - **Binary data** — CBOR has native byte arrays; JSON would need base64 encoding - **Precise decimals** — stored as tagged CBOR values, not lossy floats Values are converted to CBOR on write (via the type system's `to_cbor()`) and read back as untyped CBOR (via `from_cbor()`). The type markers from `vantage_type_system!` ensure correct binding — integers bind as `i64`, text as `&str`, booleans as `bool`. ## Type conversion reference Each database handles Rust types differently depending on the column type. See the [Type Conversions](./sql/type-conversions.md) reference for detailed round-trip tables covering: - **Chrono types** — `NaiveDate`, `NaiveTime`, `NaiveDateTime`, `DateTime`, `DateTime` - **Numeric types** — `Decimal`, `i64`, `f64` - Exact vs lossy vs truncated behaviour per column type - Cross-type coercion rules and error cases --- Source: https://vantage-ui.com/book/sql/primitives.html # SQL Primitives Primitives are reusable building blocks for constructing SQL expressions. They handle quoting, escaping, vendor-specific syntax, and logical composition so you don't have to. Conditions built with typed columns and `SqliteOperation` cover simple comparisons, but real queries need more — `OR` groups, function calls, string concatenation, date formatting. That's what primitives are for. Import them with: ```rust use vantage_sql::primitives::*; use vantage_sql::sqlite::sqlite_ident as ident; ``` Primitives are **not** part of the prelude — import them explicitly when needed. The second line pins `ident` to SQLite; [`ident()`](#ident--identifiers) explains why the examples need it. ```admonish note title="Macros and structs" Some primitives have convenience macros that accept a variable number of arguments and call `.expr()` on each one automatically: `fx!` → `Fx`, `concat_!` → `Concat`. The macros are syntactic sugar — if you need to build arguments programmatically (e.g. from a `Vec`), use the underlying struct directly. ``` ## `or_()` / `and_()` — Logical Combinators By default, multiple calls to `.with_condition()` combine with `AND`. When you need `OR`, use [`or_()`](vantage_sql::primitives::logical::or_): ```rust use vantage_sql::primitives::*; # use vantage_sql::sqlite::operation::SqliteOperation; # use vantage_sql::sqlite::sqlite_ident as ident; // role = 'admin' OR role = 'superuser' let cond = or_(ident("role").eq("admin"), ident("role").eq("superuser")); ``` For nested logic, combine with [`and_()`](vantage_sql::primitives::logical::and_): ```rust # use vantage_sql::primitives::*; # use vantage_sql::sqlite::operation::SqliteOperation; # use vantage_sql::sqlite::sqlite_ident as ident; // (price > 100 AND in_stock = 1) OR (featured = 1) let cond = or_( and_(ident("price").gt(100i64), ident("in_stock").eq(true)), ident("featured").eq(true), ); ``` Both return a [`ConditionGroup`](vantage_sql::primitives::logical::ConditionGroup), which writes its own brackets and is `Expressive`, so it plugs directly into `.with_condition()`. A chain stays flat: `a.or_(b).or_(c)` renders `(a OR b OR c)`. ## `ident()` — Identifiers Creates a quoted column or table name. Quoting adapts per backend (`"` for SQLite/Postgres, `` ` `` for MySQL). ```rust # use vantage_sql::sqlite::sqlite_ident as ident; let col = ident("price"); // "price" let qualified = ident("name").dot_of("u"); // "u"."name" let aliased = ident("total").with_alias("t"); // "total" AS "t" ``` [`ident()`](vantage_sql::primitives::identifier::ident) is a shorthand for [`Identifier::new()`](vantage_sql::primitives::identifier::Identifier::new). Every name is quoted, and a quote character inside the name is doubled, so a name can't break out of its quotes. Each backend also has a **typed** identifier: [`sqlite_ident()`](vantage_sql::sqlite::sqlite_ident), `pg_ident()` (in `vantage_sql::postgres`) and `mysql_ident()` (in `vantage_sql::mysql`). Each returns a wrapper that only renders for its own backend. ### Why not the generic `ident()` everywhere? The generic `Identifier` renders for every backend feature you enabled. With one feature on, that is unambiguous. With two, `ident("price").gt(100i64)` could build a SQLite condition or a PostgreSQL one, and the compiler stops with "type annotations needed". This book builds SQLite and PostgreSQL, and most applications that talk to two databases do too. The generic one still works where the backend is fixed by context: inside `sqlite_expr!()`, or passed where a specific `Expressive` is expected. Use a typed identifier when you call an operation on it directly. The examples on this page import `sqlite_ident as ident`, so they read the same as single-backend code. ## `fx!` — Function Calls The [`fx!`](vantage_sql::fx) macro builds a SQL function call. Arguments are passed directly — `.expr()` is called on each one automatically: ```rust # use vantage_sql::sqlite::sqlite_ident as ident; use vantage_sql::{fx, sqlite_expr}; let count = fx!("count", sqlite_expr!("*")); // => COUNT(*) let average = fx!("avg", ident("price")); // => AVG("price") // Multiple arguments let nickname = fx!("coalesce", ident("nickname"), "anonymous"); // => COALESCE("nickname", 'anonymous') // Nested let rounded = fx!("round", fx!("avg", ident("price")), 2i64); // => ROUND(AVG("price"), 2) ``` If you need to build arguments programmatically (e.g. from a `Vec`), use [`Fx::new()`](vantage_sql::primitives::fx::Fx::new) directly: ```rust # use vantage_expressions::{Expression, Expressive}; # use vantage_sql::primitives::*; # use vantage_sql::sqlite::AnySqliteType; # use vantage_sql::sqlite::sqlite_ident as ident; # let columns = [ident("nickname"), ident("name")]; let args: Vec> = columns.iter().map(|c| c.expr()).collect(); let f = Fx::new("coalesce", args); ``` ## `ternary()` — Conditional Expression Three-valued conditional. Renders as `IIF()` on SQLite, `IF()` on MySQL, and `CASE WHEN ... THEN ... ELSE ... END` on PostgreSQL: ```rust # use vantage_sql::primitives::*; # use vantage_sql::sqlite::operation::SqliteOperation; # use vantage_sql::sqlite::sqlite_ident as ident; let expr = ternary( ident("stock").gt(0i64), "in stock", "sold out", ); ``` [`ternary()`](vantage_sql::primitives::ternary::ternary) is a shorthand for [`Ternary::new()`](vantage_sql::primitives::ternary::Ternary::new). ## `Case` — CASE Expressions For more than two branches, use [`Case`](vantage_sql::primitives::case::Case) to build a full `CASE WHEN ... END` block: ```rust # use vantage_sql::primitives::*; # use vantage_sql::sqlite::operation::SqliteOperation; # use vantage_sql::sqlite::sqlite_ident as ident; let expr = Case::new() .when(ident("status").eq("active"), "yes") .when(ident("status").eq("banned"), "no") .else_("unknown"); ``` ## `concat_!` — String Concatenation Concatenates expressions. Renders as `||` on SQLite/Postgres, `CONCAT()` on MySQL. The [`concat_!`](vantage_sql::concat_) macro calls `.expr()` on each argument automatically: ```rust # use vantage_sql::sqlite::sqlite_ident as ident; use vantage_sql::concat_; let full_name = concat_!(ident("first_name"), " ", ident("last_name")); ``` Use [`.ws()`](vantage_sql::primitives::concat::Concat::ws) for a separator — it accepts any `Expressive`, including string literals: ```rust # use vantage_sql::concat_; # use vantage_sql::sqlite::sqlite_ident as ident; let listed = concat_!(ident("first_name"), ident("last_name")).ws(", "); // SQLite: "first_name" || ', ' || "last_name" // MySQL: CONCAT_WS(', ', `first_name`, `last_name`) ``` ## `Interval` — Date Intervals Portable date interval that adapts per backend: ```rust # use vantage_sql::primitives::*; let i = Interval::new(30, "day"); // SQLite: 30 (used with date functions) // MySQL: INTERVAL 30 DAY // Postgres: INTERVAL '30 day' ``` [`Interval::new(amount, unit)`](vantage_sql::primitives::interval::Interval::new) takes the unit as text: `day`, `week`, `month`, `year` or `hour`, singular or plural. Use the singular form when the query may run on MySQL, which accepts only `DAY`, `MONTH` and so on. SQLite has no interval type, so it renders the interval as a number of days: a week is 7, a month 30, a year 365, and hours round to whole days. ## `date_format()` — Date Formatting Portable strftime-style formatting. Translates format tokens per backend: ```rust # use vantage_sql::primitives::*; # use vantage_sql::sqlite::sqlite_ident as ident; let formatted = date_format(ident("created_at"), "%Y-%m-%d"); // SQLite: STRFTIME('%Y-%m-%d', "created_at") // MySQL: DATE_FORMAT("created_at", '%Y-%m-%d') // Postgres: TO_CHAR("created_at", 'YYYY-MM-DD') ``` [`date_format()`](vantage_sql::primitives::date_format::date_format) is a shorthand for [`DateFormat::new()`](vantage_sql::primitives::date_format::DateFormat::new). To skip token translation and pass a native format string, build it with [`DateFormat::raw_format()`](vantage_sql::primitives::date_format::DateFormat::raw_format) instead. --- Source: https://vantage-ui.com/book/sql/type-conversions.html # Type Conversions How Rust native types round-trip through different SQL column types. Tested via entity insert + read-back (Table API). Legend: - **exact** = lossless round-trip, value identical after insert + read - **lossy** = value stored but precision reduced (e.g. f64 rounding) - **truncated** = value stored but digits beyond column precision are cut - **err** = database rejects the insert or entity conversion fails on read ## Chrono Types ### MySQL [tests/mysql/1_chrono.rs](https://github.com/romaninsh/vantage/blob/main/vantage-sql/tests/mysql/1_chrono.rs) | Column \ Rust type | NaiveDate | NaiveTime | NaiveDateTime | DateTime\ | DateTime\ | |---|---|---|---|---|---| | VARCHAR | exact | exact | exact | exact | exact | | DATE | exact | err | err | err | — | | TIME | err | truncating µs | err | err | — | | TIME(6) | err | exact | err | err | — | | DATETIME | err | err | truncating µs | truncating µs | offset → +00:00 | | DATETIME(6) | err | err | exact | exact | — | | TIMESTAMP | err | err | truncating µs | truncating µs | offset → +00:00 | | TIMESTAMP(6) | err | err | exact | exact | — | - Format: `"2025-01-10 12:00:00"` (space separator, no T); `FixedOffset` appends `+05:30` - TIME/DATETIME/TIMESTAMP default to 0 fractional digits — use `(6)` for microseconds - `DateTime`: VARCHAR preserves offset; DATETIME/TIMESTAMP normalize to UTC - MariaDB 11 rejects `DateTime` into DATETIME/TIMESTAMP (no offset literals) — use `DateTime` - Cross-type coercions (e.g. NaiveTime → DATE) fail with variant mismatch ### PostgreSQL [tests/postgres/1_chrono.rs](https://github.com/romaninsh/vantage/blob/main/vantage-sql/tests/postgres/1_chrono.rs) | Column \ Rust type | NaiveDate | NaiveTime | NaiveDateTime | DateTime\ | DateTime\ | |---|---|---|---|---|---| | VARCHAR | exact | exact | exact | exact | offset → +00:00 | | DATE | exact | err | err | err | — | | TIME | err | exact | err | err | — | | TIMESTAMP | err | err | exact | exact | offset → +00:00 | | TIMESTAMPTZ | err | err | exact | exact | offset → +00:00 | - Format: `"2025-01-10 12:00:00+00"` (space separator, abbreviated tz offset) - Microsecond precision by default — no `(6)` suffix needed - Typed binds (chrono types, not text) required for DATE/TIME/TIMESTAMP/TIMESTAMPTZ - `DateTime`: offset always normalized to UTC (typed binds), no column preserves it - Cross-type coercions fail with variant mismatch ### SQLite [tests/sqlite/1_chrono.rs](https://github.com/romaninsh/vantage/blob/main/vantage-sql/tests/sqlite/1_chrono.rs) | Column \ Rust type | NaiveDate | NaiveTime | NaiveDateTime | DateTime\ | DateTime\ | |---|---|---|---|---|---| | TEXT | exact | exact | exact | exact | exact | - All dates stored as TEXT — format: ISO 8601 with T separator (`"2025-01-10T12:00:00Z"`) - Subsecond precision and timezone offsets preserved as-is ## Numeric Types ### MySQL [tests/mysql/1_decimal.rs](https://github.com/romaninsh/vantage/blob/main/vantage-sql/tests/mysql/1_decimal.rs) | Column \ Rust type | Decimal | i64 | f64 | |---|---|---|---| | VARCHAR | exact | exact | exact | | DECIMAL(20,6) | truncated to 6 places | err | err | | DECIMAL(38,15) | exact | err | err | | DOUBLE | lossy (~15 digits) | err | exact | | FLOAT | lossy (~7 digits) | err | lossy (~7 digits) | | BIGINT | fractional part lost | exact | err | - No cross-conversion between Integer, Float, and Decimal CBOR types - VARCHAR works for all types — `from_cbor` parses text as fallback ### PostgreSQL [tests/postgres/1_decimal.rs](https://github.com/romaninsh/vantage/blob/main/vantage-sql/tests/postgres/1_decimal.rs) | Column \ Rust type | Decimal | i64 | f64 | |---|---|---|---| | VARCHAR | exact | exact | exact | | NUMERIC(20,6) | truncated to 6 places | err | err | | NUMERIC(38,15) | exact | err | err | | DOUBLE PRECISION | lossy (~15 digits) | err | exact | | REAL | lossy (~7 digits) | err | lossy (~7 digits) | | BIGINT | fractional part lost | exact | err | - Typed binds (`rust_decimal::Decimal`) used for NUMERIC columns ### SQLite [tests/sqlite/1_decimal.rs](https://github.com/romaninsh/vantage/blob/main/vantage-sql/tests/sqlite/1_decimal.rs) | Column \ Rust type | Decimal | i64 | f64 | |---|---|---|---| | TEXT | exact | exact | exact | | NUMERIC | lossy (~15 digits, stored as REAL) | exact | exact | | REAL | lossy (~15 digits) | err | exact | | INTEGER | fractional part lost | exact | err | - SQLite infers CBOR type from stored value, not declared column type - NUMERIC/REAL affinities coerce to float — `i64` in REAL comes back as Float and fails - Store Decimal in TEXT for lossless precision --- Source: https://vantage-ui.com/book/surrealdb.html # SurrealDB The `vantage-surrealdb` crate targets [SurrealDB](https://surrealdb.com) — a document-graph database whose query language (SurrealQL) is close enough to SQL to share a vocabulary, but diverges where it matters: graph traversals instead of joins, embedded arrays with closures, record links, and a `math::`/`array::`/`string::` function namespace. Behind the `rhai` feature the crate exposes a scripting surface: a [Rhai](https://rhai.rs) engine (registered by the `register_surreal_engine!` macro) that builds `SELECT` statements from the same named primitives as `vantage-sql` where the two overlap, and surreal-specific ones where they don't. ## The Vista With the `vista` feature, `SurrealVistaFactory` builds a [Vista](./intro/step4-vista.md) from a YAML spec. It advertises the reads SurrealQL can do in the query: `Order`, `Search`, `FilterOps`, `NativeConditions` (the `with_condition` verb in `modify:` scripts), `Window` (`LIMIT … START …`, served by `fetch_window`) and `Traverse`. A table-sourced Vista adds `Insert`, `Update`, `Delete`, `ConfineWrites` and `Subscribe`, the last through a `LIVE SELECT`. A query-sourced one (`rhai:` or `base:`) has no table to write to or to subscribe to, so it advertises the reads only. A quicksearch reads the columns flagged `searchable`, or every column when none is flagged: the same rule as every other driver (`Table::search_columns`). Each column is matched as a case-insensitive substring, `string::contains(string::lowercase((col)), $text)` with the text lowercased, and the matches are joined with `OR`. A computed (`expr:`) column is searched through its expression, because its alias doesn't exist where `WHERE` runs. Imported reference columns are left out. ## Pages - [Primitives](./surrealdb/primitives.md) — the named expression vocabulary and how each lowers to SurrealQL. --- Source: https://vantage-ui.com/book/surrealdb/primitives.html # SurrealDB Primitives Primitives are the named building blocks for SurrealQL expressions. Each one does a single job and lowers to the SurrealQL it's named for — you call `surreal::coalesce(a, b)` and get `a ?? b`, not a string you had to escape yourself. The guiding rule is **one meaningful name per concept, reusing the `vantage-sql` name wherever the concept already exists.** A script that says `count()`, `avg()`, `round()`, `coalesce()`, `case_when()`, `date_format()` reads the same against SQLite, Postgres, or SurrealDB; only the lowering differs (`avg` → `AVG` on SQL, `math::mean` on SurrealDB). Where SurrealDB has no SQL analogue — graph traversals, embedded-array closures, `SPLIT` — the primitive carries SurrealDB's own term. The surface is the Rhai engine built by `register_surreal_engine!`. The examples below are Rhai; the `// →` comment shows what each renders to. ```admonish note title="The surreal:: namespace" Every free function here, and the `me` anchor, lives in the `surreal::` namespace: `surreal::count()`, `surreal::ident("name")`, `surreal::me`. The SQL builders use `sql::` for the same names, so a script names its backend once per call. Methods (`.alias`, `.when`, `.map`, `.from`, …), the `[…]` indexer and the operators are called on their receiver without a prefix. The tables below leave the prefix out. ``` ```admonish note title="Escape hatches" `surreal::expr("…")` and `surreal::fx("name", […])` drop to raw SurrealQL when no primitive fits — `surreal::expr("tags")`, `surreal::fx("crypto::md5", [surreal::ident("email")])`. Prefer a named primitive; reach for these only at the edges. ``` ## The shared vocabulary Same names as `vantage-sql`, lowered to SurrealQL: | Primitive | SurrealQL | Notes | |---|---|---| | `count()` | `count()` | zero-arg row count | | `count(expr)` | `count(expr)` | count truthy / array values | | `count_distinct(expr)` | `count(array::distinct(expr))` | | | `sum(expr)` | `math::sum(expr)` | | | `avg(expr)` | `math::mean(expr)` | SQL name, surreal lowering | | `min(expr)` / `max(expr)` | `math::min` / `math::max` | | | `round(expr)` | `math::round(expr)` | nearest integer | | `round(expr, n)` | `math::fixed(expr, n)` | round to `n` decimals — `math::round` has no places arg | | `coalesce(a, b)` | `a ?? b` | null-coalescing | | `nullif(a, b)` | `IF a = b THEN NONE ELSE a END` | | | `cast(expr, "int")` | `type::int(expr)` | also `float`/`string`/`decimal`/`datetime`/`number`/`bool` | | `date_format(expr, fmt)` | `time::format(expr, "fmt")` | | ```rhai surreal::coalesce(surreal::ident("nickname"), "anonymous") // → nickname ?? "anonymous" surreal::round(surreal::avg(surreal::ident("price")), 2) // → math::fixed(math::mean(price), 2) surreal::cast(surreal::ident("qty"), "float") // → type::float(qty) ``` ### `case_when()` — multi-branch conditional Builds a SurrealQL `IF … THEN … ELSE … END`. Note SurrealQL uses a single trailing `END`, not one per branch: ```rhai surreal::case_when() .when(surreal::avg(surreal::ident("price")) >= 250, "premium") .when(surreal::avg(surreal::ident("price")) >= 150, "mid") .else_("value") .expr() // → IF math::mean(price) >= 250 THEN "premium" ELSE IF math::mean(price) >= 150 THEN "mid" ELSE "value" END ``` ## Surreal-specific functions Stats, collection, string, and time helpers — each a single-purpose name over a `math::`/`array::`/ `object::`/`string::`/`time::` function: | Primitive | SurrealQL | |---|---| | `first(expr)` | `array::first(expr)` | | `len(expr)` | `array::len(expr)` | | `stddev(expr)` | `math::stddev(expr)` | | `median(expr)` | `math::median(expr)` | | `lower(expr)` | `string::lowercase(expr)` | | `words(expr)` | `string::words(expr)` | | `object_entries(expr)` | `object::entries(expr)` | | `object_values(expr)` | `object::values(expr)` | | `similarity(expr, term)` | `string::similarity::jaro_winkler(expr, "term")` | | `time_group(expr, unit)` | `time::group(expr, "unit")` | `similarity` and `time_group` bind their second argument (the search term, the bucket unit) as a parameter, like any other string operand. ## Field and element access Dotted paths and indexing use the `[…]` indexer rather than a separate primitive: ```rhai surreal::ident("department")["name"] // → department.name surreal::ident("nutrition")["sugar"] // → nutrition.sugar some_subquery[0] // → (…)[0] (integer index) ``` `surreal::ident("t")["col"].alias("x")` projects a path: `col_path AS x`. ## Graph traversal SurrealDB replaces joins with graph paths (`->edge->table`, `<-edge<-table`). One positional primitive expresses both directions — exactly one argument is the **anchor** (`surreal::me`, the current record, or a nested `surreal::graph(…)`), and its **position** sets direction: ```rhai surreal::graph(surreal::me, "placed", "order") // → ->placed->order (anchor left = outward) surreal::graph("reports_to", surreal::me) // → <-reports_to (anchor right = inward) surreal::graph(surreal::me, "reports_to", "employee")["name"] // → ->reports_to->employee.name ``` Mixed direction comes from nesting — each `graph()` appends one directed hop: ```rhai surreal::graph("client", "placed", surreal::graph(surreal::me, "placed", "order")) // → ->placed->order<-placed<-client ``` `recurse(path, min, max)` wraps a path in ranged recursion: ```rhai surreal::recurse(surreal::graph("employee", "reports_to", surreal::me), 1, 5)["name"] // → @.{1..5}(<-reports_to<-employee).name ``` ```admonish note title="Why position, not direction names?" An `out`/`in`/`up`/`down` vocabulary either assumes a hierarchy or names the arrow rather than the meaning. Anchoring by position keeps the call reading like the path it produces, and composition (nesting) carries mixed direction without per-edge glyph overrides. ``` ## Select clauses Surreal-only clauses on the select builder: ```rhai surreal::select().value().expression(surreal::max(surreal::ident("total"))).from("order").group_all() // → SELECT VALUE math::max(total) FROM order GROUP ALL surreal::select().field("price").from("product").split("tags") // → SELECT price FROM product SPLIT tags ``` `.subquery()` parenthesizes a select so it composes as a scalar expression — it bridges a select into the expression layer, where it pairs with the `[n]` indexer, `.alias()`, comparisons, and `from()`: ```rhai surreal::select().value().expression(surreal::max(surreal::ident("total"))) .from("order").where(surreal::ident("client") == surreal::parent("id")).group_all() .subquery()[0].alias("biggest_order") // → (SELECT VALUE math::max(total) FROM order WHERE client = $parent.id GROUP ALL)[0] AS biggest_order ``` ## Parameters `param(name)` is any SurrealDB `$`-parameter (`$auth`, `$this`, `$parent`, a `LET`-bound name). "Parameter" is SurrealDB's own term for `$`-prefixed names, so the primitive is named for it — and **not** `var`, which Rhai reserves as a keyword. ```rhai surreal::param("auth")["id"] // → $auth.id surreal::parent("id") // → $parent.id (sugar for the correlated-subquery case) ``` ## Embedded-array closures SurrealDB's `array.map`/`fold`/`filter` take an inline closure — the one place it exceeds the SQL vocabulary. These are written as **native Rhai closures**, not a closure-as-data constructor: each parameter is bound to a placeholder expression and the closure runs *symbolically*, so every operator and indexer in the body builds SurrealQL instead of computing a value. A `#{…}` map lowers to an object literal, `[…]` to an array literal, and `* + - /` render parenthesized. `map`, `fold` and `filter` are methods, so they take no namespace. ```rhai surreal::ident("lines").map(|l| #{ product: l["product"]["name"], subtotal: l["quantity"] * l["price"] }) // → lines.map(|$value| { product: $value.product.name, subtotal: ($value.quantity * $value.price) }) surreal::ident("lines").fold(0, |acc, l| acc + l["quantity"] * l["price"]) // → lines.fold(0, |$acc, $value| ($acc + ($value.quantity * $value.price))) ``` ```admonish warning title="The closure parameter name is engine-chosen" A Rhai local can't carry a `$`, so the engine never sees your `|l|` — it binds its own placeholders (`$value` for the item, `$acc` for a fold accumulator). The emitted SurrealQL therefore reads `|$value| …`, not `|$l| …`. This is cosmetic: SurrealQL doesn't care about the parameter name, and the result executes identically. ``` ## Checklist - Reach for a **named primitive** first; it handles the lowering, quoting, and escaping. - Call free functions through **`surreal::`**; call methods, indexers and operators on their receiver. - Use the **`vantage-sql` name** when the concept overlaps (`count`, `avg`, `round`, `coalesce`, `case_when`, `date_format`) — the same name works under `sql::`. - Build **field paths** with the `["col"]` indexer, **graph paths** with `surreal::graph()` + `surreal::me`, and **mixed direction** by nesting rather than glyphs. - Write **array closures** as native `|l| …` Rhai closures; expect engine-chosen `$value`/`$acc` parameter names in the output. - Drop to **`surreal::expr("…")` / `surreal::fx(…)`** only when no primitive fits. --- Source: https://vantage-ui.com/book/http-apis.html # HTTP APIs: REST & GraphQL The `vantage-api-client` crate reads tables from HTTP APIs. `RestApi` turns a table into `GET {base_url}/{endpoint}?…`, and `GraphqlApi` turns it into one query document posted to an endpoint. Both send through `vantage-api-pool`'s `ResilientClient`, which caps parallel requests, retries, rate-limits and trips a circuit breaker per API. Both drivers read a [YAML spec](./config-driven-vistas.md) through their Vista factory, the same way the database drivers do. The universal keys (`name`, `id_column`, `columns`, `references`) work as everywhere else, with two differences in references: REST takes multi-column `keys:` but not a column-level `references:`, and GraphQL takes a column-level `references:` but not `keys:`. Each driver adds one table-level block: `api:` for REST and `graphql:` for GraphQL. GraphQL also has a column-level `graphql:` block. Neither driver picks its own quicksearch columns. Both read the shared rule: the columns flagged `searchable`, or every column when none is flagged. GraphQL sends the search to the server when the table's `search` setting allows it. A REST Vista, and a GraphQL one with `search` off, refuses the search and leaves it to the consumer, which applies the same rule to the rows it holds. Settings that belong to the endpoint as a whole (base URL, auth, response envelope, paging parameter names, timeouts) are set once on the `RestApi` / `GraphqlApi` builder. Every table built from that API shares them. The YAML blocks carry only what differs per table. Both drivers read only. Their Vistas advertise no write capability, so a write is refused with an `Unsupported` error before any request, and so are `sum`, `min` and `max` on the typed tables. Send a write yourself with `RestApi::http_request` or `GraphqlApi::post_graphql`. These go through the same client, so they share its cap, breaker and observer. ## Pages - [REST: the `api:` block](./http-apis/rest-yaml.md): endpoint templates, references, column types, and the builder settings that sit beside them. - [GraphQL: the `graphql:` blocks](./http-apis/graphql-yaml.md): every table and column key, dialects, paging, counting, search and references. --- Source: https://vantage-ui.com/book/http-apis/rest-yaml.html # REST: the `api:` block A REST spec is a `RestApiVistaSpec`. Its only driver-specific key is the table-level `api:` block. Columns and references use the universal spec keys, with no `api:` block of their own. ```yaml name: albums columns: id: { type: int, flags: [id] } userId: { type: int } title: { type: string, flags: [title] } api: endpoint: users/{userId}/albums references: photos: { table: photos, kind: has_many, foreign_key: albumId } ``` ```rust,no_run # use vantage_api_client::{ResponseShape, RestApi}; # const ALBUMS_YAML: &str = r#" # name: albums # columns: # id: { type: int, flags: [id] } # userId: { type: int } # title: { type: string, flags: [title] } # api: # endpoint: users/{userId}/albums # references: # photos: { table: photos, kind: has_many, foreign_key: albumId } # "#; # const PHOTOS_YAML: &str = r#" # name: photos # columns: # id: { type: int, flags: [id] } # albumId: { type: int } # title: { type: string, flags: [title] } # "#; # fn main() -> vantage_core::Result<()> { let api = RestApi::builder("https://jsonplaceholder.typicode.com") .response_shape(ResponseShape::BareArray) .build(); let mut factory = api.vista_factory(); factory.register_yaml(ALBUMS_YAML)?; factory.register_yaml(PHOTOS_YAML)?; let albums = factory.build("albums")?; # let _ = albums; # Ok(()) # } ``` Building the factory sends nothing; the first read does. The block is compiled but not run, because `RestApi::build` makes a `reqwest` client, which reads the system's proxy settings. ## Table keys | Key | Type | Default | Meaning | |---|---|---|---| | `api.endpoint` | string | the spec `name` | Path under the API's base URL. May hold `{placeholder}` segments. | The whole `api:` block is optional. Unknown keys are a parse error. ### Endpoint templates Each `{placeholder}` is filled at request time from the eq-condition on the column of that name, and that condition is used up. Any eq-condition left over goes into the query string (`?title=x`), or is applied to the fetched rows when the API was built with `FilterStrategy::Client`. A placeholder with no condition to fill it is an error (`No eq-condition provided for URI placeholder`), so nothing is sent. So `users/{userId}/albums`, narrowed to `userId = 1`, fetches `GET /users/1/albums`. Traversal from a user to its albums arrives at that URL without any code. ## Columns | `type` | Column | |---|---| | `int`, `integer`, `i64`, `i32` | `i64` | | `float`, `double`, `f64`, `f32` | `f64` | | `bool`, `boolean` | `bool` | | `string`, `text`, `str` (default) | `String` | | `json`, `any` | raw CBOR value | Any other type fails the build. `flags:` works as in every driver (`id`, `title`, `hidden`, …). A column with a `lazy:` script is computed from each fetched record. It needs the crate's `rhai` feature. A row is keyed by its id column. A row whose id is missing, null, empty, or not a string or number is keyed `~row:`, where the position is absolute in the result. Two rows with the same id in one response are an error. ## References `references:` takes the universal form: `table`, `kind` (`has_one` / `has_many`) and `foreign_key`, or `keys:` for a multi-column join. Traversal from a row reads each join value from the parent row and gives the target one eq-condition per key. Each condition then fills a placeholder in the target's `endpoint`, becomes a query parameter, or filters rows, by the rules above. - `has_one`: the parent's `foreign_key` value narrows the target's id column. - `has_many`: the parent's id narrows the target's `foreign_key` column. - `keys:`: a list of `{ from, to }` pairs, a parent column to a target column. When given, the pairs are the whole join and `foreign_key` is ignored. A deployment narrowed by both its product and its version is `keys: [{ from: product_id, to: product_id }, { from: version_id, to: version_id }]`. The REST factory reads the `references:` map only. A column-level `references:` (the `userId: { type: int, references: users }` form other drivers accept) is parsed but doesn't become a relation, so declare REST joins in the map. Targets are found by name. `register_yaml` keeps them in the factory's own registry. If the models live across several drivers, `with_model_resolver(resolver)` takes an `Arc` that returns a built `Vista` for a name instead: a closure, or a [`Catalog`](vantage_vista::Catalog). ## Builder settings The block is deliberately small. Everything that describes the API, rather than one table, is set on `RestApi::builder(base_url)`: | Builder method | Default | Meaning | |---|---|---| | `response_shape(ResponseShape)` | `Wrapped { array_key: "data" }` | Where the rows are: `BareArray`, `Wrapped { array_key }`, `WrappedByTableName` | | `pagination_params(PaginationParams)` | `page_limit("_page", "_limit")` | Paging parameter names; `skip_limit("skip", "limit")` for 0-based offsets. Setting it enables windowed reads | | `no_pagination()` | off | Send no paging parameters; the first page is the whole set | | `total_key("count")` | none | Envelope key holding the total row count | | `filter_strategy(FilterStrategy)` | `Query` | `Client` keeps non-path conditions off the wire | | `ordering(OrderingParams::new("ordering", "-"))` | none | Sort parameter; makes every column orderable | | `auth(..)` / `auth_refresher(..)` / `bearer_source(..)` | none | `Authorization` header: fixed, refreshed after a `401`, or asked for on every request | | `max_parallel`, `rate_limit`, `request_timeout`, `observer`, `http_client` | 4, none, 30 s, none, default | Transport; see the crate's `README_transport.md` | When each signed-in session has its own token, build one `RestApi` and hand each session `api.with_bearer_source(source)`: a clone that shares the client (connection pool, cap, breaker) but asks its own source for the token on every request. Don't give the shared API an `auth_refresher`, since the client adds that header itself and a clone can't drop it. A REST Vista can always count and traverse. It advertises `Window` when the API pages (a `total_key` puts the grand total on every page), and `Order` when `ordering` is set. It never advertises `Search`, so the Vista refuses a quicksearch rather than send one the API can't apply. A Dio over it searches the rows it holds instead, over the columns flagged `searchable`, or every column when none is flagged. --- Source: https://vantage-ui.com/book/http-apis/graphql-yaml.html # GraphQL: the `graphql:` blocks A GraphQL spec is a `GraphqlApiVistaSpec`. It has a table-level `graphql:` block for the root field and what the server accepts, and an optional column-level `graphql:` block for a column whose server field has a different name. ```yaml name: launches id_column: id columns: id: { type: string, flags: [id] } mission: { type: string, flags: [title, searchable], graphql: { field: mission_name } } launch_year: { type: int } rocket_id: { type: string } rocket.rocket_name: { type: string } references: rocket: { table: rockets, kind: has_one, foreign_key: rocket_id } graphql: root_field: launches dialect: hasura paginate: true count: aggregate ``` ```rust,no_run # use vantage_api_client::{GraphqlApi, GraphqlApiVistaSpec}; # use vantage_vista::VistaFactory; # const LAUNCHES_YAML: &str = r#" # name: launches # id_column: id # columns: # id: { type: string, flags: [id] } # mission: { type: string, flags: [title, searchable], graphql: { field: mission_name } } # launch_year: { type: int } # rocket_id: { type: string } # rocket.rocket_name: { type: string } # references: # rocket: { table: rockets, kind: has_one, foreign_key: rocket_id } # graphql: # root_field: launches # dialect: hasura # paginate: true # count: aggregate # "#; # fn main() -> Result<(), Box> { let api = GraphqlApi::new("https://hasura.example.com/v1/graphql"); let spec: GraphqlApiVistaSpec = serde_yaml_ng::from_str(LAUNCHES_YAML)?; let launches = api.vista_factory().build_from_spec(spec)?; # let _ = launches; # Ok(()) # } ``` `build_from_spec` comes from the [`VistaFactory`](vantage_vista::VistaFactory) trait, so it needs `use vantage_vista::VistaFactory;`. The block is compiled but not run: building the API makes a `reqwest` client, which reads the system's proxy settings. ## Table keys | Key | Type | Default | Meaning | |---|---|---|---| | `root_field` | name | the spec `name` | Query field the rows are read from | | `dialect` | `generic` \| `hasura` | the API's dialect | How filters are spelled; lowercase only | | `filter_arg` | name | the API's, else `where` (Hasura) / `find` (Generic) | Name of the filter argument | | `args` | map | the API's `root_args` | Literal arguments always passed to the root field | | `response_path` | dotted path | the API's | Path walked into the root field's value to reach the rows (`edges.node`) | | `filter` | bool | `true` | Conditions may be sent to the server | | `order` | bool | `true` on Hasura, else `false` | `order_by:` may be sent | | `search` | bool | `true` on Hasura, else `false` | Quicksearch may be sent as `_ilike`s; never with `filter: false` | | `paginate` | bool | `false` | `limit:` / `offset:` may be sent | | `count` | `list` \| `aggregate` | `list` | How `get_count` is answered | | `count_field` | name | the API's, else `_aggregate` | Aggregate field for `count: aggregate` | The whole block is optional, and unknown keys or values are a parse error. Every name the block puts into a query is checked against the GraphQL name grammar (`[_A-Za-z][_0-9A-Za-z]*`, dotted paths segment by segment) when the Vista is built. The block's `filter`, `order`, `search`, `paginate` and `count` override the API's `Supports` for that table, key by key. A key left out keeps the builder's setting, and the default in the table above applies only when the builder left it unset too. ### Dialects | Dialect | Filter | Operators | |---|---|---| | `hasura` | `launches(where: {launch_year: {_eq: 2018}})` | the full set, plus `_and` / `_or` / `_not` | | `generic` | `launches(find: {launch_year: 2018})` | equality only | A condition the dialect can't spell fails before anything is sent. On Generic, a Vista `contains` filter is refused as unsupported, so the consumer filters locally. ### `filter: false` This is for a root field that takes no filter argument. Conditions, including traversal conditions, are applied to the fetched rows instead. Search, operators and paging switch off, and counts list. ### Paging With `paginate: true` and `filter` on, the Vista advertises `Window` and serves `fetch_window`, sent as `$limit` / `$offset` variables. It pages by offset only, so it never advertises `Cursor`. With `filter: false` a server page would be cut short by the local filter, so paging is not offered. On Hasura, a paged read with no order of its own is ordered by the id column so that pages stay stable. A Generic schema can't sort, so its pages are only as stable as the server's natural order. ### Counting `count: list` lists the rows and counts them, which works for every role. `count: aggregate` (Hasura, with `filter` on) sends `(where: …) { aggregate { count } }`. That needs a role with `allow_aggregations`. If the server rejects the query, the count fails with an error that suggests `count: list`; it never falls back silently. Generic tables always list. ### Search and ordering A quicksearch ORs `_ilike` over the columns every driver searches: those flagged `searchable`, or every column when none is flagged (`Table::search_columns`). `_ilike` takes text, so when nothing is flagged GraphQL keeps only the string-typed columns other than the id. Dotted columns are then skipped. The rule is settled before they are, so a table whose only searchable columns are dotted matches nothing rather than widening to every string column. With no column left the search matches nothing. `%`, `_` and `\` in the text are escaped. Ordering a dotted column is refused. ## Column keys | Key | Type | Default | Meaning | |---|---|---|---| | `graphql.field` | name | the column name | Server field the column reads | The query selects an aliased column as `mission: mission_name`, so rows keep the column name. Conditions, sort keys and search on the column are sent as the server field. Both names must be plain (undotted). A dotted column (`rocket.rocket_name`) renders as a nested selection, `rocket { rocket_name }`. A `lazy:` column is computed from each fetched row and never selected (crate feature `rhai`). | `type` | Column | |---|---| | `int`, `integer`, `i32` | `i32` | | `bigint`, `i64`, `long` | `i64` | | `float`, `double`, `f64`, `f32` | `f64` | | `bool`, `boolean` | `bool` | | `string`, `text`, `str`, `id` (default `string`) | `String` | | `datetime`, `timestamp`, `timestamptz` | `DateTime` | | `date`, `time` | `NaiveDate`, `NaiveTime` | | `uuid` | `Uuid` | | `json` | `serde_json::Value` | Get-by-id sends an integer literal for an int id column, and a quoted string otherwise. A row whose id is null or empty is keyed `~row:`. Two rows with the same id in one response are an error. ## References A relation is declared in one of two places: - a `references:` entry: `table`, `kind` (`has_one` / `has_many`) and `foreign_key`; - a column-level `references:`, for a has-one keyed by the column itself. `rocket_id: { type: string, references: rockets }` is a has-one named `rocket_id`, joining `rocket_id` to the id of `rockets`. An entry of the same name in the `references:` map wins. Multi-column `keys:` on a `references:` entry are refused when the Vista is built, with an unsupported error naming the relation. A GraphQL filter argument narrows by one field per relation here, so the driver refuses a join it would only half apply. Traversal gives the target an eq-condition: `has_one` narrows the target's id by the parent's `foreign_key`, and `has_many` narrows the target's `foreign_key` by the parent's id. A text key is sent as an integer when the target column is an int. A target is built from its own spec on the factory's API, without the parent table's `graphql:` overrides. Targets come from the factory's registry (`register_yaml`, then `build(name)`), or from a spec resolver: ```rust,no_run # use std::collections::HashMap; # use std::sync::Arc; # use vantage_api_client::{GraphqlApi, GraphqlApiVistaSpec}; # use vantage_vista::SpecResolver; # let api = GraphqlApi::new("https://hasura.example.com/v1/graphql"); # let specs: HashMap = HashMap::new(); let resolver: Arc> = Arc::new(move |name: &str| specs.get(name).cloned()); let factory = api.vista_factory().with_resolver(resolver); ``` Any `Fn(&str) -> Option` closure is a [`SpecResolver`](vantage_vista::SpecResolver); a spec is looked up each time a traversal builds its target, so the target always reflects the current spec. A reference to a model with no spec is an error naming the relation and the target, and no request is sent.