Concepts¶

Status¶

Legend: ✅ implemented · 🟡 partial · ❌ not implemented yet.

Feature	Status	Where
`ost://`, `file://`, `lvm://`, `zfs://`, `mds://` schemes	✅	`src/backend.cpp`, `src/*_backend.cpp`
Comma-separated location/target lists, duplicate URIs rejected	✅	`src/location.cpp`, `src/target.cpp`
`\,` escaping inside a URI	✅	`librawstd/src/uri.cpp`
Mirroring policy (`ost://a,ost://b`)	✅	`src/chunk.cpp`, see Mirroring
Data locality (`file://` on the hypervisor + `ost://`)	🟡	works as a regular mirror with local reads; no cache / primary-store asymmetry (see Multiple backends)
Object / Chunk / Slot runtime model	✅	`src/object.cpp`, `src/chunk.cpp`, `src/slot.cpp`
Internal multi-chunk form, chunk offset path segment	✅	`src/target.cpp` (`parse_target_path()`)
`rawstor_target_chunks()`	✅	`include/rawstor/target.h`
Version target (`.../<uuid>/<version_uuid>`), read-only open	✅	`src/target.cpp`
`mds://` object-level version	✅	`src/mds_backend.cpp`
Per-slot native CoW version: `zfs://`	✅	`src/zfs_backend.cpp`
Per-slot native CoW version: `ost://` (relayed to the server's backend)	✅	`src/wire_backend.cpp`
Per-slot native CoW version: `lvm://`	❌	`-ENOTSUP` (waits for an lvm-thin backend)
Per-slot native CoW version: `file://`	❌	`-ENOTSUP` by design (no CoW)

Overview¶

Rawstor addresses data through six related concepts, each with its own URI form: Location, Target, Object, Chunk, Slot, and Version. A Location names a backend store; a Target names one specific object within it; Object/Chunk/Slot are the layers a Target decomposes into once opened; Version is a bound, read-only copy of any of them. Only Location and Target are syntax a caller ever types by hand (CLI arguments, rawstor_target_*()'s own target string) — Object, Chunk and Slot are the client library's internal model of what a Target opens into, described here so the two visible forms make sense in context.

How the pieces fit¶

A caller types a Location or a Target; opening a Target gives an Object, which the library splits into Chunks, each mirrored over one or more Slots, each talking to one Backend:

classDiagram
    direction TB
    Location "1" o-- "*" Target : holds objects
    Target "1" ..> "1" Object : open()
    Object "1" *-- "1..*" Chunk : routes I/O by offset
    Chunk "1" *-- "1..*" Slot : mirrors
    Slot "1" --> "1" Backend : one connection
    Backend <|-- file
    Backend <|-- lvm
    Backend <|-- zfs
    Backend <|-- ost
    Backend <|-- mds
    class Location {
      <<typed by the caller>>
      one URI per backend
      +create(spec) Target
      +list() Targets
      +info() used, total
    }
    class Target {
      <<typed by the caller>>
      Location + object uuid
      optional version uuid
      +create(spec)
      +open(flags) Object
      +meta() / spec()
      +create_version()
      +resize(size)
      +remove()
    }
    class Object {
      <<runtime handle>>
      size, chunk_size
      +pread() / pwrite()
      +discard() / write_zeroes()
      +flush() / close()
    }
    class Chunk {
      <<mirror consistency>>
      offset = index x chunk_size
      state, epoch, sync_id
      +quorum open, degrade, resync()
    }
    class Slot {
      <<one mirror arm>>
      one bare URI
      +retry / reconnect()
    }
    class Backend {
      <<physical store>>
      +create / remove / meta()
      +read / write()
    }
    class mds {
      <<resolves, stores nothing>>
      expands one object into
      per-chunk ost URIs
    }
    note for Location "ost://host:7777
    file:///var/rawstor
    mds://host:7776
    comma-separated = mirrors"
    note for Target "ost://host:7777/UUID
    ost://a/UUID,ost://b/UUID
    mds://host:7776/UUID/VERSION_UUID"

The same layers on a concrete object: a 1.5 GiB mds:// object with 512 MiB chunks, two copies each. The caller only ever sees the Target at the top; everything below it is built by the library from the MDS's chunk map, and the leaves are plain ost:// servers:

flowchart TB
    T(["Target<br/>mds://mds:7776/U"])
    O["Object U<br/>size 1.5 GiB, chunk_size 512 MiB"]
    T -- "open()" --> O

    subgraph C0 ["Chunk @ 0"]
        S00["Slot<br/>ost://a:7777/U/0"]
        S01["Slot<br/>ost://b:7777/U/0"]
    end
    subgraph C1 ["Chunk @ 512 MiB"]
        S10["Slot<br/>ost://b:7777/U/20000000"]
        S11["Slot<br/>ost://c:7777/U/20000000"]
    end
    subgraph C2 ["Chunk @ 1 GiB"]
        S20["Slot<br/>ost://c:7777/U/40000000"]
        S21["Slot<br/>ost://d:7777/U/40000000"]
    end
    O --> C0 & C1 & C2

    A[("OST a")]
    B[("OST b")]
    C[("OST c")]
    D[("OST d")]
    S00 --> A
    S01 --> B
    S10 --> B
    S11 --> C
    S20 --> C
    S21 --> D

Location¶

A location specifies the address of a backend data store (or a list of backends). It is expressed as a comma-separated list of URIs. The URI format follows the standard scheme <scheme>://<endpoint>.

Currently, five URI schemes are supported:

Scheme	Description
`ost`	Backend server speaking the OST protocol (see Protocol)
`file`	Local filesystem backend (a folder path)
`lvm`	Local LVM volume group backend
`zfs`	Local ZFS pool backend
`mds`	Metadata server addressing a whole chunked, possibly multi-copy object rather than a single physical store (see mds.md)

Single backend examples¶

ost://<host>:<port> – an OST server at the given host and port.
file://<path_to_folder> – a folder on the local filesystem.

Multiple backends (comma‑separated)¶

When multiple URIs are listed, the client interprets the list according to specific policies:

Example	Behavior
`ost://host1:port1,ost://host2:port2`	Mirroring – both backends contain identical data.
`file:///data/folder,ost://host:port`	Data locality – the file backend serves as a local cache or fast access path, while the OST backend is the primary remote store.

Today data locality is plain mirroring (see Mirroring) with the local copy listed first: reads are served from the first IN-SYNC member in list order, so they stay on the hypervisor's own disk. The two copies are still equal mirrors, not a cache in front of a primary store:

A write is acknowledged only once it has completed on both copies, so write latency is the remote OST's.
The local copy is a full copy of the object, not a cache with eviction.
With two copies, an open needs both reachable (quorum is > N/2): if either the local disk or the OST is down, the object does not open automatically.
After a client crash with both copies DIRTY (case F5 in Mirroring), the first copy in the list, the local one, is the reference (a copy that is CLEAN is preferred to it) and the remote copy is healed from it while the object stays writable.

Syntax rules: - Do not add spaces between URIs – use a single comma: uri1,uri2 - To include a literal comma within a URI, escape it with a backslash: \, - Each URI must be a valid location (scheme + endpoint). - All URIs in the list must be unique – duplicates are not allowed.

Target¶

A target identifies a specific object a caller can open, read, write, create, remove, or create a version of. For a single-URI target, the format is <scheme>://<endpoint>/<uuid>. For multiple URIs (mirroring), the UUID is appended to each one: <scheme1>://<endpoint1>/<uuid>,<scheme2>://<endpoint2>/<uuid>,...

A target that names one version of an object appends the version's UUID: <scheme>://<endpoint>/<uuid>/<version_uuid> (on every URI of a multiple target). Such a target can only be opened read-only (RAWSTOR_READONLY, which still needs the quorum of a writable open unless combined with RAWSTOR_FORCE) or removed.

Where: - <scheme> and <endpoint> are the same as for location. - <uuid> is the unique identifier of the object (rawstor uses UUID v7). - <version_uuid> is the identifier of a version of that object (also UUID v7), optional.

Single backend target examples¶

ost://<host>:<port>/<uuid> – an object stored on a single OST server.
file://<path_to_folder>/<uuid> – an object stored as a file in a local folder.
mds://<host>:<port>/<uuid> – a whole object addressed through its MDS; always a single URI (no comma list — mirroring/locality here happen per chunk, inside the object, not at this level). See mds.md.

Multiple backend target (mirroring / locality)¶

The client uses the same policies as for locations (mirroring, locality, etc.).

Example: ost://127.0.0.1:9090/019cbfad-a389-7d42-a0f6-c29993ac8c00,file:///var/rawstor/019cbfad-a389-7d42-a0f6-c29993ac8c00

This target references the same object (UUID 019cbfad-a389-7d42-a0f6-c29993ac8c00) on two different backends.

Important: All URIs in a target list must point to the same UUID. Mixing different UUIDs in one target is not allowed.

Internal multi-chunk form (not for manual entry)¶

Opening a target (rawstor_target_open()) resolves it into an Object made of one or more Chunks (see below). For a plain target, this is always a single chunk — the target string above, in full. mds:// objects need more than one: the client library builds, internally, a single flat ,-separated list of every chunk's own URIs, one after another, with no second separator marking where one chunk's own group ends and the next begins — Target's own constructor sorts them back into their chunk groups itself, by each URI's own internal offset path segment (a chunk's byte offset within the object, logical_ index * chunk_size — URIs sharing one offset are mirrors of the same chunk, distinct chunks always differ). This form only ever exists inside the library (built by the MDS backend from the object's chunk map) — it is never part of the target syntax a caller types, appears in the CLI, or is returned by any rawstor_target_*() call. It is documented here only so the internal model below is unambiguous about where a multi-chunk Object's own per-chunk URI lists come from.

Object¶

The Object is the client-facing handle a target opens into (Architecture: "Object = group of chunks") — what rawstor_target_open() returns, and what every rawstor_object_*() I/O call (pread/pwrite/discard/write_zeroes/flush/close) acts on. It has no URI form of its own: it is derived from the target string that opened it, routing each I/O request to the Chunk that logically owns the touched byte range. A plain target's Object is always the degenerate case of exactly one Chunk; an mds:// target's Object may route across many.

Chunk¶

A Chunk is one logical piece of an Object's data — for a plain target, the whole object; for an mds:// object, one fixed-size slice of it (docs/mds.md: with 1 GiB chunks, a 1 TiB object has ≤ 1024 chunks). A Chunk owns the mirror-consistency protocol (member roles, sync_id and epoch, degrade/resync — docs/mirroring.md) across its own one or more Slots. Its URI form is exactly a plain target's own: one URI per mirror, comma-separated, all sharing the same UUID — ost://h1:p1/<uuid>,ost://h2:p2/<uuid>. A single-chunk Object's one Chunk is the target string that opened it; a multi-chunk mds:// object's chunks are each one same-offset group of the internal form above.

An OST session opens a Slot for one copy, using the object UUID, chunk offset and version supplied by SET_OBJECT. Its I/O offsets are relative to that copy. The Slot enforces the writer's configuration epoch and protects client writes during resync; the client-side Chunk coordinates the mirror and routes I/O to its Slots.

Slot¶

A Slot is one physical mirror arm of a Chunk: a single URI, backed by one connection to one backend, responsible for retrying and reconnecting on that one arm independently of its siblings. Its URI form is a single, bare URI: ost://h1:p1/<uuid>. A Chunk with N mirrors has N Slots; a plain, unmirrored target's Chunk has exactly one.

Version¶

A version is a bound, read-only copy of a target/chunk/slot, identified by a version id (version_id, a UUID -- never nil, nil always means "live"). Like every other id in this design, version_id is client-generated -- the caller picks it (or generates a fresh one, the same single point of generation a fresh object id comes from) and passes it to rawstor_target_create_version(), either as its version_id argument or already embedded in the target string (with version_id NULL); with neither, a fresh id is generated. rawstor_target_create() never creates a version -- it rejects a target that carries one with -EINVAL. There is no separate "assign" mode, mds:// included: two independent mechanisms use the same id, at different layers:

mds:// object-level: CoW-every-chunk/OBJ_COMMIT_VERSION (docs/mds.md, "Versions (stage 2)") registers the caller's id against the whole object, driven by mds::Backend::create_version().
Per-slot native CoW: a single backend's own thin-clone/version primitive (zfs::Backend today), addressed directly by target/chunk- slot URI with the version appended as a trailing path segment, in either of two equivalent shapes:
Logical — no offset segment, for a plain target (there is no chunk-offset concept to name at that level): ost://host:port/<uuid>/018f4e2a-3000-7000-8000-000000000001, or on an mds:// target, mds://host:port/<id>/018f4e2a-3000-7000-8000-000000000001.
Physical — an explicit offset segment right before the version (never omitted, even "0" — see "Chunk offset" below for why), the shape every chunk-of-a-larger-object address always uses: ost://host:port/<uuid>/0/018f4e2a-3000-7000-8000-000000000001.

Either way this segment is never part of a location and never carries a comma-separated list of its own — it binds whichever single URI it's attached to.

Chunk offset (not for manual entry)¶

A chunk's own byte offset within its parent mds:// object (logical_index * chunk_size) rides the same URI, as a path segment right after the UUID: ost://host:port/<uuid>/100000 (hexadecimal -- 0x100000, 1MiB; optionally followed by a version segment, e.g. ost://host:port/<uuid>/100000/018f4e2a-3000-7000-8000-000000000001). Offset and version are told apart from an arbitrary preceding location path by shape alone (a UUID-shaped segment vs. a hexadecimal one): parse_target_path() (src/target.hpp) reads the identity off the end of the path. A target binds at most one version, so the trailing run of UUID-shaped segments is at most two long (longer is rejected). A single trailing UUID preceded by a valid hexadecimal offset (and another UUID, the id) is the version — the physical, with a bound version shape above; otherwise the run's first segment is the id and its second (if any) the version — the logical shape, offset implied 0. A location path that itself ends in a UUID-shaped directory is therefore ambiguous and not supported. If the last segment isn't UUID-shaped at all (no version to find), it must instead be the hexadecimal offset itself, with the id right before it — ost://host:port/<uuid>/ 100000 above, the physical, live shape: an explicit offset with no version anywhere in the path. A chunk's own offset is always spelled out explicitly (one of the two physical shapes) since it's never 0-and-absent-by-convention the way a plain target's own implied offset is. Like the internal multi-chunk form above, this is built only by mds::Backend from its own chunk map and never something a caller types — Target's own constructor reads it back out to reconstruct chunk grouping. rawstor_target_chunks() returns addressable physical chunk target strings, preserving their offset in each URI directly (a single entry, 0, for a plain, non-mds:// target).

Summary table¶

Concept	Format	Purpose	Example
Location	`<scheme>://<endpoint>` or `uri1,uri2,...`	Address of a backend data store (or a set of stores)	`ost://127.0.0.1:9090` `file:///var/rawstor`
Target	`<scheme>://<endpoint>/<uuid>` or `uri1,uri2,...`	Address of a specific object a caller opens/creates/removes	`ost://127.0.0.1:9090/019cbfad-a389-7d42-a0f6-c29993ac8c00` `ost://127.0.0.1:9090/019cbfad-a389-7d42-a0f6-c29993ac8c00,file:///var/rawstor/019cbfad-a389-7d42-a0f6-c29993ac8c00`
Object	(none — derived from the target string that opened it)	The open handle `rawstor_object_*()` I/O acts on; routes to the owning Chunk	—
Chunk	Same as Target: `uri1,uri2,...` (one UUID per group)	One logical slice of an Object's data; owns mirror consistency	`ost://h1:p1/<uuid>,ost://h2:p2/<uuid>`
Slot	A single URI	One physical mirror arm of a Chunk; owns retries/reconnects	`ost://h1:p1/<uuid>`
Version	Trailing path segment on a target/chunk-slot URI (logical: right after the id; physical: after an explicit offset), `version_id` a client-generated UUID	A bound, read-only copy	`ost://host:port/<uuid>/018f4e2a-3000-7000-8000-000000000001`

Notes¶

When using the file:// scheme, the path must be absolute. Relative paths are not allowed.
The wire protocol (commands, frame layouts, error reporting) is described in the protocol specification.
On naming: this document's "Location"/"Target" pair, and the Object/Chunk/Slot model above, could arguably be called a "topology" — but that word is already taken: mds.md uses "topology" for a different, unrelated concept (the MDS's own static list of known OSTs, TOPOLOGY_PATH/topology.conf, scanned by rawstor-mds --reconstruct). Reusing it here for this document would make the two docs' shared vocabulary actively misleading, so this file keeps its current name.

Witness members¶

Witnesses use ordinary backend addresses. With width + 1 locations and an even data width, the last location is a witness, whether reachable or not. Opening checks each available member's META.member_role against its expected position; it does not search for a witness by role. width counts only data copies; config.nroles counts all members. One witness is supported only with an even number of data members. It participates in synchronous membership barriers and quorum; data I/O and the data-copy phase of resync skip it. Keep the complete target and member order. See Witness.

Addressing metadata operations¶

A logical mds://host:port/UUID addresses a whole object through its placement map. mds://host:port/UUID/OFFSET selects a logical chunk through that map. A physical member always uses its native backend scheme: ost://host:port/UUID/OFFSET also addresses a witness stored on MDS, whose server speaks the member protocol. Routing follows the URI scheme, without probing for witness records. rawstor_target_chunks() returns physical chunk targets; Python Target.chunks() returns Target instances. Metadata and member-configuration operations take a chunk target and no separate offset. Read/write offsets remain offsets within an opened object.

rawstor show -v prints a target: line under each chunk; this target can be passed directly to rawstor resolve TARGET --winner=N.

Physical member slots¶

rawstor_target_slots() returns an owned string list with one physical member URI per entry; free it with rawstor_string_list_delete(). Python exposes the same operation as Target.slots(), returning a list of single-URI Targets.

A logical mds://host:port/UUID/OFFSET resolves to that chunk's physical slots. Without an offset, all chunks are enumerated in offset order, then member order. A physical comma-separated target is split without querying the members; unreachable slots remain in the result. Version suffixes are preserved. Witnesses on MDS use ost:// at the MDS endpoint. rawstor show -v prints each slot's target: under its mirror[N], even when the member is unreachable.

For new MDS chunks with an even DATA width, placement adds a witness on an unused failure domain. The required topology role column permits data, witness or both (data,witness in either order), independently of URI. MDS backends use ost:// with role witness. With no eligible extra failure domain, placement fails instead of silently leaving out a witness.