The mathematics behind the encoding layer

Datasent is built on a small number of precise mathematical ideas. This page is where we publish our work: the papers that describe the full framework, the open problems we're working on, and the technical writing that explains how specific parts of the system work and why.
Papers

The underlying research powering our technology

A Mathematical Framework for Structured Information Encoding

Sets out the framework behind Datasent's Polygen: each window of data is fitted to a basis and stored as coefficients plus a residual, a token that is smaller than the record, reconstructs it exactly, and can be queried without decoding.

A Trusted Setup for Bandwidth-Minimal, Lossless Data Tokenization

Presents the trusted setup protocol for lossless tokenisation that enables exact reconstruction while transmitting only minimal residual information. Covers deterministic canonicalisation, integer-lossless representation, the residual-only transmission protocol, threshold and custodian reconstruction, and integrity verification. Demonstrates that raw data never needs to traverse organisational or network boundaries.

Non-Identifiability of Polynomial Token Components: Quantitative Reconstruction Limits

Establishes exactly when a coefficient-only or residual-only view fails to identify the original data, and quantifies what an ideal decoder can recover. On three exhaustively enumerated domains, exact recovery from a single component was 2.4% to 6.9%, with over 99% of inputs in non-singleton candidate pools. Extends the analysis to repeated releases, policy-selected disclosure, and coalition views, and states precisely which privacy properties the results do and do not provide.
Privacy

What the mathematics proves, and what it does not yet

For a fixed basis, exact recovery of the original from coefficients alone or residuals alone is provably non-unique: no decoder, with any amount of computation, can be right on every admissible input. The same framework supplies exact Bayes recovery probabilities, conditional entropy, min-entropy, candidate-pool distributions, and coalition views, so the ambiguity of any declared release can be measured rather than asserted. What does not yet exist is a product profile that binds those measurements to the complete token package under a declared real-world threat model. That packaging step, together with held-out attack evidence or an added randomised mechanism, is what turns the mathematics into a customer-defensible privacy claim.

Code, models and demos

The public artifacts behind the numbers on this site: adapters with their recipes and limitations, demos that run against a private Datasent endpoint, and the evaluation environment where numbers are recorded on a named host.

Qwen2-VL adapters

Rank-16 LoRA adapters for Qwen2-VL-2B and Qwen2-VL-7B, trained on 300 ScienceQA images. Each model card carries the per-tier results, the training recipe, and what the adapter is not for. Released under a Datasent evaluation licence.

Live demos

The side-by-side vision-language comparison and the encode-once RAG corpus. Thin clients on Hugging Face Spaces; inference and encoding run on a Datasent endpoint, so nothing proprietary runs in the browser. First request after a quiet period can take up to a minute.

Evaluation environment

poly.datasent.com runs the SDK on your own change streams and telemetry files; sdk-api.datasent.com exposes the encoder as an HTTP API. Every number shown is a measurement on that host or a labelled projection from one. Access is issued per evaluation.
Open problems

The questions we're still working on

Datasent is a complete, working system. But there are directions we know matter and haven't fully solved. We publish them here because intellectual honesty about what's open is more useful than pretending everything is settled.
1

Formal complexity analysis for multi-dimensional canonical forms

The primary mathematical analysis treats canonical data as a matrix partitioned along its first axis. Extending the formal analysis (complexity bounds, compression conditions, operator compatibility) to multi-dimensional canonical forms, including the image and video decompositions the system already handles in practice, is an evolving research direction.
2

Operator compatibility beyond linear maps

The token algebra defines how operations compose over the structured representation, and the current formal results cover linear operators: the cases where a transform on the data corresponds to a transform on coefficients. Extending that guarantee to nonlinear operations, including model inference and thresholding, is where token-native compute depends on open work.
3

Optimal basis selection under distribution shift

The trusted setup fixes basis families before any data is seen, and the MDL objective then picks the best fit per segment. When the data's underlying structure drifts over time, the agreed basis can become a poor fit and the residual grows. How to detect that drift and renegotiate a basis without falling back to raw transfer is unresolved.
4

Zero-knowledge proof integration

Integrity today rests on deterministic reconstruction and a hash of the canonical form, which detects tampering without any cryptographic machinery beyond a collision-resistant hash. Proving that a residual corresponds to a valid fitting, or that a reconstruction was authorised, without revealing the underlying data would require ZK proof integration. The architecture is designed to support it; the proofs themselves are not built.
5

End-to-end empirical evaluation

The compression and bandwidth figures published here come from internal measurements on specific workloads. A systematic evaluation across all supported modalities, against standard baselines such as Parquet, FLAC, and PNG on public datasets, with published methodology, has not yet been done. It is the work most likely to change the numbers on this site, in either direction.
6

Binding ambiguity measurements to the token package

The defensible privacy construction combines three guarantees. First, no extra package leakage: prove that every byte of a selected token is generated solely from a frozen allowed-leakage function, given declared side information and public configuration. Second, a token-specific risk certificate: before release, calculate or bound the optimal exact-recovery probability, conditional min-entropy, mutual information, and Bayes risk, plus identity and sensitive-attribute risks where relevant, and release only if the token passes a declared policy (for example, at least 100 compatible candidates and a posterior exact-recovery probability no greater than 1 percent). Candidate count alone is insufficient under a concentrated prior, so both multiplicity and posterior risk must be certified. Third, held-out inference evidence: run reconstruction, identification, linkage, membership, and attribute attacks against the actual serialised artifact, compared with prior-only, metadata-only, full-token, and raw-data controls. For a prior-independent formal guarantee, a differentially private selected-disclosure mechanism would be added on top: fixed or privately selected preprocessing, clipping, calibrated random noise, sanitised metadata, and cumulative privacy accounting. The deterministic mechanism cannot provide nontrivial differential privacy by itself.

Working on something related? We'd like to hear from you.

We're interested in conversations with data infrastructure engineers, ML researchers, and applied mathematicians working on problems that overlap with ours, whether that's lossless compression, efficient ML data pipelines, structured representations, trusted computation, or cryptographic verification of data properties.