Polygen
Polygen: the tokenized data layer.
A lightweight SDK that runs in your cloud, on your servers, or at the edge. It encodes any data into a structured lossless token format. One primitive handles tabular, time-series, images, video, audio, sensor, embeddings, and graphs. The same tokens drive your storage layer, your transmission layer, and your analytics and AI workloads. Raw data stays where it lives. The original is always exactly recoverable.
x = B · C + r
Per window, per column: fit a compact model, store the model and the exact leftover. Decode is bit-exact, or skip the leftover and read a coefficient-only approximation from the same token.
C: coefficients
A handful of numbers describing the window's shape. Fixed-length, so they behave like an embedding: index them, search them, train on them.
r: residual
The exact error the model missed. Its size is a free anomaly signal. Withhold it and the recipient cannot reconstruct exactly without your release.
Per-segment digest
A cryptographic fingerprint per window. Tamper-evident, verifiable across machines, and it says which window changed, not just that something did.
Vision-language benchmarks
Polygen compresses the visual tokens a vision-language model attends over, and a small LoRA adapter trained on 300 ScienceQA images teaches the model to read the compressed sequence. The results below are end to end on Qwen2-VL-7B-Instruct in fp16 on an NVIDIA H200, against an un-spliced, un-LoRA baseline, ScienceQA n=200.
Operating tier
ScienceQA accuracy
Attention FLOPs
Storage
T=4 (lossless)
+6.9pp
0% (length-preserving)
3.9x smaller
T=3
+8.4pp
41% fewer
5.1x smaller
T=2
+7.4pp
~71% fewer
7.5x smaller
T=1
+3.4pp
~91% fewer
14.3x smaller
T=0
−2.1pp
~99% fewer
283x smaller
Transfer to benchmarks the adapter never saw
Benchmark
Backbone
Sample
Accuracy delta
ScienceQA
Qwen2-VL-2B
n=2,017
+3.08pp at T=4
MMMU
Qwen2-VL-2B
n=802
+1.57pp
VQAv2
Qwen2-VL-7B
n=2,000
+2.50pp
VQAv2
Qwen2-VL-2B
n=2,000
+0.15pp (within 95% CI)
DocVQA
Qwen2-VL-2B, hook path
n=500
+0.21pp (within 95% CI)
Compute is reported as attention-FLOP reduction, a count that follows from the shorter visual sequence, not wall-clock; single-shot wall-clock savings are not claimed. The adapter was trained on ScienceQA only, so every other row is transfer without retraining. T=3 is the accuracy headline, T=2 the compression headline, and the two are a statistical tie. Adapters, method, and the live side-by-side demo are published on Hugging Face under a Datasent evaluation licence.
Demos and evaluation
Two public demos run on Hugging Face Spaces against a private Datasent endpoint; nothing proprietary runs in the browser. The evaluation environment runs the SDK on a measured host and is issued per engagement.
Polygen for Qwen2-VL
Upload an image, ask a question, and compare the un-spliced Qwen2-VL baseline with the polygen plus LoRA path side by side. Same answer, fewer visual tokens, shown per image. First request after a quiet period can take up to a minute while the endpoint wakes.
Polygen RAG: encode once, query many
An 8-page visual corpus encoded once and queried many times. Stored as polygen descriptors the index is 5.5x smaller than cached fp16 embeddings (10.4 MB against 56.7 MB, measured over the corpus), and each query's answer is checked against the full-page result.
Evaluation environment
poly.datasent.com runs the SDK on your own change streams and telemetry files; sdk-api.datasent.com exposes the same encoder as an HTTP API. Every number shown there is a measurement on that host or a labelled projection from one. Access is issued per evaluation.
For developers
Polygen runs as a hosted API during evaluation and ships as an SDK or container afterward. The API reference is public and needs no key to read; calls need a license key issued per evaluation.
API reference
OpenAPI 3.1 reference for the hosted API: health, encode, reconstruct and synthesize, each with a file-upload variant, with request and response schemas and a downloadable spec.
Models and demos
Qwen2-VL LoRA adapters trained on Polygen tokens, at 2B and 7B, with model cards and the per-tier benchmark tables, plus the running VLM and multimodal RAG demos.
Evaluation keys
License keys are issued per evaluation and cover the hosted API and the poly.datasent.com evaluation environment. Tell us the modality and the question you want answered.
Approach
How the encoding layer works
Encode
Agree on the model.
Sender and receiver establish a shared basis upfront. Data is encoded against this basis, capturing its structure while isolating what cannot be predicted.
Transmit
Only the residual moves
Raw data stays in place. Only the residual, which is the unpredictable part, and minimal metadata are transmitted across systems.
Reconstruct
Exact reconstruction
The receiver regenerates the basis locally and reconstructs the exact original when authorised. No exposure during transfer.
Advantage
A simpler alternative to moving data
Most systems assume data has to move, then add layers to protect it via encryption, duplication, and controlled environments. Each layer adds cost, latency, and operational overhead.
Classical compression
Reduces storage but produces opaque byte streams. Any computation requires full decompression. Raw data still traverses the network. No structural information preserved.
Columnar storage formats
Preserve schema but apply fixed heuristics, run-length, and delta encoding without adaptive model selection. Raw data moves in full. No raw-data-local guarantee. No governed reconstruction.
Why Datasent is different
Lossless and structurally explicit. Raw data stays local architecturally, not as a policy. Governed reconstruction. Works across every major data type. One encoding layer for storage, transmission, and computation.
One encoder. Six places the cost falls.
Storage
Lossless archive plus a coefficient-only analytics tier from the same encode, 100x to 900x smaller on the public cohorts we have measured.
Memory
Feature matrices shrink about 78x, so training fits where a cluster was needed before. Measured on TCGA RNA-seq.
Compute
Same libraries, smaller matrices: PCA and similarity search run 16x to 32x faster on tokens, measured on the 1000 Genomes cohort.
Egress
Ship coefficients, not archives: analytics-grade delivery over a thin or contested link, with raw records staying at the source.
Governance
Per-window integrity digests, tamper localisation, and custody splits between data owner and recipient.
ML fidelity
Exact mode matches raw by construction. A diagnostic reports whether coefficient-only features still predict your target before you commit; on TCGA classification they reached 97.5% against 100% for raw.
Industries
Real-world impact across industries
Transportation & Infrastructure
Lossless telemetry without the bandwidth cost
High-volume sensor streams from connected infrastructure, traffic systems, fleet telemetry, and environmental monitors are encoded and transmitted as residuals. Raw data stays on-site. Insights reach the cloud exactly.
.avif)
Financial Services
Governed data sharing across regulated boundaries
Time-series, transactional, and behavioural data shared as token exchanges and not raw transfers. The custodian model gates reconstruction behind explicit authorisation and can log every release.
Healthcare & Life Sciences
Analytics on sensitive data without moving it
Organisations can analyse, share, and build on data without transferring raw records. With a shared model basis and residual-only transmission, data stays in place while remaining exactly recoverable when required.
AI & ML Infrastructure
Training data that's already in the right format
Token components map directly to model input features, not a separate preprocessing pass. Federated training across organisations exchanges tokenised residuals, not raw datasets.
Transportation & Infrastructure
Roadside cameras to cloud AI, without moving the footage
A state department of transportation runs a real-time camera-to-AI pipeline in which roadside units encode video into compact event tokens at the edge. Only tokens cross the network to the analytics platform, end to end in under two seconds.
Financial Services
Collections propensity from consumer history, not static profiles
On one receivables provider's book (7,926 accounts with 12 months of exposure), a model built only from consumer history available at listing time reached AUC 0.781 against the incumbent score's 0.688. Working the top half of its queue reached 83.7% of all payers, against 71.0% on the incumbent queue.
Defense
FMV and telemetry over a contested link
On a thin tactical link the downlink is the binding constraint. The video stream is already near-optimal under H.264 or H.265, so Polygen leaves it alone and tokenizes the telemetry and per-frame signals beside it: about 200 bytes per minute of telemetry, 4 bytes per frame of event signal, and analytics on tokens with no decode.
Media & Satellite
Video archives and broadcast metadata, beside the codec
Polygen is not a video codec. For video and satellite distribution it tokenizes the structural signals H.264 and HEVC leave alone: archival metadata derived from the codec residual, per-segment content provenance, and the metadata channels that ride beside broadcast video.
AI & ML Infrastructure
Vision-language models and multimodal RAG on compressed visual tokens
Polygen compresses the visual tokens a vision-language model attends over. On Qwen2-VL-7B the recommended setting delivers +8.4 points of ScienceQA accuracy over the un-spliced baseline with 41% fewer attention FLOPs, and a visual corpus encoded once is 5.5x smaller to store and answers the same questions.
Technical depth
How Datasent compares
System
Lossless
Raw data stays local
Governed reconstruction
Multi-modal
Classical compression (Huffman, LZ)
Lossy transform coding (JPEG, DCT)
Columnar storage (Parquet, ORC)
Federated learning frameworks
Secure clean rooms
Datasent
Datasent is the only system that is simultaneously lossless, keeps raw data local, supports governed reconstruction, and works across every major data type. A full technical treatment is available in the white paper.
Supported data types
Every type of data your organisation produces
Tabular
Rows and columns, integer or fixed-point. The natural starting point for most data teams.
Time-series
Sequential numeric measurements over time. Sensors, metrics, telemetry, financial data.
Images
Encoded per channel. Works with any standard image format representable as pixel values.
Video
Decomposed into keyframes, motion, and residual streams, each encoded independently and composed losslessly.
Audio
PCM audio encoded as a multi-channel time-series matrix. Preserves full waveform fidelity.
Sensor and scientific data
High-volume numeric streams from IoT devices, instruments, and measurement systems.
Embeddings
Fixed-point quantised vector matrices. Lossless encoding of ML embedding spaces.
Graphs
Node features, edge lists, and adjacency matrices encoded as independent token streams.
Text
Token ID or byte sequences encoded as integer arrays.
Integration
Fits into your existing stack. Doesn’t replace it
Datasent is an encoding layer, not a database or compute platform. It sits between your raw data and the systems that store, move, and process it, adding lossless compression, trusted transmission, and governed reconstruction without requiring you to replace anything downstream.
Storage systems
Datasent-encoded files drop into any object store, data lake, or file system. S3, GCS, Azure Blob, HDFS, local disk: the format is storage-agnostic. Your existing infrastructure handles the bytes; Datasent handles what those bytes contain.
Data pipelines
Encode at ingestion and stay encoded through your pipeline. Datasent integrates at the data loader level, so your pipeline logic, orchestration, and downstream systems are unchanged. No new middleware required.
ML frameworks
Datasent-encoded datasets load directly into training pipelines. Token components map to model input features without a separate preprocessing step. Compatible with PyTorch, TensorFlow, JAX, and any framework that reads from standard data loaders.
Edge and IoT
Establish the trusted setup between the edge device and the cloud endpoint once. From that point, the device encodes and transmits only the residual. The cloud endpoint regenerates the basis locally and reconstructs exactly. Raw sensor data never crosses the network.
Cross-organisation data sharing
The custodian model allows data to be shared as governed token exchanges. One party holds residuals, the custodian holds coefficient shares. Reconstruction requires explicit authorisation from the custodian and is logged.
Start hosted, finish embedded, without redoing the integration.
Hosted API
Dedicated single-tenant instance, TLS, key-gated. Nothing installs on your systems; HTTPS egress is the only dependency. The evaluation and pilot path.
SDK or container
Compiled wheel or the same API container, deployed inside your environment, up to accredited enclaves. The production and OEM path.
The turnover contract
Every API field maps by name onto an SDK argument. Moving from hosted to embedded changes a base URL, not your adapter. Pilot integration work carries forward.
Security & Privacy Principles
Three guarantees, built into the encoding
Datasent is designed around three guarantees: raw data never leaves its environment, the original is always exactly recoverable, and reconstruction happens only under explicit control.
Join our newsletter
Get updates on research, product news, and privacy-first data insights.
