Storing time-series data from devices without a large cloud bill

Device data piles up fast. One sensor reporting every second produces 86,400 rows a day, about 31.5 million a year. Multiply by a few sensors and a few hundred devices, and a database that felt free during the pilot starts to dominate the cloud bill. Most of that growth is data nobody will ever look at again at full resolution.

Sampling rate is not reporting rate

The device can sample a temperature every second and still report once a minute. Sample fast to catch what you need, then reduce on the device: send the average, minimum and maximum for the interval, plus an event immediately when a limit is crossed. A minute of one-second data becomes one row, and the spike is still visible through the maximum.

Reducing at the source also cuts cellular data, battery use and broker load, which are often bigger costs than storage.

Downsample as data ages

Use the same data at different resolutions:

• Recent days at full reporting resolution, for troubleshooting.
• Weeks to months rolled up to, say, 5 or 15 minute buckets with min, max, mean and count.
• Years rolled up to hourly or daily values, for trends and reports.

Time-series databases such as TimescaleDB, InfluxDB and cloud equivalents do this with continuous aggregates or retention policies. Store the count along with the mean, so later aggregation stays correct.

Retention tiers

Hot storage is fast and costs the most, and holds the recent window dashboards query. Warm storage keeps the rollups. Cold storage, such as compressed Parquet or CSV files in object storage, holds raw history you must keep for audit or later analysis, at a small fraction of database cost. Decide the retention period per data type with the customer. Energy billing data may need years. A vibration waveform may need days.

Compression

Time-series columns compress well because values change slowly and timestamps are regular. Delta-of-delta encoding for timestamps and XOR-style encoding for floats are used by several time-series engines, and ratios of ten times or better are commonly reported on regular data. Columnar formats such as Parquet with zstd give similar savings in cold storage. When the sensor resolution is known, store integers with a scale factor.

Cardinality and queries

The hidden cost is often cardinality: high-cardinality tags, such as a unique ID in every label, bloat indexes. Keep tag sets small and stable. Also check what dashboards actually query, since a chart that scans a year of raw data on every refresh costs more than the data itself.

What this does not fix

Downsampling is one-way. If you roll up and delete raw data, you cannot later recover a fault pattern that was visible only at one-second resolution. Keep a short raw window, and on-device triggers that capture high-rate data around events, before you decide on retention.

---

If your device data is getting expensive to keep, book a discovery call, and see our cloud, apps and dashboards work.