When CSP Monitoring Met InfluxDB: A 7.5 Billion Series Disaster

How It Started

A while back, we wanted to build a front-end security monitoring system:

CSP (Content Security Policy) violation tracking
First-party JS integrity verification
Third-party JS hash checks
Risky script alerts

The product is a SaaS platform serving multiple countries. Combined monthly active users across all sites: close to 10 million.

The requirements sounded straightforward. Every time a user opens a page, check CSP compliance, verify JS hashes, and report anomalies. Then build trend analysis, risk statistics, real-time monitoring, and incident search on top.

We picked InfluxDB. The reasoning was, well...

"Time-series databases are fast."

Honestly, I wasn't very familiar with time-series databases at the time. But AI-assisted development moves so fast these days—the architecture isn't even figured out yet, and the system is already running.

The entire system was almost entirely written by Codex. Data structures, InfluxDB writes, dashboards, aggregation queries, front-end hash verification, query APIs—all AI-generated.

First version. Three days to production. Staging environment looked perfect.

Then we deployed.

Everything exploded.

v1: The Memory Aggregation Disaster

At first, I thought the queries were just badly written.

To be fair, the v1 logic was brutally simple: pull all raw data out of InfluxDB, then aggregate in Node.js memory.

Staging was completely fine. Small data volume, few users, low cardinality.

Production was a different planet.

About 20 minutes after deployment, alerts started firing:

Node memory spiked
CPU went through the roof
Dashboard froze
Queries kept getting slower

Turned out the bottleneck wasn't InfluxDB at all. It was me pulling every raw record into Node and crunching it in memory.

So we immediately started on v2.

v2: Still Broken

This time we made several changes:

Pushed aggregation logic into InfluxDB's native functions
Minimized Node-side computation
Optimized raw query patterns
Rewrote the query layer

Still broken.

And this time it was weirder—some queries took several minutes to return.

At this point I started wondering: is InfluxDB just not good enough?

Then it hit me. The problem wasn't InfluxDB. The problem was:

I was using a time-series database as a high-dimensional search engine.

What Time-Series Databases Actually Do

A common misconception when people first encounter InfluxDB, Prometheus, or OpenTSDB: "It's a database, so I should be able to query it like MySQL, right?"

No. What time-series databases actually solve is:

How does a metric change over time?

They excel at:

CPU usage trends
Error rate tracking
QPS fluctuations
Time-bucket aggregation
Metric-based alerting

That's Time Series Analysis.

They do not excel at:

Arbitrary multi-condition filtering
Multi-dimensional search
Pagination
Looking up individual records

When I re-examined my actual requirements, I realized what I really needed was:

Search by userId
Search by scriptUrl
Search by hash
Search by pageUrl
Multi-condition combination filters
Risk investigation
Pagination
Detailed record retrieval

This isn't "time-series analysis." This is a risk control system, a log search engine, an OLTP query system.

The Abyss: Series Cardinality

The real problem emerged when I finally understood what tag and field actually mean in InfluxDB.

At first I thought:

tag = indexed
field = regular column

Nope. That's not it at all.

InfluxDB tags aren't just "indexed fields"—they're part of the series identity. When data is written, InfluxDB has already determined the series structure.

This is a completely different universe from MySQL. In MySQL, no index today? Just run ALTER TABLE ADD INDEX tomorrow. InfluxDB doesn't work like that. A wrong tag design means backfill, rewrite, rebuild—basically rebuilding the entire time-series structure.

7.5 Billion Series

Then I started researching what series cardinality actually means.

InfluxDB combines measurement + tag set into a unique series.

For example:

measurement=csp_report
siteCode=us
env=prod
userId=123
scriptUrl=a.js
pageUrl=/checkout
hash=xxxx

This entire combination is one series.

Change the userId? New series. Change the hash? Another new series.

If you know InfluxDB, you're probably already laughing at seeing userId and hash as tags.

Then I did the math:

Dimension	Count
siteCode	30
userId	100,000
hash	500
scriptUrl	50
pageUrl	100

Theoretical series:

30 × 100,000 × 500 × 50 × 100 = 7,500,000,000

7.5 billion.

The first time I calculated this, I thought I'd made a mistake. I double-checked. Triple-checked. Nope. 7.5 billion series.

That's when it hit me: the scariest thing about InfluxDB isn't one huge dimension—it's multiple "reasonable-looking" dimensions combining into a combinatorial explosion.

The Real Cost: Metadata, Not Data

Here's the thing: we only had about 1.7 million rows per day.

Honestly, that's not even that much. MySQL with sharding, ClickHouse, Elasticsearch—they could all handle it.

But InfluxDB's most expensive resource was never data volume. It was series cardinality.

People often assume series ≈ data volume. It doesn't.

Series is more like index space. For every single series, InfluxDB:

Allocates a series ID
Builds an inverted index
Maintains metadata
Creates tag indexes

The real resource drain isn't the data itself—it's the series metadata.

This is why InfluxDB's official docs keep saying: don't put userId, hash, uuid, requestId, or random strings into tags. These fields will blow up your series cardinality.

Why Queries Got Progressively Slower

Under high cardinality, InfluxDB queries aren't just "scanning data." They're scanning massive amounts of series metadata.

Especially:

tag filters
GROUP BY operations
series expansion

These trigger:

Expensive index scans
Cross-shard traversal
Frequent mmap page faults

This isn't something you can fix by "optimizing queries." It's a fundamental architecture mismatch.

The Fix: InfluxDB + MySQL Dual Storage

We tore the whole system apart and rebuilt it.

InfluxDB stayed, but only for what it's good at:

Trend analysis
count / sum aggregation
TOP N rankings
Time-bucket aggregation
Alert trends

MySQL handled the search:

Detail queries
userId lookups
hash lookups
Multi-condition filtering
Risk investigation
Pagination

In other words:

InfluxDB for trends. MySQL for records.

The Final Tag / Field Design

The redesigned InfluxDB measurement only kept low-cardinality tags:

# Tags (low cardinality, part of series identity)
siteCode    → ~30 values
env         → prod / staging / dev
type        → csp_violation / js_integrity / third_party_hash / risk_script

# Fields (high cardinality, NOT part of series identity)
reportCount → integer (for aggregation)
riskLevel   → string (high / medium / low)

All high-cardinality fields (userId, hash, scriptUrl, pageUrl) go to MySQL only. InfluxDB only cares about "how many anomalies of type X happened today, and what's the trend."

On the write side, we do dual writes: each record goes to both InfluxDB (for aggregation) and MySQL (for detail queries). Write consistency is handled at the application layer with retries + MySQL transactions. Missing a few writes to InfluxDB doesn't significantly affect trend accuracy.

Here's what I finally understood: to decide whether a system fits a time-series database, don't ask "does the data have a timestamp?" Ask:

Are you looking at trends, or looking up records?

These are fundamentally different problems.

Looking Back

This whole project feels like a microcosm of software development in the AI era.

Codex helped me ship the entire system in 3 days. Staging passed instantly. Development velocity was insane.

But it didn't warn me about:

30 × 100,000 × 500 × 50 × 100 = 7,500,000,000

AI can accelerate development enormously. But systems still have to obey the laws of their underlying architecture.

Speed doesn't matter if you're heading in the wrong direction.