Post-Mortem: Outbid's Viral Launch broke Vemetric's Ingestion

On August 19, 2026, Jonathan Wilke launched Outbid, a very simple website where products compete for the top position on a public leaderboard by outbidding each other. Outbid quickly took off on X and Hacker News and reached a ton of attention, including a huge traffic spike.

---

Outbid used Vemetric for its analytics. Less than a day after the launch, Vemetric was receiving more traffic than it had ever handled before. It included so much traffic, that Redis filled up with jobs faster than the workers could process them. No new data was visible on the Dashboard + API anymore, and for a timeframe of about 30 minutes, some analytics data was lost.

It received a lot of attention from Hacker News and X, and many people discovered Vemetric because Jonathan openly shared its public analytics dashboard. Investigation showed that also a good amount of unusual automated traffic was involved, which made the situation for Vemetric’s ingestion system even worse.

But even with bot traffic included, it’s clear is that Vemetric should have handled it. Vemetric should be able to handle such a viral launch and also a huge traffic spike for one project shouldn’t affect the entire platform.

This post explains what happened, why the system failed, how I recovered it, and what I’m changing so that Vemetric can handle such situations in the future.

I want to clearly state that I’m very sorry this happened. Outbid brought Vemetric far more traffic and attention than usual, but serving that traffic was my responsibility. Vemetric failed Jonathan exactly when the project was taking off and that’s a big fumble on my side.

What Happened?

Between 18:00 and 22:00 UTC, Vemetric accepted 1,016,237 events associated with Outbid’s project. This included genuine analytics from the viral launch as well as automated and fabricated activity. The traffic peaked at about 2,5k accepted events per second.

As how Vemetric’s ingestion works at the moment, every accepted event created at least three separate BullMQ jobs:

1. Store the event
2. Create or update the device
3. Create or update the session

The accepted events therefore created more than 3 million core queue jobs, not including retries and additional jobs.

The workers couldn’t handle the volume fast enough anymore. Incoming traffic was therefore much faster than every part of the processing pipeline and Vemetric failed to reliably serve the latest data.

This also caused Redis memory usage and queue sizes to grow quickly. Since the ingestion server and Redis were running on the same server, they also competed for the same CPU and memory. The result was a restart loop which made the situation even worse.

Timeline

All times below are in UTC.

• August 19, 21:08: Jonathan publishes the initial Outbid launch post on X. The project starts spreading through the indie-hacker community.
• August 20, before 18:00: Outbid gains significant genuine traffic from X and Hacker News. It also brings a lot of new attention to Vemetric.
• 19:26: The first custom events which Outbid never implemented appear in the stored data, signaling some DDOS attack behavior.
• 20:02: Accepted traffic reaches its peak of 2,712 events in one second.
• Around 21:30: The ingestion and worker system is no longer making reliable progress. Redis contains a large backlog and the server is restarting repeatedly.
• 21:45: The emergency Cloudflare block stops new ingestion for the affected project.
• Around 22:05: Workers begin processing jobs again.
• Around 03:35: The event queue has caught up after running at its maximum processing rate for several hours.
• During the following hours: The device and session queues still contain hundreds of thousands of jobs. Jobs belonging to Outbid are skipped so that other Vemetric customers can return to normal processing.

Why Vemetric Failed

The combination of viral traffic and automated telemetry created the ingestion spike, but that alone is not a sufficient explanation for the incident. Vemetric should have contained the load associated with one project without affecting other customers.

Below are the main reasons why Vemetric failed to handle the traffic spike.

One Event Created at Least Three Jobs

For every event accepted by the hub, Vemetric adds an event job, a device job, and a session job.

This architecture works at normal traffic levels, but it amplifies sudden traffic spikes. During this incident, around one million accepted events turned into more than three million queue jobs.

Much of the incoming telemetry used almost unique identities for every event. This meant that the device and session jobs could not be meaningfully combined or reused.

Device and Session Processing Was Effectively Serial

The device and session queues both have a global concurrency of one. Adding more worker instances would therefore not have increased their throughput.

This was intended to avoid conflicting updates, but it created a huge bottleneck. At the peak of the spike, new jobs arrived much faster than the worker could process them.

Redis and the Hub Share One Server

Redis and the ingestion server are running on the same server. As the queues grew, Redis consumed more memory while the hub needed more CPU and memory to accept new requests.

When the server restarted, ingestion stopped making progress. Incoming requests and retries then made it harder for the system to recover.

Upgrading the server from 4 GB to 8 GB of RAM helped it boot and continue processing, but more memory alone would not have solved the underlying problem. The queues would eventually have filled the additional memory as well.

Insufficient Bot / DDoS Detection

The existing bot detection mainly checked known crawler user agents and Cloudflare’s verified bot signal.

The automated telemetry used normal-looking Chrome, Safari, and mobile user agents. It therefore passed these checks even though the overall pattern was inconsistent with ordinary visitor behavior.

Impact

The incident affected more than the project associated with the traffic.

• Vemetric’s dashboard was mostly available, but couldn’t serve the latest data
• New events were delayed and some data was lost
• Redis contained hundreds of thousands of pending device and session jobs
• The hub server ran out of resources and entered a restart loop, which made the situation worse
• Other customers’ events had to wait behind the contaminated backlog associated with Outbid’s project
• Recovery continued for several hours after the incoming traffic stopped

The ClickHouse DB itself remained healthy. Its inserts completed without recorded errors and its flush latency stayed relatively stable. The bottleneck was the ingestion and queue architecture in front of it.

How I Recovered the System

The first priority was to stop the queues from growing.

I was already in contact with Jonathan, who was very helpful throughout the incident and after it was clear that Vemetric couldn’t handle the load in it’s current form, he quickly removed the Vemetric tracking script from Outbid. I also added a Cloudflare rule which blocked ingestion requests for the Outbid project token.

After increasing the server memory, Redis and the workers were able to run reliably enough to process the existing event queue. The event queue was empty again after roughly six hours, and the delayed events appeared correctly in the Vemetric dashboard.

The device and session queues were a different problem. They still contained between 600,000 and 800,000 jobs and would have taken days to process with their existing throughput.

Since restoring normal service for other customers was more important than enriching Outbid’s data, I deployed a temporary project exclusion for the device and session workers.

What Went Well

Even though the incident was serious, a few parts worked as intended:

• ClickHouse remained stable and accepted the events which reached it
• The event backlog could be processed after incoming traffic was blocked
• Most of the delayed analytics data was recovered
• Outbid could be isolated from the remaining device and session queues
• Jonathan was responsive, patient, and helped stop the incoming traffic
• Cloudflare made it possible to apply an emergency block without waiting for an application deployment

What Did Not Go Well

The incident also exposed several weaknesses:

• One project was able to consume resources needed by every customer
• Vemetric had no hard limit before expensive queue work was created
• Workers could not scale horizontally, the ingestion failed to make progress
• Redis and the ingestion server shared the same failure domain
• Bot detection was not sufficient enough
• There was no simple emergency switch to pause one project

What I Am Changing

The most important goal is to make sure that the ingestion system of Vemetric is able to handle a lot more traffic, that it can be scaled horizontally if needed, and that fabricated traffic is easier to reject and cannot take down Vemetric for everyone else.

Per-project workers

The ability to make specific workers only process jobs for a specific cluster of projects allows Vemetric to scale horizontally and isolate projects with traffic spikes from the rest of the system.

This is more of a short-term fix, because in the future, workers should be able to scale horizontally for the whole platform and not just for a specific project. But it is a good first step to make sure that one project cannot take down the entire system.

Ingestion and Queue Improvements

The current three-jobs-per-event model needs to change.

I plan to move towards one durable raw ingestion stream. Events can then be stored in batches, while device and session data is derived asynchronously and combined by identity.

The main improvements will be:

• Handle event jobs in batches
• Combine repeated device and session updates
• Group work by project or identity, so workers can safely run in parallel
• Drop or sample low-value enrichment work before raw event ingestion is affected

Infrastructure and Observability

The Hub, Redis, and Workers should not fail together.

I will separate the Hub (our ingestion service) from Redis, make it horizontally scalable, and also add the ability to scale workers horizontally.

I’m still very happy with the basic decisions I made that led to the current architecture, but I need to do some improvements to make it handle more traffic reliably.

Final Words

I have to admit, this incident completely caught me off guard. 😅

It happend while I was on a vacation with my family, when suddenly I got the notifications that parts of Vemetric are down and not working properly. This shouldn’t be an excuse, I want Vemetric to be able to work when I’m not immediately available as well, but it was a really bad moment (like it usually is for such events).

Seeing Vemetric unavailable while queues kept growing was not fun, especially because Jonathan trusted Vemetric with Outbid at the exact moment his project went viral and other customers were affected as well.

On the other hand, Outbid also gave Vemetric a lot of visibility. Many people discovered Vemetric because Jonathan chose it for the project and openly showed its dashboard. I’m genuinely grateful for that, which makes it even more frustrating that I could not provide the service he should have received in return.

At the same time, the incident provided a very clear stress test. I now have measured limits for the current system, a much better understanding of the traffic, and a concrete list of improvements.

I want to be transparent about incidents like this, especially because Vemetric already has a few customers who trust it with their analytics infrastructure. I want to take responsibility for this, I want Vemetric to be a reliable platform, and I always try to learn from such incidents so that they don’t happen again.

I’m sorry I fumbled this one, Jonathan. Thank you for being so helpful while everything was on fire. I will work hard to make Vemetric much more resilient because of it. 🫡

If you have any kind of feedback to share, I’m always happy to hear it. You can follow the development of Vemetric on GitHub or my X profile.