Implement a ring buffer

sql, sql-server, ssms, t-sql

Solution

Unless there is something magic about 1 billion, I think you should consider other approaches.

The first that comes to mind is partitioning the data. Say, put one hour's worth of data into each partition. This will result in about 15,000*60*60 = 54 million records in a partition. About every 20 hours, you can remove a partition.

One big advantage of partitioning is that the insert performance should work well and you don't have to delete individual records. There can be additional overheads depending on the query load, indexes, and other factors. But, with no additional indexes and a query load that is primarily inserts, it should solve your problem better than trying to delete 15,000 records each second along with the inserts.

Problem

We have a table logging data. It is logging at say 15K rows per second. Question: How would we limit the table size to the 1bn newest rows? i.e. once 1bn rows is reached, it becomes a ring buffer, deleting the oldest row when adding the newest. Triggers might load the system too much. Here's a trigger example on SO. We are already using a bunch of tweaks to keep the speed up (such as stored procedures, Table Parameters etc). Edit (8 years on) : My recent question/answer here addresses a similar issue using a time series database.

Original source

Related problems