A while ago I published a short Twitter thread explaining a few of the problems I had run into while running their [enterprise] clustering solution; it’s been quite a painful process and I would like to write down the alternative architecture I ended up with while looking for improvements.

The original scenario

The original scenario I was working at was composed of a cluster of Go API servers backed by an InfluxDB database. Original service requirements demanded to deploy a fault-tolerant datastore so that’s the reason an InfluxDB cluster was put in place.

I like reading, a lot, but, sadly, I don’t read as much as I used to do a few years ago (for many different reasons). However, I keep growing the list of books I’d like to read, hoping I will be able to read them at some point in the future.

You’ll find a few different topics: databases, distributed systems, computer architecture, concurrency, or formal methods are some of them. Maybe you find something interesting to read.

Final entry on the InfluxDB storage subsystem series. This time I am going to focus on the, relatively new, indexing mechanism that the InfluxDB folks have built into their storage system.

If you just arrived here for the first time, you can find some more details about the other major components of the storage system in the previous blog posts of the series: intro and WAL component and tsm component.

The origins

If you recall from the previous posts, the TSM files have an index that allows the database to determine where a certain time series is located when a seriesKey and a time range are provided.

During this entry, we are going through the TSM part of the InfluxDB storage engine: how the contents are organized in the disk, how they are compressed or how they are compacted. This is the second entry of the series about the InfluxDB storage engine started in the previous post.

TSM files structure

The TSM files are where Influx stores the real data; these files are read-only files and are memory-mapped. If you’re familiar with any database using an LSM Tree variant this concept is very similar to the SSTable concept.

Writing on the blog about some of the technical stuff I usually enjoy and/or work with is something that’s always in my plans but I never find the right time do it. Due to all this quarantine related stuff, both my sleeping habits and the kid’s are being somehow affected and, well, here I am, staring at a blank page trying to start writing about databases. And, well, since I don’t longer work for an international company I thought it would be a good idea to somehow practice my poor English skills.