This year I’ve decided to start writing a quick summary of some of the papers and books I read. I don’t know how long this will last, but I am going to give it a try (I haven’t been reading that much lately anyway).

So, to get myself started I am going to use the first paper I have read this year: Are You Sure You Want to Use MMAP in Your Database Management System?

This year I have read/watched very few tech-related things outside of $job but I would like to share with all of you a few of the ones I’ve enjoyed the most.

As usual, the contents revolve around databases, systems engineering, cloud computing, runtimes, programming languages and low-level programming, so, if you’re not interested in any of these topics you can save some good time and stop reading here.

Tech Talks

How we build Firebolt

A really nice talk about how the Firebolt folks built their cloud analytical database; the talk covers different aspects of their architecture: how they used ClickHouse as their starting point, compute and storage separation, their metadata layer (based on FoundationDB) among a few other interesting topics.

Disclaimer: By no means, I am an expert on this topic; I would say I am a beginner on all this stuff. During my Ph.D. time (which I’ve never finished) I used some of the techniques/tools presented here trying to help me formally verify a few designs I was working on. A second attempt to take benefit of formal methods was when I tried to apply some of them during my time at one of the OSGi Alliance Experts Group, looking to formally validate the design of certain aspects in a couple of specs I was working on at that time (I never managed to make a formal proposal to the group, tho).

A while ago I published a short Twitter thread explaining a few of the problems I had run into while running their [enterprise] clustering solution; it’s been quite a painful process and I would like to write down the alternative architecture I ended up with while looking for improvements.

The original scenario

The original scenario I was working at was composed of a cluster of Go API servers backed by an InfluxDB database. Original service requirements demanded to deploy a fault-tolerant datastore so that’s the reason an InfluxDB cluster was put in place.

I like reading, a lot, but, sadly, I don’t read as much as I used to do a few years ago (for many different reasons). However, I keep growing the list of books I’d like to read, hoping I will be able to read them at some point in the future.

You’ll find a few different topics: databases, distributed systems, computer architecture, concurrency, or formal methods are some of them. Maybe you find something interesting to read.

Final entry on the InfluxDB storage subsystem series. This time I am going to focus on the, relatively new, indexing mechanism that the InfluxDB folks have built into their storage system.

If you just arrived here for the first time, you can find some more details about the other major components of the storage system in the previous blog posts of the series: intro and WAL component and tsm component.

The origins

If you recall from the previous posts, the TSM files have an index that allows the database to determine where a certain time series is located when a seriesKey and a time range are provided.

During this entry, we are going through the TSM part of the InfluxDB storage engine: how the contents are organized in the disk, how they are compressed or how they are compacted. This is the second entry of the series about the InfluxDB storage engine started in the previous post.

TSM files structure

The TSM files are where Influx stores the real data; these files are read-only files and are memory-mapped. If you’re familiar with any database using an LSM Tree variant this concept is very similar to the SSTable concept.

Writing on the blog about some of the technical stuff I usually enjoy and/or work with is something that’s always in my plans but I never find the right time do it. Due to all this quarantine related stuff, both my sleeping habits and the kid’s are being somehow affected and, well, here I am, staring at a blank page trying to start writing about databases. And, well, since I don’t longer work for an international company I thought it would be a good idea to somehow practice my poor English skills.

I would like to share with you a bunch of the tech talks I have recently watched or some of the latest books I have enjoyed.

Disclaimer: this list is just a set of resources I have gone through more or less “recently” about a few different topics I use to enjoy. It’s not my goal to create a “Best Talks/Papers/Whatever List”, just wanted to share with you all some of the things I have found interesting lately (all the resources are listed in no specific order).