Skip to main content

Posts

Postgres Index Cost-Benefit Analysis Using the Cumulative Statistics System

Index Overhead Postgres MVCC mechanism creates a new tuple version when a row is updated. This means that tuple updates, barring hot updates, have an amplifying effect on write load in addition to inserts and deletes, because all indexes that contain a pointer to the old tuple version must also get updated. Updates impose another insidious cost on query performance: index-only scans lose efficiency. If a table’s visibility map is stale due to heavy update volume, then index-only scans must fetch heap pages to confirm tuple visibility, nullifying the benefits of the index-only scan. Then, there is WAL logging. To support physical replication, index changes have to be individually tracked in the WAL file along with the heap change if WAL logging is enabled. This is another cost overhead that requires careful consideration of index use. Consequently, a single tuple operation has the potential to turn into multiple write operations depending on just table indexing choices, everything else ...
Post a Comment
Read more
Recent posts

A Configuration-Driven Test Automation Tool for Linux: PyMergen

Having dealt with system designs and issues throughout my career, I always felt the need to quickly create and perform benchmark tests on Linux systems. I have finally released a Python tool to accomplish that task. PyMergen is a configuration-driven test automation tool for Linux. It is intended to be a local command executor to automate repetitive and iterative testing actions on a Linux host to perform benchmarking and metric collection. The default collection logic is built around Linux Control Groups (cgroups) and Perf tools, and the functionality is extensible through a plugin mechanism. GitHub repository is here .  
Post a Comment
Read more

A Systems Engineering Attempt to Analyze the Linux Block Layer with BPF Compiler Collection (BCC) Toolkit and Static Tracepoints

Originally posted on medium on Jan 16th, 2023. Systems Engineering NASA defines Systems Engineering as follows : At NASA, “systems engineering” is defined as a methodical, multi-disciplinary approach for the design, realization, technical management, operations, and retirement of a system. A “system” is the combination of elements that function together to produce the capability required to meet a need. The elements include all hardware, software, equipment, facilities, personnel, processes, and procedures needed for this purpose; that is, all things required to produce system-level results. The results include system-level qualities, properties, characteristics, functions, behavior, and performance. The value added by the system as a whole, beyond that contributed independently by the parts, is primarily created by the relationship among the parts; that is, how they are interconnected. It is a way of looking at the “big picture” when making technical decisions. It is a way of achievi...
Post a Comment
Read more

Home Monitoring with a Raspberry Pi 2 Model B and ZoneMinder

Overview I had been thinking about setting up a camera system in my place for a while. My plan was to set up two cameras with motion detection; one to monitor the patio for deliveries and another to monitor the living room in case there was a break in. This plan stayed in the back burner for a long time until my colleague's house got burglarized just a couple of weeks ago. He was in a different state at the time; however, as soon as he received an alarm, the police department was called and one of the intruders got caught right after he left the house. Suddenly, I had a good reason to implement my own system. My design requirements could be described as simplicity. I just needed two cameras with wireless connectivity to minimize cable clutter. The first camera would be monitoring my living room and needed to be PTZ capable. My idea was to disable monitoring while I was at home and I needed a camera with this capability to provide a visual indication. During t...
6 comments
Read more

Automate SSH Login, TMUX Session Creation, and Window Setup

Modern Time Technology That was it... I was finally fed up with amount of time I was spending to SSH into each host manually. I was repeating the same procedure probably hundreds of times every day... Typing the ssh command and the host name, waiting for the initial prompt, entering my password, typing the sudo -i command to get root access, and entering my password again. Just as expected from a modern time worker. But wait, there was more to deal with... Working with a lot of servers, I need to use my favorite terminal emulator to organize and manage the number of sessions. This means that every window has to be properly named and assigned to a specific session. So before logging into each server, I had to switch to the correct TMUX session (or create a new one), create a new TMUX window, name the window properly, and finally move on to the SSH stage. I was aware of a script that was created by an ex-colleague. The script automated the process by storing the credentials in a ...
Post a Comment
Read more

Exploiting the /proc filesystem

It has been quite a while since I have written my last article. After pondering what to write for a couple of days, I have decided to share a couple of /proc filesystem tricks. Here is a list of real-life scenarios that illustrate how you can exploit this filesystem: Environment Variables Let's start with a simple example. You have a Java process that is misbehaving and you would like to make sure that the JAVA_HOME environment variable for the process context is pointing to the correct JDK installed on the system. Just cat the /proc/[pid]/environ file to view the list of environment variables for the process. File Descriptors To get the list of file descriptors opened by a process run an ls command for /proc/[pid]/fd. This directory would list the file descriptors as symbolic links to files or sockets with inode numbers. This is a particularly useful trick in case you are dealing with a long strace output and you have a system call with an associated file descriptor. If t...
Post a Comment
Read more

Using (R)?ex for Fame and Fortune

Overview (R)?ex , according to its website, is a tool written in Perl for configuration management and software deployment. The only requirement to have is an SSH connection to the remote server(s). Alternative Uses (R)?ex is a very capable tool to automate system administration tasks. For example, I have recently built an RPM build script that copies my local code/files to a build server, builds an RPM package, and downloads the finished RPM to my RPM repository on my local box. I have also used it to generate reports. One recent report was required to see how many of our servers were able to access a certain DNS view. In fact, the use cases are pretty much unlimited. You can use (R)?ex for any process that requires some sort of automation on a remote server. That said, as powerful as it is, (R)?ex still requires some initial setup to become usable. I have included a couple of my tips below. Tips and Tricks Code Structure Rexify.org Wiki recommends using revision control s...
Post a Comment
Read more