Skip to main content

Exploiting the /proc filesystem

It has been quite a while since I have written my last article. After pondering what to write for a couple of days, I have decided to share a couple of /proc filesystem tricks. Here is a list of real-life scenarios that illustrate how you can exploit this filesystem:

Environment Variables

Let's start with a simple example. You have a Java process that is misbehaving and you would like to make sure that the JAVA_HOME environment variable for the process context is pointing to the correct JDK installed on the system. Just cat the /proc/[pid]/environ file to view the list of environment variables for the process.

File Descriptors

To get the list of file descriptors opened by a process run an ls command for /proc/[pid]/fd. This directory would list the file descriptors as symbolic links to files or sockets with inode numbers. This is a particularly useful trick in case you are dealing with a long strace output and you have a system call with an associated file descriptor. If the file descriptor is a socket, you can gather even more information by grepping for the inode number in /proc/[pid]/net/tcp, /proc/[pid]/net/udp, or /proc/[pid]/net/unix depending on the socket type.

Swap Space

You are dealing with a server that has started running a process consuming large amounts of memory and the server has swapped some data from memory. It is not big deal for Linux if swapping is not continuous; however, it is a problem from a monitoring perspective because Linux is not going to move that data back into memory. One trick to use the swapoff command to force the data back into memory. If there is not enough space on the other hand, you have to force drop the page caches from memory by running "echo 1 > /proc/sys/vm/drop_caches" and then do a swapoff and a subsequent swapon to free up the swap space.

Connection Tracking

If you ever run into "nf_conntrack: table full, dropping packet" errors... You are using iptables and the kernel is hitting a limit on the number of connections that can be tracked. This is controlled by the net.netfilter.nf_conntrack_max system variable which defaults to 65536. You can either try to increase this default value or cat the /proc/net/nf_conntrack file to figure out what type of connection constitutes the majority of tracked connections and update your iptables rules accordingly to not track them - see the raw table with the NOTRACK option in the iptables manual.

Child Threads

You have a MySQL instance running with a very large connection limit and you have a slave I/O thread that is misbehaving. The MySQL instance is lagging behind yet there seems to be no apparent problem. The next idea you come up with is to strace the I/O thread; however, the "show processlist" output is showing connection ids not Linux process ids. Luckily, there is a solution. If you have persistent database connections, stop the I/O thread and start it again. Then, run an "ls -al" in /proc/[pid]/task which will in turn show you a list of child pids with creation dates. Because you have restarted your I/O thread, the most recently created one should be the I/O thread. You can also verify this by comparing the task creation date to the "Time" column for the I/O thread in the "show processlist" output. If you don't have persistent connections, identifying the correct thread is a bit more tricky. Simply restart the I/O and SQL threads at different times, wait for a while, and then identify the correct thread by comparing the "Time" column from the "show processlist" output again.

Process Limits

Every process has to run within certain limits set by the kernel. One of these limits is the number of child processes that can be spawned by a parent process. For example, when we started running into MySQL connection errors, the underlying problem turned out to be the connection limit (each MySQL connection is served by a child thread) being more than what the kernel allowed for. These limits can be different for each process and can be viewed by catting the /proc/[pid]/limits file. And the best part? You can change the limits dynamically by echoing to this file instead of restarting that production MySQL instance.

Labels:

Comments

Post a Comment

Popular posts from this blog

Securing Symfony2 REST services with FOSOAuthServerBundle

Overview In my previous article, I wrote about setting up a Symfony2 REST service using FOSRestBundle. However, this REST service was behind a firewall protected by a generic form_login provider. Not really ideal if you wish to open your REST API to other applications. So in this article, I will try to explain how to set up FOSOAuthServerBundle to protect your REST API methods using OAuth2. Before we start getting into the gritty details, it is a good idea to have a look at the official OAuth2 documentation . Let's begin... FOSOAuthServerBundle Installation You have to install v1.1.0 of FOSOAuthServerBundle if you are using Symfony 2.0.x. If not, see the docs . First, add the following entries to your deps file: [FOSOAuthServerBundle] git=git://github.com/FriendsOfSymfony/FOSOAuthServerBundle.git target=bundles/FOS/OAuthServerBundle version=origin/1.1.x [oauth2-php] git=git://github.com/FriendsOfSymfony/oauth2-php.git Run the vendors script to install these...
87 comments
Read more

Unexpected token "name" of value "if" ("end of statement block" expected) in "WebProfilerBundle:Collector:logger.html.twig"

Encountered this WebProfilerBundle error message when I ran the bin/vendors script to update my Symfony2 bundles. Make sure your deps file is up to date; you need to pay special attention to your version values. In this case, update your twig version to v1.2.0 as illustrated below: [twig] git=http://github.com/fabpot/Twig.git version=v1.2.0 Run the vendors script to update your bundle and the error message should disappear. You can get the most up to date deps file from the symfony-standard repository located at: https://github.com/symfony/symfony-standard/blob/master/deps
1 comment
Read more

A Parcelable Tutorial for Android

Parcelable Interface Overview In one of my earlier posts, I mentioned writing an article about FOSOAuthBundle integration with an Android client. To keep that article to the point, I need to explain some concepts beforehand. One of the important concepts is the Android Parcelable interface that allows data to be transferred between different processes/threads. Certain network operations with Android such as authentication with OAuth2 and then fetching data from a REST endpoint should be performed in the background in order not to block the UI thread. This requires data to be fetched by a service (I have opted for Intent Services in my implementation) in the background and then passed back to the calling activity/fragment with a result callback. This is where the Parcelable interface comes into play. Basically, the Parcelable interface allows your classes to be flattened inside a message container called a Parcel to facilitate high performance inter process communication. The rece...
3 comments
Read more