JavaScript Required

We're sorry, but we doesn't work properly without JavaScript enabled.

System Focussed Programming in Python

Introduction:

What is meant by “System Focussed Programming in Python”? Well, it means many things, depending on the perspective from which you look at it. For instance, for a system administrator, it is a means to monitor, debug, rectify, shutdown and start services in a system. The list for a sys admin actually may be longer than what I specified, but I just wanted to put it in a nutshell (and as you may have already realized, nutshells are not really large). From the perspective of a programmer (I call myself one, but I guess it is open to debate), it means making programs that take advantage of system resources, optimize processing power, etc. From a DB administrator's point of view, it is how to make the database she/he is administering to function in an optimized way, again, managing system resources wisely (this is a relative term – what is 'wisely' for you may not be very 'wise' for someone else). The list goes on and on, and I really can't do justice to all of them here. Hence, I have decided to address only a couple of perspectives – the perspective of a system administrator and the perspective of a programmer.

system-focussed-programming-in-python

The Commands:

Python allows you to do system programming in two different ways. Firstly, you have the two python workhorse modules – the 'os' and 'sys' (both of them have their legitimate own child modules which offer more functionality), and secondly, you can run shell commands from within python code, either using the ‘os.system’ or create a subprocess (using the 'subprocess' module). Apart from the above 2 methods, you may also use the information from the 'proc' file system (this is available only on Unix and Linux systems), may be by opening a desired file in the proc file system that contains the information you need. Last, but not the least, there is the python library named “platform” which provides valuable insights about the functioning of the system.

Here, I will first go through a list of functions/modules available from the 'os' , 'platform' and the 'sys' modules, and I will try to explain in brief what they do and what you should expect from them.

First, let us try to find a brief description of the OS you are working on. On the command line, you may run “uname -a”, which will provide you with an out similar to the one below:

Linux testyard.in 3.16.0-77-generic #99~14.04.1-Ubuntu SMP Tue Jun 28 19:17:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

The server name “testyard.in” in the above description will be different for various machines, but you will get much of the same output as above. In order to do the same with python, you may use os.system (“uname -a”). However, the “platform” module allows us to get the specific info that you need, one at a time. So, in case you wish to know the architecture of your system, you may use the following python code: (Please note that the commands are in both italics and bold, whereas the outputs are in simple italic font face.)

platform.machine()

The output of the above mentioned command will provide you with something similar to the result mentioned below:

'x86_64'

Similarly, you may run the following commands to get a whole view of the system you are working on:

platform.version()

'#99~14.04.1-Ubuntu SMP Tue Jun 28 19:17:10 UTC 2016'

platform.platform()

'Linux-3.16.0-77-generic-x86_64-with-Ubuntu-14.04-trusty'

Again, please note that your output may have different information, as it is based on the system you are using.

The next command is a replacement for the “uname -a” shell command we discussed earlier, but it provides you with the same info in pure python form.

platform.uname()

('Linux', 'testyard.in', '3.16.0-77-generic', '#99~14.04.1-Ubuntu SMP Tue Jun 28 19:17:10 UTC 2016', 'x86_64', 'x86_64')

So, in the above output, you have every bit of information that you derived from the “uname -a” shell command, but the interesting difference here is the fact that every info in the output of “platform.uname” comes as elements in a tuple. This seems to be very convenient for the programmer as she may not need all the information provided by the output. In the case of a tuple, the programmer can specific elements from the tuple and use it as required.

Now, every now and then, you may want to know the processor you are using, and the “platform.processor ()” is just the method that can provide you with that information:

platform.processor()

For me, the output is a bit disappointing, as it just displays the following:

'x86_64'

The output is definitely accurate, but it does miss a few details, and later on we will see how to extract that information using a shell command from within the “os.system” method.

In case if you are interested in the release information or the architecture, you may use the following 2 commands. The outputs from my system are provided with the commands.

platform.release()

'3.16.0-77-generic'

platform.architecture()

('64bit', 'ELF')

There are quite a few other cool things that you can do with the “platform” module, like finding about your platform you are using, the python branch, build, compiler, revision, version, etc you are using. I will leave that to you as an exercise.

Earlier, we said that the modules “os” and “sys” are workhorses, but so far we haven't got into it. Let us dive into it now and see what we can (and cannot) do with them.

The “os” module allows us to do a lot of file and directory manipulations. You can use it to create files, directories, change permissions, spawn child processes, etc. We will look at a sample of such commands here and then we will create a small program to manipulate the file system using it.

Well, having said that, it comes as a surprise, that the “os” module too provides a method name “uname()”, and the output is almost identical to the output of “platform.uname()” method.

os.uname()

('Linux', 'testyard.in', '3.16.0-77-generic', '#99~14.04.1-Ubuntu SMP Tue Jun 28 19:17:10 UTC 2016', 'x86_64')

However, as I said, the “os” module is normally used for filesystem operations. The “os” module exposes a number of methods, some of which are listed below, with some description and a sample output of those methods:

os.symlink(src, dest)

This creates a symbolic link named “dest” that points to “src”. So, if you are creating a program which needs to have a symlink created between 2 entities, you may use this powerful method.

os.mkdir(path,[perms])

This creates a directory identified by the path (first argument) with the permissions set as the second argument. By default, that is if you don't specify the second argument, then the permission is set to '0777', which basically means world readable, executable and writable. Not really a good idea. In case the path provided by the first argument doesn't exists, then an exception is raised and the method fails.

In order to handle this situation, the “os” module provides a second method that handles the case of a missing path. It is called “makedirs”. It is similar to the shell command “mkdir -p

”. So what “makedirs” does it that when it find a missing directory in the path, it creates one and finally creates the directory that is was supposed to create. You may invoke it through the following statement:

os.makedirs(path,[perms])

The arguments are similar to the “mkdir” method, and the default permissions are 777.

As an aside, I would strongly recommend that the programmer to make a habit of specifying the permissions, even when she/he is working on these things as an exercise. Without this habit, in a real world scenario where one has to meet deadlines, there is a tendency of cutting corners. And the python web developers normally cuts the corners in these situations. That poses a dreadful security risk, and hence, it should be avoided at all costs. Deadlines should be respected, but it should NOT be at the cost of making a software vulnerable in any way. If that makes the programmer to miss the deadline, so be it.

os.popen and its friends:

os.popen(command, [mode, [buffersize]]) is a method provided by the “os” module to open an pipe to or from the command being executed (the first parameter of the method). This is a very handy tool for a system admin when she/he wishes to execute a shell command and also wants to have the output of that command saved somewhere in order to use it later. Basically, what happens in case of popen is that a file object is returned to the variable that catches it, and this handle can be later on read to extract the info returned by the command.

For example, I ran the following command on my machine:

fp = os.popen("ls -la")

fp.read()

The output is as follows:

'total 108\ndrwxr-xr-x 11 supriyo root 4096 Nov 29 21:51 .\ndrwxrwxrwx 32 supriyo root 4096 Dec 29 20:02 ..\ndrwxrwxr-x 4 supriyo supriyo 4096 Jan 4 22:37 backup_files\n-rw-rw-rw- 1 root root 893 Dec 2 2017 dumppostdata.txt\n-rw-rw-r-- 1 supriyo supriyo 9209 Apr 16 2017 dump.txt\ndrwxrwxr-x 20 supriyo supriyo 4096 Nov 2 20:19 experiments\ndrwxrwxr-x 2 supriyo supriyo 4096 Nov 29 17:43 latex\ndrwxr-xr-x 2 supriyo root 4096 Jun 3 2017 logs\n-rw-r--r-- 1 root root 5659 Oct 23 2017 logs:tests_tasks.log\n-rw-rw-rw- 1 root root 77 Dec 29 19:50 logs:testyard.log\n-rw-rw-r-- 1 supriyo supriyo 423 Apr 28 2017 prerequisites.txt\n-rw-rw-r-- 1 supriyo supriyo 423 Apr 28 2017 prerequisites.txt~\ndrwxr-xr-x 8 supriyo root 4096 Jan 1 05:29 pyenv\n-rw------- 1 root root 17 Apr 23 2017 .Rhistory\n-rw-rw-r-- 1 supriyo supriyo 3124 Nov 29 21:51 sample_latex.txt\n-rw-rw-r-- 1 supriyo supriyo 3109 Nov 24 22:13 sample_latex.txt~\ndrwxrwxr-x 19 supriyo root 4096 Nov 9 00:23 softwares\n-rw-rw-r-- 1 supriyo supriyo 1728 Oct 21 02:39 ssl_cert_creation_process.txt\n-rw-rw-r-- 1 supriyo supriyo 1689 Oct 21 02:31 ssl_cert_creation_process.txt~\n-rw-rw-r-- 1 supriyo supriyo 1977 Nov 10 2015 test_responses.txt\n-rw-rw-r-- 1 supriyo supriyo 1127 Nov 6 2015 test_responses.txt~\ndrwxr-xr-x 7 supriyo root 4096 Jan 4 21:40 testyard\ndrwxrwxr-x 2 supriyo supriyo 4096 Dec 31 20:00 tmpfiles\ndrwxrwxr-x 4 supriyo supriyo 4096 Oct 25 00:36 WebRTC\n'

The above output looks a little un-nerving, but you may format it by splitting on newlines and make it look better to use it. No big deal.

The methods 'popen2()', 'popen3()' and 'popen4()' are similar, but with slight differences. I won't go into the differences here (as you can always do a os.popenx.__doc__ to find what happens, 'x' would be either 2, 3 or 4), but what I am going to do here is to explain popen2(), and you should be able to extrapolate the explanation for 3 and 4 variants of popen().

In a nutshell (again, nutshells are not big), what popen2 does is that it executes a command provided to it as a parameter as a subprocess and returns back a tuple of 2 file handles. One is a read handle (the second one), which you can read and extract the data churned out by the command provided as a parameter, and the first element of the tuple is a write handle. There isn't much that you might extract from this handle, and please also note that popen family has been deprecated since python version 6 onwards. In order to similar things, you now have the “subprocess” module. To execute a command, you may use the “subprocess.call()” method with the name of the command and its switches as a list as shown below:

subprocesss.call([“ls”, “-la”])

The above command actually takes 4 more arguments, namely, “stdin”, “stdout”, “stderr” and “shell”. The default values for the first three are “None” and the default for the fourth one (shell) is “False”.

“subprocess.call()” waits for the command execution to complete, and then returns the “returncode” attribute.

You may take a look at the elaborate capabilities of the subprocess module here:

https://docs.python.org/2/library/subprocess.html#module-subprocess

For now, I will conclude this post with a little more interesting material. There is a library in python named “psutil” and if you need to find out the availability of resources in your system dynamically from within a script, you may use it to do just that. Here is how you do it:

First, you need to install psutil in your system using “pip install psutil”. It doesn't come with core python distro.

Next, let us suppose you would like to record the status of your virtual_memory, CPU usage, and disk usage. This is what I did on my computer and the outputs are shown below:

import psutil psutil.virtual_memory() svmem(total=4051648512, available=604794880, percent=85.1, used=2922110976, free=174460928, active=2066489344, inactive=1459490816, buffers=54050816, cached=901025792, shared=313266176, slab=132530176) psutil.cpu_percent() 5.3 psutil.cpu_percent() 6.5 psutil.disk_usage('/dev/sda') sdiskusage(total=2009702400, used=4096, free=2009698304, percent=0.0) psutil.disk_partitions() [sdiskpart(device='/dev/sda4', mountpoint='/', fstype='ext4', opts='rw,errors=remount-ro'), sdiskpart(device='/dev/sda1', mountpoint='/boot/efi', fstype='vfat', opts='rw')] psutil.cpu_freq() scpufreq(current=1998.984, min=500.0, max=2000.0) psutil.cpu_count() 4

So, both from the point of view of a system administrator and a programmer, this is a very useful tool. It can provide both professionals with an adequate understanding of the system's resources and thus help them in providing better solutions.

End Game

I didn't go into a full-fledged programming example in this post as that is not what I wanted to do. Rather than that, my intention here was to provide the reader with the tools python provides us and what we can do with them. A programmer can certainly create a program that suits her/his needs based on the information given here. Of course, what I have done here is just a scratch on the surface, and anyone interested should do more research on the topics I have tried to cover here.

Ast Note

Some of our clients

team