Introduction:
What is meant by “System Focussed Programming in Python”? Well, it means many things, depending on the perspective from which you look at it. For instance, for a system administrator, it is a means to monitor, debug, rectify, shutdown and start services in a system.
The list for a sys admin actually may be longer than what I specified, but I just wanted to put it in a nutshell (and as you may have already realized, nutshells are not large).
From the perspective of a programmer (I call myself one, but I guess it is open to debate), it means making programs that take advantage of system resources, optimize processing power, etc. From a DB administrator's point of view, it is how to make the database she/he is administering function in an optimized way, again, managing system resources wisely (this is a relative term – what is 'wisely' for you may not be very 'wise' for someone else). The list goes on and on, and I really can't do justice to all of them here. Hence, I have decided to address only a couple of perspectives – the perspective of a system administrator and the perspective of a programmer.
The Commands
Python allows you to do system programming in two different ways. Firstly, you have the two python workhorse modules – the 'os' and 'sys' (both of them have their legitimate own child modules which offer more functionality), and secondly, you can run shell commands from within python code, either using the ‘os.system’ or create a subprocess (using the 'subprocess' module). Apart from the above 2 methods, you may also use the information from the 'proc' file system (this is available only on Unix and Linux systems), maybe by opening a desired file in the proc file system that contains the information you need. Last, but not least, there is the python library named “platform” which provides valuable insights into the functioning of the system.
Here, I will first go through a list of functions/modules available from the 'os' , 'platform' and the 'sys' modules, and I will try to explain in brief what they do and what you should expect from them.
First, let us try to find a brief description of the OS you are working on. On the command line, you may run “uname -a”, which will provide you with an out similar to the one below:
Linux testyard.in 3.16.0-77-generic #99~14.04.1-Ubuntu SMP Tue Jun 28 19:17:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
The server name “testyard.in” in the above description will be different for various machines, but you will get much of the same output as above. To do the same with python, you may use the os.system (“uname -a”). However, the “platform” module allows us to get the specific info that you need, one at a time. So, in case you wish to know the architecture of your system, you may use the following python code: (Please note that the commands are in both italics and bold, whereas the outputs are in a simple italic font face.)
platform.machine()
The output of the above-mentioned command will provide you with something similar to the result mentioned below:
'x86_64'
Similarly, you may run the following commands to get a whole view of the system you are working on:
platform.version()
'#99~14.04.1-Ubuntu SMP Tue Jun 28 19:17:10 UTC 2016'
platform.platform()
'Linux-3.16.0-77-generic-x86_64-with-Ubuntu-14.04-trusty'
Again, please note that your output may have different information, as it is based on the system you are using.
The next command is a replacement for the “uname -a” shell command we discussed earlier, but it provides you with the same info in pure python form.
platform.uname()
('Linux', 'testyard.in', '3.16.0-77-generic', '#99~14.04.1-Ubuntu SMP Tue Jun 28 19:17:10 UTC 2016', 'x86_64', 'x86_64')
So, in the above output, you have every bit of information that you derived from the “uname -a” shell command, but the interesting difference here is the fact that every info in the output of “platform.uname” comes as elements in a tuple. This seems to be very convenient for the programmer as she may not need all the information provided by the output. In the case of a tuple, the programmer can specific elements from the tuple and use them as required.
Now, now and then, you may want to know the processor you are using, and the “platform.processor ()” is just the method that can provide you with that information:
platform.processor()
For me, the output is a bit disappointing, as it just displays the following:
'x86_64'
The output is definitely accurate, but it does miss a few details, and later on, we will see how to extract that information using a shell command from within the “os.system” method.
In case if you are interested in the release information or the architecture, you may use the following 2 commands. The outputs from my system are provided with the commands.
platform.release()
'3.16.0-77-generic'
platform.architecture()
('64bit', 'ELF')
There are quite a few other cool things that you can do with the “platform” module, like finding the platform you are using, the python branch, build, compiler, revision, version, etc. You are using. I will leave that to you as an exercise.
Earlier, we said that the modules “os” and “sys” are workhorses, but, so far we haven't got into it. Let us dive into it now and see what we can (and cannot) do with them.
The “os” module allows us to do a lot of file and directory manipulations. You can use it to create files, directories, change permissions, spawn child processes, etc. We will look at a sample of such commands here, and then we will create a small program to manipulate the file system using it.
Well, having said that, it comes as a surprise, that the “os” module provides a method name “uname()”, and the output is almost identical to the output of the “platform.uname()” method.
os.uname()
('Linux', 'testyard.in', '3.16.0-77-generic', '#99~14.04.1-Ubuntu SMP Tue Jun 28 19:17:10 UTC 2016', 'x86_64')
However, as I said, the “os” module is normally used for filesystem operations. The “os” module exposes several methods, some of which are listed below, with some description and a sample output of those methods:
os.symlink(src, dest)
This creates a symbolic link named “dest” that points to “src”. So, if you are creating a program that needs to have a symlink created between 2 entities, you may use this powerful method.
os.mkdir(path,[perms])
This creates a directory identified by the path (first argument) with the permissions set as the second argument. By default, that is, if you don't specify the second argument, then the permission is set to '0777', which means world-readable, executable, and writable. Not a good idea. In case the path provided by the first argument doesn't exist, then an exception is raised, and the method fails.
To handle this situation, the “os” module provides a second method that handles the case of a missing path. It is called “makedirs”. It is similar to the shell command “mkdir -p”. So what “makedirs” does is that when it finds a missing directory in the path, it creates one and finally creates the directory that it was supposed to create. You may invoke it through the following statement:
os.makedirs(path,[perms])
The arguments are similar to the “mkdir” method, and the default permissions are 777.
As an aside, I would strongly recommend that the programmer make a habit of specifying the permissions, even when she/he is working on these things as an exercise. Without this habit, there is a tendency to scuttle in real-world scenarios where one has to meet deadlines. And python web developers cut corners in these situations. That poses a dreadful security risk. And hence, you should avoid it at all costs. Deadlines should be respected, but they should NOT be at the cost of making software vulnerable in any way. If that makes the programmer miss the deadline, so be it. So ensure to hire Python developers from the best company like NEX Softsys who know this.
os.popen and its friends:
os.popen(command, [mode, [buffersize]]) is a method provided by the “os” module to open a pipe to or from the command being executed (the first parameter of the method). This is a very handy tool for a system admin when she/he wishes to execute a shell command and also wants to have the output of that command saved somewhere to use later. What happens in the case of popen is that a file object is returned to the variable that catches it, and this handle can be later on read to extract the info returned by the command.
For example, I ran the following command on my machine:
fp = os.popen("ls -la")
fp.read()
The output is as follows:
The above output looks a little unnerving, but you may format it by splitting it on newlines and make it look better to use it. No big deal.
The methods 'popen2()', 'popen3()' and 'popen4()' are similar, but with slight differences. I won't go into the differences here (as you can always do an os.popenx.__doc__ to find what happens, 'X would be either 2, 3, or 4), but what I am going to do here is to explain popen2(), and you should be able to extrapolate the explanation for 3 and 4 variants of popen().
In a nutshell (again, nutshells are not big), what popen2 does is that it executes a command provided to it as a parameter as a subprocess and returns a tuple of 2 filehandles. One is a read handle (the second one), which you can read and extract the data churned out by the command provided as a parameter, and the first element of the tuple is a write handle. There isn't much that you might extract from this handle, and please also note that popen family has been deprecated since python version 6 onwards. To similar things, you now have the “subprocess” module. To execute a command, you may use the “subprocess.call()” method with the name of the command and its switches as a list as shown below:
subprocesss.call([“ls”, “-la”])
The above command actually takes 4 more arguments, namely, “stdin”, “stdout”, “stderr” and “shell”. The default values for the first three are “None” and the default for the fourth one (shell) is “False”.
“subprocess.call()” waits for the command execution to complete, and then returns the “returncode” attribute.
You may take a look at the elaborate capabilities of the subprocess module here:
https://docs.python.org/2/library/subprocess.html#module-subprocess
For now, I will conclude this post with a little more interesting material. There is a library in python named “psutil” and if you need to find out the availability of resources in your system dynamically from within a script, you may use it to do just that. Here is how you do it:
First, you need to install psutil in your system using “pip install psutil”. It doesn't come with a core python distro.
Next, let us suppose you would like to record the status of your virtual_memory, CPU usage, and disk usage. This is what I did on my computer, and the outputs are shown below:
So, both from the point of view of a system administrator and a programmer, this is a very useful tool. It can provide both professionals with an adequate understanding of the system's resources and thus help them in providing better solutions.
End Game
I didn't go into a full-fledged programming example in this post, as that is not what I wanted to do. Rather than that, my intention here was to provide the reader with the tools python provides us and what we can do with them. A programmer can certainly create a program that suits her/his needs based on the information given here. Of course, what I have done here is just a scratch on the surface, and anyone interested should do more research on the topics I have tried to cover here.
