Meta Programming and the Role of Meta Classes in Python

article

Python is a language that is used for variety of purposes and its USP is the simplicity of use by a programmer to solve all types of problems that come up. Around 99.99% of such problems can be solved by using various simple features of the language, but there will be that 0.01% of problems that would need special handling. Please note that by “simple features” I mean features that are “Pythonic” in nature, and these features conform to the philosophy of the language. Hence, decorators, generators, iterators, classes, etc. come under the head of “simple features” since they all conform to the philosophy of python, which, in short may be stated as follows: Python is a language that is easy to read and the code itself may serve as a technical documentation for the (technical) person reading the code.” Hence python code is easily maintainable. If you want to know more about the philosophy of Python, please take a look at this page: https://www.python-course.eu/python3_history_and_philosophy.php.

When I started out in the software industry some fourteen years back, I used to do a lot of perl and C coding. While C/C++ programs are moderately maintainable (that is, if the coder had left enough comments at critical and complex parts of the codebase), maintaining and debugging perl programs were a nightmare. Most of the time what I used to do was when I was given a perl code that doesn't work, I used to figure out  (from whoever gave me the code) what the code is supposed to do, and I used to write that part from scratch. I worked on perl for about 8 years, and then I started getting exposed to Python development services, and it was love at first sight. (Nevertheless, I still do perl and C/C++ intermittently nowadays).

However, whether meta programming (and meta classes) conform to that philosophy is highly debatable.

The reason for this is that m eta programming, in a certain way, manipulates the written code, and the meaning (and the purpose) of that code is dynamically decided. Hence, reading such code does not provide the reader an immediate understanding of what the code it trying to do, and the reader has to understand it by looking at it from various contexts.

Well, enough talking, let's see how we can use meta programming in python. I will start with simple examples (sample codes), and as we go, we will see more complex scenarios. But before that, let us list down some facts that we need to remember while doing meta programming and implementing meta classes:

Examples and Notes:

  • In python, almost everything is an object (Please note I wrote “almost”, so we will take a look at what other types of entities we have in python).
  • In order to implement meta classes, you need to derive that class from 'type', which is not an object. You may also derive your class from another class which is derived from 'type', and you would successfully create a meta class.
  • Quite a lot of things we would be doing using meta classes can also be done using decorators. However, as I had mentioned above, there would be a few actions that would need to be implemented using meta classes.
  • Meta classes do not perform magic. However, it is a truly powerful feature in python, and hence it should be used in cases which 'normal' python code cannot do.
  • Meta programming involves introspection, and this can be easily implemented using the 'type' function. A built-in function that helps with introspection is the 'dir' function, which lists down all attributes/methods of an object in python. I understand that some of you will argue about it, but if you look closely at the definition of introspection, then you would certainly realize that the 'dir()' function satisfies the conditions laid  down by the definition of introspection. Hence I will always consider it as a tool for introspection.
  • In python 2.4+, all classes must inherit from some other class. If there is no class that you can inherit from, then you mush inherit from the class named “object”. In case of python 3.x, classes are derived from  “classobj” by default. In our examples, we will be considering Python2.7.x, but if you are using python 3, then please replace the first line in the code with class MyCrawler(classobj):

Example #1:

class MyCrawler(object): def __init__(self, webUrl=”http://www.amazon.in”): # Do some activity here... pass def listHyperLinks(pageContent): soup = BeautifulSoup(pageContent) allanchors = soup.findAll(“a”, {'class' : 'linkclass'}) allhrefs = [] for anchor in allanchors: if anchor.has_key('href'): allhrefs.append(anchor['href']) return allhrefs if __name__ == “__main__”: crawlobj = MyCrawler() type( crawlobj) inspect.isclass(MyCrawler) type(type( crawlobj))

The results are as follows:

<class '__main__.MyCrawler'> True <type 'type'>

The first 2 outputs are expected and possibly need no explanation. The 'type' of 'crawlobj' is an instance of the class we have defined. In the second statement, we checked if 'MyCrawler' is a class or not. Turned out to be a class. No surprises there. But, hey, what does the third statement show?

Remember I wrote “In python, nearly all is an object”? Well, actually in python everything is an object, some of them are instances of classes, some are instances of meta classes, except for type.

The question that arises now is what exactly is 'type'. In order to find that out, we will be using a method of the 'inspect' module named 'isclass()'. 'isclass' returns True if its argument is a class, and False otherwise. So let us see what happens when we do that to the last line in the previous code sample:

inspect.isclass(type(type( crawlobj)))

If you execute this at the python command line interpreter (or in a codebase), the result you would get is 'True'. So 'type' itself is a class, but it is not one of our standard classes, it is a bit special. 'type' is a built-in metaclass in Python from which you can create your own meta classes. So in order to create your own meta class, you would do something like the following:

class MyOwnMetaClass(type):
    # Do whatever you want to do.

Now, we have already seen that if we call 'type' with one argument, it returns us the type of that object passed in as parameter to 'type'. However, 'type' may even be called with 3 arguments. In that case, it creates a class on the fly. The class is created with the 3 parameters provided to the 'type' function. The 3 parameters are: 1) the name of the class to be created (a string value), 2) a list of base classes from which the class will inherit attributes (this may be empty in case the class doesn't need to inherit attributes/methods from other classes), and 3) a dictionary containing the names of all attributes and methods to be included in the class. Please note that you would still need to implement the methods provided as part of the third parameter. You may also leave the third parameter empty.

Here are the examples of the calls to 'type' and their outputs. We will also show you the corresponding class that gets created by the call to the 'type' function with 3 parameters.

MyCrawler = type('MyCrawler', (), {})

In the above code, the first argument is the name of the metaclass we are creating. The second and third arguments are empty in the example. The second argument is a list of base classes (which is empty in our example, so our class MyCrawler doesn't inherit from any base class), and the third argument, a dictionary, holds all the fields and methods the class might have. The above code could simply be written as:

class MyCrawler:
pass

Now let's consider something more realistic. After all you won't be writing code to create classes that do nothing. Consider the following code:

Example #2:

def login(self, username, password): # your login logic goes here... return sessioncode # This is the session Id that we managed to have after a successful login operation. MyCrawler = type('MyCrawler', (), {}) MyFacebookCrawler = type('MyFacebookCrawler', [MyCrawler,], {'login' : login, 'credentials' : ['james', 'bond007']})

The above code is equivalent to writing the following python code:

class MyCrawler: pass class MyFacebookCrawler(MyCrawler): credentials = ['james', 'bond007'] def login(self, username, password): # your login logic goes here... return sessioncode # This is the session Id that we managed to have after a successful login operation.

On the first look, the meta class implementation might look a bit cryptic and difficult to understand what is actually happening. However, as you start using this notations (of metaclasses), you will start getting used to it and you would be able to write code that is flexible and concise. However, as I have stated before, these things should be used only in cases where they are the only solution. For instance,if you are a 'professional ethical hacker' (excuse me here, but there is actually no such thing called 'ethical hacker', since the term 'hacking' means breaking into people's private spaces which in itself is a crime. I have worked with organizations that do this, and believe me, the biggest clients of such organizations are not corporates, but government security agencies. If necessary, they can get the exact information about the clicks made by you and the keys pressed by you in a given span of time. Sorry for digressing.), you would possibly be using this type of logic in your everyday work. A web application developer or a REST API framework developer might also use this, but that would be only in one or two places, not all over the entire codebase.

Metaprogramming is also performed when you create and use decorators. So let us put a rapid show at it.

Decorators as metaprogramming tool

If you have used and created decorators, then you already know what it is and how it works. However, for those who haven't created a decorator, I will quickly go through the steps of creating one.

Firstly, a decorator is a callable that takes a function as an argument and performs some actions before and (may be) after calling the function from within itself. Here is an example of a decorator I wrote for a certain project, and it checks if a session in a web application is valid or not.

Example #3:

def is_session_valid(func): """ Decorator version of checksession to check the validity of a session. Uses the same isloggedin function above internally. """ def sessioncheck(request): if not isloggedin(request): message = error_msg('1006') response = HttpResponseRedirect(gethosturl(request) + "/" + mysettings.LOGIN_URL + "?msg=" + message) else: # Return the request response = func(request) return response return sessioncheck

Since we will be taking a look at the structure here, there is no need to follow the logic in it. “is_session_valid” is a usual python function, but preferable than getting a scalar or a list/tuple/dict as an argument, it takes a function as an argument. Next, once inside the “is_session_valid” function, we define another function named “sessioncheck” which takes a HTTP request object as an argument. For our purposes here, you may ignore where this request object comes from. Inside “sessioncheck”, it does some processing and eventually calls the function that was passed in as an argument. Finally, “is_session_valid” returns the internal function “sessioncheck”.

Now, this is an example of a simple function decorator, and in fact it doesn't have much to do with metaprogramming directly. However, what if we were to do this to an entire class? (Remember, when we defined the term decorator, we mentioned the word “callable”, and it means that an object for which the function __call__ is defined. So that brings classes into the fray too). For a single class, you could make a class decorator (I won't be explaining how to create a class decorator here as that is beyond the scope of our discussion here. However, if you want to know how to create one, there are excellent tutorials on the internet that can teach you how to create one), and wrap it with it. But what if we are considering multiple classes? What if we do not know in advance how many classes need to be wrapped with our decorator? The solution is to create a metaclass.

In order to write a meta class, we normally modify and/or define methods like __new__, __prepare__ and __call__. We might also mess with __init__, but we might not want to do that since __init__ is called when an object has already been created and we want to initialize it. The creation of the object happens inside __new__, and hence our focus is normally on that method. Let's see some code that creates a metaclass using some of the above mentioned magic methods (Please try using python3.x):

Example #4:

class MetaDemo(type): def __new__(cls, clsname, baseclasses, attribdict): a = super().__new__(cls, clsname, baseclasses, attribdictattribdict) a.attrib = 100 return a class MyClass(metaclass=MetaDemo): pass MyClass.attrib

The output of the above code would be:

100

Let's try to explain what happened above. It is simple: MyClass recognized the attrib attribute automatically from the metaclass MetaDemo.

A metaclass can be thought of as a class factory, in the same way we consider a class to be an object factory.

Wrapping Up:

Well, what we have been able to touch here is the tip of the proverbial iceberg. Metaprogramming and metaclasses are a vast and deep and a fascinating topic and I think one can write a 250 page book on it, that will be able to delve into these topics at a moderate depth. This write up is basically for inculcating interest on this topic, and I sincerely hope I have been able to do justice in that regard. We have left out topics like introspection in python which go hand in hand with metaprogramming. Specifically, one should look at how __getattribute__ and its brothers and sisters (namely __getattr__, etc) work. I would request the reader to research such things. There are a whole lot of wonderful documentation and tutorials on the internet that take up one aspect of metaprogramming and delve into the very depths of that specific aspect. So it would take time to learn it. But once learned, it will be a wonderful tool for the programmer who has to tackle problems that need very dynamic and unique solutions. So it is definitely worth the effort.

- Thanks for reading.

  • img
  • img
  • img
  • img
  • img
  • img
  • img
  • img
  • img
  • img
  • img
  • img
  • img
  • img
  • img
  • img
  • img