Python packages and plugins

By October 6, 2012Hacking, Uncategorized

While we obviously do a lot of work on the innards of Android, putting together a backend infrastructure requires a lot of engineering. We use a number of different technologies for this, and from time to time come across stuff we think could be generally interesting to others (even if it isn’t Android focused). This post is focused on Python, which we use pretty extensively. We like keeping up with the new language developments, so most of our Python posts will be related to Python 3, not the old Python 2.7.

One thing that can be a little confusing with Python is how packages work. Packages let you group your modules together and gives you a nice namespace. You can read all about them in the Python docs.

Something that’s pretty confusing is that importing a package does not mean that any modules inside that package are loaded.

Imagine a very simple package called testing, with a single foo module. E.g:


The foo module might look something like:

def bar():
    return 'bar'

Now, you might expect to be able to write code such as:

import testing

However, trying this won’t work, you end up with anAttributeError:

Traceback (most recent call last):
  File "", line 2, in
AttributeError: 'module' object has no attribute 'foo'

So, to fix this you need to actually import the module. There are (at least) two ways you can do this:

from testing import foo

Either of these put into sys.modules, will work fine.

But, what if you want to load all the modules in a package? Well, as far as I know there isn’t any built-in approach to doing this, so what we’ve come up with is a pretty simple function that, given a package, will load all the modules in the package, and return them as a dictionary keyed by the module name.

def plugin_load(pkg):
    """Load all the plugin modules in a specific package.

    A dictionary of modules is returned indexed by the module name.

    Note: This assumes packages have a single path, and will only
    find modules with a .py file extension.

    path = pkg.__path__[0]
    pkg_name = pkg.__name__
    module_names = [os.path.splitext(m)[0] for m in
                    if os.path.splitext(m)[1] == '.py' and m != '']
    imported = __import__(pkg_name, fromlist=module_names)
    return {m: getattr(imported, m) for m in module_names}

There are plenty of caveats to be aware of here. It only works with modules ending in .py, which may miss out on some cases. Also, at this point it doesn’t support packages that span multiple directories (although that would be relatively simple to add). Note: code testing on Python 3.2, probably needs some modification to work on 2.x (in particular I don’t think dictionary comprehensions work in 2.x).

If you’ve got a better way for achieving this, please let us know in the comments.

Benno Leslie

Leave a Reply