This is a Python tutorial showing how models can dynamically be added to an instance of declarative_base without triggering pep8 unused import. This example works best if the project uses a directory structure similar to the following (__init__.py has been omitted):

  • database
    • database_manager.py
    • models
      • example1.py
      • example2.py

The problem from this structure arises as the same instance of declarative_base needs to be imported by both example1.py and example2.py. While at the same time the database_manager.py would create the database with all its associated tables & relationships. The simplest solution is for the database_manager.py to have the instance declared and all the models to import it. However, this is not possible since all the models need to be imported as well. this is necessary as the database would otherwise be created without the tables & relationships defined by these models. Further complication this construction is that blind imports of the models without calling any method or accessing an attribute would trigger a pep8 unused import.

To resolve this two additional files are defined. The first file allows the definition of a declarative_base in a separate file. While the second file allows for the dynamic importing of models in the models directory. The resulting directory structure looks as follows:

  • database
    • create_database.py
    • database_manager.py
    • declarative_base.py
    • models
      • example1.py
      • example2.py

declarative_base.py

In the declarative_base.py the shared instance of declarative_base is instantiated. Additionally this file can be used to define a base class for all models to inherit automatically.

from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.ext.declarative import declared_attr


class Base(object):
    """Base object all sqlalchemy models will extend"""

    @declared_attr
    def __tablename__(cls):
        """Generate tablename based on class name

        Setting the model its __tablename__ attribute will override this
        generated name.
        """
        return cls.__name__.lower()


base = declarative_base(cls=Base)

create_database.py

Let’s cover create_database.py first as this is the most extensive file, it covers the automatic detection of files and classes inside the models directory. It could be adapted to perform the same or similar automatic detection in various other scenario’s easily. The file contains four methods starting with list_model_names. This method will return a collection of filenames inside the models directory.

import os
import pkgutil

# relative directory for models from this file
models_rel_directory = '/models'

def _list_model_names():
    """Gather collection of model names from models modules

    :return: A collection of module names iterated from the models directory
    """
    model_names = []

    # import everything from models directory relative to this file
    package_path = os.path.dirname(
        os.path.abspath(__file__)) + models_rel_directory

    # iterate over all module files and append their names to the list
    for __, modname, ispkg in pkgutil.iter_modules(path=[package_path]):
        model_names.append(modname)
    return model_names

In list_model_names the path relative to the current file is used to discover the models directory. This directory is subsequently used by pkgutil to find all the modules in this directory. Here every file that is not __init__.py will be added to the collection. Note that at this point the collection contains elements of type str and the actual modules are not yet imported.

The next method is import_models this method will actually import the modules. As argument it takes the collection from list_model_names. In addition it uses the importlib library and works under the assumption the model file names are similar to their class names.

import importlib

model_absolute_path = 'database.models.'

def _import_models(module_names):
    """Gather collection of tuples with modules and associated classes

    :return: A collection of tuples with the module and its associated class
    """
    imported_modules = []

    # for every module names import it and check that it has a class similar
    # to the name of the module. Example: if the file is named account it will
    # look for the Account class. If the file is called account_types it will
    # look for the AccountTypes class.
    for modname in module_names:
        # Capitalize first letters and subsequently remove all underscores.
        class_name = modname.title().replace('_', '')
        mod = importlib.import_module(model_absolute_path + modname)
        if not hasattr(mod, class_name):
            msg = "The module '" + model_absolute_path + ".%s' should have a" \
                  "'%s' class similar to the module name." % \
                  (modname, class_name)
            raise AttributeError(msg)
        else:
            imported_modules.append((mod, class_name))
    return imported_modules

For every str from the passed collection it will try to import it using the specified model_absolute_path. It will automatically determine the class name as long as it follows the pattern as clearly defined in the comment. If the model file is named account_types than the class must be named AccountTypes and so on. Only if the associated class for the module could be found will it be added to the collection. The collection itself consists of tuples with the loaded module and the name of the class.

Now for the method that brings most of these methods together called list_tables. This method will create a collection of all models their __table__ objects. These objects can be passed during the creation of a database when using a declarative_base instance. By accessing these properties and using it in the creation of a database having unused imports is prevented.

def _list_tables():
    """Collection of all the sqlalchemy model __table__ objects

    :return: A Collection of ``__table__`` objects from sqlalchemy models.
    """
    tables = list()
    model_names = _list_model_names()
    imported_modules = _import_models(model_names)
    for module in imported_modules:
        # Access the modules class and subsequent __table__
        tables.append(getattr(module[0], module[1]).__table__)
    return tables

The list_tables method uses getattr to access the module its class by string subsequently accessing the class its __table__ object. This is shown in: tables.append(getattr(module[0], module[1]).__table__).

The final method create_database_tables performs the actual creation of all tables & relationships by using the shared declarative_base. Any type of sqlalchemy engine can be passed to this method and in this example the database_manager.py is expected to call it.

def create_database_tables(engine):
    """Creates the database table using the specified engine"""
    tables = _list_tables()
    base.metadata.create_all(bind=engine, tables=tables)

Finally let’s put these four methods together into one large file.

final create_database.py

# -*- encoding: utf-8 -*-
# Copyright (c) 2019 Dantali0n
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.

import importlib
import os
import pkgutil

from database.declarative_base import base

# relative directory for models from this file
models_rel_directory = '/models'

# absolute path to models for imports
model_absolute_path = 'database.models.'


def create_database_tables(engine):
    """Creates the database table using the specified engine"""
    tables = _list_tables()
    base.metadata.create_all(bind=engine, tables=tables)


def _list_tables():
    """Collection of all the sqlalchemy model __table__ objects

    :return: A Collection of ``__table__`` objects from sqlalchemy models.
    """
    tables = list()
    model_names = _list_model_names()
    imported_modules = _import_models(model_names)
    for module in imported_modules:
        # Access the modules class and subsequent __table__
        tables.append(getattr(module[0], module[1]).__table__)
    return tables


def _list_model_names():
    """Gather collection of model names from models modules

    :return: A collection of module names iterated from the models directory
    """
    model_names = []

    # import everything from models directory relative to this file
    package_path = os.path.dirname(
        os.path.abspath(__file__)) + models_rel_directory

    # iterate over all module files and append their names to the list
    for __, modname, ispkg in pkgutil.iter_modules(path=[package_path]):
        model_names.append(modname)
    return model_names


def _import_models(module_names):
    """Gather collection of tuples with modules and associated classes

    :return: A collection of tuples with the module and its associated class
    """
    imported_modules = []

    # for every module names import it and check that it has a class similar
    # to the name of the module. Example: if the file is named account it will
    # look for the Account class. If the file is called account_types it will
    # look for the AccountTypes class.
    for modname in module_names:
        # Capitalize first letters and subsequently remove all underscores.
        class_name = modname.title().replace('_', '')
        mod = importlib.import_module(model_absolute_path + modname)
        if not hasattr(mod, class_name):
            msg = "The module '" + model_absolute_path + ".%s' should have a" \
                  "'%s' class similar to the module name." % \
                  (modname, class_name)
            raise AttributeError(msg)
        else:
            imported_modules.append((mod, class_name))
    return imported_modules