Sharing code using a setup.py

A 101 on how you can share code throughout a Python mission

Picture by AbsolutVision on Unsplash

You’ve gotten the next mission construction:

├── src  ├── frequent
├── __init__.py
└── utils.py
├── config
└── necessities.txt
├── project1
├── __init__.py
└── essential.py
├── project2
├── __init__.py
└── essential.py
├── README.md

The place each sub-projects <project1> and <project2> have a essential.py file that use comparable code.

You (wish to) retailer this shared code inside a utils.py file that sits in a 3rd listing <frequent>.

To make use of any operate saved in utils.py you try to jot down the next import assertion into the essential.py recordsdata of <project1> and <project2>:

from frequent.utils import <function1>, ...

and also you get the next ModuleNotFoundError:

ModuleNotFoundError: No module named 'frequent'

You then try to change the import assertion to:

from .utils import <function1>

or

from ..utils import <function1>

returning an ImportError:

ImportError: tried relative import with no recognized guardian package deal

On this article, I’ll clarify why you obtain these errors and how you can keep away from this problem through the use of a setup.py file that creates a Python package deal.

Firstly…

What are the Errors?

ModuleNotFoundError: No module named ‘frequent’

This error happens when you find yourself making an attempt to import a Python package deal/module that’s not put in in your (digital) surroundings. In the event you run the command:

pip listing

in your terminal (mac) or command line immediate (home windows) the module that you’re making an attempt to put in is not going to be within the listing outputted to the display screen. For instance, working the pip listingcommand in my digital surroundings (venv) after receiving the ModuleNotFoundError I get:

Package deal        Model
-------------- -------
pip 21.3.1
setuptools 59.0.1

This doesn’t include any reference to the utils.py contained in the <frequent> package deal folder.

ImportError: tried relative import with no recognized guardian package deal

Relative import errors come up whenever you try to import a module from a guardian package deal however Python has a unique guardian package deal saved in __main__. Basically, you might be telling Python to import from a package deal that it isn’t conscious of.

An awesome rationalization of how Python shops data in its __main__ variable will be discovered within the [Script vs. Module] StackOverflow answer here.

Notice: Relative imports aren’t pythonic and ought to be averted if doable to keep away from confusion and enhance the readability of your code.

Some doable non-pythonic options you might have tried…

1. sys.path()

You may usually bandage each of those errors by including the specified package deal guardian path to the sys.path variable earlier than the erroring imports. For instance, including the next traces into the essential.py file:

Nevertheless, usually, I might recommend avoiding this method for the next causes:

  • Messy: It’s messy.
  • Error Propagation: It must be coded correctly to keep away from errors on re-use by others resulting from totally different folder places.
  • Code Duplication: It must be added to every .py file with the import errors.

2. Restructuring Challenge

You may restructure the mission and have every sub-project’s essential.py file on the root, enabling the imports to comply with the hierarchical listing construction downwards.

Two key causes to keep away from adopting this answer:

  • Messy: As your mission grows, this method will turn into disorganized, unclear, and messy.
  • Customization: It doesn’t permit you to construction your mission in a readable and distinctive approach.
  • Naming: You aren’t allowed identify recordsdata the identical as they’re all in the identical listing that means you’ll have to conjure up distinctive names resembling, main1.py and main2.py.

setup.py to the rescue…

A very good rule of thumb is that when you have quite a lot of code in a mission that may be reused by many modules, make it right into a Python package deal.

What’s a Python Package deal?

A folder that incorporates an __init__.py will probably be recognised by Python as a package deal. A package deal folder often incorporates a number of modules.

Packages which are put in in your (digital) surroundings will be recognized utilizing pip listing command.

Establishing the setup.py

  • A setup.py file gives data on how you can create a customized Python package deal. It leverages Python’s setuptools library and a fundamental file seems one thing like this:
fundamental setup.py file

The primary three kwargs of setup() are:

  • identify — The identify of your Python package deal.
  • install_requires — Factors the setup.py at a necessities.txt file that incorporates the Python libraries which are wanted by the Python modules which are going to be part of the package deal. Traces 3 and 4 let the setup.py learn within the package deal required Python packages from a specified location.
  • packages — The packages that you really want setup.py to incorporate within the customized Python package deal. You need to use setuptools.find_packages() to find the entire packages in your mission or enter them manually. find_packages() will establish the entire folders that include a __init__.py. For the instance mission construction, working find_packages() returns the next as packages (as they’re folders that include an __init__.py)
['common', 'project1', 'project2']
  • We solely wish to package deal the code inside <frequent> and so we add project1 and project2 to the exclude kwarg:
find_packages(exclude=["project1", "project2"])

Package deal Set up

Contemplating the mission setup in “The Drawback” part of this text, the next is an up to date construction when utilizing the setup.py screenshotted above.

├── src  ├── frequent
├── __init__.py
└── utils.py
├── config
└── necessities.txt
├── project1
├── __init__.py
└── essential.py
├── project2
├── __init__.py
└── essential.py
├── setup.py <--- Added in setup.py to the /src folder├── README.md

Notice: The setup.py file must be within the location of the packages which are getting used to construct the customized package deal (within the setup() packages kwarg).

To create the package deal, it’s good to go to the guardian folder dir of the setup.py file and run the next command:

pip set up -e ./<root of setup.py dir>

For the instance mission on this article, you’d run pip set up -e ./src:

terminal stdout when working pip set up -e ./src

The -e runs the package deal putting in in editable mode (dynamically) which robotically detects any modifications you make to the code when creating, avoiding the necessity to regularly re-install the package deal.

As soon as put in, you’ll be able to verify that it was efficiently put in by re-running:

pip listing

It is best to now see your package deal in addition to these listed within the necessities.txt within the listed packages:

Package deal        Model Editable mission location
-------------- ------- -----------------------------
DateTime 4.4
pip 21.3.1
pytz 2022.1
setuptools 59.0.1
simple-package 1.0.0 /Customers/danielseal/git_local/Sharing_Code_Example/src
zope.interface 5.4.0

Discover the Editable mission location a part of the pip listing output.

If you wish to laborious set up the package deal (statically) i.e. for a secure model the place you aren’t going to make any modifications you’ll be able to merely drop the -e and run pip set up ./src:

terminal stdout when working pip set up ./src

This set up is extra specific, returning the next output after working pip listing:

Package deal        Model
-------------- -------
DateTime 4.4
pip 21.3.1
pytz 2022.1
setuptools 59.0.1
simple-package 1.0.0 <--- that is the customized package deal
zope.interface 5.4.0
wheel 0.37.1. <--- see 2nd be aware beneath

Dropping the -e can even construct a wheel for the packing in your mission in a /construct folder as you aren’t in editable mode.

Notice:

  • each static and dynamic installations will add a .egg-info file into your mission.
  • earlier than working pip set up ./src , it’s suggested to put in the wheel package deal utilizing pip set up wheel to keep away from a warning that you just do not need wheel put in.
Challenge construction after working pip intall ./src (dropped the -e to point out the express set up).

At this step, you might have efficiently put in a customized Python package deal.

Now whenever you run the beforehand erroring imports, you shouldn’t get any errors…

stdout when working project1/essential.py and project2/essential.py after package deal set up.

Feedback

  • Warning: In the event you put in the packaged statically (with out including -ein set up command) you will have to re-run the set up command, including --upgrade or -uon the finish, when you make modifications to the code contained in the package deal folder you might be putting in ( <frequent> for this instance):
pip set up ./src --upgrade
  • Editable: As talked about, putting in the package deal utilizing pip set up -e ./src will set up the package deal in editable mode. It is not going to add a /construct folder to the mission
  • Model Management: When you have applied a brand new function or a repair to the Python package deal, it’s suggested to replace the model kwarg in setup() to advertise good model management.

This StackOverflow feed on updating an area Python package deal with pip explains using -u (improve) and -e (editable) rather well.

More Posts