The pythonpath is the list of directories that python uses to search for modules, packages and other files when executing a script or importing modules.

Example:

from myfile import myfunc

Here python needs to find both myfile and myfunc. It then adds myfunc to the namespace, and we can call the function from line 2.

How does python search

1. Current Directory
2. Python Path Environment Variables
3. Standard Library
4. Site-Packages

First, Python checks the current directory. If it doesn’t find the module there, it proceeds to inspect the directories specified in the Python Path environment variable. Should the module still remain elusive, Python then looks within the Standard Library( It is installed automatically when we install python) . If it’s not located there either, Python’s final stop is the Site-Packages directory( Site-Packages are those packages which are install using pip ) . If the module remains undiscovered throughout this search process, Python raises an error indicating that the module couldn’t be found.

Set pythonpath variables manually

1. Environment Variables

2. sys.path

import sys  
sys.path.append("/path/to/your/module")

Issues arising:

Let's say we have the following structure:

project/
│
└──O.py
├── folderA/
│   └── A.py
│
└── folderB/
    └── B.py

Then inside of A.py we do not have access to B.py. However from O.py we can call both A.py and B.py. These can even be imported inside of A.py and B.py no issue.

In Data science

In Data science, during the exploratory phase we often have the following folder structure:

project/
│
└──current_approach.py
└──utils.py
|
├── approachTypeA/
│   └── A1.py
|	└── A2.py
│
└── approachTypeB/
    └── B1.py
    └── B2.py

Previous Approaches are encapsulated inside of each file. However they use functions inside of utils.py. They cannot locally import utils.py because it is not in the Pythonpath. In this case, because it is not production code and the issue gets accentuated due to the usage of the python interactive window, therefore it is acceptable to manually append to the python path.

Hacky Solution: append the system path

import sys
from pathlib import Path

# Get the parent directory of the current file
parent_dir = Path(__file__).resolve().parent.parent

# Append the parent directory to the system path
sys.path.append(str(parent_dir))

Better Solution: Create a python package and install it in edit mode:

Python packaging