Hi James H. - Clarifying how Python libraries get installed when using the Remote Python Executor (RPE).
How Python Libraries Are Installed in RPE
The installation process depends on which approach you're using:
1. Using the Standard RPE Image (Pre-built)
When you use the standard RPE Docker image with:
docker pull snaplogic/rpe:main-5
docker run --memory-swap="-1" --restart=always -dti -p 5301:5301 -e "REMOTE_PYTHON_EXECUTOR_TOKEN=" -v /opt/remote_python_executor_log/:/opt/remote_python_executor_log/ --name=rpe snaplogic/rpe:main-5
The libraries are automatically installed at runtime using the SLTool.ensure() method in your Python script. You don't need to run pip commands manually. Here's how it works:
from snaplogic.tool import SLTool as slt
# Libraries are automatically installed when ensure() is called
slt.ensure("scikit-learn", "0.20.0")
slt.ensure("keras", "2.2.4")
slt.ensure("tensorflow", "1.5.0")2. Using a Custom RPE Image
If you want to pre-install libraries or have more control, you can:
- 1.
Download the custom RPE package (contains Dockerfile and requirements.txt)
- 2.
Add your required libraries to requirements.txt
- 3.
Build your custom Docker image:
- 4.
docker build --no-cache -t snaplogic_custom_rpe .
- 5.
Run the custom container:
- 6.
docker run --memory-swap="-1" --restart=always -dti -p 5301:5301 -e "REMOTE_PYTHON_EXECUTOR_TOKEN=" -v /opt/remote_python_executor_log/:/opt/remote_python_executor_log/ --name=rpe snaplogic_custom_rpe
Key Points:
No manual pip commands needed - The RPE handles library installation automatically
Runtime installation happens when you use SLTool.ensure() in your Python scripts
Custom images allow you to pre-install libraries via requirements.txt during the Docker build process
The requirements.txt approach is used when building custom RPE images, not for runtime installation
The confusion likely comes from the fact that there are two different workflows - the standard image uses runtime installation via SLTool.ensure(), while custom images use the traditional requirements.txt + Docker build approach.
