The following is a tutorial for remotely running a Jupyter notebook on the cluster with custom compute through VS-code. Modified from here.

First, if you are using miniconda, you have to make sure that the kernels for your environments are available. 1. Make sure that ipython is installed in every env <env_name> with:

1conda activate env_name
2conda install anaconda::ipython

1. Generate the kernels by running the following command for each environment (replace <env_name>) :

1path\to\miniconda\envs\<env_name>\bin\ipython kernel install --user --name <env_name>

Assuming you want to run the notebook in the directory notebook_dir, run the following script on the cluster with sbatch or srun:

1#!/bin/bash
2
3# Change number of tasks, amount of memory and time limit according to your needs
4
5#SBATCH -n 1
6#SBATCH --time=3:0:0
7#SBATCH --mem=4G
8#SBATCH -J jupyter
9#SBATCH --output=%x-%j.out
10
11## Get host IP
12host_ip=$(hostname -i)
13
14# Select free random port
15jupyter_port=$(python3 -c 'import socket; s=socket.socket(); s.bind(("", 0)); print(s.getsockname()[1]); s.close()');
16
17# Print tunnel command
18echo $jupyter_port:$host_ip
19
20# Change to notebook directory
21cd $notebook_dir
22
23# Start Jupyter
24jupyter notebook --no-browser --ip=0.0.0.0 --port=$jupyter_port

This script will supply you with multiple addresses (in terminal if you srun it, in a log file if you sbatch it). One of these addresses looks like this:

1http://phoenix-gw-01:56631/?token=155a2a652697cda7dd1775bc97542da54b05c7d90d8249b6

On VS-code (Assuming that you are using remote-SSH to connect to the river host) open the notebook. In the top right corner press the kernel button, pick select another kernel, and then existing jupyter server. Then enter that address from before. The notebook is now running on the cluster server!

Make sure you close the job when you're done with the notebook.

Bonus

Add the following to the start of your .zshrc file to automatically ask if you want to run an interactive Jupyter server session / ZSH shell when you connect to phoenix-gw. Make sure the change <jupyter_script_path> to the path of the script above.

1if [[ -n $SSH_CONNECTION && $HOST == phoenix-gw* ]]; then
2    read "choice?What would you like to run? (j for Jupyter / z for ZSH shell / Enter to skip): "
3    echo ""
4
5    if [[ $choice == "j" || $choice == "z" ]]; then
6        # Common parameters for both options
7        read "ram?Enter RAM amount in GB (e.g., 16): "
8        read "cpus?Enter number of CPUs (e.g., 2): "
9        read -q "killable?Make job killable? (y/n) "
10        echo ""
11        read -q "gpu?Do you want GPU? (y/n) "
12        echo ""
13
14        gpu_flag=""
15        if [[ $gpu == "y" ]]; then
16            read "gpu_type?Enter GPU type (e.g., rtx2080, a10, I4): "
17            gpu_flag="--gres=gpu:${gpu_type}"
18        fi
19
20        if [[ $choice == "j" ]]; then
21            # Run Jupyter
22            if [[ $killable == "y" ]]; then
23                srun --killable --mem=${ram}g -c${cpus} ${gpu_flag} --time=12:0:0 <jupyter_script_path>
24            else
25                srun --mem=${ram}g -c${cpus} ${gpu_flag} --time=12:0:0 <jupyter_script_path>
26            fi
27        else
28            # Run ZSH
29            if [[ $killable == "y" ]]; then
30                srun --killable --mem=${ram}g -c${cpus} ${gpu_flag} --time=12:0:0 --pty zsh -i
31            else
32                srun --mem=${ram}g -c${cpus} ${gpu_flag} --time=12:0:0 --pty zsh -i
33            fi
34        fi
35    fi
36fi