Skip to end of metadata
Go to start of metadata

Table of Contents

Overview

We have chosen to implement a Jupyter Notebook user interface on our local HPC, the Cheaha supercomputer, with Anaconda, a popular data science distribution.

This implementation attempts to keep end user instructions for running the Jupyter Notebook as close to the official documentation as possible.

Prerequisites

  1. A ssh client with tunnel support.
    • OpenSSH client is recommended.
  2. A web browser.
  3. A Cheaha account.

Start a Jupyter Notebook Server

  1. Shell into Cheaha.

    $ ssh -V
    OpenSSH_7.6p1, OpenSSL 1.0.2m  2 Nov 2017
    $ ssh jelaiw@cheaha.rc.uab.edu
    Last login: Thu Jun 21 15:40:45 2018 from 138.26.21.60
    
       ___ _                _
      / __\ |__   ___  __ _| |__   __ _
     / /  | '_ \ / _ \/ _` | '_ \ / _` |
    / /___| | | |  __/ (_| | | | | (_| |
    \____/|_| |_|\___|\__,_|_| |_|\__,_|
    -------------------------------------------------------------------------------
    Cheaha is an analysis and computing  resource available to the UAB research
    community and operated by the UAB IT Research Computing group.
    
    Use of this resource is governed by the UAB Acceptable Use Policy for Computer
    and Network Resources. Please review these policies on-line:
    
    http://www.uabgrid.uab.edu/aup
    
    For more information on Cheaha and the tools available to support research
    please review the documentation:
    
    http://docs.uabgrid.uab.edu/wiki/Cheaha
    
    ----------------------    Environment Modules    ------------------------------
    
    'module avail'            - show available modules
    'module add <module>'     - adds a module to your environment for this session
    'module initadd <module>' - configure module to be loaded at every login
    
    -------------------    Available Slurm Partitions    --------------------------
    
    Partition      Max Runtime Max Compute Nodes Priority
    ----------- -------------- ----------------- --------
    express            2 hours          No Limit        2
    short             12 hours                44        4
    medium      2 days 2 hours                44        6
    long        6 days 6 hours                22        8
    interactive        2 hours                 1       10
    -------------------------------------------------------------------------------
    
    ----------------------    Disk Quota Report    -------------------------------------
    /home/jelaiw      : 15.52GB of 53.68GB
    /data/user/jelaiw : 91.84GB of 5TB
    [jelaiw@login001 ~]$
    
    
    • This example uses the OpenSSH client, which is typically installed by default on recent versions of Windows 10, Linux, and Mac.

  2. Load BioITX Jupyter module.

    [jelaiw@login001 ~]$ module load bioitx/jupyter
    [jelaiw@login001 ~]$ module list
    Currently Loaded Modulefiles:
      1) slurm/17.02.2     3) Anaconda3/5.2.0
      2) rc-base           4) bioitx/jupyter
  3. Start a Jupyter Notebook server.

    [jelaiw@login001 ~]$ jupyter notebook --no-browser
    [I 14:19:46.724 NotebookApp] The port 8888 is already in use, trying another port.
    [I 14:19:46.762 NotebookApp] JupyterLab beta preview extension loaded from /opt/anaconda3/lib/python3.6/site-packages/jupyterlab
    [I 14:19:46.762 NotebookApp] JupyterLab application directory is /opt/anaconda3/share/jupyter/lab
    [I 14:19:46.767 NotebookApp] Serving notebooks from local directory: /home/jelaiw
    [I 14:19:46.767 NotebookApp] 0 active kernels
    [I 14:19:46.767 NotebookApp] The Jupyter Notebook is running at:
    [I 14:19:46.767 NotebookApp] http://localhost:8889/?token=228d267592d9f0646a6831a8bc78f9dc6efe9b815a75e465
    [I 14:19:46.767 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
    [W 14:19:46.767 NotebookApp] No web browser found: could not locate runnable browser.
    [C 14:19:46.768 NotebookApp]
    
        Copy/paste this URL into your browser when you connect for the first time,
        to login with a token:
            http://localhost:8889/?token=228d267592d9f0646a6831a8bc78f9dc6efe9b815a75e465&token=228d267592d9f0646a6831a8bc78f9dc6efe9b815a75e465
    

Note that the Jupyter Notebook server attempts to bind to port 8888, but fails, then tries another port 8889, and is successful. 

Don't run computationally-intensive code here!

These instructions run the Jupyter Notebook server on the Cheaha head node. This is fine for the many lightweight tasks you may typically perform in a web notebook.

However, if you plan to run computationally-intensive code, please request resources with SLURM, and run the Jupyter Notebook server on Cheaha compute nodes.

See FAQ.


Establish ssh tunnel

  1. Create a ssh tunnel.

    1. This is a second ssh connection to Cheaha.
$ ssh -L 8889:localhost:8889 jelaiw@cheaha.rc.uab.edu


---- SNIP ----

[jelaiw@login001 ~]$

If the ssh tunnel is not properly configured, you will not be able to connect to the Jupyter Notebook server!!

Note the -L 8889:localhost:8889 option, which builds a ssh tunnel specifying that connections to port 8889 on the local (client) host are to be forwarded to the given host and port (i.e. localhost on port  8889) on the remote side.


Connect to Jupyter Notebook server

  1. Connect to Jupyter Notebook server with web browser.

    • For this example, we copy & paste http://localhost:8889/?token=2e101ef6fb69ba67ffb17babb08eb80dc2adbd4cbcc33f8e from the Jupyter Notebook server console log output into the address/URL bar of web browser.
    • You should see the Jupyter Dashboard, as shown in the below screenshot.

FAQ

Can you recommend a tutorial for Jupyter Notebook beginners?

See https://www.dataquest.io/blog/jupyter-notebook-tutorial/ for Jupyter Notebook for Beginners: A Tutorial.

A Jupyter notebook for this tutorial is located at https://gitlab.rc.uab.edu/U-BRITE/jupyter-demos for reference.

How do I start a Jupyter Notebook server on a compute node?

See example below.

[jelaiw@login001 ~]$ srun --ntasks=1 --cpus-per-task=1 --time=01:00:00 --partition=express --job-name=jupyter --pty /bin/bash
[jelaiw@c0046 ~]$ module load bioitx/jupyter
[jelaiw@c0046 ~]$ jupyter notebook --ip=`hostname`
[I 08:55:17.210 NotebookApp] [nb_conda_kernels] enabled, 13 kernels found
[I 08:55:18.099 NotebookApp] JupyterLab beta preview extension loaded from /share/apps/rc/software/Anaconda3/5.2.0/lib/python3.6/site-packages/jupyterlab
[I 08:55:18.099 NotebookApp] JupyterLab application directory is /share/apps/rc/software/Anaconda3/5.2.0/share/jupyter/lab
[I 08:55:18.110 NotebookApp] Serving notebooks from local directory: /home/jelaiw
[I 08:55:18.110 NotebookApp] 0 active kernels
[I 08:55:18.110 NotebookApp] The Jupyter Notebook is running at:
[I 08:55:18.110 NotebookApp] http://c0046:8888/?token=04beed2873b0e58a4ff8a58d02a72f3e17687831e026d682
[I 08:55:18.110 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 08:55:18.114 NotebookApp]

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://c0046:8888/?token=04beed2873b0e58a4ff8a58d02a72f3e17687831e026d682&token=04beed2873b0e58a4ff8a58d02a72f3e17687831e026d682

Note that we request compute time the usual way through SLURM. In this example, we request an hour of time on the express partition.

Here is how the ssh tunnel was built.

$ ssh -L 8888:c0046:8888 jelaiw@cheaha.rc.uab.edu

You should continue to connect to localhost in the web browser.

If you typically copy & paste the connect string from the Jupyter Notebook server startup logs, you will need to update the "host" in the URL from the host name of the compute node to localhost.

In this example, you should connect to http://localhost:8888, not http://c0046:8888, in your web browser.

A big thank you to Ravi, Mike, John-Paul and others in Research Computing for their guidance and support!

How do I access the JupyterLab beta UI?

[jelaiw@login001 jupyter-demos]$ jupyter lab
[I 14:49:26.116 LabApp] The port 8888 is already in use, trying another port.
[I 14:49:26.117 LabApp] The port 8889 is already in use, trying another port.
[I 14:49:26.117 LabApp] The port 8890 is already in use, trying another port.
[I 14:49:26.140 LabApp] JupyterLab beta preview extension loaded from /share/apps/rc/software/Anaconda3/5.2.0/lib/python3.6/site-packages/jupyterlab
[I 14:49:26.140 LabApp] JupyterLab application directory is /share/apps/rc/software/Anaconda3/5.2.0/share/jupyter/lab
[W 14:49:26.147 LabApp] JupyterLab server extension not enabled, manually loading...
[I 14:49:26.148 LabApp] JupyterLab beta preview extension loaded from /share/apps/rc/software/Anaconda3/5.2.0/lib/python3.6/site-packages/jupyterlab
[I 14:49:26.148 LabApp] JupyterLab application directory is /share/apps/rc/software/Anaconda3/5.2.0/share/jupyter/lab
[I 14:49:26.159 LabApp] Serving notebooks from local directory: /home/jelaiw/repos/jupyter-demos
[I 14:49:26.159 LabApp] 0 active kernels
[I 14:49:26.159 LabApp] The Jupyter Notebook is running at:
[I 14:49:26.159 LabApp] http://localhost:8891/?token=6fb504829d7725a7c4e0ac23230d2da78a4e3c3a7ad25603
[I 14:49:26.159 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 14:49:26.160 LabApp]

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://localhost:8891/?token=6fb504829d7725a7c4e0ac23230d2da78a4e3c3a7ad25603&token=6fb504829d7725a7c4e0ac23230d2da78a4e3c3a7ad25603



How do I shut down a Jupyter Notebook server?

In the Jupyter Dashboard, click the "Quit" button in the top right of the user interface to shut down the Jupyter Notebook server.

Or Ctrl-C in your Jupyter Notebook server console.

^C[I 15:58:28.040 NotebookApp] interrupted
Serving notebooks from local directory: /home/jelaiw
0 active kernels
The Jupyter Notebook is running at:
http://localhost:8889/?token=2e101ef6fb69ba67ffb17babb08eb80dc2adbd4cbcc33f8e
Shutdown this notebook server (y/[n])? y
[C 15:58:29.140 NotebookApp] Shutdown confirmed
[I 15:58:29.141 NotebookApp] Shutting down 0 kernels
[jelaiw@login001 ~]$

  • No labels