Conda Installation and Configuration to Avoid Quota Issues
Conda is a popular package manager. However, Conda creates many small files when installing packages, which can quickly fill your quota on /software. This article will tell you how to configure Conda to avoid this issue. Please note that these instructions also work for Mamba, which is a much faster drop-in replacement for Conda.
Installation
When you install Conda, it is best practice to install in your software partition, i.e. $MYSOFTWARE or `/software/projects/$PAWSEY_PROJECT/$USER/miniconda3`
$ mkdir -p $MYSOFTWARE/miniconda3 $ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O $MYSOFTWARE/miniconda3/miniconda.sh $ bash $MYSOFTWARE/miniconda3/miniconda.sh -b -u -p $MYSOFTWARE/miniconda3 $ rm -rf $MYSOFTWARE/miniconda3/miniconda.sh
.
Use links in a "Fake Home" to replace problamatic directories in your HOME
If you get a Disk Quota exceeded error, your home directory might be filled up. So try to clean your home directory and re-create those problematic directories in a "fake Home" in /software or /scratch, and then use soft links from your home directory to point towards them. (See the explanation in Pawsey Filesystems and their Use.)
Configuration
When Conda installs packages, it writes a lot of small files, which can quickly fill up your quota on /software. To avoid this issue, you can configure Conda via the .condarc file to put all those package files into the /scratch partition instead. There are two ways to update your .condarc file. The first way allows you to update your .condarc file from the command line. Ensure you edit the command below to match your project ID and user name:
After installing
$ conda config --add pkgs_dirs /scratch/pawsey0000/user-name/conda_cache
The second option is to open the .condarc file in a text editor and add the following lines, edited to match your project ID and user name:
pkgs_dirs: - /scratch/pawsey0000/user-name/conda_cache
Cleaning up
Conda supports users cleaning up old package files you no longer need. It’s important to do this regularly to avoid overburdening the Lustre filesystem on /scratch. To clean up, simply run:
$ conda clean -af