Storage full

Storage full errors can occur when users generate many files, transfer files, or perform other storage-intensive tasks. This document provides guidance to help you troubleshoot this.

Check disk usage

When encountering a storage full, the first step is to check your container’s disk usage. You can use the df -h command to display a summary of disk usage.

df -h

Example output:

root@9b8e325167b2:/# df -h
Filesystem                    Size  Used Avail Use% Mounted on
overlay                        20G   16M   20G   1% /
tmpfs                          64M     0   64M   0% /dev
tmpfs                         252G     0  252G   0% /sys/fs/cgroup
shm                            24G     0   24G   0% /dev/shm
/dev/sda2                     457G   12G  423G   3% /usr/bin/nvidia-smi
tmpfs                         252G   12K  252G   1% /proc/driver/nvidia
tmpfs                         252G  4.0K  252G   1% /etc/nvidia/nvidia-application-profiles-rc.d
tmpfs                          51G  4.4M   51G   1% /run/nvidia-persistenced/socket
tmpfs                         252G     0  252G   0% /proc/asound
tmpfs                         252G     0  252G   0% /proc/acpi
tmpfs                         252G     0  252G   0% /proc/scsi
tmpfs                         252G     0  252G   0% /sys/firmware
tmpfs                         252G     0  252G   0% /sys/devices/virtual/powercap

Key areas to check

Container Disk Usage: The primary storage area for your container is mounted on the overlay filesystem. This indicates the container’s root directory.

Filesystem                    Size  Used Avail Use% Mounted on
overlay                        20G   16M   20G   1% /

You can use the command du -sh . to check the space usage of the current directory.

By default, the volume disk or network volume is mounted at /workspace, You can check the usage with the following example::

root@9b8e325167b2:/# cd workspace/
root@9b8e325167b2:/workspace# du -sh .
194M    .

Identifying Large Files: To identify the top 10 largest files in your /workspace, you can run the following command:

root@9b8e325167b2:/# find /workspace -type f -exec du -h {} + | sort -rh | head -n 10
96M     /workspace/f.txt
96M     /workspace/e.txt
1.0K    /workspace/c.txt
512     /workspace/b.txt
512     /workspace/a.txt

Removing files and directories

Once you’ve identified large files or directories that are no longer needed, you can remove them to free up space.

This will permanently delete the file, folder. Use with caution.

# To delete a specific file, use the rm command:
rm /path/to/file

# To remove an entire directory and its contents, use the rm -r command:
rm -r /path/to/directory

JupyterLab hidden trash bin

If you deleted files through JupyterLab’s UI but storage isn’t freed, they may be in a hidden trash directory.

Quick check and fix

Check if trash directories exist:

Check for trash directories

{ ls -lah $HOME/.local/share/Trash/ 2>/dev/null && echo "✓ Found: $HOME/.local/share/Trash/"; ls -lah /workspace/.Trash* 2>/dev/null && echo "✓ Found: /workspace/.Trash*"; } || echo "✗ No trash directories found"

If trash was found above, clear it:

Clear trash directories

[ -d "$HOME/.local/share/Trash" ] && rm -rf $HOME/.local/share/Trash/* && echo "✓ Cleared home trash" || echo "✗ No home trash to clear"
ls -d /workspace/.Trash* 2>/dev/null && rm -rf /workspace/.Trash* && echo "✓ Cleared workspace trash" || echo "✗ No workspace trash to clear"

Additional storage

If your Pod needs more than 20GB of storage, consider using a network volume. For more information, see Network volumes, or refer to this blog post.