How I Resolved Docker Storage Issues on a Full Disk
Introduction
As a DevOps engineer, I often encounter various challenges that require quick and effective solutions. One such incident occurred when I noticed that Docker had stopped running due to storage issues. This blog post outlines the steps I took to diagnose and resolve the issue by reallocating Docker storage to a larger disk.
The Problem
A few days ago, I discovered that my Docker containers had stopped running. Upon investigating, I found that the root filesystem had run out of space. Here’s the output of the ```df -h``` command:
As you can see, the root filesystem (/dev/md2) was completely full, but /home had plenty of available space. My goal was to move Docker’s storage to /dev/md3, which is mounted on /home.
Resolution
Here are the steps I took to resolve the issue and ensure Docker used the available space on /home:
Stop Docker:
First, I stopped the Docker service to prevent any issues during the migration process.
```sudo systemctl stop docker```
Create a New Directory for Docker Data:
Next, I created a new directory on /home to store Docker data.
```sudo mkdir -p /home/docker```
Move Existing Docker Data:
I moved the existing Docker data from /var/lib/docker to the new directory.
```sudo mv /var/lib/docker /home/docker```
Create a Symlink:
To ensure Docker could still find its data, I created a symbolic link from the old data directory to the new one.
```sudo ln -s /home/docker/docker /var/lib/docker```
Update Docker Service File (Optional):
In some cases, it might be necessary to update the Docker service file to point to the new data directory. I checked the Docker service file:
```sudo nano /lib/systemd/system/docker.service```
Then I added the — data-root flag to the ExecStart= line:
```ExecStart=/usr/bin/dockerd — data-root=/home/docker/docker```
Reload Systemd and Restart Docker:
After making these changes, I reloaded the systemd configuration and restarted Docker.
```sudo systemctl daemon-reload```
```sudo systemctl start docker```
Verify the Change:
I verified that Docker was using the new storage directory by checking the Docker info.
```docker info | grep ‘Docker Root Dir’```
The output confirmed the change:
Docker Root Dir: /home/docker/docker
Start Docker Compose:
Finally, I restarted my Docker Compose setup.
```docker-compose up -d```
Conclusion
By reallocating Docker’s storage to a disk with ample space, I was able to resolve the storage issue and get my containers running again. This incident reminded me of the importance of monitoring disk usage and planning for scalable storage solutions, especially in a production environment.
I hope this guide helps other DevOps engineers who might encounter similar issues. If you have any questions or need further assistance, feel free to leave a comment below!
