yes - [ubuntu server 22.04. 3 LTS] 도커 에러, Cannot connect to the Docker daemon, structure needs cleaning

[ubuntu server 22.04. 3 LTS] 도커 에러, Cannot connect to the Docker daemon, structure needs cleaning

외장하드에 도커 저장소를 옮긴 후, 아래와 같은 2가지 에러가 발생했습니다.

- Cannot connect to the Docker daemon

- structure needs cleaning

발생원인을 추론컨데,

- 외장하드가 read only 로 되어 있어서, RW 로 바꿔주면서, 도커 실행이 되지 않았습니다.

$sudo docker info
Client: Docker Engine - Community
Version:    24.0.7
Context:    default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
    Version: v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
    Version: v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info

$sudo dockerd
INFO[2024-01-11T15:34:51.875822011+09:00] Starting up
INFO[2024-01-11T15:34:51.883085597+09:00] detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf
ERRO[2024-01-11T15:34:52.685635925+09:00] [graphdriver] prior storage driver overlay2 failed: lstat /vo2/image/overlay2/check-overlayfs-support2277673038: structure needs cleaning
INFO[2024-01-11T15:34:52.699653959+09:00] stopping event stream following graceful shutdown error="context canceled" module=libcontainerd namespace=plugins.moby
failed to start daemon: error initializing graphdriver: lstat /vo2/image/overlay2/check-overlayfs-support2277673038: structure needs cleaning

해결방법

- 언마운트 후, fsck 실행

$sudo umount -f /vo2
$sudo fsck /dev/sda2
fsck from util-linux 2.37.2
e2fsck 1.46.5 (30-Dec-2021)
/dev/sda2 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Deleted inode 262773 has zero dtime. Fix<y>? yes
Deleted inode 263018 has zero dtime. Fix<y>? yes
Inodes that were part of a corrupted orphan linked list found. Fix<y>? yes
Inode 263022 was part of the orphaned inode list. FIXED.
Inode 263031 was part of the orphaned inode list. FIXED.
Deleted inode 263034 has zero dtime. Fix<y>? yes
Deleted inode 263043 has zero dtime. Fix<y>? yes
Inode 263044 was part of the orphaned inode list. FIXED.
Inode 263045 was part of the orphaned inode list. FIXED.
Inode 263046 was part of the orphaned inode list. FIXED.
Inode 263277 was part of the orphaned inode list. FIXED.
Deleted inode 263285 has zero dtime. Fix<y>? yes
Inode 263328 was part of the orphaned inode list. FIXED.
Inode 263338 was part of the orphaned inode list. FIXED.
Inode 263339 was part of the orphaned inode list. FIXED.
Inode 263383 was part of the orphaned inode list. FIXED.
Inode 263389 was part of the orphaned inode list. FIXED.
Inode 263417 was part of the orphaned inode list. FIXED.
Inode 281718 was part of the orphaned inode list. FIXED.
Inode 288125 was part of the orphaned inode list. FIXED.
Pass 2: Checking directory structure
Entry 'check-overlayfs-support2277673038' in /image/overlay2 (131085) has deleted/unused inode 263018. Clear<y>? yes
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences: +524288 +(524291--524293) +524295 +(524297--524298) +(524300--524301) +(524304--524305) +(524308--524309) +524311 +524313 +(524315--524317) +(524320--524323) +(524326--524335) +(524338--524415) +(524417--524431) +(524433--524450) +(524454--524463) +(524465--524488) +(524490--524521) +(524523--524609) +(524611--524649) +(524651--524673) +(524675--524739) +(524741--524770) +(524772--524773) +(524775--524867) +(524870--524898) +(524901--524929) +524931 +524934 +(524936--524937) +(524941--524943) +524945 +524949 +524953 +524955 +(524957--524960) +(524964--524965) +(524968--524971) +(524974--524976) +524978 +524981 +(524985--524998) +525000 +525002 +525006 +(525010--525011) +525014 +(525016--525017) +(525019--525020) +525023 +(525027--525032) +525036 +(525038--525040) +525042 +525045 +(525049--525055) +525057 +(525059--525064) +525068 +(525070--525071) +525073 +525077 +525081 +525083 +(525085--525088) +(525092--525093) +(525096--525099) +(525102--525104) +525106 +525109 +(525113--525121) +525123 +525126 +(525128--525129) +(525133--525135) +525137 +525141 +525145 +525147 +(525149--525152) +(525156--525157) +(525160--525163) +(525166--525168) +525170 +525173 +(525177--525185) +(525187--525188) +(525190--525315) +(525318--525346) +(525348--525350) +(525352--525423) +(525426--525443) +(525447--525474) +(525477--525487) +(525489--525507) +(525510--525511) +(525514--525551) +(525553--525569) +525571 +525574 +(525576--525577) +(525581--525583) +525585 +525589 +525593 +525595 +(525597--525600) +(525604--525605) +(525608--525611) +(525614--525616) +525618 +525621 +(525625--525633) +525635 +525638 +(525640--525641) +(525645--525647) +525649 +525653 +525657 +525659 +(525661--525664) +(525668--525669) +(525672--525675) +(525678--525680) +525682 +525685 +(525689--525697) +525699 +525702 +(525704--525705) +(525709--525711) +525713 +525717 +525721 +525723 +(525725--525728) +(525732--525733) +(525736--525739) +(525742--525744) +525746 +525749 +(525753--525761) +525763 +525766 +(525768--525769) +(525773--525775) +525777 +525781 +525785 +525787 +(525789--525792) +(525796--525797) +(525800--525803) +(525806--525808) +525810 +525813 +(525817--525827) +(525831--525889) +525892 +(525895--525953) +(525955--525956) +(525958--526015) +526019 +(526021--526031) +(526034--526036) +(526039--526047) +(526049--526052) +(526055--526063) +526065 +526068 +(526070--526079) +(526082--526084) +(526087--526095) +526097 +(526099--526100) +(526103--526112) +(526114--526115) +(526119--526129) +526131 +(526135--526143) +526145 +526148 +(526150--526159) +(526163--526165) +(526167--526177) +526179 +526181 +(526183--526191) +(526194--526195) +526197 +(526199--526207) +526213

-

중략

-

Fix<y>? yes
Free blocks count wrong for group #22 (2137, counted=2139).
Fix<y>? yes
Free blocks count wrong for group #29 (5633, counted=5635).
Fix ('a' enables 'yes' to all) <y>? yes
Free blocks count wrong (6510410, counted=6559502).
Fix ('a' enables 'yes' to all) <y>? yes
Inode bitmap differences: -262773 -263018 -263022 -263031 -263034 -(263043--263046) -263277 -263285 -263328 -(263338--263339) -263383 -263389 -263417 -281718 -288125
Fix ('a' enables 'yes' to all) <y>? yes
Free inodes count wrong for group #32 (0, counted=17).
Fix ('a' enables 'yes' to all) <y>? yes
Free inodes count wrong for group #34 (0, counted=1).
Fix<y>? yes
Free inodes count wrong for group #35 (6787, counted=6788).
Fix<y>? yes
Free inodes count wrong (1765936, counted=1765955).
Fix<y>? yes

/dev/sda2: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sda2: 77245/1843200 files (0.2% non-contiguous), 811506/7371008 blocks

- 마운트 후, 도커 실행

$sudo mount /vo2
$sudo dockerd
INFO[2024-01-11T15:48:25.956578628+09:00] Starting up
INFO[2024-01-11T15:48:25.988710290+09:00] detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf
INFO[2024-01-11T15:48:27.287441910+09:00] [graphdriver] using prior storage driver: overlay2
INFO[2024-01-11T15:48:28.240762163+09:00] Loading containers: start.
WARN[2024-01-11T15:48:30.677513139+09:00] Could not load necessary modules for IPSEC rules: protocol not supported
INFO[2024-01-11T15:48:33.464340823+09:00] Removing stale sandbox 2bc88d222667149745a823fa18cf93a685308c6b9885e7e5edc4162f9f85a321 (abb9980fb9d3ecd3cd6e6d9c010e507ee16b87a8843a02022c018d555aa05735)
WARN[2024-01-11T15:48:33.664378208+09:00] Failed to clean up network resources on container glances disconnect: failed to set gateway while updating gateway: route for the gateway 172.18.0.1 could not be found: network is unreachable
WARN[2024-01-11T15:48:33.739497184+09:00] Error (Unable to complete atomic operation, key modified) deleting object [endpoint 87c20047aa61e3d60eaa32553adcd6114370a0603071f4e11024e5aa95a4edd1 5adaf881a989f71b1cd405ff57a7d25679f99aa944c30ec010a3c385268e1be9], retrying....
WARN[2024-01-11T15:48:33.851551830+09:00] Error (Unable to complete atomic operation, key modified) deleting object [endpoint f479dc10473c967b2f27d33070663b0ba320d3bc644228c06eb088ab94dd5c44 dcd91d432ca9125b2bc53a5ebae9ba0ef25078ff819fcbfa97bb868fd23c4196], retrying....
INFO[2024-01-11T15:48:33.920290703+09:00] Removing stale sandbox 6e4b19dc63854b6c5a26479d4ce538e21df55a3487ce0b03817042b40237f7fd (a6c2eb45b60fd6e4c472d64ac4410006d1cbf9ae384473c36cd275c6d6a04761)
WARN[2024-01-11T15:48:34.215215058+09:00] Failed to clean up network resources on container portainer disconnect: failed to set gateway while updating gateway: route for the gateway 172.18.0.1 could not be found: network is unreachable
WARN[2024-01-11T15:48:34.287493259+09:00] Error (Unable to complete atomic operation, key modified) deleting object [endpoint 87c20047aa61e3d60eaa32553adcd6114370a0603071f4e11024e5aa95a4edd1 20071051fd7a96da15c63279b606e551637a7f3454698d3061e32ebf3738ae86], retrying....
WARN[2024-01-11T15:48:34.398351155+09:00] Error (Unable to complete atomic operation, key modified) deleting object [endpoint f479dc10473c967b2f27d33070663b0ba320d3bc644228c06eb088ab94dd5c44 6d876f925afaf4b6ef323771a974f38cffec184e14578e5ff304e3ed060add41], retrying....
INFO[2024-01-11T15:48:34.493358587+09:00] Removing stale sandbox 7167764dbf20dcc065ce41d8d2f7170018a87fa1803ebf187581d6b9e676cdb0 (1f73bea4477c7ccaef59b6072d40d8666c6d4bfdcf90930091edaf2c6d579877)
WARN[2024-01-11T15:48:34.673433057+09:00] Failed to clean up network resources on container filebrowser disconnect: failed to set gateway while updating gateway: route for the gateway 172.18.0.1 could not be found: network is unreachable
WARN[2024-01-11T15:48:34.747967409+09:00] Error (Unable to complete atomic operation, key modified) deleting object [endpoint 87c20047aa61e3d60eaa32553adcd6114370a0603071f4e11024e5aa95a4edd1 afa3dd92967bfba996570df6bd0cc8113bd3459ae7708a6d4b604a0537d3c0dd], retrying....
WARN[2024-01-11T15:48:34.863477180+09:00] Error (Unable to complete atomic operation, key modified) deleting object [endpoint f479dc10473c967b2f27d33070663b0ba320d3bc644228c06eb088ab94dd5c44 d73790625312476fef5aaab078e30a95b9ae33b79832843f58e10d64b21b74d9], retrying....
INFO[2024-01-11T15:48:35.272955372+09:00] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address
INFO[2024-01-11T15:48:36.074949435+09:00] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]
INFO[2024-01-11T15:48:36.075214915+09:00] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]
INFO[2024-01-11T15:48:36.430720581+09:00] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]
INFO[2024-01-11T15:48:36.436714743+09:00] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]
INFO[2024-01-11T15:48:38.444512647+09:00] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]
INFO[2024-01-11T15:48:38.444889570+09:00] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]
INFO[2024-01-11T15:48:49.219865976+09:00] Loading containers: done.
INFO[2024-01-11T15:48:55.839188534+09:00] Docker daemon                                 commit=311b9ff graphdriver=overlay2 version=24.0.7
INFO[2024-01-11T15:48:55.977738140+09:00] Daemon has completed initialization
INFO[2024-01-11T15:48:58.740027633+09:00] API listen on /var/run/docker.sock