Issues with pre-loading images in sys container

[teja156]

Hello there, so I was following this guide https://github.com/nestybox/sysbox/blob/master/docs/quickstart/images.md#building-a-system-container-that-includes-inner-container-images--v012- to test pre-loading the sys container with some images.
I used the exact same Dockerfile and docker-pull.sh script as shown in the guide.

Dockerfile:

FROM nestybox/alpine-docker

COPY docker-pull.sh /usr/bin
RUN chmod +x /usr/bin/docker-pull.sh && docker-pull.sh && rm /usr/bin/docker-pull.sh

docker-pull.sh:

#!/bin/sh

# dockerd start
dockerd > /var/log/dockerd.log 2>&1 &
sleep 2

# pull inner images
docker pull busybox:latest
docker pull alpine:latest

# dockerd cleanup (remove the .pid file as otherwise it prevents
# dockerd from launching correctly inside sys container)
kill $(cat /var/run/docker.pid)
kill $(cat /run/docker/containerd/containerd.pid)
rm -f /var/run/docker.pid
rm -f /run/docker/containerd/containerd.pid

Command used to build the image:

sudo docker build -t alpine_test:new . --progress=plain

However, during the build process, dockerd fails to start in the sys container and as a result, the images were not pre-loaded. Here are the logs /var/log/dockerd.log:

#8 2.452 time="2024-08-30T19:26:41.436897298Z" level=info msg="Starting up"
#8 2.452 time="2024-08-30T19:26:41.438264417Z" level=info msg="containerd not running, starting managed containerd"
#8 2.452 time="2024-08-30T19:26:41.439765503Z" level=info msg="started new containerd process" address=/var/run/docker/containerd/containerd.sock module=libcontainerd pid=21
#8 2.452 time="2024-08-30T19:26:41.839883876Z" level=info msg="starting containerd" revision=3a4de459a68952ffb703bbe7f2290861a75b6b67 version=v1.7.17
#8 2.452 time="2024-08-30T19:26:41.862351970Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
#8 2.452 time="2024-08-30T19:26:41.867377883Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: can't change directory to '/lib/modules': No such file or directory\\n\"): skip plugin" type=io.containerd.snapshotter.v1
#8 2.452 time="2024-08-30T19:26:41.867427950Z" level=info msg="loading plugin \"io.containerd.event.v1.exchange\"..." type=io.containerd.event.v1
#8 2.452 time="2024-08-30T19:26:41.867442354Z" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
#8 2.452 time="2024-08-30T19:26:41.870664800Z" level=info msg="loading plugin \"io.containerd.warning.v1.deprecations\"..." type=io.containerd.warning.v1
#8 2.452 time="2024-08-30T19:26:41.870682741Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.blockfile\"..." type=io.containerd.snapshotter.v1
#8 2.452 time="2024-08-30T19:26:41.870839880Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.blockfile\"..." error="no scratch file generator: skip plugin" type=io.containerd.snapshotter.v1
#8 2.452 time="2024-08-30T19:26:41.870850084Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
#8 2.452 time="2024-08-30T19:26:41.871051381Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (overlay) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
#8 2.452 time="2024-08-30T19:26:41.871062300Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
#8 2.452 time="2024-08-30T19:26:41.871079980Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
#8 2.452 time="2024-08-30T19:26:41.871102723Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
#8 2.452 time="2024-08-30T19:26:41.871357499Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
#8 2.452 time="2024-08-30T19:26:41.872345490Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
#8 2.452 time="2024-08-30T19:26:41.872455999Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
#8 2.452 time="2024-08-30T19:26:41.872466542Z" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
#8 2.452 time="2024-08-30T19:26:41.872632169Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
#8 2.452 time="2024-08-30T19:26:41.872733248Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
#8 2.452 time="2024-08-30T19:26:41.872743510Z" level=info msg="metadata content store policy set" policy=shared
#8 2.452 time="2024-08-30T19:26:41.879597214Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
#8 2.452 time="2024-08-30T19:26:41.879755099Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
#8 2.452 time="2024-08-30T19:26:41.879802526Z" level=info msg="loading plugin \"io.containerd.lease.v1.manager\"..." type=io.containerd.lease.v1
#8 2.452 time="2024-08-30T19:26:41.879867020Z" level=info msg="loading plugin \"io.containerd.streaming.v1.manager\"..." type=io.containerd.streaming.v1
#8 2.452 time="2024-08-30T19:26:41.879901245Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
#8 2.452 time="2024-08-30T19:26:41.880354936Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
#8 2.452 time="2024-08-30T19:26:41.881266876Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
#8 2.452 time="2024-08-30T19:26:41.881746701Z" level=info msg="loading plugin \"io.containerd.runtime.v2.shim\"..." type=io.containerd.runtime.v2
#8 2.452 time="2024-08-30T19:26:41.881779007Z" level=info msg="loading plugin \"io.containerd.sandbox.store.v1.local\"..." type=io.containerd.sandbox.store.v1
#8 2.452 time="2024-08-30T19:26:41.881792088Z" level=info msg="loading plugin \"io.containerd.sandbox.controller.v1.local\"..." type=io.containerd.sandbox.controller.v1
#8 2.452 time="2024-08-30T19:26:41.881801433Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
#8 2.452 time="2024-08-30T19:26:41.881832120Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
#8 2.452 time="2024-08-30T19:26:41.881842818Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
#8 2.452 time="2024-08-30T19:26:41.881868949Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
#8 2.452 time="2024-08-30T19:26:41.881892283Z" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
#8 2.452 time="2024-08-30T19:26:41.881912524Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
#8 2.452 time="2024-08-30T19:26:41.881923155Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
#8 2.452 time="2024-08-30T19:26:41.881930514Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
#8 2.452 time="2024-08-30T19:26:41.882011295Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882023020Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882049917Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882060887Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882081764Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882092302Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882099835Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882118823Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882145355Z" level=info msg="loading plugin \"io.containerd.grpc.v1.sandbox-controllers\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882164716Z" level=info msg="loading plugin \"io.containerd.grpc.v1.sandboxes\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882175468Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882183628Z" level=info msg="loading plugin \"io.containerd.grpc.v1.streaming\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882201888Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882223253Z" level=info msg="loading plugin \"io.containerd.transfer.v1.local\"..." type=io.containerd.transfer.v1
#8 2.452 time="2024-08-30T19:26:41.882290734Z" level=info msg="loading plugin \"io.containerd.grpc.v1.transfer\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882310534Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882317375Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
#8 2.452 time="2024-08-30T19:26:41.882657991Z" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1
#8 2.452 time="2024-08-30T19:26:41.882690342Z" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="skip plugin: tracing endpoint not configured" type=io.containerd.tracing.processor.v1
#8 2.452 time="2024-08-30T19:26:41.882700267Z" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
#8 2.452 time="2024-08-30T19:26:41.882719923Z" level=info msg="skip loading plugin \"io.containerd.internal.v1.tracing\"..." error="skip plugin: tracing endpoint not configured" type=io.containerd.internal.v1
#8 2.452 time="2024-08-30T19:26:41.882728934Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882762991Z" level=info msg="loading plugin \"io.containerd.nri.v1.nri\"..." type=io.containerd.nri.v1
#8 2.452 time="2024-08-30T19:26:41.882797884Z" level=info msg="NRI interface is disabled by configuration."
#8 2.452 time="2024-08-30T19:26:41.884176280Z" level=info msg=serving... address=/var/run/docker/containerd/containerd-debug.sock
#8 2.452 time="2024-08-30T19:26:41.884416229Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock.ttrpc
#8 2.452 time="2024-08-30T19:26:41.884623747Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock
#8 2.452 time="2024-08-30T19:26:41.884658606Z" level=info msg="containerd successfully booted in 0.049604s"
#8 2.452 time="2024-08-30T19:26:42.531582249Z" level=error msg="failed to mount overlay: operation not permitted" storage-driver=overlay2
#8 2.452 time="2024-08-30T19:26:42.532260220Z" level=error msg="exec: \"fuse-overlayfs\": executable file not found in $PATH" storage-driver=fuse-overlayfs
#8 2.452 time="2024-08-30T19:26:42.533777746Z" level=warning msg="Unable to setup quota: operation not permitted\n"
#8 2.452 time="2024-08-30T19:26:42.536507551Z" level=info msg="Loading containers: start."
#8 2.452 time="2024-08-30T19:26:42.542072234Z" level=info msg="unable to detect if iptables supports xlock: 'iptables --wait -L -n': `iptables v1.8.10 (nf_tables): Could not fetch rule set generation id: Permission denied (you must be root)`" error="exit status 4"
#8 2.452 time="2024-08-30T19:26:42.564851708Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
#8 2.452 time="2024-08-30T19:26:42.565504283Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
#8 2.452 time="2024-08-30T19:26:42.565792447Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby

I have sysbox-runc set as the default docker runtime on the host.
Docker version: 27.2.0
Sysbox version: 0.6.4
Host OS: Ubuntu 22.04.4 LTS (Ubuntu Jammy)
Host Kernel Version: 6.5.0-44-generic

What am I doing wrong? Can someone guide me in the right direction?
Thanks in advance.