Skip to content

Docker and K3s with Lima VM in MacOS

Installing Lima and Docker CLI in MacOS

brew install lima

brew install docker

Configuring virtual networking with VDE_VMNET

Install vde_vmnet:

git clone https://github.com/virtualsquare/vde-2.git
cd vde-2
autoreconf -fis
./configure --prefix=/opt/vde
make
sudo make install

Note

You may need to install automake to be able to use autoreconf

brew install automake

And now installing vde_switch:

git clone https://github.com/lima-vm/vde_vmnet
cd vde_vmnet
make PREFIX=/opt/vde
sudo make PREFIX=/opt/vde install

Check following files installed:

  • /opt/vde/bin/vde_vmnet
  • /Library/LaunchDaemons/io.github.virtualsquare.vde-2.vde_switch.plist
  • /Library/LaunchDaemons/io.github.lima-vm.vde_vmnet.plist

Configuring Lima VM

Create the config file for lima vm lima-k3s.yaml:

# Example to use Docker instead of containerd & nerdctl
# $ limactl start ./docker.yaml
# $ limactl shell docker docker run -it -v $HOME:$HOME --rm alpine

# To run `docker` on the host (assumes docker-cli is installed):
# $ export DOCKER_HOST=$(limactl list docker --format 'unix://{{.Dir}}/sock/docker.sock')
# $ docker ...

# This example requires Lima v0.8.0 or later
images:
# Hint: run `limactl prune` to invalidate the "current" cache
- location: "https://cloud-images.ubuntu.com/impish/current/impish-server-cloudimg-amd64.img"
  arch: "x86_64"
- location: "https://cloud-images.ubuntu.com/impish/current/impish-server-cloudimg-arm64.img"
  arch: "aarch64"
mounts:
- location: "~"
- location: "/tmp/lima"
  writable: true
# containerd is managed by Docker, not by Lima, so the values are set to false here.
containerd:
  system: false
  user: false
provision:
- mode: system
  script: |
    #!/bin/sh
    curl -sfL https://get.k3s.io | sh -
- mode: system
  script: |
    #!/bin/sh
    sed -i 's/host.lima.internal.*/host.lima.internal host.docker.internal/' /etc/hosts
- mode: system
  script: |
    #!/bin/bash
    set -eux -o pipefail
    command -v docker >/dev/null 2>&1 && exit 0
    export DEBIAN_FRONTEND=noninteractive
    curl -fsSL https://get.docker.com | sh
    # NOTE: you may remove the lines below, if you prefer to use rootful docker, not rootless
    # systemctl disable --now docker
    apt-get install -y uidmap dbus-user-session
- mode: user
  script: |
    #!/bin/bash
    set -eux -o pipefail
    systemctl --user start dbus
    # dockerd-rootless-setuptool.sh install
    # docker context use rootless
    sudo usermod -aG docker "$USER"
- mode: user
  script: |
    #!/bin/bash
    newgrp docker
probes:
- script: |
    #!/bin/bash
    set -eux -o pipefail
    if ! timeout 30s bash -c "until command -v docker >/dev/null 2>&1; do sleep 3; done"; then
      echo >&2 "docker is not installed yet"
      exit 1
    fi
    if ! timeout 30s bash -c "until test -f /etc/rancher/k3s/k3s.yaml; do sleep 3; done"; then
      echo >&2 "k3s is not running yet"
      exit 1
    fi
  hint: |
    The k3s kubeconfig file has not yet been created.
    Run "limactl shell k3s sudo journalctl -u k3s" to check the log.
    If that is still empty, check the bottom of the log at "/var/log/cloud-init-output.log".
    See "/var/log/cloud-init-output.log". in the guest
portForwards:
- guestPortRange:
    - 1
    - 65535
  hostIP: 0.0.0.0
- guestSocket: "/var/run/docker.sock"
  hostSocket: "{{.Dir}}/sock/docker.sock"
networks:
- lima: shared
  interface: rd1
- lima: bridged
  interface: rd0
message: |
  To run docker on the host (assumes docker-cli is installed), run the following commands:
  ------
  docker context create lima --docker "host=unix://{{.Dir}}/sock/docker.sock"
  docker context use lima
  docker run hello-world
  echo "User is {{.User}}"
  ------
  To run `kubectl` on the host (assumes kubectl is installed):
  $ mkdir -p "{{.Dir}}/conf"
  $ export KUBECONFIG="{{.Dir}}/conf/k3s.yaml"
  $ limactl shell {{.Name}} sudo cat /etc/rancher/k3s/k3s.yaml >$KUBECONFIG
  $ kubectl ...

Edit $HOME/.lima/_config/networks.yaml:

paths:
  vdeSwitch: /opt/vde/bin/vde_switch
  vdeVMNet: /opt/vde/bin/vde_vmnet
  varRun: /private/var/run/lima
  sudoers: /private/etc/sudoers.d/lima

group: everyone

networks:
  shared:
    mode: shared
    gateway: 192.168.105.1
    dhcpEnd: 192.168.105.254
    netmask: 255.255.255.0
  bridged:
    mode: bridged
    interface: en0
    # bridged mode doesn't have a gateway; dhcp is managed by outside network
  host:
    mode: host
    gateway: 192.168.106.1
    dhcpEnd: 192.168.106.254
    netmask: 255.255.255.0

Since the commands to start and stop the vde_vmnet daemon requires root, the user either must have password-less sudo enabled, or add the required commands to a sudoers file. This can be done via:

limactl sudoers | sudo tee /etc/sudoers.d/lima

Starting Lima VM and using Docker from Host (MacOS)

Now you can start the lima image:

limactl start ./lima-k3s.yaml

You should check that the LIMA_HOME is created at $HOME/.lima/lima-k3s

Then, configure a Docker context with:

docker context create lima-k3s --docker "host=unix://$HOME/.lima/lima-k3s/sock/docker.sock"
docker context use lima-k3s

You should be able to list containers:

$ docker container ls
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

Configure KUBECONFIG from Host (MacOS)

Now K3s is running in the Lima VM. If we want to access from the host (MacOS), we only need to do the following from MacOS shell:

export KUBECONFIG="$HOME/.kube/k3s.yaml"
limactl shell lima-k3s sudo cat /etc/rancher/k3s/k3s.yaml >$KUBECONFIG

Change the interface in K3S to work in bridge mode

You can use the bridged IP (external IP from the same network of your host) to access your Lima VM, so you can access to your cluster from any other machine of your network. In this case, you may also want from your K3S cluster to use that IP as a NodeHost, so any LoadBalancer service type can be reached from your network.

To do this you need to change the K3S network interface to the one selected as bridged in your previous lima-k3s.yaml. Checking the lima config file should be something like this:

...
networks:
...
- lima: bridged
  interface: rd0
...

In this case we need to tell K3S to use the rd0 interface. This can be done by specifying the network parameter --flannel-iface in the k3s service in the Lima VM:

limactl shell lima-k3s sudo vim /etc/systemd/system/k3s.service

And you modify the file adding the flag --flannel-iface rd0 in the ExecStart command:

...
...
[Service]
...
ExecStart=/usr/local/bin/k3s \
    server --flannel-iface rd0