Docker on MacOS is still slow?

Table of contents

TLDR Link to heading

Two years after my first analysis of Docker performance on MacOS, things have improved significantly. VirtioFS is now much faster (bind mounts are only 3x slower instead of 5-6x), and we have new solutions in the ecosystem. Lima (open-source) performs well and sometimes better than Docker Desktop, while Docker’s new file synchronization feature offers impressive speed improvements (59% faster) but requires a paid subscription. Additionally, OrbStack has emerged as a strong contender, offering excellent performance with bind mounts and native operations. For the most stable performance, the hybrid approach (combining bind mounts with volumes) remains the best practice. Choose your setup based on your needs:

• Fast, stable, and open-source: Go with Lima.
• Maximum speed: Use Docker Desktop with file synchronization or OrbStack.
• Stable performance: Use the hybrid approach with volumes with any solution.

Introduction Link to heading

Two years ago I wrote this post Docker on MacOS is slow and how to fix it which had decent success; it was featured on the CNCF blog and trended on Hacker News.

Despite the unexpected success, the goal was to explain all the moving parts of Docker when running in a virtualized environment like MacOS or Windows. What I found was that in some circumstances, the performance was very poor, especially when using bind mounts, a common strategy when developing applications with Docker. At that time, I proposed some strategies, like using named volumes instead of bind mounts, or even using devcontainer to improve the developer experience.

Another technique I proposed was enabling the VirtioFS file sharing driver, which was still an experimental feature at that time but could improve the file system performance significantly.

Now, almost two years later, how have things changed? What is the current state of Docker on MacOS? Let’s find out together.

Linux Virtualization on MacOS Link to heading

Before diving into some benchmarks, let’s take a look at the current technologies that allow Docker to run on MacOS. We’ll explore the Virtualization Framework and the VIRTIO standard, which are used by almost all virtualization solutions on MacOS, including those specifically designed for containers like Docker for Desktop, Lima, Rancher Desktop, Podman Desktop, etc.

Virtualization Framework Link to heading

One of the key elements of Docker solutions on MacOS is the virtualization layer that allows running Linux Virtual Machines on MacOS. This layer is called Virtualization Framework and has been part of the Apple ecosystem since MacOS Big Sur, which was released in 2020 along with the first Apple Silicon Macs.

It is based on the Apple Hypervisor Framework but offers high-level APIs to manage virtual machines for MacOS and Linux. It also implements virtual devices, such as network interfaces and block devices, following the VIRTIO specifications. Another key feature is the ability to run x86_64 Linux binaries on Apple Silicon through Rosetta 2. This feature is particularly powerful as Rosetta is an extremely optimized translation layer, much faster than QEMU for this specific task and environment. Thanks to this integration, we can run linux/amd64 containers when the linux/arm64/v8 counterpart is unavailable.

VirtIO drivers Link to heading

The biggest decision Apple made with this framework was adopting the VIRTIO standard to expose the host hardware to guest VMs as VirtIO drivers. This means that on the Linux side, we can use the same drivers as other hypervisors like KVM or QEMU, which are well-optimized and maintained by the Linux community. On the MacOS side, the system leverages the VirtIO kernel drivers implemented by Apple.

The architecture looks more or less like this:

As shown in the diagram, the VIRTIO architecture comprises the following components:

1. Frontend: The driver running in the guest VM, responsible for communicating with the backend driver.
2. Backend: The driver running on the host, responsible for communicating with the frontend driver.
3. VirtIO queue: The communication channel between the frontend and backend drivers.

A real-world open-source example of this architecture can be found in QEMU. For instance, in virtio-net:

1. Frontend: https://github.com/qemu/qemu/blob/master/hw/net/virtio-net.c
2. Backend: https://github.com/qemu/qemu/blob/master/hw/net/virtio-net.c

Apple implemented the backend driver in the Virtualization Framework and exposed the VirtIO queues to guest VMs, allowing the Linux kernel to use VirtIO drivers to communicate with the host hardware.

As far as I know, the Apple VirtIO drivers are not open source. There are just a few references in the XNU open-source repository. It’s almost certain that everything is implemented in userspace, as confirmed by the currently exposed APIs: https://developer.apple.com/search/?q=virtio&type=Documentation.

Of course, I’ve simplified the architecture for clarity, but if you want to know more, I suggest reading the VIRTIO specification, the Virtio on Linux documentation, and the super-interesting RedHat article series (which co-developed the standard with IBM) Virtqueues and virtio ring: How the data travels.

libkrun and Docker VMM Link to heading

Although the Virtualization.framework is a nice library covering many use cases, it suffers from being a closed-source project that cannot be extended. This means only Apple can implement new features, such as new devices or optimizations.

Consider running an AI workload inside a container that requires GPU acceleration. Thanks to the virtio-gpu driver, we can expose the host GPU to the guest VM. However, this is not possible with the current Apple Virtualization.framework, as it only supports 2D acceleration, announced at WWDC 2022, with no further updates. Since it’s a closed-source project, no one but Apple can enhance it.

This is where a new project called libkrun comes into play. This library offers a modern, Rust-based Virtual Machine Monitor that directly integrates with the Hypervisor framework for creating Virtual Machines on MacOS. It is fully open source (Apache 2.0), allowing for extensive customization.

I can’t explain the project better than Sergio Lopez did in his blog post: “Enabling containers to access the GPU on macOS”. I highly recommend reading this super-interesting article.

Another project worth mentioning is Docker VMM, which, like libkrun, integrates with the low-level Hypervisor.framework to create and manage VMs on MacOS, instead of the native Virtualization.framework. The project is still in beta and is a closed-source project. From what the documentation says, its goal is to provide significant optimizations on both the Linux kernel and hypervisor sides.

From the official documentation:

Docker VMM brings exciting advancements specifically tailored for Apple Silicon machines. By optimizing both the Linux kernel and hypervisor layers, Docker VMM delivers significant performance enhancements across common developer tasks.

Some key performance enhancements provided by Docker VMM include:

• Faster I/O operations: With a cold cache, iterating over a large shared filesystem with find is 2x faster than when the Apple Virtualization Framework is used.

• Improved caching: With a warm cache, performance can improve by as much as 25x, even surpassing native Mac operations.

Even though the project is still in beta and closed-source, it’s worth benchmarking to see how it performs compared to the Apple Virtualization Framework.

Benchmarks Link to heading

In the previous article, I used this repository docker-for-mac-bench to test filesystem performance in various scenarios. It’s a simple project using a sample React application and runs a npm install. It’s straightforward but effective. We’ll reuse it now to compare the results with the previous ones.

Tests Link to heading

The tests that will be run are:

1. Native: Host machine without Docker.
2. Docker without volumes: A container without volumes.
3. Docker volume + mount: Bind mount ./create/react/app:/usr/src/app and a volume on nodemodules:/usr/src/app/node_modules.
4. Docker bind mount: Bind mount ./create-react-app:/usr/src/app.

This is another good excuse to make another Excalidraw diagram:

As explained in detail here, the core idea to keep in mind is that bind mounting is always the slowest and costliest option, as it requires HOST to VM file-system sharing, like VirtioFS.

System information Link to heading

To run the tests, I’ll use the following machines:

MacOS Link to heading

• OS: MacOS 15.1.1
• Processor: Apple M4 Pro 12 cores (8 performance and 4 efficiency)
• Memory: 48GB
• Storage: 512GB

Linux Link to heading

• OS: Archlinux (provisioned with this ansible playbook)
• Processor: ThreadRipper PRO 5945WX
• Memory: 64GB
• Storage: 1TB NVMe (Samsung 980 PRO) on BTRFS mounted with the following options noatime,compress=zstd:1

Container runtime platforms Link to heading

I’ll use the following container runtime platforms:

MacOS Link to heading

• Docker for Desktop: 4.37.1
  • Closed source and free for small businesses (fewer than 250 employees AND less than $10 million in annual revenue).
  • Apple Virtualization Framework + VirtioFS
  • Docker VMM + VirtioFS
  • Apple Virtualization Framework + VirtioFS + Synchronized file sharing
• Lima: 1.0.3
  • Open source (Apache 2.0)
  • Apple Virtualization Framework + VirtioFS
• OrbStack: 1.9.2
  • Closed source - Pricing and Licensing
  • As it’s a closed source project I can only guess that is based on Apple Virtualization Framework (I guess it because it supports Rosetta 2 out of the box). You can find more information here: https://docs.orbstack.dev/faq

Linux Link to heading

• Docker: 27.3.1

1

limactl create --name=default --vm-type=vz --mount-type=virtiofs --mount-writable --memory=16 --cpus=8 --disk=100 template://docker-desktop

1
2

docker context create lima-default --docker "host=unix:///Users/paolomainardi/.lima/default/sock/docker.sock"
docker context use lima-default

1
2
3
4
5

✉ docker context ls
NAME             DESCRIPTION                               DOCKER ENDPOINT                                              ERROR
default          Current DOCKER_HOST based configuration   unix:///var/run/docker.sock
desktop-linux    Docker Desktop                            unix:///Users/paolomainardi/.docker/run/docker.sock
lima-default *

I’m especially interested in the performance of Lima as it’s a CNCF Open Source project. In Sparkfabrik, we are still defaulting to Docker for Desktop, but we are evaluating switching to other solutions like Lima, to keep our stack as Open Source as possible.

Results Link to heading

Despite being limited to a single test scenario, this benchmark reveals several interesting patterns. Let’s analyze the key findings:

1. Linux Performance Consistency The most striking observation is the consistency of Docker performance on Linux. Regardless of the configuration (bind mounts, volumes, or native operations), execution times remain stable around 5 seconds, with minimal variations. This consistency is expected since no virtualization layer is involved.

1. Lima vs Docker Desktop Performance Interestingly, Lima outperforms standard Docker Desktop with Apple Virtualization Framework in bind mount operations, clocking at 8.99 seconds compared to 9.53 seconds. This difference, while modest, demonstrates Lima’s efficiency in handling filesystem operations and that it is already a viable alternative to Docker Desktop.

2. OrbStack’s Notable Performance Among the tested platforms, OrbStack stands out with its impressive performance, especially in scenarios involving bind mounts and native operations. With a bind mount time of 4.22 seconds, OrbStack outperforms both standard Docker Desktop and Lima in similar tests. This suggests that OrbStack leverages the Apple Virtualization Framework more efficiently, likely with additional optimizations for file-sharing mechanisms. Developers looking for a blend of speed and ease of use may find OrbStack to be a compelling option.

3. Bind Mounts Performance Impact As previously documented, bind mounts remain the slowest option, running approximately 3x slower than native operations. However, there’s notable progress: previous benchmarks showed bind mounts being 5-6x slower, indicating significant VirtIOFS performance improvements on MacOS.

4. File Synchronization Benefits The most dramatic improvement comes from Docker’s Apple Virtualization Framework with file synchronization enabled. This configuration reduces bind mount operation times to just 3.88 seconds—a 59% performance improvement over standard Docker-VZ. However, this feature is only available in Docker for Desktop’s paid version.

1
2
3

Testing: Native installation...
rm: node_modules: Permission denied
make: *** [test-native] Error 1

1. Hybrid Approach Stability The combination of bind mounts with volumes shows remarkably consistent performance across all configurations, with times ranging from 3.42 to 4.06 seconds. This stability makes it an attractive option for development environments where predictable performance is crucial.

Conclusions Link to heading

After two years from my first analysis, the Docker ecosystem on MacOS has significantly evolved. Here are the main takeaways from our benchmarks:

1. Performance Has Improved The VirtIOFS improvements are notable - bind mount operations are now only 3x slower than native operations, compared to 5-6x slower two years ago. This is a substantial improvement in daily development workflows.

2. New Solutions Are Emerging The introduction of Docker VMM, the maturity of Lima, and the strong performance of OrbStack show how the ecosystem is evolving. Lima and OrbStack prove to be viable alternatives to Docker Desktop, each excelling in different areas.

3. File Synchronization is Game-Changing Docker’s file synchronization feature shows impressive results, reducing bind mount operation times by 59%. However, being a paid feature, developers need to evaluate if the performance boost justifies the cost for their specific needs.

4. Best Practices Still Matter The hybrid approach (combining bind mounts with volumes) continues to provide the most consistent performance across all configurations. This reinforces our previous recommendations about using volumes when possible.

5. Platform Choice Matters While Docker on Linux shows consistent performance regardless of the configuration, MacOS users need to carefully consider their setup based on their specific needs:

• For hobby projects or small applications, any solution works fine
• For larger projects, either Docker with file synchronization or Lima could be the best choice
• Still the hybrid approach with volumes offers the most predictable performance.

Looking forward, we can expect further improvements in the MacOS Docker ecosystem, especially with new projects like Docker VMM and the continuous development of Lima and OrbStack. The gap between native Linux performance and MacOS virtualized environments continues to narrow, making Docker on MacOS an increasingly viable option for development workflows.

Thanks for reading all of this, and if you find something wrong or want to discuss some topics further, get in touch with me. As always you can leave your comments here: https://github.com/paolomainardi/paolomainardi.com/discussions/38 or reach me on the contacts you can find here.

Data and diagrams Link to heading

1. Excalidraw diagrams (read-only link): https://excalidraw.com/#json=FD3dYVkiXvQ9xHeZ2Pgfy,5tVt2hukDYlki3qwzMI15w
2. Benchmark results: data.csv

References Link to heading

Here a list of some very random references I’ve collected during my research:

1. What makes Docker VMM better than Apple Virtualization Framework?
2. Docker VMM
3. REDHAT - vfkit - A macOS hypervisor using Apple’s virtualization framework
4. vz - Go binding with Apple Virtualization.framework
5. vfkit - Simple command line tool to start VMs through the macOS Virtualization framework
6. What Are the Latest Docker Desktop Enterprise-Grade Performance Optimizations?
7. Tart is a virtualization toolset to build, run and manage macOS and Linux virtual machines on Apple Silicon
8. Podman 5.2 Enhances macOS VMs with GPU Support
9. Introduction to VirtIO
10. Virgl not functionning under Apple’s virtualization.framework
11. macOS Docker Provider Performance, November 2023