Nested variable interpolation in compose files is not working #4265
Description
Description
Interpolation of nested variables in compose files is not supported/bugged.
Reproduce
- take the following compose file:
services:
my-service:
image: "alpine"
command: ["echo", "${HELLO:-${UNUSED}}"]- Init swarm
docker swarm init - Deploy a stack with this compose file:
HELLO=hello docker stack deploy --compose-file docker-compose.yml foo - Show the logs of the service:
docker service logs foo_my-service
The logs print hello}, meaning that the variable wasn't properly interpolated.
Note that as per compose specification, this should be allowed and is not undefined behavior.
Expected behavior
Either interpolate to hello, just like Compose does, or at least output an error such as nested variables interpolation is not supported. Currently, the interpolation is incorrect which is confusing and surprising to the user.
docker version
Client:
Version: 23.0.5
API version: 1.42
Go version: go1.20.4
Git commit: bc4487a59e
Built: Tue May 2 21:53:09 2023
OS/Arch: linux/amd64
Context: default
Server:
Engine:
Version: 23.0.4
API version: 1.42 (minimum version 1.12)
Go version: go1.20.3
Git commit: cbce331930
Built: Fri Apr 21 22:05:37 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.7.0
GitCommit: 1fbd70374134b891f97ce19c70b6e50c7b9f4e0d.m
runc:
Version: 1.1.7
GitCommit:
docker-init:
Version: 0.19.0
GitCommit: de40ad0docker info
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: 0.10.4
Path: /usr/lib/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: 2.17.3
Path: /usr/lib/docker/cli-plugins/docker-compose
Server:
Containers: 33
Running: 3
Paused: 0
Stopped: 30
Images: 276
Server Version: 23.0.4
Storage Driver: btrfs
Btrfs:
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 1fbd70374134b891f97ce19c70b6e50c7b9f4e0d.m
runc version:
init version: de40ad0
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.2.12-arch1-1
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.34GiB
Name: WKS-79LR7G3
ID: ARE2:H3PX:4QCX:J6R2:Q4CM:2BYH:EIWM:3YRO:ZISS:XTT2:4B5Y:HLIB
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: falseAdditional Info
The problem originates from the fact that the parser is regex based, which is not well suited for recursive parsing.
One solution could to write a custom tokenizer/parser, produce an AST and all that jazz. However, Compose already has a working implementation in compose-spec/compose-go. It can be used as a drop-in replacement for the custom template module used by this project. I don't know if this solution is acceptable, but if it is, I can draft a PR.
Here is a minimal patch that can make this work:
fix_interpolation.patch
diff --git a/cli/compose/interpolation/interpolation.go b/cli/compose/interpolation/interpolation.go
index e84756dba6..183ff6273b 100644
--- a/cli/compose/interpolation/interpolation.go
+++ b/cli/compose/interpolation/interpolation.go
@@ -4,7 +4,7 @@ import (
"os"
"strings"
- "github.com/docker/cli/cli/compose/template"
+ "github.com/compose-spec/compose-go/template"
"github.com/pkg/errors"
)
diff --git a/cli/compose/interpolation/interpolation_test.go b/cli/compose/interpolation/interpolation_test.go
index 069d8d2549..f8adfe419b 100644
--- a/cli/compose/interpolation/interpolation_test.go
+++ b/cli/compose/interpolation/interpolation_test.go
@@ -11,6 +11,7 @@ import (
var defaults = map[string]string{
"USER": "jenny",
"FOO": "bar",
+ "CMD": "my-cmd",
"count": "5",
}
@@ -22,7 +23,11 @@ func defaultMapping(name string) (string, bool) {
func TestInterpolate(t *testing.T) {
services := map[string]interface{}{
"servicea": map[string]interface{}{
- "image": "example:${USER}",
+ "image": "example:${USER}",
+ "command": []interface{}{
+ "${CMD:-${UNDEF_CMD:-}}",
+ "${UNDEF:-${CMD}}",
+ },
"volumes": []interface{}{"$FOO:/target"},
"logging": map[string]interface{}{
"driver": "${FOO}",
@@ -36,6 +41,7 @@ func TestInterpolate(t *testing.T) {
"servicea": map[string]interface{}{
"image": "example:jenny",
"volumes": []interface{}{"bar:/target"},
+ "command": []interface{}{"my-cmd", "my-cmd"},
"logging": map[string]interface{}{
"driver": "bar",
"options": map[string]interface{}{
diff --git a/cli/compose/loader/loader.go b/cli/compose/loader/loader.go
index a9b8f1f1f9..bde9efbf99 100644
--- a/cli/compose/loader/loader.go
+++ b/cli/compose/loader/loader.go
@@ -9,9 +9,10 @@ import (
"strings"
"time"
interp "github.com/docker/cli/cli/compose/interpolation"
"github.com/docker/cli/cli/compose/schema"
- "github.com/docker/cli/cli/compose/template"
+ "github.com/compose-spec/compose-go/template"
"github.com/docker/cli/cli/compose/types"
"github.com/docker/cli/opts"
"github.com/docker/docker/api/types/versions"As you can see, there's not a lot to it.