Kubernetes Control Plane

In order to run container workloads, you will need a Kubernetes cluster. While it is possible to provision and manage a cluster manually on Azure, their managed offering, Azure Kubernetes Service (AKS), offers an easier way to get up and running.

See the official Kubernetes docs for more details.

The full code for this stack is on GitHub.

In order to run container workloads, you will need a Kubernetes cluster. While it is possible to provision and manage a cluster manually on Google Cloud, their managed offering, Google Kubernetes Engine (GKE), offers an easier way to get up and running.

See the official Kubernetes docs for more details.

The full code for this stack is on GitHub.

Azure Kubernetes Service can be configured to use Azure Active Directory (Azure AD) for user authentication. Cluster administrators can then configure Kubernetes role-based access control (RBAC) based on a user’s identity or directory group membership.

To provide Azure AD authentication for an AKS cluster, two Azure AD applications are created. The first application is a server component that provides user authentication. The second application is a client component that uses the server application for the actual authentication of the credentials provided by the client.

We configure applications and service principals using the @pulumi/azuread package. After the applications are created, there is manual step required to grant admin consent for API permissions.

For users, we create and use a ServicePrincipal for cluster administrators with root privileges, and a limited scope devs user group for general purpose execution of workloads. Both identities will be tied into Kubernetes RBAC in Configure Access Control.

We configure the principal identities using servicePrincipal to create the cluster, and set up roleBasedAccessControl to manage authentication into the cluster.

import * as azure from "@pulumi/azure";

// Create the AKS cluster with IAM.
const cluster = new azure.containerservice.KubernetesCluster(`${name}`, {
    resourceGroupName: config.resourceGroupName,
    servicePrincipal: {
        clientId: config.adClientAppId,
        clientSecret: config.adClientAppSecret,
    },
    roleBasedAccessControl: {
        enabled: true,
        azureActiveDirectory: {
            clientAppId: config.adClientAppId,
            serverAppId: config.adServerAppId,
            serverAppSecret: config.adServerAppSecret,
        },
    },
    ...
}

For users, we create and use a ServiceAccount for cluster administrators with root privileges, and a limited scope devs ServiceAccount for general purpose execution of workloads. Both identities will be tied into Kubernetes RBAC in Configure Access Control.

By authenticating with the ServiceAccount using gcloud, as outlined in Identity, we automatically bind the ServiceAccount to be a cluster admin and no further action is required.

How you create the network will vary on your permissions and preferences.

Typical setups will provide Kubernetes with the following resources to use for the cluster.

• Private subnets for provisioning private load balancers.
• Private subnets for use as the default subnets for workers to run in.
• Managed Pod networking.

By default, pulumi/azure will deploy workers into the private subnets without associating a public IP address. This creates workers that will not be publicly accessible from the Internet, and they’ll typically be shielded within your network.

AKS will manage Kubernetes Pod networking for us through the use of the AKS CNI Plugin. This plugin is deployed by default on worker nodes as a DaemonSet named azure-cni-networkmonitor in all clusters provisioned with pulumi/azure.

import * as azure from "@pulumi/azure";

// Create a Virtual Network for the cluster.
const vnet = new azure.network.VirtualNetwork(name, {
    resourceGroupName: config.resourceGroupName,
    addressSpaces: ["10.2.0.0/16"],
});

// Create a Subnet for the cluster.
const subnet = new azure.network.Subnet(name, {
    resourceGroupName: config.resourceGroupName,
    virtualNetworkName: vnet.name,
    addressPrefix: "10.2.1.0/24",
});

How you create the network will vary on your permissions and preferences.

Typical setups will provide Kubernetes with the following resources to use for the cluster.

• Private subnets for use as the default subnets for workers to run in.
• Managed Pod networking.

By default, pulumi/gcp will deploy workers into the private subnets without associating an external IP address. This creates workers that will not be publicly accessible from the Internet, and they’ll typically be shielded within your network.

GKE will manage Kubernetes Pod networking for us through the use of Alias IPs to address and route pods within a Google Cloud network.

// Create a new network.
const network = new gcp.compute.Network(projectName, {
    autoCreateSubnetworks: false,
});
export const networkName = network.name;

// Create a new subnet.
const subnet = new gcp.compute.Subnetwork(projectName, {
    ipCidrRange: "10.0.0.0/24",
    network: network.name,
    secondaryIpRanges: [{ rangeName: "pods", ipCidrRange: "10.1.0.0/16" }],
});
export const subnetworkName = subnet.name;

See the official AKS docs for more details.

cat > storage-classes.yaml << EOF
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: managed-premium-retain
provisioner: kubernetes.io/azure-disk
parameters:
  storageaccounttype: Premium_LRS
  kind: Managed
EOF

$ kubectl apply -f storage-classes.yaml

cat > pvc.yaml << EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mypvc
  labels:
    app: myapp
spec:
  storageClassName: managed-premium-retain
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
EOF

$ kubectl apply -f pvc.yaml

import * as k8s from "@pulumi/k8s";

// Create the premium StorageClass.
const sc = new k8s.storage.v1.StorageClass("premium",
    {
        provisioner: "kubernetes.io/azure-disk",
        parameters: {
            "storageaccounttype": "Premium_LRS",
            "kind": "Managed"
        },
    },
    { provider: provider }
);

import * as k8s from "@pulumi/k8s";

// Create a Persistent Volume Claim on the StorageClass.
const myPvc = new k8s.core.v1.PersistentVolumeClaim("mypvc", {
    spec: {
        accessModes: ["ReadWriteOnce"],
        storageClassName: sc.metadata.name,
        resources: {requests: {storage: "1Gi"}}
    }
},
    { provider: provider }
);

See the official GKE docs for more details.

cat > storage-classes.yaml << EOF
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: slow
provisioner: kubernetes.io/gce-pd
parameters:
  type: "pd-standard"
  replication-type: "none"
EOF

$ kubectl apply -f storage-classes.yaml

cat > pvc.yaml << EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mypvc
  labels:
    app: myapp
spec:
  storageClassName: slow
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
EOF

$ kubectl apply -f pvc.yaml

import * as k8s from "@pulumi/k8s";

// Create the standard StorageClass.
const sc = new k8s.storage.v1.StorageClass("standard",
    {
        provisioner: "kubernetes.io/gce-pd",
        parameters: {
            "type": "pd-standard",
            "replication-type": "none"
        },
    },
    { provider: provider }
);

import * as k8s from "@pulumi/k8s";

// Create a Persistent Volume Claim on the StorageClass.
const myPvc = new k8s.core.v1.PersistentVolumeClaim("mypvc", {
        spec: {
            accessModes: ["ReadWriteOnce"],
            storageClassName: sc.metadata.name,
            resources: {requests: {storage: "1Gi"}}
        }
    },
    { provider: provider }
);

AKS:

• Enable PodSecurityPolicies using enablePodSecurityPolicy: true

• Set Node Labels to identify nodes by attributes

• Enable Log Analytics using the omsAgent setting

  import * as azure from "@pulumi/azure";
  
  const cluster = new azure.containerservice.KubernetesCluster(`${name}`, {
          ...
          enablePodSecurityPolicy: true,
          kubernetesVersion: "1.14.8",
          addonProfile: {
              omsAgent: {
                  enabled: true,
                  logAnalyticsWorkspaceId: config.logAnalyticsWorkspaceId,
              },
          },
  });
  

  Copy

GKE:

• Enable PodSecurityPolicies using podSecurityPolicyConfig: { enabled: true }

• Skip enabling the default node group in favor of managing them separately from the control plane, as demonstrated in Create the Worker Nodes.

• Disable legacy metadata APIs that are not v1 and do not enforce internal Google Cloud metadata headers

• Enable control plane logging and monitoring through oauthScopes to have diagnostics of the control plane’s actions, and for use in debugging and auditing.

• (Optional) Configure private accessibility of the control plane / API Server endpoint to prevent it from being publicly exposed on the Internet. To enable this feature, additional networking is required, and a bastion host would be needed to access the control plane.

  import * as gcp from "@pulumi/gcp";
  
  const cluster = new gcp.container.Cluster("cluster", {
          ...
          minMasterVersion: "1.14.7-gke.10",
          podSecurityPolicyConfig: { enabled: true },
          nodeConfig: {
              // We can't create a cluster without a node pool defined, but we want to
              // only use separately managed node pools. So we create the smallest
              // possible default node pool and immediately delete it.
              removeDefaultNodePool: true,
              initialNodeCount: 1,
              metadata: {
                  "disable-legacy-endpoints": "true",
              },
              oauthScopes: [
                  "https://www.googleapis.com/auth/logging.write",
                  "https://www.googleapis.com/auth/monitoring",
              ],
          },
  });
  

  Copy