KinD Kubernetes CSI Container Storage Interface Azurefile Azure Azure File CSI Driver

Kubernetes使用Azure File CSI Driver掛載Azure Files

Posted by Polin on Sun, Nov 17, 2024

Kubernetes使用Azure File CSI Driver掛載Azure Files

Azure File CSI Driver 是 Kubernetes 上一種用於掛載 Azure Files 的解決方案，能夠在容器中提供簡單、高效且可擴展的文件存儲功能。透過 CSI Driver，使用者可以利用 Kubernetes 的 PersistentVolume (PV) 和 PersistentVolumeClaim (PVC) 動態或靜態地配置 Azure Files，共享存儲資源，實現跨容器和跨 Pod 的數據共享，並支持企業級存儲需求，如高可用性、備份及權限控制等功能。

先決條件

在開始之前，請確保您的 macOS 環境已經具備以下軟體：

  1. Homebrew：macOS 的套件管理工具，用來安裝其他必要軟體。

  2. KinD: 一種基於 Docker 的本地 Kubernetes 集群工具，用於在本地開發與測試 Kubernetes 工作負載。 可以透過 Homebrew 安裝：

    brew install kind

    更多詳細說明可參考 在本機用 KinD 建立 Kubernetes。

  3. Helm: Kubernetes 的應用程式包管理工具，用於簡化應用部署與管理。可以透過 Homebrew 安裝

    brew install helm

準備 Azure Files 資源

登入 Azure Portal 或使用 Azure CLI 建立一個 Azure Storage Account。 在 Storage Account 中建立一個 Azure Files 檔案共享。

建立 Storage Account

az storage account create --name <storage-account-name> \
  --resource-group <resource-group> \
  --location <location> \
  --sku Standard_LRS
  • storage-account-name: 指定建立的Storage Account名稱
  • resource-group: 指定Storage Account存在的Resource Group
  • location: 指定Storage Account位於的地區

範例：

az storage account create --name stdemo20241117 --resource-group rg-demo-dev-01  --location japaneast  --sku Standard_LRS

指令執行成功後會顯示以下類似訊息

{
  "accessTier": "Hot",
  "accountMigrationInProgress": null,
  "allowBlobPublicAccess": false,
  "allowCrossTenantReplication": false,
  "allowSharedKeyAccess": null,
  "allowedCopyScope": null,
  "azureFilesIdentityBasedAuthentication": null,
  "blobRestoreStatus": null,
  "creationTime": "2024-11-17T12:24:10.186484+00:00",
  "customDomain": null,
  "defaultToOAuthAuthentication": null,
  "dnsEndpointType": null,
  "enableExtendedGroups": null,
  "enableHttpsTrafficOnly": true,
  "enableNfsV3": null,
  "encryption": {
    "encryptionIdentity": null,
    "keySource": "Microsoft.Storage",
    "keyVaultProperties": null,
    "requireInfrastructureEncryption": null,
    "services": {
      "blob": {
        "enabled": true,
        "keyType": "Account",
        "lastEnabledTime": "2024-11-17T12:24:10.264618+00:00"
      },
      "file": {
        "enabled": true,
        "keyType": "Account",
        "lastEnabledTime": "2024-11-17T12:24:10.264618+00:00"
      },
      "queue": null,
      "table": null
    }
  },
  "extendedLocation": null,
  "failoverInProgress": null,
  "geoReplicationStats": null,
  "id": "/subscriptions/5393ecef-XXXX-XXXX-XXXX-8aa2c09d666b/resourceGroups/rg-demo-dev-01/providers/Microsoft.Storage/storageAccounts/stdemo20241117",
  "identity": null,
  "immutableStorageWithVersioning": null,
  "isHnsEnabled": null,
  "isLocalUserEnabled": null,
  "isSftpEnabled": null,
  "isSkuConversionBlocked": null,
  "keyCreationTime": {
    "key1": "2024-11-17T12:24:10.264618+00:00",
    "key2": "2024-11-17T12:24:10.264618+00:00"
  },
  "keyPolicy": null,
  "kind": "StorageV2",
  "largeFileSharesState": null,
  "lastGeoFailoverTime": null,
  "location": "japaneast",
  "minimumTlsVersion": "TLS1_0",
  "name": "stdemo20241117",
  "networkRuleSet": {
    "bypass": "AzureServices",
    "defaultAction": "Allow",
    "ipRules": [],
    "ipv6Rules": [],
    "resourceAccessRules": null,
    "virtualNetworkRules": []
  },
  "primaryEndpoints": {
    "blob": "https://stdemo20241117.blob.core.windows.net/",
    "dfs": "https://stdemo20241117.dfs.core.windows.net/",
    "file": "https://stdemo20241117.file.core.windows.net/",
    "internetEndpoints": null,
    "microsoftEndpoints": null,
    "queue": "https://stdemo20241117.queue.core.windows.net/",
    "table": "https://stdemo20241117.table.core.windows.net/",
    "web": "https://stdemo20241117.z11.web.core.windows.net/"
  },
  "primaryLocation": "japaneast",
  "privateEndpointConnections": [],
  "provisioningState": "Succeeded",
  "publicNetworkAccess": null,
  "resourceGroup": "rg-demo-dev-01",
  "routingPreference": null,
  "sasPolicy": null,
  "secondaryEndpoints": null,
  "secondaryLocation": null,
  "sku": {
    "name": "Standard_LRS",
    "tier": "Standard"
  },
  "statusOfPrimary": "available",
  "statusOfSecondary": null,
  "storageAccountSkuConversionStatus": null,
  "tags": {},
  "type": "Microsoft.Storage/storageAccounts"
}

建立 Azure File Shares

az storage share create --account-name <storage-account-name> --name <share-name>
  • storage-account-name: Storage Account 名稱
  • share-name: Share 名稱

範例：

az storage share create --account-name stdemo20241117 --name share-demo20241117

指令執行成功後會顯示以下類似訊息

{
  "created": true
}

建立 Microsoft Entra ID 應用程式帳戶

1. 登入 Azure Portal

確保您已登入 Azure CLI，若尚未登入，使用以下指令：

az login

2. 建立應用程式

使用以下指令來註冊應用程式：

az ad app create --display-name <application-name>
  • application-name: 應用程式名稱。

範例：

az ad app create --display-name "k8s-azurefile-app"

指令執行成功後會顯示以下類似訊息

{
  "@odata.context": "https://graph.microsoft.com/v1.0/$metadata#applications/$entity",
  "addIns": [],
  "api": {
    "acceptMappedClaims": null,
    "knownClientApplications": [],
    "oauth2PermissionScopes": [],
    "preAuthorizedApplications": [],
    "requestedAccessTokenVersion": null
  },
  "appId": "d2cf9ec9-XXXX-XXXX-XXXX-ab4b87e3411f",
  "appRoles": [],
  "applicationTemplateId": null,
  "certification": null,
  "createdDateTime": "2024-11-17T12:09:40.7297318Z",
  "defaultRedirectUri": null,
  "deletedDateTime": null,
  "description": null,
  "disabledByMicrosoftStatus": null,
  "displayName": "k8s-azurefile-app",
  "groupMembershipClaims": null,
  "id": "901a3026-XXXX-XXXX-XXXX-dc2ab32fada2",
  "identifierUris": [],
  "info": {
    "logoUrl": null,
    "marketingUrl": null,
    "privacyStatementUrl": null,
    "supportUrl": null,
    "termsOfServiceUrl": null
  },
  "isDeviceOnlyAuthSupported": null,
  "isFallbackPublicClient": null,
  "keyCredentials": [],
  "nativeAuthenticationApisEnabled": null,
  "notes": null,
  "optionalClaims": null,
  "parentalControlSettings": {
    "countriesBlockedForMinors": [],
    "legalAgeGroupRule": "Allow"
  },
  "passwordCredentials": [],
  "publicClient": {
    "redirectUris": []
  },
  "publisherDomain": "xxxxxxx.onmicrosoft.com",
  "requestSignatureVerification": null,
  "requiredResourceAccess": [],
  "samlMetadataUrl": null,
  "serviceManagementReference": null,
  "servicePrincipalLockConfiguration": null,
  "signInAudience": "AzureADMyOrg",
  "spa": {
    "redirectUris": []
  },
  "tags": [],
  "tokenEncryptionKeyId": null,
  "uniqueName": null,
  "verifiedPublisher": {
    "addedDateTime": null,
    "displayName": null,
    "verifiedPublisherId": null
  },
  "web": {
    "homePageUrl": null,
    "implicitGrantSettings": {
      "enableAccessTokenIssuance": false,
      "enableIdTokenIssuance": false
    },
    "logoutUrl": null,
    "redirectUriSettings": [],
    "redirectUris": []
  }
}

記錄以下兩項：

  1. appId (應用程式 ID / 用戶端 ID)

    "appId": "d2cf9ec9-XXXX-XXXX-XXXX-ab4b87e3411f",

  2. id (物件 ID)

    "id": "901a3026-XXXX-XXXX-XXXX-dc2ab32fada2",

3. 建立應用程式密碼

生成用戶端密碼 (應用程式金鑰)：

az ad app credential reset --id <appId> --append --display-name "k8s-secret" --years 1
  • appId: 替換為前一步中的應用程式 ID。
  • –years 1 表示密碼有效期為 1 年，可根據需求調整。

範例：

az ad app credential reset --id d2cf9ec9-XXXX-XXXX-XXXX-ab4b87e3411f --append --display-name "k8s-secret" --years 1

指令執行成功後，會輸出一組新的密碼值 (即應用程式密碼)，請妥善記錄。

{
  "appId": "d2cf9ec9-XXXX-XXXX-XXXX-ab4b87e3411f",
  "password": "SZW8Q~.........Owx8NvdxC",
  "tenant": "645d4b1e-XXXX-XXXX-XXXX-21b9fa967537"
}

4. 建立服務主體 (Service Principal)

應用程式本身無法直接使用，需為其建立一個服務主體：

az ad sp create --id <appId>
  • appId: 替換為前一步中的應用程式 ID。

範例：

az ad sp create --id d2cf9ec9-XXXX-XXXX-XXXX-ab4b87e3411f

指令執行成功後會顯示以下類似訊息

{
  "@odata.context": "https://graph.microsoft.com/v1.0/$metadata#servicePrincipals/$entity",
  "accountEnabled": true,
  "addIns": [],
  "alternativeNames": [],
  "appDescription": null,
  "appDisplayName": "k8s-azurefile-app",
  "appId": "d2cf9ec9-XXXX-XXXX-XXXX-ab4b87e3411f",
  "appOwnerOrganizationId": "645d4b1e-XXXX-XXXX-XXXX-21b9fa967537",
  "appRoleAssignmentRequired": false,
  "appRoles": [],
  "applicationTemplateId": null,
  "createdDateTime": "2024-11-17T12:35:24Z",
  "deletedDateTime": null,
  "description": null,
  "disabledByMicrosoftStatus": null,
  "displayName": "k8s-azurefile-app",
  "homepage": null,
  "id": "47489db9-XXXX-XXXX-XXXX-1e65ac1c1fa1",
  "info": {
    "logoUrl": null,
    "marketingUrl": null,
    "privacyStatementUrl": null,
    "supportUrl": null,
    "termsOfServiceUrl": null
  },
  "keyCredentials": [],
  "loginUrl": null,
  "logoutUrl": null,
  "notes": null,
  "notificationEmailAddresses": [],
  "oauth2PermissionScopes": [],
  "passwordCredentials": [],
  "preferredSingleSignOnMode": null,
  "preferredTokenSigningKeyThumbprint": null,
  "replyUrls": [],
  "resourceSpecificApplicationPermissions": [],
  "samlSingleSignOnSettings": null,
  "servicePrincipalNames": [
    "d2cf9ec9-XXXX-XXXX-XXXX-ab4b87e3411f"
  ],
  "servicePrincipalType": "Application",
  "signInAudience": "AzureADMyOrg",
  "tags": [],
  "tokenEncryptionKeyId": null,
  "verifiedPublisher": {
    "addedDateTime": null,
    "displayName": null,
    "verifiedPublisherId": null
  }
}

5. 分配存取權限

將服務主體指派至 Storage Account 的存取角色：

az role assignment create --assignee <appId> --role "Storage Account Contributor" --scope /subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account-name>
  • appId: 替換為應用程式 ID。
  • subscription-id: 替換為您的訂閱 ID。
  • resource-group: 替換為先前建立的Storage Account存在的Resource Group
  • storage-account-name: 替換為先前建立的Storage Account名稱

範例：

az role assignment create --assignee d2cf9ec9-XXXX-XXXX-XXXX-ab4b87e3411f --role "Storage File Data SMB Share Contributor" --scope /subscriptions/5393ecef-XXXX-XXXX-XXXX-8aa2c09d666b/resourceGroups/rg-demo-dev-01/providers/Microsoft.Storage/storageAccounts/stdemo20241117

指令執行成功後會顯示以下類似訊息

{
  "condition": null,
  "conditionVersion": null,
  "createdBy": null,
  "createdOn": "2024-11-17T12:41:50.650679+00:00",
  "delegatedManagedIdentityResourceId": null,
  "description": null,
  "id": "/subscriptions/5393ecef-XXXX-XXXX-XXXX-8aa2c09d666b/resourceGroups/rg-demo-dev-01/providers/Microsoft.Storage/storageAccounts/stdemo20241117/providers/Microsoft.Authorization/roleAssignments/4b26cf40-XXXX-XXXX-XXXX-259471832bfe",
  "name": "4b26cf40-XXXX-XXXX-XXXX-259471832bfe",
  "principalId": "47489db9-XXXX-XXXX-XXXX-1e65ac1c1fa1",
  "principalType": "ServicePrincipal",
  "resourceGroup": "rg-demo-dev-01",
  "roleDefinitionId": "/subscriptions/5393ecef-XXXX-XXXX-XXXX-8aa2c09d666b/providers/Microsoft.Authorization/roleDefinitions/0c867c2a-1d8c-454a-a3db-ab2ea1bdc8bb",
  "scope": "/subscriptions/5393ecef-XXXX-XXXX-XXXX-8aa2c09d666b/resourceGroups/rg-demo-dev-01/providers/Microsoft.Storage/storageAccounts/stdemo20241117",
  "type": "Microsoft.Authorization/roleAssignments",
  "updatedBy": "3e515418-XXXX-XXXX-XXXX-f24259c13744",
  "updatedOn": "2024-11-17T12:41:51.025688+00:00"
}

建立 Kubernetes Secret

為了讓 Kubernetes 認識 Azure Files，需建立一個 Secret 包含存取金鑰。

STORAGE_KEY=$(az storage account keys list --resource-group <resource-group> --account-name <storage-account-name> --query "[0].value" -o tsv)
kubectl create secret generic azure-secret --from-literal=azurestorageaccountname=<storage-account-name> --from-literal=azurestorageaccountkey=$STORAGE_KEY

範例：

STORAGE_KEY=$(az storage account keys list --resource-group rg-demo-dev-01 --account-name stdemo20241117 --query "[0].value" -o tsv)
kubectl create secret generic azure-secret --from-literal=azurestorageaccountname=stdemo20241117 --from-literal=azurestorageaccountkey=$STORAGE_KEY

指令執行成功後會顯示以下類似訊息

secret/azure-secret created

cat > azure.json <<EOF
{
  "cloud": "AzureCloud",
  "tenantId": "645d4b1e-XXXX-XXXX-XXXX-21b9fa967537 ",
  "subscriptionId": "5393ecef-XXXX-XXXX-XXXX-8aa2c09d666b",
  "aadClientId": "d2cf9ec9-XXXX-XXXX-XXXX-ab4b87e3411f",
  "aadClientSecret": "SZW8Q~.........Owx8NvdxC",
  "resourceGroup": "rg-demo-dev-01",
  "location": "JapanEast"
}
EOF

kubectl create secret generic azure-cloud-provider --from-file=cloud-config=\azure.json -n kube-system

安裝 Azure File CSI Driver

在 Kubernetes 中安裝 Azure File CSI Driver，可以透過 Helm Chart 簡化部署流程。

以下是安裝步驟：

1. 新增 Azure File CSI Driver 儲存庫：

首先，將 Azure File CSI Driver 的 Helm Chart 儲存庫加入本地 Helm 設定：

helm repo add azurefile-csi-driver https://raw.githubusercontent.com/kubernetes-sigs/azurefile-csi-driver/master/charts

指令執行成功後會顯示以下類似訊息

"azurefile-csi-driver" has been added to your repositories

2. 更新 Helm Chart 儲存庫

確保您擁有最新的 Chart 資料：

helm repo update azurefile-csi-driver

指令執行成功後會顯示以下類似訊息

Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "azurefile-csi-driver" chart repository
Update Complete. ⎈Happy Helming!⎈

3. 安裝 Azure File CSI Driver

使用 Helm Chart 安裝 Vault 至指定的命名空間，範例 vault：

helm install azurefile-csi-driver azurefile-csi-driver/azurefile-csi-driver \
  --namespace kube-system \
  --version v1.31.0

指令執行成功後會顯示以下類似訊息

NAME: azurefile-csi-driver
LAST DEPLOYED: Sun Nov 17 20:59:52 2024
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Azure File CSI Driver is getting deployed to your cluster.

To check Azure File CSI Driver pods status, please run:

  kubectl --namespace=kube-system get pods --selector="app.kubernetes.io/name=azurefile-csi-driver" --watch

4. 驗證部署

確認 Azure File CSI Driver Pod 是否成功啟動：

檢查 csi-azurefile-node

kubectl -n kube-system get pod -l app=csi-azurefile-node

指令執行成功後會顯示以下類似訊息

NAME                       READY   STATUS    RESTARTS        AGE
csi-azurefile-node-jbhmq   3/3     Running   1 (2m45s ago)   4m18s

檢查 csi-azurefile-controller

kubectl -n kube-system get pod -l app=csi-azurefile-controller

指令執行成功後會顯示以下類似訊息

NAME                                       READY   STATUS    RESTARTS       AGE
csi-azurefile-controller-78b7488ff-tn9xc   5/5     Running   2 (113s ago)   4m7s
csi-azurefile-controller-78b7488ff-tsqbz   0/5     Pending   0              4m7s

因為我們只有開一個 Node，所以有一個Pod會是處於Pending狀態，實屬正常
kubectl -n kube-system scale deploy/csi-azurefile-controller --replicas 1
可以使用該指令將Pod調整為1

建立 StorageClass

cat > storageclass.yaml <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azurefile
provisioner: file.csi.azure.com
allowVolumeExpansion: true
parameters:
  skuName: Standard_LRS
EOF

套用此配置：

kubectl apply -f storageclass.yaml

指令執行成功後會顯示以下類似訊息

storageclass.storage.k8s.io/azurefile created

定義 PersistentVolume (PV)

建立 PV 以配置存儲：

cat > pv.yaml <<EOF
apiVersion: v1
kind: PersistentVolume
metadata:
  name: azurefile-pv
spec:
  capacity:
    storage: 3Gi
  accessModes:
    - ReadWriteMany
  storageClassName: azurefile
  csi:
    driver: file.csi.azure.com
    volumeHandle: "rg-demo-dev-01#stdemo20241117#share-demo20241117"  # make sure this volumeid is unique for every identical share in the cluster
    volumeAttributes:
      resourceGroup: rg-demo-dev-01  # optional, only set this when storage account is not in the same resource group as node
      shareName: share-demo20241117
  claimRef:
    kind: PersistentVolumeClaim
    namespace: default
    name: azurefile-pvc
    apiVersion: v1
  mountOptions:
    - dir_mode=0777
    - file_mode=0777
    - uid=0
    - gid=0
    - mfsymlinks
    - cache=strict
    - actimeo=30
    - noperm

套用此配置：

kubectl apply -f pv.yaml

指令執行成功後會顯示以下類似訊息

persistentvolume/azurefile-pv created

定義 PersistentVolumeClaim (PVC)

建立 PVC 以靜態配置存儲：

cat > pvc.yaml <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: azurefile-pvc
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: azurefile
  volumeName: azurefile-pv
  resources:
    requests:
      storage: 5Gi
EOF

套用此配置：

kubectl apply -f pvc.yaml

指令執行成功後會顯示以下類似訊息

persistentvolumeclaim/azurefile-pvc created

掛載到 Pod

修改 Pod 的定義檔案，將 PVC 掛載為 Volume。

cat > pod.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: azurefile-pod
spec:
  containers:
  - name: webapp
    image: polinhou/golang-http-headers:latest
    volumeMounts:
    - name: azurefile-volume
      mountPath: /mnt/azure
  volumes:
  - name: azurefile-volume
    persistentVolumeClaim:
      claimName: azurefile-pvc
EOF

套用此配置：

kubectl apply -f pod.yaml

指令執行成功後會顯示以下類似訊息

pod/azurefile-pod created

驗證掛載

1. 在 Pod 中建立檔案

kubectl exec -ti azurefile-pod -- df -h

指令執行成功後會顯示以下類似訊息

Filesystem                Size      Used Available Use% Mounted on
overlay                  39.1G     34.3G      2.8G  93% /
tmpfs                    64.0M         0     64.0M   0% /dev
//stdemo20241117.file.core.windows.net/share-demo20241117
                          5.0G         0      5.0G   0% /mnt/azure
/dev/vda1                39.1G     34.3G      2.8G  93% /etc/hosts
/dev/vda1                39.1G     34.3G      2.8G  93% /dev/termination-log
/dev/vda1                39.1G     34.3G      2.8G  93% /etc/hostname
/dev/vda1                39.1G     34.3G      2.8G  93% /etc/resolv.conf
shm                      64.0M         0     64.0M   0% /dev/shm
tmpfs                     3.8G     12.0K      3.8G   0% /run/secrets/kubernetes.io/serviceaccount
tmpfs                     1.9G         0      1.9G   0% /proc/acpi
tmpfs                    64.0M         0     64.0M   0% /proc/kcore
tmpfs                    64.0M         0     64.0M   0% /proc/keys
tmpfs                    64.0M         0     64.0M   0% /proc/timer_list
tmpfs                     1.9G         0      1.9G   0% /sys/firmware

我們可以看到有成功掛載路徑

//stdemo20241117.file.core.windows.net/share-demo20241117
                          5.0G         0      5.0G   0% /mnt/azure

新增一個test-demo的檔案

kubectl exec -ti azurefile-pod -- touch /mnt/azure/test-demo

沒有出現錯誤訊息

2. 查看 Azurefile 檢查檔案

取得Azure File 使用金鑰進行驗證，需先獲取存取金鑰：

az storage account keys list --resource-group rg-demo-dev-01 --account-name stdemo20241117 --query "[0].value" -o tsv

指令執行成功後會顯示以下類似訊息

Wmy5No6h.........StTRutCg==

使用以下指令列出共享中的檔案：

az storage file list \
  --account-name <storage-account-name> \
  --account-key <account-key> \
  --share-name <file-share-name> \
  --output table
  • storage-account-name：您的 Azure Storage Account 名稱。
  • account-key：剛剛取得的存取金鑰。
  • share-name：Azure File 共享名稱。

範例：

az storage file list \
  --account-name stdemo20241117 \
  --account-key Wmy5No6h.........StTRutCg== \
  --share-name share-demo20241117 \
  --output table

指令執行成功後會顯示以下類似訊息

Name       Content Length    Type    Last Modified
---------  ----------------  ------  -------------------------
test-demo  0                 file    2024-11-17T14:20:53+00:00

檔案成功出現，驗證成功

參考

Azurefile 介紹

Azurefile-CSI-Driver Github網址

CATALOG
    FEATURED TAGS
    AWS Azure Azure File CSI Driver Azurefile Cloud Cloud Provider KinD CodeReady Containers Container Storage Interface CRC CSI Docker EC2 GCP Github Google Authenticator Hugo Istio KinD Kubernetes Linux LoadBalancer Memorystore MFA Observability Openshift Opentelemetry Openvpn Redhat Redis Secrets Store CSI Driver Service Mesh SSCSI Vault