Keep running into "exceeded its progress dead line" despite changing progressDeadlineSeconds

I'm new AKS, ACR, and DevOps Pipelines and I'm trying to setup a CI/CD pipeline.

I have a resource group setup that has both AKS and ACR in it. AKS is using Standard_B2s and only one node at this point since I'm just playing around.

Images are being deployed to ACR automatically on a commit to master--haven't figured out how to setup testing yet--but when it comes to deploying to AKS, I just keep getting a:

##[error]error: deployment "client-deployment" exceeded its progress deadline 

I've changed my client.yaml to include a progressDeadlineSeconds of like an hour as 10, 15, and 20 minutes didn't work:

apiVersion: apps/v1 kind: Deployment metadata: name: client-deployment spec: progressDeadlineSeconds: 3600 replicas: 1 selector: matchLabels: component: client template: metadata: labels: component: client spec: containers: - name: client image: ports: - containerPort: 3000 --- apiVersion: v1 kind: Service metadata: name: client-cluster-ip-service spec: type: ClusterIP selector: component: client ports: - port: 3000 targetPort: 3000 

I've just been modifying the azure-pipelines.yml that Pipelines generated for me, which I currently have as the following:

# Docker # Build and push an image to Azure Container Registry # trigger: - master resources: - repo: self variables: # Container registry service connection established during pipeline creation dockerRegistryServiceConnection: <dockerRegistryServiceConnection_key> imageRepository: 'testapp' containerRegistry: 'testappcontainers.azurecr.io' dockerfilePath: '$(Build.SourcesDirectory)' tag: '$(Build.BuildId)' imagePullSecret: <imagePullSecret_key> # Agent VM image name vmImageName: 'ubuntu-latest' stages: - stage: Build displayName: Build and push stage jobs: - job: Build displayName: Build pool: vmImage: $(vmImageName) steps: - task: Docker@2 displayName: Build and push client image to container registry inputs: command: buildAndPush repository: $(imageRepository)-client dockerfile: $(dockerfilePath)/client/Dockerfile containerRegistry: $(dockerRegistryServiceConnection) tags: | $(tag) - upload: manifests artifact: manifests - stage: Deploy displayName: Deploy stage dependsOn: Build jobs: - deployment: Deploy displayName: Deploy job pool: vmImage: $(vmImageName) environment: 'testapp.default' strategy: runOnce: deploy: steps: - task: KubernetesManifest@0 displayName: Create imagePullSecret inputs: action: createSecret secretName: $(imagePullSecret) dockerRegistryEndpoint: $(dockerRegistryServiceConnection) - task: KubernetesManifest@0 displayName: Deploy to Kubernetes cluster inputs: action: deploy manifests: | $(Pipeline.Workspace)/manifests/client.yaml imagePullSecrets: | $(imagePullSecret) containers: | $(containerRegistry)/$(imageRepository):$(tag) 

Here is the log too for the Task that fails:

##[debug]Evaluating condition for step: 'Deploy to Kubernetes cluster' ##[debug]Evaluating: SucceededNode() ##[debug]Evaluating SucceededNode: ##[debug]=> True ##[debug]Result: True ##[section]Starting: Deploy to Kubernetes cluster ============================================================================== Task : Deploy to Kubernetes Description : Use Kubernetes manifest files to deploy to clusters or even bake the manifest files to be used for deployments using Helm charts Version : 0.162.1 Author : Microsoft Corporation Help : ============================================================================== ##[debug]agent.TempDirectory=/home/vsts/work/_temp ##[debug]loading inputs and endpoints ##[debug]loading INPUT_ACTION ##[debug]loading INPUT_KUBERNETESSERVICECONNECTION ##[debug]loading INPUT_STRATEGY ##[debug]loading INPUT_TRAFFICSPLITMETHOD ##[debug]loading INPUT_PERCENTAGE ##[debug]loading INPUT_BASELINEANDCANARYREPLICAS ##[debug]loading INPUT_MANIFESTS ##[debug]loading INPUT_CONTAINERS ##[debug]loading INPUT_IMAGEPULLSECRETS ##[debug]loading INPUT_RENDERTYPE ##[debug]loading INPUT_DOCKERCOMPOSEFILE ##[debug]loading INPUT_HELMCHART ##[debug]loading INPUT_KUSTOMIZATIONPATH ##[debug]loading INPUT_RESOURCETOPATCH ##[debug]loading INPUT_RESOURCEFILETOPATCH ##[debug]loading INPUT_MERGESTRATEGY ##[debug]loading INPUT_SECRETTYPE ##[debug]loading ENDPOINT_AUTH_<token> ##[debug]loading ENDPOINT_AUTH_SCHEME_<token> ##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_AZUREENVIRONMENT ##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_AZURETENANTID ##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_SERVICEACCOUNTNAME ##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_ROLEBINDINGNAME ##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_SECRETNAME ##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_APITOKEN ##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_SERVICEACCOUNTCERTIFICATE ##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION ##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION ##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN ##[debug]loading SECRET_CONTAINER_PASSWORD ##[debug]loading SECRET_CONTAINER_USERNAME ##[debug]loading SECRET_SYSTEM_ACCESSTOKEN ##[debug]loaded 32 ##[debug]Agent.ProxyUrl=undefined ##[debug]Agent.CAInfo=undefined ##[debug]Agent.ClientCert=undefined ##[debug]Agent.SkipCertValidation=undefined ##[debug]SYSTEM_HOSTTYPE=build ##[debug]System.TeamFoundationCollectionUri= ##[debug]Build.BuildNumber=20191231.5 ##[debug]Build.DefinitionName=test-app ##[debug]System.DefinitionId=4 ##[debug]Agent.JobName=Deploy job ##[debug]System.TeamProject=test-app ##[debug]Build.BuildId=41 ##[debug]System.TeamProject=test-app ##[debug]namespace=null ##[debug]containers=***/testapp:41 ##[debug]imagePullSecrets=testappcontainers<key>-auth ##[debug]manifests=/home/vsts/work/1/manifests/client.yaml ##[debug]percentage=0 ##[debug]strategy=none ##[debug]trafficSplitMethod=pod ##[debug]baselineAndCanaryReplicas=0 ##[debug]arguments=null ##[debug]secretArguments=null ##[debug]secretType=dockerRegistry ##[debug]secretName=null ##[debug]dockerRegistryEndpoint=null ##[debug]kubernetesServiceConnection=<token> ##[debug]<token> data namespace = default ##[debug]System.TeamFoundationCollectionUri= ##[debug]System.HostType=build ##[debug]System.DefaultWorkingDirectory=/home/vsts/work/1/s ##[debug]Build.SourceBranchName=master ##[debug]Build.Repository.Provider=TfsGit ##[debug]Build.Repository.Uri= ##[debug]agent.proxyurl=undefined ##[debug]VSTS_ARM_REST_IGNORE_SSL_ERRORS=undefined ##[debug]AZURE_HTTP_USER_AGENT=VSTS_<hash>_build_4_0 ##[debug]Agent.ProxyUrl=undefined ##[debug]Agent.CAInfo=undefined ##[debug]Agent.ClientCert=undefined ##[debug]check path : /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/node_modules/azure-pipelines-tool-lib/lib.json ##[debug]adding resource file: /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/node_modules/azure-pipelines-tool-lib/lib.json ##[debug]system.culture=en-US ##[debug]check path : /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/task.json ##[debug]adding resource file: /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/task.json ##[debug]system.culture=en-US ##[debug]action=deploy ##[debug]kubernetesServiceConnection=<token> ##[debug]agent.tempDirectory=/home/vsts/work/_temp ##[debug]<token> data authorizationType = AzureSubscription ##[debug]<token>= ##[debug]<token> auth param serviceAccountCertificate = *** ##[debug]<token> auth param apiToken = *** ##[debug]set KUBECONFIG=/home/vsts/work/_temp/kubectlTask/1577816701759/config ##[debug]Processed: ##vso[task.setvariable variable=KUBECONFIG;issecret=false;]/home/vsts/work/_temp/kubectlTask/1577816701759/config ##[debug]<token> data acceptUntrustedCerts = undefined ##[debug]which 'kubectl' ##[debug]found: '/usr/bin/kubectl' ##[debug]which 'kubectl' ##[debug]found: '/usr/bin/kubectl' ##[debug]System.DefaultWorkingDirectory=/home/vsts/work/1/s ##[debug]defaultRoot: '/home/vsts/work/1/s' ##[debug]findOptions.allowBrokenSymbolicLinks: 'false' ##[debug]findOptions.followSpecifiedSymbolicLink: 'true' ##[debug]findOptions.followSymbolicLinks: 'true' ##[debug]matchOptions.debug: 'false' ##[debug]matchOptions.nobrace: 'true' ##[debug]matchOptions.noglobstar: 'false' ##[debug]matchOptions.dot: 'true' ##[debug]matchOptions.noext: 'false' ##[debug]matchOptions.nocase: 'false' ##[debug]matchOptions.nonull: 'false' ##[debug]matchOptions.matchBase: 'false' ##[debug]matchOptions.nocomment: 'false' ##[debug]matchOptions.nonegate: 'false' ##[debug]matchOptions.flipNegate: 'false' ##[debug]pattern: '/home/vsts/work/1/manifests/client.yaml' ##[debug]findPath: '/home/vsts/work/1/manifests/client.yaml' ##[debug]statOnly: 'true' ##[debug]found 1 paths ##[debug]applying include pattern ##[debug]1 matches ##[debug]1 final results ##[debug]agent.tempDirectory=/home/vsts/work/_temp ##[debug]New K8s objects after addin imagePullSecrets are :[{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"name":"client-deployment"},"spec":{"progressDeadlineSeconds":3600,"replicas":1,"selector":{"matchLabels":{"component":"client"}},"template":{"metadata":{"labels":{"component":"client"}},"spec":{"containers":[{"name":"client","image":"***/testapp-client","ports":[{"containerPort":3000}]}],"imagePullSecrets":[{"name":"testappcontainers1741032e-auth"}]}}}},{"apiVersion":"v1","kind":"Service","metadata":{"name":"client-cluster-ip-service"},"spec":{"type":"ClusterIP","selector":{"component":"client"},"ports":[{"port":3000,"targetPort":3000}]}}] ##[debug]agent.tempDirectory=/home/vsts/work/_temp ##[debug]agent.tempDirectory=/home/vsts/work/_temp ##[debug]which '/usr/bin/kubectl' ##[debug]found: '/usr/bin/kubectl' ##[debug]which '/usr/bin/kubectl' ##[debug]found: '/usr/bin/kubectl' ##[debug]/usr/bin/kubectl arg: apply ##[debug]/usr/bin/kubectl arg: ["-f","/home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782"] ##[debug]/usr/bin/kubectl arg: ["--namespace","default"] ##[debug]exec tool: /usr/bin/kubectl ##[debug]arguments: ##[debug] apply ##[debug] -f ##[debug] /home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782 ##[debug] --namespace ##[debug] default [command]/usr/bin/kubectl apply -f /home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782 --namespace default unchanged service/client-cluster-ip-service unchanged ##[debug]which '/usr/bin/kubectl' ##[debug]found: '/usr/bin/kubectl' ##[debug]which '/usr/bin/kubectl' ##[debug]found: '/usr/bin/kubectl' ##[debug]/usr/bin/kubectl arg: ["rollout","status"] ##[debug]/usr/bin/kubectl arg: Deployment/client-deployment ##[debug]/usr/bin/kubectl arg: ["--namespace","default"] ##[debug]exec tool: /usr/bin/kubectl ##[debug]arguments: ##[debug] rollout ##[debug] status ##[debug] Deployment/client-deployment ##[debug] --namespace ##[debug] default [command]/usr/bin/kubectl rollout status Deployment/client-deployment --namespace default error: deployment "client-deployment" exceeded its progress deadline ##[debug]which '/usr/bin/kubectl' ##[debug]found: '/usr/bin/kubectl' ##[debug]which '/usr/bin/kubectl' ##[debug]found: '/usr/bin/kubectl' ##[debug]/usr/bin/kubectl arg: get ##[debug]/usr/bin/kubectl arg: service/client-cluster-ip-service ##[debug]/usr/bin/kubectl arg: ["-o","json"] ##[debug]/usr/bin/kubectl arg: ["--namespace","default"] ##[debug]exec tool: /usr/bin/kubectl ##[debug]arguments: ##[debug] get ##[debug] service/client-cluster-ip-service ##[debug] -o ##[debug] json ##[debug] --namespace ##[debug] default [command]/usr/bin/kubectl get service/client-cluster-ip-service -o json --namespace default { "apiVersion": "v1", "kind": "Service", "metadata": { "annotations": { "": "{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"name\":\"client-cluster-ip-service\",\"namespace\":\"default\"},\"spec\":{\"ports\":[{\"port\":3000,\"targetPort\":3000}],\"selector\":{\"component\":\"client\"},\"type\":\"ClusterIP\"}}\n" }, "creationTimestamp": "name": "client-cluster-ip-service", "namespace": "default", "resourceVersion": "1234045", "selfLink": "/api/v1/namespaces/default/services/client-cluster-ip-service", "uid": "5f077159-2bdd-11ea-af20-3eaa105eb2b3" }, "spec": { "clusterIP": "10.0.181.220", "ports": [ { "port": 3000, "protocol": "TCP", "targetPort": 3000 } ], "selector": { "component": "client" }, "sessionAffinity": "None", "type": "ClusterIP" }, "status": { "loadBalancer": {} } } ##[debug]KUBECONFIG=/home/vsts/work/_temp/kubectlTask/1577816701759/config ##[debug]set KUBECONFIG= ##[debug]Processed: ##vso[task.setvariable variable=KUBECONFIG;issecret=false;] ##[debug]task result: Failed ##[error]error: deployment "client-deployment" exceeded its progress deadline ##[debug]Processed: ##vso[task.issue type=error;]error: deployment "client-deployment" exceeded its progress deadline ##[debug]Processed: ##vso[task.complete result=Failed;]error: deployment "client-deployment" exceeded its progress deadline ##[section]Finishing: Deploy to Kubernetes cluster 

Then in Azure CLI, it shows the deployment is there, but with no available pods:

eox-dev@Azure:~$ kubectl get deployments NAME READY UP-TO-DATE AVAILABLE AGE client-deployment 0/1 1 0 3h47m eox-dev@Azure:~$ kubectl describe deployment client-deployment Name: client-deployment Namespace: default CreationTimestamp: Tue, 31 Dec 2019 15:50:30 +0000 Labels: <none> Annotations: 1 {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"client-deployment","namespace":"default"},"spec":{"progre... Selector: component=client Replicas: 1 desired | 1 updated | 1 total | 0 available | 1 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 25% max unavailable, 25% max surge Pod Template: Labels: component=client Containers: client: Image: Port: 3000/TCP Host Port: 0/TCP Environment: <none> Mounts: <none> Volumes: <none> Conditions: Type Status Reason ---- ------ ------ Available False MinimumReplicasUnavailable Progressing False ProgressDeadlineExceeded OldReplicaSets: <none> NewReplicaSet: client-deployment-5688bdc69c (1/1 replicas created) Events: <none> 

So what am I doing wrong here?

5 Answers

Error from server (BadRequest): container "client" in pod "client-deployment-5688bdc69c-hxlcf" is waiting to start: trying and failing to pull image

Based on my experience, this is more relative with imagePullSecrets and the Kubernetes namespace.

In your Create imagePullSecret and Deploy to Kubernetes cluster task, I saw that you did not provide the value to task parameter: namespace. This will lead to a new namespace which name is default will be created, since you unspecified the namespace.

And, the kubernetes secret which generated by createSecret action is seperated for each namespace. In one word, different namespace has different secret value:

Secrets are stored within a given namespace and can only be accessed by pods within the same namespace.

Now, let’s back to your build compile process.

In your yml definition, Create imagePullSecret will create a secret for new namespace default which created by task automatically as you did not provide the given namespace value.

Then, in next task Deploy to Kubernetes cluster, because of the same reason, here the task will re-created a another new namespace default(Note: this is not same with the previous one). Also, you could see this progress from the log:

At this time, the secret that generated from the previous task will not available for the current namespace. BUT, as you know, the ACR is a private container registry which our system must verify whether the kubernetes secret is available.

In addition, in your Deploy to Kubernetes cluster task, you were specifying the repository as $(imageRepository) which does not same with the repository you push the image to $(imageRepository)-client.

This can also be checked in your log:

That's why there's no available node in your kubernetes, and you failed to pull the image also.

To avoid the issue, please ensure you provide the namespace value in KubernetesManifest@0 task.

 - task: KubernetesManifest@0 displayName: Create imagePullSecret inputs: action: createSecret secretName: $(imagePullSecret) namespace: $(k8sNamespace) dockerRegistryEndpoint: $(DRServiceConnection) - task: KubernetesManifest@0 displayName: Deploy to Kubernetes cluster inputs: action: deploy namespace: $(k8sNamespace) manifests: | $(System.ArtifactsDirectory)/manifests/deployment.yml imagePullSecrets: | $(imagePullSecret) containers: | $(containerRegistry)/$(imageRepository)-client:$(tag) 

secret to imagePullSecrets of each namespace

I think what is likely to cause this issue (particularly in AKS) is if your cluster runs a health check or a liveness check on your containers before deploying. Your cluster identifies that your application isn't running and enters a waiting state due to CrashLoopBackOff

This can be fixed by just creating a default GET route on your API application on "/" that returns a 200. This will indicate that your containers are healthy or live

I faced a similar issues even provided the namespace still get error "exceeded its progress deadline Waiting for deployment "XX-service" rollout to finish: 0 of 1 updated replicas are available..."

due to fail to pull image

Adding "imagePullPolicy: Always" in .yaml fixed my rollout pull issues

I've faced a similar issue because of an exception and also readiness probe failures. After solving those issues, rollout has been completed successfully.

In my case, it was issue with the worker node. So I just tried to pull the image from the node which my pod is assigned to. I came to know that the node is having some perfmance related issue so I just add a high new node with high spec and assigned the pod to that node. The deployment also got successful.