We have a Jenkins server which has successfully built our code over 200 times - until a couple of days ago.
We are now getting an error to indicate that Jenkins was unable to delete the workspace (full message to follow with identifying elements redacted.)
I have checked through the recent code changes, and can see nothing which may have contributed to this issue, and nothing on that server has changed for weeks.
The stack trace suggests that "the context class hudson.FilePath is missing", but the config has not been changed from the config which worked over 200 times.
Can anybody suggest steps which could be taken to fix this ?
Started by user <REDACTED> > git rev-parse --is-inside-work-tree # timeout=10 Setting origin to > git config remote.origin.url # timeout=10 Fetching origin... Fetching upstream changes from origin > git --version # timeout=10 using GIT_ASKPASS to set credentials <REDACTED>@bitbucket > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* Seen branch in repository origin/master Seen branch in repository origin/temp Seen 2 remote branches Obtained code/Jenkinsfile from <REDACTED> Running in Durability level: MAX_SURVIVABILITY [Pipeline] node Running on Jenkins in /var/lib/jenkins/workspace/<REDACTED> [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout Cloning the remote Git repository Cloning with configured refspecs honoured and without tags Cloning repository ERROR: Failed to clean the workspace java.io.IOException: Unable to delete '/var/lib/jenkins/workspace/<REDACTED>. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. at hudson.Util.deleteContentsRecursive(Util.java:252) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:555) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1120) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate(ACL.java:290) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.nio.file.FileSystemException: /var/lib/jenkins/workspace/<REDACTED>/code/<REDACTED>-query/target/classes/application.properties: Operation not permitted at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileAttributeViews$Posix.setMode(UnixFileAttributeViews.java:238) at sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(UnixFileAttributeViews.java:260) at java.nio.file.Files.setPosixFilePermissions(Files.java:2045) at hudson.Util.makeWritable(Util.java:332) at hudson.Util.tryOnceDeleteFile(Util.java:292) at hudson.Util.tryOnceDeleteRecursive(Util.java:383) at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402) at hudson.Util.tryOnceDeleteRecursive(Util.java:382) at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402) at hudson.Util.tryOnceDeleteRecursive(Util.java:382) at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402) at hudson.Util.tryOnceDeleteRecursive(Util.java:382) at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402) at hudson.Util.tryOnceDeleteRecursive(Util.java:382) at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402) at hudson.Util.deleteContentsRecursive(Util.java:247) ... 14 more ERROR: Error cloning remote repo 'origin' hudson.plugins.git.GitException: Failed to delete workspace at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:558) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1120) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate(ACL.java:290) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Unable to delete '/var/lib/jenkins/workspace/<REDACTED>'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. at hudson.Util.deleteContentsRecursive(Util.java:252) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:555) ... 13 more Caused by: java.nio.file.FileSystemException: /var/lib/jenkins/workspace/<REDACTED>/code/<REDACTED>-query/target/classes/application.properties: Operation not permitted at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileAttributeViews$Posix.setMode(UnixFileAttributeViews.java:238) at sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(UnixFileAttributeViews.java:260) at java.nio.file.Files.setPosixFilePermissions(Files.java:2045) at hudson.Util.makeWritable(Util.java:332) at hudson.Util.tryOnceDeleteFile(Util.java:292) at hudson.Util.tryOnceDeleteRecursive(Util.java:383) at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402) at hudson.Util.tryOnceDeleteRecursive(Util.java:382) at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402) at hudson.Util.tryOnceDeleteRecursive(Util.java:382) at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402) at hudson.Util.tryOnceDeleteRecursive(Util.java:382) at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402) at hudson.Util.tryOnceDeleteRecursive(Util.java:382) at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402) at hudson.Util.deleteContentsRecursive(Util.java:247) ... 14 more [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // node [Pipeline] stage [Pipeline] { (Declarative: Post Actions) [Pipeline] step Required context class hudson.FilePath is missing Perhaps you forgot to surround the code with a step that provides this, such as: node [Pipeline] } [Pipeline] // stage [Pipeline] End of Pipeline ERROR: Error cloning remote repo 'origin' Finished: FAILURE 12 Answers
I finally found the solution that explains everything.
Cause: I was using docker within Jenkins and mounted Jenkins work directory in Docker (-v pwd:/code:rw). During runtime my program generate few files that also goes in to Jenkins work directory as it is mounted. but the user is docker root not Jenkins user, because of that Jenkins user is unable to delete those files and cause this error.
Solution: you can also share Jenkins user with docker, So all file created is same as file created by Jenkins itself.
docker run --user "$(id -u):$(id -g)" -i --rm -v /etc/passwd:/etc/passwd:ro -v `pwd`:/code:rw docker_image:tag So Adding This basically solve the issue by Using Jenkins user inside docker.
--user "$(id -u):$(id -g)" -v /etc/passwd:/etc/passwd:ro There is something locking the file or directory in the workspace.
- Someone has logged into Jenkins and is accessing the files from the workspace directly using the file system but not via Jenkins.
- The build process left a zombie program.
Some other similar cases.
If you have the shell access to FS use something like lsof to find the locking process.
EDIT: After several votes I implemented several items on some linux and windows slaves. In my case I know the names of the possible zombies.
I added a special job/build that kills processes that might be left in linux.
In windows I wrote a CS program that kills possible processes and/or runs the 'UnLock IT' with the Jenkins job that also frees up resources.
This page still gets a lot of attention, and this is probably the correct answer to my original question.
I had the same problem, found some files with permission root:root in the workspace directory. Running "sudo chown -R jenkins:jenkins /var/lib/jenkins/workspace" solved the problem.
Thanks to Olivier Boudry
0As a temporary solution, try renaming your project.
I am getting similar issue. The cause of issue is I am mounting Jenkins workspace to docker and execute the python code, It creates some Pycache executables with different permission and Jenkins is not able to delete that.
Temp fix for this is to remove git clone command then use sudo rm -rf code_directory then build. It will delete all code. Then restore git clone and your all other commands
I deleted the directory within the workspace directory, re-ran the job, and it fixed the issue.
I just tried renaming the job and it worked fine
I deleted the directory and tried again. Worked for me.
In my case, I had an error
Caused by: jenkins.util.io.CompositeIOException: Unable to delete '/home/jenkins/workspace/my_project'. Tried 3 times
But in fact, when I ssh'ed to that folder, it was empty.
Real location of the problematic folder was
/media/jenkins_home/workspace/my_project
When I deleted there and restarted the build, all went successfully. /home/jenkins/workspace/ is definitely not a symlink and I still don't understand why the folder in error log is different but guessing it is related to Docker volumes or smth..
I ran into this problem too within my Jenkins pipeline, specifically when integrating with Flywaydb before I had any migration files so the directory I was mounting didn't exist. What fixed it for me was to create the empty directory with .gitkeep and then the volume was able to be removed afterwards.
I am using docker within Jenkins and mounted Jenkins work directory in Docker container.
I saw the solution above
Solution: you can also share Jenkins user with docker, So all file created is same as file created by Jenkins itself.
docker run --user "$(id -u):$(id -g)" -i --rm -v /etc/passwd:/etc/passwd:ro -v `pwd`:/code:rw docker_image:tag So Adding This basically solve the issue by Using Jenkins user inside docker.
--user "$(id -u):$(id -g)" -v /etc/passwd:/etc/passwd:ro" But it is not clear for me where should I run this commands in the Jenkinsfile or on the machine/node where I run my job.
I had the same problem, found some files with permission root:root in the workspace directory. Tried running find . -user root -exec chmod 777 {} ; to change the permission of files but it failed because of some dangling symbolic links. Tried deleting the dangling symbolic link by find <folder name> -xtype l -delete but it also failed for some symbolic link (Since those symbolic giving Too many levels of symbolic links)
Running
sudo chown -R jenkins:jenkins /var/lib/jenkins/workspace solved the problem.
