Loading ...
/bin/rm annoying limitation: Argument list too long
I recently had to write a small shell script to clean up some temporary files that were left over in the homes of all the users in one system. No problem, all very simple, but in the process of running the script, I was hit by one very annoying limitation in the /bin/rm command that was failing if there were too many files to delete. Now since I have seen this previously and solved it very easy this was not a real problem, but I thought it might be useful to other readers of the site that are having similar problems.
Now in the loop of the script I have used the simple command to delete the files in cause like this:
rm -f /home/$u/tmp/*.wrkThis command was failing in case there were too many files in the respective folder, with the following error:
-bash: /bin/rm: Argument list too longHow many files were in there?
ls -1 | wc -l
2855Actually this is not a limitation of the rm command, but a kernel limitation on the size of the parameters of the command. Since I was performing shell globbing (selecting all the files with extension .wrk), this meant that the size of the command line arguments became bigger with the number of the files involved. For who cares this is defined by:
egrep ARG_MAX /usr/include/linux/limits.h
#define ARG_MAX 131072 /* # bytes of args + environ for exec() */The possible solutions are to either run the rm command inside a loop and delete each individual result, or to use find with the xargs parameter to pass the delete command. I prefer the find solution so I had changed the rm line inside the script to:
find /home/$u/tmp/ -name '*.wrk' -print0 | xargs -0 rmthis does the trick and solves the problem. One final touch was to not receive warnings if there were no actual files to delete, like:
rm: too few arguments
Try `rm --help' for more information.For this I have added the -f parameter to rm (-f, –force = ignore nonexistent files, never prompt). Since this was running in a shell script from cron the prompt was not needed also so no problem here. The final line I used to replace the rm one was:
find /home/$u/tmp/ -name '*.wrk' -print0 | xargs -0 rm -f
28th May 2006, 22:05
You can also do this in the directory:
ls | xargs rm
That’ll take care of it as well.
29th May 2006, 12:26
Alternatively you could have used a one line find:
find /home/$u/tmp/ -name ‘*.wrk’ -exec rm {} \; -print
Cheers,
Dez
—
Dez Blanchfield
http://WebSearch.COM.AU
http://TheStorageForum.COM
http://www.Blanchfield.COM.AU
http://CradleTechnologies.COM
25th July 2006, 19:38
Actually, MAX_ARG_PAGES, which…
“defines the number of pages allocated for arguments and envelope for the new program”
…is in /usr/src/linux/include/linux/binfmts.h
Not sure what effect ARG_MAX in /usr/include/linux/limits.h has, if any, on the problem at hand.
8th August 2006, 16:47
Thanks for posting this. Very helpful!
3rd September 2006, 22:16
Was running up against this problem while trying to delete several thousand files, and I’m not exactly a Linux expert - thanks for posting the solution.
5th November 2006, 20:44
Why not use:
find . -exec rm {} \;
18th December 2006, 11:45
really its helpful post. i would to write a perl script to overcome with such problem.
20th January 2007, 18:03
[...] Akhirnya saya temukan solusinya, tulisan marius di http://www.ducea.com. Solusi dari marius : [...]
12th March 2007, 14:29
You can also use find . -delete
13th March 2007, 15:15
I didn’t know about find -delete. That must be new though. It’s on find 4.2.27, but not version 4.1.20, so might not be available.
find -exec vs xargs can affect performance. As far as I understand it, find -exec creates a new process for each file, where xargs runs the command once.
24th March 2007, 20:19
“find . -name ‘foo*’ | xargs rm *” No Workie. Boohoo.
Same annoying message, no matter what args I use.
Now using -exec rm, which is hella slow.
30th March 2007, 13:55
Thanks, I have the same problem with 15.000 files in a directory and thanks to your post I have solved it.
Thanks very much!
2nd April 2007, 11:21
Personally I use for i in `ls `; do rm -v $i; done;
6th April 2007, 23:34
foobar, You should not have a * in the rm at the end. It should be just “find . -name ‘foo*’ | xargs rmâ€, otherwise you’re repeating the same problem.
25th April 2007, 08:07
Thanks, it was really helpful
15th May 2007, 03:26
Thank you! That was indeed helpful.
17th May 2007, 07:31
[...] Akhirnya saya temukan solusinya, tulisan marius di http://www.ducea.com. Solusi dari marius : [...]
25th June 2007, 05:46
I found this solution is the fastest
find /some/directory -name ‘*.wrk’ -print0 | xargs -n1000 -0 rm -f
The difference is the -n1000 which tells xargs to put 1000 arguments onto each invocation of rm. Therefore rm is run only 3 times for the 2855 files instead of being run 2855 times.
Not that -print0 and -0 is required if any file or path in the list has spaces in it. You should always use it to be safe.
17th August 2007, 12:48
Maybe tmpreaper is a better to clean up user’s tmp directories.
27th November 2007, 22:54
I have used
find /some/directory -name ‘*.wrk’ -print0 | xargs -n1000 -0 rm -f
and
find . ‘*.wrk’ -print0 | xargs -0 rm -f
Is there any way to do this with -mtime to delet files older than sya 30 days? (i can’t get it to work)
29th November 2007, 19:49
kp1269: that should work just by adding something like:
find . ‘*.wrk’ -mtime 30 -print0 | xargs -0 rm -f
If for some reason that doesn’t work for you, you might want to try -mmin +xx and translate in minutes the time you need. hth.
29th November 2007, 20:17
Thanks Marius..
I found that there is a limitation on mtime due to it having to sort… My problem stems from there being roughly 250,000 files in the directory.
I ended up just piping ls -rt to a file.sh and modding that to rm the files… took a long time, but worked.
29th January 2008, 20:23
[...] then, I found another hint (the comments are the useful part), and tried: ls post*.jpg | xargs chmod [...]
9th August 2008, 19:00
Often the reason why there are loads of files in a folder is because it is a logs folder and you haven’t been doing your house keeping properly. In these situations you rarely want to delete all of them. Usually you want to keep the last 30 days worth of files. So find is a far better solution, or bigdel documented here:
http://www.bigsoft.co.uk/blog/index.php/2008/08/09/rm-argument-list-too-long
24th October 2008, 13:14
cd /var/spool/mqueue/
find . -name ‘dfm3*’ | xargs rm