Loading ...
/bin/rm annoying limitation: Argument list too long
I recently had to write a small shell script to clean up some temporary files that were left over in the homes of all the users in one system. No problem, all very simple, but in the process of running the script, I was hit by one very annoying limitation in the /bin/rm command that was failing if there were too many files to delete. Now since I have seen this previously and solved it very easy this was not a real problem, but I thought it might be useful to other readers of the site that are having similar problems.
Now in the loop of the script I have used the simple command to delete the files in cause like this:
rm -f /home/$u/tmp/*.wrkThis command was failing in case there were too many files in the respective folder, with the following error:
-bash: /bin/rm: Argument list too longHow many files were in there?
ls -1 | wc -l
2855Actually this is not a limitation of the rm command, but a kernel limitation on the size of the parameters of the command. Since I was performing shell globbing (selecting all the files with extension .wrk), this meant that the size of the command line arguments became bigger with the number of the files involved. For who cares this is defined by:
egrep ARG_MAX /usr/include/linux/limits.h
#define ARG_MAX 131072 /* # bytes of args + environ for exec() */The possible solutions are to either run the rm command inside a loop and delete each individual result, or to use find with the xargs parameter to pass the delete command. I prefer the find solution so I had changed the rm line inside the script to:
find /home/$u/tmp/ -name '*.wrk' -print0 | xargs -0 rmthis does the trick and solves the problem. One final touch was to not receive warnings if there were no actual files to delete, like:
rm: too few arguments
Try `rm --help' for more information.For this I have added the -f parameter to rm (-f, –force = ignore nonexistent files, never prompt). Since this was running in a shell script from cron the prompt was not needed also so no problem here. The final line I used to replace the rm one was:
find /home/$u/tmp/ -name '*.wrk' -print0 | xargs -0 rm -f
28th May 2006, 22:05
You can also do this in the directory:
ls | xargs rm
That’ll take care of it as well.
29th May 2006, 12:26
Alternatively you could have used a one line find:
find /home/$u/tmp/ -name ‘*.wrk’ -exec rm {} \; -print
Cheers,
Dez
—
Dez Blanchfield
http://WebSearch.COM.AU
http://TheStorageForum.COM
http://www.Blanchfield.COM.AU
http://CradleTechnologies.COM
25th July 2006, 19:38
Actually, MAX_ARG_PAGES, which…
“defines the number of pages allocated for arguments and envelope for the new program”
…is in /usr/src/linux/include/linux/binfmts.h
Not sure what effect ARG_MAX in /usr/include/linux/limits.h has, if any, on the problem at hand.
8th August 2006, 16:47
Thanks for posting this. Very helpful!
3rd September 2006, 22:16
Was running up against this problem while trying to delete several thousand files, and I’m not exactly a Linux expert – thanks for posting the solution.
5th November 2006, 20:44
Why not use:
find . -exec rm {} \;
18th December 2006, 11:45
really its helpful post. i would to write a perl script to overcome with such problem.
20th January 2007, 18:03
[...] Akhirnya saya temukan solusinya, tulisan marius di http://www.ducea.com. Solusi dari marius : [...]
12th March 2007, 14:29
You can also use find . -delete
13th March 2007, 15:15
I didn’t know about find -delete. That must be new though. It’s on find 4.2.27, but not version 4.1.20, so might not be available.
find -exec vs xargs can affect performance. As far as I understand it, find -exec creates a new process for each file, where xargs runs the command once.
24th March 2007, 20:19
“find . -name ‘foo*’ | xargs rm *” No Workie. Boohoo.
Same annoying message, no matter what args I use.
Now using -exec rm, which is hella slow.
30th March 2007, 13:55
Thanks, I have the same problem with 15.000 files in a directory and thanks to your post I have solved it.
Thanks very much!
2nd April 2007, 11:21
Personally I use for i in `ls `; do rm -v $i; done;
6th April 2007, 23:34
foobar, You should not have a * in the rm at the end. It should be just “find . -name ‘foo*’ | xargs rmâ€, otherwise you’re repeating the same problem.
25th April 2007, 08:07
Thanks, it was really helpful
15th May 2007, 03:26
Thank you! That was indeed helpful.
17th May 2007, 07:31
[...] Akhirnya saya temukan solusinya, tulisan marius di http://www.ducea.com. Solusi dari marius : [...]
25th June 2007, 05:46
I found this solution is the fastest
find /some/directory -name ‘*.wrk’ -print0 | xargs -n1000 -0 rm -f
The difference is the -n1000 which tells xargs to put 1000 arguments onto each invocation of rm. Therefore rm is run only 3 times for the 2855 files instead of being run 2855 times.
Not that -print0 and -0 is required if any file or path in the list has spaces in it. You should always use it to be safe.
17th August 2007, 12:48
Maybe tmpreaper is a better to clean up user’s tmp directories.
27th November 2007, 22:54
I have used
find /some/directory -name ‘*.wrk’ -print0 | xargs -n1000 -0 rm -f
and
find . ‘*.wrk’ -print0 | xargs -0 rm -f
Is there any way to do this with -mtime to delet files older than sya 30 days? (i can’t get it to work)
29th November 2007, 19:49
kp1269: that should work just by adding something like:
find . ‘*.wrk’ -mtime 30 -print0 | xargs -0 rm -f
If for some reason that doesn’t work for you, you might want to try -mmin +xx and translate in minutes the time you need. hth.
29th November 2007, 20:17
Thanks Marius..
I found that there is a limitation on mtime due to it having to sort… My problem stems from there being roughly 250,000 files in the directory.
I ended up just piping ls -rt to a file.sh and modding that to rm the files… took a long time, but worked.
29th January 2008, 20:23
[...] then, I found another hint (the comments are the useful part), and tried: ls post*.jpg | xargs chmod [...]
9th August 2008, 19:00
Often the reason why there are loads of files in a folder is because it is a logs folder and you haven’t been doing your house keeping properly. In these situations you rarely want to delete all of them. Usually you want to keep the last 30 days worth of files. So find is a far better solution, or bigdel documented here:
http://www.bigsoft.co.uk/blog/index.php/2008/08/09/rm-argument-list-too-long
24th October 2008, 13:14
cd /var/spool/mqueue/
find . -name ‘dfm3*’ | xargs rm
8th February 2009, 12:08
command works on my 7k+ swf files, thanks for the solution.
2nd April 2009, 16:08
thanks to much really nice …. but i need your help in this question
The goal of this assignment is to write a shell script to replace the rm command andprovide undelete capability similar to MS-DOS.Write a shell program to duplicate the UNIX rm command with the following features:1.It will have a switch -i that will act in the same manner as in the rm command.2.Instead of deleting the files, it will move them to a wastebasket directory. If thefile already exists in the wastebasket directory, then the existing file (in thewastebasket) will have the version number zero appended to it and the newlydeleted file will have version number one appended to it. If the version numbersetup is already in use for that file then the newest one will simply have the nextversion number in the series appended to it. It’s your choice as to what system touse for this. Just be sure that it is a unique system and will not be part of anynormal file naming convention you might use.3.It will have a switch -c that will clear the entire wastebasket after asking forconfirmation. The -i and -c switches may be combined to ask for confirmation onindividual files.Don’t forget that wildcards and multiple files are allowed on the command line. Theswitches when combined may be in any order but must be before any filenames. Yourprogram must also catch any operator errors.
i hope to answer me in my email
thanks …
4th September 2009, 13:49
thanx for sharing to the netiziens