You get an “argument list too long” error while trying to do an operation involving shell wildcard expansion.
Use the xargs command, possibly in conjunction with find, to break up your argument list.
For simple cases, just use a for loop or find instead of ls:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
$ ls /path/with/many/many/files/*e* -/bin/bash: /bin/ls: Argument list too long # Short demo, surrounding ~ are for illustration only $ for i in ./some_files/*e*; do echo "~$i~"; done ~./some_files/A file with (parens)~ ~./some_files/A file with [brackets]~ ~./some_files/File with embedded newline~ ~./some_files/file with = sign~ ~./some_files/file with spaces~ ~./some_files/file with |~ ~./some_files/file with:~ ~./some_files/file with;~ ~./some_files/regular_file~ $ find ./some_files -name '*e*' -exec echo ~{}~ \; ~./some_files~ ~./some_files/A file with [brackets]~ ~./some_files/A file with (parens)~ ~./some_files/regular_file~ ~./some_files/file with spaces~ ~./some_files/file with = sign~ ~./some_files/File with embedded newline~ ~./some_files/file with;~ ~./some_files/file with:~ ~./some_files/file with |~ $ for i in /path/with/many/many/files/*e*; do echo "$i"; done [This works, but the output is too long to list] $ find /path/with/many/many/files/ -name '*e*' [This works, but the output is too long to list] |
The example above works correctly with the echo command, but when you feed that “$i” into other programs, especially other shell constructs, $IFS and other parsing may come into play.
The GNU find and xargs take that into account with find – print0 and xargs -0. (No, we don’t know why it’s -print0 and -0 instead of being consistent.)
These arguments cause find to use the null character (which can’t appear in a filename) instead of whitespace as an output record separator, and xargs to use null as its input record separator.
That will correctly parse files containing odd characters.
|
1 |
$ find /path/with/many/many/files/ -name '*e*' -print0 | xargs -0 proggy |
Note that the default behavior of bash (and sh) is to return unmatched patterns
unchanged.
That means you could end up with your for loop setting $i to ./some_ files/*e* if no files match the wildcard pattern.
You can set the shopt -s nullglob option to cause filename patterns that match no files to expand to a null string, rather than expand to themselves.
You might assume that the for loop solution in the simple case would run into the same problem as the ls command, but it doesn’t.
Chet Ramey tells us:
|
1 2 3 4 5 6 7 |
ARG_MAX bounds the total space requirement of the exec* family of system calls, so the kernel knows the largest buffer it will have to allocate. This is all three arguments to execve: program name, argument vector, and environment. The [ls command] fails because the total bytes taken up by the arguments to execve exceeds ARG_MAX. The [for loop] succeeds because everything is done internally: though the entire list is generated and stored, execve is never called. |
Be careful that find doesn’t find too many files, since it will recursively descend into all subdirectories by default while ls will not.
Some versions of find have a -d option to control how deep it goes. Using the for loop is probably easier.
Use the getconf ARG_MAX command to see what the limit is on your system.
It varies wildly (see also getconf LINE_MAX; see Table 15-1).
Table 15-1. System limits
| System | ARG_MAX limits (bytes) |
| HP-UX11
Solarid(1,9,10) NetBSD 2.0.2, OpenBSD 3.7, OS/X Linux (Red Hat, Debian, Ubuntu) FreeBSD 5.4 |
2048000
1048320 262144 131072 65536 |
