InfinityQuest - Programming Code Tutorials and Examples with Python, C++, Java, PHP, C#, JavaScript, Swift and more

Menu
  • Home
  • Sitemap

Python Programming Language Best Tutorials and Code Examples

Learn Python Right Now!
Home
Bash
Working Around “argument list too long” Errors in bash
Bash

Working Around “argument list too long” Errors in bash

InfinityCoder April 4, 2017

You get an “argument list too long” error while trying to do an operation involving shell wildcard expansion.

Use the xargs command, possibly in conjunction with find, to break up your argument list.
For simple cases, just use a for loop or find instead of ls:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
$ ls /path/with/many/many/files/*e*
-/bin/bash: /bin/ls: Argument list too long
 
# Short demo, surrounding ~ are for illustration only
$ for i in ./some_files/*e*; do echo "~$i~"; done
~./some_files/A file with (parens)~
~./some_files/A file with [brackets]~
~./some_files/File with embedded
newline~
~./some_files/file with = sign~
~./some_files/file with spaces~
~./some_files/file with |~
~./some_files/file with:~
~./some_files/file with;~
~./some_files/regular_file~
 
$ find ./some_files -name '*e*' -exec echo ~{}~ \;
~./some_files~
~./some_files/A file with [brackets]~
~./some_files/A file with (parens)~
~./some_files/regular_file~
~./some_files/file with spaces~
~./some_files/file with = sign~
~./some_files/File with embedded
newline~
~./some_files/file with;~
~./some_files/file with:~
~./some_files/file with |~
 
$ for i in /path/with/many/many/files/*e*; do echo "$i"; done
[This works, but the output is too long to list]
 
$ find /path/with/many/many/files/ -name '*e*'
[This works, but the output is too long to list]

The example above works correctly with the echo command, but when you feed that “$i” into other programs, especially other shell constructs, $IFS and other parsing may come into play.

The GNU find and xargs take that into account with find – print0 and xargs -0. (No, we don’t know why it’s -print0 and -0 instead of being consistent.)

These arguments cause find to use the null character (which can’t appear in a filename) instead of whitespace as an output record separator, and xargs to use null as its input record separator.

That will correctly parse files containing odd characters.

1
$ find /path/with/many/many/files/ -name '*e*' -print0 | xargs -0 proggy

Note that the default behavior of bash (and sh) is to return unmatched patterns
unchanged.

That means you could end up with your for loop setting $i to ./some_ files/*e* if no files match the wildcard pattern.

You can set the shopt -s nullglob option to cause filename patterns that match no files to expand to a null string, rather than expand to themselves.
You might assume that the for loop solution in the simple case would run into the same problem as the ls command, but it doesn’t.

Chet Ramey tells us:

1
2
3
4
5
6
7
ARG_MAX bounds the total space requirement of the exec* family of system calls, so the
kernel knows the largest buffer it will have to allocate. This is all three arguments to
execve: program name, argument vector, and environment.
 
The [ls command] fails because the total bytes taken up by the arguments to execve
exceeds ARG_MAX. The [for loop] succeeds because everything is done internally:
though the entire list is generated and stored, execve is never called.

Be careful that find doesn’t find too many files, since it will recursively descend into all subdirectories by default while ls will not.

Some versions of find have a -d option to control how deep it goes. Using the for loop is probably easier.
Use the getconf ARG_MAX command to see what the limit is on your system.

It varies wildly (see also getconf LINE_MAX; see Table 15-1).

Table 15-1. System limits

System ARG_MAX limits (bytes)
HP-UX11

Solarid(1,9,10)

NetBSD 2.0.2, OpenBSD 3.7, OS/X

Linux (Red Hat, Debian, Ubuntu)

FreeBSD 5.4

2048000

1048320

262144

131072

65536

Share
Tweet
Email
Prev Article
Next Article

Related Articles

Using bash Net-Redirection in bash
You need to send or receive very simple network traffic …

Using bash Net-Redirection in bash

Separating Variable Names from Surrounding Text in bash
You need to print a variable along with other text. …

Separating Variable Names from Surrounding Text in bash

About The Author

InfinityCoder
InfinityCoder

Leave a Reply

Cancel reply

Recent Tutorials InfinityQuest

  • Adding New Features to bash Using Loadable Built-ins in bash
    Adding New Features to bash Using Loadable …
    June 27, 2017 0
  • Getting to the Bottom of Things in bash
    Getting to the Bottom of Things in …
    June 27, 2017 0

Recent Comments

  • fer on Turning a Dictionary into XML in Python
  • mahesh on Turning a Dictionary into XML in Python

Categories

  • Bash
  • PHP
  • Python
  • Uncategorized

InfinityQuest - Programming Code Tutorials and Examples with Python, C++, Java, PHP, C#, JavaScript, Swift and more

About Us

Start learning your desired programming language with InfinityQuest.com.

On our website you can access any tutorial that you want with video and code examples.

We are very happy and honored that InfinityQuest.com has been listed as a recommended learning website for students.

Popular Tags

binary data python CIDR convert string into datetime python create xml from dict python dictionary into xml python how to create xml with dict in Python how to write binary data in Python IP Address read binary data python tutorial string as date object python string to datetime python

Archives

  • June 2017
  • April 2017
  • February 2017
  • January 2017
  • December 2016
  • November 2016
Copyright © 2021 InfinityQuest - Programming Code Tutorials and Examples with Python, C++, Java, PHP, C#, JavaScript, Swift and more
Programming Tutorials | Sitemap