InfinityQuest - Programming Code Tutorials and Examples with Python, C++, Java, PHP, C#, JavaScript, Swift and more

Menu
  • Home
  • Sitemap

Python Programming Language Best Tutorials and Code Examples

Learn Python Right Now!
Home
Bash
Isolating Specific Fields in Data in bash
Bash

Isolating Specific Fields in Data in bash

InfinityCoder February 21, 2017

You need to extract one or more fields from each line of output.

Use cut if there are delimiters you can easily pick out, even if they are different for the beginning and end of the field you need:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# Here's an easy one, what users, home directories and shells do
# we have on this NetBSD system
$ cut -d':' -f1,6,7 /etc/passwd
root:/root:/bin/csh
toor:/root:/bin/sh
daemon:/:/sbin/nologin
operator:/usr/guest/operator:/sbin/nologin
bin:/:/sbin/nologin
games:/usr/games:/sbin/nologin
postfix:/var/spool/postfix:/sbin/nologin
named:/var/chroot/named:/sbin/nologin
ntpd:/var/chroot/ntpd:/sbin/nologin
sshd:/var/chroot/sshd:/sbin/nologin
smmsp:/nonexistent:/sbin/nologin
uucp:/var/spool/uucppublic:/usr/libexec/uucp/uucico
nobody:/nonexistent:/sbin/nologin
jp:/home/jp:/usr/pkg/bin/bash
 
# What is the most popular shell on the system?
$ cut -d':' -f7 /etc/passwd | sort | uniq -c | sort -rn
  10 /sbin/nologin
   2 /usr/pkg/bin/bash
   1 /bin/csh
   1 /bin/sh
   1 /usr/libexec/uucp/uucico
 
# Now let's see the first two directory levels
$ cut -d':' -f6 /etc/passwd | cut -d'/' -f1-3 | sort -u
/
/home/jp
/nonexistent
/root
/usr/games
/usr/guest
/var/chroot
/var/spool

Use awk to split on multiples of whitespace, or if you need to rearrange the order of the output fields.

Note the ➝ denotes a tab character in the output. The default is space but you can change that using $OFS:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# Users, home directories and shells, but swap the last two
# and use a tab delimiter
$ awk 'BEGIN {FS=":"; OFS="\t"; } { print $1,$7,$6; }' /etc/passwd
root ➝ /bin/csh ➝ /root
toor ➝ /bin/sh ➝ /root
daemon ➝ /sbin/nologin ➝ /
operator ➝ /sbin/nologin ➝ /usr/guest/operator
bin ➝ /sbin/nologin ➝ /
games ➝ /sbin/nologin ➝ /usr/games
postfix ➝ /sbin/nologin ➝ /var/spool/postfix
named ➝ /sbin/nologin ➝ /var/chroot/named
ntpd ➝ /sbin/nologin ➝ /var/chroot/ntpd
sshd ➝ /sbin/nologin ➝ /var/chroot/sshd
smmsp ➝ /sbin/nologin ➝ /nonexistent
uucp ➝ /usr/libexec/uucp/uucico ➝ /var/spool/uucppublic
nobody ➝ /sbin/nologin ➝ /nonexistent
jp ➝ /usr/pkg/bin/bash ➝ /home/jp
 
# Multiples of whitespace and swapped, first field removed
$ grep '^# [1-9]' /etc/hosts | awk '{print $3,$2}'
10.255.255.255 10.0.0.0
172.31.255.255 172.16.0.0
192.168.255.255 192.168.0.0

Use grep -o to display just the part that matched your pattern.

This is particularly handy when you can’t express delimiters in a way that lends itself to the above solutions.
For example, say you need to extract all IP addresses from a file, no matter where they are.

Note we use egrep because of the regular expression (regex), but -o should work with whichever GNU grep flavor you use, but it is probably not supported on non-GNU versions.

Check your documentation.

1
2
3
4
5
6
7
8
9
10
$ cat has_ipas
This is line 1 with 1 IPA: 10.10.10.10
Line 2 has 2; they are 10.10.10.11 and 10.10.10.12.
Line three is ftp_server=10.10.10.13:21.
 
$ egrep -o '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' has_ipas
10.10.10.10
10.10.10.11
10.10.10.12
10.10.10.13

The possibilities are endless, and we haven’t even scratched the surface here.

This is the very essence of what the Unix toolchain idea is all about.

Take a number of small tools that do one thing well and combine them as needed to solve problems.

Also, the regex we used for IP addresses is naive and could match other things, including invalid addresses.

For a much better pattern, use the Perl Compatible Regular Expressions (PCRE) regex from Mastering Regular Expressions by Jeffrey E. F. Friedl (O’Reilly), if your grep supports -P. Or use Perl.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ grep -oP '([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\
d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])' has_ipas
10.10.10.10
10.10.10.11
10.10.10.12
10.10.10.13
 
$ perl -ne 'while ( m/([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.
([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])/g ) { print qq($1.$2.$3.
$4\n); }' has_ipas
10.10.10.10
10.10.10.11
10.10.10.12
10.10.10.13

 

Share
Tweet
Email
Prev Article
Next Article

Related Articles

Looping for a While
You want your shell script to perform some actions repeatedly …

Looping for a While

Seeing All Variable Values in bash
How can I see which variables have been exported and …

Seeing All Variable Values in bash

About The Author

InfinityCoder
InfinityCoder

Leave a Reply

Cancel reply

Recent Tutorials InfinityQuest

  • Adding New Features to bash Using Loadable Built-ins in bash
    Adding New Features to bash Using Loadable …
    June 27, 2017 0
  • Getting to the Bottom of Things in bash
    Getting to the Bottom of Things in …
    June 27, 2017 0

Recent Comments

  • fer on Turning a Dictionary into XML in Python
  • mahesh on Turning a Dictionary into XML in Python

Categories

  • Bash
  • PHP
  • Python
  • Uncategorized

InfinityQuest - Programming Code Tutorials and Examples with Python, C++, Java, PHP, C#, JavaScript, Swift and more

About Us

Start learning your desired programming language with InfinityQuest.com.

On our website you can access any tutorial that you want with video and code examples.

We are very happy and honored that InfinityQuest.com has been listed as a recommended learning website for students.

Popular Tags

binary data python CIDR convert string into datetime python create xml from dict python dictionary into xml python how to create xml with dict in Python how to write binary data in Python IP Address read binary data python tutorial string as date object python string to datetime python

Archives

  • June 2017
  • April 2017
  • February 2017
  • January 2017
  • December 2016
  • November 2016
Copyright © 2021 InfinityQuest - Programming Code Tutorials and Examples with Python, C++, Java, PHP, C#, JavaScript, Swift and more
Programming Tutorials | Sitemap