I'm beginning to think Linux Really!: August 2013

Monday, August 5, 2013

Advanced examples with sed command,

Sed - Stream Editor is a powerful command to modify the files in linux. Sed command is one of most useful commands used in shell scripting. Some of the sed examples

[root@machine2 ~]# cat test.txt
Linux is a Unix-like computer operating system. Linux is assembled under the model of free and open source software development and distribution.

Search and replace certain patterns in a file

Replaces first occurance of the pattern in file

[root@machine2 ~]# sed 's/Linux/unix/1' test.txt
unix is a Unix-like computer operating system. Linux is assembled under the model of free and open source software development and distribution.

Replaces second occurance of pattern in the file

[root@machine2 ~]# sed 's/Linux/unix/2' test.txt
Linux is a Unix-like computer operating system. unix is assembled under the model of free and open source software development and distribution.

Even this syntax replaces the first occurance.

[root@machine2 ~]# sed 's/Linux/unix/' test.txt
unix is a Unix-like computer operating system. Linux is assembled under the model of free and open source software development and distribution.

Replaces all the matching pattern in the file.

[root@machine2 ~]# sed 's/Linux/unix/g' test.txt
unix is a Unix-like computer operating system. unix is assembled under the model of free and open source software development and distribution.

[root@machine2 ~]# cat test.txt
Linux is a Unix-like computer operating system. Linux is assembled under the model of free and open source software development and distribution.Love to work with Linux.

Replaces the matching pattern from the second occurances to the last.

[root@machine2 ~]# sed 's/Linux/unix/2g' test.txt
Linux is a Unix-like computer operating system. unix is assembled under the model of free and open source software development and distribution.Love to work with unix.

[root@machine2 ~]# cat > test1.txt
http://google.com
http://yahoo.com

[root@machine2 ~]# cat test1.txt
http://google.com
http://yahoo.com

Replacing a url. Here \ is the delimiter

[root@machine2 ~]# sed 's/http:\/\//www/' test1.txt
wwwgoogle.com
wwwyahoo.com

[root@machine2 ~]# sed 's/http:\/\//www./' test1.txt
www.google.com
www.yahoo.com

We can also use _ as a delimiter

[root@machine2 ~]# sed 's_http://_www._' test1.txt
www.google.com
www.yahoo.com

Here | is the delimiter.

[root@machine2 ~]# sed 's|http:|_www.|' test1.txt
_www.//google.com
_www.//yahoo.com

[root@machine2 ~]# sed 's|http:|www.|' test1.txt
www.//google.com
www.//yahoo.com

[root@machine2 ~]# sed 's|http://|www.|' test1.txt
www.google.com
www.yahoo.com

[root@machine2 ~]# cat test.txt
Linux is a Unix-like computer operating system. Linux is assembled under the model of free and open source software development and distribution.Love to work with Linux.

Searches for a specific pattern and add some extra characters, symbols using &

[root@machine2 ~]# sed 's/Linux/{&}/' test.txt
{Linux} is a Unix-like computer operating system. Linux is assembled under the model of free and open source software development and distribution.Love to work with Linux.

[root@machine2 ~]# sed 's/Linux/,&/' test.txt
,Linux is a Unix-like computer operating system. Linux is assembled under the model of free and open source software development and distribution.Love to work with Linux.

[root@machine2 ~]# sed 's/Linux/&,/' test.txt
Linux, is a Unix-like computer operating system. Linux is assembled under the model of free and open source software development and distribution.Love to work with Linux.

To include the same patter once again use &&

only in First occurance.

[root@machine2 ~]# sed 's/Linux/{&&}/' test.txt
{LinuxLinux} is a Unix-like computer operating system. Linux is assembled under the model of free and open source software development and distribution.Love to work with Linux.

Includes the extra characters to all the matching pattern.

[root@machine2 ~]# sed 's/Linux/{&}/g' test.txt
{Linux} is a Unix-like computer operating system. {Linux} is assembled under the model of free and open source software development and distribution.Love to work with {Linux}.

Search and replaces the matching pattern and prints line double.

[root@machine2 ~]# sed 's/Linux/unix/p' test.txt
unix is a Unix-like computer operating system. Linux is assembled under the model of free and open source software development and distribution.Love to work with Linux.
unix is a Unix-like computer operating system. Linux is assembled under the model of free and open source software development and distribution.Love to work with Linux.

Search, replace and prints the line.

[root@machine2 ~]# sed -n 's/Linux/unix/p' test.txt
unix is a Unix-like computer operating system. Linux is assembled under the model of free and open source software development and distribution.Love to work with Linux.

Executing multiple sed commands in single line using | symbol and -e option

[root@machine2 ~]# sed 's/Linux/LINUXOS/' test.txt | sed 's/Unix/UnixOS/'
LINUXOS is a UnixOS-like computer operating system. Linux is assembled under the model of free and open source software development and distribution.Love to work with Linux.

[root@machine2 ~]# sed -e 's/Linux/LINUXOperatingSystem/' -e 's/Unix/UnixOperatingSystem/' test.txt
LINUXOperatingSystem is a UnixOperatingSystem-like computer operating system. Linux is assembled under the model of free and open source software development and distribution.Love to work with Linux.

[root@machine2 ~]# cat test.txt
Linux is a Unix-like computer operating system.
Linux is assembled under the model of free and open source software development and distribution.Love to work with Linux.
Learning Linux is very easy.

Replaces word Linux with unix in the second line of the file

[root@machine2 ~]# sed '2 s/Linux/unix/' test.txt
Linux is a Unix-like computer operating system.
unix is assembled under the model of free and open source software development and distribution.Love to work with Linux.
Learning Linux is very easy.

Replaces word Linux with unix in the first and second line of the file. We can also specify the line range.

[root@machine2 ~]# sed '1,2 s/Linux/unix/' test.txt
unix is a Unix-like computer operating system.
unix is assembled under the model of free and open source software development and distribution.Love to work with Linux.
Learning Linux is very easy.

Replaces word Linux with unix from first to last line of the file

[root@machine2 ~]# sed '1,$ s/Linux/unix/' test.txt
unix is a Unix-like computer operating system.
unix is assembled under the model of free and open source software development and distribution.Love to work with Linux.
Learning unix is very easy.

Search for a specific pattern and replaces it.

[root@machine2 ~]# sed '/Linux/ s/Linux/centos/' test.txt
centos is a Unix-like computer operating system.
centos is assembled under the model of free and open source software development and distribution.Love to work with Linux.
Learning centos is very easy.

Here, Search for the occurances a matching pattern and replaces it with the given word.

[root@machine2 ~]# sed '/Linux/ s/Unix/Fedora/' test.txt
Linux is a Fedora-like computer operating system.
Linux is assembled under the model of free and open source software development and distribution.Love to work with Linux.
Learning Linux is very easy.

In above searches for Unix and replaces with Fedora.

Prints the line twice in a file.

[root@machine2 ~]# sed 'p' test.txt
Linux is a Unix-like computer operating system.
Linux is a Unix-like computer operating system.
Linux is assembled under the model of free and open source software development and distribution.Love to work with Linux.
Linux is assembled under the model of free and open source software development and distribution.Love to work with Linux.
Learning Linux is very easy.
Learning Linux is very easy.

Like grep command fetches and prints the line where the matching string found.

[root@machine2 ~]# sed -n '/free/ p' test.txt
Linux is assembled under the model of free and open source software development and distribution.Love to work with Linux.

Prints the line other than the matching string line.

[root@machine2 ~]# sed -n '/free/ !p' test.txt
Linux is a Unix-like computer operating system.
Learning Linux is very easy.

Add a new line after the matching pattern.

[root@machine2 ~]# sed '/Learning/ a "Adding a new line" ' test.txt
Linux is a Unix-like computer operating system.
Linux is assembled under the model of free and open source software development and distribution.Love to work with Linux.
Learning Linux is very easy.
"Adding a new line"

Add a new line before the matching pattern.

[root@machine2 ~]# sed '/Learning/ i "Adding a new line" ' test.txt
Linux is a Unix-like computer operating system.
Linux is assembled under the model of free and open source software development and distribution.Love to work with Linux.
"Adding a new line"
Learning Linux is very easy.

Change the line where the matching string found.

[root@machine2 ~]# sed '/Learning/ c "chaning with a new line" ' test.txt
Linux is a Unix-like computer operating system.
Linux is assembled under the model of free and open source software development and distribution.Love to work with Linux.
"chaning with a new line"

[root@machine2 ~]# sed '/Learning/ c "changing with a new line" ' test.txt
Linux is a Unix-like computer operating system.
Linux is assembled under the model of free and open source software development and distribution.Love to work with Linux.
"changing with a new line"

Transfrom with the specific letters like tr command.

[root@machine2 ~]# sed 'y/L/l' test.txt
sed: -e expression #1, char 5: unterminated `y' command

[root@machine2 ~]# sed 'y/L/l/' test.txt
linux is a Unix-like computer operating system.
linux is assembled under the model of free and open source software development and distribution.love to work with linux.
learning linux is very easy

Remove specific lines in a file using sed command.

sed command examples to delete or remove specific lines in linux
Here file is the file name.

[root@machine1 ~]# cat > file
first
second
third
fourth
fifth
END

[root@machine1 ~]# cat file
first
second
third
fourth
fifth
END

Here 1 is the line number and d is the option to delete first line

[root@machine1~]# sed '1d' file
second
third
fourth
fifth
END

[root@machine1 ~]# cat file
first
second
third
fourth
fifth
END

This will not remove in the orignal file, to remove in original use -i option

[root@machine1 ~]# sed -i '1d' file

[root@machine1 ~]# cat file
second
third
fourth
fifth
END

We also have the option to redirect it to the other file
[root@machine1 ~]# sed '2d' file > file1

To delete a last line in linux.

[root@machine1 ~]# sed '$d' file
second
third
fourth
fifth
[root@machine1 ~]# cat file
second
third
fourth
fifth
END
[root@machine1 ~]# sed -i '$d' file

[root@machine1 ~]# cat file
second
third
fourth
fifth

Delete for specific range. here i delete from 2nd to 4th line

[root@machine1 ~]# sed '2,4d' file
second

[root@machine1 ~]# cat file
second
third
fourth
fifth

Delete except first line

[root@machine1 ~]# sed '1!d' file
second

Delete except last line

[root@machine1 ~]# sed '$!d' file
fifth

Delete except specific line range from 2 to 4

[root@machine1 ~]# sed '2,4!d' file
third
fourth
fifth

Delete first and last line, here ; specify multiple conditions

[root@machine1 ~]# sed '1d;$d' file
third
fourth

Delete blank or empty lines

[root@machine1 ~]# sed '/^$/d' file
second
third
fourth
fifth

Delete line starts with pariticular characters

[root@machine1 ~]# sed '/^f/d' file
second
third

Delete line end with particular character

[root@machine1 ~]# sed '/d$/d' file
fourth
fifth

[root@machine1 ~]# cat file
second
third
fourth
fifth
LINUX
UBUNTU

Delete lines in upper case

[root@machine1 ~]# sed '/^[A-Z]*$/d' file
second
third
fourth
fifth

Delete lines that contain a specific pattern here i delete a line starts with a pattern UBUNTU

[root@machine1 ~]# sed '/UBUNTU/d' file
second
third
fourth
fifth
LINUX

Delete line starting from a pattern till the last line.

[root@machine1 ~]# sed '/fifth/,$d' file
second
third
fourth

Delete last line only if it contains a specific pattern

[root@machine1 ~]# sed '${/UBUNTU/d;}' file
second
third
fourth
fifth
LINUX

prints specific line which contain a pattern fifth

[root@machine1 ~]# cat file
second
third
fourth
fifth
LINUX
UBUNTU

[root@machine1 ~]# sed -n '/fifth/p' file
fifth

To print between 2 patterns

[root@machine1 ~]# sed -n '/fifth/,/UBUNTU/p' file
fifth
LINUX
UBUNTU

Add blank lines using sed command.

Add a double blank lines after each line

sed G test.txt

Linux is a Unix-like computer operating system.

Linux is assembled under the model of free and open source software development and distribution.Love to work with Linux.

Learning Linux is very easy.

Add triple blank lines after each lines.

[root@machine2 ~]# sed 'G;G' test.txt
Linux is a Unix-like computer operating system.

Linux is assembled under the model of free and open source software development and distribution.Love to work with Linux.

Learning Linux is very easy.

[root@machine2 ~]# sed -i 'G;G' test.txt

[root@machine2 ~]# cat test.txt
Linux is a Unix-like computer operating system.

Linux is assembled under the model of free and open source software development and distribution.Love to work with Linux.

Learning Linux is very easy.

Undo double space.

[root@machine2 ~]# sed 'n;d' test.txt
Linux is a Unix-like computer operating system.

Learning Linux is very easy.

Add a single blank line above the matching pattern.

[root@machine2 ~]# sed '/Redhat/{x;p;x}' test.txt
Linux is a Unix-like computer operating system.

Learning Linux is very easy.

Redhat Enterprise linux

[root@machine2 ~]# cat test.txt
Linux is a Unix-like computer operating system.

Learning Linux is very easy.
Redhat Enterprise linux

Add a blank line below the matching pattern.

[root@machine2 ~]# sed '/Redhat/G' test.txt
Linux is a Unix-like computer operating system.

Learning Linux is very easy.
Redhat Enterprise linux

Add a blank line above and below the matching pattern.

[root@machine2 ~]# sed '/Redhat/{x;p;x;G}' test.txt
Linux is a Unix-like computer operating system.

Learning Linux is very easy.

Redhat Enterprise linux

Number the lines in linux

[root@machine2 ~]# sed = test.txt | sed 'N;s/\n/\t/'
1 Linux is a Unix-like computer operating system.
2
3 Learning Linux is very easy.
4 Redhat Enterprise linux

[root@machine2 ~]# sed = test.txt
1
Linux is a Unix-like computer operating system.
2

3
Learning Linux is very easy.
4
Redhat Enterprise linux

N join the lines.

The 's/\n/\t/' replaces the newline chars with tabs,

1 Linux is a Unix-like computer operating system.
2
3 Learning Linux is very easy.
4 Redhat Enterprise linux

Cound the number of lines in a file

[root@machine2 ~]# sed -n '$=' test.txt
4

Print number of lines in a file.

[root@machine2 ~]# sed 2q test.txt
Linux is a Unix-like computer operating system.

Prints specific line number in a file

Advanced examples with awk command.

[root@machine2 test]# awk '1;{print ""}' testdoc
total 4.0K

-rw-r--r-- 1 root root 166 Jan 25 18:53 for.sh

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing1

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing2

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing4

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing3

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing6

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing5

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing8

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing7

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing9

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing10

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing12

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing11

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing13

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing14

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing15

-rw-r--r-- 1 root root 0 Jun 9 17:46 testdoc

[root@machine2 test]# awk 'BEGIN{ORS="\n\n"}' testdoc
ORS- output row separator

[root@machine2 test]# awk 'BEGIN{ORS="\n\n"};1' testdoc
total 4.0K

-rw-r--r-- 1 root root 166 Jan 25 18:53 for.sh

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing1

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing2

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing4

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing3

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing6

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing5

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing8

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing7

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing9

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing10

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing12

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing11

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing13

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing14

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing15

-rw-r--r-- 1 root root 0 Jun 9 17:46 testdoc

[root@machine2 test]# awk 'NF{print $0 "\n"}' testdoc

triple space a file

[root@machine2 test]# awk '1;{print "\n"}' testdoc
total 4.0K

-rw-r--r-- 1 root root 166 Jan 25 18:53 for.sh

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing1

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing2

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing4

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing3

-rw-r--r-- 1 root root 0 Jun 9 17:43 testing6

Print each line with its line number using thebelow syntax

[root@machine2 test]# awk '{print NR "\t" $0}' testdoc
1 total 4.0K
2 -rw-r--r-- 1 root root 166 Jan 25 18:53 for.sh
3 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing1
4 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing2
5 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing4
6 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing3
7 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing6
8 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing5
9 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing8
10 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing7
11 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing9
12 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing10
13 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing12
14 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing11
15 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing13
16 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing14
17 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing15
18 -rw-r--r-- 1 root root 0 Jun 9 17:46 testdoc
[root@machine2 test]# awk '{print FNR "\t" $0}' testdoc
1 total 4.0K
2 -rw-r--r-- 1 root root 166 Jan 25 18:53 for.sh
3 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing1
4 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing2
5 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing4
6 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing3
7 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing6
8 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing5
9 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing8
10 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing7
11 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing9
12 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing10
13 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing12
14 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing11
15 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing13
16 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing14
17 -rw-r--r-- 1 root root 0 Jun 9 17:43 testing15
18 -rw-r--r-- 1 root root 0 Jun 9 17:46 testdoc

[root@machine2 test]# awk '{printf("%5d : %s\n", NR,$0)}' testdoc
1 : total 4.0K
2 : -rw-r--r-- 1 root root 166 Jan 25 18:53 for.sh
3 : -rw-r--r-- 1 root root 0 Jun 9 17:43 testing1
4 : -rw-r--r-- 1 root root 0 Jun 9 17:43 testing2
5 : -rw-r--r-- 1 root root 0 Jun 9 17:43 testing4
6 : -rw-r--r-- 1 root root 0 Jun 9 17:43 testing3
7 : -rw-r--r-- 1 root root 0 Jun 9 17:43 testing6
8 : -rw-r--r-- 1 root root 0 Jun 9 17:43 testing5
9 : -rw-r--r-- 1 root root 0 Jun 9 17:43 testing8
10 : -rw-r--r-- 1 root root 0 Jun 9 17:43 testing7
11 : -rw-r--r-- 1 root root 0 Jun 9 17:43 testing9
12 : -rw-r--r-- 1 root root 0 Jun 9 17:43 testing10
13 : -rw-r--r-- 1 root root 0 Jun 9 17:43 testing12
14 : -rw-r--r-- 1 root root 0 Jun 9 17:43 testing11
15 : -rw-r--r-- 1 root root 0 Jun 9 17:43 testing13
16 : -rw-r--r-- 1 root root 0 Jun 9 17:43 testing14
17 : -rw-r--r-- 1 root root 0 Jun 9 17:43 testing15
18 : -rw-r--r-- 1 root root 0 Jun 9 17:46 testdoc

Left align and right align using the above syntax

[root@machine2 test]# awk '{ total = total + NF }; END {print total}' testdoc
155
print the total number of fields ("words") in all lines

[root@machine2 test]# awk '{print $NF}' testdoc
4.0K
for.sh
testing1
testing2
testing4
testing3
testing6
testing5
testing8
testing7
testing9
testing10
testing12
testing11
testing13
testing14
testing15
testdoc

Prints the last filed in every line with above syntax
[root@machine2 test]# awk '{print NF ":" $0 }' testdoc
2:total 4.0K
9:-rw-r--r-- 1 root root 166 Jan 25 18:53 for.sh
9:-rw-r--r-- 1 root root 0 Jun 9 17:43 testing1
9:-rw-r--r-- 1 root root 0 Jun 9 17:43 testing2
9:-rw-r--r-- 1 root root 0 Jun 9 17:43 testing4
9:-rw-r--r-- 1 root root 0 Jun 9 17:43 testing3
9:-rw-r--r-- 1 root root 0 Jun 9 17:43 testing6
9:-rw-r--r-- 1 root root 0 Jun 9 17:43 testing5
9:-rw-r--r-- 1 root root 0 Jun 9 17:43 testing8
9:-rw-r--r-- 1 root root 0 Jun 9 17:43 testing7
9:-rw-r--r-- 1 root root 0 Jun 9 17:43 testing9
9:-rw-r--r-- 1 root root 0 Jun 9 17:43 testing10
9:-rw-r--r-- 1 root root 0 Jun 9 17:43 testing12
9:-rw-r--r-- 1 root root 0 Jun 9 17:43 testing11
9:-rw-r--r-- 1 root root 0 Jun 9 17:43 testing13
9:-rw-r--r-- 1 root root 0 Jun 9 17:43 testing14
9:-rw-r--r-- 1 root root 0 Jun 9 17:43 testing15
9:-rw-r--r-- 1 root root 0 Jun 9 17:46 testdoc

Prints number of fields followed by a character in each line.

[root@machine2 test]# awk '{test=$NF}; END{print test }' testdoc
testdoc

[root@machine2 test]# awk 'END{print $NF}' testdoc
testdoc

Find the last field of the last line using the above syntaxes

[root@machine2 test]# awk 'NF > 3' testdoc
-rw-r--r-- 1 root root 166 Jan 25 18:53 for.sh
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing1
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing2
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing4
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing3
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing6
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing5
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing8
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing7
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing9
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing10
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing12
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing11
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing13
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing14
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing15
-rw-r--r-- 1 root root 0 Jun 9 17:46 testdoc

Prints number of fields larger than 3

[root@machine2 test]# awk 'NF < 3' testdoc
total 4.0K

Prints number of fields less than 3

[root@machine2 test]# awk 'NR < 4' testdoc
total 4.0K
-rw-r--r-- 1 root root 166 Jan 25 18:53 for.sh
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing1

[root@machine2 test]# awk 'NR <=4' testdoc
total 4.0K
-rw-r--r-- 1 root root 166 Jan 25 18:53 for.sh
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing1
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing2

Prints number of rows less than 4 and equal to 4 using above syntax.

[root@machine2 test]# awk '/total/' testdoc
total 4.0K
search for a pattern using above syntax.

[root@machine2 test]# awk '!/total/' testdoc
-rw-r--r-- 1 root root 166 Jan 25 18:53 for.sh
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing1
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing2
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing4
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing3
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing6
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing5
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing8
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing7
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing9
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing10
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing12
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing11
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing13
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing14
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing15
-rw-r--r-- 1 root root 0 Jun 9 17:46 testdoc

Prints other than the searching pattern using above syntax.

[root@machine2 test]# awk '/total/; /for/' testdoc
total 4.0K
-rw-r--r-- 1 root root 166 Jan 25 18:53 for.sh

search multiple pattern using above syntax.

[root@machine2 test]# awk 'length> 48' testdoc
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing10
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing12
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing11
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing13
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing14
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing15

print only line above 48 characters through above syntax.

[root@machine2 test]# awk 'length< 15' testdoc
total 4.0K
prints less than 15 characters

[root@machine2 test]# awk 'NR==8,NR==12' testdoc
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing5
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing8
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing7
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing9
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing10

Prints specific rows from 8 to 12 with the above syntaxes

[root@machine2 test]# awk 'NR==12' testdoc
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing10

prints specific line.

awk command with examples

[root@machine2 test]# for i in `seq 1 1 15`; do touch testing$i; done
[root@machine2 test]# ls
for.sh testing10 testing12 testing14 testing2 testing4 testing6 testing8
testing1 testing11 testing13 testing15 testing3 testing5 testing7 testing9
[root@machine2 test]# ls -lrth
total 4.0K
-rw-r--r-- 1 root root 166 Jan 25 18:53 for.sh
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing1
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing2
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing4
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing3
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing6
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing5
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing8
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing7
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing9
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing10
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing12
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing11
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing13
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing14
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing15

[root@machine2 test]# ls -lrth > testdoc

To fetch only the column type $followed by the column number.

[root@machine2 test]# cat testdoc | awk '{print$1}'
total
-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--

[root@machine2 test]# cat testdoc | awk '{print$1,$2}'
total 4.0K
-rw-r--r-- 1
-rw-r--r-- 1
-rw-r--r-- 1
-rw-r--r-- 1
-rw-r--r-- 1
-rw-r--r-- 1
-rw-r--r-- 1
-rw-r--r-- 1
-rw-r--r-- 1
-rw-r--r-- 1
-rw-r--r-- 1
-rw-r--r-- 1
-rw-r--r-- 1
-rw-r--r-- 1
-rw-r--r-- 1
-rw-r--r-- 1
-rw-r--r-- 1

[root@machine2 test]# cat testdoc | awk '{print$1,$5}'
total
-rw-r--r-- 166
-rw-r--r-- 0
-rw-r--r-- 0
-rw-r--r-- 0
-rw-r--r-- 0
-rw-r--r-- 0
-rw-r--r-- 0
-rw-r--r-- 0
-rw-r--r-- 0
-rw-r--r-- 0
-rw-r--r-- 0
-rw-r--r-- 0
-rw-r--r-- 0
-rw-r--r-- 0
-rw-r--r-- 0
-rw-r--r-- 0
-rw-r--r-- 0

[root@machine2 test]# cat testdoc | awk '{print$1,$6}'
total
-rw-r--r-- Jan
-rw-r--r-- Jun
-rw-r--r-- Jun
-rw-r--r-- Jun
-rw-r--r-- Jun
-rw-r--r-- Jun
-rw-r--r-- Jun
-rw-r--r-- Jun
-rw-r--r-- Jun
-rw-r--r-- Jun
-rw-r--r-- Jun
-rw-r--r-- Jun
-rw-r--r-- Jun
-rw-r--r-- Jun
-rw-r--r-- Jun
-rw-r--r-- Jun
-rw-r--r-- Jun

[root@machine2 test]# cat testdoc | awk '{print$1,$9}'
total
-rw-r--r-- for.sh
-rw-r--r-- testing1
-rw-r--r-- testing2
-rw-r--r-- testing4
-rw-r--r-- testing3
-rw-r--r-- testing6
-rw-r--r-- testing5
-rw-r--r-- testing8
-rw-r--r-- testing7
-rw-r--r-- testing9
-rw-r--r-- testing10
-rw-r--r-- testing12
-rw-r--r-- testing11
-rw-r--r-- testing13
-rw-r--r-- testing14
-rw-r--r-- testing15
-rw-r--r-- testdoc

[root@machine2 test]# awk 'BEGIN {total=0} {total=total+$5} END {print total}' testdoc
166

The above syntax will print the sum of 5th column in the testdoc file. Here total is the variable assigned to zero in the begin syntax.
In the action block, the value of 5th column is passed for each row .After all values are processed, it prints the total value in END block.

[root@machine2 test]# awk -f totalscript testdoc
166

[root@machine2 test]# cat totalscript
#!/usr/bin/awk -f
BEGIN {total=0} {total=total+$5} END {print total}

We can use the script in a file and use -f option to run it.

[root@machine2 test]# awk '{if($9 == "testing9") print$0;}' testdoc
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing9

[root@machine2 test]# awk 'BEGIN{for(i=1;i<=10;i++) print "square of", i, "is", i*i;}'
square of 1 is 1
square of 2 is 4
square of 3 is 9
square of 4 is 16
square of 5 is 25
square of 6 is 36
square of 7 is 49
square of 8 is 64
square of 9 is 81
square of 10 is 100

[root@machine2 test]# awk -F: '{print$3}' /etc/passwd
0
1
2
3
4
5
6
7
8
10
11
12
13
14
99
81
170
69
32
499
173
68
38
498
89
27
70
29
65534
497
74
72
55
28
65
96
52
496
495
48
50022
75
107
141
14
Here -F is the field separator . We can also use awk 'BEGIN {FS=":"} {print $3}' /etc/passwd

[root@machine2 test]# awk '{print$8,$9}' testdoc

18:53 for.sh
17:43 testing1
17:43 testing2
17:43 testing4
17:43 testing3
17:43 testing6
17:43 testing5
17:43 testing8
17:43 testing7
17:43 testing9
17:43 testing10
17:43 testing12
17:43 testing11
17:43 testing13
17:43 testing14
17:43 testing15
17:46 testdoc
0

[root@machine2 test]# awk 'BEGIN{OFS=":"}{print$8,$9}' testdoc
:
18:53:for.sh
17:43:testing1
17:43:testing2
17:43:testing4
17:43:testing3
17:43:testing6
17:43:testing5
17:43:testing8
17:43:testing7
17:43:testing9
17:43:testing10
17:43:testing12
17:43:testing11
17:43:testing13
17:43:testing14
17:43:testing15
17:46:testdoc

OFS- Output Field separator , we add any character with this option

[root@machine2 test]# ls -l
total 12
-rw-r--r-- 1 root root 166 Jan 25 18:53 for.sh
-rw-r--r-- 1 root root 847 Jun 9 17:46 testdoc
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing1
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing10
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing11
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing12
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing13
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing14
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing15
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing2
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing3
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing4
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing5
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing6
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing7
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing8
-rw-r--r-- 1 root root 0 Jun 9 17:43 testing9
-rw-r--r-- 1 root root 71 Jun 9 17:59 totalscript

[root@machine2 test]# awk '{print NF}' testdoc
2
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9

NF - Number of field variables This will print number of columns in each row

[root@machine2 test]# awk '{print NR}' testdoc
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
NR- Number of records, prints count of lines in a file or line number

For total line numbers type the below syntax

[root@machine2 test]# awk 'END{print NR}' testdoc
18

to search the contents in awk with out casesensitive IGNORECASE=1

Ethtool - Display or change ethernet card settings

ethtool - Display or change ethernet card settings

Full duplex -sending and receiving data simultaneously.
half duplex - half duplex, you can either send or receive data at a time (
to find nic card half or full duplex

[root@machine1 ~]# dmesg | grep -i duplex
e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx

[root@machine1 ~]# ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 2
Transceiver: internal
Auto-negotiation: on
MDI-X: off
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
Link detected: yes

[root@machine1 ~]# mii-tool eth0
eth0: negotiated 100baseTx-FD flow-control, link ok

You can change the ethernet setttings using ethtool -s option

ethtool -s eth0 speed 100 duplex full
ethtool -s eth0 speed 10 duplex half

OR

mii-tool -F 100baseTx-HD
mii-tool -F 10baseT-HD

I'm beginning to think Linux Really!