All About Linux

Network Configuration

Set static IP address

The primary network interface

1
2
3
4
5
6
7
$ sudo nano /etc/network/interfaces 
auto eth0
iface eth0 inet static
address 192.168.1.X
netmask 255.255.255.0
gateway 192.168.1.X
dns-nameservers X.X.X.X
1
$ sudo ifdown eth0 && sudo ifup eth0

DNS nameserver configuration (Ubuntu 18.04)

Solution 1

Keep it mind that Ubuntu appends a tail file to the resolv.conf file that it generates. Editing the tail file instead of the actual resolv.conf file will prevent your change from being lost upon a system reboot.

1
2
$ sudo nano /etc/resolvconf/resolv.conf.d/tail
$ nameserver 8.8.8.8

You will need to activate the change via

1
$ sudo resolvconf -u

Solution 2

1
2
3
4
$ sudo nano /etc/resolv.conf
# Generated by resolvconf
nameserver 8.8.8.8
nameserver 8.8.4.4
1
$ sudo nano /etc/resolv.conf

Dual NIC configuration via NAT

Set static IP address

This subnet 192.168.1.0/24 will act as a Internet Facing interface
auto ens33

1
2
3
4
5
6
7
8
$ sudo nano /etc/network/interfaces
iface ens33 inet static
address 192.168.1.220
netmask 255.255.255.0
network 192.168.1.0
broadcast 192.168.1.255
gateway 192.168.1.1
dns-nameservers 8.8.8.8

This subnet 10.0.0.0/24 will act as a gateway of LAN PCs

1
2
3
4
5
6
7
8
$ sudo nano /etc/network/interfaces
auto ens38
iface ens38 inet static
address 10.0.0.1
netmask 255.255.255.0
network 10.0.0.0
broadcast 10.0.0.255
dns-nameservers 8.8.8.8

Restart the networking function

1
2
3
4
5
6
$ sudo /etc/init.d/networking restart
```
Alternatively, you can
```
$ sudo ifup eth0
$ sudo ifdown eth0
Enable IP Forwarding
1
2
$ sudo nano /etc/sysctl.conf
net.ipv4.ip_forward=1

Alternatively, you can

1
$ sudo echo 1 > /proc/sys/net/ipv4/ip_forward

Set iptable Rules

Step 1: There’re four rule sets availabe and we will use the first two

  • FILTER (Default)
  • NAT
  • MANGLE
  • RAW
1
2
3
4
$ sudo iptables -X
$ sudo iptables -F
$ sudo iptables -t nat -X
$ sudo iptables -t nat -F

Step 2: Set accepted destination of forwarding chain

1
2
$ sudo iptables -I INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
$ sudo iptables -I FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT

Step 3: Enable IP Masquerade

1
$ sudo iptables -t nat -I POSTROUTING -o ens33 -j MASQUERADE

Alternative Method

1
2
3
4
5
6
$ sudo iptables -F
$ sudo iptables -t nat -F
$ sudo iptables -t mangle -F
$ sudo iptables -P INPUT ACCEPT
$ sudo iptables -P FORWARD ACCEPT
$ sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

Enable DNAT

External Client via [NIC_1: 192.168.1.220:22030 | NIC_2: 10.0.0.1] –> 10.0.0.200:22

1
$ sudo iptables -t nat -A PREROUTING -d 192.168.1.220 -p tcp --dport 22030 -j DNAT --to 10.0.0.200:22

Save iptables Rules persistently

On Ubuntu, the easiest way to save iptables rules, so they will survive a reboot, is to use the iptables-persistent package. Install it with apt-get like this:

1
$ sudo apt-get install iptables-persistent

During the installation, you will asked if you want to save your current firewall rules. If you update your firewall rules and want to save the changes, run this command:

1
$ sudo netfilter-persistent save

On versions of Ubuntu prior to 16.04, run this command instead:

1
$ sudo invoke-rc.d iptables-persistent save

SSH without using password

Step 1: Generate a public/private key pair using ssh-kengen
1
2
3
$ ssh-keygen
Generating public/private rsa key pair.
... ...
Step 2: Copy the key to the destination server

ssh-copy-id 命令可以把本地主机的公钥复制到远程主机的authorized_keys文件上

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ ssh-copy-id server01
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/vagrant/.ssh/id_rsa.pub"
The authenticity of host 'server01 (10.9.8.11)' can't be established.
ECDSA key fingerprint is SHA256:Fi4FisVgFyEkos9NgKz0q+zzZwe3+xhCHWGrXL+jZck.
ECDSA key fingerprint is MD5:b6:04:55:d7:db:3c:a8:a1:b6:f6:15:1f:be:7e:48:41.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
vagrant@server01's password:

Number of key(s) added: 1

Now try logging into the machine, with: "ssh 'server01'"
and check to make sure that only the key(s) you wanted were added.

[vagrant@admin01 ~]$ ssh server01
[vagrant@server01 ~]$ exit
Easy for system admin to manage various servers
1
2
3
4
5
6
7
8
9
$ for SERVER in $(cat server)
> do
> ssh ${SERVER} hostname
> ssh ${SERVER} uptime
> done
server01
03:29:44 up 26 min, 0 users, load average: 0.00, 0.01, 0.05
server02
03:29:44 up 25 min, 0 users, load average: 0.00, 0.01, 0.04

One time command running with SSH

1
2
3
$ ssh server01 'hostname ; hostname'
server01
server01

Remote system health checking with SSH

1
2
3
4
$ ssh server01 "ps -ef | head -3"
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 03:03 ? 00:00:00 /usr/lib/systemd/systemd --switched-root --system --deserialize 21
root 2 0 0 03:03 ? 00:00:00 [kthreadd]

System Running

Start a service when booting system

For ubuntu 16.04 and above, creat a service file named startup.service

1
2
3
4
5
6
7
8
9
$ sudo nano /etc/systemd/system/startup.service
[Unit]
Description=Startup Applications
[Service]
Type=oneshot
RemainAfterExit=false
ExecStart=/etc/rc.local start
[Install]
WantedBy=multi-user.target

Create a rc.local file

1
2
$ cd /etc
$ sudo touch rc.local

For example, auto-start IP Masquerade service for NAT

1
2
3
#!/bin/sh
iptables -t nat -I POSTROUTING -o ens33 -j MASQUERADE
exit 0

To make it executable

1
$ sudo chmod +x /etc/rc.local

Then go back to /etc/systemd/system

1
2
3
$ sudo systemctl enable startup
$ sudo systemctl start startup
$ sudo systemctl status startup

System runnig process and port number

To find which port the system is listening

1
$ netstat -a

To find a process number associated with port 12345 on TCP

1
2
$ fuser 12345/tcp
12345/tcp: 490

Next, to find what (service/program) that particular process number is runnning

1
2
$ ps -ef | grep 490
root 490 483 0 Sep19 ? 00:02:17 /usr/local/bin/license_manager

Kill a running port

1
2
3
$ sudo netstat -lpn | grep 4000
$ sudo fuser -k 4000/tcp
$ sudo kill -9 PID

[LINUX] Other commands to display programs associated with ports

1
2
3
4
## To find the process ID associated with the service listening on port 901
$ lsof -i TCP:901
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
inetd 387 root 9u IPv4 369 TCP *:swat (LISTEN)

[WINDOWS]

A Windows tool known as fport (available with a tool search on www.securityfocus.com) can
be used to associate processes with ports on which they run.

File System

Using ‘df’ command

Show information about the linux file system on which each FILE resides,
or all file systems by default.

1
2
3
4
5
6
7
8
9
10
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 30G 7.4G 21G 27% /
devtmpfs 460M 0 460M 0% /dev
tmpfs 464M 0 464M 0% /dev/shm
tmpfs 464M 19M 446M 4% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 464M 0 464M 0% /sys/fs/cgroup
/dev/mmcblk0p1 41M 21M 21M 51% /boot
tmpfs 93M 0 93M 0% /run/user/1000

Using ‘du’ command

Summarize disk usage of the set of FILEs, recursively for directories.

Size of a file:

1
$ du -h xxx

Size of a directory (recursively):

1
$ du -h /tmp

Cron Jobs

Cron Job Types

  • You should use crontab -e to edit user crontabs (including root’s);
  • User crontabs don’t have a user field, but the system crontabs do.

User cron jobs (e.g. root) are located in /var/spool/cron/crontabs

1
2
3
4
5
6
7
$ cat /var/spool/cron/crontabs/xx(root)
... ...
# m h dom mon dow command
*/5 * * * * broctl cron
59 23 * * * rm -rf /nsm/bro/logs/20??-*-*
59 23 * * * rm -rf /nsm/bro/extracted/
... ...

We can use crontab to modify the file

1
$ sudo crontab -e

System-wide Cron Directory (there’s a ‘user’ field in the file) /etc/crontab

1
2
3
4
5
6
$ sudo cat /etc/crontab
... ...
# m h dom mon dow user command
*/5 * * * * root /nsm/scripts/cleanup
30 * * * * root /nsm/scripts/update
... ...

Bash Shell

Shelll variables, useful commands

Bash Shell Built-in Variables

1
2
3
4
5
6
7
8
${0} -> Display the command just executed
${#} -> Total number of positional parameters
$ echo "Parameter 1: ${1}"
$ echo "Parameter 2: ${2}"
$ echo "Parameter 3: ${3}"
${?} -> The return/exit number of last execution
${@} -> Using in FOR LOOP to represent all the arguments one by one | When used in quotes: "$1" "$2"...
${*} -> Represent the arguments as a whole ${1} | When used in quotes: "$1 $2..."

1
2
$ echo ${UID}
501
1
2
$ echo ${RANDOM}
15718
1
2
$ echo ${HOST}
MBP2017.local

Command which shows the full path of (shell) commands.

1
2
$ which head
/usr/bin/head

Command locate (N.B. need to run $ sudo updatedb to reflect the current searching list)

1
2
system admin command —> /usr/sbin/*
normal command —> /usr/sbin/*

Command find with /dev/null to ignore any “find: ‘xxx’: Permission denied”

1
2
3
4
5
6
$ find /usr/sbin/ -name userdel
/usr/sbin/userdel

$ find / -name userdel 2> /dev/null
/usr/sbin/userdel
/usr/share/bash-completion/completions/userdel

Command basename returns the file without directory path

1
2
$ basename /vagrant/luser-demo06.sh
$ luser-demo06.sh

Command dirname returns the directory path without file name

1
2
$ dirname /vagrant/luser-demo06.sh
$ /vagrant

Command let can be used as double parentheses such as NUM=$(( NUM += 5 ))

1
2
3
$ let NUM='2 + 3'
$ echo ${NUM}
5

Command expr can be used as a calculator (with white space)

1
2
3
4
$ expr 1 + 1
2
$ expr 5 * 2
10

The if statement, for loop, while loop and getops

‘if’ statement

1
2
3
4
5
6
if [[ COMMANDS ]]
then
COMMANDS
else
COMMANDS
fi

‘for’ loop

1
2
3
4
for VARIABLE in LIST
do
COMMANDS
done

‘while’ loop

1
2
3
4
while [[ COMMANDS ]]
do
COMMANDS
done

‘getopts’ with while

1
2
3
4
5
6
7
8
9
10
11
while getopts vl:s OPTION
do
case ${OPTION} in
v) … ;;
l) … ;;
s) … ;;
?)
usage
;;
esac
done

Program exit statues

1
2
0 = true / successful
1 = false / unsuccessful

Any non-zero exit status represents a failure.

1
2
exit 1
echo ${?} # You can also use $?

I/O Redirection - Pipes

File Descriptor

1
2
3
FD 0 - STDIN
FD 1 - STDOUT
FD 2 - STDERR

File (Command) I/O Redirection

1
2
3
4
5
6
7
8
9
COMMAND > /path/to/file
COMMAND >> /path/to/file
COMMAND < /path/to/file
COMMAND 2> /path/to/file
COMMAND &> /path/to/file

COMMAND |& COMMAND
COMMAND >&2
COMMAND > /dev/null

Sending STDOUT as STDIN

1
echo ${PWORD} | passwd --stdin ${NAME}

String manipulation & data munging with pipes

1
echo '!@#$%^' | fold -w1 | shuf | head -c1

In a pipe, the standard input (>) comes from the output of command

1
$ echo "secret" > my_passwd_in_file

With the help of less-than-simple (<), the standard input comes from a file

1
$ sudo passwd --stdin einstein < my_passwd_in_file

You cannot echo a FILE but you can echo a VARIABLE such as ${FILE} acting as a file name

1
2
3
4
FILE="/tmp/data"
head -n3 /etc/passwd > ${FILE}
echo "The file name (path) ${FILE} has the content of:"
cat ${FILE}

Examples

1
2
3
4
5
6
7
$ read X < /etc/centos-release
$ echo ${X}
CentOS Linux release 7.4.1708 (Core)

$ read X 0< /etc/centos-release
$ echo ${X}
CentOS Linux release 7.4.1708 (Core)

Using 2>&1 to redirect the stderr to the front stdout (2>&1), which is stdboth

1
2
[New Method] $ head -n1 /etc/passwd /etc/hosts /nofilelol &> stdboth
[Old Method] $ head -n1 /etc/passwd /etc/hosts /nofilelol 1> stdboth 2>&1 (&1 stands for stdout put)

Using cat to show stdout and stderrPlease note stderr does not float to the pipe … only stdout …

1
2
$ head -n1 /etc/passwd /etc/hosts /nofilelol 2>&1 | cat -n
$ head -n1 /etc/passwd /etc/hosts /nofilelol |& cat -n

Send output to stderr

1
echo "This is STDERR!" 1>&2

To throw whatever to that you don’t want to see using /dev/null

1
2
$ head -n1 /etc/passwd /etc/hosts /nofilelol 2> /dev/null 
(so that we don't see any error messages)

File Editing (grep, cut, awk, sort, wc, tww)

Using “echo -e” along with “cut” command

Option “-e” enable interpretation of backslash escapes.

1
2
3
4
5
6
7
# For example, the "\t" stands for horizontal tab
$ echo -e "one\ttwo\tthree" | cut -f 1
one
$ echo -e "one\ttwo\tthree" | cut -f 2
two
$ echo -e "one\ttwo\tthree" | cut -f 3
three
The “grep” command with “Reg”

Match every line which starts with ‘first’

1
$ grep '^first' /etc/passwd

Match every line which ends with ‘bash’

1
$ grep 'bash$' /etc/passwd

Match every line which starts with ‘first’ and ends with ‘end’

1
$ cat /etc/passwd | grep '^test' | grep 'bash$'

Use ‘-v’ option to inverse the match result (unmatch the reg below)

1
$ grep -Ev '^root|bash$' /etc/passwd

The “cut” command

Change the delimiter from ‘:’ to ‘,’

1
2
3
4
$ cut -d ':' -f 1,2 --output-delimiter=',' /etc/passwd
root,x
bin,x
... ...

The “awk” command

Scenario 1 - When we need more than one character as a delimiter

[Example 1] Using ‘/home’ as a delimiter

1
2
3
4
5
6
7
8
9
$ cat /etc/passwd | grep '^test0' | grep 'bash$'
test01:x:1027:1028::/home/test01:/bin/bash
test02:x:1028:1029::/home/test02:/bin/bash
$ cat /etc/passwd | grep '^test0' | grep 'bash$' | awk -F '/home' '{print $1}'
test01:x:1027:1028::
test02:x:1028:1029::
$ cat /etc/passwd | grep '^test0' | grep 'bash$' | awk -F '/home' '{print $2}'
/test01:/bin/bash
/test02:/bin/bash

[Example 2] See the differece between ‘cut’ and ‘awk’

1
2
3
4
$ cut -d ':' -f 1,3 /etc/passwd
root:0
$ awk -F ':' '{print $1, $3}' /etc/passwd
root 0

[Example 3] Modify the delimiter between $1 and $3 flexibly

1
2
3
4
5
6
7
8
$ awk -F ':' -v OFS=‘_#_’ '{print $1, $3}' /etc/passwd
root_#_0
$ awk -F ':' '{print $1 "_#_" $3}' /etc/passwd
root_#_0
$ awk -F ':' '{print " NAME:" $1 " UID:" $3}' /etc/passwd
NAME:root UID:0
$ awk -F ':' '{print "UID:" $3 " NAME:" $1}' /etc/passwd
UID:0 NAME:root

[Example 4] Print the last field using $NF, along with math within parentheses ()

1
2
3
4
5
6
$ awk -F ':' '{print $NF}' /etc/passwd
/bin/bash
$ awk -F ':' '{print "Shell: " $NF}' /etc/passwd
Shell: /bin/bash
$ awk -F ':' '{print $(NF-1)}' /etc/passwd
/root

Scenario 2 - Consider non-white space characters to be a field by default, so we can use ‘awk’ to easily seperate file via white space

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
$ echo 'L1      L2' > lines
$ echo 'L3 L4' >> lines
$ echo 'L5 L6' >> lines
$ echo 'L7 L8' >> lines
$ cat lines
L1 L2
L3 L4
L5 L6
L7 L8
$ awk '{print $1 $2}' lines
L1L2
L3L4
L5L6
L7L8
$ awk '{print $1, $2}' lines
L1 L2
L3 L4
L5 L6
L7 L8
The “sort” command with –field-separator=SEP and –key=KEYDEF
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# [option -t, --field-separator=SEP] use SEP instead of non-blank to blank transition
# [option -k, --key=KEYDEF] sort via a key; KEYDEF gives location and type
# [option -n, --numeric-sort] compare according to string numerical value
# [option -r, --reverse] reverse the result of comparisons

# For example, file "/etc/passwd" is sorted by the third numeric filed, which is "0", "1", "2", etc.
$ cat /etc/passwd | sort -t ':' -k 3 -n
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin

$ zcat access.log.9.gz | cut -d '"' -f 2 | cut -d ' ' -f 2 | sort | uniq -c | sort -r | head -3
171 /cgi-bin/soctools/http.css
131 /cgi-bin/soctools/http_get.html
32 /cgi-bin/soctools/http.py?domain=ss&ip=122.122.12.111&sid=1&google=1
The “sort” with “du”command
1
2
3
4
5
6
7
8
9
# [option -h] 
# --human-numeric-sort

$ sudo du -h /var | sort -h | tail -5
63M /var/cache/yum/x86_64/7
64M /var/cache
98M /var/lib/rpm
106M /var/lib
171M /var
The “sort” with ‘uniq’ command
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# [option -u] 
# -- to be unique

$ netstat -nutl | grep ':' | awk '{print $4}' | awk -F ':' '{print $NF}' | sort -n
22
22
25
25
68
323
323
28010
38207
$ netstat -nutl | grep ':' | awk '{print $4}' | awk -F ':' '{print $NF}' | sort -n | uniq -c
2 22
2 25
1 68
2 323
1 28010
1 38207

$ sudo cat /var/log/messages | awk '{print $5}' | sort | uniq -c | sort -n
1 polkitd[609]:
1 polkitd[612]:
1 sm-notify[905]:
1 systemd-journald[93]:
1 systemd-journald[95]:
1 systemd-journald[96]:
3 dhclient[1149]:
3 dhclient[696]:
... ...
58 NetworkManager[610]:
58 NetworkManager[615]:
105 systemd[1]:
791 systemd:
1125 kernel:
The “wc” command
1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ wc /etc/passwd
53 84 2634 /etc/passwd
$ wc -l /etc/passwd
53 /etc/passwd
$ wc -w /etc/passwd
84 /etc/passwd
$ wc -c /etc/passwd
2634 /etc/passwd

$ grep test0 /etc/passwd
test01:x:1027:1028::/home/test01:/bin/bash
test02:x:1028:1029::/home/test02:/bin/bash
$ grep test0 /etc/passwd | wc -l
2
The “tee” command

Sometimes we can use ’tee’ command to add a hostname in /etc/host when privilege is a concern

1
2
3
4
5
6
7
8
$ echo '10.9.8.11 server01' | sudo tee -a /etc/hosts
10.9.8.11 server01
$ cat /etc/hosts
127.0.0.1 admin01 admin01
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.9.8.11 server01
`

Because the comand echois not allowed due to priviledge issue

1
2
$ sudo echo test >> /etc/hsots
-bash: /etc/hsots: Permission denied

VIM Cheatsheet

Cursor movement

1
2
3
4
5
6
7
8
9
10
11
12
:set nu	        # Display Line Numbers
gg (or 1G) # Move the cursor to the first line of the document
G (Shift + g) # Move the cursor to the last line of the document
58G # Move the cursor to Line 58
40-> # Move the cursor right 40 characters
^ (Shift + 6) # jump to the first non-blank character of the line
$ (Shift + 4) # Quickly move the cursor the the last bit of one line
Control + g # Show the total lines and the current position using percentage
Ctrl + b # move back one full screen
Ctrl + f # move forward one full screen
Ctrl + d # move forward 1/2 a screen
Ctrl + u # move back 1/2 a screen

Editing

1
2
3
u               # undo
Ctrl + r # redo
. # repeat

Copy and paste

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
yy              # yank (copy) a line
Nyy -> p # Copy multiple lines and paste all of them
yw # yank (copy) the characters of the word from the cursor position to the next start
y$ # yank (copy) to end of line
p # put (paste) the clipboard after cursor
P # put (paste) before cursor
dd # delete (cut) a line
2dd # delete (cut) 2 lines
dw # delete (cut) the characters of the word from the cursor position to next start
D # delete (cut) to the end of the line
d$ # delete (cut) to the end of the line
d^ # delete (cut) to the first non-blank character of the line
d0 # delete (cut) to the begining of the line
x # delete (cut) character
15x # delete (cut) 15 characters from the current cursor position

Search and replace

1
2
3
4
5
6
:%s/old/new/g   # Replace all old with new throughout file
:%s/old/new/gc # Replace all old with new throughout file with confirmations
:50,100s/old/new/gc
# From Line 50 to Line 100, replace all old with new throughout file
/pattern # Search for pattern
n # Repeat search in same direction

Comment multple lines In/Out

注释多行文本

1
2
3
4
5
6
7
1. 使用vim打开要操作的文件
2. Ctrl+V,进入VISUAL BLOCK模式
3. 使用上下的方向键移动光标,选择你需要注释的行
3. shift+i
4. 输入注释符号 #
5. esc,退出visual block模式,
稍微等待一下(其他行的注释可能会有一些延迟),是不是已经将多行注释掉了啊?

取消注释多行文本

1
2
3
4
1. 使用vim打开要操作的文件
2. Ctrl+V,进入VISUAL BLOCK模式
3. 使用上下的方向键移动光标,选择你需要取消注释的行
3. 输入 x,删除所有的 #,同时退出visual block模式

Save as a new File

1
:w <new filename>    # Save the current file as a new File