Linux删除重复行

清华大佬耗费三个月吐血整理的几百G的资源,免费分享!....>>>

# !/bin/sh 

file='test.txt' 

sort -n $file | uniq 

sort -n $file | awk '{if($0!=line)print; line=$0}' 

sort -n $file | sed '$!N; /^\(.*\)\n\1$/!P; D' 

测试文件:

yanggang@barry$ cat test.txt

aaa

bbbbb

ccccc

123

aaaaa

123

bbb

aaa

执行结果:

yanggang@barry$ ./diffRow.sh

aaa

aaaaa

bbb

bbbbb

ccccc

123