I just want data those have unique value, say we write log file for mobile users

<mobile_number1>|20141006 06:15:26||AKQY6LYACZAA4O|12|3|BIHAR| <mobile_number2>|20141006 06:16:05||AKQY6MAYAEQALE|12|22|UP EAST| <mobile_number3>|20141006 06:16:39||AKQY6MQICY4BEQ|12|2|ASSAM| <mobile_number4>|20141006 06:16:49||AKQY6LUIAE4ACI|12|1|ANDHRA PRADESH| <mobile_number1>|20141006 06:17:53||AKQY6NFAAEYAJS|12|23|UP WEST| <mobile_number6>|20141006 06:18:09||AKQY6M7ACY4ANG|12|18|ORISSA| <mobile_number7>|20141006 06:18:54||AKQY6MWQCZAAME|12|20|RAJASTHAN| <mobile_number1>|20141006 06:19:50||AKQY6N2ACZMA2K|12|1|ANDHRA PRADESH|

Now we need to fetch unique mobile number like this.

<mobile_number1> <mobile_number2> <mobile_number3> <mobile_number4> <mobile_number6> <mobile_number7>
4

3 Answers

Try this:

 sort -t '|' -k 1,1 -u yourFile | awk -F "|" '{print $1}'

where:

  • -t '|' use | as separator
  • -k 1,1 use first column as key
  • -u get unique line using key

awk print first column and the result is:

<mobile_number1> <mobile_number2> <mobile_number3> <mobile_number4> <mobile_number6> <mobile_number7>

This script works if each line in yourFile has mobile_number as first column separated from the rest by |.

6

To throw away all columns but the first:

cut -d '|' -f 1

To omit recurring lines:

sort -u

Together (with input from a file):

cut -d '|' -f 1 <FILE> | sort -u

This uses the most simple shell utilities to perform the task. No command interpreters like awk necessary.

 awk '!x[$1]++ {print $1}' FS="|" /path/to/file

example:

[aneesh@mylap /]$ awk '!x[$1]++ {print $1}' FS="|" /tmp/test.txt <mobile_number1> <mobile_number2> <mobile_number3> <mobile_number4> <mobile_number6> <mobile_number7>