Sfdwhf

I have a text file as follows;

ALIC00AUS_R_20183350000.gz -4052052.7 4212835.9 -2545104.6

ALIC00AUS_R_20183350000.gz -4052052.7 4212836.0 -2545104.6

ALIC00AUS_R_20183350000.gz -4052052.7 4212836.0 -2545104.6

ALIC00AUS_R_20183350000.gz -4052052.7 4212835.9 -2545104.6

ALIC00AUS_R_20183350000.gz -4052052.5 4212836.0 -2545104.6

ALIC00AUS_R_20183350000.gz -4052052.6 4212835.9 -2545104.6

CPVG00CPV_R_20183460000.gz 5626883.4 -2380932.3 1824483.9

CPVG00CPV_R_20183460000.gz 5626883.4 -2380932.3 1824483.9

CPVG00CPV_R_20183460000.gz 5626883.3 -2380932.2 1824483.1

In this file, ALIC00AUS_R_20183350000.gz and CPVG00CPV_R_20183460000.gz repeat six and three times, respectively. I need to remove the last repeated lines of each string in column 1 and the output should be as follows;

ALIC00AUS_R_20183350000.gz -4052052.7 4212835.9 -2545104.6

ALIC00AUS_R_20183350000.gz -4052052.7 4212836.0 -2545104.6

ALIC00AUS_R_20183350000.gz -4052052.7 4212836.0 -2545104.6

ALIC00AUS_R_20183350000.gz -4052052.7 4212835.9 -2545104.6

ALIC00AUS_R_20183350000.gz -4052052.5 4212836.0 -2545104.6

CPVG00CPV_R_20183460000.gz 5626883.4 -2380932.3 1824483.9

CPVG00CPV_R_20183460000.gz 5626883.4 -2380932.3 1824483.9

Awk is a go to program for doing this kind of replacemen.

To remove lines that have a duplicate in the first column this should do it.

awk '!seen[$1]++' filename > outputfile

If you need to remove full duplication of lines use this instead.

awk '!seen[$0]++' filename > outputfile

As seen in this answer: https://unix.stackexchange.com/questions/171091/remove-lines-based-on-duplicates-within-one-column-without-sort

Here is a brief explanation. awk is used for pattern scanning and text processing. First, it checks if the value in column 1 ($1) is in the map seen. If it isn't it prints the line to the output file, or screen if you don't redirect (> outputfile). The next part is that it adds the viewed column info to seen (++) so it can search with the next pattern.

If you are sure that each $1 (first column) is duplicated at least once, then you can:

1. reverse the order of lines


2. only select those lines whose $1 has been seen before


3. reverse the result

Ex.

$ tac file | awk 'seen[$1]++' | tac

ALIC00AUS_R_20183350000.gz -4052052.7 4212835.9 -2545104.6

ALIC00AUS_R_20183350000.gz -4052052.7 4212836.0 -2545104.6

ALIC00AUS_R_20183350000.gz -4052052.7 4212836.0 -2545104.6

ALIC00AUS_R_20183350000.gz -4052052.7 4212835.9 -2545104.6

ALIC00AUS_R_20183350000.gz -4052052.5 4212836.0 -2545104.6

CPVG00CPV_R_20183460000.gz 5626883.4 -2380932.3 1824483.9

CPVG00CPV_R_20183460000.gz 5626883.4 -2380932.3 1824483.9

If you can meet the guarantee in the answer provided by @steeldriver that is the better solution, if not use this script.

##!/usr/bin/awk -f

{

  if (!seen[$1]++) {

    line = $0

    prev = $1

  }

  else {

    if (prev == $1) print line

    line = $0

  }

}

END { if (prev != $1) print line }