Ytukyg

I have multiple revisions of a text file in separate files in the same folder.

How can I grep all files in that folder without listing any duplicate of lines with identical text?

How about

cat * | grep exampletext | sort -u

I use:

grep -h test files* | puniq

puniq is: perl -ne '$seen{$_}++ or print;'

It is similar to sort -u but it does not sort the input and it gives output while running.

If you want the file name and avoid duplicate lines in each file:

parallel --tag --lb 'grep string {} | puniq' ::: files*

If you want the file name and do not want duplicate lines from any of the files (File names must not contain TAB (t)):

parallel --tag --lb grep string {} ::: files* |

  perl -ne '/^[^t]+(.*)/ and $seen{$1}++ or print;'

Maybe something like this could be close to what you imagine (works with gnu awk):

cat file1

1

2

3

22



cat file11

1

2

3

8

9



cat file111

1

2

3

5

6



awk '{seen[$0]++;fname[$0]=FILENAME};END{for (k in seen) {if (seen[k]==1) print fname[k],":",k}}' file1*

file111 : 5

file111 : 6

file11 : 8

file11 : 9

file1 : 22

Pipe the result into sort to filter duplicates.

grep -re pattern files and dirs ... | sort -ut: -k2

The -t: and -k2 options of sort will cause it to ignore the file name when doing the sorting and merging.

Or, if you don't want the filenames, simply:

grep -hre pattern files and dirs ... | sort -u 

If what you need is to find which file(s) match some text, use:

$ grep -rl 'text to find' ./dir

If you need only the first match of each file:

$ for file in ./*; do sed -n '/text to match/{p,q}' "$file"; done

which will not print the name of the files matching, but will be fast.

Or:

$ find ../* -type f -exec sh -c '

      a=$(sed -n "/echo/{p;q}" "$1");

      [ "$a" ] && printf "%sn" "$1 : $a"

  ' findsh {} ;

If you need the filename also (separated with :).

I would use something like:

#!/bin/bash

echo "What is the files base name? e.g. testfile"

read filename

for i in *$filename*; do 

    cat $i >> tempfile

    cat tempfile | sort -u -o tempfile

done

or

cat *filename* | sort -u

The placement of the *'s is based on how you name your itterations.

• When done read tempfile.


• Rename to whatever.

Use with caution in a test directory with test files first

• Tailor to your needs.

My script turns:

testfile1         testfile2          testfile3



This is line 1    This is line 5     This is line 5   

This is line 2    This is line 6     This is line 2

This is line 3    This is line 7     This is line 7

This is line 4    This is line 8     This is line 3

Into :

This is line 1

This is line 2

This is line 3

This is line 4

This is line 5

This is line 6

This is line 7

This is line 8