Start with list.txt like:
server7858 server7858 server7858 server7858 server7861 server7860 server8310 server8310 server7863 server8311
server7859 server7859 server7859 server7859 server8781 server8676 server8677 server8677 server8679 server8782
Which has duplicates and long lines and crap. Run this:
rm list2.txt
rm list3.txt
for word in `cat list.txt`; do echo $word ; done |sort |uniq >> list2.txt
sed -ibak -e ‘s/ //g’ list2.txt
cat list2.txt |sort|uniq > list3.txt
vi list3.txt
ta da!
if you need word counts and such, pipe it through wc before running uniq
If you need to collapse multi line (multi-line multiline) data like this:
fldcvisla8524:
packages.MQSeriesServer.installdate: 1439579830
fldcvfsla13746:
packages.MQSeriesServer.installdate: 1486575523
Into:
fldcvisla8524: packages.MQSeriesServer.installdate: 1439579830
fldcvfsla13746: packages.MQSeriesServer.installdate: 1486575523
Then try this:
paste -s -d ‘ \n’ list-mq.txt >> list-mq-out.txt
If you have to adapt to collapsing every 3 lines or 4th or 5 or whatever, add more spaces in the ‘ \n’ and try again until it loops properly
this one is useful, you can search for a pattern and then combine the following lines, needs tweaking for your use case:
cat temp.txt |sed -n ‘/+version/ {s/.*//; N; N; s/\n//g; p;}’
Remember that there is the paste command that combines two files together!
Join too.
file1.txt:
January
February
file2.txt
01 1970
07 1967
paste file1.txt file2.txt > file3.txt
gives file3:
January 01 1970
February 07 1967
Join will match up fields in two places from two separate files, it’s a sql join across text files!
more metadata, this one is hard to find:
bash
list
word
parse
parsing
text
columns
sorting
collapse
words
shell
script
split
line
lines
grep
collapse
multi
multiple
multiline
multi-line
multi line
cut
paste