my favorite awk snippet is !x[$0]++ which is like uniq but doesn’t care about order. basically, it’s equivalent to print_this_line = line_cache[$current_line] == 0; line_cache[$current_line] += 1; if$print_this_linethenprint$current_line end.
Looking at the above awk snippet, it’ll retain order, though. So, sort will normally change the order. The awk snippet won’t, just skip occurrences of a given line after the first. Depending upon the use case, that order retention could be pretty desireable.
my favorite awk snippet is
!x[$0]++which is likeuniqbut doesn’t care about order. basically, it’s equivalent toprint_this_line = line_cache[$current_line] == 0; line_cache[$current_line] += 1; if $print_this_line then print $current_line end.really useful for those long spammy logs.
same, that statement saved me so much effort
Oh that’s very interesting. I usually do
sort --uniqueorsort [...] | uniqif I need specific sorting logic (like by size on disk, etc).Looking at the above awk snippet, it’ll retain order, though. So, sort will normally change the order. The awk snippet won’t, just skip occurrences of a given line after the first. Depending upon the use case, that order retention could be pretty desireable.