• lime!@feddit.nu
    link
    fedilink
    English
    arrow-up
    20
    ·
    16 hours ago

    my favorite awk snippet is !x[$0]++ which is like uniq but doesn’t care about order. basically, it’s equivalent to print_this_line = line_cache[$current_line] == 0; line_cache[$current_line] += 1; if $print_this_line then print $current_line end.

    really useful for those long spammy logs.

    • grrgyle@slrpnk.net
      link
      fedilink
      arrow-up
      3
      ·
      2 hours ago

      Oh that’s very interesting. I usually do sort --unique or sort [...] | uniq if I need specific sorting logic (like by size on disk, etc).

      • tal@lemmy.today
        link
        fedilink
        English
        arrow-up
        3
        ·
        edit-2
        1 hour ago

        Looking at the above awk snippet, it’ll retain order, though. So, sort will normally change the order. The awk snippet won’t, just skip occurrences of a given line after the first. Depending upon the use case, that order retention could be pretty desireable.