• 0 Posts
  • 49 Comments
Joined 1 year ago
cake
Cake day: October 18th, 2024

help-circle


  • Hehe, true! I left the field about 4 years ago when it became obvious that “more GPUs!” was better than any architectural design changes…

    Most of the image generation made by the products you mention are based on a mix of LLMs (for processing of user inputs) and some other modality for other media types. Last time I checked, ChatGPT was capable of handling images only because it offloaded the image processing to a branch of the architecture that was not a transformer, or at least not a classical transformer. They did have to grift CNN parts to the LLM to make progress.

    Maybe in the last 4 years they reorganised it to completely remove CNN blocks, but I think people call these models “LLMs” only as a shorthand for the core of the architecture.

    Again, you said that a new benchmark is set every few months, but considering they’re just consuming more power and water, it’s quite boring and I’d argue it’s not really progress in the academical/theoretical sense. That attitude is exactly why I don’t work with NN anymore.



  • The main breakthrough of LLM happened when they figured out how to tokenize words… The subsequent transformer architecture was already being tested on various data types and struggled compared to similarly advanced CNN.

    When they figured out word encoding, it created a buzz because transformers could work well with words. They never quite worked as well on images. For that, stable diffusion (a variation on CNN) has always been better.

    It’s only because of the buzz on LLMs that they tried applying them to other data types, mostly because that’s how they could get funding. By throwing in disproportionate amount of resources, it works… But it would have been so much more efficient to use different architectures.













  • I once was reported by the high school programming class for “plagiarism” because I used visual studio’s auto-generated template to start my homework.

    The teacher reported it to my parents, he wanted to make me fail. I was also reported for creating a “hidden” chat app that I shared with my friends. (It didn’t show in the taskbar.)

    Next Christmas, they bought me a visual studio license at home!