• ben@lemmy.zip
    link
    fedilink
    English
    arrow-up
    43
    ·
    4 months ago

    I skimmed the article, but it seems to be assuming that Google’s LLM is using the same architecture as everyone else. I’m pretty sure Google uses their TPU chips instead of a regular GPU like everyone else. Those are generally pretty energy efficient.

    That and they don’t seem to be considering how much data is just being cached for questions that are the same. And a lot of Google searches are going to be identical just because of the search suggestions funneling people into the same form of a question.

    • kromem@lemmy.world
      link
      fedilink
      English
      arrow-up
      17
      arrow-down
      1
      ·
      4 months ago

      Exactly. The difference between a cached response and a live one even for non-AI queries is an OOM difference.

      At this point, a lot of people just care about the ‘feel’ of anti-AI articles even if the substance is BS though.

      And then people just feed whatever gets clicks and shares.

    • AlecSadler@sh.itjust.works
      link
      fedilink
      English
      arrow-up
      12
      ·
      4 months ago

      I hadn’t really heard of the TPU chips until a couple weeks ago when my boss told me about how he uses USB versions for at-home ML processing of his closed network camera feeds. At first I thought he was using NVIDIA GPUs in some sort of desktop unit and just burning energy…but I looked the USB things up and they’re wildly efficient and he says they work just fine for his applications. I was impressed.

      • ben@lemmy.zip
        link
        fedilink
        English
        arrow-up
        8
        ·
        4 months ago

        Yeah they’re pretty impressive for some at home stuff and they’re not even that costly.

      • dan@upvote.au
        link
        fedilink
        English
        arrow-up
        8
        ·
        4 months ago

        The Coral is fantastic for use cases that don’t need large models. Object recognition for security cameras (using Blue Iris or Frigate) is a common use case, but you can also do things like object tracking (track where individual objects move in a video), pose estimation, keyphrase detection, sound classification, and more.

        It runs Tensorflow Lite, so you can also build your own models.

        Pretty good for a $25 device!

    • dan@upvote.au
      link
      fedilink
      English
      arrow-up
      6
      ·
      edit-2
      4 months ago

      I’m pretty sure Google uses their TPU chips

      The Coral ones? They don’t have nearly enough RAM to handle LLMs - they only have 8MB RAM and only support small Tensorflow Lite models.

      Google might have some custom-made non-public chips though - a lot of the big tech companies are working on that.

      instead of a regular GPU

      I wouldn’t call them regular GPUs… AI use cases often use products like the Nvidia H100, which are specifically designed for AI. They don’t have any video output ports.