Ask HN: Google Colab alternatives for large lang models?

by lovelearning on 7/20/2023, 2:49:11 PM with 5 comments
I just want to run some LLM experiments on GPUs. Since these are 10-20GB models, I need something like an A100 40GB GPU at a minimum.

But I'm finding that most of the paid Google Colab options don't work. Instead of working on the LLM experiments, I'm wasting considerable time on infrastructure.

First, I purchased pay-as-you-go credits[1]. But about 80% of the time, I get a "Selected GPU unavailable" for A100.

Next, I tried the custom GCE VM option. But the Colab deployment marketplace app[2] keeps hitting me with quota denials for A2-CPUs and A100-GPUs.

Requesting quota increases in GCP is equally frustrating. They auto-approved 12 A2-CPUs for a zone but denied one A100-GPU for that zone!

There are some 50 or so zones and apparently I have to individually request quota increases each of them but they get denied.

-----

QUESTION 1:

How are you people testing LLMs like GPT-NeoX, LLaMA 66B, etc?

------

QUESTION 2:

Is there a simple, non-time-wasting notebook alternative for Colab where I can get A100 or better GPUs easily?

Is there some arrangement of using Colab with local runtime which somehow proxies to a better GPU service provider?

Kaggle NB is also frustrating in other ways. Their GPUs don't seem to work with these models. Also difficult to connect to GDrive, difficult to transfer files in and out, etc.

-----

QUESTION 3:

If there's no alternative to Colab, anybody here has some practical tips to avoid these quota denials?

------

[1]: https://colab.research.google.com/signup/pricing

[2]: https://research.google.com/colaboratory/marketplace.html

  • by verdverm on 7/20/2023, 3:26:00 PM

    Getting access to A100s seems to be a general problem for everyone, not limited to you or GCP.

    The denial is probably because they don't have any to rent you. Many clouds facing this issue.

    When submitting a GCP quota increase, be wordy and explain the details of your need for the new quota. Also, only ask for what you actually need and make sure they have them for the zone. Not all GPUs are in all zones.

    Did you ask for more than one? Are you spending actual money on GCP or just using credits?

    Also, just a heads up, even with quota, you can still get a no GPU available error if they are all rented at the time