Memory sharing/model switching solution for image and text generators ?

[shodan]

Hi,

I run in two CT containers the following

oobabooga/text-generation-webui
AUTOMATIC1111/stable-diffusion-webui

On their own they each work great, however, only one can be using the VRAM at one time.
As I only have 12gb in my 3060 LHR

I am wondering if you have any script to automate, or other solution, to make the user experience better.
Without having to go into both interfaces and manually unload the model in one then load it in the other.
Something to automate that ?

 

[LnxBil]

In this setup, just start only the one container that needs to have access to your GPU and shutdown the other. This is a simple resource problem in your application where PVE can nothing do about it.

 

[shodan]

I am asking if someone has a better solution than that.

Since that is a problem for everyone running both a text generator and image generator.

You don't want to shut them off and disconnect the users, instead they have to coordinate across the containers, with something in the hypervisor so that they use the vram one at a time.

That could take the form of a web page with a toggle switch between text or image.
Surely someone here has had and already dealt with this issue ?
In a way that does not mean explaining to the users how to operate the proxmox webui.

 

[LnxBil]

with something in the hypervisor so that they use the vram one at a time.

That's not a hypervisor problem, you don't virtualize the GPU, you just passthrough it. Therefore the hypervisor is not involved and you need to do it manually.

Surely someone here has had and already dealt with this issue ?

Of course, by stopping the running kernel on the gpu and using another one. Like it would be without any hypervisor invovled.