running VMs from templates saturates NAS

[sdettmer]

Hi,

I created a few VMs using a thin clone template from NAS. This makes it simple to clone VMs on any of the nodes. However, I think the VMs share the I/O bandwidth, so for 80 VMs every just gets ~1.2%, On nodes, I see dmesg messages like:

[99033.583208] rpc_check_timeout: 108 callbacks suppressed
[99033.615466] nfs: server denas-c1 not responding, still trying
[99033.645409] nfs: server denas-c1 not responding, still trying
[99033.645444] nfs: server denas-c1 not responding, still trying
[99033.668942] nfs: server denas-c1 not responding, still trying
[99033.668981] nfs: server denas-c1 not responding, still trying
[99033.668985] nfs: server denas-c1 not responding, still trying
[99033.668986] nfs: server denas-c1 not responding, still trying
[99033.669011] call_decode: 37 callbacks suppressed
[99033.669012] nfs: server denas-c1 OK
[99033.669016] nfs: server denas-c1 OK
[99033.669016] nfs: server denas-c1 OK
[99033.669017] nfs: server denas-c1 OK

For this test, I have approximately 10 nodes with approx 8 VMs each, and a single shared NAS (spinning disks with Gigabit Ethernet).

Using a shared storage for the template eases cloning.

I think thin clone means that written bytes are stored in VM-specific files, on the shared storage.

Can I somehow have the VM-specific files locally (i.e. read shared, write local)? I think most I/O load I have from locally created data (linking C++ compiled objects and stuff) and having it on local SSD storage should help a lot.

Otherwise I think I should full clone to local SSD storage.

Or is there a better approach?

Any suggestions are welcome!