3. Or should I throw everything I have at it? (Assuming that my tasks can utilize all 16T fully)
You wont, because you can't use 16T within the VM on a 16Thread system.
You will always compete with the host. Whatever you think the system is doing - that is not how a hypervisor operates. Please - I don't want to mess with you or play games. I just try to explain you have a fundamental glitch in your thought.
2. But when there is something to be done, do I only spend 20% resources and let the remaining 80% sit idle?
Kind off, because Virtualization means: Exchange flexibility with performance.
You can't have both. There is a trade-off.
Even though you think the system "does nothing" - it is not true. There is always something every VM does. It has an OS, there are tasks running.
And those tasks will interfer with your "big VM".
This is why cloud hyperscalers use very complex mechanisms to balance and optimize their environments. If you think a single Hypervisor can do this - you are wrong. Autoscaling is difficult and this is why your expectation is wishfull thinking.
5. Should I use PVE or instead double my cost and get 2 machines to install Linux and Windows so that they can get 100% resources when required?
if you want to have 100% ressources for one job, yes. But you also could use the same hardware and dual-boot. Depending on what you want to do.
Remember - flexibility vs. performance vs. cost. This all is a triangle - you can't have all.
6. A feasible (we can aim for optimal later) approach is to give both my VMs access to about roughly 60% of the resources and let them decide on what to do with them
usually less is more in virtualization environments. That also depends on the application. Means, if a VM always can get ressources when it needs, without waiting for it (e.g. your big VM being in a race condition by default) a small VM even might get the job done faster. There is a rule of thumb to "size to the minimum" rather than "maximum". Because every vCPU creates overhead.
7. 50% RAM is easy, but what is 60% CPU? The simple answer is 60% of 16T (and while 6% of 160T is also ~60% CPU, what's the point of such an example?)
RAM is relatively easy, unless you use oversubscription. But there as well. Memory of 16GB in guest means more on the host. You need memory to amanage the guest memory. Also don't forget the host OS (PVE) and services running there, ZFS for instance has a significant memory requirement.
1. My machine is idle about 70% of the time, 120 of 168 hours
This might look like it - but also can be caused through wait-states. So an overwhelmed system can also look like there is nothing going on it.
The performance analyses and tuning is a very difficult discipline. I am not saying that this is the case in your situation, but I have experienced those situations as well...
8. Of course I understand the final performance is dependent on the algo being used, but the question is, have the VMs been given access to ~60% f the computing power in the first place?
Forget about the host is my advice. Check what the VM needs to perform a proper operation. If you have cycles on the host to spare - that is ok. He is using them wisely and maybe prioritize another VM.
9. I already found my answer several posts ago, 16T and I just have to decide how many concurrent threads (on the hardware) to allow for each VM.
Not sure which question you exactly mean. You have 8 Cores. Real cores.
You should not use more than 6 vCPUs because everything else is a lie. It is no real core. 2 Cores (in mind) for the OS (PVE) help to prevent congestion.
And then use as minimal core count per VM as possible.
Linux might be even 1 vCPU
Windows typically needs at least 2 vCPUs.
Start with that and see how your VMs behave.
