LXC and tvheadend - Continuity counter error

[ojaksch]

I have a tvheadend LXC on my PVE7 which is (was) working excellent for years. Since a few months I'm expecting "Continuity counter errors" within tvheadend's log after watching or recording live TV for ~10 minutes, which are visible/hearable as hickups, garbage and artefacts in the video and audio stream.
I've read in some tvheadend and VDR forums that in most cases this comes from bad coax or LAN cables, a damaged/dying receiver or LNC. While hunting down this errors I've extracted and "expanded" the LXC to a physical NUC-like machine at the first step, but there tvheadend is working for hours without any flaws.

This brings me to the finding that the LXC might have problems with "network buffer underruns" or something like that. But where to turn which screws?

EDIT: This setup is using an external SAT-IP receiver, so no USB etc involved.

 

[kristian79]

Hi,

did you solve the probem? i have exactly the same problem. Since i am upgraded to the latest Proxmox Version (7.0-11) i have "Continuity counter errors".

Best regards

 

[ojaksch]

No, I didn't. The "physicalized" setup ran for ~2 weeks without any flaws, so I re-virtualized the whole thing to a QM and it's running there greatful since then.
Now I'm very sure that there are quirks or bugs regarding PVE7/LXC as the used software, setup and settings of the tvheadend system are the original ones (except that is now has it's own kernel etc.) .

 

[kristian79]

thank you for the answer. I bought a cheap usb3 Gigabit LAN Adapter and assign them physically to the lxc Container

/etc/pve/nodes/pve/lxc/CTID.conf

lxc.net.X.type: phys
lxc.net.X.link: enx00XXXXX
lxc.net.X.ipv4.address: 192.168.yyy.zzz/24
lxc.net.X.flags: up

since one hour its working quite well.

 

[ojaksch]

Any news on that? Is it still working? What brand is your internal network adapter from?
Meanwhile I bridged one port of my quad NIC (Intel igb) exclusively to the tvh lxc, but without luck - same "Continuity counter errors" after 8 minutes. I then "plugged" this eth port directly to the lxc by similar commands of your example (thanks your this hint!), eth is working fine, but, alas, the same "continuity errors" again after 8 minutes.
I've ordered an USB2<->eth adapter now and report back the next days.

 

[kristian79]

Hi,

bad news... after an hour (!) the "Continuity counter errors" came back. My two internal NIC are both from Intel the first one is an I219-LM (E1000E Driver) and the other one is a I211 (IGB Driver), in my USB Adapter work an Realtek RTL8188EE (RTL8188EE driver).

I configured a LACP bond between the both Intel which work with V. 6. very well. Maybe i try to change to openvswitch...

 

[ojaksch]

Looks like that it's somewhat hopeless for now, as long as nobody from the PVE team assist us...
But the re-virtualized QM of my LXC is working great. No issues at all.

 

[herrJones]

Looks like that it's somewhat hopeless for now, as long as nobody from the PVE team assist us...
But the re-virtualized QM of my LXC is working great. No issues at all.

I updated my Debian 10 LXC to Debian 11
Seems to resolve the problem

 

[ojaksch]

I updated my Debian 10 LXC to Debian 11
Seems to resolve the problem

Thanks for the headup. Mine is still at Stretch. Cloned and updated this LXC to buster->bullseye in a few minutes, upgraded tvheadend to 4.3 but issues are still the same after 8 minutes

Nov 18 06:13:20 tvheadend tvheadend[192]: TS: DVB-S Network/11582.25H/phoenix HD: H264 @ #5261 Continuity counter error (total 1)
Nov 18 06:13:23 tvheadend tvheadend[192]: TS: DVB-S Network/11582.25H/phoenix HD: MPEG2AUDIO @ #5263 Continuity counter error (total 1)
Nov 18 06:13:34 tvheadend tvheadend[192]: TS: DVB-S Network/11582.25H/phoenix HD: H264 @ #5261 Continuity counter error (total 4)
Nov 18 06:14:03 tvheadend tvheadend[192]: TS: DVB-S Network/11582.25H/phoenix HD: H264 @ #5261 Continuity counter error (total 5)
Nov 18 06:14:10 tvheadend tvheadend[192]: TS: DVB-S Network/11582.25H/phoenix HD: TELETEXT @ #5264 Continuity counter error (total 1)
Nov 18 06:14:11 tvheadend tvheadend[192]: TS: DVB-S Network/11582.25H/phoenix HD: MPEG2AUDIO @ #5263 Continuity counter error (total 2)
Nov 18 06:14:14 tvheadend tvheadend[192]: TS: DVB-S Network/11582.25H/phoenix HD: H264 @ #5261 Continuity counter error (total 21)

Same setup and configuration is still working w/o any flaws in the QM.

 

[herrJones]

Thanks for the headup. Mine is still at Stretch. Cloned and updated this LXC to buster->bullseye in a few minutes, upgraded tvheadend to 4.3 but issues are still the same after 8 minutes


Same setup and configuration is still working w/o any flaws in the QM.

indeed

I was optimistic after seeing BBC One HD for 2h without a single continuity error.
Same for a simultaneous recording.

But then a Channel4 recording was almost unwatchable due to the continuity errors...

Now trying to find some debug logging for LXC containers. Must be some performance/queueing error

 

[DerMerowinger]

As I can see your problems (and mine), I should not have updated to pve7 as there is this absolutely disturbing problem.
It also worked for years here and now after upgrading to pve7 it does not. Anybody tried tvheadend in a vm? Maybe qemu does not have this problem?!

 

[kristian79]

Hi,

@DerMerowinger

Maybe qemu does not have this problem?!

In Qemu run TVHeadheand without a problem.

 

[mm553]

Take a look at "free -m" and the "free" column, I made the observation that the memory of the LTX container can't free up the memory and when it arrives at 0, you'll get Continuity counter error.
This is really disappointing. I tried everything but can't get it working. One (really crappy) workaround is to drop the memory cache on the host system with a cronjob for example: " echo 3 > /proc/sys/vm/drop_caches".
Anyways this is a general problem of the LTX containers in Proxmox, it's not just an issue with Tvheadend!

As I'm new to Proxmox I can't reproduce this on an older version. Did the problem appear with version 7.x or did it exist in version 6.4 already?

Thanks!

 

[DerMerowinger]

I observed it only after updating to version 7

 

[mm553]

Ok, i've made a test with Proxmox 6.4.13 now and the issue has gone away.
I think I found the main problem for the memory issue.
In 6.4.13 you can get the consumed resources, like cpu and memory (you can view with ltc-top). In 7.x all the stats are missing, ltc-top shows zero in every column.

 

[deffcon]

Dear members,

Any news on this ? I’m experiencing the same issues.

Dave

 

[mm553]

The problem still exists with Proxmox 7.2.7... very frustrating...

 

[mathx]

Check your tmpfs usage, its charged to ram for the container. My culprit was systemd-journald, had to put in a crontab to vacuum the logs out.

 

[mm553]

Check your tmpfs usage, its charged to ram for the container. My culprit was systemd-journald, had to put in a crontab to vacuum the logs out.

Thanks for the advice!
Hmm, I don‘t think logs are the problem here…
The buffer/cache grows and grows whole I‘m recording to the harddisk. When free = 0 the recording stops

 

[mathx]

interesting. Dont think I've had that problem, so far every time it's been tmpfs with journald. tmpfs ram shows up as 'buffers' for some reason (since its disk related i guess? though flushing buffers of course wont solve it! thus I feel it's misreported in the wrong column and sent me goose chasing down the wrong way!)

your issue is different. you can dump caches as explained in another thread, but that costs the whole server by having no read cache for a while before it's repopulated again - suddenly reads on your drives will jump up til it is.

https://www.tecmint.com/clear-ram-memory-cache-buffer-and-swap-space-on-linux/