NFS Unknown symbol errors

[ianmbetts]

I have a four node cluster, 3 Ceph nodes, plus a node hosting NFS for backups.

One of the three Ceph nodes has broken such that it cant access the NFS share.
The other two are fine.
All the nodes are identical hardware and software is pve install on top of a debian buster minimal net install.

Investigation reveals a shed load of unresolved symbols reported in kern.log on the fauity node.
Further if I boot that node with the orginal debian 4.19 kernel NFS works fine.

Below a snip from kern.log

May 8 12:31:06 mfscn02 kernel: [ 1634.543990] nfs: Unknown symbol rpc_create (err -2)
May 8 12:31:06 mfscn02 kernel: [ 1634.544018] nfs: Unknown symbol __fscache_uncache_all_inode_pages (err -2)
May 8 12:31:06 mfscn02 kernel: [ 1634.544034] nfs: Unknown symbol __fscache_read_or_alloc_page (err -2)
May 8 12:31:06 mfscn02 kernel: [ 1634.544051] nfs: Unknown symbol rpc_call_sync (err -2)
May 8 12:31:06 mfscn02 kernel: [ 1634.544073] nfs: Unknown symbol rpc_put_task_async (err -2)
May 8 12:31:06 mfscn02 kernel: [ 1634.544091] nfs: Unknown symbol xdr_init_decode_pages (err -2)
May 8 12:31:06 mfscn02 kernel: [ 1634.544121] nfs: Unknown symbol rpc_exit (err -2)
May 8 12:31:06 mfscn02 kernel: [ 1634.544148] nfs: Unknown symbol rpc_killall_tasks (err -2)
May 8 12:31:06 mfscn02 kernel: [ 1634.544167] nfs: Unknown symbol xdr_reserve_space (err -2)
May 8 12:31:06 mfscn02 kernel: [ 1634.544190] nfs: Unknown symbol __fscache_uncache_page (err -2)
May 8 12:31:06 mfscn02 kernel: [ 1634.544231] nfs: Unknown symbol __fscache_register_netfs (err -2)
May 8 12:31:06 mfscn02 kernel: [ 1634.544272] nfs: Unknown symbol __fscache_write_page (err -2)
May 8 12:31:06 mfscn02 kernel: [ 1634.544295] nfs: Unknown symbol rpc_init_wait_queue (err -2)
May 8 12:31:06 mfscn02 kernel: [ 1634.544317] nfs: Unknown symbol put_rpccred (err -2)
May 8 12:31:06 mfscn02 kernel: [ 1634.544333] nfs: Unknown symbol rpc_clnt_swap_deactivate (err -2)
May 8 12:31:06 mfscn02 kernel: [ 1634.544355] nfs: Unknown symbol __fscache_invalidate (err -2)
May 8 12:31:06 mfscn02 kernel: [ 1634.544383] nfs: Unknown symbol rpc_clnt_xprt_switch_has_addr (err -2)
May 8 12:31:06 mfscn02 kernel: [ 1634.544405] nfs: Unknown symbol nlmclnt_init (err -2)
May 8 12:31:06 mfscn02 kernel: [ 1634.544429] nfs: Unknown symbol nlmclnt_done (err -2)
May 8 12:31:06 mfscn02 kernel: [ 1634.544461] nfs: Unknown symbol __fscache_wait_on_page_write (err -2)
May 8 12:31:06 mfscn02 kernel: [ 1634.544481] nfs: Unknown symbol nfs_debug (err -2)
May 8 12:31:06 mfscn02 kernel: [ 1634.544501] nfs: Unknown symbol __fscache_disable_cookie (err -2)
May 8 12:31:06 mfscn02 kernel: [ 1634.544525] nfs: Unkno

Any ideas ?
Thanks

 

[ianmbetts]

OK solved.
For the benefit of anyone else, I had upgraded all nodes but the upgrade for this particular mode must have failed.
The following dirs were missing

/lib/modules/5.3.18-3-pve/kernel/kernel
/lib/modules/5.3.18-3-pve/kernel/lib
/lib/modules/5.3.18-3-pve/kernel/mm
/lib/modules/5.3.18-3-pve/kernel/net

I copied them across from another node and everything seems fine now.