Windows 2016 - KB4013418 - NetBackup 7.x 8.x

Test environment:
OS Name: Microsoft Windows Server 2016 Standard
OS Version: 10.0.14393 N/A Build 14393
NetBackup 7.x and 8.x

Error from job details :
Dec 20, 2017 10:15:52 PM - Warning bpbrm (pid=20340) from client test-windows-2016.netbackup.local: WRN - can't open object: System State:\System Files\System Files (WIN32 3758134305: Unknown error)
Dec 20, 2017 10:15:52 PM - Error bpbrm (pid=20340) from client test-windows-2016.netbackup.local: ERR - Error encountered while attempting to get additional files for System State:\

Enabled debug logs in NetBackup - 2 General, 5 Verbose
Create folders bpkar under Netbackup\logs\
Now let's read 31MB of logs :)
Error does not appear under code 8, 16 or 32 so it should be fun to find it.


Digging :
09:37:53.484 [548.3096] <2> ov_log::V_GlobalLog: INF - Informational: calling IVssBackupComponents::SetBackupSucceeded with status 'SUCCESS (0x00000000)' for Component 'TasksStore' in SHADOW::CloseComponent
09:37:53.854 [548.3096] <2> ov_log::V_GlobalLog: INF - Informational: calling IVssBackupComponents::SetBackupSucceeded with status 'SUCCESS (0x00000000)' for Component 'WriterMetadataStore' in SHADOW::CloseComponent
09:37:54.388 [548.3096] <2> ov_log::V_GlobalLog: INF - Informational: calling IVssBackupComponents::SetBackupSucceeded with status 'SUCCESS (0x00000000)' for Component 'PerformanceCounters' in SHADOW::CloseComponent
09:43:06.123 [548.3096] <2> ov_log::V_GlobalLog: INF - Informational: calling IVssBackupComponents::SetBackupSucceeded with status 'SUCCESS (0x00000000)' for Component 'Volume{8cbe8c15-0000-0000-0000-100000000000}' in SHADOW::CloseComponent
09:43:09.260 [548.3096] <2> ov_log::V_GlobalLog: INF - Informational: calling IVssBackupComponents::SetBackupSucceeded with status 'SUCCESS (0x00000000)' for Component 'BCD' in SHADOW::CloseComponent
09:43:09.559 [548.3096] <2> ov_log::V_GlobalLog: INF - Informational: calling IVssBackupComponents::SetBackupSucceeded with status 'SUCCESS (0x00000000)' for Component 'COM+ REGDB' in SHADOW::CloseComponent
09:43:11.615 [548.3096] <2> ov_log::V_GlobalLog: INF - Informational: calling IVssBackupComponents::SetBackupSucceeded with status 'SUCCESS (0x00000000)' for Component 'Registry' in SHADOW::CloseComponent
09:43:12.352 [548.3096] <2> ov_log::V_GlobalLog: INF - Informational: calling IVssBackupComponents::SetBackupSucceeded with status 'SUCCESS (0x00000000)' for Component 'WMI' in SHADOW::CloseComponent
......
Everything looks good so far.
.....
09:43:12.419 [548.3096] <4> tar_backup::backup_endarg_state: INF - finish error count: 1
......
So it finished with status 1.

Now up again to find the related error:
.....
09:41:50.768 [548.3096] <2> ov_log::V_GlobalLog: INF - Status 0x00000003 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy29\windows\systemroot\system32\drivers\vsock.sys in brUtil::GenerateFileList
09:41:50.768 [548.3096] <2> ov_log::V_GlobalLog: INF - Status 0x00000003 for file c:\windows\\systemroot\system32\drivers\vsock.sys in brUtil::GenerateFileList
09:42:06.521 [548.3096] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy29\windows\servicing\version\10.0.14393.0\amd64_installed in brUtil::GenerateFileList
09:42:06.521 [548.3096] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\servicing\version\10.0.14393.0\amd64_installed in brUtil::GenerateFileList
09:42:52.975 [548.3096] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy29\windows\servicing\version\10.0.14393.0\x86_installed in brUtil::GenerateFileList
09:42:52.975 [548.3096] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\servicing\version\10.0.14393.0\x86_installed in brUtil::GenerateFileList
09:43:04.172 [548.3096] <2> ov_log::V_GlobalLog: INF - Informational: Component:System Files no files are present to protect.
09:43:04.172 [548.3096] <2> ov_log::V_GlobalLog: INF - brUtil::SetBackupMethod(): Backup method set to BRU_BACKUP_METHOD_NORMAL(0) [Time=0]
09:43:04.172 [548.3096] <2> ov_log::V_GlobalLog: INF - Component Set System Files has 3 files missing on the snapshot
.......

3 files missing from the snapshot .... iuhuu ...

Microsoft documentation shows only 2 of them, 3th one is related to VMware Tools.

File List: Path = c:\windows\\systemroot\system32\drivers, Filespec = vsock.sys (Image path issue)

Microsoft link : https://support.microsoft.com/en-us/help/4052556/error-the-system-writer-is-not-found-in-the-backup-windows-server-2016

On a computer that is running Windows Server 2016 (Version 1607, build 14393.693), you install a servicing stack update. For example, the March 14, 2017 Servicing Stack Update (KB 4013418).

The folder C:\Windows\servicing\Version\10.0.14393.1051 is automatically created, and the amd64_installed and x86_installed files are saved to this folder.

For example, the following files are removed:

C:\Windows\servicing\Version\10.0.14393.0\amd64_installed
C:\Windows\servicing\Version\10.0.14393.0\x86_installed

Resolution: To fix the issue create new placeholder files with the same name for the files that are reported as not found.

Contig (https://docs.microsoft.com/en-us/sysinternals/downloads/contig) will help you create the placeholders files.

@update - Vmware released a note and a fix - this affects all Windows 2012/2016 running VMware tools 10.1.x
https://kb.vmware.com/s/article/2149178