NetBackup 8.X - Windows 2016 - KB4048953 - AddLegacyDriverFiles: Unable to back up image of binary Microsoft Link-Layer Discovery Protocol.

Test environment:
OS Name: Microsoft Windows Server 2016 Standard
OS Version: 10.0.14393 N/A Build 14393
NetBackup 8.X


Error from job details :

Dec 20, 2017 10:15:52 PM - Warning bpbrm (pid=20340) from client test-windows-2016.netbackup.local: WRN - can't open object: System State:\System Files\System Files (WIN32 3758134305: Unknown error)
Dec 20, 2017 10:15:52 PM - Error bpbrm (pid=20340) from client test-windows-2016.netbackup.local: ERR - Error encountered while attempting to get additional files for System State:\

Windows error :
Cryptographic Services failed while processing the OnIdentity() call in the System Writer Object.
Details: AddLegacyDriverFiles: Unable to back up image of binary Microsoft Link-Layer Discovery Protocol.
System Error:Access is denied.

Details :

This problem occurs because VSS System Writer does not have permission to read the NT AUTHORITY\SERVICE (service account). When System Writer runs as a cryptographic service and tries to read the Mslldp.sys information from a Microsoft Link-Layer Discovery Protocol driver, the "access denied" error is generated.

The binary security descriptor for the driver is located here:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MsLldp\Security

Microsoft KB - https://support.microsoft.com/en-us/help/3209092/event-id-513-when-running-vss-in-windows-server

This event log entry can be safely ignored. To prevent this entry from being logged, grant the required permission to the Microsoft Link-Layer Discovery Protocol driver (Mslldp.dll) to process System Writer.

To do this, follow these steps:

Open an administrative Command Prompt window, and then run the following command to check the current permissions:
sc sdshow mslldp
Copy the output string from step 1, append it with (A;;CCLCSWLOCRRC;;;SU), and then run the following command to add the access permission to Mslldp.dll:
sc sdset mslldp <string>

For example, run the following command:
sc sdset mslldp D:(D;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;BG)(A;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;SY)(A;;CCDCLCSWRPDTLOCRSDRCWDWO;;;BA)(A;;CCLCSWRPWPDTLOCRRC;;;SO)(A;;LCRPWP;;;S-1-5-80-3141615172-2057878085-1754447212-2405740020-3916490453)(A;;CCLCSWLOCRRC;;;SU)

NetBackup 8.1 - Performance issue (during upgrade but also after)

At this moment the note is not public so here is a little preview.

I have seen performance issues when I tested NetBackup 8.1 under lab and I will detail them later.

Now, back to the article.

Well, I will tell you the cause, it's all those "senior bat experts" working at Veritas.

Recommendation would be to upgrade everything to 8.1 but since there are performance issue why I would upgrade ? Leaving clients on 8.0 will also create issue with restores.

It was not enough that the "focus on security" has created millions of issues with these client certificates ....

There are also performance issues after the upgrade not only during.

@update - link is here : https://www.veritas.com/support/en_US/article.100041390

Windows 2016 - KB4013418 - NetBackup 7.x 8.x

Test environment:
OS Name: Microsoft Windows Server 2016 Standard
OS Version: 10.0.14393 N/A Build 14393
NetBackup 7.x and 8.x

Error from job details :
Dec 20, 2017 10:15:52 PM - Warning bpbrm (pid=20340) from client test-windows-2016.netbackup.local: WRN - can't open object: System State:\System Files\System Files (WIN32 3758134305: Unknown error)
Dec 20, 2017 10:15:52 PM - Error bpbrm (pid=20340) from client test-windows-2016.netbackup.local: ERR - Error encountered while attempting to get additional files for System State:\

Enabled debug logs in NetBackup - 2 General, 5 Verbose
Create folders bpkar under Netbackup\logs\
Now let's read 31MB of logs :)
Error does not appear under code 8, 16 or 32 so it should be fun to find it.


Digging :
09:37:53.484 [548.3096] <2> ov_log::V_GlobalLog: INF - Informational: calling IVssBackupComponents::SetBackupSucceeded with status 'SUCCESS (0x00000000)' for Component 'TasksStore' in SHADOW::CloseComponent
09:37:53.854 [548.3096] <2> ov_log::V_GlobalLog: INF - Informational: calling IVssBackupComponents::SetBackupSucceeded with status 'SUCCESS (0x00000000)' for Component 'WriterMetadataStore' in SHADOW::CloseComponent
09:37:54.388 [548.3096] <2> ov_log::V_GlobalLog: INF - Informational: calling IVssBackupComponents::SetBackupSucceeded with status 'SUCCESS (0x00000000)' for Component 'PerformanceCounters' in SHADOW::CloseComponent
09:43:06.123 [548.3096] <2> ov_log::V_GlobalLog: INF - Informational: calling IVssBackupComponents::SetBackupSucceeded with status 'SUCCESS (0x00000000)' for Component 'Volume{8cbe8c15-0000-0000-0000-100000000000}' in SHADOW::CloseComponent
09:43:09.260 [548.3096] <2> ov_log::V_GlobalLog: INF - Informational: calling IVssBackupComponents::SetBackupSucceeded with status 'SUCCESS (0x00000000)' for Component 'BCD' in SHADOW::CloseComponent
09:43:09.559 [548.3096] <2> ov_log::V_GlobalLog: INF - Informational: calling IVssBackupComponents::SetBackupSucceeded with status 'SUCCESS (0x00000000)' for Component 'COM+ REGDB' in SHADOW::CloseComponent
09:43:11.615 [548.3096] <2> ov_log::V_GlobalLog: INF - Informational: calling IVssBackupComponents::SetBackupSucceeded with status 'SUCCESS (0x00000000)' for Component 'Registry' in SHADOW::CloseComponent
09:43:12.352 [548.3096] <2> ov_log::V_GlobalLog: INF - Informational: calling IVssBackupComponents::SetBackupSucceeded with status 'SUCCESS (0x00000000)' for Component 'WMI' in SHADOW::CloseComponent
......
Everything looks good so far.
.....
09:43:12.419 [548.3096] <4> tar_backup::backup_endarg_state: INF - finish error count: 1
......
So it finished with status 1.

Now up again to find the related error:
.....
09:41:50.768 [548.3096] <2> ov_log::V_GlobalLog: INF - Status 0x00000003 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy29\windows\systemroot\system32\drivers\vsock.sys in brUtil::GenerateFileList
09:41:50.768 [548.3096] <2> ov_log::V_GlobalLog: INF - Status 0x00000003 for file c:\windows\\systemroot\system32\drivers\vsock.sys in brUtil::GenerateFileList
09:42:06.521 [548.3096] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy29\windows\servicing\version\10.0.14393.0\amd64_installed in brUtil::GenerateFileList
09:42:06.521 [548.3096] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\servicing\version\10.0.14393.0\amd64_installed in brUtil::GenerateFileList
09:42:52.975 [548.3096] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy29\windows\servicing\version\10.0.14393.0\x86_installed in brUtil::GenerateFileList
09:42:52.975 [548.3096] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\servicing\version\10.0.14393.0\x86_installed in brUtil::GenerateFileList
09:43:04.172 [548.3096] <2> ov_log::V_GlobalLog: INF - Informational: Component:System Files no files are present to protect.
09:43:04.172 [548.3096] <2> ov_log::V_GlobalLog: INF - brUtil::SetBackupMethod(): Backup method set to BRU_BACKUP_METHOD_NORMAL(0) [Time=0]
09:43:04.172 [548.3096] <2> ov_log::V_GlobalLog: INF - Component Set System Files has 3 files missing on the snapshot
.......

3 files missing from the snapshot .... iuhuu ...

Microsoft documentation shows only 2 of them, 3th one is related to VMware Tools.

File List: Path = c:\windows\\systemroot\system32\drivers, Filespec = vsock.sys (Image path issue)

Microsoft link : https://support.microsoft.com/en-us/help/4052556/error-the-system-writer-is-not-found-in-the-backup-windows-server-2016

On a computer that is running Windows Server 2016 (Version 1607, build 14393.693), you install a servicing stack update. For example, the March 14, 2017 Servicing Stack Update (KB 4013418).

The folder C:\Windows\servicing\Version\10.0.14393.1051 is automatically created, and the amd64_installed and x86_installed files are saved to this folder.

For example, the following files are removed:

C:\Windows\servicing\Version\10.0.14393.0\amd64_installed
C:\Windows\servicing\Version\10.0.14393.0\x86_installed

Resolution: To fix the issue create new placeholder files with the same name for the files that are reported as not found.

Contig (https://docs.microsoft.com/en-us/sysinternals/downloads/contig) will help you create the placeholders files.

@update - Vmware released a note and a fix - this affects all Windows 2012/2016 running VMware tools 10.1.x
https://kb.vmware.com/s/article/2149178

NetBackup 8.1 and MySQL protection (Windows)

Test environment:
- NetBackup version 8.1
- NetBackup for MYSQL version 8.1.0.0
- Windows 2016 x64
- MySQL 5.7.20

Veritas documentation related to MySQL protection can be found here: https://www.veritas.com/content/support/en_US/doc/127649594-127651726-1


First we see on Veritas website the amazing new feature:

"
Open Source is the wave of the future
Future-proof your RDBMS data protection and ride the wave
By 2018, more than 70% of new in-house applications will be developed on an OSDBMS, and 50% of existing commercial RDBMS instances will have been converted or will be in process of being converted. Whether your RDBMS is traditional or Open Source, rely on NetBackup to protect them.
Protect and harness the speed and economics of Open Source RDBMS
Uniquely integrates leading open source RDBMS database workloads with NetBackup
Get new workload support with open APIs for third party integration for NetBackup
Check back soon for MySQL, PostgreSQL, MariaDB, and SQLite for plugin availability
"

Windows testing:

To fully test this I have used a test database which can be found here : https://github.com/datacharmer/test_db

Simply download the zip and import the database using "mysql -uroot -p < employees.sql"

Perform a check on the database to be sure the import was ok "mysql -uroot -p -t < test_employees_md5.sql"

1. About Netbackup for MySQL

NetBackup supports full instance backups of the MySQL database
NetBackup supports full instance restores of MySQL backups


2. Installing NetBackup for MySQL agent

Download NBMySQLagent.zip from Veritas website
Run setup.exe - press 2 next and one finish. Agent will install by default under "C:\Program Files\Veritas\NBMySQLAgent"


3. Configuring NetBackup for MySQL agent

If you are planning to use another user to perform backups then add it.
GRANT SELECT, INSERT, UPDATE, CREATE, DROP, RELOAD, SHUTDOWN, FILE,
INDEX, ALTER, SUPER, LOCK TABLES, CREATE VIEW, SHOW VIEW, TRIGGER,
CREATE ROUTINE, DELETE, EVENT, ALTER ROUTINE ON *.* TO
'THE_USERNAME'@'localhost' IDENTIFIED BY 'THE_PASSWORD'

Edit the environmental path and include NetBackup binary folder. If you used the default path when Netbackup client was installed, then it should be "C:\Program Files\VERITAS\NetBackup\bin"

Edit nbmysql.conf carefully as following:
- DB_PORT : port on which the MySQL listens (default 3306)
- DB_USER : user that will be able to connect to MySQL
- MySQL_LIB_INSTALL_PATH : path to MySQL library
- NETBACKUP_MASTER_SERVER : NetBackup master server FQDN
- NETBACKUP_POLICY_NAME : name of policy name
- NETBACKUP_SCHEDULE_NAME : only Default-Application-Backup is supported so don't waist your time creating new schedules.
- NETBACKUP_CLIENT_NAME : name of NetBackup client
- SNAPSHOT_SIZE : not needed on Windows
- COPY_ID : only used for restores so it can be ignored for the moment
- MYSQL_TARGET_DIRECTORY : only used for restores so it can be ignored for the moment
- NBMYSQL_LOG_LEVEL : Logging level 4 - for testing purpose
- NBMYSQL_LOG_SIZE : 50MB for testing purpose

Veritas documentation contains a mistake as follows:

4. Create a policy in NetBackup:

Attributes : Policy type is Data Store

Schedules : Edit "Default-Application-Backup" retention

Clients : add NetBackup client name

Backup Selection : there is nothing specified in the documentation of what should be added here.

Notes:

Additional schedules are not supported.

Backup selection needs to be EMPTY.

5. Backup MySQL

From cmd trigger : "nbmysql -o backup -S master_server_name -P policy_name -s schedule_name -l mysql_lib_path -p mysql_port -u mysl_user"

Even if we added all these data in nbmysql.conf you still need to enter them as they are mandatory fields.

Notes:

You cannot backup a single database, it's all or nothing.

You cannot exclude a database from backup, it's all or nothing.

You cannot trigger differential or transaction log backup.

Automatic schedule for MySQL backups cannot be done on Windows. (Documentation says otherwise but looks like nobody from Veritas really tested it)

6. Query master server and display all available backups
From cmd trigger : "nbmysql -o query"

Notes :
Query works but BackupTime is wrong !!!!!!!

7. Restore MySQL
From cmd trigger : nbmysql -o restore -S master_server_name -t target_directory -p
db_port -i copy_id -C client_name
Again, even if we did add all this information in nbmysql.conf it's again needed.

Notes : 

You cannot restore a single database, it's all or nothing.

Everytime you want to restore from a different date you need to modify nbmysql.conf and specify the CopyID and destination path (make sure the folder exists and it's empty)
After restore is done you need to shutdown MySQL and overwrite manually the data folder with the restored data, afterwards you can start MySQL.

More fun:
Let's assume that when the backup is triggered there is an update done on several tables/columns.

NetBackup will do a flush/lock on all databases tables and afterwards it will create a shadow using SYSTEM WRITER but since our update ir running it will wait until it finishes.
After the backup finished the shadow copy is still there and will remain there forever and ever causing the shadow storage to get full at some point and creating more issues.

Final note : NetBackup MySQL agent for Windows is simple a waiste of time, mysqldump is way better from all points of view.