FFFDFFF9 SMS Error When Attempting To Backup The File System

The customer made changes to their NSS pools by expanding them.  After this change they noticed that they could no longer backup their file system on this server.

 

Troubleshooting done:

1.  Investigate whether the pool expansion had any problems using partfix.nlm log files.

2.  Use TSATEST.NLM to simplify backup testing.

 

NLM available here:

 

3.  Use multiple different user identities (all admin equivalent) so as to rule out a rights issue.

4.  Apply the latest SMS updates (tsa5up23 currently).

4.  Enable tsafs debugging.  Gather the log files and investigate the findings.

 

Cool Solution found here:

 

Everytime the backup was initiated, the error FFFDFFF9 was returned.

 

Here is a sample of the log file received:

 

a02f03a0[TSATEST.NLM     1        ]000063b0: Start
a02f03a0[TSATEST.NLM     1        ]63b0    [358  ]:IN: name=<SERVER NAME>
96ada500[TSA CONNECTION Idle Threa]4ddb0   [683  ]:[TSAFS.NLM       2] to [TSA CONNECTION Idle Thread : 8FC3E2C0]
a02f03a0[TSATEST.NLM     1        ]00006180: Start
a02f03a0[TSATEST.NLM     1        ]00006210: Start
a02f03a0[TSATEST.NLM     1        ]00006210: Start
a02f03a0[TSATEST.NLM     1        ]0004fdf0: Start
a02f03a0[TSATEST.NLM     1        ]000113f0[00101]: zOpen                     failed. ret: 00005141.
a02f03a0[TSATEST.NLM     1        ]113f0   [124  ]:Volume : DATA: : COWEnabled : FALSE
a02f03a0[TSATEST.NLM     1        ]113f0   [124  ]:Volume : PRINTQ: : COWEnabled : FALSE
a02f03a0[TSATEST.NLM     1        ]00007190: Start
a02f03a0[TSATEST.NLM     1        ]00007190= fffdfff9
a02f03a0[TSATEST.NLM     1        ]00006f20: Start
96ada500[TSAFS.NLM       2        ]4ddb0   [683  ]:[NULL] to [TSAFS.NLM       2]
a02f03a0[TSATEST.NLM     1        ]63b0    [556  ]:OUT: handle=0
a02f03a0[TSATEST.NLM     1        ]000063b0= fffdfff9
a02f03a0[TSATEST.NLM     1        ]000063b0: Start
a02f03a0[TSATEST.NLM     1        ]63b0    [358  ]:IN: name=<SERVER NAME>
9fa96260[TSA CONNECTION Idle Threa]4ddb0   [683  ]:[TSAFS.NLM       3] to [TSA CONNECTION Idle Thread : 8FC3E2C0]

The FFFDFFF9 error means that the SMS layer cannot create a file.  NWSMTS_CREATE_ERROR:  Cannot create a file.

The zOpen return code of 5141 is in hex, so it is 20801 in decimal which translates to zERR_VOLUME_NOT_FOUND.  This is an error from the NSS file system.  In this instance, the NSS volume couldn't be accessed due to a failure to create a connection with the SMS engine.

 

We also found that the NSS pool expansion was a red herring and had nothing to do with this issue.

 

For this customer the issue concerned the number of temporary files in the SYS:SYSTEM\TSA directory.  TSAFS will create a temporary file for each connection used during a backup.  As the engine connection is terminated, the temporary file should be deleted.  The TSA has a maximum limit of 4096 files it can create in this directory, and it needs to create one file per connection.  If that limit on the number of files has been reached, it will not create any more files, and thus connections for the backup process will fail as will the backup itself, with the resulting above error returned.  We found that the temporary files in the before mentioned directory had not been deleted.

 

After deleting all temporary files from the SYS:SYSTEM\TSA directory (file pattern:  TSA$*), this problem was solved.  The file deletion allowed the backup program to continue without errors.