Unable to search events in secondary storage

Unable to search events in secondary storage. If searching partitions in primary storage, events are returned. If the search needs to access partitions from secondary storage no events are found.

The server0.0.log shows the following error when trying to access the secondary storage. 

Tue Nov 17 15:55:36 CET 2020|SEVERE|pool-7920785-thread-1|esecurity.ccs.comp.event.indexedlog.IndexedLogSearchJob$PartitionHitsRetrieverTask.call

Error IXSEARCH0002(IndexedLog-Search): Unexpected error performing search for the day Nov 16, 2020 (UTC). Search job id: c6f520315a6b671eD34070C00B1210399084005056B648C0

Root cause: Unable to mount squashfs index '/var/opt/novell/sentinel/data/eventdata/events/20201116_408E7E50-C02E-4325-B7C5-2B9FE4853476/index.sqfs': sudo: PAM account management error: Permission denied (java.io.IOException)

com.novell.sentinel.util.logging.error.indexedlog.IXSEARCH0002: Unexpected error performing search for the day Nov 16, 2020 (UTC). Search job id: c6f520315a6b671eD34070C00B1210399084005056B648C0

at esecurity.ccs.comp.event.indexedlog.IndexedLogSearchJob$PartitionHitsRetrieverTask.call(IndexedLogSearchJob.java:716)

at esecurity.ccs.comp.event.indexedlog.IndexedLogSearchJob$PartitionHitsRetrieverTask.call(IndexedLogSearchJob.java:639)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.io.IOException: Unable to mount squashfs index '/var/opt/novell/sentinel/data/eventdata/events/20201116_408E7E50-C02E-4325-B7C5-2B9FE4853476/index.sqfs': sudo: PAM account management error: Permission denied

Go to /etc/pam.d/sudo  and confirm that the sudo file is similar to the output below. 

sentinel:/etc/pam.d # cat sudo

#%PAM-1.0

auth include common-auth

account include common-account

password include common-password

session optional pam_keyinit.so revoke

session include common-session

# session optional pam_xauth.so

If the sudo file looks okay the next step would be to get the PAM administrator involved to help troubleshoot what else might have been recently updated in PAM to cause this permissions issue. 

The customer is using the PAM application and something has changed that has impacted permissions. In order for Sentinel to search data in the secondary storage location we must first mount the squashfs index and then unsquash the index file. This process works in conjunction with the root and novell accounts. Typically something has changed in the /etc/sudoers file that causes this type of permissions problem. In this case the user has the PAM application installed in their environment and there was a change done in the /etc/pam.d/sudo file that caused the permissions issue.