Unable to search events in secondary storage. If searching partitions in primary storage, events are returned. If the search needs to access partitions from secondary storage no events are found.
The server0.0.log shows the following error when trying to access the secondary storage.
Tue Nov 17 15:55:36 CET 2020|SEVERE|pool-7920785-thread-1|esecurity.ccs.comp.event.indexedlog.IndexedLogSearchJob$PartitionHitsRetrieverTask.callError IXSEARCH0002(IndexedLog-Search): Unexpected error performing search for the day Nov 16, 2020 (UTC). Search job id: c6f520315a6b671eD34070C00B1210399084005056B648C0Root cause: Unable to mount squashfs index '/var/opt/novell/sentinel/data/eventdata/events/20201116_408E7E50-C02E-4325-B7C5-2B9FE4853476/index.sqfs': sudo: PAM account management error: Permission denied (java.io.IOException)com.novell.sentinel.util.logging.error.indexedlog.IXSEARCH0002: Unexpected error performing search for the day Nov 16, 2020 (UTC). Search job id: c6f520315a6b671eD34070C00B1210399084005056B648C0at esecurity.ccs.comp.event.indexedlog.IndexedLogSearchJob$PartitionHitsRetrieverTask.call(IndexedLogSearchJob.java:716)at esecurity.ccs.comp.event.indexedlog.IndexedLogSearchJob$PartitionHitsRetrieverTask.call(IndexedLogSearchJob.java:639)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Caused by: java.io.IOException: Unable to mount squashfs index '/var/opt/novell/sentinel/data/eventdata/events/20201116_408E7E50-C02E-4325-B7C5-2B9FE4853476/index.sqfs': sudo: PAM account management error: Permission denied
Go to /etc/pam.d/sudo and confirm that the sudo file is similar to the output below.sentinel:/etc/pam.d # cat sudo#%PAM-1.0auth include common-authaccount include common-accountpassword include common-passwordsession optional pam_keyinit.so revokesession include common-session# session optional pam_xauth.soIf the sudo file looks okay the next step would be to get the PAM administrator involved to help troubleshoot what else might have been recently updated in PAM to cause this permissions issue.
The customer is using the PAM application and something has changed that has impacted permissions. In order for Sentinel to search data in the secondary storage location we must first mount the squashfs index and then unsquash the index file. This process works in conjunction with the root and novell accounts. Typically something has changed in the /etc/sudoers file that causes this type of permissions problem. In this case the user has the PAM application installed in their environment and there was a change done in the /etc/pam.d/sudo file that caused the permissions issue.