DeepSea deployment results in errors for the iSCSI Gateway

  • 7018668
  • 27-Feb-2017
  • 27-Feb-2017

Environment

SUSE Enterprise Storage 4

Situation

When deploying SUSE Enterprise Storage using Salt/DeepSea and having an iSCSI Gateway configured, the following errors are seen when running stage 4:

:~ # salt-run state.orch ceph.stage.4
[WARNING ] Could not write out jid file for job 20170221120717480962. Retrying.
[WARNING ] Could not write out jid file for job 20170221120717480962. Retrying.
[WARNING ] Could not write out jid file for job 20170221120717480962. Retrying.
[WARNING ] Could not write out jid file for job 20170221120717480962. Retrying.
[WARNING ] Could not write out jid file for job 20170221120717480962. Retrying.
[ERROR   ] prep_jid could not store a jid after 5 tries.
[ERROR   ] Could not store job cache info. Job details for this run may be unavailable.
[ERROR   ] Run failed on minions: server.lab.suse.de
Failures:
    server.lab.suse.de:
      Name: lrbd - Function: pkg.installed - Result: Clean Started: - 12:07:27.126075 Duration: 484.869 ms
    ----------
              ID: enable lrbd
        Function: service.running
            Name: lrbd
          Result: False
         Comment: Service lrbd has been enabled, and is dead
         Started: 12:07:27.611338
        Duration: 894.987 ms
         Changes:   
                  ----------
                  lrbd:
                      True
      Name: service.restart - Function: module.run - Result: Changed Started: - 12:07:28.506752 Duration: 1359.962 ms
    
    Summary for server.lab.suse.de                                                                                                                                    
    ------------                                                                                                                                                           
    Succeeded: 2 (changed=2)
    Failed:    1
    ------------
    Total states run:     3                                                                                                                                                
    Total run time:   2.740 s

No minions matched the target. No command was sent, no jid was assigned.
adminserver.lab.suse.de_master:
  Name: auth - Function: salt.state - Result: Changed Started: - 12:07:18.813741 Duration: 1248.03 ms
  Name: keyring - Function: salt.state - Result: Changed Started: - 12:07:20.061930 Duration: 255.429 ms
  Name: sysconfig - Function: salt.state - Result: Changed Started: - 12:07:20.317510 Duration: 358.052 ms
  Name: iscsi import - Function: salt.state - Result: Changed Started: - 12:07:20.675734 Duration: 6089.31 ms
----------
          ID: iscsi apply
    Function: salt.state
      Result: False
     Comment: Run failed on minions: server.lab.suse.de
              Failures:                                                                                                                                                    
                  server.lab.suse.de:                                                                                                                                 
                    Name: lrbd - Function: pkg.installed - Result: Clean Started: - 12:07:27.126075 Duration: 484.869 ms
                  ----------
                            ID: enable lrbd
                      Function: service.running
                          Name: lrbd
                        Result: False
                       Comment: Service lrbd has been enabled, and is dead
                       Started: 12:07:27.611338
                      Duration: 894.987 ms
                       Changes:   
                                ----------
                                lrbd:
                                    True
                    Name: service.restart - Function: module.run - Result: Changed Started: - 12:07:28.506752 Duration: 1359.962 ms
                  
                  Summary for server.lab.suse.de                                                                                                                      
                  ------------                                                                                                                                             
                  Succeeded: 2 (changed=2)
                  Failed:    1
                  ------------
                  Total states run:     3
                  Total run time:   2.740 s
     Started: 12:07:26.765193
    Duration: 3112.397 ms
     Changes:   
  Name: cephfs pools - Function: salt.state - Result: Changed Started: - 12:07:29.877742 Duration: 3778.575 ms
  Name: mds auth - Function: salt.state - Result: Changed Started: - 12:07:33.656472 Duration: 1072.477 ms
  Name: mds - Function: salt.state - Result: Changed Started: - 12:07:34.729093 Duration: 525.992 ms
  Name: rgw auth - Function: salt.state - Result: Clean Started: - 12:07:35.255234 Duration: 821.704 ms
  Name: rgw - Function: salt.state - Result: Clean Started: - 12:07:36.077085 Duration: 104.533 ms
  Name: openattic auth - Function: salt.state - Result: Changed Started: - 12:07:36.181747 Duration: 502.064 ms
  Name: openattic - Function: salt.state - Result: Changed Started: - 12:07:36.683972 Duration: 37952.848 ms
  Name: openattic keyring - Function: salt.state - Result: Changed Started: - 12:08:14.636991 Duration: 517.841 ms
  Name: openattic oaconfig - Function: salt.state - Result: Changed Started: - 12:08:15.155038 Duration: 13705.437 ms

Summary for adminserver.lab.suse.de_master
-------------
Succeeded: 13 (changed=11)
Failed:     1
-------------
Total states run:     14
Total run time:   70.045 s

Resolution

The errors are cosmetic and can be ignored.

Additional Information

The errors are more often encountered when running the Admin node as a virtual machine. To verify the configuration was successful run only the iSCSI specific subset (that forms part of Stage 4) again:

salt-run state.orch ceph.stage.iscsi

This should return something like the following:

:~ # salt-run state.orch ceph.stage.iscsi
adminserver.lab.suse.de_master:
  Name: auth - Function: salt.state - Result: Changed Started: - 17:26:33.706143 Duration: 1090.991 ms
  Name: keyring - Function: salt.state - Result: Clean Started: - 17:26:34.797278 Duration: 356.39 ms
  Name: sysconfig - Function: salt.state - Result: Changed Started: - 17:26:35.153830 Duration: 252.811 ms
  Name: iscsi import - Function: salt.state - Result: Changed Started: - 17:26:35.406796 Duration: 5123.672 ms
  Name: iscsi apply - Function: salt.state - Result: Changed Started: - 17:26:40.530622 Duration: 6884.202 ms

Summary for adminserver.hwlab.suse.de_master
------------
Succeeded: 5 (changed=4)
Failed:    0
------------
Total states run:     5
Total run time:  13.708 s

Alternatively to manually verify the installation take the following steps on the node designated to run the iSCSI Gateway:

1. Verify the lrbd service is running:

:~ # systemctl status lrbd.service 
● lrbd.service - configures target.service from Ceph
   Loaded: loaded (/usr/lib/systemd/system/lrbd.service; enabled; vendor preset: disabled)
   Active: active (exited) since Tue 2017-02-21 12:07:29 CET; 2h 55min ago
  Process: 32216 ExecStart=/usr/sbin/lrbd $LRBD_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 32216 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 512)
   CGroup: /system.slice/lrbd.service

2. Verify the configured RBD images were mapped, by default and with the below example an image demo is created:

:~ # rbd showmapped
id pool image snap device    
0  rbd  demo  -    /dev/rbd0

3. Verify the iSCSI Gateway is listening (the second option uses the default target port):

:~ # netstat -plt | grep iscsi
tcp        0      0 server.:iscsi-target *:*                     LISTEN      -         
          
:~ # netstat -plnt | grep 3260
tcp        0      0 192.100.100.10:3260      0.0.0.0:*               LISTEN      -  

Feedback service temporarily unavailable. For content questions or problems, please contact Support.