Summary
Question
A. launch individual sm-rte debug pod that is not a part of original SM-RTE cluster
B. login to sm-rte pod directly through external network
C. add debug parameters into sm-rte pod manually
Answer
# All steps are verified in SMA-SM 2017-Nov GA
# Contents
A. launch individual sm-rte debug pod that is not a part of original SM-RTE cluster
B. login to sm-rte pod directly through external network
B1. select one of sm-rte pods
B2. enable port-forward for selected pod on master node ( master -> sm-rte pod )
B3. make SSL tunnel between your laptop and Master
B4. connect sm-rte pod directly through external network(your laptop)
C. add debug parameters into sm-rte pod manually
A. launch individual sm-rte debug pod that is not a part of original SM-RTE cluster
- we will launch temporary sm-rte debug pod for troubleshooting purpose.
1) make a copy of original sm-rte's yaml file
$ cd /var/vols/itom/core/suite-install/itsma/output/itom-sm-9.52.2021.12/yamls
$ cp sm-rte.yaml sm-rte2.yaml
$ vi sm-rte2.yaml
a) remove Service section, line 1 to 18
~~~
apiVersion: v1
kind: Service
metadata:
name: sm-rte-svc
labels:
app: sm-rte-svc
itsmaService: itom-sm
namespace: itsma3
spec:
# type: NodePort
# sessionAffinity: ClientIP
ports:
- port: 13080
# nodePort: 30180
selector:
app: sm-rte
---
~~~
b) change label for separating it from existing SM-RTE cluster
~~~
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: sm-rte2
labels:
app: sm-rte2
itsmaService: itom-sm2
namespace: itsma3
spec:
replicas: 1
template:
metadata:
labels:
app: sm-rte2
itsmaService: itom-sm2
~~~
2) check current sm-rte pods before launching new debug pod
$ kubectl get pods --all-namespaces --selector="app in (sm-rte,sm-rte2)" --show-labels
3) launch new sm-rte that is for troubleshooting
$ kubectl create -f sm-rte2.yaml
$ kubectl create -f sm-rte2.yaml
--> for delete pod $ kubectl delete -f sm-rte2.yaml
4) verify all sm-rte pods and its label
$ kubectl get pods --all-namespaces --selector="app in (sm-rte,sm-rte2)" --show-labels
[root@btp01sm64 rte]# kubectl get pods --all-namespaces --selector="app in (sm-rte,sm-rte2)" --show-labels
NAMESPACE NAME READY STATUS RESTARTS AGE LABELS
itsma3 sm-rte-1549193434-c56sb 2/2 Running 0 7d app=sm-rte,itsmaService=itom-sm,pod-template-hash=1549193434
itsma3 sm-rte-1549193434-smhxb 2/2 Running 0 3d app=sm-rte,itsmaService=itom-sm,pod-template-hash=1549193434
itsma3 sm-rte2-2042745150-3hfzz 2/2 Running 0 13m app=sm-rte2,itsmaService=itom-sm2,pod-template-hash=2042745150
--> sm-rte2-2042745150-3hfzz is newly deployed
B. login to sm-rte pod directly through external network
B1. select one of sm-rte pods
- lists all of sm-rte pods
$ kubectl get pods --all-namespaces --selector="app=sm-rte"
$ kubectl get pods --all-namespaces --selector="app in (sm-rte,sm-rte2)" --show-labels
ex)
[root@btp01sm64 rte]# kubectl get pods --all-namespaces --selector="app=sm-rte"
NAMESPACE NAME READY STATUS RESTARTS AGE
itsma3 sm-rte-1549193434-c56sb 2/2 Running 0 9d
itsma3 sm-rte-1549193434-smhxb 2/2 Running 0 5d
[root@btp01sm64 rte]# kubectl get pods --all-namespaces --selector="app in (sm-rte,sm-rte2)" --show-labels
NAMESPACE NAME READY STATUS RESTARTS AGE LABELS
itsma3 sm-rte-1549193434-c56sb 2/2 Running 0 9d app=sm-rte,itsmaService=itom-sm,pod-template-hash=1549193434
itsma3 sm-rte-1549193434-smhxb 2/2 Running 0 5d app=sm-rte,itsmaService=itom-sm,pod-template-hash=1549193434
itsma3 sm-rte2-2042745150-3hfzz 2/2 Running 0 2d app=sm-rte2,itsmaService=itom-sm2,pod-template-hash=2042745150
- select one of sm-rte pods
> we pick up "sm-rte2-2042745150-3hfzz"
B2. enable port-forward for selected pod on master node ( master -> sm-rte pod )
- on master node, find available port to use
$ netstat -na | grep 13080
ex)
[root@btp01sm64 rte]# netstat -na | grep 13081
[root@btp01sm64 rte]#
// no result because 13081 is available to use
// we can use any available port
- enable port-forward on master node
$ kubectl port-forward {pod_name} {portA}:13080 -n {namespace}
//
ex)
[root@btp01sm64 bin]# kubectl port-forward sm-rte2-2042745150-3hfzz 13081:13080 -n itsma3
Forwarding from 127.0.0.1:13081 -> 13080
Forwarding from [::1]:13081 -> 13080
// sm-rte pod use 13080 by default in kubernetes
// we will connect it through master:13081
master:13081 --> sm-rte-pod:13080
- verification on master node
[root@btp01sm64 rte]# netstat -na | grep 13081
tcp 0 0 127.0.0.1:13081 0.0.0.0:* LISTEN
tcp6 0 0 ::1:13081 :::* LISTEN
B3. make SSL tunnel between your laptop and Master
- install putty into your laptop
http://www.putty.org/
- launch putty & configure SSH-tunnel & login to Master
1) launch putty on your laptop
2) Category > Connection > SSH > Auth > Tunnels
-> add new forwarded port:
source port : 13082 // you can use any port available on your laptop
destination : 127.0.0.1:13081 // 13081 is master's port that is forwarding to sm-rte's pod
(*) Local
(*) Auto
--> click 'Add' button
3) Category > Sessions
-> SSH login to Master node
-> now SSL Tunnel is made between your laptop and Master
4) verify Putty's event log
-> click right mouse button on the putty's head window
-> select 'Event Log'
-> you should find this log,
2017-12-17 16:33:58 Local port 13082 forwarding to 127.0.0.1:13081
# if session forwarding is not shown in putty's session log,
clean putty's configuration and then, try again from step-1)
cmd> putty -cleanup
( it will remove windows registery data of putty )
B4. connect sm-rte pod directly through external network(your laptop)
- launch sm-win client on your laptop
> change profile,
Advanced tab :
*check : Connect to External Load Balance
Connection tab :
username : falcon
password : 123456 // it is default password
server host name : 127.0.0.1
server port number : 13082
--> try login
# if master node show this error, then install socat on all master and workers
E1214 23:57:13.863287 26245 portforward.go:332] an error occurred forwarding 13081 -> 13080:
error forwarding port 13081 to pod sm-rte-1549193434-c56sb_itsma3, uid : unable to do port forwarding: socat not found.
$ yum install socat
- monitor sm's log on master
/var/vols/itom/itsma/itsma-itsma1-global/logs/sm-9.52/rte
sm_sm-rte-1549193434-c56sb.log
sm_sm-rte-1549193434-smhxb.log
sm_sm-rte2-2042745150-3hfzz.log <---------------- we use this pod
$ tail -f sm_sm-rte2-2042745150-3hfzz.log
- Summary
your laptop(127.0.0.1:13082)
<--- SSL Tunneling by Putty --->
master-node(127.0.0.1:13081) <--- kubernets port-forward --->
sm-rte-pod(sm_sm-rte2-2042745150-3hfzz:13080)
C. add debug parameters into sm-rte pod manually
- It is for temporary investigation especially for support engineers.
updated sm.ini is disappeared if redeploying sm-rte because we just updated on pod, not image.
1) sm-rte pod's shell and make a backup for sm.ini
- lists all of sm-rte pods
$ kubectl get pods --all-namespaces --selector="app=sm-rte"
$ kubectl get pods --all-namespaces --selector="app in (sm-rte,sm-rte2)" --show-labels
ex)
[root@btp01sm64 rte]# kubectl get pods --all-namespaces --selector="app=sm-rte"
NAMESPACE NAME READY STATUS RESTARTS AGE
itsma3 sm-rte-1549193434-c56sb 2/2 Running 0 9d
itsma3 sm-rte-1549193434-smhxb 2/2 Running 0 5d
[root@btp01sm64 rte]# kubectl get pods --all-namespaces --selector="app in (sm-rte,sm-rte2)" --show-labels
NAMESPACE NAME READY STATUS RESTARTS AGE LABELS
itsma3 sm-rte-1549193434-c56sb 2/2 Running 0 9d app=sm-rte,itsmaService=itom-sm,pod-template-hash=1549193434
itsma3 sm-rte-1549193434-smhxb 2/2 Running 0 5d app=sm-rte,itsmaService=itom-sm,pod-template-hash=1549193434
itsma3 sm-rte2-2042745150-3hfzz 2/2 Running 0 2d app=sm-rte2,itsmaService=itom-sm2,pod-template-hash=2042745150
- select one of sm-rte pods
> we pick up "sm-rte2-2042745150-3hfzz"
// we use pod-name, sm-rte2-2042745150-3hfzz
$ kubectl exec -ti -n itsma3 sm-rte2-2042745150-3hfzz -c sm-rte /bin/bash
pod> cd /opt/SM_SERVER/RUN
pod> cat sm.ini
pod> cp sm.ini sm.ini.org
pod> cp sm.cfg sm.cfg.org
ex)
[root@btp01sm64 rte]# kubectl exec -ti -n itsma3 sm-rte2-2042745150-3hfzz -c sm-rte /bin/bash
sm@sm-rte2-2042745150-3hfzz:/> ls
bin dev home lib64 opt root sbin srv tmp var
boot etc lib mnt proc run selinux sys usr
sm@sm-rte2-2042745150-3hfzz:/> cd /opt/SM_SERVER/RUN
sm@sm-rte2-2042745150-3hfzz:/opt/SM_SERVER/RUN> cp sm.ini sm.ini.org
2) copy pod's sm.ini, sm.cfg into local shell and then, modify it
- in pakcaged sm-rte image, vi command is not included, so we extract file externally and modify it.
$ kubectl cp itsma3/sm-rte2-2042745150-3hfzz:opt/SM_SERVER/RUN/sm.ini -c sm-rte ./sm.ini
$ kubectl cp itsma3/sm-rte2-2042745150-3hfzz:opt/SM_SERVER/RUN/sm.cfg -c sm-rte ./sm.cfg
$ vi sm.ini
~~~ add two parameters ~~~
RTM:3
debugdbquery:999
~~~
$ vi sm.cfg
~~~ disable system.start ~~~
sm
#sm system.start
~~~
3) copy modified sm.ini, smcfg into pod
$ kubectl cp ./sm.ini itsma3/sm-rte2-2042745150-3hfzz:opt/SM_SERVER/RUN/sm.ini -c sm-rte
$ kubectl cp ./sm.cfg itsma3/sm-rte2-2042745150-3hfzz:opt/SM_SERVER/RUN/sm.cfg -c sm-rte
4) restart SM on the specific pod
$ kubectl exec -ti -n itsma3 sm-rte2-2042745150-3hfzz -c sm-rte /bin/bash
pod> cd /opt/SM_SERVER/RUN
pod> ./smstatus
pod> ./smstop
pod> ./smstart
5) verify logs with direct connections
- for direct connection, please follow step of "B. login to sm-rte pod directly through external network"
- verify logs
$ cd /var/vols/itom/itsma/itsma-itsma1-global/logs/sm-9.52/rte
$ tail -f sm_sm-rte2-2042745150-3hfzz.log
6) after troubleshooting, remove sm-rte debug pod that we built individually
$ cd /var/vols/itom/core/suite-install/itsma/output/itom-sm-9.52.2021.12/yamls
$ kubectl delete -f sm-rte2.yaml