Summary
Logstash pod goes down in Arcsight Interset 6.10, throwing the
exception=>"LogStash::ConfigurationError", message=>"Expected one of #, { at line ... "
Error
I have arcsight interset 6.10 installation and after reinstall of interset components I have an error in logstash pod:
[2020-12-03T14:51:50,521][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:kafka_cef_es, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 40, column 9 (byte 1314) after output {\n elasticsearch { \n hosts => [\"elasticsearch-svc:9200\"] # example: http://devil-search1.ad.interset.com:9200\n ssl => true\n cacert => \"/vault-crt/RE/issue_ca.crt\"\n user => elastic\n password => 1qaz2wsx\n ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:22:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:90:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:43:in `block in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in `block in exclusive'", "org/jruby/ext/thread/Mutex.java:165:in `synchronize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in `exclusive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:39:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:334:in `block in converge_state'"]}
After that error logstash pod goes down every time. All other pods (including analytics) working fine
Cause
Logstach pod at 6.1 will not accept elasticsearch passwords with special characters or starting with a number.
In the 6.2 new version logstash was upgraded to a newer version and doesn't have the same problem.
Fix
There are 3 ways to overcome this probelm:
1) Reinstall Arcsight Interset 6.1 without changing the default password for elasticsearch or changing it to something very simple, without special characters and starting with a letter, not a number.
2) Upgrade to Arcsight Interset (Intelligence) 6.2
3) Uninstall Arcsight Interset 6.1 and Install Arcsight Interset (Intelligence) 6.2