Admin: Created page with " =Purpose= Setup a demo Lab with a clustered Splunk environment, based on CLI commands '''
NOTE: Only to be used temporary in test environments''' =TestSetup= ==Host..."

2023-07-05T07:17:41Z

Created page with " =Purpose= *Setup a demo Lab with a clustered Splunk environment, based on CLI commands '''<br> *NOTE: Only to be used temporary in test environments''' =TestSetup= ==Host..."

New page

=Purpose=
*Setup a demo Lab with a clustered Splunk environment, based on CLI commands '''<br>
*NOTE: Only to be used temporary in test environments'''

=TestSetup=

==Host Environment==
*/etc/hosts
#List of required Splunk instances with IP and Hostname in order of the installation
#-----------------------------------------------------------------------------------
#Index Cluster
192.168.178.109 splunk-cluster-master
192.168.178.110 splunk-idx-cluster-indexer-0
192.168.178.111 splunk-idx-cluster-indexer-1
192.168.178.112 splunk-idx-cluster-indexer-2
#Search Head Cluster
192.168.178.113 splunk-sh-cluster-search-head-0
192.168.178.114 splunk-sh-cluster-search-head-1
192.168.178.115 splunk-sh-cluster-search-head-2
192.168.178.116 splunk-sh-cluster-deployer
#Monitoring
192.168.178.117 splunk-mc-monitoring-console
#Universal Forwarder
192.168.178.118 splunk-uf-01
192.168.178.120 splunk-uf-02

==Templates==
The one in bold is used
'''splunk-8.2.11-e73c56f930c5-linux-2.6-amd64.deb'''
splunk-8.2.11-e73c56f930c5-x64-release.msi
splunk-9.0.5-e9494146ae5c-linux-2.6-amd64.deb
'''splunkforwarder-9.0.5-e9494146ae5c-linux-2.6-amd64.deb'''
splunkforwarder-9.0.5-e9494146ae5c-x64-release.msi

==General==

===Environment requirements===
*Debian Bookworm, Bridged VMs for each Splunk instance (x 11), each with 2CPUs, 4GB RAM, 64GB HD
*Working as root
*Fully network access to each other, no port restrictions
*Internet access everywhere (No Proxies!)

===Splunk disclaimer===
'''Important Note''' that this installation is based on the '''trial License''' which Splunk set at the first start. <br>
Once started it counts for '''60 days''' before the setup becomes unusable anymore.

===Quickstart Reference===
*Debian specific but may work on Ubuntu too
apt-get install curl
dpkg -i splunk-8.2.11-e73c56f930c5-linux-2.6-amd64.deb
/opt/splunk/bin/splunk start --accept-license --answer-yes
/opt/splunk/bin/splunk enable web-ssl
/opt/splunk/bin/splunk set web-port 443
/opt/splunk/bin/splunk restart

===Quickdelete Reference===
*Debian specific but may work on Ubuntu too
dpkg --purge --force-all splunk
rm -rf /opt/splunk

==Cluster==
===Cluster Master===
*Start with the cluster master, note replication_factor and search_factor, the secret requires a min length
root@splunk-cluster-master:'''/opt/splunk/bin/splunk edit cluster-config -mode master -replication_factor 3 -search_factor 2 -secret idxSymmKey'''
root@splunk-cluster-master:'''/opt/splunk/bin/splunk restart

===Cluster Peer 0 ===
root@splunk-idx-cluster-indexer-0:/opt/splunk/bin/splunk enable listen 9997
root@splunk-idx-cluster-indexer-0:/opt/splunk/bin/splunk edit cluster-config -mode peer -manager_uri https://splunk-cluster-master:8089 -secret idxSymmKey -replication_port 9100
root@splunk-idx-cluster-indexer-0:/opt/splunk/bin/splunk restart
===Cluster Peer 1 ===
root@splunk-idx-cluster-indexer-1:/opt/splunk/bin/splunk enable listen 9997
root@splunk-idx-cluster-indexer-1:/opt/splunk/bin/splunk edit cluster-config -mode peer -manager_uri https://splunk-cluster-master:8089 -secret idxSymmKey -replication_port 9100
root@splunk-idx-cluster-indexer-1:/opt/splunk/bin/splunk restart

===Cluster Peer 2 ===
root@splunk-idx-cluster-indexer-2:/opt/splunk/bin/splunk enable listen 9997
root@splunk-idx-cluster-indexer-2:/opt/splunk/bin/splunk edit cluster-config -mode peer -manager_uri https://splunk-cluster-master:8089 -secret idxSymmKey -replication_port 9100
root@splunk-idx-cluster-indexer-2:/opt/splunk/bin/splunk restart

==Search Head==
'''Note: the secret requires a min length otherwise the setup will not work'''
===SH0===
root@splunk-sh-cluster-search-head-0:/opt/splunk/bin/splunk init shcluster-config -mgmt_uri https://splunk-sh-cluster-search-head-0:8089 -replication_port 9200 -secret shclusterabcd
root@splunk-sh-cluster-search-head-0:/opt/splunk/bin/splunk restart
===SH1===
root@splunk-sh-cluster-search-head-1:/opt/splunk/bin/splunk init shcluster-config -mgmt_uri https://splunk-sh-cluster-search-head-1:8089 -replication_port 9200 -secret shclusterabcd
root@splunk-sh-cluster-search-head-1:/opt/splunk/bin/splunk restart
===SH2===
root@splunk-sh-cluster-search-head-2:/opt/splunk/bin/splunk init shcluster-config -mgmt_uri https://splunk-sh-cluster-search-head-2:8089 -replication_port 9200 -secret shclusterabcd
root@splunk-sh-cluster-search-head-2:/opt/splunk/bin/splunk restart
===Bootstrap===
root@splunk-sh-cluster-search-head-0:/opt/splunk/bin/splunk bootstrap shcluster-captain -servers_list https://splunk-sh-cluster-search-head-0:8089,https://splunk-sh-cluster-search-head-1:8089,https://splunk-sh-cluster-search-head-2:8089
root@splunk-sh-cluster-search-head-0:/opt/splunk/bin/splunk restart
*It makes sense to check the setup after the above was applied. Check kvstore status and the mc

===Add SH0 to Index Cluster===
root@splunk-sh-cluster-search-head-0:/opt/splunk/bin/splunk edit cluster-config -mode searchhead -manager_uri https://splunk-cluster-master:8089 -secret idxSymmKey
root@splunk-sh-cluster-search-head-0:/opt/splunk/bin/splunk restart

===Add SH1 to Index Cluster===
root@splunk-sh-cluster-search-head-1:/opt/splunk/bin/splunk edit cluster-config -mode searchhead -manager_uri https://splunk-cluster-master:8089 -secret idxSymmKey
root@splunk-sh-cluster-search-head-1:/opt/splunk/bin/splunk restart
===Add SH2 to Index Cluster===
root@splunk-sh-cluster-search-head-2:/opt/splunk/bin/splunk edit cluster-config -mode searchhead -manager_uri https://splunk-cluster-master:8089 -secret idxSymmKey
root@splunk-sh-cluster-search-head-2:/opt/splunk/bin/splunk restart

==Universal Forwarder==
Check This: https://docs.splunk.com/Documentation/Splunk/9.0.5/DistSearch/SHCandindexercluster
===UF0===
*/opt/splunkforwarder/etc/system/local/inputs.conf
[monitor:///var/log/auth.log]
disabled=false
sourcetype=auth

[monitor:///var/log/syslog]
disabled=false
sourcetype=syslog

*/opt/splunkforwarder/etc/system/local/outputs.conf
[tcpout]
defaultGroup=indexer1

[tcpout:indexer1]
server=192.168.178.110:9997,192.168.178.111:9997,192.168.178.112:9997

===UF1===
'''NOTE THAT UF1 USES ONLY ONE INDEXER'''
*/opt/splunkforwarder/etc/system/local/inputs.conf
[monitor:///var/log/auth.log]
disabled=false
sourcetype=auth

[monitor:///var/log/syslog]
disabled=false
sourcetype=syslog

*/opt/splunkforwarder/etc/system/local/outputs.conf
[tcpout]
defaultGroup=indexer1

[tcpout:indexer1]
server=192.168.178.110:9997

==Testing==
Happy testing
*Aftermath
**Switch to '''Distributed Mode''' from the mc web page of the '''https://splunk-cluster-master'''
**Switch off the MC on all Splunk Instances except the cluster-master
/opt/splunk/bin/splunk disable app splunk_monitoring_console
*consideration
**Remove the MC app from the SH Cluster(s) beforehand, just to make sure its being removed
**The MC must run only once clusterwide on: '''https://splunk-cluster-master''' - nowwhere else

==Reference==
*https://github.com/juanlittledevil/splunk-cluster-training/blob/master/README.md
*https://docs.splunk.com/Documentation/Splunk/9.0.4/Indexer/MultisiteCLI
*Nice: https://www.aplura.com/assets/pdf/shc_cheatsheet.pdf
*https://kinneygroup.com/blog/the-rules-of-the-search-head-cluster-shc/
*https://blog.avotrix.com/splunk-search-head-cluster-configuration/
*https://ashvinpandey.medium.com/steps-to-create-search-head-cluster-in-splunk-19106909e148

Setup Splunk with Index and SearchHead Cluster - Revision history

Admin: Created page with " =Purpose= *Setup a demo Lab with a clustered Splunk environment, based on CLI commands ''' *NOTE: Only to be used temporary in test environments''' =TestSetup= ==Host..."

Admin: Created page with " =Purpose= Setup a demo Lab with a clustered Splunk environment, based on CLI commands '''
NOTE: Only to be used temporary in test environments''' =TestSetup= ==Host..."