First, you You need to be on edX VPN to access splunk at https://splunk.edx.org and will also need to file a ticket to create an Splunk Account.
What is Splunk?
Splunk is a query interface for users to retrieve specific log messages on our web servers. Whenever you see in code "logger.info", "logger.warning" or "logger.exception". Those logs gets written into the log files onto the web server instance.
...
- The first clause "index=prod-edx" specify which environment we are querying against. The index key word points to an environment. Common values are:
- prod-edx
- stage-edx
- The second clause "service_variant=ecommerce" specify which IDA/service we are querying for. Common values are:
- lms
- cms
- ecommerce
- credentials
- discovery
- The second claus clause can also be "source=/edx/var/log/lms/edx.log", which specify the log file we are querying against. Some common log file values are
- /edx/var/log/lms/edx.log
- /edx/var/log/cms/edx.log
- /edx/var/log/ecomworker/edx.log
- /edx/var/log/ecommerce/edx.log
- /edx/var/log/credentials/edx.log
- /edx/var/log/discovery/edx.log
- The third clause is the key word within logs you want to search for.
- Blockstore currently uses a different scheme for the first two clauses. service_variant can be omitted, and instead must be included in the index:
- yes:
- index=prod-edx-blockstore
- index=stage-edx-blockstore
- no:
- index=prod-edx service_variant=blockstore
- index=stage-edx service_variant=blockstore
- yes:
Once you created the query like above, then you should select the time range you are limiting the query for. The default is "All Time". The default takes a long time to finish, so I recommend limit your query time range to something reasonable.
...
You can also create alerts yourself. If you do not have the permission in Splunk to create alerts, please create an ITSupport Devops Support ticket to to get yourself permissions