So – I wanted to get Splunk but in my organisation that was never going to happen (you want something that costs MONEY?! Ludicrous!) So we had to come up with a compromise. A colleague of mine went hunting for some open source logging software and found that the combination of Elastic Search, LogStash, Kibana and nxLog worked well. He tested it on his PC, wrote a few lines on how to get it roughly working and then sent it through to me to get it working from a server perspective. (Hi Ken! ^_^)<\/p>\n
I’ve just recently finished the base setup (with a little assistance) and it’s getting information from our production, test and development\u00a0AD DC’s\u00a0and WOAH do they waffle a lot. So after posting on Twitter that I’d got this working (because I was excited I’d got it working…duh!) someone asked me to do a blog post on the setup. So this is that blog post. I’ve written this in a way that anyone else with basic Windows Server knowledge could install this if required – yes, it is dumbed down a bit in some areas, but that’s because I wrote it to be as idiot-proof as possible.<\/p>\n
UPDATE\u00a0(3rd August 2016)<\/strong> – This document has now been updated with details regarding the most recent ELK stack. I have recently done an install using the instructions of ElasticSearch 2.3.4, LogStash 2.3.4 & Kibana 4.5.3.<\/p>\n Software used:<\/p>\n Other basic setup (specific to the environment we setup):<\/p>\n Java is now installed and the variables required for the LogServer set.<\/p>\n ElasticSearch will now be running on the server successfully as a service.<\/p>\n (Unlike the previous version, Kibana 4.* no longer requires IIS to run and instead runs inside it’s own webserver, yay!)<\/p>\n The Kibana webserver will now be running and can be successfully accessed.<\/p>\n LogStash will now be running on the server successfully without a user needing to be logged in.<\/p>\n The server will now be sending logs to the Log Server<\/p>\n I tried really hard to get LogStash to run as a service…and I failed miserably. If someone knows how to get this working, please enlighten me as my batch file does work, but it’s not quite as clean and lovely as a service.<\/p>\n A really nifty command that we’ve found is that sometimes you may need to delete the logs you’ve collected – either because there’s too many, or you’ve changed your config and want to collect something else, or you were testing and want to get rid of the test logs you’d collected. In this case, the way we were deleting things was via PowerShell (all hail PowerShell!):<\/p>\n Invoke-WebRequest -Uri http:\/\/[FQDN of log server]:9200\/[name of log file folder] -Method DELETE<\/p>\n In order to not be absolutely FLOODED with events, we also modified the nxLog conf file to only collect what we wanted. You may want to tweak this yourself, depending on what you’re interested in collecting.<\/p>\n Changes we made to the nxlog.conf file:<\/p>\n This is giving us a few things:<\/p>\n Our default dashboard<\/a>\u00a0(I’ve removed any proprietary info from the image so it’s safe to view!) has also been customised a bit (thanks again to Ken! ^_^) to include some of the information most useful to us and to make it look nice and shiny to management. In particular:<\/p>\n So yes – that’s our log server. Very exciting. If there are any updates or tweaks, I’ll do an updated post.<\/p>\n UPDATE (23rd June 2014)\u00a0<\/strong>– I was requested to give information on our logstash.conf file as well as the dashboard we use.<\/p>\n The edit logstash.conf file:<\/p>\n I was also asked for a copy of our dashboard.json file, which I’ve uploaded here: Kibana Dashboard (default.json)<\/a>. Due to security restrictions on my blog, I’ve uploaded it as a .txt file. When you’ve downloaded it, just change the .txt to .json and away you go!<\/p>\n","protected":false},"excerpt":{"rendered":" So – I wanted to get Splunk but in my organisation that was never going to happen (you want something that costs MONEY?! Ludicrous!) So we had to come up with a compromise. A colleague of mine went hunting for some open source logging software and found that the combination of Elastic Search, LogStash, Kibana and nxLog worked well. He… Read more »<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[5,328,2,7],"tags":[163,164,167,165,125,166],"class_list":["post-438","post","type-post","status-publish","format-standard","hentry","category-sys-admin","category-techstuff","category-technology","category-work","tag-elastic-search","tag-kibana","tag-log","tag-logstash","tag-monitoring","tag-nxlog"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p2Tmk1-74","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/girl-germs.com\/index.php?rest_route=\/wp\/v2\/posts\/438","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/girl-germs.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/girl-germs.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/girl-germs.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/girl-germs.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=438"}],"version-history":[{"count":17,"href":"https:\/\/girl-germs.com\/index.php?rest_route=\/wp\/v2\/posts\/438\/revisions"}],"predecessor-version":[{"id":1571,"href":"https:\/\/girl-germs.com\/index.php?rest_route=\/wp\/v2\/posts\/438\/revisions\/1571"}],"wp:attachment":[{"href":"https:\/\/girl-germs.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=438"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/girl-germs.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=438"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/girl-germs.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=438"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n
\n
Log Server Installation Instructions\u00a0<\/strong><\/h2>\n
Folder creations<\/span><\/h3>\n
\n
\n
\n
\n
Create a Service Account<\/span><\/h3>\n
\n
\n
\n
Install Java JRE<\/span><\/h3>\n
\n
\n
\n
Install ElasticSearch<\/span><\/h3>\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
Install Kibana<\/span><\/h3>\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
Install Logstash<\/span><\/h3>\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
Client Server Installation Instructions<\/strong><\/h2>\n
nxLog<\/h3>\n
\n
\n
\n
\n
\n
\n
Other tips\/configuration<\/h2>\n
Query <QueryList>\\\r\n<Query Id=\"0\">\\\r\n<Select Path=\"Security\">*<\/Select>\\\r\n<Suppress Path=\"Security\">*[System[(EventID=4624 or EventID=4776 or EventID=4634 or EventID=4672 or EventID=4688)]]<\/Suppress>\\\r\n<Select Path=\"System\">*[System[(EventID=1074 or (EventID >= 6005 and EventID <= 6009) or EventID=6013)]]<\/Select>\\\r\n<Select Path=\"Microsoft-Windows-TerminalServices-LocalSessionManager\/Operational\">*<\/Select>\\\r\n<\/Query>\\\r\n<\/QueryList>\r\n<\/pre>\n
\n
\n
input {\r\n\t# Accept messages in on tcp\/3515\r\n\t# Incoming messages will be in json format, one per line\r\n\t# Tag these messages as windows and eventlog so we can filter on them later on\r\n tcp {\r\n port => 3515\r\n\tcodec => json_lines\r\n\ttags => [\"windows\",\"eventlog\"]\r\n }\r\n}\r\n\r\nfilter {\r\n # If it is an eventlog message, change some fields to lower case, and rename some fields so they match logstash's default\r\n\tif \"eventlog\" in [tags] {\r\n mutate {\r\n lowercase => [ \"EventType\", \"FileName\", \"Hostname\", \"Severity\", \"host\" ]\r\n rename => [ \"Hostname\", \"host\" ]\r\n rename => [ \"Message\", \"message\" ]\r\n }\r\n }\r\n}\r\n\r\noutput { \r\n\t# Send all the output to the elasticsearch cluster listed below\r\n\telasticsearch { \r\n\t\thost => localhost\r\n\t\tcluster => \"YourClusterName\"\r\n\t} \r\n}\r\n<\/pre>\n
![default dashboard<\/a>\u00a0(I’ve removed any proprietary info from the image so it’s safe to view!) has also been customised a bit (thanks again to Ken! ^_^) to include some of the information most useful to us and to make it look nice and shiny to management. In particular:<\/p>\n\nPie chart breaking down Event ID’s<\/li>\nBar chart showing our most active DC’s<\/li>\nPie chart of the accounts that are being locked out the most – this, for me right now, is one of the more interesting charts…<\/li>\nStandard bar chart, showing logs over time<\/li>\nA sorted column list displaying all events but with limited columns, in particular: EventTime, EventID, SourceName, message, SubjectUserName, TargetUserName – this may not include every bit of information we need for certain events, but it fits for most events.<\/li>\n<\/ul>\nSo yes – that’s our log server. Very exciting. If there are any updates or tweaks, I’ll do an updated post.<\/p>\nUPDATE (23rd June 2014)\u00a0<\/strong>– I was requested to give information on our logstash.conf file as well as the dashboard we use.<\/p>\nThe edit logstash.conf file:<\/p>\ninput {\r\n\t# Accept messages in on tcp\/3515\r\n\t# Incoming messages will be in json format, one per line\r\n\t# Tag these messages as windows and eventlog so we can filter on them later on\r\n tcp {\r\n port => 3515\r\n\tcodec => json_lines\r\n\ttags => [\"windows\",\"eventlog\"]\r\n }\r\n}\r\n\r\nfilter {\r\n # If it is an eventlog message, change some fields to lower case, and rename some fields so they match logstash's default\r\n\tif \"eventlog\" in [tags] {\r\n mutate {\r\n lowercase => [ \"EventType\", \"FileName\", \"Hostname\", \"Severity\", \"host\" ]\r\n rename => [ \"Hostname\", \"host\" ]\r\n rename => [ \"Message\", \"message\" ]\r\n }\r\n }\r\n}\r\n\r\noutput { \r\n\t# Send all the output to the elasticsearch cluster listed below\r\n\telasticsearch { \r\n\t\thost => localhost\r\n\t\tcluster => \"YourClusterName\"\r\n\t} \r\n}\r\n<\/pre>\nI was also asked for a copy of our dashboard.json file, which I’ve uploaded here: Kibana Dashboard (default.json)<\/a>. Due to security restrictions on my blog, I’ve uploaded it as a .txt file. When you’ve downloaded it, just change the .txt to .json and away you go!<\/p>\n","protected":false},"excerpt":{"rendered":"So – I wanted to get Splunk but in my organisation that was never going to happen (you want something that costs MONEY?! Ludicrous!) So we had to come up with a compromise. A colleague of mine went hunting for some open source logging software and found that the combination of Elastic Search, LogStash, Kibana and nxLog worked well. He… Read more »<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[5,328,2,7],"tags":[163,164,167,165,125,166],"class_list":["post-438","post","type-post","status-publish","format-standard","hentry","category-sys-admin","category-techstuff","category-technology","category-work","tag-elastic-search","tag-kibana","tag-log","tag-logstash","tag-monitoring","tag-nxlog"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p2Tmk1-74","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/girl-germs.com\/index.php?rest_route=\/wp\/v2\/posts\/438","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/girl-germs.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/girl-germs.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/girl-germs.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/girl-germs.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=438"}],"version-history":[{"count":17,"href":"https:\/\/girl-germs.com\/index.php?rest_route=\/wp\/v2\/posts\/438\/revisions"}],"predecessor-version":[{"id":1571,"href":"https:\/\/girl-germs.com\/index.php?rest_route=\/wp\/v2\/posts\/438\/revisions\/1571"}],"wp:attachment":[{"href":"https:\/\/girl-germs.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=438"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/girl-germs.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=438"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/girl-germs.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=438"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}](\"http:\/\/i.imgur.com\/uTsyWN0.jpg\"){kind=link}