Kibana/ElasticSearch Logging
A listener that writes the request and response logs to the ELK Stack .
For efficiency, it batches multiple logs together.
A helper script is provided at scripts/kibana_index_mapping.sh
for setting up the Index Mappings in ElasticSearch.
// Domain Object
{
domain - name : "example.com"
listeners : [
{ id : log_kibana , builder : KibanaBuilder }
]
log : {
// A list of request headers to capture in the logging output
request - headers : [
"user-agent" ,
"client-address"
]
// A list of response headers to capture in the logging output
response - headers : [
"content-type"
]
}
listener - config : {
// id used in the listener list
log_kibana : {
// the host (and port) of the ElasticSearch service
host : "localhost:9200"
// The log forwarder sends the logs to the index named "-yyyy.MM.dd",
// mimicking the behavior of LogStash
index - prefix : "logstash"
// The "type" to use when writing to ElasticSearch
type : "access_log"
// To bleed off pressure if ElasticSearch is backed up, this limits the number of outstanding
// requests to ElasticSearch. If there are more than `max-outstanding` requests to ElasticSearch,
// future requests are dropped until one of the in flight requests completes.
max - outstanding : 10
// The logger will wait until there are `buffer-size` entries ready to go, or will
// flush every 100 ms to ElasticSearch
buffer - size : 100
// Optional: Signs the request before writing to the AWS ElasticSearch Service
AWSSigning : {
// Whether or not to sign the request
active : true
// Used in AWS's signing algorithm. Should pretty much always be "es"
service : "es"
// The region that the ElasticSearch service is running in
region : "us-west-2"
}
}
}
}
Json Log
{
"responding_host" : "http://thor:9001" ,
"method" : "GET" ,
"@timestamp" : "2016-09-05T20:57:27.214-05:00" ,
"cache_status" : "nocache" ,
"middleware_time" : {},
"path" : "http://localhost:9001/healthcheck" ,
"response_headers" : { "content-type" : "application/json; charset=UTF-8" },
"response_size" : 387 ,
"responding_service" : "shield" ,
"request_headers" : { "client-address" : "0:0:0:0:0:0:0:1" , "User-Agent" : "curl/7.47.1" },
"response_status" : 200 ,
"shield_host" : "http://thor:9001" ,
"template" : "/healthcheck" ,
"overall_time" : 24
}
Authentication
If the AWSSigning
configuration is present and active, Shield uses the AWS Java SDK for signing the request to
ElasticSearch. See the documentation on how to set up Shield with
AWS Credentials .