Recently, I had to track down some bugs which were happening in the current system that I am working on. The problem was, the issue was occurring enough to raise the flags, but could not reproduce very easily. The system I am currently working on it has dozens of HTTP application, and it is not easy to track down the exact origin of the problem. We are stashing our logs in elastic search, and over kibana it was visible that users were receiving HTTP 500 – Internal Server Error. After getting exhausted, I want it to tap into the HTTP communication and get more detail, so at least, I can identify the issue.

Nginx is in front of all services, so it would be a relatively easy trick. But that meant logging request and response body of every request, which are unnecessary, so the natural question came into mind was “Can I log request and response body of failures only?”

I have asked my colleague, who has immediately said “why not”, after some reading and googling, we both have accepted yes it is doable. Lucky me, he actually did stick it to the job until he completed, while me, well that is irrelevant…

To log the request and response body, we needed ngx_lua module, which you can find detailed information about it here. So the task was to, assign request and response body to nginx variables, in a case of failures. As we already had an overview incoming request and HTTP statutes, we took a lazy approach and filtered anything that is not a healthy response, (e.g. 200, 201 or 302). This allowed us to see anything that is smelling fishy. So the end configuration did look like this for server block of nginx.conf :

If HTTP status codes were to be 200, 201 or 302 nginx variables error_request_body and error_response_body would stay empty therefore there would be no log for health responses from the HTTP applications. As you can see location block contains the log format which is to be specified as follow:

From this point on, remaining work was to change to grok pattern, restart logstash and so on, so logs can be parsed and persisted in elastic search.

While this is might be a good approach to figuring out some problems, I would suggest that it should be used wisely, so you do not start logging any sensitive user information.