How to send logs from a file to Graylog?

Igor Teleychuk

• 4 min read
How to send logs from a file to Graylog?

When I first installed Graylog Server on my Linux, the question almost immediately arose, what should I do next? Graylog has a whole bunch of settings, sections, parameters. However, it is not specified anywhere exactly how to move the logs from the file to Graylog.

One of the parts of my job is to organize the collection of logs of various applications. These are mostly Java applications deployed on Linux (CentOS) servers.

And here's a simple question, a Java application saves logs to a file, how do I send them from the CentOS server to Graylog? There are two main options for sending logs from the server, but first we need to create an Input in the Graylog itself. We need to set up the log entry point in the service, and let's get on with it.

Select the System tab > Input

Next, Select input > Syslog UDP and Launch new input

The main fields here are as follows:

Node - we leave the default, in this case, one node, of the Greylog server itself.
Title - we specify the name, it will participate in the extended summary of logs, so we specify it wisely.
Port - which port will listen to the log stream on.
Bind address - optional, but you can specify which address to listen from.
You can leave the rest by default and click Launch Input at the bottom.

There are two main options for sending logs from a file to Graylog, which I use in my work.:

rsyslog

sudo yum info rsyslog
sudo yum install rsyslog
sudo systemctl status rsyslog

The basic syslog configuration is located in the /etc/rsyslog.conf file by default, and additional configuration can be created in the /etc/rsyslog.d folder, for example:

  sudo nano /etc/rsyslog.d/graylog.conf

rsyslog has a rich system of settings for tracking, interacting, sending logs, and more, but we are interested in simply transferring logs from a file to Graylog, so we can use the following template:

$ModLoad imfile
$InputFileName /<path>/<to>/<logs>/your_app.log
$InputFileTag your-app-general
$InputFileStateFile stat-your-app-logs
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor
local7.* @<Graylog_server_IP>:<port>;RSYSLOG_SyslogProtocol23Format

There are many parameters here, but the main ones are these:

  • $InputFileName is the path to the file;
  • $InputFileTag is the name of the marker, which will become the application_name parameter in Greylog;
  • $InputFileStateFile - must also have a unique name;
  • local7.* @Graylog_server_IP:port;RSYSLOG_SyslogProtocol23Format - specify the IP:Port of the Graylog server.

Next, you need to restart the rsyslog service and it will immediately start sending logs from the file to the specified Graylog port.

In addition, if you need Linux server system logs, you can send them in just one line.:

*.*@Graylog_server_IP:port;RSYSLOG_SyslogProtocol23Format

filebeat

Is a lightweight open-source data shipper for forwarding and centralizing log data. It is part of the Elastic Stack and is designed to monitor log files on a server, read those logs, and send the collected data to Elasticsearch or Logstash for further processing and analysis.

By efficiently handling log files and managing their data flow, Filebeat helps in extracting valuable insights and ensuring that log data is readily available for search and visualization. Its ability to parse and structure logs makes it a vital tool for any logging and monitoring solution.

I got to know him quickly enough when sending logs using rsyslog to Graylog, we have one problem. This is a very simple and non-configurable system. For example, a Java application can generate logs in a non-standard format, such as errors.

A stack trace is a set of individual lines of information about an error and its location. Syslog sends each line as a separate message, and this interferes with the normal reading of logs.

Filebeam solves this problem because it supports multi-line log sending to Graylog. But we'll talk about this in a separate article. Let's look at how to install Filebeat on CentOS. It is not always possible to install directly through the package installer, so it is better to download the desired version of the application from the official website.

Install and go to the configuration file:

sudo sudo rpm -ivh /path_to_package/filebeat-8.16.0-x86_64.rpm
sudo nano /etc/filebeat/filebeat.yml

For the basic sending of logs from a file to Graylog, only two parameters need to be changed.:

filebeat.inputs:
  paths:
    - /path_to_log/your_app.log
output.logstash:
  hosts: ["Graylog_server_IP:port"]

The first parameter is the path to the log file, and the second is the IP address and port of Graylog. We specify the port that we set in the input settings at the very beginning of the article.

And run the file bat with the commands:

sudo chown root:root /etc/filebeat/filebeat.yml
sudo chmod go-w /etc/filebeat/filebeat.yml
sudo systemctl start filebeat.service
sudo systemctl enable filebeat.service
sudo systemctl status filebeat.service

For Filebeat, it is important that the settings file belongs to root and has certain read and write settings. As a result, we launch the service, turn on autorun and check the operation status.

That's all. In the following articles, we'll look at configuring filebeat, the Graylog server, and other parts of the system.

Recommended articles: