Simplifying IT, Empowering Businesses

If you’ve been following us, you know that we consistently post well-researched technical articles in a relatively easier language for all aspiring developers to understand. Today, as well, we have taken up a very interesting topic (like always) and something that we have been asked a lot of times through various social media channels.

In this post, we will explore the simplest ways to upload a CSV file via Logstash. However, before that, let’s quickly look at the basics and understand what Logstash is. 

What Is Logstash?

In simple words, Logstash is a free and open-source lightweight server-side data processor that allows us to get data from any source, transform or modify it and send it to a desired location. It’s most commonly used in Elasticsearch.

The Architecture of Logstash

How Logstash Works?

Install Logstash on windows

To install Logstash on Windows, there are some prerequisites involved:

• ElasticSearch must be installed
• JDK 8 and JVM is installed
• Power shell

You also need to download the Logstash zip package for windows. You can do it from here: https://www.elastic.co/downloads/logstash

Install the Logstash. With Logstash, it is compulsory to give a meaningful name to your project. Give it and save it to the bin folder, where the Logstash is installed. For instance, “logstash.config“

Now, Logstash is installed on your windows. We can go ahead and take one CSV file and import it to elastic search via Logstash.

To do this, we first need to create the Logstash elasticsearch file, which we need to upload. Remember we already assigned a name to the file called csv.config?

Here is the config file:

input {
  file {
    path => "/home/ubuntu/Documents/esearch/test.csv"
    sincedb_path => "NULL"   --Windows     "/dev/null" --LINUX    
     start_position => "beginning"
  }
}

filter {
      csv {
	separator => ","
        columns => [ "id", "FirstName", "LastName", "DOB"]
remove_field => ["@timestamp","@version","message","path","host"]
     }
    }

output {
  elasticsearch { 
  hosts => ["http://localhost:9200"] 
  index => "person"  
  }
}

Note: the index name is always written in small letters to avoid an error while uploading.

To upload a CSV file using the Logstash config while we need to run this command on our cmd prompt.

bin/logstash -f config/csv.conf

This command will upload the CSV file in the personal index on elastic search. To check whether the data is uploaded or not, we need to check it on a browser using the below url. You can use any browser of your choice.

http://localhost:9200/person/_search?pretty

The final result will be in json format in the following way:

"_index" : "person",
        "_type" : "_doc",
        "_id" : "dmx9emwB7Q7sfK_2g0Zo",
        "_score" : 1.0,
        "_source" : {
          "id" : "1",
          "FirstName" : "Jose",
          "LastName" : "KOSTE",
          "host" : "localhost",
          "message" : "72552,0,297,9317",
          "@version" : "1",
          "@timestamp" : "2019-08-10T07:45:41.642Z",
          "DOB" : "11061988",
          "path" : "/home/ubuntu/Documents/esearch/test.csv"
        }

Wrapping Up

So, this is how you import a CSV via Logstash to elastic search. Like always, we believe this tutorial was simple and useful. Follow the exact commands and steps and you’ll be able to accomplish the task.