Tumlare Software Services Pvt. Ltd. (TSS) developed an in-house solution by combining three different technologies: Elasticsearch, Logstash server, and Kibana / ELK stack, to ingest raw legacy data into a search engine, making data easily accessible and consumable via APIs (Application Program Interfaces) in JSON file structure, and digitally transformed to integrate with any other platform that wants to use operation data.
Elasticsearch is a distributed, free and open search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured. Elasticsearch is built on Apache Lucene and was first released in 2010 by Elasticsearch N.V. (now known as Elastic). Known for its simple REST APIs, distributed nature, speed, and scalability, Elasticsearch is the central component of the Elastic Stack, a set of free and open tools for data ingestion, enrichment, storage, analysis, and visualization. Commonly referred to as the ELK Stack (after Elasticsearch, Logstash, and Kibana), the Elastic Stack now includes a rich collection of lightweight shipping agents known as Beats for sending data to Elasticsearch.
Raw data flows into Elasticsearch from a variety of sources, including legacy applications ,logs, system metrics, and web applications. Data ingestion is the process by which this raw data is parsed, normalized, and enriched before it is indexed in Elasticsearch. Once indexed in Elasticsearch, users can run complex queries against their data and use aggregations to retrieve complex summaries of their data. From Kibana, users can create powerful visualizations of their data, share dashboards, and manage the Elastic Stack.
The speed and scalability of Elasticsearch and its ability to index many types of content mean that it can be used for a number of use cases:
• Application search
• Website search
• Enterprise search
• Logging and log analytics
• Infrastructure metrics and container monitoring
• Application performance monitoring
• Geospatial data analysis and visualization
• Security analytics
• Business analytics
Logstash, one of the core products of the Elastic Stack, is used to aggregate and process data and send it to Elasticsearch. Logstash is an open source, server-side data processing pipeline that enables you to ingest data from multiple sources simultaneously and enrich and transform it before it is indexed into Elasticsearch.
Kibana is a data visualization and management tool for Elasticsearch that provides real-time histograms, line graphs, pie charts, and maps. Kibana also includes advanced applications such as Canvas, which allows users to create custom dynamic infographics based on their data, and Elastic Maps for visualizing geospatial data.
Elastic recognized as a Challenger in the 2021 Gartner Magic Quadrant for Insight Engines
https://www.elastic.co/blog/elastic-recognized-as-a-challenger-in-the-2021-gartner-magic-quadrant-for-insight-engines.
Amit Mahajan
Project Manager