Training Example: Fluentd – Review the Data, Give Your Score & Compare to the Real AI Evaluation

[H4] Unified Logging Layer
Fluentd decouples data sources from backend systems by providing a unified logging layer in between.

[H4] Simple yet Flexible
Fluentd's 500+ plugins connect it to many data sources and outputs while keeping its core simple.

[H4] Proven
5,000+ data-driven companies rely on Fluentd. Its largest user currently collects logs from 50,000+ servers.

[IMG: line]

[IMG: pplive]

[IMG: amazon webservices]

[IMG: microsoft]

[IMG: atlassian]

[IMG: nintendo]

[IMG: twilio]

[IMG: gree]

[IMG: toastexchange]

[IMG: gungho]

[IMG: namcobandai studios]

[IMG: wish]

[IMG: ukengames]

We are a Cloud Native Computing Foundation (CNCF) member project.

[IMG: Cloud Native Computing Foundation]

[H2] Quotes

[IMG: Matz]

"Fluentd proves you can achieve programmer happiness and performance at the same time. A great example of Ruby beyond the Web."
Yukihiro Matsumoto (Matz), creator of Ruby

[IMG: Adam Wiggins]

"Logs are streams, not files. I love that Fluentd puts this concept front-and-center, with a developer-friendly approach for distributed systems logging."
Adam Wiggins, Heroku co-founder

[H2] Featured Video
RubyConf 2014: "Build the Unified Logging Layer with Fluentd"

Want to learn more about Fluentd?

[IMG: Before Fluentd]

[H3] Before Fluentd

[IMG: After Fluentd]

[H3] After Fluentd

[H3] Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data.
Here're the key features.

[IMG: Unified Logging with JSON]

[H2] Unified Logging with JSON
Fluentd tries to structure data as JSON as much as possible: this allows Fluentd to unify all facets of processing log data: collecting, filtering, buffering, and outputting logs across multiple sources and destinations (Unified Logging Layer). The downstream data processing is much easier with JSON, since it has enough structure to be accessible while retaining flexible schemas.

[IMG: Pluggable Architecture]

[H2] Pluggable Architecture
Fluentd has a flexible plugin system that allows the community to extend its functionality. Our 500+ community-contributed plugins connect dozens of data sources and data outputs. By leveraging the plugins, you can start making better use of your logs right away.

[IMG: Minimum Resources Required]

[H2] Minimum Resources Required
Fluentd is written in a combination of C language and Ruby, and requires very little system resource. The vanilla instance runs on 30-40MB of memory and can process 13,000 events/second/core. If you have tighter memory requirements (-450kb), check out Fluent Bit, the lightweight forwarder for Fluentd.

[IMG: Built-in Reliability]

[H2] Built-in Reliability
Fluentd supports memory- and file-based buffering to prevent inter-node data loss. Fluentd also supports robust failover and can be set up for high availability. 2,000+ data-driven companies rely on Fluentd to differentiate their products and services through a better use and understanding of their log data.

[H3] Fluentd History
Fluentd was conceived by Sadayuki "Sada" Furuhashi in 2011. Sada is a co-founder of Treasure Data, Inc., a primary sponsor of the Fluentd project. Since being open-sourced in October 2011, the Fluentd project has grown dramatically: dozens of contributors, hundreds of community-contributed plugins, thousands of users, and trillions of events collected, filtered and stored. Fluentd was accepted to Cloud Native Computing Foundation on November 8, 2016 and is at the Graduated project maturity level on 2019. And its development is maintained with the community.

Learn why Fluentd is for you

[H2] Learn
Want to learn the basics of Fluentd? Check out these pages.
What is Fluentd?
Why use Fluentd?
Who is using Fluentd?
What is Unified Logging Layer?
FAQs
[H2] Ask the Community
Couldn't find enough information? Let's ask the community!
Forum
Google Group
News Letter
Stack Overflow
Issue Tracker
Twitter
Facebook
Source Code
[H2] Ask the Experts
You need commercial-grade support from Fluentd committers and experts?
Enterprise Services
Ecosystem Survey/Interview

Let's share your Fluentd use-case/testimonials in this page!
If your service/organization is not listed here, please feedback via
Creating GitHub Issue!

[IMG: Atlassian]

"At Atlassian, we've been impressed by Fluentd and have chosen to use it in Atlassian Cloud's logging and analytics pipeline."

[IMG: PITS Global Data Recovery Services]

PITS Global Data Recovery Services is a leading data recovery center in the United States specializing in recovering data, including lost or damaged files, from hard drives, SSDs, RAID arrays, flash drives, and other storage media. Fluentd is used to handle highly-loaded internal infrastructure and provide reliable, fully-managed business solutions.

Download Fluentd Now

[IMG: Amazon Web Services]

The architecture of Fluentd (Sponsored by Treasure Data) is very similar to Apache Flume or Facebook’s Scribe. Fluentd is easier to install and maintain and has better documentation and support than Flume and Scribe.

[IMG: Backplane]

When I first started at Backplane,
I wanted to get us to start logging everything and aggregating it in JSON,
and I wanted that system up and running fast. Fluentd got me there in less than a day.
I was skeptical of reliability at first, but it takes so little time to setup, there's no reason not to try it.
It hasn't crashed once yet, works exactly as I need it to thanks to the huge plugin library,
and likely saved us a ton of time. As an added bonus: the user list for Fluentd was also way more responsive
when I ran into initial setup questions than those of any competitors.
If you want to start logging your data fast, Fluentd is a great way to go.

[IMG: Change.org]

Change.org has been able to implement multi-source, multi-destination event streaming quickly and easily using fluentd. We're very appreciative.

[IMG: CyberAgent]

CyberAgent (TYO:4751) is a Tokyo based company that specializes in internet business. We are expanding our businesses in the field of Social Network Services and Web advertising.
Though collecting and analyzing log data is quite important in our services, we had no choice but to conduct in conventional methods such as using Hadoop to analyze log data or rsync and script.
That is why we began to use Fluentd. Fluentd made it possible to collect existing log with significantly lower resources, while it is not affecting the analyzing process. We introduced this system successfully and safely.
Fluentd helps us collect and analyze an app log and midleware log efficiently. And as a result, it improves our daily operation and services.

[IMG: DeNA]

DeNA (pronounced “D-N-A”) is a global Internet company (TSE:2432)
focused on social games and e-commerce.
DeNA operates the Mobage platform, which is one of the most successful gaming service in Japan with more than49 million registered players as of March, 2013.
We use Fluentd for log monitoring, visualization of metrics, and realtime log collection.
It can deal with the log of our large-scaled servers. We are very happy with its reliability, and flexibility provided by its plugin mechanism.

[IMG: Drecom]

Drecom (TYO:3793), based in Tokyo, is planning and developing many types of services which are focusing on word "communication" such as: mobile contents, internet advertising, etc. In our business, Log collection and analysis is essential in order to continue to provide services of the highest satisfaction to the user.
Fluentd is easy to install, reliable, and provides support with out interfering with our regular tasks. If anything, we are paying attention to not forget the presence of the silent daemon. It can not only provide many plugins, but also have and can develop add plugins with necessary functions. We are pleased to fluent work with the Fluentd daemon.

[IMG: Elastisys]

Elastisys, founded in 2011 and based in Sweden, specializes in cloud native technologies. We offer a cloud native Kubernetes platform, optimized for security and compliance with EU regulations for software critical to our society. Our solutions enable customers to accelerate innovation, reduce costs, and enhance security using our leading cloud native technology and expertise.
We utilize Fluentd as part of our offering to provide our customers with out of the box logging for Kubernetes environments.

[IMG: GREE]

Data Science is at the core of GREE (TYO:3632)'s business, and it starts with logging user events and Advertisement related data. We currently deploy Fluentd for a number of our games and ad services to federate user event data, and use that data to inform our product decisions.
So far, we've been pleased with Fluentd's robustness and performance: it has held up strongly against our massive data, allowing us to focus on data analytics instead of logging.

[IMG: GungHo]

We use fluentd to tally action logs and monitor action logs for our hit puzzle games. We're extremely satisfied, as fluentd delivers a simple architecture with high availability.

[IMG: Knowlarity]

At Knowlarity Communications, we process anywhere between 2 ~ 10 Million calls on a daily basis. This needs us to be Fault tolerant & Resilient. We use Fluentd to collect Call Detail Records (CDRs) from various machines and feed it to our analytics tools & alerting systems in real time. Fluentd was very easy to setup & use. It also provides a variety of plugins which lets you get started quickly. Additionally It has also proved to be very reliable and scalable.

[IMG: LINE]

LINE Corp., based in Tokyo, is web services company. Log collection and analysis is one the most important missions in our day-to-day operations. As such we have spent quite a long time to search for the right tool that can process and collect data from various sources with great flexibility and high throughput -- and we came up with Fluentd.
We utilize Fluentd to collect a very large amount of logs. The logs are written into Hadoop HDFS clusters, and are also used to analyze various service statuses in realtime. We also use many plugins from rubygems.org to further enhance this mechanism. - Satoshi Tagomori

[IMG: Livesense]

Since we adopted Fluentd, our services have acquired to integrate collecting various structured events and log data. It is not only for collecting logs, it could preprocess logs with many swiss-army knife plugins to make it easy to analyze them. Also we could easily to create input/output plugins as needed.
Fluentd has been worked in our business to help day-by-day data driven development for our Growth Hacker. Therefore, I felt Fluentd is revolutionary log collection tool I have ever seen.

[IMG: M3]

M3 is a Tokyo-based, global company whose mission is to help more people around the world live healthier, happier lives by reducing the cost of health care in the world using technology and the Internet. M3, together with its subsidiaries, operate online physician sites in Japan, China, the UK, USA, Europe, and South Korea, comprised of more than 3.5 million doctors worldwide.
We have been using Fluentd to support monitoring and realtime log analysis since 2012. Many thanks to Fluentd and its ecosystem!

[IMG: Microsoft]

Operations Management Suite uses Fluentd: "The core of the agent uses an existing open source data aggregator called Fluentd. Fluentd has hundreds of existing plugins, which will make it really easy for you to add new data sources."

[IMG: NAMCO BANDAI Studios]

NAMCO BANDAI Studios Inc. is a platform-agnostic development studio that mainly develops console games, arcade games, online games, and mobile games.
Besides games, NBSI also produces music, movie, and other entertainment content.
Log file analysis is key to examining user trends for our game services. We keep numerous logs such as access logs, error logs, and action logs. In order to efficiently analyze the information, we needed a solution flexible enough to aggregate the data from all the various logs.
The Fluentd concept fits our needs perfectly. For a large-scale service with hundreds of thousands of users, Fluentd's scalability has proven to be valuable.

[IMG: Nintendo]

We use Fluentd to collect massive data logs for our platforms. Having developed a system based on Fluentd, we are now effectively monitoring and analyzing our services in real-time. We are very much satisfied with its flexibility, especially how easy it is to use in tandem with other systems.

[IMG: PPLive]

PPTV.com is one of the largest Chinese online TV provider, which has 35 million active users. PPTV clients can be installed on multi-platforms (Windows, Mac, Android, iOS, WP, etc).
We (PPTV site operation team) tried to find a solution to do realtime log-collection and analyse based on the massive logfiles. Finally, we found that Fluentd is the perfect solution to do real-time log collection. Log can be JSON-structured to MongoDB cluster.
Realtime monitoring/reporting/charting can be generated from MongoDB data, which is helpful to our operation work. Fluentd is a great tool!

[IMG: SlideShare]

SlideShare uses Fluentd to stream logs and events in the cloud. We were looking for a versatile yet lightweight logger in Ruby, and Fluentd fit the bill perfectly.

[IMG: SMARTSTUDY]

SMARTSTUDY creates educational mobile apps for kids. We use Fluentd to bridge our Docker instances and Elasticsearch for log analysis. We love Fluentd!

[IMG: Sophotech]

At Sophotech we rely on Fluentd to streamline log collection across our systems, ensuring consistent, reliable observability at scale. Its flexibility and ease of integration allow us to simplify operations and focus on delivering value, while reducing the overhead traditionally associated with managing logs.

[IMG: TOAST eXchange]

TOAST eXchange, an advertisement platform service made by NHN Entertainment, requires ingesting tremendous volume of data to target customers based on their interest fast and accurately. Fluentd provides flexible configuration and easy installation so that we could adopt it with low cost and resources and could focus our core competency on business layer. Fluentd is really helpful to construct expandable streaming data pipelines for large scale event handling.

[IMG: Twilio]

Twilio, the leading cloud communications platform, provides building blocks to add messaging, voice, and video in your web and mobile applications. At Twilio, we use Fluentd to forward billions of logs messages per day from our instances in Amazon AWS to Google's BigQuery service.
Having operated Fluentd on thousands of production instances for the past several months, we are very happy with Fluentd's reliability, flexibility, and performance. It has proven to be one of the more dependable parts of our log aggregation solution.

[IMG: Uken Games]

Uken Games is a Canadian cross-platform gaming company. Our users generate a lot of logs, and Fluentd made it really easy for us to manage them. It also allowed us to painlessly try and compare new solutions for log analysis, simply by modifying configs and zero-downtime restarting the client. Deploying Fluentd was a breeze with Chef.

[IMG: Viki]

At Viki, we use Fluentd deployed on Heroku to collect analytics events from various apps which then go into Treasure Data and in-house Hadoop Cluster for analysis. Fluentd integrates seamlessly with Heroku, and we've been impressed with its versatility and reliability.

[IMG: Wish]

At wish.com, we started using Fluentd from early on to
collect user event data. Its reliability and ease of use from application code (Python for us) make it
easy for us to keep track of the key metrics.

Download Fluentd Now

[H2] Learn
Want to learn the basics of Fluentd? Check out these pages.
What is Fluentd?
Why use Fluentd?
Who is using Fluentd?
What is Unified Logging Layer?
FAQs
[H2] Ask the Community
Couldn't find enough information? Let's ask the community!
Forum
Google Group
News Letter
Stack Overflow
Issue Tracker
Twitter
Facebook
Source Code
[H2] Ask the Experts
You need commercial-grade support from Fluentd committers and experts?
Enterprise Services
Ecosystem Survey/Interview

[H4] Application Logs

[IMG: Java]

[H5] Java

[IMG: Ruby]

[H5] Ruby

[IMG: Ruby on Rails]

[H5] Ruby on Rails

[IMG: Python]

[H5] Python

[IMG: PHP]

[H5] PHP

[IMG: Perl]

[H5] Perl

[IMG: Node.js]

[H5] Node.js

[IMG: Scala]

[H5] Scala

[IMG: .NET]

[H5] .NET

[H4] Network Protocols

[IMG: Syslog]

[H5] Syslog

[IMG: HTTP]

[H5] HTTP

[IMG: TCP]

[H5] TCP

[IMG: UDP]

[H5] UDP

[IMG: Unix Domain Socket]

[H5] Unix Domain Socket

[IMG: SNMP]

[H5] SNMP

[IMG: MQTT]

[H5] MQTT

[H4] IoT Devices

[IMG: RasperryPi]

[H5] RasperryPi

[H4] Others

[IMG: Docker]

[H5] Docker

[IMG: Twitter]

[H5] Twitter

[IMG: MySQL Slow Query Log]

[H5] MySQL Slow Query Log

[IMG: PostgreSQL Slow Query Log]

[H5] PostgreSQL Slow Query Log

[IMG: Gluster FS]

[H5] Gluster FS

[IMG: Kafka]

[H5] Kafka

[IMG: Kestrel]

[H5] Kestrel

[IMG: Munin]

[H5] Munin

[IMG: DStat]

[H5] DStat

[IMG: Windows Event Log]

[H5] Windows Event Log

[IMG: Netflow]

[H5] Netflow

[IMG: Postfix]

[H5] Postfix

[IMG: AWS Cloud Watch]

[H5] AWS Cloud Watch

[IMG: AWS ELB Log]

[H5] AWS ELB Log

[IMG: AWS SQS]

[H5] AWS SQS

[IMG: OpenTelemetry]

[H5] OpenTelemetry

Check List of All Plugins

[H2] Learn
Want to learn the basics of Fluentd? Check out these pages.
What is Fluentd?
Why use Fluentd?
Who is using Fluentd?
What is Unified Logging Layer?
FAQs
[H2] Ask the Community
Couldn't find enough information? Let's ask the community!
Forum
Google Group
News Letter
Stack Overflow
Issue Tracker
Twitter
Facebook
Source Code
[H2] Ask the Experts
You need commercial-grade support from Fluentd committers and experts?
Enterprise Services
Ecosystem Survey/Interview