Training Example: Apache Flink – Review the Data, Give Your Score & Compare to the Real AI Evaluation

Flink Capabilities

Correctness guarantees
Exactly-once state consistency
Event-time processing
Sophisticated late data handling

Learn More

Layered APIs
SQL on Stream & Batch Data
DataStream API
ProcessFunction (Time & State)

Learn More

Operational focus
Flexible deployment
High-availability setup
Savepoints

Learn More

Scalability
Scale-out architecture
Support for very large state
Incremental Checkpoints

Learn More

Performance
Low latency
High throughput
In-Memory computing

Learn More

[H3]
Apache Flink Kubernetes Operator 1.15.0 Release Announcement
May 26, 2026 -
Gyula Fora
The Apache Flink community is excited to announce the release of Flink Kubernetes Operator 1.15.0!
This release brings Kubernetes-native Conditions to FlinkDeployment, Logback logging support, bundled metric reporters, Flink 2.2 compatibility, and a number of important reliability fixes across session jobs, savepoints, and the mutating webhook.
We encourage you to download the release and share your experience with the community through the Flink mailing lists or JIRA! We’re looking forward to your feedback!
...

Continue reading »

[H3]
Apache Flink 2.2.1 Release Announcement
May 15, 2026 -
Sergey Nuyanzin
The Apache Flink Community is pleased to announce the first bug fix release of the Flink 2.2 series.
This release includes 44 bug fixes, vulnerability fixes, and minor improvements for Flink 2.2. Below you will find a list of all bug fixes and improvements (excluding improvements to the build infrastructure and build stability). For a complete list of all changes see: JIRA.
We highly recommend all users upgrade to Flink 2.
...

Continue reading »

[H3]
Apache Flink 2.0.2 Release Announcement
May 11, 2026 -
Sergey Nuyanzin
The Apache Flink Community is pleased to announce the second bug fix release of the Flink 2.0 series.
This release includes 34 bug fixes, vulnerability fixes, and minor improvements for Flink 2.0. Below you will find a list of all bug fixes and improvements (excluding improvements to the build infrastructure and build stability). For a complete list of all changes, please see: JIRA.
We highly recommend all users upgrade to Flink 2.
...

Continue reading »

[H3]
Apache Flink 2.1.2 Release Announcement
May 11, 2026 -
Sergey Nuyanzin
The Apache Flink Community is pleased to announce the second bug fix release of the Flink 2.1 series.
This release includes 45 bug fixes, vulnerability fixes, and minor improvements for Flink 2.1. Below you will find a list of all bug fixes and improvements (excluding improvements to the build infrastructure and build stability). For a complete list of all changes see: JIRA.
We highly recommend all users upgrade to Flink 2.
...

Continue reading »

[H3]
Apache Flink 1.20.4 Release Announcement
April 22, 2026 -
Sergey Nuyanzin
The Apache Flink Community is pleased to announce the fourth bug fix release of the Flink 1.20 series.
This release includes 41 bug fixes, vulnerability fixes, and minor improvements for Flink 1.20. Below you will find a list of all bug fixes and improvements (excluding improvements to the build infrastructure and build stability). For a complete list of all changes see: JIRA.
We highly recommend all users upgrade to Flink 1.
...

Continue reading »

[H3]
Flink Community update for April 2026
April 13, 2026 -
Julian Payne
David Radley
This is the monthly Flink Community update for April 2026. We scour the latest updates from the Flink community so you don’t have to!
...

Continue reading »

[H3]
Apache Flink CDC 3.6.0 Release Announcement
March 30, 2026 -
Yanquan Lv
The Apache Flink Community is excited to announce the release of Flink CDC 3.6.0! This release extends Flink version support to 1.20.x and 2.2.x, upgrades JDK version to 11, introduces new Oracle Source and Apache Hudi Sink Pipeline connectors, adds Lenient mode schema evolution support for Fluss Pipeline connector, introduces PostgreSQL Schema Evolution support, enhances Schema Evolution capabilities for better multi-table synchronization scenarios and table name mapping flexibility, and strengthens Transform framework with VARIANT type and JSON parsing support.
...

Continue reading »

[H3]
Apache Flink Agents 0.2.1 Release Announcement
March 26, 2026 -
Wenjin Xie
The Apache Flink Community is pleased to announce the first bug fix release of the Flink-Agents 0.2 series.
This release includes 3 bug fixes, vulnerability fixes, and minor improvements for Flink-Agents 0.2. Below you will find a list of all bug fixes and improvements (excluding improvements to the build infrastructure and build stability). For a complete list of all changes, please see: ISSUE.
We highly recommend all users upgrade to Flink-Agents 0.
...

Continue reading »

[H3]
Flink Community update for March 2026
March 1, 2026 -
David Radley
This is the monthly Flink Community update for March 2026. It covers the
main developments in the community over the past month, including technical updates,
user-facing updates, and governance and community.
...

Continue reading »

[H3]
Apache Flink Kubernetes Operator 1.14.0 Release Announcement
February 15, 2026 -
Sergio Chong Loo / Daniel Rossos
The Apache Flink community is excited to announce the release of Flink Kubernetes Operator 1.14.0!
This release introduces a major new capability for production deployments: native Blue/Green deployment support. Deploy stateful streaming applications without interruption with automated Blue/Green deployments directly in your Kubernetes clusters.
We encourage you to download the release and share your experience with the community through the Flink mailing lists or JIRA! We’re looking forward to your feedback!
...

Continue reading »

««

«

1

2

3

4

5

»

»»

[H1]
What is Apache Flink? — Architecture
#
Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale.
Here, we explain important aspects of Flink’s architecture.
[H2]
Process Unbounded and Bounded Data
#
Any kind of data is produced as a stream of events. Credit card transactions, sensor measurements, machine logs, or user interactions on a website or mobile application, all of these data are generated as a stream.
Data can be processed as unbounded or bounded streams.
Unbounded streams have a start but no defined end. They do not terminate and provide data as it is generated. Unbounded streams must be continuously processed, i.e., events must be promptly handled after they have been ingested. It is not possible to wait for all input data to arrive because the input is unbounded and will not be complete at any point in time. Processing unbounded data often requires that events are ingested in a specific order, such as the order in which events occurred, to be able to reason about result completeness.
Bounded streams have a defined start and end. Bounded streams can be processed by ingesting all data before performing any computations. Ordered ingestion is not required to process bounded streams because a bounded data set can always be sorted. Processing of bounded streams is also known as batch processing.
Apache Flink excels at processing unbounded and bounded data sets. Precise control of time and state enable Flink’s runtime to run any kind of application on unbounded streams. Bounded streams are internally processed by algorithms and data structures that are specifically designed for fixed sized data sets, yielding excellent performance.
Convince yourself by exploring the use cases that have been built on top of Flink.
[H2]
Deploy Applications Anywhere
#
Apache Flink is a distributed system and requires compute resources in order to execute applications. Flink integrates with all common cluster resource managers such as Hadoop YARN and Kubernetes but can also be setup to run as a stand-alone cluster.
Flink is designed to work well with each of the previously listed resource managers. This is achieved by resource-manager-specific deployment modes that allow Flink to interact with each resource manager in its idiomatic way.
When deploying a Flink application, Flink automatically identifies the required resources based on the application’s configured parallelism and requests them from the resource manager. In case of a failure, Flink replaces the failed container by requesting new resources. All communication to submit or control an application happens via REST calls. This eases the integration of Flink in many environments.
[H2]
Run Applications at any Scale
#
Flink is designed to run stateful streaming applications at any scale. Applications are parallelized into possibly thousands of tasks that are distributed and concurrently executed in a cluster. Therefore, an application can leverage virtually unlimited amounts of CPUs, main memory, disk and network IO. Moreover, Flink easily maintains very large application state. Its asynchronous and incremental checkpointing algorithm ensures minimal impact on processing latencies while guaranteeing exactly-once state consistency.
Users reported impressive scalability numbers for Flink applications running in their production environments, such as
applications processing multiple trillions of events per day,
applications maintaining multiple terabytes of state, and
applications running on thousands of cores.
[H2]
Leverage In-Memory Performance
#
Stateful Flink applications are optimized for local state access. Task state is always maintained in memory or, if the state size exceeds the available memory, in access-efficient on-disk data structures. Hence, tasks perform all computations by accessing local, often in-memory, state yielding very low processing latencies. Flink guarantees exactly-once state consistency in case of failures by periodically and asynchronously checkpointing the local state to durable storage.

Want to contribute translation?

Edit This Page

[H1]
What is Apache Flink? — Applications
#
Apache Flink is a framework for stateful computations over unbounded and bounded data streams. Flink provides multiple APIs at different levels of abstraction and offers dedicated libraries for common use cases.
Here, we present Flink’s easy-to-use and expressive APIs and libraries.
[H2]
Building Blocks for Streaming Applications
#
The types of applications that can be built with and executed by a stream processing framework are defined by how well the framework controls streams, state, and time. In the following, we describe these building blocks for stream processing applications and explain Flink’s approaches to handle them.
[H3]
Streams
#
Obviously, streams are a fundamental aspect of stream processing. However, streams can have different characteristics that affect how a stream can and should be processed. Flink is a versatile processing framework that can handle any kind of stream.
Bounded and unbounded streams: Streams can be unbounded or bounded, i.e., fixed-sized data sets. Flink has sophisticated features to process unbounded streams, but also dedicated operators to efficiently process bounded streams.
Real-time and recorded streams: All data are generated as streams. There are two ways to process the data. Processing it in real-time as it is generated or persisting the stream to a storage system, e.g., a file system or object store, and processed it later. Flink applications can process recorded or real-time streams.
[H3]
State
#
Every non-trivial streaming application is stateful, i.e., only applications that apply transformations on individual events do not require state. Any application that runs basic business logic needs to remember events or intermediate results to access them at a later point in time, for example when the next event is received or after a specific time duration.

Application state is a first-class citizen in Flink. You can see that by looking at all the features that Flink provides in the context of state handling.
Multiple State Primitives: Flink provides state primitives for different data structures, such as atomic values, lists, or maps. Developers can choose the state primitive that is most efficient based on the access pattern of the function.
Pluggable State Backends: Application state is managed in and checkpointed by a pluggable state backend. Flink features different state backends that store state in memory or in RocksDB, an efficient embedded on-disk data store. Custom state backends can be plugged in as well.
Exactly-once state consistency: Flink’s checkpointing and recovery algorithms guarantee the consistency of application state in case of a failure. Hence, failures are transparently handled and do not affect the correctness of an application.
Very Large State: Flink is able to maintain application state of several terabytes in size due to its asynchronous and incremental checkpoint algorithm.
Scalable Applications: Flink supports scaling of stateful applications by redistributing the state to more or fewer workers.
[H3]
Time
#
Time is another important ingredient of streaming applications. Most event streams have inherent time semantics because each event is produced at a specific point in time. Moreover, many common stream computations are based on time, such as windows aggregations, sessionization, pattern detection, and time-based joins. An important aspect of stream processing is how an application measures time, i.e., the difference between event-time and processing-time.
Flink provides a rich set of time-related features.
Event-time Mode: Applications that process streams with event-time semantics compute results based on timestamps of the events. Thereby, event-time processing allows for accurate and consistent results regardless whether recorded or real-time events are processed.
Watermark Support: Flink employs watermarks to reason about time in event-time applications. Watermarks are also a flexible mechanism to trade-off the latency and completeness of results.
Late Data Handling: When processing streams in event-time mode with watermarks, it can happen that a computation was considered completed before all associated events have arrived. Such events are called late events. Flink features multiple options to handle late events, such as rerouting them via side outputs and updating previously completed results.
Processing-time Mode: In addition to its event-time mode, Flink also supports processing-time semantics which performs computations as triggered by the wall-clock time of the processing machine. The processing-time mode can be suitable for certain applications with strict low-latency requirements that can tolerate approximate results.
[H2]
Layered APIs
#
Flink provides three layered APIs. Each API offers a different trade-off between conciseness and expressiveness and targets different use cases.

We briefly present each API, discuss its applications, and show a code example.
[H3]
The ProcessFunctions
#
ProcessFunctions
are the most expressive function interfaces that Flink offers. Flink provides ProcessFunctions to process individual events from one or two input streams or events that were grouped in a window. ProcessFunctions provide fine-grained control over time and state. A ProcessFunction can arbitrarily modify its state and register timers that will trigger a callback function in the future. Hence, ProcessFunctions can implement complex per-event business logic as required for many stateful event-driven applications.
The following example shows a KeyedProcessFunction that operates on a KeyedStream and matches START and END events. When a START event is received, the function remembers its timestamp in state and registers a timer in four hours. If an END event is received before the timer fires, the function computes the duration between END and START event, clears the state, and returns the value. Otherwise, the timer just fires and clears the state.
/**
* Matches keyed START and END events and computes the difference between
* both elements' timestamps. The first String field is the key attribute,
* the second String attribute marks START and END events.
*/
public static class StartEndDuration
extends KeyedProcessFunction<String, Tuple2<String, String>, Tuple2<String, Long>> {
private ValueState<Long> startTime;
@Override
public void open(Configuration conf) {
// obtain state handle
startTime = getRuntimeContext()
.getState(new ValueStateDescriptor<Long>("startTime", Long.class));
}
/** Called for each processed event. */
@Override
public void processElement(
Tuple2<String, String> in,
Context ctx,
Collector<Tuple2<String, Long>> out) throws Exception {
switch (in.f1) {
case "START":
// set the start time if we receive a start event.
startTime.update(ctx.timestamp());
// register a timer in four hours from the start event.
ctx.timerService()
.registerEventTimeTimer(ctx.timestamp() + 4 * 60 * 60 * 1000);
break;
case "END":
// emit the duration between start and end event
Long sTime = startTime.value();
if (sTime != null) {
out.collect(Tuple2.of(in.f0, ctx.timestamp() - sTime));
// clear the state
startTime.clear();
}
default:
// do nothing
}
}
/** Called when a timer fires. */
@Override
public void onTimer(
long timestamp,
OnTimerContext ctx,
Collector<Tuple2<String, Long>> out) {
// Timeout interval exceeded. Cleaning up the state.
startTime.clear();
}
}
The example illustrates the expressive power of the KeyedProcessFunction but also highlights that it is a rather verbose interface.
[H3]
The DataStream API
#
The
DataStream API
provides primitives for many common stream processing operations, such as windowing, record-at-a-time transformations, and enriching events by querying an external data store. The DataStream API is available for Java and is based on functions, such as map(), reduce(), and aggregate(). Functions can be defined by extending interfaces or as Java lambda functions.
The following example shows how to sessionize a clickstream and count the number of clicks per session.
// a stream of website clicks
DataStream<Click> clicks = ...
DataStream<Tuple2<String, Long>> result = clicks
// project clicks to userId and add a 1 for counting
.map(
// define function by implementing the MapFunction interface.
new MapFunction<Click, Tuple2<String, Long>>() {
@Override
public Tuple2<String, Long> map(Click click) {
return Tuple2.of(click.userId, 1L);
}
})
// key by userId (field 0)
.keyBy(0)
// define session window with 30 minute gap
.window(EventTimeSessionWindows.withGap(Time.minutes(30L)))
// count clicks per session. Define function as lambda function.
.reduce((a, b) -> Tuple2.of(a.f0, a.f1 + b.f1));
[H3]
SQL & Table API
#
Flink features two relational APIs, the
Table API and SQL
. Both APIs are unified APIs for batch and stream processing, i.e., queries are executed with the same semantics on unbounded, real-time streams or bounded, recorded streams and produce the same results. The Table API and SQL leverage Apache Calcite for parsing, validation, and query optimization. They can be seamlessly integrated with the DataStream API and support user-defined scalar, aggregate, and table-valued functions.
Flink’s relational APIs are designed to ease the definition of data analytics, data pipelining, and ETL applications.
The following example shows the SQL query to sessionize a clickstream and count the number of clicks per session. This is the same use case as in the example of the DataStream API.
SELECT userId, COUNT(*)
FROM clicks
GROUP BY SESSION(clicktime, INTERVAL '30' MINUTE), userId
[H2]
Libraries
#
Flink features several libraries for common data processing use cases. The libraries are typically embedded in an API and not fully self-contained. Hence, they can benefit from all features of the API and be integrated with other libraries.
Complex Event Processing (CEP)
: Pattern detection is a very common use case for event stream processing. Flink’s CEP library provides an API to specify patterns of events (think of regular expressions or state machines). The CEP library is integrated with Flink’s DataStream API, such that patterns are evaluated on DataStreams. Applications for the CEP library include network intrusion detection, business process monitoring, and fraud detection.

Want to contribute translation?

Edit This Page