Training Example: Apache Lucene – Review the Data, Give Your Score & Compare to the Real AI Evaluation

[H1] Welcome to Apache Lucene

The Apache Lucene™ project develops open-source search
software. The project releases a core search library, named Lucene™ core,
as well as PyLucene, a python binding for Lucene.
Lucene Core is a Java library providing powerful indexing and search features,
as well as spellchecking, hit highlighting and advanced analysis/tokenization capabilities. The PyLucene
sub project provides Python bindings for Lucene Core.

[H2] Latest Lucene Core News

Apache Lucene™ 10.4.0 available (25.Feb)

Apache Lucene™ 10.3.2 available (17.Nov)

Apache Lucene™ 10.3.1 available (06.Oct)

[IMG: Solr Logo]

ANNOUNCEMENT: The Solr™
sub project has moved to a separate Top Level Project (TLP). All things Solr can now be found at
https://solr.apache.org/. Mailing lists and git repositories have changed,
please see details on the Solr website.

[H2] The Apache Software Foundation
¶
The Apache Software Foundation provides support for the Apache community of open-source software projects. The Apache projects are defined by collaborative consensus based processes, an open, pragmatic software license and a desire to create high quality software that leads the way in its field. Apache Lucene, Apache Solr, Apache PyLucene, Apache Open Relevance Project and their respective logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.

Download
Click to
Apache Lucene 10.4.0
Apache Lucene 10.4.0
[H1] Projects¶
Lucene Core (Java)
PyLucene
Open Relevance (Discontinued)
[H1] About¶
License
Who We are
TLP News
Code of Conduct
[H1] Events¶
[H1] ASF links¶
Apache Software Foundation
Thanks
Become a Sponsor
Security
[H1] Editing This Site¶
Instructions
Markdown
[H1] Related Projects¶
Apache Solr
Apache Hadoop
Apache ManifoldCF
Apache Lucene.Net
Apache Mahout
Apache Nutch
Apache OpenNLP
Apache Tika
Apache Zookeeper

[H1] Apache Lucene Core
Apache Lucene™ is a
high-performance, full-featured search engine library written entirely in Java.
It is a technology suitable for nearly any application that requires structured
search, full-text search, faceting, nearest-neighbor search across
high-dimensionality vectors, spell correction or query suggestions.
Apache Lucene is an open source project available for free download. Please use the
links on the right to access Lucene.
[H1] Lucene™ Features
Lucene offers powerful features through a simple API:
[H2] Scalable, High-Performance Indexing
over 800GB/hour on modern hardware
small RAM requirements -- only 1MB heap
incremental indexing as fast as batch indexing
index size roughly 20-30% the size of text indexed
[H2] Powerful, Accurate and Efficient Search Algorithms
ranked searching -- best results returned first
many powerful query types: phrase queries, wildcard queries, proximity
queries, range queries and more
fielded searching (e.g. title, author, contents)
nearest-neighbor search for high-dimensionality vectors
sorting by any field
multiple-index searching with merged results
allows simultaneous update and searching
flexible faceting, highlighting, joins and result grouping
fast, memory-efficient and typo-tolerant suggesters
pluggable ranking models, including the Vector Space Model and Okapi BM25
configurable storage engine (codecs)
Search performance of Apache Lucene is tracked in muliple places. Check out
Mike McCandless' nightly benchmarks for Lucene for an historical view of Lucene's query performance, going back to 2011 for some queries
Search Benchmark, the Game for a comparison of Lucene with other search engines
[H2] Cross-Platform Solution
Available as Open Source software under the Apache License which lets you use
Lucene in both commercial and Open Source programs
100%-pure Java
Implementations in other programming languages available

Download
Click to
Apache Lucene 10.4.0
Apache Lucene 10.4.0
[H1] Resources
Mailing Lists
Developer
Features
Releases
System Requirements
[H1] Release Docs
10.4.0
[H1] About
License
Who We are
[H1] Events
[H1] ASF links
Apache Software Foundation
Thanks
Become a Sponsor
Security
[H1] Related Projects
Apache Solr
Apache Hadoop
Apache ManifoldCF
Apache Lucene.Net
Apache Mahout
Apache Nutch
Apache OpenNLP
Apache Tika
Apache Zookeeper

[H1] Lucene™ Core News
Apache Lucene is a high-performance, full-featured search engine library written entirely in Java.
It is a technology suitable for nearly any application that requires structured search, full-text search, faceting, nearest-neighbor search across high-dimensionality vectors, spell correction or query suggestions.
You may also read these news as an ATOM feed.

[H2]
25 February 2026 - Apache Lucene™ 10.4.0 available
¶
The Lucene PMC is pleased to announce the release of Apache Lucene 10.4.0.
Apache Lucene is a high-performance, full-featured search engine library written entirely in Java. It is a technology suitable for nearly any application that requires structured search, full-text search, faceting, nearest-neighbor search across high-dimensionality vectors, spell correction or query suggestions.
This release contains numerous bug fixes, optimizations, and improvements, some of which are highlighted below. The release is available for immediate download at:
https://lucene.apache.org/core/downloads.html
[H3] Lucene 10.4.0 Release Highlights:
Lucene 10.4 brings some significant performance improvements and a brand new knn vectors format.
Many lucene queries should see a performance improvement of 10-15%, some might even see a 35% improvement! This is due to increasing the block size of the terms postings and better utilization of SIMD optimized code.
Additionally, there is a new scalar quantized format for dense vectors and knn search. Lucene104ScalarQuantizedVectorsFormat and Lucene104HnswScalarQuantizedVectorsFormat allow custom quantized bits for 1, 2, 4, 7, and 8. The recall has improved significantly and for many vector types, quantizing to 2 bits will achieve even better recall than older formats at the 4 bit level. This improves latency while increasing recall for various vector work loads.
New Features
Provides new and improved scalar quantized formats Lucene104ScalarQuantizedVectorsFormat  and Lucene104HnswScalarQuantizedVectorsFormat  for dense vectors. Allowing for quantizing to 1, 2, 4, 7, and 8 bits. For reference, the new 2 bit quantization technique provides better recall and speed than the old 4 bit.
API Changes
New bulk operation APIs for dense vectors and numeric doc values
Improvements and Optimizations
HNSW graphs can now delay being built for tiny segments and will prevent completely rebuilding the graphs when handling deletes
Handling of deletes in general got much faster and cheaper, improving storage costs significantly when there are very few deleted docs
Block size increased for terms postings, significantly improving query latency for many types of queries
Use a coarser-grained competitive iterator with lower construction costs for numeric sorts against fields with DocValuesSkippers.
Runtime Behavior Changes and Bug Fixes
Fix tessellator failure by preferring the shared vertex that is the leftmost vertex of the hole
The "reverse" field of SortField is now final. If you have subclassed SortField, you should set "reverse" in the super constructor.
Align float vectors on disk to 64 bytes, for optimal performance on Arm Neoverse machines
Please read CHANGES.txt for a full list of new features and changes:
https://lucene.apache.org/core/10_4_0/changes/Changes.html
[H2]
17 November 2025 - Apache Lucene™ 10.3.2 available
¶
The Lucene PMC is pleased to announce the release of Apache Lucene 10.3.2.
Apache Lucene is a high-performance, full-featured search engine library written entirely in Java. It is a technology suitable for nearly any application that requires structured search, full-text search, faceting, nearest-neighbor search across high-dimensionality vectors, spell correction or query suggestions.
This release contains numerous bug fixes, optimizations, and improvements, some of which are highlighted below. The release is available for immediate download at:
https://lucene.apache.org/core/downloads.html
[H3] Lucene 10.3.2 Release Highlights:
Bug fixes
Fix potential EOF introduced by optimized filter iterations in MaxScoreBulkScorer.
Please read CHANGES.txt for a full list of changes:
https://lucene.apache.org/core/10_3_2/changes/Changes.html
[H2]
6 October 2025 - Apache Lucene™ 10.3.1 available
¶
The Lucene PMC is pleased to announce the release of Apache Lucene 10.3.1.
Apache Lucene is a high-performance, full-featured search engine library written entirely in Java. It is a technology suitable for nearly any application that requires structured search, full-text search, faceting, nearest-neighbor search across high-dimensionality vectors, spell correction or query suggestions.
This release contains numerous bug fixes, optimizations, and improvements, some of which are highlighted below. The release is available for immediate download at:
https://lucene.apache.org/core/downloads.html
[H3] Lucene 10.3.1 Release Highlights:
Bug Fixes
Fix the Impact returned from Lucene103PostingsReader when frequencies are not indexed.
Please read CHANGES.txt for a full list of changes:
https://lucene.apache.org/core/10_3_1/changes/Changes.html
[H2]
27 September 2025 - Apache Lucene™ 9.12.3 available
¶
The Lucene PMC is pleased to announce the release of Apache Lucene 9.12.3.
Apache Lucene is a high-performance, full-featured search engine library written entirely in Java. It is a technology suitable for nearly any application that requires structured search, full-text search, faceting, nearest-neighbor search across high-dimensionality vectors, spell correction or query suggestions.
This patch release contains bug fixes that are highlighted below. The release is available for immediate download at:
https://lucene.apache.org/core/downloads.html
[H3] Lucene 9.12.3 Release Highlights:
Bug fixes
Support for JDK24+
Reduce sharedArenaMaxPermits from 1024 to 64
Use READONCE to read segment infos
Fix rare spin-loop in RefCountedSharedArena
Ensure vector queries handle advanceShallow correctly
Fix failure due to hole bridge being coplanar with polygon edge
Further details of changes are available in the change log available at:
https://lucene.apache.org/core/9_12_3/changes/Changes.html
[H2]
13 September 2025 - Apache Lucene™ 10.3.0 available
¶
The Lucene PMC is pleased to announce the release of Apache Lucene 10.3.0.
Apache Lucene is a high-performance, full-featured search engine library written entirely in Java. It is a technology suitable for nearly any application that requires structured search, full-text search, faceting, nearest-neighbor search across high-dimensionality vectors, spell correction or query suggestions.
This release contains numerous bug fixes, optimizations, and improvements, some of which are highlighted below. The release is available for immediate download at:
https://lucene.apache.org/core/downloads.html
[H3] Lucene 10.3.0 Release Highlights:
Lucene 10.3 brings major performance improvements.
Lexical search is now vectorized to better take advantage of SIMD instructions, more efficient memory access patterns, CPU pipelining and amortize the cost of virtual function calls. Lucene's nightly benchmarks report a 40% speedup compared with Lucene 10.2 when computing top-100 hits by score on disjunctive and conjunctive queries.
Vector search now better parallelizes fetching vectors into the CPU cache. Lucene's nightly benchmarks report a 15%-20% speedup compared with Lucene 10.2.
The terms dictionary performs about 30% faster than in Lucene 10.2 on primary-key lookups according to Lucene's nightly benchmarks. This should help speed up workloads that rely on terms dictionary lookup performance including primary-key lookups, indexing operations that specify an ID and TermInSet queries.
New Features
Supports reranking with late interaction model multi-vectors, full precision vector similarity scores, or any provided DoubleValuesSource, enabling improved ranking of search results.
Adds a MultiIndexMergeScheduler – a multi-tenant wrapper that allows sharing a common merge scheduler across multiple instances.
API Changes
Adds API to fetch the size of off-heap memory required by a KNN field. This size can be used to help determine the memory requirements for optimal search performance, which can be greatly affected by page faults when not enough memory is available.
RandomVectorScorer now supports a bulk scoring interface.
LeafReader#searchNearestVectors now accepts an AcceptDocs instance instead of a Bits instance to identify document IDs to filter.
Collectors can now take advantage of pre-aggregated data to speed up faceting using LeafCollector#collectRange.
Improvements and Optimizations
Adds optimistic knn search to vector queries. Optimistic knn search addresses a major issue where we return inconsistent results due to race conditions in the shared queue previously used over multi-segment search.
Faster vector search on HNSW graphs through GroupVarInt encoding.
Searcher managers now support 'Adaptive Refresh', enabling users to control the commit points it refreshes on. This helps with graceful handling of large replication payloads in segment-replicated systems.
Runtime Behavior Changes and Bug Fixes
The default ReadAdvice has been changed from RANDOM to NORMAL. MMapDirectory will no longer set any specific read advice out-of-the-box.
Default RefCountedSharedArena.DEFAULT_MAX_PERMITS are reduced to 64. Also fixes the infinite loop when RefCountedSharedArena's underlying Arena#close fails due to concurrent usage of segments.
Uses READONCE when reading segment infos, to fix mmap leaks on segment info files. Includes fixes for multiple other resource leaks.
Please read CHANGES.txt for a full list of new features and changes:
https://lucene.apache.org/core/10_3_0/changes/Changes.html
[H2]
20 June 2025 - Apache Lucene™ 10.2.2 available
¶
The Lucene PMC is pleased to announce the release of Apache Lucene 10.2.2.
Apache Lucene is a high-performance, full-featured search engine library written entirely in Java. It is a technology suitable for nearly any application that requires structured search, full-text search, faceting, nearest-neighbor search across high-dimensionality vectors, spell correction or query suggestions.
This patch release contains bug fixes that are highlighted below. The release is available for immediate download at:
https://lucene.apache.org/core/downloads.html
[H3] Lucene 10.2.2 Release Highlights
[H4] Bug fixes
Reduce NeighborArray on-heap memory during HNSW graph building
Fix IndexSortSortedNumericDocValuesRangeQuery for int sort
ValueSource.fromDoubleValuesSource(dvs).getSortField() would throw errors when used if the DoubleValuesSource needed scores
Further details of changes are available in the change log available at:
http://lucene.apache.org/core/10_2_2/changes/Changes.html
[H2]
20 June 2025 - Apache Lucene™ 9.12.2 available
¶
The Lucene PMC is pleased to announce the release of Apache Lucene 9.12.2.
Apache Lucene is a high-performance, full-featured search engine library written entirely in Java. It is a technology suitable for nearly any application that requires structured search, full-text search, faceting, nearest-neighbor search across high-dimensionality vectors, spell correction or query suggestions.
This patch release contains bug fixes that are highlighted below. The release is available for immediate download at:
https://lucene.apache.org/core/downloads.html
[H3] Lucene 9.12.2 Release Highlights
Reduce NeighborArray on-heap memory during HNSW graph building
Fix IndexSortSortedNumericDocValuesRangeQuery for int sort
ValueSource.fromDoubleValuesSource(dvs).getSortField() would throw errors when used if the DoubleValuesSource needed scores
Disable connectedComponents logic in HNSW graph building.
Further details of changes are available in the change log available at:
https://lucene.apache.org/core/9_12_2/changes/Changes.html
[H2]
1 May 2025 - Apache Lucene™ 10.2.1 available
¶
The Lucene PMC is pleased to announce the release of Apache Lucene 10.2.1.
Apache Lucene is a high-performance, full-featured search engine library written entirely in Java. It is a technology suitable for nearly any application that requires structured search, full-text search, faceting, nearest-neighbor search across high-dimensionality vectors, spell correction or query suggestions.
This patch release contains bug fixes that are highlighted below. The release is available for immediate download at:
https://lucene.apache.org/core/downloads.html
[H3] Lucene 10.2.1 Release Highlights
[H4] Bug fixes
Fix DISIDocIdStream::count so that it does not try to count beyond max.
Correct TermOrdValComparator competitive iterator so that it forces sparse field iteration to be at least scoring window baseline when doing intoBitSet.
Provide better impacts for fields indexed with IndexOptions.DOCS
Fixed lead cost computations for bulk scorers of conjunctive queries that mix MUST and FILTER clauses, and disjunctive queries that configure a minimum number of matching SHOULD clauses.
Further details of changes are available in the change log available at:
http://lucene.apache.org/core/10_2_1/changes/Changes.html
[H2]
10 April 2025 - Apache Lucene™ 10.2.0 available
¶
The Lucene PMC is pleased to announce the release of Apache Lucene 10.2.0.
Apache Lucene is a high-performance, full-featured search engine library written entirely in Java. It is a technology suitable for nearly any application that requires structured search, full-text search, faceting, nearest-neighbor search on high-dimensionality vectors, spell correction or query suggestions.
This release contains numerous features, optimizations, and improvements, some of which are highlighted below. The release is available for immediate download at:
https://lucene.apache.org/core/downloads.html
[H3] Lucene 10.2.0 Release Highlights
Lucene 10.2 includes major search-time performance improvements for a wide variety of queries. This is most notably due to:
Improved storage format of doc IDs in BKD trees for faster decoding.
More vectorization when processing PointRangeQuerys and non-scoring BooleanQuerys.
Encoding of dense blocks of postings lists as bit sets instead of FOR-delta. This change also saves a bit of storage.
Merging matches of dense conjunctive clauses using bitwise ANDs. This especially helps on postings blocks that are encoded as bit sets.
Implementing the ACORN-1 algorithm for pre-filtered vector searches.
Searches that don't require scores and match many docs should generally see good speedups, depending on how expensive the Collector is. Compared with Lucene 10.1.0, Lucene's nightly benchmarks report the following speedups when counting the number of hits of a the following queries:
Disjunctions of term queries: 77% to 4x faster
Conjunctions of term queries: 38% to 5x faster
Filtered disjunctions of term queries: 2.5x to 4x faster
Filtered PointRangeQuery: 3.5x faster
And the following speedup when computing top-100 hits:
Pre-filtered vector search: 3.5x faster
[H4] Ch

[H1] Welcome to PyLucene

[H2] What is PyLucene ?
PyLucene is a Python extension for accessing Java
Lucene™. Its goal
is to allow you to use Lucene's text indexing and searching capabilities from
Python. It is API compatible with Java Lucene version 10.0.0 as of October 20th,
2024.
PyLucene is not a Lucene port but a Python wrapper around Java Lucene. PyLucene
embeds a Java VM with Lucene into a Python process. The PyLucene Python extension,
a Python module called lucene is machine-generated by JCC.
PyLucene is built with JCC, a C++ code generator that makes it
possible to call into Java classes from Python via Java's Native Invocation
Interface (JNI). Sources for JCC are included with the PyLucene sources.
See here for more information and documentation about PyLucene.
[H2] Requirements
PyLucene is supported on macOS, Linux, Solaris and Windows.
PyLucene requires Python 3.x (x >= 3), and Java x (x >= 8 for PyLucene 8 and
earlier, x >= 11 for PyLucene 9, x >= 21 for PyLucene 10).
Python 2 support is untested since PyLucene 9.4.1 / JCC 3.13.
Building PyLucene requires GNU Make and a C++ compiler. Use of
setuptools is recommended.
See the JCC installation instructions for more information
about building JCC from sources.
See the PyLucene installation instructions for more information
about building PyLucene from sources.
[H1] Latest News
¶
[H2]
20 October 2024 - PyLucene 10.0.0 available
¶
This release tracks Lucene Core's 10.0.0 release and requires Java 21 or greater.
See PyLucene 10.0.0 CHANGES and JCC 3.15 CHANGES for details.
Source distributions are available here.
[H2]
11 October 2024 - PyLucene 9.12.0 available
¶
This release tracks Lucene Core's 9.12.0 release and requires Java 11 or greater.
See PyLucene 9.12.0 CHANGES and JCC 3.14 CHANGES for details.
Source distributions are available here.
[H2]
4 March 2024 - PyLucene 9.10.0 available
¶
This release tracks Lucene Core's 9.10.0 release and requires Java 11.
See PyLucene 9.10.0 CHANGES and JCC 3.14 CHANGES for details.
Source distributions are available here.
For all news items, see PyLucene News

Download
Click to begin
of Apache PyLucene
[H1] Documentation¶
License
Features
Install
[H1] Events¶
[H1] ASF links¶
Apache Software Foundation
Thanks
Become a Sponsor
Security
[H1] Related Projects¶
Apache Solr
Apache Hadoop
Apache ManifoldCF
Apache Lucene.Net
Apache Mahout
Apache Nutch
Apache OpenNLP
Apache Tika
Apache Zookeeper