This is the 16th release of Apache BookKeeper!

The 4.9.0 release incorporates hundreds of bug fixes, improvements, and features since previous major release, 4.8.0, which was released four months ago. It is a new milestone in Apache BookKeeper community.

Apache BookKeeper/DistributedLog users are encouraged to upgrade to 4.9.0. The technical details of this release are summarized below.

Highlights

The main features in 4.9.0 cover are around following areas:

  • Dependencies Changes
  • Public API
  • Configuration
  • Metadata
  • Table Service
  • Operations
  • Builds & Testing
  • Enhancements
  • Bug Fixes

Dependencies Changes

Here is a list of dependencies changed in 4.9.0:

Public API

There are multiple new client features introduced in 4.9.0. Here are two highlighted features:

  • LedgerHandleAdv exposes asyncAddEntry variant that takes ByteBuf

Configuration

There are bunch of new settings introduced in both bookie and client in 4.9.0. Here are those settings:

Bookie

  • serverNumIOThreads: configures the number of IO threads for bookies (see #1612)
  • The default value of fileInfoFormatVersionToWrite is bumped from 0 to 1. (see #1689)
  • The default value of journalFormatVersionToWrite is bumped from 5 to 6. (see #1689)

Client

  • numIOThreads: configures the number of IO threads for client (see #1612)

Metadata

There are a few big changes around metadata in 4.9.0. They are:

  • Refactor ledger metadata in LedgerHandle to make ledger metadata instance immutable (see #281)
  • Store ledger metadata in binary protobuf format (see details at #723)
  • Etcd based metadata driver implementation is in BETA release (see details at #1639)

Additionally, there are bunch of new interfaces introduced in the metadata driver API.

Table Service

There are a lot of improvements and features introduced into the table service. The maturity of table service is moving from alpha to beta, and has started to be used as the state storage for Pulsar Functions. More table service usage will come in Pulsar’s future releases.

Starting from 4.9.0, bookkeeper will release a python client for table service. See details at #1691

Operations

HTTP Admin REST Endpoint

  • /api/v1/bookie/gc_details is introduced to retrieve the GC details.
  • /api/v1/bookie/gc is introduced to trigger GC through HTTP REST endpoint.

BookieShell

There are are multiple new commands are added in BookieShell. Here are a few highlighted:

  • regenerate-interleaved-storage-index-file command is introduced for rebuilding the index files for interleaved based ledger storage. (#1642)
  • ledgermetadata command now supports dumping/restoring ledger metadata to/from file.
  • localconsistencycheck command is introduce for running consistency check on bookies locally. (#1819)
  • a new bk-perf script is introduced for running performance benchmark on bookkeeper. (1697)

A new BookKeeper CLI package is released as bkctl. This bkctl package includes both the existing bookie shell and the new bkctl tool.

MDC

Mapped Diagnostic Context (MDC) is now supported at both bookie and client sides. Application request context can be passed as context and being logged through slf4j/log4j. This simplifies throubleshooting of request-level failures/errors. See details at #1672.

Stats Annotation

StatsDoc annotation is introduced in BP-36. The StatsDoc annotation is used for documenting stats added across the project.

Builds & Testing

  • Java 11 is supported for building bookkeeper.

Enhancements

Bug Fixes

Bookie

AutoRecovery

Client

Full list of changes

Apache BookKeeper 4.9.0 Release Notes

An entry is a sequence of bytes (plus some metadata) written to a BookKeeper ledger. Entries are also known as records.

A ledger is a sequence of entries written to BookKeeper. Entries are written sequentially to ledgers and at most once, giving ledgers append-only semantics.

A bookie is an individual BookKeeper storage server.

Bookies store the content of ledgers and act as a distributed ensemble.

A subsystem that runs in the background on bookies to ensure that ledgers are fully replicated even if one bookie from the ensemble is down.

Striping is the process of distributing BookKeeper ledgers to sub-groups of bookies rather than to all bookies in a BookKeeper ensemble.

Striping is essential to ensuring fast performance.

Striping is the process of distributing BookKeeper ledgers to sub-groups of bookies rather than to all bookies in a BookKeeper ensemble.

Striping is essential to ensuring fast performance.

A journal file stores BookKeeper transaction logs.

When a reader forces a ledger to close, preventing any further entries from being written to the ledger.

A record is a sequence of bytes (plus some metadata) written to a BookKeeper ledger. Records are also known as entries.