Dragonfly v1.39.0

v1.39.0
Added 13
  • Add FT.HYBRID command combining full-text and vector similarity search with LINEAR and RRF score fusion strategies, HNSW KNN and range-based vector queries, YIELD_SCORE_AS aliasing, LOAD projection, and FT.PROFILE support
  • Add exact phrase queries with syntax "hello world" and approximate phrase matching with slop "..."~N backed by per-document token positions in compressed posting lists
  • Add BM25STD (Okapi BM25) relevance scoring engine with per-field TF and document-length tracking
  • Add TFIDF and TFIDF.DOCNORM scorer options for FT.SEARCH, FT.PROFILE, and FT.AGGREGATE
  • Add WITHSCORES and SCORER parameters to FT.SEARCH
  • Add ADDSCORES parameter to FT.AGGREGATE to inject a __score field for downstream GROUPBY/SORTBY/REDUCE on relevance
Changed 5
  • PubSub, Monitor, and Invalidation replies are now batched per wakeup instead of one sendmsg per message, delivering up to +12% RPS and 24–70% fewer syscalls
  • INFO REPLICATION and replica memory stats are now served lock-free via per-proactor thread-local snapshots
  • Replace global RCU-based ChannelStore with ShardedHashMap using 16 independent shard locks for pub/sub
  • Tiered hashes are now mutable
  • Cluster config validation now rejects invalid migration configurations at parse time including wrong target masters, duplicate targets, empty or out-of-range slot sets, and overlapping migration ranges
Fixed 2
  • Fix NotifyWatchQueue to exit in O(1) when a watched key is absent instead of O(N) over all waiters
  • Fix zombie-key cleanup across SORT, ZUNIONSTORE, ZINTERSTORE, SHRINK, and hash TTL commands to prevent empty containers in keyspace
Security 1
  • Apply multiple Lua sandbox hardening patches and RESP injection prevention

Dragonfly v1.39.0

This release delivers a significant leap in full-text search capability — FT.HYBRID fuses vector similarity with text queries, exact phrase matching with slop arrives, Porter stemming is on by default for TEXT fields, and BM25STD/TFIDF scoring with WITHSCORES support rounds out a deeply improved search engine. Performance improves across the board: PubSub reply batching cuts syscalls by 24–70% and raises throughput up to +12% RPS, INFO REPLICATION becomes fully lock-free, and a new ShardedHashMap replaces the global RCU ChannelStore for pub/sub. Security is tightened with multiple Lua sandbox hardening patches, RESP injection prevention, and PII-safe error logging. Tiered hashes are now mutable. A significant sweep of zombie-key cleanup fixes closes a class of bugs where lazy field expiry across SORT, ZUNIONSTORE, ZINTERSTORE, SHRINK, and hash TTL commands left empty containers in the keyspace.

Highlights
  • Added FT.HYBRID combining full-text and vector similarity search with LINEAR and RRF score fusion strategies, capping a major release cycle of search improvements (#7420)
  • PubSub, Monitor, and Invalidation replies are now batched per wakeup, delivering up to +12% RPS and 24–70% fewer syscalls at pipeline depths 1–500 (#7479)
  • Tagged chunk serialization splits large objects across snapshot and replication boundaries so they can be reassembled correctly even with interleaved journal entries (#7416, #7070)
Search
  • Adds FT.HYBRID combining full-text and VSIM search via LINEAR (ALPHA/BETA weighted sum of normalized scores) and RRF (Reciprocal Rank Fusion) strategies, with HNSW KNN and range-based vector queries, YIELD_SCORE_AS aliasing, LOAD projection, and FT.PROFILE support (#7420)
  • Adds exact phrase queries ("hello world") and approximate phrase matching with slop ("..."~N), backed by per-document token positions stored in compressed posting lists; BM25 scoring updated for phrase semantics (#7319)
  • TEXT fields now apply Porter stemming by default via the vendored Snowball/libstemmer library; per-field NOSTEM, index-level LANGUAGE (default english), and per-document LANGUAGE_FIELD are all supported; FT.INFO surfaces the new flags (#7295)
  • Adds BM25STD (Okapi BM25) relevance scoring engine with per-field TF and document-length tracking for accurate normalization; results can be ranked when a scorer is activated (#7101)
  • Adds TFIDF (TF × ln(N/n)) and TFIDF.DOCNORM (additionally normalized by field length) as new scorer options for FT.SEARCH, FT.PROFILE, and FT.AGGREGATE; scorer dispatch refactored to a function-pointer table (#7200)
  • FT.SEARCH now supports WITHSCORES and SCORER parameters; FT.AGGREGATE gains ADDSCORES to inject a __score field enabling downstream GROUPBY/SORTBY/REDUCE on relevance (#7181)
  • HNSW vector index memory is now tracked in the search_used memory class and reported in dragonfly_memory_used_bytes via an atomic capacity-based footprint counter (#7205)
Performance
  • PubSub, Monitor, and Invalidation control-path replies are now accumulated in the send buffer and flushed together per wakeup rather than one sendmsg per message; single-message wakeups still flush immediately [+3% RPS at pipeline=1, +12% at pipeline=10, −24% to −70% syscalls at pipeline=1–500] (#7479)
  • INFO REPLICATION and replica memory stats are now served lock-free via per-proactor thread-local snapshots updated on every mutation, eliminating global mutex and per-replica shared-lock fan-out (#7348)
  • Replaced the global RCU-based ChannelStore with a ShardedHashMap (16 independent shard locks), removing the pool-wide broadcast on every subscribe/unsubscribe (#7174)
  • NotifyWatchQueue exits in O(1) when a watched key is absent (previously O(N) over all waiters), directly benefiting BLPOP, BZPOPMIN, and XREAD workloads (#7225)
  • Long-running container iteration commands (LRANGE, SMEMBERS, ZRANGE, HGETALL, etc.) now yield the shard fiber periodically; configurable via --container_iteration_yield_interval_usec (default 500 µs, 0 to disable) (#7391)
  • Integrates OAHSet as an alternative in-memory set backend selectable via --use_oah_set; threaded through RDB save/load, async deletion, defragmentation, and SHRINK (#7246)
  • Adds foundational zero-copy GET infrastructure to CompactObj and LargeString: TryBorrow() returns an encoded view with Copy-on-Write semantics; a thread-local pin registry defers buffer deallocation until all readers finish (#7412)
  • Connections that occasionally receive large commands now trim parse buffers exceeding 256 KiB at more than 2× actual utilization, preventing permanent oversized memory holds (#7378)
Replication
  • Adds serialization_tagged_chunks flag enabling the RDB serializer to split large entries into tagged chunks with envelope IDs; compression is applied per-blob; a related list-decoding crash during chunked load is also fixed (#7416)
  • Implements tagged chunk loading in the RDB loader with per-stream continuation state, supporting interleaved chunks from different objects and cross-database-boundary reassembly (#7070)
  • Four new Prometheus metrics are now emitted on replica instances at /metrics: dragonfly_master_link_status, dragonfly_master_last_io_seconds_ago, dragonfly_sync_in_progress, and dragonfly_slave_repl_offset (#7069)
Cluster
  • Adds --cluster_coordinator_connect_timeout_ms and --cluster_coordinator_response_timeout_ms flags to control coordinator-to-shard timeouts (previously hard-coded at 3000 ms) (#7221)
  • Cluster config validation now rejects invalid migration configurations at parse time: wrong target masters, duplicate targets, empty or out-of-range slot sets, and overlapping migration ranges (#7210)
Commands
  • CLIENT LIST now supports TYPE <normal|master|replica|slave|pubsub> and ID <id> [id …] filter arguments; outbound replication links are surfaced as TYPE master entries; CLIENT KILL ID on a master-link ID returns a descriptive error directing users to REPLICAOF NO ONE (#7377)
  • DEBUG TRAFFIC now supports per-listener recording (LISTENER <main|memcache|admin>), full Memcache protocol capture and replay (SET/GET/CAS/INCR/GAT/etc. with flags, expiry, and CAS tokens), and a v3 file format; v2 files remain readable (#7192)
  • DEBUG TRAFFIC START <path> REPLICA records the replication stream received by a replica from its master; fails with a clear error on standalone/master instances (#7217)
  • BF.SCANDUMP iteratively exports a Scalable Bloom Filter in chunks of up to 16 MiB, returning (cursor, data) pairs until cursor=0 signals end-of-stream (#7092)
  • BF.LOADCHUNK reconstructs an SBF from chunks produced by BF.SCANDUMP, completing the SCANDUMP/LOADCHUNK round-trip; guards prevent other BF commands from operating on a partially-loaded filter (#7169)
  • BGSAVE now accepts the optional SCHEDULE subcommand (case-insensitive) for Redis client compatibility; concurrent saves are still rejected rather than queued (#7286)
  • dfly_bench gains --json_out_file to write a memtier_benchmark-compatible latency JSON report including per-operation aggregate stats (count, ops/sec, average/min/max, p50/p99/p99.9) and a per-second time-series (#7269)
Security
  • The Lua load() built-in is now wrapped to enforce text-only mode ("t"), closing a sandbox escape vector where a malicious script could supply binary Lua bytecode to bypass restrictions (#7376)
  • Lua sandbox hardened: rawset, setmetatable, and getmetatable are overridden to block access to _G and all global library tables; guard metatables are attached to prevent replacement or corruption across script executions (#7370)
  • Error logs from failed commands no longer include user-supplied PII: arguments are truncated at 128 bytes with a byte-count suffix, at most 31 args are logged, EVAL KEYS/ARGV are omitted, and AUTH/ACL subcommands are fully redacted (#7338)
  • INFO ACL section added, exposing aggregate registry metrics (acl_num_users, password/glob counts, estimated acl_total_bytes) maintained incrementally without holding the registry write lock (#7149)
Cloud & Storage
  • S3 snapshot storage is now available in all builds using helio's native S3 client by default; the AWS SDK backend remains available via WITH_AWS + --s3_use_helio_client=false; builds without WITH_AWS no longer exit on s3:// paths (#7180)
  • Azure Blob Storage snapshot backend fully implemented: az:// URIs are correctly routed, OpenWriteFile/OpenReadFile use the cloud::azure API, and cursor-based pagination fixes truncation for large buckets (#7131)
  • Hash write commands (HSET, HDEL) now operate on hashes offloaded to tiered (disk-backed) storage: the serialized listpack is read from disk, mutated in-place, and re-stashed with I/O backpressure (#6085)
  • S3 endpoint override (e.g. for MinIO) is now passed directly to AwsCredsProvider instead of setting AWS_S3_ENDPOINT in the process environment, eliminating side effects for other processes (#7408)
Bug Fixes

Replication & Persistence

  • Fixed slave_repl_offset reporting zero in INFO REPLICATION when replicating from a Redis/Valkey master; offset is now correctly populated from repl_offs_ (#7517)
  • Fixed a TOCTOU race where a DflyConn_ fiber could violate the serializer's fiber-identity check during BGSAVE while Lua EVAL scripts ran concurrently (#7499)
  • Fixed single-shard batch scheduling (ScheduleBatchInShard) allowing a journal DEL entry to execute before the RDB loader finished creating the key on a shard (#7507)
  • Fixed SBF (Scalable Bloom Filter) serialization mismatch where SaveString encoding was read back as raw bytes, crashing Dragonfly in Renamer::FinalizeRename on COPY/DUMP (#7475)
  • Fixed data loaded via DFLY LOAD not propagating to replicas; the journal ring buffer is now cleared and reconnecting replicas are forced into full sync (#6740)
  • Fixed slot MOVED redirections becoming corrupted after replica takeover due to REPLCONF CLIENT-ID sending the replica's internal ID instead of its cluster node ID (#7263)
  • Fixed a race in cluster slot migration restart where StartNewSlotMigrations was called before stale slot data was flushed, causing incoming DFLYMIGRATE FLOW writes to be wiped (#7333)
  • Fixed a SIGSEGV during HNSW index replication where per-shard restoration raced with new HSET operations on still-restoring shards; a three-state lifecycle (kProhibit → kRestoring → kBuilding) now defers all graph mutations until all shards complete (#7126)

Security & Lua

  • Fixed a RESP protocol injection vulnerability where redis.error_reply() / redis.status_reply() strings containing \r\n could inject arbitrary frames into the response stream (#7332)
  • Fixed dragonfly.randstr() accepting unbounded arguments; input is now validated for count (1–32768), size (1–16 MiB), and type (#7368)
  • Fixed EVAL/EVALSHA crashing with a CHECK abort on negative or non-numeric numkeys arguments (e.g. "-0") instead of returning an error reply (#7203)

Search

  • Fixed full-text search scoring (BM25, TF-IDF, TFIDF.DOCNORM) using per-shard document stats instead of global aggregates, causing non-deterministic top-K rankings on multi-shard deployments; a two-phase stats-collection pass now precedes the search (#7250)
  • Fixed TEXT field tokenization not honoring backslash escapes (e.g. \., \-, \:) symmetrically at index time and query time, causing escaped-punctuation queries to fail to match (#7438)
  • Fixed a crash during SAVE when FT.SEARCH lazy field expiry left behind an empty zombie hash key; StringMapAccessor's destructor now deletes the key if the hash is empty after access (#7166)

Commands & Data Structures

  • RESTORE now performs deep (entry-by-entry) integrity validation on listpack, intset, quicklist, and stream payloads, preventing crafted payloads with malformed interior entries from causing OOB reads or crashes (#7502)
  • Fixed DenseSet::SetExpiryTime not setting expiration_used_, causing RandomPairsUnique to skip expiry filtering and return already-expired keys (#7392)
  • Fixed two use-after-free bugs in DenseSet expiry chain walks (ExpireIfNeededInternal, Find2) where deleting a node left node/curr dangling for subsequent iterations (#7393)
  • Fixed single-field hashes incorrectly converting to dense_set encoding when the value exceeded max_map_field_len; listpack encoding is now preserved for single-field hashes regardless of value size (#7257)
  • Fixed an out-of-bounds read in BITCOUNT with BIT index mode when the end index falls exactly on a byte boundary, and incorrect handling of both-negative inverted ranges on short strings (#7190)
  • Fixed per-type memory accounting underflow when AutoUpdater::Run() processed a value whose object type changed (e.g., SORT STORE overwriting a SET with a LIST) (#7142)
  • Fixed XADD computing an incorrect memory diff when RecordJournal preempted and triggered evictions before the stream size update; UpdateStreamSize is now called before RecordJournal (#7235)
  • Fixed MEMORY DEFRAGMENT silently ignoring parse errors for the threshold float argument and proceeding with defaults (#7346)
  • Fixed MULTI/EXEC ghost connections (addr=0.0.0.0, phase=scheduled) permanently stalling when a socket RST prevented CancelBlocking() from being reached (#7273)
  • Fixed a FIN_WAIT_2/CLOSE_WAIT socket leak where connections blocked on BLPOP and similar commands did not observe EOF when the client closed its socket (#7224)
  • Fixed a crash during BGSAVE when a lazily-expired hash field caused an empty container mid-snapshot; DFATAL downgraded to ERROR so the entry is safely skipped (#7452)
  • Fixed a crash in DEBUG OBJHIST/UNIQ-STRS from lazy expiry triggered in an unexpected fiber context, and a second bug where the wrong DB index was used when deleting the resulting zombie key (#7241)
  • Fixed a bad-input parse error in the RESP server parser logging at ERROR level, which allowed malformed/adversarial clients to flood the error log; downgraded to rate-limited WARNING (#7352)

Zombie Key / Empty Container Cleanup — Fixed a widespread class of bugs where lazy field or member expiry left empty containers as zombie keys, causing DFATAL crashes in subsequent SAVE or incorrect command behavior:

  • Empty hash not deleted after: FIELDTTL (#7137), FIELDEXPIRE (#7158), HTTL/HPTTL (#7159), DEBUG OBJHIST/UNIQ-STRS (#7165)
  • Empty set not deleted after: SHRINK compaction (#7160), SORT (#7161), SORT BY pattern (#7162), ZUNIONSTORE (#7163), ZINTERSTORE (#7164)
  • Lazy expiry disabled during cluster migration serialization to prevent mid-iteration deletions that caused DFATAL on empty-container saves (#7167)
What's Changed
Huge thanks to all the contributors! ❤️
New Contributors

Full Changelog: https://github.com/dragonflydb/dragonfly/compare/v1.38.0...v1.39.0

View original

Upgraded? How did it go?

Discussion