Dragonfly v1.36.0

Dragonfly v1.36.0

Important stability fixes and changes:

1. Support Global Vector index #6103
2. Support unicode tags in #6127
3. django-cacheops compatibility fixes: #6123 #6129
4. better memory defragmentation for json objects: #6240 #6276

What's Changed

• actions: Add a volume mount to container by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6048
• chore: remove RecvProvided from dragonfly connection by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/6050
• fix(rdb_load): Store streamed items per rdb lodaer by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6049
• fix(search): Result set cutoff by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5906
• fix: remove duplicate expired/evicted metric header by @romange in https://github.com/dragonflydb/dragonfly/pull/6058
• fix(test): ssl test to work with pytest 3.12 by @romange in https://github.com/dragonflydb/dragonfly/pull/6059
• feat(hset): Load tiered values in read-only by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5996
• chore(tiering): Add throttling metric + more tests by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6040
• fix: macOS build by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6065
• server: Parse JSON on destination shard by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6061
• feat: add CSS option for FT.CREATE by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6057
• chore(facade): Handle Socket::Recv() returning 0 explicitly by @romange in https://github.com/dragonflydb/dragonfly/pull/6066
• fix: Remove index docs on eviction and cluster migration by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6062
• fix(tests): Disable huge val streaming by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6074
• fix(replication): Add detailed error message for REPLTAKEOVER failures by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6071
• fix(tiering): Fix upload strategy by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6067
• fix(replication): use correct replica pointer in takeover catchup loop by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6068
• fix(replication): add socket timeout to REPLTAKEOVER command by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6070
• fix(takeover): pick correct replica to reconcile slots by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/6034
• test: fix flaky test_takeover_bug_wrong_replica_checked_in_logs by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6075
• feat(helm): adds common label support to the helm chart by @gaganhr94 in https://github.com/dragonflydb/dragonfly/pull/6042
• Revert "feat(helm): adds common label support to the helm chart (#6042)" by @romange in https://github.com/dragonflydb/dragonfly/pull/6077
• fix(server): Move LpGetView to core target by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6076
• feat: adds common label support to the helm chart by @gaganhr94 in https://github.com/dragonflydb/dragonfly/pull/6080
• docs(build-from-source): update LTO instructions to use USE_MOLD option by @glevkovich in https://github.com/dragonflydb/dragonfly/pull/6081
• Update build-from-source.md by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6087
• feat: add ftsearch result parsing for css by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6090
• chore: pull the latest helio by @romange in https://github.com/dragonflydb/dragonfly/pull/6096
• fix: suppress BUSYGROUP warning logs by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6095
• feat(tiering): Basic metrics by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6086
• chore: change the resolution of reply latency to measurement to ns by @romange in https://github.com/dragonflydb/dragonfly/pull/6092
• fix: test_reply_count regression by @romange in https://github.com/dragonflydb/dragonfly/pull/6099
• chore(hooks): add commit-msg hook to enforce Signed-off-by line in commit messages by @glevkovich in https://github.com/dragonflydb/dragonfly/pull/6082
• fix(server): handle INFO command with multiple sections correctly by @glevkovich in https://github.com/dragonflydb/dragonfly/pull/6093
• chore: pull helio by @romange in https://github.com/dragonflydb/dragonfly/pull/6100
• fix: smashing stack on Huffman test by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6102
• tools: Add scripts and supporting files to generate os packge repo by @abhijat in https://github.com/dragonflydb/dragonfly/pull/5923
• chore(deps): bump the actions group across 1 directory with 4 updates by @dependabot[bot] in https://github.com/dragonflydb/dragonfly/pull/6107
• chore(core): Refactoring to prepare for using stateless allocator by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6111
• tools: Download ARM deb package, disable fedora ARM test by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6110
• fix: replica of self by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/6097
• chore: relax pytest requirements by @romange in https://github.com/dragonflydb/dragonfly/pull/6113
• feat(JSON): Use stateless allocator by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6060
• fix(dfly_bench): fix parsing for memcache by @romange in https://github.com/dragonflydb/dragonfly/pull/6116
• chore: update vector-bench README by @romange in https://github.com/dragonflydb/dragonfly/pull/6117
• chore: add mi_malloc memory test by @romange in https://github.com/dragonflydb/dragonfly/pull/6118
• docs: add copilot-instructions.md for AI coding agent onboarding by @Copilot in https://github.com/dragonflydb/dragonfly/pull/6112
• fix: Handle django-cache script so it could run in dragonfly by @romange in https://github.com/dragonflydb/dragonfly/pull/6123
• chore: replace PMR_NS with StatelessAllocator in dense_set by @romange in https://github.com/dragonflydb/dragonfly/pull/6121
• chore: implement background self-identification of huffman encoding by @romange in https://github.com/dragonflydb/dragonfly/pull/6106
• chore: gcc14 for fedora 30 by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6124
• chore: expose more information when takeover fails on master. by @romange in https://github.com/dragonflydb/dragonfly/pull/6130
• feat(search): Global HNSW vector index by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6103
• fix(search): Support unicode for tags by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6127
• chore: use our own vector for hnsw distance computation by @romange in https://github.com/dragonflydb/dragonfly/pull/6132
• chore: more fixes for django-cacheops compatibility by @romange in https://github.com/dragonflydb/dragonfly/pull/6129
• chore: refactor hnsw index into separate files by @romange in https://github.com/dragonflydb/dragonfly/pull/6134
• chore: move HierarchicalNSW class to our codebase by @romange in https://github.com/dragonflydb/dragonfly/pull/6136
• feat(tiering): Serialize hashes by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6015
• fix(stream): Allow XINFO in script by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6128
• fix(core): Remove async delete flag by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6139
• feat: implement DenseSet::Shrink by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6094
• chore(core): Move page_usage related code to its own library by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6146
• chore: quietly ignore LATENCY HISTOGRAM command by @romange in https://github.com/dragonflydb/dragonfly/pull/6151
• fix(tiering): missing comma in dashboard json by @goochs in https://github.com/dragonflydb/dragonfly/pull/6150
• chore: add a tool that creates charts based on memtier metrics data by @romange in https://github.com/dragonflydb/dragonfly/pull/6154
• chore: add latency graph to local dashboard by @romange in https://github.com/dragonflydb/dragonfly/pull/6153
• fix: incorrect stalled heartbeat warning by @romange in https://github.com/dragonflydb/dragonfly/pull/6155
• feat(lua): add legacy-float flag for Lua 5.1 number compatibility by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6149
• chore(dfly_bench): properly handle 0 from Recv by @romange in https://github.com/dragonflydb/dragonfly/pull/6159
• chore: minor cleanups around the eval flow by @romange in https://github.com/dragonflydb/dragonfly/pull/6152
• tests: Use AWS public docker registry by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6161
• fix(search): Fix numeric tree splits by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6143
• fix(tiering): Small test fixes by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6133
• chore: introduce common directory by @romange in https://github.com/dragonflydb/dragonfly/pull/6170
• chore: introduce ParsedArguments class by @romange in https://github.com/dragonflydb/dragonfly/pull/6169
• chore: rename WRITE to JOURNALED by @romange in https://github.com/dragonflydb/dragonfly/pull/6172
• chore: fix data corruption bug by @romange in https://github.com/dragonflydb/dragonfly/pull/6174
• chore: add ability for ParsedArgs to reference BackedArguments by @romange in https://github.com/dragonflydb/dragonfly/pull/6171
• chore: use BackedArguments for implementing a pipeline message by @romange in https://github.com/dragonflydb/dragonfly/pull/6167
• feat(search): Defragment search indices by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6144
• chore(core): Refactor SmallString by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6138
• chore: clean up sanitizers checks by @romange in https://github.com/dragonflydb/dragonfly/pull/6173
• feat(fuzz): improve fuzzing effectiveness, better timeouts, and comprehensive seeds, reproduction tooling by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6166
• chore: use BackedArguments for memcache_parser by @romange in https://github.com/dragonflydb/dragonfly/pull/6176
• CI: download docker for correct architecture by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6180
• fix(search): Fix numeric tree finalization by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6188
• chore: improve backed_args by allowing random access to its arguments by @romange in https://github.com/dragonflydb/dragonfly/pull/6189
• Add coordinator connection reusing by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6179
• chore(server): Don't decode size for empty blob by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6194
• chore: hold memcache value in BackedArguments by @romange in https://github.com/dragonflydb/dragonfly/pull/6193
• chore(core): Remove some unused CompactObj code by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6141
• feat(search): Simple multi-reader multi-writer mutex for hnsw index by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6156
• chore: asynchronous IO for connection fiber by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/6069
• fix: macOS test by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6195
• Integrate hiredis by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6198
• fix(search): Use thread lock for hnsw multi reader/multi writer mutex by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6205
• chore: to move some of the command-specific state into CommandContext by @romange in https://github.com/dragonflydb/dragonfly/pull/6199
• hset_family: Preserve ttl on increment command by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6186
• Pass CommandContext as a mutable pointer to all the commands. by @romange in https://github.com/dragonflydb/dragonfly/pull/6206
• fix: macOS compilation error by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6208
• chore(rax): add move ctor to rax tree, cleanup defragmap by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6178
• docs: add antigravity agent instructions by @romange in https://github.com/dragonflydb/dragonfly/pull/6209
• chore: reduce boilerplate by removing redundant function declarations by @romange in https://github.com/dragonflydb/dragonfly/pull/6207
• chore: move some SendError calls to CommandContext by @romange in https://github.com/dragonflydb/dragonfly/pull/6211
• chore(server): polish CommandContext data structure by @romange in https://github.com/dragonflydb/dragonfly/pull/6214
• fix: regression tests by @romange in https://github.com/dragonflydb/dragonfly/pull/6212
• feat: add WITHSORTKEYS option to FT.SEARCH by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6210
• chore(tiering): Optimize serialization by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6164
• chore: Introduce ParsedCommand by @romange in https://github.com/dragonflydb/dragonfly/pull/6219
• fix(search): remove invalid DCHECK in FT.INFO for concurrent index operations by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6204
• fix(server): Fix EXPIRE bugs, support non-exclusive options by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6203
• chore: pass ParsedCommand to DispatchMC by @romange in https://github.com/dragonflydb/dragonfly/pull/6221
• fix(server): Rename with tiering by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6217
• feat(search): Yield when rebuilding index by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6224
• fix: load search index from old snapshots with TAG SORTABLE SEPARATOR order by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6220
• chore: switch dev docker to GCC 14 by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6225
• fix: fix build when search is off by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6229
• chore: further clean ups around ParsedCommand. by @romange in https://github.com/dragonflydb/dragonfly/pull/6223
• docs: create unified AI agent instruction system with AGENTS.md by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6231
• fix(core): Account for size change on json object defragment by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6053
• feat: create a wrapper for hiredis by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6233
• chore: further replace SendError calls with CommandContext by @romange in https://github.com/dragonflydb/dragonfly/pull/6236
• chore: simplify memcache pipelining flow by @romange in https://github.com/dragonflydb/dragonfly/pull/6230
• chore: reduce reply builder interface by @romange in https://github.com/dragonflydb/dragonfly/pull/6237
• chore: swap SendError calls in json_family by @romange in https://github.com/dragonflydb/dragonfly/pull/6238
• feat: add open addressing hash set by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/4705
• fix: macOS build by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6239
• chore: reduce compilation dependencies between server and facade by @romange in https://github.com/dragonflydb/dragonfly/pull/6242
• chore: switch SendError calls to CommandContext by @romange in https://github.com/dragonflydb/dragonfly/pull/6245
• chore: minor fixes around async execution by @romange in https://github.com/dragonflydb/dragonfly/pull/6241
• chore: Split CompactObj into Key&Value by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6218
• chore(core): remove unused num_used_buckets_ from DenseSet by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6250
• fix: cross-shard search tests by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6244
• chore: refactor ParsedCommand implementation into a separate file by @romange in https://github.com/dragonflydb/dragonfly/pull/6251
• fix: reset obj_malloc_used_ in DenseSet::Clear by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6254
• chore: simplify memcache_parser_test by @romange in https://github.com/dragonflydb/dragonfly/pull/6259
• chore(server): Remove more family header definitions by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6255
• chore: MemcacheParser now consumes all the input. by @romange in https://github.com/dragonflydb/dragonfly/pull/6260
• chore: an issue of destruction of in-flight parsed commands by @romange in https://github.com/dragonflydb/dragonfly/pull/6258
• chore: switch SendError calls in zset_family by @romange in https://github.com/dragonflydb/dragonfly/pull/6248
• feat(json): Improve json defragmentation by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6240
• chore: align parse errors with deferred replies by @romange in https://github.com/dragonflydb/dragonfly/pull/6261
• chore(server): Split tiering into own cmake target by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6252
• chore(server): Remove some manual acl by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6256
• chore(common): Include cstdint to fix build error on newer fedora versions by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6268
• chore(server): Multi consistency test by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6215
• chore: move memcache reply rendering into MCRender by @romange in https://github.com/dragonflydb/dragonfly/pull/6267
• chore: reduce compilation dependencies between facade and server by @romange in https://github.com/dragonflydb/dragonfly/pull/6269
• chore(deps): bump the actions group with 4 updates by @dependabot[bot] in https://github.com/dragonflydb/dragonfly/pull/6222
• chore: get rid of command specific flags in MCReplyBuilder by @romange in https://github.com/dragonflydb/dragonfly/pull/6273
• feat(transaction): Blocking unlock multi by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6246
• fix: macOS build by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6275
• Add alloc size to OAHSet by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6272
• chore: switch SendError calls to CommandContext API by @romange in https://github.com/dragonflydb/dragonfly/pull/6270
• chore: implement async dispatching for memcache SET command by @romange in https://github.com/dragonflydb/dragonfly/pull/6247
• chore(helio): bump for improved TLS EOF handling by @glevkovich in https://github.com/dragonflydb/dragonfly/pull/6274
• chore(server): Unite family headers by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6266
• chore(server): Use cycle based quota for defragmentation by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6262
• chore: swap ParsedEntry::CmdData with BackedArguments by @romange in https://github.com/dragonflydb/dragonfly/pull/6278
• chore(json): Use new json defragmentation by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6276
• fix: loading of a snapshot from a cloud folder by @romange in https://github.com/dragonflydb/dragonfly/pull/6280
• chore: fix build error by @romange in https://github.com/dragonflydb/dragonfly/pull/6289
• chore: simplify GAT command arguments by @romange in https://github.com/dragonflydb/dragonfly/pull/6288
• chore: combine GAT and MGET code together by @romange in https://github.com/dragonflydb/dragonfly/pull/6290
• fix: allow Cm_Cache_Backend_Redis Lua scripts by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6292
• fix: CONFIG GET returns numeric bytes for memory configs by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6291
• chore: remove Stored and Deleted interfaces from ParsedCommand by @romange in https://github.com/dragonflydb/dragonfly/pull/6287
• fix: macOS build by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6293
• chore: change DispatchCommand interface by @romange in https://github.com/dragonflydb/dragonfly/pull/6283
• chore: implement support for deferred mget by @romange in https://github.com/dragonflydb/dragonfly/pull/6286
• chore: update .gitignore and clarify TRAFFIC help by @glevkovich in https://github.com/dragonflydb/dragonfly/pull/6294
• chore: optimize experimental_io_loop_v2 flow by @romange in https://github.com/dragonflydb/dragonfly/pull/6295
• feat(build): rename cxx_test to helio_cxx_test by @glevkovich in https://github.com/dragonflydb/dragonfly/pull/6297
• chore(server): Remove list family header file by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6296
• Sorting for cluster search by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6301
• feat(server): Allow SELECT inside global/non-atomic exec/eval by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6302
• chore: introduce RespSrvParser by @romange in https://github.com/dragonflydb/dragonfly/pull/6300
• chore: move more reply calls to CommandContext API by @romange in https://github.com/dragonflydb/dragonfly/pull/6303
• fix: Handle expired hash fields during RENAME without crashing by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6306
• chore: refresh release settings for dragonfly by @romange in https://github.com/dragonflydb/dragonfly/pull/6304
• chore: fix release action by @romange in https://github.com/dragonflydb/dragonfly/pull/6308
• test(geo): relax fp checks for Helio Haswell upgrade by @glevkovich in https://github.com/dragonflydb/dragonfly/pull/6307
• Fix fiber stack overflow and improve JSON safety by @romange in https://github.com/dragonflydb/dragonfly/pull/6316
• CI: sync valkey tests by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6319
• chore: Add debug information to investigate memcache regtest failure. by @romange in https://github.com/dragonflydb/dragonfly/pull/6322
• fix: handle control messages during SquashPipeline iteration by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6313
• chore(defrag): Add defragmentation of search indices by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6281
• CI: Extract common step for syncing valkey tests into action by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6333
• tests: make valkey_search integration tests optional by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6332
• chore: Reused parsed messages for memcache workloads by @romange in https://github.com/dragonflydb/dragonfly/pull/6324
• fix: memcache parser CRLF handling by @romange in https://github.com/dragonflydb/dragonfly/pull/6330

Huge thanks to all the contributors! ❤️

New Contributors

• @gaganhr94 made their first contribution in https://github.com/dragonflydb/dragonfly/pull/6042
• @glevkovich made their first contribution in https://github.com/dragonflydb/dragonfly/pull/6081
• @Copilot made their first contribution in https://github.com/dragonflydb/dragonfly/pull/6112
• @goochs made their first contribution in https://github.com/dragonflydb/dragonfly/pull/6150

Full Changelog: https://github.com/dragonflydb/dragonfly/compare/v1.35.0...v1.36.0