[SERVER-16119] Hang during initial sync of 2.8.0-rc0 (wiredtiger) Created: 13/Nov/14  Updated: 11/Jul/16  Resolved: 19/Nov/14

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 2.8.0-rc0
Fix Version/s: 2.8.0-rc1

Type: Bug Priority: Critical - P2
Reporter: Cailin Nelson Assignee: Kaloian Manassiev
Resolution: Done Votes: 0
Labels: wiredtiger
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File log.txt     File wt.bt.gz    
Issue Links:
Depends
depends on SERVER-16236 failed to create WiredTiger bulk curs... Closed
Related
is related to SERVER-16118 MongoD 2.8rc0 terminates with "WiredT... Closed
Operating System: ALL
Participants:

 Description   

Trying to do an initial sync of a 2.8.0rc0 secondary from a 2.6.3 primary, got the following error:

2014-11-13T02:36:07.106+0000 I INDEXING [rsSync] 	 building index using bulk method
2014-11-13T02:36:07.107+0000 I INDEXING [rsSync] build index done.  scanned 28 total records. 0 secs
2014-11-13T02:36:07.112+0000 I INDEXING [rsSync] build index on: mmsdbconfig.app.systemDbStats properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.app.systemDbStats" }
2014-11-13T02:36:07.112+0000 I INDEXING [rsSync] 	 building index using bulk method
2014-11-13T02:36:07.113+0000 I INDEXING [rsSync] build index done.  scanned 16 total records. 0 secs
2014-11-13T02:36:07.119+0000 I INDEXING [rsSync] build index on: mmsdbconfig.config.alertConfigHistory properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.config.alertConfigHistory" }
2014-11-13T02:36:07.119+0000 I INDEXING [rsSync] 	 building index using bulk method
2014-11-13T02:36:07.120+0000 I INDEXING [rsSync] build index done.  scanned 41 total records. 0 secs
2014-11-13T02:36:07.126+0000 I INDEXING [rsSync] build index on: mmsdbconfig.config.hostChartSelects properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.config.hostChartSelects" }
2014-11-13T02:36:07.126+0000 I INDEXING [rsSync] 	 building index using bulk method
2014-11-13T02:36:07.126+0000 I INDEXING [rsSync] build index done.  scanned 15 total records. 0 secs
2014-11-13T02:36:07.132+0000 I INDEXING [rsSync] build index on: mmsdbconfig.config.hostClusterHistory properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "mmsdbconfig.config.hostClusterHistory" }
2014-11-13T02:36:07.132+0000 I INDEXING [rsSync] 	 building index using bulk method
2014-11-13T02:36:07.133+0000 I INDEXING [rsSync] build index done.  scanned 14 total records. 0 secs
2014-11-13T02:36:07.133+0000 I REPLSETS [rsSync] initial sync cloning db: 540bb74fe4b039a8d31a04a7_A
2014-11-13T02:36:14.680+0000 F -        Invalid access at address: 0x30818
2014-11-13T02:36:14.723+0000 F -        Got signal: 11 (Segmentation fault).
 
 0xf6ffb9 0xf6f922 0xf6fc4e 0x7f134261b340 0x132bf70 0x132cb6e 0x1330165 0x13184e5 0x139a574 0x138f556 0x1341898 0x7f1342613182 0x7f1341713fbd
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B6FFB9"},{"b":"400000","o":"B6F922"},{"b":"400000","o":"B6FC4E"},{"b":"7F134260B000","o":"10340"},{"b":"400000","o":"F2BF70"},{"b":"400000","o":"F2CB6E"},{"b":"400000","o":"F30165"},{"b":"400000","o":"F184E5"},{"b":"400000","o":"F9A574"},{"b":"400000","o":"F8F556"},{"b":"400000","o":"F41898"},{"b":"7F134260B000","o":"8182"},{"b":"7F1341619000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc0", "gitVersion" : "b6c4e2491c1442b05a160acda0d78001f56a2ade", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFF2D0FE000", "elfType" : 3 }, { "b" : "7F134260B000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7F1342403000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7F13421FF000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7F1341EFB000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F1341BF5000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7F13419DF000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F1341619000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7F1342829000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf6ffb9]
 mongod(+0xB6F922) [0xf6f922]
 mongod(+0xB6FC4E) [0xf6fc4e]
 libpthread.so.0(+0x10340) [0x7f134261b340]
 mongod(+0xF2BF70) [0x132bf70]
 mongod(+0xF2CB6E) [0x132cb6e]
 mongod(__wt_rec_write+0x1B5) [0x1330165]
 mongod(__wt_cache_op+0x5B5) [0x13184e5]
 mongod(__wt_txn_checkpoint+0x1F4) [0x139a574]
 mongod(+0xF8F556) [0x138f556]
 mongod(+0xF41898) [0x1341898]
 libpthread.so.0(+0x8182) [0x7f1342613182]
 libc.so.6(clone+0x6D) [0x7f1341713fbd]
-----  END BACKTRACE  -----

Command line options for the 2.8.0rc0 secondary:

    "cmdLineOpts": {
        "parsed": {
            "replication": {"replSetName": "mmsdev"},
            "processManagement": {"fork": true},
            "systemLog": {
                "path": "/data/mmsdev_4/mongodb.log",
                "destination": "file"
            },
            "config": "/data/mmsdev_4/automation-mongod.conf",
            "net": {"port": 27018},
            "storage": {
                "engine": "wiredtiger",
                "dbPath": "/data/mmsdev_4"
            }
        },
        "argv": [
            "/var/lib/mongodb-mms-automation/mongodb-linux-x86_64-2.8.0-rc0/bin/mongod",
            "-f",
            "/data/mmsdev_4/automation-mongod.conf"
        ],
        "ok": 1
    }



 Comments   
Comment by Githook User [ 02/Dec/14 ]

Author:

{u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'}

Message: Revert "SERVER-16119: faster WT shutdown race avoidance"

This reverts commit fe363010e4b2b953c9876608861c02df4c8662ec, which stalls
WT parallel tests.
Branch: master
https://github.com/mongodb/mongo/commit/43e478efd441bde1f45a716c3da2d39e37566e55

Comment by Githook User [ 02/Dec/14 ]

Author:

{u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'}

Message: Revert "SERVER-16119: faster WT shutdown race avoidance"

This reverts commit fe363010e4b2b953c9876608861c02df4c8662ec, which stalls
WT parallel tests.
Branch: master
https://github.com/mongodb/mongo/commit/43e478efd441bde1f45a716c3da2d39e37566e55

Comment by Githook User [ 27/Nov/14 ]

Author:

{u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-16119: faster WT shutdown race avoidance
Branch: master
https://github.com/mongodb/mongo/commit/fe363010e4b2b953c9876608861c02df4c8662ec

Comment by Kaloian Manassiev [ 19/Nov/14 ]

The hang has been fixed, but we now hit the issue in SERVER-16236. Resolving in place of this ticket.

Comment by Eric Milkie [ 18/Nov/14 ]

I figured out why the initial sync is fasserting with "oplog back in time". Part of the problem was introduced when we added initial sync op batching.
I'll see if this affects 2.6, and I'll have a fix shortly.

Comment by Githook User [ 15/Nov/14 ]

Author:

{u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'}

Message: SERVER-16119 Do not cleanup cached sessions under the session lock
Branch: master
https://github.com/mongodb/mongo/commit/d12f3728b725615cb62b89396efbd3c8c059524f

Comment by Kaloian Manassiev [ 14/Nov/14 ]

OK, I think I see the problem. It looks like we use the point where we return sessions to the cache as an opportunity to update the size of the collection. Which means we are trying to commit transactions under that lock:

Thread 353 (Thread 0x7f634a583700 (LWP 16428)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000000001379a0f in __wt_cond_wait ()
#2  0x0000000001308aeb in __wt_btcur_insert ()
#3  0x0000000001351416 in ?? ()
#4  0x0000000000d97d96 in mongo::WiredTigerSizeStorer::storeInto(mongo::WiredTigerSession*, std::string const&) ()
#5  0x0000000000d8b65e in mongo::WiredTigerKVEngine::syncSizeInfo(bool) const ()
#6  0x0000000000d8b882 in mongo::WiredTigerKVEngine::haveDropsQueued() const ()
#7  0x0000000000d95599 in mongo::WiredTigerSessionCache::releaseSession(mongo::WiredTigerSession*) ()
#8  0x0000000000d930f4 in mongo::WiredTigerRecoveryUnit::~WiredTigerRecoveryUnit() ()
#9  0x0000000000d931d9 in mongo::WiredTigerRecoveryUnit::~WiredTigerRecoveryUnit() ()
#10 0x0000000000af2d91 in mongo::OperationContextImpl::~OperationContextImpl() ()
#11 0x0000000000c79220 in mongo::repl::OplogInfoServerStatus::generateSection(mongo::BSONElement const&) const ()

This happens while the original WT transaction is still active, I believe. Thus if there are a lot of these active, commit will block waiting for them to drain, but they will never drain because they are blocked on writing the size information.

I think the correct fix is like Eliot suggested, to first make this lock only protect the actual removal from the session pool (this would allow pending transactions to drain). And also it would be best if the size storing happens after we have dropped the session so we don't hold any active WiredTiger state.

Comment by Cailin Nelson [ 14/Nov/14 ]

Here is a longer log, of another run, left overnight: https://dropbox.10gen.com/cailin/2014-11-14-08-36/wiredtiger_sync_3.log

Comment by Scott Hernandez (Inactive) [ 14/Nov/14 ]

It is still building the index as of the end of that log file. Parts of serverstatus status require a dblock which is probably why it is blocking. db.currentOp() should yield more info, or just watching the log also.

Comment by Cailin Nelson [ 14/Nov/14 ]

Tried this again.

  • First, I tried a sync of the same data set on 2.8.0rc0/mmapv1 with both the MMS Monitoring and Automation Agents running. This sync took 11 minutes.
  • Next, I retried the sync on 2.8.0rc1-pre/wiredtiger with both the MMs Monitoring and Automation Agents turned off within ~30 seconds of starting the sync. (In order to explore that possibility the one or both agents makes a request to mongod which was causing some sort of lock contention.)

At the conclusion of this log file the process was up and responsive in the shell, but db.serverStatus() would not return results:

ubuntu@ip-10-169-132-210:/data/mmsdev_6$ /var/lib/mongodb-mms-automation/mongodb-linux-x86_64-2.8.0-rc1-pre-/bin/mongo localhost:27018
MongoDB shell version: 2.8.0-rc1-pre-
connecting to: localhost:27018/test
Server has startup warnings: 
2014-11-14T02:19:02.962+0000 I CONTROL  [initandlisten] 
2014-11-14T02:19:02.962+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2014-11-14T02:19:02.962+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2014-11-14T02:19:02.962+0000 I CONTROL  [initandlisten] 
mmsdev:STARTUP2> 
mmsdev:STARTUP2> 
mmsdev:STARTUP2> use admin
switched to db admin
mmsdev:STARTUP2> db.runCommand({getCmdLineOpts:1})
{
	"argv" : [
		"/var/lib/mongodb-mms-automation/mongodb-linux-x86_64-2.8.0-rc1-pre-/bin/mongod",
		"-f",
		"/data/mmsdev_6/automation-mongod.conf"
	],
	"parsed" : {
		"config" : "/data/mmsdev_6/automation-mongod.conf",
		"net" : {
			"port" : 27018
		},
		"processManagement" : {
			"fork" : true
		},
		"replication" : {
			"replSetName" : "mmsdev"
		},
		"storage" : {
			"dbPath" : "/data/mmsdev_6",
			"engine" : "wiredtiger"
		},
		"systemLog" : {
			"destination" : "file",
			"path" : "/data/mmsdev_6/mongodb.log"
		}
	},
	"ok" : 1
}
mmsdev:STARTUP2> db.runCommand({buildInfo:1})
{
	"version" : "2.8.0-rc1-pre-",
	"gitVersion" : "7bdca162807d6436d469a352a129226252cb451d",
	"OpenSSLVersion" : "",
	"sysInfo" : "Linux build9.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49",
	"loaderFlags" : "-fPIC -pthread -Wl,-z,now -rdynamic",
	"compilerFlags" : "-Wnon-virtual-dtor -Woverloaded-virtual -std=c++11 -fPIC -fno-strict-aliasing -ggdb -pthread -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -pipe -Werror -O3 -Wno-unused-local-typedefs -Wno-unused-function -Wno-deprecated-declarations -Wno-unused-but-set-variable -fno-builtin-memcmp -std=c99",
	"allocator" : "tcmalloc",
	"versionArray" : [
		2,
		8,
		0,
		-9
	],
	"javascriptEngine" : "V8",
	"bits" : 64,
	"debug" : false,
	"maxBsonObjectSize" : 16777216,
	"ok" : 1
}
mmsdev:STARTUP2> db.serverStatus()

EDIT: I was looking through the log file and noticed that there were still some serverStatus calls that looked like they were coming from an Automation Agent. I realized I had forgotten to turn off all the Automation Agents associated with this replica set. So, I tried it one more time with all Automation Agents off. I got the same result as before - the LockManager warnings start after about 10 minutes. I'll let this one go for another 30 minutes, so that we can satisfy ourselves that it never recovers from this state.

Comment by Daniel Pasette (Inactive) [ 13/Nov/14 ]

log segment from the end of the file.

Should be noted that this is a build after RC0:

2014-11-13T19:46:51.559+0000 I CONTROL [conn36] db version v2.8.0-rc1-pre-
2014-11-13T19:46:51.559+0000 I CONTROL [conn36] git version: 7bdca162807d6436d469a352a129226252cb451d

Comment by Daniel Pasette (Inactive) [ 13/Nov/14 ]

attaching backtrace for all threads

Comment by Cailin Nelson [ 13/Nov/14 ]

I tried the sync again, and it failed again, but this time at a different point, and with a different backtrace:

2014-11-13T02:47:38.719+0000 I INDEXING [rsSync] build index on: 5401c3c2f64b2377fcd0aff2.oplog_config-e6b08e99dd3588f30baeb3342bca10a0 properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "5401c3c2f64b2377fcd0aff2.oplog_config-e6b08e99dd3588f30baeb3342bca10a0" }
2014-11-13T02:47:38.719+0000 I INDEXING [rsSync] 	 building index using bulk method
2014-11-13T02:47:38.902+0000 I INDEXING [rsSync] build index done.  scanned 5079 total records. 0 secs
2014-11-13T02:47:38.903+0000 I REPLSETS [rsSync] initial sync cloning db: mmsdbagentlog
2014-11-13T02:47:38.993+0000 I NETWORK  [conn87] end connection 10.218.181.240:36146 (21 connections now open)
2014-11-13T02:47:38.994+0000 I NETWORK  [initandlisten] connection accepted from 10.218.181.240:36157 #90 (22 connections now open)
2014-11-13T02:47:40.564+0000 I NETWORK  [conn88] end connection 10.169.132.210:43010 (21 connections now open)
2014-11-13T02:47:40.564+0000 I NETWORK  [initandlisten] connection accepted from 10.169.132.210:43029 #91 (22 connections now open)
2014-11-13T02:47:42.839+0000 F -        [rsSync] Invalid access at address: 0
2014-11-13T02:47:42.861+0000 F -        [rsSync] Got signal: 11 (Segmentation fault).
 
 0xf6ffb9 0xf6f922 0xf6fc4e 0x7f2b4ed92340 0xfd0bd3 0xfd10c0 0x140abb7 0x13092da 0x13224f4 0x1321a8e 0x1309e7d 0x130f21f 0x1338b9e 0x130344f 0x134cea6 0xd8a53e 0x8fa8ce 0x8fb1c0 0x9242d0 0x83729d 0x91ff90 0x921ae6 0xcb8df8 0xcb9742 0xcba13d 0xcc2825 0xfbd5c4 0x7f2b4ed8a182 0x7f2b4de8afbd
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B6FFB9"},{"b":"400000","o":"B6F922"},{"b":"400000","o":"B6FC4E"},{"b":"7F2B4ED82000","o":"10340"},{"b":"400000","o":"BD0BD3"},{"b":"400000","o":"BD10C0"},{"b":"400000","o":"100ABB7"},{"b":"400000","o":"F092DA"},{"b":"400000","o":"F224F4"},{"b":"400000","o":"F21A8E"},{"b":"400000","o":"F09E7D"},{"b":"400000","o":"F0F21F"},{"b":"400000","o":"F38B9E"},{"b":"400000","o":"F0344F"},{"b":"400000","o":"F4CEA6"},{"b":"400000","o":"98A53E"},{"b":"400000","o":"4FA8CE"},{"b":"400000","o":"4FB1C0"},{"b":"400000","o":"5242D0"},{"b":"400000","o":"43729D"},{"b":"400000","o":"51FF90"},{"b":"400000","o":"521AE6"},{"b":"400000","o":"8B8DF8"},{"b":"400000","o":"8B9742"},{"b":"400000","o":"8BA13D"},{"b":"400000","o":"8C2825"},{"b":"400000","o":"BBD5C4"},{"b":"7F2B4ED82000","o":"8182"},{"b":"7F2B4DD90000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc0", "gitVersion" : "b6c4e2491c1442b05a160acda0d78001f56a2ade", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFF138FE000", "elfType" : 3 }, { "b" : "7F2B4ED82000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7F2B4EB7A000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7F2B4E976000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7F2B4E672000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F2B4E36C000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7F2B4E156000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F2B4DD90000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7F2B4EFA0000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf6ffb9]
 mongod(+0xB6F922) [0xf6f922]
 mongod(+0xB6FC4E) [0xf6fc4e]
 libpthread.so.0(+0x10340) [0x7f2b4ed92340]
 mongod(_ZN8tcmalloc11ThreadCache21ReleaseToCentralCacheEPNS0_8FreeListEmi+0xD3) [0xfd0bd3]
 mongod(_ZN8tcmalloc11ThreadCache8ScavengeEv+0x40) [0xfd10c0]
 mongod(free+0x1D7) [0x140abb7]
 mongod(__wt_page_out+0x23A) [0x13092da]
 mongod(__wt_split_evict+0x194) [0x13224f4]
 mongod(__wt_rec_evict+0x41E) [0x1321a8e]
 mongod(__wt_evict_page+0x2D) [0x1309e7d]
 mongod(__wt_page_in_func+0x56F) [0x130f21f]
 mongod(__wt_row_search+0x9AE) [0x1338b9e]
 mongod(__wt_btcur_insert+0x29F) [0x130344f]
 mongod(+0xF4CEA6) [0x134cea6]
 mongod(_ZN5mongo21WiredTigerRecordStore12insertRecordEPNS_16OperationContextEPKcib+0xDE) [0xd8a53e]
 mongod(_ZN5mongo10Collection15_insertDocumentEPNS_16OperationContextERKNS_7BSONObjEb+0x4E) [0x8fa8ce]
 mongod(_ZN5mongo10Collection14insertDocumentEPNS_16OperationContextERKNS_7BSONObjEb+0x70) [0x8fb1c0]
 mongod(_ZN5mongo6Cloner3FunclERNS_27DBClientCursorBatchIteratorE+0x470) [0x9242d0]
 mongod(_ZN5mongo18DBClientConnection5queryESt8functionIFvRNS_27DBClientCursorBatchIteratorEEERKSsNS_5QueryEPKNS_7BSONObjEi+0x1AD) [0x83729d]
 mongod(_ZN5mongo6Cloner4copyEPNS_16OperationContextERKSsRKNS_15NamespaceStringES7_bbbbbNS_5QueryE+0x4F0) [0x91ff90]
 mongod(_ZN5mongo6Cloner2goEPNS_16OperationContextERKSsS4_RKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0xB16) [0x921ae6]
 mongod(+0x8B8DF8) [0xcb8df8]
 mongod(+0x8B9742) [0xcb9742]
 mongod(_ZN5mongo4repl17syncDoInitialSyncEv+0x3D) [0xcba13d]
 mongod(_ZN5mongo4repl13runSyncThreadEv+0x185) [0xcc2825]
 mongod(+0xBBD5C4) [0xfbd5c4]
 libpthread.so.0(+0x8182) [0x7f2b4ed8a182]
 libc.so.6(clone+0x6D) [0x7f2b4de8afbd]
-----  END BACKTRACE  -----

Generated at Thu Feb 08 03:40:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.