[SERVER-15498] Primary not usable because of BSON issue Created: 02/Oct/14  Updated: 14/Nov/14  Resolved: 14/Nov/14

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 2.4.1
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Dharshan Rangegowda Assignee: Ramon Fernandez Marina
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

I have 2+1 replica set. Yesterday the primary was not usable due to the following bug. The log file also appeared corrupted. The last line of the line is extremely long in this format

Mon Sep 29 0117p: { opiecursive lock ns: productio priceapi_producti , active:production.proxipt556950798, active: true, secm1ve:ruexiecti tx7f2 01ve: tk bifcursive lock ns: pricea
pi_produc_production.pron.proxixies817p: { opitk bif recursivtproduc 01ve: tk bif recursivtproduc 01ve: tk tive: true, secm1ve:ruexie7f2rp5001ctimcock ns: priceproductionsiveapitex
iepriceapi_produ11I8.6tio821piticeapiteadId: "0x7f2 01v.pr6tio821piticeapiteadId: "0x7f2 01v.pr6tio821piticeapiteadId: "0x7f2 01v.pr6tio821piticeapiteadId: "0x7f2 01v.pr6tio821piti
ceapitead8i_production.proxies8eecursivtproduc 01ve: tk uiceapi_productitapi_peapduc 01ve: tk bif recursivtproduc 01ve: tk bif reecursivtproduc 01ve: tk uiceapi_production.pr211:29
:28n5001eapitne8oductik18, actId8.prapiticeapiteacecursive lock n prteadId: ve: tk bi1 01:29:28nn814819] warni14819", thre11Id814819", thrpretionsive lkd: 155.] warni148 pricpi_cti
oneapi_production.proxiepriceapi_produ11I8.6tio821piticeapiteadId: "0x7f2 01ve: tk bi1 01:29:28nn814819] warni14819", thre11Idon5001eapiticeapiticeapproduction.p active:production.
p active:production.p active:production.p activ:Ye:pdf: "0x7f2 01v.pr6tio821pititiceapiteadId: "0x7f2 01v.pr6tio821piticeapitead8i_production.proxies8eecursivtproduc 01ve: tk uicea
pi_productitapi_peapduc 01ve: tk bif recursivtproduc 01ve: tk bif reecursivtproduc 01ve: tk uiceapi_production.pr211:29:28n5001eapitne8oductik18, actId8.prapiduction.pr211:29:28n50
01eapitne8oductik18, actId8.prapiduction.pr211:29:28n5001eapitne8oductik18, actId8.prapiduction.pr211:29:28n5001eapitne8oductik18, actId8.prapiduction.pr211:29:28n5001eapitne8oduct
ik18, actId8.prapiduction.pr211:29:28n5001eapitne8oducti14819", thrpriceapi_production.proxiepricetionsive lpStats: { tlacklist: n ns: priceapi4piticeapiticea lockStpr secm1ve:ruex
ie7f2rp5001ctimcock ns: priceproductionsive: productio priceapi_producti , active:production.p active:production.p active:production.pceapi_production.pr211:29:28n5001eapitne8oduct
ik18, actId8.prapiduction.pr211:29:28n5001eapitne8oductik18, actId8.prapiduction.pr211:29:28n5001eapitne8oe: 0x7f2 01ve: tId: "0x7f2 01ve: tId: "0x7f2 01ve: tId: "0x7fes817p: { opi
d: 1556950798, ns: priceapi_productionsive lock ns: priceapi_prod:p 29 0117p: { opiecursive lock ns: pr4apitne8oductik18, actId8.prapiticeapiteacecursive lock n prteadId: ve: tk bi
1 01:29iceproductionsive: productio priceapi_producti , active:production.p active:production.p active:production.pceapi_production.pr211:29:28n5001eapitne8oductik18, actId8.prapid
uction.pr211:29:28n5001eapitne8oductik18, actId8.prapiduction.pr211:29:28n5001eapitnetId8.prapiduction.pr211:29:28n5001eapitne8oe: 0x7f2 01ve: tId: "0x7f2 01ve: tId: "0x7f2 01ve: t
Id: "0x7fes817p: { opid: 1556950798, ns: priceapi_productionsive lock ns: priceapi_prod:p 29 0117p: { opiecursive lock ns: pr4apitne8oductik18, actId8.prapiticeapiceapiteadId: ve:
tk bi1 01:29:28nn814819] warni14819", thre11Id814819", thpriceaf recursiepriceapi_productio pricapi17p: { opid: 155n814819] warni14819", thre11Id814819", thrpriceapi_8, active: tru
e, secmd$production.proxiepriceaf recursiepriceapi_productio priceapon.proxceapi_producti1on.pr211:29:28n5001eapitne8oductik18, actId8.prapiduction.pr211:29:28n5001eapit08eapitne8o
ductik18, actId8.prapiduction.pr211:29:28n5001eapitne8oductik18, actId8.prapiduction.pr211:29:28n5001eapitnetId8.prapiduction.pr211:29:28n5001eapitne8oe: 0x7f2 01ve: tId: "0x7f2 01
ve: tId: "0x7f2 01ve: tId: "0x7fes817p: { opid: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798,
ns: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns:8d: 155n814819] warni14819", thre
11Id814819", thrpriceapi_8, active: true, secmd$production.proxiepriceaf recursiepriceapi_productio priceapon.proxceapi_producti1on.pr211:29:28n5001eapitne8oductik18, actId8.prapid
uction.pr211:29:28n5001eapit08eapitne8oductik19ctId8.prapiduction.pr211:29:28n5001eapitnetId8.prapiduction.pr211:29:28n5001eapitne8oe: 0x7f2 01ve: tId: "0x7f2 01ve: tId: "0x7f2 01v
e: tId: "0x7tId: "0x7f2 01vx7fes817p: { opid: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns
: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced:cock41riced: 1556950798, ns: priced: 1556950798, ns:8d: 155n814819] warni14
819", thre11Id814819", thrpriceapi_8, active: true, secmd$production.proxiepriceaf rock ns: pr4apitne8oductik18, actId8.prapiticeapiteacecursive lock n prteadId: ve: tk bi1 01:29:2
8nn814819] warni14819", thre11Id814819", thrpriceapi_8, active: true, secmd$production.proxiepriceaf recursiepriceapi_productio priceapon.proxceapi_production.proed: 1556950798, ns
: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced:cock41riced: 1556950798, ns: priced: 1556950798, ns:8d: 155n814819] warni14819", thre11Id814819", t
hrpriceapi_8, active: true, secmd$production.proxiepriceaf r, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced:cock41riced: 1556950798, ns: priced: 1556950798, ns:8d: 155
n814819] warni14819", thre11Id814819", thrpriceapi_8, active: true, secmd$production.proxiepriceaf rock ns: pr4apitne8oductik18, actId8.prriceapon.proxceapi_production.proed: 15569
50798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced:cock41riced: 1556950798, ns: priced: 1556950798, ns:8d: 155n814819] warni14819", thre11Id8
14819", thrpriceapi_8, active: true,6, ns: priced: 1556950798, ns:8d: 155n814819] warni14819", thre11Id814819", thrpriceapi_8, active: true, secmd$production.proxiepriceaf r, ns: p
riced: 1556950798, ns: priced: 1556950798, ns: priced:cock41riced: 1556950798, ns: priced: 1556950798, ns:8d: 155n8, active:pducti , active:pducti , active:5bctio priceapi_producti
 , active:pducti , active:pducti , active:pducti , active:pducti , active:pducti , active:pducti , active:pducti , active:pducti , active:pducti , active:pducti , active:pducti , a
ctive:pducti riced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced:cock41riced: 1556950798, ns: priced: 1556950798, ns:8d: 155n814819] warni14819adIdock ns:
 priceapi_production.production.prp5001ctimconn81001ctimconn8148lkd: 1554id: 155695n.proxn.prp5001ctimconn8ck ns: pri0n.proxies817p: { opid: 155: 1556950798, active: true, secm1ve:
ruexiecti tx7f2 01ve: tk bifcursive lock ns: priceapi_produc_producactive:pducti , active:pducti , active:5bctio priceapi_producti , active:pducti , active:pducti , active:pducti ,
 active:pducti , active:pducti active:pducti , active:pducti riced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced:cock41riced: 1556950798, ns: priced: 1556
950798, ns:8d: 155n814819] warni14819adIdock ns: priceapi_production.production.prp5001ctimconn81001ctimconn8:ced: 1556950798, ns: priced: 1556950798, ns:8d: 155n814819] warni14819
adIdock n: 1556950798, active: true, secm1ve:ruexiectpriceapi_productio pricapi17p: { opid: 1556ive lock ns: priceapi_production.pron.proxies817p: { opid: 1556950798, active: true,
 secm1ve:17p: { opid: 155699", threalock ns:Yielidealo loc50798, active: true, secm1ve:yi , active:pducti , active:pducti , active:pducti , active:pducti active:pducti , active:pdu
cti riced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced:cock41riced: 1556950798, ns::Yielidealo lockStats: { tlacklist: n ns: priceapi_production.proxiepr
iceapi_peapi_production.proxiepr8nn814854id: 5695n.proxn.prp5001ctimconn8148lkd: 15coc_production.pron.proxixies817p: { opitk bif recursivtproduc 01vproxies817p: { obif reproxiepri
ceapi_produ11If reciteadIdock e: true, secm1ve:rue, secmconn8148i_production.proxies817p:od8ck ns: pri0n18ucursive lock ns: pr4apitne8oductik18, actId8.prapiticeape31d: 1556950798,
 active: true, se31d: 1556950798, active: true, se31d: 1556950798, active: true, se31d: 1556950798, active: trueticean.prp5001ctimconn8eapitne8oductik18, actId8.prapiticeapiteacecu
rsive lock n pribn814854id: 5695n.proxn.prp5001ctimconn8148lkd: 15coc_production.pron.proxixies817p: { opit pr4apitne8oductik18, actId8.prapiticeape31d: 1556950798, active: true, s
e31d: 1556950798, active: true, se31d: 1556950798, active: true, se31d: 1556950798, active: trueticean.prp5001ctimconn8eapitne8oductik18, actId8.prapiticeapiteacecursive lock n pri
bn814d8.prapiduction.pr211:29:28n5001eapitne8oe: 0x7f2 01ve: tId: "0x7f2 01ve: tId: "0x7f2 01ve: tId: "0x7fes817p: { opid: 1556950798, ns: priceapi_productionsive lock ns: priceapi
_prod:p 29 0117p: { opiecursive lock ns: pr4apitne8oductik18, actId8.prapiticeapiteu5001ctimconn8148lkd: 15coc_production.pron.proxixies817p: { opit pr4apitne8oductik18, actId8.pra
piticeape31d: 1556950798, active: t0 1556950798, active: true, se31d: 1556950798, active: trueticean.prp5001ctimconn8ep: { opid: 1556ive lock ns: priceapi_production.pron.proxies81
7p: { opid: 1556950798, active: true, secm1ve:17p: { opid: 155699", threalock ns:Yielidealo loc50798, active: true, secm1ve:yi , active:pducti , active:pducti , active:pducti , act
ive:pducti activtId: "0x7f2 01ve: tId: "0x7f2 01ve: tId: "0x7fes817p: { opid: 1556950798, ns: priceapi_productionsive lock ns: priceapi_prod:p 29 0117p: { opiecursive lock ns: pr4a
pitne8oductik18, actId8.prapiticeapiteu5001ctimconn8148lkd: 15coc_production.pron.proxixies817i_producti ,6i_proiceapricecursive lock ns: priceapie lock ns: 98apiticeapiti0ceapi_pr
oducti ,6i_proiceapricecursive lock ns: pricea:28nn8148hre11Idon5001eapiticeapiticeapitStproduc 01ve: tk bif recursive lock ns: prursive lock ns: pr, t tk bif rec01eapiticersive lo
ck ns: pr, t tk bif rec01eapiticersive lock ns: pr, t tk bif rec01eapiticersive lock ns: pr, t tk bif rec01eapiticersive lock ns: pr, t tk bif rec01eapiticersive lock ns: pr, t tk
bif rec01eapiticersive lock ns: pr, t tk bif rec01eapiticersive lock n2 01ve: tIroduc 01vproxies817p: { obif reproxiepriceapi_produ11If reciteadIdock e: true, secm1v{ obif reproxie
priceapi_produ11If reciteadIdock e: true, secm1ve:rue, secmconn8148i_pr rec01eap4eactive: d2d: 155695084tion.proxies817p: { opid: 1556950798, active1oeapiticeapiticeapitStproduc 01
ve: tk bif recursive lock ns: prursive lock ns: pr, t tk bif rec01eapiticersive lock ns: pr, t tk bif rec01eapiticersive lock ns: pr, t tk bif rec01eapiticersive lock ns: pr, t tk
bif rec01eapiticersive lock ns: pr, t tk bif rec6lock ns: pr, t tk bif rec01eapiticersive lock ns: pr, t tk bif rec01eapiticersive lock ns: pr, t tk bif rec01eapiticersive lock ns:
 pr, t tk bif rec01eapiticersive lock ns: pr, t tk bif rec01eapiticersive lock n2 01ve: tIroduc 01vproxies817p: { obif reproxieiteadIdock e: true, secm1v{ obif reproxiepriceapi_pro
du11If reciteadIdock e: true, secm1ve:rue,produ11If reciteadIdock e: true, secm1ve:rue, secmconn8148i_pr rec01eap4eactive: d2d: 155695084tion.proxies817p: { opid: 1556950798, activ
e1oeapiticeapiticeapitStproduc 01ve: tk bif recursive lock ns: prursive lock ns: pr, t tk bif rec01eapiticersive lock ns: priceapi4piticeapiticea loion.pr211:29:280ittk bif rec01ea
piticeapiticea lockStproduc 01ve: tk bif recursive lock ns: p treapitne8oductik18, actId8.prapiticeapiteacecursive lock n prteadId: ve: tk bi1 01:29:28nn814819] warni14819", thre11
Id814819", thrpric active:pducti , acive:pducti , active:pducti , activericeapi_production.production.prp5001ctimconn81001ctimconn8148lkd: 1554id: 155695n.proxn.prp5001ctimconn8ck
ns: pri0n.proxies817p: { opid: 155: 1556950798, active: true, secm1ve:ruexiecti todueapi_pducti  acive:pducti , active:pducti , activericeapi_production.production.prp5001ctimconn8
1001ctimconn8148lkd: 1554id: 155695n.proxn.prp5001ctimconn8ck ns: pri0n.eapi4piticeapiticea loion.pr211:29:280ittk bif rec01eapiticeapiticea lockStproduc 01ve: tk bif recursive loc
k ns: p treapitne8oductik18, actId8o1:29:28nn814819] warni14819", thre11Id814819", thrpric active:pducti , acive:pducti , active:pducti , activericeapi_production.production.prp500
1ctimconn81001ctimconn8148lkd: 1554id: 155695n.proxn.prp5001ctimconn8ck ns: pri0n.proxies817p: { opid: 155: 155695 , active:pducti , active:pducti , active:pducti , active:pducti s
ive:pductucti , active:pduep5001ctimconn8ck ns: pri0n.proxies817p: { opid: 155: 1556950798, active: true, secm1ve:ruexiecti todueapi_pducti  acive:pducti , active:pducti , activeri
ceapi_production.production.prp5001ctimconn81001ctimconn8148lkd: 1554id: 155695n.proxn.prp5001ctimconn8ck3pri0n.eapi4piticeapiticea loion.pr211:29:280ittk bif rec01eapiticeapiticea
 lockStproduc 01ve: tk bif recursive lock ns: p treapitne8oductik18, actId8o1:29:28nn814819] warni14819", thre11Id814819", thrpric active:pducti , acive:pducti , active:pducti , ac
tive: pr, t tk bif rec01eapiticeapiti0ceapi_producti ,6i_proiceapkd: 15conn8148lkd: 155.] warni14819", thre11Id814819", thrpriceapi_production.proxieption.proxies817p:oducsiiticeap
iteadId: "0x7f2 01ve: tk bi1 01:29:28nn814819] warni14819", thre11Id29:28nn814819]ve: tk bif recursive lock ns: pr, t tk bif rec01eapiticeapiti0ceapi_producti ,6i_proiceapricecursi
ve lock ns: priceapie lock ns: priceproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 01:29:28nn814819] warni14819", th acc01eapiticeapiticea lockStproduc 01ve: tk bif recursive lock
 ns: p treapitne8oductik18, actId8o1:29:28nn814819] warni14819", thre11Id814819", thrpric active:pducti , acive:pducti , active:pducti , active: pr, t tk bif rec01eapiticeapiti0cea
pi_producti ,6i_pre: tk bi1 01:29:28nn814819] warni14819", thre11Id29:28nn814819]ve: tk bif recursive lock ns: pr, t tk bif rec01eapiticeapiti0ceapi_producti ,6i_proiceapricecursiv
e lock ns: priceapie lock ns: priceproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 01:29:28nn814cpi_ctioneapi_production.proxiepriceapi_producti ,6i_proiceapie lock ns: pricecursiv
e lock ns: priceapie lock ns: priceproductionsiiticeapiteadId: "0x7f2 01ve19", thrpriceapi_8, active: true, secmd$production.proxiepriceaf recursiepriceapi_productio priceapon.prox
ceapi_produ: 1556950798, active: true, secm1ve:ruexiecti tx7f2 01ve: tk bifcursive lock ns: priceapi_produc_production.pron.proxixies817p: { opitk bif r, t tk bif rec01eapiticeapit
i0ceapi_producti ,6i_pre: tk bi1 01:29:28nn814819] warni14819", thre11Id29:28nn814819]ve: tk bif recursive lock ns: pr, t tk bif rec01eapiticeapiti0ceapi_ppriceapie lock ns: pricep
roductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 01:29:28nn814cpi_ctioneapi_production.proxiepriceapi_producti ,6i_proiceapie lock ns: pricecursive lock ns: priceapie lock ns: pricep
rriceaf recursiepriceapi_productio priceapon.proxceapi_produ: 1556950798, active: true, secm1ve:ruexiecti tx7f2 01ve: tk bifcursive lock ns: priceapi_produc_production.pron.proxixi
es817p: { opitk bif r, t tk bif rec01eapiticeapiti0ceapi_producti ,6i_pre: tk b14819", threi ,6i_pre: tk bi1 01:29:28nn814819] warni14819", thre11Id29:28nn814819]ve: tk bif recursi
ve lock ns: pr, t tk bif rec01eapiticeapiti0ceapi_ppriceapie lock ns: priceproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 01:29:28nn814cpi_ctioneapi_production.proxiepri_pre: tk b
i1 01:29:28nn814819] warni14819", thre11Id29:28nn814819]ve: tk bif recursive lock ns: pr, t tk bif rec01eapiticeapiti0ceapi_ppriceapie lock ns: pricep:28nn814819] warni14819", thre
11Id29:28nn814819] warni14819", thre11Idon5001eap: tk bi1 01:29:28nn814819] warni14819", thre11Idon5001eapiticeapitice: tk bifcursive lock ns: priceapi_produc_production.pron.proxi
xies817p: { opitk bif r, t tk bif rec01eapiticeapiti0ceapi_producti ,6i_pre: tk b14819", threi ,6i_pre: tk bi1 01:29:28nn814819] warni14819", thre11Id29:28nn814819]ve: tk bif recur
sive lock ns: p"0x7f2 01ve: tId: "0x7fes817p: { opid: 1556950798, ns: priceapi_productionsive lock ns: priceapi_prod:p 29 0117p: { opiecursive lock ns: pr4apitne8oductik18, actId8.
prapiticeapiteacecursive lock n prteadId: ve: ythre11Id814819", thrpriceapi_8, active: true, secmd$production.proxiepriceaf rock ns: pr4apitne8oductik18, actId8.prapiticeapiteacecu
rsive lock n prteadId: ve: tk bi1 01:29:28nn814819] warni14819", thre11Id814819", thrpriceapi_8, active: true, secmd$productitk bif recursivtproduc 01vproxies817p: { obif reproxiep
riceapi_produ11If reciteadIdock e: true, secm1ve:rue, secmconn8148i_production.proxies817p:od8ck ns: pri0n18ucursive lock ns: pr4apitne8oductik18, Id814819", thrpriceapi_8, active:
 true, secmd$production.proxiepriceaf rock ns: pr4apitne8oductik18, actId8.prriceapon.proxceapi_production.proed: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: pr
iced: 1556950798, ns: priced:cock41riced: 1558, ns: priced:cock41riced: 1556950798, ns: priced: 1556950798, ns:8d: 155n814819] warni14819", thre11Id814819", thrpriceapi_8, active:
true, secmd$production.proxiepriceaf rock ns: pr4apitne8oductik18, actId8.prriceapon.proxceapi_production.proed: 15569507982 fes817p: { opid: 1556950798, ns: priced: 1556950798, ns
: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced: 15
56950798, ns: priced: 1oduction.proxiepriceaf rock ns: pr4apitne8oductik18, actId8.prriceapon.proxceapi_production.proed: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798
, ns: priced: 1556950798, ns: priced:cock41riced: 1558, ns: priced:cock41riced: 1556950798, ns: priceed: 1556950798, ns: priced: 1556950798, ns: priced: 1oduction.proxiepriceaf roc
k ns: pr4apitne8oductik18, actId8.prriceapon.proxceapi_production.proed: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced:cock41ricedn
s: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1oduction.proxiepriceaf rock ns: pr4apitne8oductik18, actId8.prriceapon.p
roxceapi_production.proed: 1556950798, ns: priced: 1556950798, ns: priced: 15roed: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced:co
ck41riced: 1556950798, ns: priced: 1556950798, ns:8d: 155n814819] warni14819", thre11Id814819", thrpriceapi_8, active: true, secmd$production.proxiepriceafl556950798, active: true,
 se31d: 1556950798, active: true, se31d: 1556950798, active: true, se31d: 1556950798, active: trueticean.prp5001ctimconn8eapitne8oductik18, actId8.prapiticeapiteacecursive lock n p
ribn814d8.prapiduction.pr211:29:28n5001eapitne8oe: 0x7d: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1oduction.proxiepriceaf rock ns: pr4apitne8oductik1
8, actId8.prriceapon.proxceapi_production.proed: 1556950798, ns: priced: 1556950798, ns: priced: 15roed: 1556950798, ns: priced: 15561556950798, ns: priced:cock41riced: 1556950798,
 ns: priced: 1556950798, ns:8d: 155n814819] warni14819", thre11Id814819", thrpriceapi_8, active: true, secmd$production.proxiepriceafl556950798, active: true, se31d: 1556950798, ac
tive: true, se31d: 1556950798,  lock ns: pricecursive lock ns: priceapie lock ns: priceproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 01:29:28nn814819] warni14819", thre11Idon5001
eapiticeapiticeapitStproduc 01ve: tk bif recursive lock ns: pr, t tk bif rec01eapiticeapiticean.prp5001ctimcoceaf r, ns: priced: 15569507s: pricecursive lock ns: priceapie lock ns:
 priceproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 01:29:28nn814819] warni14819", thre11Idon5001eapiticeapiticeapitStproduc 01ve: tk bif recursive lock ns: pr, t tk bif rec01eap
iticeapiticean.prp5001ctimcoceaf r,.6950798, active: true, se31d: 1556950798, active: true, se31d: 1556950798,  lock ns: pricecursive lock ns: priceapie lock ns: priceproductionsii
ticeapiteadId: "0x7f2 01ve: tk bi1 01:29:28nn814819] warni14819", thre11Idon5001eapiticeapiticeapitStproduc 01ve: k bif rec01eapiticeapiticean.prp5001ctimcoceaf r, ns: priced: 1556
9507s: pricecursive lock ns: priceapie lock ns: priceproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 01:29:28nn814819] warni14819", thre11Idon5001eapiticeapiticeapitStproduc 01ve:
tk bif recurtStproduc 01ve: tk bif recursive lock ns: pr, t tk bif rec01eapiticeapiticean.prp5001ctimcoceaf r,.6950798, active: true, se31d: 1556950798, $pductiticeapiticean.prp500
1ctimcoceaf r,.6950798, active: true, se31d: 1556950798, active: true, se31d: 1556950798,  lock ns: pricecursive lock ns: priceapie lock ns: priceproductionsiiticeapiteadId: "0x7f2
 01ve: tk bi1 01:29:28nn814819] warni14819", thre11Idon50( ns: priced: 15569507s: pitStproduc 01ve: tk bif recursive lock ns: prursive lock ns: pr, t tk bif rec01eapiticersive lock
 ns: pr, t tk bif rec01eapiticersive lock ns: pr, t tk bif rec01eapiticersive lock ns: pr, t tk bif rec01eapiticersive lock ns: pr, t tk bif rec6lock ns: pr, t tced: 1556950798, ns
: priced: 1556950798, ns: priced: 1556950798, ns: priced:cock41riced: 1556950798, ns: priced: 1556950798, ns:8d: 155n814819] warni14819", thre11Id814819", thrpriceapi_8, active: tr
ue,6, ns: priced: 1556950798, ns:8d: 155n814819] warni14819r priceproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 01:29:28nn814819] warni14819", thre11Idon50( ns: priced: 15569507s
: pitStproduc 01ve: tk bif recursive lock ns: prursive lock ns: pr, t tk bif rec01eapiticersive lock ns: pr, t tk bif rec01eapiticersive lrsive lock ns: pr, t tk bif rec01eapiticer
sive lock ns: pr, t tk bif rec01eapiticersive lock ns: pr, t tk bif rec6lock ns: pr, t tced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798, ns: priced:cock41riced: 155
6950798, ns: priced: 1556950798, ns:6950798, ns: priced:cock41riced: 1556950798, ns: priced: 1556950798, ns:8d: 155n814819] warni14819", thre11Id814819", thrpriceapi_8, active: tru
e,6, ns: priced: 1556950798, ns:8d: 155n814819] warni14819r priceproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 02 01v.pr6tio821piticeapitead8i_production.proxies8eecursivtproduc
01ve: tk uiceapi_productitapi_peapduc 01ve: tk bif recursivtproduc 01ve: tk bif reecursivtproduc 01ve: tk uiceapi_production.pr211:29:28n5001eapitne8oductik18, actId8.prapiduction.
pr211:29:28n5 secmd$production.proxiepriceaf recursiepriceapi_productio priceapon.proxceapi_producti1on.pr211:29:28n5001eapitne8oductik18, actId8.prapiduction.pr211:29:28n5001eapit
08eapitne8oductik18, actId8.prapiduction.pr211:29:28n5001eapitne8oductik18, actId8.prapiducId814819", thrpriceapi_8, active: true,6, ns: priced: 1556950798, ns:8d: 155n814819] warn
i14819r priceproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 02 01v.pr6tio821piticeapitead8i_production.proxies8eecursivtproduc 01ve: tk uiceapi_productitapi_peapduc 01vduction.pro
xies8eecursivtproduc 01ve: tk uiceapi_productitapi_peapduc 01ve: tk biroduction.pr211:29:28n5001eapitne8oductik18, actId8.prapiduction.pr211:29:28n5 secmd$production.proxiepriceaf
recursiepriceapi_productio priceapon.proxceapi_producti1on.pr211:29:28n5001eapitne8oductik18, actId8.prapiduction.pr211:29:28n5001eapit08eapitne8odu8, active: true,6, ns: priced: 1
556950798, ns:8d: 155n814819] warni14819r priceproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 02 01v.pr6tio821piticeapitead8i_production.proxies8eecursivtproduc 01ve: tk uiceapi_p
roductitapi_peapduc 01ve: tk bif recursivtprod 1556950798, ns:8d: 155n814819] warni14819r priceproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 02 01v.pr6tio821piticeapitead8i_produ
ction.proxies8eecursivtproduc 01ve: tk uiceapi_productitapi_peapduc 01ve: tk bif recursivtprod 1556950798, ns:8d: 155n814819ceapie lock ns: pricecursive lock ns: priceapie lock ns:
 priceprriceaf recursiepriceapi_productio priceapon.proxceapi_produ: 1556950798, active: true, secm1ve:ruexiecti tx7f2 01ve: tk bifcursive lock ns: priceapi_produc_production.pron.
proxixies817p: { opitk ns:8d: 155n814819] warni14819r priceproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 02 01v.pr6tio821piticeapitead8i_production.proxies8eecursivtproduc 01ve:
tk uiceapi_productitapi_peapduc 01ve: tk bif recursivtproduc 01ve: tk bif reecursivtproduc 01ve: tk 21pi_productitapi_peapduc 01ve: tk bif recursivtprod 1556950798, ns:8d: 155n8148
19ceapie lock ns: pricecursive lock ns: priceapie lock ns: priceprriceaf recursiepriceapi_productio priceapon.proxceapi_produ: 1556950798, active: true, secm1ve:ruexiecti tx7f2 01v
oduction.proxiepriceapi_peapi_production.proxiepr8nn814854id: 5695n.proxn.prp5001ctimconn8148lkd: 15coc_production.pron.proxixies817p: { opitk bif recursivtproduc 01vproxies817p: {
 obif reproxiepriceapi_produ11If reciteadIdock e: true, secm1ve:rue, secmconn8oxiepriceapi_produ11If reciteadIdock e: true, secm1ve:rue,produ11If reciteadIdock e: true, secm1ve:rue
, secmconn8148i_pr rec01eap4eactive: d2d: 155695084tion.proxies817p: { opid: 1556950798, active1oeapiticeapiticeapitStproduc 01ve: tk bif recursive lock ns:c 01ve: tk bif recursive
 lock ns: prursive lock ns: pr, t tk bif rec01eapiticersive lock ns: priceapi4piticeapiticea loion.pr211:29:280ittk bif rec01eapiticeapiticea lockStproduc 01ve: tk bif recursive lo
ck ns: p treapitne8oductik18, actId8.prapiticeapiteace3c_production.pron.proxixies817p: { opitk bif recursivtproduc 01vproxies817p: { obif reproxiepriceapi_produ11If reciteadIdock
e: true, secm1ve:rue, secmconn8oxiepriceapi_produ11If reciteadIdock e: true, secm1ve:rue,produ11If reciteadIdock e: true, secm1ve:rue355695084tion.proxies817p: { opid: 1556950798,
active1oeapiticeapiticeapitStproduc 01ve: tk bif recursive lock ns:c 01ve: tk bif recursive lock ns: pr5ceapiticea lockStproduc 01ve: tk bif recursive lock ns: p treapitne8oductik1
8, actId8.prapiticeapiteace3c_production.pron.proxixies817p: { opitk bifapi4pns: priceapi4piticeapiticea loion.pr211:29:280ittk bif rec01eapiticeapiticea lockStproduc 01ve: tk bif
recursive lock ns: p treapitne8oductik18, actId8.prapiticeapiteace3c_production.pron.proxixies817p: { opitk bif recursivtproduc 01vproxies817p: { obif repracecursive lock n prteadI
d: ve: tk bi1 01:29:28nn814819] warni14819", thre11Id814819", thrpric active:pducti , acive:pducti , active:pducti , activericeapi_production.production.prp5001ctimconn81001ctimcon
n8148lkd: 1554id: 155695n.proxn.prp5001ctimconn8ck nso5001ctimconn8148lkd: 15coc_production.pron.proxixies817p: { opit pr4apitne8oductik18, actId8.prapiticeape31d: 1556950798, acti
ve: true, se31d: 1556950798, active: true, se31d: 1556950798, active: true, se31d: 1556950798, active: trueticean.prp5001ctimconn8ea: 155695n.proxn.prp5001ctimconn8ck ns: pri0n.eap
i4piticeapiticea loion.pr211:29:280ittk bif rec01eapiticeapiticea lockStproduc 01ve: tk bif recursive lock ns: p treapitne8oductik18, actId8o1:29:28nn814819] warni14819", thre11Id8
14819", thrpric active:pducti 9", thrpricproduction.production.prp5001ctimconn81001ctimconn8148lkd: 1554id: 155695n.proxn.56950798, ns: priced: 1556950798, ns: priced: 1556950798,
ns: priced: 1556950798, ns: priced: 1556950798, ns: priced: 1oduction.proxiepriceaf rock ns: pr4apitne8oductik18, actId8.prriceapon.proxceapi_production.proed: 1556950798, ns: pric
ed: 1556950798, ns: tive: true, se31d: 1556950798, active: true, se31d: 1556950798, active: trueticean.prp5001ctimconn8ea: 155695n.proxn.prp5001ctimconn8ck ns: pri0n.eapi4piticeapi
ticea loion.pr211:29:280ittk bif rec01eapiticeapiticea lockStproduc 01ve: tk bif recursive lock ns1d: 1556950798, $pductiticeapiticean.prp5001ctimcoceaf r,.6950798, active: true, s
e31d: 1556950798, active: true, se31d: 1556950798,  lock ns: pricecursive2_oxn.prp5001ctimconn8ck ns: pri0n.eapi4piticeapiticea loion.pr211:29:280ittk bif rec01eapiticeapiticea loc
kStproduc 01ve: tk bif recursive lock ns: p treapitne8oductik18, actId8o1:29:28nn814819] warni14819", thre11Id814819", thrpric active:pducti 9", thrpricponn8:ced: 1556950798, ns: p
riced: 1556950798, ns:8d: 155n814819] warni14819adIdock n: 1556950798, active: true, secm1ve:ruexiectpriceapi_productio pricapi17p: { opid: 1556ive lock ns: priceapi_production.pro
n.proxies817p: { opid: 1556950798, active: true, secursive2_oxn.prp5001ctimconn8ck ns: pri0n.eapi4piticeapiticea loion.pr211:29:280ittk bif rec01eapiticeapiticea lockStproduc 01ve:
 tk bif recursive lock ns: p treapitne8oductik18, actId8o1:29:28nn814819] warni14819", thre11Id814819", thrpric active:pducti 9",i14819", thre11Id29:28nn814819]ve: tk bif recursive
 lock ns: pr, t tk bif rec01eapiticeapiti0ceapi_ppriceapie lock ns: priceproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 01:29:28nn814cpi_ctioneapi_production.proxiepri_pre: tk bi1
 01:29:28nn814819] warni148 ns:8d: 155n814819] warni14819adIdock n: 1556950798, active: true, secm1ve:ruexiectpriceapi_productio pricapi17p: { opid: 1556ive lock ns: priceapi_produ
ction.pron.proxies817p: { opid: 1556950798, active: true, secursive2_oxn.prp5001ctimconn8ck ns: pri0n.eapctimconn8ck ns: pri0n.eapctimconn8ck ns: pri0n.eapctimconn8ck ns: pre: tk b
if recursive lock ns: pr, t tk bif rec01eapiticeapiti0ceapi_ppriceapie lock ns: priceproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 01:29:28nn814cpi_ctioneapi_production.proxiepri
_pre: tk bi1 01:29:28nn814819] wpi_productio pricapi17p: { opid: 1556ive lock ns: priceapi_production.pron.proxies817p: { opid: 1556950798, active: true, secursive2_oxn98, active:
true, secursive2_oxn98, active: true, secursive2_oxn98, active: true, secursive2_oxn98, active: true, secursive2_oxn98, active: true, secursive2_oxn98, active: true, secursive2_oxn
98, active: true, secursive2_oxn98, active: true, secursive2_oxn98001ctimconn8ck ns: pri0n.eapctimconn8ck ns: pri0n.eapctimconn8ck ns: pri0n.eapctimconn8ck ns: pre: tk bif recursiv
e lock ns: pr, t tk bif rec01eapiticeapiti0ceapi_ppriceapie lock ns: priceproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 01:29:28nn814cpi_ctionock ns: pr, t tk bif rec01eapiticeap
iticean.prp5001ctimcoceaf r,.6950798, active: true, se31d: 1556950798, active: true, se31d: 1556950798,  lock ns: pricecursive lock ns: priceapie lock ns: priceproductionsiiticeapi
teadId: "0x7f2 01ve: tk bi1 01:29:28nn8148id: 1556950798, active: true, secursive2_oxn98, active: true, secursive2_oxn98, active: true, secursive2_oxn98, active: true, secursive2_o
xn98, active: true, secursive2_oxn98, active: true, secursive2_oxn98, active: true, secursive2_oxn98, active: true, secuproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 01:29:28nn81
4819] warni14819", thre11Idon50( ns: priced: 15569507s: pitStproduc 01ve: tk bif recursive lock ns: prursive lock ns: pr, t tk bif rec01eapiticersive lock ns: pr, t tk bif rec01eap
iticersive lrsive r, t tk bif rec01eapiticeapiticean.prp5001ctimcoceaf r,.6950798, active: true, se31d: 1556950798, active: true, se31d: 1556950798,  lock ns: pricecursive lock ns:
 priceapie lock ns: priceproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 01:29:28nn8148id: 15569cursive2_oxn98, active: true, secursive2_oxn98, active: true, secursive2_oxn98, acti
ve: true, secuproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 01:29:28nn814819] warni14819", thre11Idon50( ns: priced: 15569507s: pitStproduc 01ve: tk bif recursive lock ns: cursiv
e lock ns: prursive lock ns: pr, t tk bif rec01eapiticersive lock ns: pr, t tk bif rec01eapiticersive lrsive r, t tk bif rec01eapiticeapiticean.prp5001ctimcoceaf r,.6950798, active
: true, se31d: 155695productionsiiticeapiteadId: "0x7f2 01ve: tk bi1 01:29:28nn8148id: 15569cursive2_oxn98, active: true, secursive2_oxn98, active: true, secursive2_oxn98, active:
true, secuproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 01:29:28nn814819] warni14819", thre11Idon tk bi1 01:29:28nn814819] warni14819", thre11Idon5001eapiticeapiticeapitStproduc
01ve: k bif rec01eapiticeapiticean.prp5001ctimcoceaf r, ns: priced: 15569507s: pricecursive lock ns: priceapie lock ns: priceproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 01:2tic
ean.prp5001ctimcoceaf r,.6950798, active: true, se31d: 155695productionsiiock ns: pr, t tk bif rec6lock ns: pr, t tced: 1556950798, ns: priced: 1556950798, ns: priced: 1556950798,
ns: priced:cock41riced: 1556950798,14819]ve: tk bif recursive lock ns: pr, t tk bif rec01eapiticeapiti0ceapi_ppriceapie lock ns: priceproductionsiiticeapiteadId: "0x7f2 01ve: tk bi
1 01:2k bif rec01eapiticersive lrsive lock ns: pr, t tk bif rec01eapiticersive lock ns: pr, t tk bif rec01eapiticersive lock ns: pr, t tk bif rec6lock ns: pr, t tced: 1556950798, n
s: priced: 1556950798, ns: priced: 1556950798, ns: priced:cock41riced: 1556950798, nive lock ns: p treapitne8oductik18, actId8o1:29:28nn814819] warni14819", thre11Id814819", thrpri
c active:pducti , acive:pducti , active:pducti , active: pr, t tk bif rec01eapiticeapiti0ceapi_producti ,6i_pre: tk bi1 01:29:28nn814819] warni14819", thre11Id29:s: priced: 1556950
798, ns: priced: 1556950798, ns: priced:cock41riced: 1556950798,14819]ve: tk bif recursive lock ns: pr, t tk bif rec01eapiticeapiti0ceapi_ppriceapie lock ns: priceproductionsiitice
apiteadId: "0x7f2 01ve: tk bi1 01:2k bif rec01eapiticersive priceapi_productionsive lock ns: priceapi_prod:p 29 0117p: { opiecursive lock ns: pr4apitne8oductik18, actId8.prapiticea
piteu5001ctimconn8148lkd: 15coc_production.pron.proxixies817i_producti ,6i_proiceapricecursive lock ns: priceapie lock ns: 98apiticeapiti0 lock ns: priceproductionsiiticeapiteadId:
 "0x7f2 01ve: tk bi1 01:29:28nn814819] warni14819", thre11Idon5001eapiticeapiticeapitStproduc 01ve: tk bif recursive lock ns: pr, t tk bif rec01eapiticeapiticean.prp5001ctimcoceaf
r,.6950798, active: true, se31d: 155iced:cock41riced: 1556950798,14819]ve: tk bif recursive lock ns: pr, t tk bif rec01eapiticeapiti0ceapi_ppriceapie lock ns: priceproductionsiitic
_ppriceapie lock ns: priceproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 01:2k bif rec01eapiticersive priceapi_productionsive lock ns: priceapi_prod:p 29 0117p: { opiecursive lock
 ns: pr4apitne8oductik18, actId8.prapiticeapiteu5001ctimconn8148lkd: 15coc_prooice secmd$production.proxiepriceaf recursiepriceapi_productio priceapon.proxceapi_producti1on.pr211:2
9:28n5001eapitne8oductik18, actId8.prapiduction.pr211e8oductik18, actId8.prriceapon.proxceapi_production.proed: 1556950798, ns: priced: 1556950798, ns: priced: 15roed: 1556950798,
ns: priced: 15561556950798, ns: priced:cock41riced: 1556950798, ns: priced: 1556950798, ns:8d: 155n814819] warni14819", thre11Id814i1 01:2k bif rec01eapiticersive priceapiwarni1481
9", thre11Id814819", thrpric active:pducti , acive:pducti , active:pducti , activericeapi_production.production.prp5001ctimconn81001ctimconn8ion.proxiepriceaf recursiepriceapi_prod
uctio priceapon.proxceapi_producti1on.pr211:29:28n5001eapitne8oductik18, actId8.prapiduction.pr211e8oductik18, actId8.prriceapon.proxceapi_production.proed: 1556950798, ns: priced:
 1556950798, ns: priced: 15roed: 155695, ns: priced: 15roed: 1556950798, ns: priced: 15561556950798, ns: priced:cock41riced: 1556950798, ns: priced: 1556950798, ns:8d: 155n814819]
warni14819ns:8d: 155n814819] warni14819ns:8d: 155n814819] warni14819ns:8d: 155n814819] warni14819ns:8d: 155n814819] warni14819ns:8d: 155n814819] warni14819ns:8d: 155n814819] warni1
4819ns:8d: 155n814819] warni14819ns:8d: 155n814819] warni14819ns:8d: 155n814819] warni14ve:pducti , active:pducti , activericeapi_production.production.prp5001ctimconn81001ctimconn
8ion.proxiepriceaf recursiepriceapi_productio priceapon.proxceapi_producti1on.pr211:29:28n5001eapitne8oductik18, actId8.prapiduction.pr211e8oductik18, actId8.prriceap155n814819] wa
rni14819ns:8d: 155n814819] warni14819ns:8d: 155n814819] warni14819ns:8d: 155n814819] warni14ve:pducti , active:pducti , activericeapi_production.production.prp5001ctimconn81001ctim
conn8ion.proxiepriceaf recursiepriceapi_productio priceapon.prox556950798, ns: priced: 1556950798, ns:8d: 155n814819] warni14819ns:8d: 155n814819] warni14819ns:8d: 155n814819] warn
i14819ns:8d: 155n41riced: 1556950798, ns: priced: 1556950798, ns:8d: 155n814819] warni14819", thre11Id814819", thrpriceapi_8, active: true,6, ns: priced: 1556950798, ns:8d: 155n814
819] warni14819r priceproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 02 01v.pr6tio821piticeapioproed: 1556950798, ns: priced: 1556950798, ns: tive: true, se31d: 1556950798, active
: true, se31d: 1556950798, active: trueticean.prp5001ctimconn8ea: 155695n.proxn.prp5001ctimconn8ck ns: pri0n.eapi4piticeapiticea loion.pr211:29:280ittk bif rec01eapiticeapitioxiepr
iceaf recursiepriceapi_productio priceapon.prox556950798, ns: priced: 1556950798, ns:8d: 155n814819] warni14819ns:8d: 155n814819] warni14819ns:8d: 155n814819] warni14819ns:8d: 155n
41riced: 1556950798, ns: priced: 1556950798, ns:8d: 155n814819] warni1480798, ns:8d: 155n814819] warni14819ns:8d: 155n814819] warni14819ns:8d: 155n814819] warni14819ns:8d: 155n41ri
ced: 1556950798, ns: priced: 1556950798, ns:8d: 155n814819] warni14819", thre11Id814819", thrpriceapi_8, active: true,6, ns: priced: 1556950798, ns:8d4d: 1556950798, active: true,
se31d: 1556950798, active: trueticean.prp50ckStproduc 01ve: tk bif recursive lock ns: p treapitne8oductik18, actId8.prapiticeapiteace3c_production.pron.proxixies817p: { opitk bif r
ecursivtproduc 01vproxies817p: { obif repracecursive lock n prteadId: ve: tk bi1 01:29:28nn814819] warni14819", thre11Id8oduc 01ve: tk bif recursive lock ns: p treapitne8oductik18,
 actId8.prapiticeapiteace3c_priti0ceapi_producti ,6i_pre: tk bi1 01:29:28nn814819] warni14819", thre11Id29:28nn814819]ve: tk bif recursive lock ns: pr, t tk bif rec01eapiticeapiti0
ceapi_ppriceapie lock ns: priceproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 01:29:28nn814cpi_ctioneapi_prne8oductik18, actId8o1:29:28nn814819] warni14819", thre11Id814819", thrp
ric active:pducti , acive:pducti , active:pducti , active: pr, t tk bif rec01eapiticnn814819] warni14819", thre11Id8oduc 01ve: tk bif recursive lock ns: p treapitne8oductik18, actI
d8.prapiticeapiteace3c_priti0ceapi_producti ,6i_pre: tk bi1 01:29:28nn814819] warni14819", thre11Id29:28nn814819]ve: tk bif recursive lock ns: pr, t tk bif rec01e9]ve: tk7 tk bif r
ecursive lock ns: p treapitne8oductik18, actId8.prapiticeapiteace3c_priti0ceapi_producti ,6i_pre: tk bi1 01:29:28nn814819] warni14819", thre11Id29:28nn814819]ve: tk bif recursive l
ock ns: pr, t tk bif rec01eapiticeapiti0ceapi_ppriceapie lock ns: pri4819] warni14819", thre11Id814819", thrpric active:pducti , acive:pducti , active:pducti , actirec01eapiticnn81
4819] warni14819", thre11Id8oduc 01ve: tk bif recursive lock ns: p treapitne8oductik18, actId8.prapiticeapiteace3c_priti0ceapi_producti ,6i_pre: tpriceapi_productio p0recursivtprod
uc 01vproxies817p: { obif reproxiepriceapi_produ11If reciteadIdock e: true, secm1ve:rue, secmconn8oxiepriceapi_produ11If reciteadIdock e: true, secm1ve:rue,produ11If reciteadIdock
e: true, secm1ve:rue355695084tion.proxies817p: { opid: 1556950798ns: priceapie lock ns: priceprriceaf recursiepriceapi_productio priceapon.proxceapi_produ: 1556950798, active: true
, secm1ve:ruexiecti tx7f2 01voduction.proxiepriceapi_peapi_production.proxiepr8nn814854id: 5695nlock ns: pri4819] warni14819", thre11Id814819", thrpric active:pducti , acive:pducti
 , active:pducti , actirec01eapiticnn814819] warni14819", thre11Id8oduc 01ve: tk bif recursive lock ns: p treapitne8oductik18, actId8.prapiticeapiteace3c_priti0ceapi_productiv4819"
, thrpric active:pducti 9", thrpricponn8:ced: 1556950798, ns: priced: 1556950798, ns:8d: 155n814819] warni14819adIdock n: 1556950798, active: true, secm1ve:ruexiectpriceapi_product
io pricapi17p: { opid: 1556ive lock ns: priceapi_production.pron.proxies8ioneapi_prne8oductik18, actId8o1:29:28nn814819] warni14819", thre11Id814819", thrpric active:pducti , acive
:pducti , active:pducti , active: pr, t tk bif rec01eapiticnn814819] warni14819", thre11Id8oduc 01ve: tk bif recurs 01ve: tk bi1 01:29:28nn814819] warni14819", thre11Idon5001eapiti
ceapiticeapitStproduc 01ve: tk bif recurtStproduc 01ve: tk bif recursive lock ns: pr, t tk bif 41thre11Idon50( ns: priced: 15569507s: pitStproduc 01ve: tk bif recursive lock ns: cu
rsive lock ns: prursive lock ns: pr, t tk bif rec01eapiticersive lock ns: pr, t tk bif rec01eapiticersive lrsive r, t tk bif rec01eapiticeapiticean.prp5001ctimcoceaf r,.6950798, ac
tive: true, secmd$production.proxiepriceafl556950798, active: true, se31d: 1556950798, active: true, se31d: 1556950798,  lock ns: pricecursive lock ns: priceapie lock ns: priceprod
uctionsiiticeapiteadId: "0x7f2 01ve: tk bi1 01:29:28nn814819] warni14819"bif rec01eapiticnn814819] warni14819", thre11Id8oduc 01ve: tk bif recurs 01ve: tk bi1 01:29:28nn814819] war
ni14819", thre11Idon5001eapiticeapiticeapitStproduc 01ve: tk bif recurtStproduc 01ve: tk bif recermconne5569cursive2_oxn98, active: true, secursive2_oxn98, active: true, secursive2
_oxn98, active: true, secuproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 01se: tk bi1 01se: tk bi1 01se: tk bi1 01se: tk bi1 01se: tk bi1 01se: tk bi1 01se: tk bi1 01se: tk bi1 01
se: tk bi1 01se: tk bi1 01se: tk bi1 01se: tk bi1 01se: tk bi1 01se: tk bi1 01se: tk bi1 01se: tk bi1 01se: tk bi1 01se: tk bi1 01se: tk bi1 01se: tk bi1 0 icnn814819] warni14819",
 thre11Id8oduc 01ve: tk bif recurs 01ve: tk bi1 01:29:28nn814819] warni14819", thre11Idon5001eapiticeapiticeapitStproduc 01ve: tk bif recurtStproduc 01ve: tk bif recermconne5569cur
sive2_oxn98, active: true, secursive2_oxn98, active: t:14819r priceproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 02 01v.pr6tio821piticeapitead8i_production.proxies8eecursivtprodu
c 01ve: tk uiceapi_productitapi_peapd:, active: true, secuproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 01se: tk bi1 01se: tk bi1 01se: tk bi1 01se: tk bi1 01se: tk bi1 01se: tk
bi1 01se: tk bi1 01se: tk bi1 01se: tk bi1 01se: tk bi1 01se: tk bi1 01se: tk bi1 01se: tk bi1 01se: tk bi1 01se: tkapiticeapitStproduc 01ve: k bif rec01eapiticeapiticean.prp5001ct
imcoceaf r, ns: priced: 15569507s: pricecursive lock ns: priceapie lock ns: priceproductionsiiticeapiteadId: "0x7f2 01ve: tk bi1 01:2ticean.prp5001ctimcoceaf r,.6950798, active: tr
ue, se31d: 155sive2_oxn98, active: true, secursive2_oxn98, active: true, secursive2_oxn98, active: true, secursive2_oxn98, active: true, secursive2_oxn98, actron.proxixies817p: { o
pit pr4apitne8oductik18, actId8.prapiticeape31d: 1556950798, active: true, se31d: 1556950798, active: true, se31d: 1556950798, active: true, se31d: 1556950798, active: trueticean.p
rp5001ctimconn8ea: 155695n.proxn.prp5001ctimconn8ck ns: piticeapiticean.prp5001ctimcoceaf r, ns: priced: 15569507s: pricecursive lock ns: priceap798, active: true, se31d: 155695079..... 

Here is the status I saw from the secondary

Wed Oct  1 15:12:01.538 [rsBackgroundSync] replSet db exception in producer: 13106 nextSafe(): { $err: "BSONObj size: 1207454100 (0x9449F847) is invalid. Size must be between 0 and
 16793600(16MB) First element: 8c·UsEA♣↨<94I:8a¶,88Fû&82959d♀_Dw9bùè80
£@?I♣[...", code: 10334 }

In order to recover I failed over to the other node of the replica set, removed all the data on this node and did a full resync. Let me know what data you need to identify the issue.



 Comments   
Comment by Ramon Fernandez Marina [ 14/Nov/14 ]

dharshanr@scalegrid.net, we haven't heard back from you for some time so we're going to close this ticket. If you feel there's any further information you can provide that helps investigate this issue feel free to re-open the ticket and upload such information.

Regards,
Ramón.

Comment by Ramon Fernandez Marina [ 17/Oct/14 ]

dharshanr@scalegrid.net, this could be two separate issues, but neither of them with enough information to investigate. In the log line it seems as if multiple processes were writing to the same log file, is that the case? Are you logging via syslog and/or over the network (including NFS) perhaps?

As for the BSON error on the secondary, it seems the replication thread found corrupt data. This data corruption could have come from a flaky network, or it could also be due to defective memory on this particular secondary, so I'd recommend checking the memory on this node as this issue may happen again if it's defective. I'd also suggest checking your network and the network card on this node, as these are not uncommon sources of BSON corruption.

This being said, there have been numerous improvements in replication and other areas since 2.4.1, so I'd strongly encourage you to upgrade to 2.4.12 at your earliest convenience.

Unless there's a way to reproduce this issue I'm not sure there's anything we can do to understand what triggered this behavior. Is there any further information you can provide that can help us understand this issue?

Thanks,
Ramón.

Generated at Thu Feb 08 03:38:11 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.