Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2114

Improve fairness of application eviction

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT2.7.0
    • Labels:
      None
    • # Replies:
      5
    • Last comment by Customer:
      true

      Description

      The current way an application thread contributes to eviction when the cache is full can be unfair. Each thread will evict up to 10 pages, if a thread keeps failing to evict (for whatever reason) it will keep waiting.

      It makes more sense to track global evictions, rather than evictions completed by an individual thread.

        Issue Links

          Activity

          Hide
          wangy wangy added a comment -

          Everybody is good:
          I am from China,Is using the mongodb3.0.2 .
          I met a very confused error exception,The error log shape such as:
          2015-09-18T16:26:11.159+0800 I NETWORK [ReplicaSetMonitorWatcher] Socket recv() timeout 172.31.0.121:27017
          2015-09-18T16:26:11.159+0800 I NETWORK [ReplicaSetMonitorWatcher] SocketException: remote: 172.31.0.121:27017 error: 9001 socket exception [RECV_TIMEOUT] server [172.31.0.121:27017]
          2015-09-18T16:26:11.159+0800 I NETWORK [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed
          2015-09-18T16:26:11.160+0800 I NETWORK [ReplicaSetMonitorWatcher] Detected bad connection created at 1442481272820041 microSec, clearing pool for 172.31.0.121:27017 of 0 connections
          2015-09-18T16:26:11.160+0800 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set rs1
          2015-09-18T16:26:13.878+0800 I NETWORK [ReplExecNetThread-271] Socket recv() timeout 172.31.0.124:27018
          2015-09-18T16:26:17.872+0800 I NETWORK [initandlisten] connection accepted from 172.31.0.121:35709 #6985 (42 connections now open)
          2015-09-18T16:26:17.921+0800 I NETWORK [conn6983] end connection 172.31.0.124:33222 (40 connections now open)
          2015-09-18T16:26:18.323+0800 I NETWORK [ReplExecNetThread-271] SocketException: remote: 172.31.0.124:27018 error: 9001 socket exception [RECV_TIMEOUT] server [172.31.0.124:27018]
          2015-09-18T16:26:18.323+0800 I NETWORK [initandlisten] connection accepted from 172.31.0.124:33224 #6986 (43 connections now open)
          2015-09-18T16:26:18.323+0800 I NETWORK [ReplExecNetThread-271] DBClientCursor::init call() failed
          2015-09-18T16:26:18.324+0800 I REPL [ReplicationExecutor] Error in heartbeat request to 172.31.0.124:27018; Location10276 DBClientBase::findN: transport error: 172.31.0.124:27018 ns: admin.$cmd query:

          { replSetHeartbeat: "rs2", pv: 1, v: 1, from: "172.31.0.123:27018", fromId: 0, checkEmpty: false }

          I can't solve this error, when the server is a lot of pressure, appear very frequent,And a very big effect.
          The main cause is the effect of master-slave switching,Or can't detect the heart,Cause my Java programs written to failure.

          Some error log:
          com.mongodb.WriteConcernException: Write failed with error code 8 and error message 'not master'
          at com.mongodb.operation.BaseWriteOperation.convertBulkWriteException(BaseWriteOperation.java:207)
          at com.mongodb.operation.BaseWriteOperation.access$200(BaseWriteOperation.java:58)
          at com.mongodb.operation.BaseWriteOperation$1.call(BaseWriteOperation.java:116)
          at com.mongodb.operation.BaseWriteOperation$1.call(BaseWriteOperation.java:106)
          at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:186)
          at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:177)
          at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:106)
          at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:58)
          at com.mongodb.Mongo.execute(Mongo.java:745)
          at com.mongodb.Mongo$2.execute(Mongo.java:728)
          at com.mongodb.DBCollection.executeWriteOperation(DBCollection.java:327)
          at com.mongodb.DBCollection.insert(DBCollection.java:323)
          at com.mongodb.DBCollection.insert(DBCollection.java:314)
          at com.mongodb.DBCollection.insert(DBCollection.java:284)
          at com.mongodb.DBCollection.insert(DBCollection.java:250)
          at com.mongodb.DBCollection.insert(DBCollection.java:187)
          at build.BuildData.main(BuildData.java:77)

          or

          com.mongodb.WriteConcernException: Write failed with error code 83 and error message 'write results unavailable from 172.16.130.11:27028 :: caused by :: Location17255 error receiving write command response, possible socket exception - see logs'
          at com.mongodb.operation.BaseWriteOperation.convertBulkWriteException(BaseWriteOperation.java:207)
          at com.mongodb.operation.BaseWriteOperation.access$200(BaseWriteOperation.java:58)
          at com.mongodb.operation.BaseWriteOperation$1.call(BaseWriteOperation.java:116)
          at com.mongodb.operation.BaseWriteOperation$1.call(BaseWriteOperation.java:106)
          at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:186)
          at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:177)
          at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:106)
          at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:58)
          at com.mongodb.Mongo.execute(Mongo.java:745)
          at com.mongodb.Mongo$2.execute(Mongo.java:728)
          at com.mongodb.DBCollection.executeWriteOperation(DBCollection.java:327)
          at com.mongodb.DBCollection.insert(DBCollection.java:323)
          at com.mongodb.DBCollection.insert(DBCollection.java:314)
          at com.mongodb.DBCollection.insert(DBCollection.java:284)
          at com.mongodb.DBCollection.insert(DBCollection.java:250)
          at com.mongodb.DBCollection.insert(DBCollection.java:187)
          at build.BuildData.main(BuildData.java:77)

          I check the Internet connection, and without any problems.
          3.0.6 version also do the test for new online, also can appear such problems.
          Hope problems can be solved, and give me some advice, I thank you very much.

          My English is not standard, please forgive me.

          If you have more want to know, please contact me.

          Email address:
          18996262637@163.com
          I am in China.

          thanks
          Beijing time:2015-09-18 17:00:00

          Show
          wangy wangy added a comment - Everybody is good: I am from China,Is using the mongodb3.0.2 . I met a very confused error exception,The error log shape such as: 2015-09-18T16:26:11.159+0800 I NETWORK [ReplicaSetMonitorWatcher] Socket recv() timeout 172.31.0.121:27017 2015-09-18T16:26:11.159+0800 I NETWORK [ReplicaSetMonitorWatcher] SocketException: remote: 172.31.0.121:27017 error: 9001 socket exception [RECV_TIMEOUT] server [172.31.0.121:27017] 2015-09-18T16:26:11.159+0800 I NETWORK [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed 2015-09-18T16:26:11.160+0800 I NETWORK [ReplicaSetMonitorWatcher] Detected bad connection created at 1442481272820041 microSec, clearing pool for 172.31.0.121:27017 of 0 connections 2015-09-18T16:26:11.160+0800 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set rs1 2015-09-18T16:26:13.878+0800 I NETWORK [ReplExecNetThread-271] Socket recv() timeout 172.31.0.124:27018 2015-09-18T16:26:17.872+0800 I NETWORK [initandlisten] connection accepted from 172.31.0.121:35709 #6985 (42 connections now open) 2015-09-18T16:26:17.921+0800 I NETWORK [conn6983] end connection 172.31.0.124:33222 (40 connections now open) 2015-09-18T16:26:18.323+0800 I NETWORK [ReplExecNetThread-271] SocketException: remote: 172.31.0.124:27018 error: 9001 socket exception [RECV_TIMEOUT] server [172.31.0.124:27018] 2015-09-18T16:26:18.323+0800 I NETWORK [initandlisten] connection accepted from 172.31.0.124:33224 #6986 (43 connections now open) 2015-09-18T16:26:18.323+0800 I NETWORK [ReplExecNetThread-271] DBClientCursor::init call() failed 2015-09-18T16:26:18.324+0800 I REPL [ReplicationExecutor] Error in heartbeat request to 172.31.0.124:27018; Location10276 DBClientBase::findN: transport error: 172.31.0.124:27018 ns: admin.$cmd query: { replSetHeartbeat: "rs2", pv: 1, v: 1, from: "172.31.0.123:27018", fromId: 0, checkEmpty: false } I can't solve this error, when the server is a lot of pressure, appear very frequent,And a very big effect. The main cause is the effect of master-slave switching,Or can't detect the heart,Cause my Java programs written to failure. Some error log: com.mongodb.WriteConcernException: Write failed with error code 8 and error message 'not master' at com.mongodb.operation.BaseWriteOperation.convertBulkWriteException(BaseWriteOperation.java:207) at com.mongodb.operation.BaseWriteOperation.access$200(BaseWriteOperation.java:58) at com.mongodb.operation.BaseWriteOperation$1.call(BaseWriteOperation.java:116) at com.mongodb.operation.BaseWriteOperation$1.call(BaseWriteOperation.java:106) at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:186) at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:177) at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:106) at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:58) at com.mongodb.Mongo.execute(Mongo.java:745) at com.mongodb.Mongo$2.execute(Mongo.java:728) at com.mongodb.DBCollection.executeWriteOperation(DBCollection.java:327) at com.mongodb.DBCollection.insert(DBCollection.java:323) at com.mongodb.DBCollection.insert(DBCollection.java:314) at com.mongodb.DBCollection.insert(DBCollection.java:284) at com.mongodb.DBCollection.insert(DBCollection.java:250) at com.mongodb.DBCollection.insert(DBCollection.java:187) at build.BuildData.main(BuildData.java:77) or com.mongodb.WriteConcernException: Write failed with error code 83 and error message 'write results unavailable from 172.16.130.11:27028 :: caused by :: Location17255 error receiving write command response, possible socket exception - see logs' at com.mongodb.operation.BaseWriteOperation.convertBulkWriteException(BaseWriteOperation.java:207) at com.mongodb.operation.BaseWriteOperation.access$200(BaseWriteOperation.java:58) at com.mongodb.operation.BaseWriteOperation$1.call(BaseWriteOperation.java:116) at com.mongodb.operation.BaseWriteOperation$1.call(BaseWriteOperation.java:106) at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:186) at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:177) at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:106) at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:58) at com.mongodb.Mongo.execute(Mongo.java:745) at com.mongodb.Mongo$2.execute(Mongo.java:728) at com.mongodb.DBCollection.executeWriteOperation(DBCollection.java:327) at com.mongodb.DBCollection.insert(DBCollection.java:323) at com.mongodb.DBCollection.insert(DBCollection.java:314) at com.mongodb.DBCollection.insert(DBCollection.java:284) at com.mongodb.DBCollection.insert(DBCollection.java:250) at com.mongodb.DBCollection.insert(DBCollection.java:187) at build.BuildData.main(BuildData.java:77) I check the Internet connection, and without any problems. 3.0.6 version also do the test for new online, also can appear such problems. Hope problems can be solved, and give me some advice, I thank you very much. My English is not standard, please forgive me. If you have more want to know, please contact me. Email address: 18996262637@163.com I am in China. thanks Beijing time:2015-09-18 17:00:00
          Hide
          alexander.gorrod Alexander Gorrod added a comment -

          wangy This is the wrong place for your question. A better place is the MongoDB user group:
          https://groups.google.com/forum/#!forum/mongodb-user

          Please ask there instead.

          Show
          alexander.gorrod Alexander Gorrod added a comment - wangy This is the wrong place for your question. A better place is the MongoDB user group: https://groups.google.com/forum/#!forum/mongodb-user Please ask there instead.
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexg@wiredtiger.com'}

          Message: WT-2114 Update application eviction loop.

          Have it use a global evict count rather than a count of evictions
          done by this thread.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/9c7823141754c8099b784867d2dbed6f2dd28562

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexg@wiredtiger.com'} Message: WT-2114 Update application eviction loop. Have it use a global evict count rather than a count of evictions done by this thread. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/9c7823141754c8099b784867d2dbed6f2dd28562
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexg@wiredtiger.com'}

          Message: WT-2114 Fix a bug testing against wrong variable.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/30824b710849574b9d6644193d86f447fa2af3cc

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexg@wiredtiger.com'} Message: WT-2114 Fix a bug testing against wrong variable. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/30824b710849574b9d6644193d86f447fa2af3cc
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: Merge pull request #2203 from wiredtiger/app-evict-fairer

          WT-2114 Make application eviction fairer
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/dcb0ddb247d2e7ab0a8e082cfa784d889d3f00ae

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Merge pull request #2203 from wiredtiger/app-evict-fairer WT-2114 Make application eviction fairer Branch: develop https://github.com/wiredtiger/wiredtiger/commit/dcb0ddb247d2e7ab0a8e082cfa784d889d3f00ae

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since reply:
                1 year, 38 weeks, 2 days ago
                Date of 1st Reply: