Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-11320

Crash in mongodb secondary

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Critical - P2 Critical - P2
    • None
    • Affects Version/s: 2.4.1
    • Component/s: Stability
    • Labels:
      None
    • Linux

      I have a Replica set running on SSD environment. One of my secondaries crashed with the following message in the syslog. Is this a memory related crash?

      kernel: mongod[11538] general protection ip:e244b9 sp:7fa94522b2e0 error:0 in mongod[400000+10d9000]

      Here are the necessary traces
      Oct 22 10:51:29 SG-haystackdb-518 yum[1581]: Installed: mailcap-2.1.31-2.el6.noarch
      Oct 22 10:51:29 SG-haystackdb-518 yum[1581]: Installed: perl-libwww-perl-5.833-2.el6.noarch
      Oct 22 10:51:29 SG-haystackdb-518 yum[1581]: Installed: perl-XML-Parser-2.36-7.el6.x86_64
      Oct 22 10:51:29 SG-haystackdb-518 yum[1581]: Installed: hdparm-9.16-3.4.el6.x86_64
      Oct 22 10:51:33 SG-haystackdb-518 yum[1581]: Installed: munin-node-2.0.17-6.el6.noarch
      Oct 22 11:08:21 SG-haystackdb-518 kernel: hrtimer: interrupt took 6838768 ns
      Oct 22 11:17:57 SG-haystackdb-518 kernel: kswapd0: page allocation failure. order:1, mode:0x20
      Oct 22 11:17:57 SG-haystackdb-518 kernel: Pid: 58, comm: kswapd0 Not tainted 2.6.32-358.6.2.el6.x86_64 #1
      Oct 22 11:17:57 SG-haystackdb-518 kernel: Call Trace:
      Oct 22 11:17:57 SG-haystackdb-518 kernel: <IRQ> [<ffffffff8112c207>] ? __alloc_pages_nodemask+0x757/0x8d0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81166ab2>] ? kmem_getpages+0x62/0x170
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff811676ca>] ? fallback_alloc+0x1ba/0x270
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8116711f>] ? cache_grow+0x2cf/0x320
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81167449>] ? ____cache_alloc_node+0x99/0x160
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff811683cb>] ? kmem_cache_alloc+0x11b/0x190
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81439eb8>] ? sk_prot_alloc+0x48/0x1c0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8143af92>] ? sk_clone+0x22/0x2e0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81489ec6>] ? inet_csk_clone+0x16/0xd0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff814a2e13>] ? tcp_create_openreq_child+0x23/0x450
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff814a060d>] ? tcp_v4_syn_recv_sock+0x4d/0x310
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff814a2bb6>] ? tcp_check_req+0x226/0x460
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff814a00ab>] ? tcp_v4_do_rcv+0x35b/0x430
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffffa0213557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff814a18be>] ? tcp_v4_rcv+0x4fe/0x8d0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8147f590>] ? ip_local_deliver_finish+0x0/0x2d0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8147f66d>] ? ip_local_deliver_finish+0xdd/0x2d0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8147f8f8>] ? ip_local_deliver+0x98/0xa0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8147edbd>] ? ip_rcv_finish+0x12d/0x440
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8147f345>] ? ip_rcv+0x275/0x350
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff810912d1>] ? __queue_work+0x41/0x50
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8144851b>] ? __netif_receive_skb+0x4ab/0x750
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8144a8f8>] ? netif_receive_skb+0x58/0x60
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffffa00f055d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8103c7b8>] ? pvclock_clocksource_read+0x58/0xd0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8144d0c3>] ? net_rx_action+0x103/0x2f0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffffa00ef0b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81076fb1>] ? __do_softirq+0xc1/0x1e0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff810e1720>] ? handle_IRQ_event+0x60/0x170
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8107700f>] ? __do_softirq+0x11f/0x1e0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81076d95>] ? irq_exit+0x85/0x90
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81517145>] ? do_IRQ+0x75/0xf0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
      Oct 22 11:17:57 SG-haystackdb-518 kernel: <EOI> [<ffffffff8127769a>] ? find_next_bit+0x1a/0xb0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81277331>] ? cpumask_any_but+0x31/0x50
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8104c7e8>] ? flush_tlb_page+0x48/0xb0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8104b740>] ? ptep_clear_flush_young+0x50/0x70
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8114d544>] ? page_referenced_one+0xa4/0x1e0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8114df48>] ? page_referenced+0x148/0x360
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8113248d>] ? shrink_page_list.clone.3+0x17d/0x650
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8117086b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81132bf7>] ? isolate_lru_pages.clone.0+0xd7/0x170
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81133343>] ? shrink_inactive_list+0x343/0x830
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81133bde>] ? shrink_mem_cgroup_zone+0x3ae/0x610
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8117257d>] ? mem_cgroup_iter+0xfd/0x280
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81133ea3>] ? shrink_zone+0x63/0xb0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81135265>] ? balance_pgdat+0x705/0x820
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff811354b4>] ? kswapd+0x134/0x3c0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81096ca0>] ? autoremove_wake_function+0x0/0x40
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81135380>] ? kswapd+0x0/0x3c0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81096936>] ? kthread+0x96/0xa0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff810968a0>] ? kthread+0x0/0xa0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
      Oct 22 11:17:57 SG-haystackdb-518 kernel: kswapd0: page allocation failure. order:1, mode:0x20
      Oct 22 11:17:57 SG-haystackdb-518 kernel: Pid: 58, comm: kswapd0 Not tainted 2.6.32-358.6.2.el6.x86_64 #1
      Oct 22 11:17:57 SG-haystackdb-518 kernel: Call Trace:
      Oct 22 11:17:57 SG-haystackdb-518 kernel: <IRQ> [<ffffffff8112c207>] ? __alloc_pages_nodemask+0x757/0x8d0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81166ab2>] ? kmem_getpages+0x62/0x170
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff811676ca>] ? fallback_alloc+0x1ba/0x270
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8116711f>] ? cache_grow+0x2cf/0x320
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81167449>] ? ____cache_alloc_node+0x99/0x160
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff811683cb>] ? kmem_cache_alloc+0x11b/0x190
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81439eb8>] ? sk_prot_alloc+0x48/0x1c0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8143af92>] ? sk_clone+0x22/0x2e0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81489ec6>] ? inet_csk_clone+0x16/0xd0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff814a2e13>] ? tcp_create_openreq_child+0x23/0x450
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff814a060d>] ? tcp_v4_syn_recv_sock+0x4d/0x310
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff814a2bb6>] ? tcp_check_req+0x226/0x460
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff814a00ab>] ? tcp_v4_do_rcv+0x35b/0x430
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffffa0213557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff814a18be>] ? tcp_v4_rcv+0x4fe/0x8d0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8147f590>] ? ip_local_deliver_finish+0x0/0x2d0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8147f66d>] ? ip_local_deliver_finish+0xdd/0x2d0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8147f8f8>] ? ip_local_deliver+0x98/0xa0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8147edbd>] ? ip_rcv_finish+0x12d/0x440
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8147f345>] ? ip_rcv+0x275/0x350
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff810912d1>] ? __queue_work+0x41/0x50
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8144851b>] ? __netif_receive_skb+0x4ab/0x750
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8144a8f8>] ? netif_receive_skb+0x58/0x60
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffffa00f055d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8103c7b8>] ? pvclock_clocksource_read+0x58/0xd0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8144d0c3>] ? net_rx_action+0x103/0x2f0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffffa00ef0b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81076fb1>] ? __do_softirq+0xc1/0x1e0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff810e1720>] ? handle_IRQ_event+0x60/0x170
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8107700f>] ? __do_softirq+0x11f/0x1e0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81076d95>] ? irq_exit+0x85/0x90
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81517145>] ? do_IRQ+0x75/0xf0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
      Oct 22 11:17:57 SG-haystackdb-518 kernel: <EOI> [<ffffffff8127769a>] ? find_next_bit+0x1a/0xb0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81277331>] ? cpumask_any_but+0x31/0x50
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8104c7e8>] ? flush_tlb_page+0x48/0xb0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8104b740>] ? ptep_clear_flush_young+0x50/0x70
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8114d544>] ? page_referenced_one+0xa4/0x1e0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8114df48>] ? page_referenced+0x148/0x360
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8113248d>] ? shrink_page_list.clone.3+0x17d/0x650
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8117086b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81132bf7>] ? isolate_lru_pages.clone.0+0xd7/0x170
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81133343>] ? shrink_inactive_list+0x343/0x830
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81133bde>] ? shrink_mem_cgroup_zone+0x3ae/0x610
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8117257d>] ? mem_cgroup_iter+0xfd/0x280
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81133ea3>] ? shrink_zone+0x63/0xb0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81135265>] ? balance_pgdat+0x705/0x820
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff811354b4>] ? kswapd+0x134/0x3c0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81096ca0>] ? autoremove_wake_function+0x0/0x40
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81135380>] ? kswapd+0x0/0x3c0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81096936>] ? kthread+0x96/0xa0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff810968a0>] ? kthread+0x0/0xa0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
      Oct 22 11:17:57 SG-haystackdb-518 kernel: kswapd0: page allocation failure. order:1, mode:0x20
      Oct 22 11:17:57 SG-haystackdb-518 kernel: Pid: 58, comm: kswapd0 Not tainted 2.6.32-358.6.2.el6.x86_64 #1
      Oct 22 11:17:57 SG-haystackdb-518 kernel: Call Trace:
      Oct 22 11:17:57 SG-haystackdb-518 kernel: <IRQ> [<ffffffff8112c207>] ? __alloc_pages_nodemask+0x757/0x8d0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81166ab2>] ? kmem_getpages+0x62/0x170
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff811676ca>] ? fallback_alloc+0x1ba/0x270
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8116711f>] ? cache_grow+0x2cf/0x320
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81167449>] ? ____cache_alloc_node+0x99/0x160
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff811683cb>] ? kmem_cache_alloc+0x11b/0x190
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81439eb8>] ? sk_prot_alloc+0x48/0x1c0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8143af92>] ? sk_clone+0x22/0x2e0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81489ec6>] ? inet_csk_clone+0x16/0xd0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff814a2e13>] ? tcp_create_openreq_child+0x23/0x450
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff814a060d>] ? tcp_v4_syn_recv_sock+0x4d/0x310
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff814a2bb6>] ? tcp_check_req+0x226/0x460
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff814a00ab>] ? tcp_v4_do_rcv+0x35b/0x430
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffffa0213557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff814a18be>] ? tcp_v4_rcv+0x4fe/0x8d0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8147f590>] ? ip_local_deliver_finish+0x0/0x2d0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8147f66d>] ? ip_local_deliver_finish+0xdd/0x2d0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8147f8f8>] ? ip_local_deliver+0x98/0xa0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8147edbd>] ? ip_rcv_finish+0x12d/0x440
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8147f345>] ? ip_rcv+0x275/0x350
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff810912d1>] ? __queue_work+0x41/0x50
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8144851b>] ? __netif_receive_skb+0x4ab/0x750
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8144a8f8>] ? netif_receive_skb+0x58/0x60
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffffa00f055d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8103c7b8>] ? pvclock_clocksource_read+0x58/0xd0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8144d0c3>] ? net_rx_action+0x103/0x2f0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffffa00ef0b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81076fb1>] ? __do_softirq+0xc1/0x1e0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff810e1720>] ? handle_IRQ_event+0x60/0x170
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8107700f>] ? __do_softirq+0x11f/0x1e0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81076d95>] ? irq_exit+0x85/0x90
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81517145>] ? do_IRQ+0x75/0xf0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
      Oct 22 11:17:57 SG-haystackdb-518 kernel: <EOI> [<ffffffff8104c64a>] ? flush_tlb_others_ipi+0x11a/0x130
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8104c6d6>] ? native_flush_tlb_others+0x76/0x90
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8104c7fe>] ? flush_tlb_page+0x5e/0xb0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8104b740>] ? ptep_clear_flush_young+0x50/0x70
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8114d544>] ? page_referenced_one+0xa4/0x1e0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8114df48>] ? page_referenced+0x148/0x360
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8113248d>] ? shrink_page_list.clone.3+0x17d/0x650
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8117086b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81132bf7>] ? isolate_lru_pages.clone.0+0xd7/0x170
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81133343>] ? shrink_inactive_list+0x343/0x830
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8112d04a>] ? determine_dirtyable_memory+0x1a/0x30
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8112d0f7>] ? get_dirty_limits+0x27/0x2f0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81133bde>] ? shrink_mem_cgroup_zone+0x3ae/0x610
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8117257d>] ? mem_cgroup_iter+0xfd/0x280
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81133ea3>] ? shrink_zone+0x63/0xb0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81135265>] ? balance_pgdat+0x705/0x820
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff811354b4>] ? kswapd+0x134/0x3c0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81096ca0>] ? autoremove_wake_function+0x0/0x40
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81135380>] ? kswapd+0x0/0x3c0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff81096936>] ? kthread+0x96/0xa0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff810968a0>] ? kthread+0x0/0xa0
      Oct 22 11:17:57 SG-haystackdb-518 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
      Oct 22 11:22:52 SG-haystackdb-518 kernel: swapper: page allocation failure. order:1, mode:0x20
      Oct 22 11:22:52 SG-haystackdb-518 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-358.6.2.el6.x86_64 #1
      Oct 22 11:22:52 SG-haystackdb-518 kernel: Call Trace:
      Oct 22 11:22:52 SG-haystackdb-518 kernel: <IRQ> [<ffffffff8112c207>] ? __alloc_pages_nodemask+0x757/0x8d0
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff81166ab2>] ? kmem_getpages+0x62/0x170
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff811676ca>] ? fallback_alloc+0x1ba/0x270
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff8116711f>] ? cache_grow+0x2cf/0x320
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff81167449>] ? ____cache_alloc_node+0x99/0x160
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff811683cb>] ? kmem_cache_alloc+0x11b/0x190
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff81439eb8>] ? sk_prot_alloc+0x48/0x1c0
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff8143af92>] ? sk_clone+0x22/0x2e0
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff81489ec6>] ? inet_csk_clone+0x16/0xd0
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff814a2e13>] ? tcp_create_openreq_child+0x23/0x450
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff814a060d>] ? tcp_v4_syn_recv_sock+0x4d/0x310
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff814a2bb6>] ? tcp_check_req+0x226/0x460
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff814a00ab>] ? tcp_v4_do_rcv+0x35b/0x430
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffffa0213557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff814a18be>] ? tcp_v4_rcv+0x4fe/0x8d0
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff8147f590>] ? ip_local_deliver_finish+0x0/0x2d0
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff8147f66d>] ? ip_local_deliver_finish+0xdd/0x2d0
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff8147f8f8>] ? ip_local_deliver+0x98/0xa0
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff8147edbd>] ? ip_rcv_finish+0x12d/0x440
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff8147f345>] ? ip_rcv+0x275/0x350
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff8144851b>] ? __netif_receive_skb+0x4ab/0x750
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff8144a8f8>] ? netif_receive_skb+0x58/0x60
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffffa00f055d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff81255203>] ? elv_queue_empty+0x33/0x40
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff8144d0c3>] ? net_rx_action+0x103/0x2f0
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffffa00ef0b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff81076fb1>] ? __do_softirq+0xc1/0x1e0
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff810e1720>] ? handle_IRQ_event+0x60/0x170
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff81076d95>] ? irq_exit+0x85/0x90
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff81517145>] ? do_IRQ+0x75/0xf0
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
      Oct 22 11:22:52 SG-haystackdb-518 kernel: <EOI> [<ffffffff8103b90b>] ? native_safe_halt+0xb/0x10
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff810149ad>] ? default_idle+0x4d/0xb0
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff814f33da>] ? rest_init+0x7a/0x80
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff81c27f7b>] ? start_kernel+0x424/0x430
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff81c2733a>] ? x86_64_start_reservations+0x125/0x129
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff81c27438>] ? x86_64_start_kernel+0xfa/0x109
      Oct 22 11:22:52 SG-haystackdb-518 kernel: swapper: page allocation failure. order:1, mode:0x20
      Oct 22 11:22:52 SG-haystackdb-518 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-358.6.2.el6.x86_64 #1
      Oct 22 11:22:52 SG-haystackdb-518 kernel: Call Trace:
      Oct 22 11:22:52 SG-haystackdb-518 kernel: <IRQ> [<ffffffff8112c207>] ? __alloc_pages_nodemask+0x757/0x8d0
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffffa00ef75e>] ? start_xmit+0x6e/0x1d0 [virtio_net]
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff81166ab2>] ? kmem_getpages+0x62/0x170
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff811676ca>] ? fallback_alloc+0x1ba/0x270
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff8116711f>] ? cache_grow+0x2cf/0x320
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff81167449>] ? ____cache_alloc_node+0x99/0x160
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff811683cb>] ? kmem_cache_alloc+0x11b/0x190
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff81439eb8>] ? sk_prot_alloc+0x48/0x1c0
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff8143af92>] ? sk_clone+0x22/0x2e0
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff81489ec6>] ? inet_csk_clone+0x16/0xd0
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff814a2e13>] ? tcp_create_openreq_child+0x23/0x450
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff814a060d>] ? tcp_v4_syn_recv_sock+0x4d/0x310
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff814a2bb6>] ? tcp_check_req+0x226/0x460
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff814a00ab>] ? tcp_v4_do_rcv+0x35b/0x430
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffffa0213557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff814a18be>] ? tcp_v4_rcv+0x4fe/0x8d0
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff8147f590>] ? ip_local_deliver_finish+0x0/0x2d0
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff8147f66d>] ? ip_local_deliver_finish+0xdd/0x2d0
      Oct 22 11:22:52 SG-haystackdb-518 kernel: [<ffffffff8147f8f8>] ? ip_local_deliver+0x98/0xa0
      Tue Oct 22 22:34:19.768 [conn11514] end connection 50.19.152.124:56247 (17 connections now open)
      Tue Oct 22 22:34:19.768 [conn11512] end connection 50.19.152.124:56245 (17 connections now open)
      Tue Oct 22 22:34:20.134 [initandlisten] connection accepted from 50.19.152.124:56630 #11529 (16 connections now open)
      Tue Oct 22 22:34:20.154 [initandlisten] connection accepted from 50.19.152.124:56632 #11530 (17 connections now open)
      Tue Oct 22 22:34:20.197 [initandlisten] connection accepted from 50.19.152.124:56633 #11531 (18 connections now open)
      Tue Oct 22 22:34:20.217 [initandlisten] connection accepted from 50.19.152.124:56634 #11532 (19 connections now open)
      Tue Oct 22 22:34:20.234 [initandlisten] connection accepted from 50.19.152.124:56635 #11533 (20 connections now open)

            Assignee:
            Unassigned Unassigned
            Reporter:
            dharshanr@scalegrid.net Dharshan Rangegowda
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: