Hello all
I am facing a problem with my centos server with oracle database
application on it. The server gets hang and I have to do a force
reboot. The server is a vm. On examining the logs I have found below
in /var/log/messages. I want a guidance on this., what can be
concluded and what action should be taken now.
Nov 21 10:41:08 server kernel:
Nov 21 10:41:08 server kernel: Call Trace:
Nov 21 10:41:08 server kernel: [
+0x8e/0x2f5
Nov 21 10:41:08 server kernel: [
+0x245/0x2ce
Nov 21 10:41:08 server kernel: [
__do_page_cache_readahead+0x95/0x1d9
Nov 21 10:41:08 server kernel: [
+0x5b/0x66
Nov 21 10:41:08 server kernel:
[
Nov 21 10:41:08 server kernel: [
+0x148/0x322
Nov 21 10:41:08 server kernel: [
+0x1f8/0xe23
Nov 21 10:41:08 server kernel: [
+0x4cb/0x830
Nov 21 10:41:08 server kernel: [
+0x0/0x84
Nov 21 10:41:08 server kernel:
Nov 21 10:41:13 server kernel: Mem-info:
Nov 21 10:41:13 server kernel: Node 0 DMA per-cpu:
Nov 21 10:41:13 server kernel: cpu 0 hot: high 0, batch 1 used:0
Nov 21 10:41:13 server kernel: cpu 0 cold: high 0, batch 1 used:0
Nov 21 10:41:13 server kernel: cpu 1 hot: high 0, batch 1 used:0
Nov 21 10:41:13 server kernel: cpu 1 cold: high 0, batch 1 used:0
Nov 21 10:41:13 server kernel: Node 0 DMA32 per-cpu:
Nov 21 10:41:13 server kernel: cpu 0 hot: high 186, batch 31 used:42
Nov 21 10:41:13 server kernel: cpu 0 cold: high 62, batch 15 used:14
Nov 21 10:41:13 server kernel: cpu 1 hot: high 186, batch 31 used:21
Nov 21 10:41:13 server kernel: cpu 1 cold: high 62, batch 15 used:59
Nov 21 10:41:13 server kernel: Node 0 Normal per-cpu: empty
Nov 21 10:41:13 server kernel: Node 0 HighMem per-cpu: empty
Nov 21 10:41:13 server kernel: Free pages: 7856kB (0kB HighMem)
Nov 21 10:41:13 server kernel: Active:258059 inactive:234203 dirty:0
writeback:0 unstable:0 free:1964 slab:7285 mapped-file:994 mapped-anon:
413490 pageta
bles:7672
Nov 21 10:41:13 server kernel: Node 0 DMA free:2180kB min:28kB low:
32kB high:40kB active:0kB inactive:0kB present:10780kB pages_scanned:0
all_unreclaimab
le? yes
Nov 21 10:41:13 server kernel: lowmem_reserve[]: 0 2004 2004 2004
Nov 21 10:41:13 server kernel: Node 0 DMA32 free:5676kB min:5712kB low:
7140kB high:8568kB active:1032236kB inactive:936812kB present:
2052256kB pages_scan
ned:5132543 all_unreclaimable? yes
Nov 21 10:41:13 server kernel: lowmem_reserve[]: 0 0 0 0
Nov 21 10:41:13 server kernel: Node 0 Normal free:0kB min:0kB low:0kB
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Nov 21 10:41:13 server kernel: lowmem_reserve[]: 0 0 0 0
Nov 21 10:41:13 server kernel: Node 0 HighMem free:0kB min:128kB low:
128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimab
le? no
Nov 21 10:41:13 server kernel: lowmem_reserve[]: 0 0 0 0
Nov 21 10:41:14 server kernel: Node 0 DMA: 3*4kB 5*8kB 3*16kB 3*32kB
1*64kB 3*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2180kB
Nov 21 10:41:14 server kernel: Node 0 DMA32: 13*4kB 3*8kB 2*16kB
0*32kB 1*64kB 1*128kB 1*256kB 2*512kB 0*1024kB 0*2048kB 1*4096kB =
5676kB
Nov 21 10:41:14 server kernel: Node 0 Normal: empty
Nov 21 10:41:14 server kernel: Node 0 HighMem: empty
Nov 21 10:41:14 server kernel: 79782 pagecache pages
Nov 21 10:41:14 server kernel: Swap cache: add 2637883, delete
2637386, find 389368/637880, race 0+2628
Nov 21 10:41:14 server kernel: Free swap = 0kB
Nov 21 10:41:14 server kernel: Total swap = 2031608kB
Nov 21 10:41:14 server kernel: Free swap: 0kB
Nov 21 10:41:14 server kernel: 524288 pages of RAM
Nov 21 10:41:14 server kernel: 9412 reserved pages
Nov 21 10:41:14 server kernel: 11493 pages shared
Nov 21 10:41:14 server kernel: 528 pages swap cached
Nov 21 10:41:14 server kernel: Out of memory: Killed process 27777
(oracle).
Nov 21 10:41:14 server kernel: audispd invoked oom-killer:
gfp_mask=0x201d2, order=0, oomkilladj=0
Nov 21 10:41:14 server kernel:
Please note that the server memory remains
total used free shared
buffers cached
Mem: 2011 1977 33 0 43
1743
-/+ buffers/cache: 190 1820
Swap: 1983 77 1906
But this means enough memory is free, and swap size is also
sufficient. please suggest
Thanks
Niyati
Centos server gets hang and kernel errors in logs
Nirmal Pathak |On Nov 22, 11:07 am, Niyati Dave wrote:] out_of_memory] __alloc_pages]] __wait_on_bit_lock] :dm_mod:dm_any_congested+0x38/0x3f] filemap_nopage] __handle_mm_fault] do_page_fault] error_exit
> Hello all
>
> I am facing a problem with my centos server with oracle database
> application on it. The server gets hang and I have to do a force
> reboot. The server is a vm. On examining the logs I have found below
> in /var/log/messages. I want a guidance on this., what can be
> concluded and what action should be taken now.
>
> Nov 21 10:41:08 server kernel:
> Nov 21 10:41:08 server kernel: Call Trace:
> Nov 21 10:41:08 server kernel: [
> +0x8e/0x2f5
> Nov 21 10:41:08 server kernel: [
> +0x245/0x2ce
> Nov 21 10:41:08 server kernel: [
> __do_page_cache_readahead+0x95/0x1d9
> Nov 21 10:41:08 server kernel: [
> +0x5b/0x66
> Nov 21 10:41:08 server kernel:
> [
> Nov 21 10:41:08 server kernel: [
> +0x148/0x322
> Nov 21 10:41:08 server kernel: [
> +0x1f8/0xe23
> Nov 21 10:41:08 server kernel: [
> +0x4cb/0x830
> Nov 21 10:41:08 server kernel: [
> +0x0/0x84
> Nov 21 10:41:08 server kernel:
> Nov 21 10:41:13 server kernel: Mem-info:
> Nov 21 10:41:13 server kernel: Node 0 DMA per-cpu:
> Nov 21 10:41:13 server kernel: cpu 0 hot: high 0, batch 1 used:0
> Nov 21 10:41:13 server kernel: cpu 0 cold: high 0, batch 1 used:0
> Nov 21 10:41:13 server kernel: cpu 1 hot: high 0, batch 1 used:0
> Nov 21 10:41:13 server kernel: cpu 1 cold: high 0, batch 1 used:0
> Nov 21 10:41:13 server kernel: Node 0 DMA32 per-cpu:
> Nov 21 10:41:13 server kernel: cpu 0 hot: high 186, batch 31 used:42
> Nov 21 10:41:13 server kernel: cpu 0 cold: high 62, batch 15 used:14
> Nov 21 10:41:13 server kernel: cpu 1 hot: high 186, batch 31 used:21
> Nov 21 10:41:13 server kernel: cpu 1 cold: high 62, batch 15 used:59
> Nov 21 10:41:13 server kernel: Node 0 Normal per-cpu: empty
> Nov 21 10:41:13 server kernel: Node 0 HighMem per-cpu: empty
> Nov 21 10:41:13 server kernel: Free pages: 7856kB (0kB HighMem)
> Nov 21 10:41:13 server kernel: Active:258059 inactive:234203 dirty:0
> writeback:0 unstable:0 free:1964 slab:7285 mapped-file:994 mapped-anon:
> 413490 pageta
> bles:7672
> Nov 21 10:41:13 server kernel: Node 0 DMA free:2180kB min:28kB low:
> 32kB high:40kB active:0kB inactive:0kB present:10780kB pages_scanned:0
> all_unreclaimab
> le? yes
> Nov 21 10:41:13 server kernel: lowmem_reserve[]: 0 2004 2004 2004
> Nov 21 10:41:13 server kernel: Node 0 DMA32 free:5676kB min:5712kB low:
> 7140kB high:8568kB active:1032236kB inactive:936812kB present:
> 2052256kB pages_scan
> ned:5132543 all_unreclaimable? yes
> Nov 21 10:41:13 server kernel: lowmem_reserve[]: 0 0 0 0
> Nov 21 10:41:13 server kernel: Node 0 Normal free:0kB min:0kB low:0kB
> high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0
> all_unreclaimable? no
> Nov 21 10:41:13 server kernel: lowmem_reserve[]: 0 0 0 0
> Nov 21 10:41:13 server kernel: Node 0 HighMem free:0kB min:128kB low:
> 128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0
> all_unreclaimab
> le? no
> Nov 21 10:41:13 server kernel: lowmem_reserve[]: 0 0 0 0
> Nov 21 10:41:14 server kernel: Node 0 DMA: 3*4kB 5*8kB 3*16kB 3*32kB
> 1*64kB 3*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2180kB
> Nov 21 10:41:14 server kernel: Node 0 DMA32: 13*4kB 3*8kB 2*16kB
> 0*32kB 1*64kB 1*128kB 1*256kB 2*512kB 0*1024kB 0*2048kB 1*4096kB =
> 5676kB
> Nov 21 10:41:14 server kernel: Node 0 Normal: empty
> Nov 21 10:41:14 server kernel: Node 0 HighMem: empty
> Nov 21 10:41:14 server kernel: 79782 pagecache pages
> Nov 21 10:41:14 server kernel: Swap cache: add 2637883, delete
> 2637386, find 389368/637880, race 0+2628
> Nov 21 10:41:14 server kernel: Free swap = 0kB
> Nov 21 10:41:14 server kernel: Total swap = 2031608kB
> Nov 21 10:41:14 server kernel: Free swap: 0kB
> Nov 21 10:41:14 server kernel: 524288 pages of RAM
> Nov 21 10:41:14 server kernel: 9412 reserved pages
> Nov 21 10:41:14 server kernel: 11493 pages shared
> Nov 21 10:41:14 server kernel: 528 pages swap cached
> Nov 21 10:41:14 server kernel: Out of memory: Killed process 27777
> (oracle).
> Nov 21 10:41:14 server kernel: audispd invoked oom-killer:
> gfp_mask=0x201d2, order=0, oomkilladj=0
> Nov 21 10:41:14 server kernel:
>
The server was out of memory. Check following logs,
===
Nov 21 10:41:14 server kernel: Node 0 Normal: empty
Nov 21 10:41:14 server kernel: Node 0 HighMem: empty
Nov 21 10:41:14 server kernel: 79782 pagecache pages
Nov 21 10:41:14 server kernel: Swap cache: add 2637883, delete
2637386, find 389368/637880, race 0+2628
Nov 21 10:41:14 server kernel: Free swap = 0kB
Nov 21 10:41:14 server kernel: Total swap = 2031608kB
Nov 21 10:41:14 server kernel: Free swap: 0kB
==
You can see that the system had no Physical or Swap Memory left & it
got hung! This situation is called OOM (Out of Memory) & not kernel
panic. The system is under memory starvation.
I would recommend you to increase swap space on the system. Setting
memory tunable in kernel can be done but first try to add some swap &
monitor the system.
You can checkout following link for memory tunable in kernel.
http://www.cyberciti.biz/faq/linux-kernel-tuning-virtual-memory-subsystem/
You need to understand the each parameter before making any changes in
VM subsystem.
> Please note that the server memory remains
>
> total used free shared
> buffers cached
> Mem: 2011 1977 33 0 43
> 1743
> -/+ buffers/cache: 190 1820
> Swap: 1983 77 1906
>
> But this means enough memory is free, and swap size is also
> sufficient. please suggest
>
The 'free' command's output which you have given seems to be after
reboot of the system and not when the system was in hung state.
> Thanks
> Niyati
Have FuN!
-- Nirmal Pathak.
Centos server gets hang and kernel errors in logs
Sachin Gujral |Hello,
I gone throgh out put of the free -m command we can see there 1743 MB of your memory store in cache so u have to release the memory from the cache after then your server will get free memory and also increase your SWAP space. as per my understanding the virtual memory space should always double of your RAM space.
to release your cache memory u have to follow these steps listed below............
# free -m
# sync
# echo 3 > /proc/sys/vm/drop_caches
at last please let me know if i m doing any mistake in this manner.
--
Sachin Gujral
RHCE MCSA
System Administrator
Centos server gets hang and kernel errors in logs
Nirmal Pathak |On 23 November 2011 10:36, Sachin Gujral
wrote:
> Hello,
Please do not Top Post on mailing list! :(
> I gone throgh out put of the free -m command we can see there
> 1743 MB of your memory store in cache so u have to release the memory from
> the cache after then your server will get free memory and also increase your
> SWAP space. as per my understanding the virtual memory space should always
> double of your RAM space.
This was thumb rule when there were systems with less memory. Now days
memory & storage is cheap. We get systems with 32/64 GB RAM so there
we do not need Swap Space double of RAM, it is off NO use.
I would like to mention here that Memory Caching is in fact a good for
applications to run faster. Dropping memory cache will give
performance hit on the running applications.
> to release your cache memory u have to follow these steps listed
> below............
>
> # free -m
>
> # sync
>
> # echo 3 > /proc/sys/vm/drop_caches
>
> at last please let me know if i m doing any mistake in this manner.
>
Dropping memory caches may effect applications currently running on
the systems hence it is not good idea to drop the caches unless you
really need it!
The 'free -m' output shown seems to be captured after the system
reboot hence it will not solve the issue of OOM.
> wrote: wrote:] out_of_memory] __alloc_pages]] __wait_on_bit_lock] :dm_mod:dm_any_congested+0x38/0x3f] filemap_nopage] __handle_mm_fault] do_page_fault] error_exit
> On Tue, Nov 22, 2011 at 11:47 AM, Nirmal
>>
>>
>> On Nov 22, 11:07 am, Niyati Dave
>> > Hello all
>> >
>> > I am facing a problem with my centos server with oracle database
>> > application on it. The server gets hang and I have to do a force
>> > reboot. The server is a vm. On examining the logs I have found below
>> > in /var/log/messages. I want a guidance on this., what can be
>> > concluded and what action should be taken now.
>> >
>> > Nov 21 10:41:08 server kernel:
>> > Nov 21 10:41:08 server kernel: Call Trace:
>> > Nov 21 10:41:08 server kernel: [
>> > +0x8e/0x2f5
>> > Nov 21 10:41:08 server kernel: [
>> > +0x245/0x2ce
>> > Nov 21 10:41:08 server kernel: [
>> > __do_page_cache_readahead+0x95/0x1d9
>> > Nov 21 10:41:08 server kernel: [
>> > +0x5b/0x66
>> > Nov 21 10:41:08 server kernel:
>> > [
>> > Nov 21 10:41:08 server kernel: [
>> > +0x148/0x322
>> > Nov 21 10:41:08 server kernel: [
>> > +0x1f8/0xe23
>> > Nov 21 10:41:08 server kernel: [
>> > +0x4cb/0x830
>> > Nov 21 10:41:08 server kernel: [
>> > +0x0/0x84
>> > Nov 21 10:41:08 server kernel:
>> > Nov 21 10:41:13 server kernel: Mem-info:
>> > Nov 21 10:41:13 server kernel: Node 0 DMA per-cpu:
>> > Nov 21 10:41:13 server kernel: cpu 0 hot: high 0, batch 1 used:0
>> > Nov 21 10:41:13 server kernel: cpu 0 cold: high 0, batch 1 used:0
>> > Nov 21 10:41:13 server kernel: cpu 1 hot: high 0, batch 1 used:0
>> > Nov 21 10:41:13 server kernel: cpu 1 cold: high 0, batch 1 used:0
>> > Nov 21 10:41:13 server kernel: Node 0 DMA32 per-cpu:
>> > Nov 21 10:41:13 server kernel: cpu 0 hot: high 186, batch 31 used:42
>> > Nov 21 10:41:13 server kernel: cpu 0 cold: high 62, batch 15 used:14
>> > Nov 21 10:41:13 server kernel: cpu 1 hot: high 186, batch 31 used:21
>> > Nov 21 10:41:13 server kernel: cpu 1 cold: high 62, batch 15 used:59
>> > Nov 21 10:41:13 server kernel: Node 0 Normal per-cpu: empty
>> > Nov 21 10:41:13 server kernel: Node 0 HighMem per-cpu: empty
>> > Nov 21 10:41:13 server kernel: Free pages: 7856kB (0kB HighMem)
>> > Nov 21 10:41:13 server kernel: Active:258059 inactive:234203 dirty:0
>> > writeback:0 unstable:0 free:1964 slab:7285 mapped-file:994 mapped-anon:
>> > 413490 pageta
>> > bles:7672
>> > Nov 21 10:41:13 server kernel: Node 0 DMA free:2180kB min:28kB low:
>> > 32kB high:40kB active:0kB inactive:0kB present:10780kB pages_scanned:0
>> > all_unreclaimab
>> > le? yes
>> > Nov 21 10:41:13 server kernel: lowmem_reserve[]: 0 2004 2004 2004
>> > Nov 21 10:41:13 server kernel: Node 0 DMA32 free:5676kB min:5712kB low:
>> > 7140kB high:8568kB active:1032236kB inactive:936812kB present:
>> > 2052256kB pages_scan
>> > ned:5132543 all_unreclaimable? yes
>> > Nov 21 10:41:13 server kernel: lowmem_reserve[]: 0 0 0 0
>> > Nov 21 10:41:13 server kernel: Node 0 Normal free:0kB min:0kB low:0kB
>> > high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0
>> > all_unreclaimable? no
>> > Nov 21 10:41:13 server kernel: lowmem_reserve[]: 0 0 0 0
>> > Nov 21 10:41:13 server kernel: Node 0 HighMem free:0kB min:128kB low:
>> > 128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0
>> > all_unreclaimab
>> > le? no
>> > Nov 21 10:41:13 server kernel: lowmem_reserve[]: 0 0 0 0
>> > Nov 21 10:41:14 server kernel: Node 0 DMA: 3*4kB 5*8kB 3*16kB 3*32kB
>> > 1*64kB 3*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2180kB
>> > Nov 21 10:41:14 server kernel: Node 0 DMA32: 13*4kB 3*8kB 2*16kB
>> > 0*32kB 1*64kB 1*128kB 1*256kB 2*512kB 0*1024kB 0*2048kB 1*4096kB =
>> > 5676kB
>> > Nov 21 10:41:14 server kernel: Node 0 Normal: empty
>> > Nov 21 10:41:14 server kernel: Node 0 HighMem: empty
>> > Nov 21 10:41:14 server kernel: 79782 pagecache pages
>> > Nov 21 10:41:14 server kernel: Swap cache: add 2637883, delete
>> > 2637386, find 389368/637880, race 0+2628
>> > Nov 21 10:41:14 server kernel: Free swap = 0kB
>> > Nov 21 10:41:14 server kernel: Total swap = 2031608kB
>> > Nov 21 10:41:14 server kernel: Free swap: 0kB
>> > Nov 21 10:41:14 server kernel: 524288 pages of RAM
>> > Nov 21 10:41:14 server kernel: 9412 reserved pages
>> > Nov 21 10:41:14 server kernel: 11493 pages shared
>> > Nov 21 10:41:14 server kernel: 528 pages swap cached
>> > Nov 21 10:41:14 server kernel: Out of memory: Killed process 27777
>> > (oracle).
>> > Nov 21 10:41:14 server kernel: audispd invoked oom-killer:
>> > gfp_mask=0x201d2, order=0, oomkilladj=0
>> > Nov 21 10:41:14 server kernel:
>> >
>>
>> The server was out of memory. Check following logs,
>>
>> ===
>> Nov 21 10:41:14 server kernel: Node 0 Normal: empty
>> Nov 21 10:41:14 server kernel: Node 0 HighMem: empty
>> Nov 21 10:41:14 server kernel: 79782 pagecache pages
>> Nov 21 10:41:14 server kernel: Swap cache: add 2637883, delete
>> 2637386, find 389368/637880, race 0+2628
>> Nov 21 10:41:14 server kernel: Free swap = 0kB
>> Nov 21 10:41:14 server kernel: Total swap = 2031608kB
>> Nov 21 10:41:14 server kernel: Free swap: 0kB
>> ==
>>
>> You can see that the system had no Physical or Swap Memory left & it
>> got hung! This situation is called OOM (Out of Memory) & not kernel
>> panic. The system is under memory starvation.
>>
>> I would recommend you to increase swap space on the system. Setting
>> memory tunable in kernel can be done but first try to add some swap &
>> monitor the system.
>>
>> You can checkout following link for memory tunable in kernel.
>> http://www.cyberciti.biz/faq/linux-kernel-tuning-virtual-memory-subsystem/
>>
>> You need to understand the each parameter before making any changes in
>> VM subsystem.
>>
>> > Please note that the server memory remains
>> >
>> > total used free shared
>> > buffers cached
>> > Mem: 2011 1977 33 0 43
>> > 1743
>> > -/+ buffers/cache: 190 1820
>> > Swap: 1983 77 1906
>> >
>> > But this means enough memory is free, and swap size is also
>> > sufficient. please suggest
>> >
>>
>> The 'free' command's output which you have given seems to be after
>> reboot of the system and not when the system was in hung state.
>>
>> > Thanks
>> > Niyati
>>
>> Have FuN!
>> -- Nirmal Pathak.
>>
>> --
>> Please read http://www.catb.org/~esr/faqs/smart-questions.html before
>> posting.
>> You received this message because you are subscribed to the "Vibrant
>> GNU/Linux User Group".
>> To stop receiving emails from this group, mail to
>> VGLUG+unsubscribe@googlegroups.com
>>
>>
>
>
>
> --
> Sachin Gujral
> RHCE MCSA
> System Administrator
>
> --
> Please read http://www.catb.org/~esr/faqs/smart-questions.html before
> posting.
> You received this message because you are subscribed to the "Vibrant
> GNU/Linux User Group".
> To stop receiving emails from this group, mail to
> VGLUG+unsubscribe@googlegroups.com
>
>
>
--
Nirmal D Pathak.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
What I want is freedom, not life, nor pleasure, nor good!
FOSS gives that.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Centos server gets hang and kernel errors in logs
Niyati Dave |Thanks for the immediate responses. It is helpful to me.
currently I am monitoring the server to find what uses all the memory, later planning to try out with increasing swap.
Thanks again
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
> 1743 MB of your memory store in cache so u have to release the memory from
> the cache after then your server will get free memory and also increase your
> SWAP space. as per my understanding the virtual memory space should always
> double of your RAM space.
This was thumb rule when there were systems with less memory. Now days
memory & storage is cheap. We get systems with 32/64 GB RAM so there
we do not need Swap Space double of RAM, it is off NO use.
I would like to mention here that Memory Caching is in fact a good for
applications to run faster. Dropping memory cache will give
performance hit on the running applications.
> to release your cache memory u have to follow these steps listed
> below............
>
> # free -m
>
> # sync
>
> # echo 3 > /proc/sys/vm/drop_caches
>
> at last please let me know if i m doing any mistake in this manner.
>
Dropping memory caches may effect applications currently running on
the systems hence it is not good idea to drop the caches unless you
really need it!
The 'free -m' output shown seems to be captured after the system
reboot hence it will not solve the issue of OOM.
>
> On Tue, Nov 22, 2011 at 11:47 AM, Nirmal <nirmal.pathak@gmail.com> wrote:
>>
>>
>> On Nov 22, 11:07 am, Niyati Dave <niyati...@gmail.com> wrote:
>> > Hello all
>> >
>> > I am facing a problem with my centos server with oracle database
>> > application on it. The server gets hang and I have to do a force
>> > reboot. The server is a vm. On examining the logs I have found below
>> > in /var/log/messages. I want a guidance on this., what can be
>> > concluded and what action should be taken now.
>> >
>> > Nov 21 10:41:08 server kernel:
>> > Nov 21 10:41:08 server kernel: Call Trace:
>> > Nov 21 10:41:08 server kernel: [<ffffffff800c0b09>] out_of_memory
>> > +0x8e/0x2f5
>> > Nov 21 10:41:08 server kernel: [<ffffffff8000f263>] __alloc_pages
>> > +0x245/0x2ce
>> > Nov 21 10:41:08 server kernel: [<ffffffff80012946>]
>> > __do_page_cache_readahead+0x95/0x1d9
>> > Nov 21 10:41:08 server kernel: [<ffffffff800638ab>] __wait_on_bit_lock
>> > +0x5b/0x66
>> > Nov 21 10:41:08 server kernel:
>> > [<ffffffff88141d4d>] :dm_mod:dm_any_congested+0x38/0x3f
>> > Nov 21 10:41:08 server kernel: [<ffffffff800132e5>] filemap_nopage
>> > +0x148/0x322
>> > Nov 21 10:41:08 server kernel: [<ffffffff80008848>] __handle_mm_fault
>> > +0x1f8/0xe23
>> > Nov 21 10:41:08 server kernel: [<ffffffff8006681f>] do_page_fault
>> > +0x4cb/0x830
>> > Nov 21 10:41:08 server kernel: [<ffffffff8005dde9>] error_exit
>> > +0x0/0x84
>> > Nov 21 10:41:08 server kernel:
>> > Nov 21 10:41:13 server kernel: Mem-info:
>> > Nov 21 10:41:13 server kernel: Node 0 DMA per-cpu:
>> > Nov 21 10:41:13 server kernel: cpu 0 hot: high 0, batch 1 used:0
>> > Nov 21 10:41:13 server kernel: cpu 0 cold: high 0, batch 1 used:0
>> > Nov 21 10:41:13 server kernel: cpu 1 hot: high 0, batch 1 used:0
>> > Nov 21 10:41:13 server kernel: cpu 1 cold: high 0, batch 1 used:0
>> > Nov 21 10:41:13 server kernel: Node 0 DMA32 per-cpu:
>> > Nov 21 10:41:13 server kernel: cpu 0 hot: high 186, batch 31 used:42
>> > Nov 21 10:41:13 server kernel: cpu 0 cold: high 62, batch 15 used:14
>> > Nov 21 10:41:13 server kernel: cpu 1 hot: high 186, batch 31 used:21
>> > Nov 21 10:41:13 server kernel: cpu 1 cold: high 62, batch 15 used:59
>> > Nov 21 10:41:13 server kernel: Node 0 Normal per-cpu: empty
>> > Nov 21 10:41:13 server kernel: Node 0 HighMem per-cpu: empty
>> > Nov 21 10:41:13 server kernel: Free pages: 7856kB (0kB HighMem)
>> > Nov 21 10:41:13 server kernel: Active:258059 inactive:234203 dirty:0
>> > writeback:0 unstable:0 free:1964 slab:7285 mapped-file:994 mapped-anon:
>> > 413490 pageta
>> > bles:7672
>> > Nov 21 10:41:13 server kernel: Node 0 DMA free:2180kB min:28kB low:
>> > 32kB high:40kB active:0kB inactive:0kB present:10780kB pages_scanned:0
>> > all_unreclaimab
>> > le? yes
>> > Nov 21 10:41:13 server kernel: lowmem_reserve[]: 0 2004 2004 2004
>> > Nov 21 10:41:13 server kernel: Node 0 DMA32 free:5676kB min:5712kB low:
>> > 7140kB high:8568kB active:1032236kB inactive:936812kB present:
>> > 2052256kB pages_scan
>> > ned:5132543 all_unreclaimable? yes
>> > Nov 21 10:41:13 server kernel: lowmem_reserve[]: 0 0 0 0
>> > Nov 21 10:41:13 server kernel: Node 0 Normal free:0kB min:0kB low:0kB
>> > high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0
>> > all_unreclaimable? no
>> > Nov 21 10:41:13 server kernel: lowmem_reserve[]: 0 0 0 0
>> > Nov 21 10:41:13 server kernel: Node 0 HighMem free:0kB min:128kB low:
>> > 128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0
>> > all_unreclaimab
>> > le? no
>> > Nov 21 10:41:13 server kernel: lowmem_reserve[]: 0 0 0 0
>> > Nov 21 10:41:14 server kernel: Node 0 DMA: 3*4kB 5*8kB 3*16kB 3*32kB
>> > 1*64kB 3*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2180kB
>> > Nov 21 10:41:14 server kernel: Node 0 DMA32: 13*4kB 3*8kB 2*16kB
>> > 0*32kB 1*64kB 1*128kB 1*256kB 2*512kB 0*1024kB 0*2048kB 1*4096kB =
>> > 5676kB
>> > Nov 21 10:41:14 server kernel: Node 0 Normal: empty
>> > Nov 21 10:41:14 server kernel: Node 0 HighMem: empty
>> > Nov 21 10:41:14 server kernel: 79782 pagecache pages
>> > Nov 21 10:41:14 server kernel: Swap cache: add 2637883, delete
>> > 2637386, find 389368/637880, race 0+2628
>> > Nov 21 10:41:14 server kernel: Free swap = 0kB
>> > Nov 21 10:41:14 server kernel: Total swap = 2031608kB
>> > Nov 21 10:41:14 server kernel: Free swap: 0kB
>> > Nov 21 10:41:14 server kernel: 524288 pages of RAM
>> > Nov 21 10:41:14 server kernel: 9412 reserved pages
>> > Nov 21 10:41:14 server kernel: 11493 pages shared
>> > Nov 21 10:41:14 server kernel: 528 pages swap cached
>> > Nov 21 10:41:14 server kernel: Out of memory: Killed process 27777
>> > (oracle).
>> > Nov 21 10:41:14 server kernel: audispd invoked oom-killer:
>> > gfp_mask=0x201d2, order=0, oomkilladj=0
>> > Nov 21 10:41:14 server kernel:
>> >
>>
>> The server was out of memory. Check following logs,
>>
>> ===
>> Nov 21 10:41:14 server kernel: Node 0 Normal: empty
>> Nov 21 10:41:14 server kernel: Node 0 HighMem: empty
>> Nov 21 10:41:14 server kernel: 79782 pagecache pages
>> Nov 21 10:41:14 server kernel: Swap cache: add 2637883, delete
>> 2637386, find 389368/637880, race 0+2628
>> Nov 21 10:41:14 server kernel: Free swap = 0kB
>> Nov 21 10:41:14 server kernel: Total swap = 2031608kB
>> Nov 21 10:41:14 server kernel: Free swap: 0kB
>> ==
>>
>> You can see that the system had no Physical or Swap Memory left & it
>> got hung! This situation is called OOM (Out of Memory) & not kernel
>> panic. The system is under memory starvation.
>>
>> I would recommend you to increase swap space on the system. Setting
>> memory tunable in kernel can be done but first try to add some swap &
>> monitor the system.
>>
>> You can checkout following link for memory tunable in kernel.
>> http://www.cyberciti.biz/faq/linux-kernel-tuning-virtual-memory-subsystem/
>>
>> You need to understand the each parameter before making any changes in
>> VM subsystem.
>>
>> > Please note that the server memory remains
>> >
>> > total used free shared
>> > buffers cached
>> > Mem: 2011 1977 33 0 43
>> > 1743
>> > -/+ buffers/cache: 190 1820
>> > Swap: 1983 77 1906
>> >
>> > But this means enough memory is free, and swap size is also
>> > sufficient. please suggest
>> >
>>
>> The 'free' command's output which you have given seems to be after
>> reboot of the system and not when the system was in hung state.
>>
>> > Thanks
>> > Niyati
>>
>> Have FuN!
>> -- Nirmal Pathak.
>>
>> --
>> Please read http://www.catb.org/~esr/faqs/smart-questions.html before
>> posting.
>> You received this message because you are subscribed to the "Vibrant
>> GNU/Linux User Group".
>> To stop receiving emails from this group, mail to
>> VGLUG+unsubscribe@googlegroups.com
>>
>>
>
>
>
> --
> Sachin Gujral
> RHCE MCSA
> System Administrator
>
> --
> Please read http://www.catb.org/~esr/faqs/smart-questions.html before
> posting.
> You received this message because you are subscribed to the "Vibrant
> GNU/Linux User Group".
> To stop receiving emails from this group, mail to
> VGLUG+unsubscribe@googlegroups.com
>
>
>
--
Nirmal D Pathak.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
What I want is freedom, not life, nor pleasure, nor good!
FOSS gives that.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
Centos server gets hang and kernel errors in logs
Nirmal Pathak |On Nov 22, 11:07 am, Niyati Dave wrote:] out_of_memory] __alloc_pages]] __wait_on_bit_lock] :dm_mod:dm_any_congested+0x38/0x3f] filemap_nopage] __handle_mm_fault] do_page_fault] error_exit
> Hello all
>
> I am facing a problem with my centos server with oracle database
> application on it. The server gets hang and I have to do a force
> reboot. The server is a vm. On examining the logs I have found below
> in /var/log/messages. I want a guidance on this., what can be
> concluded and what action should be taken now.
>
> Nov 21 10:41:08 server kernel:
> Nov 21 10:41:08 server kernel: Call Trace:
> Nov 21 10:41:08 server kernel: [
> +0x8e/0x2f5
> Nov 21 10:41:08 server kernel: [
> +0x245/0x2ce
> Nov 21 10:41:08 server kernel: [
> __do_page_cache_readahead+0x95/0x1d9
> Nov 21 10:41:08 server kernel: [
> +0x5b/0x66
> Nov 21 10:41:08 server kernel:
> [
> Nov 21 10:41:08 server kernel: [
> +0x148/0x322
> Nov 21 10:41:08 server kernel: [
> +0x1f8/0xe23
> Nov 21 10:41:08 server kernel: [
> +0x4cb/0x830
> Nov 21 10:41:08 server kernel: [
> +0x0/0x84
> Nov 21 10:41:08 server kernel:
> Nov 21 10:41:13 server kernel: Mem-info:
> Nov 21 10:41:13 server kernel: Node 0 DMA per-cpu:
> Nov 21 10:41:13 server kernel: cpu 0 hot: high 0, batch 1 used:0
> Nov 21 10:41:13 server kernel: cpu 0 cold: high 0, batch 1 used:0
> Nov 21 10:41:13 server kernel: cpu 1 hot: high 0, batch 1 used:0
> Nov 21 10:41:13 server kernel: cpu 1 cold: high 0, batch 1 used:0
> Nov 21 10:41:13 server kernel: Node 0 DMA32 per-cpu:
> Nov 21 10:41:13 server kernel: cpu 0 hot: high 186, batch 31 used:42
> Nov 21 10:41:13 server kernel: cpu 0 cold: high 62, batch 15 used:14
> Nov 21 10:41:13 server kernel: cpu 1 hot: high 186, batch 31 used:21
> Nov 21 10:41:13 server kernel: cpu 1 cold: high 62, batch 15 used:59
> Nov 21 10:41:13 server kernel: Node 0 Normal per-cpu: empty
> Nov 21 10:41:13 server kernel: Node 0 HighMem per-cpu: empty
> Nov 21 10:41:13 server kernel: Free pages: 7856kB (0kB HighMem)
> Nov 21 10:41:13 server kernel: Active:258059 inactive:234203 dirty:0
> writeback:0 unstable:0 free:1964 slab:7285 mapped-file:994 mapped-anon:
> 413490 pageta
> bles:7672
> Nov 21 10:41:13 server kernel: Node 0 DMA free:2180kB min:28kB low:
> 32kB high:40kB active:0kB inactive:0kB present:10780kB pages_scanned:0
> all_unreclaimab
> le? yes
> Nov 21 10:41:13 server kernel: lowmem_reserve[]: 0 2004 2004 2004
> Nov 21 10:41:13 server kernel: Node 0 DMA32 free:5676kB min:5712kB low:
> 7140kB high:8568kB active:1032236kB inactive:936812kB present:
> 2052256kB pages_scan
> ned:5132543 all_unreclaimable? yes
> Nov 21 10:41:13 server kernel: lowmem_reserve[]: 0 0 0 0
> Nov 21 10:41:13 server kernel: Node 0 Normal free:0kB min:0kB low:0kB
> high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0
> all_unreclaimable? no
> Nov 21 10:41:13 server kernel: lowmem_reserve[]: 0 0 0 0
> Nov 21 10:41:13 server kernel: Node 0 HighMem free:0kB min:128kB low:
> 128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0
> all_unreclaimab
> le? no
> Nov 21 10:41:13 server kernel: lowmem_reserve[]: 0 0 0 0
> Nov 21 10:41:14 server kernel: Node 0 DMA: 3*4kB 5*8kB 3*16kB 3*32kB
> 1*64kB 3*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2180kB
> Nov 21 10:41:14 server kernel: Node 0 DMA32: 13*4kB 3*8kB 2*16kB
> 0*32kB 1*64kB 1*128kB 1*256kB 2*512kB 0*1024kB 0*2048kB 1*4096kB =
> 5676kB
> Nov 21 10:41:14 server kernel: Node 0 Normal: empty
> Nov 21 10:41:14 server kernel: Node 0 HighMem: empty
> Nov 21 10:41:14 server kernel: 79782 pagecache pages
> Nov 21 10:41:14 server kernel: Swap cache: add 2637883, delete
> 2637386, find 389368/637880, race 0+2628
> Nov 21 10:41:14 server kernel: Free swap = 0kB
> Nov 21 10:41:14 server kernel: Total swap = 2031608kB
> Nov 21 10:41:14 server kernel: Free swap: 0kB
> Nov 21 10:41:14 server kernel: 524288 pages of RAM
> Nov 21 10:41:14 server kernel: 9412 reserved pages
> Nov 21 10:41:14 server kernel: 11493 pages shared
> Nov 21 10:41:14 server kernel: 528 pages swap cached
> Nov 21 10:41:14 server kernel: Out of memory: Killed process 27777
> (oracle).
> Nov 21 10:41:14 server kernel: audispd invoked oom-killer:
> gfp_mask=0x201d2, order=0, oomkilladj=0
> Nov 21 10:41:14 server kernel:
>
This situation is OOM(Out Of Memory) and not Kernel Panic.
The system was facing memory starvation, which can be clearly seen by
following logs. The Physical Memory and Swap Space was completely
empty, which caused the system to hung itself.
--snip--
Nov 21 10:41:14 server kernel: Node 0 Normal: empty
Nov 21 10:41:14 server kernel: Node 0 HighMem: empty
Nov 21 10:41:14 server kernel: 79782 pagecache pages
Nov 21 10:41:14 server kernel: Swap cache: add 2637883, delete
2637386, find 389368/637880, race 0+2628
Nov 21 10:41:14 server kernel: Free swap = 0kB
Nov 21 10:41:14 server kernel: Total swap = 2031608kB
Nov 21 10:41:14 server kernel: Free swap: 0kB
--snip--
I would recommend you to increase some swap space initially before
going to fine tune any memory related kernel parameters.
> Please note that the server memory remains
>
> total used free shared
> buffers cached
> Mem: 2011 1977 33 0 43
> 1743
> -/+ buffers/cache: 190 1820
> Swap: 1983 77 1906
>
> But this means enough memory is free, and swap size is also
> sufficient. please suggest
>
It seems the 'free' command output which you have given was captured
after system reboot and not when the system was in hung state. You
have aorund 2 GB of swap space and I will recommend to increase it to
4 GB.
> Thanks
> Niyati
Have FuN!
- Nirmal Pathak.