[LUNI] Redhat swap memory crash

Andy_Bach at wiwb.uscourts.gov Andy_Bach at wiwb.uscourts.gov
Mon May 7 16:44:56 CDT 2007 

Hey,

Sorry, for the logs but - we've got 
Linux version 2.4.21-40.ELsmp (bhcompile at hs20-bc1-7.build.redhat.com) (gcc 
version 3.2.3 20030502 (Red Hat Linux 3.2.3-54))

w/ 2 Gig of ram
# cat /proc/meminfo
        total:    used:    free:  shared: buffers:  cached:
Mem:  2104774656 1760980992 343793664        0 159186944 1419739136
Swap: 2147442688        0 2147442688
MemTotal:      2055444 kB
MemFree:        335736 kB
MemShared:           0 kB
Buffers:        155456 kB
Cached:        1386464 kB
SwapCached:          0 kB
Active:         831792 kB
ActiveAnon:      27212 kB
ActiveCache:    804580 kB
Inact_dirty:    566508 kB
Inact_laundry:  170388 kB
Inact_clean:         0 kB
Inact_target:   313736 kB
HighTotal:     1179596 kB
HighFree:        27636 kB
LowTotal:       875848 kB
LowFree:        308100 kB
SwapTotal:     2097112 kB
SwapFree:      2097112 kB
CommitLimit:   3124832 kB
Committed_AS:   339612 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

twice now, in the last week, the box has locked up. messages shows the 8 
or so minutes worth of "Mem-info" entries (see below) before becoming 
frozen.  A webserver (Apache, nearly constant access) starts kicking back 
"500" pages about a minute after the first Mem-info entry (perhaps as the 
child was killed?) but seemed to recover and continue serving until 
reboot. 
2 questions (at least): 1) what is all the Mem-info info and is there 
anything useful in there (aside from zero swap ;-).  2. What kicks off the 
mem-info situation and how does it choose what processes to kill?

Any hints, pointers etc. much apreciated.

a

Apr 30 15:45:03 wiwbleo kernel: Mem-info:
Apr 30 15:45:03 wiwbleo kernel: Zone:DMA freepages:  2880 min:     0 low:  
  0 high:     0
Apr 30 15:45:03 wiwbleo kernel: Zone:Normal freepages:  1276 min:  1278 
low:  4543 high:  6303
Apr 30 15:45:03 wiwbleo kernel: Zone:HighMem freepages:   207 min:   255 
low:  4606 high:  6909
Apr 30 15:45:03 wiwbleo kernel: Free pages:        4363 (   207 HighMem)
Apr 30 15:45:03 wiwbleo kernel: ( Active: 470301/7922, inactive_laundry: 
0, inactive_clean: 0, free: 4363 )
Apr 30 15:45:03 wiwbleo kernel:   aa:0 ac:0 id:0 il:0 ic:0 fr:2880
Apr 30 15:45:03 wiwbleo kernel:   aa:192808 ac:1711 id:3616 il:0 ic:0 
fr:1276
Apr 30 15:45:03 wiwbleo kernel:   aa:273269 ac:2484 id:4335 il:0 ic:0 
fr:207
Apr 30 15:45:03 wiwbleo kernel: 0*4kB 0*8kB 2*16kB 3*32kB 4*64kB 1*128kB 
1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11520kB)
Apr 30 15:45:03 wiwbleo kernel: 0*4kB 2*8kB 2*16kB 0*32kB 1*64kB 13*128kB 
7*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 5104kB)
Apr 30 15:45:03 wiwbleo kernel: 19*4kB 0*8kB 1*16kB 1*32kB 1*64kB 1*128kB 
0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 828kB)
Apr 30 15:45:05 wiwbleo kernel: Swap cache: add 2137150, delete 2137074, 
find 5527644/5746138, race 0+12
Apr 30 15:45:05 wiwbleo kernel: 7873 pages of slabcache
Apr 30 15:45:06 wiwbleo kernel: 2328 pages of kernel stacks
Apr 30 15:45:06 wiwbleo kernel: 0 lowmem pagetables, 13825 highmem 
pagetables
Apr 30 15:45:06 wiwbleo kernel: 32 bounce buffer pages, 32 are on the 
emergency list
Apr 30 15:45:06 wiwbleo kernel: Free swap:            0kB
Apr 30 15:45:06 wiwbleo kernel: 524275 pages of RAM
Apr 30 15:45:06 wiwbleo kernel: 294899 pages of HIGHMEM
Apr 30 15:45:06 wiwbleo kernel: 10414 reserved pages
Apr 30 15:45:06 wiwbleo kernel: 387065 pages shared
Apr 30 15:45:06 wiwbleo kernel: 90 pages swap cached
Apr 30 15:45:06 wiwbleo kernel: Out of Memory: Killed process 23209 
(hpsmhd).
...
Apr 30 15:46:52 wiwbleo kernel: Mem-info:
Apr 30 15:46:52 wiwbleo kernel: Zone:DMA freepages:  2880 min:     0 low:  
  0 high:     0
Apr 30 15:46:52 wiwbleo kernel: Zone:Normal freepages:  1277 min:  1278 
low:  4543 high:  6303
Apr 30 15:46:52 wiwbleo kernel: Zone:HighMem freepages:   254 min:   255 
low:  4606 high:  6909
Apr 30 15:46:52 wiwbleo kernel: Free pages:        4411 (   254 HighMem)
Apr 30 15:46:52 wiwbleo kernel: ( Active: 477887/384, inactive_laundry: 
126, inactive_clean: 0, free: 4411 )
Apr 30 15:46:52 wiwbleo kernel:   aa:0 ac:0 id:0 il:0 ic:0 fr:2880
Apr 30 15:46:52 wiwbleo kernel:   aa:196156 ac:1940 id:190 il:112 ic:0 
fr:1277
Apr 30 15:46:52 wiwbleo kernel:   aa:277064 ac:2744 id:192 il:0 ic:0 
fr:254
Apr 30 15:46:52 wiwbleo kernel: 0*4kB 0*8kB 2*16kB 3*32kB 4*64kB 1*128kB 
1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11520kB)
Apr 30 15:46:52 wiwbleo kernel: 17*4kB 2*8kB 0*16kB 1*32kB 0*64kB 13*128kB 
7*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 5108kB)
Apr 30 15:46:52 wiwbleo kernel: 4*4kB 5*8kB 8*16kB 0*32kB 3*64kB 1*128kB 
0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1016kB)
Apr 30 15:46:52 wiwbleo kernel: Swap cache: add 2194745, delete 2194683, 
find 5537099/5771879, race 0+78
Apr 30 15:46:52 wiwbleo kernel: 7642 pages of slabcache
Apr 30 15:46:52 wiwbleo kernel: 2288 pages of kernel stacks
Apr 30 15:46:52 wiwbleo kernel: 0 lowmem pagetables, 13869 highmem 
pagetables
Apr 30 15:46:52 wiwbleo kernel: 32 bounce buffer pages, 32 are on the 
emergency list
Apr 30 15:46:52 wiwbleo kernel: Free swap:            0kB
Apr 30 15:46:53 wiwbleo kernel: 524275 pages of RAM
Apr 30 15:46:53 wiwbleo kernel: 294899 pages of HIGHMEM
Apr 30 15:46:53 wiwbleo kernel: 10414 reserved pages
Apr 30 15:46:53 wiwbleo kernel: 388858 pages shared
Apr 30 15:46:53 wiwbleo kernel: 80 pages swap cached
Apr 30 15:46:53 wiwbleo kernel: Out of Memory: Killed process 32399 
(httpd).
...
Apr 30 15:52:52 wiwbleo kernel: Mem-info:
Apr 30 15:52:52 wiwbleo kernel: Zone:DMA freepages:  2880 min:     0 low:  
  0 high:     0
Apr 30 15:52:52 wiwbleo kernel: Zone:Normal freepages:  1276 min:  1278 
low:  4543 high:  6303
Apr 30 15:52:52 wiwbleo kernel: Zone:HighMem freepages:   254 min:   255 
low:  4606 high:  6909
Apr 30 15:52:52 wiwbleo kernel: Free pages:        4410 (   254 HighMem)
Apr 30 15:52:52 wiwbleo kernel: ( Active: 478356/738, inactive_laundry: 0, 
inactive_clean: 0, free: 4410 )
Apr 30 15:52:52 wiwbleo kernel:   aa:0 ac:0 id:0 il:0 ic:0 fr:2880
Apr 30 15:52:52 wiwbleo kernel:   aa:196622 ac:1948 id:639 il:0 ic:0 
fr:1276
Apr 30 15:52:52 wiwbleo kernel:   aa:276938 ac:2740 id:209 il:0 ic:0 
fr:254
Apr 30 15:52:52 wiwbleo kernel: 0*4kB 0*8kB 2*16kB 3*32kB 4*64kB 1*128kB 
1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11520kB)
Apr 30 15:52:52 wiwbleo kernel: 1*4kB 1*8kB 2*16kB 6*32kB 0*64kB 12*128kB 
7*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 5100kB)
Apr 30 15:52:52 wiwbleo kernel: 0*4kB 5*8kB 7*16kB 1*32kB 3*64kB 1*128kB 
0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1016kB)
Apr 30 15:52:52 wiwbleo kernel: Swap cache: add 2324655, delete 2324567, 
find 5546679/5828169, race 0+134
Apr 30 15:52:52 wiwbleo kernel: 6776 pages of slabcache
Apr 30 15:52:52 wiwbleo kernel: 2304 pages of kernel stacks
Apr 30 15:52:52 wiwbleo kernel: 0 lowmem pagetables, 13979 highmem 
pagetables
Apr 30 15:52:52 wiwbleo kernel: 32 bounce buffer pages, 32 are on the 
emergency list
Apr 30 15:52:52 wiwbleo kernel: Free swap:            0kB
Apr 30 15:52:52 wiwbleo kernel: 524275 pages of RAM
Apr 30 15:52:52 wiwbleo kernel: 294899 pages of HIGHMEM
Apr 30 15:52:52 wiwbleo kernel: 10414 reserved pages
Apr 30 15:52:52 wiwbleo kernel: 394816 pages shared
Apr 30 15:52:52 wiwbleo kernel: 110 pages swap cached
Apr 30 15:52:52 wiwbleo kernel: Out of Memory: Killed process 1695 
(hpsmhd).

Andy Bach
Systems Mangler
Internet: andy_bach at wiwb.uscourts.gov
VOICE: (608) 261-5738  FAX 264-5932

"So it goes ...."
Kurt Vonnegut, Jr. (November 11, 1922 ? April 11, 2007)

More information about the luni mailing list