Strange rerults from a DreamHost cluster/server
Please read this thread: http://webhostingtalk.com/showthread.php?t=576487
The current stats of this server
Uptime_At_DreamHostIf you are interested in looking at the current stats of this server (and another one hosted on DreamHost).
Load average and CPU usage
Results of uptime;mpstat 1 10;uptime
First, typical output from one of the server at DreamHost.15:19:19 up 1:53, 0 users, load average: 5.74, 4.33, 3.76 Linux 2.4.32-grsec+f6b+gr217+nfs+a32+fuse23+tg+++p4+c8+gr2b-v6.194 01/10/07 15:19:19 CPU %user %nice %system %iowait %irq %soft %idle intr/s 15:19:20 all 88.50 0.00 4.00 0.00 0.00 0.00 7.50 2362.00 15:19:21 all 87.75 0.00 5.00 0.00 0.00 0.00 7.25 2743.00 15:19:22 all 88.25 0.00 7.50 0.00 0.00 0.00 4.25 4064.00 15:19:23 all 83.50 0.00 10.00 0.00 0.00 0.00 6.50 7730.00 15:19:24 all 82.75 0.00 4.00 0.00 0.00 0.00 13.25 2830.00 15:19:25 all 34.75 0.00 3.75 0.00 0.00 0.00 61.50 3692.00 15:19:26 all 54.25 0.00 6.00 0.00 0.00 0.00 39.75 2938.00 15:19:27 all 43.00 0.00 3.50 0.00 0.00 0.00 53.50 2243.00 15:19:28 all 29.50 0.00 1.25 0.00 0.00 0.00 69.25 2135.00 15:19:29 all 40.50 0.00 2.50 0.00 0.00 0.00 57.00 2282.00 Average: all 63.27 0.00 4.75 0.00 0.00 0.00 31.98 3301.90 15:19:29 up 1:54, 0 users, load average: 5.23, 4.26, 3.74
Judging from the results bellow, these server attain 100% CPU usage arount the load average of 10.10:19:57 up 23:00, 1 user, load average: 10.50, 10.54, 10.26 Linux 2.4.32-grsec+f6b+gr217+nfs+a32+fuse23+tg+++p4+c8+gr2b-v6.194 01/07/07 10:19:57 CPU %user %nice %system %iowait %irq %soft %idle intr/s 10:19:58 all 90.50 0.00 9.25 0.00 0.00 0.00 0.25 3119.00 10:19:59 all 89.50 0.00 10.50 0.00 0.00 0.00 0.00 2744.00 10:20:00 all 90.75 0.00 9.00 0.00 0.00 0.00 0.25 2551.00 10:20:01 all 83.75 0.00 16.00 0.00 0.00 0.00 0.25 2464.00 10:20:02 all 85.75 0.25 17.00 0.00 0.00 0.00 0.00 2637.00 10:20:03 all 85.50 0.00 14.50 0.00 0.00 0.00 0.00 3322.00 10:20:04 all 90.75 0.00 9.25 0.00 0.00 0.00 0.00 2633.00 10:20:05 all 92.00 0.00 8.00 0.00 0.00 0.00 0.00 3308.00 10:20:06 all 87.50 0.00 14.50 0.00 0.00 0.00 0.00 3067.00 10:20:07 all 91.75 0.00 8.25 0.00 0.00 0.00 0.00 3597.00 Average: all 88.78 0.03 11.62 0.00 0.00 0.00 0.00 2944.20 10:20:07 up 23:00, 1 user, load average: 10.52, 10.54, 10.26
06:00:08 up 1 day, 1:12, 2 users, load average: 13.44, 10.29, 8.93 Linux 2.4.32-grsec+f6b+gr217+nfs+a32+fuse23+tg+++p4+c8+gr2b-v6.194 01/09/07 06:00:08 CPU %user %nice %system %iowait %irq %soft %idle intr/s 06:00:09 all 82.50 0.00 17.50 0.00 0.00 0.00 0.00 3540.00 06:00:10 all 82.50 0.00 20.50 0.00 0.00 0.00 0.00 3488.00 06:00:11 all 80.25 0.00 19.75 0.00 0.00 0.00 0.00 5506.00 06:00:12 all 88.25 0.00 11.75 0.00 0.00 0.00 0.00 2909.00 06:00:13 all 88.25 0.00 11.75 0.00 0.00 0.00 0.00 3582.00 06:00:14 all 84.50 0.00 15.50 0.00 0.00 0.00 0.00 3701.00 06:00:15 all 91.25 0.00 10.75 0.00 0.00 0.00 0.00 3452.00 06:00:16 all 93.00 0.00 7.00 0.00 0.00 0.00 0.00 2836.00 06:00:17 all 93.25 0.00 6.75 0.00 0.00 0.00 0.00 5101.00 06:00:18 all 93.50 0.00 6.50 0.00 0.00 0.00 0.00 3603.00 Average: all 87.72 0.00 12.78 0.00 0.00 0.00 0.00 3771.80 06:00:18 up 1 day, 1:13, 2 users, load average: 13.91, 10.50, 9.02
Althought the loadaverage is high, and CPU is 100% useed, the server is responding without noticable delay.07:14:41 up 1 day, 2:27, 0 users, load average: 13.66, 10.42, 8.93 Linux 2.4.32-grsec+f6b+gr217+nfs+a32+fuse23+tg+++p4+c8+gr2b-v6.194 01/09/07 07:14:41 CPU %user %nice %system %iowait %irq %soft %idle intr/s 07:14:42 all 88.00 0.00 14.75 0.00 0.00 0.00 0.00 4304.00 07:14:43 all 89.50 0.00 13.50 0.00 0.00 0.00 0.00 3659.00 07:14:44 all 88.50 0.00 12.50 0.00 0.00 0.00 0.00 3073.00 07:14:45 all 94.50 0.00 9.50 0.00 0.00 0.00 0.00 2512.00 07:14:46 all 91.00 0.25 16.75 0.00 0.00 0.00 0.00 2361.00 07:14:47 all 80.00 0.00 22.00 0.00 0.00 0.00 0.00 4480.00 07:14:48 all 87.14 0.24 19.42 0.00 0.00 0.00 0.00 2814.56 07:14:49 all 85.25 0.25 14.50 0.00 0.00 0.00 0.00 2804.00 07:14:50 all 89.25 0.00 12.75 0.00 0.00 0.00 0.00 2913.00 07:14:51 all 86.25 0.50 15.00 0.00 0.00 0.00 0.00 2651.00 Average: all 87.94 0.12 15.08 0.00 0.00 0.00 0.00 3156.13 07:14:51 up 1 day, 2:27, 0 users, load average: 14.08, 10.62, 9.01 07:15:10 up 1 day, 2:27, 0 users, load average: 14.87, 11.02, 9.18 Linux 2.4.32-grsec+f6b+gr217+nfs+a32+fuse23+tg+++p4+c8+gr2b-v6.194 01/09/07 07:15:10 CPU %user %nice %system %iowait %irq %soft %idle intr/s 07:15:11 all 90.50 0.25 19.25 0.00 0.00 0.00 0.00 2448.00 07:15:12 all 83.17 0.00 15.84 0.00 0.00 0.00 0.99 2231.68 07:15:13 all 88.50 0.00 15.25 0.00 0.00 0.00 0.00 2547.00 07:15:14 all 90.59 0.00 13.61 0.00 0.00 0.00 0.00 2833.66 07:15:15 all 71.50 0.25 31.00 0.00 0.00 0.00 0.00 1849.00 07:15:16 all 78.22 0.25 21.78 0.00 0.00 0.00 0.00 2681.19 07:15:17 all 85.50 0.25 13.00 0.00 0.00 0.00 1.25 3344.00 07:15:18 all 86.25 0.00 13.75 0.00 0.00 0.00 0.00 3634.00 07:15:19 all 85.75 0.25 14.00 0.00 0.00 0.00 0.00 3515.00 07:15:20 all 79.75 0.25 20.00 0.00 0.00 0.00 0.00 3182.00 Average: all 83.97 0.15 17.75 0.00 0.00 0.00 0.00 2825.82 07:15:20 up 1 day, 2:27, 0 users, load average: 14.59, 11.08, 9.22 07:15:37 up 1 day, 2:28, 0 users, load average: 14.24, 11.19, 9.28 Linux 2.4.32-grsec+f6b+gr217+nfs+a32+fuse23+tg+++p4+c8+gr2b-v6.194 01/09/07 07:15:37 CPU %user %nice %system %iowait %irq %soft %idle intr/s 07:15:38 all 92.25 0.25 6.50 0.00 0.00 0.00 1.00 2838.00 07:15:39 all 94.25 0.75 5.00 0.00 0.00 0.00 0.00 2422.00 07:15:40 all 90.50 1.00 10.50 0.00 0.00 0.00 0.00 2229.00 07:15:41 all 89.25 0.50 14.25 0.00 0.00 0.00 0.00 3080.00 07:15:42 all 93.00 0.25 6.75 0.00 0.00 0.00 0.00 3502.00 07:15:43 all 94.25 0.25 5.50 0.00 0.00 0.00 0.00 4216.00 07:15:44 all 91.25 0.25 8.50 0.00 0.00 0.00 0.00 3412.00 07:15:45 all 94.55 0.25 10.40 0.00 0.00 0.00 0.00 2953.47 07:15:46 all 91.00 0.75 10.00 0.00 0.00 0.00 0.00 2607.00 07:15:47 all 96.75 0.50 2.75 0.00 0.00 0.00 0.00 2731.00 Average: all 92.71 0.47 8.02 0.00 0.00 0.00 0.00 2999.00 07:15:47 up 1 day, 2:28, 0 users, load average: 13.74, 11.18, 9.30 07:16:04 up 1 day, 2:28, 0 users, load average: 12.88, 11.15, 9.33 Linux 2.4.32-grsec+f6b+gr217+nfs+a32+fuse23+tg+++p4+c8+gr2b-v6.194 01/09/07 07:16:04 CPU %user %nice %system %iowait %irq %soft %idle intr/s 07:16:05 all 80.94 9.16 34.90 0.00 0.00 0.00 0.00 1622.77 07:16:06 all 85.10 5.30 34.60 0.00 0.00 0.00 0.00 1573.74 07:16:07 all 76.75 10.75 32.75 0.00 0.00 0.00 0.00 1713.00 07:16:08 all 74.50 10.40 35.89 0.00 0.00 0.00 0.00 1748.51 07:16:09 all 86.00 11.75 31.75 0.00 0.00 0.00 0.00 1698.00 07:16:10 all 94.75 4.25 36.25 0.00 0.00 0.00 0.00 1727.00 07:16:11 all 85.25 1.50 31.75 0.00 0.00 0.00 0.00 1666.00 07:16:12 all 91.25 0.00 36.25 0.00 0.00 0.00 0.00 1698.00 07:16:13 all 98.25 0.00 35.00 0.00 0.00 0.00 0.00 1871.00 07:16:14 all 93.50 0.00 34.75 0.00 0.00 0.00 0.00 1898.00 Average: all 86.61 5.32 34.39 0.00 0.00 0.00 0.00 1721.68 07:16:14 up 1 day, 2:28, 0 users, load average: 14.86, 11.62, 9.50 07:16:31 up 1 day, 2:28, 0 users, load average: 14.13, 11.62, 9.54 Linux 2.4.32-grsec+f6b+gr217+nfs+a32+fuse23+tg+++p4+c8+gr2b-v6.194 01/09/07 07:16:31 CPU %user %nice %system %iowait %irq %soft %idle intr/s 07:16:32 all 72.75 11.50 8.25 0.00 0.00 0.00 7.50 2236.00 07:16:33 all 82.25 6.00 9.50 0.00 0.00 0.00 2.25 2473.00 07:16:34 all 71.75 11.50 9.75 0.00 0.00 0.00 7.00 3023.00 07:16:35 all 86.88 4.70 10.40 0.00 0.00 0.00 0.00 2547.52 07:16:36 all 70.83 14.22 29.66 0.00 0.00 0.00 0.00 1394.12 07:16:37 all 79.46 10.40 35.15 0.00 0.00 0.00 0.00 1789.11 07:16:38 all 74.00 9.25 28.50 0.00 0.00 0.00 0.00 1696.00 07:16:39 all 78.00 9.75 7.00 0.00 0.00 0.00 5.25 2970.00 07:16:40 all 56.00 14.25 30.75 0.00 0.00 0.00 0.00 1788.00 07:16:41 all 55.45 18.56 30.69 0.00 0.00 0.00 0.00 1659.41 Average: all 72.74 11.02 20.00 0.00 0.00 0.00 0.00 2155.62 07:16:41 up 1 day, 2:29, 0 users, load average: 14.24, 11.72, 9.59 07:16:59 up 1 day, 2:29, 0 users, load average: 15.84, 12.25, 9.81 Linux 2.4.32-grsec+f6b+gr217+nfs+a32+fuse23+tg+++p4+c8+gr2b-v6.194 01/09/07 07:16:59 CPU %user %nice %system %iowait %irq %soft %idle intr/s 07:17:00 all 80.75 3.25 15.25 0.00 0.00 0.00 0.75 3542.00 07:17:01 all 95.25 0.75 23.00 0.00 0.00 0.00 0.00 3002.00 07:17:02 all 81.00 0.75 18.25 0.00 0.00 0.00 0.00 3068.00 07:17:03 all 87.75 1.75 10.50 0.00 0.00 0.00 0.00 3063.00 07:17:04 all 89.75 2.00 8.25 0.00 0.00 0.00 0.00 3226.00 07:17:05 all 87.00 4.50 8.50 0.00 0.00 0.00 0.00 2760.00 07:17:06 all 78.71 7.67 29.70 0.00 0.00 0.00 0.00 1958.42 07:17:07 all 71.25 13.50 10.50 0.00 0.00 0.00 4.75 2907.00 07:17:08 all 81.75 2.50 15.00 0.00 0.00 0.00 0.75 2410.00 07:17:09 all 88.50 2.25 9.00 0.00 0.00 0.00 0.25 3051.00 Average: all 84.17 3.90 14.81 0.00 0.00 0.00 0.00 2897.80 07:17:09 up 1 day, 2:29, 0 users, load average: 15.17, 12.23, 9.83
07:20:54 up 1 day, 2:33, 0 users, load average: 10.27, 11.07, 9.83 Linux 2.4.32-grsec+f6b+gr217+nfs+a32+fuse23+tg+++p4+c8+gr2b-v6.194 01/09/07 07:20:54 CPU %user %nice %system %iowait %irq %soft %idle intr/s 07:20:55 all 95.00 0.00 4.25 0.00 0.00 0.00 0.75 1816.00 07:20:56 all 95.50 0.00 4.50 0.00 0.00 0.00 0.00 1968.00 07:20:57 all 95.25 0.00 4.75 0.00 0.00 0.00 0.00 2483.00 07:20:58 all 94.50 0.00 6.50 0.00 0.00 0.00 0.00 1900.00 07:20:59 all 91.75 0.00 8.25 0.00 0.00 0.00 0.00 2482.00 07:21:00 all 94.25 0.00 5.75 0.00 0.00 0.00 0.00 2190.00 07:21:01 all 95.00 0.00 5.50 0.00 0.00 0.00 0.00 2451.00 07:21:02 all 91.75 0.00 7.75 0.00 0.00 0.00 0.50 2147.00 07:21:03 all 86.50 0.00 13.50 0.00 0.00 0.00 0.00 3623.00 07:21:04 all 91.50 0.00 8.50 0.00 0.00 0.00 0.00 1877.00 Average: all 93.10 0.00 6.92 0.00 0.00 0.00 0.00 2293.70 07:21:04 up 1 day, 2:33, 0 users, load average: 11.03, 11.20, 9.89I noticed the increase of Nice % when the load average goes up. This host(DreamHost) might be changing the priority (nice level) of some processes depending on the server load.
Server crash
This is the output of uptime ran by cron, each minuteAs you can see, the server crashed arount 13:13.The site must have been unavailable from 12:46 till 13:31.Clearly, the server monitoring/controlling system couldn't handle the situation, properly.12:33:01 up 2 days, 7:39, 0 users, load average: 4.84, 7.25, 7.46 12:34:01 up 2 days, 7:40, 0 users, load average: 5.29, 6.93, 7.33 12:35:01 up 2 days, 7:41, 0 users, load average: 4.79, 6.57, 7.19 12:36:01 up 2 days, 7:42, 0 users, load average: 7.00, 6.85, 7.24 12:37:01 up 2 days, 7:43, 0 users, load average: 7.75, 7.03, 7.27 12:38:02 up 2 days, 7:44, 0 users, load average: 8.62, 7.42, 7.39 12:39:01 up 2 days, 7:45, 0 users, load average: 8.67, 7.63, 7.46 12:40:01 up 2 days, 7:46, 0 users, load average: 8.23, 7.72, 7.50 12:41:01 up 2 days, 7:47, 0 users, load average: 6.45, 7.26, 7.35 12:42:01 up 2 days, 7:48, 0 users, load average: 7.98, 7.59, 7.46 12:43:56 up 2 days, 7:50, 0 users, load average: 208.77, 69.47, 29.14 12:46:26 up 2 days, 7:52, 0 users, load average: 596.31, 266.96, 107.82 12:46:44 up 2 days, 7:52, 0 users, load average: 726.53, 363.20, 150.80 12:53:25 up 2 days, 7:59, 0 users, load average: 915.28, 286.69, 581.75 12:57:34 up 2 days, 8:03, 0 users, load average: 47.13, 933.70, 780.02 12:55:30 up 2 days, 8:01, 0 users, load average: 188.07, 133.87, 841.74 13:05:31 up 2 days, 8:11, 0 users, load average: 79.39, 847.23, 712.73 13:13:31 up 2 days, 8:20, 0 users, load average: 47.24, 67.93, 762.62 13:31:01 up 5 min, 2 users, load average: 4.84, 2.94, 1.22 13:32:01 up 6 min, 2 users, load average: 3.98, 3.07, 1.37 13:33:01 up 7 min, 2 users, load average: 3.93, 3.14, 1.49 13:34:01 up 8 min, 2 users, load average: 4.02, 3.36, 1.67 13:35:01 up 9 min, 2 users, load average: 3.26, 3.25, 1.74 13:36:01 up 10 min, 2 users, load average: 5.16, 3.69, 1.97 13:37:01 up 11 min, 2 users, load average: 5.38, 4.06, 2.21 13:38:02 up 12 min, 2 users, load average: 4.11, 3.98, 2.29 13:39:01 up 13 min, 2 users, load average: 4.40, 4.08, 2.44 13:40:01 up 14 min, 2 users, load average: 4.62, 4.24, 2.59 13:41:01 up 15 min, 2 users, load average: 4.24, 4.30, 2.72
Server reboot
As announced by the DreamHost status page:http://www.dreamhoststatus.com/2007/01/24/server-relocation-bonanza/#commentsIt took exactly 20 minutes as they said.cerritos 22:34:00 up 18 days, 6:36, 1 user, load average: 5.15, 4.24, 3.70 cerritos 22:35:00 up 18 days, 6:37, 1 user, load average: 4.81, 4.43, 3.81 cerritos 22:36:00 up 18 days, 6:38, 1 user, load average: 4.15, 4.29, 3.80 cerritos 22:37:00 up 18 days, 6:39, 1 user, load average: 3.21, 4.01, 3.73 cerritos 22:38:01 up 18 days, 6:40, 1 user, load average: 3.91, 4.04, 3.75 cerritos 22:38:01 up 18 days, 6:40, 1 user, load average: 3.91, 4.04, 3.75 cerritos 22:38:00 up 18 days, 6:40, 1 user, load average: 3.91, 4.04, 3.75 cerritos 22:39:01 up 18 days, 6:41, 1 user, load average: 2.70, 3.67, 3.64 cerritos 22:39:01 up 18 days, 6:41, 1 user, load average: 2.70, 3.67, 3.64 cerritos 22:39:01 up 18 days, 6:41, 1 user, load average: 2.70, 3.67, 3.64 cerritos 23:19:01 up 6 min, 0 users, load average: 5.63, 3.39, 1.41 cerritos 23:20:01 up 7 min, 0 users, load average: 5.93, 3.84, 1.69 cerritos 23:21:01 up 8 min, 0 users, load average: 4.00, 3.72, 1.78 cerritos 23:22:02 up 9 min, 0 users, load average: 4.79, 4.02, 2.00 cerritos 23:23:01 up 10 min, 0 users, load average: 4.86, 4.23, 2.20 cerritos 23:24:02 up 11 min, 0 users, load average: 5.64, 4.61, 2.46
Load average and other resources
Results of "uptime;vmstat 1 10;uptime"07:10:48 up 17:43, 2 users, load average: 8.25, 7.43, 7.23 procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 54032 121364 4468 946384 0 0 5 20 116 84 50 6 44 0 0 0 54032 86064 4452 944896 0 0 0 0 4119 5619 92 8 0 0 0 0 54032 85136 4436 939452 0 0 0 0 4018 4485 92 8 0 0 0 0 54032 103448 4412 921336 0 0 0 0 3985 4129 90 10 0 0 0 0 54032 91856 4412 920892 0 0 0 0 3242 4265 95 5 0 0 0 0 54032 85832 4336 904364 0 0 0 368 3860 3821 89 11 0 0 0 0 54032 94112 4280 895176 0 0 0 0 4037 4233 93 7 0 0 0 0 54032 95288 4280 891236 0 0 0 0 3580 3617 94 6 0 0 0 0 54032 98440 4280 892304 0 0 0 0 3795 3804 94 6 0 0 0 0 54032 111788 4280 893324 0 0 0 0 3547 5191 94 6 0 0 07:10:58 up 17:43, 2 users, load average: 9.53, 7.73, 7.33
Hostwick.com --- Generated on 2007-01-25_07:34:30