I am running a job in R that forks multiple processes. Every once in a while it seems that when the forked processes return and the next job starts, the previously forked processes terminate, but the CPU for those processes moves from 'User' to 'System'. So my activity monitor looks like this:
How can I debug this? How can I determine what processes are using the 'System' section in the screenshot? They do not show up if I sort by CPU in activity monitor, or on another tab (e.g., memory).
My activity monitor is showing 'all processes', so I do see running processes for the root user. However, all of the 'System' CPU is unaccounted for in activity monitor. And I tried showing 'system processes', but nothing with high cpu shows up there (which makes sense since these processes also show up for 'all processes').
The R use case may not be relevant. Or it may be. But generally, what I'd like to know is how to determine what is using the 'System' portion of the CPU load in OS X Mavericks.
EDIT: Here's an output of the top command. Same story. All % CPU under System is unnaccounted for in top, as well as in activity monitor. I have 12 physical cores (24 with hyper threading) on the machine, so 8 forked R processes each taking 100% cpu only accounts for 33% of CPU (what's under User).
Processes: 202 total, 10 running, 9 stuck, 183 sleeping, 845 threads 16:40:46
Load Avg: 16.81, 14.89, 11.47 CPU usage: 32.97% user, 35.27% sys, 31.74% idle SharedLibs: 10M resident, 3332K data, 0B linkedit. MemRegions: 74215 total, 8797M resident, 35M private, 14G shared.
PhysMem: 45G used (5360M wired), 12G unused. VM: 770G vsize, 1043M framework vsize, 905949(0) swapins, 1927120(0) swapouts. Networks: packets: 786361/1025M in, 669041/904M out. Disks: 1780468/188G read, 1536871/30G written.
PID COMMAND %CPU TIME #TH #WQ #PORT #MREG MEM RPRVT PURG CMPRS VPRVT VSIZE PGRP PPID STATE UID FAULTS COW MSGSENT MSGRECV SYSBSD SYSMACH CSW PAGEINS KPRVT KSHRD
27694 R 101.5 07:26.65 1/1 0 7 3055+ 7116M+ 812M+ 0B 0B 822M+ 22G+ 10309 15033 running 502 2429348+ 138079+ 36 16 11676+ 11415+ 29506+ 0 23M+ 231K+
27693 R 101.1 07:26.79 1/1 0 7 2917+ 7049M+ 751M+ 0B 0B 763M+ 21G+ 10309 15033 running 502 2326962+ 136821+ 34 14 11259+ 10022+ 23910+ 0 23M+ 201K+
27692 R 101.0 07:26.73 1/1 0 7 2850- 7132M- 781M- 0B 0B 797M- 21G- 10309 15033 running 502 2404303+ 141462+ 35 15 11751+ 11105+ 27449+ 0 23M- 182K-
27691 R 100.8 07:26.77 1/1 0 7 3310+ 7291M+ 957M+ 0B 0B 977M+ 22G+ 10309 15033 running 502 2374283+ 139669+ 34 15 11202+ 10358+ 25394+ 0 23M+ 284K+
27689 R 100.8 07:26.70 1/1 0 7 3155+ 7195M+ 881M+ 0B 0B 892M+ 22G+ 10309 15033 running 502 2449497+ 139793+ 37+ 17+ 11943+ 11515+ 29045+ 0 23M+ 251K+
27695 R 100.5 07:26.70 1/1 0 7 3401+ 7320M+ 988M+ 0B 0B 1007M+ 22G+ 10309 15033 running 502 2472322+ 144544+ 42 17 13105+ 11710+ 29017+ 0 23M+ 304K+
27690 R 100.1 07:26.71 1/1 0 7 3052+ 7261M+ 893M+ 0B 0B 913M+ 22G+ 10309 15033 running 502 2432181+ 140952+ 36 16 12080+ 11257+ 27092 0 23M+ 229K+
27688 R 99.5 07:26.66 1/1 0 7 3307+ 7366M+ 1013M+ 0B 0B 1036M+ 22G+ 10309 15033 running 502 2482511+ 139454+ 36 16 11331+ 11598+ 29773+ 0 23M+ 289K+
27720 top 20.2 00:15.51 1/1 0 22 74+ 3808K+ 3576K+ 0B 0B 207M+ 2566M+ 27720 470 running 0 83438+ 111 9267209+ 4633523+ 36139+ 4744288+ 1870+ 12 329K+ 138K+
133 WindowServer 3.7 24:37.23 4 0 435 2734 41M- 46M 51M+ 57M 479M 4043M 133 1 sleeping 88 1106029 3616 71798681+ 33280535+ 293955 47953977+ 13597819+ 2379 10M- 44M+
327 Terminal 2.3 27:26.62 8 1 198 1578 96M 90M 0B 271M 1805M 4258M 327 265 sleeping 502 3826356 17691 7388415+ 181612 42406285+ 15097554+ 9358587+ 2176 10M 57M+
26685 R 1.9 14:22.11 3 0 45 2434 35M 35M 0B 4682M 4988M 7395M 26685 472 sleeping 502 44308891 75552 1177048+ 1055 12394750+ 9626458+ 3543849+ 0 28M 16M