Trace number 4396853

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
cosoco 2.0? (TO) 2519.94 2520.01

General information on the benchmark

NameLatinSquare/LatinSquare-m1-gp/
qwh-o30-h374-08.xml
MD5SUMa38db4eec3f77542c2f3ff7a616038a8
Bench CategoryCSP (decision problem)
Best result obtained on this benchmarkSAT
Best value of the objective obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark9.15382
Satisfiable
(Un)Satisfiability was proved
Number of variables900
Number of constraints2
Number of domains1
Minimum domain size30
Maximum domain size30
Distribution of domain sizes[{"size":30,"count":900}]
Minimum variable degree1
Maximum variable degree2
Distribution of variable degrees[{"degree":1,"count":374},{"degree":2,"count":526}]
Minimum constraint arity526
Maximum constraint arity900
Distribution of constraint arities[{"arity":526,"count":1},{"arity":900,"count":1}]
Number of extensional constraints0
Number of intensional constraints0
Distribution of constraint types[{"type":"allDifferent","count":1},{"type":"instantiation","count":1}]
Optimization problemNO
Type of objective

Solver Data 

0.00/0.02	c root propagations     : 4686        
0.00/0.02	c restarts enabled, print stats at every restarts
0.00/0.02	c
0.00/0.02	c
0.00/0.02	c
0.00/0.02	c This is cosoco 2.00 --  
0.00/0.02	c
0.00/0.02	c command line: cosoco -lc -model -restarts -stick HOME/instance-4396853-1568593524.xml 
0.00/0.02	c ========================================[ Problem Statistics ]===========================================
0.00/0.02	c |                                                                                                       
0.00/0.02	c |  Parse time        :         0.01 s 
0.00/0.02	c |
0.00/0.02	c |               Variables: 900
0.00/0.02	c |            Domain Sizes: 30..30
0.00/0.02	c |
0.00/0.02	c |             Constraints: 586
0.00/0.02	c |                   Arity: 1..30  -- Unary: 526
0.00/0.02	c | 
0.00/0.02	c |                   Types: 
0.00/0.02	c |                          All Different: 60
0.00/0.02	c |                          Unary: 526
0.00/0.02	c |                          
0.00/0.02	c =========================================================================================================
0.00/0.02	c conflicts      decisions      filterCalls    propagations   Root Prop.     Nogoods        min/max/avg depth
0.00/0.07	c 101            1093           13179          45982          894            0              582   684   642
0.10/0.12	c 211            2209           21862          73523          894            0              280   506   465
0.21/0.23	c 332            3925           42218          158680         894            0              310   703   625
0.21/0.27	c 465            5037           50503          186124         894            0              522   666   637
0.31/0.36	c 611            6331           66496          253747         894            0              522   695   661
0.41/0.42	c 771            7517           77822          296290         894            0              562   621   577
0.51/0.52	c 947            9306           97385          370959         894            0              506   678   649
0.51/0.56	c 1140           10550          105568         396370         894            0              516   576   555
0.60/0.69	c 1352           12581          128214         489334         894            0              516   688   648
0.79/0.84	c 1585           14716          156171         614146         894            0              543   706   659
1.00/1.05	c 1841           17218          194760         783460         894            0              580   707   684
1.29/1.34	c 2122           20331          247599         1034962        894            0              642   752   715
1.59/1.62	c 2431           23324          298698         1299131        894            0              576   746   709
1.69/1.78	c 2770           25398          326845         1440641        894            0              544   709   676
2.10/2.12	c 3142           28427          389079         1741414        894            0              587   720   690
2.49/2.54	c 3551           32016          462560         2119324        894            0              644   732   699
3.00/3.03	c 4000           36457          552587         2562692        894            0              570   717   686
3.39/3.46	c 4493           39740          628175         2929314        894            0              601   715   674
3.89/3.95	c 5035           43424          713948         3388845        894            0              588   698   672
4.39/4.45	c 5631           47805          804171         3883509        894            0              570   721   691
5.00/5.07	c 6286           52882          914900         4422622        894            0              590   711   683
5.49/5.53	c 7006           57331          997513         4837039        894            0              611   710   685
6.29/6.31	c 7798           63117          1137548        5548190        894            0              604   717   690
6.89/6.96	c 8669           68076          1253230        6109807        894            0              640   709   675
7.69/7.74	c 9627           74425          1393676        6865590        894            0              631   723   700
8.79/8.85	c 10680          84034          1591802        7896626        894            0              593   743   694
9.59/9.60	c 11838          90620          1725594        8632495        894            0              585   743   697
10.69/10.71	c 13111          99367          1921184        9674150        894            0              566   736   693
11.49/11.51	c 14511          106451         2065414        10372524       894            0              561   724   677
12.59/12.68	c 16051          117297         2276027        11380671       894            0              563   710   676
14.10/14.17	c 17745          129285         2543245        12597062       894            0              562   709   663
15.29/15.35	c 19608          139272         2752956        13683694       894            0              571   700   665
16.49/16.53	c 21657          150719         2966387        14718880       894            0              572   718   679
18.10/18.19	c 23910          163318         3260931        16122336       894            0              553   710   676
20.10/20.13	c 26388          178554         3609261        17904852       894            0              554   725   684
22.59/22.60	c 29113          200914         4051777        20080261       894            0              565   726   678
24.59/24.66	c 32110          222530         4421327        21991471       894            0              551   728   693
26.49/26.55	c 35406          240589         4758849        23612235       894            0              552   707   658
29.89/29.98	c 39031          267783         5368391        26779771       894            0              543   732   687
31.39/31.45	c 43018          284031         5633109        28118275       894            0              547   719   666
34.99/35.05	c 47403          312524         6276704        31390952       894            0              539   726   695
37.58/37.65	c 52226          332779         6742031        33725318       894            0              542   707   681
41.68/41.74	c 57531          365655         7481687        37328983       894            0              549   746   697
46.09/46.17	c 63366          395361         8261631        41222321       894            0              557   707   670
52.38/52.46	c 69784          438717         9375925        46994191       894            0              576   730   689
59.99/60.05	c 76843          492616         10718315       54186006       894            0              616   743   700
66.48/66.58	c 84607          539562         11883628       60091614       894            0              559   740   687
73.99/74.10	c 93147          607140         13247922       67202530       894            0              580   751   715
81.58/81.66	c 102541         675233         14604652       74402834       894            0              592   745   706
89.99/90.03	c 112874         736598         16086664       81805777       894            0              614   710   670
102.10/102.11	c 124240         814294         18220341       92850582       894            0              602   736   692
112.49/112.54	c 136742         901967         20086955       101915225      894            0              606   742   692
122.19/122.20	c 150494         980651         21799836       111032583      894            0              612   731   686
134.39/134.41	c 165621         1074856        23973181       122105217      894            0              625   735   687
152.59/152.66	c 182260         1220816        27244000       139087546      894            0              613   759   697
170.00/170.03	c 200562         1354632        30344866       155726009      894            0              632   754   704
189.79/189.87	c 220694         1486604        33877357       173662186      894            0              582   731   693
212.99/213.05	c 242839         1672946        38015143       194706694      894            0              640   736   689
238.38/238.48	c 267198         1843422        42515795       218129558      894            0              642   735   692
262.58/262.69	c 293992         2009026        46765832       240721885      894            0              644   745   692
288.28/288.33	c 323465         2210167        51322635       262961959      894            0              622   735   687
310.88/310.97	c 355885         2398919        55349503       284488759      894            0              573   745   700
335.28/335.34	c 391547         2606422        59702011       306401289      894            0              591   735   690
365.38/365.40	c 430775         2863963        65039171       335721613      894            0              613   735   690
398.68/398.77	c 473925         3135394        70996386       365041203      894            0              640   751   699
432.88/432.95	c 521390         3413700        77117504       397177528      894            0              581   737   693
471.37/471.47	c 573601         3709413        84005626       431073944      894            0              650   747   698
517.27/517.35	c 631033         4086463        92155871       474408759      894            0              642   740   690
565.87/565.95	c 694208         4479298        100807282      521956334      894            0              584   751   691
640.98/641.06	c 763700         4953998        114094316      592149836      894            0              622   742   686
707.56/707.67	c 840141         5457953        125922565      653287654      894            0              594   743   695
776.77/776.82	c 924226         5912485        138193367      714223958      894            0              637   752   683
837.17/837.21	c 1016719        6364658        148861929      772175319      894            0              611   723   681
914.87/914.93	c 1118461        6959384        162606608      844979640      894            0              626   740   682
1003.07/1003.18	c 1230377        7635687        178177998      926718954      894            0              587   753   689
1106.46/1106.52	c 1353484        8401368        196482658      1017956693     894            0              627   742   684
1207.07/1207.14	c 1488901        9206619        214362789      1108108898     894            0              563   739   687
1288.85/1288.94	c 1637859        9915562        228907286      1187204812     894            0              558   746   696
1418.15/1418.23	c 1801712        11028792       251995542      1307307049     894            0              574   750   692
1567.64/1567.71	c 1981950        12275670       278694134      1447862538     894            0              550   745   695
1727.34/1727.48	c 2180211        13475042       307128856      1608528943     894            0              557   753   702
1916.73/1916.89	c 2398298        14937815       340727938      1790374016     894            0              564   752   697
2099.83/2099.93	c 2638193        16275468       373280813      1966164653     894            0              616   738   684
2304.82/2304.91	c 2902077        17854637       409954314      2156328423     894            0              600   730   683
2519.92/2520.00	
2519.92/2520.00	c *** INTERRUPTED ***
2519.92/2520.00	c restarts              : 84          
2519.92/2520.00	c conflicts             : 3175682      (1260 /sec)
2519.92/2520.00	c decisions             : 19428174     (7710 /sec)
2519.92/2520.00	c propagations          : 2365445146  (938702 /sec)
2519.92/2520.00	c root propagations     : 4686        
2519.92/2520.00	c filter calls          : 448095325      (177822 /sec)
2519.92/2520.00	c useless filter calls  : 111036947      (24 %)
2519.92/2520.00	c Memory used           : 32.52 MB
2519.92/2520.00	c CPU time              : 2519.91 s
2519.92/2520.00	c real time             : 2520 s
2519.92/2520.00	
2519.92/2520.00	s UNKNOWN

Verifier Data


Watcher Data 

runsolver version 3.4.0 (svn: 3012) Copyright (C) 2010-2013 Olivier ROUSSEL

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

NUMA information:
  number of nodes: 2
  memory of node 0: 16374 MiB (13002 MiB free)
  memory of node 1: 16384 MiB (15532 MiB free)
  node distances:
     10  21
     21  10

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-4396853-1568593524/watcher-4396853-1568593524 -o /tmp/evaluation-result-4396853-1568593524/solver-4396853-1568593524 -C 2520 -W 2520 -M 15500 --bin-var /tmp/runsolver-exch-node103-1568593336-23101 --watchdog 2580 cosoco -lc -model -restarts -stick HOME/instance-4396853-1568593524.xml 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 2520 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 2550 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 2520 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Linux 3.10.0-514.16.1.el7.x86_64
Starting watchdog thread
Current StackSize limit: 8192 KiB
solver pid=23201, runsolver pid=23198


[startup+0.100136 s]*
/proc/loadavg: 1.89 1.38 1.46 3/199 23202
/proc/meminfo: memFree=29215040/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=32504 memory=8932 CPUtime=0.1 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 2423 0 0 0 10 0 0 0 20 0 1 0 435605507 33284096 2233 33554432000 4194304 6211092 140720340203840 140720340200376 5409424 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8126 2233 864 493 0 1431 0
Current children cumulated CPU time: 0.1 s
Current children cumulated vsize: 32504 KiB
Current children cumulated memory: 8932 KiB

[startup+0.209658 s]*
/proc/loadavg: 1.89 1.38 1.46 3/199 23202
/proc/meminfo: memFree=29214332/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=32900 memory=9460 CPUtime=0.21 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 2538 0 0 0 21 0 0 0 20 0 1 0 435605507 33689600 2365 33554432000 4194304 6211092 140720340203840 140720340200376 5075694 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8225 2365 866 493 0 1530 0
Current children cumulated CPU time: 0.21 s
Current children cumulated vsize: 32900 KiB
Current children cumulated memory: 9460 KiB

[startup+0.309061 s]*
/proc/loadavg: 1.89 1.38 1.46 3/199 23202
/proc/meminfo: memFree=29214332/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33032 memory=9460 CPUtime=0.31 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 2552 0 0 0 31 0 0 0 20 0 1 0 435605507 33824768 2365 33554432000 4194304 6211092 140720340203840 140720340200376 5079261 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8258 2365 866 493 0 1563 0
Current children cumulated CPU time: 0.31 s
Current children cumulated vsize: 33032 KiB
Current children cumulated memory: 9460 KiB

[startup+0.700285 s]
/proc/loadavg: 1.89 1.38 1.46 3/199 23202
/proc/meminfo: memFree=29214332/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33032 memory=9460 CPUtime=0.69 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 2565 0 0 0 69 0 0 0 20 0 1 0 435605507 33824768 2365 33554432000 4194304 6211092 140720340203840 140720340201800 5193733 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8258 2365 866 493 0 1563 0
Current children cumulated CPU time: 0.69 s
Current children cumulated vsize: 33032 KiB
Current children cumulated memory: 9460 KiB

[startup+1.50027 s]
/proc/loadavg: 1.89 1.39 1.47 3/199 23202
/proc/meminfo: memFree=29213544/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33032 memory=9460 CPUtime=1.49 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 2575 0 0 0 149 0 0 0 20 0 1 0 435605507 33824768 2365 33554432000 4194304 6211092 140720340203840 140720340201800 5074703 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8258 2365 866 493 0 1563 0
Current children cumulated CPU time: 1.49 s
Current children cumulated vsize: 33032 KiB
Current children cumulated memory: 9460 KiB

[startup+3.10951 s]
/proc/loadavg: 1.89 1.39 1.47 3/199 23202
/proc/meminfo: memFree=29206048/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33032 memory=9652 CPUtime=3.1 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 4030 0 0 0 310 0 0 0 20 0 1 0 435605507 33824768 2413 33554432000 4194304 6211092 140720340203840 140720340201800 5081706 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8258 2413 866 493 0 1563 0
Current children cumulated CPU time: 3.1 s
Current children cumulated vsize: 33032 KiB
Current children cumulated memory: 9652 KiB

[startup+6.30026 s]
/proc/loadavg: 1.90 1.40 1.47 3/199 23202
/proc/meminfo: memFree=29206000/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33164 memory=9672 CPUtime=6.29 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 5496 0 0 0 629 0 0 0 20 0 1 0 435605507 33959936 2418 33554432000 4194304 6211092 140720340203840 140720340201800 5079333 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8291 2418 866 493 0 1596 0
Current children cumulated CPU time: 6.29 s
Current children cumulated vsize: 33164 KiB
Current children cumulated memory: 9672 KiB

[startup+12.7003 s]
/proc/loadavg: 1.91 1.41 1.47 3/199 23202
/proc/meminfo: memFree=29202776/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33164 memory=9688 CPUtime=12.69 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 6963 0 0 0 1269 0 0 0 20 0 1 0 435605507 33959936 2422 33554432000 4194304 6211092 140720340203840 140720340201800 5075563 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8291 2422 866 493 0 1596 0
Current children cumulated CPU time: 12.69 s
Current children cumulated vsize: 33164 KiB
Current children cumulated memory: 9688 KiB

[startup+25.5003 s]
/proc/loadavg: 1.92 1.43 1.48 3/199 23204
/proc/meminfo: memFree=29198000/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33164 memory=9712 CPUtime=25.49 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 9897 0 0 0 2549 0 0 0 20 0 1 0 435605507 33959936 2428 33554432000 4194304 6211092 140720340203840 140720340201800 5081375 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8291 2428 866 493 0 1596 0
Current children cumulated CPU time: 25.49 s
Current children cumulated vsize: 33164 KiB
Current children cumulated memory: 9712 KiB

[startup+51.1094 s]
/proc/loadavg: 1.95 1.48 1.50 3/199 23206
/proc/meminfo: memFree=29194808/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33164 memory=9720 CPUtime=51.09 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 11369 0 0 0 5109 0 0 0 20 0 1 0 435605507 33959936 2430 33554432000 4194304 6211092 140720340203840 140720340201800 5081706 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8291 2430 866 493 0 1596 0
Current children cumulated CPU time: 51.09 s
Current children cumulated vsize: 33164 KiB
Current children cumulated memory: 9720 KiB

[startup+102.3 s]
/proc/loadavg: 1.98 1.56 1.52 3/199 23209
/proc/meminfo: memFree=29149040/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33164 memory=9752 CPUtime=102.29 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 12853 0 0 0 10228 1 0 0 20 0 1 0 435605507 33959936 2438 33554432000 4194304 6211092 140720340203840 140720340201800 5393282 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8291 2438 866 493 0 1596 0
Current children cumulated CPU time: 102.29 s
Current children cumulated vsize: 33164 KiB
Current children cumulated memory: 9752 KiB

[startup+162.3 s]
/proc/loadavg: 1.99 1.64 1.55 3/200 23216
/proc/meminfo: memFree=29122848/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33164 memory=9764 CPUtime=162.29 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 14331 0 0 0 16228 1 0 0 20 0 1 0 435605507 33959936 2441 33554432000 4194304 6211092 140720340203840 140720340201800 5076360 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8291 2441 866 493 0 1596 0
Current children cumulated CPU time: 162.29 s
Current children cumulated vsize: 33164 KiB
Current children cumulated memory: 9764 KiB

[startup+222.3 s]
/proc/loadavg: 2.15 1.74 1.59 3/200 23221
/proc/meminfo: memFree=29116988/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33164 memory=9772 CPUtime=222.28 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 15809 0 0 0 22227 1 0 0 20 0 1 0 435605507 33959936 2443 33554432000 4194304 6211092 140720340203840 140720340201800 5081205 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8291 2443 866 493 0 1596 0
Current children cumulated CPU time: 222.28 s
Current children cumulated vsize: 33164 KiB
Current children cumulated memory: 9772 KiB

[startup+282.3 s]
/proc/loadavg: 2.40 1.90 1.66 3/200 23225
/proc/meminfo: memFree=29113892/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33164 memory=9780 CPUtime=282.28 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 17285 0 0 0 28227 1 0 0 20 0 1 0 435605507 33959936 2445 33554432000 4194304 6211092 140720340203840 140720340201800 5081280 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8291 2445 866 493 0 1596 0
Current children cumulated CPU time: 282.28 s
Current children cumulated vsize: 33164 KiB
Current children cumulated memory: 9780 KiB

[startup+342.3 s]
/proc/loadavg: 2.23 1.94 1.69 3/201 23239
/proc/meminfo: memFree=29111144/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33164 memory=9788 CPUtime=342.28 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 18765 0 0 0 34227 1 0 0 20 0 1 0 435605507 33959936 2447 33554432000 4194304 6211092 140720340203840 140720340201800 5075228 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8291 2447 866 493 0 1596 0
Current children cumulated CPU time: 342.28 s
Current children cumulated vsize: 33164 KiB
Current children cumulated memory: 9788 KiB

[startup+402.3 s]
/proc/loadavg: 2.08 1.95 1.70 3/201 23243
/proc/meminfo: memFree=29109176/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33164 memory=9788 CPUtime=402.28 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 20245 0 0 0 40227 1 0 0 20 0 1 0 435605507 33959936 2447 33554432000 4194304 6211092 140720340203840 140720340201800 5074837 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8291 2447 866 493 0 1596 0
Current children cumulated CPU time: 402.28 s
Current children cumulated vsize: 33164 KiB
Current children cumulated memory: 9788 KiB

[startup+462.3 s]
/proc/loadavg: 2.03 1.96 1.72 3/200 23247
/proc/meminfo: memFree=29108916/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33164 memory=9788 CPUtime=462.27 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 21723 0 0 0 46226 1 0 0 20 0 1 0 435605507 33959936 2447 33554432000 4194304 6211092 140720340203840 140720340201800 5076013 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8291 2447 866 493 0 1596 0
Current children cumulated CPU time: 462.27 s
Current children cumulated vsize: 33164 KiB
Current children cumulated memory: 9788 KiB


################
# More data... #
################

/proc/23201/statm: 8291 2453 866 493 0 1596 0
Current children cumulated CPU time: 1722.24 s
Current children cumulated vsize: 33164 KiB
Current children cumulated memory: 9812 KiB

[startup+1782.3 s]
/proc/loadavg: 2.00 2.01 1.94 3/201 23368
/proc/meminfo: memFree=29099868/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33164 memory=9812 CPUtime=1782.24 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 42523 0 0 0 178222 2 0 0 20 0 1 0 435605507 33959936 2453 33554432000 4194304 6211092 140720340203840 140720340201800 5078748 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8291 2453 866 493 0 1596 0
Current children cumulated CPU time: 1782.24 s
Current children cumulated vsize: 33164 KiB
Current children cumulated memory: 9812 KiB

[startup+1842.3 s]
/proc/loadavg: 2.00 2.01 1.94 3/199 23373
/proc/meminfo: memFree=29100160/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33164 memory=9812 CPUtime=1842.24 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 42822 0 0 0 184222 2 0 0 20 0 1 0 435605507 33959936 2453 33554432000 4194304 6211092 140720340203840 140720340201800 5081422 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8291 2453 866 493 0 1596 0
Current children cumulated CPU time: 1842.24 s
Current children cumulated vsize: 33164 KiB
Current children cumulated memory: 9812 KiB

[startup+1902.3 s]
/proc/loadavg: 2.00 2.01 1.95 3/199 23377
/proc/meminfo: memFree=29100384/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33164 memory=9812 CPUtime=1902.23 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 43125 0 0 0 190221 2 0 0 20 0 1 0 435605507 33959936 2453 33554432000 4194304 6211092 140720340203840 140720340201800 5193696 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8291 2453 866 493 0 1596 0
Current children cumulated CPU time: 1902.23 s
Current children cumulated vsize: 33164 KiB
Current children cumulated memory: 9812 KiB

[startup+1962.3 s]
/proc/loadavg: 2.00 2.01 1.95 3/199 23383
/proc/meminfo: memFree=29099848/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33296 memory=9812 CPUtime=1962.23 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 43601 0 0 0 196221 2 0 0 20 0 1 0 435605507 34095104 2453 33554432000 4194304 6211092 140720340203840 140720340201432 5081322 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8324 2453 866 493 0 1629 0
Current children cumulated CPU time: 1962.23 s
Current children cumulated vsize: 33296 KiB
Current children cumulated memory: 9812 KiB

[startup+2022.3 s]
/proc/loadavg: 2.00 2.01 1.95 3/199 23388
/proc/meminfo: memFree=29099884/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33296 memory=9816 CPUtime=2022.23 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 43904 0 0 0 202221 2 0 0 20 0 1 0 435605507 34095104 2454 33554432000 4194304 6211092 140720340203840 140720340201432 5076431 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8324 2454 866 493 0 1629 0
Current children cumulated CPU time: 2022.23 s
Current children cumulated vsize: 33296 KiB
Current children cumulated memory: 9816 KiB

[startup+2082.3 s]
/proc/loadavg: 2.00 2.01 1.95 3/199 23392
/proc/meminfo: memFree=29099860/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33296 memory=9816 CPUtime=2082.23 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 44206 0 0 0 208221 2 0 0 20 0 1 0 435605507 34095104 2454 33554432000 4194304 6211092 140720340203840 140720340201432 5082405 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8324 2454 866 493 0 1629 0
Current children cumulated CPU time: 2082.23 s
Current children cumulated vsize: 33296 KiB
Current children cumulated memory: 9816 KiB

[startup+2142.3 s]
/proc/loadavg: 2.00 2.01 1.95 3/200 23406
/proc/meminfo: memFree=29098252/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33296 memory=9816 CPUtime=2142.23 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 44682 0 0 0 214221 2 0 0 20 0 1 0 435605507 34095104 2454 33554432000 4194304 6211092 140720340203840 140720340201800 5081956 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8324 2454 866 493 0 1629 0
Current children cumulated CPU time: 2142.23 s
Current children cumulated vsize: 33296 KiB
Current children cumulated memory: 9816 KiB

[startup+2202.3 s]
/proc/loadavg: 2.00 2.01 1.95 3/201 23432
/proc/meminfo: memFree=29097676/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33296 memory=9816 CPUtime=2202.23 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 44983 0 0 0 220221 2 0 0 20 0 1 0 435605507 34095104 2454 33554432000 4194304 6211092 140720340203840 140720340201800 5079302 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8324 2454 866 493 0 1629 0
Current children cumulated CPU time: 2202.23 s
Current children cumulated vsize: 33296 KiB
Current children cumulated memory: 9816 KiB

[startup+2262.3 s]
/proc/loadavg: 2.05 2.03 1.96 3/201 23436
/proc/meminfo: memFree=29097800/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33296 memory=9816 CPUtime=2262.22 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 45284 0 0 0 226220 2 0 0 20 0 1 0 435605507 34095104 2454 33554432000 4194304 6211092 140720340203840 140720340201800 5075234 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8324 2454 866 493 0 1629 0
Current children cumulated CPU time: 2262.22 s
Current children cumulated vsize: 33296 KiB
Current children cumulated memory: 9816 KiB

[startup+2322.3 s]
/proc/loadavg: 2.02 2.02 1.96 3/201 23442
/proc/meminfo: memFree=29098004/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33296 memory=9816 CPUtime=2322.22 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 45762 0 0 0 232220 2 0 0 20 0 1 0 435605507 34095104 2454 33554432000 4194304 6211092 140720340203840 140720340201800 5079432 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8324 2454 866 493 0 1629 0
Current children cumulated CPU time: 2322.22 s
Current children cumulated vsize: 33296 KiB
Current children cumulated memory: 9816 KiB

[startup+2382.3 s]
/proc/loadavg: 2.01 2.02 1.96 3/201 23446
/proc/meminfo: memFree=29098108/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33296 memory=9820 CPUtime=2382.22 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 46062 0 0 0 238220 2 0 0 20 0 1 0 435605507 34095104 2455 33554432000 4194304 6211092 140720340203840 140720340201800 5082427 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8324 2455 866 493 0 1629 0
Current children cumulated CPU time: 2382.22 s
Current children cumulated vsize: 33296 KiB
Current children cumulated memory: 9820 KiB

[startup+2442.3 s]
/proc/loadavg: 1.44 1.86 1.91 2/193 23495
/proc/meminfo: memFree=31263220/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33296 memory=9820 CPUtime=2442.22 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 46363 0 0 0 244220 2 0 0 20 0 1 0 435605507 34095104 2455 33554432000 4194304 6211092 140720340203840 140720340201800 5081375 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8324 2455 866 493 0 1629 0
Current children cumulated CPU time: 2442.22 s
Current children cumulated vsize: 33296 KiB
Current children cumulated memory: 9820 KiB

[startup+2502.3 s]
/proc/loadavg: 1.16 1.70 1.85 2/193 23501
/proc/meminfo: memFree=31262928/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33296 memory=9820 CPUtime=2502.22 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 46660 0 0 0 250220 2 0 0 20 0 1 0 435605507 34095104 2455 33554432000 4194304 6211092 140720340203840 140720340201800 5082393 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8324 2455 866 493 0 1629 0
Current children cumulated CPU time: 2502.22 s
Current children cumulated vsize: 33296 KiB
Current children cumulated memory: 9820 KiB



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+2520 s]*
/proc/loadavg: 1.12 1.67 1.84 2/193 23503
/proc/meminfo: memFree=31262556/32770624 swapFree=66997024/67108860
[pid=23201] ppid=23198 vsize=33296 memory=9820 CPUtime=2519.92 cores=1,3,5,7
/proc/23201/stat : 23201 (cosoco) R 23198 23201 23069 0 -1 4202496 46951 0 0 0 251990 2 0 0 20 0 1 0 435605507 34095104 2455 33554432000 4194304 6211092 140720340203840 140720340201800 5082405 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 21778432 140720340210699 140720340210801 140720340210801 140720340213710 0
/proc/23201/statm: 8324 2455 866 493 0 1629 0
Current children cumulated CPU time: 2519.92 s
Current children cumulated vsize: 33296 KiB
Current children cumulated memory: 9820 KiB

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

[startup+2520.01 s]
# the end of solver process 23201 was just reported to runsolver
# cumulated CPU time of all completed processes:  user=2519.91 s, system=0.027999 s

Solver just ended.
??? end of watcher thread
Dumping a history of the last processes samples

Child status: 1
Real time (s): 2520.01
CPU time (s): 2519.94
CPU user time (s): 2519.91
CPU system time (s): 0.027999
CPU usage (%): 99.9972
Max. virtual memory (cumulated for all children) (KiB): 33296
Max. memory (cumulated for all children) (KiB): 9820

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2519.91
system time used= 0.027999
maximum resident set size= 9884
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 47002
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 16
involuntary context switches= 642


# summary of solver processes directly reported to runsolver:
#   pid: 23201
#   total CPU time (s): 2519.94
#   total CPU user time (s): 2519.91
#   total CPU system time (s): 0.027999

??? join timestamper begins
??? join timestamper ends
??? end of timestamper thread
runsolver used 9.11059 second user time and 15.9581 second system time

The end

Launcher Data 

Begin job on node103 at 2019-09-16 02:24:25
IDJOB=4396853
IDBENCH=138965
IDSOLVER=2848
FILE ID=node103/4396853-1568593524
RUNJOBID= node103-1568593336-23101
PBS_JOBID= 21703342
Free space on /tmp= 61716 MiB

SOLVER NAME= cosoco 2.0
BENCH NAME= XCSP17/LatinSquare/LatinSquare-m1-gp/qwh-o30-h374-08.xml
COMMAND LINE= cosoco -lc -model -restarts -stick BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-4396853-1568593524/watcher-4396853-1568593524 -o /tmp/evaluation-result-4396853-1568593524/solver-4396853-1568593524 -C 2520 -W 2520 -M 15500 --bin-var /tmp/runsolver-exch-node103-1568593336-23101 --watchdog 2580  cosoco -lc -model -restarts -stick HOME/instance-4396853-1568593524.xml

TIME LIMIT= 2520 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= a38db4eec3f77542c2f3ff7a616038a8
RANDOM SEED=60285845

node103.alineos.net Linux 3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017

/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor:
performance

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32770624 kB
MemFree:        29219984 kB
MemAvailable:   29835108 kB
Buffers:          128716 kB
Cached:           778704 kB
SwapCached:         9168 kB
Active:          2641792 kB
Inactive:         321780 kB
Active(anon):    2050372 kB
Inactive(anon):    55256 kB
Active(file):     591420 kB
Inactive(file):   266524 kB
Unevictable:      111168 kB
Mlocked:          111168 kB
SwapTotal:      67108860 kB
SwapFree:       66997024 kB
Dirty:              2920 kB
Writeback:             0 kB
AnonPages:       2225176 kB
Mapped:            49272 kB
Shmem:             40856 kB
Slab:             175776 kB
SReclaimable:     146032 kB
SUnreclaim:        29744 kB
KernelStack:        3760 kB
PageTables:         9408 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83494172 kB
Committed_AS:    2579396 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      350232 kB
VmallocChunk:   34341910524 kB
HardwareCorrupted:     0 kB
AnonHugePages:    137216 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      105856 kB
DirectMap2M:    33439744 kB

Free space on /tmp at the end= 61716 MiB
End job on node103 at 2019-09-16 03:06:26