Trace number 4242012

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
R1: cosoco 1.1? (TO) 2400.04 2400.31

General information on the benchmark

NameOrtholatin/Ortholatin-m1-s1/
Ortholatin-028.xml
MD5SUM0d73506f0318cc77856f25ccca7d4c63
Bench CategoryCSP (decision problem)
Best result obtained on this benchmark
Best value of the objective obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2352
Number of constraints789
Number of domains2
Minimum domain size28
Maximum domain size784
Distribution of domain sizes[{"size":28,"count":1568},{"size":784,"count":784}]
Minimum variable degree2
Maximum variable degree3
Distribution of variable degrees[{"degree":2,"count":2296},{"degree":3,"count":56}]
Minimum constraint arity3
Maximum constraint arity784
Distribution of constraint arities[{"arity":3,"count":784},{"arity":28,"count":2},{"arity":784,"count":3}]
Number of extensional constraints784
Number of intensional constraints0
Distribution of constraint types[{"type":"extension","count":784},{"type":"allDifferent","count":3},{"type":"instantiation","count":2}]
Optimization problemNO
Type of objective

Solver Data 

0.09/0.15	c root propagations     : 27324       
0.09/0.15	c restarts enabled, print stats at every restarts
0.09/0.15	c
0.09/0.15	c
0.09/0.15	c
0.09/0.15	c This is cosoco 1.0 --  
0.09/0.15	c
0.09/0.15	c ========================================[ Problem Statistics ]===========================================
0.09/0.15	c |                                                                                                       
0.09/0.15	c |  Parse time        :         0.05 s 
0.09/0.15	c |
0.09/0.15	c |               Variables: 2352
0.09/0.15	c |            Domain Sizes: 28..784
0.09/0.15	c |
0.09/0.15	c |             Constraints: 953
0.09/0.15	c |                   Arity: 1..784  -- Unary: 56  -- Ternary: 784
0.09/0.15	c | 
0.09/0.15	c |                   Types: 
0.09/0.15	c |                          All Different: 113
0.09/0.15	c |                          Extension: 784  (nb tuples: 784..784) -- (shared: 783)
0.09/0.15	c |                          Unary: 56
0.09/0.15	c |                          
0.09/0.15	c =========================================================================================================
0.09/0.15	c conflicts      decisions      filterCalls    propagations   Root Prop.     Nogoods        min/max/avg depth
0.19/0.22	c 101            389            7233           93235          948            0              233   289   253
0.59/0.66	c 211            1226           30037          366031         948            0              240   484   468
0.69/0.77	c 332            1479           34516          382180         948            0              133   133   131
0.79/0.86	c 465            1744           38155          396875         948            0              133   133   132
0.89/0.95	c 611            2025           41887          412350         948            0              136   136   135
0.99/1.05	c 771            2323           45736          428634         948            0              139   139   138
1.10/1.15	c 947            2640           49686          445705         948            0              142   142   141
1.20/1.25	c 1140           2977           53758          463588         948            0              145   145   144
1.29/1.35	c 1352           3336           57938          482267         948            0              148   148   147
1.39/1.46	c 1585           3719           62248          501764         948            0              151   151   150
1.49/1.58	c 1841           4128           66679          522067         948            0              154   154   153
1.69/1.70	c 2122           4565           71248          543194         948            0              157   157   156
1.79/1.82	c 2431           5066           75662          568720         948            0              181   193   187
1.89/1.96	c 2770           5603           81027          598483         948            0              193   199   194
2.09/2.10	c 3142           6179           86087          629876         948            0              202   205   202
2.20/2.26	c 3551           6798           91961          663560         948            0              208   211   208
2.39/2.41	c 4000           7466           97602          698963         948            0              214   220   215
2.69/2.79	c 4493           8450           112959         844079         948            0              262   432   358
2.99/3.02	c 5035           9304           122155         906660         948            0              307   313   308
4.49/4.58	c 5631           11156          192998         1561459        948            0              332   500   445
4.79/4.83	c 6286           12051          201764         1607391        948            0              238   241   238
4.99/5.09	c 7006           13014          210859         1652671        948            0              241   244   241
5.29/5.36	c 7798           14055          220276         1700320        948            0              247   250   247
5.59/5.65	c 8669           15181          230001         1749647        948            0              253   256   253
5.89/5.95	c 9627           16406          240165         1800123        948            0              262   268   263
6.19/6.25	c 10680          17735          250161         1852464        948            0              271   277   272
6.59/6.70	c 11838          19217          263336         1919897        948            0              283   325   295
7.19/7.21	c 13111          20886          279481         2008978        948            0              322   397   340
7.98/8.07	c 14511          22856          312493         2210769        948            0              383   457   412
8.98/9.06	c 16051          25186          353632         2530150        948            0              440   463   445
9.89/9.97	c 17745          27505          388151         2749802        948            0              399   464   442
10.68/10.73	c 19608          29905          412626         2902703        948            0              363   481   473
12.19/12.26	c 21657          33262          470558         3588478        948            0              493   504   495
12.88/12.93	c 23910          35903          489376         3703414        948            0              352   389   371
14.18/14.30	c 26388          39338          539454         4097340        948            0              371   421   410
15.49/15.53	c 29113          42613          581686         4335068        948            0              398   442   426
19.98/20.03	c 32110          49032          785560         6427682        948            0              412   481   449
21.18/21.27	c 35406          52964          818473         6627214        948            0              341   554   475
30.58/30.68	c 39031          60702          1238769        9109210        948            0              339   401   353
32.08/32.13	c 43018          65972          1287841        9631378        948            0              348   412   376
34.08/34.12	c 47403          71403          1350574        10092207       948            0              307   583   510
37.88/37.99	c 52226          79117          1517555        11534535       948            0              354   432   414
40.08/40.19	c 57531          85227          1570962        11821927       948            0              393   691   562
52.88/52.91	c 63366          100124         2197050        16961302       948            0              357   431   416
56.67/56.76	c 69784          110333         2383980        19366737       948            0              368   422   411
60.77/60.84	c 76843          119458         2565365        20239916       948            0              327   417   400
64.87/64.99	c 84607          129449         2754343        21375150       948            0              379   418   407
67.17/67.27	c 93147          140270         2860138        22601909       948            0              209   422   408
73.16/73.24	c 102541         152221         3093618        24030073       948            0              335   784   565
76.76/76.81	c 112874         164105         3224933        24993083       948            0              280   425   353
80.16/80.26	c 124240         176642         3306139        25432939       948            0              248   883   674
83.96/84.05	c 136742         190824         3407636        26250794       948            0              251   845   608
88.56/88.62	c 150494         206381         3519769        27044385       948            0              260   1043   796
92.37/92.49	c 165621         226638         3637508        29256565       948            0              265   583   429
97.75/97.90	c 182260         244906         3761563        29923462       948            0              276   1205   882
104.57/104.65	c 200562         266525         3951138        31536434       948            0              279   1450   1084
114.26/114.34	c 220694         294826         4376115        35178621       948            0              290   925   911
120.56/120.65	c 242839         327928         4683422        40158977       948            0              303   506   496
127.76/127.81	c 267198         355609         4881270        41704034       948            0              364   975   617
137.57/137.61	c 293992         388179         5222897        45213672       948            0              340   1430   943
145.56/145.65	c 323465         426855         5586999        49343274       948            0              298   487   475
160.35/160.45	c 355885         470251         6183537        54808887       948            0              373   1182   683
173.76/173.86	c 391547         516720         6677581        60883718       948            0              332   1251   942
190.75/190.86	c 430775         567372         7268175        69401177       948            0              1   1451   1191
205.45/205.50	c 473925         620486         7757920        74818280       948            0              323   1302   801
219.95/220.03	c 521390         673789         8290643        77323382       948            0              354   512   499
239.44/239.56	c 573601         742398         8959787        90481867       948            0              299   1425   1033
260.54/260.70	c 631033         817792         9720883        99516860       948            0              372   1069   835
276.24/276.31	c 694208         898185         10267869       107088617      948            0              404   601   588
318.24/318.39	c 763700         994355         11971982       127288511      948            0              468   1396   1164
342.34/342.42	c 840141         1085631        12839612       134873136      948            0              382   650   638
363.43/363.51	c 924226         1190225        13541441       142582686      948            0              328   605   592
389.34/389.49	c 1016719        1305156        14339616       151851668      948            0              326   951   905
420.93/421.04	c 1118461        1464781        15882227       177434246      948            0              419   703   688
450.23/450.35	c 1230377        1595408        16735796       186432997      948            0              326   1370   789
484.63/484.79	c 1353484        1751168        17836339       202047247      948            0              319   1056   791
523.63/523.71	c 1488901        1950069        19580002       228056813      948            0              339   594   582
569.72/569.82	c 1637859        2124025        21249150       238604477      948            0              486   555   542
621.61/621.70	c 1801712        2341065        23439549       262977377      948            0              292   697   610
683.70/683.87	c 1981950        2575952        25886199       289191163      948            0              359   636   572
737.30/737.40	c 2180211        2822230        27582196       310743083      948            0              404   916   697
808.40/808.52	c 2398298        3081473        30207855       330281175      948            0              512   629   571
872.68/872.86	c 2638193        3379201        32570562       355203025      948            0              417   673   607
966.78/966.96	c 2902077        3781334        36310970       415993751      948            0              345   1318   697
1074.86/1075.01	c 3192349        4152169        40390203       453632225      948            0              363   639   616
1220.95/1221.11	c 3511648        4674332        46234583       551151769      948            0              282   1533   1300
1308.24/1308.48	c 3862876        5115287        49411144       587747246      948            0              390   660   646
1432.03/1432.28	c 4249226        5579350        53868935       628433582      948            0              328   806   784
1559.62/1559.86	c 4674211        6135764        59059098       692975337      948            0              297   697   619
1730.90/1731.11	c 5141694        6776185        65633373       774904542      948            0              333   1130   878
1851.69/1851.93	c 5655925        7439041        70550448       830542043      948            0              372   489   478
1987.98/1988.24	c 6221579        8123654        75161719       883547569      948            0              233   665   644
2273.74/2274.08	c 6843798        8972635        86516240       1006761308     948            0              246   879   814

Verifier Data 

One s line expected

Watcher Data 

runsolver Copyright (C) 2010-2013 Olivier ROUSSEL

This is runsolver version 3.3.7 (svn: 2553)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-4242012-1500656577/watcher-4242012-1500656577 -o /tmp/evaluation-result-4242012-1500656577/solver-4242012-1500656577 -C 2400 -W 2500 -M 15500 --bin-var /tmp/runsolver-exch-node103-1500648178-9851 --watchdog 2560 HOME/cosoco -lc -model -restarts HOME/instance-4242012-1500656577.xml 

pid=13423
running on 8 cores: 1,3,5,7,9,11,13,15

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 2400 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 2430 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 2500 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 8192 KiB


[startup+0 s]
/proc/loadavg: 2.38 2.15 2.09 3/244 13426
/proc/meminfo: memFree=32025340/32770624 swapFree=5992/191416
[pid=13426] ppid=13423 vsize=27648 CPUtime=0 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 1108 0 0 0 0 0 0 0 20 0 1 0 927729 28311552 939 33554432000 4194304 5970956 140728564224960 140728564221560 139979031473125 0 0 4096 0 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 6912 939 703 434 0 276 0

[startup+0.100161 s]
/proc/loadavg: 2.38 2.15 2.09 3/244 13426
/proc/meminfo: memFree=32025340/32770624 swapFree=5992/191416
[pid=13426] ppid=13423 vsize=53564 CPUtime=0.09 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 7662 0 0 0 8 1 0 0 20 0 1 0 927729 54849536 7472 33554432000 4194304 5970956 140728564224960 140728564222216 4934432 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 13391 7472 743 434 0 6755 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 53564

[startup+0.200403 s]
/proc/loadavg: 2.38 2.15 2.09 3/244 13426
/proc/meminfo: memFree=32025340/32770624 swapFree=5992/191416
[pid=13426] ppid=13423 vsize=54752 CPUtime=0.19 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 7966 0 0 0 18 1 0 0 20 0 1 0 927729 56066048 7802 33554432000 4194304 5970956 140728564224960 140728564221880 4969951 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 13688 7802 774 434 0 7052 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 54752

[startup+0.300335 s]
/proc/loadavg: 2.38 2.15 2.09 3/244 13426
/proc/meminfo: memFree=32025340/32770624 swapFree=5992/191416
[pid=13426] ppid=13423 vsize=54884 CPUtime=0.29 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 7989 0 0 0 28 1 0 0 20 0 1 0 927729 56201216 7802 33554432000 4194304 5970956 140728564224960 140728564222296 4977781 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 13721 7802 774 434 0 7085 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 54884

[startup+0.700286 s]
/proc/loadavg: 2.38 2.15 2.09 3/244 13426
/proc/meminfo: memFree=32025340/32770624 swapFree=5992/191416
[pid=13426] ppid=13423 vsize=58184 CPUtime=0.69 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 8820 0 0 0 67 2 0 0 20 0 1 0 927729 59580416 8594 33554432000 4194304 5970956 140728564224960 140728564222296 4934366 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 14546 8594 777 434 0 7910 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 58184

[startup+1.50029 s]
/proc/loadavg: 2.38 2.15 2.09 3/245 13427
/proc/meminfo: memFree=31993704/32770624 swapFree=5992/191416
[pid=13426] ppid=13423 vsize=58184 CPUtime=1.49 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 8820 0 0 0 147 2 0 0 20 0 1 0 927729 59580416 8594 33554432000 4194304 5970956 140728564224960 140728564222296 4934366 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 14546 8594 777 434 0 7910 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 58184

[startup+3.10023 s]
/proc/loadavg: 2.38 2.15 2.09 3/245 13427
/proc/meminfo: memFree=31993264/32770624 swapFree=5992/191416
[pid=13426] ppid=13423 vsize=58912 CPUtime=3.09 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 15142 0 0 0 307 2 0 0 20 0 1 0 927729 60325888 8820 33554432000 4194304 5970956 140728564224960 140728564222296 4974652 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 14728 8820 777 434 0 8092 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 58912

[startup+6.30021 s]
/proc/loadavg: 2.35 2.15 2.09 3/245 13427
/proc/meminfo: memFree=31990860/32770624 swapFree=5992/191448
[pid=13426] ppid=13423 vsize=61024 CPUtime=6.29 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 22522 0 0 0 627 2 0 0 20 0 1 0 927729 62488576 9362 33554432000 4194304 5970956 140728564224960 140728564222296 4974147 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 15256 9362 777 434 0 8620 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 61024

[startup+12.7002 s]
/proc/loadavg: 2.33 2.15 2.09 3/245 13427
/proc/meminfo: memFree=31989996/32770624 swapFree=5992/191448
[pid=13426] ppid=13423 vsize=61740 CPUtime=12.68 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 29657 0 0 0 1266 2 0 0 20 0 1 0 927729 63221760 9493 33554432000 4194304 5970956 140728564224960 140728564222296 4970936 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 15435 9493 777 434 0 8799 0
Current children cumulated CPU time (s) 12.68
Current children cumulated vsize (KiB) 61740

[startup+25.5004 s]
/proc/loadavg: 2.25 2.14 2.09 3/245 13427
/proc/meminfo: memFree=31989260/32770624 swapFree=5992/191472
[pid=13426] ppid=13423 vsize=62136 CPUtime=25.48 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 42341 0 0 0 2546 2 0 0 20 0 1 0 927729 63627264 9639 33554432000 4194304 5970956 140728564224960 140728564222296 4934292 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 15534 9639 777 434 0 8898 0
Current children cumulated CPU time (s) 25.48
Current children cumulated vsize (KiB) 62136

[startup+51.1004 s]
/proc/loadavg: 2.17 2.13 2.08 3/245 13427
/proc/meminfo: memFree=31988880/32770624 swapFree=5992/191472
[pid=13426] ppid=13423 vsize=62988 CPUtime=51.07 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 51292 0 0 0 5104 3 0 0 20 0 1 0 927729 64499712 9832 33554432000 4194304 5970956 140728564224960 140728564222296 5098928 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 15747 9832 777 434 0 9111 0
Current children cumulated CPU time (s) 51.07
Current children cumulated vsize (KiB) 62988

[startup+102.3 s]
/proc/loadavg: 2.07 2.11 2.08 3/245 13428
/proc/meminfo: memFree=31981128/32770624 swapFree=5992/191556
[pid=13426] ppid=13423 vsize=70140 CPUtime=102.26 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 53902 0 0 0 10223 3 0 0 20 0 1 0 927729 71823360 11619 33554432000 4194304 5970956 140728564224960 140728564221656 4969782 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 17535 11619 777 434 0 10899 0
Current children cumulated CPU time (s) 102.26
Current children cumulated vsize (KiB) 70140

[startup+162.3 s]
/proc/loadavg: 2.03 2.09 2.07 3/245 13429
/proc/meminfo: memFree=31978304/32770624 swapFree=5992/191624
[pid=13426] ppid=13423 vsize=72416 CPUtime=162.25 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 57445 0 0 0 16221 4 0 0 20 0 1 0 927729 74153984 12188 33554432000 4194304 5970956 140728564224960 140728564222296 4934354 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 18104 12188 777 434 0 11468 0
Current children cumulated CPU time (s) 162.25
Current children cumulated vsize (KiB) 72416

[startup+222.308 s]
/proc/loadavg: 2.01 2.07 2.07 3/243 13430
/proc/meminfo: memFree=31978044/32770624 swapFree=5992/191676
[pid=13426] ppid=13423 vsize=72952 CPUtime=222.25 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 58155 0 0 0 22221 4 0 0 20 0 1 0 927729 74702848 12341 33554432000 4194304 5970956 140728564224960 140728564222296 4974750 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 18238 12341 777 434 0 11602 0
Current children cumulated CPU time (s) 222.25
Current children cumulated vsize (KiB) 72952

[startup+282.3 s]
/proc/loadavg: 2.00 2.06 2.06 3/243 13431
/proc/meminfo: memFree=31978224/32770624 swapFree=5992/191740
[pid=13426] ppid=13423 vsize=73084 CPUtime=282.24 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 58924 0 0 0 28220 4 0 0 20 0 1 0 927729 74838016 12378 33554432000 4194304 5970956 140728564224960 140728564222296 4976788 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 18271 12378 777 434 0 11635 0
Current children cumulated CPU time (s) 282.24
Current children cumulated vsize (KiB) 73084

[startup+342.3 s]
/proc/loadavg: 2.00 2.05 2.05 3/243 13432
/proc/meminfo: memFree=31978188/32770624 swapFree=5992/191804
[pid=13426] ppid=13423 vsize=73084 CPUtime=342.24 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 59630 0 0 0 34220 4 0 0 20 0 1 0 927729 74838016 12386 33554432000 4194304 5970956 140728564224960 140728564222296 4974415 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 18271 12386 777 434 0 11635 0
Current children cumulated CPU time (s) 342.24
Current children cumulated vsize (KiB) 73084

[startup+402.3 s]
/proc/loadavg: 2.00 2.04 2.05 3/243 13433
/proc/meminfo: memFree=31978364/32770624 swapFree=5992/191860
[pid=13426] ppid=13423 vsize=73084 CPUtime=402.23 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 60305 0 0 0 40219 4 0 0 20 0 1 0 927729 74838016 12386 33554432000 4194304 5970956 140728564224960 140728564222296 4970936 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 18271 12386 777 434 0 11635 0
Current children cumulated CPU time (s) 402.23
Current children cumulated vsize (KiB) 73084

[startup+462.3 s]
/proc/loadavg: 2.00 2.04 2.05 3/244 13442
/proc/meminfo: memFree=31976936/32770624 swapFree=5992/191900
[pid=13426] ppid=13423 vsize=73352 CPUtime=462.23 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 61108 0 0 0 46219 4 0 0 20 0 1 0 927729 75112448 12445 33554432000 4194304 5970956 140728564224960 140728564222296 4970046 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 18338 12445 777 434 0 11702 0
Current children cumulated CPU time (s) 462.23
Current children cumulated vsize (KiB) 73352

[startup+522.3 s]
/proc/loadavg: 2.00 2.03 2.05 3/244 13443
/proc/meminfo: memFree=31976872/32770624 swapFree=5992/191956
[pid=13426] ppid=13423 vsize=73352 CPUtime=522.22 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 61823 0 0 0 52218 4 0 0 20 0 1 0 927729 75112448 12446 33554432000 4194304 5970956 140728564224960 140728564222296 4977082 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 18338 12446 777 434 0 11702 0
Current children cumulated CPU time (s) 522.22
Current children cumulated vsize (KiB) 73352

[startup+582.3 s]
/proc/loadavg: 2.00 2.02 2.05 3/244 13444
/proc/meminfo: memFree=31977412/32770624 swapFree=5992/191988
[pid=13426] ppid=13423 vsize=73352 CPUtime=582.22 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 62477 0 0 0 58218 4 0 0 20 0 1 0 927729 75112448 12446 33554432000 4194304 5970956 140728564224960 140728564222296 4974240 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 18338 12446 777 434 0 11702 0
Current children cumulated CPU time (s) 582.22
Current children cumulated vsize (KiB) 73352


################
# More data... #
################


[startup+1542.31 s]
/proc/loadavg: 2.01 2.02 2.05 3/243 13470
/proc/meminfo: memFree=31975160/32770624 swapFree=5992/192664
[pid=13426] ppid=13423 vsize=73892 CPUtime=1542.13 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 71002 0 0 0 154209 4 0 0 20 0 1 0 927729 75665408 12580 33554432000 4194304 5970956 140728564224960 140728564222296 4974808 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 18473 12580 777 434 0 11837 0
Current children cumulated CPU time (s) 1542.13
Current children cumulated vsize (KiB) 73892

[startup+1602.3 s]
/proc/loadavg: 2.00 2.01 2.05 3/243 13471
/proc/meminfo: memFree=31973916/32770624 swapFree=5992/192716
[pid=13426] ppid=13423 vsize=73892 CPUtime=1602.11 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 71384 0 0 0 160207 4 0 0 20 0 1 0 927729 75665408 12580 33554432000 4194304 5970956 140728564224960 140728564222296 5322288 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 18473 12580 777 434 0 11837 0
Current children cumulated CPU time (s) 1602.11
Current children cumulated vsize (KiB) 73892

[startup+1662.3 s]
/proc/loadavg: 2.00 2.01 2.05 3/244 13480
/proc/meminfo: memFree=31973928/32770624 swapFree=5992/192760
[pid=13426] ppid=13423 vsize=73892 CPUtime=1662.1 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 71738 0 0 0 166206 4 0 0 20 0 1 0 927729 75665408 12580 33554432000 4194304 5970956 140728564224960 140728564222296 4974124 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 18473 12580 777 434 0 11837 0
Current children cumulated CPU time (s) 1662.1
Current children cumulated vsize (KiB) 73892

[startup+1722.3 s]
/proc/loadavg: 2.08 2.03 2.05 3/244 13481
/proc/meminfo: memFree=31974396/32770624 swapFree=5992/192796
[pid=13426] ppid=13423 vsize=73892 CPUtime=1722.1 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 72064 0 0 0 172206 4 0 0 20 0 1 0 927729 75665408 12580 33554432000 4194304 5970956 140728564224960 140728564222296 4977138 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 18473 12580 777 434 0 11837 0
Current children cumulated CPU time (s) 1722.1
Current children cumulated vsize (KiB) 73892

[startup+1782.3 s]
/proc/loadavg: 2.03 2.02 2.05 3/244 13548
/proc/meminfo: memFree=31986556/32770624 swapFree=5992/192880
[pid=13426] ppid=13423 vsize=73892 CPUtime=1782.09 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 72460 0 0 0 178205 4 0 0 20 0 1 0 927729 75665408 12580 33554432000 4194304 5970956 140728564224960 140728564222296 4976725 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 18473 12580 777 434 0 11837 0
Current children cumulated CPU time (s) 1782.09
Current children cumulated vsize (KiB) 73892

[startup+1842.3 s]
/proc/loadavg: 2.01 2.02 2.05 3/244 13549
/proc/meminfo: memFree=31986684/32770624 swapFree=5992/192944
[pid=13426] ppid=13423 vsize=73892 CPUtime=1842.09 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 72771 0 0 0 184205 4 0 0 20 0 1 0 927729 75665408 12580 33554432000 4194304 5970956 140728564224960 140728564222296 4971273 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 18473 12580 777 434 0 11837 0
Current children cumulated CPU time (s) 1842.09
Current children cumulated vsize (KiB) 73892

[startup+1902.3 s]
/proc/loadavg: 2.00 2.01 2.05 3/244 13550
/proc/meminfo: memFree=31985472/32770624 swapFree=5992/193004
[pid=13426] ppid=13423 vsize=73892 CPUtime=1902.09 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 73158 0 0 0 190205 4 0 0 20 0 1 0 927729 75665408 12580 33554432000 4194304 5970956 140728564224960 140728564222296 4974720 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 18473 12580 777 434 0 11837 0
Current children cumulated CPU time (s) 1902.09
Current children cumulated vsize (KiB) 73892

[startup+1962.3 s]
/proc/loadavg: 2.00 2.01 2.05 3/243 13551
/proc/meminfo: memFree=31985444/32770624 swapFree=5992/193036
[pid=13426] ppid=13423 vsize=73892 CPUtime=1962.08 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 73466 0 0 0 196204 4 0 0 20 0 1 0 927729 75665408 12580 33554432000 4194304 5970956 140728564224960 140728564222296 4971765 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 18473 12580 777 434 0 11837 0
Current children cumulated CPU time (s) 1962.08
Current children cumulated vsize (KiB) 73892

[startup+2022.3 s]
/proc/loadavg: 2.00 2.01 2.05 3/243 13552
/proc/meminfo: memFree=31985768/32770624 swapFree=5992/193076
[pid=13426] ppid=13423 vsize=73892 CPUtime=2022.08 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 73855 0 0 0 202204 4 0 0 20 0 1 0 927729 75665408 12580 33554432000 4194304 5970956 140728564224960 140728564222296 4934327 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 18473 12580 777 434 0 11837 0
Current children cumulated CPU time (s) 2022.08
Current children cumulated vsize (KiB) 73892

[startup+2082.3 s]
/proc/loadavg: 2.00 2.01 2.05 3/243 13553
/proc/meminfo: memFree=31985092/32770624 swapFree=5992/193112
[pid=13426] ppid=13423 vsize=73892 CPUtime=2082.07 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 74168 0 0 0 208203 4 0 0 20 0 1 0 927729 75665408 12580 33554432000 4194304 5970956 140728564224960 140728564222296 4976581 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 18473 12580 777 434 0 11837 0
Current children cumulated CPU time (s) 2082.07
Current children cumulated vsize (KiB) 73892

[startup+2142.3 s]
/proc/loadavg: 2.03 2.03 2.05 3/243 13554
/proc/meminfo: memFree=31985524/32770624 swapFree=5992/193168
[pid=13426] ppid=13423 vsize=73892 CPUtime=2142.05 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 74480 0 0 0 214201 4 0 0 20 0 1 0 927729 75665408 12580 33554432000 4194304 5970956 140728564224960 140728564222296 4971926 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 18473 12580 777 434 0 11837 0
Current children cumulated CPU time (s) 2142.05
Current children cumulated vsize (KiB) 73892

[startup+2202.31 s]
/proc/loadavg: 2.01 2.02 2.05 3/243 13555
/proc/meminfo: memFree=31986116/32770624 swapFree=5992/193204
[pid=13426] ppid=13423 vsize=73892 CPUtime=2202.05 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 74795 0 0 0 220201 4 0 0 20 0 1 0 927729 75665408 12580 33554432000 4194304 5970956 140728564224960 140728564222296 4977807 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 18473 12580 777 434 0 11837 0
Current children cumulated CPU time (s) 2202.05
Current children cumulated vsize (KiB) 73892

[startup+2262.3 s]
/proc/loadavg: 2.00 2.01 2.05 3/243 13563
/proc/meminfo: memFree=31984944/32770624 swapFree=5992/193244
[pid=13426] ppid=13423 vsize=73892 CPUtime=2262.04 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 75115 0 0 0 226200 4 0 0 20 0 1 0 927729 75665408 12580 33554432000 4194304 5970956 140728564224960 140728564222296 4934272 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 18473 12580 777 434 0 11837 0
Current children cumulated CPU time (s) 2262.04
Current children cumulated vsize (KiB) 73892

[startup+2322.3 s]
/proc/loadavg: 2.00 2.01 2.05 3/243 13564
/proc/meminfo: memFree=31985036/32770624 swapFree=5992/193284
[pid=13426] ppid=13423 vsize=73892 CPUtime=2322.03 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 75508 0 0 0 232199 4 0 0 20 0 1 0 927729 75665408 12580 33554432000 4194304 5970956 140728564224960 140728564222296 4976725 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 18473 12580 777 434 0 11837 0
Current children cumulated CPU time (s) 2322.03
Current children cumulated vsize (KiB) 73892

[startup+2382.3 s]
/proc/loadavg: 2.00 2.01 2.05 3/243 13565
/proc/meminfo: memFree=31985780/32770624 swapFree=5992/193320
[pid=13426] ppid=13423 vsize=73892 CPUtime=2382.02 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 75819 0 0 0 238198 4 0 0 20 0 1 0 927729 75665408 12580 33554432000 4194304 5970956 140728564224960 140728564222296 4978160 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 18473 12580 777 434 0 11837 0
Current children cumulated CPU time (s) 2382.02
Current children cumulated vsize (KiB) 73892



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+2400.31 s]
/proc/loadavg: 2.00 2.01 2.05 3/243 13565
/proc/meminfo: memFree=31985468/32770624 swapFree=5992/193324
[pid=13426] ppid=13423 vsize=73892 CPUtime=2400.02 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 76123 0 0 0 239998 4 0 0 20 0 1 0 927729 75665408 12580 33554432000 4194304 5970956 140728564224960 140728564222296 4974816 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 18473 12580 777 434 0 11837 0
Current children cumulated CPU time (s) 2400.02
Current children cumulated vsize (KiB) 73892

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended.
??? exit watcher main loop
??? sigkill at end of watcher main loop
??? kill 13426 sig 9
??? end of watcher thread
Dumping a history of the last processes samples

[startup+2400.31 s]
/proc/loadavg: 2.00 2.01 2.05 3/243 13565
/proc/meminfo: memFree=31985468/32770624 swapFree=5992/193324
[pid=13426] ppid=13423 vsize=73892 CPUtime=2400.02 cores=1,3,5,7,9,11,13,15
/proc/13426/stat : 13426 (cosoco) R 13423 13426 9819 0 -1 4202496 76123 0 0 0 239998 4 0 0 20 0 1 0 927729 75665408 12580 33554432000 4194304 5970956 140728564224960 140728564222296 4974816 0 0 4096 8405026 0 0 0 17 11 0 0 0 0 0 8070896 8075904 8871936 140728564233409 140728564233539 140728564233539 140728564236238 0
/proc/13426/statm: 18473 12580 777 434 0 11837 0
Current children cumulated CPU time (s) 2400.02
Current children cumulated vsize (KiB) 73892

Child status: 1
Real time (s): 2400.31
CPU time (s): 2400.04
CPU user time (s): 2399.99
CPU system time (s): 0.047995
CPU usage (%): 99.9886
Max. virtual memory (cumulated for all children) (KiB): 73892

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2399.99
system time used= 0.047995
maximum resident set size= 50364
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 76150
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 23
involuntary context switches= 120717

??? join timestamper begins
??? join timestamper ends
??? end of timestamper thread
runsolver used 7.77443 second user time and 11.6308 second system time

The end

Launcher Data 

Begin job on node103 at 2017-07-21 19:02:57
IDJOB=4242012
IDBENCH=133176
IDSOLVER=2620
FILE ID=node103/4242012-1500656577
RUNJOBID= node103-1500648178-9851
PBS_JOBID= 20623311
Free space on /tmp= 62012 MiB

SOLVER NAME= cosoco 1.1
BENCH NAME= XCSP17/Ortholatin/Ortholatin-m1-s1/Ortholatin-028.xml
COMMAND LINE= DIR/cosoco -lc -model -restarts BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-4242012-1500656577/watcher-4242012-1500656577 -o /tmp/evaluation-result-4242012-1500656577/solver-4242012-1500656577 -C 2400 -W 2500 -M 15500 --bin-var /tmp/runsolver-exch-node103-1500648178-9851 --watchdog 2560  HOME/cosoco -lc -model -restarts HOME/instance-4242012-1500656577.xml

TIME LIMIT= 2400 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 8

MD5SUM BENCH= 0d73506f0318cc77856f25ccca7d4c63
RANDOM SEED=2145866130

node103.alineos.net Linux 3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 1600.000
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.12
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 1600.000
cache size	: 8192 KB
physical id	: 1
siblings	: 8
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 1600.000
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.12
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 1600.000
cache size	: 8192 KB
physical id	: 1
siblings	: 8
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 1600.000
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.12
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 1600.000
cache size	: 8192 KB
physical id	: 1
siblings	: 8
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 1600.000
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.12
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 1600.000
cache size	: 8192 KB
physical id	: 1
siblings	: 8
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 1600.000
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 4
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.12
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2533.000
cache size	: 8192 KB
physical id	: 1
siblings	: 8
core id		: 0
cpu cores	: 4
apicid		: 17
initial apicid	: 17
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 1600.000
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 4
apicid		: 3
initial apicid	: 3
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.12
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 1867.000
cache size	: 8192 KB
physical id	: 1
siblings	: 8
core id		: 1
cpu cores	: 4
apicid		: 19
initial apicid	: 19
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 12
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 4
apicid		: 5
initial apicid	: 5
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.12
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 13
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2000.000
cache size	: 8192 KB
physical id	: 1
siblings	: 8
core id		: 2
cpu cores	: 4
apicid		: 21
initial apicid	: 21
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 14
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 1733.000
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 4
apicid		: 7
initial apicid	: 7
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.12
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 1600.000
cache size	: 8192 KB
physical id	: 1
siblings	: 8
core id		: 3
cpu cores	: 4
apicid		: 23
initial apicid	: 23
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32770624 kB
MemFree:        32024784 kB
MemAvailable:   32016924 kB
Buffers:           34332 kB
Cached:           267800 kB
SwapCached:            0 kB
Active:           168908 kB
Inactive:         200064 kB
Active(anon):      69732 kB
Inactive(anon):     8648 kB
Active(file):      99176 kB
Inactive(file):   191416 kB
Unevictable:        5992 kB
Mlocked:            5996 kB
SwapTotal:      67108860 kB
SwapFree:       67108860 kB
Dirty:              2448 kB
Writeback:            36 kB
AnonPages:         72756 kB
Mapped:            31184 kB
Shmem:              8852 kB
Slab:              67400 kB
SReclaimable:      34768 kB
SUnreclaim:        32632 kB
KernelStack:        4320 kB
PageTables:         5248 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83494172 kB
Committed_AS:     339648 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      350232 kB
VmallocChunk:   34341910524 kB
HardwareCorrupted:     0 kB
AnonHugePages:     16384 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      112000 kB
DirectMap2M:    33433600 kB

Free space on /tmp at the end= 62012 MiB
End job on node103 at 2017-07-21 19:42:58