Error updating to 11.0.30

JMB365 · 2011-10-27 02:10:15

Hello,

I ran into an error (Ubuntu 11.04 64bit on QuadCore Intel CPU) compiling mpi version of aster. The error from the mpi version is shown below:

--------------------------------------------------------------------------------
 Prepare cata_ele.pickled
build cata_ele.pickled with make_capy_offi.py                           [  OK  ]
<F>_PICKLED_ERROR  error during making cata_ele.pickled
--------------------------------------------------------------------------------
<F> errors previously raised :
<F>_PICKLED_ERROR  error during making cata_ele.pickled
<E>_UPDATE_FAILED  Exception raised by AutoUpdate:
------------------------------------------------------------
Traceback:
Traceback (most recent call last):
  File "/opt/aster/lib/python2.7/site-packages/asrun/maintenance.py", line 1309, in AutoUpdate
    Update(run, num_update=num_update, *lpkg)
  File "/opt/aster/lib/python2.7/site-packages/asrun/maintenance.py", line 1189, in Update
    _build_aster(run, conf, True, REPREF, None, reptrav_built, lardv=lardv)
  File "/opt/aster/lib/python2.7/site-packages/asrun/maintenance.py", line 962, in _build_aster
    kret = build.MakePickled(REPREF, reptrav, repdest=destdir, **kargs)
  File "/opt/aster/lib/python2.7/site-packages/asrun/build.py", line 1170, in MakePickled
    '<F>_PICKLED_ERROR')
  File "/opt/aster/lib/python2.7/site-packages/asrun/build.py", line 156, in _mess
    self.run.Mess(msg, cod, store)
  File "/opt/aster/lib/python2.7/site-packages/asrun/run.py", line 640, in Mess
    self.Sortie(4)
  File "/opt/aster/lib/python2.7/site-packages/asrun/run.py", line 679, in Sortie
    raise RunAsterError, exit_code
RunAsterError: 4
------------------------------------------------------------
Exit code : 4
--------------------------------------------------------------------------------
<F> errors previously raised :
<F>_PICKLED_ERROR  error during making cata_ele.pickled
 ---------------------------------------------------------------------------------
                                            cpu     system    cpu+sys    elapsed
 ---------------------------------------------------------------------------------
   Download archives                       0.40       0.65       1.05     461.57
   Extraction of archives                  2.44       0.52       2.96       3.42
   Copy of updated files                   0.15       0.59       0.74       5.10
   Deletion of old files                   0.40       0.70       1.10       2.96
   Compilation in debug mode               6.26       2.49       8.75      12.16
   Add object files to library             0.61       0.92       1.53       9.06
   Build executables                       5.75       2.03       7.78       9.56
   Test executables                        0.69       0.22       0.91       5.08
   Compilation in nodebug mode            13.10       1.82      14.92       7.70
   Compilation of commands catalogue       0.39       0.26       0.65       3.72
   Make pickled of elements                0.38       0.07       0.45       2.52
 ---------------------------------------------------------------------------------
   Total                                  30.60      10.28      40.88     522.89
 ---------------------------------------------------------------------------------
   (*) cpu and system times are not correctly counted using mpirun.
as_run 1.10.2
------------------------------------------------------------
--- DIAGNOSTIC JOB : <F>_PICKLED_ERROR
------------------------------------------------------------
EXIT_CODE=4

Same everything (except SingleCore AMD Athlon PC "ubuntu31") updated with no problems! Why / what is happening? How can I diagnose or help diagnose the problem?

uname -a
Linux ubuntu31 2.6.38-11-generic #50-Ubuntu SMP Mon Sep 12 21:17:25 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux  -OK
Linux ubuntu32 2.6.38-11-generic #50-Ubuntu SMP Mon Sep 12 21:17:25 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux  -NOK
Linux ubuntu33 2.6.38-11-generic #50-Ubuntu SMP Mon Sep 12 21:17:25 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux - NOK

Regards, JMB

Last edited by JMB365 (2011-10-27 02:47:24)

JMB365 · 2011-10-27 02:23:16

Hello,

As an update, when I ran:

cd /opt/aster/NEW11.0
/opt/aster/bin/as_run --make > /tmp/output 2> /tmp/error
cd /opt/aster/NEW11.0_mpi
/opt/aster/bin/as_run  --vers=NEW11.0_mpi --make > /tmp/output 2> /tmp/error

The /tmp/output shows that everything compiled fine on the sequential version! But the error remains when compiling the mpi version. So there is an elusive BUG somewhere...

Maybe this topic gives us the clue: http://www.code-aster.org/forum2/viewtopic.php?id=12856 Hopefully a clean solution to the problem can be found...

Regards, JMB

Last edited by JMB365 (2011-10-27 03:26:02)

mathieu.courtois · 2011-10-27 08:06:11

Please attach the output and error files.

JMB365 · 2011-10-27 13:43:00

Hello courtois,

Thanks for looking into it. The files are attached.

Regards, JMB

Attachments:
output.zip, Size: 8.46 KiB, Downloads: 518

JMB365 · 2011-10-27 17:05:48

Hello,

It does not seem normal that ALL the 30 test (list /opt/aster/NEW11.0_mpi/astest/liste_short) would fail with INCORRECT_PARA, ERROR or NO_RESU_FILE; as I am getting. Can somebody comment on whether the changes being made to the test cases in the latest versions (11.0.30) are causing this or is my compiled asteru_mpi at fault? Also a key indicator is that the full suite of tests has no [OK] tests. This makes me suspect my asteru_mpi Thanks!

Regards, JMB

Last edited by JMB365 (2011-10-27 17:06:57)

sovsep · 2011-10-27 20:30:50

Hi,
I had a similar error and when I wanted to checked the files, there were none. All the files that gave ERROR_PARAM weren't in the astout folder. Did you checked those files?

Cheers
S.

mathieu.courtois · 2011-10-27 20:35:55

Hi,

It seems that some files (at least in the astest directory) have not been correctly updated.
Perhaps because the forma* testcases have been heavily changed (removed and re-added later)...
...a bug in as_run ? I can't believe it

MC

JMB365 · 2011-10-27 20:57:54

courtois wrote:

It seems that some files (at least in the astest directory) have not been correctly updated.
Perhaps because the forma* testcases have been heavily changed (removed and re-added later)...
...a bug in as_run ? I can't believe it ;)MC

I hear what you are saying, but I have a situation were every one of the 30 (liste_short) tests fail and every one of the 2400+ tests are failing for one reason or another wherein after 500 errors the testing just stops. Now that is abnormal... So I continue to suspect my .../NEW11.0_mpi/asteru_mpi

Regards, JMB

JMB365 · 2011-10-27 21:01:28

sovsep wrote:

Did you checked those files?.

Thanks for the suggestion, but those files do exist. The mpi version of CodeAster is failing on every one of them, while the sequential version passes OK with most of them.

Regards, JMB

mathieu.courtois · 2011-10-27 21:48:41

JMB365 wrote:

The mpi version of CodeAster is failing on every one of them

No, your mpi version fails and that's what we have to understand !

JMB365 · 2011-10-27 23:59:47

courtois wrote:

No, your mpi version fails and that's what we have to understand !

Before you read too much into what you think I may be implying; please note that two postings ago I stated "So I continue to suspect my .../NEW11.0_mpi/asteru_mpi

-JMB

mathieu.courtois · 2011-10-28 14:09:11

We may have more infos by adding the option '-g' :

as_run -g --make --vers=NEW11_mpi > /tmp/output

I checked it, it will print the Code_Aster output when compiling the file of elements.
Hope this will show us the problem !

MC

JMB365 · 2011-10-28 22:10:18

Hello courtois,

Thanks for looking into the problem. The files are attached below.

Regards, JMB

Attachments:
files_mpi2.zip, Size: 88.7 KiB, Downloads: 558

mathieu.courtois · 2011-10-29 08:51:46

There is yet the problem you reported earlier :

testing executable /opt/aster/NEW11.0_mpi/asteru_mpi...                 [  OK  ]
mpiexec: unable to start all procs; may have invalid machine names
    remaining specified hosts:
        192.168.1.231 (ubuntu31.mshome)

But it exits with exit code = 0 (OK) so I don't know if it starts or not and it prints no output ?!

This thread is probably not solved :-) : http://www.code-aster.org/forum2/viewtopic.php?id=15853

Try running your Pi executable with the same command lines as used by as_run (see the error file).

Do you have several MPI installation on the machines ? Check that /usr/local/bin/mpirun comes with mpdboot ?

(Probably not relevant here but) mpiexec and mpirun point to the same target but nevertheless they may have a different behavior (show and get point together to as_run...).

MC

JMB365 · 2011-10-29 18:44:49

Hello courtois,

Thank for all the suggestions. Let me look into them more carefully. I am sure that the calculation of Pi worked correctly on the 3-noded cluster, as reported by me in Msg#8 of http://www.code-aster.org/forum2/viewtopic.php?id=15853

I may have zipped up the wrong output/error files in my previous message. Currently I am trying to install CAELinux2011 on two other PCs so that I can compare configurations, in an effort to better understand what I need to do for the problems I am experiencing. After I investigate as thoroughly as I can, I will respond back. Have a good weekend!

Regards,
JMB

mathieu.courtois · 2011-10-29 19:41:56

I just suggested to

courtois wrote:

Try running your Pi executable with the same command lines as used by as_run (see the error file).

to see what does not work on your installation : comparing the command line (yours) which works and the one (as_run) it does not !

MC

#1 2011-10-27 02:10:15

Error updating to 11.0.30

#2 2011-10-27 02:23:16

Re: Error updating to 11.0.30

#3 2011-10-27 08:06:11

Re: Error updating to 11.0.30

#4 2011-10-27 13:43:00

Re: Error updating to 11.0.30

#5 2011-10-27 17:05:48

Re: Error updating to 11.0.30

#6 2011-10-27 20:30:50

Re: Error updating to 11.0.30

#7 2011-10-27 20:35:55

Re: Error updating to 11.0.30

#8 2011-10-27 20:57:54

Re: Error updating to 11.0.30

#9 2011-10-27 21:01:28

Re: Error updating to 11.0.30

#10 2011-10-27 21:48:41

Re: Error updating to 11.0.30

#11 2011-10-27 23:59:47

Re: Error updating to 11.0.30

#12 2011-10-28 14:09:11

Re: Error updating to 11.0.30

#13 2011-10-28 22:10:18

Re: Error updating to 11.0.30

#14 2011-10-29 08:51:46

Re: Error updating to 11.0.30

#15 2011-10-29 18:44:49

Re: Error updating to 11.0.30

#16 2011-10-29 19:41:56

Re: Error updating to 11.0.30

Board footer