code_aster v13.6 parallel version

This page is a note of the work to build parallel version.

0.preparation

Nothing.

1.OpenBLAS

To get source code from author’s HP. Then execute following procedure.

$ tar xfvz OpenBLAS-0.2.20.tar.gz
$ cd OpenBLAS-0.2.20
$ make NO_AFFINITY=1 USE_OPENMP=1
$ make PREFIX=/opt/OpenBLAS install
$ echo /opt/OpenBLAS/lib | sudo tee -a /etc/ld.so.conf.d/openblas.conf
$ sudo ldconfig
2.Code_Aster eith OpenBLAS
First, the source code of code_aster is unpacked. Next, “setup.py” is modified according to the reference.
$ cd aster-full-src-13.6.0
$ sed -i "s:PREFER_COMPILER\ =\ 'GNU':PREFER_COMPILER\ =\'GNU_without_MATH'\nMATHLIB=\ '/opt/OpenBLAS/lib/libopenblas.a':g" setup.cfg

And

$ python setup.py install

After the build complete, to make host file for parallel calculation.

$ echo "$HOSTNAME cpu=$(cat /proc/cpuinfo | grep processor | wc -l)" > /opt/aster/etc/codeaster/mpi_hostfile
3.ScaLAPACK
$ tar xfvz scalapack_installer.tgz
$ cd scalapack_installer
$ ./setup.py --lapacklib=/opt/OpenBLAS/lib/libopenblas.a --mpicc=mpicc --mpif90=mpif90 --mpiincdir=/usr/lib/x86_64-linux-gnu/openmpi/include --ldflags_c=-fopenmp --ldflags_fc=-fopenmp --prefix=/opt/scalapack
Over.
4.Parmetis

Get source code and unpack it.

$ tar xvf parmetis-4.0.3.tar.gz
$ cd parmetis-4.0.3
$ make config prefix=/opt/parmetis-4.0.3
$ make

Next, checking.

$ cd Graphs
$ mpirun -np 8 ptest rotor.graph rotor.graph.xyz

When any error is reported, go to~/parmetis-4.0.3 and execute follows:

$ make install

This part is finished.

5.Scotch and Ptscotch

Unpack scotch-6.0.4-aster5.tar.gz which is included the package of code_aster in /opt.

$ cd /opt/scotch-6.0.4/src

Makefile.inc is here.

$ make scotch esmumps ptscotch ptesmumps CCD=mpicc

After building, check it.

$make check
$make ptcheck

This part is finised.

6.MUMPS

Pick mumps-5.1.1-aster2.tar.gz from the package of code_aster out and unpack it in /opt. The name of folder is changed to mumps-5.1.1-mpi.

$ cd /opt/mumps-5.1.1-mpi

Makefile.inc is here.

$ make all

Next, to go to ~/examples and check the MUMPS.

7.Petsc

petsc-3.7.7.tar.gz is downloaded from author’s HP. Then unpack it in /opt. First step is “config”.

$ cd /opt/petsc-3.7.7
$ ./config/configure.py --with-mumps-dir=/opt/mumps-5.1.1-mpi --with-openmpi-dir=/usr/lib/x86_64-linux-gnu/openmpi --with-blas-lapack-lib=[/opt/OpenBLAS/lib/libopenblas.a] --download-hypre=yes --download-ml=yes --with-debugging=0 COPTFLAGS=-O1 CXXOPTFLAGS=-O1 FOPTFLAGS=-O1 --with-x=0 --with-shared-libraries=0 --with-scalapack-dir=/opt/scalapack --PETSC_ARCH=linux-metis-mumps --with-metis-dir=/opt/aster/public/metis-5.1.0 --with-parmetis-dir=/opt/parmetis-4.0.3 --with-scotch-dir=/opt/scotch-6.0.4 --with-ptscotch-dir=/opt/scotch-6.0.4 LIBS=-lgomp --configModules=PETSc.Configure --optionsModule=PETSc.compilerOptions

Next step is “make”

$ make PETSC_DIR=/opt/petsc-3.7.7 PETSC_ARCH=arch-linux2-c-opt all

Then a check of function.

$ make PETSC_DIR=/opt/petsc-3.7.7 PETSC_ARCH=arch-linux2-c-opt test
$ make PETSC_DIR=/opt/petsc-3.7.7 PETSC_ARCH=arch-linux2-c-opt streams

This part is finished.

8.Parallel version Code_Aster

Go to the folder stored the source code of code_aster. And unpack aster-13.6.0.

$ cd ~/Install_Files
$ cd aster-full-src-13.6.0/SRC
$ tar xfvz aster-13.6.0.tgz
$ cd aster-13.6.0

Next a part of ~/waftools/mathematics.py is modified to skip the checking of blacs.

# program testing a blacs call, output is 0 and 1
blacs_fragment = r"""
program test_blacs
    integer iam, nprocs
#    call blacs_pinfo (iam, nprocs)
#    print *,iam
#    print *,nprocs
end program test_blacs
"""

Then copy Ubuntu_gnu.py and Ubuntu_gnu_mpi.py to the folder. Now preparation of build is ready.

$ export ASTER_ROOT=/opt/aster
$ ./waf configure --use-config-dir=$ASTER_ROOT/13.6/share/aster --use-config=Ubuntu_gnu_mpi --prefix=$ASTER_ROOT/PAR13.6MUPT2
$ ./waf install -p

When build is finised. add register the parallel version to /opt/aster/etc/codeaster/aster.

# Code_Aster versions
# versions can be absolute paths or relative to ASTER_ROOT
# examples : NEW11, /usr/lib/codeaster/NEW11

# default version (overridden by --vers option)
default_vers : stable

# available versions
# DO NOT EDIT FOLLOWING LINE !
#?vers : VVV?
vers : stable:/opt/aster/13.6/share/aster
vers : 13.6MUPT:/opt/aster/PAR13.6MUPT2/share/aster
7.Change the setting
Open /opt/aster/etc/codeaster/asrun and edit the line of “mpi_get_procid_cmd”.
mpi_get_procid_cmd : echo $OMPI_COMM_WORLD_RANK
Next, the line regarding to “mpirun” is modified as follows.
mpirun_cmd : mpirun -np %(mpi_nbcpu)s %(program)s

Whole work is finished. Let’s Enjoy!

9 Replies to “code_aster v13.6 parallel version”

  1. Dear Mr. Hitori,

    thank you for publishing your efforts to compile a parallel version of CA13.6.

    I have followed your instructions on this page faithfully, but yet I have stumbled at two issues, and got stuck in the latter:

    1) The first problem is encountered with PARMETIS:

    When I try to check the result of “make” according to your instructions by

    make config prefix=/opt/parmetis-4.0.3
    make
    cd Graphs
    mpirun -np 8 ptest rotor.graph rotor.graph.xyz

    the following error message was received:

    ————————————————————————–
    mpirun was unable to find the specified executable file, and therefore
    did not launch the job. This error was first reported for process
    rank 0; it may have occurred for other processes as well.

    NOTE: A common cause for this error is misspelling a mpirun command
    line parameter option (remember that mpirun interprets the first
    unrecognized command line token as the executable).

    Node: ip-172-31-21-63
    Executable: ptest
    ————————————————————————–
    8 total processes failed to start
    ubuntu@ip-172-31-21-63:/opt/parmetis-4.0.3/Graphs$

    Since I am not an expert in these things I could do nothing but ignore it and proceed with the installation.

    2) The next issue was with your item 8, Parallel version Code_Aster:

    I managed to proceed to the point where the two files Ubuntu_gnu.py and Ubuntu_gnu_mpi.py must be downloaded. After

    export ASTER_ROOT=/opt/aster

    I tried to launch the “waf configure” operation, according to the next command line in your example. But this failed, producing an error message

    ./waf configure –use-config-dir=$ASTER_ROOT/13.6/share/aster –use-config=Ubuntu_gnu_mpi –prefix=$ASTER_ROOT/PAR13.6MUPT2
    checking environment… no found
    Setting top to : /home/ubuntu/code_aster/aster-full-src-13.6.0/SRC/aster-13.6.0
    Setting out to : /home/ubuntu/code_aster/aster-full-src-13.6.0/SRC/aster-13.6.0/build
    Setting prefix to : /opt/aster/PAR13.6MUPT2
    Searching configuration ‘Ubuntu_gnu_mpi’…
    Traceback (most recent call last):
    File “/home/ubuntu/code_aster/aster-full-src-13.6.0/SRC/aster-13.6.0/.waf-1.8.22-5fa41dedfc4a80e01a7ca1d1c9d570c0/waflib/Scripting.py”, line 114, in waf_entry_point
    run_commands()
    File “/home/ubuntu/code_aster/aster-full-src-13.6.0/SRC/aster-13.6.0/.waf-1.8.22-5fa41dedfc4a80e01a7ca1d1c9d570c0/waflib/Scripting.py”, line 175, in run_commands
    ctx=run_command(cmd_name)
    File “/home/ubuntu/code_aster/aster-full-src-13.6.0/SRC/aster-13.6.0/.waf-1.8.22-5fa41dedfc4a80e01a7ca1d1c9d570c0/waflib/Scripting.py”, line 166, in run_command
    ctx.execute()
    File “/home/ubuntu/code_aster/aster-full-src-13.6.0/SRC/aster-13.6.0/.waf-1.8.22-5fa41dedfc4a80e01a7ca1d1c9d570c0/waflib/Configure.py”, line 87, in execute
    super(ConfigurationContext,self).execute()
    File “/home/ubuntu/code_aster/aster-full-src-13.6.0/SRC/aster-13.6.0/.waf-1.8.22-5fa41dedfc4a80e01a7ca1d1c9d570c0/waflib/Context.py”, line 93, in execute
    self.recurse([os.path.dirname(g_module.root_path)])
    File “/home/ubuntu/code_aster/aster-full-src-13.6.0/SRC/aster-13.6.0/.waf-1.8.22-5fa41dedfc4a80e01a7ca1d1c9d570c0/waflib/Context.py”, line 134, in recurse
    user_function(self)
    File “/home/ubuntu/code_aster/aster-full-src-13.6.0/SRC/aster-13.6.0/wscript”, line 157, in configure
    self.load(‘use_config’)
    File “/home/ubuntu/code_aster/aster-full-src-13.6.0/SRC/aster-13.6.0/.waf-1.8.22-5fa41dedfc4a80e01a7ca1d1c9d570c0/waflib/Configure.py”, line 153, in load
    if type(func)is type(Utils.readf):func(self)
    File “/home/ubuntu/code_aster/aster-full-src-13.6.0/SRC/aster-13.6.0/.waf-1.8.22-5fa41dedfc4a80e01a7ca1d1c9d570c0/waflib/extras/use_config.py”, line 85, in configure
    self.load(cfg,tooldir=tooldir,**kwargs)
    File “/home/ubuntu/code_aster/aster-full-src-13.6.0/SRC/aster-13.6.0/.waf-1.8.22-5fa41dedfc4a80e01a7ca1d1c9d570c0/waflib/Configure.py”, line 141, in load
    module=Context.load_tool(tool,tooldir,ctx=self,with_sys_path=with_sys_path)
    File “/home/ubuntu/code_aster/aster-full-src-13.6.0/SRC/aster-13.6.0/.waf-1.8.22-5fa41dedfc4a80e01a7ca1d1c9d570c0/waflib/extras/use_config.py”, line 59, in load_tool
    module=Context.load_tool_default(tool,tooldir,ctx,with_sys_path)
    File “/home/ubuntu/code_aster/aster-full-src-13.6.0/SRC/aster-13.6.0/.waf-1.8.22-5fa41dedfc4a80e01a7ca1d1c9d570c0/waflib/Context.py”, line 370, in load_tool
    __import__(tool)
    File “/home/ubuntu/code_aster/aster-full-src-13.6.0/SRC/aster-13.6.0/Ubuntu_gnu_mpi.py”, line 12, in
    import Ubuntu_gnu
    File “/home/ubuntu/code_aster/aster-full-src-13.6.0/SRC/aster-13.6.0/Ubuntu_gnu.py”, line 22
    ISCOTCH = -I/opt/scotch-6.0.4/include
    ^
    SyntaxError: invalid syntax

    As you can see, a syntax error in the file Ubuntu_gnu.py is reported.

    I am unable to discover which error, if any, is there, and cannot therefore manage this situation by myself.

    Can I kindly request your comments on the two issues above. Thank you very much for your efforts.

    With kind regards
    Hanno Hammer

    1. Hello Dr. Hammer,

      Thank you for your comment.
      regarding your 1st comment, a pass to “ptest” was incorrect.
      In my case, the following command was successful.
      $> mpirun -np 8 /home/kk/Downloads/parmetis-4.0.3/build/Linux-x86_64/programs/ptest rotor.graph rotor.graph.xyz

      And I’m checking your 2nd comment now.
      Thank you,
      Hitori.

      1. Hi Hitori,

        I have just produced another error which I may have overlooked in the first run. It occurs at

        Item 6, MUMPS:

        I unpack the mumps package that is delivered with the code_aster files in /opt and then download the Makefile.inc from your website,

        https://hitoricae.com/wp1/wp-content/uploads/2019/06/bl2-ca136P_MUMPS.txt

        When I issue

        make all

        I get the following error:

        ubuntu@ip-172-31-40-120:/opt/mumps-5.1.1-mpi$ make all
        make ARITH=c mumps_lib
        make[1]: Entering directory ‘/opt/mumps-5.1.1-mpi’
        (cd src ; make c)
        make[2]: Entering directory ‘/opt/mumps-5.1.1-mpi/src’
        make ARITH=c mumps_lib
        make[3]: Entering directory ‘/opt/mumps-5.1.1-mpi/src’
        make[3]: Nothing to be done for ‘mumps_lib’.
        make[3]: Leaving directory ‘/opt/mumps-5.1.1-mpi/src’
        make[2]: Leaving directory ‘/opt/mumps-5.1.1-mpi/src’
        make[1]: Leaving directory ‘/opt/mumps-5.1.1-mpi’
        (cd examples ; make c)
        make[1]: Entering directory ‘/opt/mumps-5.1.1-mpi/examples’
        mpif90 -o csimpletest -O -fopenmp csimpletest.o ../lib/libcmumps.a ../lib/libmumps_common.a -L/opt/aster/public/metis-5.1.0/lib -lmetis -L../PORD/lib/ -lpord -L/opt/scotch-6.0.4/lib -lptesmumps -lptscotch -lptscotcherr -lesmumps -lscotch -lscotcherr -L/opt/parmetis-4.0.3/lib -lparmetis /opt/scalapack/lib/libscalapack.a /opt/OpenBLAS/lib/libopenblas.a -L/usr/local -lmpi -lmpi_mpifh -L/opt/OpenBLAS/lib/ -lopenblas -lpthread
        /usr/bin/ld: cannot find -lparmetis
        collect2: error: ld returned 1 exit status
        Makefile:42: recipe for target ‘csimpletest’ failed
        make[1]: *** [csimpletest] Error 1
        make[1]: Leaving directory ‘/opt/mumps-5.1.1-mpi/examples’
        Makefile:43: recipe for target ‘cexamples’ failed
        make: *** [cexamples] Error 2
        ubuntu@ip-172-31-40-120:/opt/mumps-5.1.1-mpi$

        Maybe this will help you in finding the other bug.

        Kind regards,
        Hanno

        1. Hi,

          there is a follow-up to my last post: The error with MUMPS is produced only when, in the prior installation of PARMETIS, the command

          make install

          was NOT given.

    2. Hello Hanno-san,

      For you 2nd problem, This cause is a incorrect linked file.
      I re-upload Ubuntu_gnu.py. Please recompile with using the file.
      Sorry for your inconvenience.

      Thank you,
      Hitori

  2. I have found yet another error:

    In item 7, PETSC, the command

    make PETSC_DIR=/opt/petsc-3.7.7 PETSC_ARCH=arch-linux2-c-opt all

    does not work. It should be replaced by

    make PETSC_DIR=/opt/petsc-3.7.7 PETSC_ARCH=linux-metis-mumps all

    This latter passes the command

    make PETSC_DIR=/opt/petsc-3.7.7 PETSC_ARCH=linux-metis-mumps test

    but the next command

    make PETSC_DIR=/opt/petsc-3.7.7 PETSC_ARCH=linux-metis-mumps streams

    still terminates with an error message

    makefile:52: recipe for target ‘streams’ failed
    make[1]: [streams] Error 1 (ignored)

    Kind regards
    Hanno

    1. Hello Hanno-san,

      I recompiled petsc according to the article. But I didn’t have any error.
      Can you success config command?

      Thank you.
      Hitori

  3. Hello Hitori,

    thank you for your responses so far!

    I forgot to tell you that, in my previous attempts, I have downloaded all required packages from the internet, since I could not locate their links, or your homepage, for that matter. The reason is that I cannot read the Japanese letters on your site.

    Now it seems that you have created your own compilations of at least OpenBLAS and PETSC. Could you please explain to me how I can download these packages, and from where?

    Thanks a lot for your efforts,
    Hanno

コメントを残す

メールアドレスが公開されることはありません。 * が付いている欄は必須項目です