This page is a note of the work to build parallel version.
０．preparation
Preparation is same as the case of CodeAster 13.6.
Some packages are installed as follows.
$sudo apt-get install gfortran g++ python-dev python-numpy liblapack-dev libblas-dev tcl tk zlib1g-dev bison flex checkinstall openmpi-bin libx11-dev cmake grace gettext libboost-all-dev swig
１．OpenBLAS
To get source code from author’s HP. Then execute following procedure.
$ tar xfvz OpenBLAS-0.2.20.tar.gz
$ cd OpenBLAS-0.2.20
$ make NO_AFFINITY=1 USE_OPENMP=1
$ make PREFIX=/opt/OpenBLAS install
$ echo /opt/OpenBLAS/lib | sudo tee -a /etc/ld.so.conf.d/openblas.conf
$ sudo ldconfig
２．Code_Aster eith OpenBLAS
First, the source code of code_aster is unpacked. Next, “setup.py” is modified according to the reference.
$ cd aster-full-src-13.6.0
$ sed -i "s:PREFER_COMPILER\ =\ 'GNU':PREFER_COMPILER\ =\'GNU_without_MATH'\nMATHLIB=\ '/opt/OpenBLAS/lib/libopenblas.a':g" setup.cfg
And
$ python setup.py install
After the build complete, to make host file for parallel calculation.
$ echo "$HOSTNAME cpu=$(cat /proc/cpuinfo | grep processor | wc -l)" > /opt/aster/etc/codeaster/mpi_hostfile
３．ScaLAPACK
$ tar xfvz scalapack_installer.tgz
$ cd scalapack_installer
$ ./setup.py --lapacklib=/opt/OpenBLAS/lib/libopenblas.a --mpicc=mpicc --mpif90=mpif90 --mpiincdir=/usr/lib/x86_64-linux-gnu/openmpi/include --ldflags_c=-fopenmp --ldflags_fc=-fopenmp --prefix=/opt/scalapack
Over.
４．Parmetis
Get source code and unpack it.
$ tar xvf parmetis-4.0.3.tar.gz
$ cd parmetis-4.0.3
$ make config prefix=/opt/parmetis-4.0.3
$ make
Next, checking.
$ cd Graphs
$ mpirun -np 8 ptest rotor.graph rotor.graph.xyz
When any error is reported, go to~/parmetis-4.0.3 and execute follows:
$ make install
This part is finished.
５．Scotch and Ptscotch
Unpack scotch-6.0.4-aster5.tar.gz which is included the package of code_aster in /opt.
$ cd /opt/scotch-6.0.4/src
Makefile.inc is here.
$ make scotch esmumps ptscotch ptesmumps CCD=mpicc
After building, check it.
$make check
$make ptcheck
This part is finised.
６．MUMPS
Pick mumps-5.1.1-aster2.tar.gz from the package of code_aster out and unpack it in /opt. The name of folder is changed to mumps-5.1.1-mpi.
$ cd /opt/mumps-5.1.1-mpi
Makefile.inc is here.
$ make all
Next, to go to ~/examples and check the MUMPS.
７．Petsc
petsc-3.7.7.tar.gz is downloaded from author’s HP. Then unpack it in /opt. First step is “config”.
$ cd /opt/petsc-3.7.7
$ ./config/configure.py --with-mumps-dir=/opt/mumps-5.1.1-mpi --with-openmpi-dir=/usr/lib/x86_64-linux-gnu/openmpi --with-blas-lapack-lib=[/opt/OpenBLAS/lib/libopenblas.a] --download-hypre=yes --download-ml=yes --with-debugging=0 COPTFLAGS=-O1 CXXOPTFLAGS=-O1 FOPTFLAGS=-O1 --with-x=0 --with-shared-libraries=0 --with-scalapack-dir=/opt/scalapack --PETSC_ARCH=linux-metis-mumps --with-metis-dir=/opt/aster/public/metis-5.1.0 --with-parmetis-dir=/opt/parmetis-4.0.3 --with-scotch-dir=/opt/scotch-6.0.4 --with-ptscotch-dir=/opt/scotch-6.0.4 LIBS=-lgomp --configModules=PETSc.Configure --optionsModule=PETSc.compilerOptions
Next step is “make”
$ make PETSC_DIR=/opt/petsc-3.7.7 PETSC_ARCH=linux-metis-mumps all
Then a check of function.
$ make PETSC_DIR=/opt/petsc-3.7.7 PETSC_ARCH=linux-metis-mumps test
$ make PETSC_DIR=/opt/petsc-3.7.7 PETSC_ARCH=linux-metis-mumps streams
This part is finished.
８．Parallel version Code_Aster
Go to the folder stored the source code of code_aster. And unpack aster-13.6.0.
$ cd ~/Install_Files
$ cd aster-full-src-13.6.0/SRC
$ tar xfvz aster-13.6.0.tgz
$ cd aster-13.6.0
Next a part of ~/waftools/mathematics.py is modified to skip the checking of blacs.
# program testing a blacs call, output is 0 and 1
blacs_fragment = r"""
program test_blacs
integer iam, nprocs
# call blacs_pinfo (iam, nprocs)
# print *,iam
# print *,nprocs
end program test_blacs
"""
Then copy Ubuntu_gnu.py and Ubuntu_gnu_mpi.py to the folder. Now preparation of build is ready.
$ export ASTER_ROOT=/opt/aster
$ ./waf configure --use-config-dir=$ASTER_ROOT/13.6/share/aster --use-config=Ubuntu_gnu_mpi --prefix=$ASTER_ROOT/PAR13.6MUPT2
$ ./waf install -p
When build is finised. add register the parallel version to /opt/aster/etc/codeaster/aster.
# Code_Aster versions
# versions can be absolute paths or relative to ASTER_ROOT
# examples : NEW11, /usr/lib/codeaster/NEW11
# default version (overridden by --vers option)
default_vers : stable
# available versions
# DO NOT EDIT FOLLOWING LINE !
#?vers : VVV?
vers : stable:/opt/aster/13.6/share/aster
vers : 13.6MUPT:/opt/aster/PAR13.6MUPT2/share/aster
７．Change the setting
Open /opt/aster/etc/codeaster/asrun and edit the line of “mpi_get_procid_cmd”.
mpi_get_procid_cmd : echo $OMPI_COMM_WORLD_RANK
Next, the line regarding to “mpirun” is modified as follows.
mpirun_cmd : mpirun -np %(mpi_nbcpu)s %(program)s
Whole work is finished. Let’s Enjoy!
