Code Aster 15.2 MPI outside Singularity

This is a note of the work.
The linux box is xUbuntu 18.04LTS on VMware.
Python is python 2.7.17 and Python3 is python 3.6.9.

Preparataion

$sudo apt-get install gfortran g++ python-dev python-numpy liblapack-dev libblas-dev tcl tk zlib1g-dev bison flex checkinstall openmpi-bin libx11-dev cmake grace gettext libboost-all-dev swig

And install superlu package.

$sudo apt-get install libsuperlu-dev libopenmpi-dev python3-mpi4py cython

And I build Code aster 15.2.0 sequential version with –prefi x=/opt/aster/15.2.

OpenBLAS

To get source code from author’s HP. Then execute following procedure.

$ tar xfvz OpenBLAS-0.2.20.tar.gz$ cd OpenBLAS-0.2.20$ make NO_AFFINITY=1 USE_OPENMP=1$ make PREFIX=/opt/OpenBLAS install$ echo /opt/OpenBLAS/lib | sudo tee -a /etc/ld.so.conf.d/openblas.conf$ sudo ldconfig

Scalapack

$./setup.py --lapacklib=/opt/OpenBLAS/lib/libopenblas.a --mpicc=mpicc --mpif90=mpif90 --mpiincdir=/usr/lib/x86_64-linux-gnu/openmpi/include --ldflags_c=-fopenmp --ldflags_fc=-fopenmp --prefix=/opt/scalapack-n --ccflag=-fPIC --fcflag=-fPIC

Parmetis

Get source code and unpack it.

$ tar xvf parmetis-4.0.3.tar.gz
$ cd parmetis-4.0.3

Integer size is 8 byte in this work. #defi ne IDXTYPEWIDTH 64″ and “#defi ne REALTYPEWIDTH 64”is written in ~/parmetis-4.0.3/metis/include/metis.h.

Then

$ make config shared=1 prefix=/opt/parmetis-4.0.3
$ make

checking,

$ export LD_LIBRARY_PATH=/opt/parmetis-4.0.3/lib/:$LD_LIBRARY_PATH$ cd Graphs$ mpirun -np 8 ptest rotor.graph rotor.graph.xyz

When any error is not reported, go to~/parmetis-4.0.3 and execute follows:

$ make install

Scotch and Ptscotch

Unpack scotch-6.0.4-aster5.tar.gz which is included the package of code_aster in /opt.

$ cd /opt/scotch-6.0.4/src

Makefi le.inc is xxx

$ make scotch esmumps ptscotch ptesmumps CCD=mpicc

After building, check it.

$ make check
$ make ptcheck

MUMPS

Pick mumps-5.2.1-aster2.tar.gz from the package of code_aster out and unpack it. The name offolder is changed to mumps-5.2.1_mob.

$ cd /opt/mumps-5.2.1_mob

Makefi le.inc ダウンロード

$ make all

Next, to go to ~/examples and check the MUMPS.

PETSC

petsc 3.12.3.tar.gz is downloaded from author’s HP. Then unpack it in /opt. First , a few line (line43 to 48) is commented out in /opt/petsc-3.12.3/confi g/BuildSystem/confi g/packages/metis.py.

def configureLibrary(self):
config.package.Package.configureLibrary(self)
oldFlags = self.compilers.CPPFLAGS
self.compilers.CPPFLAGS += ' '+self.headers.toString(self.include)
# if not self.checkCompile('#include "metis.h"', '#if (IDXTYPEWIDTH != '+ str(self.getDefaultIndexSize())+')\n#error incompatible IDXTYPEWIDTH\n#endif'):
# if self.defaultIndexSize == 64:
# msg= '--with-64-bit-indices option requires a metis build with IDXTYPEWIDTH=64.\n'
# else:
# msg= 'IDXTYPEWIDTH=64 metis build appears to be specified for a default 32-bit-indices build of PETSc.\n'
# raise RuntimeError('Metis specified is incompatible!\n'+msg+'Suggest using --download-metis for a compatible metis')

And when openmpi is used, path to openmpi library is added to LD_LIBRARY_PATH. Next is asample.

$ export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/openmpi/lib/:$LD_LIBRARY_PATH

Then, “confi gure” is done.

$ ./configure --with-debugging=0 COPTFLAGS='-O -fPIC' CXXOPTFLAGS='-O -fPIC' FOPTFLAGS='-O -fPIC' --with-shared-libraries=yes --with-scalapack-dir=/opt/scalapack-n --PETSC_ARCH=linux-metis-mumps --with-metis-dir=/opt/aster/public/metis-5.1.0 --with-ptscotch-dir=/opt/scotch-6.0.4 --LIBS="-lgomp" --with-mumps-dir=/opt/mumps-5.2.1_mob -with-x=0 --with-blas-lapack-lib=[/opt/OpenBLAS/lib/libopenblas.a] --force --download-petsc4py=yes

and “make”

$ make PETSC_DIR=/opt/petsc-3.12.3 PETSC_ARCH=linux-metis-mumps all

Code Aster 15.2 MPI

Go to source code folder. And unpack aster-15.2.0.

>cd ..Downloads/aster-full-src-15.2.0/SRC
>tar -xvf aster-15.2.0
>cd aster-15.2.0

Next a part of ~/waftools/mathematics.py is modifi ed to skip the checking of blacs.

program testing a blacs call, output is 0 and 1
blacs_fragment = r"""
program test_blacs
 integer iam, nprocs
# call blacs_pinfo (iam, nprocs)
# print *,iam
# print *,nprocs
end program test_blacs
"""

Modify profi le.sh in /opt/aster/15.2/share/aster.

profi le.sh ダウンロード

>source /opt/aster/15.2/share/aster/profile.sh

Download following fi les and put into the source code folder.

Ubuntu_gnu_mpi.py ダウンロード
Ubuntu_gnu.py ダウンロード

>./waf clean

>./waf configure --use-config-dir=/opt/aster/15.2/share/aster --use-config=Ubuntu_gnu_mpi --prefix=/opt/aster/15.2MP

>./waf build --jobs=1

>./waf install

When build is fi nished, sign up the parallel version in /opt/aster/etc/codeaster/aster.

Testing

In case of petsc01a.

ress file is

petsc01a.mess ダウンロード

Enjoy!