ATPESC 2013 PUMI Tutorial

Exercise 1 - Mesh partitioning

afosr geometric model

  • Demonstrates
  • Link to the source code
    • pumi_mesh/test/partitioning/main.cc
  • Example data files
    • Executable: /gpfs/vesta-fs0/projects/FASTMath/examples/pumi/ex1/pumi_mesh_testpartitioning
    • Input files: /gpfs/vesta-fs0/projects/FASTMath/examples/pumi/ex1/afosr_60e3tets_2p.tar.gz
    • Output files: /gpfs/vesta-fs0/projects/FASTMath/examples/pumi/ex1/output/
  • Executing the example
    • Execution time: 1-2 mins per tool
    • Number of cores: 2
    • Setup
      soft delete +mpiwrapper-gcc
      soft add +mpiwrapper-xl
      mkdir $HOME/pumi
      cd $HOME/pumi
      cp -r /gpfs/vesta-fs0/projects/FASTMath/examples/pumi/ex1 .
      cd ex1
      tar xzf afosr_60e3tets_2p.tar.gz 
      
    • Run the partitioning tools and write Parview (*VTK/*vtu) files.
      # run local partitioning with ParMETIS
      mkdir localParMetisVTK/
      qsub -A ATPESC2013 -q Q.ATPESC2013 -O localParMetisPtn -t 10 --mode c2 -n 1 $PWD/pumi_mesh_testpartitioning -m afosr_60e3tets_2p/afosr_60e3tets_2p.sms -t 3 -p 8 -w localParMetisVTK/afosr_60e3tets_localParMetis_16p.vtk 
      # run local partitioning with Zoltan RCB
      mkdir localGeoVTK/
      qsub -A ATPESC2013 -q Q.ATPESC2013 -O localGeoPtn -t 10 --mode c2 -n 1 $PWD/pumi_mesh_testpartitioning -m afosr_60e3tets_2p/afosr_60e3tets_2p.sms -t 7 -p 8 -w localGeoVTK/afosr_60e3tets_localGeoPtn_16p.vtk 
      # run global partitioning with ParMETIS multi-level 
      mkdir globalParMetisVTK/
      qsub -A ATPESC2013 -q Q.ATPESC2013 -O globalParMetisPtn -t 10 --mode c2 -n 1 $PWD/pumi_mesh_testpartitioning -m afosr_60e3tets_2p/afosr_60e3tets_2p.sms -t 6 -p 8 -w globalParMetisVTK/afosr_60e3tets_globalParMetisPtn_16p.vtk
      
  • Expected/Sample Output
    • see /gpfs/vesta-fs0/projects/FASTMath/examples/pumi/ex1/output/*.output
  • Examining results
    • Note the differences between the three partitioning methods in the final mesh imbalance and the execution time
      # The first instance is the initial element imbalance, the last is the final element imbalance.
      grep "Imbalance ratio \[max/avg weight\]" *.output
      # wall run time
      grep "elapsed time" *.output
      
    • see the next two exercises for viewing meshes in ParaView

Exercise 2 - Mesh Visualization

Serial Visualization

  • Download ParaView
  • Transfer the *VTK directories to your machine
  • Load all three meshes
    • Select the folder icon in the top left then browse to your home directory and select one of the *VTK/.pvtu files.
    • Click 'Apply' to display the mesh.
  • Expected/Sample Output
    • Each part is shaded with a different color. Note how structured the geometric partitioning is compared to the graph based partitions.
      paraview localGeoPtn paraview localParMetisPtn paraview globalParMetisPtn
      localGeoPtn localParMetisPtn globalParMetisPtn

Parallel Mesh Visualization - OPTIONAL

Exercise 3 - Mesh adaptation

  • Demonstrates
  • Link to the source code
    • meshadapt/meshAdapt/meshAdapt/test/simModel/adaptSimModel.cc
  • Example data files
    • Executable: /gpfs/vesta-fs0/projects/FASTMath/examples/pumi/ex3/adaptSimModel
    • Input files: /gpfs/vesta-fs0/projects/FASTMath/examples/pumi/ex3/afosr_60e3tets_16p.tar.gz
    • Output files: /gpfs/vesta-fs0/projects/FASTMath/examples/pumi/ex3/output/
  • Executing the example
    • Execution time: 5 mins
    • Number of cores: 16
    • Setup
      # assuming exercise 1 was already ran 
      cd $HOME/pumi
      cp -r /gpfs/vesta-fs0/projects/FASTMath/examples/pumi/ex3 .
      cd ex3
      tar xzf afosr_60e3tets_16p.tar.gz 
      
    • Run uniform refinement followed by partitioning to 128 parts.
      The following will generate vtk files for the adapted mesh (adaptedVTK/*vtu).
      mkdir adaptedVTK
      qsub -A ATPESC2013 -q Q.ATPESC2013 -t 10 -O adapt --mode c8 -n 2 $PWD/adaptSimModel afosr_60e3tets_16p/afosr_60e3tets_16p.sms afosr_60e3tets_16p/afosr.smd 0 8 adaptedVTK/afosr_480e3tets_128p.vtk 
      
  • Expected/Sample Output
    • see /gpfs/vesta-fs0/projects/FASTMath/examples/pumi/ex3/output/adapt.output
  • Examining results
    • Notice the 8x increase in mesh elements; the result of splitting all mesh edges in an all tetrahedron mesh.
      # The first instance is the initial element count, the second is the final element count.
      grep "#region" adapt.output
      
    • Copy the adaptedVTK/*vtu files to Tukey. Use two Tukey nodes to visualize this mesh. Open the mesh files in ParaView and notice that the new mesh vertices classified on geometric model vertices/edges/faces are 'snapped' to the geometric model.
      pre-adapt paraview preAdapt zoom1 paraview preAdapt zoom2 paraview preAdapt zoom3
      adapted paraview Adapted zoom1 paraview Adapted zoom2 paraview Adapted zoom3
      Zoom1 Zoom2 Zoom3

Exercise 4 - Size field based mesh adaptation

analytic size field

  • Demonstrates
  • Link to the source code
    • meshadapt/meshAdapt/meshAdapt/test/simModel/adaptSimModel.cc
  • Example data files
    • Executable: /gpfs/vesta-fs0/projects/FASTMath/examples/pumi/ex4/adaptSimModel
    • Input files: /gpfs/vesta-fs0/projects/FASTMath/examples/pumi/ex4/afosr_3.8e6tets_512p.tar.gz
    • Output files: /gpfs/vesta-fs0/projects/FASTMath/examples/pumi/ex4/output/
  • Executing the example
    • Execution time: 18 mins
    • Number of cores: 512
    • Setup
      # assuming exercise 3 was already ran 
      cd $HOME/pumi
      cp -r /gpfs/vesta-fs0/projects/FASTMath/examples/pumi/ex4 .
      cd ex4
      tar xzf afosr_3.8e6tets_512p.tar.gz 
      
    • Run size field based adaptation preceeded by predictive load balancing and followed by dynamic load balancing.
      The following will generate adapted ParaView mesh files (adaptedVTK/*vtu).
      mkdir adaptedVTK
      qsub -A ATPESC2013 -q Q.ATPESC2013 -O adapt -t 30 --mode c16 -n 32 $PWD/adaptSimModel afosr_3.8e6tets_512p/afosr_3.8e6tets_512p.sms afosr_3.8e6tets_512p/afosr.smd 1 1 adaptedVTK/afosr_3.8e6tets_adapted_512p.vtk 
      
  • Expected/Sample Output
    • see /gpfs/vesta-fs0/projects/FASTMath/examples/pumi/ex4/output/adapt.output
  • Examining results
    • Note the imbalance before and after adaptation. Running predictive load balancing prior to adaptation prevents exhaustion of available memory during adaptation.
      # The first instance is the element imbalance after adaptation, the last is the element imbalance after dynamic load balancing.
      grep "Imbalance ratio \[max/avg weight\]" *.output
      
    • Copy the adaptedVTK/*vtu files to Tukey and load them in ParaView. Use four Tukey nodes to visualize this mesh. Notice the refined zones at 60% coord length, the actuator, and into the far field.
      mesh generated from analytic sz fld near actuator mesh generated from analytic sz fld near 60% coord mesh generated from analytic sz fld into the far field
      actuator 60% coord farField

Demo 1 - Mesh Generation with Simmetrix Tools

Additional References

  • PUMI tools
    • M. Zhou, O. Sahni, T. Xie, M.S. Shephard and K.E. Jansen, Unstructured Mesh Partition Improvement for Implicit Finite Element at Extreme Scale, Journal of Supercomputing, 59(3): 1218-1228, 2012. DOI 10.1007s11227-010-0521-0
    • M. Zhou, T. Xie, S. Seol, M.S. Shephard, O. Sahni and K.E. Jansen, Tools to Support Mesh Adaptation on Massively Parallel Computers, Engineering with Computers, 28(3):287-301, 2012. DOI: 10.1007s00366-011-0218-x
    • M. Zhou, O. Sahni, M.S. Shephard, K.D. Devine and K.E. Jansen, Controlling unstructured mesh partitions for massively parallel simulations, SIAM J. Sci. Comp., 32(6):3201-3227, 2010. DOI: 10.1137090777323
    • M. Zhou, O. Sahni, H.J. Kim, C.A. Figueroa, C.A. Taylor, M.S. Shephard, and K.E. Jansen, Cardiovascular Flow Simulation at Extreme Scale, Computational Mechanics, 46:71-82, 2010. DOI: 10.1007s00466-009-0450-z
  • Mesh data and geometry interactions
    • Seol, E.S. and Shephard, M.S., Efficient distributed mesh data structure for parallel automated adaptive analysis, Engineering with Computers, 22(3-4):197-213, 2006, DOI: 10.1007s00366-006-0048-4
    • Beall, M.W., Walsh, J. and Shephard, M.S, A comparison of techniques for geometry access related to mesh generation, Engineering with Computers, 20(3):210-221, 2004, DOI: 10.1007s00366-004-0289-z.
    • Beall, M.W. and Shephard, M.S., A general topology-based mesh data structure, Int. J. Numer. Meth. Engng., 40(9):1573-1596, 1997, DOI: 10.1002(SICI)1097-0207(19970515)40:9<1573::AID-NME128>3.0.CO;2-9.
  • Adaptivity
    • Aleksandr Ovcharenko, Parallel Anisotropic Mesh Adaptation with Boundary Layers, Ph.D. Dissertation, RPI, 2012
    • Q. Lu, M.S. Shephard, S. Tendulkar and M.W. Beall, Parallel Curved Mesh Adaptation for Large Scale High-Order Finite Element Simulations, Proc. 21 Roundtable, Springer, NY, pp. 419-436, 2012, DOI 10.1007978-3-642-33573-0.
    • A. Ovcharenko, K. Chitale, O. Sahni, K.E. Jansen and M.S. Shephard, S. Tendulkar and M.W. Beall, Parallel Adaptive Boundary Layer Meshing for CFD Analysis, Proc. 21st International Meshing Roundtable, Springer, NY, pp. 437-455, 2012, DOI 10.1007978-3-642-33573-0
    • X.-J. Luo, M.S. Shephard, L.-Q. Lee and C. Ng, Moving Curved Mesh Adaption for Higher Order Finite Element Simulations, Engineering with Computers, 27(1):41-50, 2011. DOI: 10.1007/s00366-010-0179-5
    • O. Sahni, X.J. Luo, K.E. Jansen, M.S. Shephard, Curved Boundary Layer Meshing for Adaptive Viscous Flow Simulations, Finite Elements in Analysis and Design, 46:132-139, 2010. DOI: 10.1007/s00366-008-0095-0
    • Alauzet, F., Li, X., Seol, E.S. and Shephard, M.S., Parallel Anisotropic 3D Mesh Adaptation by Mesh Modification, Engineering with Computers, 21(3):247-258, 2006, DOI: 10.1007s00366-005-0009-3
    • Li, X., Shephard, M.S. and Beall, M.W., 3-D Anisotropic Mesh Adaptation by Mesh Modifications, Comp. Meth. Appl. Mech. Engng., 194(48-49):4915-4950, 2005, doi:10.1016/j.cma.2004.11.019
    • Li, X., Shephard, M.S. and Beall, M.W., Accounting for curved domains in mesh adaptation, International Journal for Numerical Methods in Engineering, 58:246-276, 2003, DOI: 10.1002/nme.772

Attachments

afosr_1stpair_0aoa.png - afosr geometric model (33.8 KB) Cameron Smith, 07/31/2013 02:04 am

paraviewLocalGeoPtn.png - paraview localGeoPtn (128 KB) Cameron Smith, 07/31/2013 04:17 am

paraviewLocalParMetisPtn.png - paraview localParMetisPtn (131 KB) Cameron Smith, 07/31/2013 04:17 am

paraviewGlobalParMetisPtn.png - paraview globalParMetisPtn (136 KB) Cameron Smith, 07/31/2013 04:17 am

paraviewApply.png - paraview apply button (106 KB) Cameron Smith, 07/31/2013 04:24 am

paraviewPreAdaptZoom.png - paraview preAdapt zoom1 (164 KB) Cameron Smith, 07/31/2013 04:49 am

paraviewPreAdaptZoom2.png - paraview preAdapt zoom2 (276 KB) Cameron Smith, 07/31/2013 04:49 am

paraviewPreAdaptZoom3.png - paraview preAdapt zoom3 (284 KB) Cameron Smith, 07/31/2013 04:49 am

paraviewAdaptedZoom.png - paraview Adapted zoom1 (174 KB) Cameron Smith, 07/31/2013 04:49 am

paraviewAdaptedZoom2.png - paraview Adapted zoom2 (345 KB) Cameron Smith, 07/31/2013 04:49 am

paraviewAdaptedZoom3.png - paraview Adapted zoom3 (376 KB) Cameron Smith, 07/31/2013 04:49 am

SJ_inst_vortX_speed_backplane.mov - actuator animation (262 KB) Cameron Smith, 08/01/2013 12:27 pm

analyticSizeField.png - analytic size field (225 KB) Cameron Smith, 08/02/2013 12:47 am

actCrdFar45e6viewAct.png - mesh generated from analytic sz fld near actuator (280 KB) Cameron Smith, 08/02/2013 04:12 pm

actCrdFar45e6viewCrd.png - mesh generated from analytic sz fld near 60% coord (297 KB) Cameron Smith, 08/02/2013 04:12 pm

actCrdFar45e6viewFar.png - mesh generated from analytic sz fld into the far field (299 KB) Cameron Smith, 08/02/2013 04:12 pm

scramjetAdaptiveExample.pdf (925 KB) Cameron Smith, 08/03/2013 11:17 am