This post documents how to compile the latest release of WRF (version 3.8.1) on the ARCHER HPC service. There is already a compiled version available on archer that be accessed using the modules funciton (see this guide), but you will need to be able to compile from the source code if you are making any modifications to the code, or if you need to compile the idealised cases, or need to compile it with a different nesting set up. (The pre-compiled code is set up for basic nesting only).

## Compilation with the default ARCHER compiler (Cray CCE)

This is relatively straightforward as the configure script already works as expected. However, compiling with the Cray compiler (usually the default loaded compiler when you login to ARCHER) can take upwards of 6-8 hours, depending on the options selected in configure, and the load on the login or serial nodes.

### Setting up your ARCHER environment

First, check that you do actually have the Cray compiler environment loaded. You can look for PrgEnv-cray in the output from running the module list command, or you can do echo $PE_ENV from a login session. Because it takes so long to compile using the Cray compiler, you need to run the compilation task as a serial job on the serial nodes. If you attempt to run on the login nodes, the compilation process will time out well before completion. So we are going to prepare three things: 1. A ‘pre build’ script that will load the correct modules on ARCHER and set up the environment variables 2. The configure.wrf file. This is prepared using the configure script, so you should not need to do anything different to the normal WRF compilation instructions for this part. 3. A compliation job submission (.pbs) script. This will be submitted as a serial node job to do the actual compilation bit. #### The pre-build script This is a shell script (of the bash flavour) that loads the relevant modules for WRF to compile. pre-build.bash NetCDF-4 is the default netcdf module on the ARCHER environment, so I am assuming you want to compile it with netcdf-4 (it has extra features like supporting file compression etc…) I attempted this with the gcc/5.x.x module, but ran into compilation errors. Using GCC v6 seemed to fix them. Note that you may need to switch a load statement for a swap statement in some places, depending on what default modules are loaded in your ARCHER environment. See your .profile and .bashrc scripts. NOTE: .profile (in the $HOME directory is only loaded on the login shells. If you launch any other interactive shells (like an interactive job mode), then .bashrc will get loaded.

pre-build.bash (continued)

Some things to note:

1. You cannot compile WRF in parallel (it’s a bug in the Makefile, apparrently)
2. There are some environment variables that are just set equal to some other environmnet variables, e.g. NETCDF=$NETCDF_DIR. This works because when you use the module system to load, say, netCDF, ARCHER will automatically set its own environment variables that we can use to initialise the WRF configure variables, e.g. $NETCDF.

### Run configure

For the CRAY compiler, this can be run as normal e.g. ./configure from the WRFV3 directory.

### Compilation job script

At this stage, you can either request an interactive mode job on the serial nodes, and then run compile in the usual way (after running the prebuild script and the configure commands), or you can submit a serial job with the PBS job scheduling system to run when a node becomes available. If you are going down the interactive job mode, be sure to request enough walltime as the Cray compiler takes a long time to compile everything. I would ask for 12 hours to be on the safe side.

If you want to submit a job to run without having to wait for an interactive-mode job, prepare the following job submission script:

### Putting it all together.

1. Make sure the pre-build.bash script and the compile.pbs script are in a directory above /WRFV3. I called it WRF_build381

2. Use qsub as normal to submit the compile.pbs script. E.g. qsub compile.pbs

Your job should run and the compile logs will be written to compileCray.log (or whatever you named them in the compile.log script above.

## Compiling WRF using the GNU compilers on ARCHER

You may have reason to want to compile WRF with the GNU compilers on ARCHER or another Cray XC30 system. Unfortunately I found that the configure script supplied with version 3.8.1 did not generate a correct configure.wrf script for the GNU compilers in a Cray enviroment. Namely, it used compilation flags specific to the Cray compiler, rather than the gfortran compilation flags (which are incompatible). To rectify this you can either run the configure script as normal, and then correct the compiler flags in the configure.wrf output script that is generated. Or if you want a more re-usable soultion you can edit the file in the WRFV3/arch/configure_new.defaults file.

I did this by opening the configure_new.defaults file and adding a new entry. The purpose of the file is to generate the menu entries that you see when running the configure script, and then populate the Makefile with the correct compilation options.

Find the CRAY CCE entry in the configure_new.defaults file and insert a new entry below it called GNU on CRAY XC30 system or similar. The entry should contain the following:

###########################################################
#ARCH    Cray XE and XC CLE/Linux x86_64, GNU Compiler on Cray System # serial dmpar smpar dm+sm
# Use this when you are using the GNU programming environment on ARCHER (a Cray system)

DESCRIPTION     =       GNU on Cray system ($SFC/$SCC): Cray XE and XC
# OpenMP is enabled by default for Cray CCE compiler
# This turns it off
DMPARALLEL      =       # 1
OMPCPP          =       # -D_OPENMP
OMP             =       # -fopenmp
OMPCC           =       # -fopenmp
SFC             =       ftn
SCC             =       cc
CCOMP           =       gcc
DM_FC           =       ftn
DM_CC           =       cc
FC              =       CONFIGURE_FC
CC              =       CONFIGURE_CC
LD              =       $(FC) RWORDSIZE = CONFIGURE_RWORDSIZE PROMOTION = #-fdefault-real-8 ARCH_LOCAL = -DNONSTANDARD_SYSTEM_SUBR -DWRF_USE_CLM CFLAGS_LOCAL = -O3 LDFLAGS_LOCAL = CPLUSPLUSLIB = ESMF_LDFLAG =$(CPLUSPLUSLIB)

FCOPTIM         =       -O2 -ftree-vectorize -funroll-loops
FCREDUCEDOPT    =       $(FCOPTIM) FCNOOPT = -O0 FCDEBUG = # -g$(FCNOOPT) # -ggdb -fbacktrace -fcheck=bounds,do,mem,pointer -ffpe-trap=invalid,zero,overflow
FORMAT_FIXED    =       -ffixed-form
FORMAT_FREE     =       -ffree-form -ffree-line-length-none
FCSUFFIX        =
BYTESWAPIO      =       -fconvert=big-endian -frecord-marker=4
FCBASEOPTS_NO_G =       -w $(FORMAT_FREE)$(BYTESWAPIO)
FCBASEOPTS      =       $(FCBASEOPTS_NO_G)$(FCDEBUG)
FCBASEOPTS_NO_G =       -N1023 $(FORMAT_FREE)$(BYTESWAPIO) #-ra
FCBASEOPTS      =       $(FCBASEOPTS_NO_G)$(FCDEBUG)
MODULE_SRCH_FLAG =

You can run the configure script as normal once these changes have been made and you will get a configure.wrf suitable for using the GNU compilers on ARCHER to build WRF v3.8.1.