openEMS & MPI

How to use openEMS. Discussion on examples, tutorials etc

Moderator: thorsten

Olf
Posts: 69
Joined: Thu 13 Sep 2012, 07:53

openEMS & MPI

Post by Olf » Fri 26 Apr 2013, 08:09

Hi,
Is there any example how to use openEMS with MPI? I would like to do some simulations on a linux cluster but don't quite understand how to do that from the matlab help...
Any example or explanation would be greatly appreciated :)
Olf

thorsten
Posts: 1438
Joined: Mon 27 Jun 2011, 12:26

Re: openEMS & MPI

Post by thorsten » Fri 26 Apr 2013, 08:41

Hi Olf,

MPI in openEMS is highly experimental, expect crashes and other problems :D
The reason is, that I only very rarely have tested using MPI and in some cases it did not work, but I didn't figure out why...

Tell me something about your cluster, is it a collection of different Linux PC's connected by a network or is it a HPC?
I have only experience with a cluster of network-connected PC's .

Firstly, have you tired to simulate using remote ssh on that machines? http://openems.de/index.php/Remote_SSH
It is very handy for sweeps...

MPI Setup:
First you need to specify how to split the simulation domain:

Code: Select all

FDTD = InitFDTD();
FDTD = SetupMPI(FDTD,'SplitN_X',2 ,'SplitPos_Z', '-500,500');
That above would split equally in 2 parts x-direction, and split the domain in z-direction at z=-500 and z=500 --> 3 parts...

Finally setup the MPI options:

Code: Select all

  % mpi binary path on all nodes needed
  Settings.MPI.Binary = '/opt/openEMS/openEMS';
  % number of processes to run
  Settings.MPI.NrProc = 6;
  % define the mpi hosts :
  Settings.MPI.Hosts = {'host1','host2','host3','host4','host5',host6'};
RunOpenEMS(Sim_Path,Sim_File,'-v',Settings);
Since I think you need ssh access, try the remote SSH approach first.
But again, I'm not an MPI expert... Any bug-reports, improvement hints are welcome.

regards
Thorsten

Olf
Posts: 69
Joined: Thu 13 Sep 2012, 07:53

Re: openEMS & MPI

Post by Olf » Fri 26 Apr 2013, 10:08

Hi,

The cluster is an HPC (SUSE Linux Enterprise Server 11 (x86_64)). I currently start the simulations directly on the machines (using ssh & putty). I always need to do both FDTD + NF2FF, so I am not sure if that is easily possible with the remote ssh method. It is already about three times faster than on my local PC but I am planning some larger simulations in the future where the MPI approach would be very useful. From the little knowledge I have about MPI, I thought the nodes would be assigned automatically?
I'll try to follow your instructions and see how it goes. I assume I should use the latest openEMS version from the repository? At the moment I have v0.0.30 installed.

Cheers,
Olf

thorsten
Posts: 1438
Joined: Mon 27 Jun 2011, 12:26

Re: openEMS & MPI

Post by thorsten » Fri 26 Apr 2013, 10:46

Hi,
I always need to do both FDTD + NF2FF, so I am not sure if that is easily possible with the remote ssh method.
It is, just give it a try with the remote ssh. What it does is it copies everything to the HPC, runs the simulation and copies all results back and performs the nf2ff and postprocessing locally. You don't even need Matlab/Octave on the HPC... only openEMS is executed.

You just need a password-less login, which is not that trivial with putty.

About the MPI, I think you only need the SetupMPI to tell openEMS how to split the simulation.
Than you run it with (on the HPC):

Code: Select all

mpirun -np 4 openEMS sim.xml > openEMS.log
where 4 is the number of parallel jobs. You don't need the settings and list of hosts etc...
But regarding the nf2ff we will have to implement some missing parts. Currently this will not work, but should be easy to fix.

Keep me informed...

rgeards
Thorsten

sebastian
Posts: 114
Joined: Mon 27 Jun 2011, 12:36
Contact:

Re: openEMS & MPI

Post by sebastian » Fri 26 Apr 2013, 13:02

using putty passwordless is easy to accomplish: generate a key pair vith the putty tools, copy the public key to the linux system and start pageant on the windows side (add the previously generated key). From now on, putty does not require a password and openEMS will start directly.

Sebastian

thorsten
Posts: 1438
Joined: Mon 27 Jun 2011, 12:26

Re: openEMS & MPI

Post by thorsten » Fri 26 Apr 2013, 13:33

@Seb, that is mostly what I describe on the wiki page. But without pageant, just with defining where to find the private key in the openEMS Settings...

But compared to a "ssh-copy-id" on Linux, I just called it not trivial ;)

regards
Thorsten

Olf
Posts: 69
Joined: Thu 13 Sep 2012, 07:53

Re: openEMS & MPI

Post by Olf » Mon 29 Apr 2013, 03:36

Hi guys,

thanks for the replies! I actually want to run both FDTD and NF2FF on the cluster so I guess it would be the easiest to just run the whole script there. ALso then I can just let things run over night or over the weekend :)
I'll try to use MPI and see how it goes. I am a bit confused because there is the RunOpenEMS_MPI function where I have to specify the host names. If I understand you correctly, you suggest to create the xml file with octave/matlab and then start mpirun from the console?

Regards,
Olf

thorsten
Posts: 1438
Joined: Mon 27 Jun 2011, 12:26

Re: openEMS & MPI

Post by thorsten » Mon 29 Apr 2013, 08:44

Hi,

RunOpenEMS_MPI is executing this mpirun etc. and on my cluster here (or maybe I call it a bunch of Linux PC) I needed to specify the list of hosts. In your case you don't and maybe we have to modify the RunOpenEMS_MPI to accept a missing host list. Maybe it already works? I'm not sure.

Give it a try...

regards
Thorsten

thorsten
Posts: 1438
Joined: Mon 27 Jun 2011, 12:26

Re: openEMS & MPI

Post by thorsten » Mon 29 Apr 2013, 09:40

Hi,

I just made a few changes to RunOpenEMS_MPI to accept a missing host-list and run on the local machine.
One problem left open is the nf2ff calculations... Because what I do with MPI is that every node is dumping his fields into his own dump-box-files... The nf2ff would therefore need to read all this files, which it currently doesn't know anything about...

One question, what do you want to do? Because the total-field/scattered-field approach is (currently) not compatible with MPI as well. But it is not a good idea to use with very large domains anyways...

regards
Thorsten

Olf
Posts: 69
Joined: Thu 13 Sep 2012, 07:53

Re: openEMS & MPI

Post by Olf » Tue 30 Apr 2013, 01:38

Ah I see, well that's [scattered filed/total field -> nf2ff] what I wanted do... Well, that's not a problem actually, I just change my strategy. I'll figure out how large structures I'll be able to simulate and for the rest I just start several jobs parallel :)
Cheers,
Olf

Post Reply