• Central Processing Unit: Executes instructions of a computer program.
• Modern CPUs have multiple processor cores (“processes”).
  • Different processes can execute different tasks at same time (multi-processing).
  • Each core has its own memory.
• Modern cores/CPU execute several tasks at the same time (multi-threading)
  • Each task is called a thread
  • Threads will share memory.

• When is multi-threading better than multi-processing?
• Multi-threading: All workers share same memory.
  • Faster set-up and faster to share memory across workers.
  • Need to be careful when workers edit same memory (“data race”).
  • All workers need to be on same computer/server.
• Multi-processing: All workers have their own memory
  • Slower to set up and share memory.
  • Need to be careful about what information each worker needs.
  • Workers can exist across computers/servers.

• Multi-processing and multi-threading allow you to complete tasks in parallel and speed up your code if done well.
• So far, you are not taking full advantage of computation resources available to you.
• Your laptop likely has several cores that can have two threads of execution.
• UW-Madison has a lot of computing power available for you to use.

https://sscc.wisc.edu/sscc/pubs/grc/resources.html

1. Linstat: 4 interactive Linux servers
   • Each server has 36 or 48 cores.
   • Good for checking that your code actually works on SSCC’s servers before submitting to slurm.
2. Slurm: 40 Servers with a total of 5,000 cores.
   • Most servers have 128 cores.
   • You can let your code run for up to 10 days.

Be mindful that other people need to use Linstat and Slurm, too.

• SSCC has other resources for jobs that need lots of memory (Winstat BIG) and jobs that use confidential data (SILO).

https://chtc.cs.wisc.edu/

• High Performance Computing (HPC): Multi-processing tasks.
• High Throughput Computing Cluster (HTC): Multi-threading tasks.
• I am not so familiar with these resources.

• Value function iteration to solve for \(V\) \[ V_{n+1}(z,k) = \max_{c,k'} \:\:\: \log(c) + \beta \textbf{E}_{z'}[V_{n}(z',k')|z]. \] \[ c + k' = zk^\alpha + (1-\delta)k. \]
• We are solving on some capital grid \(\{k_1, k_2, ..., k_n\}\).
• Can ask different cores/workers to solve for \(\{k'(z,k), c(z,k), V_{n+1}(z,k)\}\) at different initial \(k\).
• Then gather all information together, and go again.

• Now we have solved for \(V\).
• Suppose we want to simulates lots of histories of \(c\) and \(k\).
• We generate a bunch of random paths for productivity \(\{z_t\}\).
• Simulate paths of \(c\) and \(k\).
• Instead of doing this all on one core, we can simulate many different paths at the same time using parallel computing.

• Parallelization is not a free lunch because of over-head costs.
• Having many cores running uses more memory.
• Sharing memory across cores takes time.
• If you are doing lots of calculations that take a long time to run, parallelization can help.
• There are also diminishing returns that can kick in quickly for smaller problems.

https://www.sas.upenn.edu/~jesusfv/Guide_Parallel.pdf

• The easiest way to log in to Linstat/Slurm is to first log on to Winstat.
• You should have made a Winstat account/password at first-year orientation.

https://kb.wisc.edu/sscc/using-winstat

• You can log into Winstat with Citrix Workspace or at the following website. Web log-in cannot access local files.

https://winstat.ssc.wisc.edu/vpn/index.html

• If you need to reset your password:

https://www.sscc.wisc.edu/sscc_jsp/password/reset.jsp

• Open interactive julia REPL.

  julia

• Run a julia script file and send save output to text file.

  julia script.jl > output.txt & 

• Change working directory to a folder.

  cd FolderPath

• See jobs running on current Linstat server.

  top

• To close top, press “q”.
• Stop a job PID that is currently running

  kill PID

• To run a script file on slurm with N cores and M gb of memory:

  ssubmit --cores=N --mem=Mg "julia script.jl"

• To figure out how much memory you need:
  1. Run your job on Linstat and check its memory use (RES).
  2. Then kill your job and send it to slurm.
     
• Slurm will send you an email when your job is done running.
• You can submit to specific partitions of the server using the option --part.

https://sscc.wisc.edu/sscc/pubs/grc/slurm.html#partitions-priorities

• To see jobs that are running on slurm, use

squeue -a

• To cancel your job:

scancel JOBID

• Linstat is good for:
  1. Interactive coding.
  2. Checking that your code runs on SSCC servers.
  3. See how much memory you are using.
  4. Sending jobs to slurm.
• Slurm is good for big jobs that need lots of cores and run for a long time.