T 09240

Setup aips on ares:

 source /usr/local/pkg/AIPS_31DEC06/LOGIN.CSH
 $CDTST

Run aips

 aips tv=local

 NUM CHANNELS:       512

NOTE: 1024 chans doesn’t seem to work!

When the correlation is finished

1. hex5: vsib_send -H hex5 -p 52100 *.lba but don’t press [Enter] yet!
2. hex4: vsib_send -H hex4 -p 52101 *.lba but don’t press [Enter] yet!
3. hex0: cd ~/correlations/t09240
4. hex0: ./run.sh
5. When you see this:

[ -1] INFO MPI Process 0 is running on host hex0.phys.utas.edu.au Press [Enter] in the hex5 and hex4 windows. Data should start moving and the RPF file should appear and grow.

Press Ctrl-C to stop correlation. Before re-starting, change the input file to give a different output RPF file, or delete the existing RPF file, otherwise DiFX won’t start.

1. copy machines, run.sh2 and .threads file from a previous experiment. note the threads file should now be named “threads”, not “<expt_code>.threads”

 -cross indicates create cross-products
 -evlbi indicates data will be streamed using vsib_send and vsib_recv
 -new for the new version of DiFX

This creates t09240.input

First time with data to both telescopes is 05:04:20. Run time2sec.pl to convert that time to seconds:

Now make the model

 vex2model_updated.pl t09240.skd

Edit the machines file. We’re not using hex1, so it looks like this:

 hex0
 hex5
 hex4
 hex2
 hex3
 hex4
 hex5

Hobart data on hex5, Ceduna on hex4

edit the threads file. Note number of spaces between text is important:

 NUMBER OF CORES:    4
 8
 8
 7
 7

edit run.sh

 -np 7 (where 7 is # processes, = wc -l machines)

Then start correlation:

1. Open 3 windows, one on hex0 (head node), one on hex5 (Hobart data), one on hex4 (Ceduna data)
   1. hex5: cd /data/glast_agn/t09240
   2. hex4: cd /data_fermi/t09240/cd
   3. in one of the above (if necessary), delete any initial data files where there’s no overlap
   4. hex0:

edit run.sh

 -np 8 (where 8 is # processes, = wc -l machines)

VEX2config evlbi mode to use vsib_send and recv instead of specifying files by hand. Also, selecting three antennas only:

 vex2config.pl -evlbi -ant Pa -ant At -ant Mp vt12a.skd 

Example

1. on hex0:

    cd ~/correlations
    mkdir t08232 (or whatever the experiment name is)

2. copy machines, run.sh2] and [@.threads file from a previous experiment
3. copy .skd file across from hobart

    vex2config.pl -cross -evlbi t08232.skd

4. edit the .input file for a fringe search / Ceduna check. e.g First 20 mins is 1921–293:

 EXECUTE TIME: 1200
 INT TIME (SEC):     0.5
 FREQ (MHZ) 0:       6640.00

remove the semi-colon from end of READ FROM FILE If pols swapped at Ceduna, for CED:

 INPUT BAND 0 POL:   L
 INPUT BAND 1 POL:   R

1. vex2model_updated.pl t08232.skd
2. edit machines:

 hex1
 hex2
 hex3
 hex4
 hex5

1. edit t08232.threads:

 7

Test with Agilent and SMY on 3c273

Correlation notes

512 chan delay fix pols

1. on hex0:

    cd ~/correlations
    mkdir t09240

2. copy machines, run.sh2 and .threads file from a previous experiment. note the threads file should now be named “threads”, not “<expt_code>.threads”%
3. copy .skd file across from hobart

    vex2config.pl -cross -evlbi -new t09240.skd

-cross indicates create cross-products -evlbi indicates data will be streamed using vsib_send and vsib_recv -new for the new version of DiFX

creates <blah>.input

edit the input file and check the following:

 EXECUTE TIME (SEC): 
 START MJD:          
 START SECONDS:      

First time with data to both telescopes is 05:04:20

 observer@hex0:~/correlations/t09240$ time2sec.pl 05:04:20
  05:04:20 → 18260

so in the input file:

 START SECONDS:      18260

For an initial fringe check, set

 EXECUTE TIME (SEC): 300
 OUTPUT FILENAME: /home/observer/correlations/t09240/t09240_1.rpf

Lots of channels for initial search

 NUM CHANNELS:       1024

It turned out that default clocks were way out. Try these for a start:

 # TELESCOPE TABLE ##!
 TELESCOPE ENTRIES:  2
 TELESCOPE NAME 0:   HOB
 CLOCK DELAY (us) 0: −3
 CLOCK RATE(us/s) 0: 0.0
 TELESCOPE NAME 1:   CED
 CLOCK DELAY (us) 1: 1.05
 CLOCK RATE(us/s) 1: 0.0

Polarisations are swapped at Ceduna, so in the DATASTREAM TABLE, swap pols for second antenna (Ceduna):

 TELESCOPE INDEX:    1
 …
 INPUT BAND 0 POL:   L
 INPUT BAND 0 INDEX: 0
 INPUT BAND 1 POL:   L
 INPUT BAND 1 INDEX: 1
 INPUT BAND 2 POL:   R
 INPUT BAND 2 INDEX: 0
 INPUT BAND 3 POL:   R
 INPUT BAND 3 INDEX: 1

Also in the baseline table:

 # BASELINE TABLE ###!
 …
 POL PRODUCTS 0/1:   4
 D/STREAM A BAND 0:  1
 D/STREAM B BAND 0:  3
 D/STREAM A BAND 1:  3
 D/STREAM B BAND 1:  1
 D/STREAM A BAND 2:  1
 D/STREAM B BAND 2:  1
 D/STREAM A BAND 3:  3
 D/STREAM B BAND 3:  3

Lastly, the port numbers for data

 # NETWORK TABLE ####!
 PORT NUM 0:         52100
 TCP WINDOW (KB) 0:  512
 PORT NUM 1:         52101
 TCP WINDOW (KB) 1:  512

time2sec

Times in seconds are from the start of the day. Need to change these if correlating a subset of the experiment

Configurations

 NUM CONFIGURATIONS: 1        leave
 CONFIG SOURCE:      DEFAULT   leave
 INT TIME (SEC):     2        change if you want to
 NUM CHANNELS:       128      cahnge
 BLOCKS PER SEND:    1250     change in extreme cases (e.g. lots of spectral pts) for performance
 GUARD BLOCKS:       1        leave don’t know what this is
 POST-F FRINGE ROT:  FALSE    TRUE is faster but avoid when high spec res, high fringe rate
 QUAD DELAY INTERP:  TRUE     leave
 WRITE AUTOCORRS:    TRUE     change (if you want to)
 PULSAR BINNING:     FALSE    leave (unless you’re interested in pulsars)
 DATASTREAM 0 INDEX: 0        leave
 DATASTREAM 1 INDEX: 1      leave
 BASELINE 0 INDEX:   0   leave 

 # FREQ TABLE #######! leave but check it agrees with obs

 # TELESCOPE TABLE ##!    Change. Use GPS-Maser as first guess. Correlate on a fringe-finder, fringe fit and apply in production run. If monitoring GPS-Maser during the experiment use the rate from that, it’s probably better.
 TELESCOPE ENTRIES:  2
 TELESCOPE NAME 0:   HOB
 CLOCK DELAY (us) 0: −10.6
 CLOCK RATE(us/s) 0: 0.0
 TELESCOPE NAME 1:   CED
 CLOCK DELAY (us) 1: 1.05
 CLOCK RATE(us/s) 1: 0.0 

 # DATASTREAM TABLE #!           leave, except maybe Tsys (in Jy)
 DATASTREAM ENTRIES: 2
 DATA BUFFER FACTOR: 256
 NUM DATA SEGMENTS:  16
 TELESCOPE INDEX:    0
 TSYS:               1240
 DATA FORMAT:        LBAVSOP
 QUANTISATION BITS:  2
 FILTERBANK USED:    FALSE
 READ FROM FILE:     TRUE
 NUM FREQS:          1
 FREQ TABLE INDEX 0: 0
 CLK OFFSET 0 (us):  0.000       leave but this is the clock offset per freq. may be used after fringe-fit to correct for differing delays. 
 NUM POLS 0:         2
 INPUT BAND 0 POL:   R  check this is correct.
 INPUT BAND 0 INDEX: 0
 INPUT BAND 1 POL:   L
 INPUT BAND 1 INDEX: 0


 # BASELINE TABLE ###!    defines what to process
 BASELINE ENTRIES:   1
 D/STREAM A INDEX 0: 0
 D/STREAM B INDEX 0: 1    baseline 0 correlates antennas 0 and 1
 NUM FREQS 0:        1
 POL PRODUCTS 0/0:   2
 D/STREAM A BAND 0:  0    A0-B0 (RCP)
 D/STREAM B BAND 0:  0
 D/STREAM A BAND 1:  1    A1-B1 (LCP)
 D/STREAM B BAND 1:  1 

For cross-products too

 vex2config.pl -cross chsb6324.skd 

and the above section will look like this:

 POL PRODUCTS 0/0:   4
 D/STREAM A BAND 0:  0
 D/STREAM B BAND 0:  0
 D/STREAM A BAND 1:  1
 D/STREAM B BAND 1:  1
 D/STREAM A BAND 2:  0
 D/STREAM B BAND 2:  1
 D/STREAM A BAND 3:  1
 D/STREAM B BAND 3:  0


 # DATA TABLE #######     Need to edit this bit if not using evlbi (vsib_send/recv) mode:
 should be the number of raw data files in each directory e.g.:

 # DATA TABLE #######
 D/STREAM 0 FILES:   6
 FILE 0/0:           /data/sb6324/ho/sb6324_Ho_323_230000.lba
 FILE 0/0:           /data/sb6324/ho/sb6324_Ho_323_230010.lba
 FILE 0/0:           /data/sb6324/ho/sb6324_Ho_323_230020.lba
 FILE 0/0:           /data/sb6324/ho/sb6324_Ho_323_230030.lba
 FILE 0/0:           /data/sb6324/ho/sb6324_Ho_323_230040.lba
 FILE 0/0:           /data/sb6324/ho/sb6324_Ho_323_230050.lba
 D/STREAM 1 FILES:   6
 FILE 0/0:           /data/sb6324/cd/sb6324_Cd_323_230000.lba
 FILE 0/0:           /data/sb6324/cd/sb6324_Cd_323_230010.lba
 FILE 0/0:           /data/sb6324/cd/sb6324_Cd_323_230020.lba
 FILE 0/0:           /data/sb6324/cd/sb6324_Cd_323_230030.lba
 FILE 0/0:           /data/sb6324/cd/sb6324_Cd_323_230040.lba
 FILE 0/0:           /data/sb6324/cd/sb6324_Cd_323_230050.lba


 # NETWORK TABLE ####!      for eVLBI. 
 PORT NUM 0:         52100       port number for data stream 1
 TCP WINDOW (KB) 0:  512         TCP window size
 PORT NUM 1:         52101      port number for data stream 2
 TCP WINDOW (KB) 1:  −1

cd to directory with VEX file

 vex2model_updated.pl chsb6324.skd 

machines file

 first line is head node (hex0)
 next is data stream processes:
 hex0


 hex0   head node
 hex0   data stream 0
 hex0   data stream 1
  (if e.g. data were on hex4):
 hex4   data stream 0
 hex4   data stream 1
  next are processing (core) nodes 
 hex1
 hex2
 hex3
 hex4
 hex5
 (leaving hex0 to deal with IO (data streams) and head (manager)). We could specify hex0 as another core node and use the remaining 5 threads (1+4) but in this case there’s no need)

edit the threads file. Note number of spaces between text is important:

 NUMBER OF CORES:    5
 8
 8
 8
 8
 8

These would be reduced if running IO nodes on other machines. E.g. if data on hex4:

 8
 8
 8
 6
 8
 … and may have used hex0 too, but anyway….

edit run.sh

 -np 8 (where 8 is # processes, = wc -l machines)

VEX2config evlbi mode to use vsib_send and recv instead of specifying files by hand. Also, selecting three antennas only:

 vex2config.pl -evlbi -ant Pa -ant At -ant Mp vt12a.skd 

Example

1. on hex0:

    cd ~/correlations
    mkdir t08232 (or whatever the experiment name is)

2. copy machines, run.sh2] and [@.threads file from a previous experiment
3. copy .skd file across from hobart

    vex2config.pl -cross -evlbi t08232.skd

4. edit the .input file for a fringe search / Ceduna check. e.g First 20 mins is 1921–293:

 EXECUTE TIME: 1200
 INT TIME (SEC):     0.5
 FREQ (MHZ) 0:       6640.00

remove the semi-colon from end of READ FROM FILE If pols swapped at Ceduna, for CED:

 INPUT BAND 0 POL:   L
 INPUT BAND 1 POL:   R

1. vex2model_updated.pl t08232.skd
2. edit machines:

 hex0
 hex2
 hex5
 hex1
 hex2
 hex3
 hex4
 hex5

1. edit t08232.threads:

 NUMBER OF CORES:    5
 8
 7
 8
 8
 7

Test with Agilent and SMY on 3c273