Recent Changes - Search:
Schedules /

LBA Cd

Pre-setup things to do or check before setting up for LBA observations:

  1. Make sure you are part of the LBA chat on Mattermost. Any sudden change in scheduling for experiment or any other information will be on Mattermost. You can access that by pointing a browser at https://chat.atnf.csiro.au/. If you don't have a personal account you can login as hbobserver@gmail.com (usual observer password). Keep an eye on messages here. The other LBA observers at different stations are a good first port of call for problems at Hobart/Ceduna. Chris Phillips is the main LBA coordinator at the ATNF. Jamie Stevens also knows a lot about the systems here. There are also Mattermost apps for smartphones etc if you want to use them.
  2. Check the information email for the session. If in any doubt you can cross-check against the information for the experiment at ATNF LBA wiki

Preparing schedule files

Instructions on preparing LBA schedule files for use at Ceduna 30m are discussed here:

  1. First of all, Access Hb vnc by vncviewer pcfscd:0 from ops8. You can also use 'vnc://pcfscd.phys.utas.edu.au:5900' from an inbuilt mac vnc. you'll need UTAS VPN to connect to the vncs if your are observing outside of UTAS network.
  2. Now find out the experiment name and code which you are going to setup for (eg. v255ah). Find out experiments start time, end time in UTC format. This info can be found either on UTAS observing wiki page ra-wiki page, or offical LBA website page ATNF LBA wiki
  3. Your next job now is to make sure that proc file and snp file the particular experiment are already there or not. From oper@pcfscd, go to /usr2/proc and /usr2/sched folders to check whether these files are there or not. e.g for exp v255ah, i'll go (from oper@pcfscd):
    cd /usr2/proc/
    ls v255ahcd.prc
    cd /usr2/sched/
    ls v255ahcd.snp

    Most probably the files will be drudged and checked before the session/exp starts by the main UTAS LBA support person. so, if these files are there in their relevant folders, then it's ok. If they are not there, then you need to download and drudge the exp to make these files.

How to drudge

NOTE: There is a chance that we need to edit proc and snap files for Ceduna, but thats the job for the Main person in-charge for the session. Most probably the main UTAS LBA support person will make these proc files already before the start of the session and you will find them in proc folder. But in case if they are not there, contact the on-call person.

everything is done from observer@newsmerd. experiment vex file will be in the home area.

for ceduna, from observer@newsmerd, run: ~/slogit.lbacd.v5.sh v255ah.vex

now remember few things:

  1. you only need to downlaod vex file once
  2. you only need to overwrite vex file for once (do it for first time, i mean 1st telescope)
  3. you need to make proc and snp files every time, so enter 'yes'
  4. 'yes' for stow command

If this drudge method fails, you can also try it with 2nd method, which is run drudge for each antenna. example

copy vex file for particular experiment from observer@newsmerd to oper@pcfscd:/usr2/sched/

scp v255ah.vex oper@pcfscd:/us2/sched/v255ah.vex

Now, from oper@pcfscd, run:

drudg /usr2/sched/v255ah.vex
cd (choose antenna)
11 (show/set equipment type)
19 16 1 1 (inputs for equipment type)
3 (make a .snp file)
5 (print a summary file)
12 (make a proc file) (again remember, it is not working for ke and hb for the moment, so skip making proc file for the moment being)
0 (done with drudge)

Now move the proc file to /usr2/proc/ directory
mv /usr2/sched/v255ahcd.prc /usr2/proc/v255ahcd.prc

After this is done, make sure you have proc and snp file in their relevant directories on pcfscd (eg, v255ahcd.prc in /usr2/proc/ and v255ahcd.snp & v255ahcd.sum files in /usr2/sched/)

Pre-checkups before setting up for the experiment

  1. Ceduna field system is fs-flexbuf. If you are setting up for a new experiment from the start, close the field system which is running by typing 'terminate' in Oprin window, and start the right field system by typing fs-flexbuf in field system window (or any window on oper@pcfscd). But if there is already an experiment going on and you are doing a change-over setup, then you don't need to kill the fs (because most prob right field system is running)
  2. Now you should check whether jive is running or not: from oper@pcfscd, do:
    ssh observer@flexbuffcd

    Now, check jive by doing
    ps -ef | grep -i jive

    if it returns jive5ab-3.1.0-64bit-Release, then it is running and good. but if jive is not running, then from observer@flexbuffcd, run:

    StartJ5

    Which automatically runs jive5ab-3.1.0-64bit-Release and logs the debug information into a log file under the /home/observer/jive5ab.logs/. That is convenient if we need to review what went wrong on a session. if that doesn't work, you can also start the jive5ab as:
    jive5ab-3.1.0-64bit-Release
  3. Make sure that dbbc is running by doing vncviewer dbbccd from oper@pcfscd. Check the right version of dbbc is open (DBBC2 Control DDC v106_2.exe). If not, close the one that is open and open the right dbbc version. it should be somewhere in the middle of desktop. If you want to reconfigure the dbbc or stated it again, then you'll need to synchronise the dbbc clocks.

    go to the Oprin window, type dbbc=pps_delay , if the field system returns 61959, then its fine, otherwise in Oprin, type dbbc=pps_sync until it returns that number.
  4. make sure that Serial.Server.sem.py is running on dbbccd. if not, start it. it will be somewhere on desktop.

Loading the procedure files:

Now we want to load in the procedure files. Do following steps:

proc=v255ahcd
sched_initi
setup01
bread
iread
lo

  1. from lo output on fs(field system): check the lo is correctly set.\\ format is /lo/loa,<lofreq>,<sideband>,rcp,off and /lo/lob,<lofreq>,<sideband>,lcp,off. lofreq & sideband freq should be the same for both cases

    example : /lo/loa,7300,lsb,rcp,off. and /lo/lob,7300,lsb,lcp,off
  2. from bread output on fs(field system): check that bbcs values that you see in fs are same as what you have in proc file.
  3. from iread output on fs(field system): compare the desired power levels and the current power levels. format is /ifa/1,agc,1,<desired>,<atten>,<current> and /ifb/1,agc,1,<desired>,<atten>,<current>

    example : /ifa/1,agc,1,48000, 23, 48066 and /ifb/1,agc,1,48000, 18, 47066

Now, there is a chance that you need to calculate the mode for the experiment. Generally the Main UTAS Support person will do these calculations early on and edit the proc file before the start of the experiment or session. So the proc file will have all these edits when you are setting up for the experiment. If the proc file for the experiment is not there, call the on-call person. But in case you need the math, here it is:

To figure out mode: check the proc file : (we are looking for bitmask, which looks something like 0xffffffff)

from oper@pcfscd, run grep mode= /usr2/proc/<exp>cd.prc or less /usr2/proc/<exp>cd.prc

Mode will be mark5b-<rate>-<numchans>-2 where the <numchans> going to be 2*number of bbc's' if there are f's in the bitmask, or 'number of bbc's' if there are only 3's and/or c's.

example1: if bitmask is 0xffffffff, then it's 1024-16-2 - 1Gbps (each entry after 'x' represent 1 BBC, 'f' represent both sides of BBC band, and each bbc has has 16Mz bandwidth for ceduna)(f stands for both side bands, 3 for upper side band, and c for lower side band)

example2: if bitmask is 0x0f0f0000, then it's 256-4-2 - 256Mbps

you can check by giving following inputs in Oprin window:

scan_name=test,v255ah,cd
disk_record=on
mk5=evlbi?
mk5=evlbi?
mk5=evlbi?
mk5=evlbi?
disk_record=off

check that output number in fs keeps increasing each time when you run mk5=evlbi? and if we see that output number in fs keeps increasing each time, that means that data is being recorded.

also, from observer@flexbuffcd, do:
vbs_ls -lrth 'v255ah*' | tail -10
this will show your recorded scan. This check will confirm that data is being recorded at right place.

If the number in mk5=evlbi? does not increase, this is most likely because the ports are not correctly set by the configure script.
\\ To solve that, Firstly, type mk5=net_port? in Oprin window, which should return as 46277 in fs

now check whether configure the fila10g has worked or not: run Fila10g background and check the bitmask:
nc dbbcho 6789
sysstat
check that it gives right bitmask

^C to exit

check mode with mk5=mode? in Oprin window. if it is any how not looks right, set it up with mk5=mode=mark5b-256-4-2 in Oprin window

also check:

dbbc=pps_delay mk5=mode?
it everything looks right, proceed further:

now, do the test recording and to generate the autocorrs graphs. from oper@pcfscd, run:

~/autocorr_test.flexbuffcd.sh mark5b-256-4-2

If the autocorr graphs looks bad, try to reconfigure the fila10g and/or dbbccd. If you are reconfiguring the dbbc, remember to check dbbc=pps_delay and then to reconfigure the fila10g again afterwards. normally, it runs without any errors (it generally should).

Now setup is basically done. Usually for LBA, there are two schedules per experiment, a fringe check denoted with vc or vt and a main experiment, always v. We do our setups and fringe checks during the fringe check session (first one). After the fringe check is over or your telescope has fringes, you can put it on the main schedule with:

schedule=v594ahcd,#1
onsource
list

OR if late start, do:
schedule=v255ahhb,2021.210.03:00:00 (any time particular time from where you want to start the schedule, hint: time right now)

NOTE: DO NOT CHANGE THE PROC FILE OR MESS WITH ANY OF THE FREQUENCY STUFF AFTER YOU HAVE FRINGES!! This would negate the fringes since now you are uncertain the setup is correct. You can ONLY get fringes if the setup is 100% correct.

Monitoring:

  1. Make sure that antenna is tracking and slewing to next source as well, and not struck. this can be done by doing onsource command in Oprin window.
  2. Make sure that data is being recorded by running watch -n60 "vbs_ls -lrth 'v255ah*' | tail -30" on observer@flexbuffcd to check whether all scans are being recorded or not. also, run mk5=evlbi? multiple times in oprin window which scan recording is going on to check whether data is actually being recorded or not (the number in fs should increase every time)
  3. Check the delays and samplers are good. if not, you need to halt the schedule and reconfig the dbbc3.
  4. Monitor for wind stows, should be obvious if you are monitoring the antenna moving. make sure that wind is ok from antenna_monitor window.
  5. Eyes out for ERRORS in the field system log. There are a few benign ones that will regularly come up.
  6. Check how much space is left in the disk to record by doing mk5=rtime? , to check how much free space you have. if free space is less than 4-5%, let your on-call person know about it.
  7. Record experiment details to the appropriate ATNF wiki page. (Go here, click the session and then experiment number link, scroll down to "observing comments for each antenna" section and click Hb to update the Hobart 12 log and Cd to update the Ceduna log.) The login username and password can be found here.
  8. These web pages should always be open:
    Grafana monitoring Page
    ATNF VLBI monitoring page
    ATNF recorder monitoring page
    Local Mt Pleasant live page
    Local Ceduna live page

Also, it is good to know how to halt and continue the setup during the experiment, in case if you need to do it during your observations. To halt the setup, do following from Oprin window:

halt
disk_record=off
and then to continue the setup, do following from oprin window:
setup01
cont
list

few points for observers:

  1. Check every 30 mins during day shift and every 1 hour during night shifts
  2. Always try to use vnc for monitoring
  3. Also can use Grafana to do checks as well but vncs are more relaible
  4. Make sure that you atleast check once every 30 mins that everything is running smooth or not
  5. Always have LBA mattermost chat open when you are on observing duty so other people can interact with you

Trouble-shooting some common errors and probelms:

stuck antenna:

If antenna is stuck, and you see following errors in fs window:
2012.297.03:58:54.01?ERROR st -998 reading SystemClock1
2012.297.03:58:54.01?ERROR st -999 TCP/IP connection was closed by remote peer
2012.297.03:58:54.01?ERROR st -5 Error return from antenna, see Mbus error

OR

2012.297.03:58:54.01?ERROR st -998 antenna stuck

then you can do following things:
first, in Oprin window, type:

antenna=open
antenna=operate
antenna=status

If this doesn't solve the issue, then:

  1. Halt the schedule using step described earlier on this page
  2. Go to timehb vnc by doing vncviewer timehb

    then, over there:

    Go to the antenna control window. Look for red buttons there. turn off the antenna manually by doing from operate to standby. give it sometime to catch up after changing. then switch Drives off.
    wait some time, then switch Drives back to on. After turning the drives on, it is recommended to wait a little while before putting the antenna in operate

    This step is crucial if the previous steps haven't resolved the issue

    This will generally bring back the antenna form stuck. but if this doesn't solve it, do the RESETS at the bottom. Basically at the end, we want green buttons at operate, drives on in antenna control window.

    Note: sometimes, when antenna goes to windstow, it does not come back automatically after wind becomes fine. for that, you need to halt the schedule, reboot the timehb machine using the ipswitch website, then login again to timehb machine and open all the programs that were open before and make sure that everything is all right. Then terminate the field system, start it again and do a quick re-setup. this will solve the issue.

For change over setups:

first: source=stow

then check whether we have to change receivers or not? if yes, then you need to call 'on-site' person to change the receiver. then, check the right receive is in place by checking the Ceduna30m live page, Local Ceduna live page

then, do the following procedures:

if there was no receiver change:

proc=v255ahcd
sched_initi
setup01
bread
iread
lo
mk5=mode?

scan_name=test,v255ah,cd
disk_record=on
mk5=evlbi?
mk5=evlbi?
mk5=evlbi?
mk5=evlbi?
disk_record=off
schedule=v594ahcd,#1 onsource list

if there was a receiver change: Then, you need to do everything again from start.