Ceduna Monitoring Checklist
Ceduna is run through a VNC server on cdvsi. The cdvsi VNC session should look something like this:
(As for the Hobart screenshot above, in this example the left hand windows are the field system and the right hand ones are the cdisko2 gui)
Check every 2–3 hours:
- Is recorder running? (ATNF recorder monitoring page, cdisko2 main window)
- Is BIGBUF near to 100%? (ATNF recorder monitoring page, cdisko2 recording status window)
- Are the bit sampler levels OK? (ATNF recorder monitoring page, cdisko2 recording status window). These should be around 17, 33, 33, 17 for VSOP modes and 33, 33, 17, 17 for BG3 modes.
- Are the numbers for missed PPS OK? (ATNF recorder monitoring page, cdisko2 recording status window). Cannot miss more than 5/100.
- Is the telescope on source? (type “onsource” in the operator input window of FS)
- Is the weather being logged? (type “wth” in the operator input window of FS)
- Is the clock offset being measured? (type “clkoff” in the operator input window of FS)
- Is the Tsys sensible? (scroll through the FS log)
As in the case of Hobart, Ceduna alarms are dealt with by an instance of the pmSTALM script running from newsmerd (see Hobart example screenshot above). For Ceduna errors the alarm sound will play from the newsmerd speakers only (i.e. the main Mt Pleasant station alarms will not sound.)
What to do if an alarm sounds:
- Commence panicking.
- Cease panicking.
- Turn off alarm (ctrl-C in the alarm window).
- Attempt to diagnose/solve the problem: Check error log in main pmSTALM terminal. Search for the error code on the wiki. If there’s nothing there ask the LBA chat. If they don’t know what to do (fairly likely) call the on-call person.
What to do if the Ceduna network falls over
Sometimes the Ceduna VNC window will freeze and pings will stop getting through to sille (a computer at Ceduna). Usually this will right itself after about a minute. If it does not, try these steps to revive the network.
If you lose the Ceduna VNC window, restart in a newsmerd terminal window with the following command:
The password can be found here.
What to do if the Mark5 crashes:
- Power cycle the mark5
- once it’s back up (can be pinged), ssh into mk5cd and run jive5ab