HPF Operator's manual for Astronomers at HET

HPF control system in a nutshell

All the subsystems of HPF are controlled via an asynchronous server-client network architecture. There exist different clients like Enviro (for environmental control), Calib (for calibration bench), Detect (for controlling detector), etc., in the HPF control system. One can talk to all the clients in HPF control system via commands in following format.

tims ClientToContact device:arguments  

For the ease of operation, we have a suite of bash scripts which does all the necessary communications. You will find all the scripts in ~/bin and ~/Scripts/InstConfig/CalScripts/ on hpf-server

How HPF detector is different from optical CCDs

HPF uses H2RG near-infrared array. We are operating it at a 10.45 second cycle (4 channel mode). While Detector is idle, it keeps resetting itself at every 10.45 seconds. This is important to keep in mind. Unlike other instruments, HPF is always exposing and there is no internal shutter. Hence, one should avoid bright lights on the Telescope's focal plane when Pfip shutter is open. None of the lamps on the FCU or in the dome are bright enough to cause a serious problem when detector is idle. But beware of bright sunlight before twilight, or the moon. If the detector comes close to saturation, it can lead to persistence which will last for many many hours.

When the detector client is asked to start an exposure, by default, it will first do two resets (20.1 sec), and will immediately start reading out the counts in each pixel from one corner of the channel to another corner. Unlike CCDs these readouts are non-destructive since it is only measuring the voltage of the photo diode pixel which is accumulating the charges. After finishing one sweep of reads across the detector (which takes 10.45 seconds) it will start all over again without resetting the array. Thus we obtain an increasing count each time we read the pixel. This is called the up-the-ramp readout. One traditional exposure is a full ramp in the case of HPF. For example, a 104.5 seconds exposure/ramp will have 10 (=104.5/10.45) non-destructive reads (NDRs). Each of these NDRs are saved as separate fits file image in the data directory (~/HPFData/LinuxGigE/YYYYMMDD/OBSNUM/ )

Once the detector finishes the requested number of NDRs, it will reset the array, and go back to the idle mode of constant resetting.

How to start up the terminals to control HPF

On the RA's desktop in control room, open two terminals, and keep them side by side. From here onward, we will be calling the left side terminal as the HPF Monitoring terminal, and left side terminal as the HPF Control terminal.

Starting up HPF Monitoring terminal

Login to HPF computer

$ ssh -X hpf@hpf

Open the ever running hpf screen session inside the hpf-server

$ screen -x hpf

You will see the outputs of the HPF's Detect client inside this window.

Starting up HPF Control terminal

Login to HPF computer

$ ssh -X hpf@hpf

You can type any of the commands explained in the rest of the document in this window to control the hpf computer.

TIP: Optionally, you can also open another third terminal and ssh into hpf-server. This terminal can then be used like a service road for the main highway. You can use this to send various interrupt commands to control hpf while the HPF control terminal is busy doing something. (Eg: sc, pausehpf, playhpf, pausecal, playcal, showhpfspec)

Taking an exposure with HPF

RA should decide based on observer's request whether they need simultaneous laser frequency comb (LFC) in the Cal fiber. Typically LFC in cal fiber is recommended for bright precision RV star science (Exposures < ~420 seconds or ~40 reads). They are not recommended for faint targets which need long exposures, nor for non-RV targets to reduce the scattered background light contamination.

While TO finishes setting up the target on the HPF's HR science fiber. If this exposure needs LFC, give the following commands in the HPF Control terminal to open LFC shutter

lfc_open

If the exposure do not need LFC, give the following command to close LFC shutter

lfc_close

Now, the exposure can be started by the following command.

hpfseconds "OBJECT NAME" EXPTIMEseconds NoOfExposures

where "OBJECT NAME" is the object name (The Quotes are necessary if there is space in the name)
EXPTIMEseconds is the exposure time in seconds
NoOfExposures is the number of multiple exposures to take.
Optionally, one can also set the Observation number manually by giving it as an optional 4th argument

If all the three mandatory inputs are not provided the command will simply print a Usage documentation of the command.

Inside the script, it will automatically convert the request EXPTIMEseconds to the nearest multiple of No of NDRs.

One can also directly give the Number of NDRS in a ramp instead of the EXPTIME in seconds using the following command.

hpframp "OBJECT NAME" NoOFNDRs NoOfExposures

where "OBJECT NAME" is the object name (The Quotes are necessary if there is space in the name)
NoOFNDRs is the number of Non-destructive readouts to take in a single ramp.
NoOfExposures is the number of multiple exposures to take.
Optionally, one can also set the Observation number manually by giving it as an optional 4th argument.

Once the exposure starts, you can see the activity in the HPF Monitoring terminal on the left.

You will also see estimated countdown to finish the ongoing exposure in the HPF Control terminal from where the exposure command was issued. (Eg: 680. .676. .672. ...)

TIP: If you see these countdown numbers going to large negative values ( say, < -30 ), then most likely the Detector client has crashed and no exposures are going on in the HPF Monitoring window. See the section below on how to restart Detect client.

Once a requested sequence of exposure is done, a beep sound will be played in the control room to alert RA.

How to check the quality of the data

As soon as the first two Frames are done in the HPF Monitoring window, one can start taking a look at the data quality from a third hpf terminal using the following command.

showhpfspec

It will open a quick look image of the data obtained so far in a ds9 window. And after a few seconds, it will also open a plot window which shows a quick extracted spectrum of Science, Sky and Cal fiber, with total accumulated counts marked on the left axis. Green dots at each order along with the values on the green right axis shows the Signal to Noise per extracted pixel achieved in each order of the Science fiber spectrum. The orange line on the top shows the hard saturation limit. Don't let the flux counts come near that line. (Recommended level is 50% to 80% below that line). Reduce the exposure time in the next exposure if the flux comes that high.

One can also open any of the older completed spectrum for comparison using the following command

showhpfspec YYYYMMDD OBSNUM

How to stop before the next exposure starts

One can stop a multiple exposure command before it starts a new exposure by the following command in a third hpf terminal

sc

If an exposure has already started, it will stop only after that exposure is finished.

Pause and Play of hpf

One can momentarily pause an exposure of any script before it starts the next exposure by the following commands

pausehpf

This will pause the script, at the beginning of the next exposure, until the following command is given to start

playhpf

These will be handy, in situations of one forgetting to open the LFC shutter, or needs to go inside the dome, or any other unexpected interruptions.

How to run Evening cals

Evening Cals are split into two parts. Part A and Part C. Part A requires the Dome to be dark.

The Day OPS RA can start the evening cals on the HPF Control terminal using the following commands at 5:30 pm local time.

$ cd ~/Scripts/InstConfig/CalScripts/

$ ./hpf_evening_cals_PartA.sh  # To run only Part A of the evening cals

Or (recommended) 

$ ./hpf_evening_cals_PartAC.sh  # To run both Part A and C of the evening cals

If only Part A was run at 5:30 pm, the night RA can start the Part C when the telescope starts stacking or even using the LRS or VIRUS cals. (However, not during twilight flats).

$ cd ~/Scripts/InstConfig/CalScripts/

$ ./hpf_evening_cals_PartC.sh  # To run the last Part C of the evening cals

How to run Morning cals

Morning Cals also have two parts internally. Part A requires the Dome to be dark and will finish before 9 am if night RA starts the script by 7 am. Internal Part B will continue for another 50 minutes, which only requires the Pfip shutter to be closed.

To start the morning cals, the night RA can give the following command

$ cd ~/Scripts/InstConfig/CalScripts/

$ ./hpf_morning_cals.sh  # To run both Part A and B of the morning cals

Pausing the cal script momentarily before next time it uses the FCU head.

Give the following command to pause the cal script just before it uses the FCU head for calibration.

pausecal  # wait for ongoing sequence to pause after issuing this command

This is handy, if one needs to switch on the Dome lights for some unexpected reason. After Domes lights are turned off, resume the cal script using the following command

playcal

TIP: At the completion of each parts of the cals, an email will be send out alerting the completion of that part of the cals.

Error states to watch for

The detector control software can experience several different error states, which will require the observer to restart the detector software. It is critically important to monitor the text output from the detector client in the 'hpf' screen session. Any 'ERROR', 'FAILED', or 'WARNING' notes should not be ignored. Once these start occurring, they will continue to occur frequently, so if the first instance is not noticed by the observer (and scrolls off the screen) additional instances will follow shortly.

The following error is very serious, but also rare. If it occurs, the detector will appear to continue operating. New observations can be started, new data directories will be created, and the exposure loop will complete. However, no data is being saved to disk. If this error occurs, the detector should be restarted at the end of the current ramp.

Resetting FIFO ...
**ERROR**: FifoReset status=65525, Addr:5200000d, ... Success
Waiting for Asic to be IDLE ... Confirmed
Resetting FIFO ...
**ERROR**: FifoReset status=65525, Addr:5200000d, ... Success

Additional error states to be added here as they occur...

How to restart Detect client when it stops working

Type the following command in the HPF Control terminal to power down the detector

$ tims Detect pyhxrg:PowerDown

You will see message in the HPF Monitoring terminal that Detector is Powered Down Then give the following command to shutdown the Detector client

$ tims Detect shutdown

Once the Detector client is shutdown, you will receive a terminal prompt back int he HPF monitoring terminal. (If not, Press Cntrl+C)

Once, you have a terminal prompt in the HPF monitoring terminal, wait for a few seconds and give the following command in the HPF monitoring terminal to start the Detect client

$ cd ~/HPFics/
$ python -m TIMS.clients.tims_detect

This will take a minute or two, and it will bring up the detector and you are ready to go.

Extra Notes

If one ever wants to close the screen session hpf which is running in the HPF Monitoring window, Press Cntrl+a d. (Never press Cntrl+d)