In the late afternoon shifters noticed the semi-expert drive GUI was not updating, several restarts did not help. Checking into the issue with the drive team connecting from remote it was found that the MOS server is not running and cannot be restarted. This issue could not be solved and will be revisited tomorrow, Jan 17, during the day.

Actions applied for solving this issue: (T. Le Flour)

The status is the following

• The MOS OPCUA Server cannot be relauched since the port used by server was not properly released after the last stop.
• We decided to reboot the VM running on TCS02 but after this operation , the concerned port was still in use.
• By using the commend lsof , we relased the port and relaunched successfully the server.
• Actions : releasing the port has been included in the script used to launch the server to prevent such an issue.

When parking in yesterday, the telescope drove to the nominal position (approx. 95° zenith angle and -1° azimuth), but then it took a long time and never showed the "park out" button again.

In the driver container, we saw two alerts to acknowledge, then the GUI indicated "telescope parked". We confirmed visually that the position was correct and we could lock azimuth and elevation. Informed experts on WhatsApp and got confirmation that everything was done right.

This morning we were asked by Ino to rearm fuse Q21 (24V), which was the reason for the wrong statement from the UI.

We tracked stars at various zenith angles and changed the Az and Zd offsets, taking images with the PSF camera for each position. After Armand's changes during the day the offset is the same for all Zenith angles.

Star Zd Az
dAz, dZd image

Polaris 61.5 0.7
0, 0 1568923765.fits
0.5, 0 1568923985.fits
0, 0.5 1568924088.fits
0.5, -0.5 1568924324.fits

Eltanin 29.2 327.9
0, 0 1568924558.fits
0.5, 0 1568924683.fits
0, 0.5 1568924747.fits

Benetnasch 66.6 314.7
Vega 15.6 313.8
0, 0 1568925023.fits
0.5, 0 1568925090.fits
0, 0.5

Kochab 57.3 342.0
Sulafat 8.7 299.5
0, 0 1568925265.fits
0.5, 0 1568925420.fits
0, 0.5 1568925485.fits

Alphecca 52.6 283.0
Arcturus 72.2 282.5
Oph Alpha-55 32.2 246.0
Albireo 1.2 229.1
0, 0 1568925734.fits
0.5, 0 1568925842.fits
0, 0.5 1568925917.fits

Nunki 55.8 190.6
Altair 20.2 168.3
Enif 35.7 115.2
Alpheratz 59.0 71.4
Caph 54.8 35.6
Deneb 21.2 33.9
Polaris 61.5 0.7

1. Many parkin and parkout operations with different torque  (with the clamping system of the access tower (from 18% to 8% = 26N/m). With 8%, the locking position cannot be reached
2. Some minors have been corrected

• Different ParkOut and Parkin done with different values for testing the torque limitation. 
  • By decreasing the torque and the position , the parkin failed because we are not in the CSS locking sensor (in locking position)
  • By setting back the original values, th parkin is ok but the bolt cannot be properly inserted.

After tracking Mk421, we abort current action (acknowledged) and try to go to a new source: RA=308.1, Dec=40.96. Drive Control refuses these parameters ("check your parameters").

General Abort without effect for 2 minutes. Called expert (Ino). He will look into it.

Ino explained: MOS was totally desynchronised with PLC. (It was assuming that the telescope was parked.) He parked the telescope manually. Thanks!
Experts will investigate on Monday. But the shift crews can continue to use the  Drive Control and tracking.

Tonight, we tracked Vega to test (a correction of) the pointing offset of the Drive.

Around 22:30UTC, we tried different Az offsets of the pointing in the Semi-expert GUI, followed by taking an image with the PSF camera to see the effect of the offset choice.

We tried delta_az = 0.3°, 0.1°, 0.26°, 0.6°. Strangely, we saw some unexpected (much too large) shifting of the spot for delta_az = 0.3° and delta_az = 0.6° (but looked reasonable for delta_az = 0.26°), which appeared quite puzzling to us (see specific Camera Elog here).

After having tracked with delta_az = 0.6°, we stopped again, and selected delta_az = 0.4°. We submitted the command ("Start"). Some seconds after submission, we became unsure whether we used the right field in the GUI where we entered the delta_az values. We hit "abort current action" (to check our previous command), when probably the slewing to the new offset position wasn't finished yet (about 5 seconds after the previous "Start" command). Then, the GUI became stuck displaying "abort current action", and not displaying the field to enter new positions/targets. According to the GUI, the telescope had in fact stopped tracking, but without any possibility to receive further commands.

We restarted the GUI, but the state remained the same, and checked the telescope outside. Then we called the experts, who could solve the problems after some non-trivial intervention.

• Test at 60% maximal speed from -90 to 0
  •  Check of the FW discharge : No error (FW 2 : 95% FW1 : 100%)
• Test at 60% maximal speed from -90 to 270
  • Check of the FW discharge ==> Error (FW 2 : 90 % FW1 : 100%)
  • Drive stops at 250 degree.
  • Power consumption = 60kw
•  

Thierry&Ino on site

Ievgen, Vitalli and Jean-Luc remotely.

TCU tests to perform the parkin-in and park-out procedure from the Drive.

Results :

• Actions can be triggered from the TCU and correctly executed thru the ACS Drive Bridge-MOS-Drive PLC.
• Monitoring information is not correctly pushed from the bridge to the TCU (positions , ...)
  • Investigation brings many options. For the time being, a simple change in the ACS xml description will fix the problem. Anyway, this solution is not satifactory and a new solution will be implement asap.

Plans : Monday September 16th (2pm localtime) : new TCU tests to validate the intermediate solution for park-in, park-out and probably for some others action like Tracking,  GotoCircularPosition, ...

Tests and debugging of the tracking procedure. Many components are involved : PLC MOS GUIs.

Each component needs to be well debbuged to achieve a safe movement of the telescope

==> IN PROGRESS

We sent a tracking with with RA211,08 and DEC29,52

1st time the goto position of azimuth was sent at +42° and the trajectory requierts  -52° !!! (speed limit of the tracking 0,001 rad/s)

After a cancel the azimuth was sent at -54° !!!

Yesterday we had the same problem with the same RAD/DEC coordinates. -31° and after a cancel +270° !!!

Is it a problem of trajectory with the 1st point positionning or with this coordinates does we need a higher speed in tracking ?

Start of the tracking debug ==> PLC and MOS 

Debugging of the tracking sequence :

• PLC sequence : trajectory tracking data correctly managed
• PLC tuning : regulation OK ==> need to refine.
• Tracking algorithm :
  • Changes in Drive.ccp to allow azimuth negative values.
  • Tracking duration : even by changing the duration of he tracking (from 1'to 5') ==> not correctly managed (still 1' tracking) ==> to be checked
•  

• Tracking procedure has been debugged from the GUI-MOS (OPCUA Server with tracking plugin)-PLC
• Still some improvments to apply but globally the procedure is running well.

• Regulation of the motors (PID) for slow and fast tracking and motor oscillations.
• Implementation and Debugging of the Semi-Expert control software for the drive.
  • GUIs + MOS plugin (OPCUA Server)
•  Results are quite encouraging in good wheather condiftions.
  • additionnal tests in different (wind) conditions are envisaged to validate completly the process

• Tracking at position :
  • RA = 101.288541 RA Offset = -0.6
  • DEC = -16.71343 DEC Offset = 0.7

• Debugging of the MOS plugin corresponding to the tracking 

- During shutter tests, we were requested to move the telescope many times between two positions at different zenith. Sometimes, once the position is reached and in the log I got the message 'GoToTelescopePosition Done received', the command panel stay stack with the "action in progress" bar without possibility to perform other atctions. I need to click to 'general abort' to stop the action and  get the panel comand back. It happened 3 times

- During tracking, the message on the screen switch from "tracking in progress" to "action in progress, please wait..." The latter is quite misleading for non-familiar user of the gui.