Firefly-2058: HIPS handling underlying FITS, with extract and readout

[robyww]

Firefly-2058: HIPS handling underlying FITS, with extract and readout

• Fits file extraction
• Fits readout
• Fits cube integration
• Wcs lock will support syncing cube planes
• Some refactor related to FitsReadUtil.java
• Some retactor related to HiPSListUtil.java
• refactor names of cube related function in PlotViewUtil.js and add support for hips in those functions
• Mouse readout direct fits access data no longer sent back and forth in the readout call
• A file download call can connect to one if it is active

Files with big changes

• HiPSRetrieve.java - much more advance to support fits retrieve. Major cleanup.
• FluxValueUtil.java - move all the flux calls into one file.
• URIFileRetriever.java
• ExtractHiPSTileTool.js - drawing layer for extracting HiPS
• HiPSTileExtractionDialog.jsx - dialog for extracting HiPS
• HiPSUtil.js - to setup and make the FITS extraction call
• MouseReadoutWatch.js - to handle fits flux readout with HiPS
• FitsCacher - improve to also cache small DirectFileAccess objects

Testing

https://firefly-2058-hips.irsakubedev.ipac.caltech.edu/firefly

Test HiPS, there have underlying FITS images as well

1. HiPS tab -> uncheck IRSA featured -> search for PanSTARRS -> choose PansSTARRS DR1 z
2. HiPS tab -> choose URL -> enter url below
   • https://irsadev.ipac.caltech.edu/data/hips/SPHEREx/spherex_allsky_cube_alpha1

Testing after the hips is loaded

• zoom in a little
• Test 1
  • move around on the HiPS, there should now the a Flux readout, it might save retrieving... at the beginning
• Test 2
  • From the image tools menu -> go to the extract rows -> choose
  • You should see a popup, click anywhere on the HiPS and choose Extract Tile
  • The underlying fits should be shown matched to the HiPS
• For the SPHEREx cube
  • the plane should change with the HiPS cube plane
  • the wavelength should show up on the plane control

[lrebull]

With the overall caveat that I have the same heebie-jeebies that Vandana has (who will use this? who will trust this? how will people find this? how will people know what (data sets, visualizations, readouts, files) to trust (or initiate this behavior with) and what not to trust?)...

• you click on the hips image, click extract tile, it attempts to do so, and then right away you get an error "Not a HiPS image: HiPS tile extraction only available for a HiPS display." I know why it does this -- you're in 'extraction' mode and now the currently selected image is fits not hips. But I'm, like, three clicks into this interaction and i've done exactly what the tool is telling me to do and now i have an error? every time? annoying. vaguely rude. There is another pop-up that the tool shows me that i can never see because the error is on top, and if I try to move the error, it goes away.
• I find this confusing (see screenshot): i've zoomed, i clicked on a tile, and hit extract. you see the error. why is there a black cutout? why is the blue tile not matching one of the presumably hips-shaped tiles? (heebie-jeebies: why is the tile a diamond polygon and the FITS a square?) MOST importantly -- why is this nowhere near where I clicked? The center of the blue polygon and the FITS image is approx 4h52m52.25s, +49d55m17.5s EQ_J2000. The place on the sky i was looking when i clicked was 18h18m48.20s, -13d48m25.9s EQ_J2000. that is superduper different. like "not even the same hemisphere" different.

• i zoomed out to see how far away i was from M16 (where'd been initially), and the tile i had clicked on briefly matched the magenta tiles. I zoomed away by using the target pulldown to get back to M16, and then used the 'recenter' button in the FITS pane. so that means i clicked away and then came back. now i have this (see screenshot). heebie-jeebies: how can i understand what part of the sky it is when the polygons keep changing? is it or is it not matched to the magenta tiles on the sky, distortions not withstanding? if it really is the square that the FITS file is, why is it having such a hard time with the blue polygon on the hips image? something isn't right somewhere, and this behavior makes me mistrust what it's doing with the coordinates. (I remember that we had problems with the PanSTARRS wcs, but i'm working on the assumption that we've run that to ground, because I think I remember doing that, but this series of screenshots has made me remember that whole thing.)

• the readout at the top of each extracted tile is telling me that there are large rotation angles and a flip applied. If I undo that by telling it unflip and north-up, as far as i can tell, it changes the FITS but not the hips which i think is weird. I'm confused about what exactly it's doing and this just further makes me suspicious of the fits files it's showing me.

• upload tab -> the hips URL tells me "could not recognize file type." is that expected?

• when i use the hips tab to upload the spherex cube, it's empty for me; i never get a readout ("retrieving value", no flux or wavelength shown at the bottom; honestly it took me a while to find the wavelength at the top, but i should have known better), and the image is just black.

further heebie-jeebies questions include but are not limited to: what will the readout give me in a 3-color hips image? where is it pulling the fits files from (truly original fits files or the pieces that were used to construct the hips image)? why are the point sources often but not always distorted in the pan starrs tiles? how will we convey to users what this is, when it's available, how and when to trust it?

(i'm still an advocate for getting this tool's functionality: https://firefly-2058-hips.irsakubedev.ipac.caltech.edu/firefly/onlinehelp/#id=visualization.selectregion
treated as another kind of 'extract' tool and placed here in the menu.)

[lrebull]

ok, updating now that i know how to see the data that appear in the SPHEREx hips image:

• i like that when you change the wavelength plane in one (hips or tile), everyone changes, this is good.
• if you extract 1 tile, everything is NOT wcs linked. if you extract 2 tiles, everything IS wcs linked. if you close that second tile, then everything NOT wcs linked again. i found this behavior odd. either it's all WCS linked or not.
• the artifacts (and satellite streaks) show me that i'm looking at the same data but the distortions (square vs. diamond) still throw me for a loop (heebie-jeebies)
• i still don't understand the "flip y" that is necessary for the FITS to match the HiPS.

copying from slack conversation so that i don't lose it - more heebie-jeebies from working with PanSTARRS z: Trey says that it works better if you pull tiles from a further zoom in rather than zoom out. this.. doesn't make any sense to me.

[robyww]

@lrebull

i still don't understand the "flip y" that is necessary for the FITS to match the HiPS.

The fits files are east right, so the image has to be fliped to WCS match

[lrebull]

But the HiPS aren't East right?

[robyww]

The hips are east left, so the FITS has to be flipped to WCS match them

[robyww]

@lrebull I made some tweeks to the extraction

• moving it to HiPS debug along with the hips grid.
• fixed the wcs issue when you delete a image
• Made more friendly wording for the extraction dialog when off a FITS image
• I can duplicate the first image not coming up WCS matched. It always does for me so I will need to understand what you are doing.

The new build is ready.

[lrebull]

pop-up window punctuation:

• "...valid WCS information." (add period at end)
• "...HEALPix curvature; they will always appear square." (semicolon instead of comma; period at end)

pop-up when you're back on the FITS is much better!

wcs is now linked for me too. i don't know exactly what was happening for me before, but it wasn't just me before. (we saw it on someone else's screen too!)

I'm not sure "hips debug" is as clear as i thought it would be in my head. i bet the others will Have Opinions on what it should be..

[aventura121]

Hi, overall looks good, with a few errors spotted.

[aventura121]

Trey, will this work on a non-equatorial survey (I'm assuming it's intended to)? Claude pointed me to this code because it doesn't do the conversion from the worldPt. I tried it on IRAC1 for M81's nucleus and it reads 0.058, which is too low; panning around also doesn't read out the values you'd expect (i.e. from lighter to darker, etc.).

[robyww]

I will look.

[robyww]

After evaluation. HiPS always has a projection and we can compute a WorldPt. A FITS image may not have projection information and we do support that in other context. When interacting with a HiPS we should always have a valid WorldPt or it is outside of the projection which is also meaningful.

I had not tested with IRAC1. After testing, it mostly seems that brighter source had greater values, though I did see some that did not make sense. Some of the old HiPS repositories seem to have some problems with their backing FITS. Overall I am fairly (sort of ?) confident that is is working.

[aventura121]

The bug is that on a non-equatorial HiPS, the helper function getHealpixPixelAtNorder returns an offset based on the wrong tile, because it never converts the world point coordinates when identifying the tile.

The general flow is that the worldPt comes into makeHiPSPlotAsyncReadout as EQ J2000.

On line 276 getHealpixCellAtNorder(norderForFitsReadout, worldPt, plot.dataCoordSys) converts the coordinates (it calls convertCelestial(wp, dataCoordSys)) before returning the cell from which we construct the tile url.

So far so good because the non-equatorial coordinates were converted and now we have the correct tile url.

But on line 279, getHealpixPixelAtNorder(norderForFitsReadout, worldPt) gets the tile and computes the offset in that cell, but this time it doesn't do the coordinate conversion. It just skips it. So in this helper the tile is wrong.

Fix would be to give getHealpixPixelAtNorder the same conversion the cell version uses, then pass plot.dataCoordSys at the call site:

// HiPSUtil.js
export function getHealpixPixelAtNorder(tileNorder, wp, dataCoordSys) {
    if (tileNorder>MAX_SUPPORTED_HIPS_LEVEL-9)
    const dataWp = convertCelestial(wp, dataCoordSys);
    const polar = radecToPolar(dataWp.x, dataWp.y);
    ...
}

// MouseReadoutWatch.js:279
const {tileImagePt}= getHealpixPixelAtNorder(norderForFitsReadout, worldPt, plot.dataCoordSys);

[robyww]

ok, thanks, I will fix it

[robyww]

After studying the problem I realized I was mis-understanding what you were saying. For whatever reason I misread non-equatorial to mean "hips without a projection". After re-reading you were saying non-equatorial! In the HiPS case you were saying "it is not working when the data coordinate system is galactic".

Yes, you are correct. The point has to converted to the data coordinate system, in case J2000 to galactic, before it can investigate the tile pixel offsets. Most HiPS are in J2000 (an equatorial projection), however the IRAC ones are in galactic.

Actually this has happened before, I need to remember to test against galactic based HiPS.

Thanks for pursuing this issue.

[robyww]

@aventura121 - made a new build with the response to feedback

[robyww]

@aventura121 - another new build.