Add support for HI and APC-ARP in rover mode

[gdt]

This is an enhancement request inspired by #269.

NMEA from the F9P gives positions of the APC. It's an interesting question what people really want for positions, and it's a system design problem to allocate responsibility among the F9P, the RTK Express/etc. ESP32 and something else that might be "data collector". (This is complicated by the current situation where no Free Software data collectors are usable with the RTK Express, but I'm considering that future world.)

The position of the bottom of the pole is below the APC by the APC-ARP height and the ARP->mark height, the second being HI. I'll call (in this issue) the sum HAPC, the height of the APC over the mark.

Options are

• the ESP32 subtracts HAPC so the NMEA stream is for the mark (could reuse the UI code and previous values for HI/APC)
• the data collector subtracts HAPC before logging
• user postprocesses some NMEA/GPX/geopackage to subtract HAPC

In all of these, despite NMEA trying to encode orthometric height and geoid offset, I think it's best to back that out to HAE promptly, mostly because the geoid model in the F9P is terrible. That's sort of separate, but I think it's the path we should design to.

Once going down this road, geoid models come up. A brief digression, even though I think that should be in a separate ticket:

• we don't know what datum the user is using. Once you do RTK, you aren't in WGS84, and often you are in a regional/national frame (e.g. NAD83(2011) epoch 2010.0 in the US).
• built-in geoid models are terrible. And once you aren't in WGS84, WGS84 orthometric height doesn't make sense, and you certainly can't use EGM2008 with NAD83 HAE. There you probably want NAVD88 instead of WGS84 Orthometric height, and that model done right can't possibly fit in the ESP32.
  All of this argues for recording RTK data in HAE and letting the user do orthometric conversion in GIS or maybe data collector.

[tonycanike]

Well-written request! Totally agreed that there are system design questions to answer about what people want for positions and which system components should perform transformations to give people the positions they want, how they want them.

And yes, geoid model questions quickly arise, perhaps along with the tilt sensor functionality (see issue #245 )

We probably need to answer the "what do people want" question first, and what is the vision and roadmap for the product.

Greg, what do you need? Do you want height above the WGS84 ellipsoid of the ground you are standing on? Do you want this for GIS collection, for land surveying, for mapping a hiking trail for Open Street Map, for something else?

The SparkFun RTK products look like commercial surveying gear, so are the majority of the users using it for applications that commercial surveying gear could be used for, such as GIS and Surveying applications? And they want it to interface with s/w like ESRI GIS data collectors, land survey data collectors, etc.?

The commercial data collectors seem to work quite pretty well getting WGS84 LLh (ellip height) of the APC via NMEA data, and converting it to what their users need, either on the fly or back in the office. APC->mark, WGS84->NAD83/SPC, HAE->NAVD88, etc. Things get complicated as the GNSS receivers get more features (tilt) and need complicated configurations (e.g. NTRIP connections, fixed rtk base coordinate entry, configuring RTK base-rover radios, etc.)

Maybe one answer is in GTD's comment on a free data collector being future work. Does this mean an open source data collector that has specific functionality to interface with the Sparkfun RTK open source software? Would be very useful, though it sounds like a big lift, as a user of a commercial data collector, I'm often astounded at the functionality in the expensive applications, both GIS and land surveying.

Maybe one answer is working with SW Maps developer to
(1) incorporate SparkFun ARP-APC offsets (I don't recall it doing that, but it already does HI), and
(2) providing the resulting ___location (of presumably the mark) to the OS ___location services for all apps on that phone to use.

Tony.

[gdt]

First, for me this is hobby activity. I am working in NAD83(2011) epoch 2010.0 because that is what is stable around me and because that is what MaCORS provides. I do a number of different things:

• trails, sidewalks, point features, etc. for openstreetmap
• above, but for maps (NAD83 native) in qgis
• getting good coordinates for property corners and town boundary monuments (IANA RPLS! But I was a town official for boundary checking under a 1600s law), which means 30s averages, with return visits on different days. I want vertical, just because nerds want to know, even though this is very much a horizontal control sort of situation.
• measuring benchmarks and checking GNSS-derived heights vs leveled heights (actually got 10 cm agreement with a mark leveled in the 1960s in NGVD29)
• not yet, topo measurements, meaning a bunch of spot elevations.

Interesting comment about commercial surveying: I see it differently, that it is serious-hobbyist-grade gear that is affordable. Just as a Trimble R20 is well outside the budget, so is an ESRI license and buying a windows computer just to run that. And, sparkfun products are sold in a context which pitches Open Source.

Certainly it's easy to get LLh from NMEA; it provides LLH and N, except that it is isn't really N, it's some very coarse approximation, and thus H isn't really H, but h-coarse(N). But if you back that out to h right there and thus use the same wrong N that the F9P used to make H from h, I am pretty sure you get back the right h.

SW Maps, while zero cost, is not Open Source.

I agree that sparkfun developing an open source data collector is too much and well beyond the scope and I didn't mean to say that.

What I mean is that there are a number of Open Source (and Free Software, almost the same set of programs, avoiding the philosophical distinction as unhelpful in this context) programs which can function as data collectors -- but which aren't currently usable with the sparkfun products. They include at least:
- Vespucci
- QFIeld
- Mergin (formerly input)
- SMASH (formerly geopapparazi)\

I have been able to use Vespucci with the Ardusimple NTRIP master, because that does wifi and over it does NTRIP itself and provides the NMEA stream on a socket. So one puts their phone in hotspot mode, and runs Vespucci that connects to the socket and gets NMEA and uses it. This works well, except it's a bit fiddly with DHCP and Vespucci has no datum support, or LLH,N->LLh support. So my height workflow is messy and/or manual. I contributed higher-precision support to Vespucci and will probably add LLh support somehow; the author has been accepting of this sort of thing.

I have used QField with the RTK Express, but it lacks NTRIP, and the RTK Express can't do NTRIP on WiFi and bluetooth at the same time.

[nseidle]

Good discussion, but forgive me, I don't see marching orders. What is it that you want added, and where?

I don't mean to be cross, it's just that I am a EE that has been thrust into the incredible GIS world and do not swim as easily as you both do.

[tonycanike]

Hey, EE - Cool! I went to school for EE! But spent my career in software and then application security.

I use a gazillion-dollar SurvPC-based data collector that handles horizontal and vertical data transformations, geiods, etc., so I'm able to contribute how the "big boys" do this. SurvPC reads the NMEA messages, factors out the geoid correction reported in the NMEA string to get the height above the WGS84 ellipsoid, and saves that (WGS84 LLh) in its .rw5 "raw" file for future analysis. (The .rw5 file is essentially much of what a surveyor would have written down on paper.) It then applies the geoid model and other transformations that I specify, and saves that result in the point coordinate file, all in the field on my tablet. But I spent way more on that software than I had hoped to and did it because I'm working with folks who use it daily. I'm not wishing that cost on anyone.

If collection s/w is recording the "H" in the NMEA string that was calculated using the crude ZED geoid, I think I understand some of Greg's request. It would be a huge bother to back out the ZED's crude geoid to apply a better geoid. I'd rather have the h above the WGS84 and apply my choice of geoid, HI, antenna offset later back in the office.

-- Can the ZED be configured (as an option) to not use its internal crude geoid and simply report the h, height above the WGS84 ellipsoid in the NMEA messages? Would that help Greg?

I use QGIS but not QField; I just now browsed the GField manual and didn't see any handling for instrument height (HI) or antenna offset, which surprises me. I would say that's a needed feature in QGIS.

QGIS has ample facilities for converting between horizontal and vertical coordinate datums, so I'm not surprised that QField leaves things in WGS84 LL...but the h vs. H-from-crude-ZED-geoid could be a significant issue.
-- Does QField just accept the H in the NMEA string or does it back out the crude ZED geoid and record h?

I wonder (and don't have time to check now) what SW Maps does with the h vs. H-from-crude-ZED-geoid situation?

[gdt]

QField does have HI in settings/positioning as "antenna height compensation". I'm not really sure what it does with the crude N. Vespucci records H in GPX, and H and N in points.

The situation with data collectors is complicated. I see the Surveyor users as being multiple personas:

1. professionals using sub-professional (but low cost and arguably good enough) equipment with expensive proprietary data collectors that have the features we are talking about already
2. serious amateurs (with professional-level understanding) attempting to get professional-grade results with Free Software data collectors (because they have an amateur budget, or because they object to proprietary software, or both)
3. normal amateurs doing far better positioning than phone/etc.

I'm in the persona 2 set. My real goal is to gather data in an accurate way such that there is a sane, reliable path to future processing. Compared to just "log output values", there are two issues:

1. The builtin geoid model is terrible, to the point that nobody should ever use it with RTK. It is around 4m off in Massachusetts.
2. The receiver reports an APC position (blend of L1/L2, complicated, details out of scope in this ticket), but the user is measuring the position of the bottom of the prism pole (or monopod). The difference is the APC-ARP height, a fixed characteristic of the antenna, and the ARP-mark height (HI), which is about the prism pole configuration (pin in which stop, which foot, known value at the time).

There is the further backdrop that the datum of the positions is almost never WGS84 (and that's an ensemble anyway and shouldn't be used): it's the datum of the reference station. So there's already a workflow issue of labeling the data from the collector in that datum. And, EGM96/EGM2008 (a crude EGM96 is in the F9P) are models from WGS84 h to WGS84 orthometric height. But if your RTK reference is NAD83, and you want NAVD88, you'd use GEOID18.

Also professional RTK will be using an antenna calibration file in the solution and thus will output the position of the ARP not the APC (and have lower residuals and better lock because it's applying elevation-dependent phase shifts). It's a deficiency of the F9P's builtin RTK that it doesn't do that, but that's not a fair complaint for $200 vs $10K.

This leads to two paths:

1. a data collector that is aware of this mess and expects to get standard NMEA (with H and a terrible value of N, from which it can calculate the original h), expects to label the coordinates with a configured datum, and either records in LLh or applies a user-requested high-quality geoid model, eg. to obtain NAD83+NAVD88. This data collector might also have HI/APC corrections (or expect a single value for the sum).
2. a data collector that isn't really aware, and the user coping later in some combination of gdal/proj/qgis.

For path 2, to make things sane/easy/lower-error-likelihood I'd like to see:

1. option to "report HAE in NMEA". If enabled, NMEA sentence is parsed, H and N used to calculate h, and then h put in H field and N field set to 0 (and checksum fix of course). This is technically non-conforming but is basically "report LLh in NMEA". But note that it is also nonconforming to report positions in a non-WGS84 datum in NMEA without setting datum tags, and NMEA has no real concept that I know of of using alternative orthometric height definitions and geoids. People who care just don't do that in NMEA; I think they recover LLh, relabel it, and the process in a sane post-NMEA environment.
2. option to set "HI". If non-zero, H field in NMEA is set to H-HI, checksum fixed. User has to input HI+APC; really it's the delta from pole tip to solution. So perhaps call it "Antenna height compensation" and describe it as "value to subtract from reported height to obtain height of pole tip". I'll use AHC below.

With both, I could say "GPX file from data collector is EPSG:6319 (which is NAD83(2011) epoch 2010.0 LLh)" and have it be the coordinates of the pole tip, with no geoid fuzz. If I want NAVD88 I just transform in qgis.

I wouldn't care if option 2 can only be enabled when option 1 is enabled, but it's basically (modulo a possible sign error)

 if (option1 || option2)
  unpackNMEA():
if (option1) { H += N; N = 0; }
if (option2) { H -= AHC; }
if (option1 || option2)
  repackNMEA()

I expect pro data collectors can be configured for either of these, and people with those can do it one way or the other. People with QField, Vespucci, or something homegrown, can get better data straight from the Surveyor.

[gdt]

I do not know of any way to configure the F9P to not use the internal geoid for NMEA.

[nseidle]

I'm starting to understand, thanks.

I want to avoid building a data collector into a receiver. We have to draw the line somewhere and I'm thinking it's 'don't modify the NMEA'. I'm all for adding HI and ARP offsets so the user can avoid messing up the math, but this sounds like it will lead to users who assume they are getting WGS84 based NMEA and their data collector is doing a 2nd modification to the data (and thus corrupting it).

gdt - of your two paths, I generally think the data collectors, or user post processing, are what should be relied upon. I doubt we can re-create SurvPC; we should instead allow people to use SurvPC. Or, nudge, you should start your own OS project to create a data collector :)

I'm building out #269. If that's useful, and you still really think we need to support NMEA modification in situ, then we can continue the conversation.

[edgecase14]

I wonder if the same place where these antenna height calculations take place, would make sense for inclinometer based pole slope compensation to take place? And the inclinometer would definitely have to be physically mounted in either the antenna or the pole.

[bendauphinee]

To glom onto this thread (and let me know if I should break out a new issue for a feature request), I have the Facet, and after taking my first official RTK compensated shots today, it got me to thinking about tilt.

What kind of work would it be to add a tilt sensor to the Facet, and update the firmware to factor tilt into reported NEMA measurements if a sensor is present? I did double shots at all of my control points, and the variance in my shots was as much as 33mm, just from not having a completely level pole (I need to get a bubble level at least).

[gdt]

Indeed, things that are off topic for a feature request really should have their own issue.