[chrony-users] leapsectz and phc refclock offset

[Chris Perl]

I have chrony servers that are using phc refclocks. Those refclocks are
kept in TAI, so I have to specify the current TAI-UTC offset in the
`refclock' directive ("offset -37").

I cross check that the current value I have set there seems sane via some
monitoring that uses the timezone files to compute the difference between
posix/UTC and right/UTC. So, at least when it changes, I'll find out about
it (I also make sure my timezone files are up to date). The downside is
obviously that I have to go update a bunch of config files and restart
chronyd when the next leap second occurs.

I was hoping that I'd be able to use the `leapsectz' directive to ask
chrony to automatically calculate the TAI-UTC offset for my refclock, but
after some testing, I'm not sure that is possible (or, I'm just doing
something wrong).

Is the idea of the leapsectz directive just that your refclock gives you
whatever time it gives you (perhaps TAI, perhaps UTC) and the timezone
files are just used so chrony can, for example, arm the kernel (if
leapsecmode is system) and advertise the impending leap second to
downstream clients?

Assuming I'm not misunderstanding, would it be possible to add something to
allow the TAI-UTC offset to be dynamically calculated using the timezone
files for refclocks?

[Bill Unruh]

chrony assumes that the time delivered by your server clocks is correct. It
makes no assumptions about what sort of time is delivered. Now, you could
modify chrony to have it deliver utc rather than TAI. You could alster your
server to deliver UTC or you couuld have your timezonefiles do the conversion.
Or you could use the /usr/share/zoneinfo/right files to convert to your
favourite time assuming you system clock is in TAI rather than UTC. You could
have to make sure that you kept your leapsecond file up to date since it uses
that to figure out how may seconds it should alter

Mind you the "right" files only count leap seconds since 1970 (UT1), and do not
include the 10 seconds that was the TAI difference between U and TAI in 1972.
So you would have to edit the tzdata so that the "right" files included that
10 sec if you really want TAI( eg add 10 to each of the times. That way your system clock could be in TAI and
the date funtions still returns the correct time for your timezone.



William G. Unruh __| Canadian Institute for|____ Tel: [phone]
Physics&Astronomy _|___ Advanced Research _|____ Fax: [phone]
UBC, Vancouver,BC _|_ Program in Cosmology |____ ***@physics.ubc.ca
Canada V6T 1Z1 ____|____ and Gravity ______|_ www.theory.physics.ubc.ca/

I have chrony servers that are using phc refclocks.  Those refclocks are kept in TAI, so I have to specify the current TAI-UTC
offset in the `refclock' directive ("offset -37").
I cross check that the current value I have set there seems sane via some monitoring that uses the timezone files to compute the
difference between posix/UTC and right/UTC.  So, at least when it changes, I'll find out about it (I also make sure my timezone
files are up to date).  The downside is obviously that I have to go update a bunch of config files and restart chronyd when the
next leap second occurs.
I was hoping that I'd be able to use the `leapsectz' directive to ask chrony to automatically calculate the TAI-UTC offset for
my refclock, but after some testing, I'm not sure that is possible (or, I'm just doing something wrong).
Is the idea of the leapsectz directive just that your refclock gives you whatever time it gives you (perhaps TAI, perhaps UTC)
and the timezone files are just used so chrony can, for example, arm the kernel (if leapsecmode is system) and advertise the
impending leap second to downstream clients?
Assuming I'm not misunderstanding, would it be possible to add something to allow the TAI-UTC offset to be dynamically
calculated using the timezone files for refclocks?

[Miroslav Lichvar]

Post by Chris Perl
I have chrony servers that are using phc refclocks. Those refclocks are
kept in TAI, so I have to specify the current TAI-UTC offset in the
`refclock' directive ("offset -37").

Is the PHC synchronized by ptp4l? You could use phc2sys -E ntpshm to
get the reference time in UTC in a SHM segment and use that in chrony
with the SHM refclock instead.

An advantage of this approach is that chronyd will stop accumulating
new samples when the PHC is no longer synchronized as phc2sys is
continuously checking state of the PTP port.

Another possibility is to configure the PHC refclock with the pps
option, so it would use only the sub-second offset. This assumes an
additional source (NTP server or non-PPS refclock) is available.

Post by Chris Perl
Is the idea of the leapsectz directive just that your refclock gives you
whatever time it gives you (perhaps TAI, perhaps UTC) and the timezone
files are just used so chrony can, for example, arm the kernel (if
leapsecmode is system) and advertise the impending leap second to
downstream clients?

Yes, currently leapsectz is just an additional source of leap seconds
and it sets the TAI-UTC offset of the system clock, so applications
using clock_gettime(CLOCK_TAI) get correct time.

Post by Chris Perl
Assuming I'm not misunderstanding, would it be possible to add something to
allow the TAI-UTC offset to be dynamically calculated using the timezone
files for refclocks?

I think a new "tai" option could be added for refclocks. If specified,
the refclock code would call something like REF_GetTaiOffset() and add
(or substract?) the value to offset in all samples.

[Chris Perl]

Post by Miroslav Lichvar
Is the PHC synchronized by ptp4l? You could use phc2sys -E ntpshm to
get the reference time in UTC in a SHM segment and use that in chrony
with the SHM refclock instead.
An advantage of this approach is that chronyd will stop accumulating
new samples when the PHC is no longer synchronized as phc2sys is
continuously checking state of the PTP port.

Yes, the PHC is synchronized by ptp4l.

I had originally set things up using phc2sys (using timemaster,
similar to your blog post at [1]), but removed it from the equation
when I saw some weird behavior.

When I was using phc2sys, I would observe that when left running for a
long period of time (not very long, like 24h would be sufficient),
chrony's reported offsets, and root dispersion would go crazy for a
while and then come back into line (they would go crazy for a few
hours or so).

I happened to observe one of these episodes in real time and restarted
phc2sys, which immediately caused the issues to stop. I figured
perhaps it was some kind of feedback loop between phc2sys and chrony.

Its possible I had phc2sys misconfigured, but, I was using timemaster,
so I'm not sure how I would have managed that. When I realized that I
could point chrony at the PHC directly, I decided to just remove
phc2sys from the equation and stopped looking as that seemed to give
me stable results.

I didn't realize that phc2sys was checking the state of the port, that
is an interesting thing to consider.

We also have monitoring that is checking various aspects of ptp4l
(including the state of the port), so our monitoring would report any
issues with ptp4l as well, but the idea that chrony wouldn't
accumulate the samples is interesting.

Post by Miroslav Lichvar
Another possibility is to configure the PHC refclock with the pps
option, so it would use only the sub-second offset. This assumes an
additional source (NTP server or non-PPS refclock) is available.

I had been looking at this as well. I'll have to think about this a
bit harder and test it out.

Post by Miroslav Lichvar
I think a new "tai" option could be added for refclocks. If specified,
the refclock code would call something like REF_GetTaiOffset() and add
(or substract?) the value to offset in all samples.

In my particular case (assuming I don't ultimately wind up using one
of the alternatives you proposed), that would be very helpful.

What would you imagine the time frame to get something like that going
would be? I'm not trying to ask you to do it now, just curious when
it might land or if attempting to provide patches for such a thing
would be worthwhile.

[1] http://rhelblog.redhat.com/2016/07/20/combining-ptp-with-ntp-to-get-the-best-of-both-worlds/

[Miroslav Lichvar]

Post by Chris Perl
I happened to observe one of these episodes in real time and restarted
phc2sys, which immediately caused the issues to stop. I figured
perhaps it was some kind of feedback loop between phc2sys and chrony.

You are the first one reporting this problem. If you have collected
the phc2sys output and chrony's refclock+tracking logs from that time,
I'd like to see that.

Post by Chris Perl

Post by Miroslav Lichvar
I think a new "tai" option could be added for refclocks. If specified,
the refclock code would call something like REF_GetTaiOffset() and add
(or substract?) the value to offset in all samples.

In my particular case (assuming I don't ultimately wind up using one
of the alternatives you proposed), that would be very helpful.
What would you imagine the time frame to get something like that going
would be? I'm not trying to ask you to do it now, just curious when
it might land or if attempting to provide patches for such a thing
would be worthwhile.

Unless I'm missing some important detail, it should be straightforward.
I'm working on other things now, but I think I can get to it in the
next few weeks. If you can prepare a patch for that, including
documentation, I'll take it :).

[Chris Perl]

Post by Miroslav Lichvar
You are the first one reporting this problem. If you have collected
the phc2sys output and chrony's refclock+tracking logs from that time,
I'd like to see that.

I don't have the output or logs handy, but I do have notes about how I
had the system configured, so I'll see if I can still reproduce the
issue and if I can, I'll get you the relevant info.

[Chris Perl]

Post by Miroslav Lichvar
An advantage of this approach is that chronyd will stop accumulating
new samples when the PHC is no longer synchronized as phc2sys is
continuously checking state of the PTP port.

This works to some extent, although not as robustly as I would have
hoped. If I use iptables to stop updates from getting to ptp4l, I see
phc2sys notice this and stop updating the SHM, which then causes
chrony to increment its root dispersion and then downstream clients
can see that and make proper decisions.

Unfortunately, if ptp4l isn't running (perhaps it crashed, or was
accidentally killed by an administrator), whether it leaves behind its
unix domain socket or not, it seems that phc2sys is happy to continue
taking samples in that case.

But, this sounds like an issue to be raised on the linuxptp lists.

I'm going to leave this setup running for a while and see if I can
reproduce the weird behavior I was seeing before.

[Chris Perl]

Post by Chris Perl
This works to some extent, although not as robustly as I would have
hoped.

Thinking about this a bit more, I was wondering what you would think
about having some manual way to tell chronyd that a given refclock is
currently in a bad state and should not be used (e.g. some chronyc
command to do so). I thought perhaps using something like `offline'
might just work, but I don't think I'm able to offline a refclock (or,
at least my first few attempts failed with errors about trying to
resolve the name).

If I had such a command, I could write my own (simple) monitoring
daemon that could decide if the PHC refclock was in a good state or
not and then update chronyd with that information.

[Miroslav Lichvar]

Post by Chris Perl
Thinking about this a bit more, I was wondering what you would think
about having some manual way to tell chronyd that a given refclock is
currently in a bad state and should not be used (e.g. some chronyc
command to do so). I thought perhaps using something like `offline'
might just work, but I don't think I'm able to offline a refclock (or,
at least my first few attempts failed with errors about trying to
resolve the name).

A new command that could be used to suspend and resume a reference
clock would make sense to me. I'm not sure what would be the best
syntax. I think it has to separate command from the online/offline
commands as those are specific to NTP sources and the argument is
resolved to an IP address.

Please note that suspending a reference clock would not force a
reselection. When a source doesn't have new samples, it can stay
selected until other sources have better statistics.

[Chris Perl]

Post by Miroslav Lichvar
A new command that could be used to suspend and resume a reference
clock would make sense to me. I'm not sure what would be the best
syntax. I think it has to separate command from the online/offline
commands as those are specific to NTP sources and the argument is
resolved to an IP address.

What about "suspend" and "resume"?

Post by Miroslav Lichvar
Please note that suspending a reference clock would not force a
reselection. When a source doesn't have new samples, it can stay
selected until other sources have better statistics.

Yes, noted.

[Miroslav Lichvar]

Post by Chris Perl

Post by Miroslav Lichvar
A new command that could be used to suspend and resume a reference
clock would make sense to me. I'm not sure what would be the best
syntax. I think it has to separate command from the online/offline
commands as those are specific to NTP sources and the argument is
resolved to an IP address.

What about "suspend" and "resume"?

I think we might want to reserve a suspend/resume command for
suspending/resuming the chrony operation before/after system suspend.

[Chris Perl]

Post by Miroslav Lichvar
I think we might want to reserve a suspend/resume command for
suspending/resuming the chrony operation before/after system suspend.

Good point. I'm going to think on it a while longer and will try to
come up with something.