[chrony-users] Re: Locking PPS refclock with NTP source

[Thibaut BEYLER]

I'm trying to synchronise chrony with a pps signal coming from a bnc output
on a gps timeserver.

The pulses seems to arrive properly according to ppstest, but i don't
figure out how to combine this pulse with a ntp time source to complete the
samples (according to the documentation, it seems to be possible)

All the exemples i see on the Internet use NMEA (which i don't think I have
out of my pps output) or another reference clock, any way to do that ?

[Miroslav Lichvar]

Post by Thibaut BEYLER
I'm trying to synchronise chrony with a pps signal coming from a bnc output
on a gps timeserver.
The pulses seems to arrive properly according to ppstest, but i don't
figure out how to combine this pulse with a ntp time source to complete the
samples (according to the documentation, it seems to be possible)
All the exemples i see on the Internet use NMEA (which i don't think I have
out of my pps output) or another reference clock, any way to do that ?

The lock option can work only with other reference clocks. With an NTP
source you wouldn't specify the option, e.g.

server foo.example.com iburst
refclock PPS /dev/pps0

After start, chronyd will synchronize the clock to the NTP server.
This will allow the PPS refclock to complete the pulses and then it
can take over as the main time source for synchronization of the
system clock.

[Thibaut BEYLER]

Thanks, this is working indeed, but should i get good results (like, not
more than a few of micros from my gps time) with these settings ?

Right now it is locking on PPS (*) but my clock seem worse (offset to utc)
than when using only the ntp server as source.

Post by Thibaut BEYLER

Post by Thibaut BEYLER
I'm trying to synchronise chrony with a pps signal coming from a bnc

output

Post by Thibaut BEYLER
on a gps timeserver.
The pulses seems to arrive properly according to ppstest, but i don't
figure out how to combine this pulse with a ntp time source to complete

the

Post by Thibaut BEYLER
samples (according to the documentation, it seems to be possible)
All the exemples i see on the Internet use NMEA (which i don't think I

have

Post by Thibaut BEYLER
out of my pps output) or another reference clock, any way to do that ?

The lock option can work only with other reference clocks. With an NTP
source you wouldn't specify the option, e.g.
server foo.example.com iburst
refclock PPS /dev/pps0
After start, chronyd will synchronize the clock to the NTP server.
This will allow the PPS refclock to complete the pulses and then it
can take over as the main time source for synchronization of the
system clock.
--
Miroslav Lichvar
--
with "unsubscribe" in the subject.
with "help" in the subject.

[Miroslav Lichvar]

Post by Thibaut BEYLER
Thanks, this is working indeed, but should i get good results (like, not
more than a few of micros from my gps time) with these settings ?

You should get the same results as if the PPS refclock was locked to a
NMEA source for instance. The lock option is mainly useful to speed up
the initial synchronization and allow the noselect option to be
specified with the NMEA or other low quality refclocks.

Post by Thibaut BEYLER
Right now it is locking on PPS (*) but my clock seem worse (offset to utc)
than when using only the ntp server as source.

Can you please post the output of "chronyc -n -m sources sourcestats" ?

What hardware and what PPS driver (e.g. serial, gpio) do you use?
Typically, on a x86_64 machine with a GPS PPS and real serial port the
offset should be stable to a microsecond or better.

[Thibaut BEYLER]

210 Number of sources = 2
MS Name/IP address Stratum Poll Reach LastRx Last sample

===============================================================================

#* PPS0 0 4 377 11 +4829ns[+6161ns]

+/- 3021ns

^- 10.30.121.241 1 6 377 6 -51us[ -50us ]

+/- 154us

210 Number of sources = 2
Name/IP Address NP NR Span Frequency Freq Skew Offset Std

Dev
==============================================================================

PPS0 23 17 383 +0.001 0.052

+5ns 5548ns

10.30.121.241 6 3 325 -0.085 0.323

-41us 7468ns

I am using pps_ldisc, i tried pps-gpio but it was not better.

I am using a custom serial cable (pin 1 of the serial cable is linked to
the bnc ouput on the gps timeserver), same setup had given good results
using an other software.

Thanks, this is working indeed, but should i get good results (like, not
more than a few of micros from my gps time) with these settings ?

You should get the same results as if the PPS refclock was locked to a
NMEA source for instance. The lock option is mainly useful to speed up
the initial synchronization and allow the noselect option to be
specified with the NMEA or other low quality refclocks.

Right now it is locking on PPS (*) but my clock seem worse (offset to

utc)

than when using only the ntp server as source.

Can you please post the output of "chronyc -n -m sources sourcestats" ?
What hardware and what PPS driver (e.g. serial, gpio) do you use?
Typically, on a x86_64 machine with a GPS PPS and real serial port the
offset should be stable to a microsecond or better.
--
Miroslav Lichvar
--
with "unsubscribe" in the subject.
with "help" in the subject.

[Miroslav Lichvar]

Post by Thibaut BEYLER

#* PPS0 0 4 377 11 +4829ns[+6161ns]

+/- 3021ns

PPS0 23 17 383 +0.001 0.052

+5ns 5548ns

I am using pps_ldisc, i tried pps-gpio but it was not better.
I am using a custom serial cable (pin 1 of the serial cable is linked to
the bnc ouput on the gps timeserver), same setup had given good results
using an other software.

The chronyc output suggests the PPS input is very jittery. What
architecture is that? If it has GPIO I'd assume it's not just an
ordinary x86_64 computer. Also, what kernel version is it running? Is
the NOHZ config option enabled? Disabling it with nohz=off might help.

[Thibaut BEYLER]

No I tested it on ordinary x64 kernel (4.9 , 3.16 etc..) coming from debian
& fc repos with no custom options.

I just installed pps-linux and invoked 'ldattach 18 /dev/ttyS0' (which also
loads pps_ldisc module) , maybe i'm missing something ?

Post by Thibaut BEYLER

#* PPS0 0 4 377 11

+4829ns[+6161ns]

Post by Thibaut BEYLER

+/- 3021ns

PPS0 23 17 383 +0.001 0.052

+5ns 5548ns

I am using pps_ldisc, i tried pps-gpio but it was not better.
I am using a custom serial cable (pin 1 of the serial cable is linked to
the bnc ouput on the gps timeserver), same setup had given good results
using an other software.

The chronyc output suggests the PPS input is very jittery. What
architecture is that? If it has GPIO I'd assume it's not just an
ordinary x86_64 computer. Also, what kernel version is it running? Is
the NOHZ config option enabled? Disabling it with nohz=off might help.
--
Miroslav Lichvar
--
with "unsubscribe" in the subject.
with "help" in the subject.

[Miroslav Lichvar]

Post by Thibaut BEYLER
No I tested it on ordinary x64 kernel (4.9 , 3.16 etc..) coming from debian
& fc repos with no custom options.
I just installed pps-linux and invoked 'ldattach 18 /dev/ttyS0' (which also
loads pps_ldisc module) , maybe i'm missing something ?

I'm not sure. Maybe the hardware cannot do better. From what I have
seen, with a real serial port (no USB, etc) the PPS is usually stable to around
1 microsecond. For example, this is what I get with a Garmin 18x I have here:

MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
#* PPS0 0 4 377 8 +200ns[ +391ns] +/- 281ns

Name/IP Address NP NR Span Frequency Freq Skew Offset Std Dev
==============================================================================
PPS0 11 6 161 +0.001 0.015 +18ns 415ns

[Thibaut BEYLER]

Well, I found out that if i run a cpu stress test (i am using mprime
'torture test') while running chrony, suddenly my PPS source std dev goes
stable below 300ns.

So i guess this is realted to some kind of cpu scheduling. However i
disabled the c-state options in the bios, my CPU frequency is not altered
according to /proc/cpuinfo, and yet i still have bad results unless i run
this stress test.

Any idea ?

Post by Thibaut BEYLER

Post by Thibaut BEYLER
No I tested it on ordinary x64 kernel (4.9 , 3.16 etc..) coming from

debian

Post by Thibaut BEYLER
& fc repos with no custom options.
I just installed pps-linux and invoked 'ldattach 18 /dev/ttyS0' (which

also

Post by Thibaut BEYLER
loads pps_ldisc module) , maybe i'm missing something ?

I'm not sure. Maybe the hardware cannot do better. From what I have
seen, with a real serial port (no USB, etc) the PPS is usually stable to around
MS Name/IP address Stratum Poll Reach LastRx Last sample
============================================================
===================
#* PPS0 0 4 377 8 +200ns[ +391ns] +/-
281ns
Name/IP Address NP NR Span Frequency Freq Skew Offset Std Dev
============================================================
==================
PPS0 11 6 161 +0.001 0.015 +18ns
415ns
--
Miroslav Lichvar
--
with "unsubscribe" in the subject.
with "help" in the subject.

[Chris Perl]

Post by Thibaut BEYLER
Well, I found out that if i run a cpu stress test (i am using mprime
'torture test') while running chrony, suddenly my PPS source std dev goes
stable below 300ns.
So i guess this is realted to some kind of cpu scheduling. However i
disabled the c-state options in the bios, my CPU frequency is not altered
according to /proc/cpuinfo, and yet i still have bad results unless i run
this stress test.
Any idea ?

I found similar results when stressing the cpus (I saw the
synchronization get much more stable, although it wasn't with a pps
source).

In my case it was due to cpu c-states, even though I had also had the
bios set to "performance" mode (or something similar).

You can see if its c-states with `turbostat --debug' (in CentOS 6 its
part of the `cpupowerutils' package, in CentOS 7 its part of
`kernel-tools'). For other distro's, I'm not sure.

It'll show you where each processor is spending its time.

If it is, in fact, c-states, then you can either disable them via the
kernel (some combination of `processor.max_cstate',
`intel_idle.max_cstate' and maybe writing "performance" to
`/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor') or you can
use the PM QOS interface from user space [1].

Hope some of that is useful.

[1] https://access.redhat.com/articles/65410

[Chris Perl]

Post by Thibaut BEYLER
So i guess this is realted to some kind of cpu scheduling. However i

And, just to help rule out actual scheduler latency, you could run
chronyd with realtime priority (SCHED_FIFO or SCHED_RR) to make sure
it always has an absolute scheduling advantage over other processes
that aren't running with realtime priority.

[Thibaut BEYLER]

I tested some other BIOS settings and found out that the disabling APIC
support improve a lot chrony perfomances : PPS is good and i get much
better performance with my ntp sources : better std dev and test C is not
failing anymore ( like it did 90% of the time on some sources before with
chrony 3)

However, it also disable SMP support, so it's not a great solution.. I'm
currently trying to play with scheduling settings but so far no luck.

Post by Chris Perl

Post by Thibaut BEYLER
So i guess this is realted to some kind of cpu scheduling. However i

And, just to help rule out actual scheduler latency, you could run
chronyd with realtime priority (SCHED_FIFO or SCHED_RR) to make sure
it always has an absolute scheduling advantage over other processes
that aren't running with realtime priority.
--
with "unsubscribe" in the subject.
with "help" in the subject.

[Bill Unruh]

And how do you know it is worse. The best way would be for you to have an even
better clock source to compare it to.
Do you have that?


William G. Unruh __| Canadian Institute for|____ Tel: [phone]
Physics&Astronomy _|___ Advanced Research _|____ Fax: [phone]
UBC, Vancouver,BC _|_ Program in Cosmology |____ ***@physics.ubc.ca
Canada V6T 1Z1 ____|____ and Gravity ______|_ www.theory.physics.ubc.ca/

Thanks, this is working indeed, but should i get good results (like, not more than a few of micros
from my gps time) with these settings ?
Right now it is locking on PPS (*) but my clock seem worse (offset to utc) than when using only the
ntp server as source.

Post by Thibaut BEYLER
I'm trying to synchronise chrony with a pps signal coming from a bnc output
on a gps timeserver.
The pulses seems to arrive properly according to ppstest, but i don't
figure out how to combine this pulse with a ntp time source to complete the
samples (according to the documentation, it seems to be possible)
All the exemples i see on the Internet use NMEA (which i don't think I have
out of my pps output) or another reference clock, any way to do that ?

The lock option can work only with other reference clocks. With an NTP
source you wouldn't specify the option, e.g.
server foo.example.com iburst
refclock PPS /dev/pps0
After start, chronyd will synchronize the clock to the NTP server.
This will allow the PPS refclock to complete the pulses and then it
can take over as the main time source for synchronization of the
system clock.
--
Miroslav Lichvar