[chrony-users] xleave and HW timestamping

[Joe Williams]

Hello list,

I have chrony working great using interleaved mode between two machines
that have both RX and TX hardware timestamping. It's far more common for
NICs to support TX timestamps in HW rather than both TX and RX. It seems
like in that case the RX timestamps would fall back to happening in the
kernel, while TX would stay in hardware. The xleave docs only mention TX
timestamps, while the RFC draft mentions both RX and TX. So it's not
totally clear to me how much HW RX timestamps matter to interleaved mode.

For clients or servers that don't support HW RX timestamping (only TX) how
much precision would I be giving up?

Thanks!
-Joe

[Miroslav Lichvar]

Post by Joe Williams
Hello list,
I have chrony working great using interleaved mode between two machines
that have both RX and TX hardware timestamping. It's far more common for
NICs to support TX timestamps in HW rather than both TX and RX. It seems
like in that case the RX timestamps would fall back to happening in the
kernel, while TX would stay in hardware. The xleave docs only mention TX
timestamps, while the RFC draft mentions both RX and TX. So it's not
totally clear to me how much HW RX timestamps matter to interleaved mode.

Interleaved mode matters only for TX timestamps. RX timestamps are
the same in basic and interleaved mode.

Post by Joe Williams
For clients or servers that don't support HW RX timestamping (only TX) how
much precision would I be giving up?

As compared to a fully HW timestamping? The main issue would be the
asymmetry. RX timestamps will likely have a much larger error, e.g.
few microseconds vs sub-microsecond in the TX timestamp. The loss of
stability will depend on which direction has more network jitter. If
it's mostly on the RX path, it might not be so bad as chrony would
stick to the more stable TX direction.
--
Miroslav Lichvar

[Joe Williams]

Thanks for the response!

Post by Miroslav Lichvar
As compared to a fully HW timestamping?

Yes, that's right. Having both RX and TX HW timestamping vs only having TX
HW timestamping.

It sounds like having both RX and TX HW timestamps reduces the chance of
asymmetry since both are done in HW, rather than RX done in the kernel and
TX done in HW which ensures some amount of difference. In that case does it
make sense to disable HW timestamping completely if both RX and TX cannot
be done in HW or is there still an advantage of only having TX in HW? Seems
like the same asymmetry would happen in interleave mode when only one side
has full HW timestamping, is that correct?

-Joe

Post by Miroslav Lichvar

Post by Joe Williams
Hello list,
I have chrony working great using interleaved mode between two machines
that have both RX and TX hardware timestamping. It's far more common for
NICs to support TX timestamps in HW rather than both TX and RX. It seems
like in that case the RX timestamps would fall back to happening in the
kernel, while TX would stay in hardware. The xleave docs only mention TX
timestamps, while the RFC draft mentions both RX and TX. So it's not
totally clear to me how much HW RX timestamps matter to interleaved mode.

Interleaved mode matters only for TX timestamps. RX timestamps are
the same in basic and interleaved mode.

Post by Joe Williams
For clients or servers that don't support HW RX timestamping (only TX)

how

Post by Joe Williams
much precision would I be giving up?

As compared to a fully HW timestamping? The main issue would be the
asymmetry. RX timestamps will likely have a much larger error, e.g.
few microseconds vs sub-microsecond in the TX timestamp. The loss of
stability will depend on which direction has more network jitter. If
it's mostly on the RX path, it might not be so bad as chrony would
stick to the more stable TX direction.
--
Miroslav Lichvar
--
with "unsubscribe" in the subject.
with "help" in the subject.

[Miroslav Lichvar]

Post by Joe Williams
It sounds like having both RX and TX HW timestamps reduces the chance of
asymmetry since both are done in HW, rather than RX done in the kernel and
TX done in HW which ensures some amount of difference. In that case does it
make sense to disable HW timestamping completely if both RX and TX cannot
be done in HW or is there still an advantage of only having TX in HW? Seems
like the same asymmetry would happen in interleave mode when only one side
has full HW timestamping, is that correct?

It depends on what is more important for you, accuracy or stability?
With partial HW timestamping you will likely have a better stability,
but accuracy may be worse. It doesn't have to be, it depends on the
server and client HW and configuration.

Usually, there always is some asymmetry. Even when both RX and TX are
HW-timestamped, there will likely be an error of a few hundred
nanoseconds. An exception is the I210, which has the timestamping
errors compensated in the driver.

What matters for accuracy is how those errors cancel out between the
server and client. The best case is that they both have the same
hardware and configuration, so any asymmetries cancel out.

Maybe you have a number of identical clients and you need their clocks
to be close to each other, but don't care much about accuracy wrt the
server? In such case you would prefer partial HW timestamping
over no HW timestamping.
--
Miroslav Lichvar

[Joe Williams]

This is great info, thank you!

Post by Miroslav Lichvar

Post by Joe Williams
It sounds like having both RX and TX HW timestamps reduces the chance of
asymmetry since both are done in HW, rather than RX done in the kernel

and

Post by Joe Williams
TX done in HW which ensures some amount of difference. In that case does

it

Post by Joe Williams
make sense to disable HW timestamping completely if both RX and TX cannot
be done in HW or is there still an advantage of only having TX in HW?

Seems

Post by Joe Williams
like the same asymmetry would happen in interleave mode when only one

side

Post by Joe Williams
has full HW timestamping, is that correct?

It depends on what is more important for you, accuracy or stability?
With partial HW timestamping you will likely have a better stability,
but accuracy may be worse. It doesn't have to be, it depends on the
server and client HW and configuration.
Usually, there always is some asymmetry. Even when both RX and TX are
HW-timestamped, there will likely be an error of a few hundred
nanoseconds. An exception is the I210, which has the timestamping
errors compensated in the driver.
What matters for accuracy is how those errors cancel out between the
server and client. The best case is that they both have the same
hardware and configuration, so any asymmetries cancel out.
Maybe you have a number of identical clients and you need their clocks
to be close to each other, but don't care much about accuracy wrt the
server? In such case you would prefer partial HW timestamping
over no HW timestamping.
--
Miroslav Lichvar
--
with "unsubscribe" in the subject.
with "help" in the subject.