ptp: Support hardware clocks with additional free running time
Subsystems (12)
cover ptp: Support hardware clocks with additional free running time
ptp vclocks require a clock with free running time for the timecounter. Currently only a physical clock forced to free running is supported. If vclocks are used, then the physical clock cannot be synchronized anymore. The synchronized time is not available in hardware in this case. As a result, timed transmission with ETF/TAPRIO hardware support is not possible anymore.
If hardware would support a free running time additionally to the physical clock, then the physical clock does not need to be forced to free running. Thus, the physical clocks can still be synchronized while vclocks are in use.
The physical clock could be used to synchronize the time domain of the TSN network and trigger ETF/TAPRIO. In parallel vclocks can be used to synchronize other time domains.
One year ago I thought for two time domains within a TSN network also two physical clocks are required. This would lead to new kernel interfaces for asking for the second clock, ... . But actually for a time triggered system like TSN there can be only one time domain that controls the system itself. All other time domains belong to other layers, but not to the time triggered system itself. So other time domains can be based on a free running counter if similar mechanisms to 2 step synchronisation are used.
Synchronisation was tested with two time domains between two directly connected hosts. Each host run two ptp4l instances, the first used the physical clock and the second used the virtual clock. I used my FPGA based network controller as network device. ptp4l was used in combination with the virtual clock support patch set from Miroslav Lichvar.
Please give me some feedback. For me it seems like a straight forward extension for ptp vclocks, but I'm new to this topic.
Gerhard Engleder (6):
bpf: Access hwtstamp field of hwtstamps directly
ptp: Initialize skb_shared_hwtstamps
ptp: Add free running time support
ptp: Support time stamps based on free running time
ptp: Allow vclocks without free running physical clock
tsnep: Add free running time support
.../net/ethernet/cavium/liquidio/lio_main.c | 1 + .../ethernet/cavium/liquidio/lio_vf_main.c | 1 + .../net/ethernet/chelsio/cxgb4/cxgb4_ptp.c | 1 + drivers/net/ethernet/engleder/tsnep_hw.h | 9 ++++-- drivers/net/ethernet/engleder/tsnep_main.c | 6 ++++ drivers/net/ethernet/engleder/tsnep_ptp.c | 28 +++++++++++++++++++ .../hisilicon/hns3/hns3pf/hclge_ptp.c | 1 + .../ethernet/mellanox/mlxsw/spectrum_ptp.c | 1 + drivers/net/ethernet/sfc/tx_common.c | 1 + drivers/ptp/ptp_clock.c | 8 +++--- drivers/ptp/ptp_ines.c | 1 + drivers/ptp/ptp_private.h | 9 ++++++ drivers/ptp/ptp_sysfs.c | 11 ++++---- drivers/ptp/ptp_vclock.c | 25 +++++++++++++---- include/linux/ptp_clock_kernel.h | 27 ++++++++++++++++++ include/linux/skbuff.h | 3 ++ net/core/filter.c | 3 +- 17 files changed, 117 insertions(+), 19 deletions(-)
Signature · 1 line
2.20.1
RFC PATCH 1/6 bpf: Access hwtstamp field of hwtstamps directly
skb_shared_hwtstamps contains only the field hwtstamp. That property is hard-coded and checked during build in BPF. bpf_target_off() gets the whole structure as argument (hwtstamps) instead of the actually accessed field hwtstamp.
Access hwtstamp field directly and allow future extensions of skb_shared_hwtstamps.
--- net/core/filter.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
net/core/filter.c +1 -2
@@ -9364,13 +9364,12 @@ static u32 bpf_convert_ctx_access(enum bpf_access_type type,break ;case offsetof(struct __sk_buff , hwtstamp): BUILD_BUG_ON(sizeof_field(struct skb_shared_hwtstamps, hwtstamp) != 8); -BUILD_BUG_ON (offsetof(struct skb_shared_hwtstamps , hwtstamp) !=0 ); insn =bpf_convert_shinfo_access (si, insn); *insn++ =BPF_LDX_MEM (BPF_DW , si->dst_reg, si->dst_reg, bpf_target_off(struct skb_shared_info, - hwtstamps,8 , + hwtstamps.hwtstamp,8 , target_size));break ; }
Signature · 1 line
2.20.1
RFC PATCH 2/6 ptp: Initialize skb_shared_hwtstamps
In reply to Gerhard EnglederThere are only a few locations, which do not initialize the whole skb_shared_hwtstamps structure and write the only field of the structure. This is ok as long as skb_shared_hwtstamps is not extended with additional fields.
Always initialize the whole skb_shared_hwtstamps structure to prepare for future extensions of this structure.
--- drivers/net/ethernet/cavium/liquidio/lio_main.c | 1 + drivers/net/ethernet/cavium/liquidio/lio_vf_main.c | 1 + drivers/net/ethernet/chelsio/cxgb4/cxgb4_ptp.c | 1 + drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_ptp.c | 1 + drivers/net/ethernet/mellanox/mlxsw/spectrum_ptp.c | 1 + drivers/net/ethernet/sfc/tx_common.c | 1 + drivers/ptp/ptp_ines.c | 1 + 7 files changed, 7 insertions(+)
drivers/net/ethernet/cavium/liquidio/lio_main.c +1 -0
@@ -2212,6 +2212,7 @@ static void handle_timestamp(struct octeon_device *oct,netif_info (lio, tx_done, lio->netdev,"Got resulting SKBTX_HW_TSTAMP skb=%p ns=%016llu\n" , skb, (unsigned long long )ns); +memset (&ts,0 ,sizeof (ts)); ts.hwtstamp =ns_to_ktime (ns + lio->ptp_adjust);skb_tstamp_tx (skb, &ts); }
drivers/net/ethernet/cavium/liquidio/lio_vf_main.c +1 -0
@@ -1340,6 +1340,7 @@ static void handle_timestamp(struct octeon_device *oct, u32 status, void *buf)netif_info (lio, tx_done, lio->netdev,"Got resulting SKBTX_HW_TSTAMP skb=%p ns=%016llu\n" , skb, (unsigned long long )ns); +memset (&ts,0 ,sizeof (ts)); ts.hwtstamp =ns_to_ktime (ns + lio->ptp_adjust);skb_tstamp_tx (skb, &ts); }
drivers/net/ethernet/chelsio/cxgb4/cxgb4_ptp.c +1 -0
@@ -109,6 +109,7 @@ void cxgb4_ptp_read_hwstamp(struct adapter *adapter, struct port_info *pi) tx_ts |= (u64 )t4_read_reg (adapter,T5_PORT_REG (pi->port_id,MAC_PORT_TX_TS_VAL_HI )) <<32 ; +memset (skb_ts,0 ,sizeof (*skb_ts)); skb_ts->hwtstamp =ns_to_ktime (tx_ts);skb_tstamp_tx (adapter->ptp_tx_skb, skb_ts);dev_kfree_skb_any (adapter->ptp_tx_skb);
drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_ptp.c +1 -0
@@ -103,6 +103,7 @@ void hclge_ptp_clean_tx_hwts(struct hclge_dev *hdev) hdev->ptp->tx_cleaned++; ns += (((u64 )hi) <<32 | lo) *NSEC_PER_SEC ; +memset (&hwts,0 ,sizeof (hwts)); hwts.hwtstamp =ns_to_ktime (ns);skb_tstamp_tx (skb, &hwts);dev_kfree_skb_any (skb);
drivers/net/ethernet/mellanox/mlxsw/spectrum_ptp.c +1 -0
@@ -445,6 +445,7 @@ static void mlxsw_sp1_packet_timestamp(struct mlxsw_sp *mlxsw_sp, nsec =timecounter_cyc2time (&mlxsw_sp->clock->tc, timestamp);spin_unlock_bh (&mlxsw_sp->clock->lock); +memset (&hwtstamps,0 ,sizeof (hwtstamps)); hwtstamps.hwtstamp =ns_to_ktime (nsec);mlxsw_sp1_ptp_packet_finish (mlxsw_sp, skb, key.local_port, key.ingress, &hwtstamps);
drivers/net/ethernet/sfc/tx_common.c +1 -0
@@ -170,6 +170,7 @@ void efx_dequeue_buffer(struct efx_tx_queue *tx_queue, tx_queue->completed_timestamp_minor)) {struct skb_shared_hwtstamps hwtstamp; +memset (&hwtstamp,0 ,sizeof (hwtstamp)); hwtstamp.hwtstamp =efx_ptp_nic_to_kernel_time (tx_queue);skb_tstamp_tx (skb, &hwtstamp);
drivers/ptp/ptp_ines.c +1 -0
@@ -659,6 +659,7 @@ static void ines_txtstamp_work(struct work_struct *work)kfree_skb (skb);return ; } +memset (&ssh,0 ,sizeof (ssh)); ssh.hwtstamp =ns_to_ktime (ns);skb_complete_tx_timestamp (skb, &ssh); }
Signature · 1 line
2.20.1
RFC PATCH 3/6 ptp: Add free running time support
In reply to Gerhard Englederptp vclocks require a clock with free running time for the timecounter. Currently only a physical clock forced to free running is supported. If vclocks are used, then the physical clock cannot be synchronized anymore. The synchronized time is not available in hardware in this case. As a result, timed transmission with ETF/TAPRIO hardware support is not possible anymore.
If hardware would support a free running time additionally to the physical clock, then the physical clock does not need to be forced to free running. Thus, the physical clocks can still be synchronized while vclocks are in use.
The physical clock could be used to synchronize the time domain of the TSN network and trigger ETF/TAPRIO. In parallel vclocks can be used to synchronize other time domains.
Allow read and cross time stamp of additional free running time for physical clocks. Let vclocks use free running time if available.
--- drivers/ptp/ptp_vclock.c | 20 +++++++++++++++----- include/linux/ptp_clock_kernel.h | 27 +++++++++++++++++++++++++++ 2 files changed, 42 insertions(+), 5 deletions(-)
drivers/ptp/ptp_vclock.c +15 -5
@@ -68,7 +68,10 @@ static int ptp_vclock_gettimex(struct ptp_clock_info *ptp,int err;u64 ns; - err = pptp->info->gettimex64 (pptp->info, &pts, sts); +if (pptp->info->getfreeruntimex64) + err = pptp->info->getfreeruntimex64 (pptp->info, &pts, sts); +else + err = pptp->info->gettimex64 (pptp->info, &pts, sts);if (err)return err;
@@ -104,7 +107,10 @@ static int ptp_vclock_getcrosststamp(struct ptp_clock_info *ptp,int err;u64 ns; - err = pptp->info->getcrosststamp (pptp->info, xtstamp); +if (pptp->info->getfreeruncrosststamp) + err = pptp->info->getfreeruncrosststamp (pptp->info, xtstamp); +else + err = pptp->info->getcrosststamp (pptp->info, xtstamp);if (err)return err;
@@ -143,7 +149,9 @@ static u64 ptp_vclock_read(const struct cyclecounter *cc)struct ptp_clock *ptp = vclock->pclock;struct timespec64 ts = {}; -if (ptp->info->gettimex64) +if (ptp->info->getfreeruntimex64) + ptp->info->getfreeruntimex64 (ptp->info, &ts,NULL ); +else if (ptp->info->gettimex64) ptp->info->gettimex64 (ptp->info, &ts,NULL );else ptp->info->gettime64 (ptp->info, &ts);
@@ -168,11 +176,13 @@ struct ptp_vclock *ptp_vclock_register(struct ptp_clock *pclock) vclock->pclock = pclock; vclock->info = ptp_vclock_info; -if (pclock->info->gettimex64) +if (pclock->info->getfreeruntimex64 || pclock->info->gettimex64) vclock->info.gettimex64 = ptp_vclock_gettimex;else vclock->info.gettime64 = ptp_vclock_gettime; -if (pclock->info->getcrosststamp) +if ((pclock->info->getfreeruntimex64 && + pclock->info->getfreeruncrosststamp) || + pclock->info->getcrosststamp) vclock->info.getcrosststamp = ptp_vclock_getcrosststamp; vclock->cc = ptp_vclock_cc;
include/linux/ptp_clock_kernel.h +27 -0
@@ -108,6 +108,28 @@ struct ptp_system_timestamp {* @settime64: Set the current time on the hardware clock. * parameter ts: Time value to set. * +* @getfreeruntimex64: Reads the current free running time from the hardware +* clock and optionally also the system clock. This +* operation requires hardware support for an additional +* free running time including support for hardware time +* stamps based on that free running time. +* The free running time must be completely independet from +* the actual time of the PTP clock. It must be monotonic +* and its frequency must be constant. +* parameter ts: Holds the PHC free running timestamp. +* parameter sts: If not NULL, it holds a pair of +* timestamps from the system clock. The first reading is +* made right before reading the lowest bits of the PHC +* free running timestamp and the second reading +* immediately follows that. +* +* @getfreeruncrosststamp: Reads the current time from the free running +* hardware clock and system clock simultaneously. +* parameter cts: Contains timestamp (device,system) +* pair, where device time is the free running time +* also used for @getfreeruntimex64 and system time is +* realtime and monotonic. +* * @enable: Request driver to enable or disable an ancillary feature. * parameter request: Desired resource to enable or disable. * parameter on: Caller passes one to enable or zero to disable.
@@ -155,6 +177,11 @@ struct ptp_clock_info {int (*getcrosststamp)(struct ptp_clock_info *ptp,struct system_device_crosststamp *cts);int (*settime64)(struct ptp_clock_info *p,const struct timespec64 *ts); +int (*getfreeruntimex64)(struct ptp_clock_info *ptp, +struct timespec64 *ts, +struct ptp_system_timestamp *sts); +int (*getfreeruncrosststamp)(struct ptp_clock_info *ptp, +struct system_device_crosststamp *cts);int (*enable)(struct ptp_clock_info *ptp,struct ptp_clock_request *request,int on); int (*verify)(struct ptp_clock_info *ptp, unsigned int pin,
Signature · 1 line
2.20.1
Richard Cochran
maintainerRe: [RFC PATCH net-next 3/6] ptp: Add free running time support
Quoted 30 lines · Gerhard Engleder
ptp vclocks require a clock with free running time for the timecounter. Currently only a physical clock forced to free running is supported. If vclocks are used, then the physical clock cannot be synchronized anymore. The synchronized time is not available in hardware in this case. As a result, timed transmission with ETF/TAPRIO hardware support is not possible anymore.
If hardware would support a free running time additionally to the physical clock, then the physical clock does not need to be forced to free running. Thus, the physical clocks can still be synchronized while vclocks are in use.
The physical clock could be used to synchronize the time domain of the TSN network and trigger ETF/TAPRIO. In parallel vclocks can be used to synchronize other time domains.
Allow read and cross time stamp of additional free running time for physical clocks. Let vclocks use free running time if available.
Signed-off-by Gerhard Engleder <[email protected]>--- drivers/ptp/ptp_vclock.c | 20 +++++++++++++++----- include/linux/ptp_clock_kernel.h | 27 +++++++++++++++++++++++++++ 2 files changed, 42 insertions(+), 5 deletions(-)drivers/ptp/ptp_vclock.c +0
-0
drivers/ptp/ptp_vclock.c static int ptp_vclock_gettimex(struct ptp_clock_info *ptp, · line 74
Quoted 8 lines
drivers/ptp/ptp_vclock.c +4
-1@@ -68,7 +68,10 @@ static int ptp_vclock_gettimex(struct ptp_clock_info *ptp,int err;u64 ns; - err = pptp->info->gettimex64 (pptp->info, &pts, sts); +if (pptp->info->getfreeruntimex64) + err = pptp->info->getfreeruntimex64 (pptp->info, &pts, sts); +else + err = pptp->info->gettimex64 (pptp->info, &pts, sts);
Why all this extra if/then/else here and at registration? Just provide new ptp_vclock_ helpers and drop the run time conditionals.
drivers/ptp/ptp_vclock.c static int ptp_vclock_getcrosststamp(struct ptp_clock_info *ptp, · line 113
Quoted 12 lines
drivers/ptp/ptp_vclock.c +4
-1if (err)return err;@@ -104,7 +107,10 @@ static int ptp_vclock_getcrosststamp(struct ptp_clock_info *ptp,int err;u64 ns; - err = pptp->info->getcrosststamp (pptp->info, xtstamp); +if (pptp->info->getfreeruncrosststamp) + err = pptp->info->getfreeruncrosststamp (pptp->info, xtstamp); +else + err = pptp->info->getcrosststamp (pptp->info, xtstamp);
same here
Quoted 64 lines
drivers/ptp/ptp_vclock.c +7
-3if (err)return err;@@ -143,7 +149,9 @@ static u64 ptp_vclock_read(const struct cyclecounter *cc)struct ptp_clock *ptp = vclock->pclock;struct timespec64 ts = {}; -if (ptp->info->gettimex64) +if (ptp->info->getfreeruntimex64) + ptp->info->getfreeruntimex64 (ptp->info, &ts,NULL ); +else if (ptp->info->gettimex64) ptp->info->gettimex64 (ptp->info, &ts,NULL );else ptp->info->gettime64 (ptp->info, &ts);@@ -168,11 +176,13 @@ struct ptp_vclock *ptp_vclock_register(struct ptp_clock *pclock) vclock->pclock = pclock; vclock->info = ptp_vclock_info; -if (pclock->info->gettimex64) +if (pclock->info->getfreeruntimex64 || pclock->info->gettimex64) vclock->info.gettimex64 = ptp_vclock_gettimex;else vclock->info.gettime64 = ptp_vclock_gettime; -if (pclock->info->getcrosststamp) +if ((pclock->info->getfreeruntimex64 && + pclock->info->getfreeruncrosststamp) || + pclock->info->getcrosststamp) vclock->info.getcrosststamp = ptp_vclock_getcrosststamp; vclock->cc = ptp_vclock_cc;include/linux/ptp_clock_kernel.h +22
-0@@ -108,6 +108,28 @@ struct ptp_system_timestamp {* @settime64: Set the current time on the hardware clock. * parameter ts: Time value to set. * +* @getfreeruntimex64: Reads the current free running time from the hardware +* clock and optionally also the system clock. This +* operation requires hardware support for an additional +* free running time including support for hardware time +* stamps based on that free running time. +* The free running time must be completely independet from +* the actual time of the PTP clock. It must be monotonic +* and its frequency must be constant. +* parameter ts: Holds the PHC free running timestamp. +* parameter sts: If not NULL, it holds a pair of +* timestamps from the system clock. The first reading is +* made right before reading the lowest bits of the PHC +* free running timestamp and the second reading +* immediately follows that. +* +* @getfreeruncrosststamp: Reads the current time from the free running +* hardware clock and system clock simultaneously. +* parameter cts: Contains timestamp (device,system) +* pair, where device time is the free running time +* also used for @getfreeruntimex64 and system time is +* realtime and monotonic. +* * @enable: Request driver to enable or disable an ancillary feature. * parameter request: Desired resource to enable or disable. * parameter on: Caller passes one to enable or zero to disable.
include/linux/ptp_clock_kernel.h struct ptp_clock_info { · line 184
Quoted 8 lines
include/linux/ptp_clock_kernel.h +5
-0@@ -155,6 +177,11 @@ struct ptp_clock_info {int (*getcrosststamp)(struct ptp_clock_info *ptp,struct system_device_crosststamp *cts);int (*settime64)(struct ptp_clock_info *p,const struct timespec64 *ts); +int (*getfreeruntimex64)(struct ptp_clock_info *ptp, +struct timespec64 *ts, +struct ptp_system_timestamp *sts); +int (*getfreeruncrosststamp)(struct ptp_clock_info *ptp, +struct system_device_crosststamp *cts);
Wow, that is really hard to read. Suggest freerun_gettimex64 and freerun_crosststamp instead.
Quoted 6 lines
include/linux/ptp_clock_kernel.h +0
-0int (*enable)(struct ptp_clock_info *ptp,struct ptp_clock_request *request,int on); int (*verify)(struct ptp_clock_info *ptp, unsigned int pin,Signature · 1 line
2.20.1
Thanks,
Richard
RFC PATCH 4/6 ptp: Support time stamps based on free running time
In reply to Gerhard EnglederPhysical clocks are used for hardware time stamping. Also ptp vclocks support hardware time stamps. If a physical clock additionally supports a free running time and this time is used as base for ptp vclocks, then also hardware time stamps based on that free running time are required.
Add hardware time stamp of additional free running time to skb_shared_hwtstamps and use it if physical clock supports an additional free running time.
--- drivers/ptp/ptp_vclock.c | 5 ++++- include/linux/skbuff.h | 3 +++ 2 files changed, 7 insertions(+), 1 deletion(-)
drivers/ptp/ptp_vclock.c +4 -1
@@ -268,7 +268,10 @@ ktime_t ptp_convert_timestamp(const struct skb_shared_hwtstamps *hwtstamps, vclock =info_to_vclock (ptp->info); - ns =ktime_to_ns (hwtstamps->hwtstamp); +if (vclock->pclock->info->getfreeruntimex64) + ns =ktime_to_ns (hwtstamps->hwfreeruntstamp); +else + ns =ktime_to_ns (hwtstamps->hwtstamp);spin_lock_irqsave (&vclock->lock, flags); ns =timecounter_cyc2time (&vclock->tc, ns);
include/linux/skbuff.h +3 -0
@@ -521,6 +521,8 @@ static inline bool skb_frag_must_loop(struct page *p)* struct skb_shared_hwtstamps - hardware time stamps * @hwtstamp: hardware time stamp transformed into duration * since arbitrary point in time +* @hwfreeruntstamp: hardware time stamp based on free running time +* transformed into duration since arbitrary point in time * * Software time stamps generated by ktime_get_real() are stored in * skb->tstamp.
@@ -533,6 +535,7 @@ static inline bool skb_frag_must_loop(struct page *p)*/ struct skb_shared_hwtstamps {ktime_t hwtstamp; +ktime_t hwfreeruntstamp; };/* Definitions for tx_flags in struct skb_shared_info */
Signature · 1 line
2.20.1
Richard Cochran
maintainerRe: [RFC PATCH net-next 4/6] ptp: Support time stamps based on free running time
Quoted 14 lines · Gerhard Engleder
include/linux/skbuff.h +2
-0@@ -521,6 +521,8 @@ static inline bool skb_frag_must_loop(struct page *p)* struct skb_shared_hwtstamps - hardware time stamps * @hwtstamp: hardware time stamp transformed into duration * since arbitrary point in time +* @hwfreeruntstamp: hardware time stamp based on free running time +* transformed into duration since arbitrary point in time * * Software time stamps generated by ktime_get_real() are stored in * skb->tstamp.
include/linux/skbuff.h static inline bool skb_frag_must_loop(struct page *p) · line 538
Quoted 4 lines
include/linux/skbuff.h +1
-0@@ -533,6 +535,7 @@ static inline bool skb_frag_must_loop(struct page *p)*/ struct skb_shared_hwtstamps {ktime_t hwtstamp; +ktime_t hwfreeruntstamp;
You are adding eight bytes per frame for what is arguably an extreme niche case. I personally wouldn't mind, but expect push back from the rest of the world!
Maybe this should hide behind a Kconfig option with default off.
@davem, @kuba, what do you think?
Quoted 6 lines
include/linux/skbuff.h +0
-0};/* Definitions for tx_flags in struct skb_shared_info */ Signature · 1 line
2.20.1
Thanks,
Richard
RFC PATCH 5/6 ptp: Allow vclocks without free running physical clock
In reply to Gerhard EnglederIf a physical clock supports an additional free running time, then there is no need to force the physical clock itself to free running.
Guarantee free running physical clock only if additional free running time is not supported.
--- drivers/ptp/ptp_clock.c | 8 ++++---- drivers/ptp/ptp_private.h | 9 +++++++++ drivers/ptp/ptp_sysfs.c | 11 ++++++----- 3 files changed, 19 insertions(+), 9 deletions(-)
drivers/ptp/ptp_clock.c +4 -4
@@ -77,8 +77,8 @@ static int ptp_clock_settime(struct posix_clock *pc, const struct timespec64 *tp { struct ptp_clock *ptp = container_of(pc, struct ptp_clock, clock); -if (ptp_vclock_in_use (ptp)) { -pr_err ("ptp: virtual clock in use\n" ); +if (ptp_vclock_free_run (ptp)) { +pr_err ("ptp: virtual clock requires free run\n" );return -EBUSY ; }
@@ -103,8 +103,8 @@ static int ptp_clock_adjtime(struct posix_clock *pc, struct __kernel_timex *tx)struct ptp_clock_info *ops;int err = -EOPNOTSUPP ; -if (ptp_vclock_in_use (ptp)) { -pr_err ("ptp: virtual clock in use\n" ); +if (ptp_vclock_free_run (ptp)) { +pr_err ("ptp: virtual clock requires free run\n" );return -EBUSY ; }
drivers/ptp/ptp_private.h +9 -0
@@ -96,6 +96,15 @@ static inline bool ptp_vclock_in_use(struct ptp_clock *ptp)return in_use; } +/* Check if free run for virtual clock is required */ +static inline bool ptp_vclock_free_run (struct ptp_clock *ptp) +{ +if (ptp->info->getfreeruntimex64) +return false; +else +return ptp_vclock_in_use (ptp); +} +extern struct class *ptp_class;/*
drivers/ptp/ptp_sysfs.c +6 -5
@@ -230,11 +230,12 @@ static ssize_t n_vclocks_store(struct device *dev,for (i =1 ; i <= ptp->n_vclocks - num; i++) *(ptp->vclock_index + ptp->n_vclocks - i) =-1 ; } - -if (num ==0 ) -dev_info (dev,"only physical clock in use now\n" ); -else -dev_info (dev,"guarantee physical clock free running\n" ); +if (!ptp->info->getfreeruntimex64) { +if (num ==0 ) +dev_info (dev,"only physical clock in use now\n" ); +else +dev_info (dev,"guarantee physical clock free running\n" ); + } ptp->n_vclocks = num;mutex_unlock (&ptp->n_vclocks_mux);
Signature · 1 line
2.20.1
RFC PATCH 6/6 tsnep: Add free running time support
In reply to Gerhard EnglederThe TSN endpoint Ethernet MAC supports a free running counter additionally to its clock. This free running counter can be read and hardware time stamps are supported. As the name implies, this counter cannot be set and its frequency cannot be adjusted.
Add free running time support based on free running counter to physical clock. This also requires hardware time stamps based on that free running time.
--- drivers/net/ethernet/engleder/tsnep_hw.h | 9 +++++-- drivers/net/ethernet/engleder/tsnep_main.c | 6 +++++ drivers/net/ethernet/engleder/tsnep_ptp.c | 28 ++++++++++++++++++++++ 3 files changed, 41 insertions(+), 2 deletions(-)
drivers/net/ethernet/engleder/tsnep_hw.h +7 -2
@@ -43,6 +43,10 @@#define ECM_RESET_CHANNEL 0x00000100#define ECM_RESET_TXRX 0x00010000 +/* counter */ +#define ECM_COUNTER_LOW 0x0028 +#define ECM_COUNTER_HIGH 0x002C +/* control and status */ #define ECM_STATUS 0x0080#define ECM_LINK_MODE_OFF 0x01000000
@@ -190,7 +194,8 @@ struct tsnep_tx_desc {/* tsnep TX descriptor writeback */ struct tsnep_tx_desc_wb {__le32 properties; -__le32 reserved1[3 ]; +__le32 reserved1; +__le64 counter;__le64 timestamp;__le32 dma_delay;__le32 reserved2;
@@ -221,7 +226,7 @@ struct tsnep_rx_desc_wb {/* tsnep RX inline meta */ struct tsnep_rx_inline { -__le64 reserved; +__le64 counter;__le64 timestamp; };
drivers/net/ethernet/engleder/tsnep_main.c +6 -0
@@ -472,9 +472,12 @@ static bool tsnep_tx_poll(struct tsnep_tx *tx, int napi_budget)struct skb_shared_hwtstamps hwtstamps;u64 timestamp =__le64_to_cpu (entry->desc_wb->timestamp); +u64 counter = +__le64_to_cpu (entry->desc_wb->counter);memset (&hwtstamps,0 ,sizeof (hwtstamps)); hwtstamps.hwtstamp =ns_to_ktime (timestamp); + hwtstamps.hwfreeruntstamp =ns_to_ktime (counter);skb_tstamp_tx (entry->skb, &hwtstamps); }
@@ -706,9 +709,12 @@ static int tsnep_rx_poll(struct tsnep_rx *rx, struct napi_struct *napi, (struct tsnep_rx_inline *)skb->data;u64 timestamp =__le64_to_cpu (rx_inline->timestamp); +u64 counter = +__le64_to_cpu (rx_inline->counter);memset (hwtstamps,0 ,sizeof (*hwtstamps)); hwtstamps->hwtstamp =ns_to_ktime (timestamp); + hwtstamps->hwfreeruntstamp =ns_to_ktime (counter); }skb_pull (skb,TSNEP_RX_INLINE_METADATA_SIZE ); skb->protocol = eth_type_trans(skb,
drivers/net/ethernet/engleder/tsnep_ptp.c +28 -0
@@ -175,6 +175,33 @@ static int tsnep_ptp_settime64(struct ptp_clock_info *ptp,return 0 ; } +static int tsnep_ptp_getfreeruntimex64 (struct ptp_clock_info *ptp, +struct timespec64 *ts, +struct ptp_system_timestamp *sts) +{ + struct tsnep_adapter *adapter = container_of(ptp, struct tsnep_adapter, + ptp_clock_info); +u32 high_before; +u32 low; +u32 high; +u64 counter; + +/* read high dword twice to detect overrun */ + high =ioread32 (adapter->addr +ECM_COUNTER_HIGH ); +do { +ptp_read_system_prets (sts); + low =ioread32 (adapter->addr +ECM_COUNTER_LOW ); +ptp_read_system_postts (sts); + high_before = high; + high =ioread32 (adapter->addr +ECM_COUNTER_HIGH ); + }while (high != high_before); + counter = (((u64 )high) <<32 ) | ((u64 )low); + + *ts =ns_to_timespec64 (counter); + +return 0 ; +} +int tsnep_ptp_init (struct tsnep_adapter *adapter) {int retval =0 ;
@@ -192,6 +219,7 @@ int tsnep_ptp_init(struct tsnep_adapter *adapter) adapter->ptp_clock_info.adjtime = tsnep_ptp_adjtime; adapter->ptp_clock_info.gettimex64 = tsnep_ptp_gettimex64; adapter->ptp_clock_info.settime64 = tsnep_ptp_settime64; + adapter->ptp_clock_info.getfreeruntimex64 = tsnep_ptp_getfreeruntimex64;spin_lock_init (&adapter->ptp_lock);
Signature · 1 line
2.20.1
Richard Cochran
maintainerQuoted 4 lines · Gerhard Engleder
If hardware would support a free running time additionally to the physical clock, then the physical clock does not need to be forced to free running. Thus, the physical clocks can still be synchronized while vclocks are in use.
So... the HW must provide frame time stamps using the one clock and ancillary operations using the other, right?
Thanks,
Richard
Richard Cochran
maintainerQuoted 9 lines · Richard Cochran
Quoted 4 lines · Gerhard Engleder
If hardware would support a free running time additionally to the physical clock, then the physical clock does not need to be forced to free running. Thus, the physical clocks can still be synchronized while vclocks are in use.
So... the HW must provide frame time stamps using the one clock and ancillary operations using the other, right?
Looking at your tsnep driver, the requirement is that the HW provide each frame with TWO time stamps, one from each clock.
Thanks,
Richard
Quoted 6 lines · Gerhard Engleder
ptp vclocks require a clock with free running time for the timecounter. Currently only a physical clock forced to free running is supported. If vclocks are used, then the physical clock cannot be synchronized anymore. The synchronized time is not available in hardware in this case. As a result, timed transmission with ETF/TAPRIO hardware support is not possible anymore.
NAK.
I don't see why you don't simply provide two PHC instances from this one device.
AFAICT, this series is not needed at all, as the existing API covers this use case already.
Thanks,
Richard
Quoted 11 lines · Richard Cochran
Quoted 6 lines
ptp vclocks require a clock with free running time for the timecounter. Currently only a physical clock forced to free running is supported. If vclocks are used, then the physical clock cannot be synchronized anymore. The synchronized time is not available in hardware in this case. As a result, timed transmission with ETF/TAPRIO hardware support is not possible anymore.
NAK.
I don't see why you don't simply provide two PHC instances from this one device.
Because with vclocks the user space interface is already available. Also In my opinion it is a good fit. The second PHC would be based on the free running hardware counter. So it would not provide any additional functionality compared to the vclocks based on it.
Are two PHC instances supported? At least for ethtool there is only a single phc_index field.
Quoted 2 lines
AFAICT, this series is not needed at all, as the existing API covers this use case already.
I assume that you mean for ETF the transmission time can be converted, similar like for time stamps. So for ETF you are right. It was too quick to mention ETF along with TAPRIO.
My use case is TAPRIO with hardware support. For TAPRIO the hardware has to act based on the synchronized time within the TSN network. No transmission times, which could be converted, are used. The hardware is in charge to transmit frames from a certain queue only during defined intervals. These intervals are based on the synchronized time. So the hardware must be synchronized somehow. This is my solution to keep the hardware synchronized while vclocks are in use.
How can I cover my use case with the existing API? I had no idea so far.
Thanks!
Gerhard
Richard Cochran
maintainerQuoted 1 line · Gerhard Engleder
How can I cover my use case with the existing API? I had no idea so far.
Okay, so 2 PHCs doesn't help, but still all you need is:
1. a different method to convert time stamps to vclock time base
2. a different method for vclocks' gettime
So let me suggest a much smaller change to the phc/vclock api... stay tuned
Thanks,
Richard
Richard Cochran
maintainerQuoted 10 lines · Richard Cochran
Quoted 1 line · Gerhard Engleder
How can I cover my use case with the existing API? I had no idea so far.
Okay, so 2 PHCs doesn't help, but still all you need is:
1. a different method to convert time stamps to vclock time base
2. a different method for vclocks' gettime
So let me suggest a much smaller change to the phc/vclock api... stay tuned
For #1:
On the receive path, the stack calls ptp_convert_timestamp() if the socket has the SOF_TIMESTAMPING_RAW_HARDWARE option. In that method, you need only get the raw cycle count if supported by the pclock.
So instead of:
vclock = info_to_vclock(ptp->info); ns = ktime_to_ns(hwtstamps->hwtstamp); spin_lock_irqsave(&vclock->lock, flags); ns = timecounter_cyc2time(&vclock->tc, ns); spin_unlock_irqrestore(&vclock->lock, flags);
something like this:
vclock = info_to_vclock(ptp->info); cycles = pclock->ktime_to_cycles(hwtstamps->hwtstamp); spin_lock_irqsave(&vclock->lock, flags); ns = timecounter_cyc2time(&vclock->tc, cycles); spin_unlock_irqrestore(&vclock->lock, flags);
This new class method, ktime_to_cycles, can simply do ktime_to_ns() by default for all of the existing drivers, but your special driver can look up the hwtstamp in a cache of {hwtstamp, cycles} pairs.
(No need to bloat skbuff by another eight bytes!)
For #2:
Similarly, add a new class method, say, pclock.get_cycles that does
if (ptp->info->gettimex64) ptp->info->gettimex64(ptp->info, &ts, NULL); else ptp->info->gettime64(ptp->info, &ts);
by default, but in your driver will read the special counter.
Thanks,
Richard
Quoted 36 lines
Quoted 9 lines
Quoted 1 line
How can I cover my use case with the existing API? I had no idea so far.
Okay, so 2 PHCs doesn't help, but still all you need is:
1. a different method to convert time stamps to vclock time base
2. a different method for vclocks' gettime
So let me suggest a much smaller change to the phc/vclock api... stay tuned
For #1:
On the receive path, the stack calls
ptp_convert_timestamp()if the socket has theSOF_TIMESTAMPING_RAW_HARDWAREoption. In that method, you need only get the raw cycle count if supported by the pclock.So instead of:
vclock = info_to_vclock(ptp->info); ns = ktime_to_ns(hwtstamps->hwtstamp); spin_lock_irqsave(&vclock->lock, flags); ns = timecounter_cyc2time(&vclock->tc, ns); spin_unlock_irqrestore(&vclock->lock, flags);something like this:
vclock = info_to_vclock(ptp->info); cycles = pclock->ktime_to_cycles(hwtstamps->hwtstamp); spin_lock_irqsave(&vclock->lock, flags); ns = timecounter_cyc2time(&vclock->tc, cycles); spin_unlock_irqrestore(&vclock->lock, flags);
Quoted 3 lines
This new class method,
ktime_to_cycles, can simply doktime_to_ns()by default for all of the existing drivers, but your special driver can look up the hwtstamp in a cache of {hwtstamp, cycles} pairs.
ktime_to_cycles uses hwtstamp as key for the cache lookup. As long as the PHC is monotonic, the key is unique. If the time of the PHC is set, then the cache would be invalidated. I'm afraid that setting the PHC could lead to wrong or missing timestamps. Setting the PHC in hardware, timestamp generation in hardware, and cache invalidation in software would need to be synchronized somehow.
Practically the PHC should be set only once. It would be also ok to use vclocks for PTP after PHC has been set. So it should work. But it would not be the generic PHC/vclock user space interface anymore.
Quoted 1 line
(No need to bloat skbuff by another eight bytes!)
I understand that bloating skbuff shall be prevented. Actually I need to find a way
to generate the correct timestamp within ptp_convert_timestamp and without bloating skbuff. The cache is one possibility. What do you think about the following idea?
For TX it is known which timestamp is required. So I would have to find a way to detect which timestamp shall be filled into hwtstamp.
For RX both timestamps are already available within skbuff, because they are stored in front of the Ethernet header by the hardware. So I have to find a way to detect the RX case and copy the right timestamp to hwtstamp.
This would prevent the cache and does not bloat skbuff.
Quoted 10 lines
For #2:
Similarly, add a new class method, say, pclock.get_cycles that does
if (ptp->info->gettimex64) ptp->info->gettimex64(ptp->info, &ts, NULL); else ptp->info->gettime64(ptp->info, &ts);by default, but in your driver will read the special counter.
Looks better than my getfreeruntimex64.
Thanks for your suggestion!
Gerhard
Richard Cochran
maintainerQuoted 4 lines · Gerhard Engleder
ktime_to_cyclesuses hwtstamp as key for the cache lookup. As long as the PHC is monotonic, the key is unique. If the time of the PHC is set, then the cache would be invalidated. I'm afraid that setting the PHC could lead to wrong or missing timestamps.
"Wrong" timestamps happen with normal PHCs anyhow. Not a big deal. Your driver can drop the cache and force racing vclock frames to omit the Rx timestamp. User space must deal with this in any case.
Quoted 3 lines
For RX both timestamps are already available within skbuff, because they are stored in front of the Ethernet header by the hardware. So I have to find a way to detect the RX case and copy the right timestamp to hwtstamp.
This works for your driver, but not for the generic vclock. Maybe you could let the ptp_convert_timestamp() method pass the skb back into your driver.
Thanks,
Richard
Richard Cochran
maintainerQuoted 6 lines · Gerhard Engleder
ktime_to_cyclesuses hwtstamp as key for the cache lookup. As long as the PHC is monotonic, the key is unique. If the time of the PHC is set, then the cache would be invalidated. I'm afraid that setting the PHC could lead to wrong or missing timestamps. Setting the PHC in hardware, timestamp generation in hardware, and cache invalidation in software would need to be synchronized somehow.
You can avoid errors even with a time jump:
Make a variant (union) of skb_shared_hwtstamps to allow driver to provide an address or cookie instead of ktime_t. Set a flag in the skbuff to signal this.
Let the Rx path check the flag and fetch the time stamp by callback with the address/cookie.
Thanks,
Richard
Quoted 13 lines
Quoted 6 lines
ktime_to_cyclesuses hwtstamp as key for the cache lookup. As long as the PHC is monotonic, the key is unique. If the time of the PHC is set, then the cache would be invalidated. I'm afraid that setting the PHC could lead to wrong or missing timestamps. Setting the PHC in hardware, timestamp generation in hardware, and cache invalidation in software would need to be synchronized somehow.You can avoid errors even with a time jump:
Make a variant (union) of
skb_shared_hwtstampsto allow driver to provide an address or cookie instead ofktime_t. Set a flag in the skbuff to signal this.
Quoted 7 lines
Let the Rx path check the flag and fetch the time stamp by callback with the address/cookie.
Quoted 2 lines
For TX it is known which timestamp is required. So I would have to find a way to detect which timestamp shall be filled into hwtstamp.
How about
tx_flagsin structskb_shared_info?
I will try to make an implementation!
Shall I use tx_flags in struct skb_shared_info for both, TX and RX? Or should I use flags in struct skb_shared_info for address/cookie signalisation?
3 of 8 flags are unused in tx_flags. It may be possible to use the same flag for TX and RX. Is it worth it trying to save flags?
For TX it signals "fill cycle based timestamp" and could be called SKBTX_HW_CSTAMP.
For RX it signals "hwtstamp is an address/cookie" and could be called SKBFL_TSTAMP_COOKIE.
Thank you!
Gerhard
Richard Cochran
maintainerQuoted 2 lines · Gerhard Engleder
3 of 8 flags are unused in
tx_flags. It may be possible to use the same flag for TX and RX. Is it worth it trying to save flags?
Yes, every bit is precious.
Thanks,
Richard
Richard Cochran
maintainerQuoted 2 lines · Gerhard Engleder
For TX it is known which timestamp is required. So I would have to find a way to detect which timestamp shall be filled into hwtstamp.
How about tx_flags in struct skb_shared_info ?
Thanks,
Richard
Hi,
Quoted 32 lines
ptp vclocks require a clock with free running time for the timecounter. Currently only a physical clock forced to free running is supported. If vclocks are used, then the physical clock cannot be synchronized anymore. The synchronized time is not available in hardware in this case. As a result, timed transmission with ETF/TAPRIO hardware support is not possible anymore.
If hardware would support a free running time additionally to the physical clock, then the physical clock does not need to be forced to free running. Thus, the physical clocks can still be synchronized while vclocks are in use.
The physical clock could be used to synchronize the time domain of the TSN network and trigger ETF/TAPRIO. In parallel vclocks can be used to synchronize other time domains.
One year ago I thought for two time domains within a TSN network also two physical clocks are required. This would lead to new kernel interfaces for asking for the second clock, ... . But actually for a time triggered system like TSN there can be only one time domain that controls the system itself. All other time domains belong to other layers, but not to the time triggered system itself. So other time domains can be based on a free running counter if similar mechanisms to 2 step synchronisation are used.
Synchronisation was tested with two time domains between two directly connected hosts. Each host run two ptp4l instances, the first used the physical clock and the second used the virtual clock. I used my FPGA based network controller as network device. ptp4l was used in combination with the virtual clock support patch set from Miroslav Lichvar.
Quoted 2 lines
Please give me some feedback. For me it seems like a straight forward extension for ptp vclocks, but I'm new to this topic.
From what I understand, you have a second PHC which is just a second PHC you cannot control; i.e. it is equivalent to a PHC in free running mode. This PHC will take the timestamps for the PTP frames. You can create multiple vclocks and you can use ptp4l to synchronize these.
The first (controlable) PHC is used to do the Qbv scheduling, thus needs a synchronized time.
How do you synchronize the vclock with this PHC? And how precise is it? I know that some cards can do cross timestamping in hardware to aid the synchronization (but I think that is not supported right now in linux).
Quoted 2 lines · Richard Cochran
You are adding eight bytes per frame for what is arguably an extreme niche case.
I don't think it is a niche case, btw. I was always wondering why NXP introduced the vclock thingy. And apparently it is for 802.1AS-rev, but one use case for that is 802.1Qbv and there you'll need a (synchronized) hardware clock to control the gates. So while we can have multiple time domain support with the vclock, we cannot use Qbv with them. That was something I have always wondered about. Or.. maybe I'm missing something here.
-michael
Richard Cochran
maintainerQuoted 11 lines · Michael Walle
Quoted 2 lines · Richard Cochran
You are adding eight bytes per frame for what is arguably an extreme niche case.
I don't think it is a niche case, btw. I was always wondering why NXP introduced the vclock thingy. And apparently it is for 802.1AS-rev, but one use case for that is 802.1Qbv and there you'll need a (synchronized) hardware clock to control the gates. So while we can have multiple time domain support with the vclock, we cannot use Qbv with them. That was something I have always wondered about. Or.. maybe I'm missing something here.
Niche is relative.
Believe it or not, the overwhelmingly great majority of people using Linux have no interest in TSN.
Thanks,
Richard
Quoted 7 lines · Richard Cochran
Quoted 6 lines · Michael Walle
Quoted 2 lines · Richard Cochran
You are adding eight bytes per frame for what is arguably an extreme niche case.
I don't think it is a niche case, btw. I was always wondering why NXP introduced the vclock thingy. And apparently it is for
...
Quoted 5 lines
Niche is relative.
Believe it or not, the overwhelmingly great majority of people using Linux have no interest in TSN.
Is this not the same issue as what was recently discussed about some devices being able to provide two (e.g. PHY+MAC) or even more timestamps at the same time?
There is a need to have multiple PHCs per device and for that to work the drivers need to be able to save multiple timestamps per packet.
In this series it is an additional freerunning clock. That seems too specific. I think we need a more general approach that will support two and more physical PHCs per device. Virtual clocks are not involved here.
Signature · 1 line
Miroslav Lichvar
Richard Cochran
maintainerQuoted 2 lines · Miroslav Lichvar
There is a need to have multiple PHCs per device and for that to work the drivers need to be able to save multiple timestamps per packet.
That is a big job that entails a new user API.
For the present, this device can be supported with only minimal changes in the PHC class layer. See my other reply.
Thanks,
Richard
Richard Cochran
maintainerQuoted 6 lines · Richard Cochran
Quoted 2 lines · Miroslav Lichvar
There is a need to have multiple PHCs per device and for that to work the drivers need to be able to save multiple timestamps per packet.
That is a big job that entails a new user API.
On second thought, maybe the address/cookie idea will at least open up multiple driver timestamps in a practical way...
Thanks,
Richard
Quoted 5 lines
From what I understand, you have a second PHC which is just a second PHC you cannot control; i.e. it is equivalent to a PHC in free running mode. This PHC will take the timestamps for the PTP frames. You can create multiple vclocks and you can use ptp4l to synchronize these.
Quoted 7 lines
The first (controlable) PHC is used to do the Qbv scheduling, thus needs a synchronized time.
How do you synchronize the vclock with this PHC? And how precise is it? I know that some cards can do cross timestamping in hardware to aid the synchronization (but I think that is not supported right now in linux).
There is no need to synchronize the first (controlable) PHC with the vclock. A ptp4l instance is running and synchronizing the time for Qbv/TAPRIO. vclocks are used for other time domains, which do not affect Qbv/TAPRIO.
Quoted 9 lines
Quoted 2 lines
You are adding eight bytes per frame for what is arguably an extreme niche case.
I don't think it is a niche case, btw. I was always wondering why NXP introduced the vclock thingy. And apparently it is for 802.1AS-rev, but one use case for that is 802.1Qbv and there you'll need a (synchronized) hardware clock to control the gates. So while we can have multiple time domain support with the vclock, we cannot use Qbv with them. That was something I have always wondered about.
I agree that most people using Linux have no interest in TSN. For the few people who are interested in TSN, I assume using two time domains in combination with Qbv/TAPRIO is a common goal. Is there anyone else who wants to use two time domains in combination with Qbv/TAPRIO?
Gerhard