zfcp: retrieve local RDP data, fix and cleanup
Subsystems (4)
v2 cover zfcp: retrieve local RDP data, fix and cleanup
Hello all,
this is the second version of my RDP patchset for zfcp, after I noticed a memory-leak in the first version earlier this year. Here is the original description, which remains valid:
With FC-LS-3 Fibre Channel got support for the RDP Extended Link Service, which we can use to gather diagnostics data (health values from the optics and such). To gather those data from our local link, the FCP Channel firmware (for FCP and Ficon Express adapters) added some new fields in our Exchange Port Data firmware command (primarily SFP data).
This patchset adds a new sysfs-directory below our
adapter-object and several files in that to read those new fields.
Additionally this also adds a file to read the buffer-to-buffer
credit value from our local port (also something that can be gathered using the RDP ELS).
The main purpose of the data from RDP requests (at least in my mind) are to catch some otherwise hard to track down errors, like dirt on the fibre and such (I would expect optics data to degraded with such conditions). These things are a pain to track down because it doesn't cause the link to go away, just some sporadic errors that can result in incomplete exchanges and such.
The patchset is somewhat big because I don't want to issue a single Exchange Port/Config Data command for each read of those files, which would generate unnecessary traffic (and possibly inconsistent data) on our command-queue (we don't have an admin-queue or some such). So it also adds a (time-based) mechanism to cache data from Exchange Config/Port Data.
We also want to prevent our adapter-structure from growing,
which would further grow Scsi_Host, which is already too damn big, so these data are decoupled.
More details are in the patches themselves.
Additionally to this, there are two more patches from Steffen that we want to include, the first (patch 10) is just a cleanup for better readability, the second (patch 11) is a fix/improvement for our tracing-feature to make traces of failed FCP requests better. Even though Steffen flagged the later as fix, I don't think we need extra handling for it, if it goes in with the rest, thats fine - its nothing pressing.
Reviews are highly welcome :-).
James, Martin, if there are no objections, it would be nice if you could still include this in the merge for 5.5.
FYI: I also plan to/am in the middle of writing a simple command-line
tool to send RDP to all/certain available remote/local FC-Ports in the System and present them in a sane format (sane to me ;-) ). This could be used for logging, or while gathering debug-data from a system.
The RDP ELS' to remote ports will be send using BSG. Local data (for zFCP) would be read using this new interface.
If anyone is interested, I can point it out on the list once I have something that has at least a minimum set of functionality.
Benjamin Block (9):
zfcp: signal incomplete or error for sync exchange config/port data
zfcp: diagnostics buffer caching and use for exchange port data
zfcp: add diagnostics buffer for exchange config data
zfcp: support retrieval of SFP Data via Exchange Port Data
zfcp: introduce SysFS interface for diagnostics of local SFP
transceiver
zfcp: implicitly refresh port-data diagnostics when reading SysFS
zfcp: introduce SysFS interface to read the local B2B-Credit
zfcp: implicitly refresh config-data diagnostics when reading SysFS
zfcp: move maximum age of diagnostic buffers into a per-adapter
variable
Steffen Maier (2):
zfcp: proper indentation to reduce confusion in zfcp_erp_required_act
zfcp: trace channel log even for FCP command responses
drivers/s390/scsi/Makefile | 2 +- drivers/s390/scsi/zfcp_aux.c | 12 +- drivers/s390/scsi/zfcp_dbf.c | 8 +- drivers/s390/scsi/zfcp_def.h | 4 +- drivers/s390/scsi/zfcp_diag.c | 305 +++++++++++++++++++++++++++++++++ drivers/s390/scsi/zfcp_diag.h | 101 +++++++++++ drivers/s390/scsi/zfcp_erp.c | 4 +- drivers/s390/scsi/zfcp_ext.h | 1 + drivers/s390/scsi/zfcp_fsf.c | 73 +++++++- drivers/s390/scsi/zfcp_fsf.h | 21 ++- drivers/s390/scsi/zfcp_scsi.c | 4 +- drivers/s390/scsi/zfcp_sysfs.c | 170 +++++++++++++++++- 12 files changed, 683 insertions(+), 22 deletions(-) create mode 100644 drivers/s390/scsi/zfcp_diag.c create mode 100644 drivers/s390/scsi/zfcp_diag.h
Signature · 1 line
2.21.0
PATCH v2 1/11 zfcp: signal incomplete or error for sync exchange config/port data
Adds a new FSF-Request status flag (ZFCP_STATUS_FSFREQ_XDATAINCOMPLETE) that signal that the data received using Exchange Config Data or Exchange Port Data was incomplete. This new flags is set in the respective handlers during the response path.
With this patch, only the synchronous FSF-functions for each command got support for the new flag, otherwise it is transparent.
Together with this new flag and already existing status flags the synchronous FSF-functions are extended to now detect whether the received data is complete, incomplete or completely invalid (this includes cases where a command ran into a timeout). This is now signaled back to the caller, where previously only failures on the request path would result in a bad return-code.
For complete data the return-code remains 0. For incomplete data a new return-code -EAGAIN is added to the function-interface. For completely invalid data the already existing return-code -EIO is reused - formerly this was used to signal failures on the request path.
Existing callers of the FSF-functions are adjusted so that they behave as before for return-code 0 and -EAGAIN, to not change the user-interface. As -EIO existed all along, it was already exposed to the user - and needed handling - and will now also be exposed in this new special case.
--- drivers/s390/scsi/zfcp_def.h | 1 + drivers/s390/scsi/zfcp_fsf.c | 46 ++++++++++++++++++++++++++++++---- drivers/s390/scsi/zfcp_scsi.c | 4 +-- drivers/s390/scsi/zfcp_sysfs.c | 4 +-- 4 files changed, 46 insertions(+), 9 deletions(-)
drivers/s390/scsi/zfcp_def.h +1 -0
@@ -86,6 +86,7 @@#define ZFCP_STATUS_FSFREQ_ABORTNOTNEEDED 0x00000080#define ZFCP_STATUS_FSFREQ_TMFUNCFAILED 0x00000200#define ZFCP_STATUS_FSFREQ_DISMISSED 0x00001000 +#define ZFCP_STATUS_FSFREQ_XDATAINCOMPLETE 0x00020000/************************* STRUCTURE DEFINITIONS *****************************/
drivers/s390/scsi/zfcp_fsf.c +41 -5
88 lines not on this page · raw message
drivers/s390/scsi/zfcp_scsi.c +2 -2
@@ -605,7 +605,7 @@ zfcp_scsi_get_fc_host_stats(struct Scsi_Host *host)return NULL ; ret =zfcp_fsf_exchange_port_data_sync (adapter->qdio, data); -if (ret) { +if (ret !=0 && ret != -EAGAIN ) {kfree (data);return NULL ; }
@@ -634,7 +634,7 @@ static void zfcp_scsi_reset_fc_host_stats(struct Scsi_Host *shost)return ; ret =zfcp_fsf_exchange_port_data_sync (adapter->qdio, data); -if (ret) +if (ret !=0 && ret != -EAGAIN )kfree (data);else { adapter->stats_reset = jiffies/HZ ;
drivers/s390/scsi/zfcp_sysfs.c +2 -2
@@ -577,7 +577,7 @@ static ssize_t zfcp_sysfs_adapter_util_show(struct device *dev,return -ENOMEM ; retval =zfcp_fsf_exchange_port_data_sync (adapter->qdio, qtcb_port); -if (!retval) +if (retval ==0 || retval == -EAGAIN ) retval =sprintf (buf,"%u %u %u\n" , qtcb_port->cp_util, qtcb_port->cb_util, qtcb_port->a_util);kfree (qtcb_port);
@@ -603,7 +603,7 @@ static int zfcp_sysfs_adapter_ex_config(struct device *dev,return -ENOMEM ; retval =zfcp_fsf_exchange_config_data_sync (adapter->qdio, qtcb_config); -if (!retval) +if (retval ==0 || retval == -EAGAIN ) *stat_inf = qtcb_config->stat_info;kfree (qtcb_config);
Signature · 1 line
2.21.0
PATCH v2 2/11 zfcp: diagnostics buffer caching and use for exchange port data
In reply to Benjamin BlockThe FCP channel exposes two central interfaces to receive information about the local FCP-Adapter/-Port: Exchange Port and Exchange Config Data. Using these commands can negatively impact the adapter if we allow them to be sent at a very high rate.
The later parts of this patchset will introduce new user-interfaces to receive more diagnostics from the adapter. To prevent any negative impact from using those, this patch adds a simple caching-mechanism that will prevent a malicious/faulty userspace-application from generating an abnormal high amount of Exchange Port/Config Data traffic.
Relevant diagnostic data that is received via Exchange Config/Port Data is cached in buffers associated with the corresponding adapter-struct. Each buffer is associated with a timestamp that signals how old the data is, and, added via a following patch in this series, lets userspace-interfaces determine when the data is too old and needs to be updated.
Buffer-updates are made during the normal response path of the corresponding command. With this patch only the output of the Exchange Port Data command is captured.
--- drivers/s390/scsi/Makefile | 2 +- drivers/s390/scsi/zfcp_aux.c | 8 ++- drivers/s390/scsi/zfcp_def.h | 3 +- drivers/s390/scsi/zfcp_diag.c | 93 +++++++++++++++++++++++++++++++++++ drivers/s390/scsi/zfcp_diag.h | 60 ++++++++++++++++++++++ drivers/s390/scsi/zfcp_fsf.c | 12 +++++ 6 files changed, 175 insertions(+), 3 deletions(-) create mode 100644 drivers/s390/scsi/zfcp_diag.c create mode 100644 drivers/s390/scsi/zfcp_diag.h
drivers/s390/scsi/Makefile +1 -1
@@ -5,6 +5,6 @@ zfcp-objs := zfcp_aux.o zfcp_ccw.o zfcp_dbf.o zfcp_erp.o \ zfcp_fc.o zfcp_fsf.o zfcp_qdio.o zfcp_scsi.o zfcp_sysfs.o \ - zfcp_unit.o + zfcp_unit.o zfcp_diag.o obj-$(CONFIG_ZFCP) += zfcp.o
drivers/s390/scsi/zfcp_aux.c +7 -1
@@ -4,7 +4,7 @@* * Module interface and handling of zfcp data structures. * -* Copyright IBM Corp. 2002, 2017 +* Copyright IBM Corp. 2002, 2018 */ /*
@@ -25,6 +25,7 @@* Martin Petermann * Sven Schuetz * Steffen Maier +* Benjamin Block */ #define KMSG_COMPONENT "zfcp"
@@ -36,6 +37,7 @@#include "zfcp_ext.h" #include "zfcp_fc.h" #include "zfcp_reqlist.h" +#include "zfcp_diag.h" #define ZFCP_BUS_ID_SIZE 20
@@ -356,6 +358,9 @@ struct zfcp_adapter *zfcp_adapter_enqueue(struct ccw_device *ccw_device) adapter->erp_action.adapter = adapter; +if (zfcp_diag_adapter_setup (adapter)) + goto failed; +if (zfcp_qdio_setup (adapter)) goto failed;
@@ -449,6 +454,7 @@ void zfcp_adapter_release(struct kref *ref)dev_set_drvdata (&adapter->ccw_device->dev,NULL );zfcp_fc_gs_destroy (adapter);zfcp_free_low_mem_buffers (adapter); +zfcp_diag_adapter_free (adapter);kfree (adapter->req_list);kfree (adapter->fc_stats);kfree (adapter->stats_reset_data);
drivers/s390/scsi/zfcp_def.h +2 -1
@@ -4,7 +4,7 @@* * Global definitions for the zfcp device driver. * -* Copyright IBM Corp. 2002, 2017 +* Copyright IBM Corp. 2002, 2018 */ #ifndef ZFCP_DEF_H
@@ -198,6 +198,7 @@ struct zfcp_adapter {struct device_dma_parameters dma_parms;struct zfcp_fc_events events;unsigned long next_port_scan; +struct zfcp_diag_adapter *diagnostics; };struct zfcp_port {
drivers/s390/scsi/zfcp_diag.c new file +93 -0
93 lines not on this page · raw message
drivers/s390/scsi/zfcp_diag.h new file +60 -0
60 lines not on this page · raw message
drivers/s390/scsi/zfcp_fsf.c +12 -0
@@ -11,6 +11,7 @@#define pr_fmt (fmt) KMSG_COMPONENT ": " fmt#include <linux/blktrace_api.h> +#include <linux/jiffies.h> #include <linux/types.h> #include <linux/slab.h> #include <scsi/fc/fc_els.h>
@@ -19,6 +20,7 @@#include "zfcp_dbf.h" #include "zfcp_qdio.h" #include "zfcp_reqlist.h" +#include "zfcp_diag.h" /* timeout for FSF requests sent during scsi_eh: abort or FCP TMF */ #define ZFCP_FSF_SCSI_ER_TIMEOUT (10*HZ)
@@ -655,16 +657,26 @@ static void zfcp_fsf_exchange_port_evaluate(struct zfcp_fsf_req *req)static void zfcp_fsf_exchange_port_data_handler (struct zfcp_fsf_req *req) { +struct zfcp_diag_header *const diag_hdr = + &req->adapter->diagnostics->port_data.header;struct fsf_qtcb *qtcb = req->qtcb; +struct fsf_qtcb_bottom_port *bottom = &qtcb->bottom.port;if (req->status &ZFCP_STATUS_FSFREQ_ERROR )return ;switch (qtcb->header.fsf_status) {case FSF_GOOD : +/* +* usually we wait with an update till the cache is too old, +* but because we have the data available, update it anyway +*/ +zfcp_diag_update_xdata (diag_hdr, bottom, false); +zfcp_fsf_exchange_port_evaluate (req);break ;case FSF_EXCHANGE_CONFIG_DATA_INCOMPLETE : +zfcp_diag_update_xdata (diag_hdr, bottom, true); req->status |=ZFCP_STATUS_FSFREQ_XDATAINCOMPLETE ;zfcp_fsf_exchange_port_evaluate (req);
Signature · 1 line
2.21.0
PATCH v2 3/11 zfcp: add diagnostics buffer for exchange config data
In reply to Benjamin BlockIn the same vein as the previous patch, add diagnostic data capture for the Exchange Config Data command.
--- drivers/s390/scsi/zfcp_diag.c | 14 ++++++++++++-- drivers/s390/scsi/zfcp_diag.h | 7 +++++++ drivers/s390/scsi/zfcp_fsf.c | 9 +++++++++ 3 files changed, 28 insertions(+), 2 deletions(-)
drivers/s390/scsi/zfcp_diag.c +12 -2
@@ -34,6 +34,9 @@*/ int zfcp_diag_adapter_setup (struct zfcp_adapter *const adapter) { +/* set the timestamp so that the first test on age will always fail */ +const unsigned long initial_timestamp = + jiffies -msecs_to_jiffies (ZFCP_DIAG_MAX_AGE );struct zfcp_diag_adapter *diag;struct zfcp_diag_header *hdr;
@@ -47,8 +50,15 @@ int zfcp_diag_adapter_setup(struct zfcp_adapter *const adapter)spin_lock_init (&hdr->access_lock); hdr->buffer = &diag->port_data.data; hdr->buffer_size =sizeof (diag->port_data.data); -/* set the timestamp so that the first test on age will always fail */ - hdr->timestamp = jiffies -msecs_to_jiffies (ZFCP_DIAG_MAX_AGE ); + hdr->timestamp = initial_timestamp; + +/* setup header for config_data */ + hdr = &diag->config_data.header; + +spin_lock_init (&hdr->access_lock); + hdr->buffer = &diag->config_data.data; + hdr->buffer_size =sizeof (diag->config_data.data); + hdr->timestamp = initial_timestamp; adapter->diagnostics = diag;return 0 ;
drivers/s390/scsi/zfcp_diag.h +7 -0
@@ -43,12 +43,19 @@ struct zfcp_diag_header {* @port_data: data retrieved using exchange port data. * @port_data.header: header with metadata for the cache in @port_data.data. * @port_data.data: cached QTCB Bottom of command exchange port data. +* @config_data: data retrieved using exchange config data. +* @config_data.header: header with metadata for the cache in @config_data.data. +* @config_data.data: cached QTCB Bottom of command exchange config data. */ struct zfcp_diag_adapter {struct {struct zfcp_diag_header header;struct fsf_qtcb_bottom_port data; } port_data; +struct { +struct zfcp_diag_header header; +struct fsf_qtcb_bottom_config data; + } config_data; };int zfcp_diag_adapter_setup (struct zfcp_adapter *const adapter);
drivers/s390/scsi/zfcp_fsf.c +9 -0
@@ -556,6 +556,8 @@ static int zfcp_fsf_exchange_config_evaluate(struct zfcp_fsf_req *req)static void zfcp_fsf_exchange_config_data_handler (struct zfcp_fsf_req *req) {struct zfcp_adapter *adapter = req->adapter; +struct zfcp_diag_header *const diag_hdr = + &adapter->diagnostics->config_data.header;struct fsf_qtcb *qtcb = req->qtcb;struct fsf_qtcb_bottom_config *bottom = &qtcb->bottom.config;struct Scsi_Host *shost = adapter->scsi_host;
@@ -572,6 +574,12 @@ static void zfcp_fsf_exchange_config_data_handler(struct zfcp_fsf_req *req)switch (qtcb->header.fsf_status) {case FSF_GOOD : +/* +* usually we wait with an update till the cache is too old, +* but because we have the data available, update it anyway +*/ +zfcp_diag_update_xdata (diag_hdr, bottom, false); +if (zfcp_fsf_exchange_config_evaluate (req))return ;
@@ -587,6 +595,7 @@ static void zfcp_fsf_exchange_config_data_handler(struct zfcp_fsf_req *req) &adapter->status);break ;case FSF_EXCHANGE_CONFIG_DATA_INCOMPLETE : +zfcp_diag_update_xdata (diag_hdr, bottom, true); req->status |=ZFCP_STATUS_FSFREQ_XDATAINCOMPLETE ;fc_host_node_name (shost) =0 ;
Signature · 1 line
2.21.0
PATCH v2 4/11 zfcp: support retrieval of SFP Data via Exchange Port Data
In reply to Benjamin BlockA new FCP channel feature allows us to read the diagnostics from our local SFP transceivers. To make use of that add a flag (FSF_FEATURE_REQUEST_SFP_DATA) to the feature-set we request from the FCP channel. Whether the channel actually implements this can be determined via an other new flag (FSF_FEATURE_REPORT_SFP_DATA), that is set in the adapter_features field of the adapter structure after Exchange Config Data finished.
Also add the corresponding definitions in the QTCB Bottom for Exchange Port Data. These new definitions are only valid, if FSF_FEATURE_REPORT_SFP_DATA is set.
--- drivers/s390/scsi/zfcp_fsf.c | 6 ++++-- drivers/s390/scsi/zfcp_fsf.h | 21 ++++++++++++++++++++- 2 files changed, 24 insertions(+), 3 deletions(-)
drivers/s390/scsi/zfcp_fsf.c +4 -2
@@ -1286,7 +1286,8 @@ int zfcp_fsf_exchange_config_data(struct zfcp_erp_action *erp_action) req->qtcb->bottom.config.feature_selection =FSF_FEATURE_NOTIFICATION_LOST | -FSF_FEATURE_UPDATE_ALERT ; +FSF_FEATURE_UPDATE_ALERT | +FSF_FEATURE_REQUEST_SFP_DATA ; req->erp_action = erp_action; req->handler = zfcp_fsf_exchange_config_data_handler; erp_action->fsf_req_id = req->req_id;
@@ -1339,7 +1340,8 @@ int zfcp_fsf_exchange_config_data_sync(struct zfcp_qdio *qdio, req->qtcb->bottom.config.feature_selection =FSF_FEATURE_NOTIFICATION_LOST | -FSF_FEATURE_UPDATE_ALERT ; +FSF_FEATURE_UPDATE_ALERT | +FSF_FEATURE_REQUEST_SFP_DATA ;if (data) req->data = data;
drivers/s390/scsi/zfcp_fsf.h +20 -1
@@ -163,6 +163,8 @@#define FSF_FEATURE_ELS_CT_CHAINED_SBALS 0x00000020#define FSF_FEATURE_UPDATE_ALERT 0x00000100#define FSF_FEATURE_MEASUREMENT_DATA 0x00000200 +#define FSF_FEATURE_REQUEST_SFP_DATA 0x00000200 +#define FSF_FEATURE_REPORT_SFP_DATA 0x00000800#define FSF_FEATURE_DIF_PROT_TYPE1 0x00010000#define FSF_FEATURE_DIX_PROT_TCPIP 0x00020000
@@ -407,7 +409,24 @@ struct fsf_qtcb_bottom_port {u8 cp_util;u8 cb_util;u8 a_util; -u8 res2[253 ]; +u8 res2; +u16 temperature; +u16 vcc; +u16 tx_bias; +u16 tx_power; +u16 rx_power; +union { +u16 raw; +struct { +u16 fec_active :1 ; + u16:7; +u16 connector_type :2 ; +u16 sfp_invalid :1 ; +u16 optical_port :1 ; +u16 port_tx_type :4 ; + }; + } sfp_flags; +u8 res3[240 ]; } __attribute__ ((packed));union fsf_qtcb_bottom {
Signature · 1 line
2.21.0
PATCH v2 5/11 zfcp: introduce SysFS interface for diagnostics of local SFP transceiver
In reply to Benjamin BlockThis adds an interface to read the diagnostics of the local SFP transceiver of an FCP-Channel from userspace. This comes in the form of new SysFS entries that are attached to the CCW device representing the FCP device. Each type of data gets its own SysFS entry; the whole collection of entries is pooled into a new child-directory of the CCW device node: "diagnostics".
Adds SysFS entries for:
* sfp_invalid: boolean value evaluating to whether the following 5
fields are invalid; {0, 1}; 1 - invalid
* temperature: transceiver temp.; unit 1/256°C;
range [-128°C, +128°C]
* vcc: supply voltage; unit 100μV; range [0, 6.55V]
* tx_bias: transmitter laser bias current; unit 2μA;
range [0, 131mA] * tx_power: coupled TX output power; unit 0.1μW; range [0, 6.5mW]
* rx_power: received optical power; unit 0.1μW; range [0, 6.5mW]
* optical_port: boolean value evaluating to whether the FCP-Channel has
an optical port; {0, 1}; 1 - optical
* fec_active: boolean value evaluating to whether 16G FEC is active;
{0, 1}; 1 - active * port_tx_type: nibble describing the port type; {0, 1, 2, 3};
0 - unknown, 1 - short wave,
2 - long wave LC 1310nm, 3 - long wave LL 1550nm
* connector_type: two bits describing the connector type; {0, 1};
0 - unknown, 1 - SFP+
This is only supported if the FCP-Channel in turn supports reporting the SFP Diagnostic Data, otherwise read() on these new entries will return EOPNOTSUPP (this affects only adapters older than FICON Express8S, on Mainframe generations older than z14). Other possible errors for read() include ENOLINK, ENODEV and ENOMEM.
With this patch the userspace-interface will only read data stored in the corresponding "diagnostic buffer" (that was stored during completion of an previous Exchange Port Data command). Implicit updating will follow later in this series.
--- drivers/s390/scsi/zfcp_aux.c | 4 ++ drivers/s390/scsi/zfcp_diag.c | 42 ++++++++++++++++++++ drivers/s390/scsi/zfcp_diag.h | 18 +++++++++ drivers/s390/scsi/zfcp_ext.h | 1 + drivers/s390/scsi/zfcp_sysfs.c | 71 ++++++++++++++++++++++++++++++++++ 5 files changed, 136 insertions(+)
drivers/s390/scsi/zfcp_aux.c +4 -0
@@ -407,6 +407,9 @@ struct zfcp_adapter *zfcp_adapter_enqueue(struct ccw_device *ccw_device) &zfcp_sysfs_adapter_attrs)) goto failed; +if (zfcp_diag_sysfs_setup (adapter)) + goto failed; +/* report size limit per scatter-gather segment */ adapter->ccw_device->dev.dma_parms = &adapter->dma_parms;
@@ -431,6 +434,7 @@ void zfcp_adapter_unregister(struct zfcp_adapter *adapter)zfcp_fc_wka_ports_force_offline (adapter->gs);zfcp_scsi_adapter_unregister (adapter); +zfcp_diag_sysfs_destroy (adapter);sysfs_remove_group (&cdev->dev.kobj, &zfcp_sysfs_adapter_attrs);zfcp_erp_thread_kill (adapter);
drivers/s390/scsi/zfcp_diag.c +42 -0
54 lines not on this page · raw message
drivers/s390/scsi/zfcp_diag.h +18 -0
@@ -40,6 +40,8 @@ struct zfcp_diag_header {/** * struct zfcp_diag_adapter - central storage for all diagnostics concerning an * adapter. +* @sysfs_established: flag showing that the associated sysfs-group was created +* during run of zfcp_adapter_enqueue(). * @port_data: data retrieved using exchange port data. * @port_data.header: header with metadata for the cache in @port_data.data. * @port_data.data: cached QTCB Bottom of command exchange port data.
@@ -48,6 +50,8 @@ struct zfcp_diag_header {* @config_data.data: cached QTCB Bottom of command exchange config data. */ struct zfcp_diag_adapter { +u64 sysfs_established :1 ; +struct {struct zfcp_diag_header header;struct fsf_qtcb_bottom_port data;
@@ -61,7 +65,21 @@ struct zfcp_diag_adapter {int zfcp_diag_adapter_setup (struct zfcp_adapter *const adapter);void zfcp_diag_adapter_free (struct zfcp_adapter *const adapter); +int zfcp_diag_sysfs_setup (struct zfcp_adapter *const adapter); +void zfcp_diag_sysfs_destroy (struct zfcp_adapter *const adapter); +void zfcp_diag_update_xdata (struct zfcp_diag_header *const hdr,const void *const data,const bool incomplete); +/** +* zfcp_diag_support_sfp() - Return %true if the @adapter supports reporting +* SFP Data. +* @adapter: adapter to test the availability of SFP Data reporting for. +*/ +static inline bool +zfcp_diag_support_sfp (const struct zfcp_adapter *const adapter) +{ +return !!(adapter->adapter_features &FSF_FEATURE_REPORT_SFP_DATA ); +} +#endif /* ZFCP_DIAG_H */
drivers/s390/scsi/zfcp_ext.h +1 -0
@@ -167,6 +167,7 @@ extern const struct attribute_group *zfcp_port_attr_groups[];extern struct mutex zfcp_sysfs_port_units_mutex;extern struct device_attribute *zfcp_sysfs_sdev_attrs[];extern struct device_attribute *zfcp_sysfs_shost_attrs[]; +extern const struct attribute_group zfcp_sysfs_diag_attr_group;bool zfcp_sysfs_port_is_removing (const struct zfcp_port *const port);/* zfcp_unit.c */
drivers/s390/scsi/zfcp_sysfs.c +71 -0
80 lines not on this page · raw message
Signature · 1 line
2.21.0
PATCH v2 6/11 zfcp: implicitly refresh port-data diagnostics when reading SysFS
In reply to Benjamin BlockThis patch adds implicit updates to the SysFS entries that read the diagnostic data stored in the "caching buffer" for Exchange Port Data.
An update is triggered once the buffer is older than ZFCP_DIAG_MAX_AGE milliseconds (5s). This entails sending an Exchange Port Data command to the FCP-Channel, and during its ingress path updating the cached data and the timestamp. To prevent multiple concurrent userspace-applications from triggering this update in parallel we synchronize all of them using a wait-queue (waiting threads are interruptible; the updating thread is not).
--- drivers/s390/scsi/zfcp_diag.c | 129 +++++++++++++++++++++++++++++++++ drivers/s390/scsi/zfcp_diag.h | 11 +++ drivers/s390/scsi/zfcp_sysfs.c | 5 ++ 3 files changed, 145 insertions(+)
drivers/s390/scsi/zfcp_diag.c +129 -0
@@ -22,6 +22,8 @@/* Max age of data in a diagnostics buffer before it needs a refresh (in ms). */ #define ZFCP_DIAG_MAX_AGE (5 * 1000) +static DECLARE_WAIT_QUEUE_HEAD (__zfcp_diag_publish_wait); +/** * zfcp_diag_adapter_setup() - Setup storage for adapter diagnostics. * @adapter: the adapter to setup diagnostics for.
@@ -143,3 +145,130 @@ void zfcp_diag_update_xdata(struct zfcp_diag_header *const hdr, out:spin_unlock_irqrestore (&hdr->access_lock, flags); } + +/** +* zfcp_diag_update_port_data_buffer() - Implementation of +* &typedef zfcp_diag_update_buffer_func +* to collect and update Port Data. +* @adapter: Adapter to collect Port Data from. +* +* This call is SYNCHRONOUS ! It blocks till the respective command has +* finished completely, or has failed in some way. +* +* Return: +* * 0 - Successfully retrieved new Diagnostics and Updated the buffer; +* this also includes cases where data was retrieved, but +* incomplete; you'll have to check the flag ``incomplete`` +* of &struct zfcp_diag_header. +* * see zfcp_fsf_exchange_port_data_sync() for possible error-codes ( +* excluding -EAGAIN) +*/ +int zfcp_diag_update_port_data_buffer (struct zfcp_adapter *const adapter) +{ +int rc; + + rc =zfcp_fsf_exchange_port_data_sync (adapter->qdio,NULL ); +if (rc == -EAGAIN ) + rc =0 ;/* signaling incomplete via struct zfcp_diag_header */ + +/* buffer-data was updated in zfcp_fsf_exchange_port_data_handler() */ + +return rc; +} + +static int __zfcp_diag_update_buffer (struct zfcp_adapter *const adapter, +struct zfcp_diag_header *const hdr, +zfcp_diag_update_buffer_func buffer_update, +unsigned long *const flags) + __must_hold(hdr->access_lock) +{ +int rc; + +if (hdr->updating ==1 ) { + rc =wait_event_interruptible_lock_irq (__zfcp_diag_publish_wait, + hdr->updating ==0 , + hdr->access_lock); + rc = (rc ==0 ? -EAGAIN : -EINTR ); + }else { + hdr->updating =1 ; +spin_unlock_irqrestore (&hdr->access_lock, *flags); + +/* unlocked, because update function sleeps */ + rc =buffer_update (adapter); + +spin_lock_irqsave (&hdr->access_lock, *flags); + hdr->updating =0 ; + +/* +* every thread waiting here went via an interruptible wait, +* so its fine to only wake those +*/ +wake_up_interruptible_all (&__zfcp_diag_publish_wait); + } + +return rc; +} + +static bool +__zfcp_diag_test_buffer_age_isfresh (const struct zfcp_diag_header *const hdr) + __must_hold(hdr->access_lock) +{ +const unsigned long now = jiffies; + +/* +* Should not happen (data is from the future).. if it does, still +* signal that it needs refresh +*/ +if (!time_after_eq (now, hdr->timestamp)) +return false; + +if (jiffies_to_msecs (now - hdr->timestamp) >=ZFCP_DIAG_MAX_AGE ) +return false; + +return true; +} + +/** +* zfcp_diag_update_buffer_limited() - Collect diagnostics and update a +* diagnostics buffer rate limited. +* @adapter: Adapter to collect the diagnostics from. +* @hdr: buffer-header for which to update with the collected diagnostics. +* @buffer_update: Specific implementation for collecting and updating. +* +* This function will cause an update of the given @hdr by calling the also +* given @buffer_update function. If called by multiple sources at the same +* time, it will synchornize the update by only allowing one source to call +* @buffer_update and the others to wait for that source to complete instead +* (the wait is interruptible). +* +* Additionally this version is rate-limited and will only exit if either the +* buffer is fresh enough (within the limit) - it will do nothing if the buffer +* is fresh enough to begin with -, or if the source/thread that started this +* update is the one that made the update (to prevent endless loops). +* +* Return: +* * 0 - If the update was successfully published and/or the buffer is +* fresh enough +* * -EINTR - If the thread went into the wait-state and was interrupted +* * whatever @buffer_update returns +*/ +int zfcp_diag_update_buffer_limited (struct zfcp_adapter *const adapter, +struct zfcp_diag_header *const hdr, +zfcp_diag_update_buffer_func buffer_update) +{ +unsigned long flags; +int rc; + +spin_lock_irqsave (&hdr->access_lock, flags); + +for (rc =0 ; !__zfcp_diag_test_buffer_age_isfresh (hdr); rc =0 ) { + rc =__zfcp_diag_update_buffer (adapter, hdr, buffer_update, + &flags); +if (rc != -EAGAIN ) +break ; + } + +spin_unlock_irqrestore (&hdr->access_lock, flags); + +return rc; +}
drivers/s390/scsi/zfcp_diag.h +11 -0
@@ -71,6 +71,17 @@ void zfcp_diag_sysfs_destroy(struct zfcp_adapter *const adapter);void zfcp_diag_update_xdata (struct zfcp_diag_header *const hdr,const void *const data,const bool incomplete); +/* +* Function-Type used in zfcp_diag_update_buffer_limited() for the function +* that does the buffer-implementation dependent work. +*/ +typedef int (*zfcp_diag_update_buffer_func )(struct zfcp_adapter *const adapter); + +int zfcp_diag_update_port_data_buffer (struct zfcp_adapter *const adapter); +int zfcp_diag_update_buffer_limited (struct zfcp_adapter *const adapter, +struct zfcp_diag_header *const hdr, +zfcp_diag_update_buffer_func buffer_update); +/** * zfcp_diag_support_sfp() - Return %true if the @adapter supports reporting * SFP Data.
drivers/s390/scsi/zfcp_sysfs.c +5 -0
@@ -693,6 +693,11 @@ struct device_attribute *zfcp_sysfs_shost_attrs[] = { \ diag_hdr = &adapter->diagnostics->port_data.header; \ \ + rc =zfcp_diag_update_buffer_limited ( \ + adapter, diag_hdr, zfcp_diag_update_port_data_buffer); \ +if (rc !=0 ) \ + goto out; \ + \spin_lock_irqsave (&diag_hdr->access_lock, flags); \ rc = scnprintf( \ buf, (_prtsize) + 2, _prtfmt "\n", \
Signature · 1 line
2.21.0
PATCH v2 7/11 zfcp: introduce SysFS interface to read the local B2B-Credit
In reply to Benjamin BlockIn addition to the diagnostic data from the local SFP transceiver this patch adds an interface to read the advertised buffer-to-buffer credit from the local FC_Port.
With this patch the userspace-interface will only read data stored in the corresponding "diagnostic buffer" (that was stored during completion of a previous Exchange Config Data command). Implicit updating will follow later in this series.
--- drivers/s390/scsi/zfcp_sysfs.c | 40 ++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+)
drivers/s390/scsi/zfcp_sysfs.c +40 -0
@@ -666,6 +666,45 @@ struct device_attribute *zfcp_sysfs_shost_attrs[] = {NULL }; +static ssize_t zfcp_sysfs_adapter_diag_b2b_credit_show ( +struct device *dev,struct device_attribute *attr,char *buf) +{ +struct zfcp_adapter *adapter =zfcp_ccw_adapter_by_cdev (to_ccwdev (dev)); +struct zfcp_diag_header *diag_hdr; +struct fc_els_flogi *nsp; +ssize_t rc = -ENOLINK ; +unsigned long flags; +unsigned int status; + +if (!adapter) +return -ENODEV ; + + status =atomic_read (&adapter->status); +if (0 == (status &ZFCP_STATUS_COMMON_OPEN ) || +0 == (status &ZFCP_STATUS_COMMON_UNBLOCKED ) || +0 != (status &ZFCP_STATUS_COMMON_ERP_FAILED )) + goto out; + + diag_hdr = &adapter->diagnostics->config_data.header; + +spin_lock_irqsave (&diag_hdr->access_lock, flags); +/* nport_serv_param doesn't contain the ELS_Command code */ + nsp = (struct fc_els_flogi *)((unsigned long ) + adapter->diagnostics->config_data + .data.nport_serv_param - +sizeof (u32)); + + rc =scnprintf (buf,5 +2 ,"%hu\n" , +be16_to_cpu (nsp->fl_csp.sp_bb_cred)); +spin_unlock_irqrestore (&diag_hdr->access_lock, flags); + +out: +zfcp_ccw_adapter_put (adapter); +return rc; +} +static ZFCP_DEV_ATTR(adapter_diag, b2b_credit, 0400, + zfcp_sysfs_adapter_diag_b2b_credit_show, NULL); +#define ZFCP_DEFINE_DIAG_SFP_ATTR (_name, _qtcb_member, _prtsize, _prtfmt) \ static ssize_t zfcp_sysfs_adapter_diag_sfp_##_name##_show( \ struct device *dev, struct device_attribute *attr, char *buf) \
@@ -733,6 +772,7 @@ static struct attribute *zfcp_sysfs_diag_attrs[] = { &dev_attr_adapter_diag_sfp_sfp_invalid.attr, &dev_attr_adapter_diag_sfp_connector_type.attr, &dev_attr_adapter_diag_sfp_fec_active.attr, + &dev_attr_adapter_diag_b2b_credit.attr, NULL, };
Signature · 1 line
2.21.0
PATCH v2 8/11 zfcp: implicitly refresh config-data diagnostics when reading SysFS
In reply to Benjamin BlockAdds implicit updates of cached diagnostics via Exchange Config Data when reading SysFS attributes interfacing them. Right now this only affects the new B2B-Credit diagnostic attribute.
This uses the same mechanism previously also used for cached diagnostics of Exchange Port Data.
--- drivers/s390/scsi/zfcp_diag.c | 30 ++++++++++++++++++++++++++++++ drivers/s390/scsi/zfcp_diag.h | 1 + drivers/s390/scsi/zfcp_sysfs.c | 5 +++++ 3 files changed, 36 insertions(+)
drivers/s390/scsi/zfcp_diag.c +30 -0
@@ -176,6 +176,36 @@ int zfcp_diag_update_port_data_buffer(struct zfcp_adapter *const adapter)return rc; } +/** +* zfcp_diag_update_config_data_buffer() - Implementation of +* &typedef zfcp_diag_update_buffer_func +* to collect and update Config Data. +* @adapter: Adapter to collect Config Data from. +* +* This call is SYNCHRONOUS ! It blocks till the respective command has +* finished completely, or has failed in some way. +* +* Return: +* * 0 - Successfully retrieved new Diagnostics and Updated the buffer; +* this also includes cases where data was retrieved, but +* incomplete; you'll have to check the flag ``incomplete`` +* of &struct zfcp_diag_header. +* * see zfcp_fsf_exchange_config_data_sync() for possible error-codes ( +* excluding -EAGAIN) +*/ +int zfcp_diag_update_config_data_buffer (struct zfcp_adapter *const adapter) +{ +int rc; + + rc =zfcp_fsf_exchange_config_data_sync (adapter->qdio,NULL ); +if (rc == -EAGAIN ) + rc =0 ;/* signaling incomplete via struct zfcp_diag_header */ + +/* buffer-data was updated in zfcp_fsf_exchange_config_data_handler() */ + +return rc; +} + static int __zfcp_diag_update_buffer(struct zfcp_adapter *const adapter, struct zfcp_diag_header *const hdr, zfcp_diag_update_buffer_func buffer_update,
drivers/s390/scsi/zfcp_diag.h +1 -0
@@ -77,6 +77,7 @@ void zfcp_diag_update_xdata(struct zfcp_diag_header *const hdr,*/ typedef int (*zfcp_diag_update_buffer_func )(struct zfcp_adapter *const adapter); +int zfcp_diag_update_config_data_buffer (struct zfcp_adapter *const adapter);int zfcp_diag_update_port_data_buffer (struct zfcp_adapter *const adapter); int zfcp_diag_update_buffer_limited(struct zfcp_adapter *const adapter, struct zfcp_diag_header *const hdr,
drivers/s390/scsi/zfcp_sysfs.c +5 -0
@@ -687,6 +687,11 @@ static ssize_t zfcp_sysfs_adapter_diag_b2b_credit_show( diag_hdr = &adapter->diagnostics->config_data.header; + rc =zfcp_diag_update_buffer_limited ( + adapter, diag_hdr, zfcp_diag_update_config_data_buffer); +if (rc !=0 ) + goto out; +spin_lock_irqsave (&diag_hdr->access_lock, flags);/* nport_serv_param doesn't contain the ELS_Command code */ nsp = (struct fc_els_flogi *)((unsigned long)
Signature · 1 line
2.21.0
PATCH v2 9/11 zfcp: move maximum age of diagnostic buffers into a per-adapter variable
In reply to Benjamin BlockReplace the static define (ZFCP_DIAG_MAX_AGE) with a per-adapter variable (${adapter}->diagnostics->max_age). This new variable is exported via SysFS, along with other, already existing adapter variables, and can both be read and written. This way users can choose how much time should pass between refreshes of diagnostic buffers. The default value for the age remains to be five seconds.
By setting this new variable to 0, the caching of diagnostic buffers for userspace accesses can also be completely removed.
All diagnostic buffers of a given adapter are subject to this setting in the same way.
--- drivers/s390/scsi/zfcp_diag.c | 23 ++++++++--------- drivers/s390/scsi/zfcp_diag.h | 4 +++ drivers/s390/scsi/zfcp_sysfs.c | 45 ++++++++++++++++++++++++++++++++++ 3 files changed, 61 insertions(+), 11 deletions(-)
drivers/s390/scsi/zfcp_diag.c +12 -11
@@ -19,9 +19,6 @@#include "zfcp_ext.h" #include "zfcp_def.h" -/* Max age of data in a diagnostics buffer before it needs a refresh (in ms). */ -#define ZFCP_DIAG_MAX_AGE (5 * 1000) -static DECLARE_WAIT_QUEUE_HEAD (__zfcp_diag_publish_wait);/**
@@ -38,9 +35,6 @@ static DECLARE_WAIT_QUEUE_HEAD(__zfcp_diag_publish_wait);*/ int zfcp_diag_adapter_setup (struct zfcp_adapter *const adapter) { -/* set the timestamp so that the first test on age will always fail */ -const unsigned long initial_timestamp = - jiffies -msecs_to_jiffies (ZFCP_DIAG_MAX_AGE );struct zfcp_diag_adapter *diag;struct zfcp_diag_header *hdr;
@@ -48,13 +42,16 @@ int zfcp_diag_adapter_setup(struct zfcp_adapter *const adapter)if (diag ==NULL )return -ENOMEM ; + diag->max_age = (5 *1000 );/* default value: 5 s */ +/* setup header for port_data */ hdr = &diag->port_data.header;spin_lock_init (&hdr->access_lock); hdr->buffer = &diag->port_data.data; hdr->buffer_size =sizeof (diag->port_data.data); - hdr->timestamp = initial_timestamp; +/* set the timestamp so that the first test on age will always fail */ + hdr->timestamp = jiffies -msecs_to_jiffies (diag->max_age);/* setup header for config_data */ hdr = &diag->config_data.header;
@@ -62,7 +59,8 @@ int zfcp_diag_adapter_setup(struct zfcp_adapter *const adapter)spin_lock_init (&hdr->access_lock); hdr->buffer = &diag->config_data.data; hdr->buffer_size =sizeof (diag->config_data.data); - hdr->timestamp = initial_timestamp; +/* set the timestamp so that the first test on age will always fail */ + hdr->timestamp = jiffies -msecs_to_jiffies (diag->max_age); adapter->diagnostics = diag;return 0 ;
@@ -240,7 +238,8 @@ static int __zfcp_diag_update_buffer(struct zfcp_adapter *const adapter, }static bool -__zfcp_diag_test_buffer_age_isfresh (const struct zfcp_diag_header *const hdr) +__zfcp_diag_test_buffer_age_isfresh (const struct zfcp_diag_adapter *const diag, +const struct zfcp_diag_header *const hdr) __must_hold(hdr->access_lock) {const unsigned long now = jiffies;
@@ -252,7 +251,7 @@ __zfcp_diag_test_buffer_age_isfresh(const struct zfcp_diag_header *const hdr)if (!time_after_eq (now, hdr->timestamp))return false; -if (jiffies_to_msecs (now - hdr->timestamp) >=ZFCP_DIAG_MAX_AGE ) +if (jiffies_to_msecs (now - hdr->timestamp) >= diag->max_age)return false;return true;
@@ -291,7 +290,9 @@ int zfcp_diag_update_buffer_limited(struct zfcp_adapter *const adapter,spin_lock_irqsave (&hdr->access_lock, flags); -for (rc =0 ; !__zfcp_diag_test_buffer_age_isfresh (hdr); rc =0 ) { +for (rc =0 ; + !__zfcp_diag_test_buffer_age_isfresh (adapter->diagnostics, hdr); + rc =0 ) { rc =__zfcp_diag_update_buffer (adapter, hdr, buffer_update, &flags);if (rc != -EAGAIN )
drivers/s390/scsi/zfcp_diag.h +4 -0
@@ -42,6 +42,8 @@ struct zfcp_diag_header {* adapter. * @sysfs_established: flag showing that the associated sysfs-group was created * during run of zfcp_adapter_enqueue(). +* @max_age: maximum age of data in diagnostic buffers before they need to be +* refreshed (in ms). * @port_data: data retrieved using exchange port data. * @port_data.header: header with metadata for the cache in @port_data.data. * @port_data.data: cached QTCB Bottom of command exchange port data.
@@ -52,6 +54,8 @@ struct zfcp_diag_header {struct zfcp_diag_adapter {u64 sysfs_established :1 ; +unsigned long max_age; +struct {struct zfcp_diag_header header;struct fsf_qtcb_bottom_port data;
drivers/s390/scsi/zfcp_sysfs.c +45 -0
57 lines not on this page · raw message
Signature · 1 line
2.21.0
PATCH v2 10/11 zfcp: proper indentation to reduce confusion in zfcp_erp_required_act
In reply to Benjamin BlockFrom: Steffen Maier <[email protected]>
No functional change.
The unary not operator only applies to the sub expression before the logical or. So we return early if (not running) or failed.
--- drivers/s390/scsi/zfcp_erp.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/s390/scsi/zfcp_erp.c +2 -2
@@ -174,7 +174,7 @@ static enum zfcp_erp_act_type zfcp_erp_required_act(enum zfcp_erp_act_type want,return 0 ; p_status =atomic_read (&port->status);if (!(p_status &ZFCP_STATUS_COMMON_RUNNING ) || - p_status &ZFCP_STATUS_COMMON_ERP_FAILED ) + p_status &ZFCP_STATUS_COMMON_ERP_FAILED )return 0 ;if (!(p_status &ZFCP_STATUS_COMMON_UNBLOCKED )) need =ZFCP_ERP_ACTION_REOPEN_PORT ;
@@ -190,7 +190,7 @@ static enum zfcp_erp_act_type zfcp_erp_required_act(enum zfcp_erp_act_type want,return 0 ; a_status =atomic_read (&adapter->status);if (!(a_status &ZFCP_STATUS_COMMON_RUNNING ) || - a_status &ZFCP_STATUS_COMMON_ERP_FAILED ) + a_status &ZFCP_STATUS_COMMON_ERP_FAILED )return 0 ;if (p_status &ZFCP_STATUS_COMMON_NOESC )return need;
Signature · 1 line
2.21.0
PATCH v2 11/11 zfcp: trace channel log even for FCP command responses
In reply to Benjamin BlockFrom: Steffen Maier <[email protected]>
While v2.6.26 commit b75db73159cc
("[SCSI] zfcp: Add qtcb dump to hba debug trace")
is right that we don't want to flood the (payload) trace ring buffer, we don't trace successful FCP command responses by default. So we can include the channel log for problem determination with failed responses of any FSF request type.
--- drivers/s390/scsi/zfcp_dbf.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
drivers/s390/scsi/zfcp_dbf.c +3 -5
@@ -95,11 +95,9 @@ void zfcp_dbf_hba_fsf_res(char *tag, int level, struct zfcp_fsf_req *req)memcpy (rec->u.res.fsf_status_qual, &q_head->fsf_status_qual,FSF_STATUS_QUALIFIER_SIZE ); -if (q_head->fsf_command !=FSF_QTCB_FCP_CMND ) { - rec->pl_len = q_head->log_length; -zfcp_dbf_pl_write (dbf, (char *)q_pref + q_head->log_start, - rec->pl_len,"fsf_res" , req->req_id); - } + rec->pl_len = q_head->log_length; +zfcp_dbf_pl_write (dbf, (char *)q_pref + q_head->log_start, + rec->pl_len,"fsf_res" , req->req_id);debug_event (dbf->hba, level, rec,sizeof (*rec));spin_unlock_irqrestore (&dbf->hba_lock, flags);
Signature · 1 line
2.21.0
Martin K. Petersen
AppliedBenjamin,
Quoted 3 lines
this is the second version of my RDP patchset for zfcp, after I noticed a memory-leak in the first version earlier this year. Here is the original description, which remains valid:
Applied to 5.5/scsi-queue, thanks!
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
Quoted 9 lines · Martin K. Petersen
Benjamin,
Quoted 3 lines
this is the second version of my RDP patchset for zfcp, after I noticed a memory-leak in the first version earlier this year. Here is the original description, which remains valid:
Applied to 5.5/scsi-queue, thanks!
Thanks, Martin.
Signature · 4 lines
With Best Regards, Benjamin Block / Linux on IBM Z Kernel Development IBM Systems & Technology Group / IBM Deutschland Research & Development GmbH Vorsitz. AufsR.: Matthias Hartmann / Gesch�ftsf�hrung: Dirk Wittkopp Sitz der Gesellschaft: B�blingen / Registergericht: AmtsG Stuttgart, HRB 243294